From patchwork Thu Oct 14 08:02:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vasily Averin X-Patchwork-Id: 12557971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 27701C433EF for ; Thu, 14 Oct 2021 08:03:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 892C5610E8 for ; Thu, 14 Oct 2021 08:03:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 892C5610E8 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=virtuozzo.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 243E994002A; Thu, 14 Oct 2021 04:03:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F3D494001C; Thu, 14 Oct 2021 04:03:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E35D94002A; Thu, 14 Oct 2021 04:03:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id F262C94001C for ; Thu, 14 Oct 2021 04:03:23 -0400 (EDT) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B7B471824196E for ; Thu, 14 Oct 2021 08:03:23 +0000 (UTC) X-FDA: 78694303086.06.3D346D3 Received: from relay.sw.ru (relay.sw.ru [185.231.240.75]) by imf21.hostedemail.com (Postfix) with ESMTP id B1E9FD040567 for ; Thu, 14 Oct 2021 08:03:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=virtuozzo.com; s=relay; h=Content-Type:MIME-Version:Date:Message-ID:Subject :From; bh=ml/asmO2M55pRoF1Gh0wA+ZD2la4BPG3z6+NcpeHyVE=; b=d3eBUq3lpjOVTy9cmKM mpNWN6wrh9dFOEUw7oJMxHrTxIl4YT/DybLGVtK8YInhHntC56JRz95JAZCsDl5TBfPpll+nMXaqG eTiseE2yNAFYEekMwK+/bv4gp+QDyPVg7p/sMB5vtYLQGy8wsWmx0YEhCJGo9QlvBWYsAm39/io=; Received: from [172.29.1.17] by relay.sw.ru with esmtp (Exim 4.94.2) (envelope-from ) id 1mavhu-005z98-9a; Thu, 14 Oct 2021 11:03:18 +0300 From: Vasily Averin Subject: [PATCH mm v5] memcg: enable memory accounting in __alloc_pages_bulk To: Michal Hocko , Johannes Weiner , Vladimir Davydov , Andrew Morton , Shakeel Butt , Mel Gorman Cc: Roman Gushchin , Uladzislau Rezki , Vlastimil Babka , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, kernel@openvz.org References: <0baa2b26-a41b-acab-b75d-72ec241f5151@virtuozzo.com> Message-ID: <65c1afaf-7947-ce28-55b7-06bde7aeb278@virtuozzo.com> Date: Thu, 14 Oct 2021 11:02:57 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.13.0 MIME-Version: 1.0 In-Reply-To: <0baa2b26-a41b-acab-b75d-72ec241f5151@virtuozzo.com> Content-Language: en-US Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=virtuozzo.com header.s=relay header.b=d3eBUq3l; dmarc=pass (policy=quarantine) header.from=virtuozzo.com; spf=pass (imf21.hostedemail.com: domain of vvs@virtuozzo.com designates 185.231.240.75 as permitted sender) smtp.mailfrom=vvs@virtuozzo.com X-Stat-Signature: 56orajar8dxk7yn5uykmezj3g148qjrd X-Rspamd-Queue-Id: B1E9FD040567 X-Rspamd-Server: rspam01 X-HE-Tag: 1634198602-352071 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Bulk page allocator is used in vmalloc where it can be called with __GFP_ACCOUNT and must charge allocated pages into memory cgroup. Fixes: 387ba26fb1cb ("mm/page_alloc: add a bulk page allocator") Cc: Signed-off-by: Vasily Averin --- v5: remove pre-charge, use post-allocation per-page charges according to Michal Hocko's remarks v4: updated according to Shakeel Butt's remarks, fixed wrong location of hooks declaration in memcontrol.h v3: added comments, removed call of post charge hook for nr_pages = 0 v2: modified according to Shakeel Butt's remarks --- include/linux/memcontrol.h | 9 +++++++ mm/memcontrol.c | 50 ++++++++++++++++++++++++++++++++++++++ mm/page_alloc.c | 12 +++++++-- 3 files changed, 69 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 3096c9a0ee01..6bad3d6efd03 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1692,6 +1692,9 @@ static inline int memcg_cache_id(struct mem_cgroup *memcg) struct mem_cgroup *mem_cgroup_from_obj(void *p); +int memcg_charge_bulk_pages(gfp_t gfp, int nr_pages, + struct list_head *page_list, + struct page **page_array); #else static inline bool mem_cgroup_kmem_disabled(void) { @@ -1744,6 +1747,12 @@ static inline struct mem_cgroup *mem_cgroup_from_obj(void *p) return NULL; } +int memcg_charge_bulk_pages(gfp_t gfp, int nr_pages, + struct list_head *page_list, + struct page **page_array) +{ + return 0; +} #endif /* CONFIG_MEMCG_KMEM */ #endif /* _LINUX_MEMCONTROL_H */ diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 87e41c3cac10..568e594179f5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3239,6 +3239,56 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) refill_obj_stock(objcg, size, true); } +/* + * memcg_charge_bulk_pages - Charge pages allocated by bulk allocator + * @gfp: GFP flags for the allocation + * @nr_pages: The number of pages added into the list or array + * @page_list: Optional list of allocated pages + * @page_array: Optional array of allocated pages + * + * Walks through array or list of allocated pages. + * For each page tries to charge it. + * If charge fails removes page from of array/list, frees it, + * and repeat it till end of array/list + * + * Returns the number of freed pages. + */ +int memcg_charge_bulk_pages(gfp_t gfp, int nr_pages, + struct list_head *page_list, + struct page **page_array) +{ + struct page *page, *np = NULL; + bool charge = true; + int i, nr_freed = 0; + + if (page_list) + page = list_first_entry(page_list, struct page, lru); + + for (i = 0; i < nr_pages; i++) { + if (page_list) { + if (np) + page = np; + np = list_next_entry(page, lru); + } else { + page = page_array[i]; + } + /* some pages in incoming array can be charged already */ + if (!page->memcg_data) { + if (charge && __memcg_kmem_charge_page(page, gfp, 0)) + charge = false; + + if (!charge) { + if (page_list) + list_del(&page->lru); + else + page_array[i] = NULL; + __free_pages(page, 0); + nr_freed++; + } + } + } + return nr_freed; +} #endif /* CONFIG_MEMCG_KMEM */ /* diff --git a/mm/page_alloc.c b/mm/page_alloc.c index b37435c274cf..71c7f29ff8dc 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5203,10 +5203,11 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, struct zoneref *z; struct per_cpu_pages *pcp; struct list_head *pcp_list; + LIST_HEAD(tpl); struct alloc_context ac; gfp_t alloc_gfp; unsigned int alloc_flags = ALLOC_WMARK_LOW; - int nr_populated = 0, nr_account = 0; + int nr_populated = 0, nr_account = 0, nr_freed = 0; /* * Skip populated array elements to determine if any pages need @@ -5300,7 +5301,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, prep_new_page(page, 0, gfp, 0); if (page_list) - list_add(&page->lru, page_list); + list_add(&page->lru, &tpl); else page_array[nr_populated] = page; nr_populated++; @@ -5308,6 +5309,13 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, local_unlock_irqrestore(&pagesets.lock, flags); + if (memcg_kmem_enabled() && (gfp & __GFP_ACCOUNT) && nr_account) + nr_freed = memcg_charge_bulk_pages(gfp, nr_populated, + page_list ? &tpl : NULL, + page_array); + nr_account -= nr_freed; + nr_populated -= nr_freed; + list_splice(&tpl, page_list); __count_zid_vm_events(PGALLOC, zone_idx(zone), nr_account); zone_statistics(ac.preferred_zoneref->zone, zone, nr_account);