From patchwork Thu Feb 17 00:22:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12749178 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD695C433FE for ; Thu, 17 Feb 2022 00:22:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60BA86B007B; Wed, 16 Feb 2022 19:22:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BCCB6B007D; Wed, 16 Feb 2022 19:22:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45D056B007E; Wed, 16 Feb 2022 19:22:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0211.hostedemail.com [216.40.44.211]) by kanga.kvack.org (Postfix) with ESMTP id 36C696B007B for ; Wed, 16 Feb 2022 19:22:51 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E4B01181AC9C6 for ; Thu, 17 Feb 2022 00:22:50 +0000 (UTC) X-FDA: 79150371300.13.446D363 Received: from outbound-smtp20.blacknight.com (outbound-smtp20.blacknight.com [46.22.139.247]) by imf10.hostedemail.com (Postfix) with ESMTP id 6552FC0003 for ; Thu, 17 Feb 2022 00:22:50 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp20.blacknight.com (Postfix) with ESMTPS id 097861C4249 for ; Thu, 17 Feb 2022 00:22:49 +0000 (GMT) Received: (qmail 22567 invoked from network); 17 Feb 2022 00:22:48 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.223]) by 81.17.254.9 with ESMTPA; 17 Feb 2022 00:22:48 -0000 From: Mel Gorman To: Andrew Morton Cc: Aaron Lu , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM , Mel Gorman Subject: [PATCH 1/6] mm/page_alloc: Fetch the correct pcp buddy during bulk free Date: Thu, 17 Feb 2022 00:22:22 +0000 Message-Id: <20220217002227.5739-2-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220217002227.5739-1-mgorman@techsingularity.net> References: <20220217002227.5739-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 6552FC0003 X-Stat-Signature: sg73ug1foj9qwtnz4tciaemr6kib6cys X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf10.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.139.247 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-HE-Tag: 1645057370-585471 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pcppages_bulk() prefetches buddies about to be freed but the order must also be passed in as PCP lists store multiple orders. Fixes: 44042b449872 ("mm/page_alloc: allow high-order pages to be stored on the per-cpu lists") Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka Reviewed-by: Aaron Lu --- mm/page_alloc.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3589febc6d31..08de32cfd9bb 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1432,10 +1432,10 @@ static bool bulkfree_pcp_prepare(struct page *page) } #endif /* CONFIG_DEBUG_VM */ -static inline void prefetch_buddy(struct page *page) +static inline void prefetch_buddy(struct page *page, unsigned int order) { unsigned long pfn = page_to_pfn(page); - unsigned long buddy_pfn = __find_buddy_pfn(pfn, 0); + unsigned long buddy_pfn = __find_buddy_pfn(pfn, order); struct page *buddy = page + (buddy_pfn - pfn); prefetch(buddy); @@ -1512,7 +1512,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, * prefetch buddy for the first pcp->batch nr of pages. */ if (prefetch_nr) { - prefetch_buddy(page); + prefetch_buddy(page, order); prefetch_nr--; } } while (count > 0 && --batch_free && !list_empty(list)); From patchwork Thu Feb 17 00:22:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12749179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9003FC433EF for ; Thu, 17 Feb 2022 00:23:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C5CF6B007D; Wed, 16 Feb 2022 19:23:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 175276B007E; Wed, 16 Feb 2022 19:23:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03D7F6B0080; Wed, 16 Feb 2022 19:23:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id E8A796B007D for ; Wed, 16 Feb 2022 19:23:00 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id BC0D4810B6 for ; Thu, 17 Feb 2022 00:23:00 +0000 (UTC) X-FDA: 79150371720.04.BA43205 Received: from outbound-smtp04.blacknight.com (outbound-smtp04.blacknight.com [81.17.249.35]) by imf12.hostedemail.com (Postfix) with ESMTP id 4BB1440004 for ; Thu, 17 Feb 2022 00:23:00 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp04.blacknight.com (Postfix) with ESMTPS id 3A0C5EC081 for ; Thu, 17 Feb 2022 00:22:59 +0000 (GMT) Received: (qmail 22882 invoked from network); 17 Feb 2022 00:22:58 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.223]) by 81.17.254.9 with ESMTPA; 17 Feb 2022 00:22:58 -0000 From: Mel Gorman To: Andrew Morton Cc: Aaron Lu , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM , Mel Gorman Subject: [PATCH 2/6] mm/page_alloc: Track range of active PCP lists during bulk free Date: Thu, 17 Feb 2022 00:22:23 +0000 Message-Id: <20220217002227.5739-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220217002227.5739-1-mgorman@techsingularity.net> References: <20220217002227.5739-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 4BB1440004 X-Stat-Signature: bh5hazdkyniuk9s5hqquoz1b6zorr9pc Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of mgorman@techsingularity.net designates 81.17.249.35 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none X-HE-Tag: 1645057380-276565 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pcppages_bulk() frees pages in a round-robin fashion. Originally, this was dealing only with migratetypes but storing high-order pages means that there can be many more empty lists that are uselessly checked. Track the minimum and maximum active pindex to reduce the search space. Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 08de32cfd9bb..85cc1fe8bcc5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1450,6 +1450,8 @@ static void free_pcppages_bulk(struct zone *zone, int count, struct per_cpu_pages *pcp) { int pindex = 0; + int min_pindex = 0; + int max_pindex = NR_PCP_LISTS - 1; int batch_free = 0; int nr_freed = 0; unsigned int order; @@ -1475,13 +1477,20 @@ static void free_pcppages_bulk(struct zone *zone, int count, */ do { batch_free++; - if (++pindex == NR_PCP_LISTS) - pindex = 0; + if (++pindex > max_pindex) + pindex = min_pindex; list = &pcp->lists[pindex]; - } while (list_empty(list)); + if (!list_empty(list)) + break; + + if (pindex == max_pindex) + max_pindex--; + if (pindex == min_pindex) + min_pindex++; + } while (1); /* This is the only non-empty list. Free them all. */ - if (batch_free == NR_PCP_LISTS) + if (batch_free >= max_pindex - min_pindex) batch_free = count; order = pindex_to_order(pindex); From patchwork Thu Feb 17 00:22:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12749180 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37793C433F5 for ; Thu, 17 Feb 2022 00:23:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB1526B007E; Wed, 16 Feb 2022 19:23:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B5FCE6B0080; Wed, 16 Feb 2022 19:23:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A27D36B0081; Wed, 16 Feb 2022 19:23:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id 940A76B007E for ; Wed, 16 Feb 2022 19:23:11 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5A4EC9094F for ; Thu, 17 Feb 2022 00:23:11 +0000 (UTC) X-FDA: 79150372182.24.C4B2BD3 Received: from outbound-smtp29.blacknight.com (outbound-smtp29.blacknight.com [81.17.249.32]) by imf13.hostedemail.com (Postfix) with ESMTP id E63C820007 for ; Thu, 17 Feb 2022 00:23:10 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp29.blacknight.com (Postfix) with ESMTPS id C09F918E02E for ; Thu, 17 Feb 2022 00:23:09 +0000 (GMT) Received: (qmail 23170 invoked from network); 17 Feb 2022 00:23:09 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.223]) by 81.17.254.9 with ESMTPA; 17 Feb 2022 00:23:09 -0000 From: Mel Gorman To: Andrew Morton Cc: Aaron Lu , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM , Mel Gorman Subject: [PATCH 3/6] mm/page_alloc: Simplify how many pages are selected per pcp list during bulk free Date: Thu, 17 Feb 2022 00:22:24 +0000 Message-Id: <20220217002227.5739-4-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220217002227.5739-1-mgorman@techsingularity.net> References: <20220217002227.5739-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspamd-Queue-Id: E63C820007 X-Rspam-User: Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of mgorman@techsingularity.net designates 81.17.249.32 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none X-Stat-Signature: kw64huqz34oju68n1hu4g944tr38t84t X-Rspamd-Server: rspam03 X-HE-Tag: 1645057390-156325 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pcppages_bulk() selects pages to free by round-robining between lists. Originally this was to evenly shrink pages by migratetype but uneven freeing is inevitable due to high pages. Simplify list selection by starting with a list that definitely has pages on it in free_unref_page_commit() and for drain, it does not matter where draining starts as all pages are removed. Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 34 +++++++++++----------------------- 1 file changed, 11 insertions(+), 23 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 85cc1fe8bcc5..dfc347a58ea6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1447,13 +1447,11 @@ static inline void prefetch_buddy(struct page *page, unsigned int order) * count is the number of pages to free. */ static void free_pcppages_bulk(struct zone *zone, int count, - struct per_cpu_pages *pcp) + struct per_cpu_pages *pcp, + int pindex) { - int pindex = 0; int min_pindex = 0; int max_pindex = NR_PCP_LISTS - 1; - int batch_free = 0; - int nr_freed = 0; unsigned int order; int prefetch_nr = READ_ONCE(pcp->batch); bool isolated_pageblocks; @@ -1467,16 +1465,10 @@ static void free_pcppages_bulk(struct zone *zone, int count, count = min(pcp->count, count); while (count > 0) { struct list_head *list; + int nr_pages; - /* - * Remove pages from lists in a round-robin fashion. A - * batch_free count is maintained that is incremented when an - * empty list is encountered. This is so more pages are freed - * off fuller lists instead of spinning excessively around empty - * lists - */ + /* Remove pages from lists in a round-robin fashion. */ do { - batch_free++; if (++pindex > max_pindex) pindex = min_pindex; list = &pcp->lists[pindex]; @@ -1489,18 +1481,15 @@ static void free_pcppages_bulk(struct zone *zone, int count, min_pindex++; } while (1); - /* This is the only non-empty list. Free them all. */ - if (batch_free >= max_pindex - min_pindex) - batch_free = count; - order = pindex_to_order(pindex); + nr_pages = 1 << order; BUILD_BUG_ON(MAX_ORDER >= (1<lru); - nr_freed += 1 << order; - count -= 1 << order; + count -= nr_pages; + pcp->count -= nr_pages; if (bulkfree_pcp_prepare(page)) continue; @@ -1524,9 +1513,8 @@ static void free_pcppages_bulk(struct zone *zone, int count, prefetch_buddy(page, order); prefetch_nr--; } - } while (count > 0 && --batch_free && !list_empty(list)); + } while (count > 0 && !list_empty(list)); } - pcp->count -= nr_freed; /* * local_lock_irq held so equivalent to spin_lock_irqsave for @@ -3095,7 +3083,7 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) batch = READ_ONCE(pcp->batch); to_drain = min(pcp->count, batch); if (to_drain > 0) - free_pcppages_bulk(zone, to_drain, pcp); + free_pcppages_bulk(zone, to_drain, pcp, 0); local_unlock_irqrestore(&pagesets.lock, flags); } #endif @@ -3116,7 +3104,7 @@ static void drain_pages_zone(unsigned int cpu, struct zone *zone) pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); if (pcp->count) - free_pcppages_bulk(zone, pcp->count, pcp); + free_pcppages_bulk(zone, pcp->count, pcp, 0); local_unlock_irqrestore(&pagesets.lock, flags); } @@ -3397,7 +3385,7 @@ static void free_unref_page_commit(struct page *page, unsigned long pfn, if (pcp->count >= high) { int batch = READ_ONCE(pcp->batch); - free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp); + free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp, pindex); } } From patchwork Thu Feb 17 00:22:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12749181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E358FC433EF for ; Thu, 17 Feb 2022 00:23:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 792776B0080; Wed, 16 Feb 2022 19:23:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 741426B0081; Wed, 16 Feb 2022 19:23:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 608936B0082; Wed, 16 Feb 2022 19:23:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0131.hostedemail.com [216.40.44.131]) by kanga.kvack.org (Postfix) with ESMTP id 4F8CF6B0080 for ; Wed, 16 Feb 2022 19:23:22 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1967B8249980 for ; Thu, 17 Feb 2022 00:23:22 +0000 (UTC) X-FDA: 79150372644.19.5F6E789 Received: from outbound-smtp22.blacknight.com (outbound-smtp22.blacknight.com [81.17.249.190]) by imf07.hostedemail.com (Postfix) with ESMTP id 90EC040004 for ; Thu, 17 Feb 2022 00:23:21 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp22.blacknight.com (Postfix) with ESMTPS id 771ABBACE1 for ; Thu, 17 Feb 2022 00:23:20 +0000 (GMT) Received: (qmail 23542 invoked from network); 17 Feb 2022 00:23:20 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.223]) by 81.17.254.9 with ESMTPA; 17 Feb 2022 00:23:20 -0000 From: Mel Gorman To: Andrew Morton Cc: Aaron Lu , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM , Mel Gorman Subject: [PATCH 4/6] mm/page_alloc: Drain the requested list first during bulk free Date: Thu, 17 Feb 2022 00:22:25 +0000 Message-Id: <20220217002227.5739-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220217002227.5739-1-mgorman@techsingularity.net> References: <20220217002227.5739-1-mgorman@techsingularity.net> MIME-Version: 1.0 Authentication-Results: imf07.hostedemail.com; dkim=none; spf=pass (imf07.hostedemail.com: domain of mgorman@techsingularity.net designates 81.17.249.190 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none X-Rspamd-Server: rspam07 X-Rspam-User: X-Rspamd-Queue-Id: 90EC040004 X-Stat-Signature: a9zxd1gi5q99yahsmzqz9dy1ttjz8i88 X-HE-Tag: 1645057401-196001 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Prior to the series, pindex 0 (order-0 MIGRATE_UNMOVABLE) was always skipped first and the precise reason is forgotten. A potential reason may have been to artificially preserve MIGRATE_UNMOVABLE but there is no reason why that would be optimal as it depends on the workload. The more likely reason is that it was less complicated to do a pre-increment instead of a post-increment in terms of overall code flow. As free_pcppages_bulk() now typically receives the pindex of the PCP list that exceeded high, always start draining that list. Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index dfc347a58ea6..635a4e0f70b4 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1463,6 +1463,10 @@ static void free_pcppages_bulk(struct zone *zone, int count, * below while (list_empty(list)) loop. */ count = min(pcp->count, count); + + /* Ensure requested pindex is drained first. */ + pindex = pindex - 1; + while (count > 0) { struct list_head *list; int nr_pages; From patchwork Thu Feb 17 00:22:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12749182 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0BE8C433EF for ; Thu, 17 Feb 2022 00:23:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 483EF6B0081; Wed, 16 Feb 2022 19:23:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 434CE6B0082; Wed, 16 Feb 2022 19:23:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2FB3B6B0083; Wed, 16 Feb 2022 19:23:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 1C8396B0081 for ; Wed, 16 Feb 2022 19:23:33 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D5398181AC9C6 for ; Thu, 17 Feb 2022 00:23:32 +0000 (UTC) X-FDA: 79150373064.15.DE49831 Received: from outbound-smtp51.blacknight.com (outbound-smtp51.blacknight.com [46.22.136.235]) by imf20.hostedemail.com (Postfix) with ESMTP id 329C31C0005 for ; Thu, 17 Feb 2022 00:23:32 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp51.blacknight.com (Postfix) with ESMTPS id E4155FAE22 for ; Thu, 17 Feb 2022 00:23:30 +0000 (GMT) Received: (qmail 23898 invoked from network); 17 Feb 2022 00:23:30 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.223]) by 81.17.254.9 with ESMTPA; 17 Feb 2022 00:23:30 -0000 From: Mel Gorman To: Andrew Morton Cc: Aaron Lu , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM , Mel Gorman Subject: [PATCH 5/6] mm/page_alloc: Free pages in a single pass during bulk free Date: Thu, 17 Feb 2022 00:22:26 +0000 Message-Id: <20220217002227.5739-6-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220217002227.5739-1-mgorman@techsingularity.net> References: <20220217002227.5739-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 329C31C0005 X-Stat-Signature: 9zepr6ui1g4guxpuj9jm3mpkwppfy59z X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf20.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.136.235 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net X-HE-Tag: 1645057412-33009 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_pcppages_bulk() has taken two passes through the pcp lists since commit 0a5f4e5b4562 ("mm/free_pcppages_bulk: do not hold lock when picking pages to free") due to deferring the cost of selecting PCP lists until the zone lock is held. Now that list selection is simplier, the main cost during selection is bulkfree_pcp_prepare() which in the normal case is a simple check and prefetching. As the list manipulations have cost in itself, go back to freeing pages in a single pass. The series up to this point was evaulated using a trunc microbenchmark that is truncating sparse files stored in page cache (mmtests config config-io-trunc). Sparse files were used to limit filesystem interaction. The results versus a revert of storing high-order pages in the PCP lists is 1-socket Skylake 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3 vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2 Min elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%) Amean elapsed 543.00 ( 0.00%) 530.00 * 2.39%* 530.00 * 2.39%* Stddev elapsed 4.83 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%) CoeffVar elapsed 0.89 ( 0.00%) 0.00 ( 100.00%) 0.00 ( 100.00%) Max elapsed 550.00 ( 0.00%) 530.00 ( 3.64%) 530.00 ( 3.64%) BAmean-50 elapsed 540.00 ( 0.00%) 530.00 ( 1.85%) 530.00 ( 1.85%) BAmean-95 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%) BAmean-99 elapsed 542.22 ( 0.00%) 530.00 ( 2.25%) 530.00 ( 2.25%) 2-socket CascadeLake 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3 vanilla mm-reverthighpcp-v1 mm-highpcpopt-v2 Min elapsed 510.00 ( 0.00%) 500.00 ( 1.96%) 500.00 ( 1.96%) Amean elapsed 529.00 ( 0.00%) 521.00 ( 1.51%) 510.00 * 3.59%* Stddev elapsed 16.63 ( 0.00%) 12.87 ( 22.64%) 11.55 ( 30.58%) CoeffVar elapsed 3.14 ( 0.00%) 2.47 ( 21.46%) 2.26 ( 27.99%) Max elapsed 550.00 ( 0.00%) 540.00 ( 1.82%) 530.00 ( 3.64%) BAmean-50 elapsed 516.00 ( 0.00%) 512.00 ( 0.78%) 500.00 ( 3.10%) BAmean-95 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%) BAmean-99 elapsed 526.67 ( 0.00%) 518.89 ( 1.48%) 507.78 ( 3.59%) The original motivation for multi-passes was will-it-scale page_fault1 using $nr_cpu processes. 2-socket CascadeLake (40 cores, 80 CPUs HT enabled) 5.17.0-rc3 5.17.0-rc3 vanilla mm-highpcpopt-v2 Hmean page_fault1-processes-2 2694662.26 ( 0.00%) 2695780.35 ( 0.04%) Hmean page_fault1-processes-5 6425819.34 ( 0.00%) 6435544.57 * 0.15%* Hmean page_fault1-processes-8 9642169.10 ( 0.00%) 9658962.39 ( 0.17%) Hmean page_fault1-processes-12 12167502.10 ( 0.00%) 12190163.79 ( 0.19%) Hmean page_fault1-processes-21 15636859.03 ( 0.00%) 15612447.26 ( -0.16%) Hmean page_fault1-processes-30 25157348.61 ( 0.00%) 25169456.65 ( 0.05%) Hmean page_fault1-processes-48 27694013.85 ( 0.00%) 27671111.46 ( -0.08%) Hmean page_fault1-processes-79 25928742.64 ( 0.00%) 25934202.02 ( 0.02%) <-- Hmean page_fault1-processes-110 25730869.75 ( 0.00%) 25671880.65 * -0.23%* Hmean page_fault1-processes-141 25626992.42 ( 0.00%) 25629551.61 ( 0.01%) Hmean page_fault1-processes-172 25611651.35 ( 0.00%) 25614927.99 ( 0.01%) Hmean page_fault1-processes-203 25577298.75 ( 0.00%) 25583445.59 ( 0.02%) Hmean page_fault1-processes-234 25580686.07 ( 0.00%) 25608240.71 ( 0.11%) Hmean page_fault1-processes-265 25570215.47 ( 0.00%) 25568647.58 ( -0.01%) Hmean page_fault1-processes-296 25549488.62 ( 0.00%) 25543935.00 ( -0.02%) Hmean page_fault1-processes-320 25555149.05 ( 0.00%) 25575696.74 ( 0.08%) The differences are mostly within the noise and the difference close to $nr_cpus is negligible. Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka Signed-off-by: Mel Gorman --- mm/page_alloc.c | 56 +++++++++++++++++++------------------------------ 1 file changed, 21 insertions(+), 35 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 635a4e0f70b4..68e2132717c5 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1455,8 +1455,7 @@ static void free_pcppages_bulk(struct zone *zone, int count, unsigned int order; int prefetch_nr = READ_ONCE(pcp->batch); bool isolated_pageblocks; - struct page *page, *tmp; - LIST_HEAD(head); + struct page *page; /* * Ensure proper count is passed which otherwise would stuck in the @@ -1467,6 +1466,13 @@ static void free_pcppages_bulk(struct zone *zone, int count, /* Ensure requested pindex is drained first. */ pindex = pindex - 1; + /* + * local_lock_irq held so equivalent to spin_lock_irqsave for + * both PREEMPT_RT and non-PREEMPT_RT configurations. + */ + spin_lock(&zone->lock); + isolated_pageblocks = has_isolate_pageblock(zone); + while (count > 0) { struct list_head *list; int nr_pages; @@ -1489,7 +1495,11 @@ static void free_pcppages_bulk(struct zone *zone, int count, nr_pages = 1 << order; BUILD_BUG_ON(MAX_ORDER >= (1<lru); count -= nr_pages; @@ -1498,12 +1508,6 @@ static void free_pcppages_bulk(struct zone *zone, int count, if (bulkfree_pcp_prepare(page)) continue; - /* Encode order with the migratetype */ - page->index <<= NR_PCP_ORDER_WIDTH; - page->index |= order; - - list_add_tail(&page->lru, &head); - /* * We are going to put the page back to the global * pool, prefetch its buddy to speed up later access @@ -1517,36 +1521,18 @@ static void free_pcppages_bulk(struct zone *zone, int count, prefetch_buddy(page, order); prefetch_nr--; } - } while (count > 0 && !list_empty(list)); - } - /* - * local_lock_irq held so equivalent to spin_lock_irqsave for - * both PREEMPT_RT and non-PREEMPT_RT configurations. - */ - spin_lock(&zone->lock); - isolated_pageblocks = has_isolate_pageblock(zone); + /* MIGRATE_ISOLATE page should not go to pcplists */ + VM_BUG_ON_PAGE(is_migrate_isolate(mt), page); + /* Pageblock could have been isolated meanwhile */ + if (unlikely(isolated_pageblocks)) + mt = get_pageblock_migratetype(page); - /* - * Use safe version since after __free_one_page(), - * page->lru.next will not point to original list. - */ - list_for_each_entry_safe(page, tmp, &head, lru) { - int mt = get_pcppage_migratetype(page); - - /* mt has been encoded with the order (see above) */ - order = mt & NR_PCP_ORDER_MASK; - mt >>= NR_PCP_ORDER_WIDTH; - - /* MIGRATE_ISOLATE page should not go to pcplists */ - VM_BUG_ON_PAGE(is_migrate_isolate(mt), page); - /* Pageblock could have been isolated meanwhile */ - if (unlikely(isolated_pageblocks)) - mt = get_pageblock_migratetype(page); - - __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE); - trace_mm_page_pcpu_drain(page, order, mt); + __free_one_page(page, page_to_pfn(page), zone, order, mt, FPI_NONE); + trace_mm_page_pcpu_drain(page, order, mt); + } while (count > 0 && !list_empty(list)); } + spin_unlock(&zone->lock); } From patchwork Thu Feb 17 00:22:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12749183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 26840C433F5 for ; Thu, 17 Feb 2022 00:23:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A77D16B0082; Wed, 16 Feb 2022 19:23:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A26996B0083; Wed, 16 Feb 2022 19:23:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EF5C6B0085; Wed, 16 Feb 2022 19:23:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0161.hostedemail.com [216.40.44.161]) by kanga.kvack.org (Postfix) with ESMTP id 803076B0082 for ; Wed, 16 Feb 2022 19:23:43 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2D1F095C8C for ; Thu, 17 Feb 2022 00:23:43 +0000 (UTC) X-FDA: 79150373526.17.AE0E704 Received: from outbound-smtp52.blacknight.com (outbound-smtp52.blacknight.com [46.22.136.236]) by imf14.hostedemail.com (Postfix) with ESMTP id 84B3C100004 for ; Thu, 17 Feb 2022 00:23:42 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail05.blacknight.ie [81.17.254.26]) by outbound-smtp52.blacknight.com (Postfix) with ESMTPS id 6C41DFAE1A for ; Thu, 17 Feb 2022 00:23:41 +0000 (GMT) Received: (qmail 24173 invoked from network); 17 Feb 2022 00:23:41 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.17.223]) by 81.17.254.9 with ESMTPA; 17 Feb 2022 00:23:41 -0000 From: Mel Gorman To: Andrew Morton Cc: Aaron Lu , Dave Hansen , Vlastimil Babka , Michal Hocko , Jesper Dangaard Brouer , LKML , Linux-MM , Mel Gorman Subject: [PATCH 6/6] mm/page_alloc: Limit number of high-order pages on PCP during bulk free Date: Thu, 17 Feb 2022 00:22:27 +0000 Message-Id: <20220217002227.5739-7-mgorman@techsingularity.net> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220217002227.5739-1-mgorman@techsingularity.net> References: <20220217002227.5739-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspamd-Queue-Id: 84B3C100004 X-Rspam-User: Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.136.236 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none X-Stat-Signature: 6frg5tbayhqwpo5gygqwwkqb7rrfymua X-Rspamd-Server: rspam03 X-HE-Tag: 1645057422-266754 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a PCP is mostly used for frees then high-order pages can exist on PCP lists for some time. This is problematic when the allocation pattern is all allocations from one CPU and all frees from another resulting in colder pages being used. When bulk freeing pages, limit the number of high-order pages that are stored on the PCP lists. Netperf running on localhost exhibits this pattern and while it does not matter for some machines, it does matter for others with smaller caches where cache misses cause problems due to reduced page reuse. Pages freed directly to the buddy list may be reused quickly while still cache hot where as storing on the PCP lists may be cold by the time free_pcppages_bulk() is called. Using perf kmem:mm_page_alloc, the 5 most used page frames were 5.17-rc3 13041 pfn=0x111a30 13081 pfn=0x5814d0 13097 pfn=0x108258 13121 pfn=0x689598 13128 pfn=0x5814d8 5.17-revert-highpcp 192009 pfn=0x54c140 195426 pfn=0x1081d0 200908 pfn=0x61c808 243515 pfn=0xa9dc20 402523 pfn=0x222bb8 5.17-full-series 142693 pfn=0x346208 162227 pfn=0x13bf08 166413 pfn=0x2711e0 166950 pfn=0x2702f8 The spread is wider as there is still time before pages freed to one PCP get released with a tradeoff between fast reuse and reduced zone lock acquisition. From the machine used to gather the traces, the headline performance was equivalent. netperf-tcp 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3 vanilla mm-reverthighpcp-v1r1 mm-highpcplimit-v2 Hmean 64 839.93 ( 0.00%) 840.77 ( 0.10%) 841.02 ( 0.13%) Hmean 128 1614.22 ( 0.00%) 1622.07 * 0.49%* 1636.41 * 1.37%* Hmean 256 2952.00 ( 0.00%) 2953.19 ( 0.04%) 2977.76 * 0.87%* Hmean 1024 10291.67 ( 0.00%) 10239.17 ( -0.51%) 10434.41 * 1.39%* Hmean 2048 17335.08 ( 0.00%) 17399.97 ( 0.37%) 17134.81 * -1.16%* Hmean 3312 22628.15 ( 0.00%) 22471.97 ( -0.69%) 22422.78 ( -0.91%) Hmean 4096 25009.50 ( 0.00%) 24752.83 * -1.03%* 24740.41 ( -1.08%) Hmean 8192 32745.01 ( 0.00%) 31682.63 * -3.24%* 32153.50 * -1.81%* Hmean 16384 39759.59 ( 0.00%) 36805.78 * -7.43%* 38948.13 * -2.04%* From a 1-socket skylake machine with a small CPU cache that suffers more if cache misses are too high netperf-tcp 5.17.0-rc3 5.17.0-rc3 5.17.0-rc3 vanilla mm-reverthighpcp-v1 mm-highpcplimit-v2 Hmean 64 938.95 ( 0.00%) 941.50 * 0.27%* 943.61 * 0.50%* Hmean 128 1843.10 ( 0.00%) 1857.58 * 0.79%* 1861.09 * 0.98%* Hmean 256 3573.07 ( 0.00%) 3667.45 * 2.64%* 3674.91 * 2.85%* Hmean 1024 13206.52 ( 0.00%) 13487.80 * 2.13%* 13393.21 * 1.41%* Hmean 2048 22870.23 ( 0.00%) 23337.96 * 2.05%* 23188.41 * 1.39%* Hmean 3312 31001.99 ( 0.00%) 32206.50 * 3.89%* 31863.62 * 2.78%* Hmean 4096 35364.59 ( 0.00%) 36490.96 * 3.19%* 36112.54 * 2.11%* Hmean 8192 48497.71 ( 0.00%) 49954.05 * 3.00%* 49588.26 * 2.25%* Hmean 16384 58410.86 ( 0.00%) 60839.80 * 4.16%* 62282.96 * 6.63%* Note that this was a machine that did not benefit from caching high-order pages and performance is almost restored with the series applied. It's not fully restored as cache misses are still higher. This is a trade-off between optimising for a workload that does all allocs on one CPU and frees on another or more general workloads that need high-order pages for SLUB and benefit from avoiding zone->lock for every SLUB refill/drain. Signed-off-by: Mel Gorman Reviewed-by: Vlastimil Babka --- mm/page_alloc.c | 26 +++++++++++++++++++++----- 1 file changed, 21 insertions(+), 5 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 68e2132717c5..de9f072d23bd 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3317,10 +3317,15 @@ static bool free_unref_page_prepare(struct page *page, unsigned long pfn, return true; } -static int nr_pcp_free(struct per_cpu_pages *pcp, int high, int batch) +static int nr_pcp_free(struct per_cpu_pages *pcp, int high, int batch, + bool free_high) { int min_nr_free, max_nr_free; + /* Free everything if batch freeing high-order pages. */ + if (unlikely(free_high)) + return pcp->count; + /* Check for PCP disabled or boot pageset */ if (unlikely(high < batch)) return 1; @@ -3341,11 +3346,12 @@ static int nr_pcp_free(struct per_cpu_pages *pcp, int high, int batch) return batch; } -static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone) +static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, + bool free_high) { int high = READ_ONCE(pcp->high); - if (unlikely(!high)) + if (unlikely(!high || free_high)) return 0; if (!test_bit(ZONE_RECLAIM_ACTIVE, &zone->flags)) @@ -3365,17 +3371,27 @@ static void free_unref_page_commit(struct page *page, unsigned long pfn, struct per_cpu_pages *pcp; int high; int pindex; + bool free_high; __count_vm_event(PGFREE); pcp = this_cpu_ptr(zone->per_cpu_pageset); pindex = order_to_pindex(migratetype, order); list_add(&page->lru, &pcp->lists[pindex]); pcp->count += 1 << order; - high = nr_pcp_high(pcp, zone); + + /* + * As high-order pages other than THP's stored on PCP can contribute + * to fragmentation, limit the number stored when PCP is heavily + * freeing without allocation. The remainder after bulk freeing + * stops will be drained from vmstat refresh context. + */ + free_high = (pcp->free_factor && order && order <= PAGE_ALLOC_COSTLY_ORDER); + + high = nr_pcp_high(pcp, zone, free_high); if (pcp->count >= high) { int batch = READ_ONCE(pcp->batch); - free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp, pindex); + free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch, free_high), pcp, pindex); } }