From patchwork Fri Oct 18 10:56:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 11198211 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2F5118A6 for ; Fri, 18 Oct 2019 10:56:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C0605222C9 for ; Fri, 18 Oct 2019 10:56:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C0605222C9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6724E8E0007; Fri, 18 Oct 2019 06:56:11 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5FADA8E0008; Fri, 18 Oct 2019 06:56:11 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42B498E0007; Fri, 18 Oct 2019 06:56:11 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0230.hostedemail.com [216.40.44.230]) by kanga.kvack.org (Postfix) with ESMTP id 01ACC8E0005 for ; Fri, 18 Oct 2019 06:56:10 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with SMTP id 897F05DD7 for ; Fri, 18 Oct 2019 10:56:10 +0000 (UTC) X-FDA: 76056600900.10.jail15_18c27e3305e31 X-Spam-Summary: 2,0,0,e4ae2ac8383e7a2e,d41d8cd98f00b204,mgorman@techsingularity.net,:akpm@linux-foundation.org:mhocko@suse.com:vbabka@suse.cz:tglx@linutronix.de:matt@codeblueprint.co.uk:bp@alien8.de::linux-kernel@vger.kernel.org:mgorman@techsingularity.net,RULES_HIT:41:355:379:421:541:800:960:966:973:988:989:1260:1345:1359:1437:1534:1543:1711:1730:1747:1777:1792:1801:2196:2198:2199:2200:2393:2559:2562:2693:2895:3138:3139:3140:3141:3142:3355:3740:3865:3866:3867:3868:3870:3871:3872:3874:4250:4321:4385:4605:5007:6117:6119:6261:7774:7875:7903:8603:8634:8666:9010:9040:9121:10004:11026:11473:11537:11658:11914:12043:12297:12438:12517:12519:12522:12555:12895:13151:13161:13228:13229:13255:14181:14394:14721:14819:21080:21222:21451:21611:21627:21740:21795:21809:30034:30051:30054:30060,0,RBL:46.22.139.15:@techsingularity.net:.lbl8.mailshell.net-62.2.114.100 64.100.201.201,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fs,MSBL:0,DNSBL:neutral,Custom_rules:0 :0:0,LFt X-HE-Tag: jail15_18c27e3305e31 X-Filterd-Recvd-Size: 4769 Received: from outbound-smtp10.blacknight.com (outbound-smtp10.blacknight.com [46.22.139.15]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Fri, 18 Oct 2019 10:56:10 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail03.blacknight.ie [81.17.254.16]) by outbound-smtp10.blacknight.com (Postfix) with ESMTPS id 79DA31C32FE for ; Fri, 18 Oct 2019 11:56:08 +0100 (IST) Received: (qmail 30789 invoked from network); 18 Oct 2019 10:56:08 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.19.210]) by 81.17.254.9 with ESMTPA; 18 Oct 2019 10:56:08 -0000 From: Mel Gorman To: Andrew Morton Cc: Michal Hocko , Vlastimil Babka , Thomas Gleixner , Matt Fleming , Borislav Petkov , Linux-MM , Linux Kernel Mailing List , Mel Gorman Subject: [PATCH 2/3] mm, meminit: Recalculate pcpu batch and high limits after init completes Date: Fri, 18 Oct 2019 11:56:05 +0100 Message-Id: <20191018105606.3249-3-mgorman@techsingularity.net> X-Mailer: git-send-email 2.16.4 In-Reply-To: <20191018105606.3249-1-mgorman@techsingularity.net> References: <20191018105606.3249-1-mgorman@techsingularity.net> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Deferred memory initialisation updates zone->managed_pages during the initialisation phase but before that finishes, the per-cpu page allocator (pcpu) calculates the number of pages allocated/freed in batches as well as the maximum number of pages allowed on a per-cpu list. As zone->managed_pages is not up to date yet, the pcpu initialisation calculates inappropriately low batch and high values. This increases zone lock contention quite severely in some cases with the degree of severity depending on how many CPUs share a local zone and the size of the zone. A private report indicated that kernel build times were excessive with extremely high system CPU usage. A perf profile indicated that a large chunk of time was lost on zone->lock contention. This patch recalculates the pcpu batch and high values after deferred initialisation completes on each node. It was tested on a 2-socket AMD EPYC 2 machine using a kernel compilation workload -- allmodconfig and all available CPUs. mmtests configuration: config-workload-kernbench-max Configuration was modified to build on a fresh XFS partition. kernbench 5.4.0-rc3 5.4.0-rc3 vanilla resetpcpu-v1r1 Amean user-256 13249.50 ( 0.00%) 15928.40 * -20.22%* Amean syst-256 14760.30 ( 0.00%) 4551.77 * 69.16%* Amean elsp-256 162.42 ( 0.00%) 118.46 * 27.06%* Stddev user-256 42.97 ( 0.00%) 50.83 ( -18.30%) Stddev syst-256 336.87 ( 0.00%) 33.70 ( 90.00%) Stddev elsp-256 2.46 ( 0.00%) 0.81 ( 67.01%) 5.4.0-rc3 5.4.0-rc3 vanillaresetpcpu-v1r1 Duration User 39766.24 47802.92 Duration System 44298.10 13671.93 Duration Elapsed 519.11 387.65 The patch reduces system CPU usage by 69.16% and total build time by 27.06%. The variance of system CPU usage is also much reduced. Cc: stable@vger.kernel.org # v4.15+ Signed-off-by: Mel Gorman Tested-by: Matt Fleming Acked-by: Michal Hocko Signed-off-by: Mel Gorman Tested-by: Matt Fleming Acked-by: Michal Hocko Signed-off-by: Andrew Morton --- mm/page_alloc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cafe568d36f6..0a0dd74edc83 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1818,6 +1818,14 @@ static int __init deferred_init_memmap(void *data) */ while (spfn < epfn) nr_pages += deferred_init_maxorder(&i, zone, &spfn, &epfn); + + /* + * The number of managed pages has changed due to the initialisation + * so the pcpu batch and high limits needs to be updated or the limits + * will be artificially small. + */ + zone_pcp_update(zone); + zone_empty: pgdat_resize_unlock(pgdat, &flags); @@ -8516,7 +8524,6 @@ void free_contig_range(unsigned long pfn, unsigned int nr_pages) WARN(count != 0, "%d pages are still in use!\n", count); } -#ifdef CONFIG_MEMORY_HOTPLUG /* * The zone indicated has a new number of managed_pages; batch sizes and percpu * page high values need to be recalulated. @@ -8527,7 +8534,6 @@ void __meminit zone_pcp_update(struct zone *zone) __zone_pcp_update(zone); mutex_unlock(&pcp_batch_high_lock); } -#endif void zone_pcp_reset(struct zone *zone) {