From patchwork Wed Sep 20 06:18:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13392112 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19074CE79AD for ; Wed, 20 Sep 2023 06:19:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A47FF6B0113; Wed, 20 Sep 2023 02:19:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F43B6B0114; Wed, 20 Sep 2023 02:19:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8BCD86B0115; Wed, 20 Sep 2023 02:19:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 76C406B0113 for ; Wed, 20 Sep 2023 02:19:55 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 504DCB41E5 for ; Wed, 20 Sep 2023 06:19:55 +0000 (UTC) X-FDA: 81255975150.01.62E490B Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by imf29.hostedemail.com (Postfix) with ESMTP id 45D91120013 for ; Wed, 20 Sep 2023 06:19:53 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Gp7YBX0n; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695190793; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=B04gbBYfG2Z+ZG2AaIcVwZBgZPaKp1IkffSjK1YB3eQ=; b=fnXUc05Lrhf+Rvt1vzsMp2C5jpU2Uqlsyc6nYcaaCqHjW2pgmAKcxdaDzwqs8wqRbm0VMH jS2W3lIAxrQLXlCA7bCZNzqX/fdMhC8CQOYJxXPrDYf4SbCceLH7R1jR80tnBeyo6v0d0N Wda1zdJgN802CV5sdyiCtXOrdp8FVls= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Gp7YBX0n; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf29.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695190793; a=rsa-sha256; cv=none; b=LI2xjPGh+AwCcC3ifE0Hrqm4iYFfhdjWsF5B+lE9cHXesS+WouUqTV5yqohWDuEYpeDtMD 3FZmczpp8VBpCun4TBGkpCj+597KnlrGt4WASWBy/3IAI86MUT83jDT8qEXYk80SoL/PE+ cK6abLRVsfaQ228szQwG3f690TMplsU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695190793; x=1726726793; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=34wdsgvd2w4CDiR/vU6T5wdv61OCb5iOnSh2X+G58Mc=; b=Gp7YBX0nMiS5QX/TBV6p3G9xBLqXObK3Vvn25jn0OFU44htoSWmGhBO8 xv+t8h02FgEdi0dLRwcRuNl+mmOHsEUyEh19kd0qiPcLb1h9X9r+m6g+P 46KrzmOYM6RpgA/E5m+Ss6UN+SmSG2y2EZXtAFsP+YNEKXaVLb8pIKvQw f2vNEubh7v0tTgZd06YUAYshMqmz01kfTsWsgDY/ZE8yUaBoZ8b39gR26 yvYxuPJhy/kgH6wXcGB43jfeTboGe301YHenTKyzBvXwNivj3kxFNzKF9 Ot6gSC5Ku1KrPsELcyzePifubGKcLVYcgHfzN3Q2K/gww5copN/jP0P2G Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10838"; a="365187684" X-IronPort-AV: E=Sophos;i="6.02,161,1688454000"; d="scan'208";a="365187684" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Sep 2023 23:19:52 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10838"; a="740060606" X-IronPort-AV: E=Sophos;i="6.02,161,1688454000"; d="scan'208";a="740060606" Received: from yhuang6-mobl2.sh.intel.com ([10.238.6.133]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Sep 2023 23:19:49 -0700 From: Huang Ying To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Arjan Van De Ven , Huang Ying , Mel Gorman , Andrew Morton , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Michal Hocko , Pavel Tatashin , Matthew Wilcox , Christoph Lameter Subject: [PATCH 05/10] mm, page_alloc: scale the number of pages that are batch allocated Date: Wed, 20 Sep 2023 14:18:51 +0800 Message-Id: <20230920061856.257597-6-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230920061856.257597-1-ying.huang@intel.com> References: <20230920061856.257597-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 45D91120013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: c46oeyoi1qaajq3hxtqdzz6z4mpgbnz9 X-HE-Tag: 1695190793-606164 X-HE-Meta: U2FsdGVkX1/VfS66kpHpbgyki/Fwi9Zb9E/kI8xKxZXOugTzOdgytHD6AA8dKNnzEun2QmMdkyZvgJeYdfW2Avs0OFj++nnBvdQ5DmrDfrT3+ePvcka5dye9j7sYP3WcJRzc98QrgaiUjzfcrgQC+plaHS63UzKZDmFzWPTWZnkGQ749VPBUUKXw7uDUJ7bhJCBVKXUlwwGjPx0NIjevbKVV+2q9GBjnoXsiUWhCS7r0vmNXknIacKEUfBPywRMvXgW7OyyjRUB8IPDEaKz2JAosehFFRLc2tL/umaG7RrU1utgMDz8K8Ajvh6NjWMtf15164RUZ7CO/LdK+sUDXRzVGbw7tsL1OgQonarme/vzF03y+HGSdfz1wZ4lt6Wzy1fAulZHXAlm5dCJRgVYW90zp9ml3hBNPCLi/LVGYxPMvFLXcBTCiRxMk8S8xuZlj+rCEWXRhnTfKJDgm8bVXiidogk3VOOnaLfhl20u1RF061lOuAJgfW+gsc8ubrVOvga3AspI6sA6/u4hpj4IE2tIlxK5FOQPV40ZC8DyMyFKUlMR9RRAfcr+nt4FlVcz9ZAy7C3+yzbLEKVXWIrM0UzatVcCagzJmP8CDHNhhaeG51kD7MQDBuzf5r9fEoi1O1xGtdmhJ26NCLCox60Z6aAE5zMVP7R4V3nzHZJX3mQD6LJm7gxlT2jaG9h+SKkBHESvn6hXHaKt1m0HQzwh1hWZDSPe0E5fE5eHnVLbrfE5oU7JMbX5OaYzQYpVe6GLUJNebzhqwQ70yjl81urUudHb+TT1tyPenY2t+8TTExr4b5JTnKFKjJrabiEOhWcIDkvRGSDo7LKMQNaNjUBMD1kJu7hTE24et9YKd6q1NF7SoNj6FK+Xs658I9GkXnSK0Ggo9MY0BzVy8YlrbmNU+oiKs6tmHvS++SiXNP/W4LZSaKuEHS5GsupFafWjoxKnQUDEbOXMGYAl2BGAbZai hfa1bcAR Qb3RWSVEug7qkiqJ5zBLzsdd6gPGchmZQoPRrTzG+oG/tTJTPGof/Nb6bOidovXDlVpYGuYT3A1LqPQq0Bc9UwqybWEZlfNo08iQCuL/SPZSJIA0Dr0MUyIQkBoKQkz0bAV0ZUUoIWA1dXFmVsFH0T/ovfXVubGT/CkEOA2B0I9iAZThKmUyape2N36HstORWCKc1bvesZZKHoWQddeVwlOuns5rg9GrWQCjK7lyp/iS2hkE00VmDrFU+ArIPOBnXZnPaf7Yp0UnUCTf3kKZzH8T9SS0aaC9faQVm3yHnnhWpwBvZGroPUCmO0WVrBZVDuON0qzZQauVY9NWM2WMNxTEGBDatpSFBBRwvydAZqGesXJuQWO5Akxk/Z5W/ZSSeNHhWqTuG6zsy6Pp1KY2n52xWFo4AKWuPfQmF4Hcub0mpwoen11VlfV+FghQLq2pvKjzmYx9HeuwcQqO3qN3MRwROMzpDG/Fp9WDoLGXeHzuIf958utqljD2zWRkUkMdgHrw1kfmA75PvFdMoSnn9cdd1vhIS5ernI7jIv9b3sbbcetjATje3QdaTqyp+Ea0CHD839kBPIPZ/biMENHX7A/HVZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a task is allocating a large number of order-0 pages, it may acquire the zone->lock multiple times allocating pages in batches. This may unnecessarily contend on the zone lock when allocating very large number of pages. This patch adapts the size of the batch based on the recent pattern to scale the batch size for subsequent allocations. On a 2-socket Intel server with 224 logical CPU, we tested kbuild on one socket with `make -j 112`. With the patch, the cycles% of the spinlock contention (mostly for zone lock) decreases from 40.5% to 37.9% (with PCP size == 361). Signed-off-by: "Huang, Ying" Suggested-by: Mel Gorman Cc: Andrew Morton Cc: Vlastimil Babka Cc: David Hildenbrand Cc: Johannes Weiner Cc: Dave Hansen Cc: Michal Hocko Cc: Pavel Tatashin Cc: Matthew Wilcox Cc: Christoph Lameter Acked-by: Mel Gorman --- include/linux/mmzone.h | 3 ++- mm/page_alloc.c | 52 ++++++++++++++++++++++++++++++++++-------- 2 files changed, 44 insertions(+), 11 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 4132e7490b49..4f7420e35fbb 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -685,9 +685,10 @@ struct per_cpu_pages { int high; /* high watermark, emptying needed */ int batch; /* chunk size for buddy add/remove */ u8 flags; /* protected by pcp->lock */ + u8 alloc_factor; /* batch scaling factor during allocate */ u8 free_factor; /* batch scaling factor during free */ #ifdef CONFIG_NUMA - short expire; /* When 0, remote pagesets are drained */ + u8 expire; /* When 0, remote pagesets are drained */ #endif /* Lists of pages, one per migrate type stored on the pcp-lists */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 30554c674349..30bb05fa5353 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2376,6 +2376,12 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, int pindex; bool free_high = false; + /* + * On freeing, reduce the number of pages that are batch allocated. + * See nr_pcp_alloc() where alloc_factor is increased for subsequent + * allocations. + */ + pcp->alloc_factor >>= 1; __count_vm_events(PGFREE, 1 << order); pindex = order_to_pindex(migratetype, order); list_add(&page->pcp_list, &pcp->lists[pindex]); @@ -2682,6 +2688,41 @@ struct page *rmqueue_buddy(struct zone *preferred_zone, struct zone *zone, return page; } +static int nr_pcp_alloc(struct per_cpu_pages *pcp, int order) +{ + int high, batch, max_nr_alloc; + + high = READ_ONCE(pcp->high); + batch = READ_ONCE(pcp->batch); + + /* Check for PCP disabled or boot pageset */ + if (unlikely(high < batch)) + return 1; + + /* + * Double the number of pages allocated each time there is subsequent + * refiling of order-0 pages without drain. + */ + if (!order) { + max_nr_alloc = max(high - pcp->count - batch, batch); + batch <<= pcp->alloc_factor; + if (batch <= max_nr_alloc && pcp->alloc_factor < PCP_BATCH_SCALE_MAX) + pcp->alloc_factor++; + batch = min(batch, max_nr_alloc); + } + + /* + * Scale batch relative to order if batch implies free pages + * can be stored on the PCP. Batch can be 1 for small zones or + * for boot pagesets which should never store free pages as + * the pages may belong to arbitrary zones. + */ + if (batch > 1) + batch = max(batch >> order, 2); + + return batch; +} + /* Remove page from the per-cpu list, caller must protect the list */ static inline struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, @@ -2694,18 +2735,9 @@ struct page *__rmqueue_pcplist(struct zone *zone, unsigned int order, do { if (list_empty(list)) { - int batch = READ_ONCE(pcp->batch); + int batch = nr_pcp_alloc(pcp, order); int alloced; - /* - * Scale batch relative to order if batch implies - * free pages can be stored on the PCP. Batch can - * be 1 for small zones or for boot pagesets which - * should never store free pages as the pages may - * belong to arbitrary zones. - */ - if (batch > 1) - batch = max(batch >> order, 2); alloced = rmqueue_bulk(zone, order, batch, list, migratetype, alloc_flags);