From patchwork Tue Sep 26 06:09:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13398717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D20AE8181F for ; Tue, 26 Sep 2023 06:10:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9C49D8D006D; Tue, 26 Sep 2023 02:10:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 973688D0005; Tue, 26 Sep 2023 02:10:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83BC38D006D; Tue, 26 Sep 2023 02:10:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5DF488D0005 for ; Tue, 26 Sep 2023 02:10:08 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2477380F76 for ; Tue, 26 Sep 2023 06:10:08 +0000 (UTC) X-FDA: 81277723296.13.D1D4505 Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.100]) by imf19.hostedemail.com (Postfix) with ESMTP id 244E71A000B for ; Tue, 26 Sep 2023 06:10:04 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=dCgIwoe1; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695708605; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OkOx4+4B6DN2IG2d9UYB5azt2eRQHHjxJuFmoFGEHqs=; b=Ov9Mvi7R2tbn2zIJULQMJ4KYX1jMaFWAPTlk435UzQff/yNZX6kOjuEw0rciYVowo1Y5QH QExkxjJXIdqjVINiaaNoYXty1jlFXoxwM8BvxlED7CYOpBYxm9NzaihCxyXt4fY5osIhV4 OZ4ahGZpN+IrMpgPV8lCipnnnlVeZ5U= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=dCgIwoe1; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695708605; a=rsa-sha256; cv=none; b=EpVpfqeLcOh2yO98gpsEpOxcXlarbmO+5i68n6gdsuWnSGd/PeXc0KpU5K2+0W9mDfSbNo VbW275iLZYSA+8SVBz9NnVjbSXH15+FPUsoyn/uZw/f9h/jXfi5zizsli14i7fGYh0a6qq MvrqXInpfJvFVeMESEMPxczEdoFAdm0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695708605; x=1727244605; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Y9KtfnCLPgA/SXtfssQyiyzVZDDHiki6dZr93OTwpTQ=; b=dCgIwoe1eOoeY2ki9jFzwrIPTujScQUsam1dtsurIOvlBaX0g4svI54H m3auzTxnkPSM23Ndn2CnbmNIIGPJPfsZbdpzQk2ljurDyjTNsJEzlewi9 BS7BYDQcZCq+dhaCfOyCOVoPn/4wt9A/rFSll3r96C0yZKolQ/d3zPo8+ pW80QMIuMBaOj7TXzV2rSbm8k2JoMG4mPuoZ/rnNq8gNyf/hf955FxAX2 VPJ8kQDrRiHh4oy+XtUJsCzf8RwaWnX7GvRz0jUDPIcpP2XvCpK7+SZja +koj1lpoLIAC9l2tu0ZpNR8CUp8TAztTcuwI3x48uXFyoSmWYRMdTclu9 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="447991460" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="447991460" Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2023 23:10:04 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10843"; a="892076071" X-IronPort-AV: E=Sophos;i="6.03,177,1694761200"; d="scan'208";a="892076071" Received: from aozhu-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.31.94]) by fmsmga001-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 25 Sep 2023 23:08:54 -0700 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Arjan Van De Ven , Huang Ying , Mel Gorman , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Michal Hocko , Pavel Tatashin , Matthew Wilcox , Christoph Lameter Subject: [PATCH -V2 08/10] mm, pcp: decrease PCP high if free pages < high watermark Date: Tue, 26 Sep 2023 14:09:09 +0800 Message-Id: <20230926060911.266511-9-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230926060911.266511-1-ying.huang@intel.com> References: <20230926060911.266511-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 244E71A000B X-Stat-Signature: zzu485tp3kdnuebaagubuxoz96jkhw5j X-HE-Tag: 1695708604-872787 X-HE-Meta: U2FsdGVkX1+HsC5DcsgFooda7EF+bLAJwL4pQAIsUJKaKcDmzhi8iYfXOqHGJjYJjNPpgXGShUbsRk2oZOCwWFBHeElP29/KHUUpbkAlfgaDzvoRkddSSvpbtrjQxi8lxbflOa9lWUTiLu3i2NBbkDT6jbp6fL1UhDU3h6EUBWXXBkHFCqsftn4NFHHl1xsQ1EjNEkP7qlMzqoETcwPXja4atoJyELhkeErwHnQ4OadkrXxNTkXEBXcLOx+xAsrYZZAYPvEUoHDhiILMorevLiayIUJVWSR1OeVKM8Ss3TIzFurfe+ElHXKE6/smKcwMVXQM7Uahj7Jeq9wbjbJLBSCGMJepRbwGmWCcqKh41XP9LjIh8+bYk9bhfjpQsJBNrB5Ehmrsn6PZdS7PnGU+0glZdUUSbTyh2j55wLhFk9Y5sBSltc/iW2i2WZ8vo/p8I+LMqh+1x21r3mkBIUx0qTKnXBoRk7SKjAwR+/UbnchIDTYW0mR9B1uYXF7Bxga9TkDIsoFYDZES2/5UZKQSstZJfS1Ie4oqnyozymvRuGg7SqurlaPthVw3Rtfch2QMGnCn6VTcNCqYxITo+LEm9Q1/JyOvsSDhvkmnvm8K1gyQdSyoA8Eh54l6rsF5pG0GoGpvupoU2Go4Df6Lcw7b129BB21Ju7LBEc0vjsExepX7wi011vFHugRrDlYHJjnkZmSVK01m50wEHtUojv57pkMJbGoZ3iSP0Juk/2YSv0VUJ+anm9o91RU/Wufi6KLtBD9R/Ew+VGpbAocwcw7TXzLXed3LWNHTIulDrQil4VkHYYc4awrYcJxiwSRK/62LNMDgBQX6zq2Z6Nub6mSxNKvgfBsx+h7nTeKj+cxZCC97vUGognvBEyOa9fK2uAmXkcyq+AEBL0Nhocnem2nMJSzxvTBJWRtokCRh7W1Kfz92KPOEL168YW2svBcxNEWPeYZmzq2ftdrEXBE4VMz Dxj3gsc/ h4iJemxiTNKHdjmYSTOOY0S6fZfndIA6qwbRc5DI/x7PRtT80DJ7fjlDLRP3PkyhDyY2WMv7iiaZihiyCUKF8YMkMwhhPDlBVVH+UBLw4Qb23cILTCZJyKUoc7mQbyGy1iGwig9+d7QioV0paccHpEf0ZsSVqbQjKn0T75dgOsLjwi2Mg44XVMp6STh7pn5mFC7zyaLVlKvkwcK90OCJIM7MH8f4+zGNX5uqA0mBVO6D2yNUKwc7I49n/R5tfO24unx0/rf+AvWaL0g5vfpT04R/MlLS5RLrAA2U+jRL3LpFMAmdyeEOAEx2Sycm5du/uQdR7VO1eO+6Y7lYVIwP0nO/YRtxnFi6DMn1SdU7AugPHByMGsC7Fxw0Ok+ODgIMhIFLVsuQXdL+nsQSXUMAxpZEOiAZxRnnJDqjnlr+m5CurMlECCEnaDHLdubqkxMsm6NwoaseVnyf/Dwdj2h26ZrdREFXGpfsqHH9NglzHl+yAVIw3llvSRPm4ZyR1Plk4NENupIuhxMce57jeqtp3ROyoQwmoIM8B22oj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: One target of PCP is to minimize pages in PCP if the system free pages is too few. To reach that target, when page reclaiming is active for the zone (ZONE_RECLAIM_ACTIVE), we will stop increasing PCP high in allocating path, decrease PCP high and free some pages in freeing path. But this may be too late because the background page reclaiming may introduce latency for some workloads. So, in this patch, during page allocation we will detect whether the number of free pages of the zone is below high watermark. If so, we will stop increasing PCP high in allocating path, decrease PCP high and free some pages in freeing path. With this, we can reduce the possibility of the premature background page reclaiming caused by too large PCP. The high watermark checking is done in allocating path to reduce the overhead in hotter freeing path. Signed-off-by: "Huang, Ying" Cc: Andrew Morton Cc: Mel Gorman Cc: Vlastimil Babka Cc: David Hildenbrand Cc: Johannes Weiner Cc: Dave Hansen Cc: Michal Hocko Cc: Pavel Tatashin Cc: Matthew Wilcox Cc: Christoph Lameter --- include/linux/mmzone.h | 1 + mm/page_alloc.c | 22 ++++++++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index d6cfb5023f3e..8a19e2af89df 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1006,6 +1006,7 @@ enum zone_flags { * Cleared when kswapd is woken. */ ZONE_RECLAIM_ACTIVE, /* kswapd may be scanning the zone. */ + ZONE_BELOW_HIGH, /* zone is below high watermark. */ }; static inline unsigned long zone_managed_pages(struct zone *zone) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0d482a55235b..08b74c65b88a 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2409,7 +2409,13 @@ static int nr_pcp_high(struct per_cpu_pages *pcp, struct zone *zone, return min(batch << 2, pcp->high); } - if (pcp->count >= high && high_min != high_max) { + if (high_min == high_max) + return high; + + if (test_bit(ZONE_BELOW_HIGH, &zone->flags)) { + pcp->high = max(high - (batch << pcp->free_factor), high_min); + high = max(pcp->count, high_min); + } else if (pcp->count >= high) { int need_high = (batch << pcp->free_factor) + batch; /* pcp->high should be large enough to hold batch freed pages */ @@ -2459,6 +2465,10 @@ static void free_unref_page_commit(struct zone *zone, struct per_cpu_pages *pcp, if (pcp->count >= high) { free_pcppages_bulk(zone, nr_pcp_free(pcp, batch, high, free_high), pcp, pindex); + if (test_bit(ZONE_BELOW_HIGH, &zone->flags) && + zone_watermark_ok(zone, 0, high_wmark_pages(zone), + ZONE_MOVABLE, 0)) + clear_bit(ZONE_BELOW_HIGH, &zone->flags); } } @@ -2765,7 +2775,7 @@ static int nr_pcp_alloc(struct per_cpu_pages *pcp, struct zone *zone, int order) * If we had larger pcp->high, we could avoid to allocate from * zone. */ - if (high_min != high_max && !test_bit(ZONE_RECLAIM_ACTIVE, &zone->flags)) + if (high_min != high_max && !test_bit(ZONE_BELOW_HIGH, &zone->flags)) high = pcp->high = min(high + batch, high_max); if (!order) { @@ -3226,6 +3236,14 @@ get_page_from_freelist(gfp_t gfp_mask, unsigned int order, int alloc_flags, } } + mark = high_wmark_pages(zone); + if (zone_watermark_fast(zone, order, mark, + ac->highest_zoneidx, alloc_flags, + gfp_mask)) + goto try_this_zone; + else if (!test_bit(ZONE_BELOW_HIGH, &zone->flags)) + set_bit(ZONE_BELOW_HIGH, &zone->flags); + mark = wmark_pages(zone, alloc_flags & ALLOC_WMARK_MASK); if (!zone_watermark_fast(zone, order, mark, ac->highest_zoneidx, alloc_flags,