From patchwork Thu May 23 05:14:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13671272 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5CA21C25B75 for ; Thu, 23 May 2024 05:14:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97DDB6B007B; Thu, 23 May 2024 01:14:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92E626B0082; Thu, 23 May 2024 01:14:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7F5976B0083; Thu, 23 May 2024 01:14:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 6263E6B007B for ; Thu, 23 May 2024 01:14:22 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id DBD6F1602DB for ; Thu, 23 May 2024 05:14:21 +0000 (UTC) X-FDA: 82148494722.24.D8E9C50 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf15.hostedemail.com (Postfix) with ESMTP id A3C82A0005 for ; Thu, 23 May 2024 05:14:19 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf15.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1716441260; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=ZBDu2S3I3NpBlsQUbsWSRkS9+pdVshjCNLDsLTHOyrc=; b=54ITdkdjBtCzI/e4RM+nLLU4xa2J1tA7YboMIkoalpNs1+3pPDOnBs5CMG999CgYEOWILf bsuIz1VhyhuEN1O7dIH7sWkzN6A/fSfz7arD03wQWKp0b6jshOqmQZNPXhQIEFZUZZoGUr gYbouCYtqV89Mejnm8e/08KlEOhPfk0= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf15.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1716441260; a=rsa-sha256; cv=none; b=5hgMdwCD7fEpDCfqn3ZFkxIfLBH8s733BO8nqtZdsizE72UYqp99+C/kXIqucM5fhefNv0 FtewHkYDfKXPI0TgG0ZqmV46eCR0C4V19sMNv5sjv6qdt5jK7btW9H8lBmsELUwklaRT51 sKZNmlIx1NQRqwmHLUugD7srWhmg2tE= X-AuditID: a67dfc5b-d6dff70000001748-cf-664ed0a93f34 From: Byungchul Park To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com Subject: [PATCH] mm: let kswapd work again for node that used to be hopeless but may not now Date: Thu, 23 May 2024 14:14:06 +0900 Message-Id: <20240523051406.81053-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrOJMWRmVeSWpSXmKPExsXC9ZZnoe7KC35pBrv2sVvMWb+GzeLyrjls FvfW/Gd1YPbY9GkSu8eJGb9ZPD5vkgtgjuKySUnNySxLLdK3S+DKOHLDpOC8VsXtd8ENjJuV uhg5OSQETCTWnOln7GLkALO/7csGCbMJqEvcuPGTGcQWEZCVmPr3PAuIzSwQKLH80BqwuLBA jMTVr3/ZQWwWAVWJfyteMYLYvAKmEt1v1rNDjJeXWL3hAFA9F5C9ilVi+/cHbBAJSYmDK26w TGDkXsDIsIpRKDOvLDcxM8dEL6MyL7NCLzk/dxMj0LvLav9E72D8dCH4EKMAB6MSD2+HrF+a EGtiWXFl7iFGCQ5mJRHe6JW+aUK8KYmVValF+fFFpTmpxYcYpTlYlMR5jb6VpwgJpCeWpGan phakFsFkmTg4pRoYndOX+h3sSGtXS7m4uNI25eWK9Cea5ZUZOgbtKt58GYYRjfkagW1TjJUD 6i7H83ha80u7Kk00WXRqZ+1Gq98OQZPs1z/4/PtOcuaqwN44jwcPZ22buVTFNPl2/d+omzGn Ek/snb53+9wL5bJZ8cnaE147WpRf+nfsfp6py4z4rJC3EfoyatZKLMUZiYZazEXFiQBfdEPn 6gEAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrHJMWRmVeSWpSXmKPExsXC5WfdrLvigl+awa6VVhZz1q9hszg89ySr xeVdc9gs7q35z+rA4rHp0yR2jxMzfrN4LH7xgcnj8ya5AJYoLpuU1JzMstQifbsErowjN0wK zmtV3H4X3MC4WamLkYNDQsBE4tu+7C5GTg42AXWJGzd+MoPYIgKyElP/nmcBsZkFAiWWH1oD FhcWiJG4+vUvO4jNIqAq8W/FK0YQm1fAVKL7zXqwuISAvMTqDQeYJzByLGBkWMUokplXlpuY mWOqV5ydUZmXWaGXnJ+7iRHoq2W1fybuYPxy2f0QowAHoxIPb8Iv3zQh1sSy4srcQ4wSHMxK IrzRK4FCvCmJlVWpRfnxRaU5qcWHGKU5WJTEeb3CUxOEBNITS1KzU1MLUotgskwcnFINjDEf +VOfy0/xZBGOFX3kyDln5sfWCYyeVtl8qoXnk6srd3zZHv766LOmRxnf0o40BE34yZrkES3K 4jR3Hvshjd1x93wFdy+R3dmifNyfxfGcz/Kwu6nrLxYfm5ojcU+v4/mdyWHbjSI7i/UmybrG vPtx80Hmi6ui8RoBX3h++8Y8Xnvn0twFK5VYijMSDbWYi4oTAeJ4mHPRAQAA X-CFilter-Loop: Reflected X-Stat-Signature: ukkbjhxt3ck5qg4fkcgmozo7ta3b5egt X-Rspamd-Queue-Id: A3C82A0005 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1716441259-77878 X-HE-Meta: U2FsdGVkX1/g2uqvgf9VFTgcB9xZm6W11w73xcpdy31/3kDLaQILM9WxCuya6dUTf8Rte/xU5r74eL90wP7JCAhud3OaHrS486hlDdxGnkHBBBfz9FJkYvkdQyR3HvvuLzBsoPcNNjqA4poGe0j2Iirpg1+GduWDTzJAWQ7wnL1mQjSQUtbkuIriWnLO8oHLcBQ6WSTDfet9g175r/ZYzw+VKh3WwIKWWNNrRZwb1nEu/7hNSKj3nuQPgLgrcSFJx9OmD+PswODXpzBjPA6o2x1SlLa3fGFecj8ZtL96TMPgO7TcePVUGDq8kzcKEfONp20TMQwksFS9OckhyUXd8QoOZImaXZSOEaiXDg73N8vf0yTf5QnYPVQTaOWD+Fr54qln4UT+O9ZW3f6B2V7wAKdOQgf5MBLfmd7ZidRufJivFaA3OzuAzI+ysxoaGEGL4HNvNDhJPxYpamTFi6rgejPq0rCf122ILZVtRyj/6uq+7vqLadml+A72JgffXUPIo7JFIv82oPhHa/9YT7yiRMnE3WlsOfCVDJaqzYwsjP9EMNWehbFL9SODdu2zgIZX+/mXOt2F0nM+Zl3U0GZd6jDZC59TSr2BkdNv/1IPsh0ynEywRZR62EnPneBJ9wc3fCQ9YKV3l/rwJIBPxgeyWHTDdDCk5/O6tJ8jEbn/iml8nPwK6QYZ0MyrFBObid25+WSSLsH4A9HBRrfcPZ+V/5DgezxmepRmE0SIA2swTYFl6Nce9fxmzJv9Rx/Wc+hHMA0h+XJ3sJZZjnCROSlfhFcNTjjoOP8WYSPWfB7oq775b9wX/O0kLd2GkrYQjxcflhwynkyyASYJNssvZ6ZeVpSOc3I9V5k4l8qhBLPWysj879FOW39JeuY06VXZrFXK4eMi1yy9SBfEzco5zmabXkr7yNMHdxtxiFcjgqdmIfp3yQVKKKuOAIifW7dYu9T0Bu3wb6NpIMLpf/at4NH nOA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: I suffered from kswapd stopped in the following scenario: CONFIG_NUMA_BALANCING enabled sysctl_numa_balancing_mode set to NUMA_BALANCING_MEMORY_TIERING numa node0 (500GB local DRAM, 128 CPUs) numa node1 (100GB CXL memory, no CPUs) swap off 1) Run any workload using a lot of anon pages e.g. mmap(200GB). 2) Keep adding another workload using a lot of anon pages. 3) The DRAM becomes filled with only anon pages through promotion. 4) Demotion barely works due to severe memory pressure. 5) kswapd for node0 stops because of the unreclaimable anon pages. 6) Manually kill the memory hoggers. 7) kswapd is still stopped even though the system got back to normal. From now on, the system should run without reclaim service in background served by kswapd until direct reclaim will do for that. Even worse, tiering mechanism is no longer able to work because kswapd has stopped that the mechanism relies on. However, after 6), the DRAM will be filled with pages that might or might not be reclaimable, that depends on how those are going to be used. Since those are potentially reclaimable anyway, it's worth hopefully trying reclaim by allowing kswapd to work again if needed. Signed-off-by: Byungchul Park --- include/linux/mmzone.h | 4 ++++ mm/page_alloc.c | 12 ++++++++++++ mm/vmscan.c | 21 ++++++++++++++++----- 3 files changed, 32 insertions(+), 5 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index c11b7cde81ef..7c0ba90ea7b4 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1331,6 +1331,10 @@ typedef struct pglist_data { enum zone_type kswapd_highest_zoneidx; int kswapd_failures; /* Number of 'reclaimed == 0' runs */ + int nr_may_reclaimable; /* Number of pages that have been + allocated since considered the + node is hopeless due to too many + kswapd_failures. */ #ifdef CONFIG_COMPACTION int kcompactd_max_order; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 14d39f34d336..1dd2daede014 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -1538,8 +1538,20 @@ inline void post_alloc_hook(struct page *page, unsigned int order, static void prep_new_page(struct page *page, unsigned int order, gfp_t gfp_flags, unsigned int alloc_flags) { + pg_data_t *pgdat = page_pgdat(page); + post_alloc_hook(page, order, gfp_flags); + /* + * New pages might or might not be reclaimable depending on how + * these pages are going to be used. However, since these are + * potentially reclaimable, it's worth hopefully trying reclaim + * by allowing kswapd to work again even if there have been too + * many ->kswapd_failures, if ->nr_may_reclaimable is big enough. + */ + if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES) + pgdat->nr_may_reclaimable += 1 << order; + if (order && (gfp_flags & __GFP_COMP)) prep_compound_page(page, order); diff --git a/mm/vmscan.c b/mm/vmscan.c index 3ef654addd44..5b39090c4ef1 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4943,6 +4943,7 @@ static void lru_gen_shrink_node(struct pglist_data *pgdat, struct scan_control * done: /* kswapd should never fail */ pgdat->kswapd_failures = 0; + pgdat->nr_may_reclaimable = 0; } /****************************************************************************** @@ -5991,9 +5992,10 @@ static void shrink_node(pg_data_t *pgdat, struct scan_control *sc) * sleep. On reclaim progress, reset the failure counter. A * successful direct reclaim run will revive a dormant kswapd. */ - if (reclaimable) + if (reclaimable) { pgdat->kswapd_failures = 0; - else if (sc->cache_trim_mode) + pgdat->nr_may_reclaimable = 0; + } else if (sc->cache_trim_mode) sc->cache_trim_mode_failed = 1; } @@ -6636,6 +6638,11 @@ static void clear_pgdat_congested(pg_data_t *pgdat) clear_bit(PGDAT_WRITEBACK, &pgdat->flags); } +static bool may_recaimable(pg_data_t *pgdat, int order) +{ + return pgdat->nr_may_reclaimable >= 1 << order; +} + /* * Prepare kswapd for sleeping. This verifies that there are no processes * waiting in throttle_direct_reclaim() and that watermarks have been met. @@ -6662,7 +6669,8 @@ static bool prepare_kswapd_sleep(pg_data_t *pgdat, int order, wake_up_all(&pgdat->pfmemalloc_wait); /* Hopeless node, leave it to direct reclaim */ - if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES) + if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES && + !may_recaimable(pgdat, order)) return true; if (pgdat_balanced(pgdat, order, highest_zoneidx)) { @@ -6940,8 +6948,10 @@ static int balance_pgdat(pg_data_t *pgdat, int order, int highest_zoneidx) goto restart; } - if (!sc.nr_reclaimed) + if (!sc.nr_reclaimed) { pgdat->kswapd_failures++; + pgdat->nr_may_reclaimable = 0; + } out: clear_reclaim_active(pgdat, highest_zoneidx); @@ -7204,7 +7214,8 @@ void wakeup_kswapd(struct zone *zone, gfp_t gfp_flags, int order, return; /* Hopeless node, leave it to direct reclaim if possible */ - if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES || + if ((pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES && + !may_recaimable(pgdat, order)) || (pgdat_balanced(pgdat, order, highest_zoneidx) && !pgdat_watermark_boosted(pgdat, highest_zoneidx))) { /*