From patchwork Thu Feb 22 07:08:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13566778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 903F1C48BEB for ; Thu, 22 Feb 2024 07:08:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C50FD6B0075; Thu, 22 Feb 2024 02:08:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C017B6B0078; Thu, 22 Feb 2024 02:08:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC8DF6B007B; Thu, 22 Feb 2024 02:08:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9ADB46B0075 for ; Thu, 22 Feb 2024 02:08:37 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5489940CE8 for ; Thu, 22 Feb 2024 07:08:37 +0000 (UTC) X-FDA: 81818561874.30.985091D Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf29.hostedemail.com (Postfix) with ESMTP id 156A6120016 for ; Thu, 22 Feb 2024 07:08:32 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708585716; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:references; bh=dWrqA3p55spLe74XoQgXoo+/wXMhH1wq8LSzHqTJKmY=; b=fG2HkmHOZ2O+MdadgqmjZa3JNpmPtOWzIyoOApKm1IdxqNelLDIAEqBkYGZ12vdYOLRYzw jjUeO9MtaqaQjtSADV7Uk1DvRLwhAH2WERkppEJnFHE5gGK3tpNcPVcx0L2Xp/TYLmRU1A UffSRLrpruJbuHyzVB6QuZL6imnTv28= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf29.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708585716; a=rsa-sha256; cv=none; b=ukg64S7EJVgel1GwjLcBe9azj7wYQnbfEQDUvfoGtra3Fq476dSJZQxo8z7922bkSQqKU4 BeUyxpVLYui0I34e4MBKBZkVI5JpRxAOsCb9293DyUOOHUuABZmijXl4mMN0H60HioUi0n R5vtjODMscWwPASbpoBH434VYcUHsf0= X-AuditID: a67dfc5b-d6dff70000001748-7a-65d6f2eb9aed From: Byungchul Park To: akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, kernel_team@skhynix.com, yuzhao@google.com, ying.huang@intel.com Subject: [PATCH v2] mm, vmscan: don't turn on cache_trim_mode at high scan priorities Date: Thu, 22 Feb 2024 16:08:17 +0900 Message-Id: <20240222070817.70515-1-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrDJMWRmVeSWpSXmKPExsXC9ZZnke7rT9dSDZ5eVrSYs34Nm8XlXXPY LO6t+c9qcXLWZBaLdxO+sDqweizYVOqxeM9LJo9Nnyaxe5yY8ZvF4/MmuQDWKC6blNSczLLU In27BK6Mxw+fMhW0KlW8O5nYwPhKqouRk0NCwETi45oHzDD2jEnX2UBsNgF1iRs3foLFRQRk Jab+Pc8CYjML1Eks//cRLC4sECpx58JDsHoWAVWJZ807GUFsXgFTifV7NrBDzJSXWL3hAFA9 F5B9klXi954/UMskJQ6uuMEygZF7ASPDKkahzLyy3MTMHBO9jMq8zAq95PzcTYzAMFhW+yd6 B+OnC8GHGAU4GJV4eB8wXksVYk0sK67MPcQowcGsJMLLUn4lVYg3JbGyKrUoP76oNCe1+BCj NAeLkjiv0bfyFCGB9MSS1OzU1ILUIpgsEwenVANj99OpF53X7rTdb3rwiln+l/LoVjnn0xOU RMOKZVVrJaPmlWc+Suax5FT6lNqzMoav8aFLT8uMfPkNR7+XXdgSO/ebvZ6Z+ay0WSKr5l62 4txg4tNyYYuUcdQPsVUb+2/Wrzo9scew6sPXkhMbp57S4BA3u1w2p/HTrSc33BSrPt4JLPU8 /e2/EktxRqKhFnNRcSIAQgW+v/8BAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMJMWRmVeSWpSXmKPExsXC5WfdrPv607VUg++PpSzmrF/DZnF47klW i8u75rBZ3Fvzn9Xi5KzJLBbvJnxhdWDzWLCp1GPxnpdMHps+TWL3ODHjN4vH4hcfmDw+b5IL YIvisklJzcksSy3St0vgynj88ClTQatSxbuTiQ2Mr6S6GDk5JARMJGZMus4GYrMJqEvcuPGT GcQWEZCVmPr3PAuIzSxQJ7H830ewuLBAqMSdCw/B6lkEVCWeNe9kBLF5BUwl1u/ZwA4xU15i 9YYDzBMYORYwMqxiFMnMK8tNzMwx1SvOzqjMy6zQS87P3cQI9Oqy2j8TdzB+uex+iFGAg1GJ h/cB47VUIdbEsuLK3EOMEhzMSiK8LOVXUoV4UxIrq1KL8uOLSnNSiw8xSnOwKInzeoWnJggJ pCeWpGanphakFsFkmTg4pRoYV6pGfGiUVfBN+mjAn56+bbVale4N+0eP6rJ2uB/8a5m08vql eQWhC3d9ftPheibamev4388GT7fy2bnYuiz2nvz3d7SxeVrq3tNNUS+1Nt8wfrzlza2sG/Id d1+nuZ+WreWflVIdP+upfqHk2a+LM45fvKLncemASojMLlFeBf853vf17mZeVWIpzkg01GIu Kk4EAHTWec7mAQAA X-CFilter-Loop: Reflected X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 156A6120016 X-Stat-Signature: qpkb5pcs4pxa5hjqs4wmzqif5e1k1j9u X-Rspam-User: X-HE-Tag: 1708585712-552816 X-HE-Meta: U2FsdGVkX1/KTVfyRaSFbseRPCog5/X0c71tPMuymZTyPqK2Uo+/KN6i3WCXwOBE8cSX+Apn0yKcr8tXjbV8LfjrvfusLKVIcqBN6SVlqwxvnV1lahDVMHAPXm5qXuz3GwVhREYRzq7lwtGdEMgc3JFco/MTjlyI0f039+lPKNejpFvvO/y5zeFNw40f5tHiEVCoF0hRapmCBBWABuoUmSnNPZKUsBM97mDjgQn7z++dSmty/rKaehpvjMjKoXxl4Jl1sU3aMSCYTpfQhEW8tHgRCHZFkf8BGB/dq+kpZ/e2dhOVhoyO6Q6Z95YblzNyZr7H+x+T61V2QnH5emWtT6JKE+kn7PvuZhBPySTEHkjPUExHGVKSYsdf2FU2snLrPUqRYNxQwL90ddgdF/0MM4gRpVguJkB0JqWjo2+Tx5ng5KsJZrZg1h9hQbqmDPuPZ0jbDColxuHwC7vpkcOKDkGE3HYWI6HajHupxB2SspEv2r0b4bki+SF1TBbXQ8d2ACUKtlDK6hs0Jq+9AQ1NSNMSLj8L5MAprd0JJJ42Dt7S1gLY/kUr+K4+EM5Cuv/bmXnqpFip7qtfZk7c1kKo4Ax8C3vnploF4hMHr/3aD5gHprmdD5Qb96H5JUJrU6Tv51Z2lDPR/UGdJbmLm55+NruxtXr3+b2tQ7EFbqkZ9f76/06kOezwJd1VHWoVPsQXM97Yw8Od659SR36BNIqchqFBVdDg8/hbuZNG5hAUwXOA6i85uDEaXwVtJtB7SHY3 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Changes from v1: 1. Add a comment describing why this change is necessary in code and rewrite the commit message with how to reproduce and what the result is using vmstat. (feedbacked by Andrew Morton and Yu Zhao) 2. Change the condition to avoid cache_trim_mode from 'sc->priority != 1' to 'sc->priority > 1' to reflect cases where the priority goes to zero all the way. (feedbacked by Yu Zhao) --->8--- From 07e0baab368160e50b6ca35d95745168aa60e217 Mon Sep 17 00:00:00 2001 From: Byungchul Park Date: Thu, 22 Feb 2024 14:50:17 +0900 Subject: [PATCH v2] mm, vmscan: don't turn on cache_trim_mode at high scan priorities With cache_trim_mode on, reclaim logic doesn't bother reclaiming anon pages. However, it should be more careful to turn on the mode because it's going to prevent anon pages from being reclaimed even if there are a huge number of anon pages that are cold and should be reclaimed. Even worse, that can lead kswapd_failures to reach MAX_RECLAIM_RETRIES and stopping kswapd until direct reclaim eventually works to resume kswapd. So this is more like a bug fix than a performance improvement. The problematic behavior can be reproduced by: CONFIG_NUMA_BALANCING enabled sysctl_numa_balancing_mode set to NUMA_BALANCING_MEMORY_TIERING numa node0 (8GB local memory, 16 CPUs) numa node1 (8GB slow tier memory, no CPUs) Sequence: 1) echo 3 > /proc/sys/vm/drop_caches 2) To emulate the system with full of cold memory in local DRAM, run the following dummy program and never touch the region: mmap(0, 8 * 1024 * 1024 * 1024, PROT_READ | PROT_WRITE, MAP_ANONYMOUS | MAP_PRIVATE | MAP_POPULATE, -1, 0); 3) Run any memory intensive work e.g. XSBench. 4) Check if numa balancing is working e.i. promotion/demotion. 5) Iterate 1) ~ 4) until kswapd stops. With this, you could eventually see that promotion/demotion are not working because kswapd has stopped due to ->kswapd_failures >= MAX_RECLAIM_RETRIES. Interesting vmstat delta's differences between before and after are like: -nr_inactive_anon 321935 -nr_active_anon 1780700 -nr_inactive_file 30425 -nr_active_file 14961 -pgpromote_success 356 -pgpromote_candidate 21953245 -pgactivate 1844523 -pgdeactivate 50634 -pgfault 31100294 -pgdemote_kswapd 30856 -pgscan_kswapd 1861981 -pgscan_anon 1822930 -pgscan_file 39051 -pgsteal_anon 386 -pgsteal_file 30470 -pageoutrun 30 -numa_hint_faults 27418279 -numa_pages_migrated 356 +nr_inactive_anon 1662306 +nr_active_anon 440303 +nr_inactive_file 27669 +nr_active_file 1654 +pgpromote_success 1314102 +pgpromote_candidate 1892525 +pgactivate 3284457 +pgdeactivate 1527504 +pgfault 6847775 +pgdemote_kswapd 2142047 +pgscan_kswapd 7496588 +pgscan_anon 7462488 +pgscan_file 34100 +pgsteal_anon 2115661 +pgsteal_file 26386 +pageoutrun 378 +numa_hint_faults 3220891 +numa_pages_migrated 1314102 where -: before this patch, +: after this patch Signed-off-by: Byungchul Park --- mm/vmscan.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index bba207f41b14..6eda59fce5ee 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2266,9 +2266,17 @@ static void prepare_scan_control(pg_data_t *pgdat, struct scan_control *sc) * If we have plenty of inactive file pages that aren't * thrashing, try to reclaim those first before touching * anonymous pages. + * + * However, the condition 'sc->cache_trim_mode == 1' all through + * the scan priorties might lead reclaim failure. If it keeps + * MAX_RECLAIM_RETRIES times, then kswapd would get stopped even + * if there are still plenty anon pages to reclaim, which is not + * desirable. So do not use cache_trim_mode when reclaim is not + * smooth e.i. high scan priority. */ file = lruvec_page_state(target_lruvec, NR_INACTIVE_FILE); - if (file >> sc->priority && !(sc->may_deactivate & DEACTIVATE_FILE)) + if (sc->priority > 1 && file >> sc->priority && + !(sc->may_deactivate & DEACTIVATE_FILE)) sc->cache_trim_mode = 1; else sc->cache_trim_mode = 0;