From patchwork Wed Feb 26 12:03:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13992216 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55526C18E7C for ; Wed, 26 Feb 2025 12:04:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DF4F280045; Wed, 26 Feb 2025 07:03:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48F79280042; Wed, 26 Feb 2025 07:03:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2DF3B280044; Wed, 26 Feb 2025 07:03:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0E3EA280042 for ; Wed, 26 Feb 2025 07:03:59 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B91311C97CE for ; Wed, 26 Feb 2025 12:03:58 +0000 (UTC) X-FDA: 83161962156.28.36698C0 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf25.hostedemail.com (Postfix) with ESMTP id CBB6DA0019 for ; Wed, 26 Feb 2025 12:03:56 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740571437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=9vR7sZI/g9xLheO85wiiZxHP+SqzTea3rvr0rqea48g=; b=mO/txXVZrSgAP62NkmBS8B+0fOGUFLcWI+VyQrn+1C+6gr0IlAdUIPe3+Mn8kU/reqnrJo qkg2sdjYT+NFqI2Pvvg4AcSry04JYEPJQn6IeWiJKkXc24Kq5oofdagFuBeZmta9Xn+gCk djp6S8xhcYxWxovJVXuXSrEdxbcfamw= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf25.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740571437; a=rsa-sha256; cv=none; b=BSkTikndB8Tuh+zIyBTbnPydZpLwKfieF++ABaQlPEptwV+KLce8tqzut8ROWcfKC7YRdj YgLNbX/Pdt4dKwW6QTbSDLzkn157nrS3EcWOq6k5cM/Ma8cW63Iu/LcMXEbFrYnJvtMxwo VOKW38dHbmrVlIHhEZsjiwYMAxShDd8= X-AuditID: a67dfc5b-3e1ff7000001d7ae-6b-67bf03231e30 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, rjgolo@gmail.com Subject: [RFC PATCH v12 based on v6.14-rc4 24/25] mm/vmscan: apply luf mechanism to unmapping during folio reclaim Date: Wed, 26 Feb 2025 21:03:35 +0900 Message-Id: <20250226120336.29565-24-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250226120336.29565-1-byungchul@sk.com> References: <20250226113024.GA1935@system.software.com> <20250226120336.29565-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrALMWRmVeSWpSXmKPExsXC9ZZnoa4y8/50gy8TFCzmrF/DZvF5wz82 i6/rfzFbPP3Ux2JxedccNot7a/6zWpzftZbVYsfSfUwWlw4sYLI43nuAyWL+vc9sFps3TWW2 OD5lKqPF7x9z2Bz4PL639rF47Jx1l91jwaZSj80rtDw2repk89j0aRK7x7tz59g9Tsz4zeLx ft9VNo+tv+w8GqdeY/P4vEkugCeKyyYlNSezLLVI3y6BK2PGkeiCi/oVfw7uZWxgfKnWxcjB ISFgIrHyuGoXIyeYObP3PxuIzSagLnHjxk9mEFtEwEziYOsf9i5GLg5mgWVMEntPNIAVCQsU Scw7tAysiEVAVaJ/xX0WkJm8QA0XbltDzJSXWL3hAFgJJ1D407RjYK1CAskSO3//YYKouc0m cfKOH4QtKXFwxQ2WCYy8CxgZVjEKZeaV5SZm5pjoZVTmZVboJefnbmIEhvOy2j/ROxg/XQg+ xCjAwajEw/vgzN50IdbEsuLK3EOMEhzMSiK8nJl70oV4UxIrq1KL8uOLSnNSiw8xSnOwKInz Gn0rTxESSE8sSc1OTS1ILYLJMnFwSjUwlnBuvvlQl/FCLrPVDpXC9z/8t5VH3yzl9rogreWy a+ufdPXahAVnvPbs+vvh3AK7a+ULnycdZVYTD5ee1M1nax71c8EX/sVtZckuuUIegp1fjdp+ fOgRSuGKmZrUyhzGIXh+Z1VMS8r3pPxO5Rtzve1fzd3Gk+Sw+1/VSVE+39Pnl+w/tfqfEktx RqKhFnNRcSIAU/DYKmMCAAA= X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrJLMWRmVeSWpSXmKPExsXC5WfdrKvMvD/d4Mh0cYs569ewWXze8I/N 4uv6X8wWTz/1sVgcnnuS1eLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHG89wCTxfx7n9ks Nm+aymxxfMpURovfP+awOfB7fG/tY/HYOesuu8eCTaUem1doeWxa1cnmsenTJHaPd+fOsXuc mPGbxeP9vqtsHotffGDy2PrLzqNx6jU2j8+b5AJ4o7hsUlJzMstSi/TtErgyZhyJLrioX/Hn 4F7GBsaXal2MnBwSAiYSM3v/s4HYbALqEjdu/GQGsUUEzCQOtv5h72Lk4mAWWMYksfdEA1iR sECRxLxDy8CKWARUJfpX3GfpYuTg4AVquHDbGmKmvMTqDQfASjiBwp+mHQNrFRJIltj5+w/T BEauBYwMqxhFMvPKchMzc0z1irMzKvMyK/SS83M3MQIDdFntn4k7GL9cdj/EKMDBqMTD++DM 3nQh1sSy4srcQ4wSHMxKIrycmXvShXhTEiurUovy44tKc1KLDzFKc7AoifN6hacmCAmkJ5ak ZqemFqQWwWSZODilGhiLZwdfDuLnLb1rvft3XuJ+b+ktxnnHWh5OXZt8JmfBwTzfxHMnq94d vii+K2bBlPN9F5Zfk1I1eHDBb6WdHlut2LKCqhcdDIzOt2f8uXPrk9J6rdcvpY0jr0+RjrF+ lxsukePCrdScYvJhQgjn3tAy8flWSxxf5HtHX16ocnDSMq2Fgv83V/sosRRnJBpqMRcVJwIA 9alR/EwCAAA= X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: CBB6DA0019 X-Stat-Signature: p3cfxggzwgrxrrmtjasxd8xo1zypjxan X-HE-Tag: 1740571436-742508 X-HE-Meta: U2FsdGVkX1/OYCw4+hG/LmL/8AnUG/AbeV8xe11WSe1aRfP/C4DLthWfnop2lGUVbq6M4bjx2O4ANQ2+5YWaCkGL18rtdBDsNFUOMaJV2CbygHUavFweQEO5U93NEQ2i1DQVilNrE/0uf//OHPsBkeJL1gVphpkMbQqeaNBuCyZSaOT7zuTMS/BBAuCnn4BCxJrJ5M66rSF1tW9xLbTVFz00UOuYAs041TihiDc25tmhm9yN8pBJrqvb9Wl2QrBXdVimakKYQ5XR66v9VNiUob3mf/K1KNlL5oveIllS9uDovyK+VHGbuMCM1Sj5M+B1uhrwRtaJpVWmDHr9CxA8xKQHOjygqu3MRRN6gOp7/OkiQtxqpDtz2luaYpLIk/+GZHWxfc0SI8zQyH8il2wmbdjuKsfosG33ntryIMoYvF1uXXSYg3kl2MJQ4AhJtMqbqASHX12cIbr4haE3iHyhfV6zicWmixP14uTmM1Eoi6R5t7/O2VNr/XiqI3Q6ODTii+77yv3QLjtoDC2VPUunVMzP9CinmSM0tBJ2TfwwEErDkGRc9DwLJ7XYXvoJYLYCLhqkJlHqEZ3GfeK2MWqZyUXFWdKEoRqhMIF+U8E6l+7+U8HJx55WvHmWkduOk2Z0MtN6nOfKOMkwI9bPrhZS2xfwgNEImV42vIQoJpkKq4tXvmi8qdYN7nykl6TBStmD6WM71UOTU+LNDbv75YTP7DMnRZAouJ5O6H5eHNeA6SscGeIZ3BvGqml6MLsD4jGjNJRlxTkzB0B7grl0iU0uaKs9BYEkMtfynQ2gsPyK9uFSg0OW9l9VK3xVj+9bve1uk3j2bXu7teEjFETvxiq8bCDbLJLBP+Lpb18ItQDnO9V6RwCojHcCpwUo1xeDbceVk3mc2vowMAz2rucCRdBw93jLVCjbJF9Ekvy2/fscnKJaAZu4tIJO3mpc3mcZIU/UNOS54aJD6lYCkHR4WwS ylg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. Applied the mechanism to unmapping during folio reclaim. Signed-off-by: Byungchul Park --- include/linux/rmap.h | 5 +++-- mm/rmap.c | 11 +++++++---- mm/vmscan.c | 37 ++++++++++++++++++++++++++++++++----- 3 files changed, 42 insertions(+), 11 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index cedba4812ccc7..854b41441d466 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -661,7 +661,7 @@ int folio_referenced(struct folio *, int is_locked, struct mem_cgroup *memcg, unsigned long *vm_flags); bool try_to_migrate(struct folio *folio, enum ttu_flags flags); -void try_to_unmap(struct folio *, enum ttu_flags flags); +bool try_to_unmap(struct folio *, enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, unsigned long end, struct page **pages, @@ -794,8 +794,9 @@ static inline int folio_referenced(struct folio *folio, int is_locked, return 0; } -static inline void try_to_unmap(struct folio *folio, enum ttu_flags flags) +static inline bool try_to_unmap(struct folio *folio, enum ttu_flags flags) { + return false; } static inline int folio_mkclean(struct folio *folio) diff --git a/mm/rmap.c b/mm/rmap.c index 284fc48aef2de..df350b4dfddd0 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2386,10 +2386,11 @@ static int folio_not_mapped(struct folio *folio) * Tries to remove all the page table entries which are mapping this * folio. It is the caller's responsibility to check if the folio is * still mapped if needed (use TTU_SYNC to prevent accounting races). + * Return true if all the mappings are read-only, otherwise false. * * Context: Caller must hold the folio lock. */ -void try_to_unmap(struct folio *folio, enum ttu_flags flags) +bool try_to_unmap(struct folio *folio, enum ttu_flags flags) { struct rmap_walk_control rwc = { .rmap_one = try_to_unmap_one, @@ -2408,10 +2409,12 @@ void try_to_unmap(struct folio *folio, enum ttu_flags flags) else rmap_walk(folio, &rwc); - if (can_luf_test()) + if (can_luf_test()) { fold_batch(tlb_ubc_luf, tlb_ubc_ro, true); - else - fold_batch(tlb_ubc, tlb_ubc_ro, true); + return true; + } + fold_batch(tlb_ubc, tlb_ubc_ro, true); + return false; } /* diff --git a/mm/vmscan.c b/mm/vmscan.c index a31a7cf87315f..065b40f36bbdd 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -1092,14 +1092,17 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, struct reclaim_stat *stat, bool ignore_references) { struct folio_batch free_folios; + struct folio_batch free_folios_luf; LIST_HEAD(ret_folios); LIST_HEAD(demote_folios); unsigned int nr_reclaimed = 0, nr_demoted = 0; unsigned int pgactivate = 0; bool do_demote_pass; struct swap_iocb *plug = NULL; + unsigned short luf_key; folio_batch_init(&free_folios); + folio_batch_init(&free_folios_luf); memset(stat, 0, sizeof(*stat)); cond_resched(); do_demote_pass = can_demote(pgdat->node_id, sc); @@ -1111,6 +1114,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, enum folio_references references = FOLIOREF_RECLAIM; bool dirty, writeback; unsigned int nr_pages; + bool can_luf = false; cond_resched(); @@ -1344,7 +1348,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, if (folio_test_large(folio)) flags |= TTU_SYNC; - try_to_unmap(folio, flags); + can_luf = try_to_unmap(folio, flags); if (folio_mapped(folio)) { stat->nr_unmap_fail += nr_pages; if (!was_swapbacked && @@ -1488,6 +1492,8 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, * leave it off the LRU). */ nr_reclaimed += nr_pages; + if (can_luf) + luf_flush(fold_unmap_luf()); continue; } } @@ -1520,6 +1526,19 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, nr_reclaimed += nr_pages; folio_unqueue_deferred_split(folio); + + if (can_luf) { + if (folio_batch_add(&free_folios_luf, folio) == 0) { + mem_cgroup_uncharge_folios(&free_folios); + mem_cgroup_uncharge_folios(&free_folios_luf); + luf_key = fold_unmap_luf(); + try_to_unmap_flush(); + free_unref_folios(&free_folios, 0); + free_unref_folios(&free_folios_luf, luf_key); + } + continue; + } + if (folio_batch_add(&free_folios, folio) == 0) { mem_cgroup_uncharge_folios(&free_folios); try_to_unmap_flush(); @@ -1554,9 +1573,21 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, list_add(&folio->lru, &ret_folios); VM_BUG_ON_FOLIO(folio_test_lru(folio) || folio_test_unevictable(folio), folio); + if (can_luf) + luf_flush(fold_unmap_luf()); } /* 'folio_list' is always empty here */ + /* + * Finalize this turn before demote_folio_list(). + */ + mem_cgroup_uncharge_folios(&free_folios); + mem_cgroup_uncharge_folios(&free_folios_luf); + luf_key = fold_unmap_luf(); + try_to_unmap_flush(); + free_unref_folios(&free_folios, 0); + free_unref_folios(&free_folios_luf, luf_key); + /* Migrate folios selected for demotion */ nr_demoted = demote_folio_list(&demote_folios, pgdat); nr_reclaimed += nr_demoted; @@ -1590,10 +1621,6 @@ static unsigned int shrink_folio_list(struct list_head *folio_list, pgactivate = stat->nr_activate[0] + stat->nr_activate[1]; - mem_cgroup_uncharge_folios(&free_folios); - try_to_unmap_flush(); - free_unref_folios(&free_folios, 0); - list_splice(&ret_folios, folio_list); count_vm_events(PGACTIVATE, pgactivate);