From patchwork Wed Feb 26 12:03:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13992215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDAABC021B8 for ; Wed, 26 Feb 2025 12:04:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A822F280043; Wed, 26 Feb 2025 07:03:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A3572280042; Wed, 26 Feb 2025 07:03:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F6A5280043; Wed, 26 Feb 2025 07:03:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 34673280042 for ; Wed, 26 Feb 2025 07:03:58 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id F044D1C9C9B for ; Wed, 26 Feb 2025 12:03:57 +0000 (UTC) X-FDA: 83161962114.17.4C6D055 Received: from invmail4.hynix.com (exvmail4.skhynix.com [166.125.252.92]) by imf12.hostedemail.com (Postfix) with ESMTP id E418C40015 for ; Wed, 26 Feb 2025 12:03:55 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740571436; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=XXMUXL/Lk5kaMkA3arvEEUUX2KAeYLr0m6D2vVkax2Y=; b=6WF7DnGZgY/b9WuZCfwJ/1BUgr9qfH7G1zPjV1dj9bQB+r3Z21x5mLMslqklCw2cE1JF7B jActJYwmCP5C5XgfHppBFNCZf4UqWeWZU/aUzfezMXlRhjHsf9cRbLUn2GY4Cj4sgO67PU FtktJ/XCMwY+/IljhSdtQymLdPtE58s= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; spf=pass (imf12.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740571436; a=rsa-sha256; cv=none; b=xZiVXrL26+Ph4vzpDcJrc5q6FrJbwWVMNmemQRq4Bn0XG5PFVNCJKKWu78pk0BQxpe+UBq OfjMFHlth3aA1jITFhypIfIr8AZ0WCSmzlJ9LArMVDXz5ttklbqHuaPV6T+2E6N8HRGiwG AMcMiPSd8A+sOl8LbUs1GO299wonrN0= X-AuditID: a67dfc5b-3e1ff7000001d7ae-66-67bf032324af From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, rjgolo@gmail.com Subject: [RFC PATCH v12 based on v6.14-rc4 23/25] mm/migrate: apply luf mechanism to unmapping during migration Date: Wed, 26 Feb 2025 21:03:34 +0900 Message-Id: <20250226120336.29565-23-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250226120336.29565-1-byungchul@sk.com> References: <20250226113024.GA1935@system.software.com> <20250226120336.29565-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFrrMLMWRmVeSWpSXmKPExsXC9ZZnka4y8/50gw29shZz1q9hs/i84R+b xdf1v5gtnn7qY7G4vGsOm8W9Nf9ZLc7vWstqsWPpPiaLSwcWMFkc7z3AZDH/3mc2i82bpjJb HJ8yldHi9485bA58Ht9b+1g8ds66y+6xYFOpx+YVWh6bVnWyeWz6NInd4925c+weJ2b8ZvF4 v+8qm8fWX3YejVOvsXl83iQXwBPFZZOSmpNZllqkb5fAlfFxwV+Wgn8BFX/37GZpYJzo2MXI ySEhYCLx9tctFhh71s92JhCbTUBd4saNn8wgtoiAmcTB1j/sXYxcHMwCy5gk9p5oYANJCAvk S1ye+AyogYODRUBV4uplP5AwL1D9q/utzBAz5SVWbzgAZnMCxT9NOwbWKiSQLLHz9x8mkJkS ArfZJJ4cPgR1hKTEwRU3WCYw8i5gZFjFKJSZV5abmJljopdRmZdZoZecn7uJERjUy2r/RO9g /HQh+BCjAAejEg/vgzN704VYE8uKK3MPMUpwMCuJ8HJm7kkX4k1JrKxKLcqPLyrNSS0+xCjN waIkzmv0rTxFSCA9sSQ1OzW1ILUIJsvEwSnVwLjmyq818vf3sfz67aZbLvr3tfg2Ka2Vc9an u7WLzZ/xhiHwwzwZo9ZjU9cUcrwolBXkkC6yUPGuN978jKVDWf/kfAWmi08+Pz0e1s6zyn7p AU7j67xnXvRxupw2WZnKfrxLmPFRDcvfdU95nR+wSfn1nmF6dPOddfu38Dr3kypxP6/vcD78 6aASS3FGoqEWc1FxIgCdmyVNZgIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrFLMWRmVeSWpSXmKPExsXC5WfdrKvMvD/d4MYBIYs569ewWXze8I/N 4uv6X8wWTz/1sVgcnnuS1eLyrjlsFvfW/Ge1OL9rLavFjqX7mCwuHVjAZHG89wCTxfx7n9ks Nm+aymxxfMpURovfP+awOfB7fG/tY/HYOesuu8eCTaUem1doeWxa1cnmsenTJHaPd+fOsXuc mPGbxeP9vqtsHotffGDy2PrLzqNx6jU2j8+b5AJ4o7hsUlJzMstSi/TtErgyPi74y1LwL6Di 757dLA2MEx27GDk5JARMJGb9bGcCsdkE1CVu3PjJDGKLCJhJHGz9w97FyMXBLLCMSWLviQY2 kISwQL7E5YnPgBo4OFgEVCWuXvYDCfMC1b+638oMMVNeYvWGA2A2J1D807RjYK1CAskSO3// YZrAyLWAkWEVo0hmXlluYmaOqV5xdkZlXmaFXnJ+7iZGYIguq/0zcQfjl8vuhxgFOBiVeHgf nNmbLsSaWFZcmXuIUYKDWUmElzNzT7oQb0piZVVqUX58UWlOavEhRmkOFiVxXq/w1AQhgfTE ktTs1NSC1CKYLBMHp1QDIwtLfybnL7GM1RkLL8z/sTH6I/P2ovVMDFvKOzKM5stf28n92lpt er5M2JGLD7Oaz7fZH567zVJtVUJdis/TTXNVHgldiPVpDYt+X8pSpLjd5xd/qtmH05/WXmY7 6ORjGHGkK0c0MCX5oPnjIxy2dc77lVW3XWi4t8Le4rbFt/uJD3XOzu+ZrcRSnJFoqMVcVJwI AOzi2bhNAgAA X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Queue-Id: E418C40015 X-Stat-Signature: r4j7qcdxw9qs94zxhknzrcdm46145gdm X-Rspamd-Server: rspam03 X-HE-Tag: 1740571435-647598 X-HE-Meta: U2FsdGVkX18tTxkLeVC5JrJXPhDmtPRDr6PFRnvatBm5dhBYFnHJd3CFsMQJx92s/s06i7Am9E5/QvrYQUYVbgFdj65NgVrZUWKnyJ3nX/SvvO3Wrg1H+w9WrzOGUMClMOGtKkqVU12YsXhniHEi800orp32he/8YwsT7lrLpC33bwrO9spqUHHXu9zwSqdNIj8CNC7/9AaycoratrlmTYMi9jwr94JOScjVewHYpJq+xFW2KgRz7lS+sEqb3huzT+tWKKydQdA7Cnem8ZjTPD4vV+qAlq+18fMs1w+rCP60QG8uSWLMZvuooy2PwGamFlnMpDIBIAwLuW8/8EhR34oJ5YHJQhNNA2rm13+XbdjpqDDlMPZHVBmFFhj2g2cIpKDIKAFxew4HhlthYJml7EC7fQ11cIQ07rWM72GUKARMoztmaPC4DyebTbDsl/7JYwF6S+LxigJbbRkd1XuefYMLm3u35cEy3MLT8mcjdNnPSCYWyfN49gkpxKyhy7NAvh/wuOi6Txff17/hTRwbDWOOv4Ziu+8f+iqv+VcBld9udKpHyyEP67yF46b29s6Vm3tp8UGe5vs72ei0gUU3T7NorCJGcqmOMoZGUoDZayGy6U6Cx1CUuRLFX/TMN+5TToH+YB1LaJGdUcL8R6tkXPGnLp7ZC0xtbbwJ5BmYLesMO5Ml8W6s6+hneX7Rw1cUONRKA6LlUUu8KBuowxPTFK9sm0yIJPZSSvPLql8FC5wKUU29EoD3gumz+OaWwFCkl+y0rIMXxfKc/Ue3T5+dm2H3cY+8V1vq52z9lHjuGovzGFURKxk1yi/qgEMj1zDLLQE3ew8WBb3E+iqm8oBUgxmiwa51Gq7Yq4uAaCLh0k4DVY6t0ksfAtvnHh9cTUMxwCZzSm5VX3Yp/wX9/8BGxB76Wd7BE/d5SX7J8I2pL8bzqcYfFK794gTThN17LaN57d6viXDT9G1ytts0r6v PYw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. Applied the mechanism to unmapping during migration. Signed-off-by: Byungchul Park --- include/linux/mm.h | 2 ++ include/linux/rmap.h | 2 +- mm/migrate.c | 66 ++++++++++++++++++++++++++++++++++---------- mm/rmap.c | 15 ++++++---- mm/swap.c | 2 +- 5 files changed, 64 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e8e6562abc77d..1577bc8b743fe 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1489,6 +1489,8 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +void page_cache_release(struct folio *folio); + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 683a04088f3f2..cedba4812ccc7 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -660,7 +660,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, int folio_referenced(struct folio *, int is_locked, struct mem_cgroup *memcg, unsigned long *vm_flags); -void try_to_migrate(struct folio *folio, enum ttu_flags flags); +bool try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, diff --git a/mm/migrate.c b/mm/migrate.c index fb19a18892c89..7ce4d3dbcb1af 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1164,7 +1164,8 @@ static void migrate_folio_undo_dst(struct folio *dst, bool locked, /* Cleanup src folio upon migration success */ static void migrate_folio_done(struct folio *src, - enum migrate_reason reason) + enum migrate_reason reason, + unsigned short luf_key) { /* * Compaction can migrate also non-LRU pages which are @@ -1175,16 +1176,31 @@ static void migrate_folio_done(struct folio *src, mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + folio_is_file_lru(src), -folio_nr_pages(src)); - if (reason != MR_MEMORY_FAILURE) - /* We release the page in page_handle_poison. */ + /* We release the page in page_handle_poison. */ + if (reason == MR_MEMORY_FAILURE) + luf_flush(luf_key); + else if (!luf_key) folio_put(src); + else { + /* + * Should be the last reference. + */ + if (unlikely(!folio_put_testzero(src))) + VM_WARN_ON(1); + + page_cache_release(src); + folio_unqueue_deferred_split(src); + mem_cgroup_uncharge(src); + free_frozen_pages(&src->page, folio_order(src), luf_key); + } } /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_folio_t get_new_folio, free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio **dstp, enum migrate_mode mode, - enum migrate_reason reason, struct list_head *ret) + enum migrate_reason reason, struct list_head *ret, + bool *can_luf) { struct folio *dst; int rc = -EAGAIN; @@ -1200,7 +1216,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ list_del(&src->lru); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, 0); return MIGRATEPAGE_SUCCESS; } @@ -1317,7 +1333,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); + *can_luf = try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); old_page_state |= PAGE_WAS_MAPPED; } @@ -1345,7 +1361,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, unsigned short luf_key) { int rc; int old_page_state = 0; @@ -1399,7 +1415,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, luf_key); return rc; out: @@ -1694,7 +1710,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, unsigned short luf_key) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1711,7 +1727,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, luf_key); /* * The rules are: * Success: folio will be freed @@ -1788,7 +1804,11 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_luf); + LIST_HEAD(dst_folios_luf); bool nosplit = (reason == MR_NUMA_MISPLACED); + unsigned short luf_key; + bool can_luf; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); @@ -1863,9 +1883,11 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_luf = false; rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, - ret_folios); + ret_folios, &can_luf); + /* * The rules are: * Success: folio will be freed @@ -1911,7 +1933,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_luf)) goto out; else goto move; @@ -1925,8 +1948,13 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_luf) { + list_move_tail(&folio->lru, &unmap_folios_luf); + list_add_tail(&dst->lru, &dst_folios_luf); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1946,6 +1974,8 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + luf_key = fold_unmap_luf(); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1959,7 +1989,11 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, 0); + migrate_folios_move(&unmap_folios_luf, &dst_folios_luf, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, luf_key); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1970,6 +2004,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_luf, &dst_folios_luf, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/rmap.c b/mm/rmap.c index b6613b48669ac..284fc48aef2de 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2750,8 +2750,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, * * Tries to remove all the page table entries which are mapping this folio and * replace them with special swap entries. Caller must hold the folio lock. + * Return true if all the mappings are read-only, otherwise false. */ -void try_to_migrate(struct folio *folio, enum ttu_flags flags) +bool try_to_migrate(struct folio *folio, enum ttu_flags flags) { struct rmap_walk_control rwc = { .rmap_one = try_to_migrate_one, @@ -2769,11 +2770,11 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) */ if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | TTU_SYNC | TTU_BATCH_FLUSH))) - return; + return false; if (folio_is_zone_device(folio) && (!folio_is_device_private(folio) && !folio_is_device_coherent(folio))) - return; + return false; /* * During exec, a temporary VMA is setup and later moved. @@ -2793,10 +2794,12 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) else rmap_walk(folio, &rwc); - if (can_luf_test()) + if (can_luf_test()) { fold_batch(tlb_ubc_luf, tlb_ubc_ro, true); - else - fold_batch(tlb_ubc, tlb_ubc_ro, true); + return true; + } + fold_batch(tlb_ubc, tlb_ubc_ro, true); + return false; } #ifdef CONFIG_DEVICE_PRIVATE diff --git a/mm/swap.c b/mm/swap.c index 0c6198e4a8ee4..e322670c30041 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -84,7 +84,7 @@ static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, * This path almost never happens for VM activity - pages are normally freed * in batches. But it gets used by networking - and for compound pages. */ -static void page_cache_release(struct folio *folio) +void page_cache_release(struct folio *folio) { struct lruvec *lruvec = NULL; unsigned long flags;