From patchwork Thu Feb 20 05:20:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13983340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA0C3C021B1 for ; Thu, 20 Feb 2025 05:21:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C8792801D3; Thu, 20 Feb 2025 00:20:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5522B2801CA; Thu, 20 Feb 2025 00:20:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26B0728018F; Thu, 20 Feb 2025 00:20:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CC2E628013A for ; Thu, 20 Feb 2025 00:20:50 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 95D8BC0B2B for ; Thu, 20 Feb 2025 05:20:50 +0000 (UTC) X-FDA: 83139173460.26.FC32AA5 Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf01.hostedemail.com (Postfix) with ESMTP id 6856A4000F for ; Thu, 20 Feb 2025 05:20:48 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740028848; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=5p5ntld4g8OjOWaErdoALvFDdNmR4Src3R46ZMd0RRY=; b=PQZJQGV4pen8+VsZOyzEg6Ughd9v0zl6rv6YP5sTtD6ah97aB0gpijQuelN0CS1S6H6G47 k1sPnVFk175u45hE2X4fdncjkGqAfazkRRD1Z1V31blzfgK8omSLwhqg6Nr++TRFd7GdXB u68YlNt8ejPWrLABHsTnCKhLh6mxbiM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740028848; a=rsa-sha256; cv=none; b=f6u7udexMXZ4GRcADjTq91y884ZlbHZczTAdk8YJSGOXB6RUp602rADsO54XZeYMIXHBue vUa39VCqtN1RSWw0eD77B2Y0ML5ZUi/4h4rvYR5bQkrvZpKYRIKW91MG+9+apzY2qMCBlg ifijqMJfU/CHyEIqVmEdzgNpgHf0HPg= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com X-AuditID: a67dfc5b-3c9ff7000001d7ae-26-67b6bba76383 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [RFC PATCH v12 24/26] mm/migrate: apply luf mechanism to unmapping during migration Date: Thu, 20 Feb 2025 14:20:25 +0900 Message-Id: <20250220052027.58847-25-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20250220052027.58847-1-byungchul@sk.com> References: <20250220052027.58847-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrGLMWRmVeSWpSXmKPExsXC9ZZnoe7y3dvSDbbfU7CYs34Nm8XnDf/Y LF5saGe0+Lr+F7PF0099LBaXd81hs7i35j+rxflda1ktdizdx2Rx6cACJovjvQeYLObf+8xm sXnTVGaL41OmMlr8/gFUfHLWZBYHAY/vrX0sHjtn3WX3WLCp1GPzCi2PxXteMnlsWtXJ5rHp 0yR2j3fnzrF7nJjxm8Vj3slAj/f7rrJ5bP1l59E49Rqbx+dNcgF8UVw2Kak5mWWpRfp2CVwZ a+ZsZCq4GlCx6sx8pgbGNw5djJwcEgImEt2vpjLD2Ee+T2MFsdkE1CVu3PgJFhcRMJM42PqH HcRmFrjLJHGgnw3EFhaIkdh8oAeshkVAVWL2h32MIDYvUP3CiefZIWbKS6zecACshhMo/mNG L1ivkICpxLsFl5i6GLmAat6zSfS/fccE0SApcXDFDZYJjLwLGBlWMQpl5pXlJmbmmOhlVOZl Vugl5+duYgSG/rLaP9E7GD9dCD7EKMDBqMTDO6N1W7oQa2JZcWXuIUYJDmYlEd62+i3pQrwp iZVVqUX58UWlOanFhxilOViUxHmNvpWnCAmkJ5akZqemFqQWwWSZODilGhjVTWtvvjIv25Tn LrrsZ0FYmYfr2UY1pbZzv9eE3lRy0WV9mMlgrqWqujx0d7bGJPWi+ltxH/Nj97zwfSIZfvXT ymvbGXyuO7+8fe7kzFOCnboV3A8d7mcekZQyXbj2f8hHqVtpIduKNLP3xjPnO6rc1hc3/MUi 19V90LjTUEOt+NqCXuVHBUosxRmJhlrMRcWJAF4LOdZ5AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrLLMWRmVeSWpSXmKPExsXC5WfdrLt897Z0g8Z/ohZz1q9hs/i84R+b xYsN7YwWX9f/YrZ4+qmPxeLw3JOsFpd3zWGzuLfmP6vF+V1rWS12LN3HZHHpwAImi+O9B5gs 5t/7zGaxedNUZovjU6YyWvz+AVR8ctZkFgdBj++tfSweO2fdZfdYsKnUY/MKLY/Fe14yeWxa 1cnmsenTJHaPd+fOsXucmPGbxWPeyUCP9/uusnksfvGByWPrLzuPxqnX2Dw+b5IL4I/isklJ zcksSy3St0vgylgzZyNTwdWAilVn5jM1ML5x6GLk5JAQMJE48n0aK4jNJqAucePGT2YQW0TA TOJg6x92EJtZ4C6TxIF+NhBbWCBGYvOBHrAaFgFVidkf9jGC2LxA9QsnnmeHmCkvsXrDAbAa TqD4jxm9YL1CAqYS7xZcYprAyLWAkWEVo0hmXlluYmaOqV5xdkZlXmaFXnJ+7iZGYCAvq/0z cQfjl8vuhxgFOBiVeHgfPN6aLsSaWFZcmXuIUYKDWUmEt61+S7oQb0piZVVqUX58UWlOavEh RmkOFiVxXq/w1AQhgfTEktTs1NSC1CKYLBMHp1QDI4PEzeR3q75x/Pd+/fmll+1bdZ+87Gt6 P017ZdpuXi5ZLhdkNY1n18KEHaea2jaZWVjkJrxqNY20m85muV2qY8YB5q+Xfk9dsXRR3Ud3 NpaOQ5bfAx6Gr9j5+lDjb/ko+dzzCSUac82t/keH1k23j2WbwHXt89ubwVNa757c4P9yVV+3 +O3wM0osxRmJhlrMRcWJAO3GVDlgAgAA X-CFilter-Loop: Reflected X-Rspam-User: X-Rspamd-Queue-Id: 6856A4000F X-Rspamd-Server: rspam07 X-Stat-Signature: bpqstbj9f57q1q91b6rhrbuofos55ses X-HE-Tag: 1740028848-9593 X-HE-Meta: U2FsdGVkX1+P7mKz6nPncqfbyUZygyOrQCpmUtkDL+JiSbwz4ibLOOlTpfrmOWp2+tNf/K1HlaaDdcDCa7MGXvXGmqdzQShEZZ5lAZ7ek4AY9MbnWtam+aoRkz8f+gHJTmltw5rS8TLWMCvrDqOKZcSKxCLwZz0R9K2ETi8DG08GCUX2sm2dE0DfxjfrmJhgYhiB+nwEKCUQ7Dqstbvq86FrUI9sOHvsFSealjGICcCDuYyDw+CKch+4ukp36eTZ7hlaNspwriIYzos1vpLLG6qwEreWffl8zOhVV61RHo+NRDh3oaQCEr84LMa8zv4ZJg+bo1rTI9AcGqfIZfMzGLJFjKPYqpc+JDZDA3nEaSm70svIn3cLheedNuX/ML/DAYt68S+YGeihd6CJCBONSXi6guvLEXgoE7Mb5Od3mni+WBPoyHnxsLWF3MQQvQsSfg6A4pJD9feohA/kEreJ9RDKLtSfxLxsWidEtR0JI+xEhDkBE8UBMSpWoasvYXNnnJ1Wz3/DCtJgKSW8zwhKEcm1PqKx1DQD+OtUMc9Yr3xOl+ZaEJA09Ypp59stDUFES/B82/8j42SCGXDnAiT1W9l+kzlkK7K+wBrk5RDH6LWLyc9Qz6zoUZrIfPH7elXBo8caIZTEYogSmIkfHcY28+a9XNqe1M2neU9PZeGDWrQ/s1P8avoYgnVCvax24ncBpRY4zvRDMwbhw6oLgA0n+DQsjES/mbwoXzBOm3x/ujQ0inaeAlc7khZeoBrCBB86Kwr2BQ60HHuiQdKgaNw6dXO333q61UBi+iJrw0yAuTFA0PybLjcFesJxb8tbEujWWCyVTU0tBqMP8WqCiquKOM38Q1HF5LfYi+pgZOHEJX5bccYg0GTylMWRqV3CjX38sdf2eVyguFhyxgWNjDYyJh++eXNUqOQUWo6ak3rCQei1c/LcBwXqjTavnIo2zBFImYYWayYBWdKGDOPGZ9Q Fcw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. Applied the mechanism to unmapping during migration. Signed-off-by: Byungchul Park --- include/linux/mm.h | 2 ++ include/linux/rmap.h | 2 +- mm/migrate.c | 65 ++++++++++++++++++++++++++++++++++---------- mm/rmap.c | 15 ++++++---- mm/swap.c | 2 +- 5 files changed, 63 insertions(+), 23 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 46638e86e8073..5c81c9831bc5d 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -1476,6 +1476,8 @@ static inline void folio_put(struct folio *folio) __folio_put(folio); } +void page_cache_release(struct folio *folio); + /** * folio_put_refs - Reduce the reference count on a folio. * @folio: The folio. diff --git a/include/linux/rmap.h b/include/linux/rmap.h index 683a04088f3f2..cedba4812ccc7 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -660,7 +660,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, int folio_referenced(struct folio *, int is_locked, struct mem_cgroup *memcg, unsigned long *vm_flags); -void try_to_migrate(struct folio *folio, enum ttu_flags flags); +bool try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, diff --git a/mm/migrate.c b/mm/migrate.c index 5e12023dbc75a..6b77efee4ebd7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1172,7 +1172,8 @@ static void migrate_folio_undo_dst(struct folio *dst, bool locked, /* Cleanup src folio upon migration success */ static void migrate_folio_done(struct folio *src, - enum migrate_reason reason) + enum migrate_reason reason, + unsigned short luf_key) { /* * Compaction can migrate also non-LRU pages which are @@ -1183,16 +1184,30 @@ static void migrate_folio_done(struct folio *src, mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + folio_is_file_lru(src), -folio_nr_pages(src)); - if (reason != MR_MEMORY_FAILURE) - /* We release the page in page_handle_poison. */ + /* We release the page in page_handle_poison. */ + if (reason == MR_MEMORY_FAILURE) + luf_flush(luf_key); + else if (!luf_key) folio_put(src); + else { + /* + * Should be the last reference. + */ + if (unlikely(!folio_put_testzero(src))) + VM_WARN_ON(1); + + page_cache_release(src); + mem_cgroup_uncharge(src); + free_unref_page(&src->page, folio_order(src), luf_key); + } } /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_folio_t get_new_folio, free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio **dstp, enum migrate_mode mode, - enum migrate_reason reason, struct list_head *ret) + enum migrate_reason reason, struct list_head *ret, + bool *can_luf) { struct folio *dst; int rc = -EAGAIN; @@ -1208,7 +1223,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ list_del(&src->lru); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, 0); return MIGRATEPAGE_SUCCESS; } @@ -1325,7 +1340,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); + *can_luf = try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); old_page_state |= PAGE_WAS_MAPPED; } @@ -1353,7 +1368,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, unsigned short luf_key) { int rc; int old_page_state = 0; @@ -1407,7 +1422,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, luf_key); return rc; out: @@ -1702,7 +1717,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, unsigned short luf_key) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1719,7 +1734,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, luf_key); /* * The rules are: * Success: folio will be freed @@ -1796,7 +1811,11 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_luf); + LIST_HEAD(dst_folios_luf); bool nosplit = (reason == MR_NUMA_MISPLACED); + unsigned short luf_key; + bool can_luf; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); @@ -1871,9 +1890,11 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_luf = false; rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, - ret_folios); + ret_folios, &can_luf); + /* * The rules are: * Success: folio will be freed @@ -1919,7 +1940,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_luf)) goto out; else goto move; @@ -1933,8 +1955,13 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_luf) { + list_move_tail(&folio->lru, &unmap_folios_luf); + list_add_tail(&dst->lru, &dst_folios_luf); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1954,6 +1981,8 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + luf_key = fold_unmap_luf(); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1967,7 +1996,11 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, 0); + migrate_folios_move(&unmap_folios_luf, &dst_folios_luf, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, luf_key); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1978,6 +2011,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_luf, &dst_folios_luf, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/rmap.c b/mm/rmap.c index ebe91ff1bcb16..b6b61b8103655 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2750,8 +2750,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, * * Tries to remove all the page table entries which are mapping this folio and * replace them with special swap entries. Caller must hold the folio lock. + * Return true if all the mappings are read-only, otherwise false. */ -void try_to_migrate(struct folio *folio, enum ttu_flags flags) +bool try_to_migrate(struct folio *folio, enum ttu_flags flags) { struct rmap_walk_control rwc = { .rmap_one = try_to_migrate_one, @@ -2769,11 +2770,11 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) */ if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | TTU_SYNC | TTU_BATCH_FLUSH))) - return; + return false; if (folio_is_zone_device(folio) && (!folio_is_device_private(folio) && !folio_is_device_coherent(folio))) - return; + return false; /* * During exec, a temporary VMA is setup and later moved. @@ -2793,10 +2794,12 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) else rmap_walk(folio, &rwc); - if (can_luf_test()) + if (can_luf_test()) { fold_batch(tlb_ubc_luf, tlb_ubc_ro, true); - else - fold_batch(tlb_ubc, tlb_ubc_ro, true); + return true; + } + fold_batch(tlb_ubc, tlb_ubc_ro, true); + return false; } #ifdef CONFIG_DEVICE_PRIVATE diff --git a/mm/swap.c b/mm/swap.c index 54b0ba10dbb86..d6c29fdc67ca5 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -84,7 +84,7 @@ static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, * This path almost never happens for VM activity - pages are normally freed * in batches. But it gets used by networking - and for compound pages. */ -static void page_cache_release(struct folio *folio) +void page_cache_release(struct folio *folio) { struct lruvec *lruvec = NULL; unsigned long flags;