From patchwork Fri May 31 09:20:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Byungchul Park X-Patchwork-Id: 13681407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3D4CAC25B75 for ; Fri, 31 May 2024 09:20:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 601716B00A4; Fri, 31 May 2024 05:20:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53CC56B00A5; Fri, 31 May 2024 05:20:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 38DC46B00A6; Fri, 31 May 2024 05:20:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1BF7F6B00A4 for ; Fri, 31 May 2024 05:20:23 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CCF1541169 for ; Fri, 31 May 2024 09:20:22 +0000 (UTC) X-FDA: 82178145084.30.061317F Received: from invmail4.hynix.com (exvmail4.hynix.com [166.125.252.92]) by imf08.hostedemail.com (Postfix) with ESMTP id CA9CA16000E for ; Fri, 31 May 2024 09:20:20 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717147221; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to: references:references; bh=DWL/qYrEsADsHqpSLfnBPP2hWqV8vJfMQfSqpZua+C0=; b=gADcQzxYnyO+s8MdDZxR1JuONCwqarnaKOXCmca2EjYZvSmPD14sXxZ8i/9EDZnAoQ5neT 50b6R7g1QcjH+aaF5oBIEr7ksS+gMYT5wJFamN3/mbq8LVfxH/6a5GRoS0GWnE3GHad9cJ yKkQY46Y/VXP6UDqEEd/cV6KJ8BEXjo= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf08.hostedemail.com: domain of byungchul@sk.com designates 166.125.252.92 as permitted sender) smtp.mailfrom=byungchul@sk.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717147221; a=rsa-sha256; cv=none; b=2lWbin0vhCjdesaqnNCGPrzZBWIPXm3vBw+V8O0Bve6ej0YJJaO05IK6/t5xPo+0F9mMbF qKToeR5QPe2Ho9itWavJ/sIfofaYFn11ftXSRZR+zAwLC+YhaEh4YSNgMaQsfGfsQ3PhOu gbTWvsD92N0lYOvA2rxeTW8pk0w065c= X-AuditID: a67dfc5b-d85ff70000001748-76-6659964c68c9 From: Byungchul Park To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: kernel_team@skhynix.com, akpm@linux-foundation.org, ying.huang@intel.com, vernhao@tencent.com, mgorman@techsingularity.net, hughd@google.com, willy@infradead.org, david@redhat.com, peterz@infradead.org, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, rjgolo@gmail.com Subject: [PATCH v11 11/12] mm, migrate: apply luf mechanism to unmapping during migration Date: Fri, 31 May 2024 18:20:00 +0900 Message-Id: <20240531092001.30428-12-byungchul@sk.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20240531092001.30428-1-byungchul@sk.com> References: <20240531092001.30428-1-byungchul@sk.com> X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFnrCLMWRmVeSWpSXmKPExsXC9ZZnka7PtMg0g+dTVSzmrF/DZvF5wz82 ixcb2hktvq7/xWzx9FMfi8XlXXPYLO6t+c9qcX7XWlaLHUv3MVlcOrCAyeJ47wEmi/n3PrNZ bN40ldni+JSpjBa/fwAVn5w1mcVBwON7ax+Lx85Zd9k9Fmwq9di8Qstj8Z6XTB6bVnWyeWz6 NInd4925c+weJ2b8ZvGYdzLQ4/2+q2weW3/ZeTROvcbm8XmTXABfFJdNSmpOZllqkb5dAlfG ivkTWAqa3SqmnX3E2sC40KKLkZNDQsBE4uf/E4ww9pO+YywgNpuAusSNGz+ZQWwRATOJg61/ 2EFsZoG7TBIH+tm6GDk4hAUiJVb9SwMxWQRUJT53GoBU8AJVX23sY4WYKC+xesMBsCmcQPED f++AbRISMJVY9L8XyOYCqnnPJnHy3T42iAZJiYMrbrBMYORdwMiwilEoM68sNzEzx0QvozIv s0IvOT93EyMw7JfV/onewfjpQvAhRgEORiUe3oCKiDQh1sSy4srcQ4wSHMxKIry/0oFCvCmJ lVWpRfnxRaU5qcWHGKU5WJTEeY2+lacICaQnlqRmp6YWpBbBZJk4OKUaGK3/aSwp6u+uLisV Cd725dPDnmsmFcV87JOzbvk9DtLmvvd6eqv1/F3r9NQuxUz7sbU46YNKTG3WlaqseO6S5uk9 dlOmMmWvbnjFuiyM5db9m3cZQnvWplt7PMwL+jMjt+N3zeplDpqeL64IPrt1MVC1WEQmNHHh nSuFx5inWuoemR6YdFH8mBJLcUaioRZzUXEiAOcOLfV3AgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrNLMWRmVeSWpSXmKPExsXC5WfdrOszLTLNoPmlpMWc9WvYLD5v+Mdm 8WJDO6PF1/W/mC2efupjsTg89ySrxeVdc9gs7q35z2pxftdaVosdS/cxWVw6sIDJ4njvASaL +fc+s1ls3jSV2eL4lKmMFr9/ABWfnDWZxUHQ43trH4vHzll32T0WbCr12LxCy2PxnpdMHptW dbJ5bPo0id3j3blz7B4nZvxm8Zh3MtDj/b6rbB6LX3xg8tj6y86jceo1No/Pm+QC+KO4bFJS czLLUov07RK4MlbMn8BS0OxWMe3sI9YGxoUWXYycHBICJhJP+o6xgNhsAuoSN278ZAaxRQTM JA62/mEHsZkF7jJJHOhn62Lk4BAWiJRY9S8NxGQRUJX43GkAUsELVH21sY8VYqK8xOoNB8Cm cALFD/y9wwhiCwmYSiz638s4gZFrASPDKkaRzLyy3MTMHFO94uyMyrzMCr3k/NxNjMAgXlb7 Z+IOxi+X3Q8xCnAwKvHwBlREpAmxJpYVV+YeYpTgYFYS4f2VDhTiTUmsrEotyo8vKs1JLT7E KM3BoiTO6xWemiAkkJ5YkpqdmlqQWgSTZeLglGpgvDH3m6fsxadPdXK/X/Z2O8Z0kOHE/O6Z GtEfWWX/d7MUlVqvlY1/lPUuo1rhqF+PaMy2zODzFs/sfn6S3nbvEq/n3bki7ybZ/qkWlr2+ kW8h8+QYv3ibgKDaurkKrd+59xww2tSUwf+24uyluA7zq0avffRPp3r+4J11ni8yaElWkZqE 1O9zSizFGYmGWsxFxYkAOqgFhV4CAAA= X-CFilter-Loop: Reflected X-Stat-Signature: p3g73k1w5aebmzunq4pz9mkgwmut3w1a X-Rspamd-Queue-Id: CA9CA16000E X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1717147220-803930 X-HE-Meta: U2FsdGVkX18isTSonzsYf3y+A1sgWCZjOen/ZeJIi/BfmHtaYLOtOIszzSvs3+BE36F0fhpt6yqMpF5/ml17c4JHZNtCRc6sThyXxp/BjKN8cySTQPj4dtnbGKJVw8t4PNw/1CZ3I/AOGDz3AaxsucjrUaHNtkIzIDbtnJzbFF7fHhcxiF3wouO7LNMJbGHPMiRaIrzFsBq/SFCHHYjqqHPFyc+/hd4zODqFj0MtNJq36DwfYH9FGsf4MOPF/OB3dkW9FXERG4gCA3jisWKR63e7eyr77+tg9cqX9Ad/fqcdyM0aRS6mUeDH02+WwAXcd0mTGsqOAwULbtc8ArMrUtNB70AdX+KuSUwoSUNWRWAxpdkWW4YsPv8b8hMQYmjMC7kNC8j/NXf+rbsIhZ35jRa7fQ4KJZ0dSUCN1quWoCG1DY+VmaLPiM9/sKsf7y3sRsYrZU/gHaalBiD6Px0UxNqXOcKDfO1rtoWNv6W5sW5RSdzFn7YHFg4SKGe4BPTpxmKbfFuiL3rQbId3+3oGYrc7oIoN2zc2bzAwKWbLTXw3sSWTtkVz3rIjovKkoQKrlCol8VNIC3rnFG+WV5ndAS3kwBKCzHj73A4NvrYrtrwKLaEsJp/QekObtNJt1COMKWSFRcLEsjYfdREg2qjHx1BZlS1HTHlL4xxlRunMy7J08A6iZPFY8toa6FkmyaBKs1PjdSJbcL/czNdRXg0wAnBlhnsJ7gpR4IYnA/1FB7TogaUg3WBj++bgVZmCS67qIwbJDUVmOPZ2tupWcqZZZ2qJy0m7ReKhevEbcBjirIA2d5EAGSWeCqeLfpRZN3XYjsEiQ4r7USVI7U6mPTG0WUI4COElgDPJEY6sprX/9nY86R5jHb0cfA/DLq+oWzmCVkDz+fxUr2fqxkzRv3QcOjnDZ+ArbVNmmpkwq4T+cE6JFQBUmQtfXPwQ4c30aDi8IzHPBfkKidg6n0Nw3jB Zng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: A new mechanism, LUF(Lazy Unmap Flush), defers tlb flush until folios that have been unmapped and freed, eventually get allocated again. It's safe for folios that had been mapped read only and were unmapped, since the contents of the folios don't change while staying in pcp or buddy so we can still read the data through the stale tlb entries. Applied the mechanism to unmapping during migration. Signed-off-by: Byungchul Park --- include/linux/rmap.h | 2 +- mm/migrate.c | 56 ++++++++++++++++++++++++++++++++------------ mm/rmap.c | 9 ++++--- 3 files changed, 48 insertions(+), 19 deletions(-) diff --git a/include/linux/rmap.h b/include/linux/rmap.h index bb53e5920b88..6aca569e342b 100644 --- a/include/linux/rmap.h +++ b/include/linux/rmap.h @@ -660,7 +660,7 @@ static inline int folio_try_share_anon_rmap_pmd(struct folio *folio, int folio_referenced(struct folio *, int is_locked, struct mem_cgroup *memcg, unsigned long *vm_flags); -void try_to_migrate(struct folio *folio, enum ttu_flags flags); +bool try_to_migrate(struct folio *folio, enum ttu_flags flags); void try_to_unmap(struct folio *, enum ttu_flags flags); int make_device_exclusive_range(struct mm_struct *mm, unsigned long start, diff --git a/mm/migrate.c b/mm/migrate.c index 6c22a1402923..6da8335cdf4c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1067,7 +1067,8 @@ static void migrate_folio_undo_dst(struct folio *dst, bool locked, /* Cleanup src folio upon migration success */ static void migrate_folio_done(struct folio *src, - enum migrate_reason reason) + enum migrate_reason reason, + unsigned short int ugen) { /* * Compaction can migrate also non-LRU pages which are @@ -1078,8 +1079,12 @@ static void migrate_folio_done(struct folio *src, mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + folio_is_file_lru(src), -folio_nr_pages(src)); - if (reason != MR_MEMORY_FAILURE) - /* We release the page in page_handle_poison. */ + /* We release the page in page_handle_poison. */ + if (reason == MR_MEMORY_FAILURE) + check_luf_flush(ugen); + else if (ugen) + folio_put_ugen(src, ugen); + else folio_put(src); } @@ -1087,7 +1092,8 @@ static void migrate_folio_done(struct folio *src, static int migrate_folio_unmap(new_folio_t get_new_folio, free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio **dstp, enum migrate_mode mode, - enum migrate_reason reason, struct list_head *ret) + enum migrate_reason reason, struct list_head *ret, + bool *can_luf) { struct folio *dst; int rc = -EAGAIN; @@ -1103,7 +1109,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ list_del(&src->lru); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, 0); return MIGRATEPAGE_SUCCESS; } @@ -1220,7 +1226,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); + *can_luf = try_to_migrate(src, mode == MIGRATE_ASYNC ? TTU_BATCH_FLUSH : 0); old_page_state |= PAGE_WAS_MAPPED; } @@ -1248,7 +1254,7 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, struct folio *src, struct folio *dst, enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct list_head *ret, unsigned short int ugen) { int rc; int old_page_state = 0; @@ -1302,7 +1308,7 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); - migrate_folio_done(src, reason); + migrate_folio_done(src, reason, ugen); return rc; out: @@ -1591,7 +1597,7 @@ static void migrate_folios_move(struct list_head *src_folios, struct list_head *ret_folios, struct migrate_pages_stats *stats, int *retry, int *thp_retry, int *nr_failed, - int *nr_retry_pages) + int *nr_retry_pages, unsigned short int ugen) { struct folio *folio, *folio2, *dst, *dst2; bool is_thp; @@ -1608,7 +1614,7 @@ static void migrate_folios_move(struct list_head *src_folios, rc = migrate_folio_move(put_new_folio, private, folio, dst, mode, - reason, ret_folios); + reason, ret_folios, ugen); /* * The rules are: * Success: folio will be freed @@ -1685,7 +1691,11 @@ static int migrate_pages_batch(struct list_head *from, int rc, rc_saved = 0, nr_pages; LIST_HEAD(unmap_folios); LIST_HEAD(dst_folios); + LIST_HEAD(unmap_folios_luf); + LIST_HEAD(dst_folios_luf); bool nosplit = (reason == MR_NUMA_MISPLACED); + unsigned short int ugen; + bool can_luf; VM_WARN_ON_ONCE(mode != MIGRATE_ASYNC && !list_empty(from) && !list_is_singular(from)); @@ -1748,9 +1758,11 @@ static int migrate_pages_batch(struct list_head *from, continue; } + can_luf = false; rc = migrate_folio_unmap(get_new_folio, put_new_folio, private, folio, &dst, mode, reason, - ret_folios); + ret_folios, &can_luf); + /* * The rules are: * Success: folio will be freed @@ -1796,7 +1808,8 @@ static int migrate_pages_batch(struct list_head *from, /* nr_failed isn't updated for not used */ stats->nr_thp_failed += thp_retry; rc_saved = rc; - if (list_empty(&unmap_folios)) + if (list_empty(&unmap_folios) && + list_empty(&unmap_folios_luf)) goto out; else goto move; @@ -1810,8 +1823,13 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_succeeded += is_thp; break; case MIGRATEPAGE_UNMAP: - list_move_tail(&folio->lru, &unmap_folios); - list_add_tail(&dst->lru, &dst_folios); + if (can_luf) { + list_move_tail(&folio->lru, &unmap_folios_luf); + list_add_tail(&dst->lru, &dst_folios_luf); + } else { + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + } break; default: /* @@ -1831,6 +1849,8 @@ static int migrate_pages_batch(struct list_head *from, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Should be before try_to_unmap_flush() */ + ugen = try_to_unmap_luf(); /* Flush TLBs for all unmapped folios */ try_to_unmap_flush(); @@ -1844,7 +1864,11 @@ static int migrate_pages_batch(struct list_head *from, migrate_folios_move(&unmap_folios, &dst_folios, put_new_folio, private, mode, reason, ret_folios, stats, &retry, &thp_retry, - &nr_failed, &nr_retry_pages); + &nr_failed, &nr_retry_pages, 0); + migrate_folios_move(&unmap_folios_luf, &dst_folios_luf, + put_new_folio, private, mode, reason, + ret_folios, stats, &retry, &thp_retry, + &nr_failed, &nr_retry_pages, ugen); } nr_failed += retry; stats->nr_thp_failed += thp_retry; @@ -1855,6 +1879,8 @@ static int migrate_pages_batch(struct list_head *from, /* Cleanup remaining folios */ migrate_folios_undo(&unmap_folios, &dst_folios, put_new_folio, private, ret_folios); + migrate_folios_undo(&unmap_folios_luf, &dst_folios_luf, + put_new_folio, private, ret_folios); return rc; } diff --git a/mm/rmap.c b/mm/rmap.c index 459d4d1631f0..b8b977278a1b 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2635,8 +2635,9 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, * * Tries to remove all the page table entries which are mapping this folio and * replace them with special swap entries. Caller must hold the folio lock. + * Return true if all the mappings are read-only, otherwise false. */ -void try_to_migrate(struct folio *folio, enum ttu_flags flags) +bool try_to_migrate(struct folio *folio, enum ttu_flags flags) { struct rmap_walk_control rwc = { .rmap_one = try_to_migrate_one, @@ -2655,11 +2656,11 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) */ if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | TTU_SYNC | TTU_BATCH_FLUSH))) - return; + return false; if (folio_is_zone_device(folio) && (!folio_is_device_private(folio) && !folio_is_device_coherent(folio))) - return; + return false; /* * During exec, a temporary VMA is setup and later moved. @@ -2684,6 +2685,8 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) fold_ubc(tlb_ubc_luf, tlb_ubc_ro); else fold_ubc(tlb_ubc, tlb_ubc_ro); + + return can_luf; } #ifdef CONFIG_DEVICE_PRIVATE