From patchwork Tue Mar 19 15:47:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13596811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 393A5CD11BF for ; Tue, 19 Mar 2024 15:48:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2AC8D6B0085; Tue, 19 Mar 2024 11:48:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25BCB6B0087; Tue, 19 Mar 2024 11:48:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1246C6B0088; Tue, 19 Mar 2024 11:48:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 04B986B0085 for ; Tue, 19 Mar 2024 11:48:05 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id C4EA640196 for ; Tue, 19 Mar 2024 15:48:04 +0000 (UTC) X-FDA: 81914219688.12.8F5D93B Received: from wfout3-smtp.messagingengine.com (wfout3-smtp.messagingengine.com [64.147.123.146]) by imf04.hostedemail.com (Postfix) with ESMTP id F32964000E for ; Tue, 19 Mar 2024 15:48:00 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b="J 0vcsUt"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=X0tz1iv0; spf=pass (imf04.hostedemail.com: domain of zi.yan@sent.com designates 64.147.123.146 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710863281; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=nessaiOBTaNCwP7gjUOn86nFinuzdOOpq2kbEfd5EGw=; b=eCUh4b6BQ1bIVP31Ih0fosIr4EVziIMEIQ5XVbq5zUmEYlu7WCMqmXBiI1wA8phqgYZMX1 MTs+Tp5/elyW2kwvNXRk1Ee1/uAhRvLHvCv2mektstDqkAo8kRTlEZcTHijtjzTekCrUU0 DSAjGp8zuoDWzJmXWz9WFy2tKnV7cbk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710863281; a=rsa-sha256; cv=none; b=dU+Hb4JSxKYf2VncSU2eIl38VzdKpzeJLMbtsRnRvuLhMzJkXcSrGVg5i0KaQV/Fb7lKJY 5jB85NBDflfnqv+xngh5ZuT/asnZRlYTRhngZNf1O/6aswpJ4K6HS6lsF9N9/eYkcGCo9O AyIW67KvsPrsZwGZsbenA96qxKDTsB0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=sent.com header.s=fm2 header.b="J 0vcsUt"; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=X0tz1iv0; spf=pass (imf04.hostedemail.com: domain of zi.yan@sent.com designates 64.147.123.146 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (policy=none) header.from=sent.com Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfout.west.internal (Postfix) with ESMTP id 790B21C000F7; Tue, 19 Mar 2024 11:47:58 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Tue, 19 Mar 2024 11:47:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :reply-to:subject:subject:to:to; s=fm2; t=1710863277; x= 1710949677; bh=nessaiOBTaNCwP7gjUOn86nFinuzdOOpq2kbEfd5EGw=; b=J 0vcsUthTSQYrwXZb6x4NqkHmEFK8yNAKPH0eud1LI9GxXiA4y6Mh+lULGpL352Z8 6Ck0j00psKFiBfpR6y/Bh48vki09ys+eSlehhtyprmArhs+X2kO+jALDpO/6bL8M gh6VuMoCSrFea8mcbIDfWqPsj4wHeqWAzMOu9tZMGKAH9LABV2ZI8tiDa1Dg/E98 kgfdPdu/VF34hy/G/UDENvx3OPytMZpFr/ZZXG2zpjx0gA0JuKdjBEiM9L4YFlrg 0DaYh6Wka0wt1j6NZvzLqcEMGGbbAN+iWBCZQndZazIOWHxy0ff04YvnO4TXRMG9 qpsJ9p4WHuS+5616exHFA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:message-id:mime-version:reply-to:reply-to :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; t=1710863277; x=1710949677; bh=n essaiOBTaNCwP7gjUOn86nFinuzdOOpq2kbEfd5EGw=; b=X0tz1iv04y3dkcwvP OLigajGJCLsEEzXoX99mR0e/P/WBdFXnAeyY8pqKsydTTcuSlQu1IQHo1zr0kvA+ aoSO+AamEqs2vocidbud0M3pCfJQvdUMOakknueQcV+YAk6YaQCoNmaRTriJYkK1 gub+QxUCMSoYPxzp6He5dTlfAHtqpzV+aDqdPs95DE0FVh74BKFI6PvcySUXKUu/ mn6i0dQbUh4EqnjyCZuuo/ndrbhO86oed4bBUKAvIDN9pgxOwjJNfSLhioGKxzYg gzv3fm2x+oAuHGe68LnJX79hs2+sfce+KgOo+2xdk8SmBSMSBqfbvIE1LCBOsr84 KBW+w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrledtgdehgecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvfevufffkfforhggtgfgsehtkeertdertdejnecuhfhrohhmpegkihcujggr nhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucggtffrrghtthgvrhhnpeetudevhe etheehieetiefhjeevjeeltdfgvdeiueeiudetffdtvedthfetvedtffenucffohhmrghi nhepkhgvrhhnvghlrdhorhhgnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghmpe hmrghilhhfrhhomhepiihirdihrghnsehsvghnthdrtghomh X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Tue, 19 Mar 2024 11:47:56 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org Cc: Zi Yan , Andrew Morton , "Matthew Wilcox (Oracle)" , Yang Shi , Huang Ying , "Kirill A . Shutemov" , Ryan Roberts , Baolin Wang , "Yin, Fengwei" , linux-kernel@vger.kernel.org Subject: [PATCH v3] mm/migrate: split source folio if it is on deferred split list Date: Tue, 19 Mar 2024 11:47:53 -0400 Message-ID: <20240319154753.253262-1-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 Reply-To: Zi Yan MIME-Version: 1.0 X-Stat-Signature: bqjqpwpsqdipb5qdmdan5fo6dgzoqrzf X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: F32964000E X-Rspam-User: X-HE-Tag: 1710863280-114013 X-HE-Meta: U2FsdGVkX1/zMoFFbBO5oWZZXG8USR0o71Pj5a7XK/vZ7wtWQLwTFPeqTBy1c5mf3Kg8EAeYjbwAE596yo1nYQkwIexxK/1JPsjkO4SlG12xlfxjy+uaQMkbG5js1Nfl0zr7hRERSi0rPuA8iq65aSgZMJFT+d9gZ3KNmRij88GqKVOd47crBnTqx0Mpf91REn4IUe0BYlufPAWm7ZUdu0yRgvWoyZDVBRtS03WAeb2x+da9QPMnyyUrePj4ZVSQ9R1SKDElEJVs/HWLbL6DynIlZmTzW6S/adPFxh1ddHi5nfOedFlrLgnYiE6RGdNJSEUCDIMWXodveQVUIiV2jn+FnReJsqsUF6K+8yyIO3qplgvIw5fhFVZyvJpLVFgl4knxzqB5+3J/2b1NIYmI8BOTXV5ZRXV9flC/iG9a8dIUnZlp+0FBSuB7PKee5uPEz6UmvBWY32qj/X5RXpTvNsF6ev67kdIGdqPSySB7Mlc11LCsnnxYx5E1pHtZv3G/aMbodfgmGOfQ2Z7ASP6D7F2wNvPeTeNglINJRszsfdgJNiv+ecqbSqCA4rLMhRJEZvsrl1enj44LxptPG1qbJzAzgQuH8KHn6yhSoNPOWRRjmAKsJ+9WnzP/u+TBT0t1Pi7T3M1T0rvN8lHFFjyvfJbC+/VQUKZ3Zdaql9MCWI4bJGujmGJFPUGvW+FUxh2zjz6rMvHz5gaU8E3RrCyknp2LxiCSszENbz/YUH4Jhk4GUGctNszthsGITf+LBjjPccJqbXtOTIe+Hb8X7Yf2qF5LJonYVfk65Jm4ROpKz7JqfYn8ijkTvS0O+Tbv2RxkNno6NEsazW5nTzdcTK4RGKj4mb4edp9cTs+91x3572MUhhiz9qPBbB2NEXLE2Q+5A8R2IQdrOoQrPwqAAzDlmLSmahSiX1rGKn4oqwOZsEVAuzEP66l7CcD8DVUZGRWBgTOG1WLTHiz4p8IND5r 5Zi9PMYG /gwkRGosXD136orSzirgVZy6WxFbTnis/qO1orm+8/lK7u/fsyeAMivejmagYgUkZrxVqVv7oB5Y4t8vjbEqdW1btF64mRhAEmcBGWB8oT8g5AEDbf+8wZdLzbXF1M+S6NUBVN3TWVfN6U0+k9TIsmKfK1UzGTldw0Zxm8nor3acV1kmuwaswwFAFYycl0flu1wCRaKZ9KuCJkut9D3f0Da9SxyO4kwV/oyWLJcPlRTgVcRdEmJAjsYxldf9X/2FkNhMQ8C5UuW3T5Q2VjMubY7fhPPutJuirM0//iWe7+SAEVMHNdnXfQ4RBNbRy3JX9hoegeHYfm+WVqJ0iqc0nErZYP3fldrD98EFIzk3XnfkeUMcoangSxscES1UAtYIdKAyuL5MjIJNEtFN3cqEF9HOeu+QXB76rCSDfxE9wFqmRtZTud5uHzDhXJ6v6EYyydhxhwnYdBY7bJXU4kRGecez/OX3Ay6OuLR/7P0NEzu+bcUVRgaJrav/9QodYE18oM/NyICNEaAuarQMmuiCmXajr7nBMXRGkO4c9luWN9LAWoTu/KnMQ3mvLlVLFEvz7edXF X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan If the source folio is on deferred split list, it is likely some subpages are not used. Split it before migration to avoid migrating unused subpages. Commit 616b8371539a6 ("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1]. From v2: 1. Split the source folio instead of migrating it (per Matthew Wilcox)[2]. From v1: 1. Used dst to get correct deferred split list after migration (per Ryan Roberts). [1]: https://lore.kernel.org/linux-mm/03CE3A00-917C-48CC-8E1C-6A98713C817C@nvidia.com/ [2]: https://lore.kernel.org/linux-mm/Ze_P6xagdTbcu1Kz@casper.infradead.org/ Fixes: 616b8371539a ("mm: thp: enable thp migration in generic path") Signed-off-by: Zi Yan --- mm/huge_memory.c | 22 ------------------ mm/internal.h | 23 +++++++++++++++++++ mm/migrate.c | 60 +++++++++++++++++++++++++++++++++++++++--------- 3 files changed, 72 insertions(+), 33 deletions(-) base-commit: 384df0d61b071bb7a38ae63bae9b59d088335911 diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9859aa4f7553..c6d4d0cdf4b3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct mem_cgroup *memcg = folio_memcg(folio); - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - if (memcg) - return &memcg->deferred_split_queue; - else - return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - return &pgdat->deferred_split_queue; -} -#endif - void folio_prep_large_rmappable(struct folio *folio) { if (!folio || !folio_test_large(folio)) diff --git a/mm/internal.h b/mm/internal.h index d1c69119b24f..8fa36e84463a 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct mem_cgroup *memcg = folio_memcg(folio); + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + if (memcg) + return &memcg->deferred_split_queue; + else + return &pgdat->deferred_split_queue; +} +#else +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + return &pgdat->deferred_split_queue; +} +#endif + + /* * mm/mmap.c */ diff --git a/mm/migrate.c b/mm/migrate.c index 73a052a382f1..3fd58b267e37 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1654,25 +1654,63 @@ static int migrate_pages_batch(struct list_head *from, /* * Large folio migration might be unsupported or - * the allocation might be failed so we should retry - * on the same folio with the large folio split + * the folio is on deferred split list so we should + * retry on the same folio with the large folio split * to normal folios. * * Split folios are put in split_folios, and * we will migrate them after the rest of the * list is processed. */ - if (!thp_migration_supported() && is_thp) { - nr_failed++; - stats->nr_thp_failed++; - if (!try_split_folio(folio, split_folios)) { - stats->nr_thp_split++; - stats->nr_split++; + if (is_thp) { + bool is_on_deferred_list = false; + + /* + * Check without taking split_queue_lock to + * reduce locking overheads. The worst case is + * that if the folio is put on the deferred + * split list after the check, it will be + * migrated and not put back on the list. + * The migrated folio will not be split + * via shrinker during memory pressure. + */ + if (!data_race(list_empty(&folio->_deferred_list))) { + struct deferred_split *ds_queue; + unsigned long flags; + + ds_queue = + get_deferred_split_queue(folio); + spin_lock_irqsave(&ds_queue->split_queue_lock, + flags); + /* + * Only check if the folio is on + * deferred split list without removing + * it. Since the folio can be on + * deferred_split_scan() local list and + * removing it can cause the local list + * corruption. Folio split process + * below can handle it with the help of + * folio_ref_freeze(). + */ + is_on_deferred_list = + !list_empty(&folio->_deferred_list); + spin_unlock_irqrestore(&ds_queue->split_queue_lock, + flags); + } + if (!thp_migration_supported() || + is_on_deferred_list) { + nr_failed++; + stats->nr_thp_failed++; + if (!try_split_folio(folio, + split_folios)) { + stats->nr_thp_split++; + stats->nr_split++; + continue; + } + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); continue; } - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - continue; } rc = migrate_folio_unmap(get_new_folio, put_new_folio,