From patchwork Mon Mar 11 19:58:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 13589299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17ED2C5475B for ; Mon, 11 Mar 2024 19:58:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AB29F6B00FE; Mon, 11 Mar 2024 15:58:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A601B6B0101; Mon, 11 Mar 2024 15:58:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9014C6B00FF; Mon, 11 Mar 2024 15:58:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7E0716B011E for ; Mon, 11 Mar 2024 15:58:54 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 23BDF1C0C2A for ; Mon, 11 Mar 2024 19:58:54 +0000 (UTC) X-FDA: 81885821388.17.8CAD3DD Received: from fhigh3-smtp.messagingengine.com (fhigh3-smtp.messagingengine.com [103.168.172.154]) by imf30.hostedemail.com (Postfix) with ESMTP id 5443180018 for ; Mon, 11 Mar 2024 19:58:52 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b="o wxgzEg"; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="d0/uvwRh"; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf30.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.154 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710187132; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=A/MBFnXXdSSgAn+JMHfZprV2OLIqaCRbl8i6tPL49MY=; b=VZjaIoodfCIWuuR0GcQ1VQs5dUBth6JmMRU6R24G1zdzJTNAfcU6qqpk4af62HyuL15sO1 Vtg3gKS0Mpu7jBbJauD3w0iztzO3QU/LNPlE7IxcH5a1tKZial+JSBGBEd01zWIgCuZt3Q QrDpLZZuXkwzadKVgmJ2OAPABIqO+c0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=sent.com header.s=fm1 header.b="o wxgzEg"; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="d0/uvwRh"; dmarc=pass (policy=none) header.from=sent.com; spf=pass (imf30.hostedemail.com: domain of zi.yan@sent.com designates 103.168.172.154 as permitted sender) smtp.mailfrom=zi.yan@sent.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710187132; a=rsa-sha256; cv=none; b=2xPrc4z5tLLDkfQuBEGx8vysZfVPTeRpB+XsWDU14cBsVWUtawb0F9ZjH3JEvDOEmEP3MS snGYvqN6ivwBxi+hbsEeC6PjUCihFulHZuGzQ7YImf/tFIlMAO1dmGvok9RgyqbhfxC/iK sRmntQJEYkIBTNx9Vim34HfvIksGvB8= Received: from compute7.internal (compute7.nyi.internal [10.202.2.48]) by mailfhigh.nyi.internal (Postfix) with ESMTP id A6EB411400E2; Mon, 11 Mar 2024 15:58:51 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute7.internal (MEProxy); Mon, 11 Mar 2024 15:58:51 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=cc :cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :reply-to:subject:subject:to:to; s=fm1; t=1710187131; x= 1710273531; bh=A/MBFnXXdSSgAn+JMHfZprV2OLIqaCRbl8i6tPL49MY=; b=o wxgzEgPviaIEemket13I0jAVdGZHZxIDDtw/tQDKpuGKZMhiLYHwwZJoO/1l2coz Oa/A478hyUdh4SiG31PGTNntuctQuzT85RXwY76sBHqWiG+JRk/UPIEoSioNFEY0 RyQ5LaQsyv9Cv8ebPHUOzQVerfANAJXbdcxKD6OQzmKym0StFPr1PJWw7XulUCiC d3mt4qrQiSyJyo7M2oKw0bVn5M7bAg49bd2UUMjhv23tFMGrnfO16NNTf6/NxZqg cwvEgqR4kNxgq/Qm0jrGZAfxyaogKkz4azmhzddxaSFj1TJfYnOxA2sAecq4cNmg kPABiC1PILMJluCdOoFrw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:content-type:date:date:feedback-id:feedback-id :from:from:in-reply-to:message-id:mime-version:reply-to:reply-to :subject:subject:to:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; t=1710187131; x=1710273531; bh=A /MBFnXXdSSgAn+JMHfZprV2OLIqaCRbl8i6tPL49MY=; b=d0/uvwRhXJgD9iqcn WBcsjvIJvpUjudTVTN5GbDN9kf4Nqa7A+v05yqcaDHtbr+zoq93ruVn/WjvzYCjW 3jahNJJ63N5q9rVWzuGscUNM6U8KCR4RWKzmJtChIxlV/gsxdCLTkJz1Tz5UspTy RfHRmgQ9uJjvqowS+wAJsWpm/waTr9L4EBZRCycIz0QxtiDa+sYHCo4szuW4GA9p PoW2RpQ1MsWKLBWoCXT4uX99IKUCT2aeT4VxbLnqIhgtMMaMvTjYvECPl5mvLigm RVbLtyHLMTIqjhcmFx5m6bCKUVDUD0ArJub6eQiYCsIYwNeebPRr4ePIwAH71mqf H3R9w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrjedugddufedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhephffvvefufffkofhrgggtgfesthekredtredtjeenucfhrhhomhepkghiucgj rghnuceoiihirdihrghnsehsvghnthdrtghomheqnecuggftrfgrthhtvghrnhepteduve ehteehheeiteeihfejveejledtgfdvieeuiedutefftdevtdfhteevtdffnecuffhomhgr ihhnpehkvghrnhgvlhdrohhrghenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmh epmhgrihhlfhhrohhmpeiiihdrhigrnhesshgvnhhtrdgtohhm X-ME-Proxy: Feedback-ID: iccd040f4:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 11 Mar 2024 15:58:50 -0400 (EDT) From: Zi Yan To: linux-mm@kvack.org Cc: Zi Yan , Andrew Morton , "Matthew Wilcox (Oracle)" , Yang Shi , Huang Ying , "Kirill A . Shutemov" , Ryan Roberts , linux-kernel@vger.kernel.org Subject: [PATCH v2] mm/migrate: put dest folio on deferred split list if source was there. Date: Mon, 11 Mar 2024 15:58:48 -0400 Message-ID: <20240311195848.135067-1-zi.yan@sent.com> X-Mailer: git-send-email 2.43.0 Reply-To: Zi Yan MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: gbahug7ftfnrdsiirxmyibdpnj783uqy X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 5443180018 X-HE-Tag: 1710187132-368519 X-HE-Meta: U2FsdGVkX1+pq4BwoYGbwejoJawe0pLwdci+KBu2HIxtXVvL0WGKb4TXZs6IetFEjOb+FbxEyl1PZh3NsnpXgu0iKVjy9YYETlN8chfvlDdGqx+oSyWNrX662cGH6uK/AlqlPJ1DWj3CRcatDHlFqSUYJqiOFSIe70WohMwLOR7ITh7pqJJK+atsXTRDRqROPtVHSCu6NdqKjlRPwUbWsrB6RjM1FvCPSX0j/I7xRNh5+YBG4byGGXw/qlXCKERCZneV7gHGDDsB2rJWhGIN0su2e2sbFdbBufwNg2IruuRiI1QICqsh13YKgMLsq/jm0qmbGYjd/0xaNP7yt0z80C+tkZinf3HqCcjrtuRgcpGAGSKGQVVetPTbi6rHIh6CjTcuw4HwgErXubYzGQ0TDI0a9dHq+bitu9/+prujdXuDiEZujlwzX6AIofZUu3Xv3YQJBSAJQHPD31+r2UXSYaDSWaGeRkHhx+TjvqCf+7FjQvCvqi5P3oLboiXnK17o2nEys3ri/aQwIEOv/kxISLioWRNZSUvhO4hgU+Lbw6pmMJ5NyF9GCrd4SGVudck8tpPvOOUOCHDXUs7/KWQFj+wJlFnOcAffxMYJQLmXMdQd+lFcdW0XbTcamjrQwsF8qqoUP7UWOoUAwElYzRqnDOyoZxm84/noIhxjOwAJq1hhgW7ExjAWkfHK8PAIBZ3DICNFHiDKO6Ib7BrjnJy0Y19XV63/05nTClFxNX3fh9f0pG4VpRsX2nv1ImI9kJAnUg8mYSxUpAwdd4RWI4TjorYRLb5kyuA35oePGAsrBIkN9O0U9kDLX21eQDqPCnXku7YrEJFzLEAzah8F2TgGSMtqnR/fbt9Tg2sONeZrs7p1p4siEcZs8cVJgM+xQHZZ5tezBh5w5vTsh3ZTkZEzDnfi0gelOLfb3WYO5HrGvqz7p1StekiD3K+LPhZRQlqTsuhYH2AedmHNGRCXEXa sUbS9vWt JLEKvdEl5lLUvlwSJjk6GMtEKAGBO3w2n23FPBEy5R3j2WHpe+i62qKysqxpzJ4K5j/wCR9okpy1nJabHXuojT8Whka8IcGliDpEs7tFNkEvDJ8I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Zi Yan Commit 616b8371539a6 ("mm: thp: enable thp migration in generic path") did not check if a THP is on deferred split list before migration, thus, the destination THP is never put on deferred split list even if the source THP might be. The opportunity of reclaiming free pages in a partially mapped THP during deferred list scanning is lost, but no other harmful consequence is present[1]. Checking source folio deferred split list status before page unmapped and add destination folio to the list if source was after migration. [1]: https://lore.kernel.org/linux-mm/03CE3A00-917C-48CC-8E1C-6A98713C817C@nvidia.com/ From v1: 1. Used dst to get correct deferred split list after migration (per Ryan Roberts). Fixes: 616b8371539a ("mm: thp: enable thp migration in generic path") Signed-off-by: Zi Yan --- mm/huge_memory.c | 22 ---------------------- mm/internal.h | 23 +++++++++++++++++++++++ mm/migrate.c | 26 +++++++++++++++++++++++++- 3 files changed, 48 insertions(+), 23 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 9859aa4f7553..c6d4d0cdf4b3 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -766,28 +766,6 @@ pmd_t maybe_pmd_mkwrite(pmd_t pmd, struct vm_area_struct *vma) return pmd; } -#ifdef CONFIG_MEMCG -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct mem_cgroup *memcg = folio_memcg(folio); - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - if (memcg) - return &memcg->deferred_split_queue; - else - return &pgdat->deferred_split_queue; -} -#else -static inline -struct deferred_split *get_deferred_split_queue(struct folio *folio) -{ - struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); - - return &pgdat->deferred_split_queue; -} -#endif - void folio_prep_large_rmappable(struct folio *folio) { if (!folio || !folio_test_large(folio)) diff --git a/mm/internal.h b/mm/internal.h index d1c69119b24f..8fa36e84463a 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1107,6 +1107,29 @@ struct page *follow_trans_huge_pmd(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmd, unsigned int flags); +#ifdef CONFIG_MEMCG +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct mem_cgroup *memcg = folio_memcg(folio); + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + if (memcg) + return &memcg->deferred_split_queue; + else + return &pgdat->deferred_split_queue; +} +#else +static inline +struct deferred_split *get_deferred_split_queue(struct folio *folio) +{ + struct pglist_data *pgdat = NODE_DATA(folio_nid(folio)); + + return &pgdat->deferred_split_queue; +} +#endif + + /* * mm/mmap.c */ diff --git a/mm/migrate.c b/mm/migrate.c index 73a052a382f1..591e65658535 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1037,7 +1038,10 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, enum { PAGE_WAS_MAPPED = BIT(0), PAGE_WAS_MLOCKED = BIT(1), - PAGE_OLD_STATES = PAGE_WAS_MAPPED | PAGE_WAS_MLOCKED, + PAGE_WAS_ON_DEFERRED_LIST = BIT(2), + PAGE_OLD_STATES = PAGE_WAS_MAPPED | + PAGE_WAS_MLOCKED | + PAGE_WAS_ON_DEFERRED_LIST, }; static void __migrate_folio_record(struct folio *dst, @@ -1168,6 +1172,17 @@ static int migrate_folio_unmap(new_folio_t get_new_folio, folio_lock(src); } locked = true; + if (folio_test_large_rmappable(src) && + !list_empty(&src->_deferred_list)) { + struct deferred_split *ds_queue = get_deferred_split_queue(src); + + spin_lock(&ds_queue->split_queue_lock); + ds_queue->split_queue_len--; + list_del_init(&src->_deferred_list); + spin_unlock(&ds_queue->split_queue_lock); + old_page_state |= PAGE_WAS_ON_DEFERRED_LIST; + } + if (folio_test_mlocked(src)) old_page_state |= PAGE_WAS_MLOCKED; @@ -1307,6 +1322,15 @@ static int migrate_folio_move(free_folio_t put_new_folio, unsigned long private, if (old_page_state & PAGE_WAS_MAPPED) remove_migration_ptes(src, dst, false); + if (old_page_state & PAGE_WAS_ON_DEFERRED_LIST) { + struct deferred_split *ds_queue = get_deferred_split_queue(dst); + + spin_lock(&ds_queue->split_queue_lock); + ds_queue->split_queue_len++; + list_add(&dst->_deferred_list, &ds_queue->split_queue); + spin_unlock(&ds_queue->split_queue_lock); + } + out_unlock_both: folio_unlock(dst); set_page_owner_migrate_reason(&dst->page, reason);