From patchwork Tue Jan 10 07:53:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094776 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83444C46467 for ; Tue, 10 Jan 2023 07:53:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D0CE8E0003; Tue, 10 Jan 2023 02:53:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 00C078E0001; Tue, 10 Jan 2023 02:53:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8F3D8E0003; Tue, 10 Jan 2023 02:53:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B596E8E0001 for ; Tue, 10 Jan 2023 02:53:41 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 7D7071206E3 for ; Tue, 10 Jan 2023 07:53:41 +0000 (UTC) X-FDA: 80338125042.30.95DD9B2 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf17.hostedemail.com (Postfix) with ESMTP id A6AC140004 for ; Tue, 10 Jan 2023 07:53:39 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=a6XzUIgn; spf=pass (imf17.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337219; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GkLj8bodM1qeeMksi5L9FrB60k43damwvJgqq8uLrLg=; b=AJwgziwkxLsAxw4AmR4suQW7WhhBXKYpLEZyZgixFAPb/69KeOG4tDDGkfb21Bu3v5mmZA qC9uOMbxY/JidHlp8JxRX0ftI6/VooCocwzWxnIRWg88Ot+MBJWSbfSPNMDoCWGkukFVLF mH6k3QA48j99TW5+0Ya/ymOawKr4wD0= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=a6XzUIgn; spf=pass (imf17.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337219; a=rsa-sha256; cv=none; b=KHepESETDU0l9QyaR7h3230xzjkPwq5NMS761d5CBA6n9g0Tzd7TqvuK5zV7vWV/Xt9N3T CrRHETOdQQM4nlr/2Qg5Z+lvuybzbdailcg4JFsqADDAmNv0jSZdr5X0Ds5iHfoH6qaW3S WGgiUAX9I6BytEX3g/kdl1KppqwQ46E= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337219; x=1704873219; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cjJltQK7JFH99xWbvsedhf2YXMfqZWRjPAsent7jXKU=; b=a6XzUIgnhvgpUF3svjVeLl9Jgq7wyDmqgz2L/LW5iOusp2E0jhMjV40+ DfXISrmUccdZ0kDaUXBCTLZ8/enkslmes+ZWeQXLRuZsllCKl5QE47K3L jJQk0PDLbH9TF95iUuFDMwsgPxS4hiKa3C4hlv8VXL4tfF1d5vaA30Oki 7GuzWA4NWl+0r2CJw7CAHX7BBK4fRXBJSCOipx5VB7hONvv3e5Trt8QFE CeNYZVLnxFTFgHijEWL4cYSR2S5KdUd9ZaoWiIblWdpcfYTK6ES18Yl/3 AWgr48rZ7XTXmIWJIkoUn31x1tZRjyiwDZ/5gD3DK2GzSvFbcuJP3YNVh Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449267" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449267" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:38 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287143" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287143" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:35 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Alistair Popple , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , haoxin Subject: [PATCH -v2 1/9] migrate_pages: organize stats with struct migrate_pages_stats Date: Tue, 10 Jan 2023 15:53:19 +0800 Message-Id: <20230110075327.590514-2-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A6AC140004 X-Stat-Signature: dc7349ozp34ykfzyx4dcoysoapb6f38t X-Rspam-User: X-HE-Tag: 1673337219-327209 X-HE-Meta: U2FsdGVkX1/AAFtVwAANWiQgTf4arsiVjvcf3LSYEE7/rjNhvgZ4+mJb+/ugPolsYordWvvnrnIqpjkKJaaa6z0CsjO6KuC6ozJ68jnI0rkpCiP4UjMaRlSUo9m87T9R36AJI3Y0D3C6ZMvsLCBnNMgJ8abSFyT6JoNHylnfIHo+aJI1XfZ0xScQOq6ryPqEsdembZND8nOn2fSAPJS1VNk+4mt5Q4/0tPtYzJbv/1K0FXI04XE5osiHjqNERhKq5vY0CePHOrYUq39F528/7G/BEQauxumEK1fWY5xKaJUWyaTr1V3GXyh4T03DYzvnGPA09iHo+X+SgPyFAvQPjW9sEx4Qbfn+tPo9FqigmQF0fKbc1xYhZ4YVas6wVFIXZjGAAw2uAXH9bQFC9UfqbHI63XrG1wXgj5DfyajuGRtcllipqUVzd84hyarbur6q/qVxuXvhvpbr5U4HkC1UUtVuYOGZ8ZP3PcbM3Q27hQ/IvM6RR4XnKTL4AuFOXosMuIZ2YVruoF2Dd1oQnjGw22tsy/ODsT6ogyXVNx0rerFLzyDLKXhiYWNAmtMXVNpdmVFnIajLOP/bCCECUDQC9k+fXT7mdMGsTKfFS4S/FI0E9FyFRKhqwrRF2fRy5srEUKQEshUwP1PUFnxgNnjpPO4/NVGw1U3m2UBMGhPBUPWRwNPPmV6lmqGRM+GNmxYoKAtOOFrYVxgANadcJApHSqIvPih4wddI4YVE6dhDj6dkRImcPO25nVUuJTGMKkfXPFUtT8vm6gbiQ2d2YpTDK0cB7KMRAaqkcfIPDUZ0c82hYXUoJNqTxQt6u+8sT64JLtSykjWT8FMKeiD2S2VSsAiMBoPBt+hpol4h1f9ZXXEa29hij15ruYegGFzlYyEoTXkwTDrDW6zJhtqibQeBGa8/WpPVZm4Cq3nvnUzqb0wV0AB4Vpv4kTEjUCWdua82 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define struct migrate_pages_stats to organize the various statistics in migrate_pages(). This makes it easier to collect and consume the statistics in multiple functions. This will be needed in the following patches in the series. Signed-off-by: "Huang, Ying" Reviewed-by: Alistair Popple Reviewed-by: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: haoxin Reviewed-by: Baolin Wang --- mm/migrate.c | 60 +++++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index a4d3fc65085f..d21de40861a0 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1396,6 +1396,16 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +struct migrate_pages_stats { + int nr_succeeded; /* Normal pages and THP migrated successfully, in units + of base pages */ + int nr_failed_pages; /* Normal pages and THP failed to be migrated, in units + of base pages. Untried pages aren't counted */ + int nr_thp_succeeded; /* THP migrated successfully */ + int nr_thp_failed; /* THP failed to be migrated */ + int nr_thp_split; /* THP split before migrating */ +}; + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1430,13 +1440,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int large_retry = 1; int thp_retry = 1; int nr_failed = 0; - int nr_failed_pages = 0; int nr_retry_pages = 0; - int nr_succeeded = 0; - int nr_thp_succeeded = 0; int nr_large_failed = 0; - int nr_thp_failed = 0; - int nr_thp_split = 0; int pass = 0; bool is_large = false; bool is_thp = false; @@ -1446,9 +1451,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + struct migrate_pages_stats stats; trace_mm_migrate_pages_start(mode, reason); + memset(&stats, 0, sizeof(stats)); split_folio_migration: for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; @@ -1502,9 +1509,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } /* Hugetlb migration is unsupported */ @@ -1512,7 +1519,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; list_move_tail(&folio->lru, &ret_folios); break; case -ENOMEM: @@ -1522,13 +1529,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1546,7 +1553,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages + nr_retry_pages; + stats.nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large * folios left in split_folios list. Move them back to migration @@ -1556,7 +1563,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, list_splice_init(&split_folios, from); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - nr_thp_failed += thp_retry; + stats.nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1568,8 +1575,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - nr_succeeded += nr_pages; - nr_thp_succeeded += is_thp; + stats.nr_succeeded += nr_pages; + stats.nr_thp_succeeded += is_thp; break; default: /* @@ -1580,20 +1587,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - nr_thp_failed += thp_retry; - nr_failed_pages += nr_retry_pages; + stats.nr_thp_failed += thp_retry; + stats.nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1626,16 +1633,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (list_empty(from)) rc = 0; - count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded); - count_vm_events(PGMIGRATE_FAIL, nr_failed_pages); - count_vm_events(THP_MIGRATION_SUCCESS, nr_thp_succeeded); - count_vm_events(THP_MIGRATION_FAIL, nr_thp_failed); - count_vm_events(THP_MIGRATION_SPLIT, nr_thp_split); - trace_mm_migrate_pages(nr_succeeded, nr_failed_pages, nr_thp_succeeded, - nr_thp_failed, nr_thp_split, mode, reason); + count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); + count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); + count_vm_events(THP_MIGRATION_SUCCESS, stats.nr_thp_succeeded); + count_vm_events(THP_MIGRATION_FAIL, stats.nr_thp_failed); + count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split); + trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages, + stats.nr_thp_succeeded, stats.nr_thp_failed, + stats.nr_thp_split, mode, reason); if (ret_succeeded) - *ret_succeeded = nr_succeeded; + *ret_succeeded = stats.nr_succeeded; return rc; } From patchwork Tue Jan 10 07:53:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21600C46467 for ; Tue, 10 Jan 2023 07:53:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A6D0A8E0005; Tue, 10 Jan 2023 02:53:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9F6348E0001; Tue, 10 Jan 2023 02:53:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 824B78E0005; Tue, 10 Jan 2023 02:53:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 717C18E0001 for ; Tue, 10 Jan 2023 02:53:45 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 49C12AED9F for ; Tue, 10 Jan 2023 07:53:45 +0000 (UTC) X-FDA: 80338125210.03.3B75C59 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf20.hostedemail.com (Postfix) with ESMTP id 67F101C0014 for ; Tue, 10 Jan 2023 07:53:43 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=L+16QUHT; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf20.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337223; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=35494nDnkMXz4koO2JhR0eCSJtOg4Z8qRFBMG4pROWc=; b=sxCuvOZtOSHky+t8RJb2hrY3NrvsyZk/c/jdE5X15260PibBrdqdl/MAGC77ybhwmgKcMu AieK0vCV/mB09xl1umkJWFmLXGNqdN2zPJGgWTCAjlapR4Cl6twr/hjTXaWbimD+fL128L GGGkbIAi/vty7q9T0vqHdCKqUlUoH6M= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=L+16QUHT; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf20.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337223; a=rsa-sha256; cv=none; b=GTTTtcmLDMtws6rbsBTgMVv8mqMqvxxflVbq2A7UzYPU6qOEzkxLmsP+lRTjSkszURlkXU fBIZb27tcr6lytiEC4m6Jrwt/+EklN0cV6YmhNn/HPaPfh+uCePKtzQBoWi+Blj440Zcc+ bXfKrhKmopEDsqVzJTuIJOJ76PAUVRI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337223; x=1704873223; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rrmNLGIxnXBvmPk6ddHap7Pzun7U4AV1fL0pxDIYoqM=; b=L+16QUHTIa6TPEXmy8nvPwJF79d5gt2Zs9SrFclH+fTj0f9E7FLxO7Z1 HbujGFZxFhm4ZNZkMNsFvvEgh3hVre+31XqwdOvxnrkt6a94tpd6AVNbz hS9wGKJ3wNUSm1p3jFbz50t9Tjj80tMeadg+TyWXwIpLc0REhkMAska3M kRnIzr2DyHRhMVpa1Rr3dMcwogFNhlsAfITJrC2uxl24AVCDp5/JpCk+E FkesFCQ36Nnc4fGoxI62RGxFpjr2o+OfGuN0j0RKkhyPpO4MoGDE8P8dI gs7pCStfmUSIi0YlwqUBQG0YnVi2IT04Zil5AZJ1nmuCjEiLuiF1VxDfB g==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449282" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449282" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:41 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287147" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287147" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:38 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 2/9] migrate_pages: separate hugetlb folios migration Date: Tue, 10 Jan 2023 15:53:20 +0800 Message-Id: <20230110075327.590514-3-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 67F101C0014 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: om813fiekzjzzhhyyiwbcatf35e3qg4z X-HE-Tag: 1673337223-511398 X-HE-Meta: U2FsdGVkX1+XCnPA0SzOMH8rsvyTmH9eYAUjAVhZUVAV9FnXTpCwQu/E11OVF+NEUqJk00AayEYke/G3mCPIIFR+iyygljbqZAKk1bZgVhff2SH7+RkTyKMf+UyeiiGyTaxidI0UUzZh0pcFacqOBQEizUSk16MQJ7YKGYx/ZVRTwGl29fmJ5PQYAXfcQKI53zrnF1MXnYfFYFiu4KmYpK+C8a5ZE25kATlQDIK8Uq4Aj6bme7JXLfMn9vjtYTwY5421vA3FZRZbqQZY5awpuHhqCLSVGo+2uUBY/UdtxtwIfDYQluQGV/dauYLJoXMQ8w9bNyDuoK8fGd1AWgiQCMEw1SQHApKiqOMSXtLfAUOMCGlGrWEDCyjNPGW2z4SscWnUm0Ltgx3nFqIxeTnwMU2DO7O5DSlNY5N4+I8mEpSrXKRJ+UFcSza9f8ApEA7BHC9v1WUQN3IrP/9KjjzfFH7ADDJaPyeF2u+ZEObudb/0f27QfgY/fShmb4hUpP77kpeRGaNi0hoetHnEYsvL5NoZUk/oYwMldySGNRWbMMdWv5rzK4P30elYl2zdY5Y/yfGcCWGcCyraCOJMrrpwhTvSCO72ncPw91xsktU6jXzTd7bWHjVwrOp2o+AEwZdA5GaqAc+YDdjjA4BDI6q+Ro0QOnHQckZVVtKAqpBLufNpPmsrqsIhFED8i0JCyd9BroXFlYTHNaoYxYGw3AkHXephEXcViPU1Stnyuw3GE2NUVvEENN0+LeaBcuQI4nsklq8tQNbM78gHQ/akftEvtesPgzcHjTHVBVrb85NzgpPRRm6Nnh3CgFsMGCypWLMisDgQQX/0Yt5ygL74erEI2OtwaIexed830AOivSoQma7yg+Mlmqmh4bvRkAPe1LGuSrQ5ugALIPxiPhXbsYYdIoq77Ct/TZsm+r8gbr57m6h8gDD2K9lt96aa8bAidldZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving for the non-hugetlb folios. Based on that we can batch the TLB shootdown during the folio migration and make it possible to use some hardware accelerator for the folio copying. In this patch the hugetlb folios and non-hugetlb folios migration is separated in migrate_pages() to make it easy to change the non-hugetlb folios migration implementation. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Reviewed-by: Baolin Wang --- mm/migrate.c | 141 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 119 insertions(+), 22 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index d21de40861a0..04e6802c236c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1396,6 +1396,8 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +#define NR_MAX_MIGRATE_PAGES_RETRY 10 + struct migrate_pages_stats { int nr_succeeded; /* Normal pages and THP migrated successfully, in units of base pages */ @@ -1406,6 +1408,95 @@ struct migrate_pages_stats { int nr_thp_split; /* THP split before migrating */ }; +/* + * Returns the number of hugetlb folios that were not migrated, or an error code + * after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no hugetlb folios are movable + * any more because the list has become empty or no retryable hugetlb folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * to return hugetlb folios to the LRU or free list only if ret != 0. + */ +static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, + struct migrate_pages_stats *stats, + struct list_head *ret_folios) +{ + int retry = 1; + int nr_failed = 0; + int nr_retry_pages = 0; + int pass = 0; + struct folio *folio, *folio2; + int rc, nr_pages; + + for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && retry; pass++) { + retry = 0; + nr_retry_pages = 0; + + list_for_each_entry_safe(folio, folio2, from, lru) { + if (!folio_test_hugetlb(folio)) + continue; + + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = unmap_and_move_huge_page(get_new_page, + put_new_page, private, + &folio->page, pass > 2, mode, + reason, ret_folios); + /* + * The rules are: + * Success: hugetlb folio will be put back + * -EAGAIN: stay on the from list + * -ENOMEM: stay on the from list + * -ENOSYS: stay on the from list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -ENOSYS: + /* Hugetlb migration is unsupported */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + break; + case -ENOMEM: + /* + * When memory is low, don't bother to try to migrate + * other folios, just exit. + */ + stats->nr_failed_pages += nr_pages + nr_retry_pages; + return -ENOMEM; + case -EAGAIN: + retry++; + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + break; + default: + /* + * Permanent failure (-EBUSY, etc.): + * unlike -EAGAIN case, the failed folio is + * removed from migration folio list and not + * retried in the next outer loop. + */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + break; + } + } + } + /* + * nr_failed is number of hugetlb folios failed to be migrated. After + * NR_MAX_MIGRATE_PAGES_RETRY attempts, give up and count retried hugetlb + * folios as failed. + */ + nr_failed += retry; + stats->nr_failed_pages += nr_retry_pages; + + return nr_failed; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1422,10 +1513,10 @@ struct migrate_pages_stats { * @ret_succeeded: Set to the number of folios migrated successfully if * the caller passes a non-NULL pointer. * - * The function returns after 10 attempts or if no folios are movable any more - * because the list has become empty or no retryable folios exist any more. - * It is caller's responsibility to call putback_movable_pages() to return folios - * to the LRU or free list only if ret != 0. + * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios + * are movable any more because the list has become empty or no retryable folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * to return folios to the LRU or free list only if ret != 0. * * Returns the number of {normal folio, large folio, hugetlb} that were not * migrated, or an error code. The number of large folio splits will be @@ -1439,7 +1530,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed = 0; + int nr_failed; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1456,38 +1547,45 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, trace_mm_migrate_pages_start(mode, reason); memset(&stats, 0, sizeof(stats)); + rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, + &stats, &ret_folios); + if (rc < 0) + goto out; + nr_failed = rc; + split_folio_migration: - for (pass = 0; pass < 10 && (retry || large_retry); pass++) { + for (pass = 0; + pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); + pass++) { retry = 0; large_retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + /* Retried hugetlb folios will be kept in list */ + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + /* * Large folio statistics is based on the source large * folio. Capture required information that might get * lost during migration. */ - is_large = folio_test_large(folio) && !folio_test_hugetlb(folio); + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); + cond_resched(); - if (folio_test_hugetlb(folio)) - rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, - &folio->page, pass > 2, mode, - reason, - &ret_folios); - else - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, &ret_folios); + rc = unmap_and_move(get_new_page, put_new_page, + private, folio, pass > 2, mode, + reason, &ret_folios); /* * The rules are: - * Success: non hugetlb folio will be freed, hugetlb - * folio will be put back + * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list @@ -1514,7 +1612,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, stats.nr_thp_split += is_thp; break; } - /* Hugetlb migration is unsupported */ } else if (!no_split_folio_counting) { nr_failed++; } @@ -1608,8 +1705,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (!list_empty(&split_folios)) { /* - * Move non-migrated folios (after 10 retries) to ret_folios - * to avoid migrating them again. + * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY + * retries) to ret_folios to avoid migrating them again. */ list_splice_init(from, &ret_folios); list_splice_init(&split_folios, from); From patchwork Tue Jan 10 07:53:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56382C54EBE for ; Tue, 10 Jan 2023 07:53:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C6B748E0006; Tue, 10 Jan 2023 02:53:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BCAE18E0001; Tue, 10 Jan 2023 02:53:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D0598E0006; Tue, 10 Jan 2023 02:53:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8C4088E0001 for ; Tue, 10 Jan 2023 02:53:47 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 664031A07EF for ; Tue, 10 Jan 2023 07:53:47 +0000 (UTC) X-FDA: 80338125294.21.53E86FE Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf20.hostedemail.com (Postfix) with ESMTP id 79EA91C0011 for ; Tue, 10 Jan 2023 07:53:45 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Zm7CUrpk; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf20.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337225; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=jdAL0C+qbB0ZnrrpYIl6W4eKPWtrpsrzAsk/jEkw9Qs=; b=5/ppuW2P303IuXZxXzuEKyy8APvOfrJJeHbFu0pYhbt6K3aw2TBZUxqTPBkfChR5IcScp3 x5z1BADkKHungxUaWXz2VcZ9O3IlymcvMf2X1MiHViqODn0GZZPsVsGiV0kDRql2pBXKb2 DHd42y0TMmi1pa2neNZdyfdSCM6+SIk= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Zm7CUrpk; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf20.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337225; a=rsa-sha256; cv=none; b=ipYJsZoj6U/9JObF2Xh9kCxOV3r8oE30NmtmiXdi7TLx9yOFuMrvVA2sRPTbjMr6DJ3Z8p 0WGcqwi8ozbg3Sw8CnlY2AXhAFKh4ZZACYDwXzd1aVRhC8l+hdkFBAE+BKanh5zsxMkpHl MH+iZdtQWl27jcM30W89CtZmZDaLky0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337225; x=1704873225; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=leS5TV6HqI6Qv0q8QFcoZ1Fc2DDHqNooq7QiIFOtAUg=; b=Zm7CUrpkvYNJ9qSBmOCFOiyQaNQHBiDLB2nT6jVZwU34HtvY/lQvH7ra em/Ilyg7IZwFvMASFEmg0+5xPgEEsbtvbln99uH4YMqwAwlnVzgM9nPTZ 5tsIgb8BCVsf6Fgo8eRMRL8sx+eJpcRaN6mks/J5dOsUlUsKd2m62IEQR 8thCOmNLs5DzbBr1KYHofrhOhlMz2dN6SanO6zCX40YpCqi/9A4FJs1la TLJ6z1aRGZEY7zU/oQwgeWT4e2u4MLuh769QrU+xTqQa931tYdWP3h42Z CuodcASG5R5kObX1CT9jCN3m6cwWsZ1RVMfgao7wR8bAPamv6CGM6Sdvp g==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449298" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449298" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:44 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287151" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287151" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:41 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 3/9] migrate_pages: restrict number of pages to migrate in batch Date: Tue, 10 Jan 2023 15:53:21 +0800 Message-Id: <20230110075327.590514-4-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 79EA91C0011 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: cehw161wx6c5hqmsnuidarhdnji45q8r X-HE-Tag: 1673337225-693067 X-HE-Meta: U2FsdGVkX1+7vSFUsqUU9rsQO1qdab6HhNSen2JyxGIn5EO0xx7byPc6IL1KwCtY3hmAIBO+ORdOlQ0lIKiXrIPwOhom1jnHu6tcFMKZ1UJSGdk5IIymhTB2V0KjsdDBMPtOQnO1+XBx7SX0Zt1yndcYy/OqEhKMuzJz0w4g9W1+Yv4yYrzYpTGc0ZLiUFgfxYcLo+vUpnbEwT51iJAHNlYC5t0vBgBIr/C6PDAZdqlKgyvB+PA0+WqrZSxsrWc9SIA7RbkPWqpcUsHstdDsXPBVRZgGeVyVUY+dI4j2uJoNzRKhUs03RLGBxrkAqp8uiAw224lxE6O4CGTMIWxxQa1umz7BEcZHxL0mWeFYNLDx57MyPri+iBl85gUHrUb++5dwiIvb8we6vsPtk+fxUr/xz2Fyf3S/x4YZGQu3gLG0NipVz6lXnBh0ozpHNwnMFGd1guQAkm1Gku4BU1uhAb8SVJRCVS5v+3o0rRRKAFFdgOsnySUMvaoOi4/vcpEl6Axnwq60xGvTgiNgd0sMbSAOxxv0xX+gXhBkH+lrFZNvulX0pjNnQ6obeZmxKH7Q1Kx13RQUv4JJTl6ovEujld1ezlSO6UR+bR+7jjfdqjhJmM7yjw37MvbH2PMvmN4oNAYNrzPzbWUtSDJWFP6fJlSvd2gUcM6kIRdqMh+fvDRoNtadjnsD3jf3fXlPkprsotV+zmTjyfnEn5TEj9qzvHp0BbNaw61aiaHjq4k1HHSeSsyUGRW6KRZaKhjFJ/IJxFeV/slFvumfW9/+V6rgG++I0v+El5rL8TY5g0IB3K2tUlYrsIO33y4OibLWCwajaKYYGXkzXqjy4aySRo7Hp23T3aiXEypmWt+bxUUZkwGNDbhonl6ntDLl/4iKJtBsGv+kt3HZU9M7MaSSKxjQL4cZKVV/fdWlV5Cl7c363yuZCH2VCRcVwRH6xDzbhf+qczGN0dGBIPs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving for non-hugetlb folios. If we had batched the folio unmapping, all folios to be migrated would be unmapped before copying the contents and flags of the folios. If the folios that were passed to migrate_pages() were too many in unit of pages, the execution of the processes would be stopped for too long time, thus too long latency. For example, migrate_pages() syscall will call migrate_pages() with all folios of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Reviewed-by: Baolin Wang --- mm/migrate.c | 174 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 106 insertions(+), 68 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 04e6802c236c..4931dc4c1215 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1396,6 +1396,11 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#else +#define NR_MAX_BATCHED_MIGRATION 512 +#endif #define NR_MAX_MIGRATE_PAGES_RETRY 10 struct migrate_pages_stats { @@ -1497,40 +1502,15 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, return nr_failed; } -/* - * migrate_pages - migrate the folios specified in a list, to the free folios - * supplied as the target for the page migration - * - * @from: The list of folios to be migrated. - * @get_new_page: The function used to allocate free folios to be used - * as the target of the folio migration. - * @put_new_page: The function used to free target folios if migration - * fails, or NULL if no special handling is necessary. - * @private: Private data to be passed on to get_new_page() - * @mode: The migration mode that specifies the constraints for - * folio migration, if any. - * @reason: The reason for folio migration. - * @ret_succeeded: Set to the number of folios migrated successfully if - * the caller passes a non-NULL pointer. - * - * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios - * are movable any more because the list has become empty or no retryable folios - * exist any more. It is caller's responsibility to call putback_movable_pages() - * to return folios to the LRU or free list only if ret != 0. - * - * Returns the number of {normal folio, large folio, hugetlb} that were not - * migrated, or an error code. The number of large folio splits will be - * considered as the number of non-migrated large folio, no matter how many - * split folios of the large folio are migrated successfully. - */ -int migrate_pages(struct list_head *from, new_page_t get_new_page, +static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, - enum migrate_mode mode, int reason, unsigned int *ret_succeeded) + enum migrate_mode mode, int reason, struct list_head *ret_folios, + struct migrate_pages_stats *stats) { int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed; + int nr_failed = 0; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1538,20 +1518,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, bool is_thp = false; struct folio *folio, *folio2; int rc, nr_pages; - LIST_HEAD(ret_folios); LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; - struct migrate_pages_stats stats; - - trace_mm_migrate_pages_start(mode, reason); - - memset(&stats, 0, sizeof(stats)); - rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, - &stats, &ret_folios); - if (rc < 0) - goto out; - nr_failed = rc; split_folio_migration: for (pass = 0; @@ -1563,12 +1532,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { - /* Retried hugetlb folios will be kept in list */ - if (folio_test_hugetlb(folio)) { - list_move_tail(&folio->lru, &ret_folios); - continue; - } - /* * Large folio statistics is based on the source large * folio. Capture required information that might get @@ -1582,15 +1545,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, rc = unmap_and_move(get_new_page, put_new_page, private, folio, pass > 2, mode, - reason, &ret_folios); + reason, ret_folios); /* * The rules are: * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list - * Other errno: put on ret_folios list then splice to - * from list + * Other errno: put on ret_folios list */ switch(rc) { /* @@ -1607,17 +1569,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, &ret_folios); + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); break; case -ENOMEM: /* @@ -1626,13 +1588,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1650,17 +1612,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - stats.nr_failed_pages += nr_pages + nr_retry_pages; + stats->nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large - * folios left in split_folios list. Move them back to migration + * folios left in split_folios list. Move them to ret_folios * list so that they could be put back to the right list by * the caller otherwise the folio refcnt will be leaked. */ - list_splice_init(&split_folios, from); + list_splice_init(&split_folios, ret_folios); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; + stats->nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1672,8 +1634,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - stats.nr_succeeded += nr_pages; - stats.nr_thp_succeeded += is_thp; + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; break; default: /* @@ -1684,20 +1646,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; + stats->nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; - stats.nr_failed_pages += nr_retry_pages; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1708,7 +1670,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY * retries) to ret_folios to avoid migrating them again. */ - list_splice_init(from, &ret_folios); + list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; retry = 1; @@ -1716,6 +1678,82 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } rc = nr_failed + nr_large_failed; +out: + return rc; +} + +/* + * migrate_pages - migrate the folios specified in a list, to the free folios + * supplied as the target for the page migration + * + * @from: The list of folios to be migrated. + * @get_new_page: The function used to allocate free folios to be used + * as the target of the folio migration. + * @put_new_page: The function used to free target folios if migration + * fails, or NULL if no special handling is necessary. + * @private: Private data to be passed on to get_new_page() + * @mode: The migration mode that specifies the constraints for + * folio migration, if any. + * @reason: The reason for folio migration. + * @ret_succeeded: Set to the number of folios migrated successfully if + * the caller passes a non-NULL pointer. + * + * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios + * are movable any more because the list has become empty or no retryable folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * to return folios to the LRU or free list only if ret != 0. + * + * Returns the number of {normal folio, large folio, hugetlb} that were not + * migrated, or an error code. The number of large folio splits will be + * considered as the number of non-migrated large folio, no matter how many + * split folios of the large folio are migrated successfully. + */ +int migrate_pages(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, unsigned int *ret_succeeded) +{ + int rc, rc_gather; + int nr_pages; + struct folio *folio, *folio2; + LIST_HEAD(folios); + LIST_HEAD(ret_folios); + struct migrate_pages_stats stats; + + trace_mm_migrate_pages_start(mode, reason); + + memset(&stats, 0, sizeof(stats)); + + rc_gather = migrate_hugetlbs(from, get_new_page, put_new_page, private, + mode, reason, &stats, &ret_folios); + if (rc_gather < 0) + goto out; +again: + nr_pages = 0; + list_for_each_entry_safe(folio, folio2, from, lru) { + /* Retried hugetlb folios will be kept in list */ + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + + nr_pages += folio_nr_pages(folio); + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + break; + } + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + list_cut_before(&folios, from, &folio->lru); + else + list_splice_init(from, &folios); + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &stats); + list_splice_tail_init(&folios, &ret_folios); + if (rc < 0) { + rc_gather = rc; + goto out; + } + rc_gather += rc; + if (!list_empty(from)) + goto again; out: /* * Put the permanent failure folio back to migration list, they @@ -1728,7 +1766,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * are migrated successfully. */ if (list_empty(from)) - rc = 0; + rc_gather = 0; count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); @@ -1742,7 +1780,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (ret_succeeded) *ret_succeeded = stats.nr_succeeded; - return rc; + return rc_gather; } struct page *alloc_migration_target(struct page *page, unsigned long private) From patchwork Tue Jan 10 07:53:22 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB50DC54EBE for ; Tue, 10 Jan 2023 07:53:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69CF08E0007; Tue, 10 Jan 2023 02:53:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 625AA8E0001; Tue, 10 Jan 2023 02:53:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4A8958E0007; Tue, 10 Jan 2023 02:53:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 32CAA8E0001 for ; Tue, 10 Jan 2023 02:53:51 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 03A0FAEDBB for ; Tue, 10 Jan 2023 07:53:50 +0000 (UTC) X-FDA: 80338125462.05.3B7772A Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf22.hostedemail.com (Postfix) with ESMTP id 3A234C0011 for ; Tue, 10 Jan 2023 07:53:48 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bqLnmeyK; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337229; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IsvuFzuKd/PJaSSO4IpH38fq4iKF0uiswXlJwwYO6Dk=; b=D+FZm7d/I03IhaIbPNjDuxcH/HPx4A+dWMQBdgZqKFT6e1+55HE11lBZrMJ2F5PGdXEm6m ob99W0iC0uvyEyG25IPU9ck+SpISr4Ur7UBKOH07HVU4dmxYzChPMK3xzOZYuTzxK6Z0Z9 HQu0OZfE70F0UHzcG5WIdJcKEkX0JU0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=bqLnmeyK; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337229; a=rsa-sha256; cv=none; b=H8NL6+fARbMWJp8JeO4I28LoOqoi3bMT0iFdhAuaPg+M2cfVITpX65X6Q7eATJe39aWCKK O1eBnX+x7pmRp7aoaugv6FE+V+0ChLDIu6ClsohWNFxz3/0ODHCbdBvBiCDm/aIG/wAePZ vjXq50d+fDaLXK6XQ6pdbGVhnNnvkt0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337229; x=1704873229; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hES0vkW1UnMqzZSwMsa4s0nsv90UXV8dO5UmZ7jk8Co=; b=bqLnmeyKrIYwAhZwJq1eZlWdLLcMhXCKJ1sTNk4j2gm78AsuJcyerglp mPckm7M0j5aBCUcACNFdDlWD2WxQd+D8mJrrKwdvAh9D3OFnd2nYvVH1O cbbrT5z1HnlPg9Sgjy0QFp7nZccKppZcuIf4Xxx3APm+tu/s/oThYEJF+ YKf3lYZCnE+T4KkD+ERt3KYnYJsnRrUPhobsBB1EHHsVDJWOKqsfVn/MX p3xlDMmlxFVLZ3vmNIMaj8pEqfOLhJYiwt/mHApbPJ9AdwVLQjexZSe2a LjMhkUgDOs2am+swsESbWd2XDFTI27z3WIrDWMQXFCg9KxtKi39UyvAiw A==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449313" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449313" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287154" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287154" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:44 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 4/9] migrate_pages: split unmap_and_move() to _unmap() and _move() Date: Tue, 10 Jan 2023 15:53:22 +0800 Message-Id: <20230110075327.590514-5-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3A234C0011 X-Stat-Signature: t1ypnh3nza67tmcoazsck5p93k894i59 X-HE-Tag: 1673337228-934117 X-HE-Meta: U2FsdGVkX1+M2eMNQwfJ5MJzxevhFYTqS3ARDqn37gfwf54VqhSN4sPQQ/BVhmEgWPVNzFS8DT/IYPVVpjdfVIqQz3NFTfWPGPjJ1rH0rIL/k2bV2X7RkUctK4k20gvWyT39ei/vrV1VdjIcUXh0Bgo0/uoIKnEtx5c77OqgHy/kOD2zu4nb87wkHG3kXFK/LWFsQKkMj5+EcDgPw42KEU3JZyKsBeX3A68G4EUW6Tp7F/wzJr0smi9nbEDVwACjesAAHyFCLQZ/2zyNDgG5EeF7MCEVI3pCy3PExaznn2oI3Az6Rhx643Wd2+u+HrVC7/MzhMxf5uro5oE1JMnj03tJZJiHeonXN9FONxk+ycvZWvuVGi6teh3SK7fOG4bTKh1hylKtymFD8Pm53JCai3QBAvTpCMVIymrEL/oQU39BUiLSvfQifcUMWw1jCdQcBlDnK/TqXOmPTsI2fHEzLmgbAZkPXQLmaLPBnXjmFxklaDc3PAVDX5ssWqncbOmYWEolMRmgsdzSObQW+XiBjbw3xVZY66U3ii8La1Xjhl1ROXkfEALglde5ZJgvAzvlMdj+bfQhdA5sUyZRRwPG21hygTv8o4OUUjFe6bMqogozUe5sgmofB7zMGKB2SFscpVp0keE7Vn820dCy16JUGF7a/6fNtV7oSumZr0WiTZtZz7sOQ2cdamgzgA+or/Q43pyZxTFKsGJOaW7eRZpF1BcWbeuTPgsOANjHaHGFCztY1tWCUKfzJ7X4LVQaqVlcNFy6HEf6B8i/IZuNWn76NPMyLZS90C4JtXTuCGs9urV5h4HX5d4qJfDjZbMxWqQ8TYSixllfcMhfRPQ9E3Di+nBatH9ZLjM14aZf2OBHQk4ZUUmANTAczgRDR2IIUyyOqspjwZ9blFvCQMBerNpoxv43qH6t0FnQJFd8otmqGNSElsEfYlHDSQzmYOhUvHdJ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving. In this patch, unmap_and_move() is split to migrate_folio_unmap() and migrate_folio_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused dst->mapping and dst->private are used. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin Reviewed-by: Baolin Wang --- include/linux/migrate.h | 1 + mm/migrate.c | 169 ++++++++++++++++++++++++++++++---------- 2 files changed, 128 insertions(+), 42 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 3ef77f52a4f0..7376074f2e1e 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -18,6 +18,7 @@ struct migration_target_control; * - zero on page migration success; */ #define MIGRATEPAGE_SUCCESS 0 +#define MIGRATEPAGE_UNMAP 1 /** * struct movable_operations - Driver page migration diff --git a/mm/migrate.c b/mm/migrate.c index 4931dc4c1215..4c35c2a49574 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1009,11 +1009,53 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, return rc; } -static int __unmap_and_move(struct folio *src, struct folio *dst, +/* + * To record some information during migration, we uses some unused + * fields (mapping and private) of struct folio of the newly allocated + * destination folio. This is safe because nobody is using them + * except us. + */ +static void __migrate_folio_record(struct folio *dst, + unsigned long page_was_mapped, + struct anon_vma *anon_vma) +{ + dst->mapping = (void *)anon_vma; + dst->private = (void *)page_was_mapped; +} + +static void __migrate_folio_extract(struct folio *dst, + int *page_was_mappedp, + struct anon_vma **anon_vmap) +{ + *anon_vmap = (void *)dst->mapping; + *page_was_mappedp = (unsigned long)dst->private; + dst->mapping = NULL; + dst->private = NULL; +} + +/* Cleanup src folio upon migration success */ +static void migrate_folio_done(struct folio *src, + enum migrate_reason reason) +{ + /* + * Compaction can migrate also non-LRU pages which are + * not accounted to NR_ISOLATED_*. They can be recognized + * as __PageMovable + */ + if (likely(!__folio_test_movable(src))) + mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + + folio_is_file_lru(src), -folio_nr_pages(src)); + + if (reason != MR_MEMORY_FAILURE) + /* We release the page in page_handle_poison. */ + folio_put(src); +} + +static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force, enum migrate_mode mode) { int rc = -EAGAIN; - bool page_was_mapped = false; + int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); @@ -1089,8 +1131,8 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, goto out_unlock; if (unlikely(!is_lru)) { - rc = move_to_new_folio(dst, src, mode); - goto out_unlock_both; + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; } /* @@ -1115,11 +1157,40 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); try_to_migrate(src, 0); - page_was_mapped = true; + page_was_mapped = 1; } - if (!folio_mapped(src)) - rc = move_to_new_folio(dst, src, mode); + if (!folio_mapped(src)) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; + } + + + if (page_was_mapped) + remove_migration_ptes(src, src, false); + +out_unlock_both: + folio_unlock(dst); +out_unlock: + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); +out: + + return rc; +} + +static int __migrate_folio_move(struct folio *src, struct folio *dst, + enum migrate_mode mode) +{ + int rc; + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + + rc = move_to_new_folio(dst, src, mode); /* * When successful, push dst to LRU immediately: so that if it @@ -1140,14 +1211,11 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, remove_migration_ptes(src, rc == MIGRATEPAGE_SUCCESS ? dst : src, false); -out_unlock_both: folio_unlock(dst); -out_unlock: /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); -out: /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased @@ -1159,19 +1227,15 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, return rc; } -/* - * Obtain the lock on folio, remove all ptes and migrate the folio - * to the newly allocated folio in dst. - */ -static int unmap_and_move(new_page_t get_new_page, - free_page_t put_new_page, - unsigned long private, struct folio *src, - int force, enum migrate_mode mode, - enum migrate_reason reason, - struct list_head *ret) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { struct folio *dst; - int rc = MIGRATEPAGE_SUCCESS; + int rc = MIGRATEPAGE_UNMAP; struct page *newpage = NULL; if (!thp_migration_supported() && folio_test_transhuge(src)) @@ -1182,20 +1246,50 @@ static int unmap_and_move(new_page_t get_new_page, folio_clear_active(src); folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ - goto out; + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; } newpage = get_new_page(&src->page, private); if (!newpage) return -ENOMEM; dst = page_folio(newpage); + *dstp = dst; dst->private = NULL; - rc = __unmap_and_move(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A page that has not been migrated will have kept its + * references and be restored. + */ + /* restore the folio to right list. */ + if (rc != -EAGAIN) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + + rc = __migrate_folio_move(src, dst, mode); if (rc == MIGRATEPAGE_SUCCESS) set_page_owner_migrate_reason(&dst->page, reason); -out: if (rc != -EAGAIN) { /* * A folio that has been migrated has all references @@ -1211,20 +1305,7 @@ static int unmap_and_move(new_page_t get_new_page, * we want to retry. */ if (rc == MIGRATEPAGE_SUCCESS) { - /* - * Compaction can migrate also non-LRU folios which are - * not accounted to NR_ISOLATED_*. They can be recognized - * as __folio_test_movable - */ - if (likely(!__folio_test_movable(src))) - mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + - folio_is_file_lru(src), -folio_nr_pages(src)); - - if (reason != MR_MEMORY_FAILURE) - /* - * We release the folio in page_handle_poison. - */ - folio_put(src); + migrate_folio_done(src, reason); } else { if (rc != -EAGAIN) list_add_tail(&src->lru, ret); @@ -1516,7 +1597,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2; + struct folio *folio, *folio2, *dst = NULL; int rc, nr_pages; LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); @@ -1543,9 +1624,13 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, ret_folios); + rc = migrate_folio_unmap(get_new_page, put_new_page, private, + folio, &dst, pass > 2, mode, + reason, ret_folios); + if (rc == MIGRATEPAGE_UNMAP) + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); /* * The rules are: * Success: folio will be freed From patchwork Tue Jan 10 07:53:23 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F13D6C61DB3 for ; Tue, 10 Jan 2023 07:53:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8989D8E0008; Tue, 10 Jan 2023 02:53:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7FA5B8E0001; Tue, 10 Jan 2023 02:53:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 628298E0008; Tue, 10 Jan 2023 02:53:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 4CDB08E0001 for ; Tue, 10 Jan 2023 02:53:53 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 259451C8D6C for ; Tue, 10 Jan 2023 07:53:53 +0000 (UTC) X-FDA: 80338125546.13.0A9276E Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf22.hostedemail.com (Postfix) with ESMTP id 48A2EC0010 for ; Tue, 10 Jan 2023 07:53:51 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jYQkAWbf; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337231; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=78Jmy2wQmRyFY2FNKBTQ8HkInu8fVVqfAfR/8RsC21w=; b=BAL7guCLZkDHEKXvnUyuys7orIkZA0BRYJw6q7hOkQkB4N17C2FOHTrmQW2RQanYtcQToM QOIQoBw5VSke3pNLgVMQHKjF5QDLp7WA6GqK8h8anGjHWzohEKRag8/C+n4D5bZx+8b64O HcZlQ+HXpc8lGxMvku/tBDUODeY77C0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=jYQkAWbf; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337231; a=rsa-sha256; cv=none; b=3z0B+hVxDpp1Jo0L3xkMPF6+Y+clXoLqDLV5gi57/KrtiSainun3Qd5HByMsLOqrqqvpW/ YzOG1zL1uNx4oHo5uAfFUuwH5l+QGYkC8BVqslzSnEAPxgm/Em979n/NFDsowzbD6QwhRR 3/q6TG0uxXRyhc16MIksVJFBVZNwo1o= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337231; x=1704873231; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8sqENHGY6DNVA+KJ5mUSSEkar1m6VwPzX6i/JVkVePc=; b=jYQkAWbfrUk1aabCeSBS8jHJCy2rZsUc63v5dzFrlgGiXQOmqWsJGun8 pxpAzvC8oPZFixCmQCk1OkiU6CkWbra0z+Ca6Eny9Fg0q6dWVVU1nlYCd Z/0XivCdlS8ZIgZETg3TC1SeeSGUtdZDtG3eS3BBGRi9uWEHjc42pw4N2 BWgxBiuuqJ+nHemLZDV5x5nQt6rGu7jKk3JB7o2mdr3Qg8njS+EuS46AX S4J9QM7GMN7fHLVivyc9pp+eMlnp702efVrbzzVEf3PaDzga4zbOCJDln mCJZCaTIlelz69H8t6GodlESQBdyJMI3oohs9VH+euiqsO5LIJQEzlFaL Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449328" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449328" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:50 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287157" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287157" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:47 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 5/9] migrate_pages: batch _unmap and _move Date: Tue, 10 Jan 2023 15:53:23 +0800 Message-Id: <20230110075327.590514-6-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 48A2EC0010 X-Stat-Signature: n3xz1j65e98r76ubnmuuew5m9j76uo3m X-HE-Tag: 1673337231-832369 X-HE-Meta: U2FsdGVkX19u3UDE5TljgKE8z76e+jZw/zN5c9rnERV/oxrw98fWnckiX6KQ6FPVn5jZwDNHg5rTJgMmbZ/Bqwka1ckvXpQNUdyNwr5YdzLhr18MiM0YC46aWWE84xN91ut4nHGoRK6xxaKrU3f7v77UPucR2AMRFsnTGviiVCs8Cdnu5ALLSF6lwMH0yBB5suZ3wDArs45FVTo6mDXyF6w/xSPx3VnvjhOzIksbMDQVvBt+0VB+ioZVZB/Fwr1R6uF5+f6slxMc+erXBYcPtBMzTDkwgw1GM9ZNGQT3obWCuKLB3DxcXr0JjlekBvkDbt6qcG1qk8tj0DxoKJ61FFudFYgMwhiRtRpuV0K92zOjqs0HTJjIBAKmLEn0x1JtEVXbut0WxuROuHNhGlcRgbPbP43ykT2uV+Sg30iB+YmElNJ46qxBMje21GYBWBDhCL+8/IwXW204rxkT11zDgQTbdw21iDqZmvgk1oVQJh/akg/JxO5YYmAvsi2d5/N/NhrBZok01XhzisJ+ydpsVulcxfBMRmbIaUzF/jFCisWhkjHky9yVjhppuQOAZakMmaH2I0Ne07zyNSAoji3DKTzMt+j4aa0YvmMGuHVCm57/cdNfi+AxO7RWfBkZfkCd5n67B/G21+RXMTyUtqz52SRpcsYeckeCOysxAV1V1MR19Q6Nle6NJO9vN4BKqvBvGwx0pftq+uIdVDtiwCwiFFQl7iOJriBkEzIxti6+UrxA3+cMsYX+bEtp790cIJbN6hYieesa+w3BkqkcXy9kRWVrgrDo92NioDetPkWOwE0/o2gDYtkfIwO/a560o+zVhPbV/Bie6l/mcejblf+kNK/c8lxP1vyUZNQI6dHtNVT4fBApRisy+CMnq4pP+f+0gP2vMqtYLAezCSsEeAIkJt+n3krbXucZlw708jn+fuqIP3BpCrkLJP0w+L0sV16x3EhhiCmB2Js= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In this patch the _unmap and _move stage of the folio migration is batched. That for, previously, it is, for each folio _unmap() _move() Now, it is, for each folio _unmap() for each folio _move() Based on this, we can batch the TLB flushing and use some hardware accelerator to copy folios between batched _unmap and batched _move stages. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 205 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 181 insertions(+), 24 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 4c35c2a49574..2983662cff75 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1033,6 +1033,33 @@ static void __migrate_folio_extract(struct folio *dst, dst->private = NULL; } +/* Restore the source folio to the original state upon failure */ +static void migrate_folio_undo_src(struct folio *src, + int page_was_mapped, + struct anon_vma *anon_vma, + struct list_head *ret) +{ + if (page_was_mapped) + remove_migration_ptes(src, src, false); + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + list_move_tail(&src->lru, ret); +} + +/* Restore the destination folio to the original state upon failure */ +static void migrate_folio_undo_dst(struct folio *dst, + free_page_t put_new_page, + unsigned long private) +{ + folio_unlock(dst); + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); +} + /* Cleanup src folio upon migration success */ static void migrate_folio_done(struct folio *src, enum migrate_reason reason) @@ -1052,7 +1079,7 @@ static void migrate_folio_done(struct folio *src, } static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, enum migrate_mode mode) + int force, bool force_lock, enum migrate_mode mode) { int rc = -EAGAIN; int page_was_mapped = 0; @@ -1079,6 +1106,17 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (current->flags & PF_MEMALLOC) goto out; + /* + * We have locked some folios, to avoid deadlock, we cannot + * lock the folio synchronously. Go out to process (and + * unlock) all the locked folios. Then we can lock the folio + * synchronously. + */ + if (!force_lock) { + rc = -EDEADLOCK; + goto out; + } + folio_lock(src); } @@ -1192,6 +1230,8 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, rc = move_to_new_folio(dst, src, mode); + if (rc != -EAGAIN) + list_del(&dst->lru); /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1207,6 +1247,11 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, lru_add_drain(); } + if (rc == -EAGAIN) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; + } + if (page_was_mapped) remove_migration_ptes(src, rc == MIGRATEPAGE_SUCCESS ? dst : src, false); @@ -1230,7 +1275,7 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, + struct folio **dstp, int force, bool force_lock, enum migrate_mode mode, enum migrate_reason reason, struct list_head *ret) { @@ -1258,7 +1303,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page *dstp = dst; dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); if (rc == MIGRATEPAGE_UNMAP) return rc; @@ -1267,7 +1312,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page * references and be restored. */ /* restore the folio to right list. */ - if (rc != -EAGAIN) + if (rc != -EAGAIN && rc != -EDEADLOCK) list_move_tail(&src->lru, ret); if (put_new_page) @@ -1306,9 +1351,8 @@ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, */ if (rc == MIGRATEPAGE_SUCCESS) { migrate_folio_done(src, reason); - } else { - if (rc != -EAGAIN) - list_add_tail(&src->lru, ret); + } else if (rc != -EAGAIN) { + list_add_tail(&src->lru, ret); if (put_new_page) put_new_page(&dst->page, private); @@ -1588,7 +1632,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason, struct list_head *ret_folios, struct migrate_pages_stats *stats) { - int retry = 1; + int retry; int large_retry = 1; int thp_retry = 1; int nr_failed = 0; @@ -1597,13 +1641,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2, *dst = NULL; - int rc, nr_pages; + struct folio *folio, *folio2, *dst = NULL, *dst2; + int rc, rc_saved, nr_pages; LIST_HEAD(split_folios); + LIST_HEAD(unmap_folios); + LIST_HEAD(dst_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + bool force_lock; -split_folio_migration: +retry: + rc_saved = 0; + force_lock = true; + retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); pass++) { @@ -1625,16 +1675,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, mode, - reason, ret_folios); - if (rc == MIGRATEPAGE_UNMAP) - rc = migrate_folio_move(put_new_page, private, - folio, dst, mode, - reason, ret_folios); + folio, &dst, pass > 2, force_lock, + mode, reason, ret_folios); /* * The rules are: * Success: folio will be freed + * Unmap: folio will be put on unmap_folios list, + * dst folio put on dst_folios list * -EAGAIN: stay on the from list + * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list * Other errno: put on ret_folios list @@ -1669,7 +1718,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, case -ENOMEM: /* * When memory is low, don't bother to try to migrate - * other folios, just exit. + * other folios, move unmapped folios, then exit. */ if (is_large) { nr_large_failed++; @@ -1708,7 +1757,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; - goto out; + rc_saved = rc; + if (list_empty(&unmap_folios)) + goto out; + else + goto move; + case -EDEADLOCK: + /* + * The folio cannot be locked for potential deadlock. + * Go move (and unlock) all locked folios. Then we can + * try again. + */ + rc_saved = rc; + goto move; case -EAGAIN: if (is_large) { large_retry++; @@ -1722,6 +1783,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_succeeded += nr_pages; stats->nr_thp_succeeded += is_thp; break; + case MIGRATEPAGE_UNMAP: + /* + * We have locked some folios, don't force lock + * to avoid deadlock. + */ + force_lock = false; + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + break; default: /* * Permanent failure (-EBUSY, etc.): @@ -1745,12 +1815,95 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; +move: + retry = 1; + for (pass = 0; + pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); + pass++) { + retry = 0; + large_retry = 0; + thp_retry = 0; + nr_retry_pages = 0; + + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + is_large = folio_test_large(folio); + is_thp = is_large && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + if (is_large) { + large_retry++; + thp_retry += is_thp; + } else if (!no_split_folio_counting) { + retry++; + } + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + if (is_large) { + nr_large_failed++; + stats->nr_thp_failed += is_thp; + } else if (!no_split_folio_counting) { + nr_failed++; + } + + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + } + nr_failed += retry; + nr_large_failed += large_retry; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; + + if (rc_saved) + rc = rc_saved; + else + rc = nr_failed + nr_large_failed; +out: + /* Cleanup remaining folios */ + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + migrate_folio_undo_src(folio, page_was_mapped, anon_vma, + ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, put_new_page, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a * large folio is counted as 1 failure in the first round. */ - if (!list_empty(&split_folios)) { + if (rc >= 0 && !list_empty(&split_folios)) { /* * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY * retries) to ret_folios to avoid migrating them again. @@ -1758,12 +1911,16 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; - retry = 1; - goto split_folio_migration; + goto retry; } - rc = nr_failed + nr_large_failed; -out: + /* + * We have unlocked all locked folios, so we can force lock now, let's + * try again. + */ + if (rc == -EDEADLOCK) + goto retry; + return rc; } From patchwork Tue Jan 10 07:53:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E35A6C54EBE for ; Tue, 10 Jan 2023 07:53:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7AE8E8E0009; Tue, 10 Jan 2023 02:53:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7374F8E0001; Tue, 10 Jan 2023 02:53:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B0C68E0009; Tue, 10 Jan 2023 02:53:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 421728E0001 for ; Tue, 10 Jan 2023 02:53:56 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1023540BCC for ; Tue, 10 Jan 2023 07:53:56 +0000 (UTC) X-FDA: 80338125672.23.0FD4E2C Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf22.hostedemail.com (Postfix) with ESMTP id 3248BC000C for ; Tue, 10 Jan 2023 07:53:54 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=l1iVHLI5; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337234; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lLV4TWCZsGHvy8bTmJiNBKRkyzMepfk67LuPaElxASU=; b=RQ0JPWUyXYV13fpAXqSZD/L9qaVBfn2lnSCgSbxfQq0ocyXjanLCUKRltMtedq7rDykk66 cDCY0468joKzk08MVzFG/ptts/tjFI+p7qeU2CXB1sSQSxXHfMZUXbgzdG1G1xXpKhsvRv yvA3WhlR6iryw/JZb8eymrMFKsWhICs= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=l1iVHLI5; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337234; a=rsa-sha256; cv=none; b=oB5XfP+SqX3TzR6UVB8ahX/v1dP39cS0fRXLA1jRLPpRVy8vAffHiwObUs4MeAtWm7Bmk9 vT1WYWhFd5C7oL6JuMSSMe8AWY6A+ircWnTEw9g/qYwJqp0sMpaP0w1YujE91Dp3vzkw9v PsQvpTHunhUnFLufaYj8yCmrZX7pCmg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337234; x=1704873234; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1seMGdAxQCuDtz4CTF8kF1CgdzVLCUmbg8rfqSN0s20=; b=l1iVHLI5CbCht401ANL+Sp2UcdI4O8fmBxNMGShPiWQLS5iV7sHq8ieY GakfrU8LUTrQ5tO+CZ7ViNIJfNqItFS1IGj3P2S5VUl9f7CLVa53bdbHJ /ZVN6TVyedfak6r8Jj5lwaDY1kygWITgHlK6z1YiRTHq/vIxYkyC0mncv 6p15yg0OXr7iiaj1oDvGu2kjm3YsEd3r94sRuUYkTcoDprrnnyUuu5L4b fa5fJOrbfu8x5CwnKFMKrcer3KxVEsC71IOUBTSNE/fZ8Zrjzlq1K+LPa P6EhFCrK1sJ2rBXe+FZj3iFqj+mlQDfHmP9fU7q3oniHXC+aT60Yp9wOj A==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449349" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449349" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:53 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287160" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287160" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:50 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 6/9] migrate_pages: move migrate_folio_unmap() Date: Tue, 10 Jan 2023 15:53:24 +0800 Message-Id: <20230110075327.590514-7-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 3248BC000C X-Stat-Signature: hdah58wrbnxe4ezikqbhdkoqw8r1kzqr X-HE-Tag: 1673337234-496250 X-HE-Meta: U2FsdGVkX19CTzJLEVLXuwni1rfZBx7cn/uYs+15DijdrkBIvjv97HzbG90JOwPQR+qLjpDGxQ4u3GhbNCTOY1ro/LN2gEcnnkQCZFN2nYTPKKoJj0mlSckm9DEwbhTs4G8L11WO6/ey1EgQYj/gWBBnyvwEcOZvy38LWMkRpeeVrODdw9WyuYzhsQC03iY8TdRkbiZ1ihMd2JSzdRXbZHCjYujLdVyyvqRrUytbwAbc0gioCHXYZQ77VQEtgenbanmI5Q5IJR+xxGJwgn+aXwIN53BhUxSsXfBPKdEULRRvwtPevWTrDLHOrmDLHk7o51HblzmTZfFvhwwv5SxfOBL28nXCjHeyvuGavXKzFE8Y9TaTpjkDTpyLypFTGwU7azpZl/SSotAZehO6ZQBSIPpiEjgnNxu7BlTbE9ARz5x1F2Hy5da+plDy3DqD3euOzhUqZ+2+uG0Q/6A+BVZFrkXsiuU5hmHTFM6bUYsdJAypgtaOV2FRRmqXskq1W5V0LZdhYeA9daDzeINnQD4M/VQ4jK/BO/+j7jyt5X76ZnfJqvN6TriwC9BG1ibT+Sr+g5/UDaSI78O308psmZxx8tvoddjppCVG9zYvRv1ffJqgxaZvyemaINDe6vHr73Kdi7aO2TB3PVoi+oevTGlyQWBI1xQyBSN2JcZOzMT11hixu04bLg1wZSQKtU89KZE9707Kw+pDXoV9dULnAvYkEjw93lhf08HXxosAAkgD9MUtMZwtofl9anoCXa4hlSlcXAPx3TLQW746ruQeZzaYzD1EVpuNKUh+qyxNgSDsiN0tzdEMBrxSMDigmf6XJZMj2z6ZseZJrZePnrt454kCN2NI/rppkwVwVmncR1fr6P2fHYSgW09y07M6ic8pG1Yl1Cz5z8aTBJsT7QLk3IlrT3/Po1/VvYBJd6eSBxHofW/W9DaOzsYZcjfKMIQz/tNHLml5GXgXm0o= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just move the position of the functions. There's no any functionality change. This is to make it easier to review the next patch via putting code near its position in the next patch. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 102 +++++++++++++++++++++++++-------------------------- 1 file changed, 51 insertions(+), 51 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 2983662cff75..08ce68ba5085 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1219,6 +1219,57 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, return rc; } +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + struct folio *dst; + int rc = MIGRATEPAGE_UNMAP; + struct page *newpage = NULL; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; + rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A page that has not been migrated will have kept its + * references and be restored. + */ + /* restore the folio to right list. */ + if (rc != -EAGAIN && rc != -EDEADLOCK) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + static int __migrate_folio_move(struct folio *src, struct folio *dst, enum migrate_mode mode) { @@ -1272,57 +1323,6 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, return rc; } -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - - /* - * A page that has not been migrated will have kept its - * references and be restored. - */ - /* restore the folio to right list. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); - - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); - - return rc; -} - /* Migrate the folio to the newly allocated folio in dst. */ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, struct folio *src, struct folio *dst, From patchwork Tue Jan 10 07:53:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66D51C54EBE for ; Tue, 10 Jan 2023 07:54:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E3D628E000A; Tue, 10 Jan 2023 02:53:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DC6938E0001; Tue, 10 Jan 2023 02:53:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C40168E000A; Tue, 10 Jan 2023 02:53:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AE2978E0001 for ; Tue, 10 Jan 2023 02:53:59 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 750E8803A9 for ; Tue, 10 Jan 2023 07:53:59 +0000 (UTC) X-FDA: 80338125798.05.23434B2 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf22.hostedemail.com (Postfix) with ESMTP id 5F141C0004 for ; Tue, 10 Jan 2023 07:53:57 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="IL/CmecD"; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337237; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nvlOY5thobMMfBBwSIL7UZ3rHYZV4yteEfs4Uw0/IUk=; b=5drwduAFbCOJFO+pvtdzYLLU/JV1TpZjRqVKQmNiSH18Zws897AH3tO/oqMboivvyimjm9 Em847t2BscMKtUrhUsssJdqrSkdp5InNT+7lMzfX/DfFMYGCFnUuR+HUB7PebcUVkaEKXP 7nlT4pA+LNT+wtJsC1miloAASFkQOo8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="IL/CmecD"; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337237; a=rsa-sha256; cv=none; b=xoST9sTmaDmS0Dq5ln0kwiClBE+LfFj4yj/1APxCJJh7xw7/hqkNSTsOPc3LUtFEqCJ+Fk Au5klUNWrHz9l8R6hXeKfn4LXWePCS/iTcVnNPkTKTvzMwX2t+ut4bTy7xJWtYLE87DwRP 9RcE0F7OpkF93pbPZLWnvkdpIB0i2Q4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337237; x=1704873237; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Jj7YUOuBj4VP8ir6ICoR9mdplgusD9YiP4zbuT3Oc1o=; b=IL/CmecDNoCcF1oE+B0g7/3xxPVCJHhBUGFPMqzLE/87yAfMYdmbB9ms IY+fcKwAzc+Kh9CZSH8F70wZjltFuJKHRrhlus+8qcuBe60jV30rdCrH+ IN5YbMz6xigfWrgM+pSDyVvdXa5YfRQ8hMdwyjtmdqeeozmt1Wf41TIjM F+ZGflIbYdCpOaAgH7tl6vb2UuTwasMK4LcLQQZ654YvLW+NX+qtrwXZ0 2fGzEzMw65yG74ExdStiqTO+kGPXmxHwEykYbXLj4bpBdUIwuJivXAd26 qhqFlH6xXTU/9YaoxIIh46Gp9vQhh+lT/yzCeFOvXuiFsPakke4TQcjWL Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449364" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449364" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:56 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287176" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287176" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:53 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 7/9] migrate_pages: share more code between _unmap and _move Date: Tue, 10 Jan 2023 15:53:25 +0800 Message-Id: <20230110075327.590514-8-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5F141C0004 X-Stat-Signature: bz6idtnuqk1o6z68pfcfiod5ztfckgta X-HE-Tag: 1673337237-322933 X-HE-Meta: U2FsdGVkX19gmdhGtk+z/EzAgWEQA8H36Zv2YO067CmRG6qGhEVNqiNwHSy3UdBF6SQhECx7+NweEfOz+xFaItdn4TpZuSJyJqzbK6A5ZmSq17t4SqMGCog6WZ8azCOrDSt3GozuXpa3iEcunVjPuPfcW86pIYgkhXL+4sTcLfkDI4jF8G2vs78wdvlBkUpcVEUAC3FyqaulqSD/ydIOUVERFl5iyCDWPKhuXrmEHLMvJ9p8IdAjJldSQ6TQHpu1iG6tqwdEpnTeotlAr3AYpPPcyS5KNY9OxpSVtxrjDSvd8bpkML4pgnOCC3hNJZG/Yj4BBiEq++i3g68QSDYSnlrmOdDMmEulGEuzUT2fGxqHSBvEI3m1BM5/gGb+0Hpzl05g0I3rPBw+Pq2EsA0TfWJ2Y4MzEyPE2ajE63UNhUBHP+7peZOBfMPwUopkcaQiftdmKqJwmBDzIOPiersya4p57nU3wQ6DKvzOEeshHs+60TTo9vg0amdcz7aL1eSofbZPZeNCFPYDRy+ANxYT1e7Bh3qg/iPJYLHuD54rKrS/4UQGSZ4lD570zYp2OFElJR3R+LAYCg8ML3MzBRJtrNpSZ8Lex0CcJb8LLqofNJpszzS09H15jmaI4uh9Z77AqA5GA+/oFwCclvnCL+lkHyDoE9+1AYYjGGKmbsLkcccL4852zeJn+eyKYVKf/pLoWenK4m0wlrsIcifQhyz5kdyh0ad7SN0kY8bQkN8E26iAEc0Ff84pjLfS9Z9BGtQvGiljc4IWbUuJxj8q9txhR4ZoRF47+xIHQrVXoLzGmBnt0UTfYUk/OMtPR3ykzoSl9YxtdB7YrmHldKaED1vf/M/cb0HT8BR9zRbMhmjwGYhdJu8D6qpulOx1/qKdWEOXjY3h7QZFtHSwCbwnatxYMO7DIANx9tX3mOb+J2uIUXG90qdB0BzjYDZO+dDxxTzeEJezMipJWwE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a code cleanup patch to reduce the duplicated code between the _unmap and _move stages of migrate_pages(). No functionality change is expected. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 207 ++++++++++++++++++++------------------------------- 1 file changed, 82 insertions(+), 125 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 08ce68ba5085..6c721b897efd 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1037,6 +1037,7 @@ static void __migrate_folio_extract(struct folio *dst, static void migrate_folio_undo_src(struct folio *src, int page_was_mapped, struct anon_vma *anon_vma, + bool locked, struct list_head *ret) { if (page_was_mapped) @@ -1044,16 +1045,20 @@ static void migrate_folio_undo_src(struct folio *src, /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); - folio_unlock(src); - list_move_tail(&src->lru, ret); + if (locked) + folio_unlock(src); + if (ret) + list_move_tail(&src->lru, ret); } /* Restore the destination folio to the original state upon failure */ static void migrate_folio_undo_dst(struct folio *dst, + bool locked, free_page_t put_new_page, unsigned long private) { - folio_unlock(dst); + if (locked) + folio_unlock(dst); if (put_new_page) put_new_page(&dst->page, private); else @@ -1078,13 +1083,42 @@ static void migrate_folio_done(struct folio *src, folio_put(src); } -static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, bool force_lock, enum migrate_mode mode) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { + struct folio *dst; int rc = -EAGAIN; + struct page *newpage = NULL; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); + bool locked = false; + bool dst_locked = false; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; if (!folio_trylock(src)) { if (!force || mode == MIGRATE_ASYNC) @@ -1119,6 +1153,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, folio_lock(src); } + locked = true; if (folio_test_writeback(src)) { /* @@ -1133,10 +1168,10 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, break; default: rc = -EBUSY; - goto out_unlock; + goto out; } if (!force) - goto out_unlock; + goto out; folio_wait_writeback(src); } @@ -1166,7 +1201,8 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, * This is much like races on refcount of oldpage: just don't BUG(). */ if (unlikely(!folio_trylock(dst))) - goto out_unlock; + goto out; + dst_locked = true; if (unlikely(!is_lru)) { __migrate_folio_record(dst, page_was_mapped, anon_vma); @@ -1188,7 +1224,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (!src->mapping) { if (folio_test_private(src)) { try_to_free_buffers(src); - goto out_unlock_both; + goto out; } } else if (folio_mapped(src)) { /* Establish migration ptes */ @@ -1203,75 +1239,26 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, return MIGRATEPAGE_UNMAP; } - - if (page_was_mapped) - remove_migration_ptes(src, src, false); - -out_unlock_both: - folio_unlock(dst); -out_unlock: - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); out: - - return rc; -} - -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - /* * A page that has not been migrated will have kept its * references and be restored. */ /* restore the folio to right list. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); + if (rc == -EAGAIN || rc == -EDEADLOCK) + ret = NULL; - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); + migrate_folio_undo_dst(dst, dst_locked, put_new_page, private); return rc; } -static int __migrate_folio_move(struct folio *src, struct folio *dst, - enum migrate_mode mode) +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { int rc; int page_was_mapped = 0; @@ -1280,9 +1267,10 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); rc = move_to_new_folio(dst, src, mode); + if (rc) + goto out; - if (rc != -EAGAIN) - list_del(&dst->lru); + list_del(&dst->lru); /* * When successful, push dst to LRU immediately: so that if it * turns out to be an mlocked page, remove_migration_ptes() will @@ -1292,74 +1280,43 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, * unsuccessful, and other cases when a page has been temporarily * isolated from the unevictable LRU: but this case is the easiest. */ - if (rc == MIGRATEPAGE_SUCCESS) { - folio_add_lru(dst); - if (page_was_mapped) - lru_add_drain(); - } - - if (rc == -EAGAIN) { - __migrate_folio_record(dst, page_was_mapped, anon_vma); - return rc; - } - + folio_add_lru(dst); if (page_was_mapped) - remove_migration_ptes(src, - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); + lru_add_drain(); + if (page_was_mapped) + remove_migration_ptes(src, dst, false); folio_unlock(dst); - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); + set_page_owner_migrate_reason(&dst->page, reason); /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased * refcounter. */ - if (rc == MIGRATEPAGE_SUCCESS) - folio_put(dst); - - return rc; -} - -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_page_t put_new_page, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - int rc; - - rc = __migrate_folio_move(src, dst, mode); - if (rc == MIGRATEPAGE_SUCCESS) - set_page_owner_migrate_reason(&dst->page, reason); - - if (rc != -EAGAIN) { - /* - * A folio that has been migrated has all references - * removed and will be freed. A folio that has not been - * migrated will have kept its references and be restored. - */ - list_del(&src->lru); - } + folio_put(dst); /* - * If migration is successful, releases reference grabbed during - * isolation. Otherwise, restore the folio to right list unless - * we want to retry. + * A page that has been migrated has all references removed + * and will be freed. */ - if (rc == MIGRATEPAGE_SUCCESS) { - migrate_folio_done(src, reason); - } else if (rc != -EAGAIN) { - list_add_tail(&src->lru, ret); + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + migrate_folio_done(src, reason); - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + return rc; +out: + if (rc == -EAGAIN) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; } + migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, true, put_new_page, private); + return rc; } @@ -1891,9 +1848,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); migrate_folio_undo_src(folio, page_was_mapped, anon_vma, - ret_folios); + true, ret_folios); list_del(&dst->lru); - migrate_folio_undo_dst(dst, put_new_page, private); + migrate_folio_undo_dst(dst, true, put_new_page, private); dst = dst2; dst2 = list_next_entry(dst, lru); } From patchwork Tue Jan 10 07:53:26 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EBFFC61DB3 for ; Tue, 10 Jan 2023 07:54:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A9608E000B; Tue, 10 Jan 2023 02:54:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9324F8E0001; Tue, 10 Jan 2023 02:54:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7ACA98E000B; Tue, 10 Jan 2023 02:54:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 602058E0001 for ; Tue, 10 Jan 2023 02:54:02 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 14E4AAEDB3 for ; Tue, 10 Jan 2023 07:54:02 +0000 (UTC) X-FDA: 80338125924.18.9D67B47 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf22.hostedemail.com (Postfix) with ESMTP id 195FCC0016 for ; Tue, 10 Jan 2023 07:53:59 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WqPn9YDC; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337240; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3zVlQ/eFw4o90gDmtfcrBAcKY3HCYoo+98Jb/ucefC8=; b=RywOBrPvKV9Zd2VOkO+tnnuT1dtusP9qfp2KLoCdxHdFrudse5jQMCMUmcgsTPZzw1DrhJ ZIV+Wde7z++r9UWWHSuxPI/RvRl9XuCdi9s00eddzgTjX/ntMRQcqGfiSKEqB9V6htv6bx +5lQWMxfh+N3TvIacwdxMBVFe7Nowvc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=WqPn9YDC; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337240; a=rsa-sha256; cv=none; b=ViFVn5p8q1KcSziYRQIPyoOn27ZXRXxq0Gl4b8I06OtWnw2xy7RNUUal/HdfGCUzvVc9LO eF00g4fFbVcUUOVhOA8hIDXzqduoP9Dawjyaq8ZzXIe6P38/aCFV0Z8qc9T4dtpX4OTZwy jU6jpPXL1gHu8ivxrrblT5niYxgZJbk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337240; x=1704873240; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+JXFxO9igzjBIvznA/wHPFCJpglE8nncKGoF1baKSuw=; b=WqPn9YDC+dd6mUMiidwjIunRCgPOAdhVdpnSlUzMBipBKA8pg4elfl8K /r4WbLlYj0HuXeXqKdr+HegC+KIisZA7rqkgDCEs6+d6u/xi64+cXCsiD qm3Ou4BjzDGKK65QlYypwlUnJnvK58IfWSPQEZNRfMyAoeu2/1kaPD2AI mIs9tOC1CDJdMPSPaXdSr3qSHOyEQFelNd7qzGSiET3WUlLpgSjZrWaRV dQqkIncEYxOFeCejpDe8Xf/QVmeDEXjM/Svo4+SATTajPJIOkDomWVM9F fJFRqveRBBIfgQGAEe8vF6G2DA1oCsTp8l8VtKhdeZpN/RFlfk2q4IzHj Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449378" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449378" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:59 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287180" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287180" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:56 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH -v2 8/9] migrate_pages: batch flushing TLB Date: Tue, 10 Jan 2023 15:53:26 +0800 Message-Id: <20230110075327.590514-9-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 195FCC0016 X-Stat-Signature: pme96bbamsoc9onazy1bizeu753ai5cp X-HE-Tag: 1673337239-926191 X-HE-Meta: U2FsdGVkX18AA+FGoTV5NF5Kv6cC/mTK1cQycxd0CXzeOZZKWUOa8FCBF4cebM779yQVTLKibTH1XEJ0cW5O6N0iBW2aE1fiNxNOzYB0ba+TaoEoJcBB966q/LJGC2LCKMzqRAIwFiNYgrgZfBqJADAX23dWg4TkOI74/UslEWMRVwtBbLYwaUUkdFhphGBlOZW955jShLLAYIHVJkWjwGXDIXxsQAT33VWYPtEnU8+EyOmO/s+/DgedI8PSvm+Dq3Ock6clWok7WTKRJjk7zDsx9I8TBCNTOxnPkPJ3X2E2JgNXsvG2urTR4pgvOIAph09sz3mpBPdKtJFsmNI1LFwr7jlwRqox8DOXoTPkhLDnVTX64KFsfr9C02eZQDlBK9hF4CFql3u87AuV4eh7hnpfnt085ylvcsmAl2l43hawE3ywEjIXPuFEOHkzILoW74mPMcJiPZ0l+XDmLtrx9jMPdQNumjhnso3Cr1Fe2qHROj4zNgL5a6OE9cuHwocsQar10/vIW4Qve9D8gK06lEx3oeXgThfWFwDOM0koSfIqDlc65Bzy7zvxMZTJsrTKPhvmDmS3PB5YEwsb4NLic1/TwfA1wUvWVjXhyD1XvLeVeaZ6c9gjdbmLSu2OuHx2BK0DI4E+xul2EShW1vzJKxqxTE3bXAZRzdoKQ6LI5PyjpWL6IHVBl9yBf7kK53aOsZBJ0F609wGupfccP4EBlClx4d2dGfwnkcXETcIwz3eX6+Z9xwaQpwQ4kqZRdaresmtW9cVK//JjGnBCUGKlc+E5xU1BCiGVr5uzaTupU3hvv96RjEuTxlohhG0HkjKOwE+9XVVIab8mL6WhdxbIMfixfUNFLfCXRHDb/FTOLUxff9ngbSNUIfAk3xWgjbLsV3oKF5ierslnIx8aEoHwtWzwfRXk+gq3zY5srtxpp6iwMsFNmCR1xrRQfwH9TW90GUqjhIEOMTs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The TLB flushing will cost quite some CPU cycles during the folio migration in some situations. For example, when migrate a folio of a process with multiple active threads that run on multiple CPUs. After batching the _unmap and _move in migrate_pages(), the TLB flushing can be batched easily with the existing TLB flush batching mechanism. This patch implements that. We use the following test case to test the patch. On a 2-socket Intel server, - Run pmbench memory accessing benchmark - Run `migratepages` to migrate pages of pmbench between node 0 and node 1 back and forth. With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%. NOTE: TLB flushing is batched only for normal folios, not for THP folios. Because the overhead of TLB flushing for THP folios is much lower than that for normal folios (about 1/512 on x86 platform). Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 4 +++- mm/rmap.c | 20 +++++++++++++++++--- 2 files changed, 20 insertions(+), 4 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 6c721b897efd..6adaea05b80a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1230,7 +1230,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, 0); + try_to_migrate(src, TTU_BATCH_FLUSH); page_was_mapped = 1; } @@ -1773,6 +1773,8 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + try_to_unmap_flush(); + retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); diff --git a/mm/rmap.c b/mm/rmap.c index b616870a09be..2e125f3e462e 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1976,7 +1976,21 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } else { flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); /* Nuke the page table entry. */ - pteval = ptep_clear_flush(vma, address, pvmw.pte); + if (should_defer_flush(mm, flags)) { + /* + * We clear the PTE but do not flush so potentially + * a remote CPU could still be writing to the folio. + * If the entry was previously clean then the + * architecture must guarantee that a clear->dirty + * transition on a cached TLB entry is written through + * and traps if the PTE is unmapped. + */ + pteval = ptep_get_and_clear(mm, address, pvmw.pte); + + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + } else { + pteval = ptep_clear_flush(vma, address, pvmw.pte); + } } /* Set the dirty flag on the folio now the pte is gone. */ @@ -2148,10 +2162,10 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) /* * Migration always ignores mlock and only supports TTU_RMAP_LOCKED and - * TTU_SPLIT_HUGE_PMD and TTU_SYNC flags. + * TTU_SPLIT_HUGE_PMD, TTU_SYNC, and TTU_BATCH_FLUSH flags. */ if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC))) + TTU_SYNC | TTU_BATCH_FLUSH))) return; if (folio_is_zone_device(folio) && From patchwork Tue Jan 10 07:53:27 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13094784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E78A7C46467 for ; Tue, 10 Jan 2023 07:54:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 822B08E000C; Tue, 10 Jan 2023 02:54:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7AB698E0001; Tue, 10 Jan 2023 02:54:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5FE808E000C; Tue, 10 Jan 2023 02:54:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 47F6A8E0001 for ; Tue, 10 Jan 2023 02:54:05 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1785FC02BB for ; Tue, 10 Jan 2023 07:54:05 +0000 (UTC) X-FDA: 80338126050.02.C108E75 Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by imf22.hostedemail.com (Postfix) with ESMTP id 2B170C000C for ; Tue, 10 Jan 2023 07:54:02 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Ks2QbPWd; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673337243; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mTIu/ye9xgr8iuDmI1qF4Y4bUkFIxhLO9oLQsAT1aRw=; b=WFjjoM878fLkNaq+gU6nTe0voe3fM7oNR3jPJGEDCLrxySFCp0AlR3AZWJcCDB4rbfq3Sj WQTuyARaCUec0N21FthDVov378jkNxwq1zxtGcmQgwaaflgzVndaaNfOgRCmXNAcawNtFu gAn6qQaraLOmlsEqffz2j+6f0piOPsc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Ks2QbPWd; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.151 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673337243; a=rsa-sha256; cv=none; b=2As2Z5JaPZpiAutmRQ+hyHnrJP7WVsdwvAfmxZhm9L1vKBGu/mJtFkFW96roT6IoHRpslN 7onKmgD2hKnZhSbuLV0dnkcRuT6lj82y2EymrNOG07ntTGrMnQUEG7Dy8Ms6s4eJDhrXTp vXoYbPIreoJrc/7LP6Kd74jjn8oNBJQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1673337243; x=1704873243; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=apP3JU34KkC4d0lE7LZcTXIY/rJds9BktgkJCe0SGdY=; b=Ks2QbPWdXY6BmhyGhcsJM+bdKieWYo/7gBahoVoYWwk/TdObDRsKstnt T4RoynyCRzKfm2BL1jPXmpV01xllkxQdaEJ9ihZxIDATOF+Yq+7MDA9j+ s3ITq0TVY9ra3Q7YTUgUrZrDxgSxPOYHSdYtSZvcLFZLBOY8GCQYM+vx4 8MJw8s85s2/UmtetbnMbuevdgMWxb5wrmXEBR/J7XHGLUXCXwqpNjTtRu aEsGOFoyAxvQ74I3ZmyaSI1jug2uCi8LHXw27Lydqnu1vK6xdVGljOhx/ UVAh1ViKqN1NMc3+CuS8fZOpVcQH+XRalmlZo8BjYv6ezlmZegMogUwo9 A==; X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="303449396" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="303449396" Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga107.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:54:02 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10585"; a="902287197" X-IronPort-AV: E=Sophos;i="5.96,314,1665471600"; d="scan'208";a="902287197" Received: from juxinli-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.214.35]) by fmsmga006-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 09 Jan 2023 23:53:59 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Alistair Popple , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , haoxin Subject: [PATCH -v2 9/9] migrate_pages: move THP/hugetlb migration support check to simplify code Date: Tue, 10 Jan 2023 15:53:27 +0800 Message-Id: <20230110075327.590514-10-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230110075327.590514-1-ying.huang@intel.com> References: <20230110075327.590514-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 2B170C000C X-Stat-Signature: eqzz6ngtj7cde4uoxfjz3ydn8rfw464m X-HE-Tag: 1673337242-830097 X-HE-Meta: U2FsdGVkX1/erGbMYkC8GHpPc8YEjfM9EVh6NysZgdOxPndkLLhQeOqRJQNqm4YU47JIcCoYQOZ7/UHX8yvZ92LuZTF9MY2JMgp5E14i6f6urlQsCT8I/23tY4nkZG0NsdzBtv864T0/emoUdIty1URq9dIJ7C4G5vWHCkwVImBIMYjIxmLuSVZeJvx5qpa6zCdUVOH2ByykIp27FmCMwAHF9mvijdKhPoNgsrZ2Ui8nNJe1julgsJeO2vImY32fRUr9EcNIgzCZ18r94dnsQeOgwc7nY35gyMZZSsZY6VpZYChjp3kdiQsFRkCPP9mBDEZfYClT/PPHKE/+ESs+Np6M478r831W7QmqpPhyC6jyuEwGbzIDtYPBvKTownduNFF2HzNNFgWIGZKBzRRBdILU+vI3MgmJ4XdRe4x1weX0m3/xnfCdo6O0fyXrXJlyA1tAWdy63Xmuuw8WwWC+stUW1wB+ofL0DGkPqb5UFPbdTJK7je/b/iiHxWIvKVxx0auYxLhhGCsrI5Yt0TT+b7VO5fPizqEIHSlWVKsG9okGV6plqSXouxpVfvDcLMAiAck9YTqCC9R4o+FAqBPVjE2OhTe1DrwrmfFPHrIOhG/jMIbrT81hx3C6XkMY3aHwcU6atUAP15seO5qXCpgSdH1GzwFhV9lkY6AjIFzNhNhrsIiz9L+lEDqv59burn/ar/RKrehumOXmUzwkTJzygCs2L41hvxdk7RLFtiz7pU7WLCTGaAQlvsqBQklE7OfNh2DAsT9Bai/1m3vs3T2F2IdfgRuE6m3YNQiUVjvnaOZZawznvXUkQYmcm38r4LM8AYItbcON0JqRsdwRkG5BrH0L75sCoWW0VVzA+/YLhKEyIqPqZj2uYwGi4p/ndvrPAr8HU/k+odwQvBASkSJT4o5/iLMYyv+csyZAU33HoT092370p46Zwm0VLRi8z05V X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a code cleanup patch, no functionality change is expected. After the change, the line number reduces especially in the long migrate_pages_batch(). Signed-off-by: "Huang, Ying" Suggested-by: Alistair Popple Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: haoxin --- mm/migrate.c | 83 +++++++++++++++++++++++----------------------------- 1 file changed, 36 insertions(+), 47 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 6adaea05b80a..7d4047be5a4a 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1099,9 +1099,6 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page bool locked = false; bool dst_locked = false; - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - if (folio_ref_count(src) == 1) { /* Folio was freed from under us. So we are done. */ folio_clear_active(src); @@ -1351,16 +1348,6 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; - /* - * Migratability of hugepages depends on architectures and their size. - * This check is necessary because some callers of hugepage migration - * like soft offline and memory hotremove don't walk through page - * tables or check whether the hugepage is pmd-based or not before - * kicking migration. - */ - if (!hugepage_migration_supported(page_hstate(hpage))) - return -ENOSYS; - if (folio_ref_count(src) == 1) { /* page was freed from under us. So we are done. */ putback_active_hugepage(hpage); @@ -1527,6 +1514,20 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, cond_resched(); + /* + * Migratability of hugepages depends on architectures and + * their size. This check is necessary because some callers + * of hugepage migration like soft offline and memory + * hotremove don't walk through page tables or check whether + * the hugepage is pmd-based or not before kicking migration. + */ + if (!hugepage_migration_supported(folio_hstate(folio))) { + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + continue; + } + rc = unmap_and_move_huge_page(get_new_page, put_new_page, private, &folio->page, pass > 2, mode, @@ -1536,16 +1537,9 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, * Success: hugetlb folio will be put back * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list - * -ENOSYS: stay on the from list * Other errno: put on ret_folios list */ switch(rc) { - case -ENOSYS: - /* Hugetlb migration is unsupported */ - nr_failed++; - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - break; case -ENOMEM: /* * When memory is low, don't bother to try to migrate @@ -1631,6 +1625,28 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); + /* + * Large folio migration might be unsupported or + * the allocation might be failed so we should retry + * on the same folio with the large folio split + * to normal folios. + * + * Split folios are put in split_folios, and + * we will migrate them after the rest of the + * list is processed. + */ + if (!thp_migration_supported() && is_thp) { + nr_large_failed++; + stats->nr_thp_failed++; + if (!try_split_folio(folio, &split_folios)) { + stats->nr_thp_split++; + continue; + } + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + continue; + } + rc = migrate_folio_unmap(get_new_page, put_new_page, private, folio, &dst, pass > 2, force_lock, mode, reason, ret_folios); @@ -1642,36 +1658,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, * -EAGAIN: stay on the from list * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list - * -ENOSYS: stay on the from list * Other errno: put on ret_folios list */ switch(rc) { - /* - * Large folio migration might be unsupported or - * the allocation could've failed so we should retry - * on the same folio with the large folio split - * to normal folios. - * - * Split folios are put in split_folios, and - * we will migrate them after the rest of the - * list is processed. - */ - case -ENOSYS: - /* Large folio migration is unsupported */ - if (is_large) { - nr_large_failed++; - stats->nr_thp_failed += is_thp; - if (!try_split_folio(folio, &split_folios)) { - stats->nr_thp_split += is_thp; - break; - } - } else if (!no_split_folio_counting) { - nr_failed++; - } - - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - break; case -ENOMEM: /* * When memory is low, don't bother to try to migrate