From patchwork Mon Feb 13 12:34:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4F96C636D4 for ; Mon, 13 Feb 2023 12:35:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10A2E6B0073; Mon, 13 Feb 2023 07:35:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 093D36B0078; Mon, 13 Feb 2023 07:35:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E50D96B007B; Mon, 13 Feb 2023 07:35:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D280E6B0073 for ; Mon, 13 Feb 2023 07:35:23 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 98E5EC0216 for ; Mon, 13 Feb 2023 12:35:23 +0000 (UTC) X-FDA: 80462214126.26.20A44B5 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id 83982C0007 for ; Mon, 13 Feb 2023 12:35:20 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KO0L1mA6; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291720; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X/7w6AU0dVQ4ona2w+LdKDayTkxXkGEmhWg+x18hNdg=; b=jrAoPz5svtVVwv/ylrpVUHvJSPkDOcuhNpGzfa54Fy+Kg2vGzX6x8h71Se5Y08vgY9GKv4 KXvP1T19N6+0C8nR9yA4UabmUDuw87myhVm7V9Z9sHlFIxcoU+SL6YDPn0jEkSKaEpaRlZ Bou56ZvlrVYtB0ZGDC+5Mv7xaXuUlxM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=KO0L1mA6; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291720; a=rsa-sha256; cv=none; b=ZQ0cf5tklqTuMDXcWiAoUK5pahJazu3hrqu5PPRzrwmiENcs3woYXgZVCl5Uv+glBkCXSt rIDwVALOlRYsWn5QDfEjgTLH8NYEy9tZa0QwP8Ci6StaXLZEYO2dCgQ3cGjBWJzTzTAZF/ W/qG7QvJxwYSTq/czKq3jl1HocOUZfg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291720; x=1707827720; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=l+3xR+mdu+KvJS/5Kh2raKgnX+cbZNTvoyGmwX5GiZQ=; b=KO0L1mA6OVqg7RUdUazUmMHJJ8JpbotvvRZ62r6D86VRGYs67bITmXB9 1k4CDfws5HSMU2CRReNuznsO6aTSLF+6Vsgqa6D73LUrwkKHRiwAf50ba QczcBSEAZf6jrP6B0TA+RE1ftEeg1G9ccLXLiCqoDuQ5E6WDtv794IBky s/tqsv4Kwbq8KEm1CfFOA9zi+8tah3v+px6GhSiS8ck1brUil4+dwqSb+ iOCGxCuugV37cOxNh4l/hRjm4xXmvFZwc9IxnxO0VJNE5m+KbGqICOtGK D+S7XSixo7HMhSH1Jr/2IMCVX2aOlQI8BJcK45F4X7Pmd3MK3QOw0pWmh Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513154" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513154" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:19 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366593" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366593" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:15 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Alistair Popple , Zi Yan , Baolin Wang , Xin Hao , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 1/9] migrate_pages: organize stats with struct migrate_pages_stats Date: Mon, 13 Feb 2023 20:34:36 +0800 Message-Id: <20230213123444.155149-2-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: f1pyf13pjajd3qa53p8fwzuj1dia4yha X-Rspamd-Queue-Id: 83982C0007 X-HE-Tag: 1676291720-875829 X-HE-Meta: U2FsdGVkX19hF44AfhfsCxrPUsG4tqhXsp9cXDkSqnrdGvf6YrEpNoIJrRFjsh9qGCxe0VssfoEVLiF4Ee5JlC5UwQ4wWkmA4fgQU2t2NPT8R7kFoVW3ZqPC1EyuLeRPDMJ0LmSSys992tNp3W/jJfdikFRq86ukO7srq5vGn7WwiUTM4oJk+X8MJQtYzZGY4npV+iOFcA0M6P1UQe3gaSd6JtD1H+Vo8HqcB63dBzW29cFbbndVCbIRuY5t9OOY+AC6WLZHWPiIik2JDYhwDUg9x1+OX4TMCVylSzUxbTdPYYH6Vj/kwgaZYp0Nu/QfkyZXOqzdaFCFJy3BqlJ8rdj7rlgWEAVdJyT0vHWlFjewgjgiD9Kf0R229x5RbE59C0TrvE6RONRqgt8GTjSBjrkkndXcVKjQE3Gaq1dongzhEQOEEapuyslcMjg3mZGGBUG2UUV0n2X5E2ul7CuTWrVoBketNMvua726ongUkpDdi2X2PHHraWR/U7cdgQ84J855Yz27QjZlnULRX2OhZu/d9YIbV3jsQOw/zlGdpIV2am/tNpnkzz9v7xuN1iVRnQGERwFBWCTuwph71rEEjStBhGSyh01QApThTV1aTHtPzAXHqVDkx1OYYmIE1AW542MYiRV1t4DdhcxXWq34zr7W74mM7+j9dsfglsvzP/eH+fD2Fxn7NNol20myM4H1uiqSVKMbt/DcV/iZ3LQSQez/tUaVFoOYusDU5YvCz9zz6rOh8IPk5Iv7WkivCaffokYk3hTS/2izJxqfJRNw3TzIrKCMIzVu459ZiG3d/RqgMGW0alDElx80eJlfvTdv5b7mzY6SncF1rI38ehgxnHRcZ8O1JrsDW3bUgVScm/OYiUXh/AYAk13uP1iduKUjF+yPQA6Pra0V8c24D3i1KH0p/7ne/EDow6a4kiXNctjN6i8z5hRPqf4Rv9RLaCREDYc44k4cyikRFWGV+qV 3dTBZKpa k/Nx8NAsB4jNbGWeuJ/Gr1I4W8+2Hctt/ar/bnuSTjWMCkNWUmMqrl9YPTvEBH17H1k7qtHfkdxWHz2l1Qlyq7hNdcCaEBseL8hP5JVbGEVXSC5GuLisL3qVgthIWbt9sVajK+jV4gT4HaFom2IFH1/aolh9Ss/9IIi7fnSLAjQCgqyRRu5ZaFWVY5yxIJHIWnnwfWOncxizvZFVGJ587Kjy04fRXVWhM0mE7Nn+ZxlblIEtinhyPspX1qDmxaK8rK+GZScrF7QL31CxJ8OTON5LWyIWbNQbxt/oCaOmoQNnDEqvomaL3jBgLKO9/gjne+Bu6N0i8MOM8wBNqa94kly7CHV/FDGIZKi0Ef3hRbn99K+9twKA8Ye2xkbxgH124sua3Tf9Iz+QBF5Up7Qh3oDUY5XHxukCmBYaEYZTcFp9tVyNiigiOCZqOKRuiVrmnAHEs+KCvHYn9piUXwScWL+fRW/e43SKMoSoBVwgddfwPC6VTgWCR1Tp7bhnfmKCB2GMz2TWBFiSIyO9jDFDN/l6vfw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Define struct migrate_pages_stats to organize the various statistics in migrate_pages(). This makes it easier to collect and consume the statistics in multiple functions. This will be needed in the following patches in the series. Signed-off-by: "Huang, Ying" Reviewed-by: Alistair Popple Reviewed-by: Zi Yan Reviewed-by: Baolin Wang Reviewed-by: Xin Hao Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 60 +++++++++++++++++++++++++++++----------------------- 1 file changed, 34 insertions(+), 26 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 5b40b9040ba6..1a9cfcf857d2 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1414,6 +1414,16 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +struct migrate_pages_stats { + int nr_succeeded; /* Normal and large folios migrated successfully, in + units of base pages */ + int nr_failed_pages; /* Normal and large folios failed to be migrated, in + units of base pages. Untried folios aren't counted */ + int nr_thp_succeeded; /* THP migrated successfully */ + int nr_thp_failed; /* THP failed to be migrated */ + int nr_thp_split; /* THP split before migrating */ +}; + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1448,13 +1458,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int large_retry = 1; int thp_retry = 1; int nr_failed = 0; - int nr_failed_pages = 0; int nr_retry_pages = 0; - int nr_succeeded = 0; - int nr_thp_succeeded = 0; int nr_large_failed = 0; - int nr_thp_failed = 0; - int nr_thp_split = 0; int pass = 0; bool is_large = false; bool is_thp = false; @@ -1464,9 +1469,11 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + struct migrate_pages_stats stats; trace_mm_migrate_pages_start(mode, reason); + memset(&stats, 0, sizeof(stats)); split_folio_migration: for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; @@ -1520,9 +1527,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } /* Hugetlb migration is unsupported */ @@ -1530,7 +1537,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; list_move_tail(&folio->lru, &ret_folios); break; case -ENOMEM: @@ -1540,13 +1547,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - nr_thp_split += is_thp; + stats.nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1564,7 +1571,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - nr_failed_pages += nr_pages + nr_retry_pages; + stats.nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large * folios left in split_folios list. Move them back to migration @@ -1574,7 +1581,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, list_splice_init(&split_folios, from); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - nr_thp_failed += thp_retry; + stats.nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1586,8 +1593,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - nr_succeeded += nr_pages; - nr_thp_succeeded += is_thp; + stats.nr_succeeded += nr_pages; + stats.nr_thp_succeeded += is_thp; break; default: /* @@ -1598,20 +1605,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - nr_thp_failed += is_thp; + stats.nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - nr_failed_pages += nr_pages; + stats.nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - nr_thp_failed += thp_retry; - nr_failed_pages += nr_retry_pages; + stats.nr_thp_failed += thp_retry; + stats.nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1644,16 +1651,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (list_empty(from)) rc = 0; - count_vm_events(PGMIGRATE_SUCCESS, nr_succeeded); - count_vm_events(PGMIGRATE_FAIL, nr_failed_pages); - count_vm_events(THP_MIGRATION_SUCCESS, nr_thp_succeeded); - count_vm_events(THP_MIGRATION_FAIL, nr_thp_failed); - count_vm_events(THP_MIGRATION_SPLIT, nr_thp_split); - trace_mm_migrate_pages(nr_succeeded, nr_failed_pages, nr_thp_succeeded, - nr_thp_failed, nr_thp_split, mode, reason); + count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); + count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); + count_vm_events(THP_MIGRATION_SUCCESS, stats.nr_thp_succeeded); + count_vm_events(THP_MIGRATION_FAIL, stats.nr_thp_failed); + count_vm_events(THP_MIGRATION_SPLIT, stats.nr_thp_split); + trace_mm_migrate_pages(stats.nr_succeeded, stats.nr_failed_pages, + stats.nr_thp_succeeded, stats.nr_thp_failed, + stats.nr_thp_split, mode, reason); if (ret_succeeded) - *ret_succeeded = nr_succeeded; + *ret_succeeded = stats.nr_succeeded; return rc; } From patchwork Mon Feb 13 12:34:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8D78C636D4 for ; Mon, 13 Feb 2023 12:35:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85D3F6B0078; Mon, 13 Feb 2023 07:35:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E7EC6B007B; Mon, 13 Feb 2023 07:35:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 611A06B007D; Mon, 13 Feb 2023 07:35:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 50A366B0078 for ; Mon, 13 Feb 2023 07:35:27 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0FA1714023F for ; Mon, 13 Feb 2023 12:35:27 +0000 (UTC) X-FDA: 80462214294.22.1E5117D Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id ED4F5C0008 for ; Mon, 13 Feb 2023 12:35:24 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=hniG6EvQ; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291725; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kKTCNkv9GqWNS7mKqiYpyyO31OvJb7yE0UqXMZGOTnA=; b=Tl5tihmZ5BxngNkbun6NAKwPjzY/GjFqeCwisJnmdN0OLMyDFQ5mR8RBPbHBVc9XUL/4+W RASPHszgek5sKsEjU/tq/r5p51/cjUx1B44HkiaLeopQ3HdwaMk61LZBLbuKeYuMmwBIFw Kleq4VWpE9Wm98FbkOYSwO1nnuSa8DY= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=hniG6EvQ; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291725; a=rsa-sha256; cv=none; b=2f+puUiCZUIrsTtPfjWTzRbZPQea5+JJgavmU+0Yyc+SmMgD+PS+RqNY6w1BA03ZpSGLcz zhuVpz12ufFXFYWfRoqK9HZVzKk3A964/3YkLKWgWgIVt3wMifeZ2KiUNgjCzHN+2eYXdf uygnI04peO0DzDBHMMP9/WDLMxCeZpw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291725; x=1707827725; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6XvIwqCGGbQdtVE2GfY6YLHxNbOVZ88E0KPcEuieCq0=; b=hniG6EvQnXC3heAO5+8LOlUkGhekQNL1+P84WoWQwVfELDusndxrHFA2 SZ0NN8HDXHdBYPkWEpPE8+7WS49fiPh+DiNWExzmc6a1NXDAY458peFRk IqFNDSL54dX1ksSqvNZaMzTgL93+zo2Tc6YpVMbphW15cT/7L1GgcvtBT 9OVRQFUpslevzxNKnobYBuF+/ncIeyNcOrrkkt3M2UX1YRXbpIOTQmd/7 nv8TK706jm6SFm0+OMZhPjcnWy4zpLZRwaQHLPZqGUDcx5F8s0D0dPBkH 5ejgSvpAleSsLwpBeeFxp3LixasRUXSlw+Cwj42+nLrfEfoEA7TahLn+m A==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513182" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513182" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:23 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366601" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366601" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:19 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Baolin Wang , Xin Hao , Zi Yan , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 2/9] migrate_pages: separate hugetlb folios migration Date: Mon, 13 Feb 2023 20:34:37 +0800 Message-Id: <20230213123444.155149-3-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: ED4F5C0008 X-Stat-Signature: e8ncxuic5hm7ygzo51oj1na55msy7ry8 X-HE-Tag: 1676291724-956534 X-HE-Meta: U2FsdGVkX1/ZIvTtJFbr/Q/xjGLn4UPWF0ujwvkrYPs3fCnx2Y+ThGrlttJVRLag+Znxn1bpjoZsvINB0S4SU70KLZxEz1Cbdgd76qX+9CGsuHYg+A81FHF34vFgVOO9PrZpnb1QGu9ti6OfzBYkyRAF4xf2L8Ld+0pAoYA1qgyappRyAlLE1e+GY9klaIOcnX3h5CZh+H1VuZzh1VsMf+N7V62wr2p4Fq0YE0HcnvTiZ7Xh5sLPiLAKBQeTiPTZhTizlxHJdc2N0yB82Fn5t003gwzOK5bYQEhQlXIqszfMzc3+XZBt6Au6Hyq5wOioi9C+I3RraqV+SiUcZJx7KQgfScDmaViw8KK46a5mjOpcIDjfulEK+If4h8vsS8fPGmA0mpjickSCp2+vIXc6m9c0mYWDILY56scMV1r7kFsWD8zez0ZR2KKOQEeBrC5R+Gq2EAx+dNyP6qMnG54NOjsHGiJS8KPRkpk8Y4pr4ymmsj144wSRslODGHMtGB+gJwRROQGMmNYfdCIVL6Qm9xO7S6k4MXe2Ux9O1v9V+sg+jm2oBD+l9eIW1mex0p9VdLB7koJJE650pi5Z5L6ORIskWLqz9HxnPVOVsw05yH9waSVkUfWGAZe8i1HHWJ5i1BGfnrCxa7gur6l5QM/+VTFCr5HubkL53IAUfIBP2Lr7bLyVsYinPei6XUOlQrwSNPpI9ONteuayuHBs6z6iWU7nwuzn6D5c8h6jaZFsl8hFveE/G27VrlL0P3xzmoc/pLRI92HP9Xu8tzJJxMHQAUWa9/qjAKvdTG0+yXe3TrUaktbNKCjFmbdBRzm7oI2RBwYVuDvNijsE0yywnrggIQzhq8r8KjdvcjFSuj/w4Y9VD7QllZj4f8rb5aGA+jah2Rr/g/s2KZpBrTiLM6Cny41/xABY8bzwIbjYpjDWS+gBZ9y3AgI1aTLFDu/NYGXtn3iFXXxF8uMmj9agJIo h9exjisw 4bcP+Qw20GpU/LvU4sKXl7BCXqkRIPuvCloTAchRAtu5DD16GvvDDZItJd6Rd2X4QX/jX0Jz/cmmQejZlHgdcjQcDuOHxaoobSHZ9FU8SkRJmTCnG4yqocl5ZwhlYqkUdBj/O1poTIoA4fU3PSM+2LX8umecjrIQOwGAvCS70p4GKNydkFzBcpdGta8185yz+I1rOLPagfJdMfdpqTGowZR19Vx9Li68s4Yy5VOrgZXGqzJSvZiRlrUh4Ky9nc8v149l1OYRAoVC+oi/S9s9qAZX2mveB9I/4SHN4WylLt404vw227wV6fC/i5Sxwfidi111duR0y37fK7qLmij4bSuxUtxIJE7FUROdkvL1iAM9kHh7uyOj2532woNRZhbCrv0/6+e/Z2OSnIQjVNaiHMu6H7fNK8kI2OBTKbAGDhNraIsFzwb2D6CecQi/HqB4xS/+WwcxVGSKSIDxvyBg2wItSri2ixbctxkFIOg2oPkzM1JnGl08Ryram9kT2lsIA/h8bxkOTGiT5rxESsn2KuNpZOQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving for the non-hugetlb folios. Based on that we can batch the TLB shootdown during the folio migration and make it possible to use some hardware accelerator for the folio copying. In this patch the hugetlb folios and non-hugetlb folios migration is separated in migrate_pages() to make it easy to change the non-hugetlb folios migration implementation. Signed-off-by: "Huang, Ying" Reviewed-by: Baolin Wang Reviewed-by: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 141 +++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 119 insertions(+), 22 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 1a9cfcf857d2..586a32bdaa71 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1414,6 +1414,8 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +#define NR_MAX_MIGRATE_PAGES_RETRY 10 + struct migrate_pages_stats { int nr_succeeded; /* Normal and large folios migrated successfully, in units of base pages */ @@ -1424,6 +1426,95 @@ struct migrate_pages_stats { int nr_thp_split; /* THP split before migrating */ }; +/* + * Returns the number of hugetlb folios that were not migrated, or an error code + * after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no hugetlb folios are movable + * any more because the list has become empty or no retryable hugetlb folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * only if ret != 0. + */ +static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, + struct migrate_pages_stats *stats, + struct list_head *ret_folios) +{ + int retry = 1; + int nr_failed = 0; + int nr_retry_pages = 0; + int pass = 0; + struct folio *folio, *folio2; + int rc, nr_pages; + + for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && retry; pass++) { + retry = 0; + nr_retry_pages = 0; + + list_for_each_entry_safe(folio, folio2, from, lru) { + if (!folio_test_hugetlb(folio)) + continue; + + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = unmap_and_move_huge_page(get_new_page, + put_new_page, private, + &folio->page, pass > 2, mode, + reason, ret_folios); + /* + * The rules are: + * Success: hugetlb folio will be put back + * -EAGAIN: stay on the from list + * -ENOMEM: stay on the from list + * -ENOSYS: stay on the from list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -ENOSYS: + /* Hugetlb migration is unsupported */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + break; + case -ENOMEM: + /* + * When memory is low, don't bother to try to migrate + * other folios, just exit. + */ + stats->nr_failed_pages += nr_pages + nr_retry_pages; + return -ENOMEM; + case -EAGAIN: + retry++; + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + break; + default: + /* + * Permanent failure (-EBUSY, etc.): + * unlike -EAGAIN case, the failed folio is + * removed from migration folio list and not + * retried in the next outer loop. + */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + break; + } + } + } + /* + * nr_failed is number of hugetlb folios failed to be migrated. After + * NR_MAX_MIGRATE_PAGES_RETRY attempts, give up and count retried hugetlb + * folios as failed. + */ + nr_failed += retry; + stats->nr_failed_pages += nr_retry_pages; + + return nr_failed; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1440,10 +1531,10 @@ struct migrate_pages_stats { * @ret_succeeded: Set to the number of folios migrated successfully if * the caller passes a non-NULL pointer. * - * The function returns after 10 attempts or if no folios are movable any more - * because the list has become empty or no retryable folios exist any more. - * It is caller's responsibility to call putback_movable_pages() to return folios - * to the LRU or free list only if ret != 0. + * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios + * are movable any more because the list has become empty or no retryable folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * only if ret != 0. * * Returns the number of {normal folio, large folio, hugetlb} that were not * migrated, or an error code. The number of large folio splits will be @@ -1457,7 +1548,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed = 0; + int nr_failed; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1474,38 +1565,45 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, trace_mm_migrate_pages_start(mode, reason); memset(&stats, 0, sizeof(stats)); + rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, + &stats, &ret_folios); + if (rc < 0) + goto out; + nr_failed = rc; + split_folio_migration: - for (pass = 0; pass < 10 && (retry || large_retry); pass++) { + for (pass = 0; + pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); + pass++) { retry = 0; large_retry = 0; thp_retry = 0; nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + /* Retried hugetlb folios will be kept in list */ + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + /* * Large folio statistics is based on the source large * folio. Capture required information that might get * lost during migration. */ - is_large = folio_test_large(folio) && !folio_test_hugetlb(folio); + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); + cond_resched(); - if (folio_test_hugetlb(folio)) - rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, - &folio->page, pass > 2, mode, - reason, - &ret_folios); - else - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, &ret_folios); + rc = unmap_and_move(get_new_page, put_new_page, + private, folio, pass > 2, mode, + reason, &ret_folios); /* * The rules are: - * Success: non hugetlb folio will be freed, hugetlb - * folio will be put back + * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list @@ -1532,7 +1630,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, stats.nr_thp_split += is_thp; break; } - /* Hugetlb migration is unsupported */ } else if (!no_split_folio_counting) { nr_failed++; } @@ -1626,8 +1723,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (!list_empty(&split_folios)) { /* - * Move non-migrated folios (after 10 retries) to ret_folios - * to avoid migrating them again. + * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY + * retries) to ret_folios to avoid migrating them again. */ list_splice_init(from, &ret_folios); list_splice_init(&split_folios, from); From patchwork Mon Feb 13 12:34:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE75EC6379F for ; Mon, 13 Feb 2023 12:35:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D2A06B007B; Mon, 13 Feb 2023 07:35:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 85BA46B007D; Mon, 13 Feb 2023 07:35:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6871B280001; Mon, 13 Feb 2023 07:35:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 56B856B007B for ; Mon, 13 Feb 2023 07:35:30 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3470914032C for ; Mon, 13 Feb 2023 12:35:30 +0000 (UTC) X-FDA: 80462214420.24.D7C871D Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id 07E68C0012 for ; Mon, 13 Feb 2023 12:35:27 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GfmGWoTf; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1+ZHJ0rLiUNbRG7hVjliBVTg3A3l6FGhfIhApQcVfrQ=; b=pO4MTQxq4WMWCLRGtuws7/sd7tJ+mJYS5vcI//iVJBOsaLYiiJS9TFoB+/KtbILC1XgoES XGWbAyK8U3nz8MwouNpyil+iEBJ3bSysdx4VWabvBnLWt5ryKsnT67ysMMqT51Lu2Pw+gY +Gc76aMwVdOYC5rA5Wo8wmSlcJKKCC8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GfmGWoTf; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291728; a=rsa-sha256; cv=none; b=GVmYRc6FmbLqT068cSPjip7ORQpbS1A8XVdRKKM/QAQ7w1eEy6SPB5EdN0OE0evWdVBEr9 YPuRA7o0gP4/W+50b1MZ+oPYtqnG8BcCUHmTKmQFidmqr82d2sgNkP0uRL/l9XyPfgux64 iUVbUJLsjutRtFXc7JPCpLY2I6qc9lk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291728; x=1707827728; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6l4ikwjy8UmsuY4ji+NzBGTMG5CFHZs/zfizs/IbAY0=; b=GfmGWoTfiEJi3BNxIICr16aOc2rDvSzMzXzqEeW0JyNkmLA9cx9/ZiLK /Dmsr8qgGcO/qLQKR8MdYBrQ1GsJlbGnKMPBws3h6CxaYCJNgMLtB6QSf U0bWMsl/1Fh00OSfX3fHIcH9vluaPDMeQOH7inTgeGVV81iULI9dKLaXT wBc+QUOWLzrQs0nipo2nIpdo8eyMAMAvZnocXw/JcZ54t7zIUVihQojOH rHhLg3ngIyJzGub2UMjQriuKmpN9FhiNEYFhmmUKwU/y07kwtPIxnjCTL F9HO7ERnPx3na437eGbxAoyyFXUc7hd4Iyv3VnI2c4MTjA+VLO9LbAbmh A==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513204" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513204" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:27 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366614" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366614" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:23 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Baolin Wang , Zi Yan , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Xin Hao , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 3/9] migrate_pages: restrict number of pages to migrate in batch Date: Mon, 13 Feb 2023 20:34:38 +0800 Message-Id: <20230213123444.155149-4-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 07E68C0012 X-Stat-Signature: fwc4rurs4suojc719rf8pyam8i7hi7y5 X-HE-Tag: 1676291727-398604 X-HE-Meta: U2FsdGVkX1+0XWxGAfQTVm5vVpnIOFcEefNu4IP14qmL4dbZFev5uAokIvL8Uu9PlN9CRP01KktM4C5+KxaO5HEQ5tcEYFb0Zuti4tpFUYOn1JwRLOoL5DtCK83o3Tl2yeJpehla2z8iQimJET1oGB0b9sYeBw8gOFzC/U0pXd8STtEzRoIHzn4Hwfsz+vDO7/QqAwbi8uhHzYoh3NdIAIJLCfWzsSnM5K3o7ZFpGfUCUFSEJ2gaokJ82OY0qrtodO22cX2w1g+zzZl57AaF7F9gKq/P8/E8gV25OuZRmilO785pxkZuywkrH/InGblUMtIH2DSjEQukD9NV5k80tg9gCGLg0dXBmKGXAFp28n0cOSe2i77fQKZoSUvmEDR75ZqjHf6bKcsnFWaCNMUbzmfEC69Y+e6BbjTmW4NtOfhz+oJikSjJKHAQ/q1CGjIAlTFLkwNN/xlf70b2x/9O+lJasbH/TEsfm1AEBp+TvwiUlJ5Cb2KmaNU/rVSiUucqqhCcXxzB4hj1qJK5/f1qiTl+NtcA0PO2pq5mTvSXt33w2HD/qI7ZABoeRSfRt/RtooLcFYi5ZVx9/bMjS6f86L8DErgIa4Rar6KHr9p3FTQCycFQQOqVZn0S5Uz1Qdp5nM2Sp13/fuNvn4UwdkjhdgHBs0NGVRtz37ysoDZJKG3eZ0Hhen4ECB1MhgH3frmDtLfeZfDG2w9lbbCXV2Aifutdk5ej2ow7RMk/rbnNBqUJmlcDPU3jen5nsC+JMaumBlpXVcZzvWffluriFK6lP+tledi7tTiibwf0/zCFOVbVxiepd+ag6C98x8aXodzaCfbk45z5usEJMr2NaEWrjnzVk7VPYuC8CFu8ekKmOsUM2IaDQCN4UPHTFBZuLRlflsyBOLuJMzKzkHXXTXMBZAG6BZtfJ17MvlNapbrHTlJdGKGxUnn/1KJBnklbWMLUdOEuoEOYFPDlEx2cRcq 22AZOA8x tqJJb2jZItjF8ICeKaGKSPhhiAHzlaoHKs49FnqNQaj5sCPZ5Q0/lKL3VNeFb2I/eMU83W/z7la/OFMTrqyhyplUtZW3jZMpyggQZrxapm9cJSKhDebqEt2XlP+1fC1OPR48vwDLC+05gU+dTaDHtu5cQhvvEKGri8H5SPyA9BKtjMbCNbLbWn0277OuU/yftMPOlPGsVVGIS9q6Rxj64hjkswrHmY6UUMhA3HGZ4wGvj18Ldn5VC7DjZ14HNMFZdY4OYlBlsGChqhyFEJ1eNlGMFFTR5bmBzAu9xdSqYb3lXpv3slPD5uGs/jpx+a1lh4Lj4sHaHHL5EW/jk0FVbGo4iRVRlkwnKY2gyEGJxw57dzWbrlHlTRge5rJpqbDfrkRUwHMB9H70O6Z3M9gV+zVJMrLd4+H48eamf2AFCWUnxT3eHkDmrIxvNF/fWClp7TsX0rvNv3JyKUReMftl8AmTjNK7xPyed3MFuXKO5tuu21Myc4y6Xue2HN2gQZclsSWOhITdSCKsvHarQT0B/Iph/5w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving for non-hugetlb folios. If we had batched the folio unmapping, all folios to be migrated would be unmapped before copying the contents and flags of the folios. If the folios that were passed to migrate_pages() were too many in unit of pages, the execution of the processes would be stopped for too long time, thus too long latency. For example, migrate_pages() syscall will call migrate_pages() with all folios of a process. To avoid this possible issue, in this patch, we restrict the number of pages to be migrated to be no more than HPAGE_PMD_NR. That is, the influence is at the same level of THP migration. Signed-off-by: "Huang, Ying" Reviewed-by: Baolin Wang Cc: Zi Yan Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: Xin Hao Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 174 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 106 insertions(+), 68 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 586a32bdaa71..d436f35fa145 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1414,6 +1414,11 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f return rc; } +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#define NR_MAX_BATCHED_MIGRATION HPAGE_PMD_NR +#else +#define NR_MAX_BATCHED_MIGRATION 512 +#endif #define NR_MAX_MIGRATE_PAGES_RETRY 10 struct migrate_pages_stats { @@ -1515,40 +1520,15 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, return nr_failed; } -/* - * migrate_pages - migrate the folios specified in a list, to the free folios - * supplied as the target for the page migration - * - * @from: The list of folios to be migrated. - * @get_new_page: The function used to allocate free folios to be used - * as the target of the folio migration. - * @put_new_page: The function used to free target folios if migration - * fails, or NULL if no special handling is necessary. - * @private: Private data to be passed on to get_new_page() - * @mode: The migration mode that specifies the constraints for - * folio migration, if any. - * @reason: The reason for folio migration. - * @ret_succeeded: Set to the number of folios migrated successfully if - * the caller passes a non-NULL pointer. - * - * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios - * are movable any more because the list has become empty or no retryable folios - * exist any more. It is caller's responsibility to call putback_movable_pages() - * only if ret != 0. - * - * Returns the number of {normal folio, large folio, hugetlb} that were not - * migrated, or an error code. The number of large folio splits will be - * considered as the number of non-migrated large folio, no matter how many - * split folios of the large folio are migrated successfully. - */ -int migrate_pages(struct list_head *from, new_page_t get_new_page, +static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, - enum migrate_mode mode, int reason, unsigned int *ret_succeeded) + enum migrate_mode mode, int reason, struct list_head *ret_folios, + struct migrate_pages_stats *stats) { int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed; + int nr_failed = 0; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1556,20 +1536,9 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, bool is_thp = false; struct folio *folio, *folio2; int rc, nr_pages; - LIST_HEAD(ret_folios); LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; - struct migrate_pages_stats stats; - - trace_mm_migrate_pages_start(mode, reason); - - memset(&stats, 0, sizeof(stats)); - rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, - &stats, &ret_folios); - if (rc < 0) - goto out; - nr_failed = rc; split_folio_migration: for (pass = 0; @@ -1581,12 +1550,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { - /* Retried hugetlb folios will be kept in list */ - if (folio_test_hugetlb(folio)) { - list_move_tail(&folio->lru, &ret_folios); - continue; - } - /* * Large folio statistics is based on the source large * folio. Capture required information that might get @@ -1600,15 +1563,14 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, rc = unmap_and_move(get_new_page, put_new_page, private, folio, pass > 2, mode, - reason, &ret_folios); + reason, ret_folios); /* * The rules are: * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list - * Other errno: put on ret_folios list then splice to - * from list + * Other errno: put on ret_folios list */ switch(rc) { /* @@ -1625,17 +1587,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, /* Large folio migration is unsupported */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; if (!try_split_folio(folio, &split_folios)) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, &ret_folios); + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); break; case -ENOMEM: /* @@ -1644,13 +1606,13 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; /* Large folio NUMA faulting doesn't split to retry. */ if (!nosplit) { int ret = try_split_folio(folio, &split_folios); if (!ret) { - stats.nr_thp_split += is_thp; + stats->nr_thp_split += is_thp; break; } else if (reason == MR_LONGTERM_PIN && ret == -EAGAIN) { @@ -1668,17 +1630,17 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_failed++; } - stats.nr_failed_pages += nr_pages + nr_retry_pages; + stats->nr_failed_pages += nr_pages + nr_retry_pages; /* * There might be some split folios of fail-to-migrate large - * folios left in split_folios list. Move them back to migration + * folios left in split_folios list. Move them to ret_folios * list so that they could be put back to the right list by * the caller otherwise the folio refcnt will be leaked. */ - list_splice_init(&split_folios, from); + list_splice_init(&split_folios, ret_folios); /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; + stats->nr_thp_failed += thp_retry; goto out; case -EAGAIN: if (is_large) { @@ -1690,8 +1652,8 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages += nr_pages; break; case MIGRATEPAGE_SUCCESS: - stats.nr_succeeded += nr_pages; - stats.nr_thp_succeeded += is_thp; + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; break; default: /* @@ -1702,20 +1664,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, */ if (is_large) { nr_large_failed++; - stats.nr_thp_failed += is_thp; + stats->nr_thp_failed += is_thp; } else if (!no_split_folio_counting) { nr_failed++; } - stats.nr_failed_pages += nr_pages; + stats->nr_failed_pages += nr_pages; break; } } } nr_failed += retry; nr_large_failed += large_retry; - stats.nr_thp_failed += thp_retry; - stats.nr_failed_pages += nr_retry_pages; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a @@ -1726,7 +1688,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY * retries) to ret_folios to avoid migrating them again. */ - list_splice_init(from, &ret_folios); + list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; retry = 1; @@ -1734,6 +1696,82 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } rc = nr_failed + nr_large_failed; +out: + return rc; +} + +/* + * migrate_pages - migrate the folios specified in a list, to the free folios + * supplied as the target for the page migration + * + * @from: The list of folios to be migrated. + * @get_new_page: The function used to allocate free folios to be used + * as the target of the folio migration. + * @put_new_page: The function used to free target folios if migration + * fails, or NULL if no special handling is necessary. + * @private: Private data to be passed on to get_new_page() + * @mode: The migration mode that specifies the constraints for + * folio migration, if any. + * @reason: The reason for folio migration. + * @ret_succeeded: Set to the number of folios migrated successfully if + * the caller passes a non-NULL pointer. + * + * The function returns after NR_MAX_MIGRATE_PAGES_RETRY attempts or if no folios + * are movable any more because the list has become empty or no retryable folios + * exist any more. It is caller's responsibility to call putback_movable_pages() + * only if ret != 0. + * + * Returns the number of {normal folio, large folio, hugetlb} that were not + * migrated, or an error code. The number of large folio splits will be + * considered as the number of non-migrated large folio, no matter how many + * split folios of the large folio are migrated successfully. + */ +int migrate_pages(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, unsigned int *ret_succeeded) +{ + int rc, rc_gather; + int nr_pages; + struct folio *folio, *folio2; + LIST_HEAD(folios); + LIST_HEAD(ret_folios); + struct migrate_pages_stats stats; + + trace_mm_migrate_pages_start(mode, reason); + + memset(&stats, 0, sizeof(stats)); + + rc_gather = migrate_hugetlbs(from, get_new_page, put_new_page, private, + mode, reason, &stats, &ret_folios); + if (rc_gather < 0) + goto out; +again: + nr_pages = 0; + list_for_each_entry_safe(folio, folio2, from, lru) { + /* Retried hugetlb folios will be kept in list */ + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + + nr_pages += folio_nr_pages(folio); + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + break; + } + if (nr_pages > NR_MAX_BATCHED_MIGRATION) + list_cut_before(&folios, from, &folio->lru); + else + list_splice_init(from, &folios); + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &stats); + list_splice_tail_init(&folios, &ret_folios); + if (rc < 0) { + rc_gather = rc; + goto out; + } + rc_gather += rc; + if (!list_empty(from)) + goto again; out: /* * Put the permanent failure folio back to migration list, they @@ -1746,7 +1784,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, * are migrated successfully. */ if (list_empty(from)) - rc = 0; + rc_gather = 0; count_vm_events(PGMIGRATE_SUCCESS, stats.nr_succeeded); count_vm_events(PGMIGRATE_FAIL, stats.nr_failed_pages); @@ -1760,7 +1798,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (ret_succeeded) *ret_succeeded = stats.nr_succeeded; - return rc; + return rc_gather; } struct page *alloc_migration_target(struct page *page, unsigned long private) From patchwork Mon Feb 13 12:34:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21547C636CC for ; Mon, 13 Feb 2023 12:35:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96F1B6B007D; Mon, 13 Feb 2023 07:35:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8FB4A280001; Mon, 13 Feb 2023 07:35:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7718B6B0080; Mon, 13 Feb 2023 07:35:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 644406B007D for ; Mon, 13 Feb 2023 07:35:34 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3E0E71A05F0 for ; Mon, 13 Feb 2023 12:35:34 +0000 (UTC) X-FDA: 80462214588.14.1C02532 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id D4CF3C0007 for ; Mon, 13 Feb 2023 12:35:31 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Pm74s98j; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291732; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZFIi0s5drFbPlmDXPFosTCFwtCRMcMEqzhBaJWzq/ss=; b=mKcehGYwNBxPv79QQlQ+L27PbigyxLz+WNn+FqKjkiO1Xt68GVM9FB0vnZApezQOoj7FU+ zit97xPF8ufYzfsdpWZEKsZUkYdmMyC1byplxwAon73cJYtr/Iy7NIw25LV1fwbGRH+kDe 6G4iDY4APoTi2+PlmKXTXxk7QnQSjgQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Pm74s98j; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291732; a=rsa-sha256; cv=none; b=EX77N7G/KC7L3bhUiNw0ToQ1TL/6hTPQ9QynYx13we54Y81zR8sZbG0RCzL2AqtPCT8cH6 n3y498q+IL1NJU1UgG7zAj2ZyYeSTsOOlasyiPysMAULOvdr0IWr9IpdjuYEFHDLOYgKA4 UUr1iDr5EZLl/8e9oc712WW6jmOlfO8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291731; x=1707827731; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3XEbG6JD2LdI1CjVNNq0obsv14/SFsm/rzfKm+kf4eA=; b=Pm74s98jQmkqJKLjbHgQZV/Tj3/8F4Z05naZJNZGCZuhngUPPjAheQn2 BJNlg5jtpm8/wADUt1f1kCido3hXdz0yovnqrB2liI4ILCFoQLqn7Ujzo UFi4pZoC83OCcxf0CAKyX6n0uxw6YHJCVhSWCDwEBtizShzMm6brgKgnM TqQRYdX+rA1F4HfyAxRP9sX4u3CcFec48ZcOnc90X+qMCCpwVdXlYRFgd JDLYKBuxIGi6R8NOptMM1xfZXLayWdvBbtxhEdPJFMK5Q3afc3D25yu/U yQGk6ZctyidsqOrvs7cG5JeHyTSWkGt/vGfyxrw3Cdke4+rc6VLKN31rG g==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513224" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513224" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:31 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366631" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366631" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:27 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Baolin Wang , Xin Hao , Zi Yan , Yang Shi , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 4/9] migrate_pages: split unmap_and_move() to _unmap() and _move() Date: Mon, 13 Feb 2023 20:34:39 +0800 Message-Id: <20230213123444.155149-5-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D4CF3C0007 X-Stat-Signature: 8rb33pzwcensm9oj94yb49zb1oypm3q5 X-HE-Tag: 1676291731-154814 X-HE-Meta: U2FsdGVkX1/bDRF8T/OiAi5emGzEtP2WXpGJcJKvjMphrXvXwGlLuFv7tt48ynvGlvB75OTCkyB6YIvxaeCOeGnBQHUjLzrgv60RA4jxq0+e9n5U438nlSd7rtCrrAppVdG1GnhVUj0K3jD42MI46U9POH6572G6s4WSoxLNE5YwrvenR92tk4j7X8k0CJEttvXuCEa8ZWjvqvAzhvDvfj0xrNAcVI7B3fbeIlvS7crk5E2Bq/RuwemGKe3BIZLLqIM3MOt2hsrFLc41nrUcOEyLpHJDfq8pShc/TlXpKI9bG551o14HGjDdOK8vW6+bQJiNoH3+yIAiqA1HCm7U1D5Ls1rmPZKLtKxAO9EUiQxYVSFPxOgx7M6awKTeKwLFbD66LNJoZMMkBTH081DRSjBc1pC1Xk/YQilNjoQqU6+rrezCtokrOb5BXfkt/4OzUs2hjKfswut8eQsMidhBZLt8e2+BnGrs2mAJAlu+OrbkDZGYLd/7qm4y7IV1+l05ab/Xkknbzzs0GKhmAKaKpvOzpX8bpXeFeyBYofHnfQF9Fdu6iMRKhkBpiCO/uRui4fcF++bNtd+m+LlnPXm+8aGzyGYSix0oh4XR7wMWb6ntiDtD1PlsHT/lNxAaWvFaBePNCy7y6nCa7BUZAZNM+RrV7bSiZdeWFhV+A03T8iBQ4AEB1jjKbHJ1GfalZS5lJGu0uUGsOkKEuGZRfK/Q3xFZ4zLI35JheT9ovyCwijXAjeIxhAqkTYC17Z7Dpa3S+viDQFRPPWOHGaT8+2vgoeSBo4EU8UrxoD0C8PUfTVziVM919fkuC62OW4aZAwsjzydGpJTV8rgvGHfpppylrhyhF+ksXWipTTbam2b/X2mjzNgq7zSGI2zV5G0YvsB5kLGUZKVpGWdDq80S1l1SpbsrpugbP/CBO6cTRYE/NqWoegdTJPRVChZmOZvAXFT3g9dPQh0NCTdb+Ss63W4 GQ+HecLI VsAD8t2N8ZBMAvRA9nMGb+hCm7qTuXOe9hOdsIUoiufBzT8ZjKqGoUsM59sAXa1hEwkBHNlS84OvU0KGa49ReQFMwXDG6HLFmkRPVuQUCiNY2iqrKwSK4dSsqQAObafjA5SHJQQutXsXmO+RRuHA0BfWNdx24gm8QsTwRhuShcqpu/53anR/Iwzwd2a43wzua1ffXCfAemBxI2pL+oqq276/S2dDoSVMbYBBuMDlpinATtqkzZZKgOOAz+/HtKg17KoHVBzIjO622MzoJ+ooejkN1RY/rC4lCqlunj+yILski+i8ZEO6P/H+rPhqo4rSDNOeN9eAVovLqrzSVwiRiZuSxhIVJoJbqCYRoVvk9zMsb4QhmF5TXDM4UAJ3Z7rXBhFKf+0eCY/HNDukx9xxTBlaPLAf2DKzgf93YMLbKCtnBr6no49LHKJw7RYCVeN57ycnh7a6vf2c+cbw7llLO25VxswJBKvlXxUlsqqBW+ynXjK2WWdrHQGS+yPQYIs+mNnvnrO088baEY5VhKwhevCOKsQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving. In this patch, unmap_and_move() is split to migrate_folio_unmap() and migrate_folio_move(). So, we can batch _unmap() and _move() in different loops later. To pass some information between unmap and move, the original unused dst->mapping and dst->private are used. Signed-off-by: "Huang, Ying" Reviewed-by: Baolin Wang Reviewed-by: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- include/linux/migrate.h | 1 + mm/migrate.c | 169 ++++++++++++++++++++++++++++++---------- 2 files changed, 129 insertions(+), 41 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index bdff950a8bb4..c88b96b48be7 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -18,6 +18,7 @@ struct migration_target_control; * - zero on page migration success; */ #define MIGRATEPAGE_SUCCESS 0 +#define MIGRATEPAGE_UNMAP 1 /** * struct movable_operations - Driver page migration diff --git a/mm/migrate.c b/mm/migrate.c index d436f35fa145..5fd18a7cce62 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1027,11 +1027,53 @@ static int move_to_new_folio(struct folio *dst, struct folio *src, return rc; } -static int __unmap_and_move(struct folio *src, struct folio *dst, +/* + * To record some information during migration, we use some unused + * fields (mapping and private) of struct folio of the newly allocated + * destination folio. This is safe because nobody is using them + * except us. + */ +static void __migrate_folio_record(struct folio *dst, + unsigned long page_was_mapped, + struct anon_vma *anon_vma) +{ + dst->mapping = (void *)anon_vma; + dst->private = (void *)page_was_mapped; +} + +static void __migrate_folio_extract(struct folio *dst, + int *page_was_mappedp, + struct anon_vma **anon_vmap) +{ + *anon_vmap = (void *)dst->mapping; + *page_was_mappedp = (unsigned long)dst->private; + dst->mapping = NULL; + dst->private = NULL; +} + +/* Cleanup src folio upon migration success */ +static void migrate_folio_done(struct folio *src, + enum migrate_reason reason) +{ + /* + * Compaction can migrate also non-LRU pages which are + * not accounted to NR_ISOLATED_*. They can be recognized + * as __PageMovable + */ + if (likely(!__folio_test_movable(src))) + mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + + folio_is_file_lru(src), -folio_nr_pages(src)); + + if (reason != MR_MEMORY_FAILURE) + /* We release the page in page_handle_poison. */ + folio_put(src); +} + +static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force, enum migrate_mode mode) { int rc = -EAGAIN; - bool page_was_mapped = false; + int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); @@ -1107,8 +1149,8 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, goto out_unlock; if (unlikely(!is_lru)) { - rc = move_to_new_folio(dst, src, mode); - goto out_unlock_both; + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; } /* @@ -1133,11 +1175,42 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); try_to_migrate(src, 0); - page_was_mapped = true; + page_was_mapped = 1; } - if (!folio_mapped(src)) - rc = move_to_new_folio(dst, src, mode); + if (!folio_mapped(src)) { + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return MIGRATEPAGE_UNMAP; + } + + if (page_was_mapped) + remove_migration_ptes(src, src, false); + +out_unlock_both: + folio_unlock(dst); +out_unlock: + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); +out: + + return rc; +} + +static int __migrate_folio_move(struct folio *src, struct folio *dst, + enum migrate_mode mode) +{ + int rc; + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + bool is_lru = !__PageMovable(&src->page); + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + + rc = move_to_new_folio(dst, src, mode); + if (unlikely(!is_lru)) + goto out_unlock_both; /* * When successful, push dst to LRU immediately: so that if it @@ -1160,12 +1233,10 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, out_unlock_both: folio_unlock(dst); -out_unlock: /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); folio_unlock(src); -out: /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased @@ -1177,19 +1248,15 @@ static int __unmap_and_move(struct folio *src, struct folio *dst, return rc; } -/* - * Obtain the lock on folio, remove all ptes and migrate the folio - * to the newly allocated folio in dst. - */ -static int unmap_and_move(new_page_t get_new_page, - free_page_t put_new_page, - unsigned long private, struct folio *src, - int force, enum migrate_mode mode, - enum migrate_reason reason, - struct list_head *ret) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { struct folio *dst; - int rc = MIGRATEPAGE_SUCCESS; + int rc = MIGRATEPAGE_UNMAP; struct page *newpage = NULL; if (!thp_migration_supported() && folio_test_transhuge(src)) @@ -1200,20 +1267,49 @@ static int unmap_and_move(new_page_t get_new_page, folio_clear_active(src); folio_clear_unevictable(src); /* free_pages_prepare() will clear PG_isolated. */ - goto out; + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; } newpage = get_new_page(&src->page, private); if (!newpage) return -ENOMEM; dst = page_folio(newpage); + *dstp = dst; dst->private = NULL; - rc = __unmap_and_move(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A folio that has not been unmapped will be restored to + * right list unless we want to retry. + */ + if (rc != -EAGAIN) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + int rc; + + rc = __migrate_folio_move(src, dst, mode); if (rc == MIGRATEPAGE_SUCCESS) set_page_owner_migrate_reason(&dst->page, reason); -out: if (rc != -EAGAIN) { /* * A folio that has been migrated has all references @@ -1229,20 +1325,7 @@ static int unmap_and_move(new_page_t get_new_page, * we want to retry. */ if (rc == MIGRATEPAGE_SUCCESS) { - /* - * Compaction can migrate also non-LRU folios which are - * not accounted to NR_ISOLATED_*. They can be recognized - * as __folio_test_movable - */ - if (likely(!__folio_test_movable(src))) - mod_node_page_state(folio_pgdat(src), NR_ISOLATED_ANON + - folio_is_file_lru(src), -folio_nr_pages(src)); - - if (reason != MR_MEMORY_FAILURE) - /* - * We release the folio in page_handle_poison. - */ - folio_put(src); + migrate_folio_done(src, reason); } else { if (rc != -EAGAIN) list_add_tail(&src->lru, ret); @@ -1534,7 +1617,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2; + struct folio *folio, *folio2, *dst = NULL; int rc, nr_pages; LIST_HEAD(split_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); @@ -1561,9 +1644,13 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, ret_folios); + rc = migrate_folio_unmap(get_new_page, put_new_page, private, + folio, &dst, pass > 2, mode, + reason, ret_folios); + if (rc == MIGRATEPAGE_UNMAP) + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); /* * The rules are: * Success: folio will be freed From patchwork Mon Feb 13 12:34:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138386 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ED40C636CC for ; Mon, 13 Feb 2023 12:35:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2D6E6B007E; Mon, 13 Feb 2023 07:35:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB69E6B0080; Mon, 13 Feb 2023 07:35:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E3386B0081; Mon, 13 Feb 2023 07:35:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 8937B6B007E for ; Mon, 13 Feb 2023 07:35:38 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 4B34C1202AE for ; Mon, 13 Feb 2023 12:35:38 +0000 (UTC) X-FDA: 80462214756.05.96D8A09 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id E58BDC0015 for ; Mon, 13 Feb 2023 12:35:35 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Z8t1xy5h; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291736; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7J2aZsgD8e6HhZAtVKpHpVkvKBz6sCgNze0CWH7rcdY=; b=xRUU1GsGHUC5kSsyLPjBDt21szBWwVkVHSsHy0b1ALPuZpxqFyd5Iv+MomZTTv13h67911 4YFHHrnN2TxIEioED3YOBo4xjQLJAGHNQSPlgTUq/TZfKVi36looS7M/HqHvIHhAO/suCb az59SReMqifRGBH6HdKUZDrleZ6ip/E= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Z8t1xy5h; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291736; a=rsa-sha256; cv=none; b=LZOIB1tXaCfKJX2sALmS2RAiDzX8gQfTx5SZmXS7Kn1jBfPSXiRqz5rWl47BOq50ZM1OW+ ZOQo7sZ7N/ubbSIr10E0bCMMF1WQdBx4RdiIf9kiElLoX9K9pXWHNKcpRKiiaJg3+fEJeo HMDx/7wQuqMAO/Q21sTva4wPrBClJq8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291735; x=1707827735; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=ylCTF0e7vPed9oWJVrpJ/3xILUJhaWMREBCKtLbxdi8=; b=Z8t1xy5hQ5bmARU0mvzSikCYbjq0vkEqz+8IZphpuRhYTh83m/VPhM0/ ImoMx4/vgl9arVZj6eh1ufILoaE4DD/Id/sXK2gtBoNymypS056K6XEua fhGo2c1xAGpu3/jKWa+Y4Cs5M8fW1hi9nHCNp5QA+uWdRIWz8rIDPf+/d ufdivld9XltS9Qkrko5XTNsKB45tXWOWrxCUgwb2/8wwD7FlDknW7btZ0 TY/cUVImov2FzTXR++VWiw0nOGGUqaoQah1Xq/3yzWoTRtu0bLtXkX7Rh QU4C4C0gm9TZjwJKBygdJwj7f7aZ77SRdXY2Mgx0EFaDKEuP0RxvtLduJ w==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513250" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513250" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:35 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366645" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366645" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:31 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Xin Hao , Minchan Kim , Mike Kravetz Subject: [PATCH -v5 5/9] migrate_pages: batch _unmap and _move Date: Mon, 13 Feb 2023 20:34:40 +0800 Message-Id: <20230213123444.155149-6-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E58BDC0015 X-Stat-Signature: nxrcf3w5m4fxkdrfonzmmgysf3sdpk6h X-HE-Tag: 1676291735-811875 X-HE-Meta: U2FsdGVkX1+WQmT7T50/UxAvfe5OwJT51WTNWkKHjAVWIuyc4hzgAv/rzQr2g4EjNZGUgpJlUeg3dlVEVHuKSYLHH5E8aMeTCMfpl7teSGiYUOHxRUFbjVBgCnoaRmdKxRU0Ih16ooxtQGTOHvXvfJmOqO7ofJwIMRpuzHs1YV2Upm4Jnumm2mm25kb9lbNO3UIeHMkfoyFeuiistQrA9lyGCZOKNHNXDzlkKjAHuUDWjvQdXKCsBw6OkgNLtX03IX0O7Cts9s9byHUcLzeR/bgcpluYvEmQG2fEZrz09CGMAezAG1s3N1HS7tvy3hIT9PjdgQxIN3/3+n7tilHJ2A+X0kc6lbUx6J83exJaKidjq0udMPsFILfdi3zkox71yhSd4+U8aFd8DUweBeZqocli0OjikBI8V9G14/FTLkVeyBCYzrEoYkrxdyy/shpgsWjezTgNhysgssGPAK9oBTqKaA/228wB2yElIKSNYvRVhdMyBIBihOSFYuwVaCkcYW4PmjpBnitmScceZMA+7bSOBlItuhyOVbQBCYHI4zbZMCYN767mpdR+QeHmbpx6yICb4xPhMcVGNOUhqCx13v5Zr+45H7pEVqeoFhjRarEupW6t/d1ebOJZAm9NiUo1MrtM7KK83nB1e3AWW7VjPkXNtdi/qsyq8ifsmwx5jps4EzG9Qf3tcQqTzK5G2UgZyeHhNP8r5NFdm9+BfNRGUwikcbZas3AqtrKadYdI0EbZQLxNwsS16vtl40qzZNuiLP9USmc6K+6zS/Qq43R0Zx6TcHDVhYy8v7SWUmUeV5iX223h08LmlMRtFIijqFs6a6kvjMGHFaF/4oXKBw7ao9m2IwgpagoNCbsPy3DPmb2cblT7uAzX4dbuQ8nSyAxM2EYbwv5oiZa+D50/sTrwKtByR+7ro4cMswFRhGgyqdW0fKyPXcDfSggWjQnOQor5PSZs9wu53I6vl4I3ZS8 k24zWiZP ZEFviw5aYv0l86cLrTE/zoWcbIKjkBtou4vtlwW6LhrqpY47/+DOMVhXykO9bdZdDlMsxcxNJCai3GXZVjivE7OEnCoKu4dQlOUQruLfBzyfBBABcKx8vkjkbCCH3Bi14uZ3EOd+LTkiCKyMD1DfrSpt34IxmrgmQB5hsnptO3lRR6pukrRaVir4y/twsOiOd95JFoqkcaltLnRIM5DCV9kNnGbsdmGC0uf/kMNOXDpBCdAYiWqivumoLwgGZE2MaawoNXVLKMmo2x/YGYU1hZH0i0XxsMXn9JGsCfuO6TLOgq87wOf+oAu5hLllxVv+EhTkq+G0FzeL5/R7vUJfqQ5JpYCCqFHrcmJNbNZOxij0GOdDRpMm29ypzi4H4oZclnQ3ubE9GZ8jSdI9c/gW8NMKRJkPTdx2fuhJqGOIgqfwKZkVg1gtTDH+ertI9EjknDg+xFh6ep4iiPx0RTJPfctlDsa4c0upC7AEj6nkXeyxy2+JgQPR4iOZcpdiWi3sD6Y2IqQeDdrBG/D/ExaYJQn3pPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In this patch the _unmap and _move stage of the folio migration is batched. That for, previously, it is, for each folio _unmap() _move() Now, it is, for each folio _unmap() for each folio _move() Based on this, we can batch the TLB flushing and use some hardware accelerator to copy folios between batched _unmap and batched _move stages. Signed-off-by: "Huang, Ying" Tested-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: Xin Hao Cc: Minchan Kim Cc: Mike Kravetz --- mm/migrate.c | 214 +++++++++++++++++++++++++++++++++++++++++++++------ 1 file changed, 189 insertions(+), 25 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 5fd18a7cce62..ee3e21f1061c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1051,6 +1051,33 @@ static void __migrate_folio_extract(struct folio *dst, dst->private = NULL; } +/* Restore the source folio to the original state upon failure */ +static void migrate_folio_undo_src(struct folio *src, + int page_was_mapped, + struct anon_vma *anon_vma, + struct list_head *ret) +{ + if (page_was_mapped) + remove_migration_ptes(src, src, false); + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + list_move_tail(&src->lru, ret); +} + +/* Restore the destination folio to the original state upon failure */ +static void migrate_folio_undo_dst(struct folio *dst, + free_page_t put_new_page, + unsigned long private) +{ + folio_unlock(dst); + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); +} + /* Cleanup src folio upon migration success */ static void migrate_folio_done(struct folio *src, enum migrate_reason reason) @@ -1069,8 +1096,8 @@ static void migrate_folio_done(struct folio *src, folio_put(src); } -static int __migrate_folio_unmap(struct folio *src, struct folio *dst, - int force, enum migrate_mode mode) +static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force, + bool avoid_force_lock, enum migrate_mode mode) { int rc = -EAGAIN; int page_was_mapped = 0; @@ -1097,6 +1124,17 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, if (current->flags & PF_MEMALLOC) goto out; + /* + * We have locked some folios and are going to wait to lock + * this folio. To avoid a potential deadlock, let's bail + * out and not do that. The locked folios will be moved and + * unlocked, then we can wait to lock this folio. + */ + if (avoid_force_lock) { + rc = -EDEADLOCK; + goto out; + } + folio_lock(src); } @@ -1205,10 +1243,20 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); + struct list_head *prev; __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + prev = dst->lru.prev; + list_del(&dst->lru); rc = move_to_new_folio(dst, src, mode); + + if (rc == -EAGAIN) { + list_add(&dst->lru, prev); + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; + } + if (unlikely(!is_lru)) goto out_unlock_both; @@ -1251,7 +1299,7 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, + struct folio **dstp, int force, bool avoid_force_lock, enum migrate_mode mode, enum migrate_reason reason, struct list_head *ret) { @@ -1279,7 +1327,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page *dstp = dst; dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, mode); + rc = __migrate_folio_unmap(src, dst, force, avoid_force_lock, mode); if (rc == MIGRATEPAGE_UNMAP) return rc; @@ -1287,7 +1335,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page * A folio that has not been unmapped will be restored to * right list unless we want to retry. */ - if (rc != -EAGAIN) + if (rc != -EAGAIN && rc != -EDEADLOCK) list_move_tail(&src->lru, ret); if (put_new_page) @@ -1326,9 +1374,8 @@ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, */ if (rc == MIGRATEPAGE_SUCCESS) { migrate_folio_done(src, reason); - } else { - if (rc != -EAGAIN) - list_add_tail(&src->lru, ret); + } else if (rc != -EAGAIN) { + list_add_tail(&src->lru, ret); if (put_new_page) put_new_page(&dst->page, private); @@ -1603,12 +1650,16 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, return nr_failed; } +/* + * migrate_pages_batch() first unmaps folios in the from list as many as + * possible, then move the unmapped folios. + */ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, free_page_t put_new_page, unsigned long private, enum migrate_mode mode, int reason, struct list_head *ret_folios, struct migrate_pages_stats *stats) { - int retry = 1; + int retry; int large_retry = 1; int thp_retry = 1; int nr_failed = 0; @@ -1617,13 +1668,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, int pass = 0; bool is_large = false; bool is_thp = false; - struct folio *folio, *folio2, *dst = NULL; - int rc, nr_pages; + struct folio *folio, *folio2, *dst = NULL, *dst2; + int rc, rc_saved, nr_pages; LIST_HEAD(split_folios); + LIST_HEAD(unmap_folios); + LIST_HEAD(dst_folios); bool nosplit = (reason == MR_NUMA_MISPLACED); bool no_split_folio_counting = false; + bool avoid_force_lock; -split_folio_migration: +retry: + rc_saved = 0; + avoid_force_lock = false; + retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); pass++) { @@ -1645,16 +1702,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, mode, - reason, ret_folios); - if (rc == MIGRATEPAGE_UNMAP) - rc = migrate_folio_move(put_new_page, private, - folio, dst, mode, - reason, ret_folios); + folio, &dst, pass > 2, avoid_force_lock, + mode, reason, ret_folios); /* * The rules are: * Success: folio will be freed + * Unmap: folio will be put on unmap_folios list, + * dst folio put on dst_folios list * -EAGAIN: stay on the from list + * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list * Other errno: put on ret_folios list @@ -1689,7 +1745,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, case -ENOMEM: /* * When memory is low, don't bother to try to migrate - * other folios, just exit. + * other folios, move unmapped folios, then exit. */ if (is_large) { nr_large_failed++; @@ -1728,7 +1784,19 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, /* nr_failed isn't updated for not used */ nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; - goto out; + rc_saved = rc; + if (list_empty(&unmap_folios)) + goto out; + else + goto move; + case -EDEADLOCK: + /* + * The folio cannot be locked for potential deadlock. + * Go move (and unlock) all locked folios. Then we can + * try again. + */ + rc_saved = rc; + goto move; case -EAGAIN: if (is_large) { large_retry++; @@ -1742,6 +1810,15 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_succeeded += nr_pages; stats->nr_thp_succeeded += is_thp; break; + case MIGRATEPAGE_UNMAP: + /* + * We have locked some folios, don't force lock + * to avoid deadlock. + */ + avoid_force_lock = true; + list_move_tail(&folio->lru, &unmap_folios); + list_add_tail(&dst->lru, &dst_folios); + break; default: /* * Permanent failure (-EBUSY, etc.): @@ -1765,12 +1842,95 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, nr_large_failed += large_retry; stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; +move: + retry = 1; + for (pass = 0; + pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); + pass++) { + retry = 0; + large_retry = 0; + thp_retry = 0; + nr_retry_pages = 0; + + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + is_large = folio_test_large(folio); + is_thp = is_large && folio_test_pmd_mappable(folio); + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = migrate_folio_move(put_new_page, private, + folio, dst, mode, + reason, ret_folios); + /* + * The rules are: + * Success: folio will be freed + * -EAGAIN: stay on the unmap_folios list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -EAGAIN: + if (is_large) { + large_retry++; + thp_retry += is_thp; + } else if (!no_split_folio_counting) { + retry++; + } + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + stats->nr_thp_succeeded += is_thp; + break; + default: + if (is_large) { + nr_large_failed++; + stats->nr_thp_failed += is_thp; + } else if (!no_split_folio_counting) { + nr_failed++; + } + + stats->nr_failed_pages += nr_pages; + break; + } + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + } + nr_failed += retry; + nr_large_failed += large_retry; + stats->nr_thp_failed += thp_retry; + stats->nr_failed_pages += nr_retry_pages; + + if (rc_saved) + rc = rc_saved; + else + rc = nr_failed + nr_large_failed; +out: + /* Cleanup remaining folios */ + dst = list_first_entry(&dst_folios, struct folio, lru); + dst2 = list_next_entry(dst, lru); + list_for_each_entry_safe(folio, folio2, &unmap_folios, lru) { + int page_was_mapped = 0; + struct anon_vma *anon_vma = NULL; + + __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); + migrate_folio_undo_src(folio, page_was_mapped, anon_vma, + ret_folios); + list_del(&dst->lru); + migrate_folio_undo_dst(dst, put_new_page, private); + dst = dst2; + dst2 = list_next_entry(dst, lru); + } + /* * Try to migrate split folios of fail-to-migrate large folios, no * nr_failed counting in this round, since all split folios of a * large folio is counted as 1 failure in the first round. */ - if (!list_empty(&split_folios)) { + if (rc >= 0 && !list_empty(&split_folios)) { /* * Move non-migrated folios (after NR_MAX_MIGRATE_PAGES_RETRY * retries) to ret_folios to avoid migrating them again. @@ -1778,12 +1938,16 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, list_splice_init(from, ret_folios); list_splice_init(&split_folios, from); no_split_folio_counting = true; - retry = 1; - goto split_folio_migration; + goto retry; } - rc = nr_failed + nr_large_failed; -out: + /* + * We have unlocked all locked folios, so we can force lock now, let's + * try again. + */ + if (rc == -EDEADLOCK) + goto retry; + return rc; } From patchwork Mon Feb 13 12:34:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138387 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1561C64EC7 for ; Mon, 13 Feb 2023 12:35:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6AF366B0071; Mon, 13 Feb 2023 07:35:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 639096B0080; Mon, 13 Feb 2023 07:35:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B1DB6B0081; Mon, 13 Feb 2023 07:35:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3A7826B0071 for ; Mon, 13 Feb 2023 07:35:42 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0B9B81608A4 for ; Mon, 13 Feb 2023 12:35:41 +0000 (UTC) X-FDA: 80462214924.02.6853765 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id D6639C0015 for ; Mon, 13 Feb 2023 12:35:39 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Z5IL4tn+; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291740; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WjTzw4PSTBXrymmqX83J4G/fs1zgCKHC1Xe1yoEujwM=; b=QnjSHNVIlbAbj3QGLRS1UTp+dBUDJ+dgOhrDa4YCRzj8e64JFKS9hmfJSFYNSwN6H5azul QjV22teVoNmgJHgWdJrapGKEhUQfBHzn9SqCVQ1hsCKcYd2H296FyAbNL8D6p/olgewMdk re4Uqpt21qZlKhuCycwhxengzRSlkNQ= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Z5IL4tn+; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291740; a=rsa-sha256; cv=none; b=cKSOn/isfpIZC+mhbc5wEgbpgnA/WeFBgkUmKhnsR0RmQdRw1NGFSYzpx7JcNAYq61iAY1 UVz344Od8KijEYcHCkRbR8/1Z5xhHy41Xv+huudOggU9w7OtdWQZtF2I1GKB0Iqc37JqgP PFgKxBhPD2SnOShxnIp/MdgVHAAJQsI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291739; x=1707827739; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=K4XDKropeMr7/ju8eX1ZAYnD0sjioz6DP3VREqBtsko=; b=Z5IL4tn+SAjmtk9Cb0hGD6JeBSGVeCuX3kdGF92hkdMoza6R3npUTXiO vVeb47nCxQsa01C4PK8xprXf/wR0pAsmDXVbnD5XUllFhoFVe8up2b1I3 eoQgDARSTFARF2f2XdMVlBzWyRkTM7S2XVXGHBBLjJs05EcweBaVT05Zt xteyeli2vK/1s9slGqlE+i5tgrgj+MpUx1qpWP2M6Tr+PMkdJbIzAOf+n oHlxMv03DHs0dC64lQR6wZXJ7QCkbDZB475ohgAkAp8RgTUMwGNMTlwbc h47bqVIvdxK8jKJzfNMUhvCUX0Ax2TyQMrDSG5fST2I9UsiqD+HEo6Xx1 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513272" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513272" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:39 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366651" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366651" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:35 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Xin Hao , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 6/9] migrate_pages: move migrate_folio_unmap() Date: Mon, 13 Feb 2023 20:34:41 +0800 Message-Id: <20230213123444.155149-7-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D6639C0015 X-Stat-Signature: sdiiymkcrpzoxke8mjybir8is4gairot X-HE-Tag: 1676291739-512490 X-HE-Meta: U2FsdGVkX19UB+Xs/PyuVMpI3phhkoDmhftJ04wHwZECAp4RRnrJDVpZGAvyjHTXz1DM1Fih13Pl5SNMLfjQl7bAgMIk9fTAisSoqWkUbRFzyde7VTrkb3NeTsV7Nsr+9SR0DMuRK7GAWr0Wy8+Vx3qcQ5VeAKvDEE7k4ntE3+QVb464Zk3uBpfqi8gtUpxVhJnEGYnAzF80wfjEHDy0ptyEVxw76i96bJTioibp60UtGapi4mSCm0eUlS0IAA1M8i6OriE4NJXedvTPO2ra0+1Ihs0eJL58lXn9FLGrtSPQkDm4wD+4LXVyevhkHLQHdNVW1UnI3xOIrpLBoViuIdYh4fR+1QDbr24/oZSvs13dVZaosyjyc+Bc8OM5if6AlQG30vs446A8pDyu5PtHCiWTlVlI2sijMEhv7A8XIF7sY7wvJSQRcnu1l8s44dlBCbMRoaKaACWxDVvo9EvfML+6KmIn+zGGwFHE7jult8RMLfgwSTCvDB94EBv1oUouLC74U78v3EJd/xtUWh4c+mq2o3o9jp0cg0cS4va1X/IfyMlFQsrOimkwjdUhdvCgNp9Dv36JzstBi4gbodJej4HbvBPyGFcGIy/uS9GStg7+9SpqF391ZSGe+iRpmkMtPlXJouOryQREdsaLyz8wqg8NpQPm4EWKaYP3IIjtN7U6kYgskTOW+TbYrS++uQuBrOYKXaFY5mu9+Qf/S2AMwjbfuM5iXgfmNDtdOKmcmL0G+u/eBvMLorBkbC4U7dU5/ljskxLWn6ZxTor6KmQdSd4fgGyjbA654Zckv9+hh8abQiQ2U7pD3kREqX0zXqvtINaLNphN7CfV7ihBMPe8+B8T2wG02Mo1n9Zbn/08OQ5OFzKNPOZGlc9pEJnYJuhHzxjeaUuoucacKcPPnXT75cHwmmBJsE1yhCBiD0I/L44r4wJqjzeG2TcKGkXm7LWctOyAJGdEFu1bDS6al5r 8iqJsx4t gDVrOVJxSSD4WzQ50+5V22ilZe8rMMIoacA3czdYjaM762n5Yduy7oMjO9BiWwOCiUlHSk/9APtsdWypdvCc8/R5tpAbhz32I0UM4KfgRgfXy0gvcBujvqrRsnFJ7PdvsZQg+0GHxs73HMotfHIwljasRRG+pLYqhyL6HbnZ6VjWzPH2a/Hg1vC84DoS5PHqVrKhhD7Ryux899i3t+0f4WmgrXdsKf2ofXLU3Lao+zYtckuJbwzJInyIdIaImxF8ga1tr92dz1NoayjQznCqQAgQhXE64pEkoji+ngL0YFa5Lc0J4szmMQHC0k/oDqw5mbIGDOLC4XXCplvkV7KOsig1oaZcOl2C1xmz5YIfAO5dZ9WHIq3XGGuwOMCCexGIgCmgCT6IBzxWy5Sljb7waC+9aPIpJcUaXfRghzfGx86vFLiJNaM9bSIW4NoHeLSVO9cGjWitK4orozFabtrxEbcoXWmTD4IJt0RNiKSFywGg8SqCMBo97bKez+xXXSKJab6NXj6LJqFPncfJIeSUP5MLvLQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just move the position of the functions. There's no any functionality change. This is to make it easier to review the next patch via putting code near its position in the next patch. Signed-off-by: "Huang, Ying" Reviewed-by: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: Xin Hao Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 100 +++++++++++++++++++++++++-------------------------- 1 file changed, 50 insertions(+), 50 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index ee3e21f1061c..0c7488ebe248 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1236,6 +1236,56 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force return rc; } +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool avoid_force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) +{ + struct folio *dst; + int rc = MIGRATEPAGE_UNMAP; + struct page *newpage = NULL; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; + rc = __migrate_folio_unmap(src, dst, force, avoid_force_lock, mode); + if (rc == MIGRATEPAGE_UNMAP) + return rc; + + /* + * A folio that has not been unmapped will be restored to + * right list unless we want to retry. + */ + if (rc != -EAGAIN && rc != -EDEADLOCK) + list_move_tail(&src->lru, ret); + + if (put_new_page) + put_new_page(&dst->page, private); + else + folio_put(dst); + + return rc; +} + static int __migrate_folio_move(struct folio *src, struct folio *dst, enum migrate_mode mode) { @@ -1296,56 +1346,6 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, return rc; } -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool avoid_force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, avoid_force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - - /* - * A folio that has not been unmapped will be restored to - * right list unless we want to retry. - */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); - - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); - - return rc; -} - /* Migrate the folio to the newly allocated folio in dst. */ static int migrate_folio_move(free_page_t put_new_page, unsigned long private, struct folio *src, struct folio *dst, From patchwork Mon Feb 13 12:34:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138388 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14F99C636CC for ; Mon, 13 Feb 2023 12:35:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8AABF6B007B; Mon, 13 Feb 2023 07:35:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 834396B0080; Mon, 13 Feb 2023 07:35:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6ADDE6B0081; Mon, 13 Feb 2023 07:35:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 58D7C6B007B for ; Mon, 13 Feb 2023 07:35:46 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 399A71406D4 for ; Mon, 13 Feb 2023 12:35:46 +0000 (UTC) X-FDA: 80462215092.25.2455E87 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id E8E58C0008 for ; Mon, 13 Feb 2023 12:35:43 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HJyRi8kD; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291744; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XQwhEs+XZRIDnah2mU2EUx0wmK8jRwxbNBjhZ0XNldA=; b=bXLIhBGNPmzqoOyFf/vPgvVVHwjOG71aUBO7w17LOrq1Y3npMCrj1eBrP88bQuOArm48q/ oTzK2b3inu4wdRJld3UY9vRVxJEC2M9QlCQFyKFVAye/8/mGy296gqq1cfL33iF9VQ6RmK 2kohKYPJq3oM8+kAs3yxEprd2NvgwLc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=HJyRi8kD; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291744; a=rsa-sha256; cv=none; b=EpG+efgwdPkH9DgIc1JAPXWsTd3pDjImroRJVww2JmRdDfYikulPhaSVCrPIzmOZebvvT1 5Bpv0g0IJn8xjaSfjq3a2qhXeGj4deZXSi7sd5HAz8PK1LY7I8k3TtP/W5Nq64nWgSx5Zx naTOVVk1CrnMU0JuWLhaHlVg7ZWbTP0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291744; x=1707827744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LJBQcFuwCDNelXry1gr5Vxbs1ZO8LcRLXJ+2vWMUK9M=; b=HJyRi8kDL3FHZSlcMHs9/T73kj8+zXaPbtyxtD5Fzsc9alGO2ulgigSr UP/F7lyKwPa7iyEZbPmBCcbendvJMol5J5CGCai7vqCYy+xE96qcNqvVa FJ937Xv3QWTDZ9YTPXHB3VHtAw2WYUnzqWFR6XG0cdgLAjT6qbHAUO3TL Ug9V7tkNRCiFhVs8z+48HHwwLQQlXqnSWYalUhrt9AIkfY/PM71Yu8q/I 3chzMdNakTcn4yk0Hp32//+qm9ZnW2FArirti9iwCLJYsDKo0TXwPnqO+ 5rKcwT9FN/3OsSZbkA/WnAWOW3iEn+lKWm3tnI45eHCAFQpm3wUtOeCA2 Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513296" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513296" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:43 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366659" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366659" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:39 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Xin Hao , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 7/9] migrate_pages: share more code between _unmap and _move Date: Mon, 13 Feb 2023 20:34:42 +0800 Message-Id: <20230213123444.155149-8-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E8E58C0008 X-Stat-Signature: 78z8rizgxrcfjj4368penm6h1ozm8d88 X-HE-Tag: 1676291743-141978 X-HE-Meta: U2FsdGVkX188N6NqKBg4MVDLNs4WashG+x+UHvBSXL0IocKgzPJimI6YsDn6RGga4z0KPCSkksWU3BVsXgawyKhC4Rx+WwZfQtPcmC2CFPstKGr/Y8slXdnt+Zo1J9Jd1lzTyHM82sm0+FwNlOfLhc6Ck3YOg6CjPHi42eutCsNqJLiQSPrvjCdEVqJj2kAp7m91bqgoDSxU63SRRh8VcH+fF0jgQRgJNGRNq+9lt1neHE88eKfVxGe15FLF0dWXRUG4wiQvdvT2MZzVMJKnsjZEgovYB1/y8ZmQaWt+d9VKiUFtmkPzLUpEt0vKOBXwmOFYta6biTDscqsYKI5xIzCgRYlrWCRGv2zIKW8s7jI3mljiZ6OK/pXil/VrLv5to6XD1K8AO1ihiFxzYVYQspXDsvva2mEZlD6/qNx8iO/EG06aN+Bd8YiRLorXkNl321vJAlxWlVkybr+XbSazMeQKaXKGwifW1v1+hoqvJDhmk+bWxb7R1UNhsQHJ6HNBe1/boJaVr6cMh7HcsLaXvJBtyRdlX+lrejYXE5OTHFYuMIP/Gcubv1Vn6V0IXu6srtKRuHGy7ayljbrcqYuXTAlT8VQ+88qZ4qzW5rRpZtMO0r+WDmTJm0EPIs8X8K7IZwuhPOq+2cJjP0fz+Bp19SjfW+4SEei4xmP1tlsxhIspXmuUWiRib3LIvwK+vwOhJ32GYVaCEvkh/AjNOyzp7cIfoHgIvN7MwSg7MdUMf1+p8zpExV6vCtS8g+3XyMsU38Qdqr1xHX+KHPWM/Dvj1Wap2S99EL8t/kw/E+J42OTda90jiQz39+4MaV8yFmtjN0MmXVX605MlwMcMHUM6TOdZUiUgRxxnNZpVt/MNMpQ2vsWaxtDBGJrALtZ9gjjyU3AidusXkEyiAaxYykTmI0LheeW4U71oyOBAQwuoBelVmzoOv8ad/TdSDnNwfyCuPG+OTarmEvwpNkOXPLY qxoeXWTu 7Y2K9iUphslgrUl1RyR8wcmzAcGoqqTziP0lUomOLTuLBl/1GqWrMr6oP+JGEjsmw9gFDUSWx6PwJBlHAsToXZamR7CTl5hz6QZQ28GdvgleaLqlgaHHF/hlTPTDaF5En48oLyL1UPbCzKo06TaHdqlGGEoLY0iDm/IbX27RhK84L70cCPN4i0go9jWu1XX4uaIqjJ337Iod8UWCYxp6OJ16a3EDgXReT91tCRhYBuXmLU9T54lJ1iSzp1y+L4Q+YqXJ7wtUAHOE1aHP9/pwOyjJsxIw6o0a1tej+gQ+tRDmLxuFWYlY0W3HDWaCYru5kWkhOplwCYUHxVo46ikAIL5b0g36xJqw8u1t647qvITMSd1wNHfVA9m9D80ZkDxO1zBR2uHBeUkV1NrHlznvYzRX/fy3HE5B+3zNNBHHp2aIMKOztgilYlrdwm4YIyBq/Aul+rSBamMbSD2NNcetqZfAJQfD07c7jUuN9yZ2PfKIjxdq198nC/labo0JigfRIwYXy92Qw1os4RGuZ/3klNni0Mw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a code cleanup patch to reduce the duplicated code between the _unmap and _move stages of migrate_pages(). No functionality change is expected. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: Xin Hao Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 207 +++++++++++++++++++++------------------------------ 1 file changed, 85 insertions(+), 122 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 0c7488ebe248..00713ccb6643 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1055,6 +1055,7 @@ static void __migrate_folio_extract(struct folio *dst, static void migrate_folio_undo_src(struct folio *src, int page_was_mapped, struct anon_vma *anon_vma, + bool locked, struct list_head *ret) { if (page_was_mapped) @@ -1062,16 +1063,20 @@ static void migrate_folio_undo_src(struct folio *src, /* Drop an anon_vma reference if we took one */ if (anon_vma) put_anon_vma(anon_vma); - folio_unlock(src); - list_move_tail(&src->lru, ret); + if (locked) + folio_unlock(src); + if (ret) + list_move_tail(&src->lru, ret); } /* Restore the destination folio to the original state upon failure */ static void migrate_folio_undo_dst(struct folio *dst, + bool locked, free_page_t put_new_page, unsigned long private) { - folio_unlock(dst); + if (locked) + folio_unlock(dst); if (put_new_page) put_new_page(&dst->page, private); else @@ -1096,13 +1101,42 @@ static void migrate_folio_done(struct folio *src, folio_put(src); } -static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force, - bool avoid_force_lock, enum migrate_mode mode) +/* Obtain the lock on page, remove all ptes. */ +static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, + unsigned long private, struct folio *src, + struct folio **dstp, int force, bool avoid_force_lock, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { + struct folio *dst; int rc = -EAGAIN; + struct page *newpage = NULL; int page_was_mapped = 0; struct anon_vma *anon_vma = NULL; bool is_lru = !__PageMovable(&src->page); + bool locked = false; + bool dst_locked = false; + + if (!thp_migration_supported() && folio_test_transhuge(src)) + return -ENOSYS; + + if (folio_ref_count(src) == 1) { + /* Folio was freed from under us. So we are done. */ + folio_clear_active(src); + folio_clear_unevictable(src); + /* free_pages_prepare() will clear PG_isolated. */ + list_del(&src->lru); + migrate_folio_done(src, reason); + return MIGRATEPAGE_SUCCESS; + } + + newpage = get_new_page(&src->page, private); + if (!newpage) + return -ENOMEM; + dst = page_folio(newpage); + *dstp = dst; + + dst->private = NULL; if (!folio_trylock(src)) { if (!force || mode == MIGRATE_ASYNC) @@ -1137,6 +1171,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force folio_lock(src); } + locked = true; if (folio_test_writeback(src)) { /* @@ -1151,10 +1186,10 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force break; default: rc = -EBUSY; - goto out_unlock; + goto out; } if (!force) - goto out_unlock; + goto out; folio_wait_writeback(src); } @@ -1184,7 +1219,8 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force * This is much like races on refcount of oldpage: just don't BUG(). */ if (unlikely(!folio_trylock(dst))) - goto out_unlock; + goto out; + dst_locked = true; if (unlikely(!is_lru)) { __migrate_folio_record(dst, page_was_mapped, anon_vma); @@ -1206,7 +1242,7 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force if (!src->mapping) { if (folio_test_private(src)) { try_to_free_buffers(src); - goto out_unlock_both; + goto out; } } else if (folio_mapped(src)) { /* Establish migration ptes */ @@ -1221,73 +1257,25 @@ static int __migrate_folio_unmap(struct folio *src, struct folio *dst, int force return MIGRATEPAGE_UNMAP; } - if (page_was_mapped) - remove_migration_ptes(src, src, false); - -out_unlock_both: - folio_unlock(dst); -out_unlock: - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); out: - - return rc; -} - -/* Obtain the lock on page, remove all ptes. */ -static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, - unsigned long private, struct folio *src, - struct folio **dstp, int force, bool avoid_force_lock, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - struct folio *dst; - int rc = MIGRATEPAGE_UNMAP; - struct page *newpage = NULL; - - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - - if (folio_ref_count(src) == 1) { - /* Folio was freed from under us. So we are done. */ - folio_clear_active(src); - folio_clear_unevictable(src); - /* free_pages_prepare() will clear PG_isolated. */ - list_del(&src->lru); - migrate_folio_done(src, reason); - return MIGRATEPAGE_SUCCESS; - } - - newpage = get_new_page(&src->page, private); - if (!newpage) - return -ENOMEM; - dst = page_folio(newpage); - *dstp = dst; - - dst->private = NULL; - rc = __migrate_folio_unmap(src, dst, force, avoid_force_lock, mode); - if (rc == MIGRATEPAGE_UNMAP) - return rc; - /* * A folio that has not been unmapped will be restored to * right list unless we want to retry. */ - if (rc != -EAGAIN && rc != -EDEADLOCK) - list_move_tail(&src->lru, ret); + if (rc == -EAGAIN || rc == -EDEADLOCK) + ret = NULL; - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + migrate_folio_undo_src(src, page_was_mapped, anon_vma, locked, ret); + migrate_folio_undo_dst(dst, dst_locked, put_new_page, private); return rc; } -static int __migrate_folio_move(struct folio *src, struct folio *dst, - enum migrate_mode mode) +/* Migrate the folio to the newly allocated folio in dst. */ +static int migrate_folio_move(free_page_t put_new_page, unsigned long private, + struct folio *src, struct folio *dst, + enum migrate_mode mode, enum migrate_reason reason, + struct list_head *ret) { int rc; int page_was_mapped = 0; @@ -1300,12 +1288,8 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, list_del(&dst->lru); rc = move_to_new_folio(dst, src, mode); - - if (rc == -EAGAIN) { - list_add(&dst->lru, prev); - __migrate_folio_record(dst, page_was_mapped, anon_vma); - return rc; - } + if (rc) + goto out; if (unlikely(!is_lru)) goto out_unlock_both; @@ -1319,70 +1303,49 @@ static int __migrate_folio_move(struct folio *src, struct folio *dst, * unsuccessful, and other cases when a page has been temporarily * isolated from the unevictable LRU: but this case is the easiest. */ - if (rc == MIGRATEPAGE_SUCCESS) { - folio_add_lru(dst); - if (page_was_mapped) - lru_add_drain(); - } + folio_add_lru(dst); + if (page_was_mapped) + lru_add_drain(); if (page_was_mapped) - remove_migration_ptes(src, - rc == MIGRATEPAGE_SUCCESS ? dst : src, false); + remove_migration_ptes(src, dst, false); out_unlock_both: folio_unlock(dst); - /* Drop an anon_vma reference if we took one */ - if (anon_vma) - put_anon_vma(anon_vma); - folio_unlock(src); + set_page_owner_migrate_reason(&dst->page, reason); /* * If migration is successful, decrease refcount of dst, * which will not free the page because new page owner increased * refcounter. */ - if (rc == MIGRATEPAGE_SUCCESS) - folio_put(dst); - - return rc; -} - -/* Migrate the folio to the newly allocated folio in dst. */ -static int migrate_folio_move(free_page_t put_new_page, unsigned long private, - struct folio *src, struct folio *dst, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) -{ - int rc; - - rc = __migrate_folio_move(src, dst, mode); - if (rc == MIGRATEPAGE_SUCCESS) - set_page_owner_migrate_reason(&dst->page, reason); - - if (rc != -EAGAIN) { - /* - * A folio that has been migrated has all references - * removed and will be freed. A folio that has not been - * migrated will have kept its references and be restored. - */ - list_del(&src->lru); - } + folio_put(dst); /* - * If migration is successful, releases reference grabbed during - * isolation. Otherwise, restore the folio to right list unless - * we want to retry. + * A folio that has been migrated has all references removed + * and will be freed. */ - if (rc == MIGRATEPAGE_SUCCESS) { - migrate_folio_done(src, reason); - } else if (rc != -EAGAIN) { - list_add_tail(&src->lru, ret); + list_del(&src->lru); + /* Drop an anon_vma reference if we took one */ + if (anon_vma) + put_anon_vma(anon_vma); + folio_unlock(src); + migrate_folio_done(src, reason); - if (put_new_page) - put_new_page(&dst->page, private); - else - folio_put(dst); + return rc; +out: + /* + * A folio that has not been migrated will be restored to + * right list unless we want to retry. + */ + if (rc == -EAGAIN) { + list_add(&dst->lru, prev); + __migrate_folio_record(dst, page_was_mapped, anon_vma); + return rc; } + migrate_folio_undo_src(src, page_was_mapped, anon_vma, true, ret); + migrate_folio_undo_dst(dst, true, put_new_page, private); + return rc; } @@ -1918,9 +1881,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, __migrate_folio_extract(dst, &page_was_mapped, &anon_vma); migrate_folio_undo_src(folio, page_was_mapped, anon_vma, - ret_folios); + true, ret_folios); list_del(&dst->lru); - migrate_folio_undo_dst(dst, put_new_page, private); + migrate_folio_undo_dst(dst, true, put_new_page, private); dst = dst2; dst2 = list_next_entry(dst, lru); } From patchwork Mon Feb 13 12:34:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138389 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEDD1C636CC for ; Mon, 13 Feb 2023 12:35:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C0BA6B0080; Mon, 13 Feb 2023 07:35:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 749126B0081; Mon, 13 Feb 2023 07:35:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 59C846B0082; Mon, 13 Feb 2023 07:35:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 41DF66B0080 for ; Mon, 13 Feb 2023 07:35:50 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 15E1E12042B for ; Mon, 13 Feb 2023 12:35:50 +0000 (UTC) X-FDA: 80462215260.22.DD65D4E Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id F035DC0012 for ; Mon, 13 Feb 2023 12:35:47 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XM+rg5ZK; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291748; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=g5jW1/79Tv0OPDamwb6MdcM8qsaeNohjOfSEKq5e9pg=; b=hye3rQzQ9qOMlVy4OoxygsS/5LRDA3XmUTbuh1rcQReeCI2OUZs8BcTKQNyj0qRxv/3bjW DUiGRg/Q2xDnhw3/l5TbxPW3zYa7+D/ApCrnFHfFteASmfWfjdf5qRU7v1/Ufkg0IADgzm iy6ihNCQoohGqHS44N4+Gjpv2v5Bb04= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=XM+rg5ZK; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291748; a=rsa-sha256; cv=none; b=KsWe1Nd06VEjEaN68wJXGXgstkyo+08q5tBUZRU8kCSgRlaYTp7H0z4dzxgHmqKmu/tfXU fKSmjjEMP6/HY1dfljmqR+zeTFWRQdpf3AU9nm7HJr3MNpcQ3Lq9ImNLFI0y9/5WqUZPtB 1R0aSBK6t9wkmI/DnAeDTHIR2QwlzDY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291748; x=1707827748; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VYkt38A9z8Pq4upkiF6Pp5/5nTEJdBdjh8yFl0FvjyA=; b=XM+rg5ZKTHufyu94tHWEcYZp03nAHVZiLewqM9FgrnrnPoRERIBYXSJ9 dIGQkPpsmkDQ+oe/30UCd6bKTDTVUEWPGnx2es5WLy8XYkfReWJfjen6F 8+c3Lhe7lrMc5OaEojuvqEUo9uYvv4xNs+/mpBTG7+0dg4jyL0udXlY1c ehOggZ/AXeUT80db+VDX9GxhRpjTAt2OaGCRiBPqGrmLgv8gAPlKFSzk0 Wl/JdU5bPy2Nvf4n48QkIAZWkokkz0MSbo7NW/pvYgmRh4PygwP4dGBMH KPHXKQ4VmS6wwknTAfuRvmZ3sdfu+0+LWvtmU9zc8zYfy2tgaMj/ci5is g==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513314" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513314" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:47 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366663" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366663" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:43 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Xin Hao , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 8/9] migrate_pages: batch flushing TLB Date: Mon, 13 Feb 2023 20:34:43 +0800 Message-Id: <20230213123444.155149-9-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: F035DC0012 X-Stat-Signature: hmfugsquncobtbuudabgmhs8aahebznz X-HE-Tag: 1676291747-411301 X-HE-Meta: U2FsdGVkX18KT1NrJ7LK7it8Vax18kIdZZsPZXdTmdBsdAlTTwFsiXw3aBYIrqd26a4OT9yij4k3MHv7wzPzorvbEDPtnK1iroZ0eeX8DqrddVlGCePdjkf/kKYbD8Y1ZMOgry5rwCe8udYGJ4Q+lWN5ZicSSoXVYgadPGK6Z37e85cTdTCqFTeu3QfjzaH6sQq0ziVvtyim+swUNkcop92SqNLr4do2e+5qAOy97dF1CQC/lywnGFF6KhTzExZVShuTSsKhTKvvP85JR3zrHnK99IVmfPNeu5nf6eXnZMYieYuXvqgPHadOii5okX7lmr+9mUl+snE7LQUFGVeO79+PW8FfSIrALRlwOlW/wD5UQ7Uo0Fr2cky5M8EDaeWpUrjseqmY3BtG9BiDn2Hard+SScdtl+8BHP6EwBoAav+sroO/szhg+XQWZLhOq6qqozcuIuVnZRNmpnclwisPAxXQMueyFysMxBqyBxOn8bNPOMTq/bYXeI2R+el33kilZ2tDm4yLWxV+LxM0849/ruti0d+yQNDIx0DnA0e+DT9x/xnhEuq3AZjiSyH7KvxY8k84QGMNTJfuPBRWE0FJF5lw6OtgjC/6L05ZvL0Gkzx8yBryPCxUfaT+Qn4SId88jAE6lngYrzehXc+dwLng7UgWohrNots/J91Ofb/KFNSxk161goRdylxmMygwH7JgirrDuNwFLpKiWN/xWj8PasTaXjqO+D8sK/K9bvuFzr9IqSaX8J4CSNOx1VFqqRSsfn54UHSH+k0JEnUbRqUo6C8/zhbPFgvKwSzoBk9eg1vW1vpVCpeRghsSMBEbAYWTjMR9MIj2OkUTwSKAksdumIiM2+kPlRuC2IyQgvZ3qWtMR1yVAGwpr3S/73RuPmcAeTdAnLilUeRSdhNNW3Htf/C8843s4zfgrgyk3UPCG1Qlp4PUkq4YhnLPx0VhD8iescew+Sq0cKmlR6IlsCQ FgOmzQ4p l9dMF/21FitG1/+0SaecRWAktg3CCHAlQUs0vXxZM+NQXPCAlzXnQz8rXL2M173bI7h3smDmnc65PUiSUlCpyZwhu/jLsJl/MZ7Gq9cWjdjkQmuhQLEWoRUkU9zFPKEghmbhnoPjGdBMXB622CoiNDkWMaivr4Fq2qY6kcy4ksAcm44tDnrZEs3OR9NXiC7wJmjY2asxOC7VY2p47BLVl3cR+UrAeisd0AS1r3Wpmkhe4oOh8GqXB92AZX9DaF05epEbR9k0GZXF8dmKxMMjadDRNL8tQDtzWCkeT7nUbjnEGCXduA2YZhK46SHMePmlnJMj65AS+zI/S7Abo7zVT/fdfwvd5M6rEv+hdc2reVbvVNSuti4yVhTC9G2eagCKUHYCGtfg9W1+5yrZHD4ui+Gvr13FCzbz4AwMofy8IYvQAAo3sbMOnYeawi3wINoS2z0EX7WCvFwvm2QgK/hFS1Fyp5ruHtIK5QjP6hH8TgvlkVzwou9PMtlYuE4hvdrmXUlZQP1seZyMUJJ19mMzzn+yZiA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The TLB flushing will cost quite some CPU cycles during the folio migration in some situations. For example, when migrate a folio of a process with multiple active threads that run on multiple CPUs. After batching the _unmap and _move in migrate_pages(), the TLB flushing can be batched easily with the existing TLB flush batching mechanism. This patch implements that. We use the following test case to test the patch. On a 2-socket Intel server, - Run pmbench memory accessing benchmark - Run `migratepages` to migrate pages of pmbench between node 0 and node 1 back and forth. With the patch, the TLB flushing IPI reduces 99.1% during the test and the number of pages migrated successfully per second increases 291.7%. Haoxin helped to test the patchset on an ARM64 server with 128 cores, 2 NUMA nodes. Test results show that the page migration performance increases up to 78%. NOTE: TLB flushing is batched only for normal folios, not for THP folios. Because the overhead of TLB flushing for THP folios is much lower than that for normal folios (about 1/512 on x86 platform). Signed-off-by: "Huang, Ying" Tested-by: Xin Hao Reviewed-by: Zi Yan Reviewed-by: Xin Hao Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 5 ++++- mm/rmap.c | 20 +++++++++++++++++--- 2 files changed, 21 insertions(+), 4 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 00713ccb6643..2fa420e4f68c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1248,7 +1248,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page /* Establish migration ptes */ VM_BUG_ON_FOLIO(folio_test_anon(src) && !folio_test_ksm(src) && !anon_vma, src); - try_to_migrate(src, 0); + try_to_migrate(src, TTU_BATCH_FLUSH); page_was_mapped = 1; } @@ -1806,6 +1806,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, stats->nr_thp_failed += thp_retry; stats->nr_failed_pages += nr_retry_pages; move: + /* Flush TLBs for all unmapped folios */ + try_to_unmap_flush(); + retry = 1; for (pass = 0; pass < NR_MAX_MIGRATE_PAGES_RETRY && (retry || large_retry); diff --git a/mm/rmap.c b/mm/rmap.c index 8287f2cc327d..15ae24585fc4 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -1952,7 +1952,21 @@ static bool try_to_migrate_one(struct folio *folio, struct vm_area_struct *vma, } else { flush_cache_page(vma, address, pte_pfn(*pvmw.pte)); /* Nuke the page table entry. */ - pteval = ptep_clear_flush(vma, address, pvmw.pte); + if (should_defer_flush(mm, flags)) { + /* + * We clear the PTE but do not flush so potentially + * a remote CPU could still be writing to the folio. + * If the entry was previously clean then the + * architecture must guarantee that a clear->dirty + * transition on a cached TLB entry is written through + * and traps if the PTE is unmapped. + */ + pteval = ptep_get_and_clear(mm, address, pvmw.pte); + + set_tlb_ubc_flush_pending(mm, pte_dirty(pteval)); + } else { + pteval = ptep_clear_flush(vma, address, pvmw.pte); + } } /* Set the dirty flag on the folio now the pte is gone. */ @@ -2124,10 +2138,10 @@ void try_to_migrate(struct folio *folio, enum ttu_flags flags) /* * Migration always ignores mlock and only supports TTU_RMAP_LOCKED and - * TTU_SPLIT_HUGE_PMD and TTU_SYNC flags. + * TTU_SPLIT_HUGE_PMD, TTU_SYNC, and TTU_BATCH_FLUSH flags. */ if (WARN_ON_ONCE(flags & ~(TTU_RMAP_LOCKED | TTU_SPLIT_HUGE_PMD | - TTU_SYNC))) + TTU_SYNC | TTU_BATCH_FLUSH))) return; if (folio_is_zone_device(folio) && From patchwork Mon Feb 13 12:34:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13138390 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01BA3C636D4 for ; Mon, 13 Feb 2023 12:35:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96E456B007E; Mon, 13 Feb 2023 07:35:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F8256B0081; Mon, 13 Feb 2023 07:35:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7712B6B0082; Mon, 13 Feb 2023 07:35:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 65EE06B007E for ; Mon, 13 Feb 2023 07:35:54 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3AF27C05A6 for ; Mon, 13 Feb 2023 12:35:54 +0000 (UTC) X-FDA: 80462215428.17.CC05351 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by imf22.hostedemail.com (Postfix) with ESMTP id 09421C0015 for ; Mon, 13 Feb 2023 12:35:51 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="USK/pRUU"; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676291752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Za/An/+pIf91H6aGxbJOURUY4wcorEleUW4nfQS+vus=; b=sRmLE9y6JFQ/VuZZ24B82dhzKdrgw/zQDzv2uVOC67+NFo5YwUKVZ3UJfLbyinPUgLj38S 2pX/dxdBGm1lTNlwt43pmcShqmi1dxEndsOVMLmf4s+YZqOP5Atr6AtLvoeOV8rL/Is/Rq olCx6+F/uZTZaoGuJ1CvD/RoYnsTSLU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b="USK/pRUU"; spf=pass (imf22.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.136 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676291752; a=rsa-sha256; cv=none; b=cnwhxpcXa41aWHhYgCLiByCPVj2oceZVuP7O/JKoXqBcc4TTjne3F2oa4/T3atK4GEhNK/ 84pBcyP/MaUlfDerMXDGdSdwFFODzyOEYqAl0LMf71VGBC/wdzGh4QO2DPEN75WZL9R3Wy /ill21/idLPWXNLCXF3Igpdhpx8JIQk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676291752; x=1707827752; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dO4m0t0CfUL8xaVC72bSSWz5Mdmx3h1ISiajGHuEsMg=; b=USK/pRUUN969BQ8clF7lXKUc1xRrKMZXDiFEGDc/QTwbXk1GXGa0+fgC PzsZHlFJTohGWcff75T3XtYDcEGqENaTqA9zv6pxje83EVlrrGDKUUNYU rmzqM148LqqIfiJYHSvAqLP0FvBt28tSUPGjDKcVaFPlyEPvRF4TncLJy atOR/msskl5YgloiqOmz/g024Jj6AoNKRdQH83bsHiEAEukE1g399DRRS hDpdKV5rGVW/CsWi9sk08hzw3XlAk90V93LyjnFshxsswimwvc+nxCWoo MKCdnTEgD8ylC4nj8vQ0ZTjjoDf1YKwV9NB1Yky/k1iyoW8LDMe9y686+ g==; X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="310513333" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="310513333" Received: from orsmga006.jf.intel.com ([10.7.209.51]) by fmsmga106.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:51 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10619"; a="646366670" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="646366670" Received: from changxin-mobl2.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.28.171]) by orsmga006-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 04:35:47 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Alistair Popple , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Xin Hao , Minchan Kim , Mike Kravetz , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH -v5 9/9] migrate_pages: move THP/hugetlb migration support check to simplify code Date: Mon, 13 Feb 2023 20:34:44 +0800 Message-Id: <20230213123444.155149-10-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20230213123444.155149-1-ying.huang@intel.com> References: <20230213123444.155149-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 09421C0015 X-Stat-Signature: 14eseddpynky5inkqw9jp9fihzo9w193 X-HE-Tag: 1676291751-353171 X-HE-Meta: U2FsdGVkX18mclHfDWp9K2xQLUZT2Dcu5rbkH+RYpuwdB4DczoeoKAbiMJhy1kbDTDpTUIUTjsD5KtBxkXKjIKEOQCjsgLEIxCkpCJ98Ew2ANk3m3eOCWtJ+tl9QhXDOefZAB4V0fJ6ObB6ym2eVy37Az4mfRb1Oi6GdEk8O+g3+5E1Go82IxWkUl92DrfYv3S40bHUVn4+mUXmpO7kX03w6CXLiWiPZUM0kuXUQPPxtPB907rsJ+bEK86w/JDMvOv3N6v8cJtTK6G4QoShhjY2hm+SJBvGX5630aUV/ypU5WN7AAhwlZ/yIy21lGCeyvLXHC6v/yF0l/babYqMTQO2t+XzVONuKbpA8TAc6i8HM9EWxHa5g2sUnffwm+OhFFat6KU+qzBN9lML+aijmeXRQN9i+wXtAiOsB7K/hSqL4UR+j5NK3oS6PwkM39KYXYaVzbE4vBOfgOdo/EhLEaAk7DNfu9H5uv1w/yjjnGcnI5OcxMwrIwNS247eIUzXB1bDDFaOJAjy8VItysorcRYYqAlQ8bZZUnurr72te3pE7K7qLPZGUf3tLuVK5q8eQe8yIurEtxPc3zGJgFL0I8dnLygGhrcdjG8U2tXVDR8YjYjgZnFpCiIEfJ4cQnt9K4DUR2E9QoOIMFfFz2gazlya5Ohl0UfBAC5nYemr3Ur5DQzclUQuRuiFWGuuLtptqvWwB8VGhO1Cs5NOugmoHvYmtT1qRGo5q0j8BfZU1itoCPkgWzF9ZbJbQH5lQvebK4+CgAy6Jv2c6u5oxViEq4xZ/8bZJv82Mcm3aOHDnUBWpBTPjiVtslb553wd8BZjYvbGUtLKPawW2Qntfb/YJ8ROODkAo9zXEJwKdfbCem7Z5DgezOqtmwN6izGUWITX4XA1NguyaUh8H9sNGf0S9HHvUMRmKXulsl3V0B3/J6wpcsRfI8YYOMqibWhpCFYxC8EoGNnM65W7uXNqkvTD l6mlLEYA XG8n7zpghfUhPT1d6EmYf6gc0a17H/+Do1JGfaDE46aW62X82odWA1S+qtbVp9sIlXpOzXOUomHlLAe19gMZQvDRqF8/nyaA4ijEqle6rCU3s8TwOXfhweYKWmrUNMFd3FhXHDDRPvM316Ihof6npeJubND397rLAzfgjPC3KgHNC2rtH4utRTjbskIMVIRLSrg16yt04XP3cyIl3FVsmKTPx8mTqSwtL6O/s3NKWtzalNfvSaLiBp8bitaC/QGLTtq6gw4QurngqnVCmAFtpGj3/F2IdfgsBOo8K/aElJEvFxWYQAUiNU31m1Qri2pn4Y3TrxzWJ/O3GMD3FsMZ3kP8+j/p0BfZbDH1yhRYey7FnQqxgpg82It5diKIhcu448lQ4VczHU8ESm4b9bM/ZrWvDMYfwpd9bYt7HU4y9LNVcnNRr6IVS3Rs+wvCDNYoWS3+CpOjfdd0oT8dWQqln9sPoSeD6JnGrjI7un0F7GrUGeMiRcB+QL8Pf8ZiRmOaulTLuWuFh4G8mIqQycCcJbjrroQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a code cleanup patch, no functionality change is expected. After the change, the line number reduces especially in the long migrate_pages_batch(). Signed-off-by: "Huang, Ying" Suggested-by: Alistair Popple Reviewed-by: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Xin Hao Cc: Minchan Kim Cc: Mike Kravetz Cc: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/migrate.c | 83 +++++++++++++++++++++++----------------------------- 1 file changed, 36 insertions(+), 47 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 2fa420e4f68c..ef68a1aff35c 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1117,9 +1117,6 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page bool locked = false; bool dst_locked = false; - if (!thp_migration_supported() && folio_test_transhuge(src)) - return -ENOSYS; - if (folio_ref_count(src) == 1) { /* Folio was freed from under us. So we are done. */ folio_clear_active(src); @@ -1380,16 +1377,6 @@ static int unmap_and_move_huge_page(new_page_t get_new_page, struct anon_vma *anon_vma = NULL; struct address_space *mapping = NULL; - /* - * Migratability of hugepages depends on architectures and their size. - * This check is necessary because some callers of hugepage migration - * like soft offline and memory hotremove don't walk through page - * tables or check whether the hugepage is pmd-based or not before - * kicking migration. - */ - if (!hugepage_migration_supported(page_hstate(hpage))) - return -ENOSYS; - if (folio_ref_count(src) == 1) { /* page was freed from under us. So we are done. */ folio_putback_active_hugetlb(src); @@ -1556,6 +1543,20 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, cond_resched(); + /* + * Migratability of hugepages depends on architectures and + * their size. This check is necessary because some callers + * of hugepage migration like soft offline and memory + * hotremove don't walk through page tables or check whether + * the hugepage is pmd-based or not before kicking migration. + */ + if (!hugepage_migration_supported(folio_hstate(folio))) { + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + continue; + } + rc = unmap_and_move_huge_page(get_new_page, put_new_page, private, &folio->page, pass > 2, mode, @@ -1565,16 +1566,9 @@ static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, * Success: hugetlb folio will be put back * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list - * -ENOSYS: stay on the from list * Other errno: put on ret_folios list */ switch(rc) { - case -ENOSYS: - /* Hugetlb migration is unsupported */ - nr_failed++; - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - break; case -ENOMEM: /* * When memory is low, don't bother to try to migrate @@ -1664,6 +1658,28 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, cond_resched(); + /* + * Large folio migration might be unsupported or + * the allocation might be failed so we should retry + * on the same folio with the large folio split + * to normal folios. + * + * Split folios are put in split_folios, and + * we will migrate them after the rest of the + * list is processed. + */ + if (!thp_migration_supported() && is_thp) { + nr_large_failed++; + stats->nr_thp_failed++; + if (!try_split_folio(folio, &split_folios)) { + stats->nr_thp_split++; + continue; + } + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + continue; + } + rc = migrate_folio_unmap(get_new_page, put_new_page, private, folio, &dst, pass > 2, avoid_force_lock, mode, reason, ret_folios); @@ -1675,36 +1691,9 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, * -EAGAIN: stay on the from list * -EDEADLOCK: stay on the from list * -ENOMEM: stay on the from list - * -ENOSYS: stay on the from list * Other errno: put on ret_folios list */ switch(rc) { - /* - * Large folio migration might be unsupported or - * the allocation could've failed so we should retry - * on the same folio with the large folio split - * to normal folios. - * - * Split folios are put in split_folios, and - * we will migrate them after the rest of the - * list is processed. - */ - case -ENOSYS: - /* Large folio migration is unsupported */ - if (is_large) { - nr_large_failed++; - stats->nr_thp_failed += is_thp; - if (!try_split_folio(folio, &split_folios)) { - stats->nr_thp_split += is_thp; - break; - } - } else if (!no_split_folio_counting) { - nr_failed++; - } - - stats->nr_failed_pages += nr_pages; - list_move_tail(&folio->lru, ret_folios); - break; case -ENOMEM: /* * When memory is low, don't bother to try to migrate