From patchwork Tue Dec 27 00:28:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13081972 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16264C3DA79 for ; Tue, 27 Dec 2022 00:29:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52B13940008; Mon, 26 Dec 2022 19:29:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 50A39940007; Mon, 26 Dec 2022 19:29:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EFDD940008; Mon, 26 Dec 2022 19:29:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2D347940007 for ; Mon, 26 Dec 2022 19:29:34 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E6F191201F0 for ; Tue, 27 Dec 2022 00:29:33 +0000 (UTC) X-FDA: 80286202626.22.60DCDBF Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by imf09.hostedemail.com (Postfix) with ESMTP id 1534814000A for ; Tue, 27 Dec 2022 00:29:30 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gxbO6+bv; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1672100971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EmUgQlD9tnG4P9yimIg6vRqU1RZS8A7fonhRfLBzs4c=; b=lKHdxsQmtj093IJlC5fWBiYa8BJws7SDJexnk8st4/+mqZ9U2LIu2j2JaUXwcl6HkVYrRt TbK+Sicpj0Z+4Cli2w9rSDhBoBnHeefm6eF0kuaKq0+KqIewDVlaU4yqpeOWU5ICAEo8pv 2v9x7ekWIeeqtNmfe00vcq0Fc5/6lHk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=gxbO6+bv; spf=pass (imf09.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.65 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1672100971; a=rsa-sha256; cv=none; b=H9XZccMH2lIG5Ui/ZxP0y8Oo3CE5pDv+5TNlN+u893Nfff7AsfE2clgzI4aBSkyFT21pCs KzZe9dYI2asaE7BOfkAl8GQ1nKRR8Fm2d9rPit7Ltj0evMF+q4LkNG4+zDsOphTTHbhGuj aUWkJqTuiUqLolfGKtLtk6cWt6sBWCQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1672100971; x=1703636971; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=fdSCkAQLw2skvmVvw4V6UUPknrIjo/5xEY/WxCdRv2U=; b=gxbO6+bvSuyUPfTcg0pZ9XtPVYqaJli/zJbqxttBTG2kwY5q+3Lezob0 /uTVs9QIriRzsZKL2/jEyriRWlQ/gyg4xu5DK0x/QnleqpjXpC+4ALwjJ kZH8BWjSN8ajs+wogEq8z3yCAeOCFGdNcxckglLVD4kkjA1x0WeHQwNBN 6BDQmHJZpK2COh7i2MLfezCc1XNw2sgV+Oa78sSn0fC3+YBTqCGLOzhVM KdF6VdfJ4jGmAGAH/2B2qYCMzPvdgtJ4dyQGKr1Jx1yrkcIL3Fq+Tx6/G jSW2/L1GQAN3Z0W5HVE0BafdpxzfvnQH4v84n1HuEKPWWuQeK2+2xZkIc g==; X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="322597216" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="322597216" Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga103.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:29 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10572"; a="760172194" X-IronPort-AV: E=Sophos;i="5.96,277,1665471600"; d="scan'208";a="760172194" Received: from yyang3-mobl1.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.254.212.104]) by fmsmga002-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 26 Dec 2022 16:29:25 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Zi Yan , Yang Shi , Baolin Wang , Oscar Salvador , Matthew Wilcox , Bharata B Rao , Alistair Popple , haoxin Subject: [PATCH 2/8] migrate_pages: separate hugetlb folios migration Date: Tue, 27 Dec 2022 08:28:53 +0800 Message-Id: <20221227002859.27740-3-ying.huang@intel.com> X-Mailer: git-send-email 2.35.1 In-Reply-To: <20221227002859.27740-1-ying.huang@intel.com> References: <20221227002859.27740-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1534814000A X-Stat-Signature: uudbkw4bhsngsrcixryuh814y7pbc548 X-HE-Tag: 1672100970-997455 X-HE-Meta: U2FsdGVkX1+SXy5aZ6oVduwykqWeet/nC4Jbnd98JayyTpyiMymJt/JD0cIkAroJHw+ulkwwi10wU0d5URPlvhGvoq0/3tEMaU1oAKe59K4dNP4Hh8lj47WiptEJvk4lF/V2hiveqR02rTEIV38G7YMjAN+b8Bc3z621ImI4NqpBoHSiuMNkGg/NLxWPCWN4l3IW4EB4rLf49F5oA9vKn67m82uO6M/p3Vj2WfIatYXV2Qy+R4v2lI9TMEz23soQ7927P13mfD+i4rqcNwRE0V4YsuFI1CvCIJzyfbX29+hnyeHnBHbRUQ4l7pfy/wAX2er0d+zqdPaKrB6rsZfvd9ULA07x/u4+frPZa6xl2DOxZjBkWfp/bXWrtDSfVt4szup0UYiv2yBu1lkLZk1xT+m45J90Dn/OkuGeClaJ41HnV+l/qfWhOqgfZsdVr4jV8GqwUrtYZ0e7PDCLaBdU5Z0pX4sjebDYP0DRyCurhkcLPaMQ+C8VP3Uy81mtGI+Zlzh9ENC7NttRGsSLRw0GLMIeUNaHQWCxy7jRqoXfs6/CAreXUUhI8YzFFa+qZkouAeYBfTxid949jM9jfHcyLnklu2sEIbEssxZz1XRa5NnL6LOpdbNZSjWBZO53fyOBMBdxYFsNUi+nQLjSvMHlTl1QBqtAeA1AarKuEuZykc88j9azVhFzWJnnP0BmeyjARTehdnAemy+vHzANoPabhPC9j+wIaAerodS3xLo7TcmXfeVsh/aPhkmhroOXyPDZ61y2vqzI2zsQ9Sm/Rp4IQzkAHcK5WDw+8zs9bQQYCldFVvCkIFYF49PaVvMQ1XCKY7xOt68JUq4OsAyf5S7H8FfY4bPw+yE9aeTdUyOZn9Ltlqt3/5g4fjYXUQbj5e2Ib/qUolwhNkQ+dIo+ACpaGF806WlcI7d1g3zg6WMqX64i6EoEEoLPpjHgqjjfxRRR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a preparation patch to batch the folio unmapping and moving for the non-hugetlb folios. Based on that we can batch the TLB shootdown during the folio migration and make it possible to use some hardware accelerator for the folio copying. In this patch the hugetlb folios and non-hugetlb folios migration is separated in migrate_pages() to make it easy to change the non-hugetlb folios migration implementation. Signed-off-by: "Huang, Ying" Cc: Zi Yan Cc: Yang Shi Cc: Baolin Wang Cc: Oscar Salvador Cc: Matthew Wilcox Cc: Bharata B Rao Cc: Alistair Popple Cc: haoxin --- mm/migrate.c | 114 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 99 insertions(+), 15 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index ec9263a33d38..bdbe73fe2eb7 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1404,6 +1404,87 @@ struct migrate_pages_stats { int nr_thp_split; }; +static int migrate_hugetlbs(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, + struct migrate_pages_stats *stats, + struct list_head *ret_folios) +{ + int retry = 1; + int nr_failed = 0; + int nr_retry_pages = 0; + int pass = 0; + struct folio *folio, *folio2; + int rc = 0, nr_pages; + + for (pass = 0; pass < 10 && retry; pass++) { + retry = 0; + nr_retry_pages = 0; + + list_for_each_entry_safe(folio, folio2, from, lru) { + if (!folio_test_hugetlb(folio)) + continue; + + nr_pages = folio_nr_pages(folio); + + cond_resched(); + + rc = unmap_and_move_huge_page(get_new_page, + put_new_page, private, + &folio->page, pass > 2, mode, + reason, ret_folios); + /* + * The rules are: + * Success: hugetlb folio will be put back + * -EAGAIN: stay on the from list + * -ENOMEM: stay on the from list + * -ENOSYS: stay on the from list + * Other errno: put on ret_folios list + */ + switch(rc) { + case -ENOSYS: + /* Hugetlb migration is unsupported */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + list_move_tail(&folio->lru, ret_folios); + break; + case -ENOMEM: + /* + * When memory is low, don't bother to try to migrate + * other folios, just exit. + */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + goto out; + case -EAGAIN: + retry++; + nr_retry_pages += nr_pages; + break; + case MIGRATEPAGE_SUCCESS: + stats->nr_succeeded += nr_pages; + break; + default: + /* + * Permanent failure (-EBUSY, etc.): + * unlike -EAGAIN case, the failed folio is + * removed from migration folio list and not + * retried in the next outer loop. + */ + nr_failed++; + stats->nr_failed_pages += nr_pages; + break; + } + } + } +out: + nr_failed += retry; + stats->nr_failed_pages += nr_retry_pages; + if (rc != -ENOMEM) + rc = nr_failed; + + return rc; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1437,7 +1518,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, int retry = 1; int large_retry = 1; int thp_retry = 1; - int nr_failed = 0; + int nr_failed; int nr_retry_pages = 0; int nr_large_failed = 0; int pass = 0; @@ -1454,6 +1535,12 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, trace_mm_migrate_pages_start(mode, reason); memset(&stats, 0, sizeof(stats)); + rc = migrate_hugetlbs(from, get_new_page, put_new_page, private, mode, reason, + &stats, &ret_folios); + if (rc < 0) + goto out; + nr_failed = rc; + split_folio_migration: for (pass = 0; pass < 10 && (retry || large_retry); pass++) { retry = 0; @@ -1462,30 +1549,28 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, nr_retry_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { + if (folio_test_hugetlb(folio)) { + list_move_tail(&folio->lru, &ret_folios); + continue; + } + /* * Large folio statistics is based on the source large * folio. Capture required information that might get * lost during migration. */ - is_large = folio_test_large(folio) && !folio_test_hugetlb(folio); + is_large = folio_test_large(folio); is_thp = is_large && folio_test_pmd_mappable(folio); nr_pages = folio_nr_pages(folio); + cond_resched(); - if (folio_test_hugetlb(folio)) - rc = unmap_and_move_huge_page(get_new_page, - put_new_page, private, - &folio->page, pass > 2, mode, - reason, - &ret_folios); - else - rc = unmap_and_move(get_new_page, put_new_page, - private, folio, pass > 2, mode, - reason, &ret_folios); + rc = unmap_and_move(get_new_page, put_new_page, + private, folio, pass > 2, mode, + reason, &ret_folios); /* * The rules are: - * Success: non hugetlb folio will be freed, hugetlb - * folio will be put back + * Success: folio will be freed * -EAGAIN: stay on the from list * -ENOMEM: stay on the from list * -ENOSYS: stay on the from list @@ -1512,7 +1597,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, stats.nr_thp_split += is_thp; break; } - /* Hugetlb migration is unsupported */ } else if (!no_split_folio_counting) { nr_failed++; }