From patchwork Fri Mar 3 03:01:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Huang, Ying" X-Patchwork-Id: 13158304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18191C6FA8E for ; Fri, 3 Mar 2023 03:02:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1C4C6B007E; Thu, 2 Mar 2023 22:02:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 99A306B007D; Thu, 2 Mar 2023 22:02:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 861986B007E; Thu, 2 Mar 2023 22:02:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 78F1F6B007B for ; Thu, 2 Mar 2023 22:02:36 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3586F4103F for ; Fri, 3 Mar 2023 03:02:36 +0000 (UTC) X-FDA: 80526089112.17.EC7D39F Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf19.hostedemail.com (Postfix) with ESMTP id 0AD9C1A000C for ; Fri, 3 Mar 2023 03:02:33 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GqAGxdf3; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677812554; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+fy/EEWe3O7Z5hNnH3nFYSMYo/2ZVU+27kdL2LJsKso=; b=z5O/n5MpeWrlczH8CvArZZ2147IQb6UxtkjICz4OvDBQeG3hi7eWfVKJNdE2qzChNNWWDj 1uQd00su5Grd9YOZDh90ltNGUj5uoXg/y7ZDJa0Am+hfGanmF4ScTdU2jpmEbEM9ZThV7a OofVN3g+aokaKrsvJM5IJWPBERR+KX4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=GqAGxdf3; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of ying.huang@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=ying.huang@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677812554; a=rsa-sha256; cv=none; b=ta3oGdlA6IXAZzt83g2K2XveDQbm84T2djCcQsKyf0rl/yPD9w+4zYUfXU43rIh2fDyBbw 1dfjG65/ebfz/JVC3YzLR8624mnm8xJmAJSZ6Pq4GYTiIEtbMl5HiOHIK1FujWWfoClq9V kQe9G+vMIQKuRO5WsqIjOXUko/dUTSk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1677812554; x=1709348554; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=6dOm2+hfvH1d/Mz6q4OoqdRr+NjQWevzknZQQER47XU=; b=GqAGxdf37NKOuwbcvcnn4iHt0gaD9Zdj91/bD79P4jtY2SNxI1iKBumi pFSMtNR4II7vOSgf8CO0/WfvhwNZYVe/BpBxGiD8fuyKcnjGbK3Cl41y0 ZyXB9ciqOz1hndbRrvxJi8bm5+isa+ZFItMTr5hLpkiBBKQ49GEQamNam psIWb+no0zdKs4Y3hDDUp79AoNiULfgBBiplo6VXuOgl/U14TdS9vHV7h 9/k/99IG2qdo07qT/zzegajD27yv8UA1jMrRAhz8qXVHR/jGJrpqWkuxt DwTyJKl2HGy1FH0kMwUTKBjxQqbFlhayv11Lb3Z5XNWZaKzPKs0zJvmzc Q==; X-IronPort-AV: E=McAfee;i="6500,9779,10637"; a="421207146" X-IronPort-AV: E=Sophos;i="5.98,229,1673942400"; d="scan'208";a="421207146" Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2023 19:02:33 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6500,9779,10637"; a="668497706" X-IronPort-AV: E=Sophos;i="5.98,229,1673942400"; d="scan'208";a="668497706" Received: from xinyumao-mobl.ccr.corp.intel.com (HELO yhuang6-mobl2.ccr.corp.intel.com) ([10.255.31.231]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Mar 2023 19:02:29 -0800 From: Huang Ying To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Huang Ying , Hugh Dickins , Baolin Wang , "Xu, Pengfei" , Christoph Hellwig , Stefan Roesch , Tejun Heo , Xin Hao , Zi Yan , Yang Shi , Matthew Wilcox , Mike Kravetz Subject: [PATCH -V2 3/3] migrate_pages: try migrate in batch asynchronously firstly Date: Fri, 3 Mar 2023 11:01:55 +0800 Message-Id: <20230303030155.160983-4-ying.huang@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230303030155.160983-1-ying.huang@intel.com> References: <20230303030155.160983-1-ying.huang@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0AD9C1A000C X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 1hzxpyefhczoq85z87a8bcerpjtukqf4 X-HE-Tag: 1677812553-628456 X-HE-Meta: U2FsdGVkX18GKr3Zf/hCq3eNUDzViCaJP2C3YD5fR8VPe2HsVrqroy/LE4zruC8Wc847a0iV7cJlv3clqRkKLMU6G2lXD7qJ9V9X3HfD9qvutIvnkKdXbv5EKt7XlPeNv+IpyLI8sYra4bJ8wgmCgX3O8/hP660VEZz4D4ncVkmcFAct5DY99ajdfRfRDxmizduaxEZKcjurtBYIA3/B5A/OJfWs02qxuKptCZe8FIVwSiZFJgA0YZwgqmF4yyNJtFnl7D3mmNuvMEEAS4wrBr58U1RJMi39QBNBXGpcwhWe3aXMUTCbO2++AAmB6pLUIlX6aPAEhPOEd/Bo7V6gtdIKMc7c2627s1F2B1KoLtCjnPCKqVooeRYSWA0o37/ycvxAnM9y5CXgqLkSB/CEpP/kGO9P62mj4/Z6bqJqvXuw2bSfbpLlOXisXnlAWnwT0GFAWJphUWzkr9xhxZ4wWywPg/uX4SUs67jABkv0Sso03Q8Fgmm5jCQ3UL6HZM1aFZ3JBN/FCGIcK60RxuMIorbxbZge1uFBmXKTXSPd/lunTIVYdkvSgkwtwI8f8nNZRHWYlwWGfHhw/lB/AdQ+dtgLXAlCx1FduXgMd2CpNTqzlGqT/zjlMmZhNYnDHvSAk6/53nVWcPROw7KEbdBUnZjth63/z9HbwqE730Wf5oETHXxF2U7ehGhO2tuCGFtySWkvahLiLtGCA1+x15hgrKTNry009CIpJwQy9vzvrNycYOQbetmZIYZlAjJYU0ZMrMdos5oWb4UJX0ge1TVExvDGM/LtwZ+U4qnKt+QDKInXd98fBNMVoJ86Sg8xjyCom/R1FKxujTwrPbdip8Nwi2ngfaASC5Zr0AxfL39JS+c/jFsMg6q3cRORm+VZWftK0hYxqVEBpgfsLnjc+oGklHguPO+FGE7JhGNDxj/RuJuhCNcBnuRzzsVjuPaQlXSxF4L4P4nSxhAF264NzaP l0wxdd4c TXzCQXTw5d3nn7Y9kxGfP/1dtAMh/zQ2ndYca9Qzg32BpyeWH5IEy3hxruTtgV2VatIpR9pNQN5RgACWpnP/AjRcamqouB5rG0eLaHN0VsFoyX4/y6gH6vwR9ZwIx9u35e4fo1mbjU4gMy6Oq7VZ/H7/jQNdhE2Q4CJaCbFCovaQoxfYSMtyIMXHiD7DPsawFiJ9ZSnFQ7GdO4Zef0/ZJS4JSmQlKxP5x53AaRRiH/UZ/cIeDIMiCKobEyBhfxgn8VB9UO2itHrr3sfnY+V5b4f2/g3lGGukrA7KdXBOJ0ZOi5GJ5GXtrxKECUZ8zevzE438XVBDqzZaTZ0FtOwFUOdXLSIVPUbMS/Ivpw9EvPVk1zdCl7sLmtcbC9DShUOSovAJr0NuM/Pb1NtzjFW5BJ5+NsS4BHKvOt/E3cPo3LymaM8EKd9k35VO6DEhZLKcys7bqFWChSWw348c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When we have locked more than one folios, we cannot wait the lock or bit (e.g., page lock, buffer head lock, writeback bit) synchronously. Otherwise deadlock may be triggered. This make it hard to batch the synchronous migration directly. This patch re-enables batching synchronous migration via trying to migrate in batch asynchronously firstly. And any folios that are failed to be migrated asynchronously will be migrated synchronously one by one. Test shows that this can restore the TLB flushing batching performance for synchronous migration effectively. Fixes: 5dfab109d519 ("migrate_pages: batch _unmap and _move") Signed-off-by: "Huang, Ying" Tested-by: Hugh Dickins Reviewed-by: Baolin Wang Cc: "Xu, Pengfei" Cc: Christoph Hellwig Cc: Stefan Roesch Cc: Tejun Heo Cc: Xin Hao Cc: Zi Yan Cc: Yang Shi Cc: Matthew Wilcox Cc: Mike Kravetz --- mm/migrate.c | 80 ++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 62 insertions(+), 18 deletions(-) diff --git a/mm/migrate.c b/mm/migrate.c index 61009e2ae2be..ae86345e12a5 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -1115,9 +1115,8 @@ static void migrate_folio_done(struct folio *src, /* Obtain the lock on page, remove all ptes. */ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page, unsigned long private, struct folio *src, - struct folio **dstp, int force, - enum migrate_mode mode, enum migrate_reason reason, - struct list_head *ret) + struct folio **dstp, enum migrate_mode mode, + enum migrate_reason reason, struct list_head *ret) { struct folio *dst; int rc = -EAGAIN; @@ -1147,7 +1146,7 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page dst->private = NULL; if (!folio_trylock(src)) { - if (!force || mode == MIGRATE_ASYNC) + if (mode == MIGRATE_ASYNC) goto out; /* @@ -1185,8 +1184,6 @@ static int migrate_folio_unmap(new_page_t get_new_page, free_page_t put_new_page rc = -EBUSY; goto out; } - if (!force) - goto out; folio_wait_writeback(src); } @@ -1500,6 +1497,9 @@ static inline int try_split_folio(struct folio *folio, struct list_head *split_f #define NR_MAX_BATCHED_MIGRATION 512 #endif #define NR_MAX_MIGRATE_PAGES_RETRY 10 +#define NR_MAX_MIGRATE_ASYNC_RETRY 3 +#define NR_MAX_MIGRATE_SYNC_RETRY \ + (NR_MAX_MIGRATE_PAGES_RETRY - NR_MAX_MIGRATE_ASYNC_RETRY) struct migrate_pages_stats { int nr_succeeded; /* Normal and large folios migrated successfully, in @@ -1681,8 +1681,7 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, } rc = migrate_folio_unmap(get_new_page, put_new_page, private, - folio, &dst, pass > 2, mode, - reason, ret_folios); + folio, &dst, mode, reason, ret_folios); /* * The rules are: * Success: folio will be freed @@ -1860,6 +1859,51 @@ static int migrate_pages_batch(struct list_head *from, new_page_t get_new_page, return rc; } +static int migrate_pages_sync(struct list_head *from, new_page_t get_new_page, + free_page_t put_new_page, unsigned long private, + enum migrate_mode mode, int reason, struct list_head *ret_folios, + struct list_head *split_folios, struct migrate_pages_stats *stats) +{ + int rc, nr_failed = 0; + LIST_HEAD(folios); + struct migrate_pages_stats astats; + + memset(&astats, 0, sizeof(astats)); + /* Try to migrate in batch with MIGRATE_ASYNC mode firstly */ + rc = migrate_pages_batch(from, get_new_page, put_new_page, private, MIGRATE_ASYNC, + reason, &folios, split_folios, &astats, + NR_MAX_MIGRATE_ASYNC_RETRY); + stats->nr_succeeded += astats.nr_succeeded; + stats->nr_thp_succeeded += astats.nr_thp_succeeded; + stats->nr_thp_split += astats.nr_thp_split; + if (rc < 0) { + stats->nr_failed_pages += astats.nr_failed_pages; + stats->nr_thp_failed += astats.nr_thp_failed; + list_splice_tail(&folios, ret_folios); + return rc; + } + stats->nr_thp_failed += astats.nr_thp_split; + nr_failed += astats.nr_thp_split; + /* + * Fall back to migrate all failed folios one by one synchronously. All + * failed folios except split THPs will be retried, so their failure + * isn't counted + */ + list_splice_tail_init(&folios, from); + while (!list_empty(from)) { + list_move(from->next, &folios); + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, + private, mode, reason, ret_folios, + split_folios, stats, NR_MAX_MIGRATE_SYNC_RETRY); + list_splice_tail_init(&folios, ret_folios); + if (rc < 0) + return rc; + nr_failed += rc; + } + + return nr_failed; +} + /* * migrate_pages - migrate the folios specified in a list, to the free folios * supplied as the target for the page migration @@ -1891,7 +1935,7 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, enum migrate_mode mode, int reason, unsigned int *ret_succeeded) { int rc, rc_gather; - int nr_pages, batch; + int nr_pages; struct folio *folio, *folio2; LIST_HEAD(folios); LIST_HEAD(ret_folios); @@ -1907,10 +1951,6 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, if (rc_gather < 0) goto out; - if (mode == MIGRATE_ASYNC) - batch = NR_MAX_BATCHED_MIGRATION; - else - batch = 1; again: nr_pages = 0; list_for_each_entry_safe(folio, folio2, from, lru) { @@ -1921,16 +1961,20 @@ int migrate_pages(struct list_head *from, new_page_t get_new_page, } nr_pages += folio_nr_pages(folio); - if (nr_pages >= batch) + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) break; } - if (nr_pages >= batch) + if (nr_pages >= NR_MAX_BATCHED_MIGRATION) list_cut_before(&folios, from, &folio2->lru); else list_splice_init(from, &folios); - rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, - mode, reason, &ret_folios, &split_folios, &stats, - NR_MAX_MIGRATE_PAGES_RETRY); + if (mode == MIGRATE_ASYNC) + rc = migrate_pages_batch(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &split_folios, &stats, + NR_MAX_MIGRATE_PAGES_RETRY); + else + rc = migrate_pages_sync(&folios, get_new_page, put_new_page, private, + mode, reason, &ret_folios, &split_folios, &stats); list_splice_tail_init(&folios, &ret_folios); if (rc < 0) { rc_gather = rc;