From patchwork Thu Apr 4 02:00:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zi Yan X-Patchwork-Id: 10884741 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 33BCB1708 for ; Thu, 4 Apr 2019 02:01:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 16F95288E4 for ; Thu, 4 Apr 2019 02:01:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0B20528928; Thu, 4 Apr 2019 02:01:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4C104288E4 for ; Thu, 4 Apr 2019 02:01:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1DE4A6B0266; Wed, 3 Apr 2019 22:01:21 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 18CDD6B0269; Wed, 3 Apr 2019 22:01:21 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 007E86B026A; Wed, 3 Apr 2019 22:01:20 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by kanga.kvack.org (Postfix) with ESMTP id D09416B0266 for ; Wed, 3 Apr 2019 22:01:20 -0400 (EDT) Received: by mail-qk1-f199.google.com with SMTP id 77so942259qkd.9 for ; Wed, 03 Apr 2019 19:01:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:dkim-signature:from:to:cc:subject :date:message-id:in-reply-to:references:reply-to:mime-version :content-transfer-encoding; bh=sTcFdQ0uTFmNEqF/buqEuv4aNZlA40xkHnCTG+zeMnI=; b=BKFFXFSUXH2aIGQDPot0UOjIwSse2CrkO+PsrPzsWpzaZJTud1brDEScaIB7b8TrGw QsFUPIOAeH6VmWP+ezgRu38NzvwuvORUW0BlBRyCns+sWsyxD+MC1XXRxn4Cy0cTOCfU ROdO8lFY5QsC1VwXgceKAJfmsWcLUZHD1Fw+GZFyFVBzLULi25YzNlpOFwgid6lG6Fop /QfE1iEfXHk+ilqo12FS25dgESMtSFMoSLAA3et24d+V9TEBotT0gqyiPwaeho8stfJz 669CTV1/eOClYZHj26pIBZUqesDKim3/QjEe/dBaJT4OIEqjGy1xydWI+AdobDvqr0n5 OpDA== X-Gm-Message-State: APjAAAUJFISRGHiWjP08g5Q+gf2lgTw8htzTjqaeIFzmaASgj5S1m2Oy rAqIJ9GNnsNmBvboTlCsglbr/nafN7odZwofjiPOX4NDfyofw2ei23FRg7HLxBPstR4V60Fctoy nkdLEV7BSfwHKKf6ykf1XEYvuIMAVFwhV5mBPt9afl5PIicZgi68X54gNg2/gFt7wKQ== X-Received: by 2002:ac8:2de7:: with SMTP id q36mr3023094qta.3.1554343280557; Wed, 03 Apr 2019 19:01:20 -0700 (PDT) X-Google-Smtp-Source: APXvYqy39vXIqXru2GmNHldv70bAIrmJnzf1lYkt7slQ6+loiMlhxTFvjQfz5Wty6PaD/ihSg79p X-Received: by 2002:ac8:2de7:: with SMTP id q36mr3023061qta.3.1554343279862; Wed, 03 Apr 2019 19:01:19 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1554343279; cv=none; d=google.com; s=arc-20160816; b=nZY4PGIWWvDW2Vtc+y43zp2EqUwTY7Tlmo22vfJqusQ8APXh4gPnN5bbozGT7iYyzY +pQXzWqo8pLfKjIdpIi3l4WYE6GIs1041EWj9GWW8QpY+ogjATFS9lAQfnCgjR31UK6k Ys6+QvstL8qybM/KdX7/5yIAymg4CssfiS/Yr0Rf3m+4XMO4TnyFwCWJn4jIAIKk72nx 6+RHDv9utm28pcHHQtASkqxaaU06OPzHeD8yKAuO4cwYlaR+ns5GpTUvO8ITv5MQ9u8S erpbBGNMK2MysvsyF/3098e6eDrXl0PkW9oenQKhnouWnoFfcphDFHHBM++YfqcJJMcO zdNw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:dkim-signature :dkim-signature; bh=sTcFdQ0uTFmNEqF/buqEuv4aNZlA40xkHnCTG+zeMnI=; b=khQNHX8DhKs/JH/dR32QC0I+b4ilOv5MGFiinFecFigNwVkKBNi+FxMbK69qZib8Zu ODnbhNjPZO85I/VtWmBau0nA3Jy/HD6VdSirEMXdORrHm3q3nSEORLUq0EIeobYfH5Oe NGl7By+BLikt/TPn3kNQsyEyfLRmJbIazzamK8NJNHRlzXO+prKx1lmqFD7vdr6fkrtJ 1W10rUozRJxeSONiuRTY85YXJ8sksWBblH+OxNmoHMfk3cT5sPbT1q9Clt+WPpfmNbFk tqs37TUIWA8xyUmsV+7QhYaYbPLfEAW3UtD1jRlyNtPWi3sEgekgU9F1+t/DinLXnytR z1HQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@sent.com header.s=fm3 header.b=V0JIpLVR; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=kwN41DXj; spf=pass (google.com: domain of zi.yan@sent.com designates 66.111.4.29 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from out5-smtp.messagingengine.com (out5-smtp.messagingengine.com. [66.111.4.29]) by mx.google.com with ESMTPS id a29si8169888qte.337.2019.04.03.19.01.19 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Apr 2019 19:01:19 -0700 (PDT) Received-SPF: pass (google.com: domain of zi.yan@sent.com designates 66.111.4.29 as permitted sender) client-ip=66.111.4.29; Authentication-Results: mx.google.com; dkim=pass header.i=@sent.com header.s=fm3 header.b=V0JIpLVR; dkim=pass header.i=@messagingengine.com header.s=fm2 header.b=kwN41DXj; spf=pass (google.com: domain of zi.yan@sent.com designates 66.111.4.29 as permitted sender) smtp.mailfrom=zi.yan@sent.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=sent.com Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.nyi.internal (Postfix) with ESMTP id 8F98A2257C; Wed, 3 Apr 2019 22:01:19 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 03 Apr 2019 22:01:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sent.com; h=from :to:cc:subject:date:message-id:in-reply-to:references:reply-to :mime-version:content-transfer-encoding; s=fm3; bh=sTcFdQ0uTFmNE qF/buqEuv4aNZlA40xkHnCTG+zeMnI=; b=V0JIpLVR1+jFUK3V21cuqVcW5pHEz EjA/Azwlmjyub9N3bs1MsXPgbBsO2TbK48M6mumUTV6WCI5VuqZYbRO+GKb/F2T2 Vjd2VxpLT5dcKrq5JWA4CIlwCOot8ZdU6Zmwu5DyJLLe1fAQfq+XaO0/mHN97L7p YC4X04nk0pRDz0tbycrlOugeHPKN7jLOUHyb9Wl6UCug0pM0yQ8SbaCQ/vvCdVRP OHuhIP2qIUYE/YZrDf/47drANb1XAr4qHcJ1ChcZ/Z1cqry5k+pBRCptIjE/yqZZ Lxni0AmzKX5tCCHhHXao1ySlxQRlPOnWX9cXwelHJVeYAdC6lFbQMbqJw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:date:from :in-reply-to:message-id:mime-version:references:reply-to:subject :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=sTcFdQ0uTFmNEqF/buqEuv4aNZlA40xkHnCTG+zeMnI=; b=kwN41DXj 0DXVNlAxGxdTXxrF5A2KNJ+rdQQp1auwloTcMOilZmB5wftzWBgF7+BDjm+haTXZ wY29smEaDNOa3uKbyjcnfL8JvksBch6T2C7Ad10foa0wKFC7NGFnytKrGpR5BF1j oLXtBZaMW3XgQa44IumNHFDXQQYLe4L12argM29pFDY/tnqldRBoMba9+CmEEWNx R2A2J4fg64nzem4SqoPRSUl+IvAHSHBse7s1+SairkNqiGloxy0aNrqhwaUv/ah4 7zWfKW37ypooI9QzHt3wWiNkuwK3fkvoK75RkCnuRRm3eJhdgW66K2a4BC3uPOzv 0A3gJOEARFO6gg== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrtdeggdehudculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivg hnthhsucdlqddutddtmdenucfjughrpefhvffufffkofgjfhhrggfgsedtkeertdertddt necuhfhrohhmpegkihcujggrnhcuoeiiihdrhigrnhesshgvnhhtrdgtohhmqeenucfkph epvdduiedrvddvkedrudduvddrvddvnecurfgrrhgrmhepmhgrihhlfhhrohhmpeiiihdr higrnhesshgvnhhtrdgtohhmnecuvehluhhsthgvrhfuihiivgepud X-ME-Proxy: Received: from nvrsysarch5.nvidia.com (thunderhill.nvidia.com [216.228.112.22]) by mail.messagingengine.com (Postfix) with ESMTPA id B33B51030F; Wed, 3 Apr 2019 22:01:17 -0400 (EDT) From: Zi Yan To: Dave Hansen , Yang Shi , Keith Busch , Fengguang Wu , linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Daniel Jordan , Michal Hocko , "Kirill A . Shutemov" , Andrew Morton , Vlastimil Babka , Mel Gorman , John Hubbard , Mark Hairgrove , Nitin Gupta , Javier Cabezas , David Nellans , Zi Yan Subject: [RFC PATCH 04/25] mm: migrate: Add copy_page_multithread into migrate_pages. Date: Wed, 3 Apr 2019 19:00:25 -0700 Message-Id: <20190404020046.32741-5-zi.yan@sent.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20190404020046.32741-1-zi.yan@sent.com> References: <20190404020046.32741-1-zi.yan@sent.com> Reply-To: ziy@nvidia.com MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Zi Yan An option is added to move_pages() syscall to use multi-threaded page migration. Signed-off-by: Zi Yan --- include/linux/migrate_mode.h | 1 + include/uapi/linux/mempolicy.h | 2 ++ mm/migrate.c | 29 +++++++++++++++++++---------- 3 files changed, 22 insertions(+), 10 deletions(-) diff --git a/include/linux/migrate_mode.h b/include/linux/migrate_mode.h index da44940..5bc8a77 100644 --- a/include/linux/migrate_mode.h +++ b/include/linux/migrate_mode.h @@ -22,6 +22,7 @@ enum migrate_mode { MIGRATE_MODE_MASK = 3, MIGRATE_SINGLETHREAD = 0, + MIGRATE_MT = 1<<4, }; #endif /* MIGRATE_MODE_H_INCLUDED */ diff --git a/include/uapi/linux/mempolicy.h b/include/uapi/linux/mempolicy.h index 3354774..890269b 100644 --- a/include/uapi/linux/mempolicy.h +++ b/include/uapi/linux/mempolicy.h @@ -48,6 +48,8 @@ enum { #define MPOL_MF_LAZY (1<<3) /* Modifies '_MOVE: lazy migrate on fault */ #define MPOL_MF_INTERNAL (1<<4) /* Internal flags start here */ +#define MPOL_MF_MOVE_MT (1<<6) /* Use multi-threaded page copy routine */ + #define MPOL_MF_VALID (MPOL_MF_STRICT | \ MPOL_MF_MOVE | \ MPOL_MF_MOVE_ALL) diff --git a/mm/migrate.c b/mm/migrate.c index 2b2653e..dd6ccbe 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -572,6 +572,7 @@ static void copy_huge_page(struct page *dst, struct page *src, { int i; int nr_pages; + int rc = -EFAULT; if (PageHuge(src)) { /* hugetlbfs page */ @@ -588,10 +589,14 @@ static void copy_huge_page(struct page *dst, struct page *src, nr_pages = hpage_nr_pages(src); } - for (i = 0; i < nr_pages; i++) { - cond_resched(); - copy_highpage(dst + i, src + i); - } + if (mode & MIGRATE_MT) + rc = copy_page_multithread(dst, src, nr_pages); + + if (rc) + for (i = 0; i < nr_pages; i++) { + cond_resched(); + copy_highpage(dst + i, src + i); + } } /* @@ -1500,7 +1505,7 @@ static int store_status(int __user *status, int start, int value, int nr) } static int do_move_pages_to_node(struct mm_struct *mm, - struct list_head *pagelist, int node) + struct list_head *pagelist, int node, bool migrate_mt) { int err; @@ -1508,7 +1513,8 @@ static int do_move_pages_to_node(struct mm_struct *mm, return 0; err = migrate_pages(pagelist, alloc_new_node_page, NULL, node, - MIGRATE_SYNC, MR_SYSCALL); + MIGRATE_SYNC | (migrate_mt ? MIGRATE_MT : MIGRATE_SINGLETHREAD), + MR_SYSCALL); if (err) putback_movable_pages(pagelist); return err; @@ -1629,7 +1635,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, current_node = node; start = i; } else if (node != current_node) { - err = do_move_pages_to_node(mm, &pagelist, current_node); + err = do_move_pages_to_node(mm, &pagelist, current_node, + flags & MPOL_MF_MOVE_MT); if (err) goto out; err = store_status(status, start, current_node, i - start); @@ -1652,7 +1659,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, if (err) goto out_flush; - err = do_move_pages_to_node(mm, &pagelist, current_node); + err = do_move_pages_to_node(mm, &pagelist, current_node, + flags & MPOL_MF_MOVE_MT); if (err) goto out; if (i > start) { @@ -1667,7 +1675,8 @@ static int do_pages_move(struct mm_struct *mm, nodemask_t task_nodes, return err; /* Make sure we do not overwrite the existing error */ - err1 = do_move_pages_to_node(mm, &pagelist, current_node); + err1 = do_move_pages_to_node(mm, &pagelist, current_node, + flags & MPOL_MF_MOVE_MT); if (!err1) err1 = store_status(status, start, current_node, i - start); if (!err) @@ -1763,7 +1772,7 @@ static int kernel_move_pages(pid_t pid, unsigned long nr_pages, nodemask_t task_nodes; /* Check flags */ - if (flags & ~(MPOL_MF_MOVE|MPOL_MF_MOVE_ALL)) + if (flags & ~(MPOL_MF_MOVE|MPOL_MF_MOVE_ALL|MPOL_MF_MOVE_MT)) return -EINVAL; if ((flags & MPOL_MF_MOVE_ALL) && !capable(CAP_SYS_NICE))