From patchwork Wed Mar 3 10:20:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 12113251 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2687CC433DB for ; Wed, 3 Mar 2021 10:21:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A0FD2601FF for ; Wed, 3 Mar 2021 10:21:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A0FD2601FF Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2CE5C8D0152; Wed, 3 Mar 2021 05:21:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27DEB8D0135; Wed, 3 Mar 2021 05:21:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 11F868D0152; Wed, 3 Mar 2021 05:21:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0137.hostedemail.com [216.40.44.137]) by kanga.kvack.org (Postfix) with ESMTP id E58048D0135 for ; Wed, 3 Mar 2021 05:21:56 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id A321F8249980 for ; Wed, 3 Mar 2021 10:21:56 +0000 (UTC) X-FDA: 77878172232.11.43324CE Received: from mga05.intel.com (mga05.intel.com [192.55.52.43]) by imf10.hostedemail.com (Postfix) with ESMTP id 7B8D6407F8F7 for ; Wed, 3 Mar 2021 10:21:53 +0000 (UTC) IronPort-SDR: DoXhW6xIyvY68BfBit3FBPrhdVAnXiEkZqBwDavH56gxQQZgbSoK01+z1pWo1YjakWs1Vzce7B 6QtisXxJmGlw== X-IronPort-AV: E=McAfee;i="6000,8403,9911"; a="272162880" X-IronPort-AV: E=Sophos;i="5.81,219,1610438400"; d="scan'208";a="272162880" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Mar 2021 02:21:54 -0800 IronPort-SDR: wq82e/lOEyykZxs9PW7u5fkZ50pf6gS06CUPn93kDOM5AOVzmbNuwLtWO4w8kTqTiRzCt2CZ29 sgrE/QkcAxlA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.81,219,1610438400"; d="scan'208";a="445200421" Received: from shbuild999.sh.intel.com ([10.239.146.165]) by orsmga001.jf.intel.com with ESMTP; 03 Mar 2021 02:21:50 -0800 From: Feng Tang To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrew Morton Cc: Michal Hocko , Andrea Arcangeli , David Rientjes , Mel Gorman , Mike Kravetz , Randy Dunlap , Vlastimil Babka , Dave Hansen , Ben Widawsky , Andi leen , Dan Williams , Feng Tang Subject: [PATCH v3 RFC 14/14] mm: speedup page alloc for MPOL_PREFERRED_MANY by adding a NO_SLOWPATH gfp bit Date: Wed, 3 Mar 2021 18:20:58 +0800 Message-Id: <1614766858-90344-15-git-send-email-feng.tang@intel.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1614766858-90344-1-git-send-email-feng.tang@intel.com> References: <1614766858-90344-1-git-send-email-feng.tang@intel.com> X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 7B8D6407F8F7 X-Stat-Signature: h4r4a6765zibxgdaby4z6rnbwkcctgig Received-SPF: none (intel.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=mga05.intel.com; client-ip=192.55.52.43 X-HE-DKIM-Result: none/none X-HE-Tag: 1614766913-925600 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When doing broader test, we noticed allocation slowness in one test case that malloc memory with size which is slightly bigger than free memory of targeted nodes, but much less then the total free memory of system. The reason is the code enters the slowpath of __alloc_pages_nodemask(), which takes quite some time. As alloc_pages_policy() will give it a 2nd try with NULL nodemask, so there is no need to enter the slowpath for the first try. Add a new gfp bit to skip the slowpath, so that user cases like this can leverage. With it, the malloc in such case is much accelerated as it never enters the slowpath. Adding a new gfp_mask bit is generally not liked, and another idea is to add another nodemask to struct 'alloc_context', so it has 2: 'preferred-nmask' and 'fallback-nmask', and they will be tried in turn if not NULL, with it we can call __alloc_pages_nodemask() only once. Signed-off-by: Feng Tang Signed-off-by: Feng Tang --- include/linux/gfp.h | 9 +++++++-- mm/mempolicy.c | 2 +- mm/page_alloc.c | 2 +- 3 files changed, 9 insertions(+), 4 deletions(-) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 6e479e9..81bacbe 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -39,8 +39,9 @@ struct vm_area_struct; #define ___GFP_HARDWALL 0x100000u #define ___GFP_THISNODE 0x200000u #define ___GFP_ACCOUNT 0x400000u +#define ___GFP_NO_SLOWPATH 0x800000u #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x800000u +#define ___GFP_NOLOCKDEP 0x1000000u #else #define ___GFP_NOLOCKDEP 0 #endif @@ -220,11 +221,15 @@ struct vm_area_struct; #define __GFP_COMP ((__force gfp_t)___GFP_COMP) #define __GFP_ZERO ((__force gfp_t)___GFP_ZERO) +/* Do not go into the slowpath */ +#define __GFP_NO_SLOWPATH ((__force gfp_t)___GFP_NO_SLOWPATH) + /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) + /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (23 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (24 + IS_ENABLED(CONFIG_LOCKDEP)) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/mm/mempolicy.c b/mm/mempolicy.c index d66c1c0..e84b56d 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2206,7 +2206,7 @@ static struct page *alloc_pages_policy(struct mempolicy *pol, gfp_t gfp, * +-------------------------------+---------------+------------+ */ if (pol->mode == MPOL_PREFERRED_MANY) - gfp_mask |= __GFP_RETRY_MAYFAIL | __GFP_NOWARN; + gfp_mask |= __GFP_RETRY_MAYFAIL | __GFP_NOWARN | __GFP_NO_SLOWPATH; page = __alloc_pages_nodemask(gfp_mask, order, policy_node(gfp, pol, preferred_nid), diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 519a60d..969e3a1 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4993,7 +4993,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, int preferred_nid, /* First allocation attempt */ page = get_page_from_freelist(alloc_mask, order, alloc_flags, &ac); - if (likely(page)) + if (likely(page) || (gfp_mask & __GFP_NO_SLOWPATH)) goto out; /*