From patchwork Fri Jun 19 16:24:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ben Widawsky X-Patchwork-Id: 11614585 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A2F2D90 for ; Fri, 19 Jun 2020 16:24:27 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 799222168B for ; Fri, 19 Jun 2020 16:24:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 799222168B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8F1828D00CF; Fri, 19 Jun 2020 12:24:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 87C0F8D00CE; Fri, 19 Jun 2020 12:24:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 744298D00CF; Fri, 19 Jun 2020 12:24:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0250.hostedemail.com [216.40.44.250]) by kanga.kvack.org (Postfix) with ESMTP id 4E1038D00CE for ; Fri, 19 Jun 2020 12:24:23 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 0B294180AD822 for ; Fri, 19 Jun 2020 16:24:23 +0000 (UTC) X-FDA: 76946484006.19.sort00_4e1563926e1a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id B62F81AD1B1 for ; Fri, 19 Jun 2020 16:24:22 +0000 (UTC) X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,ben.widawsky@intel.com,,RULES_HIT:30012:30054:30064:30070,0,RBL:134.134.136.100:@intel.com:.lbl8.mailshell.net-62.18.0.100 64.95.201.95,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:17,LUA_SUMMARY:none X-HE-Tag: sort00_4e1563926e1a X-Filterd-Recvd-Size: 6374 Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Fri, 19 Jun 2020 16:24:21 +0000 (UTC) IronPort-SDR: sXhD0NOB0B2DAD6KNmWKwCrQE4lEqN8IqO5EWVT+uU7ssshMMoyq3eWak/Ae2RT9gGcSU2tnX3 xvHGy4g6CGwg== X-IronPort-AV: E=McAfee;i="6000,8403,9657"; a="208241115" X-IronPort-AV: E=Sophos;i="5.75,255,1589266800"; d="scan'208";a="208241115" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2020 09:24:17 -0700 IronPort-SDR: QyzmVislWiGMSX9RlqRypaXmCqqXRntrqn26yzGwbYA/cJDX4siRn12I1Q75uMXVe11wintGcN pvbrBy++bW5g== X-IronPort-AV: E=Sophos;i="5.75,255,1589266800"; d="scan'208";a="264366425" Received: from sjiang-mobl2.ccr.corp.intel.com (HELO bwidawsk-mobl5.local) ([10.252.131.131]) by orsmga007-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 19 Jun 2020 09:24:17 -0700 From: Ben Widawsky To: linux-mm Subject: [PATCH 11/18] mm: Extract THP hugepage allocation Date: Fri, 19 Jun 2020 09:24:07 -0700 Message-Id: <20200619162414.1052234-12-ben.widawsky@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20200619162414.1052234-1-ben.widawsky@intel.com> References: <20200619162414.1052234-1-ben.widawsky@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B62F81AD1B1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The next patch is going to rework this code to support MPOL_PREFERRED_MANY. This refactor makes the that change much more readable. After the extraction, the resulting code makes it apparent that this can be converted to a simple if ladder and thus allows removing the goto. There is not meant to be any functional or behavioral changes. Note that still at this point MPOL_PREFERRED_MANY isn't specially handled for huge pages. Cc: Andrew Morton Cc: Dave Hansen Cc: Michal Hocko Signed-off-by: Ben Widawsky --- mm/mempolicy.c | 96 ++++++++++++++++++++++++++------------------------ 1 file changed, 49 insertions(+), 47 deletions(-) diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 408ba78c8424..3ce2354fed44 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -2232,6 +2232,48 @@ static struct page *alloc_page_interleave(gfp_t gfp, unsigned order, return page; } +static struct page *alloc_pages_vma_thp(gfp_t gfp, struct mempolicy *pol, + int order, int node) +{ + nodemask_t *nmask; + struct page *page; + int hpage_node = node; + + /* + * For hugepage allocation and non-interleave policy which allows the + * current node (or other explicitly preferred node) we only try to + * allocate from the current/preferred node and don't fall back to other + * nodes, as the cost of remote accesses would likely offset THP + * benefits. + * + * If the policy is interleave or multiple preferred nodes, or does not + * allow the current node in its nodemask, we allocate the standard way. + */ + if (pol->mode == MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL)) + hpage_node = first_node(pol->v.preferred_nodes); + + nmask = policy_nodemask(gfp, pol); + + /* + * First, try to allocate THP only on local node, but don't reclaim + * unnecessarily, just compact. + */ + page = __alloc_pages_nodemask(gfp | __GFP_THISNODE | __GFP_NORETRY, + order, hpage_node, nmask); + + /* + * If hugepage allocations are configured to always synchronous compact + * or the vma has been madvised to prefer hugepage backing, retry + * allowing remote memory with both reclaim and compact as well. + */ + if (!page && (gfp & __GFP_DIRECT_RECLAIM)) + page = __alloc_pages_nodemask(gfp, order, hpage_node, nmask); + + VM_BUG_ON(page && nmask && !node_isset(page_to_nid(page), *nmask)); + + return page; +} + /** * alloc_pages_vma - Allocate a page for a VMA. * @@ -2272,57 +2314,17 @@ alloc_pages_vma(gfp_t gfp, int order, struct vm_area_struct *vma, nid = interleave_nid(pol, vma, addr, PAGE_SHIFT + order); mpol_cond_put(pol); page = alloc_page_interleave(gfp, order, nid); - goto out; - } - - if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && hugepage)) { - int hpage_node = node; - - /* - * For hugepage allocation and non-interleave policy which - * allows the current node (or other explicitly preferred - * node) we only try to allocate from the current/preferred - * node and don't fall back to other nodes, as the cost of - * remote accesses would likely offset THP benefits. - * - * If the policy is interleave or multiple preferred nodes, or - * does not allow the current node in its nodemask, we allocate - * the standard way. - */ - if (pol->mode == MPOL_PREFERRED && !(pol->flags & MPOL_F_LOCAL)) - hpage_node = first_node(pol->v.preferred_nodes); - + } else if (unlikely(IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE) && + hugepage)) { + page = alloc_pages_vma_thp(gfp, pol, order, node); + mpol_cond_put(pol); + } else { nmask = policy_nodemask(gfp, pol); + preferred_nid = policy_node(gfp, pol, node); + page = __alloc_pages_nodemask(gfp, order, preferred_nid, nmask); mpol_cond_put(pol); - - /* - * First, try to allocate THP only on local node, but - * don't reclaim unnecessarily, just compact. - */ - page = __alloc_pages_nodemask(gfp | __GFP_THISNODE | - __GFP_NORETRY, - order, hpage_node, nmask); - - /* - * If hugepage allocations are configured to always synchronous - * compact or the vma has been madvised to prefer hugepage - * backing, retry allowing remote memory with both reclaim and - * compact as well. - */ - if (!page && (gfp & __GFP_DIRECT_RECLAIM)) - page = __alloc_pages_nodemask(gfp, order, hpage_node, - nmask); - - VM_BUG_ON(page && nmask && - !node_isset(page_to_nid(page), *nmask)); - goto out; } - nmask = policy_nodemask(gfp, pol); - preferred_nid = policy_node(gfp, pol, node); - page = __alloc_pages_nodemask(gfp, order, preferred_nid, nmask); - mpol_cond_put(pol); -out: return page; } EXPORT_SYMBOL(alloc_pages_vma);