From patchwork Mon Oct 18 10:20:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12565963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C69F9C433EF for ; Mon, 18 Oct 2021 10:25:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5055B60FED for ; Mon, 18 Oct 2021 10:25:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5055B60FED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id E52B0900004; Mon, 18 Oct 2021 06:25:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E0245900003; Mon, 18 Oct 2021 06:25:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCAF7900004; Mon, 18 Oct 2021 06:25:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0187.hostedemail.com [216.40.44.187]) by kanga.kvack.org (Postfix) with ESMTP id BF36E900003 for ; Mon, 18 Oct 2021 06:25:37 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 790AC1838DC12 for ; Mon, 18 Oct 2021 10:25:37 +0000 (UTC) X-FDA: 78709176714.07.9AAB08A Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf29.hostedemail.com (Postfix) with ESMTP id 20205900024E for ; Mon, 18 Oct 2021 10:25:34 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id lk8-20020a17090b33c800b001a0a284fcc2so14151213pjb.2 for ; Mon, 18 Oct 2021 03:25:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yhMXohaipBke6xOtWf2whLCJBoi6CVAwEzN4wXOFZSY=; b=hnUO9pplQhl5S3GsNvI6yx4RjlSwCCnaMcQvqhLf2TBKOUUDkg47/v6/xUW+vvgv0x 84rkV2kA7bARAel5+WQbpyBZDisbABA88id207Wq7htpPwWyQO1/d2oGWdKr5yn2KyAa Aiu1uQ1A18cw+wOtyl8SPJw+pYQpTEy60dbkLS3FvzF5KPLKKBe31CNb+8r7lQ1BE5hN 4+16f9sfEcscRWhfNbyg/bFRiIo2jgB2DtTGl5IN1unyxS+LCB8t/6h4vOPAb/Jso+Oj Xml9ecpione7lx2zqqQ7l8Hiq1awF8tVJqD1oGFqcOH1Kb+sRwG8yf0XWgLPsXZqIdya JMGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yhMXohaipBke6xOtWf2whLCJBoi6CVAwEzN4wXOFZSY=; b=kLxollpVRlV+yqdVXmY7kvTQW5ovnD72k5W16BxM8d0Ey9P/3AuvDjOAXR+ZzsWWea gwjNTXfrtPy9E2F3dGka8vEsWknGO2nhdqASiwheksDWFnzlK9ag6iktoQL1BAMllx/z J5CfjuueXJ2U3ZtZLB60Ra1dKvfppU/RgVJUgxXO1nRbXCRM3woVAJJes7AZa3OT2J6C a11vvbE7qI1ldkCIMGahWdU2wjw2xfjRzUFI9LI9ZjmlQkkqd2nMOOXLNrmZ0KTNLpSc WTqFfyt4T4zh7qSiexVIlCdua7QDKV7qw51nbwi8x22GTAoa8HaBXvgpy7Irl6FJJIIA LXLA== X-Gm-Message-State: AOAM533OHfogeFOQNHgLQyidX2X3Yhi+HEqKKR1AsT0qawz80WPT+2tF 3PNy/YFqBdtLzGpkp+GvKMEvYQ== X-Google-Smtp-Source: ABdhPJzmhXBehF57sDToISHgBpJeXGOTenrQtWrZvDbe/BybmHbhc2G3sAd4iD96pWqgNZheVhDwMQ== X-Received: by 2002:a17:903:234a:b0:13e:f01a:24f with SMTP id c10-20020a170903234a00b0013ef01a024fmr26554320plh.43.1634552736229; Mon, 18 Oct 2021 03:25:36 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.237]) by smtp.gmail.com with ESMTPSA id nn14sm12762232pjb.27.2021.10.18.03.25.29 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 18 Oct 2021 03:25:36 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net, willy@infradead.org, 21cnbao@gmail.com Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, smuchun@gmail.com, zhengqi.arch@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v6 2/5] mm: hugetlb: replace hugetlb_free_vmemmap_enabled with a static_key Date: Mon, 18 Oct 2021 18:20:40 +0800 Message-Id: <20211018102043.78685-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20211018102043.78685-1-songmuchun@bytedance.com> References: <20211018102043.78685-1-songmuchun@bytedance.com> MIME-Version: 1.0 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=hnUO9ppl; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.41 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 20205900024E X-Stat-Signature: c37ueta4utsnoohyrtxeigaxsips76h9 X-HE-Tag: 1634552734-453653 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_head_if_fake() is used throughout memory management and the conditional check requires checking a global variable, although the overhead of this check may be small, it increases when the memory cache comes under pressure. Also, the global variable will not be modified after system boot, so it is very appropriate to use static key machanism. Signed-off-by: Muchun Song Reviewed-by: Barry Song --- include/linux/hugetlb.h | 6 ------ include/linux/page-flags.h | 16 ++++++++++++++-- mm/hugetlb_vmemmap.c | 12 ++++++------ mm/memory_hotplug.c | 2 +- 4 files changed, 21 insertions(+), 15 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 44c2ab0dfa59..27a2adff0db7 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1077,12 +1077,6 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr } #endif /* CONFIG_HUGETLB_PAGE */ -#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; -#else -#define hugetlb_free_vmemmap_enabled false -#endif - static inline spinlock_t *huge_pte_lock(struct hstate *h, struct mm_struct *mm, pte_t *pte) { diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 7cd386538d0c..26e540fd3393 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -185,7 +185,14 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); + +static __always_inline bool hugetlb_free_vmemmap_enabled(void) +{ + return static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + &hugetlb_free_vmemmap_enabled_key); +} /* * If the feature of freeing some vmemmap pages associated with each HugeTLB @@ -205,7 +212,7 @@ extern bool hugetlb_free_vmemmap_enabled; */ static __always_inline const struct page *page_fixed_fake_head(const struct page *page) { - if (!hugetlb_free_vmemmap_enabled) + if (!hugetlb_free_vmemmap_enabled()) return page; /* @@ -233,6 +240,11 @@ static inline const struct page *page_fixed_fake_head(const struct page *page) { return page; } + +static inline bool hugetlb_free_vmemmap_enabled(void) +{ + return false; +} #endif static __always_inline int page_is_fake_head(struct page *page) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index f4a8fca691ee..22ecb5e21686 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -188,9 +188,9 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) -bool hugetlb_free_vmemmap_enabled __read_mostly = - IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); -EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled); +DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); +EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled_key); static int __init early_hugetlb_free_vmemmap_param(char *buf) { @@ -204,9 +204,9 @@ static int __init early_hugetlb_free_vmemmap_param(char *buf) return -EINVAL; if (!strcmp(buf, "on")) - hugetlb_free_vmemmap_enabled = true; + static_branch_enable(&hugetlb_free_vmemmap_enabled_key); else if (!strcmp(buf, "off")) - hugetlb_free_vmemmap_enabled = false; + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); else return -EINVAL; @@ -284,7 +284,7 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled) + if (!hugetlb_free_vmemmap_enabled()) return; vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 3de7933e5302..587a8fc61fc8 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1326,7 +1326,7 @@ bool mhp_supports_memmap_on_memory(unsigned long size) * populate a single PMD. */ return memmap_on_memory && - !hugetlb_free_vmemmap_enabled && + !hugetlb_free_vmemmap_enabled() && IS_ENABLED(CONFIG_MHP_MEMMAP_ON_MEMORY) && size == memory_block_size_bytes() && IS_ALIGNED(vmemmap_size, PMD_SIZE) &&