From patchwork Wed Sep 22 10:24:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12510021 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3AF3C433F5 for ; Wed, 22 Sep 2021 10:27:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 20D52611C9 for ; Wed, 22 Sep 2021 10:27:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 20D52611C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id B76E76B0073; Wed, 22 Sep 2021 06:27:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AFFEE900002; Wed, 22 Sep 2021 06:27:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97B006B0075; Wed, 22 Sep 2021 06:27:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0241.hostedemail.com [216.40.44.241]) by kanga.kvack.org (Postfix) with ESMTP id 8763D6B0073 for ; Wed, 22 Sep 2021 06:27:15 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3E398180357C2 for ; Wed, 22 Sep 2021 10:27:15 +0000 (UTC) X-FDA: 78614832030.11.211D864 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf03.hostedemail.com (Postfix) with ESMTP id 061B130000A4 for ; Wed, 22 Sep 2021 10:27:14 +0000 (UTC) Received: by mail-pf1-f169.google.com with SMTP id e16so2333370pfc.6 for ; Wed, 22 Sep 2021 03:27:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=z9Y75SXjQ4OjFnf6AL18GnX+kpqvEJUigzCibLGNWc0=; b=GoaNh0eyBtI0BAqbHw0AIPyMRmRMogyXs6ENtuhJGZRh7QLB6qZhAYmlhRt3JpTkmg w+FjzBOWFidlgTJxxCS1rlaNtWkqw9hU8SIgyItfljjyCa6vLtSpriQ/VDeRt7FHf2Ju P4yVKFidSEWAYw2n4yMjkX8Ry9IX/tvlbBZ2l+/C61KUcldmySBCsVL2S65jZFVPNOKo q5z4V9kz6d12molI4J1Q7OB3ZpjEXV0Sr1JZL5LXOVOikW/niitVMPYeW5l0l6P1KRfl zVG34ggd/XbKZoP34T11Le4LU7XbyNHtnHWUEksOrEJ996dvlTq6gFCEDK8WbtuQbh6+ qqbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=z9Y75SXjQ4OjFnf6AL18GnX+kpqvEJUigzCibLGNWc0=; b=fUJQPXUS0z3LOcakCKk6HIYGOUnlfK0LOX2F7GeZ+oUgCiLOuiCtBgqlx31ubX71Ja 2eqk0WiYMNxTJPBpOVdRluKP08VHsfaIJo7UwlgetGicRUKHopxCYrMoQZ1QwPG1zf1Q P+Vo+nMkWrwdynInVd7z0ct6Md5X9ua2qIigjIZByBKfLHIjjte9R6Z9YtU9EpjS8vwJ UHwzN39+4V1ZcNlViG0xTbZgSv3VGrQbgNQSBF15EPXY2TbSUdh/+ibDB/WFU9f+Xz8R 4Htalpq6QOcGalVIX7hxFOqqcQLTVM+o5UkE1DuuPPrnlKceEHVDEobSkHj6WCS6AQ/e V9dw== X-Gm-Message-State: AOAM532xgr8d4zQ7FPGHdtwbrln61Q440GMSBCKxOhm1EmoQUlbF5YyZ 8YgMli/sqz2LHnVAq6iURjq9mg== X-Google-Smtp-Source: ABdhPJwj+m8/3Sco/DMCLFXrMXDs8yZjN0k1vj0JTznwjQRew1gI7Jr5pVDALF/DRfwJHuGQAjB39w== X-Received: by 2002:a63:1f24:: with SMTP id f36mr32764347pgf.6.1632306434072; Wed, 22 Sep 2021 03:27:14 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id s89sm1821929pjj.43.2021.09.22.03.27.08 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 22 Sep 2021 03:27:13 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net, willy@infradead.org, 21cnbao@gmail.com Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, smuchun@gmail.com, zhengqi.arch@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Muchun Song Subject: [PATCH v3 2/4] mm: hugetlb: replace hugetlb_free_vmemmap_enabled with a static_key Date: Wed, 22 Sep 2021 18:24:09 +0800 Message-Id: <20210922102411.34494-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210922102411.34494-1-songmuchun@bytedance.com> References: <20210922102411.34494-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: p7oyezwi9u1mxbyc19iy7qhtsw7wagd3 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=GoaNh0ey; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 061B130000A4 X-HE-Tag: 1632306434-750205 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_head_if_fake() is used throughout memory management and the conditional check requires checking a global variable, although the overhead of this check may be small, it increases when the memory cache comes under pressure. Also, the global variable will not be modified after system boot, so it is very appropriate to use static key machanism. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 10 ++++++++-- include/linux/page-flags.h | 6 ++++-- mm/hugetlb_vmemmap.c | 12 ++++++------ mm/memory_hotplug.c | 2 +- 4 files changed, 19 insertions(+), 11 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 1faebe1cd0ed..4cc647a5dbf8 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1066,9 +1066,15 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr #endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +static inline bool hugetlb_free_vmemmap_enabled(void) +{ + return static_key_enabled(&hugetlb_free_vmemmap_enabled_key); +} #else -#define hugetlb_free_vmemmap_enabled false +static inline bool hugetlb_free_vmemmap_enabled(void) +{ + return false; +} #endif static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index b47a7f51d2c3..54e119e44496 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -185,7 +185,8 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); /* * If the feature of freeing some vmemmap pages associated with each HugeTLB @@ -205,7 +206,8 @@ extern bool hugetlb_free_vmemmap_enabled; */ static __always_inline const struct page *page_fixed_fake_head(const struct page *page) { - if (!hugetlb_free_vmemmap_enabled) + if (!static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + &hugetlb_free_vmemmap_enabled_key)) return page; /* * Only addresses aligned with PAGE_SIZE of struct page may be fake head diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 527bcaa44a48..47517e878ed5 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -188,9 +188,9 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) -bool hugetlb_free_vmemmap_enabled __read_mostly = - IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); -EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled); +DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); +EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled_key); static int __init early_hugetlb_free_vmemmap_param(char *buf) { @@ -204,9 +204,9 @@ static int __init early_hugetlb_free_vmemmap_param(char *buf) return -EINVAL; if (!strcmp(buf, "on")) - hugetlb_free_vmemmap_enabled = true; + static_branch_enable(&hugetlb_free_vmemmap_enabled_key); else if (!strcmp(buf, "off")) - hugetlb_free_vmemmap_enabled = false; + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); else return -EINVAL; @@ -284,7 +284,7 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled) + if (!hugetlb_free_vmemmap_enabled()) return; vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 0488eed3327c..89c1fde02162 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1341,7 +1341,7 @@ bool mhp_supports_memmap_on_memory(unsigned long size) * populate a single PMD. */ return memmap_on_memory && - !hugetlb_free_vmemmap_enabled && + !hugetlb_free_vmemmap_enabled() && IS_ENABLED(CONFIG_MHP_MEMMAP_ON_MEMORY) && size == memory_block_size_bytes() && IS_ALIGNED(vmemmap_size, PMD_SIZE) &&