From patchwork Wed Jul 14 09:17:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12376359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5DBCEC07E9C for ; Wed, 14 Jul 2021 09:21:18 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0B5D96128B for ; Wed, 14 Jul 2021 09:21:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0B5D96128B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4F4F98D0001; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CB226B0095; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 36C4B8D0001; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0103.hostedemail.com [216.40.44.103]) by kanga.kvack.org (Postfix) with ESMTP id 15CEC6B0092 for ; Wed, 14 Jul 2021 05:21:18 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 1820018587528 for ; Wed, 14 Jul 2021 09:21:17 +0000 (UTC) X-FDA: 78360649794.22.5C88D28 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) by imf29.hostedemail.com (Postfix) with ESMTP id 9559F9000256 for ; Wed, 14 Jul 2021 09:21:16 +0000 (UTC) Received: by mail-pg1-f180.google.com with SMTP id 62so1794049pgf.1 for ; Wed, 14 Jul 2021 02:21:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=CkdbfyChYU3bqFZgwG+U5i+8HOhTg3puUmTJxwvU+7E=; b=U6HfgZ+T/eT56e1XbZEvevYHqqXpNVMhY4FWPTeZZeapWt4nKyvve9g7cSTGlRaEgx Pz/ACDbeddOp2tDYW6pgVgxNjM3yy76a14fwbXT+0hznC6V0PQAicWAhywPZzR3ZvB+t M2RyXS+z6J8shiA0tuFWYJv+L0F+EjgqHmshUyXV8DH5Gi8HxShdiciPI6HbL7DHkSb+ KxtUS6NJP6NWxkhn+G0mViqf0S6oeZ2DnaaLhSUxg90Z+r/23t6OIUyxKThlnr3ZMlaS Pdd/itKuHLZlLFy4Uc6Z1SK4qnvVculUrR0+MLUJf6lic7Vv2JPnEcAC5a0wlhQgRSjS fbWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CkdbfyChYU3bqFZgwG+U5i+8HOhTg3puUmTJxwvU+7E=; b=R8U+iGRwnrqQGj4VcFBvIfGEMnyQTXeYDn5JYKq2GXDC4dkzU49QJ0B9jEKDS4a3qz wWdMZaLGN8n86e0QPpUOVoYd+1y4ykT+ISghga8WvPxNqB0kCAnUFwSPKjpVpEAWmbx8 D6UCK+44DxOHjXnob8PmqOENRYVV1AVTcvTG69U0l4s234jeBdyiqZbmLuj0VrYPlxyt Tp1T95GhxtOFyZp4N194nq7a3nEYoRk315i64LV2gLJ0MShq7z93CcQg05uQRhGM6WIw xgAM7A7nqKNtKNw2wf4ZUex+cnX1eiYbrJ8hViyfcplSvkAIH3nhkkL3Sf9ftLyOAtD1 aC7g== X-Gm-Message-State: AOAM5319CZPcxVk7J2oG8cla/rqJqZ+pvkBeQ1X2Z8/66li/rPbMYfuG Ah7r0jLSU4bWe0SxnI0GkL2B5w== X-Google-Smtp-Source: ABdhPJzoLfhTDsf9vvjpt27Db4UVjn18L2E3wvZk35t27v/G6ikZT65cAao/86OzOUyqtO55T+nmdA== X-Received: by 2002:a65:690a:: with SMTP id s10mr6674972pgq.99.1626254475855; Wed, 14 Jul 2021 02:21:15 -0700 (PDT) Received: from localhost.localdomain ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id k19sm1742540pji.32.2021.07.14.02.21.10 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 14 Jul 2021 02:21:15 -0700 (PDT) From: Muchun Song To: mike.kravetz@oracle.com, akpm@linux-foundation.org, osalvador@suse.de, mhocko@suse.com, song.bao.hua@hisilicon.com, david@redhat.com, chenhuang5@huawei.com, bodeddub@amazon.com, corbet@lwn.net Cc: duanxiongchun@bytedance.com, fam.zheng@bytedance.com, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH 4/5] mm: hugetlb: replace hugetlb_free_vmemmap_enabled with a static_key Date: Wed, 14 Jul 2021 17:17:59 +0800 Message-Id: <20210714091800.42645-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210714091800.42645-1-songmuchun@bytedance.com> References: <20210714091800.42645-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9559F9000256 X-Stat-Signature: e9675jboacarm4kussesmtqro7brp1f1 Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20150623.gappssmtp.com header.s=20150623 header.b=U6HfgZ+T; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.180 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1626254476-849452 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The page_head_if_fake() is used throughout memory management and the conditional check requires checking a global variable, although the overhead of this check may be small, it increases when the memory cache comes under pressure. Also, the global variable will not be modified after system boot, so it is very appropriate to use static key machanism. Signed-off-by: Muchun Song --- include/linux/hugetlb.h | 6 +++++- include/linux/page-flags.h | 6 ++++-- mm/hugetlb_vmemmap.c | 13 +++++++------ 3 files changed, 16 insertions(+), 9 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index f7ca1a3870ea..876a1af73bec 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1057,7 +1057,11 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr #endif /* CONFIG_HUGETLB_PAGE */ #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); +#define hugetlb_free_vmemmap_enabled \ + static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, \ + &hugetlb_free_vmemmap_enabled_key) #else #define hugetlb_free_vmemmap_enabled false #endif diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index d29c16f7b176..f05f806b98b4 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -183,7 +183,8 @@ enum pageflags { #ifndef __GENERATING_BOUNDS_H #ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP -extern bool hugetlb_free_vmemmap_enabled; +DECLARE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); /* * If the feature of freeing some vmemmap pages associated with each HugeTLB @@ -222,7 +223,8 @@ extern bool hugetlb_free_vmemmap_enabled; */ static __always_inline struct page *page_head_if_fake(const struct page *page) { - if (!hugetlb_free_vmemmap_enabled) + if (!static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + &hugetlb_free_vmemmap_enabled_key)) return NULL; /* diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 527bcaa44a48..a382e12f6ff0 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -188,9 +188,9 @@ #define RESERVE_VMEMMAP_NR 1U #define RESERVE_VMEMMAP_SIZE (RESERVE_VMEMMAP_NR << PAGE_SHIFT) -bool hugetlb_free_vmemmap_enabled __read_mostly = - IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON); -EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled); +DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + hugetlb_free_vmemmap_enabled_key); +EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled_key); static int __init early_hugetlb_free_vmemmap_param(char *buf) { @@ -204,9 +204,9 @@ static int __init early_hugetlb_free_vmemmap_param(char *buf) return -EINVAL; if (!strcmp(buf, "on")) - hugetlb_free_vmemmap_enabled = true; + static_branch_enable(&hugetlb_free_vmemmap_enabled_key); else if (!strcmp(buf, "off")) - hugetlb_free_vmemmap_enabled = false; + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); else return -EINVAL; @@ -284,7 +284,8 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled) + if (!static_branch_maybe(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, + &hugetlb_free_vmemmap_enabled_key)) return; vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT;