From patchwork Wed Mar 23 12:55:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12789758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF593C433EF for ; Wed, 23 Mar 2022 12:57:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43D926B0073; Wed, 23 Mar 2022 08:57:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EBED6B0074; Wed, 23 Mar 2022 08:57:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28C8E6B0075; Wed, 23 Mar 2022 08:57:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 1AC066B0073 for ; Wed, 23 Mar 2022 08:57:22 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id EB28D60FB1 for ; Wed, 23 Mar 2022 12:57:21 +0000 (UTC) X-FDA: 79275651882.13.F28D189 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf11.hostedemail.com (Postfix) with ESMTP id 3E2F94000F for ; Wed, 23 Mar 2022 12:57:21 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id k14so1073438pga.0 for ; Wed, 23 Mar 2022 05:57:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8U++b74ADZPjIiOkZZbim39PJyVay6HYO7+jpHyJ7BU=; b=fuvlOqlnx4uwam7PbWcvOe9tU8gNtjEOl+4ijCOT65bO9wP7Hl7SprBh6d1rQVhw3t 7XW8w8EomUGuwR04Iik6xJW8mJUKc/FISyN3Tlfxr4qxfANF/Ar1WO9NW7oI1PXVWXKM Z3uOIqqJRTxopgmzOVGCHLh48mVCZwsMx+8MxQJV5b8ywKU/uUY34MsDjePT98oshcrM lg+vV1H4wiCBvksP8h9SWYwWpi7nCgJn2cjEAPOnXdMRCDSRaIILcVKQz0wYvy46EKtS hNpvq2cHKJIGg85cYYd9u10qBt0aURe1Oh/c/Co/ouAxxzcDZFqay0SvL5oBOxsCVHu0 a12w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8U++b74ADZPjIiOkZZbim39PJyVay6HYO7+jpHyJ7BU=; b=gclqUA4DS68fxQukJlnNPsg0n7JVyj+HU74FMPXYDuXMxzFKPsBjEW2D3sQSnAPbVL Xdz6rm7WyKJ72GhIGpxMEiVmZMeDjIjEKBGnbH3VEc0wHh6RtxdC3IF7SSlNOxqfzKme URL5ex9c9IVbzH6t5r5F9Qobdpac1stGOMysntQAu9hGC+bB5RuNOUuFhYwnF3p7EhSb 3yBTLxifEHtEXA4BNiXVyNPGMCueSMsFm3vEEo5g7YNd6iH/oLGHtDSiymcBCI37ODG+ UwrsEw/LQkGIXjPxAMmyBBeQjcVlHlKotfdqtBheNvmKvFB01s8C7feafGuqsV9EYH7/ X5/g== X-Gm-Message-State: AOAM532Zrelz1FHpigjkKdtX5WFK9vPhirsiQoQC173bX9QFn00x4JaJ 2bfp4a4L+l9MDPIwT1pImspqTA== X-Google-Smtp-Source: ABdhPJxME2zOZXrnFN9UyKcNEF/huqi8q5lIdipbwtt+diUrlSh84HhmazStaDoumJy/UyhW5GLunw== X-Received: by 2002:a62:6101:0:b0:4fa:7ff4:b543 with SMTP id v1-20020a626101000000b004fa7ff4b543mr22479202pfb.75.1648040240170; Wed, 23 Mar 2022 05:57:20 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id k185-20020a6384c2000000b003821dcd9020sm15716517pgd.27.2022.03.23.05.57.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 05:57:19 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v5 1/4] mm: hugetlb_vmemmap: introduce STRUCT_PAGE_SIZE_IS_POWER_OF_2 Date: Wed, 23 Mar 2022 20:55:20 +0800 Message-Id: <20220323125523.79254-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220323125523.79254-1-songmuchun@bytedance.com> References: <20220323125523.79254-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3E2F94000F X-Stat-Signature: bqxd7dny9okdaydroqi9859a84rq7a1g X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=fuvlOqln; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf11.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-HE-Tag: 1648040241-610676 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If the size of "struct page" is not the power of two and this feature is enabled, then the vmemmap pages of HugeTLB will be corrupted after remapping (panic is about to happen in theory). But this only exists when !CONFIG_MEMCG && !CONFIG_SLUB on x86_64. However, it is not a conventional configuration nowadays. So it is not a real word issue, just the result of a code review. But we have to prevent anyone from configuring that combined configuration. In order to avoid many checks like "is_power_of_2 (sizeof(struct page))" through mm/hugetlb_vmemmap.c. Introduce STRUCT_PAGE_SIZE_IS_POWER_OF_2 to detect if the size of struct page is power of 2 and make this feature depends on this new config. Then we could prevent anyone do any unexpected configuration. Signed-off-by: Muchun Song Suggested-by: Luis Chamberlain Reported-by: kernel test robot Reported-by: kernel test robot --- Kbuild | 14 ++++++++++++++ fs/Kconfig | 1 + include/linux/mm_types.h | 2 ++ mm/Kconfig | 3 +++ mm/hugetlb_vmemmap.c | 6 ------ mm/struct_page_size.c | 19 +++++++++++++++++++ scripts/check_struct_page_po2.sh | 9 +++++++++ 7 files changed, 48 insertions(+), 6 deletions(-) create mode 100644 mm/struct_page_size.c create mode 100755 scripts/check_struct_page_po2.sh diff --git a/Kbuild b/Kbuild index fa441b98c9f6..21415c3b2728 100644 --- a/Kbuild +++ b/Kbuild @@ -37,6 +37,20 @@ $(offsets-file): arch/$(SRCARCH)/kernel/asm-offsets.s FORCE $(call filechk,offsets,__ASM_OFFSETS_H__) ##### +# Generate struct_page_size.h. + +struct_page_size-file := include/generated/struct_page_size.h + +always-y := $(struct_page_size-file) +targets := mm/struct_page_size.s + +mm/struct_page_size.s: $(timeconst-file) $(bounds-file) + +$(struct_page_size-file): mm/struct_page_size.s FORCE + $(call filechk,offsets,__LINUX_STRUCT_PAGE_SIZE_H__) + $(Q)$(MAKE) -f $(srctree)/Makefile syncconfig + +##### # Check for missing system calls always-y += missing-syscalls diff --git a/fs/Kconfig b/fs/Kconfig index 7f2455e8e18a..856d2e9f5aef 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -249,6 +249,7 @@ config HUGETLB_PAGE_FREE_VMEMMAP def_bool HUGETLB_PAGE depends on X86_64 depends on SPARSEMEM_VMEMMAP + depends on STRUCT_PAGE_SIZE_IS_POWER_OF_2 config HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON bool "Default freeing vmemmap pages of HugeTLB to on" diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 8834e38c06a4..5fbff44a4310 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -223,6 +223,7 @@ struct page { #endif } _struct_page_alignment; +#ifndef __GENERATING_STRUCT_PAGE_SIZE_IS_POWER_OF_2_H /** * struct folio - Represents a contiguous set of bytes. * @flags: Identical to the page flags. @@ -844,5 +845,6 @@ enum fault_flag { FAULT_FLAG_INSTRUCTION = 1 << 8, FAULT_FLAG_INTERRUPTIBLE = 1 << 9, }; +#endif /* !__GENERATING_STRUCT_PAGE_SIZE_IS_POWER_OF_2_H */ #endif /* _LINUX_MM_TYPES_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 034d87953600..9314bd34f49e 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -2,6 +2,9 @@ menu "Memory Management options" +config STRUCT_PAGE_SIZE_IS_POWER_OF_2 + def_bool $(success,test "$(shell, $(srctree)/scripts/check_struct_page_po2.sh)" = 1) + config SELECT_MEMORY_MODEL def_bool y depends on ARCH_SELECT_MEMORY_MODEL diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 791626983c2e..33ecb77c2b2a 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -194,12 +194,6 @@ EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled_key); static int __init early_hugetlb_free_vmemmap_param(char *buf) { - /* We cannot optimize if a "struct page" crosses page boundaries. */ - if (!is_power_of_2(sizeof(struct page))) { - pr_warn("cannot free vmemmap pages because \"struct page\" crosses page boundaries\n"); - return 0; - } - if (!buf) return -EINVAL; diff --git a/mm/struct_page_size.c b/mm/struct_page_size.c new file mode 100644 index 000000000000..5749609aa1b3 --- /dev/null +++ b/mm/struct_page_size.c @@ -0,0 +1,19 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Generate definitions needed by the preprocessor. + * This code generates raw asm output which is post-processed + * to extract and format the required data. + */ + +#define __GENERATING_STRUCT_PAGE_SIZE_IS_POWER_OF_2_H +/* Include headers that define the enum constants of interest */ +#include +#include +#include + +int main(void) +{ + DEFINE(STRUCT_PAGE_SIZE_IS_POWER_OF_2, is_power_of_2(sizeof(struct page))); + + return 0; +} diff --git a/scripts/check_struct_page_po2.sh b/scripts/check_struct_page_po2.sh new file mode 100755 index 000000000000..1764ef9a4f1d --- /dev/null +++ b/scripts/check_struct_page_po2.sh @@ -0,0 +1,9 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# +# Check if the size of "struct page" is power of 2 + +file="include/generated/struct_page_size.h" +if [ -f "$file" ]; then + grep STRUCT_PAGE_SIZE_IS_POWER_OF_2 "$file" | cut -d' ' -f3 +fi From patchwork Wed Mar 23 12:55:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12789759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C704C433FE for ; Wed, 23 Mar 2022 12:57:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D0266B0074; Wed, 23 Mar 2022 08:57:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 87F3F6B0075; Wed, 23 Mar 2022 08:57:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F97D6B0078; Wed, 23 Mar 2022 08:57:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.27]) by kanga.kvack.org (Postfix) with ESMTP id 610F26B0074 for ; Wed, 23 Mar 2022 08:57:28 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2CAAA21F88 for ; Wed, 23 Mar 2022 12:57:28 +0000 (UTC) X-FDA: 79275652176.10.8DACF1D Received: from mail-pg1-f170.google.com (mail-pg1-f170.google.com [209.85.215.170]) by imf17.hostedemail.com (Postfix) with ESMTP id B29C340027 for ; Wed, 23 Mar 2022 12:57:27 +0000 (UTC) Received: by mail-pg1-f170.google.com with SMTP id s72so1049590pgc.5 for ; Wed, 23 Mar 2022 05:57:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WwEC4WbmrPNUQyCrBW01Ge/3AIxEgFtA8vMHcjbrgUI=; b=klQHErqFx0PL/Fv5xPsDY5FCp9h1WAWHtQY8CkLJqSnWWxlf7n8gZPbF1jU67rWUkO RiNMKhxX/Xg1v5vX52kPo/YZ9LUvdvabSQyz+QHeO8CRTXNQ+zFyjfInSR4JoPxA3Z5F SRAfE1BxdQg1gFD8ogKft8NnWyg8oM1opsAaki/A9v6Mb9WFi+MI/e3hCeDvBkSzggEO DZzHUo3JWrwfdiuJg5dS4p02oU2qGMsbQSGu32tpXh9I/KPjvq97JrKDey8w9cKoviVw TOVj2mC/CUcDvDZg5H2V5bgy2VpnFGnoDgr4naFnzqFOiFOKszSHtWEqrrgq4Z63avHo UiJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WwEC4WbmrPNUQyCrBW01Ge/3AIxEgFtA8vMHcjbrgUI=; b=NJAplGRXcvk4tll23bwt+DqDmnD99yqy/ZpxzG3KwadKFWK3AsAgTmLLHxX7wtfcs5 dSM+YWYsWqEYlSnotPKLJE8O+REXcTv14iZfbcq9K4CcfsZucyHHRHmUNLgZXcgpyRLO 9mDbYRbQQpAJSBcmGwNoopA2itUhEeOzSs7NQuZd+aOCMJH2cUyB+QINfIcN906XqJL6 s52jonqXQ651MJqq+JDT4RNkQJu/c4wbMkU4M6AWk9sxNsKNsF0sBekH3CPkZmCGoxI9 4wjKfnUJjI8a25x8kuUS6osBVsQSz2mkxNw4InG/eL0RkJIDiXN4E+XFBkyTKXwijrtM sqRA== X-Gm-Message-State: AOAM531xe9ShjvCvv/5zYpRuTwPMnGUVbzNg0svkwc9zkqxx0avlkx+q U6wHEts4iBhZGNnh2huYklw2ag== X-Google-Smtp-Source: ABdhPJxBpZnTaoYXrcelzj7ZCP8izarLpVDuw5/fdNopmLw6AjGmH4Z7zl4e+9aOupHblaaznliQNw== X-Received: by 2002:a05:6a00:22c3:b0:4f7:7cb:26b0 with SMTP id f3-20020a056a0022c300b004f707cb26b0mr33799357pfj.47.1648040246799; Wed, 23 Mar 2022 05:57:26 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id k185-20020a6384c2000000b003821dcd9020sm15716517pgd.27.2022.03.23.05.57.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 05:57:26 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v5 2/4] mm: memory_hotplug: override memmap_on_memory when hugetlb_free_vmemmap=on Date: Wed, 23 Mar 2022 20:55:21 +0800 Message-Id: <20220323125523.79254-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220323125523.79254-1-songmuchun@bytedance.com> References: <20220323125523.79254-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: k844ao36trrtub1ci55xekynzks4a7yx Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=klQHErqF; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf17.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.170 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Queue-Id: B29C340027 X-HE-Tag: 1648040247-624967 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When "hugetlb_free_vmemmap=on" and "memory_hotplug.memmap_on_memory" are both passed to boot cmdline, the variable of "memmap_on_memory" will be set to 1 even if the vmemmap pages will not be allocated from the hotadded memory since the former takes precedence over the latter. In the next patch, we want to enable or disable the feature of freeing vmemmap pages of HugeTLB via sysctl. We need a way to know if the feature of memory_hotplug.memmap_on_memory is enabled when enabling the feature of freeing vmemmap pages since those two features are not compatible, however, the variable of "memmap_on_memory" cannot indicate this nowadays. Do not set "memmap_on_memory" to 1 when both parameters are passed to cmdline, in this case, "memmap_on_memory" could indicate if this feature is enabled by the users. Also introduce mhp_memmap_on_memory() helper to move the definition of "memmap_on_memory" to the scope of CONFIG_MHP_MEMMAP_ON_MEMORY. In the next patch, mhp_memmap_on_memory() will also be exported to be used in hugetlb_vmemmap.c. Signed-off-by: Muchun Song --- mm/memory_hotplug.c | 32 ++++++++++++++++++++++++++------ 1 file changed, 26 insertions(+), 6 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 416b38ca8def..da594b382829 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -42,14 +42,36 @@ #include "internal.h" #include "shuffle.h" +#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY +static int memmap_on_memory_set(const char *val, const struct kernel_param *kp) +{ + if (hugetlb_free_vmemmap_enabled()) + return 0; + return param_set_bool(val, kp); +} + +static const struct kernel_param_ops memmap_on_memory_ops = { + .flags = KERNEL_PARAM_OPS_FL_NOARG, + .set = memmap_on_memory_set, + .get = param_get_bool, +}; /* * memory_hotplug.memmap_on_memory parameter */ static bool memmap_on_memory __ro_after_init; -#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY -module_param(memmap_on_memory, bool, 0444); +module_param_cb(memmap_on_memory, &memmap_on_memory_ops, &memmap_on_memory, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug"); + +static inline bool mhp_memmap_on_memory(void) +{ + return memmap_on_memory; +} +#else +static inline bool mhp_memmap_on_memory(void) +{ + return false; +} #endif enum { @@ -1288,9 +1310,7 @@ bool mhp_supports_memmap_on_memory(unsigned long size) * altmap as an alternative source of memory, and we do not exactly * populate a single PMD. */ - return memmap_on_memory && - !hugetlb_free_vmemmap_enabled() && - IS_ENABLED(CONFIG_MHP_MEMMAP_ON_MEMORY) && + return mhp_memmap_on_memory() && size == memory_block_size_bytes() && IS_ALIGNED(vmemmap_size, PMD_SIZE) && IS_ALIGNED(remaining_size, (pageblock_nr_pages << PAGE_SHIFT)); @@ -2074,7 +2094,7 @@ static int __ref try_remove_memory(u64 start, u64 size) * We only support removing memory added with MHP_MEMMAP_ON_MEMORY in * the same granularity it was added - a single memory block. */ - if (memmap_on_memory) { + if (mhp_memmap_on_memory()) { nr_vmemmap_pages = walk_memory_blocks(start, size, NULL, get_nr_vmemmap_pages_cb); if (nr_vmemmap_pages) { From patchwork Wed Mar 23 12:55:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12789760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 023B6C433EF for ; Wed, 23 Mar 2022 12:57:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 834696B0075; Wed, 23 Mar 2022 08:57:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E2CB6B0078; Wed, 23 Mar 2022 08:57:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6AB4A8D0001; Wed, 23 Mar 2022 08:57:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0027.hostedemail.com [216.40.44.27]) by kanga.kvack.org (Postfix) with ESMTP id 5C6496B0075 for ; Wed, 23 Mar 2022 08:57:35 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 14BBBA325A for ; Wed, 23 Mar 2022 12:57:35 +0000 (UTC) X-FDA: 79275652470.30.0FD9C10 Received: from mail-pg1-f174.google.com (mail-pg1-f174.google.com [209.85.215.174]) by imf17.hostedemail.com (Postfix) with ESMTP id A9B8940027 for ; Wed, 23 Mar 2022 12:57:34 +0000 (UTC) Received: by mail-pg1-f174.google.com with SMTP id s72so1049880pgc.5 for ; Wed, 23 Mar 2022 05:57:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VeiakI6ZdfDqy26wAEKtPtiKSfYNO4Hqsr8Ua8hMV6s=; b=zeeWwcCD9//7csZdk9BkAb8ksK2GIgn57HPaFohCWXiiP8aYQSHKB2Hpfdb13YuOEX KIJ4aNjZW7yk8vagCxd46XF6rNCQ7YkPfMZ97gOgF2rlWWD8XKvcBGg0RSxWP7JyZUWY DFzfyXxLUFe3c+70ocA8kagHeHwgeIA5WneZCToDLSzWhq9fjq5q+/1YPv/4n/HHs5Fn Yx7FUpK7u7Y8o+TsJ0QitplE3fSmqJ1rqiYwmoCSHOy3wpsTfpbXUWeOvHBIArgLk837 BZUvNuawJx/tk5w8zW96S7+aT3Ubv3REKDr6/izxMpR6qsjkj9OdlhRloavP8QytBtii ZsHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VeiakI6ZdfDqy26wAEKtPtiKSfYNO4Hqsr8Ua8hMV6s=; b=qzme9NlGD+YtNnccrA+bNigHuDQO1kCzEN7f8hhnYl2HRZedVQN6bopR+Tt+PwT2/b ON/3z1/8/AH03kfN/v3dyxuhUoBI5EWuP9gYNxxw6YuPQKOGwAXaHVTPnjqtHFG3jHRR VxLyPIeB5ugwdcoKKnne5jfH9KVSvE8uFsoM7NFlQpLrIRlegbU9zLjynHyPe0ThLGTS 0ApEHY52MMMqEChkTrz3oRXfi9YZYxyteGZWOGJG5pAKQyrmt9tb1s7NN62klLjb0ksW 5LktYW4CkNMFd3pfcnmBywKweNCMnLmNNLeu8yxZKFJHATotCl4/ft0Vm4Mct4gwRynx H3JA== X-Gm-Message-State: AOAM530avvnW1inmvDlXDr+Cor/WkF8kb5j2oBUU7ujFvm7oYJQavT9/ ah512dEw7FYSNRjaWwVwgYC+rRxbJxMAgw== X-Google-Smtp-Source: ABdhPJzfb3QxTbZHF2sE8H1JPQSnUAsBsuCcE1dCXSiB41/ce3nustfqBUnhgXYLIVEMuV9Ic1mN6A== X-Received: by 2002:a63:475c:0:b0:381:ef1e:7099 with SMTP id w28-20020a63475c000000b00381ef1e7099mr25853767pgk.225.1648040253862; Wed, 23 Mar 2022 05:57:33 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id k185-20020a6384c2000000b003821dcd9020sm15716517pgd.27.2022.03.23.05.57.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 05:57:33 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v5 3/4] sysctl: allow to set extra1 to SYSCTL_ONE Date: Wed, 23 Mar 2022 20:55:22 +0800 Message-Id: <20220323125523.79254-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220323125523.79254-1-songmuchun@bytedance.com> References: <20220323125523.79254-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A9B8940027 X-Stat-Signature: g47o8mubdzrz1i86f97txuyadj8m4h3d Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=zeeWwcCD; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf17.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.174 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspam-User: X-HE-Tag: 1648040254-71101 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: proc_do_static_key() does not consider the situation where a sysctl is only allowed to be enabled and cannot be disabled under certain circumstances since it set "->extra1" to SYSCTL_ZERO unconditionally. This patch add the functionality to set "->extra1" accordingly. Signed-off-by: Muchun Song --- kernel/sysctl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 770d5f7c7ae4..1e89c3e428ad 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1638,7 +1638,7 @@ int proc_do_static_key(struct ctl_table *table, int write, .data = &val, .maxlen = sizeof(val), .mode = table->mode, - .extra1 = SYSCTL_ZERO, + .extra1 = table->extra1 == SYSCTL_ONE ? SYSCTL_ONE : SYSCTL_ZERO, .extra2 = SYSCTL_ONE, }; From patchwork Wed Mar 23 12:55:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12789761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EC7B4C433F5 for ; Wed, 23 Mar 2022 12:57:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C0168D0001; Wed, 23 Mar 2022 08:57:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 76E316B007B; Wed, 23 Mar 2022 08:57:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5EB0C8D0001; Wed, 23 Mar 2022 08:57:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id 4F8B86B0078 for ; Wed, 23 Mar 2022 08:57:42 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 04E44182056C6 for ; Wed, 23 Mar 2022 12:57:42 +0000 (UTC) X-FDA: 79275652764.30.B4D5350 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf03.hostedemail.com (Postfix) with ESMTP id 84AB020014 for ; Wed, 23 Mar 2022 12:57:41 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id mr5-20020a17090b238500b001c67366ae93so6317273pjb.4 for ; Wed, 23 Mar 2022 05:57:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=yxPYjtXpfBf0hnIpFUPE7iclhmNThzJGoUWla3RxJ4c=; b=PPsk6+P6EYSEz/wZPX6BiOS1i+8fH+kbvejTaqLWjYXj5Zmh2lLYrqc6hRsYhVr+Xp JgP+CP5wxGsICtk1UshhuR++Lau0tLTEaQsSJI7alk5/SgbcgDXjgcxpYXyQ1nGPQa/S BGVFy+0LN50cZvhWnpM+hxDjrCZpMNUj8rxPdwPFCixkvv2XFKfPHRN8DNnpZVVt87jh XuACPA7cX5ytduAI8hUGLcnSgtbQMV62SIjzo4mjoA2wM7Oq+mg0JY314H85sGFp1xZC 2LfwRuqNs8R05aoX419ZJDvDKP/V15OAAGnU9V2IFZLn7kDI2xujzDV8YyI4wFRMg6BR twsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=yxPYjtXpfBf0hnIpFUPE7iclhmNThzJGoUWla3RxJ4c=; b=bo6nQheIL8jiAQEFUK8IInpe6FRmQ62Yvcuus5Lqim1IUnyOS1yoNy6VTgp9UxetnT jqnVfTA8tPLbfYko3mNYi46154890uy7ZnezAJawKz77pe6mEftS57/oLXIjHwSZXpVf AH0ZHx8dxpyHZ3oMxrluW8jjsFJyWWZF8TOaqdBvQ1QOT/7iRcygmpd0PxhdUuFhz1wP ng533TDcpf5MmvOAVqCFCOEQ9vwkz7/Pl8jmvMeyPtlG5UuERZYEvaPUOE6N4IubSpTY jNITps7MiEHaRFYfkomy2W25ZFchxE2ShJddzVmwdhI7P51h1N1mt4KUfDj4+gxCwHT8 Be8w== X-Gm-Message-State: AOAM533YN5ZCsiTPf1IOecTEuuuvxTr52UsWjSVm8bGRPudZHFnr7pa6 vSoJjBo1LQPp5P0rLWbOya+5ceaQ5iSSBA== X-Google-Smtp-Source: ABdhPJx7Yh9faKkk9Uyk1f5LbSPSRuAR1Jw+V+pKb3kI8IKNhoOjGo86Tis7hcep7OcoXAtXplb5pg== X-Received: by 2002:a17:902:8a91:b0:14f:969b:f6be with SMTP id p17-20020a1709028a9100b0014f969bf6bemr23762649plo.161.1648040260501; Wed, 23 Mar 2022 05:57:40 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id k185-20020a6384c2000000b003821dcd9020sm15716517pgd.27.2022.03.23.05.57.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 23 Mar 2022 05:57:40 -0700 (PDT) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com, osalvador@suse.de, david@redhat.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v5 4/4] mm: hugetlb_vmemmap: add hugetlb_free_vmemmap sysctl Date: Wed, 23 Mar 2022 20:55:23 +0800 Message-Id: <20220323125523.79254-5-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220323125523.79254-1-songmuchun@bytedance.com> References: <20220323125523.79254-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Stat-Signature: qa3tfja3rg9uaghz3bpo6d464etdqycz Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=PPsk6+P6; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 84AB020014 X-HE-Tag: 1648040261-896755 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We must add "hugetlb_free_vmemmap=on" to boot cmdline and reboot the server to enable the feature of freeing vmemmap pages of HugeTLB pages. Rebooting usually takes a long time. Add a sysctl to enable or disable the feature at runtime without rebooting. Disabling requires there is no any optimized HugeTLB page in the system. If you fail to disable it, you can set "nr_hugepages" to 0 and then retry. Signed-off-by: Muchun Song --- Documentation/admin-guide/sysctl/vm.rst | 14 +++++ include/linux/memory_hotplug.h | 9 +++ mm/hugetlb_vmemmap.c | 101 +++++++++++++++++++++++++------- mm/hugetlb_vmemmap.h | 4 +- mm/memory_hotplug.c | 7 +-- 5 files changed, 108 insertions(+), 27 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index f4804ce37c58..9e0e153ed935 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -561,6 +561,20 @@ Change the minimum size of the hugepage pool. See Documentation/admin-guide/mm/hugetlbpage.rst +hugetlb_free_vmemmap +==================== + +Enable (set to 1) or disable (set to 0) the feature of optimizing vmemmap +pages associated with each HugeTLB page. Once true, the vmemmap pages of +subsequent allocation of HugeTLB pages from buddy system will be optimized, +whereas already allocated HugeTLB pages will not be optimized. If you fail +to disable this feature, you can set "nr_hugepages" to 0 and then retry +since it is only allowed to be disabled after there is no any optimized +HugeTLB page in the system. + +See Documentation/admin-guide/mm/hugetlbpage.rst + + nr_hugepages_mempolicy ====================== diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 1ce6f8044f1e..9b015b254e86 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -348,4 +348,13 @@ void arch_remove_linear_mapping(u64 start, u64 size); extern bool mhp_supports_memmap_on_memory(unsigned long size); #endif /* CONFIG_MEMORY_HOTPLUG */ +#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY +bool mhp_memmap_on_memory(void); +#else +static inline bool mhp_memmap_on_memory(void) +{ + return false; +} +#endif + #endif /* __LINUX_MEMORY_HOTPLUG_H */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 33ecb77c2b2a..f920073d52ba 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -176,6 +176,7 @@ */ #define pr_fmt(fmt) "HugeTLB: " fmt +#include #include "hugetlb_vmemmap.h" /* @@ -192,6 +193,10 @@ DEFINE_STATIC_KEY_MAYBE(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON, hugetlb_free_vmemmap_enabled_key); EXPORT_SYMBOL(hugetlb_free_vmemmap_enabled_key); +/* How many HugeTLB pages with vmemmap pages optimized. */ +static atomic_long_t optimized_pages = ATOMIC_LONG_INIT(0); +static DECLARE_RWSEM(sysctl_rwsem); + static int __init early_hugetlb_free_vmemmap_param(char *buf) { if (!buf) @@ -208,11 +213,6 @@ static int __init early_hugetlb_free_vmemmap_param(char *buf) } early_param("hugetlb_free_vmemmap", early_hugetlb_free_vmemmap_param); -static inline unsigned long free_vmemmap_pages_size_per_hpage(struct hstate *h) -{ - return (unsigned long)free_vmemmap_pages_per_hpage(h) << PAGE_SHIFT; -} - /* * Previously discarded vmemmap pages will be allocated and remapping * after this function returns zero. @@ -221,14 +221,18 @@ int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) { int ret; unsigned long vmemmap_addr = (unsigned long)head; - unsigned long vmemmap_end, vmemmap_reuse; + unsigned long vmemmap_end, vmemmap_reuse, vmemmap_pages; if (!HPageVmemmapOptimized(head)) return 0; - vmemmap_addr += RESERVE_VMEMMAP_SIZE; - vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); - vmemmap_reuse = vmemmap_addr - PAGE_SIZE; + vmemmap_addr += RESERVE_VMEMMAP_SIZE; + vmemmap_pages = free_vmemmap_pages_per_hpage(h); + vmemmap_end = vmemmap_addr + (vmemmap_pages << PAGE_SHIFT); + vmemmap_reuse = vmemmap_addr - PAGE_SIZE; + + VM_BUG_ON_PAGE(!vmemmap_pages, head); + /* * The pages which the vmemmap virtual address range [@vmemmap_addr, * @vmemmap_end) are mapped to are freed to the buddy allocator, and @@ -238,8 +242,14 @@ int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) */ ret = vmemmap_remap_alloc(vmemmap_addr, vmemmap_end, vmemmap_reuse, GFP_KERNEL | __GFP_NORETRY | __GFP_THISNODE); - if (!ret) + if (!ret) { ClearHPageVmemmapOptimized(head); + /* + * Paired with acquire semantic in + * hugetlb_free_vmemmap_handler(). + */ + atomic_long_dec_return_release(&optimized_pages); + } return ret; } @@ -247,22 +257,28 @@ int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) void free_huge_page_vmemmap(struct hstate *h, struct page *head) { unsigned long vmemmap_addr = (unsigned long)head; - unsigned long vmemmap_end, vmemmap_reuse; + unsigned long vmemmap_end, vmemmap_reuse, vmemmap_pages; - if (!free_vmemmap_pages_per_hpage(h)) - return; + down_read(&sysctl_rwsem); + vmemmap_pages = free_vmemmap_pages_per_hpage(h); + if (!vmemmap_pages) + goto out; - vmemmap_addr += RESERVE_VMEMMAP_SIZE; - vmemmap_end = vmemmap_addr + free_vmemmap_pages_size_per_hpage(h); - vmemmap_reuse = vmemmap_addr - PAGE_SIZE; + vmemmap_addr += RESERVE_VMEMMAP_SIZE; + vmemmap_end = vmemmap_addr + (vmemmap_pages << PAGE_SHIFT); + vmemmap_reuse = vmemmap_addr - PAGE_SIZE; /* * Remap the vmemmap virtual address range [@vmemmap_addr, @vmemmap_end) * to the page which @vmemmap_reuse is mapped to, then free the pages * which the range [@vmemmap_addr, @vmemmap_end] is mapped to. */ - if (!vmemmap_remap_free(vmemmap_addr, vmemmap_end, vmemmap_reuse)) + if (!vmemmap_remap_free(vmemmap_addr, vmemmap_end, vmemmap_reuse)) { SetHPageVmemmapOptimized(head); + atomic_long_inc(&optimized_pages); + } +out: + up_read(&sysctl_rwsem); } void __init hugetlb_vmemmap_init(struct hstate *h) @@ -278,9 +294,6 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled()) - return; - vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; /* * The head page is not to be freed to buddy allocator, the other tail @@ -296,3 +309,51 @@ void __init hugetlb_vmemmap_init(struct hstate *h) pr_info("can free %d vmemmap pages for %s\n", h->nr_free_vmemmap_pages, h->name); } + +static int hugetlb_free_vmemmap_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, + loff_t *ppos) +{ + int ret; + + down_write(&sysctl_rwsem); + /* + * Cannot be disabled when there is at lease one optimized + * HugeTLB in the system. + * + * The acquire semantic is paired with release semantic in + * alloc_huge_page_vmemmap(). If we saw the @optimized_pages + * with 0, all the operations of vmemmap pages remapping from + * alloc_huge_page_vmemmap() are visible too so that we can + * safely disable static key. + */ + table->extra1 = atomic_long_read_acquire(&optimized_pages) ? + SYSCTL_ONE : SYSCTL_ZERO; + ret = proc_do_static_key(table, write, buffer, length, ppos); + up_write(&sysctl_rwsem); + + return ret; +} + +static struct ctl_table hugetlb_vmemmap_sysctls[] = { + { + .procname = "hugetlb_free_vmemmap", + .data = &hugetlb_free_vmemmap_enabled_key.key, + .mode = 0644, + .proc_handler = hugetlb_free_vmemmap_handler, + }, + { } +}; + +static __init int hugetlb_vmemmap_sysctls_init(void) +{ + /* + * The vmemmap pages cannot be optimized if + * "memory_hotplug.memmap_on_memory" is enabled. + */ + if (!mhp_memmap_on_memory()) + register_sysctl_init("vm", hugetlb_vmemmap_sysctls); + + return 0; +} +late_initcall(hugetlb_vmemmap_sysctls_init); diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index cb2bef8f9e73..b67a159027f4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -21,7 +21,9 @@ void hugetlb_vmemmap_init(struct hstate *h); */ static inline unsigned int free_vmemmap_pages_per_hpage(struct hstate *h) { - return h->nr_free_vmemmap_pages; + if (hugetlb_free_vmemmap_enabled()) + return h->nr_free_vmemmap_pages; + return 0; } #else static inline int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index da594b382829..793c04cfe46f 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -63,15 +63,10 @@ static bool memmap_on_memory __ro_after_init; module_param_cb(memmap_on_memory, &memmap_on_memory_ops, &memmap_on_memory, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug"); -static inline bool mhp_memmap_on_memory(void) +bool mhp_memmap_on_memory(void) { return memmap_on_memory; } -#else -static inline bool mhp_memmap_on_memory(void) -{ - return false; -} #endif enum {