From patchwork Mon Feb 28 07:10:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12762468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFCD1C433EF for ; Mon, 28 Feb 2022 07:11:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D24B8D0007; Mon, 28 Feb 2022 02:11:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 682398D0001; Mon, 28 Feb 2022 02:11:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 524D68D0007; Mon, 28 Feb 2022 02:11:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 428508D0001 for ; Mon, 28 Feb 2022 02:11:42 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0CE48518 for ; Mon, 28 Feb 2022 07:11:42 +0000 (UTC) X-FDA: 79191318444.13.7F4B491 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf30.hostedemail.com (Postfix) with ESMTP id 7EA5180004 for ; Mon, 28 Feb 2022 07:11:41 +0000 (UTC) Received: by mail-pl1-f169.google.com with SMTP id l9so9485886pls.6 for ; Sun, 27 Feb 2022 23:11:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/8ft55r/VTfoT4ZinxhB51u9HRB18drBxk6cMGVkRIc=; b=QPUd6SMtlL0lgH1f6RussrMAtI4S+e2NVZ4z1/cLuYCljBgasiYx6HN63o1FJmT3mB EyERA9+htOPxv14o3q8QvwJsmjpHWrbsY4D9/F3IqZQVtTdSQjgOHYIoizN+awrObS+b o2mqmQMWwaP9vT80WgYs90x14FldeS7DT225M0TViqTER9xsB49CZ19VLK0rtgs7bYAO 880eTRkSqRu/r7E1L3noMD1KcmGHufV9nWspTVwy6deuZwsaBvPqkRSSCBZhRkjKyLnq QSWEj7WLbh8fbWTAI0GZPeBUprgYQt0jZeV+T307CtT3eVTyVwzCq9i4XXeVZ3jIhskr BeCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/8ft55r/VTfoT4ZinxhB51u9HRB18drBxk6cMGVkRIc=; b=fTJ58Zv4f0xaKsrPtgQ3anSHg+F4012zQXA+CY9UTy0+oUx2JWvWr6Uup+EMv/9bjg J4osmLbB9w60pp/X4HqfrCjrV+WA3QPchwiJeYFyqS641GwxrwyZqYV+onE+7kaGT6F6 JtKjA+HDFfm+N/YKIlVmGOJNwCUt09Y6LZvPKGBuuXVpqWm7SAQUFYV021Ua3GgMQRCt BOnAdxpFzSmd492SuBgBkPGGxFLtu6xqhShVA1s3YaJqcK6Xvx2avmy0KBDyVbhJdFh2 2a2dluLDx8Pio5xSJBGZaZYn5YFtMvNdktkCSjC3B61Vi1mm5QFdZhFWYS+NO0nK3/tn FqFg== X-Gm-Message-State: AOAM531Yk2pCxn8J4CXCHMhTsoOpiZFyhoritTjbiXagm/QdqQc4zyPm H40N3lZ2fc1le/wEOnRTwp4ATg== X-Google-Smtp-Source: ABdhPJy8a01fRIV4nGG4Fscsyj42sS294OeFwsAc/3btG49ySTvEgsDMP+HVgRNeNNfgTB6q6/T4XA== X-Received: by 2002:a17:90a:5995:b0:1bc:be7f:f5a1 with SMTP id l21-20020a17090a599500b001bcbe7ff5a1mr15573946pji.84.1646032300573; Sun, 27 Feb 2022 23:11:40 -0800 (PST) Received: from FVFYT0MHHV2J.tiktokcdn.com ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id i7-20020a17090a65c700b001b936b8abe0sm16240390pjs.7.2022.02.27.23.11.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Feb 2022 23:11:40 -0800 (PST) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH 3/3] mm: hugetlb: add hugetlb_free_vmemmap sysctl Date: Mon, 28 Feb 2022 15:10:22 +0800 Message-Id: <20220228071022.26143-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220228071022.26143-1-songmuchun@bytedance.com> References: <20220228071022.26143-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7EA5180004 X-Stat-Signature: xzyft6655ykfk4ink551epcwnj5msyk6 X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=QPUd6SMt; spf=pass (imf30.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam07 X-HE-Tag: 1646032301-162169 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We must add "hugetlb_free_vmemmap=on" to boot cmdline and reboot the server to enable the feature of freeing vmemmap pages of HugeTLB pages. Rebooting usually taske a long time. Add a sysctl to enable the feature at runtime and do not need to reboot. Signed-off-by: Muchun Song Reported-by: kernel test robot Reported-by: kernel test robot --- Documentation/admin-guide/sysctl/vm.rst | 13 +++++++++++++ include/linux/hugetlb.h | 5 +++++ include/linux/memory_hotplug.h | 1 + kernel/sysctl.c | 11 +++++++++++ mm/hugetlb_vmemmap.c | 23 +++++++++++++++++------ mm/hugetlb_vmemmap.h | 4 +++- mm/memory_hotplug.c | 2 +- 7 files changed, 51 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index f4804ce37c58..01f18e6cc227 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -561,6 +561,19 @@ Change the minimum size of the hugepage pool. See Documentation/admin-guide/mm/hugetlbpage.rst +hugetlb_free_vmemmap +==================== + +A toggle value indicating if vmemmap pages are allowed to be optimized. +If it is off (0), then it can be set true (1). Once true, the vmemmap +pages associated with each HugeTLB page will be optimized, and the toggle +cannot be set back to false. It only optimizes the subsequent allocation +of HugeTLB pages from buddy system, while already allocated HugeTLB pages +will not be optimized. + +See Documentation/admin-guide/mm/hugetlbpage.rst + + nr_hugepages_mempolicy ====================== diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 53c1b6082a4c..cc4ab21892f5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1080,6 +1080,11 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr } #endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP +int hugetlb_vmemmap_sysctl_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, loff_t *ppos); +#endif + static inline spinlock_t *huge_pte_lock(struct hstate *h, struct mm_struct *mm, pte_t *pte) { diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index e0b2209ab71c..b30f9fdaed73 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -349,6 +349,7 @@ extern int arch_create_linear_mapping(int nid, u64 start, u64 size, struct mhp_params *params); void arch_remove_linear_mapping(u64 start, u64 size); extern bool mhp_supports_memmap_on_memory(unsigned long size); +extern bool memmap_on_memory; #endif /* CONFIG_MEMORY_HOTPLUG */ #endif /* __LINUX_MEMORY_HOTPLUG_H */ diff --git a/kernel/sysctl.c b/kernel/sysctl.c index ab3e9c937268..77f039849b2a 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -2223,6 +2223,17 @@ static struct ctl_table vm_table[] = { .mode = 0644, .proc_handler = hugetlb_sysctl_handler, }, +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP + { + .procname = "hugetlb_free_vmemmap", + .data = &hugetlb_free_vmemmap_enabled_key.key, + .maxlen = sizeof(hugetlb_free_vmemmap_enabled_key.key), + .mode = 0644, + /* only handle a transition from default "0" to "1" */ + .proc_handler = hugetlb_vmemmap_sysctl_handler, + .extra1 = SYSCTL_ONE, + }, +#endif #ifdef CONFIG_NUMA { .procname = "nr_hugepages_mempolicy", diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 836d1117f08b..3167021055d6 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -10,6 +10,7 @@ #define pr_fmt(fmt) "HugeTLB: " fmt +#include #include "hugetlb_vmemmap.h" /* @@ -118,17 +119,14 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled()) - return; - - if (IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON) && - !is_power_of_2(sizeof(struct page))) { + if (!is_power_of_2(sizeof(struct page))) { /* * The hugetlb_free_vmemmap_enabled_key can be enabled when * CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON. It should * be disabled if "struct page" crosses page boundaries. */ - static_branch_disable(&hugetlb_free_vmemmap_enabled_key); + if (IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON)) + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); return; } @@ -147,3 +145,16 @@ void __init hugetlb_vmemmap_init(struct hstate *h) pr_info("can free %d vmemmap pages for %s\n", h->nr_free_vmemmap_pages, h->name); } + +int hugetlb_vmemmap_sysctl_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, loff_t *ppos) +{ + /* + * The vmemmap pages cannot be optimized if a "struct page" crosses page + * boundaries or memory_hotplug.memmap_on_memory is enabled. + */ + if (write && (!is_power_of_2(sizeof(struct page)) || memmap_on_memory)) + return -EPERM; + + return proc_do_static_key(table, write, buffer, length, ppos); +} diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index cb2bef8f9e73..b67a159027f4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -21,7 +21,9 @@ void hugetlb_vmemmap_init(struct hstate *h); */ static inline unsigned int free_vmemmap_pages_per_hpage(struct hstate *h) { - return h->nr_free_vmemmap_pages; + if (hugetlb_free_vmemmap_enabled()) + return h->nr_free_vmemmap_pages; + return 0; } #else static inline int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index c226a337c1ef..b5cc5abde05a 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -46,7 +46,7 @@ /* * memory_hotplug.memmap_on_memory parameter */ -static bool memmap_on_memory __ro_after_init; +bool memmap_on_memory __ro_after_init; #ifdef CONFIG_MHP_MEMMAP_ON_MEMORY module_param(memmap_on_memory, bool, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug");