From patchwork Mon Feb 28 07:10:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12762466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92C7FC433EF for ; Mon, 28 Feb 2022 07:11:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 32A4E8D0005; Mon, 28 Feb 2022 02:11:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D97E8D0001; Mon, 28 Feb 2022 02:11:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A0EB8D0005; Mon, 28 Feb 2022 02:11:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0073.hostedemail.com [216.40.44.73]) by kanga.kvack.org (Postfix) with ESMTP id 0B04F8D0001 for ; Mon, 28 Feb 2022 02:11:33 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id BDC598249980 for ; Mon, 28 Feb 2022 07:11:32 +0000 (UTC) X-FDA: 79191318024.19.7FE85CA Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf29.hostedemail.com (Postfix) with ESMTP id 479AD120004 for ; Mon, 28 Feb 2022 07:11:32 +0000 (UTC) Received: by mail-pl1-f173.google.com with SMTP id p17so9912811plo.9 for ; Sun, 27 Feb 2022 23:11:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oqE92Z30jLRB5qj639p7ReXjqJAONBgi3Xr8rqugwWc=; b=e2F/6vYuG7H/7yU5OibJdQw/ZPCcUdpXzvY1YCsl6iX6fv1pAnjbnW2Mjddjky6LK9 uzjUZZZJBhd2peqU+T3t4mdTPJv5YZaODOD3iKFHa6tL7PTAKR8rzj0atcTRNL0EMx/b NbZzF1mPZkhDn9fxn/XJ8FzqQbVEApgBQi+jQ8HqKqg0dmNOFHHpLCLs7R09wtLpqTNm RjIVCwzjknwpWcPrrul95klarohrfak/wSmWb1LisWOtHrPyj/27/KHtUnQhdxvdhF6u +fPqkPBrOhlguOPAehK38KtyA/LpERXHqg7GVrbY6tHr+WXEbixjZ3Hey8zQvbEjkJht DAXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oqE92Z30jLRB5qj639p7ReXjqJAONBgi3Xr8rqugwWc=; b=TbFQzuHj3MPz2ThycfS72qSE5/hYNOxDNiAP5fcccGdwDsJq8kcCKbPJSBxUt632y5 Wa3zIcmlJlf8tKP/GAnhNpPMzoQjMrA+TyM3sbsTKFqiWG1NzgQZgOA0gmPy1t7Y7+xO vWzjlcsbfxhH2voRuVZBra0pgRg82lU+M6vG4sQTC8sNl4fWQAx0mJTLKIj1iUQEGSMg /Dc4myK2cglNidiebL73N6kmtyKMdeXTME1yLKpW3ByTQz2ajRa+yydBuEk7W4uu/vk1 2zrAG1lELS3/8scBUOg6ixAL0gRNebje5jq+ltmHDTThGhGN3LvLUZrEV0HuGIa3i5eq Kl2Q== X-Gm-Message-State: AOAM532y+orUxS8O5khPWZz+pYaTNM4tE3NKWn8OO1JZssaQqMy6gPuW sWRwmKTEV/v/sg0ChVfaaYptsA== X-Google-Smtp-Source: ABdhPJzSxBRm2Jc+FR3haOpBiZvOJNWnMLPpmqPSQ2uOi05bt+bgbNRdMNtr3yqqrTJeLgu2AeKf1g== X-Received: by 2002:a17:902:da8a:b0:151:6240:b877 with SMTP id j10-20020a170902da8a00b001516240b877mr5078007plx.22.1646032291347; Sun, 27 Feb 2022 23:11:31 -0800 (PST) Received: from FVFYT0MHHV2J.tiktokcdn.com ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id i7-20020a17090a65c700b001b936b8abe0sm16240390pjs.7.2022.02.27.23.11.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Feb 2022 23:11:31 -0800 (PST) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH 1/3] mm: hugetlb: disable freeing vmemmap pages when struct page crosses page boundaries Date: Mon, 28 Feb 2022 15:10:20 +0800 Message-Id: <20220228071022.26143-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220228071022.26143-1-songmuchun@bytedance.com> References: <20220228071022.26143-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 479AD120004 X-Stat-Signature: scy99tsfupwuk416ay9jqnd1tmg9994r Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b="e2F/6vYu"; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf29.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.173 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-HE-Tag: 1646032292-500465 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON is enabled and the size of "struct page" is not power of two, we cannot optimize vmemmap pages of HugeTLB pages. We should disable this feature in this case. Signed-off-by: Muchun Song --- mm/hugetlb_vmemmap.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b3118dba0518..836d1117f08b 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -121,6 +121,17 @@ void __init hugetlb_vmemmap_init(struct hstate *h) if (!hugetlb_free_vmemmap_enabled()) return; + if (IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON) && + !is_power_of_2(sizeof(struct page))) { + /* + * The hugetlb_free_vmemmap_enabled_key can be enabled when + * CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON. It should + * be disabled if "struct page" crosses page boundaries. + */ + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); + return; + } + vmemmap_pages = (nr_pages * sizeof(struct page)) >> PAGE_SHIFT; /* * The head page is not to be freed to buddy allocator, the other tail From patchwork Mon Feb 28 07:10:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12762467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15F7EC433EF for ; Mon, 28 Feb 2022 07:11:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B9718D0006; Mon, 28 Feb 2022 02:11:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 967DA8D0001; Mon, 28 Feb 2022 02:11:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8578A8D0006; Mon, 28 Feb 2022 02:11:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id 779FB8D0001 for ; Mon, 28 Feb 2022 02:11:37 -0500 (EST) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 325438249980 for ; Mon, 28 Feb 2022 07:11:37 +0000 (UTC) X-FDA: 79191318234.17.527955B Received: from mail-pg1-f182.google.com (mail-pg1-f182.google.com [209.85.215.182]) by imf10.hostedemail.com (Postfix) with ESMTP id A32D1C0005 for ; Mon, 28 Feb 2022 07:11:36 +0000 (UTC) Received: by mail-pg1-f182.google.com with SMTP id o23so10618169pgk.13 for ; Sun, 27 Feb 2022 23:11:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6H6PLOlODlELuUrXrCLYoGeXhxxcGUB9kEy/fShHjn4=; b=yuJj80QeewJ2+adT1VdgjIVhQ6FRG5fnHsSu0frIUV2+jgz5s32ulViVic7VWe/w8B JbHJ3/v5Ykyq1Zju+8bFuuNvbWHQM2PGRjEmGZMFPBzzBpLDX1lsmlk223md9Xzl9mFd zV68wuqOqLAtIXKFGOZD4dFP2SM1fxjX/6Y8HiVq+IhJwRGVCcTJfP8QHOX7m076OAQE JA6/AgeQ0Pea2J3StVRYnW6zkowHDcMIi2W0ApNiSCZXyTKBh82GLmYttvWAioktT0g7 eRKw3nD0jNANDcoAi5rFrYM2UYMMRZ/0wnXqLXjUDLqn+PEZ1fplx7I9eFCxIDFv4EOW 89eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6H6PLOlODlELuUrXrCLYoGeXhxxcGUB9kEy/fShHjn4=; b=x4YU3ZaFkaAO/aTuJud0a8Vm+k2a9PtaN/WwPvMjV2s/IyPCdCoZTHXcLbhEK/ADu/ evKUBiNWaNlWXAmFAEaTMv1a1qyVWlvZrUqbigu0+VQNTndiBgEjheUJy2e7NGsIsKDE NQFt7T9yU4S8x+MGFo9G2rvFch0zc9VSfrA2oZW+0meZxHPs3dBNO/M5ulLHQkZlH0gz F4ywYv2pktY0PdK+ePvCcjOIV4fzWk1MU6nVAvAXKb+pxnWufHvsmkahuxvA450BR/Mw /OSwCaovDTWZCcTb+3T9H+pz5VmKWztQrS8/+7VA4OvyL/y85SD/EQFf55+PPGGuoWS/ NldQ== X-Gm-Message-State: AOAM533q/h9aiWvObnomIiabibJo6o+o32oN7M+YWtVU5pzFYwbvtE1O nNNam0jxhpXgeHj5/DgWmshBtg== X-Google-Smtp-Source: ABdhPJwvXcp5CL86BH67oNCcmHurbmKiXoJXFoBN4tPzC2nolXf9EU8whvLTaQsnxZ9c3gxr0Ef0LQ== X-Received: by 2002:a63:2204:0:b0:378:9f08:206d with SMTP id i4-20020a632204000000b003789f08206dmr3409181pgi.3.1646032295754; Sun, 27 Feb 2022 23:11:35 -0800 (PST) Received: from FVFYT0MHHV2J.tiktokcdn.com ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id i7-20020a17090a65c700b001b936b8abe0sm16240390pjs.7.2022.02.27.23.11.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Feb 2022 23:11:35 -0800 (PST) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH 2/3] sysctl: allow to set extra1 to SYSCTL_ONE Date: Mon, 28 Feb 2022 15:10:21 +0800 Message-Id: <20220228071022.26143-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220228071022.26143-1-songmuchun@bytedance.com> References: <20220228071022.26143-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: A32D1C0005 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=yuJj80Qe; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf10.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.215.182 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: ztnfmgpr8n55e5bja4k83m7zct3oy9oa X-HE-Tag: 1646032296-797861 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some sysctls only allow to be enabled and cannot be set back to be disabled. But proc_do_static_key() does not consider this situation, which set ->extra1 to SYSCTL_ZERO unconditionally. This patch add the ability to set ->extra1 to SYSCTL_ONE, which will be used in the next patch. Signed-off-by: Muchun Song --- kernel/sysctl.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/sysctl.c b/kernel/sysctl.c index 64065abf361e..ab3e9c937268 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1631,7 +1631,7 @@ int proc_do_static_key(struct ctl_table *table, int write, .data = &val, .maxlen = sizeof(val), .mode = table->mode, - .extra1 = SYSCTL_ZERO, + .extra1 = table->extra1 == SYSCTL_ONE ? SYSCTL_ONE : SYSCTL_ZERO, .extra2 = SYSCTL_ONE, }; From patchwork Mon Feb 28 07:10:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12762468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CFCD1C433EF for ; Mon, 28 Feb 2022 07:11:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6D24B8D0007; Mon, 28 Feb 2022 02:11:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 682398D0001; Mon, 28 Feb 2022 02:11:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 524D68D0007; Mon, 28 Feb 2022 02:11:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.28]) by kanga.kvack.org (Postfix) with ESMTP id 428508D0001 for ; Mon, 28 Feb 2022 02:11:42 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 0CE48518 for ; Mon, 28 Feb 2022 07:11:42 +0000 (UTC) X-FDA: 79191318444.13.7F4B491 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf30.hostedemail.com (Postfix) with ESMTP id 7EA5180004 for ; Mon, 28 Feb 2022 07:11:41 +0000 (UTC) Received: by mail-pl1-f169.google.com with SMTP id l9so9485886pls.6 for ; Sun, 27 Feb 2022 23:11:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/8ft55r/VTfoT4ZinxhB51u9HRB18drBxk6cMGVkRIc=; b=QPUd6SMtlL0lgH1f6RussrMAtI4S+e2NVZ4z1/cLuYCljBgasiYx6HN63o1FJmT3mB EyERA9+htOPxv14o3q8QvwJsmjpHWrbsY4D9/F3IqZQVtTdSQjgOHYIoizN+awrObS+b o2mqmQMWwaP9vT80WgYs90x14FldeS7DT225M0TViqTER9xsB49CZ19VLK0rtgs7bYAO 880eTRkSqRu/r7E1L3noMD1KcmGHufV9nWspTVwy6deuZwsaBvPqkRSSCBZhRkjKyLnq QSWEj7WLbh8fbWTAI0GZPeBUprgYQt0jZeV+T307CtT3eVTyVwzCq9i4XXeVZ3jIhskr BeCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/8ft55r/VTfoT4ZinxhB51u9HRB18drBxk6cMGVkRIc=; b=fTJ58Zv4f0xaKsrPtgQ3anSHg+F4012zQXA+CY9UTy0+oUx2JWvWr6Uup+EMv/9bjg J4osmLbB9w60pp/X4HqfrCjrV+WA3QPchwiJeYFyqS641GwxrwyZqYV+onE+7kaGT6F6 JtKjA+HDFfm+N/YKIlVmGOJNwCUt09Y6LZvPKGBuuXVpqWm7SAQUFYV021Ua3GgMQRCt BOnAdxpFzSmd492SuBgBkPGGxFLtu6xqhShVA1s3YaJqcK6Xvx2avmy0KBDyVbhJdFh2 2a2dluLDx8Pio5xSJBGZaZYn5YFtMvNdktkCSjC3B61Vi1mm5QFdZhFWYS+NO0nK3/tn FqFg== X-Gm-Message-State: AOAM531Yk2pCxn8J4CXCHMhTsoOpiZFyhoritTjbiXagm/QdqQc4zyPm H40N3lZ2fc1le/wEOnRTwp4ATg== X-Google-Smtp-Source: ABdhPJy8a01fRIV4nGG4Fscsyj42sS294OeFwsAc/3btG49ySTvEgsDMP+HVgRNeNNfgTB6q6/T4XA== X-Received: by 2002:a17:90a:5995:b0:1bc:be7f:f5a1 with SMTP id l21-20020a17090a599500b001bcbe7ff5a1mr15573946pji.84.1646032300573; Sun, 27 Feb 2022 23:11:40 -0800 (PST) Received: from FVFYT0MHHV2J.tiktokcdn.com ([139.177.225.243]) by smtp.gmail.com with ESMTPSA id i7-20020a17090a65c700b001b936b8abe0sm16240390pjs.7.2022.02.27.23.11.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 27 Feb 2022 23:11:40 -0800 (PST) From: Muchun Song To: corbet@lwn.net, mike.kravetz@oracle.com, akpm@linux-foundation.org, mcgrof@kernel.org, keescook@chromium.org, yzaikin@google.com Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH 3/3] mm: hugetlb: add hugetlb_free_vmemmap sysctl Date: Mon, 28 Feb 2022 15:10:22 +0800 Message-Id: <20220228071022.26143-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220228071022.26143-1-songmuchun@bytedance.com> References: <20220228071022.26143-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7EA5180004 X-Stat-Signature: xzyft6655ykfk4ink551epcwnj5msyk6 X-Rspam-User: Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=QPUd6SMt; spf=pass (imf30.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Rspamd-Server: rspam07 X-HE-Tag: 1646032301-162169 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We must add "hugetlb_free_vmemmap=on" to boot cmdline and reboot the server to enable the feature of freeing vmemmap pages of HugeTLB pages. Rebooting usually taske a long time. Add a sysctl to enable the feature at runtime and do not need to reboot. Signed-off-by: Muchun Song Reported-by: kernel test robot Reported-by: kernel test robot --- Documentation/admin-guide/sysctl/vm.rst | 13 +++++++++++++ include/linux/hugetlb.h | 5 +++++ include/linux/memory_hotplug.h | 1 + kernel/sysctl.c | 11 +++++++++++ mm/hugetlb_vmemmap.c | 23 +++++++++++++++++------ mm/hugetlb_vmemmap.h | 4 +++- mm/memory_hotplug.c | 2 +- 7 files changed, 51 insertions(+), 8 deletions(-) diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index f4804ce37c58..01f18e6cc227 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -561,6 +561,19 @@ Change the minimum size of the hugepage pool. See Documentation/admin-guide/mm/hugetlbpage.rst +hugetlb_free_vmemmap +==================== + +A toggle value indicating if vmemmap pages are allowed to be optimized. +If it is off (0), then it can be set true (1). Once true, the vmemmap +pages associated with each HugeTLB page will be optimized, and the toggle +cannot be set back to false. It only optimizes the subsequent allocation +of HugeTLB pages from buddy system, while already allocated HugeTLB pages +will not be optimized. + +See Documentation/admin-guide/mm/hugetlbpage.rst + + nr_hugepages_mempolicy ====================== diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 53c1b6082a4c..cc4ab21892f5 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -1080,6 +1080,11 @@ static inline void set_huge_swap_pte_at(struct mm_struct *mm, unsigned long addr } #endif /* CONFIG_HUGETLB_PAGE */ +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP +int hugetlb_vmemmap_sysctl_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, loff_t *ppos); +#endif + static inline spinlock_t *huge_pte_lock(struct hstate *h, struct mm_struct *mm, pte_t *pte) { diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index e0b2209ab71c..b30f9fdaed73 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -349,6 +349,7 @@ extern int arch_create_linear_mapping(int nid, u64 start, u64 size, struct mhp_params *params); void arch_remove_linear_mapping(u64 start, u64 size); extern bool mhp_supports_memmap_on_memory(unsigned long size); +extern bool memmap_on_memory; #endif /* CONFIG_MEMORY_HOTPLUG */ #endif /* __LINUX_MEMORY_HOTPLUG_H */ diff --git a/kernel/sysctl.c b/kernel/sysctl.c index ab3e9c937268..77f039849b2a 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -2223,6 +2223,17 @@ static struct ctl_table vm_table[] = { .mode = 0644, .proc_handler = hugetlb_sysctl_handler, }, +#ifdef CONFIG_HUGETLB_PAGE_FREE_VMEMMAP + { + .procname = "hugetlb_free_vmemmap", + .data = &hugetlb_free_vmemmap_enabled_key.key, + .maxlen = sizeof(hugetlb_free_vmemmap_enabled_key.key), + .mode = 0644, + /* only handle a transition from default "0" to "1" */ + .proc_handler = hugetlb_vmemmap_sysctl_handler, + .extra1 = SYSCTL_ONE, + }, +#endif #ifdef CONFIG_NUMA { .procname = "nr_hugepages_mempolicy", diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 836d1117f08b..3167021055d6 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -10,6 +10,7 @@ #define pr_fmt(fmt) "HugeTLB: " fmt +#include #include "hugetlb_vmemmap.h" /* @@ -118,17 +119,14 @@ void __init hugetlb_vmemmap_init(struct hstate *h) BUILD_BUG_ON(__NR_USED_SUBPAGE >= RESERVE_VMEMMAP_SIZE / sizeof(struct page)); - if (!hugetlb_free_vmemmap_enabled()) - return; - - if (IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON) && - !is_power_of_2(sizeof(struct page))) { + if (!is_power_of_2(sizeof(struct page))) { /* * The hugetlb_free_vmemmap_enabled_key can be enabled when * CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON. It should * be disabled if "struct page" crosses page boundaries. */ - static_branch_disable(&hugetlb_free_vmemmap_enabled_key); + if (IS_ENABLED(CONFIG_HUGETLB_PAGE_FREE_VMEMMAP_DEFAULT_ON)) + static_branch_disable(&hugetlb_free_vmemmap_enabled_key); return; } @@ -147,3 +145,16 @@ void __init hugetlb_vmemmap_init(struct hstate *h) pr_info("can free %d vmemmap pages for %s\n", h->nr_free_vmemmap_pages, h->name); } + +int hugetlb_vmemmap_sysctl_handler(struct ctl_table *table, int write, + void *buffer, size_t *length, loff_t *ppos) +{ + /* + * The vmemmap pages cannot be optimized if a "struct page" crosses page + * boundaries or memory_hotplug.memmap_on_memory is enabled. + */ + if (write && (!is_power_of_2(sizeof(struct page)) || memmap_on_memory)) + return -EPERM; + + return proc_do_static_key(table, write, buffer, length, ppos); +} diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index cb2bef8f9e73..b67a159027f4 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -21,7 +21,9 @@ void hugetlb_vmemmap_init(struct hstate *h); */ static inline unsigned int free_vmemmap_pages_per_hpage(struct hstate *h) { - return h->nr_free_vmemmap_pages; + if (hugetlb_free_vmemmap_enabled()) + return h->nr_free_vmemmap_pages; + return 0; } #else static inline int alloc_huge_page_vmemmap(struct hstate *h, struct page *head) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index c226a337c1ef..b5cc5abde05a 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -46,7 +46,7 @@ /* * memory_hotplug.memmap_on_memory parameter */ -static bool memmap_on_memory __ro_after_init; +bool memmap_on_memory __ro_after_init; #ifdef CONFIG_MHP_MEMMAP_ON_MEMORY module_param(memmap_on_memory, bool, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug");