From patchwork Fri Jun 17 13:56:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12885725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C215C433EF for ; Fri, 17 Jun 2022 13:58:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8C3476B0075; Fri, 17 Jun 2022 09:58:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 872C16B0078; Fri, 17 Jun 2022 09:58:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 73AE76B007B; Fri, 17 Jun 2022 09:58:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 64A4D6B0075 for ; Fri, 17 Jun 2022 09:58:20 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 1879735630 for ; Fri, 17 Jun 2022 13:58:20 +0000 (UTC) X-FDA: 79587882360.19.3428A7E Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf03.hostedemail.com (Postfix) with ESMTP id BBAF820082 for ; Fri, 17 Jun 2022 13:58:19 +0000 (UTC) Received: by mail-pl1-f181.google.com with SMTP id i15so3964388plr.1 for ; Fri, 17 Jun 2022 06:58:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=EMatSV1+t4qFy1NZ5GBQuBSsvAWCaC/xn6RiPVNR2iY=; b=DLgDdvgAiDYnxxGK0kizD0rVOn7tYa/8ez/L+HVhsX01j4WfdiAS7MSgpst303DfHZ L6bwt6PTV4zPEnsPg+50xx+ona2i2Z4YhG2RPzA+8j5v9tsVAUEMNYA3Yre14T6XU3pM degBA1XhN9KMMYkat8iu4BZrQ7o5IkOcQvskdrwGor6Zq+0VUBRw16zTxnzhEIcJrKPl 4rNIyaqVLioAyijg/u5Z7aYJALFk1ywKDQg91qhVk+b3uE9fjUzdfhf5YAps+gnpNfwb K4DLEfTi+hOkveRn5qccdXKv5l6xGzrZh6COVf3d6zLzpf/byrjioEGsEC3Ot3dCoAvL zg0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=EMatSV1+t4qFy1NZ5GBQuBSsvAWCaC/xn6RiPVNR2iY=; b=YCfCjYYIW8p9uqu1M7hXccqggRn3u+ziryujVCm5tfZP8H49P0fOfhfcQ7yOnUJFmc 7EAgxSJNbxnRD/GypEl0zzkZEHiXXS6iULbgmMJX1jmUjUOWSKFp3XYfjq0Xf53LqZvI oZQqjBilE3HtlC73oaY8Oq7bt2vUQJs0VY89pyP57txyetiWT1USrqj9JPROcQ+i6zUB NRiAltZ/ZyK8VxReknlI/PqwzNkcwlZx5INYxj1t4+MT4035iqmBijpkBoQ5hdGVjsaR Xuxs2T7uwrsjC57gK4bekNgagjoYFY7cGhT/G1blNx+zh7uMlttulvAvXquYoi+NDjq6 NrHg== X-Gm-Message-State: AJIora8i96ZPw/DdOzrhCLFCxSFAankyDISzDvAiwdkQ3QVvvNBY4eTc fNDeZcjySgOSTtiymVgQ3vtkCA== X-Google-Smtp-Source: AGRyM1vJNmuLAuXSWdMYIgHaMxFecDj98dF84vb69573D8juQRY+AEtOMLzRXelhvc3vm0JdgDSJ+w== X-Received: by 2002:a17:90b:164a:b0:1e8:8e0c:88dc with SMTP id il10-20020a17090b164a00b001e88e0c88dcmr10823433pjb.237.1655474298733; Fri, 17 Jun 2022 06:58:18 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015e8d4eb277sm3605869plo.193.2022.06.17.06.58.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Jun 2022 06:58:12 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, mike.kravetz@oracle.com, osalvador@suse.de, paulmck@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v3 1/2] mm: memory_hotplug: enumerate all supported section flags Date: Fri, 17 Jun 2022 21:56:49 +0800 Message-Id: <20220617135650.74901-2-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220617135650.74901-1-songmuchun@bytedance.com> References: <20220617135650.74901-1-songmuchun@bytedance.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655474299; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EMatSV1+t4qFy1NZ5GBQuBSsvAWCaC/xn6RiPVNR2iY=; b=IBjemn8VMo2pIlpanQttdKEsQJ4M5DbXUekU6kGdmVIUjejBHCqhgKTRowutKhhuowv9ZH Ndz9M/9D7FqUw+6qmu3FBBo6+JNZXtZI7bjTQoyVOWqOzRdNo2wk/7fhm6x9VPg566LPKg zoT1XYbx+mR3yFaPZLNT0IwwtmRcBOw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655474299; a=rsa-sha256; cv=none; b=UDyhQ3vlMsxmTRGT5BSBQ2KH0cTbxS/x6Uh/AAW73ITRX82S0AADIVHagv6Ka6TbCHujHp V2QAc8SOrhMzGHSMXwuRJxioG6Ix/s5Tv+i3jmdIkTpY3g6eDGEfHZ4qkGvi3qo5cDKaI0 HnGglcwSyRC2xKELgQtLCWLP2HBxd/g= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=DLgDdvgA; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Stat-Signature: w4ngshuxwuuxzkma7iko9d1f5tpesjyb X-Rspamd-Queue-Id: BBAF820082 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=DLgDdvgA; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf03.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.214.181 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam01 X-Rspam-User: X-HE-Tag: 1655474299-554887 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We are almost running out of section flags, only one bit is available in the worst case (powerpc with 256k pages). However, there are still some free bits (in ->section_mem_map) on other architectures (e.g. x86_64 has 10 bits available, arm64 has 8 bits available with worst case of 64K pages). We have hard coded those numbers in code, it is inconvenient to use those bits on other architectures except powerpc. So transfer those section flags to enumeration to make it easy to add new section flags in the future. Also, move SECTION_TAINT_ZONE_DEVICE into the scope of CONFIG_ZONE_DEVICE to save a bit on non-zone-device case. Signed-off-by: Muchun Song --- include/linux/mmzone.h | 44 +++++++++++++++++++++++++++++++++++--------- mm/memory_hotplug.c | 6 ++++++ mm/sparse.c | 2 +- 3 files changed, 42 insertions(+), 10 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index aab70355d64f..932843c6459b 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1418,16 +1418,35 @@ extern size_t mem_section_usage_size(void); * (equal SECTION_SIZE_BITS - PAGE_SHIFT), and the * worst combination is powerpc with 256k pages, * which results in PFN_SECTION_SHIFT equal 6. - * To sum it up, at least 6 bits are available. + * To sum it up, at least 6 bits are available on all architectures. + * However, we can exceed 6 bits on some other architectures except + * powerpc (e.g. 15 bits are available on x86_64, 13 bits are available + * with the worst case of 64K pages on arm64) if we make sure the + * exceeded bit is not applicable to powerpc. */ -#define SECTION_MARKED_PRESENT (1UL<<0) -#define SECTION_HAS_MEM_MAP (1UL<<1) -#define SECTION_IS_ONLINE (1UL<<2) -#define SECTION_IS_EARLY (1UL<<3) -#define SECTION_TAINT_ZONE_DEVICE (1UL<<4) -#define SECTION_MAP_LAST_BIT (1UL<<5) -#define SECTION_MAP_MASK (~(SECTION_MAP_LAST_BIT-1)) -#define SECTION_NID_SHIFT 6 +enum { + SECTION_MARKED_PRESENT_BIT, + SECTION_HAS_MEM_MAP_BIT, + SECTION_IS_ONLINE_BIT, + SECTION_IS_EARLY_BIT, +#ifdef CONFIG_ZONE_DEVICE + SECTION_TAINT_ZONE_DEVICE_BIT, +#endif + SECTION_MAP_LAST_BIT, +}; + +enum { + SECTION_MARKED_PRESENT = BIT(SECTION_MARKED_PRESENT_BIT), + SECTION_HAS_MEM_MAP = BIT(SECTION_HAS_MEM_MAP_BIT), + SECTION_IS_ONLINE = BIT(SECTION_IS_ONLINE_BIT), + SECTION_IS_EARLY = BIT(SECTION_IS_EARLY_BIT), +#ifdef CONFIG_ZONE_DEVICE + SECTION_TAINT_ZONE_DEVICE = BIT(SECTION_TAINT_ZONE_DEVICE_BIT), +#endif +}; + +#define SECTION_MAP_MASK (~(BIT(SECTION_MAP_LAST_BIT) - 1)) +#define SECTION_NID_SHIFT SECTION_MAP_LAST_BIT static inline struct page *__section_mem_map_addr(struct mem_section *section) { @@ -1466,12 +1485,19 @@ static inline int online_section(struct mem_section *section) return (section && (section->section_mem_map & SECTION_IS_ONLINE)); } +#ifdef CONFIG_ZONE_DEVICE static inline int online_device_section(struct mem_section *section) { unsigned long flags = SECTION_IS_ONLINE | SECTION_TAINT_ZONE_DEVICE; return section && ((section->section_mem_map & flags) == flags); } +#else +static inline int online_device_section(struct mem_section *section) +{ + return 0; +} +#endif static inline int online_section_nr(unsigned long nr) { diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 1f1a730c4499..6662b86e9e64 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -670,12 +670,18 @@ static void __meminit resize_pgdat_range(struct pglist_data *pgdat, unsigned lon } +#ifdef CONFIG_ZONE_DEVICE static void section_taint_zone_device(unsigned long pfn) { struct mem_section *ms = __pfn_to_section(pfn); ms->section_mem_map |= SECTION_TAINT_ZONE_DEVICE; } +#else +static inline void section_taint_zone_device(unsigned long pfn) +{ +} +#endif /* * Associate the pfn range with the given zone, initializing the memmaps diff --git a/mm/sparse.c b/mm/sparse.c index cb3bfae64036..e5a8a3a0edd7 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -281,7 +281,7 @@ static unsigned long sparse_encode_mem_map(struct page *mem_map, unsigned long p { unsigned long coded_mem_map = (unsigned long)(mem_map - (section_nr_to_pfn(pnum))); - BUILD_BUG_ON(SECTION_MAP_LAST_BIT > (1UL< PFN_SECTION_SHIFT); BUG_ON(coded_mem_map & ~SECTION_MAP_MASK); return coded_mem_map; } From patchwork Fri Jun 17 13:56:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12885726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5253AC43334 for ; Fri, 17 Jun 2022 13:58:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E90076B0078; Fri, 17 Jun 2022 09:58:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3F416B007B; Fri, 17 Jun 2022 09:58:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB9D76B007D; Fri, 17 Jun 2022 09:58:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BE8206B0078 for ; Fri, 17 Jun 2022 09:58:29 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 96E6E1205B3 for ; Fri, 17 Jun 2022 13:58:29 +0000 (UTC) X-FDA: 79587882738.14.57A738C Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf20.hostedemail.com (Postfix) with ESMTP id 3E6281C009A for ; Fri, 17 Jun 2022 13:58:29 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id y196so4250956pfb.6 for ; Fri, 17 Jun 2022 06:58:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=fMp2Yji0h8sfYM7J5lFAZRMAw6Jq1VbU0SE1jLcEqEs=; b=L9UV1ZoSzp59Hlir3y1ZhuzMspnMdC0v+qw0W3k4ueDLzl+5S9vaVODKrpeIHznJIq K5aAIMzfCf4UmTN0A+S1X1DNhzZJDrpLRCkn6OeTPAJ+MlRnOSx08x2IZ6SXBfzuA6Fo r9L9ibIqbtI9g0jSTmSEMetzGf9xEX8ckuKM4AtXtLKDSWX6gNvfnZXeb+sNIBA4e2nd bIyfSRyfdOJXZs01A84JV0ktwCle6wE218Oiy83HUPeNvBSkz2zJu1wr/e4hoKqYVB+A 2Os3NdV9t8R9bRbwrRkXYdrEkD0JlixEnevlaopo5njcz/Gv8z8GAZUpVFU47Nrj1gNW RqqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=fMp2Yji0h8sfYM7J5lFAZRMAw6Jq1VbU0SE1jLcEqEs=; b=VlcABzj4GkQJ7ROTyxFDtwu4oUSykkwDb4QQfNR5o7R3phsApW3q7qnAG2jtIrRmKM GXUgOsvneyZaCsK4Vmypx0qNoyApSHP+1DRdlj57fRvX5bPM6+hcK9+PgqyD4HXG66gV IpY8z8f4U0lmzbX0SqeWcIdCjAI0mqxB8C9mTUh96edlXpH/7TirjEKVwcf7217pDKQ3 HN/WIBHWvHMYI8bhAOAemxqgLCJr94IWw6QOMLFeqsV3MEB+I/da40n+s246ZBkcxHl+ IZmuYHgFZsq19it2MeJ1EsEvv1ow/8lrZAmJ3KyXWbgMaSmWNBATDzpiGNa0TbrxG0LT sa0Q== X-Gm-Message-State: AJIora/W9nvRpIEOBo5P7/vk4J2BmAJGhkKfLq5vC5WDjpqKo+Shn3kD zAiSoHSo6Zzr8hcafe2mQvYD/g== X-Google-Smtp-Source: AGRyM1t2u542es5M1iiHQrwjDy17In3/uqKkxNDga71rYGgextPxVInGRvHAW44WhOFK7GJq/6DliQ== X-Received: by 2002:a63:788:0:b0:3fc:6922:704 with SMTP id 130-20020a630788000000b003fc69220704mr9189754pgh.6.1655474308200; Fri, 17 Jun 2022 06:58:28 -0700 (PDT) Received: from FVFYT0MHHV2J.bytedance.net ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id e3-20020a170903240300b0015e8d4eb277sm3605869plo.193.2022.06.17.06.58.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 17 Jun 2022 06:58:27 -0700 (PDT) From: Muchun Song To: akpm@linux-foundation.org, corbet@lwn.net, david@redhat.com, mike.kravetz@oracle.com, osalvador@suse.de, paulmck@kernel.org Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, smuchun@gmail.com, Muchun Song Subject: [PATCH v3 2/2] mm: memory_hotplug: make hugetlb_optimize_vmemmap compatible with memmap_on_memory Date: Fri, 17 Jun 2022 21:56:50 +0800 Message-Id: <20220617135650.74901-3-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.1 (Apple Git-133) In-Reply-To: <20220617135650.74901-1-songmuchun@bytedance.com> References: <20220617135650.74901-1-songmuchun@bytedance.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655474309; a=rsa-sha256; cv=none; b=KR2Wsryf94sQso+V6sJNDA/ZCEJMtFAxd9ErZL2LI6/DV3fm7PZy1uMEtzijwsoIIhrD00 36LZY7xiqqjWWZMUGhYCdDEK17V7ABJBwI7pZyXLmWYwFD1btX9oAIvsuNVeRD1KdCuEsS W3U4gIP21jlYWXhkgvmsr2lFsEB9BKU= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=L9UV1ZoS; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655474309; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fMp2Yji0h8sfYM7J5lFAZRMAw6Jq1VbU0SE1jLcEqEs=; b=2ZiXfkkqiKVR+m4Kn8YtR9T+WZ5f5o9nFkMik/mX/UVIrahbfd/z4YxSbm6XvciZ+x6DAT H9+lgdjiSGrw7gH3N+4D+ldsehfXBhmtRx2edX9PAmETYuht5YkwiWRIZs3Xl7GIwhNo80 JUZU0zx9rSiHIq3WzLN2xP1w42FBfsM= X-Rspamd-Queue-Id: 3E6281C009A X-Rspam-User: Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=L9UV1ZoS; dmarc=pass (policy=none) header.from=bytedance.com; spf=pass (imf20.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com X-Rspamd-Server: rspam06 X-Stat-Signature: an3bew91dakt31mqiuu61uz4so595b37 X-HE-Tag: 1655474309-181642 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: For now, the feature of hugetlb_free_vmemmap is not compatible with the feature of memory_hotplug.memmap_on_memory, and hugetlb_free_vmemmap takes precedence over memory_hotplug.memmap_on_memory. However, someone wants to make memory_hotplug.memmap_on_memory takes precedence over hugetlb_free_vmemmap since memmap_on_memory makes it more likely to succeed memory hotplug in close-to-OOM situations. So the decision of making hugetlb_free_vmemmap take precedence is not wise and elegant. The proper approach is to have hugetlb_vmemmap.c do the check whether the section which the HugeTLB pages belong to can be optimized. If the section's vmemmap pages are allocated from the added memory block itself, hugetlb_free_vmemmap should refuse to optimize the vmemmap, otherwise, do the optimization. Then both kernel parameters are compatible. So this patch introduces VmemmapSelfHosted to mask any non-optimizable vmemmap pages. The hugetlb_vmemmap can use this flag to detect if a vmemmap page can be optimized. Signed-off-by: Muchun Song Co-developed-by: Oscar Salvador Signed-off-by: Oscar Salvador --- Documentation/admin-guide/kernel-parameters.txt | 22 ++++++------ Documentation/admin-guide/sysctl/vm.rst | 5 ++- include/linux/memory_hotplug.h | 9 ----- include/linux/page-flags.h | 11 ++++++ mm/hugetlb_vmemmap.c | 47 ++++++++++++++++++++----- mm/memory_hotplug.c | 27 ++++++-------- 6 files changed, 74 insertions(+), 47 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 8090130b544b..d740e2ed0e61 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1722,9 +1722,11 @@ Built with CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON=y, the default is on. - This is not compatible with memory_hotplug.memmap_on_memory. - If both parameters are enabled, hugetlb_free_vmemmap takes - precedence over memory_hotplug.memmap_on_memory. + Note that the vmemmap pages may be allocated from the added + memory block itself when memory_hotplug.memmap_on_memory is + enabled, those vmemmap pages cannot be optimized even if this + feature is enabled. Other vmemmap pages not allocated from + the added memory block itself do not be affected. hung_task_panic= [KNL] Should the hung task detector generate panics. @@ -3069,10 +3071,12 @@ [KNL,X86,ARM] Boolean flag to enable this feature. Format: {on | off (default)} When enabled, runtime hotplugged memory will - allocate its internal metadata (struct pages) - from the hotadded memory which will allow to - hotadd a lot of memory without requiring - additional memory to do so. + allocate its internal metadata (struct pages, + those vmemmap pages cannot be optimized even + if hugetlb_free_vmemmap is enabled) from the + hotadded memory which will allow to hotadd a + lot of memory without requiring additional + memory to do so. This feature is disabled by default because it has some implication on large (e.g. GB) allocations in some configurations (e.g. small @@ -3082,10 +3086,6 @@ Note that even when enabled, there are a few cases where the feature is not effective. - This is not compatible with hugetlb_free_vmemmap. If - both parameters are enabled, hugetlb_free_vmemmap takes - precedence over memory_hotplug.memmap_on_memory. - memtest= [KNL,X86,ARM,M68K,PPC,RISCV] Enable memtest Format: default : 0 diff --git a/Documentation/admin-guide/sysctl/vm.rst b/Documentation/admin-guide/sysctl/vm.rst index 5c9aa171a0d3..d7374a1e8ac9 100644 --- a/Documentation/admin-guide/sysctl/vm.rst +++ b/Documentation/admin-guide/sysctl/vm.rst @@ -565,9 +565,8 @@ See Documentation/admin-guide/mm/hugetlbpage.rst hugetlb_optimize_vmemmap ======================== -This knob is not available when memory_hotplug.memmap_on_memory (kernel parameter) -is configured or the size of 'struct page' (a structure defined in -include/linux/mm_types.h) is not power of two (an unusual system config could +This knob is not available when the size of 'struct page' (a structure defined +in include/linux/mm_types.h) is not power of two (an unusual system config could result in this). Enable (set to 1) or disable (set to 0) the feature of optimizing vmemmap pages diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 20d7edf62a6a..e0b2209ab71c 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -351,13 +351,4 @@ void arch_remove_linear_mapping(u64 start, u64 size); extern bool mhp_supports_memmap_on_memory(unsigned long size); #endif /* CONFIG_MEMORY_HOTPLUG */ -#ifdef CONFIG_MHP_MEMMAP_ON_MEMORY -bool mhp_memmap_on_memory(void); -#else -static inline bool mhp_memmap_on_memory(void) -{ - return false; -} -#endif - #endif /* __LINUX_MEMORY_HOTPLUG_H */ diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index e66f7aa3191d..2aa5dcbfe468 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -193,6 +193,11 @@ enum pageflags { /* Only valid for buddy pages. Used to track pages that are reported */ PG_reported = PG_uptodate, + +#ifdef CONFIG_MEMORY_HOTPLUG + /* For self-hosted memmap pages */ + PG_vmemmap_self_hosted = PG_owner_priv_1, +#endif }; #define PAGEFLAGS_MASK ((1UL << NR_PAGEFLAGS) - 1) @@ -628,6 +633,12 @@ PAGEFLAG_FALSE(SkipKASanPoison, skip_kasan_poison) */ __PAGEFLAG(Reported, reported, PF_NO_COMPOUND) +#ifdef CONFIG_MEMORY_HOTPLUG +PAGEFLAG(VmemmapSelfHosted, vmemmap_self_hosted, PF_ANY) +#else +PAGEFLAG_FALSE(VmemmapSelfHosted, vmemmap_self_hosted) +#endif + /* * On an anonymous page mapped into a user virtual memory area, * page->mapping points to its anon_vma, not to a struct address_space; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 1089ea8a9c98..982f27c24a95 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -10,7 +10,6 @@ */ #define pr_fmt(fmt) "HugeTLB: " fmt -#include #include "hugetlb_vmemmap.h" /* @@ -97,18 +96,50 @@ int hugetlb_vmemmap_alloc(struct hstate *h, struct page *head) return ret; } +static unsigned int vmemmap_optimizable_pages(struct hstate *h, + struct page *head) +{ + struct mem_section *ms; + struct page *vmemmap_page; + unsigned long pfn = page_to_pfn(head); + + if (READ_ONCE(vmemmap_optimize_mode) == VMEMMAP_OPTIMIZE_OFF) + return 0; + + ms = __pfn_to_section(pfn); + vmemmap_page = sparse_decode_mem_map(ms->section_mem_map, + pfn_to_section_nr(pfn)); + /* + * Only the vmemmap pages' vmemmap may be marked as VmemmapSelfHosted. + * + * Due to HugeTLB alignment requirements, and the vmemmap pages being + * at the start of the hotplugged memory region. Checking any vmemmap + * page's vmemmap is fine. + * + * [ hotplugged memory ] + * [ vmemmap ][ usable memory ] + * ^ | | | + * +---+ | | + * ^ | | + * +--------+ | + * ^ | + * +-----------------+ + */ + if (PageVmemmapSelfHosted(vmemmap_page)) + return 0; + + return hugetlb_optimize_vmemmap_pages(h); +} + void hugetlb_vmemmap_free(struct hstate *h, struct page *head) { unsigned long vmemmap_addr = (unsigned long)head; unsigned long vmemmap_end, vmemmap_reuse, vmemmap_pages; - vmemmap_pages = hugetlb_optimize_vmemmap_pages(h); + vmemmap_pages = vmemmap_optimizable_pages(h, head); if (!vmemmap_pages) return; - if (READ_ONCE(vmemmap_optimize_mode) == VMEMMAP_OPTIMIZE_OFF) - return; - static_branch_inc(&hugetlb_optimize_vmemmap_key); vmemmap_addr += RESERVE_VMEMMAP_SIZE; @@ -199,10 +230,10 @@ static struct ctl_table hugetlb_vmemmap_sysctls[] = { static __init int hugetlb_vmemmap_sysctls_init(void) { /* - * If "memory_hotplug.memmap_on_memory" is enabled or "struct page" - * crosses page boundaries, the vmemmap pages cannot be optimized. + * If "struct page" crosses page boundaries, the vmemmap pages cannot + * be optimized. */ - if (!mhp_memmap_on_memory() && is_power_of_2(sizeof(struct page))) + if (is_power_of_2(sizeof(struct page))) register_sysctl_init("vm", hugetlb_vmemmap_sysctls); return 0; diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 6662b86e9e64..3a59d4e97c03 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -43,30 +43,22 @@ #include "shuffle.h" #ifdef CONFIG_MHP_MEMMAP_ON_MEMORY -static int memmap_on_memory_set(const char *val, const struct kernel_param *kp) -{ - if (hugetlb_optimize_vmemmap_enabled()) - return 0; - return param_set_bool(val, kp); -} - -static const struct kernel_param_ops memmap_on_memory_ops = { - .flags = KERNEL_PARAM_OPS_FL_NOARG, - .set = memmap_on_memory_set, - .get = param_get_bool, -}; - /* * memory_hotplug.memmap_on_memory parameter */ static bool memmap_on_memory __ro_after_init; -module_param_cb(memmap_on_memory, &memmap_on_memory_ops, &memmap_on_memory, 0444); +module_param(memmap_on_memory, bool, 0444); MODULE_PARM_DESC(memmap_on_memory, "Enable memmap on memory for memory hotplug"); -bool mhp_memmap_on_memory(void) +static inline bool mhp_memmap_on_memory(void) { return memmap_on_memory; } +#else +static inline bool mhp_memmap_on_memory(void) +{ + return false; +} #endif enum { @@ -1035,7 +1027,7 @@ int mhp_init_memmap_on_memory(unsigned long pfn, unsigned long nr_pages, struct zone *zone) { unsigned long end_pfn = pfn + nr_pages; - int ret; + int ret, i; ret = kasan_add_zero_shadow(__va(PFN_PHYS(pfn)), PFN_PHYS(nr_pages)); if (ret) @@ -1043,6 +1035,9 @@ int mhp_init_memmap_on_memory(unsigned long pfn, unsigned long nr_pages, move_pfn_range_to_zone(zone, pfn, nr_pages, NULL, MIGRATE_UNMOVABLE); + for (i = 0; i < nr_pages; i++) + SetPageVmemmapSelfHosted(pfn_to_page(pfn + i)); + /* * It might be that the vmemmap_pages fully span sections. If that is * the case, mark those sections online here as otherwise they will be