From patchwork Tue Feb 18 18:16:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13980399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA3B8C021AA for ; Tue, 18 Feb 2025 18:17:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5AE4A280169; Tue, 18 Feb 2025 13:17:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5122128015C; Tue, 18 Feb 2025 13:17:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B36D280169; Tue, 18 Feb 2025 13:17:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1898A28015C for ; Tue, 18 Feb 2025 13:17:11 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 97397803CA for ; Tue, 18 Feb 2025 18:17:10 +0000 (UTC) X-FDA: 83133872220.20.401D051 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf13.hostedemail.com (Postfix) with ESMTP id E9D9B20004 for ; Tue, 18 Feb 2025 18:17:07 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XfjyW7Sf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3os60ZwQKCFc4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3os60ZwQKCFc4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739902628; a=rsa-sha256; cv=none; b=kHfNcScA1A3AZlYBczW/O5Vuf3Eo89kyWDjG3+1BWx/AG5mQy5Jg0jTNRtJPJCE5ZK4kTI T6FquFKK2UBFSncms8/YpzmVmeN+vc68ssPZU4UvZypA9dIhOdZyKz+rd1v+VONJEw41gU nninEE0ELGKXiQlWNQgiFyRLoJoY1u8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XfjyW7Sf; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3os60ZwQKCFc4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3os60ZwQKCFc4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739902628; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=jisVN5bTJe7Fi8QsBTJqEyml9eIdvwz50hIBvYWlrsk=; b=Zctl7m9zF7qlCjTJSOObcD8OWT5bCvrhaqF9h9z3l7i+SKULR/CXvHcnwt76rgiWi24vuA M8II7CUA6oZHcJteTxJwFxVQzhW1peF9E4oiNi2j6mJW0fLtkw4zvxLRhvz+bFeeglFO8g K54S4h3KRx4PuGCm/olKfB5zBN3b8tw= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2fc3e239675so11691857a91.0 for ; Tue, 18 Feb 2025 10:17:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1739902626; x=1740507426; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=jisVN5bTJe7Fi8QsBTJqEyml9eIdvwz50hIBvYWlrsk=; b=XfjyW7Sf9+bSAbgRg3abxqdnxnLeV3hAfHvy2ZRV2zO69zyJWnWYEbCIQaqzGYbYST SoSMgTUmvhEUZgaJYWHCd3YasKVGan19UPQ5y48nBzhmoFnJ0/Bq8ZwIofUyX/ANij3q HJSh/wpgehLPF2f22+SUhyV9Gf4Eut8npLSyzTJTNR0nrpLx2vrfbCeqaoxM2rIffnVi ZID9BcKKz5TAzi5KzWGfw8DfD+jz9Jd5Z7s9zY4l7PoDgVB07o7iaq+gsSBsJMWrlHX3 GjTue15altU3fFRBB75gEPmcHkuN/WZhBNfCumIIqQbqY7GRBsWcHhTZPXXYd+LY7Y1P tgEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739902626; x=1740507426; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=jisVN5bTJe7Fi8QsBTJqEyml9eIdvwz50hIBvYWlrsk=; b=fuzh+23d2RX7juQnNDPMYTWL0GeLTQj0+zL+QQKjGDL0rqeenJZ6h9jYOz3opys6Xb xviPxPUVBSiaNEQrpnnY/P1cQIViUr8onh7+l9qXUJchQBHDpYHZdo5p5evBM16l33W0 spX5iixssrmk34JM0l2tp6AvVPqa3zkJEm/AS+An4nyFukvou2ZZ4JEtGDNHcfgCsVAu qX8mP0OmpNmztHUeTMfFU6IHAGErdhDaCsUBQceTyi+VkBS8y7X8uffEPABimg6hY2BW 1YSzhuAZyU5RlOn2BAlaQqtxxyCs3RubZYhz+/eoRBq6KLNKNCbYJj118JHyLOGJtCto J4ww== X-Forwarded-Encrypted: i=1; AJvYcCWbehIKo/3i6uFMqTBlFMqEZoBEkBQ3vKLp7SYaO+SCK8t6Mq5aI0f+bap9tEm7RJJ1RjIWNVuTQQ==@kvack.org X-Gm-Message-State: AOJu0YwtQdwcIA8gizvx3XLQall49U7Xce+QJfSFi4ee+tmRc6K6APfD 0Ua+zACsy81q/6j/w5IftE5CbreaA0ZTy2paiodC5Rj5A1HMTAQlUo8bURvTF1uL+LPFmQ== X-Google-Smtp-Source: AGHT+IGaLH0jnj/yKNp2m9q9TkjCOe3LzVl4lYPJZ/pJ2hRpG0FKvat+KHpmOpia+azxtVdQ4BRqLybW X-Received: from pfbce6.prod.google.com ([2002:a05:6a00:2a06:b0:730:8e17:ed06]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:778a:b0:1ee:c093:e235 with SMTP id adf61e73a8af0-1eec093e4f5mr7917342637.41.1739902626668; Tue, 18 Feb 2025 10:17:06 -0800 (PST) Date: Tue, 18 Feb 2025 18:16:28 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog Message-ID: <20250218181656.207178-1-fvdl@google.com> Subject: [PATCH v4 00/27] hugetlb/CMA improvements for large systems From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Queue-Id: E9D9B20004 X-Rspamd-Server: rspam12 X-Stat-Signature: 1dtgcsbn438z5d47hmpzd1s3761icb7s X-HE-Tag: 1739902627-252740 X-HE-Meta: U2FsdGVkX18KltiXmjkhJ/Mgr4fWWq/FpiIJXlMmWr6SuuC9Ooc8qrqFSxMyXr9n+A8s7W8DzgE7tK/9WQ5P9TTzZJknOF0vG1pLtKz7DqUEg4AsbNeQ0QwOR5JvQGxeHD2NRLDj4vWU/pRlACbLlrGYAB5TKAxkVzzp7E7/ni24YUEnt4SS36GiUT5YZUDuXAzB+iMncaeiJ0QFE8LUVLOxp3uNraSLJgdsZmVRJvJCRfBwuzTRvDNaqLRRLndVmAXMJHeB8+10Cu5ZMJLeW0Gq3iocTxppDrUfbmtKFH+z2wn0RRh30/Bk1CHA0/pnMaags1+fLyzjzXsb0z48bS7Uqakoyyg2v6/GZ17WMmilXpIQklI3Z0zPlUn06BBR39dP7n5PdQJmInk3z/HG8jjdY1Kc55qcE8+4M/EQ1CdnS98SWYy2vYohe8jGIsArMsEN+JlwmgvUQ+Ph5xBiidNnCgUjop6y7qDDL4UbKqieb9ZRkUwFjoR8jZpEbIleHG2h5fnjtLNNbMaFyu4Doq45YdXFqd3wcWLn4uf6SKGUblKSt2ZcrAUTSiaDE6wv2R15wSs75u/N7Ea2SzCpX7fcE2w166uIjWYyZVYW4UQvsIlyPk/xGDbTa3cR9RssiyQOTnsLu67FuLB2GNeduSW4wuN7MO8B5bKBb2hN9aqzTSTqQtLbep1jspOgjAPOE32MS9O8YGrBlYkOHJiIruodL+v8s/EAUQPwJw8laWyLTDAv6Ooe/mntYDJDuN9zvhHTn9wpMuGyFxs4CFM7ujraGL7eRe2oF6cb/JCj3u3h6vZKGk9BEzvMssqrN6LOQoveQbYOm7CqIFMOxzhRp3KfINtTZ085mGYTuIjejDLXSAqkREnMYhg5wh43p/qV/nJ4Jxha9TSyaR3ss8Bie2UvPoowDU0XcX8byKYfChtWg90MMFWhY3+W4wZCuZE2ODkv7g7jnV0XgJ2PfGA YNbrmwOw AlD14d6CVPNxq3BNSh9HVyVpM9nRAoX2Q0uznJt8PmgPLuidyL0qS3J2uAeWKI4NbGXFtmLSON+tbIT1oehyELp223HBYWi5mhD7EgP5f+Qj5k/QsgsJgrxzsDD4P8QAsjxUA10U6FH8ssJgoknXyOWvsVNML6o4BMafzS7ORTiQozWoaEpMxqmokG6pBDsACml8hmzCk4/tZqA0F8dQ3Y7Si7PyaTPtw28g7Q4nLfG4VmFlhjtKtjLbmwDrwJCI84SG+dYJ7Y+1HudN/peMHHJ3PkvR8HObFM+55+xZDVdSS4EyQWOsbd16v1JLYC4nDvZIbwH+F4vIkBDSX0aQk1eNXKg+3NAPrUQ4612GEJoUvrsmW5q+3SlPHboZ4SO1EG+iO2yRHQXt9/QS9a3z5ZZF9kS8SFmBFbS0lqyFpdvsyDPL6zbrvC3XHm2cXG5kJ81NbSAuzPq1MYNf/fs43i5SdvhXPob+aliI5N+kCqwMPLScvv/wEeqIkGahB4ICtyt091MwcZ17GDzser/7b9AF/s/k+pfUcJjRfdpUSyCh9l31oh3aC4lZ0hoSxVR9WpAvNnyn8MFWl4PTARZFjdp6wqCnkAohQ4Y5cipupmiwfKvV/mVAsIRqAT9xDUfNibTTqv9NHdJgj8tw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: v4: * This series assumes that the following patch has been applied: https://lore.kernel.org/linux-mm/1f9f5fb4-27cd-4981-aa60-789a33376598@redhat.com/ * Drop memblock alloc loop patch that wasn't needed. * Update documentation to reflect that hugepagesz and hugepages are now early parameters. * Fix uninitialized var warning (from Dan Carpenter ) * In the case of node fallback by memblock_alloc_try_nid_raw, make sure to get the actual NUMA node it was allocated from and put it on the list for that node, as this is what the pre-HVO code needs. * Moved some empty static inline function decls to the right patch. * Fix incorrect memory stats for hugetlb_cma= combined with hugepages= v3: * Fix SPDX comment include file format. * Add new hugetlb_cma.* files to MAINTAINERS * Document new ranges/ subdir in CMA debugfs. * Fix powerpc compilation for config without HAVE_BOOTMEM_INFO_NODE * Fix various other nits found by kernel test robot. * Use a PFN value of -1 to indicate a non-mirrored mapping in sparse-vmemmap.c, not 0. * Fix incorrect if() statement that got mangled in cma.c v2: * Add missing CMA debugfs code. * Minor cleanups in hugetlb_cma changes. * Move hugetlb_cma code to its own file to further clean things up. On large systems, we observed some issues with hugetlb and CMA: 1) When specifying a large number of hugetlb boot pages (hugepages= on the commandline), the kernel may run out of memory before it even gets to HVO. For example, if you have a 3072G system, and want to use 3024 1G hugetlb pages for VMs, that should leave you plenty of space for the hypervisor, provided you have the hugetlb vmemmap optimization (HVO) enabled. However, since the vmemmap pages are always allocated first, and then later in boot freed, you will actually run yourself out of memory before you can do HVO. This means not getting all the hugetlb pages you want, and worse, failure to boot if there is an allocation failure in the system from which it can't recover. 2) There is a system setup where you might want to use hugetlb_cma with a large value (say, again, 3024 out of 3072G like above), and then lower that if system usage allows it, to make room for non-hugetlb processes. For this, a variation of the problem above applies: the kernel runs out of unmovable space to allocate from before you finish boot, since your CMA area takes up all the space. 3) CMA wants to use one big contiguous area for allocations. Which fails if you have the aforementioned 3T system with a gap in the middle of physical memory (like the < 40bits BIOS DMA area seen on some AMD systems). You then won't be able to set up a CMA area for one of the NUMA nodes, leading to loss of half of your hugetlb CMA area. 4) Under the scenario mentioned in 2), when trying to grow the number of hugetlb pages after dropping it for a while, new CMA allocations may fail occasionally. This is not unexpected, some transient references on pages may prevent cma_alloc from succeeding under memory pressure. However, the hugetlb code then falls back to a normal contiguous alloc, which may end up succeeding. This is not always desired behavior. If you have a large CMA area, then the kernel has a restricted amount of memory it can do unmovable allocations from (a well known issue). A normal contiguous alloc may eat further in to this space. To resolve these issues, do the following: * Add hooks to the section init code to do custom initialization of memmap pages. Hugetlb bootmem (memblock) allocated pages can then be pre-HVOed. This avoids allocating a large number of vmemmap pages early in boot, only to have them be freed again later, and also avoids running out of memory as described under 1). Using these hooks for hugetlb is optional. It requires moving hugetlb bootmem allocation to an earlier spot by the architecture. This has been enabled on x86. * hugetlb_cma doesn't care about the CMA area it uses being one large contiguous range. Multiple smaller ranges are fine. The only requirements are that the areas should be on one NUMA node, and individual gigantic pages should be allocatable from them. So, implement multi-range support for CMA, avoiding issue 3). * Introduce a hugetlb_cma_only option on the commandline. This only allows allocations from CMA for gigantic pages, if hugetlb_cma= is also specified. * With hugetlb_cma_only active, it also makes sense to be able to pre-allocate gigantic hugetlb pages at boot time from the CMA area(s). Add a rudimentary early CMA allocation interface, that just grabs a piece of memblock-allocated space from the CMA area, which gets marked as allocated in the CMA bitmap when the CMA area is initialized. With this, hugepages= can be supported with hugetlb_cma=, making scenario 2) work. Additionally, fix some minor bugs, with one worth mentioning: since hugetlb gigantic bootmem pages are allocated by memblock, they may span multiple zones, as memblock doesn't (and mostly can't) know about zones. This can cause problems. A hugetlb page spanning multiple zones is bad, and it's worse with HVO, when the de-HVO step effectively sneakily re-assigns pages to a different zone than originally configured, since the tail pages all inherit the zone from the first 60 tail pages. This condition is not common, but can be easily reproduced using ZONE_MOVABLE. To fix this, add checks to see if gigantic bootmem pages intersect with multiple zones, and do not use them if they do, giving them back to the page allocator instead. The first patch is kind of along for the ride, except that maintaining an available_count for a CMA area is convenient for the multiple range support. Frank van der Linden (27): mm/cma: export total and free number of pages for CMA areas mm, cma: support multiple contiguous ranges, if requested mm/cma: introduce cma_intersects function mm, hugetlb: use cma_declare_contiguous_multi mm/hugetlb: remove redundant __ClearPageReserved mm/hugetlb: use online nodes for bootmem allocation mm/hugetlb: convert cmdline parameters from setup to early x86/mm: make register_page_bootmem_memmap handle PTE mappings mm/bootmem_info: export register_page_bootmem_memmap mm/sparse: allow for alternate vmemmap section init at boot mm/hugetlb: set migratetype for bootmem folios mm: define __init_reserved_page_zone function mm/hugetlb: check bootmem pages for zone intersections mm/sparse: add vmemmap_*_hvo functions mm/hugetlb: deal with multiple calls to hugetlb_bootmem_alloc mm/hugetlb: move huge_boot_pages list init to hugetlb_bootmem_alloc mm/hugetlb: add pre-HVO framework mm/hugetlb_vmemmap: fix hugetlb_vmemmap_restore_folios definition mm/hugetlb: do pre-HVO for bootmem allocated pages x86/setup: call hugetlb_bootmem_alloc early x86/mm: set ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT mm/cma: simplify zone intersection check mm/cma: introduce a cma validate function mm/cma: introduce interface for early reservations mm/hugetlb: add hugetlb_cma_only cmdline option mm/hugetlb: enable bootmem allocation from CMA areas mm/hugetlb: move hugetlb CMA code in to its own file Documentation/ABI/testing/sysfs-kernel-mm-cma | 13 + .../admin-guide/kernel-parameters.txt | 21 +- Documentation/admin-guide/mm/cma_debugfs.rst | 10 +- MAINTAINERS | 2 + arch/powerpc/include/asm/book3s/64/hugetlb.h | 6 + arch/powerpc/mm/hugetlbpage.c | 1 + arch/powerpc/mm/init_64.c | 4 + arch/s390/mm/init.c | 13 +- arch/x86/Kconfig | 1 + arch/x86/kernel/setup.c | 4 +- arch/x86/mm/init_64.c | 18 +- include/linux/bootmem_info.h | 7 + include/linux/cma.h | 9 + include/linux/hugetlb.h | 35 + include/linux/mm.h | 13 +- include/linux/mmzone.h | 35 + mm/Kconfig | 8 + mm/Makefile | 3 + mm/bootmem_info.c | 4 +- mm/cma.c | 739 +++++++++++++++--- mm/cma.h | 46 +- mm/cma_debug.c | 61 +- mm/cma_sysfs.c | 20 + mm/hugetlb.c | 582 ++++++++------ mm/hugetlb_cma.c | 275 +++++++ mm/hugetlb_cma.h | 57 ++ mm/hugetlb_vmemmap.c | 199 ++++- mm/hugetlb_vmemmap.h | 23 +- mm/internal.h | 19 + mm/mm_init.c | 78 +- mm/sparse-vmemmap.c | 168 +++- mm/sparse.c | 87 ++- 32 files changed, 2066 insertions(+), 495 deletions(-) create mode 100644 mm/hugetlb_cma.c create mode 100644 mm/hugetlb_cma.h