From patchwork Fri Feb 28 18:29:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13996927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C6EDC282D0 for ; Fri, 28 Feb 2025 18:30:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 685C828001B; Fri, 28 Feb 2025 13:30:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6369F280001; Fri, 28 Feb 2025 13:30:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 48A3D28001B; Fri, 28 Feb 2025 13:30:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 26F81280001 for ; Fri, 28 Feb 2025 13:30:34 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C56EEC1160 for ; Fri, 28 Feb 2025 18:30:33 +0000 (UTC) X-FDA: 83170193946.24.2379246 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf23.hostedemail.com (Postfix) with ESMTP id D5699140020 for ; Fri, 28 Feb 2025 18:30:31 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gqSRIi1L; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3xgDCZwQKCBU0Gy619916z.x97638FI-775Gvx5.9C1@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3xgDCZwQKCBU0Gy619916z.x97638FI-775Gvx5.9C1@flex--fvdl.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740767432; a=rsa-sha256; cv=none; b=A0RVKKz3xB9AYaJ+jl5GX/8rTrzMvwmCuambkh4FyIvNiV8/2t0yljjMIsuB+Plh3pJJyc nXcarzxiPjHyj1NzkkSanPGwXIqE0qm+oe7M+MQMLbjbUmx95dj9hWgPzrSG80kb5tUJuj PyJs91xPgi4wIRTeaE33gQqtEN2N6IY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gqSRIi1L; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf23.hostedemail.com: domain of 3xgDCZwQKCBU0Gy619916z.x97638FI-775Gvx5.9C1@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3xgDCZwQKCBU0Gy619916z.x97638FI-775Gvx5.9C1@flex--fvdl.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740767432; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TMe19yWV86sRu83AfjzN0OeG5+sON+m39CVh+mgprHg=; b=GzhALx6AGRNKcrXAhHgBQI5jK6rDC2ykfGkcUDCm4bJ05X5mbA2UZocu/m3BjEd4BSD2FB Lkvm15c+GtDm6CeULvFFyU+m2bf3uBjx5GFa9/okqXfbjuf8xTQ1Z4X8tKHp87l8xLG7JC vy7x6uR/i4qHs8dNgXeqAmkLvITotHs= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2fec1f46678so2976757a91.1 for ; Fri, 28 Feb 2025 10:30:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1740767431; x=1741372231; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TMe19yWV86sRu83AfjzN0OeG5+sON+m39CVh+mgprHg=; b=gqSRIi1Lr6sSbJkpUl+srL0Pe8obGQzJ7Wq3uOp+XfIFfBlzCT6aJ3HD8nEBAUUZSH 1pboEhg2/whlhvngGJ089j+4GXRceTSA6zuND6HZTvKn09B7sXss8Uj0hT0J0oA7rqw/ 7j5jVC+P1GpSMFoLhu0DwBR8j5ylfTncZwnbBLqwl9nM/zZGdSwr/qJ67Ua8umMpYf2f df+RkEwlUpbw5rzvUqu9A2pBMgOWrxh0wKdQspg1InyRg36OrKJqw/HBkqeBZo+aZS/u 829TtIGvOZ0aF9kQA/5DWlITuana9Lc85G0d+TacCFbI/Tthr0eTss4uvk6f/EUjg9Ud ufCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740767431; x=1741372231; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TMe19yWV86sRu83AfjzN0OeG5+sON+m39CVh+mgprHg=; b=BwIo1nqn4rcQtSIHoLbdy9ehwdeVOm4n320WQN8jZ+zAS/xSwrysPHFO3QAMgGukfA UBQNMbjhHWbZZC0u6sxhT+qB2CYQ11ddnNBhteY7yaDADI+lhMoouTgzuQmGXOWuuczN Ano6t/jeQuNFQpKHxSreYZLIq38HKwU0iO46cDqpYzuciqeF3OuVKNubqjsdWik0cPPn LI2NO9f5V2sxUReweuYmGDPI9I38DecLg5Go4Wu+X40DMzzXSd2lgnG670FO0Cy+1C9j cTtAFUuDizczFrZgEKnKLvoxB638LUGPlPHdjFnOU7uUJO2EHagJZ9IFQWUmCdadvX9j +8lQ== X-Forwarded-Encrypted: i=1; AJvYcCV8zn5Cpa2Up5U2+vZFHcmARZ4BIwSB3tEX7EfnwP+x2k07jWI1C7zYS/AAmBh+qp5JlByXsx71jA==@kvack.org X-Gm-Message-State: AOJu0Yw2S4e5wg30wGyDcmrq5aQt2abJrctX00d3a9Zmt3lEVpNWguXE KQEXFqehJi8hNrlrQc4vIGUx+IjEbSQIMMEHUX4UsrvtNChOKeAVF56AxnAbtBDdnozO1A== X-Google-Smtp-Source: AGHT+IFdDufRUVuOA2NiFtFscHtFYi1z0zZ+G49tzlylohpjZ77dDf1/vb2nPMiEU/5Q5ipJa4U2fDbm X-Received: from pjbsr16.prod.google.com ([2002:a17:90b:4e90:b0:2fa:2891:e310]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3d0a:b0:2f6:be57:49d2 with SMTP id 98e67ed59e1d1-2febab7459bmr8199787a91.17.1740767430763; Fri, 28 Feb 2025 10:30:30 -0800 (PST) Date: Fri, 28 Feb 2025 18:29:27 +0000 In-Reply-To: <20250228182928.2645936-1-fvdl@google.com> Mime-Version: 1.0 References: <20250228182928.2645936-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.711.g2feabab25a-goog Message-ID: <20250228182928.2645936-27-fvdl@google.com> Subject: [PATCH v5 26/27] mm/hugetlb: enable bootmem allocation from CMA areas From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, ziy@nvidia.com, david@redhat.com, Frank van der Linden , Madhavan Srinivasan , Michael Ellerman , linuxppc-dev@lists.ozlabs.org X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D5699140020 X-Stat-Signature: ypap9x9mjwrfaicuubnsczfme3eufano X-HE-Tag: 1740767431-405728 X-HE-Meta: U2FsdGVkX19QhQ1apYNqw9iieJ0MAUz7bultHquHW3jZ/jc3HulIeJn4nOrICB7HM89Mw7pBBU6ezwhtfocGRoLlpE+pSWrV6kSSqY4uqGVAOTurodLtPw75WDXAPC1oX18kGyXwD+gkewRUk9VF5bS2F0zbYeysAUbBKbzvusUHYRvR0tZNGjpqd5EBMSaVg/sotpzw25yd1rxxLnLP/lERrlhN3eO2uyIAwZiOWbc2IHoYqCNvif0nPB/gCdZIvKMhh7ouIB4HwsiuRFWKPrzaZn6RozfyX8Uq4qGw4ZLzjZ+HVSUqxco5ylmTXTVS7EK1qGkh6zSdc9jkjknEpEjeZZXmf+EdoUkmj4OjB4bGbAg0MhlMWPvLYw6eWZq3LQnlEFo8LoSO36xWZ03DCa6OvVstCxGLrl8uzp1JG3fS9zBcPznPdywPCR+cNNmfKLYVC6zfyNxkApBfPtsBoLvPXfLqpExzdZtJRkfnAHqBUpv/5xn/AFwRe6vNDobbO1I3dRqEyWp5c3tMmFVm8wd1jMVLNe42zFhGRhqUMCHChQ+yeMdIRjU0caE+eXXc14Lpe5O2FGkW08xM1ecboXmLyLr2VtJG0pIgPQkK2nnSW1Bu+nNjQfIZyf+MtRe8HpjbaZpogdnxciwiRIjSdRQeUFw4v4Lhbnu+ntanQOpAv94OAoscc6xANqc8T6TwoQYSh1ID4sUAPy8XLFSgpYuNCyUeS1/LMjXKGC7NSlfzGnt8sIJJmARcFv+uVDNjoQHGYuKku596etq/DeyDuy5XO+TAJvTk+fPFlHlbiD3+jM2W3cPRTpPjZ6/gaBwNXDD2c18mE2FJuoRQmBjukrnw1N1mQSFdYHvYnDk2lk2sH/sTWbuoLl79ILKpI9vhz3okq02M6OA4oA/cr9eXq0//QL88Jy0S3DY6jX8y/YOmWn1wiBiVry0tC+/+2TeeVUwK3oweRsO1NROZx77 VBvURyKV vIeElBJ99sczuiyUs3RDajUYSe/xx3Fcf46ZPYBtHKfxsqvR4Lfi8fIfkyvb+A7GSEtwaA+oavoe9V84WvbQp+BlqRjAcGUTUmdqfwumPfGCwvMGfkUnDUM5DIk8ZH9Io+J1LkeFAhFPAwMDD3aWY+sHdwvT/3CNJMHQtVw7qtOTe7IFe+5QqW6i1sj+gtZXTwDvLOKeta75tpAO0+P2fMkx1kps5LXI+YQaWqRAJVRSEGvkiEoEHXYEHlOaaLxgjhaXB+nudHnpMoNNU/Zh1nOKyiINdJaSpCFUyfbfosHWybH5EtKZQaQf4Qu3+c18IbFggY4Uh+1KvOkuGnLW5HrDIo283mVcfAAcuNxfSQ82CwUXHMrR9lfoxp87WrbqtNA1LynSxN9r7ATFCmLX+SdMNldVp6QKQh9iQ6cgofSHZVSXCmoFXTCJcwMl3arW5SDKjOW1Vpf0JcfW9ftgSbDPNv0OwHQf1UfUfP+rxgXiPcGCFHAT3UR2gybDbLgS2pJzX/iLFjDzlTaAs8brRWX3cwdU1h5oelpqsNIVTsLppZi9MluT9Bdb00K7ssVEC/TYzn/i5Fn1rxtoDWhH83iezA1Dg1rJTU37MOwEPhVKYcXQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If hugetlb_cma_only is enabled, we know that hugetlb pages can only be allocated from CMA. Now that there is an interface to do early reservations from a CMA area (returning memblock memory), it can be used to allocate hugetlb pages from CMA. This also allows for doing pre-HVO on these pages (if enabled). Make sure to initialize the page structures and associated data correctly. Create a flag to signal that a hugetlb page has been allocated from CMA to make things a little easier. Some configurations of powerpc have a special hugetlb bootmem allocator, so introduce a boolean arch_specific_huge_bootmem_alloc that returns true if such an allocator is present. In that case, CMA bootmem allocations can't be used, so check that function before trying. Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Frank van der Linden --- arch/powerpc/include/asm/book3s/64/hugetlb.h | 6 + include/linux/hugetlb.h | 17 ++ mm/hugetlb.c | 168 ++++++++++++++----- 3 files changed, 152 insertions(+), 39 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/hugetlb.h b/arch/powerpc/include/asm/book3s/64/hugetlb.h index f0bba9c5f9c3..bb786694dd26 100644 --- a/arch/powerpc/include/asm/book3s/64/hugetlb.h +++ b/arch/powerpc/include/asm/book3s/64/hugetlb.h @@ -94,4 +94,10 @@ static inline int check_and_get_huge_psize(int shift) return mmu_psize; } +#define arch_has_huge_bootmem_alloc arch_has_huge_bootmem_alloc + +static inline bool arch_has_huge_bootmem_alloc(void) +{ + return (firmware_has_feature(FW_FEATURE_LPAR) && !radix_enabled()); +} #endif diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 2512463bca49..6c6546b54934 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -591,6 +591,7 @@ enum hugetlb_page_flags { HPG_freed, HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, + HPG_cma, __NR_HPAGEFLAGS, }; @@ -650,6 +651,7 @@ HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) +HPAGEFLAG(Cma, cma) #ifdef CONFIG_HUGETLB_PAGE @@ -678,14 +680,18 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; +struct cma; + struct huge_bootmem_page { struct list_head list; struct hstate *hstate; unsigned long flags; + struct cma *cma; }; #define HUGE_BOOTMEM_HVO 0x0001 #define HUGE_BOOTMEM_ZONES_VALID 0x0002 +#define HUGE_BOOTMEM_CMA 0x0004 bool hugetlb_bootmem_page_zones_valid(int nid, struct huge_bootmem_page *m); @@ -823,6 +829,17 @@ static inline pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, } #endif +#ifndef arch_has_huge_bootmem_alloc +/* + * Some architectures do their own bootmem allocation, so they can't use + * early CMA allocation. + */ +static inline bool arch_has_huge_bootmem_alloc(void) +{ + return false; +} +#endif + static inline struct hstate *folio_hstate(struct folio *folio) { VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0b483c466656..664ccaaa717a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -131,8 +131,10 @@ static void hugetlb_free_folio(struct folio *folio) #ifdef CONFIG_CMA int nid = folio_nid(folio); - if (cma_free_folio(hugetlb_cma[nid], folio)) + if (folio_test_hugetlb_cma(folio)) { + WARN_ON_ONCE(!cma_free_folio(hugetlb_cma[nid], folio)); return; + } #endif folio_put(folio); } @@ -1508,6 +1510,9 @@ static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, break; } } + + if (folio) + folio_set_hugetlb_cma(folio); } #endif if (!folio) { @@ -3174,6 +3179,86 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, return ERR_PTR(-ENOSPC); } +static bool __init hugetlb_early_cma(struct hstate *h) +{ + if (arch_has_huge_bootmem_alloc()) + return false; + + return (hstate_is_gigantic(h) && hugetlb_cma_only); +} + +static __init void *alloc_bootmem(struct hstate *h, int nid, bool node_exact) +{ + struct huge_bootmem_page *m; + unsigned long flags; + struct cma *cma; + int listnode = nid; + +#ifdef CONFIG_CMA + if (hugetlb_early_cma(h)) { + flags = HUGE_BOOTMEM_CMA; + cma = hugetlb_cma[nid]; + m = cma_reserve_early(cma, huge_page_size(h)); + if (!m) { + int node; + + if (node_exact) + return NULL; + for_each_online_node(node) { + cma = hugetlb_cma[node]; + if (!cma || node == nid) + continue; + m = cma_reserve_early(cma, huge_page_size(h)); + if (m) { + listnode = node; + break; + } + } + } + } else +#endif + { + flags = 0; + cma = NULL; + if (node_exact) + m = memblock_alloc_exact_nid_raw(huge_page_size(h), + huge_page_size(h), 0, + MEMBLOCK_ALLOC_ACCESSIBLE, nid); + else { + m = memblock_alloc_try_nid_raw(huge_page_size(h), + huge_page_size(h), 0, + MEMBLOCK_ALLOC_ACCESSIBLE, nid); + /* + * For pre-HVO to work correctly, pages need to be on + * the list for the node they were actually allocated + * from. That node may be different in the case of + * fallback by memblock_alloc_try_nid_raw. So, + * extract the actual node first. + */ + if (m) + listnode = early_pfn_to_nid(PHYS_PFN(virt_to_phys(m))); + } + } + + if (m) { + /* + * Use the beginning of the huge page to store the + * huge_bootmem_page struct (until gather_bootmem + * puts them into the mem_map). + * + * Put them into a private list first because mem_map + * is not up yet. + */ + INIT_LIST_HEAD(&m->list); + list_add(&m->list, &huge_boot_pages[listnode]); + m->hstate = h; + m->flags = flags; + m->cma = cma; + } + + return m; +} + int alloc_bootmem_huge_page(struct hstate *h, int nid) __attribute__ ((weak, alias("__alloc_bootmem_huge_page"))); int __alloc_bootmem_huge_page(struct hstate *h, int nid) @@ -3183,22 +3268,15 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) /* do node specific alloc */ if (nid != NUMA_NO_NODE) { - m = memblock_alloc_exact_nid_raw(huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); + m = alloc_bootmem(h, node, true); if (!m) return 0; goto found; } + /* allocate from next node when distributing huge pages */ for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_ONLINE]) { - m = memblock_alloc_try_nid_raw( - huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); - /* - * Use the beginning of the huge page to store the - * huge_bootmem_page struct (until gather_bootmem - * puts them into the mem_map). - */ + m = alloc_bootmem(h, node, false); if (!m) return 0; goto found; @@ -3216,21 +3294,6 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE), huge_page_size(h) - PAGE_SIZE); - /* - * Put them into a private list first because mem_map is not up yet. - * - * For pre-HVO to work correctly, pages need to be on the list for - * the node they were actually allocated from. That node may be - * different in the case of fallback by memblock_alloc_try_nid_raw. - * So, extract the actual node first. - */ - if (nid == NUMA_NO_NODE) - node = early_pfn_to_nid(PHYS_PFN(virt_to_phys(m))); - - INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages[node]); - m->hstate = h; - m->flags = 0; return 1; } @@ -3271,13 +3334,25 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) +{ + return m->flags & HUGE_BOOTMEM_HVO; +} + +static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page *m) +{ + return m->flags & HUGE_BOOTMEM_CMA; +} + /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) - * here. + * here, or MIGRATE_CMA if this was a page allocated through an early CMA + * reservation. * - * Note that this will not write the page struct, it is ok (and necessary) - * to do this on vmemmap optimized folios. + * In case of vmemmap optimized folios, the tail vmemmap pages are mapped + * read-only, but that's ok - for sparse vmemmap this does not write to + * the page structure. */ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, struct hstate *h) @@ -3286,9 +3361,13 @@ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio))); - for (i = 0; i < nr_pages; i += pageblock_nr_pages) - set_pageblock_migratetype(folio_page(folio, i), + for (i = 0; i < nr_pages; i += pageblock_nr_pages) { + if (folio_test_hugetlb_cma(folio)) + init_cma_pageblock(folio_page(folio, i)); + else + set_pageblock_migratetype(folio_page(folio, i), MIGRATE_MOVABLE); + } } static void __init prep_and_add_bootmem_folios(struct hstate *h, @@ -3334,10 +3413,16 @@ bool __init hugetlb_bootmem_page_zones_valid(int nid, return true; } + if (hugetlb_bootmem_page_earlycma(m)) { + valid = cma_validate_zones(m->cma); + goto out; + } + start_pfn = virt_to_phys(m) >> PAGE_SHIFT; valid = !pfn_range_intersects_zones(nid, start_pfn, pages_per_huge_page(m->hstate)); +out: if (!valid) hstate_boot_nrinvalid[hstate_index(m->hstate)]++; @@ -3366,11 +3451,6 @@ static void __init hugetlb_bootmem_free_invalid_page(int nid, struct page *page, } } -static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) -{ - return (m->flags & HUGE_BOOTMEM_HVO); -} - /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. @@ -3420,14 +3500,21 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) */ folio_set_hugetlb_vmemmap_optimized(folio); + if (hugetlb_bootmem_page_earlycma(m)) + folio_set_hugetlb_cma(folio); + list_add(&folio->lru, &folio_list); /* * We need to restore the 'stolen' pages to totalram_pages * in order to fix confusing memory reports from free(1) and * other side-effects, like CommitLimit going negative. + * + * For CMA pages, this is done in init_cma_pageblock + * (via hugetlb_bootmem_init_migratetype), so skip it here. */ - adjust_managed_page_count(page, pages_per_huge_page(h)); + if (!folio_test_hugetlb_cma(folio)) + adjust_managed_page_count(page, pages_per_huge_page(h)); cond_resched(); } @@ -3612,8 +3699,11 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long allocated; - /* skip gigantic hugepages allocation if hugetlb_cma enabled */ - if (hstate_is_gigantic(h) && hugetlb_cma_size) { + /* + * Skip gigantic hugepages allocation if early CMA + * reservations are not available. + */ + if (hstate_is_gigantic(h) && hugetlb_cma_size && !hugetlb_early_cma(h)) { pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n"); return; }