From patchwork Thu Feb 6 18:50:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4F9CC0219B for ; Thu, 6 Feb 2025 18:51:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71CAF6B0085; Thu, 6 Feb 2025 13:51:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 67F63280001; Thu, 6 Feb 2025 13:51:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D1726B008A; Thu, 6 Feb 2025 13:51:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2E7396B0085 for ; Thu, 6 Feb 2025 13:51:21 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DE406A143E for ; Thu, 6 Feb 2025 18:51:20 +0000 (UTC) X-FDA: 83090412720.10.8765D98 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf05.hostedemail.com (Postfix) with ESMTP id 124A8100019 for ; Thu, 6 Feb 2025 18:51:18 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HNJLIqvk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3pgSlZwQKCIcq6owrzzrwp.nzxwty58-xxv6lnv.z2r@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3pgSlZwQKCIcq6owrzzrwp.nzxwty58-xxv6lnv.z2r@flex--fvdl.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867879; a=rsa-sha256; cv=none; b=n4sA7dSkld+kU2LwtumAGyRhIwID8JiuNWVxmgSJ+46FyyID4nSY1zpVN9W90fOoSjx5wj NUEZLi1kSrYSdHBi8sDPL8Wt/oXiU0a167/sRPPovB28TZrj4nfWL8jQY9tc5XFTbe716l dDw8zNJe7ypncBDCij7P83CoWXz/e8M= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HNJLIqvk; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf05.hostedemail.com: domain of 3pgSlZwQKCIcq6owrzzrwp.nzxwty58-xxv6lnv.z2r@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3pgSlZwQKCIcq6owrzzrwp.nzxwty58-xxv6lnv.z2r@flex--fvdl.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867879; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NJPFpuaCa1L0JdfMq2mnIjylFDa72i51MvIbj3cBAAk=; b=NNCSvqa4hmyxXADuVV29tLEzdTakBe+aI+8uQzbXufDJCMo67Fv3TOwwMZGmNKTkyaJYn0 +7OEcg7FFPRznVGyEV3+aRiMPiyJJM68+g0xQVvDgmLodqGOGBFRijQ6y/5NwzOimwxaU4 psSsWrrPn7d2Qk6EgpgcnaM9CWNIQMw= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f46b7851fcso3581513a91.1 for ; Thu, 06 Feb 2025 10:51:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867878; x=1739472678; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NJPFpuaCa1L0JdfMq2mnIjylFDa72i51MvIbj3cBAAk=; b=HNJLIqvkmkmzl+FvX/hCOdGbRDL5wR6CpmbCVr5Tdj9wADZynLwAB5kX3kcQ7D4Yy7 BNT1cmgUlchaMOmzkR9E7nHVKmbuVsoLrYYkCQ69BY9MW+LDBj8euqO56B+pcFiLyv6d bdA9Inwlaa6sjzfNRiPUJ/Slr/yp2vs3JBidlcQrnW2EHTbwmmoaz6iQngzr1WGVv8le Uv09KT92HWa/l1FevYC7j5URwyssn8vmPhyfHc2kj7gln/ANx762LMg0bnO/v7tv0m2I B7Xe2PebuQBAsclKBsMYVkeWiZDPG6hcIIfHv1oeMN22n55njMwXBuf1sG3211ASIGC4 m9zA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867878; x=1739472678; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NJPFpuaCa1L0JdfMq2mnIjylFDa72i51MvIbj3cBAAk=; b=CODcaHkrRf9RZF7e2yJZCMB1lYnisjPKIwWq6x2Q8faU3t5dMqTAoJNgkmKxFvLDtA Xjwx73Vof1ovx0qgukJtg+sdF1pcSvHX1fWf4RTKQyAmze/XLrEjotkOAQtJWKH8cMh0 cTImEeJsCIpmxyNiv3P49ay3gsLtVrTcoFy0aP52kO4OaHLAVcwL4DadIKRZE4TmRjjC 8fcSuHpGdm4U3Y0itQOMbRUMRAFeLPyzMYyph1jInpjdDzIpU59ngd55ihkW5df03p5F wOgWO5pAZb2Gf2HEPCaXDASmTdmOCTc0SwKzh9VUGG+M0FvO4kC7UoD9uw4te/tkboPO j/ww== X-Forwarded-Encrypted: i=1; AJvYcCVKPaJWsE+Xd5Mt3r0gjSRPksC3iebY/ZD3oAS1x6TKfDv9Fg7fALJbWUTY2yaCrhvj6Mtxrl7pZA==@kvack.org X-Gm-Message-State: AOJu0YzzagvoaDrADqRgqfcJtFOLAPGYgJnMF0Jk3FlPDO4fKWKtNqd+ yvmyxZD/tV8jrsdCuHeU12NQRZYczz9VFgE8QClh44aw5PkLhra84PhVUdv95T5avMCblg== X-Google-Smtp-Source: AGHT+IE32zObckxQH9j+bb16bIAOnvT1L5KvON9aMD8I/isc7mdoYBPbnHRBSRCen9t4NlEX4I7KXcS/ X-Received: from pjbpm18.prod.google.com ([2002:a17:90b:3c52:b0:2ea:448a:8cd1]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3d03:b0:2ea:bf1c:1e3a with SMTP id 98e67ed59e1d1-2fa24069ec6mr240439a91.12.1738867878036; Thu, 06 Feb 2025 10:51:18 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:41 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-2-fvdl@google.com> Subject: [PATCH v3 01/28] mm/cma: export total and free number of pages for CMA areas From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 124A8100019 X-Stat-Signature: dy8xwwwo1oc7mqtfiooay8eqdicpcwxk X-Rspam-User: X-HE-Tag: 1738867878-688959 X-HE-Meta: U2FsdGVkX1//+3CRRACdrsRWf2MAfbNAK43ws2suJio3jDqw0GhI3gbbf5VzJ2D8aaMVFwunxrvvpydoklmTwAsdvYf5eqSSbYs4OKAa9npmeNgEsQqx/iVKd3Juf+KWyzGcfd8eFTTEeEQqkP8hAR1kmECjaEMG22gC1unQ1aFJ5jETu457HsS3P9apXmj6nmbH95Cfo3Xy4eAwUFPb429ArOqHjA86Hu2lX+i/xVmBYuxxCGNg83CGmxLM039WgBUTFKlaoIbBZmodQYyodhjyPZTqdgAyVbnBBQtNkkVww+Sv0MMQlIoSN/laUUdbkrtjcxWrkhGLzV1Fr6OIDoODv6up2czqFo1e7OD2pIzlsQuAM2rFXdjUCTrdgyK5hYnjkMu9C+z+INoQZC+VkYg6OYs2kLo3iZsIbzgr18YM6eA8LioVF8fOj615p4whrGc+LapjmRABn5RpNEIbLAXNiQaSneoNGmcBSAN4Yagphc7eboScJjuz9QJ/SLad6uJlW/Jq0pdlRHVLc6XMd3XpJQzNAgL37JGiGX0dTRrzphgyejWuhMMcPEOFkquXa5x/qc3SUrQdsV/ynqxQkX3+eUtjKRmoc7JESHMajBRnpGLGjQXurNwo27EPoaAXdwPw1oLtcT1K4Q6SA4pHWkFQl1TrVU2hcN3+iiaAF66riE8bBNlk3iI1AthbRTXiq9rbp9KOE11WggoEwWqRwSZ1iWv7+mhCA3Ld79mAt7ZEXscXqJzRtkAMouvyCHx5Ys+bWgwc4BK7R3SCwkdx70o41MIvhVOezqOYDIvF6yOGAUnlDQ/o4KaTotdpNEmuBtZGszOH8chIri0I1x7awlRGCqs/ayD5ZHOhLJxfHGHgLfU+2Iu6QxleqXqpMOUItqRWSkycVbd/Pk5rROXryLlAiyb2dQBsc4I15MhPtRpVyOyd7j7htXmk4duY1m74xBu6orO0y721Gfp0EHj jjvjG3la mSxzD/4xrswfDkkOlZCbsDfnF76ThkpIeaqIrquUUUDy8LSa2h2uVSe/nYZAVBiEoHl2hSjz/p3+VMui0bOcvVznF9dzGgkxB7zEoWHpoYDlGbNYtE0ZYdv8CiE/kSIlso7woNMvoSWlTOLGEZ4hBx0G/E8hKJUMqz+4qgkuyRtYqybhVaT2KmGOY3bVosxUxz2vZjoM4VfVqgFaJIQgkF103wLN5z/hk+XfcJikGq4r6Eb6qKZCO+vxOjyEe89KuAAwvtCrQzFQLaDA/IClAbjlpFnqDwkTTtjNjTyxoQkzpSpABJN9cetQyrF+15HJXEFDEsdTAKmjpGHTn3KTlqFur94qotPddPBTb67wibAH6gzpPV0HKKtYSYvU35SuAtmhdw33hekf6B/o1QF7BM4mDFWxqBYxoAPASo81WxRpFUSCA2ClO55Gb6JcPoXyeweoQ1YC8qamyTk81gacBAigtppFyNn3JS8VB31arcJ4sVdCN4gzodASUZTOb4jwBtFkLxKSplmZnNXT6zHNNFqT/LVo479YFhaRlVV/BbA8fNET1veCtIhK8XiXKQ8uI6qsvCLYpe0ZGhsySL2GrG/YJBjRZq+nEFmPj00gVEkzM4SU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In addition to the number of allocations and releases, system management software may like to be aware of the size of CMA areas, and how many pages are available in it. This information is currently not available, so export it in total_page and available_pages, respectively. The name 'available_pages' was picked over 'free_pages' because 'free' implies that the pages are unused. But they might not be, they just haven't been used by cma_alloc The number of available pages is tracked regardless of CONFIG_CMA_SYSFS, allowing for a few minor shortcuts in the code, avoiding bitmap operations. Signed-off-by: Frank van der Linden Reviewed-by: Oscar Salvador --- Documentation/ABI/testing/sysfs-kernel-mm-cma | 13 +++++++++++ mm/cma.c | 22 ++++++++++++++----- mm/cma.h | 1 + mm/cma_debug.c | 5 +---- mm/cma_sysfs.c | 20 +++++++++++++++++ 5 files changed, 51 insertions(+), 10 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-kernel-mm-cma b/Documentation/ABI/testing/sysfs-kernel-mm-cma index dfd755201142..aaf2a5d8b13b 100644 --- a/Documentation/ABI/testing/sysfs-kernel-mm-cma +++ b/Documentation/ABI/testing/sysfs-kernel-mm-cma @@ -29,3 +29,16 @@ Date: Feb 2024 Contact: Anshuman Khandual Description: the number of pages CMA API succeeded to release + +What: /sys/kernel/mm/cma//total_pages +Date: Jun 2024 +Contact: Frank van der Linden +Description: + The size of the CMA area in pages. + +What: /sys/kernel/mm/cma//available_pages +Date: Jun 2024 +Contact: Frank van der Linden +Description: + The number of pages in the CMA area that are still + available for CMA allocation. diff --git a/mm/cma.c b/mm/cma.c index de5bc0c81fc2..95a8788e54d3 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -86,6 +86,7 @@ static void cma_clear_bitmap(struct cma *cma, unsigned long pfn, spin_lock_irqsave(&cma->lock, flags); bitmap_clear(cma->bitmap, bitmap_no, bitmap_count); + cma->available_count += count; spin_unlock_irqrestore(&cma->lock, flags); } @@ -133,7 +134,7 @@ static void __init cma_activate_area(struct cma *cma) free_reserved_page(pfn_to_page(pfn)); } totalcma_pages -= cma->count; - cma->count = 0; + cma->available_count = cma->count = 0; pr_err("CMA area %s could not be activated\n", cma->name); } @@ -206,7 +207,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count); cma->base_pfn = PFN_DOWN(base); - cma->count = size >> PAGE_SHIFT; + cma->available_count = cma->count = size >> PAGE_SHIFT; cma->order_per_bit = order_per_bit; *res_cma = cma; cma_area_count++; @@ -390,7 +391,7 @@ static void cma_debug_show_areas(struct cma *cma) { unsigned long next_zero_bit, next_set_bit, nr_zero; unsigned long start = 0; - unsigned long nr_part, nr_total = 0; + unsigned long nr_part; unsigned long nbits = cma_bitmap_maxno(cma); spin_lock_irq(&cma->lock); @@ -402,12 +403,12 @@ static void cma_debug_show_areas(struct cma *cma) next_set_bit = find_next_bit(cma->bitmap, nbits, next_zero_bit); nr_zero = next_set_bit - next_zero_bit; nr_part = nr_zero << cma->order_per_bit; - pr_cont("%s%lu@%lu", nr_total ? "+" : "", nr_part, + pr_cont("%s%lu@%lu", start ? "+" : "", nr_part, next_zero_bit); - nr_total += nr_part; start = next_zero_bit + nr_zero; } - pr_cont("=> %lu free of %lu total pages\n", nr_total, cma->count); + pr_cont("=> %lu free of %lu total pages\n", cma->available_count, + cma->count); spin_unlock_irq(&cma->lock); } @@ -444,6 +445,14 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, for (;;) { spin_lock_irq(&cma->lock); + /* + * If the request is larger than the available number + * of pages, stop right away. + */ + if (count > cma->available_count) { + spin_unlock_irq(&cma->lock); + break; + } bitmap_no = bitmap_find_next_zero_area_off(cma->bitmap, bitmap_maxno, start, bitmap_count, mask, offset); @@ -452,6 +461,7 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, break; } bitmap_set(cma->bitmap, bitmap_no, bitmap_count); + cma->available_count -= count; /* * It's safe to drop the lock here. We've marked this region for * our exclusive use. If the migration fails we will take the diff --git a/mm/cma.h b/mm/cma.h index 8485ef893e99..3dd3376ae980 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -13,6 +13,7 @@ struct cma_kobject { struct cma { unsigned long base_pfn; unsigned long count; + unsigned long available_count; unsigned long *bitmap; unsigned int order_per_bit; /* Order of pages represented by one bit */ spinlock_t lock; diff --git a/mm/cma_debug.c b/mm/cma_debug.c index 602fff89b15f..89236f22230a 100644 --- a/mm/cma_debug.c +++ b/mm/cma_debug.c @@ -34,13 +34,10 @@ DEFINE_DEBUGFS_ATTRIBUTE(cma_debugfs_fops, cma_debugfs_get, NULL, "%llu\n"); static int cma_used_get(void *data, u64 *val) { struct cma *cma = data; - unsigned long used; spin_lock_irq(&cma->lock); - /* pages counter is smaller than sizeof(int) */ - used = bitmap_weight(cma->bitmap, (int)cma_bitmap_maxno(cma)); + *val = cma->count - cma->available_count; spin_unlock_irq(&cma->lock); - *val = (u64)used << cma->order_per_bit; return 0; } diff --git a/mm/cma_sysfs.c b/mm/cma_sysfs.c index f50db3973171..97acd3e5a6a5 100644 --- a/mm/cma_sysfs.c +++ b/mm/cma_sysfs.c @@ -62,6 +62,24 @@ static ssize_t release_pages_success_show(struct kobject *kobj, } CMA_ATTR_RO(release_pages_success); +static ssize_t total_pages_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct cma *cma = cma_from_kobj(kobj); + + return sysfs_emit(buf, "%lu\n", cma->count); +} +CMA_ATTR_RO(total_pages); + +static ssize_t available_pages_show(struct kobject *kobj, + struct kobj_attribute *attr, char *buf) +{ + struct cma *cma = cma_from_kobj(kobj); + + return sysfs_emit(buf, "%lu\n", cma->available_count); +} +CMA_ATTR_RO(available_pages); + static void cma_kobj_release(struct kobject *kobj) { struct cma *cma = cma_from_kobj(kobj); @@ -75,6 +93,8 @@ static struct attribute *cma_attrs[] = { &alloc_pages_success_attr.attr, &alloc_pages_fail_attr.attr, &release_pages_success_attr.attr, + &total_pages_attr.attr, + &available_pages_attr.attr, NULL, }; ATTRIBUTE_GROUPS(cma); From patchwork Thu Feb 6 18:50:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963550 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0527AC0219C for ; Thu, 6 Feb 2025 18:51:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28AF2280001; Thu, 6 Feb 2025 13:51:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 19F61280002; Thu, 6 Feb 2025 13:51:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EBE24280001; Thu, 6 Feb 2025 13:51:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B94B16B0089 for ; Thu, 6 Feb 2025 13:51:22 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6FC2C1C9177 for ; Thu, 6 Feb 2025 18:51:22 +0000 (UTC) X-FDA: 83090412804.01.DCC15D5 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf15.hostedemail.com (Postfix) with ESMTP id 7D495A000B for ; Thu, 6 Feb 2025 18:51:20 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=EQRps6Ml; spf=pass (imf15.hostedemail.com: domain of 3pwSlZwQKCIgr7pxs00sxq.o0yxuz69-yyw7mow.03s@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3pwSlZwQKCIgr7pxs00sxq.o0yxuz69-yyw7mow.03s@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867880; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b6zbvBC7sDTC9/vWH7N0h9lqjmBU1iu0Iy1mp9Qfvhc=; b=kPg9Jgz9GEzwxITZ428TaaVaoxQETBCrUFXTtt+2Sf+rEZjFeDCLIBuOt6Sd3CY38wGYgd rtmtHCavdUAXuarQDLLP1TkiSOWoXnpx9dW+L4cr1COXecv8zrmY7dNPUy8LkKueijZWrf ymauxcDoL2/80tbdKYqQtaXaKGafhWU= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=EQRps6Ml; spf=pass (imf15.hostedemail.com: domain of 3pwSlZwQKCIgr7pxs00sxq.o0yxuz69-yyw7mow.03s@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3pwSlZwQKCIgr7pxs00sxq.o0yxuz69-yyw7mow.03s@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867880; a=rsa-sha256; cv=none; b=5/8i11RizGZnMDAebxpEljUCQHMetf2QPoub+Q6yQIUOmETPil9lhPWLU/aLD9c58HT5FI hbxbeTk7bRp1qSOpLqs4Yr3jd77a1/cVRnZNzn1yqFja4iT4eveo3TsKU15b8M00HI+edd yvLd9wP5na9rRNrp6NvZKNCd4uWlw3M= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f9bf2d1c7eso2501881a91.1 for ; Thu, 06 Feb 2025 10:51:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867879; x=1739472679; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=b6zbvBC7sDTC9/vWH7N0h9lqjmBU1iu0Iy1mp9Qfvhc=; b=EQRps6MlTQ7sPbfi8Gqty6hwZLk3DPxfS/3ZhqTaHShTSb6mPvUFI0wlRT9BuNp92E HZq04zAelDN1oDjhcjcTV5c1yib1Y2sDL3j7fMtGRoSBQLU2ozf7mmJq1HCx70YZGKUn jV+s+mYo9PCLbpz8VKa0F/5gZU4CQYcZPd0W0S88G6wVTtFvIebv87IvmbD53ULnZro9 v5Q124DVZkb1ZgZCWk5o0Frl8onqv/pGap/5D9T7MCv2yYt7NtUVK6WueYBf+qIBYTZq oQrjGK3BryIBx8RGFmQc+UCJAwUhnM0iKfV6B/Ni3piKnyuDbouRxE+hfigDaUXsihOX TyZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867879; x=1739472679; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b6zbvBC7sDTC9/vWH7N0h9lqjmBU1iu0Iy1mp9Qfvhc=; b=ghh/FL6zhAZenlTOGQUcUK1wSf3cETvDzYDzA+ZNr47650rkMjCEshm6fy3xIwgy/P SrP7ZWYE+CYiCBcGIFpxhuGY611da4yNTiPcsNc11ZL9b9v9dHv6UKVJ8ZFiaJWz0Hyk ryAF82MV580Dv/XOWbDg+TTI52uWH0c1pPZC0rW1GRQdhthsrpmTdHRkTQ1pUKJbtEN6 4lFe19wvln3ez4fK1Rr8Fv8WrHa2xz7W/17WeqqUeQOkz2C5mZEQSWXH5gvo9LTGttA0 t9ywu7rz4knmUwo1TZ5K72tRnJvsmtL04dxaU5+JwuWCGsqJv9WGikSFjI+qzSFQ9t0C HnOg== X-Forwarded-Encrypted: i=1; AJvYcCWh2x0HSYv1pslUjEMRg+lShwbRBs0cmrHGHwR2KRiUfGfRlMjeQtIHXOTU5sGX1b/zqAuo0ZQ/DA==@kvack.org X-Gm-Message-State: AOJu0YxIi+LwmHGOAepOtBBtCONLuW6zc1VIkjgFqXYcncuwtqlitDR9 HYFj3lp3xjcCBPI2pO/slcnUYaaNSctIvBBWd5a5W002MOsHDALxpY7uLMxPI/rhZ9Idxg== X-Google-Smtp-Source: AGHT+IEKvPQ7lm0Rn5b7uX4UIGLM9ngAdhktQMTfPtHK7wAo3+YelYon7sVG6argp0J2+d6Xf0MHG2W9 X-Received: from pjbsi17.prod.google.com ([2002:a17:90b:5291:b0:2f9:d9fe:e723]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:19c4:b0:2fa:1e3e:9be5 with SMTP id 98e67ed59e1d1-2fa22b8bbb5mr346931a91.0.1738867879459; Thu, 06 Feb 2025 10:51:19 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:42 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-3-fvdl@google.com> Subject: [PATCH v3 02/28] mm, cma: support multiple contiguous ranges, if requested From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 7D495A000B X-Stat-Signature: toxi11xborskef7cq44jpxp7wq35pksn X-HE-Tag: 1738867880-138171 X-HE-Meta: U2FsdGVkX18AU7YdNfMrm/BCfnDK2I4gJNNZPEokbdAiisc67S7uGA/zPxXOtgYzUVCLv7Y5LxSzmUkrCaRaLajuG57/EOAoZyw1ZR0TsJHn3iAc+MpNsMvGY+XMp+MNazyBl11urBBaqpUpM9hT+JJoHZJqOaaczxTWXSSwWefj/bF4Zoz2BbK7AQ1LHJUb4QaBLOm3N9lG4G7SbocLRvGCz28Rgw7Lfu0hDbWe0n76WqL17Gb4euu4yFmXanHoH9JKwFLHEY8KeBrqAjdMJbTcZgdTuJFCDPPJ6saC6EcsR6aunjVjgmh1ii+weax0Na9tCtO07IXR45RP6zpFYftIcAkD+XTUqJDX65/RL4SYei3e3WKiLogPA29BKl1ConlHSuFZFiz9JT804wlFexNQY2lHAuJmeOZnsAzPrDguiPUvGrRxXC9X3NXHiz/1RrwTYej6PerGqh8VGbWYeuzYzvMx/WyvU8a1E7V7I0TBrDJifQ73OIJ5l+dDb141XM5i7Diot88qXNDw7gWS/psRxjT+IdbYaawlI8kjmwbJAV99OUnDtqaCwBiNaPhhAw/kbfg9h/PWt3z1i1BIe2JFusR1Tr7d/93A2E3jMly8iIS4tX6d4tQsPR8MmOiiWHQvDdrZASMpUHvbhpPPrD2qUqDb+m3OnjiMW4Jv3HwayeHRAonyNxECFWX8LoNabMsPgztyoUMq5LZemLAOHR8CEp3OdcZ9+9aTAllekqEIefIApBF1nJ035T28UIG3b9GImy6qYGX/TbTou9X6ZfV1yUOkg+Wltf5G+TdxyblmakP9BsxnrsKRJNVX6+pxQEHLCDM1LIQNaChGIIZDQdFxdlU+mwEbswgPsbbw/CJOH0ARoAI4VBj78ACAXkLlc+hzcjeZ+tmhL90ANvXwzI+2XaTLuAYo796Nk8uoROBlah2b89WTuuKK/JmeeLf3ZW/Q35vTzP4W8zPgIss PK2c+oG4 CrcX2/gzpiWHvxY1fwQwLD4UyXbehDSzXeEfMyi9FZ2VnTu+4i3iS5+xuDM1pF2s7d8x7eAGoOgSyX8KSfOxzyOFEJ7aUTLSBsijdE405KdwVp9ZJ9hRNUBAPpVWKJ2tFY8YI9/832pN9z/9sRPnSZlmxLMDGVjSbrsM15g/Wl3A94AfhxN6ZLKlV1DSxWOKBcXWNMK9lhH8n43XxBGuYLYEVz26LFb/Xacmo5MvAXHWN7hU/r9d3yo7jhH5t6k3D5NTt6fYhrf5NAn9nppzMc5HnARFf9a8o91jefxQgRP936GTAL9nEvDtoCsR8a6QBwIOV0jS78wB7hhm9L81mctgK7RICw4+NaSdW95peY3Tv6Ks4S4KJdKEqWeb+Kmf4ExctQkXxpJmOKoh6PPplKUO8rerRxnB9BYG7sSES8M3lmAUUzS4xIoT1T+z1TTvxJY0hW3K0uDVf7ZWYJy+JbGbF9xqL6OX1bmyO26V2e2DUqWbbiJi43tXwGjf8UUETr4qPI5MOK7twBvogDp6xt/PbQPTE5HgIhq1VKGyjWRUJL08= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, CMA manages one range of physically contiguous memory. Creation of larger CMA areas with hugetlb_cma may run in to gaps in physical memory, so that they are not able to allocate that contiguous physical range from memblock when creating the CMA area. This can happen, for example, on an AMD system with > 1TB of memory, where there will be a gap just below the 1TB (40bit DMA) line. If you have set aside most of memory for potential hugetlb CMA allocation, cma_declare_contiguous_nid will fail. hugetlb_cma doesn't need the entire area to be one physically contiguous range. It just cares about being able to get physically contiguous chunks of a certain size (e.g. 1G), and it is fine to have the CMA area backed by multiple physical ranges, as long as it gets 1G contiguous allocations. Multi-range support is implemented by introducing an array of ranges, instead of just one big one. Each range has its own bitmap. Effectively, the allocate and release operations work as before, just per-range. So, instead of going through one large bitmap, they now go through a number of smaller ones. The maximum number of supported ranges is 8, as defined in CMA_MAX_RANGES. Since some current users of CMA expect a CMA area to just use one physically contiguous range, only allow for multiple ranges if a new interface, cma_declare_contiguous_nid_multi, is used. The other interfaces will work like before, creating only CMA areas with 1 range. cma_declare_contiguous_nid_multi works as follows, mimicking the default "bottom-up, above 4G" reservation approach: 0) Try cma_declare_contiguous_nid, which will use only one region. If this succeeds, return. This makes sure that for all the cases that currently work, the behavior remains unchanged even if the caller switches from cma_declare_contiguous_nid to cma_declare_contiguous_nid_multi. 1) Select the largest free memblock ranges above 4G, with a maximum number of CMA_MAX_RANGES. 2) If we did not find at most CMA_MAX_RANGES that add up to the total size requested, return -ENOMEM. 3) Sort the selected ranges by base address. 4) Reserve them bottom-up until we get what we wanted. Signed-off-by: Frank van der Linden --- Documentation/admin-guide/mm/cma_debugfs.rst | 10 +- include/linux/cma.h | 3 + mm/cma.c | 594 +++++++++++++++---- mm/cma.h | 27 +- mm/cma_debug.c | 56 +- 5 files changed, 550 insertions(+), 140 deletions(-) diff --git a/Documentation/admin-guide/mm/cma_debugfs.rst b/Documentation/admin-guide/mm/cma_debugfs.rst index 7367e6294ef6..4120e9cb0cd5 100644 --- a/Documentation/admin-guide/mm/cma_debugfs.rst +++ b/Documentation/admin-guide/mm/cma_debugfs.rst @@ -12,10 +12,16 @@ its CMA name like below: The structure of the files created under that directory is as follows: - - [RO] base_pfn: The base PFN (Page Frame Number) of the zone. + - [RO] base_pfn: The base PFN (Page Frame Number) of the CMA area. + This is the same as ranges/0/base_pfn. - [RO] count: Amount of memory in the CMA area. - [RO] order_per_bit: Order of pages represented by one bit. - - [RO] bitmap: The bitmap of page states in the zone. + - [RO] bitmap: The bitmap of allocated pages in the area. + This is the same as ranges/0/base_pfn. + - [RO] ranges/N/base_pfn: The base PFN of contiguous range N + in the CMA area. + - [RO] ranges/N/bitmap: The bit map of allocated pages in + range N in the CMA area. - [WO] alloc: Allocate N pages from that CMA area. For example:: echo 5 > /cma//alloc diff --git a/include/linux/cma.h b/include/linux/cma.h index d15b64f51336..863427c27dc2 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -40,6 +40,9 @@ static inline int __init cma_declare_contiguous(phys_addr_t base, return cma_declare_contiguous_nid(base, size, limit, alignment, order_per_bit, fixed, name, res_cma, NUMA_NO_NODE); } +extern int __init cma_declare_contiguous_multi(phys_addr_t size, + phys_addr_t align, unsigned int order_per_bit, + const char *name, struct cma **res_cma, int nid); extern int cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, unsigned int order_per_bit, const char *name, diff --git a/mm/cma.c b/mm/cma.c index 95a8788e54d3..6ac9173d3a7d 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -35,9 +36,16 @@ struct cma cma_areas[MAX_CMA_AREAS]; unsigned int cma_area_count; static DEFINE_MUTEX(cma_mutex); +static int __init __cma_declare_contiguous_nid(phys_addr_t base, + phys_addr_t size, phys_addr_t limit, + phys_addr_t alignment, unsigned int order_per_bit, + bool fixed, const char *name, struct cma **res_cma, + int nid); + phys_addr_t cma_get_base(const struct cma *cma) { - return PFN_PHYS(cma->base_pfn); + WARN_ON_ONCE(cma->nranges != 1); + return PFN_PHYS(cma->ranges[0].base_pfn); } unsigned long cma_get_size(const struct cma *cma) @@ -63,9 +71,10 @@ static unsigned long cma_bitmap_aligned_mask(const struct cma *cma, * The value returned is represented in order_per_bits. */ static unsigned long cma_bitmap_aligned_offset(const struct cma *cma, + const struct cma_memrange *cmr, unsigned int align_order) { - return (cma->base_pfn & ((1UL << align_order) - 1)) + return (cmr->base_pfn & ((1UL << align_order) - 1)) >> cma->order_per_bit; } @@ -75,46 +84,57 @@ static unsigned long cma_bitmap_pages_to_bits(const struct cma *cma, return ALIGN(pages, 1UL << cma->order_per_bit) >> cma->order_per_bit; } -static void cma_clear_bitmap(struct cma *cma, unsigned long pfn, - unsigned long count) +static void cma_clear_bitmap(struct cma *cma, const struct cma_memrange *cmr, + unsigned long pfn, unsigned long count) { unsigned long bitmap_no, bitmap_count; unsigned long flags; - bitmap_no = (pfn - cma->base_pfn) >> cma->order_per_bit; + bitmap_no = (pfn - cmr->base_pfn) >> cma->order_per_bit; bitmap_count = cma_bitmap_pages_to_bits(cma, count); spin_lock_irqsave(&cma->lock, flags); - bitmap_clear(cma->bitmap, bitmap_no, bitmap_count); + bitmap_clear(cmr->bitmap, bitmap_no, bitmap_count); cma->available_count += count; spin_unlock_irqrestore(&cma->lock, flags); } static void __init cma_activate_area(struct cma *cma) { - unsigned long base_pfn = cma->base_pfn, pfn; + unsigned long pfn, base_pfn; + int allocrange, r; struct zone *zone; + struct cma_memrange *cmr; + + for (allocrange = 0; allocrange < cma->nranges; allocrange++) { + cmr = &cma->ranges[allocrange]; + cmr->bitmap = bitmap_zalloc(cma_bitmap_maxno(cma, cmr), + GFP_KERNEL); + if (!cmr->bitmap) + goto cleanup; + } - cma->bitmap = bitmap_zalloc(cma_bitmap_maxno(cma), GFP_KERNEL); - if (!cma->bitmap) - goto out_error; + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + base_pfn = cmr->base_pfn; - /* - * alloc_contig_range() requires the pfn range specified to be in the - * same zone. Simplify by forcing the entire CMA resv range to be in the - * same zone. - */ - WARN_ON_ONCE(!pfn_valid(base_pfn)); - zone = page_zone(pfn_to_page(base_pfn)); - for (pfn = base_pfn + 1; pfn < base_pfn + cma->count; pfn++) { - WARN_ON_ONCE(!pfn_valid(pfn)); - if (page_zone(pfn_to_page(pfn)) != zone) - goto not_in_zone; - } + /* + * alloc_contig_range() requires the pfn range specified + * to be in the same zone. Simplify by forcing the entire + * CMA resv range to be in the same zone. + */ + WARN_ON_ONCE(!pfn_valid(base_pfn)); + zone = page_zone(pfn_to_page(base_pfn)); + for (pfn = base_pfn + 1; pfn < base_pfn + cmr->count; pfn++) { + WARN_ON_ONCE(!pfn_valid(pfn)); + if (page_zone(pfn_to_page(pfn)) != zone) + goto cleanup; + } - for (pfn = base_pfn; pfn < base_pfn + cma->count; - pfn += pageblock_nr_pages) - init_cma_reserved_pageblock(pfn_to_page(pfn)); + for (pfn = base_pfn; pfn < base_pfn + cmr->count; + pfn += pageblock_nr_pages) + init_cma_reserved_pageblock(pfn_to_page(pfn)); + } spin_lock_init(&cma->lock); @@ -125,13 +145,19 @@ static void __init cma_activate_area(struct cma *cma) return; -not_in_zone: - bitmap_free(cma->bitmap); -out_error: +cleanup: + for (r = 0; r < allocrange; r++) + bitmap_free(cma->ranges[r].bitmap); + /* Expose all pages to the buddy, they are useless for CMA. */ if (!cma->reserve_pages_on_error) { - for (pfn = base_pfn; pfn < base_pfn + cma->count; pfn++) - free_reserved_page(pfn_to_page(pfn)); + for (r = 0; r < allocrange; r++) { + cmr = &cma->ranges[r]; + for (pfn = cmr->base_pfn; + pfn < cmr->base_pfn + cmr->count; + pfn++) + free_reserved_page(pfn_to_page(pfn)); + } } totalcma_pages -= cma->count; cma->available_count = cma->count = 0; @@ -154,6 +180,43 @@ void __init cma_reserve_pages_on_error(struct cma *cma) cma->reserve_pages_on_error = true; } +static int __init cma_new_area(const char *name, phys_addr_t size, + unsigned int order_per_bit, + struct cma **res_cma) +{ + struct cma *cma; + + if (cma_area_count == ARRAY_SIZE(cma_areas)) { + pr_err("Not enough slots for CMA reserved regions!\n"); + return -ENOSPC; + } + + /* + * Each reserved area must be initialised later, when more kernel + * subsystems (like slab allocator) are available. + */ + cma = &cma_areas[cma_area_count]; + cma_area_count++; + + if (name) + snprintf(cma->name, CMA_MAX_NAME, name); + else + snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count); + + cma->available_count = cma->count = size >> PAGE_SHIFT; + cma->order_per_bit = order_per_bit; + *res_cma = cma; + totalcma_pages += cma->count; + + return 0; +} + +static void __init cma_drop_area(struct cma *cma) +{ + totalcma_pages -= cma->count; + cma_area_count--; +} + /** * cma_init_reserved_mem() - create custom contiguous area from reserved memory * @base: Base address of the reserved area @@ -172,13 +235,9 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, struct cma **res_cma) { struct cma *cma; + int ret; /* Sanity checks */ - if (cma_area_count == ARRAY_SIZE(cma_areas)) { - pr_err("Not enough slots for CMA reserved regions!\n"); - return -ENOSPC; - } - if (!size || !memblock_is_region_reserved(base, size)) return -EINVAL; @@ -195,25 +254,261 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, if (!IS_ALIGNED(base | size, CMA_MIN_ALIGNMENT_BYTES)) return -EINVAL; + ret = cma_new_area(name, size, order_per_bit, &cma); + if (ret != 0) + return ret; + + cma->ranges[0].base_pfn = PFN_DOWN(base); + cma->ranges[0].count = cma->count; + cma->nranges = 1; + + *res_cma = cma; + + return 0; +} + +/* + * Structure used while walking physical memory ranges and finding out + * which one(s) to use for a CMA area. + */ +struct cma_init_memrange { + phys_addr_t base; + phys_addr_t size; + struct list_head list; +}; + +/* + * Work array used during CMA initialization. + */ +static struct cma_init_memrange memranges[CMA_MAX_RANGES] __initdata; + +static bool __init revsizecmp(struct cma_init_memrange *mlp, + struct cma_init_memrange *mrp) +{ + return mlp->size > mrp->size; +} + +static bool __init basecmp(struct cma_init_memrange *mlp, + struct cma_init_memrange *mrp) +{ + return mlp->base < mrp->base; +} + +/* + * Helper function to create sorted lists. + */ +static void __init list_insert_sorted( + struct list_head *ranges, + struct cma_init_memrange *mrp, + bool (*cmp)(struct cma_init_memrange *lh, struct cma_init_memrange *rh)) +{ + struct list_head *mp; + struct cma_init_memrange *mlp; + + if (list_empty(ranges)) + list_add(&mrp->list, ranges); + else { + list_for_each(mp, ranges) { + mlp = list_entry(mp, struct cma_init_memrange, list); + if (cmp(mlp, mrp)) + break; + } + __list_add(&mrp->list, mlp->list.prev, &mlp->list); + } +} + +/* + * Create CMA areas with a total size of @total_size. A normal allocation + * for one area is tried first. If that fails, the biggest memblock + * ranges above 4G are selected, and allocated bottom up. + * + * The complexity here is not great, but this function will only be + * called during boot, and the lists operated on have fewer than + * CMA_MAX_RANGES elements (default value: 8). + */ +int __init cma_declare_contiguous_multi(phys_addr_t total_size, + phys_addr_t align, unsigned int order_per_bit, + const char *name, struct cma **res_cma, int nid) +{ + phys_addr_t start, end; + phys_addr_t size, sizesum, sizeleft; + struct cma_init_memrange *mrp, *mlp, *failed; + struct cma_memrange *cmrp; + LIST_HEAD(ranges); + LIST_HEAD(final_ranges); + struct list_head *mp, *next; + int ret, nr = 1; + u64 i; + struct cma *cma; + /* - * Each reserved area must be initialised later, when more kernel - * subsystems (like slab allocator) are available. + * First, try it the normal way, producing just one range. */ - cma = &cma_areas[cma_area_count]; + ret = __cma_declare_contiguous_nid(0, total_size, 0, align, + order_per_bit, false, name, res_cma, nid); + if (ret != -ENOMEM) + goto out; - if (name) - snprintf(cma->name, CMA_MAX_NAME, name); - else - snprintf(cma->name, CMA_MAX_NAME, "cma%d\n", cma_area_count); + /* + * Couldn't find one range that fits our needs, so try multiple + * ranges. + * + * No need to do the alignment checks here, the call to + * cma_declare_contiguous_nid above would have caught + * any issues. With the checks, we know that: + * + * - @align is a power of 2 + * - @align is >= pageblock alignment + * - @size is aligned to @align and to @order_per_bit + * + * So, as long as we create ranges that have a base + * aligned to @align, and a size that is aligned to + * both @align and @order_to_bit, things will work out. + */ + nr = 0; + sizesum = 0; + failed = NULL; - cma->base_pfn = PFN_DOWN(base); - cma->available_count = cma->count = size >> PAGE_SHIFT; - cma->order_per_bit = order_per_bit; + ret = cma_new_area(name, total_size, order_per_bit, &cma); + if (ret != 0) + goto out; + + align = max_t(phys_addr_t, align, CMA_MIN_ALIGNMENT_BYTES); + /* + * Create a list of ranges above 4G, largest range first. + */ + for_each_free_mem_range(i, nid, MEMBLOCK_NONE, &start, &end, NULL) { + if (start < SZ_4G) + continue; + + start = ALIGN(start, align); + if (start >= end) + continue; + + end = ALIGN_DOWN(end, align); + if (end <= start) + continue; + + size = end - start; + size = ALIGN_DOWN(size, (PAGE_SIZE << order_per_bit)); + if (!size) + continue; + sizesum += size; + + pr_debug("consider %016llx - %016llx\n", (u64)start, (u64)end); + + /* + * If we don't yet have used the maximum number of + * areas, grab a new one. + * + * If we can't use anymore, see if this range is not + * smaller than the smallest one already recorded. If + * not, re-use the smallest element. + */ + if (nr < CMA_MAX_RANGES) + mrp = &memranges[nr++]; + else { + mrp = list_last_entry(&ranges, + struct cma_init_memrange, list); + if (size < mrp->size) + continue; + list_del(&mrp->list); + sizesum -= mrp->size; + pr_debug("deleted %016llx - %016llx from the list\n", + (u64)mrp->base, (u64)mrp->base + size); + } + mrp->base = start; + mrp->size = size; + + /* + * Now do a sorted insert. + */ + list_insert_sorted(&ranges, mrp, revsizecmp); + pr_debug("added %016llx - %016llx to the list\n", + (u64)mrp->base, (u64)mrp->base + size); + pr_debug("total size now %llu\n", (u64)sizesum); + } + + /* + * There is not enough room in the CMA_MAX_RANGES largest + * ranges, so bail out. + */ + if (sizesum < total_size) { + cma_drop_area(cma); + ret = -ENOMEM; + goto out; + } + + /* + * Found ranges that provide enough combined space. + * Now, sorted them by address, smallest first, because we + * want to mimic a bottom-up memblock allocation. + */ + sizesum = 0; + list_for_each_safe(mp, next, &ranges) { + mlp = list_entry(mp, struct cma_init_memrange, list); + list_del(mp); + list_insert_sorted(&final_ranges, mlp, basecmp); + sizesum += mlp->size; + if (sizesum >= total_size) + break; + } + + /* + * Walk the final list, and add a CMA range for + * each range, possibly not using the last one fully. + */ + nr = 0; + sizeleft = total_size; + list_for_each(mp, &final_ranges) { + mlp = list_entry(mp, struct cma_init_memrange, list); + size = min(sizeleft, mlp->size); + if (memblock_reserve(mlp->base, size)) { + /* + * Unexpected error. Could go on to + * the next one, but just abort to + * be safe. + */ + failed = mlp; + break; + } + + pr_debug("created region %d: %016llx - %016llx\n", + nr, (u64)mlp->base, (u64)mlp->base + size); + cmrp = &cma->ranges[nr++]; + cmrp->base_pfn = PHYS_PFN(mlp->base); + cmrp->count = size >> PAGE_SHIFT; + + sizeleft -= size; + if (sizeleft == 0) + break; + } + + if (failed) { + list_for_each(mp, &final_ranges) { + mlp = list_entry(mp, struct cma_init_memrange, list); + if (mlp == failed) + break; + memblock_phys_free(mlp->base, mlp->size); + } + cma_drop_area(cma); + ret = -ENOMEM; + goto out; + } + + cma->nranges = nr; *res_cma = cma; - cma_area_count++; - totalcma_pages += cma->count; - return 0; +out: + if (ret != 0) + pr_err("Failed to reserve %lu MiB\n", + (unsigned long)total_size / SZ_1M); + else + pr_info("Reserved %lu MiB in %d range%s\n", + (unsigned long)total_size / SZ_1M, nr, + nr > 1 ? "s" : ""); + + return ret; } /** @@ -241,6 +536,26 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, phys_addr_t alignment, unsigned int order_per_bit, bool fixed, const char *name, struct cma **res_cma, int nid) +{ + int ret; + + ret = __cma_declare_contiguous_nid(base, size, limit, alignment, + order_per_bit, fixed, name, res_cma, nid); + if (ret != 0) + pr_err("Failed to reserve %ld MiB\n", + (unsigned long)size / SZ_1M); + else + pr_info("Reserved %ld MiB at %pa\n", + (unsigned long)size / SZ_1M, &base); + + return ret; +} + +static int __init __cma_declare_contiguous_nid(phys_addr_t base, + phys_addr_t size, phys_addr_t limit, + phys_addr_t alignment, unsigned int order_per_bit, + bool fixed, const char *name, struct cma **res_cma, + int nid) { phys_addr_t memblock_end = memblock_end_of_DRAM(); phys_addr_t highmem_start; @@ -273,10 +588,9 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, /* Sanitise input arguments. */ alignment = max_t(phys_addr_t, alignment, CMA_MIN_ALIGNMENT_BYTES); if (fixed && base & (alignment - 1)) { - ret = -EINVAL; pr_err("Region at %pa must be aligned to %pa bytes\n", &base, &alignment); - goto err; + return -EINVAL; } base = ALIGN(base, alignment); size = ALIGN(size, alignment); @@ -294,10 +608,9 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, * low/high memory boundary. */ if (fixed && base < highmem_start && base + size > highmem_start) { - ret = -EINVAL; pr_err("Region at %pa defined on low/high memory boundary (%pa)\n", &base, &highmem_start); - goto err; + return -EINVAL; } /* @@ -309,18 +622,16 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, limit = memblock_end; if (base + size > limit) { - ret = -EINVAL; pr_err("Size (%pa) of region at %pa exceeds limit (%pa)\n", &size, &base, &limit); - goto err; + return -EINVAL; } /* Reserve memory */ if (fixed) { if (memblock_is_region_reserved(base, size) || memblock_reserve(base, size) < 0) { - ret = -EBUSY; - goto err; + return -EBUSY; } } else { phys_addr_t addr = 0; @@ -357,10 +668,8 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, if (!addr) { addr = memblock_alloc_range_nid(size, alignment, base, limit, nid, true); - if (!addr) { - ret = -ENOMEM; - goto err; - } + if (!addr) + return -ENOMEM; } /* @@ -373,75 +682,67 @@ int __init cma_declare_contiguous_nid(phys_addr_t base, ret = cma_init_reserved_mem(base, size, order_per_bit, name, res_cma); if (ret) - goto free_mem; - - pr_info("Reserved %ld MiB at %pa on node %d\n", (unsigned long)size / SZ_1M, - &base, nid); - return 0; + memblock_phys_free(base, size); -free_mem: - memblock_phys_free(base, size); -err: - pr_err("Failed to reserve %ld MiB on node %d\n", (unsigned long)size / SZ_1M, - nid); return ret; } static void cma_debug_show_areas(struct cma *cma) { unsigned long next_zero_bit, next_set_bit, nr_zero; - unsigned long start = 0; + unsigned long start; unsigned long nr_part; - unsigned long nbits = cma_bitmap_maxno(cma); + unsigned long nbits; + int r; + struct cma_memrange *cmr; spin_lock_irq(&cma->lock); pr_info("number of available pages: "); - for (;;) { - next_zero_bit = find_next_zero_bit(cma->bitmap, nbits, start); - if (next_zero_bit >= nbits) - break; - next_set_bit = find_next_bit(cma->bitmap, nbits, next_zero_bit); - nr_zero = next_set_bit - next_zero_bit; - nr_part = nr_zero << cma->order_per_bit; - pr_cont("%s%lu@%lu", start ? "+" : "", nr_part, - next_zero_bit); - start = next_zero_bit + nr_zero; + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + + start = 0; + nbits = cma_bitmap_maxno(cma, cmr); + + pr_info("range %d: ", r); + for (;;) { + next_zero_bit = find_next_zero_bit(cmr->bitmap, + nbits, start); + if (next_zero_bit >= nbits) + break; + next_set_bit = find_next_bit(cmr->bitmap, nbits, + next_zero_bit); + nr_zero = next_set_bit - next_zero_bit; + nr_part = nr_zero << cma->order_per_bit; + pr_cont("%s%lu@%lu", start ? "+" : "", nr_part, + next_zero_bit); + start = next_zero_bit + nr_zero; + } + pr_info("\n"); } pr_cont("=> %lu free of %lu total pages\n", cma->available_count, cma->count); spin_unlock_irq(&cma->lock); } -static struct page *__cma_alloc(struct cma *cma, unsigned long count, - unsigned int align, gfp_t gfp) +static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr, + unsigned long count, unsigned int align, + struct page **pagep, gfp_t gfp) { unsigned long mask, offset; unsigned long pfn = -1; unsigned long start = 0; unsigned long bitmap_maxno, bitmap_no, bitmap_count; - unsigned long i; + int ret = -EBUSY; struct page *page = NULL; - int ret = -ENOMEM; - const char *name = cma ? cma->name : NULL; - - trace_cma_alloc_start(name, count, align); - - if (!cma || !cma->count || !cma->bitmap) - return page; - - pr_debug("%s(cma %p, name: %s, count %lu, align %d)\n", __func__, - (void *)cma, cma->name, count, align); - - if (!count) - return page; mask = cma_bitmap_aligned_mask(cma, align); - offset = cma_bitmap_aligned_offset(cma, align); - bitmap_maxno = cma_bitmap_maxno(cma); + offset = cma_bitmap_aligned_offset(cma, cmr, align); + bitmap_maxno = cma_bitmap_maxno(cma, cmr); bitmap_count = cma_bitmap_pages_to_bits(cma, count); if (bitmap_count > bitmap_maxno) - return page; + goto out; for (;;) { spin_lock_irq(&cma->lock); @@ -453,14 +754,14 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, spin_unlock_irq(&cma->lock); break; } - bitmap_no = bitmap_find_next_zero_area_off(cma->bitmap, + bitmap_no = bitmap_find_next_zero_area_off(cmr->bitmap, bitmap_maxno, start, bitmap_count, mask, offset); if (bitmap_no >= bitmap_maxno) { spin_unlock_irq(&cma->lock); break; } - bitmap_set(cma->bitmap, bitmap_no, bitmap_count); + bitmap_set(cmr->bitmap, bitmap_no, bitmap_count); cma->available_count -= count; /* * It's safe to drop the lock here. We've marked this region for @@ -469,7 +770,7 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, */ spin_unlock_irq(&cma->lock); - pfn = cma->base_pfn + (bitmap_no << cma->order_per_bit); + pfn = cmr->base_pfn + (bitmap_no << cma->order_per_bit); mutex_lock(&cma_mutex); ret = alloc_contig_range(pfn, pfn + count, MIGRATE_CMA, gfp); mutex_unlock(&cma_mutex); @@ -478,7 +779,7 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, break; } - cma_clear_bitmap(cma, pfn, count); + cma_clear_bitmap(cma, cmr, pfn, count); if (ret != -EBUSY) break; @@ -490,6 +791,38 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, /* try again with a bit different memory target */ start = bitmap_no + mask + 1; } +out: + *pagep = page; + return ret; +} + +static struct page *__cma_alloc(struct cma *cma, unsigned long count, + unsigned int align, gfp_t gfp) +{ + struct page *page = NULL; + int ret = -ENOMEM, r; + unsigned long i; + const char *name = cma ? cma->name : NULL; + + trace_cma_alloc_start(name, count, align); + + if (!cma || !cma->count) + return page; + + pr_debug("%s(cma %p, name: %s, count %lu, align %d)\n", __func__, + (void *)cma, cma->name, count, align); + + if (!count) + return page; + + for (r = 0; r < cma->nranges; r++) { + page = NULL; + + ret = cma_range_alloc(cma, &cma->ranges[r], count, align, + &page, gfp); + if (ret != -EBUSY || page) + break; + } /* * CMA can allocate multiple page blocks, which results in different @@ -508,7 +841,8 @@ static struct page *__cma_alloc(struct cma *cma, unsigned long count, } pr_debug("%s(): returned %p\n", __func__, page); - trace_cma_alloc_finish(name, pfn, page, count, align, ret); + trace_cma_alloc_finish(name, page ? page_to_pfn(page) : 0, + page, count, align, ret); if (page) { count_vm_event(CMA_ALLOC_SUCCESS); cma_sysfs_account_success_pages(cma, count); @@ -551,20 +885,31 @@ struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned long count) { - unsigned long pfn; + unsigned long pfn, end; + int r; + struct cma_memrange *cmr; + bool ret; - if (!cma || !pages) + if (!cma || !pages || count > cma->count) return false; pfn = page_to_pfn(pages); + ret = false; - if (pfn < cma->base_pfn || pfn >= cma->base_pfn + cma->count) { - pr_debug("%s(page %p, count %lu)\n", __func__, - (void *)pages, count); - return false; + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + end = cmr->base_pfn + cmr->count; + if (pfn >= cmr->base_pfn && pfn < end) { + ret = pfn + count <= end; + break; + } } - return true; + if (!ret) + pr_debug("%s(page %p, count %lu)\n", + __func__, (void *)pages, count); + + return ret; } /** @@ -580,19 +925,32 @@ bool cma_pages_valid(struct cma *cma, const struct page *pages, bool cma_release(struct cma *cma, const struct page *pages, unsigned long count) { - unsigned long pfn; + struct cma_memrange *cmr; + unsigned long pfn, end_pfn; + int r; + + pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count); if (!cma_pages_valid(cma, pages, count)) return false; - pr_debug("%s(page %p, count %lu)\n", __func__, (void *)pages, count); - pfn = page_to_pfn(pages); + end_pfn = pfn + count; + + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + if (pfn >= cmr->base_pfn && + pfn < (cmr->base_pfn + cmr->count)) { + VM_BUG_ON(end_pfn > cmr->base_pfn + cmr->count); + break; + } + } - VM_BUG_ON(pfn + count > cma->base_pfn + cma->count); + if (r == cma->nranges) + return false; free_contig_range(pfn, count); - cma_clear_bitmap(cma, pfn, count); + cma_clear_bitmap(cma, cmr, pfn, count); cma_sysfs_account_release_pages(cma, count); trace_cma_release(cma->name, pfn, pages, count); diff --git a/mm/cma.h b/mm/cma.h index 3dd3376ae980..5f39dd1aac91 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -10,19 +10,35 @@ struct cma_kobject { struct cma *cma; }; +/* + * Multi-range support. This can be useful if the size of the allocation + * is not expected to be larger than the alignment (like with hugetlb_cma), + * and the total amount of memory requested, while smaller than the total + * amount of memory available, is large enough that it doesn't fit in a + * single physical memory range because of memory holes. + */ +struct cma_memrange { + unsigned long base_pfn; + unsigned long count; + unsigned long *bitmap; +#ifdef CONFIG_CMA_DEBUGFS + struct debugfs_u32_array dfs_bitmap; +#endif +}; +#define CMA_MAX_RANGES 8 + struct cma { - unsigned long base_pfn; unsigned long count; unsigned long available_count; - unsigned long *bitmap; unsigned int order_per_bit; /* Order of pages represented by one bit */ spinlock_t lock; #ifdef CONFIG_CMA_DEBUGFS struct hlist_head mem_head; spinlock_t mem_head_lock; - struct debugfs_u32_array dfs_bitmap; #endif char name[CMA_MAX_NAME]; + int nranges; + struct cma_memrange ranges[CMA_MAX_RANGES]; #ifdef CONFIG_CMA_SYSFS /* the number of CMA page successful allocations */ atomic64_t nr_pages_succeeded; @@ -39,9 +55,10 @@ struct cma { extern struct cma cma_areas[MAX_CMA_AREAS]; extern unsigned int cma_area_count; -static inline unsigned long cma_bitmap_maxno(struct cma *cma) +static inline unsigned long cma_bitmap_maxno(struct cma *cma, + struct cma_memrange *cmr) { - return cma->count >> cma->order_per_bit; + return cmr->count >> cma->order_per_bit; } #ifdef CONFIG_CMA_SYSFS diff --git a/mm/cma_debug.c b/mm/cma_debug.c index 89236f22230a..fdf899532ca0 100644 --- a/mm/cma_debug.c +++ b/mm/cma_debug.c @@ -46,17 +46,26 @@ DEFINE_DEBUGFS_ATTRIBUTE(cma_used_fops, cma_used_get, NULL, "%llu\n"); static int cma_maxchunk_get(void *data, u64 *val) { struct cma *cma = data; + struct cma_memrange *cmr; unsigned long maxchunk = 0; - unsigned long start, end = 0; - unsigned long bitmap_maxno = cma_bitmap_maxno(cma); + unsigned long start, end; + unsigned long bitmap_maxno; + int r; spin_lock_irq(&cma->lock); - for (;;) { - start = find_next_zero_bit(cma->bitmap, bitmap_maxno, end); - if (start >= bitmap_maxno) - break; - end = find_next_bit(cma->bitmap, bitmap_maxno, start); - maxchunk = max(end - start, maxchunk); + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + bitmap_maxno = cma_bitmap_maxno(cma, cmr); + end = 0; + for (;;) { + start = find_next_zero_bit(cmr->bitmap, + bitmap_maxno, end); + if (start >= bitmap_maxno) + break; + end = find_next_bit(cmr->bitmap, bitmap_maxno, + start); + maxchunk = max(end - start, maxchunk); + } } spin_unlock_irq(&cma->lock); *val = (u64)maxchunk << cma->order_per_bit; @@ -159,24 +168,41 @@ DEFINE_DEBUGFS_ATTRIBUTE(cma_alloc_fops, NULL, cma_alloc_write, "%llu\n"); static void cma_debugfs_add_one(struct cma *cma, struct dentry *root_dentry) { - struct dentry *tmp; + struct dentry *tmp, *dir, *rangedir; + int r; + char rdirname[12]; + struct cma_memrange *cmr; tmp = debugfs_create_dir(cma->name, root_dentry); debugfs_create_file("alloc", 0200, tmp, cma, &cma_alloc_fops); debugfs_create_file("free", 0200, tmp, cma, &cma_free_fops); - debugfs_create_file("base_pfn", 0444, tmp, - &cma->base_pfn, &cma_debugfs_fops); debugfs_create_file("count", 0444, tmp, &cma->count, &cma_debugfs_fops); debugfs_create_file("order_per_bit", 0444, tmp, &cma->order_per_bit, &cma_debugfs_fops); debugfs_create_file("used", 0444, tmp, cma, &cma_used_fops); debugfs_create_file("maxchunk", 0444, tmp, cma, &cma_maxchunk_fops); - cma->dfs_bitmap.array = (u32 *)cma->bitmap; - cma->dfs_bitmap.n_elements = DIV_ROUND_UP(cma_bitmap_maxno(cma), - BITS_PER_BYTE * sizeof(u32)); - debugfs_create_u32_array("bitmap", 0444, tmp, &cma->dfs_bitmap); + rangedir = debugfs_create_dir("ranges", tmp); + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + snprintf(rdirname, sizeof(rdirname), "%d", r); + dir = debugfs_create_dir(rdirname, rangedir); + debugfs_create_file("base_pfn", 0444, dir, + &cmr->base_pfn, &cma_debugfs_fops); + cmr->dfs_bitmap.array = (u32 *)cmr->bitmap; + cmr->dfs_bitmap.n_elements = + DIV_ROUND_UP(cma_bitmap_maxno(cma, cmr), + BITS_PER_BYTE * sizeof(u32)); + debugfs_create_u32_array("bitmap", 0444, dir, + &cmr->dfs_bitmap); + } + + /* + * Backward compatible symlinks to range 0 for base_pfn and bitmap. + */ + debugfs_create_symlink("base_pfn", tmp, "ranges/0/base_pfn"); + debugfs_create_symlink("bitmap", tmp, "ranges/0/bitmap"); } static int __init cma_debugfs_init(void) From patchwork Thu Feb 6 18:50:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2680EC02194 for ; Thu, 6 Feb 2025 18:51:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98E6A280003; Thu, 6 Feb 2025 13:51:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8C85F280002; Thu, 6 Feb 2025 13:51:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 71F5D280003; Thu, 6 Feb 2025 13:51:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4CDBB280002 for ; Thu, 6 Feb 2025 13:51:24 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0BB4DB255A for ; Thu, 6 Feb 2025 18:51:24 +0000 (UTC) X-FDA: 83090412888.07.5C8E2F0 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf01.hostedemail.com (Postfix) with ESMTP id 3ADA640011 for ; Thu, 6 Feb 2025 18:51:22 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3b2Ze1UN; spf=pass (imf01.hostedemail.com: domain of 3qQSlZwQKCIot9rzu22uzs.q20zw18B-00y9oqy.25u@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3qQSlZwQKCIot9rzu22uzs.q20zw18B-00y9oqy.25u@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867882; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GxP6S0l3dQ+6CkO+pIY1HsOnc8ShdfnkIOi767vYzT8=; b=dpJTKwUuYeoyo5v5wtGLdzVV0Bl7d1teHk5103h6GoppzrLVYYGjQyzePKIsU2BwWNZ45D zg+u0VzmYYnPrHrrxc23BmFfAMhnpc+s/xsJOKsaOcJE0uJ2bJMQ0vWy1CNRAga3K3FXXT +fihQEU1hieoGkG1ap1zFk1h0yb3npU= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=3b2Ze1UN; spf=pass (imf01.hostedemail.com: domain of 3qQSlZwQKCIot9rzu22uzs.q20zw18B-00y9oqy.25u@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3qQSlZwQKCIot9rzu22uzs.q20zw18B-00y9oqy.25u@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867882; a=rsa-sha256; cv=none; b=oTAUX/uXiEtODMCSo8wdT+/GkrWQ2uA1a7vkmvscM/GE6lIeiV8TtVHWABNK+Kgv5WQRFq 1Z0j2eody3jUQR26FGareCa0fXnRPD+F6jX3qILO6kb6jGS53LgJeXnchUdNBN53GPX85U 6tzJ2SCKedcRrbvQcZrEfHn3acvBSKs= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2fa228b4143so147541a91.0 for ; Thu, 06 Feb 2025 10:51:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867881; x=1739472681; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GxP6S0l3dQ+6CkO+pIY1HsOnc8ShdfnkIOi767vYzT8=; b=3b2Ze1UN9qAOebg9IX86+yRAzOAACgHQfNKnfZajlUdV2ljL2SaliFi3PzPh2sAYRR /GzQLEvXbByhUS82dI+Tuyfv5Hgj+0gG+3P15NGH0XrPIWx1BweZY1ZhrAhpSKcEXgQL kAnSxf7rD+0lpDBwEWU+zej5XbpdL2qiKeXBFA2Rv0eJ5LiD0x/01IX++Jg/znTQaOCd /bM2rLyPy19aol7UynBA2GmMmTXOp16nlVE//Dh9xd4u4SXBtO2kTUlLj/578o02cP4L ctBvw2s0A1G92dGBUNqQPeh1a98MFpze/ndz7tqwNE9kRuHtXyg0yj9rY1DRa10v5zgL MNeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867881; x=1739472681; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GxP6S0l3dQ+6CkO+pIY1HsOnc8ShdfnkIOi767vYzT8=; b=Tlp8CxWt4UxmrFzFBHwohiSB5YmtsxrzRZxsHLfKDJpoilv1KvVPgg0DUNRN4xzsuz 1rwUS7p0oPPKbpZbAPtkbwoewymKtj0ywhcc+Dknt9+qNQ56sP0xBTKChn7NWGSmle// cjx2Ho+rXdmLmE4TA73J8CqLnQbD1r7lvUnq/R2pNt49qnrdOeI9pwcERZzE6C16xkjn fxar83w0M41MUdR76fVWqFcgZDyDIAhekRfersk+9+WIkSdUBamd30RU89J5oaOZx2u2 887xJD7vqWXt0kCgNzH8HdsuHK7gJpiYPPABpyQRFNy1Z2Wixz2QBfkkf2eaJcgz90bM vQ6w== X-Forwarded-Encrypted: i=1; AJvYcCWBTD6+/duJumBLNY/2HEbKMyGSWLrg3pAqvDxh7r0SpCViy2ww9XGPMLQGtflCkr7Tyx6hqNC1VA==@kvack.org X-Gm-Message-State: AOJu0Yyh1i50VqEssBlDpRhyW9qTnjjFJ1MNXQoMqLjn9pk8o7MAba0b R81zmd58B50myxPz+8Uqo8+dlP8jZirfoxS+54kYBDv29XdbPd4sKUYm9q6C/za56PsGLg== X-Google-Smtp-Source: AGHT+IFhAWowVYwEoYtx6mfFOivywbt3Piw8sNnJL4ZJYNh4M1FuUx2ERPDZanvYPgCeFa64BAkgYsQp X-Received: from pjbse15.prod.google.com ([2002:a17:90b:518f:b0:2f4:3e59:8bb1]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:1946:b0:2fa:17e4:b1cf with SMTP id 98e67ed59e1d1-2fa247f9fcdmr44198a91.2.1738867881130; Thu, 06 Feb 2025 10:51:21 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:43 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-4-fvdl@google.com> Subject: [PATCH v3 03/28] mm/cma: introduce cma_intersects function From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden , Heiko Carstens , Vasily Gorbik , Alexander Gordeev , linux-s390@vger.kernel.org X-Rspam-User: X-Rspamd-Queue-Id: 3ADA640011 X-Stat-Signature: ggj7wqashna1ofuf9w6ongdb63bjpktm X-Rspamd-Server: rspam03 X-HE-Tag: 1738867882-766853 X-HE-Meta: U2FsdGVkX1/aDV4fBgqRsfyrwX14prMSbRej+xhLXCBDqWfj29wxZ+bvFNVIXiVB9al+DeLu4koOlXO10nIbQ5Kj6FzDcjawmnHYnYDZx/569dimNCyo2JuhrShm/U0Vs/fUTaRBu9nvOCZBJQeWH3LZP09iaFAyqIckLfFMswNrfQxpgugvskdp/Hd/4J/zVwbrgCqImtK7YyAuE1e3Pk5TjTxbirwgf7wwUA/uxP/HAGSLsrYcu/TsbOxZCSifwADLjEzFoWxckCnr4lRHKaoI60TirDyCFDYOB5bDWjsc+uy//VdHIUzU0+wWt3RhYAMFx5gVZDSC7fWgeB94OoYkrs+3uPSLfDgvD3f2HDAtrWjBDnsO1O1/DLCyJt0WYGVid0/w6yZkKz5SjS2FkVArkwuYzsb50yahOQCWTOK7B4wZJQXASb5GOiCjK/5PlaUr5yKNuBbG6d4djelJv0Su6zYpefUeOSgc0ZyhSOxJKWAvfjaUjPW26zeUpYnPJYHD7aVKNEeZLDWjQmri4PUrqJnTmfCtij8CkF23g5npL4oHvZ1SvhdmIMOCsDwNQYFK1rNLHkeHq2Clpz6nu8i7BcWUrjmdmMhdU+2i9todymO5OLMNWlpOK6SIJ8RVsM8Hc+WJnlhVIVB+OE8mKh+X/e/ovL1Tpt8vtVR0LOsq49Pv+be9Qi6bLJZrY1agD+ZRBhVgu27T8jKLoauIjvTRCpijySURwqjWHLF+Fv5CGbN2phK4kkQMMVtwCUF/LkO1IKS2FexDd+QEfFjxUzTGF/p0bzR0tV6HhCcWiodwSPemeK4hcrqQP05zhiBXyREb2x2xEygn8NAGbYxpb9ZPvQ3Y8d8kNDbu6ouxPCWQAGBBkSbJ2dGoxbo7pF9WgLP70VB2zb56Gkbv8GOz+/1q8hTcwFUnG6yd8Pxkhtqpwu3UoMxtnht7VyFVmPrebxRlPOAOushm3dV57dF SxpT7Kw8 Wd55oC/iXkosxXFUk+oIvk7P4UyKhWxXCy9XmWIhGtfkrpEnkyv4VGckPuIkFsjJ9YxkgwKb9pvKPdQOpHIRVxEijDlvoP7ph3yjmX9b2rDiacc9o8uCKNwnE5qixhJuS+lv9xR1mn9Kxyb0U/EQy6fXDgRzGgICcwgKUZWdasG2x/raHMy/z3MkW3jFxFqRaDqpdWWHcpjYZTFZb+II9MVO67x8RyI4YSnXICgJzmwOZely35jl/UjD0Urc+kwKrK7ca+OxxSfzVVlYDgxPVbO7760wdGoYMGo8Rzd395EKbcgz/PV5si2BaDh+HGhMNEXiQUYoC+kWRcInhdULNC6VC82LIKFEprzaS5ZHik/M8/Uuz2NyFczNHcALzD0mH0lAG8sQmaxqqvqzadeqXFlZOGS6eFecV+1V2wV0MnZAleLO5Z5yKYG2b8yT4szmBxcb5/EpAPQA8mC/RtpoBsgi+0o8CvJhR2/1wPHCYC4vTUcrJ1N1967vC2gDKwueRcLIUvpaRisoENxCF5sUCZjIeBIYr1NUDhiADsyNSnqiaoOGSjrHRC5UOGmo0gUyDx55jhuULjvR/Ukz/D81e8WqWtqhJBUCOenwYAOr6OCoVMzcJ8HyrqVJGxQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000852, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now that CMA areas can have multiple physical ranges, code can't assume a CMA struct represents a base_pfn plus a size, as returned from cma_get_base. Most cases are ok though, since they all explicitly refer to CMA areas that were created using existing interfaces (cma_declare_contiguous_nid or cma_init_reserved_mem), which guarantees they have just one physical range. An exception is the s390 code, which walks all CMA ranges to see if they intersect with a range of memory that is about to be hotremoved. So, in the future, it might run in to multi-range areas. To keep this check working, define a cma_intersects function. This just checks if a physaddr range intersects any of the ranges. Use it in the s390 check. Cc: Heiko Carstens Cc: Vasily Gorbik Cc: Alexander Gordeev Cc: linux-s390@vger.kernel.org Signed-off-by: Frank van der Linden Acked-by: Alexander Gordeev --- arch/s390/mm/init.c | 13 +++++-------- include/linux/cma.h | 1 + mm/cma.c | 21 +++++++++++++++++++++ 3 files changed, 27 insertions(+), 8 deletions(-) diff --git a/arch/s390/mm/init.c b/arch/s390/mm/init.c index f2298f7a3f21..d88cb1c13f7d 100644 --- a/arch/s390/mm/init.c +++ b/arch/s390/mm/init.c @@ -239,16 +239,13 @@ struct s390_cma_mem_data { static int s390_cma_check_range(struct cma *cma, void *data) { struct s390_cma_mem_data *mem_data; - unsigned long start, end; mem_data = data; - start = cma_get_base(cma); - end = start + cma_get_size(cma); - if (end < mem_data->start) - return 0; - if (start >= mem_data->end) - return 0; - return -EBUSY; + + if (cma_intersects(cma, mem_data->start, mem_data->end)) + return -EBUSY; + + return 0; } static int s390_cma_mem_notifier(struct notifier_block *nb, diff --git a/include/linux/cma.h b/include/linux/cma.h index 863427c27dc2..03d85c100dcc 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -53,6 +53,7 @@ extern bool cma_pages_valid(struct cma *cma, const struct page *pages, unsigned extern bool cma_release(struct cma *cma, const struct page *pages, unsigned long count); extern int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data); +extern bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end); extern void cma_reserve_pages_on_error(struct cma *cma); diff --git a/mm/cma.c b/mm/cma.c index 6ac9173d3a7d..c7116a5819c5 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -978,3 +978,24 @@ int cma_for_each_area(int (*it)(struct cma *cma, void *data), void *data) return 0; } + +bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end) +{ + int r; + struct cma_memrange *cmr; + unsigned long rstart, rend; + + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + + rstart = PFN_PHYS(cmr->base_pfn); + rend = PFN_PHYS(cmr->base_pfn + cmr->count); + if (end < rstart) + continue; + if (start >= rend) + continue; + return true; + } + + return false; +} From patchwork Thu Feb 6 18:50:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963552 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCC58C0219B for ; Thu, 6 Feb 2025 18:51:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B2B9280004; Thu, 6 Feb 2025 13:51:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 74552280002; Thu, 6 Feb 2025 13:51:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5BCD0280004; Thu, 6 Feb 2025 13:51:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 30A55280002 for ; Thu, 6 Feb 2025 13:51:26 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B46D81C90BA for ; Thu, 6 Feb 2025 18:51:25 +0000 (UTC) X-FDA: 83090412930.03.9551194 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf06.hostedemail.com (Postfix) with ESMTP id E80C518000B for ; Thu, 6 Feb 2025 18:51:23 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=V2wguMqd; spf=pass (imf06.hostedemail.com: domain of 3qgSlZwQKCIsuAs0v33v0t.r310x29C-11zAprz.36v@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3qgSlZwQKCIsuAs0v33v0t.r310x29C-11zAprz.36v@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867884; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RHIk1zAzNzZzzd4zq0SrZCaGEXgiIYu2uBl1eqTVpf4=; b=wLd45LKvpQEs9HD0i8/trSmf+i1JHe0TkTD8scp5TYbM/cUnoEC7+jOzIZsvtbBoNv6odE Y2m9yviLW7OVgxIKd00bedBobR8kCM++L/P2xfOB8nLsV96jJ/6a+u1/c9t+L6a2uk9+NQ qlxmbi2jRHUPptEMb+kMJcWzPaPsuWY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=V2wguMqd; spf=pass (imf06.hostedemail.com: domain of 3qgSlZwQKCIsuAs0v33v0t.r310x29C-11zAprz.36v@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3qgSlZwQKCIsuAs0v33v0t.r310x29C-11zAprz.36v@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867884; a=rsa-sha256; cv=none; b=DiduvxenDG7BCvwmue3YWLTOHG6rIi7jisc1QxxcgpVM2E69dumSorlexzWri0youSELqB CnHY3XUTeWEVPJ4PLsAKtwQvqOeWOdf2CQe2aUx4vZOMLFM886pKHu6KFJICcsyeD9smEY grNDh0j8fFLJDi5pa62VJHzeiAarinM= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f816a85facso2747663a91.3 for ; Thu, 06 Feb 2025 10:51:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867883; x=1739472683; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RHIk1zAzNzZzzd4zq0SrZCaGEXgiIYu2uBl1eqTVpf4=; b=V2wguMqd/OhM4j+u/J+yczwN97hyluhDSMkJIDeO4GDVOxRbnBZg+ZpNOPiXh7nHBy zKs3aixz82djKA9m0OMCERzH7TXQbSlyR6Tuauo+Byk/nP4XDmLJx5ZFU2H5fej/PZ1U w0nXzZ8MolBJB3alDDWPjkawqankTrie/YTMQqlueuVYfxjnqw5tp3p/bAju5fobtpS9 YGvzHqUYEPlHj/mSCAW3PGkhYDDkRCuvXZRiNQBbZQi57DAWoLtqkPPbMVIltLU4JhKJ JI1ECmtDP8FlLplxmUnpltoZIYfVLgg0/oumCuDOKZofCPd+QfPUNYUT9kAy4HdKTimH f2Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867883; x=1739472683; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RHIk1zAzNzZzzd4zq0SrZCaGEXgiIYu2uBl1eqTVpf4=; b=rCqidxIB1LlAE+ZrFfPWpnHHKOwEFK6Uopae5xa+s9+DVtMKY1nS6WNaUgY5lNnYe0 OIfYkT8nk0JAx+59V8IV6UErkgMuPtf1XrpohgLlmKpMNmd/FU8zf++u115PDiu2/yVA z5Ls8d1AoYNcavp0XIl9mt6C5ea9AKRxrD25lQ1fxTLOXN7hIav6uicDfmEX/C2ZFOnN X0XiB4zBt9YDHcr6nm/nsn1mKHAGkh1JVCNxU2a5B70/e7r4wpQ5+tvgm5NTUWN/tk6Q disxDjykXYxh3b/GWbeLFAtk4lMVATmSxplNOUlJluc+8hJcHOUL3wo7Nzk2dRXNTXvO RmfQ== X-Forwarded-Encrypted: i=1; AJvYcCV9yKTucztax7HtKs9OK7LZWN8+Kod1IUGs4hJdZdwud3hmvdFOsWfYaaDR4iTBj8wEPEpKPnSTHg==@kvack.org X-Gm-Message-State: AOJu0Yx7bw+r1gBHmzYwC4xfvjAZcZfsOsDQrhN6rOiCCxPXAOZ9g6QJ OTAqXjrRC23gtkM8+f9XXll4Ri0RH4tIHIDNUbPYBSUegje7a3xbOuOKBZfJuwkQddH8lQ== X-Google-Smtp-Source: AGHT+IG2C6Tqj4tol1eUNf+3cFodAFg6bV6h55pukBTkPfnQj74WsqcAVhR4si2RwXlaV64ND0j3yoTj X-Received: from pjbsi17.prod.google.com ([2002:a17:90b:5291:b0:2f9:d9fe:e723]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4fc6:b0:2ee:cd83:8fe7 with SMTP id 98e67ed59e1d1-2fa243f666emr158067a91.35.1738867882861; Thu, 06 Feb 2025 10:51:22 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:44 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-5-fvdl@google.com> Subject: [PATCH v3 04/28] mm, hugetlb: use cma_declare_contiguous_multi From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E80C518000B X-Stat-Signature: tw4z6a8jpwaz893moy6mzndxsmzytehz X-HE-Tag: 1738867883-854151 X-HE-Meta: U2FsdGVkX192Dz+KJFs70GuMMuu732IPHlZCSiC51iW+Agp4t1WZnEQYxS67aeEWQKAEWu7Gqsrq7CicCImyzZXUickAYmTaVZLjd2UR/mFmw2j7DMB5PUe7zWbFu59lOOMpkiVmCAbg6gvqqsgM/5kjZSRPfxnzm59lO2GgE1wKoo+FIaOW2E/ruBdwjTzeEoEeJtAKyX9XHao/HRhso3drrSWeLLcG9BwS8jzpHMzqZd4b61eJA2A8FKVvNb0oxXQo/02KODwuIE+yDcn0qOjHOjR6keXGLfaXqtOcEqxv/am++zpyXcjxoS8eP1RYybVyYwJUb1P+k3SaKgBbhtUHIrJcxyOlKdinpCTCFG3w/qMdAMF9xnOhZQzdVKiwwJyC+OmKTp4Su7O5ZTTA06ikPyskY7cl1U5a3u0KNs+opztmPqorxiIWRdT5CODby4GU4ygyA4s0lKrMkIOxiyVT415TDMztlPy7052aqezhY29unyBA6Y8YEFaK0Xjv8Yv4YmmdGKlsSulkfOHYZTN4rAS/CfIWJ51Im0pBZslKls2eJttWhQEkE01Mq+MuIDtvPsui5/dPTaHlTARQVXEzLBHYGWhBoVKjLDSXtvrh3zkxtKrLkS+vPHi8OtvvjsAYZ/OKv9F5Q/W8MpcwBQHx25ge74k9Qks0Kje4FCLe973dSQfdOIgPIuTz1cSo2+LcqNPgn5ya2ZSmJw6eMSMK4W/Ht0sz4gNigkRVYU8rpFH7ivdf8CyBG6xvTsO78BBv3+2b06NYhvbAhghKfM2VbyLtBX6Mjv4aPRF78bg5FckmJaeJ9J4wRttDiTsdz7B1RbHiWU+fB21XHKhGtKpgHOWYbdkt7ClZ/9fnhexEcRdpQeURJBri3aXAQkeqiF2jLs0RFbFxTnN1P9PlHTrvfpyRLCLRSRqd1M4GUm1ZFmsWV56k0FJHgeWnm3y/iKT8AFhJjsJsr0d996o GIrdEFeI O0woxZUzG/QQVnQlzrcIyPDaDBIEQNgWm/91qoofY8ZUI/PSnBbzdxtqCxU58jwU260ImkKiVXVdB5d6mVQez30i4lwIGxgfOzSoMshYlgyTSfnUcyLo/T6mOJ74bWuW/sX+0sW03S3oiy3CXq3ZBVK1YVrY2oV7UIzg1Eq5m1SVAuNjm6FlmaRDPyG6HQ47pn9pPM75EOmEeMYMHykTZOMYal/Odu3qJibOvGmPDd2qY8imdSCYwqSbprLkKjv4J9jXxARaK3Ixlf6YFsR5b77bO1kTBHxfLJ9QeetxkX4UD9pT50euiQeCSprKSQpiLC2F3eg6M/AB3vzW3l8gXoPnOZo8AHkWHPF1Cf6fQFHlFTab2VdTJs/eGrcL5vfIp3+mFTuJTHibnJKLXLcjmGEW+afXqH03C4zabOHpLQ3FG7F3pQfTjf9S6vYAKJDchcF2WbJGDjFTt2IwUjEk9kM91pG0mQ7wN2ukdy+jsWJznAOTlvRoHksaupmIu1D4SQVezjBkEgHeTsHodnI+lPO/IVjO/ZQ9XKVzW++NFwKfAXkyrZvEDQfEA77PDX0IREYos X-Bogosity: Ham, tests=bogofilter, spamicity=0.000143, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb_cma is fine with using multiple CMA ranges, as long as it can get its gigantic pages allocated from them. So, use cma_declare_contiguous_multi to allow for multiple ranges, increasing the chances of getting what we want on systems with gaps in physical memory. Signed-off-by: Frank van der Linden --- mm/hugetlb.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 65068671e460..828ae0080ab5 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7738,9 +7738,8 @@ void __init hugetlb_cma_reserve(int order) * may be returned to CMA allocator in the case of * huge page demotion. */ - res = cma_declare_contiguous_nid(0, size, 0, - PAGE_SIZE << order, - HUGETLB_PAGE_ORDER, false, name, + res = cma_declare_contiguous_multi(size, PAGE_SIZE << order, + HUGETLB_PAGE_ORDER, name, &hugetlb_cma[nid], nid); if (res) { pr_warn("hugetlb_cma: reservation failed: err %d, node %d", From patchwork Thu Feb 6 18:50:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 745F7C02199 for ; Thu, 6 Feb 2025 18:51:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2B2B9280005; Thu, 6 Feb 2025 13:51:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 23A4A280002; Thu, 6 Feb 2025 13:51:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 018AB280005; Thu, 6 Feb 2025 13:51:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C05A3280002 for ; Thu, 6 Feb 2025 13:51:27 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 7A521C1408 for ; Thu, 6 Feb 2025 18:51:27 +0000 (UTC) X-FDA: 83090413014.01.93CE20A Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf18.hostedemail.com (Postfix) with ESMTP id A79A91C0009 for ; Thu, 6 Feb 2025 18:51:25 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zy2k50f7; spf=pass (imf18.hostedemail.com: domain of 3rASlZwQKCI0wCu2x55x2v.t532z4BE-331Crt1.58x@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3rASlZwQKCI0wCu2x55x2v.t532z4BE-331Crt1.58x@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867885; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hzoPqdOikoObjXWU9SvKmiO4GzZxuupFnL9M3tsuTWc=; b=mndFF4JjBjv8/6kVeOonF5n7fjaxOaw9VeoEBzzub1+bBaIJyVpnnvY86hkxW3S4rvyaLi MaNQ5w7T1fvgiEt4KWBsb7j2yV0ah2IenXiJdXxDR9j5TXF2O2WWof1+ce+YHj1fIw5wVj WRbm3IUMqUsBGTCqWfab7O2K8YdtK2c= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zy2k50f7; spf=pass (imf18.hostedemail.com: domain of 3rASlZwQKCI0wCu2x55x2v.t532z4BE-331Crt1.58x@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3rASlZwQKCI0wCu2x55x2v.t532z4BE-331Crt1.58x@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867885; a=rsa-sha256; cv=none; b=aWkW/FI8m/eo+jVn8+Ba46gqPsYdUBPT5Y6Uj1jaJDitcXowURL6XB14a6AFe/8Azt1S1W 5JIdCQdAgX9NFaIZJJnFsC10jvLdyUPf9sTIJ2xJIEtVRLysQdQTZcXQJt1HU5R7qKkXUD JF71e5C0TlYFe4erF8t0+rDT6p78gwc= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2fa0204ad21so2418053a91.0 for ; Thu, 06 Feb 2025 10:51:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867884; x=1739472684; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=hzoPqdOikoObjXWU9SvKmiO4GzZxuupFnL9M3tsuTWc=; b=zy2k50f7MEJc/j6GLa9rL5aVpSQyKC6IP/PWgVcvlxQVHzwYEKSLYc+dhMxi7LUfDv Xp34TZANb/dbMzzKT3yKhETd2dsNAsMvHMfA+vzMEm17/YUI1ABG8NGYhGTikCepCIhI uXNcrav6FsbMyL2FR+OrVuHil+IbIV0ncVUrS61qh3Eq/XrgPA4KBEz8khHBcoMABkye ffUfK/TC8QETmMvHCvK2aiM7rdmZcp4MFGKm/wgnGjnTwPfy84WzWXNIu/4Mog6jF9m+ VOeiIqioT9rlMcSzBiJEDKrOsJsL+ZYTPFfU7A5ruGUiDMfFxCYX/3jOcYAL+0Wor3oJ SR6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867884; x=1739472684; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hzoPqdOikoObjXWU9SvKmiO4GzZxuupFnL9M3tsuTWc=; b=fd9ZHuIb5rP7GXfELQS+vNJ1y30sfIQaev0XWvaRMXVzIiQ43Y9VDe5y7QVE4ayQyb tZglnkxztc7geyB7IEF6WdsujUBSoAZmhinc6QR8a2gyUJ3vfvf5caT9wZNsYYzIeXNb h/fOyM0evcclv4VpGd6CEuADaD4Y7Sp8mUMaGFHISJqGRgpN/UmBf8to8ch2YUSvxcRy 4dizqvrKNdJEToaxrAK6Hd2doS34Kx1dVnt9jBvjj9T8OPJYb75zSszTx8gG2B7lf/ul AG50j095h2ulm8YjpVdB90DqioE8hcki0eI5mUsxgiPPZe9mcJ0Tf3ZSmC3sztHMtvXc ezVA== X-Forwarded-Encrypted: i=1; AJvYcCUoZYNIUyRHrv/Rqvi0XxK6oDmxWJKZ8XW/NQJAvpLJtFdeRggj18UD6nktbXQ2SQQ1GyoZ6Gv4pQ==@kvack.org X-Gm-Message-State: AOJu0YxAY8KFU6wIXKu89t9UPc67K9+g7aGKOd85EtFqIGaz9K784+3z YGjvJgCgj4vRtC6pAgkur7ljr+JGiKvip/aoagXcg9KLvkNnc1zHfFg0WBkxEjQFU9bbhg== X-Google-Smtp-Source: AGHT+IEL2yhkPaKLLzs2ZR25Ntw2PD6TefD2snZgeFGU7MfGyptfuCLCB6W/h56WjY/xARRU7WJJptnX X-Received: from pjbpl9.prod.google.com ([2002:a17:90b:2689:b0:2ea:aa56:49c]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3c0e:b0:2f2:ab09:c256 with SMTP id 98e67ed59e1d1-2fa243ee52dmr179111a91.33.1738867884627; Thu, 06 Feb 2025 10:51:24 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:45 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-6-fvdl@google.com> Subject: [PATCH v3 05/28] mm/hugetlb: fix round-robin bootmem allocation From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden , Zhenguo Yao X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A79A91C0009 X-Stat-Signature: qggaexm81nb1ft57dx48jmcugjqkg7ko X-HE-Tag: 1738867885-864575 X-HE-Meta: U2FsdGVkX1+rXEWEWLDDJeypxJg8ZyCXetSVYVx//JPmE7KZQ4i/iS8OtQYj2pIUEDdFulqV2OzhNRTYjiJ03WuEqLDAsH1gUMFzei2GSuXvc8OB6QV4IzhXdDMnLBWqULuKtrex3b5GUrKGc1mUTxFZjjtTl3l9d2SII1I78auv8as6YIKTWo1Of2aYQvc3gqvGfUhv2PkWiMO2mjhUEvLxMsBHVx0Rj0QEzPqwdPfYur+QvAIAnVS4PpIWyniuNBf9QNMIod5sjRQd9Yqrpbp23PRFud8XmVGU6FYP/yOdFmq2yJV0jozoa+cKAeoSyRjPAWorDxPaHSdQS5TZMfQV2OwdN8jJwstfQb3Z1lduEkkEETBnz8fK8YM8jUFmk9qqtMDT/m+RRRQ46KAZaLjpWVomYEkWdxq6TgEiOrd/+nQ9McEBFR5LBO4/Cgvo9TVIigg/4vWWBAcD2T9dg74xdAy6YnGNEmblJtEw692nBgfoD3SPuaTQDrl6RNwX2Nt7JMrVsWWYfTDKzAFyB182DnrPt4rgYtHGNI0hghRyd5++zkxNkifXmkrRQIDU2JWigrS+SotxtL+W/vpUSZr97qE940xYlufvjf1iMtmuDHusIUUFlzFBIwb++TxdXOyQ310MmpwMVHUrEakS2pC8WqFaA6+G8/qAzXKQMbz7vmgYyJudoXqP386mPTBDrkhY44azlLzXl0BFFmLYSWc3uEW42qyTzC859FzBZzT4uS0KtvbDanrbs5J2Grb0Kf/oZztOkW8lgE0/XWAMVcJTWDfSvdCxVkyoLUcNV5d45iUl9SdN0XT7KzQ9vGUFPaGnw8chNhwLU3CDHhqS/BS0nwiCHEglrWlO6C5eT4DIMYaC3XumoOJaGy0i4jpAt84XNhgz/yjG8FS5Xvwj9XN+Mi/shG+WpC2jF5UXW+MwAoFPWQAwCNNo3Kkuy0JyDsWp+VnJv66LRXtKS0w mQYkJtuN gdHb3GXjoR8s5vRnLrBtQRH+yBZ4mfjze44OEW5w/118enfdb6eGMWxDD133phSsL0OiWlITwB8f4nlaw2ETceEZlsET5ouiCvZbP9Jh6vNKylnYHNw4a9O81xemKiZvop047t1nG2zRQcRGYZugnzj6MuZK3iKa0ft1gSKEfQ816LGpAOClMIvN1vyMVvTmvM7yfgM7mjRGySVcnyZ65is8Aa9f/3zrZWhVxu4LbSMGeY/67SvFsO3NOKDVXPJTFnprKJXqgn+NR8+INMGlvBd28KiAi3lH+j12XIxSTJvBTb6le3Zok9Z9yi8euSJbFnB+0YKko4IEuK3ftb+lbO11La4sl0FdZG82SYjM+IXelUp9JGkvWzJPETyF9JPI+xNASf6yuBnxpyMleGCDo0SgryfMkOKba/NOt9tUR8ZPmGSkgYBZwrgFDUT1i5bTEb0CuDrQ/yEy7vrtmZRXT4+kLPjGSXP4102x0pqhlAGozI6tanTXoBWxY58LaEkMnZ0ebYDIWMZHsqjUuDSk/g/k/kLVCbM9E90lVJlnzFjpY2iE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit b5389086ad7b ("hugetlbfs: extend the definition of hugepages parameter to support node allocation") changed the NUMA_NO_NODE round-robin allocation behavior in case of a failure to allocate from one NUMA node. The code originally moved on to the next node to try again, but now it immediately breaks out of the loop. Restore the original behavior. Fixes: b5389086ad7b ("hugetlbfs: extend the definition of hugepages parameter to support node allocation") Cc: Zhenguo Yao Signed-off-by: Frank van der Linden --- mm/hugetlb.c | 22 +++++++++++++--------- 1 file changed, 13 insertions(+), 9 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 828ae0080ab5..1d8ec21dc2c2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3156,16 +3156,13 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) m = memblock_alloc_try_nid_raw( huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); - /* - * Use the beginning of the huge page to store the - * huge_bootmem_page struct (until gather_bootmem - * puts them into the mem_map). - */ - if (!m) - return 0; - goto found; + if (m) + break; } + if (!m) + return 0; + found: /* @@ -3177,7 +3174,14 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE), huge_page_size(h) - PAGE_SIZE); - /* Put them into a private list first because mem_map is not up yet */ + /* + * Use the beginning of the huge page to store the + * huge_bootmem_page struct (until gather_bootmem + * puts them into the mem_map). + * + * Put them into a private list first because mem_map + * is not up yet. + */ INIT_LIST_HEAD(&m->list); list_add(&m->list, &huge_boot_pages[node]); m->hstate = h; From patchwork Thu Feb 6 18:50:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963554 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 134ACC02194 for ; Thu, 6 Feb 2025 18:51:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 151C1280006; Thu, 6 Feb 2025 13:51:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0D9C0280002; Thu, 6 Feb 2025 13:51:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D5E59280006; Thu, 6 Feb 2025 13:51:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AAB3E280002 for ; Thu, 6 Feb 2025 13:51:29 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 282891404F4 for ; Thu, 6 Feb 2025 18:51:29 +0000 (UTC) X-FDA: 83090413098.04.AA557B5 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf17.hostedemail.com (Postfix) with ESMTP id 4F5FC40010 for ; Thu, 6 Feb 2025 18:51:27 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1cyBNem4; spf=pass (imf17.hostedemail.com: domain of 3rgSlZwQKCI8yEw4z77z4x.v75416DG-553Etv3.7Az@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3rgSlZwQKCI8yEw4z77z4x.v75416DG-553Etv3.7Az@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867887; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3krrgaxsXaFa4uf9iEDSe21apCBKUA0Oh8FsqZLKWjo=; b=3nYdSIRCwzmEbRs/+ee95KAtrfMMm9aiRSEeYB6bdd79XC9jiEdWblle6dMM3D2QSGlBlL tv9Ke7UcsziZ8IyRLelXIg/r9rNDi7dNNuYuYdFZ1abpgElhREvD/uQoytg4seTNv0hfLG pQKdTeZ4HNaHOFLrQzFqZGBbqoiX3ZY= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1cyBNem4; spf=pass (imf17.hostedemail.com: domain of 3rgSlZwQKCI8yEw4z77z4x.v75416DG-553Etv3.7Az@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3rgSlZwQKCI8yEw4z77z4x.v75416DG-553Etv3.7Az@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867887; a=rsa-sha256; cv=none; b=j2eIDl+YcGm5ZjOi6C17+5dAg2joV6/qw6upLG/bYMfJgJ6Nkb65tD0US3hsn3knC6od+r IxfBLWMwsi6IUMiTnXR5IloAwmenvg21U4SMdUkbvHg023DC6O2Q/TaU8LqY9EdRkdp25u fwRPbWv7KKuC1V8BwF3vASs6FRbgbV0= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2fa228b4143so147780a91.0 for ; Thu, 06 Feb 2025 10:51:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867886; x=1739472686; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3krrgaxsXaFa4uf9iEDSe21apCBKUA0Oh8FsqZLKWjo=; b=1cyBNem49UTWqwNMq+x6BfnW1/X/a9ZkL/bOApU6maYywumUfezgYX17KsoLcI0ru7 KwTF62LVo60XWo4k1JQEdOXmcNb6efyjv4EcUL2WT0k6uwbCxzlT/ba4rvfJ0nnRNMeX kJ88YKsGFZl9O24blLqKLuidyqhb8T3dW/0sGqBgjbwvsZks3WDLJssRqkoStXgIM25R PMaKm3o7fQQJf5VsC4R6btAtrj7ns33y7FTNvYZjf/a9+xNCCjv7Y74IwvLpMF1YPGre aT4wl7dcfk+TMPxaRR1CnFSHz8YsuCS8rn+BHWE4yDse4N9sAuFWXWMTKUCbAuLVeLOJ Tz2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867886; x=1739472686; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3krrgaxsXaFa4uf9iEDSe21apCBKUA0Oh8FsqZLKWjo=; b=G4cOasBMQ7imYXp/AUw5XazV1F0u1ZpaEG9+Q8TZ/sHQxvYmxJAbFRIuF33YJapJm7 P5IF/Lrr0/aoMdUKBpGjecL19LNFhKJ9cqOlgrF09QP4j/UXs1sN8gOJlvu4++W6oYPb 86pmNJUSsVJP02heLrwZBShexc1WFfQd+gE3l3DQCKLYTV5dmdXDRdbGORbJNL+OJUHo +w7Muw/cDplzcItwQZbJIOlAJei/OxnxZobXOEGy3upAT1AzTUKcSWI0MzNTTkrl0y04 bBkxRZ+uPbaP1Psce4S3FzHP8VC5D7jNUxpx65KET74wLHYouYB+jiasW+bVwB3EyDzQ 7D2g== X-Forwarded-Encrypted: i=1; AJvYcCWTqAb9rWTgsbXumga3R2CLq2w6+L13hehVQsDE242u3q1cvV/OIqqyn/PO7t3EfGPc9q32irez8g==@kvack.org X-Gm-Message-State: AOJu0Yxr9N4BSXPeTks+JSxBQ61SUjZBvjj6lIap5UUGtzOhhsWRzt1B ICLoJ6zvmbqa+hHcI/GL8J2HfntKEIdxn3XoP3L9fscwNaFAQN4KFgax0CY3Hi1ST/xJ6Q== X-Google-Smtp-Source: AGHT+IHdn9ARX0BKU2aoxnDMJSfDfQ2JxdDOMx8aC4l0xtpEE2SmhHzHCIxuEt3YV/CF1/5TaCD9NFnV X-Received: from pjbst7.prod.google.com ([2002:a17:90b:1fc7:b0:2ef:95f4:4619]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5282:b0:2fa:9c9:20a3 with SMTP id 98e67ed59e1d1-2fa24700cbfmr107801a91.0.1738867886262; Thu, 06 Feb 2025 10:51:26 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:46 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-7-fvdl@google.com> Subject: [PATCH v3 06/28] mm/hugetlb: remove redundant __ClearPageReserved From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 4F5FC40010 X-Stat-Signature: nybk567kh5frtcwrqy4uqbc8u7qst7rx X-HE-Tag: 1738867887-905659 X-HE-Meta: U2FsdGVkX1+uNIAZ4clttDkSxbEIoah5qwhHI6aTW3mML3wpTdcUbZCGF6HhLGlfJnAohgG1UfRxom1pAWQA/xRnwDDUsWSt+fLgUDyXO8cEIXKfnJiMS6MpHWoIpkMUEWNO3EOA7Uq46AljkGe5mrumtdmJKwoCiQ1e8S6Ae/W79eF05VuCjR617lFrxUgo6u7tk7HsSVTUOY6vyjiOtSvSE5QJzAdDvjrGJWswVuPBpQX26p2kf+Xa/dScwNSFwc9XxNmdzjxuT+tXKcuZLRO8DcM1vttzhyqd5A+bnoNNYFApH2MIkgJkt8nAqgjDrnNmXFRUVFbmZIGAmyC9TQO83UC3l8P6jScuekjY/hmD47B+BHRjBwdch2k1RNgXtrSWEQpmIBDXw8vVXF6lYbBK4DR4Tdjt2lcDfOgSzcGB2t0ETpY5NZsapDkJJNFshFwD/OsseQFz0fQb7h2ZWbNbLxoBWUPjWT4AXzzlg40jgGgoO2cpkUaOjPz0uOxh6RKVSVfm3tD0RqeOW/Lb5nJYKSW2QY7UZWbV03C1s3dqlxkWjv0aGzY41N1fA1BSc9p8HAvqWdOslQbxo17+Pz3WthbnUNrLGeccQVpsVnLPH2DjLwl3EZ+lDmB7JTKuBrotWhgVBf6lKMriExdxs+C+H4n355e4x/8mQBRmfCQMJtE3RuQ1YQZFpH5eIS8OME+hSfX371UNaZ5sm5CGHHvZ/DhTV+omkza6cc14EwDWDT15mbByzmb32NRt2c38rioVA8yo8wODfjepqGIc/RgoR37zU8ghY9BgLFP5MzcQ2SWtXNXxt4ojK9Pbj0C03uYPaF4LvPHg1b6ooFQ7H63I9Lz8nbTB0VHoDOdS7ZuEneknwzZpBqJjU+EikYYdjsGhWI3QhasrhLqmR9Dd5oV3gios3MvgcyFlyNunNqZpIPs9jlJQyZwdCmdZ7fdbhnhoI+MkZT21+/mqrw1 OxR2oF2p r0YhuuZltyBr6M3GjF24XfSeh+rQpadOVoycSmjLDQqG131L499cgKxexSbYbHW8lwNRWwxOzc1AL+HkJGtDEz5ukI3YLKCaQtGuAqsEdi9YJI5mUHXmdcllTkoFLRq8DZAdVbI0txo26Pmclll8ZTrEzH9308cZLKv21Ir2Hcnq3wj8gq8z/Gtb2yXktbOQTGVaxxYzcTQt4IQJlR1wW7t35sP+oJM18IkFrbJ6TfsL3hrXhHOD8iHMLza2ajP7oKMe4vGx2N6u0hmvdZOs+A5iPOdd/+zmfgl+knW1EILwWAWNF/Qo2/p9Y/qja6+U3D6WCax//U3p06aeOfQBv0qJ3MagvJjpN9cDkQL4YJ83KwltdUnwugpboDnRR9cOctcJCZvNNJXo/gXlWyL5nV8DB3nZ83dQtkrMBIkbi9KTa539Q3cRBhM/KRmOxlRBKXPusodR5jOohuAzkQQ85xdLVryQHXUIkFSI6s7RuqOKurYd5VFzK9xg7itqsx2Yd5/Mc9Ik7Gx3oBWns45yJ+N6taSUYxpdEbIjEqxn0hxYTzXWKLbPSXVjHwJz0PlIc6tQk X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In hugetlb_folio_init_tail_vmemmap, the reserved flag is cleared for the tail page just before it is zeroed out, which is redundant. Remove the __ClearPageReserved call. Signed-off-by: Frank van der Linden Reviewed-by: Oscar Salvador --- mm/hugetlb.c | 1 - 1 file changed, 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1d8ec21dc2c2..0836e3231774 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3202,7 +3202,6 @@ static void __init hugetlb_folio_init_tail_vmemmap(struct folio *folio, for (pfn = head_pfn + start_page_number; pfn < end_pfn; pfn++) { struct page *page = pfn_to_page(pfn); - __ClearPageReserved(folio_page(folio, pfn - head_pfn)); __init_single_page(page, pfn, zone, nid); prep_compound_tail((struct page *)folio, pfn - head_pfn); ret = page_ref_freeze(page, 1); From patchwork Thu Feb 6 18:50:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BB57C02199 for ; Thu, 6 Feb 2025 18:51:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BB8F280007; Thu, 6 Feb 2025 13:51:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F4F2280002; Thu, 6 Feb 2025 13:51:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 522D9280007; Thu, 6 Feb 2025 13:51:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 24F2A280002 for ; Thu, 6 Feb 2025 13:51:31 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id E0A011C9180 for ; Thu, 6 Feb 2025 18:51:30 +0000 (UTC) X-FDA: 83090413140.08.812AD7F Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf15.hostedemail.com (Postfix) with ESMTP id 1EF38A000C for ; Thu, 6 Feb 2025 18:51:28 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=f0fwbHsP; spf=pass (imf15.hostedemail.com: domain of 3rwSlZwQKCJAzFx508805y.w86527EH-664Fuw4.8B0@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3rwSlZwQKCJAzFx508805y.w86527EH-664Fuw4.8B0@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6PxACdTioptMBNslx7g+3VfoPnhQJsGOs8ZAh4yAkb8=; b=aXoOlUf9zx7GlF1O8eHU1eG0dFgJf6v4YW6dwo48xY4XZWRR2Ry1CejZKvHkFp1jd/BkQr Km4yZSi7kJYKNWiYvXrykBTqnbxaKLECZH+WRtpg+XuUQPlKGDFIUUIJ+hyZGT3NHsMX88 2RsguSk7/UP98cu8WXRY/4FB1cd9n0c= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=f0fwbHsP; spf=pass (imf15.hostedemail.com: domain of 3rwSlZwQKCJAzFx508805y.w86527EH-664Fuw4.8B0@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3rwSlZwQKCJAzFx508805y.w86527EH-664Fuw4.8B0@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867889; a=rsa-sha256; cv=none; b=vaDPjy1Zm1t7GerwIchNAptdL2lPX61RhJ7YCBYMyzO0Yh7NgButo41bsxbBexu8bi6Nyo 6Cyn4tYJlyCNQFNvE9S6alVAid+XflpdJZZG4gmxtjWZy7r5yRI3Jpg2tWRRiGGprNp1fq rTjwgDLfY+s5WVdhrAx/1Htnpn2/mqs= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f2a9f056a8so2425489a91.2 for ; Thu, 06 Feb 2025 10:51:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867888; x=1739472688; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6PxACdTioptMBNslx7g+3VfoPnhQJsGOs8ZAh4yAkb8=; b=f0fwbHsPln4e2C9NxRd969TiP6S5B4z/D/dqp/sv9Vqs3HZc+llH3Y+uImA/dVDe2y Mh2c/tfWijNLmYckiQXwK9/dV08GHMABpENmi8hGCAHzsbf/KG+l0Np9zkMdUUFnWLKy SSYeE/y/uA5wTDplBy/ioKmKEWtWJl84SvVXhU0wcM9r/02pMVXzeCfmj5VtjKJ5A/ki IyKYqigauYxnCnj7kgVY3ZsWF/RcmTPfrJJMyrOkuSjg1P/ydgBMtNBW+hv6IsvJjgy8 ONYG5vFU4fre7HjA/W0RUdOuxADUR2S/pH9iY85BJnL7hbE1wG5F2ivZ4HbbgeOeC9OJ OaJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867888; x=1739472688; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6PxACdTioptMBNslx7g+3VfoPnhQJsGOs8ZAh4yAkb8=; b=kOtmGdsttjY0/6Xww6qCLk/ZIITi3H2nhjPAybSYDUDG+GIG6T9ewAfMemuPx6O0Rs zTZrYqePDmmtseYu7EG+2Wbjpl3xocXtvPrG1cTKpf7/O25bAMeOfOFQn6pJ5+A7A6xh cGMgV7ckGWY2GbVmzt/E37cEHOeJLxNbwGqKUJlzBfRpPXHIy7A048oMKYKOlKtLV1Ae zwNXx6oEdw1T2puDeXmtVwTyQMYpMM6EQp60D29jtb0Tg1kWXjxxVgTYi0Ff2jQrHm/4 LW0GMfE2MDbVXJxKIOYU8LOp7xTZia0yjLcJyP4ZPRBCVe6YWFKD6o7p1XR230Jb+WS9 dZLw== X-Forwarded-Encrypted: i=1; AJvYcCXtdr4zNOSu9wQHemD0kSLTDL+DCUzz92VmHNDMYQbfL5YYGWFzYDayh+vPIGdP+AkDJE977eaxKw==@kvack.org X-Gm-Message-State: AOJu0YwnX9atYkJmwED0Zb/gUQqKL6OvyxJ6rFuAomSLLvIoAUkO7KEh fNEC0RbR8ggx6IhcA4XfmJF07ueu4ZQGCFf8pl2uq5lK8QS0U+ctvoXjkIZg/5ALXv0iFg== X-Google-Smtp-Source: AGHT+IHyKxzmRP4eOhwx4mD9ZcZrcL1FTQwwbx+QtNyhga2EwWZzDYY9rIMSfx1FziNod6CZsidSIYk3 X-Received: from pjbpq3.prod.google.com ([2002:a17:90b:3d83:b0:2ef:701e:21c1]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:4a86:b0:2f9:c144:9d13 with SMTP id 98e67ed59e1d1-2fa242e5dc0mr174642a91.24.1738867887870; Thu, 06 Feb 2025 10:51:27 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:47 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-8-fvdl@google.com> Subject: [PATCH v3 07/28] mm/hugetlb: use online nodes for bootmem allocation From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 1EF38A000C X-Stat-Signature: qts4kxgxn3qsy7kub4rkacdatjk4dqhi X-HE-Tag: 1738867888-347915 X-HE-Meta: U2FsdGVkX19YHYzKiL+mtOmZ39N2OhVXtFDBQZjAI8YnR7OewyHBsg4L8eZEjtLw0foeelUjTRB15ZEA0Hy0mFwJvsnxS1kB8BjqUIVQtVVH+PuDipn0Q9b2tk/Hm9mBkhH0XSqXu6e5CPUEPis8mqf2FPHcm2hogz7ZgL5buITaD3BaCkbguNUKkk9WeKqLAyswYB+DcrtAGKpR82+TT0U4/OnLt34wHHVS0YJ50MCvB9r++OtFstFq8P9UbSZlL0LR+TeYm1/EDk7XXo86aINSAvd8c1ggGnOpzKOGSw7QjUziBHlQEAMYnhA+u+TGFGdOH/3c/HzwgiBB0U7xG5Fb7VJY12gjExT//h6Rs5/kCFomDSrvymSFN0mfIAvXnsdHo6dxjBPShXdimfAAtzcBpguiRGOJTpxIBmRJlI479yXlWxXWbSXScMK56kxS6WuAcO+hTFVys6h6mS9qEWYS4O0Qlq5XBXxE6fpxqFqGsLI/U5WZEJsGFE3RX7+r0/e98qlc3IdBrESB2c5dLAvWxu7WfdEfUsbL6dQKAm8IU4khqOeSXaP9OxfcZcIymZLcAOqr45HDPVg3jx3ATXFsBR6C0isFWeGfai0YYvmVnOCNxdocHKPD7KTRrlueqJ2QQZzH0XWqKU4KlqJw7ehZxOAV7qznKm/hFBXavdw8UgzVYmV+hACsZQjWwsPiZT/eb/L7yfjYlUFA1tFVHv39z+ZaGD2IvJdSorXONE4EVdKpBGXHGf3nWMs7yN5+EmCsqY23EmAAAuXLZ2UNsA84EOLDrbZKWEohdozyObhbRje9tmy4eDVVMVYabZciDvDijVbTUv564P4bj2Wvr6CYStShjeLRHBmifwvvChPwJVkgHxCiGWqZGsi9fSRS2XmeeJyhSwPcfMXjwG7XMbUMOqjeNmEhkYZHYNUMGnZggpjMxqQqQKSfOrHEW1VUc+bivvswVqSm4KksKdU 0cTiqAoM bkOH5LjJpxMbpQGIYb4UskKPS6iyA15m0XT3sB/rB2kEEcSDk+cyAEupjHwD4pCS2teThunPVtz1dJGsKrb6B+eEdgjWWENfGf0Mu925IogdiXM4Dcd1pl0A4T7AxRLHuj6lXRE+YHWSZ4rmDyGYKPHfB4qmhRMl+Ge2IwoNMaVLurEgiNuVxt5+P9ZE4ZJsodBOlMycCgTY6rfgn4dkAJuZDO67+NNotzHwCY3OMyQ6dic5rV8X7CA+Yn8BEVXVrlvvaoSJSoimfqQZ9ToVPKy7Eqy9PZD2RaB9rCkNYkZmvmpyswXG9HRlhDVHWG67CGmNy8X3Z1WRsCFKSHtUpA/1bk4UH4YT/upPnfk5AinRjcTiDEXOnC4jOzNmHcNHnVXn7qzQOGd54mteCKFu21tJ10/k0Y2sz+4vKYHfilJvg8lgGYnH0OYReFN2QmdKzYKbf4OGfgTKCpzo4VymyNbzeGC5BxEEKkm7rQLRSHpN75c1euzKPgomnuwnLvZB/WayPdwqYeeJnfTYHgYqXb0+AAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000014, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Later commits will move hugetlb bootmem allocation to earlier in init, when N_MEMORY has not yet been set on nodes. Use online nodes instead. At most, this wastes just a few cycles once during boot (and most likely none). Signed-off-by: Frank van der Linden --- mm/hugetlb.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0836e3231774..b4de3bbd010d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3152,7 +3152,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) goto found; } /* allocate from next node when distributing huge pages */ - for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_MEMORY]) { + for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_ONLINE]) { m = memblock_alloc_try_nid_raw( huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); @@ -4550,8 +4550,8 @@ void __init hugetlb_add_hstate(unsigned int order) for (i = 0; i < MAX_NUMNODES; ++i) INIT_LIST_HEAD(&h->hugepage_freelists[i]); INIT_LIST_HEAD(&h->hugepage_activelist); - h->next_nid_to_alloc = first_memory_node; - h->next_nid_to_free = first_memory_node; + h->next_nid_to_alloc = first_online_node; + h->next_nid_to_free = first_online_node; snprintf(h->name, HSTATE_NAME_LEN, "hugepages-%lukB", huge_page_size(h)/SZ_1K); From patchwork Thu Feb 6 18:50:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963556 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15DCFC02199 for ; Thu, 6 Feb 2025 18:51:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52690280008; Thu, 6 Feb 2025 13:51:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 48805280002; Thu, 6 Feb 2025 13:51:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C79C280008; Thu, 6 Feb 2025 13:51:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E5E45280002 for ; Thu, 6 Feb 2025 13:51:32 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 997F11404F4 for ; Thu, 6 Feb 2025 18:51:32 +0000 (UTC) X-FDA: 83090413224.14.784FF38 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf30.hostedemail.com (Postfix) with ESMTP id A51A080018 for ; Thu, 6 Feb 2025 18:51:30 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=K5lJ49fX; spf=pass (imf30.hostedemail.com: domain of 3sQSlZwQKCJI1Hz72AA270.yA8749GJ-886Hwy6.AD2@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3sQSlZwQKCJI1Hz72AA270.yA8749GJ-886Hwy6.AD2@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867890; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nTJNRf6LTiMkMlXPEOIwPpP06vsojDmaDWHwE95+Iic=; b=ynCiLfYBIJVdf4hLJxpbCKmGBGSX/yrTXFz65QbSpLyWZaWAVU3RYu4+gJojmrv2pz70IL tB/CQcc7+zckcJL1bzRi6dTAYihhNc3XeUEsE9GAwRSsndA/a+tPn7iEvpks8ZR/f7Ai5c I/AREQYdiWqaeisZr82Td+9smXmdQpE= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=K5lJ49fX; spf=pass (imf30.hostedemail.com: domain of 3sQSlZwQKCJI1Hz72AA270.yA8749GJ-886Hwy6.AD2@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3sQSlZwQKCJI1Hz72AA270.yA8749GJ-886Hwy6.AD2@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867890; a=rsa-sha256; cv=none; b=nyamIkuXD17Ul5tkNWVSdcorlbqFjgJBTJPdvlhutjCGCW8JJBjIpi0jSToAJWCZ5R4nkB XxW1J9X+Mgu6L8qsuGO651P7hHt2cyyCUeEUxW2XQZi/kSpgYo++Wj3YObWh/zShrEizpr 6ELsHsmOgNm4+pBr2ZDZ6LuX5nRSXi4= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f780a3d6e5so2670682a91.0 for ; Thu, 06 Feb 2025 10:51:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867889; x=1739472689; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nTJNRf6LTiMkMlXPEOIwPpP06vsojDmaDWHwE95+Iic=; b=K5lJ49fXXPD7CIghLKJy548aiptnxl8cj6h8bdbXHOsKf0iE13ztyPHkFjZpt1F7N1 +Gxvxqg86pEU7apZJdwqYOfCuL5zquB17pK4o3U0iGvSgHBGOOSh2QlhbUAK4l7gPXG5 DiTtfDqqVxFskvzQuyTriVWUBOxOzhcl1x+qfpPmfMZBGxd4EVu53AR2ceMPto3e4kai f3SFsZwXBGgRSM60sH5rfVU7HarkcxWsZ65hkuVahmjuHxp1NM5hICJ7CnXTKlz3jorP uUHkrichwDT4ts5IWscfg99uorPtay8FMbR7pT0HlbmiO1u1xD3c7Mby5y/zHU5m1nEh 3sHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867889; x=1739472689; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nTJNRf6LTiMkMlXPEOIwPpP06vsojDmaDWHwE95+Iic=; b=LTCsHdf0M8CQk/CEe9/6TYEBoht51/2TlafAJ8dKbSmRALTdH8LXN7OH81d3Ve2VTa hgKf2zTtJxYUiCrrBVK0YoitXjFVwX4x8utQeE8/Mkk6qjOnk2f3YpMlivue0ATNj2Zl 9MsrQKEHUrnj16HG4qapZBDRcltcjNc6ilyyaH0puR/oHWPXQGqhLQLF65EKg50RuA/L VE1ceNsmoxCc71oeA14WjzSMcMunWJvrOPNHJttX+r9FpHT3W9ls77tRlg4nS8K1veNb 606rsUg/U6kOKQ3bV0U8gN14/7j2lHhDZJlbzNImXl2sjk1THQl0wCm5f3AV7sMrKMu5 MLTw== X-Forwarded-Encrypted: i=1; AJvYcCVn3fEIZRBSf5VrYD7hPqPo/meZV/B8qTdhREwAXntjFKGwqKJV87liSv4wz/MkE9kGC542QbUN2Q==@kvack.org X-Gm-Message-State: AOJu0YyKkQ176W/yZQu3QBgoCfhu4q/IYvSI3RM8Q0qciJjYpsbx/y+L rnmRrYSJnIiap3xj8VO5Lx0I3HFG+qrwO9+cwHRN1Txjm9be5SRI5m5DMOT6AVK15yJpGw== X-Google-Smtp-Source: AGHT+IH3ZOjV9MQ9vVNRnnO7zW4cnOheRu/dHPje2DoECTojb4J2x2zX0jaHZ8pQaiYA+/kja/RHuYZU X-Received: from pjbpl16.prod.google.com ([2002:a17:90b:2690:b0:2e9:ee22:8881]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:2ccf:b0:2ee:b666:d14a with SMTP id 98e67ed59e1d1-2fa24177fa6mr225873a91.17.1738867889615; Thu, 06 Feb 2025 10:51:29 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:48 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-9-fvdl@google.com> Subject: [PATCH v3 08/28] mm/hugetlb: convert cmdline parameters from setup to early From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A51A080018 X-Stat-Signature: hkdiraykwghbmhydi8k1i998sgwh4pug X-Rspam-User: X-HE-Tag: 1738867890-87887 X-HE-Meta: U2FsdGVkX1/JI/xxseFZl3/+ar9rk3WMZzBm9R5udN+YlPSzp7e01kP02eXG9cCfPQUjXC9/smEpdvXnkMHx2Jps7fmQxdCbt596AlMDD3jDzv8inefpFGKrIZThl/Xvrq/FMo/oNTISldaqW0WarYI5gDCHkPMQFWx6AAv+75autkpdcAuK3BfRcq3ls0e5B2XcvswZQjnLjAIwBIkSJfRkcLo9b5T3KPfay9p1HGR5rScSjLFRez2gff7dTE4l/O+SC8o85Oct8T75EQQbl9l3qfENKA64+oTuKT/OGMH7tXZy7JQD9X0HIqA7E/+LYE7gNYZT/Y/w9jwVWzKHgkDTe02cxgjYZVuidySbixOSxMj53aQJyxZwsQ0itwDnjJ6JcoKFbxZVhwIQ6ZWAPjojIWfuIrqji0PPHiMxH8tBIhFJH7tIiDT5a9NQIbKYapkcBk8IsKR8H6mDtBUV5PYTE8wed1BSinInpQ2dV8FR4Ks1y9qqCqyVtz2XexSvLsnbsKkOenM4te/VLqFZUx4rs4a3mthdeIv6DUus+q4D+i3ycnOKcKgZT/BXKEK5+/D9+7RuoCnvck/vIcRbwKB4muDzxb08orvlIPZDV7eEIPbfvpivfBKQQ59rC5+hCFPUyDOtergG/Cx1lfK/3lz5uVU0McCyndFym+KfHOEUgSoftAeFsm+kr0capMvmxQNrcNUu+gsoxCktGci5OwA8PosN0lcygumpejp061d4/MQNWJYG6SAeaxcWMx2m1C7tgXzjV9oIYT7DXwNYMY3aY1+2809u+EmgGs2EwLEH1Cr/5GXXjHilukvg3cStOBzE2NUd/2FoVVtWsMZEI2m41NGJQSswGsu6DFy+UyXScefrYBlKbUGqv1e8kx37HuH5raru3xK4GdxwK7sRHatmnRxSXh0e/M0yVQ4lB7wDu9Xrsmm1d55JZkfi8X4DRvKuxN34CK5dW009+L9 ZfzfXprM Uam1EF8WAXNyNYXaHU6OHv/UEZsckcIQRB9oylU8/oY0v7yHcjpG44Igri7PoFLzMrkGk3PsBxjRM2acUoafGpH3e/1dCDtpP2+KVHkiEl6dFXpfc4/LQ7ODCf+AQD78Ld/PSbFCRlJOEosFhT50gA6LvGp5REmNn7JQ9RxfkfH3UfxLv4eWq34LlIsQRdhOlPr9vrfFLSImmg+Q6gt0VCPrRRpnc8T/MazGgMPAYEBqgtrA3XsPTgfut41B3iVtXcObIvlCJ244HVerK4ev2KKzT8e0P7ju17VMWex8aHQR8f/kvg+oi6rYzqqcnMTVhRqhIEKHogi0HSOKdBGmuUGznvefUnHVpL4AL4byMtzh6Rs7rYokjpjUJBjzKCXGDdFMQOX8gXakAUz4K0TgoFpmvm8O11D3IBxRKM8TQVItxzTNMyCAd+o8oduifDVNPzcWJSZN5joZQxNAlX34WX9iDyYo7pp36/easNV+IFEWhXW8YZJeiAiBuDGLwS0GRerHV06pyQXawxqq68Xu+IYmrlg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Convert the cmdline parameters (hugepagesz, hugepages, default_hugepagesz and hugetlb_free_vmemmap) to early parameters. Since parse_early_param might run before MMU setups on some platforms (powerpc), validation of huge page sizes as specified in command line parameters would fail. So instead, for the hstate-related values, just record the them and parse them on demand, from hugetlb_bootmem_alloc. The allocation of hugetlb bootmem pages is now done in hugetlb_bootmem_alloc, which is called explicitly at the start of mm_core_init(). core_initcall would be too late, as that happens with memblock already torn down. This change will allow earlier allocation and initialization of bootmem hugetlb pages later on. No functional change intended. Signed-off-by: Frank van der Linden --- include/linux/hugetlb.h | 6 ++ mm/hugetlb.c | 133 +++++++++++++++++++++++++++++++--------- mm/hugetlb_vmemmap.c | 6 +- mm/mm_init.c | 3 + 4 files changed, 119 insertions(+), 29 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index ec8c0ccc8f95..9cd7c9dacb88 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -174,6 +174,8 @@ struct address_space *hugetlb_folio_mapping_lock_write(struct folio *folio); extern int sysctl_hugetlb_shm_group; extern struct list_head huge_boot_pages[MAX_NUMNODES]; +void hugetlb_bootmem_alloc(void); + /* arch callbacks */ #ifndef CONFIG_HIGHPTE @@ -1250,6 +1252,10 @@ static inline bool hugetlbfs_pagecache_present( { return false; } + +static inline void hugetlb_bootmem_alloc(void) +{ +} #endif /* CONFIG_HUGETLB_PAGE */ static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index b4de3bbd010d..5a4b322e2bb2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -40,6 +40,7 @@ #include #include #include +#include #include #include @@ -62,6 +63,24 @@ static unsigned long hugetlb_cma_size __initdata; __initdata struct list_head huge_boot_pages[MAX_NUMNODES]; +/* + * Due to ordering constraints across the init code for various + * architectures, hugetlb hstate cmdline parameters can't simply + * be early_param. early_param might call the setup function + * before valid hugetlb page sizes are determined, leading to + * incorrect rejection of valid hugepagesz= options. + * + * So, record the parameters early and consume them whenever the + * init code is ready for them, by calling hugetlb_parse_params(). + */ + +/* one (hugepagesz=,hugepages=) pair per hstate, one default_hugepagesz */ +#define HUGE_MAX_CMDLINE_ARGS (2 * HUGE_MAX_HSTATE + 1) +struct hugetlb_cmdline { + char *val; + int (*setup)(char *val); +}; + /* for command line parsing */ static struct hstate * __initdata parsed_hstate; static unsigned long __initdata default_hstate_max_huge_pages; @@ -69,6 +88,20 @@ static bool __initdata parsed_valid_hugepagesz = true; static bool __initdata parsed_default_hugepagesz; static unsigned int default_hugepages_in_node[MAX_NUMNODES] __initdata; +static char hstate_cmdline_buf[COMMAND_LINE_SIZE] __initdata; +static int hstate_cmdline_index __initdata; +static struct hugetlb_cmdline hugetlb_params[HUGE_MAX_CMDLINE_ARGS] __initdata; +static int hugetlb_param_index __initdata; +static __init int hugetlb_add_param(char *s, int (*setup)(char *val)); +static __init void hugetlb_parse_params(void); + +#define hugetlb_early_param(str, func) \ +static __init int func##args(char *s) \ +{ \ + return hugetlb_add_param(s, func); \ +} \ +early_param(str, func##args) + /* * Protects updates to hugepage_freelists, hugepage_activelist, nr_huge_pages, * free_huge_pages, and surplus_huge_pages. @@ -3488,6 +3521,8 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) for (i = 0; i < MAX_NUMNODES; i++) INIT_LIST_HEAD(&huge_boot_pages[i]); + h->next_nid_to_alloc = first_online_node; + h->next_nid_to_free = first_online_node; initialized = true; } @@ -4550,8 +4585,6 @@ void __init hugetlb_add_hstate(unsigned int order) for (i = 0; i < MAX_NUMNODES; ++i) INIT_LIST_HEAD(&h->hugepage_freelists[i]); INIT_LIST_HEAD(&h->hugepage_activelist); - h->next_nid_to_alloc = first_online_node; - h->next_nid_to_free = first_online_node; snprintf(h->name, HSTATE_NAME_LEN, "hugepages-%lukB", huge_page_size(h)/SZ_1K); @@ -4576,6 +4609,42 @@ static void __init hugepages_clear_pages_in_node(void) } } +static __init int hugetlb_add_param(char *s, int (*setup)(char *)) +{ + size_t len; + char *p; + + if (hugetlb_param_index >= HUGE_MAX_CMDLINE_ARGS) + return -EINVAL; + + len = strlen(s) + 1; + if (len + hstate_cmdline_index > sizeof(hstate_cmdline_buf)) + return -EINVAL; + + p = &hstate_cmdline_buf[hstate_cmdline_index]; + memcpy(p, s, len); + hstate_cmdline_index += len; + + hugetlb_params[hugetlb_param_index].val = p; + hugetlb_params[hugetlb_param_index].setup = setup; + + hugetlb_param_index++; + + return 0; +} + +static __init void hugetlb_parse_params(void) +{ + int i; + struct hugetlb_cmdline *hcp; + + for (i = 0; i < hugetlb_param_index; i++) { + hcp = &hugetlb_params[i]; + + hcp->setup(hcp->val); + } +} + /* * hugepages command line processing * hugepages normally follows a valid hugepagsz or default_hugepagsz @@ -4595,7 +4664,7 @@ static int __init hugepages_setup(char *s) if (!parsed_valid_hugepagesz) { pr_warn("HugeTLB: hugepages=%s does not follow a valid hugepagesz, ignoring\n", s); parsed_valid_hugepagesz = true; - return 1; + return -EINVAL; } /* @@ -4649,24 +4718,16 @@ static int __init hugepages_setup(char *s) } } - /* - * Global state is always initialized later in hugetlb_init. - * But we need to allocate gigantic hstates here early to still - * use the bootmem allocator. - */ - if (hugetlb_max_hstate && hstate_is_gigantic(parsed_hstate)) - hugetlb_hstate_alloc_pages(parsed_hstate); - last_mhp = mhp; - return 1; + return 0; invalid: pr_warn("HugeTLB: Invalid hugepages parameter %s\n", p); hugepages_clear_pages_in_node(); - return 1; + return -EINVAL; } -__setup("hugepages=", hugepages_setup); +hugetlb_early_param("hugepages", hugepages_setup); /* * hugepagesz command line processing @@ -4685,7 +4746,7 @@ static int __init hugepagesz_setup(char *s) if (!arch_hugetlb_valid_size(size)) { pr_err("HugeTLB: unsupported hugepagesz=%s\n", s); - return 1; + return -EINVAL; } h = size_to_hstate(size); @@ -4700,7 +4761,7 @@ static int __init hugepagesz_setup(char *s) if (!parsed_default_hugepagesz || h != &default_hstate || default_hstate.max_huge_pages) { pr_warn("HugeTLB: hugepagesz=%s specified twice, ignoring\n", s); - return 1; + return -EINVAL; } /* @@ -4710,14 +4771,14 @@ static int __init hugepagesz_setup(char *s) */ parsed_hstate = h; parsed_valid_hugepagesz = true; - return 1; + return 0; } hugetlb_add_hstate(ilog2(size) - PAGE_SHIFT); parsed_valid_hugepagesz = true; - return 1; + return 0; } -__setup("hugepagesz=", hugepagesz_setup); +hugetlb_early_param("hugepagesz", hugepagesz_setup); /* * default_hugepagesz command line input @@ -4731,14 +4792,14 @@ static int __init default_hugepagesz_setup(char *s) parsed_valid_hugepagesz = false; if (parsed_default_hugepagesz) { pr_err("HugeTLB: default_hugepagesz previously specified, ignoring %s\n", s); - return 1; + return -EINVAL; } size = (unsigned long)memparse(s, NULL); if (!arch_hugetlb_valid_size(size)) { pr_err("HugeTLB: unsupported default_hugepagesz=%s\n", s); - return 1; + return -EINVAL; } hugetlb_add_hstate(ilog2(size) - PAGE_SHIFT); @@ -4755,17 +4816,33 @@ static int __init default_hugepagesz_setup(char *s) */ if (default_hstate_max_huge_pages) { default_hstate.max_huge_pages = default_hstate_max_huge_pages; - for_each_online_node(i) - default_hstate.max_huge_pages_node[i] = - default_hugepages_in_node[i]; - if (hstate_is_gigantic(&default_hstate)) - hugetlb_hstate_alloc_pages(&default_hstate); + /* + * Since this is an early parameter, we can't check + * NUMA node state yet, so loop through MAX_NUMNODES. + */ + for (i = 0; i < MAX_NUMNODES; i++) { + if (default_hugepages_in_node[i] != 0) + default_hstate.max_huge_pages_node[i] = + default_hugepages_in_node[i]; + } default_hstate_max_huge_pages = 0; } - return 1; + return 0; +} +hugetlb_early_param("default_hugepagesz", default_hugepagesz_setup); + +void __init hugetlb_bootmem_alloc(void) +{ + struct hstate *h; + + hugetlb_parse_params(); + + for_each_hstate(h) { + if (hstate_is_gigantic(h)) + hugetlb_hstate_alloc_pages(h); + } } -__setup("default_hugepagesz=", default_hugepagesz_setup); static unsigned int allowed_mems_nr(struct hstate *h) { diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 7735972add01..5b484758f813 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -444,7 +444,11 @@ DEFINE_STATIC_KEY_FALSE(hugetlb_optimize_vmemmap_key); EXPORT_SYMBOL(hugetlb_optimize_vmemmap_key); static bool vmemmap_optimize_enabled = IS_ENABLED(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP_DEFAULT_ON); -core_param(hugetlb_free_vmemmap, vmemmap_optimize_enabled, bool, 0); +static int __init hugetlb_vmemmap_optimize_param(char *buf) +{ + return kstrtobool(buf, &vmemmap_optimize_enabled); +} +early_param("hugetlb_free_vmemmap", hugetlb_vmemmap_optimize_param); static int __hugetlb_vmemmap_restore_folio(const struct hstate *h, struct folio *folio, unsigned long flags) diff --git a/mm/mm_init.c b/mm/mm_init.c index 2630cc30147e..d2dee53e95dd 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -2641,6 +2642,8 @@ static void __init mem_init_print_info(void) */ void __init mm_core_init(void) { + hugetlb_bootmem_alloc(); + /* Initializations relying on SMP setup */ BUILD_BUG_ON(MAX_ZONELISTS > 2); build_all_zonelists(NULL); From patchwork Thu Feb 6 18:50:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F3C4DC02194 for ; Thu, 6 Feb 2025 18:51:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DFDCF280009; Thu, 6 Feb 2025 13:51:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D85FB280002; Thu, 6 Feb 2025 13:51:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BAF34280009; Thu, 6 Feb 2025 13:51:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 92D38280002 for ; Thu, 6 Feb 2025 13:51:34 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4A74D8130C for ; Thu, 6 Feb 2025 18:51:34 +0000 (UTC) X-FDA: 83090413308.17.4DD307A Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf17.hostedemail.com (Postfix) with ESMTP id 5628C40012 for ; Thu, 6 Feb 2025 18:51:32 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WfXlphAv; spf=pass (imf17.hostedemail.com: domain of 3swSlZwQKCJQ3J194CC492.0CA96BIL-AA8Jy08.CF4@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3swSlZwQKCJQ3J194CC492.0CA96BIL-AA8Jy08.CF4@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867892; a=rsa-sha256; cv=none; b=k+C+gMnciAJ1ExsLQJpZZdTFrtBOF42V4+BAjrD9l1nOYU6zWG76714EeulXnHq0cM/FWO 4XFAtXoqA+/dcihK7MVBeaXwRyxcwrhF77SJM8y715sZ0nkdXaYNk6K4hdDMvWfqMGeYRX EchPYeWtv4G3iKMtlh7ffNKNVyL2hLw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WfXlphAv; spf=pass (imf17.hostedemail.com: domain of 3swSlZwQKCJQ3J194CC492.0CA96BIL-AA8Jy08.CF4@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3swSlZwQKCJQ3J194CC492.0CA96BIL-AA8Jy08.CF4@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867892; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Avpy7BuEP02ffYq1Owfnvcdb9scYtoUfsZ01E274s1M=; b=C6t7VJtnMMmCS/mbCZS797O+ix3J5hXoKnL25FRqe2x8fT3PQq340QeObPiAqx5wibZDlc cmucdYv4mZXAxxrL8xN7eYXRZgS4/8AtLMFHYYklwlt7jzZAAEE3ssUce37E1seMfG4t9v per7zfEzXobEAK9aTVdf8X1wVcFBt8E= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f9e3a00fcdso2787448a91.0 for ; Thu, 06 Feb 2025 10:51:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867891; x=1739472691; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Avpy7BuEP02ffYq1Owfnvcdb9scYtoUfsZ01E274s1M=; b=WfXlphAvczjiKEGntnA8/oFIDek19XgcioYxZjLN0vG0mzut8twDGnhDCtCBhvmNdz cU/PfMnYWdH0sRehuCjmSYnpitnpVY0zO5iX6zZfb75qgsgjDCFRHBgTTw1VZ80lP7Uf qHhZbETXaUNJAihjh3QU1JIZBxGcxEqDgmwBno/m9oE/GWfq2J0/t0pAV6gha90aBIt0 d9nMTVH0BNiiDSf+oJcaepgOn6swqVhptqCZYSCCUOsABhrteNWGqST7mvaPXSq8OXFa LO5O7/9Nvs07GozDbiKAtJFrGrhNszE83GbmbvAjXhmjFydv+iKUss0qi464XfPKjmim l81g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867891; x=1739472691; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Avpy7BuEP02ffYq1Owfnvcdb9scYtoUfsZ01E274s1M=; b=QZLC7Q1u72559gBmca6hqCtGoLbenYwC0acirBNHhoSqwaJWnqvXZv5Fg4372IVddn ZJEgHBLHd58B8l988Kkj7HfJzWkvxE37+PTU/yAO7shZiAukYjn6RsVYqNaR4WaaVmED VYqUj62XZ9bfDLzVrvVVo/rK7eV36rpOiLthf/bTJmJIaCEW7AhRJO5dwPLsnmj0q2l+ 0XENTL5nHGKx8uu/VSBVuZetx6eh4l37xLLy3seZTmfVc0IeKnlF97rzqWN9gAKqO6Xv VQAQ688c2SXHnpGT30ukWBai0Q7ZuVOrZFYMqDDnUHGs/0n9ZU7BTpflxKkMaAjl0RLq 40Pw== X-Forwarded-Encrypted: i=1; AJvYcCWiGnIb3ECWu2iXqJVA6gvsWHVU7fNHPOJ6d23ANYdO1C5vVxD8szGr7clC8Ano92fNd0TlRQxf9g==@kvack.org X-Gm-Message-State: AOJu0YwDROuioI8Z+UBWEwcs5FlS7OFr7qt3FrFxfqjBCnlzxmeSj4JZ F7V5J0ssYwj4jCAoAWKBf4bKniZ0ngMlElityXDj7cE5eEH0xaLCdPcmig8fKK5GPlxKEQ== X-Google-Smtp-Source: AGHT+IFX3NvACBJ1xhrHGkex/mQRvyXzqKTeTXis5hJniSsUBBE3cj7dQzcaYViSr3/nKGrkfs3yIvZW X-Received: from pjbpq3.prod.google.com ([2002:a17:90b:3d83:b0:2ef:701e:21c1]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:510e:b0:2ee:8430:b831 with SMTP id 98e67ed59e1d1-2fa23f55c3bmr201823a91.2.1738867891288; Thu, 06 Feb 2025 10:51:31 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:49 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-10-fvdl@google.com> Subject: [PATCH v3 09/28] x86/mm: make register_page_bootmem_memmap handle PTE mappings From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden , Dave Hansen , Andy Lutomirski , Peter Zijlstra X-Rspamd-Queue-Id: 5628C40012 X-Stat-Signature: gceufqshbmkn8om8783xnf4szbjcjzse X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1738867892-739375 X-HE-Meta: U2FsdGVkX18MFzqDAsekDH3JIFpBwg8mCO3jS5f4uVpuwE78cs10hRiY5bcfGAjK6UQ1Ui1szzKh2eueEkd8VjY8fKyj2F2o90goXpPA2lOsMW50dzIW62Yllyr6HizfgkNFGkVntbMuF0qyMNukqbq/404ti8291G2tfTenxgxQz9bwqrYGeSaJtduOmPKj++xdzEG0E6eE43DUJjgEq0NWVEe/NaAsY8KQpBX4qoiygUle1tz+oZIY3+NHNHgB/OC4wRbN2tJn1De63tARWQSV4Ui2rETQViiZW43cLiVz4CCCd1bZY8oURHazIGWlQmVEdVsX/F0XQKVYd477MKgpolx7pm2ClzJGDSC4ETNZxsoCPFswLhGaNb6D1gcH/+xVenr3ffl1BdwH+gn0LquYDpbISEi1Rd6W4xOH/FuqYGWo++H0ueBpC5ZDbTZQfiHBRNS/6BC87zDQMc9YxHDpo8n0G5YtQBRbZHYkQnDOCjJ7GHsEd7cUsdtzb95I16N1HbotZbQiwR9ra0uRwuypG0grfYNRYLmQvqoaCc3KkjG1JytGQqVtXFW4lZpRPDWkU7DkUaSJyPrfB/YzCUeariMNCACXp7po2J0rxccGHdaqNAhLBdIBj8L58ca/DHpEXHXoT1L2/mQEJlS4XRiPT9bYlxAQgIv84+BuiUQsAmWMvyXdl+yUQkduHVoI3gSgOlDTS3YpKuLrURGmIo4u1Xsu/TDXm3Msj9sH1s/q17rIBVxytvRqgHqmdbkb7jKbl0S+3zMjj2rGqKY51fewTKkDXAM62eD4i7Um/LijuYy/UtFly+Nbv+UsRRp4TRoxwNwLmrV1wb/MqcIVNtXdxw8cqPUxIPUBWIuWI0funWjPQo3XxUoOfIJiKz199En5FZftQpp/DZDBA/otn+XBZXvzMhYtIbqxAqJ1GzQXUIosXJMCwEBJFFxe2hb+nR5W9/DM5/+vjCGHPmq MNb6Mrm7 C0osvkCgVTI1r8whngZ+MzMx0RsOUtjKgwv0ODlZpm9WbbvRGRybQXwPeVcH2dgqpDTNgeajSt8DgcW5166Dw+yxHxz4AZ91ELS7KqOPNSwgpdhd3UIkzVIpwHye5xN6fHC6eENewnvkL0VASRLi0z3YYp6Lo04cHdCnF3HZfyhHiYZcxKMcdzPD7HmTe/Z8/2npvcuv65SXYFW3NAu/GUqN2iTLgJ/MpcFPRLrKkoED6HHjzHesJCPojg5/Q/nObALtNAdDf1CaamgXEe70eJqDwKCs+pSvoPPICRtBupTNGylF+ugOaZnElyYwbE6/nHFCOuGupwVgT59Zg7rMYlqg1zrx3nh2G0D2dh+22fhsLEwR4CvyzvrBIH9K4DBO9EfcEsjB/aYEGxi9Xaq53MKofcT9BAkxzZhH45cRYwLXwgXSZul5g9oOlsmyuDYp1JXKhs9A2OFNAsv+IweeRtU0aGz5kX4PHnhfQl8iTgnNiuVxch/V5KQjKsGZ5xJflKL+KvqatXxO//YQP7SwJv4P1DwgPJGWeUoq7RkYpwjYUTURGagfCx72Yc+GyiS8+eW8zKeweJtOIZRsuZEl9ox93H7oesmcfWILkfDQKk/irtIjHLUl+k1CZmGqmIk9vHlJ6vAiiwLt2Z1bnShsGva0PP13Lr+AWlJG2k81YfPaYFGFqGsWK0QzrAA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: register_page_bootmem_memmap expects that vmemmap pages handed to it are PMD-mapped, and that the number of pages to call get_page_bootmem on is PMD-aligned. This is currently a correct assumption, but will no longer be true once pre-HVO of hugetlb pages is implemented. Make it handle PTE-mapped vmemmap pages and a nr_pages argument that is not necessarily PAGES_PER_SECTION. Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Signed-off-by: Frank van der Linden --- arch/x86/mm/init_64.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index 01ea7c6df303..e7572af639a4 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1599,11 +1599,12 @@ void register_page_bootmem_memmap(unsigned long section_nr, } get_page_bootmem(section_nr, pud_page(*pud), MIX_SECTION_INFO); - if (!boot_cpu_has(X86_FEATURE_PSE)) { + pmd = pmd_offset(pud, addr); + if (pmd_none(*pmd)) + continue; + + if (!boot_cpu_has(X86_FEATURE_PSE) || !pmd_leaf(*pmd)) { next = (addr + PAGE_SIZE) & PAGE_MASK; - pmd = pmd_offset(pud, addr); - if (pmd_none(*pmd)) - continue; get_page_bootmem(section_nr, pmd_page(*pmd), MIX_SECTION_INFO); @@ -1614,12 +1615,7 @@ void register_page_bootmem_memmap(unsigned long section_nr, SECTION_INFO); } else { next = pmd_addr_end(addr, end); - - pmd = pmd_offset(pud, addr); - if (pmd_none(*pmd)) - continue; - - nr_pmd_pages = 1 << get_order(PMD_SIZE); + nr_pmd_pages = (next - addr) >> PAGE_SHIFT; page = pmd_page(*pmd); while (nr_pmd_pages--) get_page_bootmem(section_nr, page++, From patchwork Thu Feb 6 18:50:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963558 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD020C0219C for ; Thu, 6 Feb 2025 18:51:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BC8F28000A; Thu, 6 Feb 2025 13:51:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 743C9280002; Thu, 6 Feb 2025 13:51:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 56DB828000A; Thu, 6 Feb 2025 13:51:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 33FF4280002 for ; Thu, 6 Feb 2025 13:51:36 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E61D01613B8 for ; Thu, 6 Feb 2025 18:51:35 +0000 (UTC) X-FDA: 83090413350.24.56064C1 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf14.hostedemail.com (Postfix) with ESMTP id E593A10001B for ; Thu, 6 Feb 2025 18:51:33 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XzyJETcl; spf=pass (imf14.hostedemail.com: domain of 3tASlZwQKCJU4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3tASlZwQKCJU4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867894; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZM2K7Fz2wY4XMJIGW8UIEY9GhsBMo7TuugczH48eoOc=; b=mcPMtgHLWIFN2i4avpqBJx//znD+qIRem9d8SopOaQr+dOhhUBxu/Q01RFAWKnWCvL2X7d tE3vsLymSu47SoVPb9vqf06M4Yd0VZOodRj/NaBjF/lNVka8SjBL+3HBWEs0Ek8Ma+rv0K LfnBzjzsvdzn0qbcrtThIZdFXkm0t+Y= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=XzyJETcl; spf=pass (imf14.hostedemail.com: domain of 3tASlZwQKCJU4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3tASlZwQKCJU4K2A5DD5A3.1DBA7CJM-BB9Kz19.DG5@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867894; a=rsa-sha256; cv=none; b=uTGmAZ4fbv9dFQMZUbdH0FKKSFZeopxLsTT4EMkvqnvXIONN21BL8Qjx9IZ/4vwLX/8zT2 ZynuaLQJ2qA8rS+KmfZeEl9W08MKEIS7g6a9gCAoibjFpI96KnAbMr6cQJbD8Sa0572jFz QinS6Xfa74z1DC1ONI9ZkHLPcU5GLUA= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f9eb31238aso3867185a91.1 for ; Thu, 06 Feb 2025 10:51:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867893; x=1739472693; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZM2K7Fz2wY4XMJIGW8UIEY9GhsBMo7TuugczH48eoOc=; b=XzyJETclK0UFvflHcow6TLJMQQQDQ7LuucbNU6WDzuhMD2MXZ0015eMumDFV3T75QR i/f07zvnfoBLmEuQo/LGIoKgJR7wDRaZFmYJLsTx8BGOwLaEHCZz+CeIdjQYAH7sIBOT LBVhuy6pD0A1UdfBVDKBnqBi5TaptjybYtVxrBz49mmepnPxG5yX46miSxk2DTJgDU4J Rat6h2m7+7iF22sN/voemxpd6boyhKLVTEU02HhZtf+kHT7ygAr8sLuawUPr5Q+GWZY9 B4oe2A8vnVsKrbDlp4X+WXhNygxYIAYaa2LVihUbUU1WlRIfzn+02xDqE+rWPwEG/iyl ZfSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867893; x=1739472693; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZM2K7Fz2wY4XMJIGW8UIEY9GhsBMo7TuugczH48eoOc=; b=oAqg/210SNibOTFQ5UirQUAqkuE9KlC3s0qnFo3GHIsS1qvvKsniEPj2G0P/BLkNPq tthn+97WfOx0YXK0SCSl3tVS9XzWl+703UM1FLJJk3q9pRg2CBuhqXymoAmkdeDL70dU Q6fy4L6d/bzPRy6qiLkKk2rZnGJg5Mk5FEkRyyFviY89Z0YVhn5ycERputPOk7PtKmT9 6cR64XxKkUte9N4FjVfeldBO31Dh1sSvI8D9hiQ8oyo6T6oVRGjlTDV4/yxd5KB2wQF2 5S0hfSzNGuKH+i/d1HIF6+c0YcG1WudKamlSEcbnORTApyJyyUBnOmE8SI9h81wyUG/j zXGg== X-Forwarded-Encrypted: i=1; AJvYcCUzgv3p8FlYDQ+HW4lGflg5HCUONT3b/CMbIHypuWsdHmCK9ePfnA5P2tZ/lzCuYbSgVjqhU4+Ojw==@kvack.org X-Gm-Message-State: AOJu0Yz9LPtb3fh5YXLGxVcVQNlL/jYNmwfYHDlza4pWhVGai4b2fOwM IORH46ba/cWXJRhNjL+S8/GMXK3YiWvBfOC3SR+WxOaTfDjTiUKlOknqkp7Xnf3IHT8aWw== X-Google-Smtp-Source: AGHT+IFVHLzvQXhhOOlHiOL7BAHshrKvK37YtrOlJQtEUP1k+IFRDA6RXf7GAghzr6oe2lWRYNCvHguW X-Received: from pfbha22.prod.google.com ([2002:a05:6a00:8516:b0:728:e76c:253f]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:84c:b0:72d:80da:aff with SMTP id d2e1a72fcca58-7305d463bb9mr573392b3a.9.1738867892913; Thu, 06 Feb 2025 10:51:32 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:50 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-11-fvdl@google.com> Subject: [PATCH v3 10/28] mm/bootmem_info: export register_page_bootmem_memmap From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E593A10001B X-Stat-Signature: g7z6cthzdm6ss7wyfe1kusnzr6joicy3 X-HE-Tag: 1738867893-671618 X-HE-Meta: U2FsdGVkX18kok6q7G7/Ua91469Fxbm2Gsr/wAuIfime8rXB/v8Urx25jo+x8Z0PqITkoiszBES/TnL5WsMbcwR6SrfGV8tuIGEGl0qLtu3QVou1ll5CXnZzWn7PmC4Ck99Q6ZO2p5PmZkgfVsr5u71uwUeA5xuKSsioPnuC0wvXlOqtSUW1twCdmRBJxDgb7yrcjSFSFmaCuzwPe2dRRWQamS4cPs3eYYSSEZ6miRtcZM8hzTcyuncpMHU2sBZzPSMpLWAUsojJpr3Agtolct8UKHlwHZpelLc+54Uwv0KiwxmHhy3ftLaSYhzyaorMWdLDtegs3fhTreYerbmXERtYnpuh0a6wwUWTRxNwWE11DDYJyk7Px0bzHqxmIu2OkeW7D1cbz0uLz4ms0AQYpeqWichk8h+B2qzbKAe0I1GHgCTjIMHiGbMGVKfz9vLuhx2CvdsI0XNd7lLa1Q4QlDu8E8PDseH7USHeyFa3R6YWAwwKo1hiI/K5U4wuCDT2MN95t0Qd+lN6/p3u/y2HuHLMinYguYBEeN1SJi7Ka7XHCv+bIDBCG9zd1iud7j1JAwdtf7SHMPDvVOtHrxvwnggvcy+Si1ZUIV9rR3dtIaZk0wHJ3xAFS2Nul9X+393IBLdkq8c7aFyIfUucOudI+JW/M32DzwGpFtm7jh2LPlL1cJVuqwKcfA6XLRb3FOKmyClEMvUvXTBC4jW+Tywi0lgWHt82F0pZg77hc8XJEDdcTjsLCv3w/f7W7/T59Gu1v/Bo5UZcgZBKHZsUBtYbIlPn3vPl+mx8jbL34NVijPezU+wuwYlOKeWYiU1etEOoCpSLJo0pfoQu+6orDpuWjaczf1ZS7BzHusmeOiFGreHEIBVzw012vI2KBKsqejhWjpT41x35MGaPNQSLf8QDNCVxkJV+ptt4UWM34gktX4Q3tntho8G+5U0UhuiyLkHYfybf58WtTmepYm/25po wvjQufRc j5Rj2orfNoNlxRm6dqpKBmaYO5G/YhmVl1fAnBxvLrHFCKXv6T8YwdHWDfEZKuWUPjbuc5WZSEJjQaOssQ2juWTPAIHcb6bI/kXc36t8/gHHXfwjyd8vh6kf+v2JRyddX2hnt2v3RQk+MwpxR1BaWnqoM/Ke+8EYCLGnLzgjQwM1z78rTvb2g/4KFK7EBod2/lEKKFjWvksqwLfGQNzYPZoDGZvBHMQKT/Dj7xHTTyiKO6aa+YDtGqfC8Hbz7yhXWnd0n+cyXNv/Zifz0tLuJzufrOKF5DeWb+O7WhLqvEdDW+HGj/kJTe9jeT/v5V3H7iGJMkbh2kbKmKVIT+U8jCc4hNF55TP3JAo/Zhkqc/1wt8mxBT7sBGaQSSLCt8pYfDkHikuvpUMWvPZVDisybGDgzJQtjz7y2SEfYxit47Vod3AyK3IZeAjPK+WIjRemnWUaeTGtRm+vxcoItqKPjaao9dxYoYoTnk6KtJmIWHU4uKBzL1WBjmyPxzC0WiCqE2IIi2vVYEb1BuOwLNGPRegCwhkbqMX3s9knpQdl+CTXFraTS6vnrKT51+NoZxASrzWMh X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If other mm code wants to use this function for early memmap inialization (on the platforms that have it), it should be made available properly, not just unconditionally in mm.h Make this function available for such cases. Signed-off-by: Frank van der Linden --- arch/powerpc/mm/init_64.c | 4 ++++ include/linux/bootmem_info.h | 7 +++++++ include/linux/mm.h | 3 --- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/mm/init_64.c b/arch/powerpc/mm/init_64.c index d96bbc001e73..b6f3ae03ca9e 100644 --- a/arch/powerpc/mm/init_64.c +++ b/arch/powerpc/mm/init_64.c @@ -41,6 +41,7 @@ #include #include #include +#include #include #include @@ -386,10 +387,13 @@ void __ref vmemmap_free(unsigned long start, unsigned long end, } #endif + +#ifdef CONFIG_HAVE_BOOTMEM_INFO_NODE void register_page_bootmem_memmap(unsigned long section_nr, struct page *start_page, unsigned long size) { } +#endif /* CONFIG_HAVE_BOOTMEM_INFO_NODE */ #endif /* CONFIG_SPARSEMEM_VMEMMAP */ diff --git a/include/linux/bootmem_info.h b/include/linux/bootmem_info.h index d8a8d245824a..4c506e76a808 100644 --- a/include/linux/bootmem_info.h +++ b/include/linux/bootmem_info.h @@ -18,6 +18,8 @@ enum bootmem_type { #ifdef CONFIG_HAVE_BOOTMEM_INFO_NODE void __init register_page_bootmem_info_node(struct pglist_data *pgdat); +void register_page_bootmem_memmap(unsigned long section_nr, struct page *map, + unsigned long nr_pages); void get_page_bootmem(unsigned long info, struct page *page, enum bootmem_type type); @@ -58,6 +60,11 @@ static inline void register_page_bootmem_info_node(struct pglist_data *pgdat) { } +static inline void register_page_bootmem_memmap(unsigned long section_nr, + struct page *map, unsigned long nr_pages) +{ +} + static inline void put_page_bootmem(struct page *page) { } diff --git a/include/linux/mm.h b/include/linux/mm.h index 7b1068ddcbb7..6dfc41b461af 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3918,9 +3918,6 @@ static inline bool vmemmap_can_optimize(struct vmem_altmap *altmap, } #endif -void register_page_bootmem_memmap(unsigned long section_nr, struct page *map, - unsigned long nr_pages); - enum mf_flags { MF_COUNT_INCREASED = 1 << 0, MF_ACTION_REQUIRED = 1 << 1, From patchwork Thu Feb 6 18:50:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB31BC02194 for ; Thu, 6 Feb 2025 18:51:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17CB428000B; Thu, 6 Feb 2025 13:51:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 10219280002; Thu, 6 Feb 2025 13:51:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DFA8728000B; Thu, 6 Feb 2025 13:51:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B387D280002 for ; Thu, 6 Feb 2025 13:51:37 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6C11C1A140A for ; Thu, 6 Feb 2025 18:51:37 +0000 (UTC) X-FDA: 83090413434.18.A6D2982 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf15.hostedemail.com (Postfix) with ESMTP id 9660FA0013 for ; Thu, 6 Feb 2025 18:51:35 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fGjuBVwt; spf=pass (imf15.hostedemail.com: domain of 3tgSlZwQKCJc6M4C7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3tgSlZwQKCJc6M4C7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867895; a=rsa-sha256; cv=none; b=E/Kg6PuKjxoXGS65zeWmpyCN7qgRx6ZrYbpD/89DqyA34MzTNqFN5FQOK/Mp31YHpKr8lD se077wAwy4ox0xhpcY1hd/35DzsboKuB7FqwnNizlR1BJ258x2TMofK0V6JPiIXl7AWgvo jM/k6D9r4UjInFY/BVRL5HlyKLEqmR4= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=fGjuBVwt; spf=pass (imf15.hostedemail.com: domain of 3tgSlZwQKCJc6M4C7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3tgSlZwQKCJc6M4C7FF7C5.3FDC9ELO-DDBM13B.FI7@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867895; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=laTVDvhr5jKOxEf5d1BFEkjR6DjNz8pI1Rzto2J5iEQ=; b=DpbogtL+UyejAzDE5lkkS8VR6ulGriWm84EhccDmSGngbmulVoV4UgFHPlrAF+Ka5LRoI2 kpsyRyq3FhpkYAzZrEYKUXm+H+gY2kXbHNJ4APafBE63V4grnWvvRPaf0dWrKPlFMjDpvA muMtnzaBGFyN5hnORWPHUzeAgipNvZc= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f9ee46968cso2422769a91.3 for ; Thu, 06 Feb 2025 10:51:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867894; x=1739472694; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=laTVDvhr5jKOxEf5d1BFEkjR6DjNz8pI1Rzto2J5iEQ=; b=fGjuBVwt3xXDr4yvULkJq5Rm/gEShGndGMi86NOsOWu5rscvY+C843x6WNgJVX9KZD RE5UTyshoJ7aOdVpllA21a5zXV+BpS2HyQxMdrdKFHj++lMM2YArMKgmQhe+gZv3JAkW 8wmxL4Jl/uI2ed8R+YoPuHRX2ZsPj/+zI1YeYgARBXta3TlLfstYbzDTsIlILht60PwA VaznkqXoL18k3yHzNaOftleZzifn4+g42ONTRZwYJYpPmzeYUjD0ZFYH0abRhaYH7Giu w03p8J3P+AC7zx6NcM4LTuQm3cGkZ76hf/t0bPJCQtZQhrBgt4oG6vgBu5zEagy2e7Vz g9Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867894; x=1739472694; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=laTVDvhr5jKOxEf5d1BFEkjR6DjNz8pI1Rzto2J5iEQ=; b=TjVrHfJCBsk0YVU++nGnwRRfq2TIHVM9iueOCw7fLUArG9XuXqwBlmTfhCGVzwAWiA 374f88TpNypDhyp6GAJJ7kfOmiYBc2UuAIzHaSMTss95wdVqWF0n7HBPVccFDeEzQBPc N8XPzqEOQA1bn2FdYoGW04yWocYizEzwwmRICvTS6vXW8cBpr5aVsbsObgJ4FXTbDZBR joAq7a9WJe1KojR1bw963qDiuvUD9fYfBYKTJUEkgOvyoVG3YEAzgHrjXMZFYajA00rO Ucbw6587hCZCdDEZtKocyU791Iw4xGMYPP6G/KZv7PqCZlDuw825Ren3FkuW5PhQUDSY Jvyw== X-Forwarded-Encrypted: i=1; AJvYcCWXLu3UERN73I+7cyF3dKUcyQhyf+QQd+lYzwZHrJ8YnybWnfi851wX9qeQsw1nMz2zQyAWIhej/w==@kvack.org X-Gm-Message-State: AOJu0Yy+nMv07ZWNHu+7tV4Q5P4y1a0kunNZXpW3uSQKU21y3Fb7qjtv jP1XZqVakYzYQWb/cc7ybGyC6pMQB72vHXGu5dYclcYF7AJCpGYYb8C3wQ90er0A+hloxQ== X-Google-Smtp-Source: AGHT+IFWXu22XDiYyc0nTMJwiopxxnqaKl40GlxcarOlGhTscpEPLMqJOzkRyOrxdabf2F7/pMajnLN0 X-Received: from pjur16.prod.google.com ([2002:a17:90a:d410:b0:2f2:e5c9:de99]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:38d2:b0:2ee:48bf:7dc3 with SMTP id 98e67ed59e1d1-2fa241646afmr193581a91.15.1738867894502; Thu, 06 Feb 2025 10:51:34 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:51 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-12-fvdl@google.com> Subject: [PATCH v3 11/28] mm/sparse: allow for alternate vmemmap section init at boot From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Stat-Signature: 7p4f6h6i5zqb7t4nffq98ztd51xrao3y X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9660FA0013 X-Rspam-User: X-HE-Tag: 1738867895-533214 X-HE-Meta: U2FsdGVkX1+4K7CMrbUtzVcEoMNmS69znTwh1h6WDdAb+EgPWzGR5tfPqQQewqDmPV1Ixphu4u5TgdoOfE9KmlKo8SN5cnDFTiMw2JeJlP5k1jYFk8Kh+6agJuUVXb3lzeL3bNxfqXOa24znlz43wG14mL3QuXowBmSLSoz483sDsISu/247tPWKRf1yLmbppzjJDyv62NwCZuuXVvtolS1jfCrhh64ujvuNpYEmsiHRhNVlLo8dz5O4Z5d60ChvDp9WEB17xwn+WZ1TCBZmc2T4YLvbUJ2UL4CyT39KoDZL72dmy23nvxAY4l9mFRuHceRhAsLWWuB1ufBibugHwSLlWWVOcVl8+dJbOn0p5vmB6z2O1LsUdBx1BdbFSF1m/Ck9HApRRmU/Jdq/OIgsXsoAFj3P6A+SrD7M2BqtTPG4eL7nGcnqDLzwGdpYui924peTQfxKJvq8Q6VaMu3UdL9PbuLwvJdSiouLt1zTpwgKqVhtvIUPvkA4HRPTk0Gvz2/+sUu575BT08ce7TqyUrlQuxc92mIBEM5sm5WljP1kLqynX5eKLl2TjwTMYTtQSJf21F5QZnKV4/aET3POMyTIXLp6YqtzR5U/YfJqz6j1tzoHfXrfmoSroyd4asQhdsmF3STK+RhCxEHwIgbfE1e+Owc2sFfSk38qLQUX+XqQESFeZwPKkmevtnHFOIhmaDYeYSWtQr0ZOBIHPm7vX2/AojNGUWYXf9T91knSrBm4/IFEdurIk7IqZ9hptR2r/HSxE+gvArUMmK0uXQpFyA7d6wiQVyLdfVBTkXQX/c3JVp8nQ8Bgijw7ezY/UVRCT8uZgV/DWGMqGsj1AyM+FIyD8GJyIGXWk2EPoXcCPeQ8I0/snc3KfghpUnwZ+P+vDwxwAx6WY+rxoSmd8WpuGFLcwIoPRFw2J/3t8VuFIfJOCaYtxYhuRrX7jIIs4Opoo6Ex+dbeY7fsJ8bHiuh udZWIZ+Z dvABdkUE7ho+4wbtbxfk0mgUj7hIOwgKQZuA40c48T8bzd/TdxcBZ6wv/FdwYmPQXurf8/fsqSCX47LatwJI4R6kHGHkn9tk50Mff6rS6WItIApbesTe4Vl0+GwOsHqwb0qbQn8ryv165KfqIhvLjJLY0pRhq3GcRbifhgX7jxmUzOK7cruIzDM4EzdpteMbBTgApiJpSA/epiQRJJntvSCvtKsh0ER5Rqc4csN0MmDBhVOCoN+NB46f3dxNen/uuRRAvbZixRbxlDd/OMEL8vhlN4CfbD1mkaJGpkfpecofQlJHVS6oMoDwsMAN2vdmnFQfzLtGJbXF/UzvQSZyUxCF7JQcqeauNd05lwOjTdaEtomwDwbaFropU8LG/ouFNnI3Y0tjRgAS2xv4x3IBLlOVbnhxDCHsIVpRXdLQtg8/BKb1NQlxklncqPicm5KUvHLrTWnkUSHF4Msshi4dTbdR5aEDyCcJ3q7dw5+rt/yadEiBuXekLBayfsSxXJBElEXiNFObrBLeUfag= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add functions that are called just before the per-section memmap is initialized and just before the memmap page structures are initialized. They are called sparse_vmemmap_init_nid_early and sparse_vmemmap_init_nid_late, respectively. This allows for mm subsystems to add calls to initialize memmap and page structures in a specific way, if using SPARSEMEM_VMEMMAP. Specifically, hugetlb can pre-HVO bootmem allocated pages that way, so that no time and resources are wasted on allocating vmemmap pages, only to free them later (and possibly unnecessarily running the system out of memory in the process). Refactor some code and export a few convenience functions for external use. In sparse_init_nid, skip any sections that are already initialized, e.g. they have been initialized by sparse_vmemmap_init_nid_early already. The hugetlb code to use these functions will be added in a later commit. Export section_map_size, as any alternate memmap init code will want to use it. THe config option to enable this is SPARSEMEM_VMEMMAP_PREINIT, which is dependent on and architecture-specific option, ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT. This is done because a section flag is used, and the number of flags available is architecture-dependent (see mmzone.h). Architecures can decide if there is room for the flag and enable the option. Fortunately, as of right now, all sparse vmemmap using architectures do have room. Signed-off-by: Frank van der Linden --- include/linux/mm.h | 1 + include/linux/mmzone.h | 35 +++++++++++++++++ mm/Kconfig | 8 ++++ mm/bootmem_info.c | 4 +- mm/mm_init.c | 3 ++ mm/sparse-vmemmap.c | 23 +++++++++++ mm/sparse.c | 87 ++++++++++++++++++++++++++++++++---------- 7 files changed, 139 insertions(+), 22 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6dfc41b461af..df83653ed6e3 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3828,6 +3828,7 @@ static inline void print_vma_addr(char *prefix, unsigned long rip) #endif void *sparse_buffer_alloc(unsigned long size); +unsigned long section_map_size(void); struct page * __populate_section_memmap(unsigned long pfn, unsigned long nr_pages, int nid, struct vmem_altmap *altmap, struct dev_pagemap *pgmap); diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 9540b41894da..44ecb2f90db4 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -1933,6 +1933,9 @@ enum { SECTION_IS_EARLY_BIT, #ifdef CONFIG_ZONE_DEVICE SECTION_TAINT_ZONE_DEVICE_BIT, +#endif +#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT + SECTION_IS_VMEMMAP_PREINIT_BIT, #endif SECTION_MAP_LAST_BIT, }; @@ -1944,6 +1947,9 @@ enum { #ifdef CONFIG_ZONE_DEVICE #define SECTION_TAINT_ZONE_DEVICE BIT(SECTION_TAINT_ZONE_DEVICE_BIT) #endif +#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT +#define SECTION_IS_VMEMMAP_PREINIT BIT(SECTION_IS_VMEMMAP_PREINIT_BIT) +#endif #define SECTION_MAP_MASK (~(BIT(SECTION_MAP_LAST_BIT) - 1)) #define SECTION_NID_SHIFT SECTION_MAP_LAST_BIT @@ -1998,6 +2004,30 @@ static inline int online_device_section(struct mem_section *section) } #endif +#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT +static inline int preinited_vmemmap_section(struct mem_section *section) +{ + return (section && + (section->section_mem_map & SECTION_IS_VMEMMAP_PREINIT)); +} + +void sparse_vmemmap_init_nid_early(int nid); +void sparse_vmemmap_init_nid_late(int nid); + +#else +static inline int preinited_vmemmap_section(struct mem_section *section) +{ + return 0; +} +static inline void sparse_vmemmap_init_nid_early(int nid) +{ +} + +static inline void sparse_vmemmap_init_nid_late(int nid) +{ +} +#endif + static inline int online_section_nr(unsigned long nr) { return online_section(__nr_to_section(nr)); @@ -2035,6 +2065,9 @@ static inline int pfn_section_valid(struct mem_section *ms, unsigned long pfn) } #endif +void sparse_init_early_section(int nid, struct page *map, unsigned long pnum, + unsigned long flags); + #ifndef CONFIG_HAVE_ARCH_PFN_VALID /** * pfn_valid - check if there is a valid memory map entry for a PFN @@ -2116,6 +2149,8 @@ void sparse_init(void); #else #define sparse_init() do {} while (0) #define sparse_index_init(_sec, _nid) do {} while (0) +#define sparse_vmemmap_init_nid_early(_nid, _use) do {} while (0) +#define sparse_vmemmap_init_nid_late(_nid) do {} while (0) #define pfn_in_present_section pfn_valid #define subsection_map_init(_pfn, _nr_pages) do {} while (0) #endif /* CONFIG_SPARSEMEM */ diff --git a/mm/Kconfig b/mm/Kconfig index 1b501db06417..f984dd928ce7 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -489,6 +489,14 @@ config SPARSEMEM_VMEMMAP SPARSEMEM_VMEMMAP uses a virtually mapped memmap to optimise pfn_to_page and page_to_pfn operations. This is the most efficient option when sufficient kernel resources are available. + +config ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT + bool + +config SPARSEMEM_VMEMMAP_PREINIT + bool "Early init of sparse memory virtual memmap" + depends on SPARSEMEM_VMEMMAP && ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT + default y # # Select this config option from the architecture Kconfig, if it is preferred # to enable the feature of HugeTLB/dev_dax vmemmap optimization. diff --git a/mm/bootmem_info.c b/mm/bootmem_info.c index 95f288169a38..b0e2a9fa641f 100644 --- a/mm/bootmem_info.c +++ b/mm/bootmem_info.c @@ -88,7 +88,9 @@ static void __init register_page_bootmem_info_section(unsigned long start_pfn) memmap = sparse_decode_mem_map(ms->section_mem_map, section_nr); - register_page_bootmem_memmap(section_nr, memmap, PAGES_PER_SECTION); + if (!preinited_vmemmap_section(ms)) + register_page_bootmem_memmap(section_nr, memmap, + PAGES_PER_SECTION); usage = ms->usage; page = virt_to_page(usage); diff --git a/mm/mm_init.c b/mm/mm_init.c index d2dee53e95dd..9f1e41c3dde6 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1862,6 +1862,9 @@ void __init free_area_init(unsigned long *max_zone_pfn) } } + for_each_node_state(nid, N_MEMORY) + sparse_vmemmap_init_nid_late(nid); + calc_nr_kernel_pages(); memmap_init(); diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 3287ebadd167..8751c46c35e4 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -470,3 +470,26 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, return pfn_to_page(pfn); } + +#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT +/* + * This is called just before initializing sections for a NUMA node. + * Any special initialization that needs to be done before the + * generic initialization can be done from here. Sections that + * are initialized in hooks called from here will be skipped by + * the generic initialization. + */ +void __init sparse_vmemmap_init_nid_early(int nid) +{ +} + +/* + * This is called just before the initialization of page structures + * through memmap_init. Zones are now initialized, so any work that + * needs to be done that needs zone information can be done from + * here. + */ +void __init sparse_vmemmap_init_nid_late(int nid) +{ +} +#endif diff --git a/mm/sparse.c b/mm/sparse.c index 133b033d0cba..ee0234a77c7f 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -408,13 +408,13 @@ static void __init check_usemap_section_nr(int nid, #endif /* CONFIG_MEMORY_HOTREMOVE */ #ifdef CONFIG_SPARSEMEM_VMEMMAP -static unsigned long __init section_map_size(void) +unsigned long __init section_map_size(void) { return ALIGN(sizeof(struct page) * PAGES_PER_SECTION, PMD_SIZE); } #else -static unsigned long __init section_map_size(void) +unsigned long __init section_map_size(void) { return PAGE_ALIGN(sizeof(struct page) * PAGES_PER_SECTION); } @@ -495,6 +495,44 @@ void __weak __meminit vmemmap_populate_print_last(void) { } +static void *sparse_usagebuf __meminitdata; +static void *sparse_usagebuf_end __meminitdata; + +/* + * Helper function that is used for generic section initialization, and + * can also be used by any hooks added above. + */ +void __init sparse_init_early_section(int nid, struct page *map, + unsigned long pnum, unsigned long flags) +{ + BUG_ON(!sparse_usagebuf || sparse_usagebuf >= sparse_usagebuf_end); + check_usemap_section_nr(nid, sparse_usagebuf); + sparse_init_one_section(__nr_to_section(pnum), pnum, map, + sparse_usagebuf, SECTION_IS_EARLY | flags); + sparse_usagebuf = (void *)sparse_usagebuf + mem_section_usage_size(); +} + +static int __init sparse_usage_init(int nid, unsigned long map_count) +{ + unsigned long size; + + size = mem_section_usage_size() * map_count; + sparse_usagebuf = sparse_early_usemaps_alloc_pgdat_section( + NODE_DATA(nid), size); + if (!sparse_usagebuf) { + sparse_usagebuf_end = NULL; + return -ENOMEM; + } + + sparse_usagebuf_end = sparse_usagebuf + size; + return 0; +} + +static void __init sparse_usage_fini(void) +{ + sparse_usagebuf = sparse_usagebuf_end = NULL; +} + /* * Initialize sparse on a specific node. The node spans [pnum_begin, pnum_end) * And number of present sections in this node is map_count. @@ -503,47 +541,54 @@ static void __init sparse_init_nid(int nid, unsigned long pnum_begin, unsigned long pnum_end, unsigned long map_count) { - struct mem_section_usage *usage; unsigned long pnum; struct page *map; + struct mem_section *ms; - usage = sparse_early_usemaps_alloc_pgdat_section(NODE_DATA(nid), - mem_section_usage_size() * map_count); - if (!usage) { + if (sparse_usage_init(nid, map_count)) { pr_err("%s: node[%d] usemap allocation failed", __func__, nid); goto failed; } + sparse_buffer_init(map_count * section_map_size(), nid); + + sparse_vmemmap_init_nid_early(nid); + for_each_present_section_nr(pnum_begin, pnum) { unsigned long pfn = section_nr_to_pfn(pnum); if (pnum >= pnum_end) break; - map = __populate_section_memmap(pfn, PAGES_PER_SECTION, - nid, NULL, NULL); - if (!map) { - pr_err("%s: node[%d] memory map backing failed. Some memory will not be available.", - __func__, nid); - pnum_begin = pnum; - sparse_buffer_fini(); - goto failed; + ms = __nr_to_section(pnum); + if (!preinited_vmemmap_section(ms)) { + map = __populate_section_memmap(pfn, PAGES_PER_SECTION, + nid, NULL, NULL); + if (!map) { + pr_err("%s: node[%d] memory map backing failed. Some memory will not be available.", + __func__, nid); + pnum_begin = pnum; + sparse_usage_fini(); + sparse_buffer_fini(); + goto failed; + } + sparse_init_early_section(nid, map, pnum, 0); } - check_usemap_section_nr(nid, usage); - sparse_init_one_section(__nr_to_section(pnum), pnum, map, usage, - SECTION_IS_EARLY); - usage = (void *) usage + mem_section_usage_size(); } + sparse_usage_fini(); sparse_buffer_fini(); return; failed: - /* We failed to allocate, mark all the following pnums as not present */ + /* + * We failed to allocate, mark all the following pnums as not present, + * except the ones already initialized earlier. + */ for_each_present_section_nr(pnum_begin, pnum) { - struct mem_section *ms; - if (pnum >= pnum_end) break; ms = __nr_to_section(pnum); + if (!preinited_vmemmap_section(ms)) + ms->section_mem_map = 0; ms->section_mem_map = 0; } } From patchwork Thu Feb 6 18:50:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963560 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09692C02199 for ; Thu, 6 Feb 2025 18:51:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F1EF28000C; Thu, 6 Feb 2025 13:51:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A246280002; Thu, 6 Feb 2025 13:51:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A61428000C; Thu, 6 Feb 2025 13:51:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 44054280002 for ; Thu, 6 Feb 2025 13:51:39 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 066A71C9171 for ; Thu, 6 Feb 2025 18:51:39 +0000 (UTC) X-FDA: 83090413518.20.D57489F Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf30.hostedemail.com (Postfix) with ESMTP id 3E32580002 for ; Thu, 6 Feb 2025 18:51:37 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="UU07Xw/O"; spf=pass (imf30.hostedemail.com: domain of 3uASlZwQKCJk8O6E9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3uASlZwQKCJk8O6E9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867897; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3NeQMOnzxPkjyqTJozldr6OrEIjuFMWwE6WQJOgcCAA=; b=VyvxwualuUTGB3jvCunK0Mix3KhVGq6m0dCuzNDOssBUphRgxFULBs7gKGAHDBPiiwv5nz QlEwUNNV5CoEhAzVc0vW8eU5d5+69sQciJ2TEP9F9cJrle2e/s9riJRzVzmPqHdn/OPg0H yovWM91RS4Y33hF/ljv6hiS3c9hINY0= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="UU07Xw/O"; spf=pass (imf30.hostedemail.com: domain of 3uASlZwQKCJk8O6E9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3uASlZwQKCJk8O6E9HH9E7.5HFEBGNQ-FFDO35D.HK9@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867897; a=rsa-sha256; cv=none; b=oNv9zK9XBAB2pvNNyoXMOTyheoOfhwa5Rv1Df0/GT2TIoeQrWIQqLSNf5u+1b8eybnY0qE ecr3tTqfv7kRFaOcZp/Dj1d2Je2GJHuieHtUERxGLohLrVLgkh7bC3l5IOmNT3WSZOrz4z leP1bmTkevhhWrE9nxvtYGwyF2IBbnE= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2fa0204ad21so2418523a91.0 for ; Thu, 06 Feb 2025 10:51:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867896; x=1739472696; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3NeQMOnzxPkjyqTJozldr6OrEIjuFMWwE6WQJOgcCAA=; b=UU07Xw/OObHEnQNNUn1sJaYIhYA0Y4n+jjdp3syBY+g6XS45XgO4QOVIShIkmUf9ea 7WjNKM9Mo8pmfLYDlBY7B9zhP8httjGSq67d0TfGK/7nBRZa1Sbgb0lSkFFeUZPSCaih vu6VyNo4WQsVP422VVsfoZSfDx84WKeTY0IiwamQRFVhtqQocvgPpuJZ5yoFceXTuSKY z7zZ6IBESIht3QRZXWUi0VA+3Rvb63JEmmbSVEjCgPvpXn7Gr0ty4cANN7ODjAMYtmiW tSELM5iaW50DH6T9Cx8SGskkbDWMHMTam1XsSB2ep0mW84GT3+wNRE4K9Fd+FQVHBJhw 3bpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867896; x=1739472696; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3NeQMOnzxPkjyqTJozldr6OrEIjuFMWwE6WQJOgcCAA=; b=QKErz2O9OL2urRbV6z5an1y3g8cK4r/+xUKYyTmsU3T+gUcrqtoimwGN4DiytdcMP+ qmR5oxnQPd7h6P6hB1nmwVNZJmYulT5xiXDeFsE1pjCcNMEY27vOxOmm5GUW26/UhZIq Fu+ye0nElHDWp+5BoAVpGwgp68bORJyUXs2sAnSJDCcf6jlKTsvgefCBFZuh9UMv9ylj bJZQiQS9ktnRuY9DSjvlrhNMT15vXdCMX8pcyKO8EofopG6/15HrPx9aXOsk+9s+ZU4B 2lU6SIY854i+NED92lHlt85OHSxjw5Gc0NsxbIFCRvPaO/quy7n2wtxar4fut40DnU0M jO6g== X-Forwarded-Encrypted: i=1; AJvYcCViqPlNE5VBoQncvStPWeksl+8/YH3b7FWH9ChSb8R/hf6JQeOTcv6LuQKmu3NPzXt5RWqd/qRvqw==@kvack.org X-Gm-Message-State: AOJu0YwTYqiO6UnVpQgiWAxYcvWGaDzqfVYsg2asaOyig266tXVI9pUC kjwH3/yIOWr0L4jRCxGAzGwKLsSa5QdPN+liVNBIa9MjcNl/tl8bVbBefPDefg4/6sqOHQ== X-Google-Smtp-Source: AGHT+IH7ZN2LcrXXo7ZGhTfjqav29224bEfR3b4dopinjdsKyhAOEliIZ1YG+HBMVTQXtNzOsKEFpkdd X-Received: from pjuw5.prod.google.com ([2002:a17:90a:d605:b0:2ef:eb2f:ebe]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90a:f94e:b0:2f2:a664:df19 with SMTP id 98e67ed59e1d1-2fa23f436d6mr217716a91.7.1738867896108; Thu, 06 Feb 2025 10:51:36 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:52 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-13-fvdl@google.com> Subject: [PATCH v3 12/28] mm/hugetlb: set migratetype for bootmem folios From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Queue-Id: 3E32580002 X-Stat-Signature: 1ggdw735z9fb3rp6nz5o5qa4qq6i1qoc X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1738867897-141534 X-HE-Meta: U2FsdGVkX19cAz0A76xgOrEPFfzr0iKarSIpk+rL92a7e4bZUt/1i6WP7rXCH7Tb3wrKnxHet406hNNvu9kUxoVym49ep15hifQFp8hJZG58evyMro4O5EtRTdZgVQX04M1XLJzChFMRkWYICoEiBuhjBBteLpvpGmwNiTilqWRDa8fBDdA/PW+MWRvPxdfRmbju4jm+3KQWiXBdlJVczNgYFATpUcNMwu0Y8qKhaMvjKDtKISGTUR4LReezcBvDMgHMAtzMYkZxEJS5BgJy+UVK1sOyOY0znkASobcSGDTgKfmm2y4i/E1eIEarAV41fWmOw0lU42VLGptlzcSbcbR3aXA3gQJpcFmT4FN3wq3hW4hJqIJsPSfz+ERsN4NAM4BqiExtbB2E271Wqk+2WejFJn9lhaPjToh7WNTo5go5Mf3Jx4cOMhGub45nRw5/vNtTg1cWrNnOoA+Cs6ApQINEvszk7LCVj/0j0ejmJVFcpKxoOI3sSaA5rPg4m86TmOb1nr7Xn4VBTZ2NTzqU80Y4SBLEBxkuyN86CoEzqZX7oZl8tVzbPq9sAmDVGpaPx0ToI4UmLu7jg7SwTh5jjOP5SNp5WJWllQ5IrsAPAYDV0idwoxaFVu8uF5DGoJXs0sNONyTwLCX1eNukfPbCHE7iATwXqkSCq6VluVIl63FTdwc6JxrHg8CoqU/bvW/EjAN4p8Y8Sb6hVAptu2IUxgK/2Oxw4T9aqbdlK6wIQnxj0YLzoJYh0+FUb2yOR+eKGlLiEp72j49R6bzJ8dnNL9I7TorJfZzBbLkyH174t7xgmVP4MtoZikW2N2Jjc0NkD3GDBKnCplIPotWgzVqPa5APKLVXWXRFVfJykHr63bSp6Jc2OM6pci8oEsGO7p8MR+97UhSL1osl4RdpA6HG++FGgvmcVWOXT1oqyMzWg8i8tc6OFGRzTe5VBFRi/Yf1lFHzMiG6FxJZy2ogr12 4x5s76Qu un/WzNy4vmmP1fN5HfuBRONfq2oac9hzeyKm4mGZjGncclLqU9SwQMO+MBOyn0Jr2f/5WZy0Au8WMjC9YafvxrsUEH8mLKdkF3SSyyH/4H/dqbczfqmR0OBeDKbXX/SSkmMlFEdbx57iYrODRQHwDmZVue4EZ3UyFjae+/gHpHgVp+NEdWjCwmCZWYhVHP8Tz1LkP1yoRu4y4fKe6MldDvSVLNJWz5VNf38pKUKQnSHoqVKBdQcxu7RrhtcujozYDNunSqd3KimdgW7c5NRy3fdGsiSGzgHEaRIwwMzA3pVtMJxm9ezEH+SUmoaLTlF63ymPdLzzytgPviXT2r+NkL49xPi093Rdd7ywSITx0XzUE6asr4qO6TEsWMGDUVInt4GMelfZjd/ZMLrevdd/Rr/W2jEnHXZyFXdNbVJADTUD3khQP0oBv3o38Au7au3lHzpGWm1GBt6bElyL44sxCf7dkaeIxAKt08gWm7+XwPkNB0IIQVY0i4pS3KiQH/9U6QGnD1izXkZtpb4g/SrHIpS0B3Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The pageblocks that back memblock allocated hugetlb folios might not have the migrate type set, in the CONFIG_DEFERRED_STRUCT_PAGE_INIT case. memblock allocated hugetlb folios might be given to the buddy allocator eventually (if nr_hugepages is lowered), so make sure that the migrate type for the pageblocks contained in them is set when initializing them. Set it to the default that memmap init also uses (MIGRATE_MOVABLE). Signed-off-by: Frank van der Linden --- mm/hugetlb.c | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 5a4b322e2bb2..de8adfb487f4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3258,6 +3258,26 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +/* + * memblock-allocated pageblocks might not have the migrate type set + * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) + * here. + * + * Note that this will not write the page struct, it is ok (and necessary) + * to do this on vmemmap optimized folios. + */ +static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, + struct hstate *h) +{ + unsigned long nr_pages = pages_per_huge_page(h), i; + + WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio))); + + for (i = 0; i < nr_pages; i += pageblock_nr_pages) + set_pageblock_migratetype(folio_page(folio, i), + MIGRATE_MOVABLE); +} + static void __init prep_and_add_bootmem_folios(struct hstate *h, struct list_head *folio_list) { @@ -3279,6 +3299,7 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, HUGETLB_VMEMMAP_RESERVE_PAGES, pages_per_huge_page(h)); } + hugetlb_bootmem_init_migratetype(folio, h); /* Subdivide locks to achieve better parallel performance */ spin_lock_irqsave(&hugetlb_lock, flags); __prep_account_new_huge_page(h, folio_nid(folio)); From patchwork Thu Feb 6 18:50:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4FA55C02194 for ; Thu, 6 Feb 2025 18:58:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DD8436B0096; Thu, 6 Feb 2025 13:58:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D87EF6B0099; Thu, 6 Feb 2025 13:58:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C4FE06B009A; Thu, 6 Feb 2025 13:58:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A655A6B0096 for ; Thu, 6 Feb 2025 13:58:35 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 460E2465CF for ; Thu, 6 Feb 2025 18:58:35 +0000 (UTC) X-FDA: 83090430990.26.D976D35 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf30.hostedemail.com (Postfix) with ESMTP id 70FAA80019 for ; Thu, 6 Feb 2025 18:58:33 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q+qSoiUh; spf=pass (imf30.hostedemail.com: domain of 3uQSlZwQKCJo9P7FAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3uQSlZwQKCJo9P7FAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738868313; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=N1nizw8LTgFJ6Hunu1baoXowLm72wInJnvRTI+vB3dI=; b=hK05sxm0Ytz+fS0cpCmGFzodFEAU4vpLbYaGiJtXpGwB07/8lY+vP5oUGM3HPlrNF9vvPB Ncn8RIk4FRIR96yJBxeEhRAUqBNHMqmR1k823tgzujcvFqeyHBjaqJcaeL5ii9H3K3mFyM usi3qv5TmlXJi+a7GH+UJqTwFPiSewI= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Q+qSoiUh; spf=pass (imf30.hostedemail.com: domain of 3uQSlZwQKCJo9P7FAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3uQSlZwQKCJo9P7FAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738868313; a=rsa-sha256; cv=none; b=SKm2h7PoUd4Uxs6G8T81t5f30xzlzaQu+jU863ldpU7disfEdImb6QarLM1rer/8dik1CQ 656iH00i1hA5MPETmeQmoll+QNuO2cVKuGOgzNYaFd6dCGqd2d1U0dR8EspNwNSX6NMYlc xdv4LH6nG0LVYiXvpWuZgOGBfcpUk6M= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-2163dc0f689so40538025ad.1 for ; Thu, 06 Feb 2025 10:58:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738868312; x=1739473112; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=N1nizw8LTgFJ6Hunu1baoXowLm72wInJnvRTI+vB3dI=; b=Q+qSoiUhM676fV/iBCZvQTb907b3TPTZ74vBAOsMVS3I+EuaMAzmpptIXLsb3wLeI1 u/FbnPO3/iw5/pJgsSqTuNnfN6388R1bqs8+zfG8ThEJYSHdwgx4fg5Ya1kuiPpBEuVb YK6nhcY+3PSMdzCYxdb6tcJeC+i0h9Hui2qmuGv1j+bT+g2Hin0r/WLmJC7DozbN/YQM wR4SWh+WWgBYKXwqDWZI5KpaV3ECK2NgYKUOptM0xlElAXgrpZ6VmaGJCWEmaW0m7wfs dChamDkBsUUIdFfF4OaFIfl6olY0Ig3rcIMkIpuKPF0iMPx50Zqmf2E9ebHODAL1VuM+ Sf+w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738868312; x=1739473112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=N1nizw8LTgFJ6Hunu1baoXowLm72wInJnvRTI+vB3dI=; b=sXWfYPJTbHrbHZP3kVRpihIWD8SO3NcscsMZaIHmi8fsjhuMANpwk3E7CQoIpxHoDM qsdalWAvAaJ5Ws3eA/SYT6sN9Ye2n4p5v69If4WAKYV5G74EaFStsAb3QZVenx/pFz6d 9XS7fiRMahkqpnL0N11+EHWC0uZ6jQT3VXrU22a82tjCQijatkDkc12AvGLoKv6CVXZu nqxK6mWFB8Xxfe0kG5TD8Kqe8kiEdgwPQvcizhsrB9q3834+/foQw2Je6f0HRXrg5ROe cCadPnGWgpaBR/dw5EyzLZaKF+FBsnMKKf+J0x0UC90fPwjavWdth363ZB5ba8YI8ILf xqMQ== X-Forwarded-Encrypted: i=1; AJvYcCWeu4ppXOqN9Qn6scWaV13Y/35u5NzV5H1Q1FMmlIcI7bB6rEuzGQdbH6YGPfhJt028kHN94CHX+Q==@kvack.org X-Gm-Message-State: AOJu0YySU3zjqq+NuXL0TEsHqHzrkWxtwguqrfdkNc4f41iB9t5FxKdR Al/12PHC3RutWAqMzPN7fQ8yQmcVttMjYP40t9vITqTFfwNsW8OxRBDGQ8x2slgS1GDeeA== X-Google-Smtp-Source: AGHT+IHCvMErKFRJySbjcM7qJ570UaYkF9tu8lbCYFmm8LplntCVFlwxB6ni+y+lUDS7htgcQtwDW2+L X-Received: from pfwy2.prod.google.com ([2002:a05:6a00:1c82:b0:72f:f548:3e09]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:2d27:b0:1e1:b105:87b with SMTP id adf61e73a8af0-1ee03a8f413mr928165637.23.1738867897906; Thu, 06 Feb 2025 10:51:37 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:53 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-14-fvdl@google.com> Subject: [PATCH v3 13/28] mm: define __init_reserved_page_zone function From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Queue-Id: 70FAA80019 X-Stat-Signature: s9prk1wqog7u4xm1mdjp5ahywdbu7ee6 X-Rspamd-Server: rspam03 X-HE-Tag: 1738868313-471881 X-HE-Meta: U2FsdGVkX18mYBAizZwAtH4ohHHppjYQg+XK7jd48I9BCp5TirpX9Xm4x/wOPdTiwa8goClLIKesv+ZLjNUXoyvQKEmwN9S+hnVEwDAMhzeOUc10cA0xVt1oweekAtLqZg0ZF/kSAzK+EXZ93RVpGsush5fp4wZ9XHn3SQ9Lg4m47NgmsQMuImV0Hy/enfIzilc4BggspsSozluv3HVDg1ewdSSdpDQHEy9GyF2ZqiWoatDUkaCSiudKKEFL7NUNluMjVN9/Wn60IHSuvwjbaA6snGRHKcFqUBQWphwS7kobjDk2l5aqWjspU2j1TUATwALJ9Y93ZwV6dGOlMQ2r0khZx2bIG+yX4Kwa8sKrHmYph5dYCpuBDEBbiWcrpmIU/M/gBdgsHD1G54Jm1/C9p0bCUweWyoxeelg7iGpe5XoGUpG9Dw6fNz3w9Lao+WR7CT0yCMFqrgOgXFRvCU542llS+oGTs2Wm7an02BgrvupwKVTk9j28ktc0Ol8Z3W4WKPjtXVxzP135ZhIz3OptIOPGB2mXP6hO8schBHMYklEPZUgmVkNk2MPPVtAYudNbe0tSQHotyAX4RJlR43clPidWF5J/8NZUFfNIGNsEDX6wDcFPDnY7+VhflJuyZVpXx7O2gsDyFWI3vlgMiC8CtxX+8kbXn0QENGlpEe8YayMfiuW0XpXKQeylS78L9lz/6cj+kMpJqbtVQn/gK/bC90WdiiF4A+JCdDzn2Mq3/B7cfgS3EfMXldA47p9WLZt3PMIlpi32pXtvPt/jwGs1aCIS+TW2MrYq/Mz7XgAlhMCJC3I336kRwTZS8kik3mh5UFuDypaXqBaaz9NMrY4AVReVx02ya0nkb5STcrc/WMUnlw44bscDeo5GRX7SUn93g7VHQ/muKvTjGSk0NS8vN7qx/V3KAnbuEMCFZivUFy5kKgMdVeJhImsKFXIkokfkE/bNmJkHi8e7BGDtRQv IC76ldz3 GNprYDTyeXYU1DldGuxbb8R+UxlIYWma+BhWDNY+cGZrfZ9P/v9WTHeuxq9NrrEjz4cZ9fzJgHLsb/neLv/4yriMC4p7q/rHqsAFtDJGJGuR3XDKFwuTk6WuwQpMNFg2efzk38OrmRIGrQAVfkHcr9gaFOj1c0oCbIo0Aj9lG8Kx84J6diIHB+0wVCIqJCb6QUTZsNNJKsnhG94xiTE9VsyHLjX4fKUpjGrdSP5TzDMFjXVSXNMEF+Dcc6p3TeNp8YnKYtIw3TW31butbv97Vg+tqwA3Pq2WCcUpjuC4oygtZMCxUUdhyyis2Rznbe6fmskthCnSuOJ+CIfsY6Ap9HRVOHWTMXBYJu0C2NXWEpw9a/raywbWZqR/ZZavg1SGmpcETzgbTmiXTMyv6S1qlN1yA9TSKTTvJUfpvCFYSj2OzuJdpm4EsYhbQ0cjS7D13eN7LNeudT7BNuu9Msk7CFUuxux9MD1JUSvnfx/prcpMALvMQaROa+sav/8SehTISHFO0taR7K4/jAz8GPOWE2VwObVMGNsc+wvUiD35upsEKaju2hAjyJiGhHfSdbHZyOq8RsybgOjJpsD7z1pE99GNaRANo1tNqnuxF6AD3Yj/ZWi0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Sometimes page structs must be unconditionally initialized as reserved, regardless of DEFERRED_STRUCT_PAGE_INIT. Define a function, __init_reserved_page_zone, containing code that already did all of the work in init_reserved_page, and make it available for use. Signed-off-by: Frank van der Linden --- mm/internal.h | 1 + mm/mm_init.c | 38 +++++++++++++++++++++++--------------- 2 files changed, 24 insertions(+), 15 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 109ef30fee11..57662141930e 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -1448,6 +1448,7 @@ static inline bool pte_needs_soft_dirty_wp(struct vm_area_struct *vma, pte_t pte void __meminit __init_single_page(struct page *page, unsigned long pfn, unsigned long zone, int nid); +void __meminit __init_reserved_page_zone(unsigned long pfn, int nid); /* shrinker related functions */ unsigned long shrink_slab(gfp_t gfp_mask, int nid, struct mem_cgroup *memcg, diff --git a/mm/mm_init.c b/mm/mm_init.c index 9f1e41c3dde6..925ed6564572 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -650,6 +650,28 @@ static inline void fixup_hashdist(void) static inline void fixup_hashdist(void) {} #endif /* CONFIG_NUMA */ +/* + * Initialize a reserved page unconditionally, finding its zone first. + */ +void __meminit __init_reserved_page_zone(unsigned long pfn, int nid) +{ + pg_data_t *pgdat; + int zid; + + pgdat = NODE_DATA(nid); + + for (zid = 0; zid < MAX_NR_ZONES; zid++) { + struct zone *zone = &pgdat->node_zones[zid]; + + if (zone_spans_pfn(zone, pfn)) + break; + } + __init_single_page(pfn_to_page(pfn), pfn, zid, nid); + + if (pageblock_aligned(pfn)) + set_pageblock_migratetype(pfn_to_page(pfn), MIGRATE_MOVABLE); +} + #ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT static inline void pgdat_set_deferred_range(pg_data_t *pgdat) { @@ -708,24 +730,10 @@ defer_init(int nid, unsigned long pfn, unsigned long end_pfn) static void __meminit init_reserved_page(unsigned long pfn, int nid) { - pg_data_t *pgdat; - int zid; - if (early_page_initialised(pfn, nid)) return; - pgdat = NODE_DATA(nid); - - for (zid = 0; zid < MAX_NR_ZONES; zid++) { - struct zone *zone = &pgdat->node_zones[zid]; - - if (zone_spans_pfn(zone, pfn)) - break; - } - __init_single_page(pfn_to_page(pfn), pfn, zid, nid); - - if (pageblock_aligned(pfn)) - set_pageblock_migratetype(pfn_to_page(pfn), MIGRATE_MOVABLE); + __init_reserved_page_zone(pfn, nid); } #else static inline void pgdat_set_deferred_range(pg_data_t *pgdat) {} From patchwork Thu Feb 6 18:50:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963561 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 13301C02194 for ; Thu, 6 Feb 2025 18:51:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C026728000D; Thu, 6 Feb 2025 13:51:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B89F7280002; Thu, 6 Feb 2025 13:51:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98C8228000D; Thu, 6 Feb 2025 13:51:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 66F4F280002 for ; Thu, 6 Feb 2025 13:51:42 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2389EB255A for ; Thu, 6 Feb 2025 18:51:42 +0000 (UTC) X-FDA: 83090413644.04.4092E5E Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf12.hostedemail.com (Postfix) with ESMTP id 4CB1340007 for ; Thu, 6 Feb 2025 18:51:40 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=H0b8KFIE; spf=pass (imf12.hostedemail.com: domain of 3uwSlZwQKCJwBR9HCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3uwSlZwQKCJwBR9HCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867900; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1XzNWdK53DK5kTMFUNWfsYU5y9JEDM1DtaDFYklZD4s=; b=FRBVWzIjRMq22VYdZKPqMRDiVz48FJeg/3bsrYMB5i7jnn5WZDpfbCLDlalbkZXuCMEH5h /72hmoWF8ygxetTp8HItjY+u5MF8o9d49thWaf/nuts+DOH6UbVPybmqNeFKlcIshpKdU3 wVywYzGt90RWnIpUEnS3lf4QfVHthmg= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=H0b8KFIE; spf=pass (imf12.hostedemail.com: domain of 3uwSlZwQKCJwBR9HCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3uwSlZwQKCJwBR9HCKKCHA.8KIHEJQT-IIGR68G.KNC@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867900; a=rsa-sha256; cv=none; b=MkT1MBKpxNgghNuGBvw+EWWS4dyJ/+GQunfU/pEbU3imsIwXUljpg53i8Vp078niLEt2Tl V9uKsC1UBCLvjEdf2D8uXONj5zNUHRqPmJh5Mjn5lx8Yuk5EFlHpx7MtIBU1j7TGNyL1V9 HIJChmF+peirkT/5GCL/eqA3bZHufIU= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-21f1e5c641aso43151035ad.2 for ; Thu, 06 Feb 2025 10:51:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867899; x=1739472699; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1XzNWdK53DK5kTMFUNWfsYU5y9JEDM1DtaDFYklZD4s=; b=H0b8KFIEuub4K07VUUyjU3VTQ1xhjWazuZPStyWh0fv9Qo++SwRUqDOZnZYvgj/7NC PZGu4DTxRxKAORdhXNDnUK8h0ShOLMChqtZ3HDol+PpimZMtsDU6YCn3bgF6rbQf9LCH rso5nTsux/hNSzvGfgu2W+eSR5tsEHweERsUak0xLs6m4/dCP7H3umCqAtfnEZZsGxMU 5vVVk28DeexCtE1IMxU1DiQkndWom70qfm96b6sMq0BBkmMDRfEtCBRHGGFMLo7qE7se iHi3YmO6P9Xekl4su/r6cy2qcHXfPZPmZj6bIRNPPwRGUvIJwpQ2jylyq8JftIwgbSnc uwOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867899; x=1739472699; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1XzNWdK53DK5kTMFUNWfsYU5y9JEDM1DtaDFYklZD4s=; b=X835oC4RS8y3O9q+Abau3Raybclf74FXMNcDfIu04pnAuSu4l5maM0XRRAVW0Jimbs X6gngJN3InpljGNTalsNW5RycXbJL9YOtllK/5ym9txOBIMM+/OGOEKObr3Z78M9vicF ZhGAcELq80G2H3grQRR6/hJEqVJUhXFREzhMmsSXAyoeYthj9cXUN1KO1LA3+okix8/1 HXo/295rG/a/zuOm3QqNEPTU8yI40fv/JW3Kz99MsrYnVBRz6APSTQzyE++G4OnT2q/5 1fc/yHXRCxU9gfz6sPscFYgxEc0YxIdd4nTHLQ2U9bX0kQwD2E5foH2oMGVv3Ukoy3Df BHpQ== X-Forwarded-Encrypted: i=1; AJvYcCVdfnzdlCYPKS9xHJNemQovANx/MStHSusmoRehUgr8cg+aXVpiIJTQXTcuB3jvFKe4pYN/bliKkQ==@kvack.org X-Gm-Message-State: AOJu0YywUTRkRn97utCMYrEqHvP8l5eSrCz0lBo7WSoOsIC4Y2JPLsaN 9RnpiPLP+mQbere0HZFoaVSKzMZkoY/7d/DmdwukQ0dCf0ULYafNWURaBazM76uxrYxYjg== X-Google-Smtp-Source: AGHT+IGIkx9zow3SNmjwLwkWX732+NMszmOw5UwG52/2qtVEFt/yi4VJkAhAKbmBsmMF+ToCMr1/WRbP X-Received: from pgbdl12.prod.google.com ([2002:a05:6a02:d0c:b0:ad5:1a11:9dae]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:3a87:b0:1ed:a565:f980 with SMTP id adf61e73a8af0-1ee03bb030cmr937467637.27.1738867899243; Thu, 06 Feb 2025 10:51:39 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:54 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-15-fvdl@google.com> Subject: [PATCH v3 14/28] mm/hugetlb: check bootmem pages for zone intersections From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 4CB1340007 X-Stat-Signature: 338oqr545egwhsqic5z3eczbbtk6bsxq X-HE-Tag: 1738867900-668602 X-HE-Meta: U2FsdGVkX1/e7JEKjR4TzpqU2THaU4Wq6kYPAb6xvfT2B6BRD1nLAI0iE/Uk/ApAQgSICmQ+OUoUGR8YiCozCt0WbIKhurtLklN9ZO9n4vpMOIjgVVbtnCglLL+pb7ukXwn3xQdYd+NvsMB/wwHUV0GumXmXm7jDzLWT7s2yoqcSJY/WNH7bKD2+7hDRdQfPhwAFtMGyYisV+OFZMLcJcspYsVuTfezqg333nTjDH5OwC2ZN45SmAZQPEH3kgMWW7xEackbnDuk0e1TAGPYT/6z2LUEadQ3Em3EulAH0vHJn/lox0kWqb4uPUkOQnwnpFWON/69EmYk5gmWSzo+g6esP3O8nhvx/PG1oCBUeOFL4YD6zuS8hXwHqr2DAy1nmImM1RPJjK6+FaZgNk/tgePJyxwldIDEaPo0r63gSWqNBw7DSdNVZbbgVKeF/r0YOjdXlvl4vyDBFMEMEGUI1B7P9mvVatBAJAwhGHq00HJZueFaXS8bQpxdHyhVD7idJ61Dme/6A7HCB6foPH3w8ufB0OF7QULJiD/erGJ6vB6lJ0sMVd60dQVP60aD78mL+0+P5S6103AUtgxZQkRZENHqHC5rdot/ct/MFNqYGvkcUC7mX5nDWUY6eheE8CZVNf3mjEY9/ct6mn5yafEy6S+GWdMJ/lYtgSI+GoiP+Dful2arwZr/xdqcc5mRCAtlhGa7CoBa3dnSTkH5GZVvQoab6i9oDGHjidQto0dLq7KyymmA3o2fYPDUvR5T3q5t8QQmWUEIXQMDAOh4VIRtM+RmTRCnSKJkud6/VhdJufpt4rgupfiAT4fhKAtFjtWZoRnLP/OqInWmhbMzB7lnphUYqCJc3/IySFPFNRPZyYxWXUTq6Lt31Ejtw4qCdUeRXdlqwFa0sk8K+pxgG5DuqHbQaPpJNPxYwIcmW2704wZGnEgDXFklin7DS8Mns+ZtG+i5jZ9gW9XKeTeCpwhD kn5ro3w6 U0FX3/KEGwwjtTpud0yf36va/t8OSAMbyf8M6pTSJZ8kUrZ4WUVgqUI2wgawQjI0brq0oWgNLLR9c/KxMDFrTR5ChqOusqBh2S5ZnVm2F/xdRDW3vE013O613roV6EMYWX51XeBcfgn5zHmlVM6MTBTFYMj3tobqo9avc66P7N10gJ+93jj6w3LCM5AfHX0kwzQ3Ys6z7/kBa7AB7CgXV0s6oFD/z47ufeCyqPuox7vMFhuCSzjkB5g98W/owda55hLDCSr37cb64cBT4reWVMxYCNVBRKbu4f/lHKGPZDmCb2+H9UEkdqu/CvG0hLlCGxobPO0ueDG7obdJPLjGaV4QEvEfMYj0lWlMhDyvb0Pw4Ha+OtDvYRDSU019SObdx8iPh38GOETDUunoxr9FWDJOI7DfhNW9DoKEgYnjiqOLxWZpPXagZME+6K5zNAn2h/eQGkXjkx/wqyArlMjG9qf806vgfcfMqSkQDIG/1arIGR+Yl7HfMCVmibnLXDKxl8Iv7MqTRfSmap1XUYYcHgROfeA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Bootmem hugetlb pages are allocated using memblock, which isn't (and mostly can't be) aware of zones. So, they may end up crossing zone boundaries. This would create confusion, a hugetlb page that is part of multiple zones is bad. Worse, HVO might then end up stealthily re-assigning pages to a different zone when a hugetlb page is freed, since the tail page structures beyond the first vmemmap page would inherit the zone of the first page structures. While the chance of this happening is low, you can definitely create a configuration where this happens (especially using ZONE_MOVABLE). To avoid this issue, check if bootmem hugetlb pages intersect with multiple zones during the gather phase, and discard them, handing them to the page allocator, if they do. Record the number of invalid bootmem pages per node and subtract them from the number of available pages at the end, making it easier to do these checks in multiple places later on. Signed-off-by: Frank van der Linden --- mm/hugetlb.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++-- mm/internal.h | 2 ++ mm/mm_init.c | 25 +++++++++++++++++++++ 3 files changed, 86 insertions(+), 2 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index de8adfb487f4..789341158ef6 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -62,6 +62,7 @@ static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; static unsigned long hugetlb_cma_size __initdata; __initdata struct list_head huge_boot_pages[MAX_NUMNODES]; +static unsigned long hstate_boot_nrinvalid[HUGE_MAX_HSTATE] __initdata; /* * Due to ordering constraints across the init code for various @@ -3308,6 +3309,44 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, } } +static bool __init hugetlb_bootmem_page_zones_valid(int nid, + struct huge_bootmem_page *m) +{ + unsigned long start_pfn; + bool valid; + + start_pfn = virt_to_phys(m) >> PAGE_SHIFT; + + valid = !pfn_range_intersects_zones(nid, start_pfn, + pages_per_huge_page(m->hstate)); + if (!valid) + hstate_boot_nrinvalid[hstate_index(m->hstate)]++; + + return valid; +} + +/* + * Free a bootmem page that was found to be invalid (intersecting with + * multiple zones). + * + * Since it intersects with multiple zones, we can't just do a free + * operation on all pages at once, but instead have to walk all + * pages, freeing them one by one. + */ +static void __init hugetlb_bootmem_free_invalid_page(int nid, struct page *page, + struct hstate *h) +{ + unsigned long npages = pages_per_huge_page(h); + unsigned long pfn; + + while (npages--) { + pfn = page_to_pfn(page); + __init_reserved_page_zone(pfn, nid); + free_reserved_page(page); + page++; + } +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. @@ -3315,14 +3354,25 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, static void __init gather_bootmem_prealloc_node(unsigned long nid) { LIST_HEAD(folio_list); - struct huge_bootmem_page *m; + struct huge_bootmem_page *m, *tm; struct hstate *h = NULL, *prev_h = NULL; - list_for_each_entry(m, &huge_boot_pages[nid], list) { + list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) { struct page *page = virt_to_page(m); struct folio *folio = (void *)page; h = m->hstate; + if (!hugetlb_bootmem_page_zones_valid(nid, m)) { + /* + * Can't use this page. Initialize the + * page structures if that hasn't already + * been done, and give them to the page + * allocator. + */ + hugetlb_bootmem_free_invalid_page(nid, page, h); + continue; + } + /* * It is possible to have multiple huge page sizes (hstates) * in this list. If so, process each size separately. @@ -3594,13 +3644,20 @@ static void __init hugetlb_init_hstates(void) static void __init report_hugepages(void) { struct hstate *h; + unsigned long nrinvalid; for_each_hstate(h) { char buf[32]; + nrinvalid = hstate_boot_nrinvalid[hstate_index(h)]; + h->max_huge_pages -= nrinvalid; + string_get_size(huge_page_size(h), 1, STRING_UNITS_2, buf, 32); pr_info("HugeTLB: registered %s page size, pre-allocated %ld pages\n", buf, h->free_huge_pages); + if (nrinvalid) + pr_info("HugeTLB: %s page size: %lu invalid page%s discarded\n", + buf, nrinvalid, nrinvalid > 1 ? "s" : ""); pr_info("HugeTLB: %d KiB vmemmap can be freed for a %s page\n", hugetlb_vmemmap_optimizable_size(h) / SZ_1K, buf); } diff --git a/mm/internal.h b/mm/internal.h index 57662141930e..63fda9bb9426 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -658,6 +658,8 @@ static inline struct page *pageblock_pfn_to_page(unsigned long start_pfn, } void set_zone_contiguous(struct zone *zone); +bool pfn_range_intersects_zones(int nid, unsigned long start_pfn, + unsigned long nr_pages); static inline void clear_zone_contiguous(struct zone *zone) { diff --git a/mm/mm_init.c b/mm/mm_init.c index 925ed6564572..f7d5b4fe1ae9 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2287,6 +2287,31 @@ void set_zone_contiguous(struct zone *zone) zone->contiguous = true; } +/* + * Check if a PFN range intersects multiple zones on one or more + * NUMA nodes. Specify the @nid argument if it is known that this + * PFN range is on one node, NUMA_NO_NODE otherwise. + */ +bool pfn_range_intersects_zones(int nid, unsigned long start_pfn, + unsigned long nr_pages) +{ + struct zone *zone, *izone = NULL; + + for_each_zone(zone) { + if (nid != NUMA_NO_NODE && zone_to_nid(zone) != nid) + continue; + + if (zone_intersects(zone, start_pfn, nr_pages)) { + if (izone != NULL) + return true; + izone = zone; + } + + } + + return false; +} + static void __init mem_init_print_info(void); void __init page_alloc_init_late(void) { From patchwork Thu Feb 6 18:50:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963562 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0143C02199 for ; Thu, 6 Feb 2025 18:51:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EB3B28000E; Thu, 6 Feb 2025 13:51:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34C6A280002; Thu, 6 Feb 2025 13:51:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 066B828000E; Thu, 6 Feb 2025 13:51:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D7B65280002 for ; Thu, 6 Feb 2025 13:51:43 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A01C21C90A9 for ; Thu, 6 Feb 2025 18:51:43 +0000 (UTC) X-FDA: 83090413686.03.A5412F7 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf05.hostedemail.com (Postfix) with ESMTP id D4A17100012 for ; Thu, 6 Feb 2025 18:51:41 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GtLBRzsg; spf=pass (imf05.hostedemail.com: domain of 3vASlZwQKCJ0CSAIDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3vASlZwQKCJ0CSAIDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867901; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nFoWWRyCHF1Iin3pK7rG6BcJvDoZ6S/iIkIiqdRP1Hc=; b=bdi5qw3Y2a0kOm9Q15DQV4nMiJIk2sI9DHXuBsBLN/EghMr30JpteLVMJa8/j0wVWYGsP7 fkyrJ9BHfEbJeBtW+OZGDWuo//ic3b1OOf4O2GguBFRIKdgkpTgnFfAmKjWhUydJfpGkWh qyNWkQIqXQsiIsjgdo9AhjyB+vne57I= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=GtLBRzsg; spf=pass (imf05.hostedemail.com: domain of 3vASlZwQKCJ0CSAIDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3vASlZwQKCJ0CSAIDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867901; a=rsa-sha256; cv=none; b=rDprWo4JYjbFbFydhORdgjYYz4lqoIP+obYzcEZdNGkzo75CMdGdC/BCc3Y91Wy09ZQd12 bI/iSVwBkPSh8D58Ye8TdVeu7GRxqrFLnmPOETcL5EVMUiZtCfeE9OfP62+1qbB8Q3Xnek eSH4FggNO0N3nZ/DreTgr8GFEYf1uis= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-21661949f23so41401955ad.3 for ; Thu, 06 Feb 2025 10:51:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867901; x=1739472701; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=nFoWWRyCHF1Iin3pK7rG6BcJvDoZ6S/iIkIiqdRP1Hc=; b=GtLBRzsgV0IvGL6uAX9sUVRHAy2kIA4rjewMSVKsPGUADTWEb0wfEwdCkdUB6HGKi+ UbMsaM3Wqrc+pmLjRZfAc1bj0g5afozw22EVJBVlMCVpTxh1d9w9fpD964QsKrA8Z/MV 5QLRYkfxs6qgCzCYwNT8MVsNE8p7MTgVWc+2LtNLzT8h7s+3FGgqHS63hwUSQqLgAa2S sBtdlSLGSx3sWUZjw7dhyp+mbpWypqo3nVhBYal/O1IoFrNT+kqbhz7Jz9uDo72zSFGq Tfoj8hunnunrQW6t0s4FCaWlyszPMCM56Kfa1Lo3hJ8o9ytBzYTjO1ck2hc2DlCEPVUg B6YA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867901; x=1739472701; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=nFoWWRyCHF1Iin3pK7rG6BcJvDoZ6S/iIkIiqdRP1Hc=; b=njSz3dECn3PJjo+rDaMPdn8pZUpSb8NuVx90v1kXCHD3R2E6+nzoI5MXvJqLgqxK40 OzCkY5Y7/SdXHIDYnKc8LZmRyJz0n9Sf3eaDkJfO2aYKgGxSmIW2bEk1SjANz6VfvtIu WYx9TDRfXTOyTqB79p3lNK+OhnSvyp9EPh28vmW4tBSQPmRq+Q+0ddNtTnH+308DlJa3 I9I/lKEGpw6E4uwBIe3CkL59IW1g7OPPo7SV/MrTFzJ4LYuOnqCECoI3uMb+PBfjEDNb cyKBddQEauqYoqePN7oAMyPsOXXQa3OiVgP8nesS2dhlLOXtTIahYIfXJkcGLdZMTdqw hhoQ== X-Forwarded-Encrypted: i=1; AJvYcCUReb+TOzHjzeZ8gQWnTI44M3FzB6Van/YJ757I8P07h6ik6UP12cw3Ai+ATzR2jI7vRGphkKgY/w==@kvack.org X-Gm-Message-State: AOJu0YwS/Uh/zTapmyZ163qK9UqPqPIiDQjpp541VG5dpw7PfQ7jwsnk vILYEzFarwatFJz36VV/bOPy3isaESqKKXwRCPZOQEWHYDxfp68kAc1MXF6crskBHWtBgA== X-Google-Smtp-Source: AGHT+IEGjRoSpQn8/xkl3lhb6Wk2y9VhZAipET65ydINQje9MB9/NNORs9N8PFdtkfHgKJaqcoUjp9yu X-Received: from pfvx11.prod.google.com ([2002:a05:6a00:270b:b0:725:dec7:dd47]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:6f91:b0:1dc:2365:4d7d with SMTP id adf61e73a8af0-1ee03a4744fmr811464637.16.1738867900833; Thu, 06 Feb 2025 10:51:40 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:55 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-16-fvdl@google.com> Subject: [PATCH v3 15/28] mm/sparse: add vmemmap_*_hvo functions From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: D4A17100012 X-Stat-Signature: 7hce67pmqqxnup4bondmranmt64afi6b X-HE-Tag: 1738867901-288276 X-HE-Meta: U2FsdGVkX1/7DxhXC6EOpfU8j5AZPVYofCcCCV4UVlTKcg6qLGAJ+jXOg/rp6f/UV4bGpXgRAmc+vmL24xkkL51CUG836bjh219lWaJLdVxav/RGyvJWeTWriQJ1oRJ8HvVm9g89wWKVLwU4eZ3akhxVlpzCBHwg5dSBATDYUuHNaDcpJVcEDsE30Xx1hR4z/u5bSAKWWtLA2OOrhg+DBU2LpAGvbzrL0DleMcsdg7UNdvgYr8sTshvJRiQc8Ku7Nh0w1VVjkZxsmuHcYiSutvAhjwJQUp0jBL/EHAhPl1FzCvzQx53Ptg3Onxm8igdm6TAlilSA1rG2Urrd0hynF4HerpOqSPrMmDyoci5ZZAvWIL5OOJ5Vdfl1SYJJYxO+VsV3uREHwjE2el42k2e2ayP1vmq6VMhPQeOm6fn3m9eB/hyw2UQKDNNX3/oBZtg4VmEaR/+Bx8vudLMH089w/i5c0VJHjGWpVYeeTDpprFbzk4e3/WI9cZVaqD7gwkpSHDmq6CrCOdga1xLhrrVLmJb70F/hb6F0Key1aEU57tRhASYKYM8azHYYYfFPdCImK4uzQ/pYIJf/GVXQ/3G6S2bwpr5ZQ0485oLXXxJAyREVy5IhI7YIxgKA0qGrI2zC5QkFmcgtb4PdtGyIAal1xEUdhMlzhFDB5em7ya40zGj1mxfOLkIb/LopKGQnRlCRXd7QV7PRDAxWdRXd1t/GkI/g+FhBflVumUjhWn53n5QaOaLU08QOg5KBy0qP6uGwDM5jiH6wrr6iyrJ1d+jSPQySgMf37u8L8gR5xp6NTOUEmY9+8xkYYDLbXa4f06fc4UZEmEAtCmBGffVTgzm3bTykc/yjkn4sUl8EYocEDjwxmMpx9RDQlSZfovUv4cn0Wuj0cPmJrtMRNe2hJWVjjpTejiapAL+theZg+YTIIgakLGL8EY65u0H34jMqSnZEiiMVXO6kZzX6WCaj0Ik W1OaL4aJ q9Wwk2BFAoG5sU75QLeRequcghhGwA6iL+VFQB3Io1K/iMAE6eVEs4qOPYBie54p5ROuEdfssLWVwjl5wxz+9DgdEKhKKVUaVpyS2jhxVN1ko3ftKAKcM0M4D98HdGDpjh+64fGfJmKUXL8ymZUMVGKJAc6KPD+TRb4v4UPmr2LbyIP0MUHmD2A/FxlRnA92o75kf+LYxquB7aDtBFkG2jh8kYiEXh6zMWHUnWW9boONcLMJwa0kDVJ+MJdwXY+f+nWm6V8v+Sy71uQ0mUQlCzGJu/18V3t5QW1FhYqF/PRcgWbgtnln+aqmB06kIcOYR1woY0VsxrB+R0dBfKqUd7JZeeGUkju7Wptalz+9O6/n1LWjLyw9jY2MUnNsdqibd45wuN7+JPuiwalLvmr8nRcBEQBcWtiCkXtX6Qs/UFMEYudJNPIaF1aE/VAqb0ebEPWD7aDeVpT1dZGC4cfNX2Qqfmu/Xb0B6YcafnX/12Eh6Dby7ZlPZdZQQkMKeDm2Q+nZbFQSmHrzTPpqdDXGySDrF4+epkchloRC1wBfF11JFj8TLCcVyFsahUZrtaGJhOJYkfSdqYGjSK2E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a few functions to enable early HVO: vmemmap_populate_hvo vmemmap_undo_hvo vmemmap_wrprotect_hvo The populate and undo functions are expected to be used in early init, from the sparse_init_nid_early() function. The wrprotect function is to be used, potentially, later. To implement these functions, mostly re-use the existing compound pages vmemmap logic used by DAX. vmemmap_populate_address has its argument changed a bit in this commit: the page structure passed in to be reused in the mapping is replaced by a PFN and a flag. The flag indicates whether an extra ref should be taken on the vmemmap page containing the head page structure. Taking the ref is appropriate to for DAX / ZONE_DEVICE, but not for HugeTLB HVO. The HugeTLB vmemmap optimization maps tail page structure pages read-only. The vmemmap_wrprotect_hvo function that does this is implemented separately, because it cannot be guaranteed that reserved page structures will not be write accessed during memory initialization. Even with CONFIG_DEFERRED_STRUCT_PAGE_INIT, they might still be written to (if they are at the bottom of a zone). So, vmemmap_populate_hvo leaves the tail page structure pages RW initially, and then later during initialization, after memmap init is fully done, vmemmap_wrprotect_hvo must be called to finish the job. Subsequent commits will use these functions for early HugeTLB HVO. Signed-off-by: Frank van der Linden --- include/linux/mm.h | 9 ++- mm/sparse-vmemmap.c | 141 +++++++++++++++++++++++++++++++++++++++----- 2 files changed, 135 insertions(+), 15 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index df83653ed6e3..0463c062fd7a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3837,7 +3837,8 @@ p4d_t *vmemmap_p4d_populate(pgd_t *pgd, unsigned long addr, int node); pud_t *vmemmap_pud_populate(p4d_t *p4d, unsigned long addr, int node); pmd_t *vmemmap_pmd_populate(pud_t *pud, unsigned long addr, int node); pte_t *vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, - struct vmem_altmap *altmap, struct page *reuse); + struct vmem_altmap *altmap, unsigned long ptpfn, + unsigned long flags); void *vmemmap_alloc_block(unsigned long size, int node); struct vmem_altmap; void *vmemmap_alloc_block_buf(unsigned long size, int node, @@ -3853,6 +3854,12 @@ int vmemmap_populate_hugepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); int vmemmap_populate(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap); +int vmemmap_populate_hvo(unsigned long start, unsigned long end, int node, + unsigned long headsize); +int vmemmap_undo_hvo(unsigned long start, unsigned long end, int node, + unsigned long headsize); +void vmemmap_wrprotect_hvo(unsigned long start, unsigned long end, int node, + unsigned long headsize); void vmemmap_populate_print_last(void); #ifdef CONFIG_MEMORY_HOTPLUG void vmemmap_free(unsigned long start, unsigned long end, diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 8751c46c35e4..8cc848c4b17c 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -30,6 +30,13 @@ #include #include +#include + +/* + * Flags for vmemmap_populate_range and friends. + */ +/* Get a ref on the head page struct page, for ZONE_DEVICE compound pages */ +#define VMEMMAP_POPULATE_PAGEREF 0x0001 #include "internal.h" @@ -144,17 +151,18 @@ void __meminit vmemmap_verify(pte_t *pte, int node, pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, struct vmem_altmap *altmap, - struct page *reuse) + unsigned long ptpfn, unsigned long flags) { pte_t *pte = pte_offset_kernel(pmd, addr); if (pte_none(ptep_get(pte))) { pte_t entry; void *p; - if (!reuse) { + if (ptpfn == (unsigned long)-1) { p = vmemmap_alloc_block_buf(PAGE_SIZE, node, altmap); if (!p) return NULL; + ptpfn = PHYS_PFN(__pa(p)); } else { /* * When a PTE/PMD entry is freed from the init_mm @@ -165,10 +173,10 @@ pte_t * __meminit vmemmap_pte_populate(pmd_t *pmd, unsigned long addr, int node, * and through vmemmap_populate_compound_pages() when * slab is available. */ - get_page(reuse); - p = page_to_virt(reuse); + if (flags & VMEMMAP_POPULATE_PAGEREF) + get_page(pfn_to_page(ptpfn)); } - entry = pfn_pte(__pa(p) >> PAGE_SHIFT, PAGE_KERNEL); + entry = pfn_pte(ptpfn, PAGE_KERNEL); set_pte_at(&init_mm, addr, pte, entry); } return pte; @@ -238,7 +246,8 @@ pgd_t * __meminit vmemmap_pgd_populate(unsigned long addr, int node) static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, struct vmem_altmap *altmap, - struct page *reuse) + unsigned long ptpfn, + unsigned long flags) { pgd_t *pgd; p4d_t *p4d; @@ -258,7 +267,7 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, pmd = vmemmap_pmd_populate(pud, addr, node); if (!pmd) return NULL; - pte = vmemmap_pte_populate(pmd, addr, node, altmap, reuse); + pte = vmemmap_pte_populate(pmd, addr, node, altmap, ptpfn, flags); if (!pte) return NULL; vmemmap_verify(pte, node, addr, addr + PAGE_SIZE); @@ -269,13 +278,15 @@ static pte_t * __meminit vmemmap_populate_address(unsigned long addr, int node, static int __meminit vmemmap_populate_range(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap, - struct page *reuse) + unsigned long ptpfn, + unsigned long flags) { unsigned long addr = start; pte_t *pte; for (; addr < end; addr += PAGE_SIZE) { - pte = vmemmap_populate_address(addr, node, altmap, reuse); + pte = vmemmap_populate_address(addr, node, altmap, + ptpfn, flags); if (!pte) return -ENOMEM; } @@ -286,7 +297,107 @@ static int __meminit vmemmap_populate_range(unsigned long start, int __meminit vmemmap_populate_basepages(unsigned long start, unsigned long end, int node, struct vmem_altmap *altmap) { - return vmemmap_populate_range(start, end, node, altmap, NULL); + return vmemmap_populate_range(start, end, node, altmap, -1, 0); +} + +/* + * Undo populate_hvo, and replace it with a normal base page mapping. + * Used in memory init in case a HVO mapping needs to be undone. + * + * This can happen when it is discovered that a memblock allocated + * hugetlb page spans multiple zones, which can only be verified + * after zones have been initialized. + * + * We know that: + * 1) The first @headsize / PAGE_SIZE vmemmap pages were individually + * allocated through memblock, and mapped. + * + * 2) The rest of the vmemmap pages are mirrors of the last head page. + */ +int __meminit vmemmap_undo_hvo(unsigned long addr, unsigned long end, + int node, unsigned long headsize) +{ + unsigned long maddr, pfn; + pte_t *pte; + int headpages; + + /* + * Should only be called early in boot, so nothing will + * be accessing these page structures. + */ + WARN_ON(!early_boot_irqs_disabled); + + headpages = headsize >> PAGE_SHIFT; + + /* + * Clear mirrored mappings for tail page structs. + */ + for (maddr = addr + headsize; maddr < end; maddr += PAGE_SIZE) { + pte = virt_to_kpte(maddr); + pte_clear(&init_mm, maddr, pte); + } + + /* + * Clear and free mappings for head page and first tail page + * structs. + */ + for (maddr = addr; headpages-- > 0; maddr += PAGE_SIZE) { + pte = virt_to_kpte(maddr); + pfn = pte_pfn(ptep_get(pte)); + pte_clear(&init_mm, maddr, pte); + memblock_phys_free(PFN_PHYS(pfn), PAGE_SIZE); + } + + flush_tlb_kernel_range(addr, end); + + return vmemmap_populate(addr, end, node, NULL); +} + +/* + * Write protect the mirrored tail page structs for HVO. This will be + * called from the hugetlb code when gathering and initializing the + * memblock allocated gigantic pages. The write protect can't be + * done earlier, since it can't be guaranteed that the reserved + * page structures will not be written to during initialization, + * even if CONFIG_DEFERRED_STRUCT_PAGE_INIT is enabled. + * + * The PTEs are known to exist, and nothing else should be touching + * these pages. The caller is responsible for any TLB flushing. + */ +void vmemmap_wrprotect_hvo(unsigned long addr, unsigned long end, + int node, unsigned long headsize) +{ + unsigned long maddr; + pte_t *pte; + + for (maddr = addr + headsize; maddr < end; maddr += PAGE_SIZE) { + pte = virt_to_kpte(maddr); + ptep_set_wrprotect(&init_mm, maddr, pte); + } +} + +/* + * Populate vmemmap pages HVO-style. The first page contains the head + * page and needed tail pages, the other ones are mirrors of the first + * page. + */ +int __meminit vmemmap_populate_hvo(unsigned long addr, unsigned long end, + int node, unsigned long headsize) +{ + pte_t *pte; + unsigned long maddr; + + for (maddr = addr; maddr < addr + headsize; maddr += PAGE_SIZE) { + pte = vmemmap_populate_address(maddr, node, NULL, -1, 0); + if (!pte) + return -ENOMEM; + } + + /* + * Reuse the last page struct page mapped above for the rest. + */ + return vmemmap_populate_range(maddr, end, node, NULL, + pte_pfn(ptep_get(pte)), 0); } void __weak __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, @@ -409,7 +520,8 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, * with just tail struct pages. */ return vmemmap_populate_range(start, end, node, NULL, - pte_page(ptep_get(pte))); + pte_pfn(ptep_get(pte)), + VMEMMAP_POPULATE_PAGEREF); } size = min(end - start, pgmap_vmemmap_nr(pgmap) * sizeof(struct page)); @@ -417,13 +529,13 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, unsigned long next, last = addr + size; /* Populate the head page vmemmap page */ - pte = vmemmap_populate_address(addr, node, NULL, NULL); + pte = vmemmap_populate_address(addr, node, NULL, -1, 0); if (!pte) return -ENOMEM; /* Populate the tail pages vmemmap page */ next = addr + PAGE_SIZE; - pte = vmemmap_populate_address(next, node, NULL, NULL); + pte = vmemmap_populate_address(next, node, NULL, -1, 0); if (!pte) return -ENOMEM; @@ -433,7 +545,8 @@ static int __meminit vmemmap_populate_compound_pages(unsigned long start_pfn, */ next += PAGE_SIZE; rc = vmemmap_populate_range(next, last, node, NULL, - pte_page(ptep_get(pte))); + pte_pfn(ptep_get(pte)), + VMEMMAP_POPULATE_PAGEREF); if (rc) return -ENOMEM; } From patchwork Thu Feb 6 18:50:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963563 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83E77C0219B for ; Thu, 6 Feb 2025 18:52:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB1ED28000F; Thu, 6 Feb 2025 13:51:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3984280002; Thu, 6 Feb 2025 13:51:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 93EE928000F; Thu, 6 Feb 2025 13:51:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 70D4D280002 for ; Thu, 6 Feb 2025 13:51:45 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 33D1AA13CF for ; Thu, 6 Feb 2025 18:51:45 +0000 (UTC) X-FDA: 83090413770.01.030CACD Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf16.hostedemail.com (Postfix) with ESMTP id 5200C18000E for ; Thu, 6 Feb 2025 18:51:43 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=I+TWC4CH; spf=pass (imf16.hostedemail.com: domain of 3vgSlZwQKCJ8EUCKFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3vgSlZwQKCJ8EUCKFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HYCnhKfOpNHoXGLJGoUQUcx2tHg42NkQx3e7q6fph54=; b=qswhgFWut2Mhd+c0q9EPJaghX3PSIJn3RJS/A24P12X7hvz2Kht86CYBlMHUuEuAMlEnOG xGxxbDk1MnGeeXth/l+C0EMTKCWOoiwKOQ8iD9kBei5gftzOj3ed9Au+4kkC8SuXqVmwyo 2o8lHxAHHgbtJc2RCqlWMHVEVBJE7ek= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=I+TWC4CH; spf=pass (imf16.hostedemail.com: domain of 3vgSlZwQKCJ8EUCKFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3vgSlZwQKCJ8EUCKFNNFKD.BNLKHMTW-LLJU9BJ.NQF@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867903; a=rsa-sha256; cv=none; b=BMJryFVqC6RMhhnbqOjx/r/CgVsZgU0aJ/dr+0M89In3qn+WKF/Ac0DUZ+Xd/k4mTfoN1X xi/EgZYuxRK4dxLCBW62V3tq6FWIw3gPSzoAAlMwXJOvZ+rvPa8fSNFCjdMNv9PPY0ATn7 w1LDYP4mmT6WsMZ53XeRhTf1hFp0ajk= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-216387ddda8so27457345ad.3 for ; Thu, 06 Feb 2025 10:51:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867902; x=1739472702; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HYCnhKfOpNHoXGLJGoUQUcx2tHg42NkQx3e7q6fph54=; b=I+TWC4CHOemCspNBozFHlL2+jUq9MdpRNysDp9Gh9cxvLesdwzcDtbNzYjZnzcQp0Y 9JF1awgz7j6p+09Su8x9l9F708mhT/hwIUq78CbBUjX3zwlz4M+KFQgUjYiPpajQsG8E W2EWlhgS7wXVxlHAc2c2suebSsixlXFt7aByCE/FxOlfDbP3hMQ/3lEzXp6idsIVRHRg yU297lwRXuFfGj0KTutLC98mnZX/1bGp65ua9CT7Nd6ZZuRFovRN/gRmFpXKjh5NtxW2 orXKPcvK985I5tNnmxWJ0n7FhZRmS+tNEODXTYPT3Rpl+KTl/QPsSM1qwJUv+a0HstSf 5yeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867902; x=1739472702; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HYCnhKfOpNHoXGLJGoUQUcx2tHg42NkQx3e7q6fph54=; b=jCAyvdUYt2M67hM9jXOQQpSC9EPZ+TwzvoFddppau0V0gyIG9NHXkBf/Q4M1jSIE0k rl9nhok845VoxpsSR/GTxT/WxxHlyiXw3vYc1R7FuGVDMTVk5F0E//9EtRYVJcl5+d3M vg8KMUTL3EgT0wXkj5Hm9oo7BCJ72YpPYhlOz8DpX7xWuG9Irzceyebl/toqiK87bwAC G+Jwr3X0lP6i917dnZxFF6EZNyR99hnxjoNxKRstIE0oA6mqYkleeMzaXrJY62ROoTCF s1E/y2DRvfULdGd3bS3TnQdPXNjXX/NBu53czuLfaieEK0UFn03tcjzIqyOObTKRXdHJ YRsg== X-Forwarded-Encrypted: i=1; AJvYcCVUPHRfOD+19XhkWi/dq9geJqZJmEdb0qdCibRBsMNIqjNqC4TtfaV0hGCLDeCOE9TIR8dFCprzCg==@kvack.org X-Gm-Message-State: AOJu0YzTNFyqzMsiuf+wvasZLf8H64cSycHkB054PpthhygdIg7pKcVp /FC7cgSSR35OUuhR+aTGZkgTLfzkhJpbB/TEvQbQ0O+xDgSkcmZrvRFu4Ssswb+vRUvQLA== X-Google-Smtp-Source: AGHT+IGOwjdyUKmu33OfICAjkR5uTyQ60fkGxL3DuN6UZt1dw+rarr7Qj2spOWMme9SCtmLchvIHmNez X-Received: from pgbfq6.prod.google.com ([2002:a05:6a02:2986:b0:ad0:f8ff:b90d]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:d70a:b0:1e0:dcc5:164d with SMTP id adf61e73a8af0-1ee03a21ca9mr898719637.8.1738867902197; Thu, 06 Feb 2025 10:51:42 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:56 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-17-fvdl@google.com> Subject: [PATCH v3 16/28] mm/hugetlb: deal with multiple calls to hugetlb_bootmem_alloc From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 5200C18000E X-Stat-Signature: 7zeho4ei6z9r3xtf35g6ehecea3g1res X-HE-Tag: 1738867903-561331 X-HE-Meta: U2FsdGVkX19H25SYKxU9OoTSYU8nLfVgt2juhSgEs6/CgeLJ29GAphquJLl5xuo9NpEv9TvD0u2nPMWSVOB2vo3gMKIvVyKUU64eD3jlXRauWv5fpjRXwxaFLm5HB2wb+X2n67juvMkL+z4dxVbqc8lNc2qDkZYvseMtuMLC4PF1zgAflo9g20RwMCFhn5/0d0uGYOp0PlXqruZqnW51zqn2t832aIVvOZFljQpGXR0kgCto3dISCip3+cwcZzDaMDDCNa5wyjzys2tOxuscZc0bA/WYBSujHuNGDS+DX+xms0sUHrjq4gvHEjArq838JrNC29y35tJHY54QibG9bV2JB3O9mGQsxetkV+imt9F2QSv5vYxA0j38/tWgYUQpoKPiJ/b1ypn38lcyYOfDBf5ZihUHstTDxSUGwEeegJpQrMRKmF6hPafZJfEE46byEBdNOp3s/V4WSFpwC087/rQk9IP8Ei3/MYjBg9kZb+ADNrjTLaiZ3k2sylVHtsPR0uVXyc+lePYxCRHF0AQUN7KbPF3ejPPzCyC3zPlvs0/NOfIXzvKC5BtXSK00gqWp/jbQIenXmNKtGIxv4ZvHevDv7eiZHcHToSOexnkRb4T5rI04Rz3Uw3h09rgMks41HPWorzBVafrA/EOoBcy1QzINNzI7uG7BSLcPPJcQxdyZuBVzTcTGEv4V3gCOnuPfZTRhLH+nMYGqqSQaTXD43HdUUpgeHcJc/to5jeHF4aFEihPrNTqIGp3SxhgbNYdr1uWhCmJ580HynsKo5wQIWcKChdz41INXoRf97iXX8TgHWBjqBgw2ufWGoWYtjU+eeWlgATwsCGRlY8+XjjaWeW+efR6R+oBkPCuU12yY8t+cUEnJS0nBnlyfUMozOdpHyK5HJpUKzODoQhu9ngaUYdH49Kp04LU8/fz4GJwZV49pZX782sC7g93DmhJK+wV5X9ICyQ5U6+8NNffGnIK oSw5R1Aw KTqx8UasNzSNNHudx05B546lE86VzxRhzSfDNcZ5yyvrHvyEB5ZhESYkPXP+DldK21Tk2yO++3ujZtoEpf4QaTtc/OXUOujQ0tIhGNfa3WUfuDC0lNrzXiEysmQI23D/ljWkW6e1h9AxTvlGGmLqDhguhhv8ZtT8S8VqmZmKHEuDyFMsQzxhpyrwojHGpj0D2Tyc4pJKbvf7l4a6YArgvzv2tqkiw2NbCziSjgTv3hYclZZp3TM8VbAFP3Ym4vQ+U72IsfhmGRfEK7FfGbv1WxNp+hmiLm4YlAxErgJM/uLvmN3ijUc/NYgrVb0i1HOaDOWqq9E0xkquxv9Q4H3DTqOl17QDyEurnLq8TMxrQLctxPcg4PuxZIXNGhWxQJdvA0oCLwW+4xqrym2uMwUOAKPrKy3yyyH94yO1EQWfhkxIVwhkmkFrftXH/TcBQsgiE9xFeEdALnp8/ljnbxHq4E7xzyWoT/o/JJ8uF6Dm8Rw3Jpf8W1vBWyzZCTmOnE9oXYQGhdV514qvQk6Xcwe0FkpEt2fgxs/owblpIPwcQddEiK2UYT6fSngWrcy81YOU9FUxo+18FK7goU2c= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Architectures that want pre-HVO of hugetlb vmemmap pages will need to call hugetlb_bootmem_alloc from an earlier spot in boot (before sparse_init). To facilitate some architectures doing this, protect hugetlb_bootmem_alloc against multiple calls. Also provide a helper function to check if it's been called, so that the early HVO code, to be added later, can see if there is anything to do. Signed-off-by: Frank van der Linden --- include/linux/hugetlb.h | 6 ++++++ mm/hugetlb.c | 12 ++++++++++++ 2 files changed, 18 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 9cd7c9dacb88..5061279e5f73 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -175,6 +175,7 @@ extern int sysctl_hugetlb_shm_group; extern struct list_head huge_boot_pages[MAX_NUMNODES]; void hugetlb_bootmem_alloc(void); +bool hugetlb_bootmem_allocated(void); /* arch callbacks */ @@ -1256,6 +1257,11 @@ static inline bool hugetlbfs_pagecache_present( static inline void hugetlb_bootmem_alloc(void) { } + +static inline bool hugetlb_bootmem_allocated(void) +{ + return false; +} #endif /* CONFIG_HUGETLB_PAGE */ static inline spinlock_t *huge_pte_lock(struct hstate *h, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 789341158ef6..3db980db15ec 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4910,16 +4910,28 @@ static int __init default_hugepagesz_setup(char *s) } hugetlb_early_param("default_hugepagesz", default_hugepagesz_setup); +static bool __hugetlb_bootmem_allocated __initdata; + +bool __init hugetlb_bootmem_allocated(void) +{ + return __hugetlb_bootmem_allocated; +} + void __init hugetlb_bootmem_alloc(void) { struct hstate *h; + if (__hugetlb_bootmem_allocated) + return; + hugetlb_parse_params(); for_each_hstate(h) { if (hstate_is_gigantic(h)) hugetlb_hstate_alloc_pages(h); } + + __hugetlb_bootmem_allocated = true; } static unsigned int allowed_mems_nr(struct hstate *h) From patchwork Thu Feb 6 18:50:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963564 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F26D3C02194 for ; Thu, 6 Feb 2025 18:52:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C1D0280010; Thu, 6 Feb 2025 13:51:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34A09280002; Thu, 6 Feb 2025 13:51:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 19FCD280010; Thu, 6 Feb 2025 13:51:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D7CD8280002 for ; Thu, 6 Feb 2025 13:51:46 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9D461141408 for ; Thu, 6 Feb 2025 18:51:46 +0000 (UTC) X-FDA: 83090413812.30.42C3861 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf02.hostedemail.com (Postfix) with ESMTP id D547D8000E for ; Thu, 6 Feb 2025 18:51:44 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2OufGpIF; spf=pass (imf02.hostedemail.com: domain of 3vwSlZwQKCKAFVDLGOOGLE.COMLINUX-MMKVACK.ORG@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3vwSlZwQKCKAFVDLGOOGLE.COMLINUX-MMKVACK.ORG@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867904; a=rsa-sha256; cv=none; b=dCHAFNopk0XSvqoU7ieN8/9HhQE+kbCP0cLhr/FDybHKuY5Ogee2cdfMZ9yw8RkPO2x49H tO9IU6pb0IEnBr/0kMKY5xt2/MfYPAwEAmPB/Wr8J/BB0Ss1E9QqqxU2uzjhR6sxzYh9Lv Q8Ox+NmXdaue7ayD49u09Dar/+qJ0yg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2OufGpIF; spf=pass (imf02.hostedemail.com: domain of 3vwSlZwQKCKAFVDLGOOGLE.COMLINUX-MMKVACK.ORG@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3vwSlZwQKCKAFVDLGOOGLE.COMLINUX-MMKVACK.ORG@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867904; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mjgqfpoeBSlwN57iYwWF6PWmjsHMNrUorj8RHJy/6t4=; b=a5KrdM06rVpMl3u3HRKhL1P6Qvb//0Y+2hM9t027PnSrFqDV+R5QeM0AFULDOBThd9Lg+h f1Ir6mXPyHswV94ODHinWn5OVv69KMfW53Ksi7bIuQssrFviuLF0Abyd2bSMVxUr9c1sjN i6G4bhXq25ZeUtgYUegengp8UBgpd2Y= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f9c02f54f2so2486697a91.3 for ; Thu, 06 Feb 2025 10:51:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867904; x=1739472704; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mjgqfpoeBSlwN57iYwWF6PWmjsHMNrUorj8RHJy/6t4=; b=2OufGpIFiA2yJQ9EJtYhnldFpiZAoIkRGv3t1BLTMQJu3+wo+/FC+Ih5lGeqvhNf4D 5DzSd8KKffFYqq4A250TEsN7SOFOfzvad/1Cp1m+72/l2LjbHMpDVC0exgenuYnK8NOT jlSU8uG7HVvjGwwUc++qZCLyUwguB4mJawZxcj5tO0fgMm0UB3MMqu+NgcHJk2YFhETh JlBsWKHcmKoiJJGRgZpqdw+3sIKRS6XE/fQIhkJJV8Z8LRv1ZgoYameHU91ZTWWkp/gI TyAVUxV5Mwiha/9QpcHrwUrV87gRt9vIb2tAjOfzUOAW9OvZeoG8y2B1SoB/q16g0el6 fiMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867904; x=1739472704; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mjgqfpoeBSlwN57iYwWF6PWmjsHMNrUorj8RHJy/6t4=; b=Gm2VD62FwX+7UUJqgTBa0gtyG/69D/scom5L+2Dn4yG7s3xbILiXUMJLjXO3qxqHHy 4+9Fr6RZCfZ0jYaT3pXTj4xEJNWAgFGsKgYcTkcNm/ABtRyESFmQjQbonKGFOBDVhJXc fytJOftwALK6HfIBSAxwVvuBNlLRYVvW5jKMcNtATGofFY7EGLwOPNCFfmPESvqhpD+L lRCHgHpotBSOMdJBH9O07S41B7uuyG/cz9yEjTalyY8GfvqiTskn/5h2am3Kk76Vgh4a LgGjqspDQMCqUQ8TXO9H/4ht3ys0bzOlTWJi71mW9+44MAa936U1mrgbzPLsy0JYlruI CsEA== X-Forwarded-Encrypted: i=1; AJvYcCWRS/dRQd52XtVT21dyoqOtZ0gT+zUIMZH2xJ9U9wBlMMAaKAeowzshsOiIqSAwpxYTb5wQlf8a5Q==@kvack.org X-Gm-Message-State: AOJu0YxxCBBuYDZHLKGW0MMXPoSM6OiiBXjwh9/ZH0TLBJN9twyLL9XJ bWafHBCv1gWppqe4nkzI/EfZRf1yNbwQ7cPJzTNHAuNr0/zNwmIllJr6XAKEVI/6lQggJQ== X-Google-Smtp-Source: AGHT+IGsb48koloxCQu/0NbdMR5X4NhMSEmqGncLGIVHRlgshaXB4vQQkIYCmgOftKiKB7i67AnhOHrr X-Received: from pjboe12.prod.google.com ([2002:a17:90b:394c:b0:2ef:9239:aab1]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3d03:b0:2f6:d266:f45e with SMTP id 98e67ed59e1d1-2fa23f42672mr204592a91.2.1738867903837; Thu, 06 Feb 2025 10:51:43 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:57 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-18-fvdl@google.com> Subject: [PATCH v3 17/28] mm/hugetlb: move huge_boot_pages list init to hugetlb_bootmem_alloc From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Stat-Signature: k74459wk1ng5ow314p9p63hg7akwdhoz X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D547D8000E X-Rspam-User: X-HE-Tag: 1738867904-722520 X-HE-Meta: U2FsdGVkX18isg+Q9SeXxE3cAXP9YHX72b7yVCFxJrQ4ycxg2tWPBKtsBbk4Eppf6YBt/xCJ47LR9YnyIgOYpaebTEIdKNwe9LGHUSPzVjS0jPQ1o8T+GzOEB3A4e77OHsaiaBrzmSib3kOZh/tmse+QQVdxCMGXPMJ/BB10zzmLip0IBp9U2jskgBCCUr9npF/SxhZ+6BzvuwlnTVy0JkAavbRfU8/uwC1aRxJk6B3i39F1gjPWv9ZXb/xn231LSgB1IVWNzP/e3v5KOWIrsQ/+cAUXBs1h0IDTJTtybmdjIF72CGYRUf+BHE/swwtU30BWqc8whMb0tEONEhFhFuAWa6NWSOo2yk5SorI6GoJNZqx9jH3wfE/MLmXMKOmRvVA4azlC9aa7uNo3/ASTYS2Xv9zyxPVvBBP6+XbUxhh4/Htd+4rDKvTn4IiqJqJhrwOiUIei7j7j1UC18o3QNu49PuamnNdzaKHtH57ZSDVEUDcSPFKJzpBjx4AFMHuXRTIdDD8aF/MymhYhChqhTiD9/CTX7X3uJwprP9QruOlbgQQZ5F4BFzYln6QmzxU95uH2GgDYPJ6hNIWRjYy+L61ScqVVMrrWZ/XiVCpDxLqFjkIomQUnmkU7CjorjG1yv4G42DJsXcdhvbgbiNyQKtTjba2ccEF3FmoeCEmbN3+Mfse+Zt0OR62cb+Vl+tuXvhPjZvhsIXFz4x5j81xtwqo6FMRcgqcyM2SrcRFNaroU7if3W1Ug0+PtEPpIY4pY5KDZjSl2uj2+AkBdGp3QH9tAWp9TcAvDLd7MSNB1lzUyT18qUl7a/q2oltSa3UZ+2IOmyVHLuWU773NHVZn+bw3FlMiqHOdgNLMKRxaB0HTdHKgqa5RTvEXw829/Y1qapzkCHM5j4SGZmTYTVu/YGsoH4re2Ton8eLIWtATn0b8PiAYYFWeScBxuFEFU+OHV6MDYDBi37az+vJYrNvg G1/MTkGY kSs2l/kah8PMl1pLmJrAbNo/btdWD9W4sl083ilwmXqmDsKynIiRNtLrmLnFaggXntP2UhYTlX86NKiu/ZmuZZy/N045zSSek/EHEzGuLianvwk/9VPno/K7xI/SjxNXPzbzbOjGhI0AI112HhhJYl5PSXUaKdS6aHioKGrNfvTfZwOHz5/BbT/Q6lVAiPMevOCQFE3LnW8ad1ZwOSINN5QQNjz0xWEB6l8rhU0l6Ob6CQSnLKn+H0epcWPbdzMSKmhJqdhvXh0BB2Kh6m0zdb5IrtakURWTtp51dP3SvbhSTFT1w7XUE9IykvXD53bAGzveUGI0dvb7PrdIkPpXNor81cFRIfDmNTLb2liDcE24N/To1h6V+KugSSwJwWpdHjk0vI3EmwqZ8e4u0zq5nC39bwGna1DswdjTnN+Pp4QX1vjGMHozhgnhk/36sEkfSYifJQasAMLDw7EtFur8tTaAcADAsPaSaQznC/16kn7JOJVxghNKU+N6air3RTacwR2yclBJdoBjKqp85nJjOK/gKut7e7Sj9dg+gf5HwukXUxhfqfMFf8x1R84rbomu2ku/nT7MS6MvdEDo1HYcE680YPBT1MYkc3gqhlhQxkYSRjo8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Instead of initializing the per-node hugetlb bootmem pages list from the alloc function, we can now do it in a somewhat cleaner way, since there is an explicit hugetlb_bootmem_alloc function. Initialize the lists there. Signed-off-by: Frank van der Linden --- mm/hugetlb.c | 19 +++++++------------ 1 file changed, 7 insertions(+), 12 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3db980db15ec..52ee5514a1f2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3578,7 +3578,6 @@ static unsigned long __init hugetlb_pages_alloc_boot(struct hstate *h) static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long allocated; - static bool initialized __initdata; /* skip gigantic hugepages allocation if hugetlb_cma enabled */ if (hstate_is_gigantic(h) && hugetlb_cma_size) { @@ -3586,17 +3585,6 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) return; } - /* hugetlb_hstate_alloc_pages will be called many times, initialize huge_boot_pages once */ - if (!initialized) { - int i = 0; - - for (i = 0; i < MAX_NUMNODES; i++) - INIT_LIST_HEAD(&huge_boot_pages[i]); - h->next_nid_to_alloc = first_online_node; - h->next_nid_to_free = first_online_node; - initialized = true; - } - /* do node specific alloc */ if (hugetlb_hstate_alloc_pages_specific_nodes(h)) return; @@ -4920,13 +4908,20 @@ bool __init hugetlb_bootmem_allocated(void) void __init hugetlb_bootmem_alloc(void) { struct hstate *h; + int i; if (__hugetlb_bootmem_allocated) return; + for (i = 0; i < MAX_NUMNODES; i++) + INIT_LIST_HEAD(&huge_boot_pages[i]); + hugetlb_parse_params(); for_each_hstate(h) { + h->next_nid_to_alloc = first_online_node; + h->next_nid_to_free = first_online_node; + if (hstate_is_gigantic(h)) hugetlb_hstate_alloc_pages(h); } From patchwork Thu Feb 6 18:50:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42D06C0219B for ; Thu, 6 Feb 2025 18:52:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 350E8280011; Thu, 6 Feb 2025 13:51:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2D5D4280002; Thu, 6 Feb 2025 13:51:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08DE8280011; Thu, 6 Feb 2025 13:51:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D9E48280002 for ; Thu, 6 Feb 2025 13:51:48 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 99DF2B2650 for ; Thu, 6 Feb 2025 18:51:48 +0000 (UTC) X-FDA: 83090413896.18.9054576 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf06.hostedemail.com (Postfix) with ESMTP id 6D2DD180007 for ; Thu, 6 Feb 2025 18:51:46 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gm8nmc3A; spf=pass (imf06.hostedemail.com: domain of 3wQSlZwQKCKIHXFNIQQING.EQONKPWZ-OOMXCEM.QTI@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3wQSlZwQKCKIHXFNIQQING.EQONKPWZ-OOMXCEM.QTI@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867906; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1RY4Sckn7GWnsNuNzDlgUqe5LNqypC0myFrtZsSR9cE=; b=vFRJSwcRRiqkg+6mRAW4KfUcguwKCFTzqSiWV+lxxGekebeW34W691/peekhXELRWaSSA9 cwB8qAHG61EQet3kdYZD7ecvDXDRB7t79ul7v2mXp1zai7vlAbLyBc3IGXQUazmKUO1HtO K+/hUOY2+NbBpeauWgEx41pz+j8xIHQ= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gm8nmc3A; spf=pass (imf06.hostedemail.com: domain of 3wQSlZwQKCKIHXFNIQQING.EQONKPWZ-OOMXCEM.QTI@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3wQSlZwQKCKIHXFNIQQING.EQONKPWZ-OOMXCEM.QTI@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867906; a=rsa-sha256; cv=none; b=6d3793WLaBgmEsPsfIkTpzeT9gsz2ccjPmyhj/BoMVTMXwGN83ZqrnLYQPJiDe8KxFI4iL xWSyKHW8iVa4bIfbFO2c6ypTuSvmvOb+uh6A1hwNU7iIWo2HaF3nu3nnaOATwNoijlYJpf dAVKA1Lqf2yixvoufLQG8R6KjvaUoYI= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2fa166cf693so1877754a91.1 for ; Thu, 06 Feb 2025 10:51:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867905; x=1739472705; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1RY4Sckn7GWnsNuNzDlgUqe5LNqypC0myFrtZsSR9cE=; b=gm8nmc3AizK+7xoNt7vWXTgASiOf2OEPxjch8g4OqfKT3lhedYGXJ+IUFaep8lpsih FMB3h5bIT+ljj/k5f33obNomnBXCY3c3POQWXahB3NJKzeplHLhE0IDSGVdhbQQXxKO7 FYE5spBaWRYodZnArnZOgeL0uYLnjMV1ljT3arqJiZo3yLFqAeXC5pcNCxaFanbrlK/8 yhioGTdT1g5Jj/PK5aVwL8sF+GZ4N12uc+gKS3/FHZKHGyJr8Qb2g7ODIrGsOKlLSCw4 r6RQBMsT8dBv8LpXlTnczzXRTsLo17Puc+QLU5b7P3MTSApOLorSU2cGccNVCUQ4FFHf 2D9A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867905; x=1739472705; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1RY4Sckn7GWnsNuNzDlgUqe5LNqypC0myFrtZsSR9cE=; b=WkUXtZccjSDxmYz/ISlKyUTQdX5pkYa+Urmy42vIRjD1y125Aa53wDdjDTGy8tWV2L 90nzBJJcj30M0pDW2JzarFNETx3wRdvvVihm+og8Vn17GCCp0DyUBbLgPRI3O1y1b79U Nyf8RLeltjYLcVeG/Q2tq1o9xKo6dRNwGWSNH3vZ3u1cx1kQPO2XcU2aY+jHD+pH4BfX ZiKiI7g9OSKhiKX5XHIVpzQctwadQ0eLT9NoTGj4vdivErArBqgd7VurKJzHJKWkiuix k3B7XgZK17RULRICRtK9VRlB/WGl28m+KJy9qp01C6Jd/TRBxSWQCnH/ZZb52FVO3zgd arTw== X-Forwarded-Encrypted: i=1; AJvYcCWCSpydsqfIHKQdFf8LQQFC/eQ38esfM+uX7UrC7nBkTCXLSAXfV85cXKLbiea2h1ZSHT8jbe7rRw==@kvack.org X-Gm-Message-State: AOJu0YxHt1iMz6y0v57zTg+EhrLrN22dovTzJ5OEQLW6OJAR5xlk7QnG jGTReYHtqlooqAbgFpbxkcYTGBCR7Hrpsi4J/yHFYVbsFzXhdAZrDsgPZxR/ccDFX25piQ== X-Google-Smtp-Source: AGHT+IGv5wn5p5nXx7RKxFjYk6wjYFPCzyPzBbZhXtDcOU21vQXSokgZmbD3Gr14JvKdrzDwy2bARR7t X-Received: from pgev18.prod.google.com ([2002:a63:ac12:0:b0:abb:a12f:81a8]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a21:9985:b0:1eb:7da4:305e with SMTP id adf61e73a8af0-1ee03b70cbcmr733791637.39.1738867905422; Thu, 06 Feb 2025 10:51:45 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:58 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-19-fvdl@google.com> Subject: [PATCH v3 18/28] mm/hugetlb: add pre-HVO framework From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 6D2DD180007 X-Stat-Signature: n5x9989pkq7asdymgofn1c33odkh7x6u X-Rspam-User: X-HE-Tag: 1738867906-628166 X-HE-Meta: U2FsdGVkX1+qun+6DUWBYM0StrURBeWF0ypp0531kqaQRuC6JgjK7AwNdSmVkIZSvwHJo7zEXg/o6uevmzuOIXrOE7jYVzLubxiKa2IS4XfdD1D5dA0inVpMxRakm9TmT35QULVekpohchnx54Tez+8dihNe4FRYNfa6tkygpqTQmSh4UOe+RZPb8+GOSlExLJvF3+TaWH36HjspPaUlNMZWj7eQUkLFpoNnogy6mwFpAoUQ5asE7dgHICbvjWfHaL9pFTZvc+YztvVJe9O93oODA/mPy09cWP3xWQWkKkHsHYG0wN6mIJeUM048wtLNTerVKflWygFK4skKvqnenoG+Mziy4utS2SsmN7aNJOfwFrZDckds8oTDzZ4pCp7wBVuTzN6rk1hD8Wp3KODXMU5xeLdGap5MYC3Dr5sm9G6y/xY0gIuxQABDnEAI+iPHvyOcc8x6jpNhVTOUFoE93lENjVoTaE3ZgVr+rRMft6YQiOTOs55sVOY2rmpQUn2RBVf/meEf3jAR8pdyAeNWVwFPZkAaBxvHw8f6AV8TfxM8LmsXwXHM2bSWOE77Xcuv/eXjm2DJ941bl0oQUDfn1q2/4bm5SGsqI2IHs11YzT+z3v8bV2rClezBZbfSzM4WdQ03kCvLHAP0suNW2H5ayqV4WG1Xgthqx94xBzWFObq6I1GteSLqTQyBiW41T3lXfyX/RWjOtEue6XTfhVgz8wBCKpijBx7yE7T82DbObTsO2L78bwKYf0yLY9wB1wCd6k8p+OhQkMTzGWAnOfjrB4nxjZGpttrh7s4951sl921UIx5s82U89GkE06BgdsqO5LPF0e8uBecDOiJFaw98bdtjIezaxGQUIO5GpAdwZP3aUfj8AA9CDOX+XCa8GMSM4rzc+CnEM4C5NC+mtbB5hy6mcUV/7Pz/lgk+n0/01KzXP/Hi3yBpMDXF8xzTtdrUaMd90VY9q0HgYhlUzHO 0o4zP03A mWue5EiKIs84ytNW6PECB3KJgo5Z9+J0W8DncR6GkhCoIunlDj9gbB25fo0q7Lcz5VGv3xsF/moIg7iDVBfRV14SX8+/tKzuvDFKSaMkCkPm+tptfjWqVdri5kse02dcri3szNWVH1a9KbYgbZn7f9GAPgpp9YjDwh8/RrgqGiDM8ODAnHfeTROJ8wTPgB9u4QA7KqpIKxrAqrm2B2XyShOb2DfpD+Y3U6egWAh4hGG3ZGzcEuRef4OEsrXcUCTvAzSe3aynFztoNkPkLddoRFEOElIJ4uOODHSKShKMfF2KPs1ExOZh1vzGMaOfn5k3aXHX7nbC4prdidGhXMpDF+SyYNJvbAp33imIcPQwNlThiSp+97A88miCIS2L4y4lAynNwob/GAl8+2Hb5O6jpE746aqMy1d2aNlWQv+bGQiPvgq4PKcmRUH6iOKFWzjbGdo6DS/T6YE9CMeEYKLOSunPQ6+9qMUxjrVPEZHoR1gbxCvRDCTgFC1PRMbxPsI2FD8A0kWndd+DE4wlVplVffCKfXg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Define flags for pre-HVOed bootmem hugetlb pages, and act on them. The most important flag is the HVO flag, signalling that a bootmem allocated gigantic page has already been HVO-ed. If this flag is seen by the hugetlb bootmem gather code, the page is marked as HVO optimized. The HVO code will then not try to optimize it again. Instead, it will just map the tail page mirror pages read-only, completing the HVO steps. No functional change, as nothing sets the flags yet. Signed-off-by: Frank van der Linden --- arch/powerpc/mm/hugetlbpage.c | 1 + include/linux/hugetlb.h | 4 +++ mm/hugetlb.c | 24 ++++++++++++++++- mm/hugetlb_vmemmap.c | 50 +++++++++++++++++++++++++++++++++-- mm/hugetlb_vmemmap.h | 15 +++++++++++ 5 files changed, 91 insertions(+), 3 deletions(-) diff --git a/arch/powerpc/mm/hugetlbpage.c b/arch/powerpc/mm/hugetlbpage.c index 6b043180220a..d3c1b749dcfc 100644 --- a/arch/powerpc/mm/hugetlbpage.c +++ b/arch/powerpc/mm/hugetlbpage.c @@ -113,6 +113,7 @@ static int __init pseries_alloc_bootmem_huge_page(struct hstate *hstate) gpage_freearray[nr_gpages] = 0; list_add(&m->list, &huge_boot_pages[0]); m->hstate = hstate; + m->flags = 0; return 1; } diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 5061279e5f73..10a7ce2b95e1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -681,8 +681,12 @@ struct hstate { struct huge_bootmem_page { struct list_head list; struct hstate *hstate; + unsigned long flags; }; +#define HUGE_BOOTMEM_HVO 0x0001 +#define HUGE_BOOTMEM_ZONES_VALID 0x0002 + int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn); struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 52ee5514a1f2..7e0810c217ce 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3219,6 +3219,7 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) INIT_LIST_HEAD(&m->list); list_add(&m->list, &huge_boot_pages[node]); m->hstate = h; + m->flags = 0; return 1; } @@ -3286,7 +3287,7 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, struct folio *folio, *tmp_f; /* Send list for bulk vmemmap optimization processing */ - hugetlb_vmemmap_optimize_folios(h, folio_list); + hugetlb_vmemmap_optimize_bootmem_folios(h, folio_list); list_for_each_entry_safe(folio, tmp_f, folio_list, lru) { if (!folio_test_hugetlb_vmemmap_optimized(folio)) { @@ -3315,6 +3316,13 @@ static bool __init hugetlb_bootmem_page_zones_valid(int nid, unsigned long start_pfn; bool valid; + if (m->flags & HUGE_BOOTMEM_ZONES_VALID) { + /* + * Already validated, skip check. + */ + return true; + } + start_pfn = virt_to_phys(m) >> PAGE_SHIFT; valid = !pfn_range_intersects_zones(nid, start_pfn, @@ -3347,6 +3355,11 @@ static void __init hugetlb_bootmem_free_invalid_page(int nid, struct page *page, } } +static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) +{ + return (m->flags & HUGE_BOOTMEM_HVO); +} + /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. @@ -3387,6 +3400,15 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) hugetlb_folio_init_vmemmap(folio, h, HUGETLB_VMEMMAP_RESERVE_PAGES); init_new_hugetlb_folio(h, folio); + + if (hugetlb_bootmem_page_prehvo(m)) + /* + * If pre-HVO was done, just set the + * flag, the HVO code will then skip + * this folio. + */ + folio_set_hugetlb_vmemmap_optimized(folio); + list_add(&folio->lru, &folio_list); /* diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index 5b484758f813..be6b33ecbc8e 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -649,14 +649,39 @@ static int hugetlb_vmemmap_split_folio(const struct hstate *h, struct folio *fol return vmemmap_remap_split(vmemmap_start, vmemmap_end, vmemmap_reuse); } -void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +static void __hugetlb_vmemmap_optimize_folios(struct hstate *h, + struct list_head *folio_list, + bool boot) { struct folio *folio; + int nr_to_optimize; LIST_HEAD(vmemmap_pages); unsigned long flags = VMEMMAP_REMAP_NO_TLB_FLUSH | VMEMMAP_SYNCHRONIZE_RCU; + nr_to_optimize = 0; list_for_each_entry(folio, folio_list, lru) { - int ret = hugetlb_vmemmap_split_folio(h, folio); + int ret; + unsigned long spfn, epfn; + + if (boot && folio_test_hugetlb_vmemmap_optimized(folio)) { + /* + * Already optimized by pre-HVO, just map the + * mirrored tail page structs RO. + */ + spfn = (unsigned long)&folio->page; + epfn = spfn + pages_per_huge_page(h); + vmemmap_wrprotect_hvo(spfn, epfn, folio_nid(folio), + HUGETLB_VMEMMAP_RESERVE_SIZE); + register_page_bootmem_memmap(pfn_to_section_nr(spfn), + &folio->page, + HUGETLB_VMEMMAP_RESERVE_SIZE); + static_branch_inc(&hugetlb_optimize_vmemmap_key); + continue; + } + + nr_to_optimize++; + + ret = hugetlb_vmemmap_split_folio(h, folio); /* * Spliting the PMD requires allocating a page, thus lets fail @@ -668,6 +693,16 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l break; } + if (!nr_to_optimize) + /* + * All pre-HVO folios, nothing left to do. It's ok if + * there is a mix of pre-HVO and not yet HVO-ed folios + * here, as __hugetlb_vmemmap_optimize_folio() will + * skip any folios that already have the optimized flag + * set, see vmemmap_should_optimize_folio(). + */ + goto out; + flush_tlb_all(); list_for_each_entry(folio, folio_list, lru) { @@ -693,10 +728,21 @@ void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_l } } +out: flush_tlb_all(); free_vmemmap_page_list(&vmemmap_pages); } +void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list) +{ + __hugetlb_vmemmap_optimize_folios(h, folio_list, false); +} + +void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head *folio_list) +{ + __hugetlb_vmemmap_optimize_folios(h, folio_list, true); +} + static const struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 2fcae92d3359..a6354a27e63f 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -24,6 +24,8 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *non_hvo_folios); void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); +void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head *folio_list); + static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) { @@ -64,6 +66,19 @@ static inline void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list { } +static inline void hugetlb_vmemmap_init_early(int nid) +{ +} + +static inline void hugetlb_vmemmap_init_late(int nid) +{ +} + +static inline void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, + struct list_head *folio_list) +{ +} + static inline unsigned int hugetlb_vmemmap_optimizable_size(const struct hstate *h) { return 0; From patchwork Thu Feb 6 18:50:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3B6FC0219C for ; Thu, 6 Feb 2025 18:52:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 438F8280012; Thu, 6 Feb 2025 13:51:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C190280002; Thu, 6 Feb 2025 13:51:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 214B9280012; Thu, 6 Feb 2025 13:51:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E5AE1280002 for ; Thu, 6 Feb 2025 13:51:49 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B26D2B26C2 for ; Thu, 6 Feb 2025 18:51:49 +0000 (UTC) X-FDA: 83090413938.12.5C0C19C Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf22.hostedemail.com (Postfix) with ESMTP id E92C4C0011 for ; Thu, 6 Feb 2025 18:51:47 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sh+N8s4W; spf=pass (imf22.hostedemail.com: domain of 3wgSlZwQKCKMIYGOJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3wgSlZwQKCKMIYGOJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867908; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PNNc4e9k2qfZPyKSSlBoWZ8VBSivbX9t64nIb//yCrE=; b=jL3KhRIDftYMWDq7Y/ULcnGz8mYLHiJkjyu+AwfLan3ARN0vyOvReQiLrMWMdkKklnTkbV NrmkrStRKnY3zRKqQ2+MNklGsR9LDMXJ4X7uNrnUKVB9A8unYbiMf7QeKblcSZgNQQHILI UUncISG3ElPqVl9lNRAZLVUUBan+PHo= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sh+N8s4W; spf=pass (imf22.hostedemail.com: domain of 3wgSlZwQKCKMIYGOJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3wgSlZwQKCKMIYGOJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867908; a=rsa-sha256; cv=none; b=JF3ogWx0CzpkOheDlA7lisXjmI3udAzey1v8GVlB+qMlQ9eH0dzfJc4RfCI/HAY6o5fDBA QzOTkET39dkpK7EGpHJS+wwerxq/muazdtM+9zabVqNb954+SRhE6nmyZIxNNOAfDw8pGE TWhUDDsdXaUDLolw70pe0tl3Lrp9KIU= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f9bf2d1c7eso2502732a91.1 for ; Thu, 06 Feb 2025 10:51:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867907; x=1739472707; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=PNNc4e9k2qfZPyKSSlBoWZ8VBSivbX9t64nIb//yCrE=; b=sh+N8s4WJGWn9BVdTdrZysfxYjcL5FSQEIZY/1+PWm5oqaSTDFpiFBlP+cu+5cwymy mIcLEiwnl6OzhhUlZktKTTAK0LWJ6pUYyGVQkrx+JC/dJpgRObYB4zakYvvhSMxletvX x/s+NF3JxE0XTLcDznCaJHre9LDJYvXbJEAsT81Sdu9PSlNqEYxIfOzlpPcPDSzxBt5G 9zZAmYKzJxik2NBdFFQUcBv/ke73imD8Pg/pp4wm5OPEcBOd421qHGcattjQMFKnaaUC WgfgmTr85iyu0MYoU3YH2zCDnGVscxP+ShXBtXewDVKmtkBW4qfQ24Ht0ic0nO56qDx0 7rGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867907; x=1739472707; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PNNc4e9k2qfZPyKSSlBoWZ8VBSivbX9t64nIb//yCrE=; b=gWEHp3ClAVsdRKNVt6/8eJyZhsu3Y49/jmuxKQg8ycuz4lV0hwsCnjA2Qh5xZy3R15 3MW4cZq2at56VQsMWUKNcRCfqjfKYhG5/128eZhzQZV35xCo4NJ3AfRqdnYkT8F3CrP8 3c6oNPJlgHXHap+Q2zJELqYSY2qBR34l9ezxI9cL1tMWyiJxwc9z7wi6wU5J80BROPrb H1dIm6ALjYx5ZSP0qe6wLDreWcGpKFIKFqmNXFpqRHA7vLgXOQs5tfo3S0jeGpEPXwH0 8F8N/LHqgI4RVuQK8VqEWVSQydzY3fN4Jji2xMKEgBRutWucsQgkCUUWp20O2QDYOSs6 AdwA== X-Forwarded-Encrypted: i=1; AJvYcCUtXoqNp5G02baSKsA7YVR1AIInLrV/d13NuO90j7CV10gevinDft1nK0DcoTcc+waVjDOEYNEtNQ==@kvack.org X-Gm-Message-State: AOJu0YzpBINmsLjTt3uGKa03hQT+NmILLD8Xz7JI8DmPoy1xINA+SS8/ FgFmiJh7NAPzVywXIgZ3cqfy16WSNcixgYqZS/zrK4SJZGg05JqwKvpWFfn30o6QZwaBaQ== X-Google-Smtp-Source: AGHT+IHGGdqW8NvChqfMo/KsvB7iHoprf/cZ/BIgo0+z0Q3eFlECPhwcrH/qX3NKeNb05ShwfwU55FNn X-Received: from pfbdh8.prod.google.com ([2002:a05:6a00:4788:b0:730:4c72:3583]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:aa7:88cb:0:b0:72f:d7ce:500f with SMTP id d2e1a72fcca58-7305d525eb4mr475558b3a.21.1738867906868; Thu, 06 Feb 2025 10:51:46 -0800 (PST) Date: Thu, 6 Feb 2025 18:50:59 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-20-fvdl@google.com> Subject: [PATCH v3 19/28] mm/hugetlb_vmemmap: fix hugetlb_vmemmap_restore_folios definition From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: E92C4C0011 X-Stat-Signature: akt8f1hr37uhixkd8q47s8y4htbkeamz X-HE-Tag: 1738867907-934669 X-HE-Meta: U2FsdGVkX199lSvsWkL8UuceERQXpMnpMHdliG1ZpVjpJORCz2yotrolD7uTE4NH87oA4mUho86nMzLDj4grwn8NJ16ui8xHvJC7X+6Vr//Gl74Sx0FfHjfv0+7ga6gWTlVP4vkAmyaCY4zqmQvkZXCQr7GbHShqA4//DdKo3nqOBP47ZLEm5ujT/Yml2hppxQ7MQFrK4CEELw0rxWrPzbdVNqaYCRNXCcbZVh3GkOrfQg1bYFdV4jxCVEVYbRhKHFXyymVhOpF1b+k9mxgRpiFHa9HoaeA1yrBI5ZfsWuIlf7lFtFywPywB/j9rRTMxgf7zsSJJMUiv7eSDihG9o0zx0AHR0VShJ6GRNzzk9xTWb9Fe/Zw+k5ED0eIMzFasZyyYnmvzIdTowsttIPz+3EBQiMmK6ukgFc2twdJb0Rs6FkbwAXU12pq9RXGUhe/4rEtl88x9SLi+nvdkzHGOME6L/4ugnqsPbMa6iqDA14nnsnY30EHSXTg2G0FE5hkFCmmMUfTx2Nhq4g0BxTbl5ZNQF5cHl4LLRESsZbmhdaHjmUohUU5oWq7Eu13THxVVczAPZy8AiQbAUqh8IYZZ1eJg/47Jfv6pe1DPBUZ4asCBUvxFoIdX1jledSUDfCVL6upyC88LaRShBTvDgQfD2NIDM0YOMKBjaeMnenxoC1sZH7S7yzk20zJWxN1JRv28DvYfDodDIwAYg8CcfvuAbmCzADlPgkTexyFx7Z9r9rU3hrr9bhWo97Fbm+dSby/MA9GqK7pA2SvWysw+CRhkuVS6doycnhOrQMFjC4hX4n9TFcxD/NGeEM4+cAweKOMOJlloIr09R4/wFcRiwjXuQ4ulYGGIrFG50pvfag7W4llzIMVXymf36m6a1HtCC1DhzFXWt4i698Mt0diWua4mJk6ncqiRix1P8tTIZX3JUc63vtIBR/WGELJc9xUyy5GWX5/xl0sljfRnicykvYN NnewUaSa YVxvJvPWFkKUg5WPGjf3d/SaEyyW0Spof3LPgL3hTsw1IK/h+Umm41GLS0ecUZOIAIyvBJfJdpfyXyEVncn2XqAWnCZFpi66DMszDk8+wbtmEyYpExNnCcZRQW5Y9yWrg5YrJEHDQ1Xu5K/mfI6g1IE6TBf9b7gR+Jx7dyx9I2L4ObpD0NoCLzmTWs8Wie5GYC/OUgYkISbbyzVmD/y8Z4YPrPF5G6ITQMy/Ua34Q2QIylDth4o/pETXCzwRqf1xJGPYSlWyUxUCWtdGxmATtmpf30CYAt5i1lRjk1htHzDdxHvmTHtru3B0Tri7g95ArL8p88XxzfJvPEhpgvkn/WMP3CGb7gldYtlGFBtezmavqeCxhyqa9B7CYYTU6iAQz0e8IQpkpD+DJGOAuY4xOqotAed6BQ4Eas3GvXWiliRZxpUM9HhrMk2g9xDu9ZLOIJv3W5qMTczrYlmRA5LQNuNqbO338wyLTbSx+23LVRzQVpDsWbkHYBdGzqWrb1KMzFpuDm8vb3333Xt/kjM/QQRcvblpXgeRMAlcSyv4X6Lnlje+z73iTwA17VQ1FgLafRhROfVqTvP4AFSY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make the hugetlb_vmemmap_restore_folios definition inline for the !CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP case, so that including this file in files other than hugetlb_vmemmap.c will work. Fixes: cfb8c75099db ("hugetlb: perform vmemmap restoration on a list of pages") Signed-off-by: Frank van der Linden --- mm/hugetlb_vmemmap.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index a6354a27e63f..926b8b27b5cb 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -50,7 +50,7 @@ static inline int hugetlb_vmemmap_restore_folio(const struct hstate *h, struct f return 0; } -static long hugetlb_vmemmap_restore_folios(const struct hstate *h, +static inline long hugetlb_vmemmap_restore_folios(const struct hstate *h, struct list_head *folio_list, struct list_head *non_hvo_folios) { From patchwork Thu Feb 6 18:51:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C783C0219B for ; Thu, 6 Feb 2025 18:52:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26495280013; Thu, 6 Feb 2025 13:51:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EC1F280002; Thu, 6 Feb 2025 13:51:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F31CF280013; Thu, 6 Feb 2025 13:51:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D1C9C280002 for ; Thu, 6 Feb 2025 13:51:51 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 949581C917F for ; Thu, 6 Feb 2025 18:51:51 +0000 (UTC) X-FDA: 83090414022.11.BB646CA Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf28.hostedemail.com (Postfix) with ESMTP id C4A36C0007 for ; Thu, 6 Feb 2025 18:51:49 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Xd978mEs; spf=pass (imf28.hostedemail.com: domain of 3xASlZwQKCKUKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3xASlZwQKCKUKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867909; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=41QLNtbtqhWSGZ/oObOknHg0NviKslOOeQUP783wlTM=; b=0ordxWKAMODxxG/YTGUwQPFldYUCzF5ly2sCHGhcvgRlmyXlePZrlz3u2aFEJD/8+RupUw mPLBpJOKHegveq7/dRTjEkGFo42NzRYsFGiNkEmoP6JJZOmzpBKS6JNOFSG5IVXb3opBlI d8a3TkqcFbnxqa0efrv6a5uhpwYCU3Q= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Xd978mEs; spf=pass (imf28.hostedemail.com: domain of 3xASlZwQKCKUKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3xASlZwQKCKUKaIQLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867909; a=rsa-sha256; cv=none; b=ydz+PvIw/oFvN8J6qwlcjDiHneqTyA1Vj1h5R159R72qIJI4XhpGcf7btE8rCn6vMRxhBa +vssvYAhiQIy7qUof31n/ykWqfEyXuwuoOJPVkcYYPctuyY9/OkEXH2dnfsx3iqlkR5yjx KYAEaVDO+dnGnyrLsbaYi1X9Sh2m1Kw= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f9c02f54f2so2486921a91.3 for ; Thu, 06 Feb 2025 10:51:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867909; x=1739472709; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=41QLNtbtqhWSGZ/oObOknHg0NviKslOOeQUP783wlTM=; b=Xd978mEsfuej7zGvJH2cei0SbQmidlk15mYmyN/DkKTijkTlxN1gVF6sg3bm/OQQFf 3q4GUJJaFK1PFgUYMETEe06djKM8JagGZ/Wn6XR/LY1LBkyz1U2HUm5LF0YpLEhQGTRL G3vWwhTYqtmrtUGdw0fGo59ErnEHK0GUNBUyyQkjOr4I+t8zj+z7zAPEnYQ6kVMVll6T BEdG8Q6NaUwCTsDkp0PkkvHfbMRBFSSJjV3uhaq9p8Hisy4/W6ZmJKs6TSwzFlX0zotk Ku/Unt5W+/STQbgurujHLA5ylbVYJXKxiyPV2F0/ii94vrzDDXzXywieFEW4NxKAXI5W bBZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867909; x=1739472709; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=41QLNtbtqhWSGZ/oObOknHg0NviKslOOeQUP783wlTM=; b=s0sBL+V304nXjYped2TlJe7I0P1/Jdt4O5XM3Hc/r4SuV8gls5aX9xbXcZe/ekwfQb wLo7IpgG18d4FacWPa0kmtz7ovzIRTtd+81SHjxO0t4ntJtuaW4klR7gFQgeJJmMCCbI Oci7DyEonkfjcqKqSPptHOxNa1URt+i3QIIBQadSJIFmLIEtxAFz18vm1kRwZqcvxLD4 wqf8DNhAMvIhGvok71rXPQOqGVvDgDmrKyrHNq/CQ/lo7HsS9vTi1WCPfpUpEJAdRoFA d46If4GiFqcEWS9Ixk6lzFNFquWCf7pivb8hPiN+hDT632vKq/dhFr5bIRVdDf1aEu2g XoRA== X-Forwarded-Encrypted: i=1; AJvYcCWC4+TZvzoDkZULpyXfy2152gkJKI9Lvoy9QQkHyLvXDBBckdU0MS2SKsC2sz74Rvtf8tL8Gl6UeA==@kvack.org X-Gm-Message-State: AOJu0YyyvT1epvX3+gRi3FA9aWVe3YOguCAMo4BuF8mbNC4H7BD25OOQ dTLJYCOp1DtRDYjbJSu59D5Jy3A2VZguu3ljSJhT+TE3BTVUZNZGPPCbx4yUbB9d+UsYVg== X-Google-Smtp-Source: AGHT+IGVxYFLO6PMMj5YOBtKJIkBsk+UPteT4nuImFK20RZLySxCL3GaHTYjR4JwUAp1YZGI3vapG7Y+ X-Received: from pfbhd4.prod.google.com ([2002:a05:6a00:6584:b0:72f:8e60:adb8]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2e9c:b0:725:e405:6df7 with SMTP id d2e1a72fcca58-7305d46f7bemr390731b3a.10.1738867908791; Thu, 06 Feb 2025 10:51:48 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:00 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-21-fvdl@google.com> Subject: [PATCH v3 20/28] mm/hugetlb: do pre-HVO for bootmem allocated pages From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Queue-Id: C4A36C0007 X-Stat-Signature: yudxww566cns4jezte1ye4oxtx1w3npj X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1738867909-410919 X-HE-Meta: U2FsdGVkX1+W14e334cup0TtVnGLIcZIwq9CKFqAMFsOML0HMWoGhK0ZxuEa6v5T08TnILIPbuky/CXMmGx4PNnepjfCUOnRZzmyGE+RvslwYk2eh04XOeYkKx8uT3r5W4ZaXsn3IvXhD+42cpuJcqJpeyrDHGlYQiK0QwJaw2pbB8g1cVJJV65rxiEVNcSQB/rmpJN+R6zjvfRFGAtpioWB1o7MpY5MAOT9HKEJ9F5rJn0KiVHtpWNEkjycf72YJzyB9XsmadtIDWtHwi9UGucgLkVW+ttcK1ECWrzSUnvopl9J7ZOvhgExkjXCabzfTKzqWU6KQknwvO+226H/SGhlawpiytXgBNr15M+fjbgyg6hJ+B32+6Mu9zlwsnyIMGQ5uMcq61rVFQwz9uZCjRrozidjdhwxN2q6h+oOpc5eLRznsuCEmPk0pWGCpaqDo0aks/oScBDWDiRv+hBQssjDi2hAAl8PfMdzycVK+8ZAcpmb8tRfhDAyFi3cYSy2oJCuXvLiIdtm4+CNAOi+0cd8HqwlzukXO5GURGcC2FCGRqSPS8beuULCo/YmxM/VkrnBHZK8aci2kznXTsTxU5UykGjdScogZj4UkOjzwX5XppvvtxBT1yCSKvsODEz3lauH7lWIaSBoOZnWklG5Oi/bzPoO6XtXPZKqSl1II51ahE6HGpV9NsmTClVnSrcsaPsK0NZm+exzeNdfBUXv7m973kpW1zwWCM8jvGht13g6itKC4GREtMlfM1O/Y120swWBW3oLohBuucgr6XSw7N/ZmsghT2ikcYWzwSuNR6V9L4UEEmwYeFM8UkpC08SvM8gocdkDn624S1MQMlS+Bkg/S945Hg7XRWLRJT2gphTgJwvcwLFvsQwcu09NSP3FZ27K/CDleRuDCyzaDTIRmsXInkYjLSompjFFiiCFCuip+4wlyRtfxFH0hg+ISChO+Ofqp1CZdqZ5rTDkQ6e f2kjzGWM DxSbL591FuN9rPdvLYz245WPxLAPRE9Y5fj/OHwtUCdHvYSvqR//ziGG/LbN2W5NpjJdUlRXjDzaevlWo+9kNrLSFNLbFFrhCN0JFsOlszrVfRonSuISvx5tZeZp8QhrrXRXqNeHwfsgs+chD9fLxEOy9WbtGI44xKbdpojjZhJQAbBBshpvYkJbv0DB8MDyf17jkH84fp/K8GhwicFmt4z6Jt2KYE8voRlG4JI2bEq+jpeeYuwiyi4EzQyrFYWS6I+eHNGy+JegevteMOYl5DxHcOPUX4LY1yGTOTTXW98xsg9zXr6+S2ZeW4sYbblcms2uDvaD8tOLrn4r1yHx0ZxRY9KsKjgwB3mt659RmsInpSalt+aihskUj7NTxDOURG1Qy/OJk8mUM3ikbFq8JF5jKlxjxYENwblErr/RvjkJsPD8UqDiOqNN8gWn+DNXZ/tbfTjjbNs8voZ5unfOdo54r3M6mAEyxyoWnJjKQLSyBPcrwgClKp3GmIofFK7mc5E+lsC6SZaSUfSNkqQqninN9yg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For large systems, the overhead of vmemmap pages for hugetlb is substantial. It's about 1.5% of memory, which is about 45G for a 3T system. If you want to configure most of that system for hugetlb (e.g. to use as backing memory for VMs), there is a chance of running out of memory on boot, even though you know that the 45G will become available later. To avoid this scenario, and since it's a waste to first allocate and then free that 45G during boot, do pre-HVO for hugetlb bootmem allocated pages ('gigantic' pages). pre-HVO is done by adding functions that are called from sparse_init_nid_early and sparse_init_nid_late. The first is called before memmap allocation, so it takes care of allocating memmap HVO-style. The second verifies that all bootmem pages look good, specifically it checks that they do not intersect with multiple zones. This can only be done from sparse_init_nid_late path, when zones have been initialized. The hugetlb page size must be aligned to the section size, and aligned to the size of memory described by the number of page structures contained in one PMD (since pre-HVO is not prepared to split PMDs). This should be true for most 'gigantic' pages, it is for 1G pages on x86, where both of these alignment requirements are 128M. This will only have an effect if hugetlb_bootmem_alloc was called early in boot. If not, it won't do anything, and HVO for bootmem hugetlb pages works as before. Signed-off-by: Frank van der Linden --- include/linux/hugetlb.h | 2 + mm/hugetlb.c | 4 +- mm/hugetlb_vmemmap.c | 143 ++++++++++++++++++++++++++++++++++++++++ mm/hugetlb_vmemmap.h | 6 ++ mm/sparse-vmemmap.c | 4 ++ 5 files changed, 157 insertions(+), 2 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 10a7ce2b95e1..2512463bca49 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -687,6 +687,8 @@ struct huge_bootmem_page { #define HUGE_BOOTMEM_HVO 0x0001 #define HUGE_BOOTMEM_ZONES_VALID 0x0002 +bool hugetlb_bootmem_page_zones_valid(int nid, struct huge_bootmem_page *m); + int isolate_or_dissolve_huge_page(struct page *page, struct list_head *list); int replace_free_hugepage_folios(unsigned long start_pfn, unsigned long end_pfn); struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 7e0810c217ce..29b3a6e70a53 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3310,8 +3310,8 @@ static void __init prep_and_add_bootmem_folios(struct hstate *h, } } -static bool __init hugetlb_bootmem_page_zones_valid(int nid, - struct huge_bootmem_page *m) +bool __init hugetlb_bootmem_page_zones_valid(int nid, + struct huge_bootmem_page *m) { unsigned long start_pfn; bool valid; diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index be6b33ecbc8e..9a99dfa3c495 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -743,6 +743,149 @@ void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head __hugetlb_vmemmap_optimize_folios(h, folio_list, true); } +#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT + +/* Return true of a bootmem allocated HugeTLB page should be pre-HVO-ed */ +static bool vmemmap_should_optimize_bootmem_page(struct huge_bootmem_page *m) +{ + unsigned long section_size, psize, pmd_vmemmap_size; + phys_addr_t paddr; + + if (!READ_ONCE(vmemmap_optimize_enabled)) + return false; + + if (!hugetlb_vmemmap_optimizable(m->hstate)) + return false; + + psize = huge_page_size(m->hstate); + paddr = virt_to_phys(m); + + /* + * Pre-HVO only works if the bootmem huge page + * is aligned to the section size. + */ + section_size = (1UL << PA_SECTION_SHIFT); + if (!IS_ALIGNED(paddr, section_size) || + !IS_ALIGNED(psize, section_size)) + return false; + + /* + * The pre-HVO code does not deal with splitting PMDS, + * so the bootmem page must be aligned to the number + * of base pages that can be mapped with one vmemmap PMD. + */ + pmd_vmemmap_size = (PMD_SIZE / (sizeof(struct page))) << PAGE_SHIFT; + if (!IS_ALIGNED(paddr, pmd_vmemmap_size) || + !IS_ALIGNED(psize, pmd_vmemmap_size)) + return false; + + return true; +} + +/* + * Initialize memmap section for a gigantic page, HVO-style. + */ +void __init hugetlb_vmemmap_init_early(int nid) +{ + unsigned long psize, paddr, section_size; + unsigned long ns, i, pnum, pfn, nr_pages; + unsigned long start, end; + struct huge_bootmem_page *m = NULL; + void *map; + + /* + * Noting to do if bootmem pages were not allocated + * early in boot, or if HVO wasn't enabled in the + * first place. + */ + if (!hugetlb_bootmem_allocated()) + return; + + if (!READ_ONCE(vmemmap_optimize_enabled)) + return; + + section_size = (1UL << PA_SECTION_SHIFT); + + list_for_each_entry(m, &huge_boot_pages[nid], list) { + if (!vmemmap_should_optimize_bootmem_page(m)) + continue; + + nr_pages = pages_per_huge_page(m->hstate); + psize = nr_pages << PAGE_SHIFT; + paddr = virt_to_phys(m); + pfn = PHYS_PFN(paddr); + map = pfn_to_page(pfn); + start = (unsigned long)map; + end = start + nr_pages * sizeof(struct page); + + if (vmemmap_populate_hvo(start, end, nid, + HUGETLB_VMEMMAP_RESERVE_SIZE) < 0) + continue; + + memmap_boot_pages_add(HUGETLB_VMEMMAP_RESERVE_SIZE / PAGE_SIZE); + + pnum = pfn_to_section_nr(pfn); + ns = psize / section_size; + + for (i = 0; i < ns; i++) { + sparse_init_early_section(nid, map, pnum, + SECTION_IS_VMEMMAP_PREINIT); + map += section_map_size(); + pnum++; + } + + m->flags |= HUGE_BOOTMEM_HVO; + } +} + +void __init hugetlb_vmemmap_init_late(int nid) +{ + struct huge_bootmem_page *m, *tm; + unsigned long phys, nr_pages, start, end; + unsigned long pfn, nr_mmap; + struct hstate *h; + void *map; + + if (!hugetlb_bootmem_allocated()) + return; + + if (!READ_ONCE(vmemmap_optimize_enabled)) + return; + + list_for_each_entry_safe(m, tm, &huge_boot_pages[nid], list) { + if (!(m->flags & HUGE_BOOTMEM_HVO)) + continue; + + phys = virt_to_phys(m); + h = m->hstate; + pfn = PHYS_PFN(phys); + nr_pages = pages_per_huge_page(h); + + if (!hugetlb_bootmem_page_zones_valid(nid, m)) { + /* + * Oops, the hugetlb page spans multiple zones. + * Remove it from the list, and undo HVO. + */ + list_del(&m->list); + + map = pfn_to_page(pfn); + + start = (unsigned long)map; + end = start + nr_pages * sizeof(struct page); + + vmemmap_undo_hvo(start, end, nid, + HUGETLB_VMEMMAP_RESERVE_SIZE); + nr_mmap = end - start - HUGETLB_VMEMMAP_RESERVE_SIZE; + memmap_boot_pages_add(DIV_ROUND_UP(nr_mmap, PAGE_SIZE)); + + memblock_phys_free(phys, huge_page_size(h)); + continue; + } else + m->flags |= HUGE_BOOTMEM_ZONES_VALID; + } +} +#endif + static const struct ctl_table hugetlb_vmemmap_sysctls[] = { { .procname = "hugetlb_optimize_vmemmap", diff --git a/mm/hugetlb_vmemmap.h b/mm/hugetlb_vmemmap.h index 926b8b27b5cb..0031e49b12f7 100644 --- a/mm/hugetlb_vmemmap.h +++ b/mm/hugetlb_vmemmap.h @@ -9,6 +9,8 @@ #ifndef _LINUX_HUGETLB_VMEMMAP_H #define _LINUX_HUGETLB_VMEMMAP_H #include +#include +#include /* * Reserve one vmemmap page, all vmemmap addresses are mapped to it. See @@ -25,6 +27,10 @@ long hugetlb_vmemmap_restore_folios(const struct hstate *h, void hugetlb_vmemmap_optimize_folio(const struct hstate *h, struct folio *folio); void hugetlb_vmemmap_optimize_folios(struct hstate *h, struct list_head *folio_list); void hugetlb_vmemmap_optimize_bootmem_folios(struct hstate *h, struct list_head *folio_list); +#ifdef CONFIG_SPARSEMEM_VMEMMAP_PREINIT +void hugetlb_vmemmap_init_early(int nid); +void hugetlb_vmemmap_init_late(int nid); +#endif static inline unsigned int hugetlb_vmemmap_size(const struct hstate *h) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index 8cc848c4b17c..fd2ab5118e13 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -32,6 +32,8 @@ #include #include +#include "hugetlb_vmemmap.h" + /* * Flags for vmemmap_populate_range and friends. */ @@ -594,6 +596,7 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, */ void __init sparse_vmemmap_init_nid_early(int nid) { + hugetlb_vmemmap_init_early(nid); } /* @@ -604,5 +607,6 @@ void __init sparse_vmemmap_init_nid_early(int nid) */ void __init sparse_vmemmap_init_nid_late(int nid) { + hugetlb_vmemmap_init_late(nid); } #endif From patchwork Thu Feb 6 18:51:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C4982C02199 for ; Thu, 6 Feb 2025 18:52:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E400280014; Thu, 6 Feb 2025 13:51:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 064FA280002; Thu, 6 Feb 2025 13:51:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF908280014; Thu, 6 Feb 2025 13:51:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BDA16280002 for ; Thu, 6 Feb 2025 13:51:53 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 77C7A4BDA0 for ; Thu, 6 Feb 2025 18:51:53 +0000 (UTC) X-FDA: 83090414106.19.DD8D2D9 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf16.hostedemail.com (Postfix) with ESMTP id B03FD18000E for ; Thu, 6 Feb 2025 18:51:51 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nayv+wA1; spf=pass (imf16.hostedemail.com: domain of 3xgSlZwQKCKcMcKSNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3xgSlZwQKCKcMcKSNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867911; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2/3TeRATauBC6P9iV7Z+/8og+I9Z6a+H92nsoJ8VemE=; b=2Eb3zbG1PtZPTTMtnibu02wFr0qLlleYS1VpvlHax9bgDisszkFK67VWVvyjtjk/WKg5Jd 2bVcf2HcumS3s2yBmejSubscWMZcXJ+BA/r8KX9D+x5WzkADELI3KSBJ7Xk1kBSGBiWvwn T1w74ZWOYrsdB2Q2B1qF1QCd2VRoh6Y= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nayv+wA1; spf=pass (imf16.hostedemail.com: domain of 3xgSlZwQKCKcMcKSNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3xgSlZwQKCKcMcKSNVVNSL.JVTSPUbe-TTRcHJR.VYN@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867911; a=rsa-sha256; cv=none; b=YfkWZj1Q7HuoRojJdCR8xdaMEt8l2qI+mrMFCVuiayI4yYKS5Ca8+x2Idu8PpKZvogS3np nZuegtOdFV5XpYpFJtWupwYeO92hTqx7qZ2ZFS7s471TfvnQ1MMhl4o78uErRdxn9iBpDk N3R/SbMTO40rAAdd7Y886G3TvCCkyNI= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f83e54432dso3848148a91.2 for ; Thu, 06 Feb 2025 10:51:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867910; x=1739472710; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2/3TeRATauBC6P9iV7Z+/8og+I9Z6a+H92nsoJ8VemE=; b=nayv+wA1U3kNCHxNWHUH2eW/MgT2eWf4usC47pekxa4zi/7RqPdZnNZG4erVPqT73O 1P5v3VabkkmKVypTw+MsgtC5wBqvY58CzwzKRhbTAlx+DhNocsCreCIhoCUpnpFIt/bE tFZayOE/h/b3NagFOtKdpTsqgB6Udnz3bN4N9s7M8qo6tQDYJ63MgeZ45W6MAGcF14yW DxGq50pxEzaOoKEOD+1G/EpzIkyABZpudm+E8GApb/2UH+m+OJ64jPA7FmbUKpl1Si2S 5p+j9wpCTS296q2O6OgQfvi0Dpjwjk6CiAZo39WxNoFi1n1s75b7d0NE+e45Ph7SJMhS VAcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867910; x=1739472710; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2/3TeRATauBC6P9iV7Z+/8og+I9Z6a+H92nsoJ8VemE=; b=ljQBGdFAheKORyGjUFHSTodNDLgSzqXCcOt+pJ2jAhigrDIzCggL6OnWIgNus3SylE N2SIuvsr4SrX1kVRQr80iBw8bJt/t/7SvtDTKquw6kno33OD25ixu4tUzwEGnj1b3Taa WOXpBCKwl81o4omvv1y85FE8a6YptHNz596QpSdZ+GzRJUdE+GryUbMeLhtOFlQr66B/ gZW+2Sm7ZQjfdPdXMk7HLbSbbWLN2EAIcFJmkA0K3JhFM/eGNNrvf3CiRE81wUN6jCd7 7AMJVp4BXLBc5CG9fUOdYpLX0u/4ej4xr5BO0S9DuzRmquSSnbFRIqy0saik6MkRRRaO lxxw== X-Forwarded-Encrypted: i=1; AJvYcCUyBu/aH1I0Lu2GX4UIABraOi6sDexGKjRFJJB4vkEMmHw6cJ0XFCpYOb3EjQ0IbIw2zuapSNIS3A==@kvack.org X-Gm-Message-State: AOJu0YxupVazM4G8tYrprlggHd2SG1Hdomw7K4tJn0UQ5i+e7Vt3Z44M 0ZuozbFLtPbI0ub0tWdZbMnoN/CHIgPDBu56GUwpiWU9jDMgWtYujhsuunYYg7k2N15r4Q== X-Google-Smtp-Source: AGHT+IFeJiIMbGUP+s5i04ssCNNkL93EOBt0wJtECDoXfHqJC0vwkVyoIA7fvHTV0zvzaIRZ3vC9thCy X-Received: from pjbpl16.prod.google.com ([2002:a17:90b:2690:b0:2e9:ee22:8881]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:3dc3:b0:2ee:edae:75e with SMTP id 98e67ed59e1d1-2fa24067d7amr209175a91.13.1738867910671; Thu, 06 Feb 2025 10:51:50 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:01 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-22-fvdl@google.com> Subject: [PATCH v3 21/28] x86/setup: call hugetlb_bootmem_alloc early From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden , Dave Hansen , Andy Lutomirski , Peter Zijlstra X-Rspamd-Queue-Id: B03FD18000E X-Stat-Signature: xog6eif8r531gntx4tjhqemafo947ykz X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1738867911-548565 X-HE-Meta: U2FsdGVkX1+WmJBwrokuVHsAvP4mm57myRBwY74avxucG8BrvYi0VIAFchyxZgz+wS/R866k8OignZ7bPLS7n7ZvXRKRGwcTHJqf0WrlRE9I+SHVVFZEEDtjhJzShSeLxx8e6YWGW6Sf/QBQ752sRgJw0s65gOSKoVGWfWjETK3VWQ4O9EV9wsfOzVLtc0MoAzhxnC4sYuFtnNIknPEQxuetEaMcW4pORsI2UeY4QAuMpDHMW/KnplsIhbN1Wnx5HlMJx8xtQABmv3H9egHHUdB9ADoQOMtLW7l2n9PT3GJE23ipS+/kWnCjlGwOrpfFXM3UALUyuY7GvgulrVapAvdlHwsDSFLLm+s6722XGezKy7cXiFEFlJlRZF7MjngRDvAPIdw8SWdknIZvrxC6az/wCmHZZ3JQyNDLUzDWX6NLv8qIdW1iO0sslZnxm3mtgzPOvoCPUhOW0KZfnKsgWb0t4EHYwXCs5vr9nJFhUZ5I94c4I7+g0LAMfDbREfua3sTIl0LsrFjUZKByGkxOyfKb5/qslSKBhcDsZy2jN/ixeaFBSxHmZQgV//8K9KEwOGWZvqpQKMj/JKVZbpU8CJ/dZo21kwWVJ8qzvehAJRDE5NdI2N4cuvAhELb35tdaabaaDy8qeJ78ku1MtHF9yM9ewdiF04fvWalb+IhaxpxTXYWlTn7ilZUmKN1h18E9MWB4HhHNG3159tEXbDor8m8dtt5QO5GprlsQdmyflxrSVRX6ijiJk0E/Uum9RvE8+vQCKDAGMfXwpu506ywUU79bxC9s+oUDvVOjUnwHseiXxiGZg67TwoUsaq4yvDrDUd3CrKS25//8ehu6Dub/uEQUDKVtZHanKaTKLFlkrYJHKPAF7L5fzB+uehFA7srhJyaE+mwwguYSNU1gD7LZ+Vgfj7+lW9Z33XsC7nR5nV5INYhvPkqQ6S17a3wFtzN50R6MG0Llz/zGmXOBUPZ +wgndS/f IbclnEqOTHGl1C1HynSlSIHCttF6t+YWpPCbCg9a44nj7ThRiesp0Hw45sVfb9BK1yQVQciJ0zsjjLxrZdjJr/xr2klvO9mmA53J2GhSh9RHqjDwyLi2KXq47DUS96gnQ8oFcYcbvpgG5H8lgZGnfgOl0BE0YpVsN8JyAYmIzSau89GoVU9j5/TXVYyGB5SroJePljqt81NpnncY2IdOqGzbKysPtbK5QUHFQDuoDFL3plZDKLITPeChJkyTnUwFCODhdquqLEr/2E5W3sOKq4yKZgfBaHuKgiEF+nOYAXTeUvQjYgAQ11dNjxKT/xw4i9CgnSjEidyd9H8BiNs9XtOewVgZZ+artvhgvloWuZ+ye2Ymh1mBGJBa5JbLvjYSuX7exVkgR4+S/R0e7aojA42ycLLtwhuikJ+IJNlPn3sMMUb+/UCMZob61nA1ZWPI3HLeBIImOOD8p0b0Kv7j70iQNmtzhZKgEovHSwMG6bukYODYoMLWt5yUHT7UPYEmjHXIqt8+Wl+l3mwKLpu4S5af6CZkzDiX8ZaDuIELvtxSeCBoxeB5eCKHb7PtCjHVxnpJA5KMmzToMrJu79rBvZASue2FULo3wBC7GppYm/dBUfERQX6TmP2E/9KDVMQ3No/bJmUGgGLyydwS12SmefgqiO4x4IQh2m/45cnyAw49LkuBmeZCYEk5aTQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Call hugetlb_bootmem_allloc in an earlier spot in setup, after hugelb_cma_reserve. This will make vmemmap preinit of the sections covered by the allocated hugetlb pages possible. Cc: Dave Hansen Cc: Andy Lutomirski Cc: Peter Zijlstra Signed-off-by: Frank van der Linden --- arch/x86/kernel/setup.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/arch/x86/kernel/setup.c b/arch/x86/kernel/setup.c index cebee310e200..ff8604007b08 100644 --- a/arch/x86/kernel/setup.c +++ b/arch/x86/kernel/setup.c @@ -1108,8 +1108,10 @@ void __init setup_arch(char **cmdline_p) initmem_init(); dma_contiguous_reserve(max_pfn_mapped << PAGE_SHIFT); - if (boot_cpu_has(X86_FEATURE_GBPAGES)) + if (boot_cpu_has(X86_FEATURE_GBPAGES)) { hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT); + hugetlb_bootmem_alloc(); + } /* * Reserve memory for crash kernel after SRAT is parsed so that it From patchwork Thu Feb 6 18:51:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07D0AC0219B for ; Thu, 6 Feb 2025 18:52:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A52A4280015; Thu, 6 Feb 2025 13:51:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DA9B280002; Thu, 6 Feb 2025 13:51:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7E12C280015; Thu, 6 Feb 2025 13:51:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 56200280002 for ; Thu, 6 Feb 2025 13:51:55 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0E78DB2650 for ; Thu, 6 Feb 2025 18:51:55 +0000 (UTC) X-FDA: 83090414190.26.60F6033 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 446D812000A for ; Thu, 6 Feb 2025 18:51:53 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oMeAvuw8; spf=pass (imf29.hostedemail.com: domain of 3yASlZwQKCKkOeMUPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--fvdl.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3yASlZwQKCKkOeMUPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867913; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p7TeecwDbtqxeCEQgMqbe8f8xJagmIFoeC0bQBtv3nE=; b=Vm3m9ZFTraDam+34EH7AQS4iR/ayeWopIPD8+bmaqxC1lD14g70J1MuC6p1L3yeu/x8+Ci rRUR/qhUfQyR1NmOUrcj2AJMmpszrhjNlcz7ka+1zLcZ9EEpQbDUWDW++N/1RlOBS12hQ7 3DXv3ZERNNQwvAVySGoRNLCKtkUK3Nc= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=oMeAvuw8; spf=pass (imf29.hostedemail.com: domain of 3yASlZwQKCKkOeMUPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--fvdl.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3yASlZwQKCKkOeMUPXXPUN.LXVURWdg-VVTeJLT.XaP@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867913; a=rsa-sha256; cv=none; b=WfI6vpn+Y5HAGbUC6o+imQXVKdIMSNmDCQEgCcyDApmbYykqKNOhaUn9hxUGkFNlTp8VQD S4Rie9gDd8ViJ8hBj6w1LrHmR8FOYelcmtQCYGgzksOuvO3jIswY2zQhJb0xJ8HVIqozWp /4ot+SXUUoMTvFFUWfuCLGKkhNRgueI= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-21f1e5c641aso43158205ad.2 for ; Thu, 06 Feb 2025 10:51:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867912; x=1739472712; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=p7TeecwDbtqxeCEQgMqbe8f8xJagmIFoeC0bQBtv3nE=; b=oMeAvuw8HFPH7hR/J92OaR10k7HGdSUhT8ni/MvX8gCL409SpcXnAwgQdBGfBrSw5c +yxH2y9zJcgHDwYROgX14J30WaJ7lgTmkLEA0JmBbaTXqUMrHka+BmdwcNOE5KyRckNT I6s6XL6H85XgQsHlCsb9UdIaPeoScGCd+2kDaa4Wp7IHSuJexlnD3PCVg6eZDII2nntc vzyc8C6gIF0vC+f+m9QeuMEtVCcmiSqNe0gboNS/g75hje19TaSgvWQRbUFwr4q8pFFf RtfvNtNBrsq5Pg3V6HgBtXxs/ijJyyaXRrjrXaZXpngv0KLTbG4CXH+fbCqVMcBoIGEY wOnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867912; x=1739472712; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=p7TeecwDbtqxeCEQgMqbe8f8xJagmIFoeC0bQBtv3nE=; b=UjIt0N9g68zWKhxI13VKHArHAG5Zoned2buDPDprB7oEh6KG/pHvU26Nrnq98l1uQn pdT+eFQypDUnK5Jw/7iD8ga8zK1A3VWGdMfAhVKsqCAV/WECS4chGbvqHUpr/2nHVcPT afivjLubPV+d0yNUdqnbTQxHeIYrFWCRkSfxbxEgvWQ0T2jvEJQs/JdR3+ekLxx+j3HC Kta8Poe/e6nxJXcq1Pil3m9+HYj1D5M37v1DlaHRtjpqWaL32qoWUmraUAAqwS+F1EY3 gxweRx8I+86mpKsJdgKgrdVbmWpHXmO+AabPaopU2RnmFhfSCtEkgJVku6a8H3zMqRLs 10UQ== X-Forwarded-Encrypted: i=1; AJvYcCVy4umo/3K9qtJA00uggwOgJiIEQVgGrVTw9n62pbFKAoXjUDoj3gFezt0ZXJ97BpFo53iYGf2fQQ==@kvack.org X-Gm-Message-State: AOJu0YxbDktcVkTx0YJaWXKX6HaJhdv6sfOzOv/IdK7RMus89CXoX0lr iU1NAJV31wNzxIKP58FYGFyTd6gx4xulSTo8AmS63vqwu1r8RIUIUMzrKB14Q0cHEmrukQ== X-Google-Smtp-Source: AGHT+IGTkHaS4WsaD1MB5yDYw0Ec61JJnhUDeKTqMYnnbssbsNlTFClrJPdlXWxU8MqAbVgIpap4v6N7 X-Received: from plec12.prod.google.com ([2002:a17:902:f30c:b0:216:5566:3c11]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a17:903:1c8:b0:215:b74c:d7ad with SMTP id d9443c01a7336-21f4e73b574mr2943595ad.36.1738867912218; Thu, 06 Feb 2025 10:51:52 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:02 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-23-fvdl@google.com> Subject: [PATCH v3 22/28] x86/mm: set ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 446D812000A X-Stat-Signature: 4nqz81aojcxwzax8fc1643asgbx51ymt X-HE-Tag: 1738867913-981640 X-HE-Meta: U2FsdGVkX18NgfW6ZSDG9GHBnR5AaLI2rLQEl5dCn5fE3D2bzXhxllHaUJnZ0q97nvk3QsFaQTFEIn9Ru86CaUb5+piNI/3AQl/SpEJA8vzI8yTUmiSmmlF1tS/gAa1vywSMgg975HqKhJfwFOBTs41UYFEPwVoeJq7Xd3joVYTlftrP84hOuZkOtf6G2BnSkZMG1C/CgTG5HWjFn785AcwBs51NX+G4vT50aquYDPcXid48FzEfWAzXB1DZj3UG2Zto6z3tSVkcfvO3f2/a7brvV3MeDKp1zWTyNyBFIdfPEaR9c42Kl3+99QYKPEtTHKRiDqvb4T0G/dooci/r1vbxDORC3qzvhXucDcOKSt9TgnY3FFQE+IK8n0bADMGWwBhjFwfstgerfFI62vylPmixCU2+kjmB6g4hvacV+KcVZyb1HXHNQFDJnNRTmBNWXUWN7mq8S+qlGS5tk4UMnmbOudrV2fulEt6/m3NKsD076CB51lfqyMcvhKN7Rufvjm0auwIw42FUOusSKANri1D6zR6jrU+sJlYgTo605tvTRhQJNvGTFJyP939SfHIG3LPhW/CoTTCJ94OXCpBV8VQHUfWAtCOuGqzqwqJbt8086pU06J76E1z4QhrYXkBZT+yrgjUdfco0L3moKVfhYaYoP2H885VZXB3TlkQIxqm0WxmuWQQXh+Qcj5Aac3ABbhhru1FPd6vPgzOX9UQxCRGSbeSr0Ey4pIWRyCQAAT/VJVn37/rbDsJM95P03hKLqoXoVuLroFLX+7e9Q5v9/MUS1ZSCkG7LX0yeOZSAjojSTsLDk4U9dNiFUwXvsbvR7PFA7aopdeEqC336xt7LYJgeFdk4m9Zq9WhIpCkcy02SlS6ak6NXdMW9cm8z+L+IKyCKyv8MWAWqWCloO7SHiloAdfhM4u65orAjccAjIoM8iQUVldk/gm61BFy7GJUyFOX48fVPQyRb6ZYFOd5 Xf45lOFu NojVGJEkFPn1h78RvuoAmllAqeF5Szxc21mbWaskrmSKcFmAj4ZYGAzgejTADVVLuT4ZjTUF4RhMq6YGfUQ8He+TSqVN9Mk5UJLFd+3hA4nm0IiJwwKILNVuLT6A8yg6412N8HvYik28PD/Zw/CDCrYf34sWA+DXcnBhzqrr69HfU2ObZiNyV8x2/Svu67ZpGOH0aOf8uK6tQ1osotM5dS8CJ8lFpPHpTQTNmFN+c4t2jxAwHpCjtnISHdS7W8wxTx0SWDb2RVJ5qAgkV8uSXVYICdWT1AzB0y7ATpGxjfTG/5VuaLhZsGu0hX6Ry1oDo5sYoG7RtUZkPehE82r8XpuZJN1xUm0G9C5MuGiZjNzpR/HCRhEYxKCEbrhidKLIAiikKKjkzS0b96v0lGyBrLuG3hUB9uv9BLR4au0WbcodOYYVhKV1S6RuW5cT9QM0sPXCUmtyKckWZETO2VYwclFouPRmIP9HebuWeE0jn+Cn72ZdcxQ6eOnr1X3+hWE0EAkYbf6WTrPecs3omSpgy4Rei0jsGpk+2KES3djjh2vcxyh9YdOchSQr8d53fIcbl1+/T X-Bogosity: Ham, tests=bogofilter, spamicity=0.000709, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now that hugetlb bootmem pages are allocated earlier, and available for section preinit (HVO-style), set ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT for x86_64, so that is can be done. This enables pre-HVO on x86_64. Signed-off-by: Frank van der Linden --- arch/x86/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 87198d957e2f..ccef99c0a2ba 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -146,6 +146,7 @@ config X86 select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP if X86_64 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 + select ARCH_WANT_SPARSEMEM_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT From patchwork Thu Feb 6 18:51:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01234C02194 for ; Thu, 6 Feb 2025 18:52:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5E795280016; Thu, 6 Feb 2025 13:51:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4D3CF280002; Thu, 6 Feb 2025 13:51:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D5C4280016; Thu, 6 Feb 2025 13:51:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 0ABA8280002 for ; Thu, 6 Feb 2025 13:51:57 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id BD62D1A13C5 for ; Thu, 6 Feb 2025 18:51:56 +0000 (UTC) X-FDA: 83090414232.29.EE23A30 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf20.hostedemail.com (Postfix) with ESMTP id EFB991C000B for ; Thu, 6 Feb 2025 18:51:54 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JNLkVsKl; spf=pass (imf20.hostedemail.com: domain of 3yQSlZwQKCKoPfNVQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3yQSlZwQKCKoPfNVQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867915; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LNVYMIarRsr0t8rcUQ946hUXqx5cMeLkMOmL07Z29aU=; b=7ri015VG2EQsaWyRkSg0VVyteyo+9vLUva7NeTBS6F2mi9wBu2LTvray6rApDEaNQwkNwL xZ305KBDcR14k3fPND9/25nKVBlKO33bI3E/oV5/efM4TY6+jJ7QiE+1ISqipVMLxyFTcd 4RfOt3f5boXyjuAE2f3wirtUTxpoG8U= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JNLkVsKl; spf=pass (imf20.hostedemail.com: domain of 3yQSlZwQKCKoPfNVQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=3yQSlZwQKCKoPfNVQYYQVO.MYWVSXeh-WWUfKMU.YbQ@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867915; a=rsa-sha256; cv=none; b=pP81kU4Iuj1q8OPCG6pxtEvlcih271iRi7SyQmAcVoHILOnXkXMpg8oslC27DKsMA/dDIr MZVMCEyDtrW4X3CLPNBK6l359BDgZ1SZLAZIFJuSwQ2TgshZj+FfSmCRZGoNSZbZH4UBjd tpJJDsFmiOaQTILPJcW0MZvDeqetcNE= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f9da17946fso3843539a91.3 for ; Thu, 06 Feb 2025 10:51:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867914; x=1739472714; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=LNVYMIarRsr0t8rcUQ946hUXqx5cMeLkMOmL07Z29aU=; b=JNLkVsKlk4Txp9swjm0PoTYC2Gro5OaQWDdTlebEn97SA03fvtIcPH4GWX3StLlJOk lmhgW0G9WS3bhJIYjO14xdvDWrynJprSYoBWEywcupNrq4DZpic+omh6HdZhiKoVXzrL w+GkJX3OsKFuHTU+QE6UhycXUDHNFD60+oy+JT4gQnVOom6W2sdpRBONg34o1pKVVJSL RJEHL6OYv0Wo8WVXWuYOUc/DfU6YFj0N788S/XIZIfOL6Ee/lVKRxyq0hmiJq0SEk4z2 4EjrpJlRKaSwozJDz5V17AaXDKlXKcPgTysCZdXIlrrdg7xRDttkJ1UQgZJiTvFjVh6K F+Bw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867914; x=1739472714; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=LNVYMIarRsr0t8rcUQ946hUXqx5cMeLkMOmL07Z29aU=; b=DIv2EzvZpystBxEkb1+HViO9z42Oy3FgjP1QMzjP6YgauqFWvrNX7zjeZJ6CL2YtNn pEJmkwe89hhvHY0RWDwMg6pwhvHJL+zkcoa4Hbdau8RkLoAguXf1+H9WwbocXsQmBFX8 imnvIT3vLJUd/BldvkQmQvAs3Jth812Qt4ZAXUMKNG4irAZjkejWdZljxHCrP2NEji6B RP7/Dc95jzEPK0PqE+LGlC8zyBGndAAbUzp/2sY6AH/LqFklgqTcKtGmPOyPaz+HQ69E LztBeMvN7CQJLKj60EfTy1CX1Qwj9VThxOHnBJ99CpMy/wjqEkTtDUIzN8d0OybUL92e XxIA== X-Forwarded-Encrypted: i=1; AJvYcCVtYQcyTxHJ47+5AoaZVxSgGjvdheHSsbIj+DEJLOLQ87pcaYlAMN1O0r0ir+OC2Zac1M7skp5fWg==@kvack.org X-Gm-Message-State: AOJu0YxYs89h/mFQ+/8xMqk8MgiaZQCYHwUnONKVBK7uEYbS+Vr6dfq2 ND2eFTnHFXbrYKpy5b9mxwghXkxXltmtVI7mp10reyGxIYsqrJeq5zvUvdje28CmFy0OVA== X-Google-Smtp-Source: AGHT+IFAHneXOb/4XfmdG9sW63Is4KRKWVRpRsXVSKXLK7gMZJrhr70i+wLMYHxanmPA9luzKj98Soc+ X-Received: from pfbko5.prod.google.com ([2002:a05:6a00:4605:b0:730:5607:bee4]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:139c:b0:72a:a7a4:b4cd with SMTP id d2e1a72fcca58-7305d54712emr393600b3a.21.1738867913918; Thu, 06 Feb 2025 10:51:53 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:03 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-24-fvdl@google.com> Subject: [PATCH v3 23/28] mm/cma: simplify zone intersection check From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Stat-Signature: 3g399yqqqgbhsnu7twqoiu8iiuw5wpq6 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: EFB991C000B X-HE-Tag: 1738867914-238220 X-HE-Meta: U2FsdGVkX1/7CFRcIUUXto40xglc8HkPEUXN/J9/nOrtsTEqwMfzVOvS/u+yUwlqTM7ZmsrlYs33grWve1e53tA3r3717WMuYxAA1BUH/NeqcV7T49tCKj7httNxZvbtqqMKWLbLpb8phtNr/9mA5SscH57lBhRcfsW/XOkF5gWVg/CSSLGHZ/3XIUUjGKbbMQaL48PHsn/sRzuVZqP/m1+rR7Vc5dJt65ZTnDH2GjiB4McvXbTGynB57Mv/xeQr8yD/T3YOShrb6/GV4ZXuoQZmP94n1qIgSDNncvD9unoATo490F0IqlNERUYPiDoCC/dbjTMF5tcIWw39l5a9U5iDaP/3hwYm1IiXciLzP8tjScJkEWLhDeSc/RwrZc2ki8E0p1rO96h3I0MUFgwPMmlWm/ndf2EKRH0h8s864Z+qGFOmIpBsu7hjMJaBT1Mwt8UIiBwiYil5EDoGskUCKmoTu76V+DD6KBbDiTu8HJE2cnRqYJyasKaLU3t+/2w+iqLAR9zSxq7eIY8YnEJbODzUdVnBYHlQ5sAmCu/6qbXwH9RYLJmFPmb2t+sJSLEaEcEBv+Wqxw1zg0g7BZzzk5d0sJbTBATQu/7hKzk/FfsLz0pXQkXtzfhFbKc53i8ShNcDlJWXU3/ed1Ja8Ny9a+a0LbgO7aRoR3Vz5QeNeTrFTvynvX4eNOQg/G11w/zXFvhhtEbz1gwp/QSSgJoV0Z4b/wupnD8tdeztag4JyOtuo49IMGz6B3RTdcxpCbogWV6KEs7hWrt7oRT634X/feKhhdAmsXV9dEeECLGy9oZynLCL6Szr8YNF0et1eLpQFF64lcFkCzj2qtNEOM70EPNeS6BhcpL5tZVVApFZtGsbCwqGfVZTPnduQxAdTaSOiz2edjZ4tQLAmW/n06dsB2IEC+bWevB0wxnN17Jf1km93UgudspwhakooFBsJZt6kr2zE0YuVTt+cendcay OhWhODKw pzDGV+JH5xJ8d7bC5A3pkyoQekAKjgngedY2nISP1AyZTPi3NKgGbzogIDTSXeyr26IX9UKi74z+JIcItx8pXWFWyj8yjebelH8nlsXQ1C6TFp6h/zp3WkkdwEIFJ5MLeRnoKyMiJYfYEZB9aqkyJhuF8t4ZwuMcFu/Iz1Uub+rq4A2zM48I15azAXCDIWFfr+E/tB6FF9H7O0bBrx2+Cmnt1idEbLhnsKHzMLxtWCK7PoFbv+S2yUsZ8z4AViTeQLPiOALzTQPDM5R8p6Ds4zuEcimeXtm5iaP86b4xkEAvG8m7J7/ZMELk8Hrls4TftzMiutlkU/S7MW1Mu4PKsp44S/dIfEiGuljtkx4wzYdN40tppf+pf3aAfN0idwY2IgQfw/uHjpmEZDz1Mi7t5ex2pDDp1ObsHgJg1do9RLxcvD/R6q3PRgEaKIPlInOZ1rCsu9IcL01M/ExcO9sbyPJgh6M3elW5n7gUHs2Vpdi0T5uRdnwfpbRzEmCjuJ4I2+0uRuD8biIPQLtyxqR85qTtQTg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: cma_activate_area walks all pages in the area, checking their zone individually to see if the area resides in more than one zone. Make this a little more efficient by using the recently introduced pfn_range_intersects_zones() function. Store the NUMA node id (if any) in the cma structure to facilitate this. Signed-off-by: Frank van der Linden --- mm/cma.c | 13 ++++++------- mm/cma.h | 2 ++ 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index c7116a5819c5..d0227af4bb71 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -103,7 +103,6 @@ static void __init cma_activate_area(struct cma *cma) { unsigned long pfn, base_pfn; int allocrange, r; - struct zone *zone; struct cma_memrange *cmr; for (allocrange = 0; allocrange < cma->nranges; allocrange++) { @@ -124,12 +123,8 @@ static void __init cma_activate_area(struct cma *cma) * CMA resv range to be in the same zone. */ WARN_ON_ONCE(!pfn_valid(base_pfn)); - zone = page_zone(pfn_to_page(base_pfn)); - for (pfn = base_pfn + 1; pfn < base_pfn + cmr->count; pfn++) { - WARN_ON_ONCE(!pfn_valid(pfn)); - if (page_zone(pfn_to_page(pfn)) != zone) - goto cleanup; - } + if (pfn_range_intersects_zones(cma->nid, base_pfn, cmr->count)) + goto cleanup; for (pfn = base_pfn; pfn < base_pfn + cmr->count; pfn += pageblock_nr_pages) @@ -261,6 +256,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, cma->ranges[0].base_pfn = PFN_DOWN(base); cma->ranges[0].count = cma->count; cma->nranges = 1; + cma->nid = NUMA_NO_NODE; *res_cma = cma; @@ -497,6 +493,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, } cma->nranges = nr; + cma->nid = nid; *res_cma = cma; out: @@ -684,6 +681,8 @@ static int __init __cma_declare_contiguous_nid(phys_addr_t base, if (ret) memblock_phys_free(base, size); + (*res_cma)->nid = nid; + return ret; } diff --git a/mm/cma.h b/mm/cma.h index 5f39dd1aac91..ff79dba5508c 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -50,6 +50,8 @@ struct cma { struct cma_kobject *cma_kobj; #endif bool reserve_pages_on_error; + /* NUMA node (NUMA_NO_NODE if unspecified) */ + int nid; }; extern struct cma cma_areas[MAX_CMA_AREAS]; From patchwork Thu Feb 6 18:51:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84A81C02199 for ; Thu, 6 Feb 2025 18:52:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E23CE280017; Thu, 6 Feb 2025 13:51:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D8480280002; Thu, 6 Feb 2025 13:51:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB0CE280017; Thu, 6 Feb 2025 13:51:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 98A9D280002 for ; Thu, 6 Feb 2025 13:51:58 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 635C21C8FE6 for ; Thu, 6 Feb 2025 18:51:58 +0000 (UTC) X-FDA: 83090414316.12.E2F6023 Received: from mail-pl1-f202.google.com (mail-pl1-f202.google.com [209.85.214.202]) by imf28.hostedemail.com (Postfix) with ESMTP id 920CAC000D for ; Thu, 6 Feb 2025 18:51:56 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=u8Co7Qc1; spf=pass (imf28.hostedemail.com: domain of 3ywSlZwQKCKwRhPXSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--fvdl.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3ywSlZwQKCKwRhPXSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867916; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yt0ZJmO4eMnkUw1+mDBNXTwvRT3PLwjDe9mC0zKYM+w=; b=eguPSwWei/TQyEGXe3tN8k/yt+2P+cYZhtb34gZCyUgizjBeb1JW1buQxpug4tdBZtVamZ KaKFaou+dMjt9nacmNaw4DQ8d68ESjFflBV76C1u6FDy8CLPHKmCXTjDTYYNdTST71z45A EJL4WPZElLupyaa5f5aBPcalxJm/5hA= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=u8Co7Qc1; spf=pass (imf28.hostedemail.com: domain of 3ywSlZwQKCKwRhPXSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--fvdl.bounces.google.com designates 209.85.214.202 as permitted sender) smtp.mailfrom=3ywSlZwQKCKwRhPXSaaSXQ.OaYXUZgj-YYWhMOW.adS@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867916; a=rsa-sha256; cv=none; b=lhh7rkeI9CI1XAvhmZrLnae6vAGqL4Cvx8dRrNHpGf2rbeXzPj4S9fphsblUXEm3GndMTF KhPkcECGNJa4es+YmUpD+6NMDPFUY1z7o1Vc5hFm1dGmETeo+g8G33iegBK1sVQtEpPPc6 ZLW9kWxwy3BjoWOMJSpsfHoZDkOXQFw= Received: by mail-pl1-f202.google.com with SMTP id d9443c01a7336-2161d5b3eb5so27180915ad.3 for ; Thu, 06 Feb 2025 10:51:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867915; x=1739472715; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=yt0ZJmO4eMnkUw1+mDBNXTwvRT3PLwjDe9mC0zKYM+w=; b=u8Co7Qc1086Y10XfLv05OgKhWNFUSJTlAvGIN6cpDzCbpR4/+Q3Q7pKARul/B+a/2j 0N+mcYvKYlXje5gqrqBqfP8gq3LOrQ0nVqgNfFxQxk0F6Xc1XDVUADnHVr29nLScQn2Q t6Be1onE+U16CvBbXL2gJUNyB7kVjGxV+FvM97wyyIRatsT1ZtNdt77hAEyRIBuYQ70P hHRjuwm9x76fYHasXZLvHNZcYziZS2TvXYEjG9n+G/abZrhMQEzCTfvV2kfyQEdyQUYh 6QhpaZOTN/wkM0rWFDlBYsiLm1Tlv6SWix6uZn/2kGeplCG/LiOmgImR4jLM2iUGGrtR Hdgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867915; x=1739472715; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=yt0ZJmO4eMnkUw1+mDBNXTwvRT3PLwjDe9mC0zKYM+w=; b=u/sZtdEBkCh/53MihBNKB3A4bPSrwp4bvHbihanQhLx3QgKKK1VbydpI7Z/5nvihyO OpUnsEwWhapXty6I2Timi31asa58mA9FQxm2LZa6UB3sciGgNME9a7NROSbkh0xBbNM6 6LAUGH0fkiLjo9+qRWMHsETc0YFZHQg8T9/qKiUUEfPHnZdg+L6xf5Ez8LgScwmJVXUt 6CVkBmBFkpwtrQXl8dVo/udNlkoogxcg5jGNAhPJSbiFWwFlbGsEiXd1bzW0SiGT2x/m Ko0hvuqkUWO0MqHUMgBAJEGx6Nhxw5ANbwuOTEIGeNYWYLA8cNwtipyisR11LZumsZW2 MdcA== X-Forwarded-Encrypted: i=1; AJvYcCUQNZP/x4dWjBI8Qi6aP6BiEbn13Il9R017tTzQtYWFcSqVmAKyn2ZzqCC1+JJxEZe2M2LAYfuT/A==@kvack.org X-Gm-Message-State: AOJu0YzaXa18Y2m0H53XANd8jyKU1MQ9Smedu9fTHn07RVtoU3vMJlKs o9nq2J7v3OYvEVZ70dYqEx1WshYnBbOA8wZUi2X0zJGeUNBclOXWbNDoxNB75yUlSAGy9Q== X-Google-Smtp-Source: AGHT+IFZhH2u1+M+xFH2j0CpMHxi5A7C45tXR32gxTZbZ8iZD/tY8JR/54qpQpgRoE2PyZBUqmPt1YR+ X-Received: from pgcv13.prod.google.com ([2002:a05:6a02:530d:b0:801:d724:5abd]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:9f8e:b0:1e1:94a2:275c with SMTP id adf61e73a8af0-1ee03a3f4camr941358637.18.1738867915504; Thu, 06 Feb 2025 10:51:55 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:04 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-25-fvdl@google.com> Subject: [PATCH v3 24/28] mm/cma: introduce a cma validate function From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 920CAC000D X-Stat-Signature: h8p5rg6tuczmqwazr65g8a7pybs4bna1 X-Rspam-User: X-HE-Tag: 1738867916-577436 X-HE-Meta: U2FsdGVkX181KCQbKYIy0JrPMzwbRYws7pj+qLRaD9PPwC9ldzyo/+yIdkg4xkpnKQ8d5nYfLj220c+YCELgAnf9F/jMvtBFxYIDBa0p1xaM6dwZREs9UcZcyUghxS+MubKP0rmuZ4xg37z38iKZShr0ZFEjVh9T+yvaR4kF9LIiSuaf4ahO44+Wv2l2bcmbz9gjJcQrm9hsHinC+qTAznGBOOnzXOmVIx4JDMSI0nkVRPx3C0lOLG8W+tjBe7zcWZk8xIM67IBd6Nn66rMhSk74tMVgwrjx5jJ4yvbYQ0OaAPPGUCHNV/NTYvCBZ4f7RIpm8Q7TjlhZZyaqwy/iftrN9xgKj53WrOBPKNrtvtvhciJULjmFWuoBELg/yruEvhrk357JJ6jYrolnoNFfe2qTymqMaRNN9nS+3eDGfDJcRhTWfqHFiEq63N3nhdLcSFTuaxtFVkzMNXmLhUXna74V+xDHDpVhZGG0TwnEfodHCgOZ8F1p/wfS+mMHGI7xnNqddvozyTjwEyjgeNcj8HPgJ0IU5Znu95pKqxxJoG6FpuR54a/Vg5lfc4x/lJjyTS1azxxL0/tZSFowRPwyr9o+84tx6M/0iR+BIx4gnkI6LGldTOqH1M2ci0QmCcKvtv+Exw+GhqbfqX+XynWGW1q6JPhVXEQLyHNrOF2rghP5zqVlWphJBYmq3TKuBzQQhHK3erKaue08J7MJlX4sTInyULoJeyXW0/Bt1SMTOP5VbwwcfxAclgbLjFm3Goucv+37dI6IZdN/oBSlFNq6tFw6nSQwYx/hS/0At06mxT0vcAM4igpd0vyGNWBsDa8kX/r/IUtIcK9F1jEeTIqdIbQDa2Zk+N5a21Is9JoyLtLY1UWHZ4hkV11YY6ZPzZacINSoEKNmcuI6Wc4UKMvl+itLv7pVpkebn+j2nOYgJu6IUuJntHv8br7+kTGJHljYdxMM7dRIerR+0jwL9Iz VTPAFbjz GfZAKfI7ytyX2D4TtxrvqUBUMsL1XW/vDuSpb2ghCAFEDYm66iQYacreZ5+Tj/blTLNIswn58z0ePXHzzicRddaMjOdWitQtpooqcDQCJsV2pjTWD9AjrwaRlYpKdSYPU7DQCsOJJzm+I3aRyF10g0sonWPKv7vVEZDZbO3JYqvbRg9ETrfH8MFQvhh7W/AP2aeBYpzoXuRXqdFFfXduP3QBWDIcyMr370qAg86CKWNqtP1iNMRDZoxrqifIfKXs90WnuWiEE48f3zfH3gDQc5s64Lk1R8v4mA7HK6eJI3mVDoK67qYUvfOjFBj0kay40IiFojkFRm0qI4PzH01byWOI3z1KJ3+eLst66F8FzH9aMcDuOi/rBPEtAqOEWkCGjym+elBVbe7cStzCxWh8regrykhqZqHvG/L4wsQI7Y+YhNh2UZnOUVISNEFk32N9e1cTYaS7M2jdu6CooLEAX34N1Zr/gNmnIYpsylDGhuxuyXSxf+hru23vOh0mvkw4oj/Xe8qo6tN5aqqwVp+oQ1Og/Gg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Define a function to check if a CMA area is valid, which means: do its ranges not cross any zone boundaries. Store the result in the newly created flags for each CMA area, so that multiple calls are dealt with. This allows for checking the validity of a CMA area early, which is needed later in order to be able to allocate hugetlb bootmem pages from it with pre-HVO. Signed-off-by: Frank van der Linden --- include/linux/cma.h | 5 ++++ mm/cma.c | 60 ++++++++++++++++++++++++++++++++++++--------- mm/cma.h | 8 +++++- 3 files changed, 60 insertions(+), 13 deletions(-) diff --git a/include/linux/cma.h b/include/linux/cma.h index 03d85c100dcc..62d9c1cf6326 100644 --- a/include/linux/cma.h +++ b/include/linux/cma.h @@ -60,6 +60,7 @@ extern void cma_reserve_pages_on_error(struct cma *cma); #ifdef CONFIG_CMA struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp); bool cma_free_folio(struct cma *cma, const struct folio *folio); +bool cma_validate_zones(struct cma *cma); #else static inline struct folio *cma_alloc_folio(struct cma *cma, int order, gfp_t gfp) { @@ -70,6 +71,10 @@ static inline bool cma_free_folio(struct cma *cma, const struct folio *folio) { return false; } +static inline bool cma_validate_zones(struct cma *cma) +{ + return false; +} #endif #endif diff --git a/mm/cma.c b/mm/cma.c index d0227af4bb71..4388d941d381 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -99,6 +99,49 @@ static void cma_clear_bitmap(struct cma *cma, const struct cma_memrange *cmr, spin_unlock_irqrestore(&cma->lock, flags); } +/* + * Check if a CMA area contains no ranges that intersect with + * multiple zones. Store the result in the flags in case + * this gets called more than once. + */ +bool cma_validate_zones(struct cma *cma) +{ + int r; + unsigned long base_pfn; + struct cma_memrange *cmr; + bool valid_bit_set; + + /* + * If already validated, return result of previous check. + * Either the valid or invalid bit will be set if this + * check has already been done. If neither is set, the + * check has not been performed yet. + */ + valid_bit_set = test_bit(CMA_ZONES_VALID, &cma->flags); + if (valid_bit_set || test_bit(CMA_ZONES_INVALID, &cma->flags)) + return valid_bit_set; + + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + base_pfn = cmr->base_pfn; + + /* + * alloc_contig_range() requires the pfn range specified + * to be in the same zone. Simplify by forcing the entire + * CMA resv range to be in the same zone. + */ + WARN_ON_ONCE(!pfn_valid(base_pfn)); + if (pfn_range_intersects_zones(cma->nid, base_pfn, cmr->count)) { + set_bit(CMA_ZONES_INVALID, &cma->flags); + return false; + } + } + + set_bit(CMA_ZONES_VALID, &cma->flags); + + return true; +} + static void __init cma_activate_area(struct cma *cma) { unsigned long pfn, base_pfn; @@ -113,19 +156,12 @@ static void __init cma_activate_area(struct cma *cma) goto cleanup; } + if (!cma_validate_zones(cma)) + goto cleanup; + for (r = 0; r < cma->nranges; r++) { cmr = &cma->ranges[r]; base_pfn = cmr->base_pfn; - - /* - * alloc_contig_range() requires the pfn range specified - * to be in the same zone. Simplify by forcing the entire - * CMA resv range to be in the same zone. - */ - WARN_ON_ONCE(!pfn_valid(base_pfn)); - if (pfn_range_intersects_zones(cma->nid, base_pfn, cmr->count)) - goto cleanup; - for (pfn = base_pfn; pfn < base_pfn + cmr->count; pfn += pageblock_nr_pages) init_cma_reserved_pageblock(pfn_to_page(pfn)); @@ -145,7 +181,7 @@ static void __init cma_activate_area(struct cma *cma) bitmap_free(cma->ranges[r].bitmap); /* Expose all pages to the buddy, they are useless for CMA. */ - if (!cma->reserve_pages_on_error) { + if (!test_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags)) { for (r = 0; r < allocrange; r++) { cmr = &cma->ranges[r]; for (pfn = cmr->base_pfn; @@ -172,7 +208,7 @@ core_initcall(cma_init_reserved_areas); void __init cma_reserve_pages_on_error(struct cma *cma) { - cma->reserve_pages_on_error = true; + set_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags); } static int __init cma_new_area(const char *name, phys_addr_t size, diff --git a/mm/cma.h b/mm/cma.h index ff79dba5508c..bddc84b3cd96 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -49,11 +49,17 @@ struct cma { /* kobject requires dynamic object */ struct cma_kobject *cma_kobj; #endif - bool reserve_pages_on_error; + unsigned long flags; /* NUMA node (NUMA_NO_NODE if unspecified) */ int nid; }; +enum cma_flags { + CMA_RESERVE_PAGES_ON_ERROR, + CMA_ZONES_VALID, + CMA_ZONES_INVALID, +}; + extern struct cma cma_areas[MAX_CMA_AREAS]; extern unsigned int cma_area_count; From patchwork Thu Feb 6 18:51:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C852C02194 for ; Thu, 6 Feb 2025 18:52:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 98F68280018; Thu, 6 Feb 2025 13:52:00 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91749280002; Thu, 6 Feb 2025 13:52:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 769FD280018; Thu, 6 Feb 2025 13:52:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 503A8280002 for ; Thu, 6 Feb 2025 13:52:00 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 11854B255A for ; Thu, 6 Feb 2025 18:52:00 +0000 (UTC) X-FDA: 83090414400.24.EA1D2BD Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf13.hostedemail.com (Postfix) with ESMTP id 4081A20005 for ; Thu, 6 Feb 2025 18:51:58 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0hl+6DcX; spf=pass (imf13.hostedemail.com: domain of 3zQSlZwQKCK4TjRZUccUZS.QcaZWbil-aaYjOQY.cfU@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3zQSlZwQKCK4TjRZUccUZS.QcaZWbil-aaYjOQY.cfU@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867918; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Os2cqKctxCUcRS8FYnguBqsdmbx6qbrypyMDT5BCAmU=; b=zBmyo60uBu0WTHjY8+NTS53X6mQHSUyBq3jFrXSVK8bnfYt5dqAzwbrvkodVmYCe8Ghz1q t0gbHq7YBTz64E8XDE4yHSr1RtI30yDtByiUzAFHgN89aSgtxR04bU5Zjo5hz6d0pYW1km lqkDYmWoUuQn4nALpcrhvi7A8Jgj7JM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=0hl+6DcX; spf=pass (imf13.hostedemail.com: domain of 3zQSlZwQKCK4TjRZUccUZS.QcaZWbil-aaYjOQY.cfU@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3zQSlZwQKCK4TjRZUccUZS.QcaZWbil-aaYjOQY.cfU@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867918; a=rsa-sha256; cv=none; b=g3n4lseLUWuUmo+tPSOFtKKp1EcI6G6fcBPkv3JPopLWYYk1G6yJrW2Uv6Xk0th2WKOnXO xhCeVz6Nem6yLCtvTi9EEOOkLJZj2iK4QP9tXVdOHnXGATZavsb/2He1vgKV554X5QKkTI WGQejY/r9UQ+FNLU34ACLeNhEnKobTo= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2f9fbe02121so2523521a91.0 for ; Thu, 06 Feb 2025 10:51:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867917; x=1739472717; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Os2cqKctxCUcRS8FYnguBqsdmbx6qbrypyMDT5BCAmU=; b=0hl+6DcXQsrLwquY7KQK5Ap1E7sKlcGxCI6ruyYugAdYuMngqFiII/A7KMeGQzr1oP fGjhQvGCUmXyRZpX5f2F6EHL6nmOOdXEvgk8D1F5PgI5nMg0iiEe8CKZcb93p35FjCJo 2XCUegCLqffIFjzLmhGUV3VVCWft5efh6GKhGBUqs6cZrSaDJQAVWHjZ3nh2Fv/uHN6h U6Ronu3vUbgW4p0mIVfWKDMV2RJmB5CqYU5jvqtEpMsl4AMwNl4tQB0Kv3jDV0WH8MVz Ug5IicO0mqyjht5HOxSvWlMB3RLRBsj8cQuu9qECf0OPG95UqRq6FR/83I/h/N69ZuvU hIHA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867917; x=1739472717; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Os2cqKctxCUcRS8FYnguBqsdmbx6qbrypyMDT5BCAmU=; b=c41+GVkfozn3b7IUK/oXU1nErvhUNCVKBHILJNfEGxQIlPNE2f7e+O967gxAvMAqKN ZVHQMmd9U9fEkBV62QECoDNweBdGFqd1SYKY0qBDP49OgIvTO2tS55QF49HMEi5MNxKZ pMF/NEMnmKTkjOyEVXWuMdub6nDE2d3pF6XMGGmHELrzZJtz53s0iY9lktbsaD05B8Ir WcWOpDUn9j8Oxe4YbauS1gyf+E2MkBPXplCmgncg41dt/RVBPd05kqkdSk3dfCk0cH1y cpykPE1laL+PDCrae9Qq44WMRuiSaw3LtIquO3FL9J2mX03/7MR5IFSqFGAbDFnuws8x cRcQ== X-Forwarded-Encrypted: i=1; AJvYcCXeEt+bpc2Yeu8f8shxV5avinIvPeuMia35T0QHKXbKNK4f54OdNx7mcfGVNpS1XEH9C5DV9PCwIA==@kvack.org X-Gm-Message-State: AOJu0YxbYAlWYG6Am4AUpVeiIeBcFoNaixqZ1qrmf+wl88IjsmxbaNed Vji6bGcXNiPB8IaUn6x5DlKymvOHP6VEP0kqxl96QsmCt7QVmAT2IeWn4e9zyKKWNb/E7Q== X-Google-Smtp-Source: AGHT+IHVX/1oNg+jUxgfnaihB0fSHJ3u8Hg2NHEhvkhZ5xKtCbM1RyTOrbuLCZ62fn2ArYVFlFmi+NRu X-Received: from pfmb20.prod.google.com ([2002:a05:6a00:1154:b0:730:4a3b:49ca]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:420b:b0:725:e499:5b8a with SMTP id d2e1a72fcca58-7305d505b69mr532061b3a.15.1738867917188; Thu, 06 Feb 2025 10:51:57 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:05 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-26-fvdl@google.com> Subject: [PATCH v3 25/28] mm/cma: introduce interface for early reservations From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 4081A20005 X-Stat-Signature: ipcwkjad4xhkui857chio6oqmkh9f7xy X-HE-Tag: 1738867918-81929 X-HE-Meta: U2FsdGVkX19UByiuVRnuFHZ5AVcGVMYsdwu/erFcKCWVvHk1RJ7kz5VvlFI7vWF4u2NjO+I45pKG/3TGTx+TvVaRtbDR1YKYCoAFAwWPmRauyoAaPWUJaCrx698Un/gbBaPj9YeA/lcJdDldK7FErzq/Vo3uTXHwT6IPemb7DOsxwE1dLo+u2+ZwIFAc68bhYUIIxA5WU+JB/D8OpsAx3m9J2TP/Y9MGc/oiWZuPQtPwObbLL9RWIyhyl5nK6XFhtGat/EZsPl0LqS6cPVg68bJ1PfHqTqM6ZA0u3i7yn59yqKWtTIr+E9q3CLHyy8O8TVGd528/FPtVIaprUSgMtyWk+cN1tG1ZFA+65MnJ+z3lRp+dl7+opfQEwzeT462I3qxb8RUOhbASbuOM3wmjULw3otRfSDLLEJ5dW4MOvlJZsswSDK1YxG8dkKjPf+CXy1mlXdbDI8oNDtug5PYDjxpqIe6+pgAiRufAgnSSLrqYZM2Y75UggBUThLoiurNpMOBghSjNbty+anbnELjFyy8Qkw7ZDr3gj6QhBYmgRTpkXZYefb4PwMb92xuqzUQbf8qAn/iEblvULOJGwf4o0HzYMVLh0y5OgyLEeAKWxwPkw2pY6Mq10crnDH7iZeXDinBx89LXiwBTpbflS/xthRO2Dayzd1I93+HJpjHMgigIZiy8HoEzv3Gh8M9YNszqMvdKMgBpX3LslaBmWeRFKd3rjULcHlkHWYJb9mozYXoAOBs8vXBlBN8gKnM5jepSfVb3oLblZRi+egrvr7iPhy/NDSGTHVO9ZKDX+BUAsh5HRwrKvNijPaLa+u7FzpLIz30dRvB7HK80Zoxgjo1uTw5yBmEfZJLl6LQnyGok002Z42ydYb8zDhXA3EoKfEaoPmf0YpkIcCE2uSfg2d+L2zPQ3LJcmkGdnxOdzqb6zRFuNyCl4rTuJ/KxgSkmyCoH0lILqtO/CtbLc8pDos9 EZjQ+t9B UoTuCTX8X1KHL/QjKwJ+UD8DMS35LevBYPlyWmtc9yaxhQxs43wnUHt0twm0ppL8X33R3PS/XEp9/4DKbv6tBkxda+NHkJC4/fn9RVV9xtQ5L6xIfltjmwUdVd5UQHqhhvJ2C9lVGGkg1ZcBlSwKTnV7mBwmlvCrMf2kwKNi7/j/PaB3lKuh5Wv0qZDAuQhQx9Q8FCv/iR/2Ab7+LkwLE20xDYhYfteJqJ0qhocJ/zfQaoJ/34GC8EGpxeVRnTDsmff+nN/qU39wpQGYhwKTQMMdvRxZZTOC2grxOr5KCkumX1F2u2wGRRNR5uONYSk+StFcWX/DKBMwew9117zL8gTqVKibamtm2VkrfPc+kJIcDvtgeTtfw5kE1XP/ZUy3dVoQ7pwDByMHQQE51/Shn6aI3qNOMNd3Ly0F/CzFsuhhJNid+Jd4DBLC04ACii7AizE9hj5W9WFfLXVFkp1Lg5+2iKo9ojFG6zsUf1gcaMKyEiB7tEvN+ePS9j8uVUpY9vZpeChw6M+CNKZ6kjhTVfXTEeg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: It can be desirable to reserve memory in a CMA area before it is activated, early in boot. Such reservations would effectively be memblock allocations, but they can be returned to the CMA area later. This functionality can be used to allow hugetlb bootmem allocations from a hugetlb CMA area. A new interface, cma_reserve_early is introduced. This allows for pageblock-aligned reservations. These reservations are skipped during the initial handoff of pages in a CMA area to the buddy allocator. The caller is responsible for making sure that the page structures are set up, and that the migrate type is set correctly, as with other memblock allocations that stick around. If the CMA area fails to activate (because it intersects with multiple zones), the reserved memory is not given to the buddy allocator, the caller needs to take care of that. Signed-off-by: Frank van der Linden --- mm/cma.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++----- mm/cma.h | 8 +++++ mm/internal.h | 16 ++++++++++ mm/mm_init.c | 9 ++++++ 4 files changed, 109 insertions(+), 7 deletions(-) diff --git a/mm/cma.c b/mm/cma.c index 4388d941d381..34a4df29af72 100644 --- a/mm/cma.c +++ b/mm/cma.c @@ -144,9 +144,10 @@ bool cma_validate_zones(struct cma *cma) static void __init cma_activate_area(struct cma *cma) { - unsigned long pfn, base_pfn; + unsigned long pfn, end_pfn; int allocrange, r; struct cma_memrange *cmr; + unsigned long bitmap_count, count; for (allocrange = 0; allocrange < cma->nranges; allocrange++) { cmr = &cma->ranges[allocrange]; @@ -161,8 +162,13 @@ static void __init cma_activate_area(struct cma *cma) for (r = 0; r < cma->nranges; r++) { cmr = &cma->ranges[r]; - base_pfn = cmr->base_pfn; - for (pfn = base_pfn; pfn < base_pfn + cmr->count; + if (cmr->early_pfn != cmr->base_pfn) { + count = cmr->early_pfn - cmr->base_pfn; + bitmap_count = cma_bitmap_pages_to_bits(cma, count); + bitmap_set(cmr->bitmap, 0, bitmap_count); + } + + for (pfn = cmr->early_pfn; pfn < cmr->base_pfn + cmr->count; pfn += pageblock_nr_pages) init_cma_reserved_pageblock(pfn_to_page(pfn)); } @@ -173,6 +179,7 @@ static void __init cma_activate_area(struct cma *cma) INIT_HLIST_HEAD(&cma->mem_head); spin_lock_init(&cma->mem_head_lock); #endif + set_bit(CMA_ACTIVATED, &cma->flags); return; @@ -184,9 +191,8 @@ static void __init cma_activate_area(struct cma *cma) if (!test_bit(CMA_RESERVE_PAGES_ON_ERROR, &cma->flags)) { for (r = 0; r < allocrange; r++) { cmr = &cma->ranges[r]; - for (pfn = cmr->base_pfn; - pfn < cmr->base_pfn + cmr->count; - pfn++) + end_pfn = cmr->base_pfn + cmr->count; + for (pfn = cmr->early_pfn; pfn < end_pfn; pfn++) free_reserved_page(pfn_to_page(pfn)); } } @@ -290,6 +296,7 @@ int __init cma_init_reserved_mem(phys_addr_t base, phys_addr_t size, return ret; cma->ranges[0].base_pfn = PFN_DOWN(base); + cma->ranges[0].early_pfn = PFN_DOWN(base); cma->ranges[0].count = cma->count; cma->nranges = 1; cma->nid = NUMA_NO_NODE; @@ -509,6 +516,7 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, nr, (u64)mlp->base, (u64)mlp->base + size); cmrp = &cma->ranges[nr++]; cmrp->base_pfn = PHYS_PFN(mlp->base); + cmrp->early_pfn = cmrp->base_pfn; cmrp->count = size >> PAGE_SHIFT; sizeleft -= size; @@ -540,7 +548,6 @@ int __init cma_declare_contiguous_multi(phys_addr_t total_size, pr_info("Reserved %lu MiB in %d range%s\n", (unsigned long)total_size / SZ_1M, nr, nr > 1 ? "s" : ""); - return ret; } @@ -1034,3 +1041,65 @@ bool cma_intersects(struct cma *cma, unsigned long start, unsigned long end) return false; } + +/* + * Very basic function to reserve memory from a CMA area that has not + * yet been activated. This is expected to be called early, when the + * system is single-threaded, so there is no locking. The alignment + * checking is restrictive - only pageblock-aligned areas + * (CMA_MIN_ALIGNMENT_BYTES) may be reserved through this function. + * This keeps things simple, and is enough for the current use case. + * + * The CMA bitmaps have not yet been allocated, so just start + * reserving from the bottom up, using a PFN to keep track + * of what has been reserved. Unreserving is not possible. + * + * The caller is responsible for initializing the page structures + * in the area properly, since this just points to memblock-allocated + * memory. The caller should subsequently use init_cma_pageblock to + * set the migrate type and CMA stats the pageblocks that were reserved. + * + * If the CMA area fails to activate later, memory obtained through + * this interface is not handed to the page allocator, this is + * the responsibility of the caller (e.g. like normal memblock-allocated + * memory). + */ +void __init *cma_reserve_early(struct cma *cma, unsigned long size) +{ + int r; + struct cma_memrange *cmr; + unsigned long available; + void *ret = NULL; + + if (!cma || !cma->count) + return NULL; + /* + * Can only be called early in init. + */ + if (test_bit(CMA_ACTIVATED, &cma->flags)) + return NULL; + + if (!IS_ALIGNED(size, CMA_MIN_ALIGNMENT_BYTES)) + return NULL; + + if (!IS_ALIGNED(size, (PAGE_SIZE << cma->order_per_bit))) + return NULL; + + size >>= PAGE_SHIFT; + + if (size > cma->available_count) + return NULL; + + for (r = 0; r < cma->nranges; r++) { + cmr = &cma->ranges[r]; + available = cmr->count - (cmr->early_pfn - cmr->base_pfn); + if (size <= available) { + ret = phys_to_virt(PFN_PHYS(cmr->early_pfn)); + cmr->early_pfn += size; + cma->available_count -= size; + return ret; + } + } + + return ret; +} diff --git a/mm/cma.h b/mm/cma.h index bddc84b3cd96..df7fc623b7a6 100644 --- a/mm/cma.h +++ b/mm/cma.h @@ -16,9 +16,16 @@ struct cma_kobject { * and the total amount of memory requested, while smaller than the total * amount of memory available, is large enough that it doesn't fit in a * single physical memory range because of memory holes. + * + * Fields: + * @base_pfn: physical address of range + * @early_pfn: first PFN not reserved through cma_reserve_early + * @count: size of range + * @bitmap: bitmap of allocated (1 << order_per_bit)-sized chunks. */ struct cma_memrange { unsigned long base_pfn; + unsigned long early_pfn; unsigned long count; unsigned long *bitmap; #ifdef CONFIG_CMA_DEBUGFS @@ -58,6 +65,7 @@ enum cma_flags { CMA_RESERVE_PAGES_ON_ERROR, CMA_ZONES_VALID, CMA_ZONES_INVALID, + CMA_ACTIVATED, }; extern struct cma cma_areas[MAX_CMA_AREAS]; diff --git a/mm/internal.h b/mm/internal.h index 63fda9bb9426..8318c8e6e589 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -848,6 +848,22 @@ void init_cma_reserved_pageblock(struct page *page); #endif /* CONFIG_COMPACTION || CONFIG_CMA */ +struct cma; + +#ifdef CONFIG_CMA +void *cma_reserve_early(struct cma *cma, unsigned long size); +void init_cma_pageblock(struct page *page); +#else +static inline void *cma_reserve_early(struct cma *cma, unsigned long size) +{ + return NULL; +} +static inline void init_cma_pageblock(struct page *page) +{ +} +#endif + + int find_suitable_fallback(struct free_area *area, unsigned int order, int migratetype, bool only_stealable, bool *can_steal); diff --git a/mm/mm_init.c b/mm/mm_init.c index f7d5b4fe1ae9..f31260fd393e 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2263,6 +2263,15 @@ void __init init_cma_reserved_pageblock(struct page *page) adjust_managed_page_count(page, pageblock_nr_pages); page_zone(page)->cma_pages += pageblock_nr_pages; } +/* + * Similar to above, but only set the migrate type and stats. + */ +void __init init_cma_pageblock(struct page *page) +{ + set_pageblock_migratetype(page, MIGRATE_CMA); + adjust_managed_page_count(page, pageblock_nr_pages); + page_zone(page)->cma_pages += pageblock_nr_pages; +} #endif void set_zone_contiguous(struct zone *zone) From patchwork Thu Feb 6 18:51:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F38FFC0219B for ; Thu, 6 Feb 2025 18:52:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 573F7280019; Thu, 6 Feb 2025 13:52:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 50798280002; Thu, 6 Feb 2025 13:52:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 37222280019; Thu, 6 Feb 2025 13:52:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 11269280002 for ; Thu, 6 Feb 2025 13:52:02 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C2F381C9195 for ; Thu, 6 Feb 2025 18:52:01 +0000 (UTC) X-FDA: 83090414442.18.32152B8 Received: from mail-pl1-f201.google.com (mail-pl1-f201.google.com [209.85.214.201]) by imf02.hostedemail.com (Postfix) with ESMTP id EA0A280014 for ; Thu, 6 Feb 2025 18:51:59 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="sJbwZ/l8"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3zgSlZwQKCK8UkSaVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3zgSlZwQKCK8UkSaVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--fvdl.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867920; a=rsa-sha256; cv=none; b=WfkEzRcGnVacyouvhtgrPVy8CW1OjijOwFRYROjwVwBj6IWUujHg7LrjSf4jo6521YM3HS sLElxDUwSFeakgDSIkF6pCMQJsauBh4SQLLqBAFx5VWoRf8mDpgbiuAYyFgYtB12GMWs78 jp12PEYA9Mni93Y2EQD02lKdOgT4Mzk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="sJbwZ/l8"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of 3zgSlZwQKCK8UkSaVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--fvdl.bounces.google.com designates 209.85.214.201 as permitted sender) smtp.mailfrom=3zgSlZwQKCK8UkSaVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--fvdl.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867920; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HzKNFokGTQRMQwkaxAdlaexKlVOh5cMMgkZKyufFdYg=; b=FcIp/trw6dqz5MMqCKb1CcE6YdHS7jBvPWuQcR4S7Gn4aQnPbYcz40Zjld0nOw485qF1Sy 0wkyQCgjygtNqMa/68b+4x1BQPv36gizLwxmEFrnA/RGTld/5XUrKkKB83XJDF7HRVomE+ mhr0ZGwZ4TrD53PTnif9S72QCGemTGE= Received: by mail-pl1-f201.google.com with SMTP id d9443c01a7336-216430a88b0so27719895ad.0 for ; Thu, 06 Feb 2025 10:51:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867919; x=1739472719; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=HzKNFokGTQRMQwkaxAdlaexKlVOh5cMMgkZKyufFdYg=; b=sJbwZ/l87/O/S58T20i2GhdExtl/JKQHS2cgasD1cJiDelPuyiH4DiLBlaf3N0l+hv nEGHqvT7LvZfzlBIQLthE2b5M1+9ISkQ6O7nelEn6lShsvVN5Cj6CaPGnbR6U0b5HaPH G+kcHwVHMFHckm28VTHS1zIAblIrj8+Rwii0V1vX6z8wJEbKP7Fa7oQNlrhpl2jKWhk9 kptk+ZJUnhRD11FSCuDqWqu/3LfxvyfRIZvDQHyf7w44q21BTKGV/hxbEWwUBs5N2Ofy 7E/EmAF0voB2y1BC3PorsGmxhs20LPwSIAanzF9/ffEsWITkQejY6UXY8581RFvI1gQc l8Bg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867919; x=1739472719; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=HzKNFokGTQRMQwkaxAdlaexKlVOh5cMMgkZKyufFdYg=; b=aPIZNGFIAX9LLznegJHOq+AAjA1W1c+cjnVKgbctYkgs+z1sJKHoD8EL5UTfHFR4Dz +Gv7+CkEOH+CK6wf7NgE3Vv2NCqGWu8oQxCujU3RurYwW/dGwScm2M0hxLYWJCUOZdnx GqBtVMmjLJ6FdGb0TXJSf9KVCc5w6Zgy+NEvZ6vOsw6uJ3K6Y9m40YoLoHfFeqxMWgO2 BZhxQnOWir0OGLeAeKf9NcvLUdw/LwxFNLt+k2VjwW+FghJbtxxmjvgGLPHXmDXSjxg/ H96pJCWln28jgK+Z+epzzy0iglnbX2bjR5BszrrzQxQJ8C/QOteQR61hwltaEhKljmRu bCPQ== X-Forwarded-Encrypted: i=1; AJvYcCVnael8hblXq0DPs8c+jA7nQGCRr2Z5qFFcCE9eiM4iKSuI2/ja073Ec3gJilP04JqOyxrZXC0IaA==@kvack.org X-Gm-Message-State: AOJu0YwD7g1X4fYf+BnunwDZkFSixM/KKhW2Arr1GNXGJK1De/KkKAAq N6VJBAZGJ/neIqnyMVQNHzS+zU/28C3KPUxOEuj2yWHe9SC8Cvp/cfYT/TJ+K4jD1veRjg== X-Google-Smtp-Source: AGHT+IE7WeCaaWwdtjLGXnFjJ3HV2HveYH8iYSHnW1lYFmfFYzgFw1cgzb7HpN3mgbnk50S1qEYYDK+c X-Received: from pfblg21.prod.google.com ([2002:a05:6a00:7095:b0:730:46e0:7ad0]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:9f8b:b0:1db:c20f:2c4d with SMTP id adf61e73a8af0-1ee03a22c01mr874861637.2.1738867918849; Thu, 06 Feb 2025 10:51:58 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:06 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-27-fvdl@google.com> Subject: [PATCH v3 26/28] mm/hugetlb: add hugetlb_cma_only cmdline option From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: EA0A280014 X-Stat-Signature: n7mdrmrphycguh3i31o6jxbkgn4sc3y1 X-Rspam-User: X-HE-Tag: 1738867919-335029 X-HE-Meta: U2FsdGVkX1+y/Zr3r5nmnMuAYv99htPYkiiLhVqEV+LNysoxpV2zuY9bljLyXhHmrpRD2A3urTbzovoPlUPp9N786KfqcPna/NyDP+u9D0jQn3cazHgLSGimYXeMXI/Ufyi4UMmVRxMZefTtCdlvOlz/jr6TDXc/PbIbrI637X8cePwLOAgxs9sVlqeOrDWY9XvNRxbpLNKYqDYoN8QEz/RVm/Hfbzapsz9Y14TYCaZAKHn7mKvCKd3LXkdU06iL10x5SVqQ3GtY6VkGnohtd1C1dE29I6lVOgaC+yBBxuHLbaOFiGNikwRsF+x6BzwL9gNvLIRiKbztSc7bMqS9KCafYtAAxQP45nLtD140CLsRbPQIFGMPlp080ZSzFc+uwWQ3ZdgHX4nNW7KsnWiQ4TRMPkYHB3XGoikqb1K5Epvu7pHiBdoBbvaqbPwhoTePCMWeZh4CKaGVLtatU6P4gm+z9C3Uno9aYViVtNpW/05lMIJ1fZx53/pKs4+gNUZ4aEG4gTNSi5c+pM7bBoiasfHs/EhFAqtPsna0F023gOmAqOrta3StJeKKqrwcFQsPTDFP2WbxAP4DLacy3YMWx2qQ36QWgo5KYVLanaQ6wZb/ibblrad/tyIQ3usH/S9cwyr2GBU81lXb8+LfkhoGtBxHv8ZFlY/v0JjD3TQfodxXKSaLreOfyBJjyROVbBvPnRvcYLHwM9k8nUnkGXu8sFjiyMBpNvxXGC8BQJz7w2K+88VcGU/vEZrxBGa3kJOQ7npAxh1Qxp02zuEipOkcSavQmT9+pn9g63iqidIXRWAY4Nmu9uwJ6NbAYoUIbjPJ0qkB4hlGyBxLr3Mkds8giG0p7zSw7ens9kJnulSZESmGpdGT9qN6ILJvW5LnIMpbHYIfPJkRIzQmMhE/iRgEnppXLs9XQ1WSnYV6ifdpXHMD7UXjKPQcii443+4EeJmw3ZqVQJ8kTuKk3BKYCw2 JbpvB/3i v5dc9Ixl/zg270H6vWvm+UiilcrOvggddzbAlaMoGudPujYBvjF1003Xq8MWoCbK1yBWINhESYVjFcVsRu2obnU7OW4/l8lPUpJ26/bjQ5CJKmw4zZdfmY9VWcpd1QL8pQaH7AwRDAQD9yXK7R56vPOrjKNdWHIEPVR6qWFsZ1SUrfNmjE5GIjsPN3khN/k6Mi6BmcN39Gm0cilunvOsJNh5g/ZoTxxhCskTIPXAtGxMdXwNRaA18/d817zJe4KEpbJc8gAKyDC0cUu2McbOAWIdTBZ63Sw2QPnRB/JgjXoKQcuCucXCeOmg2lT7UpUOLvySgAP/FiK0uecRVQsohHAhmCeFsJd/zSnEIE+TPcBbeQDXAInV08m3PuseUyoE/MBVrW9xnwi3bC+Ry0BG8iqtGYZu/J13amKRC/bwyTSeN8RYpm19zuZ3nVrscJJtJOrvjstGQVO3iV0YO3kRjqt546XBx7RJelqxQndgwXgHYrwx8DUpK8frDeciS/8kIDvOrk6nrnEdBUGG9RG0hVj4iiBaCkuh6P3RRGdNd3D2pCfNQfiE/IqQJZROBuMmlcq0Y X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add an option to force hugetlb gigantic pages to be allocated using CMA only (if hugetlb_cma is enabled). This avoids a fallback to allocation from the rest of system memory if the CMA allocation fails. This makes the size of hugetlb_cma a hard upper boundary for gigantic hugetlb page allocations. This is useful because, with a large CMA area, the kernel's unmovable allocations will have less room to work with and it is undesirable for new hugetlb gigantic page allocations to be done from that remaining area. It will eat in to the space available for unmovable allocations, leading to unwanted system behavior (OOMs because the kernel fails to do unmovable allocations). So, with this enabled, an administrator can force a hard upper bound for runtime gigantic page allocations, and have more predictable system behavior. Signed-off-by: Frank van der Linden --- Documentation/admin-guide/kernel-parameters.txt | 7 +++++++ mm/hugetlb.c | 14 ++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index fb8752b42ec8..eb56b251ce10 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -1892,6 +1892,13 @@ hugepages using the CMA allocator. If enabled, the boot-time allocation of gigantic hugepages is skipped. + hugetlb_cma_only= + [HW,CMA,EARLY] When allocating new HugeTLB pages, only + try to allocate from the CMA areas. + + This option does nothing if hugetlb_cma= is not also + specified. + hugetlb_free_vmemmap= [KNL] Requires CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP enabled. diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 29b3a6e70a53..24a06d6d3617 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -59,6 +59,7 @@ struct hstate hstates[HUGE_MAX_HSTATE]; static struct cma *hugetlb_cma[MAX_NUMNODES]; static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; #endif +static bool hugetlb_cma_only; static unsigned long hugetlb_cma_size __initdata; __initdata struct list_head huge_boot_pages[MAX_NUMNODES]; @@ -1510,6 +1511,9 @@ static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, } #endif if (!folio) { + if (hugetlb_cma_only) + return NULL; + folio = folio_alloc_gigantic(order, gfp_mask, nid, nodemask); if (!folio) return NULL; @@ -4731,6 +4735,9 @@ static __init void hugetlb_parse_params(void) hcp->setup(hcp->val); } + + if (!hugetlb_cma_size) + hugetlb_cma_only = false; } /* @@ -7843,6 +7850,13 @@ static int __init cmdline_parse_hugetlb_cma(char *p) early_param("hugetlb_cma", cmdline_parse_hugetlb_cma); +static int __init cmdline_parse_hugetlb_cma_only(char *p) +{ + return kstrtobool(p, &hugetlb_cma_only); +} + +early_param("hugetlb_cma_only", cmdline_parse_hugetlb_cma_only); + void __init hugetlb_cma_reserve(int order) { unsigned long size, reserved, per_node; From patchwork Thu Feb 6 18:51:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B27D5C02199 for ; Thu, 6 Feb 2025 18:52:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 01CF128001A; Thu, 6 Feb 2025 13:52:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EE3D4280002; Thu, 6 Feb 2025 13:52:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D370128001A; Thu, 6 Feb 2025 13:52:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A9A7C280002 for ; Thu, 6 Feb 2025 13:52:03 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6178B12149D for ; Thu, 6 Feb 2025 18:52:03 +0000 (UTC) X-FDA: 83090414526.29.789EB8D Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf28.hostedemail.com (Postfix) with ESMTP id 92902C0004 for ; Thu, 6 Feb 2025 18:52:01 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=i+45Cplx; spf=pass (imf28.hostedemail.com: domain of 30ASlZwQKCLEWmUcXffXcV.TfdcZelo-ddbmRTb.fiX@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=30ASlZwQKCLEWmUcXffXcV.TfdcZelo-ddbmRTb.fiX@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867921; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9FHf7X3ntolDh58znF+Bk2cZ6pAkHciPEwA7XP/h9Ew=; b=JH6m8Qn4lMC/v6dkafcAXpYWv+1BnVb7IT/9Xi4XJ7b/8TAjplkrGPgeCsJKHhPvU7G4Bu bUhfpz5OgiRo8Ta0cOSrvWjVvePq5W5edeYr9K7wCzRyr/8sWnvEMaGeHizLZBQrObABg/ z938IiNUje2AxXEMD0AoLt8qOCdAWv4= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=i+45Cplx; spf=pass (imf28.hostedemail.com: domain of 30ASlZwQKCLEWmUcXffXcV.TfdcZelo-ddbmRTb.fiX@flex--fvdl.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=30ASlZwQKCLEWmUcXffXcV.TfdcZelo-ddbmRTb.fiX@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867921; a=rsa-sha256; cv=none; b=RAbrWWkPsBqzXVm5NhOzzdnnSZ68CZjpsGOFVQceZ6UN7Va5kX2L/iFwBSeNd/7/TJ5ksC bfbxYLRAD4vxQnFZ/R6KFKwG95Euq8AsWIO+ay+N8ma5MGsWXZx5TKjvJzKsQbvM3qA7MY hKa7TFYtnOBVvglJKKvFTS5kPGysSlE= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef35de8901so2568924a91.3 for ; Thu, 06 Feb 2025 10:52:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867920; x=1739472720; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9FHf7X3ntolDh58znF+Bk2cZ6pAkHciPEwA7XP/h9Ew=; b=i+45Cplxh2jZfcmEgq/JsB3LvybZpgNQWZ1ykcT9p0qTK+DUegAIhOODJ8b4Vhd6Fj agQZZk/YAj8zBrAeAhyY+MXOmC9l8ubAw13Xv1a55JFx95fbGpH8QHe5lLm4IrsALz1X D3wfcDJLZ6qSn7e/U/CbnFQ3VjhtK6eQA0CcUbhA13UNex9xb4Gj07nekhPGE/ekzNAG 0tjcfJcZyEFTXDkFfQMz5bMrOFOyD88/TuCqrs4FfeZQODzZyx2Jnd4FEPI2jN7Rcwv8 doR3+44L8qiy8Di6irOmKFQr7ZgnMpDaMXxajo630oG5AIuOkZTRVIVY+kp+MvBH9Cmd 5lDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867920; x=1739472720; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9FHf7X3ntolDh58znF+Bk2cZ6pAkHciPEwA7XP/h9Ew=; b=r+mWCXa33A4GmupV7jNexf4jvmDKt0NtkSnIpbrSj9y3/GlXTNMeQDGnRFgnzhoHzI ztwv7X7PcW20zvU8gQkeM42g5Yf66vP3QcNoVrh9i0TLjLtbPYQmtC4xTQdceICHh3e5 eL1relJMzfL+Nu5Q6ngr4dD3DK2gm1U/FhVwz4DGCSb9YSey7SrxPA6GbvtHhPfj8Ux8 K4UPIjcHV/fbDR5JnQsSju6GBeUnT4oh23XAUKhXhH9FKY72oqmT/MjZ9FmM6XL7dyBZ YBrqCBhMiD+bqjjMaqIZlFtFh7CTWBkw4VpAjuUweMPM/Ur5YZsXg/cr68/uPAyvVl+C w5FQ== X-Forwarded-Encrypted: i=1; AJvYcCVVD+8n1Wf2oztbU5SeAKpvruc0c/axYWPlmv8rA+XJ/0ftEjWWfJ4yRLJjQ1/D/JwDcw6SWR0DXQ==@kvack.org X-Gm-Message-State: AOJu0YxXzQI4okoto+4lPuZHu47HIrVgHaXFzz52Lx//rk60fhiI1VRP w+HTm5qy/M9lrs1jGgSB9s1HkFfZc06dNxaQZkhxUcH5ykA2FppW90XjSTV0jXZ0bnIpPQ== X-Google-Smtp-Source: AGHT+IF0hEN8V7uU06C9tElOiOiDXvl7WJ62royjKbG+oprlqtroJ24zLedDhZAejMw2Fg4HCqLgipPv X-Received: from pfej4.prod.google.com ([2002:aa7:8d04:0:b0:725:c7de:e052]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:4615:b0:72d:35ed:214b with SMTP id d2e1a72fcca58-7305d515e41mr566746b3a.24.1738867920500; Thu, 06 Feb 2025 10:52:00 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:07 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-28-fvdl@google.com> Subject: [PATCH v3 27/28] mm/hugetlb: enable bootmem allocation from CMA areas From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden , Madhavan Srinivasan , Michael Ellerman , linuxppc-dev@lists.ozlabs.org X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 92902C0004 X-Stat-Signature: 44owc1t7q38axaa8gp16oy36qyemdynf X-HE-Tag: 1738867921-963032 X-HE-Meta: U2FsdGVkX1+NSrNTy7XECL3VzyxcSdjqS7HJw3wCvhMHP4g7dp4kpHNl8Nt0w/prCSns30E0MyqZLng4aUZIAK33J+0oPGnSDMDZGp19EqeqeyR8HlJaO0nShJEztCelpIbc/w0+odq7p8N2JD/+ReDjm8rrZ8URC/UkfD9r0QU9Ximhnwd1xDtBOAJntIKE9OQ7bnFmAa08V/PfHAJz8faP2BIDPlPzpNfRzh8p7V22NCQeRTvpCtYHiJSP4J2RWCcYZ+zqe2ugz/GBtj1bNsMiNyb87pO6VIyOxF3qw6sHxhHv9QoQzTHxyAULImIm4j4s3pAQNAQempXCMPRY6rmCi+LI1hE7qdrmnPDLsA9v8QonPAQPNYX/QbnaZqCeaPK49MfSWi6wI7JR5YUuKvcSyeh9mmmVQTmL0IWBuxcBEIhWxQysBF1+5EPgRwL8+Mg/tfS2U7p2B3kOEgBZ3jg2pQ3lSd8gRhKpZXXAAOKfj7gtgw6MCl9g9dwTNqWMh6eAVdjNaySBlKzTDlfDsrnNiI5Uq3jUXutNfGZxjKb5lj06GBNKTjONlKCC+Df9ROeo3IY/tlXJ4Rp6aWgwvtJ5PyehSn2e82OWxLtljgpdSUWW9BbAMutBMqqkbskZkcdMJ4pjN5JIkrBeEJi1QNb0wWAWH57J/g22vuPgK66KabRHJ8fjiNrmSLJf0UUCQMXqDk9bRcIBhSx2b8f0AOLD60hLp+UzVyFggudjL/IYc0HgAuIqZidZHr4qItpmNo1hZwVn1PivQCx+VgnHCWScFg50bce5GkR+9+j+oGVV+mJZpru0PPfYcJKZOxqKbJuQK+Hc+v4BlossUnJbwfOILy9DQMWCZQqCHSimlTAJuO4q156kpdz+7CpDXiwm+73ggmPyWpIBuQHd+UolWYcucTfONE1utpMcQ0U/aY4TDRykbifaDT9spCPbRxsFJLMqBm1F2z6S1Q5hd3s W1l1gZIJ k3ASUxold9dfUJ+2fnCtowesAjyqGyBJxSrQF2mcIlKfBvkb9iIGkze7fJLw0OZW+D2Vfffgi2VhWY/h4FLOhYZgLW0MD99FKlAeujcKak95FgXGGwuJ4M/k7JvZROlZy2Ic7dGTSkBFFG7UiTmRs6aOvWqLdI5oglFT9VvOJYB+jKtbDPJqciQsYQdKiisut18vAGdElPrP62TLcJh7jrXWpHaG4ycAyDMXztNCsAVie+gUixwd+FobQMND2LoMZ5cY7LIF3XfXD52jDhTn4nEiKKpqLeed0AG6RdTywzfsn2anJrRj5yNgYxbsB4gHVrwoffmNTZg742p38jzzXQ09sakp9GgACD+MG7P+xzVPFIwEW4UFlN/h2r1HOGd6lX7BMJBZ+ZkAsHM1tWTPI/H5qzzGvWiwMctINoPUYqHlpVDJ5gkEc9/hmNyUWrZR1jDIEG5T61zfhJYJ8FrFuO+Cp7ST1/c7pJKRlAuKlicsjAP8f4PjghNuJyCoaNkrYSHzZZshLgRqwXzItziLVn24/1BHvF4kbBBJTTL6RYkNj03AFs1gkagT1RBj/chSyi5UGHdUnt0TP7c+o4X9ok6n+50rH7XVse9p5H9j5qoDbzbs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If hugetlb_cma_only is enabled, we know that hugetlb pages can only be allocated from CMA. Now that there is an interface to do early reservations from a CMA area (returning memblock memory), it can be used to allocate hugetlb pages from CMA. This also allows for doing pre-HVO on these pages (if enabled). Make sure to initialize the page structures and associated data correctly. Create a flag to signal that a hugetlb page has been allocated from CMA to make things a little easier. Some configurations of powerpc have a special hugetlb bootmem allocator, so introduce a boolean arch_specific_huge_bootmem_alloc that returns true if such an allocator is present. In that case, CMA bootmem allocations can't be used, so check that function before trying. Cc: Madhavan Srinivasan Cc: Michael Ellerman Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Frank van der Linden --- arch/powerpc/include/asm/book3s/64/hugetlb.h | 6 + include/linux/hugetlb.h | 17 +++ mm/hugetlb.c | 121 ++++++++++++++----- 3 files changed, 113 insertions(+), 31 deletions(-) diff --git a/arch/powerpc/include/asm/book3s/64/hugetlb.h b/arch/powerpc/include/asm/book3s/64/hugetlb.h index f0bba9c5f9c3..bb786694dd26 100644 --- a/arch/powerpc/include/asm/book3s/64/hugetlb.h +++ b/arch/powerpc/include/asm/book3s/64/hugetlb.h @@ -94,4 +94,10 @@ static inline int check_and_get_huge_psize(int shift) return mmu_psize; } +#define arch_has_huge_bootmem_alloc arch_has_huge_bootmem_alloc + +static inline bool arch_has_huge_bootmem_alloc(void) +{ + return (firmware_has_feature(FW_FEATURE_LPAR) && !radix_enabled()); +} #endif diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 2512463bca49..6c6546b54934 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -591,6 +591,7 @@ enum hugetlb_page_flags { HPG_freed, HPG_vmemmap_optimized, HPG_raw_hwp_unreliable, + HPG_cma, __NR_HPAGEFLAGS, }; @@ -650,6 +651,7 @@ HPAGEFLAG(Temporary, temporary) HPAGEFLAG(Freed, freed) HPAGEFLAG(VmemmapOptimized, vmemmap_optimized) HPAGEFLAG(RawHwpUnreliable, raw_hwp_unreliable) +HPAGEFLAG(Cma, cma) #ifdef CONFIG_HUGETLB_PAGE @@ -678,14 +680,18 @@ struct hstate { char name[HSTATE_NAME_LEN]; }; +struct cma; + struct huge_bootmem_page { struct list_head list; struct hstate *hstate; unsigned long flags; + struct cma *cma; }; #define HUGE_BOOTMEM_HVO 0x0001 #define HUGE_BOOTMEM_ZONES_VALID 0x0002 +#define HUGE_BOOTMEM_CMA 0x0004 bool hugetlb_bootmem_page_zones_valid(int nid, struct huge_bootmem_page *m); @@ -823,6 +829,17 @@ static inline pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, } #endif +#ifndef arch_has_huge_bootmem_alloc +/* + * Some architectures do their own bootmem allocation, so they can't use + * early CMA allocation. + */ +static inline bool arch_has_huge_bootmem_alloc(void) +{ + return false; +} +#endif + static inline struct hstate *folio_hstate(struct folio *folio) { VM_BUG_ON_FOLIO(!folio_test_hugetlb(folio), folio); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 24a06d6d3617..737ffd223d45 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -131,8 +131,10 @@ static void hugetlb_free_folio(struct folio *folio) #ifdef CONFIG_CMA int nid = folio_nid(folio); - if (cma_free_folio(hugetlb_cma[nid], folio)) + if (folio_test_hugetlb_cma(folio)) { + WARN_ON_ONCE(!cma_free_folio(hugetlb_cma[nid], folio)); return; + } #endif folio_put(folio); } @@ -1508,6 +1510,9 @@ static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, break; } } + + if (folio) + folio_set_hugetlb_cma(folio); } #endif if (!folio) { @@ -3174,6 +3179,53 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, return ERR_PTR(-ENOSPC); } +static bool __init hugetlb_early_cma(struct hstate *h) +{ + if (arch_has_huge_bootmem_alloc()) + return false; + + return (hstate_is_gigantic(h) && hugetlb_cma_only); +} + +static __init void *alloc_bootmem(struct hstate *h, int nid) +{ + struct huge_bootmem_page *m; + unsigned long flags; + struct cma *cma; + +#ifdef CONFIG_CMA + if (hugetlb_early_cma(h)) { + flags = HUGE_BOOTMEM_CMA; + cma = hugetlb_cma[nid]; + m = cma_reserve_early(cma, huge_page_size(h)); + } else +#endif + { + flags = 0; + cma = NULL; + m = memblock_alloc_try_nid_raw(huge_page_size(h), + huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); + } + + if (m) { + /* + * Use the beginning of the huge page to store the + * huge_bootmem_page struct (until gather_bootmem + * puts them into the mem_map). + * + * Put them into a private list first because mem_map + * is not up yet. + */ + INIT_LIST_HEAD(&m->list); + list_add(&m->list, &huge_boot_pages[nid]); + m->hstate = h; + m->flags = flags; + m->cma = cma; + } + + return m; +} + int alloc_bootmem_huge_page(struct hstate *h, int nid) __attribute__ ((weak, alias("__alloc_bootmem_huge_page"))); int __alloc_bootmem_huge_page(struct hstate *h, int nid) @@ -3183,17 +3235,14 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) /* do node specific alloc */ if (nid != NUMA_NO_NODE) { - m = memblock_alloc_try_nid_raw(huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); + m = alloc_bootmem(h, node); if (!m) return 0; goto found; } /* allocate from next node when distributing huge pages */ for_each_node_mask_to_alloc(&h->next_nid_to_alloc, nr_nodes, node, &node_states[N_ONLINE]) { - m = memblock_alloc_try_nid_raw( - huge_page_size(h), huge_page_size(h), - 0, MEMBLOCK_ALLOC_ACCESSIBLE, node); + m = alloc_bootmem(h, node); if (m) break; } @@ -3202,7 +3251,6 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) return 0; found: - /* * Only initialize the head struct page in memmap_init_reserved_pages, * rest of the struct pages will be initialized by the HugeTLB @@ -3212,18 +3260,6 @@ int __alloc_bootmem_huge_page(struct hstate *h, int nid) */ memblock_reserved_mark_noinit(virt_to_phys((void *)m + PAGE_SIZE), huge_page_size(h) - PAGE_SIZE); - /* - * Use the beginning of the huge page to store the - * huge_bootmem_page struct (until gather_bootmem - * puts them into the mem_map). - * - * Put them into a private list first because mem_map - * is not up yet. - */ - INIT_LIST_HEAD(&m->list); - list_add(&m->list, &huge_boot_pages[node]); - m->hstate = h; - m->flags = 0; return 1; } @@ -3264,13 +3300,25 @@ static void __init hugetlb_folio_init_vmemmap(struct folio *folio, prep_compound_head((struct page *)folio, huge_page_order(h)); } +static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) +{ + return m->flags & HUGE_BOOTMEM_HVO; +} + +static bool __init hugetlb_bootmem_page_earlycma(struct huge_bootmem_page *m) +{ + return m->flags & HUGE_BOOTMEM_CMA; +} + /* * memblock-allocated pageblocks might not have the migrate type set * if marked with the 'noinit' flag. Set it to the default (MIGRATE_MOVABLE) - * here. + * here, or MIGRATE_CMA if this was a page allocated through an early CMA + * reservation. * - * Note that this will not write the page struct, it is ok (and necessary) - * to do this on vmemmap optimized folios. + * In case of vmemmap optimized folios, the tail vmemmap pages are mapped + * read-only, but that's ok - for sparse vmemmap this does not write to + * the page structure. */ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, struct hstate *h) @@ -3279,9 +3327,13 @@ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio, WARN_ON_ONCE(!pageblock_aligned(folio_pfn(folio))); - for (i = 0; i < nr_pages; i += pageblock_nr_pages) - set_pageblock_migratetype(folio_page(folio, i), + for (i = 0; i < nr_pages; i += pageblock_nr_pages) { + if (folio_test_hugetlb_cma(folio)) + init_cma_pageblock(folio_page(folio, i)); + else + set_pageblock_migratetype(folio_page(folio, i), MIGRATE_MOVABLE); + } } static void __init prep_and_add_bootmem_folios(struct hstate *h, @@ -3327,10 +3379,16 @@ bool __init hugetlb_bootmem_page_zones_valid(int nid, return true; } + if (hugetlb_bootmem_page_earlycma(m)) { + valid = cma_validate_zones(m->cma); + goto out; + } + start_pfn = virt_to_phys(m) >> PAGE_SHIFT; valid = !pfn_range_intersects_zones(nid, start_pfn, pages_per_huge_page(m->hstate)); +out: if (!valid) hstate_boot_nrinvalid[hstate_index(m->hstate)]++; @@ -3359,11 +3417,6 @@ static void __init hugetlb_bootmem_free_invalid_page(int nid, struct page *page, } } -static bool __init hugetlb_bootmem_page_prehvo(struct huge_bootmem_page *m) -{ - return (m->flags & HUGE_BOOTMEM_HVO); -} - /* * Put bootmem huge pages into the standard lists after mem_map is up. * Note: This only applies to gigantic (order > MAX_PAGE_ORDER) pages. @@ -3413,6 +3466,9 @@ static void __init gather_bootmem_prealloc_node(unsigned long nid) */ folio_set_hugetlb_vmemmap_optimized(folio); + if (hugetlb_bootmem_page_earlycma(m)) + folio_set_hugetlb_cma(folio); + list_add(&folio->lru, &folio_list); /* @@ -3605,8 +3661,11 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) { unsigned long allocated; - /* skip gigantic hugepages allocation if hugetlb_cma enabled */ - if (hstate_is_gigantic(h) && hugetlb_cma_size) { + /* + * Skip gigantic hugepages allocation if early CMA + * reservations are not available. + */ + if (hstate_is_gigantic(h) && hugetlb_cma_size && !hugetlb_early_cma(h)) { pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n"); return; } From patchwork Thu Feb 6 18:51:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Frank van der Linden X-Patchwork-Id: 13963575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1745C02194 for ; Thu, 6 Feb 2025 18:52:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB38D28001B; Thu, 6 Feb 2025 13:52:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3C78280002; Thu, 6 Feb 2025 13:52:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A62928001B; Thu, 6 Feb 2025 13:52:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 5FF55280002 for ; Thu, 6 Feb 2025 13:52:05 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1ADCE121463 for ; Thu, 6 Feb 2025 18:52:05 +0000 (UTC) X-FDA: 83090414610.18.CCC48E5 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf04.hostedemail.com (Postfix) with ESMTP id 418E340006 for ; Thu, 6 Feb 2025 18:52:03 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="nNQOeI/U"; spf=pass (imf04.hostedemail.com: domain of 30gSlZwQKCLMYoWeZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=30gSlZwQKCLMYoWeZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738867923; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RA4qMEgFAFc3FubUWoNTLw2qbOjPU8qUi4LYfmqlluQ=; b=NhEmadQ7bOW8gCDV+MU0NrVeYZAYX5jGN3cvCUvM7sndKa+yLF727hjHkQcS2HPhMaMj5m cDqAzyJ9GoKBr3RYWeuPML1YmYHt3RmV1uDb7GsMz5Rxr26cbEpaSkd/aUmBqDCN+BAE/j CdMWLKn8NrhCY93K1g4/d2EYVUcG7cY= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="nNQOeI/U"; spf=pass (imf04.hostedemail.com: domain of 30gSlZwQKCLMYoWeZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--fvdl.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=30gSlZwQKCLMYoWeZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--fvdl.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738867923; a=rsa-sha256; cv=none; b=n8cSboFMdU/WQZhv9QyGZvRw8YDydQ/u9uVDV8JZ9lFLCKCsPCszmg5fWTMfP7k3AfAHK6 2evb7e3TiUIwfh+ZkGu06+0HtYCpnkC6p5AXsLbDDUKiMw/a/tjlkn7Ir5lH8AhgCa96ae EX9r0S2p3HXTXQIVUTjBg4Mee2le5vE= Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2f9f42d98e3so2528074a91.0 for ; Thu, 06 Feb 2025 10:52:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1738867922; x=1739472722; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RA4qMEgFAFc3FubUWoNTLw2qbOjPU8qUi4LYfmqlluQ=; b=nNQOeI/Umoh9Qk/rmiidKXh1C/s1e8IzTRmdmT+K0DM4SVtHtv139JwC/m11jfU1im k7eIfIinKXcJ1VjmfVYd5qfimcMS8ArzlgZGvFF62nbft589sn5pKKlGwmdeLsdjqo3C q0Xyn1Ioag3H9LynfFq5AtuvNtnYTamKC4e7ebQLtbNfu3HKoIeUFu6Qvxha0N7WOLnO 2VPDXFiiZnHAkrSv8P/szkFpUb5BLp9QKQjj2Iq9pRapyCDFPcjmTKksgoMF05vWTw5Y a3LweETo0xNipMZc9pYY2VEy5QIlK4cIv0gbUynb71Z+/Vjfa9Iea+n6x0AegbLdTcl9 lkzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738867922; x=1739472722; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RA4qMEgFAFc3FubUWoNTLw2qbOjPU8qUi4LYfmqlluQ=; b=lvAdjkvQko4eYfEHt+FxDiOLhSUfjKhRDBb5YJ8T/g3yo8UrtYI2Qu+M0G4cW0kn0S 2kfmd5g9uX0dwLX5PkKstN6PAdW+h9awKLUOppWt6WzO3PHnpzwcywnOVKZIL4XXXCCn Xi+wbwh6ftMUZM9AedzfptU4ogFIA9Vb5/9Y/XVosMvJBjxeK3H0SUC5osiu3DemFLzx jsyazw6UNelnskTSnUBwN8nnbQgW8q/0/VX7jKsFU7/6IktrtnL5Bs9AZyZ29dLj9brf 1W8ILCyWA41TcYN3vAVxNAjmCH8vBcq22Yl+nggXmvfNh+Fij97e88mFxAPI4pf6pLp/ //Qw== X-Forwarded-Encrypted: i=1; AJvYcCUCvo637P5AD1er/0HUwd+6+ookeA4lYIdfkD36Vz7jgkllvFycYrdjLw0K1acbUWOUNAxOnkwT8w==@kvack.org X-Gm-Message-State: AOJu0YwL4OCq2uLe8DMcqAYSvnjFooTevQAjk0uUCUZsB7wVwrkOKq45 FixmVC4Gt04AqZBi6FymMYnBB4h+QxLsa5oalbJuqu0Sou3NVbryx8QIZquX4arkvg8Kig== X-Google-Smtp-Source: AGHT+IFe94pSmQByeR+90nxbijLeSbDXRsGX0IDwX7sg28d/h13OpEfzDDRzoANf9kA7MGn2kI9lVpbH X-Received: from pfiy10.prod.google.com ([2002:a05:6a00:190a:b0:725:df7a:f4e3]) (user=fvdl job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a00:2289:b0:727:3fd5:b530 with SMTP id d2e1a72fcca58-7305d4f01f2mr468389b3a.15.1738867922247; Thu, 06 Feb 2025 10:52:02 -0800 (PST) Date: Thu, 6 Feb 2025 18:51:08 +0000 In-Reply-To: <20250206185109.1210657-1-fvdl@google.com> Mime-Version: 1.0 References: <20250206185109.1210657-1-fvdl@google.com> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog Message-ID: <20250206185109.1210657-29-fvdl@google.com> Subject: [PATCH v3 28/28] mm/hugetlb: move hugetlb CMA code in to its own file From: Frank van der Linden To: akpm@linux-foundation.org, muchun.song@linux.dev, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: yuzhao@google.com, usamaarif642@gmail.com, joao.m.martins@oracle.com, roman.gushchin@linux.dev, Frank van der Linden X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 418E340006 X-Stat-Signature: czsfp6twyb5jqdbewuixx13f9s8q53yq X-Rspam-User: X-HE-Tag: 1738867923-744827 X-HE-Meta: U2FsdGVkX19nU6QtcZKbyB5GhDsHs+RoRktVO0CtU+24T5eHM3yiidYCV6jfF9u8Bf1+RQWgK+Fx7RfxyF9rV1rXLca9rzEyyDx6GDgUwZCVirr9faMmjqOHwgONyV3dQV4x+ssnfetpGNbg3BxYwTlNdA25mco2M3P7KaoeQ0YHSgNhSnlRVzzLDF91U4OxG78v7ecUH4aW14YnFNglR10m+vmF/7B+J2LC5rhvTI0KdmYpw3Kr6Xc9EP3duV3IAmnOgaQh9jAdE8RNj/3xzJQKqA0TMiGa6n2/NpdijS3N7BlvMXx+fMaRRc5Z0r3ch+FxsS3953QRWJlyh6s/4nxii4ccWuYs9r9XGiLLJNBdEqD462qKOXfTysr17fJ/lDgiRTMMcMZNTIhPeSRCOoWp2ysSE9pLquO9LzcgPDjkuxlvI7HJXtzzrwTbO2/0NabhkttqEqfujIemLY+UOCaL4HeIwwoM7thvBYdlxYjOYvwcHnhENfQCgr4iU/D5NBjjKEcAYifqei9O4qssaTJdKaJ7HzcUU7EvtrCdWelbJGBJ8WnQzaAyDfCM76fIKMooLMVQvoDHl414LaxSjAgjHOaN8w8VOtTCzyKeticgKfOnKclr34xktlq2akJMGGeEm/ROiZBukk2b07luscOAnvMi68K2Wu2AD6MVMIXp584Jg6cgR7QO8mm05xmuljtctDLkCN3eWrjQYty5osjbbnax05FP8L3IfCIoOV+asjveDpE7ly1JTsQ40c3St7+MjIWxwnOGkfWOl9RyiiQxV+TxCGvXYYCvSEhGGOuifWmiRO0KvQcozKeRAH36QcqAxneDNVMk25qd6jyfx/SWpXmA7lB0YUse52q0l8kU5NH2DG7cM0valLuYTEJhf5NVRnXV3kHioLKqbg07xAYitbuA9HJ3FLHU099RwSvHWQi+aWvaCrIFvGx0s/ILz8s3VPa0OxqmbyZ91kd swENTKcU 1HiLMpwiZ618rxEtR/18Y6/LgIwlsiSmL7HTC7VSG6BKI9MymYBWM/3bH0jqk9EPh4Y/sJVW2x8wBUfK/vb7NfJ9Dk9KyuqUD+FxABkAEAh7EEmyiTe3CV+yPT3BC/NtKckxB2kc/Yn9VnrScR6JCofW6YvsvLjOY+Nark6xCdIuSgBvUx2DCVKVnPoix6qPyqCHS5cqrx2LeFdKdpyfWwxtKmngUneldJIf0RnGLCd5h1x5FH79ZwQ73Qb2BdYi0yGupdQ+C6ts9Vvh9oBQw5aiWlcfI/HFmuoH28VNHkfzFdOJ92QDGv4YujYFO2Qt0dON+LSAYBKpgKsV2njsAn/c6Y+n8gmssCtI99xzD8VBM0cxcxL48SuWo6HuL8RMv0evwOilCNvdjP97ybkyUdqdfS6J3sQ+MOmxSKG7Wtpieim2DUL9gd//MA9XfcD/ytSlOzGmv5fJkf0A3wRR7lUneNA0/hxDeB0KZytm+u6PV6CIf69p/bjx1B0TG3P2U9HAuH/vA+cS9k9pV2XtHRyotUQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: hugetlb.c contained a number of CONFIG_CMA ifdefs, and the code inside them was large enough to merit being in its own file, so move it, cleaning up things a bit. Hide some direct variable access behind functions to accommodate the move. No functional change intended. Signed-off-by: Frank van der Linden --- MAINTAINERS | 2 + mm/Makefile | 3 + mm/hugetlb.c | 252 +++------------------------------------------ mm/hugetlb_cma.c | 258 +++++++++++++++++++++++++++++++++++++++++++++++ mm/hugetlb_cma.h | 55 ++++++++++ 5 files changed, 334 insertions(+), 236 deletions(-) create mode 100644 mm/hugetlb_cma.c create mode 100644 mm/hugetlb_cma.h diff --git a/MAINTAINERS b/MAINTAINERS index 896a307fa065..e72a1f247b80 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -10708,6 +10708,8 @@ F: fs/hugetlbfs/ F: include/linux/hugetlb.h F: include/trace/events/hugetlbfs.h F: mm/hugetlb.c +F: mm/hugetlb_cma.c +F: mm/hugetlb_cma.h F: mm/hugetlb_vmemmap.c F: mm/hugetlb_vmemmap.h F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c diff --git a/mm/Makefile b/mm/Makefile index 850386a67b3e..810ccd45d270 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -79,6 +79,9 @@ obj-$(CONFIG_SWAP) += page_io.o swap_state.o swapfile.o swap_slots.o obj-$(CONFIG_ZSWAP) += zswap.o obj-$(CONFIG_HAS_DMA) += dmapool.o obj-$(CONFIG_HUGETLBFS) += hugetlb.o +ifdef CONFIG_CMA +obj-$(CONFIG_HUGETLBFS) += hugetlb_cma.o +endif obj-$(CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP) += hugetlb_vmemmap.o obj-$(CONFIG_NUMA) += mempolicy.o obj-$(CONFIG_SPARSEMEM) += sparse.o diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 737ffd223d45..30bc34d43fbf 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -49,19 +49,13 @@ #include #include "internal.h" #include "hugetlb_vmemmap.h" +#include "hugetlb_cma.h" #include int hugetlb_max_hstate __read_mostly; unsigned int default_hstate_idx; struct hstate hstates[HUGE_MAX_HSTATE]; -#ifdef CONFIG_CMA -static struct cma *hugetlb_cma[MAX_NUMNODES]; -static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; -#endif -static bool hugetlb_cma_only; -static unsigned long hugetlb_cma_size __initdata; - __initdata struct list_head huge_boot_pages[MAX_NUMNODES]; static unsigned long hstate_boot_nrinvalid[HUGE_MAX_HSTATE] __initdata; @@ -128,14 +122,11 @@ static struct resv_map *vma_resv_map(struct vm_area_struct *vma); static void hugetlb_free_folio(struct folio *folio) { -#ifdef CONFIG_CMA - int nid = folio_nid(folio); - if (folio_test_hugetlb_cma(folio)) { - WARN_ON_ONCE(!cma_free_folio(hugetlb_cma[nid], folio)); + hugetlb_cma_free_folio(folio); return; } -#endif + folio_put(folio); } @@ -1492,31 +1483,9 @@ static struct folio *alloc_gigantic_folio(struct hstate *h, gfp_t gfp_mask, if (nid == NUMA_NO_NODE) nid = numa_mem_id(); retry: - folio = NULL; -#ifdef CONFIG_CMA - { - int node; - - if (hugetlb_cma[nid]) - folio = cma_alloc_folio(hugetlb_cma[nid], order, gfp_mask); - - if (!folio && !(gfp_mask & __GFP_THISNODE)) { - for_each_node_mask(node, *nodemask) { - if (node == nid || !hugetlb_cma[node]) - continue; - - folio = cma_alloc_folio(hugetlb_cma[node], order, gfp_mask); - if (folio) - break; - } - } - - if (folio) - folio_set_hugetlb_cma(folio); - } -#endif + folio = hugetlb_cma_alloc_folio(h, gfp_mask, nid, nodemask); if (!folio) { - if (hugetlb_cma_only) + if (hugetlb_cma_exclusive_alloc()) return NULL; folio = folio_alloc_gigantic(order, gfp_mask, nid, nodemask); @@ -3179,32 +3148,19 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, return ERR_PTR(-ENOSPC); } -static bool __init hugetlb_early_cma(struct hstate *h) -{ - if (arch_has_huge_bootmem_alloc()) - return false; - - return (hstate_is_gigantic(h) && hugetlb_cma_only); -} - static __init void *alloc_bootmem(struct hstate *h, int nid) { struct huge_bootmem_page *m; - unsigned long flags; - struct cma *cma; -#ifdef CONFIG_CMA - if (hugetlb_early_cma(h)) { - flags = HUGE_BOOTMEM_CMA; - cma = hugetlb_cma[nid]; - m = cma_reserve_early(cma, huge_page_size(h)); - } else -#endif - { - flags = 0; - cma = NULL; + if (hugetlb_early_cma(h)) + m = hugetlb_cma_alloc_bootmem(h, nid); + else { m = memblock_alloc_try_nid_raw(huge_page_size(h), huge_page_size(h), 0, MEMBLOCK_ALLOC_ACCESSIBLE, nid); + if (m) { + m->flags = 0; + m->cma = NULL; + } } if (m) { @@ -3219,8 +3175,6 @@ static __init void *alloc_bootmem(struct hstate *h, int nid) INIT_LIST_HEAD(&m->list); list_add(&m->list, &huge_boot_pages[nid]); m->hstate = h; - m->flags = flags; - m->cma = cma; } return m; @@ -3665,7 +3619,8 @@ static void __init hugetlb_hstate_alloc_pages(struct hstate *h) * Skip gigantic hugepages allocation if early CMA * reservations are not available. */ - if (hstate_is_gigantic(h) && hugetlb_cma_size && !hugetlb_early_cma(h)) { + if (hstate_is_gigantic(h) && hugetlb_cma_total_size() && + !hugetlb_early_cma(h)) { pr_warn_once("HugeTLB: hugetlb_cma is enabled, skip boot time allocation\n"); return; } @@ -3702,7 +3657,7 @@ static void __init hugetlb_init_hstates(void) */ if (hstate_is_gigantic(h) && !gigantic_page_runtime_supported()) continue; - if (hugetlb_cma_size && h->order <= HUGETLB_PAGE_ORDER) + if (hugetlb_cma_total_size() && h->order <= HUGETLB_PAGE_ORDER) continue; for_each_hstate(h2) { if (h2 == h) @@ -4604,14 +4559,6 @@ static void hugetlb_register_all_nodes(void) { } #endif -#ifdef CONFIG_CMA -static void __init hugetlb_cma_check(void); -#else -static inline __init void hugetlb_cma_check(void) -{ -} -#endif - static void __init hugetlb_sysfs_init(void) { struct hstate *h; @@ -4795,8 +4742,7 @@ static __init void hugetlb_parse_params(void) hcp->setup(hcp->val); } - if (!hugetlb_cma_size) - hugetlb_cma_only = false; + hugetlb_cma_validate_params(); } /* @@ -7866,169 +7812,3 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) hugetlb_unshare_pmds(vma, ALIGN(vma->vm_start, PUD_SIZE), ALIGN_DOWN(vma->vm_end, PUD_SIZE)); } - -#ifdef CONFIG_CMA -static bool cma_reserve_called __initdata; - -static int __init cmdline_parse_hugetlb_cma(char *p) -{ - int nid, count = 0; - unsigned long tmp; - char *s = p; - - while (*s) { - if (sscanf(s, "%lu%n", &tmp, &count) != 1) - break; - - if (s[count] == ':') { - if (tmp >= MAX_NUMNODES) - break; - nid = array_index_nospec(tmp, MAX_NUMNODES); - - s += count + 1; - tmp = memparse(s, &s); - hugetlb_cma_size_in_node[nid] = tmp; - hugetlb_cma_size += tmp; - - /* - * Skip the separator if have one, otherwise - * break the parsing. - */ - if (*s == ',') - s++; - else - break; - } else { - hugetlb_cma_size = memparse(p, &p); - break; - } - } - - return 0; -} - -early_param("hugetlb_cma", cmdline_parse_hugetlb_cma); - -static int __init cmdline_parse_hugetlb_cma_only(char *p) -{ - return kstrtobool(p, &hugetlb_cma_only); -} - -early_param("hugetlb_cma_only", cmdline_parse_hugetlb_cma_only); - -void __init hugetlb_cma_reserve(int order) -{ - unsigned long size, reserved, per_node; - bool node_specific_cma_alloc = false; - int nid; - - /* - * HugeTLB CMA reservation is required for gigantic - * huge pages which could not be allocated via the - * page allocator. Just warn if there is any change - * breaking this assumption. - */ - VM_WARN_ON(order <= MAX_PAGE_ORDER); - cma_reserve_called = true; - - if (!hugetlb_cma_size) - return; - - for (nid = 0; nid < MAX_NUMNODES; nid++) { - if (hugetlb_cma_size_in_node[nid] == 0) - continue; - - if (!node_online(nid)) { - pr_warn("hugetlb_cma: invalid node %d specified\n", nid); - hugetlb_cma_size -= hugetlb_cma_size_in_node[nid]; - hugetlb_cma_size_in_node[nid] = 0; - continue; - } - - if (hugetlb_cma_size_in_node[nid] < (PAGE_SIZE << order)) { - pr_warn("hugetlb_cma: cma area of node %d should be at least %lu MiB\n", - nid, (PAGE_SIZE << order) / SZ_1M); - hugetlb_cma_size -= hugetlb_cma_size_in_node[nid]; - hugetlb_cma_size_in_node[nid] = 0; - } else { - node_specific_cma_alloc = true; - } - } - - /* Validate the CMA size again in case some invalid nodes specified. */ - if (!hugetlb_cma_size) - return; - - if (hugetlb_cma_size < (PAGE_SIZE << order)) { - pr_warn("hugetlb_cma: cma area should be at least %lu MiB\n", - (PAGE_SIZE << order) / SZ_1M); - hugetlb_cma_size = 0; - return; - } - - if (!node_specific_cma_alloc) { - /* - * If 3 GB area is requested on a machine with 4 numa nodes, - * let's allocate 1 GB on first three nodes and ignore the last one. - */ - per_node = DIV_ROUND_UP(hugetlb_cma_size, nr_online_nodes); - pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n", - hugetlb_cma_size / SZ_1M, per_node / SZ_1M); - } - - reserved = 0; - for_each_online_node(nid) { - int res; - char name[CMA_MAX_NAME]; - - if (node_specific_cma_alloc) { - if (hugetlb_cma_size_in_node[nid] == 0) - continue; - - size = hugetlb_cma_size_in_node[nid]; - } else { - size = min(per_node, hugetlb_cma_size - reserved); - } - - size = round_up(size, PAGE_SIZE << order); - - snprintf(name, sizeof(name), "hugetlb%d", nid); - /* - * Note that 'order per bit' is based on smallest size that - * may be returned to CMA allocator in the case of - * huge page demotion. - */ - res = cma_declare_contiguous_multi(size, PAGE_SIZE << order, - HUGETLB_PAGE_ORDER, name, - &hugetlb_cma[nid], nid); - if (res) { - pr_warn("hugetlb_cma: reservation failed: err %d, node %d", - res, nid); - continue; - } - - reserved += size; - pr_info("hugetlb_cma: reserved %lu MiB on node %d\n", - size / SZ_1M, nid); - - if (reserved >= hugetlb_cma_size) - break; - } - - if (!reserved) - /* - * hugetlb_cma_size is used to determine if allocations from - * cma are possible. Set to zero if no cma regions are set up. - */ - hugetlb_cma_size = 0; -} - -static void __init hugetlb_cma_check(void) -{ - if (!hugetlb_cma_size || cma_reserve_called) - return; - - pr_warn("hugetlb_cma: the option isn't supported by current arch\n"); -} - -#endif /* CONFIG_CMA */ diff --git a/mm/hugetlb_cma.c b/mm/hugetlb_cma.c new file mode 100644 index 000000000000..3ea9cd0f6b9f --- /dev/null +++ b/mm/hugetlb_cma.c @@ -0,0 +1,258 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include + +#include +#include + +#include +#include "internal.h" +#include "hugetlb_cma.h" + + +static struct cma *hugetlb_cma[MAX_NUMNODES]; +static unsigned long hugetlb_cma_size_in_node[MAX_NUMNODES] __initdata; +static bool hugetlb_cma_only; +static unsigned long hugetlb_cma_size __initdata; + +void hugetlb_cma_free_folio(struct folio *folio) +{ + int nid = folio_nid(folio); + + WARN_ON_ONCE(!cma_free_folio(hugetlb_cma[nid], folio)); +} + + +struct folio *hugetlb_cma_alloc_folio(struct hstate *h, gfp_t gfp_mask, + int nid, nodemask_t *nodemask) +{ + int node; + int order = huge_page_order(h); + struct folio *folio = NULL; + + if (hugetlb_cma[nid]) + folio = cma_alloc_folio(hugetlb_cma[nid], order, gfp_mask); + + if (!folio && !(gfp_mask & __GFP_THISNODE)) { + for_each_node_mask(node, *nodemask) { + if (node == nid || !hugetlb_cma[node]) + continue; + + folio = cma_alloc_folio(hugetlb_cma[node], order, gfp_mask); + if (folio) + break; + } + } + + if (folio) + folio_set_hugetlb_cma(folio); + + return folio; +} + +struct huge_bootmem_page * __init +hugetlb_cma_alloc_bootmem(struct hstate *h, int nid) +{ + struct cma *cma; + struct huge_bootmem_page *m; + + cma = hugetlb_cma[nid]; + m = cma_reserve_early(cma, huge_page_size(h)); + if (m) { + m->flags = HUGE_BOOTMEM_CMA; + m->cma = cma; + } + + return m; +} + + +static bool cma_reserve_called __initdata; + +static int __init cmdline_parse_hugetlb_cma(char *p) +{ + int nid, count = 0; + unsigned long tmp; + char *s = p; + + while (*s) { + if (sscanf(s, "%lu%n", &tmp, &count) != 1) + break; + + if (s[count] == ':') { + if (tmp >= MAX_NUMNODES) + break; + nid = array_index_nospec(tmp, MAX_NUMNODES); + + s += count + 1; + tmp = memparse(s, &s); + hugetlb_cma_size_in_node[nid] = tmp; + hugetlb_cma_size += tmp; + + /* + * Skip the separator if have one, otherwise + * break the parsing. + */ + if (*s == ',') + s++; + else + break; + } else { + hugetlb_cma_size = memparse(p, &p); + break; + } + } + + return 0; +} + +early_param("hugetlb_cma", cmdline_parse_hugetlb_cma); + +static int __init cmdline_parse_hugetlb_cma_only(char *p) +{ + return kstrtobool(p, &hugetlb_cma_only); +} + +early_param("hugetlb_cma_only", cmdline_parse_hugetlb_cma_only); + +void __init hugetlb_cma_reserve(int order) +{ + unsigned long size, reserved, per_node; + bool node_specific_cma_alloc = false; + int nid; + + /* + * HugeTLB CMA reservation is required for gigantic + * huge pages which could not be allocated via the + * page allocator. Just warn if there is any change + * breaking this assumption. + */ + VM_WARN_ON(order <= MAX_PAGE_ORDER); + cma_reserve_called = true; + + if (!hugetlb_cma_size) + return; + + for (nid = 0; nid < MAX_NUMNODES; nid++) { + if (hugetlb_cma_size_in_node[nid] == 0) + continue; + + if (!node_online(nid)) { + pr_warn("hugetlb_cma: invalid node %d specified\n", nid); + hugetlb_cma_size -= hugetlb_cma_size_in_node[nid]; + hugetlb_cma_size_in_node[nid] = 0; + continue; + } + + if (hugetlb_cma_size_in_node[nid] < (PAGE_SIZE << order)) { + pr_warn("hugetlb_cma: cma area of node %d should be at least %lu MiB\n", + nid, (PAGE_SIZE << order) / SZ_1M); + hugetlb_cma_size -= hugetlb_cma_size_in_node[nid]; + hugetlb_cma_size_in_node[nid] = 0; + } else { + node_specific_cma_alloc = true; + } + } + + /* Validate the CMA size again in case some invalid nodes specified. */ + if (!hugetlb_cma_size) + return; + + if (hugetlb_cma_size < (PAGE_SIZE << order)) { + pr_warn("hugetlb_cma: cma area should be at least %lu MiB\n", + (PAGE_SIZE << order) / SZ_1M); + hugetlb_cma_size = 0; + return; + } + + if (!node_specific_cma_alloc) { + /* + * If 3 GB area is requested on a machine with 4 numa nodes, + * let's allocate 1 GB on first three nodes and ignore the last one. + */ + per_node = DIV_ROUND_UP(hugetlb_cma_size, nr_online_nodes); + pr_info("hugetlb_cma: reserve %lu MiB, up to %lu MiB per node\n", + hugetlb_cma_size / SZ_1M, per_node / SZ_1M); + } + + reserved = 0; + for_each_online_node(nid) { + int res; + char name[CMA_MAX_NAME]; + + if (node_specific_cma_alloc) { + if (hugetlb_cma_size_in_node[nid] == 0) + continue; + + size = hugetlb_cma_size_in_node[nid]; + } else { + size = min(per_node, hugetlb_cma_size - reserved); + } + + size = round_up(size, PAGE_SIZE << order); + + snprintf(name, sizeof(name), "hugetlb%d", nid); + /* + * Note that 'order per bit' is based on smallest size that + * may be returned to CMA allocator in the case of + * huge page demotion. + */ + res = cma_declare_contiguous_multi(size, PAGE_SIZE << order, + HUGETLB_PAGE_ORDER, name, + &hugetlb_cma[nid], nid); + if (res) { + pr_warn("hugetlb_cma: reservation failed: err %d, node %d", + res, nid); + continue; + } + + reserved += size; + pr_info("hugetlb_cma: reserved %lu MiB on node %d\n", + size / SZ_1M, nid); + + if (reserved >= hugetlb_cma_size) + break; + } + + if (!reserved) + /* + * hugetlb_cma_size is used to determine if allocations from + * cma are possible. Set to zero if no cma regions are set up. + */ + hugetlb_cma_size = 0; +} + +void __init hugetlb_cma_check(void) +{ + if (!hugetlb_cma_size || cma_reserve_called) + return; + + pr_warn("hugetlb_cma: the option isn't supported by current arch\n"); +} + +bool hugetlb_cma_exclusive_alloc(void) +{ + return hugetlb_cma_only; +} + +unsigned long __init hugetlb_cma_total_size(void) +{ + return hugetlb_cma_size; +} + +void __init hugetlb_cma_validate_params(void) +{ + if (!hugetlb_cma_size) + hugetlb_cma_only = false; +} + +bool __init hugetlb_early_cma(struct hstate *h) +{ + if (arch_has_huge_bootmem_alloc()) + return false; + + return hstate_is_gigantic(h) && hugetlb_cma_only; +} diff --git a/mm/hugetlb_cma.h b/mm/hugetlb_cma.h new file mode 100644 index 000000000000..ea400294bddd --- /dev/null +++ b/mm/hugetlb_cma.h @@ -0,0 +1,55 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_HUGETLB_CMA_H +#define _LINUX_HUGETLB_CMA_H + +#ifdef CONFIG_CMA +void hugetlb_cma_free_folio(struct folio *folio); +struct folio *hugetlb_cma_alloc_folio(struct hstate *h, gfp_t gfp_mask, + int nid, nodemask_t *nodemask); +struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int nid); +void hugetlb_cma_check(void); +bool hugetlb_cma_exclusive_alloc(void); +unsigned long hugetlb_cma_total_size(void); +void hugetlb_cma_validate_params(void); +bool hugetlb_early_cma(struct hstate *h); +#else +static inline void hugetlb_cma_free_folio(struct folio *folio) +{ +} + +static inline struct folio *hugetlb_cma_alloc_folio(struct hstate *h, + gfp_t gfp_mask, int nid, nodemask_t *nodemask) +{ + return NULL; +} + +static inline +struct huge_bootmem_page *hugetlb_cma_alloc_bootmem(struct hstate *h, int nid) +{ + return NULL; +} + +static inline void hugetlb_cma_check(void) +{ +} + +static inline bool hugetlb_cma_exclusive_alloc(void) +{ + return false; +} + +static inline unsigned long hugetlb_cma_total_size(void) +{ + return 0; +} + +static inline void hugetlb_cma_validate_params(void) +{ +} + +static inline bool hugetlb_early_cma(struct hstate *h) +{ + return false; +} +#endif +#endif