From patchwork Wed Jun 5 22:27:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sourav Panda X-Patchwork-Id: 13687633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4644C25B76 for ; Wed, 5 Jun 2024 22:28:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28B776B00A1; Wed, 5 Jun 2024 18:28:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 214686B00A2; Wed, 5 Jun 2024 18:28:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08E266B00A3; Wed, 5 Jun 2024 18:28:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id DC92D6B00A1 for ; Wed, 5 Jun 2024 18:28:01 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 4A216160A72 for ; Wed, 5 Jun 2024 22:28:01 +0000 (UTC) X-FDA: 82198273962.14.8A784B1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf04.hostedemail.com (Postfix) with ESMTP id 7764140003 for ; Wed, 5 Jun 2024 22:27:58 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kRNcuH7Z; spf=pass (imf04.hostedemail.com: domain of 3beZgZgsKCPYqmspYtnYlbYemmejc.amkjglsv-kkitYai.mpe@flex--souravpanda.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3beZgZgsKCPYqmspYtnYlbYemmejc.amkjglsv-kkitYai.mpe@flex--souravpanda.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717626478; a=rsa-sha256; cv=none; b=KYurAW4qzO09qRBSYD8i3lSKCd9XpFokwrlSOkowR10MbLxcoO/g/jXlW0oZ/JhwlfHLk8 WAEppU8dCN5yKzlbXmxQnkJKw0A1pjgHEMKUWJ1JJNJcueMIg6ZrGdPRuv3uEuchEF3XSn 9qYJA5jQYsp5jbg1Dw9u+w1JlJ1DIEc= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kRNcuH7Z; spf=pass (imf04.hostedemail.com: domain of 3beZgZgsKCPYqmspYtnYlbYemmejc.amkjglsv-kkitYai.mpe@flex--souravpanda.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3beZgZgsKCPYqmspYtnYlbYemmejc.amkjglsv-kkitYai.mpe@flex--souravpanda.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717626478; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=5Q0Q0wa3tfXSBoL1NsxTXiZdJjSmKxrHbLhP2Xd5cpY=; b=7PQDo/jlgFeRaOHvMaUIq7wRxXOtUBiRzXeFtaVf//S1QBtUXvyjpTWLqGKyUVk4QSqlq7 78zMPRrfP4ns9v53jdHfiHbGMGDz9aYLM5bS/Hz0qh360mX5i3qhMDaawCM14/m7V8Dw7l WvJP+byJN3ixA0sPTd1x/xTLahCYn/U= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-df773f9471fso694631276.3 for ; Wed, 05 Jun 2024 15:27:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1717626477; x=1718231277; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=5Q0Q0wa3tfXSBoL1NsxTXiZdJjSmKxrHbLhP2Xd5cpY=; b=kRNcuH7Z2piX0yKX8kX4hCGZVS28DPuLFVi0Ui7czfMAhYQE0p4T1xIMFHyDSX55HS 19XuIfEW8ckNpMfz9VkhOVuMmHLWWgMjo2uwyXbRYGMD/eZvVNgeE3+EitLc670m2DXR NsrH5UhpBtZYKm4U2Ao4Q7z7AJsdy4iSeec7Sr//GHS31Bab4vulRhUwhXKQpuFBWbQi nnHuUGMiAGpmiRQ8eKR+uS28AOirM3A6DxFlmlir71FxTXg6tDGXL85r8do4LPvQ1Y7R PICfB7JpnljhaKyPNAVgy501MgJnNUGj/ktcdfm5Qk4uwDJA3if3KcULR0AkfVU4/Lsw od3A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717626477; x=1718231277; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=5Q0Q0wa3tfXSBoL1NsxTXiZdJjSmKxrHbLhP2Xd5cpY=; b=LebJyT2SDsdyzuk2EQsBH8zPeAC/aMNtI9S2MUiCnpl5jl2OtFGOMGZXjSnngWxTdr CYOVLfpIVcq4xwWEn/YXmAu2XBYvqvjieCcGMcDaUZKWzbTSTESEQl+WRA86/Xkry5Qq UA8DqOW37zzjLY/1gf9H6eJafH7JFVgU1GcVwqmN9PjRGeiKvyMZaBf3lIOFVV4abH02 Hnzy7CGwgBRHIIiFO0jGsRL6Dihlqm073TVV7NUA3gzCoTkbz5QffqD07Csixd9ctZoF +bUU7UthVl2WURyx81hsRKnhY+O4uV37D9K8Vb2bhnWGrAWT4kTOBDTu0ekhKLZUO5Vc Dffg== X-Forwarded-Encrypted: i=1; AJvYcCVatQC/7gkNb8a75Cft0SVUx8puAwS0qFsCuQ06N2Rd/JRCZjcpXbB0dUP/6IeZ04Eb77w2t0ETXemoZBVGZRaXRQM= X-Gm-Message-State: AOJu0YwT/KC4GR/Bb4INzIa7C6HgDwuglJo4KGAAiUuf1ml75DKrVu8p v5mmmeH9nMkFExUy+ZDm9H1kT2y5+kXsmFum/t3ERt0lhgWMPTPSuPD3fdj4Q6bwcebUlX6vZWm AJjOuu91eo3heClg1BGgQVg== X-Google-Smtp-Source: AGHT+IGXPjMjFJ10zc2uvGKtEO6W0CWWKU5zbG/sUKhreQen3kRGgueDSN0isP4pHhblea5BrhqzoA7QHfMiINiLmA== X-Received: from souravbig.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3b3a]) (user=souravpanda job=sendgmr) by 2002:a05:6902:18d3:b0:dfa:6e39:95c6 with SMTP id 3f1490d57ef6-dfacad065a2mr1151837276.8.1717626477317; Wed, 05 Jun 2024 15:27:57 -0700 (PDT) Date: Wed, 5 Jun 2024 22:27:51 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.45.1.467.gbab1589fc0-goog Message-ID: <20240605222751.1406125-1-souravpanda@google.com> Subject: [PATCH v13] mm: report per-page metadata information From: Sourav Panda To: corbet@lwn.net, gregkh@linuxfoundation.org, rafael@kernel.org, akpm@linux-foundation.org, mike.kravetz@oracle.com, muchun.song@linux.dev, rppt@kernel.org, david@redhat.com, rdunlap@infradead.org, chenlinxuan@uniontech.com, yang.yang29@zte.com.cn, souravpanda@google.com, tomas.mudrunka@gmail.com, bhelgaas@google.com, ivan@cloudflare.com, pasha.tatashin@soleen.com, yosryahmed@google.com, hannes@cmpxchg.org, shakeelb@google.com, kirill.shutemov@linux.intel.com, wangkefeng.wang@huawei.com, adobriyan@gmail.com, vbabka@suse.cz, Liam.Howlett@Oracle.com, surenb@google.com, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, willy@infradead.org, weixugc@google.com Cc: David Rientjes X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7764140003 X-Stat-Signature: jtx6ubnwxjnj5wj3xhp8qj36ucxasqp6 X-HE-Tag: 1717626478-647895 X-HE-Meta: U2FsdGVkX1+LrT1orlX/kNLxhEQioGR8gsSkyG+MECO4ACBCAZ+6AV6KoMKVtmtjIw87HaVznatGFacMDZlvS5wf5sHawxTzZq0s/9vFJLyxDbXBBhvwJIyg0JlYvG3PRD8ln/oaJ1D8d1PhoTN1Lqc3+LTjaoPKeDlycGsYtrzJyJre1SqnYc+iI8r5m43tKfEHK/dlDrCpXmtQbyCCVo9yX2pK9F1kKswWUF2cw8MV/7HBEn/1MRZujmXX7UUOEKpxz1pKhsX3ptcYEQ3PqgZ2erRLPx/nZqVKpRGp7TGppwceWydTiSGoaAKNi5CapNDolfnRk2v2MRzc5p8eHP/DV0sOnK+jr1dbG+5JXhafIirdJsCwIE8la9lTTMGr+R3ccTJSM+KLYlMl8dVBu+dB07RaNIw0geDWZKB4zQVjA4VTbCOpgm31WpLy7+MTp7cXA+QJ133uJqpPf2Rjo3+yHwfwUCvtdjzFtqxLCsfQexgT+YIXplLFQvw4SISMIjN0YDEvTTjFTNVfhmHOY+MDS3BzTog7c7JR6nrVpvMr4evM9CdKoPyt8hQRhewLYP011+QZO7d8DB0v18FR7ht2XffemE3qN4MudBjy3/4iukiq1vsPvHVfHAfaRM8/DwGqQEKtEd8oCDH9YzMy+TWlvRbSqeWuB824EK8qvhi+DsdaEQ/LkFQnWw3osnzB+jJ+rInm5dWwbhj27QV2MQ+iuo129oOYj7rjIuqv7yxbKw1QWnF76bMa93dRfbuXUbXMTqEWmj+9V4/Q5opkZ4r4exZgdUycRHh/UzGLEZb27X6FErWZKhjcxQJxawf1AyZXYTibVNEnKwY2BgwEV0w/Jt0UvqFWZD1G0OcFDSYpcipdJNiTYAkDyqPe2wgEZfzhbDEzgXVhxNq8ER1oq2Q6eJRAsRNF/2u+EF4+fC1+W87G9iDOiZm/8lM4JW1CpQoLvrI9JasZH/BiRB7 T16Wfd2y 2ygOwEulqmcJLLU/lZL9JwjXTafccoaNBE76DiKzH1eRZrvUwQIDLROBALF2oKlmXBGMo1rbbUYWCYdRG+OBZTzBURqx+fAJJg7n4NuHacU93Zt/m0ds2PGtIRN3sgEMAkp5lTAVuf2DHD27000rQ5ce20RGQY1o7g98L07IcFuYWZw+lIuUas/Nf4bwHxkug9Xce6jSd7BLjQVb7I6ogFxtElzrTKZj7BRp7Q8c6nIOe4+UdoJI4gbD7C6v0dkEGhMGAetTHSqQxs5pqMmXqMqhSjyY4TK7HR5B6TPr/md0iU7TFdump/uokEuVGO1nkWwEfScqD4PbcxrKFBQb7weRYQkiKASi1UzZfcMuz+4sPUcYQ2TSS+3wlUc22ILmF+jvy09CfoSuwfb+WQd6viE7IlQcc6uaSmubgsWUWe1KLYEzZ9xLYYQLQxz7MQB+hy95R2t0cTRS2eiV+JgF5HFPqEeqebBxXnRfJD6Z91d4jyk9/mhpmJq6UOS3dYDYX7WvRkItySArb+9KG+ZZzpRhrULZstlkpP1BwP/ovTv5kfLz4M+64DfbWdUwYsDRAejHhrr9U4KT4bSWFfygdV+zc14GUdIVWDCt3DNeMcy0FAxeXgVTNbozVdenhtA9+WpCAzeP+YNLFaNcl18vJ4w4SaAT8AwNi8GVYltW7UsaDP+pMI88E62KVClJ+D3mnTvOXO6P27KqcDFpg4ZEDCPmh68lf11ZKw4IQoVOIF7tr5m5dRNU6DoiAwMoBd26uMBwTfzEH/9C6ci/Y80Yc5rZbZIfN27YzG3XAZRz1Bw3lgPS4ynsTfA52NQdxxgaHqBGpnjWYam+RTCSQ1cGxXum/EA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Today, we do not have any observability of per-page metadata and how much it takes away from the machine capacity. Thus, we want to describe the amount of memory that is going towards per-page metadata, which can vary depending on build configuration, machine architecture, and system use. This patch adds 2 fields to /proc/vmstat that can used as shown below: Accounting per-page metadata allocated by boot-allocator: /proc/vmstat:nr_memmap_boot * PAGE_SIZE Accounting per-page metadata allocated by buddy-allocator: /proc/vmstat:nr_memmap * PAGE_SIZE Accounting total Perpage metadata allocated on the machine: (/proc/vmstat:nr_memmap_boot + /proc/vmstat:nr_memmap) * PAGE_SIZE Utility for userspace: Observability: Describe the amount of memory overhead that is going to per-page metadata on the system at any given time since this overhead is not currently observable. Debugging: Tracking the changes or absolute value in struct pages can help detect anomalies as they can be correlated with other metrics in the machine (e.g., memtotal, number of huge pages, etc). page_ext overheads: Some kernel features such as page_owner page_table_check that use page_ext can be optionally enabled via kernel parameters. Having the total per-page metadata information helps users precisely measure impact. Furthermore, page-metadata metrics will reflect the amount of struct pages reliquished (or overhead reduced) when hugetlbfs pages are reserved which will vary depending on whether hugetlb vmemmap optimization is enabled or not. For background and results see: lore.kernel.org/all/20240220214558.3377482-1-souravpanda@google.com Acked-by: David Rientjes Signed-off-by: Sourav Panda Reviewed-by: Pasha Tatashin --- Changelog: Synchronized with 6.10-rc2. Added David Rientjes's Ack. v12: https://lore.kernel.org/all/20240512010611.290464-1-souravpanda@google.com --- include/linux/mmzone.h | 2 ++ include/linux/vmstat.h | 4 ++++ mm/hugetlb_vmemmap.c | 17 +++++++++++++---- mm/mm_init.c | 3 +++ mm/page_alloc.c | 1 + mm/page_ext.c | 32 +++++++++++++++++++++++--------- mm/sparse-vmemmap.c | 8 ++++++++ mm/sparse.c | 7 ++++++- mm/vmstat.c | 26 +++++++++++++++++++++++++- 9 files changed, 85 insertions(+), 15 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8f9c9590a42c..b7546dd8c298 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -220,6 +220,8 @@ enum node_stat_item { PGDEMOTE_KSWAPD, PGDEMOTE_DIRECT, PGDEMOTE_KHUGEPAGED, + NR_MEMMAP, /* page metadata allocated through buddy allocator */ + NR_MEMMAP_BOOT, /* page metadata allocated through boot allocator */ NR_VM_NODE_STAT_ITEMS }; diff --git a/include/linux/vmstat.h b/include/linux/vmstat.h index 735eae6e272c..16b0cfa80502 100644 --- a/include/linux/vmstat.h +++ b/include/linux/vmstat.h @@ -624,4 +624,8 @@ static inline void lruvec_stat_sub_folio(struct folio *folio, { lruvec_stat_mod_folio(folio, idx, -folio_nr_pages(folio)); } + +void __meminit mod_node_early_perpage_metadata(int nid, long delta); +void __meminit store_early_perpage_metadata(void); + #endif /* _LINUX_VMSTAT_H */ diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c index b9a55322e52c..fa00d61b6c5a 100644 --- a/mm/hugetlb_vmemmap.c +++ b/mm/hugetlb_vmemmap.c @@ -184,10 +184,13 @@ static int vmemmap_remap_range(unsigned long start, unsigned long end, */ static inline void free_vmemmap_page(struct page *page) { - if (PageReserved(page)) + if (PageReserved(page)) { free_bootmem_page(page); - else + mod_node_page_state(page_pgdat(page), NR_MEMMAP_BOOT, -1); + } else { __free_page(page); + mod_node_page_state(page_pgdat(page), NR_MEMMAP, -1); + } } /* Free a list of the vmemmap pages */ @@ -338,6 +341,7 @@ static int vmemmap_remap_free(unsigned long start, unsigned long end, copy_page(page_to_virt(walk.reuse_page), (void *)walk.reuse_addr); list_add(&walk.reuse_page->lru, vmemmap_pages); + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, 1); } /* @@ -384,14 +388,19 @@ static int alloc_vmemmap_page_list(unsigned long start, unsigned long end, unsigned long nr_pages = (end - start) >> PAGE_SHIFT; int nid = page_to_nid((struct page *)start); struct page *page, *next; + int i; - while (nr_pages--) { + for (i = 0; i < nr_pages; i++) { page = alloc_pages_node(nid, gfp_mask, 0); - if (!page) + if (!page) { + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, i); goto out; + } list_add(&page->lru, list); } + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, nr_pages); + return 0; out: list_for_each_entry_safe(page, next, list, lru) diff --git a/mm/mm_init.c b/mm/mm_init.c index f72b852bd5b8..ca2daf0a6993 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -29,6 +29,7 @@ #include #include #include +#include #include "internal.h" #include "slab.h" #include "shuffle.h" @@ -1618,6 +1619,8 @@ static void __init alloc_node_mem_map(struct pglist_data *pgdat) panic("Failed to allocate %ld bytes for node %d memory map\n", size, pgdat->node_id); pgdat->node_mem_map = map + offset; + mod_node_early_perpage_metadata(pgdat->node_id, + DIV_ROUND_UP(size, PAGE_SIZE)); pr_debug("%s: node %d, pgdat %08lx, node_mem_map %08lx\n", __func__, pgdat->node_id, (unsigned long)pgdat, (unsigned long)pgdat->node_mem_map); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 2e22ce5675ca..1f747319f76d 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -5715,6 +5715,7 @@ void __init setup_per_cpu_pageset(void) for_each_online_pgdat(pgdat) pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat); + store_early_perpage_metadata(); } __meminit void zone_pcp_init(struct zone *zone) diff --git a/mm/page_ext.c b/mm/page_ext.c index 95dd8ffeaf81..c191e490c401 100644 --- a/mm/page_ext.c +++ b/mm/page_ext.c @@ -214,6 +214,8 @@ static int __init alloc_node_page_ext(int nid) return -ENOMEM; NODE_DATA(nid)->node_page_ext = base; total_usage += table_size; + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP_BOOT, + DIV_ROUND_UP(table_size, PAGE_SIZE)); return 0; } @@ -268,12 +270,15 @@ static void *__meminit alloc_page_ext(size_t size, int nid) void *addr = NULL; addr = alloc_pages_exact_nid(nid, size, flags); - if (addr) { + if (addr) kmemleak_alloc(addr, size, 1, flags); - return addr; - } + else + addr = vzalloc_node(size, nid); - addr = vzalloc_node(size, nid); + if (addr) { + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, + DIV_ROUND_UP(size, PAGE_SIZE)); + } return addr; } @@ -316,18 +321,27 @@ static int __meminit init_section_page_ext(unsigned long pfn, int nid) static void free_page_ext(void *addr) { + size_t table_size; + struct page *page; + struct pglist_data *pgdat; + + table_size = page_ext_size * PAGES_PER_SECTION; + if (is_vmalloc_addr(addr)) { + page = vmalloc_to_page(addr); + pgdat = page_pgdat(page); vfree(addr); } else { - struct page *page = virt_to_page(addr); - size_t table_size; - - table_size = page_ext_size * PAGES_PER_SECTION; - + page = virt_to_page(addr); + pgdat = page_pgdat(page); BUG_ON(PageReserved(page)); kmemleak_free(addr); free_pages_exact(addr, table_size); } + + mod_node_page_state(pgdat, NR_MEMMAP, + -1L * (DIV_ROUND_UP(table_size, PAGE_SIZE))); + } static void __free_page_ext(unsigned long pfn) diff --git a/mm/sparse-vmemmap.c b/mm/sparse-vmemmap.c index a2cbe44c48e1..1dda6c53370b 100644 --- a/mm/sparse-vmemmap.c +++ b/mm/sparse-vmemmap.c @@ -469,5 +469,13 @@ struct page * __meminit __populate_section_memmap(unsigned long pfn, if (r < 0) return NULL; + if (system_state == SYSTEM_BOOTING) { + mod_node_early_perpage_metadata(nid, DIV_ROUND_UP(end - start, + PAGE_SIZE)); + } else { + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP, + DIV_ROUND_UP(end - start, PAGE_SIZE)); + } + return pfn_to_page(pfn); } diff --git a/mm/sparse.c b/mm/sparse.c index de40b2c73406..973d66a15062 100644 --- a/mm/sparse.c +++ b/mm/sparse.c @@ -14,7 +14,7 @@ #include #include #include - +#include #include "internal.h" #include @@ -465,6 +465,9 @@ static void __init sparse_buffer_init(unsigned long size, int nid) */ sparsemap_buf = memmap_alloc(size, section_map_size(), addr, nid, true); sparsemap_buf_end = sparsemap_buf + size; +#ifndef CONFIG_SPARSEMEM_VMEMMAP + mod_node_early_perpage_metadata(nid, DIV_ROUND_UP(size, PAGE_SIZE)); +#endif } static void __init sparse_buffer_fini(void) @@ -643,6 +646,8 @@ static void depopulate_section_memmap(unsigned long pfn, unsigned long nr_pages, unsigned long start = (unsigned long) pfn_to_page(pfn); unsigned long end = start + nr_pages * sizeof(struct page); + mod_node_page_state(page_pgdat(pfn_to_page(pfn)), NR_MEMMAP, + -1L * (DIV_ROUND_UP(end - start, PAGE_SIZE))); vmemmap_free(start, end, altmap); } static void free_map_bootmem(struct page *memmap) diff --git a/mm/vmstat.c b/mm/vmstat.c index 8507c497218b..73d791d1caad 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1255,7 +1255,8 @@ const char * const vmstat_text[] = { "pgdemote_kswapd", "pgdemote_direct", "pgdemote_khugepaged", - + "nr_memmap", + "nr_memmap_boot", /* enum writeback_stat_item counters */ "nr_dirty_threshold", "nr_dirty_background_threshold", @@ -2282,4 +2283,27 @@ static int __init extfrag_debug_init(void) } module_init(extfrag_debug_init); + #endif + +/* + * Page metadata size (struct page and page_ext) in pages + */ +static unsigned long early_perpage_metadata[MAX_NUMNODES] __meminitdata; + +void __meminit mod_node_early_perpage_metadata(int nid, long delta) +{ + early_perpage_metadata[nid] += delta; +} + +void __meminit store_early_perpage_metadata(void) +{ + int nid; + struct pglist_data *pgdat; + + for_each_online_pgdat(pgdat) { + nid = pgdat->node_id; + mod_node_page_state(NODE_DATA(nid), NR_MEMMAP_BOOT, + early_perpage_metadata[nid]); + } +}