From patchwork Wed Dec 18 13:04:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13913613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70424E77188 for ; Wed, 18 Dec 2024 13:05:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E776F6B0089; Wed, 18 Dec 2024 08:05:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E288B6B008A; Wed, 18 Dec 2024 08:05:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA15A6B008C; Wed, 18 Dec 2024 08:05:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 747CF6B0089 for ; Wed, 18 Dec 2024 08:05:26 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 0654744B7B for ; Wed, 18 Dec 2024 13:05:26 +0000 (UTC) X-FDA: 82908098784.17.935510E Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf01.hostedemail.com (Postfix) with ESMTP id BA28440020 for ; Wed, 18 Dec 2024 13:04:59 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bSRY8JED; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734527109; a=rsa-sha256; cv=none; b=pEJof8WoKiJ+1+RDP+ncPEocOZbBvg7JO8EIDSCr3Ew8K1mxD/xx5UBAvg2hEq1bmMSmK0 4c3L4v/3hXdyglIisvj4XLIVYcgGiTmgbuhj538SB9lN8/G/EJ38DSJ1QVvTf539hWYogn A/SzPtKlZ02bHJlPdw5wLg+UP/8MAqA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=bSRY8JED; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf01.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734527109; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=N9tnOfMNAyKlwt7tLvm2UTXKxz5i7XrY+re9Me1pdlHgCrYV4bH+cZfjS9BjYUPL5+Cn5c FD4fhDjHQC3KjPxsK8CDAJ8Si1OB1oQ55cKq424qpClce6f7i//Yz+MTD3bOx6F5fItb6Y 8cfc8xOwMMM0r6ghYJd56z9su1lSrP0= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-215770613dbso43925135ad.2 for ; Wed, 18 Dec 2024 05:05:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1734527123; x=1735131923; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=bSRY8JEDczSTeZWaVlY7dkVISVW5Nkx3tF4YWtDGyGd1V0VYNwa9V3kvelbW8anbCY mPcJQOmsnfkqXNyfgaq9dX0kj6tpO+6HX5605NuwqykheZUNZzPENXMH19kgS4xI243q Oub9AVxdJpX63fwbao5Lj43dDkzod7s6eIeKdBL2mlG2wzn5aiU4wIiyPT03nxs/Z+nA JsklgcVdpu3sFbxjbwus08b87wIE3NIulVUq/vTqATR3hpm+m2OwvvSJVfviAHfugQOo ms0g6yuYKYxV5A6A6TCO7A04ON4x7Qg7S7dngFHGWg2QUP41TrAl137RfTmcZxAWyb0C CEJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734527123; x=1735131923; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V1U3ICCbfOFNMbev/k3EdvOWA+vXPbG14S+K01SHhE0=; b=Uv7TgQS7LJZk+AvpJTPEcMkQCu2+S9gXcdsyDPZxxuBXO2YoVCSBw0TTp1ndpwIQNj 1LE2N7T6gLFe1CWunH40Q3CORs9ogo0Yh7cmSG0XHnHQ+i0LfDGHmVCCnueg2NNPf+Ee FB/2mn/naQ+YjymCzITvqBBRNgTLWNEYb6fxvzk4RsAuQGZIcRmiMucKF993i9NCSboK TkOSqkhSLS50mplaq3Gr/T4279zZu6xcUimNseDot0lMZ0QnUke9BY+yNYL0eOI/SWDi kaG5bgIZTeqDd5m/4wThjhXnErwK2FLnxohPHRLiriYXfS5JpSSZf7uyzM3KuybanKnI 0DJw== X-Gm-Message-State: AOJu0YyowPUyLUrt09qSt3lZFme/iu1vl6qXJw08skHUX7Y5Y9o/1RLu DFekpK+60ArFcvk3AAJ0x47GJ4hvWlwcoA+iqJSW3iFmTdVJ7jQ42KwaT9YKouQ= X-Gm-Gg: ASbGnctjDxOGdn+oUYE3ETdoIkqk8KTMNjzEXNjm3SSrMNQfKbREAOYByCTF6o9+2BS JF8SMF1nn/vTbF3JZyPUJwsIQdFB4OPHzm/jesiutdcEym7mOt3LOWDNu4G2/00db2kv6oDxDlR JgnsR9rGcJWdAUIGV9MGBjAsBwLRWMgF+5cudS2dIPohUXygvGDnl7JzvxMOYQK8DFQUszN389r Mhfg30Vkx7NsdPA4fkV6GTZu6GGy0rzutfG8POJVHFT79Cntvw2zC4jd4WtSE9h1Kpu3IF2AGVg K0uh5q60gHZPlR5socpSbw== X-Google-Smtp-Source: AGHT+IHf2m598v0e4o8VjiXswnpFdHX9IrqnwvlyprzzHZEN7MlhmaMV6JPA0vL7pLwTvqcblsUT+g== X-Received: by 2002:a17:902:fac5:b0:216:6769:9ed7 with SMTP id d9443c01a7336-218d724960fmr35428695ad.40.1734527122502; Wed, 18 Dec 2024 05:05:22 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.238]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-218a1db755dsm75751825ad.42.2024.12.18.05.05.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2024 05:05:22 -0800 (PST) From: Qi Zheng To: peterz@infradead.org, tglx@linutronix.de, david@redhat.com, jannh@google.com, hughd@google.com, yuzhao@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, lorenzo.stoakes@oracle.com, akpm@linux-foundation.org, rientjes@google.com, vishal.moola@gmail.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v2 02/15] mm: pgtable: introduce generic p4d_alloc_one() and p4d_free() Date: Wed, 18 Dec 2024 21:04:38 +0800 Message-Id: X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: BA28440020 X-Stat-Signature: kgdfczndjqzg7uspqqsezxxdbs4z1ibb X-HE-Tag: 1734527099-951714 X-HE-Meta: U2FsdGVkX18cWN4UoatP8r3ZXVZtR2qej5XMEnqbWGF3aVXYsR3/IfS/tlHfOjU9gBrHQU2uKgT26o6irAFAAZE/of2iolNPyKqTjfxzdz0TKBlizwQ5R+lBLAz2Jxl5Mo3DbqrKiXtbkyKvqHKmw3szJ/DdeehV8lpfl2knZOjOqkXnNnnxLd6vG0ZVrqHmkEJseo07vEOZqs12J+bAZda9BGDYHoQcCl6JusrSZmwYuWzWFZOJySWUza2exgWuWvoShO4JSburX9cGYcEm2ebfa8CuzynhX0FLoAHS22R2ZgiJ/qE7VLwEzjSTzD3DWjb3VHxXRMUBx1bYP4IhTqG8GBACFo7KsGaW08H6Ea5Wa4HO2TG6QjNZpUeq09PEz4EzEDZVQVF+GIvYHeK86X77hXbpEHjRP723hHkdCQpqLoFJMwWjIzBmXY/gnCXbB6FFRsRUlftvBUDtB36VfKdfq7NPZLr6FO6ZFjLZgLadE6WSNN3WodOq+MoqTgKiOL90HVJcP87ZhFf3Tm0pQeEOAryy2pb4nJN3jMCVQCLRmx3xhOPUnT2s3HVV1ZjE3ErrVofltPjrWzxajMP8/eGKeqCWsIAsJDRVpsIr/uIvxN7AY7GsRKQGKjA8FkCGfYZ/m2IaGjiWayMgk/6ISCJPGoasXMEPM+9ZtO3e+2yXX+Jk/Er5mrs7FpbbLgmUqsJNJ2sEoStZEax9oZqXrnPXaeN4bzWaLJtYkd0JlzMhg6viVzxCDDuZR2T6xX6lD5/YHrOwfuA5tnmZolWWvwCToxPUf55M0akpnSTqgS7vWlzrkldlZ3fYVM2DxT1yBMrU5ANEv5qsikQNagkKQj/3u/qXfuekdQR44j2yQXGhdUUKxi2LhXDCVWA7bkxUETNWleYSt1wG7Ga4g9Av9ixgrFEolKNbG4w8aN2+vZfZiBDXYyPhqHAItnUiWsohzXfD/rRotEauIiNY4Ej NL/GKvv+ 0UFmrL9zAzOf/DG21wok/3La7CXY2+PRY9PDFDu0stz+AGyoBnkwgoVKQlAJcjsNphEK4V/LRXnd2B3VN+m6c/sBoZpNvcidFsZhTkfw5cLxfFboFUzLrxKMirG/uLaSUQfSwZbtSF+GNdKJCSQO5/+aY8Wto0z2MQp3apiu0zZFxVjMkBcVDhe3s7UZ8gBFzJsJa2zTIUDnO+xk8JCje8xzXVrzyXDYQOHHVLgNdrqjlujr0Hf2gzqWBk249BsqJNHLP9z2YCWfuVBxlInrGXE9P6xAi+lJ/tsvKyV4GbF2pOzNcSkTHDlAmGELjtEzwSB95ofCswdunn4XinqwMSItCkhITU0xIB8u+z/5XEAjphKAKbJDYWs1THwsEK7iaUqxRqYZg6aAcTSZWhakdTGDwL4qljiY86JfYBftIQ9P8VKKLtvMXoidcblX9tyPcSdP50JkK3lBqLF0pqXunT2ZDsWZ4kpyl2Ci7plW3BuCE/wc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Several architectures (arm64, riscv, x86) define p4d_alloc_one() as a wrapper for get_zeroed_page() and p4d_free() as a wrapper for free_page(). For these architectures, provide a generic implementation in asm-generic/pgalloc.h and convert them to use it. And like other levels of page tables, add statistics for P4D level page table. For s390, it also defines p4d_alloc_one() and p4d_free(), but it uses its own logic, so skip it. Signed-off-by: Qi Zheng Originally-by: Peter Zijlstra (Intel) --- arch/arm64/include/asm/pgalloc.h | 15 ++++----- arch/riscv/include/asm/pgalloc.h | 25 ++++++--------- arch/x86/include/asm/pgalloc.h | 16 ++++------ arch/x86/mm/pgtable.c | 3 ++ include/asm-generic/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ include/linux/mm.h | 16 ++++++++++ 6 files changed, 98 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index e75422864d1bd..679c530549327 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -15,6 +15,8 @@ #define __HAVE_ARCH_PGD_FREE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include #define PGD_SIZE (PTRS_PER_PGD * sizeof(pgd_t)) @@ -87,19 +89,16 @@ static inline void pgd_populate(struct mm_struct *mm, pgd_t *pgdp, p4d_t *p4dp) static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - gfp_t gfp = GFP_PGTABLE_USER; + if (!pgtable_l5_enabled()) + return NULL; - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + __p4d_free(mm, p4d); } #define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index f52264304f772..bb6e1c5f1fb19 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -14,6 +14,8 @@ #ifdef CONFIG_MMU #define __HAVE_ARCH_PUD_ALLOC_ONE #define __HAVE_ARCH_PUD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include static inline void riscv_tlb_remove_ptdesc(struct mmu_gather *tlb, void *pt) @@ -118,21 +120,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - if (pgtable_l5_enabled) { - gfp_t gfp = GFP_PGTABLE_USER; - - if (mm == &init_mm) - gfp = GFP_PGTABLE_KERNEL; - return (p4d_t *)get_zeroed_page(gfp); - } + if (!pgtable_l5_enabled) + return NULL; - return NULL; -} - -static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) -{ - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + return __p4d_alloc_one(mm, addr); } #define p4d_free p4d_free @@ -145,8 +136,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) + if (pgtable_l5_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); riscv_tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ diff --git a/arch/x86/include/asm/pgalloc.h b/arch/x86/include/asm/pgalloc.h index dcd836b59bebd..d9bc6cae77c9e 100644 --- a/arch/x86/include/asm/pgalloc.h +++ b/arch/x86/include/asm/pgalloc.h @@ -8,6 +8,8 @@ #define __HAVE_ARCH_PTE_ALLOC_ONE #define __HAVE_ARCH_PGD_FREE +#define __HAVE_ARCH_P4D_ALLOC_ONE +#define __HAVE_ARCH_P4D_FREE #include static inline int __paravirt_pgd_alloc(struct mm_struct *mm) { return 0; } @@ -149,20 +151,16 @@ static inline void pgd_populate_safe(struct mm_struct *mm, pgd_t *pgd, p4d_t *p4 static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) { - gfp_t gfp = GFP_KERNEL_ACCOUNT; + if (!pgtable_l5_enabled()) + return NULL; - if (mm == &init_mm) - gfp &= ~__GFP_ACCOUNT; - return (p4d_t *)get_zeroed_page(gfp); + return __p4d_alloc_one(mm, addr); } static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) { - if (!pgtable_l5_enabled()) - return; - - BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); - free_page((unsigned long)p4d); + if (pgtable_l5_enabled()) + return __p4d_free(mm, p4d); } extern void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d); diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 69a357b15974a..3d6e84da45b24 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -94,6 +94,9 @@ void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) #if CONFIG_PGTABLE_LEVELS > 4 void ___p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d) { + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + pagetable_p4d_dtor(ptdesc); paravirt_release_p4d(__pa(p4d) >> PAGE_SHIFT); paravirt_tlb_remove_table(tlb, virt_to_page(p4d)); } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 7c48f5fbf8aa7..dbf61819b3581 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -215,6 +215,61 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) #endif /* CONFIG_PGTABLE_LEVELS > 3 */ +#if CONFIG_PGTABLE_LEVELS > 4 + +static inline p4d_t *__p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + struct ptdesc *ptdesc; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + gfp &= ~__GFP_HIGHMEM; + + ptdesc = pagetable_alloc_noprof(gfp, 0); + if (!ptdesc) + return NULL; + + pagetable_p4d_ctor(ptdesc); + return ptdesc_address(ptdesc); +} +#define __p4d_alloc_one(...) alloc_hooks(__p4d_alloc_one_noprof(__VA_ARGS__)) + +#ifndef __HAVE_ARCH_P4D_ALLOC_ONE +/** + * p4d_alloc_one - allocate memory for a P4D-level page table + * @mm: the mm_struct of the current context + * + * Allocate memory for a page table using %GFP_PGTABLE_USER for user context + * and %GFP_PGTABLE_KERNEL for kernel context. + * + * Return: pointer to the allocated memory or %NULL on error + */ +static inline p4d_t *p4d_alloc_one_noprof(struct mm_struct *mm, unsigned long addr) +{ + return __p4d_alloc_one_noprof(mm, addr); +} +#define p4d_alloc_one(...) alloc_hooks(p4d_alloc_one_noprof(__VA_ARGS__)) +#endif + +static inline void __p4d_free(struct mm_struct *mm, p4d_t *p4d) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(p4d); + + BUG_ON((unsigned long)p4d & (PAGE_SIZE-1)); + pagetable_p4d_dtor(ptdesc); + pagetable_free(ptdesc); +} + +#ifndef __HAVE_ARCH_P4D_FREE +static inline void p4d_free(struct mm_struct *mm, pud_t *p4d) +{ + __p4d_free(mm, p4d); +} +#endif + +#endif + #ifndef __HAVE_ARCH_PGD_FREE static inline void pgd_free(struct mm_struct *mm, pgd_t *pgd) { diff --git a/include/linux/mm.h b/include/linux/mm.h index 9d58f89e9643e..7168f46ad4770 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3269,6 +3269,22 @@ static inline void pagetable_pud_dtor(struct ptdesc *ptdesc) lruvec_stat_sub_folio(folio, NR_PAGETABLE); } +static inline void pagetable_p4d_ctor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_set_pgtable(folio); + lruvec_stat_add_folio(folio, NR_PAGETABLE); +} + +static inline void pagetable_p4d_dtor(struct ptdesc *ptdesc) +{ + struct folio *folio = ptdesc_folio(ptdesc); + + __folio_clear_pgtable(folio); + lruvec_stat_sub_folio(folio, NR_PAGETABLE); +} + extern void __init pagecache_init(void); extern void free_initmem(void);