From patchwork Wed Jun 22 08:58:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12890324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D81BC433EF for ; Wed, 22 Jun 2022 08:59:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1058B8E00A0; Wed, 22 Jun 2022 04:59:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 08F918E008A; Wed, 22 Jun 2022 04:59:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E99898E00A0; Wed, 22 Jun 2022 04:59:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D46DD8E008A for ; Wed, 22 Jun 2022 04:59:32 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7F35B21418 for ; Wed, 22 Jun 2022 08:59:32 +0000 (UTC) X-FDA: 79605273384.16.3F1F845 Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) by imf26.hostedemail.com (Postfix) with ESMTP id A9DAF1400BA for ; Wed, 22 Jun 2022 08:59:26 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VH5BRmX_1655888343; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VH5BRmX_1655888343) by smtp.aliyun-inc.com; Wed, 22 Jun 2022 16:59:04 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 1/3] mm: Factor out the pagetable pages account into new helper function Date: Wed, 22 Jun 2022 16:58:52 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655888372; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=7YXhTi/3nPAbv/D37EtgqnZnexmlrpjb9zjbPV8w/FU=; b=VsL6xBqCUQLhwHGWLVlfdeZyX1b3JhscTzhGIbuS+t52IZIeM+Wzl80ZydqYoimFfc37zr ou0cmwM/063MosfTCrcy5XtihlP6G4endEdybmd+pCtsJdXYyDEITky3C0uz3jox4QjeQ4 U/ihiDaEMGYOh5o0DavYhwgjO+hSYKU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655888372; a=rsa-sha256; cv=none; b=GEdMPnphmfaUM6sFF4VSkbUaV2/s0DU9A8S9j5GjS0+Ic3iuXNOIWDhiHrCvxaNkHo4WOt aNzruPbFGFvemvvS/i1+8UBgficZc3i7tEoZHP/t4KSQx7yPerYWeGwh9/AgkvULKfT8B+ nV1s+BPZ2gH6jrvqHuOj8VfmyUap6zI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=temperror reason="SPF/DKIM temp error" header.from=alibaba.com (policy=temperror); spf=temperror (imf26.hostedemail.com: error in processing during lookup of baolin.wang@linux.alibaba.com: DNS error) smtp.mailfrom=baolin.wang@linux.alibaba.com X-Stat-Signature: 8j5zm7s4wg85o6fzwazeenzwsrds7c5z X-Rspamd-Queue-Id: A9DAF1400BA Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=temperror reason="SPF/DKIM temp error" header.from=alibaba.com (policy=temperror); spf=temperror (imf26.hostedemail.com: error in processing during lookup of baolin.wang@linux.alibaba.com: DNS error) smtp.mailfrom=baolin.wang@linux.alibaba.com X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1655888366-165433 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Factor out the pagetable pages account into new helper functions to avoid duplicated code. Meanwhile these helper functions also will be used to account pagetable pages which do not need split pagetale lock. Signed-off-by: Baolin Wang --- include/linux/mm.h | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 6d4e9ce1..6da6634 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2377,20 +2377,30 @@ static inline void pgtable_init(void) pgtable_cache_init(); } +static inline void pgtable_set_and_inc(struct page *page) +{ + __SetPageTable(page); + inc_lruvec_page_state(page, NR_PAGETABLE); +} + +static inline void pgtable_clear_and_dec(struct page *page) +{ + __ClearPageTable(page); + dec_lruvec_page_state(page, NR_PAGETABLE); +} + static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + pgtable_set_and_inc(page); return true; } static inline void pgtable_pte_page_dtor(struct page *page) { ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + pgtable_clear_and_dec(page); } #define pte_offset_map_lock(mm, pmd, address, ptlp) \ @@ -2476,16 +2486,14 @@ static inline bool pgtable_pmd_page_ctor(struct page *page) { if (!pmd_ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + pgtable_set_and_inc(page); return true; } static inline void pgtable_pmd_page_dtor(struct page *page) { pmd_ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + pgtable_clear_and_dec(page); } /* From patchwork Wed Jun 22 08:58:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12890323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB2CFC43334 for ; Wed, 22 Jun 2022 08:59:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 858E68E009F; Wed, 22 Jun 2022 04:59:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 807D28E008A; Wed, 22 Jun 2022 04:59:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F7388E009F; Wed, 22 Jun 2022 04:59:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 611F38E008A for ; Wed, 22 Jun 2022 04:59:32 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id 32A0D1210DB for ; Wed, 22 Jun 2022 08:59:32 +0000 (UTC) X-FDA: 79605273384.29.30F09E8 Received: from out30-45.freemail.mail.aliyun.com (out30-45.freemail.mail.aliyun.com [115.124.30.45]) by imf29.hostedemail.com (Postfix) with ESMTP id 4F8EA12000F for ; Wed, 22 Jun 2022 08:59:23 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VH5BRms_1655888344; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VH5BRms_1655888344) by smtp.aliyun-inc.com; Wed, 22 Jun 2022 16:59:05 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 2/3] mm: Add PUD level pagetable account Date: Wed, 22 Jun 2022 16:58:53 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655888371; a=rsa-sha256; cv=none; b=VUjuBlQ3WIi34m6BaYJS9Ps0ijb7EOsvVfDsTcwfy4gAiqxoVwvhT8kCN6J1Gt5YmezTdl 3Cz9cYf7EIq/JUmlLyWIN/pPQ7eqREM5Sfz5jWg4+8dCQm/8uZRjpftfcYXN7Ek7/zdtY4 n4ESdtmzGZwK0C1j+rSuMvRPSwh3xgA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.45 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655888371; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=5P5FlacjuaxdAX0aLukNJa0Kb6bO9gKlyFDFRiRAx18=; b=n91jtUvMkpcZw1sgoTv8vIIGbrzc/iDfjxEFqETvR21GQycz9jqjBH0JNrhqwJ3MSnA8gy ZmZ1wD3ieBAp2cPWmGOKvu64upAOXiVccfDa4NW91pu3q3cXWRdvbXSRPRZ2Cujdwxn6Jv lEeXepplMTSD+a8xMfSkjliYr6q0wLM= Authentication-Results: imf29.hostedemail.com; dkim=none; spf=pass (imf29.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.45 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4F8EA12000F X-Stat-Signature: i1ow8u81acbz77okxgs73kmkddihi1ak X-Rspam-User: X-HE-Tag: 1655888363-544326 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the PUD level ptes are always protected by mm->page_table_lock, which means no split pagetable lock needed. So the generic PUD level pagetable pages allocation will not call pgtable_pte_page_ctor/dtor(), that means we will miss to account PUD level pagetable pages. Adding pagetable account by calling pgtable_set_and_inc() or pgtable_clear_and_dec() when allocating or freeing PUD level pagetable pages to help to get an accurate pagetable accounting. Moreover this patch will also mark the PUD level pagetable with PG_table flag, which will help to do sanity validation in unpoison_memory() and get more accurate pagetable accounting by /proc/kpageflags interface. Meanwhile converting the architectures with using generic PUD pagatable allocation to add corresponding pgtable_set_and_inc() or pgtable_clear_and_dec() to account PUD level pagetable. Signed-off-by: Baolin Wang --- arch/arm64/include/asm/tlb.h | 5 ++++- arch/loongarch/include/asm/pgalloc.h | 11 ++++++++--- arch/mips/include/asm/pgalloc.h | 11 ++++++++--- arch/s390/include/asm/tlb.h | 1 + arch/x86/mm/pgtable.c | 5 ++++- include/asm-generic/pgalloc.h | 12 ++++++++++-- 6 files changed, 35 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index c995d1f..47e0623 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -94,7 +94,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pudp)); + struct page *page = virt_to_page(pudp); + + pgtable_clear_and_dec(page); + tlb_remove_table(tlb, page); } #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index b0a57b2..50a896f 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -89,10 +89,15 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { pud_t *pud; + struct page *pg; + + pg = alloc_pages(GFP_KERNEL & ~__GFP_HIGHMEM, PUD_ORDER); + if (!pg) + return NULL; - pud = (pud_t *) __get_free_pages(GFP_KERNEL, PUD_ORDER); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + pgtable_set_and_inc(pg); + pud = (pud_t *)page_address(pg); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); return pud; } diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 867e9c3..0950f5f 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -89,11 +89,16 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { + struct page *pg; pud_t *pud; - pud = (pud_t *) __get_free_pages(GFP_KERNEL, PUD_ORDER); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + pg = alloc_pages(GFP_KERNEL & ~__GFP_HIGHMEM, PUD_ORDER); + if (!pg) + return NULL; + + pgtable_set_and_inc(pg); + pud = (pud_t *)page_address(pg); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); return pud; } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index fe6407f..45f9541 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -125,6 +125,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pgtable_clear_and_dec(virt_to_page(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a932d77..a8ab3f9 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -76,8 +76,11 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { + struct page *page = virt_to_page(pud); + + pgtable_clear_and_dec(page); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(pud)); + paravirt_tlb_remove_table(tlb, page); } #if CONFIG_PGTABLE_LEVELS > 4 diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 977bea1..328a714 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -149,11 +149,16 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) { + struct page *page; gfp_t gfp = GFP_PGTABLE_USER; if (mm == &init_mm) gfp = GFP_PGTABLE_KERNEL; - return (pud_t *)get_zeroed_page(gfp); + page = alloc_pages((gfp | __GFP_ZERO) & ~__GFP_HIGHMEM, 0); + if (!page) + return NULL; + pgtable_set_and_inc(page); + return (pud_t *)page_address(page); } #ifndef __HAVE_ARCH_PUD_ALLOC_ONE @@ -174,8 +179,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { + struct page *page = virt_to_page(pud); + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - free_page((unsigned long)pud); + pgtable_clear_and_dec(page); + __free_page(page); } #ifndef __HAVE_ARCH_PUD_FREE From patchwork Wed Jun 22 08:58:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12890325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C25FCCA47E for ; Wed, 22 Jun 2022 08:59:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 525A88E00A1; Wed, 22 Jun 2022 04:59:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 439478E008A; Wed, 22 Jun 2022 04:59:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20B4C8E00A1; Wed, 22 Jun 2022 04:59:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 105658E008A for ; Wed, 22 Jun 2022 04:59:34 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id DD00234168 for ; Wed, 22 Jun 2022 08:59:33 +0000 (UTC) X-FDA: 79605273426.02.0BFBA9B Received: from out30-56.freemail.mail.aliyun.com (out30-56.freemail.mail.aliyun.com [115.124.30.56]) by imf07.hostedemail.com (Postfix) with ESMTP id 034ED4001D for ; Wed, 22 Jun 2022 08:59:27 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R351e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046049;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VH55S7r_1655888345; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VH55S7r_1655888345) by smtp.aliyun-inc.com; Wed, 22 Jun 2022 16:59:06 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v2 3/3] mm: Add kernel PTE level pagetable pages account Date: Wed, 22 Jun 2022 16:58:54 +0800 Message-Id: <7882bbf467440f9a3ebe41d96ba5b6f384081bb7.1655887440.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655888372; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=5WS/921Au9to8/YlGn7HoKpMtcJronokX/y9cxI3cNM=; b=DiHPMKlcB2HeqYauzTOeQUQgTKL0f49nAVSU5TFZf9751emwGhoJ9OlRx1ZC+JCuMUkNrd UFUvXFHW5VP6VhnoJat0eVaUnmMymGN5Fxd4d8Ye8U8GYdUXkSS+SDehyPPqKfD6AIWnDu a0yByo3jgXf7e03k/sIc78zEZHybcLM= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; spf=temperror (imf07.hostedemail.com: error in processing during lookup of baolin.wang@linux.alibaba.com: DNS error) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=temperror reason="SPF/DKIM temp error" header.from=alibaba.com (policy=temperror) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655888372; a=rsa-sha256; cv=none; b=rTuDOLFn35P9E6r5Ex7pkmxCkxv1u//7Lx21XQqwTuPrB7VjjXAUnySUVuHqbMkyUq1gqu bz94U9UIMMX0yjXYpqGRIE0DZrr3FrVGnj5Yzw2PFxAYRmwWKT1uA+xMMJGItlLgrEkHXF t896JSgP3dI6aP53PlyLWEr4wf5cKY0= X-Rspamd-Queue-Id: 034ED4001D Authentication-Results: imf07.hostedemail.com; dkim=none; spf=temperror (imf07.hostedemail.com: error in processing during lookup of baolin.wang@linux.alibaba.com: DNS error) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=temperror reason="SPF/DKIM temp error" header.from=alibaba.com (policy=temperror) X-Rspam-User: X-Stat-Signature: 8pmykgxxnyt7brk4nb51w4y5dx58dgh8 X-Rspamd-Server: rspam04 X-HE-Tag: 1655888367-202963 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the kernel PTE level ptes are always protected by mm->page_table_lock instead of split pagetable lock, so the kernel PTE level pagetable pages are not accounted. Especially the vmalloc()/vmap() can consume lots of kernel pagetable, so to get an accurate pagetable accounting, calling new helpers pgtable_set_and_inc()/pgtable_clear_and_dec() when allocating or freeing a kernel PTE level pagetable page. Meanwhile converting architectures to use corresponding generic PTE pagetable allocation and freeing functions. Signed-off-by: Baolin Wang Reported-by: kernel test robot Reported-by: kernel test robot --- arch/csky/include/asm/pgalloc.h | 2 +- arch/microblaze/mm/pgtable.c | 2 +- arch/openrisc/mm/ioremap.c | 2 +- arch/x86/mm/pgtable.c | 5 ++++- include/asm-generic/pgalloc.h | 14 ++++++++++++-- 5 files changed, 19 insertions(+), 6 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index bbbd069..2443226 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; unsigned long i; - pte = (pte_t *) __get_free_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); if (!pte) return NULL; diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 9f73265..e96dd1b 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -245,7 +245,7 @@ unsigned long iopa(unsigned long addr) __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { if (mem_init_done) - return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + return __pte_alloc_one_kernel(mm); else return memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE, MEMBLOCK_LOW_LIMIT, diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index daae13a..3453acc 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -118,7 +118,7 @@ pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; if (likely(mem_init_done)) { - pte = (pte_t *)get_zeroed_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); } else { pte = memblock_alloc(PAGE_SIZE, PAGE_SIZE); if (!pte) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a8ab3f9..fc2b9ef 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -851,6 +851,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) { pte_t *pte; + struct page *page; pte = (pte_t *)pmd_page_vaddr(*pmd); pmd_clear(pmd); @@ -858,7 +859,9 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) /* INVLPG to clear all paging-structure caches */ flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); - free_page((unsigned long)pte); + page = virt_to_page(pte); + pgtable_clear_and_dec(page); + __free_page(page); return 1; } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 328a714..2e20e9e 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -18,7 +18,14 @@ */ static inline pte_t *__pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(GFP_PGTABLE_KERNEL); + struct page *page; + gfp_t gfp = GFP_PGTABLE_KERNEL; + + page = alloc_pages(gfp & ~__GFP_HIGHMEM, 0); + if (!page) + return NULL; + pgtable_set_and_inc(page); + return (pte_t *)page_address(page); } #ifndef __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL @@ -41,7 +48,10 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { - free_page((unsigned long)pte); + struct page *page = virt_to_page(pte); + + pgtable_clear_and_dec(page); + __free_page(page); } /**