From patchwork Thu Jun 30 11:11:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12901697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D71BC433EF for ; Thu, 30 Jun 2022 11:11:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 594CC6B0073; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5468C8E0002; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E4758E0001; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2C9C56B0072 for ; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 00FCB34A8F for ; Thu, 30 Jun 2022 11:11:30 +0000 (UTC) X-FDA: 79634636382.19.C4E1DDA Received: from out199-15.us.a.mail.aliyun.com (out199-15.us.a.mail.aliyun.com [47.90.199.15]) by imf21.hostedemail.com (Postfix) with ESMTP id 7D7801C0035 for ; Thu, 30 Jun 2022 11:11:29 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R711e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046060;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VHrqagJ_1656587484; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VHrqagJ_1656587484) by smtp.aliyun-inc.com; Thu, 30 Jun 2022 19:11:25 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v3 1/3] mm: Factor out the pagetable pages account into new helper function Date: Thu, 30 Jun 2022 19:11:14 +0800 Message-Id: <9c527d4d2eb1f457306e575ce16c6acdd8141e02.1656586863.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656587490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=11VI4BaJdVZVCzF5re1KQ1hS58clfYoLG4I4dlmXXqs=; b=46bHUR2cA/BQdejdcHl+mpDByZKntgzu8Oc3beZex5ZZ1nDSr6/bT8ilgGX2HmTiEPACYt Yv1O8QT2yzWDWE2N1UjyOjIpl7JMWd/oM2VohjDjmCBYo3CzMCs4h+OGYPdwkN2CMAkP6o zOB+Ub+Fa2pKm4286w7HPII9zCpeO+k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656587490; a=rsa-sha256; cv=none; b=2kZjCwELUEtQezyUhejpmGf217TDNp0ZqCZKlMBpLknjv8FAMe+gR/evnJBJ/228nYh4xk Id8IxyEyg8Oa0QGvQzs+isWdQsjPCWGY/iDg5y12fgs/gC5BEBozmdLutbZK04dt2h4u4R TLRw5xZckg2sq9Jy/v2epiaC1Ft4YPs= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 47.90.199.15 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 47.90.199.15 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Rspamd-Server: rspam03 X-Rspam-User: X-Stat-Signature: gh9sm7can49zjs178k8zcq5c5o3n5sog X-Rspamd-Queue-Id: 7D7801C0035 X-HE-Tag: 1656587489-181562 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Factor out the pagetable pages account into new helper functions to avoid duplicated code. Meanwhile these helper functions also will be used to account pagetable pages which do not need split pagetale lock. Meanwhile convert to use mod_lruvec_page_state() in case of non-order-0 page table allocation. Signed-off-by: Baolin Wang --- include/linux/mm.h | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index a2270e3..3be6d2c 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2353,20 +2353,30 @@ static inline void pgtable_init(void) pgtable_cache_init(); } +static inline void pgtable_page_inc(struct page *page) +{ + __SetPageTable(page); + mod_lruvec_page_state(page, NR_PAGETABLE, compound_nr(page)); +} + +static inline void pgtable_page_dec(struct page *page) +{ + __ClearPageTable(page); + mod_lruvec_page_state(page, NR_PAGETABLE, -compound_nr(page)); +} + static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + pgtable_page_inc(page); return true; } static inline void pgtable_pte_page_dtor(struct page *page) { ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + pgtable_page_dec(page); } #define pte_offset_map_lock(mm, pmd, address, ptlp) \ @@ -2452,16 +2462,14 @@ static inline bool pgtable_pmd_page_ctor(struct page *page) { if (!pmd_ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + pgtable_page_inc(page); return true; } static inline void pgtable_pmd_page_dtor(struct page *page) { pmd_ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + pgtable_page_dec(page); } /* From patchwork Thu Jun 30 11:11:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12901699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63700CCA47F for ; Thu, 30 Jun 2022 11:11:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A7E6E6B0072; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 95AD48E0001; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 743AF6B0072; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 3BE696B0075 for ; Thu, 30 Jun 2022 07:11:31 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 0C3EE8125C for ; Thu, 30 Jun 2022 11:11:31 +0000 (UTC) X-FDA: 79634636382.13.FFE5258 Received: from out30-57.freemail.mail.aliyun.com (out30-57.freemail.mail.aliyun.com [115.124.30.57]) by imf11.hostedemail.com (Postfix) with ESMTP id 6082A4000A for ; Thu, 30 Jun 2022 11:11:28 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046050;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VHrjYip_1656587485; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VHrjYip_1656587485) by smtp.aliyun-inc.com; Thu, 30 Jun 2022 19:11:26 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v3 2/3] mm: Add PUD level pagetable account Date: Thu, 30 Jun 2022 19:11:15 +0800 Message-Id: <6a6a768634b9ce8537154264e35e6a66a79b6ca8.1656586863.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656587490; a=rsa-sha256; cv=none; b=vCI0CgwhLcmSvmXh7myDJzLcCAwQscY3y6F6uBfyWFDU/3sTTu4G05sCyajj2udoxirloh 9o+hqcMOoElj7dR6zFyV3QRS13JN7bYbZr1f3RDY4HdxCMcxyUOzTWJxEOr3hqIN3+6SlA zroVC5CCuLQOhUKiJBfFzbI7la8zM8o= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf11.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.57 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656587490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=d8XVE+e8g/crSh3oLBw5ILlEBKmphms77lH8li6SrKA=; b=6sIdwMmgKH7f8JqZJjiaK4nix9a0EBvl3j+zap7hZI2dRHifhy9KdB8053i1tKroZGcZw5 68EUbQ2fOv/m+xZWMC09pahr9iXzD/8SQPMF5BhxsX3hg0WI4Gnx75H4p6KeB7gluSOj5f wSuIDZeupgmw+ZnJGncq0fUj6qzS0ro= Authentication-Results: imf11.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf11.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.57 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6082A4000A X-Stat-Signature: y7oezdcjc8b9ygeina9fak9dc19iy9fr X-Rspam-User: X-HE-Tag: 1656587488-765725 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the PUD level ptes are always protected by mm->page_table_lock, which means no split pagetable lock needed. So the generic PUD level pagetable pages allocation will not call pgtable_pte_page_ctor/dtor(), that means we will miss to account PUD level pagetable pages. Adding pagetable account by calling pgtable_page_inc() or pgtable_page_dec() when allocating or freeing PUD level pagetable pages to help to get an accurate pagetable accounting. Moreover this patch will also mark the PUD level pagetable with PG_table flag, which will help to do sanity validation in unpoison_memory() and get more accurate pagetable accounting by /proc/kpageflags interface. Meanwhile converting the architectures with using generic PUD pagatable allocation to add corresponding pgtable_page_inc() or pgtable_page_dec() to account PUD level pagetable. Signed-off-by: Baolin Wang --- arch/arm64/include/asm/tlb.h | 5 ++++- arch/loongarch/include/asm/pgalloc.h | 11 ++++++++--- arch/mips/include/asm/pgalloc.h | 11 ++++++++--- arch/s390/include/asm/tlb.h | 1 + arch/x86/mm/pgtable.c | 5 ++++- include/asm-generic/pgalloc.h | 12 ++++++++++-- 6 files changed, 35 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index c995d1f..1772df9 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -94,7 +94,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pudp)); + struct page *page = virt_to_page(pudp); + + pgtable_page_dec(page); + tlb_remove_table(tlb, page); } #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index b0a57b2..19bfe14 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -89,10 +89,15 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { pud_t *pud; + struct page *page; - pud = (pud_t *) __get_free_pages(GFP_KERNEL, PUD_ORDER); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + page = alloc_pages(GFP_KERNEL, PUD_ORDER); + if (!page) + return NULL; + + pgtable_page_inc(page); + pud = (pud_t *)page_address(page); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); return pud; } diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 867e9c3..990f614 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -89,11 +89,16 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { + struct page *page; pud_t *pud; - pud = (pud_t *) __get_free_pages(GFP_KERNEL, PUD_ORDER); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + page = alloc_pages(GFP_KERNEL, PUD_ORDER); + if (!page) + return NULL; + + pgtable_page_inc(page); + pud = (pud_t *)page_address(page); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); return pud; } diff --git a/arch/s390/include/asm/tlb.h b/arch/s390/include/asm/tlb.h index fe6407f..744e2d7 100644 --- a/arch/s390/include/asm/tlb.h +++ b/arch/s390/include/asm/tlb.h @@ -125,6 +125,7 @@ static inline void pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, { if (mm_pud_folded(tlb->mm)) return; + pgtable_page_dec(virt_to_page(pud)); tlb->mm->context.flush_mm = 1; tlb->freed_tables = 1; tlb->cleared_p4ds = 1; diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a932d77..5e46e31 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -76,8 +76,11 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { + struct page *page = virt_to_page(pud); + + pgtable_page_dec(page); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(pud)); + paravirt_tlb_remove_table(tlb, page); } #if CONFIG_PGTABLE_LEVELS > 4 diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 977bea1..11350f7 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -149,11 +149,16 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) { + struct page *page; gfp_t gfp = GFP_PGTABLE_USER; if (mm == &init_mm) gfp = GFP_PGTABLE_KERNEL; - return (pud_t *)get_zeroed_page(gfp); + page = alloc_pages(gfp, 0); + if (!page) + return NULL; + pgtable_page_inc(page); + return (pud_t *)page_address(page); } #ifndef __HAVE_ARCH_PUD_ALLOC_ONE @@ -174,8 +179,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { + struct page *page = virt_to_page(pud); + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - free_page((unsigned long)pud); + pgtable_page_dec(page); + __free_page(page); } #ifndef __HAVE_ARCH_PUD_FREE From patchwork Thu Jun 30 11:11:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12901700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 046B0C43334 for ; Thu, 30 Jun 2022 11:11:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7B7566B0075; Thu, 30 Jun 2022 07:11:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 743246B0078; Thu, 30 Jun 2022 07:11:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5929F8E0001; Thu, 30 Jun 2022 07:11:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4A6756B0075 for ; Thu, 30 Jun 2022 07:11:32 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 2A68F2155F for ; Thu, 30 Jun 2022 11:11:32 +0000 (UTC) X-FDA: 79634636424.16.8A33258 Received: from out199-16.us.a.mail.aliyun.com (out199-16.us.a.mail.aliyun.com [47.90.199.16]) by imf19.hostedemail.com (Postfix) with ESMTP id 40CB21A0039 for ; Thu, 30 Jun 2022 11:11:30 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R191e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045170;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---0VHsIczw_1656587486; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VHsIczw_1656587486) by smtp.aliyun-inc.com; Thu, 30 Jun 2022 19:11:27 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v3 3/3] mm: Add kernel PTE level pagetable pages account Date: Thu, 30 Jun 2022 19:11:16 +0800 Message-Id: <2d09ded7a78fdd531d3c6fa86787ba310bc19dc4.1656586863.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 47.90.199.16 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656587491; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=KTfTVpXq1pC+wRI6aXE2JaobuoCvmon14iSoqu6YGrk=; b=B1mwrVagybPHkzw8QOKdFUAn/WBqPop6sZChTlZ5ZMJb1ZthyyDRDzZj7sS/46ldcc/ZlS Zfcs7VgkK1y+jfKBtF0nZ/1tVXcSJxCF3vWfvoKghQ5p+hN14EdNoe9p2WudJ+YsA0gt6y fFny3wqZsocnMfM9Glc8x0cgx1D+X2o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656587491; a=rsa-sha256; cv=none; b=7uM7PcXwlLg5SFvpp8DScnuYKDRhvxZ2UklF9/A2xB3QyWxvoJj9/L6EFS+Ac1+c75zGeH dIBwxBeDfEE8AP2Lwo54uLIgv3AJKyUR34fPwLkortmTNN5/qJnbaG9Y6CsecDv0F5I49B 2yWMUwCEVgsALSXumZus7WNoy6OVbTs= X-Stat-Signature: 1pz49b7tihyxpujdkkku59fscptt9qfb X-Rspamd-Queue-Id: 40CB21A0039 Authentication-Results: imf19.hostedemail.com; dkim=none; spf=pass (imf19.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 47.90.199.16 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Rspamd-Server: rspam12 X-Rspam-User: X-HE-Tag: 1656587490-697671 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the kernel PTE level ptes are always protected by mm->page_table_lock instead of split pagetable lock, so the kernel PTE level pagetable pages are not accounted. Especially the vmalloc()/vmap() can consume lots of kernel pagetable, so to get an accurate pagetable accounting, calling new helpers pgtable_page_inc()/pgtable_page_dec() when allocating or freeing a kernel PTE level pagetable page. Meanwhile converting architectures to use corresponding generic PTE pagetable allocation and freeing functions. Note this patch only adds accounting to the page tables allocated after boot. Signed-off-by: Baolin Wang Reported-by: kernel test robot --- arch/csky/include/asm/pgalloc.h | 2 +- arch/microblaze/mm/pgtable.c | 2 +- arch/openrisc/mm/ioremap.c | 2 +- arch/x86/mm/pgtable.c | 5 ++++- include/asm-generic/pgalloc.h | 14 ++++++++++++-- 5 files changed, 19 insertions(+), 6 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index bbbd069..2443226 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; unsigned long i; - pte = (pte_t *) __get_free_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); if (!pte) return NULL; diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 9f73265..e96dd1b 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -245,7 +245,7 @@ unsigned long iopa(unsigned long addr) __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { if (mem_init_done) - return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + return __pte_alloc_one_kernel(mm); else return memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE, MEMBLOCK_LOW_LIMIT, diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index daae13a..3453acc 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -118,7 +118,7 @@ pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; if (likely(mem_init_done)) { - pte = (pte_t *)get_zeroed_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); } else { pte = memblock_alloc(PAGE_SIZE, PAGE_SIZE); if (!pte) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 5e46e31..645868b 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -851,6 +851,7 @@ int pud_free_pmd_page(pud_t *pud, unsigned long addr) int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) { pte_t *pte; + struct page *page; pte = (pte_t *)pmd_page_vaddr(*pmd); pmd_clear(pmd); @@ -858,7 +859,9 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) /* INVLPG to clear all paging-structure caches */ flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); - free_page((unsigned long)pte); + page = virt_to_page(pte); + pgtable_page_dec(page); + __free_page(page); return 1; } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 11350f7..e1a6771 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -18,7 +18,14 @@ */ static inline pte_t *__pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(GFP_PGTABLE_KERNEL); + struct page *page; + gfp_t gfp = GFP_PGTABLE_KERNEL; + + page = alloc_pages(gfp, 0); + if (!page) + return NULL; + pgtable_page_inc(page); + return (pte_t *)page_address(page); } #ifndef __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL @@ -41,7 +48,10 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { - free_page((unsigned long)pte); + struct page *page = virt_to_page(pte); + + pgtable_page_dec(page); + __free_page(page); } /**