From patchwork Wed Jul 6 08:59:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12907611 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5534ECCA473 for ; Wed, 6 Jul 2022 08:59:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1B7D18E0001; Wed, 6 Jul 2022 04:59:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 167816B0074; Wed, 6 Jul 2022 04:59:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02D3C8E0001; Wed, 6 Jul 2022 04:59:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E93BA6B0073 for ; Wed, 6 Jul 2022 04:59:31 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id AC53D8032E for ; Wed, 6 Jul 2022 08:59:31 +0000 (UTC) X-FDA: 79656076542.22.A7F4A23 Received: from out30-54.freemail.mail.aliyun.com (out30-54.freemail.mail.aliyun.com [115.124.30.54]) by imf01.hostedemail.com (Postfix) with ESMTP id A9E9F4000D for ; Wed, 6 Jul 2022 08:59:30 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R721e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=33;SR=0;TI=SMTPD_---0VIXnCQL_1657097963; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VIXnCQL_1657097963) by smtp.aliyun-inc.com; Wed, 06 Jul 2022 16:59:24 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, will@kernel.org, aneesh.kumar@linux.ibm.com, npiggin@gmail.com, peterz@infradead.org, catalin.marinas@arm.com, chenhuacai@kernel.org, kernel@xen0n.name, tsbogend@alpha.franken.de, dave.hansen@linux.intel.com, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, arnd@arndb.de, guoren@kernel.org, monstr@monstr.eu, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, baolin.wang@linux.alibaba.com, x86@kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-csky@vger.kernel.org, openrisc@lists.librecores.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] mm: Factor out the pagetable pages account into new helper function Date: Wed, 6 Jul 2022 16:59:15 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657097971; a=rsa-sha256; cv=none; b=UF5vVfH1QidDtpy1BdJF9Nc8aDqMCBR7MpUUq8WccIMcnk3X4W9SQ8+zsJH0BPfcTwCrPi 5OJThJaoy1F6bvCDkrmWMI0Qdikt6VRWqtKZGxZ1txelfl/bzIpux1Spr9X8JNqmCiScHB cHNbuXnJSyQITcj0m6ELDRcqNKnU0cE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.54 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657097971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=ZmiUTmLc/9slSyYIwDxU7koFVooiXbLhYPhlCaL32mA=; b=p2j4yfENu3H7hzfHB29SzmKxJFoufvhIS7ymXYLmifi9aLyIW4Y/6Wte6OxAcGS95b7O2p qJYIuz3QelX9Mh3YZyIubWHO9F3pQ3DRNbT/NqqkRTXtm0Bo2g4cGbrqytHkOcpw4N6w2U Jsw20X8fdbFSA0k/F03RYB0IoQ76vgo= X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A9E9F4000D X-Rspam-User: Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.54 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Stat-Signature: 99psgmmipbtf1yks1twa95cmuy1x1wjt X-HE-Tag: 1657097970-830086 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Factor out the pagetable pages account and pagetable setting into new helper functions to avoid duplicated code. Meanwhile these helper functions also will be used to account pagetable pages which do not need split pagetable lock. Signed-off-by: Baolin Wang --- include/linux/mm.h | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d084ce5..7894bc5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2352,20 +2352,30 @@ static inline void pgtable_init(void) pgtable_cache_init(); } +static inline void page_set_pgtable(struct page *page) +{ + __SetPageTable(page); + inc_lruvec_page_state(page, NR_PAGETABLE); +} + +static inline void page_clear_pgtable(struct page *page) +{ + __ClearPageTable(page); + dec_lruvec_page_state(page, NR_PAGETABLE); +} + static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + page_set_pgtable(page); return true; } static inline void pgtable_pte_page_dtor(struct page *page) { ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + page_clear_pgtable(page); } #define pte_offset_map_lock(mm, pmd, address, ptlp) \ @@ -2451,16 +2461,14 @@ static inline bool pgtable_pmd_page_ctor(struct page *page) { if (!pmd_ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + page_set_pgtable(page); return true; } static inline void pgtable_pmd_page_dtor(struct page *page) { pmd_ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + page_clear_pgtable(page); } /* From patchwork Wed Jul 6 08:59:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12907612 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7BCBC43334 for ; Wed, 6 Jul 2022 08:59:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3660B8E0002; Wed, 6 Jul 2022 04:59:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 253866B0074; Wed, 6 Jul 2022 04:59:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A5098E0002; Wed, 6 Jul 2022 04:59:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DCA6F6B0073 for ; Wed, 6 Jul 2022 04:59:34 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id AC5B81339 for ; Wed, 6 Jul 2022 08:59:34 +0000 (UTC) X-FDA: 79656076668.25.A449B22 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf26.hostedemail.com (Postfix) with ESMTP id BC2EA140028 for ; Wed, 6 Jul 2022 08:59:33 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045168;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=33;SR=0;TI=SMTPD_---0VIXgrTG_1657097965; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VIXgrTG_1657097965) by smtp.aliyun-inc.com; Wed, 06 Jul 2022 16:59:26 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, will@kernel.org, aneesh.kumar@linux.ibm.com, npiggin@gmail.com, peterz@infradead.org, catalin.marinas@arm.com, chenhuacai@kernel.org, kernel@xen0n.name, tsbogend@alpha.franken.de, dave.hansen@linux.intel.com, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, arnd@arndb.de, guoren@kernel.org, monstr@monstr.eu, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, baolin.wang@linux.alibaba.com, x86@kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-csky@vger.kernel.org, openrisc@lists.librecores.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] mm: Add PUD level pagetable account Date: Wed, 6 Jul 2022 16:59:16 +0800 Message-Id: <6c7316e17ccf8d9343c5f703c272df51346b44fa.1657096412.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657097974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=yvin7AH2CNY9YqP+KRAyKuqjif7PVfQGPzYW0l6ytYs=; b=BAdcC7R4AELwJpV6M+E3M0Vk+/HG9ZvF3jbPDd3MTgq+p3vZd0Cs3K9Cw6kU4ik+q3yrZ4 hbkBQ5p6z2Diqmyh6UReHjFML+cJYFHNHg+Vh/hrM6cXqH/83ZbsAXyRE3zUePpuVE5jMx coZEkvaRXPjLy3QxNtfZ+g9yVxFrWSQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657097974; a=rsa-sha256; cv=none; b=hGxMp2hcdgbAQTRMjvUtoclcynExG2HrXbA8VqPWXdViaAYkQIBzijfQYr0izwPn3DiUgH gqE/PoU/ZOpWnGs0GcqJCNfHc8P6O+TlVJCKGdNZnw3FWyKF5xOvbgAT9+FWogWDV7XD6a 79vKUI+pHEFYeX4cwH3eXXNt1LGAMSw= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf26.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com X-Stat-Signature: 7hsjpod6jqxg6dtpd11t1mkqntqzofpw X-Rspamd-Queue-Id: BC2EA140028 X-Rspam-User: Authentication-Results: imf26.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=alibaba.com; spf=pass (imf26.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.130 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com X-Rspamd-Server: rspam10 X-HE-Tag: 1657097973-369638 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the PUD level ptes are always protected by mm->page_table_lock, which means no split pagetable lock needed. So the generic PUD level pagetable pages allocation will not call pgtable_pte_page_ctor/dtor(), that means we will miss to account PUD level pagetable pages. So introducing pgtable_pud_page_ctor/dtor(), which are just wrappers of page_{set,clear}_pgtable() to help to get an accurate PUD pagetable accounting, when allocating or freeing PUD level pagetable pages. Moreover this patch will also mark the PUD level pagetable with PG_table flag, which will help to do sanity validation in unpoison_memory() and get more accurate pagetable accounting by /proc/kpageflags interface. Meanwhile converting the architectures with using generic PUD pagatable allocation to add corresponding pgtable_pud_page_ctor() or pgtable_pud_page_dtor() to account PUD level pagetable. Signed-off-by: Baolin Wang --- arch/arm64/include/asm/tlb.h | 5 ++++- arch/loongarch/include/asm/pgalloc.h | 12 +++++++++--- arch/mips/include/asm/pgalloc.h | 12 +++++++++--- arch/x86/mm/pgtable.c | 5 ++++- include/asm-generic/pgalloc.h | 12 ++++++++++-- include/linux/mm.h | 10 ++++++++++ 6 files changed, 46 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index c995d1f..6665f33 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -94,7 +94,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pudp)); + struct page *page = virt_to_page(pudp); + + pgtable_pud_page_dtor(page); + tlb_remove_table(tlb, page); } #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 4bfeb3c..8138101 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -89,10 +89,16 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { pud_t *pud; + struct page *page; + + page = alloc_page(GFP_KERNEL); + if (!page) + return NULL; + + pgtable_pud_page_ctor(page); + pud = (pud_t *)page_address(page); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); - pud = (pud_t *) __get_free_page(GFP_KERNEL); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); return pud; } diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 7960357..5da5880 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -89,11 +89,17 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { + struct page *page; pud_t *pud; - pud = (pud_t *) __get_free_pages(GFP_KERNEL, PUD_TABLE_ORDER); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + page = alloc_pages(GFP_KERNEL, PUD_TABLE_ORDER); + if (!page) + return NULL; + + pgtable_pud_page_ctor(page); + pud = (pud_t *)page_address(page); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + return pud; } diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a932d77..ea39670 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -76,8 +76,11 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { + struct page *page = virt_to_page(pud); + + pgtable_pud_page_dtor(page); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(pud)); + paravirt_tlb_remove_table(tlb, page); } #if CONFIG_PGTABLE_LEVELS > 4 diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 977bea1..8ce8d7c 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -149,11 +149,16 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) { + struct page *page; gfp_t gfp = GFP_PGTABLE_USER; if (mm == &init_mm) gfp = GFP_PGTABLE_KERNEL; - return (pud_t *)get_zeroed_page(gfp); + page = alloc_pages(gfp, 0); + if (!page) + return NULL; + pgtable_pud_page_ctor(page); + return (pud_t *)page_address(page); } #ifndef __HAVE_ARCH_PUD_ALLOC_ONE @@ -174,8 +179,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { + struct page *page = virt_to_page(pud); + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - free_page((unsigned long)pud); + pgtable_pud_page_dtor(page); + __free_page(page); } #ifndef __HAVE_ARCH_PUD_FREE diff --git a/include/linux/mm.h b/include/linux/mm.h index 7894bc5..54ed6f7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2364,6 +2364,16 @@ static inline void page_clear_pgtable(struct page *page) dec_lruvec_page_state(page, NR_PAGETABLE); } +static inline void pgtable_pud_page_ctor(struct page *page) +{ + page_set_pgtable(page); +} + +static inline void pgtable_pud_page_dtor(struct page *page) +{ + page_clear_pgtable(page); +} + static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) From patchwork Wed Jul 6 08:59:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12907613 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91C36CCA47C for ; Wed, 6 Jul 2022 08:59:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 316388E0003; Wed, 6 Jul 2022 04:59:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C5FC6B0074; Wed, 6 Jul 2022 04:59:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 18E978E0003; Wed, 6 Jul 2022 04:59:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 090E76B0073 for ; Wed, 6 Jul 2022 04:59:39 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay12.hostedemail.com (Postfix) with ESMTP id CAA51120C01 for ; Wed, 6 Jul 2022 08:59:38 +0000 (UTC) X-FDA: 79656076836.04.323C570 Received: from out30-43.freemail.mail.aliyun.com (out30-43.freemail.mail.aliyun.com [115.124.30.43]) by imf23.hostedemail.com (Postfix) with ESMTP id AB63A140014 for ; Wed, 6 Jul 2022 08:59:37 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=33;SR=0;TI=SMTPD_---0VIXd7l-_1657097967; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VIXd7l-_1657097967) by smtp.aliyun-inc.com; Wed, 06 Jul 2022 16:59:29 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, will@kernel.org, aneesh.kumar@linux.ibm.com, npiggin@gmail.com, peterz@infradead.org, catalin.marinas@arm.com, chenhuacai@kernel.org, kernel@xen0n.name, tsbogend@alpha.franken.de, dave.hansen@linux.intel.com, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, arnd@arndb.de, guoren@kernel.org, monstr@monstr.eu, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, baolin.wang@linux.alibaba.com, x86@kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-csky@vger.kernel.org, openrisc@lists.librecores.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] mm: Add kernel PTE level pagetable pages account Date: Wed, 6 Jul 2022 16:59:17 +0800 Message-Id: <398ead25695e530f766849be5edafaf62c1c864d.1657096412.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1657097978; a=rsa-sha256; cv=none; b=x0KzpjFY669OIL9rkIyrfAVeEYeMXASFnHLtnsIA3+6/1C7wMa7kNeiB0zsxjlaZwTNAVL Q6OO8Lkciq3ifGmiQVuHVF/uzXvWwaOQOl+3+2BN65zBOe8hAXAAbCyD3aWzX1ImPMTZRQ bpKsjS7v/n1hUb5HwjpWUmUbnWW5Eho= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none; spf=pass (imf23.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.43 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1657097978; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:content-type: content-transfer-encoding:in-reply-to:in-reply-to:in-reply-to: references:references:references; bh=V1sF18Otxk1pJCFyw2S8lFHKn6XclY/sq7caOiaZ+1E=; b=WxlRlaSfiEc1knIssypqR85beAAwxGRUhRMaAPbCVMjqqB4X9qAy/vBtWZJjZJyOB806NY h/nt5930QjFWoXdiSP3+3+heU/HZAHQ1T9RS0Vg59x3fGV8LQMOJbwONb5HLjJBQ455ljV V2+jgW7+idB27oEPTNeh8UxjeohJiNc= Authentication-Results: imf23.hostedemail.com; dkim=none; spf=pass (imf23.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.43 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=alibaba.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: kd7iidnrw9hef5jjmmz757fmihhq79tk X-Rspamd-Queue-Id: AB63A140014 X-HE-Tag: 1657097977-924211 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now the kernel PTE level ptes are always protected by mm->page_table_lock instead of split pagetable lock, so the kernel PTE level pagetable pages are not accounted. Especially the vmalloc()/vmap() can consume lots of kernel pagetable, so to get an accurate pagetable accounting, calling new helpers page_{set,clear}_pgtable() when allocating or freeing a kernel PTE level pagetable page. Meanwhile converting architectures to use corresponding generic PTE pagetable allocation and freeing functions. Note this patch only adds accounting to the page tables allocated after boot. Signed-off-by: Baolin Wang Reported-by: kernel test robot --- arch/csky/include/asm/pgalloc.h | 2 +- arch/microblaze/mm/pgtable.c | 2 +- arch/openrisc/mm/ioremap.c | 2 +- arch/x86/mm/pgtable.c | 2 +- include/asm-generic/pgalloc.h | 14 ++++++++++++-- 5 files changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 7d57e5d..56f8d25 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; unsigned long i; - pte = (pte_t *) __get_free_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); if (!pte) return NULL; diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 9f73265..e96dd1b 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -245,7 +245,7 @@ unsigned long iopa(unsigned long addr) __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { if (mem_init_done) - return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + return __pte_alloc_one_kernel(mm); else return memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE, MEMBLOCK_LOW_LIMIT, diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index daae13a..3453acc 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -118,7 +118,7 @@ pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; if (likely(mem_init_done)) { - pte = (pte_t *)get_zeroed_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); } else { pte = memblock_alloc(PAGE_SIZE, PAGE_SIZE); if (!pte) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index ea39670..20f3076 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -858,7 +858,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) /* INVLPG to clear all paging-structure caches */ flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); - free_page((unsigned long)pte); + pte_free_kernel(NULL, pte); return 1; } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 8ce8d7c..cd8420f 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -18,7 +18,14 @@ */ static inline pte_t *__pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(GFP_PGTABLE_KERNEL); + struct page *page; + gfp_t gfp = GFP_PGTABLE_KERNEL; + + page = alloc_pages(gfp, 0); + if (!page) + return NULL; + page_set_pgtable(page); + return (pte_t *)page_address(page); } #ifndef __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL @@ -41,7 +48,10 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { - free_page((unsigned long)pte); + struct page *page = virt_to_page(pte); + + page_clear_pgtable(page); + __free_page(page); } /**