From patchwork Wed Jul 6 08:59:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12907614 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4C858C43334 for ; Wed, 6 Jul 2022 09:00:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Fk+W9t7F9Ot8799GX0ZK21gV+FPlsWTUeW91JnsRG0A=; b=v7ieDNgN32EuhM iTkY6RBQHuoU7O1WPJe2X/g+LR10UULPo/KVoTcG1jHIOSDy80PLWZNxZEk/Kr3SbrESNQ5BuOgrG WXUxdj7BxiCiYycJMWC+gYwsaX6cOiNSSil4OfvPWuiiIdoaaBA/wtmNL/2GrPCCQAUyoM0y1Aq2L 0WQZgZqm3KsNGzURKJl+RbN2o1+z8qvbMq9+EtV8rNcmOOxwMuzveo4w/X3u1K7cLbfm+lXGBT6ej TKeY6qzML/x+KqnU6UhSIsqaXkS0ScmwhNeYcBXdxWLha3TRnI+7ILVzk2uBVmmXIztk7dI5VxJIJ xKMymPcR0tjkbpuDh2ZQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o90sj-007ZFc-3s; Wed, 06 Jul 2022 08:59:37 +0000 Received: from out30-132.freemail.mail.aliyun.com ([115.124.30.132]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o90sd-007ZCA-7i for linux-arm-kernel@lists.infradead.org; Wed, 06 Jul 2022 08:59:33 +0000 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R721e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046051;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=33;SR=0;TI=SMTPD_---0VIXnCQL_1657097963; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VIXnCQL_1657097963) by smtp.aliyun-inc.com; Wed, 06 Jul 2022 16:59:24 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, will@kernel.org, aneesh.kumar@linux.ibm.com, npiggin@gmail.com, peterz@infradead.org, catalin.marinas@arm.com, chenhuacai@kernel.org, kernel@xen0n.name, tsbogend@alpha.franken.de, dave.hansen@linux.intel.com, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, arnd@arndb.de, guoren@kernel.org, monstr@monstr.eu, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, baolin.wang@linux.alibaba.com, x86@kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-csky@vger.kernel.org, openrisc@lists.librecores.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 1/3] mm: Factor out the pagetable pages account into new helper function Date: Wed, 6 Jul 2022 16:59:15 +0800 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220706_015931_488711_A09EC926 X-CRM114-Status: UNSURE ( 8.89 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Factor out the pagetable pages account and pagetable setting into new helper functions to avoid duplicated code. Meanwhile these helper functions also will be used to account pagetable pages which do not need split pagetable lock. Signed-off-by: Baolin Wang --- include/linux/mm.h | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index d084ce5..7894bc5 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2352,20 +2352,30 @@ static inline void pgtable_init(void) pgtable_cache_init(); } +static inline void page_set_pgtable(struct page *page) +{ + __SetPageTable(page); + inc_lruvec_page_state(page, NR_PAGETABLE); +} + +static inline void page_clear_pgtable(struct page *page) +{ + __ClearPageTable(page); + dec_lruvec_page_state(page, NR_PAGETABLE); +} + static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + page_set_pgtable(page); return true; } static inline void pgtable_pte_page_dtor(struct page *page) { ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + page_clear_pgtable(page); } #define pte_offset_map_lock(mm, pmd, address, ptlp) \ @@ -2451,16 +2461,14 @@ static inline bool pgtable_pmd_page_ctor(struct page *page) { if (!pmd_ptlock_init(page)) return false; - __SetPageTable(page); - inc_lruvec_page_state(page, NR_PAGETABLE); + page_set_pgtable(page); return true; } static inline void pgtable_pmd_page_dtor(struct page *page) { pmd_ptlock_free(page); - __ClearPageTable(page); - dec_lruvec_page_state(page, NR_PAGETABLE); + page_clear_pgtable(page); } /* From patchwork Wed Jul 6 08:59:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12907616 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E3DDC43334 for ; Wed, 6 Jul 2022 09:01:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=YOcppxs1qWFpcUc8TWhy83uQQpPyXYn2H51wcdaR5Vs=; b=lrkK5NitQOxhkH qbBp4jAZ4szofHtyyD5vVZLW159GaW8xHV7CaLUdcPJcm8Hqn33+cJtVtPW8nNODzzyImUEi3gJzv tg5hNyEE+6w47eS5KmIL7j2FuTSiVDNs6ViBeqpna+60/WDSejCXct9OLkFpnX2tbtllYudv3RGnD uSzgoBj2bclDtHtKvAnI7I75NctTy/WOuItFUyiUaV6KxSMMkKdvEUIw+p71RJBJee1ERo6RiraMj yyRqWQaHLvFmumiLn1RBn9qgZlxKU39IiKF64DOC4c+duiAvP6xqyhftvDvgXAHCO34FWGU+LZDKO 7lTCHoEPQqkeKktLBMgQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o90tD-007ZWc-Nf; Wed, 06 Jul 2022 09:00:07 +0000 Received: from out30-57.freemail.mail.aliyun.com ([115.124.30.57]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o90sg-007ZCi-SW for linux-arm-kernel@lists.infradead.org; Wed, 06 Jul 2022 08:59:36 +0000 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R131e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018045168;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=33;SR=0;TI=SMTPD_---0VIXgrTG_1657097965; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VIXgrTG_1657097965) by smtp.aliyun-inc.com; Wed, 06 Jul 2022 16:59:26 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, will@kernel.org, aneesh.kumar@linux.ibm.com, npiggin@gmail.com, peterz@infradead.org, catalin.marinas@arm.com, chenhuacai@kernel.org, kernel@xen0n.name, tsbogend@alpha.franken.de, dave.hansen@linux.intel.com, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, arnd@arndb.de, guoren@kernel.org, monstr@monstr.eu, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, baolin.wang@linux.alibaba.com, x86@kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-csky@vger.kernel.org, openrisc@lists.librecores.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 2/3] mm: Add PUD level pagetable account Date: Wed, 6 Jul 2022 16:59:16 +0800 Message-Id: <6c7316e17ccf8d9343c5f703c272df51346b44fa.1657096412.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220706_015935_348725_9AABB52D X-CRM114-Status: GOOD ( 11.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now the PUD level ptes are always protected by mm->page_table_lock, which means no split pagetable lock needed. So the generic PUD level pagetable pages allocation will not call pgtable_pte_page_ctor/dtor(), that means we will miss to account PUD level pagetable pages. So introducing pgtable_pud_page_ctor/dtor(), which are just wrappers of page_{set,clear}_pgtable() to help to get an accurate PUD pagetable accounting, when allocating or freeing PUD level pagetable pages. Moreover this patch will also mark the PUD level pagetable with PG_table flag, which will help to do sanity validation in unpoison_memory() and get more accurate pagetable accounting by /proc/kpageflags interface. Meanwhile converting the architectures with using generic PUD pagatable allocation to add corresponding pgtable_pud_page_ctor() or pgtable_pud_page_dtor() to account PUD level pagetable. Signed-off-by: Baolin Wang --- arch/arm64/include/asm/tlb.h | 5 ++++- arch/loongarch/include/asm/pgalloc.h | 12 +++++++++--- arch/mips/include/asm/pgalloc.h | 12 +++++++++--- arch/x86/mm/pgtable.c | 5 ++++- include/asm-generic/pgalloc.h | 12 ++++++++++-- include/linux/mm.h | 10 ++++++++++ 6 files changed, 46 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index c995d1f..6665f33 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -94,7 +94,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pudp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pudp)); + struct page *page = virt_to_page(pudp); + + pgtable_pud_page_dtor(page); + tlb_remove_table(tlb, page); } #endif diff --git a/arch/loongarch/include/asm/pgalloc.h b/arch/loongarch/include/asm/pgalloc.h index 4bfeb3c..8138101 100644 --- a/arch/loongarch/include/asm/pgalloc.h +++ b/arch/loongarch/include/asm/pgalloc.h @@ -89,10 +89,16 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { pud_t *pud; + struct page *page; + + page = alloc_page(GFP_KERNEL); + if (!page) + return NULL; + + pgtable_pud_page_ctor(page); + pud = (pud_t *)page_address(page); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); - pud = (pud_t *) __get_free_page(GFP_KERNEL); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); return pud; } diff --git a/arch/mips/include/asm/pgalloc.h b/arch/mips/include/asm/pgalloc.h index 7960357..5da5880 100644 --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -89,11 +89,17 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long address) static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long address) { + struct page *page; pud_t *pud; - pud = (pud_t *) __get_free_pages(GFP_KERNEL, PUD_TABLE_ORDER); - if (pud) - pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + page = alloc_pages(GFP_KERNEL, PUD_TABLE_ORDER); + if (!page) + return NULL; + + pgtable_pud_page_ctor(page); + pud = (pud_t *)page_address(page); + pud_init((unsigned long)pud, (unsigned long)invalid_pmd_table); + return pud; } diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index a932d77..ea39670 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -76,8 +76,11 @@ void ___pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd) #if CONFIG_PGTABLE_LEVELS > 3 void ___pud_free_tlb(struct mmu_gather *tlb, pud_t *pud) { + struct page *page = virt_to_page(pud); + + pgtable_pud_page_dtor(page); paravirt_release_pud(__pa(pud) >> PAGE_SHIFT); - paravirt_tlb_remove_table(tlb, virt_to_page(pud)); + paravirt_tlb_remove_table(tlb, page); } #if CONFIG_PGTABLE_LEVELS > 4 diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 977bea1..8ce8d7c 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -149,11 +149,16 @@ static inline void pmd_free(struct mm_struct *mm, pmd_t *pmd) static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) { + struct page *page; gfp_t gfp = GFP_PGTABLE_USER; if (mm == &init_mm) gfp = GFP_PGTABLE_KERNEL; - return (pud_t *)get_zeroed_page(gfp); + page = alloc_pages(gfp, 0); + if (!page) + return NULL; + pgtable_pud_page_ctor(page); + return (pud_t *)page_address(page); } #ifndef __HAVE_ARCH_PUD_ALLOC_ONE @@ -174,8 +179,11 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) static inline void __pud_free(struct mm_struct *mm, pud_t *pud) { + struct page *page = virt_to_page(pud); + BUG_ON((unsigned long)pud & (PAGE_SIZE-1)); - free_page((unsigned long)pud); + pgtable_pud_page_dtor(page); + __free_page(page); } #ifndef __HAVE_ARCH_PUD_FREE diff --git a/include/linux/mm.h b/include/linux/mm.h index 7894bc5..54ed6f7 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2364,6 +2364,16 @@ static inline void page_clear_pgtable(struct page *page) dec_lruvec_page_state(page, NR_PAGETABLE); } +static inline void pgtable_pud_page_ctor(struct page *page) +{ + page_set_pgtable(page); +} + +static inline void pgtable_pud_page_dtor(struct page *page) +{ + page_clear_pgtable(page); +} + static inline bool pgtable_pte_page_ctor(struct page *page) { if (!ptlock_init(page)) From patchwork Wed Jul 6 08:59:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 12907617 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31B27C43334 for ; Wed, 6 Jul 2022 09:01:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:List-Subscribe:List-Help: List-Post:List-Archive:List-Unsubscribe:List-Id:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=TVrmvaKMkrK2WGMon86n+m1dYTO9lri1GZiusVvBnzw=; b=c/F57F5x8+UV7S 06f23AaU8WIeJQA440af4CYvBxTvZbB4/DKfXGeCbwN6iUCzaUuiyPS2lIObFd5vWpFewpy7wmL/S F46sX0rE7h7soBTu70HFQ2rxCkLc6JxdbHTPPE0AiVZdYW7pd5Vv0TJGuTAld6KQkDj+W1QUdVO4m rwo9ZUChJGWXdOkYU4Iquzth8LSTrJCdMpzHUP98BYRVF4WrTuHi8EIa+/SMFeaC+ql2bgNFp25n0 /ervwjym6EG/74nAffmnWP92nNlYnKxQcY7VL3pkjArd1pq24Dllr4Q968zPLuHDT3xO4WoBzSy3G LG70KRdsB26al+UrDuOg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1o90tP-007Zal-E2; Wed, 06 Jul 2022 09:00:19 +0000 Received: from out199-2.us.a.mail.aliyun.com ([47.90.199.2]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1o90sm-007ZGq-RG for linux-arm-kernel@lists.infradead.org; Wed, 06 Jul 2022 08:59:44 +0000 X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R151e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018046059;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=33;SR=0;TI=SMTPD_---0VIXd7l-_1657097967; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0VIXd7l-_1657097967) by smtp.aliyun-inc.com; Wed, 06 Jul 2022 16:59:29 +0800 From: Baolin Wang To: akpm@linux-foundation.org Cc: rppt@linux.ibm.com, willy@infradead.org, will@kernel.org, aneesh.kumar@linux.ibm.com, npiggin@gmail.com, peterz@infradead.org, catalin.marinas@arm.com, chenhuacai@kernel.org, kernel@xen0n.name, tsbogend@alpha.franken.de, dave.hansen@linux.intel.com, luto@kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, arnd@arndb.de, guoren@kernel.org, monstr@monstr.eu, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, baolin.wang@linux.alibaba.com, x86@kernel.org, linux-arch@vger.kernel.org, linux-arm-kernel@lists.infradead.org, loongarch@lists.linux.dev, linux-mips@vger.kernel.org, linux-csky@vger.kernel.org, openrisc@lists.librecores.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 3/3] mm: Add kernel PTE level pagetable pages account Date: Wed, 6 Jul 2022 16:59:17 +0800 Message-Id: <398ead25695e530f766849be5edafaf62c1c864d.1657096412.git.baolin.wang@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: In-Reply-To: References: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220706_015941_218244_B13F5C0B X-CRM114-Status: GOOD ( 12.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now the kernel PTE level ptes are always protected by mm->page_table_lock instead of split pagetable lock, so the kernel PTE level pagetable pages are not accounted. Especially the vmalloc()/vmap() can consume lots of kernel pagetable, so to get an accurate pagetable accounting, calling new helpers page_{set,clear}_pgtable() when allocating or freeing a kernel PTE level pagetable page. Meanwhile converting architectures to use corresponding generic PTE pagetable allocation and freeing functions. Note this patch only adds accounting to the page tables allocated after boot. Signed-off-by: Baolin Wang Reported-by: kernel test robot --- arch/csky/include/asm/pgalloc.h | 2 +- arch/microblaze/mm/pgtable.c | 2 +- arch/openrisc/mm/ioremap.c | 2 +- arch/x86/mm/pgtable.c | 2 +- include/asm-generic/pgalloc.h | 14 ++++++++++++-- 5 files changed, 16 insertions(+), 6 deletions(-) diff --git a/arch/csky/include/asm/pgalloc.h b/arch/csky/include/asm/pgalloc.h index 7d57e5d..56f8d25 100644 --- a/arch/csky/include/asm/pgalloc.h +++ b/arch/csky/include/asm/pgalloc.h @@ -29,7 +29,7 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; unsigned long i; - pte = (pte_t *) __get_free_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); if (!pte) return NULL; diff --git a/arch/microblaze/mm/pgtable.c b/arch/microblaze/mm/pgtable.c index 9f73265..e96dd1b 100644 --- a/arch/microblaze/mm/pgtable.c +++ b/arch/microblaze/mm/pgtable.c @@ -245,7 +245,7 @@ unsigned long iopa(unsigned long addr) __ref pte_t *pte_alloc_one_kernel(struct mm_struct *mm) { if (mem_init_done) - return (pte_t *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + return __pte_alloc_one_kernel(mm); else return memblock_alloc_try_nid(PAGE_SIZE, PAGE_SIZE, MEMBLOCK_LOW_LIMIT, diff --git a/arch/openrisc/mm/ioremap.c b/arch/openrisc/mm/ioremap.c index daae13a..3453acc 100644 --- a/arch/openrisc/mm/ioremap.c +++ b/arch/openrisc/mm/ioremap.c @@ -118,7 +118,7 @@ pte_t __ref *pte_alloc_one_kernel(struct mm_struct *mm) pte_t *pte; if (likely(mem_init_done)) { - pte = (pte_t *)get_zeroed_page(GFP_KERNEL); + pte = __pte_alloc_one_kernel(mm); } else { pte = memblock_alloc(PAGE_SIZE, PAGE_SIZE); if (!pte) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index ea39670..20f3076 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -858,7 +858,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) /* INVLPG to clear all paging-structure caches */ flush_tlb_kernel_range(addr, addr + PAGE_SIZE-1); - free_page((unsigned long)pte); + pte_free_kernel(NULL, pte); return 1; } diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index 8ce8d7c..cd8420f 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -18,7 +18,14 @@ */ static inline pte_t *__pte_alloc_one_kernel(struct mm_struct *mm) { - return (pte_t *)__get_free_page(GFP_PGTABLE_KERNEL); + struct page *page; + gfp_t gfp = GFP_PGTABLE_KERNEL; + + page = alloc_pages(gfp, 0); + if (!page) + return NULL; + page_set_pgtable(page); + return (pte_t *)page_address(page); } #ifndef __HAVE_ARCH_PTE_ALLOC_ONE_KERNEL @@ -41,7 +48,10 @@ static inline pte_t *pte_alloc_one_kernel(struct mm_struct *mm) */ static inline void pte_free_kernel(struct mm_struct *mm, pte_t *pte) { - free_page((unsigned long)pte); + struct page *page = virt_to_page(pte); + + page_clear_pgtable(page); + __free_page(page); } /**