From patchwork Fri Jul 17 11:10:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 11669885 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3CE121392 for ; Fri, 17 Jul 2020 11:14:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 088F8207E8 for ; Fri, 17 Jul 2020 11:14:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="Y/MXDI0t" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 088F8207E8 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1250D8D0034; Fri, 17 Jul 2020 07:14:25 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 0AEFA8D0030; Fri, 17 Jul 2020 07:14:25 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB7898D0034; Fri, 17 Jul 2020 07:14:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0233.hostedemail.com [216.40.44.233]) by kanga.kvack.org (Postfix) with ESMTP id D09848D0030 for ; Fri, 17 Jul 2020 07:14:24 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 861E1180EAE3F for ; Fri, 17 Jul 2020 11:14:24 +0000 (UTC) X-FDA: 77047309248.30.wish22_09162f926f0a Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 5D6DE181B7E63 for ; Fri, 17 Jul 2020 11:14:24 +0000 (UTC) X-Spam-Summary: 1,0,0,cd9ca5d2b1d3cec8,d41d8cd98f00b204,peterz@infradead.org,,RULES_HIT:41:152:355:379:800:960:965:966:968:973:988:989:1042:1183:1260:1261:1277:1313:1314:1345:1431:1437:1516:1518:1535:1543:1593:1594:1711:1730:1747:1777:1792:1981:2194:2196:2198:2199:2200:2201:2393:2559:2562:2731:2903:3138:3139:3140:3141:3142:3355:3865:3867:3870:3871:3872:3874:4117:4250:4385:4390:4395:4605:5007:6119:6261:6653:6742:7875:7903:8634:10004:10400:11026:11473:11657:11658:11914:12043:12048:12291:12296:12297:12438:12517:12519:12683:14181:14394:14659:14721:14877:21080:21450:21451:21627:21772:21990:30003:30054:30055,0,RBL:205.233.59.134:@infradead.org:.lbl8.mailshell.net-64.201.201.201 62.8.55.100;04y8ih5s9rcqa8m3xciqkebroo9swycj85m53wqr653tnwoae5twyaxi3f4jwga.45yatbotwptbohxc5hu8o41t9we569orwpgqojxyp1sboohbdgtua6ywuy6ozdz.r-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fn,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23, LUA_SUMM X-HE-Tag: wish22_09162f926f0a X-Filterd-Recvd-Size: 6341 Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Fri, 17 Jul 2020 11:14:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=gOQYLPqlkXaE5f1BcTQenQfqzoyHPAVuIDnigWhSoRE=; b=Y/MXDI0tbjGV1H6k+/3UUA8Lu5 zcG4hZO3jyz9bLcRi2dIJysN79dO3dnN/J6fbvIK4gxxQiyBlOKQuuCn0E4ddYCbGpPOTo3VVtlki b8indkgQ/FVRbEsFBMBPCZcy3KunV3UanWjKfz3yzA2GvqjF7q6418pyW6NM0aW8XTHtuYN9wNG74 sxLzg+HiCTzpnr/zc2fq3XNvfm31YDJdhEXKopYgPN6tBPKbu/nVWCyNI6khi3A9VUpNMTP2aF7eI 3elFTvXbPIuOkzZ1J7mS4tlF7nVaUtIA8IAJ8rLPZMHlzNolSp/mIGsc5xtO6SOhkwzpM6jWmaDOb UHPdiS5w==; Received: from j217100.upc-j.chello.nl ([24.132.217.100] helo=noisy.programming.kicks-ass.net) by merlin.infradead.org with esmtpsa (Exim 4.92.3 #3 (Red Hat Linux)) id 1jwOJZ-0001Zn-LT; Fri, 17 Jul 2020 11:14:05 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id 6F914306CCF; Fri, 17 Jul 2020 13:14:04 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id C0DDF203D4095; Fri, 17 Jul 2020 13:14:02 +0200 (CEST) Message-ID: <20200717111349.708289813@infradead.org> User-Agent: quilt/0.66 Date: Fri, 17 Jul 2020 13:10:11 +0200 From: Peter Zijlstra To: Will Deacon , "Aneesh Kumar K.V" , Andrew Morton , Nick Piggin , Peter Zijlstra Cc: linux-arch@vger.kernel.org, linux-sh@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yoshinori Sato , Rich Felker , "David S. Miller" , Helge Deller , Geert Uytterhoeven , Paul Burton , Tony Luck , Richard Henderson , Nick Hu , Paul Walmsley , John Paul Adrian Glaubitz , Christoph Hellwig Subject: [PATCH v2 06/11] mips/tlb: Fix __p*_free_tlb() References: <20200717111005.024867618@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5D6DE181B7E63 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Just like regular pages, page directories need to observe the following order: 1) unhook 2) TLB invalidate 3) free to ensure it is safe against concurrent accesses. MIPS has (thanks to Paul Burton for setting me straight): - HAVE_FAST_GUP, which means we need to consider software walkers (many MIPS also have software TLB-fill, which is a similar software walker). - non-page, page directories, which requires a custom table free. - Mostly IPI-based TLB invalidate, but it wouldn't be MIPS if it didn't also have broadcast TLB invalidate (I6500, GINVT). This means that in generic MIPS needs HAVE_RCU_TABLE_FREE. Signed-off-by: Peter Zijlstra (Intel) --- arch/mips/Kconfig | 1 + arch/mips/include/asm/pgalloc.h | 13 ++++++------- arch/mips/mm/pgtable.c | 34 ++++++++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+), 7 deletions(-) --- a/arch/mips/Kconfig +++ b/arch/mips/Kconfig @@ -84,6 +84,7 @@ config MIPS select HAVE_VIRT_CPU_ACCOUNTING_GEN if 64BIT || !SMP select IRQ_FORCED_THREADING select ISA if EISA + select MMU_GATHER_RCU_TABLE_FREE select MODULES_USE_ELF_REL if MODULES select MODULES_USE_ELF_RELA if MODULES && 64BIT select PERF_USE_VMALLOC --- a/arch/mips/include/asm/pgalloc.h +++ b/arch/mips/include/asm/pgalloc.h @@ -41,6 +41,9 @@ static inline void pud_populate(struct m } #endif +extern void __tlb_remove_table(void *table); +extern void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int idx); + /* * Initialize a new pgd / pmd table with invalid pointers. */ @@ -52,11 +55,7 @@ static inline void pgd_free(struct mm_st free_pages((unsigned long)pgd, PGD_ORDER); } -#define __pte_free_tlb(tlb,pte,address) \ -do { \ - pgtable_pte_page_dtor(pte); \ - tlb_remove_page((tlb), pte); \ -} while (0) +#define __pte_free_tlb(tlb,pte,address) pgtable_free_tlb((tlb), (pte), 0) #ifndef __PAGETABLE_PMD_FOLDED @@ -75,7 +74,7 @@ static inline void pmd_free(struct mm_st free_pages((unsigned long)pmd, PMD_ORDER); } -#define __pmd_free_tlb(tlb, x, addr) pmd_free((tlb)->mm, x) +#define __pmd_free_tlb(tlb, x, addr) pgtable_free_tlb((tlb), (x), 1) #endif @@ -101,7 +100,7 @@ static inline void p4d_populate(struct m set_p4d(p4d, __p4d((unsigned long)pud)); } -#define __pud_free_tlb(tlb, x, addr) pud_free((tlb)->mm, x) +#define __pud_free_tlb(tlb, x, addr) pgtable_free_tlb((tlb), (x), 2) #endif /* __PAGETABLE_PUD_FOLDED */ --- a/arch/mips/mm/pgtable.c +++ b/arch/mips/mm/pgtable.c @@ -7,6 +7,7 @@ #include #include #include +#include pgd_t *pgd_alloc(struct mm_struct *mm) { @@ -23,3 +24,36 @@ pgd_t *pgd_alloc(struct mm_struct *mm) return ret; } EXPORT_SYMBOL_GPL(pgd_alloc); + +#define TLB_IDX_MASK 0x3 + +void pgtable_free_tlb(struct mmu_gather *tlb, void *table, int idx) +{ + unsigned long pgf = (unsigned long)table; + BUG_ON(idx > TLB_IDX_MASK); + BUG_ON(pgf & TLB_IDX_MASK); + pgf |= idx; + tlb_remove_table(tlb, (void *)pgf); +} + +void __tlb_remove_table(void *table) +{ + void *dir = (void *)((unsigned long)table & ~TLB_IDX_MASK); + int idx = (unsigned long)table & TLB_IDX_MASK; + + switch (idx) { +#ifndef __PAGETABLE_PUD_FOLDED + case 2: /* PUD */ + pud_free(NULL, dir); + break; +#endif +#ifndef __PAGETABLE_PMD_FOLDED + case 1: /* PMD */ + pmd_free(NULL, dir); + break; +#endif + case 0: /* PTE */ + pte_free(NULL, dir); + break; + } +}