From patchwork Tue Dec 19 17:50:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13498797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 021F6C46CCD for ; Tue, 19 Dec 2023 18:03:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 72C448D0006; Tue, 19 Dec 2023 13:03:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6DE308D0001; Tue, 19 Dec 2023 13:03:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C3218D0006; Tue, 19 Dec 2023 13:03:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 397FA6B0078 for ; Tue, 19 Dec 2023 13:03:31 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 088B7A208F for ; Tue, 19 Dec 2023 18:03:31 +0000 (UTC) X-FDA: 81584340222.29.7E31D3D Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf27.hostedemail.com (Postfix) with ESMTP id EE8FC40013 for ; Tue, 19 Dec 2023 18:03:28 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NZ+HplTE; spf=pass (imf27.hostedemail.com: domain of jszhang@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703009009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RQg7suMb/9Jnz+wAQ9WdVfRyNeBfCryQ9+DO8mQ1BQw=; b=G7+u82//Di0GJdjLjttmZjACjBe5IWNEBssUOjIGWxTFLPy1S1FeKcPzeT59syA/SyVs+W 4ToQWwMY/9NkIDaYonVsxTpPTEuhTxyg74Ui0nzP5SCxPLon4z2W5trUAJuE8WEhsUHo1b MOBn1L0b273kD5RTTZnfPfzD1n/TAm4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703009009; a=rsa-sha256; cv=none; b=sQYQA4GaQHlKdSQH3a1f3PQ6oejvRewyqSrqco1GOwBGCcljIwFo3b/ZTmN2V7VNABeCf4 +CVJvivBiq55vucuNn+BsELjpBH5aRX5bQx4hXWB2rhnPRfyxCByWZ70Tp9DZn8X5rI+qD idDWfHkL6KdqmltsSxJcc8wuJy4vs3Q= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=NZ+HplTE; spf=pass (imf27.hostedemail.com: domain of jszhang@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=jszhang@kernel.org; dmarc=pass (policy=none) header.from=kernel.org Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 03F7BCE1AA9; Tue, 19 Dec 2023 18:03:26 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D62CDC433C9; Tue, 19 Dec 2023 18:03:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703009005; bh=39JAuH598M6uo4CmVtLFeafqNHIttPmqjwfQDvfhOjs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NZ+HplTEbztnxahsANz3uYQlJxV91QZtNCH9i3pK9wbRFkjwTt4VzJXgiEfG1XQyS BXqJkmD+ZuHI3pJbq7hpWh4FazAdi4Y3yKeIPf67IpkIahu9t2PTGodx/3z22CFrV7 m+cNydw6UBPt2db6BACnS7usZ86CqOzEjfRnK1vUXTwLBuz8h9WArwgkfMCsadNvpZ ADKvoV3VZLzOxA8w0qKo27CtkfmEvEQi0IGRe7dYSov3cTESJzZ5pYlOHNLP8W51Mp QrorLyrRGypL6qE5j1GbuY+013Qauw6noSlblzldFoZ80Yl2nqBBjJIFW2VycqEPjS p8/QIHPpMR03g== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 1/4] riscv: tlb: fix __p*d_free_tlb() Date: Wed, 20 Dec 2023 01:50:43 +0800 Message-Id: <20231219175046.2496-2-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231219175046.2496-1-jszhang@kernel.org> References: <20231219175046.2496-1-jszhang@kernel.org> MIME-Version: 1.0 X-Stat-Signature: f5em9x4s5pr8ezest8mc4d13ahro36ee X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: EE8FC40013 X-Rspam-User: X-HE-Tag: 1703009008-604613 X-HE-Meta: U2FsdGVkX19PrHbHdgzhro9bhGbD9OwqnMQN3jQhdHDw5NzI5DctBVGyVJyYD8nThm+DeJXsqybx1+r4692lfEiX0nMa1Ktepb0nl1LJoDmMy/yuQW0nEkIVQ2OFrFQYeiaXyhEU9Bz1298ddEvYVPW4sqoGqtD3pFkiNzcbHgiuurOQb3EqRlkra8S7gn07gGG4a2C20YG8zgq6aQsh+0lLvxY8xmS/CFeMGyVSgDYAO2s8x3XW7zu8setCdL/v+UawibrmZBv7RRwqD4Y0eTYVm6d/MOhAlA8CVZKoQT1i1lUiYwCgGIxzmbkf+aHmG/QSZI5IG83tGHpAadEEunv76ECfsj5cAG9JdwHwK5FMzJ2mVUp1kt6iAXk/34Rx2ZHltD629bxwYrkzcQToM3L2JZy0E/QqSh0FSWA80HffILLD5ZueSWfmHwRYqF8xZAfCWsii17nPgLxs9ZPaFvD3SOuAnKNsO6mAKodo2jxKXCQp/T/xkVyzeiUQW9TnErUQILFV1zODi30VerDws9lVhy+SSxkfPucvHlX63FqgjTfGCMul265aEOr30SB+veb4LZpeI04wUwnctOG73j8d7ZnANRaq6CPMl1SziIFWfkxSzTHu/YMBeL+DIqLyDen3JuSEAN5HoaJdYu9rntaNl71OVyT6tWciLihswwqvJ2dgKfV2LXHzB9Ds4kU6z2Fkyo84JjnPChPnmHdTU8ssQfsf5dKsRk0J1RkpMAzhKWR25VuoWD9hJ3AlzxUWLh1IEU9daexI3kgeVh7ruYRnUwufhAVyE8xrHLkoRxKacD8RKl6p/+Y5jcSqqgsEBv4ICR6a7faOqqU8aQe4Xjxqu4lf/QqiIFAoccsbDD3PlcDLxDLI2IDsAYogGCgdIwFO2jlze1VrnCSGqwiVfnwJWcsKAr9AFVfyP8sViAbdedhtVZp95fHZOcekm792i2Dk6ck+NHySgbnuaMi PhcER3Bq a6noK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If non-leaf PTEs I.E pmd, pud or p4d is modified, a sfence.vma is a must for safe, imagine if an implementation caches the non-leaf translation in TLB, although I didn't meet this HW so far, but it's possible in theory. Signed-off-by: Jisheng Zhang Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/pgalloc.h | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index d169a4f41a2e..a12fb83fa1f5 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -95,7 +95,13 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) __pud_free(mm, pud); } -#define __pud_free_tlb(tlb, pud, addr) pud_free((tlb)->mm, pud) +#define __pud_free_tlb(tlb, pud, addr) \ +do { \ + if (pgtable_l4_enabled) { \ + pagetable_pud_dtor(virt_to_ptdesc(pud)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ + } \ +} while (0) #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) @@ -124,7 +130,11 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb(tlb, p4d, addr) p4d_free((tlb)->mm, p4d) +#define __p4d_free_tlb(tlb, p4d, addr) \ +do { \ + if (pgtable_l5_enabled) \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ +} while (0) #endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings(pgd_t *pgd) @@ -149,7 +159,11 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb(tlb, pmd, addr) pmd_free((tlb)->mm, pmd) +#define __pmd_free_tlb(tlb, pmd, addr) \ +do { \ + pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ + tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ +} while (0) #endif /* __PAGETABLE_PMD_FOLDED */ From patchwork Tue Dec 19 17:50:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13498798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0951C46CA2 for ; Tue, 19 Dec 2023 18:03:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A25566B0078; Tue, 19 Dec 2023 13:03:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 97D058D0007; Tue, 19 Dec 2023 13:03:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 68C4A6B0081; Tue, 19 Dec 2023 13:03:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 48BB88D0001 for ; Tue, 19 Dec 2023 13:03:31 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 15F381A0E4E for ; Tue, 19 Dec 2023 18:03:31 +0000 (UTC) X-FDA: 81584340222.29.DBD1A1E Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf20.hostedemail.com (Postfix) with ESMTP id 1A7981C002B for ; Tue, 19 Dec 2023 18:03:28 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qZeIMWFF; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of jszhang@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jszhang@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703009009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ozqlHfd+iDCV8pBHY/VR7pPpuWXuYSgWYa3OF+jTDco=; b=XeOxFNsWbUg3NsJVSxiP1agQ8xIMDp9b68GJFsvLWpz5lxIEjZeSCZCva+k6GMk/eCcvrp /5sE5f3nn+KRpLZgMqWO9WODbPjN2DiG+7X9V+f/AF8wSM9jee412VEmpuvlInxKc08xRu QDjMuXz3T0cOpeXOVtjMgzZAxGe2PF4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qZeIMWFF; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf20.hostedemail.com: domain of jszhang@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jszhang@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703009009; a=rsa-sha256; cv=none; b=QwHBdjxJeTodTsggqq1ScMUUcWrwE719wofl2o0zLCz1PzrQ8MuDJ7mXzm8TbbL9bO+Cic 5qAe/9LGozK6rOGm8FNllkRwxRUHIRr3YPToN1vZ1z4sRHUPpNMB4ImPpdYhsXWWOrTM9l eIGhlXkFBAxHhyXDcl46ZhT6fgIXU0s= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 12B3B61472; Tue, 19 Dec 2023 18:03:28 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8F40EC433CB; Tue, 19 Dec 2023 18:03:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703009007; bh=DsAEkljb9nELe7pOkNsFuedvY7TlIkqM0TVa9ONTLHQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qZeIMWFFttxnxsz70F8UouiwH0y4TqZqUR0au/SLYyPTk1mvMoENsBc4ab890iYVP BLByYZiQvFaJaD70giLF0L/HZSuSnQoA27jZ9O0sTHDMS+CV7k5vLv971aK+ISHT0h Q723BOdVl4693wzHnanJbWjMnp8eMfvvBuZL+MqIdTyv4cvHil7xR5nCM9zK5aZIBl duobbluzAihDUQKnsNK7gbSfhRYl4rIktzJyXgaa/afOM+eBTBTEvSSO+NmqV+Zt0X kyvovCVxXTiaOcuXTRROcZOvZHg5P/fqLv8479zyhTaEDOiXIlorhkYKPkcN5VeNZg uIXfSnq44E5/Q== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 2/4] riscv: tlb: convert __p*d_free_tlb() to inline functions Date: Wed, 20 Dec 2023 01:50:44 +0800 Message-Id: <20231219175046.2496-3-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231219175046.2496-1-jszhang@kernel.org> References: <20231219175046.2496-1-jszhang@kernel.org> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1A7981C002B X-Stat-Signature: ttftcr7zzdw3cjnr86kic7pmyn695bbp X-Rspam-User: X-HE-Tag: 1703009008-856630 X-HE-Meta: U2FsdGVkX1+gHLeHErnUms+UpvTeaskq9mSOVO5R9wxiOJRCI5f7/YLOwHJGWXqUC7mBXgkD+pQn0jnH19jfTWtqCCIEJVvqlNTCTjUJG2GRChRRwheUWvqKSC/0UYNELMN9QzxdWyE8FheIWjsZjyuu2v4ycmlvH8T92gbHg+8AI0cJ0oWcPhsuUT3eYzhHprmaF4lsfR3CQwLimztpGQtaSSewFwdtr4NErdBLWSTGGoOeEva3ciJdY6mv/fTpTIF10iwWYyr6+bG96mu567cDenI/ttf3u11JHveBsTILkh9xV2fH+TQtjrODLnHQXwPv68frBZ/oq5Aupbs3xUf+fJLhEEPwX2CHXj+jTwwXOhHppppmy6N0rIz3ITu4VsO6+K9voeZ8bMIWP/WrnQcP2zON0QcwNEug+U7urjUKAZHi1+Ueyp4pkBzFWKRGDMc5RFOlZQfTKv7SnxHCrXbv7mNPDDvjvMbGk96Ucv3c3GMMKm4/6CDt0k6UdpQRo6cnEQXTOcvcjAaD8ubSB75DegWNZlf0RlCzOodpAzPuM6yVjoUKWx8jw09dABqbbaI3INWP7x0BgdtQP8lkj8yAlilVjaUtggd4wU3VFnxBqqchAXoi/ZLApBTxizjVb7mLU4XGs3vuSvSx4HCl8c7uwWZDZQgH0JpmtwWaSgj9Cw/FU7UYK/ckNx1sYaB6h4pcNQwb0ujXZt21vbGS0qNp3canv6N+xy6DB8kiKuS8whUli1RlqspvlCM6M0sKYEZDzQESueU1RZ+Px0SsxXh35xWew+9WGSmkuM5g4hQHVMnzegw6pB2YORiRKLWcYrwqxS+g1q+ssMqEEXY0g4Pl92Cz02pkgfPYlgQleg93jbneARSXlmu3fz2rpgE7bU+WzI5GEGcomh+UlLvaRs+Fhh4EQmRplm1sI24LWlF+pYybtXkK6sfKcbGAwiiFs8RxmvhU3/gQXRLNiHL xVMIJ4gs vvT+MpT+ULkVVffXuhTvT6AHiUSsHnb58VHg6lzFWOL5BBOopHM8JnqmnPrOge8b1p5dqYeq8I2qWpopgJwToUyjIg0c2BIc37oI7aqn6uXblkyPlY4ytGfv8nRqKn1qxJgWHOTpyZaXq3CcgXQPZhWju1exg1FSzbcyyYk4COLQOt6NQapY/FjJe/1ndwjniLzM49+7PD61uLb9dJJKNiF0AsHBtm5rWpxx4dohemhiYacyiLKPEv0v+NhcmMo/VBIuF9NVSiSC7+q77Mq6JKfwmHahJy2Ya758Y X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is to prepare for enabling MMU_GATHER_RCU_TABLE_FREE. No functionality changes. Signed-off-by: Jisheng Zhang Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/pgalloc.h | 54 +++++++++++++++++++------------- 1 file changed, 32 insertions(+), 22 deletions(-) diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index a12fb83fa1f5..3c5e3bd15f46 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -95,13 +95,16 @@ static inline void pud_free(struct mm_struct *mm, pud_t *pud) __pud_free(mm, pud); } -#define __pud_free_tlb(tlb, pud, addr) \ -do { \ - if (pgtable_l4_enabled) { \ - pagetable_pud_dtor(virt_to_ptdesc(pud)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pud)); \ - } \ -} while (0) +static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, + unsigned long addr) +{ + if (pgtable_l4_enabled) { + struct ptdesc *ptdesc = virt_to_ptdesc(pud); + + pagetable_pud_dtor(ptdesc); + tlb_remove_page_ptdesc(tlb, ptdesc); + } +} #define p4d_alloc_one p4d_alloc_one static inline p4d_t *p4d_alloc_one(struct mm_struct *mm, unsigned long addr) @@ -130,11 +133,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) __p4d_free(mm, p4d); } -#define __p4d_free_tlb(tlb, p4d, addr) \ -do { \ - if (pgtable_l5_enabled) \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(p4d)); \ -} while (0) +static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, + unsigned long addr) +{ + if (pgtable_l5_enabled) + tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d)); +} #endif /* __PAGETABLE_PMD_FOLDED */ static inline void sync_kernel_mappings(pgd_t *pgd) @@ -159,19 +163,25 @@ static inline pgd_t *pgd_alloc(struct mm_struct *mm) #ifndef __PAGETABLE_PMD_FOLDED -#define __pmd_free_tlb(tlb, pmd, addr) \ -do { \ - pagetable_pmd_dtor(virt_to_ptdesc(pmd)); \ - tlb_remove_page_ptdesc((tlb), virt_to_ptdesc(pmd)); \ -} while (0) +static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, + unsigned long addr) +{ + struct ptdesc *ptdesc = virt_to_ptdesc(pmd); + + pagetable_pmd_dtor(ptdesc); + tlb_remove_page_ptdesc(tlb, ptdesc); +} #endif /* __PAGETABLE_PMD_FOLDED */ -#define __pte_free_tlb(tlb, pte, buf) \ -do { \ - pagetable_pte_dtor(page_ptdesc(pte)); \ - tlb_remove_page_ptdesc((tlb), page_ptdesc(pte));\ -} while (0) +static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, + unsigned long addr) +{ + struct ptdesc *ptdesc = page_ptdesc(pte); + + pagetable_pte_dtor(ptdesc); + tlb_remove_page_ptdesc(tlb, ptdesc); +} #endif /* CONFIG_MMU */ #endif /* _ASM_RISCV_PGALLOC_H */ From patchwork Tue Dec 19 17:50:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13498799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18077C46CA2 for ; Tue, 19 Dec 2023 18:03:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2E94C8D0007; Tue, 19 Dec 2023 13:03:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26F948D0001; Tue, 19 Dec 2023 13:03:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C2A68D0007; Tue, 19 Dec 2023 13:03:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E8E518D0001 for ; Tue, 19 Dec 2023 13:03:33 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B4A571C120A for ; Tue, 19 Dec 2023 18:03:33 +0000 (UTC) X-FDA: 81584340306.06.A3CCD08 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf02.hostedemail.com (Postfix) with ESMTP id C3FEA80033 for ; Tue, 19 Dec 2023 18:03:31 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=fIZMvRHa; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf02.hostedemail.com: domain of jszhang@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jszhang@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703009011; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xxH216JwTZl30EG274Lt8XoefjQpgY5KXQzb1yFkO9s=; b=VvjdZYG3fVwbHgHHvYrXPyCq4XWFVczBhV28FpokxxUus15fJIN336pFDOoB8bT4tIveGY 0LVB5ezL3cCh3+4f2Mez5nfFUvDVr1TbTYXC8AO8jGa5c7lHRjdKkZV0ACqqRzqMcAKFDt xLMQ/ci/r4CNBjDBHYEWTUS4a907cSE= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=fIZMvRHa; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf02.hostedemail.com: domain of jszhang@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jszhang@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703009011; a=rsa-sha256; cv=none; b=j9Q9rRbn6gcrWafzuYYRDeVi1qNVnLhTZ2fcnRchkOV6Gr1Czy0lraStQgjSsqKcciLTxV bfvroNefNoREfQy9Gb2qmyEMyq5bM7c1F9B6IAs5TwthKbi25z1MULQmFcq802asJyRTSD 0m4unm//1w4IaY8/RDdLiCqGjC3Z+Sk= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id DE1B26141D; Tue, 19 Dec 2023 18:03:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 48267C433C7; Tue, 19 Dec 2023 18:03:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703009010; bh=7HEG7NxaC+hEbKvi6V3ezlMLYmQxX8eFSiY1oLoxI38=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fIZMvRHaWT8zGpNUvDXtLuxxyNtS3fWC3iodCoNDOBXnwEoc+rjvl0JDXiUFsG9JF 6g39zjtD2RlRHz83NGBvDZhFP9DCC+xyZFyt+3RESTpahoQOFX42bsSJldjUzDdjj9 yLPv1UTkwiTLGP8tz243HzVOxqcWTkvEMmM9JfpeHT5eaT5GmJwY3atxcN7rVSj+6K MuLrMN1qzfxqovVsps8N8LtxMR7kl1Zkg95orsXK4QgvYkfd+e+rZzqpNYxx+fG2Cl Uf048r/vPqCqySf8ci8W/7sQQ9/qvAcl8d1Y/tYdpY5KC4rBLQqt2HdR1BOhpOH6P5 PRUU08tc5mcJg== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 3/4] riscv: enable MMU_GATHER_RCU_TABLE_FREE for SMP && MMU Date: Wed, 20 Dec 2023 01:50:45 +0800 Message-Id: <20231219175046.2496-4-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231219175046.2496-1-jszhang@kernel.org> References: <20231219175046.2496-1-jszhang@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C3FEA80033 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: r3sqxzbo9wibnq1wao11hey7qx9fw5ko X-HE-Tag: 1703009011-469237 X-HE-Meta: U2FsdGVkX19Nd7PyORx+F+uEIBxUhRRCt5KeVvtxdvir/Oi5OAxpiqlqFJZqZEbtFXzPqXdcN620zqGXKAOHrsNDEyydvYSr/Q4Dp65GH2+ugSZ/s03jHoMRFxKgrJKOdFboFjvr3Zm12JTNw9EHXPD4T9U/W219TDcaakk7pPInSQMCPlqPcNSLjNHHqeDLny0hbyoOLp7MZq3iVh4bJ3Hi94fT1p5gvkSzer5bzP34BlfmggL0hEYUq0El/OiIQXNKwBSzefCotUn/7a6oTq+lHk8UVgViTtcdJm0NGVIwYk/JYKE/+lWiEq348lFzEAHwweayUrTsgYodqmfFWuRAy2ZIl5KfDl1voNT0paKXaR5as4S22ltBjyCYvDjC7pFc0GunS5QZWUYAfnqiMI/4CZGHI4W51wRzv7HlIh+L11dLuAAvxxg3tyP0xpjig+E4E/FBp9vq/Kwc0MWvtwHCJFCq6kktLAti/+pmcVW8/UqgCdsv6V9duXFLpHWp+1nJD8apFcFRvYJ+O2FSVS1/BmhFbtivys6Kb0AaNwA+1LlrMfSM4BgGGWYW3Nk6ITvYV9iwDgm3YfgwpyLS3Q3XmWdCJ6R8hVzZhdv9zq2Z/ghnRfiurKZXH7OnyWUw7/BfXlTqU/OHXo4r4fRYelwaUl5IS5YfZCz/61XGcGr7XoJ5PBRSOTReYjn1Y123jnrAZ52eipob3t6v4WfJ1i25bcYJ2ZOk6qK6XCiQS1mQaOSiGMDDJ1hh2IxffGcSXQdy8O+DqiLvYnNM0WmqnIJOO6MXB/hCAQw0Ys1tTCYsgLk05XXR55czZEWhaTwLgpJqFXeoKlrwoS3DtQBa1oii5xvcmMpiK7bVxxADrD1sNEOGvz2TqZLRJVKRNFXk7+VzSwznC7sRWIjF8MUu8QxCoqOqd5Osrzr/US/LoXwqiZ3FCI0Loc3ILCE3QdKqYKTABWnEwpyz9aDwkq3 fVB+vuGe Fa2MhJNA98NJrhG2aphPBnr8hFSOuOah+xoLbqXZPPnJxQbeLJ172fHideqDuTrra7q68j967C8iW/+qVbISOsHIbHiRoDgjw11cVqHij5cVzSWMAxlEDSQjLaRmXP40QVSvdWbRaVTqn3uLchlOZCzMW1HOQXDxn06iNlMa+A0tfIzI+eM6AcH2a8gKf+dMVOXJgzpSTzJm2uUlMHtT+YegV4mtVnXNWGeCI+H/Y/fPcxFQfQkUQCck2T0TXo4Wr9pwj5wDbKyqp9x7NdVtFbMIXvqR8YQlT4DXL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to implement fast gup we need to ensure that the page table walker is protected from page table pages being freed from under it. riscv situation is more complicated than other architectures: some riscv platforms may use IPI to perform TLB shootdown, for example, those platforms which support AIA, usually the riscv_ipi_for_rfence is true on these platforms; some riscv platforms may rely on the SBI to perform TLB shootdown, usually the riscv_ipi_for_rfence is false on these platforms. To keep software pagetable walkers safe in this case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h for more details. This patch enables MMU_GATHER_RCU_TABLE_FREE, then use *tlb_remove_page_ptdesc() for those platforms which use IPI to perform TLB shootdown; *tlb_remove_ptdesc() for those platforms which use SBI to perform TLB shootdown; Both case mean that disabling interrupts will block the free and protect the fast gup page walker. Signed-off-by: Jisheng Zhang Reviewed-by: Alexandre Ghiti --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgalloc.h | 23 ++++++++++++++++++----- arch/riscv/include/asm/tlb.h | 18 ++++++++++++++++++ 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index 24c1799e2ec4..d3555173d9f4 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -147,6 +147,7 @@ config RISCV select IRQ_FORCED_THREADING select KASAN_VMALLOC if KASAN select LOCK_MM_AND_FIND_VMA + select MMU_GATHER_RCU_TABLE_FREE if SMP && MMU select MODULES_USE_ELF_RELA if MODULES select MODULE_SECTIONS if MODULES select OF diff --git a/arch/riscv/include/asm/pgalloc.h b/arch/riscv/include/asm/pgalloc.h index 3c5e3bd15f46..deaf971253a2 100644 --- a/arch/riscv/include/asm/pgalloc.h +++ b/arch/riscv/include/asm/pgalloc.h @@ -102,7 +102,10 @@ static inline void __pud_free_tlb(struct mmu_gather *tlb, pud_t *pud, struct ptdesc *ptdesc = virt_to_ptdesc(pud); pagetable_pud_dtor(ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, ptdesc); + else + tlb_remove_ptdesc(tlb, ptdesc); } } @@ -136,8 +139,12 @@ static inline void p4d_free(struct mm_struct *mm, p4d_t *p4d) static inline void __p4d_free_tlb(struct mmu_gather *tlb, p4d_t *p4d, unsigned long addr) { - if (pgtable_l5_enabled) - tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d)); + if (pgtable_l5_enabled) { + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, virt_to_ptdesc(p4d)); + else + tlb_remove_ptdesc(tlb, virt_to_ptdesc(p4d)); + } } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -169,7 +176,10 @@ static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmd, struct ptdesc *ptdesc = virt_to_ptdesc(pmd); pagetable_pmd_dtor(ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, ptdesc); + else + tlb_remove_ptdesc(tlb, ptdesc); } #endif /* __PAGETABLE_PMD_FOLDED */ @@ -180,7 +190,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, struct ptdesc *ptdesc = page_ptdesc(pte); pagetable_pte_dtor(ptdesc); - tlb_remove_page_ptdesc(tlb, ptdesc); + if (riscv_use_ipi_for_rfence()) + tlb_remove_page_ptdesc(tlb, ptdesc); + else + tlb_remove_ptdesc(tlb, ptdesc); } #endif /* CONFIG_MMU */ diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1eb5682b2af6..a0b8b853503f 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -10,6 +10,24 @@ struct mmu_gather; static void tlb_flush(struct mmu_gather *tlb); +#ifdef CONFIG_MMU +#include + +/* + * While riscv platforms with riscv_ipi_for_rfence as true require an IPI to + * perform TLB shootdown, some platforms with riscv_ipi_for_rfence as false use + * SBI to perform TLB shootdown. To keep software pagetable walkers safe in this + * case we switch to RCU based table free (MMU_GATHER_RCU_TABLE_FREE). See the + * comment below 'ifdef CONFIG_MMU_GATHER_RCU_TABLE_FREE' in include/asm-generic/tlb.h + * for more details. + */ +static inline void __tlb_remove_table(void *table) +{ + free_page_and_swap_cache(table); +} + +#endif /* CONFIG_MMU */ + #define tlb_flush tlb_flush #include From patchwork Tue Dec 19 17:50:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13498800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B221AC46CA2 for ; Tue, 19 Dec 2023 18:03:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 516C48D0008; Tue, 19 Dec 2023 13:03:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 41C7B8D0001; Tue, 19 Dec 2023 13:03:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2701E8D0008; Tue, 19 Dec 2023 13:03:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 0E06E8D0001 for ; Tue, 19 Dec 2023 13:03:38 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D36BD160A48 for ; Tue, 19 Dec 2023 18:03:37 +0000 (UTC) X-FDA: 81584340474.20.7115247 Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by imf07.hostedemail.com (Postfix) with ESMTP id D425340019 for ; Tue, 19 Dec 2023 18:03:35 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="BvEJaO/t"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf07.hostedemail.com: domain of jszhang@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=jszhang@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703009016; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZOfv5mqPVoNjWH7q2gBw3BsdWC6ZGwg86MfP6BHLzRA=; b=1/oTLbu3ly6l+MVTzIR5zNAAs0iEg6N6B9JoYdBiBmQ5aaCgvAQ5MSWL7u53MoVLUnx9qI 1KGA1eoAc7Aco87c+JDeEinCCudCpcdt9e8qYe/6Yk5/GdX2xXEdINMmAi9uDj6AesLaEW 1LfmenlNTz0MitbA3K37EoRJR85w5Iw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b="BvEJaO/t"; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf07.hostedemail.com: domain of jszhang@kernel.org designates 145.40.68.75 as permitted sender) smtp.mailfrom=jszhang@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703009016; a=rsa-sha256; cv=none; b=BwBLqj//o1RkMElqpasvnkE/i75DGx/bRiLo3telCA4k6J18LjhU2QoImtjVpNlQiI9D+x +bNihkPX+Ckoe7jJDtKQNmtsMEC/29SA1iDopHYd+fMsyRN+jVmoMDQoWrA8fNKznBn7HX htFrkakoB3V3HUi+KTDzaA/1LcsXG7I= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by ams.source.kernel.org (Postfix) with ESMTP id E5A88B81A3A; Tue, 19 Dec 2023 18:03:33 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 01013C433CC; Tue, 19 Dec 2023 18:03:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703009013; bh=jOzh15UGY8/GeKyvzomEjUoOApjigEX2YDPvhINsBhY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BvEJaO/t7fLRI9ME+g6Nf2uKvBFzhDxeGVkHtu0GNM/cuRLsGbH1Dn5794sKOGhX7 t2neLuWSxp8mcJ6f0iqpeVhqi4qYZ8ahPfVqN2Da62LUO4i/t7gL4bmvdBqH64Llyn QkRpi8dmIk/b/s2jltgElLVpM3vEGAMoladx4fbNv3xbFj5qlDKciiWQ+lStLf8ju1 pub/Wz86s+HQWQ3Ve18yw7Zyy0EcvOkhR8zgpWpEnigIMCNNdI7HcIC4rkQ9U2GhTK L7bquPxY+IC1hy+VduUcpRaujnplUMl8dZNy79gV5MHb1aIEYQXoOGfUkVwIpLYaME gcTZZsxBtN4Pg== From: Jisheng Zhang To: Paul Walmsley , Palmer Dabbelt , Albert Ou , Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 4/4] riscv: enable HAVE_FAST_GUP if MMU Date: Wed, 20 Dec 2023 01:50:46 +0800 Message-Id: <20231219175046.2496-5-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231219175046.2496-1-jszhang@kernel.org> References: <20231219175046.2496-1-jszhang@kernel.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: D425340019 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: y8b4axp4qb5ox7enkfj1fxp3xnqp3m9e X-HE-Tag: 1703009015-536020 X-HE-Meta: U2FsdGVkX1+ze6A+i/LG+2RzlBHj9xvBEqNBYJA9MtVJou4y/3TzPKW05s6vKD/i4/giigvVBImQ5UlG6MxxLo9hGIwYLlpJ54l3G8pC8p5li8jfxYeA5nVP+DAtWH2fZG3bwZ+ULcPgMulJme1Mv6X7wdyA/5atahB8sWnENtwl9PX68XZdXAb7GI319SpCnZGxlbd0hCxH30b+M1F9jkxyH3jDc5dKzHwxfhW6JnLTmtFopbi7T4wLt81FblfSTTxFJ/0fGXZv+aazAi2a3cJ30ajzIbdpTKUbo2DFw5Hnq4oTZN5cUI8FXs3c7L0omTY7SaB1SMBqBOc3w107Po35Zqj4YtDURniS6zmg//9/1mPkFICIrB4XpLE7hZwReFrrrTNwXDlxnJrKrky361fXFe0qjXcZ53RSVCRrcKw+I0jI4LL+PJQmww4/gsYnJRFtO50hni7d1SCn0HR7EuJoWM7vaa6qtGoHDHobCZHQY1UR8vr5qcpOObZNH1t6icTyM/HXLxkhaV5HCD0CKxF3M9DFsI7o0UPwMr0TF+rdsnmFVWVPYky+X3NkzD2m5pRJUGDa/f2wVd5adsH5n7ra4z0gRkvdTlEWqVongj4IBC/30ILhlO9ato4dR0ShZxWgcxcYgCPxQjMn8HLWvKNpj6ZndXjDUuNS/Dl0W6+iGnzQY6djt0SdF5nt+P5AKTxA30UO0UIJ75NUtGfWZRUftIn7of+FWlQLlPMBz/qP376x9jedTOiQtCeLlfk213dL82vATgOka4b5yTs2+vHBBn8/7VeVzYQHki9FJfMmy3I5+02X+DJoWvZIIDijXljLACh2DgBRnVwu8WmcIsMuo5vaZcFzCMkKFXznEf7SA96Scyqld2JuFL7DDOAE4IdL0Mhcdx/3+vApkUSSQECxbOvh728cpGfLH+zwD30EA2D9aQQXF0+TMTYRu8pgVYUujuuR+zRKQkT2Gps 8bZscLMF rPGo1dY3AfWlv4eebSD2OKr3rocL1oqrMMjjuNKOzvyBGkbTZ/1yjdRKhwj/8FJ1HhiIO37J/uzYJ7qPU76e4qC6Ewo3P18+w56knBCyBfzB7kTtod5JoUfXeozLeYDYbiMbeocC/5N6rt8u89O2oEgTungVngoWjCtcSHBeua64s1cbfTLFTrRfqvhHrg/Qs7j4cB/tI5l5aDXlTYr2dhUbqb3b4Io1dNa2Wvk9jNM6rgKrLhzFlavhYj8KdoonbT3xNkFbMgcPeFdgyDUs+50Q0Gwo8Ooy9nuta X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Activate the fast gup for riscv mmu platforms. Here are some GUP_FAST_BENCHMARK performance numbers: Before the patch: GUP_FAST_BENCHMARK: Time: get:53203 put:5085 us After the patch: GUP_FAST_BENCHMARK: Time: get:17711 put:5060 us The get time is reduced by 66.7%! IOW, 3x get speed! Signed-off-by: Jisheng Zhang Reviewed-by: Alexandre Ghiti --- arch/riscv/Kconfig | 1 + arch/riscv/include/asm/pgtable.h | 6 ++++++ 2 files changed, 7 insertions(+) diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d3555173d9f4..04df9920282d 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -119,6 +119,7 @@ config RISCV select HAVE_FUNCTION_GRAPH_RETVAL if HAVE_FUNCTION_GRAPH_TRACER select HAVE_FUNCTION_TRACER if !XIP_KERNEL && !PREEMPTION select HAVE_EBPF_JIT if MMU + select HAVE_FAST_GUP if MMU select HAVE_FUNCTION_ARG_ACCESS_API select HAVE_FUNCTION_ERROR_INJECTION select HAVE_GCC_PLUGINS diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index ab00235b018f..c6eb214139e6 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -673,6 +673,12 @@ static inline int pmd_write(pmd_t pmd) return pte_write(pmd_pte(pmd)); } +#define pud_write pud_write +static inline int pud_write(pud_t pud) +{ + return pte_write(pud_pte(pud)); +} + static inline int pmd_dirty(pmd_t pmd) { return pte_dirty(pmd_pte(pmd));