From patchwork Thu Dec 28 08:46:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13505711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B43CDC3DA6E for ; Thu, 28 Dec 2023 08:59:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=4OU5wwetzeUqWJQYgaebIt8iDSwk/SAHJxgoez2dGKg=; b=CynlCZyUTGROgx 467r3pjbbzRwci+DZ9yr8KEkVouEo0Yzl0gWuozNPkoLOHwrXbQc4SyXSwUd5lUBtN/l0NC8KzKoA S7uin3TgPe+VjBkVnjYn29tXhXJct6bzk24LKdVAtA9zdGvKECt8NqgthVKmQNMwCeUxWj46qaFOg MZbU4cwTZOPMtwjmvb+2NimYtxjliRBO3FhSdlEFrFUYVcDuHQxsunGWoz6bzwEvW5fNbBPeT7fuL SRRE/YnfcwogZyrAUOo+hJyGiT9fNdbIBXCWVwdz6BQhUX1P2PuRtluJiwzt5lKlSKQnnEBr2p718 5a/iwk86DyY+YACbEM2Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rImEv-00GRkk-0h; Thu, 28 Dec 2023 08:59:41 +0000 Received: from ams.source.kernel.org ([145.40.68.75]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rImEj-00GRd7-0F; Thu, 28 Dec 2023 08:59:30 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by ams.source.kernel.org (Postfix) with ESMTP id 89025B80D97; Thu, 28 Dec 2023 08:59:27 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id EB820C433CA; Thu, 28 Dec 2023 08:59:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703753966; bh=16+Nu0Wb1JkSQStq2mcU9kdq58a4r2QX4o0XnGMbW1g=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VPt/k23KQR/ek+Wk4Mb0Ffgqjpr0C4tGoF8uMnFE+csMqgfr093NXmCFQN67kLxmg 2wX671IgLVNFprhYGlOqZT9sQd45M//tMA96Y48PuyxuAUw2YYrGJheIf3jUNzegsO oZr1LwJ+Pd98LSUtkZQ1GQJGY/kKAzRUqhRAQ2LGss8txZ4sD1RLgnpeb+dxBj+bEm lTFy+2WWHg42Kl6nbbR4U7U8bNB7+1AF95XqLf72vf4OQxZI1A08+R/V8QaP95mAv4 W39UhXGY1i/064M112ShThexcH0ieACEEyC3qc6Oesf9LT7f5gDAuaHeHCYJG86mq+ AIf2i7+FdSkcQ== From: Jisheng Zhang To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Catalin Marinas , Paul Walmsley , Palmer Dabbelt , Albert Ou , Arnd Bergmann Cc: linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, Nadav Amit , Andrea Arcangeli , Andy Lutomirski , Dave Hansen , Thomas Gleixner , Yu Zhao , x86@kernel.org Subject: [PATCH 1/2] mm/tlb: fix fullmm semantics Date: Thu, 28 Dec 2023 16:46:41 +0800 Message-Id: <20231228084642.1765-2-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231228084642.1765-1-jszhang@kernel.org> References: <20231228084642.1765-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231228_005929_409351_05295C32 X-CRM114-Status: GOOD ( 15.68 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Nadav Amit fullmm in mmu_gather is supposed to indicate that the mm is torn-down (e.g., on process exit) and can therefore allow certain optimizations. However, tlb_finish_mmu() sets fullmm, when in fact it want to say that the TLB should be fully flushed. Change tlb_finish_mmu() to set need_flush_all and check this flag in tlb_flush_mmu_tlbonly() when deciding whether a flush is needed. At the same time, bring the arm64 fullmm on process exit optimization back. Signed-off-by: Nadav Amit Signed-off-by: Jisheng Zhang Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Cc: x86@kernel.org --- arch/arm64/include/asm/tlb.h | 5 ++++- include/asm-generic/tlb.h | 2 +- mm/mmu_gather.c | 2 +- 3 files changed, 6 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 846c563689a8..6164c5f3b78f 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,7 +62,10 @@ static inline void tlb_flush(struct mmu_gather *tlb) * invalidating the walk-cache, since the ASID allocator won't * reallocate our ASID without invalidating the entire TLB. */ - if (tlb->fullmm) { + if (tlb->fullmm) + return; + + if (tlb->need_flush_all) { if (!last_level) flush_tlb_mm(tlb->mm); return; diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 129a3a759976..f2d46357bcbb 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -452,7 +452,7 @@ static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) * these bits. */ if (!(tlb->freed_tables || tlb->cleared_ptes || tlb->cleared_pmds || - tlb->cleared_puds || tlb->cleared_p4ds)) + tlb->cleared_puds || tlb->cleared_p4ds || tlb->need_flush_all)) return; tlb_flush(tlb); diff --git a/mm/mmu_gather.c b/mm/mmu_gather.c index 4f559f4ddd21..79298bac3481 100644 --- a/mm/mmu_gather.c +++ b/mm/mmu_gather.c @@ -384,7 +384,7 @@ void tlb_finish_mmu(struct mmu_gather *tlb) * On x86 non-fullmm doesn't yield significant difference * against fullmm. */ - tlb->fullmm = 1; + tlb->need_flush_all = 1; __tlb_reset_range(tlb); tlb->freed_tables = 1; } From patchwork Thu Dec 28 08:46:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jisheng Zhang X-Patchwork-Id: 13505712 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 116B5C47073 for ; Thu, 28 Dec 2023 08:59:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nkWHvuWK7gmyDVa6w6kfepYbX8XQzh7sHJ/DlQqq++E=; b=Xr2z/+XbcumxoW qFdcLi91Nn/LUjAve47weZFwjnjzKPU/sd7jb4NVNsD16PWY/UylNstzlRbsjVGlO6pQCH5lt5D3K qbTsz8P26JkgohdlHvw08toa4UIkLSlYojPT3lqb8m0zCzhkGojahwet1pmP5lVWDSY6WMQcAZ7LY WU8fVC88f2QpMrRV5pi8Lq/uxQkAhKlFORf9OPg/Iy1UVF93tk9/s7y0A7Mc1ZOWbo6oROqoEJz6k F6743XxzH0tG5Irp2z7dmIbA6m+e2AoY9ebIRf1vRvQvs3SwUUB/g2/4hOXzDAmnatp5ODDjDGVov oVjsXNGoT52+9nxU8NMQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rImF3-00GRoV-2Q; Thu, 28 Dec 2023 08:59:49 +0000 Received: from dfw.source.kernel.org ([139.178.84.217]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rImEo-00GReX-1L; Thu, 28 Dec 2023 08:59:35 +0000 Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 4F39E60FE6; Thu, 28 Dec 2023 08:59:30 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5389DC433CC; Thu, 28 Dec 2023 08:59:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1703753970; bh=4x9t1f04s06EEvjvtQ7yC2/ljPKJqNDBrcj2L/hT3Js=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=TKBsmvO8700bopUsWROGXLmoCWFXsTW+dDVnNGSxOkaL/USi53RKl8TdsixW0+meq ZCWxQdbp33ikg/G9ezCtq7NkctrZnj90/OzQ0n9iUxG+YKDW4DQ0nPcXFY42wiBv+r 997dRhgKccetlwPkyjqAZY+f5mx70kEYFtXj+CoiB8mr5EJbZvQA7ejpm/TS62Kx8Z MVCzbTzHlBLytfAxGrORGDpTnrYATlbt8PPwaCWGIg2WfXt/VlliMYXrTExd8VnHSR LxHkWdaNYJGBiXawHPjLTvQ6a2cSRLkXc9Y/wtpnv3MN3QHCLVBpveSUN4P0ShLMkB BBgaja1UR4WBA== From: Jisheng Zhang To: Will Deacon , "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Catalin Marinas , Paul Walmsley , Palmer Dabbelt , Albert Ou , Arnd Bergmann Cc: linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org Subject: [PATCH 2/2] riscv: tlb: avoid tlb flushing if fullmm == 1 Date: Thu, 28 Dec 2023 16:46:42 +0800 Message-Id: <20231228084642.1765-3-jszhang@kernel.org> X-Mailer: git-send-email 2.40.0 In-Reply-To: <20231228084642.1765-1-jszhang@kernel.org> References: <20231228084642.1765-1-jszhang@kernel.org> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231228_005934_496904_27528E06 X-CRM114-Status: GOOD ( 10.80 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The mmu_gather code sets fullmm=1 when tearing down the entire address space for an mm_struct on exit or execve. So if the underlying platform supports ASID, the tlb flushing can be avoided because the ASID allocator will never re-allocate a dirty ASID. Use the performance of Process creation in unixbench on T-HEAD TH1520 platform is improved by about 4%. Signed-off-by: Jisheng Zhang Reviewed-by: Alexandre Ghiti --- arch/riscv/include/asm/tlb.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/riscv/include/asm/tlb.h b/arch/riscv/include/asm/tlb.h index 1eb5682b2af6..35f3c214332e 100644 --- a/arch/riscv/include/asm/tlb.h +++ b/arch/riscv/include/asm/tlb.h @@ -12,10 +12,19 @@ static void tlb_flush(struct mmu_gather *tlb); #define tlb_flush tlb_flush #include +#include static inline void tlb_flush(struct mmu_gather *tlb) { #ifdef CONFIG_MMU + /* + * If ASID is supported, the ASID allocator will either invalidate the + * ASID or mark it as used. So we can avoid TLB invalidation when + * pulling down a full mm. + */ + if (static_branch_likely(&use_asid_allocator) && tlb->fullmm) + return; + if (tlb->fullmm || tlb->need_flush_all) flush_tlb_mm(tlb->mm); else