From patchwork Wed Dec 11 16:01:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AD783E77180 for ; Wed, 11 Dec 2024 16:06:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=u4j+iNOuE80b5cF1isLUMFrawlYtjdat+uShb0WBEyY=; b=IYZUTGBOfYjJK1DNP4GshBUWBs kUk3JTgHwmdyV1mEVYM3QoqDfFXbntuAUhOnPbl1HvHw1UarqAu0BZj90VsCrWUYJqRoGDQYOOqpN K4Ns8RGEDH9AKx/tzsWlbfEOovIqlg6QPu8DFV+LhKAK1xag7ik1fm58FjJJaZcOYUhhfgaf3SOjs lC94uWTSFc5LBmFzTDqVSDTzdTIYeIZil6pgieg228uskPYmqyO/sY9wmGTM5qAY5/CaRVpNTZL3b d8eyXIbi+bqI7uFMzc/UaesVnwtY6495xom9gDqxPDJSDlK4SCDy/5gFMqS5euoZFI2HtwUaRJPhz ziF5wNhQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPDq-0000000FOBB-3r82; Wed, 11 Dec 2024 16:05:58 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPAz-0000000FNK8-0W2x for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 16:03:02 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1BEB415A1; Wed, 11 Dec 2024 08:03:28 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id ED4963F5A1; Wed, 11 Dec 2024 08:02:57 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: ryan.roberts@arm.com, catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev Subject: [RESEND RFC PATCH v1 1/5] arm64: Add TLB Conflict Abort Exception handler to KVM Date: Wed, 11 Dec 2024 16:01:37 +0000 Message-ID: <20241211160218.41404-2-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211160218.41404-1-miko.lenczewski@arm.com> References: <20241211160218.41404-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_080301_196332_CAF5D93F X-CRM114-Status: GOOD ( 10.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, KVM does not handle the case of a stage 2 TLB conflict abort exception. The Arm ARM specifies that the worst-case handling of such an exception requires a `tlbi vmalls12e1`. Perform such an invalidation when this exception is encountered. Signed-off-by: Mikołaj Lenczewski --- arch/arm64/include/asm/esr.h | 8 ++++++++ arch/arm64/kvm/mmu.c | 6 ++++++ 2 files changed, 14 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index d1b1a33f9a8b..8a66f81ca291 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -121,6 +121,7 @@ #define ESR_ELx_FSC_SEA_TTW(n) (0x14 + (n)) #define ESR_ELx_FSC_SECC (0x18) #define ESR_ELx_FSC_SECC_TTW(n) (0x1c + (n)) +#define ESR_ELx_FSC_TLBABT (0x30) /* Status codes for individual page table levels */ #define ESR_ELx_FSC_ACCESS_L(n) (ESR_ELx_FSC_ACCESS + (n)) @@ -464,6 +465,13 @@ static inline bool esr_fsc_is_access_flag_fault(unsigned long esr) (esr == ESR_ELx_FSC_ACCESS_L(0)); } +static inline bool esr_fsc_is_tlb_conflict_abort(unsigned long esr) +{ + esr = esr & ESR_ELx_FSC; + + return esr == ESR_ELx_FSC_TLBABT; +} + /* Indicate whether ESR.EC==0x1A is for an ERETAx instruction */ static inline bool esr_iss_is_eretax(unsigned long esr) { diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index c9d46ad57e52..c8c6f5a97a1b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1756,6 +1756,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) ipa = fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); is_iabt = kvm_vcpu_trap_is_iabt(vcpu); + if (esr_fsc_is_tlb_conflict_abort(esr)) { + // does a `tlbi vmalls12e1is` + __kvm_tlb_flush_vmid(&vcpu->kvm->arch.mmu); + return 1; + } + if (esr_fsc_is_translation_fault(esr)) { /* Beyond sanitised PARange (which is the IPA limit) */ if (fault_ipa >= BIT_ULL(get_kvm_ipa_limit())) { From patchwork Wed Dec 11 16:01:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 99B5CE7717D for ; Wed, 11 Dec 2024 16:07:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=nYv/Z59AB6Gv4K2v7SFuvKNnIt2l46rfAdCdwev4tPQ=; b=25S34SY4Y67ncd1OC+9aUfnR04 lE81FcaQUl78igiPGHqVrgiRlxcuYMfq8iJQewnSeSFlB0QmY00Xsb26s2QiE/5GuPU7PdDYQWHs2 NMugT4WJ0yJ9HVpNxLrfuQDqx5zNGtixiSSIrrzP39g1o/KmadRiYIGqljsEpNi3StZD7N+vAFUeY CKupRpEzHqGOtwMiagXcD3nOgFNo2dFZauLgUbsioTHFrnXzfFlK6UggzmuJjZBfWLaWS2V25XMv6 TV8yMua6Snai4odsYg6lB1QWo824CGowXJmpJJJMhdR86wzT3eprxNOEJXorEE2+BxgJjuMduo75k U3iVvNvw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPEs-0000000FOK0-2CXG; Wed, 11 Dec 2024 16:07:02 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPB0-0000000FNKo-381G for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 16:03:03 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7B01E169E; Wed, 11 Dec 2024 08:03:30 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 583E83F5A1; Wed, 11 Dec 2024 08:03:00 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: ryan.roberts@arm.com, catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev Subject: [RESEND RFC PATCH v1 2/5] arm64: Add BBM Level 2 cpu feature Date: Wed, 11 Dec 2024 16:01:38 +0000 Message-ID: <20241211160218.41404-3-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211160218.41404-1-miko.lenczewski@arm.com> References: <20241211160218.41404-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_080302_868103_F0DD718D X-CRM114-Status: GOOD ( 18.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for. In supporting BBM level 2, we open ourselves up to potential TLB Conflict Abort Exceptions during expected execution, instead of only in exceptional circumstances. In the case of an abort, it is implementation defined at what stage the abort is generated, and the minimal set of required invalidations is also implementation defined. The maximal set of invalidations is to do a `tlbi vmalle1` or `tlbi vmalls12e1`, depending on the stage. Such aborts should not occur on Arm hardware, and were not seen in benchmarked systems, so unless performance concerns arise, implementing the abort handlers with the worst-case invalidations seems like an alright hack. Signed-off-by: Mikołaj Lenczewski --- arch/arm64/include/asm/cpufeature.h | 14 ++++++++++++++ arch/arm64/kernel/cpufeature.c | 7 +++++++ arch/arm64/mm/fault.c | 27 ++++++++++++++++++++++++++- arch/arm64/tools/cpucaps | 1 + 4 files changed, 48 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 8b4e5a3cd24c..a9f2ac335392 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -866,6 +866,20 @@ static __always_inline bool system_supports_mpam_hcr(void) return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } +static inline bool system_supports_bbml2(void) +{ + /* currently, BBM is only relied on by code touching the userspace page + * tables, and as such we are guaranteed that caps have been finalised. + * + * if later we want to use BBM for kernel mappings, particularly early + * in the kernel, this may return 0 even if BBML2 is actually supported, + * which means unnecessary break-before-make sequences, but is still + * correct + */ + + return alternative_has_cap_unlikely(ARM64_HAS_BBML2); +} + int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 6ce71f444ed8..7cc94bd5da24 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2917,6 +2917,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, EVT, IMP) }, + { + .desc = "BBM Level 2 Support", + .capability = ARM64_HAS_BBML2, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) + }, { .desc = "52-bit Virtual Addressing for KVM (LPA2)", .capability = ARM64_HAS_LPA2, diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index ef63651099a9..dc119358cbc1 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -844,6 +844,31 @@ static int do_tag_check_fault(unsigned long far, unsigned long esr, return 0; } +static int do_conflict_abort(unsigned long far, unsigned long esr, + struct pt_regs *regs) +{ + if (!system_supports_bbml2()) + return do_bad(far, esr, regs); + + /* if we receive a TLB conflict abort, we know that there are multiple + * TLB entries that translate the same address range. the minimum set + * of invalidations to clear these entries is implementation defined. + * the maximum set is defined as either tlbi(vmalls12e1) or tlbi(alle1). + * + * if el2 is enabled and stage 2 translation enabled, this may be + * raised as a stage 2 abort. if el2 is enabled but stage 2 translation + * disabled, or if el2 is disabled, it will be raised as a stage 1 + * abort. + * + * local_flush_tlb_all() does a tlbi(vmalle1), which is enough to + * handle a stage 1 abort. + */ + + local_flush_tlb_all(); + + return 0; +} + static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "ttbr address size fault" }, { do_bad, SIGKILL, SI_KERNEL, "level 1 address size fault" }, @@ -893,7 +918,7 @@ static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "unknown 45" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 46" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 47" }, - { do_bad, SIGKILL, SI_KERNEL, "TLB conflict abort" }, + { do_conflict_abort, SIGKILL, SI_KERNEL, "TLB conflict abort" }, { do_bad, SIGKILL, SI_KERNEL, "Unsupported atomic hardware update fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 50" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 51" }, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index eb17f59e543c..4ee0fbb7765b 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -26,6 +26,7 @@ HAS_ECV HAS_ECV_CNTPOFF HAS_EPAN HAS_EVT +HAS_BBML2 HAS_FPMR HAS_FGT HAS_FPSIMD From patchwork Wed Dec 11 16:01:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 35950E77180 for ; Wed, 11 Dec 2024 16:08:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9qRe+YWoR/7llDeJ38JyW1xhw7ge9Tde3t65LUVrgr4=; b=rOGBdTEFp7tH1wdz2Xc/a+Tmxa RSSUq8hO79iITcbKldLItXo+YN4F+6qTuKx6oDlfl31zY4bPC2QjDSgYNMklIMrjI06PbmpA6o1lo buIQQyBcPd/7mABvmtmt7DD8I9OG/RTFJSxHtIHkqpg6al9xyCneS1x+AgRWJoVlZkXOKy9Gwd+jg JwAyw2oRL91uD95XDL8YgmBpJamPHonBQAcZ88aQsJe4cU95nb37iwCKq8bvrADMeKNOOHlpEObu2 FHqxjA5+YZYc9N6X1tZiXMjhEtIyF7bhs4OrfcCI2s72+4eDfxSpn1ogI1FbGJz5b+DTSf5hVYKqf OQkHMnUw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPFu-0000000FOYr-0Xv0; Wed, 11 Dec 2024 16:08:06 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPB3-0000000FNMA-2B3m for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 16:03:06 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0695F1063; Wed, 11 Dec 2024 08:03:33 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B7CAC3F5A1; Wed, 11 Dec 2024 08:03:02 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: ryan.roberts@arm.com, catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev Subject: [RESEND RFC PATCH v1 3/5] arm64: Add errata and workarounds for systems with broken BBML2 Date: Wed, 11 Dec 2024 16:01:39 +0000 Message-ID: <20241211160218.41404-4-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211160218.41404-1-miko.lenczewski@arm.com> References: <20241211160218.41404-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_080305_639942_B14E29F4 X-CRM114-Status: GOOD ( 10.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There are systems which claim support for BBML2, but whose implementation of this support is broken. Add a Kconfig erratum for each of these systems, and a cpufeature workaround that forces the supported BBM level on these systems to 0. Signed-off-by: Mikołaj Lenczewski --- Documentation/arch/arm64/silicon-errata.rst | 32 ++++ arch/arm64/Kconfig | 164 ++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 32 +++- 3 files changed, 227 insertions(+), 1 deletion(-) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index b42fea07c5ce..4b4c1dd9b671 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -126,16 +126,26 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A76 | #3324349 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A76 | #3696297 | ARM64_ERRATUM_3696297 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #1491015 | N/A | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #1508412 | ARM64_ERRATUM_1508412 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #3324348 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A77 | #3696294 | ARM64_ERRATUM_3696294 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A78 | #3324344 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78 | #3696287 | ARM64_ERRATUM_3696287 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A78C | #3324346,3324347| ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78C | #3696291 | ARM64_ERRATUM_3696291 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78C | #3696292 | ARM64_ERRATUM_3696292 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #2119858 | ARM64_ERRATUM_2119858 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #2054223 | ARM64_ERRATUM_2054223 | @@ -144,6 +154,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #3324338 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A710 | #3696244 | ARM64_ERRATUM_3696244 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A715 | #2645198 | ARM64_ERRATUM_2645198 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A715 | #3456084 | ARM64_ERRATUM_3194386 | @@ -156,6 +168,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1 | #3324344 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X1 | #3696287 | ARM64_ERRATUM_3696287 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1C | #3324346 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X2 | #2119858 | ARM64_ERRATUM_2119858 | @@ -164,10 +178,18 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X2 | #3324338 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X2 | #3696244 | ARM64_ERRATUM_3696244 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X3 | #3324335 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X3 | #3696239 | ARM64_ERRATUM_3696239 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X4 | #3043263 | ARM64_ERRATUM_3043263 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X4 | #3194386 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X925 | #3056274 | ARM64_ERRATUM_3056274 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X925 | #3324334 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #1188873,1418040| ARM64_ERRATUM_1418040 | @@ -180,6 +202,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #3324349 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-N1 | #3696297 | ARM64_ERRATUM_3696297 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #2139208 | ARM64_ERRATUM_2139208 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #2067961 | ARM64_ERRATUM_2067961 | @@ -188,14 +212,22 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #3324339 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-N2 | #3696250 | ARM64_ERRATUM_3696250 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N3 | #3456111 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V1 | #1619801 | N/A | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V1 | #3324341 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V1 | #3696285 | ARM64_ERRATUM_3696285 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V2 | #3324336 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V2 | #3696242 | ARM64_ERRATUM_3696242 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V3 | #3053180 | ARM64_ERRATUM_3053180 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V3 | #3312417 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | MMU-500 | #841119,826419 | N/A | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 100570a048c5..9ef8418e8410 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1127,6 +1127,170 @@ config ARM64_ERRATUM_3194386 If unsure, say Y. +config ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + bool + +config ARM64_ERRATUM_3696250 + bool "Neoverse-N2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-N2 cores (r0p0, r0p1, r0p2, r0p3) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696244 + bool "Cortex-A710/Cortex-X2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A710 and Cortex-X2 cores (r0p0, r1p0, r2p0, r2p1) + declare break-before-make level 2 support, but changing the block + size without utilising a break-before-make sequence, or + mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696297 + bool "Cortex-A76/Neoverse-N1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + This option adds a workaround for ARM Cortex-A76/Neoverse-N1 erratum + 3696297. + + Affected Cortex-A76 and Neoverse-N1 cores (r0p0, r1p0, r2p0, r3p0, + r3p1, r4p0, r4p1) declare break-before-make level 2 support, but + changing the block size without utilising a break-before-make sequence, + or mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696294 + bool "Cortex-A77: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + This option adds a workaround for ARM Cortex-A77 erratum 3696294. + + Affected Cortex-A77 cores (r0p0, r1p0, r1p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696239 + bool "Cortex-X3: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X3 cores (r0p0, r1p0, r1p1, r1p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696242 + bool "Neoverse-V2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V2 cores (r0p0, r0p1, r0p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696285 + bool "Neoverse-V1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V1 cores (r0p0, r1p0, r1p1, r1p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696287 + bool "Cortex-A78/Cortex-X1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78 and Cortex-X1 cores (r0p0, r1p0, r1p1, r1p2) + declare break-before-make level 2 support, but changing the block + size without utilising a break-before-make sequence, or + mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696291 + bool "Cortex-A78C: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78C cores (r0p0, r0p1, r0p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696292 + bool "Cortex-A78C: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78C cores (r0p1, r0p2) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3056274 + bool "Cortex-X925: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X925 cores (r0p0, r0p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3043263 + bool "Cortex-X4: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X4 cores (r0p0, r0p1, r0p2) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3053180 + bool "Neoverse-V3: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V3 cores (r0p0, r0p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + config CAVIUM_ERRATUM_22375 bool "Cavium erratum 22375, 24313" default y diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7cc94bd5da24..e6c05b330e0f 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2167,6 +2167,36 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } +static bool has_bbml2(const struct arm64_cpu_capabilities *entry, + int scope) +{ + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT)) { + static const struct midr_range broken_bbml2_list[] = { + MIDR_ALL_VERSIONS(MIDR_CORTEX_A76), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A77), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78C), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A710), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X1), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X2), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X3), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X4), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X925), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V3), + {} + }; + + if (is_midr_in_range_list(read_cpuid_id(), broken_bbml2_list)) + return false; + } + + return has_cpuid_feature(entry, scope); +} + #ifdef CONFIG_ARM64_PAN static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused) { @@ -2921,7 +2951,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .desc = "BBM Level 2 Support", .capability = ARM64_HAS_BBML2, .type = ARM64_CPUCAP_SYSTEM_FEATURE, - .matches = has_cpuid_feature, + .matches = has_bbml2, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) }, { From patchwork Wed Dec 11 16:01:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6999EE77180 for ; Wed, 11 Dec 2024 16:09:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=aO5Lef1xz4D8yiqC49HGFl/t5fdWfkTzV7VCX9dzZSQ=; b=AT/yazGR16k5cKNpIoC/DF7FF1 WwpZhnSGXNZ5MtwZTpr43cvyLmyPw2NzZOtLD2akzH31+I+ZzDSO26tSh7UgF33VGdwhZgKBwJFz3 Aj/w0VAE8ii0eimcvJ8W+OljKhDsP6lJ+wxSiIitq+73qs3qh2eidgnbiTWYFYB6Ltieif/YRyVnK 4CWlz00+abf8kNxIFDnZWRrt25o018oyUpXcEbwokJtg9PWonob34uAudZap+jVhFPAYqr0rSLpyj JMboWLnRGlF2BG5TotNidpaOWQcF995J5a6rrAz4DML2aivTqoJGGieEUABVnxG9vOOR2vVgslnG+ vnYiCpHg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPGv-0000000FOog-38V2; Wed, 11 Dec 2024 16:09:09 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPB5-0000000FNMd-3qhO for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 16:03:08 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 64CCA15A1; Wed, 11 Dec 2024 08:03:35 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 42A103F5A1; Wed, 11 Dec 2024 08:03:05 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: ryan.roberts@arm.com, catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev Subject: [RESEND RFC PATCH v1 4/5] arm64/mm: Delay tlbi in contpte_convert() under BBML2 Date: Wed, 11 Dec 2024 16:01:40 +0000 Message-ID: <20241211160218.41404-5-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211160218.41404-1-miko.lenczewski@arm.com> References: <20241211160218.41404-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_080307_989370_8C93171E X-CRM114-Status: GOOD ( 11.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When converting a region via contpte_convert() to use mTHP, we have two different goals. We have to mark each entry as contiguous, and we would like to smear the dirty and young (access) bits across all entries in the contiguous block. Currently, we do this by first accumulating the dirty and young bits in the block, using an atomic __ptep_get_and_clear() and the relevant pte_{dirty,young}() calls, performing a tlbi, and finally smearing the correct bits across the block using __set_ptes(). This approach works fine for BBM level 0, but with support for BBM level 2 we are allowed to reorder the tlbi to after setting the pagetable entries. This reordering means that other threads will not see an invalid pagetable entry, instead operating on stale data, until we have performed our smearing and issued the invalidation. Avoiding this invalid entry reduces faults in other threads, and thus improves performance marginally (more so when there are more threads). Signed-off-by: Mikołaj Lenczewski --- arch/arm64/mm/contpte.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 55107d27d3f8..fc927be800ee 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -68,9 +68,13 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte = pte_mkyoung(pte); } - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); + if (!system_supports_bbml2()) + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); + + if (system_supports_bbml2()) + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, From patchwork Wed Dec 11 16:01:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DAE6DE77180 for ; Wed, 11 Dec 2024 16:10:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=iEBINFMezgkT8kFwn0hqoIa1/VcW9GASEhKC5a7WGaY=; b=lOFt3kG2ImIY5lad1siiPSnT39 BqhpI4j/Lq3E2vmdeCqGcGEfTeajTzbt2iCuf6Z/PD4qFAz3kZVJmW6TolPdjBlJKeuIDQcyXsbqe L5sWjmwog2G8/cveiVgSosDNKGEGMi8t9vNUve2u8SjfWxVRnwKIbkUkwRkiQDlgoeeROe51sr7SI KwhNJoqu+M9Xd2DCrjn3q6CotS8fWafghttskm7mRrhXmsIdaZp5hfWEd4bvPxZhyEbnNjxIMpI5g 5FSQUu2UOKS036bdVv/k9V3kNQy5OmHj2TUwBEpwJxSoitHDOGpoGvR8Kprn+rrX5ZCYwDyjejslw AqtlIlEQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPHx-0000000FOzp-2Jaw; Wed, 11 Dec 2024 16:10:13 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLPB8-0000000FNNL-0P6P for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 16:03:11 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id C32071713; Wed, 11 Dec 2024 08:03:37 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A0CF63F5A1; Wed, 11 Dec 2024 08:03:07 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: ryan.roberts@arm.com, catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@lists.linux.dev Subject: [RESEND RFC PATCH v1 5/5] arm64/mm: Elide tlbi in contpte_convert() under BBML2 Date: Wed, 11 Dec 2024 16:01:41 +0000 Message-ID: <20241211160218.41404-6-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211160218.41404-1-miko.lenczewski@arm.com> References: <20241211160218.41404-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_080310_174217_F6FD3A06 X-CRM114-Status: UNSURE ( 9.37 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If we support BBM level 2, we can potentially avoid an intermediate TLB invalidation, as hardware is capable of managing the TLB itself in this situation. Hardware will either silently clear out the offending entry, or will take a TLB Conflict Abort Exception. Note that such aborts should not occur on Arm hardware and indeed were not seen on any of the benchmarked systems. Eliding the invalidation results in a 12% improvement on a microbenchmark which targeted the worst case of contpte_convert(), which represents an 80% reduction in the overhead of contpte_convert(). Note also that this patch is pending review to ensure that it is architecturally valid, and we are working with Arm architects to validate this patch. Signed-off-by: Mikołaj Lenczewski --- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index fc927be800ee..009690770415 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); - - if (system_supports_bbml2()) - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,