From patchwork Wed Dec 11 15:45:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903698 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6E0DCE7717D for ; Wed, 11 Dec 2024 15:51:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=u4j+iNOuE80b5cF1isLUMFrawlYtjdat+uShb0WBEyY=; b=eI+qpdjQoZfnzhI6TmCBEcvnj2 CJGU3j30SEFJf86X6HWrd95G76efVGrFIB4F5rMyPvtMm3bzETzFcAZy2LvKsMWd+SZNmP0JdqvL9 GCfxRZEO/zOfjAZmRbkXJetD/Cq3tDcN9mBzoE80Y8S+XzhWnU9+nxvpFjpmAC5dmrlKOTvJG+QV8 mEUSWS7J6PhNsUzx4OUgMp9ehbjz7HXgNOyn5gHjIAsfW3hBvtxXBcK24x9fHuqsI9gtEEIjzR7Ej N8NxolICLWU2EoJo6z7qZZNtqedpQlL5YYM7UZY9O+U1tVX0KZncuzrm7vo7GF3E2e9Zck6XMP05Y tWXFr5/w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLOzs-0000000FKw8-0Piv; Wed, 11 Dec 2024 15:51:32 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLOvJ-0000000FJlC-1gMy for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 15:46:50 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1C87EFEC; Wed, 11 Dec 2024 07:47:17 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 78DAA3F5A1; Wed, 11 Dec 2024 07:46:47 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 1/5] arm64: Add TLB Conflict Abort Exception handler to KVM Date: Wed, 11 Dec 2024 15:45:02 +0000 Message-ID: <20241211154611.40395-2-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_074649_480580_9B8EEAD5 X-CRM114-Status: GOOD ( 10.52 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, KVM does not handle the case of a stage 2 TLB conflict abort exception. The Arm ARM specifies that the worst-case handling of such an exception requires a `tlbi vmalls12e1`. Perform such an invalidation when this exception is encountered. Signed-off-by: Mikołaj Lenczewski --- arch/arm64/include/asm/esr.h | 8 ++++++++ arch/arm64/kvm/mmu.c | 6 ++++++ 2 files changed, 14 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index d1b1a33f9a8b..8a66f81ca291 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -121,6 +121,7 @@ #define ESR_ELx_FSC_SEA_TTW(n) (0x14 + (n)) #define ESR_ELx_FSC_SECC (0x18) #define ESR_ELx_FSC_SECC_TTW(n) (0x1c + (n)) +#define ESR_ELx_FSC_TLBABT (0x30) /* Status codes for individual page table levels */ #define ESR_ELx_FSC_ACCESS_L(n) (ESR_ELx_FSC_ACCESS + (n)) @@ -464,6 +465,13 @@ static inline bool esr_fsc_is_access_flag_fault(unsigned long esr) (esr == ESR_ELx_FSC_ACCESS_L(0)); } +static inline bool esr_fsc_is_tlb_conflict_abort(unsigned long esr) +{ + esr = esr & ESR_ELx_FSC; + + return esr == ESR_ELx_FSC_TLBABT; +} + /* Indicate whether ESR.EC==0x1A is for an ERETAx instruction */ static inline bool esr_iss_is_eretax(unsigned long esr) { diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index c9d46ad57e52..c8c6f5a97a1b 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1756,6 +1756,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu) ipa = fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); is_iabt = kvm_vcpu_trap_is_iabt(vcpu); + if (esr_fsc_is_tlb_conflict_abort(esr)) { + // does a `tlbi vmalls12e1is` + __kvm_tlb_flush_vmid(&vcpu->kvm->arch.mmu); + return 1; + } + if (esr_fsc_is_translation_fault(esr)) { /* Beyond sanitised PARange (which is the IPA limit) */ if (fault_ipa >= BIT_ULL(get_kvm_ipa_limit())) { From patchwork Wed Dec 11 15:45:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 269F5E7717D for ; Wed, 11 Dec 2024 15:52:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=nYv/Z59AB6Gv4K2v7SFuvKNnIt2l46rfAdCdwev4tPQ=; b=WaF6uLKgjSRBn71KWWcD7nVVu2 I17AiFJixjbfJkSibpRxcvJS2k7jVlznW3+a3RNDjsV8omdMypS+wEXpWfFsM3PD7I7GZKxAVyRDm Pt1w2Lx3ZRb9ET98XkKvlbREpZUBPGJ+hl4ItO0ERRhzGs0Yxi5KmVQm6aZDmTnbybW8HrW3sRmjE xW2iqylYJu9ahnjksFEcA2jJ1sHk4WL4HeDFc7boRcMk3ozhHf3x3hJVaDfVJTvnT1jtsVXlx3gN9 bqkpf64mg2J3VbIqVLWrperub630W4pFGJDa4CLhS7JCoXU30eJ914PMst/3byPILDftxGue7XtHl /dRnG5qg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLP0t-0000000FL7k-3BjA; Wed, 11 Dec 2024 15:52:35 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLOvL-0000000FJli-3H2h for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 15:46:52 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E49AC1007; Wed, 11 Dec 2024 07:47:18 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 4C4993F5A1; Wed, 11 Dec 2024 07:46:49 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 2/5] arm64: Add BBM Level 2 cpu feature Date: Wed, 11 Dec 2024 15:45:03 +0000 Message-ID: <20241211154611.40395-3-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_074651_907747_52A74067 X-CRM114-Status: GOOD ( 19.21 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org The Break-Before-Make cpu feature supports multiple levels (levels 0-2), and this commit adds a dedicated BBML2 cpufeature to test against support for. In supporting BBM level 2, we open ourselves up to potential TLB Conflict Abort Exceptions during expected execution, instead of only in exceptional circumstances. In the case of an abort, it is implementation defined at what stage the abort is generated, and the minimal set of required invalidations is also implementation defined. The maximal set of invalidations is to do a `tlbi vmalle1` or `tlbi vmalls12e1`, depending on the stage. Such aborts should not occur on Arm hardware, and were not seen in benchmarked systems, so unless performance concerns arise, implementing the abort handlers with the worst-case invalidations seems like an alright hack. Signed-off-by: Mikołaj Lenczewski --- arch/arm64/include/asm/cpufeature.h | 14 ++++++++++++++ arch/arm64/kernel/cpufeature.c | 7 +++++++ arch/arm64/mm/fault.c | 27 ++++++++++++++++++++++++++- arch/arm64/tools/cpucaps | 1 + 4 files changed, 48 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 8b4e5a3cd24c..a9f2ac335392 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -866,6 +866,20 @@ static __always_inline bool system_supports_mpam_hcr(void) return alternative_has_cap_unlikely(ARM64_MPAM_HCR); } +static inline bool system_supports_bbml2(void) +{ + /* currently, BBM is only relied on by code touching the userspace page + * tables, and as such we are guaranteed that caps have been finalised. + * + * if later we want to use BBM for kernel mappings, particularly early + * in the kernel, this may return 0 even if BBML2 is actually supported, + * which means unnecessary break-before-make sequences, but is still + * correct + */ + + return alternative_has_cap_unlikely(ARM64_HAS_BBML2); +} + int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); bool try_emulate_mrs(struct pt_regs *regs, u32 isn); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 6ce71f444ed8..7cc94bd5da24 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2917,6 +2917,13 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, EVT, IMP) }, + { + .desc = "BBM Level 2 Support", + .capability = ARM64_HAS_BBML2, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_cpuid_feature, + ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) + }, { .desc = "52-bit Virtual Addressing for KVM (LPA2)", .capability = ARM64_HAS_LPA2, diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index ef63651099a9..dc119358cbc1 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -844,6 +844,31 @@ static int do_tag_check_fault(unsigned long far, unsigned long esr, return 0; } +static int do_conflict_abort(unsigned long far, unsigned long esr, + struct pt_regs *regs) +{ + if (!system_supports_bbml2()) + return do_bad(far, esr, regs); + + /* if we receive a TLB conflict abort, we know that there are multiple + * TLB entries that translate the same address range. the minimum set + * of invalidations to clear these entries is implementation defined. + * the maximum set is defined as either tlbi(vmalls12e1) or tlbi(alle1). + * + * if el2 is enabled and stage 2 translation enabled, this may be + * raised as a stage 2 abort. if el2 is enabled but stage 2 translation + * disabled, or if el2 is disabled, it will be raised as a stage 1 + * abort. + * + * local_flush_tlb_all() does a tlbi(vmalle1), which is enough to + * handle a stage 1 abort. + */ + + local_flush_tlb_all(); + + return 0; +} + static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "ttbr address size fault" }, { do_bad, SIGKILL, SI_KERNEL, "level 1 address size fault" }, @@ -893,7 +918,7 @@ static const struct fault_info fault_info[] = { { do_bad, SIGKILL, SI_KERNEL, "unknown 45" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 46" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 47" }, - { do_bad, SIGKILL, SI_KERNEL, "TLB conflict abort" }, + { do_conflict_abort, SIGKILL, SI_KERNEL, "TLB conflict abort" }, { do_bad, SIGKILL, SI_KERNEL, "Unsupported atomic hardware update fault" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 50" }, { do_bad, SIGKILL, SI_KERNEL, "unknown 51" }, diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index eb17f59e543c..4ee0fbb7765b 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -26,6 +26,7 @@ HAS_ECV HAS_ECV_CNTPOFF HAS_EPAN HAS_EVT +HAS_BBML2 HAS_FPMR HAS_FGT HAS_FPSIMD From patchwork Wed Dec 11 15:45:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903700 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A9526E7717D for ; Wed, 11 Dec 2024 15:53:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=9qRe+YWoR/7llDeJ38JyW1xhw7ge9Tde3t65LUVrgr4=; b=Pi80tE85rXClkm9Uf384nYHgSe akLeKysSnSxRVWiugqgt6ZiscfiTRMKTJsw3Ku37MbRDI9mYC0uE7nGKCZEaF76Fa+OSiLGWNwNRN 1UD3Xa3QLMPwenqgOqMDxho9vAgheLCwpFm6MbpOhwTwevReZaBB6dFKrm2S3ZIG0H+pfnIgoZwKC FVHMo+rKixL4PO/73tIiTzM4Zcq23zwnUZq8Ncluucvqc3CCr0jvhUObVWI7ZlC1RR7TgY9gSzXFT ChzizAIypIIxUvf5aIWw9cYO/RS3yYXyG9S8xYwiBUcPnqKz5u6DuSnWSawAkoJ6LUFNVp6uPYWMo f7U29Qcw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLP1v-0000000FLHC-1Pwp; Wed, 11 Dec 2024 15:53:39 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLOvN-0000000FJmj-24CL for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 15:46:54 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id D329F1713; Wed, 11 Dec 2024 07:47:20 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 20DFE3F5A1; Wed, 11 Dec 2024 07:46:51 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 3/5] arm64: Add errata and workarounds for systems with broken BBML2 Date: Wed, 11 Dec 2024 15:45:04 +0000 Message-ID: <20241211154611.40395-4-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_074653_626740_D7293D3F X-CRM114-Status: GOOD ( 11.17 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org There are systems which claim support for BBML2, but whose implementation of this support is broken. Add a Kconfig erratum for each of these systems, and a cpufeature workaround that forces the supported BBM level on these systems to 0. Signed-off-by: Mikołaj Lenczewski --- Documentation/arch/arm64/silicon-errata.rst | 32 ++++ arch/arm64/Kconfig | 164 ++++++++++++++++++++ arch/arm64/kernel/cpufeature.c | 32 +++- 3 files changed, 227 insertions(+), 1 deletion(-) diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index b42fea07c5ce..4b4c1dd9b671 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -126,16 +126,26 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A76 | #3324349 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A76 | #3696297 | ARM64_ERRATUM_3696297 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #1491015 | N/A | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #1508412 | ARM64_ERRATUM_1508412 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A77 | #3324348 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A77 | #3696294 | ARM64_ERRATUM_3696294 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A78 | #3324344 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78 | #3696287 | ARM64_ERRATUM_3696287 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A78C | #3324346,3324347| ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78C | #3696291 | ARM64_ERRATUM_3696291 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A78C | #3696292 | ARM64_ERRATUM_3696292 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #2119858 | ARM64_ERRATUM_2119858 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #2054223 | ARM64_ERRATUM_2054223 | @@ -144,6 +154,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A710 | #3324338 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A710 | #3696244 | ARM64_ERRATUM_3696244 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A715 | #2645198 | ARM64_ERRATUM_2645198 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A715 | #3456084 | ARM64_ERRATUM_3194386 | @@ -156,6 +168,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1 | #3324344 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X1 | #3696287 | ARM64_ERRATUM_3696287 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1C | #3324346 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X2 | #2119858 | ARM64_ERRATUM_2119858 | @@ -164,10 +178,18 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X2 | #3324338 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X2 | #3696244 | ARM64_ERRATUM_3696244 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X3 | #3324335 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X3 | #3696239 | ARM64_ERRATUM_3696239 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X4 | #3043263 | ARM64_ERRATUM_3043263 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X4 | #3194386 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-X925 | #3056274 | ARM64_ERRATUM_3056274 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X925 | #3324334 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #1188873,1418040| ARM64_ERRATUM_1418040 | @@ -180,6 +202,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N1 | #3324349 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-N1 | #3696297 | ARM64_ERRATUM_3696297 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #2139208 | ARM64_ERRATUM_2139208 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #2067961 | ARM64_ERRATUM_2067961 | @@ -188,14 +212,22 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N2 | #3324339 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-N2 | #3696250 | ARM64_ERRATUM_3696250 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-N3 | #3456111 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V1 | #1619801 | N/A | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V1 | #3324341 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V1 | #3696285 | ARM64_ERRATUM_3696285 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V2 | #3324336 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V2 | #3696242 | ARM64_ERRATUM_3696242 | ++----------------+-----------------+-----------------+-----------------------------+ +| ARM | Neoverse-V3 | #3053180 | ARM64_ERRATUM_3053180 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Neoverse-V3 | #3312417 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | MMU-500 | #841119,826419 | N/A | diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 100570a048c5..9ef8418e8410 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1127,6 +1127,170 @@ config ARM64_ERRATUM_3194386 If unsure, say Y. +config ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + bool + +config ARM64_ERRATUM_3696250 + bool "Neoverse-N2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-N2 cores (r0p0, r0p1, r0p2, r0p3) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696244 + bool "Cortex-A710/Cortex-X2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A710 and Cortex-X2 cores (r0p0, r1p0, r2p0, r2p1) + declare break-before-make level 2 support, but changing the block + size without utilising a break-before-make sequence, or + mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696297 + bool "Cortex-A76/Neoverse-N1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + This option adds a workaround for ARM Cortex-A76/Neoverse-N1 erratum + 3696297. + + Affected Cortex-A76 and Neoverse-N1 cores (r0p0, r1p0, r2p0, r3p0, + r3p1, r4p0, r4p1) declare break-before-make level 2 support, but + changing the block size without utilising a break-before-make sequence, + or mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696294 + bool "Cortex-A77: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + This option adds a workaround for ARM Cortex-A77 erratum 3696294. + + Affected Cortex-A77 cores (r0p0, r1p0, r1p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696239 + bool "Cortex-X3: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X3 cores (r0p0, r1p0, r1p1, r1p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696242 + bool "Neoverse-V2: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V2 cores (r0p0, r0p1, r0p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696285 + bool "Neoverse-V1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V1 cores (r0p0, r1p0, r1p1, r1p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696287 + bool "Cortex-A78/Cortex-X1: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78 and Cortex-X1 cores (r0p0, r1p0, r1p1, r1p2) + declare break-before-make level 2 support, but changing the block + size without utilising a break-before-make sequence, or + mis-programming the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696291 + bool "Cortex-A78C: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78C cores (r0p0, r0p1, r0p2) declare + break-before-make level 2 support, but changing the block size + without utilising a break-before-make sequence, or mis-programming + the contiguous hint bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3696292 + bool "Cortex-A78C: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-A78C cores (r0p1, r0p2) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3056274 + bool "Cortex-X925: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X925 cores (r0p0, r0p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3043263 + bool "Cortex-X4: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Cortex-X4 cores (r0p0, r0p1, r0p2) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + +config ARM64_ERRATUM_3053180 + bool "Neoverse-V3: workaround for broken BBM level 2 support" + default y + select ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT + help + Affected Neoverse-V3 cores (r0p0, r0p1) declare break-before-make + level 2 support, but changing the block size without utilising a + break-before-make sequence, or mis-programming the contiguous hint + bit can lead to a livelock. + + If unsure, say Y. + config CAVIUM_ERRATUM_22375 bool "Cavium erratum 22375, 24313" default y diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 7cc94bd5da24..e6c05b330e0f 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2167,6 +2167,36 @@ static bool hvhe_possible(const struct arm64_cpu_capabilities *entry, return arm64_test_sw_feature_override(ARM64_SW_FEATURE_OVERRIDE_HVHE); } +static bool has_bbml2(const struct arm64_cpu_capabilities *entry, + int scope) +{ + if (IS_ENABLED(CONFIG_ARM64_WORKAROUND_BROKEN_BBML2_SUPPORT)) { + static const struct midr_range broken_bbml2_list[] = { + MIDR_ALL_VERSIONS(MIDR_CORTEX_A76), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A77), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A78C), + MIDR_ALL_VERSIONS(MIDR_CORTEX_A710), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X1), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X2), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X3), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X4), + MIDR_ALL_VERSIONS(MIDR_CORTEX_X925), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_N2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V1), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V2), + MIDR_ALL_VERSIONS(MIDR_NEOVERSE_V3), + {} + }; + + if (is_midr_in_range_list(read_cpuid_id(), broken_bbml2_list)) + return false; + } + + return has_cpuid_feature(entry, scope); +} + #ifdef CONFIG_ARM64_PAN static void cpu_enable_pan(const struct arm64_cpu_capabilities *__unused) { @@ -2921,7 +2951,7 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .desc = "BBM Level 2 Support", .capability = ARM64_HAS_BBML2, .type = ARM64_CPUCAP_SYSTEM_FEATURE, - .matches = has_cpuid_feature, + .matches = has_bbml2, ARM64_CPUID_FIELDS(ID_AA64MMFR2_EL1, BBM, 2) }, { From patchwork Wed Dec 11 15:45:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903703 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15757E77180 for ; Wed, 11 Dec 2024 15:54:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=aO5Lef1xz4D8yiqC49HGFl/t5fdWfkTzV7VCX9dzZSQ=; b=oxGSfA5Na71RPbj/9yoriJBJ3A d+G0/AJOechW6Vga/eYsvCMZW8eg8r46ssUpqkiNoepycciD8gG1nCpVFgaZSYICy1XGoORni60lW LZNKvuPmoKNUzDDIJA1Lv3zsaKhLjmfhO8bdehXnQK2MCwH/7X8QytW61fp4jJ7Y8ftpOztVTNKPp x9/XrfO3m3lnHZuN8Yyg8gmBnK4OK4jgEi0J7o9+xfh2zpRvP4cSJlStsEjF7sCWUJp8K4ciyDM+y olpPtYa0fKVJWo0Y4N0/ZHubsswmpjZSlD3XBHBy+Rtlw8a4B1FNKUn1ZNtoXB6LnoYR/HGZ1KZYs vxqBp1sQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLP2x-0000000FLRg-0HgD; Wed, 11 Dec 2024 15:54:43 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLOvP-0000000FJo4-1Eoe for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 15:46:56 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A6D481758; Wed, 11 Dec 2024 07:47:22 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0E76B3F5A1; Wed, 11 Dec 2024 07:46:52 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 4/5] arm64/mm: Delay tlbi in contpte_convert() under BBML2 Date: Wed, 11 Dec 2024 15:45:05 +0000 Message-ID: <20241211154611.40395-5-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_074655_371542_0AC2C01A X-CRM114-Status: GOOD ( 11.57 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When converting a region via contpte_convert() to use mTHP, we have two different goals. We have to mark each entry as contiguous, and we would like to smear the dirty and young (access) bits across all entries in the contiguous block. Currently, we do this by first accumulating the dirty and young bits in the block, using an atomic __ptep_get_and_clear() and the relevant pte_{dirty,young}() calls, performing a tlbi, and finally smearing the correct bits across the block using __set_ptes(). This approach works fine for BBM level 0, but with support for BBM level 2 we are allowed to reorder the tlbi to after setting the pagetable entries. This reordering means that other threads will not see an invalid pagetable entry, instead operating on stale data, until we have performed our smearing and issued the invalidation. Avoiding this invalid entry reduces faults in other threads, and thus improves performance marginally (more so when there are more threads). Signed-off-by: Mikołaj Lenczewski --- arch/arm64/mm/contpte.c | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 55107d27d3f8..fc927be800ee 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -68,9 +68,13 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte = pte_mkyoung(pte); } - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); + if (!system_supports_bbml2()) + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); + + if (system_supports_bbml2()) + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, From patchwork Wed Dec 11 15:45:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Miko=C5=82aj_Lenczewski?= X-Patchwork-Id: 13903704 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56109E7717D for ; Wed, 11 Dec 2024 15:55:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc: To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=iEBINFMezgkT8kFwn0hqoIa1/VcW9GASEhKC5a7WGaY=; b=4zP+GBuMW3KMuNDQPr7KFOKPJk y0LcmhHMYsgx6+pKFyth88RMBmLglQNvFqCSmW+QSyBe2xqx+lMBVOoJnneFEfsCMvpXvRH5TdE5a Kc51+Sq0XNkG+YojAKa1aahFWT9xarUPU0Rfo2atoZ4odC8EjV8esRvBhc/5mdqCMfy9WM4PYT7Xn 7f8ncnN6aUzDmHRlpll4/AUHFafIXyJdemtVzdLxgmSKedAVpreaq1kMlIRexou5JE4/csCw1pMzX dNTj0aXSc5P3RjmEwxQlU39hHwHKBGq7BosHARyCZHijjOtFY2Wpj9bZyNTqkwNoe9QjyP7Cu+kFj P/27xaxw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLP3y-0000000FLkr-2pRu; Wed, 11 Dec 2024 15:55:46 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tLOvR-0000000FJoj-0dFa for linux-arm-kernel@lists.infradead.org; Wed, 11 Dec 2024 15:46:58 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 79F5DFEC; Wed, 11 Dec 2024 07:47:24 -0800 (PST) Received: from mazurka.cambridge.arm.com (mazurka.cambridge.arm.com [10.1.196.66]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D5FD93F5A1; Wed, 11 Dec 2024 07:46:54 -0800 (PST) From: =?utf-8?q?Miko=C5=82aj_Lenczewski?= To: catalin.marinas@arm.com, will@kernel.org, corbet@lwn.net, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com Cc: =?utf-8?q?Miko=C5=82aj_Lenczewski?= , linux-arm-kernel@lists.infradead.org, liunx-doc@vger.kernel.org, linux-kernel@vger.kernel.org, kvmarm@vger.kernel.org Subject: [RFC PATCH v1 5/5] arm64/mm: Elide tlbi in contpte_convert() under BBML2 Date: Wed, 11 Dec 2024 15:45:06 +0000 Message-ID: <20241211154611.40395-6-miko.lenczewski@arm.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241211154611.40395-1-miko.lenczewski@arm.com> References: <20241211154611.40395-1-miko.lenczewski@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20241211_074657_226612_5AA0E4D4 X-CRM114-Status: UNSURE ( 9.59 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If we support BBM level 2, we can potentially avoid an intermediate TLB invalidation, as hardware is capable of managing the TLB itself in this situation. Hardware will either silently clear out the offending entry, or will take a TLB Conflict Abort Exception. Note that such aborts should not occur on Arm hardware and indeed were not seen on any of the benchmarked systems. Eliding the invalidation results in a 12% improvement on a microbenchmark which targeted the worst case of contpte_convert(), which represents an 80% reduction in the overhead of contpte_convert(). Note also that this patch is pending review to ensure that it is architecturally valid, and we are working with Arm architects to validate this patch. Signed-off-by: Mikołaj Lenczewski --- arch/arm64/mm/contpte.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index fc927be800ee..009690770415 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -72,9 +72,6 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); - - if (system_supports_bbml2()) - __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); } void __contpte_try_fold(struct mm_struct *mm, unsigned long addr,