From patchwork Thu Apr 17 00:25:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054700 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91BCE1547C9; Thu, 17 Apr 2025 00:25:49 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849549; cv=none; b=WUH1ryq8S8R1MmAcesdWlw9RfvCm+2YmNMGEo77jnZ3rv81RQyH9AIsFzvSYWRU8acBykk0kTrPQCin5FOAflHJO9ejyjhlZOofGreftUWgp7rHWMiZSE/vxM8xw5ZtvYjdgnTxtDmzFtlitRwRTgPZq9eOVUJCoFnP6QqKJu2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849549; c=relaxed/simple; bh=lXf5fjxlXJSXw0bQss56C2Kk2uzTn1+I+C1/cG9yGdE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=i4yGmJAQ9HSOOoX8sjVi/XXTDoDZKqvz63cZwRvsj46dZzFTdnvOMKol5hIXsD+mDvZJNpjLMBnSxMoCVQRQk090E+0qjVqkTwra5KuoSwLZOar+pkoKac877snan48CMtr0CeR25Y5tUXVvN8BOLfXgiKmK5rW86c1GBEVDUbI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SkSmsnyk; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SkSmsnyk" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C12CFC4CEED; Thu, 17 Apr 2025 00:25:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849549; bh=lXf5fjxlXJSXw0bQss56C2Kk2uzTn1+I+C1/cG9yGdE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=SkSmsnykSLQJXpJqkW0L6ScXq/ukJOIEr+pCc7tLQqUDaY/uCRpCwGNZkDS6A0TOn ul4hgRQFnFClmanXZWDPqRgFjl0aNn7Zx2pfEBA35l2wkyE39BEXLQkVqz48Sy17Ty s8pYFU0Xh97JevdTqRp9rqhVy95lme7KzloSXu+jt1IZHlpIId5H/AFkVBUF9ycYze dGi1SKv5Rrqb6TJmbXLJdu4PrS23GHofsrL76LvFa0tr7v3lKfv7omIyafL8JoLGFQ 1r7jx/FOR9pUaWfhXzWSaevz9hJliCzWFpxJd63HfREkUfHxu2/N0mwUEPvOOcMMfd i+zlipQ3fuHiA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:05 +0100 Subject: [PATCH v5 01/28] arm64/fpsimd: Update FA64 and ZT0 enables when loading SME state Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-1-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=5728; i=broonie@kernel.org; h=from:subject:message-id; bh=lXf5fjxlXJSXw0bQss56C2Kk2uzTn1+I+C1/cG9yGdE=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEpv6H+hI6UIeRrfw15CgRRSHGgPwlOHhJw2DMJT 6HiJLSaJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKbwAKCRAk1otyXVSH0LwvB/ 9c9+Rz/XOqwTvVuZyq+p5QpwKkth2gscMmnkmnnJUYiiSz/m70GFIELbmGF8PzGSNy8xPGmcVRbbWu 0RMx9A/se1wFP1DHnV+dk6XSKhjcltc0huIFlf9R7QLxMQFiqkh8VQ59BY7BwGJbF9yGcyGbpd7Wwv kDhreEV5UJQfxO3L0LPbmqyM2HSPpqVnAKpYRCZiXRTTmYiMdqt99jmoEjHAAuIiBT1hw5dGTZdunC iuMZ+Fw2XKAbjJT0L9tIaTIgfS9IBQx59aH4ztHICMDBuHWa5UFM0wHa+iDR95eAtOYUNBMN9zrCUT BnbA+PUjfhUfw1MW78F/bZzzaEtfBJ X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Currently we enable EL0 and EL1 access to FA64 and ZT0 at boot and leave them enabled throughout the runtime of the system. When we add KVM support we will need to make this configuration dynamic, these features may be disabled for some KVM guests. Since the host kernel saves the floating point state for non-protected guests and we wish to avoid KVM having to reload the floating point state needlessly on guest reentry let's move the configuration of these enables to the floating point state reload. We provide a helper which does the configuration as part of a read/modify/write operation along with the configuration of the task VL, then update the floating point state load and SME access trap to use it. We also remove the setting of the enable bits from the CPU feature identification and resume paths. There will be a small overhead from setting the enables one at a time but this should be negligable in the context of the state load or access trap. In order to avoid compiler warnings due to unused variables in !CONFIG_ARM64_SME cases we avoid storing the vector length in temporary variables. Signed-off-by: Mark Brown df --- arch/arm64/include/asm/fpsimd.h | 14 ++++++++++++ arch/arm64/kernel/cpufeature.c | 2 -- arch/arm64/kernel/fpsimd.c | 47 +++++++++++------------------------------ 3 files changed, 26 insertions(+), 37 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index b7adb2c72204..8279565826ad 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -402,6 +402,18 @@ static inline size_t sme_state_size(struct task_struct const *task) return size; } +#define sme_cond_update_smcr(vl, fa64, zt0, reg) \ + do { \ + u64 __old = read_sysreg_s((reg)); \ + u64 __new = vl; \ + if (fa64) \ + __new |= SMCR_ELx_FA64; \ + if (zt0) \ + __new |= SMCR_ELx_EZT0; \ + if (__old != __new) \ + write_sysreg_s(__new, (reg)); \ + } while (0) + #else static inline void sme_user_disable(void) { BUILD_BUG(); } @@ -425,6 +437,8 @@ static inline size_t sme_state_size(struct task_struct const *task) return 0; } +#define sme_cond_update_smcr(val, fa64, zt0, reg) do { } while (0) + #endif /* ! CONFIG_ARM64_SME */ /* For use by EFI runtime services calls only */ diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 9c4d6d552b25..94b34366da37 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -2891,7 +2891,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .type = ARM64_CPUCAP_SYSTEM_FEATURE, .capability = ARM64_SME_FA64, .matches = has_cpuid_feature, - .cpu_enable = cpu_enable_fa64, ARM64_CPUID_FIELDS(ID_AA64SMFR0_EL1, FA64, IMP) }, { @@ -2899,7 +2898,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .type = ARM64_CPUCAP_SYSTEM_FEATURE, .capability = ARM64_SME2, .matches = has_cpuid_feature, - .cpu_enable = cpu_enable_sme2, ARM64_CPUID_FIELDS(ID_AA64PFR1_EL1, SME, SME2) }, #endif /* CONFIG_ARM64_SME */ diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index b0874402f7ec..d3a901afff72 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -392,11 +392,15 @@ static void task_fpsimd_load(void) /* Restore SME, override SVE register configuration if needed */ if (system_supports_sme()) { - unsigned long sme_vl = task_get_sme_vl(current); - - /* Ensure VL is set up for restoring data */ + /* + * Ensure VL is set up for restoring data. KVM might + * disable subfeatures so we reset them each time. + */ if (test_thread_flag(TIF_SME)) - sme_set_vq(sve_vq_from_vl(sme_vl) - 1); + sme_cond_update_smcr(sve_vq_from_vl(task_get_sme_vl(current)) - 1, + system_supports_fa64(), + system_supports_sme2(), + SYS_SMCR_EL1); write_sysreg_s(current->thread.svcr, SYS_SVCR); @@ -1242,26 +1246,6 @@ void cpu_enable_sme(const struct arm64_cpu_capabilities *__always_unused p) isb(); } -void cpu_enable_sme2(const struct arm64_cpu_capabilities *__always_unused p) -{ - /* This must be enabled after SME */ - BUILD_BUG_ON(ARM64_SME2 <= ARM64_SME); - - /* Allow use of ZT0 */ - write_sysreg_s(read_sysreg_s(SYS_SMCR_EL1) | SMCR_ELx_EZT0_MASK, - SYS_SMCR_EL1); -} - -void cpu_enable_fa64(const struct arm64_cpu_capabilities *__always_unused p) -{ - /* This must be enabled after SME */ - BUILD_BUG_ON(ARM64_SME_FA64 <= ARM64_SME); - - /* Allow use of FA64 */ - write_sysreg_s(read_sysreg_s(SYS_SMCR_EL1) | SMCR_ELx_FA64_MASK, - SYS_SMCR_EL1); -} - void __init sme_setup(void) { struct vl_info *info = &vl_info[ARM64_VEC_SME]; @@ -1305,17 +1289,9 @@ void __init sme_setup(void) void sme_suspend_exit(void) { - u64 smcr = 0; - if (!system_supports_sme()) return; - if (system_supports_fa64()) - smcr |= SMCR_ELx_FA64; - if (system_supports_sme2()) - smcr |= SMCR_ELx_EZT0; - - write_sysreg_s(smcr, SYS_SMCR_EL1); write_sysreg_s(0, SYS_SMPRI_EL1); } @@ -1430,9 +1406,10 @@ void do_sme_acc(unsigned long esr, struct pt_regs *regs) WARN_ON(1); if (!test_thread_flag(TIF_FOREIGN_FPSTATE)) { - unsigned long vq_minus_one = - sve_vq_from_vl(task_get_sme_vl(current)) - 1; - sme_set_vq(vq_minus_one); + sme_cond_update_smcr(sve_vq_from_vl(task_get_sme_vl(current)) - 1, + system_supports_fa64(), + system_supports_sme2(), + SYS_SMCR_EL1); fpsimd_bind_task_to_cpu(); } else { From patchwork Thu Apr 17 00:25:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054701 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3EF2C1547C9; Thu, 17 Apr 2025 00:25:52 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849553; cv=none; b=LBKC3zx1Fn4dtVwFBIPTPQL46w9E6I8AI4xuvkvcG5WgEDQIe9RQZpkRYBrs6sAq+gVZhVH0Q0syvmJAx/qgM/ScHOHy5pZqDS3MiJHKEdsQ2AF7+Ypz+JUCIE6/ni8FQex/dZWySvt7CsejkLOhMyFQr2AH7S/dvM0qv/8ME8M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849553; c=relaxed/simple; bh=oRt3lYFMkjnSjiKx9sDZB+3QRpB709eowurjKxguJTA=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=TTrMo9xz/5xTwrsOl/crXGM+1MmJjUW+eSfx+lpuQTvBAER1cz7lB/u5O9wQ89C7sKm02WP3qStqbFjTIPCf1jVJKB4ihBtoMTKiBrbBL1qHcQzmMzrMqhLy1gTvebVSvPhgkekAK1t8Um7j5sbt8UJiRByjfB5R0Cij9XXRjLY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Dzwy2xTc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Dzwy2xTc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6BD04C4CEE2; Thu, 17 Apr 2025 00:25:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849552; bh=oRt3lYFMkjnSjiKx9sDZB+3QRpB709eowurjKxguJTA=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Dzwy2xTcrjU/683hGez8zlDaDWQQ1RO+JDjpuMEFEQbpg6LgYMI6sKAm6Ocx8f1yC i4t6zsBw4Co3avfBQygF3vWV3XABMgUUutilSyYAeNJPztsktNfzC23AZANlvJRqHG lQr7k9v+oXrpMCYQT4NvncCRxlIJXimVyx4gUIWFxppNOUFrYb/kfTWl2zYno3jpv0 TkEkap7elGLzy/kguXUovRM07gZMmOPDayJ1Jy0TB82qFhDoaht51iF+xjR2ifAXK8 XXVj6dKupL17ahRoCccM1I+QXKzRyeueFNmK08JE4Jt58wYJnOFS7+qRdqdY9Pxpvx c7tLbi0lH0tsA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:06 +0100 Subject: [PATCH v5 02/28] arm64/fpsimd: Decide to save ZT0 and streaming mode FFR at bind time Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-2-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3233; i=broonie@kernel.org; h=from:subject:message-id; bh=oRt3lYFMkjnSjiKx9sDZB+3QRpB709eowurjKxguJTA=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEpwozZXsAd5qJB21CA4CiCtC30WEvhJ7v0tYfEB LUVgom+JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKcAAKCRAk1otyXVSH0D68B/ 9M69whFUWMXybJGWIhTbqH8+ujooutUhhbaK2hydw7hO46/aGqeqndM6p8RFHY1GvSsSF5wa0rjgkG CCZ9kYwIAksf54WacwaQ0QwTo7qDzLn3s7laitE06tg22cIY6B3HTWNbmXyr/b9DvKk0USDwzs8qbT oFLJUNDxoZYM/nLAzQvk/wbcDctkuiLuoZ22W6v5oLrneyLfqrJb3t0eyZSuj/aZlbPfr6n/fxWbFC 8uHMEaK+rwzJu139FcqQCNF6JqDjK3B1QSQ/rdX87LZwEPo+S6wEu39p8cPDlvtuQnGaArHPK4zoHB ywHb4TIGRl2XmDiNgrtPN3fSrniwif X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Some parts of the SME state are optional, enabled by additional features on top of the base FEAT_SME and controlled with enable bits in SMCR_ELx. We unconditionally enable these for the host but for KVM we will allow the feature set exposed to guests to be restricted by the VMM. These are the FFR register (FEAT_SME_FA64) and ZT0 (FEAT_SME2). We defer saving of guest floating point state for non-protected guests to the host kernel. We also want to avoid having to reconfigure the guest floating point state if nothing used the floating point state while running the host. If the guest was running with the optional features disabled then traps will be enabled for them so the host kernel will need to skip accessing that state when saving state for the guest. Support this by moving the decision about saving this state to the point where we bind floating point state to the CPU, adding a new variable to the cpu_fp_state which uses the enable bits in SMCR_ELx to flag which features are enabled. Signed-off-by: Mark Brown --- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/kernel/fpsimd.c | 10 ++++++++-- arch/arm64/kvm/fpsimd.c | 1 + 3 files changed, 10 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 8279565826ad..573bd6a219f2 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -86,6 +86,7 @@ struct cpu_fp_state { void *sme_state; u64 *svcr; u64 *fpmr; + u64 sme_features; unsigned int sve_vl; unsigned int sme_vl; enum fp_type *fp_type; diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index d3a901afff72..266ce7a9f195 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -474,12 +474,12 @@ static void fpsimd_save_user_state(void) if (*svcr & SVCR_ZA_MASK) sme_save_state(last->sme_state, - system_supports_sme2()); + last->sme_features & SMCR_ELx_EZT0); /* If we are in streaming mode override regular SVE. */ if (*svcr & SVCR_SM_MASK) { save_sve_regs = true; - save_ffr = system_supports_fa64(); + save_ffr = last->sme_features & SMCR_ELx_FA64; vl = last->sme_vl; } } @@ -1660,6 +1660,12 @@ static void fpsimd_bind_task_to_cpu(void) last->to_save = FP_STATE_CURRENT; current->thread.fpsimd_cpu = smp_processor_id(); + last->sme_features = 0; + if (system_supports_fa64()) + last->sme_features |= SMCR_ELx_FA64; + if (system_supports_sme2()) + last->sme_features |= SMCR_ELx_EZT0; + /* * Toggle SVE and SME trapping for userspace if needed, these * are serialsied by ret_to_user(). diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 7f6e43d25691..7fc14b7349a0 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -106,6 +106,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) fp_state.svcr = &__vcpu_sys_reg(vcpu, SVCR); fp_state.fpmr = &__vcpu_sys_reg(vcpu, FPMR); fp_state.fp_type = &vcpu->arch.fp_type; + fp_state.sme_features = 0; if (vcpu_has_sve(vcpu)) fp_state.to_save = FP_STATE_SVE; From patchwork Thu Apr 17 00:25:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054702 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7F55386352; Thu, 17 Apr 2025 00:25:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849556; cv=none; b=rRv3aXE+izwdZVUEj1c75VUJ15dx9Z3H6Z1KSW05FPycG8+yF2IPn1K0zJxPijG8ODVY1gWqJL/N6SMXxJXXzGoKm7CjZnKi1zCXThWYpHOCwIU+2I8YgGlVEapFFhruAWpel5oNNm5tjKIjyDfQObDj1CGiZTY8/0h74+thVLs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849556; c=relaxed/simple; bh=tPu3T/ssrLPDpFX8WvOwNRnWMZeoT9OAjrJWDn8Vwx0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=ChMDC51LxTroJ6oHZfGm/GF71cSeH9FTjAw0Dau73e/X1kaDvinIUGtz3aWbkPa/2TR6SyfpCZZRPqaj2INamjhpVFGufOFPC3UuRwOZrHjoxRnhps/4a/d8/CRzpai51532TAUGINSV1g4+oYym0C0mSo1Bt8HSVPEfbcXgzUA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oZQLo9yd; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oZQLo9yd" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 26E07C4CEEA; Thu, 17 Apr 2025 00:25:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849556; bh=tPu3T/ssrLPDpFX8WvOwNRnWMZeoT9OAjrJWDn8Vwx0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=oZQLo9ydMVLxzdYzQ9RK60r1FW7dPEbCTAswkPteIJsmUr9GgXFv4hORixeSsURCy 1nA2hduyaKSogcRqAh4udgKbcrbK+aAWA4TSkbXQV3oIOa07NuoMFEXI0iPXYsd+Ra j8OIsoTdAXXGIX9mK/P5fi27y85Z6FqHoRpwKLGIdemJpWddIfyExaGTObjnNWH63T PQQyLLohAloBiBEBcFMEbqkNxYL/oJocYeaih47rsJFS8bxK4UeI2p1+D8ld2TCUaG ptYMwS8zZ4O4kLQ+Uh6ge9YZYqzfFjgjCIv62eGnLxrO5wKxx3fB1u2iwTSJi2GrYb 0Iu6G1L+bXzCQ== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:07 +0100 Subject: [PATCH v5 03/28] arm64/fpsimd: Check enable bit for FA64 when saving EFI state Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-3-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=1468; i=broonie@kernel.org; h=from:subject:message-id; bh=tPu3T/ssrLPDpFX8WvOwNRnWMZeoT9OAjrJWDn8Vwx0=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEpxxYFzzmZ8wZSn8UroEkSMQbZKdAPgqAK1wThy CmeEtziJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKcQAKCRAk1otyXVSH0Lh7B/ 40Li96Qf1d/u6urBOWrA+jCl0rvogHJVY5H5GpHIDuVmr6+HdGdtQrG5obR2zPCrprSLLQe90fg7hR DFvslYEmd6jP6muZG32DInOuNHGtkcKpiwj2wMETd03Z+140KCFgGIPqiqnUYDZS5lqjjvziCOC0Lj WM8zMEKwcLDTbHjSLBZFFUmVX/VurzizBZQ8LFk3Sv5LvD52SFBalF7jP2KcfPJWxMjnvExXOR3vgs GCfJd/llnvJfqLsntJSmNUryVoxx8kwgqGGek+nIhC5kbQ382lH0bwXQq0Tzzw8yuvfJDPCGGqehS+ b1VMi9LCt97sPMvJHga3V/84q/gvR/ X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Currently when deciding if we need to save FFR when in streaming mode prior to EFI calls we check if FA64 is supported by the system. Since KVM guest support will mean that FA64 might be enabled and disabled at runtime switch to checking if traps for FA64 are enabled in SMCR_EL1 instead. Signed-off-by: Mark Brown --- arch/arm64/kernel/fpsimd.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 266ce7a9f195..83a4f8d3e895 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1913,6 +1913,11 @@ static DEFINE_PER_CPU(bool, efi_sm_state); * either doing something wrong or you need to propose some refactoring. */ +static bool fa64_enabled(void) +{ + return read_sysreg_s(SYS_SMCR_EL1) & SMCR_ELx_FA64; +} + /* * __efi_fpsimd_begin(): prepare FPSIMD for making an EFI runtime services call */ @@ -1947,7 +1952,7 @@ void __efi_fpsimd_begin(void) * Unless we have FA64 FFR does not * exist in streaming mode. */ - if (!system_supports_fa64()) + if (!fa64_enabled()) ffr = !(svcr & SVCR_SM_MASK); } @@ -1998,7 +2003,7 @@ void __efi_fpsimd_end(void) * Unless we have FA64 FFR does not * exist in streaming mode. */ - if (!system_supports_fa64()) + if (!fa64_enabled()) ffr = false; } } From patchwork Thu Apr 17 00:25:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054703 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8246586352; Thu, 17 Apr 2025 00:26:00 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849560; cv=none; b=XN3daDy/N+pZSSvZLfzJ5qmTBzfPXExDsVPAkBfRxoJjKqnMud+AtUNJk+lCDTVSjgIiz+unLTwp76IGKGoN+jvfc5JZ0H3cEgYI1qrwfGfLGVThMvxIQRHWEZVtpLlOIMkYxeVL+j2PxWKcC+Q1ZBWGBAV0I7OJxXt1wsNeB64= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849560; c=relaxed/simple; bh=pnmYCJgE/nkEzGjN1xsLpA3poTKMJm9x0l7BvusSFAg=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=pqdYzEaesIQKmUioMdrQJxR9gvnFFKwYxgwqv7wUBHAukS7/WcNqiWh1tqOWVF6rmJ/nRxWiFyaFRu/zCghuaoRlAVhxc67+/P3GLEVvNx2tR7B+BbJ8/NOeW4J9dwj48wmDbJFbXctRCzBTurXFhc+bJhqMZZpqBHF7g9VYJEU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=fDCEzTo6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="fDCEzTo6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C40CAC4CEED; Thu, 17 Apr 2025 00:25:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849560; bh=pnmYCJgE/nkEzGjN1xsLpA3poTKMJm9x0l7BvusSFAg=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=fDCEzTo6k8vauX6QZxob4zR5IdBqfQG9/LBGploL4Rw+4x37UEeadqXly3TOiCVkk +rc/7SpLVfLlXaWO5avohmEd0zWsW7chf+tfFiui1MOvDAKWbtRuE/bG1D45hk2nRJ q2bozicBrs/jrCYU9bW1pv5DRRPspE3j0z1YSZxkNQJc4koA51pjqCmeYNbR9TOVX8 1v2OG4o2khssaslvIoGBly2Ses1hhT5kPVqAG2CnrP7cOdcEwU7tR1dz6WM0JklhYK 9NGPQaA/Oo+z/dvz5qH0/yq/Xb7EevIr8sHdI3r7c2MU6sdabqz59NGx+xyTdoFQcK oU7g9N6g64oaA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:08 +0100 Subject: [PATCH v5 04/28] arm64/fpsimd: Determine maximum virtualisable SME vector length Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-4-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=2177; i=broonie@kernel.org; h=from:subject:message-id; bh=pnmYCJgE/nkEzGjN1xsLpA3poTKMJm9x0l7BvusSFAg=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEpyqJW6FEgBtMbN9D20fjorXEvsb+J367A9+HaO cv+12yyJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKcgAKCRAk1otyXVSH0K9iB/ 9iHNXyxCnCRC0GZoAVZ9Oz/0hMNvuJ1p2hmggRPj+Gwl4uhW6bFIUlM7MFA23oCUn3BvukVr0GHTj1 2uurVbhQ2uo01VD3KNMo95lOpwCJXGeLCB3inYPJHev8JKn4YioCNAXPBS462JwaIsp+yFCSFzPoqb +mCfioy7NElEABy/s7QWoC8oRbSaFFLtXaJ0qQbgMowrEME6NNMXCPOUG2vM85UF21DPAJhMtGE8cn nmYbHCxUNkoMLIsknnNAdVaIrBVQJ8KGi9M+Y/AxqB9h4c08CDV1VPH8xrCX9xUVZaTDz844lzC9nS gUr8MKjjYwMzWEJ1WVz6erXLVwYHBS X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB As with SVE we can only virtualise SME vector lengths that are supported by all CPUs in the system, implement similar checks to those for SVE. Since unlike SVE there are no specific vector lengths that are architecturally required the handling is subtly different, we report a system where this happens with a maximum vector length of -1. Signed-off-by: Mark Brown --- arch/arm64/kernel/fpsimd.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c index 83a4f8d3e895..31b3ca061bb2 100644 --- a/arch/arm64/kernel/fpsimd.c +++ b/arch/arm64/kernel/fpsimd.c @@ -1249,7 +1249,8 @@ void cpu_enable_sme(const struct arm64_cpu_capabilities *__always_unused p) void __init sme_setup(void) { struct vl_info *info = &vl_info[ARM64_VEC_SME]; - int min_bit, max_bit; + DECLARE_BITMAP(tmp_map, SVE_VQ_MAX); + int min_bit, max_bit, b; if (!system_supports_sme()) return; @@ -1279,12 +1280,32 @@ void __init sme_setup(void) */ set_sme_default_vl(find_supported_vector_length(ARM64_VEC_SME, 32)); + bitmap_andnot(tmp_map, info->vq_partial_map, info->vq_map, + SVE_VQ_MAX); + + b = find_last_bit(tmp_map, SVE_VQ_MAX); + if (b >= SVE_VQ_MAX) + /* All VLs virtualisable */ + info->max_virtualisable_vl = SVE_VQ_MAX; + else if (b == SVE_VQ_MAX - 1) + /* No virtualisable VLs */ + info->max_virtualisable_vl = -1; + else + info->max_virtualisable_vl = sve_vl_from_vq(__bit_to_vq(b + 1)); + + if (info->max_virtualisable_vl > info->max_vl) + info->max_virtualisable_vl = info->max_vl; + pr_info("SME: minimum available vector length %u bytes per vector\n", info->min_vl); pr_info("SME: maximum available vector length %u bytes per vector\n", info->max_vl); pr_info("SME: default vector length %u bytes per vector\n", get_sme_default_vl()); + + /* KVM decides whether to support mismatched systems. Just warn here: */ + if (info->max_virtualisable_vl < info->max_vl) + pr_warn("SME: unvirtualisable vector lengths present\n"); } void sme_suspend_exit(void) From patchwork Thu Apr 17 00:25:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054704 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3777C1BD03F; Thu, 17 Apr 2025 00:26:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849564; cv=none; b=jWuHy4wYrdD26M2FWsy1e/bsfTg9E7C1vAEBXJguj63Rm+W0EtrAJwXHwIGtmkWlIzmR0Y5vOc96rpCPzzdOk89FF/iB53PC8ofcjMGtFKEV8nWNEmBzcU4qeykjQgRUyUmeHfGa3sMCHSz+nDCVaiXqOvUoKxVzQYivCMl2qsM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849564; c=relaxed/simple; bh=O7NHBm0GfdFxS8GXkNUEVs2/+kRTAbPluup4xc6JXes=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=jRfV433ZeVvW/YN+gZe079lpA9rOaEthila9KMsqgv2XVg8r9JveVI6k38JABmhFS1h4vFdAB99y5LOCA1RO3v45f7Bzs39FV2mXl66GJ67mVH47DEZ2HtrQTYVBevSivlk+96O9VmAdwKM1/ZrsducKAzwSt+k5PS7V5xQKLf8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Cuc1nklM; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Cuc1nklM" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A4FEC4CEE2; Thu, 17 Apr 2025 00:26:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849563; bh=O7NHBm0GfdFxS8GXkNUEVs2/+kRTAbPluup4xc6JXes=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Cuc1nklMHps4oqg+RBqv/4iGdVdkA1Ywn5Cx7eSf0KFmzVPQtyLurW6K0KBmxFtjt kG9UeQ4CEwEXuidA9FzfDbfKYK37zeEOkTV3vVmVSYoJOVmdV8BJ+9qerj0AWIBDUM F5A06ielq4XzUGFT8esmSSkGGXi0Vz7B7AY6r15qM0UNffDqMqG1Aswse74VKiRtiM zeO8pT8NEEHBCQgQWVHXU5cYzY1WKVOpeqji+dv+8iVrWv9HgSrW3jDXlHJykknv7n V/qVdvJ/ureaXJTprXUe/M4N7nJiNBGlefOuwbTKmPBOHSxFY8hkRJ+ldgeZ2xza+q bJ1sG0FjYqFGw== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:09 +0100 Subject: [PATCH v5 05/28] KVM: arm64: Introduce non-UNDEF FGT control Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-5-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=2793; i=broonie@kernel.org; h=from:subject:message-id; bh=O7NHBm0GfdFxS8GXkNUEVs2/+kRTAbPluup4xc6JXes=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEpyY3NJTDN5eQWnmAxOutCt7z30HJtUVV5WkRnE gJdi0RmJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKcgAKCRAk1otyXVSH0CLsCA CEM0LWsbNf0CgKTyKVLl/fK7EgqbfJqmuBkHMLAl4oi9iK0sJMxeq/Yhp413wfgURsNqFzSgAq/QlX kqyAs5yWwt5ozh32qBqE3Ujyig92x+yxRnlX5xRSaYJc6SeGA4YYsdzDe5qv74jabbqakecBRRsA1/ Ay8AABmmmKTJQoO7YF+oDXejZsLzk30XBjkA5rTwByy4Tn+t8S9uxPQgwq8pfvy67W+X7Leth27ymT 6bI5TZGraAyxh9wuoSOvniDyOcLsLwil91LRi8CZpmzfDKsiZnHcWAme8Yqg1aV/NmT0q0uADczFdG g/kRSuwyd/44yp0x6e5QA+y4QkbNWT X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB We have support for determining a set of fine grained traps to enable for the guest which is tied to the support for injecting UNDEFs for undefined features. This means that we can't use the mechanism for system registers which should be present but need emulation, such as SMPRI_EL1 which should be accessible when SME is present but if SME priority support is absent SMPRI_EL1.Priority should be RAZ. Add an additional set of fine grained traps fgt, mirroring the existing fgu array. We use the same format where we always set the bit for the trap in the array as for FGU. This makes it clear what is being explicitly managed and keeps the code consistent. We do not convert the handling of ARM_WORKAROUND_AMPERE_ACO3_CPU_38 to this mechanism since this only enables a write trap and when implementing the existing UNDEF that we would share the read and write trap enablement (this being the overwhelmingly common case). Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 6 ++++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 7 ++++--- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e98cfe7855a6..3fc8260d55a6 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -295,6 +295,12 @@ struct kvm_arch { */ u64 fgu[__NR_FGT_GROUP_IDS__]; + /* + * Additional FGTs to enable for the guests, eg. for emulated + * registers, + */ + u64 fgt[__NR_FGT_GROUP_IDS__]; + /* * Stage 2 paging state for VMs with nested S2 using a virtual * VMID. diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index b741ea6aefa5..7c9355459648 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -98,9 +98,9 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu) id; \ }) -#define compute_undef_clr_set(vcpu, kvm, reg, clr, set) \ +#define compute_trap_clr_set(vcpu, kvm, trap, reg, clr, set) \ do { \ - u64 hfg = kvm->arch.fgu[reg_to_fgt_group_id(reg)]; \ + u64 hfg = kvm->arch.trap[reg_to_fgt_group_id(reg)]; \ set |= hfg & __ ## reg ## _MASK; \ clr |= hfg & __ ## reg ## _nMASK; \ } while(0) @@ -113,7 +113,8 @@ static inline void __activate_traps_fpsimd32(struct kvm_vcpu *vcpu) if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) \ compute_clr_set(vcpu, reg, c, s); \ \ - compute_undef_clr_set(vcpu, kvm, reg, c, s); \ + compute_trap_clr_set(vcpu, kvm, fgu, reg, c, s); \ + compute_trap_clr_set(vcpu, kvm, fgt, reg, c, s); \ \ s |= set; \ c |= clr; \ From patchwork Thu Apr 17 00:25:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054705 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7031B1BD03F; Thu, 17 Apr 2025 00:26:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849567; cv=none; b=mxrBtZqGdb/A8hNYaMcwXc/c9EhdqBtjzZ6QfP8f6vzKQfaPCaqhiV3akP9r0FVewIw9MGSZ8Bn9ajQzzAjf3s/oRavAhuypfI/dYCsFvMH/ieqzrXBBMXabfTny2lqlpiYdz2U8AQsOm8PgSVI6c+8Z8mzOzmkhIWKjP6jvfsY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849567; c=relaxed/simple; bh=LDt1xNEXDV4dCIkz7nCmkUUILCzxDoXGLDJ0dFTAhvY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=PJOEEz/H/I5apcWTkPCTQpKRMcqEIkWsS2iRDNvuHitqDVbsLTWXWQaQwiOW7nt5nvjtyPLy/d32anfEGZQmALg3DY8YmgRGcgMutgBrspZaBUIwl4MWZmlcrW4c66AD+t67LDgKSOFAfq8/otyc99KRdlZ+Y7LLczDCUUEVuSM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=MdkO4f4J; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="MdkO4f4J" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1515FC4CEEC; Thu, 17 Apr 2025 00:26:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849567; bh=LDt1xNEXDV4dCIkz7nCmkUUILCzxDoXGLDJ0dFTAhvY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=MdkO4f4JPRZABY1VGe30ZeGWmuZSaFtFjbPoiZ2JKwQKtrVsYYf+i/22bM/C+txQ3 VSO+h2w4M6PFbTJ4jJuGtA7NKmb8GQRy7fxoJvXqI8F5gsZeaih/Epir+BBRnVNxBo Hp6hUzdsIkOFIYJJjEeNVg0Ql1sx2Oy53Ymcnb6a6tGeil1b4Pi1WKRP/FZ4z5qFjp 7pPdS6TOxmVCSVTUHyU5tp+Ms+5+l1+metEEhiXl3x0QMVo21vdWGqnJXgEGOtZg0d h1BI2SPu6XavqVmm4LMiapysyCFBTz7B07OxOsVm+98XmHHC0kdhlDlBI4XpxPHEa9 dllqrRyHfmcCg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:10 +0100 Subject: [PATCH v5 06/28] KVM: arm64: Pay attention to FFR parameter in SVE save and load Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-6-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=1028; i=broonie@kernel.org; h=from:subject:message-id; bh=LDt1xNEXDV4dCIkz7nCmkUUILCzxDoXGLDJ0dFTAhvY=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEpzpbQUMEswprMpZyLy/JHBOf6XVtUUczmdbGV3 75K2OBWJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKcwAKCRAk1otyXVSH0HsWB/ 4sQSXEfCj864idBPTgxT0QVL16QLKlb2tYjFR+g2STcCPnwZtZV1LFwSJbC+OJ0RtkpxlsRUsy5QJJ WFPpm0J4Sg7vjD8aAwkFrjupW6IXioTIpwT6iNqrgVv0t5lhHq3QaUTweKPMNmhngcGpVQH7Tv0xKz CM3CmFatHvmNWXBNDUCzW3RfToFWgv6sydXRsYGZ4pXT2SPKc37OCJH0gKrmd/5pnCPhIQfFwzhDka 7wGg10+QjYKhwRxgJFm0McV6OSvvX2EegeLpZ3m8rJqinM817Vh56PEVx8wjGXJot2vVqsPOckFYjU MGfk8dfjxwzkIRO5X1aN0LPGTEUO5v X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB The hypervisor copies of the SVE save and load functions are prototyped with third arguments specifying FFR should be accessed but the assembly functions overwrite whatever is supplied to unconditionally access FFR. Remove this and use the supplied parameter. This has no effect currently since FFR is always present for SVE but will be important for SME. Signed-off-by: Mark Brown --- arch/arm64/kvm/hyp/fpsimd.S | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/fpsimd.S b/arch/arm64/kvm/hyp/fpsimd.S index e950875e31ce..6e16cbfc5df2 100644 --- a/arch/arm64/kvm/hyp/fpsimd.S +++ b/arch/arm64/kvm/hyp/fpsimd.S @@ -21,13 +21,11 @@ SYM_FUNC_START(__fpsimd_restore_state) SYM_FUNC_END(__fpsimd_restore_state) SYM_FUNC_START(__sve_restore_state) - mov x2, #1 sve_load 0, x1, x2, 3 ret SYM_FUNC_END(__sve_restore_state) SYM_FUNC_START(__sve_save_state) - mov x2, #1 sve_save 0, x1, x2, 3 ret SYM_FUNC_END(__sve_save_state) From patchwork Thu Apr 17 00:25:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054706 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 17D691553AA; Thu, 17 Apr 2025 00:26:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849571; cv=none; b=f11cxUc+ODUSzXnCHX9vFuJGgi1uhqT5shvyG7ab40u3oYOhi7fi5dkk0lxH//bqtYZTialUkyVySSRiiVqoOer5Vd1vRXA9PQSrqEL5XIWhKZeYV3ONBwKKvuGSSRunFl/YtGe4idGHHD4FQbQbgCaG3KCYMnsEWNt0lrwgw74= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849571; c=relaxed/simple; bh=eOsj1jCWW4NCg9AO5C3sxJOfLmPv36kBK4yZbeTlaYE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=GW4Q/GOQLUIx40ZzxM5KOntUmUpWP9OEkaHseFmQ51SkehUn+i1LzuhIB/FmBk9cUS/ncBKI1wmVy/BgfBhsMOHCzfLSkTyXlg6waHcu8lHTcX60tGYb+/uvcI4mo/lk64N32o3sXdRAtU2syIHWjWk5BPfZs3VNS155pYXiF2U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LglD13EB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LglD13EB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B37FDC4CEE2; Thu, 17 Apr 2025 00:26:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849571; bh=eOsj1jCWW4NCg9AO5C3sxJOfLmPv36kBK4yZbeTlaYE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=LglD13EBHLAad4Oq2DoFxnIP2NGieGPSOWLYXBsfZWgGorZcIWoKSZzGBBfvPZhP1 3PwImREMwQdqN1Fn/Sw43O6+74JOBxoCejEqznLg0aR8weuZ+8kPewHuPNWiOq1pRg qh3itCIJep/Cb8UcOQ9Y67Vvirod+bV4kkv+hghTRHv2PrvpfJcoBcmAEeas9D+kJp y+AEbSpBb0pjKwFKv7jhmjIoT5i/bkQN6ktgIXQW9HVujxPyl9lNHcTK/5UsiZ7xq2 Edxxhfk0zD3YxIPRznckFWlPSC0FdJ0yPczRJRagHpj/0LgbQTEC5VwVGKj9RaSUZL G/aOQS7YUJrNg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:11 +0100 Subject: [PATCH v5 07/28] KVM: arm64: Pull ctxt_has_ helpers to start of sysreg-sr.h Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-7-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3287; i=broonie@kernel.org; h=from:subject:message-id; bh=eOsj1jCWW4NCg9AO5C3sxJOfLmPv36kBK4yZbeTlaYE=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp0zUroL/kjbbNM9CiQy/D5jh5Xnvdv0HBSqz+p /pPd1wyJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKdAAKCRAk1otyXVSH0BF9B/ kBa4+iFGPD4jkJSIMF80lCYQHuVs59Gf7q8xDIp4KekNG8hy1pHNg8rOoPegq+8C5kCY9+zOoIdbRn CtT8iEfUf6NVDzeoOhUNKD3yhdCXUVFhWcGUZZSFnZUPtgGvx/OmjQt0NBjzCmNFBkl+ETUSgEHjiS O2kedJUqt/iFuTNfU/vLUMem4bdB995Hq5vqmfa0/BFiC2jDKDsr5wP07UVOhNIuSyr2E+snhpdTcV PsDvXqK5pf9FtIEEBUm1a+82zeuG79tttK05cjvZ5qDL+dHDlNLXdWOAsEAc9vgnIcabunWzmhSVb0 OT9cBJbUT9kSLm4OcRYkkUshWPxcSg X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Rather than add earlier prototypes of specific ctxt_has_ helpers let's just pull all their definitions to the top of sysreg-sr.h so they're all available to all the individual save/restore functions. Signed-off-by: Mark Brown --- arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 82 +++++++++++++++--------------- 1 file changed, 40 insertions(+), 42 deletions(-) diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index b9cff893bbe0..b795a7a87a93 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -16,8 +16,6 @@ #include #include -static inline bool ctxt_has_s1poe(struct kvm_cpu_context *ctxt); - static inline struct kvm_vcpu *ctxt_to_vcpu(struct kvm_cpu_context *ctxt) { struct kvm_vcpu *vcpu = ctxt->__hyp_running_vcpu; @@ -28,6 +26,46 @@ static inline struct kvm_vcpu *ctxt_to_vcpu(struct kvm_cpu_context *ctxt) return vcpu; } +static inline bool ctxt_has_mte(struct kvm_cpu_context *ctxt) +{ + struct kvm_vcpu *vcpu = ctxt_to_vcpu(ctxt); + + return kvm_has_mte(kern_hyp_va(vcpu->kvm)); +} + +static inline bool ctxt_has_s1pie(struct kvm_cpu_context *ctxt) +{ + struct kvm_vcpu *vcpu; + + if (!cpus_have_final_cap(ARM64_HAS_S1PIE)) + return false; + + vcpu = ctxt_to_vcpu(ctxt); + return kvm_has_s1pie(kern_hyp_va(vcpu->kvm)); +} + +static inline bool ctxt_has_tcrx(struct kvm_cpu_context *ctxt) +{ + struct kvm_vcpu *vcpu; + + if (!cpus_have_final_cap(ARM64_HAS_TCR2)) + return false; + + vcpu = ctxt_to_vcpu(ctxt); + return kvm_has_tcr2(kern_hyp_va(vcpu->kvm)); +} + +static inline bool ctxt_has_s1poe(struct kvm_cpu_context *ctxt) +{ + struct kvm_vcpu *vcpu; + + if (!system_supports_poe()) + return false; + + vcpu = ctxt_to_vcpu(ctxt); + return kvm_has_s1poe(kern_hyp_va(vcpu->kvm)); +} + static inline bool ctxt_is_guest(struct kvm_cpu_context *ctxt) { return host_data_ptr(host_ctxt) != ctxt; @@ -69,46 +107,6 @@ static inline void __sysreg_save_user_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, TPIDRRO_EL0) = read_sysreg(tpidrro_el0); } -static inline bool ctxt_has_mte(struct kvm_cpu_context *ctxt) -{ - struct kvm_vcpu *vcpu = ctxt_to_vcpu(ctxt); - - return kvm_has_mte(kern_hyp_va(vcpu->kvm)); -} - -static inline bool ctxt_has_s1pie(struct kvm_cpu_context *ctxt) -{ - struct kvm_vcpu *vcpu; - - if (!cpus_have_final_cap(ARM64_HAS_S1PIE)) - return false; - - vcpu = ctxt_to_vcpu(ctxt); - return kvm_has_s1pie(kern_hyp_va(vcpu->kvm)); -} - -static inline bool ctxt_has_tcrx(struct kvm_cpu_context *ctxt) -{ - struct kvm_vcpu *vcpu; - - if (!cpus_have_final_cap(ARM64_HAS_TCR2)) - return false; - - vcpu = ctxt_to_vcpu(ctxt); - return kvm_has_tcr2(kern_hyp_va(vcpu->kvm)); -} - -static inline bool ctxt_has_s1poe(struct kvm_cpu_context *ctxt) -{ - struct kvm_vcpu *vcpu; - - if (!system_supports_poe()) - return false; - - vcpu = ctxt_to_vcpu(ctxt); - return kvm_has_s1poe(kern_hyp_va(vcpu->kvm)); -} - static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) { ctxt_sys_reg(ctxt, SCTLR_EL1) = read_sysreg_el1(SYS_SCTLR); From patchwork Thu Apr 17 00:25:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054707 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2834517A2F9; Thu, 17 Apr 2025 00:26:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849575; cv=none; b=J85g7yfJYhmVm+DrRCAzKDjoH2zcpln/UsC5vEy7ukjOK/yA81fT0qwVi4B4zlQhDPn85r7AQztx0AzbglVOEKrM/zfJvdAe6KOqLscpZdfljWcoey44kSfz8qLkPuY+CtAYnTMYX7DsNul6G/9mzq+7WNKGqGp7kyU3QycAUe4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849575; c=relaxed/simple; bh=a0Ia76ikFrUJD8FigO1QWjiw7c+Y7loFA8aT5ZqeiW0=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=dAXiYa0hP1XN0/1/ITljZVSemPoPb2SJFBBvELCE2/D4i+2tWgCAHrtkxcTc8cvjmBPKJviloicUb9eD0kqJ6t4T4G+Mc1twrH+UxrhtcjZmzVUOJE0xWD4NL52NeQ9Ze78W4RlVgE5BXACxPsjAdvULO2Ox3PpBknKhNI3gzFM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Ta9GsKKB; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Ta9GsKKB" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5F542C4CEE2; Thu, 17 Apr 2025 00:26:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849574; bh=a0Ia76ikFrUJD8FigO1QWjiw7c+Y7loFA8aT5ZqeiW0=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Ta9GsKKBznileJpn86YktlmV/2ZVpEB4KwCBYsiKyDZWXyRJubcHVGl+nF+7asAsN V5UXraXUbS6/mU8PZ3bDelUkLvDGZIumJFSc7b2RSi8z3cdB1OlESEBXx7pSHVh8AW JUwbzhpiCosUVJhG9orsvKtm3Z3l9mhTUMuEWxTIRAXazeL2fxCg6pqNWhWSUKiwrH pKedc+J+eCk8WhAA2fGEMA4RMHcgdGogp5+I/cI8S309+UjgYz1TDA7N8lItdgeocV 4xD7BxFiAw+pIBEgAhQ2uPspi2K43Pij4ey7wAUEMrES6vSkKx9cauKu5VyDPkzvZ0 Iq9tk+3P/8eGA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:12 +0100 Subject: [PATCH v5 08/28] KVM: arm64: Move SVE state access macros after feature test macros Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-8-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=2566; i=broonie@kernel.org; h=from:subject:message-id; bh=a0Ia76ikFrUJD8FigO1QWjiw7c+Y7loFA8aT5ZqeiW0=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp1jatlIUZQb9blHKMZHeDBVqZrdc3aooPmUoKd 9Enk3MqJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKdQAKCRAk1otyXVSH0NmSB/ 0duzDLiieA4h9tp1fqx0R0MozC0Iiep5u4vpp5wHs0vYyxYrBSrrwa3fTLRt67q6lPFy/26m3CJ3v1 4LorI2sfQfvKloiy8rIoKCAaOYbGaHdLABTBHCxW+Tnwvj+ehpmGxLxyVFwdmqfelS/K8mgl/FGkXN 0XJmUD6cP2G0zu5k0G2HIc+fnE4ZORVSOWLKpfNgjyXkb5N2KFXrzWYignYo4w8KLlQaW0fWNvET/i yVgcT663qkewkU7BSkAlNfOXnor8Bf3HRxplprlJFzUhjMCEUHaTs7QbaHvtWmyADFa7GX4MIouIVw qprHWiaQg2wdk4VfXbdXJV4A3p7zEc X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB In preparation for SME support move the macros used to access SVE state after the feature test macros, we will need to test for SME subfeatures to determine the size of the SME state. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 46 +++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 3fc8260d55a6..fb577e10c95d 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -968,29 +968,6 @@ struct kvm_vcpu_arch { #define IN_NESTED_ERET __vcpu_single_flag(sflags, BIT(7)) -/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ -#define vcpu_sve_pffr(vcpu) (kern_hyp_va((vcpu)->arch.sve_state) + \ - sve_ffr_offset((vcpu)->arch.sve_max_vl)) - -#define vcpu_sve_max_vq(vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) - -#define vcpu_sve_zcr_elx(vcpu) \ - (unlikely(is_hyp_ctxt(vcpu)) ? ZCR_EL2 : ZCR_EL1) - -#define vcpu_sve_state_size(vcpu) ({ \ - size_t __size_ret; \ - unsigned int __vcpu_vq; \ - \ - if (WARN_ON(!sve_vl_valid((vcpu)->arch.sve_max_vl))) { \ - __size_ret = 0; \ - } else { \ - __vcpu_vq = vcpu_sve_max_vq(vcpu); \ - __size_ret = SVE_SIG_REGS_SIZE(__vcpu_vq); \ - } \ - \ - __size_ret; \ -}) - #define KVM_GUESTDBG_VALID_MASK (KVM_GUESTDBG_ENABLE | \ KVM_GUESTDBG_USE_SW_BP | \ KVM_GUESTDBG_USE_HW | \ @@ -1026,6 +1003,29 @@ struct kvm_vcpu_arch { #define vcpu_gp_regs(v) (&(v)->arch.ctxt.regs) +/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ +#define vcpu_sve_pffr(vcpu) (kern_hyp_va((vcpu)->arch.sve_state) + \ + sve_ffr_offset((vcpu)->arch.sve_max_vl)) + +#define vcpu_sve_max_vq(vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) + +#define vcpu_sve_zcr_elx(vcpu) \ + (unlikely(is_hyp_ctxt(vcpu)) ? ZCR_EL2 : ZCR_EL1) + +#define vcpu_sve_state_size(vcpu) ({ \ + size_t __size_ret; \ + unsigned int __vcpu_vq; \ + \ + if (WARN_ON(!sve_vl_valid((vcpu)->arch.sve_max_vl))) { \ + __size_ret = 0; \ + } else { \ + __vcpu_vq = vcpu_sve_max_vq(vcpu); \ + __size_ret = SVE_SIG_REGS_SIZE(__vcpu_vq); \ + } \ + \ + __size_ret; \ +}) + /* * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the * memory backed version of a register, and not the one most recently From patchwork Thu Apr 17 00:25:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054708 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 68C7517A2F9; Thu, 17 Apr 2025 00:26:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849578; cv=none; b=kZv6kfZi81pMdrh+HB5oD0UToVaF1Y0TMyWUPSygFeXnRPIKXNhccXlHHvVWYwOaSdqk9gjOBTY1NhTD/3M97lq0v+7JlS4S+ZnCoT9lX21w/fzPkyXgkxks2jYmOv4x3I68+ajPpZWSQK6CgNsfYwlBYEHgQ9HAWdBbTC2xs4c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849578; c=relaxed/simple; bh=24f0RQsbj7jx8A+wiByg8XHt3RtV8IOq4kEhiriZZxM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=LH8Xgw+9Fb6jfJbOjJ9tFJqMkUMRk0XcpG8xtik4umdX7zDCSRi1UIkh0jroc7rnrVqC08/RkRv2u0wd+sGaOpoQrv9E0t9ZgSEq0nub2M68ZVTGVytqYh1m2t1n3hwg+VO1ZYfxQytSH5PXvau4QaV7KuiNgxJ5FiYotYK3X/g= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pNMe0QOu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pNMe0QOu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0C957C4CEEA; Thu, 17 Apr 2025 00:26:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849578; bh=24f0RQsbj7jx8A+wiByg8XHt3RtV8IOq4kEhiriZZxM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=pNMe0QOuaiUpOTng9lZoC/SXTYfTZTpZtk7e/8XB7nDZsYelNEQ50U9mg9BnsZiGx ERM8+ntqDdqKyb5ine7AmirLA7nLcpgjdPdfqn3XaSa8VsRwS+Embcj1c4Fbr7vgTB z22Ld1RGhhDRwWSnJs23LaITpYURJBoSknaY1DEOm9cU0zpYwq+RGdOtIbKq9CWnJu W+NOqpKBhHCX1CIY4ldgM1nr6BVAmpTcjINJPUh5fhptpEim7UxL7/10/XJQtzKyBd f/rJ43ZWczeg795WQAYepovWclFSfXqB4Sh0CDAYxoCZpEaNg8oPQoaKkWWAaSY/cF MtBnm+RfGe+Zg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:13 +0100 Subject: [PATCH v5 09/28] KVM: arm64: Rename SVE finalization constants to be more general Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-9-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=7293; i=broonie@kernel.org; h=from:subject:message-id; bh=24f0RQsbj7jx8A+wiByg8XHt3RtV8IOq4kEhiriZZxM=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp2vSVen3Rym6pzc8jqz8w5Q3ccz3j1O//arLCs dPVykvmJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKdgAKCRAk1otyXVSH0DA9B/ 0Y72oVlFthSd4VU3erQQttFXyx26FoOriJOAAcDbRUYxdp3RPrO1sQjYPEx2hW66jUun3DLtNv3Gfe vnUwywASwTVundyJexC1PzVAybcvff4aL8Hz2oVUXIUPXFzXur24bcWpyfrvPjdBcQpfug7YNV/RdN e4aMQiikYtNUg28WXjxW20n637aMHjuc1AF53K0tepF8YjrQjbDI3wwJaJstOZ4fuJ4728uflQoS3h ttWBRrv+ig83k6kNMbqMX7zjNKu3Lv7qCajpur2+RaDjfWT+pFqUdbUJVVPuLRiDI9zw0u4rt1KPPu pRtxnDJ73TVT1z21slCAXSOzolkkF3 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Due to the overlap between SVE and SME vector length configuration created by streaming mode SVE we will finalize both at once. Rename the existing finalization to use _VEC (vector) for the naming to avoid confusion. Since this includes the userspace API we create an alias KVM_ARM_VCPU_VEC for the existing KVM_ARM_VCPU_SVE capability, existing code which does not enable SME will be unaffected and any SME only code will not need to use SVE constants. No functional change. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 8 +++++--- arch/arm64/include/uapi/asm/kvm.h | 6 ++++++ arch/arm64/kvm/guest.c | 10 +++++----- arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 +- arch/arm64/kvm/reset.c | 20 ++++++++++---------- 5 files changed, 27 insertions(+), 19 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index fb577e10c95d..e828607653c7 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -910,8 +910,8 @@ struct kvm_vcpu_arch { /* KVM_ARM_VCPU_INIT completed */ #define VCPU_INITIALIZED __vcpu_single_flag(cflags, BIT(0)) -/* SVE config completed */ -#define VCPU_SVE_FINALIZED __vcpu_single_flag(cflags, BIT(1)) +/* Vector config completed */ +#define VCPU_VEC_FINALIZED __vcpu_single_flag(cflags, BIT(1)) /* pKVM VCPU setup completed */ #define VCPU_PKVM_FINALIZED __vcpu_single_flag(cflags, BIT(2)) @@ -982,6 +982,8 @@ struct kvm_vcpu_arch { #define vcpu_has_sve(vcpu) kvm_has_sve((vcpu)->kvm) #endif +#define vcpu_has_vec(vcpu) vcpu_has_sve(vcpu) + #ifdef CONFIG_ARM64_PTR_AUTH #define vcpu_has_ptrauth(vcpu) \ ((cpus_have_final_cap(ARM64_HAS_ADDRESS_AUTH) || \ @@ -1462,7 +1464,7 @@ struct kvm *kvm_arch_alloc_vm(void); int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature); bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu); -#define kvm_arm_vcpu_sve_finalized(vcpu) vcpu_get_flag(vcpu, VCPU_SVE_FINALIZED) +#define kvm_arm_vcpu_vec_finalized(vcpu) vcpu_get_flag(vcpu, VCPU_VEC_FINALIZED) #define kvm_has_mte(kvm) \ (system_supports_mte() && \ diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index af9d9acaf997..42cfa86b96ca 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -107,6 +107,12 @@ struct kvm_regs { #define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */ #define KVM_ARM_VCPU_HAS_EL2_E2H0 8 /* Limit NV support to E2H RES0 */ +/* + * An alias for _SVE since we finalize VL configuration for both SVE and SME + * simultaneously. + */ +#define KVM_ARM_VCPU_VEC KVM_ARM_VCPU_SVE + struct kvm_vcpu_init { __u32 target; __u32 features[7]; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 2196979a24a3..73e714133bb6 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -342,7 +342,7 @@ static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (!vcpu_has_sve(vcpu)) return -ENOENT; - if (kvm_arm_vcpu_sve_finalized(vcpu)) + if (kvm_arm_vcpu_vec_finalized(vcpu)) return -EPERM; /* too late! */ if (WARN_ON(vcpu->arch.sve_state)) @@ -497,7 +497,7 @@ static int get_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (ret) return ret; - if (!kvm_arm_vcpu_sve_finalized(vcpu)) + if (!kvm_arm_vcpu_vec_finalized(vcpu)) return -EPERM; if (copy_to_user(uptr, vcpu->arch.sve_state + region.koffset, @@ -523,7 +523,7 @@ static int set_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (ret) return ret; - if (!kvm_arm_vcpu_sve_finalized(vcpu)) + if (!kvm_arm_vcpu_vec_finalized(vcpu)) return -EPERM; if (copy_from_user(vcpu->arch.sve_state + region.koffset, uptr, @@ -657,7 +657,7 @@ static unsigned long num_sve_regs(const struct kvm_vcpu *vcpu) return 0; /* Policed by KVM_GET_REG_LIST: */ - WARN_ON(!kvm_arm_vcpu_sve_finalized(vcpu)); + WARN_ON(!kvm_arm_vcpu_vec_finalized(vcpu)); return slices * (SVE_NUM_PREGS + SVE_NUM_ZREGS + 1 /* FFR */) + 1; /* KVM_REG_ARM64_SVE_VLS */ @@ -675,7 +675,7 @@ static int copy_sve_reg_indices(const struct kvm_vcpu *vcpu, return 0; /* Policed by KVM_GET_REG_LIST: */ - WARN_ON(!kvm_arm_vcpu_sve_finalized(vcpu)); + WARN_ON(!kvm_arm_vcpu_vec_finalized(vcpu)); /* * Enumerate this first, so that userspace can save/restore in diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index 5a335a51deca..eff47b58effb 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -403,7 +403,7 @@ static void pkvm_vcpu_init_sve(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu * struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; if (!vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) - vcpu_clear_flag(vcpu, VCPU_SVE_FINALIZED); + vcpu_clear_flag(vcpu, VCPU_VEC_FINALIZED); } static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index f82fcc614e13..2cf2028ab0dd 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -92,7 +92,7 @@ static void kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu) * Finalize vcpu's maximum SVE vector length, allocating * vcpu->arch.sve_state as necessary. */ -static int kvm_vcpu_finalize_sve(struct kvm_vcpu *vcpu) +static int kvm_vcpu_finalize_vec(struct kvm_vcpu *vcpu) { void *buf; unsigned int vl; @@ -122,21 +122,21 @@ static int kvm_vcpu_finalize_sve(struct kvm_vcpu *vcpu) } vcpu->arch.sve_state = buf; - vcpu_set_flag(vcpu, VCPU_SVE_FINALIZED); + vcpu_set_flag(vcpu, VCPU_VEC_FINALIZED); return 0; } int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature) { switch (feature) { - case KVM_ARM_VCPU_SVE: - if (!vcpu_has_sve(vcpu)) + case KVM_ARM_VCPU_VEC: + if (!vcpu_has_vec(vcpu)) return -EINVAL; - if (kvm_arm_vcpu_sve_finalized(vcpu)) + if (kvm_arm_vcpu_vec_finalized(vcpu)) return -EPERM; - return kvm_vcpu_finalize_sve(vcpu); + return kvm_vcpu_finalize_vec(vcpu); } return -EINVAL; @@ -144,7 +144,7 @@ int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature) bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu) { - if (vcpu_has_sve(vcpu) && !kvm_arm_vcpu_sve_finalized(vcpu)) + if (vcpu_has_vec(vcpu) && !kvm_arm_vcpu_vec_finalized(vcpu)) return false; return true; @@ -161,7 +161,7 @@ void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) kfree(vcpu->arch.ccsidr); } -static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu) +static void kvm_vcpu_reset_vec(struct kvm_vcpu *vcpu) { if (vcpu_has_sve(vcpu)) memset(vcpu->arch.sve_state, 0, vcpu_sve_state_size(vcpu)); @@ -201,11 +201,11 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) if (loaded) kvm_arch_vcpu_put(vcpu); - if (!kvm_arm_vcpu_sve_finalized(vcpu)) { + if (!kvm_arm_vcpu_vec_finalized(vcpu)) { if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) kvm_vcpu_enable_sve(vcpu); } else { - kvm_vcpu_reset_sve(vcpu); + kvm_vcpu_reset_vec(vcpu); } if (vcpu_el1_is_32bit(vcpu)) From patchwork Thu Apr 17 00:25:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054709 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EDE417E00E; Thu, 17 Apr 2025 00:26:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849582; cv=none; b=iMtrdh7L/sck3as62xtZHhRapg69DjKLS2gJmIJ3PEIXtMhtKaRd+ESjZ3NnokfExdrrpzvjZj97VgqM8xmrbuZkI6iZjmFOu4KYkOCJIx5ZCwLvO5HDvO7AjhrI7TSXQEZ24uLgP5Gviv6PsajXBgcsKAelNWFIxaWeYIJMsBE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849582; c=relaxed/simple; bh=MlzU8uizVpCnnQ1OWeHj73Xfek2PC3e+VyU2b4Ykjpw=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=R2gwfbei7zvmnzMAkQa4QAvIq7/CS9P+F8a8f5IJu48VC9VW5haK1fa8FmBGVa5OUr/jQLQuU+n9z+aUYMcihk4saDClZINASu3h05eWBto9MSddeHhgIKzk563YB2ZokTwDAQOkfYCeSQk4TnJv/iH+z4tOkLKBf7NH7g4ajMs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=PtbZrKab; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="PtbZrKab" Received: by smtp.kernel.org (Postfix) with ESMTPSA id AD50AC4CEEE; Thu, 17 Apr 2025 00:26:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849581; bh=MlzU8uizVpCnnQ1OWeHj73Xfek2PC3e+VyU2b4Ykjpw=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=PtbZrKabKg0L+VWyp42TckiQyl/mUXslRCjj/95tHbHq5a9wbQn2nAO/9wDpnBxF1 LWZ+IRcm/LTxTXIHGrECXWx7UQISs1E+7JfaEy1iNsxfsie+MMZdxqz2GlScrIhBXm JHnwtkBrqHavGrhIFaVwvvIQ48UV1Beg9Qs3C8N/yHtiaEgMkWnLjgRwVcSlpm85u8 bU5NBUVcehbC7Tsa+RIO+8cPqeg9R/c8O41rcqO5Xq56DMK4MHka3EttGGiGD6IexS DiyP6BTRnHkcocBVgbJfFclUlEBxVpIOR0qKuMGTNBtSphHiGwO+Etzz6kKTHFMqqa irnR9GykQnQzA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:14 +0100 Subject: [PATCH v5 10/28] KVM: arm64: Document the KVM ABI for SME Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-10-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=13260; i=broonie@kernel.org; h=from:subject:message-id; bh=MlzU8uizVpCnnQ1OWeHj73Xfek2PC3e+VyU2b4Ykjpw=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp2HtH279ZOZ7NF9FOV1w+BZOTUy+QEaKSW/8y+ 4kbxH5aJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKdgAKCRAk1otyXVSH0D2bB/ 0SiA+uFTJ45TVbC/oK8GVpq7WIIxBm0xqQp1uGOW0utzYtoseq38nBzXRMi+jct8X6f0LJT5koyZ2X plTcx3J6ot24FZ5vyJlMWyTVob6nfP5Czu7ADLu73e/SZrReVZIjo5KDsUBVH/D56iCvFq/a1vo5Kx lhsdEivPYIrrpqBfWp1vjZJkSjEMMjkwyn5dkwqR7vaqmQz8Lzxfesay0UFuY6YmLD4Pk4fU+e/GEk d1Vh1KxsWJ0jMdKJQ5eRyKglv2ypadVREJe+TzB43eUb9bvweIh0kQzZqrCm2jhfA8y+as2Zr0oxvp yLIYcn0xK7g2wqjXrzsxpftC9mTjO6 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME, the Scalable Matrix Extension, is an arm64 extension which adds support for matrix operations, with core concepts patterned after SVE. SVE introduced some complication in the ABI since it adds new vector floating point registers with runtime configurable size, the size being controlled by a prameter called the vector length (VL). To provide control of this to VMMs we offer two phase configuration of SVE, SVE must first be enabled for the vCPU with KVM_ARM_VCPU_INIT(KVM_ARM_VCPU_SVE), after which vector length may then be configured but the configurably sized floating point registers are inaccessible until finalized with a call to KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_SVE) after which the configurably sized registers can be accessed. SME introduces an additional independent configurable vector length which as well as controlling the size of the new ZA register also provides an alternative view of the configurably sized SVE registers (known as streaming mode) with the guest able to switch between the two modes as it pleases. There is also a fixed sized register ZT0 introduced in SME2. As well as streaming mode the guest may enable and disable ZA and (where SME2 is available) ZT0 dynamically independently of streaming mode. These modes are controlled via the system register SVCR. We handle the configuration of the vector length for SME in a similar manner to SVE, requiring initialization and finalization of the feature with a pseudo register controlling the available SME vector lengths as for SVE. Further, if the guest has both SVE and SME then finalizing one prevents further configuration of the vector length for the other. Where both SVE and SME are configured for the guest we always present the SVE registers to userspace as having the larger of the configured maximum SVE and SME vector lengths, discarding extra data at load time and zero padding on read as required if the active vector length is lower. Note that this means that enabling or disabling streaming mode while the guest is stopped will not zero Zn or Pn as it will when the guest is running, but it does allow SVCR, Zn and Pn to be read and written in any order. Userspace access to ZA and (if configured) ZT0 is always available, they will be zeroed when the guest runs if disabled in SVCR and the value read will be zero if the guest stops with them disabled. This mirrors the behaviour of the architecture, enabling access causes ZA and ZT0 to be zeroed, while allowing access to SVCR, ZA and ZT0 to be performed in any order. If SME is enabled for a guest without SVE then the FPSIMD Vn registers must be accessed via the low 128 bits of the SVE Zn registers as is the case when SVE is enabled. This is not ideal but allows access to SVCR and the registers in any order without duplication or ambiguity about which values should take effect. This may be an issue for VMMs that are unaware of SME on systems that implement it without SVE if they let SME be enabled, the lack of access to Vn may surprise them, but it seems like an unusual implementation choice. For SME unware VMMs on systems with both SVE and SME support the SVE registers may be larger than expected, this should be less disruptive than on a system without SVE as they will simply ignore the high bits of the registers. Signed-off-by: Mark Brown --- Documentation/virt/kvm/api.rst | 117 +++++++++++++++++++++++++++++------------ 1 file changed, 82 insertions(+), 35 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 47c7c3f92314..5cc30a93fdd7 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -406,7 +406,7 @@ Errors: instructions from device memory (arm64) ENOSYS data abort outside memslots with no syndrome info and KVM_CAP_ARM_NISV_TO_USER not enabled (arm64) - EPERM SVE feature set but not finalized (arm64) + EPERM SVE or SME feature set but not finalized (arm64) ======= ============================================================== This ioctl is used to run a guest virtual cpu. While there are no @@ -2590,12 +2590,12 @@ Specifically: 0x6020 0000 0010 00d5 FPCR 32 fp_regs.fpcr ======================= ========= ===== ======================================= -.. [1] These encodings are not accepted for SVE-enabled vcpus. See - :ref:`KVM_ARM_VCPU_INIT`. +.. [1] These encodings are not accepted for SVE enabled vcpus. See + :ref:`KVM_ARM_VCPU_INIT`. They are also not accepted when SME is + enabled without SVE and the vcpu is in streaming mode. The equivalent register content can be accessed via bits [127:0] of - the corresponding SVE Zn registers instead for vcpus that have SVE - enabled (see below). + the corresponding SVE Zn registers in these cases (see below). arm64 CCSIDR registers are demultiplexed by CSSELR value:: @@ -2626,24 +2626,34 @@ arm64 SVE registers have the following bit patterns:: 0x6050 0000 0015 060 FFR bits[256*slice + 255 : 256*slice] 0x6060 0000 0015 ffff KVM_REG_ARM64_SVE_VLS pseudo-register -Access to register IDs where 2048 * slice >= 128 * max_vq will fail with -ENOENT. max_vq is the vcpu's maximum supported vector length in 128-bit -quadwords: see [2]_ below. +arm64 SME registers have the following bit patterns: + + 0x6080 0000 0017 00 ZA.H[n] bits[2048*slice + 2047 : 2048*slice] + 0x60XX 0000 0017 0100 ZT0 + 0x6060 0000 0017 fffe KVM_REG_ARM64_SME_VLS pseudo-register + +Access to Z, P or ZA register IDs where 2048 * slice >= 128 * max_vq +will fail with ENOENT. max_vq is the vcpu's maximum supported vector +length in 128-bit quadwords: see [2]_ below. + +Access to the ZA and ZT0 registers is only available if SVCR.ZA is set +to 1. These registers are only accessible on vcpus for which SVE is enabled. See KVM_ARM_VCPU_INIT for details. -In addition, except for KVM_REG_ARM64_SVE_VLS, these registers are not -accessible until the vcpu's SVE configuration has been finalized -using KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_SVE). See KVM_ARM_VCPU_INIT -and KVM_ARM_VCPU_FINALIZE for more information about this procedure. +In addition, except for KVM_REG_ARM64_SVE_VLS and +KVM_REG_ARM64_SME_VLS, these registers are not accessible until the +vcpu's SVE and SME configuration has been finalized using +KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC). See KVM_ARM_VCPU_INIT and +KVM_ARM_VCPU_FINALIZE for more information about this procedure. -KVM_REG_ARM64_SVE_VLS is a pseudo-register that allows the set of vector -lengths supported by the vcpu to be discovered and configured by -userspace. When transferred to or from user memory via KVM_GET_ONE_REG -or KVM_SET_ONE_REG, the value of this register is of type -__u64[KVM_ARM64_SVE_VLS_WORDS], and encodes the set of vector lengths as -follows:: +KVM_REG_ARM64_SVE_VLS and KVM_ARM64_VCPU_SME_VLS are pseudo-registers +that allows the set of vector lengths supported by the vcpu to be +discovered and configured by userspace. When transferred to or from +user memory via KVM_GET_ONE_REG or KVM_SET_ONE_REG, the value of this +register is of type __u64[KVM_ARM64_SVE_VLS_WORDS], and encodes the +set of vector lengths as follows:: __u64 vector_lengths[KVM_ARM64_SVE_VLS_WORDS]; @@ -2655,19 +2665,25 @@ follows:: /* Vector length vq * 16 bytes not supported */ .. [2] The maximum value vq for which the above condition is true is - max_vq. This is the maximum vector length available to the guest on - this vcpu, and determines which register slices are visible through - this ioctl interface. + max_vq. This is the maximum vector length currently available to + the guest on this vcpu, and determines which register slices are + visible through this ioctl interface. + + If SME is supported then the max_vq used for the Z and P registers + then while SVCR.SM is 1 this vector length will be the maximum SME + vector length available for the guest, otherwise it will be the + maximum SVE vector length available. (See Documentation/arch/arm64/sve.rst for an explanation of the "vq" nomenclature.) -KVM_REG_ARM64_SVE_VLS is only accessible after KVM_ARM_VCPU_INIT. -KVM_ARM_VCPU_INIT initialises it to the best set of vector lengths that -the host supports. +KVM_REG_ARM64_SVE_VLS and KVM_REG_ARM_SME_VLS are only accessible +after KVM_ARM_VCPU_INIT. KVM_ARM_VCPU_INIT initialises them to the +best set of vector lengths that the host supports. -Userspace may subsequently modify it if desired until the vcpu's SVE -configuration is finalized using KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_SVE). +Userspace may subsequently modify these registers if desired until the +vcpu's SVE and SME configuration is finalized using +KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC). Apart from simply removing all vector lengths from the host set that exceed some value, support for arbitrarily chosen sets of vector lengths @@ -2675,8 +2691,8 @@ is hardware-dependent and may not be available. Attempting to configure an invalid set of vector lengths via KVM_SET_ONE_REG will fail with EINVAL. -After the vcpu's SVE configuration is finalized, further attempts to -write this register will fail with EPERM. +After the vcpu's SVE or SME configuration is finalized, further +attempts to write these registers will fail with EPERM. arm64 bitmap feature firmware pseudo-registers have the following bit pattern:: @@ -3459,6 +3475,7 @@ The initial values are defined as: - General Purpose registers, including PC and SP: set to 0 - FPSIMD/NEON registers: set to 0 - SVE registers: set to 0 + - SME registers: set to 0 - System registers: Reset to their architecturally defined values as for a warm reset to EL1 (resp. SVC) @@ -3501,7 +3518,7 @@ Possible features: - KVM_ARM_VCPU_SVE: Enables SVE for the CPU (arm64 only). Depends on KVM_CAP_ARM_SVE. - Requires KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_SVE): + Requires KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC): * After KVM_ARM_VCPU_INIT: @@ -3509,7 +3526,7 @@ Possible features: initial value of this pseudo-register indicates the best set of vector lengths possible for a vcpu on this host. - * Before KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_SVE): + * Before KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC}): - KVM_RUN and KVM_GET_REG_LIST are not available; @@ -3522,11 +3539,40 @@ Possible features: KVM_SET_ONE_REG, to modify the set of vector lengths available for the vcpu. - * After KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_SVE): + * After KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC): - the KVM_REG_ARM64_SVE_VLS pseudo-register is immutable, and can no longer be written using KVM_SET_ONE_REG. + - KVM_ARM_VCPU_SME: Enables SME for the CPU (arm64 only). + Depends on KVM_CAP_ARM_SME. + Requires KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC): + + * After KVM_ARM_VCPU_INIT: + + - KVM_REG_ARM64_SME_VLS may be read using KVM_GET_ONE_REG: the + initial value of this pseudo-register indicates the best set of + vector lengths possible for a vcpu on this host. + + * Before KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC}): + + - KVM_RUN and KVM_GET_REG_LIST are not available; + + - KVM_GET_ONE_REG and KVM_SET_ONE_REG cannot be used to access + the scalable architectural SVE registers + KVM_REG_ARM64_SVE_ZREG(), KVM_REG_ARM64_SVE_PREG() or + KVM_REG_ARM64_SVE_FFR, the matrix register + KVM_REG_ARM64_SME_ZA() or the LUT register KVM_REG_ARM64_ZT(); + + - KVM_REG_ARM64_SME_VLS may optionally be written using + KVM_SET_ONE_REG, to modify the set of vector lengths available + for the vcpu. + + * After KVM_ARM_VCPU_FINALIZE(KVM_ARM_VCPU_VEC): + + - the KVM_REG_ARM64_SME_VLS pseudo-register is immutable, and can + no longer be written using KVM_SET_ONE_REG. + 4.83 KVM_ARM_PREFERRED_TARGET ----------------------------- @@ -5096,11 +5142,12 @@ Errors: Recognised values for feature: - ===== =========================================== - arm64 KVM_ARM_VCPU_SVE (requires KVM_CAP_ARM_SVE) - ===== =========================================== + ===== ============================================================== + arm64 KVM_ARM_VCPU_VEC (requires KVM_CAP_ARM_SVE or KVM_CAP_ARM_SME) + arm64 KVM_ARM_VCPU_SVE (alias for KVM_ARM_VCPU_VEC) + ===== ============================================================== -Finalizes the configuration of the specified vcpu feature. +Finalizes the configuration of the specified vcpu features. The vcpu must already have been initialised, enabling the affected feature, by means of a successful :ref:`KVM_ARM_VCPU_INIT ` call with the From patchwork Thu Apr 17 00:25:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054710 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AC0C51F417F; Thu, 17 Apr 2025 00:26:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849585; cv=none; b=gsiEynohZ/hLstDzwcym3i+zvimRGIjP+HEdsDFnI/rw/ROloHs74DyxpKViq7WglHKN/4KkRngKUKnOz98qYiRxsfsiLUf4pjkds3KRE3Fhaut9Q37oCBYQCtP5FAZR7v5/dnigCp0OKgAxrZqdwO1o6UlNI4YrWfUOBsDGsO8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849585; c=relaxed/simple; bh=BYdob+W33p0p38mPrG7hoqJmur4cAV8L1cxc/90jdsM=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=CHzGb4r76SpyFPdpcfMCQJ/87G1CBirRqGLtPKmjezqYZZh9b52vOx/EoY3gasXu/QlwS4OpNLLR5b476WMWcGTwmsyPIs9BNxl9vLaGSLbOyW7vLzm/bOtAlGLz/i1fYidD4cs+DzNx2vWvQknBhOY8XDpzx9pYD0lkwgLTAks= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=WMd9iCFb; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="WMd9iCFb" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 565F6C4CEEC; Thu, 17 Apr 2025 00:26:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849585; bh=BYdob+W33p0p38mPrG7hoqJmur4cAV8L1cxc/90jdsM=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=WMd9iCFb4qyJ1YAYVLv4rwJJ9diJAMkdqwfehdJVANNo+mEGx0Wxsw8yM7YAvDndK fw2YHZRKB/irHhiuJWaGXlO0WytqVPZYFwzOCAgLKi06Gq8zfpQBJDNE/Za/h7qWyN hvl+b+AqitRVS8ojJIbHZEfDvV5i9k4dWidHsjozXg9IcdM+gloK3A1p48Si0z/vAz /sHCKrMXNZEJMCplNxarYV7RG/HKc/yMGgHkAR0orSDmTtvVQIeG+TD7sanrYHw9hX EDIvohL776bK6fmikjwJkEXrLL2NRO9eOzmQHVjzckbImO5DFUioikebwnvMM+yonU mLHn+VD3z75Cw== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:15 +0100 Subject: [PATCH v5 11/28] KVM: arm64: Define internal features for SME Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-11-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3485; i=broonie@kernel.org; h=from:subject:message-id; bh=BYdob+W33p0p38mPrG7hoqJmur4cAV8L1cxc/90jdsM=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp3I8DJjtzyfDKd7nlH0QFerMoK4BdwF667LOMV OrBptoOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKdwAKCRAk1otyXVSH0DRjB/ 4sZN1FtqP3EImLO6pygoyQeYddHPrz4QsBEzA7n4E4WbDcRuEO5ELi2f1ovP+mLMdhTKkKaIqbqSYZ 6H2lAS2mUWfY6Gjuk32JB50ibWxfiIyMOPwyiLIq4zKqgufsvuhEFBiBghM5GN5fFkAdUHiEoxxe1n kyhe1s0e622pdJIUbpK3rYjirr1U5qM4D5dtfyGproj/Ja2010vrF0aoCAtA4X95wsKipyJGKnu9zp SBfvkKqZ5W/aBZ5vgDvLNrRLPigLQRgg89IL6bGvn6D1QukDuk5VoZOO5RSeFXBTKnmd4e8t5sqTWP REKs8L8/aj16Jl/Xq4ktpLs0o+8ikv X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB In order to simplify interdependencies in the rest of the series define the feature detection for SME and it's subfeatures. Due to the need for vector length configuration we define a flag for SME like for SVE. We also have two subfeatures which add architectural state, FA64 and SME2, which are configured via the normal ID register scheme. Also provide helpers which check if the vCPU is in streaming mode or has ZA enabled. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 37 +++++++++++++++++++++++++++++++++++-- arch/arm64/kvm/sys_regs.c | 2 +- 2 files changed, 36 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e828607653c7..a6690893ec5e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -39,7 +39,7 @@ #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 9 #define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1) #define KVM_REQ_SLEEP \ @@ -348,6 +348,8 @@ struct kvm_arch { #define KVM_ARCH_FLAG_GUEST_HAS_SVE 9 /* MIDR_EL1, REVIDR_EL1, and AIDR_EL1 are writable from userspace */ #define KVM_ARCH_FLAG_WRITABLE_IMP_ID_REGS 10 + /* SME exposed to guest */ +#define KVM_ARCH_FLAG_GUEST_HAS_SME 11 unsigned long flags; /* VM-wide vCPU feature set */ @@ -982,7 +984,16 @@ struct kvm_vcpu_arch { #define vcpu_has_sve(vcpu) kvm_has_sve((vcpu)->kvm) #endif -#define vcpu_has_vec(vcpu) vcpu_has_sve(vcpu) +#define kvm_has_sme(kvm) (system_supports_sme() && \ + test_bit(KVM_ARCH_FLAG_GUEST_HAS_SME, &(kvm)->arch.flags)) + +#ifdef __KVM_NVHE_HYPERVISOR__ +#define vcpu_has_sme(vcpu) kvm_has_sme(kern_hyp_va((vcpu)->kvm)) +#else +#define vcpu_has_sme(vcpu) kvm_has_sme((vcpu)->kvm) +#endif + +#define vcpu_has_vec(vcpu) (vcpu_has_sve(vcpu) || vcpu_has_sme(vcpu)) #ifdef CONFIG_ARM64_PTR_AUTH #define vcpu_has_ptrauth(vcpu) \ @@ -1596,4 +1607,26 @@ void kvm_set_vm_id_reg(struct kvm *kvm, u32 reg, u64 val); #define kvm_has_s1poe(k) \ (kvm_has_feat((k), ID_AA64MMFR3_EL1, S1POE, IMP)) +#define kvm_has_fa64(k) \ + (system_supports_fa64() && \ + kvm_has_feat((k), ID_AA64SMFR0_EL1, FA64, IMP)) + +#define kvm_has_sme2(k) \ + (system_supports_sme2() && \ + kvm_has_feat((k), ID_AA64PFR1_EL1, SME, SME2)) + +#ifdef __KVM_NVHE_HYPERVISOR__ +#define vcpu_has_sme2(vcpu) kvm_has_sme2(kern_hyp_va((vcpu)->kvm)) +#define vcpu_has_fa64(vcpu) kvm_has_fa64(kern_hyp_va((vcpu)->kvm)) +#else +#define vcpu_has_sme2(vcpu) kvm_has_sme2((vcpu)->kvm) +#define vcpu_has_fa64(vcpu) kvm_has_fa64((vcpu)->kvm) +#endif + +#define vcpu_in_streaming_mode(vcpu) \ + (__vcpu_sys_reg(vcpu, SVCR) & SVCR_SM_MASK) + +#define vcpu_za_enabled(vcpu) \ + (__vcpu_sys_reg(vcpu, SVCR) & SVCR_ZA_MASK) + #endif /* __ARM64_KVM_HOST_H__ */ diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 005ad28f7306..0fc33134ac41 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1772,7 +1772,7 @@ static unsigned int sve_visibility(const struct kvm_vcpu *vcpu, static unsigned int sme_visibility(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd) { - if (kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, SME, IMP)) + if (vcpu_has_sme(vcpu)) return 0; return REG_HIDDEN; From patchwork Thu Apr 17 00:25:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054711 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A73C818FC67; Thu, 17 Apr 2025 00:26:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849589; cv=none; b=Dt57PJ0RsMFiZW6+mzhuMqqOtbexXCXJA9flFSRxCbkyNZAiN8rAbZ7Io7Z9Kv+EuwT9od8BYQ1a0A184o/gvlMSWUzR2dr4EICOEq/gFWaEHljzNa9r3RVFR7JR/OMzQbky5pXX745iWkfpIG+Fin5NAI/MIBXfQNidqMWyWq4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849589; c=relaxed/simple; bh=W9U/EIqt1ZpD/AI0aRJHQ2z1yafgwHeiK/I9nw34e+w=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=a0Tbt02ZWYQo19fIVz78MnyDwyodpTcYPov+TV5HB352DowfMokSA470pqF2H7nvtGsdFLT2T5xQQqHZP1oP7lp3iwTehlT8//4hBse0HL5ycz4O8mk5zyHzjuW4ZEM/CtmCJk9QeTW+ZYuDzfX99y7/8ykZEJXh2OaCT2CWE44= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XZB1ePwA; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XZB1ePwA" Received: by smtp.kernel.org (Postfix) with ESMTPSA id F318EC4CEEC; Thu, 17 Apr 2025 00:26:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849589; bh=W9U/EIqt1ZpD/AI0aRJHQ2z1yafgwHeiK/I9nw34e+w=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=XZB1ePwAD4xn+kJoCzFpd9PuTtr6S5KPxV1NdK5h9rwlIB1fu5BOjaSBoMLeoFljC d3hoM788Nb2Q+znfBYo4J6DY7y81WsdY+s7T9yORk++gvuL6ynpUsZPVfUE8aYIe17 /SiZPeyr00dmOmuwOsBZlignpz70ACSVrF61s+Dg5sgauSXzyjis7sSg2mdR5yChuM mieHx3FL76zZvMhquRJwjoUQRUdp4le58Y+M+/7N0alMge8/YXpohKgTZfZwp75AcQ 1b9woftk7JTlKNLalyIPwAzViDxFF1psLyyfseKG2pxHmpagbnFHMU+M/qgRi7CdXq eLtRvkvHIQr0g== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:16 +0100 Subject: [PATCH v5 12/28] KVM: arm64: Rename sve_state_reg_region Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-12-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=2352; i=broonie@kernel.org; h=from:subject:message-id; bh=W9U/EIqt1ZpD/AI0aRJHQ2z1yafgwHeiK/I9nw34e+w=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp4aYB+lTSBEvBGlKqa5ArcHrjS9+NUYvf34HDS cs96KduJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKeAAKCRAk1otyXVSH0IxnB/ oDVpFjTwsx42rNNBW7KDL0zDS+LJuIuMIdcu0iQzS3p6e6NkHvYdCO4fht5WtwUKx+dquGFKVQc5af DXiz4X8MJ4NbvNT7t1dMMrJIVYg5rcVqj3h2EXiw7HxLT9Au/UdbwfwZlLzbNs8dk+JKGfcg5piFig WLlJoOys7FyombKnLbu/Vz2KqHF5s8mIL0orW/zYKSf0iZfOM5R0AQbxlNVzckUcTAzfWzh6CP6XZI GMHWMx9RdUdUFuRXJIgJeCfpy797RhiTnFcO4vHoa0vavS9oNL+tqKluCpfKYyVENkUXF/tEFqEOQ7 IHZBdPE+BY7TnXf5jCQPFrxg9EgSVb X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB As for SVE we will need to pull parts of dynamically sized registers out of a block of memory for SME so we will use a similar code pattern for this. Rename the current struct sve_state_reg_region in preparation for this. No functional change. Signed-off-by: Mark Brown --- arch/arm64/kvm/guest.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 73e714133bb6..bb4b91e43923 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -404,9 +404,9 @@ static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) */ #define vcpu_sve_slices(vcpu) 1 -/* Bounds of a single SVE register slice within vcpu->arch.sve_state */ -struct sve_state_reg_region { - unsigned int koffset; /* offset into sve_state in kernel memory */ +/* Bounds of a single register slice within vcpu->arch.s[mv]e_state */ +struct vec_state_reg_region { + unsigned int koffset; /* offset into s[mv]e_state in kernel memory */ unsigned int klen; /* length in kernel memory */ unsigned int upad; /* extra trailing padding in user memory */ }; @@ -415,7 +415,7 @@ struct sve_state_reg_region { * Validate SVE register ID and get sanitised bounds for user/kernel SVE * register copy */ -static int sve_reg_to_region(struct sve_state_reg_region *region, +static int sve_reg_to_region(struct vec_state_reg_region *region, struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { @@ -485,7 +485,7 @@ static int sve_reg_to_region(struct sve_state_reg_region *region, static int get_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { int ret; - struct sve_state_reg_region region; + struct vec_state_reg_region region; char __user *uptr = (char __user *)reg->addr; /* Handle the KVM_REG_ARM64_SVE_VLS pseudo-reg as a special case: */ @@ -511,7 +511,7 @@ static int get_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) static int set_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { int ret; - struct sve_state_reg_region region; + struct vec_state_reg_region region; const char __user *uptr = (const char __user *)reg->addr; /* Handle the KVM_REG_ARM64_SVE_VLS pseudo-reg as a special case: */ From patchwork Thu Apr 17 00:25:17 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054712 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5DBEE78F5B; Thu, 17 Apr 2025 00:26:32 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849593; cv=none; b=qhRHG8GQ6DH3GKHwseT71TPUU9NEn/lmHLA0V71wlj9YlWB8e9jHTJrtHbklIQN9wbXR0QMQ/a7jrYSWnONE2KxnOeDxuwZ7cyuWB3+pdbGzl6/+I27hxrY1nfDKNZGAWQva39Oj0oj/Q7b5eoEznuZgKJ4lIWGODpTLsPuff18= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849593; c=relaxed/simple; bh=IqYfQW6t9ldOaFOAc2sm8Sm3afA06l36aWtBVEd2xJo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=cf13FRlzmZjZ9wQuYf19ORrQJQNwc054xIGrlD44xvEvpUoE0v7aaC5OK1QWtS8cy0QY79LGI8W6JiTNEBsggMqmqJRhGi6dwA9BafPU/5zvIoD5soe8BOTx78/gcDV8mi3f5RYF3qA5UhAzVHlQivuIzHDes6LpN59caQzyJP0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=XNLJj89d; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="XNLJj89d" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A3C3C4CEEF; Thu, 17 Apr 2025 00:26:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849592; bh=IqYfQW6t9ldOaFOAc2sm8Sm3afA06l36aWtBVEd2xJo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=XNLJj89dWCjpgAWtV/HOydNXPItY4H5FJqiN1bJnCbM2FJAnBJseMRr7wFhIP9lsW 3ELvyuxgR5Vi4H4sxxPUs3wq7Sxa1EzbwWK2VQ5zsgQKD+isFlsWBvQB34JtSS0WrS 5ZaPopsAbzAhGhWdzi78fTef7xT7qBrr0Edo5OQ8W0fEvNzc0fTIpHBa+8gdHZ72cJ PXhfpYe1pPZlH91ZWjEWB8cZjkDfanZaySM3L6HUmeTesi0stHMeRAtrfJEtz4uRJr LK2WrEJX20KnEqtYlYv9346MwTIP2JPitgn1FNeN2V4qvzQuC8OxOKqauumsfGRKtG atPDAxydg4y3A== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:17 +0100 Subject: [PATCH v5 13/28] KVM: arm64: Store vector lengths in an array Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-13-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=11859; i=broonie@kernel.org; h=from:subject:message-id; bh=IqYfQW6t9ldOaFOAc2sm8Sm3afA06l36aWtBVEd2xJo=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp5VIQv2ZwRZXIBQ0iZCnJQHfwr18enjoinmDVj ODERwIOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKeQAKCRAk1otyXVSH0LKCCA CF9j2zSmUQPTEzRQIh1oMCEwp7VG1wWvcqJoXovJqgY1HAyAEz1FWfiCxQ+Pog0/hjIMnhopJ2WGQO qtl94BbC/q1tKAn0yP1YitRYeupThjkZuBrEWSH0eoutn8d329TNdEaZQf1saPtfHX5K+pT9B2cYrc roFpkVoCgZlIBdve/ICK/IrdpjU/lzK2jfiJOcmRib3FYnK6VXdSj/AIUR7FhaaaWkerdocLG1NKle GH/0pydMwYMtUzmFNc2llXQdhPDcEP8Dk4dNR+f9hSTRlJEr6vmZMyBSBSHJvbQrwC9QUechzQ11SC PCxrDDesxAj8KY7axCceUvot6JpPhw X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME adds a second vector length configured in a very similar way to the SVE vector length, in order to facilitate future code sharing for SME refactor our storage of vector lengths to use an array like the host does. We do not yet take much advantage of this so the intermediate code is not as clean as might be. No functional change. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 17 +++++++++++------ arch/arm64/include/asm/kvm_hyp.h | 2 +- arch/arm64/include/asm/kvm_pkvm.h | 2 +- arch/arm64/kvm/fpsimd.c | 2 +- arch/arm64/kvm/guest.c | 6 +++--- arch/arm64/kvm/hyp/include/hyp/switch.h | 6 +++--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 11 +++++++---- arch/arm64/kvm/hyp/nvhe/pkvm.c | 2 +- arch/arm64/kvm/reset.c | 22 +++++++++++----------- 9 files changed, 39 insertions(+), 31 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a6690893ec5e..d5ad85478010 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -75,8 +75,10 @@ enum kvm_mode kvm_get_mode(void); static inline enum kvm_mode kvm_get_mode(void) { return KVM_MODE_NONE; }; #endif -extern unsigned int __ro_after_init kvm_sve_max_vl; -extern unsigned int __ro_after_init kvm_host_sve_max_vl; +extern unsigned int __ro_after_init kvm_max_vl[ARM64_VEC_MAX]; +extern unsigned int __ro_after_init kvm_host_max_vl[ARM64_VEC_MAX]; +DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use); + int __init kvm_arm_init_sve(void); u32 __attribute_const__ kvm_target_cpu(void); @@ -753,7 +755,7 @@ struct kvm_vcpu_arch { */ void *sve_state; enum fp_type fp_type; - unsigned int sve_max_vl; + unsigned int max_vl[ARM64_VEC_MAX]; /* Stage 2 paging state used by the hardware on next switch */ struct kvm_s2_mmu *hw_mmu; @@ -1018,9 +1020,12 @@ struct kvm_vcpu_arch { /* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ #define vcpu_sve_pffr(vcpu) (kern_hyp_va((vcpu)->arch.sve_state) + \ - sve_ffr_offset((vcpu)->arch.sve_max_vl)) + sve_ffr_offset((vcpu)->arch.max_vl[ARM64_VEC_SVE])) + +#define vcpu_vec_max_vq(vcpu, type) sve_vq_from_vl((vcpu)->arch.max_vl[type]) + +#define vcpu_sve_max_vq(vcpu) vcpu_vec_max_vq(vcpu, ARM64_VEC_SVE) -#define vcpu_sve_max_vq(vcpu) sve_vq_from_vl((vcpu)->arch.sve_max_vl) #define vcpu_sve_zcr_elx(vcpu) \ (unlikely(is_hyp_ctxt(vcpu)) ? ZCR_EL2 : ZCR_EL1) @@ -1029,7 +1034,7 @@ struct kvm_vcpu_arch { size_t __size_ret; \ unsigned int __vcpu_vq; \ \ - if (WARN_ON(!sve_vl_valid((vcpu)->arch.sve_max_vl))) { \ + if (WARN_ON(!sve_vl_valid((vcpu)->arch.max_vl[ARM64_VEC_SVE]))) { \ __size_ret = 0; \ } else { \ __vcpu_vq = vcpu_sve_max_vq(vcpu); \ diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index e6be1f5d0967..0ad5a66e0d25 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -145,6 +145,6 @@ extern u64 kvm_nvhe_sym(id_aa64smfr0_el1_sys_val); extern unsigned long kvm_nvhe_sym(__icache_flags); extern unsigned int kvm_nvhe_sym(kvm_arm_vmid_bits); -extern unsigned int kvm_nvhe_sym(kvm_host_sve_max_vl); +extern unsigned int kvm_nvhe_sym(kvm_host_max_vl[ARM64_VEC_MAX]); #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/include/asm/kvm_pkvm.h b/arch/arm64/include/asm/kvm_pkvm.h index abd693ce5b93..df64a3f7bcb4 100644 --- a/arch/arm64/include/asm/kvm_pkvm.h +++ b/arch/arm64/include/asm/kvm_pkvm.h @@ -160,7 +160,7 @@ static inline size_t pkvm_host_sve_state_size(void) return 0; return size_add(sizeof(struct cpu_sve_state), - SVE_SIG_REGS_SIZE(sve_vq_from_vl(kvm_host_sve_max_vl))); + SVE_SIG_REGS_SIZE(sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SVE]))); } struct pkvm_mapping { diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 7fc14b7349a0..72c67b067ca1 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -101,7 +101,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) */ fp_state.st = &vcpu->arch.ctxt.fp_regs; fp_state.sve_state = vcpu->arch.sve_state; - fp_state.sve_vl = vcpu->arch.sve_max_vl; + fp_state.sve_vl = vcpu->arch.max_vl[ARM64_VEC_SVE]; fp_state.sme_state = NULL; fp_state.svcr = &__vcpu_sys_reg(vcpu, SVCR); fp_state.fpmr = &__vcpu_sys_reg(vcpu, FPMR); diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index bb4b91e43923..e9f17b8a6fba 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -318,7 +318,7 @@ static int get_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (!vcpu_has_sve(vcpu)) return -ENOENT; - if (WARN_ON(!sve_vl_valid(vcpu->arch.sve_max_vl))) + if (WARN_ON(!sve_vl_valid(vcpu->arch.max_vl[ARM64_VEC_SVE]))) return -EINVAL; memset(vqs, 0, sizeof(vqs)); @@ -356,7 +356,7 @@ static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (vq_present(vqs, vq)) max_vq = vq; - if (max_vq > sve_vq_from_vl(kvm_sve_max_vl)) + if (max_vq > sve_vq_from_vl(kvm_max_vl[ARM64_VEC_SVE])) return -EINVAL; /* @@ -375,7 +375,7 @@ static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return -EINVAL; /* vcpu->arch.sve_state will be alloc'd by kvm_vcpu_finalize_sve() */ - vcpu->arch.sve_max_vl = sve_vl_from_vq(max_vq); + vcpu->arch.max_vl[ARM64_VEC_SVE] = sve_vl_from_vq(max_vq); return 0; } diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 7c9355459648..1c0ef9d6609d 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -370,8 +370,8 @@ static inline void __hyp_sve_save_host(void) struct cpu_sve_state *sve_state = *host_data_ptr(sve_state); sve_state->zcr_el1 = read_sysreg_el1(SYS_ZCR); - write_sysreg_s(sve_vq_from_vl(kvm_host_sve_max_vl) - 1, SYS_ZCR_EL2); - __sve_save_state(sve_state->sve_regs + sve_ffr_offset(kvm_host_sve_max_vl), + write_sysreg_s(sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SVE]) - 1, SYS_ZCR_EL2); + __sve_save_state(sve_state->sve_regs + sve_ffr_offset(kvm_host_max_vl[ARM64_VEC_SVE]), &sve_state->fpsr, true); } @@ -426,7 +426,7 @@ static inline void fpsimd_lazy_switch_to_host(struct kvm_vcpu *vcpu) zcr_el2 = vcpu_sve_max_vq(vcpu) - 1; write_sysreg_el2(zcr_el2, SYS_ZCR); } else { - zcr_el2 = sve_vq_from_vl(kvm_host_sve_max_vl) - 1; + zcr_el2 = sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SVE]) - 1; write_sysreg_el2(zcr_el2, SYS_ZCR); zcr_el1 = vcpu_sve_max_vq(vcpu) - 1; diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 2c37680d954c..1fb38dc64357 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -34,7 +34,7 @@ static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu) */ sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2); __sve_save_state(vcpu_sve_pffr(vcpu), &vcpu->arch.ctxt.fp_regs.fpsr, true); - write_sysreg_s(sve_vq_from_vl(kvm_host_sve_max_vl) - 1, SYS_ZCR_EL2); + write_sysreg_s(sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SVE]) - 1, SYS_ZCR_EL2); } static void __hyp_sve_restore_host(void) @@ -50,8 +50,8 @@ static void __hyp_sve_restore_host(void) * that was discovered, if we wish to use larger VLs this will * need to be revisited. */ - write_sysreg_s(sve_vq_from_vl(kvm_host_sve_max_vl) - 1, SYS_ZCR_EL2); - __sve_restore_state(sve_state->sve_regs + sve_ffr_offset(kvm_host_sve_max_vl), + write_sysreg_s(sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SVE]) - 1, SYS_ZCR_EL2); + __sve_restore_state(sve_state->sve_regs + sve_ffr_offset(kvm_host_max_vl[ARM64_VEC_SVE]), &sve_state->fpsr, true); write_sysreg_el1(sve_state->zcr_el1, SYS_ZCR); @@ -117,6 +117,7 @@ static void sync_debug_state(struct pkvm_hyp_vcpu *hyp_vcpu) static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) { struct kvm_vcpu *host_vcpu = hyp_vcpu->host_vcpu; + int i; fpsimd_sve_flush(); flush_debug_state(hyp_vcpu); @@ -125,7 +126,9 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.sve_state = kern_hyp_va(host_vcpu->arch.sve_state); /* Limit guest vector length to the maximum supported by the host. */ - hyp_vcpu->vcpu.arch.sve_max_vl = min(host_vcpu->arch.sve_max_vl, kvm_host_sve_max_vl); + for (i = 0; i < ARM64_VEC_MAX; i++) + hyp_vcpu->vcpu.arch.max_vl[i] = min(host_vcpu->arch.max_vl[i], + kvm_host_max_vl[i]); hyp_vcpu->vcpu.arch.mdcr_el2 = host_vcpu->arch.mdcr_el2; hyp_vcpu->vcpu.arch.hcr_el2 &= ~(HCR_TWI | HCR_TWE); diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index eff47b58effb..57b6d81006ae 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -20,7 +20,7 @@ unsigned long __icache_flags; /* Used by kvm_get_vttbr(). */ unsigned int kvm_arm_vmid_bits; -unsigned int kvm_host_sve_max_vl; +unsigned int kvm_host_max_vl[ARM64_VEC_MAX]; /* * The currently loaded hyp vCPU for each physical CPU. Used only when diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 2cf2028ab0dd..052a1a452f89 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -32,7 +32,7 @@ /* Maximum phys_shift supported for any VM on this host */ static u32 __ro_after_init kvm_ipa_limit; -unsigned int __ro_after_init kvm_host_sve_max_vl; +unsigned int __ro_after_init kvm_host_max_vl[ARM64_VEC_MAX]; /* * ARMv8 Reset Values @@ -46,14 +46,14 @@ unsigned int __ro_after_init kvm_host_sve_max_vl; #define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ PSR_AA32_I_BIT | PSR_AA32_F_BIT) -unsigned int __ro_after_init kvm_sve_max_vl; +unsigned int __ro_after_init kvm_max_vl[ARM64_VEC_MAX]; int __init kvm_arm_init_sve(void) { if (system_supports_sve()) { - kvm_sve_max_vl = sve_max_virtualisable_vl(); - kvm_host_sve_max_vl = sve_max_vl(); - kvm_nvhe_sym(kvm_host_sve_max_vl) = kvm_host_sve_max_vl; + kvm_max_vl[ARM64_VEC_SVE] = sve_max_virtualisable_vl(); + kvm_host_max_vl[ARM64_VEC_SVE] = sve_max_vl(); + kvm_nvhe_sym(kvm_host_max_vl[ARM64_VEC_SVE]) = kvm_host_max_vl[ARM64_VEC_SVE]; /* * The get_sve_reg()/set_sve_reg() ioctl interface will need @@ -61,16 +61,16 @@ int __init kvm_arm_init_sve(void) * order to support vector lengths greater than * VL_ARCH_MAX: */ - if (WARN_ON(kvm_sve_max_vl > VL_ARCH_MAX)) - kvm_sve_max_vl = VL_ARCH_MAX; + if (WARN_ON(kvm_max_vl[ARM64_VEC_SVE] > VL_ARCH_MAX)) + kvm_max_vl[ARM64_VEC_SVE] = VL_ARCH_MAX; /* * Don't even try to make use of vector lengths that * aren't available on all CPUs, for now: */ - if (kvm_sve_max_vl < sve_max_vl()) + if (kvm_max_vl[ARM64_VEC_SVE] < sve_max_vl()) pr_warn("KVM: SVE vector length for guests limited to %u bytes\n", - kvm_sve_max_vl); + kvm_max_vl[ARM64_VEC_SVE]); } return 0; @@ -78,7 +78,7 @@ int __init kvm_arm_init_sve(void) static void kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu) { - vcpu->arch.sve_max_vl = kvm_sve_max_vl; + vcpu->arch.max_vl[ARM64_VEC_SVE] = kvm_max_vl[ARM64_VEC_SVE]; /* * Userspace can still customize the vector lengths by writing @@ -99,7 +99,7 @@ static int kvm_vcpu_finalize_vec(struct kvm_vcpu *vcpu) size_t reg_sz; int ret; - vl = vcpu->arch.sve_max_vl; + vl = vcpu->arch.max_vl[ARM64_VEC_SVE]; /* * Responsibility for these properties is shared between From patchwork Thu Apr 17 00:25:18 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054713 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07B9A207A03; Thu, 17 Apr 2025 00:26:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849597; cv=none; b=iLWnRRxyKRbd2n1wBVuziTfKlXgCK+pJ9/cmajZXDzrQlosMiQGpZFsIQ19IFwj5N282st7Y2hL0sxGBfSq376hLEv2fMxr9kbgt9abGuEhMoG/HmRbWgDC07w6Gl5QODPhil3ncs+I/3qWS8rCgcd3ocrEIeyNH9PYKkw3azdU= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849597; c=relaxed/simple; bh=SoGNf8VLDcyGvjMMDrVy1nP3wwZE7cm2Qiw2Dqyk5zc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=h3RmMpSQ4fvCEjEG2UgeKW4bHVw3Txv97xCEvdXxEKTT9E6WNh9ON/+MHiGNjwfbfa2vOsgEsdlRPd41fOoe394uX/Yw6l2MHH/GRB3NmmF/rr+dvJBNYQ/EciVoyK1215XaDtWB8iWY5Zgc/HZYPCpuRq0dGKJmZoIjeyxaL4U= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=DAUZlGYF; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="DAUZlGYF" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 46445C4CEEA; Thu, 17 Apr 2025 00:26:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849596; bh=SoGNf8VLDcyGvjMMDrVy1nP3wwZE7cm2Qiw2Dqyk5zc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=DAUZlGYFG/jxronLcCvrfl23M7CxVm/yLwrpKUvkB54MWLvuPpCElIcE8jgKimaKg YTv5A2xYewiHd0T6bpH2iRK3OKADoLMEhtzhjKjdYES+qR0DWu6EybnVLxJRuvDdUy gCxKrnKafH33UXL7Q0rcItAwDezK1guC+u8794yE90Yw1UV1bLPOHsbFtdpVHXmfVO L2tzPHQ30BIuREHuSz3AGbtkrOdKirZjgfGwp/jaIzALhQ/e9xRnFaEGVKt4tzldfr GTQCAQ8W4gwSX/KkpkfqTrV16pxPvw77X5agY57wLhIJeWR9RDaSwUUOfajNgU9TMg OFuU7G68JRCaQ== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:18 +0100 Subject: [PATCH v5 14/28] KVM: arm64: Implement SME vector length configuration Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-14-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=10123; i=broonie@kernel.org; h=from:subject:message-id; bh=SoGNf8VLDcyGvjMMDrVy1nP3wwZE7cm2Qiw2Dqyk5zc=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp6wSzOULQFusLlMaNdJ6S5DNLevsvylJeVCmRS P+g8x8WJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKegAKCRAk1otyXVSH0DP3B/ 4kqa9dUxTw4gVdLIGbofSH0NY0viYSMReQOgg/fr/m8maV4xIm95fEiqrXtV45dNg4IrRzzY3zrU2G hABWIfTbm9RNrasgWTzULpjYCgGv86SxZkTrkrQIAUZryR9gm6CdpDHNVC299lsNRCAp1kwn3jJfl6 e+jDn0woF4lWxMP/cUvnjxEucFeXlWVeHdOJL+2ke1+yzIAVKHTU58e0DpBys9rPxMREdBV09wJH8B sYfZYeGkTi4plUS1kxakMp5byunxvm/iDVdjlYQQTzXgoi9ScfpGo4itWNeGFOhzLiT1gd7lqZLgJK PNlvp13iUs9aVGesHKKTFBKKlB1ov4 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME implements a vector length which architecturally looks very similar to that for SVE, configured in a very similar manner. This controls the vector length used for the ZA matrix register, and for the SVE vector and predicate registers when in streaming mode. The only substantial difference is that unlike SVE the architecture does not guarantee that any particular vector length will be implemented. Configuration for SME vector lengths is done using a virtual register as for SVE, hook up the implementation for the virtual register. Since we do not yet have support for any of the new SME registers stub register access functions are provided that only allow VL configuration. These will be extended as the SME specific registers, as for SVE. Since vq_available() is currently only defined for CONFIG_SVE add a stub for builds where that is disabled. Signed-off-by: Mark Brown --- arch/arm64/include/asm/fpsimd.h | 1 + arch/arm64/include/asm/kvm_host.h | 24 ++++++++++-- arch/arm64/include/uapi/asm/kvm.h | 9 +++++ arch/arm64/kvm/guest.c | 82 +++++++++++++++++++++++++++++++-------- 4 files changed, 96 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 573bd6a219f2..7af636885c04 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -321,6 +321,7 @@ static inline int sve_max_vl(void) return -EINVAL; } +static inline bool vq_available(enum vec_type type, unsigned int vq) { return false; } static inline bool sve_vq_available(unsigned int vq) { return false; } static inline void sve_user_disable(void) { BUILD_BUG(); } diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index d5ad85478010..8ac63de0a045 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -752,8 +752,15 @@ struct kvm_vcpu_arch { * low 128 bits of the SVE Z registers. When the core * floating point code saves the register state of a task it * records which view it saved in fp_type. + * + * If SME support is also present then it provides an + * alternative view of the SVE registers accessed as for the Z + * registers when PSTATE.SM is 1, plus an additional set of + * SME specific state in the matrix register ZA and LUT + * register ZT0. */ void *sve_state; + void *sme_state; enum fp_type fp_type; unsigned int max_vl[ARM64_VEC_MAX]; @@ -1018,14 +1025,23 @@ struct kvm_vcpu_arch { #define vcpu_gp_regs(v) (&(v)->arch.ctxt.regs) -/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ -#define vcpu_sve_pffr(vcpu) (kern_hyp_va((vcpu)->arch.sve_state) + \ - sve_ffr_offset((vcpu)->arch.max_vl[ARM64_VEC_SVE])) - #define vcpu_vec_max_vq(vcpu, type) sve_vq_from_vl((vcpu)->arch.max_vl[type]) #define vcpu_sve_max_vq(vcpu) vcpu_vec_max_vq(vcpu, ARM64_VEC_SVE) +#define vcpu_sme_max_vq(vcpu) vcpu_vec_max_vq(vcpu, ARM64_VEC_SME) + +#define vcpu_sve_max_vl(vcpu) ((vcpu)->arch.max_vl[ARM64_VEC_SVE]) +#define vcpu_sme_max_vl(vcpu) ((vcpu)->arch.max_vl[ARM64_VEC_SME]) +#define vcpu_max_vl(vcpu) max(vcpu_sve_max_vl(vcpu), vcpu_sme_max_vl(vcpu)) +#define vcpu_max_vq(vcpu) sve_vq_from_vl(vcpu_max_vl(vcpu)) + +#define vcpu_cur_sve_vl(vcpu) (vcpu_in_streaming_mode(vcpu) ? \ + vcpu_sme_max_vl(vcpu) : vcpu_sve_max_vl(vcpu)) + +/* Pointer to the vcpu's SVE FFR for sve_{save,load}_state() */ +#define vcpu_sve_pffr(vcpu) (kern_hyp_va((vcpu)->arch.sve_state) + \ + sve_ffr_offset(vcpu_cur_sve_vl(vcpu))) #define vcpu_sve_zcr_elx(vcpu) \ (unlikely(is_hyp_ctxt(vcpu)) ? ZCR_EL2 : ZCR_EL1) diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 42cfa86b96ca..c0a8aa8c67ea 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -354,6 +354,15 @@ struct kvm_arm_counter_offset { #define KVM_ARM64_SVE_VLS_WORDS \ ((KVM_ARM64_SVE_VQ_MAX - KVM_ARM64_SVE_VQ_MIN) / 64 + 1) +/* SME registers */ +#define KVM_REG_ARM64_SME (0x17 << KVM_REG_ARM_COPROC_SHIFT) + +/* Vector lengths pseudo-register: */ +#define KVM_REG_ARM64_SME_VLS (KVM_REG_ARM64 | KVM_REG_ARM64_SME | \ + KVM_REG_SIZE_U512 | 0xffff) +#define KVM_ARM64_SME_VLS_WORDS \ + ((KVM_ARM64_SVE_VQ_MAX - KVM_ARM64_SVE_VQ_MIN) / 64 + 1) + /* Bitmap feature firmware registers */ #define KVM_REG_ARM_FW_FEAT_BMAP (0x0016 << KVM_REG_ARM_COPROC_SHIFT) #define KVM_REG_ARM_FW_FEAT_BMAP_REG(r) (KVM_REG_ARM64 | KVM_REG_SIZE_U64 | \ diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index e9f17b8a6fba..6460bb21e01d 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -310,22 +310,20 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) #define vq_mask(vq) ((u64)1 << ((vq) - SVE_VQ_MIN) % 64) #define vq_present(vqs, vq) (!!((vqs)[vq_word(vq)] & vq_mask(vq))) -static int get_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +static int get_vec_vls(enum vec_type vec_type, struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) { unsigned int max_vq, vq; u64 vqs[KVM_ARM64_SVE_VLS_WORDS]; - if (!vcpu_has_sve(vcpu)) - return -ENOENT; - - if (WARN_ON(!sve_vl_valid(vcpu->arch.max_vl[ARM64_VEC_SVE]))) + if (WARN_ON(!sve_vl_valid(vcpu->arch.max_vl[vec_type]))) return -EINVAL; memset(vqs, 0, sizeof(vqs)); - max_vq = vcpu_sve_max_vq(vcpu); + max_vq = vcpu_vec_max_vq(vcpu, vec_type); for (vq = SVE_VQ_MIN; vq <= max_vq; ++vq) - if (sve_vq_available(vq)) + if (vq_available(vec_type, vq)) vqs[vq_word(vq)] |= vq_mask(vq); if (copy_to_user((void __user *)reg->addr, vqs, sizeof(vqs))) @@ -334,40 +332,41 @@ static int get_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return 0; } -static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +static int set_vec_vls(enum vec_type vec_type, struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) { unsigned int max_vq, vq; u64 vqs[KVM_ARM64_SVE_VLS_WORDS]; - if (!vcpu_has_sve(vcpu)) - return -ENOENT; - if (kvm_arm_vcpu_vec_finalized(vcpu)) return -EPERM; /* too late! */ - if (WARN_ON(vcpu->arch.sve_state)) + if (WARN_ON(!sve_vl_valid(vcpu->arch.max_vl[vec_type]))) return -EINVAL; if (copy_from_user(vqs, (const void __user *)reg->addr, sizeof(vqs))) return -EFAULT; + if (WARN_ON(vcpu->arch.sve_state || vcpu->arch.sme_state)) + return -EINVAL; + max_vq = 0; for (vq = SVE_VQ_MIN; vq <= SVE_VQ_MAX; ++vq) if (vq_present(vqs, vq)) max_vq = vq; - if (max_vq > sve_vq_from_vl(kvm_max_vl[ARM64_VEC_SVE])) + if (max_vq > sve_vq_from_vl(kvm_max_vl[vec_type])) return -EINVAL; /* * Vector lengths supported by the host can't currently be * hidden from the guest individually: instead we can only set a - * maximum via ZCR_EL2.LEN. So, make sure the available vector + * maximum via xCR_EL2.LEN. So, make sure the available vector * lengths match the set requested exactly up to the requested * maximum: */ for (vq = SVE_VQ_MIN; vq <= max_vq; ++vq) - if (vq_present(vqs, vq) != sve_vq_available(vq)) + if (vq_present(vqs, vq) != vq_available(vec_type, vq)) return -EINVAL; /* Can't run with no vector lengths at all: */ @@ -375,11 +374,27 @@ static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return -EINVAL; /* vcpu->arch.sve_state will be alloc'd by kvm_vcpu_finalize_sve() */ - vcpu->arch.max_vl[ARM64_VEC_SVE] = sve_vl_from_vq(max_vq); + vcpu->arch.max_vl[vec_type] = sve_vl_from_vq(max_vq); return 0; } +static int get_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +{ + if (!vcpu_has_sve(vcpu)) + return -ENOENT; + + return get_vec_vls(ARM64_VEC_SVE, vcpu, reg); +} + +static int set_sve_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +{ + if (!vcpu_has_sve(vcpu)) + return -ENOENT; + + return set_vec_vls(ARM64_VEC_SVE, vcpu, reg); +} + #define SVE_REG_SLICE_SHIFT 0 #define SVE_REG_SLICE_BITS 5 #define SVE_REG_ID_SHIFT (SVE_REG_SLICE_SHIFT + SVE_REG_SLICE_BITS) @@ -533,6 +548,39 @@ static int set_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return 0; } +static int get_sme_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +{ + if (!vcpu_has_sme(vcpu)) + return -ENOENT; + + return get_vec_vls(ARM64_VEC_SME, vcpu, reg); +} + +static int set_sme_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +{ + if (!vcpu_has_sme(vcpu)) + return -ENOENT; + + return set_vec_vls(ARM64_VEC_SME, vcpu, reg); +} + +static int get_sme_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +{ + /* Handle the KVM_REG_ARM64_SME_VLS pseudo-reg as a special case: */ + if (reg->id == KVM_REG_ARM64_SME_VLS) + return get_sme_vls(vcpu, reg); + + return -EINVAL; +} + +static int set_sme_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +{ + /* Handle the KVM_REG_ARM64_SME_VLS pseudo-reg as a special case: */ + if (reg->id == KVM_REG_ARM64_SME_VLS) + return set_sme_vls(vcpu, reg); + + return -EINVAL; +} int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) { return -EINVAL; @@ -775,6 +823,7 @@ int kvm_arm_get_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) case KVM_REG_ARM_FW_FEAT_BMAP: return kvm_arm_get_fw_reg(vcpu, reg); case KVM_REG_ARM64_SVE: return get_sve_reg(vcpu, reg); + case KVM_REG_ARM64_SME: return get_sme_reg(vcpu, reg); } if (is_timer_reg(reg->id)) @@ -795,6 +844,7 @@ int kvm_arm_set_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) case KVM_REG_ARM_FW_FEAT_BMAP: return kvm_arm_set_fw_reg(vcpu, reg); case KVM_REG_ARM64_SVE: return set_sve_reg(vcpu, reg); + case KVM_REG_ARM64_SME: return set_sme_reg(vcpu, reg); } if (is_timer_reg(reg->id)) From patchwork Thu Apr 17 00:25:19 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054714 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4A3022144B9; Thu, 17 Apr 2025 00:26:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849600; cv=none; b=LkHpJn6hrlOd20WzWZOJpkb5BVD/2i4JkI67NQBkxGEGe5Cw7SyBmPgqmLNgw4I2nI8aLM0bAk09C+brfbXwbmbqDtJB5uXigdju0hhICnbwy6un2aJ5W4wxrQvwjD7P4d6bIWPYciupQdXWxASS+h9MGXyL1rYzWtbJgeXSHa8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849600; c=relaxed/simple; bh=EjHK+Xq+IbGu7Ta3fs6RVjmba+ozioK7IwUeQp54oGE=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=rzFXE4bNZY7vM4GwdS5s6ugw3OKKJc+Iv8yaxuraN0j6lYd+ppt4vdiS0almN9wGk/PnLwwLnfwdStgKFAaFf3EVfDEbBsu5RVVTOtKy1rl/lvb6UzdV5TGZRYiYDW6jusPobp9ndhjReeA5MhSJBn7epimAS8Bd/UBlyD9gISg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=HgEMlAY4; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="HgEMlAY4" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E646EC4CEE2; Thu, 17 Apr 2025 00:26:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849600; bh=EjHK+Xq+IbGu7Ta3fs6RVjmba+ozioK7IwUeQp54oGE=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=HgEMlAY4GOXsPX4tHUuCAOI5mHLV9OtuIL5Mlv9jvwQMP9ORK2Z+tRr9aPchwnDOS E/Ra9H0INveIgrIxvGPRzJzdfnFwDomQg3pfQBGTpzD8qRm2x8X1JYSrdlcWOT65ig rAhHvOvpt0dEcK4RqGJRblwGZAGgKSAV1LbT2G+deEipPY31oQLx94aaL0ROgIB3cg 3fq9VuGcA1rg7CaEuXbM+xTS0Tc8R4Jo3nw6xNgJTwP8qQjGFEgYh4ClZkS1yIWl1s ezeXeUQ4P4+FUKBMa4Xi1Y8/pdxtl8TuHEiXgPh2ghItdWL74H7Oig84eV8B8j46v8 ae30LG5HeGvYg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:19 +0100 Subject: [PATCH v5 15/28] KVM: arm64: Support SME control registers Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-15-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=4764; i=broonie@kernel.org; h=from:subject:message-id; bh=EjHK+Xq+IbGu7Ta3fs6RVjmba+ozioK7IwUeQp54oGE=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp6wHNYkZp95SjvgCDIyxYpztZRY032tQC32QXu Ykr1e5eJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKegAKCRAk1otyXVSH0ObzB/ 4rPpDfza+6+QcDVqs+YmqyDKvD90oSlCvOq1GbhMMmyS+tiIDD7dHsojHWB0i85S843Wa9BEArVdG/ 4LDypRTBf45TubAfVUhsYlKkGOfohJp4AZnCrmXFG9VLI3O7Bzcqv1IUjDtLb3G4wUDeoVipbDlsQ/ ZJyMCEx6NRmQj5PwM3I78vy7oLEyUUvTvAmdyLnmlPBTD8SpJg5D6FxgA++LCzbK3Tcvf510y+by8s E4cCFKf0HwzwRYr6MQ+czH8Fjyc1Qd93czrZCMGRk4tYZZMhWTlExj7qqjVhQyhQr65Cmj/G3D4goX d94NfkbWTX8fcWaGlQ44peaRrDcSwX X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME is configured by the system registers SMCR_EL1 and SMCR_EL2, add definitions and userspace access for them. These control the SME vector length in a manner similar to that for SVE and also have feature enable bits for SME2 and FA64. A subsequent patch will add management of them for guests as part of the general floating point context switch, as is done for the equivalent SVE registers. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/include/asm/vncr_mapping.h | 1 + arch/arm64/kvm/sys_regs.c | 37 ++++++++++++++++++++++++++++++++++- 3 files changed, 39 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 8ac63de0a045..b15de8319a65 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -496,6 +496,7 @@ enum vcpu_sysreg { CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ HACR_EL2, /* Hypervisor Auxiliary Control Register */ ZCR_EL2, /* SVE Control Register (EL2) */ + SMCR_EL2, /* SME Control Register (EL2) */ TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ TCR_EL2, /* Translation Control Register (EL2) */ @@ -533,6 +534,7 @@ enum vcpu_sysreg { VNCR(ACTLR_EL1),/* Auxiliary Control Register */ VNCR(CPACR_EL1),/* Coprocessor Access Control */ VNCR(ZCR_EL1), /* SVE Control */ + VNCR(SMCR_EL1), /* SME Control */ VNCR(TTBR0_EL1),/* Translation Table Base Register 0 */ VNCR(TTBR1_EL1),/* Translation Table Base Register 1 */ VNCR(TCR_EL1), /* Translation Control Register */ diff --git a/arch/arm64/include/asm/vncr_mapping.h b/arch/arm64/include/asm/vncr_mapping.h index 4f9bbd4d6c26..74fc7400efbc 100644 --- a/arch/arm64/include/asm/vncr_mapping.h +++ b/arch/arm64/include/asm/vncr_mapping.h @@ -42,6 +42,7 @@ #define VNCR_HDFGWTR_EL2 0x1D8 #define VNCR_ZCR_EL1 0x1E0 #define VNCR_HAFGRTR_EL2 0x1E8 +#define VNCR_SMCR_EL1 0x1F0 #define VNCR_TTBR0_EL1 0x200 #define VNCR_TTBR1_EL1 0x210 #define VNCR_FAR_EL1 0x220 diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 0fc33134ac41..58ec44aac5c9 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -143,6 +143,7 @@ static bool get_el2_to_el1_mapping(unsigned int reg, MAPPED_EL2_SYSREG(ELR_EL2, ELR_EL1, NULL ); MAPPED_EL2_SYSREG(SPSR_EL2, SPSR_EL1, NULL ); MAPPED_EL2_SYSREG(ZCR_EL2, ZCR_EL1, NULL ); + MAPPED_EL2_SYSREG(SMCR_EL2, SMCR_EL1, NULL ); MAPPED_EL2_SYSREG(CONTEXTIDR_EL2, CONTEXTIDR_EL1, NULL ); default: return false; @@ -2521,6 +2522,37 @@ static bool access_gic_elrsr(struct kvm_vcpu *vcpu, return true; } +static unsigned int sme_el2_visibility(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd) +{ + return __el2_visibility(vcpu, rd, sme_visibility); +} + +static bool access_smcr_el2(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + unsigned int vq; + u64 smcr; + + if (guest_hyp_sve_traps_enabled(vcpu)) { + kvm_inject_nested_sve_trap(vcpu); + return true; + } + + if (!p->is_write) { + p->regval = vcpu_read_sys_reg(vcpu, SMCR_EL2); + return true; + } + + smcr = p->regval; + vq = SYS_FIELD_GET(SMCR_ELx, LEN, smcr) + 1; + vq = min(vq, vcpu_sme_max_vq(vcpu)); + vcpu_write_sys_reg(vcpu, SYS_FIELD_PREP(SMCR_ELx, LEN, vq - 1), + SMCR_EL2); + return true; +} + static unsigned int s1poe_visibility(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd) { @@ -2902,7 +2934,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_ZCR_EL1), NULL, reset_val, ZCR_EL1, 0, .visibility = sve_visibility }, { SYS_DESC(SYS_TRFCR_EL1), undef_access }, { SYS_DESC(SYS_SMPRI_EL1), undef_access }, - { SYS_DESC(SYS_SMCR_EL1), undef_access }, + { SYS_DESC(SYS_SMCR_EL1), NULL, reset_val, SMCR_EL1, 0, .visibility = sme_visibility }, { SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 }, { SYS_DESC(SYS_TTBR1_EL1), access_vm_reg, reset_unknown, TTBR1_EL1 }, { SYS_DESC(SYS_TCR_EL1), access_vm_reg, reset_val, TCR_EL1, 0 }, @@ -3256,6 +3288,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG_VNCR(HCRX_EL2, reset_val, 0), + EL2_REG_FILTERED(SMCR_EL2, access_smcr_el2, reset_val, 0, + sme_el2_visibility), + EL2_REG(TTBR0_EL2, access_rw, reset_val, 0), EL2_REG(TTBR1_EL2, access_rw, reset_val, 0), EL2_REG(TCR_EL2, access_rw, reset_val, TCR_EL2_RES1), From patchwork Thu Apr 17 00:25:20 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054715 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EF0692144B9; Thu, 17 Apr 2025 00:26:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849604; cv=none; b=kjnnDkGuKWtVFVRRrFvpQKV+1e08i7HleSeWeLLhzFTqywyWSomnka3gJKsp8Ks4WU3OCkHEIZ9Zqee8vGoupmPBp58cpJxtdllUWNuQrPmub6OPyyp8KILY4N6zODPu5CewjnxZItHmBtWBbPlCS3bwaSdEhW812R8O5H6QcbY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849604; c=relaxed/simple; bh=suQVxCyhMLYX8/wX26fCQf5eYg++OX5MZcwBCHNmvkc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=eAP0M6tcGQiyK+yeZeqzjLqQpnj3eslqs0LqnimuC574QslRudYmwigoeokQOqN6pMFg2rLdOY9l2lspLmLE2GIOhCfuOSfEBv37Nn/zDb7cWe1FGrL36oHKhxw37KTbnASpReRh/AIr6B/CxcFH4l+1kIOceE/fkC745RMqq7Y= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=htMZzg3m; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="htMZzg3m" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 90139C4CEE2; Thu, 17 Apr 2025 00:26:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849603; bh=suQVxCyhMLYX8/wX26fCQf5eYg++OX5MZcwBCHNmvkc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=htMZzg3mt17fnEvR0KBcSxGzmwsXuThdo9yFAiS9gdKhSq5kvm67mF0OiOlEOKb+N cXaXcTD0dyLXJqXnsGLtOIM4xbqj00sWV5/i2szCIjn/e44yo6PrR+1Fm7a3JJ5RP/ KfdXsMf9OozJgvTEZibC3vIKlFZVADxxQzlvFdPqshSeYiANoW1hz8uwL+N+3pnaMa PYoWXPDB8I+afpo3bOAPAQItw276Uv8GRsFBh/2pMRJuBfE2ZHluwYOI6zlkMmduGG fm6FzLtbQoEQHv2oWzMQKj5KD8dfqffWH5EM+R7JAMF6Q/x2vFZqoZtQjqkRVEtbhI ENhT+3rtAicFg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:20 +0100 Subject: [PATCH v5 16/28] KVM: arm64: Support TPIDR2_EL0 Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-16-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=4237; i=broonie@kernel.org; h=from:subject:message-id; bh=suQVxCyhMLYX8/wX26fCQf5eYg++OX5MZcwBCHNmvkc=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp7W0peKP++HkrB3Vo/bYbj162IyYwKJ6Sfgccq vAZI0Z6JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKewAKCRAk1otyXVSH0JBzB/ oD9iycjI6MM3meBb3H18BNPB5QOwa4eVbotXXRZBc2zrhAkhVwxb6w8byeZXy9ykL/NsdwJBXawVSJ e/25VtQLumRwK7CDO5pQJXpNJuVBcnfbTMyqGkkx0cAm5NKTjXgXx8v1yH429bM0h3QewnF1qdZMg2 dVZHcDieq4GNM5+yEMiZDqfmZ5dJkS6xbc2EJAayFs9UT2W6heDxS9CAiga1oZS22c7EIj9blOfaAF ucB7em0D9WHwr+xzZwH8ja54VlxYUssx8wT8Eh3uNZP0GYBumfEDgZqZ5OOFTNMQhYBIkJvBfTEqXW FW4xhKtm+4MIZe7umsD3rfBIIFjdY6 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME adds a new thread ID register, TPIDR2_EL0. This is used in userspace for delayed saving of the ZA state but in terms of the architecture is not really connected to SME other than being part of FEAT_SME. It has an independent fine grained trap and the runtime connection with the rest of SME is purely software defined. Expose the register as a system register if the guest supports SME, context switching it along with the other EL0 TPIDRs. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h | 15 +++++++++++++++ arch/arm64/kvm/sys_regs.c | 9 ++++++--- 3 files changed, 22 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index b15de8319a65..645c7c694cf2 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -440,6 +440,7 @@ enum vcpu_sysreg { CSSELR_EL1, /* Cache Size Selection Register */ TPIDR_EL0, /* Thread ID, User R/W */ TPIDRRO_EL0, /* Thread ID, User R/O */ + TPIDR2_EL0, /* Thread ID, Register 2 */ TPIDR_EL1, /* Thread ID, Privileged */ CNTKCTL_EL1, /* Timer Control Register (EL1) */ PAR_EL1, /* Physical Address Register */ diff --git a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h index b795a7a87a93..7b5ba37b4d19 100644 --- a/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h +++ b/arch/arm64/kvm/hyp/include/hyp/sysreg-sr.h @@ -66,6 +66,17 @@ static inline bool ctxt_has_s1poe(struct kvm_cpu_context *ctxt) return kvm_has_s1poe(kern_hyp_va(vcpu->kvm)); } +static inline bool ctxt_has_sme(struct kvm_cpu_context *ctxt) +{ + struct kvm_vcpu *vcpu; + + if (!system_supports_sme()) + return false; + + vcpu = ctxt_to_vcpu(ctxt); + return kvm_has_sme(kern_hyp_va(vcpu->kvm)); +} + static inline bool ctxt_is_guest(struct kvm_cpu_context *ctxt) { return host_data_ptr(host_ctxt) != ctxt; @@ -105,6 +116,8 @@ static inline void __sysreg_save_user_state(struct kvm_cpu_context *ctxt) { ctxt_sys_reg(ctxt, TPIDR_EL0) = read_sysreg(tpidr_el0); ctxt_sys_reg(ctxt, TPIDRRO_EL0) = read_sysreg(tpidrro_el0); + if (ctxt_has_sme(ctxt)) + ctxt_sys_reg(ctxt, TPIDR2_EL0) = read_sysreg_s(SYS_TPIDR2_EL0); } static inline void __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) @@ -174,6 +187,8 @@ static inline void __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) { write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL0), tpidr_el0); write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0), tpidrro_el0); + if (ctxt_has_sme(ctxt)) + write_sysreg_s(ctxt_sys_reg(ctxt, TPIDR2_EL0), SYS_TPIDR2_EL0); } static inline void __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt, diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 58ec44aac5c9..84a745e392d4 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3110,7 +3110,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { .visibility = s1poe_visibility }, { SYS_DESC(SYS_TPIDR_EL0), NULL, reset_unknown, TPIDR_EL0 }, { SYS_DESC(SYS_TPIDRRO_EL0), NULL, reset_unknown, TPIDRRO_EL0 }, - { SYS_DESC(SYS_TPIDR2_EL0), undef_access }, + { SYS_DESC(SYS_TPIDR2_EL0), NULL, reset_unknown, TPIDR2_EL0, + .visibility = sme_visibility}, { SYS_DESC(SYS_SCXTNUM_EL0), undef_access }, @@ -5186,8 +5187,7 @@ void kvm_calculate_traps(struct kvm_vcpu *vcpu) HFGxTR_EL2_nMAIR2_EL1 | HFGxTR_EL2_nS2POR_EL1 | HFGxTR_EL2_nACCDATA_EL1 | - HFGxTR_EL2_nSMPRI_EL1_MASK | - HFGxTR_EL2_nTPIDR2_EL0_MASK); + HFGxTR_EL2_nSMPRI_EL1_MASK); if (!kvm_has_feat(kvm, ID_AA64ISAR0_EL1, TLB, OS)) kvm->arch.fgu[HFGITR_GROUP] |= (HFGITR_EL2_TLBIRVAALE1OS| @@ -5242,6 +5242,9 @@ void kvm_calculate_traps(struct kvm_vcpu *vcpu) HFGITR_EL2_nBRBIALL); } + if (!kvm_has_feat(kvm, ID_AA64PFR1_EL1, SME, IMP)) + kvm->arch.fgu[HFGxTR_GROUP] |= HFGxTR_EL2_nTPIDR2_EL0; + set_bit(KVM_ARCH_FLAG_FGU_INITIALIZED, &kvm->arch.flags); out: mutex_unlock(&kvm->arch.config_lock); From patchwork Thu Apr 17 00:25:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054716 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 98D2E1A314B; Thu, 17 Apr 2025 00:26:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849607; cv=none; b=EzI9d5ILaMCjotvOWS6oZvVp3H4/pSa+3zVnc2irLhlPsG5rAKwc+ZX/nKyPb6r2H1DXmxtKJrkfSqO/Hv7pdnjhVDZOP4c8JWyjQnzPwef+V80XhAAx93pZKv/KwIwCDtzzwy3lSgIGrttMwj2/C1QTGNwridh0EPJdKrcz6HI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849607; c=relaxed/simple; bh=kajy2/JMlSVJ/et6GDghuNi02jkWS35H2h+5DKrfDHI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=K02EfS1nBMDaa4ZR/jYfTBKWdnd1Ld9RpKsQxHOJQImYgIxDHWLasuoR6prQ0JCKQigwsiTOyv7M/wSI5IUyY7r5POHOTF5TvOJqi/oHMMx4xdEDw111vDFQQ9dUeClzvUWeMqEALPuCyKIe8b/A2DjyGxddUt3I9Lswv0+2qOQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=LdTs82RG; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="LdTs82RG" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3764AC4CEEE; Thu, 17 Apr 2025 00:26:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849607; bh=kajy2/JMlSVJ/et6GDghuNi02jkWS35H2h+5DKrfDHI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=LdTs82RGRwYs0Tc+M7S8iwwJiccyIHzotUdLEBGtd6nDaFoKWxWcxEMDhQZwE0sF2 zPwksC3RLn0j2uUeBRCT74Z67LlX4bPWlmxEBUeqlFhV7XcOxYLprhwLCBHhazWtLi 40Lwrf9YhanP2BJM2/JqtIpSATlqc6BbXrcUtmkuBCH9qj2g25DtSker1llZnPqI5p uEwp15MWcAS3bmCkRx/QK56UfZ/cGfL4l1ttBViPUB9aOoCpokn8eyNriGWNUWnpSC gTlg1AOWynu2XOq0XrwC+ccDRJrs7HzHgM6cJMD7tO/j4BZnSMKdcLBVQd7sIeGzhe ogivutGZPU2tg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:21 +0100 Subject: [PATCH v5 17/28] KVM: arm64: Support SME identification registers for guests Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-17-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=5418; i=broonie@kernel.org; h=from:subject:message-id; bh=kajy2/JMlSVJ/et6GDghuNi02jkWS35H2h+5DKrfDHI=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp8Tk1JXoXy0dd8Noa27wLHoNhQeelxsnr2x6rL cAmmigWJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKfAAKCRAk1otyXVSH0GCaB/ 9DwSFt/FqoTE5/y6Y6yf8iaYKKT+HIwxqWuzYOlhDvYh14vOco+k9CRN5EWPpRnLCGsoB6WGSiy1mK phXMm4o7mwDt870V91WYvRCBRYM+tkdhxT4O3AQ1tlRHcI2iiw4459qNJ6jJ4Ia2Vpnz+fU8BWfU9J dF0NKs8gQWYVLZn+EarthL7JSTIDvrWUeA4lqrRiTCuBfiIVBM/obNUoC7aM/86wvp9HPISngulJ3F WeHfp9ooTlOmKEADgBuInekwkYaaesg6HA3HOaLMDxdZCDxa6FhpRv97B35x/LY0Z4kteK0BAFWrcb OXCT/b93rmIh7WGtG+p1Ea35p76GLQ X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB The primary register for identifying SME is ID_AA64PFR1_EL1.SME. This is hidden from guests unless SME is enabled by the VMM. When it is visible it is writable and can be used to control the availability of SME2. There is also a new register ID_AA64SMFR0_EL1 which we make writable, forcing it to all bits 0 if SME is disabled. This includes the field SMEver giving the SME version, userspace is responsible for ensuring the value is consistent with ID_AA64PFR1_EL1.SME. It also includes FA64, a separately enableable extension which provides the full FPSIMD and SVE instruction set including FFR in streaming mode. Userspace can control the availability of FA64 by writing to this field. The other features enumerated there only add new instructions, there are no architectural controls for these. There is a further identification register SMIDR_EL1 which provides a basic description of the SME microarchitecture, in a manner similar to MIDR_EL1 for the PE. It also describes support for priority management and a basic affinity description for shared SME units, plus some RES0 space. We do not support priority management and affinity is not meaningful for guests so we mask out everything except for the microarchitecture description. As for MIDR_EL1 and REVIDR_EL1 we expose the implementer and revision information to guests with the raw value from the CPU we are running on, this may present issues for asymmetric systems or for migration as it does for the existing registers. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 1 + arch/arm64/kvm/sys_regs.c | 46 +++++++++++++++++++++++++++++++++++---- 2 files changed, 43 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 645c7c694cf2..d86396b1b1e7 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -484,6 +484,7 @@ enum vcpu_sysreg { /* FP/SIMD/SVE */ SVCR, FPMR, + SMIDR_EL1, /* Streaming Mode Identification Register */ /* 32bit specific registers. */ DACR32_EL2, /* Domain Access Control Register */ diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 84a745e392d4..96888554fef5 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -774,6 +774,38 @@ static u64 reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) return mpidr; } +static u64 reset_smidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + u64 smidr = 0; + + if (!system_supports_sme()) + return smidr; + + smidr = read_sysreg_s(SYS_SMIDR_EL1); + + /* + * Mask out everything except for the implementer and revison, + * in particular priority management is not implemented. + */ + smidr &= SMIDR_EL1_IMPLEMENTER_MASK | SMIDR_EL1_REVISION_MASK; + + vcpu_write_sys_reg(vcpu, smidr, SMIDR_EL1); + + return smidr; +} + +static bool access_smidr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + + return true; +} + static unsigned int pmu_visibility(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { @@ -1604,7 +1636,9 @@ static u64 __kvm_read_sanitised_id_reg(const struct kvm_vcpu *vcpu, if (!kvm_has_mte(vcpu->kvm)) val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_MTE); - val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_SME); + if (!vcpu_has_sme(vcpu)) + val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_SME); + val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_RNDR_trap); val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_NMI); val &= ~ARM64_FEATURE_MASK(ID_AA64PFR1_EL1_MTE_frac); @@ -1721,6 +1755,10 @@ static unsigned int id_visibility(const struct kvm_vcpu *vcpu, if (!vcpu_has_sve(vcpu)) return REG_RAZ; break; + case SYS_ID_AA64SMFR0_EL1: + if (!vcpu_has_sme(vcpu)) + return REG_RAZ; + break; } return 0; @@ -2845,7 +2883,6 @@ static const struct sys_reg_desc sys_reg_descs[] = { ID_AA64PFR1_EL1_MTE_frac | ID_AA64PFR1_EL1_NMI | ID_AA64PFR1_EL1_RNDR_trap | - ID_AA64PFR1_EL1_SME | ID_AA64PFR1_EL1_RES0 | ID_AA64PFR1_EL1_MPAM_frac | ID_AA64PFR1_EL1_RAS_frac | @@ -2853,7 +2890,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { ID_WRITABLE(ID_AA64PFR2_EL1, ID_AA64PFR2_EL1_FPMR), ID_UNALLOCATED(4,3), ID_WRITABLE(ID_AA64ZFR0_EL1, ~ID_AA64ZFR0_EL1_RES0), - ID_HIDDEN(ID_AA64SMFR0_EL1), + ID_WRITABLE(ID_AA64SMFR0_EL1, ~ID_AA64SMFR0_EL1_RES0), ID_UNALLOCATED(4,6), ID_WRITABLE(ID_AA64FPFR0_EL1, ~ID_AA64FPFR0_EL1_RES0), @@ -3052,7 +3089,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CLIDR_EL1), access_clidr, reset_clidr, CLIDR_EL1, .set_user = set_clidr, .val = ~CLIDR_EL1_RES0 }, { SYS_DESC(SYS_CCSIDR2_EL1), undef_access }, - { SYS_DESC(SYS_SMIDR_EL1), undef_access }, + { SYS_DESC(SYS_SMIDR_EL1), .access = access_smidr, .reset = reset_smidr, + .reg = SMIDR_EL1, .visibility = sme_visibility }, IMPLEMENTATION_ID(AIDR_EL1, GENMASK_ULL(63, 0)), { SYS_DESC(SYS_CSSELR_EL1), access_csselr, reset_unknown, CSSELR_EL1 }, ID_FILTERED(CTR_EL0, ctr_el0, From patchwork Thu Apr 17 00:25:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054717 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 97A5121B8E1; Thu, 17 Apr 2025 00:26:51 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849611; cv=none; b=Qt6wFUWx938mfPibA8Mm4jPjL0zisBkTCYUgvTV5RQVihVxZZbNp9XRgRGVy6CNUU+8Qc6W6BYu6zK3fmQTPxxYc8EM+PxIQ2jad3mBef6x1SoWuSdF6kw9PTOI7AFVwb+712a5LyvqrM6kpCStpMPiNnJ8UGNanDL3vUBwc17g= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849611; c=relaxed/simple; bh=ZrPgsgXSIivggYmWYF/84mM6YJZOxWoZ6RzbVIJK9/Y=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=FfGalvCk+lgGat4TbCoOJEd7pjcVUXfrSvIxKvQHHWK5tYUNtlX/F7XuELwg0td11vPhEg5FiNhrwS7SWJagQUFnj/MGAt6HtoDFmmoMh5jgfVYNGmCrWilQ5ZBPDz43YLW9ldQ5Iyd1ectswIfpJKDrHL5TigW3ng+n05ACUxs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=oaIpOfTi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="oaIpOfTi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DB72FC4CEE2; Thu, 17 Apr 2025 00:26:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849611; bh=ZrPgsgXSIivggYmWYF/84mM6YJZOxWoZ6RzbVIJK9/Y=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=oaIpOfTifIpb87JvInv5ApYLwwz9fS+exnLzQnp+688OwFXZV2RWnhzM4lrlo9IGk K+E0XItIpgdqsfJYsgMIcUXVoDX56+ypaAXT/ypHQashqN/+GWGCSTvcsB6U4XWrZd /Ghz98Z0W3ClA2DZrovXrSY+fqv6MLIkK6wPtmAU2OAdtXlooGrTt2sHz0tCcTwMbv vmpC8MuUk57SDpa9Mvr4/KLhtRrd5aty6wbdrXZUa8u9TL+dA2bpKtDtJUbwyqBZTo OXetP4Eqs0UltsJN7/FJD0V2TYe/pFZ4PegrqldQfwnYyjcNfXwlxt/Hq8Tg6R8rG3 iU55Lqr5iCmiA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:22 +0100 Subject: [PATCH v5 18/28] KVM: arm64: Support SME priority registers Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-18-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=5358; i=broonie@kernel.org; h=from:subject:message-id; bh=ZrPgsgXSIivggYmWYF/84mM6YJZOxWoZ6RzbVIJK9/Y=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp9ewW5gcC4zICag7nMF6f54qxeJJD/Z8gn6AuP 7o2NuM+JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKfQAKCRAk1otyXVSH0AwXB/ 44tQOzGsFLO8SpHn06+delWlSd84N7zCg/+Xlo5wsB6oP3wgrait5WX4bMRsYrZpME9RzuSJZMx3It dwpKtNmXrTo9Zlm7MQGT+r5PeN/p6ogQXGiH6dXV2c5daHlrIQ5cUFU1cSDXCx6R4gaecR9DHAm4f2 la/ohVTT/mP2uub924VHtsfBwp0uHAahgU8xPbgf8lVQzgI0KqtdOHhPFhFivr8hUqoO2/O2LISa2i Ip5ywZP3+0D8fH6DMioPEkOaXmWsK6J0U5dKcneemvS9+dhApkVdRVMVlVJ78ybutbPCqR75lP8viK vJ1uP1H9tXU6cX3iq/BMDfh/FWmsbc X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME has optional support for configuring the relative priorities of PEs in systems where they share a single SME hardware block, known as a SMCU. Currently we do not have any support for this in Linux and will also hide it from KVM guests, pending experience with practical implementations. The interface for configuring priority support is via two new system registers, these registers are always defined when SME is available. The register SMPRI_EL1 allows control of SME execution priorities. Since we disable SME priority support for guests this register is RES0, define it as such and enable fine grained traps for SMPRI_EL1 to ensure that guests can't write to it even if the hardware supports priorites. We could with some adjustment to the generic FGT code allow untrapped reads but since we don't currently advertise priority support to guests there should be no reason for frequent accesses. There is also an EL2 register SMPRIMAP_EL2 for virtualisation of priorities, this is RES0 when priority configuration is not supported but has no specific traps available. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 2 ++ arch/arm64/include/asm/vncr_mapping.h | 1 + arch/arm64/kvm/sys_regs.c | 30 +++++++++++++++++++++++++----- 3 files changed, 28 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index d86396b1b1e7..09351c14032f 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -485,6 +485,7 @@ enum vcpu_sysreg { SVCR, FPMR, SMIDR_EL1, /* Streaming Mode Identification Register */ + SMPRI_EL1, /* Streaming Mode Priority Register */ /* 32bit specific registers. */ DACR32_EL2, /* Domain Access Control Register */ @@ -537,6 +538,7 @@ enum vcpu_sysreg { VNCR(CPACR_EL1),/* Coprocessor Access Control */ VNCR(ZCR_EL1), /* SVE Control */ VNCR(SMCR_EL1), /* SME Control */ + VNCR(SMPRIMAP_EL2), /* Streaming Mode Priority Mapping Register */ VNCR(TTBR0_EL1),/* Translation Table Base Register 0 */ VNCR(TTBR1_EL1),/* Translation Table Base Register 1 */ VNCR(TCR_EL1), /* Translation Control Register */ diff --git a/arch/arm64/include/asm/vncr_mapping.h b/arch/arm64/include/asm/vncr_mapping.h index 74fc7400efbc..1685df741294 100644 --- a/arch/arm64/include/asm/vncr_mapping.h +++ b/arch/arm64/include/asm/vncr_mapping.h @@ -43,6 +43,7 @@ #define VNCR_ZCR_EL1 0x1E0 #define VNCR_HAFGRTR_EL2 0x1E8 #define VNCR_SMCR_EL1 0x1F0 +#define VNCR_SMPRIMAP_EL2 0x1F0 #define VNCR_TTBR0_EL1 0x200 #define VNCR_TTBR1_EL1 0x210 #define VNCR_FAR_EL1 0x220 diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 96888554fef5..7cfc0c2801ed 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1826,6 +1826,15 @@ static unsigned int fp8_visibility(const struct kvm_vcpu *vcpu, return REG_HIDDEN; } +static unsigned int sme_raz_visibility(const struct kvm_vcpu *vcpu, + const struct sys_reg_desc *rd) +{ + if (vcpu_has_sme(vcpu)) + return REG_RAZ; + + return REG_HIDDEN; +} + static u64 sanitise_id_aa64pfr0_el1(const struct kvm_vcpu *vcpu, u64 val) { if (!vcpu_has_sve(vcpu)) @@ -2970,7 +2979,14 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_ZCR_EL1), NULL, reset_val, ZCR_EL1, 0, .visibility = sve_visibility }, { SYS_DESC(SYS_TRFCR_EL1), undef_access }, - { SYS_DESC(SYS_SMPRI_EL1), undef_access }, + + /* + * SMPRI_EL1 is UNDEF when SME is disabled, the UNDEF is + * handled via FGU which is handled without consulting this + * table. + */ + { SYS_DESC(SYS_SMPRI_EL1), trap_raz_wi, .visibility = sme_raz_visibility }, + { SYS_DESC(SYS_SMCR_EL1), NULL, reset_val, SMCR_EL1, 0, .visibility = sme_visibility }, { SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 }, { SYS_DESC(SYS_TTBR1_EL1), access_vm_reg, reset_unknown, TTBR1_EL1 }, @@ -3327,6 +3343,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { EL2_REG_VNCR(HCRX_EL2, reset_val, 0), + EL2_REG_FILTERED(SMPRIMAP_EL2, trap_raz_wi, reset_val, 0, + sme_el2_visibility), EL2_REG_FILTERED(SMCR_EL2, access_smcr_el2, reset_val, 0, sme_el2_visibility), @@ -5224,8 +5242,7 @@ void kvm_calculate_traps(struct kvm_vcpu *vcpu) kvm->arch.fgu[HFGxTR_GROUP] = (HFGxTR_EL2_nAMAIR2_EL1 | HFGxTR_EL2_nMAIR2_EL1 | HFGxTR_EL2_nS2POR_EL1 | - HFGxTR_EL2_nACCDATA_EL1 | - HFGxTR_EL2_nSMPRI_EL1_MASK); + HFGxTR_EL2_nACCDATA_EL1); if (!kvm_has_feat(kvm, ID_AA64ISAR0_EL1, TLB, OS)) kvm->arch.fgu[HFGITR_GROUP] |= (HFGITR_EL2_TLBIRVAALE1OS| @@ -5280,8 +5297,11 @@ void kvm_calculate_traps(struct kvm_vcpu *vcpu) HFGITR_EL2_nBRBIALL); } - if (!kvm_has_feat(kvm, ID_AA64PFR1_EL1, SME, IMP)) - kvm->arch.fgu[HFGxTR_GROUP] |= HFGxTR_EL2_nTPIDR2_EL0; + if (kvm_has_feat(kvm, ID_AA64PFR1_EL1, SME, IMP)) + kvm->arch.fgt[HFGxTR_GROUP] |= HFGxTR_EL2_nSMPRI_EL1_MASK; + else + kvm->arch.fgu[HFGxTR_GROUP] |= (HFGxTR_EL2_nTPIDR2_EL0 | + HFGxTR_EL2_nSMPRI_EL1_MASK); set_bit(KVM_ARCH_FLAG_FGU_INITIALIZED, &kvm->arch.flags); out: From patchwork Thu Apr 17 00:25:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054718 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6F29421B8E1; Thu, 17 Apr 2025 00:26:54 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849615; cv=none; b=TtId0Ytin0P40NZCPeNt4AeQ1WTYU6HVXbMGYg8uWc4pTDcHcs97gkXBf5q8j/ib+lHqLh9262NN7zW94RaozLDcQ8NAPpTTFS4OVXVbOZBe5uIxdjoie8+6w7i4tPpRDV7Yj3iZJJb3Irm5vd0CPdfbJ4qHPDPsXV3Q7vb+pYs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849615; c=relaxed/simple; bh=eko0M/1IoNzmkVmBa1f5aR4hGKuIltnrGwhCOTIwAnI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=WABB6cl/IVOAX3vA7LgJ4paRgNItbfB21JH+u9XzzrzF/FMdd3I4NS1xiRZJwb4v0A9WSd0dAlCHEBGi6jg/1Dosj4EgE6XLG1iB/Q9kjhwo5/t4+tTzdM/7mgZl1qIu8cmh114Gu91jKcI5dC4kNklqT6gD9ZBikKJF+tHlIhk= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=BX0Mw45v; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="BX0Mw45v" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 82884C4CEEA; Thu, 17 Apr 2025 00:26:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849614; bh=eko0M/1IoNzmkVmBa1f5aR4hGKuIltnrGwhCOTIwAnI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=BX0Mw45vSTx342fI8cvZFKX1P7nSBn+kZ8V5lb526uVn1eu9Th/J3GvbKm3+Jkt85 pX9ts+Ej/303cnQG7OUm+LqJCvDgc+au9ATQQkK2K2IjCwgiZcPELtktCyZBGqsrsz Hgqf+fWgPoP7gKOE0jCMI3hTBi8UBxWOW/rATZMx3EkmLxyl9WMDROSd1LWOJhXFMs BZpDZXmpxb9Illk0EM0hdgEVppuX8Pfm1BiAWGDeHo6lYoWPEms6dve79EJ8uJX2rC d/jUa3Xt6cwsqP4x1RT/7kFRZpfb/+4fvXjsK4IURp++D807XUwi59UkPI150at/er DETHI6Z09zFpg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:23 +0100 Subject: [PATCH v5 19/28] KVM: arm64: Provide assembly for SME register access Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-19-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=1851; i=broonie@kernel.org; h=from:subject:message-id; bh=eko0M/1IoNzmkVmBa1f5aR4hGKuIltnrGwhCOTIwAnI=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp9Drn0eyy5meekv9Nerff2giIQZ4JCURh9L+K+ 3E1uyEOJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKfQAKCRAk1otyXVSH0K9TB/ 9WaiHu5mQZtI6uD/X6tcGxe7I+BWnfa+auZ3VgiXZdzBAssAtuZ7ADLJl/0ddCvf1iPW5C6uP0n7Sk T7Ki6zBA1VQQOxlkANbuyRM3mu9T9NkHQqA6xrZJTyjN3pzBd9gze/ysepVx2dkuO9jFOU06gID4MV prq3xFABE4wAFYqYzEAURgZGKzVkgFVlT1IyrlHF6gB8im8BMEneZtq1TWM8OnaVDFmfVJiG2KJFry OOHQablltXo8j1I0BUCM2B03tblLOTZHoEk5ra42+WwTorWdfKtdqMGp7iqqKWc5EL2RFqEDUklfiT Fdx5Ydse/LX/kJbXZOX+JcTdG26lMb X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Provide versions of the SME state save and restore functions for the hypervisor to allow it to restore ZA and ZT for guests. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_hyp.h | 3 +++ arch/arm64/kvm/hyp/fpsimd.S | 26 ++++++++++++++++++++++++++ 2 files changed, 29 insertions(+) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 0ad5a66e0d25..1c31d8b26aa9 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -115,6 +115,9 @@ void __fpsimd_save_state(struct user_fpsimd_state *fp_regs); void __fpsimd_restore_state(struct user_fpsimd_state *fp_regs); void __sve_save_state(void *sve_pffr, u32 *fpsr, int save_ffr); void __sve_restore_state(void *sve_pffr, u32 *fpsr, int restore_ffr); +int __sve_get_vl(void); +void __sme_save_state(void const *state, bool restore_zt); +void __sme_restore_state(void const *state, bool restore_zt); u64 __guest_enter(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/hyp/fpsimd.S b/arch/arm64/kvm/hyp/fpsimd.S index 6e16cbfc5df2..44a1b0a483da 100644 --- a/arch/arm64/kvm/hyp/fpsimd.S +++ b/arch/arm64/kvm/hyp/fpsimd.S @@ -29,3 +29,29 @@ SYM_FUNC_START(__sve_save_state) sve_save 0, x1, x2, 3 ret SYM_FUNC_END(__sve_save_state) + +SYM_FUNC_START(__sve_get_vl) + _sve_rdvl 0, 1 + ret +SYM_FUNC_END(__sve_get_vl) + +SYM_FUNC_START(__sme_save_state) + _sme_rdsvl 2, 1 // x2 = VL/8 + sme_save_za 0, x2, 12 // Leaves x0 pointing to the end of ZA + + cbz x1, 1f + _str_zt 0 +1: + ret +SYM_FUNC_END(__sme_save_state) + +SYM_FUNC_START(__sme_restore_state) + _sme_rdsvl 2, 1 // x2 = VL/8 + sme_load_za 0, x2, 12 // Leaves x0 pointing to end of ZA + + cbz x1, 1f + _ldr_zt 0 + +1: + ret +SYM_FUNC_END(__sme_restore_state) From patchwork Thu Apr 17 00:25:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054719 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DB3791AAA10; Thu, 17 Apr 2025 00:26:58 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849619; cv=none; b=DC2ej84woNJtyAZplQgvPl6mMAnBmOfMUYCDWPz9q2u4YvpVvKpN6Kd4DQNRX7RK9TMw/0O1iMa9Wb4RakeAcKSIMWd0ETVc6nnHxbb6OcFfi+AcRYacx4xJb8fN4mNIiLW4GUT3xYZQ/fAFvnIKKXAIJAmEjARCNSrlO6Bs98A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849619; c=relaxed/simple; bh=4WGcB0ofjwJdg4f88xsT0zg0K56yny7JB2qYbeq6hCY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Tzjx+ddW36tHsYm9ANGwTkqs8PMVWlO9AxozyND6xqOTt2recyY/+H3jHtImk2SvYvM6oi4Q2ygW7K1qnD9+34ZotE954dNxxB0yvHHz+B9V6XmWzAOLdcdz0GMmsFksownpBmVPz1IWvE/EkTWa+Ef7bAIc+8CPphRTIcP2nd4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RtKJ3Rbt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RtKJ3Rbt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 28995C4CEE2; Thu, 17 Apr 2025 00:26:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849618; bh=4WGcB0ofjwJdg4f88xsT0zg0K56yny7JB2qYbeq6hCY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=RtKJ3RbttaaEa3TR0ee8LeP3Ccfm2FdTCpkaaSzSaeXmKBW8NbwALjvrKdiKOvuRc D0KIpOsARODsaFvNvA1wJLFP1R5moTxNLjUHGbSJmsJqftEW9iNDXANHGdMEDHb3an ZAsVMzGpJChHPA7ENs+B7m1PU0iP+y225pqSdHuKZMiA6NnqTSBk63VL+BroD4XKNv xd4Xo/R2x/319QyJFpFr0JfQOv+yIvIyEwXMQeOIY3Y/z8RZkFvO0WFW64qDCe2Zte Zib/EIf+fj6nSFbPRaIHX50I1XjXBBmvTtpW0LGxlGsiynhdMrCX0qiZVZ+SwiCgXK wlje5nisKUuPA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:24 +0100 Subject: [PATCH v5 20/28] KVM: arm64: Support userspace access to streaming mode Z and P registers Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-20-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=4047; i=broonie@kernel.org; h=from:subject:message-id; bh=4WGcB0ofjwJdg4f88xsT0zg0K56yny7JB2qYbeq6hCY=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp+w7pAYMlYRr1A+p04D92h1n1M1EGga4AOSy+z zh6fxmWJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKfgAKCRAk1otyXVSH0Py8B/ 47uNcssQNqa9VunE0G1r8LByZhK45A+biOyw7d0RQJYUSZrfczbN3t6nPaTBsFIB9e0pg14etDdWOk bt7uoDToYm+da3YJff/qzdSo1gBLQ1SUcoQ+aT+pgsRto1d85WhLndHDcG+bIQMy6hsu2KxTTm13HV 0wRjkFjpX+KO2jcboNllWMV4JN1a5jW7GCwSBbBs5ZI0G2mNNXlpzV1E3DUD/XyFOeTQjUU6j2WyPB LAjrhkUrZnq9ypGRHkGeWA0A2F0dFlCDQo0e+lWBWUx+8ZA84uQjU7BubPapAvN3YpUJhpuIq7ZLKW LCVu2cPhs2wWZ2NsXmK9zjcFzXI/Rz X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME introduces a mode called streaming mode where the Z, P and optionally FFR registers can be accessed using the SVE instructions but with the SME vector length. Reflect this in the ABI for accessing the guest registers by making the vector length for the vcpu reflect the vector length that would be seen by the guest were it running, using the SME vector length when the guest is configured for streaming mode. Since SME may be present without SVE we also update the existing checks for access to the Z, P and V registers to check for either SVE or streaming mode. When not in streaming mode the guest floating point state may be accessed via the V registers. Any VMM that supports SME must be aware of the need to configure streaming mode prior to writing the floating point registers that this creates. Signed-off-by: Mark Brown --- arch/arm64/kvm/guest.c | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 6460bb21e01d..4ba0afa369d5 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -73,6 +73,11 @@ static u64 core_reg_offset_from_id(u64 id) return id & ~(KVM_REG_ARCH_MASK | KVM_REG_SIZE_MASK | KVM_REG_ARM_CORE); } +static bool vcpu_has_sve_regs(const struct kvm_vcpu *vcpu) +{ + return vcpu_has_sve(vcpu) || vcpu_in_streaming_mode(vcpu); +} + static int core_reg_size_from_offset(const struct kvm_vcpu *vcpu, u64 off) { int size; @@ -110,9 +115,10 @@ static int core_reg_size_from_offset(const struct kvm_vcpu *vcpu, u64 off) /* * The KVM_REG_ARM64_SVE regs must be used instead of * KVM_REG_ARM_CORE for accessing the FPSIMD V-registers on - * SVE-enabled vcpus: + * SVE-enabled vcpus or when a SME enabled vcpu is in + * streaming mode: */ - if (vcpu_has_sve(vcpu) && core_reg_offset_is_vreg(off)) + if (vcpu_has_sve_regs(vcpu) && core_reg_offset_is_vreg(off)) return -EINVAL; return size; @@ -426,6 +432,24 @@ struct vec_state_reg_region { unsigned int upad; /* extra trailing padding in user memory */ }; +/* + * We represent the Z and P registers to userspace using either the + * SVE or SME vector length, depending on which features the guest has + * and if the guest is in streaming mode. + */ +static unsigned int vcpu_sve_cur_vq(struct kvm_vcpu *vcpu) +{ + unsigned int vq = 0; + + if (vcpu_has_sve(vcpu)) + vq = vcpu_sve_max_vq(vcpu); + + if (vcpu_in_streaming_mode(vcpu)) + vq = vcpu_sme_max_vq(vcpu); + + return vq; +} + /* * Validate SVE register ID and get sanitised bounds for user/kernel SVE * register copy @@ -466,7 +490,7 @@ static int sve_reg_to_region(struct vec_state_reg_region *region, if (!vcpu_has_sve(vcpu) || (reg->id & SVE_REG_SLICE_MASK) > 0) return -ENOENT; - vq = vcpu_sve_max_vq(vcpu); + vq = vcpu_sve_cur_vq(vcpu); reqoffset = SVE_SIG_ZREG_OFFSET(vq, reg_num) - SVE_SIG_REGS_OFFSET; @@ -476,7 +500,7 @@ static int sve_reg_to_region(struct vec_state_reg_region *region, if (!vcpu_has_sve(vcpu) || (reg->id & SVE_REG_SLICE_MASK) > 0) return -ENOENT; - vq = vcpu_sve_max_vq(vcpu); + vq = vcpu_sve_cur_vq(vcpu); reqoffset = SVE_SIG_PREG_OFFSET(vq, reg_num) - SVE_SIG_REGS_OFFSET; @@ -515,6 +539,9 @@ static int get_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (!kvm_arm_vcpu_vec_finalized(vcpu)) return -EPERM; + if (!vcpu_has_sve_regs(vcpu)) + return -EBUSY; + if (copy_to_user(uptr, vcpu->arch.sve_state + region.koffset, region.klen) || clear_user(uptr + region.klen, region.upad)) @@ -541,6 +568,9 @@ static int set_sve_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (!kvm_arm_vcpu_vec_finalized(vcpu)) return -EPERM; + if (!vcpu_has_sve_regs(vcpu)) + return -EBUSY; + if (copy_from_user(vcpu->arch.sve_state + region.koffset, uptr, region.klen)) return -EFAULT; From patchwork Thu Apr 17 00:25:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054720 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 24758226D17; Thu, 17 Apr 2025 00:27:02 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849622; cv=none; b=OFhW6ba6IJaKfWMVS7uKI5biAF3FR+aL6Or/OVW5A7vOd4lJs0qFo6xH5Br0kCWG2/F23tPevi4SWaMRsFbPfVxgIeZhJo0dXAeq7wDrHDbhXAKzfrHVuIkRp4+HsrMY5xjX1w9iEOjOJW7i3VUjAdkvhyKA/o0Yr/Wc85fPP6s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849622; c=relaxed/simple; bh=Cvq6eABdDqP2hcq/eRmzqfXa00dzXj2dZhiT5XmdxBo=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=gRTa2UJ1JohKoq90/BvZo3KVgnXxiwRseuUmmjUuxndRIKB8hauLEY2LJUkSaErj4ZhsT+83IlnLZdeGvUquhu0SPPSHKUt94ZDkXg/eUcXR0DlLUro6BqEA16XLT8kn3ud8gTYXt9pihkRtJTi7bboVuWTcDdofwjzKDn029Ts= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SW6XkpCW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SW6XkpCW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C4633C4CEED; Thu, 17 Apr 2025 00:26:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849622; bh=Cvq6eABdDqP2hcq/eRmzqfXa00dzXj2dZhiT5XmdxBo=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=SW6XkpCWvc+XEbG8iE7WT1gmZ58DpBgpmmyUmeLTjzpYOU6jOtzsZwcBgIP9Tsr0L KQ1eQfa+NG7K4llLcQrD+NdmaOSr53ZdcukkJOo9o7y0643/eyjBTvnmyFCwztkZ/f dsYXbZiXn9F/vf9+7BbLI8c1BZ8M4liTneCk+jQKsIaZExABvvkzyO1SKu2N3YartF a1t+sWemSpJmd/qpq2QeB99iApUnZPpU9PsMyqpWMmsSw2EiqfwHF1r0OfR43vtGUq hLNDPhSX/cK8eNFhPnn6O3xWlX8q0+UshiO+GBHsBcqMpJIboy4sVrXFD6dXpMGR9r yb0AtUJ2xeIZQ== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:25 +0100 Subject: [PATCH v5 21/28] KVM: arm64: Flush register state on writes to SVCR.SM and SVCR.ZA Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-21-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3370; i=broonie@kernel.org; h=from:subject:message-id; bh=Cvq6eABdDqP2hcq/eRmzqfXa00dzXj2dZhiT5XmdxBo=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEp/rjAQoyRjhEAHaxvC9+m6jLrGDd2SZVIFUxUe 1scs86OJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKfwAKCRAk1otyXVSH0PLbB/ 908/iRrkOt7rAypIV4yIV1NhagBAAwKxHrrJe2L5XGQfUDa1RILLmW9AibuqrlL2LFsi9LuG7UnMRZ vUISSTyOYGTOwPf51nN1DW9lKUXiaiMC1OKTlzZYRPKye+J4ulrLGSyx+k0IiDoPBaKTJ07QqOuKsE GIlIxFfEUGMyNzDD8qKlWyoWRsCLao/FjS+vx8jFYfGXK4Urkr1pO8DadS2eBH297o633Ye7Z0Fr8y /boGga7O1O9yjcTpTfM5uh3IOdIEtpTK7KUzyUr+ESoL4Mh/fb+CWIXpkpaYW2EIL976+BvcWAj93n dJp77Fr/n0RHITnkcjawRL7uLLOETN X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Writes to the physical SVCR.SM and SVCR.ZA change the state of PSTATE.SM and PSTATE.ZA, causing other floating point state to reset. Emulate this behaviour for writes done via the KVM userspace ABI. Setting PSTATE.ZA to 1 causes ZA and ZT0 to be reset to 0, these are stored in sme_state. Setting PSTATE.ZA to 0 causes ZA and ZT0 to become inaccesible so no reset is needed. Any change in PSTATE.SM causes the V, Z, P, FFR and FPMR registers to be reset to 0 and FPSR to be reset to 0x800009f. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 18 ++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 29 ++++++++++++++++++++++++++++- 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 09351c14032f..ef8b60222ac9 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1066,6 +1066,24 @@ struct kvm_vcpu_arch { __size_ret; \ }) +#define vcpu_sme_state(vcpu) (kern_hyp_va((vcpu)->arch.sme_state)) + +#define vcpu_sme_state_size(vcpu) ({ \ + size_t __size_ret; \ + unsigned int __vcpu_vq; \ + \ + if (WARN_ON(!sve_vl_valid((vcpu)->arch.max_vl[ARM64_VEC_SME]))) { \ + __size_ret = 0; \ + } else { \ + __vcpu_vq = vcpu_sme_max_vq(vcpu); \ + __size_ret = ZA_SIG_REGS_SIZE(__vcpu_vq); \ + if (vcpu_has_sme2(vcpu)) \ + __size_ret += ZT_SIG_REG_SIZE; \ + } \ + \ + __size_ret; \ +}) + /* * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the * memory backed version of a register, and not the one most recently diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 7cfc0c2801ed..0f08f791e3fe 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -806,6 +806,33 @@ static bool access_smidr(struct kvm_vcpu *vcpu, return true; } +static int set_svcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, + u64 val) +{ + u64 old = __vcpu_sys_reg(vcpu, rd->reg); + + if (val & SVCR_RES0) + return -EINVAL; + + if ((val & SVCR_ZA) && !(old & SVCR_ZA) && vcpu->arch.sme_state) + memset(vcpu->arch.sme_state, 0, vcpu_sme_state_size(vcpu)); + + if ((val & SVCR_SM) != (old & SVCR_SM)) { + memset(vcpu->arch.ctxt.fp_regs.vregs, 0, + sizeof(vcpu->arch.ctxt.fp_regs.vregs)); + + if (vcpu->arch.sve_state) + memset(vcpu->arch.sve_state, 0, + vcpu_sve_state_size(vcpu)); + + __vcpu_sys_reg(vcpu, FPMR) = 0; + vcpu->arch.ctxt.fp_regs.fpsr = 0x800009f; + } + + __vcpu_sys_reg(vcpu, rd->reg) = val; + return 0; +} + static unsigned int pmu_visibility(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) { @@ -3115,7 +3142,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { CTR_EL0_DminLine_MASK | CTR_EL0_L1Ip_MASK | CTR_EL0_IminLine_MASK), - { SYS_DESC(SYS_SVCR), undef_access, reset_val, SVCR, 0, .visibility = sme_visibility }, + { SYS_DESC(SYS_SVCR), undef_access, reset_val, SVCR, 0, .visibility = sme_visibility, .set_user = set_svcr }, { SYS_DESC(SYS_FPMR), undef_access, reset_val, FPMR, 0, .visibility = fp8_visibility }, { PMU_SYS_REG(PMCR_EL0), .access = access_pmcr, .reset = reset_pmcr, From patchwork Thu Apr 17 00:25:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054721 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 33E951ADFFE; Thu, 17 Apr 2025 00:27:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849626; cv=none; b=HvDgmzFTvGRyokyKlA+qTCnz5/wBj5+RN29Ip7rFfV5xIVNFoU/hZbVD4no09c/o/vwqxVytwIbbSS1+dVNx1bbEtRsGUBmpEtpVqwU8gC/K5hPDbYwF9gkkJ+6gQ+MwgMdnojs7/GHc0csUkWTQAU0gfW7JUYhASrKBAz6rh4c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849626; c=relaxed/simple; bh=nFaCx//F7H/LvnEphuyPhU8UFj/so9Qf29YNe93/sWQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=AlmoHlIlJWXNkBeHvHVdMqPvKuW8hwWa0DCRBp02j1UdLU4I82vXqgbTt1tOThwsDJUq0MAMbB0fRN94JXoSqduEjUQE/Pbb5OdyQb7ww+MksbI2mIbIisYVove0gazUNC73gYxk5JKrPEQgzDqMkqQB4OVjrYwTr14naX9B3ec= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=RJlXr4Kh; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="RJlXr4Kh" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6A9ADC4CEE2; Thu, 17 Apr 2025 00:27:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849626; bh=nFaCx//F7H/LvnEphuyPhU8UFj/so9Qf29YNe93/sWQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=RJlXr4Khiboh+fiAYNqZCRuXD/y9FGmhIYm5DWrVWvWVqjygql3XDRAo5+iI0/TeM VX12E07JppVfduQ3lhaBTV5Vh2SEkwt0p+ih6MiKOdIvAHRGiTdv/FcmGzmUIZBmgf VMAevwWtnYLrJTXd3Xl90XfDZhRGoazjrQ/BrPbxINuROkQmvwj02PEG1uGgWBDX2j NQ/yGta7YYw/tCjYTaolYWjmXDRUA2V7iamywATLdd8P2zMGzfHG6BvVOjL4CktY7l FlLea1qVtpn0WlLS4xD3k0Zta7D0j2qv+PK5Ad7qcOQMG7WYVOEcMC4sby3dIfx1B8 bQKERHGAE/tkA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:26 +0100 Subject: [PATCH v5 22/28] KVM: arm64: Expose SME specific state to userspace Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-22-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=6845; i=broonie@kernel.org; h=from:subject:message-id; bh=nFaCx//F7H/LvnEphuyPhU8UFj/so9Qf29YNe93/sWQ=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEqA9rdk2gHwsdmjE8ZYQh2D1iWyiLDTQEbm/+DJ dhUnMauJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKgAAKCRAk1otyXVSH0FlGB/ 47tSz1P5azhKmNsskckhwGWpsyDBWyr5YrTuTQem1nqfN8gInK9OldcvtWQeMO028sFEAImMOBFiWf 8KjDe+BkqG0tWmVqGt6803nErLBHx+7pJukMgPtoLpdofiGOCReBx7CQeyjIiAUQED/BWFLMHTe6KL maX+DaFJxs6+oAFTFfJv/6W2O1TiP3bLY6uW2Dha8M9KcVTNxudk4zAYWc2bO/3yNycYjNRy/v/nj8 5oZT0hMyLYNXuQG5Ve0uDapq2jHo+QNpXdUvJSYd+eWMA0KEmYdiekV6zh5UcsbLsSCfvKSgYEdJBQ nRI1suOCrT5DW2DXDKl+3x6/etTRd8 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME introduces two new registers, the ZA matrix register and the ZT0 LUT register. Both of these registers are only accessible when PSTATE.ZA is set and ZT0 is only present if SME2 is enabled for the guest. Provide support for configuring these from VMMs. The ZA matrix is a single SVL*SVL register which is available when PSTATE.ZA is set. We follow the pattern established by the architecture itself and expose this to userspace as a series of horizontal SVE vectors with the streaming mode vector length, using the format already established for the SVE vectors themselves. ZT0 is a single register with a refreshingly fixed size 512 bit register which is like ZA accessible only when PSTATE.ZA is set. Add support for it to the userspace API, as with ZA we allow the register to be read or written regardless of the state of PSTATE.ZA in order to simplify userspace usage. The value will be reset to 0 whenever PSTATE.ZA changes from 0 to 1, userspace can read stale values but these are not observable by the guest without manipulation of PSTATE.ZA by userspace. While there is currently only one ZT register the naming as ZT0 and the instruction encoding clearly leave room for future extensions adding more ZT registers. This encoding can readily support such an extension if one is introduced. Signed-off-by: Mark Brown --- arch/arm64/include/uapi/asm/kvm.h | 17 ++++++ arch/arm64/kvm/guest.c | 114 +++++++++++++++++++++++++++++++++++++- 2 files changed, 129 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index c0a8aa8c67ea..2ecdd29c0677 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -357,6 +357,23 @@ struct kvm_arm_counter_offset { /* SME registers */ #define KVM_REG_ARM64_SME (0x17 << KVM_REG_ARM_COPROC_SHIFT) +#define KVM_ARM64_SME_VQ_MIN __SVE_VQ_MIN +#define KVM_ARM64_SME_VQ_MAX __SVE_VQ_MAX + +/* ZA and ZTn occupy blocks at the following offsets within this range: */ +#define KVM_REG_ARM64_SME_ZA_BASE 0 +#define KVM_REG_ARM64_SME_ZT_BASE 0x600 + +#define KVM_ARM64_SME_MAX_ZAHREG (__SVE_VQ_BYTES * KVM_ARM64_SME_VQ_MAX) + +#define KVM_REG_ARM64_SME_ZAHREG(n, i) \ + (KVM_REG_ARM64 | KVM_REG_ARM64_SME | KVM_REG_ARM64_SME_ZA_BASE | \ + KVM_REG_SIZE_U2048 | \ + (((n) & (KVM_ARM64_SME_MAX_ZAHREG - 1)) << 5) | \ + ((i) & (KVM_ARM64_SVE_MAX_SLICES - 1))) + +#define KVM_REG_ARM64_SME_ZTREG_SIZE (512 / 8) + /* Vector lengths pseudo-register: */ #define KVM_REG_ARM64_SME_VLS (KVM_REG_ARM64 | KVM_REG_ARM64_SME | \ KVM_REG_SIZE_U512 | 0xffff) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 4ba0afa369d5..6ed5f4fe6126 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -594,23 +594,133 @@ static int set_sme_vls(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return set_vec_vls(ARM64_VEC_SME, vcpu, reg); } +/* + * Validate SVE register ID and get sanitised bounds for user/kernel SVE + * register copy + */ +static int sme_reg_to_region(struct vec_state_reg_region *region, + struct kvm_vcpu *vcpu, + const struct kvm_one_reg *reg) +{ + /* reg ID ranges for ZA.H[n] registers */ + unsigned int vq = vcpu_sme_max_vq(vcpu) - 1; + const u64 za_h_max = vq * __SVE_VQ_BYTES; + const u64 zah_id_min = KVM_REG_ARM64_SME_ZAHREG(0, 0); + const u64 zah_id_max = KVM_REG_ARM64_SME_ZAHREG(za_h_max - 1, + SVE_NUM_SLICES - 1); + unsigned int reg_num; + + unsigned int reqoffset, reqlen; /* User-requested offset and length */ + unsigned int maxlen; /* Maximum permitted length */ + + size_t sme_state_size; + + reg_num = (reg->id & SVE_REG_ID_MASK) >> SVE_REG_ID_SHIFT; + + if (reg->id >= zah_id_min && reg->id <= zah_id_max) { + if (!vcpu_has_sme(vcpu) || (reg->id & SVE_REG_SLICE_MASK) > 0) + return -ENOENT; + + /* ZA is exposed as SVE vectors ZA.H[n] */ + reqoffset = ZA_SIG_ZAV_OFFSET(vq, reg_num) - + ZA_SIG_REGS_OFFSET; + reqlen = KVM_SVE_ZREG_SIZE; + maxlen = SVE_SIG_ZREG_SIZE(vq); + } else if (reg->id == KVM_REG_ARM64_SME_ZT_BASE) { + /* ZA is exposed as SVE vectors ZA.H[n] */ + if (!kvm_has_feat(vcpu->kvm, ID_AA64PFR1_EL1, SME, SME2) || + (reg->id & SVE_REG_SLICE_MASK) > 0 || + reg_num > 0) + return -ENOENT; + + /* ZT0 is stored after ZA */ + reqlen = KVM_REG_ARM64_SME_ZTREG_SIZE; + maxlen = KVM_REG_ARM64_SME_ZTREG_SIZE; + } else { + return -EINVAL; + } + + sme_state_size = vcpu_sme_state_size(vcpu); + if (WARN_ON(!sme_state_size)) + return -EINVAL; + + region->koffset = array_index_nospec(reqoffset, sme_state_size); + region->klen = min(maxlen, reqlen); + region->upad = reqlen - region->klen; + + return 0; +} + +/* + * ZA is exposed as an array of horizontal vectors with the same + * format as SVE, mirroring the architecture's LDR ZA[Wv, offs], [Xn] + * instruction. + */ + static int get_sme_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { + int ret; + struct vec_state_reg_region region; + char __user *uptr = (char __user *)reg->addr; + /* Handle the KVM_REG_ARM64_SME_VLS pseudo-reg as a special case: */ if (reg->id == KVM_REG_ARM64_SME_VLS) return get_sme_vls(vcpu, reg); - return -EINVAL; + /* Try to interpret reg ID as an architectural SME register... */ + ret = sme_reg_to_region(®ion, vcpu, reg); + if (ret) + return ret; + + if (!kvm_arm_vcpu_vec_finalized(vcpu)) + return -EPERM; + + /* + * None of the SME specific registers are accessible unless + * PSTATE.ZA is set. + */ + if (!vcpu_za_enabled(vcpu)) + return -EINVAL; + + if (copy_from_user(vcpu->arch.sme_state + region.koffset, uptr, + region.klen)) + return -EFAULT; + + return 0; } static int set_sme_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { + int ret; + struct vec_state_reg_region region; + char __user *uptr = (char __user *)reg->addr; + /* Handle the KVM_REG_ARM64_SME_VLS pseudo-reg as a special case: */ if (reg->id == KVM_REG_ARM64_SME_VLS) return set_sme_vls(vcpu, reg); - return -EINVAL; + /* Try to interpret reg ID as an architectural SME register... */ + ret = sme_reg_to_region(®ion, vcpu, reg); + if (ret) + return ret; + + if (!kvm_arm_vcpu_vec_finalized(vcpu)) + return -EPERM; + + /* + * None of the SME specific registers are accessible unless + * PSTATE.ZA is set. + */ + if (!vcpu_za_enabled(vcpu)) + return -EINVAL; + + if (copy_from_user(vcpu->arch.sme_state + region.koffset, uptr, + region.klen)) + return -EFAULT; + + return 0; } + int kvm_arch_vcpu_ioctl_get_regs(struct kvm_vcpu *vcpu, struct kvm_regs *regs) { return -EINVAL; From patchwork Thu Apr 17 00:25:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054722 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D6B4E1B6CE9; Thu, 17 Apr 2025 00:27:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849634; cv=none; b=MAuax8OukqC+PG+oFEAIqpga18WWxYTV8xj70jqu45vZlYUQRUJ5sHOBegcd+gYRV18iaV1g6l6Nj/ihzorfl7f9dsf1kv+gWtV0kGKadFWGEak2D0Tj8ILRXLtzOgGhR2mcCK4BQiVEN+n3Lo0nt482Xqfa5VER3AOid6Qq7bI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849634; c=relaxed/simple; bh=b7kGdAfUth0T1NTyXq4Cq6Cp/WpJOurZv0+8UuWRaKc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=qoh2fBFa9c85pJJuXFMTx2LlbBLnYtveCH6lU9n3a72jzaMxrdXJEmcwO0pkCgnBLkilDfa5kzTRNfOR/9ir7LUfcWVzUf5+AWF905X16ieB5dSiP2wgaN4PreK4Z3jIV1DvSCmRKjM6nB12NrNH7smXIryRSf4IkQripLcmEL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=GJmfrk8O; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="GJmfrk8O" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A72EC4CEE2; Thu, 17 Apr 2025 00:27:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849633; bh=b7kGdAfUth0T1NTyXq4Cq6Cp/WpJOurZv0+8UuWRaKc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=GJmfrk8OJhGoS85xmYZEOhwbG2yy3hX6v+VAt45EFmDIzZbl30BswZhX9w0NhHH21 nApAM0uyBkK/uZDVFNwz4nOF0eADCxnXkDWJy26+V5qLYZ0qVlof89YQVmcUFfPTKl CoVpn6lBHMLIpxLfeJEdh5NhlRN1aG7DOJq/ZQyXkBkdhu6W+6RGYvsmGn1V2+W5cT rhqKpu6gAqPooI3+3mTFTtA7ENCBPNU+swgq7VSEbtXCTrKwWEMD12TuXyJqB04trD jiNCS95vg+D047mCB1gb6a2T3YAQVVArkNI3kFRoKbcrD/BEvsi1fB9rm1BIi8BrCw U8La1s4ubfXZg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:27 +0100 Subject: [PATCH v5 23/28] KVM: arm64: Context switch SME state for guests Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-23-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=17467; i=broonie@kernel.org; h=from:subject:message-id; bh=b7kGdAfUth0T1NTyXq4Cq6Cp/WpJOurZv0+8UuWRaKc=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEqBWZTcjpDadBZj5NYWLEZ0XJ2scPfqiDxG5I/e RmyhqJSJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKgQAKCRAk1otyXVSH0P0DB/ 9k8/YJgAZxL7imFmL04e4lsYcCJ0TcubJWUNAAGd4+q/xZEO/9GeaSQwKyIKMiWprBUW/DqO87+Tbt 1ZGYpYyj4q7NIJ3Bt0K/vzKLo258zWoKAIqrUzThJE85MJLa/4gAM0kYjwh5eCToX23c4sDyVgOuoM p7QvU5tb1PnzO5KnTcQKJWnv9bd4n1SBxyRaJKTHUB/1Z0488V2dIu73qdU/xNDx0zm6rvFOumKp9X ahoEDYEPa6DREMzS6d7LRp6K0QjSZA7Lr8/6H1N0ieSH80GIcWEIknp6hKzoRjr/oq++exUBbhnJOt p65cZPXgghvTmNpS/kq7+eYYqLlnzY X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB If the guest has SME state we need to context switch that state, provide support for that for normal guests. SME has three sets of registers, ZA, ZT (only present for SME2) and also streaming SVE which replaces the standard floating point registers when active. The first two are fairly straightforward, they are accessible only when PSTATE.ZA is set and we can reuse the assembly from the host to save and load them from a single contiguous buffer. When PSTATE.ZA is not set then these registers are inaccessible, when the guest enables PSTATE.ZA all bits will be set to 0 by that and nothing is required on restore. Streaming mode is slightly more complicated, when enabled via PSTATE.SM it provides a version of the SVE registers using the SME vector length and may optionally omit the FFR register. SME may also be present without SVE. The register state is stored in sve_state as for non-streaming SVE mode, we make an initial selection of registers to update based on the guest SVE support and then override this when loading SVCR if streaming mode is enabled. Since in order to avoid duplication with SME we now restore the register state outside of the SVE specific restore function we need to move the restore of the effective VL for nested guests to a separate restore function run after loading the floating point register state, along with the similar handling required for SME. The selection of which vector length to use is handled by vcpu_sve_pffr(). Signed-off-by: Mark Brown --- arch/arm64/include/asm/fpsimd.h | 10 +++ arch/arm64/include/asm/kvm_emulate.h | 12 +-- arch/arm64/include/asm/kvm_host.h | 4 + arch/arm64/kvm/fpsimd.c | 16 ++-- arch/arm64/kvm/hyp/include/hyp/switch.h | 142 +++++++++++++++++++++++++++++--- arch/arm64/kvm/hyp/nvhe/hyp-main.c | 48 ++++++++++- 6 files changed, 208 insertions(+), 24 deletions(-) diff --git a/arch/arm64/include/asm/fpsimd.h b/arch/arm64/include/asm/fpsimd.h index 7af636885c04..ca6b02c71c14 100644 --- a/arch/arm64/include/asm/fpsimd.h +++ b/arch/arm64/include/asm/fpsimd.h @@ -416,6 +416,15 @@ static inline size_t sme_state_size(struct task_struct const *task) write_sysreg_s(__new, (reg)); \ } while (0) +#define sme_cond_update_smcr_vq(val, reg) \ + do { \ + u64 __smcr = read_sysreg_s((reg)); \ + u64 __new = __smcr & ~SMCR_ELx_LEN_MASK; \ + __new |= (val) & SMCR_ELx_LEN_MASK; \ + if (__smcr != __new) \ + write_sysreg_s(__new, (reg)); \ + } while (0) + #else static inline void sme_user_disable(void) { BUILD_BUG(); } @@ -440,6 +449,7 @@ static inline size_t sme_state_size(struct task_struct const *task) } #define sme_cond_update_smcr(val, fa64, zt0, reg) do { } while (0) +#define sme_cond_update_smcr_vq(val, reg) do { } while (0) #endif /* ! CONFIG_ARM64_SME */ diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index bd020fc28aa9..c50c692373e8 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -608,12 +608,6 @@ static __always_inline void kvm_incr_pc(struct kvm_vcpu *vcpu) do { \ BUILD_BUG_ON((set) & CPTR_VHE_EL2_RES0); \ BUILD_BUG_ON((clr) & CPACR_EL1_E0POE); \ - __build_check_all_or_none((clr), CPACR_EL1_FPEN); \ - __build_check_all_or_none((set), CPACR_EL1_FPEN); \ - __build_check_all_or_none((clr), CPACR_EL1_ZEN); \ - __build_check_all_or_none((set), CPACR_EL1_ZEN); \ - __build_check_all_or_none((clr), CPACR_EL1_SMEN); \ - __build_check_all_or_none((set), CPACR_EL1_SMEN); \ \ if (has_vhe() || has_hvhe()) \ sysreg_clear_set(cpacr_el1, clr, set); \ @@ -691,4 +685,10 @@ static inline void vcpu_set_hcrx(struct kvm_vcpu *vcpu) vcpu->arch.hcrx_el2 |= HCRX_EL2_EnFPM; } } + +static inline bool guest_hyp_sme_traps_enabled(const struct kvm_vcpu *vcpu) +{ + return __guest_hyp_cptr_xen_trap_enabled(vcpu, SMEN); +} + #endif /* __ARM64_KVM_EMULATE_H__ */ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index ef8b60222ac9..e3d9e241a46f 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -682,6 +682,7 @@ struct kvm_host_data { /* Used by pKVM only. */ u64 fpmr; + u64 smcr_el1; /* Ownership of the FP regs */ enum { @@ -1052,6 +1053,9 @@ struct kvm_vcpu_arch { #define vcpu_sve_zcr_elx(vcpu) \ (unlikely(is_hyp_ctxt(vcpu)) ? ZCR_EL2 : ZCR_EL1) +#define vcpu_sme_smcr_elx(vcpu) \ + (unlikely(is_hyp_ctxt(vcpu)) ? SMCR_EL2 : SMCR_EL1) + #define vcpu_sve_state_size(vcpu) ({ \ size_t __size_ret; \ unsigned int __vcpu_vq; \ diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 72c67b067ca1..1cc3bf437e64 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -95,19 +95,25 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) WARN_ON_ONCE(!irqs_disabled()); if (guest_owns_fp_regs()) { - /* - * Currently we do not support SME guests so SVCR is - * always 0 and we just need a variable to point to. - */ fp_state.st = &vcpu->arch.ctxt.fp_regs; fp_state.sve_state = vcpu->arch.sve_state; fp_state.sve_vl = vcpu->arch.max_vl[ARM64_VEC_SVE]; - fp_state.sme_state = NULL; + fp_state.sme_vl = vcpu->arch.max_vl[ARM64_VEC_SME]; + fp_state.sme_state = vcpu->arch.sme_state; fp_state.svcr = &__vcpu_sys_reg(vcpu, SVCR); fp_state.fpmr = &__vcpu_sys_reg(vcpu, FPMR); fp_state.fp_type = &vcpu->arch.fp_type; + fp_state.sme_features = 0; + if (kvm_has_fa64(vcpu->kvm)) + fp_state.sme_features |= SMCR_ELx_FA64; + if (kvm_has_sme2(vcpu->kvm)) + fp_state.sme_features |= SMCR_ELx_EZT0; + /* + * For SME only hosts fpsimd_save() will override the + * state selection if we are in streaming mode. + */ if (vcpu_has_sve(vcpu)) fp_state.to_save = FP_STATE_SVE; else diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 1c0ef9d6609d..c21e1448ae7d 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -343,6 +343,29 @@ static inline bool kvm_hyp_handle_mops(struct kvm_vcpu *vcpu, u64 *exit_code) return true; } +static inline void __hyp_sme_restore_guest(struct kvm_vcpu *vcpu, + bool *restore_sve, + bool *restore_ffr) +{ + bool has_fa64 = vcpu_has_fa64(vcpu); + bool has_sme2 = vcpu_has_sme2(vcpu); + + sme_cond_update_smcr(vcpu_sme_max_vq(vcpu) - 1, has_fa64, has_sme2, + SYS_SMCR_EL2); + + write_sysreg_el1(__vcpu_sys_reg(vcpu, SMCR_EL1), SYS_SMCR); + + write_sysreg_s(__vcpu_sys_reg(vcpu, SVCR), SYS_SVCR); + + if (vcpu_in_streaming_mode(vcpu)) { + *restore_sve = true; + *restore_ffr = has_fa64; + } + + if (vcpu_za_enabled(vcpu)) + __sme_restore_state(vcpu_sme_state(vcpu), has_sme2); +} + static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) { /* @@ -350,19 +373,25 @@ static inline void __hyp_sve_restore_guest(struct kvm_vcpu *vcpu) * vCPU. Start off with the max VL so we can load the SVE state. */ sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2); - __sve_restore_state(vcpu_sve_pffr(vcpu), - &vcpu->arch.ctxt.fp_regs.fpsr, - true); + write_sysreg_el1(__vcpu_sys_reg(vcpu, vcpu_sve_zcr_elx(vcpu)), SYS_ZCR); +} + +static inline void __hyp_nv_restore_guest_vls(struct kvm_vcpu *vcpu) +{ /* * The effective VL for a VM could differ from the max VL when running a * nested guest, as the guest hypervisor could select a smaller VL. Slap * that into hardware before wrapping up. */ - if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) + if (!(vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu))) + return; + + if (vcpu_has_sve(vcpu)) sve_cond_update_zcr_vq(__vcpu_sys_reg(vcpu, ZCR_EL2), SYS_ZCR_EL2); - write_sysreg_el1(__vcpu_sys_reg(vcpu, vcpu_sve_zcr_elx(vcpu)), SYS_ZCR); + if (vcpu_has_sme(vcpu)) + sme_cond_update_smcr_vq(__vcpu_sys_reg(vcpu, SMCR_EL2), SYS_SMCR_EL2); } static inline void __hyp_sve_save_host(void) @@ -378,6 +407,7 @@ static inline void __hyp_sve_save_host(void) static inline void fpsimd_lazy_switch_to_guest(struct kvm_vcpu *vcpu) { + u64 smcr_el1, smcr_el2; u64 zcr_el1, zcr_el2; if (!guest_owns_fp_regs()) @@ -395,10 +425,29 @@ static inline void fpsimd_lazy_switch_to_guest(struct kvm_vcpu *vcpu) zcr_el1 = __vcpu_sys_reg(vcpu, vcpu_sve_zcr_elx(vcpu)); write_sysreg_el1(zcr_el1, SYS_ZCR); } + + if (vcpu_has_sme(vcpu)) { + /* A guest hypervisor may restrict the effective max VL. */ + if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) + smcr_el2 = __vcpu_sys_reg(vcpu, SMCR_EL2); + else + smcr_el2 = vcpu_sme_max_vq(vcpu) - 1; + + if (vcpu_has_fa64(vcpu)) + smcr_el2 |= SMCR_ELx_FA64; + if (vcpu_has_sme2(vcpu)) + smcr_el2 |= SMCR_ELx_EZT0; + + write_sysreg_el2(smcr_el2, SYS_SMCR); + + smcr_el1 = __vcpu_sys_reg(vcpu, vcpu_sme_smcr_elx(vcpu)); + write_sysreg_el1(smcr_el1, SYS_SMCR); + } } static inline void fpsimd_lazy_switch_to_host(struct kvm_vcpu *vcpu) { + u64 smcr_el1, smcr_el2; u64 zcr_el1, zcr_el2; if (!guest_owns_fp_regs()) @@ -433,10 +482,54 @@ static inline void fpsimd_lazy_switch_to_host(struct kvm_vcpu *vcpu) write_sysreg_el1(zcr_el1, SYS_ZCR); } } + + if (vcpu_has_sme(vcpu)) { + smcr_el1 = read_sysreg_el1(SYS_SMCR); + __vcpu_sys_reg(vcpu, vcpu_sme_smcr_elx(vcpu)) = smcr_el1; + + smcr_el2 = 0; + if (system_supports_fa64()) + smcr_el2 |= SMCR_ELx_FA64; + if (system_supports_sme2()) + smcr_el2 |= SMCR_ELx_EZT0; + + /* + * The guest's state is always saved using the guest's max VL. + * Ensure that the host has the guest's max VL active such that + * the host can save the guest's state lazily, but don't + * artificially restrict the host to the guest's max VL. + */ + if (has_vhe()) { + smcr_el2 |= vcpu_sme_max_vq(vcpu) - 1; + write_sysreg_el2(smcr_el2, SYS_SMCR); + } else { + smcr_el1 = smcr_el2; + smcr_el2 |= sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SME]) - 1; + write_sysreg_el2(smcr_el2, SYS_SMCR); + + smcr_el1 |= vcpu_sve_max_vq(vcpu) - 1; + write_sysreg_el1(smcr_el1, SYS_SMCR); + } + } } static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu) { + /* + * The hypervisor refuses to run if streaming mode or ZA is + * enabled, we only need to save SMCR_EL1 for SME. For pKVM + * we will restore this, reset SMCR_EL2 to a fixed value and + * disable streaming mode and ZA to avoid any state being + * leaked. + */ + if (system_supports_sme()) { + *host_data_ptr(smcr_el1) = read_sysreg_el1(SYS_SMCR); + + /* Re-enable SME traps if not supported for the guest vcpu. */ + if (!vcpu_has_sme(vcpu)) + cpacr_clear_set(CPACR_EL1_SMEN, 0); + } + /* * Non-protected kvm relies on the host restoring its sve state. * Protected kvm restores the host's sve state as not to reveal that @@ -466,14 +559,18 @@ static void kvm_hyp_save_fpsimd_host(struct kvm_vcpu *vcpu) */ static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) { - bool sve_guest; - u8 esr_ec; + u64 cpacr; + bool restore_sve, restore_ffr; + bool sve_guest, sme_guest; + u8 esr_ec, esr_iss_smtc; if (!system_supports_fpsimd()) return false; sve_guest = vcpu_has_sve(vcpu); + sme_guest = vcpu_has_sme(vcpu); esr_ec = kvm_vcpu_trap_get_class(vcpu); + esr_iss_smtc = ESR_ELx_SME_ISS_SMTC((kvm_vcpu_get_esr(vcpu))); /* Only handle traps the vCPU can support here: */ switch (esr_ec) { @@ -492,6 +589,15 @@ static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) if (guest_hyp_sve_traps_enabled(vcpu)) return false; break; + case ESR_ELx_EC_SME: + if (!sme_guest) + return false; + if (guest_hyp_sme_traps_enabled(vcpu)) + return false; + if (!kvm_has_sme2(vcpu->kvm) && + (esr_iss_smtc == ESR_ELx_SME_ISS_SMTC_ZT_DISABLED)) + return false; + break; default: return false; } @@ -499,10 +605,12 @@ static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) /* Valid trap. Switch the context: */ /* First disable enough traps to allow us to update the registers */ + cpacr = CPACR_EL1_FPEN; if (sve_guest || (is_protected_kvm_enabled() && system_supports_sve())) - cpacr_clear_set(0, CPACR_EL1_FPEN | CPACR_EL1_ZEN); - else - cpacr_clear_set(0, CPACR_EL1_FPEN); + cpacr |= CPACR_EL1_ZEN; + if (sme_guest || (is_protected_kvm_enabled() && system_supports_sme())) + cpacr |= CPACR_EL1_SMEN; + cpacr_clear_set(0, cpacr); isb(); /* Write out the host state if it's in the registers */ @@ -510,8 +618,20 @@ static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) kvm_hyp_save_fpsimd_host(vcpu); /* Restore the guest state */ + + /* These may be overridden for a SME guest */ + restore_sve = sve_guest; + restore_ffr = sve_guest; + if (sve_guest) __hyp_sve_restore_guest(vcpu); + if (sme_guest) + __hyp_sme_restore_guest(vcpu, &restore_sve, &restore_ffr); + + if (restore_sve) + __sve_restore_state(vcpu_sve_pffr(vcpu), + &vcpu->arch.ctxt.fp_regs.fpsr, + restore_ffr); else __fpsimd_restore_state(&vcpu->arch.ctxt.fp_regs); @@ -522,6 +642,8 @@ static inline bool kvm_hyp_handle_fpsimd(struct kvm_vcpu *vcpu, u64 *exit_code) if (!(read_sysreg(hcr_el2) & HCR_RW)) write_sysreg(__vcpu_sys_reg(vcpu, FPEXC32_EL2), fpexc32_el2); + __hyp_nv_restore_guest_vls(vcpu); + *host_data_ptr(fp_owner) = FP_STATE_GUEST_OWNED; return true; diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 1fb38dc64357..d7dfc76198a5 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -26,6 +26,8 @@ void __kvm_hyp_host_forward_smc(struct kvm_cpu_context *host_ctxt); static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu) { + bool save_ffr = !vcpu_in_streaming_mode(vcpu) || vcpu_has_fa64(vcpu); + __vcpu_sys_reg(vcpu, ZCR_EL1) = read_sysreg_el1(SYS_ZCR); /* * On saving/restoring guest sve state, always use the maximum VL for @@ -33,7 +35,7 @@ static void __hyp_sve_save_guest(struct kvm_vcpu *vcpu) * on the VL, so use a consistent (i.e., the maximum) guest VL. */ sve_cond_update_zcr_vq(vcpu_sve_max_vq(vcpu) - 1, SYS_ZCR_EL2); - __sve_save_state(vcpu_sve_pffr(vcpu), &vcpu->arch.ctxt.fp_regs.fpsr, true); + __sve_save_state(vcpu_sve_pffr(vcpu), &vcpu->arch.ctxt.fp_regs.fpsr, save_ffr); write_sysreg_s(sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SVE]) - 1, SYS_ZCR_EL2); } @@ -57,6 +59,38 @@ static void __hyp_sve_restore_host(void) write_sysreg_el1(sve_state->zcr_el1, SYS_ZCR); } +static void __hyp_sme_save_guest(struct kvm_vcpu *vcpu) +{ + __vcpu_sys_reg(vcpu, SMCR_EL1) = read_sysreg_el1(SYS_SMCR); + __vcpu_sys_reg(vcpu, SVCR) = read_sysreg_s(SYS_SVCR); + + sme_cond_update_smcr(vcpu_sme_max_vq(vcpu) - 1, vcpu_has_fa64(vcpu), + vcpu_has_sme2(vcpu), SYS_SMCR_EL2); + + if (vcpu_za_enabled(vcpu)) + __sme_save_state(vcpu_sme_state(vcpu), vcpu_has_sme2(vcpu)); +} + +static void __hyp_sme_restore_host(void) +{ + /* + * The hypervisor refuses to run if we are in streaming mode + * or have ZA enabled so there is no SME specific state to + * restore other than the system registers. + * + * Note that this constrains the PE to the maximum shared VL + * that was discovered, if we wish to use larger VLs this will + * need to be revisited. + */ + sme_cond_update_smcr(sve_vq_from_vl(kvm_host_max_vl[ARM64_VEC_SME]) - 1, + cpus_have_final_cap(ARM64_SME_FA64), + cpus_have_final_cap(ARM64_SME2), SYS_SMCR_EL2); + + write_sysreg_el1(*host_data_ptr(smcr_el1), SYS_SMCR); + + sme_smstop(); +} + static void fpsimd_sve_flush(void) { *host_data_ptr(fp_owner) = FP_STATE_HOST_OWNED; @@ -69,10 +103,13 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) if (!guest_owns_fp_regs()) return; - cpacr_clear_set(0, CPACR_EL1_FPEN | CPACR_EL1_ZEN); + cpacr_clear_set(0, CPACR_EL1_FPEN | CPACR_EL1_ZEN | CPACR_EL1_SMEN); isb(); - if (vcpu_has_sve(vcpu)) + if (vcpu_has_sme(vcpu)) + __hyp_sme_save_guest(vcpu); + + if (vcpu_has_sve(vcpu) || vcpu_in_streaming_mode(vcpu)) __hyp_sve_save_guest(vcpu); else __fpsimd_save_state(&vcpu->arch.ctxt.fp_regs); @@ -81,6 +118,9 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) if (has_fpmr) __vcpu_sys_reg(vcpu, FPMR) = read_sysreg_s(SYS_FPMR); + if (system_supports_sme()) + __hyp_sme_restore_host(); + if (system_supports_sve()) __hyp_sve_restore_host(); else @@ -125,6 +165,8 @@ static void flush_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu) hyp_vcpu->vcpu.arch.ctxt = host_vcpu->arch.ctxt; hyp_vcpu->vcpu.arch.sve_state = kern_hyp_va(host_vcpu->arch.sve_state); + hyp_vcpu->vcpu.arch.sme_state = kern_hyp_va(host_vcpu->arch.sme_state); + /* Limit guest vector length to the maximum supported by the host. */ for (i = 0; i < ARM64_VEC_MAX; i++) hyp_vcpu->vcpu.arch.max_vl[i] = min(host_vcpu->arch.max_vl[i], From patchwork Thu Apr 17 00:25:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054723 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0EEC21C3F30; Thu, 17 Apr 2025 00:27:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849638; cv=none; b=qZy/15Lh6Tja3xnnz0wlHBlvTTMaCq5TvUoyoPeQ+hWSZwdCzwjnFJrbPaIvf3VWahqJtj/I4rmOZTBcodRLq0bYpvEJb0PYmODhpzxxuQMvCEEiC6HFy/FFaG+BBrzAx1XbKWlWzGSyITDseOrjqMQPXC8jdpusTTfyA7CGuqc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849638; c=relaxed/simple; bh=pe5aI37YwedWzR6XP4KVR8L0CbNpDRvIXsy/vJcYfoI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=jxE8zXSDS/Nh6A+zYlGf0JWNnniPPx7fWszxsAuiRkULVu8x87zphG23Mhc6bsj/dLXflrWKbAci/aWke4RhfWnIoF8e3VawdXaaizuFtdeaBvx6qMBsTWUqoNp9XGSkNzoZZEVpGvCVa3J+ao1iGsHRL6RtocCMJRlMgL75XEc= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Qn2UP3ce; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Qn2UP3ce" Received: by smtp.kernel.org (Postfix) with ESMTPSA id C6551C4CEEE; Thu, 17 Apr 2025 00:27:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849637; bh=pe5aI37YwedWzR6XP4KVR8L0CbNpDRvIXsy/vJcYfoI=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=Qn2UP3ce+NDDRx975aSO2yZEf3+Ie0B9BupB1NCJwcrAijYfZiwMKQNtzzAoZDLaF dftxI337gmIxFJBCtDPh784f4RPMb3nHK53jrN35gckOFkDe+R8/EzVAsoMbNBCyWI YEZumZADCBsd0L/ct2rvWS1INVLj70M4PRDhgDBOCrMDkgxYCYyYAUPJ1QCh5DyaVE vwIdxhNZOWO1e+xDAtGR8KaMb3mI6rR8YlG5Wzfzr3T20yaVptQziwtP2pZklvjYSD yOT4ZZVbf5UntOuXrzVTLko+UGWHjpiFK7N3OK5eWEv5pBxZtSX4kbGOoKyZQFTgSc gNjIhYLqM8eFg== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:28 +0100 Subject: [PATCH v5 24/28] KVM: arm64: Handle SME exceptions Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-24-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=9675; i=broonie@kernel.org; h=from:subject:message-id; bh=pe5aI37YwedWzR6XP4KVR8L0CbNpDRvIXsy/vJcYfoI=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEqBhCmUmWSDkXi1qqNwc9qof49aABQf9d2SonRF 6xkW1D+JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKgQAKCRAk1otyXVSH0DF4B/ 9qSK6BOlvAbecTiGsouhvmBoO1IpmjYs0U1JxdLx+mFKe1PXSfVgSgl0wacRKZ8MNfuuGzbv/E8TNU nuBMVPgz7rPx3NrrNMG6u9Ml+I6N9Cah0jfc+ibrHOZxfS0YaMgxSz7SqAPG+J/sPJwymLxjJ+m8CU UhPCqT/ecN/CF2cylnKsqTmp2En2Oco9HfxP5kJqw0/7ZhBLqewoNWkbr55CnoS3urfx/EokydVTYZ jEIR5fQcPGoEt5wIIa5C94jEjVmDPamGTNCRlJsGlOMxmZ1wDNmFqUzABrEdoYibRbTMiVHTskayqP CbL2gSgnS3cp9Ko5px6c6GKblI2Q72 X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB The access control for SME follows the same structure as for the base FP and SVE extensions, with control being via CPACR_ELx.SMEN and CPTR_EL2.TSM mirroring the equivalent FPSIMD and SVE controls in those registers. Add handling for these controls and exceptions mirroring the existing handling for FPSIMD and SVE. When the hardware is in streaming mode guest operations that are invalid in in streaming mode will generate SME exceptions. In many situations these exceptions are routed directly to the lower ELs with no opportunity for the hypervisor to intercept. So that guests do not see unexpected exception types due to the hardware streaming mode configuration not being what the guest configured we update SVCR when enabling floating point traps for the guest. For pKVM this means that when changing SVCR we need to save the host floating point state before returning to the guest. Signed-off-by: Mark Brown squash ctxtsync --- arch/arm64/kvm/handle_exit.c | 14 +++++++++++++ arch/arm64/kvm/hyp/nvhe/hyp-main.c | 22 ++++++++++++++++----- arch/arm64/kvm/hyp/nvhe/switch.c | 40 +++++++++++++++++++++++++++++++------- arch/arm64/kvm/hyp/vhe/switch.c | 29 ++++++++++++++++++++++----- 4 files changed, 88 insertions(+), 17 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index b73dc26bc44b..6b7f83cc5a20 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -231,6 +231,19 @@ static int handle_sve(struct kvm_vcpu *vcpu) return 1; } +/* + * Guest access to SME registers should be routed to this handler only + * when the system doesn't support SME. + */ +static int handle_sme(struct kvm_vcpu *vcpu) +{ + if (guest_hyp_sme_traps_enabled(vcpu)) + return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + + kvm_inject_undefined(vcpu); + return 1; +} + /* * Two possibilities to handle a trapping ptrauth instruction: * @@ -314,6 +327,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SVC64] = handle_svc, [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, [ESR_ELx_EC_SVE] = handle_sve, + [ESR_ELx_EC_SME] = handle_sme, [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort, diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index d7dfc76198a5..a2132682913a 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -100,8 +100,21 @@ static void fpsimd_sve_sync(struct kvm_vcpu *vcpu) { bool has_fpmr; - if (!guest_owns_fp_regs()) + if (!guest_owns_fp_regs()) { + /* + * We always restore SVCR for SME guests to ensure + * exceptions within the guest are delivered with the + * right type, always reset it to the fixed host + * value. + */ + if (vcpu_has_sme(vcpu)) { + cpacr_clear_set(0, CPACR_EL1_SMEN); + isb(); + + sme_smstop(); + } return; + } cpacr_clear_set(0, CPACR_EL1_FPEN | CPACR_EL1_ZEN | CPACR_EL1_SMEN); isb(); @@ -249,10 +262,9 @@ static void handle___kvm_vcpu_run(struct kvm_cpu_context *host_ctxt) struct pkvm_hyp_vcpu *hyp_vcpu = pkvm_get_loaded_hyp_vcpu(); /* - * KVM (and pKVM) doesn't support SME guests for now, and - * ensures that SME features aren't enabled in pstate when - * loading a vcpu. Therefore, if SME features enabled the host - * is misbehaving. + * KVM (and pKVM) refuses to run if PSTATE.{SM,ZA} are + * enabled. Therefore, if SME features enabled the + * host is misbehaving. */ if (unlikely(system_supports_sme() && read_sysreg_s(SYS_SVCR))) { ret = -EINVAL; diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index 7d2ba6ef0261..d945e6c79504 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -35,12 +35,37 @@ DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); extern void kvm_nvhe_prepare_backtrace(unsigned long fp, unsigned long pc); +static void __activate_traps_sme(struct kvm_vcpu *vcpu) +{ + if (!vcpu_has_sme(vcpu)) + return; + + if (__vcpu_sys_reg(vcpu, SVCR) == read_sysreg_s(SYS_SVCR)) + return; + + /* + * Write out the host state if it's in the registers, + * updating SVCR will invalidate it. + */ + if (host_owns_fp_regs()) + kvm_hyp_save_fpsimd_host(vcpu); + + /* + * Always restore SVCR to ensure that exceptions delivered + * directly within the guest have the correct type. + */ + write_sysreg_s(__vcpu_sys_reg(vcpu, SVCR), SYS_SVCR); + *host_data_ptr(fp_owner) = FP_STATE_FREE; +} + static void __activate_cptr_traps(struct kvm_vcpu *vcpu) { u64 val = CPTR_EL2_TAM; /* Same bit irrespective of E2H */ - if (!guest_owns_fp_regs()) + if (!guest_owns_fp_regs()) { __activate_traps_fpsimd32(vcpu); + __activate_traps_sme(vcpu); + } if (has_hvhe()) { val |= CPACR_EL1_TTA; @@ -49,17 +74,16 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) val |= CPACR_EL1_FPEN; if (vcpu_has_sve(vcpu)) val |= CPACR_EL1_ZEN; + if (vcpu_has_sme(vcpu)) + val |= CPACR_EL1_SMEN; } write_sysreg(val, cpacr_el1); } else { val |= CPTR_EL2_TTA | CPTR_NVHE_EL2_RES1; - /* - * Always trap SME since it's not supported in KVM. - * TSM is RES1 if SME isn't implemented. - */ - val |= CPTR_EL2_TSM; + if (!vcpu_has_sme(vcpu) || !guest_owns_fp_regs()) + val |= CPTR_EL2_TSM; if (!vcpu_has_sve(vcpu) || !guest_owns_fp_regs()) val |= CPTR_EL2_TZ; @@ -222,6 +246,7 @@ static const exit_handler_fn hyp_exit_handlers[] = { [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg, [ESR_ELx_EC_SVE] = kvm_hyp_handle_fpsimd, + [ESR_ELx_EC_SME] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, [ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low, @@ -233,7 +258,8 @@ static const exit_handler_fn pvm_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = NULL, [ESR_ELx_EC_SYS64] = kvm_handle_pvm_sys64, [ESR_ELx_EC_SVE] = kvm_handle_pvm_restricted, - [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, + [ESR_ELx_EC_SME] = kvm_handle_pvm_restricted, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_pvm_restricted, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, [ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low, [ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low, diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 731a0378ed13..eb50e13d7013 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -67,6 +67,7 @@ static u64 __compute_hcr(struct kvm_vcpu *vcpu) static void __activate_cptr_traps(struct kvm_vcpu *vcpu) { + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; u64 cptr; /* @@ -83,8 +84,17 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) val |= CPACR_EL1_FPEN; if (vcpu_has_sve(vcpu)) val |= CPACR_EL1_ZEN; + if (vcpu_has_sme(vcpu)) + val |= CPACR_EL1_SMEN; } else { __activate_traps_fpsimd32(vcpu); + + /* + * Streaming mode affects exception types delivered + * directly to lower ELs for FP operations, configure it. + */ + if (vcpu_has_sme(vcpu)) + write_sysreg_s(ctxt_sys_reg(ctxt, SVCR), SYS_SVCR); } if (!vcpu_has_nv(vcpu)) @@ -126,6 +136,8 @@ static void __activate_cptr_traps(struct kvm_vcpu *vcpu) val &= ~CPACR_EL1_FPEN; if (!(SYS_FIELD_GET(CPACR_EL1, ZEN, cptr) & BIT(0))) val &= ~CPACR_EL1_ZEN; + if (!(SYS_FIELD_GET(CPACR_EL1, SMEN, cptr) & BIT(0))) + val &= ~CPACR_EL1_SMEN; if (kvm_has_feat(vcpu->kvm, ID_AA64MMFR3_EL1, S2POE, IMP)) val |= cptr & CPACR_EL1_E0POE; @@ -486,22 +498,28 @@ static bool kvm_hyp_handle_cpacr_el1(struct kvm_vcpu *vcpu, u64 *exit_code) return true; } -static bool kvm_hyp_handle_zcr_el2(struct kvm_vcpu *vcpu, u64 *exit_code) +static bool kvm_hyp_handle_vec_cr_el2(struct kvm_vcpu *vcpu, u64 *exit_code) { u32 sysreg = esr_sys64_to_sysreg(kvm_vcpu_get_esr(vcpu)); if (!vcpu_has_nv(vcpu)) return false; - if (sysreg != SYS_ZCR_EL2) + switch (sysreg) { + case SYS_ZCR_EL2: + case SYS_SMCR_EL2: + break; + default: return false; + } if (guest_owns_fp_regs()) return false; /* - * ZCR_EL2 traps are handled in the slow path, with the expectation - * that the guest's FP context has already been loaded onto the CPU. + * ZCR_EL2 and SMCR_EL2 traps are handled in the slow path, + * with the expectation that the guest's FP context has + * already been loaded onto the CPU. * * Load the guest's FP context and unconditionally forward to the * slow path for handling (i.e. return false). @@ -521,7 +539,7 @@ static bool kvm_hyp_handle_sysreg_vhe(struct kvm_vcpu *vcpu, u64 *exit_code) if (kvm_hyp_handle_cpacr_el1(vcpu, exit_code)) return true; - if (kvm_hyp_handle_zcr_el2(vcpu, exit_code)) + if (kvm_hyp_handle_vec_cr_el2(vcpu, exit_code)) return true; return kvm_hyp_handle_sysreg(vcpu, exit_code); @@ -550,6 +568,7 @@ static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = NULL, [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, [ESR_ELx_EC_SYS64] = kvm_hyp_handle_sysreg_vhe, + [ESR_ELx_EC_SME] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_SVE] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_FP_ASIMD] = kvm_hyp_handle_fpsimd, [ESR_ELx_EC_IABT_LOW] = kvm_hyp_handle_iabt_low, From patchwork Thu Apr 17 00:25:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054724 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C3E661C3F30; Thu, 17 Apr 2025 00:27:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849642; cv=none; b=LjCX9NngCIkFg6EqM/0E7ztkNy+kKP7fq5Tf1SEPXpLo1YmB0KtQ/4fG2py8nunOvPLB0jsxpqlpsy34H1b4gc9fEchk3BWN2N+jHeqe/3MxBT1a6hMhfHWxSMniiuUuxfbz4p2rlhXknNr/QBUj4PWSg++3x1TRQ5zYnsUcguA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849642; c=relaxed/simple; bh=h3hcTuhJ/enbUfNVFyfDycfLuRL7LaoTm3WlqJwa4DQ=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=Oa4LEHXMlnsDN3tgPGJXoQmnC6H1yLYKwQ3Lhk/eAwzqyAxZV45hUMypruYcnjx8qzGsXyFygpEksDJQXA2SKs4W2exAKWZ/BD72aOrTLqUDzbC+2sbwv2OLR5wXXx6c105mje1D8rIDttcXENmgUQIqQuvst1rjURFoOmv90dw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=husKvgdp; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="husKvgdp" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 2587AC4CEE2; Thu, 17 Apr 2025 00:27:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849642; bh=h3hcTuhJ/enbUfNVFyfDycfLuRL7LaoTm3WlqJwa4DQ=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=husKvgdpNGac4aEmz2a8LrtekbLfHboqcUZTISl25gegAlsSISd2QxneSp4ORGFED fAjrjtPZHo3rBf04uAuXwwvla5fntn/XrZ82Ke/EwuQQnJYiy5M7eh1iN1aqhar+Zt PkqHf1WIUt8dsTwzoZa3rHsHccxTe67whG8iutYEvQs2noL8l0/IcidhhQlTunX7uZ FcoK25ca2+Sc3OB77m9CtMmEeYENFpGjhhcgBqAKYvmdsRgUKFZCUteZWaL/HtXv0p ztd6V3Knvnp20un3cFDtg8z84fZFDOjVCHKvNftyvCixxluxVioAX92QyrbTF14w74 Z/MbxIPlt6sSQ== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:29 +0100 Subject: [PATCH v5 25/28] KVM: arm64: Expose SME to nested guests Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-25-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=1232; i=broonie@kernel.org; h=from:subject:message-id; bh=h3hcTuhJ/enbUfNVFyfDycfLuRL7LaoTm3WlqJwa4DQ=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEqCQNyqY3LZseLPc/PyUZbteRvSD1dXOrGL020K YedjAN2JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKggAKCRAk1otyXVSH0LCbB/ 9BF3Mn5ovMuDqziZZPCiLH51J+v/o26aqdZXm7BidvPRR+1FKOtjOli2O62XceQ9oZxtM4+OatkK8X JO9l5Olh4UHg5AxMD1SqwPamsY9cGN0qSk3EuoxHDm6IOTZtpZY81egCAQlJzgTj6uYE5au84H0KdL 2y9s7VFUIT6+JXKBZfYT1d/0DFqo7VgvFoy4AqnVHTGI8QRqNvKAfdLwaYNFxt5vZE8fp/J7JDY1QU 0mzm3gp7pIvoy/O73+hqGn/SlmOcZUiQQSwMsWDugQLhL24QeKPHpNJzIEpKfcwtTHj2Dsb6z2HSGm PHXHtBAvUK0TBqV7tj/kdmTktd8MjL X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB With support for context switching SME state in place allow access to SME in nested guests. The SME floating point state is handled along with all the other floating point state, SME specific floating point exceptions are directed into the same handlers as other floating point exceptions with NV specific handling for the vector lengths already in place. TPIDR2_EL0 is context switched along with the other TPIDRs as part of the main guest register context switch. SME priority support is currently masked from all guests including nested ones. Signed-off-by: Mark Brown --- arch/arm64/kvm/nested.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 4a3fc11f7ecf..a599216ce1ce 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -841,9 +841,10 @@ u64 limit_nv_id_reg(struct kvm *kvm, u32 reg, u64 val) break; case SYS_ID_AA64PFR1_EL1: - /* Only support BTI, SSBS, CSV2_frac */ + /* Only support BTI, SSBS, SME, CSV2_frac */ val &= (ID_AA64PFR1_EL1_BT | ID_AA64PFR1_EL1_SSBS | + ID_AA64PFR1_EL1_SME | ID_AA64PFR1_EL1_CSV2_frac); break; From patchwork Thu Apr 17 00:25:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054725 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 672E41C84D4; Thu, 17 Apr 2025 00:27:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849647; cv=none; b=Ky4SPdE44CHl4k76qdHt8WS9PhrayKqoHNtDafPSvfxQAV0oOtXhzocweZU88LpeOP9aBGcehzU6D9X5IDJ0qHuqIumMKpuB2fC13m7E81LsCclKyBsMXHeu/GI6vdg1+p0JK6VMkmhFfhPKt2Xo/vtfVQLgRMcUPLMKK14cYfw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849647; c=relaxed/simple; bh=q0Sr7+n6vZkOXsRfHInh2ZvyWEHQCHphx4n5fKiw7Nc=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=NG3W/hXFygDtTGZeZUJCqibwfuoZtKNNH/rxXxjuwrF/F1DvXy9JoX366nefosToZjYJi12VNL25GqamwBvWyJjxQv/cXCpFu0pip0Bbo/0qmDgzskCRR24UxEHgAgAT/ONfQ0BaBgvUnvCmaEbpXWwbDrBbPZ3soKwTNm7v5qM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=izYOQyFs; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="izYOQyFs" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 36E87C4CEEC; Thu, 17 Apr 2025 00:27:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849647; bh=q0Sr7+n6vZkOXsRfHInh2ZvyWEHQCHphx4n5fKiw7Nc=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=izYOQyFs95FGUIC9Johp3pMEKWgqxrdWOgr6/u12NmCpp5E43BpX1XnKIYQZSBFNZ aHQD+4acsjWUy9FgOjOqT5gaXowc9lZrA7oK1bARxHhp7ltet66olmq+BEOXGOJ41/ uSwbpPdEN4vGXBcMo589xFtkK4mQU2wA6rg3Tus2qrlIv4a8kA94oxOJi6B0NAvDoy KSRzY8DXq3/+ODmFsp2rIsJmmm9dU4MannhYKyVl4vdq/a5PNmCgOQiuaA+/g8qdDs /ldWNfsULJwWYZRuyI+tRamOEcqiyKKJtqjlQ5QTcMyeE5sgsRfhkEswfwD4OwtKKA O2xnAqKSP55pQ== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:30 +0100 Subject: [PATCH v5 26/28] KVM: arm64: Provide interface for configuring and enabling SME for guests Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-26-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=13016; i=broonie@kernel.org; h=from:subject:message-id; bh=q0Sr7+n6vZkOXsRfHInh2ZvyWEHQCHphx4n5fKiw7Nc=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEqDp8UH+NVAgyNF+8WRT+0bnRcdAJpd9GhC2fOR 0SUBHymJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKgwAKCRAk1otyXVSH0CCOB/ 4wmNVB6jDJ5Jt2wWhTAHEIhjD8Fng7j4gs5ch88jceDlxaT5BR150kYf7k9/A6kYe/Y72ohzxfrhzt 8IZashprUGf1B9igun4LuJveRDc5s0Ldrki//HTjjHT7FI5rAzsX/K3Jez8XE47dj/IZ9zRC7naAfQ 3VQc335qRfqnduKsS6CJLbzzqgNyCHTwzJvYT98JRaVVgKmtEa8BMPugjbFac3O2aa6fdvPXFUHWCv Bjg516vJQ5kVPaEzKu673jbGbnKjfn2clMnksRlFMB05oPeQj5JQOXxMEVBGf6WwkY5dDvhlw426aR TF8Slr41ote/CNoURomTtEYsUd5tyR X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Since SME requires configuration of a vector length in order to know the size of both the streaming mode SVE state and ZA array we implement a capability for it and require that it be enabled and finalized before the SME specific state can be accessed, similarly to SVE. Due to the overlap with sizing the SVE state we finalise both SVE and SME with a single finalization, preventing any further changes to the SVE and SME configuration once KVM_ARM_VCPU_VEC (an alias for _VCPU_SVE) has been finalised. This is not a thing of great elegance but it ensures that we never have a state where one of SVE or SME is finalised and the other not, avoiding complexity. SME is supported for normal and protected guests. Signed-off-by: Mark Brown --- arch/arm64/include/asm/kvm_host.h | 9 +-- arch/arm64/include/uapi/asm/kvm.h | 1 + arch/arm64/kvm/arm.c | 10 ++++ arch/arm64/kvm/hyp/nvhe/pkvm.c | 16 ++--- arch/arm64/kvm/hyp/nvhe/sys_regs.c | 6 ++ arch/arm64/kvm/reset.c | 116 +++++++++++++++++++++++++++++++------ include/uapi/linux/kvm.h | 1 + 7 files changed, 130 insertions(+), 29 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e3d9e241a46f..b821ff68c169 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -39,7 +39,7 @@ #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 9 +#define KVM_VCPU_MAX_FEATURES 10 #define KVM_VCPU_VALID_FEATURES (BIT(KVM_VCPU_MAX_FEATURES) - 1) #define KVM_REQ_SLEEP \ @@ -80,6 +80,7 @@ extern unsigned int __ro_after_init kvm_host_max_vl[ARM64_VEC_MAX]; DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use); int __init kvm_arm_init_sve(void); +int __init kvm_arm_init_sme(void); u32 __attribute_const__ kvm_target_cpu(void); void kvm_reset_vcpu(struct kvm_vcpu *vcpu); @@ -1060,10 +1061,10 @@ struct kvm_vcpu_arch { size_t __size_ret; \ unsigned int __vcpu_vq; \ \ - if (WARN_ON(!sve_vl_valid((vcpu)->arch.max_vl[ARM64_VEC_SVE]))) { \ + if (WARN_ON(!sve_vl_valid(vcpu_max_vl(vcpu)))) { \ __size_ret = 0; \ } else { \ - __vcpu_vq = vcpu_sve_max_vq(vcpu); \ + __vcpu_vq = vcpu_max_vq(vcpu); \ __size_ret = SVE_SIG_REGS_SIZE(__vcpu_vq); \ } \ \ @@ -1076,7 +1077,7 @@ struct kvm_vcpu_arch { size_t __size_ret; \ unsigned int __vcpu_vq; \ \ - if (WARN_ON(!sve_vl_valid((vcpu)->arch.max_vl[ARM64_VEC_SME]))) { \ + if (WARN_ON(!sve_vl_valid(vcpu_sme_max_vl(vcpu)))) { \ __size_ret = 0; \ } else { \ __vcpu_vq = vcpu_sme_max_vq(vcpu); \ diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 2ecdd29c0677..71bfdd05de19 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -106,6 +106,7 @@ struct kvm_regs { #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */ #define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */ #define KVM_ARM_VCPU_HAS_EL2_E2H0 8 /* Limit NV support to E2H RES0 */ +#define KVM_ARM_VCPU_SME 9 /* enable SME for this CPU */ /* * An alias for _SVE since we finalize VL configuration for both SVE and SME diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 68fec8c95fee..ab707f608456 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -386,6 +386,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) case KVM_CAP_ARM_SVE: r = system_supports_sve(); break; + case KVM_CAP_ARM_SME: + r = system_supports_sme(); + break; case KVM_CAP_ARM_PTRAUTH_ADDRESS: case KVM_CAP_ARM_PTRAUTH_GENERIC: r = kvm_has_full_ptr_auth(); @@ -1424,6 +1427,9 @@ static unsigned long system_supported_vcpu_features(void) if (!system_supports_sve()) clear_bit(KVM_ARM_VCPU_SVE, &features); + if (!system_supports_sme()) + clear_bit(KVM_ARM_VCPU_SME, &features); + if (!kvm_has_full_ptr_auth()) { clear_bit(KVM_ARM_VCPU_PTRAUTH_ADDRESS, &features); clear_bit(KVM_ARM_VCPU_PTRAUTH_GENERIC, &features); @@ -2832,6 +2838,10 @@ static __init int kvm_arm_init(void) if (err) return err; + err = kvm_arm_init_sme(); + if (err) + return err; + err = kvm_arm_vmid_alloc_init(); if (err) { kvm_err("Failed to initialize VMID allocator.\n"); diff --git a/arch/arm64/kvm/hyp/nvhe/pkvm.c b/arch/arm64/kvm/hyp/nvhe/pkvm.c index 57b6d81006ae..8f75f517ca49 100644 --- a/arch/arm64/kvm/hyp/nvhe/pkvm.c +++ b/arch/arm64/kvm/hyp/nvhe/pkvm.c @@ -148,10 +148,6 @@ static int pkvm_check_pvm_cpu_features(struct kvm_vcpu *vcpu) !kvm_has_feat(kvm, ID_AA64PFR0_EL1, AdvSIMD, IMP)) return -EINVAL; - /* No SME support in KVM right now. Check to catch if it changes. */ - if (kvm_has_feat(kvm, ID_AA64PFR1_EL1, SME, IMP)) - return -EINVAL; - return 0; } @@ -362,6 +358,11 @@ static void pkvm_init_features_from_host(struct pkvm_hyp_vm *hyp_vm, const struc kvm->arch.flags |= host_arch_flags & BIT(KVM_ARCH_FLAG_GUEST_HAS_SVE); } + if (kvm_pvm_ext_allowed(KVM_CAP_ARM_SME)) { + set_bit(KVM_ARM_VCPU_SME, allowed_features); + kvm->arch.flags |= host_arch_flags & BIT(KVM_ARCH_FLAG_GUEST_HAS_SME); + } + bitmap_and(kvm->arch.vcpu_features, host_kvm->arch.vcpu_features, allowed_features, KVM_VCPU_MAX_FEATURES); } @@ -398,11 +399,12 @@ static void init_pkvm_hyp_vm(struct kvm *host_kvm, struct pkvm_hyp_vm *hyp_vm, pkvm_init_features_from_host(hyp_vm, host_kvm); } -static void pkvm_vcpu_init_sve(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu *host_vcpu) +static void pkvm_vcpu_init_vec(struct pkvm_hyp_vcpu *hyp_vcpu, struct kvm_vcpu *host_vcpu) { struct kvm_vcpu *vcpu = &hyp_vcpu->vcpu; - if (!vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) + if (!vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE) && + !vcpu_has_feature(vcpu, KVM_ARM_VCPU_SME)) vcpu_clear_flag(vcpu, VCPU_VEC_FINALIZED); } @@ -432,7 +434,7 @@ static int init_pkvm_hyp_vcpu(struct pkvm_hyp_vcpu *hyp_vcpu, if (ret) goto done; - pkvm_vcpu_init_sve(hyp_vcpu, host_vcpu); + pkvm_vcpu_init_vec(hyp_vcpu, host_vcpu); done: if (ret) unpin_host_vcpu(host_vcpu); diff --git a/arch/arm64/kvm/hyp/nvhe/sys_regs.c b/arch/arm64/kvm/hyp/nvhe/sys_regs.c index 1ddd9ed3cbb3..eed177dfcb96 100644 --- a/arch/arm64/kvm/hyp/nvhe/sys_regs.c +++ b/arch/arm64/kvm/hyp/nvhe/sys_regs.c @@ -66,6 +66,11 @@ static bool vm_has_ptrauth(const struct kvm *kvm) kvm_vcpu_has_feature(kvm, KVM_ARM_VCPU_PTRAUTH_GENERIC); } +static bool vm_has_sme(const struct kvm *kvm) +{ + return system_supports_sme() && kvm_vcpu_has_feature(kvm, KVM_ARM_VCPU_SME); +} + static bool vm_has_sve(const struct kvm *kvm) { return system_supports_sve() && kvm_vcpu_has_feature(kvm, KVM_ARM_VCPU_SVE); @@ -102,6 +107,7 @@ static const struct pvm_ftr_bits pvmid_aa64pfr0[] = { }; static const struct pvm_ftr_bits pvmid_aa64pfr1[] = { + MAX_FEAT_FUNC(ID_AA64PFR1_EL1, SME, SME2, vm_has_sme), MAX_FEAT(ID_AA64PFR1_EL1, BT, IMP), MAX_FEAT(ID_AA64PFR1_EL1, SSBS, SSBS2), MAX_FEAT_ENUM(ID_AA64PFR1_EL1, MTE_frac, NI), diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 052a1a452f89..069bc9ab88d5 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -76,6 +76,34 @@ int __init kvm_arm_init_sve(void) return 0; } +int __init kvm_arm_init_sme(void) +{ + if (system_supports_sme()) { + kvm_max_vl[ARM64_VEC_SME] = sme_max_virtualisable_vl(); + kvm_host_max_vl[ARM64_VEC_SME] = sme_max_vl(); + kvm_nvhe_sym(kvm_host_max_vl[ARM64_VEC_SME]) = kvm_host_max_vl[ARM64_VEC_SME]; + + /* + * The get_sve_reg()/set_sve_reg() ioctl interface will need + * to be extended with multiple register slice support in + * order to support vector lengths greater than + * VL_ARCH_MAX: + */ + if (WARN_ON(kvm_max_vl[ARM64_VEC_SME] > VL_ARCH_MAX)) + kvm_max_vl[ARM64_VEC_SME] = VL_ARCH_MAX; + + /* + * Don't even try to make use of vector lengths that + * aren't available on all CPUs, for now: + */ + if (kvm_max_vl[ARM64_VEC_SME] < sme_max_vl()) + pr_warn("KVM: SME vector length for guests limited to %u bytes\n", + kvm_max_vl[ARM64_VEC_SME]); + } + + return 0; +} + static void kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu) { vcpu->arch.max_vl[ARM64_VEC_SVE] = kvm_max_vl[ARM64_VEC_SVE]; @@ -88,42 +116,86 @@ static void kvm_vcpu_enable_sve(struct kvm_vcpu *vcpu) set_bit(KVM_ARCH_FLAG_GUEST_HAS_SVE, &vcpu->kvm->arch.flags); } +static void kvm_vcpu_enable_sme(struct kvm_vcpu *vcpu) +{ + vcpu->arch.max_vl[ARM64_VEC_SME] = kvm_max_vl[ARM64_VEC_SME]; + + /* + * Userspace can still customize the vector lengths by writing + * KVM_REG_ARM64_SME_VLS. Allocation is deferred until + * kvm_arm_vcpu_finalize(), which freezes the configuration. + */ + set_bit(KVM_ARCH_FLAG_GUEST_HAS_SME, &vcpu->kvm->arch.flags); +} + /* - * Finalize vcpu's maximum SVE vector length, allocating - * vcpu->arch.sve_state as necessary. + * Finalize vcpu's maximum vector lengths, allocating + * vcpu->arch.sve_state and vcpu->arch.sme_state as necessary. */ static int kvm_vcpu_finalize_vec(struct kvm_vcpu *vcpu) { - void *buf; + void *sve_state, *sme_state; unsigned int vl; - size_t reg_sz; int ret; - vl = vcpu->arch.max_vl[ARM64_VEC_SVE]; - /* * Responsibility for these properties is shared between * kvm_arm_init_sve(), kvm_vcpu_enable_sve() and * set_sve_vls(). Double-check here just to be sure: */ - if (WARN_ON(!sve_vl_valid(vl) || vl > sve_max_virtualisable_vl() || - vl > VL_ARCH_MAX)) - return -EIO; + if (vcpu_has_sve(vcpu)) { + vl = vcpu->arch.max_vl[ARM64_VEC_SVE]; + if (WARN_ON(!sve_vl_valid(vl) || + vl > sve_max_virtualisable_vl() || + vl > VL_ARCH_MAX)) + return -EIO; + } - reg_sz = vcpu_sve_state_size(vcpu); - buf = kzalloc(reg_sz, GFP_KERNEL_ACCOUNT); - if (!buf) + /* Similarly for SME */ + if (vcpu_has_sme(vcpu)) { + vl = vcpu->arch.max_vl[ARM64_VEC_SME]; + if (WARN_ON(!sve_vl_valid(vl) || + vl > sme_max_virtualisable_vl() || + vl > VL_ARCH_MAX)) + return -EIO; + } + + sve_state = kzalloc(vcpu_sve_state_size(vcpu), GFP_KERNEL_ACCOUNT); + if (!sve_state) return -ENOMEM; - ret = kvm_share_hyp(buf, buf + reg_sz); - if (ret) { - kfree(buf); - return ret; + ret = kvm_share_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu)); + if (ret) + goto err_sve_alloc; + + if (vcpu_has_sme(vcpu)) { + sme_state = kzalloc(vcpu_sme_state_size(vcpu), + GFP_KERNEL_ACCOUNT); + if (!sme_state) { + ret = -ENOMEM; + goto err_sve_map; + } + + ret = kvm_share_hyp(sme_state, + sme_state + vcpu_sme_state_size(vcpu)); + if (ret) + goto err_sme_alloc; + } else { + sme_state = NULL; } - - vcpu->arch.sve_state = buf; + + vcpu->arch.sve_state = sve_state; + vcpu->arch.sme_state = sme_state; vcpu_set_flag(vcpu, VCPU_VEC_FINALIZED); return 0; + +err_sme_alloc: + kfree(sme_state); +err_sve_map: + kvm_unshare_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu)); +err_sve_alloc: + kfree(sve_state); + return ret; } int kvm_arm_vcpu_finalize(struct kvm_vcpu *vcpu, int feature) @@ -153,11 +225,15 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu) void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) { void *sve_state = vcpu->arch.sve_state; + void *sme_state = vcpu->arch.sme_state; kvm_unshare_hyp(vcpu, vcpu + 1); if (sve_state) kvm_unshare_hyp(sve_state, sve_state + vcpu_sve_state_size(vcpu)); kfree(sve_state); + if (sme_state) + kvm_unshare_hyp(sme_state, sme_state + vcpu_sme_state_size(vcpu)); + kfree(sme_state); kfree(vcpu->arch.ccsidr); } @@ -165,6 +241,8 @@ static void kvm_vcpu_reset_vec(struct kvm_vcpu *vcpu) { if (vcpu_has_sve(vcpu)) memset(vcpu->arch.sve_state, 0, vcpu_sve_state_size(vcpu)); + if (vcpu_has_sme(vcpu)) + memset(vcpu->arch.sme_state, 0, vcpu_sme_state_size(vcpu)); } /** @@ -204,6 +282,8 @@ void kvm_reset_vcpu(struct kvm_vcpu *vcpu) if (!kvm_arm_vcpu_vec_finalized(vcpu)) { if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_SVE)) kvm_vcpu_enable_sve(vcpu); + if (vcpu_has_feature(vcpu, KVM_ARM_VCPU_SME)) + kvm_vcpu_enable_sme(vcpu); } else { kvm_vcpu_reset_vec(vcpu); } diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h index b6ae8ad8934b..7e1a03b4e0d9 100644 --- a/include/uapi/linux/kvm.h +++ b/include/uapi/linux/kvm.h @@ -930,6 +930,7 @@ struct kvm_enable_cap { #define KVM_CAP_X86_APIC_BUS_CYCLES_NS 237 #define KVM_CAP_X86_GUEST_MODE 238 #define KVM_CAP_ARM_WRITABLE_IMP_ID_REGS 239 +#define KVM_CAP_ARM_SME 240 struct kvm_irq_routing_irqchip { __u32 irqchip; From patchwork Thu Apr 17 00:25:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054726 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2CC7D22E00A; Thu, 17 Apr 2025 00:27:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849651; cv=none; b=mpTHFDRvPyySss2lsO9i3JLtQyLmsX+PtJ7CQ/dzEW+Ms1U0uQ3YY7A4cRd1rZo/hoyhWhxJPc/T2rPxdsDrtu3EOOf7CLfPl86cRdFWDrakAJV4WNHsEJiMPX97ngc+hSetTLycepudTZGRByaR4WDH78a7jwIwhgZW13kFNO0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849651; c=relaxed/simple; bh=8wKbF7ujl1hlDs76hV4PzHiP9Uo97M2NjFijaiGNkbY=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=PXoHbWiQHebRLnbeAnqFEG0Cv8II/MZpwz5zSqsnzqhs0pv7qowy6bj2DdoTP/4NLjpQBGSU7u+G0CppOpPT6Th3z2SsMOG5X8U9QoD3vnqlA+RR2L7f15swW+fmIcq2kFxV7Vg1gnbrfT5RVqXD4R7ZnyYMSWMg8cFfN1tUcRo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=khJ9RHw5; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="khJ9RHw5" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 605D2C4CEEA; Thu, 17 Apr 2025 00:27:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849650; bh=8wKbF7ujl1hlDs76hV4PzHiP9Uo97M2NjFijaiGNkbY=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=khJ9RHw5mLTikw30MT0YwWM39KMMLBLBT6Krpr0l3eyfDmvICrx4DZuKChTJkS0VC lb/WPUBX1dt85xLJyLEZF5n5qNGHauyTaJhB0jdMvfpyATUeFDQJqj6vRTgoNGuFlr hLSnA/3A39QZBkq7Mz1qQm/5h7gk6H8NmUxCRqyIrniKuQyc2hlkCsrEShLXSxyoj2 7Oq6LaZXVwWTC0tNGKjkbGhjlEr7ztDsTTZ4HjsGTnyOmMj/KEOUjSYSS5aaWlnYK0 OjrIZ39GMQaoDOVLZs44Pv2jJwjNonqxK9WSLxnYsX5zmpMylbJ7rsT1gIniSz+1HH vqrh8W0LDWTiQ== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:31 +0100 Subject: [PATCH v5 27/28] KVM: arm64: selftests: Add SME system registers to get-reg-list Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-27-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=1636; i=broonie@kernel.org; h=from:subject:message-id; bh=8wKbF7ujl1hlDs76hV4PzHiP9Uo97M2NjFijaiGNkbY=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEqEN0PBi+5sh+Ufd6YUKPCQBIGQCBFHgZHOLVxt MwC5Lq+JATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKhAAKCRAk1otyXVSH0IbxB/ 9kt29k0vrzqc7H+CKPAvHqHz2pM9oN8EKRlhOKwQxgaYKF86p42RWIW8AHBlgiRWuZBZP5CBQVs3av nd2ICQPqXMvFnHPCcfkAgtEVUJ+wMhvmX2+aAPYaCSc6L5WMZqTYZ9D6oy2mXOC2sqF6nCXTvICFM9 T+QU8ahW5T1Exv31Drzd9sumQoIzb8KkWXGxRkay6FJsrgbbJjIw6plFpvlRsbiTYtkuP0+Zgmre2u Ne4dKicSIPFfHHvH4gz9SXuOvhAwmxLhSXVPodE/Lq/P0IW+FurG36AX2rbVIf06UvAWtzQL0BV9f6 o2nzjXBnLBkjIPYFFikRXU14ERVMDC X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB SME adds a number of new system registers, update get-reg-list to check for them based on the visibility of SME. Signed-off-by: Mark Brown --- tools/testing/selftests/kvm/arm64/get-reg-list.c | 32 +++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) diff --git a/tools/testing/selftests/kvm/arm64/get-reg-list.c b/tools/testing/selftests/kvm/arm64/get-reg-list.c index d01798b6b3b4..920784aa2838 100644 --- a/tools/testing/selftests/kvm/arm64/get-reg-list.c +++ b/tools/testing/selftests/kvm/arm64/get-reg-list.c @@ -23,6 +23,18 @@ struct feature_id_reg { }; static struct feature_id_reg feat_id_regs[] = { + { + ARM64_SYS_REG(3, 0, 1, 2, 4), /* SMPRI_EL1 */ + ARM64_SYS_REG(3, 0, 0, 4, 1), /* ID_AA64PFR1_EL1 */ + 24, + 1 + }, + { + ARM64_SYS_REG(3, 0, 1, 2, 6), /* SMCR_EL1 */ + ARM64_SYS_REG(3, 0, 0, 4, 1), /* ID_AA64PFR1_EL1 */ + 24, + 1 + }, { ARM64_SYS_REG(3, 0, 2, 0, 3), /* TCR2_EL1 */ ARM64_SYS_REG(3, 0, 0, 7, 3), /* ID_AA64MMFR3_EL1 */ @@ -52,7 +64,25 @@ static struct feature_id_reg feat_id_regs[] = { ARM64_SYS_REG(3, 0, 0, 7, 3), /* ID_AA64MMFR3_EL1 */ 16, 1 - } + }, + { + ARM64_SYS_REG(3, 1, 0, 0, 6), /* SMIDR_EL1 */ + ARM64_SYS_REG(3, 0, 0, 4, 1), /* ID_AA64PFR1_EL1 */ + 24, + 1 + }, + { + ARM64_SYS_REG(3, 3, 4, 2, 2), /* SVCR */ + ARM64_SYS_REG(3, 0, 0, 4, 1), /* ID_AA64PFR1_EL1 */ + 24, + 1 + }, + { + ARM64_SYS_REG(3, 3, 13, 0, 5), /* TPIDR2_EL0 */ + ARM64_SYS_REG(3, 0, 0, 4, 1), /* ID_AA64PFR1_EL1 */ + 24, + 1 + }, }; bool filter_reg(__u64 reg) From patchwork Thu Apr 17 00:25:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Brown X-Patchwork-Id: 14054727 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5F21922E00A; Thu, 17 Apr 2025 00:27:34 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849654; cv=none; b=idplwAkvxYVR2DhzCVyhDHSIKQZOgi7mXJRtql291clwR/fpznirTTm0sNUgZvn/n8MLtHLyqFq1TygRHciD9EIBTOOaD++DftM/pTjl6j9LYG4fdX788Y/DmPhMT6JHhcwLYN0gAzbJARBqh0kAQJSpnNc6YQt37Uck0Cj+JYQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1744849654; c=relaxed/simple; bh=c22FTsDBl8on7JkOPsVEpDLeW97hqa6VyOk90f5/da4=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=K5APvQ5iNSLXyy/PTM7gycOZHvHVTJfNXU3QUI1TfXDymJFMCqfYxs6jIR+wbPy59gFsZEVVjuXIotaDiPyDbs+hwELghW2r1+UXQO32CrLt+l7xHeSE9Q+di/3sfxevhbpUWQaIRgBmDh+1qn1Zk6qPsPbmuD/XyVbSvIz9TMY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=pdRpeEn7; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="pdRpeEn7" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0A729C4CEED; Thu, 17 Apr 2025 00:27:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1744849654; bh=c22FTsDBl8on7JkOPsVEpDLeW97hqa6VyOk90f5/da4=; h=From:Date:Subject:References:In-Reply-To:To:Cc:From; b=pdRpeEn7CrcVPx+kUfDsoI5/OAvjUj7WFrbF4LkM3BYxCHjzqplZNqkJMkrUaTzuV c7my1t3zDLG8eWNQTgNcya0XNY9zCVVJB2htQNW5WV/9RzpmC86gmY4X1sjpCsDPx5 ZisOkzq8p2/j69nPr1xnLvRJicSLEIHSGDVvdd3hZNuKn44zioZjnZzuZT+Qr/Pu1H xbGnvNP3pqhkZHUA6p3+i7j11Sp8t+8f5Te4Hs6CWoNgLg4hvgrJE1HdN9li+qKPVV frAjLJt8WYVZZLAlF4fGIROL7u0mf+Cdt9+gEQlK28A6q2/45GUcRa5Y6ORTcFrzye G0p7B2RoL+IhA== From: Mark Brown Date: Thu, 17 Apr 2025 01:25:32 +0100 Subject: [PATCH v5 28/28] KVM: arm64: selftests: Add SME to set_id_regs test Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20250417-kvm-arm64-sme-v5-28-f469a2d5f574@kernel.org> References: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> In-Reply-To: <20250417-kvm-arm64-sme-v5-0-f469a2d5f574@kernel.org> To: Marc Zyngier , Oliver Upton , Joey Gouly , Catalin Marinas , Suzuki K Poulose , Will Deacon , Paolo Bonzini , Jonathan Corbet , Shuah Khan Cc: Dave Martin , Fuad Tabba , Mark Rutland , linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, kvm@vger.kernel.org, linux-doc@vger.kernel.org, linux-kselftest@vger.kernel.org, Mark Brown X-Mailer: b4 0.15-dev-c25d1 X-Developer-Signature: v=1; a=openpgp-sha256; l=3711; i=broonie@kernel.org; h=from:subject:message-id; bh=c22FTsDBl8on7JkOPsVEpDLeW97hqa6VyOk90f5/da4=; b=owEBbQGS/pANAwAKASTWi3JdVIfQAcsmYgBoAEqEPmF1UWC0EBNqc6FOONN5AgLp5u4ISmCdeQ3d +1tXd0qJATMEAAEKAB0WIQSt5miqZ1cYtZ/in+ok1otyXVSH0AUCaABKhAAKCRAk1otyXVSH0Gh/B/ 9HAFpSzIdzlDrDNhb9algwishiTaXUKWuASjSDPBm1wCg6xVIGVPOWp9/N83PHSqzHCeYjpo4X4K0u gyHfVWPM9eDHZ8m9XrAzvEVRLv7Zr+HE3obFad8s3tkORdQfil/0/vGWXhpXQwXUiMyUHJM8f9QcKo sSs5j37wkt+KRS4anKOc8HrgvRR7MaxEnREFXI7II9vJmqNVAmSuvZC4mYCeHR3uZs4HOudiOBLt4N C2RJwtqS9Q+PQliixkCFALO12w7K5QPcAfdORLnmXwdjDIT7OaeIRO91nDZfQpdXkKHVgn8d2W0G56 s1Ln1aXjPH01uX1yqB7S2fjKUu+5Ej X-Developer-Key: i=broonie@kernel.org; a=openpgp; fpr=3F2568AAC26998F9E813A1C5C3F436CA30F5D8EB Add coverage of the SME ID registers to set_id_regs, ID_AA64PFR1_EL1.SME becomes writable and we add ID_AA64SMFR_EL1 and it's subfields. Signed-off-by: Mark Brown --- tools/testing/selftests/kvm/arm64/set_id_regs.c | 30 +++++++++++++++++++++++-- 1 file changed, 28 insertions(+), 2 deletions(-) diff --git a/tools/testing/selftests/kvm/arm64/set_id_regs.c b/tools/testing/selftests/kvm/arm64/set_id_regs.c index 322b9d3b0125..bcb00bb6be2d 100644 --- a/tools/testing/selftests/kvm/arm64/set_id_regs.c +++ b/tools/testing/selftests/kvm/arm64/set_id_regs.c @@ -138,6 +138,7 @@ static const struct reg_ftr_bits ftr_id_aa64pfr0_el1[] = { static const struct reg_ftr_bits ftr_id_aa64pfr1_el1[] = { REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64PFR1_EL1, CSV2_frac, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64PFR1_EL1, SME, 0), REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64PFR1_EL1, SSBS, ID_AA64PFR1_EL1_SSBS_NI), REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64PFR1_EL1, BT, 0), REG_FTR_END, @@ -185,6 +186,28 @@ static const struct reg_ftr_bits ftr_id_aa64mmfr2_el1[] = { REG_FTR_END, }; +static const struct reg_ftr_bits ftr_id_aa64smfr0_el1[] = { + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, FA64, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, LUTv2, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, SMEver, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, I16I64, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, F64F64, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, I16I32, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, B16B16, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, F16F16, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, F8F16, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, F8F32, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, I8I32, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, F16F32, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, B16F32, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, BI32I32, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, F32F32, 0) +, REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, SF8FMA, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, SF8DP4, 0), + REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64SMFR0_EL1, SF8DP2, 0), + REG_FTR_END, +}; + static const struct reg_ftr_bits ftr_id_aa64zfr0_el1[] = { REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64ZFR0_EL1, F64MM, 0), REG_FTR_BITS(FTR_LOWER_SAFE, ID_AA64ZFR0_EL1, F32MM, 0), @@ -215,6 +238,7 @@ static struct test_feature_reg test_regs[] = { TEST_REG(SYS_ID_AA64MMFR0_EL1, ftr_id_aa64mmfr0_el1), TEST_REG(SYS_ID_AA64MMFR1_EL1, ftr_id_aa64mmfr1_el1), TEST_REG(SYS_ID_AA64MMFR2_EL1, ftr_id_aa64mmfr2_el1), + TEST_REG(SYS_ID_AA64SMFR0_EL1, ftr_id_aa64smfr0_el1), TEST_REG(SYS_ID_AA64ZFR0_EL1, ftr_id_aa64zfr0_el1), }; @@ -231,6 +255,7 @@ static void guest_code(void) GUEST_REG_SYNC(SYS_ID_AA64MMFR0_EL1); GUEST_REG_SYNC(SYS_ID_AA64MMFR1_EL1); GUEST_REG_SYNC(SYS_ID_AA64MMFR2_EL1); + GUEST_REG_SYNC(SYS_ID_AA64SMFR0_EL1); GUEST_REG_SYNC(SYS_ID_AA64ZFR0_EL1); GUEST_REG_SYNC(SYS_CTR_EL0); GUEST_REG_SYNC(SYS_MIDR_EL1); @@ -700,8 +725,9 @@ int main(void) ARRAY_SIZE(ftr_id_aa64isar2_el1) + ARRAY_SIZE(ftr_id_aa64pfr0_el1) + ARRAY_SIZE(ftr_id_aa64pfr1_el1) + ARRAY_SIZE(ftr_id_aa64mmfr0_el1) + ARRAY_SIZE(ftr_id_aa64mmfr1_el1) + ARRAY_SIZE(ftr_id_aa64mmfr2_el1) + - ARRAY_SIZE(ftr_id_aa64zfr0_el1) - ARRAY_SIZE(test_regs) + 3 + - MPAM_IDREG_TEST; + + ARRAY_SIZE(ftr_id_aa64zfr0_el1) + ARRAY_SIZE(ftr_id_aa64smfr0_el1) - + ARRAY_SIZE(test_regs) + 3 + MPAM_IDREG_TEST; ksft_set_plan(test_cnt);