From patchwork Wed Nov 17 15:38:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandru Elisei X-Patchwork-Id: 12692961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC054C433EF for ; Wed, 17 Nov 2021 15:57:34 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 74B9560EE4 for ; Wed, 17 Nov 2021 15:57:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 74B9560EE4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CA94M1MBZl2WDgIB1xa14fAD/cL9C9UOs4EggHfdEb0=; b=Kq63tS0Q4wJODf NE3v40S11YqMFbuuU0IkIGWSr3bzucW2VAxcEADq4PlsrCr2TNzjinXAiEmOIJcQsySUMeLE0Gzq4 f8w33KBr854dhMYIwdKyOGUI5eVJVG/0+3hX/0IrvHvE0ml3AkocL202I6W+9f58UENKpPeKTctrd hx543miYVkPtFKjWhSU28u1dT+W08p92SpldHGXmGZ5KqjO9/E27LEeYUgPcHWjcPe1CXehvN2TpY To/a0tOmfh2QAB+vrg3u9J03KSn01l0zBXkOlMXTpCk6UVNnVBMfs+k65G1kQkSjKaUR45BH0Bz5i wSW1OI56thvrl71TU/XA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnNIA-005Yu1-Ht; Wed, 17 Nov 2021 15:56:11 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mnN0U-005QD4-O0 for linux-arm-kernel@lists.infradead.org; Wed, 17 Nov 2021 15:37:56 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 61A39ED1; Wed, 17 Nov 2021 07:37:54 -0800 (PST) Received: from monolith.localdoman (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 554203F5A1; Wed, 17 Nov 2021 07:37:52 -0800 (PST) From: Alexandru Elisei To: maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, will@kernel.org, mark.rutland@arm.com Subject: [RFC PATCH v5 29/38] KVM: arm64: nVHE: Context switch SPE state if VCPU has SPE Date: Wed, 17 Nov 2021 15:38:33 +0000 Message-Id: <20211117153842.302159-30-alexandru.elisei@arm.com> X-Mailer: git-send-email 2.33.1 In-Reply-To: <20211117153842.302159-1-alexandru.elisei@arm.com> References: <20211117153842.302159-1-alexandru.elisei@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211117_073754_929568_FA4750D0 X-CRM114-Status: GOOD ( 22.49 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org For non-VHE systems, make the SPE register state part of the context that is saved and restored at each world switch. The SPE buffer management interrupt will be handled in a later patch. Signed-off-by: Alexandru Elisei --- arch/arm64/include/asm/kvm_hyp.h | 19 ++++++ arch/arm64/kvm/hyp/include/hyp/spe-sr.h | 32 +++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 1 + arch/arm64/kvm/hyp/nvhe/debug-sr.c | 6 +- arch/arm64/kvm/hyp/nvhe/spe-sr.c | 87 +++++++++++++++++++++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 29 +++++++-- 6 files changed, 165 insertions(+), 9 deletions(-) create mode 100644 arch/arm64/kvm/hyp/include/hyp/spe-sr.h create mode 100644 arch/arm64/kvm/hyp/nvhe/spe-sr.c diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 0a5ff4361069..08f020912103 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -88,6 +88,25 @@ void __debug_save_host_buffers_nvhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt); void __debug_restore_host_buffers_nvhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt); +#ifdef CONFIG_KVM_ARM_SPE +void __spe_save_host_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *host_ctxt); +void __spe_save_guest_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *guest_ctxt); +void __spe_restore_host_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *host_ctxt); +void __spe_restore_guest_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *guest_ctxt); +#else +static inline void __spe_save_host_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *host_ctxt) {} +static inline void __spe_save_guest_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *guest_ctxt) {} +static inline void __spe_restore_host_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *host_ctxt) {} +static inline void __spe_restore_guest_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *guest_ctxt) {} +#endif #endif void __fpsimd_save_state(struct user_fpsimd_state *fp_regs); diff --git a/arch/arm64/kvm/hyp/include/hyp/spe-sr.h b/arch/arm64/kvm/hyp/include/hyp/spe-sr.h new file mode 100644 index 000000000000..d5f8f3ffc7d4 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/hyp/spe-sr.h @@ -0,0 +1,32 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2021 - ARM Ltd + * Author: Alexandru Elisei + */ + +#ifndef __ARM64_KVM_HYP_SPE_SR_H__ +#define __ARM64_KVM_HYP_SPE_SR_H__ + +#include + +#include + +static inline void __spe_save_common_state(struct kvm_cpu_context *ctxt) +{ + ctxt_sys_reg(ctxt, PMSICR_EL1) = read_sysreg_s(SYS_PMSICR_EL1); + ctxt_sys_reg(ctxt, PMSIRR_EL1) = read_sysreg_s(SYS_PMSIRR_EL1); + ctxt_sys_reg(ctxt, PMSFCR_EL1) = read_sysreg_s(SYS_PMSFCR_EL1); + ctxt_sys_reg(ctxt, PMSEVFR_EL1) = read_sysreg_s(SYS_PMSEVFR_EL1); + ctxt_sys_reg(ctxt, PMSLATFR_EL1) = read_sysreg_s(SYS_PMSLATFR_EL1); +} + +static inline void __spe_restore_common_state(struct kvm_cpu_context *ctxt) +{ + write_sysreg_s(ctxt_sys_reg(ctxt, PMSICR_EL1), SYS_PMSICR_EL1); + write_sysreg_s(ctxt_sys_reg(ctxt, PMSIRR_EL1), SYS_PMSIRR_EL1); + write_sysreg_s(ctxt_sys_reg(ctxt, PMSFCR_EL1), SYS_PMSFCR_EL1); + write_sysreg_s(ctxt_sys_reg(ctxt, PMSEVFR_EL1), SYS_PMSEVFR_EL1); + write_sysreg_s(ctxt_sys_reg(ctxt, PMSLATFR_EL1), SYS_PMSLATFR_EL1); +} + +#endif /* __ARM64_KVM_HYP_SPE_SR_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index c3c11974fa3b..06e66945eaab 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -17,6 +17,7 @@ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o \ cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o +obj-$(CONFIG_KVM_ARM_SPE) += spe-sr.o obj-y += $(lib-objs) ## diff --git a/arch/arm64/kvm/hyp/nvhe/debug-sr.c b/arch/arm64/kvm/hyp/nvhe/debug-sr.c index adabdcbbd753..02171dcf29c3 100644 --- a/arch/arm64/kvm/hyp/nvhe/debug-sr.c +++ b/arch/arm64/kvm/hyp/nvhe/debug-sr.c @@ -85,7 +85,8 @@ void __debug_save_host_buffers_nvhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt) { /* Disable and flush SPE data generation */ - if (vcpu->arch.flags & KVM_ARM64_DEBUG_SAVE_SPE_BUFFER) + if (!kvm_vcpu_has_spe(vcpu) && + vcpu->arch.flags & KVM_ARM64_DEBUG_SAVE_SPE_BUFFER) __debug_save_spe(__ctxt_sys_reg(host_ctxt, PMSCR_EL1)); /* Disable and flush Self-Hosted Trace generation */ if (vcpu->arch.flags & KVM_ARM64_DEBUG_SAVE_TRBE_BUFFER) @@ -100,7 +101,8 @@ void __debug_switch_to_guest(struct kvm_vcpu *vcpu) void __debug_restore_host_buffers_nvhe(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt) { - if (vcpu->arch.flags & KVM_ARM64_DEBUG_SAVE_SPE_BUFFER) + if (!kvm_vcpu_has_spe(vcpu) && + vcpu->arch.flags & KVM_ARM64_DEBUG_SAVE_SPE_BUFFER) __debug_restore_spe(ctxt_sys_reg(host_ctxt, PMSCR_EL1)); if (vcpu->arch.flags & KVM_ARM64_DEBUG_SAVE_TRBE_BUFFER) __debug_restore_trace(vcpu->arch.host_debug_state.trfcr_el1); diff --git a/arch/arm64/kvm/hyp/nvhe/spe-sr.c b/arch/arm64/kvm/hyp/nvhe/spe-sr.c new file mode 100644 index 000000000000..46e47c9fd08f --- /dev/null +++ b/arch/arm64/kvm/hyp/nvhe/spe-sr.c @@ -0,0 +1,87 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2021 - ARM Ltd + * Author: Alexandru Elisei + */ + +#include + +#include + +#include + +/* + * The owning exception level remains unchange from EL1 during the world switch, + * which means that profiling is disabled for as long as we execute at EL2. KVM + * does not need to explicitely disable profiling, like it does when the VCPU + * does not have SPE and we change buffer owning exception level, nor does it + * need to do any synchronization around sysreg save/restore. + */ + +void __spe_save_host_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *host_ctxt) +{ + u64 pmblimitr; + + pmblimitr = read_sysreg_s(SYS_PMBLIMITR_EL1); + if (pmblimitr & BIT(SYS_PMBLIMITR_EL1_E_SHIFT)) { + psb_csync(); + dsb(nsh); + /* + * The buffer performs indirect writes to system registers, a + * context synchronization event is needed before the new + * PMBPTR_EL1 value is visible to subsequent direct reads. + */ + isb(); + } + + ctxt_sys_reg(host_ctxt, PMBPTR_EL1) = read_sysreg_s(SYS_PMBPTR_EL1); + ctxt_sys_reg(host_ctxt, PMBSR_EL1) = read_sysreg_s(SYS_PMBSR_EL1); + ctxt_sys_reg(host_ctxt, PMBLIMITR_EL1) = pmblimitr; + ctxt_sys_reg(host_ctxt, PMSCR_EL1) = read_sysreg_s(SYS_PMSCR_EL1); + ctxt_sys_reg(host_ctxt, PMSCR_EL2) = read_sysreg_el2(SYS_PMSCR); + + __spe_save_common_state(host_ctxt); +} + +void __spe_save_guest_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *guest_ctxt) +{ + if (read_sysreg_s(SYS_PMBLIMITR_EL1) & BIT(SYS_PMBLIMITR_EL1_E_SHIFT)) { + psb_csync(); + dsb(nsh); + /* Ensure hardware updates to PMBPTR_EL1 are visible. */ + isb(); + } + + ctxt_sys_reg(guest_ctxt, PMBPTR_EL1) = read_sysreg_s(SYS_PMBPTR_EL1); + ctxt_sys_reg(guest_ctxt, PMBSR_EL1) = read_sysreg_s(SYS_PMBSR_EL1); + /* PMBLIMITR_EL1 is updated only on a trapped write. */ + ctxt_sys_reg(guest_ctxt, PMSCR_EL1) = read_sysreg_s(SYS_PMSCR_EL1); + + __spe_save_common_state(guest_ctxt); +} + +void __spe_restore_host_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *host_ctxt) +{ + __spe_restore_common_state(host_ctxt); + + write_sysreg_s(ctxt_sys_reg(host_ctxt, PMBPTR_EL1), SYS_PMBPTR_EL1); + write_sysreg_s(ctxt_sys_reg(host_ctxt, PMBSR_EL1), SYS_PMBSR_EL1); + write_sysreg_s(ctxt_sys_reg(host_ctxt, PMBLIMITR_EL1), SYS_PMBLIMITR_EL1); + write_sysreg_s(ctxt_sys_reg(host_ctxt, PMSCR_EL1), SYS_PMSCR_EL1); + write_sysreg_el2(ctxt_sys_reg(host_ctxt, PMSCR_EL2), SYS_PMSCR); +} + +void __spe_restore_guest_state_nvhe(struct kvm_vcpu *vcpu, + struct kvm_cpu_context *guest_ctxt) +{ + __spe_restore_common_state(guest_ctxt); + + write_sysreg_s(ctxt_sys_reg(guest_ctxt, PMBPTR_EL1), SYS_PMBPTR_EL1); + write_sysreg_s(ctxt_sys_reg(guest_ctxt, PMBSR_EL1), SYS_PMBSR_EL1); + write_sysreg_s(ctxt_sys_reg(guest_ctxt, PMBLIMITR_EL1), SYS_PMBLIMITR_EL1); + write_sysreg_s(ctxt_sys_reg(guest_ctxt, PMSCR_EL1), SYS_PMSCR_EL1); + write_sysreg_el2(0, SYS_PMSCR); +} diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index b6489e244025..d97b56559e50 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -284,12 +284,16 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) __sysreg_save_state_nvhe(host_ctxt); /* - * We must flush and disable the SPE buffer for nVHE, as - * the translation regime(EL1&0) is going to be loaded with - * that of the guest. And we must do this before we change the - * translation regime to EL2 (via MDCR_EL2_E2PB == 0) and - * before we load guest Stage1. + * If the VCPU has the SPE feature bit set, then we save the host's SPE + * context. + * + * Otherwise, we only flush and disable the SPE buffer for nVHE, as the + * translation regime(EL1&0) is going to be loaded with that of the + * guest. And we must do this before we change the translation regime to + * EL2 (via MDCR_EL2_E2PB == 0) and before we load guest Stage1. */ + if (kvm_vcpu_has_spe(vcpu)) + __spe_save_host_state_nvhe(vcpu, host_ctxt); __debug_save_host_buffers_nvhe(vcpu, host_ctxt); __kvm_adjust_pc(vcpu); @@ -309,6 +313,9 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) __load_stage2(mmu, kern_hyp_va(mmu->arch)); __activate_traps(vcpu); + if (kvm_vcpu_has_spe(vcpu)) + __spe_restore_guest_state_nvhe(vcpu, guest_ctxt); + __hyp_vgic_restore_state(vcpu); __timer_enable_traps(vcpu); @@ -326,6 +333,10 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) __sysreg_save_state_nvhe(guest_ctxt); __sysreg32_save_state(vcpu); + + if (kvm_vcpu_has_spe(vcpu)) + __spe_save_guest_state_nvhe(vcpu, guest_ctxt); + __timer_disable_traps(vcpu); __hyp_vgic_save_state(vcpu); @@ -338,10 +349,14 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) __fpsimd_save_fpexc32(vcpu); __debug_switch_to_host(vcpu); + /* - * This must come after restoring the host sysregs, since a non-VHE - * system may enable SPE here and make use of the TTBRs. + * Restoring the host context must come after restoring the host + * sysregs, since a non-VHE system may enable SPE here and make use of + * the TTBRs. */ + if (kvm_vcpu_has_spe(vcpu)) + __spe_restore_host_state_nvhe(vcpu, host_ctxt); __debug_restore_host_buffers_nvhe(vcpu, host_ctxt); if (pmu_switch_needed)