From patchwork Tue Feb 11 17:48:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376111 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 831C014B4 for ; Tue, 11 Feb 2020 17:49:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6134E208C3 for ; Tue, 11 Feb 2020 17:49:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443398; bh=CDfmi4wjycuxqXp0yX9NnOipKJiEr/tNVdnuxQnjckU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=QSpzdHBx1dsa25704iwsWlao000g5VouJA3dqPCq5D0jvZCdrN2gkHPtmQ4p3VlLd DeVlRHlFVrPXf47Mv0uMSrXSGlRzO1YEOdfSGtMoBkYEfJ60TbrcRc8Iw8sLv97nqp DWT6X44O1Ohh/LozMtRSjQ+ozwWtlnTTSQE/JnTI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729410AbgBKRt5 (ORCPT ); Tue, 11 Feb 2020 12:49:57 -0500 Received: from mail.kernel.org ([198.145.29.99]:52246 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728601AbgBKRt5 (ORCPT ); Tue, 11 Feb 2020 12:49:57 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1CB14206CC; Tue, 11 Feb 2020 17:49:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443396; bh=CDfmi4wjycuxqXp0yX9NnOipKJiEr/tNVdnuxQnjckU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AiLkh9rTX0sSunjQf5l8DcZvGYeqfXpGnAP+AYT82nTxYjF654jf1FVFloOgavoxc fdhGEQ3XsYP1Ju0ACwNFv7GdEOAPKKrXvXzkXrwkSlQ6zbmxiZIwkV6KluCrIQezRo 0uHQlTlODNKyYfyplOwlmuWuCnJVrFY7LM+XzI7s= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZfW-004O7k-E0; Tue, 11 Feb 2020 17:49:54 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 01/94] KVM: arm64: Move __load_guest_stage2 to kvm_mmu.h Date: Tue, 11 Feb 2020 17:48:05 +0000 Message-Id: <20200211174938.27809-2-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Having __load_guest_stage2 in kvm_hyp.h is quickly going to trigger a circular include problem. In order to avoid this, let's move it to kvm_mmu.h, where it will be a better fit anyway. In the process, drop the __hyp_text annotation, which doesn't help as the function is marked as __always_inline. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_hyp.h | 18 ------------------ arch/arm64/include/asm/kvm_mmu.h | 17 +++++++++++++++++ 2 files changed, 17 insertions(+), 18 deletions(-) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index a3a6a2ba9a63..4899a5b74fc3 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -10,7 +10,6 @@ #include #include #include -#include #include #define __hyp_text __section(.hyp.text) notrace @@ -81,22 +80,5 @@ void deactivate_traps_vhe_put(void); u64 __guest_enter(struct kvm_vcpu *vcpu, struct kvm_cpu_context *host_ctxt); void __noreturn __hyp_do_panic(unsigned long, ...); -/* - * Must be called from hyp code running at EL2 with an updated VTTBR - * and interrupts disabled. - */ -static __always_inline void __hyp_text __load_guest_stage2(struct kvm *kvm) -{ - write_sysreg(kvm->arch.vtcr, vtcr_el2); - write_sysreg(kvm_get_vttbr(kvm), vttbr_el2); - - /* - * ARM errata 1165522 and 1530923 require the actual execution of the - * above before we can switch to the EL1/EL0 translation regime used by - * the guest. - */ - asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT_VHE)); -} - #endif /* __ARM64_KVM_HYP_H__ */ diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 53d846f1bfe7..ee47f7637f28 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -604,5 +604,22 @@ static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) return kvm_phys_to_vttbr(baddr) | vmid_field | cnp; } +/* + * Must be called from hyp code running at EL2 with an updated VTTBR + * and interrupts disabled. + */ +static __always_inline void __load_guest_stage2(struct kvm *kvm) +{ + write_sysreg(kvm->arch.vtcr, vtcr_el2); + write_sysreg(kvm_get_vttbr(kvm), vttbr_el2); + + /* + * ARM erratum 1165522 requires the actual execution of the above + * before we can switch to the EL1/EL0 translation regime used by + * the guest. + */ + asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT_VHE)); +} + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ From patchwork Tue Feb 11 17:48:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376115 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B756E1395 for ; Tue, 11 Feb 2020 17:49:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9689721739 for ; Tue, 11 Feb 2020 17:49:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443399; bh=v75bdtDTtT7CfBGhCaWYCJiae6UIVY+xjVSdDNEavQs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=rVTf+q1tY6ffEmCOhgtXFQUYpoP8hoBfYubGsBPm8hpnwmWFUniRAXE0fUkNNLWfL g3OhJ4LhRXvey+P4PZhufDIjE5/YX8cZZIHctKxkjAVvYynPyAqjQTGvtfVD0wFRMY /y6nHTuGGCizBX1lpF3C6GY4ZRXZ/5i7/Du1laRc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730031AbgBKRt6 (ORCPT ); Tue, 11 Feb 2020 12:49:58 -0500 Received: from mail.kernel.org ([198.145.29.99]:52274 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728965AbgBKRt6 (ORCPT ); Tue, 11 Feb 2020 12:49:58 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AFB26206D6; Tue, 11 Feb 2020 17:49:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443396; bh=v75bdtDTtT7CfBGhCaWYCJiae6UIVY+xjVSdDNEavQs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jgZzRu0c36DUP47L4nJvqtGczn12oTEMkp9acNacaMpu7m5Ey/sdCLRRSBRpFwxoN Hjq9g40sSaKnQo8alQzAZSSLCOJKF8u05mzICDMsiNKSxwGnL8d1XXKhGK2i4S6dk9 7ozN9ngGelnWdiJjVQKC1K0NlQ5erqz1jfhrOips= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZfX-004O7k-09; Tue, 11 Feb 2020 17:49:55 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 02/94] arm64: Add ARM64_HAS_NESTED_VIRT cpufeature Date: Tue, 11 Feb 2020 17:48:06 +0000 Message-Id: <20200211174938.27809-3-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Add a new ARM64_HAS_NESTED_VIRT feature to indicate that the CPU has the ARMv8.3 nested virtualization capability. This will be used to support nested virtualization in KVM. Signed-off-by: Jintack Lim Signed-off-by: Andre Przywara Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- .../admin-guide/kernel-parameters.txt | 4 +++ arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kernel/cpufeature.c | 26 +++++++++++++++++++ 4 files changed, 33 insertions(+), 1 deletion(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index dbc22d684627..cb61ace641f1 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -2151,6 +2151,10 @@ [KVM,ARM] Allow use of GICv4 for direct injection of LPIs. + kvm-arm.nested= + [KVM,ARM] Allow nested virtualization in KVM/ARM. + Default is 0 (disabled) + kvm-intel.ept= [KVM,Intel] Disable extended page tables (virtualized MMU) support on capable Intel chips. Default is 1 (enabled) diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 865e0253fc1e..21f0067d8d40 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -58,7 +58,8 @@ #define ARM64_WORKAROUND_SPECULATIVE_AT_NVHE 48 #define ARM64_HAS_E0PD 49 #define ARM64_HAS_RNG 50 +#define ARM64_HAS_NESTED_VIRT 51 -#define ARM64_NCAPS 51 +#define ARM64_NCAPS 52 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index b91570ff9db1..00b283d6d31a 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -687,6 +687,7 @@ #define ID_AA64MMFR2_E0PD_SHIFT 60 #define ID_AA64MMFR2_FWB_SHIFT 40 #define ID_AA64MMFR2_AT_SHIFT 32 +#define ID_AA64MMFR2_NV_SHIFT 24 #define ID_AA64MMFR2_LVA_SHIFT 16 #define ID_AA64MMFR2_IESB_SHIFT 12 #define ID_AA64MMFR2_LSM_SHIFT 8 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 0b6715625cf6..f0e58450eb16 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -243,6 +243,7 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr2[] = { ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_E0PD_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_FWB_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_AT_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_NV_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_LVA_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_IESB_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_LSM_SHIFT, 4, 0), @@ -1241,6 +1242,21 @@ static void cpu_copy_el2regs(const struct arm64_cpu_capabilities *__unused) if (!alternative_is_applied(ARM64_HAS_VIRT_HOST_EXTN)) write_sysreg(read_sysreg(tpidr_el1), tpidr_el2); } + +static bool nested_param; +static bool has_nested_virt_support(const struct arm64_cpu_capabilities *cap, + int scope) +{ + return has_cpuid_feature(cap, scope) && + nested_param; +} + +static int __init kvmarm_nested_cfg(char *buf) +{ + return strtobool(buf, &nested_param); +} + +early_param("kvm-arm.nested", kvmarm_nested_cfg); #endif static void cpu_has_fwb(const struct arm64_cpu_capabilities *__unused) @@ -1419,6 +1435,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = runs_at_el2, .cpu_enable = cpu_copy_el2regs, }, + { + .desc = "Nested Virtualization Support", + .capability = ARM64_HAS_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_NV_SHIFT, + .min_field_value = 1, + }, #endif /* CONFIG_ARM64_VHE */ { .desc = "32-bit EL0 Support", From patchwork Tue Feb 11 17:48:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376131 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8A521395 for ; Tue, 11 Feb 2020 17:50:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C7EFF20661 for ; Tue, 11 Feb 2020 17:50:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443411; bh=Q45nUi+j7GZ+5bZdrZa7v6bmgIUXdcImPx3jSm4b5vM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=V2312moLqW2s8yct6Mwbmy9m/z8ojnLqLuAXMMXtnngHA8dKJMejnPZfKbnPNin/q sfQW1KF3f6d9HJZX/RgJT2itK/DkQtZdtX1VViTNuTcSiFJEoC9tfipF1LzOIRpigU YDwQiV2dCtoUvje2TDlGKZ4ac+ZYFOaXVT1yiNVA= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729989AbgBKRt6 (ORCPT ); Tue, 11 Feb 2020 12:49:58 -0500 Received: from mail.kernel.org ([198.145.29.99]:52302 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728601AbgBKRt5 (ORCPT ); Tue, 11 Feb 2020 12:49:57 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4EA22206D7; Tue, 11 Feb 2020 17:49:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443397; bh=Q45nUi+j7GZ+5bZdrZa7v6bmgIUXdcImPx3jSm4b5vM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lfg5GWl4io9VLVhsodj2YrfEPfxCT8wxzicuO2HOuuR2HtyajJsr6Uy+4JQbEh9m4 eXKqTRDVxGF2V58s9HzxO+UwIlOSGkcj3cG1E9n7v8Px9aHC6xhGLy+4E0Rb9+Jjcw k3HvM6YOQWR84p+7vpV9Z0cxWcCqmzGOCL5N6z+s= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZfX-004O7k-Ip; Tue, 11 Feb 2020 17:49:55 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 03/94] KVM: arm64: nv: Introduce nested virtualization VCPU feature Date: Tue, 11 Feb 2020 17:48:07 +0000 Message-Id: <20200211174938.27809-4-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Introduce the feature bit and a primitive that checks if the feature is set behind a static key check based on the cpus_have_const_cap check. Checking nested_virt_in_use() on systems without nested virt enabled should have neglgible overhead. We don't yet allow userspace to actually set this feature. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_nested.h | 9 +++++++++ arch/arm64/include/asm/kvm_nested.h | 13 +++++++++++++ arch/arm64/include/uapi/asm/kvm.h | 1 + 3 files changed, 23 insertions(+) create mode 100644 arch/arm/include/asm/kvm_nested.h create mode 100644 arch/arm64/include/asm/kvm_nested.h diff --git a/arch/arm/include/asm/kvm_nested.h b/arch/arm/include/asm/kvm_nested.h new file mode 100644 index 000000000000..124ff6445f8f --- /dev/null +++ b/arch/arm/include/asm/kvm_nested.h @@ -0,0 +1,9 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM_KVM_NESTED_H +#define __ARM_KVM_NESTED_H + +#include + +static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) { return false; } + +#endif /* __ARM_KVM_NESTED_H */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h new file mode 100644 index 000000000000..f243a8acb9fc --- /dev/null +++ b/arch/arm64/include/asm/kvm_nested.h @@ -0,0 +1,13 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef __ARM64_KVM_NESTED_H +#define __ARM64_KVM_NESTED_H + +#include + +static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) +{ + return cpus_have_const_cap(ARM64_HAS_NESTED_VIRT) && + test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features); +} + +#endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index ba85bb23f060..5b9c58a55537 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -106,6 +106,7 @@ struct kvm_regs { #define KVM_ARM_VCPU_SVE 4 /* enable SVE for this CPU */ #define KVM_ARM_VCPU_PTRAUTH_ADDRESS 5 /* VCPU uses address authentication */ #define KVM_ARM_VCPU_PTRAUTH_GENERIC 6 /* VCPU uses generic authentication */ +#define KVM_ARM_VCPU_HAS_EL2 7 /* Support nested virtualization */ struct kvm_vcpu_init { __u32 target; From patchwork Tue Feb 11 17:48:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376117 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 22EA714B4 for ; Tue, 11 Feb 2020 17:50:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0089C21739 for ; Tue, 11 Feb 2020 17:50:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443401; bh=vZYuRZEYB65TB5BcmIONClbvBia5wiG+UbdUv2W2Gl0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=MeZMsIa1uM0fHGExl1v5eVWPwnGQY8t0oerExewmekl+iWHKJ3j/c2IHZnE6SDj3m nOa1pNxtSEvYezQs1U53e7aXT0iaIN45bLrGOQEZ3hlwZtRcVKmx6IrhrUAeEu0HEg Ot2yETakrzaCI/ijA8LHsDAb66mMysAFAT19ql0s= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730093AbgBKRuA (ORCPT ); Tue, 11 Feb 2020 12:50:00 -0500 Received: from mail.kernel.org ([198.145.29.99]:52302 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728575AbgBKRt6 (ORCPT ); Tue, 11 Feb 2020 12:49:58 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E08DE2086A; Tue, 11 Feb 2020 17:49:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443398; bh=vZYuRZEYB65TB5BcmIONClbvBia5wiG+UbdUv2W2Gl0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bidf3TOnosFS8mKox3Y7XNJQiv3wt2HAC43WhRps4CyrGo0e0auz4bpkSPeolSFfV rU60Gda2lcRS4mEdynFEKhC8vzdj4S87iE6LeWp3dReN/Z5s+mKxf1nrxXscQ+cJ4t CM8G0+YrndZeKSP1bIiQT+7F8nuvdXYV+DAoFEK8= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZfY-004O7k-5R; Tue, 11 Feb 2020 17:49:56 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 04/94] KVM: arm64: nv: Reset VCPU to EL2 registers if VCPU nested virt is set Date: Tue, 11 Feb 2020 17:48:08 +0000 Message-Id: <20200211174938.27809-5-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Reset the VCPU with PSTATE.M = EL2h when the nested virtualization feature is enabled on the VCPU. Signed-off-by: Christoffer Dall [maz: rework register reset not to use empty data structures] Signed-off-by: Marc Zyngier --- arch/arm64/kvm/reset.c | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 30b7ea680f66..5c50df274239 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -28,6 +28,7 @@ #include #include #include +#include #include /* Maximum phys_shift supported for any VM on this host */ @@ -36,15 +37,14 @@ static u32 kvm_ipa_limit; /* * ARMv8 Reset Values */ -static const struct kvm_regs default_regs_reset = { - .regs.pstate = (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | - PSR_F_BIT | PSR_D_BIT), -}; +#define VCPU_RESET_PSTATE_EL1 (PSR_MODE_EL1h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) -static const struct kvm_regs default_regs_reset32 = { - .regs.pstate = (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | - PSR_AA32_I_BIT | PSR_AA32_F_BIT), -}; +#define VCPU_RESET_PSTATE_EL2 (PSR_MODE_EL2h | PSR_A_BIT | PSR_I_BIT | \ + PSR_F_BIT | PSR_D_BIT) + +#define VCPU_RESET_PSTATE_SVC (PSR_AA32_MODE_SVC | PSR_AA32_A_BIT | \ + PSR_AA32_I_BIT | PSR_AA32_F_BIT) static bool cpu_has_32bit_el1(void) { @@ -260,6 +260,7 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) const struct kvm_regs *cpu_reset; int ret = -EINVAL; bool loaded; + u32 pstate; /* Reset PMU outside of the non-preemptible section */ kvm_pmu_vcpu_reset(vcpu); @@ -288,18 +289,25 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) switch (vcpu->arch.target) { default: if (test_bit(KVM_ARM_VCPU_EL1_32BIT, vcpu->arch.features)) { - if (!cpu_has_32bit_el1()) + /* + * The CPU must support 32bit EL1, and 32bit + * NV is just not a thing... + */ + if (!cpu_has_32bit_el1() || nested_virt_in_use(vcpu)) goto out; - cpu_reset = &default_regs_reset32; + pstate = VCPU_RESET_PSTATE_SVC; + } else if (nested_virt_in_use(vcpu)) { + pstate = VCPU_RESET_PSTATE_EL2; } else { - cpu_reset = &default_regs_reset; + pstate = VCPU_RESET_PSTATE_EL1; } break; } /* Reset core registers */ - memcpy(vcpu_gp_regs(vcpu), cpu_reset, sizeof(*cpu_reset)); + memset(vcpu_gp_regs(vcpu), 0, sizeof(*cpu_reset)); + vcpu_gp_regs(vcpu)->regs.pstate = pstate; /* Reset system registers */ kvm_reset_sys_regs(vcpu); From patchwork Tue Feb 11 17:48:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376125 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AC7B21395 for ; Tue, 11 Feb 2020 17:50:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 869E0206D7 for ; Tue, 11 Feb 2020 17:50:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443406; bh=2lzshsjfN/k3DkUr97SZX8z9y6OmheIcQ6EfHGz4VSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=tUNdb9UUqfcpwovj38BhpekxabX3rzbMzt6JAj0WK8i1WGyFKpFBhOVieZ5DTggVl amB9/AIRgWYfkgY4lSY5PnLBO5s0dbiY6Ek2NnHqdsYivYk5EHSrKC2k6OxcbY/8nq XJ/t0urro0QTdt7YGRS5ELZzfadcw3b/v1nlDTuM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730061AbgBKRt7 (ORCPT ); Tue, 11 Feb 2020 12:49:59 -0500 Received: from mail.kernel.org ([198.145.29.99]:52364 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730022AbgBKRt7 (ORCPT ); Tue, 11 Feb 2020 12:49:59 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6136A214DB; Tue, 11 Feb 2020 17:49:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443398; bh=2lzshsjfN/k3DkUr97SZX8z9y6OmheIcQ6EfHGz4VSU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nxvylJ7UNsw1LZMzsJYRpYMI9SCwuz9BYiZYNOCOBQX0owbANZNnxbhTGkmrF65Ay /cbxDgif49u3ppBaqxxpzIsCC/3fupO9J8W6nqVhCRiab/rsr+ST9cXh/03AUZK73c uLBqVKjt5r/jo+5yyjmX4Vgoifb6qUi6FnOxigKQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZfY-004O7k-NW; Tue, 11 Feb 2020 17:49:56 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 05/94] KVM: arm64: nv: Allow userspace to set PSR_MODE_EL2x Date: Tue, 11 Feb 2020 17:48:09 +0000 Message-Id: <20200211174938.27809-6-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall We were not allowing userspace to set a more privileged mode for the VCPU than EL1, but we should allow this when nested virtualization is enabled for the VCPU. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/guest.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 2bd92301d32f..c75b5498c048 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #include "trace.h" @@ -194,6 +195,11 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (vcpu_el1_is_32bit(vcpu)) return -EINVAL; break; + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + if (vcpu_el1_is_32bit(vcpu) || !nested_virt_in_use(vcpu)) + return -EINVAL; + break; default: err = -EINVAL; goto out; From patchwork Tue Feb 11 17:48:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376129 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5722914B4 for ; Tue, 11 Feb 2020 17:50:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 221B0206CC for ; Tue, 11 Feb 2020 17:50:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443410; bh=XsaHBX6K4WcheAc2JoXTGURQrLqx29yW4CCr1mBKIs0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=y9YGe9hRX4JL7C5zpv00XAt3NUiGRtRW1EyUBAAW0oMI0xwgRVibD6IJ+u+3LdHT/ lCC5P1hv2POodNCUuRmzHvDZ+ade1/69D7Y4YWAfLwxVPgUdwFOVdq+4BHCqhpwvQh ZO74NpN6J0KH0tDzrH3+BzGxM9fZLder8+0Jq4Zs= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730176AbgBKRuG (ORCPT ); Tue, 11 Feb 2020 12:50:06 -0500 Received: from mail.kernel.org ([198.145.29.99]:52400 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730048AbgBKRt7 (ORCPT ); Tue, 11 Feb 2020 12:49:59 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EDD64208C3; Tue, 11 Feb 2020 17:49:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443399; bh=XsaHBX6K4WcheAc2JoXTGURQrLqx29yW4CCr1mBKIs0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=03u1C0yoDE9BvJBNYlGmUgROtFB9U+lGscIQ84Bv4h3Zr2XzyPeS8782KsrKWGdvi cW4Ti8UKpr802Oj8rLskTAi1ofK3LQs3SaJezVMm+6LYDB8Jcn+InwhLoAMRJtdm2v 5ijv2s6M/zYGbFifCuel5eNvkYNj2ttlzjLPBRcQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZfZ-004O7k-8T; Tue, 11 Feb 2020 17:49:57 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 06/94] KVM: arm64: nv: Add EL2 system registers to vcpu context Date: Tue, 11 Feb 2020 17:48:10 +0000 Message-Id: <20200211174938.27809-7-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim ARM v8.3 introduces a new bit in the HCR_EL2, which is the NV bit. When this bit is set, accessing EL2 registers in EL1 traps to EL2. In addition, executing the following instructions in EL1 will trap to EL2: tlbi, at, eret, and msr/mrs instructions to access SP_EL1. Most of the instructions that trap to EL2 with the NV bit were undef at EL1 prior to ARM v8.3. The only instruction that was not undef is eret. This patch sets up a handler for EL2 registers and SP_EL1 register accesses at EL1. The host hypervisor keeps those register values in memory, and will emulate their behavior. This patch doesn't set the NV bit yet. It will be set in a later patch once nested virtualization support is completed. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 35 ++++++++++++- arch/arm64/include/asm/sysreg.h | 50 +++++++++++++++++- arch/arm64/kvm/sys_regs.c | 85 +++++++++++++++++++++++++++++-- 3 files changed, 163 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index d87aa609d2b6..1bda11cc0f39 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -170,12 +170,45 @@ enum vcpu_sysreg { APGAKEYLO_EL1, APGAKEYHI_EL1, - /* 32bit specific registers. Keep them at the end of the range */ + /* 32bit specific registers. */ DACR32_EL2, /* Domain Access Control Register */ IFSR32_EL2, /* Instruction Fault Status Register */ FPEXC32_EL2, /* Floating-Point Exception Control Register */ DBGVCR32_EL2, /* Debug Vector Catch Register */ + /* EL2 registers sorted ascending by Op0, Op1, CRn, CRm, Op2 */ + VPIDR_EL2, /* Virtualization Processor ID Register */ + VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */ + SCTLR_EL2, /* System Control Register (EL2) */ + ACTLR_EL2, /* Auxiliary Control Register (EL2) */ + HCR_EL2, /* Hypervisor Configuration Register */ + MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */ + CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ + HSTR_EL2, /* Hypervisor System Trap Register */ + HACR_EL2, /* Hypervisor Auxiliary Control Register */ + TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ + TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ + TCR_EL2, /* Translation Control Register (EL2) */ + VTTBR_EL2, /* Virtualization Translation Table Base Register */ + VTCR_EL2, /* Virtualization Translation Control Register */ + SPSR_EL2, /* EL2 saved program status register */ + ELR_EL2, /* EL2 exception link register */ + AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */ + AFSR1_EL2, /* Auxiliary Fault Status Register 1 (EL2) */ + ESR_EL2, /* Exception Syndrome Register (EL2) */ + FAR_EL2, /* Hypervisor IPA Fault Address Register */ + HPFAR_EL2, /* Hypervisor IPA Fault Address Register */ + MAIR_EL2, /* Memory Attribute Indirection Register (EL2) */ + AMAIR_EL2, /* Auxiliary Memory Attribute Indirection Register (EL2) */ + VBAR_EL2, /* Vector Base Address Register (EL2) */ + RVBAR_EL2, /* Reset Vector Base Address Register */ + RMR_EL2, /* Reset Management Register */ + CONTEXTIDR_EL2, /* Context ID Register (EL2) */ + TPIDR_EL2, /* EL2 Software Thread ID Register */ + CNTVOFF_EL2, /* Counter-timer Virtual Offset register */ + CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ + SP_EL2, /* EL2 Stack Pointer */ + NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 00b283d6d31a..260c1cf683a2 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -407,17 +407,49 @@ #define SYS_PMCCFILTR_EL0 sys_reg(3, 3, 14, 15, 7) +#define SYS_VPIDR_EL2 sys_reg(3, 4, 0, 0, 0) +#define SYS_VMPIDR_EL2 sys_reg(3, 4, 0, 0, 5) + +#define SYS_SCTLR_EL2 sys_reg(3, 4, 1, 0, 0) +#define SYS_ACTLR_EL2 sys_reg(3, 4, 1, 0, 1) +#define SYS_HCR_EL2 sys_reg(3, 4, 1, 1, 0) +#define SYS_MDCR_EL2 sys_reg(3, 4, 1, 1, 1) +#define SYS_CPTR_EL2 sys_reg(3, 4, 1, 1, 2) +#define SYS_HSTR_EL2 sys_reg(3, 4, 1, 1, 3) +#define SYS_HACR_EL2 sys_reg(3, 4, 1, 1, 7) + #define SYS_ZCR_EL2 sys_reg(3, 4, 1, 2, 0) + +#define SYS_TTBR0_EL2 sys_reg(3, 4, 2, 0, 0) +#define SYS_TTBR1_EL2 sys_reg(3, 4, 2, 0, 1) +#define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) +#define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) +#define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) + #define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0) + #define SYS_SPSR_EL2 sys_reg(3, 4, 4, 0, 0) #define SYS_ELR_EL2 sys_reg(3, 4, 4, 0, 1) +#define SYS_SP_EL1 sys_reg(3, 4, 4, 1, 0) + #define SYS_IFSR32_EL2 sys_reg(3, 4, 5, 0, 1) +#define SYS_AFSR0_EL2 sys_reg(3, 4, 5, 1, 0) +#define SYS_AFSR1_EL2 sys_reg(3, 4, 5, 1, 1) #define SYS_ESR_EL2 sys_reg(3, 4, 5, 2, 0) #define SYS_VSESR_EL2 sys_reg(3, 4, 5, 2, 3) #define SYS_FPEXC32_EL2 sys_reg(3, 4, 5, 3, 0) #define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) -#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) +#define SYS_FAR_EL2 sys_reg(3, 4, 6, 0, 0) +#define SYS_HPFAR_EL2 sys_reg(3, 4, 6, 0, 4) + +#define SYS_MAIR_EL2 sys_reg(3, 4, 10, 2, 0) +#define SYS_AMAIR_EL2 sys_reg(3, 4, 10, 3, 0) + +#define SYS_VBAR_EL2 sys_reg(3, 4, 12, 0, 0) +#define SYS_RVBAR_EL2 sys_reg(3, 4, 12, 0, 1) +#define SYS_RMR_EL2 sys_reg(3, 4, 12, 0, 2) +#define SYS_VDISR_EL2 sys_reg(3, 4, 12, 1, 1) #define __SYS__AP0Rx_EL2(x) sys_reg(3, 4, 12, 8, x) #define SYS_ICH_AP0R0_EL2 __SYS__AP0Rx_EL2(0) #define SYS_ICH_AP0R1_EL2 __SYS__AP0Rx_EL2(1) @@ -459,23 +491,37 @@ #define SYS_ICH_LR14_EL2 __SYS__LR8_EL2(6) #define SYS_ICH_LR15_EL2 __SYS__LR8_EL2(7) +#define SYS_CONTEXTIDR_EL2 sys_reg(3, 4, 13, 0, 1) +#define SYS_TPIDR_EL2 sys_reg(3, 4, 13, 0, 2) + +#define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) +#define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) + /* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0) #define SYS_CPACR_EL12 sys_reg(3, 5, 1, 0, 2) #define SYS_ZCR_EL12 sys_reg(3, 5, 1, 2, 0) + #define SYS_TTBR0_EL12 sys_reg(3, 5, 2, 0, 0) #define SYS_TTBR1_EL12 sys_reg(3, 5, 2, 0, 1) #define SYS_TCR_EL12 sys_reg(3, 5, 2, 0, 2) + #define SYS_SPSR_EL12 sys_reg(3, 5, 4, 0, 0) #define SYS_ELR_EL12 sys_reg(3, 5, 4, 0, 1) + #define SYS_AFSR0_EL12 sys_reg(3, 5, 5, 1, 0) #define SYS_AFSR1_EL12 sys_reg(3, 5, 5, 1, 1) #define SYS_ESR_EL12 sys_reg(3, 5, 5, 2, 0) + #define SYS_FAR_EL12 sys_reg(3, 5, 6, 0, 0) + #define SYS_MAIR_EL12 sys_reg(3, 5, 10, 2, 0) #define SYS_AMAIR_EL12 sys_reg(3, 5, 10, 3, 0) + #define SYS_VBAR_EL12 sys_reg(3, 5, 12, 0, 0) + #define SYS_CONTEXTIDR_EL12 sys_reg(3, 5, 13, 0, 1) + #define SYS_CNTKCTL_EL12 sys_reg(3, 5, 14, 1, 0) #define SYS_CNTP_TVAL_EL02 sys_reg(3, 5, 14, 2, 0) #define SYS_CNTP_CTL_EL02 sys_reg(3, 5, 14, 2, 1) @@ -484,6 +530,8 @@ #define SYS_CNTV_CTL_EL02 sys_reg(3, 5, 14, 3, 1) #define SYS_CNTV_CVAL_EL02 sys_reg(3, 5, 14, 3, 2) +#define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44)) #define SCTLR_ELx_ENIA (BIT(31)) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 3e909b117f0c..b00a4aeb2a2e 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -173,6 +173,29 @@ static u32 get_ccsidr(u32 csselr) return ccsidr; } +static bool access_rw(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, r->reg); + else + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + + return true; +} + +static bool access_wi(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return ignore_write(vcpu, p); + + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + return true; +} + /* * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). */ @@ -383,12 +406,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - if (p->is_write) { - vcpu_write_sys_reg(vcpu, p->regval, r->reg); + access_rw(vcpu, p, r); + if (p->is_write) vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; - } else { - p->regval = vcpu_read_sys_reg(vcpu, r->reg); - } trace_trap_reg(__func__, r->reg, p->is_write, p->regval); @@ -1347,6 +1367,19 @@ static bool access_ccsidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, .set_user = set_raz_id_reg, \ } +static bool access_sp_el1(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + /* SP_EL1 is NOT maintained in sys_regs array */ + if (p->is_write) + vcpu->arch.ctxt.gp_regs.sp_el1 = p->regval; + else + p->regval = vcpu->arch.ctxt.gp_regs.sp_el1; + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -1639,9 +1672,51 @@ static const struct sys_reg_desc sys_reg_descs[] = { */ { SYS_DESC(SYS_PMCCFILTR_EL0), access_pmu_evtyper, reset_val, PMCCFILTR_EL0, 0 }, + { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 }, + { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 }, + + { SYS_DESC(SYS_SCTLR_EL2), access_rw, reset_val, SCTLR_EL2, 0 }, + { SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 }, + { SYS_DESC(SYS_HCR_EL2), access_rw, reset_val, HCR_EL2, 0 }, + { SYS_DESC(SYS_MDCR_EL2), access_rw, reset_val, MDCR_EL2, 0 }, + { SYS_DESC(SYS_CPTR_EL2), access_rw, reset_val, CPTR_EL2, CPTR_EL2_RES1 }, + { SYS_DESC(SYS_HSTR_EL2), access_rw, reset_val, HSTR_EL2, 0 }, + { SYS_DESC(SYS_HACR_EL2), access_rw, reset_val, HACR_EL2, 0 }, + + { SYS_DESC(SYS_TTBR0_EL2), access_rw, reset_val, TTBR0_EL2, 0 }, + { SYS_DESC(SYS_TTBR1_EL2), access_rw, reset_val, TTBR1_EL2, 0 }, + { SYS_DESC(SYS_TCR_EL2), access_rw, reset_val, TCR_EL2, TCR_EL2_RES1 }, + { SYS_DESC(SYS_VTTBR_EL2), access_rw, reset_val, VTTBR_EL2, 0 }, + { SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 }, + { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, + { SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 }, + { SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 }, + { SYS_DESC(SYS_SP_EL1), access_sp_el1}, + { SYS_DESC(SYS_IFSR32_EL2), NULL, reset_unknown, IFSR32_EL2 }, + { SYS_DESC(SYS_AFSR0_EL2), access_rw, reset_val, AFSR0_EL2, 0 }, + { SYS_DESC(SYS_AFSR1_EL2), access_rw, reset_val, AFSR1_EL2, 0 }, + { SYS_DESC(SYS_ESR_EL2), access_rw, reset_val, ESR_EL2, 0 }, { SYS_DESC(SYS_FPEXC32_EL2), NULL, reset_val, FPEXC32_EL2, 0x700 }, + + { SYS_DESC(SYS_FAR_EL2), access_rw, reset_val, FAR_EL2, 0 }, + { SYS_DESC(SYS_HPFAR_EL2), access_rw, reset_val, HPFAR_EL2, 0 }, + + { SYS_DESC(SYS_MAIR_EL2), access_rw, reset_val, MAIR_EL2, 0 }, + { SYS_DESC(SYS_AMAIR_EL2), access_rw, reset_val, AMAIR_EL2, 0 }, + + { SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 }, + { SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 }, + { SYS_DESC(SYS_RMR_EL2), access_wi, reset_val, RMR_EL2, 1 }, + + { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, + { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, + + { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, + { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; static bool trap_dbgidr(struct kvm_vcpu *vcpu, From patchwork Tue Feb 11 17:48:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376119 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9370814E3 for ; Tue, 11 Feb 2020 17:50:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 70B292173E for ; Tue, 11 Feb 2020 17:50:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443401; bh=bI6+KvbUvFrGlgGnlrCjM3mn28sscwzLqPwrQi1awEU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Zia4iV+SQVgtsExU8X0S8NgpLu7jnChkT2Bvbh8RvDvaqboCOPiGkpojIJ010ffOn Usdw3qO35gB4YwxnzK0y+Ek3qV1Cd+wiTDan1Bpu/+JrZTvSUT/qGiXUiaATSf7Yks QOrFBIhoTTms9Z3VP4p4cyR2I1yESZpOYW9J67ic= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730110AbgBKRuA (ORCPT ); Tue, 11 Feb 2020 12:50:00 -0500 Received: from mail.kernel.org ([198.145.29.99]:52302 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730022AbgBKRuA (ORCPT ); Tue, 11 Feb 2020 12:50:00 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 826C6215A4; Tue, 11 Feb 2020 17:49:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443399; bh=bI6+KvbUvFrGlgGnlrCjM3mn28sscwzLqPwrQi1awEU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JMFBpTCAoy0/00GkWXbvRlICovlus2ePDvw0fFtGO0RN7GYqNKzIu6FpMeFNv3Sd2 7AJuosKm8C2cYcbRObAlfffWpFaFkNmzQymDD4SH6T6aIIguGXwmoDrf8KVrv6Z0Sd N8TbCCBpOkxqe2nZzH3wHKvR/dVTW9DZJIj/vE4o= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZfZ-004O7k-Qx; Tue, 11 Feb 2020 17:49:57 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 07/94] KVM: arm64: nv: Reset VMPIDR_EL2 and VPIDR_EL2 to sane values Date: Tue, 11 Feb 2020 17:48:11 +0000 Message-Id: <20200211174938.27809-8-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The VMPIDR_EL2 and VPIDR_EL2 are architecturally UNKNOWN at reset, but let's be nice to a guest hypervisor behaving foolishly and reset these to something reasonable anyway. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index b00a4aeb2a2e..4b5310ea3bf8 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -624,7 +624,7 @@ static void reset_amair_el1(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) vcpu_write_sys_reg(vcpu, amair, AMAIR_EL1); } -static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +static u64 compute_reset_mpidr(struct kvm_vcpu *vcpu) { u64 mpidr; @@ -638,7 +638,24 @@ static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) mpidr = (vcpu->vcpu_id & 0x0f) << MPIDR_LEVEL_SHIFT(0); mpidr |= ((vcpu->vcpu_id >> 4) & 0xff) << MPIDR_LEVEL_SHIFT(1); mpidr |= ((vcpu->vcpu_id >> 12) & 0xff) << MPIDR_LEVEL_SHIFT(2); - vcpu_write_sys_reg(vcpu, (1ULL << 31) | mpidr, MPIDR_EL1); + mpidr |= (1ULL << 31); + + return mpidr; +} + +static void reset_mpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), MPIDR_EL1); +} + +static void reset_vmpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, compute_reset_mpidr(vcpu), VMPIDR_EL2); +} + +static void reset_vpidr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) +{ + vcpu_write_sys_reg(vcpu, read_cpuid_id(), VPIDR_EL2); } static void reset_pmcr(struct kvm_vcpu *vcpu, const struct sys_reg_desc *r) @@ -1672,8 +1689,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { */ { SYS_DESC(SYS_PMCCFILTR_EL0), access_pmu_evtyper, reset_val, PMCCFILTR_EL0, 0 }, - { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_val, VPIDR_EL2, 0 }, - { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_val, VMPIDR_EL2, 0 }, + { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_vpidr, VPIDR_EL2 }, + { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_vmpidr, VMPIDR_EL2 }, { SYS_DESC(SYS_SCTLR_EL2), access_rw, reset_val, SCTLR_EL2, 0 }, { SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 }, From patchwork Tue Feb 11 17:48:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376123 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7ADF61395 for ; Tue, 11 Feb 2020 17:50:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 56FFF21734 for ; Tue, 11 Feb 2020 17:50:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443404; bh=iaPvHhL1DahVBvUNkh+yyOVLWQlC65rw+u5cb6z/KaA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=saty5tNoMEEeUVJkMS3uLnDpdZx7pVAizRZHXE3NUlRA2k5sW/tpIS2JaCxphaPkP Pvsbc21PUgQn8L2aSC/XyvBPlQ/KcGKItxylgcff9P0JSe/EaJBrKmh/mtDaJp53H3 CiUIjGYkv3ogRSQ7bvv6mOlbJSAYLcwIkQIobioY= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730125AbgBKRuC (ORCPT ); Tue, 11 Feb 2020 12:50:02 -0500 Received: from mail.kernel.org ([198.145.29.99]:52364 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730085AbgBKRuB (ORCPT ); Tue, 11 Feb 2020 12:50:01 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 07C1C21569; Tue, 11 Feb 2020 17:50:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443400; bh=iaPvHhL1DahVBvUNkh+yyOVLWQlC65rw+u5cb6z/KaA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=yASocWOyK4GYSUUWqNBMHTyoOswWx8yjJaBXOdOnuSnfZ+YHD8cMbn15yRi2e4AAR olLSpWob1gnH0I+7Slp6AFi3xosleDCY2kDIF8tQAFRPTO/J8SeZXYAiKGOgD3A/rr w03PH58i1FarRw4rpV4X0zpzv9srhRKvOCd8JJKg= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfa-004O7k-CP; Tue, 11 Feb 2020 17:49:58 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 08/94] KVM: arm64: nv: Add nested virt VCPU primitives for vEL2 VCPU state Date: Tue, 11 Feb 2020 17:48:12 +0000 Message-Id: <20200211174938.27809-9-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall When running a nested hypervisor we commonly have to figure out if the VCPU mode is running in the context of a guest hypervisor or guest guest, or just a normal guest. Add convenient primitives for this. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 55 ++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 688c63412cc2..9646ad7c2640 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -194,6 +194,61 @@ static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, vcpu_gp_regs(vcpu)->regs.regs[reg_num] = val; } +static inline bool vcpu_mode_el2_ctxt(const struct kvm_cpu_context *ctxt) +{ + unsigned long cpsr = ctxt->gp_regs.regs.pstate; + + switch (cpsr & (PSR_MODE32_BIT | PSR_MODE_MASK)) { + case PSR_MODE_EL2h: + case PSR_MODE_EL2t: + return true; + default: + return false; + } +} + +static inline bool vcpu_mode_el2(const struct kvm_vcpu *vcpu) +{ + return vcpu_mode_el2_ctxt(&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt) +{ + return ctxt->sys_regs[HCR_EL2] & HCR_E2H; +} + +static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu) +{ + return __vcpu_el2_e2h_is_set(&vcpu->arch.ctxt); +} + +static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt) +{ + return ctxt->sys_regs[HCR_EL2] & HCR_TGE; +} + +static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu) +{ + return __vcpu_el2_tge_is_set(&vcpu->arch.ctxt); +} + +static inline bool __is_hyp_ctxt(const struct kvm_cpu_context *ctxt) +{ + /* + * We are in a hypervisor context if the vcpu mode is EL2 or + * E2H and TGE bits are set. The latter means we are in the user space + * of the VHE kernel. ARMv8.1 ARM describes this as 'InHost' + */ + return vcpu_mode_el2_ctxt(ctxt) || + (__vcpu_el2_e2h_is_set(ctxt) && __vcpu_el2_tge_is_set(ctxt)) || + WARN_ON(__vcpu_el2_tge_is_set(ctxt)); +} + +static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) +{ + return __is_hyp_ctxt(&vcpu->arch.ctxt); +} + static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu) { if (vcpu_mode_is_32bit(vcpu)) From patchwork Tue Feb 11 17:48:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376121 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C81F014B4 for ; Tue, 11 Feb 2020 17:50:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9B56521734 for ; Tue, 11 Feb 2020 17:50:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443403; bh=gbwM+0i09E4jMyjEdwf8olNDS0MI8+CJBoLot3mAdAs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=FqphOnLSnRIF3l2Nm3SZcl3//uMLGMFBq5LSSnGyZac8bsmuw7ct0PXhcsxpMUbkm JNC/cDFUfMCWa05NYgZR8fyzCHJ1YCr3QZRvj/53iTP9jUZgNInOrXZ7y3QqFP/2LA GOTaYeOsFp4Ri0j3MvQysPy6uJjHSUxLxBYBKNBw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730160AbgBKRuC (ORCPT ); Tue, 11 Feb 2020 12:50:02 -0500 Received: from mail.kernel.org ([198.145.29.99]:52400 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730104AbgBKRuB (ORCPT ); Tue, 11 Feb 2020 12:50:01 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9511A2168B; Tue, 11 Feb 2020 17:50:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443400; bh=gbwM+0i09E4jMyjEdwf8olNDS0MI8+CJBoLot3mAdAs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t7bx85rgAkjf3LqT4SsTSfxye8xVr3gMKlNghgoePqP3F7qS7ZBFJcd3lsiwXNmgc BK2Cm9ptXYcYA8dvMdDOfEM5e/ZX6FTeT83lsbE4h543ai9FrhRJgRAThqoSln6wqg HPdGKEFOrjjj7iICH0AXJFg+UiWxhZitTePZkMww= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfa-004O7k-Tg; Tue, 11 Feb 2020 17:49:59 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 09/94] KVM: arm64: nv: Support virtual EL2 exceptions Date: Tue, 11 Feb 2020 17:48:13 +0000 Message-Id: <20200211174938.27809-10-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Support injecting exceptions and performing exception returns to and from virtual EL2. This must be done entirely in software except when taking an exception from vEL0 to vEL2 when the virtual HCR_EL2.{E2H,TGE} == {1,1} (a VHE guest hypervisor). Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 17 +++ arch/arm64/include/asm/kvm_emulate.h | 22 ++++ arch/arm64/kvm/Makefile | 2 + arch/arm64/kvm/emulate-nested.c | 183 +++++++++++++++++++++++++++ arch/arm64/kvm/inject_fault.c | 12 -- arch/arm64/kvm/trace.h | 56 ++++++++ 6 files changed, 280 insertions(+), 12 deletions(-) create mode 100644 arch/arm64/kvm/emulate-nested.c diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 6e5d839f42b5..93469ac81fef 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -328,4 +328,21 @@ #define CPACR_EL1_TTA (1 << 28) #define CPACR_EL1_DEFAULT (CPACR_EL1_FPEN | CPACR_EL1_ZEN_EL1EN) +#define kvm_mode_names \ + { PSR_MODE_EL0t, "EL0t" }, \ + { PSR_MODE_EL1t, "EL1t" }, \ + { PSR_MODE_EL1h, "EL1h" }, \ + { PSR_MODE_EL2t, "EL2t" }, \ + { PSR_MODE_EL2h, "EL2h" }, \ + { PSR_MODE_EL3t, "EL3t" }, \ + { PSR_MODE_EL3h, "EL3h" }, \ + { PSR_AA32_MODE_USR, "32-bit USR" }, \ + { PSR_AA32_MODE_FIQ, "32-bit FIQ" }, \ + { PSR_AA32_MODE_IRQ, "32-bit IRQ" }, \ + { PSR_AA32_MODE_SVC, "32-bit SVC" }, \ + { PSR_AA32_MODE_ABT, "32-bit ABT" }, \ + { PSR_AA32_MODE_HYP, "32-bit HYP" }, \ + { PSR_AA32_MODE_UND, "32-bit UND" }, \ + { PSR_AA32_MODE_SYS, "32-bit SYS" } + #endif /* __ARM64_KVM_ARM_H__ */ diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 9646ad7c2640..282e9ddbe1bc 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -21,6 +21,24 @@ #include #include +#define CURRENT_EL_SP_EL0_VECTOR 0x0 +#define CURRENT_EL_SP_ELx_VECTOR 0x200 +#define LOWER_EL_AArch64_VECTOR 0x400 +#define LOWER_EL_AArch32_VECTOR 0x600 + +enum exception_type { + except_type_sync = 0, + except_type_irq = 0x80, + except_type_fiq = 0x100, + except_type_serror = 0x180, +}; + +#define kvm_exception_type_names \ + { except_type_sync, "SYNC" }, \ + { except_type_irq, "IRQ" }, \ + { except_type_fiq, "FIQ" }, \ + { except_type_serror, "SERROR" } + unsigned long *vcpu_reg32(const struct kvm_vcpu *vcpu, u8 reg_num); unsigned long vcpu_read_spsr32(const struct kvm_vcpu *vcpu); void vcpu_write_spsr32(struct kvm_vcpu *vcpu, unsigned long v); @@ -36,6 +54,10 @@ void kvm_inject_undef32(struct kvm_vcpu *vcpu); void kvm_inject_dabt32(struct kvm_vcpu *vcpu, unsigned long addr); void kvm_inject_pabt32(struct kvm_vcpu *vcpu, unsigned long addr); +void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu); +int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2); +int kvm_inject_nested_irq(struct kvm_vcpu *vcpu); + static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) { return !(vcpu->arch.hcr_el2 & HCR_RW); diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 5ffbdc39e780..32498becb71b 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -37,3 +37,5 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-debug.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o + +kvm-$(CONFIG_KVM_ARM_HOST) += emulate-nested.o diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c new file mode 100644 index 000000000000..328ae3723330 --- /dev/null +++ b/arch/arm64/kvm/emulate-nested.c @@ -0,0 +1,183 @@ +/* + * Copyright (C) 2016 - Linaro and Columbia University + * Author: Jintack Lim + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include +#include + +#include +#include +#include + +#include "trace.h" + +/* This is borrowed from get_except_vector in inject_fault.c */ +static u64 get_el2_except_vector(struct kvm_vcpu *vcpu, + enum exception_type type) +{ + u64 exc_offset; + + switch (*vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT)) { + case PSR_MODE_EL2t: + exc_offset = CURRENT_EL_SP_EL0_VECTOR; + break; + case PSR_MODE_EL2h: + exc_offset = CURRENT_EL_SP_ELx_VECTOR; + break; + case PSR_MODE_EL1t: + case PSR_MODE_EL1h: + case PSR_MODE_EL0t: + exc_offset = LOWER_EL_AArch64_VECTOR; + break; + default: + kvm_err("Unexpected previous exception level: aarch32\n"); + exc_offset = LOWER_EL_AArch32_VECTOR; + } + + return vcpu_read_sys_reg(vcpu, VBAR_EL2) + exc_offset + type; +} + +void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) +{ + u64 spsr, elr, mode; + bool direct_eret; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. + */ + spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + direct_eret = (mode == PSR_MODE_EL0t && + vcpu_el2_e2h_is_set(vcpu) && + vcpu_el2_tge_is_set(vcpu)); + direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + + if (direct_eret) { + *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); + *vcpu_cpsr(vcpu) = spsr; + trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); + return; + } + + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + + elr = __vcpu_sys_reg(vcpu, ELR_EL2); + + trace_kvm_nested_eret(vcpu, elr, spsr); + + /* + * Note that the current exception level is always the virtual EL2, + * since we set HCR_EL2.NV bit only when entering the virtual EL2. + */ + *vcpu_pc(vcpu) = elr; + *vcpu_cpsr(vcpu) = spsr; + + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); +} + +static void enter_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2, + enum exception_type type) +{ + trace_kvm_inject_nested_exception(vcpu, esr_el2, type); + + vcpu_write_sys_reg(vcpu, *vcpu_cpsr(vcpu), SPSR_EL2); + vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2); + vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2); + + *vcpu_pc(vcpu) = get_el2_except_vector(vcpu, type); + /* On an exception, PSTATE.SP becomes 1 */ + *vcpu_cpsr(vcpu) = PSR_MODE_EL2h; + *vcpu_cpsr(vcpu) |= PSR_A_BIT | PSR_F_BIT | PSR_I_BIT | PSR_D_BIT; +} + +/* + * Emulate taking an exception to EL2. + * See ARM ARM J8.1.2 AArch64.TakeException() + */ +static int kvm_inject_nested(struct kvm_vcpu *vcpu, u64 esr_el2, + enum exception_type type) +{ + u64 pstate, mode; + bool direct_inject; + + if (!nested_virt_in_use(vcpu)) { + kvm_err("Unexpected call to %s for the non-nesting configuration\n", + __func__); + return -EINVAL; + } + + /* + * As for ERET, we can avoid doing too much on the injection path by + * checking that we either took the exception from a VHE host + * userspace or from vEL2. In these cases, there is no change in + * translation regime (or anything else), so let's do as little as + * possible. + */ + pstate = *vcpu_cpsr(vcpu); + mode = pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + + direct_inject = (mode == PSR_MODE_EL0t && + vcpu_el2_e2h_is_set(vcpu) && + vcpu_el2_tge_is_set(vcpu)); + direct_inject |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); + + if (direct_inject) { + enter_el2_exception(vcpu, esr_el2, type); + return 1; + } + + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + + enter_el2_exception(vcpu, esr_el2, type); + + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); + + return 1; +} + +int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + return kvm_inject_nested(vcpu, esr_el2, except_type_sync); +} + +int kvm_inject_nested_irq(struct kvm_vcpu *vcpu) +{ + /* + * Do not inject an irq if the: + * - Current exception level is EL2, and + * - virtual HCR_EL2.TGE == 0 + * - virtual HCR_EL2.IMO == 0 + * + * See Table D1-17 "Physical interrupt target and masking when EL3 is + * not implemented and EL2 is implemented" in ARM DDI 0487C.a. + */ + + if (vcpu_mode_el2(vcpu) && !vcpu_el2_tge_is_set(vcpu) && + !(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO)) + return 1; + + /* esr_el2 value doesn't matter for exits due to irqs. */ + return kvm_inject_nested(vcpu, 0, except_type_irq); +} diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c index 6aafc2825c1c..105a171d7b72 100644 --- a/arch/arm64/kvm/inject_fault.c +++ b/arch/arm64/kvm/inject_fault.c @@ -14,18 +14,6 @@ #include #include -#define CURRENT_EL_SP_EL0_VECTOR 0x0 -#define CURRENT_EL_SP_ELx_VECTOR 0x200 -#define LOWER_EL_AArch64_VECTOR 0x400 -#define LOWER_EL_AArch32_VECTOR 0x600 - -enum exception_type { - except_type_sync = 0, - except_type_irq = 0x80, - except_type_fiq = 0x100, - except_type_serror = 0x180, -}; - static u64 get_except_vector(struct kvm_vcpu *vcpu, enum exception_type type) { u64 exc_offset; diff --git a/arch/arm64/kvm/trace.h b/arch/arm64/kvm/trace.h index eab91ad0effb..d6ee78bcf028 100644 --- a/arch/arm64/kvm/trace.h +++ b/arch/arm64/kvm/trace.h @@ -204,7 +204,63 @@ TRACE_EVENT(kvm_set_guest_debug, TP_printk("vcpu: %p, flags: 0x%08x", __entry->vcpu, __entry->guest_debug) ); +TRACE_EVENT(kvm_nested_eret, + TP_PROTO(struct kvm_vcpu *vcpu, unsigned long elr_el2, + unsigned long spsr_el2), + TP_ARGS(vcpu, elr_el2, spsr_el2), + TP_STRUCT__entry( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, elr_el2) + __field(unsigned long, spsr_el2) + __field(unsigned long, target_mode) + __field(unsigned long, hcr_el2) + ), + + TP_fast_assign( + __entry->vcpu = vcpu; + __entry->elr_el2 = elr_el2; + __entry->spsr_el2 = spsr_el2; + __entry->target_mode = spsr_el2 & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), + + TP_printk("elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx", + __entry->elr_el2, __entry->spsr_el2, + __print_symbolic(__entry->target_mode, kvm_mode_names), + __entry->hcr_el2) +); + +TRACE_EVENT(kvm_inject_nested_exception, + TP_PROTO(struct kvm_vcpu *vcpu, u64 esr_el2, int type), + TP_ARGS(vcpu, esr_el2, type), + + TP_STRUCT__entry( + __field(struct kvm_vcpu *, vcpu) + __field(unsigned long, esr_el2) + __field(int, type) + __field(unsigned long, spsr_el2) + __field(unsigned long, pc) + __field(unsigned long, source_mode) + __field(unsigned long, hcr_el2) + ), + + TP_fast_assign( + __entry->vcpu = vcpu; + __entry->esr_el2 = esr_el2; + __entry->type = type; + __entry->spsr_el2 = *vcpu_cpsr(vcpu); + __entry->pc = *vcpu_pc(vcpu); + __entry->source_mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); + __entry->hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + ), + + TP_printk("%s: esr_el2 0x%lx elr_el2: 0x%lx spsr_el2: 0x%08lx (M: %s) hcr_el2: %lx", + __print_symbolic(__entry->type, kvm_exception_type_names), + __entry->esr_el2, __entry->pc, __entry->spsr_el2, + __print_symbolic(__entry->source_mode, kvm_mode_names), + __entry->hcr_el2) +); #endif /* _TRACE_ARM64_KVM_H */ #undef TRACE_INCLUDE_PATH From patchwork Tue Feb 11 17:48:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376241 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 71C581395 for ; Tue, 11 Feb 2020 17:52:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 51F3E20870 for ; Tue, 11 Feb 2020 17:52:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443552; bh=azVeCUurVCbeNJqhkw5m3Pgfoy9e98pVPeDGxajBlw4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=sra8NBICzAM+RJt1ob8QOCRaTs39+gkpRVrckekfqOc0hpTv3mTLelzNb50aFHRKK pGW9OdzPdO+3PdjYebhs/DNnHHgaps4dz5FFvLWNS4vQA/MBTx5eAbqmEKt/J2xwfV W04Z091psOSUfLK8VfrnvnVv+3lrWjh2YMQpRxr4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730551AbgBKRwb (ORCPT ); Tue, 11 Feb 2020 12:52:31 -0500 Received: from mail.kernel.org ([198.145.29.99]:55500 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730540AbgBKRwb (ORCPT ); Tue, 11 Feb 2020 12:52:31 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AA6F3206D7; Tue, 11 Feb 2020 17:52:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443550; bh=azVeCUurVCbeNJqhkw5m3Pgfoy9e98pVPeDGxajBlw4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rTso0kUjxIpYKvI1oqUfgWaasE8RolvJxYB6/ubUXrkJD63uiStrgzz8rf6XeES0O bYtSLYDMlbJIkKeDeZW6qJAlJcz7R4/E+8AEP4QnqCpsJufGwLc3AkSUbqpo/XhcYS pgnJqHAZkJG2BbdIFNlOWKrK+0sPtD6EioHzdaa0= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfb-004O7k-Fz; Tue, 11 Feb 2020 17:49:59 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 10/94] KVM: arm64: nv: Inject HVC exceptions to the virtual EL2 Date: Tue, 11 Feb 2020 17:48:14 +0000 Message-Id: <20200211174938.27809-11-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim As we expect all PSCI calls from the L1 hypervisor to be performed using SMC when nested virtualization is enabled, it is clear that all HVC instruction from the VM (including from the virtual EL2) are supposed to handled in the virtual EL2. Forward these to EL2 as required. Signed-off-by: Jintack Lim [maz: add handling of HCR_EL2.HCD] Signed-off-by: Marc Zyngier --- arch/arm64/kvm/handle_exit.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index aacfc55de44c..be5acf72c81e 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include @@ -41,6 +42,16 @@ static int handle_hvc(struct kvm_vcpu *vcpu, struct kvm_run *run) kvm_vcpu_hvc_get_imm(vcpu)); vcpu->stat.hvc_exit_stat++; + /* Forward hvc instructions to the virtual EL2 if the guest has EL2. */ + if (nested_virt_in_use(vcpu)) { + if (vcpu_read_sys_reg(vcpu, HCR_EL2) & HCR_HCD) + kvm_inject_undefined(vcpu); + else + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_hsr(vcpu)); + + return 1; + } + ret = kvm_hvc_call_handler(vcpu); if (ret < 0) { vcpu_set_reg(vcpu, 0, ~0UL); From patchwork Tue Feb 11 17:48:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376251 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B6CF914B4 for ; Tue, 11 Feb 2020 17:52:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 92591208C3 for ; Tue, 11 Feb 2020 17:52:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443560; bh=OwafM765g44dtTZG0G56l5d9Hi2ub1Xv9jM+/4NgrPQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=SFgFYosJS6DIIZwg8aE8dPJs6827qieixxDC7r2E4iAKu72ailcHtfUXP8QajMmKx p7eATZVaqxWBKBHvKIPbrbxex3BgHfZvK0asRezhS3yMyrZTCTQrIRducGRFG5sf7f QwPahpmA64/BxG9JkNMqub4A8JJY0RaCPshb4t+k= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730568AbgBKRwj (ORCPT ); Tue, 11 Feb 2020 12:52:39 -0500 Received: from mail.kernel.org ([198.145.29.99]:55704 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730507AbgBKRwj (ORCPT ); Tue, 11 Feb 2020 12:52:39 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2D66B206CC; Tue, 11 Feb 2020 17:52:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443558; bh=OwafM765g44dtTZG0G56l5d9Hi2ub1Xv9jM+/4NgrPQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OwQooSZohfofco8TorjBM9oVwEOkbR1XgI4BStYPcGYGSFNoIWJN8p1q2ejnfdyK4 nY7d4jOXzbaVy5pLhO2Eup3mYMf8NpneB4N90OT9gpHA0M3pmX+B/SvD0YEDKsZO3I IXyrtFWraDVObSh3HmgF7E67w7hoRQbx9+OH7hbY= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfc-004O7k-0L; Tue, 11 Feb 2020 17:50:00 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 11/94] KVM: arm64: nv: Handle trapped ERET from virtual EL2 Date: Tue, 11 Feb 2020 17:48:15 +0000 Message-Id: <20200211174938.27809-12-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall When a guest hypervisor running virtual EL2 in EL1 executes an ERET instruction, we will have set HCR_EL2.NV which traps ERET to EL2, so that we can emulate the exception return in software. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/esr.h | 5 +++++ arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/kvm/handle_exit.c | 10 ++++++++++ 3 files changed, 16 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index cb29253ae86b..b15921dc889a 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -249,6 +249,11 @@ (((e) & ESR_ELx_SYS64_ISS_OP2_MASK) >> \ ESR_ELx_SYS64_ISS_OP2_SHIFT)) +/* ISS field definitions for ERET/ERETAA/ERETAB trapping */ + +#define ESR_ELx_ERET_ISS_ERET_ERETAx 0x2 +#define ESR_ELx_ERET_ISS_ERETA_ERATAB 0x1 + /* * ISS field definitions for floating-point exception traps * (FP_EXC_32/FP_EXC_64). diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 93469ac81fef..084844ec0b06 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -322,7 +322,7 @@ ECN(SP_ALIGN), ECN(FP_EXC32), ECN(FP_EXC64), ECN(SERROR), \ ECN(BREAKPT_LOW), ECN(BREAKPT_CUR), ECN(SOFTSTP_LOW), \ ECN(SOFTSTP_CUR), ECN(WATCHPT_LOW), ECN(WATCHPT_CUR), \ - ECN(BKPT32), ECN(VECTOR32), ECN(BRK64) + ECN(BKPT32), ECN(VECTOR32), ECN(BRK64), ECN(ERET) #define CPACR_EL1_FPEN (3 << 20) #define CPACR_EL1_TTA (1 << 28) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index be5acf72c81e..c3398e6ebbbe 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -210,6 +210,15 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu, struct kvm_run *run) return 1; } +static int kvm_handle_eret(struct kvm_vcpu *vcpu, struct kvm_run *run) +{ + if (kvm_vcpu_get_hsr(vcpu) & ESR_ELx_ERET_ISS_ERET_ERETAx) + return kvm_handle_ptrauth(vcpu, run); + + kvm_emulate_nested_eret(vcpu); + return 1; +} + static exit_handle_fn arm_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = kvm_handle_unknown_ec, [ESR_ELx_EC_WFx] = kvm_handle_wfx, @@ -224,6 +233,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC64] = handle_smc, [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, [ESR_ELx_EC_SVE] = handle_sve, + [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_DABT_LOW] = kvm_handle_guest_abort, [ESR_ELx_EC_SOFTSTP_LOW]= kvm_handle_guest_debug, From patchwork Tue Feb 11 17:48:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376229 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6D78C1395 for ; Tue, 11 Feb 2020 17:52:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4E28E20870 for ; Tue, 11 Feb 2020 17:52:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443541; bh=6SUf7Shc+/IFCYs1C6g+zQGf7SmOWSMKA9XYRA+43PI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=QW9JpnGIl94iQkr9DjE7z1KaDx9p+tK6yLcw1yOTgar2eWfuWGCNu8s80HejJP/Aa +wNuMota7E/1uUorYB6GvheVpJVwws7DGcjSOLtzIuP28g0W9jARnUCLLvWAlBdIco 0EXxj6Vevs7UffUSulCl3a+3wlbEwMvgeVUu/SL0= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730511AbgBKRwU (ORCPT ); Tue, 11 Feb 2020 12:52:20 -0500 Received: from mail.kernel.org ([198.145.29.99]:55182 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730509AbgBKRwU (ORCPT ); Tue, 11 Feb 2020 12:52:20 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7E2FF20578; Tue, 11 Feb 2020 17:52:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443539; bh=6SUf7Shc+/IFCYs1C6g+zQGf7SmOWSMKA9XYRA+43PI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YYe5WwvV9eLZIYh3pi0tYkueQOc1NtS75Xt9BCKB3kIlxkjSEym1Mygh4GrtSU8jq wXyDAlKkUxlKfy2NE9abDL6f/XPhs2ebqlBrHggTkmJ+XnL6dYI0i9zruPhY/WbJ51 crVvzn4mfzSOEvzjTp3ELbbM91HQVKcOSmX6701U= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfc-004O7k-Ib; Tue, 11 Feb 2020 17:50:00 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 12/94] KVM: arm64: nv: Add EL2->EL1 translation helpers Date: Tue, 11 Feb 2020 17:48:16 +0000 Message-Id: <20200211174938.27809-13-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Some EL2 system registers immediately affect the current execution of the system, so we need to use their respective EL1 counterparts. For this we need to define a mapping between the two. These helpers will get used in subsequent patches. Co-developed-by: Andre Przywara Signed-off-by: Andre Przywara Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 6 ++++ arch/arm64/kvm/sys_regs.c | 48 ++++++++++++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 282e9ddbe1bc..486978d0346b 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -58,6 +58,12 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu); int kvm_inject_nested_sync(struct kvm_vcpu *vcpu, u64 esr_el2); int kvm_inject_nested_irq(struct kvm_vcpu *vcpu); +u64 translate_tcr(u64 tcr); +u64 translate_cptr(u64 tcr); +u64 translate_sctlr(u64 tcr); +u64 translate_ttbr0(u64 tcr); +u64 translate_cnthctl(u64 tcr); + static inline bool vcpu_el1_is_32bit(struct kvm_vcpu *vcpu) { return !(vcpu->arch.hcr_el2 & HCR_RW); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 4b5310ea3bf8..634d3ee6799c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -65,6 +65,54 @@ static bool write_to_read_only(struct kvm_vcpu *vcpu, return false; } +static u64 tcr_el2_ips_to_tcr_el1_ps(u64 tcr_el2) +{ + return ((tcr_el2 & TCR_EL2_PS_MASK) >> TCR_EL2_PS_SHIFT) + << TCR_IPS_SHIFT; +} + +u64 translate_tcr(u64 tcr) +{ + return TCR_EPD1_MASK | /* disable TTBR1_EL1 */ + ((tcr & TCR_EL2_TBI) ? TCR_TBI0 : 0) | + tcr_el2_ips_to_tcr_el1_ps(tcr) | + (tcr & TCR_EL2_TG0_MASK) | + (tcr & TCR_EL2_ORGN0_MASK) | + (tcr & TCR_EL2_IRGN0_MASK) | + (tcr & TCR_EL2_T0SZ_MASK); +} + +u64 translate_cptr(u64 cptr_el2) +{ + u64 cpacr_el1 = 0; + + if (!(cptr_el2 & CPTR_EL2_TFP)) + cpacr_el1 |= CPACR_EL1_FPEN; + if (cptr_el2 & CPTR_EL2_TTA) + cpacr_el1 |= CPACR_EL1_TTA; + if (!(cptr_el2 & CPTR_EL2_TZ)) + cpacr_el1 |= CPACR_EL1_ZEN; + + return cpacr_el1; +} + +u64 translate_sctlr(u64 sctlr) +{ + /* Bit 20 is RES1 in SCTLR_EL1, but RES0 in SCTLR_EL2 */ + return sctlr | BIT(20); +} + +u64 translate_ttbr0(u64 ttbr0) +{ + /* Force ASID to 0 (ASID 0 or RES0) */ + return ttbr0 & ~GENMASK_ULL(63, 48); +} + +u64 translate_cnthctl(u64 cnthctl) +{ + return ((cnthctl & 0x3) << 10) | (cnthctl & 0xfc); +} + u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) { if (!vcpu->arch.sysregs_loaded_on_cpu) From patchwork Tue Feb 11 17:48:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376295 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AE2881395 for ; Tue, 11 Feb 2020 17:53:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7EF0520870 for ; Tue, 11 Feb 2020 17:53:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443600; bh=QY0vOrEBi06uaCWEKz/JMSvWF3lSjw1/mP4A3k4YHko=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=V3yjxYIC3iBeSXZHHBVJ8gpN5O7GZ8SbjOZmPfL3Jdv461ZxWSxCHfCwDdlkHoQ36 TW9iLFlgJC17x6ZSd2tg9VsW564FzuCOzKsY9EZyqIk2jJjAvTrhfRCkX06PzSE9+Y roIktc4dFiUudw7pivn6Ic/0Bw4exockkYdbbgJM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730679AbgBKRxT (ORCPT ); Tue, 11 Feb 2020 12:53:19 -0500 Received: from mail.kernel.org ([198.145.29.99]:56808 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730707AbgBKRxT (ORCPT ); Tue, 11 Feb 2020 12:53:19 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9A86E208C3; Tue, 11 Feb 2020 17:53:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443597; bh=QY0vOrEBi06uaCWEKz/JMSvWF3lSjw1/mP4A3k4YHko=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EwNNhkC9RVAuwBJwVCrb46vUMRF9z0KonhJQ0wgMJ1Wh3e8sJqewsHcGQBGTB+ssR 1nu1+0M4mu92YujrOo7X4aIkZLJssfBMF0nlriO8FTfLWHcqDwvBu9lH3CjvnO6hc+ YRNkcATJNfzuuvZb3QItsYR8hXENZAIUFirPCXdM= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfd-004O7k-3X; Tue, 11 Feb 2020 17:50:01 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 13/94] KVM: arm64: nv: Refactor vcpu_{read,write}_sys_reg Date: Tue, 11 Feb 2020 17:48:17 +0000 Message-Id: <20200211174938.27809-14-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Extract the direct HW accessors for later reuse. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 143 +++++++++++++++++++++++--------------- 1 file changed, 86 insertions(+), 57 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 634d3ee6799c..42a268c21f06 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -113,11 +113,8 @@ u64 translate_cnthctl(u64 cnthctl) return ((cnthctl & 0x3) << 10) | (cnthctl & 0xfc); } -u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) +static bool __vcpu_read_sys_reg_from_cpu(int reg, u64 *val) { - if (!vcpu->arch.sysregs_loaded_on_cpu) - goto immediate_read; - /* * System registers listed in the switch are not saved on every * exit from the guest but are only saved on vcpu_put. @@ -128,40 +125,37 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) * thread when emulating cross-VCPU communication. */ switch (reg) { - case CSSELR_EL1: return read_sysreg_s(SYS_CSSELR_EL1); - case SCTLR_EL1: return read_sysreg_s(SYS_SCTLR_EL12); - case ACTLR_EL1: return read_sysreg_s(SYS_ACTLR_EL1); - case CPACR_EL1: return read_sysreg_s(SYS_CPACR_EL12); - case TTBR0_EL1: return read_sysreg_s(SYS_TTBR0_EL12); - case TTBR1_EL1: return read_sysreg_s(SYS_TTBR1_EL12); - case TCR_EL1: return read_sysreg_s(SYS_TCR_EL12); - case ESR_EL1: return read_sysreg_s(SYS_ESR_EL12); - case AFSR0_EL1: return read_sysreg_s(SYS_AFSR0_EL12); - case AFSR1_EL1: return read_sysreg_s(SYS_AFSR1_EL12); - case FAR_EL1: return read_sysreg_s(SYS_FAR_EL12); - case MAIR_EL1: return read_sysreg_s(SYS_MAIR_EL12); - case VBAR_EL1: return read_sysreg_s(SYS_VBAR_EL12); - case CONTEXTIDR_EL1: return read_sysreg_s(SYS_CONTEXTIDR_EL12); - case TPIDR_EL0: return read_sysreg_s(SYS_TPIDR_EL0); - case TPIDRRO_EL0: return read_sysreg_s(SYS_TPIDRRO_EL0); - case TPIDR_EL1: return read_sysreg_s(SYS_TPIDR_EL1); - case AMAIR_EL1: return read_sysreg_s(SYS_AMAIR_EL12); - case CNTKCTL_EL1: return read_sysreg_s(SYS_CNTKCTL_EL12); - case PAR_EL1: return read_sysreg_s(SYS_PAR_EL1); - case DACR32_EL2: return read_sysreg_s(SYS_DACR32_EL2); - case IFSR32_EL2: return read_sysreg_s(SYS_IFSR32_EL2); - case DBGVCR32_EL2: return read_sysreg_s(SYS_DBGVCR32_EL2); + case CSSELR_EL1: *val = read_sysreg_s(SYS_CSSELR_EL1); break; + case SCTLR_EL1: *val = read_sysreg_s(SYS_SCTLR_EL12); break; + case ACTLR_EL1: *val = read_sysreg_s(SYS_ACTLR_EL1); break; + case CPACR_EL1: *val = read_sysreg_s(SYS_CPACR_EL12); break; + case TTBR0_EL1: *val = read_sysreg_s(SYS_TTBR0_EL12); break; + case TTBR1_EL1: *val = read_sysreg_s(SYS_TTBR1_EL12); break; + case TCR_EL1: *val = read_sysreg_s(SYS_TCR_EL12); break; + case ESR_EL1: *val = read_sysreg_s(SYS_ESR_EL12); break; + case AFSR0_EL1: *val = read_sysreg_s(SYS_AFSR0_EL12); break; + case AFSR1_EL1: *val = read_sysreg_s(SYS_AFSR1_EL12); break; + case FAR_EL1: *val = read_sysreg_s(SYS_FAR_EL12); break; + case MAIR_EL1: *val = read_sysreg_s(SYS_MAIR_EL12); break; + case VBAR_EL1: *val = read_sysreg_s(SYS_VBAR_EL12); break; + case CONTEXTIDR_EL1: *val = read_sysreg_s(SYS_CONTEXTIDR_EL12);break; + case TPIDR_EL0: *val = read_sysreg_s(SYS_TPIDR_EL0); break; + case TPIDRRO_EL0: *val = read_sysreg_s(SYS_TPIDRRO_EL0); break; + case TPIDR_EL1: *val = read_sysreg_s(SYS_TPIDR_EL1); break; + case AMAIR_EL1: *val = read_sysreg_s(SYS_AMAIR_EL12); break; + case CNTKCTL_EL1: *val = read_sysreg_s(SYS_CNTKCTL_EL12); break; + case PAR_EL1: *val = read_sysreg_s(SYS_PAR_EL1); break; + case DACR32_EL2: *val = read_sysreg_s(SYS_DACR32_EL2); break; + case IFSR32_EL2: *val = read_sysreg_s(SYS_IFSR32_EL2); break; + case DBGVCR32_EL2: *val = read_sysreg_s(SYS_DBGVCR32_EL2); break; + default: return false; } -immediate_read: - return __vcpu_sys_reg(vcpu, reg); + return true; } -void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) +static bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg) { - if (!vcpu->arch.sysregs_loaded_on_cpu) - goto immediate_write; - /* * System registers listed in the switch are not restored on every * entry to the guest but are only restored on vcpu_load. @@ -171,32 +165,67 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) * set once, before running the VCPU, and never changed later. */ switch (reg) { - case CSSELR_EL1: write_sysreg_s(val, SYS_CSSELR_EL1); return; - case SCTLR_EL1: write_sysreg_s(val, SYS_SCTLR_EL12); return; - case ACTLR_EL1: write_sysreg_s(val, SYS_ACTLR_EL1); return; - case CPACR_EL1: write_sysreg_s(val, SYS_CPACR_EL12); return; - case TTBR0_EL1: write_sysreg_s(val, SYS_TTBR0_EL12); return; - case TTBR1_EL1: write_sysreg_s(val, SYS_TTBR1_EL12); return; - case TCR_EL1: write_sysreg_s(val, SYS_TCR_EL12); return; - case ESR_EL1: write_sysreg_s(val, SYS_ESR_EL12); return; - case AFSR0_EL1: write_sysreg_s(val, SYS_AFSR0_EL12); return; - case AFSR1_EL1: write_sysreg_s(val, SYS_AFSR1_EL12); return; - case FAR_EL1: write_sysreg_s(val, SYS_FAR_EL12); return; - case MAIR_EL1: write_sysreg_s(val, SYS_MAIR_EL12); return; - case VBAR_EL1: write_sysreg_s(val, SYS_VBAR_EL12); return; - case CONTEXTIDR_EL1: write_sysreg_s(val, SYS_CONTEXTIDR_EL12); return; - case TPIDR_EL0: write_sysreg_s(val, SYS_TPIDR_EL0); return; - case TPIDRRO_EL0: write_sysreg_s(val, SYS_TPIDRRO_EL0); return; - case TPIDR_EL1: write_sysreg_s(val, SYS_TPIDR_EL1); return; - case AMAIR_EL1: write_sysreg_s(val, SYS_AMAIR_EL12); return; - case CNTKCTL_EL1: write_sysreg_s(val, SYS_CNTKCTL_EL12); return; - case PAR_EL1: write_sysreg_s(val, SYS_PAR_EL1); return; - case DACR32_EL2: write_sysreg_s(val, SYS_DACR32_EL2); return; - case IFSR32_EL2: write_sysreg_s(val, SYS_IFSR32_EL2); return; - case DBGVCR32_EL2: write_sysreg_s(val, SYS_DBGVCR32_EL2); return; + case CSSELR_EL1: write_sysreg_s(val, SYS_CSSELR_EL1); break; + case SCTLR_EL1: write_sysreg_s(val, SYS_SCTLR_EL12); break; + case ACTLR_EL1: write_sysreg_s(val, SYS_ACTLR_EL1); break; + case CPACR_EL1: write_sysreg_s(val, SYS_CPACR_EL12); break; + case TTBR0_EL1: write_sysreg_s(val, SYS_TTBR0_EL12); break; + case TTBR1_EL1: write_sysreg_s(val, SYS_TTBR1_EL12); break; + case TCR_EL1: write_sysreg_s(val, SYS_TCR_EL12); break; + case ESR_EL1: write_sysreg_s(val, SYS_ESR_EL12); break; + case AFSR0_EL1: write_sysreg_s(val, SYS_AFSR0_EL12); break; + case AFSR1_EL1: write_sysreg_s(val, SYS_AFSR1_EL12); break; + case FAR_EL1: write_sysreg_s(val, SYS_FAR_EL12); break; + case MAIR_EL1: write_sysreg_s(val, SYS_MAIR_EL12); break; + case VBAR_EL1: write_sysreg_s(val, SYS_VBAR_EL12); break; + case CONTEXTIDR_EL1: write_sysreg_s(val, SYS_CONTEXTIDR_EL12);break; + case TPIDR_EL0: write_sysreg_s(val, SYS_TPIDR_EL0); break; + case TPIDRRO_EL0: write_sysreg_s(val, SYS_TPIDRRO_EL0); break; + case TPIDR_EL1: write_sysreg_s(val, SYS_TPIDR_EL1); break; + case AMAIR_EL1: write_sysreg_s(val, SYS_AMAIR_EL12); break; + case CNTKCTL_EL1: write_sysreg_s(val, SYS_CNTKCTL_EL12); break; + case PAR_EL1: write_sysreg_s(val, SYS_PAR_EL1); break; + case DACR32_EL2: write_sysreg_s(val, SYS_DACR32_EL2); break; + case IFSR32_EL2: write_sysreg_s(val, SYS_IFSR32_EL2); break; + case DBGVCR32_EL2: write_sysreg_s(val, SYS_DBGVCR32_EL2); break; + default: return false; + } + + return true; +} + +u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) +{ + u64 val = 0x8badf00d8badf00d; + + if (!vcpu->arch.sysregs_loaded_on_cpu) { + goto memory_read; } -immediate_write: + /* EL1 register can't be on the CPU if the guest is in vEL2. */ + if (unlikely(is_hyp_ctxt(vcpu))) + goto memory_read; + + if (__vcpu_read_sys_reg_from_cpu(reg, &val)) + return val; + +memory_read: + return __vcpu_sys_reg(vcpu, reg); +} + +void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) +{ + if (!vcpu->arch.sysregs_loaded_on_cpu) + goto memory_write; + + /* EL1 register can't be on the CPU if the guest is in vEL2. */ + if (unlikely(is_hyp_ctxt(vcpu))) + goto memory_write; + + if (__vcpu_write_sys_reg_to_cpu(val, reg)) + return; + +memory_write: __vcpu_sys_reg(vcpu, reg) = val; } From patchwork Tue Feb 11 17:48:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376291 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E9A9D14E3 for ; Tue, 11 Feb 2020 17:53:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BF8B4206D7 for ; Tue, 11 Feb 2020 17:53:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443595; bh=FUObRX1i5yYjDDdV90Q3AHebzA2hM6GwZwnuM2X/+0A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Bz7UitUIsARhtcQ2Cs4eBkyIv7/5VaxisotW+EhNYsO/oQijOEGQVNgto/hGZGVqZ QVZYBgrJozA7nQiUhsmQjfn7VDkSNa1CPGTVHgoxGocNgzA7fGdXWLB9fQ0Z3iUdcg xzdvR9j2HmK+MUH5C9T/dXVPBGf0AEZ6Ij839MGQ= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730658AbgBKRxP (ORCPT ); Tue, 11 Feb 2020 12:53:15 -0500 Received: from mail.kernel.org ([198.145.29.99]:56706 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730657AbgBKRxO (ORCPT ); Tue, 11 Feb 2020 12:53:14 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E296D20661; Tue, 11 Feb 2020 17:53:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443594; bh=FUObRX1i5yYjDDdV90Q3AHebzA2hM6GwZwnuM2X/+0A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=HZwWvTFctOfcTpC0Hf84F/pPsaUV0lORAgSdwrWdnSI0tHczAcA676pWGeBhKuRVa uR6VMyXM2eZrQUH7Ian7wZi0TiIZ/L0CT07digHlsaUS+SYn5PJ6YetRUChtCGnTjc jWc0o6JH04mvLnUJyCI6/ec+FeFyrrUjkEpI077A= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfd-004O7k-Lp; Tue, 11 Feb 2020 17:50:01 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 14/94] KVM: arm64: nv: Handle virtual EL2 registers in vcpu_read/write_sys_reg() Date: Tue, 11 Feb 2020 17:48:18 +0000 Message-Id: <20200211174938.27809-15-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org KVM internally uses accessor functions when reading or writing the guest's system registers. This takes care of accessing either the stored copy or using the "live" EL1 system registers when the host uses VHE. With the introduction of virtual EL2 we add a bunch of EL2 system registers, which now must also be taken care of: - If the guest is running in vEL2, and we access an EL1 sysreg, we must revert to the stored version of that, and not use the CPU's copy. - If the guest is running in vEL1, and we access an EL2 sysreg, we must also use the stored version, since the CPU carries the EL1 copy. - Some EL2 system registers are supposed to affect the current execution of the system, so we need to put them into their respective EL1 counterparts. For this we need to define a mapping between the two. This is done using the newly introduced struct el2_sysreg_map. - Some EL2 system registers have a different format than their EL1 counterpart, so we need to translate them before writing them to the CPU. This is done using an (optional) translate function in the map. - There are the three special registers SP_EL2, SPSR_EL2 and ELR_EL2, which need some separate handling (SPSR_EL2 is being handled in a separate patch). All of these cases are now wrapped into the existing accessor functions, so KVM users wouldn't need to care whether they access EL2 or EL1 registers and also which state the guest is in. This handles what was formerly known as the "shadow state" dynamically, without requiring a separate copy for each vCPU EL. Co-developed-by: Andre Przywara Signed-off-by: Andre Przywara Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 117 +++++++++++++++++++++++++++++++++++++- 1 file changed, 116 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 42a268c21f06..64be9f452ad6 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -113,6 +113,56 @@ u64 translate_cnthctl(u64 cnthctl) return ((cnthctl & 0x3) << 10) | (cnthctl & 0xfc); } +#define PURE_EL2_SYSREG(el2) \ + case el2: { \ + *el1r = el2; \ + return true; \ + } + +#define MAPPED_EL2_SYSREG(el2, el1, fn) \ + case el2: { \ + *xlate = fn; \ + *el1r = el1; \ + return true; \ + } + +static bool get_el2_mapping(unsigned int reg, + unsigned int *el1r, u64 (**xlate)(u64)) +{ + switch (reg) { + PURE_EL2_SYSREG( VPIDR_EL2 ); + PURE_EL2_SYSREG( VMPIDR_EL2 ); + PURE_EL2_SYSREG( ACTLR_EL2 ); + PURE_EL2_SYSREG( HCR_EL2 ); + PURE_EL2_SYSREG( MDCR_EL2 ); + PURE_EL2_SYSREG( HSTR_EL2 ); + PURE_EL2_SYSREG( HACR_EL2 ); + PURE_EL2_SYSREG( VTTBR_EL2 ); + PURE_EL2_SYSREG( VTCR_EL2 ); + PURE_EL2_SYSREG( RVBAR_EL2 ); + PURE_EL2_SYSREG( RMR_EL2 ); + PURE_EL2_SYSREG( TPIDR_EL2 ); + PURE_EL2_SYSREG( HPFAR_EL2 ); + PURE_EL2_SYSREG( ELR_EL2 ); + PURE_EL2_SYSREG( SPSR_EL2 ); + MAPPED_EL2_SYSREG(SCTLR_EL2, SCTLR_EL1, translate_sctlr ); + MAPPED_EL2_SYSREG(CPTR_EL2, CPACR_EL1, translate_cptr ); + MAPPED_EL2_SYSREG(TTBR0_EL2, TTBR0_EL1, translate_ttbr0 ); + MAPPED_EL2_SYSREG(TTBR1_EL2, TTBR1_EL1, NULL ); + MAPPED_EL2_SYSREG(TCR_EL2, TCR_EL1, translate_tcr ); + MAPPED_EL2_SYSREG(VBAR_EL2, VBAR_EL1, NULL ); + MAPPED_EL2_SYSREG(AFSR0_EL2, AFSR0_EL1, NULL ); + MAPPED_EL2_SYSREG(AFSR1_EL2, AFSR1_EL1, NULL ); + MAPPED_EL2_SYSREG(ESR_EL2, ESR_EL1, NULL ); + MAPPED_EL2_SYSREG(FAR_EL2, FAR_EL1, NULL ); + MAPPED_EL2_SYSREG(MAIR_EL2, MAIR_EL1, NULL ); + MAPPED_EL2_SYSREG(AMAIR_EL2, AMAIR_EL1, NULL ); + MAPPED_EL2_SYSREG(CNTHCTL_EL2, CNTKCTL_EL1, translate_cnthctl); + default: + return false; + } +} + static bool __vcpu_read_sys_reg_from_cpu(int reg, u64 *val) { /* @@ -197,9 +247,42 @@ static bool __vcpu_write_sys_reg_to_cpu(u64 val, int reg) u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) { u64 val = 0x8badf00d8badf00d; + u64 (*xlate)(u64) = NULL; + unsigned int el1r; - if (!vcpu->arch.sysregs_loaded_on_cpu) { + if (!vcpu->arch.sysregs_loaded_on_cpu) goto memory_read; + + if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) { + if (!is_hyp_ctxt(vcpu)) + goto memory_read; + + /* + * ELR_EL2 is special cased for now. + */ + switch (reg) { + case ELR_EL2: + return read_sysreg_el1(SYS_ELR); + } + + /* + * If this register does not have an EL1 counterpart, + * then read the stored EL2 version. + */ + if (reg == el1r) + goto memory_read; + + /* + * If we have a non-VHE guest and that the sysreg + * requires translation to be used at EL1, use the + * in-memory copy instead. + */ + if (!vcpu_el2_e2h_is_set(vcpu) && xlate) + goto memory_read; + + /* Get the current version of the EL1 counterpart. */ + WARN_ON(!__vcpu_read_sys_reg_from_cpu(el1r, &val)); + return val; } /* EL1 register can't be on the CPU if the guest is in vEL2. */ @@ -215,9 +298,41 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) { + u64 (*xlate)(u64) = NULL; + unsigned int el1r; + if (!vcpu->arch.sysregs_loaded_on_cpu) goto memory_write; + if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) { + if (!is_hyp_ctxt(vcpu)) + goto memory_write; + + /* + * Always store a copy of the write to memory to avoid having + * to reverse-translate virtual EL2 system registers for a + * non-VHE guest hypervisor. + */ + __vcpu_sys_reg(vcpu, reg) = val; + + switch (reg) { + case ELR_EL2: + write_sysreg_el1(val, SYS_ELR); + return; + } + + /* No EL1 counterpart? We're done here.? */ + if (reg == el1r) + return; + + if (!vcpu_el2_e2h_is_set(vcpu) && xlate) + val = xlate(val); + + /* Redirect this to the EL1 version of the register. */ + WARN_ON(!__vcpu_write_sys_reg_to_cpu(val, el1r)); + return; + } + /* EL1 register can't be on the CPU if the guest is in vEL2. */ if (unlikely(is_hyp_ctxt(vcpu))) goto memory_write; From patchwork Tue Feb 11 17:48:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376283 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 503B114B4 for ; Tue, 11 Feb 2020 17:53:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2E98820870 for ; Tue, 11 Feb 2020 17:53:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443589; bh=gtQO6DnA30segjYAYUIYRQkYSFgAx9NxUhck8T27+04=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=SqEAPGF84pD3k3DJ578xyS0hO+1dX2nhOxW9ZPtoWCT9Z1QTHVe/RD2iYQmizEdjf BPXY+dhW1XJZdThWwq+PT+GvqxB2CyJPvBydrhWpy094pEnmpd5V5GOmk+xqajLirI xIi+RyAYcnhYUhuSU/kS+LY7k4YuaNzawZfuqMr0= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730604AbgBKRxI (ORCPT ); Tue, 11 Feb 2020 12:53:08 -0500 Received: from mail.kernel.org ([198.145.29.99]:56458 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730673AbgBKRxH (ORCPT ); Tue, 11 Feb 2020 12:53:07 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6A10220578; Tue, 11 Feb 2020 17:53:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443586; bh=gtQO6DnA30segjYAYUIYRQkYSFgAx9NxUhck8T27+04=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n/mDxxc3ah/TZ1sf3Rb8hEdp6xg6O/JZIuWhXYpHHhctmkHjf6G86fqdSkCauCkdE 4g6W/HOAHsO4txgrMfKpwSMQtCyW1zZvVk6ekEM8vcSkdh2qzfg6atWMH7TqEaFn79 Ee84VAd94Vryh+BEo0nkXfyhxCtgAwNzhpgOT6q0= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfe-004O7k-8B; Tue, 11 Feb 2020 17:50:02 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 15/94] KVM: arm64: nv: Handle SPSR_EL2 specially Date: Tue, 11 Feb 2020 17:48:19 +0000 Message-Id: <20200211174938.27809-16-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org SPSR_EL2 needs special attention when running nested on ARMv8.3: If taking an exception while running at vEL2 (actually EL1), the HW will update the SPSR_EL1 register with the EL1 mode. We need to track this in order to make sure that accesses to the virtual view of SPSR_EL2 is correct. To do so, we place an illegal value in SPSR_EL1.M, and patch it accordingly if required when accessing it. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 45 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 23 ++++++++++++-- 2 files changed, 66 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 486978d0346b..26552c8571cb 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -277,11 +277,51 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) return __is_hyp_ctxt(&vcpu->arch.ctxt); } +static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val) +{ + if (!__vcpu_el2_e2h_is_set(ctxt)) { + /* + * Clear the .M field when writing SPSR to the CPU, so that we + * can detect when the CPU clobbered our SPSR copy during a + * local exception. + */ + val &= ~0xc; + } + + return val; +} + +static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val) +{ + if (__vcpu_el2_e2h_is_set(ctxt)) + return val; + + /* + * SPSR.M == 0 means the CPU has not touched the SPSR, so the + * register has still the value we saved on the last write. + */ + if ((val & 0xc) == 0) + return ctxt->sys_regs[SPSR_EL2]; + + /* + * Otherwise there was a "local" exception on the CPU, + * which from the guest's point of view was being taken from + * EL2 to EL2, although it actually happened to be from + * EL1 to EL1. + * So we need to fix the .M field in SPSR, to make it look + * like EL2, which is what the guest would expect. + */ + return (val & ~0x0c) | CurrentEL_EL2; +} + static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu) { if (vcpu_mode_is_32bit(vcpu)) return vcpu_read_spsr32(vcpu); + if (unlikely(vcpu_mode_el2(vcpu))) + return vcpu_read_sys_reg(vcpu, SPSR_EL2); + if (vcpu->arch.sysregs_loaded_on_cpu) return read_sysreg_el1(SYS_SPSR); else @@ -295,6 +335,11 @@ static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v) return; } + if (unlikely(vcpu_mode_el2(vcpu))) { + vcpu_write_sys_reg(vcpu, v, SPSR_EL2); + return; + } + if (vcpu->arch.sysregs_loaded_on_cpu) write_sysreg_el1(v, SYS_SPSR); else diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 64be9f452ad6..8c7d3d410689 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -258,11 +258,14 @@ u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg) goto memory_read; /* - * ELR_EL2 is special cased for now. + * ELR_EL2 and SPSR_EL2 are special cased for now. */ switch (reg) { case ELR_EL2: return read_sysreg_el1(SYS_ELR); + case SPSR_EL2: + val = read_sysreg_el1(SYS_SPSR); + return __fixup_spsr_el2_read(&vcpu->arch.ctxt, val); } /* @@ -319,6 +322,10 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) case ELR_EL2: write_sysreg_el1(val, SYS_ELR); return; + case SPSR_EL2: + val = __fixup_spsr_el2_write(&vcpu->arch.ctxt, val); + write_sysreg_el1(val, SYS_SPSR); + return; } /* No EL1 counterpart? We're done here.? */ @@ -1589,6 +1596,18 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool access_spsr_el2(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); + else + p->regval = vcpu_read_sys_reg(vcpu, SPSR_EL2); + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -1899,7 +1918,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_VTCR_EL2), access_rw, reset_val, VTCR_EL2, 0 }, { SYS_DESC(SYS_DACR32_EL2), NULL, reset_unknown, DACR32_EL2 }, - { SYS_DESC(SYS_SPSR_EL2), access_rw, reset_val, SPSR_EL2, 0 }, + { SYS_DESC(SYS_SPSR_EL2), access_spsr_el2, reset_val, SPSR_EL2, 0 }, { SYS_DESC(SYS_ELR_EL2), access_rw, reset_val, ELR_EL2, 0 }, { SYS_DESC(SYS_SP_EL1), access_sp_el1}, From patchwork Tue Feb 11 17:48:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376269 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 28F9B14E3 for ; Tue, 11 Feb 2020 17:52:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 09B5B21569 for ; Tue, 11 Feb 2020 17:52:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443578; bh=9e053oblaYC4Xt2ZtPRijFPsHC8lgIne8+rer6142/k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=i1l3klK52uVciDdWO2MBus7CvZ5lcutL0WURA6kFXveZf9nfaJS7jMehqjFfqAGri fvI8z0scQgGdciWOHfCyAk/Bw9+TtdGQFmw31z11aDjgIz8WjUguZ8rUYui5t2PfFi KZA1L341Yr+O2kT0DxW8JSYtxvzRq+/XVnkz+qL4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728918AbgBKRwz (ORCPT ); Tue, 11 Feb 2020 12:52:55 -0500 Received: from mail.kernel.org ([198.145.29.99]:56124 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730635AbgBKRwy (ORCPT ); Tue, 11 Feb 2020 12:52:54 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4707A20848; Tue, 11 Feb 2020 17:52:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443573; bh=9e053oblaYC4Xt2ZtPRijFPsHC8lgIne8+rer6142/k=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=hvJIclTxiKcKlKy0VMbJSupzBOv4HMiSoUkTvFQYdzSrB4zPWibWt0C5wUfyt4hzE cWcomoDJ5PqbDW8XvDjDhB3UZr3LX9Nq0yAOwNGU1EFugk8W7yBF4NmZK4npZrIVlO nB5rcaKfCeTiG5NtcdmJiP7cjLWKamOKf50DU7n8= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfe-004O7k-Rb; Tue, 11 Feb 2020 17:50:02 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 16/94] KVM: arm64: nv: Handle HCR_EL2.E2H specially Date: Tue, 11 Feb 2020 17:48:20 +0000 Message-Id: <20200211174938.27809-17-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org HCR_EL2.E2H is nasty, as a flip of this bit completely changes the way we deal with a lot of the state. So when the guest flips this bit (sysregs are live), do the put/load dance so that we have a consistent state. Yes, this is slow. Don't do it. Suggested-by: Alexandru Elisei Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 8c7d3d410689..9b29ac37829b 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -308,9 +308,24 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) goto memory_write; if (unlikely(get_el2_mapping(reg, &el1r, &xlate))) { + bool need_put_load; + if (!is_hyp_ctxt(vcpu)) goto memory_write; + /* + * HCR_EL2.E2H is nasty: it changes the way we interpret a + * lot of the EL2 state, so treat is as a full state + * transition. + */ + need_put_load = ((reg == HCR_EL2) && + vcpu_el2_e2h_is_set(vcpu) != !!(val & HCR_E2H)); + + if (need_put_load) { + preempt_disable(); + kvm_arch_vcpu_put(vcpu); + } + /* * Always store a copy of the write to memory to avoid having * to reverse-translate virtual EL2 system registers for a @@ -318,6 +333,11 @@ void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg) */ __vcpu_sys_reg(vcpu, reg) = val; + if (need_put_load) { + kvm_arch_vcpu_load(vcpu, smp_processor_id()); + preempt_enable(); + } + switch (reg) { case ELR_EL2: write_sysreg_el1(val, SYS_ELR); From patchwork Tue Feb 11 17:48:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376305 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01F3B14B4 for ; Tue, 11 Feb 2020 17:53:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C82DC20870 for ; Tue, 11 Feb 2020 17:53:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443609; bh=1k+eFdeqZiyBy/ePjSfoyQp4vInj3/3+ncVL78Lab8o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=lk6bB5c9Ye/8NQJP2NjvM54Xu85R2kbiflmQZoe2lS+pJ78whBrZEkD7ZFsRXlb7Q egzFQgAMkfn6fe6PR2i0Vzuf5eUqoO8ct4987pfOK+stztNu9T1GqzV1PC8FdaXQd6 tUxM2ug/Q7aG5jaWoh7TF61E7OOhqNsWq+PHa4Fk= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730750AbgBKRx2 (ORCPT ); Tue, 11 Feb 2020 12:53:28 -0500 Received: from mail.kernel.org ([198.145.29.99]:57028 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730671AbgBKRx2 (ORCPT ); Tue, 11 Feb 2020 12:53:28 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DE55F206CC; Tue, 11 Feb 2020 17:53:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443607; bh=1k+eFdeqZiyBy/ePjSfoyQp4vInj3/3+ncVL78Lab8o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=zfT8vd3eeMpexNAyieRup0brrwNrvRI+NPZ0Jinp2w15Ndp6MOpMAKIEdVTpZIYw1 Lrz5FNVuq6q+UsVMj4h2Krj9GR6tJuSEPaBDUB/DgN66xnbboMilNLG4jAUgtpzWBS ZHMsLCfRvSGriCgBK84/pWUukULR4iorfbrV9QOc= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zff-004O7k-Cq; Tue, 11 Feb 2020 17:50:03 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 17/94] KVM: arm64: nv: Save/Restore vEL2 sysregs Date: Tue, 11 Feb 2020 17:48:21 +0000 Message-Id: <20200211174938.27809-18-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Andre Przywara Whenever we need to restore the guest's system registers to the CPU, we now need to take care of the EL2 system registers as well. Most of them are accessed via traps only, but some have an immediate effect and also a guest running in VHE mode would expect them to be accessible via their EL1 encoding, which we do not trap. Split the current __sysreg_{save,restore}_el1_state() functions into handling common sysregs, then differentiate between the guest running in vEL2 and vEL1. For vEL2 we write the virtual EL2 registers with an identical format directly into their EL1 counterpart, and translate the few registers that have a different format for the same effect on the execution when running a non-VHE guest guest hypervisor. [ Commit message reworked and many bug fixes applied by Marc Zyngier and Christoffer Dall. ] Signed-off-by: Andre Przywara Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/sysreg-sr.c | 171 +++++++++++++++++++++++++++++++-- 1 file changed, 164 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index 7672a978926c..ed22301fd2e3 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -11,6 +11,7 @@ #include #include #include +#include /* * Non-VHE: Both host and guest must save everything. @@ -40,11 +41,9 @@ static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt) ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0); } -static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) +static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) { - ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1); ctxt->sys_regs[SCTLR_EL1] = read_sysreg_el1(SYS_SCTLR); - ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1); ctxt->sys_regs[CPACR_EL1] = read_sysreg_el1(SYS_CPACR); ctxt->sys_regs[TTBR0_EL1] = read_sysreg_el1(SYS_TTBR0); ctxt->sys_regs[TTBR1_EL1] = read_sysreg_el1(SYS_TTBR1); @@ -58,14 +57,58 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) ctxt->sys_regs[CONTEXTIDR_EL1] = read_sysreg_el1(SYS_CONTEXTIDR); ctxt->sys_regs[AMAIR_EL1] = read_sysreg_el1(SYS_AMAIR); ctxt->sys_regs[CNTKCTL_EL1] = read_sysreg_el1(SYS_CNTKCTL); - ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); - ctxt->sys_regs[TPIDR_EL1] = read_sysreg(tpidr_el1); ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1); ctxt->gp_regs.elr_el1 = read_sysreg_el1(SYS_ELR); ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(SYS_SPSR); } +static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) +{ + ctxt->sys_regs[ESR_EL2] = read_sysreg_el1(SYS_ESR); + ctxt->sys_regs[AFSR0_EL2] = read_sysreg_el1(SYS_AFSR0); + ctxt->sys_regs[AFSR1_EL2] = read_sysreg_el1(SYS_AFSR1); + ctxt->sys_regs[FAR_EL2] = read_sysreg_el1(SYS_FAR); + ctxt->sys_regs[MAIR_EL2] = read_sysreg_el1(SYS_MAIR); + ctxt->sys_regs[VBAR_EL2] = read_sysreg_el1(SYS_VBAR); + ctxt->sys_regs[CONTEXTIDR_EL2] = read_sysreg_el1(SYS_CONTEXTIDR); + ctxt->sys_regs[AMAIR_EL2] = read_sysreg_el1(SYS_AMAIR); + + /* + * In VHE mode those registers are compatible between EL1 and EL2, + * and the guest uses the _EL1 versions on the CPU naturally. + * So we save them into their _EL2 versions here. + * For nVHE mode we trap accesses to those registers, so our + * _EL2 copy in sys_regs[] is always up-to-date and we don't need + * to save anything here. + */ + if (__vcpu_el2_e2h_is_set(ctxt)) { + ctxt->sys_regs[SCTLR_EL2] = read_sysreg_el1(SYS_SCTLR); + ctxt->sys_regs[CPTR_EL2] = read_sysreg_el1(SYS_CPACR); + ctxt->sys_regs[TTBR0_EL2] = read_sysreg_el1(SYS_TTBR0); + ctxt->sys_regs[TTBR1_EL2] = read_sysreg_el1(SYS_TTBR1); + ctxt->sys_regs[TCR_EL2] = read_sysreg_el1(SYS_TCR); + ctxt->sys_regs[CNTHCTL_EL2] = read_sysreg_el1(SYS_CNTKCTL); + } + + ctxt->sys_regs[SP_EL2] = read_sysreg(sp_el1); + ctxt->sys_regs[ELR_EL2] = read_sysreg_el1(SYS_ELR); + ctxt->sys_regs[SPSR_EL2] = __fixup_spsr_el2_read(ctxt, read_sysreg_el1(SYS_SPSR)); +} + +static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) +{ + ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1); + ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1); + ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); + ctxt->sys_regs[TPIDR_EL1] = read_sysreg(tpidr_el1); + + if (unlikely(__is_hyp_ctxt(ctxt))) + __sysreg_save_vel2_state(ctxt); + else + __sysreg_save_vel1_state(ctxt); +} + static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { ctxt->gp_regs.regs.pc = read_sysreg_el2(SYS_ELR); @@ -113,10 +156,102 @@ static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0); } -static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) +static void __sysreg_restore_vel2_state(struct kvm_cpu_context *ctxt) { + u64 val; + + write_sysreg(read_cpuid_id(), vpidr_el2); write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2); - write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1); + write_sysreg_el1(ctxt->sys_regs[MAIR_EL2], SYS_MAIR); + write_sysreg_el1(ctxt->sys_regs[VBAR_EL2], SYS_VBAR); + write_sysreg_el1(ctxt->sys_regs[CONTEXTIDR_EL2],SYS_CONTEXTIDR); + write_sysreg_el1(ctxt->sys_regs[AMAIR_EL2], SYS_AMAIR); + + if (__vcpu_el2_e2h_is_set(ctxt)) { + /* + * In VHE mode those registers are compatible between + * EL1 and EL2. + */ + write_sysreg_el1(ctxt->sys_regs[SCTLR_EL2], SYS_SCTLR); + write_sysreg_el1(ctxt->sys_regs[CPTR_EL2], SYS_CPACR); + write_sysreg_el1(ctxt->sys_regs[TTBR0_EL2], SYS_TTBR0); + write_sysreg_el1(ctxt->sys_regs[TTBR1_EL2], SYS_TTBR1); + write_sysreg_el1(ctxt->sys_regs[TCR_EL2], SYS_TCR); + write_sysreg_el1(ctxt->sys_regs[CNTHCTL_EL2], SYS_CNTKCTL); + } else { + write_sysreg_el1(translate_sctlr(ctxt->sys_regs[SCTLR_EL2]), + SYS_SCTLR); + write_sysreg_el1(translate_cptr(ctxt->sys_regs[CPTR_EL2]), + SYS_CPACR); + write_sysreg_el1(translate_ttbr0(ctxt->sys_regs[TTBR0_EL2]), + SYS_TTBR0); + write_sysreg_el1(translate_tcr(ctxt->sys_regs[TCR_EL2]), + SYS_TCR); + write_sysreg_el1(translate_cnthctl(ctxt->sys_regs[CNTHCTL_EL2]), + SYS_CNTKCTL); + } + + /* + * These registers can be modified behind our back by a fault + * taken inside vEL2. Save them, always. + */ + write_sysreg_el1(ctxt->sys_regs[ESR_EL2], SYS_ESR); + write_sysreg_el1(ctxt->sys_regs[AFSR0_EL2], SYS_AFSR0); + write_sysreg_el1(ctxt->sys_regs[AFSR1_EL2], SYS_AFSR1); + write_sysreg_el1(ctxt->sys_regs[FAR_EL2], SYS_FAR); + write_sysreg(ctxt->sys_regs[SP_EL2], sp_el1); + write_sysreg_el1(ctxt->sys_regs[ELR_EL2], SYS_ELR); + + val = __fixup_spsr_el2_write(ctxt, ctxt->sys_regs[SPSR_EL2]); + write_sysreg_el1(val, SYS_SPSR); +} + +static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) +{ + u64 mpidr; + + if (has_vhe()) { + struct kvm_vcpu *vcpu; + + /* + * We need to go from a context to a vcpu, but this is a + * complicated affair. + * + * On VHE, we should never be here with the host context as + * a parameter, so let's check and bail out if that's the + * case. + */ + if (WARN_ON_ONCE(ctxt->__hyp_running_vcpu)) + return; + + /* + * Now that we know for sure this is a guest context, we can + * extract the vcpu... + */ + vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + + if (nested_virt_in_use(vcpu)) { + /* + * Only set VPIDR_EL2 for nested VMs, as this is the + * only time it changes. We'll restore the MIDR_EL1 + * view on put. + */ + write_sysreg(ctxt->sys_regs[VPIDR_EL2], vpidr_el2); + + /* + * As we're restoring a nested guest, set the value + * provided by the guest hypervisor. + */ + mpidr = ctxt->sys_regs[VMPIDR_EL2]; + } else { + mpidr = ctxt->sys_regs[MPIDR_EL1]; + } + } else { + mpidr = ctxt->sys_regs[MPIDR_EL1]; + } + + write_sysreg(mpidr, vmpidr_el2); + write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); if (!cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) { write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); @@ -171,6 +306,19 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],SYS_SPSR); } +static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) +{ + write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1); + write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1); + write_sysreg(ctxt->sys_regs[PAR_EL1], par_el1); + write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1); + + if (__is_hyp_ctxt(ctxt)) + __sysreg_restore_vel2_state(ctxt); + else + __sysreg_restore_vel1_state(ctxt); +} + static void __hyp_text __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) { @@ -327,6 +475,15 @@ void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu) /* Restore host user state */ __sysreg_restore_user_state(host_ctxt); + /* + * If leaving a nesting guest, restore MPIDR_EL1 default view. It is + * slightly ugly to do it here, but the alternative is to penalize + * all non-nesting guests by forcing this on every load. Instead, we + * choose to only penalize nesting VMs. + */ + if (nested_virt_in_use(vcpu)) + write_sysreg(read_cpuid_id(), vpidr_el2); + vcpu->arch.sysregs_loaded_on_cpu = false; } From patchwork Tue Feb 11 17:48:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376209 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C980414B4 for ; Tue, 11 Feb 2020 17:52:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A825320848 for ; Tue, 11 Feb 2020 17:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443524; bh=2/8o/Nb0OGN1zviltRK0lrSBMaj2rdiiHzQj0Qa5JmY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=HIoYMSVSmE4gxRyAGvfKKBirEryXG9wugq4l2brPB0KQEDPA5MJ/zU5BZHBuHOoi0 rgW4wS33qUO9Wdk3YTXsC0xTwH/51XR7uyRC9aJB6Un9DXVfPj+WIBPS8h7AHX8VxX Gy2BeXyMJs5t/asI8iNG3yb2u7p76c3GdiZSAPvc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730422AbgBKRwD (ORCPT ); Tue, 11 Feb 2020 12:52:03 -0500 Received: from mail.kernel.org ([198.145.29.99]:54730 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730345AbgBKRwD (ORCPT ); Tue, 11 Feb 2020 12:52:03 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 47FE620578; Tue, 11 Feb 2020 17:52:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443522; bh=2/8o/Nb0OGN1zviltRK0lrSBMaj2rdiiHzQj0Qa5JmY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ad7ffBhwOzpckykfwEZcMTrlzlSu3M9TA6rhnyMXtDVdVyIydHlrQNOtCDbuxuXgx 6UDQzRjBs/LKN0sTdO1pbvi/nTZ+xsrsYz6jLciXoYH9lDBNd002KmL68h0OuSpBbY gUcylj6AREcOzJ+DvAt+cflCPFLW8UzZCx5o2iWQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zff-004O7k-W8; Tue, 11 Feb 2020 17:50:04 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 18/94] KVM: arm64: nv: Emulate PSTATE.M for a guest hypervisor Date: Tue, 11 Feb 2020 17:48:22 +0000 Message-Id: <20200211174938.27809-19-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall We can no longer blindly copy the VCPU's PSTATE into SPSR_EL2 and return to the guest and vice versa when taking an exception to the hypervisor, because we emulate virtual EL2 in EL1 and therefore have to translate the mode field from EL2 to EL1 and vice versa. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/sysreg-sr.c | 43 ++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index ed22301fd2e3..384f0da6b4f3 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -109,10 +109,32 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) __sysreg_save_vel1_state(ctxt); } +static u64 __hyp_text from_hw_pstate(const struct kvm_cpu_context *ctxt) +{ + u64 reg = read_sysreg_el2(SYS_SPSR); + + if (__is_hyp_ctxt(ctxt)) { + u64 mode = reg & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL1t: + mode = PSR_MODE_EL2t; + break; + case PSR_MODE_EL1h: + mode = PSR_MODE_EL2h; + break; + } + + return (reg & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + } + + return reg; +} + static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { ctxt->gp_regs.regs.pc = read_sysreg_el2(SYS_ELR); - ctxt->gp_regs.regs.pstate = read_sysreg_el2(SYS_SPSR); + ctxt->gp_regs.regs.pstate = from_hw_pstate(ctxt); if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN)) ctxt->sys_regs[DISR_EL1] = read_sysreg_s(SYS_VDISR_EL2); @@ -319,10 +341,27 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) __sysreg_restore_vel1_state(ctxt); } +/* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */ +static u64 __hyp_text to_hw_pstate(const struct kvm_cpu_context *ctxt) +{ + u64 mode = ctxt->gp_regs.regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + } + + return (ctxt->gp_regs.regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; +} + static void __hyp_text __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) { - u64 pstate = ctxt->gp_regs.regs.pstate; + u64 pstate = to_hw_pstate(ctxt); u64 mode = pstate & PSR_AA32_MODE_MASK; /* From patchwork Tue Feb 11 17:48:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376263 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C286714B4 for ; Tue, 11 Feb 2020 17:52:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A192D20870 for ; Tue, 11 Feb 2020 17:52:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443571; bh=uVpX/vFvfITJoEOTo3o/dRYhdnpA6CypgkaOmt8XOro=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=T4ndFQO90PXHPKKdSZJG8haU+uj2Qnpe03fupikM9wapxU9SBoF1a3jBG67LLN0rO vQ3cPxHkbzTU/dLKw0XXXbUs8HBodn9Oyj6lUan5crPJf8vmLGIUhqGuindrG4AYwD cFcmDPGRz6Q1uv/JVvmj1HPcqGv8L8uj5TglUxno= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728625AbgBKRwu (ORCPT ); Tue, 11 Feb 2020 12:52:50 -0500 Received: from mail.kernel.org ([198.145.29.99]:56030 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730614AbgBKRwu (ORCPT ); Tue, 11 Feb 2020 12:52:50 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5A4C4206D7; Tue, 11 Feb 2020 17:52:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443569; bh=uVpX/vFvfITJoEOTo3o/dRYhdnpA6CypgkaOmt8XOro=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Sr7BShg8r0uT6nqgL7R6MI/R+V05NAMbQFvBjYID+AFPqKlXBQPjnTHemmmMw3zqD efeSjMw8JR8+6GeQYAOOMoZBqmAyHgfz+YmlFVozZbAaF2mfUA3mrQaJ40xaFF2+UJ NT5R67NtSCEQzm7GfKkZIpR7MuAqURivYJAr8Ric= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfg-004O7k-IA; Tue, 11 Feb 2020 17:50:04 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 19/94] KVM: arm64: nv: Trap EL1 VM register accesses in virtual EL2 Date: Tue, 11 Feb 2020 17:48:23 +0000 Message-Id: <20200211174938.27809-20-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall When running in virtual EL2 mode, we actually run the hardware in EL1 and therefore have to use the EL1 registers to ensure correct operation. By setting the HCR.TVM and HCR.TVRM we ensure that the virtual EL2 mode doesn't shoot itself in the foot when setting up what it believes to be a different mode's system register state (for example when preparing to switch to a VM). We can leverage the existing sysregs infrastructure to support trapped accesses to these registers. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/switch.c | 4 ++++ arch/arm64/kvm/sys_regs.c | 18 +++++++++++++++--- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index dfe8dd172512..48e738e90966 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -149,6 +149,10 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM)) hcr |= HCR_TVM; + /* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ + if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) + hcr |= HCR_TVM | HCR_TRVM; + write_sysreg(hcr, hcr_el2); if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN) && (hcr & HCR_VSE)) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 9b29ac37829b..702692f064f4 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -440,8 +440,15 @@ static bool access_dcsw(struct kvm_vcpu *vcpu, /* * Generic accessor for VM registers. Only called as long as HCR_TVM - * is set. If the guest enables the MMU, we stop trapping the VM - * sys_regs and leave it in complete control of the caches. + * is set. + * + * This is set in two cases: either (1) we're running at vEL2, or (2) + * we're running at EL1 and the guest has its MMU off. + * + * (1) TVM is always set, as we need to virtualise some of the VM + * registers for the guest hypervisor + * (2) Oncethe guest enables the MMU, we stop trapping the VM sys_regs + * and leave it in complete control of the caches. */ static bool access_vm_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p, @@ -451,7 +458,12 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, u64 val; int reg = r->reg; - BUG_ON(!p->is_write); + BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write); + + if (!p->is_write) { + p->regval = vcpu_read_sys_reg(vcpu, reg); + return true; + } /* See the 32bit mapping in kvm_host.h */ if (p->is_aarch32) From patchwork Tue Feb 11 17:48:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376297 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8935914B4 for ; Tue, 11 Feb 2020 17:53:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 654C920870 for ; Tue, 11 Feb 2020 17:53:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443601; bh=vvHFl6AprWPxdXR5AIE7qOpRfP6rPjqAMgRD4Fcr3M8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=cy/1iCSJuZiinv9FhSCHwRb7xJjtiludR6nRCCKVJgzIw/XnfcX6qwJfrqaxFAR91 /09/TPsMc9XV0jHLrYJbIxuLAOViOnfV22fEy1sH8yvOc+aF0K0Evhth/wVgCvBwMw 0zetLXbgvQHn0ZjKOf2wDVySdk5er0VlduzNmPTQ= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730708AbgBKRxU (ORCPT ); Tue, 11 Feb 2020 12:53:20 -0500 Received: from mail.kernel.org ([198.145.29.99]:56852 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730681AbgBKRxU (ORCPT ); Tue, 11 Feb 2020 12:53:20 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6F8A1206CC; Tue, 11 Feb 2020 17:53:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443599; bh=vvHFl6AprWPxdXR5AIE7qOpRfP6rPjqAMgRD4Fcr3M8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0DeaAr9LQeT4gvQP8Gz7foVpRY07HjByBxKVOPQS8hSV4Z6p7Jt7VkFDnXB/4XAKp NZGSdwFq0aXYLCI1KXE+5w4LTuxtMl1rHOEdEN2rEV3ykwdQbQ9BgcbkT42yavwaY/ S9vjViK4fiIiE0EdBu8fc63MqVgVQRbWrUiKuH1I= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfh-004O7k-3Q; Tue, 11 Feb 2020 17:50:05 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 20/94] KVM: arm64: nv: Trap SPSR_EL1, ELR_EL1 and VBAR_EL1 from virtual EL2 Date: Tue, 11 Feb 2020 17:48:24 +0000 Message-Id: <20200211174938.27809-21-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim For the same reason we trap virtual memory register accesses at virtual EL2, we need to trap SPSR_EL1, ELR_EL1 and VBAR_EL1 accesses. ARM v8.3 introduces the HCR_EL2.NV1 bit to be able to trap on those register accesses in EL1. Do not set this bit until the whole nesting support is completed. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 29 ++++++++++++++++++++++++++++- 1 file changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 702692f064f4..18b14c04355a 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1628,6 +1628,30 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool access_elr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu->arch.ctxt.gp_regs.elr_el1 = p->regval; + else + p->regval = vcpu->arch.ctxt.gp_regs.elr_el1; + + return true; +} + +static bool access_spsr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + vcpu->arch.ctxt.gp_regs.spsr[KVM_SPSR_EL1] = p->regval; + else + p->regval = vcpu->arch.ctxt.gp_regs.spsr[KVM_SPSR_EL1]; + + return true; +} + static bool access_spsr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -1783,6 +1807,9 @@ static const struct sys_reg_desc sys_reg_descs[] = { PTRAUTH_KEY(APDB), PTRAUTH_KEY(APGA), + { SYS_DESC(SYS_SPSR_EL1), access_spsr}, + { SYS_DESC(SYS_ELR_EL1), access_elr}, + { SYS_DESC(SYS_AFSR0_EL1), access_vm_reg, reset_unknown, AFSR0_EL1 }, { SYS_DESC(SYS_AFSR1_EL1), access_vm_reg, reset_unknown, AFSR1_EL1 }, { SYS_DESC(SYS_ESR_EL1), access_vm_reg, reset_unknown, ESR_EL1 }, @@ -1811,7 +1838,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), NULL, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only }, From patchwork Tue Feb 11 17:48:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376289 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 17A8A1395 for ; Tue, 11 Feb 2020 17:53:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EA639214DB for ; Tue, 11 Feb 2020 17:53:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443594; bh=BA3j0ebKj60uf2wq/Z+cbQXS2wtxrMbpxGeiK17XT7I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=1dGWz5P1G1ngTO7321xjlzawYQMaXNH6Up1w8GPqiXjqD9f+rxf6AH3b+WVD8gb8P Zpr0CBVWHtv/q+DnX1/kDs1cPNerLfMFunBpET6WxK+SwNAL5dUSnN2yDz94ZfcTk9 aKt51p4fIHipOc+KJAQqGMpBKHf1+h6EruceX6Qo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730674AbgBKRxN (ORCPT ); Tue, 11 Feb 2020 12:53:13 -0500 Received: from mail.kernel.org ([198.145.29.99]:56650 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730658AbgBKRxM (ORCPT ); Tue, 11 Feb 2020 12:53:12 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 14B6720848; Tue, 11 Feb 2020 17:53:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443592; bh=BA3j0ebKj60uf2wq/Z+cbQXS2wtxrMbpxGeiK17XT7I=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T3fqg/OdNlWIEUz8//ik3hTukxQIiUsvOVWSlSJcGJzhx9fTQSVqDBswVbhGzMbWi NYYR29iDTtXmfNx3Lw5Rsx3nR4mJYiGyJ8u4aubiWfa3UkJYuJ4hX00RYExyN5wY67 p+PCROw/QWSMrLofV/M66muQNpaAksAhRm8r5OsA= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfh-004O7k-Or; Tue, 11 Feb 2020 17:50:05 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 21/94] KVM: arm64: nv: Trap CPACR_EL1 access in virtual EL2 Date: Tue, 11 Feb 2020 17:48:25 +0000 Message-Id: <20200211174938.27809-22-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim For the same reason we trap virtual memory register accesses in virtual EL2, we trap CPACR_EL1 access too; We allow the virtual EL2 mode to access EL1 system register state instead of the virtual EL2 one. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 2 +- arch/arm64/kvm/hyp/switch.c | 2 ++ arch/arm64/kvm/sys_regs.c | 2 +- 3 files changed, 4 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 084844ec0b06..a907c702de45 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -266,7 +266,7 @@ #define CPTR_EL2_TFP_SHIFT 10 /* Hyp Coprocessor Trap Register */ -#define CPTR_EL2_TCPAC (1 << 31) +#define CPTR_EL2_TCPAC (1U << 31) #define CPTR_EL2_TTA (1 << 20) #define CPTR_EL2_TFP (1 << CPTR_EL2_TFP_SHIFT) #define CPTR_EL2_TZ (1 << 8) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 48e738e90966..976b9661203b 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -105,6 +105,8 @@ static void activate_traps_vhe(struct kvm_vcpu *vcpu) val &= ~CPACR_EL1_FPEN; __activate_traps_fpsimd32(vcpu); } + if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) + val |= CPTR_EL2_TCPAC; write_sysreg(val, cpacr_el1); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 18b14c04355a..b545156c7d79 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1795,7 +1795,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { ID_UNALLOCATED(7,7), { SYS_DESC(SYS_SCTLR_EL1), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, - { SYS_DESC(SYS_CPACR_EL1), NULL, reset_val, CPACR_EL1, 0 }, + { SYS_DESC(SYS_CPACR_EL1), access_rw, reset_val, CPACR_EL1, 0 }, { SYS_DESC(SYS_ZCR_EL1), NULL, reset_val, ZCR_EL1, 0, .visibility = sve_visibility }, { SYS_DESC(SYS_TTBR0_EL1), access_vm_reg, reset_unknown, TTBR0_EL1 }, { SYS_DESC(SYS_TTBR1_EL1), access_vm_reg, reset_unknown, TTBR1_EL1 }, From patchwork Tue Feb 11 17:48:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376237 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B091014E3 for ; Tue, 11 Feb 2020 17:52:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 91B7C20870 for ; Tue, 11 Feb 2020 17:52:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443548; bh=Hzwd2V5pNv8PurnC7nBw9dzDHuozYcO82FdEXCzVPDc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Cbgc5w1qbIieqE1+1WrBZg9yrCwOflJlWdPM7Agbgss4LXPj8ploFf1rXMSMUQUZS QwomviYly8gVmcgmvV/s69ZNiCpa3hsOR2PIrsGxDN5HYN1lRYemMR0tsP13nIwBYc 2M5vuXv3S6h1R+iXZQIsIKdb7gNhSS9OYRV+NzGo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730536AbgBKRw2 (ORCPT ); Tue, 11 Feb 2020 12:52:28 -0500 Received: from mail.kernel.org ([198.145.29.99]:55396 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730485AbgBKRw1 (ORCPT ); Tue, 11 Feb 2020 12:52:27 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0577E20578; Tue, 11 Feb 2020 17:52:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443547; bh=Hzwd2V5pNv8PurnC7nBw9dzDHuozYcO82FdEXCzVPDc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ntYGo580sRVgNxSH2Wx9nFcgMwRCmt/JHyEahIcC7AtbVUd+Nk/2baZ56MNCFr68M ixAuGX8i8kejHfokJENZXt0x6vDIfpYChf1MhWOicxGNUcxOeJgXs8awVy+vHJ/Ow9 3nN5Kn5DlJEQWibjMcicohaWtW++adz3WS1yHVLA= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfi-004O7k-AH; Tue, 11 Feb 2020 17:50:06 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 22/94] KVM: arm64: nv: Handle PSCI call via smc from the guest Date: Tue, 11 Feb 2020 17:48:26 +0000 Message-Id: <20200211174938.27809-23-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim VMs used to execute hvc #0 for the psci call if EL3 is not implemented. However, when we come to provide the virtual EL2 mode to the VM, the host OS inside the VM calls kvm_call_hyp() which is also hvc #0. So, it's hard to differentiate between them from the host hypervisor's point of view. So, let the VM execute smc instruction for the psci call. On ARMv8.3, even if EL3 is not implemented, a smc instruction executed at non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than being treated as UNDEFINED. So, the host hypervisor can handle this psci call without any confusion. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/handle_exit.c | 24 ++++++++++++++++++++++-- 1 file changed, 22 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index c3398e6ebbbe..3ee5351b8561 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -63,6 +63,8 @@ static int handle_hvc(struct kvm_vcpu *vcpu, struct kvm_run *run) static int handle_smc(struct kvm_vcpu *vcpu, struct kvm_run *run) { + int ret; + /* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a @@ -70,10 +72,28 @@ static int handle_smc(struct kvm_vcpu *vcpu, struct kvm_run *run) * * We need to advance the PC after the trap, as it would * otherwise return to the same address... + * + * If imm is non-zero, it's not defined, so just skip it. + */ + if (kvm_vcpu_hvc_get_imm(vcpu)) { + vcpu_set_reg(vcpu, 0, ~0UL); + kvm_skip_instr(vcpu, kvm_vcpu_trap_il_is32bit(vcpu)); + return 1; + } + + /* + * If imm is zero, it's a psci call. + * Note that on ARMv8.3, even if EL3 is not implemented, SMC executed + * at Non-secure EL1 is trapped to EL2 if HCR_EL2.TSC==1, rather than + * being treated as UNDEFINED. */ - vcpu_set_reg(vcpu, 0, ~0UL); + ret = kvm_hvc_call_handler(vcpu); + if (ret < 0) + vcpu_set_reg(vcpu, 0, ~0UL); + kvm_skip_instr(vcpu, kvm_vcpu_trap_il_is32bit(vcpu)); - return 1; + + return ret; } /* From patchwork Tue Feb 11 17:48:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376155 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1CB7A14B4 for ; Tue, 11 Feb 2020 17:51:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F0F292086A for ; Tue, 11 Feb 2020 17:51:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443477; bh=9dIJ8KsE5ED+/qzJyjFk48QZLaVjJxY8pEcV3bLxq30=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=k4vnHkWzQHPXQxzO5ZyKiZVs5thBk9ZD4oG5mLIBTUAIEruCzd/H4xBm4Up2OayaL BPGsxXy921ORpHlvWmGRY40y2ecFmcxOy937jiVmTuXKS5lIMh82VD2oETY0yLot1q vZZLGqUggLWquduVV8xWQpbAnziKXrhq44rNQhsc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730309AbgBKRvP (ORCPT ); Tue, 11 Feb 2020 12:51:15 -0500 Received: from mail.kernel.org ([198.145.29.99]:53488 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730304AbgBKRvP (ORCPT ); Tue, 11 Feb 2020 12:51:15 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id CDEDF20870; Tue, 11 Feb 2020 17:51:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443475; bh=9dIJ8KsE5ED+/qzJyjFk48QZLaVjJxY8pEcV3bLxq30=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BoI7GKKI3YeYtInvbEM6ZkAr5XiMrucqxvId9HPBZ8ga43ngSXytUP8Adz0ASN/5/ TjlazEQCA/4txVsXrwfBqvcWAhiFPfZ5q+m6QsviG9eiJo7gZoT6bdBz2MZyo0sYYa W2DaJgQZ21SEaT/JFmB5ot/ukIMICBp3CgTrbiD8= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfi-004O7k-SQ; Tue, 11 Feb 2020 17:50:07 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 23/94] KVM: arm64: nv: Respect virtual HCR_EL2.TWX setting Date: Tue, 11 Feb 2020 17:48:27 +0000 Message-Id: <20200211174938.27809-24-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward exceptions due to WFI or WFE instructions to the virtual EL2 if they are not coming from the virtual EL2 and virtual HCR_EL2.TWX is set. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/handle_exit.c | 11 +++++++- arch/arm64/kvm/nested.c | 40 +++++++++++++++++++++++++++++ 4 files changed, 53 insertions(+), 1 deletion(-) create mode 100644 arch/arm64/kvm/nested.c diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index f243a8acb9fc..15041cda820b 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -10,4 +10,6 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features); } +int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 32498becb71b..5bd4775b44bc 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -38,4 +38,5 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o +kvm-$(CONFIG_KVM_ARM_HOST) += nested.o kvm-$(CONFIG_KVM_ARM_HOST) += emulate-nested.o diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 3ee5351b8561..03bc69b09ed4 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -120,7 +120,16 @@ static int handle_no_fpsimd(struct kvm_vcpu *vcpu, struct kvm_run *run) */ static int kvm_handle_wfx(struct kvm_vcpu *vcpu, struct kvm_run *run) { - if (kvm_vcpu_get_hsr(vcpu) & ESR_ELx_WFx_ISS_WFE) { + bool is_wfe = !!(kvm_vcpu_get_hsr(vcpu) & ESR_ELx_WFx_ISS_WFE); + + if (nested_virt_in_use(vcpu)) { + int ret = handle_wfx_nested(vcpu, is_wfe); + + if (ret != -EINVAL) + return ret; + } + + if (is_wfe) { trace_kvm_wfx_arm64(*vcpu_pc(vcpu), true); vcpu->stat.wfe_exit_stat++; kvm_vcpu_on_spin(vcpu, vcpu_mode_priv(vcpu)); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c new file mode 100644 index 000000000000..c5e695cf81fb --- /dev/null +++ b/arch/arm64/kvm/nested.c @@ -0,0 +1,40 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2017 - Columbia University and Linaro Ltd. + * Author: Jintack Lim + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include +#include + +#include + +/* + * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and + * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor + * handle this. + */ +int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) +{ + u64 hcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + if (vcpu_mode_el2(vcpu)) + return -EINVAL; + + if ((is_wfe && (hcr_el2 & HCR_TWE)) || (!is_wfe && (hcr_el2 & HCR_TWI))) + return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_hsr(vcpu)); + + return -EINVAL; +} From patchwork Tue Feb 11 17:48:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376299 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 88DFF14B4 for ; Tue, 11 Feb 2020 17:53:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6927A214DB for ; Tue, 11 Feb 2020 17:53:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443603; bh=sE8UXFGqcupDgYc8xZ/0fIPTLUj36A79kNsMxBbXn/o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=eroVvHP42UCCMSVn3BaZ38RCevZ0FVFbDBaZ5bp5AQxh3LC0Fcc+6G5qhdBcNTDOZ PgS4A+XGG20NBrLH5/ViXRwIZlnox7lno/qblmtYhx+Pfwg1uhbuffA5PEtdKRguxm 27b5wEsXDEOCkmg8oR5TPNii6u5HPzo+vKGadMj8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730662AbgBKRxW (ORCPT ); Tue, 11 Feb 2020 12:53:22 -0500 Received: from mail.kernel.org ([198.145.29.99]:56904 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730671AbgBKRxW (ORCPT ); Tue, 11 Feb 2020 12:53:22 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 48EC720661; Tue, 11 Feb 2020 17:53:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443601; bh=sE8UXFGqcupDgYc8xZ/0fIPTLUj36A79kNsMxBbXn/o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eJBh7/n4oMrBfqjZFf2O0DAofX+KFPP1XKEJJTDfjhF3L7LboA2R8IDJ2w/EgRP/c ydpCPbog6g8EEZxa1TGFTi4JOO+qpV/xJwZjNIiBYoEjiC0Sic9Ay7x/admleXhi+K +z+uqoy4UaSNUsOqnfC1ptuwMgj7lQoLfjWQYCZ4= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfj-004O7k-Ej; Tue, 11 Feb 2020 17:50:07 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 24/94] KVM: arm64: nv: Respect virtual CPTR_EL2.{TFP,FPEN} settings Date: Tue, 11 Feb 2020 17:48:28 +0000 Message-Id: <20200211174938.27809-25-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward traps due to FP/ASIMD register accesses to the virtual EL2 if virtual CPTR_EL2.TFP is set (with HCR_EL2.E2H == 0) or CPTR_EL2.FPEN is configure to do so (with HCR_EL2.E2h == 1). Signed-off-by: Jintack Lim Signed-off-by: Christoffer Dall [maz: account for HCR_EL2.E2H when testing for TFP/FPEN, with all the hard work actually being done by Chase Conklin] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 26 ++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 16 ++++++++++++---- arch/arm64/kvm/hyp/switch.c | 11 +++++++++-- 3 files changed, 47 insertions(+), 6 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 26552c8571cb..0e5f88060ecc 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -11,12 +11,14 @@ #ifndef __ARM64_KVM_EMULATE_H__ #define __ARM64_KVM_EMULATE_H__ +#include #include #include #include #include #include +#include #include #include #include @@ -392,6 +394,30 @@ static inline bool vcpu_mode_priv(const struct kvm_vcpu *vcpu) return mode != PSR_MODE_EL0t; } +static inline bool guest_hyp_fpsimd_traps_enabled(const struct kvm_vcpu *vcpu) +{ + u64 val; + + if (!nested_virt_in_use(vcpu)) + return false; + + val = vcpu_read_sys_reg(vcpu, CPTR_EL2); + + if (!vcpu_el2_e2h_is_set(vcpu)) + return (val & CPTR_EL2_TFP); + + switch (FIELD_GET(CPACR_EL1_FPEN, val)) { + case 0b00: + case 0b10: + return true; + case 0b01: + return vcpu_el2_tge_is_set(vcpu) && !vcpu_mode_el2(vcpu); + case 0b11: + default: /* GCC is dumb */ + return false; + } +} + static inline u32 kvm_vcpu_get_hsr(const struct kvm_vcpu *vcpu) { return vcpu->arch.fault.esr_el2; diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 03bc69b09ed4..e813bda18c7c 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -97,11 +97,19 @@ static int handle_smc(struct kvm_vcpu *vcpu, struct kvm_run *run) } /* - * Guest access to FP/ASIMD registers are routed to this handler only - * when the system doesn't support FP/ASIMD. + * This handles the cases where the system does not support FP/ASIMD or when + * we are running nested virtualization and the guest hypervisor is trapping + * FP/ASIMD accesses by its guest guest. + * + * All other handling of guest vs. host FP/ASIMD register state is handled in + * fixup_guest_exit(). */ -static int handle_no_fpsimd(struct kvm_vcpu *vcpu, struct kvm_run *run) +static int kvm_handle_fpasimd(struct kvm_vcpu *vcpu, struct kvm_run *run) { + if (guest_hyp_fpsimd_traps_enabled(vcpu)) + return kvm_inject_nested_sync(vcpu, kvm_vcpu_get_hsr(vcpu)); + + /* This is the case when the system doesn't support FP/ASIMD. */ kvm_inject_undefined(vcpu); return 1; } @@ -270,7 +278,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_BREAKPT_LOW]= kvm_handle_guest_debug, [ESR_ELx_EC_BKPT32] = kvm_handle_guest_debug, [ESR_ELx_EC_BRK64] = kvm_handle_guest_debug, - [ESR_ELx_EC_FP_ASIMD] = handle_no_fpsimd, + [ESR_ELx_EC_FP_ASIMD] = kvm_handle_fpasimd, [ESR_ELx_EC_PAC] = kvm_handle_ptrauth, }; diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 976b9661203b..ae967e274e08 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -371,11 +371,18 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) hsr_ec != ESR_ELx_EC_SVE) return false; - /* Don't handle SVE traps for non-SVE vcpus here: */ - if (!sve_guest) + /* + * Don't handle SVE traps for non-SVE vcpus here. This + * includes NV guests for the time beeing. + */ + if (!sve_guest) { if (hsr_ec != ESR_ELx_EC_FP_ASIMD) return false; + if (guest_hyp_fpsimd_traps_enabled(vcpu)) + return false; + } + /* Valid trap. Switch the context: */ if (vhe) { From patchwork Tue Feb 11 17:48:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376169 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C6E5A1395 for ; Tue, 11 Feb 2020 17:51:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A68CD206CC for ; Tue, 11 Feb 2020 17:51:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443488; bh=sdVlxI/GjjZN9m7eP6XWMEC4MszchDVWVMZ7842jhWI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=M0XJQZjPxJi/JtC4P1ZVMRlDdoZeAqDSVRIAnc8MZ/816EY9VmOzYkHKY5HMmMtOt XwPwTe/AKR7oEyUeS5+NBxO6A9voxmITabbG+pey/d04pe6dtg8sN4FNT4bUXMkR6l 8g8mcdz4/N2QHBM+ND0g7nl372DUJneFHny4OOzY= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730346AbgBKRv2 (ORCPT ); Tue, 11 Feb 2020 12:51:28 -0500 Received: from mail.kernel.org ([198.145.29.99]:53808 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730351AbgBKRv1 (ORCPT ); Tue, 11 Feb 2020 12:51:27 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2DA41214DB; Tue, 11 Feb 2020 17:51:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443486; bh=sdVlxI/GjjZN9m7eP6XWMEC4MszchDVWVMZ7842jhWI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=VHNc/bWFCFPcmyv3Y0zdskXWk5ukI7XDq34uRlMBBImJFFisOEhVf/OnPDdTKBq2L GTt98W8VJuCXGYdKr1Z2mbzhMu5uuYnj40UOYxheiRO2wowMAwX2XpmFpFnU/exjGR grm3Dl8w5BVA30uPO8vYZ8oye1kgZyYPm0xuShj4= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfk-004O7k-0k; Tue, 11 Feb 2020 17:50:08 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 25/94] KVM: arm64: nv: Respect the virtual HCR_EL2.NV bit setting Date: Tue, 11 Feb 2020 17:48:29 +0000 Message-Id: <20200211174938.27809-26-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward traps due to HCR_EL2.NV bit to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.NV bit is set. In addition to EL2 register accesses, setting NV bit will also make EL12 register accesses trap to EL2. To emulate this for the virtual EL2, forword traps due to EL12 register accessses to the virtual EL2 if the virtual HCR_EL2.NV bit is set. This is for recursive nested virtualization. Signed-off-by: Jintack Lim [Moved code to emulate-nested.c] Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/emulate-nested.c | 28 ++++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 7 +++++++ arch/arm64/kvm/sys_regs.c | 21 +++++++++++++++++++++ 5 files changed, 59 insertions(+) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index a907c702de45..1be4667d5e3d 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -13,6 +13,7 @@ /* Hyp Configuration Register (HCR) bits */ #define HCR_FWB (UL(1) << 46) +#define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) #define HCR_APK (UL(1) << 40) #define HCR_TEA (UL(1) << 37) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 15041cda820b..c75e4326a363 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -11,5 +11,7 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) } int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); +extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); +extern bool forward_nv_traps(struct kvm_vcpu *vcpu); #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 328ae3723330..957a4a84d802 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -24,6 +24,27 @@ #include "trace.h" +bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +{ + bool control_bit_set; + + if (!nested_virt_in_use(vcpu)) + return false; + + control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit; + if (!vcpu_mode_el2(vcpu) && control_bit_set) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_hsr(vcpu)); + return true; + } + return false; +} + +bool forward_nv_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_NV); +} + + /* This is borrowed from get_except_vector in inject_fault.c */ static u64 get_el2_except_vector(struct kvm_vcpu *vcpu, enum exception_type type) @@ -55,6 +76,13 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) u64 spsr, elr, mode; bool direct_eret; + /* + * Forward this trap to the virtual EL2 if the virtual + * HCR_EL2.NV bit is set and this is coming from !EL2. + */ + if (forward_nv_traps(vcpu)) + return; + /* * Going through the whole put/load motions is a waste of time * if this is a VHE guest hypervisor returning to its own diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index e813bda18c7c..d2a73ebf9d4a 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -65,6 +65,13 @@ static int handle_smc(struct kvm_vcpu *vcpu, struct kvm_run *run) { int ret; + /* + * Forward this trapped smc instruction to the virtual EL2 if + * the guest has asked for it. + */ + if (forward_traps(vcpu, HCR_TSC)) + return 1; + /* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index b545156c7d79..b948afd20e97 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -392,10 +392,19 @@ static u32 get_ccsidr(u32 csselr) return ccsidr; } +static bool el12_reg(struct sys_reg_params *p) +{ + /* All *_EL12 registers have Op1=5. */ + return (p->Op1 == 5); +} + static bool access_rw(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, r->reg); else @@ -458,6 +467,9 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, u64 val; int reg = r->reg; + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write); if (!p->is_write) { @@ -1632,6 +1644,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu->arch.ctxt.gp_regs.elr_el1 = p->regval; else @@ -1644,6 +1659,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu->arch.ctxt.gp_regs.spsr[KVM_SPSR_EL1] = p->regval; else @@ -1656,6 +1674,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); else From patchwork Tue Feb 11 17:48:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376197 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2DCB1395 for ; Tue, 11 Feb 2020 17:51:52 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8634F20578 for ; Tue, 11 Feb 2020 17:51:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443512; bh=utysODfDdOQMjnkhgSF20q8EYOV4H4+ozLatVEY6evg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=DBzn0RR3s3veAtR6Eti3KK5lwFzSUPSWJ7kDOmcNOH7Lcn+/ZZwIDWkDv1CuYHc/W 7e1rMOIE0vSETyFC7EIkEHkW5Wee+0y7BpLUnhopDk15re+J1V84hydoVLvsgM31Ig M+T1vRPVAWrcZF5WCqUlOtwk/NluRXqgETX69pow= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730444AbgBKRvv (ORCPT ); Tue, 11 Feb 2020 12:51:51 -0500 Received: from mail.kernel.org ([198.145.29.99]:54482 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730249AbgBKRvv (ORCPT ); Tue, 11 Feb 2020 12:51:51 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C3D7A21569; Tue, 11 Feb 2020 17:51:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443510; bh=utysODfDdOQMjnkhgSF20q8EYOV4H4+ozLatVEY6evg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q4BPrb9DQf743LGVa0RrNVeNRbm/KI2Xj+cdbrga1cJO515K7BYldWjvaRxyysRF3 1A/z8nMWlSyotGIjjIq0zxRkQ3UYOa9+nDMPf+GMUL2GhbHBseWQqgHe5seOd6WLAh nTJkRWNc0h5GTI787tmi1QZqTk6lppddQXYYZCMM= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfk-004O7k-KL; Tue, 11 Feb 2020 17:50:08 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 26/94] KVM: arm64: nv: Respect virtual HCR_EL2.TVM and TRVM settings Date: Tue, 11 Feb 2020 17:48:30 +0000 Message-Id: <20200211174938.27809-27-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward the EL1 virtual memory register traps to the virtual EL2 if they are not coming from the virtual EL2 and the virtual HCR_EL2.TVM or TRVM bit is set. This is for recursive nested virtualization. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index b948afd20e97..7cf5096bc11a 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -470,6 +470,13 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p)) { + u64 bit = p->is_write ? HCR_TVM : HCR_TRVM; + + if (forward_traps(vcpu, bit)) + return false; + } + BUG_ON(!vcpu_mode_el2(vcpu) && !p->is_write); if (!p->is_write) { From patchwork Tue Feb 11 17:48:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376171 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 213901395 for ; Tue, 11 Feb 2020 17:51:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 01076206CC for ; Tue, 11 Feb 2020 17:51:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443491; bh=niDNmWeRe+kA6MDesjc0xqObQ1M36vCURCX+5FfzILg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=UsT2EJNlYFqsfZc01GSooyR9Uh0S/65ayGHOquVfI11hk6guXeM3dZbErA6KGXK/w Fp4HpggGc3LKus7DNhVtjZvHw4b+H9BUukXNNynnOVgzcMP8RsAjSeKfy55tv4065u slC2Wfl4CkszzXBR0LqWGA5W3XJ7vHL/fULMXG/k= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730350AbgBKRva (ORCPT ); Tue, 11 Feb 2020 12:51:30 -0500 Received: from mail.kernel.org ([198.145.29.99]:53852 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730249AbgBKRv3 (ORCPT ); Tue, 11 Feb 2020 12:51:29 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 15C8D206D7; Tue, 11 Feb 2020 17:51:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443488; bh=niDNmWeRe+kA6MDesjc0xqObQ1M36vCURCX+5FfzILg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WfCD42GcorDFMuHpwBD0bY4Q7czs2MNbZDigczpPoecXuxoYUPcMYxrXw/TJB1/lo bgEFH+rS6Z1oIiTzp0l7Kz2wKfxEJ3PMcHMPAsdSRVAs7PxKMfTKuxhpLcN8hiNp7/ 8JeamcPh2ZnreKzmvqJlJyl844Qrxrob7c+IL/X8= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfl-004O7k-6S; Tue, 11 Feb 2020 17:50:09 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 27/94] KVM: arm64: nv: Respect the virtual HCR_EL2.NV1 bit setting Date: Tue, 11 Feb 2020 17:48:31 +0000 Message-Id: <20200211174938.27809-28-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim Forward ELR_EL1, SPSR_EL1 and VBAR_EL1 traps to the virtual EL2 if the virtual HCR_EL2.NV bit is set. This is for recursive nested virtualization. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/kvm/sys_regs.c | 28 +++++++++++++++++++++++++++- 2 files changed, 28 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 1be4667d5e3d..cdee623ce92f 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -13,6 +13,7 @@ /* Hyp Configuration Register (HCR) bits */ #define HCR_FWB (UL(1) << 46) +#define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) #define HCR_APK (UL(1) << 40) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 7cf5096bc11a..2ce85d5e7111 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -424,6 +424,22 @@ static bool access_wi(struct kvm_vcpu *vcpu, return true; } +/* This function is to support the recursive nested virtualization */ +static bool forward_nv1_traps(struct kvm_vcpu *vcpu, struct sys_reg_params *p) +{ + return forward_traps(vcpu, HCR_NV1); +} + +static bool access_vbar_el1(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (forward_nv1_traps(vcpu, p)) + return false; + + return access_rw(vcpu, p, r); +} + /* * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). */ @@ -1647,6 +1663,7 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } + static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -1654,6 +1671,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu, p)) + return false; + if (p->is_write) vcpu->arch.ctxt.gp_regs.elr_el1 = p->regval; else @@ -1669,6 +1689,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu, p)) + return false; + if (p->is_write) vcpu->arch.ctxt.gp_regs.spsr[KVM_SPSR_EL1] = p->regval; else @@ -1684,6 +1707,9 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, if (el12_reg(p) && forward_nv_traps(vcpu)) return false; + if (!el12_reg(p) && forward_nv1_traps(vcpu, p)) + return false; + if (p->is_write) vcpu_write_sys_reg(vcpu, p->regval, SPSR_EL2); else @@ -1866,7 +1892,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_LORC_EL1), trap_loregion }, { SYS_DESC(SYS_LORID_EL1), trap_loregion }, - { SYS_DESC(SYS_VBAR_EL1), access_rw, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_VBAR_EL1), access_vbar_el1, reset_val, VBAR_EL1, 0 }, { SYS_DESC(SYS_DISR_EL1), NULL, reset_val, DISR_EL1, 0 }, { SYS_DESC(SYS_ICC_IAR0_EL1), write_to_read_only }, From patchwork Tue Feb 11 17:48:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376247 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EC0271395 for ; Tue, 11 Feb 2020 17:52:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CB241206D7 for ; Tue, 11 Feb 2020 17:52:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443558; bh=BuazaigYiOjbw5OaE0ORfLHMkxH+BvQZJ5Hej8ql+c8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=r3G5Ol6eierJnqFGqJUhiTrubyCAGIfSoM1wSIYwryLoMQ/8wZ5BpGnyllVYWAczg tMwHuWrY+LPJf2o7VffvwEeohTC/xL+v6Ke+XsV3ZWDuVxT2LPKHXSKagcc1ytkMQ6 In9173Y0vaMXS7rRehp+o3BJLRq6VscIrta7LMes= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730583AbgBKRwh (ORCPT ); Tue, 11 Feb 2020 12:52:37 -0500 Received: from mail.kernel.org ([198.145.29.99]:55658 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730556AbgBKRwh (ORCPT ); Tue, 11 Feb 2020 12:52:37 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 533C420661; Tue, 11 Feb 2020 17:52:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443556; bh=BuazaigYiOjbw5OaE0ORfLHMkxH+BvQZJ5Hej8ql+c8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rqo6v8mWLi2mt25ST/XQVirargdmB+2vIAEAWzqahpkBU7fm62xg5sdMjZEshKMFd bRuR2+k6ll7CzKzFCndRg9N3HtclCooo0U7QN8wCmhh1G5Ga7mUOdU7JHqMqhR6uXz FesE2+uURoDm2sNXe3rHhVEs/FDilbt4aSHjyLYA= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfl-004O7k-NB; Tue, 11 Feb 2020 17:50:09 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 28/94] KVM: arm64: nv: Emulate EL12 register accesses from the virtual EL2 Date: Tue, 11 Feb 2020 17:48:32 +0000 Message-Id: <20200211174938.27809-29-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim With HCR_EL2.NV bit set, accesses to EL12 registers in the virtual EL2 trap to EL2. Handle those traps just like we do for EL1 registers. One exception is CNTKCTL_EL12. We don't trap on CNTKCTL_EL1 for non-VHE virtual EL2 because we don't have to. However, accessing CNTKCTL_EL12 will trap since it's one of the EL12 registers controlled by HCR_EL2.NV bit. Therefore, add a handler for it and don't treat it as a non-trap-registers when preparing a shadow context. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 2ce85d5e7111..dd4f8fe143ba 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2057,6 +2057,23 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + { SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, + { SYS_DESC(SYS_CPACR_EL12), access_rw, reset_val, CPACR_EL1, 0 }, + { SYS_DESC(SYS_TTBR0_EL12), access_vm_reg, reset_unknown, TTBR0_EL1 }, + { SYS_DESC(SYS_TTBR1_EL12), access_vm_reg, reset_unknown, TTBR1_EL1 }, + { SYS_DESC(SYS_TCR_EL12), access_vm_reg, reset_val, TCR_EL1, 0 }, + { SYS_DESC(SYS_SPSR_EL12), access_spsr}, + { SYS_DESC(SYS_ELR_EL12), access_elr}, + { SYS_DESC(SYS_AFSR0_EL12), access_vm_reg, reset_unknown, AFSR0_EL1 }, + { SYS_DESC(SYS_AFSR1_EL12), access_vm_reg, reset_unknown, AFSR1_EL1 }, + { SYS_DESC(SYS_ESR_EL12), access_vm_reg, reset_unknown, ESR_EL1 }, + { SYS_DESC(SYS_FAR_EL12), access_vm_reg, reset_unknown, FAR_EL1 }, + { SYS_DESC(SYS_MAIR_EL12), access_vm_reg, reset_unknown, MAIR_EL1 }, + { SYS_DESC(SYS_AMAIR_EL12), access_vm_reg, reset_amair_el1, AMAIR_EL1 }, + { SYS_DESC(SYS_VBAR_EL12), access_rw, reset_val, VBAR_EL1, 0 }, + { SYS_DESC(SYS_CONTEXTIDR_EL12), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 }, + { SYS_DESC(SYS_CNTKCTL_EL12), access_rw, reset_val, CNTKCTL_EL1, 0 }, + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; From patchwork Tue Feb 11 17:48:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376267 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA6291395 for ; Tue, 11 Feb 2020 17:52:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8B48B206D7 for ; Tue, 11 Feb 2020 17:52:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443577; bh=CA4QFDAvsJdDu0CZWxaYKhtPg1aoz71bH2Rea05f1TE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Rb77VbdLldDFL/7jIAMpEr3EMICYxU220P8qeUOZmWsWtJ2Q6ujhx2PS/1UyRu7Sh 0CrW8IIOyduGzK79KxC/VaIPFtycxFDsoHmqMvvp1g88MrxnmIjpeK/KM7qiiEvNYK GcQT5w4kgGEMfirO7PEe5VDqjBn/MYzdvJ10uIb8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730560AbgBKRw4 (ORCPT ); Tue, 11 Feb 2020 12:52:56 -0500 Received: from mail.kernel.org ([198.145.29.99]:56160 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730638AbgBKRw4 (ORCPT ); Tue, 11 Feb 2020 12:52:56 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1FBD9208C3; Tue, 11 Feb 2020 17:52:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443575; bh=CA4QFDAvsJdDu0CZWxaYKhtPg1aoz71bH2Rea05f1TE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=1PvbmRKVluhJPL3BGhvEy54kT1nMoI5UCQbstp7xIpxaqXujz4yt2md6HYkCNsG/9 PsHr+7jZdL//HiI6MPR22kDumTGQuL4R2VRcdR1AO0byKD7j4tNfohvNKD6Wge67Cy H47bpRN9affBJTJHr2HUpga9KPbciqK//Xlgl7zQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfm-004O7k-9X; Tue, 11 Feb 2020 17:50:10 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 29/94] KVM: arm64: nv: Forward debug traps to the nested guest Date: Tue, 11 Feb 2020 17:48:33 +0000 Message-Id: <20200211174938.27809-30-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On handling a debug trap, check whether we need to forward it to the guest before handling it. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 2 ++ arch/arm64/kvm/emulate-nested.c | 9 +++++++-- arch/arm64/kvm/sys_regs.c | 3 +++ 3 files changed, 12 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index c75e4326a363..5911a713c7dd 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -11,6 +11,8 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) } int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); +extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, + u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 957a4a84d802..8380ed810062 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -24,14 +24,14 @@ #include "trace.h" -bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit) { bool control_bit_set; if (!nested_virt_in_use(vcpu)) return false; - control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit; + control_bit_set = __vcpu_sys_reg(vcpu, reg) & control_bit; if (!vcpu_mode_el2(vcpu) && control_bit_set) { kvm_inject_nested_sync(vcpu, kvm_vcpu_get_hsr(vcpu)); return true; @@ -39,6 +39,11 @@ bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) return false; } +bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +{ + return __forward_traps(vcpu, HCR_EL2, control_bit); +} + bool forward_nv_traps(struct kvm_vcpu *vcpu) { return forward_traps(vcpu, HCR_NV); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index dd4f8fe143ba..435340a49634 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -672,6 +672,9 @@ static bool trap_debug_regs(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { + if (__forward_traps(vcpu, MDCR_EL2, MDCR_EL2_TDA | MDCR_EL2_TDE)) + return false; + access_rw(vcpu, p, r); if (p->is_write) vcpu->arch.flags |= KVM_ARM64_DEBUG_DIRTY; From patchwork Tue Feb 11 17:48:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376261 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D040D1395 for ; Tue, 11 Feb 2020 17:52:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AF97A208C3 for ; Tue, 11 Feb 2020 17:52:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443570; bh=2syRWuDJzfIhjmeHULYOTlZtqJYdWIZ8ljebtdFDvB8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=bhYXCbqWKpe+Gl/6IwwVbJl2kAjv/YUBxRGYWwXb4/5KZNdnJhsekrkuP8AY9my8m KhFjY+pWGC+uan2+j/h6j6qgLtuKjP5fNXyow4DyY29d5vlLjTIVULX/2e7EdztNBn /wm9yMwoH7jiqb5LHPGDdK/VgFkKmrdEcyR6yYyA= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730532AbgBKRwt (ORCPT ); Tue, 11 Feb 2020 12:52:49 -0500 Received: from mail.kernel.org ([198.145.29.99]:55986 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730601AbgBKRws (ORCPT ); Tue, 11 Feb 2020 12:52:48 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 78F65206CC; Tue, 11 Feb 2020 17:52:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443567; bh=2syRWuDJzfIhjmeHULYOTlZtqJYdWIZ8ljebtdFDvB8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=wp/LuaXqiT/UXN4ujliiqSWSxfUJHNoXMMZhWIP8qPeOZ97PnWgh5aniyzsMv33Sw KcYsDjK/qN88AtCWGBO27Zo6QvwPlaCTxD+tsp3lxg1OwJdOXoDKKE7/VXIoq/Kxlf v18T2whLoAhs2vv/qOqQMSiaBXxPQkLeNOYkqkc0= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfm-004O7k-Rb; Tue, 11 Feb 2020 17:50:11 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 30/94] KVM: arm64: nv: Configure HCR_EL2 for nested virtualization Date: Tue, 11 Feb 2020 17:48:34 +0000 Message-Id: <20200211174938.27809-31-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim We enable nested virtualization by setting the HCR NV and NV1 bit. When the virtual E2H bit is set, we can support EL2 register accesses via EL1 registers from the virtual EL2 by doing trap-and-emulate. A better alternative, however, is to allow the virtual EL2 to access EL2 register states without trap. This can be easily achieved by not traping EL1 registers since those registers already have EL2 register states. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/switch.c | 36 +++++++++++++++++++++++++++++++++--- 1 file changed, 33 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index ae967e274e08..fd4085ebf062 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -151,9 +151,39 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) if (cpus_have_const_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM)) hcr |= HCR_TVM; - /* Trap VM sysreg accesses if an EL2 guest is not using VHE. */ - if (vcpu_mode_el2(vcpu) && !vcpu_el2_e2h_is_set(vcpu)) - hcr |= HCR_TVM | HCR_TRVM; + if (is_hyp_ctxt(vcpu)) { + hcr |= HCR_NV; + + if (!vcpu_el2_e2h_is_set(vcpu)) { + /* + * For a guest hypervisor on v8.0, trap and emulate + * the EL1 virtual memory control register accesses. + */ + hcr |= HCR_TVM | HCR_TRVM | HCR_NV1; + } else { + /* + * For a guest hypervisor on v8.1 (VHE), allow to + * access the EL1 virtual memory control registers + * natively. These accesses are to access EL2 register + * states. + * Note that we still need to respect the virtual + * HCR_EL2 state. + */ + u64 vhcr_el2 = __vcpu_sys_reg(vcpu, HCR_EL2); + + /* + * We already set TVM to handle set/way cache maint + * ops traps, this somewhat collides with the nested + * virt trapping for nVHE. So turn this off for now + * here, in the hope that VHE guests won't ever do this. + * TODO: find out whether it's worth to support both + * cases at the same time. + */ + hcr &= ~HCR_TVM; + + hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM); + } + } write_sysreg(hcr, hcr_el2); From patchwork Tue Feb 11 17:48:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376271 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1D53E14B4 for ; Tue, 11 Feb 2020 17:52:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F1B74206D7 for ; Tue, 11 Feb 2020 17:52:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443579; bh=0Nz9UTGevazxDml65pFpWpvj0exBxapL78FXYgUMnz0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Wzy+ZbhyTO8l1xx/i4OVL6Wg3JJBoPe29VTS7dY26CUGU++CMwYp8SYJIMp7JdDsG mzxjWjBzS8JuzD6odpfGs5qU1V08z8iHCWsZp7tqT97nym366q+F3vYbHLjf8FCFK6 X3gLTeNpeeItVgPExxUYrJSo798ObyNYNDJ5hVCo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730569AbgBKRw6 (ORCPT ); Tue, 11 Feb 2020 12:52:58 -0500 Received: from mail.kernel.org ([198.145.29.99]:56192 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727911AbgBKRw5 (ORCPT ); Tue, 11 Feb 2020 12:52:57 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 09BF520661; Tue, 11 Feb 2020 17:52:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443577; bh=0Nz9UTGevazxDml65pFpWpvj0exBxapL78FXYgUMnz0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=IcFmD277CljVD6bmP3Wy7n2VaxV2vcftE3JKZII/M6aNiw0vusETXJ/mMll8RM3iJ opkauox9ZD21Jr/Q/fk6qUjJT+wAVs0olyUTGYowWGA69MI8ag4U4ixvviu16p9tt4 I14BqMlS+TphflWFD++Rq2bmdp+2ppObgLiIHPlA= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfn-004O7k-F6; Tue, 11 Feb 2020 17:50:11 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 31/94] KVM: arm64: nv: Only toggle cache for virtual EL2 when SCTLR_EL2 changes Date: Tue, 11 Feb 2020 17:48:35 +0000 Message-Id: <20200211174938.27809-32-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall So far we were flushing almost the entire universe whenever a VM would load/unload the SCTLR_EL1 and the two versions of that register had different MMU enabled settings. This turned out to be so slow that it prevented forward progress for a nested VM, because a scheduler timer tick interrupt would always be pending when we reached the nested VM. To avoid this problem, we consider the SCTLR_EL2 when evaluating if caches are on or off when entering virtual EL2 (because this is the value that we end up shadowing onto the hardware EL1 register). Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_mmu.h | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index ee47f7637f28..ec4de0613e7c 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -88,6 +88,7 @@ alternative_cb_end #include #include #include +#include void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); @@ -305,7 +306,10 @@ struct kvm; static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu) { - return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; + if (vcpu_mode_el2(vcpu)) + return (__vcpu_sys_reg(vcpu, SCTLR_EL2) & 0b101) == 0b101; + else + return (vcpu_read_sys_reg(vcpu, SCTLR_EL1) & 0b101) == 0b101; } static inline void __clean_dcache_guest_page(kvm_pfn_t pfn, unsigned long size) From patchwork Tue Feb 11 17:48:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376225 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CCCC414B4 for ; Tue, 11 Feb 2020 17:52:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A3C62208C3 for ; Tue, 11 Feb 2020 17:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443537; bh=ZhtAktcHS7kLOup2NvpgDOoQKcyB6JCtjzXZAdkr4Qk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=VDC57t2nPTipirn47Tk2c7c8Qbg6DwOy/w1BZgwCah9E6fN1mdgpbcSq+NIY5SFW3 KXKwEAa/XiIKJTW0ApK1dUTGEqRZVDp+SRN13ldWxznln5BvuAO9cX9qri1pqw9y2r XrI1Gfe6ISvaO5kjdQz1NFuhXvKNDYsVvVwluwAw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727966AbgBKRwR (ORCPT ); Tue, 11 Feb 2020 12:52:17 -0500 Received: from mail.kernel.org ([198.145.29.99]:55064 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730510AbgBKRwQ (ORCPT ); Tue, 11 Feb 2020 12:52:16 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AC8CE206D7; Tue, 11 Feb 2020 17:52:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443535; bh=ZhtAktcHS7kLOup2NvpgDOoQKcyB6JCtjzXZAdkr4Qk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=UgmYJQkYC3hHZJHVYsXADtdBJ9q7i1ZljH1Cn7dGgIgnysQiQloSvOcetNdf210gF Ea1E2EcQDCcWYA2yPtLqXuCONQR9i5234p48LVdWC/1k6o0j8E5vLS1Qb7n56zdq1y 7JRG3UJ6sByguvvASvGAHxQ9+rooSbNix/z5nHlQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfo-004O7k-0J; Tue, 11 Feb 2020 17:50:12 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 32/94] KVM: arm64: nv: Filter out unsupported features from ID regs Date: Tue, 11 Feb 2020 17:48:36 +0000 Message-Id: <20200211174938.27809-33-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As there is a number of features that we either can't support, or don't want to support right away with NV, let's add some basic filtering so that we don't advertize silly things to the EL2 guest. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 6 ++ arch/arm64/include/asm/sysreg.h | 11 +++ arch/arm64/kvm/nested.c | 115 ++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 5 +- 4 files changed, 136 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 5911a713c7dd..c258572b5ed4 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -16,4 +16,10 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); +struct sys_reg_params; +struct sys_reg_desc; + +void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, + const struct sys_reg_desc *r); + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 260c1cf683a2..360ef9e8dfe4 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -646,6 +646,9 @@ #define ID_AA64PFR0_CSV3_SHIFT 60 #define ID_AA64PFR0_CSV2_SHIFT 56 #define ID_AA64PFR0_DIT_SHIFT 48 +#define ID_AA64PFR0_AMU_SHIFT 44 +#define ID_AA64PFR0_MPAM_SHIFT 40 +#define ID_AA64PFR0_SEL2_SHIFT 36 #define ID_AA64PFR0_SVE_SHIFT 32 #define ID_AA64PFR0_RAS_SHIFT 28 #define ID_AA64PFR0_GIC_SHIFT 24 @@ -667,7 +670,9 @@ #define ID_AA64PFR0_EL0_32BIT_64BIT 0x2 /* id_aa64pfr1 */ +#define ID_AA64PFR1_MTE_SHIFT 8 #define ID_AA64PFR1_SSBS_SHIFT 4 +#define ID_AA64PFR1_BT_SHIFT 0 #define ID_AA64PFR1_SSBS_PSTATE_NI 0 #define ID_AA64PFR1_SSBS_PSTATE_ONLY 1 @@ -721,6 +726,8 @@ #endif /* id_aa64mmfr1 */ +#define ID_AA64MMFR1_XNX_SHIFT 28 +#define ID_AA64MMFR1_SpecSEI_SHIFT 24 #define ID_AA64MMFR1_PAN_SHIFT 20 #define ID_AA64MMFR1_LOR_SHIFT 16 #define ID_AA64MMFR1_HPD_SHIFT 12 @@ -733,9 +740,13 @@ /* id_aa64mmfr2 */ #define ID_AA64MMFR2_E0PD_SHIFT 60 +#define ID_AA64MMFR2_EVT_SHIFT 56 +#define ID_AA64MMFR2_BBM_SHIFT 52 #define ID_AA64MMFR2_FWB_SHIFT 40 #define ID_AA64MMFR2_AT_SHIFT 32 +#define ID_AA64MMFR2_ST_SHIFT 28 #define ID_AA64MMFR2_NV_SHIFT 24 +#define ID_AA64MMFR2_CCIDX_SHIFT 20 #define ID_AA64MMFR2_LVA_SHIFT 16 #define ID_AA64MMFR2_IESB_SHIFT 12 #define ID_AA64MMFR2_LSM_SHIFT 8 diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index c5e695cf81fb..cecea8d91196 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -20,6 +20,10 @@ #include #include +#include +#include + +#include "sys_regs.h" /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and @@ -38,3 +42,114 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) return -EINVAL; } + +#define FEATURE(x) (GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT)) + +/* + * Our emulated CPU doesn't support all the possible features. For the + * sake of simplicity (and probably mental sanity), wipe out a number + * of feature bits we don't intend to support for the time being. + * This list should get updated as new features get added to the NV + * support, and new extension to the architecture. + * + * Revisit: Implement as a whitelist rather than a blacklist? + */ +void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u32 id = sys_reg((u32)r->Op0, (u32)r->Op1, + (u32)r->CRn, (u32)r->CRm, (u32)r->Op2); + u64 val, tmp; + + if (!nested_virt_in_use(v)) + return; + + val = p->regval; + + switch (id) { + case SYS_ID_AA64DFR0_EL1: + /* No SPE */ + val &= ~FEATURE(ID_AA64DFR0_PMSVER); + /* Cap PMU to ARMv8.1 */ + tmp = FIELD_GET(FEATURE(ID_AA64DFR0_PMUVER), val); + if (tmp > 0b0100) { + val &= FEATURE(ID_AA64DFR0_PMUVER); + val |= FIELD_PREP(FEATURE(ID_AA64DFR0_PMUVER), 0b0100); + } + /* No trace */ + val &= FEATURE(ID_AA64DFR0_TRACEVER); + /* Cap Debug to ARMv8.1 */ + tmp = FIELD_GET(FEATURE(ID_AA64DFR0_DEBUGVER), val); + if (tmp > 0b0111) { + val &= FEATURE(ID_AA64DFR0_DEBUGVER); + val |= FIELD_PREP(FEATURE(ID_AA64DFR0_DEBUGVER), 0b0111); + } + break; + + case SYS_ID_AA64ISAR1_EL1: + /* No PtrAuth */ + val &= ~(FEATURE(ID_AA64ISAR1_APA) | + FEATURE(ID_AA64ISAR1_API) | + FEATURE(ID_AA64ISAR1_GPA) | + FEATURE(ID_AA64ISAR1_GPI)); + break; + + case SYS_ID_AA64MMFR0_EL1: + /* Cap PARange to 40bits */ + tmp = FIELD_GET(FEATURE(ID_AA64MMFR0_PARANGE), val); + if (tmp > 0b0010) { + val &= ~FEATURE(ID_AA64MMFR0_PARANGE); + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_PARANGE), 0b0010); + } + break; + + case SYS_ID_AA64MMFR1_EL1: + /* No XNX */ + val &= ~FEATURE(ID_AA64MMFR1_XNX); + /* No RAS */ + val &= ~FEATURE(ID_AA64MMFR1_SpecSEI); + /* No Hierarchical Permission Disable */ + val &= ~FEATURE(ID_AA64MMFR1_HPD); + /* No Hardward Access flags and Dirty Bit State update */ + val &= ~FEATURE(ID_AA64MMFR1_HADBS); + break; + + case SYS_ID_AA64MMFR2_EL1: + /* No ARMv8.2-EVT */ + val &= ~FEATURE(ID_AA64MMFR2_EVT); + /* No ARMv8.4-TTRem */ + val &= ~FEATURE(ID_AA64MMFR2_BBM); + /* No ARMv8.4-TTST */ + val &= ~FEATURE(ID_AA64MMFR2_ST); + /* No ARMv8.3-CCIDX */ + val &= ~FEATURE(ID_AA64MMFR2_CCIDX); + /* No ARMv8.2-LVA */ + val &= ~FEATURE(ID_AA64MMFR2_LVA); + break; + + case SYS_ID_AA64PFR0_EL1: + /* No AMU */ + val &= ~FEATURE(ID_AA64PFR0_AMU); + /* No MPAM */ + val &= ~FEATURE(ID_AA64PFR0_MPAM); + /* No Secure EL2 */ + val &= ~FEATURE(ID_AA64PFR0_SEL2); + /* No RAS */ + val &= ~FEATURE(ID_AA64PFR0_RAS); + /* No SVE */ + val &= ~FEATURE(ID_AA64PFR0_SVE); + /* EL2 is AArch64 only */ + val &= ~FEATURE(ID_AA64PFR0_EL2); + val |= FIELD_PREP(FEATURE(ID_AA64PFR0_EL2), 0b0001); + break; + + case SYS_ID_AA64PFR1_EL1: + /* No MTE */ + val &= ~FEATURE(ID_AA64PFR1_MTE); + /* No BT */ + val &= ~FEATURE(ID_AA64PFR1_BT); + break; + } + + p->regval = val; +} diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 435340a49634..966eb31a84e6 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1414,7 +1414,10 @@ static bool access_id_reg(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - return __access_id_reg(vcpu, p, r, false); + bool ret = __access_id_reg(vcpu, p, r, false); + + access_nested_id_reg(vcpu, p, r); + return ret; } static bool access_raz_id_reg(struct kvm_vcpu *vcpu, From patchwork Tue Feb 11 17:48:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376195 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE3BA14B4 for ; Tue, 11 Feb 2020 17:51:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AC83A208C3 for ; Tue, 11 Feb 2020 17:51:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443510; bh=zoNIyoYHhwbdd0bsnFoLdmLpXNlGzytX2uCWQSzCgPs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=qMKMy4nfwcsNtMhi6VQXotd+kFV0gEcjwjj03mtEqNLf4wCfm0FLOdqT+a2pkteE2 RmHfa5apxtLlrQ6WfMnCfuZNSNM+VrHCKdnzgtPFgiRuHjOBsOCyjrvPGVQM4eZkg6 D4CB03wvyJfF7XIeAY731xRHqhq4o4FRA7kuSvPc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730425AbgBKRvu (ORCPT ); Tue, 11 Feb 2020 12:51:50 -0500 Received: from mail.kernel.org ([198.145.29.99]:54424 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730304AbgBKRvt (ORCPT ); Tue, 11 Feb 2020 12:51:49 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id DE93D20848; Tue, 11 Feb 2020 17:51:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443509; bh=zoNIyoYHhwbdd0bsnFoLdmLpXNlGzytX2uCWQSzCgPs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=AJE2JSztRdtust1r5duNIrKHaokiDlXjwJZdJYyo42BV/MCTz+4cMOVVp7nvLElr+ dvOT2w3ebUF//S0TLPzTpXk5b+JF/3+0x7lg1WuX+hjKZMcGGklwFEEdqll4C2hrqo IA45nTChV0pXHHHtmQoJmeTnuwR+Zfj6mkxZmG4o= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfo-004O7k-JU; Tue, 11 Feb 2020 17:50:12 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 33/94] KVM: arm64: nv: Hide RAS from nested guests Date: Tue, 11 Feb 2020 17:48:37 +0000 Message-Id: <20200211174938.27809-34-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We don't want to expose complicated features to guests until we have a good grasp on the basic CPU emulation. So let's pretend that RAS, doesn't exist in a nested guest. We already hide the feature bits, let's now make sure VDISR_EL1 will UNDEF. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 966eb31a84e6..364020afc17c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -592,6 +592,14 @@ static bool trap_raz_wi(struct kvm_vcpu *vcpu, return read_zero(vcpu, p); } +static bool trap_undef(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + kvm_inject_undefined(vcpu); + return false; +} + /* * ARMv8.1 mandates at least a trivial LORegion implementation, where all the * RW registers are RES0 (which we can implement as RAZ/WI). On an ARMv8.0 @@ -2056,6 +2064,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_VBAR_EL2), access_rw, reset_val, VBAR_EL2, 0 }, { SYS_DESC(SYS_RVBAR_EL2), access_rw, reset_val, RVBAR_EL2, 0 }, { SYS_DESC(SYS_RMR_EL2), access_wi, reset_val, RMR_EL2, 1 }, + { SYS_DESC(SYS_VDISR_EL2), trap_undef }, { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, From patchwork Tue Feb 11 17:48:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376177 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF9401395 for ; Tue, 11 Feb 2020 17:51:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CEA4220661 for ; Tue, 11 Feb 2020 17:51:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443496; bh=nuQ0MN9/FGm7i28EpIRwCZK8BlzSNLL3qWI84G027to=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=NSDv3DzR84q7czyFOtMtMSXRBWJpG+KNJdefepQxAPLs/bh2PMxueQsp5dZHkovPG 450zkDo7++a2FppyHVe3PuGoKfG4N0P59SD68lgqOAJPC0tCpwKIAYY7HTkvamfxmw IvI8xGrKkXq8gPb91rStyjzMCWakDs/+XEAO4q10= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730364AbgBKRvf (ORCPT ); Tue, 11 Feb 2020 12:51:35 -0500 Received: from mail.kernel.org ([198.145.29.99]:53974 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730330AbgBKRvf (ORCPT ); Tue, 11 Feb 2020 12:51:35 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D14EA20870; Tue, 11 Feb 2020 17:51:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443494; bh=nuQ0MN9/FGm7i28EpIRwCZK8BlzSNLL3qWI84G027to=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rolyyoPuIhxBuyT6P+LDNWJEt7S0gbHBnRuxoO3pDpiT1W3uvPTj54mfLnzGvNJn+ KGORkhoUe9fKJI85d88XUE62BTUZwT1FxXGoyWmWoujF8nsrfh+zkCLbhaVjmtYsnp xDROX7tA1ILiT/6yP4V63hBMwenL6EBKRL7IXBU0= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfp-004O7k-5C; Tue, 11 Feb 2020 17:50:13 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 34/94] KVM: arm64: nv: Use ARMv8.5-GTG to advertise supported Stage-2 page sizes Date: Tue, 11 Feb 2020 17:48:38 +0000 Message-Id: <20200211174938.27809-35-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org ARMv8.5-GTG gives the opportunity to advertize the supported Stage-2 page sizes to hypervisors, and allow them to differ from the page sizes supported at Stage-1. As KVM cannot support guest Stage-2 page sizes that are smaller than PAGE_SIZE (it would break the guest's isolation guarantees), let's use this feature to let the guest know (assuming it has been told about ARMv8.5-GTG). Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/sysreg.h | 3 +++ arch/arm64/kvm/nested.c | 29 +++++++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 360ef9e8dfe4..a167219e42f4 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -701,6 +701,9 @@ #define ID_AA64ZFR0_SVEVER_SVE2 0x1 /* id_aa64mmfr0 */ +#define ID_AA64MMFR0_TGRAN4_2_SHIFT 40 +#define ID_AA64MMFR0_TGRAN64_2_SHIFT 36 +#define ID_AA64MMFR0_TGRAN16_2_SHIFT 32 #define ID_AA64MMFR0_TGRAN4_SHIFT 28 #define ID_AA64MMFR0_TGRAN64_SHIFT 24 #define ID_AA64MMFR0_TGRAN16_SHIFT 20 diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index cecea8d91196..c40bf753ead9 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -95,6 +95,35 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, break; case SYS_ID_AA64MMFR0_EL1: + /* Hide unsupported S2 page sizes */ + switch (PAGE_SIZE) { + case SZ_64K: + val &= ~FEATURE(ID_AA64MMFR0_TGRAN16_2); + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0001); + /* Fall through */ + case SZ_16K: + val &= ~FEATURE(ID_AA64MMFR0_TGRAN4_2); + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0001); + /* Fall through */ + case SZ_4K: + /* Support everything */ + break; + } + /* Advertize supported S2 page sizes */ + switch (PAGE_SIZE) { + case SZ_4K: + val &= ~FEATURE(ID_AA64MMFR0_TGRAN4_2); + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN4_2), 0b0010); + /* Fall through */ + case SZ_16K: + val &= ~FEATURE(ID_AA64MMFR0_TGRAN16_2); + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN16_2), 0b0010); + /* Fall through */ + case SZ_64K: + val &= ~FEATURE(ID_AA64MMFR0_TGRAN64_2); + val |= FIELD_PREP(FEATURE(ID_AA64MMFR0_TGRAN64_2), 0b0010); + break; + } /* Cap PARange to 40bits */ tmp = FIELD_GET(FEATURE(ID_AA64MMFR0_PARANGE), val); if (tmp > 0b0010) { From patchwork Tue Feb 11 17:48:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376273 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EF6BC14B4 for ; Tue, 11 Feb 2020 17:53:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D00BD20870 for ; Tue, 11 Feb 2020 17:53:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443581; bh=uSp/qawlZOx/68y9N8aqpzHmqTR1tge0KqerXZ4jmX0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Qu2ZbrNDx6ucCt3ZhTeZFCPvt04GuVjbEfuxELWs19NamsCgmKPgO8EwlqOoeDLkz LLFftkc3PDv+x53N/pPeM5XG1RO/RvNudjc8tGTOjUgZxxBh+bE1TnMaZogtTesA/l FjoS68RCI5RJJ49jxc3EN/lvFa4lfkIU3XVVWhOs= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730566AbgBKRxA (ORCPT ); Tue, 11 Feb 2020 12:53:00 -0500 Received: from mail.kernel.org ([198.145.29.99]:56258 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730656AbgBKRxA (ORCPT ); Tue, 11 Feb 2020 12:53:00 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D83FE20578; Tue, 11 Feb 2020 17:52:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443579; bh=uSp/qawlZOx/68y9N8aqpzHmqTR1tge0KqerXZ4jmX0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=1BBe8VosogcMLhIH1Cxow3e5tH7Wwclup3MQxlYhoLyks6Vq13UX5mWOjZRxZm6z5 coSjZmrZOD2IAtKdQTCTgeha/vBa5ose381LtQ1g+87ETGSrL5mZHc0a/jEt/M607O 53oOuEhj3X7V3/1Jl7MerCrqY3ORuFVMAlPQKoco= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfp-004O7k-OW; Tue, 11 Feb 2020 17:50:13 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 35/94] KVM: arm64: Check advertised Stage-2 page size capability Date: Tue, 11 Feb 2020 17:48:39 +0000 Message-Id: <20200211174938.27809-36-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org With ARMv8.5-GTG, the hardware (or more likely a hypervisor) can advertise the supported Stage-2 page sizes. Let's check this at boot time. Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_mmu.h | 2 +- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kernel/cpufeature.c | 8 +++++++ arch/arm64/kvm/reset.c | 39 ++++++++++++++++++++++++++++--- virt/kvm/arm/arm.c | 4 +--- 5 files changed, 47 insertions(+), 8 deletions(-) diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 0d84d50bf9ba..053a5f434b3e 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -418,7 +418,7 @@ static inline int hyp_map_aux_data(void) #define kvm_phys_to_vttbr(addr) (addr) -static inline void kvm_set_ipa_limit(void) {} +static inline int kvm_set_ipa_limit(void) { return 0; } static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) { diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 1bda11cc0f39..87ccceb9f5d8 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -734,7 +734,7 @@ static inline int kvm_arm_have_ssbd(void) void kvm_vcpu_load_sysregs(struct kvm_vcpu *vcpu); void kvm_vcpu_put_sysregs(struct kvm_vcpu *vcpu); -void kvm_set_ipa_limit(void); +int kvm_set_ipa_limit(void); #define __KVM_HAVE_ARCH_VM_ALLOC struct kvm *kvm_arch_alloc_vm(void); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index f0e58450eb16..157700590aa8 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -205,6 +205,14 @@ static const struct arm64_ftr_bits ftr_id_aa64zfr0[] = { }; static const struct arm64_ftr_bits ftr_id_aa64mmfr0[] = { + /* + * Page size not being supported at Stage-2 are not fatal. You + * just give up KVM if PAGE_SIZE isn't supported there. Go fix + * your favourite nesting hypervisor. + */ + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_EXACT, ID_AA64MMFR0_TGRAN4_2_SHIFT, 4, 1), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_EXACT, ID_AA64MMFR0_TGRAN64_2_SHIFT, 4, 1), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_EXACT, ID_AA64MMFR0_TGRAN16_2_SHIFT, 4, 1), /* * We already refuse to boot CPUs that don't support our configured * page size, so we can only detect mismatches for a page size other diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 5c50df274239..038765d1e60d 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -348,11 +348,42 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) return ret; } -void kvm_set_ipa_limit(void) +int kvm_set_ipa_limit(void) { - unsigned int ipa_max, pa_max, va_max, parange; + unsigned int ipa_max, pa_max, va_max, parange, tgran_2; + u64 mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1); - parange = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1) & 0x7; + /* + * Check with ARMv8.5-GTG that our PAGE_SIZE is supported at + * Stage-2. If not, things will stop very quickly. + */ + switch (PAGE_SIZE) { + default: + case SZ_4K: + tgran_2 = ID_AA64MMFR0_TGRAN4_2_SHIFT; + break; + case SZ_16K: + tgran_2 = ID_AA64MMFR0_TGRAN16_2_SHIFT; + break; + case SZ_64K: + tgran_2 = ID_AA64MMFR0_TGRAN64_2_SHIFT; + break; + } + + switch (FIELD_GET(0xFUL << tgran_2, mmfr0)) { + default: + case 1: + kvm_err("PAGE_SIZE not supported at Stage-2, giving up\n"); + return -EINVAL; + case 0: + kvm_debug("PAGE_SIZE supported at Stage-2 (default)\n"); + break; + case 2: + kvm_debug("PAGE_SIZE supported at Stage-2 (advertised)\n"); + break; + } + + parange = mmfr0 & 0x7; pa_max = id_aa64mmfr0_parange_to_phys_shift(parange); /* Clamp the IPA limit to the PA size supported by the kernel */ @@ -386,6 +417,8 @@ void kvm_set_ipa_limit(void) "KVM IPA limit (%d bit) is smaller than default size\n", ipa_max); kvm_ipa_limit = ipa_max; kvm_info("IPA Size Limit: %dbits\n", kvm_ipa_limit); + + return 0; } /* diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index d65a0faa46d8..d1359b2079df 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -1421,9 +1421,7 @@ static inline void hyp_cpu_pm_exit(void) static int init_common_resources(void) { - kvm_set_ipa_limit(); - - return 0; + return kvm_set_ipa_limit(); } static int init_subsystems(void) From patchwork Tue Feb 11 17:48:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376211 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6BDBC1395 for ; Tue, 11 Feb 2020 17:52:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 257AA21569 for ; Tue, 11 Feb 2020 17:52:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443528; bh=YpQtJoYsN2uIKX1l8jOSfdRYzh2kMb65gOe80OluOAs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=UOL3XjGuN3SiUdihhsBexp57222H9+PDWZRkRrcdwAsThGzUecP5wkJi/NdP3kDCn BOii9ChkTDK/k2HG1ov8zBzAQd4CYquAyBacv7iJkHNH5NpfxsyMXgQsl1CQkQpnrO kUcdU/ZFP7a3T0KoejM/wykKrGUThtZ2I943L+/8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730478AbgBKRwH (ORCPT ); Tue, 11 Feb 2020 12:52:07 -0500 Received: from mail.kernel.org ([198.145.29.99]:54774 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728935AbgBKRwG (ORCPT ); Tue, 11 Feb 2020 12:52:06 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2636420661; Tue, 11 Feb 2020 17:52:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443524; bh=YpQtJoYsN2uIKX1l8jOSfdRYzh2kMb65gOe80OluOAs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p7q9qe7t9SPmYbOAqnPcYib7PNGJAWdjHFlu7wezluNpRhhQKTHPhIJBIlUZJDoIL pRgrEHjoCRuOIri1s3aqZ7HWDRKiUCDg4pCejjN6q8QpsyDZtf0pQ+DBTuZpjBU4+1 +0yDMpuacUlp5Wb4m4agqP5UZg79GTmOOZtcojYs= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfq-004O7k-9O; Tue, 11 Feb 2020 17:50:14 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 36/94] KVM: arm/arm64: nv: Factor out stage 2 page table data from struct kvm Date: Tue, 11 Feb 2020 17:48:40 +0000 Message-Id: <20200211174938.27809-37-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall As we are about to reuse our stage 2 page table manipulation code for shadow stage 2 page tables in the context of nested virtualization, we are going to manage multiple stage 2 page tables for a single VM. This requires some pretty invasive changes to our data structures, which moves the vmid and pgd pointers into a separate structure and change pretty much all of our mmu code to operate on this structure instead. The new structre is called struct kvm_s2_mmu. There is no intended functional change by this patch alone. [Designed data structure layout in collaboration] Signed-off-by: Christoffer Dall Co-developed-by: Marc Zyngier Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_asm.h | 5 +- arch/arm/include/asm/kvm_host.h | 23 ++- arch/arm/include/asm/kvm_mmu.h | 10 +- arch/arm/kvm/hyp/switch.c | 3 +- arch/arm/kvm/hyp/tlb.c | 13 +- arch/arm64/include/asm/kvm_asm.h | 5 +- arch/arm64/include/asm/kvm_host.h | 24 ++- arch/arm64/include/asm/kvm_mmu.h | 16 +- arch/arm64/kvm/hyp/switch.c | 8 +- arch/arm64/kvm/hyp/tlb.c | 48 +++--- virt/kvm/arm/arm.c | 17 +- virt/kvm/arm/mmu.c | 250 ++++++++++++++++-------------- 12 files changed, 230 insertions(+), 192 deletions(-) diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h index f615830f9f57..4f85323f1290 100644 --- a/arch/arm/include/asm/kvm_asm.h +++ b/arch/arm/include/asm/kvm_asm.h @@ -49,13 +49,14 @@ #ifndef __ASSEMBLY__ struct kvm; struct kvm_vcpu; +struct kvm_s2_mmu; extern char __kvm_hyp_init[]; extern char __kvm_hyp_init_end[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa); -extern void __kvm_tlb_flush_vmid(struct kvm *kvm); +extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high); diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index c3314b286a61..f43a68d8ea50 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -55,18 +55,23 @@ struct kvm_vmid { u32 vmid; }; +struct kvm_s2_mmu { + /* The VMID generation used for the virt. memory system */ + struct kvm_vmid vmid; + + /* Stage-2 page table */ + pgd_t *pgd; + phys_addr_t pgd_phys; + + struct kvm *kvm; +}; + struct kvm_arch { + struct kvm_s2_mmu mmu; + /* The last vcpu id that ran on each physical CPU */ int __percpu *last_vcpu_ran; - /* - * Anything that is not used directly from assembly code goes - * here. - */ - - /* The VMID generation used for the virt. memory system */ - struct kvm_vmid vmid; - /* Stage-2 page table */ pgd_t *pgd; phys_addr_t pgd_phys; @@ -171,6 +176,8 @@ struct vcpu_reset_state { struct kvm_vcpu_arch { struct kvm_cpu_context ctxt; + struct kvm_s2_mmu *hw_mmu; + int target; /* Processor target */ DECLARE_BITMAP(features, KVM_VCPU_MAX_FEATURES); diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 053a5f434b3e..2d49950f8634 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -52,8 +52,8 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void free_hyp_pgds(void); void stage2_unmap_vm(struct kvm *kvm); -int kvm_alloc_stage2_pgd(struct kvm *kvm); -void kvm_free_stage2_pgd(struct kvm *kvm); +int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu); +void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); @@ -420,12 +420,12 @@ static inline int hyp_map_aux_data(void) static inline int kvm_set_ipa_limit(void) { return 0; } -static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) +static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) { - struct kvm_vmid *vmid = &kvm->arch.vmid; + struct kvm_vmid *vmid = &mmu->vmid; u64 vmid_field, baddr; - baddr = kvm->arch.pgd_phys; + baddr = mmu->pgd_phys; vmid_field = (u64)vmid->vmid << VTTBR_VMID_SHIFT; return kvm_phys_to_vttbr(baddr) | vmid_field; } diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c index 1efeef3fd0ee..0b53e79f3bfb 100644 --- a/arch/arm/kvm/hyp/switch.c +++ b/arch/arm/kvm/hyp/switch.c @@ -65,8 +65,7 @@ static void __hyp_text __deactivate_traps(struct kvm_vcpu *vcpu) static void __hyp_text __activate_vm(struct kvm_vcpu *vcpu) { - struct kvm *kvm = kern_hyp_va(vcpu->kvm); - write_sysreg(kvm_get_vttbr(kvm), VTTBR); + write_sysreg(kvm_get_vttbr(vcpu->arch.hw_mmu), VTTBR); write_sysreg(vcpu->arch.midr, VPIDR); } diff --git a/arch/arm/kvm/hyp/tlb.c b/arch/arm/kvm/hyp/tlb.c index 848f27bbad9d..8833e0b0ea97 100644 --- a/arch/arm/kvm/hyp/tlb.c +++ b/arch/arm/kvm/hyp/tlb.c @@ -24,13 +24,12 @@ * As v7 does not support flushing per IPA, just nuke the whole TLB * instead, ignoring the ipa value. */ -void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) +void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { dsb(ishst); /* Switch to requested VMID */ - kvm = kern_hyp_va(kvm); - write_sysreg(kvm_get_vttbr(kvm), VTTBR); + write_sysreg(kvm_get_vttbr(mmu), VTTBR); isb(); write_sysreg(0, TLBIALLIS); @@ -40,17 +39,15 @@ void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) write_sysreg(0, VTTBR); } -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) { - __kvm_tlb_flush_vmid(kvm); + __kvm_tlb_flush_vmid(mmu); } void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu) { - struct kvm *kvm = kern_hyp_va(kern_hyp_va(vcpu)->kvm); - /* Switch to requested VMID */ - write_sysreg(kvm_get_vttbr(kvm), VTTBR); + write_sysreg(kvm_get_vttbr(vcpu->arch.hw_mmu), VTTBR); isb(); write_sysreg(0, TLBIALL); diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 44a243754c1b..80f5633d9559 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -51,6 +51,7 @@ struct kvm; struct kvm_vcpu; +struct kvm_s2_mmu; extern char __kvm_hyp_init[]; extern char __kvm_hyp_init_end[]; @@ -58,8 +59,8 @@ extern char __kvm_hyp_init_end[]; extern char __kvm_hyp_vector[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa); -extern void __kvm_tlb_flush_vmid(struct kvm *kvm); +extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high); diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 87ccceb9f5d8..42ca60c8b4c8 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -62,12 +62,25 @@ struct kvm_vmid { u32 vmid; }; -struct kvm_arch { +struct kvm_s2_mmu { struct kvm_vmid vmid; - /* stage2 entry level table */ - pgd_t *pgd; - phys_addr_t pgd_phys; + /* + * stage2 entry level table + * + * Two kvm_s2_mmu structures in the same VM can point to the same pgd + * here. This happens when running a non-VHE guest hypervisor which + * uses the canonical stage 2 page table for both vEL2 and for vEL1/0 + * with vHCR_EL2.VM == 0. + */ + pgd_t *pgd; + phys_addr_t pgd_phys; + + struct kvm *kvm; +}; + +struct kvm_arch { + struct kvm_s2_mmu mmu; /* VTCR_EL2 value for this VM */ u64 vtcr; @@ -287,6 +300,9 @@ struct kvm_vcpu_arch { void *sve_state; unsigned int sve_max_vl; + /* Stage 2 paging state used by the hardware on next switch */ + struct kvm_s2_mmu *hw_mmu; + /* HYP configuration */ u64 hcr_el2; u32 mdcr_el2; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index ec4de0613e7c..ea1a597650f6 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -155,8 +155,8 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void free_hyp_pgds(void); void stage2_unmap_vm(struct kvm *kvm); -int kvm_alloc_stage2_pgd(struct kvm *kvm); -void kvm_free_stage2_pgd(struct kvm *kvm); +int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu); +void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); @@ -597,13 +597,13 @@ static inline u64 kvm_vttbr_baddr_mask(struct kvm *kvm) return vttbr_baddr_mask(kvm_phys_shift(kvm), kvm_stage2_levels(kvm)); } -static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) +static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) { - struct kvm_vmid *vmid = &kvm->arch.vmid; + struct kvm_vmid *vmid = &mmu->vmid; u64 vmid_field, baddr; u64 cnp = system_supports_cnp() ? VTTBR_CNP_BIT : 0; - baddr = kvm->arch.pgd_phys; + baddr = mmu->pgd_phys; vmid_field = (u64)vmid->vmid << VTTBR_VMID_SHIFT; return kvm_phys_to_vttbr(baddr) | vmid_field | cnp; } @@ -612,10 +612,10 @@ static __always_inline u64 kvm_get_vttbr(struct kvm *kvm) * Must be called from hyp code running at EL2 with an updated VTTBR * and interrupts disabled. */ -static __always_inline void __load_guest_stage2(struct kvm *kvm) +static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) { - write_sysreg(kvm->arch.vtcr, vtcr_el2); - write_sysreg(kvm_get_vttbr(kvm), vttbr_el2); + write_sysreg(kern_hyp_va(mmu->kvm)->arch.vtcr, vtcr_el2); + write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* * ARM erratum 1165522 requires the actual execution of the above diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index fd4085ebf062..08f12c96c458 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -281,9 +281,9 @@ void deactivate_traps_vhe_put(void) __deactivate_traps_common(); } -static void __hyp_text __activate_vm(struct kvm *kvm) +static void __hyp_text __activate_vm(struct kvm_s2_mmu *mmu) { - __load_guest_stage2(kvm); + __load_guest_stage2(mmu); } static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu) @@ -691,7 +691,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) * stage 2 translation, and __activate_traps clear HCR_EL2.TGE * (among other things). */ - __activate_vm(vcpu->kvm); + __activate_vm(vcpu->arch.hw_mmu); __activate_traps(vcpu); sysreg_restore_guest_state_vhe(guest_ctxt); @@ -763,7 +763,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu) __sysreg32_restore_state(vcpu); __sysreg_restore_state_nvhe(guest_ctxt); - __activate_vm(kern_hyp_va(vcpu->kvm)); + __activate_vm(kern_hyp_va(vcpu->arch.hw_mmu)); __activate_traps(vcpu); __hyp_vgic_restore_state(vcpu); diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index 92f560e3e1aa..a0d2bdba9f4e 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -16,7 +16,7 @@ struct tlb_inv_context { u64 sctlr; }; -static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm *kvm, +static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm_s2_mmu *mmu, struct tlb_inv_context *cxt) { u64 val; @@ -53,14 +53,14 @@ static void __hyp_text __tlb_switch_to_guest_vhe(struct kvm *kvm, * place before clearing TGE. __load_guest_stage2() already * has an ISB in order to deal with this. */ - __load_guest_stage2(kvm); + __load_guest_stage2(mmu); val = read_sysreg(hcr_el2); val &= ~HCR_TGE; write_sysreg(val, hcr_el2); isb(); } -static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm *kvm, +static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm_s2_mmu *mmu, struct tlb_inv_context *cxt) { if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) { @@ -79,21 +79,20 @@ static void __hyp_text __tlb_switch_to_guest_nvhe(struct kvm *kvm, isb(); } - __load_guest_stage2(kvm); + __load_guest_stage2(mmu); isb(); } -static void __hyp_text __tlb_switch_to_guest(struct kvm *kvm, +static void __hyp_text __tlb_switch_to_guest(struct kvm_s2_mmu *mmu, struct tlb_inv_context *cxt) { if (has_vhe()) - __tlb_switch_to_guest_vhe(kvm, cxt); + __tlb_switch_to_guest_vhe(mmu, cxt); else - __tlb_switch_to_guest_nvhe(kvm, cxt); + __tlb_switch_to_guest_nvhe(mmu, cxt); } -static void __hyp_text __tlb_switch_to_host_vhe(struct kvm *kvm, - struct tlb_inv_context *cxt) +static void __hyp_text __tlb_switch_to_host_vhe(struct tlb_inv_context *cxt) { /* * We're done with the TLB operation, let's restore the host's @@ -112,8 +111,7 @@ static void __hyp_text __tlb_switch_to_host_vhe(struct kvm *kvm, local_irq_restore(cxt->flags); } -static void __hyp_text __tlb_switch_to_host_nvhe(struct kvm *kvm, - struct tlb_inv_context *cxt) +static void __hyp_text __tlb_switch_to_host_nvhe(struct tlb_inv_context *cxt) { write_sysreg(0, vttbr_el2); @@ -125,24 +123,24 @@ static void __hyp_text __tlb_switch_to_host_nvhe(struct kvm *kvm, } } -static void __hyp_text __tlb_switch_to_host(struct kvm *kvm, +static void __hyp_text __tlb_switch_to_host(struct kvm_s2_mmu *mmu, struct tlb_inv_context *cxt) { if (has_vhe()) - __tlb_switch_to_host_vhe(kvm, cxt); + __tlb_switch_to_host_vhe(cxt); else - __tlb_switch_to_host_nvhe(kvm, cxt); + __tlb_switch_to_host_nvhe(cxt); } -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) { struct tlb_inv_context cxt; dsb(ishst); /* Switch to requested VMID */ - kvm = kern_hyp_va(kvm); - __tlb_switch_to_guest(kvm, &cxt); + mmu = kern_hyp_va(mmu); + __tlb_switch_to_guest(mmu, &cxt); /* * We could do so much better if we had the VA as well. @@ -185,39 +183,39 @@ void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) if (!has_vhe() && icache_is_vpipt()) __flush_icache_all(); - __tlb_switch_to_host(kvm, &cxt); + __tlb_switch_to_host(mmu, &cxt); } -void __hyp_text __kvm_tlb_flush_vmid(struct kvm *kvm) +void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { struct tlb_inv_context cxt; dsb(ishst); /* Switch to requested VMID */ - kvm = kern_hyp_va(kvm); - __tlb_switch_to_guest(kvm, &cxt); + mmu = kern_hyp_va(mmu); + __tlb_switch_to_guest(mmu, &cxt); __tlbi(vmalls12e1is); dsb(ish); isb(); - __tlb_switch_to_host(kvm, &cxt); + __tlb_switch_to_host(mmu, &cxt); } void __hyp_text __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu) { - struct kvm *kvm = kern_hyp_va(kern_hyp_va(vcpu)->kvm); + struct kvm_s2_mmu *mmu = kern_hyp_va(kern_hyp_va(vcpu)->arch.hw_mmu); struct tlb_inv_context cxt; /* Switch to requested VMID */ - __tlb_switch_to_guest(kvm, &cxt); + __tlb_switch_to_guest(mmu, &cxt); __tlbi(vmalle1); dsb(nsh); isb(); - __tlb_switch_to_host(kvm, &cxt); + __tlb_switch_to_host(mmu, &cxt); } void __hyp_text __kvm_flush_vm_context(void) diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index d1359b2079df..0df8d281a7c6 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -114,7 +114,11 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) for_each_possible_cpu(cpu) *per_cpu_ptr(kvm->arch.last_vcpu_ran, cpu) = -1; - ret = kvm_alloc_stage2_pgd(kvm); + /* Mark the initial VMID generation invalid */ + kvm->arch.mmu.vmid.vmid_gen = 0; + kvm->arch.mmu.kvm = kvm; + + ret = kvm_alloc_stage2_pgd(&kvm->arch.mmu); if (ret) goto out_fail_alloc; @@ -124,16 +128,13 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) kvm_vgic_early_init(kvm); - /* Mark the initial VMID generation invalid */ - kvm->arch.vmid.vmid_gen = 0; - /* The maximum number of VCPUs is limited by the host's GIC model */ kvm->arch.max_vcpus = vgic_present ? kvm_vgic_get_max_vcpus() : KVM_MAX_VCPUS; return ret; out_free_stage2_pgd: - kvm_free_stage2_pgd(kvm); + kvm_free_stage2_pgd(&kvm->arch.mmu); out_fail_alloc: free_percpu(kvm->arch.last_vcpu_ran); kvm->arch.last_vcpu_ran = NULL; @@ -279,6 +280,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu) kvm_arm_pvtime_vcpu_init(&vcpu->arch); + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + err = kvm_vgic_vcpu_init(vcpu); if (err) return err; @@ -671,7 +674,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) */ cond_resched(); - update_vmid(&vcpu->kvm->arch.vmid); + update_vmid(&vcpu->arch.hw_mmu->vmid); check_vcpu_requests(vcpu); @@ -720,7 +723,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) */ smp_store_mb(vcpu->mode, IN_GUEST_MODE); - if (ret <= 0 || need_new_vmid_gen(&vcpu->kvm->arch.vmid) || + if (ret <= 0 || need_new_vmid_gen(&vcpu->arch.hw_mmu->vmid) || kvm_request_pending(vcpu)) { vcpu->mode = OUTSIDE_GUEST_MODE; isb(); /* Ensure work in x_flush_hwstate is committed */ diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 19c961ac4e3c..ef46686da335 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -55,12 +55,12 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) */ void kvm_flush_remote_tlbs(struct kvm *kvm) { - kvm_call_hyp(__kvm_tlb_flush_vmid, kvm); + kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); } -static void kvm_tlb_flush_vmid_ipa(struct kvm *kvm, phys_addr_t ipa) +static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, kvm, ipa); + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa); } /* @@ -96,31 +96,33 @@ static bool kvm_is_device_pfn(unsigned long pfn) * * Function clears a PMD entry, flushes addr 1st and 2nd stage TLBs. */ -static void stage2_dissolve_pmd(struct kvm *kvm, phys_addr_t addr, pmd_t *pmd) +static void stage2_dissolve_pmd(struct kvm_s2_mmu *mmu, phys_addr_t addr, pmd_t *pmd) { if (!pmd_thp_or_huge(*pmd)) return; pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); put_page(virt_to_page(pmd)); } /** * stage2_dissolve_pud() - clear and flush huge PUD entry - * @kvm: pointer to kvm structure. + * @mmu: pointer to mmu structure to operate on * @addr: IPA * @pud: pud pointer for IPA * * Function clears a PUD entry, flushes addr 1st and 2nd stage TLBs. */ -static void stage2_dissolve_pud(struct kvm *kvm, phys_addr_t addr, pud_t *pudp) +static void stage2_dissolve_pud(struct kvm_s2_mmu *mmu, phys_addr_t addr, pud_t *pudp) { + struct kvm *kvm __maybe_unused = mmu->kvm; + if (!stage2_pud_huge(kvm, *pudp)) return; stage2_pud_clear(kvm, pudp); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); put_page(virt_to_page(pudp)); } @@ -156,31 +158,35 @@ static void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc) return p; } -static void clear_stage2_pgd_entry(struct kvm *kvm, pgd_t *pgd, phys_addr_t addr) +static void clear_stage2_pgd_entry(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addr_t addr) { + struct kvm *kvm __maybe_unused = mmu->kvm; + pud_t *pud_table __maybe_unused = stage2_pud_offset(kvm, pgd, 0UL); stage2_pgd_clear(kvm, pgd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); stage2_pud_free(kvm, pud_table); put_page(virt_to_page(pgd)); } -static void clear_stage2_pud_entry(struct kvm *kvm, pud_t *pud, phys_addr_t addr) +static void clear_stage2_pud_entry(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addr_t addr) { + struct kvm *kvm __maybe_unused = mmu->kvm; + pmd_t *pmd_table __maybe_unused = stage2_pmd_offset(kvm, pud, 0); VM_BUG_ON(stage2_pud_huge(kvm, *pud)); stage2_pud_clear(kvm, pud); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); stage2_pmd_free(kvm, pmd_table); put_page(virt_to_page(pud)); } -static void clear_stage2_pmd_entry(struct kvm *kvm, pmd_t *pmd, phys_addr_t addr) +static void clear_stage2_pmd_entry(struct kvm_s2_mmu *mmu, pmd_t *pmd, phys_addr_t addr) { pte_t *pte_table = pte_offset_kernel(pmd, 0); VM_BUG_ON(pmd_thp_or_huge(*pmd)); pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); free_page((unsigned long)pte_table); put_page(virt_to_page(pmd)); } @@ -238,7 +244,7 @@ static inline void kvm_pgd_populate(pgd_t *pgdp, pud_t *pudp) * we then fully enforce cacheability of RAM, no matter what the guest * does. */ -static void unmap_stage2_ptes(struct kvm *kvm, pmd_t *pmd, +static void unmap_stage2_ptes(struct kvm_s2_mmu *mmu, pmd_t *pmd, phys_addr_t addr, phys_addr_t end) { phys_addr_t start_addr = addr; @@ -250,7 +256,7 @@ static void unmap_stage2_ptes(struct kvm *kvm, pmd_t *pmd, pte_t old_pte = *pte; kvm_set_pte(pte, __pte(0)); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); /* No need to invalidate the cache for device mappings */ if (!kvm_is_device_pfn(pte_pfn(old_pte))) @@ -260,13 +266,14 @@ static void unmap_stage2_ptes(struct kvm *kvm, pmd_t *pmd, } } while (pte++, addr += PAGE_SIZE, addr != end); - if (stage2_pte_table_empty(kvm, start_pte)) - clear_stage2_pmd_entry(kvm, pmd, start_addr); + if (stage2_pte_table_empty(mmu->kvm, start_pte)) + clear_stage2_pmd_entry(mmu, pmd, start_addr); } -static void unmap_stage2_pmds(struct kvm *kvm, pud_t *pud, +static void unmap_stage2_pmds(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addr_t addr, phys_addr_t end) { + struct kvm *kvm __maybe_unused = mmu->kvm; phys_addr_t next, start_addr = addr; pmd_t *pmd, *start_pmd; @@ -278,24 +285,25 @@ static void unmap_stage2_pmds(struct kvm *kvm, pud_t *pud, pmd_t old_pmd = *pmd; pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); kvm_flush_dcache_pmd(old_pmd); put_page(virt_to_page(pmd)); } else { - unmap_stage2_ptes(kvm, pmd, addr, next); + unmap_stage2_ptes(mmu, pmd, addr, next); } } } while (pmd++, addr = next, addr != end); if (stage2_pmd_table_empty(kvm, start_pmd)) - clear_stage2_pud_entry(kvm, pud, start_addr); + clear_stage2_pud_entry(mmu, pud, start_addr); } -static void unmap_stage2_puds(struct kvm *kvm, pgd_t *pgd, +static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addr_t addr, phys_addr_t end) { + struct kvm *kvm __maybe_unused = mmu->kvm; phys_addr_t next, start_addr = addr; pud_t *pud, *start_pud; @@ -307,17 +315,17 @@ static void unmap_stage2_puds(struct kvm *kvm, pgd_t *pgd, pud_t old_pud = *pud; stage2_pud_clear(kvm, pud); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); kvm_flush_dcache_pud(old_pud); put_page(virt_to_page(pud)); } else { - unmap_stage2_pmds(kvm, pud, addr, next); + unmap_stage2_pmds(mmu, pud, addr, next); } } } while (pud++, addr = next, addr != end); if (stage2_pud_table_empty(kvm, start_pud)) - clear_stage2_pgd_entry(kvm, pgd, start_addr); + clear_stage2_pgd_entry(mmu, pgd, start_addr); } /** @@ -331,8 +339,9 @@ static void unmap_stage2_puds(struct kvm *kvm, pgd_t *pgd, * destroying the VM), otherwise another faulting VCPU may come in and mess * with things behind our backs. */ -static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size) +static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) { + struct kvm *kvm = mmu->kvm; pgd_t *pgd; phys_addr_t addr = start, end = start + size; phys_addr_t next; @@ -340,18 +349,18 @@ static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size) assert_spin_locked(&kvm->mmu_lock); WARN_ON(size & ~PAGE_MASK); - pgd = kvm->arch.pgd + stage2_pgd_index(kvm, addr); + pgd = mmu->pgd + stage2_pgd_index(kvm, addr); do { /* * Make sure the page table is still active, as another thread * could have possibly freed the page table, while we released * the lock. */ - if (!READ_ONCE(kvm->arch.pgd)) + if (!READ_ONCE(mmu->pgd)) break; next = stage2_pgd_addr_end(kvm, addr, end); if (!stage2_pgd_none(kvm, *pgd)) - unmap_stage2_puds(kvm, pgd, addr, next); + unmap_stage2_puds(mmu, pgd, addr, next); /* * If the range is too large, release the kvm->mmu_lock * to prevent starvation and lockup detector warnings. @@ -361,7 +370,7 @@ static void unmap_stage2_range(struct kvm *kvm, phys_addr_t start, u64 size) } while (pgd++, addr = next, addr != end); } -static void stage2_flush_ptes(struct kvm *kvm, pmd_t *pmd, +static void stage2_flush_ptes(struct kvm_s2_mmu *mmu, pmd_t *pmd, phys_addr_t addr, phys_addr_t end) { pte_t *pte; @@ -373,9 +382,10 @@ static void stage2_flush_ptes(struct kvm *kvm, pmd_t *pmd, } while (pte++, addr += PAGE_SIZE, addr != end); } -static void stage2_flush_pmds(struct kvm *kvm, pud_t *pud, +static void stage2_flush_pmds(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addr_t addr, phys_addr_t end) { + struct kvm *kvm = mmu->kvm; pmd_t *pmd; phys_addr_t next; @@ -386,14 +396,15 @@ static void stage2_flush_pmds(struct kvm *kvm, pud_t *pud, if (pmd_thp_or_huge(*pmd)) kvm_flush_dcache_pmd(*pmd); else - stage2_flush_ptes(kvm, pmd, addr, next); + stage2_flush_ptes(mmu, pmd, addr, next); } } while (pmd++, addr = next, addr != end); } -static void stage2_flush_puds(struct kvm *kvm, pgd_t *pgd, +static void stage2_flush_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addr_t addr, phys_addr_t end) { + struct kvm *kvm __maybe_unused = mmu->kvm; pud_t *pud; phys_addr_t next; @@ -404,24 +415,25 @@ static void stage2_flush_puds(struct kvm *kvm, pgd_t *pgd, if (stage2_pud_huge(kvm, *pud)) kvm_flush_dcache_pud(*pud); else - stage2_flush_pmds(kvm, pud, addr, next); + stage2_flush_pmds(mmu, pud, addr, next); } } while (pud++, addr = next, addr != end); } -static void stage2_flush_memslot(struct kvm *kvm, +static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, struct kvm_memory_slot *memslot) { + struct kvm *kvm = mmu->kvm; phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT; phys_addr_t end = addr + PAGE_SIZE * memslot->npages; phys_addr_t next; pgd_t *pgd; - pgd = kvm->arch.pgd + stage2_pgd_index(kvm, addr); + pgd = mmu->pgd + stage2_pgd_index(kvm, addr); do { next = stage2_pgd_addr_end(kvm, addr, end); if (!stage2_pgd_none(kvm, *pgd)) - stage2_flush_puds(kvm, pgd, addr, next); + stage2_flush_puds(mmu, pgd, addr, next); } while (pgd++, addr = next, addr != end); } @@ -443,7 +455,7 @@ static void stage2_flush_vm(struct kvm *kvm) slots = kvm_memslots(kvm); kvm_for_each_memslot(memslot, slots) - stage2_flush_memslot(kvm, memslot); + stage2_flush_memslot(&kvm->arch.mmu, memslot); spin_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, idx); @@ -887,35 +899,35 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, /** * kvm_alloc_stage2_pgd - allocate level-1 table for stage-2 translation. - * @kvm: The KVM struct pointer for the VM. + * @mmu: The stage 2 mmu struct pointer * * Allocates only the stage-2 HW PGD level table(s) of size defined by - * stage2_pgd_size(kvm). + * stage2_pgd_size(mmu->kvm). * * Note we don't need locking here as this is only called when the VM is * created, which can only be done once. */ -int kvm_alloc_stage2_pgd(struct kvm *kvm) +int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu) { phys_addr_t pgd_phys; pgd_t *pgd; - if (kvm->arch.pgd != NULL) { + if (mmu->pgd != NULL) { kvm_err("kvm_arch already initialized?\n"); return -EINVAL; } /* Allocate the HW PGD, making sure that each page gets its own refcount */ - pgd = alloc_pages_exact(stage2_pgd_size(kvm), GFP_KERNEL | __GFP_ZERO); + pgd = alloc_pages_exact(stage2_pgd_size(mmu->kvm), GFP_KERNEL | __GFP_ZERO); if (!pgd) return -ENOMEM; pgd_phys = virt_to_phys(pgd); - if (WARN_ON(pgd_phys & ~kvm_vttbr_baddr_mask(kvm))) + if (WARN_ON(pgd_phys & ~kvm_vttbr_baddr_mask(mmu->kvm))) return -EINVAL; - kvm->arch.pgd = pgd; - kvm->arch.pgd_phys = pgd_phys; + mmu->pgd = pgd; + mmu->pgd_phys = pgd_phys; return 0; } @@ -954,7 +966,7 @@ static void stage2_unmap_memslot(struct kvm *kvm, if (!(vma->vm_flags & VM_PFNMAP)) { gpa_t gpa = addr + (vm_start - memslot->userspace_addr); - unmap_stage2_range(kvm, gpa, vm_end - vm_start); + unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start); } hva = vm_end; } while (hva < reg_end); @@ -986,24 +998,16 @@ void stage2_unmap_vm(struct kvm *kvm) srcu_read_unlock(&kvm->srcu, idx); } -/** - * kvm_free_stage2_pgd - free all stage-2 tables - * @kvm: The KVM struct pointer for the VM. - * - * Walks the level-1 page table pointed to by kvm->arch.pgd and frees all - * underlying level-2 and level-3 tables before freeing the actual level-1 table - * and setting the struct pointer to NULL. - */ -void kvm_free_stage2_pgd(struct kvm *kvm) +void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) { + struct kvm *kvm = mmu->kvm; void *pgd = NULL; spin_lock(&kvm->mmu_lock); - if (kvm->arch.pgd) { - unmap_stage2_range(kvm, 0, kvm_phys_size(kvm)); - pgd = READ_ONCE(kvm->arch.pgd); - kvm->arch.pgd = NULL; - kvm->arch.pgd_phys = 0; + if (mmu->pgd) { + unmap_stage2_range(mmu, 0, kvm_phys_size(kvm)); + pgd = READ_ONCE(mmu->pgd); + mmu->pgd = NULL; } spin_unlock(&kvm->mmu_lock); @@ -1012,13 +1016,14 @@ void kvm_free_stage2_pgd(struct kvm *kvm) free_pages_exact(pgd, stage2_pgd_size(kvm)); } -static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, +static pud_t *stage2_get_pud(struct kvm_s2_mmu *mmu, struct kvm_mmu_memory_cache *cache, phys_addr_t addr) { + struct kvm *kvm __maybe_unused = mmu->kvm; pgd_t *pgd; pud_t *pud; - pgd = kvm->arch.pgd + stage2_pgd_index(kvm, addr); + pgd = mmu->pgd + stage2_pgd_index(kvm, addr); if (stage2_pgd_none(kvm, *pgd)) { if (!cache) return NULL; @@ -1030,13 +1035,14 @@ static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache return stage2_pud_offset(kvm, pgd, addr); } -static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, +static pmd_t *stage2_get_pmd(struct kvm_s2_mmu *mmu, struct kvm_mmu_memory_cache *cache, phys_addr_t addr) { + struct kvm *kvm __maybe_unused = mmu->kvm; pud_t *pud; pmd_t *pmd; - pud = stage2_get_pud(kvm, cache, addr); + pud = stage2_get_pud(mmu, cache, addr); if (!pud || stage2_pud_huge(kvm, *pud)) return NULL; @@ -1051,13 +1057,14 @@ static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache return stage2_pmd_offset(kvm, pud, addr); } -static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache - *cache, phys_addr_t addr, const pmd_t *new_pmd) +static int stage2_set_pmd_huge(struct kvm_s2_mmu *mmu, + struct kvm_mmu_memory_cache *cache, + phys_addr_t addr, const pmd_t *new_pmd) { pmd_t *pmd, old_pmd; retry: - pmd = stage2_get_pmd(kvm, cache, addr); + pmd = stage2_get_pmd(mmu, cache, addr); VM_BUG_ON(!pmd); old_pmd = *pmd; @@ -1090,7 +1097,7 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache * get handled accordingly. */ if (!pmd_thp_or_huge(old_pmd)) { - unmap_stage2_range(kvm, addr & S2_PMD_MASK, S2_PMD_SIZE); + unmap_stage2_range(mmu, addr & S2_PMD_MASK, S2_PMD_SIZE); goto retry; } /* @@ -1106,7 +1113,7 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache */ WARN_ON_ONCE(pmd_pfn(old_pmd) != pmd_pfn(*new_pmd)); pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); } else { get_page(virt_to_page(pmd)); } @@ -1115,13 +1122,15 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache return 0; } -static int stage2_set_pud_huge(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, +static int stage2_set_pud_huge(struct kvm_s2_mmu *mmu, + struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pud_t *new_pudp) { + struct kvm *kvm __maybe_unused = mmu->kvm; pud_t *pudp, old_pud; retry: - pudp = stage2_get_pud(kvm, cache, addr); + pudp = stage2_get_pud(mmu, cache, addr); VM_BUG_ON(!pudp); old_pud = *pudp; @@ -1140,13 +1149,13 @@ static int stage2_set_pud_huge(struct kvm *kvm, struct kvm_mmu_memory_cache *cac * the range for this block and retry. */ if (!stage2_pud_huge(kvm, old_pud)) { - unmap_stage2_range(kvm, addr & S2_PUD_MASK, S2_PUD_SIZE); + unmap_stage2_range(mmu, addr & S2_PUD_MASK, S2_PUD_SIZE); goto retry; } WARN_ON_ONCE(kvm_pud_pfn(old_pud) != kvm_pud_pfn(*new_pudp)); stage2_pud_clear(kvm, pudp); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); } else { get_page(virt_to_page(pudp)); } @@ -1161,9 +1170,10 @@ static int stage2_set_pud_huge(struct kvm *kvm, struct kvm_mmu_memory_cache *cac * leaf-entry is returned in the appropriate level variable - pudpp, * pmdpp, ptepp. */ -static bool stage2_get_leaf_entry(struct kvm *kvm, phys_addr_t addr, +static bool stage2_get_leaf_entry(struct kvm_s2_mmu *mmu, phys_addr_t addr, pud_t **pudpp, pmd_t **pmdpp, pte_t **ptepp) { + struct kvm *kvm __maybe_unused = mmu->kvm; pud_t *pudp; pmd_t *pmdp; pte_t *ptep; @@ -1172,7 +1182,7 @@ static bool stage2_get_leaf_entry(struct kvm *kvm, phys_addr_t addr, *pmdpp = NULL; *ptepp = NULL; - pudp = stage2_get_pud(kvm, NULL, addr); + pudp = stage2_get_pud(mmu, NULL, addr); if (!pudp || stage2_pud_none(kvm, *pudp) || !stage2_pud_present(kvm, *pudp)) return false; @@ -1198,14 +1208,14 @@ static bool stage2_get_leaf_entry(struct kvm *kvm, phys_addr_t addr, return true; } -static bool stage2_is_exec(struct kvm *kvm, phys_addr_t addr) +static bool stage2_is_exec(struct kvm_s2_mmu *mmu, phys_addr_t addr) { pud_t *pudp; pmd_t *pmdp; pte_t *ptep; bool found; - found = stage2_get_leaf_entry(kvm, addr, &pudp, &pmdp, &ptep); + found = stage2_get_leaf_entry(mmu, addr, &pudp, &pmdp, &ptep); if (!found) return false; @@ -1217,10 +1227,12 @@ static bool stage2_is_exec(struct kvm *kvm, phys_addr_t addr) return kvm_s2pte_exec(ptep); } -static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, +static int stage2_set_pte(struct kvm_s2_mmu *mmu, + struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pte_t *new_pte, unsigned long flags) { + struct kvm *kvm __maybe_unused = mmu->kvm; pud_t *pud; pmd_t *pmd; pte_t *pte, old_pte; @@ -1230,7 +1242,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, VM_BUG_ON(logging_active && !cache); /* Create stage-2 page table mapping - Levels 0 and 1 */ - pud = stage2_get_pud(kvm, cache, addr); + pud = stage2_get_pud(mmu, cache, addr); if (!pud) { /* * Ignore calls from kvm_set_spte_hva for unallocated @@ -1244,7 +1256,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, * on to allocate page. */ if (logging_active) - stage2_dissolve_pud(kvm, addr, pud); + stage2_dissolve_pud(mmu, addr, pud); if (stage2_pud_none(kvm, *pud)) { if (!cache) @@ -1268,7 +1280,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, * allocate page. */ if (logging_active) - stage2_dissolve_pmd(kvm, addr, pmd); + stage2_dissolve_pmd(mmu, addr, pmd); /* Create stage-2 page mappings - Level 2 */ if (pmd_none(*pmd)) { @@ -1292,7 +1304,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, return 0; kvm_set_pte(pte, __pte(0)); - kvm_tlb_flush_vmid_ipa(kvm, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr); } else { get_page(virt_to_page(pte)); } @@ -1358,8 +1370,8 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, if (ret) goto out; spin_lock(&kvm->mmu_lock); - ret = stage2_set_pte(kvm, &cache, addr, &pte, - KVM_S2PTE_FLAG_IS_IOMAP); + ret = stage2_set_pte(&kvm->arch.mmu, &cache, addr, &pte, + KVM_S2PTE_FLAG_IS_IOMAP); spin_unlock(&kvm->mmu_lock); if (ret) goto out; @@ -1439,9 +1451,10 @@ static void stage2_wp_ptes(pmd_t *pmd, phys_addr_t addr, phys_addr_t end) * @addr: range start address * @end: range end address */ -static void stage2_wp_pmds(struct kvm *kvm, pud_t *pud, +static void stage2_wp_pmds(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addr_t addr, phys_addr_t end) { + struct kvm *kvm = mmu->kvm; pmd_t *pmd; phys_addr_t next; @@ -1461,14 +1474,15 @@ static void stage2_wp_pmds(struct kvm *kvm, pud_t *pud, } /** - * stage2_wp_puds - write protect PGD range - * @pgd: pointer to pgd entry - * @addr: range start address - * @end: range end address - */ -static void stage2_wp_puds(struct kvm *kvm, pgd_t *pgd, + * stage2_wp_puds - write protect PGD range + * @pgd: pointer to pgd entry + * @addr: range start address + * @end: range end address + */ +static void stage2_wp_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addr_t addr, phys_addr_t end) { + struct kvm *kvm __maybe_unused = mmu->kvm; pud_t *pud; phys_addr_t next; @@ -1480,7 +1494,7 @@ static void stage2_wp_puds(struct kvm *kvm, pgd_t *pgd, if (!kvm_s2pud_readonly(pud)) kvm_set_s2pud_readonly(pud); } else { - stage2_wp_pmds(kvm, pud, addr, next); + stage2_wp_pmds(mmu, pud, addr, next); } } } while (pud++, addr = next, addr != end); @@ -1492,12 +1506,13 @@ static void stage2_wp_puds(struct kvm *kvm, pgd_t *pgd, * @addr: Start address of range * @end: End address of range */ -static void stage2_wp_range(struct kvm *kvm, phys_addr_t addr, phys_addr_t end) +static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { + struct kvm *kvm = mmu->kvm; pgd_t *pgd; phys_addr_t next; - pgd = kvm->arch.pgd + stage2_pgd_index(kvm, addr); + pgd = mmu->pgd + stage2_pgd_index(kvm, addr); do { /* * Release kvm_mmu_lock periodically if the memory region is @@ -1509,11 +1524,11 @@ static void stage2_wp_range(struct kvm *kvm, phys_addr_t addr, phys_addr_t end) * the lock. */ cond_resched_lock(&kvm->mmu_lock); - if (!READ_ONCE(kvm->arch.pgd)) + if (!READ_ONCE(mmu->pgd)) break; next = stage2_pgd_addr_end(kvm, addr, end); if (stage2_pgd_present(kvm, *pgd)) - stage2_wp_puds(kvm, pgd, addr, next); + stage2_wp_puds(mmu, pgd, addr, next); } while (pgd++, addr = next, addr != end); } @@ -1538,7 +1553,7 @@ void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - stage2_wp_range(kvm, start, end); + stage2_wp_range(&kvm->arch.mmu, start, end); spin_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs(kvm); } @@ -1562,7 +1577,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; - stage2_wp_range(kvm, start, end); + stage2_wp_range(&kvm->arch.mmu, start, end); } /* @@ -1668,6 +1683,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, pgprot_t mem_type = PAGE_S2; bool logging_active = memslot_is_logging(memslot); unsigned long vma_pagesize, flags = 0; + struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu; write_fault = kvm_is_write_fault(vcpu); exec_fault = kvm_vcpu_trap_is_iabt(vcpu); @@ -1796,7 +1812,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * execute permissions, and we preserve whatever we have. */ needs_exec = exec_fault || - (fault_status == FSC_PERM && stage2_is_exec(kvm, fault_ipa)); + (fault_status == FSC_PERM && stage2_is_exec(mmu, fault_ipa)); if (vma_pagesize == PUD_SIZE) { pud_t new_pud = kvm_pfn_pud(pfn, mem_type); @@ -1808,7 +1824,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (needs_exec) new_pud = kvm_s2pud_mkexec(new_pud); - ret = stage2_set_pud_huge(kvm, memcache, fault_ipa, &new_pud); + ret = stage2_set_pud_huge(mmu, memcache, fault_ipa, &new_pud); } else if (vma_pagesize == PMD_SIZE) { pmd_t new_pmd = kvm_pfn_pmd(pfn, mem_type); @@ -1820,7 +1836,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (needs_exec) new_pmd = kvm_s2pmd_mkexec(new_pmd); - ret = stage2_set_pmd_huge(kvm, memcache, fault_ipa, &new_pmd); + ret = stage2_set_pmd_huge(mmu, memcache, fault_ipa, &new_pmd); } else { pte_t new_pte = kvm_pfn_pte(pfn, mem_type); @@ -1832,7 +1848,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (needs_exec) new_pte = kvm_s2pte_mkexec(new_pte); - ret = stage2_set_pte(kvm, memcache, fault_ipa, &new_pte, flags); + ret = stage2_set_pte(mmu, memcache, fault_ipa, &new_pte, flags); } out_unlock: @@ -1861,7 +1877,7 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) spin_lock(&vcpu->kvm->mmu_lock); - if (!stage2_get_leaf_entry(vcpu->kvm, fault_ipa, &pud, &pmd, &pte)) + if (!stage2_get_leaf_entry(vcpu->arch.hw_mmu, fault_ipa, &pud, &pmd, &pte)) goto out; if (pud) { /* HugeTLB */ @@ -2035,14 +2051,14 @@ static int handle_hva_to_gpa(struct kvm *kvm, static int kvm_unmap_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) { - unmap_stage2_range(kvm, gpa, size); + unmap_stage2_range(&kvm->arch.mmu, gpa, size); return 0; } int kvm_unmap_hva_range(struct kvm *kvm, unsigned long start, unsigned long end) { - if (!kvm->arch.pgd) + if (!kvm->arch.mmu.pgd) return 0; trace_kvm_unmap_hva_range(start, end); @@ -2062,7 +2078,7 @@ static int kvm_set_spte_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data * therefore stage2_set_pte() never needs to clear out a huge PMD * through this calling path. */ - stage2_set_pte(kvm, NULL, gpa, pte, 0); + stage2_set_pte(&kvm->arch.mmu, NULL, gpa, pte, 0); return 0; } @@ -2073,7 +2089,7 @@ int kvm_set_spte_hva(struct kvm *kvm, unsigned long hva, pte_t pte) kvm_pfn_t pfn = pte_pfn(pte); pte_t stage2_pte; - if (!kvm->arch.pgd) + if (!kvm->arch.mmu.pgd) return 0; trace_kvm_set_spte_hva(hva); @@ -2096,7 +2112,7 @@ static int kvm_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) pte_t *pte; WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE); - if (!stage2_get_leaf_entry(kvm, gpa, &pud, &pmd, &pte)) + if (!stage2_get_leaf_entry(&kvm->arch.mmu, gpa, &pud, &pmd, &pte)) return 0; if (pud) @@ -2114,7 +2130,7 @@ static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void * pte_t *pte; WARN_ON(size != PAGE_SIZE && size != PMD_SIZE && size != PUD_SIZE); - if (!stage2_get_leaf_entry(kvm, gpa, &pud, &pmd, &pte)) + if (!stage2_get_leaf_entry(&kvm->arch.mmu, gpa, &pud, &pmd, &pte)) return 0; if (pud) @@ -2127,7 +2143,7 @@ static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void * int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end) { - if (!kvm->arch.pgd) + if (!kvm->arch.mmu.pgd) return 0; trace_kvm_age_hva(start, end); return handle_hva_to_gpa(kvm, start, end, kvm_age_hva_handler, NULL); @@ -2135,7 +2151,7 @@ int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end) int kvm_test_age_hva(struct kvm *kvm, unsigned long hva) { - if (!kvm->arch.pgd) + if (!kvm->arch.mmu.pgd) return 0; trace_kvm_test_age_hva(hva); return handle_hva_to_gpa(kvm, hva, hva + PAGE_SIZE, @@ -2340,9 +2356,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, spin_lock(&kvm->mmu_lock); if (ret) - unmap_stage2_range(kvm, mem->guest_phys_addr, mem->memory_size); + unmap_stage2_range(&kvm->arch.mmu, mem->guest_phys_addr, mem->memory_size); else - stage2_flush_memslot(kvm, memslot); + stage2_flush_memslot(&kvm->arch.mmu, memslot); spin_unlock(&kvm->mmu_lock); out: up_read(¤t->mm->mmap_sem); @@ -2366,7 +2382,7 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) void kvm_arch_flush_shadow_all(struct kvm *kvm) { - kvm_free_stage2_pgd(kvm); + kvm_free_stage2_pgd(&kvm->arch.mmu); } void kvm_arch_flush_shadow_memslot(struct kvm *kvm, @@ -2376,7 +2392,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, phys_addr_t size = slot->npages << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - unmap_stage2_range(kvm, gpa, size); + unmap_stage2_range(&kvm->arch.mmu, gpa, size); spin_unlock(&kvm->mmu_lock); } From patchwork Tue Feb 11 17:48:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376149 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2FF211395 for ; Tue, 11 Feb 2020 17:51:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EF74C206D7 for ; Tue, 11 Feb 2020 17:51:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443471; bh=Z3onEQv/IXIZ013PZdwmQdOHfXMbJRbewu8CHqkz7os=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=fFEZxzL98pr6SzS0PHFQQaR4xTNFhjIWuAqzj+ZcTy5mItEAz4Hj8KO8cF1H5vrYB UAk4J4mmj8VxZHgHBb8KnHT5Bs0tW6jTwIlrf+B8z4ciQg66uWauP6I9iTYKgxOP/K NUzRyia+vEFv1qK9UkJkY4JK3HJyK3l4SlMmRwss= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730165AbgBKRvK (ORCPT ); Tue, 11 Feb 2020 12:51:10 -0500 Received: from mail.kernel.org ([198.145.29.99]:53352 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730085AbgBKRvJ (ORCPT ); Tue, 11 Feb 2020 12:51:09 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A132F20661; Tue, 11 Feb 2020 17:51:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443468; bh=Z3onEQv/IXIZ013PZdwmQdOHfXMbJRbewu8CHqkz7os=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GpBHejrSDUjNoXNcLBM8yMYdUM3B3eFduNz/xHkzG3De0iQxbdcDGA7R8R9GUr0mJ AOwZeHnpohFEN3Vc/aBrdQITAtjS9R2LC/TV1Pd6zGjT5D9E3JvII63Om0/Xxbt7LB IEHgEH6Uc0oHXjoBri7KqEPZ9y3hPZIaJm1ou5pU= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfr-004O7k-1O; Tue, 11 Feb 2020 17:50:15 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 37/94] KVM: arm64: nv: Support multiple nested Stage-2 mmu structures Date: Tue, 11 Feb 2020 17:48:41 +0000 Message-Id: <20200211174938.27809-38-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add Stage-2 mmu data structures for virtual EL2 and for nested guests. We don't yet populate shadow Stage-2 page tables, but we now have a framework for getting to a shadow Stage-2 pgd. We allocate twice the number of vcpus as Stage-2 mmu structures because that's sufficient for each vcpu running two translation regimes without having to flush the Stage-2 page tables. Co-developed-by: Christoffer Dall Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_host.h | 4 + arch/arm/include/asm/kvm_mmu.h | 3 + arch/arm64/include/asm/kvm_host.h | 29 +++++ arch/arm64/include/asm/kvm_mmu.h | 8 ++ arch/arm64/include/asm/kvm_nested.h | 7 ++ arch/arm64/kvm/nested.c | 188 ++++++++++++++++++++++++++++ virt/kvm/arm/arm.c | 16 ++- virt/kvm/arm/mmu.c | 24 ++-- 8 files changed, 268 insertions(+), 11 deletions(-) diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index f43a68d8ea50..0a81c454a540 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -463,4 +463,8 @@ static inline bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu) return true; } +static inline void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) {} +static inline void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) {} +static inline int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) { return 0; } + #endif /* __ARM_KVM_HOST_H__ */ diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 2d49950f8634..8727fde21b8f 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -420,6 +420,9 @@ static inline int hyp_map_aux_data(void) static inline int kvm_set_ipa_limit(void) { return 0; } +static inline void kvm_init_s2_mmu(struct kvm_s2_mmu *mmu) {} +static inline void kvm_init_nested(struct kvm *kvm) {} + static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) { struct kvm_vmid *vmid = &mmu->vmid; diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 42ca60c8b4c8..18e12d00c16d 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -77,11 +77,40 @@ struct kvm_s2_mmu { phys_addr_t pgd_phys; struct kvm *kvm; + + /* + * For a shadow stage-2 MMU, the virtual vttbr programmed by the guest + * hypervisor. Unused for kvm_arch->mmu. Set to 1 when the structure + * contains no valid information. + */ + u64 vttbr; + + /* true when this represents a nested context where virtual HCR_EL2.VM == 1 */ + bool nested_stage2_enabled; + + /* + * 0: Nobody is currently using this, check vttbr for validity + * >0: Somebody is actively using this. + */ + atomic_t refcnt; }; +static inline bool kvm_s2_mmu_valid(struct kvm_s2_mmu *mmu) +{ + return !(mmu->vttbr & 1); +} + struct kvm_arch { struct kvm_s2_mmu mmu; + /* + * Stage 2 paging stage for VMs with nested virtual using a virtual + * VMID. + */ + struct kvm_s2_mmu *nested_mmus; + size_t nested_mmus_size; + int nested_mmus_next; + /* VTCR_EL2 value for this VM */ u64 vtcr; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index ea1a597650f6..868e3386452f 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -89,6 +89,7 @@ alternative_cb_end #include #include #include +#include void kvm_update_va_mask(struct alt_instr *alt, __le32 *origptr, __le32 *updptr, int nr_inst); @@ -154,6 +155,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr); void free_hyp_pgds(void); +void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size); void stage2_unmap_vm(struct kvm *kvm); int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); @@ -625,5 +627,11 @@ static __always_inline void __load_guest_stage2(struct kvm_s2_mmu *mmu) asm(ALTERNATIVE("nop", "isb", ARM64_WORKAROUND_SPECULATIVE_AT_VHE)); } +static inline u64 get_vmid(u64 vttbr) +{ + return (vttbr & VTTBR_VMID_MASK(kvm_get_vmid_bits())) >> + VTTBR_VMID_SHIFT; +} + #endif /* __ASSEMBLY__ */ #endif /* __ARM64_KVM_MMU_H__ */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index c258572b5ed4..88595447b598 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -10,6 +10,13 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features); } +extern void kvm_init_nested(struct kvm *kvm); +extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu); +extern void kvm_init_s2_mmu(struct kvm_s2_mmu *mmu); +extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); +extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); +extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); + int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index c40bf753ead9..65dbc1a796a1 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -19,12 +19,182 @@ #include #include +#include #include +#include #include #include #include "sys_regs.h" +void kvm_init_nested(struct kvm *kvm) +{ + kvm->arch.nested_mmus = NULL; + kvm->arch.nested_mmus_size = 0; +} + +int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + struct kvm_s2_mmu *tmp; + int num_mmus; + int ret = -ENOMEM; + + if (!test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features)) + return 0; + + if (!cpus_have_const_cap(ARM64_HAS_NESTED_VIRT)) + return -EINVAL; + + mutex_lock(&kvm->lock); + + /* + * Let's treat memory allocation failures as benign: If we fail to + * allocate anything, return an error and keep the allocated array + * alive. Userspace may try to recover by intializing the vcpu + * again, and there is no reason to affect the whole VM for this. + */ + num_mmus = atomic_read(&kvm->online_vcpus) * 2; + tmp = krealloc(kvm->arch.nested_mmus, + num_mmus * sizeof(*kvm->arch.nested_mmus), + GFP_KERNEL | __GFP_ZERO); + if (tmp) { + tmp[num_mmus - 1].kvm = kvm; + atomic_set(&tmp[num_mmus - 1].refcnt, 0); + tmp[num_mmus - 2].kvm = kvm; + atomic_set(&tmp[num_mmus - 2].refcnt, 0); + + if (kvm_alloc_stage2_pgd(&tmp[num_mmus - 1]) || + kvm_alloc_stage2_pgd(&tmp[num_mmus - 2])) { + kvm_free_stage2_pgd(&tmp[num_mmus - 1]); + kvm_free_stage2_pgd(&tmp[num_mmus - 2]); + } else { + kvm->arch.nested_mmus_size = num_mmus; + ret = 0; + } + + kvm->arch.nested_mmus = tmp; + } + + mutex_unlock(&kvm->lock); + return ret; +} + +/* Must be called with kvm->lock held */ +struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) +{ + bool nested_stage2_enabled = hcr & HCR_VM; + int i; + + /* Don't consider the CnP bit for the vttbr match */ + vttbr = vttbr & ~VTTBR_CNP_BIT; + + /* + * Two possibilities when looking up a S2 MMU context: + * + * - either S2 is enabled in the guest, and we need a context that + * is S2-enabled and matches the full VTTBR (VMID+BADDR), which + * makes it safe from a TLB conflict perspective (a broken guest + * won't be able to generate them), + * + * - or S2 is disabled, and we need a context that is S2-disabled + * and matches the VMID only, as all TLBs are tagged by VMID even + * if S2 translation is enabled. + */ + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (!kvm_s2_mmu_valid(mmu)) + continue; + + if (nested_stage2_enabled && + mmu->nested_stage2_enabled && + vttbr == mmu->vttbr) + return mmu; + + if (!nested_stage2_enabled && + !mmu->nested_stage2_enabled && + get_vmid(vttbr) == get_vmid(mmu->vttbr)) + return mmu; + } + return NULL; +} + +static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu) +{ + struct kvm *kvm = vcpu->kvm; + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + u64 hcr= vcpu_read_sys_reg(vcpu, HCR_EL2); + struct kvm_s2_mmu *s2_mmu; + int i; + + s2_mmu = lookup_s2_mmu(kvm, vttbr, hcr); + if (s2_mmu) + goto out; + + /* + * Make sure we don't always search from the same point, or we + * will always reuse a potentially active context, leaving + * free contexts unused. + */ + for (i = kvm->arch.nested_mmus_next; + i < (kvm->arch.nested_mmus_size + kvm->arch.nested_mmus_next); + i++) { + s2_mmu = &kvm->arch.nested_mmus[i % kvm->arch.nested_mmus_size]; + + if (atomic_read(&s2_mmu->refcnt) == 0) + break; + } + BUG_ON(atomic_read(&s2_mmu->refcnt)); /* We have struct MMUs to spare */ + + /* Set the scene for the next search */ + kvm->arch.nested_mmus_next = (i + 1) % kvm->arch.nested_mmus_size; + + if (kvm_s2_mmu_valid(s2_mmu)) { + /* Clear the old state */ + kvm_unmap_stage2_range(s2_mmu, 0, kvm_phys_size(kvm)); + if (s2_mmu->vmid.vmid_gen) + kvm_call_hyp(__kvm_tlb_flush_vmid, s2_mmu); + } + + /* + * The virtual VMID (modulo CnP) will be used as a key when matching + * an existing kvm_s2_mmu. + */ + s2_mmu->vttbr = vttbr & ~VTTBR_CNP_BIT; + s2_mmu->nested_stage2_enabled = hcr & HCR_VM; + +out: + atomic_inc(&s2_mmu->refcnt); + return s2_mmu; +} + +void kvm_init_s2_mmu(struct kvm_s2_mmu *mmu) +{ + mmu->vttbr = 1; + mmu->nested_stage2_enabled = false; + atomic_set(&mmu->refcnt, 0); +} + +void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu) +{ + if (is_hyp_ctxt(vcpu)) { + vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu; + } else { + spin_lock(&vcpu->kvm->mmu_lock); + vcpu->arch.hw_mmu = get_s2_mmu_nested(vcpu); + spin_unlock(&vcpu->kvm->mmu_lock); + } +} + +void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) +{ + if (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu) { + atomic_dec(&vcpu->arch.hw_mmu->refcnt); + vcpu->arch.hw_mmu = NULL; + } +} + /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor @@ -43,6 +213,24 @@ int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe) return -EINVAL; } +void kvm_arch_flush_shadow_all(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + WARN_ON(atomic_read(&mmu->refcnt)); + + if (!atomic_read(&mmu->refcnt)) + kvm_free_stage2_pgd(mmu); + } + kfree(kvm->arch.nested_mmus); + kvm->arch.nested_mmus = NULL; + kvm->arch.nested_mmus_size = 0; + kvm_free_stage2_pgd(&kvm->arch.mmu); +} + #define FEATURE(x) (GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT)) /* diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 0df8d281a7c6..dcd8d54d38ce 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -122,6 +123,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) if (ret) goto out_fail_alloc; + kvm_init_nested(kvm); + ret = create_hyp_mappings(kvm, kvm + 1, PAGE_HYP); if (ret) goto out_free_stage2_pgd; @@ -340,6 +343,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) int *last_ran; kvm_host_data_t *cpu_data; + if (nested_virt_in_use(vcpu)) + kvm_vcpu_load_hw_mmu(vcpu); + last_ran = this_cpu_ptr(vcpu->kvm->arch.last_vcpu_ran); cpu_data = this_cpu_ptr(&kvm_host_data); @@ -379,6 +385,9 @@ void kvm_arch_vcpu_put(struct kvm_vcpu *vcpu) kvm_vgic_put(vcpu); kvm_vcpu_pmu_restore_host(vcpu); + if (nested_virt_in_use(vcpu)) + kvm_vcpu_put_hw_mmu(vcpu); + vcpu->cpu = -1; } @@ -958,8 +967,13 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu, vcpu->arch.target = phys_target; + /* Prepare for nested if required */ + ret = kvm_vcpu_init_nested(vcpu); + /* Now we know what it is, we can reset it. */ - ret = kvm_reset_vcpu(vcpu); + if (!ret) + ret = kvm_reset_vcpu(vcpu); + if (ret) { vcpu->arch.target = -1; bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES); diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index ef46686da335..bead2ad59f7d 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -329,7 +329,7 @@ static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, } /** - * unmap_stage2_range -- Clear stage2 page table entries to unmap a range + * kvm_unmap_stage2_range -- Clear stage2 page table entries to unmap a range * @kvm: The VM pointer * @start: The intermediate physical base address of the range to unmap * @size: The size of the area to unmap @@ -339,7 +339,7 @@ static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, * destroying the VM), otherwise another faulting VCPU may come in and mess * with things behind our backs. */ -static void unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) +void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size) { struct kvm *kvm = mmu->kvm; pgd_t *pgd; @@ -928,6 +928,10 @@ int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu) mmu->pgd = pgd; mmu->pgd_phys = pgd_phys; + mmu->vmid.vmid_gen = 0; + + kvm_init_s2_mmu(mmu); + return 0; } @@ -966,7 +970,7 @@ static void stage2_unmap_memslot(struct kvm *kvm, if (!(vma->vm_flags & VM_PFNMAP)) { gpa_t gpa = addr + (vm_start - memslot->userspace_addr); - unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start); + kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, vm_end - vm_start); } hva = vm_end; } while (hva < reg_end); @@ -1005,7 +1009,7 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) spin_lock(&kvm->mmu_lock); if (mmu->pgd) { - unmap_stage2_range(mmu, 0, kvm_phys_size(kvm)); + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(kvm)); pgd = READ_ONCE(mmu->pgd); mmu->pgd = NULL; } @@ -1097,7 +1101,7 @@ static int stage2_set_pmd_huge(struct kvm_s2_mmu *mmu, * get handled accordingly. */ if (!pmd_thp_or_huge(old_pmd)) { - unmap_stage2_range(mmu, addr & S2_PMD_MASK, S2_PMD_SIZE); + kvm_unmap_stage2_range(mmu, addr & S2_PMD_MASK, S2_PMD_SIZE); goto retry; } /* @@ -1149,7 +1153,7 @@ static int stage2_set_pud_huge(struct kvm_s2_mmu *mmu, * the range for this block and retry. */ if (!stage2_pud_huge(kvm, old_pud)) { - unmap_stage2_range(mmu, addr & S2_PUD_MASK, S2_PUD_SIZE); + kvm_unmap_stage2_range(mmu, addr & S2_PUD_MASK, S2_PUD_SIZE); goto retry; } @@ -2051,7 +2055,7 @@ static int handle_hva_to_gpa(struct kvm *kvm, static int kvm_unmap_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) { - unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); return 0; } @@ -2356,7 +2360,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, spin_lock(&kvm->mmu_lock); if (ret) - unmap_stage2_range(&kvm->arch.mmu, mem->guest_phys_addr, mem->memory_size); + kvm_unmap_stage2_range(&kvm->arch.mmu, mem->guest_phys_addr, mem->memory_size); else stage2_flush_memslot(&kvm->arch.mmu, memslot); spin_unlock(&kvm->mmu_lock); @@ -2380,7 +2384,7 @@ void kvm_arch_memslots_updated(struct kvm *kvm, u64 gen) { } -void kvm_arch_flush_shadow_all(struct kvm *kvm) +__weak void kvm_arch_flush_shadow_all(struct kvm *kvm) { kvm_free_stage2_pgd(&kvm->arch.mmu); } @@ -2392,7 +2396,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, phys_addr_t size = slot->npages << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); spin_unlock(&kvm->mmu_lock); } From patchwork Tue Feb 11 17:48:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376223 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E3CE11395 for ; Tue, 11 Feb 2020 17:52:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BB1E820848 for ; Tue, 11 Feb 2020 17:52:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443536; bh=X/wQyZYwzawIOlHX3kBDxTL2CEKWV4RPVgmXqLaUA74=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=fMnNenCeU+pGVbbbvxdLYdbcGzC94zsoGYYyubzgpGFnZgKnxWRuTy604UieluSNe YVIuW7Tyv5rbvkfeNwidV9Aty2iXOW/zRz0Nhg+naFuFYG2CGJYtSgxy+KegGGX1Ww Dp+Y2SKBdRyc8LVEZuLFMuXyl2SgQ221j+Rm5ejc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730351AbgBKRwP (ORCPT ); Tue, 11 Feb 2020 12:52:15 -0500 Received: from mail.kernel.org ([198.145.29.99]:55016 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727966AbgBKRwO (ORCPT ); Tue, 11 Feb 2020 12:52:14 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id CCA9520661; Tue, 11 Feb 2020 17:52:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443533; bh=X/wQyZYwzawIOlHX3kBDxTL2CEKWV4RPVgmXqLaUA74=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=uIaP+L/gZs3nAeeSV2ON4Gc7J6dmz9xt4rCdSoAD6bsTrWq+l/2eLYH+a5/qTZdPV U2OJOvBCby8QyponOUJa0sXu4AjwKH0vVIEODG2EvTKvHHg4dgCGK1EPCF9pLdwMYz r1Y81wloYbxyqjSvSBGa1JM1bx3qrebj8r0i3twM= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfr-004O7k-Le; Tue, 11 Feb 2020 17:50:15 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 38/94] KVM: arm64: nv: Implement nested Stage-2 page table walk logic Date: Tue, 11 Feb 2020 17:48:42 +0000 Message-Id: <20200211174938.27809-39-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Based on the pseudo-code in the ARM ARM, implement a stage 2 software page table walker. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim [maz: heavily reworked for future ARMv8.4-TTL support] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/esr.h | 1 + arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_nested.h | 13 ++ arch/arm64/kvm/nested.c | 276 ++++++++++++++++++++++++++++ 4 files changed, 292 insertions(+) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index b15921dc889a..4373f4f3d58f 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -125,6 +125,7 @@ #define ESR_ELx_CM (UL(1) << ESR_ELx_CM_SHIFT) /* ISS field definitions for exceptions taken in to Hyp */ +#define ESR_ELx_FSC_ADDRSZ (0x00) #define ESR_ELx_CV (UL(1) << 24) #define ESR_ELx_COND_SHIFT (20) #define ESR_ELx_COND_MASK (UL(0xF) << ESR_ELx_COND_SHIFT) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index cdee623ce92f..95dfef820cf1 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -261,6 +261,8 @@ #define VTTBR_VMID_SHIFT (UL(48)) #define VTTBR_VMID_MASK(size) (_AT(u64, (1 << size) - 1) << VTTBR_VMID_SHIFT) +#define SCTLR_EE (UL(1) << 25) + /* Hyp System Trap Register */ #define HSTR_EL2_T(x) (1 << x) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 88595447b598..3881e51d5a2d 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -17,6 +17,19 @@ extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); +struct kvm_s2_trans { + phys_addr_t output; + unsigned long block_size; + bool writable; + bool readable; + int level; + u32 esr; + u64 upper_attr; +}; + +extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result); + int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 65dbc1a796a1..573bcfcfe53f 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -80,6 +80,282 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) return ret; } +struct s2_walk_info { + int (*read_desc)(phys_addr_t pa, u64 *desc, void *data); + void *data; + u64 baddr; + unsigned int max_pa_bits; + unsigned int pgshift; + unsigned int pgsize; + unsigned int ps; + unsigned int sl; + unsigned int t0sz; + bool be; + bool el1_aarch32; +}; + +static unsigned int ps_to_output_size(unsigned int ps) +{ + switch (ps) { + case 0: return 32; + case 1: return 36; + case 2: return 40; + case 3: return 42; + case 4: return 44; + case 5: + default: + return 48; + } +} + +static u32 compute_fsc(int level, u32 fsc) +{ + return fsc | (level & 0x3); +} + +static int esr_s2_fault(struct kvm_vcpu *vcpu, int level, u32 fsc) +{ + u32 esr; + + esr = kvm_vcpu_get_hsr(vcpu) & ~ESR_ELx_FSC; + esr |= compute_fsc(level, fsc); + return esr; +} + +static int check_base_s2_limits(struct s2_walk_info *wi, + int level, int input_size, int stride) +{ + int start_size; + + /* Check translation limits */ + switch (wi->pgsize) { + case SZ_64K: + if (level == 0 || (level == 1 && wi->max_pa_bits <= 42)) + return -EFAULT; + break; + case SZ_16K: + if (level == 0 || (level == 1 && wi->max_pa_bits <= 40)) + return -EFAULT; + break; + case SZ_4K: + if (level < 0 || (level == 0 && wi->max_pa_bits <= 42)) + return -EFAULT; + break; + } + + /* Check input size limits */ + if (input_size > wi->max_pa_bits && + (!wi->el1_aarch32 || input_size > 40)) + return -EFAULT; + + /* Check number of entries in starting level table */ + start_size = input_size - ((3 - level) * stride + wi->pgshift); + if (start_size < 1 || start_size > stride + 4) + return -EFAULT; + + return 0; +} + +/* Check if output is within boundaries */ +static int check_output_size(struct s2_walk_info *wi, phys_addr_t output) +{ + unsigned int output_size = ps_to_output_size(wi->ps); + + if (output_size > wi->max_pa_bits) + output_size = wi->max_pa_bits; + + if (output_size != 48 && (output & GENMASK_ULL(47, output_size))) + return -1; + + return 0; +} + +/* + * This is essentially a C-version of the pseudo code from the ARM ARM + * AArch64.TranslationTableWalk function. I strongly recommend looking at + * that pseudocode in trying to understand this. + * + * Must be called with the kvm->srcu read lock held + */ +static int walk_nested_s2_pgd(phys_addr_t ipa, + struct s2_walk_info *wi, struct kvm_s2_trans *out) +{ + int first_block_level, level, stride, input_size, base_lower_bound; + phys_addr_t base_addr; + unsigned int addr_top, addr_bottom; + u64 desc; /* page table entry */ + int ret; + phys_addr_t paddr; + + switch (wi->pgsize) { + case SZ_64K: + case SZ_16K: + level = 3 - wi->sl; + first_block_level = 2; + break; + case SZ_4K: + level = 2 - wi->sl; + first_block_level = 1; + break; + default: + /* GCC is braindead */ + unreachable(); + } + + stride = wi->pgshift - 3; + input_size = 64 - wi->t0sz; + if (input_size > 48 || input_size < 25) + return -EFAULT; + + ret = check_base_s2_limits(wi, level, input_size, stride); + if (WARN_ON(ret)) + return ret; + + base_lower_bound = 3 + input_size - ((3 - level) * stride + + wi->pgshift); + base_addr = wi->baddr & GENMASK_ULL(47, base_lower_bound); + + if (check_output_size(wi, base_addr)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + return 1; + } + + addr_top = input_size - 1; + + while (1) { + phys_addr_t index; + + addr_bottom = (3 - level) * stride + wi->pgshift; + index = (ipa & GENMASK_ULL(addr_top, addr_bottom)) + >> (addr_bottom - 3); + + paddr = base_addr | index; + ret = wi->read_desc(paddr, &desc, wi->data); + if (ret < 0) + return ret; + + /* + * Handle reversedescriptors if endianness differs between the + * host and the guest hypervisor. + */ + if (wi->be) + desc = be64_to_cpu(desc); + else + desc = le64_to_cpu(desc); + + /* Check for valid descriptor at this point */ + if (!(desc & 1) || ((desc & 3) == 1 && level == 3)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT); + out->upper_attr = desc; + return 1; + } + + /* We're at the final level or block translation level */ + if ((desc & 3) == 1 || level == 3) + break; + + if (check_output_size(wi, desc)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + out->upper_attr = desc; + return 1; + } + + base_addr = desc & GENMASK_ULL(47, wi->pgshift); + + level += 1; + addr_top = addr_bottom - 1; + } + + if (level < first_block_level) { + out->esr = compute_fsc(level, ESR_ELx_FSC_FAULT); + out->upper_attr = desc; + return 1; + } + + /* + * We don't use the contiguous bit in the stage-2 ptes, so skip check + * for misprogramming of the contiguous bit. + */ + + if (check_output_size(wi, desc)) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ADDRSZ); + out->upper_attr = desc; + return 1; + } + + if (!(desc & BIT(10))) { + out->esr = compute_fsc(level, ESR_ELx_FSC_ACCESS); + out->upper_attr = desc; + return 1; + } + + /* Calculate and return the result */ + paddr = (desc & GENMASK_ULL(47, addr_bottom)) | + (ipa & GENMASK_ULL(addr_bottom - 1, 0)); + out->output = paddr; + out->block_size = 1UL << ((3 - level) * stride + wi->pgshift); + out->readable = desc & (0b01 << 6); + out->writable = desc & (0b10 << 6); + out->level = level; + out->upper_attr = desc & GENMASK_ULL(63, 52); + return 0; +} + +static int read_guest_s2_desc(phys_addr_t pa, u64 *desc, void *data) +{ + struct kvm_vcpu *vcpu = data; + + return kvm_read_guest(vcpu->kvm, pa, desc, sizeof(*desc)); +} + +static void vtcr_to_walk_info(u64 vtcr, struct s2_walk_info *wi) +{ + wi->t0sz = vtcr & TCR_EL2_T0SZ_MASK; + + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + wi->pgshift = 12; break; + case VTCR_EL2_TG0_16K: + wi->pgshift = 14; break; + case VTCR_EL2_TG0_64K: + default: + wi->pgshift = 16; break; + } + + wi->pgsize = 1UL << wi->pgshift; + wi->ps = (vtcr & VTCR_EL2_PS_MASK) >> VTCR_EL2_PS_SHIFT; + wi->sl = (vtcr & VTCR_EL2_SL0_MASK) >> VTCR_EL2_SL0_SHIFT; + wi->max_pa_bits = VTCR_EL2_IPA(vtcr); +} + +int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct s2_walk_info wi; + int ret; + + result->esr = 0; + + if (!nested_virt_in_use(vcpu)) + return 0; + + wi.read_desc = read_guest_s2_desc; + wi.data = vcpu; + wi.baddr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + + vtcr_to_walk_info(vtcr, &wi); + + wi.be = vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_EE; + wi.el1_aarch32 = vcpu_mode_is_32bit(vcpu); + + ret = walk_nested_s2_pgd(gipa, &wi, result); + if (ret) + result->esr |= (kvm_vcpu_get_hsr(vcpu) & ~ESR_ELx_FSC); + + return ret; +} + /* Must be called with kvm->lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { From patchwork Tue Feb 11 17:48:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376153 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C16114E3 for ; Tue, 11 Feb 2020 17:51:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 489752086A for ; Tue, 11 Feb 2020 17:51:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443475; bh=nlXhKKJ34Rz7b0lfO28NB1NGDZ8MDge9YqsgbwjttLM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=nm93YR+KldED1GSKJO+6fLcveNds6v/jJtNZ4hmdA1CnkwaNh1ou7r3zM6aVhs12y fYDqODwniFkGAP2/eYgzKJiY4Z0fB4DW4Ati5Wf/Z9YA9p6102uPTduKxgcxstFQ0f e92jluzsL1x+u3P56YiCbEU2YDlKcq93BLTwX3rw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730248AbgBKRvO (ORCPT ); Tue, 11 Feb 2020 12:51:14 -0500 Received: from mail.kernel.org ([198.145.29.99]:53438 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728935AbgBKRvO (ORCPT ); Tue, 11 Feb 2020 12:51:14 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BACA1206CC; Tue, 11 Feb 2020 17:51:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443473; bh=nlXhKKJ34Rz7b0lfO28NB1NGDZ8MDge9YqsgbwjttLM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=lMQDqkXyylqWVJB3WXIc9QU8MjkfA07meh0h1+JUfmTjizDwwpScrQC5EUy6N/mLr aF0jlkq5r5qrwOXmAKtR9Zo/rc6sOZGpAB1qnbUGUH20d9xAcIsnugV/br3lvQG3vk msALfpZzsDck2/20OAiLBtvekhLNJZC3HE9klI1g= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfs-004O7k-9H; Tue, 11 Feb 2020 17:50:16 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 39/94] KVM: arm64: nv: Handle shadow stage 2 page faults Date: Tue, 11 Feb 2020 17:48:43 +0000 Message-Id: <20200211174938.27809-40-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall If we are faulting on a shadow stage 2 translation, we first walk the guest hypervisor's stage 2 page table to see if it has a mapping. If not, we inject a stage 2 page fault to the virtual EL2. Otherwise, we create a mapping in the shadow stage 2 page table. Note that we have to deal with two IPAs when we got a shadow stage 2 page fault. One is the address we faulted on, and is in the L2 guest phys space. The other is from the guest stage-2 page table walk, and is in the L1 guest phys space. To differentiate them, we rename variables so that fault_ipa is used for the former and ipa is used for the latter. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_mmu.h | 52 +++++++++++++++ arch/arm64/include/asm/kvm_emulate.h | 6 ++ arch/arm64/include/asm/kvm_nested.h | 18 ++++++ arch/arm64/kvm/nested.c | 39 ++++++++++++ virt/kvm/arm/mmio.c | 14 ++--- virt/kvm/arm/mmu.c | 94 ++++++++++++++++++++++------ 6 files changed, 197 insertions(+), 26 deletions(-) diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 8727fde21b8f..7f1fb496f435 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -423,6 +423,58 @@ static inline int kvm_set_ipa_limit(void) { return 0; } static inline void kvm_init_s2_mmu(struct kvm_s2_mmu *mmu) {} static inline void kvm_init_nested(struct kvm *kvm) {} +struct kvm_s2_trans {}; +static inline phys_addr_t kvm_s2_trans_output(struct kvm_s2_trans *trans) +{ + BUG(); +} + +static inline unsigned long kvm_s2_trans_size(struct kvm_s2_trans *trans) +{ + BUG(); +} + +static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans) +{ + BUG(); +} + +static inline int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t ipa, + struct kvm_s2_trans *trans) +{ + BUG(); +} + +static inline int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, + struct kvm_s2_trans *trans) +{ + BUG(); +} + +static inline void kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u32 esr) +{ + BUG(); +} + +static inline bool kvm_s2_trans_readable(struct kvm_s2_trans *trans) +{ + BUG(); +} + +static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans) +{ + BUG(); +} + +static inline void kvm_nested_s2_flush(struct kvm *kvm) {} +static inline void kvm_nested_s2_wp(struct kvm *kvm) {} +static inline void kvm_nested_s2_clear(struct kvm *kvm) {} + +static inline bool kvm_is_shadow_s2_fault(struct kvm_vcpu *vcpu) +{ + return false; +} + static __always_inline u64 kvm_get_vttbr(struct kvm_s2_mmu *mmu) { struct kvm_vmid *vmid = &mmu->vmid; diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 0e5f88060ecc..1ccd98b5fead 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -684,4 +684,10 @@ static inline void __hyp_text __kvm_skip_instr(struct kvm_vcpu *vcpu) write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR); } +static inline bool kvm_is_shadow_s2_fault(struct kvm_vcpu *vcpu) +{ + return (vcpu->arch.hw_mmu != &vcpu->kvm->arch.mmu && + vcpu->arch.hw_mmu->nested_stage2_enabled); +} + #endif /* __ARM64_KVM_EMULATE_H__ */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 3881e51d5a2d..b2bf82461fa6 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -27,9 +27,27 @@ struct kvm_s2_trans { u64 upper_attr; }; +static inline phys_addr_t kvm_s2_trans_output(struct kvm_s2_trans *trans) +{ + return trans->output; +} + +static inline unsigned long kvm_s2_trans_size(struct kvm_s2_trans *trans) +{ + return trans->block_size; +} + +static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans) +{ + return trans->esr; +} + extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result); +extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, + struct kvm_s2_trans *trans); +extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 573bcfcfe53f..361c3eb2032c 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -471,6 +471,45 @@ void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu) } } +/* + * Returns non-zero if permission fault is handled by injecting it to the next + * level hypervisor. + */ +int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans) +{ + unsigned long fault_status = kvm_vcpu_trap_get_fault_type(vcpu); + bool forward_fault = false; + + trans->esr = 0; + + if (fault_status != FSC_PERM) + return 0; + + if (kvm_vcpu_trap_is_iabt(vcpu)) { + forward_fault = (trans->upper_attr & PTE_S2_XN); + } else { + bool write_fault = kvm_is_write_fault(vcpu); + + forward_fault = ((write_fault && !trans->writable) || + (!write_fault && !trans->readable)); + } + + if (forward_fault) { + trans->esr = esr_s2_fault(vcpu, trans->level, ESR_ELx_FSC_PERM); + return 1; + } + + return 0; +} + +int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) +{ + vcpu_write_sys_reg(vcpu, vcpu->arch.fault.far_el2, FAR_EL2); + vcpu_write_sys_reg(vcpu, vcpu->arch.fault.hpfar_el2, HPFAR_EL2); + + return kvm_inject_nested_sync(vcpu, esr_el2); +} + /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor diff --git a/virt/kvm/arm/mmio.c b/virt/kvm/arm/mmio.c index aedfcff99ac5..7ac7b1eac1ac 100644 --- a/virt/kvm/arm/mmio.c +++ b/virt/kvm/arm/mmio.c @@ -120,7 +120,7 @@ int kvm_handle_mmio_return(struct kvm_vcpu *vcpu, struct kvm_run *run) } int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, - phys_addr_t fault_ipa) + phys_addr_t ipa) { unsigned long data; unsigned long rt; @@ -137,7 +137,7 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, if (vcpu->kvm->arch.return_nisv_io_abort_to_user) { run->exit_reason = KVM_EXIT_ARM_NISV; run->arm_nisv.esr_iss = kvm_vcpu_dabt_iss_nisv_sanitized(vcpu); - run->arm_nisv.fault_ipa = fault_ipa; + run->arm_nisv.fault_ipa = ipa; return 0; } @@ -164,22 +164,22 @@ int io_mem_abort(struct kvm_vcpu *vcpu, struct kvm_run *run, data = vcpu_data_guest_to_host(vcpu, vcpu_get_reg(vcpu, rt), len); - trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, len, fault_ipa, &data); + trace_kvm_mmio(KVM_TRACE_MMIO_WRITE, len, ipa, &data); kvm_mmio_write_buf(data_buf, len, data); - ret = kvm_io_bus_write(vcpu, KVM_MMIO_BUS, fault_ipa, len, + ret = kvm_io_bus_write(vcpu, KVM_MMIO_BUS, ipa, len, data_buf); } else { trace_kvm_mmio(KVM_TRACE_MMIO_READ_UNSATISFIED, len, - fault_ipa, NULL); + ipa, NULL); - ret = kvm_io_bus_read(vcpu, KVM_MMIO_BUS, fault_ipa, len, + ret = kvm_io_bus_read(vcpu, KVM_MMIO_BUS, ipa, len, data_buf); } /* Now prepare kvm_run for the potential return to userland. */ run->mmio.is_write = is_write; - run->mmio.phys_addr = fault_ipa; + run->mmio.phys_addr = ipa; run->mmio.len = len; vcpu->mmio_needed = 1; diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index bead2ad59f7d..c6db597b925c 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1388,7 +1388,8 @@ int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, return ret; } -static bool transparent_hugepage_adjust(kvm_pfn_t *pfnp, phys_addr_t *ipap) +static bool transparent_hugepage_adjust(kvm_pfn_t *pfnp, phys_addr_t *ipap, + phys_addr_t *fault_ipap) { kvm_pfn_t pfn = *pfnp; gfn_t gfn = *ipap >> PAGE_SHIFT; @@ -1416,6 +1417,7 @@ static bool transparent_hugepage_adjust(kvm_pfn_t *pfnp, phys_addr_t *ipap) mask = PTRS_PER_PMD - 1; VM_BUG_ON((gfn & mask) != (pfn & mask)); if (pfn & mask) { + *fault_ipap &= PMD_MASK; *ipap &= PMD_MASK; kvm_release_pfn_clean(pfn); pfn &= ~mask; @@ -1671,14 +1673,16 @@ static bool fault_supports_stage2_huge_mapping(struct kvm_memory_slot *memslot, } static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, - struct kvm_memory_slot *memslot, unsigned long hva, - unsigned long fault_status) + struct kvm_s2_trans *nested, + struct kvm_memory_slot *memslot, + unsigned long hva, unsigned long fault_status) { int ret; - bool write_fault, writable, force_pte = false; + bool write_fault, writable; bool exec_fault, needs_exec; unsigned long mmu_seq; - gfn_t gfn = fault_ipa >> PAGE_SHIFT; + phys_addr_t ipa = fault_ipa; + gfn_t gfn; struct kvm *kvm = vcpu->kvm; struct kvm_mmu_memory_cache *memcache = &vcpu->arch.mmu_page_cache; struct vm_area_struct *vma; @@ -1688,6 +1692,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, bool logging_active = memslot_is_logging(memslot); unsigned long vma_pagesize, flags = 0; struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu; + unsigned long max_map_size = PUD_SIZE; write_fault = kvm_is_write_fault(vcpu); exec_fault = kvm_vcpu_trap_is_iabt(vcpu); @@ -1715,10 +1720,22 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, vma_pagesize = 1ULL << vma_shift; if (logging_active || (vma->vm_flags & VM_PFNMAP) || - !fault_supports_stage2_huge_mapping(memslot, hva, vma_pagesize)) { - force_pte = true; - vma_pagesize = PAGE_SIZE; + !fault_supports_stage2_huge_mapping(memslot, hva, vma_pagesize)) + max_map_size = PAGE_SIZE; + + if (kvm_is_shadow_s2_fault(vcpu)) { + ipa = kvm_s2_trans_output(nested); + + /* + * If we're about to create a shadow stage 2 entry, then we + * can only create a block mapping if the guest stage 2 page + * table uses at least as big a mapping. + */ + max_map_size = min(kvm_s2_trans_size(nested), max_map_size); } + gfn = ipa >> PAGE_SHIFT; + + vma_pagesize = min(vma_pagesize, max_map_size); /* * The stage2 has a minimum of 2 level table (For arm64 see @@ -1728,8 +1745,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * 3 levels, i.e, PMD is not folded. */ if (vma_pagesize == PMD_SIZE || - (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm))) - gfn = (fault_ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT; + (vma_pagesize == PUD_SIZE && kvm_stage2_has_pmd(kvm))) { + gfn = (ipa & huge_page_mask(hstate_vma(vma))) >> PAGE_SHIFT; + } up_read(¤t->mm->mmap_sem); /* We need minimum second+third level pages */ @@ -1784,7 +1802,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (mmu_notifier_retry(kvm, mmu_seq)) goto out_unlock; - if (vma_pagesize == PAGE_SIZE && !force_pte) { + if (vma_pagesize == PAGE_SIZE && max_map_size >= PMD_SIZE) { /* * Only PMD_SIZE transparent hugepages(THP) are * currently supported. This code will need to be @@ -1794,7 +1812,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * aligned and that the block is contained within the memslot. */ if (fault_supports_stage2_huge_mapping(memslot, hva, PMD_SIZE) && - transparent_hugepage_adjust(&pfn, &fault_ipa)) + transparent_hugepage_adjust(&pfn, &ipa, &fault_ipa)) vma_pagesize = PMD_SIZE; } @@ -1919,8 +1937,10 @@ static void handle_access_fault(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa) int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) { unsigned long fault_status; - phys_addr_t fault_ipa; + phys_addr_t fault_ipa; /* The address we faulted on */ + phys_addr_t ipa; /* Always the IPA in the L1 guest phys space */ struct kvm_memory_slot *memslot; + struct kvm_s2_trans nested_trans; unsigned long hva; bool is_iabt, write_fault, writable; gfn_t gfn; @@ -1928,7 +1948,7 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) fault_status = kvm_vcpu_trap_get_fault_type(vcpu); - fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); + ipa = fault_ipa = kvm_vcpu_get_fault_ipa(vcpu); is_iabt = kvm_vcpu_trap_is_iabt(vcpu); /* Synchronous External Abort? */ @@ -1952,6 +1972,12 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) /* Check the stage-2 fault is trans. fault or write fault */ if (fault_status != FSC_FAULT && fault_status != FSC_PERM && fault_status != FSC_ACCESS) { + /* + * We must never see an address size fault on shadow stage 2 + * page table walk, because we would have injected an addr + * size fault when we walked the nested s2 page and not + * create the shadow entry. + */ kvm_err("Unsupported FSC: EC=%#x xFSC=%#lx ESR_EL2=%#lx\n", kvm_vcpu_trap_get_class(vcpu), (unsigned long)kvm_vcpu_trap_get_fault(vcpu), @@ -1961,7 +1987,36 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) idx = srcu_read_lock(&vcpu->kvm->srcu); - gfn = fault_ipa >> PAGE_SHIFT; + /* + * We may have faulted on a shadow stage 2 page table if we are + * running a nested guest. In this case, we have to resolve the L2 + * IPA to the L1 IPA first, before knowing what kind of memory should + * back the L1 IPA. + * + * If the shadow stage 2 page table walk faults, then we simply inject + * this to the guest and carry on. + */ + if (kvm_is_shadow_s2_fault(vcpu)) { + u32 esr; + + ret = kvm_walk_nested_s2(vcpu, fault_ipa, &nested_trans); + esr = kvm_s2_trans_esr(&nested_trans); + if (esr) + kvm_inject_s2_fault(vcpu, esr); + if (ret) + goto out_unlock; + + ret = kvm_s2_handle_perm_fault(vcpu, &nested_trans); + esr = kvm_s2_trans_esr(&nested_trans); + if (esr) + kvm_inject_s2_fault(vcpu, esr); + if (ret) + goto out_unlock; + + ipa = kvm_s2_trans_output(&nested_trans); + } + + gfn = ipa >> PAGE_SHIFT; memslot = gfn_to_memslot(vcpu->kvm, gfn); hva = gfn_to_hva_memslot_prot(memslot, gfn, &writable); write_fault = kvm_is_write_fault(vcpu); @@ -1994,13 +2049,13 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) * faulting VA. This is always 12 bits, irrespective * of the page size. */ - fault_ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); - ret = io_mem_abort(vcpu, run, fault_ipa); + ipa |= kvm_vcpu_get_hfar(vcpu) & ((1 << 12) - 1); + ret = io_mem_abort(vcpu, run, ipa); goto out_unlock; } /* Userspace should not be able to register out-of-bounds IPAs */ - VM_BUG_ON(fault_ipa >= kvm_phys_size(vcpu->kvm)); + VM_BUG_ON(ipa >= kvm_phys_size(vcpu->kvm)); if (fault_status == FSC_ACCESS) { handle_access_fault(vcpu, fault_ipa); @@ -2008,7 +2063,8 @@ int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run) goto out_unlock; } - ret = user_mem_abort(vcpu, fault_ipa, memslot, hva, fault_status); + ret = user_mem_abort(vcpu, fault_ipa, &nested_trans, + memslot, hva, fault_status); if (ret == 0) ret = 1; out: From patchwork Tue Feb 11 17:48:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376157 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CCD361395 for ; Tue, 11 Feb 2020 17:51:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A2A0F208C3 for ; Tue, 11 Feb 2020 17:51:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443479; bh=FZe+9NMjAl8Y/akILNFyMXdbM0h07wMHLLUthV7+Lks=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=lv0ZCBq0keTHxSuEuCTXT/64k0+lz4f+30oO0qyQKXQnUFaK4umWx/ACHpEGGPQmR yIAV7ZCBbKGr8mUvYHjMLqp71oGnz+6ojBz+gg9l3rtjQPMFvSKOE6vUgW5BsLERrr 0LvEXgV84ImxZFcrKqYz5I7rIdeexBPIO8keTPOc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730327AbgBKRvS (ORCPT ); Tue, 11 Feb 2020 12:51:18 -0500 Received: from mail.kernel.org ([198.145.29.99]:53522 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730316AbgBKRvS (ORCPT ); Tue, 11 Feb 2020 12:51:18 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B3D2120661; Tue, 11 Feb 2020 17:51:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443476; bh=FZe+9NMjAl8Y/akILNFyMXdbM0h07wMHLLUthV7+Lks=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kKmW+N/fVhbrnplOWXR+Yzu8VjuHuhGfCRNBryqjOjlACIbsStWTfR3r2e5HEP3Yk 8d1MJyXbU+phaFoKXozf16fa3/vHEGn/5/JZh0QAyQq/uaEGjefz+yY9ubHZG1c914 0YO5ndjhAsRjhqkfvbkPwj0OhHZgJPsxP2J+qrvg= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfs-004O7k-TE; Tue, 11 Feb 2020 17:50:16 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 40/94] KVM: arm64: nv: Unmap/flush shadow stage 2 page tables Date: Tue, 11 Feb 2020 17:48:44 +0000 Message-Id: <20200211174938.27809-41-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Unmap/flush shadow stage 2 page tables for the nested VMs as well as the stage 2 page table for the guest hypervisor. Note: A bunch of the code in mmu.c relating to MMU notifiers is currently dealt with in an extremely abrupt way, for example by clearing out an entire shadow stage-2 table. This will be handled in a more efficient way using the reverse mapping feature in a later version of the patch series. Signed-off-by: Christoffer Dall Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_mmu.h | 3 ++ arch/arm64/include/asm/kvm_nested.h | 3 ++ arch/arm64/kvm/nested.c | 39 +++++++++++++++++++++++++ virt/kvm/arm/mmu.c | 44 +++++++++++++++++++++++------ 4 files changed, 81 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 868e3386452f..41ff6aa728e3 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -153,6 +153,8 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size, void __iomem **haddr); int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void **haddr); +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end); void free_hyp_pgds(void); void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size); @@ -161,6 +163,7 @@ int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); +void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end); int kvm_handle_guest_abort(struct kvm_vcpu *vcpu, struct kvm_run *run); diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index b2bf82461fa6..20c7c18e99ac 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -48,6 +48,9 @@ extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, extern int kvm_s2_handle_perm_fault(struct kvm_vcpu *vcpu, struct kvm_s2_trans *trans); extern int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2); +extern void kvm_nested_s2_wp(struct kvm *kvm); +extern void kvm_nested_s2_clear(struct kvm *kvm); +extern void kvm_nested_s2_flush(struct kvm *kvm); int handle_wfx_nested(struct kvm_vcpu *vcpu, bool is_wfe); extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 361c3eb2032c..d7b2e17b54c3 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -510,6 +510,45 @@ int kvm_inject_s2_fault(struct kvm_vcpu *vcpu, u64 esr_el2) return kvm_inject_nested_sync(vcpu, esr_el2); } +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_wp(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_stage2_wp_range(mmu, 0, kvm_phys_size(kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_clear(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(kvm)); + } +} + +/* expects kvm->mmu_lock to be held */ +void kvm_nested_s2_flush(struct kvm *kvm) +{ + int i; + + for (i = 0; i < kvm->arch.nested_mmus_size; i++) { + struct kvm_s2_mmu *mmu = &kvm->arch.nested_mmus[i]; + + if (kvm_s2_mmu_valid(mmu)) + kvm_stage2_flush_range(mmu, 0, kvm_phys_size(kvm)); + } +} + /* * Inject wfx to the virtual EL2 if this is not from the virtual EL2 and * the virtual HCR_EL2.TWX is set. Otherwise, let the host hypervisor diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index c6db597b925c..054946566dc0 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -420,12 +420,10 @@ static void stage2_flush_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, } while (pud++, addr = next, addr != end); } -static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, - struct kvm_memory_slot *memslot) +void kvm_stage2_flush_range(struct kvm_s2_mmu *mmu, + phys_addr_t addr, phys_addr_t end) { struct kvm *kvm = mmu->kvm; - phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT; - phys_addr_t end = addr + PAGE_SIZE * memslot->npages; phys_addr_t next; pgd_t *pgd; @@ -437,6 +435,15 @@ static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, } while (pgd++, addr = next, addr != end); } +static void stage2_flush_memslot(struct kvm_s2_mmu *mmu, + struct kvm_memory_slot *memslot) +{ + phys_addr_t addr = memslot->base_gfn << PAGE_SHIFT; + phys_addr_t end = addr + PAGE_SIZE * memslot->npages; + + kvm_stage2_flush_range(mmu, addr, end); +} + /** * stage2_flush_vm - Invalidate cache for pages mapped in stage 2 * @kvm: The struct kvm pointer @@ -457,6 +464,8 @@ static void stage2_flush_vm(struct kvm *kvm) kvm_for_each_memslot(memslot, slots) stage2_flush_memslot(&kvm->arch.mmu, memslot); + kvm_nested_s2_flush(kvm); + spin_unlock(&kvm->mmu_lock); srcu_read_unlock(&kvm->srcu, idx); } @@ -997,6 +1006,8 @@ void stage2_unmap_vm(struct kvm *kvm) kvm_for_each_memslot(memslot, slots) stage2_unmap_memslot(kvm, memslot); + kvm_nested_s2_clear(kvm); + spin_unlock(&kvm->mmu_lock); up_read(¤t->mm->mmap_sem); srcu_read_unlock(&kvm->srcu, idx); @@ -1507,12 +1518,12 @@ static void stage2_wp_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, } /** - * stage2_wp_range() - write protect stage2 memory region range + * kvm_stage2_wp_range() - write protect stage2 memory region range * @kvm: The KVM pointer * @addr: Start address of range * @end: End address of range */ -static void stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) +void kvm_stage2_wp_range(struct kvm_s2_mmu *mmu, phys_addr_t addr, phys_addr_t end) { struct kvm *kvm = mmu->kvm; pgd_t *pgd; @@ -1559,7 +1570,8 @@ void kvm_mmu_wp_memory_region(struct kvm *kvm, int slot) phys_addr_t end = (memslot->base_gfn + memslot->npages) << PAGE_SHIFT; spin_lock(&kvm->mmu_lock); - stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_nested_s2_wp(kvm); spin_unlock(&kvm->mmu_lock); kvm_flush_remote_tlbs(kvm); } @@ -1583,7 +1595,7 @@ static void kvm_mmu_write_protect_pt_masked(struct kvm *kvm, phys_addr_t start = (base_gfn + __ffs(mask)) << PAGE_SHIFT; phys_addr_t end = (base_gfn + __fls(mask) + 1) << PAGE_SHIFT; - stage2_wp_range(&kvm->arch.mmu, start, end); + kvm_stage2_wp_range(&kvm->arch.mmu, start, end); } /* @@ -1598,6 +1610,7 @@ void kvm_arch_mmu_enable_log_dirty_pt_masked(struct kvm *kvm, gfn_t gfn_offset, unsigned long mask) { kvm_mmu_write_protect_pt_masked(kvm, slot, gfn_offset, mask); + kvm_nested_s2_wp(kvm); } static void clean_dcache_guest_page(kvm_pfn_t pfn, unsigned long size) @@ -2112,6 +2125,7 @@ static int handle_hva_to_gpa(struct kvm *kvm, static int kvm_unmap_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) { kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_nested_s2_clear(kvm); return 0; } @@ -2139,6 +2153,7 @@ static int kvm_set_spte_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data * through this calling path. */ stage2_set_pte(&kvm->arch.mmu, NULL, gpa, pte, 0); + kvm_nested_s2_clear(kvm); return 0; } @@ -2181,6 +2196,13 @@ static int kvm_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) return stage2_pmdp_test_and_clear_young(pmd); else return stage2_ptep_test_and_clear_young(pte); + + /* + * TODO: Handle nested_mmu structures here using the reverse mapping in + * a later version of patch series. + */ + + return stage2_ptep_test_and_clear_young(pte); } static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void *data) @@ -2199,6 +2221,11 @@ static int kvm_test_age_hva_handler(struct kvm *kvm, gpa_t gpa, u64 size, void * return pmd_young(*pmd); else return pte_young(*pte); + + /* + * TODO: Handle nested_mmu structures here using the reverse mapping in + * a later version of patch series. + */ } int kvm_age_hva(struct kvm *kvm, unsigned long start, unsigned long end) @@ -2453,6 +2480,7 @@ void kvm_arch_flush_shadow_memslot(struct kvm *kvm, spin_lock(&kvm->mmu_lock); kvm_unmap_stage2_range(&kvm->arch.mmu, gpa, size); + kvm_nested_s2_clear(kvm); spin_unlock(&kvm->mmu_lock); } From patchwork Tue Feb 11 17:48:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376317 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 20C361395 for ; Tue, 11 Feb 2020 17:53:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E177A206CC for ; Tue, 11 Feb 2020 17:53:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443621; bh=sMIPQs20M5KCorsTrh42SFSdn+qqoWLMSQtP8k/tiuA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=oUR9c3mWwTnmt1ttEEkXn5FV39Ov8qFKSTrhxuxWRJSUwZ4enDKI0NRSteMfORgBX owHfkwKFsLa6LWN6M9qXaj790f58QTciRK09NkEA64HbzONQqIdjC9WcLRDYmHT7hk PENfuVybzLmuR5mFlEcxZOqPiXaPY+1pXAT+cROs= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730788AbgBKRxk (ORCPT ); Tue, 11 Feb 2020 12:53:40 -0500 Received: from mail.kernel.org ([198.145.29.99]:57400 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730743AbgBKRxj (ORCPT ); Tue, 11 Feb 2020 12:53:39 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E7AA120661; Tue, 11 Feb 2020 17:53:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443618; bh=sMIPQs20M5KCorsTrh42SFSdn+qqoWLMSQtP8k/tiuA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Yr1DXlJe2kSxTYkM3iqIKOSXkUxkJK8Ul4bfmnhhfjFQUbMGYcKhi03cmWE1h4ewt yafwbdTxvJOLHn8ca6s/YzGI8uq68Tn+t1bun5a/JT1sTqUhY1yf+2YMvOSoF27/1k qvlEhAi1aoMm/0IHeBcmYhJ3wsZBYfoJ+y5LHsWk= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zft-004O7k-E0; Tue, 11 Feb 2020 17:50:17 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 41/94] KVM: arm64: nv: Move last_vcpu_ran to be per s2 mmu Date: Tue, 11 Feb 2020 17:48:45 +0000 Message-Id: <20200211174938.27809-42-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org last_vcpu_ran has to be per s2 mmu now that we can have multiple S2 per VM. Let's take this opportunity to perform some cleanup. Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_host.h | 6 +++--- arch/arm/include/asm/kvm_mmu.h | 4 ++-- arch/arm64/include/asm/kvm_host.h | 6 +++--- arch/arm64/include/asm/kvm_mmu.h | 2 +- arch/arm64/include/asm/kvm_nested.h | 2 +- arch/arm64/kvm/nested.c | 11 +++-------- virt/kvm/arm/arm.c | 25 ++++--------------------- virt/kvm/arm/mmu.c | 28 +++++++++++++++++++++------- 8 files changed, 38 insertions(+), 46 deletions(-) diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index 0a81c454a540..cb5e9b37a87a 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -63,15 +63,15 @@ struct kvm_s2_mmu { pgd_t *pgd; phys_addr_t pgd_phys; + /* The last vcpu id that ran on each physical CPU */ + int __percpu *last_vcpu_ran; + struct kvm *kvm; }; struct kvm_arch { struct kvm_s2_mmu mmu; - /* The last vcpu id that ran on each physical CPU */ - int __percpu *last_vcpu_ran; - /* Stage-2 page table */ pgd_t *pgd; phys_addr_t pgd_phys; diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 7f1fb496f435..1d0d5f00f0af 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -52,7 +52,7 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, void free_hyp_pgds(void); void stage2_unmap_vm(struct kvm *kvm); -int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu); +int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); @@ -420,7 +420,7 @@ static inline int hyp_map_aux_data(void) static inline int kvm_set_ipa_limit(void) { return 0; } -static inline void kvm_init_s2_mmu(struct kvm_s2_mmu *mmu) {} +static inline void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu) {} static inline void kvm_init_nested(struct kvm *kvm) {} struct kvm_s2_trans {}; diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 18e12d00c16d..36bb463dc16a 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -93,6 +93,9 @@ struct kvm_s2_mmu { * >0: Somebody is actively using this. */ atomic_t refcnt; + + /* The last vcpu id that ran on each physical CPU */ + int __percpu *last_vcpu_ran; }; static inline bool kvm_s2_mmu_valid(struct kvm_s2_mmu *mmu) @@ -114,9 +117,6 @@ struct kvm_arch { /* VTCR_EL2 value for this VM */ u64 vtcr; - /* The last vcpu id that ran on each physical CPU */ - int __percpu *last_vcpu_ran; - /* The maximum number of vCPUs depends on the used GIC model */ int max_vcpus; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 41ff6aa728e3..9c0bf878fb3b 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -159,7 +159,7 @@ void free_hyp_pgds(void); void kvm_unmap_stage2_range(struct kvm_s2_mmu *mmu, phys_addr_t start, u64 size); void stage2_unmap_vm(struct kvm *kvm); -int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu); +int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu); void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu); int kvm_phys_addr_ioremap(struct kvm *kvm, phys_addr_t guest_ipa, phys_addr_t pa, unsigned long size, bool writable); diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 20c7c18e99ac..147943d876ef 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -12,7 +12,7 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) extern void kvm_init_nested(struct kvm *kvm); extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu); -extern void kvm_init_s2_mmu(struct kvm_s2_mmu *mmu); +extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu); extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index d7b2e17b54c3..d20f9982ffea 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -59,13 +59,8 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) num_mmus * sizeof(*kvm->arch.nested_mmus), GFP_KERNEL | __GFP_ZERO); if (tmp) { - tmp[num_mmus - 1].kvm = kvm; - atomic_set(&tmp[num_mmus - 1].refcnt, 0); - tmp[num_mmus - 2].kvm = kvm; - atomic_set(&tmp[num_mmus - 2].refcnt, 0); - - if (kvm_alloc_stage2_pgd(&tmp[num_mmus - 1]) || - kvm_alloc_stage2_pgd(&tmp[num_mmus - 2])) { + if (kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 1]) || + kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2])) { kvm_free_stage2_pgd(&tmp[num_mmus - 1]); kvm_free_stage2_pgd(&tmp[num_mmus - 2]); } else { @@ -445,7 +440,7 @@ static struct kvm_s2_mmu *get_s2_mmu_nested(struct kvm_vcpu *vcpu) return s2_mmu; } -void kvm_init_s2_mmu(struct kvm_s2_mmu *mmu) +void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu) { mmu->vttbr = 1; mmu->nested_stage2_enabled = false; diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index dcd8d54d38ce..9bb5d5f47237 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -102,26 +102,15 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, */ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) { - int ret, cpu; + int ret; ret = kvm_arm_setup_stage2(kvm, type); if (ret) return ret; - kvm->arch.last_vcpu_ran = alloc_percpu(typeof(*kvm->arch.last_vcpu_ran)); - if (!kvm->arch.last_vcpu_ran) - return -ENOMEM; - - for_each_possible_cpu(cpu) - *per_cpu_ptr(kvm->arch.last_vcpu_ran, cpu) = -1; - - /* Mark the initial VMID generation invalid */ - kvm->arch.mmu.vmid.vmid_gen = 0; - kvm->arch.mmu.kvm = kvm; - - ret = kvm_alloc_stage2_pgd(&kvm->arch.mmu); + ret = kvm_init_stage2_mmu(kvm, &kvm->arch.mmu); if (ret) - goto out_fail_alloc; + return ret; kvm_init_nested(kvm); @@ -138,9 +127,6 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type) return ret; out_free_stage2_pgd: kvm_free_stage2_pgd(&kvm->arch.mmu); -out_fail_alloc: - free_percpu(kvm->arch.last_vcpu_ran); - kvm->arch.last_vcpu_ran = NULL; return ret; } @@ -165,9 +151,6 @@ void kvm_arch_destroy_vm(struct kvm *kvm) kvm_vgic_destroy(kvm); - free_percpu(kvm->arch.last_vcpu_ran); - kvm->arch.last_vcpu_ran = NULL; - for (i = 0; i < KVM_MAX_VCPUS; ++i) { if (kvm->vcpus[i]) { kvm_vcpu_destroy(kvm->vcpus[i]); @@ -346,7 +329,7 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) if (nested_virt_in_use(vcpu)) kvm_vcpu_load_hw_mmu(vcpu); - last_ran = this_cpu_ptr(vcpu->kvm->arch.last_vcpu_ran); + last_ran = this_cpu_ptr(vcpu->arch.hw_mmu->last_vcpu_ran); cpu_data = this_cpu_ptr(&kvm_host_data); /* diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 054946566dc0..dc3bded72363 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -907,8 +907,9 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, } /** - * kvm_alloc_stage2_pgd - allocate level-1 table for stage-2 translation. - * @mmu: The stage 2 mmu struct pointer + * kvm_init_stage2_mmu - Initialise a S2 MMU strucrure + * @kvm: The pointer to the KVM structure + * @mmu: The pointer to the s2 MMU structure * * Allocates only the stage-2 HW PGD level table(s) of size defined by * stage2_pgd_size(mmu->kvm). @@ -916,10 +917,11 @@ int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size, * Note we don't need locking here as this is only called when the VM is * created, which can only be done once. */ -int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu) +int kvm_init_stage2_mmu(struct kvm *kvm, struct kvm_s2_mmu *mmu) { phys_addr_t pgd_phys; pgd_t *pgd; + int cpu; if (mmu->pgd != NULL) { kvm_err("kvm_arch already initialized?\n"); @@ -927,19 +929,29 @@ int kvm_alloc_stage2_pgd(struct kvm_s2_mmu *mmu) } /* Allocate the HW PGD, making sure that each page gets its own refcount */ - pgd = alloc_pages_exact(stage2_pgd_size(mmu->kvm), GFP_KERNEL | __GFP_ZERO); + pgd = alloc_pages_exact(stage2_pgd_size(kvm), GFP_KERNEL | __GFP_ZERO); if (!pgd) return -ENOMEM; pgd_phys = virt_to_phys(pgd); - if (WARN_ON(pgd_phys & ~kvm_vttbr_baddr_mask(mmu->kvm))) + if (WARN_ON(pgd_phys & ~kvm_vttbr_baddr_mask(kvm))) return -EINVAL; + mmu->last_vcpu_ran = alloc_percpu(typeof(*mmu->last_vcpu_ran)); + if (!mmu->last_vcpu_ran) { + free_pages_exact(pgd, stage2_pgd_size(kvm)); + return -ENOMEM; + } + + for_each_possible_cpu(cpu) + *per_cpu_ptr(mmu->last_vcpu_ran, cpu) = -1; + + mmu->kvm = kvm; mmu->pgd = pgd; mmu->pgd_phys = pgd_phys; mmu->vmid.vmid_gen = 0; - kvm_init_s2_mmu(mmu); + kvm_init_nested_s2_mmu(mmu); return 0; } @@ -1027,8 +1039,10 @@ void kvm_free_stage2_pgd(struct kvm_s2_mmu *mmu) spin_unlock(&kvm->mmu_lock); /* Free the HW pgd, one page at a time */ - if (pgd) + if (pgd) { free_pages_exact(pgd, stage2_pgd_size(kvm)); + free_percpu(mmu->last_vcpu_ran); + } } static pud_t *stage2_get_pud(struct kvm_s2_mmu *mmu, struct kvm_mmu_memory_cache *cache, From patchwork Tue Feb 11 17:48:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376141 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4917014B4 for ; Tue, 11 Feb 2020 17:51:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 28BC021569 for ; Tue, 11 Feb 2020 17:51:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443463; bh=DvF9CnzkNhbTzrEF+ToXoN9N21241sWEqKWABFpMVgQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=1cwFTgIkCoTNXQjEuBSGzsMp3Y6CR+cioRpfLoJM4MgfBjmXVCtJglBCpBmCjtdxw wGMc+I4TFRR5elz/Vj2kJo7VbSsY3jeYSR41CSvecJ6JbLrlUCO5+v7cgi/q57ntA9 UUwNNe+JYSt/nVawUwkl4fII4lii8Hzj3x5IKvkQ= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729984AbgBKRvC (ORCPT ); Tue, 11 Feb 2020 12:51:02 -0500 Received: from mail.kernel.org ([198.145.29.99]:53124 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728935AbgBKRvC (ORCPT ); Tue, 11 Feb 2020 12:51:02 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 25C9D20661; Tue, 11 Feb 2020 17:51:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443461; bh=DvF9CnzkNhbTzrEF+ToXoN9N21241sWEqKWABFpMVgQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zw3w+SwJG1S8yl0ppRowiJ3ER3948GmI8LFDcf9YIghd2fYoGr3zOkeT/fQXy5b8s TlYzyE8zzk0GhvGILy04m2JByef8XeKFuC3hD6eLewZ5chtHmSFixpDWeBMiqjJw2M LC4lrEpPTKKOYgtF+m6PuhQDPl4OIjxu6f1iBVE0= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfu-004O7k-1q; Tue, 11 Feb 2020 17:50:18 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 42/94] KVM: arm64: nv: Introduce sys_reg_desc.forward_trap Date: Tue, 11 Feb 2020 17:48:46 +0000 Message-Id: <20200211174938.27809-43-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim This introduces a function prototype to determine if we need to forward system instruction traps to the virtual EL2. The implementation of forward_trap functions for each system instruction will be added in later patches. Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 9 +++++++++ arch/arm64/kvm/sys_regs.h | 6 ++++++ 2 files changed, 15 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 364020afc17c..fae3f4a70eeb 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -25,6 +25,7 @@ #include #include #include +#include #include #include @@ -2491,6 +2492,14 @@ static void perform_access(struct kvm_vcpu *vcpu, */ BUG_ON(!r->access); + /* + * Forward this trap to the virtual EL2 if the guest hypervisor has + * configured to trap the current instruction. + */ + if (nested_virt_in_use(vcpu) && r->forward_trap + && unlikely(r->forward_trap(vcpu))) + return; + /* Skip instruction if instructed so */ if (likely(r->access(vcpu, params, r))) kvm_skip_instr(vcpu, kvm_vcpu_trap_il_is32bit(vcpu)); diff --git a/arch/arm64/kvm/sys_regs.h b/arch/arm64/kvm/sys_regs.h index 5a6fc30f5989..65e736f7103a 100644 --- a/arch/arm64/kvm/sys_regs.h +++ b/arch/arm64/kvm/sys_regs.h @@ -54,6 +54,12 @@ struct sys_reg_desc { int (*set_user)(struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd, const struct kvm_one_reg *reg, void __user *uaddr); + /* + * Forward the trap to the virtual EL2 if the guest hypervisor has + * configured to trap the current instruction. + */ + bool (*forward_trap)(struct kvm_vcpu *vcpu); + /* Return mask of REG_* runtime visibility overrides */ unsigned int (*visibility)(const struct kvm_vcpu *vcpu, const struct sys_reg_desc *rd); From patchwork Tue Feb 11 17:48:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376221 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B86AC1395 for ; Tue, 11 Feb 2020 17:52:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8F93820870 for ; Tue, 11 Feb 2020 17:52:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443534; bh=KY+c3FxIs11wncD+b7KCS0+WsyvwPMVyQczqhhQ1+gQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=v8wDYHx30bl4iBmspO9WHHrRGaLdY338jL5UFRUl43uIu1skSKtENICRiTCWIxKiA Y3bH3h9KMj2RswtZr9BEoCnT3/908KXObvKZvA7KlBcXirL6vds0BLwSZH4pIkVIDD Irr8J4lJnbVD/QNbzpgr4SxvVJqlluNsfZ5z0BXY= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730504AbgBKRwN (ORCPT ); Tue, 11 Feb 2020 12:52:13 -0500 Received: from mail.kernel.org ([198.145.29.99]:54942 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727966AbgBKRwN (ORCPT ); Tue, 11 Feb 2020 12:52:13 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id EDAE820578; Tue, 11 Feb 2020 17:52:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443532; bh=KY+c3FxIs11wncD+b7KCS0+WsyvwPMVyQczqhhQ1+gQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=vvMLmr/vlOMikhad/4fTGQJ0UeE3qQwNzV/IHmcfdpFDHkBSseWx3DesOPfKMVHr1 QAF5Pz9ovyAB+j466+wsQvchN5bTLTDAk6RZLP5xhCFqYarOI0ROdMZujub8EzG97a MdmnKLKk3iItYuM45U51XPfWp0QiEb1rgzXVhxlg= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfu-004O7k-K8; Tue, 11 Feb 2020 17:50:18 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 43/94] KVM: arm64: nv: Set a handler for the system instruction traps Date: Tue, 11 Feb 2020 17:48:47 +0000 Message-Id: <20200211174938.27809-44-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim When HCR.NV bit is set, execution of the EL2 translation regime address aranslation instructions and TLB maintenance instructions are trapped to EL2. In addition, execution of the EL1 translation regime address aranslation instructions and TLB maintenance instructions that are only accessible from EL2 and above are trapped to EL2. In these cases, ESR_EL2.EC will be set to 0x18. Rework the system instruction emulation framework to handle potentially all system instruction traps other than MSR/MRS instructions. Those system instructions would be AT and TLBI instructions controlled by HCR_EL2.NV, AT, and TTLB bits. Signed-off-by: Jintack Lim [maz: squashed two patches together, redispatched various bits around] Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_coproc.h | 2 +- arch/arm64/kvm/handle_exit.c | 2 +- arch/arm64/kvm/sys_regs.c | 48 +++++++++++++++++++++++------ 3 files changed, 41 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/kvm_coproc.h b/arch/arm64/include/asm/kvm_coproc.h index 0185ee8b8b5e..755a6e8f907e 100644 --- a/arch/arm64/include/asm/kvm_coproc.h +++ b/arch/arm64/include/asm/kvm_coproc.h @@ -32,7 +32,7 @@ int kvm_handle_cp14_32(struct kvm_vcpu *vcpu, struct kvm_run *run); int kvm_handle_cp14_64(struct kvm_vcpu *vcpu, struct kvm_run *run); int kvm_handle_cp15_32(struct kvm_vcpu *vcpu, struct kvm_run *run); int kvm_handle_cp15_64(struct kvm_vcpu *vcpu, struct kvm_run *run); -int kvm_handle_sys_reg(struct kvm_vcpu *vcpu, struct kvm_run *run); +int kvm_handle_sys(struct kvm_vcpu *vcpu, struct kvm_run *run); #define kvm_coproc_table_init kvm_sys_reg_table_init void kvm_sys_reg_table_init(void); diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index d2a73ebf9d4a..10f13bd2e319 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -275,7 +275,7 @@ static exit_handle_fn arm_exit_handlers[] = { [ESR_ELx_EC_SMC32] = handle_smc, [ESR_ELx_EC_HVC64] = handle_hvc, [ESR_ELx_EC_SMC64] = handle_smc, - [ESR_ELx_EC_SYS64] = kvm_handle_sys_reg, + [ESR_ELx_EC_SYS64] = kvm_handle_sys, [ESR_ELx_EC_SVE] = handle_sve, [ESR_ELx_EC_ERET] = kvm_handle_eret, [ESR_ELx_EC_IABT_LOW] = kvm_handle_guest_abort, diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index fae3f4a70eeb..026395d16c9f 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1745,10 +1745,6 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, * more demanding guest... */ static const struct sys_reg_desc sys_reg_descs[] = { - { SYS_DESC(SYS_DC_ISW), access_dcsw }, - { SYS_DESC(SYS_DC_CSW), access_dcsw }, - { SYS_DESC(SYS_DC_CISW), access_dcsw }, - DBG_BCR_BVR_WCR_WVR_EL1(0), DBG_BCR_BVR_WCR_WVR_EL1(1), { SYS_DESC(SYS_MDCCINT_EL1), trap_debug_regs, reset_val, MDCCINT_EL1, 0 }, @@ -2093,6 +2089,14 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; +#define SYS_INSN_TO_DESC(insn, access_fn, forward_fn) \ + { SYS_DESC((insn)), (access_fn), NULL, 0, 0, NULL, NULL, (forward_fn) } +static struct sys_reg_desc sys_insn_descs[] = { + { SYS_DESC(SYS_DC_ISW), access_dcsw }, + { SYS_DESC(SYS_DC_CSW), access_dcsw }, + { SYS_DESC(SYS_DC_CISW), access_dcsw }, +}; + static bool trap_dbgidr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -2722,6 +2726,24 @@ static int emulate_sys_reg(struct kvm_vcpu *vcpu, return 1; } +static int emulate_sys_instr(struct kvm_vcpu *vcpu, struct sys_reg_params *p) +{ + const struct sys_reg_desc *r; + + /* Search from the system instruction table. */ + r = find_reg(p, sys_insn_descs, ARRAY_SIZE(sys_insn_descs)); + + if (likely(r)) { + perform_access(vcpu, p, r); + } else { + kvm_err("Unsupported guest sys instruction at: %lx\n", + *vcpu_pc(vcpu)); + print_sys_reg_instr(p); + kvm_inject_undefined(vcpu); + } + return 1; +} + static void reset_sys_reg_descs(struct kvm_vcpu *vcpu, const struct sys_reg_desc *table, size_t num, unsigned long *bmap) @@ -2739,11 +2761,12 @@ static void reset_sys_reg_descs(struct kvm_vcpu *vcpu, } /** - * kvm_handle_sys_reg -- handles a mrs/msr trap on a guest sys_reg access + * kvm_handle_sys-- handles a system instruction or mrs/msr instruction trap + on a guest execution * @vcpu: The VCPU pointer * @run: The kvm_run struct */ -int kvm_handle_sys_reg(struct kvm_vcpu *vcpu, struct kvm_run *run) +int kvm_handle_sys(struct kvm_vcpu *vcpu, struct kvm_run *run) { struct sys_reg_params params; unsigned long esr = kvm_vcpu_get_hsr(vcpu); @@ -2762,10 +2785,16 @@ int kvm_handle_sys_reg(struct kvm_vcpu *vcpu, struct kvm_run *run) params.regval = vcpu_get_reg(vcpu, Rt); params.is_write = !(esr & 1); - ret = emulate_sys_reg(vcpu, ¶ms); + if (params.Op0 == 1) { + /* System instructions */ + ret = emulate_sys_instr(vcpu, ¶ms); + } else { + /* MRS/MSR instructions */ + ret = emulate_sys_reg(vcpu, ¶ms); + if (!params.is_write) + vcpu_set_reg(vcpu, Rt, params.regval); + } - if (!params.is_write) - vcpu_set_reg(vcpu, Rt, params.regval); return ret; } @@ -3217,6 +3246,7 @@ void kvm_sys_reg_table_init(void) BUG_ON(check_sysreg_table(cp15_regs, ARRAY_SIZE(cp15_regs))); BUG_ON(check_sysreg_table(cp15_64_regs, ARRAY_SIZE(cp15_64_regs))); BUG_ON(check_sysreg_table(invariant_sys_regs, ARRAY_SIZE(invariant_sys_regs))); + BUG_ON(check_sysreg_table(sys_insn_descs, ARRAY_SIZE(sys_insn_descs))); /* We abuse the reset function to overwrite the table itself. */ for (i = 0; i < ARRAY_SIZE(invariant_sys_regs); i++) From patchwork Tue Feb 11 17:48:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376231 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 14A2A14B4 for ; Tue, 11 Feb 2020 17:52:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CDA6E206CC for ; Tue, 11 Feb 2020 17:52:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443544; bh=e288yNviCD4w55sXsGvgxFgNRKWiZoGCOqW7St9mC1M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Ur4PzymiTOR5HDhKWfikC/KFudsK8oG0op7S8qmYdtv6OIpSGi9AGqvMvJ7V7MJRl /45kagjie8bfS8sUJ+6hZpMhfmtMSJfTSh2ilXsWr2iyH0jervZOkdCkGVEut3yaCG 71RRfsyrrbIgE4cZz/yG6fMbZPLbFk6op0+RlpAw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730523AbgBKRwX (ORCPT ); Tue, 11 Feb 2020 12:52:23 -0500 Received: from mail.kernel.org ([198.145.29.99]:55240 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730518AbgBKRwW (ORCPT ); Tue, 11 Feb 2020 12:52:22 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 566F421569; Tue, 11 Feb 2020 17:52:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443541; bh=e288yNviCD4w55sXsGvgxFgNRKWiZoGCOqW7St9mC1M=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p8P6ra8cseEoZYcvUkZ6uwV5TiQbgxng7M8AvvVdwpvZj33vypHDBShimcgc8jacX 0GGvOHSWI1RDUp2/IjncfudT3s5UD/gubRmcyLBkbjh0NB5ARMzBP1KL8YMAKReg0h f3t7qK6lT4P0Y/dfDmHC6HFGwKZbUfzIdlXTCvj4= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfv-004O7k-6j; Tue, 11 Feb 2020 17:50:19 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 44/94] KVM: arm64: nv: Trap and emulate AT instructions from virtual EL2 Date: Tue, 11 Feb 2020 17:48:48 +0000 Message-Id: <20200211174938.27809-45-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim When supporting nested virtualization a guest hypervisor executing AT instructions must be trapped and emulated by the host hypervisor, because untrapped AT instructions operating on S1E1 will use the wrong translation regieme (the one used to emulate virtual EL2 in EL1 instead of virtual EL1) and AT instructions operating on S12 will not work from EL1. This patch does several things. 1. List and define all AT system instructions to emulate and document the emulation design. 2. Implement AT instruction handling logic in EL2. This will be used to emulate AT instructions executed in the virtual EL2. AT instruction emulation works by loading the proper processor context, which depends on the trapped instruction and the virtual HCR_EL2, to the EL1 virtual memory control registers and executing AT instructions. Note that ctxt->hw_sys_regs is expected to have the proper processor context before calling the handling function(__kvm_at_insn) implemented in this patch. 4. Emulate AT S1E[01] instructions by issuing the same instructions in EL2. We set the physical EL1 registers, NV and NV1 bits as described in the AT instruction emulation overview. 5. Emulate AT A12E[01] instructions in two steps: First, do the stage-1 translation by reusing the existing AT emulation functions. Second, do the stage-2 translation by walking the guest hypervisor's stage-2 page table in software. Record the translation result to PAR_EL1. 6. Emulate AT S1E2 instructions by issuing the corresponding S1E1 instructions in EL2. We set the physical EL1 registers and the HCR_EL2 register as described in the AT instruction emulation overview. 7. Forward system instruction traps to the virtual EL2 if the corresponding virtual AT bit is set in the virtual HCR_EL2. [ Much logic above has been reworked by Marc Zyngier ] Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier Signed-off-by: Christoffer Dall --- arch/arm64/include/asm/kvm_arm.h | 2 + arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 17 +++ arch/arm64/kvm/hyp/Makefile | 1 + arch/arm64/kvm/hyp/at.c | 231 +++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/switch.c | 13 +- arch/arm64/kvm/sys_regs.c | 201 ++++++++++++++++++++++++++- 7 files changed, 463 insertions(+), 4 deletions(-) create mode 100644 arch/arm64/kvm/hyp/at.c diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 95dfef820cf1..2e5be806a5c9 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -13,6 +13,7 @@ /* Hyp Configuration Register (HCR) bits */ #define HCR_FWB (UL(1) << 46) +#define HCR_AT (UL(1) << 44) #define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) #define HCR_API (UL(1) << 41) @@ -107,6 +108,7 @@ #define VTCR_EL2_TG0_16K TCR_TG0_16K #define VTCR_EL2_TG0_64K TCR_TG0_64K #define VTCR_EL2_SH0_MASK TCR_SH0_MASK +#define VTCR_EL2_SH0_SHIFT TCR_SH0_SHIFT #define VTCR_EL2_SH0_INNER TCR_SH0_INNER #define VTCR_EL2_ORGN0_MASK TCR_ORGN0_MASK #define VTCR_EL2_ORGN0_WBWA TCR_ORGN0_WBWA diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 80f5633d9559..8acbe64572a2 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -64,6 +64,8 @@ extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high); +extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); +extern void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); extern int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index a167219e42f4..f0bb76c7d04d 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -532,6 +532,23 @@ #define SYS_SP_EL2 sys_reg(3, 6, 4, 1, 0) +/* AT instructions */ +#define AT_Op0 1 +#define AT_CRn 7 + +#define OP_AT_S1E1R sys_insn(AT_Op0, 0, AT_CRn, 8, 0) +#define OP_AT_S1E1W sys_insn(AT_Op0, 0, AT_CRn, 8, 1) +#define OP_AT_S1E0R sys_insn(AT_Op0, 0, AT_CRn, 8, 2) +#define OP_AT_S1E0W sys_insn(AT_Op0, 0, AT_CRn, 8, 3) +#define OP_AT_S1E1RP sys_insn(AT_Op0, 0, AT_CRn, 9, 0) +#define OP_AT_S1E1WP sys_insn(AT_Op0, 0, AT_CRn, 9, 1) +#define OP_AT_S1E2R sys_insn(AT_Op0, 4, AT_CRn, 8, 0) +#define OP_AT_S1E2W sys_insn(AT_Op0, 4, AT_CRn, 8, 1) +#define OP_AT_S12E1R sys_insn(AT_Op0, 4, AT_CRn, 8, 4) +#define OP_AT_S12E1W sys_insn(AT_Op0, 4, AT_CRn, 8, 5) +#define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) +#define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44)) #define SCTLR_ELx_ENIA (BIT(31)) diff --git a/arch/arm64/kvm/hyp/Makefile b/arch/arm64/kvm/hyp/Makefile index ea710f674cb6..f7af51647079 100644 --- a/arch/arm64/kvm/hyp/Makefile +++ b/arch/arm64/kvm/hyp/Makefile @@ -19,6 +19,7 @@ obj-$(CONFIG_KVM_ARM_HOST) += entry.o obj-$(CONFIG_KVM_ARM_HOST) += switch.o obj-$(CONFIG_KVM_ARM_HOST) += fpsimd.o obj-$(CONFIG_KVM_ARM_HOST) += tlb.o +obj-$(CONFIG_KVM_ARM_HOST) += at.o obj-$(CONFIG_KVM_ARM_HOST) += hyp-entry.o # KVM code is run at a different exception code with a different map, so diff --git a/arch/arm64/kvm/hyp/at.c b/arch/arm64/kvm/hyp/at.c new file mode 100644 index 000000000000..9fdb87fa3597 --- /dev/null +++ b/arch/arm64/kvm/hyp/at.c @@ -0,0 +1,231 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2017 - Linaro Ltd + * Author: Jintack Lim + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program. If not, see . + */ + +#include +#include + +struct mmu_config { + u64 ttbr0; + u64 ttbr1; + u64 tcr; + u64 sctlr; + u64 vttbr; + u64 vtcr; + u64 hcr; +}; + +static void __mmu_config_save(struct mmu_config *config) +{ + config->ttbr0 = read_sysreg_el1(SYS_TTBR0); + config->ttbr1 = read_sysreg_el1(SYS_TTBR1); + config->tcr = read_sysreg_el1(SYS_TCR); + config->sctlr = read_sysreg_el1(SYS_SCTLR); + config->vttbr = read_sysreg(vttbr_el2); + config->vtcr = read_sysreg(vtcr_el2); + config->hcr = read_sysreg(hcr_el2); +} + +static void __mmu_config_restore(struct mmu_config *config) +{ + write_sysreg_el1(config->ttbr0, SYS_TTBR0); + write_sysreg_el1(config->ttbr1, SYS_TTBR1); + write_sysreg_el1(config->tcr, SYS_TCR); + write_sysreg_el1(config->sctlr, SYS_SCTLR); + write_sysreg(config->vttbr, vttbr_el2); + write_sysreg(config->vtcr, vtcr_el2); + write_sysreg(config->hcr, hcr_el2); + + isb(); +} + +void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + struct mmu_config config; + struct kvm_s2_mmu *mmu; + + spin_lock(&vcpu->kvm->mmu_lock); + + /* + * If HCR_EL2.{E2H,TGE} == {1,1}, the MMU context is already + * the right one (as we trapped from vEL2). + */ + if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) + goto skip_mmu_switch; + + /* + * FIXME: Obtaining the S2 MMU for a guest guest is horribly + * racy, and we may not find it (evicted by another vcpu, for + * example). + */ + mmu = lookup_s2_mmu(vcpu->kvm, + vcpu_read_sys_reg(vcpu, VTTBR_EL2), + vcpu_read_sys_reg(vcpu, HCR_EL2)); + + if (WARN_ON(!mmu)) + goto out; + + /* We've trapped, so everything is live on the CPU. */ + __mmu_config_save(&config); + + write_sysreg_el1(ctxt->sys_regs[TTBR0_EL1], SYS_TTBR0); + write_sysreg_el1(ctxt->sys_regs[TTBR1_EL1], SYS_TTBR1); + write_sysreg_el1(ctxt->sys_regs[TCR_EL1], SYS_TCR); + write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); + write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); + /* + * REVISIT: do we need anything from the guest's VTCR_EL2? If + * looks like keeping the hosts configuration is the right + * thing to do at this stage (and we could avoid save/restore + * it. Keep the host's version for now. + */ + write_sysreg((config.hcr & ~HCR_TGE) | HCR_VM, hcr_el2); + + isb(); + +skip_mmu_switch: + + switch (op) { + case OP_AT_S1E1R: + case OP_AT_S1E1RP: + asm volatile("at s1e1r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E1W: + case OP_AT_S1E1WP: + asm volatile("at s1e1w, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E0R: + asm volatile("at s1e0r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E0W: + asm volatile("at s1e0w, %0" : : "r" (vaddr)); + break; + default: + WARN_ON_ONCE(1); + break; + } + + isb(); + + ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); + + /* + * Failed? let's leave the building now. + * + * FIXME: how about a failed translation because the shadow S2 + * wasn't populated? We may need to perform a SW PTW, + * populating our shadow S2 and retry the instruction. + */ + if (ctxt->sys_regs[PAR_EL1] & 1) + goto nopan; + + /* No PAN? No problem. */ + if (!(*vcpu_cpsr(vcpu) & PSR_PAN_BIT)) + goto nopan; + + /* + * For PAN-involved AT operations, perform the same + * translation, using EL0 this time. + */ + switch (op) { + case OP_AT_S1E1RP: + asm volatile("at s1e0r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E1WP: + asm volatile("at s1e0w, %0" : : "r" (vaddr)); + break; + default: + goto nopan; + } + + /* + * If the EL0 translation has succeeded, we need to pretend + * the AT operation has failed, as the PAN setting forbids + * such a translation. + * + * FIXME: we hardcode a Level-3 permission fault. We really + * should return the real fault level. + */ + if (!(read_sysreg(par_el1) & 1)) + ctxt->sys_regs[PAR_EL1] = 0x1f; + +nopan: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + __mmu_config_restore(&config); + +out: + spin_unlock(&vcpu->kvm->mmu_lock); +} + +void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + struct mmu_config config; + struct kvm_s2_mmu *mmu; + u64 val; + + spin_lock(&vcpu->kvm->mmu_lock); + + mmu = &vcpu->kvm->arch.mmu; + + /* We've trapped, so everything is live on the CPU. */ + __mmu_config_save(&config); + + if (vcpu_el2_e2h_is_set(vcpu)) { + write_sysreg_el1(ctxt->sys_regs[TTBR0_EL2], SYS_TTBR0); + write_sysreg_el1(ctxt->sys_regs[TTBR1_EL2], SYS_TTBR1); + write_sysreg_el1(ctxt->sys_regs[TCR_EL2], SYS_TCR); + write_sysreg_el1(ctxt->sys_regs[SCTLR_EL2], SYS_SCTLR); + + val = config.hcr; + } else { + write_sysreg_el1(ctxt->sys_regs[TTBR0_EL2], SYS_TTBR0); + write_sysreg_el1(translate_tcr(ctxt->sys_regs[TCR_EL2]), + SYS_TCR); + write_sysreg_el1(translate_sctlr(ctxt->sys_regs[SCTLR_EL2]), + SYS_SCTLR); + + val = config.hcr | HCR_NV | HCR_NV1; + } + + write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); + /* FIXME: write S2 MMU VTCR_EL2? */ + write_sysreg((val & ~HCR_TGE) | HCR_VM, hcr_el2); + + isb(); + + switch (op) { + case OP_AT_S1E2R: + asm volatile("at s1e1r, %0" : : "r" (vaddr)); + break; + case OP_AT_S1E2W: + asm volatile("at s1e1w, %0" : : "r" (vaddr)); + break; + default: + WARN_ON_ONCE(1); + break; + } + + isb(); + + /* FIXME: handle failed translation due to shadow S2 */ + ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); + + __mmu_config_restore(&config); + spin_unlock(&vcpu->kvm->mmu_lock); +} diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 08f12c96c458..61686b8a992e 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -157,9 +157,10 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_e2h_is_set(vcpu)) { /* * For a guest hypervisor on v8.0, trap and emulate - * the EL1 virtual memory control register accesses. + * the EL1 virtual memory control register accesses + * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_NV1; + hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -182,6 +183,14 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) hcr &= ~HCR_TVM; hcr |= vhcr_el2 & (HCR_TVM | HCR_TRVM); + + /* + * If we're using the EL1 translation regime + * (TGE clear, then ensure that AT S1 ops are + * trapped too. + */ + if (!vcpu_el2_tge_is_set(vcpu)) + hcr |= HCR_AT; } } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 026395d16c9f..1ff2e16b1f57 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1678,6 +1678,10 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, return true; } +static bool forward_at_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_AT); +} static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, @@ -2089,12 +2093,205 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; -#define SYS_INSN_TO_DESC(insn, access_fn, forward_fn) \ - { SYS_DESC((insn)), (access_fn), NULL, 0, 0, NULL, NULL, (forward_fn) } +static bool handle_s1e01(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + __kvm_at_s1e01(vcpu, sys_encoding, p->regval); + + return true; +} + +static bool handle_s1e2(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + __kvm_at_s1e2(vcpu, sys_encoding, p->regval); + + return true; +} + +static u64 setup_par_aborted(u32 esr) +{ + u64 par = 0; + + /* S [9]: fault in the stage 2 translation */ + par |= (1 << 9); + /* FST [6:1]: Fault status code */ + par |= (esr << 1); + /* F [0]: translation is aborted */ + par |= 1; + + return par; +} + +static u64 setup_par_completed(struct kvm_vcpu *vcpu, struct kvm_s2_trans *out) +{ + u64 par, vtcr_sh0; + + /* F [0]: Translation is completed successfully */ + par = 0; + /* ATTR [63:56] */ + par |= out->upper_attr; + /* PA [47:12] */ + par |= out->output & GENMASK_ULL(11, 0); + /* RES1 [11] */ + par |= (1UL << 11); + /* SH [8:7]: Shareability attribute */ + vtcr_sh0 = vcpu_read_sys_reg(vcpu, VTCR_EL2) & VTCR_EL2_SH0_MASK; + par |= (vtcr_sh0 >> VTCR_EL2_SH0_SHIFT) << 7; + + return par; +} + +static bool handle_s12(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r, bool write) +{ + u64 par, va; + u32 esr; + phys_addr_t ipa; + struct kvm_s2_trans out; + int ret; + + /* Do the stage-1 translation */ + handle_s1e01(vcpu, p, r); + par = vcpu_read_sys_reg(vcpu, PAR_EL1); + if (par & 1) { + /* The stage-1 translation aborted */ + return true; + } + + /* Do the stage-2 translation */ + va = p->regval; + ipa = (par & GENMASK_ULL(47, 12)) | (va & GENMASK_ULL(11, 0)); + out.esr = 0; + ret = kvm_walk_nested_s2(vcpu, ipa, &out); + if (ret < 0) + return false; + + /* Check if the stage-2 PTW is aborted */ + if (out.esr) { + esr = out.esr; + goto s2_trans_abort; + } + + /* Check the access permission */ + if ((!write && !out.readable) || (write && !out.writable)) { + esr = ESR_ELx_FSC_PERM; + esr |= out.level & 0x3; + goto s2_trans_abort; + } + + vcpu_write_sys_reg(vcpu, setup_par_completed(vcpu, &out), PAR_EL1); + return true; + +s2_trans_abort: + vcpu_write_sys_reg(vcpu, setup_par_aborted(esr), PAR_EL1); + return true; +} + +static bool handle_s12r(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return handle_s12(vcpu, p, r, false); +} + +static bool handle_s12w(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + return handle_s12(vcpu, p, r, true); +} + +/* + * AT instruction emulation + * + * We emulate AT instructions executed in the virtual EL2. + * Basic strategy for the stage-1 translation emulation is to load proper + * context, which depends on the trapped instruction and the virtual HCR_EL2, + * to the EL1 virtual memory control registers and execute S1E[01] instructions + * in EL2. See below for more detail. + * + * For the stage-2 translation, which is necessary for S12E[01] emulation, + * we walk the guest hypervisor's stage-2 page table in software. + * + * The stage-1 translation emulations can be divided into two groups depending + * on the translation regime. + * + * 1. EL2 AT instructions: S1E2x + * +-----------------------------------------------------------------------+ + * | | Setting for the emulation | + * | Virtual HCR_EL2.E2H on trap |-----------------------------------------+ + * | | Phys EL1 regs | Phys NV, NV1 | Phys TGE | + * |-----------------------------------------------------------------------| + * | 0 | vEL2 | (1, 1) | 0 | + * | 1 | vEL2 | (0, 0) | 0 | + * +-----------------------------------------------------------------------+ + * + * We emulate the EL2 AT instructions by loading virtual EL2 context + * to the EL1 virtual memory control registers and executing corresponding + * EL1 AT instructions. + * + * We set physical NV and NV1 bits to use EL2 page table format for non-VHE + * guest hypervisor (i.e. HCR_EL2.E2H == 0). As a VHE guest hypervisor uses the + * EL1 page table format, we don't set those bits. + * + * We should clear physical TGE bit not to use the EL2 translation regime when + * the host uses the VHE feature. + * + * + * 2. EL0/EL1 AT instructions: S1E[01]x, S12E1x + * +----------------------------------------------------------------------+ + * | Virtual HCR_EL2 on trap | Setting for the emulation | + * |----------------------------------------------------------------------+ + * | (vE2H, vTGE) | (vNV, vNV1) | Phys EL1 regs | Phys NV, NV1 | Phys TGE | + * |----------------------------------------------------------------------| + * | (0, 0)* | (0, 0) | vEL1 | (0, 0) | 0 | + * | (0, 0) | (1, 1) | vEL1 | (1, 1) | 0 | + * | (1, 1) | (0, 0) | vEL2 | (0, 0) | 0 | + * | (1, 1) | (1, 1) | vEL2 | (1, 1) | 0 | + * +----------------------------------------------------------------------+ + * + * *For (0, 0) in the 'Virtual HCR_EL2 on trap' column, it actually means + * (1, 1). Keep them (0, 0) just for the readability. + * + * We set physical EL1 virtual memory control registers depending on + * (vE2H, vTGE) pair. When the pair is (0, 0) where AT instructions are + * supposed to use EL0/EL1 translation regime, we load the EL1 registers with + * the virtual EL1 registers (i.e. EL1 registers from the guest hypervisor's + * point of view). When the pair is (1, 1), however, AT instructions are defined + * to apply EL2 translation regime. To emulate this behavior, we load the EL1 + * registers with the virtual EL2 context. (i.e the shadow registers) + * + * We respect the virtual NV and NV1 bit for the emulation. When those bits are + * set, it means that a guest hypervisor would like to use EL2 page table format + * for the EL1 translation regime. We emulate this by setting the physical + * NV and NV1 bits. + */ + +#define SYS_INSN_TO_DESC(insn, access_fn, forward_fn) \ + { SYS_DESC(OP_##insn), (access_fn), NULL, 0, 0, \ + NULL, NULL, (forward_fn) } static struct sys_reg_desc sys_insn_descs[] = { { SYS_DESC(SYS_DC_ISW), access_dcsw }, + + SYS_INSN_TO_DESC(AT_S1E1R, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E1W, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E0R, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E0W, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E1RP, handle_s1e01, forward_at_traps), + SYS_INSN_TO_DESC(AT_S1E1WP, handle_s1e01, forward_at_traps), + { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CISW), access_dcsw }, + + SYS_INSN_TO_DESC(AT_S1E2R, handle_s1e2, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S1E2W, handle_s1e2, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E1R, handle_s12r, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E1W, handle_s12w, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E0R, handle_s12r, forward_nv_traps), + SYS_INSN_TO_DESC(AT_S12E0W, handle_s12w, forward_nv_traps), }; static bool trap_dbgidr(struct kvm_vcpu *vcpu, From patchwork Tue Feb 11 17:48:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376275 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5BE4C1395 for ; Tue, 11 Feb 2020 17:53:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 25B5520848 for ; Tue, 11 Feb 2020 17:53:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443584; bh=neKRGmp3bYgRzDrghb/Za/dnkVcGTm2yCxfYFw+yBO0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=UhGVm0h0zswB+h4HWCh2IQYfd7vW+sTgI72xi6Vw1SdoZlZc/YKmHM62SzeyagX5G Uug9dZ5uvtN/pa/Xxg7BZ0pBk2dWukF093KGGTuyZWaDJCPvg84c47MzP13fA1e2v2 +tOTMlzSTlLSU9EaqIBwzZ6lSzIPbvhbojYwIeMk= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730636AbgBKRxC (ORCPT ); Tue, 11 Feb 2020 12:53:02 -0500 Received: from mail.kernel.org ([198.145.29.99]:56296 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730570AbgBKRxC (ORCPT ); Tue, 11 Feb 2020 12:53:02 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B2F52206CC; Tue, 11 Feb 2020 17:53:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443581; bh=neKRGmp3bYgRzDrghb/Za/dnkVcGTm2yCxfYFw+yBO0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jIkqn+tmccY75fF46GcsxNsbpyMTZvbqo83vOHJJThT2TsZaFDKnBt8jwNbXzOYCG vnZgcqdsM8BdgeHQtpL3PgiPTdM2MoCR1Ogp6MP0AiFgCzCxzVVzk95tRWvP7ZN1zZ Hdt6BerOtWL+Hb/+wYpYQKL0wYJNkSwHHU7TbJSQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfv-004O7k-Se; Tue, 11 Feb 2020 17:50:20 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 45/94] KVM: arm64: nv: Trap and emulate TLBI instructions from virtual EL2 Date: Tue, 11 Feb 2020 17:48:49 +0000 Message-Id: <20200211174938.27809-46-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim When supporting nested virtualization a guest hypervisor executing TLBI instructions must be trapped and emulated by the host hypervisor, because the guest hypervisor can only affect physical TLB entries relating to its own execution environment (virtual EL2 in EL1) but not to the nested guests as required by the semantics of the instructions and TLBI instructions might also result in updates (invalidations) to shadow page tables. This patch does several things. 1. List and define all TLBI system instructions to emulate. 2. Emulate TLBI ALLE2(IS) instruction executed in the virtual EL2. Since we emulate the virtual EL2 in the EL1, we invalidate EL1&0 regime stage 1 TLB entries with setting vttbr_el2 having the VMID of the virtual EL2. 3. Emulate TLBI VAE2* instruction executed in the virtual EL2. Based on the same principle as TLBI ALLE2 instruction, we can simply emulate those instructions by executing corresponding VAE1* instructions with the virtual EL2's VMID assigned by the host hypervisor. Note that we are able to emulate TLBI ALLE2IS precisely by only invalidating stage 1 TLB entries via TLBI VMALL1IS instruction, but to make it simeple, we reuse the existing function, __kvm_tlb_flush_vmid(), which invalidates both of stage 1 and 2 TLB entries. 4. TLBI ALLE1(IS) instruction invalidates all EL1&0 regime stage 1 and 2 TLB entries (on all PEs in the same Inner Shareable domain). To emulate these instructions, we first need to clear all the mappings in the shadow page tables since executing those instructions implies the change of mappings in the stage 2 page tables maintained by the guest hypervisor. We then need to invalidate all EL1&0 regime stage 1 and 2 TLB entries of all VMIDs, which are assigned by the host hypervisor, for this VM. 5. Based on the same principle as TLBI ALLE1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. But this time we do it only for the current VMID from the guest hypervisor's perspective, not for all VMIDs. 6. Based on the same principle as TLBI ALLE1(IS) and TLBI VMALLS12E1(IS) emulation, we clear the mappings in the shadow stage-2 page tables and invalidate TLB entries. We do it only for one mapping for the current VMID from the guest hypervisor's view. 7. Forward system instruction traps to the virtual EL2 if a corresponding bit in the virtual HCR_EL2 is set. 8. Even though a guest hypervisor can execute TLBI instructions that are accesible at EL1 without trap, it's wrong; All those TLBI instructions work based on current VMID, and when running a guest hypervisor current VMID is the one for itself, not the one from the virtual vttbr_el2. So letting a guest hypervisor execute those TLBI instructions results in invalidating its own TLB entries and leaving invalid TLB entries unhandled. Therefore we trap and emulate those TLBI instructions. The emulation is simple; we find a shadow VMID mapped to the virtual vttbr_el2, set it in the physical vttbr_el2, then execute the same instruction in EL2. We don't set HCR_EL2.TTLB bit yet. [ Changes performed by Marc Zynger: The TLBI handling code more or less directly execute the same instruction that has been trapped (with an EL2->EL1 conversion in the case of an EL2 TLBI), but that's unfortunately not enough: - TLBIs must be upgraded to the Inner Shareable domain to account for vcpu migration, just like we already have with HCR_EL2.FB. - The DSB instruction that synchronises these must thus be on the Inner Shareable domain as well. - Prior to executing the TLBI, we need another DSB ISHST to make sure that the update to the page tables is now visible. Ordering of system instructions fixed - The current TLB invalidation code is pretty buggy, as it assume a page mapping. On the contrary, it is likely that TLB invalidation will cover more than a single page, and the size should be decided by the guests configuration (and not the host's). Since we don't cache the guest mapping sizes in the shadow PT yet, let's assume the worse case (a block mapping) and invalidate that. Take this opportunity to fix the decoding of the parameter (it isn't a straight IPA). - In general, we always emulate local TBL invalidations as being as upgraded to the Inner Shareable domain so that we can easily deal with vcpu migration. This is consistent with the fact that we set HCR_EL2.FB when running non-nested VMs. So let's emulate TLBI ALLE2 as ALLE2IS. ] [ Changes performed by Christoffer Dall: Sometimes when we are invalidating the TLB for a certain S2 MMU context, this context can also have EL2 context associated with it and we have to invalidate this too. ] Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier Signed-off-by: Christoffer Dall --- arch/arm64/include/asm/kvm_asm.h | 2 + arch/arm64/include/asm/sysreg.h | 36 ++++++ arch/arm64/kvm/hyp/switch.c | 4 +- arch/arm64/kvm/hyp/tlb.c | 81 ++++++++++++ arch/arm64/kvm/sys_regs.c | 216 +++++++++++++++++++++++++++++++ virt/kvm/arm/mmu.c | 18 ++- 6 files changed, 354 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 8acbe64572a2..c43160d2940d 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -62,6 +62,8 @@ extern void __kvm_flush_vm_context(void); extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); +extern void __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding); +extern void __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding); extern void __kvm_timer_set_cntvoff(u32 cntvoff_low, u32 cntvoff_high); extern void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr); diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index f0bb76c7d04d..bf465ddde59e 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -549,6 +549,42 @@ #define OP_AT_S12E0R sys_insn(AT_Op0, 4, AT_CRn, 8, 6) #define OP_AT_S12E0W sys_insn(AT_Op0, 4, AT_CRn, 8, 7) +/* TLBI instructions */ +#define TLBI_Op0 1 +#define TLBI_Op1_EL1 0 /* Accessible from EL1 or higher */ +#define TLBI_Op1_EL2 4 /* Accessible from EL2 or higher */ +#define TLBI_CRn 8 +#define tlbi_insn_el1(CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL1, TLBI_CRn, (CRm), (Op2)) +#define tlbi_insn_el2(CRm, Op2) sys_insn(TLBI_Op0, TLBI_Op1_EL2, TLBI_CRn, (CRm), (Op2)) + +#define OP_TLBI_VMALLE1IS tlbi_insn_el1(3, 0) +#define OP_TLBI_VAE1IS tlbi_insn_el1(3, 1) +#define OP_TLBI_ASIDE1IS tlbi_insn_el1(3, 2) +#define OP_TLBI_VAAE1IS tlbi_insn_el1(3, 3) +#define OP_TLBI_VALE1IS tlbi_insn_el1(3, 5) +#define OP_TLBI_VAALE1IS tlbi_insn_el1(3, 7) +#define OP_TLBI_VMALLE1 tlbi_insn_el1(7, 0) +#define OP_TLBI_VAE1 tlbi_insn_el1(7, 1) +#define OP_TLBI_ASIDE1 tlbi_insn_el1(7, 2) +#define OP_TLBI_VAAE1 tlbi_insn_el1(7, 3) +#define OP_TLBI_VALE1 tlbi_insn_el1(7, 5) +#define OP_TLBI_VAALE1 tlbi_insn_el1(7, 7) + +#define OP_TLBI_IPAS2E1IS tlbi_insn_el2(0, 1) +#define OP_TLBI_IPAS2LE1IS tlbi_insn_el2(0, 5) +#define OP_TLBI_ALLE2IS tlbi_insn_el2(3, 0) +#define OP_TLBI_VAE2IS tlbi_insn_el2(3, 1) +#define OP_TLBI_ALLE1IS tlbi_insn_el2(3, 4) +#define OP_TLBI_VALE2IS tlbi_insn_el2(3, 5) +#define OP_TLBI_VMALLS12E1IS tlbi_insn_el2(3, 6) +#define OP_TLBI_IPAS2E1 tlbi_insn_el2(4, 1) +#define OP_TLBI_IPAS2LE1 tlbi_insn_el2(4, 5) +#define OP_TLBI_ALLE2 tlbi_insn_el2(7, 0) +#define OP_TLBI_VAE2 tlbi_insn_el2(7, 1) +#define OP_TLBI_ALLE1 tlbi_insn_el2(7, 4) +#define OP_TLBI_VALE2 tlbi_insn_el2(7, 5) +#define OP_TLBI_VMALLS12E1 tlbi_insn_el2(7, 6) + /* Common SCTLR_ELx flags. */ #define SCTLR_ELx_DSSBS (BIT(44)) #define SCTLR_ELx_ENIA (BIT(31)) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 61686b8a992e..57e6e848b7a7 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -160,7 +160,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) * the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_NV1; + hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_TTLB | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -190,7 +190,7 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) * trapped too. */ if (!vcpu_el2_tge_is_set(vcpu)) - hcr |= HCR_AT; + hcr |= HCR_AT | HCR_TTLB; } } diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index a0d2bdba9f4e..aa08300774fa 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -237,3 +237,84 @@ void __hyp_text __kvm_flush_vm_context(void) dsb(ish); } + +void __hyp_text __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * Execute the EL1 version of TLBI VAE2* instruction, forcing + * an upgrade to the Inner Shareable domain in order to + * perform the invalidation on all CPUs. + */ + switch (sys_encoding) { + case OP_TLBI_VAE2: + case OP_TLBI_VAE2IS: + __tlbi(vae1is, va); + break; + case OP_TLBI_VALE2: + case OP_TLBI_VALE2IS: + __tlbi(vale1is, va); + break; + default: + break; + } + dsb(ish); + isb(); + + __tlb_switch_to_host(mmu, &cxt); +} + +void __hyp_text __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_encoding) +{ + struct tlb_inv_context cxt; + + dsb(ishst); + + /* Switch to requested VMID */ + __tlb_switch_to_guest(mmu, &cxt); + + /* + * Execute the same instruction as the guest hypervisor did, + * expanding the scope of local TLB invalidations to the Inner + * Shareable domain so that it takes place on all CPUs. This + * is equivalent to having HCR_EL2.FB set. + */ + switch (sys_encoding) { + case OP_TLBI_VMALLE1: + case OP_TLBI_VMALLE1IS: + __tlbi(vmalle1is); + break; + case OP_TLBI_VAE1: + case OP_TLBI_VAE1IS: + __tlbi(vae1is, val); + break; + case OP_TLBI_ASIDE1: + case OP_TLBI_ASIDE1IS: + __tlbi(aside1is, val); + break; + case OP_TLBI_VAAE1: + case OP_TLBI_VAAE1IS: + __tlbi(vaae1is, val); + break; + case OP_TLBI_VALE1: + case OP_TLBI_VALE1IS: + __tlbi(vale1is, val); + break; + case OP_TLBI_VAALE1: + case OP_TLBI_VAALE1IS: + __tlbi(vaale1is, val); + break; + default: + break; + } + dsb(ish); + isb(); + + __tlb_switch_to_host(mmu, &cxt); +} diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 1ff2e16b1f57..527319ff83f1 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1683,6 +1683,11 @@ static bool forward_at_traps(struct kvm_vcpu *vcpu) return forward_traps(vcpu, HCR_AT); } +static bool forward_ttlb_traps(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_TTLB); +} + static bool access_elr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) @@ -2204,6 +2209,189 @@ static bool handle_s12w(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return handle_s12(vcpu, p, r, true); } +static bool handle_alle2is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + /* + * To emulate invalidating all EL2 regime stage 1 TLB entries for all + * PEs, executing TLBI VMALLE1IS is enough. But reuse the existing + * interface for the simplicity; invalidating stage 2 entries doesn't + * affect the correctness. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, &vcpu->kvm->arch.mmu); + return true; +} + +static bool handle_vae2is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + int sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + /* + * Based on the same principle as TLBI ALLE2 instruction + * emulation, we emulate TLBI VAE2* instructions by executing + * corresponding TLBI VAE1* instructions with the virtual + * EL2's VMID assigned by the host hypervisor. + */ + kvm_call_hyp(__kvm_tlb_vae2is, &vcpu->kvm->arch.mmu, + p->regval, sys_encoding); + return true; +} + +static bool handle_alle1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct kvm_s2_mmu *mmu = &vcpu->kvm->arch.mmu; + spin_lock(&vcpu->kvm->mmu_lock); + + /* + * Clear all mappings in the shadow page tables and invalidate the stage + * 1 and 2 TLB entries via kvm_tlb_flush_vmid_ipa(). + */ + kvm_nested_s2_clear(vcpu->kvm); + + if (mmu->vmid.vmid_gen) { + /* + * Invalidate the stage 1 and 2 TLB entries for the host OS + * in a VM only if there is one. + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + } + + spin_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_vmalls12e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; + + spin_lock(&vcpu->kvm->mmu_lock); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); + if (mmu) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(vcpu->kvm)); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); + if (mmu) + kvm_unmap_stage2_range(mmu, 0, kvm_phys_size(vcpu->kvm)); + + spin_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct kvm_s2_mmu *mmu; + u64 base_addr; + int max_size; + + /* + * We drop a number of things from the supplied value: + * + * - NS bit: we're non-secure only. + * + * - TTL field: We already have the granule size from the + * VTCR_EL2.TG0 field, and the level is only relevant to the + * guest's S2PT. + * + * - IPA[51:48]: We don't support 52bit IPA just yet... + * + * And of course, adjust the IPA to be on an actual address. + */ + base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; + + /* Compute the maximum extent of the invalidation */ + switch ((vtcr & VTCR_EL2_TG0_MASK)) { + case VTCR_EL2_TG0_4K: + max_size = SZ_1G; + break; + case VTCR_EL2_TG0_16K: + max_size = SZ_32M; + break; + case VTCR_EL2_TG0_64K: + /* + * No, we do not support 52bit IPA in nested yet. Once + * we do, this should be 4TB. + */ + /* FIXME: remove the 52bit PA support from the IDregs */ + max_size = SZ_512M; + break; + default: + BUG(); + } + + spin_lock(&vcpu->kvm->mmu_lock); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); + if (mmu) + kvm_unmap_stage2_range(mmu, base_addr, max_size); + + mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); + if (mmu) + kvm_unmap_stage2_range(mmu, base_addr, max_size); + + spin_unlock(&vcpu->kvm->mmu_lock); + + return true; +} + +static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + + /* + * If we're here, this is because we've trapped on a EL1 TLBI + * instruction that affects the EL1 translation regime while + * we're running in a context that doesn't allow us to let the + * HW do its thing (aka vEL2): + * + * - HCR_EL2.E2H == 0 : a non-VHE guest + * - HCR_EL2.{E2H,TGE} == { 1, 0 } : a VHE guest in guest mode + * + * We don't expect these helpers to ever be called when running + * in a vEL1 context. + */ + + WARN_ON(!vcpu_mode_el2(vcpu)); + + mutex_lock(&vcpu->kvm->lock); + + if ((__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) { + u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; + + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); + if (mmu) + kvm_call_hyp(__kvm_tlb_el1_instr, + mmu, p->regval, sys_encoding); + + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); + if (mmu) + kvm_call_hyp(__kvm_tlb_el1_instr, + mmu, p->regval, sys_encoding); + } else { + /* + * ARMv8.4-NV allows the guest to change TGE behind + * our back, so we always trap EL1 TLBIs from vEL2... + */ + kvm_call_hyp(__kvm_tlb_el1_instr, + &vcpu->kvm->arch.mmu, p->regval, sys_encoding); + } + + mutex_unlock(&vcpu->kvm->lock); + + return true; +} + /* * AT instruction emulation * @@ -2286,12 +2474,40 @@ static struct sys_reg_desc sys_insn_descs[] = { { SYS_DESC(SYS_DC_CSW), access_dcsw }, { SYS_DESC(SYS_DC_CISW), access_dcsw }, + SYS_INSN_TO_DESC(TLBI_VMALLE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_ASIDE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAAE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VALE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAALE1IS, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VMALLE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_ASIDE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAAE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VALE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(TLBI_VAALE1, handle_tlbi_el1, forward_ttlb_traps), + SYS_INSN_TO_DESC(AT_S1E2R, handle_s1e2, forward_nv_traps), SYS_INSN_TO_DESC(AT_S1E2W, handle_s1e2, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E1R, handle_s12r, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E1W, handle_s12w, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E0R, handle_s12r, forward_nv_traps), SYS_INSN_TO_DESC(AT_S12E0W, handle_s12w, forward_nv_traps), + + SYS_INSN_TO_DESC(TLBI_IPAS2E1IS, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_IPAS2LE1IS, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE2IS, handle_alle2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VAE2IS, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE1IS, handle_alle1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VALE2IS, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VMALLS12E1IS, handle_vmalls12e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_IPAS2E1, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_IPAS2LE1, handle_ipas2e1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE2, handle_alle2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VAE2, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_ALLE1, handle_alle1is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VALE2, handle_vae2is, forward_nv_traps), + SYS_INSN_TO_DESC(TLBI_VMALLS12E1, handle_vmalls12e1is, forward_nv_traps), }; static bool trap_dbgidr(struct kvm_vcpu *vcpu, diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index dc3bded72363..0086b42237af 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -55,7 +55,23 @@ static bool memslot_is_logging(struct kvm_memory_slot *memslot) */ void kvm_flush_remote_tlbs(struct kvm *kvm) { - kvm_call_hyp(__kvm_tlb_flush_vmid, &kvm->arch.mmu); + struct kvm_s2_mmu *mmu = &kvm->arch.mmu; + + if (mmu == &kvm->arch.mmu) { + /* + * For a normal (i.e. non-nested) guest, flush entries for the + * given VMID * + */ + kvm_call_hyp(__kvm_tlb_flush_vmid, mmu); + } else { + /* + * When supporting nested virtualization, we can have multiple + * VMIDs in play for each VCPU in the VM, so it's really not + * worth it to try to quiesce the system and flush all the + * VMIDs that may be in use, instead just nuke the whole thing. + */ + kvm_call_hyp(__kvm_flush_vm_context); + } } static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) From patchwork Tue Feb 11 17:48:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376183 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 155D21395 for ; Tue, 11 Feb 2020 17:51:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E93DB20848 for ; Tue, 11 Feb 2020 17:51:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443502; bh=YQuPy/0PNfjVnioz+da5ic/CAQ3RrhXg+mZ1QDYR2Wg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=CAh1gVRhbhS8acKS2vLY3D3oslMPAHJBBbdyAOpmPST4ggzAC9gBTAdbAal28C47M Ra/VBXpwH9x5Nt0VOmO/LPXSkhdGmaeHq5I0RhJz5W3XdWaWKY5j2lAHtvymAivabW brsC6lJzrXU0JOKqAuYUK2rPKdcptsfL32Bd4pW8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730403AbgBKRvl (ORCPT ); Tue, 11 Feb 2020 12:51:41 -0500 Received: from mail.kernel.org ([198.145.29.99]:54112 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730389AbgBKRvk (ORCPT ); Tue, 11 Feb 2020 12:51:40 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 62399206CC; Tue, 11 Feb 2020 17:51:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443499; bh=YQuPy/0PNfjVnioz+da5ic/CAQ3RrhXg+mZ1QDYR2Wg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b5d6t3QlcYeV+Fzu8I2hdTjJoEtF0B+Gb+oobNZIsE75nEBQdpLVFD4Ko4XWhI9gX W1B/onssv9CejPB+EGHxWA8qPMuUnydyshUWz276ZOEivneFkbdNiGiY3e00GXA4xr crwyxddfdwfjFlrlWUf1gLROjItj61EUWeuoSITY= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfw-004O7k-GG; Tue, 11 Feb 2020 17:50:20 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 46/94] KVM: arm64: nv: Fold guest's HCR_EL2 configuration into the host's Date: Tue, 11 Feb 2020 17:48:50 +0000 Message-Id: <20200211174938.27809-47-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When entering a L2 guest (nested virt enabled, but not in hypervisor context), we need to honor the traps the L1 guest has asked enabled. For now, just OR the guest's HCR_EL2 into the host's. We may have to do some filtering in the future though. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/switch.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 57e6e848b7a7..43e3a60915b1 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -192,6 +192,8 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) if (!vcpu_el2_tge_is_set(vcpu)) hcr |= HCR_AT | HCR_TTLB; } + } else if (nested_virt_in_use(vcpu)) { + hcr |= __vcpu_sys_reg(vcpu, HCR_EL2); } write_sysreg(hcr, hcr_el2); From patchwork Tue Feb 11 17:48:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376143 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 075E91395 for ; Tue, 11 Feb 2020 17:51:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DAD752086A for ; Tue, 11 Feb 2020 17:51:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443464; bh=DnEk8Srn6GGdbVyC4cRzgxf2uqwN2HhfVy7YNKf3UHo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=2ol3pEPbXmfUCjWClChpvM0TsbLBRGVcv4hfO4t1eR2lM7f1uiO5ZxoTUqc2zQL0f vigQxsuTq+d1M33u9ZnGEKDU3xfkKRDLDbS4JKPqLQRm24B8KRdMzvKY0cphHJd9QP rT97whpMg04vQsbZYpwupNNLtlrdCwQCWEnJNyjE= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730080AbgBKRvE (ORCPT ); Tue, 11 Feb 2020 12:51:04 -0500 Received: from mail.kernel.org ([198.145.29.99]:53198 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728935AbgBKRvD (ORCPT ); Tue, 11 Feb 2020 12:51:03 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id F2B2F206D6; Tue, 11 Feb 2020 17:51:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443463; bh=DnEk8Srn6GGdbVyC4cRzgxf2uqwN2HhfVy7YNKf3UHo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=0ZioPLjitsAi9vePoTs52/mSTPgJlD3rFNzudTpx0sSey/TeOsWr4Uy1/hTXAFzEx GKbAXf6qK80yAsJbvJHqa6eBCLhPxHx6T3jW0B+88hvdYaImyn0P1QY8OySJf8JQKM fmajYC14r3vUKvVB/zmJqf/JovUXu+ywDdbjWYFY= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfx-004O7k-1n; Tue, 11 Feb 2020 17:50:21 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 47/94] KVM: arm64: nv: Handle traps for timer _EL02 and _EL2 sysregs accessors Date: Tue, 11 Feb 2020 17:48:51 +0000 Message-Id: <20200211174938.27809-48-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Andre Przywara Add trap handlers for the timer system registers accessed from a guest hypervisors using either _EL02 or _EL2 system register access instructions. Signed-off-by: Andre Przywara Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/sysreg.h | 6 ++++++ arch/arm64/kvm/sys_regs.c | 15 +++++++++++++++ 2 files changed, 21 insertions(+) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index bf465ddde59e..92c97a19369e 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -496,6 +496,12 @@ #define SYS_CNTVOFF_EL2 sys_reg(3, 4, 14, 0, 3) #define SYS_CNTHCTL_EL2 sys_reg(3, 4, 14, 1, 0) +#define SYS_CNTHP_TVAL_EL2 sys_reg(3, 4, 14, 2, 0) +#define SYS_CNTHP_CTL_EL2 sys_reg(3, 4, 14, 2, 1) +#define SYS_CNTHP_CVAL_EL2 sys_reg(3, 4, 14, 2, 2) +#define SYS_CNTHV_TVAL_EL2 sys_reg(3, 4, 14, 3, 0) +#define SYS_CNTHV_CTL_EL2 sys_reg(3, 4, 14, 3, 1) +#define SYS_CNTHV_CVAL_EL2 sys_reg(3, 4, 14, 3, 2) /* VHE encodings for architectural EL0/1 system registers */ #define SYS_SCTLR_EL12 sys_reg(3, 5, 1, 0, 0) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 527319ff83f1..5d9ca3988745 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2078,6 +2078,13 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, + { SYS_DESC(SYS_CNTHP_TVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHP_CTL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHP_CVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_TVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_CTL_EL2), access_arch_timer }, + { SYS_DESC(SYS_CNTHV_CVAL_EL2), access_arch_timer }, + { SYS_DESC(SYS_SCTLR_EL12), access_vm_reg, reset_val, SCTLR_EL1, 0x00C50078 }, { SYS_DESC(SYS_CPACR_EL12), access_rw, reset_val, CPACR_EL1, 0 }, { SYS_DESC(SYS_TTBR0_EL12), access_vm_reg, reset_unknown, TTBR0_EL1 }, @@ -2095,6 +2102,14 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CONTEXTIDR_EL12), access_vm_reg, reset_val, CONTEXTIDR_EL1, 0 }, { SYS_DESC(SYS_CNTKCTL_EL12), access_rw, reset_val, CNTKCTL_EL1, 0 }, + { SYS_DESC(SYS_CNTP_TVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTP_CTL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTP_CVAL_EL02), access_arch_timer }, + + { SYS_DESC(SYS_CNTV_TVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTV_CTL_EL02), access_arch_timer }, + { SYS_DESC(SYS_CNTV_CVAL_EL02), access_arch_timer }, + { SYS_DESC(SYS_SP_EL2), NULL, reset_unknown, SP_EL2 }, }; From patchwork Tue Feb 11 17:48:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376175 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A0B814B4 for ; Tue, 11 Feb 2020 17:51:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1E02520848 for ; Tue, 11 Feb 2020 17:51:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443495; bh=oKOt586Hw/FYzzsaPStgL8cuvs/A9mi+9t8W0g5iYgY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=tEZIXpQN/ZdG8GB18GxX079ci/Ax8+jQmxwrwQf6WW9QafaUgbvYU/Vkr/NacY6f5 2ly9T23Oy/vt+OfRLgm+NJtffjYbe3RKmcYC0x7Zi3LG3o25KTHtdYIDdPO/rXXdef TpkrFxUhUIk/6Z+jq5zzA3QLupJLb9SVgujkcBJU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730356AbgBKRve (ORCPT ); Tue, 11 Feb 2020 12:51:34 -0500 Received: from mail.kernel.org ([198.145.29.99]:53932 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730330AbgBKRvd (ORCPT ); Tue, 11 Feb 2020 12:51:33 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C7BF5206CC; Tue, 11 Feb 2020 17:51:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443492; bh=oKOt586Hw/FYzzsaPStgL8cuvs/A9mi+9t8W0g5iYgY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BEBawLp+JhAhObhrmYPBy07/lO+fYckX9jZBXxsP9IqXOTc/OFHYZYKGFFEQ89/SP LqgkC856/akeQwBWlw+yuVXfYUocAvXhx1g5njblrg1RttWPsFUxL0xqefLKGSI/Ut vUjwWi8HI3l/qXKxKeVkjAg72UbnvYihXVn7tuqA= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfx-004O7k-Kb; Tue, 11 Feb 2020 17:50:21 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 48/94] KVM: arm64: nv: arch_timer: Support hyp timer emulation Date: Tue, 11 Feb 2020 17:48:52 +0000 Message-Id: <20200211174938.27809-49-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Emulating EL2 also means emulating the EL2 timers. To do so, we expand our timer framework to deal with at most 4 timers. At any given time, two timers are using the HW timers, and the two others are purely emulated. The role of deciding which is which at any given time is left to a mapping function which is called every time we need to make such a decision. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_emulate.h | 2 + include/kvm/arm_arch_timer.h | 5 ++ include/kvm/arm_vgic.h | 1 + virt/kvm/arm/arch_timer.c | 124 ++++++++++++++++++++++++++++- virt/kvm/arm/trace.h | 6 +- virt/kvm/arm/vgic/vgic.c | 15 ++++ 6 files changed, 149 insertions(+), 4 deletions(-) diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h index 3944305e81df..bc94dcc83ffa 100644 --- a/arch/arm/include/asm/kvm_emulate.h +++ b/arch/arm/include/asm/kvm_emulate.h @@ -369,4 +369,6 @@ static inline unsigned long vcpu_data_host_to_guest(struct kvm_vcpu *vcpu, static inline void vcpu_ptrauth_setup_lazy(struct kvm_vcpu *vcpu) {} +static inline bool is_hyp_ctxt(struct kvm_vcpu *vcpu) { return false; } + #endif /* __ARM_KVM_EMULATE_H__ */ diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index d120e6c323e7..3a5d9255120e 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -13,6 +13,8 @@ enum kvm_arch_timers { TIMER_PTIMER, TIMER_VTIMER, + TIMER_HVTIMER, + TIMER_HPTIMER, NR_KVM_TIMERS }; @@ -54,6 +56,7 @@ struct arch_timer_context { struct timer_map { struct arch_timer_context *direct_vtimer; struct arch_timer_context *direct_ptimer; + struct arch_timer_context *emul_vtimer; struct arch_timer_context *emul_ptimer; }; @@ -98,6 +101,8 @@ bool kvm_arch_timer_get_input_level(int vintid); #define vcpu_get_timer(v,t) (&vcpu_timer(v)->timers[(t)]) #define vcpu_vtimer(v) (&(v)->arch.timer_cpu.timers[TIMER_VTIMER]) #define vcpu_ptimer(v) (&(v)->arch.timer_cpu.timers[TIMER_PTIMER]) +#define vcpu_hvtimer(v) (&(v)->arch.timer_cpu.timers[TIMER_HVTIMER]) +#define vcpu_hptimer(v) (&(v)->arch.timer_cpu.timers[TIMER_HPTIMER]) #define arch_timer_ctx_index(ctx) ((ctx) - vcpu_timer((ctx)->vcpu)->timers) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 9d53f545a3d5..1b477c662608 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -346,6 +346,7 @@ int kvm_vgic_inject_irq(struct kvm *kvm, int cpuid, unsigned int intid, int kvm_vgic_map_phys_irq(struct kvm_vcpu *vcpu, unsigned int host_irq, u32 vintid, bool (*get_input_level)(int vindid)); int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid); +int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid); bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int vintid); int kvm_vgic_vcpu_pending_irq(struct kvm_vcpu *vcpu); diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c index 0d9438e9de2a..c7edefced838 100644 --- a/virt/kvm/arm/arch_timer.c +++ b/virt/kvm/arm/arch_timer.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -39,6 +40,16 @@ static const struct kvm_irq_level default_vtimer_irq = { .level = 1, }; +static const struct kvm_irq_level default_hptimer_irq = { + .irq = 26, + .level = 1, +}; + +static const struct kvm_irq_level default_hvtimer_irq = { + .irq = 28, + .level = 1, +}; + static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx); static void kvm_timer_update_irq(struct kvm_vcpu *vcpu, bool new_level, struct arch_timer_context *timer_ctx); @@ -58,13 +69,27 @@ u64 kvm_phys_timer_read(void) static void get_timer_map(struct kvm_vcpu *vcpu, struct timer_map *map) { - if (has_vhe()) { + if (nested_virt_in_use(vcpu)) { + if (is_hyp_ctxt(vcpu)) { + map->direct_vtimer = vcpu_hvtimer(vcpu); + map->direct_ptimer = vcpu_hptimer(vcpu); + map->emul_vtimer = vcpu_vtimer(vcpu); + map->emul_ptimer = vcpu_ptimer(vcpu); + } else { + map->direct_vtimer = vcpu_vtimer(vcpu); + map->direct_ptimer = vcpu_ptimer(vcpu); + map->emul_vtimer = vcpu_hvtimer(vcpu); + map->emul_ptimer = vcpu_hptimer(vcpu); + } + } else if (has_vhe()) { map->direct_vtimer = vcpu_vtimer(vcpu); map->direct_ptimer = vcpu_ptimer(vcpu); + map->emul_vtimer = NULL; map->emul_ptimer = NULL; } else { map->direct_vtimer = vcpu_vtimer(vcpu); map->direct_ptimer = NULL; + map->emul_vtimer = NULL; map->emul_ptimer = vcpu_ptimer(vcpu); } @@ -237,9 +262,11 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: cnt_ctl = read_sysreg_el0(SYS_CNTV_CTL); break; case TIMER_PTIMER: + case TIMER_HPTIMER: cnt_ctl = read_sysreg_el0(SYS_CNTP_CTL); break; case NR_KVM_TIMERS: @@ -270,6 +297,7 @@ bool kvm_timer_is_pending(struct kvm_vcpu *vcpu) return kvm_timer_should_fire(map.direct_vtimer) || kvm_timer_should_fire(map.direct_ptimer) || + kvm_timer_should_fire(map.emul_vtimer) || kvm_timer_should_fire(map.emul_ptimer); } @@ -350,6 +378,7 @@ static void timer_save_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: ctx->cnt_ctl = read_sysreg_el0(SYS_CNTV_CTL); ctx->cnt_cval = read_sysreg_el0(SYS_CNTV_CVAL); @@ -359,6 +388,7 @@ static void timer_save_state(struct arch_timer_context *ctx) break; case TIMER_PTIMER: + case TIMER_HPTIMER: ctx->cnt_ctl = read_sysreg_el0(SYS_CNTP_CTL); ctx->cnt_cval = read_sysreg_el0(SYS_CNTP_CVAL); @@ -396,6 +426,7 @@ static void kvm_timer_blocking(struct kvm_vcpu *vcpu) */ if (!kvm_timer_irq_can_fire(map.direct_vtimer) && !kvm_timer_irq_can_fire(map.direct_ptimer) && + !kvm_timer_irq_can_fire(map.emul_vtimer) && !kvm_timer_irq_can_fire(map.emul_ptimer)) return; @@ -429,11 +460,13 @@ static void timer_restore_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: + case TIMER_HVTIMER: write_sysreg_el0(ctx->cnt_cval, SYS_CNTV_CVAL); isb(); write_sysreg_el0(ctx->cnt_ctl, SYS_CNTV_CTL); break; case TIMER_PTIMER: + case TIMER_HPTIMER: write_sysreg_el0(ctx->cnt_cval, SYS_CNTP_CVAL); isb(); write_sysreg_el0(ctx->cnt_ctl, SYS_CNTP_CTL); @@ -520,6 +553,40 @@ static void kvm_timer_vcpu_load_nogic(struct kvm_vcpu *vcpu) enable_percpu_irq(host_vtimer_irq, host_vtimer_irq_flags); } +static void kvm_timer_vcpu_load_nested_switch(struct kvm_vcpu *vcpu, + struct timer_map *map) +{ + int hw, ret; + + if (!irqchip_in_kernel(vcpu->kvm)) + return; + + /* + * We only ever unmap the vtimer irq on a VHE system that runs nested + * virtualization, in which case we have both a valid emul_vtimer, + * emul_ptimer, direct_vtimer, and direct_ptimer. + * + * Since this is called from kvm_timer_vcpu_load(), a change between + * vEL2 and vEL1/0 will have just happened, and the timer_map will + * represent this, and therefore we switch the emul/direct mappings + * below. + */ + hw = kvm_vgic_get_map(vcpu, map->direct_vtimer->irq.irq); + if (hw < 0) { + kvm_vgic_unmap_phys_irq(vcpu, map->emul_vtimer->irq.irq); + kvm_vgic_unmap_phys_irq(vcpu, map->emul_ptimer->irq.irq); + + ret = kvm_vgic_map_phys_irq(vcpu, + map->direct_vtimer->host_timer_irq, + map->direct_vtimer->irq.irq, + kvm_arch_timer_get_input_level); + ret = kvm_vgic_map_phys_irq(vcpu, + map->direct_ptimer->host_timer_irq, + map->direct_ptimer->irq.irq, + kvm_arch_timer_get_input_level); + } +} + void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) { struct arch_timer_cpu *timer = vcpu_timer(vcpu); @@ -531,6 +598,9 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) get_timer_map(vcpu, &map); if (static_branch_likely(&has_gic_active_state)) { + if (nested_virt_in_use(vcpu)) + kvm_timer_vcpu_load_nested_switch(vcpu, &map); + kvm_timer_vcpu_load_gic(map.direct_vtimer); if (map.direct_ptimer) kvm_timer_vcpu_load_gic(map.direct_ptimer); @@ -546,6 +616,8 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) if (map.direct_ptimer) timer_restore_state(map.direct_ptimer); + if (map.emul_vtimer) + timer_emulate(map.emul_vtimer); if (map.emul_ptimer) timer_emulate(map.emul_ptimer); } @@ -590,6 +662,8 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) * In any case, we re-schedule the hrtimer for the physical timer when * coming back to the VCPU thread in kvm_timer_vcpu_load(). */ + if (map.emul_vtimer) + soft_timer_cancel(&map.emul_vtimer->hrtimer); if (map.emul_ptimer) soft_timer_cancel(&map.emul_ptimer->hrtimer); @@ -650,10 +724,14 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) */ vcpu_vtimer(vcpu)->cnt_ctl = 0; vcpu_ptimer(vcpu)->cnt_ctl = 0; + vcpu_hvtimer(vcpu)->cnt_ctl = 0; + vcpu_hptimer(vcpu)->cnt_ctl = 0; if (timer->enabled) { kvm_timer_update_irq(vcpu, false, vcpu_vtimer(vcpu)); kvm_timer_update_irq(vcpu, false, vcpu_ptimer(vcpu)); + kvm_timer_update_irq(vcpu, false, vcpu_hvtimer(vcpu)); + kvm_timer_update_irq(vcpu, false, vcpu_hptimer(vcpu)); if (irqchip_in_kernel(vcpu->kvm)) { kvm_vgic_reset_mapped_irq(vcpu, map.direct_vtimer->irq.irq); @@ -662,6 +740,8 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) } } + if (map.emul_vtimer) + soft_timer_cancel(&map.emul_vtimer->hrtimer); if (map.emul_ptimer) soft_timer_cancel(&map.emul_ptimer->hrtimer); @@ -692,30 +772,47 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) struct arch_timer_cpu *timer = vcpu_timer(vcpu); struct arch_timer_context *vtimer = vcpu_vtimer(vcpu); struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); + struct arch_timer_context *hvtimer = vcpu_hvtimer(vcpu); + struct arch_timer_context *hptimer = vcpu_hptimer(vcpu); /* Synchronize cntvoff across all vtimers of a VM. */ update_vtimer_cntvoff(vcpu, kvm_phys_timer_read()); ptimer->cntvoff = 0; + hvtimer->cntvoff = 0; + hptimer->cntvoff = 0; hrtimer_init(&timer->bg_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); timer->bg_timer.function = kvm_bg_timer_expire; hrtimer_init(&vtimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); hrtimer_init(&ptimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + hrtimer_init(&hvtimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + hrtimer_init(&hptimer->hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); + vtimer->hrtimer.function = kvm_hrtimer_expire; ptimer->hrtimer.function = kvm_hrtimer_expire; + hvtimer->hrtimer.function = kvm_hrtimer_expire; + hptimer->hrtimer.function = kvm_hrtimer_expire; vtimer->irq.irq = default_vtimer_irq.irq; ptimer->irq.irq = default_ptimer_irq.irq; + hvtimer->irq.irq = default_hvtimer_irq.irq; + hptimer->irq.irq = default_hptimer_irq.irq; vtimer->host_timer_irq = host_vtimer_irq; ptimer->host_timer_irq = host_ptimer_irq; + hvtimer->host_timer_irq = host_vtimer_irq; + hptimer->host_timer_irq = host_ptimer_irq; vtimer->host_timer_irq_flags = host_vtimer_irq_flags; ptimer->host_timer_irq_flags = host_ptimer_irq_flags; + hvtimer->host_timer_irq_flags = host_vtimer_irq_flags; + hptimer->host_timer_irq_flags = host_ptimer_irq_flags; vtimer->vcpu = vcpu; ptimer->vcpu = vcpu; + hvtimer->vcpu = vcpu; + hptimer->vcpu = vcpu; } static void kvm_timer_init_interrupt(void *info) @@ -999,7 +1096,7 @@ void kvm_timer_vcpu_terminate(struct kvm_vcpu *vcpu) static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu) { - int vtimer_irq, ptimer_irq; + int vtimer_irq, ptimer_irq, hvtimer_irq, hptimer_irq; int i, ret; vtimer_irq = vcpu_vtimer(vcpu)->irq.irq; @@ -1012,9 +1109,21 @@ static bool timer_irqs_are_valid(struct kvm_vcpu *vcpu) if (ret) return false; + hvtimer_irq = vcpu_hvtimer(vcpu)->irq.irq; + ret = kvm_vgic_set_owner(vcpu, hvtimer_irq, vcpu_hvtimer(vcpu)); + if (ret) + return false; + + hptimer_irq = vcpu_hptimer(vcpu)->irq.irq; + ret = kvm_vgic_set_owner(vcpu, hptimer_irq, vcpu_hptimer(vcpu)); + if (ret) + return false; + kvm_for_each_vcpu(i, vcpu, vcpu->kvm) { if (vcpu_vtimer(vcpu)->irq.irq != vtimer_irq || - vcpu_ptimer(vcpu)->irq.irq != ptimer_irq) + vcpu_ptimer(vcpu)->irq.irq != ptimer_irq || + vcpu_hvtimer(vcpu)->irq.irq != hvtimer_irq || + vcpu_hptimer(vcpu)->irq.irq != hptimer_irq) return false; } @@ -1030,6 +1139,10 @@ bool kvm_arch_timer_get_input_level(int vintid) timer = vcpu_vtimer(vcpu); else if (vintid == vcpu_ptimer(vcpu)->irq.irq) timer = vcpu_ptimer(vcpu); + else if (vintid == vcpu_hvtimer(vcpu)->irq.irq) + timer = vcpu_hvtimer(vcpu); + else if (vintid == vcpu_hptimer(vcpu)->irq.irq) + timer = vcpu_hptimer(vcpu); else BUG(); @@ -1111,6 +1224,7 @@ static void set_timer_irqs(struct kvm *kvm, int vtimer_irq, int ptimer_irq) kvm_for_each_vcpu(i, vcpu, kvm) { vcpu_vtimer(vcpu)->irq.irq = vtimer_irq; vcpu_ptimer(vcpu)->irq.irq = ptimer_irq; + /* TODO: Add support for hv/hp timers */ } } @@ -1121,6 +1235,8 @@ int kvm_arm_timer_set_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) struct arch_timer_context *ptimer = vcpu_ptimer(vcpu); int irq; + /* TODO: Add support for hv/hp timers */ + if (!irqchip_in_kernel(vcpu->kvm)) return -EINVAL; @@ -1153,6 +1269,8 @@ int kvm_arm_timer_get_attr(struct kvm_vcpu *vcpu, struct kvm_device_attr *attr) struct arch_timer_context *timer; int irq; + /* TODO: Add support for hv/hp timers */ + switch (attr->attr) { case KVM_ARM_VCPU_TIMER_IRQ_VTIMER: timer = vcpu_vtimer(vcpu); diff --git a/virt/kvm/arm/trace.h b/virt/kvm/arm/trace.h index 204d210d01c2..3b08cc0376f4 100644 --- a/virt/kvm/arm/trace.h +++ b/virt/kvm/arm/trace.h @@ -271,6 +271,7 @@ TRACE_EVENT(kvm_get_timer_map, __field( unsigned long, vcpu_id ) __field( int, direct_vtimer ) __field( int, direct_ptimer ) + __field( int, emul_vtimer ) __field( int, emul_ptimer ) ), @@ -279,14 +280,17 @@ TRACE_EVENT(kvm_get_timer_map, __entry->direct_vtimer = arch_timer_ctx_index(map->direct_vtimer); __entry->direct_ptimer = (map->direct_ptimer) ? arch_timer_ctx_index(map->direct_ptimer) : -1; + __entry->emul_vtimer = + (map->emul_vtimer) ? arch_timer_ctx_index(map->emul_vtimer) : -1; __entry->emul_ptimer = (map->emul_ptimer) ? arch_timer_ctx_index(map->emul_ptimer) : -1; ), - TP_printk("VCPU: %ld, dv: %d, dp: %d, ep: %d", + TP_printk("VCPU: %ld, dv: %d, dp: %d, ev: %d, ep: %d", __entry->vcpu_id, __entry->direct_vtimer, __entry->direct_ptimer, + __entry->emul_vtimer, __entry->emul_ptimer) ); diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c index 99b02ca730a8..4cbaf0020277 100644 --- a/virt/kvm/arm/vgic/vgic.c +++ b/virt/kvm/arm/vgic/vgic.c @@ -573,6 +573,21 @@ int kvm_vgic_unmap_phys_irq(struct kvm_vcpu *vcpu, unsigned int vintid) return 0; } +int kvm_vgic_get_map(struct kvm_vcpu *vcpu, unsigned int vintid) +{ + struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, vintid); + unsigned long flags; + int ret = -1; + + raw_spin_lock_irqsave(&irq->irq_lock, flags); + if (irq->hw) + ret = irq->hwintid; + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + + vgic_put_irq(vcpu->kvm, irq); + return ret; +} + /** * kvm_vgic_set_owner - Set the owner of an interrupt for a VM * From patchwork Tue Feb 11 17:48:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376145 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F37B414B4 for ; Tue, 11 Feb 2020 17:51:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D21BF21569 for ; Tue, 11 Feb 2020 17:51:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443466; bh=rJBmEKFRZFOWJv/B4wYb9PONOF6F0qnBy4rtT5zDmRA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Q4Jvsb+j1VS0M6J4VeVX+2kt0MvZzRuF/Bcm6KHhd+OV7+NN+11ACWJ6+oQHElTRC tYrI1jGXV3cA7lIoCGNiFe7Jzn+7I5hFx38LBWn2HK6XvtL8MZp9sgTQ0NoVi10AUJ m7t4yoMLmQswHJnivAS5w1GS1/xVAgHtaVKFO1vM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730161AbgBKRvG (ORCPT ); Tue, 11 Feb 2020 12:51:06 -0500 Received: from mail.kernel.org ([198.145.29.99]:53244 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730103AbgBKRvF (ORCPT ); Tue, 11 Feb 2020 12:51:05 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D3D83206CC; Tue, 11 Feb 2020 17:51:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443465; bh=rJBmEKFRZFOWJv/B4wYb9PONOF6F0qnBy4rtT5zDmRA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BqTob0BqhPgzNi5/b9r+TN6JYuiw7rmAanEUHhQiUzVWdVX2DkldPd4AtBiWJvtEf Gr8szpPUauAVh73orHzwprtH1y9k1VxkhzMEjzryJu8vg0BaIUHTBQ4Cy5DCqAT4wk P8dthgUEXxmXo3EZivUgN6Ty9jLHRkG1TyWIWJVk= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfy-004O7k-8f; Tue, 11 Feb 2020 17:50:22 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 49/94] KVM: arm64: nv: Propagate CNTVOFF_EL2 to the virtual EL1 timer Date: Tue, 11 Feb 2020 17:48:53 +0000 Message-Id: <20200211174938.27809-50-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org We need to allow a guest hypervisor to virtualize the virtual timer. FOr that, let's propagate CNTVOFF_EL2 to the guest's view of that timer. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 1 - arch/arm64/kvm/sys_regs.c | 7 ++++++- include/kvm/arm_arch_timer.h | 1 + virt/kvm/arm/arch_timer.c | 8 ++++++++ 4 files changed, 15 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 36bb463dc16a..e35165de51a2 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -247,7 +247,6 @@ enum vcpu_sysreg { RMR_EL2, /* Reset Management Register */ CONTEXTIDR_EL2, /* Context ID Register (EL2) */ TPIDR_EL2, /* EL2 Software Thread ID Register */ - CNTVOFF_EL2, /* Counter-timer Virtual Offset register */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ SP_EL2, /* EL2 Stack Pointer */ diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 5d9ca3988745..7e2553480721 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1373,6 +1373,11 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, tmr = TIMER_PTIMER; treg = TIMER_REG_CVAL; break; + case SYS_CNTVOFF_EL2: + tmr = TIMER_VTIMER; + treg = TIMER_REG_VOFF; + break; + default: BUG(); } @@ -2075,7 +2080,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, - { SYS_DESC(SYS_CNTVOFF_EL2), access_rw, reset_val, CNTVOFF_EL2, 0 }, + { SYS_DESC(SYS_CNTVOFF_EL2), access_arch_timer }, { SYS_DESC(SYS_CNTHCTL_EL2), access_rw, reset_val, CNTHCTL_EL2, 0 }, { SYS_DESC(SYS_CNTHP_TVAL_EL2), access_arch_timer }, diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index 3a5d9255120e..3389606f3029 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -23,6 +23,7 @@ enum kvm_arch_timer_regs { TIMER_REG_CVAL, TIMER_REG_TVAL, TIMER_REG_CTL, + TIMER_REG_VOFF, }; struct arch_timer_context { diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c index c7edefced838..3c88a4c0a296 100644 --- a/virt/kvm/arm/arch_timer.c +++ b/virt/kvm/arm/arch_timer.c @@ -917,6 +917,10 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, val = kvm_phys_timer_read() - timer->cntvoff; break; + case TIMER_REG_VOFF: + val = timer->cntvoff; + break; + default: BUG(); } @@ -959,6 +963,10 @@ static void kvm_arm_timer_write(struct kvm_vcpu *vcpu, timer->cnt_cval = val; break; + case TIMER_REG_VOFF: + timer->cntvoff = val; + break; + default: BUG(); } From patchwork Tue Feb 11 17:48:54 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376227 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B3F3014B4 for ; Tue, 11 Feb 2020 17:52:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 923C4208C3 for ; Tue, 11 Feb 2020 17:52:19 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443539; bh=cczEClMTziRc0+33SLdi7SmaIvaC1h4Lkgq569j/kN0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Y0PETilzUggGgdBrNXfTWof+2VCphDqDA2NYGKTUO3YydBsQdiHPFJDvVygJ7NnAt CW3JgNNja5X2ORiYrAm3Flbh3XNLPK5jnwtSUWNgMHoBSskZmzJuWKOdeZJcYzhEZf 05YmSI03VC00qO3dhkC4dnR8n8i2UvxZVGffKld0= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730487AbgBKRwS (ORCPT ); Tue, 11 Feb 2020 12:52:18 -0500 Received: from mail.kernel.org ([198.145.29.99]:55116 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730515AbgBKRwS (ORCPT ); Tue, 11 Feb 2020 12:52:18 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 94022206CC; Tue, 11 Feb 2020 17:52:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443537; bh=cczEClMTziRc0+33SLdi7SmaIvaC1h4Lkgq569j/kN0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=cUS9QRakDybHd++bvcr9dWqz3wBRKWakiFBlQhOGo7R//P7MnmhMARhGDwm6pJCmU q5JZmbZhVmf6QN83KVupKqLv0+ZQW/ztP2wXX/pQJRGURJohmphTQzCUgZ/7QtKUIW 92YnZwM5fJ8FlAgntgTurAENJ4583WdFqPNS9V40= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfy-004O7k-Rf; Tue, 11 Feb 2020 17:50:22 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 50/94] KVM: arm64: nv: Load timer before the GIC Date: Tue, 11 Feb 2020 17:48:54 +0000 Message-Id: <20200211174938.27809-51-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order for vgic_v3_load_nested to be able to observe which timer interrupts have the HW bit set for the current context, the timers must have been loaded in the new mode and the right timer mapped to their corresponding HW IRQs. At the moment, we load the GIC first, meaning that timer interrupts injected to an L2 guest will never have the HW bit set (we see the old configuration). Swapping the two loads solves this particular problem. Signed-off-by: Marc Zyngier --- virt/kvm/arm/arm.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 9bb5d5f47237..14c2d94c367e 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -344,8 +344,8 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) vcpu->cpu = cpu; vcpu->arch.host_cpu_context = &cpu_data->host_ctxt; - kvm_vgic_load(vcpu); kvm_timer_vcpu_load(vcpu); + kvm_vgic_load(vcpu); kvm_vcpu_load_sysregs(vcpu); kvm_arch_vcpu_load_fp(vcpu); kvm_vcpu_pmu_restore_guest(vcpu); From patchwork Tue Feb 11 17:48:55 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376205 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2C8501395 for ; Tue, 11 Feb 2020 17:52:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EE7ED214DB for ; Tue, 11 Feb 2020 17:52:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443521; bh=jki1RLsu1jQkcwmmm8X8Fk8sx+uqtTR2fwjr2EWmrS4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=sb2SGlIJCVfIqPOjC99lVjY1mZpuL+l8DWi5arpM7i1MUitP3qUaPXCWaP0XJK8xB QSgdAoic2iVXiKDrxE3dpUUB1hiKyk0CBBynPOjkUoIwBiuw1ukrXYzl1Nj0JOxtmV uvUAxeYCOM8iU0l6VUfk1F/YHmu9pzbRske9sQIs= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730470AbgBKRwA (ORCPT ); Tue, 11 Feb 2020 12:52:00 -0500 Received: from mail.kernel.org ([198.145.29.99]:54648 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730345AbgBKRv7 (ORCPT ); Tue, 11 Feb 2020 12:51:59 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4ADF8206CC; Tue, 11 Feb 2020 17:51:58 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443518; bh=jki1RLsu1jQkcwmmm8X8Fk8sx+uqtTR2fwjr2EWmrS4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=M2ZDggBjR6GUKB7o/0PC2vyAs7STB8KsJxW++dKG3GHI6eYtHY2HGJ+rEiN81l+Bd SJwAuKpcO0BGQjwtsWNWk5MRn7tKE3QdXqeXUYK9ql8zq9ukVONKRPO+sPFYN5ye41 0yhF9ohWP7n4jsQKSP8tIwhB3Sv5g8QawVRxWIXA= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfz-004O7k-DE; Tue, 11 Feb 2020 17:50:23 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 51/94] KVM: arm64: nv: vgic-v3: Take cpu_if pointer directly instead of vcpu Date: Tue, 11 Feb 2020 17:48:55 +0000 Message-Id: <20200211174938.27809-52-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall If we move the used_lrs field to the version-specific cpu interface structure, the following functions only operate on the struct vgic_v3_cpu_if and not the full vcpu: __vgic_v3_save_state __vgic_v3_restore_state __vgic_v3_activate_traps __vgic_v3_deactivate_traps __vgic_v3_save_aprs __vgic_v3_restore_aprs This is going to be very useful for nested virt, so move the used_lrs field and change the prototypes and implementations of these functions to take the cpu_if parameter directly. No functional change. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_hyp.h | 12 ++++++------ arch/arm/kvm/hyp/switch.c | 8 ++++---- arch/arm64/include/asm/kvm_hyp.h | 12 ++++++------ arch/arm64/kvm/hyp/switch.c | 8 ++++---- include/kvm/arm_vgic.h | 5 ++++- virt/kvm/arm/hyp/vgic-v3-sr.c | 33 ++++++++++---------------------- virt/kvm/arm/vgic/vgic-v2.c | 10 +++++----- virt/kvm/arm/vgic/vgic-v3.c | 14 ++++++++------ virt/kvm/arm/vgic/vgic.c | 25 ++++++++++++++++-------- 9 files changed, 64 insertions(+), 63 deletions(-) diff --git a/arch/arm/include/asm/kvm_hyp.h b/arch/arm/include/asm/kvm_hyp.h index 3c1b55ecc578..5d4017c4f649 100644 --- a/arch/arm/include/asm/kvm_hyp.h +++ b/arch/arm/include/asm/kvm_hyp.h @@ -103,12 +103,12 @@ void __vgic_v2_restore_state(struct kvm_vcpu *vcpu); void __sysreg_save_state(struct kvm_cpu_context *ctxt); void __sysreg_restore_state(struct kvm_cpu_context *ctxt); -void __vgic_v3_save_state(struct kvm_vcpu *vcpu); -void __vgic_v3_restore_state(struct kvm_vcpu *vcpu); -void __vgic_v3_activate_traps(struct kvm_vcpu *vcpu); -void __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu); -void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu); -void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu); +void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if); asmlinkage void __vfp_save_state(struct vfp_hard_struct *vfp); asmlinkage void __vfp_restore_state(struct vfp_hard_struct *vfp); diff --git a/arch/arm/kvm/hyp/switch.c b/arch/arm/kvm/hyp/switch.c index 0b53e79f3bfb..272facc7f3ed 100644 --- a/arch/arm/kvm/hyp/switch.c +++ b/arch/arm/kvm/hyp/switch.c @@ -79,16 +79,16 @@ static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu) static void __hyp_text __vgic_save_state(struct kvm_vcpu *vcpu) { if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) { - __vgic_v3_save_state(vcpu); - __vgic_v3_deactivate_traps(vcpu); + __vgic_v3_save_state(&vcpu->arch.vgic_cpu.vgic_v3); + __vgic_v3_deactivate_traps(&vcpu->arch.vgic_cpu.vgic_v3); } } static void __hyp_text __vgic_restore_state(struct kvm_vcpu *vcpu) { if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) { - __vgic_v3_activate_traps(vcpu); - __vgic_v3_restore_state(vcpu); + __vgic_v3_activate_traps(&vcpu->arch.vgic_cpu.vgic_v3); + __vgic_v3_restore_state(&vcpu->arch.vgic_cpu.vgic_v3); } } diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 4899a5b74fc3..4c63a2254bf1 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -48,12 +48,12 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu); -void __vgic_v3_save_state(struct kvm_vcpu *vcpu); -void __vgic_v3_restore_state(struct kvm_vcpu *vcpu); -void __vgic_v3_activate_traps(struct kvm_vcpu *vcpu); -void __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu); -void __vgic_v3_save_aprs(struct kvm_vcpu *vcpu); -void __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu); +void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if); +void __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if); int __vgic_v3_perform_cpuif_access(struct kvm_vcpu *vcpu); void __timer_enable_traps(struct kvm_vcpu *vcpu); diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 43e3a60915b1..97c5c1a791b8 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -306,8 +306,8 @@ static void __hyp_text __deactivate_vm(struct kvm_vcpu *vcpu) static void __hyp_text __hyp_vgic_save_state(struct kvm_vcpu *vcpu) { if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) { - __vgic_v3_save_state(vcpu); - __vgic_v3_deactivate_traps(vcpu); + __vgic_v3_save_state(&vcpu->arch.vgic_cpu.vgic_v3); + __vgic_v3_deactivate_traps(&vcpu->arch.vgic_cpu.vgic_v3); } } @@ -315,8 +315,8 @@ static void __hyp_text __hyp_vgic_save_state(struct kvm_vcpu *vcpu) static void __hyp_text __hyp_vgic_restore_state(struct kvm_vcpu *vcpu) { if (static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) { - __vgic_v3_activate_traps(vcpu); - __vgic_v3_restore_state(vcpu); + __vgic_v3_activate_traps(&vcpu->arch.vgic_cpu.vgic_v3); + __vgic_v3_restore_state(&vcpu->arch.vgic_cpu.vgic_v3); } } diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 1b477c662608..f89b81327bb0 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -270,6 +270,8 @@ struct vgic_v2_cpu_if { u32 vgic_vmcr; u32 vgic_apr; u32 vgic_lr[VGIC_V2_MAX_LRS]; + + unsigned int used_lrs; }; struct vgic_v3_cpu_if { @@ -287,6 +289,8 @@ struct vgic_v3_cpu_if { * linking the Linux IRQ subsystem and the ITS together. */ struct its_vpe its_vpe; + + unsigned int used_lrs; }; struct vgic_cpu { @@ -296,7 +300,6 @@ struct vgic_cpu { struct vgic_v3_cpu_if vgic_v3; }; - unsigned int used_lrs; struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS]; raw_spinlock_t ap_list_lock; /* Protects the ap_list */ diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c index ccf1fde9836c..2ea9a0b73fc4 100644 --- a/virt/kvm/arm/hyp/vgic-v3-sr.c +++ b/virt/kvm/arm/hyp/vgic-v3-sr.c @@ -194,10 +194,9 @@ static u32 __hyp_text __vgic_v3_read_ap1rn(int n) return val; } -void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu) +void __hyp_text __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; - u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs; + u64 used_lrs = cpu_if->used_lrs; /* * Make sure stores to the GIC via the memory mapped interface @@ -230,10 +229,9 @@ void __hyp_text __vgic_v3_save_state(struct kvm_vcpu *vcpu) } } -void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu) +void __hyp_text __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; - u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs; + u64 used_lrs = cpu_if->used_lrs; int i; if (used_lrs || cpu_if->its_vpe.its_vm) { @@ -257,10 +255,8 @@ void __hyp_text __vgic_v3_restore_state(struct kvm_vcpu *vcpu) } } -void __hyp_text __vgic_v3_activate_traps(struct kvm_vcpu *vcpu) +void __hyp_text __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; - /* * VFIQEn is RES1 if ICC_SRE_EL1.SRE is 1. This causes a * Group0 interrupt (as generated in GICv2 mode) to be @@ -306,9 +302,8 @@ void __hyp_text __vgic_v3_activate_traps(struct kvm_vcpu *vcpu) write_gicreg(cpu_if->vgic_hcr, ICH_HCR_EL2); } -void __hyp_text __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu) +void __hyp_text __vgic_v3_deactivate_traps(struct vgic_v3_cpu_if *cpu_if) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; u64 val; if (!cpu_if->vgic_sre) { @@ -333,15 +328,11 @@ void __hyp_text __vgic_v3_deactivate_traps(struct kvm_vcpu *vcpu) write_gicreg(0, ICH_HCR_EL2); } -void __hyp_text __vgic_v3_save_aprs(struct kvm_vcpu *vcpu) +void __hyp_text __vgic_v3_save_aprs(struct vgic_v3_cpu_if *cpu_if) { - struct vgic_v3_cpu_if *cpu_if; u64 val; u32 nr_pre_bits; - vcpu = kern_hyp_va(vcpu); - cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; - val = read_gicreg(ICH_VTR_EL2); nr_pre_bits = vtr_to_nr_pre_bits(val); @@ -370,15 +361,11 @@ void __hyp_text __vgic_v3_save_aprs(struct kvm_vcpu *vcpu) } } -void __hyp_text __vgic_v3_restore_aprs(struct kvm_vcpu *vcpu) +void __hyp_text __vgic_v3_restore_aprs(struct vgic_v3_cpu_if *cpu_if) { - struct vgic_v3_cpu_if *cpu_if; u64 val; u32 nr_pre_bits; - vcpu = kern_hyp_va(vcpu); - cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; - val = read_gicreg(ICH_VTR_EL2); nr_pre_bits = vtr_to_nr_pre_bits(val); @@ -453,7 +440,7 @@ static int __hyp_text __vgic_v3_highest_priority_lr(struct kvm_vcpu *vcpu, u32 vmcr, u64 *lr_val) { - unsigned int used_lrs = vcpu->arch.vgic_cpu.used_lrs; + unsigned int used_lrs = vcpu->arch.vgic_cpu.vgic_v3.used_lrs; u8 priority = GICv3_IDLE_PRIORITY; int i, lr = -1; @@ -492,7 +479,7 @@ static int __hyp_text __vgic_v3_highest_priority_lr(struct kvm_vcpu *vcpu, static int __hyp_text __vgic_v3_find_active_lr(struct kvm_vcpu *vcpu, int intid, u64 *lr_val) { - unsigned int used_lrs = vcpu->arch.vgic_cpu.used_lrs; + unsigned int used_lrs = vcpu->arch.vgic_cpu.vgic_v3.used_lrs; int i; for (i = 0; i < used_lrs; i++) { diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c index 621cc168fe3f..ebf53a4e1296 100644 --- a/virt/kvm/arm/vgic/vgic-v2.c +++ b/virt/kvm/arm/vgic/vgic-v2.c @@ -56,7 +56,7 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) cpuif->vgic_hcr &= ~GICH_HCR_UIE; - for (lr = 0; lr < vgic_cpu->used_lrs; lr++) { + for (lr = 0; lr < vgic_cpu->vgic_v2.used_lrs; lr++) { u32 val = cpuif->vgic_lr[lr]; u32 cpuid, intid = val & GICH_LR_VIRTUALID; struct vgic_irq *irq; @@ -120,7 +120,7 @@ void vgic_v2_fold_lr_state(struct kvm_vcpu *vcpu) vgic_put_irq(vcpu->kvm, irq); } - vgic_cpu->used_lrs = 0; + cpuif->used_lrs = 0; } /* @@ -427,7 +427,7 @@ int vgic_v2_probe(const struct gic_kvm_info *info) static void save_lrs(struct kvm_vcpu *vcpu, void __iomem *base) { struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2; - u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs; + u64 used_lrs = cpu_if->used_lrs; u64 elrsr; int i; @@ -448,7 +448,7 @@ static void save_lrs(struct kvm_vcpu *vcpu, void __iomem *base) void vgic_v2_save_state(struct kvm_vcpu *vcpu) { void __iomem *base = kvm_vgic_global_state.vctrl_base; - u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs; + u64 used_lrs = vcpu->arch.vgic_cpu.vgic_v2.used_lrs; if (!base) return; @@ -463,7 +463,7 @@ void vgic_v2_restore_state(struct kvm_vcpu *vcpu) { struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2; void __iomem *base = kvm_vgic_global_state.vctrl_base; - u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs; + u64 used_lrs = cpu_if->used_lrs; int i; if (!base) diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c index f45635a6f0ec..a8732afb37e5 100644 --- a/virt/kvm/arm/vgic/vgic-v3.c +++ b/virt/kvm/arm/vgic/vgic-v3.c @@ -39,7 +39,7 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu) cpuif->vgic_hcr &= ~ICH_HCR_UIE; - for (lr = 0; lr < vgic_cpu->used_lrs; lr++) { + for (lr = 0; lr < cpuif->used_lrs; lr++) { u64 val = cpuif->vgic_lr[lr]; u32 intid, cpuid; struct vgic_irq *irq; @@ -111,7 +111,7 @@ void vgic_v3_fold_lr_state(struct kvm_vcpu *vcpu) vgic_put_irq(vcpu->kvm, irq); } - vgic_cpu->used_lrs = 0; + cpuif->used_lrs = 0; } /* Requires the irq to be locked already */ @@ -660,10 +660,10 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) if (likely(cpu_if->vgic_sre)) kvm_call_hyp(__vgic_v3_write_vmcr, cpu_if->vgic_vmcr); - kvm_call_hyp(__vgic_v3_restore_aprs, vcpu); + kvm_call_hyp(__vgic_v3_restore_aprs, kern_hyp_va(cpu_if)); if (has_vhe()) - __vgic_v3_activate_traps(vcpu); + __vgic_v3_activate_traps(cpu_if); WARN_ON(vgic_v4_load(vcpu)); } @@ -678,12 +678,14 @@ void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu) void vgic_v3_put(struct kvm_vcpu *vcpu) { + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + WARN_ON(vgic_v4_put(vcpu, false)); vgic_v3_vmcr_sync(vcpu); - kvm_call_hyp(__vgic_v3_save_aprs, vcpu); + kvm_call_hyp(__vgic_v3_save_aprs, kern_hyp_va(cpu_if)); if (has_vhe()) - __vgic_v3_deactivate_traps(vcpu); + __vgic_v3_deactivate_traps(cpu_if); } diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c index 4cbaf0020277..a2428b42f7e2 100644 --- a/virt/kvm/arm/vgic/vgic.c +++ b/virt/kvm/arm/vgic/vgic.c @@ -801,6 +801,7 @@ static void vgic_flush_lr_state(struct kvm_vcpu *vcpu) int count; bool multi_sgi; u8 prio = 0xff; + int i = 0; lockdep_assert_held(&vgic_cpu->ap_list_lock); @@ -842,11 +843,14 @@ static void vgic_flush_lr_state(struct kvm_vcpu *vcpu) } } - vcpu->arch.vgic_cpu.used_lrs = count; - /* Nuke remaining LRs */ - for ( ; count < kvm_vgic_global_state.nr_lr; count++) - vgic_clear_lr(vcpu, count); + for (i = count ; i < kvm_vgic_global_state.nr_lr; i++) + vgic_clear_lr(vcpu, i); + + if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) + vcpu->arch.vgic_cpu.vgic_v2.used_lrs = count; + else + vcpu->arch.vgic_cpu.vgic_v3.used_lrs = count; } static inline bool can_access_vgic_from_kernel(void) @@ -864,13 +868,13 @@ static inline void vgic_save_state(struct kvm_vcpu *vcpu) if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) vgic_v2_save_state(vcpu); else - __vgic_v3_save_state(vcpu); + __vgic_v3_save_state(&vcpu->arch.vgic_cpu.vgic_v3); } /* Sync back the hardware VGIC state into our emulation after a guest's run. */ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + int used_lrs; /* An empty ap_list_head implies used_lrs == 0 */ if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) @@ -879,7 +883,12 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) if (can_access_vgic_from_kernel()) vgic_save_state(vcpu); - if (vgic_cpu->used_lrs) + if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) + used_lrs = vcpu->arch.vgic_cpu.vgic_v2.used_lrs; + else + used_lrs = vcpu->arch.vgic_cpu.vgic_v3.used_lrs; + + if (used_lrs) vgic_fold_lr_state(vcpu); vgic_prune_ap_list(vcpu); } @@ -889,7 +898,7 @@ static inline void vgic_restore_state(struct kvm_vcpu *vcpu) if (!static_branch_unlikely(&kvm_vgic_global_state.gicv3_cpuif)) vgic_v2_restore_state(vcpu); else - __vgic_v3_restore_state(vcpu); + __vgic_v3_restore_state(&vcpu->arch.vgic_cpu.vgic_v3); } /* Flush our emulation state into the GIC hardware before entering the guest. */ From patchwork Tue Feb 11 17:48:56 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376219 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6371214E3 for ; Tue, 11 Feb 2020 17:52:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2792321569 for ; Tue, 11 Feb 2020 17:52:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443533; bh=O0LXgpJRzdmk6QndFG0WFYftv+sdhp3laQXxQVUiFVQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=S39bSzW0RwqNdr8IB7PrxK6CihqI5WNlMLkcW9eXUCiw5f18Npd/frFD63mekneEf vCqFKoTy1D6+F7BH2czRnLpRNdipWIplYvUcHyPgG3caZVEI6mOOryIOCiGkUU4n4g 3SLYollJ929fH3hXWwkZFkGoJ4kHBz7PqkWqpCjM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730481AbgBKRwL (ORCPT ); Tue, 11 Feb 2020 12:52:11 -0500 Received: from mail.kernel.org ([198.145.29.99]:54870 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730502AbgBKRwK (ORCPT ); Tue, 11 Feb 2020 12:52:10 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 2BD99215A4; Tue, 11 Feb 2020 17:52:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443528; bh=O0LXgpJRzdmk6QndFG0WFYftv+sdhp3laQXxQVUiFVQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=xyjyiPq9k/4d7jw8zb4U4EDwxvCK0VaEyfq8iK4cqMnQ5DdoZS0k8/R04toSTAyhg F/dXZey6CzDggda5nzC3RSxJGuQx9KE3lR/olxQJMVT4Ps7JOZjoYQDRoz1lzgiVYF Oh2hUqWmLcneSUTVdz8bp4ApD+0pw3mh+33BRoOU= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zfz-004O7k-WF; Tue, 11 Feb 2020 17:50:24 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 52/94] KVM: arm64: nv: Nested GICv3 Support Date: Tue, 11 Feb 2020 17:48:56 +0000 Message-Id: <20200211174938.27809-53-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Jintack Lim When entering a nested VM, we set up the hypervisor control interface based on what the guest hypervisor has set. Especially, we investigate each list register written by the guest hypervisor whether HW bit is set. If so, we translate hw irq number from the guest's point of view to the real hardware irq number if there is a mapping. Signed-off-by: Jintack Lim [Rewritten to support GICv3 instead of GICv2] Signed-off-by: Marc Zyngier [Redesigned execution flow around vcpu load/put] Signed-off-by: Christoffer Dall --- arch/arm/include/asm/kvm_emulate.h | 1 + arch/arm/include/asm/kvm_host.h | 7 +- arch/arm/include/asm/kvm_nested.h | 1 + arch/arm64/include/asm/kvm_host.h | 7 +- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/nested.c | 16 +++ arch/arm64/kvm/sys_regs.c | 178 +++++++++++++++++++++++++++- include/kvm/arm_vgic.h | 18 +++ virt/kvm/arm/arm.c | 13 +- virt/kvm/arm/vgic/vgic-v3-nested.c | 178 ++++++++++++++++++++++++++++ virt/kvm/arm/vgic/vgic-v3.c | 31 +++++ virt/kvm/arm/vgic/vgic.c | 32 +++++ 13 files changed, 475 insertions(+), 9 deletions(-) create mode 100644 virt/kvm/arm/vgic/vgic-v3-nested.c diff --git a/arch/arm/include/asm/kvm_emulate.h b/arch/arm/include/asm/kvm_emulate.h index bc94dcc83ffa..60d851f75830 100644 --- a/arch/arm/include/asm/kvm_emulate.h +++ b/arch/arm/include/asm/kvm_emulate.h @@ -370,5 +370,6 @@ static inline unsigned long vcpu_data_host_to_guest(struct kvm_vcpu *vcpu, static inline void vcpu_ptrauth_setup_lazy(struct kvm_vcpu *vcpu) {} static inline bool is_hyp_ctxt(struct kvm_vcpu *vcpu) { return false; } +static inline int kvm_inject_nested_irq(struct kvm_vcpu *vcpu) { BUG(); } #endif /* __ARM_KVM_EMULATE_H__ */ diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h index cb5e9b37a87a..c398f1ce117a 100644 --- a/arch/arm/include/asm/kvm_host.h +++ b/arch/arm/include/asm/kvm_host.h @@ -34,11 +34,12 @@ #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS #endif +/* KVM_REQ_GUEST_HYP_IRQ_PENDING is actually unused */ #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) -#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) -#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) +#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) +#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use); diff --git a/arch/arm/include/asm/kvm_nested.h b/arch/arm/include/asm/kvm_nested.h index 124ff6445f8f..2b89e6fa7323 100644 --- a/arch/arm/include/asm/kvm_nested.h +++ b/arch/arm/include/asm/kvm_nested.h @@ -5,5 +5,6 @@ #include static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) { return false; } +static inline void check_nested_vcpu_requests(struct kvm_vcpu *vcpu) {} #endif /* __ARM_KVM_NESTED_H */ diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e35165de51a2..c578ab9b4a02 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -41,9 +41,10 @@ #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) -#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) -#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) -#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) +#define KVM_REQ_IRQ_PENDING KVM_ARCH_REQ(1) +#define KVM_REQ_VCPU_RESET KVM_ARCH_REQ(2) +#define KVM_REQ_RECORD_STEAL KVM_ARCH_REQ(3) +#define KVM_REQ_GUEST_HYP_IRQ_PENDING KVM_ARCH_REQ(4) DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use); diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 147943d876ef..45c013c0c856 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -16,6 +16,7 @@ extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu); extern struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr); extern void kvm_vcpu_load_hw_mmu(struct kvm_vcpu *vcpu); extern void kvm_vcpu_put_hw_mmu(struct kvm_vcpu *vcpu); +extern void check_nested_vcpu_requests(struct kvm_vcpu *vcpu); struct kvm_s2_trans { phys_addr_t output; diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 5bd4775b44bc..4521fdfacef0 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -40,3 +40,4 @@ kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o kvm-$(CONFIG_KVM_ARM_HOST) += nested.o kvm-$(CONFIG_KVM_ARM_HOST) += emulate-nested.o +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3-nested.o diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index d20f9982ffea..0ab08ee59110 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -580,6 +580,22 @@ void kvm_arch_flush_shadow_all(struct kvm *kvm) kvm_free_stage2_pgd(&kvm->arch.mmu); } +bool vgic_state_is_nested(struct kvm_vcpu *vcpu) +{ + bool imo = __vcpu_sys_reg(vcpu, HCR_EL2) & HCR_IMO; + bool fmo = __vcpu_sys_reg(vcpu, HCR_EL2) & HCR_FMO; + + WARN_ONCE(imo != fmo, "Separate virtual IRQ/FIQ settings not supported\n"); + + return nested_virt_in_use(vcpu) && imo && fmo && !is_hyp_ctxt(vcpu); +} + +void check_nested_vcpu_requests(struct kvm_vcpu *vcpu) +{ + if (kvm_check_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu)) + kvm_inject_nested_irq(vcpu); +} + #define FEATURE(x) (GENMASK_ULL(x##_SHIFT + 3, x##_SHIFT)) /* diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 7e2553480721..c1e856cb84f1 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -15,6 +15,8 @@ #include #include +#include + #include #include #include @@ -521,6 +523,18 @@ static bool access_vm_reg(struct kvm_vcpu *vcpu, return true; } +/* + * The architecture says that non-secure write accesses to this register from + * EL1 are trapped to EL2, if either: + * - HCR_EL2.FMO==1, or + * - HCR_EL2.IMO==1 + */ +static bool sgi_traps_to_vel2(struct kvm_vcpu *vcpu) +{ + return !vcpu_mode_el2(vcpu) && + !!(__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_IMO | HCR_FMO)); +} + /* * Trap handler for the GICv3 SGI generation system register. * Forward the request to the VGIC emulation. @@ -536,6 +550,11 @@ static bool access_gic_sgi(struct kvm_vcpu *vcpu, if (!p->is_write) return read_from_write_only(vcpu, p, r); + if (sgi_traps_to_vel2(vcpu)) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_hsr(vcpu)); + return false; + } + /* * In a system where GICD_CTLR.DS=1, a ICC_SGI0R_EL1 access generates * Group0 SGIs only, while ICC_SGI1R_EL1 can generate either group, @@ -579,7 +598,13 @@ static bool access_gic_sre(struct kvm_vcpu *vcpu, if (p->is_write) return ignore_write(vcpu, p); - p->regval = vcpu->arch.vgic_cpu.vgic_v3.vgic_sre; + if (p->Op1 == 4) { /* ICC_SRE_EL2 */ + p->regval = (ICC_SRE_EL2_ENABLE | ICC_SRE_EL2_SRE | + ICC_SRE_EL1_DIB | ICC_SRE_EL1_DFB); + } else { /* ICC_SRE_EL1 */ + p->regval = vcpu->arch.vgic_cpu.vgic_v3.vgic_sre; + } + return true; } @@ -1747,6 +1772,122 @@ static bool access_spsr_el2(struct kvm_vcpu *vcpu, return true; } +static bool access_gic_apr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + u32 index, *base; + + index = r->Op2; + if (r->CRm == 8) + base = cpu_if->vgic_ap0r; + else + base = cpu_if->vgic_ap1r; + + if (p->is_write) + base[index] = p->regval; + else + p->regval = base[index]; + + return true; +} + +static bool access_gic_hcr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + + if (p->is_write) + cpu_if->vgic_hcr = p->regval; + else + p->regval = cpu_if->vgic_hcr; + + return true; +} + +static bool access_gic_vtr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = kvm_vgic_global_state.ich_vtr_el2; + + return true; +} + +static bool access_gic_misr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_misr(vcpu); + + return true; +} + +static bool access_gic_eisr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_eisr(vcpu); + + return true; +} + +static bool access_gic_elrsr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (p->is_write) + return write_to_read_only(vcpu, p, r); + + p->regval = vgic_v3_get_elrsr(vcpu); + + return true; +} + +static bool access_gic_vmcr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + + if (p->is_write) + cpu_if->vgic_vmcr = p->regval; + else + p->regval = cpu_if->vgic_vmcr; + + return true; +} + +static bool access_gic_lr(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + u32 index; + + index = p->Op2; + if (p->CRm == 13) + index += 8; + + if (p->is_write) + cpu_if->vgic_lr[index] = p->regval; + else + p->regval = cpu_if->vgic_lr[index]; + + return true; +} + /* * Architected system registers. * Important: Must be sorted ascending by Op0, Op1, CRn, CRm, Op2 @@ -2077,6 +2218,41 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_RMR_EL2), access_wi, reset_val, RMR_EL2, 1 }, { SYS_DESC(SYS_VDISR_EL2), trap_undef }, + { SYS_DESC(SYS_ICH_AP0R0_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R1_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R2_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP0R3_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R0_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R1_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R2_EL2), access_gic_apr }, + { SYS_DESC(SYS_ICH_AP1R3_EL2), access_gic_apr }, + + { SYS_DESC(SYS_ICC_SRE_EL2), access_gic_sre }, + + { SYS_DESC(SYS_ICH_HCR_EL2), access_gic_hcr }, + { SYS_DESC(SYS_ICH_VTR_EL2), access_gic_vtr }, + { SYS_DESC(SYS_ICH_MISR_EL2), access_gic_misr }, + { SYS_DESC(SYS_ICH_EISR_EL2), access_gic_eisr }, + { SYS_DESC(SYS_ICH_ELRSR_EL2), access_gic_elrsr }, + { SYS_DESC(SYS_ICH_VMCR_EL2), access_gic_vmcr }, + + { SYS_DESC(SYS_ICH_LR0_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR1_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR2_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR3_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR4_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR5_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR6_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR7_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR8_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR9_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR10_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR11_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR12_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR13_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR14_EL2), access_gic_lr }, + { SYS_DESC(SYS_ICH_LR15_EL2), access_gic_lr }, + { SYS_DESC(SYS_CONTEXTIDR_EL2), access_rw, reset_val, CONTEXTIDR_EL2, 0 }, { SYS_DESC(SYS_TPIDR_EL2), access_rw, reset_val, TPIDR_EL2, 0 }, diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index f89b81327bb0..61e8d5cb852b 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -302,6 +302,15 @@ struct vgic_cpu { struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS]; + /* CPU vif control registers for the virtual GICH interface */ + struct vgic_v3_cpu_if nested_vgic_v3; + + /* + * The shadow vif control register loaded to the hardware when + * running a nested L2 guest with the virtual IMO/FMO bit set. + */ + struct vgic_v3_cpu_if shadow_vgic_v3; + raw_spinlock_t ap_list_lock; /* Protects the ap_list */ /* @@ -358,6 +367,13 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); +void vgic_v3_load_nested(struct kvm_vcpu *vcpu); +void vgic_v3_put_nested(struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); +u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu); +u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu); +u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu); + #define irqchip_in_kernel(k) (!!((k)->arch.vgic.in_kernel)) #define vgic_initialized(k) ((k)->arch.vgic.initialized) #define vgic_ready(k) ((k)->arch.vgic.ready) @@ -401,4 +417,6 @@ int kvm_vgic_v4_unset_forwarding(struct kvm *kvm, int irq, int vgic_v4_load(struct kvm_vcpu *vcpu); int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db); +bool vgic_state_is_nested(struct kvm_vcpu *vcpu); + #endif /* __KVM_ARM_VGIC_H */ diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index 14c2d94c367e..fa73b7a464f1 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -620,6 +620,8 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu) if (kvm_check_request(KVM_REQ_RECORD_STEAL, vcpu)) kvm_update_stolen_time(vcpu); + + check_nested_vcpu_requests(vcpu); } } @@ -666,9 +668,16 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) */ cond_resched(); - update_vmid(&vcpu->arch.hw_mmu->vmid); - + /* + * A nested exeption triggered by a vcpu request (such + * as an interrupt injected in a guest hypervisor) can + * change the currently used VMID (by switching to a + * different translation regime. It is thus necesary + * to update the VMID *after* all requests have been + * processed. + */ check_vcpu_requests(vcpu); + update_vmid(&vcpu->arch.hw_mmu->vmid); /* * Preparing the interrupts to be injected also diff --git a/virt/kvm/arm/vgic/vgic-v3-nested.c b/virt/kvm/arm/vgic/vgic-v3-nested.c new file mode 100644 index 000000000000..bfe0ccde514f --- /dev/null +++ b/virt/kvm/arm/vgic/vgic-v3-nested.c @@ -0,0 +1,178 @@ +#include +#include +#include +#include +#include +#include + +#include + +#include +#include +#include + +#include "vgic.h" + +static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.vgic_cpu.nested_vgic_v3; +} + +static inline struct vgic_v3_cpu_if *vcpu_shadow_if(struct kvm_vcpu *vcpu) +{ + return &vcpu->arch.vgic_cpu.shadow_vgic_v3; +} + +static inline bool lr_triggers_eoi(u64 lr) +{ + return !(lr & (ICH_LR_STATE | ICH_LR_HW)) && (lr & ICH_LR_EOI); +} + +u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + u16 reg = 0; + int i; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + if (lr_triggers_eoi(cpu_if->vgic_lr[i])) + reg |= BIT(i); + } + + return reg; +} + +u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + u16 reg = 0; + int i; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + if (!(cpu_if->vgic_lr[i] & ICH_LR_STATE)) + reg |= BIT(i); + } + + return reg; +} + +u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + int nr_lr = kvm_vgic_global_state.nr_lr; + u64 reg = 0; + + if (vgic_v3_get_eisr(vcpu)) + reg |= ICH_MISR_EOI; + + if (cpu_if->vgic_hcr & ICH_HCR_UIE) { + int used_lrs; + + used_lrs = nr_lr - hweight16(vgic_v3_get_elrsr(vcpu)); + if (used_lrs <= 1) + reg |= ICH_MISR_U; + } + + /* TODO: Support remaining bits in this register */ + return reg; +} + +/* + * For LRs which have HW bit set such as timer interrupts, we modify them to + * have the host hardware interrupt number instead of the virtual one programmed + * by the guest hypervisor. + */ +static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + struct vgic_irq *irq; + int i, used_lrs = 0; + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + u64 lr = cpu_if->vgic_lr[i]; + int l1_irq; + + if (!(lr & ICH_LR_HW)) + goto next; + + /* We have the HW bit set */ + l1_irq = (lr & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT; + irq = vgic_get_irq(vcpu->kvm, vcpu, l1_irq); + + if (!irq || !irq->hw) { + /* There was no real mapping, so nuke the HW bit */ + lr &= ~ICH_LR_HW; + if (irq) + vgic_put_irq(vcpu->kvm, irq); + goto next; + } + + /* Translate the virtual mapping to the real one */ + lr &= ~ICH_LR_EOI; /* Why? */ + lr &= ~ICH_LR_PHYS_ID_MASK; + lr |= (u64)irq->hwintid << ICH_LR_PHYS_ID_SHIFT; + vgic_put_irq(vcpu->kvm, irq); + +next: + s_cpu_if->vgic_lr[i] = lr; + used_lrs = i + 1; + } + + s_cpu_if->used_lrs = used_lrs; +} + +/* + * Change the shadow HWIRQ field back to the virtual value before copying over + * the entire shadow struct to the nested state. + */ +static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + int lr; + + for (lr = 0; lr < kvm_vgic_global_state.nr_lr; lr++) { + s_cpu_if->vgic_lr[lr] &= ~ICH_LR_PHYS_ID_MASK; + s_cpu_if->vgic_lr[lr] |= cpu_if->vgic_lr[lr] & ICH_LR_PHYS_ID_MASK; + } +} + +void vgic_v3_load_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + + vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; + vgic_v3_create_shadow_lr(vcpu); + __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); +} + +void vgic_v3_put_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + + __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + + /* + * Translate the shadow state HW fields back to the virtual ones + * before copying the shadow struct back to the nested one. + */ + vgic_v3_fixup_shadow_lr_state(vcpu); + vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; +} + +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + + /* + * If we exit a nested VM with a pending maintenance interrupt from the + * GIC, then we need to forward this to the guest hypervisor so that it + * can re-sync the appropriate LRs and sample level triggered interrupts + * again. + */ + if (vgic_state_is_nested(vcpu) && + (cpu_if->vgic_hcr & ICH_HCR_EN) && + vgic_v3_get_misr(vcpu)) + kvm_inject_nested_irq(vcpu); +} diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c index a8732afb37e5..bca20528d5d1 100644 --- a/virt/kvm/arm/vgic/vgic-v3.c +++ b/virt/kvm/arm/vgic/vgic-v3.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include "vgic.h" @@ -289,6 +290,12 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) vgic_v3->vgic_sre = (ICC_SRE_EL1_DIB | ICC_SRE_EL1_DFB | ICC_SRE_EL1_SRE); + /* + * If nesting is allowed, force GICv3 onto the nested + * guests as well. + */ + if (nested_virt_in_use(vcpu)) + vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE; } else { vgic_v3->vgic_sre = 0; @@ -652,6 +659,13 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + /* + * vgic_v3_load_nested only affects the LRs in the shadow + * state, so it is fine to pass the nested state around. + */ + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + /* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen * is dependent on ICC_SRE_EL1.SRE, and we have to perform the @@ -665,6 +679,9 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) if (has_vhe()) __vgic_v3_activate_traps(cpu_if); + if (vgic_state_is_nested(vcpu)) + vgic_v3_load_nested(vcpu); + WARN_ON(vgic_v4_load(vcpu)); } @@ -672,6 +689,9 @@ void vgic_v3_vmcr_sync(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + if (likely(cpu_if->vgic_sre)) cpu_if->vgic_vmcr = kvm_call_hyp_ret(__vgic_v3_read_vmcr); } @@ -684,8 +704,19 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) vgic_v3_vmcr_sync(vcpu); + if (vgic_state_is_nested(vcpu)) + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + kvm_call_hyp(__vgic_v3_save_aprs, kern_hyp_va(cpu_if)); if (has_vhe()) __vgic_v3_deactivate_traps(cpu_if); + + if (vgic_state_is_nested(vcpu)) + vgic_v3_put_nested(vcpu); } + +__weak void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) {} +__weak void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) {} +__weak void vgic_v3_load_nested(struct kvm_vcpu *vcpu) {} +__weak void vgic_v3_put_nested(struct kvm_vcpu *vcpu) {} diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c index a2428b42f7e2..7be3b05dcd8d 100644 --- a/virt/kvm/arm/vgic/vgic.c +++ b/virt/kvm/arm/vgic/vgic.c @@ -876,6 +876,10 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) { int used_lrs; + /* If nesting, this is a load/put affair, not flush/sync. */ + if (vgic_state_is_nested(vcpu)) + return; + /* An empty ap_list_head implies used_lrs == 0 */ if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) return; @@ -920,6 +924,29 @@ void kvm_vgic_flush_hwstate(struct kvm_vcpu *vcpu) !vgic_supports_direct_msis(vcpu->kvm)) return; + /* + * If in a nested state, we must return early. Two possibilities: + * + * - If we have any pending IRQ for the guest and the guest + * expects IRQs to be handled in its virtual EL2 mode (the + * virtual IMO bit is set) and it is not already running in + * virtual EL2 mode, then we have to emulate an IRQ + * exception to virtual EL2. + * + * We do that by placing a request to ourselves which will + * abort the entry procedure and inject the exception at the + * beginning of the run loop. + * + * - Otherwise, do exactly *NOTHING*. The guest state is + * already loaded, and we can carry on with running it. + */ + if (vgic_state_is_nested(vcpu)) { + if (kvm_vgic_vcpu_pending_irq(vcpu)) + kvm_make_request(KVM_REQ_GUEST_HYP_IRQ_PENDING, vcpu); + + return; + } + DEBUG_SPINLOCK_BUG_ON(!irqs_disabled()); if (!list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) { @@ -1033,3 +1060,8 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int vintid) return map_is_active; } + +__weak bool vgic_state_is_nested(struct kvm_vcpu *vcpu) +{ + return false; +} From patchwork Tue Feb 11 17:48:57 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376245 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8205E14B4 for ; Tue, 11 Feb 2020 17:52:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 61F81214DB for ; Tue, 11 Feb 2020 17:52:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443556; bh=n91T9PiB+KAQWxH9dGtjS6W4D+t/xuG08bn74uAGMJ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=iTrbbPQ8LfYsL+erWpYJ2T5IQwJH+4jYavgI+QZQbZfHpVbe6WLOIv4Yaw9rMWnMa qQ5ivPB4CCd4gEzc6ANc34os/n40SMql3qSEpfOj9pZSj2DVNFTW+3pahpSnllX0KE aAeHBxnRtcuaQm18g1W41jkpKlKr3k4oG/W052kw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730576AbgBKRwf (ORCPT ); Tue, 11 Feb 2020 12:52:35 -0500 Received: from mail.kernel.org ([198.145.29.99]:55600 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730555AbgBKRwf (ORCPT ); Tue, 11 Feb 2020 12:52:35 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 76D2920578; Tue, 11 Feb 2020 17:52:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443554; bh=n91T9PiB+KAQWxH9dGtjS6W4D+t/xuG08bn74uAGMJ4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=zP+SM/odg1YVn7h9rLnBXYsUwkbhjNE+WtTz9zGNP8UpmZLa5B/3Kp85c7lQE3DHH BcJl6MjIgCQStSezTHDrvB85YMLDXTlN3QwFOznRHIgHAbpLZHe3lXt3tBOt/ApU8Y Ra316iFZHnKPE8+NAAK4a8DFjYcRbYaBsilYva58= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg0-004O7k-Lc; Tue, 11 Feb 2020 17:50:24 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 53/94] KVM: arm64: nv: vgic: Emulate the HW bit in software Date: Tue, 11 Feb 2020 17:48:57 +0000 Message-Id: <20200211174938.27809-54-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Should the guest hypervisor use the HW bit in the LRs, we need to emulate the deactivation from the L2 guest into the L1 distributor emulation, which is handled by L0. It's all good fun. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_hyp.h | 2 ++ include/kvm/arm_vgic.h | 1 + virt/kvm/arm/hyp/vgic-v3-sr.c | 2 +- virt/kvm/arm/vgic/vgic-v3-nested.c | 32 ++++++++++++++++++++++++++++++ virt/kvm/arm/vgic/vgic.c | 6 ++++-- 5 files changed, 40 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 4c63a2254bf1..de9c184d68a5 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -48,6 +48,8 @@ int __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu); +u64 __hyp_text __gic_v3_get_lr(unsigned int lr); + void __vgic_v3_save_state(struct vgic_v3_cpu_if *cpu_if); void __vgic_v3_restore_state(struct vgic_v3_cpu_if *cpu_if); void __vgic_v3_activate_traps(struct vgic_v3_cpu_if *cpu_if); diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index 61e8d5cb852b..c6e3634979b4 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -367,6 +367,7 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); void vgic_v3_load_nested(struct kvm_vcpu *vcpu); void vgic_v3_put_nested(struct kvm_vcpu *vcpu); void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); diff --git a/virt/kvm/arm/hyp/vgic-v3-sr.c b/virt/kvm/arm/hyp/vgic-v3-sr.c index 2ea9a0b73fc4..8562f628a7c8 100644 --- a/virt/kvm/arm/hyp/vgic-v3-sr.c +++ b/virt/kvm/arm/hyp/vgic-v3-sr.c @@ -16,7 +16,7 @@ #define vtr_to_nr_pre_bits(v) ((((u32)(v) >> 26) & 7) + 1) #define vtr_to_nr_apr_regs(v) (1 << (vtr_to_nr_pre_bits(v) - 5)) -static u64 __hyp_text __gic_v3_get_lr(unsigned int lr) +u64 __hyp_text __gic_v3_get_lr(unsigned int lr) { switch (lr & 0xf) { case 0: diff --git a/virt/kvm/arm/vgic/vgic-v3-nested.c b/virt/kvm/arm/vgic/vgic-v3-nested.c index bfe0ccde514f..3ee3438b5e22 100644 --- a/virt/kvm/arm/vgic/vgic-v3-nested.c +++ b/virt/kvm/arm/vgic/vgic-v3-nested.c @@ -138,6 +138,38 @@ static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) } } +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + struct vgic_irq *irq; + int i; + + for (i = 0; i < s_cpu_if->used_lrs; i++) { + u64 lr = cpu_if->vgic_lr[i]; + int l1_irq; + + if (!(lr & ICH_LR_HW) || !(lr & ICH_LR_STATE)) + continue; + + /* + * If we had a HW lr programmed by the guest hypervisor, we + * need to emulate the HW effect between the guest hypervisor + * and the nested guest. + */ + l1_irq = (lr & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT; + irq = vgic_get_irq(vcpu->kvm, vcpu, l1_irq); + if (!irq) + continue; /* oh well, the guest hyp is broken */ + + lr = __gic_v3_get_lr(i); + if (!(lr & ICH_LR_STATE)) + irq->active = false; + + vgic_put_irq(vcpu->kvm, irq); + } +} + void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c index 7be3b05dcd8d..dc2bf047556e 100644 --- a/virt/kvm/arm/vgic/vgic.c +++ b/virt/kvm/arm/vgic/vgic.c @@ -876,9 +876,11 @@ void kvm_vgic_sync_hwstate(struct kvm_vcpu *vcpu) { int used_lrs; - /* If nesting, this is a load/put affair, not flush/sync. */ - if (vgic_state_is_nested(vcpu)) + /* If nesting, emulate the HW effect from L0 to L1 */ + if (vgic_state_is_nested(vcpu)) { + vgic_v3_sync_nested(vcpu); return; + } /* An empty ap_list_head implies used_lrs == 0 */ if (list_empty(&vcpu->arch.vgic_cpu.ap_list_head)) From patchwork Tue Feb 11 17:48:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376285 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 60DB114B4 for ; Tue, 11 Feb 2020 17:53:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3EDFE20848 for ; Tue, 11 Feb 2020 17:53:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443591; bh=xK/bh22Y+Z2nnV8xAUZKINd9y3tRtuZ9gOlEJEphpMI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=KxBxLVQY+gCxQ4KldxAOKuT6YYbiC6oAEoXJJQE0LwtsFqsVkvCfP02ttLHFw7Zba lvfoilKaqH0UqQc1booYZffvwWpSIsgoUv5ksjnyowzSeKJ83a3ZdtLjxdW1c6lxyz XX5I17MlvsKEXSHXpjMO2F/18tGwzXuJzYLTRD/Q= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730570AbgBKRxK (ORCPT ); Tue, 11 Feb 2020 12:53:10 -0500 Received: from mail.kernel.org ([198.145.29.99]:56534 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730656AbgBKRxJ (ORCPT ); Tue, 11 Feb 2020 12:53:09 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5B16A206CC; Tue, 11 Feb 2020 17:53:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443588; bh=xK/bh22Y+Z2nnV8xAUZKINd9y3tRtuZ9gOlEJEphpMI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=DKG9Cwl4C9+cCqQjJqTePHQQ3c8GBaVeLuVkRxfOsltYIt/c2ooWxwYaG3tv2+8YO TcJXuZ3UbbTVyAFH2B1FVw+QNTP9aHWB1liFFZ0ehFzFX8P1m3hHxBttPDRYZr7mkZ 8/1FVJN6IzeV+XVVwmKJqE0hXRCmGKTDuO6+ZIK8= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg1-004O7k-7g; Tue, 11 Feb 2020 17:50:25 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 54/94] KVM: arm64: nv: vgic: Allow userland to set VGIC maintenance IRQ Date: Tue, 11 Feb 2020 17:48:58 +0000 Message-Id: <20200211174938.27809-55-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Andre Przywara The VGIC maintenance IRQ signals various conditions about the LRs, when the GIC's virtualization extension is used. So far we didn't need it, but nested virtualization needs to know about this interrupt, so add a userland interface to setup the IRQ number. The architecture mandates that it must be a PPI, on top of that this code only exports a per-device option, so the PPI is the same on all VCPUs. Signed-off-by: Andre Przywara [added some bits of documentation] Signed-off-by: Marc Zyngier --- .../virt/kvm/devices/arm-vgic-v3.txt | 8 +++++++ arch/arm/include/uapi/asm/kvm.h | 1 + arch/arm64/include/uapi/asm/kvm.h | 1 + include/kvm/arm_vgic.h | 3 +++ virt/kvm/arm/vgic/vgic-kvm-device.c | 22 +++++++++++++++++++ 5 files changed, 35 insertions(+) diff --git a/Documentation/virt/kvm/devices/arm-vgic-v3.txt b/Documentation/virt/kvm/devices/arm-vgic-v3.txt index ff290b43c8e5..3af2fb4d3095 100644 --- a/Documentation/virt/kvm/devices/arm-vgic-v3.txt +++ b/Documentation/virt/kvm/devices/arm-vgic-v3.txt @@ -249,3 +249,11 @@ Groups: Errors: -EINVAL: vINTID is not multiple of 32 or info field is not VGIC_LEVEL_INFO_LINE_LEVEL + + KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ + The attr field of kvm_device_attr encodes the following values: + bits: | 31 .... 5 | 4 .... 0 | + values: | RES0 | vINTID | + + The vINTID specifies which interrupt is generated when the vGIC + must generate a maintenance interrupt. This must be a PPI. diff --git a/arch/arm/include/uapi/asm/kvm.h b/arch/arm/include/uapi/asm/kvm.h index 03cd7c19a683..d5dd96902817 100644 --- a/arch/arm/include/uapi/asm/kvm.h +++ b/arch/arm/include/uapi/asm/kvm.h @@ -246,6 +246,7 @@ struct kvm_vcpu_events { #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h index 5b9c58a55537..4e4016e05fc6 100644 --- a/arch/arm64/include/uapi/asm/kvm.h +++ b/arch/arm64/include/uapi/asm/kvm.h @@ -314,6 +314,7 @@ struct kvm_vcpu_events { #define KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS 6 #define KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO 7 #define KVM_DEV_ARM_VGIC_GRP_ITS_REGS 8 +#define KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ 9 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT 10 #define KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK \ (0x3fffffULL << KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_SHIFT) diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index c6e3634979b4..b5c026a7e0ee 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -218,6 +218,9 @@ struct vgic_dist { int nr_spis; + /* The GIC maintenance IRQ for nested hypervisors. */ + u32 maint_irq; + /* base addresses in guest physical address space: */ gpa_t vgic_dist_base; /* distributor */ union { diff --git a/virt/kvm/arm/vgic/vgic-kvm-device.c b/virt/kvm/arm/vgic/vgic-kvm-device.c index 44419679f91a..dfb1d7cc66b3 100644 --- a/virt/kvm/arm/vgic/vgic-kvm-device.c +++ b/virt/kvm/arm/vgic/vgic-kvm-device.c @@ -241,6 +241,12 @@ static int vgic_get_common_attr(struct kvm_device *dev, VGIC_NR_PRIVATE_IRQS, uaddr); break; } + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: { + u32 __user *uaddr = (u32 __user *)(long)attr->addr; + + r = put_user(dev->kvm->arch.vgic.maint_irq, uaddr); + break; + } } return r; @@ -627,6 +633,21 @@ static int vgic_v3_set_attr(struct kvm_device *dev, reg = tmp32; return vgic_v3_attr_regs_access(dev, attr, ®, true); } + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: { + u32 __user *uaddr = (u32 __user *)(long)attr->addr; + u32 val; + + if (get_user(val, uaddr)) + return -EFAULT; + + /* Must be a PPI. */ + if ((val >= VGIC_NR_PRIVATE_IRQS) || (val < VGIC_NR_SGIS)) + return -EINVAL; + + dev->kvm->arch.vgic.maint_irq = val; + + return 0; + } case KVM_DEV_ARM_VGIC_GRP_CTRL: { int ret; @@ -712,6 +733,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev, case KVM_DEV_ARM_VGIC_GRP_CPU_SYSREGS: return vgic_v3_has_attr_regs(dev, attr); case KVM_DEV_ARM_VGIC_GRP_NR_IRQS: + case KVM_DEV_ARM_VGIC_GRP_MAINT_IRQ: return 0; case KVM_DEV_ARM_VGIC_GRP_LEVEL_INFO: { if (((attr->attr & KVM_DEV_ARM_VGIC_LINE_LEVEL_INFO_MASK) >> From patchwork Tue Feb 11 17:48:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376207 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 43A6D14B4 for ; Tue, 11 Feb 2020 17:52:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 24751208C3 for ; Tue, 11 Feb 2020 17:52:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443523; bh=IPlG3pg4CO1MgYK1b39GRc1ZDWAa8g8OanDrvwP2yV4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=aIKCHvrQsNpsOujlLcx2tNCwvULVz08QtZ5YIlspHqVyrjNrDNsSBvVjxrsWLjUdu H+yxpEcsLeiatwqSB2OolXnjKMnOu4qjTY3gxwtzR2OG1hZ8T3PLOtlMErNDh86I+r QNG3wRsZ2k2NyjDRpv8vCBlXt3edjSm1ME9lat/8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728895AbgBKRwC (ORCPT ); Tue, 11 Feb 2020 12:52:02 -0500 Received: from mail.kernel.org ([198.145.29.99]:54686 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730473AbgBKRwB (ORCPT ); Tue, 11 Feb 2020 12:52:01 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5B08A206D7; Tue, 11 Feb 2020 17:52:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443520; bh=IPlG3pg4CO1MgYK1b39GRc1ZDWAa8g8OanDrvwP2yV4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ayEM2m4QxvREt/HdGPA4aUx9hHxBnfxlskN/4mrUPGbNaZ8ySX+BuGmc+GBonImUs Sz6C2mwdVMOOvCu60HWJPST4ChozpLrzGKI4hlghNTt+qZ8ppk4owfa0Sg+Qu357OF kHume/zfnpXlEyjb4f6jWOJx6t4L0HOgdaKLLZgA= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg1-004O7k-Qr; Tue, 11 Feb 2020 17:50:25 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 55/94] KVM: arm64: nv: Implement maintenance interrupt forwarding Date: Tue, 11 Feb 2020 17:48:59 +0000 Message-Id: <20200211174938.27809-56-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When we take a maintenance interrupt, we need to decide whether it is generated on an action from the guest, or if it is something that needs to be forwarded to the guest hypervisor. Signed-off-by: Marc Zyngier --- virt/kvm/arm/vgic/vgic-init.c | 30 ++++++++++++++++++++++++++++++ virt/kvm/arm/vgic/vgic-v3-nested.c | 25 +++++++++++++++++++++---- 2 files changed, 51 insertions(+), 4 deletions(-) diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c index a963b9d766b7..191b284e7bee 100644 --- a/virt/kvm/arm/vgic/vgic-init.c +++ b/virt/kvm/arm/vgic/vgic-init.c @@ -6,10 +6,12 @@ #include #include #include +#include #include #include #include #include +#include #include "vgic.h" /* @@ -221,6 +223,16 @@ int kvm_vgic_vcpu_init(struct kvm_vcpu *vcpu) if (!irqchip_in_kernel(vcpu->kvm)) return 0; + if (nested_virt_in_use(vcpu)) { + /* FIXME: remove this hack */ + if (vcpu->kvm->arch.vgic.maint_irq == 0) + vcpu->kvm->arch.vgic.maint_irq = kvm_vgic_global_state.maint_irq; + ret = kvm_vgic_set_owner(vcpu, vcpu->kvm->arch.vgic.maint_irq, + vcpu); + if (ret) + return ret; + } + /* * If we are creating a VCPU with a GICv3 we must also register the * KVM io device for the redistributor that belongs to this VCPU. @@ -449,12 +461,23 @@ static int vgic_init_cpu_dying(unsigned int cpu) static irqreturn_t vgic_maintenance_handler(int irq, void *data) { + struct kvm_vcpu *vcpu = *(struct kvm_vcpu **)data; + /* * We cannot rely on the vgic maintenance interrupt to be * delivered synchronously. This means we can only use it to * exit the VM, and we perform the handling of EOIed * interrupts on the exit path (see vgic_fold_lr_state). */ + + /* If not nested, deactivate */ + if (!vcpu || !vgic_state_is_nested(vcpu)) { + irq_set_irqchip_state(irq, IRQCHIP_STATE_ACTIVE, false); + return IRQ_HANDLED; + } + + /* Assume nested from now */ + vgic_v3_handle_nested_maint_irq(vcpu); return IRQ_HANDLED; } @@ -525,6 +548,13 @@ int kvm_vgic_hyp_init(void) return ret; } + ret = irq_set_vcpu_affinity(kvm_vgic_global_state.maint_irq, + kvm_get_running_vcpus()); + if (ret) { + kvm_err("Error setting vcpu affinity\n"); + goto out_free_irq; + } + ret = cpuhp_setup_state(CPUHP_AP_KVM_ARM_VGIC_INIT_STARTING, "kvm/arm/vgic:starting", vgic_init_cpu_starting, vgic_init_cpu_dying); diff --git a/virt/kvm/arm/vgic/vgic-v3-nested.c b/virt/kvm/arm/vgic/vgic-v3-nested.c index 3ee3438b5e22..4ba426e2324d 100644 --- a/virt/kvm/arm/vgic/vgic-v3-nested.c +++ b/virt/kvm/arm/vgic/vgic-v3-nested.c @@ -173,10 +173,20 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + struct vgic_irq *irq; + unsigned long flags; vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; vgic_v3_create_shadow_lr(vcpu); __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); + + irq = vgic_get_irq(vcpu->kvm, vcpu, vcpu->kvm->arch.vgic.maint_irq); + raw_spin_lock_irqsave(&irq->irq_lock, flags); + if (irq->line_level || irq->active) + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, + IRQCHIP_STATE_ACTIVE, true); + raw_spin_unlock_irqrestore(&irq->irq_lock, flags); + vgic_put_irq(vcpu->kvm, irq); } void vgic_v3_put_nested(struct kvm_vcpu *vcpu) @@ -191,11 +201,14 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu) */ vgic_v3_fixup_shadow_lr_state(vcpu); vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, + IRQCHIP_STATE_ACTIVE, false); } void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) { struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); + bool state; /* * If we exit a nested VM with a pending maintenance interrupt from the @@ -203,8 +216,12 @@ void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) * can re-sync the appropriate LRs and sample level triggered interrupts * again. */ - if (vgic_state_is_nested(vcpu) && - (cpu_if->vgic_hcr & ICH_HCR_EN) && - vgic_v3_get_misr(vcpu)) - kvm_inject_nested_irq(vcpu); + if (!vgic_state_is_nested(vcpu)) + return; + + state = cpu_if->vgic_hcr & ICH_HCR_EN; + state &= vgic_v3_get_misr(vcpu); + + kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id, + vcpu->kvm->arch.vgic.maint_irq, state, vcpu); } From patchwork Tue Feb 11 17:49:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376257 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 49C3814B4 for ; Tue, 11 Feb 2020 17:52:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 200ED20870 for ; Tue, 11 Feb 2020 17:52:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443566; bh=Ft3LS22pw1Kczdq6lr0DMRdJJ3bt2weBSDVOe2zbADY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=oYGzAkcCN7bv8v6jVvoDzCbrUJKEtlT2DyPES/ZuD0jUKI1EOhjuWn7XnBXp/drDh 93aJLPfOFkuSxpvMrueghqSI1YqSM7Xc+ZoEU0dFiIKqhTqI9QFRwaQqOZXf2SSJOf l264y8ZBrjpABZx/DJ1sJOvRnN2mhLbJpqL8x9P8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730589AbgBKRwp (ORCPT ); Tue, 11 Feb 2020 12:52:45 -0500 Received: from mail.kernel.org ([198.145.29.99]:55902 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730519AbgBKRwo (ORCPT ); Tue, 11 Feb 2020 12:52:44 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B6D2220848; Tue, 11 Feb 2020 17:52:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443563; bh=Ft3LS22pw1Kczdq6lr0DMRdJJ3bt2weBSDVOe2zbADY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nhZ5Wq0N2gKx2XcrZ+FaYLw48Lsw4mbJkFYC31jYwpaq16ETNX+cQ7rTCJGTZ567G GQ9oWQ+ayDUcBA2+dwCTZdgqgNsulwyXTepqpj9g4kzWSC7K5wwAS7DCiDcBlTIP3+ 5UV/5pcuWDwuBLAdGWO4OuUhVKRMdtC84JT6ufJo= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg2-004O7k-D6; Tue, 11 Feb 2020 17:50:26 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 56/94] KVM: arm64: nv: Add nested GICv3 tracepoints Date: Tue, 11 Feb 2020 17:49:00 +0000 Message-Id: <20200211174938.27809-57-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Adding tracepoints to be able to peek into the shadow LRs used when running a guest guest. Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- virt/kvm/arm/vgic/vgic-nested-trace.h | 137 ++++++++++++++++++++++++++ virt/kvm/arm/vgic/vgic-v3-nested.c | 13 ++- 2 files changed, 149 insertions(+), 1 deletion(-) create mode 100644 virt/kvm/arm/vgic/vgic-nested-trace.h diff --git a/virt/kvm/arm/vgic/vgic-nested-trace.h b/virt/kvm/arm/vgic/vgic-nested-trace.h new file mode 100644 index 000000000000..69f4ec031e7c --- /dev/null +++ b/virt/kvm/arm/vgic/vgic-nested-trace.h @@ -0,0 +1,137 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#if !defined(_TRACE_VGIC_NESTED_H) || defined(TRACE_HEADER_MULTI_READ) +#define _TRACE_VGIC_NESTED_H + +#include + +#undef TRACE_SYSTEM +#define TRACE_SYSTEM kvm + +#define SLR_ENTRY_VALS(x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT, \ + (__entry->orig_lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT(vgic_create_shadow_lrs, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs, u64 *orig_lrs), + TP_ARGS(vcpu, nr_lr, lrs, orig_lrs), + + TP_STRUCT__entry( + __field( int, nr_lr ) + __array( u64, lrs, 16 ) + __array( u64, orig_lrs, 16 ) + ), + + TP_fast_assign( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + memcpy(__entry->orig_lrs, orig_lrs, 16 * sizeof(u64)); + ), + + TP_printk("nr_lr: %d\n" + "%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)\n" + "%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu (%5llu)", + __entry->nr_lr, + SLR_ENTRY_VALS(0), SLR_ENTRY_VALS(1), SLR_ENTRY_VALS(2), + SLR_ENTRY_VALS(3), SLR_ENTRY_VALS(4), SLR_ENTRY_VALS(5), + SLR_ENTRY_VALS(6), SLR_ENTRY_VALS(7), SLR_ENTRY_VALS(8), + SLR_ENTRY_VALS(9), SLR_ENTRY_VALS(10), SLR_ENTRY_VALS(11), + SLR_ENTRY_VALS(12), SLR_ENTRY_VALS(13), SLR_ENTRY_VALS(14), + SLR_ENTRY_VALS(15)) +); + +#define LR_ENTRY_VALS(x) \ + " ", \ + !!(__entry->lrs[x] & ICH_LR_HW), \ + !!(__entry->lrs[x] & ICH_LR_PENDING_BIT), \ + !!(__entry->lrs[x] & ICH_LR_ACTIVE_BIT), \ + __entry->lrs[x] & ICH_LR_VIRTUAL_ID_MASK, \ + (__entry->lrs[x] & ICH_LR_PHYS_ID_MASK) >> ICH_LR_PHYS_ID_SHIFT + +TRACE_EVENT(vgic_put_nested, + TP_PROTO(struct kvm_vcpu *vcpu, int nr_lr, u64 *lrs), + TP_ARGS(vcpu, nr_lr, lrs), + + TP_STRUCT__entry( + __field( int, nr_lr ) + __array( u64, lrs, 16 ) + ), + + TP_fast_assign( + __entry->nr_lr = nr_lr; + memcpy(__entry->lrs, lrs, 16 * sizeof(u64)); + ), + + TP_printk("nr_lr: %d\n" + "%50sLR[ 0]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 1]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 2]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 3]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 4]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 5]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 6]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 7]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 8]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[ 9]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[10]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[11]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[12]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[13]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[14]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu\n" + "%50sLR[15]: HW: %d P: %d: A: %d vINTID: %5llu pINTID: %5llu", + __entry->nr_lr, + LR_ENTRY_VALS(0), LR_ENTRY_VALS(1), LR_ENTRY_VALS(2), + LR_ENTRY_VALS(3), LR_ENTRY_VALS(4), LR_ENTRY_VALS(5), + LR_ENTRY_VALS(6), LR_ENTRY_VALS(7), LR_ENTRY_VALS(8), + LR_ENTRY_VALS(9), LR_ENTRY_VALS(10), LR_ENTRY_VALS(11), + LR_ENTRY_VALS(12), LR_ENTRY_VALS(13), LR_ENTRY_VALS(14), + LR_ENTRY_VALS(15)) +); + +TRACE_EVENT(vgic_nested_hw_emulate, + TP_PROTO(int lr, u64 lr_val, u32 l1_intid), + TP_ARGS(lr, lr_val, l1_intid), + + TP_STRUCT__entry( + __field( int, lr ) + __field( u64, lr_val ) + __field( u32, l1_intid ) + ), + + TP_fast_assign( + __entry->lr = lr; + __entry->lr_val = lr_val; + __entry->l1_intid = l1_intid; + ), + + TP_printk("lr: %d LR %llx L1 INTID: %u\n", + __entry->lr, __entry->lr_val, __entry->l1_intid) +); + +#endif /* _TRACE_VGIC_NESTED_H */ + +#undef TRACE_INCLUDE_PATH +#define TRACE_INCLUDE_PATH ../../../virt/kvm/arm/vgic +#undef TRACE_INCLUDE_FILE +#define TRACE_INCLUDE_FILE vgic-nested-trace + +/* This part must be outside protection */ +#include diff --git a/virt/kvm/arm/vgic/vgic-v3-nested.c b/virt/kvm/arm/vgic/vgic-v3-nested.c index 4ba426e2324d..94b1edb67011 100644 --- a/virt/kvm/arm/vgic/vgic-v3-nested.c +++ b/virt/kvm/arm/vgic/vgic-v3-nested.c @@ -13,6 +13,9 @@ #include "vgic.h" +#define CREATE_TRACE_POINTS +#include "vgic-nested-trace.h" + static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) { return &vcpu->arch.vgic_cpu.nested_vgic_v3; @@ -119,6 +122,9 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) used_lrs = i + 1; } + trace_vgic_create_shadow_lrs(vcpu, kvm_vgic_global_state.nr_lr, + s_cpu_if->vgic_lr, cpu_if->vgic_lr); + s_cpu_if->used_lrs = used_lrs; } @@ -163,8 +169,10 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) continue; /* oh well, the guest hyp is broken */ lr = __gic_v3_get_lr(i); - if (!(lr & ICH_LR_STATE)) + if (!(lr & ICH_LR_STATE)) { + trace_vgic_nested_hw_emulate(i, lr, l1_irq); irq->active = false; + } vgic_put_irq(vcpu->kvm, irq); } @@ -195,6 +203,9 @@ void vgic_v3_put_nested(struct kvm_vcpu *vcpu) __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, + vcpu_shadow_if(vcpu)->vgic_lr); + /* * Translate the shadow state HW fields back to the virtual ones * before copying the shadow struct back to the nested one. From patchwork Tue Feb 11 17:49:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376215 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5DE801395 for ; Tue, 11 Feb 2020 17:52:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3DC0920870 for ; Tue, 11 Feb 2020 17:52:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443532; bh=qk6x1d4NRUqw3DKQMLEUfcyqYegsG9M/hW2zPBecrX4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=jno3pqpsm0NI0WMiw2kZAzSQi3YK0lBMx5qjtu6qIuc/Boes8kj6I3EbnTu7IwJIA 6FF5hPp6x4HKNmk+bCbEX0YxjHc3g1h0V0z45BFlbAwKEJGljAq70BVdH9W/GbMRlu f+eXhJ3knh0+MrWvShPM/fY/13qZT5rfd5EAOYIQ= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730505AbgBKRwL (ORCPT ); Tue, 11 Feb 2020 12:52:11 -0500 Received: from mail.kernel.org ([198.145.29.99]:54910 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727966AbgBKRwL (ORCPT ); Tue, 11 Feb 2020 12:52:11 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 222C6206CC; Tue, 11 Feb 2020 17:52:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443530; bh=qk6x1d4NRUqw3DKQMLEUfcyqYegsG9M/hW2zPBecrX4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GsF0UjKrFzsPxCokJJaiQu0DtQz79/SS2LrJiBodRo4EB5+K/WEskOxtF6yh/gXBZ eMKq2wTbq36MAzUGDxisxGWxYJTufuzT9GvFH3u6FVl75/ij+IiNVR3xtefjggIdKH LGY9BXe6XiKQt/QxN7yMaZvT93DixEPxZApau50A= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg2-004O7k-US; Tue, 11 Feb 2020 17:50:27 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 57/94] arm64: KVM: nv: Add handling of EL2-specific timer registers Date: Tue, 11 Feb 2020 17:49:01 +0000 Message-Id: <20200211174938.27809-58-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add the required handling for EL2 and EL02 registers, as well as EL1 registers used in the E2H context. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 72 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 72 insertions(+) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index c1e856cb84f1..c62080d7742c 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1384,20 +1384,92 @@ static bool access_arch_timer(struct kvm_vcpu *vcpu, switch (reg) { case SYS_CNTP_TVAL_EL0: + if (vcpu_mode_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_TVAL; + break; + case SYS_AARCH32_CNTP_TVAL: + case SYS_CNTP_TVAL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_TVAL; break; + + case SYS_CNTV_TVAL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_TVAL; + break; + + case SYS_CNTHP_TVAL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_TVAL; + break; + + case SYS_CNTHV_TVAL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_TVAL; + break; + case SYS_CNTP_CTL_EL0: + if (vcpu_mode_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_CTL; + break; + case SYS_AARCH32_CNTP_CTL: + case SYS_CNTP_CTL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_CTL; break; + + case SYS_CNTV_CTL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_CTL; + break; + + case SYS_CNTHP_CTL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_CTL; + break; + + case SYS_CNTHV_CTL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_CTL; + break; + case SYS_CNTP_CVAL_EL0: + if (vcpu_mode_el2(vcpu) && vcpu_el2_e2h_is_set(vcpu)) + tmr = TIMER_HPTIMER; + else + tmr = TIMER_PTIMER; + treg = TIMER_REG_CVAL; + break; + case SYS_AARCH32_CNTP_CVAL: + case SYS_CNTP_CVAL_EL02: tmr = TIMER_PTIMER; treg = TIMER_REG_CVAL; break; + + case SYS_CNTV_CVAL_EL02: + tmr = TIMER_VTIMER; + treg = TIMER_REG_CVAL; + break; + + case SYS_CNTHP_CVAL_EL2: + tmr = TIMER_HPTIMER; + treg = TIMER_REG_CVAL; + break; + + case SYS_CNTHV_CVAL_EL2: + tmr = TIMER_HVTIMER; + treg = TIMER_REG_CVAL; + break; + case SYS_CNTVOFF_EL2: tmr = TIMER_VTIMER; treg = TIMER_REG_VOFF; From patchwork Tue Feb 11 17:49:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376187 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AF67714B4 for ; Tue, 11 Feb 2020 17:51:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9007D208C3 for ; Tue, 11 Feb 2020 17:51:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443505; bh=0Meg/3iJbL3MBzT1+yZerX/G5rQIdjlezfw/2Tu1x/0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=1Ae2zZUXbMohiCFTfEPGY70soQzYg8UPVnv0hADApJW+QdRsTmwUe9RZ0mZGRGCLe 0x+MA8swOYyVwWjYMXxsecDhbTGdtfUzeI8TWeOmq2eqLwa4jrTdVXI04J6mnUARjL T0HkHlrv2eo3K/Htg3KAuJvYJU++eeQAplF9uhKI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730388AbgBKRvo (ORCPT ); Tue, 11 Feb 2020 12:51:44 -0500 Received: from mail.kernel.org ([198.145.29.99]:54224 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730372AbgBKRvo (ORCPT ); Tue, 11 Feb 2020 12:51:44 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 67EB020578; Tue, 11 Feb 2020 17:51:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443503; bh=0Meg/3iJbL3MBzT1+yZerX/G5rQIdjlezfw/2Tu1x/0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=nDyF35cfeJLCZ1Exk84bNpVljETTKER1JLv+mpo8rnEe0BM1e2vTSn9itXPtIpwZN l7uwqcL1Otu37w3PjTpiIXwErNaRkKxuFPseraE3p+7JhWt5zGNR8U4JIuEYybhTDx ULHECoCu/uY1cM6HREUzmNHEFBQAGr/OIZJVS4KE= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg3-004O7k-Gv; Tue, 11 Feb 2020 17:50:27 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 58/94] arm64: KVM: nv: Honor SCTLR_EL2.SPAN on entering vEL2 Date: Tue, 11 Feb 2020 17:49:02 +0000 Message-Id: <20200211174938.27809-59-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On entering vEL2, we must honor the SCTLR_EL2.SPAN bit so that PSTATE.PAN reflect the expected setting. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/emulate-nested.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 8380ed810062..97af080b9b52 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -131,9 +131,11 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) static void enter_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2, enum exception_type type) { + u64 spsr = *vcpu_cpsr(vcpu); + trace_kvm_inject_nested_exception(vcpu, esr_el2, type); - vcpu_write_sys_reg(vcpu, *vcpu_cpsr(vcpu), SPSR_EL2); + vcpu_write_sys_reg(vcpu, spsr, SPSR_EL2); vcpu_write_sys_reg(vcpu, *vcpu_pc(vcpu), ELR_EL2); vcpu_write_sys_reg(vcpu, esr_el2, ESR_EL2); @@ -141,6 +143,15 @@ static void enter_el2_exception(struct kvm_vcpu *vcpu, u64 esr_el2, /* On an exception, PSTATE.SP becomes 1 */ *vcpu_cpsr(vcpu) = PSR_MODE_EL2h; *vcpu_cpsr(vcpu) |= PSR_A_BIT | PSR_F_BIT | PSR_I_BIT | PSR_D_BIT; + + /* + * If SPAN is clear, set the PAN bit on exception entry + * if SPAN is set, copy the PAN bit across + */ + if (!(vcpu_read_sys_reg(vcpu, SCTLR_EL2) & SCTLR_EL1_SPAN)) + *vcpu_cpsr(vcpu) |= PSR_PAN_BIT; + else + *vcpu_cpsr(vcpu) |= (spsr & PSR_PAN_BIT); } /* From patchwork Tue Feb 11 17:49:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376253 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0DF781395 for ; Tue, 11 Feb 2020 17:52:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E3376215A4 for ; Tue, 11 Feb 2020 17:52:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443562; bh=hU+8ZXqCJKxOobeRNU4DXaABOcyKqWOi8gVrtQCKmB4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=UI5zQycNUqLTJAk/lF1tmMhtpWmAxUXW9hXByt8catejSJOHYxRyrPtKdTSHimvdz 689hKVxtDoCL0BiZQXrRfSvXZ4m/P+pMUSWHhnRTl4545eSktdqapj2RuU0FDRfNGc xexnAfE/bu8XHTyKrXF7wc9uq3AZ31wu5dPFtCxI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730556AbgBKRwl (ORCPT ); Tue, 11 Feb 2020 12:52:41 -0500 Received: from mail.kernel.org ([198.145.29.99]:55756 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730586AbgBKRwk (ORCPT ); Tue, 11 Feb 2020 12:52:40 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 06173206D7; Tue, 11 Feb 2020 17:52:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443560; bh=hU+8ZXqCJKxOobeRNU4DXaABOcyKqWOi8gVrtQCKmB4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=EdhZW7l3r9pBPITz5bqtPd1CMIS/aQIr6rlitozY7j+K2VybQepugwb24fh8UVjmE 0IKsMPOBpFsJi0tuRE/Li5AFMw54PwX/ED9WfADaS8YXOe/U39Wwb8HizuHHsiZjgY c1f1w5mTgRnPnA1rb0405qetcy0+BVl1Ce+jMb80= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg4-004O7k-2O; Tue, 11 Feb 2020 17:50:28 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 59/94] arm64: KVM: nv: Handle SCTLR_EL2 RES0/RES1 bits Date: Tue, 11 Feb 2020 17:49:03 +0000 Message-Id: <20200211174938.27809-60-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Depending on the HCR_EL2.{E2H,TGE} values, SCTLR_EL2 has different RES0/RES1 constraints. Let's handle that. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index c62080d7742c..121b3f28cae2 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -443,6 +443,37 @@ static bool access_vbar_el1(struct kvm_vcpu *vcpu, return access_rw(vcpu, p, r); } +static bool access_sctlr_el2(struct kvm_vcpu *vcpu, + struct sys_reg_params *p, + const struct sys_reg_desc *r) +{ + if (el12_reg(p) && forward_nv_traps(vcpu)) + return false; + + if (p->is_write) { + u64 val = p->regval; + + if (vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu)) { + val &= ~(GENMASK_ULL(63,45) | GENMASK_ULL(34, 32) | + BIT_ULL(17)); + val |= SCTLR_EL1_RES1; + } else { + val &= ~(GENMASK_ULL(63,45) | BIT_ULL(42) | + GENMASK_ULL(39, 38) | GENMASK_ULL(35, 32) | + BIT_ULL(26) | BIT_ULL(24) | BIT_ULL(20) | + BIT_ULL(17) | GENMASK_ULL(15, 14) | + GENMASK(10, 7)); + val |= SCTLR_EL2_RES1; + } + + vcpu_write_sys_reg(vcpu, val, r->reg); + } else { + p->regval = vcpu_read_sys_reg(vcpu, r->reg); + } + + return true; +} + /* * See note at ARMv7 ARM B1.14.4 (TL;DR: S/W ops are not easily virtualized). */ @@ -2254,7 +2285,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_VPIDR_EL2), access_rw, reset_vpidr, VPIDR_EL2 }, { SYS_DESC(SYS_VMPIDR_EL2), access_rw, reset_vmpidr, VMPIDR_EL2 }, - { SYS_DESC(SYS_SCTLR_EL2), access_rw, reset_val, SCTLR_EL2, 0 }, + { SYS_DESC(SYS_SCTLR_EL2), access_sctlr_el2, reset_val, SCTLR_EL2, SCTLR_EL2_RES1 }, { SYS_DESC(SYS_ACTLR_EL2), access_rw, reset_val, ACTLR_EL2, 0 }, { SYS_DESC(SYS_HCR_EL2), access_rw, reset_val, HCR_EL2, 0 }, { SYS_DESC(SYS_MDCR_EL2), access_rw, reset_val, MDCR_EL2, 0 }, From patchwork Tue Feb 11 17:49:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376303 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9048C14B4 for ; Tue, 11 Feb 2020 17:53:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6FB6420848 for ; Tue, 11 Feb 2020 17:53:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443607; bh=6qEfnPAQpmbL6mkKK1EQQA1y8FQyNR3EaGGAZea+tbk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=N0GvW7yif5TlzjoR6nLrkcqa/YO5qnzB/Oo05hrMeYloiZNZFrJzFkQgZ7VYpnvbB mMPDopu1TvzHA9GN+7/EjdT+Sf+i21TaSKbiLqS5hJOsdbzdX1NOvcWBumpv6HXBxv CvcO9n0ha63VafXre3PkYIkU0sJmt0ffKjSivDmM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730739AbgBKRx0 (ORCPT ); Tue, 11 Feb 2020 12:53:26 -0500 Received: from mail.kernel.org ([198.145.29.99]:56986 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730637AbgBKRxZ (ORCPT ); Tue, 11 Feb 2020 12:53:25 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 07D0A20578; Tue, 11 Feb 2020 17:53:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443605; bh=6qEfnPAQpmbL6mkKK1EQQA1y8FQyNR3EaGGAZea+tbk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=C2hxYCSBr8A2cGVap21mWjhMt72gIg1HgqTKuOrdCTVW4rkT+XN8MmJXbILHounYV sku3e/hkzreGyEo5shAsbkuwL8odH4tzmlWRDYQjnYuWjGmzmZ2r4eL+Pnl7utC925 U3GQaUkOsj5HYl856LkVthyKy8D1I+RdICX9srbU= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg4-004O7k-ON; Tue, 11 Feb 2020 17:50:28 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 60/94] arm64: KVM: nv: Restrict S2 RD/WR permissions to match the guest's Date: Tue, 11 Feb 2020 17:49:04 +0000 Message-Id: <20200211174938.27809-61-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org When mapping a page in a shadow stage-2, special care must be taken not to be more permissive than the guest is (writable or readable page when the guest hasn't set that permission). Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_mmu.h | 18 ++++++++++++++++++ arch/arm64/include/asm/kvm_mmu.h | 18 ++++++++++++++++++ arch/arm64/include/asm/kvm_nested.h | 10 ++++++++++ virt/kvm/arm/mmu.c | 21 ++++++++++++++++++++- 4 files changed, 66 insertions(+), 1 deletion(-) diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index 1d0d5f00f0af..be7be6583e54 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -113,6 +113,12 @@ static inline pud_t kvm_s2pud_mkexec(pud_t pud) return pud; } +static inline pud_t kvm_s2pud_revoke_read(pud_t pud) +{ + WARN_ON(1); + return pud; +} + static inline bool kvm_s2pud_exec(pud_t *pud) { WARN_ON(1); @@ -155,6 +161,18 @@ static inline pmd_t kvm_s2pmd_mkexec(pmd_t pmd) return pmd; } +static inline pte_t kvm_s2pte_revoke_read(pte_t pte) +{ + pte_val(pte) &= ~L_PTE_S2_RDONLY; + return pte; +} + +static inline pmd_t kvm_s2pmd_revoke_read(pmd_t pmd) +{ + pmd_val(pmd) &= ~L_PMD_S2_RDONLY; + return pmd; +} + static inline void kvm_set_s2pte_readonly(pte_t *pte) { pte_val(*pte) = (pte_val(*pte) & ~L_PTE_S2_RDWR) | L_PTE_S2_RDONLY; diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h index 9c0bf878fb3b..b390bba466e9 100644 --- a/arch/arm64/include/asm/kvm_mmu.h +++ b/arch/arm64/include/asm/kvm_mmu.h @@ -228,6 +228,24 @@ static inline pud_t kvm_s2pud_mkexec(pud_t pud) return pud; } +static inline pte_t kvm_s2pte_revoke_read(pte_t pte) +{ + pte_val(pte) &= ~PTE_S2_RDONLY; + return pte; +} + +static inline pmd_t kvm_s2pmd_revoke_read(pmd_t pmd) +{ + pmd_val(pmd) &= ~PMD_S2_RDONLY; + return pmd; +} + +static inline pud_t kvm_s2pud_revoke_read(pud_t pud) +{ + pud_val(pud) &= ~PUD_S2_RDONLY; + return pud; +} + static inline void kvm_set_s2pte_readonly(pte_t *ptep) { pteval_t old_pteval, pteval; diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 45c013c0c856..7cd0c5b0fec9 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -43,6 +43,16 @@ static inline u32 kvm_s2_trans_esr(struct kvm_s2_trans *trans) return trans->esr; } +static inline bool kvm_s2_trans_readable(struct kvm_s2_trans *trans) +{ + return trans->readable; +} + +static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans) +{ + return trans->writable; +} + extern int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, struct kvm_s2_trans *result); diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 0086b42237af..7da72c2b7f0f 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1721,7 +1721,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, unsigned long hva, unsigned long fault_status) { int ret; - bool write_fault, writable; + bool write_fault, writable, readable = true; bool exec_fault, needs_exec; unsigned long mmu_seq; phys_addr_t ipa = fault_ipa; @@ -1841,6 +1841,16 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault && is_iomap(flags)) return -ENOEXEC; + /* + * Potentially reduce shadow S2 permissions to match the guest's own + * S2. For exec faults, we'd only reach this point if the guest + * actually allowed it (see kvm_s2_handle_perm_fault). + */ + if (kvm_is_shadow_s2_fault(vcpu)) { + writable &= kvm_s2_trans_writable(nested); + readable &= kvm_s2_trans_readable(nested); + } + spin_lock(&kvm->mmu_lock); if (mmu_notifier_retry(kvm, mmu_seq)) goto out_unlock; @@ -1886,6 +1896,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (writable) new_pud = kvm_s2pud_mkwrite(new_pud); + if (!readable) + new_pud = kvm_s2pud_revoke_read(new_pud); + if (needs_exec) new_pud = kvm_s2pud_mkexec(new_pud); @@ -1898,6 +1911,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (writable) new_pmd = kvm_s2pmd_mkwrite(new_pmd); + if (!readable) + new_pmd = kvm_s2pmd_revoke_read(new_pmd); + if (needs_exec) new_pmd = kvm_s2pmd_mkexec(new_pmd); @@ -1910,6 +1926,9 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, mark_page_dirty(kvm, gfn); } + if (!readable) + new_pte = kvm_s2pte_revoke_read(new_pte); + if (needs_exec) new_pte = kvm_s2pte_mkexec(new_pte); From patchwork Tue Feb 11 17:49:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376239 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 526D314B4 for ; Tue, 11 Feb 2020 17:52:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 27C73206D7 for ; Tue, 11 Feb 2020 17:52:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443550; bh=ggnlFbQSGsLLyg5YVIv0hmQsy7LefCB+e8Mw1VKL7Bs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Y56C63gtSzriOOEs0wAN8rH31LIuF7xJP400jS0fUx5RzzlrP86xjAi+FXLFpabzv HhHVgEe5+c9uL7b9hvEmf7kU9W00Jhkc7EW4spa4mePNlCRUu1juNgG9sAXiEzBGu2 FGC/xodLScduCfhUvcTNVS80sqtP3RG4nxfv4A4M= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730510AbgBKRw3 (ORCPT ); Tue, 11 Feb 2020 12:52:29 -0500 Received: from mail.kernel.org ([198.145.29.99]:55440 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730532AbgBKRw3 (ORCPT ); Tue, 11 Feb 2020 12:52:29 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id CB08C206CC; Tue, 11 Feb 2020 17:52:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443548; bh=ggnlFbQSGsLLyg5YVIv0hmQsy7LefCB+e8Mw1VKL7Bs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=l3CODoaR6tQmjcEfGVgVjrChQy6+4xWkKZKSoCbLToXQ5/ZLr99AnkpLr9o4AtoW0 DPl+0IuD9nHSLnTRfq1ZK0xrKraIAGOgnxvsRTFUU0nKS09x3Iwbh3cXfb8qa1apz7 CHTV11lzruSwgB5vdpWVxSj5e7UQXSFTszE8AG80= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg5-004O7k-Ax; Tue, 11 Feb 2020 17:50:29 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 61/94] arm64: KVM: nv: Allow userspace to request KVM_ARM_VCPU_NESTED_VIRT Date: Tue, 11 Feb 2020 17:49:05 +0000 Message-Id: <20200211174938.27809-62-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Since we're (almost) feature complete, let's allow userspace to request KVM_ARM_VCPU_NESTED_VIRT by bumping the KVM_VCPU_MAX_FEATURES up. It's going to be great... Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index c578ab9b4a02..06c4b7aaebab 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -37,7 +37,7 @@ #define KVM_MAX_VCPUS VGIC_V3_MAX_CPUS -#define KVM_VCPU_MAX_FEATURES 7 +#define KVM_VCPU_MAX_FEATURES 8 #define KVM_REQ_SLEEP \ KVM_ARCH_REQ_FLAGS(0, KVM_REQUEST_WAIT | KVM_REQUEST_NO_WAKEUP) From patchwork Tue Feb 11 17:49:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376307 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1FB181395 for ; Tue, 11 Feb 2020 17:53:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F3F7A208C3 for ; Tue, 11 Feb 2020 17:53:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443611; bh=ptv/8iU8Wqw8wnzlKzqydigm6fuDVi4E7l30mG2CJdc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=OOML/srBFuMNyJTwbUHsSpYWqB4Wl6Pq3erGa9YIxNM8djtCvRZynLsgyiEAyoWUu /CCb+61Sul8QssGZ7MP9bpT2iFNCAAJ5fm45E9t8Yy2x6wjofyRlpJ6Ds7Zrf4YyWq WBsrcDH7qQuahA/+0qWeP+oUR/0JuXEZpq/aLS/8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730745AbgBKRx3 (ORCPT ); Tue, 11 Feb 2020 12:53:29 -0500 Received: from mail.kernel.org ([198.145.29.99]:57084 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730671AbgBKRx3 (ORCPT ); Tue, 11 Feb 2020 12:53:29 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id C48C320661; Tue, 11 Feb 2020 17:53:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443608; bh=ptv/8iU8Wqw8wnzlKzqydigm6fuDVi4E7l30mG2CJdc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QStCXeDMI1gz0CtwFQ8/YKjHo89QwI/dOb/y8EMMTyUlWC0FfVYBEpOdSoDR0YPy9 LTkjoCu7emixnJ94yV8ouGu2olz46Rh5X97a6o2C44JtdElPDqV5ad6mjCBajXHI7f e/2k2F9kELFxrS6YkC9ANCaB+SjLtn/eZgXWweZY= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg5-004O7k-VT; Tue, 11 Feb 2020 17:50:30 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 62/94] arm64: Detect the ARMv8.4 TTL feature Date: Tue, 11 Feb 2020 17:49:06 +0000 Message-Id: <20200211174938.27809-63-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order to reduce the cost of TLB invalidation, the ARMv8.4 TTL feature allows TLBs to be issued with a level allowing for quicker invalidation. Let's detect the feature for now. Further patches will implement its actual usage. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kernel/cpufeature.c | 11 +++++++++++ 3 files changed, 14 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 21f0067d8d40..5736650cd0fb 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -59,7 +59,8 @@ #define ARM64_HAS_E0PD 49 #define ARM64_HAS_RNG 50 #define ARM64_HAS_NESTED_VIRT 51 +#define ARM64_HAS_ARMv8_4_TTL 52 -#define ARM64_NCAPS 52 +#define ARM64_NCAPS 53 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 92c97a19369e..a402e762c51d 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -804,6 +804,7 @@ #define ID_AA64MMFR2_E0PD_SHIFT 60 #define ID_AA64MMFR2_EVT_SHIFT 56 #define ID_AA64MMFR2_BBM_SHIFT 52 +#define ID_AA64MMFR2_TTL_SHIFT 48 #define ID_AA64MMFR2_FWB_SHIFT 40 #define ID_AA64MMFR2_AT_SHIFT 32 #define ID_AA64MMFR2_ST_SHIFT 28 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 157700590aa8..aada8c3eff1e 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -249,6 +249,7 @@ static const struct arm64_ftr_bits ftr_id_aa64mmfr1[] = { static const struct arm64_ftr_bits ftr_id_aa64mmfr2[] = { ARM64_FTR_BITS(FTR_HIDDEN, FTR_NONSTRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_E0PD_SHIFT, 4, 0), + ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_TTL_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_FWB_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_VISIBLE, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_AT_SHIFT, 4, 0), ARM64_FTR_BITS(FTR_HIDDEN, FTR_STRICT, FTR_LOWER_SAFE, ID_AA64MMFR2_NV_SHIFT, 4, 0), @@ -1557,6 +1558,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_cpuid_feature, .cpu_enable = cpu_has_fwb, }, + { + .desc = "ARMv8.4 Translation Table Level", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HAS_ARMv8_4_TTL, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_TTL_SHIFT, + .min_field_value = 1, + .matches = has_cpuid_feature, + }, #ifdef CONFIG_ARM64_HW_AFDBM { /* From patchwork Tue Feb 11 17:49:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376163 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D687A1395 for ; Tue, 11 Feb 2020 17:51:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B5D86214DB for ; Tue, 11 Feb 2020 17:51:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443482; bh=VNVEjhnfzVzUtrBs05fM+reAaQaIItDsd0jrQQAovo4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=OJ4gGDViX2ilUL2WlBbppTTWhUfBjXAW/Ug2M9lIaHqS50bwLZ1vV3iYEnp0jzgVh gyI0J5yQpzyWQrylDFdLxsLeremR0q6xzvnJgf6YyQgvjni4eC2TrycrJseD1TVT1+ uGQRWHyi4hz1LkgDe7q1Af15XaaB5XD0QArn+N5Q= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730341AbgBKRvW (ORCPT ); Tue, 11 Feb 2020 12:51:22 -0500 Received: from mail.kernel.org ([198.145.29.99]:53646 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730335AbgBKRvV (ORCPT ); Tue, 11 Feb 2020 12:51:21 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 85C54206CC; Tue, 11 Feb 2020 17:51:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443480; bh=VNVEjhnfzVzUtrBs05fM+reAaQaIItDsd0jrQQAovo4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ECaWC7uUe2kT8oRS+6kMRJaSvW+T9qlLb1UD1J7ffTmJgalJ4u2/8Ae6aopzxI2LZ ynEDUCrle0CrsqFnWL4lYeibbBeJ4f15H4fCgsVCn9QzQd4+GC6KRcRuy9QV5S2GPw GyFqXKQumuRG/F7D6ae3aYm7ufqyfttENeLmrKVQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg6-004O7k-Hj; Tue, 11 Feb 2020 17:50:30 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 63/94] arm64: KVM: nv: Add handling of ARMv8.4-TTL TLB invalidation Date: Tue, 11 Feb 2020 17:49:07 +0000 Message-Id: <20200211174938.27809-64-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Support guest-provided information information to find out about the range of required invalidation. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/nested.c | 56 ++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 79 +++++++++++++++++++---------- 3 files changed, 108 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 7cd0c5b0fec9..620296206483 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -67,6 +67,7 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); +unsigned int ttl_to_size(u8 ttl); struct sys_reg_params; struct sys_reg_desc; diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 0ab08ee59110..4cb4831d4022 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -351,6 +351,62 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return ret; } + +unsigned int ttl_to_size(u8 ttl) +{ + int level = ttl & 3; + unsigned int max_size = 0; + + switch (ttl >> 2) { + case 0: /* No size information */ + break; + case 1: /* 4kB translation granule */ + switch (level) { + case 0: + break; + case 1: + max_size = SZ_1G; + break; + case 2: + max_size = SZ_2M; + break; + case 3: + max_size = SZ_4K; + break; + } + break; + case 2: /* 16kB translation granule */ + switch (level) { + case 0: + case 1: + break; + case 2: + max_size = SZ_32M; + break; + case 3: + max_size = SZ_16K; + break; + } + break; + case 3: /* 64kB translation granule */ + switch (level) { + case 0: + case 1: + /* No 52bit IPA support */ + break; + case 2: + max_size = SZ_512M; + break; + case 3: + max_size = SZ_64K; + break; + } + break; + } + + return max_size; +} + /* Must be called with kvm->lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 121b3f28cae2..1dcde0230b87 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -9,6 +9,7 @@ * Christoffer Dall */ +#include #include #include #include @@ -2583,59 +2584,81 @@ static bool handle_vmalls12e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, return true; } +static unsigned long compute_tlb_inval_range(struct kvm_vcpu *vcpu, + struct kvm_s2_mmu *mmu, + u64 val) +{ + unsigned long max_size; + u8 ttl = 0; + + if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL)) { + ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + } + + max_size = ttl_to_size(ttl); + + if (!max_size) { + u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + + /* Compute the maximum extent of the invalidation */ + switch ((vtcr & VTCR_EL2_TG0_MASK)) { + case VTCR_EL2_TG0_4K: + max_size = SZ_1G; + break; + case VTCR_EL2_TG0_16K: + max_size = SZ_32M; + break; + case VTCR_EL2_TG0_64K: + /* + * No, we do not support 52bit IPA in nested yet. Once + * we do, this should be 4TB. + */ + /* FIXME: remove the 52bit PA support from the IDregs */ + max_size = SZ_512M; + break; + default: + BUG(); + } + } + + WARN_ON(!max_size); + return max_size; +} + static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { u64 vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - u64 vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); struct kvm_s2_mmu *mmu; u64 base_addr; - int max_size; + unsigned long max_size; /* * We drop a number of things from the supplied value: * * - NS bit: we're non-secure only. * - * - TTL field: We already have the granule size from the - * VTCR_EL2.TG0 field, and the level is only relevant to the - * guest's S2PT. - * * - IPA[51:48]: We don't support 52bit IPA just yet... * * And of course, adjust the IPA to be on an actual address. */ base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; - /* Compute the maximum extent of the invalidation */ - switch ((vtcr & VTCR_EL2_TG0_MASK)) { - case VTCR_EL2_TG0_4K: - max_size = SZ_1G; - break; - case VTCR_EL2_TG0_16K: - max_size = SZ_32M; - break; - case VTCR_EL2_TG0_64K: - /* - * No, we do not support 52bit IPA in nested yet. Once - * we do, this should be 4TB. - */ - /* FIXME: remove the 52bit PA support from the IDregs */ - max_size = SZ_512M; - break; - default: - BUG(); - } - spin_lock(&vcpu->kvm->mmu_lock); mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); - if (mmu) + if (mmu) { + max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); + base_addr &= ~(max_size - 1); kvm_unmap_stage2_range(mmu, base_addr, max_size); + } mmu = lookup_s2_mmu(vcpu->kvm, vttbr, 0); - if (mmu) + if (mmu) { + max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); + base_addr &= ~(max_size - 1); kvm_unmap_stage2_range(mmu, base_addr, max_size); + } spin_unlock(&vcpu->kvm->mmu_lock); From patchwork Tue Feb 11 17:49:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376161 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F3BE91395 for ; Tue, 11 Feb 2020 17:51:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D299421569 for ; Tue, 11 Feb 2020 17:51:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443480; bh=Ik3s7acYpQRtXUBC0I1kZbtJLduLroBrOt90QzyKh88=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=k3fTK9pvwTDyRM4EGR+5fjF0yvndf+A1vFL0vGsR626WGmfWrgEGwnNbTIfUjtFYb YSW8pLgJxdmeCMAgEHt8/CtUjFb9fncX3kutBJVMZUlCyT9K0en/Fe54fAPclXfIYG FxdrmAUCAABnrYeFMAwFbUj0ObOMmJ0a/VDKNcAc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730329AbgBKRvU (ORCPT ); Tue, 11 Feb 2020 12:51:20 -0500 Received: from mail.kernel.org ([198.145.29.99]:53594 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730328AbgBKRvT (ORCPT ); Tue, 11 Feb 2020 12:51:19 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B354D206D7; Tue, 11 Feb 2020 17:51:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443478; bh=Ik3s7acYpQRtXUBC0I1kZbtJLduLroBrOt90QzyKh88=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=u9QOzCk14izQ6zcb8aQ0L873AICmdYxrjfivFqIePdpy4XlMhbikv9mY+ViG4XuYM Ppi7jTTEEJwscxRBWVFdVzQKZH3Zjv9uPH3O9qDnsh3NpOQqLBMKGFamQ8wqO7vHy8 H60WbChBHqV9W2Gnj0lfIcghnmEdazX6QjpAv7V4= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg7-004O7k-4o; Tue, 11 Feb 2020 17:50:31 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 64/94] arm64: KVM: nv: Invalidate TLBs based on shadow S2 TTL-like information Date: Tue, 11 Feb 2020 17:49:08 +0000 Message-Id: <20200211174938.27809-65-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order to be able to make S2 TLB invalidations more performant on NV, let's use a scheme derived from the ARMv8.4 TTL extension. If bits [56:55] in the descriptor are non-zero, they indicate a level which can be used as an invalidation range. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 4 ++ arch/arm64/kvm/nested.c | 98 +++++++++++++++++++++++++++++ arch/arm64/kvm/sys_regs.c | 13 ++-- 3 files changed, 110 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 620296206483..debae814fdc5 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -67,6 +67,8 @@ extern bool __forward_traps(struct kvm_vcpu *vcpu, unsigned int reg, u64 control_bit); extern bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit); extern bool forward_nv_traps(struct kvm_vcpu *vcpu); +u8 get_guest_mapping_ttl(struct kvm_vcpu *vcpu, struct kvm_s2_mmu *mmu, + u64 addr); unsigned int ttl_to_size(u8 ttl); struct sys_reg_params; @@ -75,4 +77,6 @@ struct sys_reg_desc; void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, const struct sys_reg_desc *r); +#define KVM_NV_GUEST_MAP_SZ GENMASK_ULL(56, 55) + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 4cb4831d4022..6adff9b433e0 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -16,6 +16,7 @@ * along with this program. If not, see . */ +#include #include #include @@ -351,6 +352,29 @@ int kvm_walk_nested_s2(struct kvm_vcpu *vcpu, phys_addr_t gipa, return ret; } +static int read_host_s2_desc(phys_addr_t pa, u64 *desc, void *data) +{ + u64 *va = phys_to_virt(pa); + + *desc = *va; + + return 0; +} + +static int kvm_walk_shadow_s2(struct kvm_s2_mmu *mmu, phys_addr_t gipa, + struct kvm_s2_trans *result) +{ + struct s2_walk_info wi = { }; + + wi.read_desc = read_host_s2_desc; + wi.baddr = mmu->pgd_phys; + + vtcr_to_walk_info(mmu->kvm->arch.vtcr, &wi); + + wi.be = IS_ENABLED(CONFIG_CPU_BIG_ENDIAN); + + return walk_nested_s2_pgd(gipa, &wi, result); +} unsigned int ttl_to_size(u8 ttl) { @@ -407,6 +431,80 @@ unsigned int ttl_to_size(u8 ttl) return max_size; } +/* + * Compute the equivalent of the TTL field by parsing the shadow PT. + * The granule size is extracted from VTCR_EL2.TG0 while the level is + * retrieved from first entry carrying the level as a tag. + */ +u8 get_guest_mapping_ttl(struct kvm_vcpu *vcpu, struct kvm_s2_mmu *mmu, + u64 addr) +{ + u64 tmp, sz = 0, vtcr = vcpu_read_sys_reg(vcpu, VTCR_EL2); + struct kvm_s2_trans out; + u8 ttl, level; + + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + ttl = (1 << 2); + break; + case VTCR_EL2_TG0_16K: + ttl = (2 << 2); + break; + case VTCR_EL2_TG0_64K: + ttl = (3 << 2); + break; + default: + BUG(); + } + + tmp = addr; + +again: + /* Iteratively compute the block sizes for a particular granule size */ + switch (vtcr & VTCR_EL2_TG0_MASK) { + case VTCR_EL2_TG0_4K: + if (sz < SZ_4K) sz = SZ_4K; + else if (sz < SZ_2M) sz = SZ_2M; + else if (sz < SZ_1G) sz = SZ_1G; + else sz = 0; + break; + case VTCR_EL2_TG0_16K: + if (sz < SZ_16K) sz = SZ_16K; + else if (sz < SZ_32M) sz = SZ_32M; + else sz = 0; + break; + case VTCR_EL2_TG0_64K: + if (sz < SZ_64K) sz = SZ_64K; + else if (sz < SZ_512M) sz = SZ_512M; + else sz = 0; + break; + default: + BUG(); + } + + if (sz == 0) + return 0; + + tmp &= ~(sz - 1); + out = (struct kvm_s2_trans) { }; + kvm_walk_shadow_s2(mmu, tmp, &out); + level = FIELD_GET(KVM_NV_GUEST_MAP_SZ, out.upper_attr); + if (!level) + goto again; + + ttl |= level; + + /* + * We now have found some level information in the shadow S2. Check + * that the resulting range is actually including the original IPA. + */ + sz = ttl_to_size(ttl); + if (addr < (tmp + sz)) + return ttl; + + return 0; +} + /* Must be called with kvm->lock held */ struct kvm_s2_mmu *lookup_s2_mmu(struct kvm *kvm, u64 vttbr, u64 hcr) { diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 1dcde0230b87..8d4e67594d87 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2589,10 +2589,13 @@ static unsigned long compute_tlb_inval_range(struct kvm_vcpu *vcpu, u64 val) { unsigned long max_size; - u8 ttl = 0; + u8 ttl; - if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL)) { - ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + ttl = FIELD_GET(GENMASK_ULL(47, 44), val); + + if (!(cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL) && ttl)) { + u64 addr = (val & GENMASK_ULL(35, 0)) << 12; + ttl = get_guest_mapping_ttl(vcpu, mmu, addr); } max_size = ttl_to_size(ttl); @@ -2633,6 +2636,8 @@ static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, u64 base_addr; unsigned long max_size; + spin_lock(&vcpu->kvm->mmu_lock); + /* * We drop a number of things from the supplied value: * @@ -2644,8 +2649,6 @@ static bool handle_ipas2e1is(struct kvm_vcpu *vcpu, struct sys_reg_params *p, */ base_addr = (p->regval & GENMASK_ULL(35, 0)) << 12; - spin_lock(&vcpu->kvm->mmu_lock); - mmu = lookup_s2_mmu(vcpu->kvm, vttbr, HCR_VM); if (mmu) { max_size = compute_tlb_inval_range(vcpu, mmu, p->regval); From patchwork Tue Feb 11 17:49:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376213 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 04D9F186E for ; Tue, 11 Feb 2020 17:52:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D90F120848 for ; Tue, 11 Feb 2020 17:52:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443528; bh=+xm0GyRSVwHVcIutj4WcHndnRUt3TPwqJzrA5woUxCg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=SN0SxWwym6XezkIyf5a7cMM61zXq8bnEi5fP1vaK7q194Fy6wixCcp1QGQgRrrVn+ tdSixggXc+0fKsylZst9JPnhV1oBMNN/eW+6CUM/vm3JjDtZiqkGmxd/7VAiN5i8WV Mo0TrCcwh/4sZinBnr1RJg04jiz9plnK55Z+8i50= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730499AbgBKRwI (ORCPT ); Tue, 11 Feb 2020 12:52:08 -0500 Received: from mail.kernel.org ([198.145.29.99]:54822 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730479AbgBKRwH (ORCPT ); Tue, 11 Feb 2020 12:52:07 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 4C56A208C3; Tue, 11 Feb 2020 17:52:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443526; bh=+xm0GyRSVwHVcIutj4WcHndnRUt3TPwqJzrA5woUxCg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f5Dlin2MWX7gTEniKJd1TzD1iNW7WuDuHL9JSwSqF/8vMtCRQthMLMlEfDuGocIg+ eOfoeaGAXDKbHZdpzlVcxpTqH4oTrPthgr7i21RGS58WBGlK/FrsvAoAYnfH3tENqZ IFtwb+YPtzu7eRxmvYOgvG/ADYjnXwZ2UJAq9hPE= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg7-004O7k-O9; Tue, 11 Feb 2020 17:50:31 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 65/94] arm64: KVM: nv: Tag shadow S2 entries with nested level Date: Tue, 11 Feb 2020 17:49:09 +0000 Message-Id: <20200211174938.27809-66-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Populate bits [56:55] of the leaf entry with the level provided by the guest's S2 translation. Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_mmu.h | 5 +++++ arch/arm64/include/asm/kvm_nested.h | 6 ++++++ virt/kvm/arm/mmu.c | 20 ++++++++++++++++++++ 3 files changed, 31 insertions(+) diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h index be7be6583e54..3774a7289ef2 100644 --- a/arch/arm/include/asm/kvm_mmu.h +++ b/arch/arm/include/asm/kvm_mmu.h @@ -484,6 +484,11 @@ static inline bool kvm_s2_trans_writable(struct kvm_s2_trans *trans) BUG(); } +static inline u64 kvm_encode_nested_level(struct kvm_s2_trans *trans) +{ + BUG(); +} + static inline void kvm_nested_s2_flush(struct kvm *kvm) {} static inline void kvm_nested_s2_wp(struct kvm *kvm) {} static inline void kvm_nested_s2_clear(struct kvm *kvm) {} diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index debae814fdc5..3e3778d3cec6 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -2,6 +2,7 @@ #ifndef __ARM64_KVM_NESTED_H #define __ARM64_KVM_NESTED_H +#include #include static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) @@ -79,4 +80,9 @@ void access_nested_id_reg(struct kvm_vcpu *v, struct sys_reg_params *p, #define KVM_NV_GUEST_MAP_SZ GENMASK_ULL(56, 55) +static inline u64 kvm_encode_nested_level(struct kvm_s2_trans *trans) +{ + return FIELD_PREP(KVM_NV_GUEST_MAP_SZ, trans->level); +} + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 7da72c2b7f0f..eaa86cad2ac8 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1715,6 +1715,11 @@ static bool fault_supports_stage2_huge_mapping(struct kvm_memory_slot *memslot, (hva & ~(map_size - 1)) + map_size <= uaddr_end; } +#define set_desc_bits(which, desc, val) \ + do { \ + desc = __ ## which(which ## _val(desc) | val); \ + } while(0) + static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, struct kvm_s2_trans *nested, struct kvm_memory_slot *memslot, @@ -1736,6 +1741,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, unsigned long vma_pagesize, flags = 0; struct kvm_s2_mmu *mmu = vcpu->arch.hw_mmu; unsigned long max_map_size = PUD_SIZE; + u64 l1_s2_level; write_fault = kvm_is_write_fault(vcpu); exec_fault = kvm_vcpu_trap_is_iabt(vcpu); @@ -1845,10 +1851,18 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * Potentially reduce shadow S2 permissions to match the guest's own * S2. For exec faults, we'd only reach this point if the guest * actually allowed it (see kvm_s2_handle_perm_fault). + * + * Also encode the level of the nested translation in the SW bits of + * the PTE/PMD/PUD. This will be retrived on TLB invalidation from + * the guest. */ if (kvm_is_shadow_s2_fault(vcpu)) { writable &= kvm_s2_trans_writable(nested); readable &= kvm_s2_trans_readable(nested); + + l1_s2_level = kvm_encode_nested_level(nested); + } else { + l1_s2_level = 0; } spin_lock(&kvm->mmu_lock); @@ -1902,6 +1916,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (needs_exec) new_pud = kvm_s2pud_mkexec(new_pud); + set_desc_bits(pud, new_pud, l1_s2_level); + ret = stage2_set_pud_huge(mmu, memcache, fault_ipa, &new_pud); } else if (vma_pagesize == PMD_SIZE) { pmd_t new_pmd = kvm_pfn_pmd(pfn, mem_type); @@ -1917,6 +1933,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (needs_exec) new_pmd = kvm_s2pmd_mkexec(new_pmd); + set_desc_bits(pmd, new_pmd, l1_s2_level); + ret = stage2_set_pmd_huge(mmu, memcache, fault_ipa, &new_pmd); } else { pte_t new_pte = kvm_pfn_pte(pfn, mem_type); @@ -1932,6 +1950,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (needs_exec) new_pte = kvm_s2pte_mkexec(new_pte); + set_desc_bits(pte, new_pte, l1_s2_level); + ret = stage2_set_pte(mmu, memcache, fault_ipa, &new_pte, flags); } From patchwork Tue Feb 11 17:49:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376167 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3712B14B4 for ; Tue, 11 Feb 2020 17:51:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 174D32086A for ; Tue, 11 Feb 2020 17:51:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443486; bh=6suUVhOomHtBnTlJ4ORDfwOHu1rBDPyP9IgwjpH503U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=hZlQsfzL0yJQGL6Ms28y0SorEo1OQwo6r+QnEBMGDsmdNI/AZxxCTkXHJeHQpuYFv qaWKejFtai/5BhrIdEGCsydy804LlGv6EJXV6o3Ej/a3NUPHATp5GdHIJ4RZU80wEZ wzw5iaKKcxR7zTcg5PIksZBT6metsOhIqS7zrGGw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730348AbgBKRvZ (ORCPT ); Tue, 11 Feb 2020 12:51:25 -0500 Received: from mail.kernel.org ([198.145.29.99]:53756 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730300AbgBKRvZ (ORCPT ); Tue, 11 Feb 2020 12:51:25 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 31A78206D6; Tue, 11 Feb 2020 17:51:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443484; bh=6suUVhOomHtBnTlJ4ORDfwOHu1rBDPyP9IgwjpH503U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=y6cNfOqFWUf7IqHpjUqp8kQSpMWBU8xOV/ie2oD/rLhOfCmfR5GoriF1nLJGnrBRE womXjyyjQCepC4IW3dv8KkvmjEq6N4e1yIoujAciudiMp4UxoulMmS5+SFRICyUdd6 Tc4fFFZ7WsDcH4iSptfwJ82La9VVHRSZZdJNSuFE= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg8-004O7k-An; Tue, 11 Feb 2020 17:50:32 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 66/94] arm64: Add SW reserved PTE/PMD bits Date: Tue, 11 Feb 2020 17:49:10 +0000 Message-Id: <20200211174938.27809-67-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Advertise bits [58:55] as reserved for SW in the S2 descriptors. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/pgtable-hwdef.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index 6bf5e650da78..7eab0d23cdb5 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -177,10 +177,12 @@ #define PTE_S2_RDONLY (_AT(pteval_t, 1) << 6) /* HAP[2:1] */ #define PTE_S2_RDWR (_AT(pteval_t, 3) << 6) /* HAP[2:1] */ #define PTE_S2_XN (_AT(pteval_t, 2) << 53) /* XN[1:0] */ +#define PTE_S2_SW_RESVD (_AT(pteval_t, 15) << 55) /* Reserved for SW */ #define PMD_S2_RDONLY (_AT(pmdval_t, 1) << 6) /* HAP[2:1] */ #define PMD_S2_RDWR (_AT(pmdval_t, 3) << 6) /* HAP[2:1] */ #define PMD_S2_XN (_AT(pmdval_t, 2) << 53) /* XN[1:0] */ +#define PMD_S2_SW_RESVD (_AT(pmdval_t, 15) << 55) /* Reserved for SW */ #define PUD_S2_RDONLY (_AT(pudval_t, 1) << 6) /* HAP[2:1] */ #define PUD_S2_RDWR (_AT(pudval_t, 3) << 6) /* HAP[2:1] */ From patchwork Tue Feb 11 17:49:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376233 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 790CF14B4 for ; Tue, 11 Feb 2020 17:52:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 57A08214DB for ; Tue, 11 Feb 2020 17:52:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443545; bh=nwuR2uWCcoRqKVX0jc68PcrHdtG3RZH2Nm1tXCXwEfo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=cBY+iqO+tNf/ohUTI5IVv2R2I6j9M3Qu/J+DTH31A4TWzxwCJ90tfnVHr6PUiD+YI dSaF/Z+5NG1qCGB6iieAWG2U0OeDaVqNJHoA/z11yYfvwNrVNsd39Yv3QPrp+t5XtK vghxzkgxrvoGbNfjKwvqbPv4sEWf9I/KCeeKqV18= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730539AbgBKRwY (ORCPT ); Tue, 11 Feb 2020 12:52:24 -0500 Received: from mail.kernel.org ([198.145.29.99]:55284 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730525AbgBKRwX (ORCPT ); Tue, 11 Feb 2020 12:52:23 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5480320661; Tue, 11 Feb 2020 17:52:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443543; bh=nwuR2uWCcoRqKVX0jc68PcrHdtG3RZH2Nm1tXCXwEfo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=U/dRmCwbsNRjuMAnf0BBQZsTuNSDxWEebKAwTif4CxWRts/CgY25Ak470/qhj6xlf 2OeUtQ036cMaTKpQO0qTSsT+6URFXid05iM9A5df8Kf5HdM1hwO/QL4GfIxr+Cgy9r rURnrtQ0WEETid1RYNhuI1kknJ9j4jLjerZ1PJaQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg8-004O7k-SI; Tue, 11 Feb 2020 17:50:33 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 67/94] arm64: Add level-hinted TLB invalidation helper Date: Tue, 11 Feb 2020 17:49:11 +0000 Message-Id: <20200211174938.27809-68-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add a level-hinted TLB invalidation helper that only gets used if ARMv8.4-TTL gets detected. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/tlbflush.h | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bc3949064725..a3f70778a325 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -10,6 +10,7 @@ #ifndef __ASSEMBLY__ +#include #include #include #include @@ -59,6 +60,35 @@ __ta; \ }) +#define TLBI_TTL_MASK GENMASK_ULL(47, 44) + +#define __tlbi_level(op, addr, level) \ + do { \ + u64 arg = addr; \ + \ + if (cpus_have_const_cap(ARM64_HAS_ARMv8_4_TTL) && \ + level) { \ + u64 ttl = level; \ + \ + switch (PAGE_SIZE) { \ + case SZ_4K: \ + ttl |= 1 << 2; \ + break; \ + case SZ_16K: \ + ttl |= 2 << 2; \ + break; \ + case SZ_64K: \ + ttl |= 3 << 2; \ + break; \ + } \ + \ + arg &= ~TLBI_TTL_MASK; \ + arg |= FIELD_PREP(TLBI_TTL_MASK, ttl); \ + } \ + \ + __tlbi(op, arg); \ + } while(0) + /* * TLB Invalidation * ================ From patchwork Tue Feb 11 17:49:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376311 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 252D114B4 for ; Tue, 11 Feb 2020 17:53:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 067FD206D7 for ; Tue, 11 Feb 2020 17:53:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443616; bh=aDjnahK7imY4Wiu3rGFAEkgpWazKaqx4y/1oenhKSNQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=Z343eaaF5Fy+bvhS6keCHKsrdZ16ZYyN5No5iu49MR/T5UCV0qBKZdhxy63MWQCya /tPlgNHOpk+SpX0eV1tMKu+mosae1q0paci7ufdvaW/5K2na8i0ogQ/0c/v6Gt76DU swAFzamcdCCoJeAJHAwvRcdMH0yNPxw+rJ80kWQ0= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730778AbgBKRxe (ORCPT ); Tue, 11 Feb 2020 12:53:34 -0500 Received: from mail.kernel.org ([198.145.29.99]:57202 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730761AbgBKRxd (ORCPT ); Tue, 11 Feb 2020 12:53:33 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 71B5D208C3; Tue, 11 Feb 2020 17:53:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443612; bh=aDjnahK7imY4Wiu3rGFAEkgpWazKaqx4y/1oenhKSNQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=LmlCJPpcDwkR6+ZtT/FewyZ66I93MIGbdS5BuzQjlX23EZ7WYd041fS/gY4r12PLw dU9dLDxxpxYd30kJ0yml6iqhbcXvl2YAuVYQOLa4Uuok6OKrXXnu3jNZmnVKM0wko6 5RPoP4afTstyEEp2l3ewBjqR69k3nVXsDtB4MOvI= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1Zg9-004O7k-F0; Tue, 11 Feb 2020 17:50:33 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 68/94] arm64: KVM: Add a level hint to __kvm_tlb_flush_vmid_ipa Date: Tue, 11 Feb 2020 17:49:12 +0000 Message-Id: <20200211174938.27809-69-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduce a level hint to __kvm_tlb_flush_vmid_ipa. No functional change. Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_asm.h | 3 ++- arch/arm/kvm/hyp/tlb.c | 7 ++++--- arch/arm64/include/asm/kvm_asm.h | 3 ++- arch/arm64/kvm/hyp/tlb.c | 5 +++-- virt/kvm/arm/mmu.c | 2 +- 5 files changed, 12 insertions(+), 8 deletions(-) diff --git a/arch/arm/include/asm/kvm_asm.h b/arch/arm/include/asm/kvm_asm.h index 4f85323f1290..ee7c0aec4c87 100644 --- a/arch/arm/include/asm/kvm_asm.h +++ b/arch/arm/include/asm/kvm_asm.h @@ -55,7 +55,8 @@ extern char __kvm_hyp_init[]; extern char __kvm_hyp_init_end[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, + int level); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); diff --git a/arch/arm/kvm/hyp/tlb.c b/arch/arm/kvm/hyp/tlb.c index 8833e0b0ea97..84b4ea1955b2 100644 --- a/arch/arm/kvm/hyp/tlb.c +++ b/arch/arm/kvm/hyp/tlb.c @@ -21,8 +21,8 @@ * have to use an IPI based mechanism. Until then, we stick to the simple * hardware assisted version. * - * As v7 does not support flushing per IPA, just nuke the whole TLB - * instead, ignoring the ipa value. + * As v7 does not support flushing per IPA (let alone any level hint), + * just nuke the whole TLB instead, ignoring the ipa value. */ void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) { @@ -39,7 +39,8 @@ void __hyp_text __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu) write_sysreg(0, VTTBR); } -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, + phys_addr_t ipa, int level) { __kvm_tlb_flush_vmid(mmu); } diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index c43160d2940d..2c001aa1bdd6 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -59,7 +59,8 @@ extern char __kvm_hyp_init_end[]; extern char __kvm_hyp_vector[]; extern void __kvm_flush_vm_context(void); -extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa); +extern void __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, + int level); extern void __kvm_tlb_flush_vmid(struct kvm_s2_mmu *mmu); extern void __kvm_tlb_flush_local_vmid(struct kvm_vcpu *vcpu); extern void __kvm_tlb_vae2is(struct kvm_s2_mmu *mmu, u64 va, u64 sys_encoding); diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index aa08300774fa..71e4e86e0981 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -132,7 +132,8 @@ static void __hyp_text __tlb_switch_to_host(struct kvm_s2_mmu *mmu, __tlb_switch_to_host_nvhe(cxt); } -void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) +void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, + phys_addr_t ipa, int level) { struct tlb_inv_context cxt; @@ -148,7 +149,7 @@ void __hyp_text __kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa * whole of Stage-1. Weep... */ ipa >>= 12; - __tlbi(ipas2e1is, ipa); + __tlbi_level(ipas2e1is, ipa, level); /* * We have to ensure completion of the invalidation at Stage-2, diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index eaa86cad2ac8..67752c2a615f 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -76,7 +76,7 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa); + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa, 0); } /* From patchwork Tue Feb 11 17:49:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376279 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 520531395 for ; Tue, 11 Feb 2020 17:53:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 31229208C3 for ; Tue, 11 Feb 2020 17:53:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443585; bh=+RRWW/hkjQ++uwcIqUV7epI4LDYcCn4wfi9/FkAslBs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=PFPH7sM9JD/Li/poHu2HSk3vfoqAbJOQ2XEtWEzW8t0E2AtMdR90kVPB+YGvwbrSH 4adJX3gEOZETs981c6TfOU13haj2vXShGz5E15p/X12ArHrA0bnTfjPI69ITQXmDCG m6ayP8aOJ6BURSKw7HkUu6p9Gwl70odx9PjcqlJI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730660AbgBKRxE (ORCPT ); Tue, 11 Feb 2020 12:53:04 -0500 Received: from mail.kernel.org ([198.145.29.99]:56362 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730641AbgBKRxD (ORCPT ); Tue, 11 Feb 2020 12:53:03 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id AD4DB206D7; Tue, 11 Feb 2020 17:53:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443582; bh=+RRWW/hkjQ++uwcIqUV7epI4LDYcCn4wfi9/FkAslBs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=xzyBktmCt/WDU5W1GAQeUg6EGjoO+QxNysTaWznPdDJllLNvJIxpIC/3ChIqvqLX8 uDkZNKD388DUg1VD3BmTFDVDLDjBI2Vq6QwzWqYs1cnVE1cH6LYySjcI1SS6ndGU/v RL1q330SYVfqEev6CBt/n1536bYbdog7YEta5f2A= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgA-004O7k-3C; Tue, 11 Feb 2020 17:50:34 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 69/94] arm64: KVM: Use TTL hint in when invalidating stage-2 translations Date: Tue, 11 Feb 2020 17:49:13 +0000 Message-Id: <20200211174938.27809-70-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Since we always have a precide idea of the level we're dealing with when invalidating TLBs, we can provide it to as a hint to our invalidation helper. Signed-off-by: Marc Zyngier --- arch/arm/include/asm/stage2_pgtable.h | 9 +++++++++ arch/arm64/include/asm/stage2_pgtable.h | 9 +++++++++ virt/kvm/arm/mmu.c | 27 +++++++++++++------------ 3 files changed, 32 insertions(+), 13 deletions(-) diff --git a/arch/arm/include/asm/stage2_pgtable.h b/arch/arm/include/asm/stage2_pgtable.h index aaceec7855ec..d54ad534da20 100644 --- a/arch/arm/include/asm/stage2_pgtable.h +++ b/arch/arm/include/asm/stage2_pgtable.h @@ -72,4 +72,13 @@ static inline bool kvm_stage2_has_pmd(struct kvm *kvm) return true; } +/* + * The ARMv8.4-TTL extension doesn't apply to AArch32, so the below is + * only to keep things compiling. + */ +#define S2_NO_LEVEL_HINT 0 +#define S2_PUD_LEVEL S2_NO_LEVEL_HINT +#define S2_PMD_LEVEL S2_NO_LEVEL_HINT +#define S2_PTE_LEVEL S2_NO_LEVEL_HINT + #endif /* __ARM_S2_PGTABLE_H_ */ diff --git a/arch/arm64/include/asm/stage2_pgtable.h b/arch/arm64/include/asm/stage2_pgtable.h index 326aac658b9d..7ed5c1a769a9 100644 --- a/arch/arm64/include/asm/stage2_pgtable.h +++ b/arch/arm64/include/asm/stage2_pgtable.h @@ -230,4 +230,13 @@ stage2_pgd_addr_end(struct kvm *kvm, phys_addr_t addr, phys_addr_t end) return (boundary - 1 < end - 1) ? boundary : end; } +/* + * Level values for the ARMv8.4-TTL extension, mapping PUD/PMD/PTE and + * the architectural page-table level. + */ +#define S2_NO_LEVEL_HINT 0 +#define S2_PUD_LEVEL 1 +#define S2_PMD_LEVEL 2 +#define S2_PTE_LEVEL 3 + #endif /* __ARM64_S2_PGTABLE_H_ */ diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index 67752c2a615f..f6ea17f55712 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -74,9 +74,10 @@ void kvm_flush_remote_tlbs(struct kvm *kvm) } } -static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa) +static void kvm_tlb_flush_vmid_ipa(struct kvm_s2_mmu *mmu, phys_addr_t ipa, + int level) { - kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa, 0); + kvm_call_hyp(__kvm_tlb_flush_vmid_ipa, mmu, ipa, level); } /* @@ -118,7 +119,7 @@ static void stage2_dissolve_pmd(struct kvm_s2_mmu *mmu, phys_addr_t addr, pmd_t return; pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PMD_LEVEL); put_page(virt_to_page(pmd)); } @@ -138,7 +139,7 @@ static void stage2_dissolve_pud(struct kvm_s2_mmu *mmu, phys_addr_t addr, pud_t return; stage2_pud_clear(kvm, pudp); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PUD_LEVEL); put_page(virt_to_page(pudp)); } @@ -180,7 +181,7 @@ static void clear_stage2_pgd_entry(struct kvm_s2_mmu *mmu, pgd_t *pgd, phys_addr pud_t *pud_table __maybe_unused = stage2_pud_offset(kvm, pgd, 0UL); stage2_pgd_clear(kvm, pgd); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_NO_LEVEL_HINT); stage2_pud_free(kvm, pud_table); put_page(virt_to_page(pgd)); } @@ -192,7 +193,7 @@ static void clear_stage2_pud_entry(struct kvm_s2_mmu *mmu, pud_t *pud, phys_addr pmd_t *pmd_table __maybe_unused = stage2_pmd_offset(kvm, pud, 0); VM_BUG_ON(stage2_pud_huge(kvm, *pud)); stage2_pud_clear(kvm, pud); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_NO_LEVEL_HINT); stage2_pmd_free(kvm, pmd_table); put_page(virt_to_page(pud)); } @@ -202,7 +203,7 @@ static void clear_stage2_pmd_entry(struct kvm_s2_mmu *mmu, pmd_t *pmd, phys_addr pte_t *pte_table = pte_offset_kernel(pmd, 0); VM_BUG_ON(pmd_thp_or_huge(*pmd)); pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_NO_LEVEL_HINT); free_page((unsigned long)pte_table); put_page(virt_to_page(pmd)); } @@ -272,7 +273,7 @@ static void unmap_stage2_ptes(struct kvm_s2_mmu *mmu, pmd_t *pmd, pte_t old_pte = *pte; kvm_set_pte(pte, __pte(0)); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PTE_LEVEL); /* No need to invalidate the cache for device mappings */ if (!kvm_is_device_pfn(pte_pfn(old_pte))) @@ -301,7 +302,7 @@ static void unmap_stage2_pmds(struct kvm_s2_mmu *mmu, pud_t *pud, pmd_t old_pmd = *pmd; pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PMD_LEVEL); kvm_flush_dcache_pmd(old_pmd); @@ -331,7 +332,7 @@ static void unmap_stage2_puds(struct kvm_s2_mmu *mmu, pgd_t *pgd, pud_t old_pud = *pud; stage2_pud_clear(kvm, pud); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PUD_LEVEL); kvm_flush_dcache_pud(old_pud); put_page(virt_to_page(pud)); } else { @@ -1158,7 +1159,7 @@ static int stage2_set_pmd_huge(struct kvm_s2_mmu *mmu, */ WARN_ON_ONCE(pmd_pfn(old_pmd) != pmd_pfn(*new_pmd)); pmd_clear(pmd); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PMD_LEVEL); } else { get_page(virt_to_page(pmd)); } @@ -1200,7 +1201,7 @@ static int stage2_set_pud_huge(struct kvm_s2_mmu *mmu, WARN_ON_ONCE(kvm_pud_pfn(old_pud) != kvm_pud_pfn(*new_pudp)); stage2_pud_clear(kvm, pudp); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PUD_LEVEL); } else { get_page(virt_to_page(pudp)); } @@ -1349,7 +1350,7 @@ static int stage2_set_pte(struct kvm_s2_mmu *mmu, return 0; kvm_set_pte(pte, __pte(0)); - kvm_tlb_flush_vmid_ipa(mmu, addr); + kvm_tlb_flush_vmid_ipa(mmu, addr, S2_PTE_LEVEL); } else { get_page(virt_to_page(pte)); } From patchwork Tue Feb 11 17:49:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376191 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E08891395 for ; Tue, 11 Feb 2020 17:51:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C166F214DB for ; Tue, 11 Feb 2020 17:51:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443507; bh=/f2Xo9z6ABnZFrjbNY2DUZXX0BqaADfqp97+fkRaLFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=2EFlEFbUfovCByogHYHoN7kYvE7MPT3bNQFjgaPZlpoFiU+RKXI2Kp09jyyn0Vmhv yn9GfNwMOOgktgwsQa0gpWRLtAXZpbOT0sHFZ86ogFpFL9xNYNRZDO/2WTP0MThsV3 9dgGz95fQLDAyvX8rd1IpmDgPpLEvru1zxv/7UUg= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730436AbgBKRvr (ORCPT ); Tue, 11 Feb 2020 12:51:47 -0500 Received: from mail.kernel.org ([198.145.29.99]:54308 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730304AbgBKRvq (ORCPT ); Tue, 11 Feb 2020 12:51:46 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3E71120661; Tue, 11 Feb 2020 17:51:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443505; bh=/f2Xo9z6ABnZFrjbNY2DUZXX0BqaADfqp97+fkRaLFA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=T6XlXcVElAGuLukxHE8s/3/xKQky48ltUFJ+DGNxA5rgyeI1FJFCRokoQ+cgmGnUh tah2TFs5vNMX/sFquSH4qKSv0Coe4Uc8YkOCe/tuVGlAbm/3STT8g7RUMTKaKpfaI9 /28iyb2Cf8IRNq+3Fi5u95UkPbOk5uhbLmPebnIs= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgA-004O7k-Os; Tue, 11 Feb 2020 17:50:34 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 70/94] arm64: KVM: nv: Add include containing the VNCR_EL2 offsets Date: Tue, 11 Feb 2020 17:49:14 +0000 Message-Id: <20200211174938.27809-71-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org VNCR_EL2 points to a page containing a number of system registers accessed by a guest hypervisor when ARMv8.4-NV is enabled. Let's document the offsets in that page, as we are going to use this layout. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/vncr_mapping.h | 73 +++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100644 arch/arm64/include/asm/vncr_mapping.h diff --git a/arch/arm64/include/asm/vncr_mapping.h b/arch/arm64/include/asm/vncr_mapping.h new file mode 100644 index 000000000000..64c46d658fc8 --- /dev/null +++ b/arch/arm64/include/asm/vncr_mapping.h @@ -0,0 +1,73 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * System register offsets in the VNCR page + * All offsets are *byte* displacements! + */ + +#ifndef __ARM64_VNCR_MAPPING_H__ +#define __ARM64_VNCR_MAPPING_H__ + +#define VNCR_VTTBR_EL2 0x020 +#define VNCR_VTCR_EL2 0x040 +#define VNCR_VMPIDR_EL2 0x050 +#define VNCR_CNTVOFF_EL2 0x060 +#define VNCR_HCR_EL2 0x078 +#define VNCR_HSTR_EL2 0x080 +#define VNCR_VPIDR_EL2 0x088 +#define VNCR_TPIDR_EL2 0x090 +#define VNCR_VNCR_EL2 0x0B0 +#define VNCR_CPACR_EL1 0x100 +#define VNCR_CONTEXTIDR_EL1 0x108 +#define VNCR_SCTLR_EL1 0x110 +#define VNCR_ACTLR_EL1 0x118 +#define VNCR_TCR_EL1 0x120 +#define VNCR_AFSR0_EL1 0x128 +#define VNCR_AFSR1_EL1 0x130 +#define VNCR_ESR_EL1 0x138 +#define VNCR_MAIR_EL1 0x140 +#define VNCR_AMAIR_EL1 0x148 +#define VNCR_MDSCR_EL1 0x158 +#define VNCR_SPSR_EL1 0x160 +#define VNCR_CNTV_CVAL_EL0 0x168 +#define VNCR_CNTV_CTL_EL0 0x170 +#define VNCR_CNTP_CVAL_EL0 0x178 +#define VNCR_CNTP_CTL_EL0 0x180 +#define VNCR_SCXTNUM_EL1 0x188 +#define VNCR_ZCR_EL1 0x1E0 +#define VNCR_TTBR0_EL1 0x200 +#define VNCR_TTBR1_EL1 0x210 +#define VNCR_FAR_EL1 0x220 +#define VNCR_ELR_EL1 0x230 +#define VNCR_SP_EL1 0x240 +#define VNCR_VBAR_EL1 0x250 +#define VNCR_ICH_LR0_EL2 0x400 +// VNCR_ICH_LRN_EL2(n) VNCR_ICH_LR0_EL2+8*((n) & 7) +#define VNCR_ICH_AP0R0_EL2 0x480 +// VNCR_ICH_AP0RN_EL2(n) VNCR_ICH_AP0R0_EL2+8*((n) & 3) +#define VNCR_ICH_AP1R0_EL2 0x4A0 +// VNCR_ICH_AP1RN_EL2(n) VNCR_ICH_AP1R0_EL2+8*((n) & 3) +#define VNCR_ICH_HCR_EL2 0x4C0 +#define VNCR_ICH_VMCR_EL2 0x4C8 +#define VNCR_VDISR_EL2 0x500 +#define VNCR_PMBLIMITR_EL1 0x800 +#define VNCR_PMBPTR_EL1 0x810 +#define VNCR_PMBSR_EL1 0x820 +#define VNCR_PMSCR_EL1 0x828 +#define VNCR_PMSEVFR_EL1 0x830 +#define VNCR_PMSICR_EL1 0x838 +#define VNCR_PMSIRR_EL1 0x840 +#define VNCR_PMSLATFR_EL1 0x848 +#define VNCR_TRFCR_EL1 0x880 +#define VNCR_MPAM1_EL1 0x900 +#define VNCR_MPAMHCR_EL2 0x930 +#define VNCR_MPAMVPMV_EL2 0x938 +#define VNCR_MPAMVPM0_EL2 0x940 +#define VNCR_MPAMVPM1_EL2 0x948 +#define VNCR_MPAMVPM2_EL2 0x950 +#define VNCR_MPAMVPM3_EL2 0x958 +#define VNCR_MPAMVPM4_EL2 0x960 +#define VNCR_MPAMVPM5_EL2 0x968 +#define VNCR_MPAMVPM6_EL2 0x970 +#define VNCR_MPAMVPM7_EL2 0x978 + +#endif /* __ARM64_VNCR_MAPPING_H__ */ From patchwork Tue Feb 11 17:49:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376293 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AAA9314B4 for ; Tue, 11 Feb 2020 17:53:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 89FF9206D7 for ; Tue, 11 Feb 2020 17:53:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443597; bh=jkEJiw70D7ZC16bY1qatVu4X4hy1ieYoAyg/qR1PBio=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=iSmBx6/9D8mg9ByWz4fins9hlBa24tO3m/QYsR3c/47Xd4YQErNudkktZIUPDuoZg gI+tD2K/Mntanuk6qiioNQcbMnlmFCQ6HnzwoAyzH3SF/NP2g7kgOW7c/XF2fD7b2v oenHkiXUZbksNYfkCGP5ABpX4DM4eQtUhLBlmLXo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730677AbgBKRxR (ORCPT ); Tue, 11 Feb 2020 12:53:17 -0500 Received: from mail.kernel.org ([198.145.29.99]:56762 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730627AbgBKRxQ (ORCPT ); Tue, 11 Feb 2020 12:53:16 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B6C4420578; Tue, 11 Feb 2020 17:53:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443595; bh=jkEJiw70D7ZC16bY1qatVu4X4hy1ieYoAyg/qR1PBio=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=G5TYaHTaVBPzTTV7b/Qc06w+7dCPtYwRuSnSRBohgRy7z8JnBqS3bXNk4/4UK6onI inX3m2U1UMtPVGW3FXJWfHeJrrv/Qggp/MThzwlamAeQiEmoXk2zFYcg7iZn73XMMH PtCJ/X5YhgU1yHY5Xy6K58DZnRaLUp41tSu1V/ik= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgB-004O7k-Bw; Tue, 11 Feb 2020 17:50:35 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 71/94] KVM: arm64: Introduce accessor for ctxt->sys_reg Date: Tue, 11 Feb 2020 17:49:15 +0000 Message-Id: <20200211174938.27809-72-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In order to allow the disintegration of the per-vcpu sysreg array, let's introduce a new helper (ctxt_sys_reg()) that returns the in-memory copy of a system register, picked from a given context. __vcpu_sys_reg() is rewritten to use this helper. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 06c4b7aaebab..00748ceb9769 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -468,12 +468,17 @@ struct kvm_vcpu_arch { #define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs) /* - * Only use __vcpu_sys_reg if you know you want the memory backed version of a - * register, and not the one most recently accessed by a running VCPU. For - * example, for userspace access or for system registers that are never context - * switched, but only emulated. + * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the + * memory backed version of a register, and not the one most recently + * accessed by a running VCPU. For example, for userspace access or + * for system registers that are never context switched, but only + * emulated. */ -#define __vcpu_sys_reg(v,r) ((v)->arch.ctxt.sys_regs[(r)]) +#define __ctxt_sys_reg(c,r) (&(c)->sys_regs[(r)]) + +#define ctxt_sys_reg(c,r) (*__ctxt_sys_reg(c,r)) + +#define __vcpu_sys_reg(v,r) (ctxt_sys_reg(&(v)->arch.ctxt, (r))) u64 vcpu_read_sys_reg(const struct kvm_vcpu *vcpu, int reg); void vcpu_write_sys_reg(struct kvm_vcpu *vcpu, u64 val, int reg); From patchwork Tue Feb 11 17:49:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376265 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5CA2114B4 for ; Tue, 11 Feb 2020 17:52:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 18B1A206CC for ; Tue, 11 Feb 2020 17:52:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443575; bh=WOGfIiGa8LmQbXWRddZTVbs63o9xNFRhXD3mTs4iyeM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=XgxV2LbKW3spwNwKdNQVvBv8Ftm9PjVLS/l5JREQUJnyC41+0D+okMUck2CAsHz0a 6oePk62Sx96nSAWxXQGc4dOjNnHfGDuvVNzj2gq/lYPUxoxvSAkZGdj9yNNX0poBsf sevTErQN72YPp7NtXXjcH8FTXBYNJx0WJq0TYv+8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730615AbgBKRwy (ORCPT ); Tue, 11 Feb 2020 12:52:54 -0500 Received: from mail.kernel.org ([198.145.29.99]:56074 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730555AbgBKRwx (ORCPT ); Tue, 11 Feb 2020 12:52:53 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3B4A220578; Tue, 11 Feb 2020 17:52:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443571; bh=WOGfIiGa8LmQbXWRddZTVbs63o9xNFRhXD3mTs4iyeM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=2KyB9IqLM9pbbXbrrHK5zR8gsdUzwYUGqKNuUQJQZCyNlEIXudGqub9JlDKbwJ2Pr 0Yd94oGx97ho1E7NbgBo7a5ZGx9qWtCSp9fvXtQsuD+H/UGFSicelZmlz00Fy2bWQL M5N3fyXuT/PaG2ZWWX18K0Iky9H0vdfScxqyPDPU= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgB-004O7k-UT; Tue, 11 Feb 2020 17:50:36 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 72/94] KVM: arm64: sysreg: Use ctxt_sys_reg() instead of raw sys_regs access Date: Tue, 11 Feb 2020 17:49:16 +0000 Message-Id: <20200211174938.27809-73-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that we have a wrapper for the sysreg accesses, let's use that consistently. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 6 +- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/hyp/at.c | 30 ++--- arch/arm64/kvm/hyp/switch.c | 6 +- arch/arm64/kvm/hyp/sysreg-sr.c | 192 +++++++++++++-------------- 5 files changed, 118 insertions(+), 118 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 1ccd98b5fead..1ff1735ff2b3 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -244,7 +244,7 @@ static inline bool vcpu_mode_el2(const struct kvm_vcpu *vcpu) static inline bool __vcpu_el2_e2h_is_set(const struct kvm_cpu_context *ctxt) { - return ctxt->sys_regs[HCR_EL2] & HCR_E2H; + return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_E2H; } static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu) @@ -254,7 +254,7 @@ static inline bool vcpu_el2_e2h_is_set(const struct kvm_vcpu *vcpu) static inline bool __vcpu_el2_tge_is_set(const struct kvm_cpu_context *ctxt) { - return ctxt->sys_regs[HCR_EL2] & HCR_TGE; + return ctxt_sys_reg(ctxt, HCR_EL2) & HCR_TGE; } static inline bool vcpu_el2_tge_is_set(const struct kvm_vcpu *vcpu) @@ -303,7 +303,7 @@ static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 * register has still the value we saved on the last write. */ if ((val & 0xc) == 0) - return ctxt->sys_regs[SPSR_EL2]; + return ctxt_sys_reg(ctxt, SPSR_EL2); /* * Otherwise there was a "local" exception on the CPU, diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 00748ceb9769..83ed3865dda8 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -609,7 +609,7 @@ DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data); static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt) { /* The host's MPIDR is immutable, so let's set it up at boot time */ - cpu_ctxt->sys_regs[MPIDR_EL1] = read_cpuid_mpidr(); + ctxt_sys_reg(cpu_ctxt, MPIDR_EL1) = read_cpuid_mpidr(); } void __kvm_enable_ssbs(void); diff --git a/arch/arm64/kvm/hyp/at.c b/arch/arm64/kvm/hyp/at.c index 9fdb87fa3597..a703e811291a 100644 --- a/arch/arm64/kvm/hyp/at.c +++ b/arch/arm64/kvm/hyp/at.c @@ -83,10 +83,10 @@ void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) /* We've trapped, so everything is live on the CPU. */ __mmu_config_save(&config); - write_sysreg_el1(ctxt->sys_regs[TTBR0_EL1], SYS_TTBR0); - write_sysreg_el1(ctxt->sys_regs[TTBR1_EL1], SYS_TTBR1); - write_sysreg_el1(ctxt->sys_regs[TCR_EL1], SYS_TCR); - write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL1), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL1), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR); write_sysreg(kvm_get_vttbr(mmu), vttbr_el2); /* * REVISIT: do we need anything from the guest's VTCR_EL2? If @@ -122,7 +122,7 @@ void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) isb(); - ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); /* * Failed? let's leave the building now. @@ -131,7 +131,7 @@ void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) * wasn't populated? We may need to perform a SW PTW, * populating our shadow S2 and retry the instruction. */ - if (ctxt->sys_regs[PAR_EL1] & 1) + if (ctxt_sys_reg(ctxt, PAR_EL1) & 1) goto nopan; /* No PAN? No problem. */ @@ -162,7 +162,7 @@ void __kvm_at_s1e01(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) * should return the real fault level. */ if (!(read_sysreg(par_el1) & 1)) - ctxt->sys_regs[PAR_EL1] = 0x1f; + ctxt_sys_reg(ctxt, PAR_EL1) = 0x1f; nopan: if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) @@ -187,17 +187,17 @@ void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) __mmu_config_save(&config); if (vcpu_el2_e2h_is_set(vcpu)) { - write_sysreg_el1(ctxt->sys_regs[TTBR0_EL2], SYS_TTBR0); - write_sysreg_el1(ctxt->sys_regs[TTBR1_EL2], SYS_TTBR1); - write_sysreg_el1(ctxt->sys_regs[TCR_EL2], SYS_TCR); - write_sysreg_el1(ctxt->sys_regs[SCTLR_EL2], SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL2), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL2), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL2), SYS_SCTLR); val = config.hcr; } else { - write_sysreg_el1(ctxt->sys_regs[TTBR0_EL2], SYS_TTBR0); - write_sysreg_el1(translate_tcr(ctxt->sys_regs[TCR_EL2]), + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + write_sysreg_el1(translate_tcr(ctxt_sys_reg(ctxt, TCR_EL2)), SYS_TCR); - write_sysreg_el1(translate_sctlr(ctxt->sys_regs[SCTLR_EL2]), + write_sysreg_el1(translate_sctlr(ctxt_sys_reg(ctxt, SCTLR_EL2)), SYS_SCTLR); val = config.hcr | HCR_NV | HCR_NV1; @@ -224,7 +224,7 @@ void __kvm_at_s1e2(struct kvm_vcpu *vcpu, u32 op, u64 vaddr) isb(); /* FIXME: handle failed translation due to shadow S2 */ - ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); __mmu_config_restore(&config); spin_unlock(&vcpu->kvm->mmu_lock); diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 97c5c1a791b8..c5405fa68bc9 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -138,9 +138,9 @@ static void __hyp_text __activate_traps_nvhe(struct kvm_vcpu *vcpu) * configured and enabled. We can now restore the guest's S1 * configuration: SCTLR, and only then TCR. */ - write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR); isb(); - write_sysreg_el1(ctxt->sys_regs[TCR_EL1], SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); } } @@ -463,7 +463,7 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) sve_load_state(vcpu_sve_pffr(vcpu), &vcpu->arch.ctxt.gp_regs.fp_regs.fpsr, sve_vq_from_vl(vcpu->arch.sve_max_vl) - 1); - write_sysreg_s(vcpu->arch.ctxt.sys_regs[ZCR_EL1], SYS_ZCR_EL12); + write_sysreg_s(__vcpu_sys_reg(vcpu, ZCR_EL1), SYS_ZCR_EL12); } else { __fpsimd_restore_state(&vcpu->arch.ctxt.gp_regs.fp_regs); } diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index 384f0da6b4f3..df9ca6fdf3fb 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -26,7 +26,7 @@ static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt) { - ctxt->sys_regs[MDSCR_EL1] = read_sysreg(mdscr_el1); + ctxt_sys_reg(ctxt, MDSCR_EL1) = read_sysreg(mdscr_el1); /* * The host arm64 Linux uses sp_el0 to point to 'current' and it must @@ -37,26 +37,26 @@ static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt) static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt) { - ctxt->sys_regs[TPIDR_EL0] = read_sysreg(tpidr_el0); - ctxt->sys_regs[TPIDRRO_EL0] = read_sysreg(tpidrro_el0); + ctxt_sys_reg(ctxt, TPIDR_EL0) = read_sysreg(tpidr_el0); + ctxt_sys_reg(ctxt, TPIDRRO_EL0) = read_sysreg(tpidrro_el0); } static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) { - ctxt->sys_regs[SCTLR_EL1] = read_sysreg_el1(SYS_SCTLR); - ctxt->sys_regs[CPACR_EL1] = read_sysreg_el1(SYS_CPACR); - ctxt->sys_regs[TTBR0_EL1] = read_sysreg_el1(SYS_TTBR0); - ctxt->sys_regs[TTBR1_EL1] = read_sysreg_el1(SYS_TTBR1); - ctxt->sys_regs[TCR_EL1] = read_sysreg_el1(SYS_TCR); - ctxt->sys_regs[ESR_EL1] = read_sysreg_el1(SYS_ESR); - ctxt->sys_regs[AFSR0_EL1] = read_sysreg_el1(SYS_AFSR0); - ctxt->sys_regs[AFSR1_EL1] = read_sysreg_el1(SYS_AFSR1); - ctxt->sys_regs[FAR_EL1] = read_sysreg_el1(SYS_FAR); - ctxt->sys_regs[MAIR_EL1] = read_sysreg_el1(SYS_MAIR); - ctxt->sys_regs[VBAR_EL1] = read_sysreg_el1(SYS_VBAR); - ctxt->sys_regs[CONTEXTIDR_EL1] = read_sysreg_el1(SYS_CONTEXTIDR); - ctxt->sys_regs[AMAIR_EL1] = read_sysreg_el1(SYS_AMAIR); - ctxt->sys_regs[CNTKCTL_EL1] = read_sysreg_el1(SYS_CNTKCTL); + ctxt_sys_reg(ctxt, SCTLR_EL1) = read_sysreg_el1(SYS_SCTLR); + ctxt_sys_reg(ctxt, CPACR_EL1) = read_sysreg_el1(SYS_CPACR); + ctxt_sys_reg(ctxt, TTBR0_EL1) = read_sysreg_el1(SYS_TTBR0); + ctxt_sys_reg(ctxt, TTBR1_EL1) = read_sysreg_el1(SYS_TTBR1); + ctxt_sys_reg(ctxt, TCR_EL1) = read_sysreg_el1(SYS_TCR); + ctxt_sys_reg(ctxt, ESR_EL1) = read_sysreg_el1(SYS_ESR); + ctxt_sys_reg(ctxt, AFSR0_EL1) = read_sysreg_el1(SYS_AFSR0); + ctxt_sys_reg(ctxt, AFSR1_EL1) = read_sysreg_el1(SYS_AFSR1); + ctxt_sys_reg(ctxt, FAR_EL1) = read_sysreg_el1(SYS_FAR); + ctxt_sys_reg(ctxt, MAIR_EL1) = read_sysreg_el1(SYS_MAIR); + ctxt_sys_reg(ctxt, VBAR_EL1) = read_sysreg_el1(SYS_VBAR); + ctxt_sys_reg(ctxt, CONTEXTIDR_EL1) = read_sysreg_el1(SYS_CONTEXTIDR); + ctxt_sys_reg(ctxt, AMAIR_EL1) = read_sysreg_el1(SYS_AMAIR); + ctxt_sys_reg(ctxt, CNTKCTL_EL1) = read_sysreg_el1(SYS_CNTKCTL); ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1); ctxt->gp_regs.elr_el1 = read_sysreg_el1(SYS_ELR); @@ -65,14 +65,14 @@ static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) { - ctxt->sys_regs[ESR_EL2] = read_sysreg_el1(SYS_ESR); - ctxt->sys_regs[AFSR0_EL2] = read_sysreg_el1(SYS_AFSR0); - ctxt->sys_regs[AFSR1_EL2] = read_sysreg_el1(SYS_AFSR1); - ctxt->sys_regs[FAR_EL2] = read_sysreg_el1(SYS_FAR); - ctxt->sys_regs[MAIR_EL2] = read_sysreg_el1(SYS_MAIR); - ctxt->sys_regs[VBAR_EL2] = read_sysreg_el1(SYS_VBAR); - ctxt->sys_regs[CONTEXTIDR_EL2] = read_sysreg_el1(SYS_CONTEXTIDR); - ctxt->sys_regs[AMAIR_EL2] = read_sysreg_el1(SYS_AMAIR); + ctxt_sys_reg(ctxt, ESR_EL2) = read_sysreg_el1(SYS_ESR); + ctxt_sys_reg(ctxt, AFSR0_EL2) = read_sysreg_el1(SYS_AFSR0); + ctxt_sys_reg(ctxt, AFSR1_EL2) = read_sysreg_el1(SYS_AFSR1); + ctxt_sys_reg(ctxt, FAR_EL2) = read_sysreg_el1(SYS_FAR); + ctxt_sys_reg(ctxt, MAIR_EL2) = read_sysreg_el1(SYS_MAIR); + ctxt_sys_reg(ctxt, VBAR_EL2) = read_sysreg_el1(SYS_VBAR); + ctxt_sys_reg(ctxt, CONTEXTIDR_EL2) = read_sysreg_el1(SYS_CONTEXTIDR); + ctxt_sys_reg(ctxt, AMAIR_EL2) = read_sysreg_el1(SYS_AMAIR); /* * In VHE mode those registers are compatible between EL1 and EL2, @@ -83,25 +83,25 @@ static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) * to save anything here. */ if (__vcpu_el2_e2h_is_set(ctxt)) { - ctxt->sys_regs[SCTLR_EL2] = read_sysreg_el1(SYS_SCTLR); - ctxt->sys_regs[CPTR_EL2] = read_sysreg_el1(SYS_CPACR); - ctxt->sys_regs[TTBR0_EL2] = read_sysreg_el1(SYS_TTBR0); - ctxt->sys_regs[TTBR1_EL2] = read_sysreg_el1(SYS_TTBR1); - ctxt->sys_regs[TCR_EL2] = read_sysreg_el1(SYS_TCR); - ctxt->sys_regs[CNTHCTL_EL2] = read_sysreg_el1(SYS_CNTKCTL); + ctxt_sys_reg(ctxt, SCTLR_EL2) = read_sysreg_el1(SYS_SCTLR); + ctxt_sys_reg(ctxt, CPTR_EL2) = read_sysreg_el1(SYS_CPACR); + ctxt_sys_reg(ctxt, TTBR0_EL2) = read_sysreg_el1(SYS_TTBR0); + ctxt_sys_reg(ctxt, TTBR1_EL2) = read_sysreg_el1(SYS_TTBR1); + ctxt_sys_reg(ctxt, TCR_EL2) = read_sysreg_el1(SYS_TCR); + ctxt_sys_reg(ctxt, CNTHCTL_EL2) = read_sysreg_el1(SYS_CNTKCTL); } - ctxt->sys_regs[SP_EL2] = read_sysreg(sp_el1); - ctxt->sys_regs[ELR_EL2] = read_sysreg_el1(SYS_ELR); - ctxt->sys_regs[SPSR_EL2] = __fixup_spsr_el2_read(ctxt, read_sysreg_el1(SYS_SPSR)); + ctxt_sys_reg(ctxt, SP_EL2) = read_sysreg(sp_el1); + ctxt_sys_reg(ctxt, ELR_EL2) = read_sysreg_el1(SYS_ELR); + ctxt_sys_reg(ctxt, SPSR_EL2) = __fixup_spsr_el2_read(ctxt, read_sysreg_el1(SYS_SPSR)); } static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) { - ctxt->sys_regs[CSSELR_EL1] = read_sysreg(csselr_el1); - ctxt->sys_regs[ACTLR_EL1] = read_sysreg(actlr_el1); - ctxt->sys_regs[PAR_EL1] = read_sysreg(par_el1); - ctxt->sys_regs[TPIDR_EL1] = read_sysreg(tpidr_el1); + ctxt_sys_reg(ctxt, CSSELR_EL1) = read_sysreg(csselr_el1); + ctxt_sys_reg(ctxt, ACTLR_EL1) = read_sysreg(actlr_el1); + ctxt_sys_reg(ctxt, PAR_EL1) = read_sysreg(par_el1); + ctxt_sys_reg(ctxt, TPIDR_EL1) = read_sysreg(tpidr_el1); if (unlikely(__is_hyp_ctxt(ctxt))) __sysreg_save_vel2_state(ctxt); @@ -137,7 +137,7 @@ static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ct ctxt->gp_regs.regs.pstate = from_hw_pstate(ctxt); if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN)) - ctxt->sys_regs[DISR_EL1] = read_sysreg_s(SYS_VDISR_EL2); + ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2); } void __hyp_text __sysreg_save_state_nvhe(struct kvm_cpu_context *ctxt) @@ -163,7 +163,7 @@ NOKPROBE_SYMBOL(sysreg_save_guest_state_vhe); static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctxt) { - write_sysreg(ctxt->sys_regs[MDSCR_EL1], mdscr_el1); + write_sysreg(ctxt_sys_reg(ctxt, MDSCR_EL1), mdscr_el1); /* * The host arm64 Linux uses sp_el0 to point to 'current' and it must @@ -174,8 +174,8 @@ static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctx static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) { - write_sysreg(ctxt->sys_regs[TPIDR_EL0], tpidr_el0); - write_sysreg(ctxt->sys_regs[TPIDRRO_EL0], tpidrro_el0); + write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL0), tpidr_el0); + write_sysreg(ctxt_sys_reg(ctxt, TPIDRRO_EL0), tpidrro_el0); } static void __sysreg_restore_vel2_state(struct kvm_cpu_context *ctxt) @@ -183,33 +183,33 @@ static void __sysreg_restore_vel2_state(struct kvm_cpu_context *ctxt) u64 val; write_sysreg(read_cpuid_id(), vpidr_el2); - write_sysreg(ctxt->sys_regs[MPIDR_EL1], vmpidr_el2); - write_sysreg_el1(ctxt->sys_regs[MAIR_EL2], SYS_MAIR); - write_sysreg_el1(ctxt->sys_regs[VBAR_EL2], SYS_VBAR); - write_sysreg_el1(ctxt->sys_regs[CONTEXTIDR_EL2],SYS_CONTEXTIDR); - write_sysreg_el1(ctxt->sys_regs[AMAIR_EL2], SYS_AMAIR); + write_sysreg(ctxt_sys_reg(ctxt, MPIDR_EL1), vmpidr_el2); + write_sysreg_el1(ctxt_sys_reg(ctxt, MAIR_EL2), SYS_MAIR); + write_sysreg_el1(ctxt_sys_reg(ctxt, VBAR_EL2), SYS_VBAR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CONTEXTIDR_EL2),SYS_CONTEXTIDR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AMAIR_EL2), SYS_AMAIR); if (__vcpu_el2_e2h_is_set(ctxt)) { /* * In VHE mode those registers are compatible between * EL1 and EL2. */ - write_sysreg_el1(ctxt->sys_regs[SCTLR_EL2], SYS_SCTLR); - write_sysreg_el1(ctxt->sys_regs[CPTR_EL2], SYS_CPACR); - write_sysreg_el1(ctxt->sys_regs[TTBR0_EL2], SYS_TTBR0); - write_sysreg_el1(ctxt->sys_regs[TTBR1_EL2], SYS_TTBR1); - write_sysreg_el1(ctxt->sys_regs[TCR_EL2], SYS_TCR); - write_sysreg_el1(ctxt->sys_regs[CNTHCTL_EL2], SYS_CNTKCTL); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL2), SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CPTR_EL2), SYS_CPACR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL2), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL2), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL2), SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CNTHCTL_EL2), SYS_CNTKCTL); } else { - write_sysreg_el1(translate_sctlr(ctxt->sys_regs[SCTLR_EL2]), + write_sysreg_el1(translate_sctlr(ctxt_sys_reg(ctxt, SCTLR_EL2)), SYS_SCTLR); - write_sysreg_el1(translate_cptr(ctxt->sys_regs[CPTR_EL2]), + write_sysreg_el1(translate_cptr(ctxt_sys_reg(ctxt, CPTR_EL2)), SYS_CPACR); - write_sysreg_el1(translate_ttbr0(ctxt->sys_regs[TTBR0_EL2]), + write_sysreg_el1(translate_ttbr0(ctxt_sys_reg(ctxt, TTBR0_EL2)), SYS_TTBR0); - write_sysreg_el1(translate_tcr(ctxt->sys_regs[TCR_EL2]), + write_sysreg_el1(translate_tcr(ctxt_sys_reg(ctxt, TCR_EL2)), SYS_TCR); - write_sysreg_el1(translate_cnthctl(ctxt->sys_regs[CNTHCTL_EL2]), + write_sysreg_el1(translate_cnthctl(ctxt_sys_reg(ctxt, CNTHCTL_EL2)), SYS_CNTKCTL); } @@ -217,14 +217,14 @@ static void __sysreg_restore_vel2_state(struct kvm_cpu_context *ctxt) * These registers can be modified behind our back by a fault * taken inside vEL2. Save them, always. */ - write_sysreg_el1(ctxt->sys_regs[ESR_EL2], SYS_ESR); - write_sysreg_el1(ctxt->sys_regs[AFSR0_EL2], SYS_AFSR0); - write_sysreg_el1(ctxt->sys_regs[AFSR1_EL2], SYS_AFSR1); - write_sysreg_el1(ctxt->sys_regs[FAR_EL2], SYS_FAR); - write_sysreg(ctxt->sys_regs[SP_EL2], sp_el1); - write_sysreg_el1(ctxt->sys_regs[ELR_EL2], SYS_ELR); - - val = __fixup_spsr_el2_write(ctxt, ctxt->sys_regs[SPSR_EL2]); + write_sysreg_el1(ctxt_sys_reg(ctxt, ESR_EL2), SYS_ESR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR0_EL2), SYS_AFSR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR1_EL2), SYS_AFSR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, FAR_EL2), SYS_FAR); + write_sysreg(ctxt_sys_reg(ctxt, SP_EL2), sp_el1); + write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL2), SYS_ELR); + + val = __fixup_spsr_el2_write(ctxt, ctxt_sys_reg(ctxt, SPSR_EL2)); write_sysreg_el1(val, SYS_SPSR); } @@ -258,53 +258,53 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) * only time it changes. We'll restore the MIDR_EL1 * view on put. */ - write_sysreg(ctxt->sys_regs[VPIDR_EL2], vpidr_el2); + write_sysreg(ctxt_sys_reg(ctxt, VPIDR_EL2), vpidr_el2); /* * As we're restoring a nested guest, set the value * provided by the guest hypervisor. */ - mpidr = ctxt->sys_regs[VMPIDR_EL2]; + mpidr = ctxt_sys_reg(ctxt, VMPIDR_EL2); } else { - mpidr = ctxt->sys_regs[MPIDR_EL1]; + mpidr = ctxt_sys_reg(ctxt, MPIDR_EL1); } } else { - mpidr = ctxt->sys_regs[MPIDR_EL1]; + mpidr = ctxt_sys_reg(ctxt, MPIDR_EL1); } write_sysreg(mpidr, vmpidr_el2); - write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR); if (!cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE)) { - write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); - write_sysreg_el1(ctxt->sys_regs[TCR_EL1], SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); } else if (!ctxt->__hyp_running_vcpu) { /* * Must only be done for guest registers, hence the context * test. We're coming from the host, so SCTLR.M is already * set. Pairs with __activate_traps_nvhe(). */ - write_sysreg_el1((ctxt->sys_regs[TCR_EL1] | + write_sysreg_el1((ctxt_sys_reg(ctxt, TCR_EL1) | TCR_EPD1_MASK | TCR_EPD0_MASK), SYS_TCR); isb(); } - write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1); - write_sysreg_el1(ctxt->sys_regs[CPACR_EL1], SYS_CPACR); - write_sysreg_el1(ctxt->sys_regs[TTBR0_EL1], SYS_TTBR0); - write_sysreg_el1(ctxt->sys_regs[TTBR1_EL1], SYS_TTBR1); - write_sysreg_el1(ctxt->sys_regs[ESR_EL1], SYS_ESR); - write_sysreg_el1(ctxt->sys_regs[AFSR0_EL1], SYS_AFSR0); - write_sysreg_el1(ctxt->sys_regs[AFSR1_EL1], SYS_AFSR1); - write_sysreg_el1(ctxt->sys_regs[FAR_EL1], SYS_FAR); - write_sysreg_el1(ctxt->sys_regs[MAIR_EL1], SYS_MAIR); - write_sysreg_el1(ctxt->sys_regs[VBAR_EL1], SYS_VBAR); - write_sysreg_el1(ctxt->sys_regs[CONTEXTIDR_EL1],SYS_CONTEXTIDR); - write_sysreg_el1(ctxt->sys_regs[AMAIR_EL1], SYS_AMAIR); - write_sysreg_el1(ctxt->sys_regs[CNTKCTL_EL1], SYS_CNTKCTL); - write_sysreg(ctxt->sys_regs[PAR_EL1], par_el1); - write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1); + write_sysreg(ctxt_sys_reg(ctxt, ACTLR_EL1), actlr_el1); + write_sysreg_el1(ctxt_sys_reg(ctxt, CPACR_EL1), SYS_CPACR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR0_EL1), SYS_TTBR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, TTBR1_EL1), SYS_TTBR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, ESR_EL1), SYS_ESR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR0_EL1), SYS_AFSR0); + write_sysreg_el1(ctxt_sys_reg(ctxt, AFSR1_EL1), SYS_AFSR1); + write_sysreg_el1(ctxt_sys_reg(ctxt, FAR_EL1), SYS_FAR); + write_sysreg_el1(ctxt_sys_reg(ctxt, MAIR_EL1), SYS_MAIR); + write_sysreg_el1(ctxt_sys_reg(ctxt, VBAR_EL1), SYS_VBAR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CONTEXTIDR_EL1),SYS_CONTEXTIDR); + write_sysreg_el1(ctxt_sys_reg(ctxt, AMAIR_EL1), SYS_AMAIR); + write_sysreg_el1(ctxt_sys_reg(ctxt, CNTKCTL_EL1), SYS_CNTKCTL); + write_sysreg(ctxt_sys_reg(ctxt, PAR_EL1), par_el1); + write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL1), tpidr_el1); if (cpus_have_const_cap(ARM64_WORKAROUND_SPECULATIVE_AT_NVHE) && ctxt->__hyp_running_vcpu) { @@ -318,9 +318,9 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) * deconfigured and disabled. We can now restore the host's * S1 configuration: SCTLR, and only then TCR. */ - write_sysreg_el1(ctxt->sys_regs[SCTLR_EL1], SYS_SCTLR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SCTLR_EL1), SYS_SCTLR); isb(); - write_sysreg_el1(ctxt->sys_regs[TCR_EL1], SYS_TCR); + write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); } write_sysreg(ctxt->gp_regs.sp_el1, sp_el1); @@ -330,10 +330,10 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) { - write_sysreg(ctxt->sys_regs[CSSELR_EL1], csselr_el1); - write_sysreg(ctxt->sys_regs[ACTLR_EL1], actlr_el1); - write_sysreg(ctxt->sys_regs[PAR_EL1], par_el1); - write_sysreg(ctxt->sys_regs[TPIDR_EL1], tpidr_el1); + write_sysreg(ctxt_sys_reg(ctxt, CSSELR_EL1), csselr_el1); + write_sysreg(ctxt_sys_reg(ctxt, ACTLR_EL1), actlr_el1); + write_sysreg(ctxt_sys_reg(ctxt, PAR_EL1), par_el1); + write_sysreg(ctxt_sys_reg(ctxt, TPIDR_EL1), tpidr_el1); if (__is_hyp_ctxt(ctxt)) __sysreg_restore_vel2_state(ctxt); @@ -382,7 +382,7 @@ __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) write_sysreg_el2(pstate, SYS_SPSR); if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN)) - write_sysreg_s(ctxt->sys_regs[DISR_EL1], SYS_VDISR_EL2); + write_sysreg_s(ctxt_sys_reg(ctxt, DISR_EL1), SYS_VDISR_EL2); } void __hyp_text __sysreg_restore_state_nvhe(struct kvm_cpu_context *ctxt) From patchwork Tue Feb 11 17:49:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376243 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5644114B4 for ; Tue, 11 Feb 2020 17:52:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2B3AF208C3 for ; Tue, 11 Feb 2020 17:52:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443555; bh=+GxwISj8L9rgC421r+aQwDL/s2TLFKCCFgapyXZ3gcM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=pftik+5t2iiDq918rVBftwMcj2EFw4Z0XYEF9MEPhLFGOMA++XhUFjwwBJf5MIHhM jhan/nb+NkczCGis0Fj5iPbIFaxvdjvMM3cwszUFhyODeEi3xsyfQfEUjaTZlEji7L 9kLPZ4wK+vX8j/0LKPkCsbAbbXhonrq7YwdVdFis= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730552AbgBKRwd (ORCPT ); Tue, 11 Feb 2020 12:52:33 -0500 Received: from mail.kernel.org ([198.145.29.99]:55544 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730561AbgBKRwd (ORCPT ); Tue, 11 Feb 2020 12:52:33 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9539E20848; Tue, 11 Feb 2020 17:52:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443552; bh=+GxwISj8L9rgC421r+aQwDL/s2TLFKCCFgapyXZ3gcM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=f7/rcirA0+aBR5svs3tWmrs1dw7oWNx/QmvgwQJvUmS+cHpP41KuWoCMycJbHSxGN Fat5bTY8ykqrYJXjBCKe+1LC835ZKaGcKFpTfAUgOKd2ecrJcyqXWsi/1t8gzPYjN4 5jh9m3Tw2d2uZ1wRRbBowUPy6Vc6YrD9NqZzCf2w= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgC-004O7k-IW; Tue, 11 Feb 2020 17:50:36 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 73/94] KVM: arm64: sve: Use __vcpu_sys_reg() instead of raw sys_regs access Date: Tue, 11 Feb 2020 17:49:17 +0000 Message-Id: <20200211174938.27809-74-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that we have a wrapper for the sysreg accesses, let's use that consistently. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/fpsimd.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 525010504f9d..738ee805aa3c 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -110,12 +110,10 @@ void kvm_arch_vcpu_put_fp(struct kvm_vcpu *vcpu) local_irq_save(flags); if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { - u64 *guest_zcr = &vcpu->arch.ctxt.sys_regs[ZCR_EL1]; - fpsimd_save_and_flush_cpu_state(); if (guest_has_sve) - *guest_zcr = read_sysreg_s(SYS_ZCR_EL12); + __vcpu_sys_reg(vcpu, ZCR_EL1) = read_sysreg_s(SYS_ZCR_EL12); } else if (host_has_sve) { /* * The FPSIMD/SVE state in the CPU has not been touched, and we From patchwork Tue Feb 11 17:49:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376259 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E795014B4 for ; Tue, 11 Feb 2020 17:52:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C375B206D7 for ; Tue, 11 Feb 2020 17:52:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443568; bh=UoVrt09wjYGMlc6RAiUdRunAn2VHL7X7dd7By3r8axA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=NPxsehBLDESgUhsYzXvwxOIJ/WkG1ajLZlbbrzf+hSu7iaBbh68ey0EI2DoaxRMCP Y5xHDFFcjWSx0oEeaS7HXRON8KtfyU+tI79n+5ZGdT2FNx6VWcoYUxs01lTl7C+8NK soOqaziAGHjeMBrUV1aAb/+CQGySRRqSeFoYPces= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730299AbgBKRws (ORCPT ); Tue, 11 Feb 2020 12:52:48 -0500 Received: from mail.kernel.org ([198.145.29.99]:55942 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730560AbgBKRwq (ORCPT ); Tue, 11 Feb 2020 12:52:46 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 92C1D20661; Tue, 11 Feb 2020 17:52:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443565; bh=UoVrt09wjYGMlc6RAiUdRunAn2VHL7X7dd7By3r8axA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=L4uMU9VbThl1fcl4HvyyvSDn5PRdej1bQJb9B8X9dpWbnWV0QUsKHESUvQk9tIc5L g8XB1TGSBanxkRF3vPtgBiKIlmW71CiX/Wzyv8kn+hA6QKWxnPOq6KS8IitCtq1vls PtzGWPAoKwKc/LNLrxOqxQ2BOyMi7pcV2BrucIIg= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgD-004O7k-57; Tue, 11 Feb 2020 17:50:37 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 74/94] KVM: arm64: pauth: Use ctxt_sys_reg() instead of raw sys_regs access Date: Tue, 11 Feb 2020 17:49:18 +0000 Message-Id: <20200211174938.27809-75-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that we have a wrapper for the sysreg accesses, let's use that consistently. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/handle_exit.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 10f13bd2e319..9678e2389c2f 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -217,10 +217,11 @@ static int handle_sve(struct kvm_vcpu *vcpu, struct kvm_run *run) return 1; } -#define __ptrauth_save_key(regs, key) \ + +#define __ptrauth_save_key(ctxt, key) \ ({ \ - regs[key ## KEYLO_EL1] = read_sysreg_s(SYS_ ## key ## KEYLO_EL1); \ - regs[key ## KEYHI_EL1] = read_sysreg_s(SYS_ ## key ## KEYHI_EL1); \ + ctxt_sys_reg(ctxt, key ## KEYLO_EL1) = read_sysreg_s(SYS_ ## key ## KEYLO_EL1); \ + ctxt_sys_reg(ctxt, key ## KEYHI_EL1) = read_sysreg_s(SYS_ ## key ## KEYHI_EL1); \ }) /* @@ -234,11 +235,11 @@ void kvm_arm_vcpu_ptrauth_trap(struct kvm_vcpu *vcpu) if (vcpu_has_ptrauth(vcpu)) { vcpu_ptrauth_enable(vcpu); ctxt = vcpu->arch.host_cpu_context; - __ptrauth_save_key(ctxt->sys_regs, APIA); - __ptrauth_save_key(ctxt->sys_regs, APIB); - __ptrauth_save_key(ctxt->sys_regs, APDA); - __ptrauth_save_key(ctxt->sys_regs, APDB); - __ptrauth_save_key(ctxt->sys_regs, APGA); + __ptrauth_save_key(ctxt, APIA); + __ptrauth_save_key(ctxt, APIB); + __ptrauth_save_key(ctxt, APDA); + __ptrauth_save_key(ctxt, APDB); + __ptrauth_save_key(ctxt, APGA); } else { kvm_inject_undefined(vcpu); } From patchwork Tue Feb 11 17:49:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376281 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7F4DF1395 for ; Tue, 11 Feb 2020 17:53:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5C207208C3 for ; Tue, 11 Feb 2020 17:53:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443587; bh=BN3tlU9n1B0zRdOhqrbQdngEkobkGgI4KuYwe6IjLYk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=y04UJLTPfiW15z3iAk8a9coVzRzehpXfOzrpS2HShopFIH6FxL092TiH28vsUuHhP LeHvjSOmeZWh679CysIE1EacoZsiQI9USj5aphN0Aoa39kzCwBrcIm7BB7ZS8GOibR yVVZ2iKmHO7GA54jaQw+EciVCiMIYAG3dpxvg2+s= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730661AbgBKRxG (ORCPT ); Tue, 11 Feb 2020 12:53:06 -0500 Received: from mail.kernel.org ([198.145.29.99]:56414 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730604AbgBKRxF (ORCPT ); Tue, 11 Feb 2020 12:53:05 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8942C20661; Tue, 11 Feb 2020 17:53:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443584; bh=BN3tlU9n1B0zRdOhqrbQdngEkobkGgI4KuYwe6IjLYk=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=BuKsD7yyzUGPkfznB6PX1A4rkvYf29CA/nfZdbBaCewW+i9RUneZPe7q+7XjWZkra A239sBVHogTkJlGMNip3PXa1hV4nflwpQOdGyOsbBkvpM8QpoMIwcY3DmrRFpHpI2k eK6ukM2Tp2SbS8ZGuN2msJSe3qH97sLNs0eCqt7A= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgD-004O7k-Nw; Tue, 11 Feb 2020 17:50:37 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 75/94] KVM: arm64: debug: Use ctxt_sys_reg() instead of raw sys_regs access Date: Tue, 11 Feb 2020 17:49:19 +0000 Message-Id: <20200211174938.27809-76-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that we have a wrapper for the sysreg accesses, let's use that consistently. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/debug-sr.c | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/arch/arm64/kvm/hyp/debug-sr.c b/arch/arm64/kvm/hyp/debug-sr.c index 0fc9872a1467..421e547ed10f 100644 --- a/arch/arm64/kvm/hyp/debug-sr.c +++ b/arch/arm64/kvm/hyp/debug-sr.c @@ -129,8 +129,7 @@ static void __hyp_text __debug_restore_spe_nvhe(u64 pmscr_el1) write_sysreg_s(pmscr_el1, SYS_PMSCR_EL1); } -static void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu, - struct kvm_guest_debug_arch *dbg, +static void __hyp_text __debug_save_state(struct kvm_guest_debug_arch *dbg, struct kvm_cpu_context *ctxt) { u64 aa64dfr0; @@ -145,11 +144,10 @@ static void __hyp_text __debug_save_state(struct kvm_vcpu *vcpu, save_debug(dbg->dbg_wcr, dbgwcr, wrps); save_debug(dbg->dbg_wvr, dbgwvr, wrps); - ctxt->sys_regs[MDCCINT_EL1] = read_sysreg(mdccint_el1); + ctxt_sys_reg(ctxt, MDCCINT_EL1) = read_sysreg(mdccint_el1); } -static void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu, - struct kvm_guest_debug_arch *dbg, +static void __hyp_text __debug_restore_state(struct kvm_guest_debug_arch *dbg, struct kvm_cpu_context *ctxt) { u64 aa64dfr0; @@ -165,7 +163,7 @@ static void __hyp_text __debug_restore_state(struct kvm_vcpu *vcpu, restore_debug(dbg->dbg_wcr, dbgwcr, wrps); restore_debug(dbg->dbg_wvr, dbgwvr, wrps); - write_sysreg(ctxt->sys_regs[MDCCINT_EL1], mdccint_el1); + write_sysreg(ctxt_sys_reg(ctxt, MDCCINT_EL1), mdccint_el1); } void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu) @@ -190,8 +188,8 @@ void __hyp_text __debug_switch_to_guest(struct kvm_vcpu *vcpu) host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr); - __debug_save_state(vcpu, host_dbg, host_ctxt); - __debug_restore_state(vcpu, guest_dbg, guest_ctxt); + __debug_save_state(host_dbg, host_ctxt); + __debug_restore_state(guest_dbg, guest_ctxt); } void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu) @@ -212,8 +210,8 @@ void __hyp_text __debug_switch_to_host(struct kvm_vcpu *vcpu) host_dbg = &vcpu->arch.host_debug_state.regs; guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr); - __debug_save_state(vcpu, guest_dbg, guest_ctxt); - __debug_restore_state(vcpu, host_dbg, host_ctxt); + __debug_save_state(guest_dbg, guest_ctxt); + __debug_restore_state(host_dbg, host_ctxt); vcpu->arch.flags &= ~KVM_ARM64_DEBUG_DIRTY; } From patchwork Tue Feb 11 17:49:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376313 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6FC50186E for ; Tue, 11 Feb 2020 17:53:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4E170206D7 for ; Tue, 11 Feb 2020 17:53:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443616; bh=OlZdyOHdrre2bQo7COdxhqVktnxJk6q8cNv5W0KzVjY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=ZdXnhynIGaeLiMVexr0Jb7a1zAzmIzAKDYz7Um2BGqw97guBgcqLopdoeK7vcX9wh QvrZKJIvgGiIghxGOzkwfXMY1J6/pEcxohG5pKdm1w+fXSGCTDM5MXccCSTCLsLL21 FJc/KIFKy7cPU9XWv1cXnCH0RrqMwgeC1nRzWYMM= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730770AbgBKRxf (ORCPT ); Tue, 11 Feb 2020 12:53:35 -0500 Received: from mail.kernel.org ([198.145.29.99]:57280 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730742AbgBKRxf (ORCPT ); Tue, 11 Feb 2020 12:53:35 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 490C520578; Tue, 11 Feb 2020 17:53:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443614; bh=OlZdyOHdrre2bQo7COdxhqVktnxJk6q8cNv5W0KzVjY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=loqhoKYUIjzWJz/Eu5Oj/sCPMFRsidr5MYViOOS3IJ9v27E3CNa6Jrwv/McTdcCsz 1/jm0aO/U0iEOymamx0MjvyPzXu3bIgEYKLfsRQ4ZLKGs8xPyL1zVTv/Wo7DQJTR2k tC92Oxl79d9uswMuiD72P/JCW+k9b6zHX570nYPc= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgE-004O7k-Ae; Tue, 11 Feb 2020 17:50:38 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 76/94] KVM: arm64: Add missing reset handlers for PMU emulation Date: Tue, 11 Feb 2020 17:49:20 +0000 Message-Id: <20200211174938.27809-77-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As we're about to become a bit more harsh when it comes to the lack of reset callbacks, let's add the missing PMU reset handlers. Note that these only cover *CLR registers that were always covered by their *SET counterpart, so there is no semantic change here. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 8d4e67594d87..268829c73313 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2151,7 +2151,7 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_PAR_EL1), NULL, reset_unknown, PAR_EL1 }, { SYS_DESC(SYS_PMINTENSET_EL1), access_pminten, reset_unknown, PMINTENSET_EL1 }, - { SYS_DESC(SYS_PMINTENCLR_EL1), access_pminten, NULL, PMINTENSET_EL1 }, + { SYS_DESC(SYS_PMINTENCLR_EL1), access_pminten, reset_unknown, PMINTENSET_EL1 }, { SYS_DESC(SYS_MAIR_EL1), access_vm_reg, reset_unknown, MAIR_EL1 }, { SYS_DESC(SYS_AMAIR_EL1), access_vm_reg, reset_amair_el1, AMAIR_EL1 }, @@ -2190,8 +2190,8 @@ static const struct sys_reg_desc sys_reg_descs[] = { { SYS_DESC(SYS_PMCR_EL0), access_pmcr, reset_pmcr, PMCR_EL0 }, { SYS_DESC(SYS_PMCNTENSET_EL0), access_pmcnten, reset_unknown, PMCNTENSET_EL0 }, - { SYS_DESC(SYS_PMCNTENCLR_EL0), access_pmcnten, NULL, PMCNTENSET_EL0 }, - { SYS_DESC(SYS_PMOVSCLR_EL0), access_pmovs, NULL, PMOVSSET_EL0 }, + { SYS_DESC(SYS_PMCNTENCLR_EL0), access_pmcnten, reset_unknown, PMCNTENSET_EL0 }, + { SYS_DESC(SYS_PMOVSCLR_EL0), access_pmovs, reset_unknown, PMOVSSET_EL0 }, { SYS_DESC(SYS_PMSWINC_EL0), access_pmswinc, reset_unknown, PMSWINC_EL0 }, { SYS_DESC(SYS_PMSELR_EL0), access_pmselr, reset_unknown, PMSELR_EL0 }, { SYS_DESC(SYS_PMCEID0_EL0), access_pmceid }, From patchwork Tue Feb 11 17:49:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376203 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99D6114B4 for ; Tue, 11 Feb 2020 17:51:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7A2F820870 for ; Tue, 11 Feb 2020 17:51:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443519; bh=8yWpTF4iaTUbN9LOY7/rc6Dq7R5pIpQHT5f/RMVwKY4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=010ke2ze0Y7f6XF5nrxVDi8OBqdD83zciz1mCSj+Q4OPk+9Gy4nD8sifk9qKzW5aa SVp2z6MfKaLYo+lBnNawhvDNB/8bGqPvoPN9SZrZw7Bi/+vjRWthLgzE76/emIiZSD q+e17Oog6fZXDSlCewkZ/6Xr1iYUy+pkWg/HTF0k= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730463AbgBKRv6 (ORCPT ); Tue, 11 Feb 2020 12:51:58 -0500 Received: from mail.kernel.org ([198.145.29.99]:54610 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730345AbgBKRv6 (ORCPT ); Tue, 11 Feb 2020 12:51:58 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 6857220661; Tue, 11 Feb 2020 17:51:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443516; bh=8yWpTF4iaTUbN9LOY7/rc6Dq7R5pIpQHT5f/RMVwKY4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Xnf9tvHBy1TruTl5Sm8GlHHWCjQ16Uyak6LpSO+GNv+TgH50Y1d4f2SZ6UHYfGhmU 6LXT+c6DK7oMB137K/LHYYv/oVcxJw46sLGNCExd6D32Cxt7kQi9GRnoQ/EhfegG9A V032SwHm1rDg1IqKgyDWO8AIyDtYm7kqmP7lMVAk= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgE-004O7k-TP; Tue, 11 Feb 2020 17:50:39 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 77/94] KVM: arm64: nv: Move sysreg reset check to boot time Date: Tue, 11 Feb 2020 17:49:21 +0000 Message-Id: <20200211174938.27809-78-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Our sysreg reset check has become a bit silly, as it only checks whether a reset callback actually exists for a given sysreg entry, and apply the method if available. Doing the check at each vcpu reset is pretty dumb, as the tables never change. It is thus perfectly possible to do the same checks at boot time. This also allows us to introduce a sparse sys_regs[] array, something that will be required with ARMv8.4-NV. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/sys_regs.c | 74 +++++++++++++++++++-------------------- 1 file changed, 36 insertions(+), 38 deletions(-) diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 268829c73313..9d426304bfb3 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -3166,12 +3166,37 @@ static const struct sys_reg_desc cp15_64_regs[] = { { SYS_DESC(SYS_AARCH32_CNTP_CVAL), access_arch_timer }, }; +static int check_sysreg_table(const struct sys_reg_desc *table, unsigned int n, + bool is_32) +{ + unsigned int i; + + for (i = 0; i < n; i++) { + if (!is_32 && table[i].reg && !table[i].reset) { + kvm_err("sys_reg table %p entry %d has lacks reset\n", + table, i); + return 1; + } + + if (i && cmp_sys_reg(&table[i-1], &table[i]) >= 0) { + kvm_err("sys_reg table %p out of order (%d)\n", table, i - 1); + return 1; + } + } + + return 0; +} + /* Target specific emulation tables */ static struct kvm_sys_reg_target_table *target_tables[KVM_ARM_NUM_TARGETS]; void kvm_register_target_sys_reg_table(unsigned int target, struct kvm_sys_reg_target_table *table) { + if (check_sysreg_table(table->table64.table, table->table64.num, false) || + check_sysreg_table(table->table32.table, table->table32.num, true)) + return; + target_tables[target] = table; } @@ -3483,19 +3508,13 @@ static int emulate_sys_instr(struct kvm_vcpu *vcpu, struct sys_reg_params *p) } static void reset_sys_reg_descs(struct kvm_vcpu *vcpu, - const struct sys_reg_desc *table, size_t num, - unsigned long *bmap) + const struct sys_reg_desc *table, size_t num) { unsigned long i; for (i = 0; i < num; i++) - if (table[i].reset) { - int reg = table[i].reg; - + if (table[i].reset) table[i].reset(vcpu, &table[i]); - if (reg > 0 && reg < NR_SYS_REGS) - set_bit(reg, bmap); - } } /** @@ -3958,33 +3977,19 @@ int kvm_arm_copy_sys_reg_indices(struct kvm_vcpu *vcpu, u64 __user *uindices) return write_demux_regids(uindices); } -static int check_sysreg_table(const struct sys_reg_desc *table, unsigned int n) -{ - unsigned int i; - - for (i = 1; i < n; i++) { - if (cmp_sys_reg(&table[i-1], &table[i]) >= 0) { - kvm_err("sys_reg table %p out of order (%d)\n", table, i - 1); - return 1; - } - } - - return 0; -} - void kvm_sys_reg_table_init(void) { unsigned int i; struct sys_reg_desc clidr; /* Make sure tables are unique and in order. */ - BUG_ON(check_sysreg_table(sys_reg_descs, ARRAY_SIZE(sys_reg_descs))); - BUG_ON(check_sysreg_table(cp14_regs, ARRAY_SIZE(cp14_regs))); - BUG_ON(check_sysreg_table(cp14_64_regs, ARRAY_SIZE(cp14_64_regs))); - BUG_ON(check_sysreg_table(cp15_regs, ARRAY_SIZE(cp15_regs))); - BUG_ON(check_sysreg_table(cp15_64_regs, ARRAY_SIZE(cp15_64_regs))); - BUG_ON(check_sysreg_table(invariant_sys_regs, ARRAY_SIZE(invariant_sys_regs))); - BUG_ON(check_sysreg_table(sys_insn_descs, ARRAY_SIZE(sys_insn_descs))); + BUG_ON(check_sysreg_table(sys_reg_descs, ARRAY_SIZE(sys_reg_descs), false)); + BUG_ON(check_sysreg_table(cp14_regs, ARRAY_SIZE(cp14_regs), true)); + BUG_ON(check_sysreg_table(cp14_64_regs, ARRAY_SIZE(cp14_64_regs), true)); + BUG_ON(check_sysreg_table(cp15_regs, ARRAY_SIZE(cp15_regs), true)); + BUG_ON(check_sysreg_table(cp15_64_regs, ARRAY_SIZE(cp15_64_regs), true)); + BUG_ON(check_sysreg_table(invariant_sys_regs, ARRAY_SIZE(invariant_sys_regs), false)); + BUG_ON(check_sysreg_table(sys_insn_descs, ARRAY_SIZE(sys_insn_descs), false)); /* We abuse the reset function to overwrite the table itself. */ for (i = 0; i < ARRAY_SIZE(invariant_sys_regs); i++) @@ -4020,17 +4025,10 @@ void kvm_reset_sys_regs(struct kvm_vcpu *vcpu) { size_t num; const struct sys_reg_desc *table; - DECLARE_BITMAP(bmap, NR_SYS_REGS) = { 0, }; /* Generic chip reset first (so target could override). */ - reset_sys_reg_descs(vcpu, sys_reg_descs, ARRAY_SIZE(sys_reg_descs), bmap); + reset_sys_reg_descs(vcpu, sys_reg_descs, ARRAY_SIZE(sys_reg_descs)); table = get_target_table(vcpu->arch.target, true, &num); - reset_sys_reg_descs(vcpu, table, num, bmap); - - for (num = 1; num < NR_SYS_REGS; num++) { - if (WARN(!test_bit(num, bmap), - "Didn't reset __vcpu_sys_reg(%zi)\n", num)) - break; - } + reset_sys_reg_descs(vcpu, table, num); } From patchwork Tue Feb 11 17:49:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376319 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C861C14B4 for ; Tue, 11 Feb 2020 17:53:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9F3FE20870 for ; Tue, 11 Feb 2020 17:53:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443622; bh=DmBeKOo8g9nGEtxFHJrBStIc81V1f1OuSuMxpsF+zgc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=c+8fxYhCuRDKFLcNwPkALlTYpyBwC6TEmMYrM8DimnrOuNkjsVryUbNLepdOgWGa1 TV0xPR1BvqcmJuZhfQNMEpOY5gxjenKqd7ua7siZ9ScdahaPP/hohhbg3AScomrpOe CD+oIly+i2MPJmQgVQgpEMF9WHqvEnAQOA98r2rI= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730776AbgBKRxl (ORCPT ); Tue, 11 Feb 2020 12:53:41 -0500 Received: from mail.kernel.org ([198.145.29.99]:57456 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730787AbgBKRxl (ORCPT ); Tue, 11 Feb 2020 12:53:41 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BC92620848; Tue, 11 Feb 2020 17:53:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443619; bh=DmBeKOo8g9nGEtxFHJrBStIc81V1f1OuSuMxpsF+zgc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kD+FqneJLPNxsWfm9Dxu3wOYG9nF8cWB0IU2BlnpaWrrfSeaYejloK6kYk4vJSMgM gqse9oROIKluCx+/pibylv0uEwJ40Bh37hc/6JmMs9AEFKm90Td20Ty3G3siEIZSW7 YopO4xoYfeyIhUI5mk7kqpco0/AXOw7RrQb/vlP0= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgF-004O7k-GC; Tue, 11 Feb 2020 17:50:39 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 78/94] KVM: arm64: Map VNCR-capable registers to a separate page Date: Tue, 11 Feb 2020 17:49:22 +0000 Message-Id: <20200211174938.27809-79-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org With ARMv8.4-NV, registers that can be directly accessed in memory by the guest have to live at architected offsets in a special page. Let's annotate the sysreg enum to reflect the offset at which they are in this page, whith a little twist: If running on HW that doesn't have the ARMv8.4-NV feature, or even a VM that doesn't use NV, we store all the system registers in the usual sys_regs array. The only difference with the pre-8.4 situation is that VNCR-capable registers are at a "similar" offset as in the VNCR page (we can compute the actual offset at compile time), and that the sys_regs array is both bigger and sparse. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 78 +++++++++++++++++++++---------- 1 file changed, 54 insertions(+), 24 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 83ed3865dda8..e160cf283601 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -25,6 +25,7 @@ #include #include #include +#include #define __KVM_HAVE_ARCH_INTC_INITIALIZED @@ -154,6 +155,23 @@ struct kvm_vcpu_fault_info { u64 disr_el1; /* Deferred [SError] Status Register */ }; +/* + * VNCR() just places the VNCR_capable registers in the enum after + * __VNCR_START__, and the value (after correction) to be an 8-byte offset + * from the VNCR base. As we don't require the enum to be otherwise ordered, + * we need the terrible hack below to ensure that we correctly size the + * sys_regs array, no matter what. + * + * The __MAX__ macro has been lifted from Sean Eron Anderson's wonderful + * treasure trove of bit hacks: + * https://graphics.stanford.edu/~seander/bithacks.html#IntegerMinOrMax + */ +#define __MAX__(x,y) ((x) ^ (((x) ^ (y)) & -((x) < (y)))) +#define VNCR(r) \ + __before_##r, \ + r = __VNCR_START__ + ((VNCR_ ## r) / 8), \ + __after_##r = __MAX__(__before_##r - 1, r) + /* * 0 is reserved as an invalid value. * Order should be kept in sync with the save/restore code. @@ -162,27 +180,11 @@ enum vcpu_sysreg { __INVALID_SYSREG__, MPIDR_EL1, /* MultiProcessor Affinity Register */ CSSELR_EL1, /* Cache Size Selection Register */ - SCTLR_EL1, /* System Control Register */ - ACTLR_EL1, /* Auxiliary Control Register */ - CPACR_EL1, /* Coprocessor Access Control */ - ZCR_EL1, /* SVE Control */ - TTBR0_EL1, /* Translation Table Base Register 0 */ - TTBR1_EL1, /* Translation Table Base Register 1 */ - TCR_EL1, /* Translation Control Register */ - ESR_EL1, /* Exception Syndrome Register */ - AFSR0_EL1, /* Auxiliary Fault Status Register 0 */ - AFSR1_EL1, /* Auxiliary Fault Status Register 1 */ - FAR_EL1, /* Fault Address Register */ - MAIR_EL1, /* Memory Attribute Indirection Register */ - VBAR_EL1, /* Vector Base Address Register */ - CONTEXTIDR_EL1, /* Context ID Register */ TPIDR_EL0, /* Thread ID, User R/W */ TPIDRRO_EL0, /* Thread ID, User R/O */ TPIDR_EL1, /* Thread ID, Privileged */ - AMAIR_EL1, /* Aux Memory Attribute Indirection Register */ CNTKCTL_EL1, /* Timer Control Register (EL1) */ PAR_EL1, /* Physical Address Register */ - MDSCR_EL1, /* Monitor Debug System Control Register */ MDCCINT_EL1, /* Monitor Debug Comms Channel Interrupt Enable Reg */ DISR_EL1, /* Deferred Interrupt Status Register */ @@ -220,20 +222,14 @@ enum vcpu_sysreg { DBGVCR32_EL2, /* Debug Vector Catch Register */ /* EL2 registers sorted ascending by Op0, Op1, CRn, CRm, Op2 */ - VPIDR_EL2, /* Virtualization Processor ID Register */ - VMPIDR_EL2, /* Virtualization Multiprocessor ID Register */ SCTLR_EL2, /* System Control Register (EL2) */ ACTLR_EL2, /* Auxiliary Control Register (EL2) */ - HCR_EL2, /* Hypervisor Configuration Register */ MDCR_EL2, /* Monitor Debug Configuration Register (EL2) */ CPTR_EL2, /* Architectural Feature Trap Register (EL2) */ - HSTR_EL2, /* Hypervisor System Trap Register */ HACR_EL2, /* Hypervisor Auxiliary Control Register */ TTBR0_EL2, /* Translation Table Base Register 0 (EL2) */ TTBR1_EL2, /* Translation Table Base Register 1 (EL2) */ TCR_EL2, /* Translation Control Register (EL2) */ - VTTBR_EL2, /* Virtualization Translation Table Base Register */ - VTCR_EL2, /* Virtualization Translation Control Register */ SPSR_EL2, /* EL2 saved program status register */ ELR_EL2, /* EL2 exception link register */ AFSR0_EL2, /* Auxiliary Fault Status Register 0 (EL2) */ @@ -247,10 +243,35 @@ enum vcpu_sysreg { RVBAR_EL2, /* Reset Vector Base Address Register */ RMR_EL2, /* Reset Management Register */ CONTEXTIDR_EL2, /* Context ID Register (EL2) */ - TPIDR_EL2, /* EL2 Software Thread ID Register */ CNTHCTL_EL2, /* Counter-timer Hypervisor Control register */ SP_EL2, /* EL2 Stack Pointer */ + __VNCR_START__, /* Any VNCR-capable reg goes after this point */ + + VNCR(SCTLR_EL1),/* System Control Register */ + VNCR(ACTLR_EL1),/* Auxiliary Control Register */ + VNCR(CPACR_EL1),/* Coprocessor Access Control */ + VNCR(ZCR_EL1), /* SVE Control */ + VNCR(TTBR0_EL1),/* Translation Table Base Register 0 */ + VNCR(TTBR1_EL1),/* Translation Table Base Register 1 */ + VNCR(TCR_EL1), /* Translation Control Register */ + VNCR(ESR_EL1), /* Exception Syndrome Register */ + VNCR(AFSR0_EL1),/* Auxiliary Fault Status Register 0 */ + VNCR(AFSR1_EL1),/* Auxiliary Fault Status Register 1 */ + VNCR(FAR_EL1), /* Fault Address Register */ + VNCR(MAIR_EL1), /* Memory Attribute Indirection Register */ + VNCR(VBAR_EL1), /* Vector Base Address Register */ + VNCR(CONTEXTIDR_EL1), /* Context ID Register */ + VNCR(AMAIR_EL1),/* Aux Memory Attribute Indirection Register */ + VNCR(MDSCR_EL1),/* Monitor Debug System Control Register */ + VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */ + VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */ + VNCR(HCR_EL2), /* Hypervisor Configuration Register */ + VNCR(HSTR_EL2), /* Hypervisor System Trap Register */ + VNCR(VTTBR_EL2),/* Virtualization Translation Table Base Register */ + VNCR(VTCR_EL2), /* Virtualization Translation Control Register */ + VNCR(TPIDR_EL2),/* EL2 Software Thread ID Register */ + NR_SYS_REGS /* Nothing after this line! */ }; @@ -303,6 +324,9 @@ struct kvm_cpu_context { }; struct kvm_vcpu *__hyp_running_vcpu; + + /* This pointer has to be 4kB aligned. */ + u64 *vncr_array; }; struct kvm_pmu_events { @@ -474,7 +498,13 @@ struct kvm_vcpu_arch { * for system registers that are never context switched, but only * emulated. */ -#define __ctxt_sys_reg(c,r) (&(c)->sys_regs[(r)]) +static inline u64 *__ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) +{ + if (unlikely(r >= __VNCR_START__ && ctxt->vncr_array)) + return &ctxt->vncr_array[r - __VNCR_START__]; + + return (u64 *)&ctxt->sys_regs[r]; +} #define ctxt_sys_reg(c,r) (*__ctxt_sys_reg(c,r)) From patchwork Tue Feb 11 17:49:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376151 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4249F1395 for ; Tue, 11 Feb 2020 17:51:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 10A332086A for ; Tue, 11 Feb 2020 17:51:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443473; bh=iD+dO0Xdz4rLOP0a58aDH/YSqaLiXx6FsLfAjEm2ro0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=0gAwA0fW62sdpa9QCQMHef83bU4++iAjWIMlaxJYA7vPgyhWP+ffE8A3SG6sdXV4y JqT9rfg9HZ2VMSthnwJN1I5YPoBLEO6JanAFNHh8tx/wCOMGJnhKw4h6mRHZ86zMX/ pmK9Q+wWI9+1Sy9b9TWEeBneCojNm120Ty2tdZxg= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730276AbgBKRvM (ORCPT ); Tue, 11 Feb 2020 12:51:12 -0500 Received: from mail.kernel.org ([198.145.29.99]:53392 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730238AbgBKRvL (ORCPT ); Tue, 11 Feb 2020 12:51:11 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B10C5206D6; Tue, 11 Feb 2020 17:51:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443471; bh=iD+dO0Xdz4rLOP0a58aDH/YSqaLiXx6FsLfAjEm2ro0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qRjumTLmAcaRBObhPd6R0WQ25oxocU491DJLNnCogWs6VALAJi0ofN9vY+vL0WX8Y L94vIZBEslq1+CJvpuXZv0kgt11sQLzHzRURz/0BM3USpzZbm4ksgSst8q5J9blNgw Y/nQWG1V5rCYeHmHmXXkyEVuVC13JcPMPtzf1IeY= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgG-004O7k-44; Tue, 11 Feb 2020 17:50:40 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 79/94] KVM: arm64: nv: Move nested vgic state into the sysreg file Date: Tue, 11 Feb 2020 17:49:23 +0000 Message-Id: <20200211174938.27809-80-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The vgic nested state needs to be accessible from the VNCR page, and thus needs to be part of the normal sysreg file. Let's move it there. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 9 +++ arch/arm64/kvm/sys_regs.c | 53 ++++++++++++------ include/kvm/arm_vgic.h | 7 --- virt/kvm/arm/vgic/vgic-v3-nested.c | 88 +++++++++++++++--------------- virt/kvm/arm/vgic/vgic-v3.c | 18 ++++-- virt/kvm/arm/vgic/vgic.h | 10 ++++ 6 files changed, 111 insertions(+), 74 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index e160cf283601..7a66abd3df7e 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -272,6 +272,15 @@ enum vcpu_sysreg { VNCR(VTCR_EL2), /* Virtualization Translation Control Register */ VNCR(TPIDR_EL2),/* EL2 Software Thread ID Register */ + VNCR(ICH_LR0_EL2), + ICH_LR15_EL2 = ICH_LR0_EL2 + 15, + VNCR(ICH_AP0R0_EL2), + ICH_AP0R3_EL2 = ICH_AP0R0_EL2 + 3, + VNCR(ICH_AP1R0_EL2), + ICH_AP1R3_EL2 = ICH_AP1R0_EL2 + 3, + VNCR(ICH_HCR_EL2), + VNCR(ICH_VMCR_EL2), + NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 9d426304bfb3..5a41788054b7 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1880,17 +1880,17 @@ static bool access_gic_apr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - u32 index, *base; + u64 *base; + u8 index; index = r->Op2; if (r->CRm == 8) - base = cpu_if->vgic_ap0r; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_AP0R0_EL2); else - base = cpu_if->vgic_ap1r; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_AP1R0_EL2); if (p->is_write) - base[index] = p->regval; + base[index] = lower_32_bits(p->regval); else p->regval = base[index]; @@ -1901,12 +1901,10 @@ static bool access_gic_hcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - if (p->is_write) - cpu_if->vgic_hcr = p->regval; + __vcpu_sys_reg(vcpu, ICH_HCR_EL2) = lower_32_bits(p->regval); else - p->regval = cpu_if->vgic_hcr; + p->regval = __vcpu_sys_reg(vcpu, ICH_HCR_EL2); return true; } @@ -1963,12 +1961,19 @@ static bool access_gic_vmcr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; - if (p->is_write) - cpu_if->vgic_vmcr = p->regval; + __vcpu_sys_reg(vcpu, ICH_VMCR_EL2) = (p->regval & + (ICH_VMCR_ENG0_MASK | + ICH_VMCR_ENG1_MASK | + ICH_VMCR_PMR_MASK | + ICH_VMCR_BPR0_MASK | + ICH_VMCR_BPR1_MASK | + ICH_VMCR_EOIM_MASK | + ICH_VMCR_CBPR_MASK | + ICH_VMCR_FIQ_EN_MASK | + ICH_VMCR_ACK_CTL_MASK)); else - p->regval = cpu_if->vgic_vmcr; + p->regval = __vcpu_sys_reg(vcpu, ICH_VMCR_EL2); return true; } @@ -1977,17 +1982,29 @@ static bool access_gic_lr(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; u32 index; + u64 *base; + base = __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_LR0_EL2); index = p->Op2; if (p->CRm == 13) index += 8; - if (p->is_write) - cpu_if->vgic_lr[index] = p->regval; - else - p->regval = cpu_if->vgic_lr[index]; + if (p->is_write) { + u64 mask = (ICH_LR_VIRTUAL_ID_MASK | + ICH_LR_GROUP | + ICH_LR_HW | + ICH_LR_STATE); + + if (p->regval & ICH_LR_HW) + mask |= ICH_LR_PHYS_ID_MASK; + else + mask |= ICH_LR_EOI; + + base[index] = p->regval & mask; + } else { + p->regval = base[index]; + } return true; } diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h index b5c026a7e0ee..a03148ee1266 100644 --- a/include/kvm/arm_vgic.h +++ b/include/kvm/arm_vgic.h @@ -305,9 +305,6 @@ struct vgic_cpu { struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS]; - /* CPU vif control registers for the virtual GICH interface */ - struct vgic_v3_cpu_if nested_vgic_v3; - /* * The shadow vif control register loaded to the hardware when * running a nested L2 guest with the virtual IMO/FMO bit set. @@ -370,10 +367,6 @@ void kvm_vgic_load(struct kvm_vcpu *vcpu); void kvm_vgic_put(struct kvm_vcpu *vcpu); void kvm_vgic_vmcr_sync(struct kvm_vcpu *vcpu); -void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); -void vgic_v3_load_nested(struct kvm_vcpu *vcpu); -void vgic_v3_put_nested(struct kvm_vcpu *vcpu); -void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu); u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu); u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu); diff --git a/virt/kvm/arm/vgic/vgic-v3-nested.c b/virt/kvm/arm/vgic/vgic-v3-nested.c index 94b1edb67011..51f7a521e829 100644 --- a/virt/kvm/arm/vgic/vgic-v3-nested.c +++ b/virt/kvm/arm/vgic/vgic-v3-nested.c @@ -16,11 +16,6 @@ #define CREATE_TRACE_POINTS #include "vgic-nested-trace.h" -static inline struct vgic_v3_cpu_if *vcpu_nested_if(struct kvm_vcpu *vcpu) -{ - return &vcpu->arch.vgic_cpu.nested_vgic_v3; -} - static inline struct vgic_v3_cpu_if *vcpu_shadow_if(struct kvm_vcpu *vcpu) { return &vcpu->arch.vgic_cpu.shadow_vgic_v3; @@ -33,12 +28,11 @@ static inline bool lr_triggers_eoi(u64 lr) u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); u16 reg = 0; int i; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - if (lr_triggers_eoi(cpu_if->vgic_lr[i])) + if (lr_triggers_eoi(__vcpu_sys_reg(vcpu, ICH_LRN(i)))) reg |= BIT(i); } @@ -47,12 +41,11 @@ u16 vgic_v3_get_eisr(struct kvm_vcpu *vcpu) u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); u16 reg = 0; int i; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - if (!(cpu_if->vgic_lr[i] & ICH_LR_STATE)) + if (!(__vcpu_sys_reg(vcpu, ICH_LRN(i)) & ICH_LR_STATE)) reg |= BIT(i); } @@ -61,14 +54,13 @@ u16 vgic_v3_get_elrsr(struct kvm_vcpu *vcpu) u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); int nr_lr = kvm_vgic_global_state.nr_lr; u64 reg = 0; if (vgic_v3_get_eisr(vcpu)) reg |= ICH_MISR_EOI; - if (cpu_if->vgic_hcr & ICH_HCR_UIE) { + if (__vcpu_sys_reg(vcpu, ICH_HCR_EL2) & ICH_HCR_UIE) { int used_lrs; used_lrs = nr_lr - hweight16(vgic_v3_get_elrsr(vcpu)); @@ -87,13 +79,12 @@ u64 vgic_v3_get_misr(struct kvm_vcpu *vcpu) */ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); struct vgic_irq *irq; int i, used_lrs = 0; for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { - u64 lr = cpu_if->vgic_lr[i]; + u64 lr = __vcpu_sys_reg(vcpu, ICH_LRN(i)); int l1_irq; if (!(lr & ICH_LR_HW)) @@ -123,36 +114,20 @@ static void vgic_v3_create_shadow_lr(struct kvm_vcpu *vcpu) } trace_vgic_create_shadow_lrs(vcpu, kvm_vgic_global_state.nr_lr, - s_cpu_if->vgic_lr, cpu_if->vgic_lr); + s_cpu_if->vgic_lr, + __ctxt_sys_reg(&vcpu->arch.ctxt, ICH_LR0_EL2)); s_cpu_if->used_lrs = used_lrs; } -/* - * Change the shadow HWIRQ field back to the virtual value before copying over - * the entire shadow struct to the nested state. - */ -static void vgic_v3_fixup_shadow_lr_state(struct kvm_vcpu *vcpu) -{ - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); - struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); - int lr; - - for (lr = 0; lr < kvm_vgic_global_state.nr_lr; lr++) { - s_cpu_if->vgic_lr[lr] &= ~ICH_LR_PHYS_ID_MASK; - s_cpu_if->vgic_lr[lr] |= cpu_if->vgic_lr[lr] & ICH_LR_PHYS_ID_MASK; - } -} - void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); struct vgic_irq *irq; int i; for (i = 0; i < s_cpu_if->used_lrs; i++) { - u64 lr = cpu_if->vgic_lr[i]; + u64 lr = __vcpu_sys_reg(vcpu, ICH_LRN(i)); int l1_irq; if (!(lr & ICH_LR_HW) || !(lr & ICH_LR_STATE)) @@ -178,14 +153,27 @@ void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) } } +void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu) +{ + struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + int i; + + cpu_if->vgic_hcr = __vcpu_sys_reg(vcpu, ICH_HCR_EL2); + cpu_if->vgic_vmcr = __vcpu_sys_reg(vcpu, ICH_VMCR_EL2); + + for (i = 0; i < 4; i++) { + cpu_if->vgic_ap0r[i] = __vcpu_sys_reg(vcpu, ICH_AP0RN(i)); + cpu_if->vgic_ap1r[i] = __vcpu_sys_reg(vcpu, ICH_AP1RN(i)); + } + + vgic_v3_create_shadow_lr(vcpu); +} + void vgic_v3_load_nested(struct kvm_vcpu *vcpu) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; struct vgic_irq *irq; unsigned long flags; - vgic_cpu->shadow_vgic_v3 = vgic_cpu->nested_vgic_v3; - vgic_v3_create_shadow_lr(vcpu); __vgic_v3_restore_state(vcpu_shadow_if(vcpu)); irq = vgic_get_irq(vcpu->kvm, vcpu, vcpu->kvm->arch.vgic.maint_irq); @@ -199,26 +187,40 @@ void vgic_v3_load_nested(struct kvm_vcpu *vcpu) void vgic_v3_put_nested(struct kvm_vcpu *vcpu) { - struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu; + struct vgic_v3_cpu_if *s_cpu_if = vcpu_shadow_if(vcpu); + int i; - __vgic_v3_save_state(vcpu_shadow_if(vcpu)); + __vgic_v3_save_state(s_cpu_if); - trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, - vcpu_shadow_if(vcpu)->vgic_lr); + trace_vgic_put_nested(vcpu, kvm_vgic_global_state.nr_lr, s_cpu_if->vgic_lr); /* * Translate the shadow state HW fields back to the virtual ones * before copying the shadow struct back to the nested one. */ - vgic_v3_fixup_shadow_lr_state(vcpu); - vgic_cpu->nested_vgic_v3 = vgic_cpu->shadow_vgic_v3; + __vcpu_sys_reg(vcpu, ICH_HCR_EL2) = s_cpu_if->vgic_hcr; + __vcpu_sys_reg(vcpu, ICH_VMCR_EL2) = s_cpu_if->vgic_vmcr; + + for (i = 0; i < 4; i++) { + __vcpu_sys_reg(vcpu, ICH_AP0RN(i)) = s_cpu_if->vgic_ap0r[i]; + __vcpu_sys_reg(vcpu, ICH_AP1RN(i)) = s_cpu_if->vgic_ap1r[i]; + } + + for (i = 0; i < kvm_vgic_global_state.nr_lr; i++) { + u64 val = __vcpu_sys_reg(vcpu, ICH_LRN(i)); + + val &= ~ICH_LR_STATE; + val |= s_cpu_if->vgic_lr[i] & ICH_LR_STATE; + + __vcpu_sys_reg(vcpu, ICH_LRN(i)) = val; + } + irq_set_irqchip_state(kvm_vgic_global_state.maint_irq, IRQCHIP_STATE_ACTIVE, false); } void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) { - struct vgic_v3_cpu_if *cpu_if = vcpu_nested_if(vcpu); bool state; /* @@ -230,7 +232,7 @@ void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) if (!vgic_state_is_nested(vcpu)) return; - state = cpu_if->vgic_hcr & ICH_HCR_EN; + state = __vcpu_sys_reg(vcpu, ICH_HCR_EL2) & ICH_HCR_EN; state &= vgic_v3_get_misr(vcpu); kvm_vgic_inject_irq(vcpu->kvm, vcpu->vcpu_id, diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c index bca20528d5d1..e8c63c8d017a 100644 --- a/virt/kvm/arm/vgic/vgic-v3.c +++ b/virt/kvm/arm/vgic/vgic-v3.c @@ -292,10 +292,11 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu) ICC_SRE_EL1_SRE); /* * If nesting is allowed, force GICv3 onto the nested - * guests as well. + * guests as well by setting the shadow state to the + * same value. */ if (nested_virt_in_use(vcpu)) - vcpu->arch.vgic_cpu.nested_vgic_v3.vgic_sre = vgic_v3->vgic_sre; + vcpu->arch.vgic_cpu.shadow_vgic_v3.vgic_sre = vgic_v3->vgic_sre; vcpu->arch.vgic_cpu.pendbaser = INITIAL_PENDBASER_VALUE; } else { vgic_v3->vgic_sre = 0; @@ -660,11 +661,15 @@ void vgic_v3_load(struct kvm_vcpu *vcpu) struct vgic_v3_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v3; /* - * vgic_v3_load_nested only affects the LRs in the shadow - * state, so it is fine to pass the nested state around. + * If the vgic is in nested state, populate the shadow state + * from the guest's nested state. As vgic_v3_load_nested() + * will only load LRs, let's deal with the rest of the state + * here as if it was a non-nested state. Cunning. */ - if (vgic_state_is_nested(vcpu)) - cpu_if = &vcpu->arch.vgic_cpu.nested_vgic_v3; + if (vgic_state_is_nested(vcpu)) { + vgic_v3_create_shadow_state(vcpu); + cpu_if = &vcpu->arch.vgic_cpu.shadow_vgic_v3; + } /* * If dealing with a GICv2 emulation on GICv3, VMCR_EL2.VFIQen @@ -718,5 +723,6 @@ void vgic_v3_put(struct kvm_vcpu *vcpu) __weak void vgic_v3_sync_nested(struct kvm_vcpu *vcpu) {} __weak void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu) {} +__weak void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu) {} __weak void vgic_v3_load_nested(struct kvm_vcpu *vcpu) {} __weak void vgic_v3_put_nested(struct kvm_vcpu *vcpu) {} diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h index c7fefd6b1c80..a8f97e95fe31 100644 --- a/virt/kvm/arm/vgic/vgic.h +++ b/virt/kvm/arm/vgic/vgic.h @@ -317,4 +317,14 @@ bool vgic_supports_direct_msis(struct kvm *kvm); int vgic_v4_init(struct kvm *kvm); void vgic_v4_teardown(struct kvm *kvm); +void vgic_v3_sync_nested(struct kvm_vcpu *vcpu); +void vgic_v3_create_shadow_state(struct kvm_vcpu *vcpu); +void vgic_v3_load_nested(struct kvm_vcpu *vcpu); +void vgic_v3_put_nested(struct kvm_vcpu *vcpu); +void vgic_v3_handle_nested_maint_irq(struct kvm_vcpu *vcpu); + +#define ICH_LRN(n) (ICH_LR0_EL2 + (n)) +#define ICH_AP0RN(n) (ICH_AP0R0_EL2 + (n)) +#define ICH_AP1RN(n) (ICH_AP1R0_EL2 + (n)) + #endif From patchwork Tue Feb 11 17:49:24 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376173 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A6F1314B4 for ; Tue, 11 Feb 2020 17:51:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7ABA020870 for ; Tue, 11 Feb 2020 17:51:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443492; bh=LX+303+Rgi1tTyC2KS98Cz+aQmALLYRZoKsrqstynaI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=HiyNh4GD+LcreG2UchUKhy+H4at0ufIGGgUM0aEdnHxdoYC3yuQzKUHirVUlpdmLi dopAm6HzX5EYtG3ue87VjyoFnZnE5Gc1QPyO5yi6I2gF77U/EEotJHgD22lc4xP0JP eRbqdpjJC7l17LXBI9Nu52KQBWER/Oejd0r6UMiE= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730359AbgBKRvb (ORCPT ); Tue, 11 Feb 2020 12:51:31 -0500 Received: from mail.kernel.org ([198.145.29.99]:53888 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730351AbgBKRva (ORCPT ); Tue, 11 Feb 2020 12:51:30 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id E6D7820661; Tue, 11 Feb 2020 17:51:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443490; bh=LX+303+Rgi1tTyC2KS98Cz+aQmALLYRZoKsrqstynaI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=v7zstlj+OCH39Yrl6ZqpW/XuDf6iza6fP3AkLbgF8CadX4EyXI5aQCWW+ZK7okVXD OvodWB1Gu883JiyJlotyk8DFWL47H2UUZzyvVFOwd/BYcw+Dxx/QrEDtooYgXOY65f RqqMCR682gj8vat8dJX7L2xMkE5CZTwLKpqHmWf8= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgG-004O7k-Ns; Tue, 11 Feb 2020 17:50:40 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 80/94] KVM: arm64: Use accessors for timer ctl/cval/offset Date: Tue, 11 Feb 2020 17:49:24 +0000 Message-Id: <20200211174938.27809-81-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Instead of directly accessing the various fields in the timer data structures, add accessors that provide the same service. By making the weak, we'll be able to override them in a subsequent patch, making it possible to move them in the VNCR page, as they will be directly accessed by the guest. Signed-off-by: Marc Zyngier --- virt/kvm/arm/arch_timer.c | 104 +++++++++++++++++++++++++------------- 1 file changed, 68 insertions(+), 36 deletions(-) diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c index 3c88a4c0a296..d87e52f7e962 100644 --- a/virt/kvm/arm/arch_timer.c +++ b/virt/kvm/arm/arch_timer.c @@ -62,6 +62,36 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, struct arch_timer_context *timer, enum kvm_arch_timer_regs treg); +__weak u32 timer_get_ctl(struct arch_timer_context *ctxt) +{ + return ctxt->cnt_ctl; +} + +__weak u64 timer_get_cval(struct arch_timer_context *ctxt) +{ + return ctxt->cnt_cval; +} + +__weak u64 timer_get_offset(struct arch_timer_context *ctxt) +{ + return ctxt->cntvoff; +} + +__weak void timer_set_ctl(struct arch_timer_context *ctxt, u32 ctl) +{ + ctxt->cnt_ctl = ctl; +} + +__weak void timer_set_cval(struct arch_timer_context *ctxt, u64 cval) +{ + ctxt->cnt_cval = cval; +} + +__weak void timer_set_offset(struct arch_timer_context *ctxt, u64 offset) +{ + ctxt->cntvoff = offset; +} + u64 kvm_phys_timer_read(void) { return timecounter->cc->read(timecounter->cc); @@ -149,8 +179,8 @@ static u64 kvm_timer_compute_delta(struct arch_timer_context *timer_ctx) { u64 cval, now; - cval = timer_ctx->cnt_cval; - now = kvm_phys_timer_read() - timer_ctx->cntvoff; + cval = timer_get_cval(timer_ctx); + now = kvm_phys_timer_read() - timer_get_offset(timer_ctx); if (now < cval) { u64 ns; @@ -169,8 +199,8 @@ static bool kvm_timer_irq_can_fire(struct arch_timer_context *timer_ctx) { WARN_ON(timer_ctx && timer_ctx->loaded); return timer_ctx && - !(timer_ctx->cnt_ctl & ARCH_TIMER_CTRL_IT_MASK) && - (timer_ctx->cnt_ctl & ARCH_TIMER_CTRL_ENABLE); + ((timer_get_ctl(timer_ctx) & + (ARCH_TIMER_CTRL_IT_MASK | ARCH_TIMER_CTRL_ENABLE)) == ARCH_TIMER_CTRL_ENABLE); } /* @@ -283,8 +313,8 @@ static bool kvm_timer_should_fire(struct arch_timer_context *timer_ctx) if (!kvm_timer_irq_can_fire(timer_ctx)) return false; - cval = timer_ctx->cnt_cval; - now = kvm_phys_timer_read() - timer_ctx->cntvoff; + cval = timer_get_cval(timer_ctx); + now = kvm_phys_timer_read() - timer_get_offset(timer_ctx); return cval <= now; } @@ -379,8 +409,8 @@ static void timer_save_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: case TIMER_HVTIMER: - ctx->cnt_ctl = read_sysreg_el0(SYS_CNTV_CTL); - ctx->cnt_cval = read_sysreg_el0(SYS_CNTV_CVAL); + timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTV_CTL)); + timer_set_cval(ctx, read_sysreg_el0(SYS_CNTV_CVAL)); /* Disable the timer */ write_sysreg_el0(0, SYS_CNTV_CTL); @@ -389,8 +419,8 @@ static void timer_save_state(struct arch_timer_context *ctx) break; case TIMER_PTIMER: case TIMER_HPTIMER: - ctx->cnt_ctl = read_sysreg_el0(SYS_CNTP_CTL); - ctx->cnt_cval = read_sysreg_el0(SYS_CNTP_CVAL); + timer_set_ctl(ctx, read_sysreg_el0(SYS_CNTP_CTL)); + timer_set_cval(ctx, read_sysreg_el0(SYS_CNTP_CVAL)); /* Disable the timer */ write_sysreg_el0(0, SYS_CNTP_CTL); @@ -461,15 +491,15 @@ static void timer_restore_state(struct arch_timer_context *ctx) switch (index) { case TIMER_VTIMER: case TIMER_HVTIMER: - write_sysreg_el0(ctx->cnt_cval, SYS_CNTV_CVAL); + write_sysreg_el0(timer_get_cval(ctx), SYS_CNTV_CVAL); isb(); - write_sysreg_el0(ctx->cnt_ctl, SYS_CNTV_CTL); + write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTV_CTL); break; case TIMER_PTIMER: case TIMER_HPTIMER: - write_sysreg_el0(ctx->cnt_cval, SYS_CNTP_CVAL); + write_sysreg_el0(timer_get_cval(ctx), SYS_CNTP_CVAL); isb(); - write_sysreg_el0(ctx->cnt_ctl, SYS_CNTP_CTL); + write_sysreg_el0(timer_get_ctl(ctx), SYS_CNTP_CTL); break; case NR_KVM_TIMERS: BUG(); @@ -608,7 +638,7 @@ void kvm_timer_vcpu_load(struct kvm_vcpu *vcpu) kvm_timer_vcpu_load_nogic(vcpu); } - set_cntvoff(map.direct_vtimer->cntvoff); + set_cntvoff(timer_get_offset(map.direct_vtimer)); kvm_timer_unblocking(vcpu); @@ -722,10 +752,10 @@ int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu) * resets the timer to be disabled and unmasked and is compliant with * the ARMv7 architecture. */ - vcpu_vtimer(vcpu)->cnt_ctl = 0; - vcpu_ptimer(vcpu)->cnt_ctl = 0; - vcpu_hvtimer(vcpu)->cnt_ctl = 0; - vcpu_hptimer(vcpu)->cnt_ctl = 0; + timer_set_ctl(vcpu_vtimer(vcpu), 0); + timer_set_ctl(vcpu_ptimer(vcpu), 0); + timer_set_ctl(vcpu_hvtimer(vcpu), 0); + timer_set_ctl(vcpu_hptimer(vcpu), 0); if (timer->enabled) { kvm_timer_update_irq(vcpu, false, vcpu_vtimer(vcpu)); @@ -757,13 +787,13 @@ static void update_vtimer_cntvoff(struct kvm_vcpu *vcpu, u64 cntvoff) mutex_lock(&kvm->lock); kvm_for_each_vcpu(i, tmp, kvm) - vcpu_vtimer(tmp)->cntvoff = cntvoff; + timer_set_offset(vcpu_vtimer(tmp), cntvoff); /* * When called from the vcpu create path, the CPU being created is not * included in the loop above, so we just set it here as well. */ - vcpu_vtimer(vcpu)->cntvoff = cntvoff; + timer_set_offset(vcpu_vtimer(vcpu), cntvoff); mutex_unlock(&kvm->lock); } @@ -777,9 +807,9 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) /* Synchronize cntvoff across all vtimers of a VM. */ update_vtimer_cntvoff(vcpu, kvm_phys_timer_read()); - ptimer->cntvoff = 0; - hvtimer->cntvoff = 0; - hptimer->cntvoff = 0; + timer_set_offset(ptimer, 0); + timer_set_offset(hvtimer, 0); + timer_set_offset(hptimer, 0); hrtimer_init(&timer->bg_timer, CLOCK_MONOTONIC, HRTIMER_MODE_ABS_HARD); timer->bg_timer.function = kvm_bg_timer_expire; @@ -862,10 +892,12 @@ static u64 read_timer_ctl(struct arch_timer_context *timer) * UNKNOWN when ENABLE bit is 0, so we chose to set ISTATUS bit * regardless of ENABLE bit for our implementation convenience. */ + u32 ctl = timer_get_ctl(timer); + if (!kvm_timer_compute_delta(timer)) - return timer->cnt_ctl | ARCH_TIMER_CTRL_IT_STAT; - else - return timer->cnt_ctl; + ctl |= ARCH_TIMER_CTRL_IT_STAT; + + return ctl; } u64 kvm_arm_timer_get_reg(struct kvm_vcpu *vcpu, u64 regid) @@ -901,8 +933,8 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, switch (treg) { case TIMER_REG_TVAL: - val = timer->cnt_cval - kvm_phys_timer_read() + timer->cntvoff; - val &= lower_32_bits(val); + val = timer_get_cval(timer) - kvm_phys_timer_read() + timer_get_offset(timer); + val = lower_32_bits(val); break; case TIMER_REG_CTL: @@ -910,15 +942,15 @@ static u64 kvm_arm_timer_read(struct kvm_vcpu *vcpu, break; case TIMER_REG_CVAL: - val = timer->cnt_cval; + val = timer_get_cval(timer); break; case TIMER_REG_CNT: - val = kvm_phys_timer_read() - timer->cntvoff; + val = kvm_phys_timer_read() - timer_get_offset(timer); break; case TIMER_REG_VOFF: - val = timer->cntvoff; + val = timer_get_offset(timer); break; default: @@ -952,19 +984,19 @@ static void kvm_arm_timer_write(struct kvm_vcpu *vcpu, { switch (treg) { case TIMER_REG_TVAL: - timer->cnt_cval = kvm_phys_timer_read() - timer->cntvoff + (s32)val; + timer_set_cval(timer, kvm_phys_timer_read() - timer_get_offset(timer) + (s32)val); break; case TIMER_REG_CTL: - timer->cnt_ctl = val & ~ARCH_TIMER_CTRL_IT_STAT; + timer_set_ctl(timer, val & ~ARCH_TIMER_CTRL_IT_STAT); break; case TIMER_REG_CVAL: - timer->cnt_cval = val; + timer_set_cval(timer, val); break; case TIMER_REG_VOFF: - timer->cntvoff = val; + timer_set_offset(timer, val); break; default: From patchwork Tue Feb 11 17:49:25 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376147 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C899A14B4 for ; Tue, 11 Feb 2020 17:51:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A6A9B2086A for ; Tue, 11 Feb 2020 17:51:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443468; bh=vAs368NRaKbhbckeko/jjWU/YpL6QHPO+CUHr/uUINw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=rkxo/L3jJtiWLPQ9kVKcn1+GuZP1TXRO0bLOgZD5yX5M+EVaUTP8QB/5yPU7gcCGk zpAAY1HQTavFVMw9hbI4Y9qodtf3g74Wh12zMb+1uF4sbNnXxQ9v+KYoYDXQ0idlul vXjnLXdwyHTpw9uh+X9SjUQPggU5ACLhs1RrZaFo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730210AbgBKRvI (ORCPT ); Tue, 11 Feb 2020 12:51:08 -0500 Received: from mail.kernel.org ([198.145.29.99]:53290 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730184AbgBKRvH (ORCPT ); Tue, 11 Feb 2020 12:51:07 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id BC52F20870; Tue, 11 Feb 2020 17:51:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443466; bh=vAs368NRaKbhbckeko/jjWU/YpL6QHPO+CUHr/uUINw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=k3x+n34IPHf14JmYAIcLCQUXF5ebOdDALWzW+InXPWDQf/0TVYO9BtPUwiEsonFeX jQ58xgaOkfeL/EemZipxT0Xf4JzZB+xsac9u6Yz30v06m/c7Rk6m/W0qY5d8Myd0D9 ew/gLe5J3PGuxLiztF5481M8tqMGNtWBjqnowB/M= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgH-004O7k-Bb; Tue, 11 Feb 2020 17:50:41 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 81/94] KVM: arm64: Add VNCR-capable timer accessors for arm64 Date: Tue, 11 Feb 2020 17:49:25 +0000 Message-Id: <20200211174938.27809-82-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As promissed, we now add an arm64-specific set of timer accessors, which in some cases access the VNCR page instead of the struct timer fields. This requires moving the initialisation of the timer struct so that some of the helpers (such as arch_timer_ctx_index) can work correctly at an early stage. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 6 ++ arch/arm64/kvm/Makefile | 1 + virt/kvm/arm/arch_timer.c | 10 ++-- virt/kvm/arm/arch_timer_nested.c | 95 +++++++++++++++++++++++++++++++ 4 files changed, 107 insertions(+), 5 deletions(-) create mode 100644 virt/kvm/arm/arch_timer_nested.c diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 7a66abd3df7e..9e48ec1ba057 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -281,6 +281,12 @@ enum vcpu_sysreg { VNCR(ICH_HCR_EL2), VNCR(ICH_VMCR_EL2), + VNCR(CNTVOFF_EL2), + VNCR(CNTV_CVAL_EL0), + VNCR(CNTV_CTL_EL0), + VNCR(CNTP_CVAL_EL0), + VNCR(CNTP_CTL_EL0), + NR_SYS_REGS /* Nothing after this line! */ }; diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index 4521fdfacef0..c4c3eda0496e 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -40,4 +40,5 @@ kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o kvm-$(CONFIG_KVM_ARM_HOST) += nested.o kvm-$(CONFIG_KVM_ARM_HOST) += emulate-nested.o +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer_nested.o kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3-nested.o diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c index d87e52f7e962..f9ef86752630 100644 --- a/virt/kvm/arm/arch_timer.c +++ b/virt/kvm/arm/arch_timer.c @@ -805,6 +805,11 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) struct arch_timer_context *hvtimer = vcpu_hvtimer(vcpu); struct arch_timer_context *hptimer = vcpu_hptimer(vcpu); + vtimer->vcpu = vcpu; + ptimer->vcpu = vcpu; + hvtimer->vcpu = vcpu; + hptimer->vcpu = vcpu; + /* Synchronize cntvoff across all vtimers of a VM. */ update_vtimer_cntvoff(vcpu, kvm_phys_timer_read()); timer_set_offset(ptimer, 0); @@ -838,11 +843,6 @@ void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu) ptimer->host_timer_irq_flags = host_ptimer_irq_flags; hvtimer->host_timer_irq_flags = host_vtimer_irq_flags; hptimer->host_timer_irq_flags = host_ptimer_irq_flags; - - vtimer->vcpu = vcpu; - ptimer->vcpu = vcpu; - hvtimer->vcpu = vcpu; - hptimer->vcpu = vcpu; } static void kvm_timer_init_interrupt(void *info) diff --git a/virt/kvm/arm/arch_timer_nested.c b/virt/kvm/arm/arch_timer_nested.c new file mode 100644 index 000000000000..f081e13e47ad --- /dev/null +++ b/virt/kvm/arm/arch_timer_nested.c @@ -0,0 +1,95 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2019 ARM Ltd. + * Author: Marc Zyngier + * + * Override timer accessors to be nested-capable on ARMv8.4. + */ + +#include +#include + +u32 timer_get_ctl(struct arch_timer_context *ctxt) +{ + struct kvm_vcpu *vcpu = ctxt->vcpu; + + switch(arch_timer_ctx_index(ctxt)) { + case TIMER_VTIMER: + return __vcpu_sys_reg(vcpu, CNTV_CTL_EL0); + case TIMER_PTIMER: + return __vcpu_sys_reg(vcpu, CNTP_CTL_EL0); + default: + return ctxt->cnt_ctl; + } +} + +u64 timer_get_cval(struct arch_timer_context *ctxt) +{ + struct kvm_vcpu *vcpu = ctxt->vcpu; + + switch(arch_timer_ctx_index(ctxt)) { + case TIMER_VTIMER: + return __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0); + case TIMER_PTIMER: + return __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0); + default: + return ctxt->cnt_cval; + } +} + +u64 timer_get_offset(struct arch_timer_context *ctxt) +{ + struct kvm_vcpu *vcpu = ctxt->vcpu; + + switch(arch_timer_ctx_index(ctxt)) { + case TIMER_VTIMER: + return __vcpu_sys_reg(vcpu, CNTVOFF_EL2); + default: + return 0; + } +} + +void timer_set_ctl(struct arch_timer_context *ctxt, u32 ctl) +{ + struct kvm_vcpu *vcpu = ctxt->vcpu; + + switch(arch_timer_ctx_index(ctxt)) { + case TIMER_VTIMER: + __vcpu_sys_reg(vcpu, CNTV_CTL_EL0) = ctl; + break; + case TIMER_PTIMER: + __vcpu_sys_reg(vcpu, CNTP_CTL_EL0) = ctl; + break; + default: + ctxt->cnt_ctl = ctl; + } +} + +void timer_set_cval(struct arch_timer_context *ctxt, u64 cval) +{ + struct kvm_vcpu *vcpu = ctxt->vcpu; + + switch(arch_timer_ctx_index(ctxt)) { + case TIMER_VTIMER: + __vcpu_sys_reg(vcpu, CNTV_CVAL_EL0) = cval; + break; + case TIMER_PTIMER: + __vcpu_sys_reg(vcpu, CNTP_CVAL_EL0) = cval; + break; + default: + ctxt->cnt_cval = cval; + } +} + +void timer_set_offset(struct arch_timer_context *ctxt, u64 offset) +{ + struct kvm_vcpu *vcpu = ctxt->vcpu; + + switch(arch_timer_ctx_index(ctxt)) { + case TIMER_VTIMER: + __vcpu_sys_reg(vcpu, CNTVOFF_EL2) = offset; + break; + default: + WARN(offset, "timer %ld\n", arch_timer_ctx_index(ctxt)); + } +} From patchwork Tue Feb 11 17:49:26 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376185 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 907D51395 for ; Tue, 11 Feb 2020 17:51:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5D75D20870 for ; Tue, 11 Feb 2020 17:51:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443504; bh=Pp+Sia3oFFzauowV5v05awPTOI+wk4zff4Y+ELG9ZVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=0sST/m0Mm9PEia1hcXdvV1wYJBYZC5RgpLfghdiGmvPav6rBbOkhiol7Y1mKstwUk VXa7BhwoPX50kC+7gacJzagnRymTsNRGme7/HF8+7q/KzcVwZu+1h/qVtLPOjYorHS ATx3ZnGZAG/4H9HaP1oPgMvGrsXMWskXVPDgOyx0= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730393AbgBKRvn (ORCPT ); Tue, 11 Feb 2020 12:51:43 -0500 Received: from mail.kernel.org ([198.145.29.99]:54176 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730372AbgBKRvn (ORCPT ); Tue, 11 Feb 2020 12:51:43 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 52672206D7; Tue, 11 Feb 2020 17:51:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443501; bh=Pp+Sia3oFFzauowV5v05awPTOI+wk4zff4Y+ELG9ZVE=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JkPP3SqpBzxDFUrQbBfAy3VqrQ7/zs3BktYAOKA+ciUY7EUeacW4EFyOwzfuwkW3D ftq/hcUO9ISrKXfCuCo0gj1xAYALorGXOdbiES2mgq6IIL5AZl0XLYx/v+Mx0AKMms n3qOcXIyQEfypjXFs/bREo7zwHgKK7XlWdvvjR9A= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgH-004O7k-UH; Tue, 11 Feb 2020 17:50:42 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 82/94] KVM: arm64: Make struct kvm_regs userspace-only Date: Tue, 11 Feb 2020 17:49:26 +0000 Message-Id: <20200211174938.27809-83-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org struct kvm_regs is used by userspace to indicate which register gets accessed by the {GET,SET}_ONE_REG API. But as we're about to refactor the layout of the in-kernel register structures, we need the kernel to move away from it. Let's make kvm_regs userspace only, and let the kernel map it to its own internal representation. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 20 ++++---- arch/arm64/include/asm/kvm_host.h | 12 ++++- arch/arm64/kernel/asm-offsets.c | 3 +- arch/arm64/kvm/fpsimd.c | 2 +- arch/arm64/kvm/guest.c | 70 +++++++++++++++++++++++++--- arch/arm64/kvm/hyp/entry.S | 3 +- arch/arm64/kvm/hyp/switch.c | 4 +- arch/arm64/kvm/hyp/sysreg-sr.c | 30 ++++++------ arch/arm64/kvm/regmap.c | 6 +-- arch/arm64/kvm/reset.c | 2 +- arch/arm64/kvm/sys_regs.c | 12 ++--- 11 files changed, 113 insertions(+), 51 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 1ff1735ff2b3..16cd99ab7170 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -159,12 +159,12 @@ static inline void vcpu_set_vsesr(struct kvm_vcpu *vcpu, u64 vsesr) static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu) { - return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pc; + return (unsigned long *)&vcpu_gp_regs(vcpu)->pc; } static inline unsigned long *__vcpu_elr_el1(const struct kvm_vcpu *vcpu) { - return (unsigned long *)&vcpu_gp_regs(vcpu)->elr_el1; + return (unsigned long *)&vcpu->arch.ctxt.elr_el1; } static inline unsigned long vcpu_read_elr_el1(const struct kvm_vcpu *vcpu) @@ -185,7 +185,7 @@ static inline void vcpu_write_elr_el1(const struct kvm_vcpu *vcpu, unsigned long static inline unsigned long *vcpu_cpsr(const struct kvm_vcpu *vcpu) { - return (unsigned long *)&vcpu_gp_regs(vcpu)->regs.pstate; + return (unsigned long *)&vcpu_gp_regs(vcpu)->pstate; } static inline bool vcpu_mode_is_32bit(const struct kvm_vcpu *vcpu) @@ -214,19 +214,19 @@ static inline void vcpu_set_thumb(struct kvm_vcpu *vcpu) static inline unsigned long vcpu_get_reg(const struct kvm_vcpu *vcpu, u8 reg_num) { - return (reg_num == 31) ? 0 : vcpu_gp_regs(vcpu)->regs.regs[reg_num]; + return (reg_num == 31) ? 0 : vcpu_gp_regs(vcpu)->regs[reg_num]; } static inline void vcpu_set_reg(struct kvm_vcpu *vcpu, u8 reg_num, unsigned long val) { if (reg_num != 31) - vcpu_gp_regs(vcpu)->regs.regs[reg_num] = val; + vcpu_gp_regs(vcpu)->regs[reg_num] = val; } static inline bool vcpu_mode_el2_ctxt(const struct kvm_cpu_context *ctxt) { - unsigned long cpsr = ctxt->gp_regs.regs.pstate; + unsigned long cpsr = ctxt->regs.pstate; switch (cpsr & (PSR_MODE32_BIT | PSR_MODE_MASK)) { case PSR_MODE_EL2h: @@ -327,7 +327,7 @@ static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu) if (vcpu->arch.sysregs_loaded_on_cpu) return read_sysreg_el1(SYS_SPSR); else - return vcpu_gp_regs(vcpu)->spsr[KVM_SPSR_EL1]; + return vcpu->arch.ctxt.spsr[KVM_SPSR_EL1]; } static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v) @@ -345,7 +345,7 @@ static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v) if (vcpu->arch.sysregs_loaded_on_cpu) write_sysreg_el1(v, SYS_SPSR); else - vcpu_gp_regs(vcpu)->spsr[KVM_SPSR_EL1] = v; + vcpu->arch.ctxt.spsr[KVM_SPSR_EL1] = v; } /* @@ -676,11 +676,11 @@ static inline void kvm_skip_instr(struct kvm_vcpu *vcpu, bool is_wide_instr) static inline void __hyp_text __kvm_skip_instr(struct kvm_vcpu *vcpu) { *vcpu_pc(vcpu) = read_sysreg_el2(SYS_ELR); - vcpu->arch.ctxt.gp_regs.regs.pstate = read_sysreg_el2(SYS_SPSR); + vcpu_gp_regs(vcpu)->pstate = read_sysreg_el2(SYS_SPSR); kvm_skip_instr(vcpu, kvm_vcpu_trap_il_is32bit(vcpu)); - write_sysreg_el2(vcpu->arch.ctxt.gp_regs.regs.pstate, SYS_SPSR); + write_sysreg_el2(vcpu_gp_regs(vcpu)->pstate, SYS_SPSR); write_sysreg_el2(*vcpu_pc(vcpu), SYS_ELR); } diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 9e48ec1ba057..29ecd13ab4e2 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -332,7 +332,15 @@ enum vcpu_sysreg { #define NR_COPRO_REGS (NR_SYS_REGS * 2) struct kvm_cpu_context { - struct kvm_regs gp_regs; + struct user_pt_regs regs; /* sp = sp_el0 */ + + u64 sp_el1; + u64 elr_el1; + + u64 spsr[KVM_NR_SPSR]; + + struct user_fpsimd_state fp_regs; + union { u64 sys_regs[NR_SYS_REGS]; u32 copro[NR_COPRO_REGS]; @@ -504,7 +512,7 @@ struct kvm_vcpu_arch { system_supports_generic_auth()) && \ ((vcpu)->arch.flags & KVM_ARM64_GUEST_HAS_PTRAUTH)) -#define vcpu_gp_regs(v) (&(v)->arch.ctxt.gp_regs) +#define vcpu_gp_regs(v) (&(v)->arch.ctxt.regs) /* * Only use __vcpu_sys_reg/ctxt_sys_reg if you know you want the diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index a5bdce8af65b..e5ad90a2543f 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -94,13 +94,12 @@ int main(void) DEFINE(VCPU_FAULT_DISR, offsetof(struct kvm_vcpu, arch.fault.disr_el1)); DEFINE(VCPU_WORKAROUND_FLAGS, offsetof(struct kvm_vcpu, arch.workaround_flags)); DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2)); - DEFINE(CPU_GP_REGS, offsetof(struct kvm_cpu_context, gp_regs)); + DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_cpu_context, regs)); DEFINE(CPU_APIAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1])); DEFINE(CPU_APIBKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIBKEYLO_EL1])); DEFINE(CPU_APDAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APDAKEYLO_EL1])); DEFINE(CPU_APDBKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APDBKEYLO_EL1])); DEFINE(CPU_APGAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APGAKEYLO_EL1])); - DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_regs, regs)); DEFINE(HOST_CONTEXT_VCPU, offsetof(struct kvm_cpu_context, __hyp_running_vcpu)); DEFINE(HOST_DATA_CONTEXT, offsetof(struct kvm_host_data, host_ctxt)); #endif diff --git a/arch/arm64/kvm/fpsimd.c b/arch/arm64/kvm/fpsimd.c index 738ee805aa3c..b983b3f13156 100644 --- a/arch/arm64/kvm/fpsimd.c +++ b/arch/arm64/kvm/fpsimd.c @@ -86,7 +86,7 @@ void kvm_arch_vcpu_ctxsync_fp(struct kvm_vcpu *vcpu) WARN_ON_ONCE(!irqs_disabled()); if (vcpu->arch.flags & KVM_ARM64_FP_ENABLED) { - fpsimd_bind_state_to_cpu(&vcpu->arch.ctxt.gp_regs.fp_regs, + fpsimd_bind_state_to_cpu(&vcpu->arch.ctxt.fp_regs, vcpu->arch.sve_state, vcpu->arch.sve_max_vl); diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index c75b5498c048..fb416f624920 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -104,6 +104,55 @@ static int core_reg_size_from_offset(const struct kvm_vcpu *vcpu, u64 off) return size; } +static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) +{ + u64 off = core_reg_offset_from_id(reg->id); + + switch (off) { + case KVM_REG_ARM_CORE_REG(regs.regs[0]) ... + KVM_REG_ARM_CORE_REG(regs.regs[30]): + off -= KVM_REG_ARM_CORE_REG(regs.regs[0]); + off /= 2; + return &vcpu->arch.ctxt.regs.regs[off]; + + case KVM_REG_ARM_CORE_REG(regs.sp): + return &vcpu->arch.ctxt.regs.sp; + + case KVM_REG_ARM_CORE_REG(regs.pc): + return &vcpu->arch.ctxt.regs.pc; + + case KVM_REG_ARM_CORE_REG(regs.pstate): + return &vcpu->arch.ctxt.regs.pstate; + + case KVM_REG_ARM_CORE_REG(sp_el1): + return &vcpu->arch.ctxt.sp_el1; + + case KVM_REG_ARM_CORE_REG(elr_el1): + return &vcpu->arch.ctxt.elr_el1; + + case KVM_REG_ARM_CORE_REG(spsr[0]) ... + KVM_REG_ARM_CORE_REG(spsr[KVM_NR_SPSR - 1]): + off -= KVM_REG_ARM_CORE_REG(spsr[0]); + off /= 2; + return &vcpu->arch.ctxt.spsr[off]; + + case KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]) ... + KVM_REG_ARM_CORE_REG(fp_regs.vregs[31]): + off -= KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]); + off /= 4; + return &vcpu->arch.ctxt.fp_regs.vregs[off]; + + case KVM_REG_ARM_CORE_REG(fp_regs.fpsr): + return &vcpu->arch.ctxt.fp_regs.fpsr; + + case KVM_REG_ARM_CORE_REG(fp_regs.fpcr): + return &vcpu->arch.ctxt.fp_regs.fpcr; + + default: + return NULL; + } +} + static int validate_core_offset(const struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { @@ -128,8 +177,8 @@ static int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) * off the index in the "array". */ __u32 __user *uaddr = (__u32 __user *)(unsigned long)reg->addr; - struct kvm_regs *regs = vcpu_gp_regs(vcpu); - int nr_regs = sizeof(*regs) / sizeof(__u32); + int nr_regs = sizeof(struct kvm_regs) / sizeof(__u32); + void *addr; u32 off; /* Our ID is an index into the kvm_regs struct. */ @@ -141,7 +190,11 @@ static int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (validate_core_offset(vcpu, reg)) return -EINVAL; - if (copy_to_user(uaddr, ((u32 *)regs) + off, KVM_REG_SIZE(reg->id))) + addr = core_reg_addr(vcpu, reg); + if (!addr) /* Should not happen... */ + return -EINVAL; + + if (copy_to_user(uaddr, addr, KVM_REG_SIZE(reg->id))) return -EFAULT; return 0; @@ -150,10 +203,9 @@ static int get_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) { __u32 __user *uaddr = (__u32 __user *)(unsigned long)reg->addr; - struct kvm_regs *regs = vcpu_gp_regs(vcpu); - int nr_regs = sizeof(*regs) / sizeof(__u32); + int nr_regs = sizeof(struct kvm_regs) / sizeof(__u32); __uint128_t tmp; - void *valp = &tmp; + void *valp = &tmp, *addr; u64 off; int err = 0; @@ -166,6 +218,10 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) if (validate_core_offset(vcpu, reg)) return -EINVAL; + addr = core_reg_addr(vcpu, reg); + if (!addr) /* Should not happen... */ + return -EINVAL; + if (KVM_REG_SIZE(reg->id) > sizeof(tmp)) return -EINVAL; @@ -206,7 +262,7 @@ static int set_core_reg(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) } } - memcpy((u32 *)regs + off, valp, KVM_REG_SIZE(reg->id)); + memcpy(addr, valp, KVM_REG_SIZE(reg->id)); out: return err; } diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index d22d0534dd60..f86f7dc8bebb 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -16,8 +16,7 @@ #include #include -#define CPU_GP_REG_OFFSET(x) (CPU_GP_REGS + x) -#define CPU_XREG_OFFSET(x) CPU_GP_REG_OFFSET(CPU_USER_PT_REGS + 8*x) +#define CPU_XREG_OFFSET(x) (CPU_USER_PT_REGS + 8*x) .text .pushsection .hyp.text, "ax" diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index c5405fa68bc9..a1a9ca35347c 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -461,11 +461,11 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) if (sve_guest) { sve_load_state(vcpu_sve_pffr(vcpu), - &vcpu->arch.ctxt.gp_regs.fp_regs.fpsr, + &vcpu->arch.ctxt.fp_regs.fpsr, sve_vq_from_vl(vcpu->arch.sve_max_vl) - 1); write_sysreg_s(__vcpu_sys_reg(vcpu, ZCR_EL1), SYS_ZCR_EL12); } else { - __fpsimd_restore_state(&vcpu->arch.ctxt.gp_regs.fp_regs); + __fpsimd_restore_state(&vcpu->arch.ctxt.fp_regs); } /* Skip restoring fpexc32 for AArch64 guests */ diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index df9ca6fdf3fb..8b07be72f2e7 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -32,7 +32,7 @@ static void __hyp_text __sysreg_save_common_state(struct kvm_cpu_context *ctxt) * The host arm64 Linux uses sp_el0 to point to 'current' and it must * therefore be saved/restored on every entry/exit to/from the guest. */ - ctxt->gp_regs.regs.sp = read_sysreg(sp_el0); + ctxt->regs.sp = read_sysreg(sp_el0); } static void __hyp_text __sysreg_save_user_state(struct kvm_cpu_context *ctxt) @@ -58,9 +58,9 @@ static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, AMAIR_EL1) = read_sysreg_el1(SYS_AMAIR); ctxt_sys_reg(ctxt, CNTKCTL_EL1) = read_sysreg_el1(SYS_CNTKCTL); - ctxt->gp_regs.sp_el1 = read_sysreg(sp_el1); - ctxt->gp_regs.elr_el1 = read_sysreg_el1(SYS_ELR); - ctxt->gp_regs.spsr[KVM_SPSR_EL1]= read_sysreg_el1(SYS_SPSR); + ctxt->sp_el1 = read_sysreg(sp_el1); + ctxt->elr_el1 = read_sysreg_el1(SYS_ELR); + ctxt->spsr[KVM_SPSR_EL1] = read_sysreg_el1(SYS_SPSR); } static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) @@ -133,8 +133,8 @@ static u64 __hyp_text from_hw_pstate(const struct kvm_cpu_context *ctxt) static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { - ctxt->gp_regs.regs.pc = read_sysreg_el2(SYS_ELR); - ctxt->gp_regs.regs.pstate = from_hw_pstate(ctxt); + ctxt->regs.pc = read_sysreg_el2(SYS_ELR); + ctxt->regs.pstate = from_hw_pstate(ctxt); if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN)) ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2); @@ -169,7 +169,7 @@ static void __hyp_text __sysreg_restore_common_state(struct kvm_cpu_context *ctx * The host arm64 Linux uses sp_el0 to point to 'current' and it must * therefore be saved/restored on every entry/exit to/from the guest. */ - write_sysreg(ctxt->gp_regs.regs.sp, sp_el0); + write_sysreg(ctxt->regs.sp, sp_el0); } static void __hyp_text __sysreg_restore_user_state(struct kvm_cpu_context *ctxt) @@ -323,9 +323,9 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); } - write_sysreg(ctxt->gp_regs.sp_el1, sp_el1); - write_sysreg_el1(ctxt->gp_regs.elr_el1, SYS_ELR); - write_sysreg_el1(ctxt->gp_regs.spsr[KVM_SPSR_EL1],SYS_SPSR); + write_sysreg(ctxt->sp_el1, sp_el1); + write_sysreg_el1(ctxt->elr_el1, SYS_ELR); + write_sysreg_el1(ctxt->spsr[KVM_SPSR_EL1], SYS_SPSR); } static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) @@ -344,7 +344,7 @@ static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) /* Read the VCPU state's PSTATE, but translate (v)EL2 to EL1. */ static u64 __hyp_text to_hw_pstate(const struct kvm_cpu_context *ctxt) { - u64 mode = ctxt->gp_regs.regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); + u64 mode = ctxt->regs.pstate & (PSR_MODE_MASK | PSR_MODE32_BIT); switch (mode) { case PSR_MODE_EL2t: @@ -355,7 +355,7 @@ static u64 __hyp_text to_hw_pstate(const struct kvm_cpu_context *ctxt) break; } - return (ctxt->gp_regs.regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + return (ctxt->regs.pstate & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; } static void __hyp_text @@ -378,7 +378,7 @@ __sysreg_restore_el2_return_state(struct kvm_cpu_context *ctxt) if (!(mode & PSR_MODE32_BIT) && mode >= PSR_MODE_EL2t) pstate = PSR_MODE_EL2h | PSR_IL_BIT; - write_sysreg_el2(ctxt->gp_regs.regs.pc, SYS_ELR); + write_sysreg_el2(ctxt->regs.pc, SYS_ELR); write_sysreg_el2(pstate, SYS_SPSR); if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN)) @@ -413,7 +413,7 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) if (!vcpu_el1_is_32bit(vcpu)) return; - spsr = vcpu->arch.ctxt.gp_regs.spsr; + spsr = vcpu->arch.ctxt.spsr; sysreg = vcpu->arch.ctxt.sys_regs; spsr[KVM_SPSR_ABT] = read_sysreg(spsr_abt); @@ -435,7 +435,7 @@ void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu) if (!vcpu_el1_is_32bit(vcpu)) return; - spsr = vcpu->arch.ctxt.gp_regs.spsr; + spsr = vcpu->arch.ctxt.spsr; sysreg = vcpu->arch.ctxt.sys_regs; write_sysreg(spsr[KVM_SPSR_ABT], spsr_abt); diff --git a/arch/arm64/kvm/regmap.c b/arch/arm64/kvm/regmap.c index a900181e3867..b1596f314087 100644 --- a/arch/arm64/kvm/regmap.c +++ b/arch/arm64/kvm/regmap.c @@ -100,7 +100,7 @@ static const unsigned long vcpu_reg_offsets[VCPU_NR_MODES][16] = { */ unsigned long *vcpu_reg32(const struct kvm_vcpu *vcpu, u8 reg_num) { - unsigned long *reg_array = (unsigned long *)&vcpu->arch.ctxt.gp_regs.regs; + unsigned long *reg_array = (unsigned long *)&vcpu->arch.ctxt.regs; unsigned long mode = *vcpu_cpsr(vcpu) & PSR_AA32_MODE_MASK; switch (mode) { @@ -148,7 +148,7 @@ unsigned long vcpu_read_spsr32(const struct kvm_vcpu *vcpu) int spsr_idx = vcpu_spsr32_mode(vcpu); if (!vcpu->arch.sysregs_loaded_on_cpu) - return vcpu_gp_regs(vcpu)->spsr[spsr_idx]; + return vcpu->arch.ctxt.spsr[spsr_idx]; switch (spsr_idx) { case KVM_SPSR_SVC: @@ -171,7 +171,7 @@ void vcpu_write_spsr32(struct kvm_vcpu *vcpu, unsigned long v) int spsr_idx = vcpu_spsr32_mode(vcpu); if (!vcpu->arch.sysregs_loaded_on_cpu) { - vcpu_gp_regs(vcpu)->spsr[spsr_idx] = v; + vcpu->arch.ctxt.spsr[spsr_idx] = v; return; } diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 038765d1e60d..07462af1b69d 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -307,7 +307,7 @@ int kvm_reset_vcpu(struct kvm_vcpu *vcpu) /* Reset core registers */ memset(vcpu_gp_regs(vcpu), 0, sizeof(*cpu_reset)); - vcpu_gp_regs(vcpu)->regs.pstate = pstate; + vcpu_gp_regs(vcpu)->pstate = pstate; /* Reset system registers */ kvm_reset_sys_regs(vcpu); diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 5a41788054b7..d59fc246aee3 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1805,9 +1805,9 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, { /* SP_EL1 is NOT maintained in sys_regs array */ if (p->is_write) - vcpu->arch.ctxt.gp_regs.sp_el1 = p->regval; + vcpu->arch.ctxt.sp_el1 = p->regval; else - p->regval = vcpu->arch.ctxt.gp_regs.sp_el1; + p->regval = vcpu->arch.ctxt.sp_el1; return true; } @@ -1833,9 +1833,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, return false; if (p->is_write) - vcpu->arch.ctxt.gp_regs.elr_el1 = p->regval; + vcpu->arch.ctxt.elr_el1 = p->regval; else - p->regval = vcpu->arch.ctxt.gp_regs.elr_el1; + p->regval = vcpu->arch.ctxt.elr_el1; return true; } @@ -1851,9 +1851,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, return false; if (p->is_write) - vcpu->arch.ctxt.gp_regs.spsr[KVM_SPSR_EL1] = p->regval; + vcpu->arch.ctxt.spsr[KVM_SPSR_EL1] = p->regval; else - p->regval = vcpu->arch.ctxt.gp_regs.spsr[KVM_SPSR_EL1]; + p->regval = vcpu->arch.ctxt.spsr[KVM_SPSR_EL1]; return true; } From patchwork Tue Feb 11 17:49:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376315 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 521E31395 for ; Tue, 11 Feb 2020 17:53:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3216320848 for ; Tue, 11 Feb 2020 17:53:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443618; bh=/d4QEu6KExCVn1Nyc6BoOADlVZV4Bz226l2YxJeUdxQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=UTHwl46V3UOpxQvar9w6zcBleS5/FesS+qroZiXHPbuNTdxMtDe6xPxwAT8VGzdqz H3fM2F1nUQrWg9wu7caVIPlxIKL2UDkIAuKEBthjojdL/UXi9qAY/C7+ONAe5E5e/O 5nQGBQednMKhTJpx95pSXT5qum022ZzI8OCOIs04= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730779AbgBKRxh (ORCPT ); Tue, 11 Feb 2020 12:53:37 -0500 Received: from mail.kernel.org ([198.145.29.99]:57352 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730756AbgBKRxh (ORCPT ); Tue, 11 Feb 2020 12:53:37 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 1E47220870; Tue, 11 Feb 2020 17:53:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443616; bh=/d4QEu6KExCVn1Nyc6BoOADlVZV4Bz226l2YxJeUdxQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=fDNPjgXdR4X4fmVLG0KXib1sFo/eJ7toCHhIgi7nIdKzE8hMWc7Dp71j+BgEBUpak 4FPgyg+b7PchSyJiFtqaoCVZX4GQaL3WHWtHWu6rAUNzGevpbof+9qmnI1f9uaicb3 4X0gm0vDK18sR2RCEdbxvo/dELdcjaM6qyutqOms= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgI-004O7k-Hg; Tue, 11 Feb 2020 17:50:42 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 83/94] KVM: arm64: VNCR-ize ELR_EL1 Date: Tue, 11 Feb 2020 17:49:27 +0000 Message-Id: <20200211174938.27809-84-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org ELR_EL1 being a VNCR-capable register, let's flag it as such, and repaint all the accesses Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 2 +- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/kvm/hyp/sysreg-sr.c | 4 ++-- arch/arm64/kvm/sys_regs.c | 4 ++-- 5 files changed, 7 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 16cd99ab7170..5718fbc56ba5 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -164,7 +164,7 @@ static inline unsigned long *vcpu_pc(const struct kvm_vcpu *vcpu) static inline unsigned long *__vcpu_elr_el1(const struct kvm_vcpu *vcpu) { - return (unsigned long *)&vcpu->arch.ctxt.elr_el1; + return (unsigned long *)__ctxt_sys_reg(&vcpu->arch.ctxt, ELR_EL1); } static inline unsigned long vcpu_read_elr_el1(const struct kvm_vcpu *vcpu) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 29ecd13ab4e2..a497e7970418 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -264,6 +264,7 @@ enum vcpu_sysreg { VNCR(CONTEXTIDR_EL1), /* Context ID Register */ VNCR(AMAIR_EL1),/* Aux Memory Attribute Indirection Register */ VNCR(MDSCR_EL1),/* Monitor Debug System Control Register */ + VNCR(ELR_EL1), VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */ VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */ VNCR(HCR_EL2), /* Hypervisor Configuration Register */ @@ -335,7 +336,6 @@ struct kvm_cpu_context { struct user_pt_regs regs; /* sp = sp_el0 */ u64 sp_el1; - u64 elr_el1; u64 spsr[KVM_NR_SPSR]; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index fb416f624920..40ef6758266d 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -128,7 +128,7 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return &vcpu->arch.ctxt.sp_el1; case KVM_REG_ARM_CORE_REG(elr_el1): - return &vcpu->arch.ctxt.elr_el1; + return __vcpu_elr_el1(vcpu); case KVM_REG_ARM_CORE_REG(spsr[0]) ... KVM_REG_ARM_CORE_REG(spsr[KVM_NR_SPSR - 1]): diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index 8b07be72f2e7..10ce7a6a0c6c 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -59,7 +59,7 @@ static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, CNTKCTL_EL1) = read_sysreg_el1(SYS_CNTKCTL); ctxt->sp_el1 = read_sysreg(sp_el1); - ctxt->elr_el1 = read_sysreg_el1(SYS_ELR); + ctxt_sys_reg(ctxt, ELR_EL1) = read_sysreg_el1(SYS_ELR); ctxt->spsr[KVM_SPSR_EL1] = read_sysreg_el1(SYS_SPSR); } @@ -324,7 +324,7 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) } write_sysreg(ctxt->sp_el1, sp_el1); - write_sysreg_el1(ctxt->elr_el1, SYS_ELR); + write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL1), SYS_ELR); write_sysreg_el1(ctxt->spsr[KVM_SPSR_EL1], SYS_SPSR); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index d59fc246aee3..dc1b75a37f00 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1833,9 +1833,9 @@ static bool access_elr(struct kvm_vcpu *vcpu, return false; if (p->is_write) - vcpu->arch.ctxt.elr_el1 = p->regval; + *__vcpu_elr_el1(vcpu) = p->regval; else - p->regval = vcpu->arch.ctxt.elr_el1; + p->regval = *__vcpu_elr_el1(vcpu); return true; } From patchwork Tue Feb 11 17:49:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376179 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7ADD614B4 for ; Tue, 11 Feb 2020 17:51:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5A754206D7 for ; Tue, 11 Feb 2020 17:51:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443498; bh=vJgCeyNhc/zjTIzrf5MG6+BOepmc9H4if6t7gh/kTSw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=gbTxRMaw2kjl+tmmRkyRyFwuSg1vClhdZ8DMc9puhNgEgrioHOVMe0OWd6x+GmM8q 5FGKgns26Qb04iwT5HSLuyPWtwkmhVNkYx2+Ly/YCrU+eFs0DCp4sPPBi+4bm70VOq UZjU6TYLPmQjfAtDHK4VGQRZl9Yz/7bulTNMvqp8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730386AbgBKRvh (ORCPT ); Tue, 11 Feb 2020 12:51:37 -0500 Received: from mail.kernel.org ([198.145.29.99]:54036 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730330AbgBKRvg (ORCPT ); Tue, 11 Feb 2020 12:51:36 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id B11DD20578; Tue, 11 Feb 2020 17:51:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443495; bh=vJgCeyNhc/zjTIzrf5MG6+BOepmc9H4if6t7gh/kTSw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ge+7LsrMQjgNYp3YORkBfaJvbrDPYo1v/5naMcnSuHcXLhaRrtON8wAkI3dHwSLkX 81ziN2BC0cmYcNCwOpLvcxQuA1z9tF948+ntPi8gRPCGV6VAQmWmr8JotPIGO7tyGD J7gUxDJX7x4va5Hm7y7Tf+CdhP24aakywn+rO2pI= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgJ-004O7k-4I; Tue, 11 Feb 2020 17:50:43 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 84/94] KVM: arm64: VNCR-ize SP_EL1 Date: Tue, 11 Feb 2020 17:49:28 +0000 Message-Id: <20200211174938.27809-85-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org SP_EL1 being a VNCR-capable register, let's flag it as such, and repaint all the accesses Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 3 +-- arch/arm64/kvm/guest.c | 2 +- arch/arm64/kvm/hyp/sysreg-sr.c | 4 ++-- arch/arm64/kvm/sys_regs.c | 5 ++--- 4 files changed, 6 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a497e7970418..fe6d5a3bc338 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -265,6 +265,7 @@ enum vcpu_sysreg { VNCR(AMAIR_EL1),/* Aux Memory Attribute Indirection Register */ VNCR(MDSCR_EL1),/* Monitor Debug System Control Register */ VNCR(ELR_EL1), + VNCR(SP_EL1), VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */ VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */ VNCR(HCR_EL2), /* Hypervisor Configuration Register */ @@ -335,8 +336,6 @@ enum vcpu_sysreg { struct kvm_cpu_context { struct user_pt_regs regs; /* sp = sp_el0 */ - u64 sp_el1; - u64 spsr[KVM_NR_SPSR]; struct user_fpsimd_state fp_regs; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 40ef6758266d..869017e50464 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -125,7 +125,7 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return &vcpu->arch.ctxt.regs.pstate; case KVM_REG_ARM_CORE_REG(sp_el1): - return &vcpu->arch.ctxt.sp_el1; + return __ctxt_sys_reg(&vcpu->arch.ctxt, SP_EL1); case KVM_REG_ARM_CORE_REG(elr_el1): return __vcpu_elr_el1(vcpu); diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index 10ce7a6a0c6c..e5359cd39b32 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -58,7 +58,7 @@ static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, AMAIR_EL1) = read_sysreg_el1(SYS_AMAIR); ctxt_sys_reg(ctxt, CNTKCTL_EL1) = read_sysreg_el1(SYS_CNTKCTL); - ctxt->sp_el1 = read_sysreg(sp_el1); + ctxt_sys_reg(ctxt, SP_EL1) = read_sysreg(sp_el1); ctxt_sys_reg(ctxt, ELR_EL1) = read_sysreg_el1(SYS_ELR); ctxt->spsr[KVM_SPSR_EL1] = read_sysreg_el1(SYS_SPSR); } @@ -323,7 +323,7 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) write_sysreg_el1(ctxt_sys_reg(ctxt, TCR_EL1), SYS_TCR); } - write_sysreg(ctxt->sp_el1, sp_el1); + write_sysreg(ctxt_sys_reg(ctxt, SP_EL1), sp_el1); write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL1), SYS_ELR); write_sysreg_el1(ctxt->spsr[KVM_SPSR_EL1], SYS_SPSR); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index dc1b75a37f00..2c3e90e192b4 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1803,11 +1803,10 @@ static bool access_sp_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { - /* SP_EL1 is NOT maintained in sys_regs array */ if (p->is_write) - vcpu->arch.ctxt.sp_el1 = p->regval; + __vcpu_sys_reg(vcpu, SP_EL1) = p->regval; else - p->regval = vcpu->arch.ctxt.sp_el1; + p->regval = __vcpu_sys_reg(vcpu, SP_EL1); return true; } From patchwork Tue Feb 11 17:49:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376235 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12A301395 for ; Tue, 11 Feb 2020 17:52:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DDE2B20870 for ; Tue, 11 Feb 2020 17:52:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443548; bh=ZBHGkyRfdRnKRz+z/OfxfM8/L6bY0Ui9wuEIUUN1SVM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=YmC7taYvTtVdwbbPeX/6uQMTCXZIq58R7Ov2gTvV479vq1UKuNTQatme2B5IX/DkN lFwAROzn3TQrthv25AcXpNSRovUl0hX8AlHOjg5UAPQA+UmihsDlSK65Gs2oZ+uGlr rE9sj7uSrt70/SLxgWesG5gO7jBkiNUM82C8mTtw= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730535AbgBKRw0 (ORCPT ); Tue, 11 Feb 2020 12:52:26 -0500 Received: from mail.kernel.org ([198.145.29.99]:55338 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730540AbgBKRw0 (ORCPT ); Tue, 11 Feb 2020 12:52:26 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3410720848; Tue, 11 Feb 2020 17:52:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443545; bh=ZBHGkyRfdRnKRz+z/OfxfM8/L6bY0Ui9wuEIUUN1SVM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=pKLQChaKWRGnI/aGHcuXjbhZwv+vg0a13Z+4y7hQYVPLOmnvirN75iifF7jKEd7Kz 6+ZrXCjf3EAmgTDv6dxoNjc579CPSaUOO6XrP1UEK0YogRi2mauDtdTjEWIeO0UzjK 2IJd473aoUoQlA8ZQyDpIUyVNOvzXy72dLZS47R4= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgJ-004O7k-NF; Tue, 11 Feb 2020 17:50:43 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 85/94] KVM: arm64: Disintegrate SPSR array Date: Tue, 11 Feb 2020 17:49:29 +0000 Message-Id: <20200211174938.27809-86-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As we're about to move SPSR_EL1 into the VNCR page, we need to disassociate it from the rest of the 32bit cruft. Let's break the array into individual fields. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_host.h | 6 ++++- arch/arm64/kvm/guest.c | 19 +++++++++++---- arch/arm64/kvm/hyp/sysreg-sr.c | 26 ++++++++++----------- arch/arm64/kvm/regmap.c | 35 +++++++++++++++++++++++++--- arch/arm64/kvm/sys_regs.c | 4 ++-- 6 files changed, 67 insertions(+), 27 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 5718fbc56ba5..eed8bd48e60c 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -327,7 +327,7 @@ static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu) if (vcpu->arch.sysregs_loaded_on_cpu) return read_sysreg_el1(SYS_SPSR); else - return vcpu->arch.ctxt.spsr[KVM_SPSR_EL1]; + return vcpu->arch.ctxt.spsr_el1; } static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v) @@ -345,7 +345,7 @@ static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v) if (vcpu->arch.sysregs_loaded_on_cpu) write_sysreg_el1(v, SYS_SPSR); else - vcpu->arch.ctxt.spsr[KVM_SPSR_EL1] = v; + vcpu->arch.ctxt.spsr_el1 = v; } /* diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index fe6d5a3bc338..111597123c17 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -336,7 +336,11 @@ enum vcpu_sysreg { struct kvm_cpu_context { struct user_pt_regs regs; /* sp = sp_el0 */ - u64 spsr[KVM_NR_SPSR]; + u64 spsr_el1; /* aka spsr_svc */ + u64 spsr_abt; + u64 spsr_und; + u64 spsr_irq; + u64 spsr_fiq; struct user_fpsimd_state fp_regs; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 869017e50464..9710fe55c5dd 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -130,11 +130,20 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) case KVM_REG_ARM_CORE_REG(elr_el1): return __vcpu_elr_el1(vcpu); - case KVM_REG_ARM_CORE_REG(spsr[0]) ... - KVM_REG_ARM_CORE_REG(spsr[KVM_NR_SPSR - 1]): - off -= KVM_REG_ARM_CORE_REG(spsr[0]); - off /= 2; - return &vcpu->arch.ctxt.spsr[off]; + case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_EL1]): + return &vcpu->arch.ctxt.spsr_el1; + + case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_ABT]): + return &vcpu->arch.ctxt.spsr_abt; + + case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_UND]): + return &vcpu->arch.ctxt.spsr_und; + + case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_IRQ]): + return &vcpu->arch.ctxt.spsr_irq; + + case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_FIQ]): + return &vcpu->arch.ctxt.spsr_fiq; case KVM_REG_ARM_CORE_REG(fp_regs.vregs[0]) ... KVM_REG_ARM_CORE_REG(fp_regs.vregs[31]): diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index e5359cd39b32..afc603f800e9 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -60,7 +60,7 @@ static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, SP_EL1) = read_sysreg(sp_el1); ctxt_sys_reg(ctxt, ELR_EL1) = read_sysreg_el1(SYS_ELR); - ctxt->spsr[KVM_SPSR_EL1] = read_sysreg_el1(SYS_SPSR); + ctxt->spsr_el1 = read_sysreg_el1(SYS_SPSR); } static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) @@ -325,7 +325,7 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) write_sysreg(ctxt_sys_reg(ctxt, SP_EL1), sp_el1); write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL1), SYS_ELR); - write_sysreg_el1(ctxt->spsr[KVM_SPSR_EL1], SYS_SPSR); + write_sysreg_el1(ctxt->spsr_el1, SYS_SPSR); } static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) @@ -408,18 +408,17 @@ NOKPROBE_SYMBOL(sysreg_restore_guest_state_vhe); void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) { - u64 *spsr, *sysreg; + u64 *sysreg; if (!vcpu_el1_is_32bit(vcpu)) return; - spsr = vcpu->arch.ctxt.spsr; sysreg = vcpu->arch.ctxt.sys_regs; - spsr[KVM_SPSR_ABT] = read_sysreg(spsr_abt); - spsr[KVM_SPSR_UND] = read_sysreg(spsr_und); - spsr[KVM_SPSR_IRQ] = read_sysreg(spsr_irq); - spsr[KVM_SPSR_FIQ] = read_sysreg(spsr_fiq); + vcpu->arch.ctxt.spsr_abt = read_sysreg(spsr_abt); + vcpu->arch.ctxt.spsr_und = read_sysreg(spsr_und); + vcpu->arch.ctxt.spsr_irq = read_sysreg(spsr_irq); + vcpu->arch.ctxt.spsr_fiq = read_sysreg(spsr_fiq); sysreg[DACR32_EL2] = read_sysreg(dacr32_el2); sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2); @@ -430,18 +429,17 @@ void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu) { - u64 *spsr, *sysreg; + u64 *sysreg; if (!vcpu_el1_is_32bit(vcpu)) return; - spsr = vcpu->arch.ctxt.spsr; sysreg = vcpu->arch.ctxt.sys_regs; - write_sysreg(spsr[KVM_SPSR_ABT], spsr_abt); - write_sysreg(spsr[KVM_SPSR_UND], spsr_und); - write_sysreg(spsr[KVM_SPSR_IRQ], spsr_irq); - write_sysreg(spsr[KVM_SPSR_FIQ], spsr_fiq); + write_sysreg(vcpu->arch.ctxt.spsr_abt, spsr_abt); + write_sysreg(vcpu->arch.ctxt.spsr_und, spsr_und); + write_sysreg(vcpu->arch.ctxt.spsr_irq, spsr_irq); + write_sysreg(vcpu->arch.ctxt.spsr_fiq, spsr_fiq); write_sysreg(sysreg[DACR32_EL2], dacr32_el2); write_sysreg(sysreg[IFSR32_EL2], ifsr32_el2); diff --git a/arch/arm64/kvm/regmap.c b/arch/arm64/kvm/regmap.c index b1596f314087..97c110810527 100644 --- a/arch/arm64/kvm/regmap.c +++ b/arch/arm64/kvm/regmap.c @@ -147,8 +147,20 @@ unsigned long vcpu_read_spsr32(const struct kvm_vcpu *vcpu) { int spsr_idx = vcpu_spsr32_mode(vcpu); - if (!vcpu->arch.sysregs_loaded_on_cpu) - return vcpu->arch.ctxt.spsr[spsr_idx]; + if (!vcpu->arch.sysregs_loaded_on_cpu) { + switch (spsr_idx) { + case KVM_SPSR_SVC: + return vcpu->arch.ctxt.spsr_el1; + case KVM_SPSR_ABT: + return vcpu->arch.ctxt.spsr_abt; + case KVM_SPSR_UND: + return vcpu->arch.ctxt.spsr_und; + case KVM_SPSR_IRQ: + return vcpu->arch.ctxt.spsr_irq; + case KVM_SPSR_FIQ: + return vcpu->arch.ctxt.spsr_fiq; + } + } switch (spsr_idx) { case KVM_SPSR_SVC: @@ -171,7 +183,24 @@ void vcpu_write_spsr32(struct kvm_vcpu *vcpu, unsigned long v) int spsr_idx = vcpu_spsr32_mode(vcpu); if (!vcpu->arch.sysregs_loaded_on_cpu) { - vcpu->arch.ctxt.spsr[spsr_idx] = v; + switch (spsr_idx) { + case KVM_SPSR_SVC: + vcpu->arch.ctxt.spsr_el1 = v; + break; + case KVM_SPSR_ABT: + vcpu->arch.ctxt.spsr_abt = v; + break; + case KVM_SPSR_UND: + vcpu->arch.ctxt.spsr_und = v; + break; + case KVM_SPSR_IRQ: + vcpu->arch.ctxt.spsr_irq = v; + break; + case KVM_SPSR_FIQ: + vcpu->arch.ctxt.spsr_fiq = v; + break; + } + return; } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 2c3e90e192b4..0d3500a11c50 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1850,9 +1850,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, return false; if (p->is_write) - vcpu->arch.ctxt.spsr[KVM_SPSR_EL1] = p->regval; + vcpu->arch.ctxt.spsr_el1 = p->regval; else - p->regval = vcpu->arch.ctxt.spsr[KVM_SPSR_EL1]; + p->regval = vcpu->arch.ctxt.spsr_el1; return true; } From patchwork Tue Feb 11 17:49:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376287 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 82EB11395 for ; Tue, 11 Feb 2020 17:53:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 61E8B20661 for ; Tue, 11 Feb 2020 17:53:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443592; bh=1DR8MVhjtgLl66se5UHs3QX+PvNEC+nXqVOa/l9sjS8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=I82vn1I9MfJTVlIOWlyis2dzhyiey54EsD/v7rXKB1jvjUyfuDDzG02uWaVinmC5u LPB1zTt/zq+mFfGF429zO8yBntifxJGitrxzIGJqk1zXbiQe8XnXv+q2jqI+lCdVqi FBYYEI++yei+nCPbpKM4OtSJHoKpI3fhOOYG3YQc= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730651AbgBKRxL (ORCPT ); Tue, 11 Feb 2020 12:53:11 -0500 Received: from mail.kernel.org ([198.145.29.99]:56584 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730605AbgBKRxK (ORCPT ); Tue, 11 Feb 2020 12:53:10 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3BD7920870; Tue, 11 Feb 2020 17:53:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443590; bh=1DR8MVhjtgLl66se5UHs3QX+PvNEC+nXqVOa/l9sjS8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=NkNjJwlA0UByXWecN+9DcEs8lWdDkVjUPPbG02Zl7ayKBPrmqK2qwcL5jnVwwaZ1p vY0SmFQEzHXnq/rt5WmDpr0nRsYxtdODV13QZE1BMLgEPjxJa3UoI7sHXvrs3Am6sR JC7lg1SdMFfRwI592p5qmyleo4lB/vmvedK/bAjk= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgK-004O7k-AX; Tue, 11 Feb 2020 17:50:44 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 86/94] KVM: arm64: aarch32: Use __vcpu_sys_reg() instead of raw sys_regs access Date: Tue, 11 Feb 2020 17:49:30 +0000 Message-Id: <20200211174938.27809-87-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Instead of directly accessing the sysreg array, let's use convenient accessors that will hide the complexity of the register remapping. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/switch.c | 5 ++--- arch/arm64/kvm/hyp/sysreg-sr.c | 20 ++++++-------------- 2 files changed, 8 insertions(+), 17 deletions(-) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index a1a9ca35347c..3deea28794e1 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -49,7 +49,7 @@ static void __hyp_text __fpsimd_save_fpexc32(struct kvm_vcpu *vcpu) if (!vcpu_el1_is_32bit(vcpu)) return; - vcpu->arch.ctxt.sys_regs[FPEXC32_EL2] = read_sysreg(fpexc32_el2); + __vcpu_sys_reg(vcpu, FPEXC32_EL2) = read_sysreg(fpexc32_el2); } static void __hyp_text __activate_traps_fpsimd32(struct kvm_vcpu *vcpu) @@ -470,8 +470,7 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) /* Skip restoring fpexc32 for AArch64 guests */ if (!(read_sysreg(hcr_el2) & HCR_RW)) - write_sysreg(vcpu->arch.ctxt.sys_regs[FPEXC32_EL2], - fpexc32_el2); + write_sysreg(__vcpu_sys_reg(vcpu, FPEXC32_EL2), fpexc32_el2); vcpu->arch.flags |= KVM_ARM64_FP_ENABLED; diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index afc603f800e9..b009afd87322 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -408,44 +408,36 @@ NOKPROBE_SYMBOL(sysreg_restore_guest_state_vhe); void __hyp_text __sysreg32_save_state(struct kvm_vcpu *vcpu) { - u64 *sysreg; - if (!vcpu_el1_is_32bit(vcpu)) return; - sysreg = vcpu->arch.ctxt.sys_regs; - vcpu->arch.ctxt.spsr_abt = read_sysreg(spsr_abt); vcpu->arch.ctxt.spsr_und = read_sysreg(spsr_und); vcpu->arch.ctxt.spsr_irq = read_sysreg(spsr_irq); vcpu->arch.ctxt.spsr_fiq = read_sysreg(spsr_fiq); - sysreg[DACR32_EL2] = read_sysreg(dacr32_el2); - sysreg[IFSR32_EL2] = read_sysreg(ifsr32_el2); + __vcpu_sys_reg(vcpu, DACR32_EL2) = read_sysreg(dacr32_el2); + __vcpu_sys_reg(vcpu, IFSR32_EL2) = read_sysreg(ifsr32_el2); if (has_vhe() || vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY) - sysreg[DBGVCR32_EL2] = read_sysreg(dbgvcr32_el2); + __vcpu_sys_reg(vcpu, DBGVCR32_EL2) = read_sysreg(dbgvcr32_el2); } void __hyp_text __sysreg32_restore_state(struct kvm_vcpu *vcpu) { - u64 *sysreg; - if (!vcpu_el1_is_32bit(vcpu)) return; - sysreg = vcpu->arch.ctxt.sys_regs; - write_sysreg(vcpu->arch.ctxt.spsr_abt, spsr_abt); write_sysreg(vcpu->arch.ctxt.spsr_und, spsr_und); write_sysreg(vcpu->arch.ctxt.spsr_irq, spsr_irq); write_sysreg(vcpu->arch.ctxt.spsr_fiq, spsr_fiq); - write_sysreg(sysreg[DACR32_EL2], dacr32_el2); - write_sysreg(sysreg[IFSR32_EL2], ifsr32_el2); + write_sysreg(__vcpu_sys_reg(vcpu, DACR32_EL2), dacr32_el2); + write_sysreg(__vcpu_sys_reg(vcpu, IFSR32_EL2), ifsr32_el2); if (has_vhe() || vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY) - write_sysreg(sysreg[DBGVCR32_EL2], dbgvcr32_el2); + write_sysreg(__vcpu_sys_reg(vcpu, DBGVCR32_EL2), dbgvcr32_el2); } /** From patchwork Tue Feb 11 17:49:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376309 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D10861395 for ; Tue, 11 Feb 2020 17:53:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B0B992168B for ; Tue, 11 Feb 2020 17:53:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443612; bh=uuLHNfNqbhykqXJ9OzYGP0lyKUcFqC0HNltRPFdpYz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=PjimgnOiV8LiMuhs4sV70p6/uSNpDu4dJnYID0haJtIPyRBS7cxnovbYhyAt2oT3t b5BYQeQeEm/YkOZibdzmvUKKPHEvXXgs7eZYlAN9gxgT6bCv5pry12BxeVKSuuHWPb F2hp0vnE9Hm8G2TB49ToS39j3cK71PIvI8aXQ430= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730184AbgBKRxb (ORCPT ); Tue, 11 Feb 2020 12:53:31 -0500 Received: from mail.kernel.org ([198.145.29.99]:57140 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730637AbgBKRxb (ORCPT ); Tue, 11 Feb 2020 12:53:31 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9DC7020848; Tue, 11 Feb 2020 17:53:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443610; bh=uuLHNfNqbhykqXJ9OzYGP0lyKUcFqC0HNltRPFdpYz8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=g/DCsWRjqy8g4WNASBYHLgVAIpeEil/l8KpiT8y2zSrbO/Ga7rjvZPjNOcIZqbe7z Ssk0BrNQTY3FwRdNdktWOfAjZsMmwA6EAovirRiOJ0bhJLe6ZEyp3O5PXZViNPG57x jQTBS8xqv7rXDToTTencxTXIQMrUoay999JuhKhE= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgK-004O7k-SA; Tue, 11 Feb 2020 17:50:44 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 87/94] KVM: arm64: VNCR-ize SPSR_EL1 Date: Tue, 11 Feb 2020 17:49:31 +0000 Message-Id: <20200211174938.27809-88-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As promised, let's move SPSR_EL1 to the VNCR page. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 4 ++-- arch/arm64/include/asm/kvm_host.h | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/kvm/hyp/sysreg-sr.c | 4 ++-- arch/arm64/kvm/regmap.c | 4 ++-- arch/arm64/kvm/sys_regs.c | 4 ++-- 6 files changed, 10 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index eed8bd48e60c..6df684e1790e 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -327,7 +327,7 @@ static inline unsigned long vcpu_read_spsr(const struct kvm_vcpu *vcpu) if (vcpu->arch.sysregs_loaded_on_cpu) return read_sysreg_el1(SYS_SPSR); else - return vcpu->arch.ctxt.spsr_el1; + return __vcpu_sys_reg(vcpu, SPSR_EL1); } static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v) @@ -345,7 +345,7 @@ static inline void vcpu_write_spsr(struct kvm_vcpu *vcpu, unsigned long v) if (vcpu->arch.sysregs_loaded_on_cpu) write_sysreg_el1(v, SYS_SPSR); else - vcpu->arch.ctxt.spsr_el1 = v; + __vcpu_sys_reg(vcpu, SPSR_EL1) = v; } /* diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 111597123c17..657c2c5d1a39 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -266,6 +266,7 @@ enum vcpu_sysreg { VNCR(MDSCR_EL1),/* Monitor Debug System Control Register */ VNCR(ELR_EL1), VNCR(SP_EL1), + VNCR(SPSR_EL1), VNCR(VPIDR_EL2),/* Virtualization Processor ID Register */ VNCR(VMPIDR_EL2),/* Virtualization Multiprocessor ID Register */ VNCR(HCR_EL2), /* Hypervisor Configuration Register */ @@ -336,7 +337,6 @@ enum vcpu_sysreg { struct kvm_cpu_context { struct user_pt_regs regs; /* sp = sp_el0 */ - u64 spsr_el1; /* aka spsr_svc */ u64 spsr_abt; u64 spsr_und; u64 spsr_irq; diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index 9710fe55c5dd..e295368a13f2 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -131,7 +131,7 @@ static void *core_reg_addr(struct kvm_vcpu *vcpu, const struct kvm_one_reg *reg) return __vcpu_elr_el1(vcpu); case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_EL1]): - return &vcpu->arch.ctxt.spsr_el1; + return __ctxt_sys_reg(&vcpu->arch.ctxt, SPSR_EL1); case KVM_REG_ARM_CORE_REG(spsr[KVM_SPSR_ABT]): return &vcpu->arch.ctxt.spsr_abt; diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index b009afd87322..eaf78e9b3238 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -60,7 +60,7 @@ static void __hyp_text __sysreg_save_vel1_state(struct kvm_cpu_context *ctxt) ctxt_sys_reg(ctxt, SP_EL1) = read_sysreg(sp_el1); ctxt_sys_reg(ctxt, ELR_EL1) = read_sysreg_el1(SYS_ELR); - ctxt->spsr_el1 = read_sysreg_el1(SYS_SPSR); + ctxt_sys_reg(ctxt, SPSR_EL1) = read_sysreg_el1(SYS_SPSR); } static void __sysreg_save_vel2_state(struct kvm_cpu_context *ctxt) @@ -325,7 +325,7 @@ static void __hyp_text __sysreg_restore_vel1_state(struct kvm_cpu_context *ctxt) write_sysreg(ctxt_sys_reg(ctxt, SP_EL1), sp_el1); write_sysreg_el1(ctxt_sys_reg(ctxt, ELR_EL1), SYS_ELR); - write_sysreg_el1(ctxt->spsr_el1, SYS_SPSR); + write_sysreg_el1(ctxt_sys_reg(ctxt, SPSR_EL1), SYS_SPSR); } static void __hyp_text __sysreg_restore_el1_state(struct kvm_cpu_context *ctxt) diff --git a/arch/arm64/kvm/regmap.c b/arch/arm64/kvm/regmap.c index 97c110810527..accc1d5fba61 100644 --- a/arch/arm64/kvm/regmap.c +++ b/arch/arm64/kvm/regmap.c @@ -150,7 +150,7 @@ unsigned long vcpu_read_spsr32(const struct kvm_vcpu *vcpu) if (!vcpu->arch.sysregs_loaded_on_cpu) { switch (spsr_idx) { case KVM_SPSR_SVC: - return vcpu->arch.ctxt.spsr_el1; + return __vcpu_sys_reg(vcpu, SPSR_EL1); case KVM_SPSR_ABT: return vcpu->arch.ctxt.spsr_abt; case KVM_SPSR_UND: @@ -185,7 +185,7 @@ void vcpu_write_spsr32(struct kvm_vcpu *vcpu, unsigned long v) if (!vcpu->arch.sysregs_loaded_on_cpu) { switch (spsr_idx) { case KVM_SPSR_SVC: - vcpu->arch.ctxt.spsr_el1 = v; + __vcpu_sys_reg(vcpu, SPSR_EL1) = v; break; case KVM_SPSR_ABT: vcpu->arch.ctxt.spsr_abt = v; diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index 0d3500a11c50..f20f5975633f 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -1850,9 +1850,9 @@ static bool access_spsr(struct kvm_vcpu *vcpu, return false; if (p->is_write) - vcpu->arch.ctxt.spsr_el1 = p->regval; + __vcpu_sys_reg(vcpu, SPSR_EL1) = p->regval; else - p->regval = vcpu->arch.ctxt.spsr_el1; + p->regval = __vcpu_sys_reg(vcpu, SPSR_EL1); return true; } From patchwork Tue Feb 11 17:49:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376181 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D2E2E14B4 for ; Tue, 11 Feb 2020 17:51:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B2AD321569 for ; Tue, 11 Feb 2020 17:51:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443499; bh=c5I/DnWBHK7CF1Hug7c8EUPZyhvhzoIsYWcfsfMzPEo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=pqHjCB2qr/dBXdXNOvNSmw11z1hsFJQzWbUPlpw8KTOoyPo0BQmMulWJkpYdgtCx1 85UfvMjO1zYXegQmwSntx97XEPuSCQqw0YVKoNBPpUPB/EG3tZVgXVmBYHisfQQGuV 2l/RrZo/NGyCRVw+L5FDsQ2WCkLUNgoA4WUkRmb4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730397AbgBKRvj (ORCPT ); Tue, 11 Feb 2020 12:51:39 -0500 Received: from mail.kernel.org ([198.145.29.99]:54072 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730330AbgBKRvi (ORCPT ); Tue, 11 Feb 2020 12:51:38 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 84ACA20661; Tue, 11 Feb 2020 17:51:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443497; bh=c5I/DnWBHK7CF1Hug7c8EUPZyhvhzoIsYWcfsfMzPEo=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=1NNVhjng+8MtN6zoASSmNHt5TCZkZPglFujZ+pmQ1HsXWEDqvuxwzYXoM7NRAuYd+ PdGvtwsFIRccT+3waMwUvPNFQbebRWD12fTMZjWfiatJ7WseUp48snhPHHiXjreg1z 54JTpTcmoQK7v2Uvo+PqTYX+//AHEHzlP59VlY1g= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgL-004O7k-ET; Tue, 11 Feb 2020 17:50:45 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 88/94] KVM: arm64: Add ARMv8.4 Enhanced Nested Virt cpufeature Date: Tue, 11 Feb 2020 17:49:32 +0000 Message-Id: <20200211174938.27809-89-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add the detection code for the ARMv8.4-NV feature. Signed-off-by: Marc Zyngier --- arch/arm/include/asm/kvm_nested.h | 1 + arch/arm64/include/asm/cpucaps.h | 3 ++- arch/arm64/include/asm/kvm_nested.h | 6 ++++++ arch/arm64/kernel/cpufeature.c | 10 ++++++++++ 4 files changed, 19 insertions(+), 1 deletion(-) diff --git a/arch/arm/include/asm/kvm_nested.h b/arch/arm/include/asm/kvm_nested.h index 2b89e6fa7323..1b97863ae811 100644 --- a/arch/arm/include/asm/kvm_nested.h +++ b/arch/arm/include/asm/kvm_nested.h @@ -5,6 +5,7 @@ #include static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) { return false; } +static inline bool enhanced_nested_virt_in_use(const struct kvm_vcpu *vcpu) { return false; } static inline void check_nested_vcpu_requests(struct kvm_vcpu *vcpu) {} #endif /* __ARM_KVM_NESTED_H */ diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h index 5736650cd0fb..565e4878e301 100644 --- a/arch/arm64/include/asm/cpucaps.h +++ b/arch/arm64/include/asm/cpucaps.h @@ -60,7 +60,8 @@ #define ARM64_HAS_RNG 50 #define ARM64_HAS_NESTED_VIRT 51 #define ARM64_HAS_ARMv8_4_TTL 52 +#define ARM64_HAS_ENHANCED_NESTED_VIRT 53 -#define ARM64_NCAPS 53 +#define ARM64_NCAPS 54 #endif /* __ASM_CPUCAPS_H */ diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index 3e3778d3cec6..284c794a10ac 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -11,6 +11,12 @@ static inline bool nested_virt_in_use(const struct kvm_vcpu *vcpu) test_bit(KVM_ARM_VCPU_HAS_EL2, vcpu->arch.features); } +static inline bool enhanced_nested_virt_in_use(const struct kvm_vcpu *vcpu) +{ + return cpus_have_const_cap(ARM64_HAS_ENHANCED_NESTED_VIRT) && + nested_virt_in_use(vcpu); +} + extern void kvm_init_nested(struct kvm *kvm); extern int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu); extern void kvm_init_nested_s2_mmu(struct kvm_s2_mmu *mmu); diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index aada8c3eff1e..546aceb70c3f 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1454,6 +1454,16 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .field_pos = ID_AA64MMFR2_NV_SHIFT, .min_field_value = 1, }, + { + .desc = "Enhanced Nested Virtualization Support", + .capability = ARM64_HAS_ENHANCED_NESTED_VIRT, + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .matches = has_nested_virt_support, + .sys_reg = SYS_ID_AA64MMFR2_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR2_NV_SHIFT, + .min_field_value = 2, + }, #endif /* CONFIG_ARM64_VHE */ { .desc = "32-bit EL0 Support", From patchwork Tue Feb 11 17:49:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376199 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 66BB41395 for ; Tue, 11 Feb 2020 17:51:55 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 455D0206CC for ; Tue, 11 Feb 2020 17:51:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443515; bh=f33AsioL7+uzWUtpnmAwqyECVEr0La+k9kBb0QdEzio=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=K9O9iIq2isMfh+zpOsMM+u9wIKH0H/EpgNrxeKha1yywHIG9E3ivYY9befvn9v97E koOlX9oWCB9xtlswN/P0Zh25gjZTczrN4fMqESLCYsz6m4szFPgDV/GqidvsmK4P9K hcQz3abKEsIf6CSda0ZAAw//8ozSMzgCs2CwdlY4= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730451AbgBKRvy (ORCPT ); Tue, 11 Feb 2020 12:51:54 -0500 Received: from mail.kernel.org ([198.145.29.99]:54518 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730460AbgBKRvy (ORCPT ); Tue, 11 Feb 2020 12:51:54 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id A48C6208C3; Tue, 11 Feb 2020 17:51:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443512; bh=f33AsioL7+uzWUtpnmAwqyECVEr0La+k9kBb0QdEzio=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=RssqqczRl8ErSQxoAf+QfaTBvF8UEXGhf1QiEHJo1Mxn2tMIewbNiFBpab2Tdv2Wf A0z4fMM9tkq2sMYLf4hjTddHs4uj4oJmXqMnzkA+j9l2b7zDJNL49ENIvxjrsGoSnY Fc53rAzdceNzGakmzSASPPX+2d2VqWHUjZYdajHU= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgM-004O7k-0f; Tue, 11 Feb 2020 17:50:46 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 89/94] KVM: arm64: nv: Synchronize PSTATE early on exit Date: Tue, 11 Feb 2020 17:49:33 +0000 Message-Id: <20200211174938.27809-90-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The NV code relies on predicates such as is_hyp_ctxt() being reliable. In turn, is_hyp_ctxt() relies on things like PSTATE and the virtual HCR_EL2 being accurate. But with ARMv8.4-NV removing trapping for a large part of the EL2 system registers (among which HCR_EL2), we can't use such trapping to synchronize the rest of the state. Let's look at the following sequence for a VHE guest: (1) enter guest in host EL0 (2) guest traps to guest vEL2 (no hypervisor intervention) (3) guest clears virtual HCR_EL2.TGE (no trap either) (4) host interrupt fires, exit (5) is_hyp_ctxt() now says "guest" (PSTATE.M==EL1 and TGE==0) It is obvious that such behaviour would be rather unfortunate, and lead to interesting, difficult to catch bugs specially if preemption kicks in (yes, I wasted a whole week chasing this one). In order to preserve the invariant that a guest entered in host context must exit in the same context, we must make sure that is_hyp_ctxt() works correctly. Since we can always observe the guest value of HCR_EL2.{E2H,TGE} in the VNCR_EL2 page, we solely need to synchronize PSTATE as early as possible. This basically amounts to moving from_hw_pstate() as close as possible to the guest exit point, and fixup_guest_exit() seems as good a place as any. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/switch.c | 51 ++++++++++++++++++++++++++++++++-- arch/arm64/kvm/hyp/sysreg-sr.c | 26 ++--------------- 2 files changed, 51 insertions(+), 26 deletions(-) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 3deea28794e1..c35e67241d8e 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -537,8 +537,45 @@ static bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu) * the guest, false when we should restore the host state and return to the * main run loop. */ -static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) +static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code, + bool hyp_ctxt) { + /* + * Sync pstate back as early as possible, so that is_hyp_ctxt() + * reflects the exact context. It is otherwise possible to get + * confused with a VHE guest and ARMv8.4-NV, such as: + * + * (1) enter guest in host EL0 + * (2) guest traps to guest vEL2 (no hypervisor intervention) + * (3) guest clears virtual HCR_EL2.TGE (no trap either) + * (4) host interrupt fires, exit + * (5) is_hyp_ctxt() now says "guest" (pstate.M==EL1 and TGE==0) + * + * If host preemption occurs, vcpu_load/put() will be very confused. + * This of course only matters to VHE. + */ + + if (has_vhe()) { + u64 pstate = read_sysreg_el2(SYS_SPSR); + + if (unlikely(hyp_ctxt)) { + u64 mode = pstate & PSR_MODE_MASK; + + switch (mode) { + case PSR_MODE_EL1t: + mode = PSR_MODE_EL2t; + break; + case PSR_MODE_EL1h: + mode = PSR_MODE_EL2h; + break; + } + + pstate = (pstate & ~PSR_MODE_MASK) | mode; + } + + *vcpu_cpsr(vcpu) = pstate; + } + if (ARM_EXCEPTION_CODE(*exit_code) != ARM_EXCEPTION_IRQ) vcpu->arch.fault.esr_el2 = read_sysreg_el2(SYS_ESR); @@ -683,6 +720,7 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) struct kvm_cpu_context *host_ctxt; struct kvm_cpu_context *guest_ctxt; u64 exit_code; + bool hyp_ctxt; host_ctxt = vcpu->arch.host_cpu_context; host_ctxt->__hyp_running_vcpu = vcpu; @@ -709,12 +747,19 @@ int kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) __set_guest_arch_workaround_state(vcpu); + /* + * Being in HYP context or not is an invariant here. If we enter in + * a given context, we exit in the same context. We can thus only + * sample the context once. + */ + hyp_ctxt = is_hyp_ctxt(vcpu); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu, host_ctxt); /* And we're baaack! */ - } while (fixup_guest_exit(vcpu, &exit_code)); + } while (fixup_guest_exit(vcpu, &exit_code, hyp_ctxt)); __set_host_arch_workaround_state(vcpu); @@ -788,7 +833,7 @@ int __hyp_text __kvm_vcpu_run_nvhe(struct kvm_vcpu *vcpu) exit_code = __guest_enter(vcpu, host_ctxt); /* And we're baaack! */ - } while (fixup_guest_exit(vcpu, &exit_code)); + } while (fixup_guest_exit(vcpu, &exit_code, false)); __set_host_arch_workaround_state(vcpu); diff --git a/arch/arm64/kvm/hyp/sysreg-sr.c b/arch/arm64/kvm/hyp/sysreg-sr.c index eaf78e9b3238..53ece83bf062 100644 --- a/arch/arm64/kvm/hyp/sysreg-sr.c +++ b/arch/arm64/kvm/hyp/sysreg-sr.c @@ -109,32 +109,12 @@ static void __hyp_text __sysreg_save_el1_state(struct kvm_cpu_context *ctxt) __sysreg_save_vel1_state(ctxt); } -static u64 __hyp_text from_hw_pstate(const struct kvm_cpu_context *ctxt) -{ - u64 reg = read_sysreg_el2(SYS_SPSR); - - if (__is_hyp_ctxt(ctxt)) { - u64 mode = reg & (PSR_MODE_MASK | PSR_MODE32_BIT); - - switch (mode) { - case PSR_MODE_EL1t: - mode = PSR_MODE_EL2t; - break; - case PSR_MODE_EL1h: - mode = PSR_MODE_EL2h; - break; - } - - return (reg & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; - } - - return reg; -} - static void __hyp_text __sysreg_save_el2_return_state(struct kvm_cpu_context *ctxt) { + /* On VHE, PSTATE is saved in fixup_guest_exit() */ + if (!has_vhe()) + ctxt->regs.pstate = read_sysreg_el2(SYS_SPSR); ctxt->regs.pc = read_sysreg_el2(SYS_ELR); - ctxt->regs.pstate = from_hw_pstate(ctxt); if (cpus_have_const_cap(ARM64_HAS_RAS_EXTN)) ctxt_sys_reg(ctxt, DISR_EL1) = read_sysreg_s(SYS_VDISR_EL2); From patchwork Tue Feb 11 17:49:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376165 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8113714B4 for ; Tue, 11 Feb 2020 17:51:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 61A03208C3 for ; Tue, 11 Feb 2020 17:51:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443484; bh=ZVZYEbQ3L/ZhoKR/Tpg2cujpRiPoqAF6wbwuNvpBz5o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=NQYwqeIWRaTp31gqrizCdMGchNvZFI5RevozNM2VUPf2s6fChH0yEI0rHYbXjWLIx Keg/C1BAI3tt+rhdIgxfTdJ8s/8ynnxFMw8NMYc1sealpkEfUZHZ7WPXcmJbJdICoH HyQffRpLPwOIH7vRoP13na/zhefbQMYoHEeAz9jA= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730328AbgBKRvX (ORCPT ); Tue, 11 Feb 2020 12:51:23 -0500 Received: from mail.kernel.org ([198.145.29.99]:53712 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730304AbgBKRvX (ORCPT ); Tue, 11 Feb 2020 12:51:23 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 5F54E20870; Tue, 11 Feb 2020 17:51:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443482; bh=ZVZYEbQ3L/ZhoKR/Tpg2cujpRiPoqAF6wbwuNvpBz5o=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=t+vgQctMWbpQKaRhXVd3mCvkKfXcMYMwJc0ULkPZWwrqbddICINbhfDZlZF/5IwOi 18Hrp7e7wfBHUBx4xE4N9Dn1FUpx4S6zkouJCbb8XfwqQoqRwm+NFpt8bXxpXDIqRH t1Xtvgx3Ut/D43gPbyPpf2C2FBVTD0vbCLH7lmtk= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgM-004O7k-KG; Tue, 11 Feb 2020 17:50:46 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 90/94] KVM: arm64: nv: Sync nested timer state with ARMv8.4 Date: Tue, 11 Feb 2020 17:49:34 +0000 Message-Id: <20200211174938.27809-91-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org From: Christoffer Dall Emulating the ARMv8.4-NV timers is a bit odd, as the timers can be reconfigured behind our back without the hypervisor even noticing. In the VHE case, that's an actual regression in the architecture... Signed-off-by: Christoffer Dall Signed-off-by: Marc Zyngier --- include/kvm/arm_arch_timer.h | 3 ++- virt/kvm/arm/arch_timer.c | 41 +++++++++++++++++++++++++++++++++++- virt/kvm/arm/arm.c | 7 ++++-- 3 files changed, 47 insertions(+), 4 deletions(-) diff --git a/include/kvm/arm_arch_timer.h b/include/kvm/arm_arch_timer.h index 3389606f3029..2442de2c4a6e 100644 --- a/include/kvm/arm_arch_timer.h +++ b/include/kvm/arm_arch_timer.h @@ -75,7 +75,8 @@ int kvm_timer_hyp_init(bool); int kvm_timer_enable(struct kvm_vcpu *vcpu); int kvm_timer_vcpu_reset(struct kvm_vcpu *vcpu); void kvm_timer_vcpu_init(struct kvm_vcpu *vcpu); -void kvm_timer_sync_hwstate(struct kvm_vcpu *vcpu); +void kvm_timer_sync_nested(struct kvm_vcpu *vcpu); +void kvm_timer_sync_user(struct kvm_vcpu *vcpu); bool kvm_timer_should_notify_user(struct kvm_vcpu *vcpu); void kvm_timer_update_run(struct kvm_vcpu *vcpu); void kvm_timer_vcpu_terminate(struct kvm_vcpu *vcpu); diff --git a/virt/kvm/arm/arch_timer.c b/virt/kvm/arm/arch_timer.c index f9ef86752630..b95798f81764 100644 --- a/virt/kvm/arm/arch_timer.c +++ b/virt/kvm/arm/arch_timer.c @@ -710,6 +710,45 @@ void kvm_timer_vcpu_put(struct kvm_vcpu *vcpu) set_cntvoff(0); } +void kvm_timer_sync_nested(struct kvm_vcpu *vcpu) +{ +#ifdef CONFIG_ARM64 + if (!is_hyp_ctxt(vcpu)) + return; + + /* + * Guest hypervisors using ARMv8.4 enhanced nested virt support have + * their EL1 timer register accesses redirected to the VNCR page. + */ + if (!vcpu_el2_e2h_is_set(vcpu)) { + /* + * For a non-VHE guest hypervisor, we update the hardware + * timer registers with the latest value written by the guest + * to the VNCR page and let the hardware take care of the + * rest. + */ + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CTL_EL0), SYS_CNTV_CTL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTV_CVAL_EL0), SYS_CNTV_CVAL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTP_CTL_EL0), SYS_CNTP_CTL); + write_sysreg_el0(__vcpu_sys_reg(vcpu, CNTP_CVAL_EL0), SYS_CNTP_CVAL); + } else { + /* + * For a VHE guest hypervisor, the emulated state (which + * is stored in the VNCR page) could have been updated behind + * our back, and we must reset the emulation of the timers. + */ + + struct timer_map map; + get_timer_map(vcpu, &map); + + soft_timer_cancel(&map.emul_vtimer->hrtimer); + soft_timer_cancel(&map.emul_ptimer->hrtimer); + timer_emulate(map.emul_vtimer); + timer_emulate(map.emul_ptimer); + } +#endif +} + /* * With a userspace irqchip we have to check if the guest de-asserted the * timer and if so, unmask the timer irq signal on the host interrupt @@ -728,7 +767,7 @@ static void unmask_vtimer_irq_user(struct kvm_vcpu *vcpu) } } -void kvm_timer_sync_hwstate(struct kvm_vcpu *vcpu) +void kvm_timer_sync_user(struct kvm_vcpu *vcpu) { struct arch_timer_cpu *timer = vcpu_timer(vcpu); diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c index fa73b7a464f1..e8fedbfacc2c 100644 --- a/virt/kvm/arm/arm.c +++ b/virt/kvm/arm/arm.c @@ -730,7 +730,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) isb(); /* Ensure work in x_flush_hwstate is committed */ kvm_pmu_sync_hwstate(vcpu); if (static_branch_unlikely(&userspace_irqchip_in_use)) - kvm_timer_sync_hwstate(vcpu); + kvm_timer_sync_user(vcpu); kvm_vgic_sync_hwstate(vcpu); local_irq_enable(); preempt_enable(); @@ -781,7 +781,10 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu, struct kvm_run *run) * timer virtual interrupt state. */ if (static_branch_unlikely(&userspace_irqchip_in_use)) - kvm_timer_sync_hwstate(vcpu); + kvm_timer_sync_user(vcpu); + + if (enhanced_nested_virt_in_use(vcpu)) + kvm_timer_sync_nested(vcpu); kvm_arch_vcpu_ctxsync_fp(vcpu); From patchwork Tue Feb 11 17:49:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376255 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 838D71395 for ; Tue, 11 Feb 2020 17:52:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 63D8D20870 for ; Tue, 11 Feb 2020 17:52:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443564; bh=jf1mgKr2G0XkF+KTZXgNSKPUDy7zcmwp7k/gfZ+D5NQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=GY84ATZ1BqIYgS9+4m2FR+AuTzXCnTeIl5aUddSWdkWID/0ItDT7Vku9BpthvpCLw U1vszthZylahSXDSwG4GNsGfuS7qyKBVcP02G+hIeJyDRbW12urhONM+DdEQwga/KF 2f65epuG4MgUXXqZV4/IHIT7kgilAxb/NGsinJTo= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730598AbgBKRwn (ORCPT ); Tue, 11 Feb 2020 12:52:43 -0500 Received: from mail.kernel.org ([198.145.29.99]:55836 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730560AbgBKRwm (ORCPT ); Tue, 11 Feb 2020 12:52:42 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id D8B7420578; Tue, 11 Feb 2020 17:52:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443562; bh=jf1mgKr2G0XkF+KTZXgNSKPUDy7zcmwp7k/gfZ+D5NQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=p2iVJhngVaayQs+ogEg5UMk2fmKw+R2vLFGwPyllyZtwBwqx9GboWD7L9ZB9ohGgG Ti5ou0XY5O0TcDpzHNO9uG0CannBOmmR8QQuJputpDkU5C65lYbCIwP203zNQh72SB cb9LNFkv2QuYbG5/sWp10V00wNUUn6yPMAlj0TiQ= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgN-004O7k-68; Tue, 11 Feb 2020 17:50:47 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 91/94] KVM: arm64: nv: Allocate VNCR page when required Date: Tue, 11 Feb 2020 17:49:35 +0000 Message-Id: <20200211174938.27809-92-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org If running a NV guest on an ARMv8.4-NV capable system, let's allocate an additional page that will be used by the hypervisor to fulfill system register accesses. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 3 ++- arch/arm64/kvm/nested.c | 8 ++++++++ arch/arm64/kvm/reset.c | 1 + 3 files changed, 11 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 657c2c5d1a39..a26ab0fda1a7 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -526,7 +526,8 @@ struct kvm_vcpu_arch { */ static inline u64 *__ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) { - if (unlikely(r >= __VNCR_START__ && ctxt->vncr_array)) + if (unlikely(cpus_have_const_cap(ARM64_HAS_ENHANCED_NESTED_VIRT) && + r >= __VNCR_START__ && ctxt->vncr_array)) return &ctxt->vncr_array[r - __VNCR_START__]; return (u64 *)&ctxt->sys_regs[r]; diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index 6adff9b433e0..58873e4c85cb 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -47,6 +47,12 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) if (!cpus_have_const_cap(ARM64_HAS_NESTED_VIRT)) return -EINVAL; + if (cpus_have_const_cap(ARM64_HAS_ENHANCED_NESTED_VIRT)) { + vcpu->arch.ctxt.vncr_array = (u64 *)__get_free_page(GFP_KERNEL | __GFP_ZERO); + if (!vcpu->arch.ctxt.vncr_array) + return -ENOMEM; + } + mutex_lock(&kvm->lock); /* @@ -64,6 +70,8 @@ int kvm_vcpu_init_nested(struct kvm_vcpu *vcpu) kvm_init_stage2_mmu(kvm, &tmp[num_mmus - 2])) { kvm_free_stage2_pgd(&tmp[num_mmus - 1]); kvm_free_stage2_pgd(&tmp[num_mmus - 2]); + free_page((unsigned long)vcpu->arch.ctxt.vncr_array); + vcpu->arch.ctxt.vncr_array = NULL; } else { kvm->arch.nested_mmus_size = num_mmus; ret = 0; diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c index 07462af1b69d..ce8d8f53fcd0 100644 --- a/arch/arm64/kvm/reset.c +++ b/arch/arm64/kvm/reset.c @@ -207,6 +207,7 @@ bool kvm_arm_vcpu_is_finalized(struct kvm_vcpu *vcpu) void kvm_arm_vcpu_destroy(struct kvm_vcpu *vcpu) { kfree(vcpu->arch.sve_state); + free_page((unsigned long)vcpu->arch.ctxt.vncr_array); } static void kvm_vcpu_reset_sve(struct kvm_vcpu *vcpu) From patchwork Tue Feb 11 17:49:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376193 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id ACD1A14B4 for ; Tue, 11 Feb 2020 17:51:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8BFFF215A4 for ; Tue, 11 Feb 2020 17:51:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443509; bh=hnoUIwyz2ne+WiDjbigmpdDVGqHLmBjcecy2gPUXZoU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=sNC+uhPFLw+kgrCmzu71nO4n54hGMka0/8W+FLyhRUd5dQmkCoJcUl7xx6xhe9uM4 CUKasObl9Bpv1X5cuz9qYMB1U2G0+Tk3XpR6gyNkbjdXHUJUBeh+GIYPLI836HYmz/ MplBO3CQuSTEqJZomUnKoVafVybQUnhmdjmMXRV8= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730438AbgBKRvs (ORCPT ); Tue, 11 Feb 2020 12:51:48 -0500 Received: from mail.kernel.org ([198.145.29.99]:54374 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730304AbgBKRvs (ORCPT ); Tue, 11 Feb 2020 12:51:48 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 0DE8520870; Tue, 11 Feb 2020 17:51:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443507; bh=hnoUIwyz2ne+WiDjbigmpdDVGqHLmBjcecy2gPUXZoU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Zm64Xlch1/gLOYjkhzVVQw1kxqDeDmAiCYKcD7gmbKV/VobY19ZoHEfhVa00p0utB bIMTFWsE7Rtr0LV6Cm9KXv5gO84fR+43elJpKZNUtdTLFy1BObOKcwjvNJ+ACRY2qk X87CjDeTDNO5FhK0o1KrenryXTWMoCTMzS1DNHOc= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgN-004O7k-Ov; Tue, 11 Feb 2020 17:50:47 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 92/94] KVM: arm64: nv: Enable ARMv8.4-NV support Date: Tue, 11 Feb 2020 17:49:36 +0000 Message-Id: <20200211174938.27809-93-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org As all the VNCR-capable system registers are nicely separated from the rest of the crowd, let's set HCR_EL2.NV2 on and let the ball rolling. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_arm.h | 1 + arch/arm64/include/asm/kvm_emulate.h | 23 +++++++++++++---------- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/hyp/switch.c | 18 +++++++++++++++--- 4 files changed, 30 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/kvm_arm.h b/arch/arm64/include/asm/kvm_arm.h index 2e5be806a5c9..d558f8ebea7e 100644 --- a/arch/arm64/include/asm/kvm_arm.h +++ b/arch/arm64/include/asm/kvm_arm.h @@ -13,6 +13,7 @@ /* Hyp Configuration Register (HCR) bits */ #define HCR_FWB (UL(1) << 46) +#define HCR_NV2 (UL(1) << 45) #define HCR_AT (UL(1) << 44) #define HCR_NV1 (UL(1) << 43) #define HCR_NV (UL(1) << 42) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index 6df684e1790e..b87757f21c42 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -281,21 +281,24 @@ static inline bool is_hyp_ctxt(const struct kvm_vcpu *vcpu) static inline u64 __fixup_spsr_el2_write(struct kvm_cpu_context *ctxt, u64 val) { - if (!__vcpu_el2_e2h_is_set(ctxt)) { - /* - * Clear the .M field when writing SPSR to the CPU, so that we - * can detect when the CPU clobbered our SPSR copy during a - * local exception. - */ - val &= ~0xc; - } + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + + if (enhanced_nested_virt_in_use(vcpu) || __vcpu_el2_e2h_is_set(ctxt)) + return val; - return val; + /* + * Clear the .M field when writing SPSR to the CPU, so that we + * can detect when the CPU clobbered our SPSR copy during a + * local exception. + */ + return val &= ~0xc; } static inline u64 __fixup_spsr_el2_read(const struct kvm_cpu_context *ctxt, u64 val) { - if (__vcpu_el2_e2h_is_set(ctxt)) + struct kvm_vcpu *vcpu = container_of(ctxt, struct kvm_vcpu, arch.ctxt); + + if (enhanced_nested_virt_in_use(vcpu) || __vcpu_el2_e2h_is_set(ctxt)) return val; /* diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index a402e762c51d..6a466bc66901 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -425,6 +425,7 @@ #define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) #define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) #define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) +#define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0) #define SYS_DACR32_EL2 sys_reg(3, 4, 3, 0, 0) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index c35e67241d8e..2eca04ca96b6 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -160,7 +160,13 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) * the EL1 virtual memory control register accesses * as well as the AT S1 operations. */ - hcr |= HCR_TVM | HCR_TRVM | HCR_AT | HCR_TTLB | HCR_NV1; + if (enhanced_nested_virt_in_use(vcpu)) { + hcr &= ~HCR_TVM; + } else { + hcr |= HCR_TVM | HCR_TRVM | HCR_TTLB; + } + + hcr |= HCR_AT | HCR_NV1; } else { /* * For a guest hypervisor on v8.1 (VHE), allow to @@ -186,12 +192,18 @@ static void __hyp_text __activate_traps(struct kvm_vcpu *vcpu) /* * If we're using the EL1 translation regime - * (TGE clear, then ensure that AT S1 ops are - * trapped too. + * (TGE clear), then ensure that AT S1 and + * TLBI E1 ops are trapped too. */ if (!vcpu_el2_tge_is_set(vcpu)) hcr |= HCR_AT | HCR_TTLB; } + + if (enhanced_nested_virt_in_use(vcpu)) { + hcr |= HCR_AT | HCR_TTLB | HCR_NV2; + write_sysreg_s(vcpu->arch.ctxt.vncr_array, + SYS_VNCR_EL2); + } } else if (nested_virt_in_use(vcpu)) { hcr |= __vcpu_sys_reg(vcpu, HCR_EL2); } From patchwork Tue Feb 11 17:49:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376201 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87A3E14B4 for ; Tue, 11 Feb 2020 17:51:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 683B620848 for ; Tue, 11 Feb 2020 17:51:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443517; bh=BVvwjLMYZVm7q18m99ri4igIk6u8itMAhMuWbJme6zg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=1VIRd7S9Al/q8Vs810wPL82+ZhiCBBtdmeWCza2y3Jt6brDrSaKBJB0EyFNHQG9O5 ti9xMAy2D1B358UkMQP304S3nRN74DrJofRaNy5f3ilOHJ+MSX9lCTWY87Xp/XoUjA eFn0PQZaHa5hyNVjWcWI/uELlPGFiCCrrLpwr67Y= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730459AbgBKRv4 (ORCPT ); Tue, 11 Feb 2020 12:51:56 -0500 Received: from mail.kernel.org ([198.145.29.99]:54554 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730345AbgBKRvz (ORCPT ); Tue, 11 Feb 2020 12:51:55 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 8A10120578; Tue, 11 Feb 2020 17:51:54 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443514; bh=BVvwjLMYZVm7q18m99ri4igIk6u8itMAhMuWbJme6zg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=GNA8U5qzuds0iaK+s1BHhLSVjmDMx3qEPwngRkQux8elYfDSmnI9G+Pz3OR5pwb/b 0utYORagW5GDM/Lzv8khaDRPGq9fu1PC5cNYfYrTAmwMdLT8bEKCff+/tX+ZrWBBL5 dwfzDsE4KdBkRCQ4rgDQnp5TDgVIbkjSiKDDJ6fg= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgO-004O7k-C5; Tue, 11 Feb 2020 17:50:48 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 93/94] KVM: arm64: nv: Fast-track 'InHost' exception returns Date: Tue, 11 Feb 2020 17:49:37 +0000 Message-Id: <20200211174938.27809-94-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org A significant part of the ARMv8.3-NV extension is to trap ERET instructions so that the hypervisor gets a chance to switch from a vEL2 L1 guest to an EL1 L2 guest. But this also has the unfortunate consequence of trapping ERET in unsuspecting circumstances, such as staying at vEL2 (interrupt handling while being in the guest hypervisor), or returning to host userspace in the case of a VHE guest. Although we already make some effort to handle these ERET quicker by not doing the put/load dance, it is still way too far down the line for it to be efficient enough. For these cases, it would ideal to ERET directly, no question asked. Of course, we can't do that. But the next best thing is to do it as early as possible, in fixup_guest_exit(), much as we would handle FPSIMD exceptions. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/emulate-nested.c | 26 ++-------------- arch/arm64/kvm/hyp/switch.c | 54 +++++++++++++++++++++++++++++++++ 2 files changed, 56 insertions(+), 24 deletions(-) diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 97af080b9b52..eef7394e5256 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -78,8 +78,7 @@ static u64 get_el2_except_vector(struct kvm_vcpu *vcpu, void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) { - u64 spsr, elr, mode; - bool direct_eret; + u64 spsr, elr; /* * Forward this trap to the virtual EL2 if the virtual @@ -88,31 +87,10 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) if (forward_nv_traps(vcpu)) return; - /* - * Going through the whole put/load motions is a waste of time - * if this is a VHE guest hypervisor returning to its own - * userspace, or the hypervisor performing a local exception - * return. No need to save/restore registers, no need to - * switch S2 MMU. Just do the canonical ERET. - */ - spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); - mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); - - direct_eret = (mode == PSR_MODE_EL0t && - vcpu_el2_e2h_is_set(vcpu) && - vcpu_el2_tge_is_set(vcpu)); - direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); - - if (direct_eret) { - *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); - *vcpu_cpsr(vcpu) = spsr; - trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); - return; - } - preempt_disable(); kvm_arch_vcpu_put(vcpu); + spsr = __vcpu_sys_reg(vcpu, SPSR_EL2); elr = __vcpu_sys_reg(vcpu, ELR_EL2); trace_kvm_nested_eret(vcpu, elr, spsr); diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index 2eca04ca96b6..d8aae4b5f089 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -489,6 +489,57 @@ static bool __hyp_text __hyp_handle_fpsimd(struct kvm_vcpu *vcpu) return true; } +static bool __hyp_text __hyp_handle_eret(struct kvm_vcpu *vcpu) +{ + struct kvm_cpu_context *ctxt = &vcpu->arch.ctxt; + u64 spsr, mode; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. Unless the trap + * has to be forwarded further down the line, of course... + */ + if (kvm_vcpu_trap_get_class(vcpu) != ESR_ELx_EC_ERET) + return false; + + /* + * Let the trap forwarding be handled by the normal exception + * handling code. + */ + if (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) + return false; + + spsr = read_sysreg_el1(SYS_SPSR); + spsr = __fixup_spsr_el2_read(ctxt, spsr); + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL0t: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + return false; + break; + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + default: + return false; + } + + spsr = (spsr & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + + write_sysreg_el2(spsr, SYS_SPSR); + write_sysreg_el2(read_sysreg_el1(SYS_ELR), SYS_ELR); + + return true; +} + + static bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu) { u32 sysreg = esr_sys64_to_sysreg(kvm_vcpu_get_hsr(vcpu)); @@ -615,6 +666,9 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code, if (__hyp_handle_fpsimd(vcpu)) return true; + if (__hyp_handle_eret(vcpu)) + return true; + if (!__populate_fault_info(vcpu)) return true; From patchwork Tue Feb 11 17:49:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 11376301 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA03A1395 for ; Tue, 11 Feb 2020 17:53:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 88F4F20870 for ; Tue, 11 Feb 2020 17:53:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443605; bh=nQAHpVL2nQ+w5tVvVTUPy86BjEvLrpd6ejEhrIIhTy0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:List-ID:From; b=R82Mdtx4pznsAx1vNykRuckpUWvvGdL9rkf5UOZGkTyKzMwZ0VZ8LYDv+n6QZm/ju RbRiyZ19pDtTys4REp4/yolv+IULWiKNo01ZM1CLWgFrErktZLGHPai+XFm3SSeMbc frCm6lUnHy8kL/CXG/sJH9TBRmppfglyYExfuGxU= Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730730AbgBKRxY (ORCPT ); Tue, 11 Feb 2020 12:53:24 -0500 Received: from mail.kernel.org ([198.145.29.99]:56950 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730671AbgBKRxY (ORCPT ); Tue, 11 Feb 2020 12:53:24 -0500 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 3269E206D7; Tue, 11 Feb 2020 17:53:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1581443603; bh=nQAHpVL2nQ+w5tVvVTUPy86BjEvLrpd6ejEhrIIhTy0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=E6EOLZcy2osyUIfHwfxM1Yo1Ij7GHJC+4X8MClNJdq89jKUrgLgq7PLco/rhHhXDl ghOp+Y4NMlbVJWg4dSfFq4MIW6pORTSUJULq08S1v+8itBMThFwgZSOnxG4q5ruYbb 0BjvB3r2BwyNpabfh2pPalzZX5MGN53jf53ihgFo= Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1j1ZgO-004O7k-Uu; Tue, 11 Feb 2020 17:50:49 +0000 From: Marc Zyngier To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org Cc: Andre Przywara , Christoffer Dall , Dave Martin , Jintack Lim , Alexandru Elisei , James Morse , Julien Thierry , Suzuki K Poulose Subject: [PATCH v2 94/94] KVM: arm64: nv: Fast-track EL1 TLBIs for VHE guests Date: Tue, 11 Feb 2020 17:49:38 +0000 Message-Id: <20200211174938.27809-95-maz@kernel.org> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200211174938.27809-1-maz@kernel.org> References: <20200211174938.27809-1-maz@kernel.org> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, andre.przywara@arm.com, christoffer.dall@arm.com, Dave.Martin@arm.com, jintack@cs.columbia.edu, alexandru.elisei@arm.com, james.morse@arm.com, julien.thierry.kdev@gmail.com, suzuki.poulose@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Due to the way ARMv8.4-NV suppresses traps when accessing EL2 system registers, we can't track when the guest changes its HCR_EL2.TGE setting. This means we alwats trap EL1 TLBIs, even if they don't affect any guest. This obviously has a huge impact on performance, as we handle TLBI traps as a normal exit, and a normal VHE host issues thousands of TLBIs when booting (and quite a few when running userspace). A cheap way to reduce the overhead is to handle the limited case of {E2H,TGE}=={1,1} as a guest fixup, as we already have the right mmu configuration in place. Just execute the decoded instruction right away and return to the guest. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/switch.c | 35 +++++++++++++++++++++++++++++++++++ arch/arm64/kvm/hyp/tlb.c | 6 ++++-- arch/arm64/kvm/sys_regs.c | 28 +++++++++------------------- 3 files changed, 48 insertions(+), 21 deletions(-) diff --git a/arch/arm64/kvm/hyp/switch.c b/arch/arm64/kvm/hyp/switch.c index d8aae4b5f089..c892fdfecf56 100644 --- a/arch/arm64/kvm/hyp/switch.c +++ b/arch/arm64/kvm/hyp/switch.c @@ -539,6 +539,38 @@ static bool __hyp_text __hyp_handle_eret(struct kvm_vcpu *vcpu) return true; } +static bool __hyp_text __hyp_handle_tlbi_el1(struct kvm_vcpu *vcpu) +{ + u32 instr; + u64 val; + + /* + * Ideally, we would never trap on EL1 TLB invalidations when the + * guest's HCR_EL2.{E2H,TGE} == {1,1}. But "thanks" to ARMv8.4, we + * don't trap writes to HCR_EL2, meaning that we can't track + * changes to the virtual TGE bit. So we leave HCR_EL2.TTLB set on + * the host. Oopsie... + * + * In order to speed-up EL1 TLBIs from the vEL2 guest when TGE is + * set, try and handle these invalidation as quickly as possible, + * without fully exiting (unless this needs forwarding). + */ + if (kvm_vcpu_trap_get_class(vcpu) != ESR_ELx_EC_SYS64 || + !vcpu_mode_el2(vcpu) || + (__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) + return false; + + instr = esr_sys64_to_sysreg(kvm_vcpu_get_hsr(vcpu)); + if (sys_reg_Op0(instr) != TLBI_Op0 || + sys_reg_Op1(instr) != TLBI_Op1_EL1) + return false; + + val = vcpu_get_reg(vcpu, kvm_vcpu_sys_get_rt(vcpu)); + __kvm_tlb_el1_instr(NULL, val, instr); + __kvm_skip_instr(vcpu); + + return true; +} static bool __hyp_text handle_tx2_tvm(struct kvm_vcpu *vcpu) { @@ -669,6 +701,9 @@ static bool __hyp_text fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code, if (__hyp_handle_eret(vcpu)) return true; + if (__hyp_handle_tlbi_el1(vcpu)) + return true; + if (!__populate_fault_info(vcpu)) return true; diff --git a/arch/arm64/kvm/hyp/tlb.c b/arch/arm64/kvm/hyp/tlb.c index 71e4e86e0981..ef52c5a58866 100644 --- a/arch/arm64/kvm/hyp/tlb.c +++ b/arch/arm64/kvm/hyp/tlb.c @@ -278,7 +278,8 @@ void __hyp_text __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_enc dsb(ishst); /* Switch to requested VMID */ - __tlb_switch_to_guest(mmu, &cxt); + if (mmu) + __tlb_switch_to_guest(mmu, &cxt); /* * Execute the same instruction as the guest hypervisor did, @@ -317,5 +318,6 @@ void __hyp_text __kvm_tlb_el1_instr(struct kvm_s2_mmu *mmu, u64 val, u64 sys_enc dsb(ish); isb(); - __tlb_switch_to_host(mmu, &cxt); + if (mmu) + __tlb_switch_to_host(mmu, &cxt); } diff --git a/arch/arm64/kvm/sys_regs.c b/arch/arm64/kvm/sys_regs.c index f20f5975633f..c464bc3d8dad 100644 --- a/arch/arm64/kvm/sys_regs.c +++ b/arch/arm64/kvm/sys_regs.c @@ -2688,6 +2688,8 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, const struct sys_reg_desc *r) { u32 sys_encoding = sys_insn(p->Op0, p->Op1, p->CRn, p->CRm, p->Op2); + u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); + struct kvm_s2_mmu *mmu; /* * If we're here, this is because we've trapped on a EL1 TLBI @@ -2706,27 +2708,15 @@ static bool handle_tlbi_el1(struct kvm_vcpu *vcpu, struct sys_reg_params *p, mutex_lock(&vcpu->kvm->lock); - if ((__vcpu_sys_reg(vcpu, HCR_EL2) & (HCR_E2H | HCR_TGE)) != (HCR_E2H | HCR_TGE)) { - u64 virtual_vttbr = vcpu_read_sys_reg(vcpu, VTTBR_EL2); - struct kvm_s2_mmu *mmu; - - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); - if (mmu) - kvm_call_hyp(__kvm_tlb_el1_instr, - mmu, p->regval, sys_encoding); + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, HCR_VM); + if (mmu) + kvm_call_hyp(__kvm_tlb_el1_instr, + mmu, p->regval, sys_encoding); - mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); - if (mmu) - kvm_call_hyp(__kvm_tlb_el1_instr, - mmu, p->regval, sys_encoding); - } else { - /* - * ARMv8.4-NV allows the guest to change TGE behind - * our back, so we always trap EL1 TLBIs from vEL2... - */ + mmu = lookup_s2_mmu(vcpu->kvm, virtual_vttbr, 0); + if (mmu) kvm_call_hyp(__kvm_tlb_el1_instr, - &vcpu->kvm->arch.mmu, p->regval, sys_encoding); - } + mmu, p->regval, sys_encoding); mutex_unlock(&vcpu->kvm->lock);