From patchwork Mon Feb 19 09:20:02 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562404 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5651022F03; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; cv=none; b=iWF8fyt0Sp19AYLOLh+z+XSdWMVEw5gRzwRLuN9A5Uw+uoJKts9UmlJOLippKOf+EKkDgJzZJHgpArAiUSXvxx+jsrWOURrUxvd54fzBokH+Ch/nQvwDAX7x8DKOv3547h8ZDfCtD0sol9hZv3rkLRTq6TRoLpOIgnCdX83hQtA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; c=relaxed/simple; bh=KX/oVgZNALTsrMNLjBJoW2oz/1yIUZB9MBfcJb7mNbQ=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=urccuAhBJ6sRMPLdL1eg4WetZtczP20t7WnH6QiSsTCvm6mVkwFOChBlFZEaaLZIK6KEa3PIk7xPBOkFYVN0zMahIivwbgjY+A85Iq6pwHh56oc3IT8s6Xk1wVzRtR60m6yd/O0wgtxJ0UUjpzOorfS9LJW/8FYwwxw6pqx8/Eo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=a1m0F4hi; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="a1m0F4hi" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 02A69C43390; Mon, 19 Feb 2024 09:20:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334424; bh=KX/oVgZNALTsrMNLjBJoW2oz/1yIUZB9MBfcJb7mNbQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a1m0F4hiVbx60dhNCFW8c2yNdImKotiMvh6LmS9qtwB936taQ68HNw8O97prLYBiQ uhcKz2iLT67C+vAsLRIeV31JLgMK3FHNGzvTMrzZ7hb8YsbAWaW0sMArdv4QT/BKXE 79LMyUX/BZnrPrEH0rDznlWEGq38MA+VPs/IDOjwwGGUYt4EpV2Rw7KdopX+dNlYkw ku3JDoa9Y1vZ80u5YbEev7d4k8JfJAExN6Eaqef/H6Xvak+3oHx3Gu+TdMLs0LXk2N PJ6iBt8n9lkCXW1EuDibUMcOlnwD0HQIoGgYbo3CMoFJsT9BbB2eMSrb2Pbqd4lMnl 6Zihn4Hi53CIQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzoz-004WBZ-Jb; Mon, 19 Feb 2024 09:20:21 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 01/13] KVM: arm64: Harden __ctxt_sys_reg() against out-of-range values Date: Mon, 19 Feb 2024 09:20:02 +0000 Message-Id: <20240219092014.783809-2-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false The unsuspecting kernel tinkerer can be easily confused into writing something that looks like this: ikey.lo = __vcpu_sys_reg(vcpu, SYS_APIAKEYLO_EL1); which seems vaguely sensible, until you realise that the second parameter is the encoding of a sysreg, and not the index into the vcpu sysreg file... Debugging what happens in this case is an interesting exercise in head<->wall interactions. As they often say: "Any resemblance to actual persons, living or dead, or actual events is purely coincidental". In order to save people's time, add some compile-time hardening that will at least weed out the "stupidly out of range" values. This will *not* catch anything that isn't a compile-time constant. Signed-off-by: Marc Zyngier Reviewed-by: Joey Gouly --- arch/arm64/include/asm/kvm_host.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 181fef12e8e8..a5ec4c7d3966 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -895,7 +895,7 @@ struct kvm_vcpu_arch { * Don't bother with VNCR-based accesses in the nVHE code, it has no * business dealing with NV. */ -static inline u64 *__ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) +static inline u64 *___ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) { #if !defined (__KVM_NVHE_HYPERVISOR__) if (unlikely(cpus_have_final_cap(ARM64_HAS_NESTED_VIRT) && @@ -905,6 +905,13 @@ static inline u64 *__ctxt_sys_reg(const struct kvm_cpu_context *ctxt, int r) return (u64 *)&ctxt->sys_regs[r]; } +#define __ctxt_sys_reg(c,r) \ + ({ \ + BUILD_BUG_ON(__builtin_constant_p(r) && \ + (r) >= NR_SYS_REGS); \ + ___ctxt_sys_reg(c, r); \ + }) + #define ctxt_sys_reg(c,r) (*__ctxt_sys_reg(c,r)) u64 kvm_vcpu_sanitise_vncr_reg(const struct kvm_vcpu *, enum vcpu_sysreg); From patchwork Mon Feb 19 09:20:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562405 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5654822F0E; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; cv=none; b=EMzr72wQZIzjxXNb37Zk8Q7uYCZBCiwZoyUIBcmqE16oSM1mJFSPa8ruqkvy3ZzTgYb1wpgAwWumMIYsJTHwi7LtenFn2QrVrp/VRxSHNQxPIBbDiwo9IoCBsbgKD71J1Ixa1LD1kR7f2uSU9jQjoWAoBOd7etAORHibEqhPMG0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; c=relaxed/simple; bh=o1Ph3OO1c4EdF4VuyObf2uVzB4bNnlJcJufbMYmIJXA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=mBcNx1iivMIw5n98sYBEHiTFnM0/tKF/JgYx1d0KsJUhNJMolHff0vOL/D003fPaajDmXxRvfAEaoR4CSwzFdD3RKDmYmg8AfppEE7FaZOfkSB94sfNsX3BG98iHjSfBOY1zf1tH/ujdwr1WNQaOEDFiP8Uxjm6i9SBgZgWQ+38= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rmMmWLNn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rmMmWLNn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DE317C43394; Mon, 19 Feb 2024 09:20:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334423; bh=o1Ph3OO1c4EdF4VuyObf2uVzB4bNnlJcJufbMYmIJXA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rmMmWLNnjtkQ+2UxibAZipLUgaukJLI7EIA74lJdrMxEGlnZ93hie/shHjn0e4knL fQxU0qv3r/MT66AVw4gL+xSXbX4tx0YYA8Udscr0GkaGsISoVfVlRVvHQQBTcko2+6 qKxYxUcMqfbypa6m5MTU1VvK25a6TuKyVubQ0efiqMaESRhfhyRcXEOAwQi/ZYG5Zt +DTQUag20T6wENZobhbiwa1Grq0j+zd4ea7zW8YwFgEcrJmceNiOpcMiUiKDrPIjhV xUVQ9Q/VDd9jl5FUZd2XWHdiyetUdwUBY32/TJ4lFl+/YonqUmk1wSr8jmvgdmF8hv 1ODKvzQjFnUwg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzoz-004WBZ-SP; Mon, 19 Feb 2024 09:20:21 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 02/13] KVM: arm64: Clarify ESR_ELx_ERET_ISS_ERET* Date: Mon, 19 Feb 2024 09:20:03 +0000 Message-Id: <20240219092014.783809-3-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false The ESR_ELx_ERET_ISS_ERET* macros are a bit confusing: - ESR_ELx_ERET_ISS_ERET really indicates that we have trapped an ERETA* instruction, as opposed to an ERET - ESR_ELx_ERET_ISS_ERETA reallu indicates that we have trapped an ERETAB instruction, as opposed to an ERETAA. Repaint the two helpers such as: - ESR_ELx_ERET_ISS_ERET becomes ESR_ELx_ERET_ISS_ERETA - ESR_ELx_ERET_ISS_ERETA becomes ESR_ELx_ERET_ISS_ERETAB At the same time, use BIT() instead of raw values. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/esr.h | 4 ++-- arch/arm64/kvm/handle_exit.c | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 353fe08546cf..72c7810ccf2c 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -290,8 +290,8 @@ ESR_ELx_SYS64_ISS_OP2_SHIFT)) /* ISS field definitions for ERET/ERETAA/ERETAB trapping */ -#define ESR_ELx_ERET_ISS_ERET 0x2 -#define ESR_ELx_ERET_ISS_ERETA 0x1 +#define ESR_ELx_ERET_ISS_ERETA BIT(1) +#define ESR_ELx_ERET_ISS_ERETAB BIT(0) /* * ISS field definitions for floating-point exception traps diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 617ae6dea5d5..0646c623d1da 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -219,7 +219,7 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu) static int kvm_handle_eret(struct kvm_vcpu *vcpu) { - if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERET) + if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERETA) return kvm_handle_ptrauth(vcpu); /* From patchwork Mon Feb 19 09:20:04 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562403 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 564D922F02; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; cv=none; b=ma90FXyCFnTbxb/kAdBohNO4S/MrKsfkz3pJPZeYfiwnMzijHjUtaWCc5CvLaff7Q4NAP0v8m6N54CfR+NQYaYUq1Hri4gMjj1XlAfsDhYv+ZBY34vxKl0w4RXC84Ej/H4BvcRq0GVnfjDWIeid5sBtTOtOBKL51OeeUJsRWKAw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; c=relaxed/simple; bh=vgRLwq7Xgg3mkOBLCbxDTHOAAGNZox10P/aUL125qb4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=PG3B630JU7LKSwpj5Ag/mRvPNJK8ufL+tUlsZFSuPZaU0SfeyPYBOXY6qfLCiIpV+G7v0/bRBt7RFgtfKlNR/fCOYfuoMAhiXfL+m2u0dnJiMbyBXyrkm9mqN3Vov4Sj46Aqt9+3K2hhVoPjiTos/bGJ7r3T9qHWdXGje56/1lM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=rOAU80Ko; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="rOAU80Ko" Received: by smtp.kernel.org (Postfix) with ESMTPSA id EAAB4C43399; Mon, 19 Feb 2024 09:20:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334424; bh=vgRLwq7Xgg3mkOBLCbxDTHOAAGNZox10P/aUL125qb4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=rOAU80Kops03hFfKf2Qg0pH6J12Cgt64kR7juF/o0W+hjzDrqaPgy/R/SPjXnJFhD cghurczpI28Hwv/pTRxM6jRqjmvr3h8ocdNkQw+xHGgd4gGoAYM1Ssm0wk/TTLPHba apQwAkgBrm4E06nhVyDrsQzikEHCVzvfKXXBXMYhL5TaA7xCcZH+hEbYBewkRg38w+ +943CVkdGvmSFLxCIIIP1vbjM2jHy+p6//WGu4tdV3VhRd6oIyXSFvuQ3i8c+IAR2f iUQrFky1iDNHx9Gdn1XDG5PFsoc8JOmuMrJGlUrfRVOXFr+swtPmYa4ZgSkRFeNE45 Ab32fs76aSWlg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp0-004WBZ-1r; Mon, 19 Feb 2024 09:20:22 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 03/13] KVM: arm64: nv: Drop VCPU_HYP_CONTEXT flag Date: Mon, 19 Feb 2024 09:20:04 +0000 Message-Id: <20240219092014.783809-4-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false It has become obvious that HCR_EL2.NV serves the exact same use as VCPU_HYP_CONTEXT, only in an architectural way. So just drop the flag for good. Signed-off-by: Marc Zyngier Reviewed-by: Joey Gouly --- arch/arm64/include/asm/kvm_host.h | 2 -- arch/arm64/kvm/hyp/vhe/switch.c | 7 +------ 2 files changed, 1 insertion(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index a5ec4c7d3966..75eb8e170515 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -816,8 +816,6 @@ struct kvm_vcpu_arch { #define DEBUG_STATE_SAVE_SPE __vcpu_single_flag(iflags, BIT(5)) /* Save TRBE context if active */ #define DEBUG_STATE_SAVE_TRBE __vcpu_single_flag(iflags, BIT(6)) -/* vcpu running in HYP context */ -#define VCPU_HYP_CONTEXT __vcpu_single_flag(iflags, BIT(7)) /* SVE enabled for host EL0 */ #define HOST_SVE_ENABLED __vcpu_single_flag(sflags, BIT(0)) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 1581df6aec87..58415783fd53 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -197,7 +197,7 @@ static void early_exit_filter(struct kvm_vcpu *vcpu, u64 *exit_code) * If we were in HYP context on entry, adjust the PSTATE view * so that the usual helpers work correctly. */ - if (unlikely(vcpu_get_flag(vcpu, VCPU_HYP_CONTEXT))) { + if (unlikely(read_sysreg(hcr_el2) & HCR_NV)) { u64 mode = *vcpu_cpsr(vcpu) & (PSR_MODE_MASK | PSR_MODE32_BIT); switch (mode) { @@ -240,11 +240,6 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) sysreg_restore_guest_state_vhe(guest_ctxt); __debug_switch_to_guest(vcpu); - if (is_hyp_ctxt(vcpu)) - vcpu_set_flag(vcpu, VCPU_HYP_CONTEXT); - else - vcpu_clear_flag(vcpu, VCPU_HYP_CONTEXT); - do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu); From patchwork Mon Feb 19 09:20:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562406 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6E89822F11; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; cv=none; b=n0yEk/J7Og+BjViWhPNotrPhFTR++/oMfHYM/SkirOloqtIdtee305InQJ7T7DfbZokxc6Z7PhhGnL3wWyjbY3bAiI/0W4oQxkcwkP4j6oBJhdLmGo2WLLe12JyBWk/duQDjQ/hdJ7dg8mlvV776meXunpGnaGZxEytQe5Ef2EM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; c=relaxed/simple; bh=3lCs6fO4A9a0jYPyLc7yNONM1pQDZQ+vkshJ4dFCPGc=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=NiW8g6YBxHllEqz9hs/0Enmqo3Y2QvFWQZ6R/0mZUjXVEAubgdya2jeeCI4Gq79qoomX2CmFY3nm15kxhcthTnU9rQdaPlkHMCGwBLY2sOA/ksnqpn9sS+0PyGkMywYZrBYJrn7O4Y3hJfsT2VfvFm3ohNcW7tMYPTlx7ZJnJqY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=guVxLrj6; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="guVxLrj6" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1EA07C433B2; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334424; bh=3lCs6fO4A9a0jYPyLc7yNONM1pQDZQ+vkshJ4dFCPGc=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=guVxLrj67RzPAyqymw7tWCBrSaK9TH9XR3bBkifZXwLSj9/JREzB0sIV3mop+mWTY EpjRCTAb/cBZx7Mvs7s3+iGrHVPAj2FHDrQBQow4N9+RiThVoWcCprkBTcd7Zo4pA4 6d71a8wWwCWvUkMZms79GZkQvr/u6tV8Ak3yQe3spEXeqIDjZ7hgC/7hcRRTPicdh6 4zFdl7sSdtmFzSm1Hr4U2OPItpzNeuNJ97d82iOL1Y6V/refY7GMR6fC1PYiG9Gyc7 KCgnsXEuFsgB2k8WtX+btL69BqlZZo8BjwuGC9tZkgFdaQjgRJoAmcnnMoJkEHYTXH Z0JDUxfxPYraQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp0-004WBZ-7n; Mon, 19 Feb 2024 09:20:22 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 04/13] KVM: arm64: nv: Configure HCR_EL2 for FEAT_NV2 Date: Mon, 19 Feb 2024 09:20:05 +0000 Message-Id: <20240219092014.783809-5-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Add the HCR_EL2 configuration for FEAT_NV2, adding the required bits for running a guest hypervisor, and overall merging the allowed bits provided by the guest. This heavily replies on unavaliable features being sanitised when the HCR_EL2 shadow register is accessed, and only a couple of bits must be explicitly disabled. Non-NV guests are completely unaffected by any of this. Signed-off-by: Marc Zyngier Reviewed-by: Joey Gouly --- arch/arm64/include/asm/sysreg.h | 1 + arch/arm64/kvm/hyp/include/hyp/switch.h | 4 +-- arch/arm64/kvm/hyp/nvhe/switch.c | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 34 ++++++++++++++++++++++++- 4 files changed, 36 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/sysreg.h b/arch/arm64/include/asm/sysreg.h index 9e8999592f3a..a5361d9032a4 100644 --- a/arch/arm64/include/asm/sysreg.h +++ b/arch/arm64/include/asm/sysreg.h @@ -498,6 +498,7 @@ #define SYS_TCR_EL2 sys_reg(3, 4, 2, 0, 2) #define SYS_VTTBR_EL2 sys_reg(3, 4, 2, 1, 0) #define SYS_VTCR_EL2 sys_reg(3, 4, 2, 1, 2) +#define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0) #define SYS_TRFCR_EL2 sys_reg(3, 4, 1, 2, 1) #define SYS_VNCR_EL2 sys_reg(3, 4, 2, 2, 0) diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index e3fcf8c4d5b4..f5f701f309a9 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -271,10 +271,8 @@ static inline void __deactivate_traps_common(struct kvm_vcpu *vcpu) __deactivate_traps_hfgxtr(vcpu); } -static inline void ___activate_traps(struct kvm_vcpu *vcpu) +static inline void ___activate_traps(struct kvm_vcpu *vcpu, u64 hcr) { - u64 hcr = vcpu->arch.hcr_el2; - if (cpus_have_final_cap(ARM64_WORKAROUND_CAVIUM_TX2_219_TVM)) hcr |= HCR_TVM; diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index c50f8459e4fc..4103625e46c5 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -40,7 +40,7 @@ static void __activate_traps(struct kvm_vcpu *vcpu) { u64 val; - ___activate_traps(vcpu); + ___activate_traps(vcpu, vcpu->arch.hcr_el2); __activate_traps_common(vcpu); val = vcpu->arch.cptr_el2; diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 58415783fd53..29f59c374f7a 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -33,11 +33,43 @@ DEFINE_PER_CPU(struct kvm_host_data, kvm_host_data); DEFINE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); DEFINE_PER_CPU(unsigned long, kvm_hyp_vector); +/* + * HCR_EL2 bits that the NV guest can freely change (no RES0/RES1 + * semantics, irrespective of the configuration), but that cannot be + * applied to the actual HW as things would otherwise break badly. + * + * - TGE: we want to use EL1, which is incompatible with it being set + * + * - API/APK: for hysterical raisins, we enable PAuth lazily, which + * means that the guest's bits cannot be directly applied (we really + * want to see the traps). Revisit this at some point. + */ +#define NV_HCR_GUEST_EXCLUDE (HCR_TGE | HCR_API | HCR_APK) + +static u64 __compute_hcr(struct kvm_vcpu *vcpu) +{ + u64 hcr = vcpu->arch.hcr_el2; + + if (!vcpu_has_nv(vcpu)) + return hcr; + + if (is_hyp_ctxt(vcpu)) { + hcr |= HCR_NV | HCR_NV2 | HCR_AT | HCR_TTLB; + + if (!vcpu_el2_e2h_is_set(vcpu)) + hcr |= HCR_NV1; + + write_sysreg_s(vcpu->arch.ctxt.vncr_array, SYS_VNCR_EL2); + } + + return hcr | (__vcpu_sys_reg(vcpu, HCR_EL2) & ~NV_HCR_GUEST_EXCLUDE); +} + static void __activate_traps(struct kvm_vcpu *vcpu) { u64 val; - ___activate_traps(vcpu); + ___activate_traps(vcpu, __compute_hcr(vcpu)); if (has_cntpoff()) { struct timer_map map; From patchwork Mon Feb 19 09:20:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562407 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 84D9823746; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; cv=none; b=Xv4kz5XUaPb0O4qGA29AsTVHViAHe3V67877dN6+7qZqS9MVj3DENlBl6YGm0UwKRbe4nmup40uJvzMepVwGYeVazdJraXZNjhhBVYlKPkv5BE7ZiYJ02Y8gToigI6xfDR03WgmQ/GJdAgWa9+oVN7PGhKWf7CVNkLRtFpt33rM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; c=relaxed/simple; bh=sBCU4sUiSq3xPDA9YV2MrPVNgqWjyvQaOeV9pveINik=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=I5WcdaqKlGKk5erZlFAFpxOCCCY4zo0Oipazlon2qPirVYTDNSImEl+gW73Q1KocbbHq+B1qiaYTzlYW5dPDUdes3t5SCuJ2tTbiXf9ypKk/cGnv5++cD02TxbqmYkUdX1pWqEU8qn+rAchhjlsrRXUBFe6rVMdBST4DmwJuOL8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=gHP6sHos; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="gHP6sHos" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 59717C43330; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334424; bh=sBCU4sUiSq3xPDA9YV2MrPVNgqWjyvQaOeV9pveINik=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=gHP6sHos3qi5m/Adz27AeJehnp1YvIuCy9DdAjRPisyFjvNSwsXcKMTt5B3k0SVPD 8L8/K6MVWzBOVoF/PesCbHVJUMAd9LDqY+TtOf3baj6LRJwiApjdhthYU8s+ZqVqhA St90vosCa9W7kgsr2cNQGEYreZo16uj6CSoD8a2kiDLaFBESx7iCF3xyhr5U5XFOzx odzrdR4YhIz5JZ6njG1HJCkT3qhLNtxjZanP33bADHfPUVhTMHTTMj5h3Xiypa+nvG oa/tuHvOpUZMe8zSjYUxHsLLpscFoHJhRkmGy5BGAI6MWGpBQ+MtMDIdg+P7lB6l3W 7oKNIzhNy/Rcg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp0-004WBZ-DT; Mon, 19 Feb 2024 09:20:22 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 05/13] KVM: arm64: nv: Add trap forwarding for ERET and SMC Date: Mon, 19 Feb 2024 09:20:06 +0000 Message-Id: <20240219092014.783809-6-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Honor the trap forwarding bits for both ERET and SMC, using a new helper that checks for common conditions. Co-developed-by: Jintack Lim Signed-off-by: Jintack Lim Signed-off-by: Marc Zyngier Reviewed-by: Joey Gouly --- arch/arm64/include/asm/kvm_nested.h | 1 + arch/arm64/kvm/emulate-nested.c | 27 +++++++++++++++++++++++++++ arch/arm64/kvm/handle_exit.c | 7 +++++++ 3 files changed, 35 insertions(+) diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index c77d795556e1..dbc4e3a67356 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -60,6 +60,7 @@ static inline u64 translate_ttbr0_el2_to_ttbr0_el1(u64 ttbr0) return ttbr0 & ~GENMASK_ULL(63, 48); } +extern bool forward_smc_trap(struct kvm_vcpu *vcpu); int kvm_init_nv_sysregs(struct kvm *kvm); diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 4697ba41b3a9..2d80e81ae650 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -2117,6 +2117,26 @@ bool triage_sysreg_trap(struct kvm_vcpu *vcpu, int *sr_index) return true; } +static bool forward_traps(struct kvm_vcpu *vcpu, u64 control_bit) +{ + bool control_bit_set; + + if (!vcpu_has_nv(vcpu)) + return false; + + control_bit_set = __vcpu_sys_reg(vcpu, HCR_EL2) & control_bit; + if (!is_hyp_ctxt(vcpu) && control_bit_set) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return true; + } + return false; +} + +bool forward_smc_trap(struct kvm_vcpu *vcpu) +{ + return forward_traps(vcpu, HCR_TSC); +} + static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) { u64 mode = spsr & PSR_MODE_MASK; @@ -2155,6 +2175,13 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) u64 spsr, elr, mode; bool direct_eret; + /* + * Forward this trap to the virtual EL2 if the virtual + * HCR_EL2.NV bit is set and this is coming from !EL2. + */ + if (forward_traps(vcpu, HCR_NV)) + return; + /* * Going through the whole put/load motions is a waste of time * if this is a VHE guest hypervisor returning to its own diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 0646c623d1da..1ccdfe40c691 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -55,6 +55,13 @@ static int handle_hvc(struct kvm_vcpu *vcpu) static int handle_smc(struct kvm_vcpu *vcpu) { + /* + * Forward this trapped smc instruction to the virtual EL2 if + * the guest has asked for it. + */ + if (forward_smc_trap(vcpu)) + return 1; + /* * "If an SMC instruction executed at Non-secure EL1 is * trapped to EL2 because HCR_EL2.TSC is 1, the exception is a From patchwork Mon Feb 19 09:20:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562408 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B287123765; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; cv=none; b=Vg4vq+H7YqWlpZ9X73l0lsqD+r4ON2rTperag2D9+Pyd1GEelEIlLV2NZGRfV8b1Vfi6Z6SAuyoR6qhX9UX7V5pAE4TBDxVX8cv/fFxokU1/BH4lrTRk92AV1sUzXq0676W9TPfa1FFb0V45dwg6Eza7mdGMlvWNt8b58lFvjDs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334424; c=relaxed/simple; bh=gHuw6gM0kxaTDkdludLUngX+0zZfHKbDwECjHkZh6xs=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=ocZVrbWetWL1DnefTXE66Dv7ufkS0VmKuZlQT9kSvEMFIyV2JTPsLcXRQbuvRsaHhfDOQk48peOn2cj/3stKnMCrurgJ5vUt5aNgOuNEGhSQsCwJGZzWPe+u9gKeDc2J3H4y4vXMojl/7hieFf+gEd39AygBeX1TsJxvgQRJivo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=A8ok9AVu; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="A8ok9AVu" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 671B1C43141; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334424; bh=gHuw6gM0kxaTDkdludLUngX+0zZfHKbDwECjHkZh6xs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=A8ok9AVuLASRtCeECwevq3T7az8dloz4F58X2n4txtC+nzXPNCaGOB4+W9SQoRTj3 HqosrySHNBYwnlJA7ze6Vplik6S3GYFhpiw4qS1+X8FgP4kPpL7PG1rW5fOqTc6ca9 1VaFpzjWtPF4TcNeatQORbYYf0gBMSnNgr5dHK7dEmbycxSe3p14Mpj1VZJJKjN3I/ fCpWkiTCPkoBchr4+I/ge1+BGIscR9d60OrfPz/3+QitUML7CjnqNEA/3ZlEXz95d3 gXuPiT16HRLhrCNWW/TM7Peg4aMw6zdcaDYGN2jhU2aR3DXroKydy+RDhyyEbS/2ml hLqnQgg10uyCw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp0-004WBZ-J5; Mon, 19 Feb 2024 09:20:22 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 06/13] KVM: arm64: nv: Fast-track 'InHost' exception returns Date: Mon, 19 Feb 2024 09:20:07 +0000 Message-Id: <20240219092014.783809-7-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false A significant part of the FEAT_NV extension is to trap ERET instructions so that the hypervisor gets a chance to switch from a vEL2 L1 guest to an EL1 L2 guest. But this also has the unfortunate consequence of trapping ERET in unsuspecting circumstances, such as staying at vEL2 (interrupt handling while being in the guest hypervisor), or returning to host userspace in the case of a VHE guest. Although we already make some effort to handle these ERET quicker by not doing the put/load dance, it is still way too far down the line for it to be efficient enough. For these cases, it would ideal to ERET directly, no question asked. Of course, we can't do that. But the next best thing is to do it as early as possible, in fixup_guest_exit(), much as we would handle FPSIMD exceptions. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/emulate-nested.c | 29 +++------------------- arch/arm64/kvm/hyp/vhe/switch.c | 44 +++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 26 deletions(-) diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 2d80e81ae650..63a74c0330f1 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -2172,8 +2172,7 @@ static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) { - u64 spsr, elr, mode; - bool direct_eret; + u64 spsr, elr; /* * Forward this trap to the virtual EL2 if the virtual @@ -2182,33 +2181,11 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) if (forward_traps(vcpu, HCR_NV)) return; - /* - * Going through the whole put/load motions is a waste of time - * if this is a VHE guest hypervisor returning to its own - * userspace, or the hypervisor performing a local exception - * return. No need to save/restore registers, no need to - * switch S2 MMU. Just do the canonical ERET. - */ - spsr = vcpu_read_sys_reg(vcpu, SPSR_EL2); - spsr = kvm_check_illegal_exception_return(vcpu, spsr); - - mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); - - direct_eret = (mode == PSR_MODE_EL0t && - vcpu_el2_e2h_is_set(vcpu) && - vcpu_el2_tge_is_set(vcpu)); - direct_eret |= (mode == PSR_MODE_EL2h || mode == PSR_MODE_EL2t); - - if (direct_eret) { - *vcpu_pc(vcpu) = vcpu_read_sys_reg(vcpu, ELR_EL2); - *vcpu_cpsr(vcpu) = spsr; - trace_kvm_nested_eret(vcpu, *vcpu_pc(vcpu), spsr); - return; - } - preempt_disable(); kvm_arch_vcpu_put(vcpu); + spsr = __vcpu_sys_reg(vcpu, SPSR_EL2); + spsr = kvm_check_illegal_exception_return(vcpu, spsr); elr = __vcpu_sys_reg(vcpu, ELR_EL2); trace_kvm_nested_eret(vcpu, elr, spsr); diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 29f59c374f7a..0c175516d114 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -205,6 +205,49 @@ void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu) __vcpu_put_switch_sysregs(vcpu); } +static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) +{ + u64 spsr, mode; + + /* + * Going through the whole put/load motions is a waste of time + * if this is a VHE guest hypervisor returning to its own + * userspace, or the hypervisor performing a local exception + * return. No need to save/restore registers, no need to + * switch S2 MMU. Just do the canonical ERET. + * + * Unless the trap has to be forwarded further down the line, + * of course... + */ + if (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) + return false; + + spsr = read_sysreg_el1(SYS_SPSR); + mode = spsr & (PSR_MODE_MASK | PSR_MODE32_BIT); + + switch (mode) { + case PSR_MODE_EL0t: + if (!(vcpu_el2_e2h_is_set(vcpu) && vcpu_el2_tge_is_set(vcpu))) + return false; + break; + case PSR_MODE_EL2t: + mode = PSR_MODE_EL1t; + break; + case PSR_MODE_EL2h: + mode = PSR_MODE_EL1h; + break; + default: + return false; + } + + spsr = (spsr & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; + + write_sysreg_el2(spsr, SYS_SPSR); + write_sysreg_el2(read_sysreg_el1(SYS_ELR), SYS_ELR); + + return true; +} + static const exit_handler_fn hyp_exit_handlers[] = { [0 ... ESR_ELx_EC_MAX] = NULL, [ESR_ELx_EC_CP15_32] = kvm_hyp_handle_cp15_32, @@ -215,6 +258,7 @@ static const exit_handler_fn hyp_exit_handlers[] = { [ESR_ELx_EC_DABT_LOW] = kvm_hyp_handle_dabt_low, [ESR_ELx_EC_WATCHPT_LOW] = kvm_hyp_handle_watchpt_low, [ESR_ELx_EC_PAC] = kvm_hyp_handle_ptrauth, + [ESR_ELx_EC_ERET] = kvm_hyp_handle_eret, [ESR_ELx_EC_MOPS] = kvm_hyp_handle_mops, }; From patchwork Mon Feb 19 09:20:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562415 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id E173D241E4; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; cv=none; b=vCmY7tn3uIPxBLlvpZcHup/ACBUJhUupKWhgq+uHbkKAzJVcdjFk5OiNgSM5ZrVuLcqJ9i2FCGM2qNa3jzNXItCOZHVR5vG9N1RotEUFY9Xq0K3NCuJdbTRyinnW5x4f7WOpJAuulfvAo0lRsm4Hasa9p3nZuYpBLdTrYQd8+Hw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; c=relaxed/simple; bh=4kAZSZwBw7KrpIGu7/Ra/W9TXJ5J7n34iChEeNy2XZI=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=jB6JIJDnUSW5LHYT3IZbGfSjbTz30SNpZO/W1LAcnt2nf/bt/q0VM5U1snCRPFxWXUwc7lGwKg7dpQgtQfGcSNwysqIt1yNmUyQ4XWBAlZ2LoYy/HmMWDkBPglpoT7PsJYp8QefVXOta0AI7NuPjtu6Mr3kxXNS/so3a06RuGEE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=tL0BZ/Fn; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="tL0BZ/Fn" Received: by smtp.kernel.org (Postfix) with ESMTPSA id BF517C433F1; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334424; bh=4kAZSZwBw7KrpIGu7/Ra/W9TXJ5J7n34iChEeNy2XZI=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=tL0BZ/FnfSxi9h9WPq2TR8n8MgilFmZVyGO/koI7YmnjPTAWdN9sBim2tF4cZ/Iq0 WACkUXskbbAS5TSP1NEnMfZA3JhUhMEw3RZjyUbXEDQOT7FWtmgeMYAXOwhkGwIlAL Tag+CRt2pbMJowxSAO7+5by7b/EbOL0OmVv8m4IpRAHoraOF9K0ToP3v9e0diGeMKO s9XaxTZUlqQ6o/JwgBZz0nvs5YCbemUeorVfmjt8lIR/26961Xe09AzSbTV8B85YC5 VZcaHUuqmMSMKwqye3yNQwKWBx5f3RliBs/yvq/Gwy2iI4RSf09MAwH3LOCalLOBCT t7E5dfaUydzjw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp0-004WBZ-Ot; Mon, 19 Feb 2024 09:20:22 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 07/13] KVM: arm64: nv: Honor HFGITR_EL2.ERET being set Date: Mon, 19 Feb 2024 09:20:08 +0000 Message-Id: <20240219092014.783809-8-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false If the L1 hypervisor decides to trap ERETs while running L2, make sure we don't try to emulate it, just like we wouldn't if it had its NV bit set. The exception will be reinjected from the core handler. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/hyp/vhe/switch.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 0c175516d114..a6c61d2ffc35 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -219,7 +219,8 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) * Unless the trap has to be forwarded further down the line, * of course... */ - if (__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) + if ((__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_NV) || + (__vcpu_sys_reg(vcpu, HFGITR_EL2) & HFGITR_EL2_ERET)) return false; spsr = read_sysreg_el1(SYS_SPSR); From patchwork Mon Feb 19 09:20:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562411 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 07A732420C; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; cv=none; b=Bf4XYA9TB/xkC26rb3Pob9eiR8dQBcExyICQ2k0Od4SMiwEt172tXD3fnSwhQ4J9ik7NAuY1z0xJkkih1YMoA/kLhkzgFFUFdHJEwN9uJ34li6JRYTMKI0HbMlaFUdx20LfOmDBJ6GATWqdbdcNcPE4oBaottvK523wls9aC2v4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; c=relaxed/simple; bh=8mefTCTGskrcvQkNMN6umnAmckW2kWR8W3YQ9pAOy6A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=XKO7ybSigyrhlTVg0s7K0EfYin6UF31u5a2duqmegWno2nM54yb4vhjnnofQHN9NRPbE4RkhFp5TiIVe9NjEfBZYQz/gHGM5bXD1Ybb2HUoatvLR93z4yaaFV+zykjTmrCBaaXYbP5QH9Q1S/ibEteK66DZfx/3VaKqenC01vNU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=KwmZChOS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="KwmZChOS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D8D53C433B1; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334424; bh=8mefTCTGskrcvQkNMN6umnAmckW2kWR8W3YQ9pAOy6A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KwmZChOSorNCUKmU35Kl15KCa/EtcaIybpy5x+xy5ixqtoqGYSsGpIbaX3IXiRRNA ekodWPbrQ/WLMSRdtP2hir2yZbaBvSzerY+G9k91XFCrl7/Wp5s2tpDs6w8NWgmUFg qekThJTZD+O0Kjc2WtKJj1Jhbxa2oUHT2llIeegDvluHfYPG4NKbd5MM/h/U4xyAKX w0rGNm72fH8lQA6OVwekof6ZIJGWZH+hZMWuJQx2FM1amoVw2M8PG2qWXAqU3jDljC aIIPXeaFrKORUDmaqlfxjy2b9SLn9lXPo0jTBI29gwYy4XWoH3yAzxS1wPfKLdWZEO VafRY0lQ6AGrg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp0-004WBZ-Uz; Mon, 19 Feb 2024 09:20:23 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 08/13] KVM: arm64: nv: Handle HCR_EL2.{API,APK} independantly Date: Mon, 19 Feb 2024 09:20:09 +0000 Message-Id: <20240219092014.783809-9-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Although KVM couples API and APK for simplicity, the architecture makes no such requirement, and the two can be independently set or cleared. Check for which of the two possible reasons we have trapped here, and if the corresponding L1 control bit isn't set, delegate the handling for forwarding. Otherwise, set this exact bit in HCR_EL2 and resume the guest. Of course, in the non-NV case, we keep setting both bits and be done with it. Note that the entry core already saves/restores the keys should any of the two control bits be set. This result in a bit of rework, and the removal of the (trivial) vcpu_ptrauth_enable() helper. Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_emulate.h | 5 ---- arch/arm64/kvm/hyp/include/hyp/switch.h | 32 +++++++++++++++++++++---- 2 files changed, 27 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h index debc3753d2ef..d2177bc77844 100644 --- a/arch/arm64/include/asm/kvm_emulate.h +++ b/arch/arm64/include/asm/kvm_emulate.h @@ -125,11 +125,6 @@ static inline void vcpu_set_wfx_traps(struct kvm_vcpu *vcpu) vcpu->arch.hcr_el2 |= HCR_TWI; } -static inline void vcpu_ptrauth_enable(struct kvm_vcpu *vcpu) -{ - vcpu->arch.hcr_el2 |= (HCR_API | HCR_APK); -} - static inline void vcpu_ptrauth_disable(struct kvm_vcpu *vcpu) { vcpu->arch.hcr_el2 &= ~(HCR_API | HCR_APK); diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index f5f701f309a9..a0908d7a8f56 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -480,11 +480,35 @@ DECLARE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); static bool kvm_hyp_handle_ptrauth(struct kvm_vcpu *vcpu, u64 *exit_code) { struct kvm_cpu_context *ctxt; - u64 val; + u64 enable = 0; if (!vcpu_has_ptrauth(vcpu)) return false; + /* + * NV requires us to handle API and APK independently, just in + * case the hypervisor is totally nuts. Please barf >here<. + */ + if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) { + switch (ESR_ELx_EC(kvm_vcpu_get_esr(vcpu))) { + case ESR_ELx_EC_PAC: + if (!(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_API)) + return false; + + enable |= HCR_API; + break; + + case ESR_ELx_EC_SYS64: + if (!(__vcpu_sys_reg(vcpu, HCR_EL2) & HCR_APK)) + return false; + + enable |= HCR_APK; + break; + } + } else { + enable = HCR_API | HCR_APK; + } + ctxt = this_cpu_ptr(&kvm_hyp_ctxt); __ptrauth_save_key(ctxt, APIA); __ptrauth_save_key(ctxt, APIB); @@ -492,11 +516,9 @@ static bool kvm_hyp_handle_ptrauth(struct kvm_vcpu *vcpu, u64 *exit_code) __ptrauth_save_key(ctxt, APDB); __ptrauth_save_key(ctxt, APGA); - vcpu_ptrauth_enable(vcpu); - val = read_sysreg(hcr_el2); - val |= (HCR_API | HCR_APK); - write_sysreg(val, hcr_el2); + vcpu->arch.hcr_el2 |= enable; + sysreg_clear_set(hcr_el2, 0, enable); return true; } From patchwork Mon Feb 19 09:20:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562409 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5C58724B39; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; cv=none; b=iuVfcsjnmoBPaEZXCPnXF1x4AGWiEG4RYoPgxtJk+nBzVl6IxPmzy8bqRM8ogsi5BqfUhoatgwhoNq3d4lA1w2n2r9aNj2ZaKrIjtKyi1Gqb9yiEzjCZfsMPp4RPraRMVMt4QvRx9c58E6JXeEDk/Mg+FQbxhK81id//hiFIn2s= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; c=relaxed/simple; bh=BXb8tAm/95vFkuq3D2jCOy3oGWf70BBXrMbsz1oxJZA=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=OpIUEdNvr8RmwvzjV1VjNXcYSFg5y64lZ42/ZzT+SFAeQf1XytJeDrq2L/MrwrdZgOpiGud6M8D6PB51I4UvYYqzgR6ieWLAO6p499TIGKLff7D/EtF2C2UWcD8Njmwuwnk5DsKZeyT3HCgEmVtp+OGBZSY3B/oUVfuB3BQQPFw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Q89F6BMS; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Q89F6BMS" Received: by smtp.kernel.org (Postfix) with ESMTPSA id E64B8C433C7; Mon, 19 Feb 2024 09:20:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334425; bh=BXb8tAm/95vFkuq3D2jCOy3oGWf70BBXrMbsz1oxJZA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Q89F6BMSPF9NqMX3WsGiaqpVAB7KuAYncda0xr3oHsn202hR2L14VOCLek7xUEJXE jOIvqFsgUWFGNTM7nVWRai+EWBEcPy5vL00ptNeFFmnoMu273eMSD/i1GGJaTRkYZB hD32hpeGpX6RbRCkL1fpqD6Xa1LXvHNduHKMdSYJaJXyDaLsZNO+IEiVCg67dVvZRu sBhlSWXdZeKdtNn0VsrO2bpO7OXgwyJRxYx9nUa51Q+SxlLDVLPCQ4V1r5bNDjyZmd GFN4sanp2XNEdS8TxEsU7eVCgYhl+oiQOdQO8mk6dhuXr5ybpxZzoS9uDrrCdKjH37 35N5dhn7zp+qg== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp1-004WBZ-5B; Mon, 19 Feb 2024 09:20:23 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 09/13] KVM: arm64: nv: Reinject PAC exceptions caused by HCR_EL2.API==0 Date: Mon, 19 Feb 2024 09:20:10 +0000 Message-Id: <20240219092014.783809-10-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false In order for a L1 hypervisor to correctly handle PAuth instructions, it must observe traps caused by a L1 PAuth instruction when HCR_EL2.API==0. Since we already handle the case for API==1 as a fixup, only the exception injection case needs to be handled. Rework the kvm_handle_ptrauth() callback to reinject the trap in this case. Note that APK==0 is already handled by the exising triage_sysreg_trap() helper. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/handle_exit.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 1ccdfe40c691..556af771a9e9 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -214,12 +214,34 @@ static int handle_sve(struct kvm_vcpu *vcpu) } /* - * Guest usage of a ptrauth instruction (which the guest EL1 did not turn into - * a NOP). If we get here, it is that we didn't fixup ptrauth on exit, and all - * that we can do is give the guest an UNDEF. + * Two possibilities to handle a trapping ptrauth instruction: + * + * - Guest usage of a ptrauth instruction (which the guest EL1 did not + * turn into a NOP). If we get here, it is that we didn't fixup + * ptrauth on exit, and all that we can do is give the guest an + * UNDEF (as the guest isn't supposed to use ptrauth without being + * told it could). + * + * - Running an L2 NV guest while L1 has left HCR_EL2.API==0, and for + * which we reinject the exception into L1. API==1 is handled as a + * fixup so the only way to get here is when API==0. + * + * Anything else is an emulation bug (hence the WARN_ON + UNDEF). */ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu) { + if (!vcpu_has_ptrauth(vcpu)) { + kvm_inject_undefined(vcpu); + return 1; + } + + if (vcpu_has_nv(vcpu) && !is_hyp_ctxt(vcpu)) { + kvm_inject_nested_sync(vcpu, kvm_vcpu_get_esr(vcpu)); + return 1; + } + + /* Really shouldn't be here! */ + WARN_ON_ONCE(1); kvm_inject_undefined(vcpu); return 1; } From patchwork Mon Feb 19 09:20:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562414 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5AD2824B34; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; cv=none; b=XuiEsbiEXrXvBqhafroa6apwNYjhJWOiYjRTz0zdqkVlQlpUbQJp91/qaY88TCZHznbSsJ7hbENQRKitBCcfhQ1V1VibaUDla6AWdSqhyLkgBSaXgmymNKeBXl9bXV4huJ/ziQZNLZ8ZPaXqrmNCPxh0FYWT5Hudm9xPA5OtiKQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; c=relaxed/simple; bh=63HmN5bCw4yV1qIue0vyjz/ZbDoxT4DvgQFZU7t3gk4=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=LinVRoqzdfl5Pn49BufVmyiV7EEIHrEWYQRdpryBzp00OfAs8YQPvi/CH3ot3JqkVRyj/F4w259Ir9Ii+EPZHA7jCDT1RGtbDN2lM0hyJEbFR7It+wASobmYzCKJJdD/eB9KA/sMcCDj17yTkT0ebxf4nv8CHVt6WE8sx+YIeSE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=Dxjk0cVz; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="Dxjk0cVz" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 357C2C43390; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334425; bh=63HmN5bCw4yV1qIue0vyjz/ZbDoxT4DvgQFZU7t3gk4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=Dxjk0cVz0HulajUHnGT6D+sHWDAhmbx36//4etWG2bCziQ6290kACEQiJTZzpxlHk bmXzKICBXa4BqZprEGfH1Dph4I9NwAFWV6KAqwwR196J+Hj85eVQp8aXJPC5kVV7CF jaWnFBLhXcZdMW0+l8V8yRu5GmfJMqvzE5Trsu6j7fwlyjR+sQOHGHvhsmwzQk6mIf cuZem1XUawDmLrO0q5PTqm5BbJE9hrHMCFVauTL7W2zijmMSM2pX90s78uir6Uge4X 4DWI4Y5p1ZwtMpPVBsLxvL1hkVS4ns+Fj3cHpRhlwVkobQrWVWsss3dQWaNqtboTmM oc01fHe2QKIPw== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp1-004WBZ-B5; Mon, 19 Feb 2024 09:20:23 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 10/13] KVM: arm64: nv: Add kvm_has_pauth() helper Date: Mon, 19 Feb 2024 09:20:11 +0000 Message-Id: <20240219092014.783809-11-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Pointer Authentication comes in many flavors, and a faithful emulation relies on correctly handling the flavour implemented by the HW. For this, provide a new kvm_has_pauth() that checks whether we expose to the guest a particular level of support. This checks across all 3 possible authentication algorithms (Q5, Q3 and IMPDEF). Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_host.h | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h index 75eb8e170515..a97b092b7064 100644 --- a/arch/arm64/include/asm/kvm_host.h +++ b/arch/arm64/include/asm/kvm_host.h @@ -1334,4 +1334,19 @@ bool kvm_arm_vcpu_stopped(struct kvm_vcpu *vcpu); (get_idreg_field((kvm), id, fld) >= expand_field_sign(id, fld, min) && \ get_idreg_field((kvm), id, fld) <= expand_field_sign(id, fld, max)) +/* Check for a given level of PAuth support */ +#define kvm_has_pauth(k, l) \ + ({ \ + bool pa, pi, pa3; \ + \ + pa = kvm_has_feat((k), ID_AA64ISAR1_EL1, APA, l); \ + pa &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPA, IMP); \ + pi = kvm_has_feat((k), ID_AA64ISAR1_EL1, API, l); \ + pi &= kvm_has_feat((k), ID_AA64ISAR1_EL1, GPI, IMP); \ + pa3 = kvm_has_feat((k), ID_AA64ISAR2_EL1, APA3, l); \ + pa3 &= kvm_has_feat((k), ID_AA64ISAR2_EL1, GPA3, IMP); \ + \ + (pa + pi + pa3) == 1; \ + }) + #endif /* __ARM64_KVM_HOST_H__ */ From patchwork Mon Feb 19 09:20:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562412 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8C50125108; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; cv=none; b=nSsgVDASZWBR5ehh0hMLEdIZJY15UrZ0AFBEVqq04ahQ13E9+pNF1Q2mIOQKA3trF3lUxT2XxhgpdJHaB+sBtd8w9zxJ34wdTWFTKwFmRu9KxY8dk7hQtfFzX2E402QLcFhf4Js7cL2YWT+YZNRVI6VphsYxphhKKDCuCBYpH1A= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; c=relaxed/simple; bh=6zV0ogqAlCaFC/EIA74ric9obU6wunvsN+b/ySfTf2A=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IrU1BgyIDWhyU19ROBo9JvMZDzgyUqY5TZQ7FqMGj+0lVu+wfrnKHSU6n3JyMbuM/65RwV5Et3VatYJGKTIXodVtGDw3g1iIe4eAo0mjypk7wr74mSFZzjEqGxDhH0leiJ0i2oaxLNuYd1dph0wJ0umAWXQv459qGJrcT1onPc0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=QjFVLerW; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="QjFVLerW" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4D8B7C4166C; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334425; bh=6zV0ogqAlCaFC/EIA74ric9obU6wunvsN+b/ySfTf2A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=QjFVLerWRsI0g+l0XFMtUHlg9AXNpSgMw7CaXTEJD1x44UWJJVPhHT4lbzb7hKWWr jvATm0dksC0O7pQr9lGTsvGHV3xdw6IPPW/yx9VJ/ORpiNw2so9rCMgcOGvsRnR8p9 gUd1xwbnxHOeok3yWmC065QXFtllPVcdnEM5/lUUK5WYKZwqvbUbAQXae1rESKCY04 qG4i7+wJ959r4sp7+Yj8//W2ERzV7KI8cL3z38Xt6X7mJIfcJiLpPmGSwLYA6zOIQi n12UQNmeR5N/U33xmJgoAq85fda3xY+ZS2LiF0ceGZfyJ+fPiWQvkB9bFw2LJ+TKQQ h8zFvy9DmgmOQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp1-004WBZ-H7; Mon, 19 Feb 2024 09:20:23 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 11/13] KVM: arm64: nv: Add emulation for ERETAx instructions Date: Mon, 19 Feb 2024 09:20:12 +0000 Message-Id: <20240219092014.783809-12-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false FEAT_NV has the interesting property of relying on ERET being trapped. An added complexity is that it also traps ERETAA and ERETAB, meaning that the Pointer Authentication aspect of these instruction must be emulated. Add an emulation of Pointer Authentication, limited to ERETAx (always using SP_EL2 as the modifier and ELR_EL2 as the pointer), using the Generic Authentication instructions. The emulation, however small, is placed in its own compilation unit so that it can be avoided if the configuration doesn't include it (or the toolchan in not up to the task). Signed-off-by: Marc Zyngier --- arch/arm64/include/asm/kvm_nested.h | 12 ++ arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/kvm/Makefile | 1 + arch/arm64/kvm/pauth.c | 196 +++++++++++++++++++++++++ 4 files changed, 210 insertions(+) create mode 100644 arch/arm64/kvm/pauth.c diff --git a/arch/arm64/include/asm/kvm_nested.h b/arch/arm64/include/asm/kvm_nested.h index dbc4e3a67356..5e0ab0596246 100644 --- a/arch/arm64/include/asm/kvm_nested.h +++ b/arch/arm64/include/asm/kvm_nested.h @@ -64,4 +64,16 @@ extern bool forward_smc_trap(struct kvm_vcpu *vcpu); int kvm_init_nv_sysregs(struct kvm *kvm); +#ifdef CONFIG_ARM64_PTR_AUTH +bool kvm_auth_eretax(struct kvm_vcpu *vcpu, u64 *elr); +#else +static inline bool kvm_auth_eretax(struct kvm_vcpu *vcpu, u64 *elr) +{ + /* We really should never execute this... */ + WARN_ON_ONCE(1); + *elr = 0xbad9acc0debadbad; + return false; +} +#endif + #endif /* __ARM64_KVM_NESTED_H */ diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index e4944d517c99..bb88e9ef6296 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -277,6 +277,7 @@ #define TCR_TBI1 (UL(1) << 38) #define TCR_HA (UL(1) << 39) #define TCR_HD (UL(1) << 40) +#define TCR_TBID0 (UL(1) << 51) #define TCR_TBID1 (UL(1) << 52) #define TCR_NFD0 (UL(1) << 53) #define TCR_NFD1 (UL(1) << 54) diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index c0c050e53157..04882b577575 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -23,6 +23,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ vgic/vgic-its.o vgic/vgic-debug.o kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o pmu.o +kvm-$(CONFIG_ARM64_PTR_AUTH) += pauth.o always-y := hyp_constants.h hyp-constants.s diff --git a/arch/arm64/kvm/pauth.c b/arch/arm64/kvm/pauth.c new file mode 100644 index 000000000000..33ef0a26eb35 --- /dev/null +++ b/arch/arm64/kvm/pauth.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 - Google LLC + * Author: Marc Zyngier + * + * Primitive PAuth emulation for ERETAA/ERETAB. + * + * This code assumes that is is run from EL2, and that it is part of + * the emulation of ERETAx for a guest hypervisor. That's a lot of + * baked-in assumptions and shortcuts. + * + * Do no reuse for anything else! + */ + +#include + +#include +#include + +static u64 compute_pac(struct kvm_vcpu *vcpu, u64 ptr, + struct ptrauth_key ikey) +{ + struct ptrauth_key gkey; + u64 mod, pac = 0; + + preempt_disable(); + + if (!vcpu_get_flag(vcpu, SYSREGS_ON_CPU)) + mod = __vcpu_sys_reg(vcpu, SP_EL2); + else + mod = read_sysreg(sp_el1); + + gkey.lo = read_sysreg_s(SYS_APGAKEYLO_EL1); + gkey.hi = read_sysreg_s(SYS_APGAKEYHI_EL1); + + __ptrauth_key_install_nosync(APGA, ikey); + isb(); + + asm volatile(ARM64_ASM_PREAMBLE ".arch_extension pauth\n" + "pacga %0, %1, %2" : "=r" (pac) : "r" (ptr), "r" (mod)); + isb(); + + __ptrauth_key_install_nosync(APGA, gkey); + + preempt_enable(); + + /* PAC in the top 32bits */ + return pac; +} + +static bool effective_tbi(struct kvm_vcpu *vcpu, bool bit55) +{ + u64 tcr = vcpu_read_sys_reg(vcpu, TCR_EL2); + bool tbi, tbid; + + /* + * Since we are authenticating an instruction address, we have + * to take TBID into account. If E2H==0, ignore VA[55], as + * TCR_EL2 only has a single TBI/TBID. If VA[55] was set in + * this case, this is likely a guest bug... + */ + if (!vcpu_el2_e2h_is_set(vcpu)) { + tbi = tcr & BIT(20); + tbid = tcr & BIT(29); + } else if (bit55) { + tbi = tcr & TCR_TBI1; + tbid = tcr & TCR_TBID1; + } else { + tbi = tcr & TCR_TBI0; + tbid = tcr & TCR_TBID0; + } + + return tbi && !tbid; +} + +static int compute_bottom_pac(struct kvm_vcpu *vcpu, bool bit55) +{ + static const int maxtxsz = 39; // Revisit these two values once + static const int mintxsz = 16; // (if) we support TTST/LVA/LVA2 + u64 tcr = vcpu_read_sys_reg(vcpu, TCR_EL2); + int txsz; + + if (!vcpu_el2_e2h_is_set(vcpu) || !bit55) + txsz = FIELD_GET(TCR_T0SZ_MASK, tcr); + else + txsz = FIELD_GET(TCR_T1SZ_MASK, tcr); + + return 64 - clamp(txsz, mintxsz, maxtxsz); +} + +static u64 compute_pac_mask(struct kvm_vcpu *vcpu, bool bit55) +{ + int bottom_pac; + u64 mask; + + bottom_pac = compute_bottom_pac(vcpu, bit55); + + mask = GENMASK(54, bottom_pac); + if (!effective_tbi(vcpu, bit55)) + mask |= GENMASK(63, 56); + + return mask; +} + +static u64 to_canonical_addr(struct kvm_vcpu *vcpu, u64 ptr, u64 mask) +{ + bool bit55 = !!(ptr & BIT(55)); + + if (bit55) + return ptr | mask; + + return ptr & ~mask; +} + +static u64 corrupt_addr(struct kvm_vcpu *vcpu, u64 ptr) +{ + bool bit55 = !!(ptr & BIT(55)); + u64 mask, error_code; + int shift; + + if (effective_tbi(vcpu, bit55)) { + mask = GENMASK(54, 53); + shift = 53; + } else { + mask = GENMASK(62, 61); + shift = 61; + } + + if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERETAB) + error_code = 2 << shift; + else + error_code = 1 << shift; + + ptr &= ~mask; + ptr |= error_code; + + return ptr; +} + +/* + * Authenticate an ERETAA/ERETAB instruction, returning true if the + * authentication succeeded and false otherwise. In all cases, *elr + * contains the VA to ERET to. Potential exception injection is left + * to the caller. + */ +bool kvm_auth_eretax(struct kvm_vcpu *vcpu, u64 *elr) +{ + u64 sctlr = vcpu_read_sys_reg(vcpu, SCTLR_EL2); + u64 esr = kvm_vcpu_get_esr(vcpu); + u64 ptr, cptr, pac, mask; + struct ptrauth_key ikey; + + *elr = ptr = vcpu_read_sys_reg(vcpu, ELR_EL2); + + /* We assume we're already in the context of an ERETAx */ + if (esr & ESR_ELx_ERET_ISS_ERETAB ) { + if (!(sctlr & SCTLR_EL1_EnIB)) + return true; + + ikey.lo = __vcpu_sys_reg(vcpu, APIBKEYLO_EL1); + ikey.hi = __vcpu_sys_reg(vcpu, APIBKEYHI_EL1); + } else { + if (!(sctlr & SCTLR_EL1_EnIA)) + return true; + + ikey.lo = __vcpu_sys_reg(vcpu, APIAKEYLO_EL1); + ikey.hi = __vcpu_sys_reg(vcpu, APIAKEYHI_EL1); + } + + mask = compute_pac_mask(vcpu, !!(ptr & BIT(55))); + cptr = to_canonical_addr(vcpu, ptr, mask); + + pac = compute_pac(vcpu, cptr, ikey); + + /* + * Slightly deviate from the pseudocode: if we have a PAC + * match with the signed pointer, then it must be good. + * Anything after this point is pure error handling. + */ + if ((pac & mask) == (ptr & mask)) { + *elr = cptr; + return true; + } + + /* + * Authentication failed, corrupt the canonical address if + * PAuth2 isn't implemented, or some XORing if it is. + */ + if (!kvm_has_pauth(vcpu->kvm, PAuth2)) + cptr = corrupt_addr(vcpu, cptr); + else + cptr = ptr ^ (pac & mask); + + *elr = cptr; + return false; +} From patchwork Mon Feb 19 09:20:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562413 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id BE64E2556D; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; cv=none; b=PZqzB4j1bw9vHq+fqp+lG/IB54ONz58d7uA+D4DAAsDLuLjKT+Owqy/2ZiNh1O/QMVIbSUIhh0NXngQn325XxfY9N1IkapACvUxE5gbxjSDRhxyaP7LCzd20z8vnPZ3PDHPm5wHZBjbsqWAZk+MGcHRBFpa+zrDiwgeNhgBXZj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; c=relaxed/simple; bh=OYPSocZuCqbnboXmvuNV+6G4knkRpU04HiJL92gq5RY=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=IzRMLmaZXsLpZ1wOL0YidTMuokdRk/BsCC/JzYvdwsK+3NQCXa0yBMUZc3svNIzoVQtsUrPr6tCOJO7/s3BzAFMw3TYfeRwtYDw4/FjC/LfM5NocGJTwC5u7kZl2uuR4/6JxsmlxAQBQDlWYoqxTmj8wC1Am0N30UjMxl+xNVJU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=SLf4iyGL; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="SLf4iyGL" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 96E71C433B1; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334425; bh=OYPSocZuCqbnboXmvuNV+6G4knkRpU04HiJL92gq5RY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=SLf4iyGLDiyhA4/8Ayj8pCGhvPqLy7pOK0WfryjZQLSwDsJgM5hs47r0AK/35i88m KnlQ6l0KckmfYYLQ6Jy+0wySlgCUMxUPMB3a87gG/8OXPYlDJp2wieIaGAb5GvbtGu nGM4Dem0GX1k3D5z3wEAvb04XScjzKqwpONBxL0KDYQzFxMbLEP+mGDUt5S3mp39wF vVaBpK9/o5U85ILq+PUrBL4U82HFxnd8kmsEhUGIYRpjhM6TSB1Q3Iva/lTdGSA0Em Tuago5QIn9aMV6kW3QsB2RlnnwX/qxywsZS/tvHQyCW3miyJw3f8kQEb8MKHakpapn +lZfDdwb1z/AQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp1-004WBZ-NZ; Mon, 19 Feb 2024 09:20:23 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 12/13] KVM: arm64: nv: Handle ERETA[AB] instructions Date: Mon, 19 Feb 2024 09:20:13 +0000 Message-Id: <20240219092014.783809-13-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Now that we have some emulation in place for ERETA[AB], we can plug it into the exception handling machinery. As for a bare ERET, an "easy" ERETAx instruction is processed as a fixup, while something that requires a translation regime transition or an exception delivery is left to the slow path. Signed-off-by: Marc Zyngier --- arch/arm64/kvm/emulate-nested.c | 22 ++++++++++++++++++++-- arch/arm64/kvm/handle_exit.c | 3 ++- arch/arm64/kvm/hyp/vhe/switch.c | 13 +++++++++++-- 3 files changed, 33 insertions(+), 5 deletions(-) diff --git a/arch/arm64/kvm/emulate-nested.c b/arch/arm64/kvm/emulate-nested.c index 63a74c0330f1..6fc3b7580b24 100644 --- a/arch/arm64/kvm/emulate-nested.c +++ b/arch/arm64/kvm/emulate-nested.c @@ -2172,7 +2172,7 @@ static u64 kvm_check_illegal_exception_return(struct kvm_vcpu *vcpu, u64 spsr) void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) { - u64 spsr, elr; + u64 spsr, elr, esr; /* * Forward this trap to the virtual EL2 if the virtual @@ -2181,12 +2181,30 @@ void kvm_emulate_nested_eret(struct kvm_vcpu *vcpu) if (forward_traps(vcpu, HCR_NV)) return; + /* Check for an ERETAx */ + esr = kvm_vcpu_get_esr(vcpu); + if ((esr & ESR_ELx_ERET_ISS_ERETA) && !kvm_auth_eretax(vcpu, &elr)) { + /* + * Oh no, ERETAx failed to authenticate. If we have + * FPACCOMBINE, deliver an exception right away. If we + * don't, then let the mangled ELR value trickle down the + * ERET handling, and the guest will have a little surprise. + */ + if (kvm_has_pauth(vcpu->kvm, FPACCOMBINE)) { + esr &= ESR_ELx_ERET_ISS_ERETAB; + esr |= FIELD_PREP(ESR_ELx_EC_MASK, ESR_ELx_EC_FPAC); + kvm_inject_nested_sync(vcpu, esr); + return; + } + } + preempt_disable(); kvm_arch_vcpu_put(vcpu); spsr = __vcpu_sys_reg(vcpu, SPSR_EL2); spsr = kvm_check_illegal_exception_return(vcpu, spsr); - elr = __vcpu_sys_reg(vcpu, ELR_EL2); + if (!(esr & ESR_ELx_ERET_ISS_ERETA)) + elr = __vcpu_sys_reg(vcpu, ELR_EL2); trace_kvm_nested_eret(vcpu, elr, spsr); diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 556af771a9e9..998838da7c32 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -248,7 +248,8 @@ static int kvm_handle_ptrauth(struct kvm_vcpu *vcpu) static int kvm_handle_eret(struct kvm_vcpu *vcpu) { - if (kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERETA) + if ((kvm_vcpu_get_esr(vcpu) & ESR_ELx_ERET_ISS_ERETA) && + !vcpu_has_ptrauth(vcpu)) return kvm_handle_ptrauth(vcpu); /* diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index a6c61d2ffc35..04592cd56e4b 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -207,7 +207,8 @@ void kvm_vcpu_put_vhe(struct kvm_vcpu *vcpu) static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) { - u64 spsr, mode; + u64 esr = kvm_vcpu_get_esr(vcpu); + u64 spsr, elr, mode; /* * Going through the whole put/load motions is a waste of time @@ -241,10 +242,18 @@ static bool kvm_hyp_handle_eret(struct kvm_vcpu *vcpu, u64 *exit_code) return false; } + /* If ERETAx fails, take the slow path */ + if (esr & ESR_ELx_ERET_ISS_ERETA) { + if (!(vcpu_has_ptrauth(vcpu) && kvm_auth_eretax(vcpu, &elr))) + return false; + } else { + elr = read_sysreg_el1(SYS_ELR); + } + spsr = (spsr & ~(PSR_MODE_MASK | PSR_MODE32_BIT)) | mode; write_sysreg_el2(spsr, SYS_SPSR); - write_sysreg_el2(read_sysreg_el1(SYS_ELR), SYS_ELR); + write_sysreg_el2(elr, SYS_ELR); return true; } From patchwork Mon Feb 19 09:20:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marc Zyngier X-Patchwork-Id: 13562410 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id CF0E725577; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; cv=none; b=CKUCuuL+8YqPZO4yx/gWnZ/ls+8ePfcCxCvouI0w0WS52SHctR8irKdl4NZ1US+9oT1DNScSeV+PjPDEzy9lOXvpTcLTcjKzBwyaZf0kgP5hfvfxUK6bOnYkjxi19szA86fWRmziWHPkZqos1eBMmfhBAem9FwFHa6AMO9xeIzc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1708334425; c=relaxed/simple; bh=vqA6TJBDRvY50fGHqcQfGVUDuzRuFzAfHui2TaG7QnU=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=J/kiUh2wqtw/y0s2S1oLRTbgVlf78AjFJkVcZ1B9uABG73d5UTUA8w/b6Yl8ynqPBldJQaf8iisKHej9Slrb6WIDXjGqr3bZFVWoAyGUw2xleBPfgHYlKDqwSUoIpgs8ipJTVD7KJGppPZrx0/2q1rw3e+EPY7Ajz5zwd7iIJIE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=JITNI7nT; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="JITNI7nT" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B0E67C433F1; Mon, 19 Feb 2024 09:20:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1708334425; bh=vqA6TJBDRvY50fGHqcQfGVUDuzRuFzAfHui2TaG7QnU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=JITNI7nToZ36zcDFmHdeJ/RSjkban3qSQ6zMKFUVhWemwfwSekomFisA3qyyfzY4D V9s9nbNVbqMQR9cwURhdzQar9WMNWIXLh5VxnDhzGk8a/mTrPK5h6zze9cVwph3e1D cS0sGixwb8QXvYAV4S59XMaQaCCKKjj+OatArSXjy1fgxtUZmL64omaEKbc2D66Cnl oD9T3JwzLf+kfgAbeLCY7m3EeneIDKOpvgAgzFvGEO0dMFzNEOxSecdEuIqCxbrqtZ Qky+q9Cf4uiVCncv/IZgqKezctHZVLNnX4KHeE82A+7RcqYS75gDdNPGxcXK1yRTlq efvRWJxDbYoHA== Received: from sofa.misterjones.org ([185.219.108.64] helo=valley-girl.lan) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1rbzp1-004WBZ-UH; Mon, 19 Feb 2024 09:20:23 +0000 From: Marc Zyngier To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: James Morse , Suzuki K Poulose , Oliver Upton , Zenghui Yu , Will Deacon , Catalin Marinas Subject: [PATCH 13/13] KVM: arm64: nv: Advertise support for PAuth Date: Mon, 19 Feb 2024 09:20:14 +0000 Message-Id: <20240219092014.783809-14-maz@kernel.org> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240219092014.783809-1-maz@kernel.org> References: <20240219092014.783809-1-maz@kernel.org> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, yuzenghui@huawei.com, will@kernel.org, catalin.marinas@arm.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Now that we (hopefully) correctly handle ERETAx, drop the masking of the PAuth feature (something that was not even complete, as APA3 and AGA3 were still exposed). Signed-off-by: Marc Zyngier --- arch/arm64/kvm/nested.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/arch/arm64/kvm/nested.c b/arch/arm64/kvm/nested.c index ced30c90521a..6813c7c7f00a 100644 --- a/arch/arm64/kvm/nested.c +++ b/arch/arm64/kvm/nested.c @@ -35,13 +35,9 @@ static u64 limit_nv_id_reg(u32 id, u64 val) break; case SYS_ID_AA64ISAR1_EL1: - /* Support everything but PtrAuth and Spec Invalidation */ + /* Support everything but Spec Invalidation */ val &= ~(GENMASK_ULL(63, 56) | - NV_FTR(ISAR1, SPECRES) | - NV_FTR(ISAR1, GPI) | - NV_FTR(ISAR1, GPA) | - NV_FTR(ISAR1, API) | - NV_FTR(ISAR1, APA)); + NV_FTR(ISAR1, SPECRES)); break; case SYS_ID_AA64PFR0_EL1: