From patchwork Fri May 10 11:26:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661408 Received: from mail-ej1-f74.google.com (mail-ej1-f74.google.com [209.85.218.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0F58C171E4B for ; Fri, 10 May 2024 11:27:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340443; cv=none; b=jWel/TngpdZT/1xvYdLhbqomKh5m0S5UAvx/+20ZNqLZc56xMAFumPg2N8n6lCBTbN2Tg29f4d6/NrjE7LUPxR5BS4CdRdkA/4rkw3UkdAbm7okcPig/72nY57b/Km/Pa3A6NKG12of45IMrxJnPR4vPRmi1LFyoaAsX18PGaZ8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340443; c=relaxed/simple; bh=JHJFWyJeQVH89FDkO+GfM1Lay/KIUWopPc58GA0ph10=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=DhoMz837/AbSZl/LnIbo2Cn4oOi4EYbC1Bf99npc94vsSha9F9hMJ0wnfIkECQY8OmKDU3XMQbOBhnNP5OR4C+hdKQRKyqEG3KrDTxXZMaf+N2MwbhUg5yX8q3cgiN+HXK6EezxkhsomnwSqjmp/JezftmsRduMP0yyU/NwJB0Q= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=XaGHEiBj; arc=none smtp.client-ip=209.85.218.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="XaGHEiBj" Received: by mail-ej1-f74.google.com with SMTP id a640c23a62f3a-a59a0d2280cso148854866b.1 for ; Fri, 10 May 2024 04:27:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340440; x=1715945240; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=IlOgeEx7Dj0f9uD0T276ThAzO+p8Gd1L0Dz8H/gEavQ=; b=XaGHEiBjGIjhM5gZiDh/OQmxy2s3zJj2YPNAWn6K8cEkBgxybE3htizU5wu3k+yUnS w4fczwX0HJB9fZyy+kuBOI3jXRRXhE6KbyhLHj3Nmibdmjr4CsLuIN9wtvJnnUIm6kR3 uxZN7PFOjG0FKEdLtEyXE0dGoPllqVQvWXMpJNSobBHcSFrs5SI/tRm4Z+WOZzuEPG5i V4RCHpCWp5vBbYkeUTudp9DyUWIANDE/8gM6H6Hnq7Yi19Stoefs7uoAhudY0Tg7celu RolMLLbhINvTPJ97WR1GiVZmRvCuvH4p0dzYnMbBIZkFvNI7V1TE3HOmsN+eo5e1c4xi y1eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340440; x=1715945240; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=IlOgeEx7Dj0f9uD0T276ThAzO+p8Gd1L0Dz8H/gEavQ=; b=X4M0XaF2iyG4GWG+Uwok53cmEk+p1BOrZN93Ai2v5AhsGNbku0Qk+C1YmdCPwrMO4R /6EUim8G57m9MbthBkFGACXfHyvFrFxbyflWUfBE/hjHvHfL2zDM91ByW09IzLx+HftK 8hk+dBJ7e1Z3w2tvWK7kJ/eE+XwcdTlP8xfnH6hdNfZuozPhSUPw2QTt40hzxgflKiSD dhnFrpMP6G34QH/hcpgkMxEW39vMt4liMZTmZJ7d1g9Nl4399A2nnXXr+8Aw0Xx/NrOG WxQ+TBrfW7Do8UV95UIV1JCF+iSvkJCLB0TljsuW1+sXvzaR6pJ2/pnRGIp5n/CknN16 aCdw== X-Forwarded-Encrypted: i=1; AJvYcCUzb5dVOkcggHdWxNNjA85HlqO9XOnkPjqqpz2V8/k5nZPrTtdMN9zYBbQ/DqLcwYXy0ojFrw3pD22zGzPWmC0Cb0VA X-Gm-Message-State: AOJu0Yz5yYaSw30i40lphJuSoRyFFPK1nDxYLicy5IU4gX/q5ZUGewsu AGctM7X7Wt4BYQfdsDE6c2UMf2gCMqm9TRtnD6CT5f9hgZW4mRAJrHcZyaqwDU4Tb4UWQpIhkA= = X-Google-Smtp-Source: AGHT+IGzC6wBegF/BY4InGWvNQC2wCX5XWro3nkGNqjvsssmipwEVTEODbeq170WB3/2uigSQJYtADatpQ== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a17:906:456:b0:a59:cfcb:4973 with SMTP id a640c23a62f3a-a5a2d553699mr225366b.3.1715340439853; Fri, 10 May 2024 04:27:19 -0700 (PDT) Date: Fri, 10 May 2024 12:26:30 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-2-ptosi@google.com> Subject: [PATCH v3 01/12] KVM: arm64: Fix clobbered ELR in sync abort/SError From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort When the hypervisor receives a SError or synchronous exception (EL2h) while running with the __kvm_hyp_vector and if ELR_EL2 doesn't point to an extable entry, it panics indirectly by overwriting ELR with the address of a panic handler in order for the asm routine it returns to to ERET into the handler. However, this clobbers ELR_EL2 for the handler itself. As a result, hyp_panic(), when retrieving what it believes to be the PC where the exception happened, actually ends up reading the address of the panic handler that called it! This results in an erroneous and confusing panic message where the source of any synchronous exception (e.g. BUG() or kCFI) appears to be __guest_exit_panic, making it hard to locate the actual BRK instruction. Therefore, store the original ELR_EL2 in the per-CPU kvm_hyp_ctxt and point the sysreg to a routine that first restores it to its previous value before running __guest_exit_panic. Fixes: 7db21530479f ("KVM: arm64: Restore hyp when panicking in guest context") Signed-off-by: Pierre-Clément Tosi --- arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kvm/hyp/entry.S | 9 +++++++++ arch/arm64/kvm/hyp/include/hyp/switch.h | 5 +++-- 3 files changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 81496083c041..27de1dddb0ab 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -128,6 +128,7 @@ int main(void) DEFINE(VCPU_FAULT_DISR, offsetof(struct kvm_vcpu, arch.fault.disr_el1)); DEFINE(VCPU_HCR_EL2, offsetof(struct kvm_vcpu, arch.hcr_el2)); DEFINE(CPU_USER_PT_REGS, offsetof(struct kvm_cpu_context, regs)); + DEFINE(CPU_ELR_EL2, offsetof(struct kvm_cpu_context, sys_regs[ELR_EL2])); DEFINE(CPU_RGSR_EL1, offsetof(struct kvm_cpu_context, sys_regs[RGSR_EL1])); DEFINE(CPU_GCR_EL1, offsetof(struct kvm_cpu_context, sys_regs[GCR_EL1])); DEFINE(CPU_APIAKEYLO_EL1, offsetof(struct kvm_cpu_context, sys_regs[APIAKEYLO_EL1])); diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index f3aa7738b477..bcaaf1a11b4e 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -83,6 +83,15 @@ alternative_else_nop_endif eret sb +SYM_INNER_LABEL(__guest_exit_restore_elr_and_panic, SYM_L_GLOBAL) + // x0-x29,lr: hyp regs + + stp x0, x1, [sp, #-16]! + adr_this_cpu x0, kvm_hyp_ctxt, x1 + ldr x0, [x0, #CPU_ELR_EL2] + msr elr_el2, x0 + ldp x0, x1, [sp], #16 + SYM_INNER_LABEL(__guest_exit_panic, SYM_L_GLOBAL) // x2-x29,lr: vcpu regs // vcpu x0-x1 on the stack diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index e3fcf8c4d5b4..19a7ca2c1277 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -753,7 +753,7 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) static inline void __kvm_unexpected_el2_exception(void) { - extern char __guest_exit_panic[]; + extern char __guest_exit_restore_elr_and_panic[]; unsigned long addr, fixup; struct kvm_exception_table_entry *entry, *end; unsigned long elr_el2 = read_sysreg(elr_el2); @@ -775,7 +775,8 @@ static inline void __kvm_unexpected_el2_exception(void) } /* Trigger a panic after restoring the hyp context. */ - write_sysreg(__guest_exit_panic, elr_el2); + this_cpu_ptr(&kvm_hyp_ctxt)->sys_regs[ELR_EL2] = elr_el2; + write_sysreg(__guest_exit_restore_elr_and_panic, elr_el2); } #endif /* __ARM64_KVM_HYP_SWITCH_H__ */ From patchwork Fri May 10 11:26:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661409 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C8502171E5A for ; Fri, 10 May 2024 11:27:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340445; cv=none; b=g8tdkQCn2AAN/xiTPJ1TWlXes8TipiKgKe7yoCSeHGilm5rzEGUCa7FLTKM8+Az9iGiOQD00jnJoI7OCJwzZr3O3RkcefXh+s15l33V+mA1T6wDCaK2VYThHNOfcA8axzwdxNCLCjbGAxpaWMvtzQ5gDm2dmb8RfopMPExJTkCo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340445; c=relaxed/simple; bh=A//g+5Ndpfg+WpD9P8dw9I0KPXkUbDOvYBE0SxR39LU=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=u7BZB6VBq3XC1uyystQBlfmw4S4gh7vfSyc2l4u9RSdtCkHV9DkuP0sUcR0u17XrS629y7Lwm8ctBgDwoYaTXv/1QqQoJrrNEVapFzrz3ybA3GcaNgV2uvpupaUHInH5f+2VodMKBt6gxdsVjCf9otYxsJx7EEQEQe2nexZxbi4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=4riQuE5Y; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="4riQuE5Y" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-de74a2635e2so3081724276.3 for ; Fri, 10 May 2024 04:27:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340443; x=1715945243; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=sbYsZ3117akP+PvJr9gYUoi/CIT+2oSEgeDr5jj+0nI=; b=4riQuE5YQ22hGQY2FTloVDoIIGm1YDyoy5QorcBWuvdfIpmm80lVJargn/IdVPJ8O8 ouVmoeCqXXYGAdrMAluF0FTfraCJtFBQDQRGEtziIz16aLREme9UUdmi239JA9bF67og aZA4+8VBhv94QWfnop4wbG7JBLTlEpSUTHQj2jy7BfLew5O//U6wAhBnHmf5O6G9JO0B ogBxqUQc6T1D+RDCzMA4TlAzDVxYPuWGbzCJRJD77wW2gDBepKTS2n2dot+t+JrTg1VV 7eJnmLItZJ2kcRupZYf8BdUmOtu39wtyk1Zj59Wa+AunJCUW9iW+97RON1ELrlbRvDAY F8Xg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340443; x=1715945243; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=sbYsZ3117akP+PvJr9gYUoi/CIT+2oSEgeDr5jj+0nI=; b=uWyZl4CoHVSrQvD5EC/Q6qvcL9vk3WXfGlThNOZNWWwfJ8XYNNAbb/rzoZLEKpOmz/ nw+6eGJhx131qNbmPAAHKlLwgLxH0JV7LJPb+65YaNiRtUUu9RkuZLr8UpGs1KZ090Mk Fm0uCyy6SobrjFAxaypzWfQiThDVU1PsnpdnIOxyHrq9M7cLNHZFY+AmTtrTZjpg3DAD vPMiIYZmkYK/R+I3REcYkXrehYVMZVjuIlmQAYaIDAnh2PQkgNrmhrdb3cnTlYbn6lDS tU+z2V3XOJrbQc//ATbFpph2qp+N8xdP1CSHsK4lLB2xrHQ1eXvbDYqAwjvvT0AX1J4e SdDQ== X-Forwarded-Encrypted: i=1; AJvYcCXNsbEvZILPNHUlObfm+bKFv9r9J+NlpUp60j0DGftbnnp/Y6lDJ9tNo4soTrVYgsvuBTs8F71fOwAf1v3+cW0rLdYk X-Gm-Message-State: AOJu0YxUxet5DzU+gtcIAi+zb1sIP02pZSo66S/yPWm7xM6bNp4WJ5Pe kwpNVdOob7NsdJxB+kFv5rihf/50FQJseI0Gka2LoQWaFYTvFSTpXIZXs63lKsXdcKgcWfFNtw= = X-Google-Smtp-Source: AGHT+IFdj/b7v+ntFzmw+lLCR1C20+GpqYEhAi+SWxria6LS62T7NLsO4dOK8hYMQ4bb7TazgmL1bjHlFQ== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a05:6902:100a:b0:de0:ecc6:4681 with SMTP id 3f1490d57ef6-dee4f30f764mr166230276.1.1715340442754; Fri, 10 May 2024 04:27:22 -0700 (PDT) Date: Fri, 10 May 2024 12:26:31 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-3-ptosi@google.com> Subject: [PATCH v3 02/12] KVM: arm64: Fix __pkvm_init_switch_pgd C signature From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort Update the function declaration to match the asm implementation. Fixes: f320bc742bc2 ("KVM: arm64: Prepare the creation of s1 mappings at EL2") Signed-off-by: Pierre-Clément Tosi Acked-by: Will Deacon --- arch/arm64/include/asm/kvm_hyp.h | 3 +-- arch/arm64/kvm/hyp/nvhe/setup.c | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 3e2a1ac0c9bb..96daf7cf6802 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -123,8 +123,7 @@ void __noreturn __hyp_do_panic(struct kvm_cpu_context *host_ctxt, u64 spsr, #endif #ifdef __KVM_NVHE_HYPERVISOR__ -void __pkvm_init_switch_pgd(phys_addr_t phys, unsigned long size, - phys_addr_t pgd, void *sp, void *cont_fn); +void __pkvm_init_switch_pgd(phys_addr_t params, void (*finalize_fn)(void)); int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, unsigned long *per_cpu_base, u32 hyp_va_bits); void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index bc58d1b515af..bcaeb0fafd2d 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -316,7 +316,7 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, { struct kvm_nvhe_init_params *params; void *virt = hyp_phys_to_virt(phys); - void (*fn)(phys_addr_t params_pa, void *finalize_fn_va); + typeof(__pkvm_init_switch_pgd) *fn; int ret; BUG_ON(kvm_check_pvm_sysreg_table()); From patchwork Fri May 10 11:26:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661410 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 46A63171E5A for ; Fri, 10 May 2024 11:27:26 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340448; cv=none; b=m/R4HSBwKPFsBc1mB8b59eQ1xE61C5i8F1nlGEvuLSheoJeYz+HBEE3e5iWm3m5aWgPg8Uy/DJb9mcIcTXx4BgorwoVf/li1weDhFusbOQJjMBDvIzNw4d9vqFlSCSzH5TuavMwrQrQFni3QKJgUl54nKkVBgThvzZezz8MKzcc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340448; c=relaxed/simple; bh=FqP+W3H7ODA8nGAmitghr0+uUjjfekAcblOIn5trRhI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=IIvniinl7WFBlZGZNlBHEgUlKOFigrdC/cpUKsMS1KJGpcVczkXlquStIW3P7dZvJHJrCkFCIZ/+5vkin6dpsq6isUFCW1gsiXQSyap/w7iIMd1ca3dvcrJdmNE1yMEDHrOYf0frsN7UaXXSIDnYZE9Mk+3qnhxP3zSfn1x4COY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=ptJj/9MW; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="ptJj/9MW" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dee5f035dd6so25284276.0 for ; Fri, 10 May 2024 04:27:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340445; x=1715945245; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=1X74EzIbLgwkSZ3RCE9vimfqgbYyFErrsdDhyQP2Whk=; b=ptJj/9MWTl1bhZl9YTkdXxv+XbF/7BTzo+fPgFAVXueFeR6jGsMhEuA13nyLkQznVC U3dzd22V9YdVxfgToHjvism0QptPsm8Rxva2iiwF8FGYqA2kjk7vdCP0UrcS88voWlG6 L80xz0XRJoUzhxm1VaBD5Gtt77xjw9KyiiKkXDmI1Gb/rz07gfJSUPiAMHTv50QKAqdK fqCmmD3xf81jd/GFve+XxujMDi25SjjXTHACaMmFikO/Xs2yKfKsji/XZcspt31GhYlV biVX6qkbaFs17n+0O01a3W7GsynE2A7e2Oc20JA9YrSiSMk1qTod63xo4sq1KkgAG00w aISw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340445; x=1715945245; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=1X74EzIbLgwkSZ3RCE9vimfqgbYyFErrsdDhyQP2Whk=; b=ek+OvSxrwiGRwFj+dxAw3o9AtYaNRCnsumefRt5Yh+CyWh/VvAerzwPwbtO7T0Ch2a 64tg1GO/g2rzEm0JaKUa0LQOsFRFqRo0cac5SmScah/4Xd9NZmvUV0RO70Qr5mysNjfA y2gJBUb6fSK8ZYr0tjhKfHIstciIlF2LYD8G14p6KwlA9TS682SfL1zlK6d8bM/an0St qyFA3ypRssiUeBtlpfWjAlQcKVjEilWgTAi7dheP5RV2EGqst7UrUp7irPeq9+Vxm0XO LezXNKPlVp0OPpJFvwlwgNbgl7ZuXqr9NSomMgDhFt6eLIApprYILYe5ZcVcyoz3p4rB sYFA== X-Forwarded-Encrypted: i=1; AJvYcCWzKkN4yx6hhUBNhDwlihnnTVPlAVMBcOFV5eYChL0RrHEGC7PgSA7aANYQulouAoRQRB03eyEKfY40rjAh9HN8ex+b X-Gm-Message-State: AOJu0YxRejN15V7bjLUWLDkFgDr48DsXl630f2z2mphmi64uuruiBhLs ja7aTk/kXv5a/BnNYu4vlCmsi4Vd0lAgScJZumvzPhwOxUhXDi6SLzdWrhF8eQ19uDE5cHMkEQ= = X-Google-Smtp-Source: AGHT+IE2vI8V+31DiouhAYXxHwv2C3egPY5JlN+7fyVhlwS8yXv78dpY0cirHvo9+5p4Wx/GqO7X9Ryw0g== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a05:6902:2b88:b0:dcd:3172:7265 with SMTP id 3f1490d57ef6-dee4f3036e5mr593281276.8.1715340445332; Fri, 10 May 2024 04:27:25 -0700 (PDT) Date: Fri, 10 May 2024 12:26:32 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-4-ptosi@google.com> Subject: [PATCH v3 03/12] KVM: arm64: Pass pointer to __pkvm_init_switch_pgd From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort Make the function take a VA pointer, instead of a phys_addr_t, to fully take advantage of the high-level C language and its type checker. Perform all accesses to the kvm_nvhe_init_params before disabling the MMU, removing the need to access it using physical addresses, which was the reason for taking a phys_addr_t. Signed-off-by: Pierre-Clément Tosi --- arch/arm64/include/asm/kvm_hyp.h | 3 ++- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 12 +++++++++--- arch/arm64/kvm/hyp/nvhe/setup.c | 4 +--- 3 files changed, 12 insertions(+), 7 deletions(-) diff --git a/arch/arm64/include/asm/kvm_hyp.h b/arch/arm64/include/asm/kvm_hyp.h index 96daf7cf6802..c195e71d0746 100644 --- a/arch/arm64/include/asm/kvm_hyp.h +++ b/arch/arm64/include/asm/kvm_hyp.h @@ -123,7 +123,8 @@ void __noreturn __hyp_do_panic(struct kvm_cpu_context *host_ctxt, u64 spsr, #endif #ifdef __KVM_NVHE_HYPERVISOR__ -void __pkvm_init_switch_pgd(phys_addr_t params, void (*finalize_fn)(void)); +void __pkvm_init_switch_pgd(struct kvm_nvhe_init_params *params, + void (*finalize_fn)(void)); int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, unsigned long *per_cpu_base, u32 hyp_va_bits); void __noreturn __host_enter(struct kvm_cpu_context *host_ctxt); diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index 2994878d68ea..5a15737b4233 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -265,7 +265,15 @@ alternative_else_nop_endif SYM_CODE_END(__kvm_handle_stub_hvc) +/* + * void __pkvm_init_switch_pgd(struct kvm_nvhe_init_params *params, + * void (*finalize_fn)(void)); + */ SYM_FUNC_START(__pkvm_init_switch_pgd) + /* Load the inputs from the VA pointer before turning the MMU off */ + ldr x5, [x0, #NVHE_INIT_PGD_PA] + ldr x0, [x0, #NVHE_INIT_STACK_HYP_VA] + /* Turn the MMU off */ pre_disable_mmu_workaround mrs x2, sctlr_el2 @@ -276,15 +284,13 @@ SYM_FUNC_START(__pkvm_init_switch_pgd) tlbi alle2 /* Install the new pgtables */ - ldr x3, [x0, #NVHE_INIT_PGD_PA] - phys_to_ttbr x4, x3 + phys_to_ttbr x4, x5 alternative_if ARM64_HAS_CNP orr x4, x4, #TTBR_CNP_BIT alternative_else_nop_endif msr ttbr0_el2, x4 /* Set the new stack pointer */ - ldr x0, [x0, #NVHE_INIT_STACK_HYP_VA] mov sp, x0 /* And turn the MMU back on! */ diff --git a/arch/arm64/kvm/hyp/nvhe/setup.c b/arch/arm64/kvm/hyp/nvhe/setup.c index bcaeb0fafd2d..45b83f3ed012 100644 --- a/arch/arm64/kvm/hyp/nvhe/setup.c +++ b/arch/arm64/kvm/hyp/nvhe/setup.c @@ -314,7 +314,6 @@ void __noreturn __pkvm_init_finalise(void) int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, unsigned long *per_cpu_base, u32 hyp_va_bits) { - struct kvm_nvhe_init_params *params; void *virt = hyp_phys_to_virt(phys); typeof(__pkvm_init_switch_pgd) *fn; int ret; @@ -338,9 +337,8 @@ int __pkvm_init(phys_addr_t phys, unsigned long size, unsigned long nr_cpus, update_nvhe_init_params(); /* Jump in the idmap page to switch to the new page-tables */ - params = this_cpu_ptr(&kvm_init_params); fn = (typeof(fn))__hyp_pa(__pkvm_init_switch_pgd); - fn(__hyp_pa(params), __pkvm_init_finalise); + fn(this_cpu_ptr(&kvm_init_params), __pkvm_init_finalise); unreachable(); } From patchwork Fri May 10 11:26:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661411 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 485CD171E70 for ; Fri, 10 May 2024 11:27:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340450; cv=none; b=f0sFpliGQGg2Mq8neWADkplm8X/O+TE1u+M990APJQqSn9pe1gI4DGY2kiy5v2XjU6zhTNsFoPEhlbovGe3THcXJ2496CVtez/LyJWwO7fNWc59wvkXD71QD1bQI1zpGqhHN6DAIoZ91r+DHlKX46c4DaKml9fj4rkFPYYz+9Fw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340450; c=relaxed/simple; bh=q2bvImUo/Ebwh+dSMu1W+96qgAvfZDWE4RJ8PPb+UbE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=hwRMoYUTcSebORFg/TL/qFnGtH3bFhBNN9FXl0pOXHA+/ytw+2RiKfyAFwvDNdLMnaaYERXXddjq1RCZKjuWRX1KjrNZEAAp46MpfhM1oBC2tqmif65kVvienS7xm0Qas6Llf+yH+lfPlVMErjpvXny83yDp+4Vad7iep8IEpDA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Jge/1wvZ; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Jge/1wvZ" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-a59e9ac4c74so101971866b.2 for ; Fri, 10 May 2024 04:27:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340448; x=1715945248; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=gLg0BCgwrJPFXcSMjQCXQoWFMEc6ibsqpEm8oPH+eqc=; b=Jge/1wvZCnNxxZ0uD2ydVXad/n39Rx3Gp6ReSpx/g6+Pe/gN+IylpEeMJMVviP0Mvn 4P6MuHGDBhGQ3P4QGMhQzVfbXovJkrZdmrRL19oxAL7pORwwv272v0mSsXRZb67Wx4us uCfOhRrMMvEst/dTpmeNxa0tMttprqQm7oJG8rD22FiVpr1S5nI3UZ9Rn+O8s8zuxcL/ sAgEr+Rg67fR4bVTeV4OkRon/4gbwzyEzbp18dcdUpSmdpPquofyJFyVOOZ/k1VsOelV 0OLDPWHBid2X3bWfZhBagTrdbhz82Vheq20zegVff/93NnE43cQuApaRSHbKbB+80QYE ymvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340448; x=1715945248; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=gLg0BCgwrJPFXcSMjQCXQoWFMEc6ibsqpEm8oPH+eqc=; b=asGY16vVOyMAQu3ozwxVlknCtqgyGo6HWdfcNuz6VVGQzGw1GGHp6C7S/erh1w8ANi Fm9yQa9Yi9X1+3CEkslPeLEMB7rcw8tj3J3pniTAwXEJ/tF0TNPy2IztOiv5Jz0VFDR+ DVZ2E2Guw0qOMOMuaFPhEESCgyZDDyt9mHDhaoFqTq/My5l8p8kdZ/uQpYnE2aD9UWs6 dAN1PaWXXiKgpihl2211evzZwTpVjZFdqPFtfaqka2mXVthlBBEaFSJRChhMNU537KQ2 +mbgShRpAh6QBNzebts3dt79OUN69Zf1cFVIJJJAlfSKWdaUmo/xJZwv+nmGdvms4fmx X29w== X-Forwarded-Encrypted: i=1; AJvYcCXMbADB+89bdQsMEm58W/eNU8hqggmSN+f+JhgfXhPcZy4k5k7gUOsY8KF5vp/8EJIFyH8Scf1XY2n22wX23yjjH7mR X-Gm-Message-State: AOJu0YxE5/ooMFZfMJwJ85AsURFx1Bf0/hlepnAflK8w+NQQGDjvu+CV 7zQubGNJmicYIsBqhRQWkOfedVAoxgF1KoHgzxSJlOihlA876RD38Fjm5P5kndzJTZh/2eZOrg= = X-Google-Smtp-Source: AGHT+IFepWD3/v8BdAdYkb3xCeH3t+IvJIqzRXw8IRofPX/9PW2hDEA+udTBjtBXHF9E/IajiV/5ddxCaw== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a17:907:12c9:b0:a59:d5f7:e56c with SMTP id a640c23a62f3a-a5a2d58a6f4mr225866b.5.1715340447570; Fri, 10 May 2024 04:27:27 -0700 (PDT) Date: Fri, 10 May 2024 12:26:33 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-5-ptosi@google.com> Subject: [PATCH v3 04/12] KVM: arm64: nVHE: Remove __guest_exit_panic path From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort In invalid_host_el2_vect (i.e. EL2{t,h} handlers in nVHE guest context), remove the duplicate vCPU context check that __guest_exit_panic also performs, allowing an unconditional branch to it. Rename __guest_exit_panic to __hyp_panic to better reflect that it might not exit through the guest but will always (directly or indirectly) end up executing hyp_panic(). Fix its wrong (probably bitrotten) ABI doc to reflect the ABI expected by VHE and (now) nVHE. Use CPU_LR_OFFSET to clarify that the routine returns to hyp_panic(). Restore x0, x1 before calling hyp_panic when __hyp_panic is executed in host context (i.e. called from __kvm_hyp_vector). Signed-off-by: Pierre-Clément Tosi --- arch/arm64/kvm/hyp/entry.S | 14 +++++++++----- arch/arm64/kvm/hyp/hyp-entry.S | 2 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 ++-- arch/arm64/kvm/hyp/nvhe/host.S | 8 +------- 4 files changed, 13 insertions(+), 15 deletions(-) diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index bcaaf1a11b4e..6a1ce9d21e5b 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -83,7 +83,7 @@ alternative_else_nop_endif eret sb -SYM_INNER_LABEL(__guest_exit_restore_elr_and_panic, SYM_L_GLOBAL) +SYM_INNER_LABEL(__hyp_restore_elr_and_panic, SYM_L_GLOBAL) // x0-x29,lr: hyp regs stp x0, x1, [sp, #-16]! @@ -92,13 +92,15 @@ SYM_INNER_LABEL(__guest_exit_restore_elr_and_panic, SYM_L_GLOBAL) msr elr_el2, x0 ldp x0, x1, [sp], #16 -SYM_INNER_LABEL(__guest_exit_panic, SYM_L_GLOBAL) - // x2-x29,lr: vcpu regs - // vcpu x0-x1 on the stack +SYM_INNER_LABEL(__hyp_panic, SYM_L_GLOBAL) + // x0-x29,lr: vcpu regs + + stp x0, x1, [sp, #-16]! // If the hyp context is loaded, go straight to hyp_panic get_loaded_vcpu x0, x1 cbnz x0, 1f + ldp x0, x1, [sp], #16 b hyp_panic 1: @@ -110,10 +112,12 @@ SYM_INNER_LABEL(__guest_exit_panic, SYM_L_GLOBAL) // accurate if the guest had been completely restored. adr_this_cpu x0, kvm_hyp_ctxt, x1 adr_l x1, hyp_panic - str x1, [x0, #CPU_XREG_OFFSET(30)] + str x1, [x0, #CPU_LR_OFFSET] get_vcpu_ptr x1, x0 + // Keep x0-x1 on the stack for __guest_exit + SYM_INNER_LABEL(__guest_exit, SYM_L_GLOBAL) // x0: return code // x1: vcpu diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S index 03f97d71984c..7e65ef738ec9 100644 --- a/arch/arm64/kvm/hyp/hyp-entry.S +++ b/arch/arm64/kvm/hyp/hyp-entry.S @@ -122,7 +122,7 @@ el2_error: eret sb -.macro invalid_vector label, target = __guest_exit_panic +.macro invalid_vector label, target = __hyp_panic .align 2 SYM_CODE_START_LOCAL(\label) b \target diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 19a7ca2c1277..9387e3a0b680 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -753,7 +753,7 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) static inline void __kvm_unexpected_el2_exception(void) { - extern char __guest_exit_restore_elr_and_panic[]; + extern char __hyp_restore_elr_and_panic[]; unsigned long addr, fixup; struct kvm_exception_table_entry *entry, *end; unsigned long elr_el2 = read_sysreg(elr_el2); @@ -776,7 +776,7 @@ static inline void __kvm_unexpected_el2_exception(void) /* Trigger a panic after restoring the hyp context. */ this_cpu_ptr(&kvm_hyp_ctxt)->sys_regs[ELR_EL2] = elr_el2; - write_sysreg(__guest_exit_restore_elr_and_panic, elr_el2); + write_sysreg(__hyp_restore_elr_and_panic, elr_el2); } #endif /* __ARM64_KVM_HYP_SWITCH_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 135cfb294ee5..7397b4f1838a 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -196,19 +196,13 @@ SYM_FUNC_END(__host_hvc) tbz x0, #PAGE_SHIFT, .L__hyp_sp_overflow\@ sub x0, sp, x0 // x0'' = sp' - x0' = (sp + x0) - sp = x0 sub sp, sp, x0 // sp'' = sp' - x0 = (sp + x0) - x0 = sp - /* If a guest is loaded, panic out of it. */ - stp x0, x1, [sp, #-16]! - get_loaded_vcpu x0, x1 - cbnz x0, __guest_exit_panic - add sp, sp, #16 - /* * The panic may not be clean if the exception is taken before the host * context has been saved by __host_exit or after the hyp context has * been partially clobbered by __host_enter. */ - b hyp_panic + b __hyp_panic .L__hyp_sp_overflow\@: /* Switch to the overflow stack */ From patchwork Fri May 10 11:26:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661412 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91EFD17556F for ; Fri, 10 May 2024 11:27:30 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340453; cv=none; b=hm7vSIXFH+vpf4EpX2FIajYHPnoOgu4g0e/TNOgdv3EI626KmuHcnfOIqup1szHckhS248m067/RRhCTzpQ9W7hwLWz+rZbXvJA/o5VvAOjiZ4whoATkpkKJpGY3cN7FLMac5iTVZt1Vd0W77l6jYRhUYiplr0sbM4imHfODNNE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340453; c=relaxed/simple; bh=RuQZrcKwOHj3TG5Qgns1mzsOfW6oRw1bJHQR20VMa7g=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=mPzqjgS58by34BrAjrjOaL080fRSB/gxQbHLGDIxlAJRHIFmfE1p5fN73iwJdtvvKq4X8RjTcP4/Rb97ZyKD8YAi47FBau6pJL8YXmR3+FINBkETKDNAP9/bfwOpOFAj7owOmsoml0tGFkPx2HKxEiIgjuS2ake+KD4R5jjYBn4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=Jnzy1UI7; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Jnzy1UI7" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-61c9675ae5aso30498807b3.0 for ; Fri, 10 May 2024 04:27:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340450; x=1715945250; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=qcEzQNh9dWuFIVjzg9t8TylWGTumh3MZGtqI9gQXfso=; b=Jnzy1UI7iv9I62lwksYuhu2v/nnXCrKXmtbzoUXJq9HQdxPJATzHZDpr2s9m9KdThQ ithu8kcOjQ7FxYSaBQtDW/oMmCOq3LeqMFJ1gAwJ7i80SFP2KU2KX3/K7VN91yucBJFI 0Xw7n34EXcmWm0fjmMReokZ4VQMb/EnooaCmR8+vkKVsPv4S8YGMVk6LT21h/hEFNsKS Xmd0cIe5L7Q97evxLQZAE7jLkrEKMiFBAdZyrXDSKPYRNGWKyTdH2zVjCfYtciad7sHo EM1owmxCn/v9beyx5nt5kOObIoIQvs/yHE84oF5beVY2JrYk3Agblg+YOpsk+O07eS9v TNWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340450; x=1715945250; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=qcEzQNh9dWuFIVjzg9t8TylWGTumh3MZGtqI9gQXfso=; b=dCqbTM45KHcMckuEX0IkX+KLH7sRtnrBueBKetlM30Rcr7iFJN3kJxbqjRpeTfIyyb cSDjjz+mw1OHhBezBhXoKveyLGuwGFz8btRENZzfJO4aXXgx19S96zuXz1dbUg9usAi5 SOjG7NRQuK3bQJ2ZIIxNrutbEM3lF/X07x8bD+JaXPl/7h1Hzi5EyOO1vP/oHTpugB2x 2/kNNrBAsqOFsHPhY/6hp619hNcA78WLZtPvcl7vHym/6y0PPDG5FGh4OC4GPYcElbuw OGwzJjYB1IsMEexWUYdpjEOhWrPZHMAvoAS7nGE474pjlDm2PmMPq7TX/gsGIeBjB6ZA LQwg== X-Forwarded-Encrypted: i=1; AJvYcCWhbl7xR5FooOjLt5LI2cJ73P3jGMPUZGKcxSy5LdCoCTpJOFvxVPbUyAQRRe4twLaW9f6yLjHAXcnCp7jn1IUeIruO X-Gm-Message-State: AOJu0YyJlUddVYYyjIOjWc6Xw7kkMTOu5Q+mGPVgm1yj4bdSGUh4k7Hi hcLQjRo/x/jzFSQYh6F09HKqdEkraOdMQZveoHDsQUUOF89x/wFEFdzqyaBdv2bUaPLw3uCSuQ= = X-Google-Smtp-Source: AGHT+IF00DsgRNr4wMWPoZLxFG4BzAtNHmn3WH+94RNNB3FGneRo+0N3ZZ2Yl4THjXTnNjJION1G71wXVw== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a05:690c:4507:b0:61b:3b02:6901 with SMTP id 00721157ae682-622b001dd60mr5562717b3.9.1715340450084; Fri, 10 May 2024 04:27:30 -0700 (PDT) Date: Fri, 10 May 2024 12:26:34 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-6-ptosi@google.com> Subject: [PATCH v3 05/12] KVM: arm64: nVHE: Add EL2h sync exception handler From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort Introduce a handler for EL2h synchronous exceptions distinct from handlers for other "invalid" exceptions when running with the nVHE host vector. This will allow a future patch to handle kCFI (synchronous) errors without affecting other classes of exceptions. Signed-off-by: Pierre-Clément Tosi --- arch/arm64/kvm/hyp/nvhe/host.S | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 7397b4f1838a..0613b6e35137 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -183,7 +183,7 @@ SYM_FUNC_END(__host_hvc) .endif .endm -.macro invalid_host_el2_vect +.macro __host_el2_vect handler:req .align 7 /* @@ -202,7 +202,7 @@ SYM_FUNC_END(__host_hvc) * context has been saved by __host_exit or after the hyp context has * been partially clobbered by __host_enter. */ - b __hyp_panic + b \handler .L__hyp_sp_overflow\@: /* Switch to the overflow stack */ @@ -212,6 +212,10 @@ SYM_FUNC_END(__host_hvc) ASM_BUG() .endm +.macro host_el2_sync_vect + __host_el2_vect __hyp_panic +.endm + .macro invalid_host_el1_vect .align 7 mov x0, xzr /* restore_host = false */ @@ -221,6 +225,10 @@ SYM_FUNC_END(__host_hvc) b __hyp_do_panic .endm +.macro invalid_host_el2_vect + __host_el2_vect __hyp_panic +.endm + /* * The host vector does not use an ESB instruction in order to avoid consuming * SErrors that should only be consumed by the host. Guest entry is deferred by @@ -238,7 +246,7 @@ SYM_CODE_START(__kvm_hyp_host_vector) invalid_host_el2_vect // FIQ EL2t invalid_host_el2_vect // Error EL2t - invalid_host_el2_vect // Synchronous EL2h + host_el2_sync_vect // Synchronous EL2h invalid_host_el2_vect // IRQ EL2h invalid_host_el2_vect // FIQ EL2h invalid_host_el2_vect // Error EL2h From patchwork Fri May 10 11:26:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661413 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 0BCE5176FAE for ; Fri, 10 May 2024 11:27:33 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340455; cv=none; b=tVGS3ie+Ps8wn4++5PKbNwJ+KexHjXp79gZQ32IUckYKkVoAcydRABAPE9K8b6syKrk/kb1fK/LPgWGIVuU3uxvfZcuYDXsA8zXqFq09w7iXST0OhKC/DXqPL81DDTNafc0KVzlcahLuIjaXTBLT1BcTc78Lc7CnDJ3cjlSc4l8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340455; c=relaxed/simple; bh=WDIF/IYfsmu+ixH4sDQsPfWeDKguwD4ZsBjXEU5uvDI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=cM9WB3mFDGvzMvhYVYPfyUvLJ2QTTcbHiXs5QQSOgCSv6Rq8JyBqx71nRyHxGwACRHMYbJBpa9A0oLPDtY4iWZH982kkw3YJGwaLR5iKXrXu6pq8SKDSM7/rTk+6eTZaEZrtojMR1XAL7jv+3MPdFWQ5hIBzPNd7kn6zUvsE+Ew= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=FzC308oj; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="FzC308oj" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-a59a1fe7396so121441266b.3 for ; Fri, 10 May 2024 04:27:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340452; x=1715945252; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=49RDvmrkbnpDyrHr4vuzEjRKizGtD9K6IaD6JegXO3A=; b=FzC308ojcNMJkZM3YxXTK+w3wg5Pz6lGRrSy0zorI3EQIauOrkvl0vFGZ4vk4406Oa Cc3htdP/pz12bT7BvMCOeU3zFi6uR5Vknb8jBSEwP+sGhEu0uIJi3EjlMfcdOi4WgDG8 j8vbTZn/obkxbc69gq4K3L3CrWw+Vx7Sj61QbMgaihOuHQLWkLYH6hJ4anqBf9aPquJp uSy0YmmoB1LVVV9+0FTPgYOIoWqHs8m4uKSY/L3t1LK009NM430Vr4s3PwFQvQeS1VDz R9W8Fx5kBWEp8HD2QkHfys2/+M1iRPH8R2qAXseQWXtB87zHWd5ZvM/U313C7t4ltSwW a/fg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340452; x=1715945252; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=49RDvmrkbnpDyrHr4vuzEjRKizGtD9K6IaD6JegXO3A=; b=I5XaaR5cE0XZfa3eq9OFeartyxFz9cSO10Gs4QeT8QiQbFiWXTLJ548tTxqYqpSrca yBRskmdQ8/qYjGnYm548kWrnorXO7jxZdpFrbVi3WHJPbBs+rXZKpE5QAkV1ZHoaFufj 2O8a6RlC8sIvBZJRgVKRZkEL9yTQoWx/zsugrLJzzrvM3wgRRqGr4KSmHMB3Vi6BVxTS zbUE979SkRP0PS2HbEBOsmmxnERyJm7xmysI+WFOOdRTz0PnMErCa7GYfZCtct0xz5rQ kLhANagxgS5s4soHAPBipSem9zehWKcVxTdbOqh02EiytCIOqwzgbKopBk1Tri5p+Nu+ 9phQ== X-Forwarded-Encrypted: i=1; AJvYcCVOLmIAS8iYJKxtGCi3kLuwBcJ6KA66jRzZ7TaJlLrrMKSCgjXsQFx/hq3w78LTVo6qBe76SLbqsIDDpwkQIlxC0jB4 X-Gm-Message-State: AOJu0YyXckvRcpPvart5J3GMj8pL5tY615SaCgBPJ8cPV3xCt8z8nDet etWYFTVyfLJ6EGoOeHbDzB2BpE3AhIrOvYGe93xNMgmAsC/g+goE3HGc7i9nOmmjRJYkpHKxkA= = X-Google-Smtp-Source: AGHT+IGbgNiv89bjQrlHWpgNH9EcVBcfDrRvtOjM5hynq/y8S7do68WKAWDs0MZ3rKCKgKDcmOOmZ9+0TQ== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a17:906:1188:b0:a59:c39c:f202 with SMTP id a640c23a62f3a-a5a2d5a6a3amr235366b.5.1715340452299; Fri, 10 May 2024 04:27:32 -0700 (PDT) Date: Fri, 10 May 2024 12:26:35 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-7-ptosi@google.com> Subject: [PATCH v3 06/12] KVM: arm64: nVHE: gen-hyprel: Skip R_AARCH64_ABS32 From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort Ignore R_AARCH64_ABS32 relocations, instead of panicking, when emitting the relocation table of the hypervisor. The toolchain might produce them when generating function calls with kCFI, to allow type ID resolution across compilation units (between the call-site check and the callee's prefixed u32) at link time. They are therefore not needed in the final (runtime) relocation table. Signed-off-by: Pierre-Clément Tosi --- arch/arm64/kvm/hyp/nvhe/gen-hyprel.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c b/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c index 6bc88a756cb7..b63f4e1c1033 100644 --- a/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c +++ b/arch/arm64/kvm/hyp/nvhe/gen-hyprel.c @@ -50,6 +50,9 @@ #ifndef R_AARCH64_ABS64 #define R_AARCH64_ABS64 257 #endif +#ifndef R_AARCH64_ABS32 +#define R_AARCH64_ABS32 258 +#endif #ifndef R_AARCH64_PREL64 #define R_AARCH64_PREL64 260 #endif @@ -383,6 +386,9 @@ static void emit_rela_section(Elf64_Shdr *sh_rela) case R_AARCH64_ABS64: emit_rela_abs64(rela, sh_orig_name); break; + /* Allow 32-bit absolute relocation, for kCFI type hashes. */ + case R_AARCH64_ABS32: + break; /* Allow position-relative data relocations. */ case R_AARCH64_PREL64: case R_AARCH64_PREL32: From patchwork Fri May 10 11:26:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661414 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 82886176FC0 for ; Fri, 10 May 2024 11:27:36 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340458; cv=none; b=UOMqdXxA2tXcCLHjr7rfkXGWZb74XWgXQDQjUXTimYNJ+fKV76oqQsv7d3o9R8Rz9FmcUFjAGPI+5qmf+H1GcGoZDWXwZpAB8IlvEee7CPzvRrzpOwcGOL1E/9grQrcuu2FtHdttOqXhqVOhFapfdsgoMTQdxQYmPja0dXzmdA0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340458; c=relaxed/simple; bh=UDXy6AJXVdSAeErVZLumP8NwdL16orqGjCY+2++AVXE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=YjN5b8GFUy1XR7FHTs498OT5ClUToiv9FeRQ+y3hyGYfX+IvVTrt6ZG3mnO/F2xhmVP6H135Ui44JHof1iNVI+AIMRdxzZPey7dK7l/HVR1V/IwwkBhswPe0tj1Hj7bdmKUQQEogvl3y39LlGEksOG0ZsXRnibmXFrSWX9J7JKE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=YMgX0NO8; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="YMgX0NO8" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-a5a180153aeso122655966b.3 for ; Fri, 10 May 2024 04:27:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340455; x=1715945255; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=8DKhkb6dSizyRVlRq/XnhdKj6+BlqFPiK9mn9/+w2yc=; b=YMgX0NO8i2ruahaVrrW2lhzPt2LjHkuIuJ2hb0+wk2WexIquRAmr1ucakEZO4F1q21 Gx8X+IlF8myaM6hjbhQEK0wPHSS4Y6eawrDF/RPkkeO6aAG4lIPuE2x1MO2Ya7gmW2ku TvoIWct2ef5+ymwL+Va8qdf1VpcMGqoq9jFqBZnSDvrndzNgGh0yeVX1197I29bLto/U DVT8a2KlEoGq9cq05XSWvT63UrBg9Ag1Bj8pOfJdsEcUfrIAS26a9/oPLu2osdQco8SB BekGdFC13k+ABqjYK3rs+fgK32xBpEIqHfw6C8Y4Y8kSvnL63eYPFLiPSyUwve2byYqe KPkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340455; x=1715945255; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=8DKhkb6dSizyRVlRq/XnhdKj6+BlqFPiK9mn9/+w2yc=; b=okx/+nQ1H/OZt3oWBeH49KwIx68qlQUAw2/a8PxFU2XLCbIh3q8lQhjzgmCSUEw2s6 mwUBgLqFQbO67YQyXNdQL8+hO8eTrycwzG9adBCY+ztvfxxfSayPKgWVXlibuOPenNtn hlwXS+2PIpwTp4fol36QLB2D8bdgzpqfdr/IQ2+OQosAYAF34ln6VzXSGwNG9E4fFdGT gaHO+3B9Y2SuL3lbUYvzvT4kx4NEnveWpzOmVgaRLhTjgVqDeztdSv7CZs7Lcqi6CiSG 2ADcPMUhNCWRSCMDHg3wSRFVJTEoukFQOCBYzDCJwdClK+HAHg6/pzQm2FIf+WqsqZJc gUWw== X-Forwarded-Encrypted: i=1; AJvYcCVHVTiiHlF+IBCoWQ11o4qH+qsRLoK3GmarPU2Jz0a+BzkgX8CwwTu8nHBNcmpnPd0K3E5xxxeDYGBWA0ParGrSunTW X-Gm-Message-State: AOJu0YzKNSHyj1Ns9+70MbK+BAsxXOpxGg2grczQugxqs52IrMow7gld UWb5wZvmgVvrGbQ9wpzpgPrN+FP3XphXyVlJMevfPhkLMdVD108AXApTu+dwKm3OVHx6/pIdhQ= = X-Google-Smtp-Source: AGHT+IGkpwxqeSN6EavlDA+BXVJp1GAbnEIVQnql+hC6cCYj3jthHFCaBNvnkUk/0Xp8MBeg6nGyR3xjIA== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a17:907:d383:b0:a59:dcdc:a490 with SMTP id a640c23a62f3a-a5a2d65d604mr184166b.12.1715340454681; Fri, 10 May 2024 04:27:34 -0700 (PDT) Date: Fri, 10 May 2024 12:26:36 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-8-ptosi@google.com> Subject: [PATCH v3 07/12] KVM: arm64: VHE: Mark __hyp_call_panic __noreturn From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort Given that the sole purpose of __hyp_call_panic() is to call panic(), a __noreturn function, give it the __noreturn attribute, removing the need for its caller to use unreachable(). Signed-off-by: Pierre-Clément Tosi Acked-by: Will Deacon --- arch/arm64/kvm/hyp/vhe/switch.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 1581df6aec87..9db04a286398 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -301,7 +301,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) return ret; } -static void __hyp_call_panic(u64 spsr, u64 elr, u64 par) +static void __noreturn __hyp_call_panic(u64 spsr, u64 elr, u64 par) { struct kvm_cpu_context *host_ctxt; struct kvm_vcpu *vcpu; @@ -326,7 +326,6 @@ void __noreturn hyp_panic(void) u64 par = read_sysreg_par(); __hyp_call_panic(spsr, elr, par); - unreachable(); } asmlinkage void kvm_unexpected_el2_exception(void) From patchwork Fri May 10 11:26:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661415 Received: from mail-ej1-f73.google.com (mail-ej1-f73.google.com [209.85.218.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 01671176FD3 for ; Fri, 10 May 2024 11:27:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340460; cv=none; b=Rl1Ik3pFmKR1g0dOJvztc1fmgxtQThAbxe7oSLK+joSPfZG6lFuk3lFL4nb9Tjs2th31fYU26VUrRCCvXz8/43AkPcsbm+oHVIFrKNdO9cC8CrRRpav32GMHmaYWwV6CPyxfo8ub2ehvP+LhsKL7JQIYpqyGpkYfQ4dJobbKYQE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340460; c=relaxed/simple; bh=4OKNjh0djyGb87YHfRcKtU3RYjXJ+P/49RV4qkJxE7U=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=fh7II63lp5LUrIrsC68V+UvqsDgBmWtA0jE6ZZiL/3TNLpEIIn5Es3JBysQ9FCuJV4QS9y7yjbs92rJsQJ2Z1DBXbeYpjzq1jtZGSbUerAcygymZpia1URlOpwuldVwdD9Fb3FtXCYg81+M2LHlFLbe2aSQB0ri1BhtNCW6vGIY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=waQSF/Xd; arc=none smtp.client-ip=209.85.218.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="waQSF/Xd" Received: by mail-ej1-f73.google.com with SMTP id a640c23a62f3a-a59a5b06802so110445466b.1 for ; Fri, 10 May 2024 04:27:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340457; x=1715945257; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=BYiDxgRL4/PJcTPYc8te2M3RBep5LmiQ+KSg/yVolMc=; b=waQSF/Xd0mtcOHjGJYti3C9qDPbBs/B+B2u5+1MTaNkC5WitBHWYGQLqbJ6hGiEG4X BjdM/MggdYOsDkCV4J7miAqyTvRUO5hnShPI4pIEjsw+0dO3XnIMNrSF88A8jvuZTPQY Rach4cYom0QX7FndH2gftjaven4VOoGEHp/oCkP4YVCBE09vvr/0j5m6Z6KgOspyBMh9 ydlp7fL0tqFqQxadXIULR77k2vyV1s81lJXeoc1RYv6yEW6Kbn9Fs3Vc3lt26C4n/Fx0 it4/5nw8MZzkEtZHCTltaLe/DsWgoZQjL5mftxl4WtryVUFWqKuZ7U32ns2CBV6HiWVm MLog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340457; x=1715945257; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=BYiDxgRL4/PJcTPYc8te2M3RBep5LmiQ+KSg/yVolMc=; b=F4aJlExjYpFz+t99WwVgV5F6eUNPFuAZxLUHulWZGFpZXqJX9RCCwi1bv1X5qGTGEf pHgQow1Sr4brRsTZFBeV/adtsuPF/SF8diHQvzMs0fnT5yKVwp78PCAfHtuzROxSxxK2 gnDPnl3m3NvRAAS9Voo/qiKuD3aKV4YrytzrqN7w0r7mKq8R2GOVy8y7hEa7kMxmHtU3 cZrvEjU/XjOSs6Lh8iLFIUr81EhKqnjoDVOuir9fvDA1iEHg7aQo8haLr8U819+2gJNZ SyY3zjnek8xVZpAWsIZBRdbQC+P0udvPM1BgmR3JSqJdQ/4PIl8OFMQ1RCEkMemNpEJR ZYhA== X-Forwarded-Encrypted: i=1; AJvYcCURZYu1JkJCvb8+RhowysdtpRKimZZWi+Ufbhw+HEYY/w/6pSzuZD9rEo0sO1+udNe80H+ygrOfO159N/g68TOE2YpF X-Gm-Message-State: AOJu0Yw4ASfTYUw4Z1LuD9EgruFmESGtfxPndAok2hq1QFWrRlxuDukc Ly0IGyGA3glA0udx6NkGbFFSEmoIANM+s5qxAi0MG3+Bgn9nSoBcGEPPWEntqEJQ963VK7MWOw= = X-Google-Smtp-Source: AGHT+IHFd5o0vQu25tuDgldMkm4aPBD8ZTZa88+e/50cMnpBaWWfrW1M4Ov32MuLEgdpxktrbzGRQJLKwg== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a17:906:27cc:b0:a54:c131:8128 with SMTP id a640c23a62f3a-a5a2d6796c7mr215866b.14.1715340457059; Fri, 10 May 2024 04:27:37 -0700 (PDT) Date: Fri, 10 May 2024 12:26:37 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-9-ptosi@google.com> Subject: [PATCH v3 08/12] arm64: Move esr_comment() to From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort As it is already defined twice and is about to be needed for kCFI error detection, move esr_comment() to a header for re-use, with a clearer name. Signed-off-by: Pierre-Clément Tosi Acked-by: Will Deacon --- arch/arm64/include/asm/esr.h | 5 +++++ arch/arm64/kernel/debug-monitors.c | 4 +--- arch/arm64/kernel/traps.c | 8 +++----- arch/arm64/kvm/handle_exit.c | 2 +- 4 files changed, 10 insertions(+), 9 deletions(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 81606bf7d5ac..2bcf216be376 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -379,6 +379,11 @@ #ifndef __ASSEMBLY__ #include +static inline unsigned long esr_brk_comment(unsigned long esr) +{ + return esr & ESR_ELx_BRK64_ISS_COMMENT_MASK; +} + static inline bool esr_is_data_abort(unsigned long esr) { const unsigned long ec = ESR_ELx_EC(esr); diff --git a/arch/arm64/kernel/debug-monitors.c b/arch/arm64/kernel/debug-monitors.c index 64f2ecbdfe5c..024a7b245056 100644 --- a/arch/arm64/kernel/debug-monitors.c +++ b/arch/arm64/kernel/debug-monitors.c @@ -312,9 +312,7 @@ static int call_break_hook(struct pt_regs *regs, unsigned long esr) * entirely not preemptible, and we can use rcu list safely here. */ list_for_each_entry_rcu(hook, list, node) { - unsigned long comment = esr & ESR_ELx_BRK64_ISS_COMMENT_MASK; - - if ((comment & ~hook->mask) == hook->imm) + if ((esr_brk_comment(esr) & ~hook->mask) == hook->imm) fn = hook->fn; } diff --git a/arch/arm64/kernel/traps.c b/arch/arm64/kernel/traps.c index 215e6d7f2df8..2652247032ae 100644 --- a/arch/arm64/kernel/traps.c +++ b/arch/arm64/kernel/traps.c @@ -1105,8 +1105,6 @@ static struct break_hook ubsan_break_hook = { }; #endif -#define esr_comment(esr) ((esr) & ESR_ELx_BRK64_ISS_COMMENT_MASK) - /* * Initial handler for AArch64 BRK exceptions * This handler only used until debug_traps_init(). @@ -1115,15 +1113,15 @@ int __init early_brk64(unsigned long addr, unsigned long esr, struct pt_regs *regs) { #ifdef CONFIG_CFI_CLANG - if ((esr_comment(esr) & ~CFI_BRK_IMM_MASK) == CFI_BRK_IMM_BASE) + if ((esr_brk_comment(esr) & ~CFI_BRK_IMM_MASK) == CFI_BRK_IMM_BASE) return cfi_handler(regs, esr) != DBG_HOOK_HANDLED; #endif #ifdef CONFIG_KASAN_SW_TAGS - if ((esr_comment(esr) & ~KASAN_BRK_MASK) == KASAN_BRK_IMM) + if ((esr_brk_comment(esr) & ~KASAN_BRK_MASK) == KASAN_BRK_IMM) return kasan_handler(regs, esr) != DBG_HOOK_HANDLED; #endif #ifdef CONFIG_UBSAN_TRAP - if ((esr_comment(esr) & ~UBSAN_BRK_MASK) == UBSAN_BRK_IMM) + if ((esr_brk_comment(esr) & ~UBSAN_BRK_MASK) == UBSAN_BRK_IMM) return ubsan_handler(regs, esr) != DBG_HOOK_HANDLED; #endif return bug_handler(regs, esr) != DBG_HOOK_HANDLED; diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 617ae6dea5d5..0bcafb3179d6 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -395,7 +395,7 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, if (mode != PSR_MODE_EL2t && mode != PSR_MODE_EL2h) { kvm_err("Invalid host exception to nVHE hyp!\n"); } else if (ESR_ELx_EC(esr) == ESR_ELx_EC_BRK64 && - (esr & ESR_ELx_BRK64_ISS_COMMENT_MASK) == BUG_BRK_IMM) { + esr_brk_comment(esr) == BUG_BRK_IMM) { const char *file = NULL; unsigned int line = 0; From patchwork Fri May 10 11:26:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661416 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A928416DED7 for ; Fri, 10 May 2024 11:27:40 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340462; cv=none; b=rlH9VFN23ay79RTQpeY40bjc1x96GNNgVsHpI7b9EmOkns0uPv5j49kGs97MTYGloxsuo3MDqjisK3EbD+Pp1vGt3T89ASXSPsRHFGcvnZnVeU5fmAX7S1QqsZgfoVIkNQZmTAlu4LG0rslijzrbYEmn4R4SQ1QCqxJXczK1wcQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340462; c=relaxed/simple; bh=Li5DTfka/YMPlAybzWKY2XloO6nERNB6XPnN0FGTd5Y=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=WcegCnfpB77lWv6w/qVmnbyL3CYlgqdwWY81T2MMUYXsKbSH3o14EBf3DcwaDJlyjPRqEWlT+NeSdCGuak+EFOJVWsf5aocXkLZ0n5Qg0N+dsIpKKFWXlaXSVADBvbbJN+FMrxWzGJK1yxzPmO6U8a3qqW+pk/b/+UuSyUiWpnE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=T7t3OFh/; arc=none smtp.client-ip=209.85.219.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="T7t3OFh/" Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-de604d35ec0so3290861276.3 for ; Fri, 10 May 2024 04:27:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340459; x=1715945259; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=qeYMQHI62TLtzWSMqgKJxIUsn2T5TYiEl6TIFioPKSw=; b=T7t3OFh/qptybsm3ZHpxMnU4KcnWRP+uF9VQr1sxfNFQswL4C/sTHYZkGC2imiiXwy v2QI2BUbENfjVduo6YsuYzr15O1lOSjMGZYAHWsgfwImKoWryk35gL/1Z2g8FRAhz/ia 8VRPtAxRQ+owykXsLc9f2PebVJxv9A672nThnyCTsASSm8kR4qqyMDiMlxpbOgU75cAm L0BNkvCJn17JXNvf/OfWNkvQcG+ka5321YGBUsbh6hkQG0T+ZcBK61zFN2QJ5BPcQtRO siMuWbCtfWfBHzbspOiAvuuzjyBVU8KeIYR39V2sCVPByf8iMVDaxFqx9y+PI4qVJWj1 +QSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340459; x=1715945259; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=qeYMQHI62TLtzWSMqgKJxIUsn2T5TYiEl6TIFioPKSw=; b=L0EFGFqzqzSuwnkoVJumK2W06ct/DSFIrQ8mGYDHa3Rj6PWimSKiSR0DlnrCWh7O9d fEM51doT3cnyC0abfx2m1cZtCs5R0VfPbPvUMkCAo3Wd7rE5Zg/TtHYBD0mRo+DyO6O+ 5L7aiNIfhIRFbVfpdPU9bsH7kROZ8Z6JQH6yxK029TIhH7uDmwC2/v6ZXSYEB0+fLVxR Rw3KuaD2490S11aUv3MKxElvC8qHXTomLc2l0Qehz8lMM2Dv8pw+wViNea8B7TMl7rrP xaHWStGWsKx5KfCYZIcNFlqzWU7sRZpf5bHqzjem6nB9eb3zq8MH+t3+NQm6Wh3ScVO2 i2Gg== X-Forwarded-Encrypted: i=1; AJvYcCV8jFlZcZq8fMwt4P18WPXRDW6uDHQOdsb3t3VS9fZiB+91UPKO/fYdDVxOCSbkmEgJ5LvhSqBTTndqXNoAYDZxLf+k X-Gm-Message-State: AOJu0Yxl7Ol+xkcLM8ORo/tvBVbOPsJrFjqasiGBf4AkjxfLJale7NrK Exg7VbzVwGx4/i1UZpSjwJNXh1Gcvsy63bKdR3qn0QFAm1CIhiI86Esfs0pYuwnC4Xuof8ylaQ= = X-Google-Smtp-Source: AGHT+IG2Nk6GDvDApxzf2fQrO3hpSPR8XKLyC1KYy5HQT81VyW90ugxbfQimtrj4DVkv2r3ubwFrbsoU3Q== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a05:6902:1082:b0:dcb:b9d7:2760 with SMTP id 3f1490d57ef6-dee4f38cb25mr612041276.13.1715340459676; Fri, 10 May 2024 04:27:39 -0700 (PDT) Date: Fri, 10 May 2024 12:26:38 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-10-ptosi@google.com> Subject: [PATCH v3 09/12] KVM: arm64: VHE: Add test module for hyp kCFI From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort In order to easily periodically (and potentially automatically) validate that the hypervisor kCFI feature doesn't bitrot, introduce a way to trigger hypervisor kCFI faults from userspace on test builds of KVM. Add hooks in the hypervisor code to call registered callbacks (intended to trigger kCFI faults either for the callback call itself of from within the callback function) when running with guest or host VBAR_EL2. As the calls are issued from the KVM_RUN ioctl handling path, userspace gains control over when the actual triggering of the fault happens without needing to modify the KVM uAPI. Export kernel functions to register these callbacks from modules and introduce a kernel module intended to contain any testing logic. By limiting the changes to the core kernel to a strict minimum, this architectural split allows tests to be updated (within the module) without the need to redeploy (or recompile) the kernel (hyp) under test. Use the module parameters as the uAPI for configuring the fault condition being tested (i.e. either at insertion or post-insertion using /sys/module/.../parameters), which naturally makes it impossible for userspace to test kCFI without the module (and, inversely, makes the module only - not KVM - responsible for exposing said uAPI). As kCFI is implemented with a caller-side check of a callee-side value, make the module support 4 tests based on the location of the caller and callee (built-in or in-module), for each of the 2 hypervisor contexts (host & guest), selected by userspace using the 'guest' or 'host' module parameter. For this purpose, export symbols which the module can use to configure the callbacks for in-kernel and module-to-built-in kCFI faulting calls. Define the module-to-kernel API to allow the module to detect that it was loaded on a kernel built with support for it but which is running without a hypervisor (-ENXIO) or with one that doesn't use the VHE CPU feature (-EOPNOTSUPP), which is currently the only mode for which KVM supports hypervisor kCFI. Allow kernel build configs to set CONFIG_HYP_CFI_TEST to only support the in-kernel hooks (=y) or also build the test module (=m). Use intermediate internal Kconfig flags (CONFIG_HYP_SUPPORTS_CFI_TEST and CONFIG_HYP_CFI_TEST_MODULE) to simplify the Makefiles and #ifdefs. As the symbols for callback registration are only exported to modules when CONFIG_HYP_CFI_TEST != n, it is impossible for the test module to be non-forcefully inserted on a kernel that doesn't support it. Note that this feature must NOT result in any noticeable change (behavioral or binary size) when HYP_CFI_TEST_MODULE = n. CONFIG_HYP_CFI_TEST is intentionally independent of CONFIG_CFI_CLANG, to avoid arbitrarily limiting the number of flag combinations that can be tested with the module. Also note that, as VHE aliases VBAR_EL1 to VBAR_EL2 for the host, testing hypervisor kCFI in VHE and in host context is equivalent to testing kCFI support of the kernel itself i.e. EL1 in non-VHE and/or in non-virtualized environments. For this reason, CONFIG_CFI_PERMISSIVE **will** prevent the test module from triggering a hyp panic (although a warning still gets printed) in that context. Signed-off-by: Pierre-Clément Tosi --- arch/arm64/include/asm/kvm_cfi.h | 36 ++++++++ arch/arm64/kvm/Kconfig | 22 +++++ arch/arm64/kvm/Makefile | 3 + arch/arm64/kvm/hyp/include/hyp/cfi.h | 47 ++++++++++ arch/arm64/kvm/hyp/vhe/Makefile | 1 + arch/arm64/kvm/hyp/vhe/cfi.c | 37 ++++++++ arch/arm64/kvm/hyp/vhe/switch.c | 7 ++ arch/arm64/kvm/hyp_cfi_test.c | 43 +++++++++ arch/arm64/kvm/hyp_cfi_test_module.c | 133 +++++++++++++++++++++++++++ 9 files changed, 329 insertions(+) create mode 100644 arch/arm64/include/asm/kvm_cfi.h create mode 100644 arch/arm64/kvm/hyp/include/hyp/cfi.h create mode 100644 arch/arm64/kvm/hyp/vhe/cfi.c create mode 100644 arch/arm64/kvm/hyp_cfi_test.c create mode 100644 arch/arm64/kvm/hyp_cfi_test_module.c diff --git a/arch/arm64/include/asm/kvm_cfi.h b/arch/arm64/include/asm/kvm_cfi.h new file mode 100644 index 000000000000..13cc7b19d838 --- /dev/null +++ b/arch/arm64/include/asm/kvm_cfi.h @@ -0,0 +1,36 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 - Google Inc + * Author: Pierre-Clément Tosi + */ + +#ifndef __ARM64_KVM_CFI_H__ +#define __ARM64_KVM_CFI_H__ + +#include +#include + +#ifdef CONFIG_HYP_SUPPORTS_CFI_TEST + +int kvm_cfi_test_register_host_ctxt_cb(void (*cb)(void)); +int kvm_cfi_test_register_guest_ctxt_cb(void (*cb)(void)); + +#else + +static inline int kvm_cfi_test_register_host_ctxt_cb(void (*cb)(void)) +{ + return -EOPNOTSUPP; +} + +static inline int kvm_cfi_test_register_guest_ctxt_cb(void (*cb)(void)) +{ + return -EOPNOTSUPP; +} + +#endif /* CONFIG_HYP_SUPPORTS_CFI_TEST */ + +/* Symbols which the host can register as hyp callbacks; see . */ +void hyp_trigger_builtin_cfi_fault(void); +void hyp_builtin_cfi_fault_target(int unused); + +#endif /* __ARM64_KVM_CFI_H__ */ diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 58f09370d17e..5daa8079a120 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -65,4 +65,26 @@ config PROTECTED_NVHE_STACKTRACE If unsure, or not using protected nVHE (pKVM), say N. +config HYP_CFI_TEST + tristate "KVM hypervisor kCFI test support" + depends on KVM + help + Say Y or M here to build KVM with test hooks to support intentionally + triggering hypervisor kCFI faults in guest or host context. + + Say M here to also build a module which registers callbacks triggering + faults and selected by userspace through its parameters. + + Note that this feature is currently only supported in VHE mode. + + If unsure, say N. + +config HYP_SUPPORTS_CFI_TEST + def_bool y + depends on HYP_CFI_TEST + +config HYP_CFI_TEST_MODULE + def_tristate m if HYP_CFI_TEST = m + depends on HYP_CFI_TEST + endif # VIRTUALIZATION diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile index c0c050e53157..d42540ae3ea7 100644 --- a/arch/arm64/kvm/Makefile +++ b/arch/arm64/kvm/Makefile @@ -22,6 +22,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \ vgic/vgic-mmio-v3.o vgic/vgic-kvm-device.o \ vgic/vgic-its.o vgic/vgic-debug.o +kvm-$(CONFIG_HYP_SUPPORTS_CFI_TEST) += hyp_cfi_test.o kvm-$(CONFIG_HW_PERF_EVENTS) += pmu-emul.o pmu.o always-y := hyp_constants.h hyp-constants.s @@ -39,3 +40,5 @@ $(obj)/hyp_constants.h: $(obj)/hyp-constants.s FORCE obj-kvm := $(addprefix $(obj)/, $(kvm-y)) $(obj-kvm): $(obj)/hyp_constants.h + +obj-$(CONFIG_HYP_CFI_TEST_MODULE) += hyp_cfi_test_module.o diff --git a/arch/arm64/kvm/hyp/include/hyp/cfi.h b/arch/arm64/kvm/hyp/include/hyp/cfi.h new file mode 100644 index 000000000000..c6536040bc06 --- /dev/null +++ b/arch/arm64/kvm/hyp/include/hyp/cfi.h @@ -0,0 +1,47 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 - Google Inc + * Author: Pierre-Clément Tosi + */ + +#ifndef __ARM64_KVM_HYP_CFI_H__ +#define __ARM64_KVM_HYP_CFI_H__ + +#include +#include + +#include + +#ifdef CONFIG_HYP_SUPPORTS_CFI_TEST + +int __kvm_register_cfi_test_cb(void (*cb)(void), bool in_host_ctxt); + +extern void (*hyp_test_host_ctxt_cfi)(void); +extern void (*hyp_test_guest_ctxt_cfi)(void); + +/* Hypervisor callbacks for the host to register. */ +void hyp_trigger_builtin_cfi_fault(void); +void hyp_builtin_cfi_fault_target(int unused); + +#else + +static inline +int __kvm_register_cfi_test_cb(void (*cb)(void), bool in_host_ctxt) +{ + return -EOPNOTSUPP; +} + +#define hyp_test_host_ctxt_cfi ((void(*)(void))(NULL)) +#define hyp_test_guest_ctxt_cfi ((void(*)(void))(NULL)) + +static inline void hyp_trigger_builtin_cfi_fault(void) +{ +} + +static inline void hyp_builtin_cfi_fault_target(int __always_unused unused) +{ +} + +#endif /* CONFIG_HYP_SUPPORTS_CFI_TEST */ + +#endif /* __ARM64_KVM_HYP_CFI_H__ */ diff --git a/arch/arm64/kvm/hyp/vhe/Makefile b/arch/arm64/kvm/hyp/vhe/Makefile index 3b9e5464b5b3..19ca584cc21e 100644 --- a/arch/arm64/kvm/hyp/vhe/Makefile +++ b/arch/arm64/kvm/hyp/vhe/Makefile @@ -9,3 +9,4 @@ ccflags-y := -D__KVM_VHE_HYPERVISOR__ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o +obj-$(CONFIG_HYP_SUPPORTS_CFI_TEST) += cfi.o diff --git a/arch/arm64/kvm/hyp/vhe/cfi.c b/arch/arm64/kvm/hyp/vhe/cfi.c new file mode 100644 index 000000000000..5849f239e27f --- /dev/null +++ b/arch/arm64/kvm/hyp/vhe/cfi.c @@ -0,0 +1,37 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 - Google Inc + * Author: Pierre-Clément Tosi + */ +#include + +#include + +void (*hyp_test_host_ctxt_cfi)(void); +void (*hyp_test_guest_ctxt_cfi)(void); + +int __kvm_register_cfi_test_cb(void (*cb)(void), bool in_host_ctxt) +{ + if (in_host_ctxt) + hyp_test_host_ctxt_cfi = cb; + else + hyp_test_guest_ctxt_cfi = cb; + + return 0; +} + +void hyp_builtin_cfi_fault_target(int __always_unused unused) +{ +} + +void hyp_trigger_builtin_cfi_fault(void) +{ + /* Intentional UB cast & dereference, to trigger a kCFI fault. */ + void (*target)(void) = (void *)&hyp_builtin_cfi_fault_target; + + /* + * READ_ONCE() prevents this indirect call from being optimized out, + * forcing the compiler to generate the kCFI check before the branch. + */ + READ_ONCE(target)(); +} diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index 9db04a286398..b3268933b093 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -4,6 +4,7 @@ * Author: Marc Zyngier */ +#include #include #include @@ -221,6 +222,9 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) struct kvm_cpu_context *guest_ctxt; u64 exit_code; + if (IS_ENABLED(CONFIG_HYP_SUPPORTS_CFI_TEST) && unlikely(hyp_test_host_ctxt_cfi)) + hyp_test_host_ctxt_cfi(); + host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt; host_ctxt->__hyp_running_vcpu = vcpu; guest_ctxt = &vcpu->arch.ctxt; @@ -245,6 +249,9 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu) else vcpu_clear_flag(vcpu, VCPU_HYP_CONTEXT); + if (IS_ENABLED(CONFIG_HYP_SUPPORTS_CFI_TEST) && unlikely(hyp_test_guest_ctxt_cfi)) + hyp_test_guest_ctxt_cfi(); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu); diff --git a/arch/arm64/kvm/hyp_cfi_test.c b/arch/arm64/kvm/hyp_cfi_test.c new file mode 100644 index 000000000000..da7b25ca1b1f --- /dev/null +++ b/arch/arm64/kvm/hyp_cfi_test.c @@ -0,0 +1,43 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 - Google Inc + * Author: Pierre-Clément Tosi + */ +#include +#include +#include +#include + +#include +#include +#include + +/* For calling directly into the VHE hypervisor; see . */ +int __kvm_register_cfi_test_cb(void (*)(void), bool); + +static int kvm_register_cfi_test_cb(void (*vhe_cb)(void), bool in_host_ctxt) +{ + if (!is_hyp_mode_available()) + return -ENXIO; + + if (is_hyp_nvhe()) + return -EOPNOTSUPP; + + return __kvm_register_cfi_test_cb(vhe_cb, in_host_ctxt); +} + +int kvm_cfi_test_register_host_ctxt_cb(void (*cb)(void)) +{ + return kvm_register_cfi_test_cb(cb, true); +} +EXPORT_SYMBOL(kvm_cfi_test_register_host_ctxt_cb); + +int kvm_cfi_test_register_guest_ctxt_cb(void (*cb)(void)) +{ + return kvm_register_cfi_test_cb(cb, false); +} +EXPORT_SYMBOL(kvm_cfi_test_register_guest_ctxt_cb); + +/* Hypervisor callbacks for the test module to register. */ +EXPORT_SYMBOL(hyp_trigger_builtin_cfi_fault); +EXPORT_SYMBOL(hyp_builtin_cfi_fault_target); diff --git a/arch/arm64/kvm/hyp_cfi_test_module.c b/arch/arm64/kvm/hyp_cfi_test_module.c new file mode 100644 index 000000000000..eeda4be4d3ef --- /dev/null +++ b/arch/arm64/kvm/hyp_cfi_test_module.c @@ -0,0 +1,133 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2024 - Google Inc + * Author: Pierre-Clément Tosi + */ +#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt + +#include +#include +#include + +#include +#include +#include +#include + +static int set_host_mode(const char *val, const struct kernel_param *kp); +static int set_guest_mode(const char *val, const struct kernel_param *kp); + +#define M_DESC \ + "\n\t0: none" \ + "\n\t1: built-in caller & built-in callee" \ + "\n\t2: built-in caller & module callee" \ + "\n\t3: module caller & built-in callee" \ + "\n\t4: module caller & module callee" + +static unsigned int host_mode; +module_param_call(host, set_host_mode, param_get_uint, &host_mode, 0644); +MODULE_PARM_DESC(host, + "Hypervisor kCFI fault test case in host context:" M_DESC); + +static unsigned int guest_mode; +module_param_call(guest, set_guest_mode, param_get_uint, &guest_mode, 0644); +MODULE_PARM_DESC(guest, + "Hypervisor kCFI fault test case in guest context:" M_DESC); + +static void trigger_module2module_cfi_fault(void); +static void trigger_module2builtin_cfi_fault(void); +static void hyp_cfi_module2module_test_target(int); +static void hyp_cfi_builtin2module_test_target(int); + +static int set_param_mode(const char *val, const struct kernel_param *kp, + int (*register_cb)(void (*)(void))) +{ + unsigned int *mode = kp->arg; + int err; + + err = param_set_uint(val, kp); + if (err) + return err; + + switch (*mode) { + case 0: + return register_cb(NULL); + case 1: + return register_cb(hyp_trigger_builtin_cfi_fault); + case 2: + return register_cb((void *)hyp_cfi_builtin2module_test_target); + case 3: + return register_cb(trigger_module2builtin_cfi_fault); + case 4: + return register_cb(trigger_module2module_cfi_fault); + default: + return -EINVAL; + } +} + +static int set_host_mode(const char *val, const struct kernel_param *kp) +{ + return set_param_mode(val, kp, kvm_cfi_test_register_host_ctxt_cb); +} + +static int set_guest_mode(const char *val, const struct kernel_param *kp) +{ + return set_param_mode(val, kp, kvm_cfi_test_register_guest_ctxt_cb); +} + +static void __exit exit_hyp_cfi_test(void) +{ + int err; + + err = kvm_cfi_test_register_host_ctxt_cb(NULL); + if (err) + pr_err("Failed to unregister host context trigger: %d\n", err); + + err = kvm_cfi_test_register_guest_ctxt_cb(NULL); + if (err) + pr_err("Failed to unregister guest context trigger: %d\n", err); +} +module_exit(exit_hyp_cfi_test); + +static void trigger_module2builtin_cfi_fault(void) +{ + /* Intentional UB cast & dereference, to trigger a kCFI fault. */ + void (*target)(void) = (void *)&hyp_builtin_cfi_fault_target; + + /* + * READ_ONCE() prevents this indirect call from being optimized out, + * forcing the compiler to generate the kCFI check before the branch. + */ + READ_ONCE(target)(); + + pr_err_ratelimited("%s: Survived a kCFI violation\n", __func__); +} + +static void trigger_module2module_cfi_fault(void) +{ + /* Intentional UB cast & dereference, to trigger a kCFI fault. */ + void (*target)(void) = (void *)&hyp_cfi_module2module_test_target; + + /* + * READ_ONCE() prevents this indirect call from being optimized out, + * forcing the compiler to generate the kCFI check before the branch. + */ + READ_ONCE(target)(); + + pr_err_ratelimited("%s: Survived a kCFI violation\n", __func__); +} + +/* Use different functions, for clearer symbols in kCFI panic reports. */ +static noinline +void hyp_cfi_module2module_test_target(int __always_unused unused) +{ +} + +static noinline +void hyp_cfi_builtin2module_test_target(int __always_unused unused) +{ +} + +MODULE_LICENSE("GPL"); +MODULE_AUTHOR("Pierre-Clément Tosi "); +MODULE_DESCRIPTION("KVM hypervisor kCFI test module"); From patchwork Fri May 10 11:26:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661417 Received: from mail-ed1-f73.google.com (mail-ed1-f73.google.com [209.85.208.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 826E0179211 for ; Fri, 10 May 2024 11:27:43 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.73 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340465; cv=none; b=gMq1P1OMDBf2dB65tahrP0gP6hxn02d3Te4oebxaiQ3L8a2ZHmvvz7Z8O6LxFQ4BVY78N6itsdHYhjYCCeONXntOBgIyS121yJOgW6zcixC5kv1tomzgGNnP7H4O51pQfWbLd/en4t1bAIPQwtGhlBDpJt1Gglq10vfiy9Qq9E0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340465; c=relaxed/simple; bh=tINeLBZyoJGokm0sOYwcRkf991k7B88IwoNF+qWz8XI=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=X+RPGi13kT/E1RVQK1u3pPwLifR3L3pkF8A4ez5ExohdlvdHWS9wVzNKrhmyLgmbayCXGQN8/JZM6aPR2VzY6F9EIjHIum7quBzx88nkXmlpmCvOqlee6FwosHU7GJl7Dp4Zq0XuXThjmR0lCwHjs1LF+CmK2/y0HQS+1XqEIU4= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=O9uyZTC7; arc=none smtp.client-ip=209.85.208.73 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="O9uyZTC7" Received: by mail-ed1-f73.google.com with SMTP id 4fb4d7f45d1cf-572a175621bso1290670a12.3 for ; Fri, 10 May 2024 04:27:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340462; x=1715945262; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=kFhYs5lDqdu/+fs82ZoKuSCT3z4IC8Cn6RP491+YFBU=; b=O9uyZTC7l3jt0YYidp+d1z5jL216PRCm7fygQa6OXWZ2VMo8eBBr7uvOu8hsiH0bKj SOgXAQlYsYDmye8HJPfo68vwsQdjZf68hFJResScVPcpk0Dx/xkUmyMmr2IrZ6qpE2T3 ksBNLkCl7cyAz4Vj3USj3y6Dnw7CT1II8li0UisXDCABdw0JziRJrzktt50KJfuSQGrP fH3+YztFju8BOcpOCVVThDZut3/lSDiSdvFB0+e2+L57yW32uO7W3d5tu8SUJhyl3Aqb K18Hilitb8+4w762uPKbUNijDsEi2jJyxQ/rLs/XHciS2BIUI1gsi8eNwpdchY7viPsW Qoiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340462; x=1715945262; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=kFhYs5lDqdu/+fs82ZoKuSCT3z4IC8Cn6RP491+YFBU=; b=Qh+yyUKQ4kndwIcW0EnxZWUezedtSNFQWHeYRTiyc98x3gtq2JFICfWpi5vkxJYUHN x2lSkzNoc3Q2+4r9p+qkX/SA0SVqQMcqFzo3HDH3brbch5ancCn1gwOZ0rYAw8O4Uyt+ PiEEBoHEFPi7zx2F/CQRu9UKH4i7XU41AQTexCGg0AOJohRxNN8WAFsyVPgQ4EzBRWwB aITt7qu1Ss7Es2MzAoYnTc7aQLlWLpKeKVMRoXseyhdlnnJAjWo6aTaJgUp6PhbRdakJ iChva/imWRKQ3Zs7dleFnik5qLQdWBXiwvZMOhIEo1AMdlqmta11E9hcs/vMpTMh1Knt rdwQ== X-Forwarded-Encrypted: i=1; AJvYcCWBCa4HgB6JCdMDxkK6XI/OoLAXIMBvqQcuiDAMzrcer9MkBxHfDHmY/eSgAXWajpPnLTrAurM5tX/tpEXxgDag2is6 X-Gm-Message-State: AOJu0Yw6d+QnWt23pR3MVZBbsQS48CrNTdZtonyT42fpYJ2UMmY/mzJw 73TTisp+lWoHfkTzrz2EP/4PuoU8Owd9/eBlh24lZNw2if/5mjBZMY2lDdRUrv2H1nea34dxMQ= = X-Google-Smtp-Source: AGHT+IGJhtD/wN9N41xGH4SqpBDhacAaIk1ZGwSPDqSUff1bOTvFtTBlhk3QdXGF6oIEd68ubcVUJM9mKg== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:aa7:d4d2:0:b0:572:3273:62d5 with SMTP id 4fb4d7f45d1cf-5734d6ecb9cmr2976a12.5.1715340461789; Fri, 10 May 2024 04:27:41 -0700 (PDT) Date: Fri, 10 May 2024 12:26:39 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-11-ptosi@google.com> Subject: [PATCH v3 10/12] KVM: arm64: nVHE: Support CONFIG_CFI_CLANG at EL2 From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort The compiler implements kCFI by adding type information (u32) above every function that might be indirectly called and, whenever a function pointer is called, injects a read-and-compare of that u32 against the value corresponding to the expected type. In case of a mismatch, a BRK instruction gets executed. When the hypervisor triggers such an exception in nVHE, it panics and triggers and exception return to EL1. Therefore, teach nvhe_hyp_panic_handler() to detect kCFI errors from the ESR and report them. If necessary, remind the user that EL2 kCFI is not affected by CONFIG_CFI_PERMISSIVE. Pass $(CC_FLAGS_CFI) to the compiler when building the nVHE hyp code. Use SYM_TYPED_FUNC_START() for __pkvm_init_switch_pgd, as nVHE can't call it directly and must use a PA function pointer from C (because it is part of the idmap page), which would trigger a kCFI failure if the type ID wasn't present. Signed-off-by: Pierre-Clément Tosi --- arch/arm64/include/asm/esr.h | 6 ++++++ arch/arm64/kvm/handle_exit.c | 11 +++++++++++ arch/arm64/kvm/hyp/nvhe/Makefile | 6 +++--- arch/arm64/kvm/hyp/nvhe/hyp-init.S | 6 +++++- 4 files changed, 25 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/esr.h b/arch/arm64/include/asm/esr.h index 2bcf216be376..9eb9e6aa70cf 100644 --- a/arch/arm64/include/asm/esr.h +++ b/arch/arm64/include/asm/esr.h @@ -391,6 +391,12 @@ static inline bool esr_is_data_abort(unsigned long esr) return ec == ESR_ELx_EC_DABT_LOW || ec == ESR_ELx_EC_DABT_CUR; } +static inline bool esr_is_cfi_brk(unsigned long esr) +{ + return ESR_ELx_EC(esr) == ESR_ELx_EC_BRK64 && + (esr_brk_comment(esr) & ~CFI_BRK_IMM_MASK) == CFI_BRK_IMM_BASE; +} + static inline bool esr_fsc_is_translation_fault(unsigned long esr) { /* Translation fault, level -1 */ diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 0bcafb3179d6..0db23a6304ce 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -383,6 +383,15 @@ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) kvm_handle_guest_serror(vcpu, kvm_vcpu_get_esr(vcpu)); } +static void kvm_nvhe_report_cfi_failure(u64 panic_addr) +{ + kvm_err("nVHE hyp CFI failure at: [<%016llx>] %pB!\n", panic_addr, + (void *)(panic_addr + kaslr_offset())); + + if (IS_ENABLED(CONFIG_CFI_PERMISSIVE)) + kvm_err(" (CONFIG_CFI_PERMISSIVE ignored for hyp failures)\n"); +} + void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, u64 elr_virt, u64 elr_phys, u64 par, uintptr_t vcpu, @@ -413,6 +422,8 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, else kvm_err("nVHE hyp BUG at: [<%016llx>] %pB!\n", panic_addr, (void *)(panic_addr + kaslr_offset())); + } else if (IS_ENABLED(CONFIG_CFI_CLANG) && esr_is_cfi_brk(esr)) { + kvm_nvhe_report_cfi_failure(panic_addr); } else { kvm_err("nVHE hyp panic at: [<%016llx>] %pB!\n", panic_addr, (void *)(panic_addr + kaslr_offset())); diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 2250253a6429..2eb915d8943f 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -89,9 +89,9 @@ quiet_cmd_hyprel = HYPREL $@ quiet_cmd_hypcopy = HYPCOPY $@ cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@ -# Remove ftrace, Shadow Call Stack, and CFI CFLAGS. -# This is equivalent to the 'notrace', '__noscs', and '__nocfi' annotations. -KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS) $(CC_FLAGS_CFI), $(KBUILD_CFLAGS)) +# Remove ftrace and Shadow Call Stack CFLAGS. +# This is equivalent to the 'notrace' and '__noscs' annotations. +KBUILD_CFLAGS := $(filter-out $(CC_FLAGS_FTRACE) $(CC_FLAGS_SCS), $(KBUILD_CFLAGS)) # Starting from 13.0.0 llvm emits SHT_REL section '.llvm.call-graph-profile' # when profile optimization is applied. gen-hyprel does not support SHT_REL and # causes a build failure. Remove profile optimization flags. diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-init.S b/arch/arm64/kvm/hyp/nvhe/hyp-init.S index 5a15737b4233..33fb5732ab83 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-init.S +++ b/arch/arm64/kvm/hyp/nvhe/hyp-init.S @@ -5,6 +5,7 @@ */ #include +#include #include #include @@ -268,8 +269,11 @@ SYM_CODE_END(__kvm_handle_stub_hvc) /* * void __pkvm_init_switch_pgd(struct kvm_nvhe_init_params *params, * void (*finalize_fn)(void)); + * + * SYM_TYPED_FUNC_START() allows C to call this ID-mapped function indirectly + * using a physical pointer without triggering a kCFI failure. */ -SYM_FUNC_START(__pkvm_init_switch_pgd) +SYM_TYPED_FUNC_START(__pkvm_init_switch_pgd) /* Load the inputs from the VA pointer before turning the MMU off */ ldr x5, [x0, #NVHE_INIT_PGD_PA] ldr x0, [x0, #NVHE_INIT_STACK_HYP_VA] From patchwork Fri May 10 11:26:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661419 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 083AE17A924 for ; Fri, 10 May 2024 11:27:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.208.74 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340469; cv=none; b=URbm/syzhJ+/uvCK+kJafftK688QHG7nJrU6cqVlgCGc2MUvCWStUwe60cB76n/32Xd0Vdlx4zIo767DbsHdPOW7Qdy+2HYrj5K++y2T0jJHBqonxmpUPN8NpiXdegWviM24vtLRJrxQ42OnVC9IdIi/y6VBwFLT1fszEI8tVIo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340469; c=relaxed/simple; bh=eUzR7xK0G/IZ2Tzlbr0ebewMNKwloXKEQrAq+oA2ljE=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=Hvk0mN+TIz0+tqy3mxH7m6BXYUe+p3uKKp5laNpFUQ0M1rsTXj1inLKdKk5Vu2qcIZFzjpKdx9DvVCAFoet9CnlNkTnpmz8ZYDXqpTDqOKE6nrhHrGtkG/UjUrgZIT/oa7frxVIDE3b5OTgtAwA0NzAkqPfDe/fwiUJwgSozZ24= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=MFBXg486; arc=none smtp.client-ip=209.85.208.74 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="MFBXg486" Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-56c1ac93679so1497870a12.2 for ; Fri, 10 May 2024 04:27:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340464; x=1715945264; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=n1ymwJj0J6YIOfbRVTRhLBZ520J+M8M3fYhLYcu4SBQ=; b=MFBXg4863h/u5iicr+EvIBVmP0srBtONKv6oup13Xi1UHWlzTBS4Rel2liuPRtr0ho H8VdTgYa9Iv51dcittW9p9yFRSLQGbOid9R8Rg2k9FoxOfylBbFHhFzCGq2FguMWTCTf kJqINN98UW/rjm5xVOcvKcZx5Q9KgeKj1GhcomsTbC0xeK2kz+BIdjQAapjrhaJAAoUT Flwp4d6zemok70Y4MUwNgj7OKdfEO7G6KGii2mez6Wpb4J5zaOVpBGEjgVimhJAM7kk7 GyBVQtQsBun1gb1KML3WH9N6iGLGqouB87tlD8i0UiLF9hNVFGp/2KL7X8sCwlcTQaCx 740Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340464; x=1715945264; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=n1ymwJj0J6YIOfbRVTRhLBZ520J+M8M3fYhLYcu4SBQ=; b=fqDDzxIsPA9xIfG5/vL+bMTl7jHT1CD+HdmXxdHyaMWq0t5zNGCl/R0/6kcJdwAT2Y VAqnyCewKI77VupkL8dcM/cGE8mLRkOy+0/2ns0PYpSYKzpgyCArwYb8CV2hGahOC8qm q5BA58mZs++Kn1WnIjJmOOmccphTupVhTzircd9IZGVPUzCxKtndraZgCBlVs2pKOyBW gzDQGCqW+ZDu9MDCR/f1LYoEOJGU8VCZnw9FI/nC/jRznAhI4UXQYQLjuNmg137XFd8D ofQl9qyPDXfMj3XyW031Y3jG6ZFAL+yt4rnQgrvae9jGmqrM7ggU/+J1LCF+XHdaB04y EV+A== X-Forwarded-Encrypted: i=1; AJvYcCXbdIRVo5/1h5vRvQXAuW9HZcoKRVi75q/j5I92GyOxEZEx2SNNNtq4iq218DAwD7OI5Gk7lCXX5SJiXn1ipcOrLXgD X-Gm-Message-State: AOJu0YxPL8nQjtjqIH3IkICc8qbKw+t8ir+lOKhNekYFJSvhW/YYDLVg +9/z2L21G9hiBfmgF+jqNMYReCDdflgD82lMwOknWUBR8Fgpw8ITP11GmsdxXECc2Nahfjo3WQ= = X-Google-Smtp-Source: AGHT+IH0RrjjfZ++WGHYEq92ODwn6HK5Otkgd5vcXO+mNguBfs2v+NnqgUGPVhO4xg5oozparmcLaLbhaQ== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a17:907:940f:b0:a59:cafe:85b6 with SMTP id a640c23a62f3a-a5a2d53fafbmr224066b.2.1715340464190; Fri, 10 May 2024 04:27:44 -0700 (PDT) Date: Fri, 10 May 2024 12:26:40 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-12-ptosi@google.com> Subject: [PATCH v3 11/12] KVM: arm64: nVHE: Support test module for hyp kCFI From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort Extend support for the kCFI test module to nVHE by replicating the hooks on the KVM_RUN handler path currently existing in VHE in the nVHE code, exporting the equivalent callback targets for triggering built-in hyp kCFI faults, and exposing a new CONFIG_HYP_CFI_TEST-only host HVC to implement callback registration. Update the test module to register the nVHE equivalent callback for test case '1' (i.e. both EL2 hyp caller and callee are built-in) and document that other cases are not supported outside of VHE, as they require EL2 symbols in the module, which is not currently supported for nVHE. Note that a kernel in protected mode that doesn't support HYP_CFI_TEST will prevent the module from registering nVHE callbacks both by not exporting the necessary symbols (similar to VHE) but also by rejecting the corresponding HVC, if the module tries to issue it directly. Also note that the test module will run in pKVM (with HYP_CFI_TEST) independently of other debug Kconfig flags but that not stacktrace will be printed without PROTECTED_NVHE_STACKTRACE. This allows testing kCFI under conditions closer to release builds, if desired. Signed-off-by: Pierre-Clément Tosi --- arch/arm64/include/asm/kvm_asm.h | 3 ++ arch/arm64/include/asm/kvm_cfi.h | 6 ++-- arch/arm64/kvm/Kconfig | 2 -- arch/arm64/kvm/hyp/{vhe => }/cfi.c | 0 arch/arm64/kvm/hyp/nvhe/Makefile | 1 + arch/arm64/kvm/hyp/nvhe/hyp-main.c | 19 ++++++++++++ arch/arm64/kvm/hyp/nvhe/switch.c | 7 +++++ arch/arm64/kvm/hyp/vhe/Makefile | 2 +- arch/arm64/kvm/hyp_cfi_test.c | 44 ++++++++++++++++++++++++---- arch/arm64/kvm/hyp_cfi_test_module.c | 24 ++++++++------- 10 files changed, 86 insertions(+), 22 deletions(-) rename arch/arm64/kvm/hyp/{vhe => }/cfi.c (100%) diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h index 24b5e6b23417..3256c91ff234 100644 --- a/arch/arm64/include/asm/kvm_asm.h +++ b/arch/arm64/include/asm/kvm_asm.h @@ -81,6 +81,9 @@ enum __kvm_host_smccc_func { __KVM_HOST_SMCCC_FUNC___pkvm_init_vm, __KVM_HOST_SMCCC_FUNC___pkvm_init_vcpu, __KVM_HOST_SMCCC_FUNC___pkvm_teardown_vm, +#ifdef CONFIG_HYP_SUPPORTS_CFI_TEST + __KVM_HOST_SMCCC_FUNC___kvm_register_cfi_test_cb, +#endif }; #define DECLARE_KVM_VHE_SYM(sym) extern char sym[] diff --git a/arch/arm64/include/asm/kvm_cfi.h b/arch/arm64/include/asm/kvm_cfi.h index 13cc7b19d838..ed6422eebce5 100644 --- a/arch/arm64/include/asm/kvm_cfi.h +++ b/arch/arm64/include/asm/kvm_cfi.h @@ -12,8 +12,8 @@ #ifdef CONFIG_HYP_SUPPORTS_CFI_TEST -int kvm_cfi_test_register_host_ctxt_cb(void (*cb)(void)); -int kvm_cfi_test_register_guest_ctxt_cb(void (*cb)(void)); +int kvm_cfi_test_register_host_ctxt_cb(void (*vhe_cb)(void), void *nvhe_cb); +int kvm_cfi_test_register_guest_ctxt_cb(void (*vhe_cb)(void), void *nvhe_cb); #else @@ -31,6 +31,8 @@ static inline int kvm_cfi_test_register_guest_ctxt_cb(void (*cb)(void)) /* Symbols which the host can register as hyp callbacks; see . */ void hyp_trigger_builtin_cfi_fault(void); +DECLARE_KVM_NVHE_SYM(hyp_trigger_builtin_cfi_fault); void hyp_builtin_cfi_fault_target(int unused); +DECLARE_KVM_NVHE_SYM(hyp_builtin_cfi_fault_target); #endif /* __ARM64_KVM_CFI_H__ */ diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 5daa8079a120..715c85088c06 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -75,8 +75,6 @@ config HYP_CFI_TEST Say M here to also build a module which registers callbacks triggering faults and selected by userspace through its parameters. - Note that this feature is currently only supported in VHE mode. - If unsure, say N. config HYP_SUPPORTS_CFI_TEST diff --git a/arch/arm64/kvm/hyp/vhe/cfi.c b/arch/arm64/kvm/hyp/cfi.c similarity index 100% rename from arch/arm64/kvm/hyp/vhe/cfi.c rename to arch/arm64/kvm/hyp/cfi.c diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile index 2eb915d8943f..09039d351726 100644 --- a/arch/arm64/kvm/hyp/nvhe/Makefile +++ b/arch/arm64/kvm/hyp/nvhe/Makefile @@ -25,6 +25,7 @@ hyp-obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o host.o cache.o setup.o mm.o mem_protect.o sys_regs.o pkvm.o stacktrace.o ffa.o hyp-obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o ../pgtable.o +hyp-obj-$(CONFIG_HYP_SUPPORTS_CFI_TEST) += ../cfi.o hyp-obj-$(CONFIG_LIST_HARDENED) += list_debug.o hyp-obj-y += $(lib-objs) diff --git a/arch/arm64/kvm/hyp/nvhe/hyp-main.c b/arch/arm64/kvm/hyp/nvhe/hyp-main.c index 2385fd03ed87..431860e8a98d 100644 --- a/arch/arm64/kvm/hyp/nvhe/hyp-main.c +++ b/arch/arm64/kvm/hyp/nvhe/hyp-main.c @@ -5,6 +5,7 @@ */ #include +#include #include #include @@ -13,6 +14,8 @@ #include #include +#include + #include #include #include @@ -314,6 +317,19 @@ static void handle___pkvm_teardown_vm(struct kvm_cpu_context *host_ctxt) cpu_reg(host_ctxt, 1) = __pkvm_teardown_vm(handle); } +#ifndef CONFIG_HYP_SUPPORTS_CFI_TEST +__always_unused +#endif +static void handle___kvm_register_cfi_test_cb(struct kvm_cpu_context *host_ctxt) +{ + DECLARE_REG(phys_addr_t, cb_phys, host_ctxt, 1); + DECLARE_REG(bool, in_host_ctxt, host_ctxt, 2); + + void (*cb)(void) = cb_phys ? __hyp_va(cb_phys) : NULL; + + cpu_reg(host_ctxt, 1) = __kvm_register_cfi_test_cb(cb, in_host_ctxt); +} + typedef void (*hcall_t)(struct kvm_cpu_context *); #define HANDLE_FUNC(x) [__KVM_HOST_SMCCC_FUNC_##x] = (hcall_t)handle_##x @@ -348,6 +364,9 @@ static const hcall_t host_hcall[] = { HANDLE_FUNC(__pkvm_init_vm), HANDLE_FUNC(__pkvm_init_vcpu), HANDLE_FUNC(__pkvm_teardown_vm), +#ifdef CONFIG_HYP_SUPPORTS_CFI_TEST + HANDLE_FUNC(__kvm_register_cfi_test_cb), +#endif }; static void handle_host_hcall(struct kvm_cpu_context *host_ctxt) diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c index c50f8459e4fc..160311bf367b 100644 --- a/arch/arm64/kvm/hyp/nvhe/switch.c +++ b/arch/arm64/kvm/hyp/nvhe/switch.c @@ -4,6 +4,7 @@ * Author: Marc Zyngier */ +#include #include #include @@ -253,6 +254,9 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) bool pmu_switch_needed; u64 exit_code; + if (IS_ENABLED(CONFIG_HYP_SUPPORTS_CFI_TEST) && unlikely(hyp_test_host_ctxt_cfi)) + hyp_test_host_ctxt_cfi(); + /* * Having IRQs masked via PMR when entering the guest means the GIC * will not signal the CPU of interrupts of lower priority, and the @@ -313,6 +317,9 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) __debug_switch_to_guest(vcpu); + if (IS_ENABLED(CONFIG_HYP_SUPPORTS_CFI_TEST) && unlikely(hyp_test_guest_ctxt_cfi)) + hyp_test_guest_ctxt_cfi(); + do { /* Jump in the fire! */ exit_code = __guest_enter(vcpu); diff --git a/arch/arm64/kvm/hyp/vhe/Makefile b/arch/arm64/kvm/hyp/vhe/Makefile index 19ca584cc21e..951c8c00a685 100644 --- a/arch/arm64/kvm/hyp/vhe/Makefile +++ b/arch/arm64/kvm/hyp/vhe/Makefile @@ -9,4 +9,4 @@ ccflags-y := -D__KVM_VHE_HYPERVISOR__ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \ ../fpsimd.o ../hyp-entry.o ../exception.o -obj-$(CONFIG_HYP_SUPPORTS_CFI_TEST) += cfi.o +obj-$(CONFIG_HYP_SUPPORTS_CFI_TEST) += ../cfi.o diff --git a/arch/arm64/kvm/hyp_cfi_test.c b/arch/arm64/kvm/hyp_cfi_test.c index da7b25ca1b1f..6a02b43c45f6 100644 --- a/arch/arm64/kvm/hyp_cfi_test.c +++ b/arch/arm64/kvm/hyp_cfi_test.c @@ -6,6 +6,7 @@ #include #include #include +#include #include #include @@ -15,29 +16,60 @@ /* For calling directly into the VHE hypervisor; see . */ int __kvm_register_cfi_test_cb(void (*)(void), bool); -static int kvm_register_cfi_test_cb(void (*vhe_cb)(void), bool in_host_ctxt) +static int kvm_register_nvhe_cfi_test_cb(void *cb, bool in_host_ctxt) +{ + extern void *kvm_nvhe_sym(hyp_test_host_ctxt_cfi); + extern void *kvm_nvhe_sym(hyp_test_guest_ctxt_cfi); + + if (is_protected_kvm_enabled()) { + phys_addr_t cb_phys = cb ? virt_to_phys(cb) : 0; + + /* Use HVC as only the hyp can modify its callback pointers. */ + return kvm_call_hyp_nvhe(__kvm_register_cfi_test_cb, cb_phys, + in_host_ctxt); + } + + /* + * In non-protected nVHE, the pKVM HVC is not available but the + * hyp callback pointers can be accessed and modified directly. + */ + if (cb) + cb = kern_hyp_va(kvm_ksym_ref(cb)); + + if (in_host_ctxt) + kvm_nvhe_sym(hyp_test_host_ctxt_cfi) = cb; + else + kvm_nvhe_sym(hyp_test_guest_ctxt_cfi) = cb; + + return 0; +} + +static int kvm_register_cfi_test_cb(void (*vhe_cb)(void), void *nvhe_cb, + bool in_host_ctxt) { if (!is_hyp_mode_available()) return -ENXIO; if (is_hyp_nvhe()) - return -EOPNOTSUPP; + return kvm_register_nvhe_cfi_test_cb(nvhe_cb, in_host_ctxt); return __kvm_register_cfi_test_cb(vhe_cb, in_host_ctxt); } -int kvm_cfi_test_register_host_ctxt_cb(void (*cb)(void)) +int kvm_cfi_test_register_host_ctxt_cb(void (*vhe_cb)(void), void *nvhe_cb) { - return kvm_register_cfi_test_cb(cb, true); + return kvm_register_cfi_test_cb(vhe_cb, nvhe_cb, true); } EXPORT_SYMBOL(kvm_cfi_test_register_host_ctxt_cb); -int kvm_cfi_test_register_guest_ctxt_cb(void (*cb)(void)) +int kvm_cfi_test_register_guest_ctxt_cb(void (*vhe_cb)(void), void *nvhe_cb) { - return kvm_register_cfi_test_cb(cb, false); + return kvm_register_cfi_test_cb(vhe_cb, nvhe_cb, false); } EXPORT_SYMBOL(kvm_cfi_test_register_guest_ctxt_cb); /* Hypervisor callbacks for the test module to register. */ EXPORT_SYMBOL(hyp_trigger_builtin_cfi_fault); +EXPORT_SYMBOL(kvm_nvhe_sym(hyp_trigger_builtin_cfi_fault)); EXPORT_SYMBOL(hyp_builtin_cfi_fault_target); +EXPORT_SYMBOL(kvm_nvhe_sym(hyp_builtin_cfi_fault_target)); diff --git a/arch/arm64/kvm/hyp_cfi_test_module.c b/arch/arm64/kvm/hyp_cfi_test_module.c index eeda4be4d3ef..63a5e99cb164 100644 --- a/arch/arm64/kvm/hyp_cfi_test_module.c +++ b/arch/arm64/kvm/hyp_cfi_test_module.c @@ -20,9 +20,9 @@ static int set_guest_mode(const char *val, const struct kernel_param *kp); #define M_DESC \ "\n\t0: none" \ "\n\t1: built-in caller & built-in callee" \ - "\n\t2: built-in caller & module callee" \ - "\n\t3: module caller & built-in callee" \ - "\n\t4: module caller & module callee" + "\n\t2: built-in caller & module callee (VHE only)" \ + "\n\t3: module caller & built-in callee (VHE only)" \ + "\n\t4: module caller & module callee (VHE only)" static unsigned int host_mode; module_param_call(host, set_host_mode, param_get_uint, &host_mode, 0644); @@ -40,7 +40,7 @@ static void hyp_cfi_module2module_test_target(int); static void hyp_cfi_builtin2module_test_target(int); static int set_param_mode(const char *val, const struct kernel_param *kp, - int (*register_cb)(void (*)(void))) + int (*register_cb)(void (*)(void), void *)) { unsigned int *mode = kp->arg; int err; @@ -51,15 +51,17 @@ static int set_param_mode(const char *val, const struct kernel_param *kp, switch (*mode) { case 0: - return register_cb(NULL); + return register_cb(NULL, NULL); case 1: - return register_cb(hyp_trigger_builtin_cfi_fault); + return register_cb(hyp_trigger_builtin_cfi_fault, + kvm_nvhe_sym(hyp_trigger_builtin_cfi_fault)); case 2: - return register_cb((void *)hyp_cfi_builtin2module_test_target); + return register_cb((void *)hyp_cfi_builtin2module_test_target, + NULL); case 3: - return register_cb(trigger_module2builtin_cfi_fault); + return register_cb(trigger_module2builtin_cfi_fault, NULL); case 4: - return register_cb(trigger_module2module_cfi_fault); + return register_cb(trigger_module2module_cfi_fault, NULL); default: return -EINVAL; } @@ -79,11 +81,11 @@ static void __exit exit_hyp_cfi_test(void) { int err; - err = kvm_cfi_test_register_host_ctxt_cb(NULL); + err = kvm_cfi_test_register_host_ctxt_cb(NULL, NULL); if (err) pr_err("Failed to unregister host context trigger: %d\n", err); - err = kvm_cfi_test_register_guest_ctxt_cb(NULL); + err = kvm_cfi_test_register_guest_ctxt_cb(NULL, NULL); if (err) pr_err("Failed to unregister guest context trigger: %d\n", err); } From patchwork Fri May 10 11:26:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= X-Patchwork-Id: 13661418 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A42C6170859 for ; Fri, 10 May 2024 11:27:47 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340469; cv=none; b=d0ncJzA5/2ieeTnDa2FkKabyIaDYH0l5n5FCYaEOH6N3n0NmSzO7Zg1xPPMzpzGUBPTWbSeo5VNW+p/C+Vaf1lLn2oAZT9V2x1s7XIrBXlIgZuyn37hsHUEqXHJinFtEz2X4SB3ZY8qvNrhz/VsQ3UKZ9QngC0I7TStqdga5hdQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715340469; c=relaxed/simple; bh=olch/zFkSndigLQVwr9bYRtHWZEBGl9YaT4pV77/LIY=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=ItvTUlxVnqdkoinbmPhGkXILS5/2lVkz5RRAFzqpZGvQ3azfySfeCxHB8TWbohEHpULxBxF4Dt8fx6M3sPmqGn+wbqArTu0vx1ig5zsAHrZDBj08lhYbbh1jfjBJqrCxtkYG2WAfcYw/twV9IbDrMzKs8GyC9HE9AY01+eE3uF8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=cOa1sh6+; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--ptosi.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="cOa1sh6+" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-6207c483342so31802267b3.1 for ; Fri, 10 May 2024 04:27:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1715340466; x=1715945266; darn=vger.kernel.org; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:from:to:cc:subject:date:message-id :reply-to; bh=5CCzdSxscfzeRlLJ5wL+3zQfVKF6xgrgjNCyKCDms4U=; b=cOa1sh6+6sdyfqZUEn6sthXgPCOTrVLww6xHQpTeHoXLsE/Ky5zSn+R/AXLgFLju0K VKMzd61JpnoQYxYcWA09vLTfzQPJHoUfSei5HQSXDAFZJincnNMjZmBJzuZ2WDZAU11Z HNuZxPrmhyPY3Lfon/Xs/knioUgb3CVwIxwqGrlBx1O9apLVD5v98a8QLhMsK/xLiMbs +/4CCkkjMCn4kKi0ReR5rd1Kmle+y260VZj1EnGWsatfTGbK6j1vd1VsVQIbA48MBETO Ttxfg1vq0s6QPYgJ8EJF9LcL60ucKYpJzdgBpOZ1Pcup7tCaVM55W9Y08h6I01A/yEcp 7FHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715340466; x=1715945266; h=content-transfer-encoding:cc:to:from:subject:message-id:references :mime-version:in-reply-to:date:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=5CCzdSxscfzeRlLJ5wL+3zQfVKF6xgrgjNCyKCDms4U=; b=Y9/ghgP6arqTEG9ch+3lTcwUx1DVlsZHC0dmrl3ENfxOGkdXecBIGNpw6yITAv+Xuv TXKfwqOVvJ1SQjsMMVmIhdKQz7FABjwoaRbrNUaN3JBeTtQhJ6PUthcmLHT0bOK3TuO/ CkrkEexfrd0HyukwuObaboe+74gigwlUM1xVdv27MmmEQtfySaX+Nf4M+SsqLG2Xm2bg SJ1UIWUxV3ckPGEOsnvJOaaM7R63zZ7aSJ4Nci2IBWIga6qThG8Uup5MeCkIpzPo+408 dyRRo2Oo2LM0UpGBf2NddCYgNiSMVIbM2qkRRag5IQtfQaf5Rfpw+qQCAsTG2vtYvJdw UqJQ== X-Forwarded-Encrypted: i=1; AJvYcCV3i+E3cZPyj5/RXyMfj1nJzyo9ypni9D1qedxZnl6U/do0fN2YIleeoHPeEMbJaucoOnqh7PBupbxigmyeBKTErZZ1 X-Gm-Message-State: AOJu0YxqNCaG8PqHTZ0kao1x5v/AQbTf8QF1hre1AL5U1GYvjYOm30mK W1eHmEbm57Me5+btz0q05BCmxRIh7uCfcPPbI9qU0DQYD5/7BzNiRucpTtCvsd89uvhjGUniCA= = X-Google-Smtp-Source: AGHT+IHdS3ei2ZFY3/jZi7UOWajuQFEcULFtMW4+dDg7tAnI+MMeD/3xdux6nZQkCMsm0kZ06aqSsHCV0w== X-Received: from ptosi.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:11ec]) (user=ptosi job=sendgmr) by 2002:a05:690c:c9d:b0:618:8e4b:f49d with SMTP id 00721157ae682-622aff868c9mr6454537b3.4.1715340466673; Fri, 10 May 2024 04:27:46 -0700 (PDT) Date: Fri, 10 May 2024 12:26:41 +0100 In-Reply-To: <20240510112645.3625702-1-ptosi@google.com> Precedence: bulk X-Mailing-List: kvm@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240510112645.3625702-1-ptosi@google.com> X-Mailer: git-send-email 2.45.0.118.g7fe29c98d7-goog Message-ID: <20240510112645.3625702-13-ptosi@google.com> Subject: [PATCH v3 12/12] KVM: arm64: Improve CONFIG_CFI_CLANG error message From: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " To: kvmarm@lists.linux.dev, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org Cc: " =?utf-8?q?Pierre-Cl=C3=A9ment_Tosi?= " , Marc Zyngier , Oliver Upton , Suzuki K Poulose , Vincent Donnefort For kCFI, the compiler encodes in the immediate of the BRK (which the CPU places in ESR_ELx) the indices of the two registers it used to hold (resp.) the function pointer and expected type. Therefore, the kCFI handler must be able to parse the contents of the register file at the point where the exception was triggered. To achieve this, introduce a new hypervisor panic path that first stores the CPU context in the per-CPU kvm_hyp_ctxt before calling (directly or indirectly) hyp_panic() and execute it from all EL2 synchronous exception handlers i.e. - call it directly in host_el2_sync_vect (__kvm_hyp_host_vector, EL2t&h) - call it directly in el2t_sync_invalid (__kvm_hyp_vector, EL2t) - set ELR_EL2 to it in el2_sync (__kvm_hyp_vector, EL2h), which ERETs Teach hyp_panic() to decode the kCFI ESR and extract the target and type from the saved CPU context. In VHE, use that information to panic() with a specialized error message. In nVHE, only report it if the host (EL1) has access to the saved CPU context i.e. iff CONFIG_NVHE_EL2_DEBUG=y, which aligns with the behavior of CONFIG_PROTECTED_NVHE_STACKTRACE. Signed-off-by: Pierre-Clément Tosi --- arch/arm64/kvm/handle_exit.c | 30 +++++++++++++++++++++++-- arch/arm64/kvm/hyp/entry.S | 24 +++++++++++++++++++- arch/arm64/kvm/hyp/hyp-entry.S | 2 +- arch/arm64/kvm/hyp/include/hyp/switch.h | 4 ++-- arch/arm64/kvm/hyp/nvhe/host.S | 2 +- arch/arm64/kvm/hyp/vhe/switch.c | 26 +++++++++++++++++++-- 6 files changed, 79 insertions(+), 9 deletions(-) diff --git a/arch/arm64/kvm/handle_exit.c b/arch/arm64/kvm/handle_exit.c index 0db23a6304ce..d76e41a07df1 100644 --- a/arch/arm64/kvm/handle_exit.c +++ b/arch/arm64/kvm/handle_exit.c @@ -26,6 +26,8 @@ #define CREATE_TRACE_POINTS #include "trace_handle_exit.h" +DECLARE_KVM_NVHE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); + typedef int (*exit_handle_fn)(struct kvm_vcpu *); static void kvm_handle_guest_serror(struct kvm_vcpu *vcpu, u64 esr) @@ -383,11 +385,35 @@ void handle_exit_early(struct kvm_vcpu *vcpu, int exception_index) kvm_handle_guest_serror(vcpu, kvm_vcpu_get_esr(vcpu)); } -static void kvm_nvhe_report_cfi_failure(u64 panic_addr) +static void kvm_nvhe_report_cfi_target(struct user_pt_regs *regs, u64 esr, + u64 hyp_offset) +{ + u64 va_mask = GENMASK_ULL(vabits_actual - 1, 0); + u8 type_idx = FIELD_GET(CFI_BRK_IMM_TYPE, esr); + u8 target_idx = FIELD_GET(CFI_BRK_IMM_TARGET, esr); + u32 expected_type = (u32)regs->regs[type_idx]; + u64 target_addr = (regs->regs[target_idx] & va_mask) + hyp_offset; + + kvm_err(" (target: [<%016llx>] %ps, expected type: 0x%08x)\n", + target_addr, (void *)(target_addr + kaslr_offset()), + expected_type); +} + +static void kvm_nvhe_report_cfi_failure(u64 panic_addr, u64 esr, u64 hyp_offset) { + struct user_pt_regs *regs = NULL; + kvm_err("nVHE hyp CFI failure at: [<%016llx>] %pB!\n", panic_addr, (void *)(panic_addr + kaslr_offset())); + if (IS_ENABLED(CONFIG_NVHE_EL2_DEBUG) || !is_protected_kvm_enabled()) + regs = &this_cpu_ptr_nvhe_sym(kvm_hyp_ctxt)->regs; + + if (regs) + kvm_nvhe_report_cfi_target(regs, esr, hyp_offset); + else + kvm_err(" (no target information: !CONFIG_NVHE_EL2_DEBUG)\n"); + if (IS_ENABLED(CONFIG_CFI_PERMISSIVE)) kvm_err(" (CONFIG_CFI_PERMISSIVE ignored for hyp failures)\n"); } @@ -423,7 +449,7 @@ void __noreturn __cold nvhe_hyp_panic_handler(u64 esr, u64 spsr, kvm_err("nVHE hyp BUG at: [<%016llx>] %pB!\n", panic_addr, (void *)(panic_addr + kaslr_offset())); } else if (IS_ENABLED(CONFIG_CFI_CLANG) && esr_is_cfi_brk(esr)) { - kvm_nvhe_report_cfi_failure(panic_addr); + kvm_nvhe_report_cfi_failure(panic_addr, esr, hyp_offset); } else { kvm_err("nVHE hyp panic at: [<%016llx>] %pB!\n", panic_addr, (void *)(panic_addr + kaslr_offset())); diff --git a/arch/arm64/kvm/hyp/entry.S b/arch/arm64/kvm/hyp/entry.S index 6a1ce9d21e5b..8838b453b9be 100644 --- a/arch/arm64/kvm/hyp/entry.S +++ b/arch/arm64/kvm/hyp/entry.S @@ -83,7 +83,7 @@ alternative_else_nop_endif eret sb -SYM_INNER_LABEL(__hyp_restore_elr_and_panic, SYM_L_GLOBAL) +SYM_INNER_LABEL(__hyp_restore_elr_save_context_and_panic, SYM_L_GLOBAL) // x0-x29,lr: hyp regs stp x0, x1, [sp, #-16]! @@ -92,6 +92,28 @@ SYM_INNER_LABEL(__hyp_restore_elr_and_panic, SYM_L_GLOBAL) msr elr_el2, x0 ldp x0, x1, [sp], #16 +SYM_INNER_LABEL(__hyp_save_context_and_panic, SYM_L_GLOBAL) + // x0-x29,lr: hyp regs + + stp x0, x1, [sp, #-16]! + + adr_this_cpu x0, kvm_hyp_ctxt, x1 + + stp x2, x3, [x0, #CPU_XREG_OFFSET(2)] + + ldp x2, x3, [sp], #16 + + stp x2, x3, [x0, #CPU_XREG_OFFSET(0)] + stp x4, x5, [x0, #CPU_XREG_OFFSET(4)] + stp x6, x7, [x0, #CPU_XREG_OFFSET(6)] + stp x8, x9, [x0, #CPU_XREG_OFFSET(8)] + stp x10, x11, [x0, #CPU_XREG_OFFSET(10)] + stp x12, x13, [x0, #CPU_XREG_OFFSET(12)] + stp x14, x15, [x0, #CPU_XREG_OFFSET(14)] + stp x16, x17, [x0, #CPU_XREG_OFFSET(16)] + + save_callee_saved_regs x0 + SYM_INNER_LABEL(__hyp_panic, SYM_L_GLOBAL) // x0-x29,lr: vcpu regs diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S index 7e65ef738ec9..d0d90d598338 100644 --- a/arch/arm64/kvm/hyp/hyp-entry.S +++ b/arch/arm64/kvm/hyp/hyp-entry.S @@ -130,7 +130,7 @@ SYM_CODE_END(\label) .endm /* None of these should ever happen */ - invalid_vector el2t_sync_invalid + invalid_vector el2t_sync_invalid, __hyp_save_context_and_panic invalid_vector el2t_irq_invalid invalid_vector el2t_fiq_invalid invalid_vector el2t_error_invalid diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h index 9387e3a0b680..f3d8fbc7a77b 100644 --- a/arch/arm64/kvm/hyp/include/hyp/switch.h +++ b/arch/arm64/kvm/hyp/include/hyp/switch.h @@ -753,7 +753,7 @@ static inline bool fixup_guest_exit(struct kvm_vcpu *vcpu, u64 *exit_code) static inline void __kvm_unexpected_el2_exception(void) { - extern char __hyp_restore_elr_and_panic[]; + extern char __hyp_restore_elr_save_context_and_panic[]; unsigned long addr, fixup; struct kvm_exception_table_entry *entry, *end; unsigned long elr_el2 = read_sysreg(elr_el2); @@ -776,7 +776,7 @@ static inline void __kvm_unexpected_el2_exception(void) /* Trigger a panic after restoring the hyp context. */ this_cpu_ptr(&kvm_hyp_ctxt)->sys_regs[ELR_EL2] = elr_el2; - write_sysreg(__hyp_restore_elr_and_panic, elr_el2); + write_sysreg(__hyp_restore_elr_save_context_and_panic, elr_el2); } #endif /* __ARM64_KVM_HYP_SWITCH_H__ */ diff --git a/arch/arm64/kvm/hyp/nvhe/host.S b/arch/arm64/kvm/hyp/nvhe/host.S index 0613b6e35137..ec3e4f5c28cc 100644 --- a/arch/arm64/kvm/hyp/nvhe/host.S +++ b/arch/arm64/kvm/hyp/nvhe/host.S @@ -213,7 +213,7 @@ SYM_FUNC_END(__host_hvc) .endm .macro host_el2_sync_vect - __host_el2_vect __hyp_panic + __host_el2_vect __hyp_save_context_and_panic .endm .macro invalid_host_el1_vect diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c index b3268933b093..17df57580c77 100644 --- a/arch/arm64/kvm/hyp/vhe/switch.c +++ b/arch/arm64/kvm/hyp/vhe/switch.c @@ -18,6 +18,7 @@ #include #include +#include #include #include #include @@ -308,7 +309,24 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu) return ret; } -static void __noreturn __hyp_call_panic(u64 spsr, u64 elr, u64 par) +static void __noreturn __hyp_call_panic_for_cfi(u64 elr, u64 esr) +{ + struct user_pt_regs *regs = &this_cpu_ptr(&kvm_hyp_ctxt)->regs; + u8 type_idx = FIELD_GET(CFI_BRK_IMM_TYPE, esr); + u8 target_idx = FIELD_GET(CFI_BRK_IMM_TARGET, esr); + u32 expected_type = (u32)regs->regs[type_idx]; + u64 target = regs->regs[target_idx]; + + panic("VHE hyp CFI failure at: [<%016llx>] %pB (target: [<%016llx>] %ps, expected type: 0x%08x)\n" +#ifdef CONFIG_CFI_PERMISSIVE + " (CONFIG_CFI_PERMISSIVE ignored for hyp failures)\n" +#endif + , + elr, (void *)elr, target, (void *)target, expected_type); +} +NOKPROBE_SYMBOL(__hyp_call_panic_for_cfi); + +static void __noreturn __hyp_call_panic(u64 spsr, u64 elr, u64 par, u64 esr) { struct kvm_cpu_context *host_ctxt; struct kvm_vcpu *vcpu; @@ -319,6 +337,9 @@ static void __noreturn __hyp_call_panic(u64 spsr, u64 elr, u64 par) __deactivate_traps(vcpu); sysreg_restore_host_state_vhe(host_ctxt); + if (IS_ENABLED(CONFIG_CFI_CLANG) && esr_is_cfi_brk(esr)) + __hyp_call_panic_for_cfi(elr, esr); + panic("HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n", spsr, elr, read_sysreg_el2(SYS_ESR), read_sysreg_el2(SYS_FAR), @@ -331,8 +352,9 @@ void __noreturn hyp_panic(void) u64 spsr = read_sysreg_el2(SYS_SPSR); u64 elr = read_sysreg_el2(SYS_ELR); u64 par = read_sysreg_par(); + u64 esr = read_sysreg_el2(SYS_ESR); - __hyp_call_panic(spsr, elr, par); + __hyp_call_panic(spsr, elr, par, esr); } asmlinkage void kvm_unexpected_el2_exception(void)