| Message ID | 20230704193243.3300506-1-sudeep.holla@arm.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | KVM: arm64: Handle kvm_arm_init failure correctly in finalize_pkvm | expand |
On Tue, 04 Jul 2023 20:32:43 +0100, Sudeep Holla <sudeep.holla@arm.com> wrote: > > Currently there is no synchronisation between finalize_pkvm() and > kvm_arm_init() initcalls. The finalize_pkvm() proceeds happily even if > kvm_arm_init() fails resulting in the following warning on all the CPUs > and eventually a HYP panic: > > | kvm [1]: IPA Size Limit: 48 bits > | kvm [1]: Failed to init hyp memory protection > | kvm [1]: error initializing Hyp mode: -22 > | > | <snip> > | > | WARNING: CPU: 0 PID: 0 at arch/arm64/kvm/pkvm.c:226 _kvm_host_prot_finalize+0x30/0x50 > | Modules linked in: > | CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0 #237 > | Hardware name: FVP Base RevC (DT) > | pstate: 634020c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) > | pc : _kvm_host_prot_finalize+0x30/0x50 > | lr : __flush_smp_call_function_queue+0xd8/0x230 > | > | Call trace: > | _kvm_host_prot_finalize+0x3c/0x50 > | on_each_cpu_cond_mask+0x3c/0x6c > | pkvm_drop_host_privileges+0x4c/0x78 > | finalize_pkvm+0x3c/0x5c > | do_one_initcall+0xcc/0x240 > | do_initcall_level+0x8c/0xac > | do_initcalls+0x54/0x94 > | do_basic_setup+0x1c/0x28 > | kernel_init_freeable+0x100/0x16c > | kernel_init+0x20/0x1a0 > | ret_from_fork+0x10/0x20 > | Failed to finalize Hyp protection: -22 > | dtb=fvp-base-revc.dtb > | kvm [95]: nVHE hyp BUG at: arch/arm64/kvm/hyp/nvhe/mem_protect.c:540! > | kvm [95]: nVHE call trace: > | kvm [95]: [<ffff800081052984>] __kvm_nvhe_hyp_panic+0xac/0xf8 > | kvm [95]: [<ffff800081059644>] __kvm_nvhe_handle_host_mem_abort+0x1a0/0x2ac > | kvm [95]: [<ffff80008105511c>] __kvm_nvhe_handle_trap+0x4c/0x160 > | kvm [95]: [<ffff8000810540fc>] __kvm_nvhe___skip_pauth_save+0x4/0x4 > | kvm [95]: ---[ end nVHE call trace ]--- > | kvm [95]: Hyp Offset: 0xfffe8db00ffa0000 > | Kernel panic - not syncing: HYP panic: > | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 > | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 > | VCPU:0000000000000000 > | CPU: 3 PID: 95 Comm: kworker/u16:2 Tainted: G W 6.4.0 #237 > | Hardware name: FVP Base RevC (DT) > | Workqueue: rpciod rpc_async_schedule > | Call trace: > | dump_backtrace+0xec/0x108 > | show_stack+0x18/0x2c > | dump_stack_lvl+0x50/0x68 > | dump_stack+0x18/0x24 > | panic+0x138/0x33c > | nvhe_hyp_panic_handler+0x100/0x184 > | new_slab+0x23c/0x54c > | ___slab_alloc+0x3e4/0x770 > | kmem_cache_alloc_node+0x1f0/0x278 > | __alloc_skb+0xdc/0x294 > | tcp_stream_alloc_skb+0x2c/0xf0 > | tcp_sendmsg_locked+0x3d0/0xda4 > | tcp_sendmsg+0x38/0x5c > | inet_sendmsg+0x44/0x60 > | sock_sendmsg+0x1c/0x34 > | xprt_sock_sendmsg+0xdc/0x274 > | xs_tcp_send_request+0x1ac/0x28c > | xprt_transmit+0xcc/0x300 > | call_transmit+0x78/0x90 > | __rpc_execute+0x114/0x3d8 > | rpc_async_schedule+0x28/0x48 > | process_one_work+0x1d8/0x314 > | worker_thread+0x248/0x474 > | kthread+0xfc/0x184 > | ret_from_fork+0x10/0x20 > | SMP: stopping secondary CPUs > | Kernel Offset: 0x57c5cb460000 from 0xffff800080000000 > | PHYS_OFFSET: 0x80000000 > | CPU features: 0x00000000,1035b7a3,ccfe773f > | Memory Limit: none > | ---[ end Kernel panic - not syncing: HYP panic: > | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 > | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 > | VCPU:0000000000000000 ]--- > > Fix it by checking for the successfull initialisation of kvm_arm_init() > in finalize_pkvm() before proceeding any futher. > > Fixes: 87727ba2bb05 ("KVM: arm64: Ensure CPU PMU probes before pKVM host de-privilege") > Cc: Will Deacon <will@kernel.org> > Cc: Marc Zyngier <maz@kernel.org> > Cc: Oliver Upton <oliver.upton@linux.dev> > Cc: James Morse <james.morse@arm.com> > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > Cc: Zenghui Yu <yuzenghui@huawei.com> > Signed-off-by: Sudeep Holla <sudeep.holla@arm.com> I really dislike the additional helper, but now that the capabilities are patched, I can't see another way to achieve this so: Acked-by: Marc Zyngier <maz@kernel.org> M.
On Tue, Jul 11, 2023 at 09:36:46AM +0100, Marc Zyngier wrote: > On Tue, 04 Jul 2023 20:32:43 +0100, > Sudeep Holla <sudeep.holla@arm.com> wrote: > > > > Currently there is no synchronisation between finalize_pkvm() and > > kvm_arm_init() initcalls. The finalize_pkvm() proceeds happily even if > > kvm_arm_init() fails resulting in the following warning on all the CPUs > > and eventually a HYP panic: > > > > | kvm [1]: IPA Size Limit: 48 bits > > | kvm [1]: Failed to init hyp memory protection > > | kvm [1]: error initializing Hyp mode: -22 > > | > > | <snip> > > | > > | WARNING: CPU: 0 PID: 0 at arch/arm64/kvm/pkvm.c:226 _kvm_host_prot_finalize+0x30/0x50 > > | Modules linked in: > > | CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0 #237 > > | Hardware name: FVP Base RevC (DT) > > | pstate: 634020c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) > > | pc : _kvm_host_prot_finalize+0x30/0x50 > > | lr : __flush_smp_call_function_queue+0xd8/0x230 > > | > > | Call trace: > > | _kvm_host_prot_finalize+0x3c/0x50 > > | on_each_cpu_cond_mask+0x3c/0x6c > > | pkvm_drop_host_privileges+0x4c/0x78 > > | finalize_pkvm+0x3c/0x5c > > | do_one_initcall+0xcc/0x240 > > | do_initcall_level+0x8c/0xac > > | do_initcalls+0x54/0x94 > > | do_basic_setup+0x1c/0x28 > > | kernel_init_freeable+0x100/0x16c > > | kernel_init+0x20/0x1a0 > > | ret_from_fork+0x10/0x20 > > | Failed to finalize Hyp protection: -22 > > | dtb=fvp-base-revc.dtb > > | kvm [95]: nVHE hyp BUG at: arch/arm64/kvm/hyp/nvhe/mem_protect.c:540! > > | kvm [95]: nVHE call trace: > > | kvm [95]: [<ffff800081052984>] __kvm_nvhe_hyp_panic+0xac/0xf8 > > | kvm [95]: [<ffff800081059644>] __kvm_nvhe_handle_host_mem_abort+0x1a0/0x2ac > > | kvm [95]: [<ffff80008105511c>] __kvm_nvhe_handle_trap+0x4c/0x160 > > | kvm [95]: [<ffff8000810540fc>] __kvm_nvhe___skip_pauth_save+0x4/0x4 > > | kvm [95]: ---[ end nVHE call trace ]--- > > | kvm [95]: Hyp Offset: 0xfffe8db00ffa0000 > > | Kernel panic - not syncing: HYP panic: > > | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 > > | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 > > | VCPU:0000000000000000 > > | CPU: 3 PID: 95 Comm: kworker/u16:2 Tainted: G W 6.4.0 #237 > > | Hardware name: FVP Base RevC (DT) > > | Workqueue: rpciod rpc_async_schedule > > | Call trace: > > | dump_backtrace+0xec/0x108 > > | show_stack+0x18/0x2c > > | dump_stack_lvl+0x50/0x68 > > | dump_stack+0x18/0x24 > > | panic+0x138/0x33c > > | nvhe_hyp_panic_handler+0x100/0x184 > > | new_slab+0x23c/0x54c > > | ___slab_alloc+0x3e4/0x770 > > | kmem_cache_alloc_node+0x1f0/0x278 > > | __alloc_skb+0xdc/0x294 > > | tcp_stream_alloc_skb+0x2c/0xf0 > > | tcp_sendmsg_locked+0x3d0/0xda4 > > | tcp_sendmsg+0x38/0x5c > > | inet_sendmsg+0x44/0x60 > > | sock_sendmsg+0x1c/0x34 > > | xprt_sock_sendmsg+0xdc/0x274 > > | xs_tcp_send_request+0x1ac/0x28c > > | xprt_transmit+0xcc/0x300 > > | call_transmit+0x78/0x90 > > | __rpc_execute+0x114/0x3d8 > > | rpc_async_schedule+0x28/0x48 > > | process_one_work+0x1d8/0x314 > > | worker_thread+0x248/0x474 > > | kthread+0xfc/0x184 > > | ret_from_fork+0x10/0x20 > > | SMP: stopping secondary CPUs > > | Kernel Offset: 0x57c5cb460000 from 0xffff800080000000 > > | PHYS_OFFSET: 0x80000000 > > | CPU features: 0x00000000,1035b7a3,ccfe773f > > | Memory Limit: none > > | ---[ end Kernel panic - not syncing: HYP panic: > > | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 > > | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 > > | VCPU:0000000000000000 ]--- > > > > Fix it by checking for the successfull initialisation of kvm_arm_init() > > in finalize_pkvm() before proceeding any futher. > > > > Fixes: 87727ba2bb05 ("KVM: arm64: Ensure CPU PMU probes before pKVM host de-privilege") > > Cc: Will Deacon <will@kernel.org> > > Cc: Marc Zyngier <maz@kernel.org> > > Cc: Oliver Upton <oliver.upton@linux.dev> > > Cc: James Morse <james.morse@arm.com> > > Cc: Suzuki K Poulose <suzuki.poulose@arm.com> > > Cc: Zenghui Yu <yuzenghui@huawei.com> > > Signed-off-by: Sudeep Holla <sudeep.holla@arm.com> > > I really dislike the additional helper, but now that the capabilities > are patched, I can't see another way to achieve this so: > Yes, I too don't like it and was almost tempted to post it as RFC, but then I didn't want it to get ignored ;) as it a bug I hit on my setup with mismatched FF-A version. > Acked-by: Marc Zyngier <maz@kernel.org> > Thanks!
On Tue, 4 Jul 2023 20:32:43 +0100, Sudeep Holla wrote: > Currently there is no synchronisation between finalize_pkvm() and > kvm_arm_init() initcalls. The finalize_pkvm() proceeds happily even if > kvm_arm_init() fails resulting in the following warning on all the CPUs > and eventually a HYP panic: > > | kvm [1]: IPA Size Limit: 48 bits > | kvm [1]: Failed to init hyp memory protection > | kvm [1]: error initializing Hyp mode: -22 > | > | <snip> > | > | WARNING: CPU: 0 PID: 0 at arch/arm64/kvm/pkvm.c:226 _kvm_host_prot_finalize+0x30/0x50 > | Modules linked in: > | CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0 #237 > | Hardware name: FVP Base RevC (DT) > | pstate: 634020c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) > | pc : _kvm_host_prot_finalize+0x30/0x50 > | lr : __flush_smp_call_function_queue+0xd8/0x230 > | > | Call trace: > | _kvm_host_prot_finalize+0x3c/0x50 > | on_each_cpu_cond_mask+0x3c/0x6c > | pkvm_drop_host_privileges+0x4c/0x78 > | finalize_pkvm+0x3c/0x5c > | do_one_initcall+0xcc/0x240 > | do_initcall_level+0x8c/0xac > | do_initcalls+0x54/0x94 > | do_basic_setup+0x1c/0x28 > | kernel_init_freeable+0x100/0x16c > | kernel_init+0x20/0x1a0 > | ret_from_fork+0x10/0x20 > | Failed to finalize Hyp protection: -22 > | dtb=fvp-base-revc.dtb > | kvm [95]: nVHE hyp BUG at: arch/arm64/kvm/hyp/nvhe/mem_protect.c:540! > | kvm [95]: nVHE call trace: > | kvm [95]: [<ffff800081052984>] __kvm_nvhe_hyp_panic+0xac/0xf8 > | kvm [95]: [<ffff800081059644>] __kvm_nvhe_handle_host_mem_abort+0x1a0/0x2ac > | kvm [95]: [<ffff80008105511c>] __kvm_nvhe_handle_trap+0x4c/0x160 > | kvm [95]: [<ffff8000810540fc>] __kvm_nvhe___skip_pauth_save+0x4/0x4 > | kvm [95]: ---[ end nVHE call trace ]--- > | kvm [95]: Hyp Offset: 0xfffe8db00ffa0000 > | Kernel panic - not syncing: HYP panic: > | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 > | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 > | VCPU:0000000000000000 > | CPU: 3 PID: 95 Comm: kworker/u16:2 Tainted: G W 6.4.0 #237 > | Hardware name: FVP Base RevC (DT) > | Workqueue: rpciod rpc_async_schedule > | Call trace: > | dump_backtrace+0xec/0x108 > | show_stack+0x18/0x2c > | dump_stack_lvl+0x50/0x68 > | dump_stack+0x18/0x24 > | panic+0x138/0x33c > | nvhe_hyp_panic_handler+0x100/0x184 > | new_slab+0x23c/0x54c > | ___slab_alloc+0x3e4/0x770 > | kmem_cache_alloc_node+0x1f0/0x278 > | __alloc_skb+0xdc/0x294 > | tcp_stream_alloc_skb+0x2c/0xf0 > | tcp_sendmsg_locked+0x3d0/0xda4 > | tcp_sendmsg+0x38/0x5c > | inet_sendmsg+0x44/0x60 > | sock_sendmsg+0x1c/0x34 > | xprt_sock_sendmsg+0xdc/0x274 > | xs_tcp_send_request+0x1ac/0x28c > | xprt_transmit+0xcc/0x300 > | call_transmit+0x78/0x90 > | __rpc_execute+0x114/0x3d8 > | rpc_async_schedule+0x28/0x48 > | process_one_work+0x1d8/0x314 > | worker_thread+0x248/0x474 > | kthread+0xfc/0x184 > | ret_from_fork+0x10/0x20 > | SMP: stopping secondary CPUs > | Kernel Offset: 0x57c5cb460000 from 0xffff800080000000 > | PHYS_OFFSET: 0x80000000 > | CPU features: 0x00000000,1035b7a3,ccfe773f > | Memory Limit: none > | ---[ end Kernel panic - not syncing: HYP panic: > | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 > | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 > | VCPU:0000000000000000 ]--- > > [...] Applied to kvmarm/fixes, thanks! [1/1] KVM: arm64: Handle kvm_arm_init failure correctly in finalize_pkvm https://git.kernel.org/kvmarm/kvmarm/c/fa729bc7c9c8 -- Best, Oliver
diff --git a/arch/arm64/include/asm/virt.h b/arch/arm64/include/asm/virt.h index 5227db7640c8..261d6e9df2e1 100644 --- a/arch/arm64/include/asm/virt.h +++ b/arch/arm64/include/asm/virt.h @@ -78,6 +78,7 @@ extern u32 __boot_cpu_mode[2]; void __hyp_set_vectors(phys_addr_t phys_vector_base); void __hyp_reset_vectors(void); +bool is_kvm_arm_initialised(void); DECLARE_STATIC_KEY_FALSE(kvm_protected_mode_initialized); diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c index 173abca08b53..4e0176b44af3 100644 --- a/arch/arm64/kvm/arm.c +++ b/arch/arm64/kvm/arm.c @@ -53,11 +53,16 @@ DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params); DECLARE_KVM_NVHE_PER_CPU(struct kvm_cpu_context, kvm_hyp_ctxt); -static bool vgic_present; +static bool vgic_present, kvm_arm_initialised; static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled); DEFINE_STATIC_KEY_FALSE(userspace_irqchip_in_use); +bool is_kvm_arm_initialised(void) +{ + return kvm_arm_initialised; +} + int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu) { return kvm_vcpu_exiting_guest_mode(vcpu) == IN_GUEST_MODE; @@ -2484,6 +2489,8 @@ static __init int kvm_arm_init(void) if (err) goto out_subs; + kvm_arm_initialised = true; + return 0; out_subs: diff --git a/arch/arm64/kvm/pkvm.c b/arch/arm64/kvm/pkvm.c index 994a494703c3..6ff3ec18c925 100644 --- a/arch/arm64/kvm/pkvm.c +++ b/arch/arm64/kvm/pkvm.c @@ -244,7 +244,7 @@ static int __init finalize_pkvm(void) { int ret; - if (!is_protected_kvm_enabled()) + if (!is_protected_kvm_enabled() || !is_kvm_arm_initialised()) return 0; /*
Currently there is no synchronisation between finalize_pkvm() and kvm_arm_init() initcalls. The finalize_pkvm() proceeds happily even if kvm_arm_init() fails resulting in the following warning on all the CPUs and eventually a HYP panic: | kvm [1]: IPA Size Limit: 48 bits | kvm [1]: Failed to init hyp memory protection | kvm [1]: error initializing Hyp mode: -22 | | <snip> | | WARNING: CPU: 0 PID: 0 at arch/arm64/kvm/pkvm.c:226 _kvm_host_prot_finalize+0x30/0x50 | Modules linked in: | CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.4.0 #237 | Hardware name: FVP Base RevC (DT) | pstate: 634020c5 (nZCv daIF +PAN -UAO +TCO +DIT -SSBS BTYPE=--) | pc : _kvm_host_prot_finalize+0x30/0x50 | lr : __flush_smp_call_function_queue+0xd8/0x230 | | Call trace: | _kvm_host_prot_finalize+0x3c/0x50 | on_each_cpu_cond_mask+0x3c/0x6c | pkvm_drop_host_privileges+0x4c/0x78 | finalize_pkvm+0x3c/0x5c | do_one_initcall+0xcc/0x240 | do_initcall_level+0x8c/0xac | do_initcalls+0x54/0x94 | do_basic_setup+0x1c/0x28 | kernel_init_freeable+0x100/0x16c | kernel_init+0x20/0x1a0 | ret_from_fork+0x10/0x20 | Failed to finalize Hyp protection: -22 | dtb=fvp-base-revc.dtb | kvm [95]: nVHE hyp BUG at: arch/arm64/kvm/hyp/nvhe/mem_protect.c:540! | kvm [95]: nVHE call trace: | kvm [95]: [<ffff800081052984>] __kvm_nvhe_hyp_panic+0xac/0xf8 | kvm [95]: [<ffff800081059644>] __kvm_nvhe_handle_host_mem_abort+0x1a0/0x2ac | kvm [95]: [<ffff80008105511c>] __kvm_nvhe_handle_trap+0x4c/0x160 | kvm [95]: [<ffff8000810540fc>] __kvm_nvhe___skip_pauth_save+0x4/0x4 | kvm [95]: ---[ end nVHE call trace ]--- | kvm [95]: Hyp Offset: 0xfffe8db00ffa0000 | Kernel panic - not syncing: HYP panic: | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 | VCPU:0000000000000000 | CPU: 3 PID: 95 Comm: kworker/u16:2 Tainted: G W 6.4.0 #237 | Hardware name: FVP Base RevC (DT) | Workqueue: rpciod rpc_async_schedule | Call trace: | dump_backtrace+0xec/0x108 | show_stack+0x18/0x2c | dump_stack_lvl+0x50/0x68 | dump_stack+0x18/0x24 | panic+0x138/0x33c | nvhe_hyp_panic_handler+0x100/0x184 | new_slab+0x23c/0x54c | ___slab_alloc+0x3e4/0x770 | kmem_cache_alloc_node+0x1f0/0x278 | __alloc_skb+0xdc/0x294 | tcp_stream_alloc_skb+0x2c/0xf0 | tcp_sendmsg_locked+0x3d0/0xda4 | tcp_sendmsg+0x38/0x5c | inet_sendmsg+0x44/0x60 | sock_sendmsg+0x1c/0x34 | xprt_sock_sendmsg+0xdc/0x274 | xs_tcp_send_request+0x1ac/0x28c | xprt_transmit+0xcc/0x300 | call_transmit+0x78/0x90 | __rpc_execute+0x114/0x3d8 | rpc_async_schedule+0x28/0x48 | process_one_work+0x1d8/0x314 | worker_thread+0x248/0x474 | kthread+0xfc/0x184 | ret_from_fork+0x10/0x20 | SMP: stopping secondary CPUs | Kernel Offset: 0x57c5cb460000 from 0xffff800080000000 | PHYS_OFFSET: 0x80000000 | CPU features: 0x00000000,1035b7a3,ccfe773f | Memory Limit: none | ---[ end Kernel panic - not syncing: HYP panic: | PS:a34023c9 PC:0000f250710b973c ESR:00000000f2000800 | FAR:ffff000800cb00d0 HPFAR:000000000880cb00 PAR:0000000000000000 | VCPU:0000000000000000 ]--- Fix it by checking for the successfull initialisation of kvm_arm_init() in finalize_pkvm() before proceeding any futher. Fixes: 87727ba2bb05 ("KVM: arm64: Ensure CPU PMU probes before pKVM host de-privilege") Cc: Will Deacon <will@kernel.org> Cc: Marc Zyngier <maz@kernel.org> Cc: Oliver Upton <oliver.upton@linux.dev> Cc: James Morse <james.morse@arm.com> Cc: Suzuki K Poulose <suzuki.poulose@arm.com> Cc: Zenghui Yu <yuzenghui@huawei.com> Signed-off-by: Sudeep Holla <sudeep.holla@arm.com> --- arch/arm64/include/asm/virt.h | 1 + arch/arm64/kvm/arm.c | 9 ++++++++- arch/arm64/kvm/pkvm.c | 2 +- 3 files changed, 10 insertions(+), 2 deletions(-)