| Message ID | 20230914030600.16993-1-jniethe5@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | KVM: PPC: Nested APIv2 guest support | expand |
On Thu, 14 Sep 2023 13:05:49 +1000, Jordan Niethe wrote: > A nested-HV API for PAPR has been developed based on the KVM-specific > nested-HV API that is upstream in Linux/KVM and QEMU. The PAPR API had > to break compatibility to accommodate implementation in other > hypervisors and partitioning firmware. The existing KVM-specific API > will be known as the Nested APIv1 and the PAPR API will be known as the > Nested APIv2. > > [...] Applied to powerpc/topic/ppc-kvm. [01/11] KVM: PPC: Always use the GPR accessors https://git.kernel.org/powerpc/c/0e85b7df9cb0c65f840109159ef6754c783e07a0 [02/11] KVM: PPC: Introduce FPR/VR accessor functions https://git.kernel.org/powerpc/c/52425a3b3c11cec58cf66e4c897fc1504f3911a9 [03/11] KVM: PPC: Rename accessor generator macros https://git.kernel.org/powerpc/c/2a64bc673133346a7a3bd163f2e6048bd1788727 [04/11] KVM: PPC: Use accessors for VCPU registers https://git.kernel.org/powerpc/c/7028ac8d174f28220f0e2de0cb3346cd3c31976d [05/11] KVM: PPC: Use accessors for VCORE registers https://git.kernel.org/powerpc/c/c8ae9b3c6e7f22a4b71e42edb0fc3942aa7a7c42 [06/11] KVM: PPC: Book3S HV: Use accessors for VCPU registers https://git.kernel.org/powerpc/c/ebc88ea7a6ad0ea349df9c765357d3aa4e662aa9 [07/11] KVM: PPC: Book3S HV: Introduce low level MSR accessor https://git.kernel.org/powerpc/c/6de2e837babb411cfb3cdb570581c3a65576ddaf [08/11] KVM: PPC: Add helper library for Guest State Buffers https://git.kernel.org/powerpc/c/6ccbbc33f06adaf79acde18571c6543ad1cb4be6 [09/11] KVM: PPC: Book3s HV: Hold LPIDs in an unsigned long https://git.kernel.org/powerpc/c/dfcaacc8f970c6b4ea4e32d2186f2bea4a1d5255 [10/11] KVM: PPC: Add support for nestedv2 guests https://git.kernel.org/powerpc/c/19d31c5f115754c369c0995df47479c384757f82 [11/11] docs: powerpc: Document nested KVM on POWER https://git.kernel.org/powerpc/c/476652297f94a2e5e5ef29e734b0da37ade94110 cheers
On Thu, Sep 14, 2023 at 01:05:49PM +1000, Jordan Niethe wrote: > > A nested-HV API for PAPR has been developed based on the KVM-specific > nested-HV API that is upstream in Linux/KVM and QEMU. The PAPR API had > to break compatibility to accommodate implementation in other > hypervisors and partitioning firmware. The existing KVM-specific API > will be known as the Nested APIv1 and the PAPR API will be known as the > Nested APIv2. > > The control flow and interrupt processing between L0, L1, and L2 in > the Nested APIv2 are conceptually unchanged. Where Nested APIv1 is almost > stateless, the Nested APIv2 is stateful, with the L1 registering L2 virtual > machines and vCPUs with the L0. Supervisor-privileged register switching > duty is now the responsibility for the L0, which holds canonical L2 > register state and handles all switching. This new register handling > motivates the "getters and setters" wrappers to assist in syncing the > L2s state in the L1 and the L0. Hello, are there any machines on which this is supposed to work? On a 9105-22A with ML1050_fw1050.20 (78) and Linux 6.11.0-lp155.4.gce149d2-default I get: [ 29.228161] kvm-hv: nestedv2 get capabilities hcall failed, falling back to nestedv1 (rc=-2) [ 29.228168] kvm-hv: Parent hypervisor does not support nesting (rc=-2) Can the hardware requirements be clarified? Thanks Michal > > Broadly, the new hcalls will be used for creating and managing guests > by a regular partition in the following way: > > - L1 and L0 negotiate capabilities with > H_GUEST_{G,S}ET_CAPABILITIES > > - L1 requests the L0 create a L2 with > H_GUEST_CREATE and receives a handle to use in future hcalls > > - L1 requests the L0 create a L2 vCPU with > H_GUEST_CREATE_VCPU > > - L1 sets up the L2 using H_GUEST_SET and the > H_GUEST_VCPU_RUN input buffer > > - L1 requests the L0 runs the L2 vCPU using H_GUEST_VCPU_RUN > > - L2 returns to L1 with an exit reason and L1 reads the > H_GUEST_VCPU_RUN output buffer populated by the L0 > > - L1 handles the exit using H_GET_STATE if necessary > > - L1 reruns L2 vCPU with H_GUEST_VCPU_RUN > > - L1 frees the L2 in the L0 with H_GUEST_DELETE > > Further details are available in Documentation/powerpc/kvm-nested.rst. > > This series adds KVM support for using this hcall interface as a regular > PAPR partition, i.e. the L1. It does not add support for running as the > L0. > > The new hcalls have been implemented in the spapr qemu model for > testing. > > This is available at https://github.com/planetharsh/qemu/tree/upstream-0714-kop > > There are scripts available to assist in setting up an environment for > testing nested guests at https://github.com/iamjpn/kvm-powervm-test > > A tree with this series is available at > https://github.com/iamjpn/linux/tree/features/kvm-nestedv2-v5 > > Thanks to Amit Machhiwal, Kautuk Consul, Vaibhav Jain, Michael Neuling, > Shivaprasad Bhat, Harsh Prateek Bora, Paul Mackerras and Nicholas > Piggin. > > Change overview in v5: > - KVM: PPC: Add helper library for Guest State Buffers: > - Fix mismatched function comments > - KVM: PPC: Add support for nestedv2 guests: > - Check H_BUSY for {g,s}etting capabilities > - Message if plpar_guest_get_capabilities() fails and nestedv1 > support will be attempted. > - Remove unused amor variable > - KVM: PPC: Book3S HV: Use accessors for VCPU registers: > - Remove unneeded trailing comment for line length > > > Change overview in v4: > - Split previous "KVM: PPC: Use getters and setters for vcpu register > state" into a number of seperate patches > - Remove _hv suffix from VCORE wrappers > - Do not create arch_compat and lpcr setters, use the existing ones > - Use #ifdef ALTIVEC > - KVM: PPC: Rename accessor generator macros > - Fix typo > - KVM: PPC: Book3s HV: Hold LPIDs in an unsigned long > - Use u64 > - Change format strings instead of casting > - KVM: PPC: Add support for nestedv2 guests > - Batch H_GUEST_GET calls in kvmhv_nestedv2_reload_ptregs() > - Fix compile without CONFIG_PSERIES > - Fix maybe uninitialized 'trap' in kvmhv_p9_guest_entry() > - Extend existing setters for arch_compat and lpcr > > > Change overview in v3: > - KVM: PPC: Use getters and setters for vcpu register state > - Do not add a helper for pvr > - Use an expression when declaring variable in case > - Squash in all getters and setters > - Pass vector registers by reference > - KVM: PPC: Rename accessor generator macros > - New to series > - KVM: PPC: Add helper library for Guest State Buffers > - Use EXPORT_SYMBOL_GPL() > - Use the kvmppc namespace > - Move kvmppc_gsb_reset() out of kvmppc_gsm_fill_info() > - Comments for GSID elements > - Pass vector elements by reference > - Remove generic put and get functions > - KVM: PPC: Book3s HV: Hold LPIDs in an unsigned long > - New to series > - KVM: PPC: Add support for nestedv2 guests > - Use EXPORT_SYMBOL_GPL() > - Change to kvmhv_nestedv2 namespace > - Make kvmhv_enable_nested() return -ENODEV on NESTEDv2 L1 hosts > - s/kvmhv_on_papr/kvmhv_is_nestedv2/ > - mv book3s_hv_papr.c book3s_hv_nestedv2.c > - Handle shared regs without a guest state id in the same wrapper > - Use a static key for API version > - Add a positive test for NESTEDv1 > - Give the amor a static value > - s/struct kvmhv_nestedv2_host/struct kvmhv_nestedv2_io/ > - Propagate failure in kvmhv_vcpu_entry_nestedv2() > - WARN if getters and setters fail > - Progagate failure from kvmhv_nestedv2_parse_output() > - Replace delay with sleep in plpar_guest_{create,delete,create_vcpu}() > - Add logical PVR handling > - Replace kvmppc_gse_{get,put} with specific version > - docs: powerpc: Document nested KVM on POWER > - Fix typos > > > Change overview in v2: > - Rebase on top of kvm ppc prefix instruction support > - Make documentation an individual patch > - Move guest state buffer files from arch/powerpc/lib/ to > arch/powerpc/kvm/ > - Use kunit for testing guest state buffer > - Fix some build errors > - Change HEIR element from 4 bytes to 8 bytes > > Previous revisions: > > - v1: https://lore.kernel.org/linuxppc-dev/20230508072332.2937883-1-jpn@linux.vnet.ibm.com/ > - v2: https://lore.kernel.org/linuxppc-dev/20230605064848.12319-1-jpn@linux.vnet.ibm.com/ > - v3: https://lore.kernel.org/linuxppc-dev/20230807014553.1168699-1-jniethe5@gmail.com/ > - v4: https://lore.kernel.org/linuxppc-dev/20230905034658.82835-1-jniethe5@gmail.com/ > > Jordan Niethe (10): > KVM: PPC: Always use the GPR accessors > KVM: PPC: Introduce FPR/VR accessor functions > KVM: PPC: Rename accessor generator macros > KVM: PPC: Use accessors for VCPU registers > KVM: PPC: Use accessors for VCORE registers > KVM: PPC: Book3S HV: Use accessors for VCPU registers > KVM: PPC: Book3S HV: Introduce low level MSR accessor > KVM: PPC: Add helper library for Guest State Buffers > KVM: PPC: Book3s HV: Hold LPIDs in an unsigned long > KVM: PPC: Add support for nestedv2 guests > > Michael Neuling (1): > docs: powerpc: Document nested KVM on POWER > > Documentation/powerpc/index.rst | 1 + > Documentation/powerpc/kvm-nested.rst | 636 +++++++++++ > arch/powerpc/Kconfig.debug | 12 + > arch/powerpc/include/asm/guest-state-buffer.h | 995 ++++++++++++++++++ > arch/powerpc/include/asm/hvcall.h | 30 + > arch/powerpc/include/asm/kvm_book3s.h | 220 +++- > arch/powerpc/include/asm/kvm_book3s_64.h | 8 +- > arch/powerpc/include/asm/kvm_booke.h | 10 + > arch/powerpc/include/asm/kvm_host.h | 22 +- > arch/powerpc/include/asm/kvm_ppc.h | 102 +- > arch/powerpc/include/asm/plpar_wrappers.h | 267 ++++- > arch/powerpc/kvm/Makefile | 4 + > arch/powerpc/kvm/book3s.c | 38 +- > arch/powerpc/kvm/book3s_64_mmu_hv.c | 7 +- > arch/powerpc/kvm/book3s_64_mmu_radix.c | 31 +- > arch/powerpc/kvm/book3s_64_vio.c | 4 +- > arch/powerpc/kvm/book3s_hv.c | 358 +++++-- > arch/powerpc/kvm/book3s_hv.h | 76 ++ > arch/powerpc/kvm/book3s_hv_builtin.c | 11 +- > arch/powerpc/kvm/book3s_hv_nested.c | 44 +- > arch/powerpc/kvm/book3s_hv_nestedv2.c | 994 +++++++++++++++++ > arch/powerpc/kvm/book3s_hv_p9_entry.c | 4 +- > arch/powerpc/kvm/book3s_hv_ras.c | 4 +- > arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 +- > arch/powerpc/kvm/book3s_hv_rm_xics.c | 4 +- > arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- > arch/powerpc/kvm/book3s_xive.c | 12 +- > arch/powerpc/kvm/emulate_loadstore.c | 6 +- > arch/powerpc/kvm/guest-state-buffer.c | 621 +++++++++++ > arch/powerpc/kvm/powerpc.c | 76 +- > arch/powerpc/kvm/test-guest-state-buffer.c | 328 ++++++ > 31 files changed, 4672 insertions(+), 263 deletions(-) > create mode 100644 Documentation/powerpc/kvm-nested.rst > create mode 100644 arch/powerpc/include/asm/guest-state-buffer.h > create mode 100644 arch/powerpc/kvm/book3s_hv_nestedv2.c > create mode 100644 arch/powerpc/kvm/guest-state-buffer.c > create mode 100644 arch/powerpc/kvm/test-guest-state-buffer.c > > -- > 2.39.3 >
Hi Michal, Michal Suchánek <msuchanek@suse.de> writes: <snip> > Hello, > > are there any machines on which this is supposed to work? > > On a 9105-22A with ML1050_fw1050.20 (78) and On 9105-22A you need atleast: Firmware level: FW1060.10 > Linux 6.11.0-lp155.4.gce149d2-default I get: Kernel version is fine. ATM anything >=6.10 is good > > [ 29.228161] kvm-hv: nestedv2 get capabilities hcall failed, falling back to nestedv1 (rc=-2) > [ 29.228168] kvm-hv: Parent hypervisor does not support nesting (rc=-2) > If you are still getting this error after switching to FW1060.10 than you need to enable KVM mode for the LPAR following instructions here: https://www.ibm.com/docs/en/linux-on-systems?topic=servers-kvm-in-powervm-lpar#kvm_in_powervm_lpar__title__9 TLDR; Just need to enable 'KVM Mode' in the LPAR configuration from HMC. > Can the hardware requirements be clarified? Further clarified at https://www.ibm.com/docs/en/linux-on-systems?topic=servers-kvm-in-powervm-lpar There is a qemu model based on tcg, that was merged in v9.0.0 some time back via https://lore.kernel.org/all/20240308111940.1617660-1-harshpb@linux.ibm.com/ If you want to try this, then following can be helpful in setting up the environment: https://github.com/iamjpn/kvm-powervm-test > Thanks > > Michal > <snip>
On Thu, Sep 26, 2024 at 10:53:45AM +0530, Vaibhav Jain wrote: > Hi Michal, > > Michal Suchánek <msuchanek@suse.de> writes: > > <snip> > > > Hello, > > > > are there any machines on which this is supposed to work? > > > > On a 9105-22A with ML1050_fw1050.20 (78) and > > On 9105-22A you need atleast: > Firmware level: FW1060.10 Indeed, upgrading to FW1060 makes the KVM functionality available. Thanks Michal
A nested-HV API for PAPR has been developed based on the KVM-specific nested-HV API that is upstream in Linux/KVM and QEMU. The PAPR API had to break compatibility to accommodate implementation in other hypervisors and partitioning firmware. The existing KVM-specific API will be known as the Nested APIv1 and the PAPR API will be known as the Nested APIv2. The control flow and interrupt processing between L0, L1, and L2 in the Nested APIv2 are conceptually unchanged. Where Nested APIv1 is almost stateless, the Nested APIv2 is stateful, with the L1 registering L2 virtual machines and vCPUs with the L0. Supervisor-privileged register switching duty is now the responsibility for the L0, which holds canonical L2 register state and handles all switching. This new register handling motivates the "getters and setters" wrappers to assist in syncing the L2s state in the L1 and the L0. Broadly, the new hcalls will be used for creating and managing guests by a regular partition in the following way: - L1 and L0 negotiate capabilities with H_GUEST_{G,S}ET_CAPABILITIES - L1 requests the L0 create a L2 with H_GUEST_CREATE and receives a handle to use in future hcalls - L1 requests the L0 create a L2 vCPU with H_GUEST_CREATE_VCPU - L1 sets up the L2 using H_GUEST_SET and the H_GUEST_VCPU_RUN input buffer - L1 requests the L0 runs the L2 vCPU using H_GUEST_VCPU_RUN - L2 returns to L1 with an exit reason and L1 reads the H_GUEST_VCPU_RUN output buffer populated by the L0 - L1 handles the exit using H_GET_STATE if necessary - L1 reruns L2 vCPU with H_GUEST_VCPU_RUN - L1 frees the L2 in the L0 with H_GUEST_DELETE Further details are available in Documentation/powerpc/kvm-nested.rst. This series adds KVM support for using this hcall interface as a regular PAPR partition, i.e. the L1. It does not add support for running as the L0. The new hcalls have been implemented in the spapr qemu model for testing. This is available at https://github.com/planetharsh/qemu/tree/upstream-0714-kop There are scripts available to assist in setting up an environment for testing nested guests at https://github.com/iamjpn/kvm-powervm-test A tree with this series is available at https://github.com/iamjpn/linux/tree/features/kvm-nestedv2-v5 Thanks to Amit Machhiwal, Kautuk Consul, Vaibhav Jain, Michael Neuling, Shivaprasad Bhat, Harsh Prateek Bora, Paul Mackerras and Nicholas Piggin. Change overview in v5: - KVM: PPC: Add helper library for Guest State Buffers: - Fix mismatched function comments - KVM: PPC: Add support for nestedv2 guests: - Check H_BUSY for {g,s}etting capabilities - Message if plpar_guest_get_capabilities() fails and nestedv1 support will be attempted. - Remove unused amor variable - KVM: PPC: Book3S HV: Use accessors for VCPU registers: - Remove unneeded trailing comment for line length Change overview in v4: - Split previous "KVM: PPC: Use getters and setters for vcpu register state" into a number of seperate patches - Remove _hv suffix from VCORE wrappers - Do not create arch_compat and lpcr setters, use the existing ones - Use #ifdef ALTIVEC - KVM: PPC: Rename accessor generator macros - Fix typo - KVM: PPC: Book3s HV: Hold LPIDs in an unsigned long - Use u64 - Change format strings instead of casting - KVM: PPC: Add support for nestedv2 guests - Batch H_GUEST_GET calls in kvmhv_nestedv2_reload_ptregs() - Fix compile without CONFIG_PSERIES - Fix maybe uninitialized 'trap' in kvmhv_p9_guest_entry() - Extend existing setters for arch_compat and lpcr Change overview in v3: - KVM: PPC: Use getters and setters for vcpu register state - Do not add a helper for pvr - Use an expression when declaring variable in case - Squash in all getters and setters - Pass vector registers by reference - KVM: PPC: Rename accessor generator macros - New to series - KVM: PPC: Add helper library for Guest State Buffers - Use EXPORT_SYMBOL_GPL() - Use the kvmppc namespace - Move kvmppc_gsb_reset() out of kvmppc_gsm_fill_info() - Comments for GSID elements - Pass vector elements by reference - Remove generic put and get functions - KVM: PPC: Book3s HV: Hold LPIDs in an unsigned long - New to series - KVM: PPC: Add support for nestedv2 guests - Use EXPORT_SYMBOL_GPL() - Change to kvmhv_nestedv2 namespace - Make kvmhv_enable_nested() return -ENODEV on NESTEDv2 L1 hosts - s/kvmhv_on_papr/kvmhv_is_nestedv2/ - mv book3s_hv_papr.c book3s_hv_nestedv2.c - Handle shared regs without a guest state id in the same wrapper - Use a static key for API version - Add a positive test for NESTEDv1 - Give the amor a static value - s/struct kvmhv_nestedv2_host/struct kvmhv_nestedv2_io/ - Propagate failure in kvmhv_vcpu_entry_nestedv2() - WARN if getters and setters fail - Progagate failure from kvmhv_nestedv2_parse_output() - Replace delay with sleep in plpar_guest_{create,delete,create_vcpu}() - Add logical PVR handling - Replace kvmppc_gse_{get,put} with specific version - docs: powerpc: Document nested KVM on POWER - Fix typos Change overview in v2: - Rebase on top of kvm ppc prefix instruction support - Make documentation an individual patch - Move guest state buffer files from arch/powerpc/lib/ to arch/powerpc/kvm/ - Use kunit for testing guest state buffer - Fix some build errors - Change HEIR element from 4 bytes to 8 bytes Previous revisions: - v1: https://lore.kernel.org/linuxppc-dev/20230508072332.2937883-1-jpn@linux.vnet.ibm.com/ - v2: https://lore.kernel.org/linuxppc-dev/20230605064848.12319-1-jpn@linux.vnet.ibm.com/ - v3: https://lore.kernel.org/linuxppc-dev/20230807014553.1168699-1-jniethe5@gmail.com/ - v4: https://lore.kernel.org/linuxppc-dev/20230905034658.82835-1-jniethe5@gmail.com/ Jordan Niethe (10): KVM: PPC: Always use the GPR accessors KVM: PPC: Introduce FPR/VR accessor functions KVM: PPC: Rename accessor generator macros KVM: PPC: Use accessors for VCPU registers KVM: PPC: Use accessors for VCORE registers KVM: PPC: Book3S HV: Use accessors for VCPU registers KVM: PPC: Book3S HV: Introduce low level MSR accessor KVM: PPC: Add helper library for Guest State Buffers KVM: PPC: Book3s HV: Hold LPIDs in an unsigned long KVM: PPC: Add support for nestedv2 guests Michael Neuling (1): docs: powerpc: Document nested KVM on POWER Documentation/powerpc/index.rst | 1 + Documentation/powerpc/kvm-nested.rst | 636 +++++++++++ arch/powerpc/Kconfig.debug | 12 + arch/powerpc/include/asm/guest-state-buffer.h | 995 ++++++++++++++++++ arch/powerpc/include/asm/hvcall.h | 30 + arch/powerpc/include/asm/kvm_book3s.h | 220 +++- arch/powerpc/include/asm/kvm_book3s_64.h | 8 +- arch/powerpc/include/asm/kvm_booke.h | 10 + arch/powerpc/include/asm/kvm_host.h | 22 +- arch/powerpc/include/asm/kvm_ppc.h | 102 +- arch/powerpc/include/asm/plpar_wrappers.h | 267 ++++- arch/powerpc/kvm/Makefile | 4 + arch/powerpc/kvm/book3s.c | 38 +- arch/powerpc/kvm/book3s_64_mmu_hv.c | 7 +- arch/powerpc/kvm/book3s_64_mmu_radix.c | 31 +- arch/powerpc/kvm/book3s_64_vio.c | 4 +- arch/powerpc/kvm/book3s_hv.c | 358 +++++-- arch/powerpc/kvm/book3s_hv.h | 76 ++ arch/powerpc/kvm/book3s_hv_builtin.c | 11 +- arch/powerpc/kvm/book3s_hv_nested.c | 44 +- arch/powerpc/kvm/book3s_hv_nestedv2.c | 994 +++++++++++++++++ arch/powerpc/kvm/book3s_hv_p9_entry.c | 4 +- arch/powerpc/kvm/book3s_hv_ras.c | 4 +- arch/powerpc/kvm/book3s_hv_rm_mmu.c | 8 +- arch/powerpc/kvm/book3s_hv_rm_xics.c | 4 +- arch/powerpc/kvm/book3s_hv_uvmem.c | 2 +- arch/powerpc/kvm/book3s_xive.c | 12 +- arch/powerpc/kvm/emulate_loadstore.c | 6 +- arch/powerpc/kvm/guest-state-buffer.c | 621 +++++++++++ arch/powerpc/kvm/powerpc.c | 76 +- arch/powerpc/kvm/test-guest-state-buffer.c | 328 ++++++ 31 files changed, 4672 insertions(+), 263 deletions(-) create mode 100644 Documentation/powerpc/kvm-nested.rst create mode 100644 arch/powerpc/include/asm/guest-state-buffer.h create mode 100644 arch/powerpc/kvm/book3s_hv_nestedv2.c create mode 100644 arch/powerpc/kvm/guest-state-buffer.c create mode 100644 arch/powerpc/kvm/test-guest-state-buffer.c