| Message ID | 20240428100518.1642324-1-maobibo@loongson.cn (mailing list archive) |
|---|---|
| Headers | show |
| Series | LoongArch: Add pv ipi support on LoongArch VM | expand |
Hi, Bibo, I have done an off-list discussion with some KVM experts, and they think user-space have its right to know PV features, so cpucfg solution is acceptable. And I applied this series with some modifications at https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson.git/log/?h=loongarch-kvm You can test it now. But it seems the upstream qemu cannot enable PV IPI now. I will reply to other patches about my modifications. Huacai On Sun, Apr 28, 2024 at 6:05 PM Bibo Mao <maobibo@loongson.cn> wrote: > > On physical machine, ipi HW uses IOCSR registers, however there is trap > into hypervisor when vcpu accesses IOCSR registers if system is in VM > mode. SWI is a interrupt mechanism like SGI on ARM, software can send > interrupt to CPU, only that on LoongArch SWI can only be sent to local CPU > now. So SWI can not used for IPI on real HW system, however it can be used > on VM when combined with hypercall method. IPI can be sent with hypercall > method and SWI interrupt is injected to vcpu, vcpu can treat SWI > interrupt as IPI. > > With PV IPI supported, there is one trap with IPI sending, however with IPI > receiving there is no trap. with IOCSR HW ipi method, there will be one > trap with IPI sending and two trap with ipi receiving. > > Also IPI multicast support is added for VM, the idea comes from x86 PV ipi. > IPI can be sent to 128 vcpus in one time. With IPI multicast support, trap > will be reduced greatly. > > Here is the microbenchmarck data with "perf bench futex wake" testcase on > 3C5000 single-way machine, there are 16 cpus on 3C5000 single-way machine, > VM has 16 vcpus also. The benchmark data is ms time unit to wakeup 16 > threads, the performance is better if data is smaller. > > physical machine 0.0176 ms > VM original 0.1140 ms > VM with pv ipi patch 0.0481 ms > > It passes to boot with 128/256 vcpus, and passes to run runltp command > with package ltp-20230516. > > --- > v7 --- v8: > 1. Remove kernel PLV mode checking with cpucfg emulation for hypervisor > feature inquiry. > 2. Remove document about loongarch hypercall ABI per request of huacai, > will add English/Chinese doc at the same time in later. > > v6 --- v7: > 1. Refine LoongArch virt document by review comments. > 2. Add function kvm_read_reg()/kvm_write_reg() in hypercall emulation, > and later it can be used for other trap emulations. > > v5 --- v6: > 1. Add privilege checking when emulating cpucfg at index 0x4000000 -- > 0x400000FF, return 0 if not executed at kernel mode. > 2. Add document about LoongArch pv ipi with new creatly directory > Documentation/virt/kvm/loongarch/ > 3. Fix pv ipi handling in kvm backend function kvm_pv_send_ipi(), > where min should plus BITS_PER_LONG with second bitmap, otherwise > VM with more than 64 vpus fails to boot. > 4. Adjust patch order and code refine with review comments. > > v4 --- v5: > 1. Refresh function/macro name from review comments. > > v3 --- v4: > 1. Modfiy pv ipi hook function name call_func_ipi() and > call_func_single_ipi() with send_ipi_mask()/send_ipi_single(), since pv > ipi is used for both remote function call and reschedule notification. > 2. Refresh changelog. > > v2 --- v3: > 1. Add 128 vcpu ipi multicast support like x86 > 2. Change cpucfg base address from 0x10000000 to 0x40000000, in order > to avoid confliction with future hw usage > 3. Adjust patch order in this patchset, move patch > Refine-ipi-ops-on-LoongArch-platform to the first one. > > v1 --- v2: > 1. Add hw cpuid map support since ipi routing uses hw cpuid > 2. Refine changelog description > 3. Add hypercall statistic support for vcpu > 4. Set percpu pv ipi message buffer aligned with cacheline > 5. Refine pv ipi send logic, do not send ipi message with if there is > pending ipi message. > --- > Bibo Mao (6): > LoongArch/smp: Refine some ipi functions on LoongArch platform > LoongArch: KVM: Add hypercall instruction emulation support > LoongArch: KVM: Add cpucfg area for kvm hypervisor > LoongArch: KVM: Add vcpu search support from physical cpuid > LoongArch: KVM: Add pv ipi support on kvm side > LoongArch: Add pv ipi support on guest kernel side > > arch/loongarch/Kconfig | 9 + > arch/loongarch/include/asm/Kbuild | 1 - > arch/loongarch/include/asm/hardirq.h | 5 + > arch/loongarch/include/asm/inst.h | 1 + > arch/loongarch/include/asm/irq.h | 10 +- > arch/loongarch/include/asm/kvm_host.h | 27 +++ > arch/loongarch/include/asm/kvm_para.h | 155 ++++++++++++++++++ > arch/loongarch/include/asm/kvm_vcpu.h | 11 ++ > arch/loongarch/include/asm/loongarch.h | 11 ++ > arch/loongarch/include/asm/paravirt.h | 27 +++ > .../include/asm/paravirt_api_clock.h | 1 + > arch/loongarch/include/asm/smp.h | 31 ++-- > arch/loongarch/include/uapi/asm/Kbuild | 2 - > arch/loongarch/kernel/Makefile | 1 + > arch/loongarch/kernel/irq.c | 24 +-- > arch/loongarch/kernel/paravirt.c | 151 +++++++++++++++++ > arch/loongarch/kernel/perf_event.c | 14 +- > arch/loongarch/kernel/smp.c | 62 ++++--- > arch/loongarch/kernel/time.c | 12 +- > arch/loongarch/kvm/exit.c | 132 +++++++++++++-- > arch/loongarch/kvm/vcpu.c | 94 ++++++++++- > arch/loongarch/kvm/vm.c | 11 ++ > 22 files changed, 690 insertions(+), 102 deletions(-) > create mode 100644 arch/loongarch/include/asm/kvm_para.h > create mode 100644 arch/loongarch/include/asm/paravirt.h > create mode 100644 arch/loongarch/include/asm/paravirt_api_clock.h > delete mode 100644 arch/loongarch/include/uapi/asm/Kbuild > create mode 100644 arch/loongarch/kernel/paravirt.c > > > base-commit: 5eb4573ea63d0c83bf58fb7c243fc2c2b6966c02 > -- > 2.39.3 > >
On 2024/5/6 上午9:45, Huacai Chen wrote: > Hi, Bibo, > > I have done an off-list discussion with some KVM experts, and they > think user-space have its right to know PV features, so cpucfg > solution is acceptable. > > And I applied this series with some modifications at > https://git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson.git/log/?h=loongarch-kvm > You can test it now. But it seems the upstream qemu cannot enable PV IPI now. VM with 128/256 vcpus boots with this series in loongarch-kvm branch. And pv ipi works by information "cat /proc/interrupts". There need small modification with qemu like this, and we will submit the patch to qemu after it is merged. diff --git a/hw/loongarch/virt.c b/hw/loongarch/virt.c index 441d764843..9f7556cd93 100644 --- a/hw/loongarch/virt.c +++ b/hw/loongarch/virt.c @@ -15,6 +15,8 @@ #include "sysemu/runstate.h" #include "sysemu/reset.h" #include "sysemu/rtc.h" +#include "sysemu/tcg.h" +#include "sysemu/kvm.h" #include "hw/loongarch/virt.h" #include "exec/address-spaces.h" #include "hw/irq.h" @@ -786,12 +788,18 @@ static void loongarch_qemu_write(void *opaque, hwaddr addr, static uint64_t loongarch_qemu_read(void *opaque, hwaddr addr, unsigned size) { + uint64_t ret = 0; + switch (addr) { case VERSION_REG: return 0x11ULL; case FEATURE_REG: - return 1ULL << IOCSRF_MSI | 1ULL << IOCSRF_EXTIOI | + ret =1ULL << IOCSRF_MSI | 1ULL << IOCSRF_EXTIOI | 1ULL << IOCSRF_CSRIPI; + if (kvm_enabled()) { + ret |= 1ULL << IOCSRF_VM; + } + return ret; case VENDOR_REG: return 0x6e6f73676e6f6f4cULL; /* "Loongson" */ case CPUNAME_REG: Regards Bibo Mao > > I will reply to other patches about my modifications. > > Huacai > > On Sun, Apr 28, 2024 at 6:05 PM Bibo Mao <maobibo@loongson.cn> wrote: >> >> On physical machine, ipi HW uses IOCSR registers, however there is trap >> into hypervisor when vcpu accesses IOCSR registers if system is in VM >> mode. SWI is a interrupt mechanism like SGI on ARM, software can send >> interrupt to CPU, only that on LoongArch SWI can only be sent to local CPU >> now. So SWI can not used for IPI on real HW system, however it can be used >> on VM when combined with hypercall method. IPI can be sent with hypercall >> method and SWI interrupt is injected to vcpu, vcpu can treat SWI >> interrupt as IPI. >> >> With PV IPI supported, there is one trap with IPI sending, however with IPI >> receiving there is no trap. with IOCSR HW ipi method, there will be one >> trap with IPI sending and two trap with ipi receiving. >> >> Also IPI multicast support is added for VM, the idea comes from x86 PV ipi. >> IPI can be sent to 128 vcpus in one time. With IPI multicast support, trap >> will be reduced greatly. >> >> Here is the microbenchmarck data with "perf bench futex wake" testcase on >> 3C5000 single-way machine, there are 16 cpus on 3C5000 single-way machine, >> VM has 16 vcpus also. The benchmark data is ms time unit to wakeup 16 >> threads, the performance is better if data is smaller. >> >> physical machine 0.0176 ms >> VM original 0.1140 ms >> VM with pv ipi patch 0.0481 ms >> >> It passes to boot with 128/256 vcpus, and passes to run runltp command >> with package ltp-20230516. >> >> --- >> v7 --- v8: >> 1. Remove kernel PLV mode checking with cpucfg emulation for hypervisor >> feature inquiry. >> 2. Remove document about loongarch hypercall ABI per request of huacai, >> will add English/Chinese doc at the same time in later. >> >> v6 --- v7: >> 1. Refine LoongArch virt document by review comments. >> 2. Add function kvm_read_reg()/kvm_write_reg() in hypercall emulation, >> and later it can be used for other trap emulations. >> >> v5 --- v6: >> 1. Add privilege checking when emulating cpucfg at index 0x4000000 -- >> 0x400000FF, return 0 if not executed at kernel mode. >> 2. Add document about LoongArch pv ipi with new creatly directory >> Documentation/virt/kvm/loongarch/ >> 3. Fix pv ipi handling in kvm backend function kvm_pv_send_ipi(), >> where min should plus BITS_PER_LONG with second bitmap, otherwise >> VM with more than 64 vpus fails to boot. >> 4. Adjust patch order and code refine with review comments. >> >> v4 --- v5: >> 1. Refresh function/macro name from review comments. >> >> v3 --- v4: >> 1. Modfiy pv ipi hook function name call_func_ipi() and >> call_func_single_ipi() with send_ipi_mask()/send_ipi_single(), since pv >> ipi is used for both remote function call and reschedule notification. >> 2. Refresh changelog. >> >> v2 --- v3: >> 1. Add 128 vcpu ipi multicast support like x86 >> 2. Change cpucfg base address from 0x10000000 to 0x40000000, in order >> to avoid confliction with future hw usage >> 3. Adjust patch order in this patchset, move patch >> Refine-ipi-ops-on-LoongArch-platform to the first one. >> >> v1 --- v2: >> 1. Add hw cpuid map support since ipi routing uses hw cpuid >> 2. Refine changelog description >> 3. Add hypercall statistic support for vcpu >> 4. Set percpu pv ipi message buffer aligned with cacheline >> 5. Refine pv ipi send logic, do not send ipi message with if there is >> pending ipi message. >> --- >> Bibo Mao (6): >> LoongArch/smp: Refine some ipi functions on LoongArch platform >> LoongArch: KVM: Add hypercall instruction emulation support >> LoongArch: KVM: Add cpucfg area for kvm hypervisor >> LoongArch: KVM: Add vcpu search support from physical cpuid >> LoongArch: KVM: Add pv ipi support on kvm side >> LoongArch: Add pv ipi support on guest kernel side >> >> arch/loongarch/Kconfig | 9 + >> arch/loongarch/include/asm/Kbuild | 1 - >> arch/loongarch/include/asm/hardirq.h | 5 + >> arch/loongarch/include/asm/inst.h | 1 + >> arch/loongarch/include/asm/irq.h | 10 +- >> arch/loongarch/include/asm/kvm_host.h | 27 +++ >> arch/loongarch/include/asm/kvm_para.h | 155 ++++++++++++++++++ >> arch/loongarch/include/asm/kvm_vcpu.h | 11 ++ >> arch/loongarch/include/asm/loongarch.h | 11 ++ >> arch/loongarch/include/asm/paravirt.h | 27 +++ >> .../include/asm/paravirt_api_clock.h | 1 + >> arch/loongarch/include/asm/smp.h | 31 ++-- >> arch/loongarch/include/uapi/asm/Kbuild | 2 - >> arch/loongarch/kernel/Makefile | 1 + >> arch/loongarch/kernel/irq.c | 24 +-- >> arch/loongarch/kernel/paravirt.c | 151 +++++++++++++++++ >> arch/loongarch/kernel/perf_event.c | 14 +- >> arch/loongarch/kernel/smp.c | 62 ++++--- >> arch/loongarch/kernel/time.c | 12 +- >> arch/loongarch/kvm/exit.c | 132 +++++++++++++-- >> arch/loongarch/kvm/vcpu.c | 94 ++++++++++- >> arch/loongarch/kvm/vm.c | 11 ++ >> 22 files changed, 690 insertions(+), 102 deletions(-) >> create mode 100644 arch/loongarch/include/asm/kvm_para.h >> create mode 100644 arch/loongarch/include/asm/paravirt.h >> create mode 100644 arch/loongarch/include/asm/paravirt_api_clock.h >> delete mode 100644 arch/loongarch/include/uapi/asm/Kbuild >> create mode 100644 arch/loongarch/kernel/paravirt.c >> >> >> base-commit: 5eb4573ea63d0c83bf58fb7c243fc2c2b6966c02 >> -- >> 2.39.3 >> >>