| Message ID | 20240708114124.407797-4-Jiqian.Chen@amd.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Support device passthrough when dom0 is PVH on Xen | expand |
Hi, On 2024/7/8 19:41, Jiqian Chen wrote: > The gsi of a passthrough device must be configured for it to be > able to be mapped into a hvm domU. > But When dom0 is PVH, the gsis may not get registered(see below > clarification), it causes the info of apic, pin and irq not be > added into irq_2_pin list, and the handler of irq_desc is not set, > then when passthrough a device, setting ioapic affinity and vector > will fail. > > To fix above problem, on Linux kernel side, a new code will > need to call PHYSDEVOP_setup_gsi for passthrough devices to > register gsi when dom0 is PVH. > > So, add PHYSDEVOP_setup_gsi into hvm_physdev_op for above > purpose. > > Clarify two questions: > First, why the gsi of devices belong to PVH dom0 can work? > Because when probe a driver to a normal device, it uses the normal > probe function of pci device, in its callstack, it requests irq > and unmask corresponding ioapic of gsi, then trap into xen and > register gsi finally. > Callstack is(on linux kernel side) pci_device_probe-> > request_threaded_irq-> irq_startup-> __unmask_ioapic-> > io_apic_write, then trap into xen hvmemul_do_io-> > hvm_io_intercept-> hvm_process_io_intercept-> > vioapic_write_indirect-> vioapic_hwdom_map_gsi-> mp_register_gsi. > So that the gsi can be registered. > > Second, why the gsi of passthrough device can't work when dom0 > is PVH? > Because when assign a device to passthrough, it uses the specific > probe function of pciback, in its callstack, it doesn't install a > fake irq handler due to the ISR is not running. So that > mp_register_gsi on Xen side is never called, then the gsi is not > registered. > Callstack is(on linux kernel side) pcistub_probe->pcistub_seize-> > pcistub_init_device-> xen_pcibk_reset_device-> > xen_pcibk_control_isr->isr_on==0. > > Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > Signed-off-by: Huang Rui <ray.huang@amd.com> > Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > --- > xen/arch/x86/hvm/hypercall.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/xen/arch/x86/hvm/hypercall.c b/xen/arch/x86/hvm/hypercall.c > index 03ada3c880bd..cfe82d0f96ed 100644 > --- a/xen/arch/x86/hvm/hypercall.c > +++ b/xen/arch/x86/hvm/hypercall.c > @@ -86,6 +86,7 @@ long hvm_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) > return -ENOSYS; > break; > > + case PHYSDEVOP_setup_gsi: > case PHYSDEVOP_pci_mmcfg_reserved: > case PHYSDEVOP_pci_device_add: > case PHYSDEVOP_pci_device_remove: Do you have any other concern about this patch? If not, may I get your Reviewd-by? Then the first three patches of this series can be considered to merged once I send next version, so that I can continue to upstream the kernel patches that depend on them.
Hi all, On 2024/7/8 19:41, Jiqian Chen wrote: > The gsi of a passthrough device must be configured for it to be > able to be mapped into a hvm domU. > But When dom0 is PVH, the gsis may not get registered(see below > clarification), it causes the info of apic, pin and irq not be > added into irq_2_pin list, and the handler of irq_desc is not set, > then when passthrough a device, setting ioapic affinity and vector > will fail. > > To fix above problem, on Linux kernel side, a new code will > need to call PHYSDEVOP_setup_gsi for passthrough devices to > register gsi when dom0 is PVH. > > So, add PHYSDEVOP_setup_gsi into hvm_physdev_op for above > purpose. > > Clarify two questions: > First, why the gsi of devices belong to PVH dom0 can work? > Because when probe a driver to a normal device, it uses the normal > probe function of pci device, in its callstack, it requests irq > and unmask corresponding ioapic of gsi, then trap into xen and > register gsi finally. > Callstack is(on linux kernel side) pci_device_probe-> > request_threaded_irq-> irq_startup-> __unmask_ioapic-> > io_apic_write, then trap into xen hvmemul_do_io-> > hvm_io_intercept-> hvm_process_io_intercept-> > vioapic_write_indirect-> vioapic_hwdom_map_gsi-> mp_register_gsi. > So that the gsi can be registered. > > Second, why the gsi of passthrough device can't work when dom0 > is PVH? > Because when assign a device to passthrough, it uses the specific > probe function of pciback, in its callstack, it doesn't install a > fake irq handler due to the ISR is not running. So that > mp_register_gsi on Xen side is never called, then the gsi is not > registered. > Callstack is(on linux kernel side) pcistub_probe->pcistub_seize-> > pcistub_init_device-> xen_pcibk_reset_device-> > xen_pcibk_control_isr->isr_on==0. > > Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > Signed-off-by: Huang Rui <ray.huang@amd.com> > Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > --- > xen/arch/x86/hvm/hypercall.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/xen/arch/x86/hvm/hypercall.c b/xen/arch/x86/hvm/hypercall.c > index 03ada3c880bd..cfe82d0f96ed 100644 > --- a/xen/arch/x86/hvm/hypercall.c > +++ b/xen/arch/x86/hvm/hypercall.c > @@ -86,6 +86,7 @@ long hvm_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) > return -ENOSYS; > break; > > + case PHYSDEVOP_setup_gsi: > case PHYSDEVOP_pci_mmcfg_reserved: > case PHYSDEVOP_pci_device_add: > case PHYSDEVOP_pci_device_remove: If you still have concerns about this implementation that allow PHYSDEVOP_setup_gsi for PVH on Xen side and call PHYSDEVOP_setup_gsi when pciback probe the passthrough device. I have another method to solve this gsi is not registered problem. It is to adjust the codes of pciback on linux kernl side. See: diff --git a/drivers/xen/xen-pciback/pci_stub.c b/drivers/xen/xen-pciback/pci_stub.c index 51b3002b085b..db94529e65f9 100644 --- a/drivers/xen/xen-pciback/pci_stub.c +++ b/drivers/xen/xen-pciback/pci_stub.c @@ -445,6 +445,10 @@ static int pcistub_init_device(struct pcistub_device *psdev) err = pci_enable_device(dev); if (err) goto config_release; + else { + dev_data->enable_intx = 1; + xen_pcibk_control_isr(dev, 0); + } During pcistub_init_device, once pcidev is enabled(through pci_enable_device), I enable the isr for pciback, so that the fake irq handler can be installed and then gsi can be registered. In the end of pcistub_init_device, original code calls xen_pcibk_reset_device to disable isr and pcidev, so the fake irq handler will be freed. Then like nothing happened. Do you think this method is feasible? If so, we don't need this patch anymore. Looking forward to getting your input.
On Mon, 8 Jul 2024, Jiqian Chen wrote: > The gsi of a passthrough device must be configured for it to be > able to be mapped into a hvm domU. > But When dom0 is PVH, the gsis may not get registered(see below > clarification), it causes the info of apic, pin and irq not be > added into irq_2_pin list, and the handler of irq_desc is not set, > then when passthrough a device, setting ioapic affinity and vector > will fail. > > To fix above problem, on Linux kernel side, a new code will > need to call PHYSDEVOP_setup_gsi for passthrough devices to > register gsi when dom0 is PVH. > > So, add PHYSDEVOP_setup_gsi into hvm_physdev_op for above > purpose. > > Clarify two questions: > First, why the gsi of devices belong to PVH dom0 can work? > Because when probe a driver to a normal device, it uses the normal > probe function of pci device, in its callstack, it requests irq > and unmask corresponding ioapic of gsi, then trap into xen and > register gsi finally. > Callstack is(on linux kernel side) pci_device_probe-> > request_threaded_irq-> irq_startup-> __unmask_ioapic-> > io_apic_write, then trap into xen hvmemul_do_io-> > hvm_io_intercept-> hvm_process_io_intercept-> > vioapic_write_indirect-> vioapic_hwdom_map_gsi-> mp_register_gsi. > So that the gsi can be registered. > > Second, why the gsi of passthrough device can't work when dom0 > is PVH? > Because when assign a device to passthrough, it uses the specific > probe function of pciback, in its callstack, it doesn't install a > fake irq handler due to the ISR is not running. So that > mp_register_gsi on Xen side is never called, then the gsi is not > registered. > Callstack is(on linux kernel side) pcistub_probe->pcistub_seize-> > pcistub_init_device-> xen_pcibk_reset_device-> > xen_pcibk_control_isr->isr_on==0. > > Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> > Signed-off-by: Huang Rui <ray.huang@amd.com> > Signed-off-by: Jiqian Chen <Jiqian.Chen@amd.com> Reviewed-by: Stefano Stabellini <sstabellini@kernel.org> > --- > xen/arch/x86/hvm/hypercall.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/xen/arch/x86/hvm/hypercall.c b/xen/arch/x86/hvm/hypercall.c > index 03ada3c880bd..cfe82d0f96ed 100644 > --- a/xen/arch/x86/hvm/hypercall.c > +++ b/xen/arch/x86/hvm/hypercall.c > @@ -86,6 +86,7 @@ long hvm_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) > return -ENOSYS; > break; > > + case PHYSDEVOP_setup_gsi: > case PHYSDEVOP_pci_mmcfg_reserved: > case PHYSDEVOP_pci_device_add: > case PHYSDEVOP_pci_device_remove: > -- > 2.34.1 >
diff --git a/xen/arch/x86/hvm/hypercall.c b/xen/arch/x86/hvm/hypercall.c index 03ada3c880bd..cfe82d0f96ed 100644 --- a/xen/arch/x86/hvm/hypercall.c +++ b/xen/arch/x86/hvm/hypercall.c @@ -86,6 +86,7 @@ long hvm_physdev_op(int cmd, XEN_GUEST_HANDLE_PARAM(void) arg) return -ENOSYS; break; + case PHYSDEVOP_setup_gsi: case PHYSDEVOP_pci_mmcfg_reserved: case PHYSDEVOP_pci_device_add: case PHYSDEVOP_pci_device_remove: