| Message ID | 20220126124912.90205-1-lingshan.zhu@intel.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | vDPA/ifcvf: implement shared IRQ feature | expand |
On Wed, Jan 26, 2022 at 08:49:08PM +0800, Zhu Lingshan wrote: > It has been observed that on some platforms/devices, there may > not be enough MSI vectors for virtqueues and the config change. > Under such circumstances, the interrupt sources of a device > have to share vectors/IRQs. > > This series implemented a shared IRQ feature for ifcvf. > > Please help review. Given the history, can you please report which tests were performed with this patchset? Which configs tested? Thanks? > Changes from V2: > (1) Fix misuse of nvectors(in ifcvf_alloc_vectors return value)(Michael) > (2) Fix misuse of irq = get_vq_irq() in setup irqbypass(Michael) > (3) Coding style improvements(Michael) > (4) Better naming of device shared irq/shared vq irq > > Changes from V1: > (1) Enable config interrupt when only one vector is allocated(Michael) > (2) Clean vectors/IRQs if failed to request config interrupt > since config interrupt is a must(Michael) > (3) Keep local vdpa_ops, disable irq_bypass by setting IRQ = -EINVAL > for shared IRQ case(Michael) > (4) Improvements on error messages(Michael) > (5) Squash functions implementation patches to the callers(Michael) > > Zhu Lingshan (4): > vDPA/ifcvf: implement IO read/write helpers in the header file > vDPA/ifcvf: implement device MSIX vector allocator > vhost_vdpa: don't setup irq offloading when irq_num < 0 > vDPA/ifcvf: implement shared IRQ feature > > drivers/vdpa/ifcvf/ifcvf_base.c | 67 +++----- > drivers/vdpa/ifcvf/ifcvf_base.h | 60 +++++++- > drivers/vdpa/ifcvf/ifcvf_main.c | 260 ++++++++++++++++++++++++++++---- > drivers/vhost/vdpa.c | 4 + > 4 files changed, 312 insertions(+), 79 deletions(-) > > -- > 2.27.0
On 1/26/2022 10:14 PM, Michael S. Tsirkin wrote: > On Wed, Jan 26, 2022 at 08:49:08PM +0800, Zhu Lingshan wrote: >> It has been observed that on some platforms/devices, there may >> not be enough MSI vectors for virtqueues and the config change. >> Under such circumstances, the interrupt sources of a device >> have to share vectors/IRQs. >> >> This series implemented a shared IRQ feature for ifcvf. >> >> Please help review. > Given the history, can you please report which tests > were performed with this patchset? Which configs tested? > Thanks? Hi Michael, It is ping and netperf tests, and I have set nvectors = 1 and 2 in ifcvf_request_irq(), after ifcvf_alloc_vectors(), to hard coded the number of the allocate vectors. Thanks, Zhu Lingshan > >> Changes from V2: >> (1) Fix misuse of nvectors(in ifcvf_alloc_vectors return value)(Michael) >> (2) Fix misuse of irq = get_vq_irq() in setup irqbypass(Michael) >> (3) Coding style improvements(Michael) >> (4) Better naming of device shared irq/shared vq irq >> >> Changes from V1: >> (1) Enable config interrupt when only one vector is allocated(Michael) >> (2) Clean vectors/IRQs if failed to request config interrupt >> since config interrupt is a must(Michael) >> (3) Keep local vdpa_ops, disable irq_bypass by setting IRQ = -EINVAL >> for shared IRQ case(Michael) >> (4) Improvements on error messages(Michael) >> (5) Squash functions implementation patches to the callers(Michael) >> >> Zhu Lingshan (4): >> vDPA/ifcvf: implement IO read/write helpers in the header file >> vDPA/ifcvf: implement device MSIX vector allocator >> vhost_vdpa: don't setup irq offloading when irq_num < 0 >> vDPA/ifcvf: implement shared IRQ feature >> >> drivers/vdpa/ifcvf/ifcvf_base.c | 67 +++----- >> drivers/vdpa/ifcvf/ifcvf_base.h | 60 +++++++- >> drivers/vdpa/ifcvf/ifcvf_main.c | 260 ++++++++++++++++++++++++++++---- >> drivers/vhost/vdpa.c | 4 + >> 4 files changed, 312 insertions(+), 79 deletions(-) >> >> -- >> 2.27.0
On 1/27/2022 12:31 PM, Zhu, Lingshan wrote: > > > On 1/26/2022 10:14 PM, Michael S. Tsirkin wrote: >> On Wed, Jan 26, 2022 at 08:49:08PM +0800, Zhu Lingshan wrote: >>> It has been observed that on some platforms/devices, there may >>> not be enough MSI vectors for virtqueues and the config change. >>> Under such circumstances, the interrupt sources of a device >>> have to share vectors/IRQs. >>> >>> This series implemented a shared IRQ feature for ifcvf. >>> >>> Please help review. >> Given the history, can you please report which tests >> were performed with this patchset? Which configs tested? >> Thanks? > Hi Michael, > > It is ping and netperf tests, and I have set nvectors = 1 and 2 in > ifcvf_request_irq(), after ifcvf_alloc_vectors(), > to hard coded the number of the allocate vectors. > > Thanks, > Zhu Lingshan We can verify the tests result by checking the requested IRQs for the two VMs(one vhost-vdpa device per VM). (1)when setting nvectors = 1, only one IRQs requested per VM/device, all vqs and the config interrupt share this IRQ. [lszhu@cra01infra01 ~]$ cat /proc/interrupts | grep ifc 241: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 45 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534528-edge ifcvf[0000:01:00.5]-dev-shared-irq 251: 0 0 0 0 0 0 0 0 0 0 0 41 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536576-edge ifcvf[0000:01:00.6]-dev-shared-irq (2)when setting nvectors = 2, two IRQs requested for each VM/device, one for all vqs, the other for the config interrupt [lszhu@cra01infra01 ~]$ cat /proc/interrupts | grep ifc 241: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 39 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534528-edge ifcvf[0000:01:00.5]-vqs-shared-irq 242: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534529-edge ifcvf[0000:01:00.5]-config 251: 0 0 0 0 0 0 0 0 0 0 0 39 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536576-edge ifcvf[0000:01:00.6]-vqs-shared-irq 252: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536577-edge ifcvf[0000:01:00.6]-config (3)when remove nvectors hardcode, the driver allocates enough vectors for the queues and config interrupt, and we do see better performance because irq_bypass is enabled:(too many lines, cut off) [lszhu@cra01infra01 linux]$ cat /proc/interrupts | grep ifc 241: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534528-edge ifcvf[0000:01:00.5]-0 242: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534529-edge ifcvf[0000:01:00.5]-1 243: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534530-edge ifcvf[0000:01:00.5]-2 244: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534531-edge ifcvf[0000:01:00.5]-3 245: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534532-edge ifcvf[0000:01:00.5]-4 246: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534533-edge ifcvf[0000:01:00.5]-5 247: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534534-edge ifcvf[0000:01:00.5]-6 248: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534535-edge ifcvf[0000:01:00.5]-7 249: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534536-edge ifcvf[0000:01:00.5]-8 250: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 534537-edge ifcvf[0000:01:00.5]-config 251: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536576-edge ifcvf[0000:01:00.6]-0 252: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 IR-PCI-MSI 536577-edge ifcvf[0000:01:00.6]-1 Thanks, Zhu Lingshan >> >>> Changes from V2: >>> (1) Fix misuse of nvectors(in ifcvf_alloc_vectors return >>> value)(Michael) >>> (2) Fix misuse of irq = get_vq_irq() in setup irqbypass(Michael) >>> (3) Coding style improvements(Michael) >>> (4) Better naming of device shared irq/shared vq irq >>> >>> Changes from V1: >>> (1) Enable config interrupt when only one vector is allocated(Michael) >>> (2) Clean vectors/IRQs if failed to request config interrupt >>> since config interrupt is a must(Michael) >>> (3) Keep local vdpa_ops, disable irq_bypass by setting IRQ = -EINVAL >>> for shared IRQ case(Michael) >>> (4) Improvements on error messages(Michael) >>> (5) Squash functions implementation patches to the callers(Michael) >>> >>> Zhu Lingshan (4): >>> vDPA/ifcvf: implement IO read/write helpers in the header file >>> vDPA/ifcvf: implement device MSIX vector allocator >>> vhost_vdpa: don't setup irq offloading when irq_num < 0 >>> vDPA/ifcvf: implement shared IRQ feature >>> >>> drivers/vdpa/ifcvf/ifcvf_base.c | 67 +++----- >>> drivers/vdpa/ifcvf/ifcvf_base.h | 60 +++++++- >>> drivers/vdpa/ifcvf/ifcvf_main.c | 260 >>> ++++++++++++++++++++++++++++---- >>> drivers/vhost/vdpa.c | 4 + >>> 4 files changed, 312 insertions(+), 79 deletions(-) >>> >>> -- >>> 2.27.0 >