| Message ID | 1409550114-8186-1-git-send-email-akong@redhat.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Mon, Sep 01, 2014 at 01:41:54PM +0800, Amos Kong wrote: > One VM only has 128 msix interrupt, virtio-config interrupt > has less workload. This patch shares one normal interrupt normal == INT#x? Please don't call it normal. The proper name is "legacy INT#x". So you are trying to use legacy INT#x at the same time with MSI-X? This does not work: the PCI spec says: While enabled for MSI or MSI-X operation, a function is prohibited from using its INTx# pin (if implemented) to request service (MSI, MSI-X, and INTx# are mutually exclusive). does the patch work for you? If it does it might be a (minor) spec violation in kvm. Besides, INT#x really leads to terrible performance because sharing is forced even if there aren't many devices. Why do we need INT#x? How about setting IRQF_SHARED for the config interrupt while using MSI-X? You'd have to read ISR to check that the interrupt was intended for your device. > for configuration between virtio devices. > > Signed-off-by: Amos Kong <akong@redhat.com> > --- > drivers/virtio/virtio_pci.c | 41 ++++++++++++++++------------------------- > 1 file changed, 16 insertions(+), 25 deletions(-) > > diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c > index 3d1463c..b1263b3 100644 > --- a/drivers/virtio/virtio_pci.c > +++ b/drivers/virtio/virtio_pci.c > @@ -52,6 +52,7 @@ struct virtio_pci_device > /* Name strings for interrupts. This size should be enough, > * and I'm too lazy to allocate each name separately. */ > char (*msix_names)[256]; > + char config_msix_name[256]; > /* Number of available vectors */ > unsigned msix_vectors; > /* Vectors allocated, excluding per-vq vectors if any */ > @@ -282,12 +283,6 @@ static void vp_free_vectors(struct virtio_device *vdev) > free_cpumask_var(vp_dev->msix_affinity_masks[i]); > > if (vp_dev->msix_enabled) { > - /* Disable the vector used for configuration */ > - iowrite16(VIRTIO_MSI_NO_VECTOR, > - vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > - /* Flush the write out to device */ > - ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > - > pci_disable_msix(vp_dev->pci_dev); > vp_dev->msix_enabled = 0; > } > @@ -339,24 +334,18 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, > goto error; > vp_dev->msix_enabled = 1; > > - /* Set the vector used for configuration */ > - v = vp_dev->msix_used_vectors; > - snprintf(vp_dev->msix_names[v], sizeof *vp_dev->msix_names, > + /* Set shared IRQ for configuration */ > + snprintf(vp_dev->config_msix_name, sizeof(*vp_dev->msix_names), > "%s-config", name); > - err = request_irq(vp_dev->msix_entries[v].vector, > - vp_config_changed, 0, vp_dev->msix_names[v], > + err = request_irq(vp_dev->pci_dev->irq, > + vp_config_changed, > + IRQF_SHARED, > + vp_dev->config_msix_name, > vp_dev); > - if (err) > - goto error; > - ++vp_dev->msix_used_vectors; > - > - iowrite16(v, vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > - /* Verify we had enough resources to assign the vector */ > - v = ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > - if (v == VIRTIO_MSI_NO_VECTOR) { > - err = -EBUSY; > + if (!err) > + vp_dev->intx_enabled = 1; > + else > goto error; > - } > > if (!per_vq_vectors) { > /* Shared vector for all VQs */ > @@ -535,14 +524,16 @@ static int vp_try_to_find_vqs(struct virtio_device *vdev, unsigned nvqs, > goto error_request; > } else { > if (per_vq_vectors) { > - /* Best option: one for change interrupt, one per vq. */ > - nvectors = 1; > + /* Best option: one normal interrupt for change, > + one msix per vq. */ > + nvectors = 0; > for (i = 0; i < nvqs; ++i) > if (callbacks[i]) > ++nvectors; > } else { > - /* Second best: one for change, shared for all vqs. */ > - nvectors = 2; > + /* Second best: one normal interrupt for > + change, share one msix for all vqs. */ > + nvectors = 1; > } > > err = vp_request_msix_vectors(vdev, nvectors, per_vq_vectors); > -- > 1.9.3 -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Sep 01, 2014 at 09:37:30AM +0300, Michael S. Tsirkin wrote: > Hi Michael, > On Mon, Sep 01, 2014 at 01:41:54PM +0800, Amos Kong wrote: > > One VM only has 128 msix interrupt, virtio-config interrupt > > has less workload. This patch shares one normal interrupt Thanks for your quick reply. > normal == INT#x? Please don't call it normal. The proper name is > "legacy INT#x". OK > So you are trying to use legacy INT#x at the same time > with MSI-X? This does not work: the PCI spec says: > While enabled for MSI or MSI-X > operation, a function is prohibited from using its INTx# pin (if > implemented) to request > service (MSI, MSI-X, and INTx# are mutually exclusive). It means we can't use interrupt and triggered-interrupt together for one PCI device. I will study this problem. > does the patch work for you? If it does it might be a (minor) spec > violation in kvm. I did some basic testing (multiple nics, scp, ping, etc), it works. > Besides, INT#x really leads to terrible performance because > sharing is forced even if there aren't many devices. > > Why do we need INT#x? > How about setting IRQF_SHARED for the config interrupt > while using MSI-X? > You'd have to read ISR to check that the interrupt was > intended for your device. I have a draft patch to share one MSI-X for all virtio-config, it has some problem in hotplugging devices. I will continue this way. > > for configuration between virtio devices. > > > > Signed-off-by: Amos Kong <akong@redhat.com> > > --- > > drivers/virtio/virtio_pci.c | 41 ++++++++++++++++------------------------- > > 1 file changed, 16 insertions(+), 25 deletions(-) > > > > diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c > > index 3d1463c..b1263b3 100644 > > --- a/drivers/virtio/virtio_pci.c > > +++ b/drivers/virtio/virtio_pci.c > > @@ -52,6 +52,7 @@ struct virtio_pci_device > > /* Name strings for interrupts. This size should be enough, > > * and I'm too lazy to allocate each name separately. */ > > char (*msix_names)[256]; > > + char config_msix_name[256]; > > /* Number of available vectors */ > > unsigned msix_vectors; > > /* Vectors allocated, excluding per-vq vectors if any */ > > @@ -282,12 +283,6 @@ static void vp_free_vectors(struct virtio_device *vdev) > > free_cpumask_var(vp_dev->msix_affinity_masks[i]); > > > > if (vp_dev->msix_enabled) { > > - /* Disable the vector used for configuration */ > > - iowrite16(VIRTIO_MSI_NO_VECTOR, > > - vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > - /* Flush the write out to device */ > > - ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > - > > pci_disable_msix(vp_dev->pci_dev); > > vp_dev->msix_enabled = 0; > > } > > @@ -339,24 +334,18 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, > > goto error; > > vp_dev->msix_enabled = 1; > > > > - /* Set the vector used for configuration */ > > - v = vp_dev->msix_used_vectors; > > - snprintf(vp_dev->msix_names[v], sizeof *vp_dev->msix_names, > > + /* Set shared IRQ for configuration */ > > + snprintf(vp_dev->config_msix_name, sizeof(*vp_dev->msix_names), > > "%s-config", name); > > - err = request_irq(vp_dev->msix_entries[v].vector, > > - vp_config_changed, 0, vp_dev->msix_names[v], > > + err = request_irq(vp_dev->pci_dev->irq, > > + vp_config_changed, > > + IRQF_SHARED, > > + vp_dev->config_msix_name, > > vp_dev); > > - if (err) > > - goto error; > > - ++vp_dev->msix_used_vectors; > > - > > - iowrite16(v, vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > - /* Verify we had enough resources to assign the vector */ > > - v = ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > - if (v == VIRTIO_MSI_NO_VECTOR) { > > - err = -EBUSY; > > + if (!err) > > + vp_dev->intx_enabled = 1; > > + else > > goto error; > > - } > > > > if (!per_vq_vectors) { > > /* Shared vector for all VQs */ > > @@ -535,14 +524,16 @@ static int vp_try_to_find_vqs(struct virtio_device *vdev, unsigned nvqs, > > goto error_request; > > } else { > > if (per_vq_vectors) { > > - /* Best option: one for change interrupt, one per vq. */ > > - nvectors = 1; > > + /* Best option: one normal interrupt for change, > > + one msix per vq. */ > > + nvectors = 0; > > for (i = 0; i < nvqs; ++i) > > if (callbacks[i]) > > ++nvectors; > > } else { > > - /* Second best: one for change, shared for all vqs. */ > > - nvectors = 2; > > + /* Second best: one normal interrupt for > > + change, share one msix for all vqs. */ > > + nvectors = 1; > > } > > > > err = vp_request_msix_vectors(vdev, nvectors, per_vq_vectors); > > -- > > 1.9.3 > -- > To unsubscribe from this list: send the line "unsubscribe kvm" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Sep 01, 2014 at 03:58:02PM +0800, Amos Kong wrote: > On Mon, Sep 01, 2014 at 09:37:30AM +0300, Michael S. Tsirkin wrote: > > > > Hi Michael, > > > On Mon, Sep 01, 2014 at 01:41:54PM +0800, Amos Kong wrote: > > > One VM only has 128 msix interrupt, virtio-config interrupt > > > has less workload. This patch shares one normal interrupt > > Thanks for your quick reply. > > > normal == INT#x? Please don't call it normal. The proper name is > > "legacy INT#x". > > OK > > > So you are trying to use legacy INT#x at the same time > > with MSI-X? This does not work: the PCI spec says: > > While enabled for MSI or MSI-X > > operation, a function is prohibited from using its INTx# pin (if > > implemented) to request > > service (MSI, MSI-X, and INTx# are mutually exclusive). > > It means we can't use interrupt and triggered-interrupt together for > one PCI device. I will study this problem. > > > does the patch work for you? If it does it might be a (minor) spec > > violation in kvm. > > I did some basic testing (multiple nics, scp, ping, etc), it works. It's quite likely there were no config interrupts in your basic test. Trigger some event that requires a config interrupt to function. For example, drop link and see if guest notices this. > > Besides, INT#x really leads to terrible performance because > > sharing is forced even if there aren't many devices. > > > > Why do we need INT#x? > > How about setting IRQF_SHARED for the config interrupt > > while using MSI-X? > > You'd have to read ISR to check that the interrupt was > > intended for your device. > > I have a draft patch to share one MSI-X for all virtio-config, it has > some problem in hotplugging devices. I will continue this way. > > > > for configuration between virtio devices. > > > > > > Signed-off-by: Amos Kong <akong@redhat.com> > > > --- > > > drivers/virtio/virtio_pci.c | 41 ++++++++++++++++------------------------- > > > 1 file changed, 16 insertions(+), 25 deletions(-) > > > > > > diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c > > > index 3d1463c..b1263b3 100644 > > > --- a/drivers/virtio/virtio_pci.c > > > +++ b/drivers/virtio/virtio_pci.c > > > @@ -52,6 +52,7 @@ struct virtio_pci_device > > > /* Name strings for interrupts. This size should be enough, > > > * and I'm too lazy to allocate each name separately. */ > > > char (*msix_names)[256]; > > > + char config_msix_name[256]; > > > /* Number of available vectors */ > > > unsigned msix_vectors; > > > /* Vectors allocated, excluding per-vq vectors if any */ > > > @@ -282,12 +283,6 @@ static void vp_free_vectors(struct virtio_device *vdev) > > > free_cpumask_var(vp_dev->msix_affinity_masks[i]); > > > > > > if (vp_dev->msix_enabled) { > > > - /* Disable the vector used for configuration */ > > > - iowrite16(VIRTIO_MSI_NO_VECTOR, > > > - vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > > - /* Flush the write out to device */ > > > - ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > > - > > > pci_disable_msix(vp_dev->pci_dev); > > > vp_dev->msix_enabled = 0; > > > } > > > @@ -339,24 +334,18 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, > > > goto error; > > > vp_dev->msix_enabled = 1; > > > > > > - /* Set the vector used for configuration */ > > > - v = vp_dev->msix_used_vectors; > > > - snprintf(vp_dev->msix_names[v], sizeof *vp_dev->msix_names, > > > + /* Set shared IRQ for configuration */ > > > + snprintf(vp_dev->config_msix_name, sizeof(*vp_dev->msix_names), > > > "%s-config", name); > > > - err = request_irq(vp_dev->msix_entries[v].vector, > > > - vp_config_changed, 0, vp_dev->msix_names[v], > > > + err = request_irq(vp_dev->pci_dev->irq, > > > + vp_config_changed, > > > + IRQF_SHARED, > > > + vp_dev->config_msix_name, > > > vp_dev); > > > - if (err) > > > - goto error; > > > - ++vp_dev->msix_used_vectors; > > > - > > > - iowrite16(v, vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > > - /* Verify we had enough resources to assign the vector */ > > > - v = ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); > > > - if (v == VIRTIO_MSI_NO_VECTOR) { > > > - err = -EBUSY; > > > + if (!err) > > > + vp_dev->intx_enabled = 1; > > > + else > > > goto error; > > > - } > > > > > > if (!per_vq_vectors) { > > > /* Shared vector for all VQs */ > > > @@ -535,14 +524,16 @@ static int vp_try_to_find_vqs(struct virtio_device *vdev, unsigned nvqs, > > > goto error_request; > > > } else { > > > if (per_vq_vectors) { > > > - /* Best option: one for change interrupt, one per vq. */ > > > - nvectors = 1; > > > + /* Best option: one normal interrupt for change, > > > + one msix per vq. */ > > > + nvectors = 0; > > > for (i = 0; i < nvqs; ++i) > > > if (callbacks[i]) > > > ++nvectors; > > > } else { > > > - /* Second best: one for change, shared for all vqs. */ > > > - nvectors = 2; > > > + /* Second best: one normal interrupt for > > > + change, share one msix for all vqs. */ > > > + nvectors = 1; > > > } > > > > > > err = vp_request_msix_vectors(vdev, nvectors, per_vq_vectors); > > > -- > > > 1.9.3 > > -- > > To unsubscribe from this list: send the line "unsubscribe kvm" in > > the body of a message to majordomo@vger.kernel.org > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- > Amos. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Monday 01 September 2014 09:37:30, Michael S. Tsirkin wrote: > Why do we need INT#x? > How about setting IRQF_SHARED for the config interrupt > while using MSI-X? You'd have to read ISR to check that the > interrupt was intended for your device. The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X mode. I don't think that you can depend on the device to set the configuration changed bit. The virtio 1.0 spec seems to have fixed that. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Sep 18, 2014 at 09:18:37PM +0200, Stefan Fritsch wrote: > On Monday 01 September 2014 09:37:30, Michael S. Tsirkin wrote: > > Why do we need INT#x? > > How about setting IRQF_SHARED for the config interrupt > > while using MSI-X? You'd have to read ISR to check that the > > interrupt was intended for your device. > > The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X mode. I > don't think that you can depend on the device to set the configuration > changed bit. > The virtio 1.0 spec seems to have fixed that. Yes, virtio 0.9.5 has this bug. But in practice qemu always set this bit, so for qemu we could do that unconditionally. Pekka's lkvm tool doesn't unfortunately. It's easy to fix that, but it would be nicer to additionally probe for old versions of the tool, and disable IRQF_SHARED in that case. To complicate things, lkvm does not use a distinct subsystem vendor ID, in spite of the fact the virtio spec always required this explicitly. After poking at things, we could probably try and distinguish old lkmv based on bar sizes. I think lkvm has: #define IOPORT_SIZE 0x400 this is the size of the IO bar (bar0) correct? Qemu's BAR is smaller. So if 1. new versions of lkvm are fixed to always set ISR on config change even when msi is enabled 2 lkvm folks can promise not to make bar0 size smaller *before* fixing (1) then we could use the heuristic: bar size == 0x400 to clear IRQF_SHARED. Cc some lkvm folks for all of the above: would you guys be happier with some other heuristic? I'd like to note that lkvm really should get some vendor to request and then donate a subsystem vendor id (registered with pci sig) for their use, instead of pretending they are qemu. AFAIK a subsystem vendor id does not cost money to register, but only pci sig members can do this, and membership costs $3000. Maybe we should combine all this with checking subsystem vendor id, and only implement the optimization if it matches qemu, for now. This needs some thought.
On Sunday 21 September 2014 11:09:14, Michael S. Tsirkin wrote: > On Thu, Sep 18, 2014 at 09:18:37PM +0200, Stefan Fritsch wrote: > > On Monday 01 September 2014 09:37:30, Michael S. Tsirkin wrote: > > > Why do we need INT#x? > > > How about setting IRQF_SHARED for the config interrupt > > > while using MSI-X? You'd have to read ISR to check that the > > > interrupt was intended for your device. > > > > > > > > The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X > > mode. I don't think that you can depend on the device to set the > > configuration changed bit. > > The virtio 1.0 spec seems to have fixed that. > > Yes, virtio 0.9.5 has this bug. But in practice qemu always set this > bit, so for qemu we could do that unconditionally. Pekka's lkvm > tool doesn't unfortunately. It's easy to fix that, but it would be > nicer to additionally probe for old versions of the tool, and > disable IRQF_SHARED in that case. What about other implementations? I think Linux should try to conform to the spec so that all device implementations which conform to the spec just work. One implementation that comes to mind is virtualbox. But from a quick look at the source, it seems that it sets the ISR bit always, too. And it uses qemu's subsystem vendor id. But there are other implementations. For example bhyve. > AFAIK a subsystem vendor id does not cost money to register, but > only pci sig members can do this, and membership costs $3000. > Maybe we should combine all this with checking subsystem vendor id, > and only implement the optimization if it matches qemu, for now. > This needs some thought. Maybe the virtio spec should include a way to query the vendor that does not involve the pci sig. Maybe use a string? Then no registry would be necessary. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Sep 21, 2014 at 11:36:44AM +0200, Stefan Fritsch wrote: > On Sunday 21 September 2014 11:09:14, Michael S. Tsirkin wrote: > > On Thu, Sep 18, 2014 at 09:18:37PM +0200, Stefan Fritsch wrote: > > > On Monday 01 September 2014 09:37:30, Michael S. Tsirkin wrote: > > > > Why do we need INT#x? > > > > How about setting IRQF_SHARED for the config interrupt > > > > while using MSI-X? You'd have to read ISR to check that the > > > > interrupt was intended for your device. > > > > > > > > > > > > The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X > > > mode. I don't think that you can depend on the device to set the > > > configuration changed bit. > > > The virtio 1.0 spec seems to have fixed that. > > > > Yes, virtio 0.9.5 has this bug. But in practice qemu always set this > > bit, so for qemu we could do that unconditionally. Pekka's lkvm > > tool doesn't unfortunately. It's easy to fix that, but it would be > > nicer to additionally probe for old versions of the tool, and > > disable IRQF_SHARED in that case. > > What about other implementations? I think Linux should try to conform > to the spec so that all device implementations which conform to the > spec just work. > > One implementation that comes to mind is virtualbox. But from a quick > look at the source, it seems that it sets the ISR bit always, too. And > it uses qemu's subsystem vendor id. > > But there are other implementations. For example bhyve. I couldn't find any code in bhyve that sets VTCFG_ISR_CONF_CHANGED. Maybe it doesn't generate config changed interrupts? bhyve sets subsystem vendor to 0 apparently? We could use that to detect it. But maybe we should just make it a 1.0 only feature. > > > AFAIK a subsystem vendor id does not cost money to register, but > > only pci sig members can do this, and membership costs $3000. > > Maybe we should combine all this with checking subsystem vendor id, > > and only implement the optimization if it matches qemu, for now. > > This needs some thought. > > Maybe the virtio spec should include a way to query the vendor that > does not involve the pci sig. Maybe use a string? Then no registry > would be necessary. We can make the requirement for the vendor specific ID stronger in 1.0, SHOULD instead of MAY. But it seems that people will still copy-paste working code across hypervisors, I'm not sure this can be helped.
On 09/21/2014 04:09 AM, Michael S. Tsirkin wrote: >> The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X mode. I >> > don't think that you can depend on the device to set the configuration >> > changed bit. >> > The virtio 1.0 spec seems to have fixed that. > Yes, virtio 0.9.5 has this bug. But in practice qemu always set this > bit, so for qemu we could do that unconditionally. Pekka's lkvm tool > doesn't unfortunately. It's easy to fix that, but it would be nicer to > additionally probe for old versions of the tool, and disable IRQF_SHARED > in that case. > > To complicate things, lkvm does not use a distinct subsystem vendor ID, > in spite of the fact the virtio spec always required this explicitly. I think I may be a bit confused here, but AFAIK we do set subsystem vendor ID properly for our virtio-pci devices? vpci->pci_hdr = (struct pci_device_header) { .vendor_id = cpu_to_le16(PCI_VENDOR_ID_REDHAT_QUMRANET), .device_id = cpu_to_le16(device_id), [...] .subsys_vendor_id = cpu_to_le16(PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET), Thanks, Sasha -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Sep 21, 2014 at 09:47:51AM -0400, Sasha Levin wrote: > On 09/21/2014 04:09 AM, Michael S. Tsirkin wrote: > >> The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X mode. I > >> > don't think that you can depend on the device to set the configuration > >> > changed bit. > >> > The virtio 1.0 spec seems to have fixed that. > > Yes, virtio 0.9.5 has this bug. But in practice qemu always set this > > bit, so for qemu we could do that unconditionally. Pekka's lkvm tool > > doesn't unfortunately. It's easy to fix that, but it would be nicer to > > additionally probe for old versions of the tool, and disable IRQF_SHARED > > in that case. > > > > To complicate things, lkvm does not use a distinct subsystem vendor ID, > > in spite of the fact the virtio spec always required this explicitly. > > I think I may be a bit confused here, but AFAIK we do set subsystem vendor > ID properly for our virtio-pci devices? > > vpci->pci_hdr = (struct pci_device_header) { > .vendor_id = cpu_to_le16(PCI_VENDOR_ID_REDHAT_QUMRANET), > .device_id = cpu_to_le16(device_id), > [...] > .subsys_vendor_id = cpu_to_le16(PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET), > > > Thanks, > Sasha Yes but the spec says: The Subsystem Vendor ID should reflect the PCI Vendor ID of the environment. IOW lkvm shouldn't reuse the ID from qemu, it should have its own (qemu and lkvm hypervisors being a different environment). virtio 1.0 have weakened this requirement: The PCI Subsystem Vendor ID and the PCI Subsystem Device ID MAY reflect the PCI Vendor and Device ID of the environment (for informational purposes by the driver). I reasoned that since it's for informational purposes only, there's no reason to make it a SHOULD. It might or might not be a good idea to change it back, worth considering.
On 09/21/2014 11:02 AM, Michael S. Tsirkin wrote: > On Sun, Sep 21, 2014 at 09:47:51AM -0400, Sasha Levin wrote: >> > On 09/21/2014 04:09 AM, Michael S. Tsirkin wrote: >>>> > >> The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X mode. I >>>>> > >> > don't think that you can depend on the device to set the configuration >>>>> > >> > changed bit. >>>>> > >> > The virtio 1.0 spec seems to have fixed that. >>> > > Yes, virtio 0.9.5 has this bug. But in practice qemu always set this >>> > > bit, so for qemu we could do that unconditionally. Pekka's lkvm tool >>> > > doesn't unfortunately. It's easy to fix that, but it would be nicer to >>> > > additionally probe for old versions of the tool, and disable IRQF_SHARED >>> > > in that case. >>> > > >>> > > To complicate things, lkvm does not use a distinct subsystem vendor ID, >>> > > in spite of the fact the virtio spec always required this explicitly. >> > >> > I think I may be a bit confused here, but AFAIK we do set subsystem vendor >> > ID properly for our virtio-pci devices? >> > >> > vpci->pci_hdr = (struct pci_device_header) { >> > .vendor_id = cpu_to_le16(PCI_VENDOR_ID_REDHAT_QUMRANET), >> > .device_id = cpu_to_le16(device_id), >> > [...] >> > .subsys_vendor_id = cpu_to_le16(PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET), >> > >> > >> > Thanks, >> > Sasha > > Yes but the spec says: > The Subsystem Vendor ID should reflect the PCI Vendor ID of the environment. > > IOW lkvm shouldn't reuse the ID from qemu, it should have its own > (qemu and lkvm hypervisors being a different environment). > > virtio 1.0 have weakened this requirement: > The PCI Subsystem Vendor ID and the PCI Subsystem Device ID MAY > reflect the PCI Vendor and Device > ID of the environment (for informational purposes by the driver). > > I reasoned that since it's for informational purposes only, there's no > reason to make it a SHOULD. > > It might or might not be a good idea to change it back, worth > considering. Ow. The 0.9.5 spec also says: "(it's currently only used for informational purposes by the guest)." That and the combination of "should" rather then "must" (recommended rather than required) prompted us to just put something that works in there and leave it be. Thanks, Sasha -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Sun, Sep 21, 2014 at 11:19:36AM -0400, Sasha Levin wrote: > On 09/21/2014 11:02 AM, Michael S. Tsirkin wrote: > > On Sun, Sep 21, 2014 at 09:47:51AM -0400, Sasha Levin wrote: > >> > On 09/21/2014 04:09 AM, Michael S. Tsirkin wrote: > >>>> > >> The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X mode. I > >>>>> > >> > don't think that you can depend on the device to set the configuration > >>>>> > >> > changed bit. > >>>>> > >> > The virtio 1.0 spec seems to have fixed that. > >>> > > Yes, virtio 0.9.5 has this bug. But in practice qemu always set this > >>> > > bit, so for qemu we could do that unconditionally. Pekka's lkvm tool > >>> > > doesn't unfortunately. It's easy to fix that, but it would be nicer to > >>> > > additionally probe for old versions of the tool, and disable IRQF_SHARED > >>> > > in that case. > >>> > > > >>> > > To complicate things, lkvm does not use a distinct subsystem vendor ID, > >>> > > in spite of the fact the virtio spec always required this explicitly. > >> > > >> > I think I may be a bit confused here, but AFAIK we do set subsystem vendor > >> > ID properly for our virtio-pci devices? > >> > > >> > vpci->pci_hdr = (struct pci_device_header) { > >> > .vendor_id = cpu_to_le16(PCI_VENDOR_ID_REDHAT_QUMRANET), > >> > .device_id = cpu_to_le16(device_id), > >> > [...] > >> > .subsys_vendor_id = cpu_to_le16(PCI_SUBSYSTEM_VENDOR_ID_REDHAT_QUMRANET), > >> > > >> > > >> > Thanks, > >> > Sasha > > > > Yes but the spec says: > > The Subsystem Vendor ID should reflect the PCI Vendor ID of the environment. > > > > IOW lkvm shouldn't reuse the ID from qemu, it should have its own > > (qemu and lkvm hypervisors being a different environment). > > > > virtio 1.0 have weakened this requirement: > > The PCI Subsystem Vendor ID and the PCI Subsystem Device ID MAY > > reflect the PCI Vendor and Device > > ID of the environment (for informational purposes by the driver). > > > > I reasoned that since it's for informational purposes only, there's no > > reason to make it a SHOULD. > > > > It might or might not be a good idea to change it back, worth > > considering. > > Ow. The 0.9.5 spec also says: > > "(it's currently only used for informational purposes by the guest)." > > That and the combination of "should" rather then "must" (recommended rather than > required) prompted us to just put something that works in there and leave it be. > > > Thanks, > Sasha Note "currently" as well as "should" which means "before you don't, make sure you understand the implications".
On Sunday 21 September 2014 13:21:06, Michael S. Tsirkin wrote: > On Sun, Sep 21, 2014 at 11:36:44AM +0200, Stefan Fritsch wrote: > > On Sunday 21 September 2014 11:09:14, Michael S. Tsirkin wrote: > > > On Thu, Sep 18, 2014 at 09:18:37PM +0200, Stefan Fritsch wrote: > > > > On Monday 01 September 2014 09:37:30, Michael S. Tsirkin wrote: > > > > > Why do we need INT#x? > > > > > How about setting IRQF_SHARED for the config interrupt > > > > > while using MSI-X? You'd have to read ISR to check that the > > > > > interrupt was intended for your device. > > > > > > > > > > > > > > > > The virtio 0.9.5 spec says that ISR is "unused" when in MSI-X > > > > mode. I don't think that you can depend on the device to set > > > > the > > > > configuration changed bit. > > > > The virtio 1.0 spec seems to have fixed that. > > > > > > > > > > > > Yes, virtio 0.9.5 has this bug. But in practice qemu always set > > > this bit, so for qemu we could do that > > > unconditionally. Pekka's lkvm tool doesn't > > > unfortunately. It's easy to fix that, but it would be nicer to > > > additionally probe for old versions of the tool, and disable > > > IRQF_SHARED in that case. > > > > > > > > What about other implementations? I think Linux should try to > > conform to the spec so that all device implementations which > > conform to the spec just work. > > > > > > > > One implementation that comes to mind is virtualbox. But from a > > quick look at the source, it seems that it sets the ISR bit > > always, too. And it uses qemu's subsystem vendor id. > > > > > > > > But there are other implementations. For example bhyve. > > I couldn't find any code in bhyve that sets VTCFG_ISR_CONF_CHANGED. > Maybe it doesn't generate config changed interrupts? > > bhyve sets subsystem vendor to 0 apparently? > We could use that to detect it. My point was that there are many virtio implementations by now and you can't assume you know all of them. > But maybe we should just make it a 1.0 only feature. FWIW, I think that would be the better option. -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/virtio/virtio_pci.c b/drivers/virtio/virtio_pci.c index 3d1463c..b1263b3 100644 --- a/drivers/virtio/virtio_pci.c +++ b/drivers/virtio/virtio_pci.c @@ -52,6 +52,7 @@ struct virtio_pci_device /* Name strings for interrupts. This size should be enough, * and I'm too lazy to allocate each name separately. */ char (*msix_names)[256]; + char config_msix_name[256]; /* Number of available vectors */ unsigned msix_vectors; /* Vectors allocated, excluding per-vq vectors if any */ @@ -282,12 +283,6 @@ static void vp_free_vectors(struct virtio_device *vdev) free_cpumask_var(vp_dev->msix_affinity_masks[i]); if (vp_dev->msix_enabled) { - /* Disable the vector used for configuration */ - iowrite16(VIRTIO_MSI_NO_VECTOR, - vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); - /* Flush the write out to device */ - ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); - pci_disable_msix(vp_dev->pci_dev); vp_dev->msix_enabled = 0; } @@ -339,24 +334,18 @@ static int vp_request_msix_vectors(struct virtio_device *vdev, int nvectors, goto error; vp_dev->msix_enabled = 1; - /* Set the vector used for configuration */ - v = vp_dev->msix_used_vectors; - snprintf(vp_dev->msix_names[v], sizeof *vp_dev->msix_names, + /* Set shared IRQ for configuration */ + snprintf(vp_dev->config_msix_name, sizeof(*vp_dev->msix_names), "%s-config", name); - err = request_irq(vp_dev->msix_entries[v].vector, - vp_config_changed, 0, vp_dev->msix_names[v], + err = request_irq(vp_dev->pci_dev->irq, + vp_config_changed, + IRQF_SHARED, + vp_dev->config_msix_name, vp_dev); - if (err) - goto error; - ++vp_dev->msix_used_vectors; - - iowrite16(v, vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); - /* Verify we had enough resources to assign the vector */ - v = ioread16(vp_dev->ioaddr + VIRTIO_MSI_CONFIG_VECTOR); - if (v == VIRTIO_MSI_NO_VECTOR) { - err = -EBUSY; + if (!err) + vp_dev->intx_enabled = 1; + else goto error; - } if (!per_vq_vectors) { /* Shared vector for all VQs */ @@ -535,14 +524,16 @@ static int vp_try_to_find_vqs(struct virtio_device *vdev, unsigned nvqs, goto error_request; } else { if (per_vq_vectors) { - /* Best option: one for change interrupt, one per vq. */ - nvectors = 1; + /* Best option: one normal interrupt for change, + one msix per vq. */ + nvectors = 0; for (i = 0; i < nvqs; ++i) if (callbacks[i]) ++nvectors; } else { - /* Second best: one for change, shared for all vqs. */ - nvectors = 2; + /* Second best: one normal interrupt for + change, share one msix for all vqs. */ + nvectors = 1; } err = vp_request_msix_vectors(vdev, nvectors, per_vq_vectors);
One VM only has 128 msix interrupt, virtio-config interrupt has less workload. This patch shares one normal interrupt for configuration between virtio devices. Signed-off-by: Amos Kong <akong@redhat.com> --- drivers/virtio/virtio_pci.c | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-)