| Message ID | 20250128185705.1609038-1-david@redhat.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | s390x: support virtio-mem-pci | expand |
This series has been successfully tested in s390x. Here the detailed test steps: Boot up a VM already containing a memory object /home/qemu/build/qemu-system-s390x \ ... -m 4G,maxmem=20G \ -object memory-backend-ram,id=mem0,size=16G,reserve=off \ ... Check the memory devices (qemu) info memory-devices (qemu) Check the PCI devices in the VM [root@localhost ~]# lspci -v [root@localhost ~]# Hotplug a virtio-mem-pci device and check the memory devices (qemu) device_add virtio-mem-pci,id=vmem0,memdev=mem0,requested-size=1G (qemu) info memory-devices Memory device [virtio-mem]: "vmem0" memaddr: 0x100000000 node: 0 requested-size: 1073741824 size: 1073741824 max-size: 17179869184 block-size: 1048576 memdev: /objects/mem0 Check the PCI device is correct in the guest [root@localhost ~]# lspci -v 0001:00:00.0 Unclassified device [00ff]: Red Hat, Inc. virtio-mem (rev 01) Subsystem: Red Hat, Inc. Device 1100 Physical Slot: 00000000 Flags: bus master, fast devsel, latency 0, IOMMU group 0 Memory at 4000000000000000 (32-bit, non-prefetchable) [virtual] [size=4K] Memory at 4001000000000000 (64-bit, prefetchable) [virtual] [size=16K] Capabilities: [98] MSI-X: Enable+ Count=2 Masked- Capabilities: [84] Vendor Specific Information: VirtIO: <unknown> Capabilities: [70] Vendor Specific Information: VirtIO: Notify Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg Capabilities: [50] Vendor Specific Information: VirtIO: ISR Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg Kernel driver in use: virtio-pci Resize the virtio_mem device and verify the new size (qemu) qom-set vmem0 requested-size 4G (qemu) info memory-devices Memory device [virtio-mem]: "vmem0" memaddr: 0x100000000 node: 0 requested-size: 4294967296 size: 4294967296 max-size: 17179869184 block-size: 1048576 memdev: /objects/mem0 Finally, try to perform a failed unplug, resize the device to 0, and unplug it seamlessly: (qemu) device_del vmem0 Error: virtio-mem device cannot get unplugged while some of its memory is still plugged (qemu) qom-set vmem0 requested-size 0 (qemu) device_del vmem0 Tested-by: Mario Casquero <mcasquer@redhat.com> On Tue, Jan 28, 2025 at 7:57 PM David Hildenbrand <david@redhat.com> wrote: > > This is based-on [1], which adds MSI-X support to virtio-balloon-pci, > but can be applied independently. > > Turns out it is fairly easy to get virtio-mem-pci running on s390x. We > only have to add MSI-X support to virtio-mem-pci, and wire-up the > (un)plugging in the machine. > > Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems > to be working as expected. > > The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM > for it to work. > > [1] https://lkml.kernel.org/r/20250115161425.246348-1-arbab@linux.ibm.com > > v1 -> v2: > * There are no transitional/non_transitional devices for virtio-mem > * Spell out removal of "return;" in second patch > > Cc: Eduardo Habkost <eduardo@habkost.net> > Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com> > Cc: Richard Henderson <richard.henderson@linaro.org> > Cc: David Hildenbrand <david@redhat.com> > Cc: Ilya Leoshkevich <iii@linux.ibm.com> > Cc: Halil Pasic <pasic@linux.ibm.com> > Cc: Christian Borntraeger <borntraeger@linux.ibm.com> > Cc: Eric Farman <farman@linux.ibm.com> > Cc: Thomas Huth <thuth@redhat.com> > Cc: "Michael S. Tsirkin" <mst@redhat.com> > Cc: Cornelia Huck <cohuck@redhat.com> > Cc: Boris Fiuczynski <fiuczy@linux.ibm.com> > Cc: Michal Privoznik <mprivozn@redhat.com> > Cc: Mario Casquero <mcasquer@redhat.com> > > David Hildenbrand (2): > virtio-mem-pci: Allow setting nvectors, so we can use MSI-X > s390x/s390-virtio-ccw: Support plugging PCI-based virtio memory > devices > > hw/core/machine.c | 1 + > hw/s390x/s390-virtio-ccw.c | 20 ++++++++++++++------ > hw/virtio/virtio-mem-pci.c | 12 ++++++++++++ > 3 files changed, 27 insertions(+), 6 deletions(-) > > -- > 2.48.1 >
On Tue, Jan 28, 2025 at 07:57:03PM +0100, David Hildenbrand wrote: > This is based-on [1], which adds MSI-X support to virtio-balloon-pci, > but can be applied independently. > > Turns out it is fairly easy to get virtio-mem-pci running on s390x. We > only have to add MSI-X support to virtio-mem-pci, and wire-up the > (un)plugging in the machine. > > Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems > to be working as expected. > > The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM > for it to work. > > [1] https://lkml.kernel.org/r/20250115161425.246348-1-arbab@linux.ibm.com Fails CI: https://gitlab.com/mstredhat/qemu/-/jobs/9202574981 > v1 -> v2: > * There are no transitional/non_transitional devices for virtio-mem > * Spell out removal of "return;" in second patch > > Cc: Eduardo Habkost <eduardo@habkost.net> > Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com> > Cc: Richard Henderson <richard.henderson@linaro.org> > Cc: David Hildenbrand <david@redhat.com> > Cc: Ilya Leoshkevich <iii@linux.ibm.com> > Cc: Halil Pasic <pasic@linux.ibm.com> > Cc: Christian Borntraeger <borntraeger@linux.ibm.com> > Cc: Eric Farman <farman@linux.ibm.com> > Cc: Thomas Huth <thuth@redhat.com> > Cc: "Michael S. Tsirkin" <mst@redhat.com> > Cc: Cornelia Huck <cohuck@redhat.com> > Cc: Boris Fiuczynski <fiuczy@linux.ibm.com> > Cc: Michal Privoznik <mprivozn@redhat.com> > Cc: Mario Casquero <mcasquer@redhat.com> > > David Hildenbrand (2): > virtio-mem-pci: Allow setting nvectors, so we can use MSI-X > s390x/s390-virtio-ccw: Support plugging PCI-based virtio memory > devices > > hw/core/machine.c | 1 + > hw/s390x/s390-virtio-ccw.c | 20 ++++++++++++++------ > hw/virtio/virtio-mem-pci.c | 12 ++++++++++++ > 3 files changed, 27 insertions(+), 6 deletions(-) > > -- > 2.48.1
On 21.02.25 00:24, Michael S. Tsirkin wrote: > On Tue, Jan 28, 2025 at 07:57:03PM +0100, David Hildenbrand wrote: >> This is based-on [1], which adds MSI-X support to virtio-balloon-pci, >> but can be applied independently. >> >> Turns out it is fairly easy to get virtio-mem-pci running on s390x. We >> only have to add MSI-X support to virtio-mem-pci, and wire-up the >> (un)plugging in the machine. >> >> Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems >> to be working as expected. >> >> The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM >> for it to work. >> >> [1] https://lkml.kernel.org/r/20250115161425.246348-1-arbab@linux.ibm.com > > > Fails CI: > > https://gitlab.com/mstredhat/qemu/-/jobs/9202574981 Hm, I thought the CI was happy when Thomas sent it upstream. CRITICAL: log: qemu-system-i386: -device virtio-mem-pci: can't apply global virtio-mem-pci.vectors=0: Property 'virtio-mem-pci.vectors' not found And the same for the virtio-balloon devices with a similar change. Trying to run one offending cmdline on current upstream: ./qemu-system-i386 -display none -vga none -S -machine pc-q35-8.0,accel=tcg -device virtio-mem-pci qemu-system-i386: -device virtio-mem-pci: the configuration is not prepared for memory devices (e.g., for memory hotplug), consider specifying the maxmem option So I am not sure what is happening here? Is some commit in the pipeline breaking this?
On 21.02.25 09:26, David Hildenbrand wrote: > On 21.02.25 00:24, Michael S. Tsirkin wrote: >> On Tue, Jan 28, 2025 at 07:57:03PM +0100, David Hildenbrand wrote: >>> This is based-on [1], which adds MSI-X support to virtio-balloon-pci, >>> but can be applied independently. >>> >>> Turns out it is fairly easy to get virtio-mem-pci running on s390x. We >>> only have to add MSI-X support to virtio-mem-pci, and wire-up the >>> (un)plugging in the machine. >>> >>> Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems >>> to be working as expected. >>> >>> The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM >>> for it to work. >>> >>> [1] https://lkml.kernel.org/r/20250115161425.246348-1-arbab@linux.ibm.com >> >> >> Fails CI: >> >> https://gitlab.com/mstredhat/qemu/-/jobs/9202574981 > > Hm, I thought the CI was happy when Thomas sent it upstream. > > CRITICAL: log: qemu-system-i386: -device virtio-mem-pci: can't apply > global virtio-mem-pci.vectors=0: Property 'virtio-mem-pci.vectors' not found > > And the same for the virtio-balloon devices with a similar change. > > Trying to run one offending cmdline on current upstream: > > ./qemu-system-i386 -display none -vga none -S -machine > pc-q35-8.0,accel=tcg -device virtio-mem-pci > qemu-system-i386: -device virtio-mem-pci: the configuration is not > prepared for memory devices (e.g., for memory hotplug), consider > specifying the maxmem option > > > So I am not sure what is happening here? Is some commit in the pipeline > breaking this? Note that these two patches, and the virtio-balloon-pci MSI-X one are already upstream, Thomas included them in his s390 MR. I see that you have them included on your branch: https://gitlab.com/mstredhat/qemu/-/commits/v03b-20-02-2025?ref_type=heads The following commit in that tree is messed up: * virtio-mem-pci: Allow setting nvectors, so we can use MSI-X Likely, you can just drop these two patches from your tree; they are alreayd upstream.
This is based-on [1], which adds MSI-X support to virtio-balloon-pci, but can be applied independently. Turns out it is fairly easy to get virtio-mem-pci running on s390x. We only have to add MSI-X support to virtio-mem-pci, and wire-up the (un)plugging in the machine. Tried some simple stuff (hotplug/hotunplug/resize/reboot), and all seems to be working as expected. The kernel in the VM needs both, CONFIG_VIRTIO_PCI and CONFIG_VIRTIO_MEM for it to work. [1] https://lkml.kernel.org/r/20250115161425.246348-1-arbab@linux.ibm.com v1 -> v2: * There are no transitional/non_transitional devices for virtio-mem * Spell out removal of "return;" in second patch Cc: Eduardo Habkost <eduardo@habkost.net> Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com> Cc: Richard Henderson <richard.henderson@linaro.org> Cc: David Hildenbrand <david@redhat.com> Cc: Ilya Leoshkevich <iii@linux.ibm.com> Cc: Halil Pasic <pasic@linux.ibm.com> Cc: Christian Borntraeger <borntraeger@linux.ibm.com> Cc: Eric Farman <farman@linux.ibm.com> Cc: Thomas Huth <thuth@redhat.com> Cc: "Michael S. Tsirkin" <mst@redhat.com> Cc: Cornelia Huck <cohuck@redhat.com> Cc: Boris Fiuczynski <fiuczy@linux.ibm.com> Cc: Michal Privoznik <mprivozn@redhat.com> Cc: Mario Casquero <mcasquer@redhat.com> David Hildenbrand (2): virtio-mem-pci: Allow setting nvectors, so we can use MSI-X s390x/s390-virtio-ccw: Support plugging PCI-based virtio memory devices hw/core/machine.c | 1 + hw/s390x/s390-virtio-ccw.c | 20 ++++++++++++++------ hw/virtio/virtio-mem-pci.c | 12 ++++++++++++ 3 files changed, 27 insertions(+), 6 deletions(-)