| Message ID | bbba8cdef9f876ec6d194f3e1974347860eca732.1582576372.git.jag.raman@oracle.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Initial support for multi-process qemu | expand |
On Mon, Feb 24, 2020 at 03:55:41PM -0500, Jagannathan Raman wrote: > From: Elena Ufimtseva <elena.ufimtseva@oracle.com> > > Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com> > Signed-off-by: Jagannathan Raman <jag.raman@oracle.com> > Signed-off-by: John G Johnson <john.g.johnson@oracle.com> > --- > docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 86 insertions(+) > create mode 100644 docs/qemu-multiprocess.txt > > diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt > new file mode 100644 > index 0000000..f156177 > --- /dev/null > +++ b/docs/qemu-multiprocess.txt > @@ -0,0 +1,86 @@ > +Multi-process QEMU > +================== > + > +This document describes how to configure and use multi-process qemu. > +For the design document refer to docs/devel/qemu-multiprocess. > + > +1) Configuration > +---------------- > + > +To enable support for multi-process add --enable-mpqemu > +to the list of options for the "configure" script. > + > + > +2) Usage > +-------- > + > +To start qemu with devices intended to run in a separate emulation > +process without libvirtd support, the following should be used on QEMU > +command line. As of now, we only support the emulation of lsi53c895a > +in a separate process > + > +* Since parts of the RAM are shared between QEMU & remote process, a > + memory-backend-file is required to facilitate this, as follows: > + > + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on memory-backend-memfd is more convenient. It doesn't require a mem-path and share=on is the default. > + > +* The devices to be emulated in the separate process are defined as > + before with addition of "rid" suboption that serves as a remote group > + identificator. > + > + -device <device options>,rid="remote process id" > + > + For example, for non multi-process qemu: > + -device lsi53c895a,id=scsi0 device > + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 > + -drive id=drive0,file=data-disk.img > + > + and for multi-process qemu and no libvirt > + support (i.e. QEMU forks child processes): > + -device lsi53c895a,id=scsi0,rid=0 > + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0 This approach is invasive: * lsi53c895a should not need to be modified with a new rid= option. * QEMU should not know about the scsi-hd device or drive0. Only the device emulation process needs to know about scsi-hd. In order to cleanly separate QEMU and the device emulation process syntax like this is needed: -object remote-device,id=rid0,... -device remote-pci-device,id=scsi0,remote-device=rid0 The "remote-device" object could be part of remote-pci-device, but keeping it separate may be useful in the future in order to support things like reconnection. The generic "remote-pci-device" device handles any remote PCI device, not just the LSI SCSI controller. Do you agree with this approach? > +* The command-line options for the remote process are added to the "command" > + suboption of the newly added "-remote" option. > + > + -remote [socket],rid=0,exec="...",command="..." QEMU has been using the -object TYPE syntax instead of adding new -TYPE command-line options. This gives you object_add hotplug for free, for example. I suggest using -object remote-device,id=,exec=,command=, instead of -remote. > + > + The drives to be emulated by the remote process are specified as part of > + this command sub-option. The device to be used to connect to the monitor > + is also specified as part of this suboption. > + > + For example, the following option adds a drive and monitor to the remote > + process: > + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" > + > + Note: There's an issue with this "command" sub-option which we are in the > + process of fixing. To work around this issue, it requires additional > + "comma" characters as illustrated above, and in the example below. command= (which could be called args= for clarity) will be difficult to use not just because of comma escaping but also because of double-quote escaping. How do you pass a command-line argument that contains spaces?
On Thu, Feb 27, 2020 at 04:58:04PM +0000, Stefan Hajnoczi wrote: > On Mon, Feb 24, 2020 at 03:55:41PM -0500, Jagannathan Raman wrote: > > From: Elena Ufimtseva <elena.ufimtseva@oracle.com> > > > > Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com> > > Signed-off-by: Jagannathan Raman <jag.raman@oracle.com> > > Signed-off-by: John G Johnson <john.g.johnson@oracle.com> > > --- > > docs/qemu-multiprocess.txt | 86 ++++++++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 86 insertions(+) > > create mode 100644 docs/qemu-multiprocess.txt > > > > diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt > > new file mode 100644 > > index 0000000..f156177 > > --- /dev/null > > +++ b/docs/qemu-multiprocess.txt > > @@ -0,0 +1,86 @@ > > +Multi-process QEMU > > +================== > > + > > +This document describes how to configure and use multi-process qemu. > > +For the design document refer to docs/devel/qemu-multiprocess. > > + > > +1) Configuration > > +---------------- > > + > > +To enable support for multi-process add --enable-mpqemu > > +to the list of options for the "configure" script. > > + > > + > > +2) Usage > > +-------- > > + > > +To start qemu with devices intended to run in a separate emulation > > +process without libvirtd support, the following should be used on QEMU > > +command line. As of now, we only support the emulation of lsi53c895a > > +in a separate process > > + > > +* Since parts of the RAM are shared between QEMU & remote process, a > > + memory-backend-file is required to facilitate this, as follows: > > + > > + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on > > memory-backend-memfd is more convenient. It doesn't require a mem-path > and share=on is the default. > Will change this. > > + > > +* The devices to be emulated in the separate process are defined as > > + before with addition of "rid" suboption that serves as a remote group > > + identificator. > > + > > + -device <device options>,rid="remote process id" > > + > > + For example, for non multi-process qemu: > > + -device lsi53c895a,id=scsi0 device > > + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 > > + -drive id=drive0,file=data-disk.img > > + > > + and for multi-process qemu and no libvirt > > + support (i.e. QEMU forks child processes): > > + -device lsi53c895a,id=scsi0,rid=0 > > + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0 > > This approach is invasive: > * lsi53c895a should not need to be modified with a new rid= option. > * QEMU should not know about the scsi-hd device or drive0. Only the > device emulation process needs to know about scsi-hd. > > In order to cleanly separate QEMU and the device emulation process > syntax like this is needed: > > -object remote-device,id=rid0,... > -device remote-pci-device,id=scsi0,remote-device=rid0 > > The "remote-device" object could be part of remote-pci-device, but > keeping it separate may be useful in the future in order to support > things like reconnection. > > The generic "remote-pci-device" device handles any remote PCI device, > not just the LSI SCSI controller. > > Do you agree with this approach? > We discussed these changes and they seem to be along the lines with the future work on vfio over socket approach we will be working on later. Could we for this experimental version have the changes you propose here with one modification - instead of having generic remote-pci-device imply that that is LSI device? And while we work towards vfio over socket this will become any remote PCI device? > > +* The command-line options for the remote process are added to the "command" > > + suboption of the newly added "-remote" option. > > + > > + -remote [socket],rid=0,exec="...",command="..." > > QEMU has been using the -object TYPE syntax instead of adding new -TYPE > command-line options. This gives you object_add hotplug for free, for > example. I suggest using -object remote-device,id=,exec=,command=, > instead of -remote. > We will add these changes. > > + > > + The drives to be emulated by the remote process are specified as part of > > + this command sub-option. The device to be used to connect to the monitor > > + is also specified as part of this suboption. > > + > > + For example, the following option adds a drive and monitor to the remote > > + process: > > + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" > > + > > + Note: There's an issue with this "command" sub-option which we are in the > > + process of fixing. To work around this issue, it requires additional > > + "comma" characters as illustrated above, and in the example below. > > command= (which could be called args= for clarity) will be difficult to > use not just because of comma escaping but also because of double-quote > escaping. How do you pass a command-line argument that contains spaces? Yes, this is not great. And spaces are the problem at the moment. I am looking if the -object has some properties that can allow for arbitrary strings. Maybe such as data for "secret" object would do? Elena
On Fri, Feb 28, 2020 at 10:43:44AM -0800, Elena Ufimtseva wrote: > On Thu, Feb 27, 2020 at 04:58:04PM +0000, Stefan Hajnoczi wrote: > > On Mon, Feb 24, 2020 at 03:55:41PM -0500, Jagannathan Raman wrote: > > > +* The devices to be emulated in the separate process are defined as > > > + before with addition of "rid" suboption that serves as a remote group > > > + identificator. > > > + > > > + -device <device options>,rid="remote process id" > > > + > > > + For example, for non multi-process qemu: > > > + -device lsi53c895a,id=scsi0 device > > > + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 > > > + -drive id=drive0,file=data-disk.img > > > + > > > + and for multi-process qemu and no libvirt > > > + support (i.e. QEMU forks child processes): > > > + -device lsi53c895a,id=scsi0,rid=0 > > > + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0 > > > > This approach is invasive: > > * lsi53c895a should not need to be modified with a new rid= option. > > * QEMU should not know about the scsi-hd device or drive0. Only the > > device emulation process needs to know about scsi-hd. > > > > In order to cleanly separate QEMU and the device emulation process > > syntax like this is needed: > > > > -object remote-device,id=rid0,... > > -device remote-pci-device,id=scsi0,remote-device=rid0 > > > > The "remote-device" object could be part of remote-pci-device, but > > keeping it separate may be useful in the future in order to support > > things like reconnection. > > > > The generic "remote-pci-device" device handles any remote PCI device, > > not just the LSI SCSI controller. > > > > Do you agree with this approach? > > > > We discussed these changes and they seem to be along the lines with > the future work on vfio over socket approach we will be working on later. > > Could we for this experimental version have the changes you propose here > with one modification - instead of having generic remote-pci-device imply that that is LSI > device? And while we work towards vfio over socket this will become any remote > PCI device? Yes, that sounds good. > > > + The drives to be emulated by the remote process are specified as part of > > > + this command sub-option. The device to be used to connect to the monitor > > > + is also specified as part of this suboption. > > > + > > > + For example, the following option adds a drive and monitor to the remote > > > + process: > > > + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" > > > + > > > + Note: There's an issue with this "command" sub-option which we are in the > > > + process of fixing. To work around this issue, it requires additional > > > + "comma" characters as illustrated above, and in the example below. > > > > command= (which could be called args= for clarity) will be difficult to > > use not just because of comma escaping but also because of double-quote > > escaping. How do you pass a command-line argument that contains spaces? > > Yes, this is not great. And spaces are the problem at the moment. > I am looking if the -object has some properties that can allow for arbitrary > strings. Maybe such as data for "secret" object would do? I'm not aware of a way to avoid the comma escaping. The space escaping is a question of how the remote process is spawned. If the command-line is processed by a shell like with system(3) then backslash can be used to escape spaces. Stefan
diff --git a/docs/qemu-multiprocess.txt b/docs/qemu-multiprocess.txt new file mode 100644 index 0000000..f156177 --- /dev/null +++ b/docs/qemu-multiprocess.txt @@ -0,0 +1,86 @@ +Multi-process QEMU +================== + +This document describes how to configure and use multi-process qemu. +For the design document refer to docs/devel/qemu-multiprocess. + +1) Configuration +---------------- + +To enable support for multi-process add --enable-mpqemu +to the list of options for the "configure" script. + + +2) Usage +-------- + +To start qemu with devices intended to run in a separate emulation +process without libvirtd support, the following should be used on QEMU +command line. As of now, we only support the emulation of lsi53c895a +in a separate process + +* Since parts of the RAM are shared between QEMU & remote process, a + memory-backend-file is required to facilitate this, as follows: + + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=4096M,share=on + +* The devices to be emulated in the separate process are defined as + before with addition of "rid" suboption that serves as a remote group + identificator. + + -device <device options>,rid="remote process id" + + For example, for non multi-process qemu: + -device lsi53c895a,id=scsi0 device + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0 + -drive id=drive0,file=data-disk.img + + and for multi-process qemu and no libvirt + support (i.e. QEMU forks child processes): + -device lsi53c895a,id=scsi0,rid=0 + -device scsi-hd,drive=drive0,bus=scsi0.0,scsi-id=0,rid=0 + +* The command-line options for the remote process are added to the "command" + suboption of the newly added "-remote" option. + + -remote [socket],rid=0,exec="...",command="..." + + The drives to be emulated by the remote process are specified as part of + this command sub-option. The device to be used to connect to the monitor + is also specified as part of this suboption. + + For example, the following option adds a drive and monitor to the remote + process: + -remote rid=0,exec="qemu-scsi-dev",command="-drive id=drive0,,file=data-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" + + Note: There's an issue with this "command" sub-option which we are in the + process of fixing. To work around this issue, it requires additional + "comma" characters as illustrated above, and in the example below. + +* Example QEMU command-line to launch lsi53c895a in a remote process + + #/bin/sh + qemu-system-x86_64 \ + -name "OL7.4" \ + -machine q35,accel=kvm \ + -smp sockets=1,cores=1,threads=1 \ + -cpu host \ + -m 2048 \ + -object memory-backend-file,id=mem,mem-path=/dev/shm/,size=2G,share=on \ + -numa node,memdev=mem \ + -device virtio-scsi-pci,id=virtio_scsi_pci0 \ + -drive id=drive_image1,if=none,format=raw,file=/root/ol7.qcow2 \ + -device scsi-hd,id=image1,drive=drive_image1,bus=virtio_scsi_pci0.0 \ + -boot d \ + -monitor stdio \ + -vnc :0 \ + -device lsi53c895a,id=lsi0,remote,rid=8,command="qemu-scsi-dev" \ + -device scsi-hd,id=drive2,drive=drive_image2,bus=lsi0.0,scsi-id=0,remote,rid=8,command="qemu-scsi-dev"\ + -remote rid=8,exec="qemu-scsi-dev",command="-drive id=drive_image2,,file=/root/remote-process-disk.img -monitor unix:/home/qmp-sock,,server,,nowait" + + We could connect to the monitor using the following command: + socat /home/qmp-sock stdio + + After hotplugging disks to the remote process, please execute the + following command in the guest to refresh the list of storage devices: + rescan_scsi_bus.sh -a