mbox series

[RFC,v3,00/45] Initial support of multi-process qemu

Message ID cover.1567534653.git.jag.raman@oracle.com (mailing list archive)
Headers show
Series Initial support of multi-process qemu | expand

Message

Jag Raman Sept. 3, 2019, 8:37 p.m. UTC 
Started with the presentation in October 2017 made by Marc-Andre (Red Hat)
and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018,
the multi-process project is now a prototype and presented in this patchset.

This first series enables the emulation of lsi53c895a in a separate process.

We posted the Proof Of Concept patches [2] before the BoF session in 2018.
Subsequently, we posted RFC v1 [3] & RFC v2 [4] of this series. 

Thanks to the v1 & v2 review, we were able to incorporate the feedback into
our goals.

In the summer of 2019, we participated in a conference with folks at RedHat,
who provided us with feedback to improve the design of this project. 

We want to present version 3 of this series which incorporates the feedback
we received for v2 & the enhancements suggested in the conference. Following
are the improvements made in this series:
  - The command-line arguments are Orchestrator friendly. A new option
    named "-remote" carries the command-line of the emulation process.
    Added "rid" suboption to the "-device" option and moved "-drive"
    option to the emulation process's command-line.
  - Redesigned Monitor interface allows direct access to the remote process,
    thereby making it more scalable, maintainable & Orchestrator friendly.
    We also enabled more monitor commands for the remote process.
  - Enabled the remote process to run multiple LSI controllers
  - Redesigned the communication link to support multiple channels. MMIO
    transactions are carried out in a separate channel.
  - Fixed the bugs discovered during testing. Individual patches provide
    the details of these fixes.

Following people contributed to this patchset:

John G Johnson <john.g.johnson@oracle.com>
Jagannathan Raman <jag.raman@oracle.com>
Elena Ufimtseva <elena.ufimtseva@oracle.com>
Kanth Ghatraju <kanth.ghatraju@oracle.com>

For full concept writeup about QEMU disaggregation refer to
docs/devel/qemu-multiprocess.txt. Please refer to 
docs/qemu-multiprocess.txt for usage information.

We are planning on making the following improvements in the future:
  - Live migration
  - Performance improvements
  - Libvirt support
  - Enforcement of security policies
  - blockdev support

We welcome all your ideas, concerns, and questions for this patchset.

Thank you!

[1]: http://events17.linuxfoundation.org/sites/events/files/slides/KVM%20FORUM%20multi-process.pdf
[1]: https://www.youtube.com/watch?v=Kq1-coHh7lg
[2]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg566538.html
[3]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg602285.html
[4]: https://www.mail-archive.com/qemu-devel@nongnu.org/msg624877.html

Elena Ufimtseva (20):
  multi-process: add a command line option for debug file
  multi-process: introduce proxy object
  mutli-process: build remote command line args
  multi-process: add support of device id to communication channel
  multi-process: modify BARs read/write to support dev_id
  multi-process: support dev id in config read/write
  multi-process: configure remote side devices
  multi-process: add qdev_proxy_add to create proxy devices
  multi-process: remote: add setup_devices and setup_drive msg
    processing
  multi-process: remote: use fd for socket from parent process
  multi-process: remote: add create_done condition
  multi-process: add processing of remote drive and device command line
  multi-process: refractor vl.c code to re-use in remote
  multi-process: add remote option
  multi-process: add remote options parser
  multi-process: add parse_cmdline in remote process
  multi-process: add support for multiple devices
  multi-process: add heartbeat timer and signal handler
  multi-process: handle heartbeat messages in remote process
  multi-process: add configure and usage information

Jagannathan Raman (24):
  multi-process: memory: alloc RAM from file at offset
  multi-process: util: Add qemu_thread_cancel() to cancel running thread
  multi-process: Add stub functions to facilate build of multi-process
  multi-process: Add config option for multi-process QEMU
  multi-process: build system for remote device process
  multi-process: define proxy-link object
  multi-process: add functions to synchronize proxy and remote endpoints
  multi-process: setup PCI host bridge for remote device
  multi-process: setup a machine object for remote device process
  multi-process: setup memory manager for remote device
  multi-process: remote process initialization
  multi-process: PCI BAR read/write handling for proxy & remote
    endpoints
  multi-process: Add LSI device proxy object
  multi-process: Synchronize remote memory
  multi-process: create IOHUB object to handle irq
  multi-process: Introduce build flags to separate remote process code
  multi-process: Use separate MMIO communication channel
  multi-process: perform device reset in the remote process
  multi-process/mon: stub functions to enable QMP module for remote
    process
  multi-process/mon: build system for QMP module in remote process
  multi-process/mon: Refactor monitor/chardev functions out of vl.c
  multi-process/mon: trim HMP command set for remote storage processes
  multi-process/mon: Initialize QMP module for remote processes
  multi-process: prevent duplicate memory initialization in remote

John G Johnson (1):
  multi-process: add the concept description to
    docs/devel/qemu-multiprocess

 Makefile                            |   2 +
 Makefile.objs                       |  33 ++
 Makefile.target                     |  91 +++++-
 accel/stubs/kvm-stub.c              |   5 +
 accel/stubs/tcg-stub.c              |  95 ++++++
 backends/Makefile.objs              |   2 +
 block/Makefile.objs                 |   2 +
 configure                           |  15 +
 docs/devel/qemu-multiprocess.txt    | 627 ++++++++++++++++++++++++++++++++++++
 docs/qemu-multiprocess.txt          |  86 +++++
 exec.c                              |  14 +-
 hmp-scsi-commands-info.hx           | 167 ++++++++++
 hmp-scsi-commands.hx                | 384 ++++++++++++++++++++++
 hw/Makefile.objs                    |   9 +
 hw/block/Makefile.objs              |   2 +
 hw/core/Makefile.objs               |  16 +
 hw/nvram/Makefile.objs              |   2 +
 hw/pci/Makefile.objs                |   4 +
 hw/proxy/Makefile.objs              |   1 +
 hw/proxy/memory-sync.c              | 226 +++++++++++++
 hw/proxy/proxy-lsi53c895a.c         | 168 ++++++++++
 hw/proxy/qemu-proxy.c               | 615 +++++++++++++++++++++++++++++++++++
 hw/scsi/Makefile.objs               |   2 +
 include/exec/address-spaces.h       |   2 +
 include/exec/ram_addr.h             |   2 +-
 include/glib-compat.h               |   4 +
 include/hw/pci/pci_ids.h            |   3 +
 include/hw/proxy/memory-sync.h      |  51 +++
 include/hw/proxy/proxy-lsi53c895a.h |  42 +++
 include/hw/proxy/qemu-proxy.h       | 106 ++++++
 include/hw/qdev-core.h              |   2 +
 include/io/proxy-link.h             | 202 ++++++++++++
 include/monitor/qdev.h              |  24 ++
 include/qemu-common.h               |   8 +
 include/qemu/log.h                  |   1 +
 include/qemu/mmap-alloc.h           |   3 +-
 include/qemu/thread.h               |   1 +
 include/remote/iohub.h              |  63 ++++
 include/remote/machine.h            |  48 +++
 include/remote/memory.h             |  34 ++
 include/remote/pcihost.h            |  59 ++++
 io/Makefile.objs                    |   2 +
 io/proxy-link.c                     | 363 +++++++++++++++++++++
 memory.c                            |   2 +-
 migration/Makefile.objs             |   2 +
 monitor/Makefile.objs               |   3 +
 monitor/misc.c                      |  84 ++---
 monitor/monitor-internal.h          |  38 +++
 qapi/Makefile.objs                  |   2 +
 qdev-monitor.c                      | 257 ++++++++++++++-
 qemu-options.hx                     |  21 ++
 qom/Makefile.objs                   |   5 +
 remote/Makefile.objs                |   6 +
 remote/iohub.c                      | 159 +++++++++
 remote/machine.c                    | 133 ++++++++
 remote/memory.c                     |  99 ++++++
 remote/pcihost.c                    |  84 +++++
 remote/remote-main.c                | 626 +++++++++++++++++++++++++++++++++++
 remote/remote-opts.c                | 123 +++++++
 remote/remote-opts.h                |  31 ++
 rules.mak                           |   2 +-
 stubs/gdbstub.c                     |  21 ++
 stubs/machine-init-done.c           |   4 +
 stubs/migration.c                   | 162 ++++++++++
 stubs/monitor.c                     |  65 ++++
 stubs/net-stub.c                    | 100 ++++++
 stubs/qapi-misc.c                   |  41 +++
 stubs/qapi-target.c                 |  49 +++
 stubs/replay.c                      |  18 ++
 stubs/ui-stub.c                     | 130 ++++++++
 stubs/vl-stub.c                     | 169 ++++++++++
 stubs/vmstate.c                     |  20 ++
 stubs/xen-mapcache.c                |  22 ++
 ui/Makefile.objs                    |   2 +
 util/log.c                          |   2 +
 util/mmap-alloc.c                   |   7 +-
 util/oslib-posix.c                  |   2 +-
 util/qemu-thread-posix.c            |  10 +
 vl-parse.c                          | 249 ++++++++++++++
 vl.c                                | 298 ++++++++---------
 vl.h                                |  58 ++++
 81 files changed, 6466 insertions(+), 228 deletions(-)
 create mode 100644 docs/devel/qemu-multiprocess.txt
 create mode 100644 docs/qemu-multiprocess.txt
 create mode 100644 hmp-scsi-commands-info.hx
 create mode 100644 hmp-scsi-commands.hx
 create mode 100644 hw/proxy/Makefile.objs
 create mode 100644 hw/proxy/memory-sync.c
 create mode 100644 hw/proxy/proxy-lsi53c895a.c
 create mode 100644 hw/proxy/qemu-proxy.c
 create mode 100644 include/hw/proxy/memory-sync.h
 create mode 100644 include/hw/proxy/proxy-lsi53c895a.h
 create mode 100644 include/hw/proxy/qemu-proxy.h
 create mode 100644 include/io/proxy-link.h
 create mode 100644 include/remote/iohub.h
 create mode 100644 include/remote/machine.h
 create mode 100644 include/remote/memory.h
 create mode 100644 include/remote/pcihost.h
 create mode 100644 io/proxy-link.c
 create mode 100644 remote/Makefile.objs
 create mode 100644 remote/iohub.c
 create mode 100644 remote/machine.c
 create mode 100644 remote/memory.c
 create mode 100644 remote/pcihost.c
 create mode 100644 remote/remote-main.c
 create mode 100644 remote/remote-opts.c
 create mode 100644 remote/remote-opts.h
 create mode 100644 stubs/migration.c
 create mode 100644 stubs/net-stub.c
 create mode 100644 stubs/qapi-misc.c
 create mode 100644 stubs/qapi-target.c
 create mode 100644 stubs/ui-stub.c
 create mode 100644 stubs/vl-stub.c
 create mode 100644 stubs/xen-mapcache.c
 create mode 100644 vl-parse.c
 create mode 100644 vl.h

Comments

Daniel P. Berrangé Sept. 4, 2019, 9:18 a.m. UTC | #1 
On Tue, Sep 03, 2019 at 04:37:26PM -0400, Jagannathan Raman wrote:
> Started with the presentation in October 2017 made by Marc-Andre (Red Hat)
> and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018,
> the multi-process project is now a prototype and presented in this patchset.
> 
> This first series enables the emulation of lsi53c895a in a separate process.
> 
> We posted the Proof Of Concept patches [2] before the BoF session in 2018.
> Subsequently, we posted RFC v1 [3] & RFC v2 [4] of this series. 
> 
> Thanks to the v1 & v2 review, we were able to incorporate the feedback into
> our goals.
> 
> In the summer of 2019, we participated in a conference with folks at RedHat,
> who provided us with feedback to improve the design of this project. 
> 
> We want to present version 3 of this series which incorporates the feedback
> we received for v2 & the enhancements suggested in the conference. Following
> are the improvements made in this series:

This patch series fails to apply to git master when I attempt and confirmed
by patchew's failed attempt too

  https://patchew.org/QEMU/cover.1567534653.git.jag.raman@oracle.com/

Can you do one of the following

 - Tell us what git hash it was based on instead of master
or
 - provide a pointer to a publically accessible git repo with it applied
or
 - Repost with based on current git master

Regards,
Daniel
Jag Raman Sept. 4, 2019, 4:29 p.m. UTC | #2 
On 9/4/2019 5:18 AM, Daniel P. Berrangé wrote:
> On Tue, Sep 03, 2019 at 04:37:26PM -0400, Jagannathan Raman wrote:
>> Started with the presentation in October 2017 made by Marc-Andre (Red Hat)
>> and Konrad Wilk (Oracle) [1], and continued by Jag's BoF at KVM Forum 2018,
>> the multi-process project is now a prototype and presented in this patchset.
>>
>> This first series enables the emulation of lsi53c895a in a separate process.
>>
>> We posted the Proof Of Concept patches [2] before the BoF session in 2018.
>> Subsequently, we posted RFC v1 [3] & RFC v2 [4] of this series.
>>
>> Thanks to the v1 & v2 review, we were able to incorporate the feedback into
>> our goals.
>>
>> In the summer of 2019, we participated in a conference with folks at RedHat,
>> who provided us with feedback to improve the design of this project.
>>
>> We want to present version 3 of this series which incorporates the feedback
>> we received for v2 & the enhancements suggested in the conference. Following
>> are the improvements made in this series:
> 
> This patch series fails to apply to git master when I attempt and confirmed
> by patchew's failed attempt too
> 
>    https://patchew.org/QEMU/cover.1567534653.git.jag.raman@oracle.com/
> 
> Can you do one of the following
> 
>   - Tell us what git hash it was based on instead of master

Hi Daniel,

These patches are based on the latest release, v4.1.0. They are on top
of the following git hash:
9e06029aea3 Update version for v4.1.0 release

> or
>   - provide a pointer to a publically accessible git repo with it applied

They are also publicly available from the following gitub repo.:
https://github.com/oracle/qemu.git  [branch multi-process-qemu-v0.3]

Thanks!
--
Jag

> or
>   - Repost with based on current git master
> 
> Regards,
> Daniel
>