[v19,for-4.14,00/13] VM forking

Message

Tamas K Lengyel June 1, 2020, 1:21 p.m. UTC

The following patches are part of the series that implement VM forking for
Intel HVM guests to allow for the fast creation of identical VMs without the
assosciated high startup costs of booting or restoring the VM from a savefile.

JIRA issue: https://xenproject.atlassian.net/browse/XEN-89

The fork operation is implemented as part of the "xl fork-vm" command:
    xl fork-vm -C <config> -Q <qemu-save-file> <parent_domid>

By default a fully functional fork is created. The user is in charge however to
create the appropriate config file for the fork and to generate the QEMU save
file before the fork-vm call is made. The config file needs to give the
fork a new name at minimum but other settings may also require changes. Certain
settings in the config file of both the parent and the fork have to be set to
default. Details are documented.

The interface also allows to split the forking into two steps:
    xl fork-vm --launch-dm no \
               -m <max-vcpus> \
			   -p <parent_domid>
	xl fork-vm --launch-dm late \
	           -C <config_file_for_fork> \
			   -Q <qemu_save_file> \
			   <fork_domid>

The split creation model is useful when the VM needs to be created as fast as
possible. The forked VM can be unpaused without the device model being launched
to be monitored and accessed via VMI. Note however that without its device
model running (depending on what is executing in the VM) it is bound to
misbehave or even crash when its trying to access devices that would be
emulated by QEMU. We anticipate that for certain use-cases this would be an
acceptable situation, in case for example when fuzzing is performed of code
segments that don't access such devices.

Launching the device model requires the QEMU Xen savefile to be generated
manually from the parent VM. This can be accomplished simply by connecting to
its QMP socket and issuing the "xen-save-devices-state" command. For example
using the standard tool socat these commands can be used to generate the file:
    socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid>
	{ "execute": "qmp_capabilities" }
	{ "execute": "xen-save-devices-state", \
		"arguments": { "filename": "/path/to/save/qemu_state", \
						"live": false} }

The series has been tested with Windows VMs and functions as expected. Linux
VMs when forked from a running VM will have a frozen VNC screen. Linux VMs at
this time can only be forked with a working device model when the parent VM was
restored from a snapshot using "xl restore -p". This is a known limitation due
to Linux VMs having to be made aware of being saved/migrated.

New in v19:
	Including all the patches currently outstanding into the series
	Breaking up libxl/xl patch to many sub-patches to make it easier to review
	libxl/xl is now reduced to the bare essential to launch QEMU for a VM fork

Tamas K Lengyel (13):
  x86/mem_sharing: block interrupt injection for forks
  tools/libxc: xc_memshr_fork with interrupts blocked
  tools/libxl: Split libxl__domain_make
  tools/libxl: populate xenstore entries when launching dm for VM fork
  tools/libxl: Add checks for dm_restore_file
  tools/libxl: adjust domcreate_bootloader_done
  tools/libxl: Adjust libxl__build_pre
  tools/libxl: Adjust libxl__build_post
  tools/libxl: libxl__build_hvm_fork
  tools/libxl: set QEMU saved_state from dm_restore_file
  tools/libxl: Add VM forking public functions
  tools/xl: Add xl fork-vm command
  tools/xl: document fork-vm command

 docs/man/xl.1.pod.in             |  39 +++++++++
 tools/libxc/include/xenctrl.h    |   3 +-
 tools/libxc/xc_memshr.c          |   4 +-
 tools/libxl/libxl.h              |  10 +++
 tools/libxl/libxl_create.c       | 134 +++++++++++++++++++++++++------
 tools/libxl/libxl_dm.c           |   2 +-
 tools/libxl/libxl_dom.c          |  59 +++++++++++---
 tools/libxl/libxl_internal.h     |   5 +-
 tools/libxl/libxl_types.idl      |   1 +
 tools/xl/Makefile                |   2 +-
 tools/xl/xl.h                    |   4 +
 tools/xl/xl_cmdtable.c           |  13 +++
 tools/xl/xl_forkvm.c             | 122 ++++++++++++++++++++++++++++
 tools/xl/xl_vmcontrol.c          |  13 +++
 xen/arch/x86/hvm/vmx/intr.c      |   6 ++
 xen/arch/x86/mm/mem_sharing.c    |   6 +-
 xen/include/asm-x86/hvm/domain.h |   2 +-
 xen/include/public/memory.h      |   3 +
 18 files changed, 383 insertions(+), 45 deletions(-)
 create mode 100644 tools/xl/xl_forkvm.c

Comments

Paul Durrant June 1, 2020, 3:07 p.m. UTC | #1

> -----Original Message-----
> From: Xen-devel <[e-mail]> On Behalf Of Tamas K Lengyel
> Sent: 01 June 2020 14:22
> To: [e-mail]
> Cc: Kevin Tian <[e-mail]>; Stefano Stabellini <[e-mail]>; Tamas K Lengyel
> <[e-mail]>; Jun Nakajima <[e-mail]>; Wei Liu <[e-mail]>; Andrew Cooper
> <[e-mail]>; Ian Jackson <[e-mail]>; George Dunlap
> <[e-mail]>; Tamas K Lengyel <[e-mail]>; Jan Beulich <[e-mail]>;
> Anthony PERARD <[e-mail]>; Julien Grall <[e-mail]>; Roger Pau Monné
> <[e-mail]>
> Subject: [PATCH v19 for-4.14 00/13] VM forking

Hi,

  This series looks to be largely un-acked so, since we are now past the freeze date, I don't really think it can go into 4.14. Is there a particular reason that you think it should be considered?

  Paul

> 
> The following patches are part of the series that implement VM forking for
> Intel HVM guests to allow for the fast creation of identical VMs without the
> assosciated high startup costs of booting or restoring the VM from a savefile.
> 
> JIRA issue: https://xenproject.atlassian.net/browse/XEN-89
> 
> The fork operation is implemented as part of the "xl fork-vm" command:
>     xl fork-vm -C <config> -Q <qemu-save-file> <parent_domid>
> 
> By default a fully functional fork is created. The user is in charge however to
> create the appropriate config file for the fork and to generate the QEMU save
> file before the fork-vm call is made. The config file needs to give the
> fork a new name at minimum but other settings may also require changes. Certain
> settings in the config file of both the parent and the fork have to be set to
> default. Details are documented.
> 
> The interface also allows to split the forking into two steps:
>     xl fork-vm --launch-dm no \
>                -m <max-vcpus> \
> 			   -p <parent_domid>
> 	xl fork-vm --launch-dm late \
> 	           -C <config_file_for_fork> \
> 			   -Q <qemu_save_file> \
> 			   <fork_domid>
> 
> The split creation model is useful when the VM needs to be created as fast as
> possible. The forked VM can be unpaused without the device model being launched
> to be monitored and accessed via VMI. Note however that without its device
> model running (depending on what is executing in the VM) it is bound to
> misbehave or even crash when its trying to access devices that would be
> emulated by QEMU. We anticipate that for certain use-cases this would be an
> acceptable situation, in case for example when fuzzing is performed of code
> segments that don't access such devices.
> 
> Launching the device model requires the QEMU Xen savefile to be generated
> manually from the parent VM. This can be accomplished simply by connecting to
> its QMP socket and issuing the "xen-save-devices-state" command. For example
> using the standard tool socat these commands can be used to generate the file:
>     socat - UNIX-CONNECT:/var/run/xen/qmp-libxl-<parent_domid>
> 	{ "execute": "qmp_capabilities" }
> 	{ "execute": "xen-save-devices-state", \
> 		"arguments": { "filename": "/path/to/save/qemu_state", \
> 						"live": false} }
> 
> The series has been tested with Windows VMs and functions as expected. Linux
> VMs when forked from a running VM will have a frozen VNC screen. Linux VMs at
> this time can only be forked with a working device model when the parent VM was
> restored from a snapshot using "xl restore -p". This is a known limitation due
> to Linux VMs having to be made aware of being saved/migrated.
> 
> New in v19:
> 	Including all the patches currently outstanding into the series
> 	Breaking up libxl/xl patch to many sub-patches to make it easier to review
> 	libxl/xl is now reduced to the bare essential to launch QEMU for a VM fork
> 
> Tamas K Lengyel (13):
>   x86/mem_sharing: block interrupt injection for forks
>   tools/libxc: xc_memshr_fork with interrupts blocked
>   tools/libxl: Split libxl__domain_make
>   tools/libxl: populate xenstore entries when launching dm for VM fork
>   tools/libxl: Add checks for dm_restore_file
>   tools/libxl: adjust domcreate_bootloader_done
>   tools/libxl: Adjust libxl__build_pre
>   tools/libxl: Adjust libxl__build_post
>   tools/libxl: libxl__build_hvm_fork
>   tools/libxl: set QEMU saved_state from dm_restore_file
>   tools/libxl: Add VM forking public functions
>   tools/xl: Add xl fork-vm command
>   tools/xl: document fork-vm command
> 
>  docs/man/xl.1.pod.in             |  39 +++++++++
>  tools/libxc/include/xenctrl.h    |   3 +-
>  tools/libxc/xc_memshr.c          |   4 +-
>  tools/libxl/libxl.h              |  10 +++
>  tools/libxl/libxl_create.c       | 134 +++++++++++++++++++++++++------
>  tools/libxl/libxl_dm.c           |   2 +-
>  tools/libxl/libxl_dom.c          |  59 +++++++++++---
>  tools/libxl/libxl_internal.h     |   5 +-
>  tools/libxl/libxl_types.idl      |   1 +
>  tools/xl/Makefile                |   2 +-
>  tools/xl/xl.h                    |   4 +
>  tools/xl/xl_cmdtable.c           |  13 +++
>  tools/xl/xl_forkvm.c             | 122 ++++++++++++++++++++++++++++
>  tools/xl/xl_vmcontrol.c          |  13 +++
>  xen/arch/x86/hvm/vmx/intr.c      |   6 ++
>  xen/arch/x86/mm/mem_sharing.c    |   6 +-
>  xen/include/asm-x86/hvm/domain.h |   2 +-
>  xen/include/public/memory.h      |   3 +
>  18 files changed, 383 insertions(+), 45 deletions(-)
>  create mode 100644 tools/xl/xl_forkvm.c
> 
> --
> 2.25.1
>

George Dunlap June 1, 2020, 5:11 p.m. UTC | #2

> On Jun 1, 2020, at 4:07 PM, Paul Durrant <[e-mail]> wrote:
> 
>> -----Original Message-----
>> From: Xen-devel <[e-mail]> On Behalf Of Tamas K Lengyel
>> Sent: 01 June 2020 14:22
>> To: [e-mail]
>> Cc: Kevin Tian <[e-mail]>; Stefano Stabellini <[e-mail]>; Tamas K Lengyel
>> <[e-mail]>; Jun Nakajima <[e-mail]>; Wei Liu <[e-mail]>; Andrew Cooper
>> <[e-mail]>; Ian Jackson <[e-mail]>; George Dunlap
>> <[e-mail]>; Tamas K Lengyel <[e-mail]>; Jan Beulich <[e-mail]>;
>> Anthony PERARD <[e-mail]>; Julien Grall <[e-mail]>; Roger Pau Monné
>> <[e-mail]>
>> Subject: [PATCH v19 for-4.14 00/13] VM forking
> 
> Hi,
> 
>  This series looks to be largely un-acked so, since we are now past the freeze date, I don't really think it can go into 4.14. Is there a particular reason that you think it should be considered?

Tamas’ project itself mainly uses libxc and below, as I understand; and so getting patches 1 and 2 in would be an important milestone; both have had R-b’s before the feature freeze.  Arguably patches 1 and 2 are a bug fix.  Patch 1 is missing VMX (or a general x86).

The libxl/xl side hasn’t, as I understand it, had significant review; I think that should probably wait until 4.15.

What do you think, Tamas?

 -George

Tamas K Lengyel June 1, 2020, 6:37 p.m. UTC | #3

On Mon, Jun 1, 2020 at 11:11 AM George Dunlap <[e-mail]> wrote:
>
>
>
> > On Jun 1, 2020, at 4:07 PM, Paul Durrant <[e-mail]> wrote:
> >
> >> -----Original Message-----
> >> From: Xen-devel <[e-mail]> On Behalf Of Tamas K Lengyel
> >> Sent: 01 June 2020 14:22
> >> To: [e-mail]
> >> Cc: Kevin Tian <[e-mail]>; Stefano Stabellini <[e-mail]>; Tamas K Lengyel
> >> <[e-mail]>; Jun Nakajima <[e-mail]>; Wei Liu <[e-mail]>; Andrew Cooper
> >> <[e-mail]>; Ian Jackson <[e-mail]>; George Dunlap
> >> <[e-mail]>; Tamas K Lengyel <[e-mail]>; Jan Beulich <[e-mail]>;
> >> Anthony PERARD <[e-mail]>; Julien Grall <[e-mail]>; Roger Pau Monné
> >> <[e-mail]>
> >> Subject: [PATCH v19 for-4.14 00/13] VM forking
> >
> > Hi,
> >
> >  This series looks to be largely un-acked so, since we are now past the freeze date, I don't really think it can go into 4.14. Is there a particular reason that you think it should be considered?
>
> Tamas’ project itself mainly uses libxc and below, as I understand; and so getting patches 1 and 2 in would be an important milestone; both have had R-b’s before the feature freeze.  Arguably patches 1 and 2 are a bug fix.  Patch 1 is missing VMX (or a general x86).

Correct. The first two patches going in would decide whether we will
be able to use the 4.14 release without having to carry out-of-tree
patches. Although as things stand at the moment regarding all the bugs
being discovered in 4.13 and 4.14 we will likely still have to
backport all of these patches to 4.12 by hand.

> The libxl/xl side hasn’t, as I understand it, had significant review; I think that should probably wait until 4.15.

Correct. It has been sent 19 times so far over a period of 9 months
with no feedback from any of the maintainers other then that it's hard
to review. We had some good discussion with other community members
but evidently non of the toolstack maintainers care too much about it.
I made the last-ditch effort to make it easier to review but at this
point we started implementing our own toolstack to interact with VM
forks.

> What do you think, Tamas?

If it's not going into 4.14 then it's going to be dropped. It has been
made solely for the benefit of the community to make the new VM
forking more accessible and useful for others. Without it the only way
to use the feature is to implement your own toolstack. Initially we
were hoping that integrating support to xl/libxl would eliminate the
need for us to implement our own parallel toolstack but since we have
to do that now anyway there is no benefit for us in carrying these
patches any further. It's disheartening we had to resort to that and I
certainly will try to avoid contributing to xl/libxl in the future
since I personally consider it a waste of time.

Thanks,
Tamas

Paul Durrant June 2, 2020, 7:37 a.m. UTC | #4

> -----Original Message-----
> From: Tamas K Lengyel <[e-mail]>
> Sent: 01 June 2020 19:38
> To: George Dunlap <[e-mail]>
> Cc: [e-mail]; Tamas K Lengyel <[e-mail]>; xen-devel <xen-
> [e-mail]>; Kevin Tian <[e-mail]>; Stefano Stabellini
> <[e-mail]>; Jun Nakajima <[e-mail]>; Wei Liu <[e-mail]>; Andrew Cooper
> <[e-mail]>; Ian Jackson <[e-mail]>; Jan Beulich <[e-mail]>;
> Anthony Perard <[e-mail]>; Julien Grall <[e-mail]>; Roger Pau Monne
> <[e-mail]>
> Subject: Re: [PATCH v19 for-4.14 00/13] VM forking
> 
> On Mon, Jun 1, 2020 at 11:11 AM George Dunlap <[e-mail]> wrote:
> >
> >
> >
> > > On Jun 1, 2020, at 4:07 PM, Paul Durrant <[e-mail]> wrote:
> > >
> > >> -----Original Message-----
> > >> From: Xen-devel <[e-mail]> On Behalf Of Tamas K Lengyel
> > >> Sent: 01 June 2020 14:22
> > >> To: [e-mail]
> > >> Cc: Kevin Tian <[e-mail]>; Stefano Stabellini <[e-mail]>; Tamas K
> Lengyel
> > >> <[e-mail]>; Jun Nakajima <[e-mail]>; Wei Liu <[e-mail]>; Andrew
> Cooper
> > >> <[e-mail]>; Ian Jackson <[e-mail]>; George Dunlap
> > >> <[e-mail]>; Tamas K Lengyel <[e-mail]>; Jan Beulich
> <[e-mail]>;
> > >> Anthony PERARD <[e-mail]>; Julien Grall <[e-mail]>; Roger Pau Monné
> > >> <[e-mail]>
> > >> Subject: [PATCH v19 for-4.14 00/13] VM forking
> > >
> > > Hi,
> > >
> > >  This series looks to be largely un-acked so, since we are now past the freeze date, I don't
> really think it can go into 4.14. Is there a particular reason that you think it should be considered?
> >
> > Tamas’ project itself mainly uses libxc and below, as I understand; and so getting patches 1 and 2
> in would be an important milestone; both have had R-b’s before the feature freeze.  Arguably patches 1
> and 2 are a bug fix.  Patch 1 is missing VMX (or a general x86).
> 
> Correct. The first two patches going in would decide whether we will
> be able to use the 4.14 release without having to carry out-of-tree
> patches. Although as things stand at the moment regarding all the bugs
> being discovered in 4.13 and 4.14 we will likely still have to
> backport all of these patches to 4.12 by hand.

That sounds like a reasonable justification.

Maintainers/committers, can we please get those patches in a.s.a.p.?

  Paul

Jan Beulich June 2, 2020, 9:38 a.m. UTC | #5

On 02.06.2020 09:37, Paul Durrant wrote:
> Maintainers/committers, can we please get those patches in a.s.a.p.?

The sole reason I didn't put in at least the 1st patch on Friday
(perhaps also the 2nd, as it is suitably ack-ed) was that it's
still missing a VMX maintainer ack, and Kevin had provided
comments on v2 of the smaller (2-patch) series.

Jan

Tamas K Lengyel June 15, 2020, 7:27 p.m. UTC | #6

On Tue, Jun 2, 2020 at 3:39 AM Jan Beulich <[e-mail]> wrote:
>
> On 02.06.2020 09:37, Paul Durrant wrote:
> > Maintainers/committers, can we please get those patches in a.s.a.p.?
>
> The sole reason I didn't put in at least the 1st patch on Friday
> (perhaps also the 2nd, as it is suitably ack-ed) was that it's
> still missing a VMX maintainer ack, and Kevin had provided
> comments on v2 of the smaller (2-patch) series.

Can we get the first patches from this series merged now with Kevin's
Reviewed-by he sent last week
(https://lists.xenproject.org/archives/html/xen-devel/2020-06/msg00732.html)?

Thanks,
Tamas

Jan Beulich June 16, 2020, 7:07 a.m. UTC | #7

On 15.06.2020 21:27, Tamas K Lengyel wrote:
> On Tue, Jun 2, 2020 at 3:39 AM Jan Beulich <[e-mail]> wrote:
>>
>> On 02.06.2020 09:37, Paul Durrant wrote:
>>> Maintainers/committers, can we please get those patches in a.s.a.p.?
>>
>> The sole reason I didn't put in at least the 1st patch on Friday
>> (perhaps also the 2nd, as it is suitably ack-ed) was that it's
>> still missing a VMX maintainer ack, and Kevin had provided
>> comments on v2 of the smaller (2-patch) series.
> 
> Can we get the first patches from this series merged now with Kevin's
> Reviewed-by he sent last week
> (https://lists.xenproject.org/archives/html/xen-devel/2020-06/msg00732.html)?

Please can you be a little more patient? I've been out of the
office until yesterday afternoon, and I'm dealing with half
broken email now.

Jan

Jan Beulich June 16, 2020, 8:32 a.m. UTC | #8

On 01.06.2020 15:21, Tamas K Lengyel wrote:
> Tamas K Lengyel (13):
>   x86/mem_sharing: block interrupt injection for forks
>   tools/libxc: xc_memshr_fork with interrupts blocked

I've committed these two, and I'll leave the rest to the tool stack
maintainers.

Jan

Tamas K Lengyel June 16, 2020, 1:05 p.m. UTC | #9

On Tue, Jun 16, 2020 at 2:32 AM Jan Beulich <[e-mail]> wrote:
>
> On 01.06.2020 15:21, Tamas K Lengyel wrote:
> > Tamas K Lengyel (13):
> >   x86/mem_sharing: block interrupt injection for forks
> >   tools/libxc: xc_memshr_fork with interrupts blocked
>
> I've committed these two, and I'll leave the rest to the tool stack
> maintainers.

Thanks!

Tamas