From patchwork Wed Dec 18 19:40:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tamas K Lengyel X-Patchwork-Id: 11301777 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D1020109A for ; Wed, 18 Dec 2019 19:42:30 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B6B09206D8 for ; Wed, 18 Dec 2019 19:42:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B6B09206D8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihfC6-0007av-Em; Wed, 18 Dec 2019 19:41:14 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1ihfC5-0007aq-7L for xen-devel@lists.xenproject.org; Wed, 18 Dec 2019 19:41:13 +0000 X-Inumbo-ID: 57728e8a-21ce-11ea-90f3-12813bfff9fa Received: from mga03.intel.com (unknown [134.134.136.65]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id 57728e8a-21ce-11ea-90f3-12813bfff9fa; Wed, 18 Dec 2019 19:41:10 +0000 (UTC) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 18 Dec 2019 11:41:10 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,330,1571727600"; d="scan'208";a="210196266" Received: from tlengyel-mobl2.amr.corp.intel.com (HELO localhost.localdomain) ([10.254.103.7]) by orsmga008.jf.intel.com with ESMTP; 18 Dec 2019 11:41:08 -0800 From: Tamas K Lengyel To: xen-devel@lists.xenproject.org Date: Wed, 18 Dec 2019 11:40:37 -0800 Message-Id: X-Mailer: git-send-email 2.20.1 MIME-Version: 1.0 Subject: [Xen-devel] [PATCH v2 00/20] VM forking X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Petre Pircalabu , Tamas K Lengyel , Tamas K Lengyel , Wei Liu , Konrad Rzeszutek Wilk , George Dunlap , Andrew Cooper , Ian Jackson , Anthony PERARD , Stefano Stabellini , Jan Beulich , Alexandru Isaila , Julien Grall , =?utf-8?q?Roger_Pau_Monn=C3=A9?= Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" The following series implements VM forking for Intel HVM guests to allow for the fast creation of identical VMs without the assosciated high startup costs of booting or restoring the VM from a savefile. JIRA issue: https://xenproject.atlassian.net/browse/XEN-89 The main design goal with this series has been to reduce the time of creating the VM fork as much as possible. To achieve this the VM forking process is split into two steps: 1) forking the VM on the hypervisor side; 2) starting QEMU to handle the backed for emulated devices. Step 1) involves creating a VM using the new "xl fork-vm" command. The parent VM is expected to remain paused after forks are created from it (which is different then what process forking normally entails). During this forking operation the HVM context and VM settings are copied over to the new forked VM. This operation is fast and it allows the forked VM to be unpaused and to be monitored and accessed via VMI. Note however that without its device model running (depending on what is executing in the VM) it is bound to misbehave/crash when its trying to access devices that would be emulated by QEMU. We anticipate that for certain use-cases this would be an acceptable situation, in case for example when fuzzing is performed of code segments that don't access such devices. Step 2) involves launching QEMU to support the forked VM, which requires the QEMU Xen savefile to be generated manually from the parent VM. This can be accomplished simply by connecting to its QMP socket and issuing the "xen-save-devices-state" command as documented by QEMU: https://github.com/qemu/qemu/blob/master/docs/xen-save-devices-state.txt Once the QEMU Xen savefile is generated the new "xl fork-launch-dm" command is used to launch QEMU and load the specified savefile for it. At runtime the forked VM starts running with an empty p2m which gets lazily populated when the VM generates EPT faults, similar to how altp2m views are populated. If the memory access is a read-only access, the p2m entry is populated with a memory shared entry with its parent. For write memory accesses or in case memory sharing wasn't possible (for example in case a reference is held by a third party), a new page is allocated and the page contents are copied over from the parent VM. Forks can be further forked if needed, thus allowing for further memory savings. A VM fork reset hypercall is also added that allows the fork to be reset to the state it was just after a fork. This is an optimization for cases where the forks are very short-lived and run without a device model, so resetting saves some time compared to creating a brand new fork. The series has been tested with both Linux and Windows VMs and functions as expected. VM forking time has been measured to be 0.018s, device model launch to be around 1s depending largely on the number of devices being emulated. Patches 1-2 implement changes to existing internal Xen APIs to make VM forking possible. Patches 3-4 are simple code-formatting fixes for the toolstack and Xen for the memory sharing paths with no functional changes. Patches 5-16 are code-cleanups and adjustments of to Xen memory sharing subsystem with no functional changes. Patch 17 adds the hypervisor-side code implementing VM forking. Patch 18 is integration of mem_access with forked VMs. Patch 19 implements the VM fork reset operation hypervisor side bits. Patch 20 adds the toolstack-side code implementing VM forking and reset. Tamas K Lengyel (20): x86: make hvm_{get/set}_param accessible xen/x86: Make hap_get_allocation accessible tools/libxc: clean up memory sharing files x86/mem_sharing: cleanup code and comments in various locations x86/mem_sharing: make get_two_gfns take locks conditionally x86/mem_sharing: drop flags from mem_sharing_unshare_page x86/mem_sharing: don't try to unshare twice during page fault x86/mem_sharing: define mem_sharing_domain to hold some scattered variables x86/mem_sharing: Use INVALID_MFN and p2m_is_shared in relinquish_shared_pages x86/mem_sharing: Make add_to_physmap static and shorten name x86/mem_sharing: Convert MEM_SHARING_DESTROY_GFN to a bool x86/mem_sharing: Replace MEM_SHARING_DEBUG with gdprintk x86/mem_sharing: ASSERT that p2m_set_entry succeeds x86/mem_sharing: Enable mem_sharing on first memop x86/mem_sharing: Skip xen heap pages in memshr nominate x86/mem_sharing: check page type count earlier xen/mem_sharing: VM forking xen/mem_access: Use __get_gfn_type_access in set_mem_access x86/mem_sharing: reset a fork xen/tools: VM forking toolstack side tools/libxc/include/xenctrl.h | 30 +- tools/libxc/xc_memshr.c | 34 +- tools/libxl/libxl.h | 7 + tools/libxl/libxl_create.c | 237 +++++--- tools/libxl/libxl_dm.c | 2 +- tools/libxl/libxl_dom.c | 83 ++- tools/libxl/libxl_internal.h | 1 + tools/libxl/libxl_types.idl | 1 + tools/xl/xl.h | 5 + tools/xl/xl_cmdtable.c | 22 + tools/xl/xl_saverestore.c | 96 ++++ tools/xl/xl_vmcontrol.c | 8 + xen/arch/x86/hvm/hvm.c | 206 ++++--- xen/arch/x86/mm/hap/hap.c | 3 +- xen/arch/x86/mm/mem_access.c | 5 +- xen/arch/x86/mm/mem_sharing.c | 875 +++++++++++++++++++++--------- xen/arch/x86/mm/p2m.c | 34 +- xen/common/memory.c | 2 +- xen/drivers/passthrough/pci.c | 3 +- xen/include/asm-x86/hap.h | 1 + xen/include/asm-x86/hvm/domain.h | 6 +- xen/include/asm-x86/hvm/hvm.h | 4 + xen/include/asm-x86/mem_sharing.h | 84 ++- xen/include/asm-x86/p2m.h | 14 +- xen/include/public/memory.h | 6 + xen/include/xen/sched.h | 1 + 26 files changed, 1258 insertions(+), 512 deletions(-)