From patchwork Tue Nov 20 20:39:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 10691201 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3892716B1 for ; Tue, 20 Nov 2018 20:45:01 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 23B3F2AE44 for ; Tue, 20 Nov 2018 20:45:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 168B92AE50; Tue, 20 Nov 2018 20:45:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 839732AE44 for ; Tue, 20 Nov 2018 20:45:00 +0000 (UTC) Received: from localhost ([::1]:35829 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCtH-0000bO-R0 for patchwork-qemu-devel@patchwork.kernel.org; Tue, 20 Nov 2018 15:44:59 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:52392) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gPCq5-0005kz-AZ for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gPCq1-0003El-52 for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:41 -0500 Received: from hqemgate14.nvidia.com ([216.228.121.143]:16397) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gPCq0-0003AH-T1 for qemu-devel@nongnu.org; Tue, 20 Nov 2018 15:41:37 -0500 Received: from hqpgpgate102.nvidia.com (Not Verified[216.228.121.13]) by hqemgate14.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 20 Nov 2018 12:41:33 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate102.nvidia.com (PGP Universal service); Tue, 20 Nov 2018 12:41:32 -0800 X-PGP-Universal: processed; by hqpgpgate102.nvidia.com on Tue, 20 Nov 2018 12:41:32 -0800 Received: from HQMAIL111.nvidia.com (172.20.187.18) by HQMAIL104.nvidia.com (172.18.146.11) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:32 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 20 Nov 2018 20:41:32 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 20 Nov 2018 20:41:25 +0000 From: Kirti Wankhede To: , Date: Wed, 21 Nov 2018 02:09:38 +0530 Message-ID: <1542746383-18288-1-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1542746493; bh=0iHeKHqpLj1GrJb76W8LLRTkwUYZD5IEW2TLwE0X0Gg=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: X-NVConfidentiality:MIME-Version:Content-Type; b=hw8KZhf57Y1MbrlpP2FkFbb7dCuxjvtuHslW5FTPsPrCAzKhRlRGPLZ+D69CtJ2Z/ a+gHwnuaOMiiQGxbjPIKHMeCR/ddnkxiE84lU6acGIEGe5jml7AJ8nF2A1CTaSfT3W fDm9uWYYdfGq1sRy0sLF9jpkVq2fx59EUy+oMYZn66PtfrUAwYT5RjzS5EEd4jZT/I +RyrdzSZt4HUY2EKpn4HsDGLgTPD1K47a6wdXscvRINMPQ4AwI8c8zdgYRLq2oF/pc KcCc66XS+6InzMa2+Pm2Ow3IaqG0/WPT6WFPC10+LloiMtK7AhtqeW0UKdaCHAjw7/ 9vGItXmeEETKA== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.143 Subject: [Qemu-devel] [PATCH 0/5] Add migration support for VFIO device X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, Kirti Wankhede , eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Add migration support for VFIO device This Patch set include patches as below: - Define KABI for VFIO device for migration support. - Added save and restore functions for PCI configuration space - Generic migration functionality for VFIO device. * This patch set adds functionality only for PCI devices, but can be extended to other VFIO devices. * Added all the basic functions required for pre-copy, stop-and-copy and resume phases of migration. * Added state change notifier and from that notifier function, VFIO device's state changed is conveyed to VFIO device driver. * During save setup phase and resume/load setup phase, migration region is queried and is used to read/write VFIO device data. * .save_live_pending, .save_live_iterate and .is_active_iterate are implemented to use QEMU's functionality of iteration during pre-copy phase. * In .save_live_complete_precopy, that is in stop-and-copy phase, iteration to read data from VFIO device driver is implemented till pending bytes returned by driver are not zero. * .save_cleanup and .load_cleanup are implemented to unmap migration region that was setup duing setup phase. * Added function to get dirty pages bitmap for the pages which are used by driver. - Add vfio_listerner_log_sync to mark dirty pages. - Make VFIO PCI device migration capable. If migration region is not provided by driver, migration is blocked. Below is the flow of state change for live migration where states in brackets represent VM state, migration state and VFIO device state as: (VM state, MIGRATION_STATUS, VFIO_DEVICE_STATE) Live migration save path: QEMU normal running state (RUNNING, _NONE, _RUNNING) | migrate_init spawns migration_thread. (RUNNING, _SETUP, _MIGRATION_SETUP) Migration thread then calls each device's .save_setup() | (RUNNING, _ACTIVE, _MIGRATION_PRECOPY) If device is active, get pending bytes by .save_live_pending() if pending bytes >= threshold_size, call save_live_iterate() Data of VFIO device for pre-copy phase is copied. Iterate till pending bytes converge and are less than threshold | migration_completion() stops vCPUs and calls .save_live_complete_precopy for each active device. VFIO device is then transitioned in _MIGRATION_STOPNCOPY state. (FINISH_MIGRATE, _DEVICE, _MIGRATION_STOPNCOPY) For VFIO device, iterate in .save_live_complete_precopy until pending data is 0. Change VFIO device state. (FINISH_MIGRATE, _DEVICE, _MIGRATION_SAVE_COMPLETED) | (FINISH_MIGRATE, _COMPLETED, _MIGRATION_SAVE_COMPLETED) Migraton thread schedule cleanup bottom half and exit | (POST_MIGRATE, _COMPLETED, _MIGRATION_SAVE_COMPLETED) For each device, call .save_cleanup(). Unmap migration region. Live migration resume path: Incomming migration calls .load_setup for each device (RESTORE_VM, _ACTIVE, _MIGRATION_RESUME) | For each device, .load_state is called for that device section data | At the end, called .load_cleanup for each device and vCPUs are started. (RUNNING, _NONE, _RUNNING) Note that: - Migration post copy is not supported. - VFIO device driver version compatibility is not taken care in this series. v1 -> v2: - Defined MIGRATION region type and sub-type which should be used with region type capability. - Re-structured vfio_device_migration_info. This structure will be placed at 0th offset of migration region. - Replaced ioctl with read/write for trapped part of migration region. - Added both type of access support, trapped or mmapped, for data section of the region. - Moved PCI device functions to pci file. - Added iteration to get dirty page bitmap until bitmap for all requested pages are copied. Thanks, Kirti Kirti Wankhede (5): VFIO KABI for migration interface Add save and load functions for VFIO PCI devices Add migration functions for VFIO devices Add vfio_listerner_log_sync to mark dirty pages Make vfio-pci device migration capable. hw/vfio/Makefile.objs | 2 +- hw/vfio/common.c | 32 ++ hw/vfio/migration.c | 729 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 108 ++++++- hw/vfio/pci.h | 29 ++ include/hw/vfio/vfio-common.h | 23 ++ linux-headers/linux/vfio.h | 130 ++++++++ 7 files changed, 1045 insertions(+), 8 deletions(-) create mode 100644 hw/vfio/migration.c