From patchwork Tue Feb 19 21:23:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 10820801 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2A4531390 for ; Tue, 19 Feb 2019 21:32:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 19B3C2D589 for ; Tue, 19 Feb 2019 21:32:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0CC0E2D815; Tue, 19 Feb 2019 21:32:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,MAILING_LIST_MULTI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DB5A22D7DC for ; Tue, 19 Feb 2019 21:32:30 +0000 (UTC) Received: from localhost ([127.0.0.1]:55247 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gwD09-0007UF-Oz for patchwork-qemu-devel@patchwork.kernel.org; Tue, 19 Feb 2019 16:32:29 -0500 Received: from eggs.gnu.org ([209.51.188.92]:33466) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1gwCyP-00068w-1G for qemu-devel@nongnu.org; Tue, 19 Feb 2019 16:30:42 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1gwCyM-0005ga-6i for qemu-devel@nongnu.org; Tue, 19 Feb 2019 16:30:40 -0500 Received: from hqemgate15.nvidia.com ([216.228.121.64]:7148) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1gwCyL-0005b8-BF for qemu-devel@nongnu.org; Tue, 19 Feb 2019 16:30:37 -0500 Received: from hqpgpgate101.nvidia.com (Not Verified[216.228.121.13]) by hqemgate15.nvidia.com (using TLS: TLSv1.2, DES-CBC3-SHA) id ; Tue, 19 Feb 2019 13:25:25 -0800 Received: from hqmail.nvidia.com ([172.20.161.6]) by hqpgpgate101.nvidia.com (PGP Universal service); Tue, 19 Feb 2019 13:25:26 -0800 X-PGP-Universal: processed; by hqpgpgate101.nvidia.com on Tue, 19 Feb 2019 13:25:26 -0800 Received: from HQMAIL110.nvidia.com (172.18.146.15) by HQMAIL103.nvidia.com (172.20.187.11) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 19 Feb 2019 21:25:25 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by hqmail110.nvidia.com (172.18.146.15) with Microsoft SMTP Server (TLS) id 15.0.1395.4; Tue, 19 Feb 2019 21:25:24 +0000 Received: from kwankhede-dev.nvidia.com (10.124.1.5) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1395.4 via Frontend Transport; Tue, 19 Feb 2019 21:25:17 +0000 From: Kirti Wankhede To: , Date: Wed, 20 Feb 2019 02:53:16 +0530 Message-ID: <1550611400-13703-2-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1550611400-13703-1-git-send-email-kwankhede@nvidia.com> References: <1550611400-13703-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1550611525; bh=to18RAoY3ilgPnufDy6NrWRf4HS/Vht3Au3qBT8x4dc=; h=X-PGP-Universal:From:To:CC:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:X-NVConfidentiality:MIME-Version: Content-Type; b=fXR5CWssHdGhQj/A0kPzUlGZI+rODE3grq5zudZ9DTb0axcoaOSSj4M9ADT9IkPVo ONPUlRbHzXO3A+kcjMG8nNnF4Re0II6arkyj2q/urTWtKe2yup49Cfpfz0V6iQ9K3O cmAQQGeN6fUOKm/AOI+kHAvXpMfAJ3UCdc5/V6BcCWRy8GxBkS344ZbO5AapvZV0Ps iJ+TahBKjMXJTFIU3ppTdHCTon0BWZboIhGYOlyR3scimoM8vlWblTWoUVnrGMWPQV j8bey00Ej1XxJbJt+YBuzFwhgkB6DgRTVv4a/UgoRYJFpeoxGyvrLLtNm709fz1r4m /9RTsDt/PIwcg== X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.64 Subject: [Qemu-devel] [PATCH v3 1/5] VFIO KABI for migration interface X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Kirti Wankhede , Zhengxiao.zx@Alibaba-inc.com, kevin.tian@intel.com, yi.l.liu@intel.com, yan.y.zhao@intel.com, eskultet@redhat.com, ziye.yang@intel.com, qemu-devel@nongnu.org, cohuck@redhat.com, shuangtai.tst@alibaba-inc.com, dgilbert@redhat.com, zhi.a.wang@intel.com, mlevitsk@redhat.com, pasic@linux.ibm.com, aik@ozlabs.ru, yulei.zhang@intel.com, eauger@redhat.com, felipe@nutanix.com, jonathan.davies@nutanix.com, changpeng.liu@intel.com, Ken.Xue@amd.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP - Defined MIGRATION region type and sub-type. - Used 2 bits to define VFIO device states. Bit 0 => 0/1 => _STOPPED/_RUNNING Bit 1 => 0/1 => _RESUMING/_SAVING Combination of these bits defines VFIO device's state during migration _RUNNING => Normal VFIO device running state. _STOPPED => VFIO device stopped. _SAVING | _RUNNING => vCPUs are running, VFIO device is running but start saving state of device i.e. pre-copy state _SAVING | _STOPPED => vCPUs are stoppped, VFIO device should be stopped, and save device state,i.e. stop-n-copy state _RESUMING => VFIO device resuming state. - Defined vfio_device_migration_info structure which will be placed at 0th offset of migration region to get/set VFIO device related information. Defined members of structure and usage on read/write access: * device_state: (write only) To convey VFIO device state to be transitioned to. * pending bytes: (read only) To get pending bytes yet to be migrated for VFIO device * data_offset: (read/write) To get or set data offset in migration from where data exist during _SAVING and _RESUMING state * data_size: (write only) To convey size of data copied in migration region during _RESUMING state * start_pfn, page_size, total_pfns: (write only) To get bitmap of dirty pages from vendor driver from given start address for total_pfns. * copied_pfns: (read only) To get number of pfns bitmap copied in migration region. Vendor driver should copy the bitmap with bits set only for pages to be marked dirty in migration region. Vendor driver should return 0 if there are 0 pages dirty in requested range. Migration region looks like: ------------------------------------------------------------------ |vfio_device_migration_info| data section | | | /////////////////////////////// | ------------------------------------------------------------------ ^ ^ ^ offset 0-trapped part data.offset data.size Data section is always followed by vfio_device_migration_info structure in the region, so data.offset will always be none-0. Offset from where data is copied is decided by kernel driver, data section can be trapped or mapped depending on how kernel driver defines data section. If mmapped, then data.offset should be page aligned, where as initial section which contain vfio_device_migration_info structure might not end at offset which is page aligned. Signed-off-by: Kirti Wankhede Reviewed-by: Neo Jia --- linux-headers/linux/vfio.h | 65 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/linux-headers/linux/vfio.h b/linux-headers/linux/vfio.h index 12a7b1dc53c8..1b12a9b95e00 100644 --- a/linux-headers/linux/vfio.h +++ b/linux-headers/linux/vfio.h @@ -368,6 +368,71 @@ struct vfio_region_gfx_edid { */ #define VFIO_REGION_SUBTYPE_IBM_NVLINK2_ATSD (1) +/* Migration region type and sub-type */ +#define VFIO_REGION_TYPE_MIGRATION (2) +#define VFIO_REGION_SUBTYPE_MIGRATION (1) + +/** + * Structure vfio_device_migration_info is placed at 0th offset of + * VFIO_REGION_SUBTYPE_MIGRATION region to get/set VFIO device related migration + * information. Field accesses from this structure are only supported at their + * native width and alignment, otherwise should return error. + * + * device_state: (write only) + * To indicate vendor driver the state VFIO device should be transitioned + * to. If device state transition fails, write to this field return error. + * It consists of 2 bits. + * - If bit 0 set, indicates _RUNNING state. When its reset, that indicates + * _STOPPED state. When device is changed to _STOPPED, driver should stop + * device before write returns. + * - If bit 1 set, indicates _SAVING state. When its reset, that indicates + * _RESUMING state. + * + * pending bytes: (read only) + * Read pending bytes yet to be migrated from vendor driver + * + * data_offset: (read/write) + * User application should read data_offset in migration region from where + * user application should read data during _SAVING state. + * User application would write data_offset in migration region from where + * user application is had written data during _RESUMING state. + * + * data_size: (write only) + * User application should write size of data copied in migration region + * during _RESUMING state. + * + * start_pfn: (write only) + * Start address pfn to get bitmap of dirty pages from vendor driver duing + * _SAVING state. + * + * page_size: (write only) + * User application should write the page_size of pfn. + * + * total_pfns: (write only) + * Total pfn count from start_pfn for which dirty bitmap is requested. + * + * copied_pfns: (read only) + * pfn count for which dirty bitmap is copied to migration region. + * Vendor driver should copy the bitmap with bits set only for pages to be + * marked dirty in migration region. + * Vendor driver should return 0 if there are 0 pages dirty in requested + * range. + */ + +struct vfio_device_migration_info { + __u32 device_state; /* VFIO device state */ +#define VFIO_DEVICE_STATE_RUNNING (1 << 0) +#define VFIO_DEVICE_STATE_SAVING (1 << 1) + __u32 reserved; + __u64 pending_bytes; + __u64 data_offset; + __u64 data_size; + __u64 start_pfn; + __u64 page_size; + __u64 total_pfns; + __u64 copied_pfns; +} __attribute__((packed)); + /* * The MSIX mappable capability informs that MSIX data of a BAR can be mmapped * which allows direct access to non-MSIX registers which happened to be within