From patchwork Wed Oct 26 21:29:24 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kirti Wankhede X-Patchwork-Id: 9398471 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 3272260234 for ; Wed, 26 Oct 2016 21:43:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF4D629ABD for ; Wed, 26 Oct 2016 21:43:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E3EAA29C05; Wed, 26 Oct 2016 21:43:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=unavailable version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5A41729B3A for ; Wed, 26 Oct 2016 21:43:10 +0000 (UTC) Received: from localhost ([::1]:37581 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bzVyX-0004lc-GY for patchwork-qemu-devel@patchwork.kernel.org; Wed, 26 Oct 2016 17:43:09 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:54171) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1bzVlX-0002cg-L0 for qemu-devel@nongnu.org; Wed, 26 Oct 2016 17:29:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1bzVlU-0003Kv-EJ for qemu-devel@nongnu.org; Wed, 26 Oct 2016 17:29:43 -0400 Received: from hqemgate16.nvidia.com ([216.228.121.65]:2120) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1bzVlU-0003K2-3h for qemu-devel@nongnu.org; Wed, 26 Oct 2016 17:29:40 -0400 Received: from hqnvupgp08.nvidia.com (Not Verified[216.228.121.13]) by hqemgate16.nvidia.com id ; Wed, 26 Oct 2016 14:29:27 -0700 Received: from HQMAIL105.nvidia.com ([172.20.13.39]) by hqnvupgp08.nvidia.com (PGP Universal service); Wed, 26 Oct 2016 14:29:37 -0700 X-PGP-Universal: processed; by hqnvupgp08.nvidia.com on Wed, 26 Oct 2016 14:29:37 -0700 Received: from HQMAIL102.nvidia.com (172.18.146.10) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Wed, 26 Oct 2016 21:29:38 +0000 Received: from HQMAIL105.nvidia.com (172.20.187.12) by HQMAIL102.nvidia.com (172.18.146.10) with Microsoft SMTP Server (TLS) id 15.0.1210.3; Wed, 26 Oct 2016 21:29:38 +0000 Received: from kwankhede-dev.nvidia.com (172.20.13.39) by HQMAIL105.nvidia.com (172.20.187.12) with Microsoft SMTP Server (TLS) id 15.0.1210.3 via Frontend Transport; Wed, 26 Oct 2016 21:29:34 +0000 From: Kirti Wankhede To: , , , Date: Thu, 27 Oct 2016 02:59:24 +0530 Message-ID: <1477517366-27871-18-git-send-email-kwankhede@nvidia.com> X-Mailer: git-send-email 2.7.0 In-Reply-To: <1477517366-27871-1-git-send-email-kwankhede@nvidia.com> References: <1477517366-27871-1-git-send-email-kwankhede@nvidia.com> X-NVConfidentiality: public MIME-Version: 1.0 X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 X-Received-From: 216.228.121.65 Subject: [Qemu-devel] [PATCH v10 17/19] docs: Add Documentation for Mediated devices X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: jike.song@intel.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, kevin.tian@intel.com, qemu-devel@nongnu.org, Kirti Wankhede , bjsdjshi@linux.vnet.ibm.com Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Add file Documentation/vfio-mediated-device.txt that include details of mediated device framework. Signed-off-by: Kirti Wankhede Signed-off-by: Neo Jia Change-Id: I137dd646442936090d92008b115908b7b2c7bc5d --- Documentation/vfio-mdev/vfio-mediated-device.txt | 297 +++++++++++++++++++++++ 1 file changed, 297 insertions(+) create mode 100644 Documentation/vfio-mdev/vfio-mediated-device.txt diff --git a/Documentation/vfio-mdev/vfio-mediated-device.txt b/Documentation/vfio-mdev/vfio-mediated-device.txt new file mode 100644 index 000000000000..69660aff86e3 --- /dev/null +++ b/Documentation/vfio-mdev/vfio-mediated-device.txt @@ -0,0 +1,297 @@ +/* + * VFIO Mediated devices + * + * Copyright (c) 2016, NVIDIA CORPORATION. All rights reserved. + * Author: Neo Jia + * Kirti Wankhede + * + * This program is free software; you can redistribute it and/or modify + * it under the terms of the GNU General Public License version 2 as + * published by the Free Software Foundation. + */ + +Virtual Function I/O (VFIO) Mediated devices[1] +=============================================== + +The number of use cases for virtualizing DMA devices that do not have built-in +SR_IOV capability is increasing. Previously, to virtualize such devices, +developers had to create their own management interfaces and APIs, and then +integrate them with user space software. To simplify integration with user space +software, we have identified common requirements and a unified management +interface for such devices. + +The VFIO driver framework provides unified APIs for direct device access. It is +an IOMMU/device-agnostic framework for exposing direct device access to user +space in a secure, IOMMU-protected environment. This framework is used for +multiple devices, such as GPUs, network adapters, and compute accelerators. With +direct device access, virtual machines or user space applications have direct +access to the physical device. This framework is reused for mediated devices. + +The mediated core driver provides a common interface for mediated device +management that can be used by drivers of different devices. This module +provides a generic interface to perform these operations: + +* Create and destroy a mediated device +* Add a mediated device to and remove it from a mediated bus driver +* Add a mediated device to and remove it from an IOMMU group + +The mediated core driver also provides an interface to register a bus driver. +For example, the mediated VFIO mdev driver is designed for mediated devices and +supports VFIO APIs. The mediated bus driver adds a mediated device to and +removes it from a VFIO group. + +The following high-level block diagram shows the main components and interfaces +in the VFIO mediated driver framework. The diagram shows NVIDIA, Intel, and IBM +devices as examples, as these devices are the first devices to use this module. + + +---------------+ + | | + | +-----------+ | mdev_register_driver() +--------------+ + | | | +<------------------------+ | + | | mdev | | | | + | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user + | | driver | | probe()/remove() | | APIs + | | | | +--------------+ + | +-----------+ | + | | + | MDEV CORE | + | MODULE | + | mdev.ko | + | +-----------+ | mdev_register_device() +--------------+ + | | | +<------------------------+ | + | | | | | nvidia.ko |<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | | Physical | | + | | device | | mdev_register_device() +--------------+ + | | interface | |<------------------------+ | + | | | | | i915.ko |<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | | | | + | | | | mdev_register_device() +--------------+ + | | | +<------------------------+ | + | | | | | ccw_device.ko|<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | +-----------+ | + +---------------+ + + +Registration Interfaces +======================= + +The mediated core driver provides the following types of registration +interfaces: + +* Registration interface for a mediated bus driver +* Physical device driver interface + +Registration Interface for a Mediated Bus Driver +------------------------------------------------ + +The registration interface for a mediated bus driver provides the following +structure to represent a mediated device's driver: + + /* + * struct mdev_driver [2] - Mediated device's driver + * @name: driver name + * @probe: called when new device created + * @remove: called when device removed + * @driver: device driver structure + */ + struct mdev_driver { + const char *name; + int (*probe) (struct device *dev); + void (*remove) (struct device *dev); + struct device_driver driver; + }; + +A mediated bus driver for mdev should use this structure in the function calls +to register and unregister itself with the core driver: + +* Register: + + extern int mdev_register_driver(struct mdev_driver *drv, + struct module *owner); + +* Unregister: + + extern void mdev_unregister_driver(struct mdev_driver *drv); + +The mediated bus driver is responsible for adding mediated devices to the VFIO +group when devices are bound to the driver and removing mediated devices from +the VFIO when devices are unbound from the driver. + + +Physical Device Driver Interface +-------------------------------- + +The physical device driver interface provides the parent_ops[3] structure to +define the APIs to manage work in the mediated core driver that is related to +the physical device. + +The structures in the parent_ops structure are as follows: + +* dev_attr_groups: attributes of the parent device +* mdev_attr_groups: attributes of the mediated device +* supported_config: attributes to define supported configurations + +The functions in the parent_ops structure are as follows: + +* create: allocate basic resources in a driver for a mediated device +* remove: free resources in a driver when a mediated device is destroyed + +The callbacks in the parent_ops structure are as follows: + +* open: open callback of mediated device +* close: close callback of mediated device +* ioctl: ioctl callback of mediated device +* read : read emulation callback +* write: write emulation callback +* mmap: mmap emulation callback + +A driver should use the parent_ops structure in the function call to register +itself with the mdev core driver: + +extern int mdev_register_device(struct device *dev, + const struct parent_ops *ops); + +However, the parent_ops structure is not required in the function call that a +driver should use to unregister itself with the mdev core driver: + +extern void mdev_unregister_device(struct device *dev); + + +Mediated Device Management Interface Through sysfs +================================================== + +The management interface through sysfs enables user space software, such as +libvirt, to query and configure mediated devices in a hardware-agnostic fashion. +This management interface provides flexibility to the underlying physical +device's driver to support features such as: + +* Mediated device hot plug +* Multiple mediated devices in a single virtual machine +* Multiple mediated devices from different physical devices + +Links in the mdev_bus Class Directory +------------------------------------- +The /sys/class/mdev_bus/ directory contains links to devices that are registered +with the mdev core driver. + +Directories and files under the sysfs for Each Physical Device +-------------------------------------------------------------- + +|- [parent physical device] +|--- Vendor-specific-attributes [optional] +|--- [mdev_supported_types] +| |--- [] +| | |--- create +| | |--- name +| | |--- available_instances +| | |--- device_api +| | |--- description +| | |--- [devices] +| |--- [] +| | |--- create +| | |--- name +| | |--- available_instances +| | |--- device_api +| | |--- description +| | |--- [devices] +| |--- [] +| |--- create +| |--- name +| |--- available_instances +| |--- device_api +| |--- description +| |--- [devices] + +* [mdev_supported_types] + + The list of currently supported mediated device types and their details. + + [], device_api, and available_instances are mandatory attributes + that should be provided by vendor driver. + +* [] + + The [] name is created by adding the the device driver string as a + prefix to the string provided by the vendor driver. This format of this name + is as follows: + + sprintf(buf, "%s-%s", dev_driver_string(parent->dev), group->name); + +* device_api + + This attribute should show which device API is being created, for example, + "vfio-pci" for a PCI device. + +* available_instances + + This attribute should show the number of devices of type that can be + created. + +* [device] + + This directory contains links to the devices of type that have been +created. + +* name + + This attribute should show human readable name. This is optional attribute. + +* description + + This attribute should show brief features/description of the type. This is + optional attribute. + +Directories and Files Under the sysfs for Each mdev Device +---------------------------------------------------------- + +|- [parent phy device] +|--- [$MDEV_UUID] + |--- remove + |--- mdev_type {link to its type} + |--- vendor-specific-attributes [optional] + +* remove (write only) +Writing '1' to the 'remove' file destroys the mdev device. The vendor driver can +fail the remove() callback if that device is active and the vendor driver +doesn't support hot unplug. + +Example: + # echo 1 > /sys/bus/mdev/devices/$mdev_UUID/remove + +Mediated device Hot plug: +------------------------ + +Mediated devices can be created and assigned at runtime. The procedure to hot +plug a mediated device is the same as the procedure to hot plug a PCI device. + +Translation APIs for Mediated Devices +===================================== + +The following APIs are provided for translating user pfn to host pfn in a VFIO +driver: + + extern long vfio_pin_pages(struct device *dev, unsigned long *user_pfn, + long npage, int prot, unsigned long *phys_pfn); + + extern long vfio_unpin_pages(struct device *dev, unsigned long *pfn, + long npage); + +These functions call back into the back-end IOMMU module by using the pin_pages +and unpin_pages callbacks of the struct vfio_iommu_driver_ops[4]. Currently +these callbacks are supported in the TYPE1 IOMMU module. To enable them for +other IOMMU backend modules, such as PPC64 sPAPR module, they need to provide +these two callback functions. + +References +---------- + +[1] See Documentation/vfio.txt for more information on VFIO. +[2] struct mdev_driver in include/linux/mdev.h +[3] struct parent_ops in include/linux/mdev.h +[4] struct vfio_iommu_driver_ops in include/linux/vfio.h