| Message ID | 1472097235-6332-5-git-send-email-kwankhede@nvidia.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
Adding Eric. Eric, This is the v7 version of patch. I'll incorporate changes that you suggested here. Kirti. On 8/25/2016 9:23 AM, Kirti Wankhede wrote: > Add file Documentation/vfio-mediated-device.txt that include details of > mediated device framework. > > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com> > Signed-off-by: Neo Jia <cjia@nvidia.com> > Change-Id: I137dd646442936090d92008b115908b7b2c7bc5d > Reviewed-on: http://git-master/r/1182512 > Reviewed-by: Automatic_Commit_Validation_User > --- > Documentation/vfio-mediated-device.txt | 203 +++++++++++++++++++++++++++++++++ > 1 file changed, 203 insertions(+) > create mode 100644 Documentation/vfio-mediated-device.txt > > diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt > new file mode 100644 > index 000000000000..237d8eb630b7 > --- /dev/null > +++ b/Documentation/vfio-mediated-device.txt > @@ -0,0 +1,203 @@ > +VFIO Mediated devices [1] > +------------------------------------------------------------------------------- > + > +There are more and more use cases/demands to virtualize the DMA devices which > +doesn't have SR_IOV capability built-in. To do this, drivers of different > +devices had to develop their own management interface and set of APIs and then > +integrate it to user space software. We've identified common requirements and > +unified management interface for such devices to make user space software > +integration easier. > + > +The VFIO driver framework provides unified APIs for direct device access. It is > +an IOMMU/device agnostic framework for exposing direct device access to > +user space, in a secure, IOMMU protected environment. This framework is > +used for multiple devices like GPUs, network adapters and compute accelerators. > +With direct device access, virtual machines or user space applications have > +direct access of physical device. This framework is reused for mediated devices. > + > +Mediated core driver provides a common interface for mediated device management > +that can be used by drivers of different devices. This module provides a generic > +interface to create/destroy mediated device, add/remove it to mediated bus > +driver, add/remove device to IOMMU group. It also provides an interface to > +register bus driver, for example, Mediated VFIO mdev driver is designed for > +mediated devices and supports VFIO APIs. Mediated bus driver add/delete mediated > +device to VFIO Group. > + > +Below is the high Level block diagram, with NVIDIA, Intel and IBM devices > +as example, since these are the devices which are going to actively use > +this module as of now. > + > + +---------------+ > + | | > + | +-----------+ | mdev_register_driver() +--------------+ > + | | | +<------------------------+ | > + | | mdev | | | | > + | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user > + | | driver | | probe()/remove() | | APIs > + | | | | +--------------+ > + | +-----------+ | > + | | > + | MDEV CORE | > + | MODULE | > + | mdev.ko | > + | +-----------+ | mdev_register_device() +--------------+ > + | | | +<------------------------+ | > + | | | | | nvidia.ko |<-> physical > + | | | +------------------------>+ | device > + | | | | callbacks +--------------+ > + | | Physical | | > + | | device | | mdev_register_device() +--------------+ > + | | interface | |<------------------------+ | > + | | | | | i915.ko |<-> physical > + | | | +------------------------>+ | device > + | | | | callbacks +--------------+ > + | | | | > + | | | | mdev_register_device() +--------------+ > + | | | +<------------------------+ | > + | | | | | ccw_device.ko|<-> physical > + | | | +------------------------>+ | device > + | | | | callbacks +--------------+ > + | +-----------+ | > + +---------------+ > + > + > +Registration Interfaces > +------------------------------------------------------------------------------- > + > +Mediated core driver provides two types of registration interfaces: > + > +1. Registration interface for mediated bus driver: > +------------------------------------------------- > + /* > + * struct mdev_driver [2] - Mediated device's driver > + * @name: driver name > + * @probe: called when new device created > + * @remove: called when device removed > + * @driver: device driver structure > + */ > + struct mdev_driver { > + const char *name; > + int (*probe) (struct device *dev); > + void (*remove) (struct device *dev); > + struct device_driver driver; > + }; > + > +Mediated bus driver for mdev should use this interface to register and > +unregister with core driver respectively: > + > +extern int mdev_register_driver(struct mdev_driver *drv, struct module *owner); > +extern void mdev_unregister_driver(struct mdev_driver *drv); > + > +Mediated bus driver is responsible to add/delete mediated devices to/from VFIO > +group when devices are bound and unbound to the driver. > + > +2. Physical device driver interface: > +----------------------------------- > +This interface [3] provides a set of APIs to manage physical device related work > +in its driver. APIs are: > + > +* dev_attr_groups: attributes of the parent device. > +* mdev_attr_groups: attributes of the mediated device. > +* supported_config: to provide supported configuration list by the driver. > +* create: to allocate basic resources in driver for a mediated device. > +* destroy: to free resources in driver when mediated device is destroyed. > +* reset: to free and reallocate resources in driver on mediated device reset. > +* set_online_status: to change online status of mediated device. > +* get_online_status: to get current (online/offline) status of mediated device. > +* read : read emulation callback. > +* write: write emulation callback. > +* mmap: mmap emulation callback. > +* get_irq_info: to retrieve information about mediated device's IRQ. > +* set_irqs: gives interrupt configuration information that VMM sets. > +* get_region_info: to provide region size and its flags for the mediated device. > + Vendor driver can provide the capability id and corresponding capability > + structure if want to support a capability. > +* get_device_info: to retrieve VFIO device related flags, number of regions and > + number of IRQs supported. > + > +Drivers should use this interface to register and unregister device to mdev core > +driver respectively: > + > +extern int mdev_register_device(struct device *dev, > + const struct parent_ops *ops); > +extern void mdev_unregister_device(struct device *dev); > + > +Mediated device management interface via sysfs > +------------------------------------------------------------------------------- > +This is the interface that allows user space software, like libvirt, to query > +and configure mediated device in a HW agnostic fashion. This management > +interface provide flexibility to underlying physical device's driver to support > +mediated device hotplug, multiple mediated devices per virtual machine, multiple > +mediated devices from different physical devices, etc. > + > +Under per-physical device sysfs: > +-------------------------------- > + > +* mdev_supported_types: (read only) > + List the current supported mediated device types and its details. > + > +* mdev_create: (write only) > + Create a mediated device on target physical device. > + Input syntax: <UUID:params> > + where, > + UUID: mediated device's UUID > + params: extra parameters required by driver > + Example: > + # echo "12345678-1234-1234-1234-123456789abc:0" > > + /sys/bus/pci/devices/0000\:05\:00.0/mdev_create > + > +* mdev_destroy: (write only) > + Destroy a mediated device on a target physical device. > + Input syntax: <UUID> > + where, > + UUID: mediated device's UUID > + Example: > + # echo "12345678-1234-1234-1234-123456789abc" > > + /sys/bus/pci/devices/0000\:05\:00.0/mdev_destroy > + > +Under per mdev device: > +---------------------------------------- > + > +* online: (read write) > + Read on this file provides current status of mediated device (0 or 1). > + Write on this file (0 or 1) will change the state of mediated device. > + This trigger the registration callback to notify the driver to commit > + or free mediated device resources. This callback is blocking call, > + successful return of this call will indicate requested mdev resources > + has been fully committed, the VMM should continue. > + Example: > + # echo "1|0" > /sys/bus/mdev/devices/$mdev_UUID/online > + > + > +Mediated device Hotplug: > +----------------------- > + > +To support mediated device hotplug, <mdev_create> and <mdev_destroy> can be > +accessed during VM runtime, and the corresponding registration callback is > +invoked to allow driver to support hotplug. > + > +Translation APIs for Mediated device > +------------------------------------------------------------------------------ > + > +Below APIs are provided for user pfn to host pfn translation in VFIO driver: > + > +extern long vfio_pin_pages(struct device *dev, unsigned long *user_pfn, > + long npage, int prot, unsigned long *phys_pfn); > + > +extern long vfio_unpin_pages(struct device *dev, unsigned long *pfn, > + long npage); > + > +These functions call back into the backend IOMMU module using two callbacks of > +struct vfio_iommu_driver_ops, pin_pages and unpin_pages [4]. Currently these are > +supported in TYPE1 IOMMU module. To enable the same for other IOMMU backend > +modules, such as PPC64 sPAPR module, they need to provide these two callback > +functions. > + > +References > +------------------------------------------------------------------------------- > + > +[1] See Documentation/vfio.txt for more information on VFIO. > +[2] struct mdev_driver in include/linux/mdev.h > +[3] struct parent_ops in include/linux/mdev.h > +[4] struct vfio_iommu_driver_ops in include/linux/vfio.h > + > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/Documentation/vfio-mediated-device.txt b/Documentation/vfio-mediated-device.txt new file mode 100644 index 000000000000..237d8eb630b7 --- /dev/null +++ b/Documentation/vfio-mediated-device.txt @@ -0,0 +1,203 @@ +VFIO Mediated devices [1] +------------------------------------------------------------------------------- + +There are more and more use cases/demands to virtualize the DMA devices which +doesn't have SR_IOV capability built-in. To do this, drivers of different +devices had to develop their own management interface and set of APIs and then +integrate it to user space software. We've identified common requirements and +unified management interface for such devices to make user space software +integration easier. + +The VFIO driver framework provides unified APIs for direct device access. It is +an IOMMU/device agnostic framework for exposing direct device access to +user space, in a secure, IOMMU protected environment. This framework is +used for multiple devices like GPUs, network adapters and compute accelerators. +With direct device access, virtual machines or user space applications have +direct access of physical device. This framework is reused for mediated devices. + +Mediated core driver provides a common interface for mediated device management +that can be used by drivers of different devices. This module provides a generic +interface to create/destroy mediated device, add/remove it to mediated bus +driver, add/remove device to IOMMU group. It also provides an interface to +register bus driver, for example, Mediated VFIO mdev driver is designed for +mediated devices and supports VFIO APIs. Mediated bus driver add/delete mediated +device to VFIO Group. + +Below is the high Level block diagram, with NVIDIA, Intel and IBM devices +as example, since these are the devices which are going to actively use +this module as of now. + + +---------------+ + | | + | +-----------+ | mdev_register_driver() +--------------+ + | | | +<------------------------+ | + | | mdev | | | | + | | bus | +------------------------>+ vfio_mdev.ko |<-> VFIO user + | | driver | | probe()/remove() | | APIs + | | | | +--------------+ + | +-----------+ | + | | + | MDEV CORE | + | MODULE | + | mdev.ko | + | +-----------+ | mdev_register_device() +--------------+ + | | | +<------------------------+ | + | | | | | nvidia.ko |<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | | Physical | | + | | device | | mdev_register_device() +--------------+ + | | interface | |<------------------------+ | + | | | | | i915.ko |<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | | | | + | | | | mdev_register_device() +--------------+ + | | | +<------------------------+ | + | | | | | ccw_device.ko|<-> physical + | | | +------------------------>+ | device + | | | | callbacks +--------------+ + | +-----------+ | + +---------------+ + + +Registration Interfaces +------------------------------------------------------------------------------- + +Mediated core driver provides two types of registration interfaces: + +1. Registration interface for mediated bus driver: +------------------------------------------------- + /* + * struct mdev_driver [2] - Mediated device's driver + * @name: driver name + * @probe: called when new device created + * @remove: called when device removed + * @driver: device driver structure + */ + struct mdev_driver { + const char *name; + int (*probe) (struct device *dev); + void (*remove) (struct device *dev); + struct device_driver driver; + }; + +Mediated bus driver for mdev should use this interface to register and +unregister with core driver respectively: + +extern int mdev_register_driver(struct mdev_driver *drv, struct module *owner); +extern void mdev_unregister_driver(struct mdev_driver *drv); + +Mediated bus driver is responsible to add/delete mediated devices to/from VFIO +group when devices are bound and unbound to the driver. + +2. Physical device driver interface: +----------------------------------- +This interface [3] provides a set of APIs to manage physical device related work +in its driver. APIs are: + +* dev_attr_groups: attributes of the parent device. +* mdev_attr_groups: attributes of the mediated device. +* supported_config: to provide supported configuration list by the driver. +* create: to allocate basic resources in driver for a mediated device. +* destroy: to free resources in driver when mediated device is destroyed. +* reset: to free and reallocate resources in driver on mediated device reset. +* set_online_status: to change online status of mediated device. +* get_online_status: to get current (online/offline) status of mediated device. +* read : read emulation callback. +* write: write emulation callback. +* mmap: mmap emulation callback. +* get_irq_info: to retrieve information about mediated device's IRQ. +* set_irqs: gives interrupt configuration information that VMM sets. +* get_region_info: to provide region size and its flags for the mediated device. + Vendor driver can provide the capability id and corresponding capability + structure if want to support a capability. +* get_device_info: to retrieve VFIO device related flags, number of regions and + number of IRQs supported. + +Drivers should use this interface to register and unregister device to mdev core +driver respectively: + +extern int mdev_register_device(struct device *dev, + const struct parent_ops *ops); +extern void mdev_unregister_device(struct device *dev); + +Mediated device management interface via sysfs +------------------------------------------------------------------------------- +This is the interface that allows user space software, like libvirt, to query +and configure mediated device in a HW agnostic fashion. This management +interface provide flexibility to underlying physical device's driver to support +mediated device hotplug, multiple mediated devices per virtual machine, multiple +mediated devices from different physical devices, etc. + +Under per-physical device sysfs: +-------------------------------- + +* mdev_supported_types: (read only) + List the current supported mediated device types and its details. + +* mdev_create: (write only) + Create a mediated device on target physical device. + Input syntax: <UUID:params> + where, + UUID: mediated device's UUID + params: extra parameters required by driver + Example: + # echo "12345678-1234-1234-1234-123456789abc:0" > + /sys/bus/pci/devices/0000\:05\:00.0/mdev_create + +* mdev_destroy: (write only) + Destroy a mediated device on a target physical device. + Input syntax: <UUID> + where, + UUID: mediated device's UUID + Example: + # echo "12345678-1234-1234-1234-123456789abc" > + /sys/bus/pci/devices/0000\:05\:00.0/mdev_destroy + +Under per mdev device: +---------------------------------------- + +* online: (read write) + Read on this file provides current status of mediated device (0 or 1). + Write on this file (0 or 1) will change the state of mediated device. + This trigger the registration callback to notify the driver to commit + or free mediated device resources. This callback is blocking call, + successful return of this call will indicate requested mdev resources + has been fully committed, the VMM should continue. + Example: + # echo "1|0" > /sys/bus/mdev/devices/$mdev_UUID/online + + +Mediated device Hotplug: +----------------------- + +To support mediated device hotplug, <mdev_create> and <mdev_destroy> can be +accessed during VM runtime, and the corresponding registration callback is +invoked to allow driver to support hotplug. + +Translation APIs for Mediated device +------------------------------------------------------------------------------ + +Below APIs are provided for user pfn to host pfn translation in VFIO driver: + +extern long vfio_pin_pages(struct device *dev, unsigned long *user_pfn, + long npage, int prot, unsigned long *phys_pfn); + +extern long vfio_unpin_pages(struct device *dev, unsigned long *pfn, + long npage); + +These functions call back into the backend IOMMU module using two callbacks of +struct vfio_iommu_driver_ops, pin_pages and unpin_pages [4]. Currently these are +supported in TYPE1 IOMMU module. To enable the same for other IOMMU backend +modules, such as PPC64 sPAPR module, they need to provide these two callback +functions. + +References +------------------------------------------------------------------------------- + +[1] See Documentation/vfio.txt for more information on VFIO. +[2] struct mdev_driver in include/linux/mdev.h +[3] struct parent_ops in include/linux/mdev.h +[4] struct vfio_iommu_driver_ops in include/linux/vfio.h +