[v6,8/8] vfio/pci: Add IGD documentation

Commit Message

Alex Williamson May 17, 2016, 8:20 p.m. UTC

Document the usage modes, host primary graphics considerations, usage,
and fw_cfg ABI required for IGD assignment with vfio.

Signed-off-by: Alex Williamson <[e-mail]>
---
 docs/igd-assign.txt |  133 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)
 create mode 100644 docs/igd-assign.txt

Comments

Eric Blake May 18, 2016, 2:23 a.m. UTC | #1

On 05/17/2016 02:20 PM, Alex Williamson wrote:
> Document the usage modes, host primary graphics considerations, usage,
> and fw_cfg ABI required for IGD assignment with vfio.
> 
> Signed-off-by: Alex Williamson <[e-mail]>
> ---
>  docs/igd-assign.txt |  133 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 133 insertions(+)
>  create mode 100644 docs/igd-assign.txt
> 
> diff --git a/docs/igd-assign.txt b/docs/igd-assign.txt
> new file mode 100644
> index 0000000..e17bb50
> --- /dev/null
> +++ b/docs/igd-assign.txt
> @@ -0,0 +1,133 @@
> +Intel Graphics Device (IGD) assignment with vfio-pci
> +====================================================

Without an explicit copyright notice and license, your documentation
defaults to GPLv2+, per the top level COPYING.  If you want something
else, it may be worth explicitly calling it out.

Otherwise, it was a good read.

Alex Williamson May 18, 2016, 6:35 p.m. UTC | #2

On Tue, 17 May 2016 20:23:42 -0600
Eric Blake <[e-mail]> wrote:

> On 05/17/2016 02:20 PM, Alex Williamson wrote:
> > Document the usage modes, host primary graphics considerations, usage,
> > and fw_cfg ABI required for IGD assignment with vfio.
> > 
> > Signed-off-by: Alex Williamson <[e-mail]>
> > ---
> >  docs/igd-assign.txt |  133 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  1 file changed, 133 insertions(+)
> >  create mode 100644 docs/igd-assign.txt
> > 
> > diff --git a/docs/igd-assign.txt b/docs/igd-assign.txt
> > new file mode 100644
> > index 0000000..e17bb50
> > --- /dev/null
> > +++ b/docs/igd-assign.txt
> > @@ -0,0 +1,133 @@
> > +Intel Graphics Device (IGD) assignment with vfio-pci
> > +====================================================  
> 
> Without an explicit copyright notice and license, your documentation
> defaults to GPLv2+, per the top level COPYING.  If you want something
> else, it may be worth explicitly calling it out.
> 
> Otherwise, it was a good read.


Thanks for reviewing, Eric.  Surveying existing docs, I see a mix of a
few with explicit GPLv2+, some PROMELA code under WTFPL/public domain,
and one lone file under FreeBSD documentation license.  The vast
majority don't set an explicit license.  I don't have a strong feeling
about this and the documentation is certainly no opus.  Should I have a
strong feeling about this?  Thanks,

Alex

Eric Blake May 18, 2016, 7:28 p.m. UTC | #3

On 05/18/2016 12:35 PM, Alex Williamson wrote:

>> Without an explicit copyright notice and license, your documentation
>> defaults to GPLv2+, per the top level COPYING.  If you want something
>> else, it may be worth explicitly calling it out.
>>
>> Otherwise, it was a good read.
> 
> 
> Thanks for reviewing, Eric.  Surveying existing docs, I see a mix of a
> few with explicit GPLv2+, some PROMELA code under WTFPL/public domain,
> and one lone file under FreeBSD documentation license.  The vast
> majority don't set an explicit license.  I don't have a strong feeling
> about this and the documentation is certainly no opus.  Should I have a
> strong feeling about this?  Thanks,

My personal opinion:

I can see arguments for two different camps:

1. Will this document ever be useful to someone implementing a parallel
code base to the same thing? If so, does a GPLv2+ document taint their
implementation to also have to be GPLv2+ compatible? I'm not a lawyer,
so I have NO IDEA if someone can legally claim that an independent
implementation of code based on a GPLv2+ specification is constrained by
the license of the doc it was implementing.  If it were me answering,
I'd say that there is no tainting involved (GPL applies to source code
in its preferred original form; and how would you justify documentation
to be a preferred original form?).  But precisely because it is a fuzzy
question, being explicit makes the solution easier to reason about:
giving an explicit license looser than GPL means that no one can argue
that the documentation is forcing a particular license on implementations.

2. Why bother? I like GPLv2+ because of its strong copyleft, and want to
make life harder for anyone that doesn't believe in the same way.
Meanwhile, sticking an explicit copyright blurb in the doc makes the doc
harder to read (you have to scroll to get to the real meat),
particularly if I'm just fine with the project-default license, so
omitting the text is okay.  Of course, once a copyright blurb is
omitted, if qemu ever changes its own license (such as to GPLv3+ [hah,
we know that won't happen without a rewrite of all the GPLv2-only
stuff]), then my document follows that change in lockstep, so explicitly
stating GPLv2+ cements my intention regardless of external context.

So at the end of the day, it boils down to how comfortable are you with
the license you've chosen, how likely do you think your doc will be
important to someone writing parallel code, and how much do you care.
I'm perfectly okay with no change, and was merely pointing it out in
case you have a stronger opinion than me.  As you rightly noted, we have
a number of interesting precedents in the directory, so there is no real
common standard, so much as personal preference, although I do prefer
that if you use an explicit license, pick one already in use rather than
adding yet another license to the overall mix of qemu.

Patch

diff --git a/docs/igd-assign.txt b/docs/igd-assign.txt
new file mode 100644
index 0000000..e17bb50
--- /dev/null
+++ b/docs/igd-assign.txt
@@ -0,0 +1,133 @@ 
+Intel Graphics Device (IGD) assignment with vfio-pci
+====================================================
+
+IGD has two different modes for assignment using vfio-pci:
+
+1) Universal Pass-Through (UPT) mode:
+
+   In this mode the IGD device is added as a *secondary* (ie. non-primary)
+   graphics device in combination with an emulated primary graphics device.
+   This mode *requires* guest driver support to remove the external
+   dependencies generally associated with IGD (see below).  Those guest
+   drivers only support this mode for Broadwell and newer IGD, according to
+   Intel.  Additionally, this mode by default, and as officially supported
+   by Intel, does not support direct video output.  The intention is to use
+   this mode either to provide hardware acceleration to the emulated graphics
+   or to use this mode in combination with guest-based remote access software,
+   for example VNC (see below for optional output support).  This mode
+   theoretically has no device specific handling dependencies on vfio-pci or
+   the VM firmware.
+
+2) "Legacy" mode:
+
+   In this mode the IGD device is intended to be the primary and exclusive
+   graphics device in the VM[1], as such QEMU does not facilitate any sort
+   of remote graphics to the VM in this mode.  A connected physical monitor
+   is the intended output device for IGD.  This mode includes several
+   requirements and restrictions:
+
+    * IGD must be given address 02.0 on the PCI root bus in the VM
+    * The host kernel must support vfio extensions for IGD (v4.6)
+    * vfio VGA support very likely needs to be enabled in the host kernel
+    * The VM firmware must support specific fw_cfg enablers for IGD
+    * The VM machine type must support a PCI host bridge at 00.0 (standard)
+    * The VM machine type must provide or allow to be created a special
+      ISA/LPC bridge device (vfio-pci-igd-lpc-bridge) on the root bus at
+      PCI address 1f.0.
+    * The IGD device must have a VGA ROM, either provided via the romfile
+      option or loaded automatically through vfio (standard).  rombar=0
+      will disable legacy mode support.
+    * Hotplug of the IGD device is not supported.
+    * The IGD device must be a SandyBridge or newer model device.
+
+For either mode, depending on the host kernel, the i915 driver in the host
+may generate faults and errors upon re-binding to an IGD device after it
+has been assigned to a VM.  It's therefore generally recommended to prevent
+such driver binding unless the host driver is known to work well for this.
+There are numerous ways to do this, i915 can be blacklisted on the host,
+the driver_override option can be used to ensure that only vfio-pci can bind
+to the device on the host[2], virsh nodedev-detach can be used to bind the
+device to vfio drivers and then managed='no' set in the VM xml to prevent
+re-binding to i915, etc.  Also note that IGD is also typically the primary
+graphics in the host and special options may be required beyond simply
+blacklisting i915 or using pci-stub/vfio-pci to take ownership of IGD as a
+PCI class device.  Lower level drivers exist that may still claim the device.
+It may therefore be necessary to use kernel boot options video=vesafb:off or
+video=efifb:off (depending on host BIOS/UEFI) or these can be combined to
+a catch-all, video=vesafb:off,efifb:off.  Error messages such as:
+
+    Failed to mmap 0000:00:02.0 BAR <>. Performance may be slow
+
+are a good indicator that such a problem exists.  The host files /proc/iomem
+and /proc/ioports are often useful for identifying drivers consuming ranges
+of the device to cause such conflicts.
+
+Additionally, IGD device are known to generate small numbers of DMAR faults
+when initially assigned.  It is believed that this is simply the IGD attempting
+to access the reserved GTT space after reset, which it no longer has access to
+when accessed from userspace.  So long as the DMAR faults are small in number
+and most importantly, not ongoing, these are not an indication of an error.
+
+Additionally++, analog VGA output (as opposed to digital outputs like HDMI,
+DVI, or DisplayPort) may be unsupported in some use cases.  In the author's
+experience, even DP to VGA adapters can be troublesome while adapters between
+digital formats work well.
+
+Usage
+=====
+The intention is for IGD assignment to be transparent for users and thus for
+management tools like libvirt.  To make use of legacy mode, simply remove all
+other graphics options and use "-nographic" and either "-vga none" or
+"-nodefaults", along with adding the device using vfio-pci:
+
+    -device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2
+
+For UPT mode, retain the default emulated graphics and simply add the vfio-pci
+device making use of any other bus address other than 02.0.  libvirt will
+default to assigning the device a UPT compatible address while legacy mode
+users will need to manually edit the XML if using a tool like virt-manager
+where the VM device address is not expressly specified.
+
+An experimental vfio-pci option also exists to enable OpRegion, and thus
+external monitor support, for UPT mode.  This can be enabled by adding
+"x-igd-opregion=on" to the vfio-pci device options for the IGD device.  As
+with legacy mode, this requires the host to support features introduced in
+the v4.6 kernel.  If Intel chooses to embrace this support, the option may
+be made non-experimental in the future, opening it to libvirt support.
+
+Developer ABI
+=============
+Legacy mode IGD support imposes two fw_cfg requirements on the VM firmware:
+
+1) "etc/igd-opregion"
+
+   This fw_cfg file exposes the OpRegion for the IGD device.  A reserved
+   region should be created below 4GB (recommended 4KB alignment), sized
+   sufficient for the fw_cfg file size, and the content of this file copied
+   to it.  The dword based address of this reserved memory region must also
+   be written to the ASLS register at offset 0xFC on the IGD device.  It is
+   recommended that firmware should make use of this fw_cfg entry for any
+   PCI class VGA device with Intel vendor ID.  Multiple of such devices
+   within a VM is undefined.
+
+2) "etc/igd-bdsm-size"
+
+   This fw_cfg file contains an 8-byte, little endian integer indicating
+   the size of the reserved memory region required for IGD stolen memory.
+   Firmware must allocate a reserved memory below 4GB with required 1MB
+   alignment equal to this size.  Additionally the base address of this
+   reserved region must be written to the dword BDSM register in PCI config
+   space of the IGD device at offset 0x5C.  As this support is related to
+   running the IGD ROM, which has other dependencies on the device appearing
+   at guest address 00:02.0, it's expected that this fw_cfg file is only
+   relevant to a single PCI class VGA device with Intel vendor ID, appearing
+   at PCI bus address 00:02.0.
+
+Footnotes
+=========
+[1] Nothing precludes adding additional emulated or assigned graphics devices
+    as non-primary, other than the combination typically not working.  I only
+    intend to set user expectations, others are welcome to find working
+    combinations or fix whatever issues prevent this from working in the common
+    case.
+[2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override