From patchwork Thu May 26 18:01:20 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Williamson X-Patchwork-Id: 9137315 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 811486075A for ; Thu, 26 May 2016 18:09:57 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 72E85280B2 for ; Thu, 26 May 2016 18:09:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6410E28255; Thu, 26 May 2016 18:09:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.9 required=2.0 tests=BAYES_00,FSL_HELO_HOME, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id A1957280B2 for ; Thu, 26 May 2016 18:09:56 +0000 (UTC) Received: from localhost ([::1]:39855 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b5zjH-0008Kz-ND for patchwork-qemu-devel@patchwork.kernel.org; Thu, 26 May 2016 14:09:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:50579) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b5zb5-0001cF-64 for qemu-devel@nongnu.org; Thu, 26 May 2016 14:01:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1b5zaz-00015w-NW for qemu-devel@nongnu.org; Thu, 26 May 2016 14:01:26 -0400 Received: from mx1.redhat.com ([209.132.183.28]:50586) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1b5zaz-00015s-FH for qemu-devel@nongnu.org; Thu, 26 May 2016 14:01:21 -0400 Received: from int-mx10.intmail.prod.int.phx2.redhat.com (int-mx10.intmail.prod.int.phx2.redhat.com [10.5.11.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 032CFC04D289 for ; Thu, 26 May 2016 18:01:21 +0000 (UTC) Received: from gimli.home (ovpn-116-97.phx2.redhat.com [10.3.116.97]) by int-mx10.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id u4QI1KvT012889; Thu, 26 May 2016 14:01:20 -0400 From: Alex Williamson To: qemu-devel@nongnu.org Date: Thu, 26 May 2016 12:01:20 -0600 Message-ID: <20160526180119.13473.25061.stgit@gimli.home> In-Reply-To: <20160526175810.13473.82479.stgit@gimli.home> References: <20160526175810.13473.82479.stgit@gimli.home> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.68 on 10.5.11.23 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.31]); Thu, 26 May 2016 18:01:21 +0000 (UTC) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 209.132.183.28 Subject: [Qemu-devel] [PULL 08/11] vfio/pci: Add IGD documentation X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Gerd Hoffmann Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP Document the usage modes, host primary graphics considerations, usage, and fw_cfg ABI required for IGD assignment with vfio. Signed-off-by: Alex Williamson Reviewed-by: Gerd Hoffmann Tested-by: Gerd Hoffmann --- docs/igd-assign.txt | 133 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 133 insertions(+) create mode 100644 docs/igd-assign.txt diff --git a/docs/igd-assign.txt b/docs/igd-assign.txt new file mode 100644 index 0000000..e17bb50 --- /dev/null +++ b/docs/igd-assign.txt @@ -0,0 +1,133 @@ +Intel Graphics Device (IGD) assignment with vfio-pci +==================================================== + +IGD has two different modes for assignment using vfio-pci: + +1) Universal Pass-Through (UPT) mode: + + In this mode the IGD device is added as a *secondary* (ie. non-primary) + graphics device in combination with an emulated primary graphics device. + This mode *requires* guest driver support to remove the external + dependencies generally associated with IGD (see below). Those guest + drivers only support this mode for Broadwell and newer IGD, according to + Intel. Additionally, this mode by default, and as officially supported + by Intel, does not support direct video output. The intention is to use + this mode either to provide hardware acceleration to the emulated graphics + or to use this mode in combination with guest-based remote access software, + for example VNC (see below for optional output support). This mode + theoretically has no device specific handling dependencies on vfio-pci or + the VM firmware. + +2) "Legacy" mode: + + In this mode the IGD device is intended to be the primary and exclusive + graphics device in the VM[1], as such QEMU does not facilitate any sort + of remote graphics to the VM in this mode. A connected physical monitor + is the intended output device for IGD. This mode includes several + requirements and restrictions: + + * IGD must be given address 02.0 on the PCI root bus in the VM + * The host kernel must support vfio extensions for IGD (v4.6) + * vfio VGA support very likely needs to be enabled in the host kernel + * The VM firmware must support specific fw_cfg enablers for IGD + * The VM machine type must support a PCI host bridge at 00.0 (standard) + * The VM machine type must provide or allow to be created a special + ISA/LPC bridge device (vfio-pci-igd-lpc-bridge) on the root bus at + PCI address 1f.0. + * The IGD device must have a VGA ROM, either provided via the romfile + option or loaded automatically through vfio (standard). rombar=0 + will disable legacy mode support. + * Hotplug of the IGD device is not supported. + * The IGD device must be a SandyBridge or newer model device. + +For either mode, depending on the host kernel, the i915 driver in the host +may generate faults and errors upon re-binding to an IGD device after it +has been assigned to a VM. It's therefore generally recommended to prevent +such driver binding unless the host driver is known to work well for this. +There are numerous ways to do this, i915 can be blacklisted on the host, +the driver_override option can be used to ensure that only vfio-pci can bind +to the device on the host[2], virsh nodedev-detach can be used to bind the +device to vfio drivers and then managed='no' set in the VM xml to prevent +re-binding to i915, etc. Also note that IGD is also typically the primary +graphics in the host and special options may be required beyond simply +blacklisting i915 or using pci-stub/vfio-pci to take ownership of IGD as a +PCI class device. Lower level drivers exist that may still claim the device. +It may therefore be necessary to use kernel boot options video=vesafb:off or +video=efifb:off (depending on host BIOS/UEFI) or these can be combined to +a catch-all, video=vesafb:off,efifb:off. Error messages such as: + + Failed to mmap 0000:00:02.0 BAR <>. Performance may be slow + +are a good indicator that such a problem exists. The host files /proc/iomem +and /proc/ioports are often useful for identifying drivers consuming ranges +of the device to cause such conflicts. + +Additionally, IGD device are known to generate small numbers of DMAR faults +when initially assigned. It is believed that this is simply the IGD attempting +to access the reserved GTT space after reset, which it no longer has access to +when accessed from userspace. So long as the DMAR faults are small in number +and most importantly, not ongoing, these are not an indication of an error. + +Additionally++, analog VGA output (as opposed to digital outputs like HDMI, +DVI, or DisplayPort) may be unsupported in some use cases. In the author's +experience, even DP to VGA adapters can be troublesome while adapters between +digital formats work well. + +Usage +===== +The intention is for IGD assignment to be transparent for users and thus for +management tools like libvirt. To make use of legacy mode, simply remove all +other graphics options and use "-nographic" and either "-vga none" or +"-nodefaults", along with adding the device using vfio-pci: + + -device vfio-pci,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2 + +For UPT mode, retain the default emulated graphics and simply add the vfio-pci +device making use of any other bus address other than 02.0. libvirt will +default to assigning the device a UPT compatible address while legacy mode +users will need to manually edit the XML if using a tool like virt-manager +where the VM device address is not expressly specified. + +An experimental vfio-pci option also exists to enable OpRegion, and thus +external monitor support, for UPT mode. This can be enabled by adding +"x-igd-opregion=on" to the vfio-pci device options for the IGD device. As +with legacy mode, this requires the host to support features introduced in +the v4.6 kernel. If Intel chooses to embrace this support, the option may +be made non-experimental in the future, opening it to libvirt support. + +Developer ABI +============= +Legacy mode IGD support imposes two fw_cfg requirements on the VM firmware: + +1) "etc/igd-opregion" + + This fw_cfg file exposes the OpRegion for the IGD device. A reserved + region should be created below 4GB (recommended 4KB alignment), sized + sufficient for the fw_cfg file size, and the content of this file copied + to it. The dword based address of this reserved memory region must also + be written to the ASLS register at offset 0xFC on the IGD device. It is + recommended that firmware should make use of this fw_cfg entry for any + PCI class VGA device with Intel vendor ID. Multiple of such devices + within a VM is undefined. + +2) "etc/igd-bdsm-size" + + This fw_cfg file contains an 8-byte, little endian integer indicating + the size of the reserved memory region required for IGD stolen memory. + Firmware must allocate a reserved memory below 4GB with required 1MB + alignment equal to this size. Additionally the base address of this + reserved region must be written to the dword BDSM register in PCI config + space of the IGD device at offset 0x5C. As this support is related to + running the IGD ROM, which has other dependencies on the device appearing + at guest address 00:02.0, it's expected that this fw_cfg file is only + relevant to a single PCI class VGA device with Intel vendor ID, appearing + at PCI bus address 00:02.0. + +Footnotes +========= +[1] Nothing precludes adding additional emulated or assigned graphics devices + as non-primary, other than the combination typically not working. I only + intend to set user expectations, others are welcome to find working + combinations or fix whatever issues prevent this from working in the common + case. +[2] # echo "vfio-pci" > /sys/bus/pci/devices/0000:00:02.0/driver_override