From patchwork Thu Jun 29 05:50:37 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "lan,Tianyu" X-Patchwork-Id: 9816557 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2F8C9603D7 for ; Thu, 29 Jun 2017 12:00:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A5E2F28683 for ; Thu, 29 Jun 2017 12:00:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 993672871E; Thu, 29 Jun 2017 12:00:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=2.0 tests=BAYES_00, DATE_IN_PAST_06_12, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6F4C428683 for ; Thu, 29 Jun 2017 12:00:31 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dQY5A-00063p-Ot; Thu, 29 Jun 2017 11:58:00 +0000 Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dQY59-00062I-Aw for xen-devel@lists.xen.org; Thu, 29 Jun 2017 11:57:59 +0000 Received: from [85.158.139.211] by server-9.bemta-5.messagelabs.com id CC/EF-01994-64BE4595; Thu, 29 Jun 2017 11:57:58 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrALMWRWlGSWpSXmKPExsXS1tYhoev2OiT S4OAuboslHxezODB6HN39mymAMYo1My8pvyKBNaP1yRv2ghbdim0nvzI2MB6V7WLk4hASmM4o cfL0daYuRk4OCQFeiSPLZrBC2P4Sb47tY4YoameU+L77JDNIgk1AXeLE4omMILaIgLTEtc+XG UGKmAU2M0rM+raCHSQhLGAhsenSXrAGFgFViT0Tp7CB2LwCLhIHLj5ihtigIDHl4Xswm1PAVa Lh5hawGiGgmk89O5knMPIuYGRYxahRnFpUllqka2Sql1SUmZ5RkpuYmaNraGCql5taXJyYnpq TmFSsl5yfu4kRGBL1DAyMOxh3tfsdYpTkYFIS5TV8HhIpxJeUn1KZkVicEV9UmpNafIhRhoND SYL39EugnGBRanpqRVpmDjA4YdISHDxKIrx9IK28xQWJucWZ6RCpU4yKUuK8XK+AEgIgiYzSP Lg2WERcYpSVEuZlZGBgEOIpSC3KzSxBlX/FKM7BqCTMKw8yhSczrwRu+iugxUxAi4VngC0uSU RISTUw9st1SFf8+rdsf+CjBXZvPt4y7kpQEj/xQ3l+m/mXrQeSZ87Q+zJnKutsxvSq7nQL56D 1Bz/ECl7f6BWzbP0axwkLSxYauulZBdRyyETzb+pK8YqPuaF+KNW5VvGsyeMD582kHIUOL5c5 s6BtYt437zTJ9L+3XWzPsTbt29Z9g691vahMDGOrEktxRqKhFnNRcSIANjU8GYMCAAA= X-Env-Sender: tianyu.lan@intel.com X-Msg-Ref: server-14.tower-206.messagelabs.com!1498737475!61802085!1 X-Originating-IP: [134.134.136.24] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogMTM0LjEzNC4xMzYuMjQgPT4gMzkwOTcx\n X-StarScan-Received: X-StarScan-Version: 9.4.19; banners=-,-,- X-VirusChecked: Checked Received: (qmail 42084 invoked from network); 29 Jun 2017 11:57:57 -0000 Received: from mga09.intel.com (HELO mga09.intel.com) (134.134.136.24) by server-14.tower-206.messagelabs.com with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 29 Jun 2017 11:57:57 -0000 Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Jun 2017 04:57:55 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.40,280,1496127600"; d="scan'208";a="873010452" Received: from sky-ws.sh.intel.com (HELO localhost) ([10.239.48.74]) by FMSMGA003.fm.intel.com with ESMTP; 29 Jun 2017 04:57:53 -0700 From: Lan Tianyu To: xen-devel@lists.xen.org Date: Thu, 29 Jun 2017 01:50:37 -0400 Message-Id: <1498715457-16565-6-git-send-email-tianyu.lan@intel.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1498715457-16565-1-git-send-email-tianyu.lan@intel.com> References: <1498715457-16565-1-git-send-email-tianyu.lan@intel.com> Cc: Lan Tianyu , kevin.tian@intel.com, wei.liu2@citrix.com, andrew.cooper3@citrix.com, ian.jackson@eu.citrix.com, jbeulich@suse.com, chao.gao@intel.com Subject: [Xen-devel] [PATCH 5/25] Xen/doc: Add Xen virtual IOMMU doc X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP This patch is to add Xen virtual IOMMU doc to introduce motivation, framework, vIOMMU hypercall and xl configuration. Signed-off-by: Lan Tianyu --- docs/misc/viommu.txt | 129 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) create mode 100644 docs/misc/viommu.txt diff --git a/docs/misc/viommu.txt b/docs/misc/viommu.txt new file mode 100644 index 0000000..76d4cee --- /dev/null +++ b/docs/misc/viommu.txt @@ -0,0 +1,129 @@ +Xen virtual IOMMU + +Motivation +========== +*) Enable more than 255 vcpu support +HPC cloud service requires VM provides high performance parallel +computing and we hope to create a huge VM with >255 vcpu on one machine +to meet such requirement. Pin each vcpu to separate pcpus. + +To support >255 vcpus, X2APIC mode in guest is necessary because legacy +APIC(XAPIC) just supports 8-bit APIC ID and it only can support 255 +vcpus at most. X2APIC mode supports 32-bit APIC ID and it requires +interrupt mapping function of vIOMMU. + +The reason for this is that there is no modification to existing PCI MSI +and IOAPIC with the introduction of X2APIC. PCI MSI/IOAPIC can only send +interrupt message containing 8-bit APIC ID, which cannot address >255 +cpus. Interrupt remapping supports 32-bit APIC ID and so it's necessary +to enable >255 cpus with x2apic mode. + + +vIOMMU Architecture +=================== +vIOMMU device model is inside Xen hypervisor for following factors + 1) Avoid round trips between Qemu and Xen hypervisor + 2) Ease of integration with the rest of hypervisor + 3) HVMlite/PVH doesn't use Qemu + +* Interrupt remapping overview. +Interrupts from virtual devices and physical devices are delivered +to vLAPIC from vIOAPIC and vMSI. vIOMMU needs to remap interrupt during +this procedure. + ++---------------------------------------------------+ +|Qemu |VM | +| | +----------------+ | +| | | Device driver | | +| | +--------+-------+ | +| | ^ | +| +----------------+ | +--------+-------+ | +| | Virtual device | | | IRQ subsystem | | +| +-------+--------+ | +--------+-------+ | +| | | ^ | +| | | | | ++---------------------------+-----------------------+ +|hyperviosr | | VIRQ | +| | +---------+--------+ | +| | | vLAPIC | | +| |VIRQ +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOMMU | | +| | +---------+--------+ | +| | ^ | +| | | | +| | +---------+--------+ | +| | | vIOAPIC/vMSI | | +| | +----+----+--------+ | +| | ^ ^ | +| +-----------------+ | | +| | | ++---------------------------------------------------+ +HW |IRQ + +-------------------+ + | PCI Device | + +-------------------+ + + +vIOMMU hypercall +================ +Introduce new domctl hypercall "xen_domctl_viommu_op" to create/destroy +vIOMMU and query vIOMMU capabilities that device model can support. + +* vIOMMU hypercall parameter structure +struct xen_domctl_viommu_op { + uint32_t cmd; +#define XEN_DOMCTL_create_viommu 0 +#define XEN_DOMCTL_destroy_viommu 1 +#define XEN_DOMCTL_query_viommu_caps 2 + union { + struct { + /* IN - vIOMMU type */ + uint64_t viommu_type; + /* IN - MMIO base address of vIOMMU. */ + uint64_t base_address; + /* IN - Length of MMIO region */ + uint64_t length; + /* IN - Capabilities with which we want to create */ + uint64_t capabilities; + /* OUT - vIOMMU identity */ + uint32_t viommu_id; + } create_viommu; + + struct { + /* IN - vIOMMU identity */ + uint32_t viommu_id; + } destroy_viommu; + + struct { + /* IN - vIOMMU type */ + uint64_t viommu_type; + /* OUT - vIOMMU Capabilities */ + uint64_t caps; + } query_caps; + } u; +}; + +- XEN_DOMCTL_query_viommu_caps + Query capabilities of vIOMMU device model. vIOMMU_type specifies +which vendor vIOMMU device model(E,G Intel VTD) is targeted and hypervisor +returns capability bits(E,G interrupt remapping bit). + +- XEN_DOMCTL_create_viommu + Create vIOMMU device with vIOMMU_type, capabilities, MMIO +base address and length. Hypervisor returns viommu_id. Capabilities should +be in range of value returned by query_viommu_caps hypercall. + +- XEN_DOMCTL_destroy_viommu + Destroy vIOMMU in Xen hypervisor with viommu_id as parameters. + +xl vIOMMU configuration +======================= +viommu="type=vtd,intremap=1,x2apic=1" + +"type" - Specify vIOMMU device model type. Currently only supports Intel vtd +device model. +"intremap" - Enable vIOMMU interrupt remapping function. +"x2apic" - Support x2apic mode with interrupt remapping function.