From patchwork Mon Aug 14 14:28:42 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 9899057 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 62948602BA for ; Mon, 14 Aug 2017 14:30:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 53A7528669 for ; Mon, 14 Aug 2017 14:30:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4846E28668; Mon, 14 Aug 2017 14:30:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6460728634 for ; Mon, 14 Aug 2017 14:30:55 +0000 (UTC) Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dhGMZ-0004MO-BC; Mon, 14 Aug 2017 14:29:03 +0000 Received: from mail6.bemta3.messagelabs.com ([195.245.230.39]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1dhGMY-0004LB-17 for xen-devel@lists.xenproject.org; Mon, 14 Aug 2017 14:29:02 +0000 Received: from [85.158.137.68] by server-5.bemta-3.messagelabs.com id F2/18-02181-DA3B1995; Mon, 14 Aug 2017 14:29:01 +0000 X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFtrMIsWRWlGSWpSXmKPExsXitHRDpO6azRM jDf6uNLD4vmUykwOjx+EPV1gCGKNYM/OS8isSWDN61tUX9MRX3Lp4g62BcaZ7FyMnh4SAv8SE F5/ZQWw2AR2Ji3N3snUxcnCICKhI3N5r0MXIxcEscINRYu22v6wgcWEBP4mXc+VBylkEVCV6H 8xnAbF5BSwl5ny9xAYxUk/i7cQXjCA2p4CVxJfL18HGCwHVHO2/zghRLyhxcuYTsF5mAU2J1u 2/2SFseYnmrbOZIeoVJfrnPWCbwMg3C0nLLCQts5C0LGBkXsWoUZxaVJZapGtkoZdUlJmeUZK bmJmja2hgrJebWlycmJ6ak5hUrJecn7uJERho9QwMjDsY20/4HWKU5GBSEuVN8OmNFOJLyk+p zEgszogvKs1JLT7EqMHBIbB57eoLjFIsefl5qUoSvG82TYwUEixKTU+tSMvMAcYCTKkEB4+SC G8PSJq3uCAxtzgzHSJ1ilGX49WE/9+YhMBmSInzhoAUCYAUZZTmwY2AxeUlRlkpYV5GBgYGIZ 6C1KLczBJU+VeM4hyMSsK8D0Cm8GTmlcBtegV0BBPQEX0gv/AWlyQipKQaGE87fZn2h3X+43I Rxu9vj78uVTtfPe9vNvuz774b3052+1uyKCfYTP764TYL28zALKsP0bvXO7jemsKxTG1ti/2z dU4V+oUCV7tNOzgDj19syBaNkS+5uV7g5PFCh6U1K/Wyuxa0WVhudNjJNjXY42WnwlrWpGM6P urfYx4oO9vMnr/1ysMCOyWW4oxEQy3mouJEACLlNLnGAgAA X-Env-Sender: prvs=3925c166a=roger.pau@citrix.com X-Msg-Ref: server-8.tower-31.messagelabs.com!1502720937!110025701!2 X-Originating-IP: [66.165.176.89] X-SpamReason: No, hits=0.0 required=7.0 tests=sa_preprocessor: VHJ1c3RlZCBJUDogNjYuMTY1LjE3Ni44OSA9PiAyMDMwMDc=\n, received_headers: No Received headers X-StarScan-Received: X-StarScan-Version: 9.4.45; banners=-,-,- X-VirusChecked: Checked Received: (qmail 55972 invoked from network); 14 Aug 2017 14:29:00 -0000 Received: from smtp.citrix.com (HELO SMTP.CITRIX.COM) (66.165.176.89) by server-8.tower-31.messagelabs.com with RC4-SHA encrypted SMTP; 14 Aug 2017 14:29:00 -0000 X-IronPort-AV: E=Sophos;i="5.41,373,1498521600"; d="scan'208";a="435407660" From: Roger Pau Monne To: Date: Mon, 14 Aug 2017 15:28:42 +0100 Message-ID: <20170814142850.39133-4-roger.pau@citrix.com> X-Mailer: git-send-email 2.11.0 (Apple Git-81) In-Reply-To: <20170814142850.39133-1-roger.pau@citrix.com> References: <20170814142850.39133-1-roger.pau@citrix.com> MIME-Version: 1.0 Cc: Andrew Cooper , Paul Durrant , Jan Beulich , boris.ostrovsky@oracle.com, Roger Pau Monne Subject: [Xen-devel] [PATCH v5 03/11] x86/mmcfg: add handlers for the PVH Dom0 MMCFG areas X-BeenThere: xen-devel@lists.xen.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" X-Virus-Scanned: ClamAV using ClamSMTP Introduce a set of handlers for the accesses to the MMCFG areas. Those areas are setup based on the contents of the hardware MMCFG tables, and the list of handled MMCFG areas is stored inside of the hvm_domain struct. The read/writes are forwarded to the generic vpci handlers once the address is decoded in order to obtain the device and register the guest is trying to access. Signed-off-by: Roger Pau Monné Reviewed-by: Paul Durrant --- Cc: Jan Beulich Cc: Andrew Cooper Cc: Paul Durrant --- Changes since v4: - Change the attribute of pvh_setup_mmcfg to __hwdom_init. - Try to add as many MMCFG regions as possible, even if one fails to add. - Change some fields of the hvm_mmcfg struct: turn size into a unsigned int, segment into uint16_t and bus into uint8_t. - Convert some address parameters from unsigned long to paddr_t for consistency. - Make vpci_mmcfg_decode_addr return the decoded register in the return of the function. - Introduce a new macro to convert a MMCFG address into a BDF, and use it in vpci_mmcfg_decode_addr to clarify the logic. - In vpci_mmcfg_{read/write} unify the logic for 8B accesses and smaller ones. - Add the __hwdom_init attribute to register_vpci_mmcfg_handler. - Test that reg + size doesn't cross a device boundary. Changes since v3: - Propagate changes from previous patches: drop xen_ prefix for vpci functions, pass slot and func instead of devfn and fix the error paths of the MMCFG handlers. - s/ecam/mmcfg/. - Move the destroy code to a separate function, so the hvm_mmcfg struct can be private to hvm/io.c. - Constify the return of vpci_mmcfg_find. - Use d instead of v->domain in vpci_mmcfg_accept. - Allow 8byte accesses to the mmcfg. Changes since v1: - Added locking. --- xen/arch/x86/hvm/dom0_build.c | 22 +++++ xen/arch/x86/hvm/hvm.c | 3 + xen/arch/x86/hvm/io.c | 183 ++++++++++++++++++++++++++++++++++++++- xen/include/asm-x86/hvm/domain.h | 3 + xen/include/asm-x86/hvm/io.h | 7 ++ xen/include/asm-x86/pci.h | 2 + 6 files changed, 219 insertions(+), 1 deletion(-) diff --git a/xen/arch/x86/hvm/dom0_build.c b/xen/arch/x86/hvm/dom0_build.c index 0e7d06be95..04a8682d33 100644 --- a/xen/arch/x86/hvm/dom0_build.c +++ b/xen/arch/x86/hvm/dom0_build.c @@ -38,6 +38,8 @@ #include #include +#include "../x86_64/mmconfig.h" + /* * Have the TSS cover the ISA port range, which makes it * - 104 bytes base structure @@ -1041,6 +1043,24 @@ static int __init pvh_setup_acpi(struct domain *d, paddr_t start_info) return 0; } +static void __hwdom_init pvh_setup_mmcfg(struct domain *d) +{ + unsigned int i; + int rc; + + for ( i = 0; i < pci_mmcfg_config_num; i++ ) + { + rc = register_vpci_mmcfg_handler(d, pci_mmcfg_config[i].address, + pci_mmcfg_config[i].start_bus_number, + pci_mmcfg_config[i].end_bus_number, + pci_mmcfg_config[i].pci_segment); + if ( rc ) + printk("Unable to setup MMCFG handler at %#lx for segment %u\n", + pci_mmcfg_config[i].address, + pci_mmcfg_config[i].pci_segment); + } +} + int __init dom0_construct_pvh(struct domain *d, const module_t *image, unsigned long image_headroom, module_t *initrd, @@ -1090,6 +1110,8 @@ int __init dom0_construct_pvh(struct domain *d, const module_t *image, return rc; } + pvh_setup_mmcfg(d); + panic("Building a PVHv2 Dom0 is not yet supported."); return 0; } diff --git a/xen/arch/x86/hvm/hvm.c b/xen/arch/x86/hvm/hvm.c index cc73df8dc7..3168973820 100644 --- a/xen/arch/x86/hvm/hvm.c +++ b/xen/arch/x86/hvm/hvm.c @@ -583,6 +583,7 @@ int hvm_domain_initialise(struct domain *d, unsigned long domcr_flags, spin_lock_init(&d->arch.hvm_domain.write_map.lock); INIT_LIST_HEAD(&d->arch.hvm_domain.write_map.list); INIT_LIST_HEAD(&d->arch.hvm_domain.g2m_ioport_list); + INIT_LIST_HEAD(&d->arch.hvm_domain.mmcfg_regions); rc = create_perdomain_mapping(d, PERDOMAIN_VIRT_START, 0, NULL, NULL); if ( rc ) @@ -728,6 +729,8 @@ void hvm_domain_destroy(struct domain *d) list_del(&ioport->list); xfree(ioport); } + + destroy_vpci_mmcfg(&d->arch.hvm_domain.mmcfg_regions); } static int hvm_save_tsc_adjust(struct domain *d, hvm_domain_context_t *h) diff --git a/xen/arch/x86/hvm/io.c b/xen/arch/x86/hvm/io.c index c3b68eb257..2845dc5b48 100644 --- a/xen/arch/x86/hvm/io.c +++ b/xen/arch/x86/hvm/io.c @@ -280,7 +280,7 @@ unsigned int hvm_pci_decode_addr(unsigned int cf8, unsigned int addr, static bool vpci_access_allowed(unsigned int reg, unsigned int len) { /* Check access size. */ - if ( len != 1 && len != 2 && len != 4 ) + if ( len != 1 && len != 2 && len != 4 && len != 8 ) return false; /* Check that access is size aligned. */ @@ -391,6 +391,187 @@ void register_vpci_portio_handler(struct domain *d) handler->ops = &vpci_portio_ops; } +struct hvm_mmcfg { + struct list_head next; + paddr_t addr; + unsigned int size; + uint16_t segment; + int8_t bus; +}; + +/* Handlers to trap PCI MMCFG config accesses. */ +static const struct hvm_mmcfg *vpci_mmcfg_find(struct domain *d, paddr_t addr) +{ + const struct hvm_mmcfg *mmcfg; + + list_for_each_entry ( mmcfg, &d->arch.hvm_domain.mmcfg_regions, next ) + if ( addr >= mmcfg->addr && addr < mmcfg->addr + mmcfg->size ) + return mmcfg; + + return NULL; +} + +static unsigned int vpci_mmcfg_decode_addr(const struct hvm_mmcfg *mmcfg, + paddr_t addr, unsigned int *bus, + unsigned int *slot, + unsigned int *func) +{ + unsigned int bdf; + + addr -= mmcfg->addr; + bdf = MMCFG_BDF(addr); + *bus = PCI_BUS(bdf) + mmcfg->bus; + *slot = PCI_SLOT(bdf); + *func = PCI_FUNC(bdf); + + return addr & (PCI_CFG_SPACE_EXP_SIZE - 1); +} + +static int vpci_mmcfg_accept(struct vcpu *v, unsigned long addr) +{ + struct domain *d = v->domain; + bool found; + + vpci_rlock(d); + found = vpci_mmcfg_find(d, addr); + vpci_runlock(d); + + return found; +} + +static int vpci_mmcfg_read(struct vcpu *v, unsigned long addr, + unsigned int len, unsigned long *data) +{ + struct domain *d = v->domain; + const struct hvm_mmcfg *mmcfg; + unsigned int bus, slot, func, reg; + + *data = ~0ul; + + vpci_rlock(d); + mmcfg = vpci_mmcfg_find(d, addr); + if ( !mmcfg ) + { + vpci_runlock(d); + return X86EMUL_OKAY; + } + + reg = vpci_mmcfg_decode_addr(mmcfg, addr, &bus, &slot, &func); + + if ( !vpci_access_allowed(reg, len) || + (reg + len) > PCI_CFG_SPACE_EXP_SIZE ) + { + vpci_runlock(d); + return X86EMUL_OKAY; + } + + /* + * According to the PCIe 3.1A specification: + * - Configuration Reads and Writes must usually be DWORD or smaller + * in size. + * - Because Root Complex implementations are not required to support + * accesses to a RCRB that cross DW boundaries [...] software + * should take care not to cause the generation of such accesses + * when accessing a RCRB unless the Root Complex will support the + * access. + * Xen however supports 8byte accesses by splitting them into two + * 4byte accesses. + */ + *data = vpci_read(mmcfg->segment, bus, slot, func, reg, min(4u, len)); + if ( len == 8 ) + *data |= (uint64_t)vpci_read(mmcfg->segment, bus, slot, func, + reg + 4, 4) << 32; + vpci_runlock(d); + + return X86EMUL_OKAY; +} + +static int vpci_mmcfg_write(struct vcpu *v, unsigned long addr, + unsigned int len, unsigned long data) +{ + struct domain *d = v->domain; + const struct hvm_mmcfg *mmcfg; + unsigned int bus, slot, func, reg; + + vpci_wlock(d); + mmcfg = vpci_mmcfg_find(d, addr); + if ( !mmcfg ) + { + vpci_wunlock(d); + return X86EMUL_OKAY; + } + + reg = vpci_mmcfg_decode_addr(mmcfg, addr, &bus, &slot, &func); + + if ( !vpci_access_allowed(reg, len) || + (reg + len) > PCI_CFG_SPACE_EXP_SIZE ) + { + vpci_wunlock(d); + return X86EMUL_OKAY; + } + + vpci_write(mmcfg->segment, bus, slot, func, reg, min(4u, len), data); + if ( len == 8 ) + vpci_write(mmcfg->segment, bus, slot, func, reg + 4, 4, data >> 32); + vpci_wunlock(d); + + return X86EMUL_OKAY; +} + +static const struct hvm_mmio_ops vpci_mmcfg_ops = { + .check = vpci_mmcfg_accept, + .read = vpci_mmcfg_read, + .write = vpci_mmcfg_write, +}; + +int __hwdom_init register_vpci_mmcfg_handler(struct domain *d, paddr_t addr, + unsigned int start_bus, + unsigned int end_bus, + unsigned int seg) +{ + struct hvm_mmcfg *mmcfg; + + ASSERT(is_hardware_domain(d)); + + vpci_wlock(d); + if ( vpci_mmcfg_find(d, addr) ) + { + vpci_wunlock(d); + return -EEXIST; + } + + mmcfg = xmalloc(struct hvm_mmcfg); + if ( !mmcfg ) + { + vpci_wunlock(d); + return -ENOMEM; + } + + if ( list_empty(&d->arch.hvm_domain.mmcfg_regions) ) + register_mmio_handler(d, &vpci_mmcfg_ops); + + mmcfg->addr = addr + (start_bus << 20); + mmcfg->bus = start_bus; + mmcfg->segment = seg; + mmcfg->size = (end_bus - start_bus + 1) << 20; + list_add(&mmcfg->next, &d->arch.hvm_domain.mmcfg_regions); + vpci_wunlock(d); + + return 0; +} + +void destroy_vpci_mmcfg(struct list_head *domain_mmcfg) +{ + while ( !list_empty(domain_mmcfg) ) + { + struct hvm_mmcfg *mmcfg = list_first_entry(domain_mmcfg, + struct hvm_mmcfg, next); + + list_del(&mmcfg->next); + xfree(mmcfg); + } +} + /* * Local variables: * mode: C diff --git a/xen/include/asm-x86/hvm/domain.h b/xen/include/asm-x86/hvm/domain.h index 3a54d50606..e8dc01bc3e 100644 --- a/xen/include/asm-x86/hvm/domain.h +++ b/xen/include/asm-x86/hvm/domain.h @@ -187,6 +187,9 @@ struct hvm_domain { /* Lock for the PCI emulation layer (vPCI). */ rwlock_t vpci_lock; + /* List of MMCFG regions trapped by Xen. */ + struct list_head mmcfg_regions; + /* List of permanently write-mapped pages. */ struct { spinlock_t lock; diff --git a/xen/include/asm-x86/hvm/io.h b/xen/include/asm-x86/hvm/io.h index 01322a2e21..837046026c 100644 --- a/xen/include/asm-x86/hvm/io.h +++ b/xen/include/asm-x86/hvm/io.h @@ -163,6 +163,13 @@ void register_g2m_portio_handler(struct domain *d); /* HVM port IO handler for PCI accesses. */ void register_vpci_portio_handler(struct domain *d); +/* HVM MMIO handler for PCI MMCFG accesses. */ +int register_vpci_mmcfg_handler(struct domain *d, paddr_t addr, + unsigned int start_bus, unsigned int end_bus, + unsigned int seg); +/* Destroy tracked MMCFG areas. */ +void destroy_vpci_mmcfg(struct list_head *domain_mmcfg); + #endif /* __ASM_X86_HVM_IO_H__ */ diff --git a/xen/include/asm-x86/pci.h b/xen/include/asm-x86/pci.h index 36801d317b..ac16c8fd5d 100644 --- a/xen/include/asm-x86/pci.h +++ b/xen/include/asm-x86/pci.h @@ -6,6 +6,8 @@ #define CF8_ADDR_HI(cf8) ( ((cf8) & 0x0f000000) >> 16) #define CF8_ENABLED(cf8) (!!((cf8) & 0x80000000)) +#define MMCFG_BDF(addr) ( ((addr) & 0x0ffff000) >> 12) + #define IS_SNB_GFX(id) (id == 0x01068086 || id == 0x01168086 \ || id == 0x01268086 || id == 0x01028086 \ || id == 0x01128086 || id == 0x01228086 \