From patchwork Thu Oct 10 11:03:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Roger_Pau_Monn=C3=A9?= X-Patchwork-Id: 11183105 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEBE11668 for ; Thu, 10 Oct 2019 11:05:42 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8FB32206A1 for ; Thu, 10 Oct 2019 11:05:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=citrix.com header.i=@citrix.com header.b="R0Pf56u4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8FB32206A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=citrix.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=xen-devel-bounces@lists.xenproject.org Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iIWEp-0006yf-CJ; Thu, 10 Oct 2019 11:04:07 +0000 Received: from all-amaz-eas1.inumbo.com ([34.197.232.57] helo=us1-amaz-eas2.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.89) (envelope-from ) id 1iIWEn-0006xq-TZ for xen-devel@lists.xenproject.org; Thu, 10 Oct 2019 11:04:05 +0000 X-Inumbo-ID: abded814-eb4d-11e9-9800-12813bfff9fa Received: from esa3.hc3370-68.iphmx.com (unknown [216.71.145.155]) by us1-amaz-eas2.inumbo.com (Halon) with ESMTPS id abded814-eb4d-11e9-9800-12813bfff9fa; Thu, 10 Oct 2019 11:04:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=citrix.com; s=securemail; t=1570705444; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=VJ+eNzJg3w6G9LgrbYkzOMKvORC0WkpX/C+Ntg/gB28=; b=R0Pf56u45rMIebFgvYc3d80HdPWEdMq660rfxagtuXfblNdqMotEVVOd H95RQNOKKqHAwwVWPhd5O1VxR7sQ4+DYL0H5jjSQQKWEPrQQX+VPJhBh9 l1OsSeewrgogcBQHG4WJkSPZbRZwxQwxI00PHkp4OvhruQZWB4aP2yASx M=; Authentication-Results: esa3.hc3370-68.iphmx.com; dkim=none (message not signed) header.i=none; spf=None smtp.pra=roger.pau@citrix.com; spf=Pass smtp.mailfrom=roger.pau@citrix.com; spf=None smtp.helo=postmaster@mail.citrix.com Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of roger.pau@citrix.com) identity=pra; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible Received-SPF: Pass (esa3.hc3370-68.iphmx.com: domain of roger.pau@citrix.com designates 162.221.158.21 as permitted sender) identity=mailfrom; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="roger.pau@citrix.com"; x-conformance=sidf_compatible; x-record-type="v=spf1"; x-record-text="v=spf1 ip4:209.167.231.154 ip4:178.63.86.133 ip4:195.66.111.40/30 ip4:85.115.9.32/28 ip4:199.102.83.4 ip4:192.28.146.160 ip4:192.28.146.107 ip4:216.52.6.88 ip4:216.52.6.188 ip4:162.221.158.21 ip4:162.221.156.83 ip4:168.245.78.127 ~all" Received-SPF: None (esa3.hc3370-68.iphmx.com: no sender authenticity information available from domain of postmaster@mail.citrix.com) identity=helo; client-ip=162.221.158.21; receiver=esa3.hc3370-68.iphmx.com; envelope-from="roger.pau@citrix.com"; x-sender="postmaster@mail.citrix.com"; x-conformance=sidf_compatible IronPort-SDR: mF1CA1AWAbHUHk4MlTKuqzGUAsELROpk4KzZ60p5x0ipNdshNZz76ayx8ydzkiCF6Oe9Y5oNBu TYIjjbC0vgd+6N0a2sMyN79n+Y2uxSbYGZ4W+uns/c5af7U5nNMAfOG1LCaNYgwK3E7TJrRKbO c9pG4EAwzbDCnYtgEUTCQnie6Gk4e96vp6H4g+79AviPmzgiH/1Rmw28R9AOYuT3RSIDelI6q3 rE9Bpgls6selSdFKYQDk9+DgjUewe6wyyoUvJWFAweSI6SPziWwATJ8PteJiIJoW6VWddO4hC+ S9U= X-SBRS: 2.7 X-MesageID: 6708681 X-Ironport-Server: esa3.hc3370-68.iphmx.com X-Remote-IP: 162.221.158.21 X-Policy: $RELAYED X-IronPort-AV: E=Sophos;i="5.67,279,1566878400"; d="scan'208";a="6708681" From: Roger Pau Monne To: Date: Thu, 10 Oct 2019 13:03:39 +0200 Message-ID: <20191010110339.6447-3-roger.pau@citrix.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20191010110339.6447-1-roger.pau@citrix.com> References: <20191010110339.6447-1-roger.pau@citrix.com> MIME-Version: 1.0 Subject: [Xen-devel] [PATCH 2/2] iommu: translate IO-APIC pins when enabling interrupt remapping X-BeenThere: xen-devel@lists.xenproject.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Cc: Juergen Gross , Suravee Suthikulpanit , Wei Liu , Andrew Cooper , Jan Beulich , Roger Pau Monne Errors-To: xen-devel-bounces@lists.xenproject.org Sender: "Xen-devel" On Intel hardware there's currently no translation of already enabled IO-APIC pins when interrupt remapping is enabled on the IOMMU, hence introduce a logic similar to the one used in x2apic_bsp_setup in order to save and mask all IO-APIC pins, and then translate and restore them after interrupt remapping has been enabled. With this change the AMD specific logic to deal with enabled pins (amd_iommu_setup_ioapic_remapping) can be removed, thus unifying the handling of IO-APIC when enabling interrupt remapping regardless of the IOMMU vendor. Reported-by: Andrew Cooper Signed-off-by: Roger Pau Monné --- Cc: Juergen Gross --- xen/drivers/passthrough/amd/iommu_init.c | 11 ++- xen/drivers/passthrough/amd/iommu_intr.c | 90 +------------------ xen/drivers/passthrough/x86/iommu.c | 34 ++++++- xen/include/asm-x86/hvm/svm/amd-iommu-proto.h | 1 + 4 files changed, 40 insertions(+), 96 deletions(-) diff --git a/xen/drivers/passthrough/amd/iommu_init.c b/xen/drivers/passthrough/amd/iommu_init.c index 6f53c7ec08..3c244619b9 100644 --- a/xen/drivers/passthrough/amd/iommu_init.c +++ b/xen/drivers/passthrough/amd/iommu_init.c @@ -19,6 +19,7 @@ #include #include +#include #include #include #include @@ -1435,12 +1436,6 @@ int __init amd_iommu_init(bool xt) if ( rc ) goto error_out; - /* initialize io-apic interrupt remapping entries */ - if ( iommu_intremap ) - rc = amd_iommu_setup_ioapic_remapping(); - if ( rc ) - goto error_out; - /* Allocate and initialize device table(s). */ pci_init = !xt; rc = iterate_ivrs_mappings(amd_iommu_setup_device_table); @@ -1469,6 +1464,10 @@ int __init amd_iommu_init(bool xt) goto error_out; } + if ( iommu_intremap ) + register_keyhandler('V', &amd_iommu_dump_intremap_tables, + "dump IOMMU intremap tables", 0); + return 0; error_out: diff --git a/xen/drivers/passthrough/amd/iommu_intr.c b/xen/drivers/passthrough/amd/iommu_intr.c index fb71073c84..1eed60f265 100644 --- a/xen/drivers/passthrough/amd/iommu_intr.c +++ b/xen/drivers/passthrough/amd/iommu_intr.c @@ -21,7 +21,6 @@ #include #include #include -#include #include union irte32 { @@ -79,8 +78,6 @@ unsigned long *shared_intremap_inuse; static DEFINE_SPINLOCK(shared_intremap_lock); unsigned int nr_ioapic_sbdf; -static void dump_intremap_tables(unsigned char key); - #define intremap_page_order(irt) PFN_ORDER(virt_to_page(irt)) unsigned int amd_iommu_intremap_table_order( @@ -354,91 +351,6 @@ static int update_intremap_entry_from_ioapic( return 0; } -int __init amd_iommu_setup_ioapic_remapping(void) -{ - struct IO_APIC_route_entry rte; - unsigned long flags; - union irte_ptr entry; - int apic, pin; - u8 delivery_mode, dest, vector, dest_mode; - u16 seg, bdf, req_id; - struct amd_iommu *iommu; - spinlock_t *lock; - unsigned int offset; - - /* Read ioapic entries and update interrupt remapping table accordingly */ - for ( apic = 0; apic < nr_ioapics; apic++ ) - { - for ( pin = 0; pin < nr_ioapic_entries[apic]; pin++ ) - { - unsigned int idx; - - rte = __ioapic_read_entry(apic, pin, 1); - if ( rte.mask == 1 ) - continue; - - /* get device id of ioapic devices */ - idx = ioapic_id_to_index(IO_APIC_ID(apic)); - if ( idx == MAX_IO_APICS ) - return -EINVAL; - - bdf = ioapic_sbdf[idx].bdf; - seg = ioapic_sbdf[idx].seg; - iommu = find_iommu_for_device(seg, bdf); - if ( !iommu ) - { - AMD_IOMMU_DEBUG("Fail to find iommu for ioapic " - "device id = %04x:%04x\n", seg, bdf); - continue; - } - - req_id = get_intremap_requestor_id(iommu->seg, bdf); - lock = get_intremap_lock(iommu->seg, req_id); - - delivery_mode = rte.delivery_mode; - vector = rte.vector; - dest_mode = rte.dest_mode; - dest = rte.dest.logical.logical_dest; - - if ( iommu->ctrl.xt_en ) - { - /* - * In x2APIC mode we have no way of discovering the high 24 - * bits of the destination of an already enabled interrupt. - * We come here earlier than for xAPIC mode, so no interrupts - * should have been set up before. - */ - AMD_IOMMU_DEBUG("Unmasked IO-APIC#%u entry %u in x2APIC mode\n", - IO_APIC_ID(apic), pin); - } - - spin_lock_irqsave(lock, flags); - offset = alloc_intremap_entry(iommu, req_id, 1); - BUG_ON(offset >= INTREMAP_MAX_ENTRIES); - entry = get_intremap_entry(iommu, req_id, offset); - update_intremap_entry(iommu, entry, vector, - delivery_mode, dest_mode, dest); - spin_unlock_irqrestore(lock, flags); - - set_rte_index(&rte, offset); - ioapic_sbdf[idx].pin_2_idx[pin] = offset; - __ioapic_write_entry(apic, pin, 1, rte); - - if ( iommu->enabled ) - { - spin_lock_irqsave(&iommu->lock, flags); - amd_iommu_flush_intremap(iommu, req_id); - spin_unlock_irqrestore(&iommu->lock, flags); - } - } - } - - register_keyhandler('V', &dump_intremap_tables, - "dump IOMMU intremap tables", 0); - - return 0; -} - void amd_iommu_ioapic_update_ire( unsigned int apic, unsigned int reg, unsigned int value) { @@ -982,7 +894,7 @@ static int dump_intremap_mapping(const struct amd_iommu *iommu, return 0; } -static void dump_intremap_tables(unsigned char key) +void amd_iommu_dump_intremap_tables(unsigned char key) { if ( !shared_intremap_table ) { diff --git a/xen/drivers/passthrough/x86/iommu.c b/xen/drivers/passthrough/x86/iommu.c index 59905629e1..2cf528e760 100644 --- a/xen/drivers/passthrough/x86/iommu.c +++ b/xen/drivers/passthrough/x86/iommu.c @@ -21,6 +21,7 @@ #include #include +#include #include const struct iommu_init_ops *__initdata iommu_init_ops; @@ -28,6 +29,7 @@ struct iommu_ops __read_mostly iommu_ops; int __init iommu_hardware_setup(void) { + struct IO_APIC_route_entry **ioapic_entries = NULL; int rc; if ( !iommu_init_ops ) @@ -43,7 +45,37 @@ int __init iommu_hardware_setup(void) /* x2apic setup may have previously initialised the struct. */ ASSERT(iommu_ops.init == iommu_init_ops->ops->init); - return iommu_init_ops->setup(); + if ( !x2apic_enabled && iommu_intremap ) + { + /* + * If x2APIC is enabled interrupt remapping is already enabled, so + * there's no need to mess with the IO-APIC because the remapping + * entries are already correctly setup by x2apic_bsp_setup. + */ + ioapic_entries = alloc_ioapic_entries(); + if ( !ioapic_entries ) + return -ENOMEM; + rc = save_IO_APIC_setup(ioapic_entries); + if ( rc ) + { + free_ioapic_entries(ioapic_entries); + return rc; + } + + mask_8259A(); + mask_IO_APIC_setup(ioapic_entries); + } + + rc = iommu_init_ops->setup(); + + if ( ioapic_entries ) + { + restore_IO_APIC_setup(ioapic_entries, rc); + unmask_8259A(); + free_ioapic_entries(ioapic_entries); + } + + return rc; } int iommu_enable_x2apic(void) diff --git a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h index 07d25a585d..8ed9482791 100644 --- a/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h +++ b/xen/include/asm-x86/hvm/svm/amd-iommu-proto.h @@ -114,6 +114,7 @@ int amd_iommu_msi_msg_update_ire( void amd_iommu_read_msi_from_ire( struct msi_desc *msi_desc, struct msi_msg *msg); int amd_setup_hpet_msi(struct msi_desc *msi_desc); +void amd_iommu_dump_intremap_tables(unsigned char key); extern struct ioapic_sbdf { u16 bdf, seg;