From patchwork Wed Jan 11 09:37:36 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cornelia Huck X-Patchwork-Id: 9509707 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6F22B60231 for ; Wed, 11 Jan 2017 09:50:23 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6C36225EF7 for ; Wed, 11 Jan 2017 09:50:23 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 60BF528584; Wed, 11 Jan 2017 09:50:23 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B94ED25EF7 for ; Wed, 11 Jan 2017 09:50:22 +0000 (UTC) Received: from localhost ([::1]:52705 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cRFXx-0003iF-Ru for patchwork-qemu-devel@patchwork.kernel.org; Wed, 11 Jan 2017 04:50:21 -0500 Received: from eggs.gnu.org ([2001:4830:134:3::10]:45078) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1cRFM0-0000cR-Cv for qemu-devel@nongnu.org; Wed, 11 Jan 2017 04:38:01 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1cRFLw-0000k9-39 for qemu-devel@nongnu.org; Wed, 11 Jan 2017 04:38:00 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:34663 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1cRFLv-0000k5-Uq for qemu-devel@nongnu.org; Wed, 11 Jan 2017 04:37:56 -0500 Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.17/8.16.0.17) with SMTP id v0B9Xlvw090104 for ; Wed, 11 Jan 2017 04:37:55 -0500 Received: from e06smtp12.uk.ibm.com (e06smtp12.uk.ibm.com [195.75.94.108]) by mx0b-001b2d01.pphosted.com with ESMTP id 27w2xmchnm-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 11 Jan 2017 04:37:55 -0500 Received: from localhost by e06smtp12.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 11 Jan 2017 09:37:53 -0000 Received: from d06dlp03.portsmouth.uk.ibm.com (9.149.20.15) by e06smtp12.uk.ibm.com (192.168.101.142) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; Wed, 11 Jan 2017 09:37:52 -0000 Received: from b06cxnps3074.portsmouth.uk.ibm.com (d06relay09.portsmouth.uk.ibm.com [9.149.109.194]) by d06dlp03.portsmouth.uk.ibm.com (Postfix) with ESMTP id 297F31B08069; Wed, 11 Jan 2017 09:40:33 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06cxnps3074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v0B9bpFb13107632; Wed, 11 Jan 2017 09:37:51 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8115842052; Wed, 11 Jan 2017 08:35:47 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 559FC4204F; Wed, 11 Jan 2017 08:35:47 +0000 (GMT) Received: from gondolin.boeblingen.de.ibm.com (unknown [9.152.224.55]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTPS; Wed, 11 Jan 2017 08:35:47 +0000 (GMT) From: Cornelia Huck To: qemu-devel@nongnu.org Date: Wed, 11 Jan 2017 10:37:36 +0100 X-Mailer: git-send-email 2.11.0 In-Reply-To: <20170111093742.21946-1-cornelia.huck@de.ibm.com> References: <20170111093742.21946-1-cornelia.huck@de.ibm.com> X-TM-AS-GCONF: 00 X-Content-Scanned: Fidelis XPS MAILER x-cbid: 17011109-0008-0000-0000-000003CE3915 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 17011109-0009-0000-0000-00001C0E017E Message-Id: <20170111093742.21946-6-cornelia.huck@de.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:, , definitions=2017-01-11_08:, , signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 spamscore=0 suspectscore=3 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1612050000 definitions=main-1701110135 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [generic] [fuzzy] X-Received-From: 148.163.158.5 Subject: [Qemu-devel] [PATCH 05/11] s390x/pci: dynamically allocate iommu X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Cornelia Huck , borntraeger@de.ibm.com, jfrei@linux.vnet.ibm.com, Yi Min Zhao , agraf@suse.de Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: Yi Min Zhao When initializing a PCI device, an address space is required during PCI core initialization and before the call to the embedding object hotplug callback. To provide this AS, we allocate a S390PCIIOMMU object containing this AS. Initialization of S390PCIIOMMU object is done before the PCI device is completely created. So that we cannot associate the IOMMU with the device at the moment. To track the IOMMU object, we use g_hash functions with the PCI device's bus address as a key to provide an array of pointers indexed by the PCI device's devfn to the allocated IOMMU. Signed-off-by: Yi Min Zhao Reviewed-by: Pierre Morel Signed-off-by: Cornelia Huck --- hw/s390x/s390-pci-bus.c | 73 +++++++++++++++++++++++++++++++++++++++---------- hw/s390x/s390-pci-bus.h | 7 ++++- 2 files changed, 65 insertions(+), 15 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 11244fc93c..04a4c3508e 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -432,11 +432,48 @@ static const MemoryRegionIOMMUOps s390_iommu_ops = { .translate = s390_translate_iommu, }; +static S390PCIIOMMU *s390_pci_get_iommu(S390pciState *s, PCIBus *bus, + int devfn) +{ + uint64_t key = (unsigned long)bus; + S390PCIIOMMUTable *table = g_hash_table_lookup(s->iommu_table, &key); + S390PCIIOMMU *iommu; + + if (!table) { + table = g_malloc0(sizeof(S390PCIIOMMUTable)); + table->key = key; + g_hash_table_insert(s->iommu_table, &table->key, table); + } + + iommu = table->iommu[PCI_SLOT(devfn)]; + if (!iommu) { + iommu = S390_PCI_IOMMU(object_new(TYPE_S390_PCI_IOMMU)); + + char *mr_name = g_strdup_printf("iommu-root-%02x:%02x.%01x", + pci_bus_num(bus), + PCI_SLOT(devfn), + PCI_FUNC(devfn)); + char *as_name = g_strdup_printf("iommu-pci-%02x:%02x.%01x", + pci_bus_num(bus), + PCI_SLOT(devfn), + PCI_FUNC(devfn)); + memory_region_init(&iommu->mr, OBJECT(iommu), mr_name, UINT64_MAX); + address_space_init(&iommu->as, &iommu->mr, as_name); + table->iommu[PCI_SLOT(devfn)] = iommu; + + g_free(mr_name); + g_free(as_name); + } + + return iommu; +} + static AddressSpace *s390_pci_dma_iommu(PCIBus *bus, void *opaque, int devfn) { S390pciState *s = opaque; + S390PCIIOMMU *iommu = s390_pci_get_iommu(s, bus, devfn); - return &s->iommu[PCI_SLOT(devfn)]->as; + return &iommu->as; } static uint8_t set_ind_atomic(uint64_t ind_loc, uint8_t to_be_set) @@ -520,19 +557,22 @@ void s390_pci_iommu_disable(S390PCIIOMMU *iommu) object_unparent(OBJECT(&iommu->iommu_mr)); } -static void s390_pcihost_init_as(S390pciState *s) +static void s390_pci_iommu_free(PCIBus *bus, int32_t devfn) { - int i; - S390PCIIOMMU *iommu; - - for (i = 0; i < PCI_SLOT_MAX; i++) { - iommu = g_malloc0(sizeof(S390PCIIOMMU)); - memory_region_init(&iommu->mr, OBJECT(s), - "iommu-root-s390", UINT64_MAX); - address_space_init(&iommu->as, &iommu->mr, "iommu-pci"); + uint64_t key = (unsigned long)bus; + S390pciState *s = s390_get_phb(); + S390PCIIOMMUTable *table = g_hash_table_lookup(s->iommu_table, &key); + S390PCIIOMMU *iommu = table ? table->iommu[PCI_SLOT(devfn)] : NULL; - s->iommu[i] = iommu; + if (!table || !iommu) { + return; } + + table->iommu[PCI_SLOT(devfn)] = NULL; + address_space_destroy(&iommu->as); + object_unparent(OBJECT(&iommu->mr)); + object_unparent(OBJECT(iommu)); + object_unref(OBJECT(iommu)); } static int s390_pcihost_init(SysBusDevice *dev) @@ -548,7 +588,6 @@ static int s390_pcihost_init(SysBusDevice *dev) s390_pci_set_irq, s390_pci_map_irq, NULL, get_system_memory(), get_system_io(), 0, 64, TYPE_PCI_BUS); - s390_pcihost_init_as(s); pci_setup_iommu(b, s390_pci_dma_iommu, s); bus = BUS(b); @@ -558,6 +597,8 @@ static int s390_pcihost_init(SysBusDevice *dev) s->bus = S390_PCI_BUS(qbus_create(TYPE_S390_PCI_BUS, DEVICE(s), NULL)); qbus_set_hotplug_handler(BUS(s->bus), DEVICE(s), NULL); + s->iommu_table = g_hash_table_new_full(g_int64_hash, g_int64_equal, + NULL, g_free); QTAILQ_INIT(&s->pending_sei); return 0; } @@ -661,7 +702,7 @@ static void s390_pcihost_hot_plug(HotplugHandler *hotplug_dev, } pbdev->pdev = pdev; - pbdev->iommu = s->iommu[PCI_SLOT(pdev->devfn)]; + pbdev->iommu = s390_pci_get_iommu(s, pdev->bus, pdev->devfn); pbdev->iommu->pbdev = pbdev; pbdev->state = ZPCI_FS_STANDBY; @@ -708,8 +749,9 @@ static void s390_pcihost_timer_cb(void *opaque) static void s390_pcihost_hot_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, Error **errp) { - int i; PCIDevice *pci_dev = NULL; + PCIBus *bus; + int32_t devfn, i; S390PCIBusDevice *pbdev = NULL; S390pciState *s = s390_get_phb(); @@ -752,8 +794,11 @@ static void s390_pcihost_hot_unplug(HotplugHandler *hotplug_dev, s390_pci_generate_plug_event(HP_EVENT_STANDBY_TO_RESERVED, pbdev->fh, pbdev->fid); + bus = pci_dev->bus; + devfn = pci_dev->devfn; object_unparent(OBJECT(pci_dev)); s390_pci_msix_free(pbdev); + s390_pci_iommu_free(bus, devfn); pbdev->pdev = NULL; pbdev->state = ZPCI_FS_RESERVED; out: diff --git a/hw/s390x/s390-pci-bus.h b/hw/s390x/s390-pci-bus.h index 1d05c1a92e..fe108e9330 100644 --- a/hw/s390x/s390-pci-bus.h +++ b/hw/s390x/s390-pci-bus.h @@ -274,6 +274,11 @@ typedef struct S390PCIIOMMU { uint64_t pal; } S390PCIIOMMU; +typedef struct S390PCIIOMMUTable { + uint64_t key; + S390PCIIOMMU *iommu[PCI_SLOT_MAX]; +} S390PCIIOMMUTable; + typedef struct S390PCIBusDevice { DeviceState qdev; PCIDevice *pdev; @@ -304,7 +309,7 @@ typedef struct S390pciState { PCIHostState parent_obj; S390PCIBus *bus; S390PCIBusDevice *pbdev[PCI_SLOT_MAX]; - S390PCIIOMMU *iommu[PCI_SLOT_MAX]; + GHashTable *iommu_table; QTAILQ_HEAD(, SeiContainer) pending_sei; } S390pciState;