From patchwork Mon Sep 14 22:29:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 11775071 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7A26C6CA for ; Mon, 14 Sep 2020 22:30:43 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 119FC2065E for ; Mon, 14 Sep 2020 22:30:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="GENecKyI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 119FC2065E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:58214 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kHwzh-0004X9-Bg for patchwork-qemu-devel@patchwork.kernel.org; Mon, 14 Sep 2020 18:30:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56976) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHwyl-0003ci-5u; Mon, 14 Sep 2020 18:29:43 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:16700 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHwyj-0002rW-D1; Mon, 14 Sep 2020 18:29:42 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08EMBJqv134789; Mon, 14 Sep 2020 18:29:39 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=hrQKq18NXIKk2mOHywBIyALzLygSHnOXkBOOGDfQ2w8=; b=GENecKyI/Wr2gcSZihdavkbyLLxOWptcAqwKbOKlVdsHsO+0FsembHXjujrKiOJCmFkn nSmhR+DPvvtV280DcWg2iw8giervlk4O/QkRSnwW27U3Q+b46lC6awI3cwXlpjHzCioM zF4aBZqRDZsKcgKEQXvBtZwTXAJxowsAsU7pnGWHaiF6X1t5wYFUGiIQ8nCUtOBL8Y9X louaJIaql1JpwqNZjQpWYCFZ9EuiftW04TV9KlVKLv3JuvTfYjwaqWj7IWOjwu7UE7o9 1UUK3zKAdE/GfCXawRlNYS2KKza304H52OV/XA3X05wM9CwlHIVN7QFfs2D3elTz+Sc0 yQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 33jh3fga14-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 18:29:39 -0400 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08EMCLPV137347; Mon, 14 Sep 2020 18:29:38 -0400 Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0b-001b2d01.pphosted.com with ESMTP id 33jh3fga10-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 18:29:38 -0400 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08EMRl6S006102; Mon, 14 Sep 2020 22:29:38 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma04wdc.us.ibm.com with ESMTP id 33jgrgr6hn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 22:29:38 +0000 Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08EMTada000574 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Sep 2020 22:29:36 GMT Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id BD88BBE04F; Mon, 14 Sep 2020 22:29:36 +0000 (GMT) Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 95A4FBE051; Mon, 14 Sep 2020 22:29:35 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.163.85.51]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 14 Sep 2020 22:29:35 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com Subject: [PATCH v2 1/3] vfio: Find DMA available capability Date: Mon, 14 Sep 2020 18:29:28 -0400 Message-Id: <1600122570-12941-2-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1600122570-12941-1-git-send-email-mjrosato@linux.ibm.com> References: <1600122570-12941-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-14_09:2020-09-14, 2020-09-14 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxlogscore=833 clxscore=1015 impostorscore=0 priorityscore=1501 lowpriorityscore=0 mlxscore=0 suspectscore=0 spamscore=0 bulkscore=0 phishscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009140166 Received-SPF: pass client-ip=148.163.158.5; envelope-from=mjrosato@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/14 18:29:38 X-ACL-Warn: Detected OS = Linux 3.x [generic] X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: thuth@redhat.com, pmorel@linux.ibm.com, david@redhat.com, schnelle@linux.ibm.com, qemu-devel@nongnu.org, pasic@linux.ibm.com, borntraeger@de.ibm.com, qemu-s390x@nongnu.org, rth@twiddle.net Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" The underlying host may be limiting the number of outstanding DMA requests for type 1 IOMMU. Add helper functions to check for the DMA available capability and retrieve the current number of DMA mappings allowed. Signed-off-by: Matthew Rosato --- hw/vfio/common.c | 37 +++++++++++++++++++++++++++++++++++++ include/hw/vfio/vfio-common.h | 2 ++ 2 files changed, 39 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 3335714..7f4a338 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -844,6 +844,43 @@ vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id) return NULL; } +static struct vfio_info_cap_header * +vfio_get_iommu_type1_info_cap(struct vfio_iommu_type1_info *info, uint16_t id) +{ + struct vfio_info_cap_header *hdr; + void *ptr = info; + + if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { + return NULL; + } + + for (hdr = ptr + info->cap_offset; hdr != ptr; hdr = ptr + hdr->next) { + if (hdr->id == id) { + return hdr; + } + } + + return NULL; +} + +bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, + unsigned int *avail) +{ + struct vfio_info_cap_header *hdr; + struct vfio_iommu_type1_info_dma_avail *cap; + + /* If the capability cannot be found, assume no DMA limiting */ + hdr = vfio_get_iommu_type1_info_cap(info, + VFIO_IOMMU_TYPE1_INFO_DMA_AVAIL); + if (hdr == NULL || avail == NULL) { + return false; + } + + cap = (void *) hdr; + *avail = cap->avail; + return true; +} + static int vfio_setup_region_sparse_mmaps(VFIORegion *region, struct vfio_region_info *info) { diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index c78f3ff..661a380 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -178,6 +178,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp); void vfio_put_group(VFIOGroup *group); int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp); +bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, + unsigned int *avail); extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; From patchwork Mon Sep 14 22:29:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 11775073 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B6AF292C for ; Mon, 14 Sep 2020 22:31:01 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D01A1208DB for ; Mon, 14 Sep 2020 22:31:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="gW1BH+6W" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D01A1208DB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:58446 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kHwzz-0004fd-Gr for patchwork-qemu-devel@patchwork.kernel.org; Mon, 14 Sep 2020 18:30:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57018) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHwyo-0003fS-9A; Mon, 14 Sep 2020 18:29:46 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:25256) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHwyl-0002ri-6X; Mon, 14 Sep 2020 18:29:45 -0400 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08EM1Zsl032110; Mon, 14 Sep 2020 18:29:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=jpOYTdtLU3gE6lh5GcVNIu82TuH5cKkBPjxPH9WNNDA=; b=gW1BH+6WG1gMa7KHIRKQIXbZ2nyOBrpBgxs3W2jVih1hXP2JCB4jBYaj1LpMiKwPRs4H qncdroL5fXXbHBvzN8vAlhxudP6wK1a2pR9BCRuCryH0NlU95LkgVdfFnEx+EAZKiVny ++9ZW/q2H/mrgRYUpa3O54KTqcmRrxZUFKdYAU+ont9euIbMQQHTuMqvTlL1OhE00MGT p0VZWr5fa6VYs7ESlq6faORazqfw/K3Nb4CaB61JtL3XVWmzUu2EdV049z/UHIDtYQK7 alkogifaLXFkA9BaXclQ9EXiEo++GPpWhQ/2PLweLUQjj4zDT79pLsBI25/1XtqG5+Ws 3g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 33jb7ba4nb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 18:29:41 -0400 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08EMM1BT120195; Mon, 14 Sep 2020 18:29:40 -0400 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0a-001b2d01.pphosted.com with ESMTP id 33jb7ba4n3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 18:29:40 -0400 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08EMRfv0023230; Mon, 14 Sep 2020 22:29:39 GMT Received: from b03cxnp08026.gho.boulder.ibm.com (b03cxnp08026.gho.boulder.ibm.com [9.17.130.18]) by ppma05wdc.us.ibm.com with ESMTP id 33gny8u009-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 22:29:39 +0000 Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp08026.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08EMTXQi1769990 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Sep 2020 22:29:33 GMT Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 52671BE04F; Mon, 14 Sep 2020 22:29:38 +0000 (GMT) Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E0A7FBE051; Mon, 14 Sep 2020 22:29:36 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.163.85.51]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 14 Sep 2020 22:29:36 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com Subject: [PATCH v2 2/3] s390x/pci: Honor DMA limits set by vfio Date: Mon, 14 Sep 2020 18:29:29 -0400 Message-Id: <1600122570-12941-3-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1600122570-12941-1-git-send-email-mjrosato@linux.ibm.com> References: <1600122570-12941-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-14_09:2020-09-14, 2020-09-14 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=2 bulkscore=0 lowpriorityscore=0 phishscore=0 impostorscore=0 mlxscore=0 adultscore=0 malwarescore=0 priorityscore=1501 spamscore=0 mlxlogscore=999 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009140166 Received-SPF: pass client-ip=148.163.156.1; envelope-from=mjrosato@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/14 18:29:41 X-ACL-Warn: Detected OS = Linux 3.x [generic] [fuzzy] X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: thuth@redhat.com, pmorel@linux.ibm.com, david@redhat.com, schnelle@linux.ibm.com, qemu-devel@nongnu.org, pasic@linux.ibm.com, borntraeger@de.ibm.com, qemu-s390x@nongnu.org, rth@twiddle.net Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" When an s390 guest is using lazy unmapping, it can result in a very large number of oustanding DMA requests, far beyond the default limit configured for vfio. Let's track DMA usage similar to vfio in the host, and trigger the guest to flush their DMA mappings before vfio runs out. Signed-off-by: Matthew Rosato --- hw/s390x/s390-pci-bus.c | 99 +++++++++++++++++++++++++++++++++++++++++++++--- hw/s390x/s390-pci-bus.h | 9 +++++ hw/s390x/s390-pci-inst.c | 29 +++++++++++--- hw/s390x/s390-pci-inst.h | 3 ++ 4 files changed, 129 insertions(+), 11 deletions(-) diff --git a/hw/s390x/s390-pci-bus.c b/hw/s390x/s390-pci-bus.c index 92146a2..23474cd 100644 --- a/hw/s390x/s390-pci-bus.c +++ b/hw/s390x/s390-pci-bus.c @@ -11,6 +11,8 @@ * directory. */ +#include + #include "qemu/osdep.h" #include "qapi/error.h" #include "qapi/visitor.h" @@ -24,6 +26,9 @@ #include "qemu/error-report.h" #include "qemu/module.h" +#include "hw/vfio/pci.h" +#include "hw/vfio/vfio-common.h" + #ifndef DEBUG_S390PCI_BUS #define DEBUG_S390PCI_BUS 0 #endif @@ -737,6 +742,82 @@ static void s390_pci_iommu_free(S390pciState *s, PCIBus *bus, int32_t devfn) object_unref(OBJECT(iommu)); } +static bool s390_sync_dma_avail(int fd, unsigned int *avail) +{ + struct vfio_iommu_type1_info *info; + uint32_t argsz; + bool rval = false; + int ret; + + if (avail == NULL) { + return false; + } + + argsz = sizeof(struct vfio_iommu_type1_info); + info = g_malloc0(argsz); + info->argsz = argsz; + /* + * If the specified argsz is not large enough to contain all + * capabilities it will be updated upon return. In this case + * use the updated value to get the entire capability chain. + */ + ret = ioctl(fd, VFIO_IOMMU_GET_INFO, info); + if (argsz != info->argsz) { + argsz = info->argsz; + info = g_realloc(info, argsz); + info->argsz = argsz; + ret = ioctl(fd, VFIO_IOMMU_GET_INFO, info); + } + + if (ret) { + goto out; + } + + /* If the capability exists, update with the current value */ + rval = vfio_get_info_dma_avail(info, avail); + +out: + g_free(info); + return rval; +} + +static S390PCIDMACount *s390_start_dma_count(S390pciState *s, VFIODevice *vdev) +{ + int id = vdev->group->container->fd; + S390PCIDMACount *cnt; + uint32_t avail; + + if (!s390_sync_dma_avail(id, &avail)) { + return NULL; + } + + QTAILQ_FOREACH(cnt, &s->zpci_dma_limit, link) { + if (cnt->id == id) { + cnt->users++; + return cnt; + } + } + + cnt = g_new0(S390PCIDMACount, 1); + cnt->id = id; + cnt->users = 1; + cnt->avail = avail; + QTAILQ_INSERT_TAIL(&s->zpci_dma_limit, cnt, link); + return cnt; +} + +static void s390_end_dma_count(S390pciState *s, S390PCIDMACount *cnt) +{ + if (cnt == NULL) { + return; + } + + cnt->users--; + if (cnt->users == 0) { + QTAILQ_REMOVE(&s->zpci_dma_limit, cnt, link); + } +} + static void s390_pcihost_realize(DeviceState *dev, Error **errp) { PCIBus *b; @@ -764,6 +845,7 @@ static void s390_pcihost_realize(DeviceState *dev, Error **errp) s->bus_no = 0; QTAILQ_INIT(&s->pending_sei); QTAILQ_INIT(&s->zpci_devs); + QTAILQ_INIT(&s->zpci_dma_limit); css_register_io_adapters(CSS_IO_ADAPTER_PCI, true, false, S390_ADAPTER_SUPPRESSIBLE, errp); @@ -902,6 +984,7 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, { S390pciState *s = S390_PCI_HOST_BRIDGE(hotplug_dev); PCIDevice *pdev = NULL; + VFIOPCIDevice *vpdev = NULL; S390PCIBusDevice *pbdev = NULL; if (object_dynamic_cast(OBJECT(dev), TYPE_PCI_BRIDGE)) { @@ -941,17 +1024,20 @@ static void s390_pcihost_plug(HotplugHandler *hotplug_dev, DeviceState *dev, } } + pbdev->pdev = pdev; + pbdev->iommu = s390_pci_get_iommu(s, pci_get_bus(pdev), pdev->devfn); + pbdev->iommu->pbdev = pbdev; + pbdev->state = ZPCI_FS_DISABLED; + if (object_dynamic_cast(OBJECT(dev), "vfio-pci")) { pbdev->fh |= FH_SHM_VFIO; + vpdev = container_of(pbdev->pdev, VFIOPCIDevice, pdev); + pbdev->iommu->dma_limit = s390_start_dma_count(s, + &vpdev->vbasedev); } else { pbdev->fh |= FH_SHM_EMUL; } - pbdev->pdev = pdev; - pbdev->iommu = s390_pci_get_iommu(s, pci_get_bus(pdev), pdev->devfn); - pbdev->iommu->pbdev = pbdev; - pbdev->state = ZPCI_FS_DISABLED; - if (s390_pci_msix_init(pbdev)) { error_setg(errp, "MSI-X support is mandatory " "in the S390 architecture"); @@ -1004,6 +1090,9 @@ static void s390_pcihost_unplug(HotplugHandler *hotplug_dev, DeviceState *dev, pbdev->fid = 0; QTAILQ_REMOVE(&s->zpci_devs, pbdev, link); g_hash_table_remove(s->zpci_table, &pbdev->idx); + if (pbdev->iommu->dma_limit) { + s390_end_dma_count(s, pbdev->iommu->dma_limit); + } qdev_unrealize(dev); } } diff --git a/hw/s390x/s390-pci-bus.h b/hw/s390x/s390-pci-bus.h index 0458059..f166fd9 100644 --- a/hw/s390x/s390-pci-bus.h +++ b/hw/s390x/s390-pci-bus.h @@ -270,6 +270,13 @@ typedef struct S390IOTLBEntry { uint64_t perm; } S390IOTLBEntry; +typedef struct S390PCIDMACount { + int id; + int users; + uint32_t avail; + QTAILQ_ENTRY(S390PCIDMACount) link; +} S390PCIDMACount; + struct S390PCIIOMMU { Object parent_obj; S390PCIBusDevice *pbdev; @@ -281,6 +288,7 @@ struct S390PCIIOMMU { uint64_t pba; uint64_t pal; GHashTable *iotlb; + S390PCIDMACount *dma_limit; }; typedef struct S390PCIIOMMUTable { @@ -356,6 +364,7 @@ struct S390pciState { GHashTable *zpci_table; QTAILQ_HEAD(, SeiContainer) pending_sei; QTAILQ_HEAD(, S390PCIBusDevice) zpci_devs; + QTAILQ_HEAD(, S390PCIDMACount) zpci_dma_limit; }; S390pciState *s390_get_phb(void); diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 2f7a7d7..6af9af4 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -32,6 +32,9 @@ } \ } while (0) +#define INC_DMA_AVAIL(iommu) if (iommu->dma_limit) iommu->dma_limit->avail++; +#define DEC_DMA_AVAIL(iommu) if (iommu->dma_limit) iommu->dma_limit->avail--; + static void s390_set_status_code(CPUS390XState *env, uint8_t r, uint64_t status_code) { @@ -572,7 +575,8 @@ int pcistg_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) return 0; } -static void s390_pci_update_iotlb(S390PCIIOMMU *iommu, S390IOTLBEntry *entry) +static uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, + S390IOTLBEntry *entry) { S390IOTLBEntry *cache = g_hash_table_lookup(iommu->iotlb, &entry->iova); IOMMUTLBEntry notify = { @@ -585,14 +589,15 @@ static void s390_pci_update_iotlb(S390PCIIOMMU *iommu, S390IOTLBEntry *entry) if (entry->perm == IOMMU_NONE) { if (!cache) { - return; + goto out; } g_hash_table_remove(iommu->iotlb, &entry->iova); + INC_DMA_AVAIL(iommu); } else { if (cache) { if (cache->perm == entry->perm && cache->translated_addr == entry->translated_addr) { - return; + goto out; } notify.perm = IOMMU_NONE; @@ -606,9 +611,13 @@ static void s390_pci_update_iotlb(S390PCIIOMMU *iommu, S390IOTLBEntry *entry) cache->len = PAGE_SIZE; cache->perm = entry->perm; g_hash_table_replace(iommu->iotlb, &cache->iova, cache); + DEC_DMA_AVAIL(iommu); } memory_region_notify_iommu(&iommu->iommu_mr, 0, notify); + +out: + return iommu->dma_limit ? iommu->dma_limit->avail : 1; } int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) @@ -620,6 +629,7 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) S390PCIIOMMU *iommu; S390IOTLBEntry entry; hwaddr start, end; + uint32_t dma_avail; if (env->psw.mask & PSW_MASK_PSTATE) { s390_program_interrupt(env, PGM_PRIVILEGED, ra); @@ -675,8 +685,9 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) } start += entry.len; - while (entry.iova < start && entry.iova < end) { - s390_pci_update_iotlb(iommu, &entry); + dma_avail = 1; /* Assume non-zero dma_avail to start */ + while (entry.iova < start && entry.iova < end && dma_avail > 0) { + dma_avail = s390_pci_update_iotlb(iommu, &entry); entry.iova += PAGE_SIZE; entry.translated_addr += PAGE_SIZE; } @@ -689,7 +700,13 @@ err: s390_pci_generate_error_event(error, pbdev->fh, pbdev->fid, start, 0); } else { pbdev->fmb.counter[ZPCI_FMB_CNT_RPCIT]++; - setcc(cpu, ZPCI_PCI_LS_OK); + if (dma_avail > 0) { + setcc(cpu, ZPCI_PCI_LS_OK); + } else { + /* vfio DMA mappings are exhausted, trigger a RPCIT */ + setcc(cpu, ZPCI_PCI_LS_ERR); + s390_set_status_code(env, r1, ZPCI_RPCIT_ST_INSUFF_RES); + } } return 0; } diff --git a/hw/s390x/s390-pci-inst.h b/hw/s390x/s390-pci-inst.h index fa3bf8b..8ee3a3c 100644 --- a/hw/s390x/s390-pci-inst.h +++ b/hw/s390x/s390-pci-inst.h @@ -254,6 +254,9 @@ typedef struct ClpReqRspQueryPciGrp { #define ZPCI_STPCIFC_ST_INVAL_DMAAS 28 #define ZPCI_STPCIFC_ST_ERROR_RECOVER 40 +/* Refresh PCI Translations status codes */ +#define ZPCI_RPCIT_ST_INSUFF_RES 16 + /* FIB function controls */ #define ZPCI_FIB_FC_ENABLED 0x80 #define ZPCI_FIB_FC_ERROR 0x40 From patchwork Mon Sep 14 22:29:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 11775077 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C824192C for ; Mon, 14 Sep 2020 22:33:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 85B8C208E4 for ; Mon, 14 Sep 2020 22:33:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="HYpEIc0n" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 85B8C208E4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Received: from localhost ([::1]:35374 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1kHx2Y-0006uU-UB for patchwork-qemu-devel@patchwork.kernel.org; Mon, 14 Sep 2020 18:33:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:57008) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHwyn-0003dY-9r; Mon, 14 Sep 2020 18:29:45 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:24162 helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kHwyl-0002rk-4b; Mon, 14 Sep 2020 18:29:45 -0400 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 08EM16Pv184435; Mon, 14 Sep 2020 18:29:42 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=pp1; bh=LTJ9SCOY13WvROMsevAjTtFI/wX7cmIpINdVDEHU+Qk=; b=HYpEIc0nglQ34qQ+TSSa4CAnrIYZCFsjXjOC/26hGP3DTAk3WkbCSanx8+gaz9YFPr5d eoJ8pXUIY9EXJSywDsmWSuxp1XVePle7WccyIdhihYiNwBlP9iY3wGiVgUkK9NvV9Zye Ih++vbIzDFLKEyfMa7YFv+bhHyD/XGELrmKF2GOTPVeolnwD4QL9SSatEeMn0rVO8DiE mWPMFqXQ6/mVs/gMhuijnHfRux3e8QDGzdwR+OZrZtq8OAPb7RuUObciSZDzEOukPtjZ 0kGh3PEIV0CThPAGwFGu/TZvKR4reXjtwDeSt0QjRgz5GLroitPYbs0bMCiS3HYwoRrs aw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 33jfbab3sx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 18:29:41 -0400 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 08EMS93e070468; Mon, 14 Sep 2020 18:29:41 -0400 Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0b-001b2d01.pphosted.com with ESMTP id 33jfbab3sk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 18:29:41 -0400 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 08EMRoxb023277; Mon, 14 Sep 2020 22:29:41 GMT Received: from b03cxnp07028.gho.boulder.ibm.com (b03cxnp07028.gho.boulder.ibm.com [9.17.130.15]) by ppma05wdc.us.ibm.com with ESMTP id 33gny8u00a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 14 Sep 2020 22:29:40 +0000 Received: from b03ledav005.gho.boulder.ibm.com (b03ledav005.gho.boulder.ibm.com [9.17.130.236]) by b03cxnp07028.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 08EMTdkk47120828 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 14 Sep 2020 22:29:39 GMT Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B7811BE051; Mon, 14 Sep 2020 22:29:39 +0000 (GMT) Received: from b03ledav005.gho.boulder.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8EAC5BE04F; Mon, 14 Sep 2020 22:29:38 +0000 (GMT) Received: from oc4221205838.ibm.com (unknown [9.163.85.51]) by b03ledav005.gho.boulder.ibm.com (Postfix) with ESMTP; Mon, 14 Sep 2020 22:29:38 +0000 (GMT) From: Matthew Rosato To: alex.williamson@redhat.com, cohuck@redhat.com Subject: [PATCH v2 3/3] vfio: Create shared routine for scanning info capabilities Date: Mon, 14 Sep 2020 18:29:30 -0400 Message-Id: <1600122570-12941-4-git-send-email-mjrosato@linux.ibm.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1600122570-12941-1-git-send-email-mjrosato@linux.ibm.com> References: <1600122570-12941-1-git-send-email-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235, 18.0.687 definitions=2020-09-14_09:2020-09-14, 2020-09-14 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 phishscore=0 spamscore=0 clxscore=1015 adultscore=0 suspectscore=0 priorityscore=1501 bulkscore=0 impostorscore=0 malwarescore=0 mlxlogscore=999 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2009140166 Received-SPF: pass client-ip=148.163.158.5; envelope-from=mjrosato@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-detected-operating-system: by eggs.gnu.org: First seen = 2020/09/14 18:29:38 X-ACL-Warn: Detected OS = Linux 3.x [generic] X-Spam_score_int: -26 X-Spam_score: -2.7 X-Spam_bar: -- X-Spam_report: (-2.7 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: thuth@redhat.com, pmorel@linux.ibm.com, david@redhat.com, schnelle@linux.ibm.com, qemu-devel@nongnu.org, pasic@linux.ibm.com, borntraeger@de.ibm.com, qemu-s390x@nongnu.org, rth@twiddle.net Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" Rather than duplicating the same loop in multiple locations, create a static function to do the work. Signed-off-by: Matthew Rosato --- hw/vfio/common.c | 33 +++++++++++++++------------------ 1 file changed, 15 insertions(+), 18 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 7f4a338..bfbbfe4 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -825,17 +825,12 @@ static void vfio_listener_release(VFIOContainer *container) } } -struct vfio_info_cap_header * -vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id) +static struct vfio_info_cap_header * +vfio_get_cap(void *ptr, uint32_t cap_offset, uint16_t id) { struct vfio_info_cap_header *hdr; - void *ptr = info; - - if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS)) { - return NULL; - } - for (hdr = ptr + info->cap_offset; hdr != ptr; hdr = ptr + hdr->next) { + for (hdr = ptr + cap_offset; hdr != ptr; hdr = ptr + hdr->next) { if (hdr->id == id) { return hdr; } @@ -844,23 +839,25 @@ vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id) return NULL; } + +struct vfio_info_cap_header * +vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id) +{ + if (!(info->flags & VFIO_REGION_INFO_FLAG_CAPS)) { + return NULL; + } + + return vfio_get_cap((void *)info, info->cap_offset, id); +} + static struct vfio_info_cap_header * vfio_get_iommu_type1_info_cap(struct vfio_iommu_type1_info *info, uint16_t id) { - struct vfio_info_cap_header *hdr; - void *ptr = info; - if (!(info->flags & VFIO_IOMMU_INFO_CAPS)) { return NULL; } - for (hdr = ptr + info->cap_offset; hdr != ptr; hdr = ptr + hdr->next) { - if (hdr->id == id) { - return hdr; - } - } - - return NULL; + return vfio_get_cap((void *)info, info->cap_offset, id); } bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info,