From patchwork Fri Oct 28 19:47:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 13024257 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F481ECAAA1 for ; Fri, 28 Oct 2022 19:49:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ooVM1-0000FY-GT; Fri, 28 Oct 2022 15:49:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ooVKw-0007tB-Dn; Fri, 28 Oct 2022 15:48:15 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5] helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ooVKu-00084B-JV; Fri, 28 Oct 2022 15:48:14 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29SJIq9i013678; Fri, 28 Oct 2022 19:48:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : mime-version; s=pp1; bh=nehj1xAdeOWrD81mNDca9EgI41WgI2JQMCBbw89hpOA=; b=tDfPQEGkFtXZLXc0Zaf5I9iCRg5AOFCeMnjRb8zeCOLROHtBOiUfgSFSoCyw8ZitM64T iX44Eo9/WwJzdmnC9ptCqp1g28DW0fq0moCwp0N8QC/6e74ckvIEWpO7mRmkhJpdzn6E RPzWhWgH4Od1g2kyHtL4/EcZmcWQLkgij/qlQ22zEjo0Vo6zqYkSn//EKtS9iUML0c07 kUWcHcdao4407vZngR3lbkiBZK+9pBSlnIfXD7NhixqrM015IuToI52ytNjBPax4oio0 rh53mbT68NzS+G/jAGzhdkzJvdF5fESOYoS9aD8gFhGP2hVPCs2XSRxvXSyn0EuRRSim Uw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3kgn4p0t3p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:04 +0000 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29SJIvq7013953; Fri, 28 Oct 2022 19:48:03 GMT Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3kgn4p0t2x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:03 +0000 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29SJZQjO020125; Fri, 28 Oct 2022 19:48:03 GMT Received: from b03cxnp07027.gho.boulder.ibm.com (b03cxnp07027.gho.boulder.ibm.com [9.17.130.14]) by ppma01dal.us.ibm.com with ESMTP id 3kfahyh3ge-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:03 +0000 Received: from smtpav03.dal12v.mail.ibm.com ([9.208.128.129]) by b03cxnp07027.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29SJm1q520578896 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 28 Oct 2022 19:48:01 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 373EF5806C; Fri, 28 Oct 2022 19:48:01 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 430CC58061; Fri, 28 Oct 2022 19:48:00 +0000 (GMT) Received: from li-2311da4c-2e09-11b2-a85c-c003041e9174.ibm.com.com (unknown [9.160.93.208]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Fri, 28 Oct 2022 19:48:00 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: farman@linux.ibm.com, pmorel@linux.ibm.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, qemu-devel@nongnu.org Subject: [PATCH 1/3] s390x/pci: RPCIT second pass when mappings exhausted Date: Fri, 28 Oct 2022 15:47:56 -0400 Message-Id: <20221028194758.204007-2-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221028194758.204007-1-mjrosato@linux.ibm.com> References: <20221028194758.204007-1-mjrosato@linux.ibm.com> X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: GOY_y9PEW-AxouaCPZ6g-Q329jtVCpmk X-Proofpoint-GUID: Cw42c96Nnz0YkYIVsHiICnybpSk9xVn4 X-Proofpoint-UnRewURL: 0 URL was un-rewritten MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-28_10,2022-10-27_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 clxscore=1011 lowpriorityscore=0 impostorscore=0 bulkscore=0 priorityscore=1501 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2210280123 Received-SPF: pass client-ip=148.163.158.5; envelope-from=mjrosato@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org If we encounter a new mapping while the number of available DMA entries in vfio is 0, we are currently skipping that mapping which is a problem if we manage to free up DMA space after that within the same RPCIT -- we will return to the guest with CC0 and have not mapped everything within the specified range. This issue was uncovered while testing changes to the s390 linux kernel iommu/dma code, where a different usage pattern was employed (new mappings start at the end of the aperture and work back towards the front, making us far more likely to encounter new mappings before invalidated mappings during a global refresh). Fix this by tracking whether any mappings were skipped due to vfio DMA limit hitting 0; when this occurs, we still continue the range and unmap/map anything we can - then we must re-run the range again to pickup anything that was missed. This must occur in a loop until all requests are satisfied (success) or we detect that we are still unable to complete all mappings (return ZPCI_RPCIT_ST_INSUFF_RES). Link: https://lore.kernel.org/linux-s390/20221019144435.369902-1-schnelle@linux.ibm.com/ Fixes: 37fa32de70 ("s390x/pci: Honor DMA limits set by vfio") Reported-by: Niklas Schnelle Signed-off-by: Matthew Rosato Reviewed-by: Eric Farman --- hw/s390x/s390-pci-inst.c | 29 ++++++++++++++++++++++------- 1 file changed, 22 insertions(+), 7 deletions(-) diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 20a9bcc7af..7cc4bcf850 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -677,8 +677,9 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) S390PCIBusDevice *pbdev; S390PCIIOMMU *iommu; S390IOTLBEntry entry; - hwaddr start, end; + hwaddr start, end, sstart; uint32_t dma_avail; + bool again; if (env->psw.mask & PSW_MASK_PSTATE) { s390_program_interrupt(env, PGM_PRIVILEGED, ra); @@ -691,7 +692,7 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) } fh = env->regs[r1] >> 32; - start = env->regs[r2]; + sstart = start = env->regs[r2]; end = start + env->regs[r2 + 1]; pbdev = s390_pci_find_dev_by_fh(s390_get_phb(), fh); @@ -732,6 +733,9 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) goto err; } + retry: + start = sstart; + again = false; while (start < end) { error = s390_guest_io_table_walk(iommu->g_iota, start, &entry); if (error) { @@ -739,13 +743,24 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) } start += entry.len; - while (entry.iova < start && entry.iova < end && - (dma_avail > 0 || entry.perm == IOMMU_NONE)) { - dma_avail = s390_pci_update_iotlb(iommu, &entry); - entry.iova += TARGET_PAGE_SIZE; - entry.translated_addr += TARGET_PAGE_SIZE; + while (entry.iova < start && entry.iova < end) { + if (dma_avail > 0 || entry.perm == IOMMU_NONE) { + dma_avail = s390_pci_update_iotlb(iommu, &entry); + entry.iova += TARGET_PAGE_SIZE; + entry.translated_addr += TARGET_PAGE_SIZE; + } else { + /* + * We are unable to make a new mapping at this time, continue + * on and hopefully free up more space. Then attempt another + * pass. + */ + again = true; + break; + } } } + if (again && dma_avail > 0) + goto retry; err: if (error) { pbdev->state = ZPCI_FS_ERROR; From patchwork Fri Oct 28 19:47:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 13024258 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 75065FA3740 for ; Fri, 28 Oct 2022 19:49:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ooVLx-000067-GY; Fri, 28 Oct 2022 15:49:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ooVKw-0007tD-H0; Fri, 28 Oct 2022 15:48:16 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5] helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ooVKv-00084F-1e; Fri, 28 Oct 2022 15:48:14 -0400 Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29SJIqEI013693; Fri, 28 Oct 2022 19:48:05 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=oMxIFN5RtppHnUAuO9brjKZYjERmcYxV//XuqwD6q8A=; b=h9zDzNdgq60zkEt5XVfJBJhnBXgey4Eek1U4E2Vnayp0vcKUrgK7SheuDeIz41e1fFYD rPD3giqftMNG66MQJZmjqmSnNJuQ/YPS3McAZjt2QSxvfKxAmikET4BYBoGYgtYLfz7l APTNVqEOSQthJTqG2YS4kdKLkCp0hYk0eIvLiMC+mNaEFEemWko6QTXyQtaQ06BU9AhK 6JhgpCx3LsqoCMFCAPFKr0OXbEaXyPXHM+aUYbYcp7Rqk18716ePP63cptLjvknFDQzD vbUJasNKfH9CgqOSyj2kv1jCVLGRw3SqdCU/pTC4lldi65wvcL12rrrcOho7kVaCE5Xg XQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3kgn4p0t41-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:05 +0000 Received: from m0098419.ppops.net (m0098419.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29SJJBl7015626; Fri, 28 Oct 2022 19:48:04 GMT Received: from ppma05wdc.us.ibm.com (1b.90.2fa9.ip4.static.sl-reverse.com [169.47.144.27]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3kgn4p0t3s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:04 +0000 Received: from pps.filterd (ppma05wdc.us.ibm.com [127.0.0.1]) by ppma05wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29SJZURY030058; Fri, 28 Oct 2022 19:48:04 GMT Received: from b03cxnp07029.gho.boulder.ibm.com (b03cxnp07029.gho.boulder.ibm.com [9.17.130.16]) by ppma05wdc.us.ibm.com with ESMTP id 3kfah2p378-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:04 +0000 Received: from smtpav03.dal12v.mail.ibm.com ([9.208.128.129]) by b03cxnp07029.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29SJm1xi17957424 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 28 Oct 2022 19:48:01 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 83E9958061; Fri, 28 Oct 2022 19:48:02 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5784458063; Fri, 28 Oct 2022 19:48:01 +0000 (GMT) Received: from li-2311da4c-2e09-11b2-a85c-c003041e9174.ibm.com.com (unknown [9.160.93.208]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Fri, 28 Oct 2022 19:48:01 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: farman@linux.ibm.com, pmorel@linux.ibm.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, qemu-devel@nongnu.org Subject: [PATCH 2/3] s390x/pci: coalesce unmap operations Date: Fri, 28 Oct 2022 15:47:57 -0400 Message-Id: <20221028194758.204007-3-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221028194758.204007-1-mjrosato@linux.ibm.com> References: <20221028194758.204007-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: rKu1RKWBHGPxww929r_wvjIlMytpakZD X-Proofpoint-GUID: 1LdIqpb8KGND4TKKomYS14MdNSLL8jdO X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-28_10,2022-10-27_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 mlxlogscore=999 clxscore=1015 lowpriorityscore=0 impostorscore=0 bulkscore=0 priorityscore=1501 phishscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2210280123 Received-SPF: pass client-ip=148.163.158.5; envelope-from=mjrosato@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently, each unmapped page is handled as an individual iommu region notification. Attempt to group contiguous unmap operations into fewer notifications to reduce overhead. Signed-off-by: Matthew Rosato Reviewed-by: Eric Farman --- hw/s390x/s390-pci-inst.c | 51 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/hw/s390x/s390-pci-inst.c b/hw/s390x/s390-pci-inst.c index 7cc4bcf850..66e764f901 100644 --- a/hw/s390x/s390-pci-inst.c +++ b/hw/s390x/s390-pci-inst.c @@ -640,6 +640,8 @@ static uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, } g_hash_table_remove(iommu->iotlb, &entry->iova); inc_dma_avail(iommu); + /* Don't notify the iommu yet, maybe we can bundle contiguous unmaps */ + goto out; } else { if (cache) { if (cache->perm == entry->perm && @@ -663,15 +665,44 @@ static uint32_t s390_pci_update_iotlb(S390PCIIOMMU *iommu, dec_dma_avail(iommu); } + /* + * All associated iotlb entries have already been cleared, trigger the + * unmaps. + */ memory_region_notify_iommu(&iommu->iommu_mr, 0, event); out: return iommu->dma_limit ? iommu->dma_limit->avail : 1; } +static void s390_pci_batch_unmap(S390PCIIOMMU *iommu, uint64_t iova, + uint64_t len) +{ + uint64_t remain = len, start = iova, end = start + len - 1, mask, size; + IOMMUTLBEvent event = { + .type = IOMMU_NOTIFIER_UNMAP, + .entry = { + .target_as = &address_space_memory, + .translated_addr = 0, + .perm = IOMMU_NONE, + }, + }; + + while (remain >= TARGET_PAGE_SIZE) { + mask = dma_aligned_pow2_mask(start, end, 64); + size = mask + 1; + event.entry.iova = start; + event.entry.addr_mask = mask; + memory_region_notify_iommu(&iommu->iommu_mr, 0, event); + start += size; + remain -= size; + } +} + int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) { CPUS390XState *env = &cpu->env; + uint64_t iova, coalesce = 0; uint32_t fh; uint16_t error = 0; S390PCIBusDevice *pbdev; @@ -742,6 +773,21 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) break; } + /* + * If this is an unmap of a PTE, let's try to coalesce multiple unmaps + * into as few notifier events as possible. + */ + if (entry.perm == IOMMU_NONE && entry.len == TARGET_PAGE_SIZE) { + if (coalesce == 0) { + iova = entry.iova; + } + coalesce += entry.len; + } else if (coalesce > 0) { + /* Unleash the coalesced unmap before processing a new map */ + s390_pci_batch_unmap(iommu, iova, coalesce); + coalesce = 0; + } + start += entry.len; while (entry.iova < start && entry.iova < end) { if (dma_avail > 0 || entry.perm == IOMMU_NONE) { @@ -759,6 +805,11 @@ int rpcit_service_call(S390CPU *cpu, uint8_t r1, uint8_t r2, uintptr_t ra) } } } + if (coalesce) { + /* Unleash the coalesced unmap before finishing rpcit */ + s390_pci_batch_unmap(iommu, iova, coalesce); + coalesce = 0; + } if (again && dma_avail > 0) goto retry; err: From patchwork Fri Oct 28 19:47:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Rosato X-Patchwork-Id: 13024256 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E8C42C38A02 for ; Fri, 28 Oct 2022 19:49:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1ooVLT-00083Q-Nb; Fri, 28 Oct 2022 15:48:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ooVKu-0007sc-Tt; Fri, 28 Oct 2022 15:48:14 -0400 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5] helo=mx0a-001b2d01.pphosted.com) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1ooVKs-00084N-Gc; Fri, 28 Oct 2022 15:48:12 -0400 Received: from pps.filterd (m0098420.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 29SJiIbQ007068; Fri, 28 Oct 2022 19:48:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=pjuDXaF8RyohrXvfJmk+1y+Q4Pz6Xe0BzfZQouZ6zQA=; b=dNYGq/LvjmPDIQAsMSTt2ljoMkOB7il+M/HKQAwuLQzaQ01Y4xUee/ns2Hg6Q9z9DTTh 35/wrmGm0XVUptItz200h9mCWW0RUZdn5s93JCJwB1oEuz19smUafjtzmDQKX0TyvUZW I4SGhrHqsGvozNerxzRVnaFKISXiUgQqdwaU77W8YU5w6syL8eTLZ/I/M0NW4+xkWNqH gKb/qAUa1R9wI4lyBB/36KX8zhpBQN0+L3F9/MAetqvXbFismL+xApcKz2Dg0s7ZJQEE GS0Cz7PYPAS9iLRneO7TabraeWrrkClMCh1Fh3UPw1jtY9sZ9Lq3vgPUBqCQQ0gEIllA 0g== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3kgngc830q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:06 +0000 Received: from m0098420.ppops.net (m0098420.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 29SJkhAE016760; Fri, 28 Oct 2022 19:48:05 GMT Received: from ppma04wdc.us.ibm.com (1a.90.2fa9.ip4.static.sl-reverse.com [169.47.144.26]) by mx0b-001b2d01.pphosted.com (PPS) with ESMTPS id 3kgngc830a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:05 +0000 Received: from pps.filterd (ppma04wdc.us.ibm.com [127.0.0.1]) by ppma04wdc.us.ibm.com (8.16.1.2/8.16.1.2) with SMTP id 29SJa3Tj008805; Fri, 28 Oct 2022 19:48:05 GMT Received: from b03cxnp08025.gho.boulder.ibm.com (b03cxnp08025.gho.boulder.ibm.com [9.17.130.17]) by ppma04wdc.us.ibm.com with ESMTP id 3kfah7wjdm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Oct 2022 19:48:05 +0000 Received: from smtpav03.dal12v.mail.ibm.com ([9.208.128.129]) by b03cxnp08025.gho.boulder.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 29SJm2XV60817798 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 28 Oct 2022 19:48:02 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 98A455805A; Fri, 28 Oct 2022 19:48:03 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A494558068; Fri, 28 Oct 2022 19:48:02 +0000 (GMT) Received: from li-2311da4c-2e09-11b2-a85c-c003041e9174.ibm.com.com (unknown [9.160.93.208]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Fri, 28 Oct 2022 19:48:02 +0000 (GMT) From: Matthew Rosato To: qemu-s390x@nongnu.org Cc: farman@linux.ibm.com, pmorel@linux.ibm.com, schnelle@linux.ibm.com, cohuck@redhat.com, thuth@redhat.com, pasic@linux.ibm.com, borntraeger@linux.ibm.com, richard.henderson@linaro.org, david@redhat.com, qemu-devel@nongnu.org Subject: [PATCH 3/3] s390x/pci: shrink DMA aperture to be bound by vfio DMA limit Date: Fri, 28 Oct 2022 15:47:58 -0400 Message-Id: <20221028194758.204007-4-mjrosato@linux.ibm.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221028194758.204007-1-mjrosato@linux.ibm.com> References: <20221028194758.204007-1-mjrosato@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-GUID: 6jUOY68sg8K535nzbjugr4MLO7k65yQJ X-Proofpoint-ORIG-GUID: -GaIcLVmAIDqhjgmE7pJLvM1xcxYoipk X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.545,FMLib:17.11.122.1 definitions=2022-10-28_10,2022-10-27_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 mlxscore=0 lowpriorityscore=0 spamscore=0 mlxlogscore=989 clxscore=1015 impostorscore=0 malwarescore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2210170000 definitions=main-2210280123 Received-SPF: pass client-ip=148.163.158.5; envelope-from=mjrosato@linux.ibm.com; helo=mx0a-001b2d01.pphosted.com X-Spam_score_int: -19 X-Spam_score: -2.0 X-Spam_bar: -- X-Spam_report: (-2.0 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Qemu-devel" Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Currently, s390x-pci performs accounting against the vfio DMA limit and triggers the guest to clean up mappings when the limit is reached. Let's go a step further and also limit the size of the supported DMA aperture reported to the guest based upon the initial vfio DMA limit reported for the container (if less than than the size reported by the firmware/host zPCI layer). This avoids processing sections of the guest DMA table during global refresh that, for common use cases, will never be used anway, and makes exhausting the vfio DMA limit due to mismatch between guest aperture size and host limit far less likely and more indicitive of an error. Signed-off-by: Matthew Rosato Reviewed-by: Eric Farman --- hw/s390x/s390-pci-vfio.c | 11 +++++++++++ include/hw/s390x/s390-pci-bus.h | 1 + 2 files changed, 12 insertions(+) diff --git a/hw/s390x/s390-pci-vfio.c b/hw/s390x/s390-pci-vfio.c index 2aefa508a0..99806e2a84 100644 --- a/hw/s390x/s390-pci-vfio.c +++ b/hw/s390x/s390-pci-vfio.c @@ -84,6 +84,7 @@ S390PCIDMACount *s390_pci_start_dma_count(S390pciState *s, cnt->users = 1; cnt->avail = avail; QTAILQ_INSERT_TAIL(&s->zpci_dma_limit, cnt, link); + pbdev->iommu->max_dma_limit = avail; return cnt; } @@ -103,6 +104,7 @@ static void s390_pci_read_base(S390PCIBusDevice *pbdev, struct vfio_info_cap_header *hdr; struct vfio_device_info_cap_zpci_base *cap; VFIOPCIDevice *vpci = container_of(pbdev->pdev, VFIOPCIDevice, pdev); + uint64_t vfio_size; hdr = vfio_get_device_info_cap(info, VFIO_DEVICE_INFO_CAP_ZPCI_BASE); @@ -122,6 +124,15 @@ static void s390_pci_read_base(S390PCIBusDevice *pbdev, /* The following values remain 0 until we support other FMB formats */ pbdev->zpci_fn.fmbl = 0; pbdev->zpci_fn.pft = 0; + + /* + * If appropriate, reduce the size of the supported DMA aperture reported + * to the guest based upon the vfio DMA limit. + */ + vfio_size = pbdev->iommu->max_dma_limit << TARGET_PAGE_BITS; + if (vfio_size < (cap->end_dma - cap->start_dma + 1)) { + pbdev->zpci_fn.edma = cap->start_dma + vfio_size - 1; + } } static bool get_host_fh(S390PCIBusDevice *pbdev, struct vfio_device_info *info, diff --git a/include/hw/s390x/s390-pci-bus.h b/include/hw/s390x/s390-pci-bus.h index 0605fcea24..1c46e3a269 100644 --- a/include/hw/s390x/s390-pci-bus.h +++ b/include/hw/s390x/s390-pci-bus.h @@ -278,6 +278,7 @@ struct S390PCIIOMMU { uint64_t g_iota; uint64_t pba; uint64_t pal; + uint64_t max_dma_limit; GHashTable *iotlb; S390PCIDMACount *dma_limit; };