From patchwork Sat Jul 20 19:15:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737890 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A7EC6C3DA63 for ; Sat, 20 Jul 2024 19:18:09 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZE-0007RY-Iq; Sat, 20 Jul 2024 15:16:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYt-0006pt-CI for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYj-0000pN-2M for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:02 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJAT2N021757; Sat, 20 Jul 2024 19:15:42 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=+svMWzFd5fTMtOMPInlaYfVGm5rk0v5rTgwc8FqpUes=; b= NaYfL5HSm6maWXJrVj4vUL66k+vbZfU5/oGoXpNzxt+EczPbFBHxLPo+cE9aV9Yd GUemXQEITDHNB/mFQhfmWsGRjxMsv9K0wn7/b6KzBIke2oyVr15aV742Pc4xjv32 NEq80VL3u/AS2E3Q8NlZzqJQR94DE6qNOIpayClrEisy7grR69AOAtr/Gq0ptCMP bsWscsbaBlsLkwhHR7N80LDaBEf2BQx3GE/FquE7GYcpkUPJQR/LVDRjqFRm+e+l 7qH053MI0mJvWe7gtpsw1fTDqXrqgUH//wgM0CghzmRBuv6RSzwQDj/xlTjVHn+O yVsFlz5VXnyxGpIOh3IHkg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5jr04b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:41 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KI7MNd027535; Sat, 20 Jul 2024 19:15:41 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5putc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:41 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdf3028365; Sat, 20 Jul 2024 19:15:40 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-2; Sat, 20 Jul 2024 19:15:40 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 01/12] vfio: move cpr_exec_notifier Date: Sat, 20 Jul 2024 12:15:26 -0700 Message-Id: <1721502937-87102-2-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: AiT4tP8vkgJH0EV0qLH0UFEc2NZ00gAd X-Proofpoint-ORIG-GUID: AiT4tP8vkgJH0EV0qLH0UFEc2NZ00gAd Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Move the cpr notifier to the base container. This change will be squashed into the "live update: vfio" series. Signed-off-by: Steve Sistare --- hw/vfio/cpr-legacy.c | 10 +++++----- include/hw/vfio/vfio-common.h | 1 - include/hw/vfio/vfio-container-base.h | 1 + 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/hw/vfio/cpr-legacy.c b/hw/vfio/cpr-legacy.c index 8f6224e..91be762 100644 --- a/hw/vfio/cpr-legacy.c +++ b/hw/vfio/cpr-legacy.c @@ -107,9 +107,9 @@ static const VMStateDescription vfio_container_vmstate = { static int vfio_cpr_fail_notifier(NotifierWithReturn *notifier, MigrationEvent *e, Error **errp) { - VFIOContainer *container = - container_of(notifier, VFIOContainer, cpr_exec_notifier); - VFIOContainerBase *bcontainer = &container->bcontainer; + VFIOContainerBase *bcontainer = + container_of(notifier, VFIOContainerBase, cpr_exec_notifier); + VFIOContainer *container = VFIO_CONTAINER(bcontainer); if (e->type != MIG_EVENT_PRECOPY_FAILED) { return 0; @@ -147,7 +147,7 @@ bool vfio_legacy_cpr_register_container(VFIOContainerBase *bcontainer, vmstate_register(NULL, -1, &vfio_container_vmstate, container); - migration_add_notifier_mode(&container->cpr_exec_notifier, + migration_add_notifier_mode(&bcontainer->cpr_exec_notifier, vfio_cpr_fail_notifier, MIG_MODE_CPR_EXEC); return true; @@ -158,5 +158,5 @@ void vfio_legacy_cpr_unregister_container(VFIOContainerBase *bcontainer) VFIOContainer *container = VFIO_CONTAINER(bcontainer); vmstate_unregister(NULL, &vfio_container_vmstate, container); - migration_remove_notifier(&container->cpr_exec_notifier); + migration_remove_notifier(&bcontainer->cpr_exec_notifier); } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 1902c8f..9512a0c 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -81,7 +81,6 @@ typedef struct VFIOContainer { VFIOContainerBase bcontainer; int fd; /* /dev/vfio/vfio, empowered by the attached groups */ unsigned iommu_type; - NotifierWithReturn cpr_exec_notifier; bool vaddr_unmapped; QLIST_HEAD(, VFIOGroup) group_list; } VFIOContainer; diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index 3d30365..f8b7b26 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -52,6 +52,7 @@ typedef struct VFIOContainerBase { QLIST_HEAD(, VFIODevice) device_list; GList *iova_ranges; NotifierWithReturn cpr_reboot_notifier; + NotifierWithReturn cpr_exec_notifier; Error *cpr_blocker; } VFIOContainerBase; From patchwork Sat Jul 20 19:15:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 77C92C3DA59 for ; Sat, 20 Jul 2024 19:16:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZ9-00074U-Hg; Sat, 20 Jul 2024 15:16:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYv-0006px-O4 for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYj-0000pU-2O for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:05 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJC5S6024759; Sat, 20 Jul 2024 19:15:43 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=WWHM+SubOAqNkb6b1haldRvEsQO00N7M97eRzHUiLPc=; b= JzOFMrhF7trCRzoi1AFbiTfbosSZZC8d7OX5nUxu6Os7LG4P9+6WAEzSZWHtB+RS BQMbINdy41ItCb8QGbXdJUDlTeCdQ0zJMnmJE66Aehe91KUHSCqni01uxQqqVzro o0IgEQwQvncTlavjsl9UC3tFdz/Yybgxz5ifN9A0ddDak1XHZMM2Nj1GoiS2rW3t nZxRLDjiCUWp2KtTIcqlFp0sFi95Nq/8mTQgttnhLNfacMDFfdVycgcBUXBAwmMt r57Aor2ZCxXPlbG3UkUVpnNFv4PzdBxtCZm4+whOCQW72iG9AFJi6N+Q54sNzY2R +xMVGQBY8JTzj4W6xRGpbw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5jr04c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:43 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KI0BUH027428; Sat, 20 Jul 2024 19:15:42 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5putk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:42 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdf5028365; Sat, 20 Jul 2024 19:15:42 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-3; Sat, 20 Jul 2024 19:15:41 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 02/12] iommufd: no DMA to BARs Date: Sat, 20 Jul 2024 12:15:27 -0700 Message-Id: <1721502937-87102-3-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: HXuqmKPOJMNoYFbNfgIxL0MGKtfSuRCO X-Proofpoint-ORIG-GUID: HXuqmKPOJMNoYFbNfgIxL0MGKtfSuRCO Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Do not map VFIO PCI BARs for DMA. This stops a raft of warnings of the following form at QEMU start time when using -object iommufd: qemu-kvm: warning: IOMMU_IOAS_MAP failed: Bad address, PCI BAR? qemu-kvm: vfio_container_dma_map(0x555558282db0, 0x8800010000, 0x4000, 0x7ffff7ff0000) = -14 (Bad address) Signed-off-by: Steve Sistare --- hw/vfio/common.c | 3 ++- hw/vfio/helpers.c | 1 + include/exec/memory.h | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index da2e0ec..403d45a 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -248,7 +248,8 @@ static bool vfio_listener_skipped_section(MemoryRegionSection *section) * are never accessed by the CPU and beyond the address width of * some IOMMU hardware. TODO: VFIO should tell us the IOMMU width. */ - section->offset_within_address_space & (1ULL << 63); + section->offset_within_address_space & (1ULL << 63) || + section->mr->no_dma; } /* Called with rcu_read_lock held. */ diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index b14edd4..e4cfdd2 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -435,6 +435,7 @@ int vfio_region_mmap(VFIORegion *region) memory_region_owner(region->mem), name, region->mmaps[i].size, region->mmaps[i].mmap); + region->mmaps[i].mem.no_dma = true; g_free(name); memory_region_add_subregion(region->mem, region->mmaps[i].offset, ®ion->mmaps[i].mem); diff --git a/include/exec/memory.h b/include/exec/memory.h index ea03ef2..850cc8c 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -794,6 +794,7 @@ struct MemoryRegion { bool unmergeable; uint8_t dirty_log_mask; bool is_iommu; + bool no_dma; RAMBlock *ram_block; Object *owner; /* owner as TYPE_DEVICE. Used for re-entrancy checks in MR access hotpath */ From patchwork Sat Jul 20 19:15:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5E680C3DA49 for ; Sat, 20 Jul 2024 19:17:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZB-0007Di-7E; Sat, 20 Jul 2024 15:16:25 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYx-0006q6-JO for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYk-0000pY-Sj for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:06 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJFiNO017251; Sat, 20 Jul 2024 19:15:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=gpAaa9HSbr+6w0SO4zYiChdvbrFszVDHKwG0Gs2gAnA=; b= om6GkSBV9fU0KTC/X/UlR3e+eCZdqHvRuIJyoHTtEUB0FduZPd91RRU13EZSDYu1 9JG+bocvhOWX5KZnTe+t+yPb75h/YDGxYZsA5Rv+CC77LEtpOyWl/WsMIoysJT8X P9DFwBsPEa1S+82UxJwATTVPch3nbetUeokQyD5oBqGhD4YdnFynL3rZ1R5oPiBF rbAouQwiineI+vpcPnhSoMeNLTht27Wu8wWYir1yss1Sar7lWdPN7flY6nkxPvsu NKa+dIZ+pAZpVM13Qy1EXrpp5pKLOFbiKBRFSum6FCv71VH4kMPgnAXw2BWCQuEF ZA4vuPdqCGiN3jMliNK1nQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk7w000a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:44 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KH5ckn027509; Sat, 20 Jul 2024 19:15:44 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5putv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:43 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdf7028365; Sat, 20 Jul 2024 19:15:43 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-4; Sat, 20 Jul 2024 19:15:43 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 03/12] iommufd: pass name to connect Date: Sat, 20 Jul 2024 12:15:28 -0700 Message-Id: <1721502937-87102-4-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-ORIG-GUID: jshvbBiRpXkyEkfobyOr4P5ltYtE2vGk X-Proofpoint-GUID: jshvbBiRpXkyEkfobyOr4P5ltYtE2vGk Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Pass device name to iommufd_backend_connect and iommufd_backend_disconnect, for use by CPR in a subsequent patch. No functional change. Signed-off-by: Steve Sistare --- backends/iommufd.c | 4 ++-- hw/vfio/iommufd.c | 6 +++--- include/sysemu/iommufd.h | 5 +++-- 3 files changed, 8 insertions(+), 7 deletions(-) diff --git a/backends/iommufd.c b/backends/iommufd.c index 84fefbc..fc37386 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -72,7 +72,7 @@ static void iommufd_backend_class_init(ObjectClass *oc, void *data) object_class_property_add_str(oc, "fd", NULL, iommufd_backend_set_fd); } -bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp) +bool iommufd_backend_connect(IOMMUFDBackend *be, const char *name, Error **errp) { int fd; @@ -90,7 +90,7 @@ bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp) return true; } -void iommufd_backend_disconnect(IOMMUFDBackend *be) +void iommufd_backend_disconnect(IOMMUFDBackend *be, const char *name) { if (!be->users) { goto out; diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index c2f158e..255966a 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -71,7 +71,7 @@ static bool iommufd_cdev_connect_and_bind(VFIODevice *vbasedev, Error **errp) .flags = 0, }; - if (!iommufd_backend_connect(iommufd, errp)) { + if (!iommufd_backend_connect(iommufd, vbasedev->name, errp)) { return false; } @@ -99,7 +99,7 @@ static bool iommufd_cdev_connect_and_bind(VFIODevice *vbasedev, Error **errp) err_bind: iommufd_cdev_kvm_device_del(vbasedev); err_kvm_device_add: - iommufd_backend_disconnect(iommufd); + iommufd_backend_disconnect(iommufd, vbasedev->name); return false; } @@ -107,7 +107,7 @@ static void iommufd_cdev_unbind_and_disconnect(VFIODevice *vbasedev) { /* Unbind is automatically conducted when device fd is closed */ iommufd_cdev_kvm_device_del(vbasedev); - iommufd_backend_disconnect(vbasedev->iommufd); + iommufd_backend_disconnect(vbasedev->iommufd, vbasedev->name); } static int iommufd_cdev_getfd(const char *sysfs_path, Error **errp) diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index 9edfec6..aa195d1 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -37,8 +37,9 @@ struct IOMMUFDBackend { /*< public >*/ }; -bool iommufd_backend_connect(IOMMUFDBackend *be, Error **errp); -void iommufd_backend_disconnect(IOMMUFDBackend *be); +bool iommufd_backend_connect(IOMMUFDBackend *be, const char *name, + Error **errp); +void iommufd_backend_disconnect(IOMMUFDBackend *be, const char *name); bool iommufd_backend_alloc_ioas(IOMMUFDBackend *be, uint32_t *ioas_id, Error **errp); From patchwork Sat Jul 20 19:15:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737884 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 532A8C3DA59 for ; Sat, 20 Jul 2024 19:17:36 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZD-0007MK-DW; Sat, 20 Jul 2024 15:16:27 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYt-0006pu-CI for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYk-0000pc-Se for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:03 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJAbUS021307; Sat, 20 Jul 2024 19:15:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=7tzmnxKwo2znaMzLs9b303Ot3KrE0JHiqIjx0pM+AQk=; b= ReYqZCap+RNHb9xFUGrG0UVQyVuwu7d2uV0XUDpgJ6vk2BSt/iEsKg7199C53pTH uF/lWvpYIUOKqrUhJ5+EAXviDcpBiPBcg9mXNHjDdarzVTEKq3YAjDbwITJk1Jcf IUMmKxL/qvYtofAUJ0hmarhZAHtKG33SezIoy1+7KthoFWmW/gBEG5SnbXnna5qW gSxburRD758PHOgXfZ4fC95OhgUiuXka7Vm5CoFMAVd0wlvTlq//VESt9GpUJdF2 CaAoQPng+2NQePrurcOTzM9nO+itf9M7ANT631RM5EpB/Aoznwi4rMIdQqOdVTcK aRUS52idNMbKc4vPEd0Ajg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5gr055-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:45 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KIF8QN027510; Sat, 20 Jul 2024 19:15:45 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puty-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:45 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdf9028365; Sat, 20 Jul 2024 19:15:44 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-5; Sat, 20 Jul 2024 19:15:44 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 04/12] migration: cpr_find_fd_any Date: Sat, 20 Jul 2024 12:15:29 -0700 Message-Id: <1721502937-87102-5-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: z0rzYnRqnXyymF91NvI1MsEyXCNYAXdG X-Proofpoint-ORIG-GUID: z0rzYnRqnXyymF91NvI1MsEyXCNYAXdG Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Add a function for finding a CPR fd by name, for any value of id, and return the id. Signed-off-by: Steve Sistare --- include/migration/cpr.h | 1 + migration/cpr.c | 15 +++++++++++++++ 2 files changed, 16 insertions(+) diff --git a/include/migration/cpr.h b/include/migration/cpr.h index bfd9864..c9e6111 100644 --- a/include/migration/cpr.h +++ b/include/migration/cpr.h @@ -19,6 +19,7 @@ typedef int (*cpr_walk_fd_cb)(int fd); void cpr_save_fd(const char *name, int id, int fd); void cpr_delete_fd(const char *name, int id); int cpr_find_fd(const char *name, int id); +int cpr_find_fd_any(const char *name, int *id_p); int cpr_walk_fd(cpr_walk_fd_cb cb); void cpr_resave_fd(const char *name, int id, int fd); diff --git a/migration/cpr.c b/migration/cpr.c index 853d3a1..096e8d8 100644 --- a/migration/cpr.c +++ b/migration/cpr.c @@ -109,6 +109,21 @@ int cpr_find_fd(const char *name, int id) return fd; } +int cpr_find_fd_any(const char *name, int *id_p) +{ + CprFd *elem; + + QLIST_FOREACH(elem, &cpr_state.fds, next) { + if (!strcmp(elem->name, name)) { + trace_cpr_find_fd(name, elem->id, elem->fd); + *id_p = elem->id; + return elem->fd; + } + } + trace_cpr_find_fd(name, -1, -1); + return -1; +} + int cpr_walk_fd(cpr_walk_fd_cb cb) { CprFd *elem; From patchwork Sat Jul 20 19:15:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737882 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C726DC3DA59 for ; Sat, 20 Jul 2024 19:17:21 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZ9-00077R-Vc; Sat, 20 Jul 2024 15:16:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYv-0006py-OQ for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYj-0000pi-2M for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:03 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJC5S7024759; Sat, 20 Jul 2024 19:15:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=qRto/pvnEyoMMJwjO7J8WazkfLwLTPRBpNgqAp8ROR8=; b= PA5kzuaoH35bwTX0JBMRGQ33d5PJNrYycNTfYC+MGuzq4euCt+X1Q5Zp3+Mf6D9R SGM9WXft1Hmg6jeVNETMAVqYekgr3h/KNmz2tAsZZR+ZhJPCyj64y0eZrr40GF/J 4KIKZ0ytG51H3rn7bNDVCCV9rkCbQbNK8o5mIawXdAY0J+f24+3vSy4wkwRggyty IKZSJPp4VqUKCnD/7NXyQ9601RJOL2Pbu1YFh+AmprfYh/p+gLPSUveupN0iZ/m8 7GWUCjOk07/Z78HKUW1SqwQT7KOK+Qqo+PTEnklfRm5AEUMqm26vPqQGxpJrG9N2 JYFu49+qniiQxizg51ZHOQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5jr04d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:47 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KIfsYr027542; Sat, 20 Jul 2024 19:15:46 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puu9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:46 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfB028365; Sat, 20 Jul 2024 19:15:46 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-6; Sat, 20 Jul 2024 19:15:46 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 05/12] iommufd: preserve device fd Date: Sat, 20 Jul 2024 12:15:30 -0700 Message-Id: <1721502937-87102-6-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: I4keliZ8JNp44lv8SGujXGpB6lDuvuEu X-Proofpoint-ORIG-GUID: I4keliZ8JNp44lv8SGujXGpB6lDuvuEu Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Save the iommu and vfio device fd in CPR state when it is created, and fetch the fd from that state after CPR. Save the devid as the fd id. Remember that the fd was reused, for subsequent patches. Signed-off-by: Steve Sistare --- backends/iommufd.c | 12 +++++++++++- hw/vfio/iommufd.c | 17 ++++++++++++++++- include/sysemu/iommufd.h | 1 + 3 files changed, 28 insertions(+), 2 deletions(-) diff --git a/backends/iommufd.c b/backends/iommufd.c index fc37386..4bdbad2 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -16,6 +16,7 @@ #include "qemu/module.h" #include "qom/object_interfaces.h" #include "qemu/error-report.h" +#include "migration/cpr.h" #include "monitor/monitor.h" #include "trace.h" #include @@ -77,11 +78,17 @@ bool iommufd_backend_connect(IOMMUFDBackend *be, const char *name, Error **errp) int fd; if (be->owned && !be->users) { - fd = qemu_open_old("/dev/iommu", O_RDWR); + g_autofree char *iname = g_strdup_printf("%s_iommu", name); + fd = cpr_find_fd(iname, 0); + be->reused = (fd >= 0); + if (!be->reused) { + fd = qemu_open_old("/dev/iommu", O_RDWR); + } if (fd < 0) { error_setg_errno(errp, errno, "/dev/iommu opening failed"); return false; } + cpr_resave_fd(iname, 0, fd); be->fd = fd; } be->users++; @@ -92,6 +99,8 @@ bool iommufd_backend_connect(IOMMUFDBackend *be, const char *name, Error **errp) void iommufd_backend_disconnect(IOMMUFDBackend *be, const char *name) { + g_autofree char *iname = g_strdup_printf("%s_iommu", name); + if (!be->users) { goto out; } @@ -101,6 +110,7 @@ void iommufd_backend_disconnect(IOMMUFDBackend *be, const char *name) be->fd = -1; } out: + cpr_delete_fd(iname, 0); trace_iommufd_backend_disconnect(be->fd, be->users); } diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 255966a..cefc9e0 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -24,6 +24,7 @@ #include "sysemu/reset.h" #include "qemu/cutils.h" #include "qemu/chardev_open.h" +#include "migration/cpr.h" #include "pci.h" static int iommufd_cdev_map(const VFIOContainerBase *bcontainer, hwaddr iova, @@ -84,6 +85,11 @@ static bool iommufd_cdev_connect_and_bind(VFIODevice *vbasedev, Error **errp) goto err_kvm_device_add; } + if (vbasedev->reused) { + /* Already bound, and devid was set in iommufd_cdev_attach */ + goto skip_bind; + } + /* Bind device to iommufd */ bind.iommufd = iommufd->fd; if (ioctl(vbasedev->fd, VFIO_DEVICE_BIND_IOMMUFD, &bind)) { @@ -95,6 +101,8 @@ static bool iommufd_cdev_connect_and_bind(VFIODevice *vbasedev, Error **errp) vbasedev->devid = bind.out_devid; trace_iommufd_cdev_connect_and_bind(bind.iommufd, vbasedev->name, vbasedev->fd, vbasedev->devid); + +skip_bind: return true; err_bind: iommufd_cdev_kvm_device_del(vbasedev); @@ -305,13 +313,18 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, VFIO_IOMMU_CLASS(object_class_by_name(TYPE_VFIO_IOMMU_IOMMUFD)); if (vbasedev->fd < 0) { - devfd = iommufd_cdev_getfd(vbasedev->sysfsdev, errp); + devfd = cpr_find_fd_any(vbasedev->name, (int *)&vbasedev->devid); + vbasedev->reused = (devfd >= 0); + if (!vbasedev->reused) { + devfd = iommufd_cdev_getfd(vbasedev->sysfsdev, errp); + } if (devfd < 0) { return false; } vbasedev->fd = devfd; } else { devfd = vbasedev->fd; + vbasedev->reused = false; } if (!iommufd_cdev_connect_and_bind(vbasedev, errp)) { @@ -413,6 +426,7 @@ found_container: vbasedev->bcontainer = bcontainer; QLIST_INSERT_HEAD(&bcontainer->device_list, vbasedev, container_next); QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); + cpr_resave_fd(vbasedev->name, vbasedev->devid, devfd); trace_iommufd_cdev_device_info(vbasedev->name, devfd, vbasedev->num_irqs, vbasedev->num_regions, vbasedev->flags); @@ -452,6 +466,7 @@ static void iommufd_cdev_detach(VFIODevice *vbasedev) iommufd_cdev_container_destroy(container); vfio_put_address_space(space); + cpr_delete_fd(vbasedev->name, vbasedev->devid); iommufd_cdev_unbind_and_disconnect(vbasedev); close(vbasedev->fd); } diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index aa195d1..6955ebd 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -32,6 +32,7 @@ struct IOMMUFDBackend { /*< protected >*/ int fd; /* /dev/iommu file descriptor */ bool owned; /* is the /dev/iommu opened internally */ + bool reused; /* fd is reused after CPR */ uint32_t users; /*< public >*/ From patchwork Sat Jul 20 19:15:31 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B31D5C3DA59 for ; Sat, 20 Jul 2024 19:18:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZE-0007Po-9E; Sat, 20 Jul 2024 15:16:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYx-0006q5-I9 for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYk-0000pn-Sf for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:06 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJAbUT021307; Sat, 20 Jul 2024 19:15:48 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=Uztrps1pTcfTeC0mVuH2F8BWKUuzbgXfqQivfqijXOc=; b= cDRF1tQ5ZqpUICdHKkNjV8bMgWEeaObHXEpHd7LOcEuHI3iXr/vRtn4i3+ENaGfu sTFXmUDq5wm+HEmHRUAKs2FXpfCuhh+gzKL5aPppAbZ2RoUrR7Xqq1DBgTayFm2U HyD1cBE6Ifup2HdYxcvq1et1huu40ZbI1fOf9hEmIwcPdTOssI6SS7yZrXTu25Q7 8CfMwQ4rIXT5eOWfw4Khpd8O/J1xxs0/mrhWkb4tkeazVy183v1W2dWt9a9aM6Xh EN2XbTrHFV7dhPPVp0ZziceZs82haVI7mftrUJ0ItAeFu3XJwtF5EdpysnHp0bAa uXE2ATwlAgQVwdYIZ1UOKg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5gr058-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:48 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KHe2C6027449; Sat, 20 Jul 2024 19:15:48 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puuj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:48 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfD028365; Sat, 20 Jul 2024 19:15:47 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-7; Sat, 20 Jul 2024 19:15:47 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 06/12] iommufd: export iommufd_cdev_get_info_iova_range Date: Sat, 20 Jul 2024 12:15:31 -0700 Message-Id: <1721502937-87102-7-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: a9CwzR89gNACnChLA9S9J_wlJrWrbJnl X-Proofpoint-ORIG-GUID: a9CwzR89gNACnChLA9S9J_wlJrWrbJnl Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Export iommufd_cdev_get_info_iova_range for use by CPR. No functional change. Signed-off-by: Steve Sistare --- hw/vfio/iommufd.c | 4 ++-- include/hw/vfio/vfio-common.h | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index cefc9e0..6d77daa 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -257,8 +257,8 @@ static int iommufd_cdev_ram_block_discard_disable(bool state) return ram_block_uncoordinated_discard_disable(state); } -static bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, - uint32_t ioas_id, Error **errp) +bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, + uint32_t ioas_id, Error **errp) { VFIOContainerBase *bcontainer = &container->bcontainer; g_autofree struct iommu_ioas_iova_ranges *info = NULL; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 9512a0c..ec5b7168 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -245,6 +245,8 @@ void vfio_cpr_unregister_container(VFIOContainerBase *bcontainer); bool vfio_legacy_cpr_register_container(VFIOContainerBase *bcontainer, Error **errp); void vfio_legacy_cpr_unregister_container(VFIOContainerBase *bcontainer); +bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, + uint32_t ioas_id, Error **errp); extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; From patchwork Sat Jul 20 19:15:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 49EAEC3DA59 for ; Sat, 20 Jul 2024 19:18:02 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZ8-0006yx-7E; Sat, 20 Jul 2024 15:16:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYx-0006q8-Js for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYk-0000ps-Sf for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:06 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJFiNQ017251; Sat, 20 Jul 2024 19:15:50 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=0dXgl+LujCB3wIyI+XNOOhw1CD+oAEVN21vOCufobWU=; b= mpZmrVcy3R+26LxpCbOs7XOisNRbG6hkjFEb+5W0MwKYS3ZuP0Ah0C6SpP1ll22J 2UnhAoj1LxBDT7+ERk+eOye3qPqyXUdZ19CVXRJPueOJaVzY8DI4jIWg0IPdLwRs 6lWHUN3+ToRmgaI+UDyEPYS5R3UxbQIQLWFaYESlpFvKZloQ0GZqsztrneOXPsT8 ApX1Qjhaxc28klmR63aYyrh76acOSoOacKHCV+zq24aK4MBvX510HYS4kaKi08k8 s8VssQBfcK65jC6ipMRi9dpaKiJVjhSyc8xBAc23nwgV0gB90DX0m2jGffcK/wdw PhIBb4UaXv6k7B85GGuz3A== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk7w000d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:50 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KHe2C7027449; Sat, 20 Jul 2024 19:15:49 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puuy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:49 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfF028365; Sat, 20 Jul 2024 19:15:49 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-8; Sat, 20 Jul 2024 19:15:48 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 07/12] iommufd: change_process kernel interface Date: Sat, 20 Jul 2024 12:15:32 -0700 Message-Id: <1721502937-87102-8-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-ORIG-GUID: Mc3aYxEQ_c8TbSfcqTMZxaUfzh2CGVCz X-Proofpoint-GUID: Mc3aYxEQ_c8TbSfcqTMZxaUfzh2CGVCz Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Define IOMMU_IOAS_CHANGE_PROCESS for use by CPR. This interface is preliminary. Signed-off-by: Steve Sistare --- linux-headers/linux/iommufd.h | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/linux-headers/linux/iommufd.h b/linux-headers/linux/iommufd.h index 72e8f4b..568029a 100644 --- a/linux-headers/linux/iommufd.h +++ b/linux-headers/linux/iommufd.h @@ -50,6 +50,7 @@ enum { IOMMUFD_CMD_HWPT_SET_DIRTY_TRACKING, IOMMUFD_CMD_HWPT_GET_DIRTY_BITMAP, IOMMUFD_CMD_HWPT_INVALIDATE, + IOMMUFD_CMD_IOAS_CHANGE_PROCESS, }; /** @@ -692,4 +693,22 @@ struct iommu_hwpt_invalidate { __u32 __reserved; }; #define IOMMU_HWPT_INVALIDATE _IO(IOMMUFD_TYPE, IOMMUFD_CMD_HWPT_INVALIDATE) + +struct iommu_ioas_userspace_map { + __u64 addr_old; + __u64 addr_new; + __u64 size; /* bytes */ +}; + +struct iommu_ioas_change_process { + __u32 size; + __u32 flags; /* must be 0 */ + __u32 n_umap; + __u32 __reserved; /* must be 0 */ + __aligned_u64 umap; +}; + +#define IOMMU_IOAS_CHANGE_PROCESS \ + _IO(IOMMUFD_TYPE, IOMMUFD_CMD_IOAS_CHANGE_PROCESS) + #endif From patchwork Sat Jul 20 19:15:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 32F4AC3DA59 for ; Sat, 20 Jul 2024 19:18:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZA-000786-2x; Sat, 20 Jul 2024 15:16:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYx-0006q7-JX for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYk-0000rL-Sm for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:06 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJAjSN021362; Sat, 20 Jul 2024 19:15:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=DpTciMWL5BAj6n1iFBVHsSIkDF5ym/ImJcxqzxWgfVM=; b= HmApqsn91G1CDYkSxmlgQTxqrGQpck7Zr2TuIl2cqJ7WAW0KIq9DvXIjElG5oiQ1 9LC1OnX7gLhdsbRkhGLCb/gZarT6s5jYutewWkjOJX/+sNTXUgcWTbobnwiPv0Bu E2D9oFVBWXJfjtwna8esOHZxH4b/DONoY2P4BEzVh8tEWVjxZxsjj9bw/RYvleP5 RcV+VMNuPreneGeg0QsSYw8XdzCcK5x9MuPn02nPDId1crGMQhPX9WbH3ee6WP8w pyOvHFhBOtl2l81j5fmnWtEGCFxckNv1HZzbzfWF/RUuZF2KbKyXrbb/cHjRFsDn 5AXeL8FXtoluVWR6eGWRrw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5gr05b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:51 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KHe1JX027443; Sat, 20 Jul 2024 19:15:51 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puvf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:50 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfH028365; Sat, 20 Jul 2024 19:15:50 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-9; Sat, 20 Jul 2024 19:15:50 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 08/12] vfio/iommufd: register container for cpr Date: Sat, 20 Jul 2024 12:15:33 -0700 Message-Id: <1721502937-87102-9-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: 2CudywQa5VHt7u50sWxOlsNDjiJAKXKE X-Proofpoint-ORIG-GUID: 2CudywQa5VHt7u50sWxOlsNDjiJAKXKE Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Register a vfio iommufd container for CPR. Add a blocker if the kernel does not support IOMMU_IOAS_CHANGE_PROCESS. Signed-off-by: Steve Sistare --- backends/iommufd.c | 8 ++++++ hw/vfio/cpr-iommufd.c | 60 +++++++++++++++++++++++++++++++++++++++++++ hw/vfio/iommufd.c | 2 ++ hw/vfio/meson.build | 1 + include/hw/vfio/vfio-common.h | 3 +++ include/sysemu/iommufd.h | 1 + 6 files changed, 75 insertions(+) create mode 100644 hw/vfio/cpr-iommufd.c diff --git a/backends/iommufd.c b/backends/iommufd.c index 4bdbad2..243178e 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -73,6 +73,14 @@ static void iommufd_backend_class_init(ObjectClass *oc, void *data) object_class_property_add_str(oc, "fd", NULL, iommufd_backend_set_fd); } +bool iommufd_change_process_capable(IOMMUFDBackend *be) +{ + struct iommu_ioas_change_process args = {.n_umap = -1}; + + ioctl(be->fd, IOMMU_IOAS_CHANGE_PROCESS, &args); + return (errno != ENOTTY); +} + bool iommufd_backend_connect(IOMMUFDBackend *be, const char *name, Error **errp) { int fd; diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c new file mode 100644 index 0000000..f2e34f4 --- /dev/null +++ b/hw/vfio/cpr-iommufd.c @@ -0,0 +1,60 @@ +/* + * Copyright (c) 2021-2024 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + */ + +#include "qemu/osdep.h" +#include "qapi/error.h" +#include "hw/vfio/vfio-common.h" +#include "migration/blocker.h" +#include "migration/cpr.h" +#include "migration/migration.h" +#include "migration/vmstate.h" +#include "sysemu/iommufd.h" + +#define IOMMUFD_CONTAINER(base) \ + container_of(base, VFIOIOMMUFDContainer, bcontainer) + +static bool vfio_can_cpr_exec(VFIOIOMMUFDContainer *container, Error **errp) +{ + if (!iommufd_change_process_capable(container->be)) { + error_setg(errp, + "VFIO container does not support IOMMU_IOAS_CHANGE_PROCESS"); + return false; + } + return true; +} + +static const VMStateDescription vfio_container_vmstate = { + .name = "vfio-iommufd-container", + .version_id = 0, + .minimum_version_id = 0, + .needed = cpr_needed_for_reuse, + .fields = (VMStateField[]) { + VMSTATE_END_OF_LIST() + } +}; + +bool vfio_iommufd_cpr_register_container(VFIOContainerBase *bcontainer, + Error **errp) +{ + VFIOIOMMUFDContainer *container = IOMMUFD_CONTAINER(bcontainer); + + if (!vfio_can_cpr_exec(container, &bcontainer->cpr_blocker)) { + return migrate_add_blocker_modes(&bcontainer->cpr_blocker, errp, + MIG_MODE_CPR_EXEC, -1) == 0; + } + + vmstate_register(NULL, -1, &vfio_container_vmstate, container); + + return true; +} + +void vfio_iommufd_cpr_unregister_container(VFIOContainerBase *bcontainer) +{ + VFIOIOMMUFDContainer *container = IOMMUFD_CONTAINER(bcontainer); + + vmstate_unregister(NULL, &vfio_container_vmstate, container); +} diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 6d77daa..585bf09 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -632,6 +632,8 @@ static void vfio_iommu_iommufd_class_init(ObjectClass *klass, void *data) vioc->attach_device = iommufd_cdev_attach; vioc->detach_device = iommufd_cdev_detach; vioc->pci_hot_reset = iommufd_cdev_pci_hot_reset; + vioc->cpr_register = vfio_iommufd_cpr_register_container; + vioc->cpr_unregister = vfio_iommufd_cpr_unregister_container; }; static bool hiod_iommufd_vfio_realize(HostIOMMUDevice *hiod, void *opaque, diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index 5487815..998adb5 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -13,6 +13,7 @@ vfio_ss.add(when: 'CONFIG_IOMMUFD', if_true: files( vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'cpr.c', 'cpr-legacy.c', + 'cpr-iommufd.c', 'display.c', 'pci-quirks.c', 'pci.c', diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ec5b7168..8aa02d4 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -247,6 +247,9 @@ bool vfio_legacy_cpr_register_container(VFIOContainerBase *bcontainer, void vfio_legacy_cpr_unregister_container(VFIOContainerBase *bcontainer); bool iommufd_cdev_get_info_iova_range(VFIOIOMMUFDContainer *container, uint32_t ioas_id, Error **errp); +bool vfio_iommufd_cpr_register_container(VFIOContainerBase *bcontainer, + Error **errp); +void vfio_iommufd_cpr_unregister_container(VFIOContainerBase *bcontainer); extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; diff --git a/include/sysemu/iommufd.h b/include/sysemu/iommufd.h index 6955ebd..f80b968 100644 --- a/include/sysemu/iommufd.h +++ b/include/sysemu/iommufd.h @@ -52,6 +52,7 @@ int iommufd_backend_unmap_dma(IOMMUFDBackend *be, uint32_t ioas_id, bool iommufd_backend_get_device_info(IOMMUFDBackend *be, uint32_t devid, uint32_t *type, void *data, uint32_t len, Error **errp); +bool iommufd_change_process_capable(IOMMUFDBackend *be); #define TYPE_HOST_IOMMU_DEVICE_IOMMUFD TYPE_HOST_IOMMU_DEVICE "-iommufd" #endif From patchwork Sat Jul 20 19:15:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8CD90C3DA49 for ; Sat, 20 Jul 2024 19:17:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZA-0007BH-Ls; Sat, 20 Jul 2024 15:16:24 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYx-0006qA-MM for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYk-0000rP-Sl for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:07 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJFiNR017251; Sat, 20 Jul 2024 19:15:53 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=PXLtbP39VoPZ9SgE3nPBLw9XhUEMVA2a59Rjc1/ZxJQ=; b= RTBuJ50/RLvFX5kkE1w1bfnf1Nk23JPep4i6pVRa99r8ZoYefuIuqxFoYF63fwja Oz9I27/bDxOxfSO1cCQppGc6MDNixAvMdi6slHP31h2XAI+4c9b1PBF73txjgSPY GLja6Z56gIyBpfkAvbm+3jb2LkYmB2e/KPMNSZ3x9JNqkQ86FrOfXzN8pX2aD97v DiI8i3G/3lSqECGAb5SXbzToI0Ze9ci4W/cDqxN3A0cKyi4mub2vFvkK5foSLYSw 7nHYb+xPlcBzLrfMDMYa+Zc9xJbu3E174bWZM0k7TeiADYFah8g/OEAZPBSU0NWH 1fzwyyJ3KFRCc+xM516PaA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk7w000f-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:52 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KGUQZE027514; Sat, 20 Jul 2024 19:15:52 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puvx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:52 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfJ028365; Sat, 20 Jul 2024 19:15:51 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-10; Sat, 20 Jul 2024 19:15:51 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 09/12] vfio/iommufd: rebuild device Date: Sat, 20 Jul 2024 12:15:34 -0700 Message-Id: <1721502937-87102-10-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-ORIG-GUID: seeqehVg5KQJEa7glOrPghQDQuQg9jVz X-Proofpoint-GUID: seeqehVg5KQJEa7glOrPghQDQuQg9jVz Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Rebuild userland device state after CPR. During vfio_realize, skip all ioctls that configure the device, as it was already configured in old QEMU, and we preserved the device descriptor. Preserve the ioas_id in vmstate. Because we skip ioctl's, it is not needed at realize time. However, we do need to gather range info, so defer the call to iommufd_cdev_get_info_iova_range to a post_load handler, at which time the ioas_id is known. Registering the vfio_memory_listener causes spurious calls to map and unmap DMA, as devices are created and the address space is built. This memory was already already mapped by the device, so suppress map and unmap during CPR -- eg, if the reused flag is set. Clear the reused flag in the post_load handler. Signed-off-by: Steve Sistare --- backends/iommufd.c | 8 ++++++++ hw/vfio/cpr-iommufd.c | 24 ++++++++++++++++++++++++ hw/vfio/iommufd.c | 14 +++++++++++++- 3 files changed, 45 insertions(+), 1 deletion(-) diff --git a/backends/iommufd.c b/backends/iommufd.c index 243178e..86fd9db 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -172,6 +172,10 @@ int iommufd_backend_map_dma(IOMMUFDBackend *be, uint32_t ioas_id, hwaddr iova, .length = size, }; + if (be->reused) { + return 0; + } + if (!readonly) { map.flags |= IOMMU_IOAS_MAP_WRITEABLE; } @@ -203,6 +207,10 @@ int iommufd_backend_unmap_dma(IOMMUFDBackend *be, uint32_t ioas_id, .length = size, }; + if (be->reused) { + return 0; + } + ret = ioctl(fd, IOMMU_IOAS_UNMAP, &unmap); /* * IOMMUFD takes mapping as some kind of object, unmapping diff --git a/hw/vfio/cpr-iommufd.c b/hw/vfio/cpr-iommufd.c index f2e34f4..c38485a 100644 --- a/hw/vfio/cpr-iommufd.c +++ b/hw/vfio/cpr-iommufd.c @@ -27,12 +27,36 @@ static bool vfio_can_cpr_exec(VFIOIOMMUFDContainer *container, Error **errp) return true; } +static int vfio_container_post_load(void *opaque, int version_id) +{ + VFIOIOMMUFDContainer *container = opaque; + VFIOContainerBase *bcontainer = &container->bcontainer; + VFIODevice *vbasedev; + Error *err = NULL; + uint32_t ioas_id = container->ioas_id; + + if (!iommufd_cdev_get_info_iova_range(container, ioas_id, &err)) { + error_report_err(err); + return -1; + } + + bcontainer->reused = false; + QLIST_FOREACH(vbasedev, &bcontainer->device_list, container_next) { + vbasedev->reused = false; + } + container->be->reused = false; + + return 0; +} + static const VMStateDescription vfio_container_vmstate = { .name = "vfio-iommufd-container", .version_id = 0, .minimum_version_id = 0, + .post_load = vfio_container_post_load, .needed = cpr_needed_for_reuse, .fields = (VMStateField[]) { + VMSTATE_UINT32(ioas_id, VFIOIOMMUFDContainer), VMSTATE_END_OF_LIST() } }; diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 585bf09..186edc7 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -357,6 +357,11 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, } } + if (vbasedev->reused) { + ioas_id = -1; /* ioas_id will be sent in vmstate */ + goto skip_ioas_alloc; + } + /* Need to allocate a new dedicated container */ if (!iommufd_backend_alloc_ioas(vbasedev->iommufd, &ioas_id, errp)) { goto err_alloc_ioas; @@ -364,6 +369,7 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, trace_iommufd_cdev_alloc_ioas(vbasedev->iommufd->fd, ioas_id); +skip_ioas_alloc: container = VFIO_IOMMU_IOMMUFD(object_new(TYPE_VFIO_IOMMU_IOMMUFD)); container->be = vbasedev->iommufd; container->ioas_id = ioas_id; @@ -371,7 +377,8 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, bcontainer = &container->bcontainer; vfio_address_space_insert(space, bcontainer); - if (!iommufd_cdev_attach_container(vbasedev, container, errp)) { + if (!vbasedev->reused && + !iommufd_cdev_attach_container(vbasedev, container, errp)) { goto err_attach_container; } @@ -380,6 +387,10 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, goto err_discard_disable; } + if (vbasedev->reused) { + goto skip_info; + } + if (!iommufd_cdev_get_info_iova_range(container, ioas_id, &err)) { error_append_hint(&err, "Fallback to default 64bit IOVA range and 4K page size\n"); @@ -388,6 +399,7 @@ static bool iommufd_cdev_attach(const char *name, VFIODevice *vbasedev, bcontainer->pgsizes = qemu_real_host_page_size(); } +skip_info: bcontainer->listener = vfio_memory_listener; memory_listener_register(&bcontainer->listener, bcontainer->space->as); From patchwork Sat Jul 20 19:15:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56CABC3DA59 for ; Sat, 20 Jul 2024 19:17:25 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZ9-00074N-99; Sat, 20 Jul 2024 15:16:23 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYz-0006qE-FQ for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:15 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFYv-0000rZ-3P for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:12 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJAUWm021897; Sat, 20 Jul 2024 19:15:54 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=KEtu+Nxv6EH7Q/6YhuuppatukClo9brblJ9w6R7zxOU=; b= JubR34CvCUj+4G4NyrI5TXEht4YWjiiHsdxcLLBrGN/tlgTeXOYkRZ2H3BKIiL6Q H6B93N36aVJy1nrDf7+JQbP1tVrgpSy1kUwRiA2XEtTPp50/KNrraU8LsPJ4W7/6 cngv0s6zJglnHRPUxXKm/dRU6wbf4OfgC1JYhmdhhzEtvQJktPAkA/0cT+2X+OEp 8vN0y7Ci76TQph7XBlABwZq3wjxXXzns/KM54IzNKCVq0YvaGTHXeiB16dJCveTM gEORix8Gxg33dSuJi4AgrpJ+qRTtJcw+5owYFEY4EkO8UmZrk6dd9QiJXJgsh0Z0 NKx5wgRkqYuNLRHOC8+YzQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5jr04j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:54 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KIfsYt027542; Sat, 20 Jul 2024 19:15:53 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puwa-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:53 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfL028365; Sat, 20 Jul 2024 19:15:53 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-11; Sat, 20 Jul 2024 19:15:53 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 10/12] migration/ram: old host address Date: Sat, 20 Jul 2024 12:15:35 -0700 Message-Id: <1721502937-87102-11-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: iEqILvLT-oYpMT69ERvnhgoX8cj0UfEh X-Proofpoint-ORIG-GUID: iEqILvLT-oYpMT69ERvnhgoX8cj0UfEh Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Remember the RAMBlock host address as host_old during migration, for use by CPR. The iommufd interface to update the virtual address of DMA mappings requires it. Signed-off-by: Steve Sistare --- hw/core/machine.c | 6 ++++++ include/exec/ramblock.h | 1 + migration/migration.h | 2 ++ migration/options.c | 2 ++ migration/ram.c | 7 +++++++ 5 files changed, 18 insertions(+) diff --git a/hw/core/machine.c b/hw/core/machine.c index 9676953..0ac16b8 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -35,6 +35,12 @@ #include "hw/virtio/virtio-iommu.h" #include "audio/audio.h" +/* TBD: register hw_compat_9_1 with machines */ +GlobalProperty hw_compat_9_1[] = { + { "migration", "send-host-old", "off"}, +}; +const size_t hw_compat_9_1_len = G_N_ELEMENTS(hw_compat_9_1); + GlobalProperty hw_compat_9_0[] = { {"arm-cpu", "backcompat-cntfrq", "true" }, {"scsi-disk-base", "migrate-emulated-scsi-request", "false" }, diff --git a/include/exec/ramblock.h b/include/exec/ramblock.h index 64484cd..8f1c535 100644 --- a/include/exec/ramblock.h +++ b/include/exec/ramblock.h @@ -28,6 +28,7 @@ struct RAMBlock { struct rcu_head rcu; struct MemoryRegion *mr; uint8_t *host; + uint64_t host_old; uint8_t *colo_cache; /* For colo, VM's ram cache */ ram_addr_t offset; ram_addr_t used_length; diff --git a/migration/migration.h b/migration/migration.h index 38aa140..b5e3151 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -442,6 +442,8 @@ struct MigrationState { */ uint8_t clear_bitmap_shift; + bool send_host_old; + /* * This save hostname when out-going migration starts */ diff --git a/migration/options.c b/migration/options.c index 7526f9f..197cb86 100644 --- a/migration/options.c +++ b/migration/options.c @@ -92,6 +92,8 @@ Property migration_properties[] = { clear_bitmap_shift, CLEAR_BITMAP_SHIFT_DEFAULT), DEFINE_PROP_BOOL("x-preempt-pre-7-2", MigrationState, preempt_pre_7_2, false), + DEFINE_PROP_BOOL("send-host-old", MigrationState, + send_host_old, true), /* Migration parameters */ DEFINE_PROP_UINT8("x-throttle-trigger-threshold", MigrationState, diff --git a/migration/ram.c b/migration/ram.c index 1e1e05e..8644917 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -3030,6 +3030,9 @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp) qemu_put_byte(f, strlen(block->idstr)); qemu_put_buffer(f, (uint8_t *)block->idstr, strlen(block->idstr)); qemu_put_be64(f, block->used_length); + if (migrate_get_current()->send_host_old) { + qemu_put_be64(f, (uint64_t)block->host); + } if (migrate_postcopy_ram() && block->page_size != max_hg_page_size) { qemu_put_be64(f, block->page_size); @@ -4021,6 +4024,10 @@ static int parse_ramblock(QEMUFile *f, RAMBlock *block, ram_addr_t length) assert(block); + if (migrate_get_current()->send_host_old) { + block->host_old = qemu_get_be64(f); + } + if (migrate_mapped_ram()) { parse_ramblock_mapped_ram(f, block, length, &local_err); if (local_err) { From patchwork Sat Jul 20 19:15:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737888 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0D2E5C3DA49 for ; Sat, 20 Jul 2024 19:18:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZE-0007Qs-DS; Sat, 20 Jul 2024 15:16:28 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFZ5-0006vb-Bw for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:21 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFZ1-0000rd-Ai for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:18 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJA1iN020362; Sat, 20 Jul 2024 19:15:56 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=OdD3Vh8VrBj3S8FDaAY/ddgO1Ru7HGoZ8kU+j7nwgqM=; b= MkGw3BT8JObh76hShs77BNb2yezr3TSl+Ix0Ox8vLM4MnuvKS+AeuEjRLHk/Q9lS PF49nCSodj1HFXx1FHElJc/7dbC9dOxJcWhfio8Po4foLa5vzWTCrtRlUzaqFQJb eqgSKguaEIyvw/cEs+gFRs9xWQX4ECKOS2wRB5dWZx7IMVXcV9cCOS20G/CL9aGE XHFTrWtDvORNhOxFYK6J/PjFILbz0JMLX9u0Cn7LAUwVJBtgFoxOqNTvDAx7HXef 9VR6wyEBwBDEeqq3/aLZBfKHqZO8q+/6rgZr+RViXLWHopXX5RFiYlCjjWeyuID6 zze6xXnQJLAeSgTNWKvycQ== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk5gr05c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:55 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KI0Q70027478; Sat, 20 Jul 2024 19:15:55 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5puwh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:55 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfN028365; Sat, 20 Jul 2024 19:15:54 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-12; Sat, 20 Jul 2024 19:15:54 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 11/12] iommufd: update DMA virtual addresses Date: Sat, 20 Jul 2024 12:15:36 -0700 Message-Id: <1721502937-87102-12-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-GUID: ZoasZSdLt6aXDvSLHjc8bEeQB4d9So3o X-Proofpoint-ORIG-GUID: ZoasZSdLt6aXDvSLHjc8bEeQB4d9So3o Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Register a vmstate post_load handler to call IOMMU_IOAS_CHANGE_PROCESS and update the virtual address of all DMA mappings after CPR. Signed-off-by: Steve Sistare --- backends/iommufd.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 80 insertions(+), 1 deletion(-) diff --git a/backends/iommufd.c b/backends/iommufd.c index 86fd9db..2e72b6f 100644 --- a/backends/iommufd.c +++ b/backends/iommufd.c @@ -17,7 +17,9 @@ #include "qom/object_interfaces.h" #include "qemu/error-report.h" #include "migration/cpr.h" +#include "migration/vmstate.h" #include "monitor/monitor.h" +#include "exec/ramblock.h" #include "trace.h" #include #include @@ -81,6 +83,83 @@ bool iommufd_change_process_capable(IOMMUFDBackend *be) return (errno != ENOTTY); } +static int iommufd_change_process(IOMMUFDBackend *be, + struct iommu_ioas_change_process *args) +{ + int ret, fd = be->fd; + + ret = ioctl(fd, IOMMU_IOAS_CHANGE_PROCESS, args); + if (ret) { + ret = -errno; + error_report("IOMMU_IOAS_CHANGE_PROCESS failed: %m"); + } + return ret; +} + +static int count_umap(RAMBlock *rb, void *opaque) +{ + if (qemu_ram_is_migratable(rb)) { + (*(int *)opaque)++; + } + return 0; +} + +static int fill_umap(RAMBlock *rb, void *opaque) +{ + if (qemu_ram_is_migratable(rb)) { + struct iommu_ioas_change_process *args = opaque; + struct iommu_ioas_userspace_map *umap = (void *)args->umap; + int i = args->n_umap++; + + assert(rb->host_old && rb->host); + umap[i].addr_old = (__u64)rb->host_old; + umap[i].addr_new = (__u64)rb->host; + umap[i].size = rb->max_length; + } + return 0; +} + +static int cmp_umap(const void *elem1, const void *elem2) +{ + const struct iommu_ioas_userspace_map *e1 = elem1; + const struct iommu_ioas_userspace_map *e2 = elem2; + + return (e1->addr_old < e2->addr_old) ? -1 : + (e1->addr_old > e2->addr_old); +} + +static int iommufd_cpr_post_load(void *opaque, int version_id) +{ + IOMMUFDBackend *be = opaque; + struct iommu_ioas_change_process args = { + .size = sizeof(args), + .flags = 0, + .n_umap = 0, + .umap = 0, + }; + int n = 0; + g_autofree struct iommu_ioas_userspace_map *umap = NULL; + + RCU_READ_LOCK_GUARD(); + qemu_ram_foreach_block(count_umap, &n); + umap = g_malloc_n(n, sizeof(*umap)); + args.umap = (__u64)umap; + qemu_ram_foreach_block(fill_umap, &args); + qsort(umap, args.n_umap, sizeof(*umap), cmp_umap); + return iommufd_change_process(be, &args); +} + +static const VMStateDescription iommufd_cpr_vmstate = { + .name = "iommufd", + .version_id = 0, + .minimum_version_id = 0, + .post_load = iommufd_cpr_post_load, + .needed = cpr_needed_for_reuse, + .fields = (VMStateField[]) { + VMSTATE_END_OF_LIST() + } +}; + bool iommufd_backend_connect(IOMMUFDBackend *be, const char *name, Error **errp) { int fd; @@ -100,7 +179,7 @@ bool iommufd_backend_connect(IOMMUFDBackend *be, const char *name, Error **errp) be->fd = fd; } be->users++; - + vmstate_register(NULL, -1, &iommufd_cpr_vmstate, be); trace_iommufd_backend_connect(be->fd, be->owned, be->users); return true; } From patchwork Sat Jul 20 19:15:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Sistare X-Patchwork-Id: 13737880 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5C87BC3DA49 for ; Sat, 20 Jul 2024 19:16:53 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1sVFZC-0007Jj-Mb; Sat, 20 Jul 2024 15:16:26 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFZ5-0006vc-Bu for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:21 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1sVFZ2-0000ri-4T for qemu-devel@nongnu.org; Sat, 20 Jul 2024 15:16:18 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 46KJF2Il016270; Sat, 20 Jul 2024 19:15:57 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h= from:to:cc:subject:date:message-id:in-reply-to:references; s= corp-2023-11-20; bh=PrZpKjn22NnTRu3L/ZRkY8HK9ZTK3ILIUp25NlJXXPY=; b= RyXbMPMglk9zbN+z45Qpfr+qZJBsRw1tylEZFJE9BIifzQ81A6vxvtnm98VFY2KW il1QPw4Ni+LRDHDU5Pg/Vp8PJliTQt8mJ3sYITUelAepT2MZ89QO4WWpSOsR/p/M dLh5ip4FiMpQPzhLhsnAQxBmG5sggLlrwnpJGrFfaJpGPVl9wUK1k2C8SQz+h3mc yZmKcyTp+7z4SGng/7sU5uub+BfNpobjOc6mMsRT+nVl7RuByWwq5tf2rkpddJxI Ik0FTIiIPIPiQ9zv7huBnnNaeIlg6X4WBESJilouReI+OVX1M4dUi+1XRqICr7sD 3sLMNdwWPm8bIIx9xFvneA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 40gk7w000g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:57 +0000 (GMT) Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 46KI06tt027423; Sat, 20 Jul 2024 19:15:56 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 40g3p5pux3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Sat, 20 Jul 2024 19:15:56 +0000 Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 46KJFdfP028365; Sat, 20 Jul 2024 19:15:56 GMT Received: from ca-dev63.us.oracle.com (ca-dev63.us.oracle.com [10.211.8.221]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 40g3p5pusv-13; Sat, 20 Jul 2024 19:15:55 +0000 From: Steve Sistare To: qemu-devel@nongnu.org Cc: Yi Liu , Eric Auger , Zhenzhong Duan , Alex Williamson , Cedric Le Goater , "Michael S. Tsirkin" , Peter Xu , Fabiano Rosas , Philippe Mathieu-Daude , David Hildenbrand , Steve Sistare Subject: [RFC V1 12/12] vfio: mdev blocker Date: Sat, 20 Jul 2024 12:15:37 -0700 Message-Id: <1721502937-87102-13-git-send-email-steven.sistare@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> References: <1721502937-87102-1-git-send-email-steven.sistare@oracle.com> X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1039,Hydra:6.0.680,FMLib:17.12.28.16 definitions=2024-07-20_16,2024-07-18_01,2024-05-17_01 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 bulkscore=0 phishscore=0 suspectscore=0 adultscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2407110000 definitions=main-2407200140 X-Proofpoint-ORIG-GUID: rF5Tpq8KAvljSnaTMrZxdy-CV-FxPcgV X-Proofpoint-GUID: rF5Tpq8KAvljSnaTMrZxdy-CV-FxPcgV Received-SPF: pass client-ip=205.220.177.32; envelope-from=steven.sistare@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Block CPR if the container has any mdevs (mediated devices). CPR is not supported for legacy containers and mdevs. It will be supported for iommufd containers with mdevs in a future patch. Signed-off-by: Steve Sistare --- hw/vfio/pci.c | 10 ++++++++++ include/hw/vfio/vfio-common.h | 1 + include/hw/vfio/vfio-container-base.h | 2 ++ 3 files changed, 13 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index b5e7592..872b07c 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3100,6 +3100,13 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) is_mdev = subsys && (strcmp(subsys, "/sys/bus/mdev") == 0); free(subsys); + vbasedev->is_mdev = is_mdev; + if (is_mdev && !vbasedev->bcontainer->n_mdev++) { + error_setg(&vbasedev->bcontainer->cpr_mdev_blocker, + "CPR does not support vfio mdev"); + migrate_add_blocker_modes(&vbasedev->bcontainer->cpr_mdev_blocker, + &error_fatal, MIG_MODE_CPR_EXEC, -1); + } trace_vfio_mdev(vbasedev->name, is_mdev); if (vbasedev->ram_block_discard_allowed && !is_mdev) { @@ -3387,6 +3394,9 @@ static void vfio_exitfn(PCIDevice *pdev) vfio_teardown_msi(vdev); vfio_pci_disable_rp_atomics(vdev); vfio_bars_exit(vdev); + if (vbasedev->is_mdev && !--vbasedev->bcontainer->n_mdev) { + migrate_del_blocker(&vbasedev->bcontainer->cpr_mdev_blocker); + } vfio_migration_exit(vbasedev); pci_device_unset_iommu_device(pdev); } diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 8aa02d4..342c40f 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -133,6 +133,7 @@ typedef struct VFIODevice { OnOffAuto pre_copy_dirty_page_tracking; bool dirty_pages_supported; bool dirty_tracking; + bool is_mdev; HostIOMMUDevice *hiod; int devid; IOMMUFDBackend *iommufd; diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index f8b7b26..e29cbb8 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -45,6 +45,7 @@ typedef struct VFIOContainerBase { uint64_t max_dirty_bitmap_size; unsigned long pgsizes; unsigned int dma_max_mappings; + unsigned int n_mdev; bool dirty_pages_supported; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIORamDiscardListener) vrdl_list; @@ -54,6 +55,7 @@ typedef struct VFIOContainerBase { NotifierWithReturn cpr_reboot_notifier; NotifierWithReturn cpr_exec_notifier; Error *cpr_blocker; + Error *cpr_mdev_blocker; } VFIOContainerBase; typedef struct VFIOGuestIOMMU {