From patchwork Wed Feb 19 14:48:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982382 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 05E4DC3DA4A for ; Wed, 19 Feb 2025 14:50:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOD-0003G0-8Z; Wed, 19 Feb 2025 09:49:29 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklO8-0003FE-Nf; Wed, 19 Feb 2025 09:49:25 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklO6-000796-Ch; Wed, 19 Feb 2025 09:49:24 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBqX0u018818; Wed, 19 Feb 2025 06:49:18 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=WylXG2VuACek6zvTWwOOVI3eu3kl1mG5PKZWuLrkI 4w=; b=QdTmCqP6OTH6b8zEW7RyTBLqu12g5i7wDOOvghvivrSPckP3xig2WJ1cU u070DuQP1m9ncLnqmPTCjQoRJLD7b/dbbaeWacSCt63+NfHQCKYWTHGng3zDV2++ HxjNj0cdabyssy2WNsQS1IHZqj2fw0iUJ03xjPKxAEA3YYHONEzn/eUgZlYGEV38 G7fVvioVsyPmYvhAFU4pz1DXGDp8K5Qr6lqdoGYh4uM6Xf1HIBq363h4PjEDkNJU AWVPMsVJDrE9DUcl6x2Pjqf8r2cr73MWB9yeyyxzwsDcU9KsYzr+v0k1/TlqvOwP idNTSmZut6EY1tcxk9/oEl/OORgQQ== Received: from bn1pr04cu002.outbound.protection.outlook.com (mail-eastus2azlp17010019.outbound.protection.outlook.com [40.93.12.19]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxqq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:17 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=lxwYFMQCq6LYSMbz5MMXeMvBlcgotoPBmaI68Jf27LuMjAVXTED8pUIOkMCiBIN6tW0bC/87LxXHYD9UMVEeFyIiuwu9vaTr0xf5HFOT2S4TB+iLpOwMl3ffbKTyqY9KUUWSa3AlbPZKOJRDxm504jnpY9F8gOlpmWoNq/vHTGVbj90IBBcE/AO9oE2iZctlaWYWNNdAPVrIGe8/b7AZkZHgilu6S2zFDv0pFV7aeAwadnM0cFsfEYeLzxoLsLT2FJOk6FhCiNjEhUkQeFqd0otEetdAFGv2Unrllj/fPT4AWQcB3pDUun7WqvFTC8NNwqTFrtL+vIeW3jtf4QEyLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WylXG2VuACek6zvTWwOOVI3eu3kl1mG5PKZWuLrkI4w=; b=g7BUtPF9qkyZvkd7qJk12H2EvD8AkTxnrmMa7uN4+M7NhlYSOk25fj5TRtr4QdJ955IyeZYZpqkZhUShQBbWIkBZBCQr2gVW8uaOtAw8HbUGT2+IBgwpAl5UDeZwQ9k21OgMHah7zmfo0gFT7FatS5L5VA9Ie71U4tdbEVRFnVVrWKsskH9mZSC9W7lwKZGfXQDml1ossjJMuCAcSQVyw6hqgn8vQl9CTI+vDNznCK1hwaMrbJ+gHvx9cvYlCf3ZmqV2p8GcNanghoFcXoWnsGkt1KWpHLcdoqrK9zLd/TVLtmrH+qMQp1KYHH0PxMwRXQVmb59jYPUGRHKXIGK5iw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WylXG2VuACek6zvTWwOOVI3eu3kl1mG5PKZWuLrkI4w=; b=HmBbXfN8CVB1QVYNcLHIxM0E35DZA3eVKTi491umHui8jBSsHwq+RFdEfIsb/6hs5aGl5SyWhE2YJ7HtDzzPx1Z356eu62tu4uJ1n7Wll2pGPkZbeaKpujbB127eOY7cyGowrYH1Kz649md/3BXTfpQjSPAVnqQU9Aw7s9+l0sI1lgbw0zbsjCSc4TbGPwvr6noD3YrAHuSSsW1gglOo9PtWEJ6NXEa/PMBtmWtkE2z5bzyDGvknLW5FM6fR7kZ4wBbF3GA/4WQbBls4V/RB/m27u/KU+wvyuI+28WIkQZPysP91usJaBxnDYkrnR6e3ceZBAPwQ1H6A7Dq3FtyVCA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:15 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:15 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 01/28] vfio/container: pass MemoryRegion to DMA operations Date: Wed, 19 Feb 2025 15:48:31 +0100 Message-Id: <20250219144858.266455-2-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 51ec3733-7726-475e-644c-08dd50f4946f x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: RVHBoTMQ1Eem3eR/A9d2gauubU/g8Ay7wBEWeIQZpWnFpksV2N+VNB34IGgcKvsIS5ZUvxNJkoMKBrMJJIIza++cF6s+WX92QDFdlMUz8R9NZ9M0/T1C9qTmhwk3UGsPAwLSyEstD3vJWNaSGHrzEvF9dFngh0EJMuSH/DPEviDyJPRqOrkFhqjmbLx4oirddFh033xTO+UcUFbAgTcK6F98S6rqcdaQE9WsUpVGiXtY4r6Wd5C0VhTTZKAknQTLifzMdAAGMnM9ga6V2CXhRZQls8h8pCtnRC2duJ939PTcmB1ewu4W/EzBaWCKWlpadXd2LXXn8JlqawLdh1XoktAuZgLWYlYqTK6yu6UUV73aZqPHJSDW/zOaSXYClbIBmVanPJyDlMGW1QiknGjF0AGANnfSKsfAnY+QK+ul/iqWnPRIUewnIpij9EoqPVC6R7fWqmc8Z6hem5KPUUx+Pqb0qUS8KDlozAEpG8keNFXOrzrmLROorisxxUT4hOfykCUqyJ+Fn8G7g+aSqigY2Bq8O+jE+meAA3XYgCMvBe4e8Z4/kmGW6DGZJUowCTFOPNoJXattbG/Yu9JNV7oISP41P6Krrrbm56Zq9j+43jLP8XriyRfB2oRjhsBcrFnlDB3OWpHOeKWQA4buYWIwCLSO/ccAYsk9509C719rZvR689OaLXIIrns3YqdcfQYlEeXUt5PnWZJRyEDM+KO88hsDJ/uIs4XaQjdO+bfb2T+xrQwbkO/U7Qlx7UQV8jBlRwZt59hUKkJvsqE8iMTyPG9QmNT+i8f/G53tNW5H6sAk7Gg2wODmDgKbf3wSTsxhvVsjmumsKYNwpzD5U1IIKe5ZT9TcF1yeCJ/7VVb02lCekJ/T3l2EWoJxTCt6XUWptaHIQDunFVrYP0HYV+EBjm2vk2oqeYZDgJSgL2OhkFQqgc7xOZYYbUS0CcGTMNrFx6HJHgVp7e/Pfyh4+7KWav5I/ZL0bcY/hYckIe83QWsgqAqEoatjz+OQKCyNzTYUmdc+f107xq89tjNsKlhDbrJzmaLT+3TxUqrPe6jMyWansH3yK7wUdK8ZMHkYol+iiMxCgbhZwRS5ETJyHkpusG5EL1mVuh/cB7Us+Ri32hdbip2ZwaAZLL6B+jgUj0OWbmXTClW45fVQP+u2SQ19RTkK82F/3nIhAPGBcodAv+KnMw5qM22AmDICQNUVU2HIZ1cSL2S3FGIe54sM87ZytdmpJvZWg5nkPbMulM2/kJdUjgSWSr8GhPxe/7wmSSLM3y19GEBvylQD5sTn2WJyRP5eK+PGBk3J44hrkJguxEPA0X09itpSfyfWF1T26UzH2cWJiHzhG10AmmN7mc5LZZatVGddBbZG/8ko6XmYuE2wDWYLxFWveKxT+d9Gw1ZX X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: VUirwcQNfvwXnJC5+wq8/EL8VYBithYK1fWD3GCz9vR29Edcpy9+BFSaW6hT8tzu3nbhT/Cuw8G9X3xt8Q64SnBEHtmCvB+wZl9lF8qZiw0tLhQbSf4DKBwc0Nq/f9cdX+zNF9si6L4/1MG4ugf5ekwILw9EGbNjgu410tGzEaxw7kFTeGROJFpOZbJzmPCMmd8FDoHvAa/DmM52hq18CpUChPK6sCQsHWc8uuhENAtkIZNJBqAfB73d9uhSDEhL+3FJJ6srorSA4LEW3SAR/6qzVRjLb8Pkv7MUAZ4p3SIEDiz61TsjJ0s0JqHgGAOG4ccMRAbHduSk7rSDM78Gt3ssfPQjBU9p0C1t4KthkS13hU9f0GI8I7g5LOSzrlsfC1a1xCmnrokmGJZUh1RqfbzXCYQQwWkh/oYLOSjhJgkTVf+dFxWcaChr0CKekjrYZDmqLh9amGCtvoU84gZqc7CootZtr25csfSjVbDhV6vNw9dcZ7ZphYuZ4pHlQ1Jc0ZKcuQ7BPhxafzvnJAFwOpETkuZmvsakR83/8nDanRKEQwed/FLZPa6rjLx8B4A+hF1/w81YhXVpBo8rHUG5/cL2V4YExT1TP5voIpJiMaSuSQDmwKsGbyFrVMyaU2m/CmOAQY7YZ2h6xL6YKH2iJgGqguGSwV+t0hI9uGNP8+39dk4AyqzALj8d1JLQ/1+Tpct5wYVZMQRPb5gb+kBKZhXUlLu/By1M3LoLSoJyRz+XyMMA6BB90RX/vYRG/ncHuxoHxgc01CrvSoOXWNp0uxpcsyk8DaoF6qwWFm7nL3Zn1thtute83Lw5b08uPERZskb6OgdbkHYK40leLi/AEq4y1LyL1x+CLTzJnjHOlNEJRpVP4Il5F1/uXD3TqZZBTN6BabDs6d5hoBVStKsNK/1PMF8UZOXdqXK1wjIevJQR+fipA88Gy3AqMwtRDFw3nLxrDXlOuRWfNgNiWSVtofmU7k9Mf3Tux0W/FIw11S9NDSAnO5xP3Sy54ciuUX3RBJLOqXpOcsXKUVhX5s14Cy4OVU+WtzjpWv0EQ0QmslQ53Ty1OjMWik4unpMGYlb8PN43zbe0ju6WwK4XcQKOD+a00H7HTgpiW8OdDSrYLj2TAYQQLBRbwbPJbXryiS9F8/ef2a6ngpa6OnjJbuRA/RZC/9MeGyOqWsqRKPjniKX/A5xWMX+kE5GP877CR/DNrIECweSCJDUfv56NtdPT0mSktINhxJzp0Q+z9moB4SUiJhuhg0Fx+eUoiVKXzOrpDyMF9ggcG/mXZx38/dvdlRCi1zf+fBsbTwYK+qw6Yl53MzazZ5F0pwPAqPja0qQMdL1t+O506KWEyr0tMQ/Vtp2JPOjVHmvwgVd1GtrWJPzBU7ls47QyD5FJYBw3H5QDPGDlEjOcuDG6YELz32QrMpm1VC0iI7aGkqsP9UmwsNyhmTWR2LFQPnvNKiloL7moxWJZP+j8mHtbHBvePX/iLThWzzS1HS1ikslpeWrEJTobZXjrEuu8suya7cXZe883i5zG87dU9Div56aVjkAZ+YKKBMRL4H7Hqrj6xSjf2d65kKm4YY/hcQjgDNT3PkqH X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 51ec3733-7726-475e-644c-08dd50f4946f X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:15.0885 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yJmdVgZ0yJhScNK+i5FHQIQhHgWQg4FqinbjUq40t5V54LRAAfiryZFtOPdxxxa76lPbnjpH+Prr0c7+nJYLCg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: UjCI9QZtG5i5eKJPhovyrtaK7bLy7wVw X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5ef6d cx=c_pps a=b4GTUaH8bIYRMqSiyTmTYA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=DMMBb2U3yOgPCP8cz6YA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: UjCI9QZtG5i5eKJPhovyrtaK7bLy7wVw X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Pass through the MemoryRegion to DMA operation handlers of vfio containers. The vfio-user container will need this later. Originally-by: John Johnson Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John Levon --- hw/vfio/common.c | 17 ++++++++++------- hw/vfio/container-base.c | 4 ++-- hw/vfio/container.c | 3 ++- hw/vfio/iommufd.c | 3 ++- hw/virtio/vhost-vdpa.c | 2 +- include/exec/memory.h | 4 +++- include/hw/vfio/vfio-container-base.h | 4 ++-- system/memory.c | 7 ++++++- 8 files changed, 28 insertions(+), 16 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index abbdc56b6d..8d3d425c63 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -248,12 +248,12 @@ static bool vfio_listener_skipped_section(MemoryRegionSection *section) /* Called with rcu_read_lock held. */ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, ram_addr_t *ram_addr, bool *read_only, - Error **errp) + MemoryRegion **mrp, Error **errp) { bool ret, mr_has_discard_manager; ret = memory_get_xlat_addr(iotlb, vaddr, ram_addr, read_only, - &mr_has_discard_manager, errp); + &mr_has_discard_manager, mrp, errp); if (ret && mr_has_discard_manager) { /* * Malicious VMs might trigger discarding of IOMMU-mapped memory. The @@ -281,6 +281,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) VFIOGuestIOMMU *giommu = container_of(n, VFIOGuestIOMMU, n); VFIOContainerBase *bcontainer = giommu->bcontainer; hwaddr iova = iotlb->iova + giommu->iommu_offset; + MemoryRegion *mrp; void *vaddr; int ret; Error *local_err = NULL; @@ -300,7 +301,8 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) { bool read_only; - if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, &local_err)) { + if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, &mrp, + &local_err)) { error_report_err(local_err); goto out; } @@ -313,7 +315,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) */ ret = vfio_container_dma_map(bcontainer, iova, iotlb->addr_mask + 1, vaddr, - read_only); + read_only, mrp); if (ret) { error_report("vfio_container_dma_map(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx", %p) = %d (%s)", @@ -378,7 +380,7 @@ static int vfio_ram_discard_notify_populate(RamDiscardListener *rdl, vaddr = memory_region_get_ram_ptr(section->mr) + start; ret = vfio_container_dma_map(bcontainer, iova, next - start, - vaddr, section->readonly); + vaddr, section->readonly, section->mr); if (ret) { /* Rollback */ vfio_ram_discard_notify_discard(rdl, section); @@ -675,7 +677,7 @@ static void vfio_listener_region_add(MemoryListener *listener, } ret = vfio_container_dma_map(bcontainer, iova, int128_get64(llsize), - vaddr, section->readonly); + vaddr, section->readonly, section->mr); if (ret) { error_setg(&err, "vfio_container_dma_map(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx", %p) = %d (%s)", @@ -1232,7 +1234,8 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) } rcu_read_lock(); - if (!vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL, &local_err)) { + if (!vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL, NULL, + &local_err)) { error_report_err(local_err); goto out_unlock; } diff --git a/hw/vfio/container-base.c b/hw/vfio/container-base.c index 749a3fd29d..5e0c9700d9 100644 --- a/hw/vfio/container-base.c +++ b/hw/vfio/container-base.c @@ -17,12 +17,12 @@ int vfio_container_dma_map(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - void *vaddr, bool readonly) + void *vaddr, bool readonly, MemoryRegion *mrp) { VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); g_assert(vioc->dma_map); - return vioc->dma_map(bcontainer, iova, size, vaddr, readonly); + return vioc->dma_map(bcontainer, iova, size, vaddr, readonly, mrp); } int vfio_container_dma_unmap(VFIOContainerBase *bcontainer, diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 7c57bdd27b..0db0055f39 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -175,7 +175,8 @@ static int vfio_legacy_dma_unmap(const VFIOContainerBase *bcontainer, } static int vfio_legacy_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, - ram_addr_t size, void *vaddr, bool readonly) + ram_addr_t size, void *vaddr, bool readonly, + MemoryRegion *mrp) { const VFIOContainer *container = container_of(bcontainer, VFIOContainer, bcontainer); diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index df61edffc0..583b063707 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -28,7 +28,8 @@ #include "exec/ram_addr.h" static int iommufd_cdev_map(const VFIOContainerBase *bcontainer, hwaddr iova, - ram_addr_t size, void *vaddr, bool readonly) + ram_addr_t size, void *vaddr, bool readonly, + MemoryRegion *mrp) { const VFIOIOMMUFDContainer *container = container_of(bcontainer, VFIOIOMMUFDContainer, bcontainer); diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c index 3cdaa12ed5..a1866bb396 100644 --- a/hw/virtio/vhost-vdpa.c +++ b/hw/virtio/vhost-vdpa.c @@ -228,7 +228,7 @@ static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) { bool read_only; - if (!memory_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, NULL, + if (!memory_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, NULL, NULL, &local_err)) { error_report_err(local_err); return; diff --git a/include/exec/memory.h b/include/exec/memory.h index 9f73b59867..adaf1d1ca4 100644 --- a/include/exec/memory.h +++ b/include/exec/memory.h @@ -747,13 +747,15 @@ void ram_discard_manager_unregister_listener(RamDiscardManager *rdm, * @read_only: indicates if writes are allowed * @mr_has_discard_manager: indicates memory is controlled by a * RamDiscardManager + * @mrp: if non-NULL, fill in with MemoryRegion * @errp: pointer to Error*, to store an error if it happens. * * Return: true on success, else false setting @errp with error. */ bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, ram_addr_t *ram_addr, bool *read_only, - bool *mr_has_discard_manager, Error **errp); + bool *mr_has_discard_manager, MemoryRegion **mrp, + Error **errp); typedef struct CoalescedMemoryRange CoalescedMemoryRange; typedef struct MemoryRegionIoeventfd MemoryRegionIoeventfd; diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index 4cff9943ab..c9d339383e 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -73,7 +73,7 @@ typedef struct VFIORamDiscardListener { int vfio_container_dma_map(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - void *vaddr, bool readonly); + void *vaddr, bool readonly, MemoryRegion *mrp); int vfio_container_dma_unmap(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb); @@ -113,7 +113,7 @@ struct VFIOIOMMUClass { bool (*setup)(VFIOContainerBase *bcontainer, Error **errp); int (*dma_map)(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - void *vaddr, bool readonly); + void *vaddr, bool readonly, MemoryRegion *mrp); int (*dma_unmap)(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb); diff --git a/system/memory.c b/system/memory.c index 4c829793a0..de4f955a66 100644 --- a/system/memory.c +++ b/system/memory.c @@ -2185,7 +2185,8 @@ void ram_discard_manager_unregister_listener(RamDiscardManager *rdm, /* Called with rcu_read_lock held. */ bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, ram_addr_t *ram_addr, bool *read_only, - bool *mr_has_discard_manager, Error **errp) + bool *mr_has_discard_manager, MemoryRegion **mrp, + Error **errp) { MemoryRegion *mr; hwaddr xlat; @@ -2250,6 +2251,10 @@ bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, *read_only = !writable || mr->readonly; } + if (mrp != NULL) { + *mrp = mr; + } + return true; } From patchwork Wed Feb 19 14:48:32 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982410 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5F8C8C021AA for ; Wed, 19 Feb 2025 14:54:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOM-0003IY-Fp; Wed, 19 Feb 2025 09:49:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOI-0003Gz-U0; Wed, 19 Feb 2025 09:49:34 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOG-0007Aw-8O; Wed, 19 Feb 2025 09:49:34 -0500 Received: from pps.filterd (m0127838.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBVcSY008853; Wed, 19 Feb 2025 06:49:26 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=kz3u18mEdU21JsXHlve7g/kWa7hW13gZEodmiWHxe hY=; b=Tz8QpMH9y8rpJjDBPgDO02hEduXc99o1hb1eSiSqCBdTGf/nS/lIcrLEh CmDUoaSeIa0sdpx1GkDNA8THujh+BaZ7yl9zuh6yc5zN/CWlM56CiZ+Ygd18rfm4 p/MOTdDNoU49l7GLZAVkDC2pc+Q6tPi62a+e8vftsEgSh2ViYfVu/ZWKTkMZgMnt DWPfISlAjoHvGWH8onV8Ybi3tS+miM/M8xEZWJpAH//co/GEyDRoopDPr9HHIhlY WkjVe7hlHlAdBNp5K3fnbk1JCWv5emkuIxJtlKjecR2b2MfWFWi7KJa/tuJseWpq hgBLcgRnEfMvbAfKuuJ2C2Hf7ldNw== Received: from ch4pr04cu002.outbound.protection.outlook.com (mail-northcentralusazlp17013060.outbound.protection.outlook.com [40.93.20.60]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basy8w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:26 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=efy0pf22cBbu+C6k4chphdK4rhC9xAzso/068DOLb3LzxfX4jsHqI52R9w7a/5DvtpumItvK4yLZ7CxwCeeGbLEv/vcICOXwYjpnOmI1Xo/eA468NDJc3rNemmnkHzQvK4sZd7HMt9Ax8wLdFEpYzcmqtINy06EezbRHMwvSgj0je08ZW5GyacTTN4xgvlnlAdVQILVKm8JJ8A0MhCpIkbnrN5YuZFq6wxBjbPCYUDQx4FCDATcXigFKJlww6EBPnGHk/rBWyKrEc9XAFXWqWrhayMzh9pXH5z7jPqAWp+iohx2FoVb9oNmnDkMyn6zkkWq4FhVTEq/ExhAh0kRjIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=kz3u18mEdU21JsXHlve7g/kWa7hW13gZEodmiWHxehY=; b=Z36G5/ERnxZE8GrlRKLn69delAegfwyymHPHGuPq/HCplxJg7M5aI8ZWqhz5wN4y4BpbMuoH2oKbtavYXZJwF0xHAssGySAcz+XwMp/cLcco3jXKqM6M+xGMxV6fH6BpJ6HuYRRluXI2YFV9YLP4heWtizCRzIEyRlHS/QOPZGRnyGvo40uFs4xa5dJAu6QViMpWmOdJKULNmmWETJ2/KVVW25i7ABhyCN/e4pUlPdXCxZdZTsoBbPGUpzwYD4rBctzEnASDyLKKGyFEnYV++D2NZA6S/x/UxRC0ACUPm6BOXOQLTegxezDM/N0FAHUJcIn6HHz9f6BMZelTMxw4bw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kz3u18mEdU21JsXHlve7g/kWa7hW13gZEodmiWHxehY=; b=qERcCwF8bV3ewyHp4Sy8aGbtlaR+Joscz967tetmKs/afcf/JqHIytRBDDuq6RhNEWzsVZuK9Uncd6QtJGMy1iM3r9xfMc6BGtHlhVAycfVj4r+DO+AK+Oa07jbMUDI5HyhZWioSOZWifEUNU60BozeaVhUPlsChVArcVv/FAe49wC0VUGbjBq1+HDkWr16+7GNxU4RrBxxkKadmHn07QOHeA4anlMkW9Uu/tciyBPXhQdwQeRP1hYID7rUcwU80DxV9EAWkuikHXG6u4v0HD38VQZlbeMZgmm8X2jFccVuyHv+JHb2uovcZ/GNOgdJQ7uln0y4jepwyDdoOiTvOQg== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:18 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:18 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , John Levon Subject: [PATCH v8 02/28] vfio/container: pass listener_begin/commit callbacks Date: Wed, 19 Feb 2025 15:48:32 +0100 Message-Id: <20250219144858.266455-3-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 7ed5c253-d4fe-4e31-94ec-08dd50f49678 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: ncpHu3Eqw278G/WHrs5z2rG9LpO5M4Y0vtll1RLbxrAfiKPQlcJuemRxxtJz5t+Fp9vneDnRFlrTAkfRv8cmZvYZqSw4x26AEhj0mTyUas+HoliAINP1Z6HdVwsDYjSYU+8D7hfHM/dh9VSku5677ro0x9wMn5l3PSSD4skjlXGa9VWdVVn0VWvUA+ar1wzFUrrcXBTbK5x1WhijPZq4Tms2GRptm4OWuojE+UIUnc+UvDnE0Cese0DEoUU9+2VugqQWtLBZXVLQ086KVuLMWEY5WIY98Enyx8hD+Z28D/wot+D/gkJktzyP0S7jenJMzsOEB2f4AYf0GuwXvOBhLwNN01BVDhbmKWZgzmOdcre4nvGUYJEIcZZKNdD6Uc+DOjQ+mYwZ3tJmJWRE/sUIPhPxdZfzcvRXimIgn+Th+5UtVUY8qVfeAhnhznsmgY9ax4QIpUn/w3Re5i0HfLGxDbNs8MqJmHUBdGn+wstju4yXb9eXCB8e1aRup2FzEkIGj5cbxp2Y4N+ErxpU0IZqPHQ95yu+hK7wLPvAE6bERSXGNtvo4gmxgeJjlIuiyDoK0LCIsyIAwgEcSOLUm0ZPwSFh92eCyyWwShM1ghXb1taX8j8qAIl4a/otxdJjwSRNuOLIuXNoLGAnvAvqNvuZlvaa15VEnnGunx42yQusTQ0ghmUOgvtP1aTSrpqthE8UwFgQ7+UEBjHhQy3OkuUTXXMkkC9MVIDTuH/U+PmILfrGvaksA/iOdZM73E4WkYds6Cppb5a9JIhEU/njX5LoqzyFRAUOuL8UDs+HLVBXuYj+7Huf1KNrxIhCokIjfP5zb2RMhzVswtl9jfWFZ36WoP/aGL3SnHvZgsQPkiO/HwCnnon/yvq32aAmINTkm/alyZwffXoxKjmpGEqSbCxP0IxytM3H4M4wYyGBgPlf0vrcI5YHVsFd5WkYwt37U7DTq58ke+0jWKyTNxI5eBt5WD4EQu4cbgZFeWi+aub8tY1d3BiaZEajxQVNvvZt7RUQtKKWJ1yWafBqevita3P1X/9qdq/VKmb4j4vHqHwOoFWuqFfHYkhpAlrg8igN0l5ItsG58UjiZdMdbUmGbW7Hyo2KmMYAbA4pHrLaE6gFMxF1qV1GECjwtgS8pksAK8iiibCtUiBdaJfQvg3B6GUuh5bE7mVB4ZBtvMwO4s2piyiBshaLd3wRCFOtNJywGTaIKq8+i/UOD7mJE3WQlCvKpZ+Qu0IjwkYrqATFj0IcSD4kOP0inQtXYSPjFo9qCwFfNRt7jSgzvmMxWqoCh7TANfSehq7dwhglu259e+s3DXULiTeoZO9XcsZbvmT56BUILPiBNBekpC2R6UD87svSq5uLDEVcXcLYmtUyc/l+QfA0fVc76efim+2HmLmeLOui X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qZfEYDMKOi+9yDo7gbEFAhOTiP+gj6qxXoNzUyMpFic7LcxvBlQa3BKm25T6NWTpacGBN4NCsvXzeN4jAkFa/m9w/ZngbBZwSrM6A2/SUGoIxQFK4wyhqax5Y6BFmy5kU2h7H7BPI3Mti4jkqxdgsk4hWsUIB9lRX+Ar7KzW0EkvGrAZFg8SAGvEQZJmc9Hg+vO1h+luQH99dIjacOYnyUSQb/Ev64GcGQT1wAW6AvqrY03tfP4uLR7SaAczNzoUrTI9THtJRTiyzUT9LVYgsOyPiTa/x035IDCwRUNyXfPQBCpBv9SSKXMrVGgkzBp2pNltYRmZ4gu17HXOuPqSAF+ekQW/X9F4Xv+5xO7+eyelEGJOmGoXniaqq0gFlMZhCkJCf+409LHooPuPA0L6ggdD65DFav83mY6vrCZxDqMoZwzZ9Cn41x8D3tkN6xdumOX42U6bj4/EP5xmZ9H/dpfdWhcqPmYvFEqlub49zQNbRxk28F4eayjMDK3TprCcMLNKmN2/pvvixVxGdEzVwpu3tdWcjNUiXqW3PLhphzw8Rm6JPipEml3GpgfinzBnc7nFJBkGkjA7CgaaaX5X4MSs/4nyvqLkePlUtP1Xrh6PP26mTxDrThDZU3NRNxGYUeFeD3wzZCCTBAGtrOsKuwbN+uoGkLIMlBr8o47JR1QTRObPvig16rPNZEu31ae6mpxI7Slk6oW+McLWHo69vC9DIVNzkmy0XQLrT03ITSoCMZJVXy4x8kwnIoPuX/GMHEJf53ZFC4m968oQP+tepe+XLaNbZ0DJDqAFfR+lphz4wovO8zdnByK9NjQO0koZ5XKvXkbTc7ZB13oLQQq7YtK47Jnk5JGI8pFPLOmtBmv4/jDZ2BPUZeTmOCI6+fbMyVj5j4+SUxomFJMGsYNV4F3PiXqXwIlcGI9PFxIdT4LNl/n8l7mMyhni7wBZNVL429FllE6Y+6Exm1WGjv/f+K1+UFCHE2K4FbhoDj80TVKXBnbzZN2GUv0/h+Rp+H5WUxJVoTFyaiU/cfKougK3fJSjHUHg4jIxN26NGJ7JS4DaCEY3edTJWh5uA3RYBcvIfGLkTfBvWYqmvn9ksV5MV5jmp+XPcUeqDsrb3Xb78gF3MYjq74BtXrch3QQwF92ZIkFcXNFuAyNarHmeUkIOsPYaUXbEvvUvQnb+Fr/S5jFO5NZr3tVm3Ijde4g8PhvFzXwdcepsQ3slCNQEHXGASied6AejS8cxhiFCshpGUyijNsYCDxSIBRAYyT8BoJthZCrIDxtmuPXyzH7v8h7osevsakRbwmj4DPFK+eCZMcYvL2rNlc5NGx+9FRS3mEmmiXkFhbgeOEn05SOAWe7Gr+/XxUhZ4coGyhJNeM+jln4xQN4GNEjRKfdrfl9CJdzXb6r1/pv44VUybixb9OzyoMJUEOudYra/nuM3+IBwIfIRV0XJN1X6tWXVZsHADU6ekmVOFO6jvGHorWDxtrVqAn4UWvpQxkQMi8uh17dFot/ODo+/QhRx0GS9D7PIJkbD6fg/dKNEhqQ5N63pSJFFgZwg46SRw1c3+NHBDu6683Qb8dFCFH76M+TkI7jjpwyp X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7ed5c253-d4fe-4e31-94ec-08dd50f49678 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:18.4357 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fRQALOEq8xME3IKIs5ffsaK1QKSS3pkLenxDjf5xvovR3Kgi3m2aJGSIYpMyLOHpbSwERx9kwFFksGFWo1J9Ww== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: eOULesTb5VjrgI3V1H_xtVSJGy6vRbGa X-Proofpoint-ORIG-GUID: eOULesTb5VjrgI3V1H_xtVSJGy6vRbGa X-Authority-Analysis: v=2.4 cv=d58PyQjE c=1 sm=1 tr=0 ts=67b5ef76 cx=c_pps a=4HTA/yvmbQxnAJvhdENERA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=RQZ_2NmkAAAA:8 a=64Cc0HZtAAAA:8 a=vU0vC_IIgwTtBsNJuTcA:9 a=46pEW5UW3zrkaSsnLxuo:22 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: John Levon The vfio-user container will later need to hook into these callbacks; set up vfio to use them, and optionally pass them through to the container. Signed-off-by: John Levon --- hw/vfio/common.c | 28 +++++++++++++++++++++++++++ include/hw/vfio/vfio-container-base.h | 2 ++ 2 files changed, 30 insertions(+) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 8d3d425c63..6f106167fd 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -557,6 +557,32 @@ static bool vfio_get_section_iova_range(VFIOContainerBase *bcontainer, return true; } +static void vfio_listener_begin(MemoryListener *listener) +{ + VFIOContainerBase *bcontainer = container_of(listener, VFIOContainerBase, + listener); + void (*listener_begin)(VFIOContainerBase *bcontainer); + + listener_begin = VFIO_IOMMU_GET_CLASS(bcontainer)->listener_begin; + + if (listener_begin) { + listener_begin(bcontainer); + } +} + +static void vfio_listener_commit(MemoryListener *listener) +{ + VFIOContainerBase *bcontainer = container_of(listener, VFIOContainerBase, + listener); + void (*listener_commit)(VFIOContainerBase *bcontainer); + + listener_commit = VFIO_IOMMU_GET_CLASS(bcontainer)->listener_begin; + + if (listener_commit) { + listener_commit(bcontainer); + } +} + static void vfio_device_error_append(VFIODevice *vbasedev, Error **errp) { /* @@ -1396,6 +1422,8 @@ static void vfio_listener_log_sync(MemoryListener *listener, const MemoryListener vfio_memory_listener = { .name = "vfio", + .begin = vfio_listener_begin, + .commit = vfio_listener_commit, .region_add = vfio_listener_region_add, .region_del = vfio_listener_region_del, .log_global_start = vfio_listener_log_global_start, diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index c9d339383e..0a863df0dc 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -111,6 +111,8 @@ struct VFIOIOMMUClass { /* basic feature */ bool (*setup)(VFIOContainerBase *bcontainer, Error **errp); + void (*listener_begin)(VFIOContainerBase *bcontainer); + void (*listener_commit)(VFIOContainerBase *bcontainer); int (*dma_map)(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly, MemoryRegion *mrp); From patchwork Wed Feb 19 14:48:33 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD792C021B2 for ; Wed, 19 Feb 2025 14:50:30 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOJ-0003HE-Cq; Wed, 19 Feb 2025 09:49:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOH-0003Gf-TI; Wed, 19 Feb 2025 09:49:34 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOF-0007As-A9; Wed, 19 Feb 2025 09:49:33 -0500 Received: from pps.filterd (m0127838.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBVcSZ008853; Wed, 19 Feb 2025 06:49:27 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=gG4HXVmj9QlbGmx1vIMAGEk4uIlpHTF+8RGLijexj 14=; b=1O21s7HGsylnQAF/4SwtVoelEXW7u7uy/E1oHo07tf9ClZheIh1zuXDm9 Pp/2Qe5Kqzlt/Fww7Z5O3FlFx7CwdjLPHPncp8h2dFFkE+x0pxq2wlG3whe5D52p AmDuTGKkovWue+LLjQTyVQbNJ0xXszGLyeGAVdeici6wpc8ICOHooiApE5k66mKT fQrUrqGHKvHuJo5eZK6PqSW8SDVWgxMdw3Y/gR4by6qxddC0/Bo0hMwsswX+pRaa r1gAJKbi6Pid/0sjhLYyf1tohZuNcCBERDcSXeSvkS66uQv7k0OJPXwRWLQrpAR7 Jqkp4FEEkJC7ohIp/eoPW8OjvRO8Q== Received: from ch4pr04cu002.outbound.protection.outlook.com (mail-northcentralusazlp17013060.outbound.protection.outlook.com [40.93.20.60]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basy8w-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=PtgYip2ajbjWMJq1UBVRozncKl+u1zgjDTrwBANUybl9iMTfeJaMAVXggQiy0PsKkexFVQfAfjn+cR+pPRf9I+6rWahuRGRPGT76ktzzqs5JRjree53BAZnGJtUK82DyvURNLLJrDJ2rFymeBXXJ2Fbq3/HA4n0MJf72YrYZomlC8HUMKbGaAF2Wsx5y7XufY91ewlOwYfP+09O/3qa+oEpUjRLxEmJDwcylAWpRrqXL52tz/zzfUHj2dgDm7EMj06iQLyvmxXnnKMZeWsglBI2VBxKxKWnjGDg5damUhvmVWyaKvFMh3grrboaiYKjKzyhUDPq37+IpqALoYhq+aA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=gG4HXVmj9QlbGmx1vIMAGEk4uIlpHTF+8RGLijexj14=; b=hLWROKlpzZntDtiR0h7Ifh6oHusU+70wA0lbv7Lv3KWq2nVF/KekWd8nA1rLu1HZLkloK1EDrstslFDlKvpFZv35WML6Y7EoODeHtNh67I7/Pcru/VJpHQAZig9/GeIGFL0AVejU8ewTw/aguor55+5cbl/vuHtzctEa7PwXyRegYqQPlYff4chOHZjIaOSAdfPqrrrFGB0RhY/bzvHigQFOjjUpuKhUWu9a/pSjX4m5uxaxRM/tjtYxlr99cagTwLZwPSj2qGV8QE0TMEmWqd4StJ2G2w7mX5bWHrHIACv7s8ji58+R0yJ7wBNBdA+4WjKQVoCpms0ts16X0wC2qA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gG4HXVmj9QlbGmx1vIMAGEk4uIlpHTF+8RGLijexj14=; b=onupNvRXv8aw7C/kiusTO7Fjg91l7WQt5Q1io9PD6DNyMMuvcj3Mh4OKh8y0aY3/JUzSwJ0dfroQnj2zyZYHRshQxTixGu84Ks277dTK3hKqoAcNz0HS7yD+4sThkNZabvqkdbsIV/zDjyoNnAe6mA2SNzmN7apegZZsByHbroObA01OhdK9HxJWZFjYTpupkIOePT1ROL2/0mDMH1PP8I6IXdZurtCOJELTNzNcpaYcgdzXvjlB3gj3IoI69gxsDa/RAqegUiOdr0CRcOtDd+cqi+YgtZbSADEmf34YLalTpQrYiNO98DFjOCtmelI23j3BegNK6wvy2MTQILuraw== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:21 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:21 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , John Johnson , Elena Ufimtseva , Jagannathan Raman Subject: [PATCH v8 03/28] vfio/container: support VFIO_DMA_UNMAP_FLAG_ALL Date: Wed, 19 Feb 2025 15:48:33 +0100 Message-Id: <20250219144858.266455-4-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: c64daffc-2347-47bf-4ce0-08dd50f49870 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: LQC4GRsgVHJLA5bu/Dor/glDfg2dx4KJtb2vq23KjPruCh6kBstLYuB6J49Iu3dP32EliJFsLpAGxthHcZ/k23qpKAHXBwi7pqtB/tV7xFURwc5f6DMvRyOQqkwjJCWnaUuYkWjAxrTSYyguHOnv4SZ3x43Bmrl9fgcFRYGyRMz9875AlkhhffV2R6smy1M0aJIJJ0Uc4PdgFV1kHOROOsVgbRxEX8O0InaA7G0Op8wgUQ5MPeFaGG5xxrsdzUTrukRGHwN/nDa5mqjPw+cv3PlRsPbgW3O2k4jJ0/AdX3eUZoAK15DYGT2cy1VTTUbLwvi/ExHznr8yayQOou0SIATNdl7e78odQNZxX2bpqDSHFY+kdrB6mFcGr4Oetdaojgng98RA7VWDaQ3Lb7v/nMZEoPaDfSrhgjIt8ulO2L5pf66K84A1GXg+3JjqBTBJD0x6pakdWxtr0+0UcrUt8U4g4ON1ASClbG6OPtwWgVIgrOqNTIGNfj/mkue+fjZ65wyOhSrIKpO1ngQ0J9zO90MhwnRCnfv1KfrToTTRrYf4fXsQwAsvOyU/nYWqtlFUm8wZYOexPTGPG5ilJWmMaHeGwA44CcPmf1tk4kqtOax+H5XXgUAvqc+E9W4eTDeroubwFKQDOKNHPQkw5cw+zMtdNAONNqYhE8sbwd8aTEW0WnX+19cgraTNgrVTfdo1MvBJpMpX/SFRFSoj8ewNWdWXlwb3T2zRWhS1ygRzA0vc3iST9RxsIWD8jq7vXJg7+lch3MSpEndpdtXOZi4qr0ANns4re0IYXE8xBVh4lj0c28IIrlGBn5XTrfMI7Uhxz1u7OJ/r58uC9YNyzjeM9jI2VuA0HCWWWDGYBPDGo8pV0NG29WcudMumcFD+pDara8yg8aRbExSXmBEG+B8nstpX8I35cXwWGkbIz+F5xjqK4sK8YtvKS2ZcxSjbX6UrCSrQUnn6gF/7NPQqej/RgejcnQuEMcByFCEnXlLS8/AMb/vfKjRll5DGk577mcYEcQ4q/08iCziG1jlIhNaN8ABInV94Mybb+2+BD3hPpl54wpsL4seUDAb8QRlW76kXQi5LaF/lTyOjem+3nsuAe5DWeuhWRzvRUbGWjkf/0Tn0SYGU+OIQ2GmOGopyCIvcUSeiHupYsAbCCHr7FfG9d35vyc+C3OMvjcUSVgxjqXbcmLj5RjpPCdE3sbbaP6OBwiytBQVsC9ACGeQCRMN/5TNffM3Bd27d2FecEzEwM6zGPfRfb3H+yG0KmUnNF9j2y4tg2qV3hzmE0oA11fgHJRyCgCGeGy2kWKEp6ppTNLZp/A3uVnMDbfDlyK517nU3E6wP/G5Isv8oOMhapzxV7jrewqwoRJh4+SWs//UJwMM6rUFd0ZQAAee/K0XVSw8a X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 7DejeyOB5aVvqvXqnFYhIBVB+jB/5WLTTzaCgLR0RVe8Qkz+KT2Ky0zVSUmZ1Hptc08EgdGk4Q4zjvK6qAg87WOClxDN3LHD60a/W3SZLVKi1UGpgH19kAM64rihPYMS85BRO3F2OA0cZFZXHLAZsKxLoYY7BXI4He+IHCZnr6Ax5Aev+tz1cvwX7Vl0R1/tD0bCmPlMgGheJ6CAL9MDl9ptjt3rXpEKEQhA6SnzNnVASDLs7XJh8KY/p+OLGzY3tLRbKXbuGM66m+3E0AQcFZ6IKGqiifODFUrdcdYjfxCJZaH28gpuC+pbBQkbXlbVX9of+jazLCg2BuA8N7i+K3cs+3mirj6dvOs4GCYxKPIae3+BlRh22ayxlu30Ne5311wIrQ1Gm2KRgcR8GNagcAnY9PNucYsPKO4NMlzo4w42awKxWdN8AbuDbYqUzUPAjKZb4j4RbkuNhWTCO90AuZ/+hB+6IJl4JkPgfuJrwtzOnrub5ILxFRooXqDZQywBaWynRUW4LaawUM/tl+NALMsZruW/uvRVw7Mkc3Adz68ZMw5gET0sMDDMQiusqS0VenNiUY4WrRgi+BANMSfz5/ej+xWFrMiwTMicMsAYJks2PXMrJU8PpKVMF6GN/MIZfOe6iJSLc7VsR/Tc3hwN7PzU/pMGQLoqdQuVWClL90wB1n27FzBbYFHCqY8YQaURKHBdzyd6lwncY2OT8SbS3mTO+KlQWtWExd/ppfKScyz6eDyl2cDvaLKs4DSTwEQk1FtUQMXnjtoKd9tAH2Bc5Gjf6iETvFtJI105svopGsxVKGn0l6ENo7rwWMdLvQ5JbD9yzv/9bLrixN1eB0tzkn8XPukBtmBX31QbBdUwwEbBXF1Uch48fmbulB/Y0l8+g9J21uiEP3YCOVNpGw8PxLLpBx5b+QNM7bLVpsULiQcAiSSYZTLMaDrUy+OeOVW3w82tat1sOfMaRsAmKaevXyuhBNNsXnejwx4C9eCisXT6O61YtlX9os481o1lWpi/qRfeGCxamzfCuEgiwB6yJiXWXMZ2bDUpWfjbCG9hRAU3TrmYGIwb6pOea1Z1RvCcWuf9I7g1wFUvs/0BWHzJWCumEhpXnVLs5EcjyX9/VB9v/v0ECQqtWNH8M5RLaKcv7xMjqMYcDUKmtiaolGD4LEntg0aqWEEonmdWdaFpfPi8SmiA7bjb2I0/ZFkDgpkZi4uyNjYduZJoJdH9nzhm0JYS/RM1qv51slTAvAOrC4i2WEfPutA3L1AGzfQsjNOufzlyB3O3AlZj/KW2D2ontHpTy3Z3jlePm6qFN5RKd7OaEzawHhrcpF0A20NdVHYn7NooZCLBXTpCAsKNUjI2IwgQhqs69VhlmGOOEmv4IQlDdd8VWAuATXI9FnzXbMvHhyNs38E5EYGE6HWUV8hMVS69RPnEjkONTZtUG51UbcaL2pu+/hd6OKQgv9FurAzgY7M3axDKdxprGbbCs+vSgYO+9n9yQ4b4uhMETvmyMo+T/7I2x1JTgW3sTY2Rt6teMro7qQEcBvaO/qCsvrJFStGuG4i1WyonO/WVnQHymT/kLKFVANk6Md4S/HhKvYy9 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: c64daffc-2347-47bf-4ce0-08dd50f49870 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:21.7538 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: X7BvmueD3C5pRcqbpJPKz4Iwm+aRgvvNVOv6/hnYvF93h2AgSOtRmGJgza1e1draazS9eFF4sqxIzUKHY+3FsQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: ozHuOVhwLTlOqkUO9imbgvx7maZ_Yncl X-Proofpoint-ORIG-GUID: ozHuOVhwLTlOqkUO9imbgvx7maZ_Yncl X-Authority-Analysis: v=2.4 cv=d58PyQjE c=1 sm=1 tr=0 ts=67b5ef77 cx=c_pps a=4HTA/yvmbQxnAJvhdENERA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=rAkVWd3RjT2K1M8sKjoA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Some containers can directly implement unmapping all regions; add a new flag to support this. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio/common.c | 24 +++++++---------- hw/vfio/container-base.c | 4 +-- hw/vfio/container.c | 38 +++++++++++++++++++++++++-- hw/vfio/iommufd.c | 19 +++++++++++++- include/hw/vfio/vfio-common.h | 1 + include/hw/vfio/vfio-container-base.h | 4 +-- 6 files changed, 68 insertions(+), 22 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 6f106167fd..b49aafc40c 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -324,7 +324,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) } } else { ret = vfio_container_dma_unmap(bcontainer, iova, - iotlb->addr_mask + 1, iotlb); + iotlb->addr_mask + 1, iotlb, 0); if (ret) { error_report("vfio_container_dma_unmap(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx") = %d (%s)", @@ -348,7 +348,7 @@ static void vfio_ram_discard_notify_discard(RamDiscardListener *rdl, int ret; /* Unmap with a single call. */ - ret = vfio_container_dma_unmap(bcontainer, iova, size , NULL); + ret = vfio_container_dma_unmap(bcontainer, iova, size, NULL, 0); if (ret) { error_report("%s: vfio_container_dma_unmap() failed: %s", __func__, strerror(-ret)); @@ -806,21 +806,15 @@ static void vfio_listener_region_del(MemoryListener *listener, } if (try_unmap) { + int flags = 0; + if (int128_eq(llsize, int128_2_64())) { - /* The unmap ioctl doesn't accept a full 64-bit span. */ - llsize = int128_rshift(llsize, 1); - ret = vfio_container_dma_unmap(bcontainer, iova, - int128_get64(llsize), NULL); - if (ret) { - error_report("vfio_container_dma_unmap(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx") = %d (%s)", - bcontainer, iova, int128_get64(llsize), ret, - strerror(-ret)); - } - iova += int128_get64(llsize); + flags = VFIO_DMA_UNMAP_FLAG_ALL; } - ret = vfio_container_dma_unmap(bcontainer, iova, - int128_get64(llsize), NULL); + + ret = vfio_container_dma_unmap(bcontainer, iova, int128_get64(llsize), + NULL, flags); + if (ret) { error_report("vfio_container_dma_unmap(%p, 0x%"HWADDR_PRIx", " "0x%"HWADDR_PRIx") = %d (%s)", diff --git a/hw/vfio/container-base.c b/hw/vfio/container-base.c index 5e0c9700d9..db27e9c31d 100644 --- a/hw/vfio/container-base.c +++ b/hw/vfio/container-base.c @@ -27,12 +27,12 @@ int vfio_container_dma_map(VFIOContainerBase *bcontainer, int vfio_container_dma_unmap(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb) + IOMMUTLBEntry *iotlb, int flags) { VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); g_assert(vioc->dma_unmap); - return vioc->dma_unmap(bcontainer, iova, size, iotlb); + return vioc->dma_unmap(bcontainer, iova, size, iotlb, flags); } bool vfio_container_add_section_window(VFIOContainerBase *bcontainer, diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 0db0055f39..82987063e5 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -117,7 +117,7 @@ unmap_exit: */ static int vfio_legacy_dma_unmap(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb) + IOMMUTLBEntry *iotlb, int flags) { const VFIOContainer *container = container_of(bcontainer, VFIOContainer, bcontainer); @@ -140,6 +140,34 @@ static int vfio_legacy_dma_unmap(const VFIOContainerBase *bcontainer, need_dirty_sync = true; } + /* use unmap all if supported */ + if (flags & VFIO_DMA_UNMAP_FLAG_ALL) { + unmap.iova = 0; + unmap.size = 0; + if (container->unmap_all_supported) { + ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + } else { + /* unmap in halves */ + Int128 llsize = int128_rshift(int128_2_64(), 1); + + unmap.size = int128_get64(llsize); + + ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + + if (ret == 0) { + unmap.iova += int128_get64(llsize); + + ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + } + } + + if (ret != 0) { + return -errno; + } + + goto out; + } + while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { /* * The type1 backend has an off-by-one bug in the kernel (71a7d3d78e3c @@ -162,6 +190,7 @@ static int vfio_legacy_dma_unmap(const VFIOContainerBase *bcontainer, return -errno; } +out: if (need_dirty_sync) { ret = vfio_get_dirty_bitmap(bcontainer, iova, size, iotlb->translated_addr, &local_err); @@ -199,7 +228,7 @@ static int vfio_legacy_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, */ if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) == 0 || (errno == EBUSY && - vfio_legacy_dma_unmap(bcontainer, iova, size, NULL) == 0 && + vfio_legacy_dma_unmap(bcontainer, iova, size, NULL, 0) == 0 && ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) == 0)) { return 0; } @@ -533,6 +562,11 @@ static bool vfio_legacy_setup(VFIOContainerBase *bcontainer, Error **errp) vfio_get_info_iova_range(info, bcontainer); vfio_get_iommu_info_migration(container, info); + + ret = ioctl(container->fd, VFIO_CHECK_EXTENSION, VFIO_UNMAP_ALL); + + container->unmap_all_supported = (ret != 0); + return true; } diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index 583b063707..e295f251c0 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -41,11 +41,28 @@ static int iommufd_cdev_map(const VFIOContainerBase *bcontainer, hwaddr iova, static int iommufd_cdev_unmap(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb) + IOMMUTLBEntry *iotlb, int flags) { const VFIOIOMMUFDContainer *container = container_of(bcontainer, VFIOIOMMUFDContainer, bcontainer); + /* unmap in halves */ + if (flags & VFIO_DMA_UNMAP_FLAG_ALL) { + Int128 llsize = int128_rshift(int128_2_64(), 1); + int ret; + + ret = iommufd_backend_unmap_dma(container->be, container->ioas_id, + iova, int128_get64(llsize)); + iova += int128_get64(llsize); + + if (ret == 0) { + ret = iommufd_backend_unmap_dma(container->be, container->ioas_id, + iova, int128_get64(llsize)); + } + + return ret; + } + /* TODO: Handle dma_unmap_bitmap with iotlb args (migration) */ return iommufd_backend_unmap_dma(container->be, container->ioas_id, iova, size); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ac35136a11..f4f08eb8a6 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -84,6 +84,7 @@ typedef struct VFIOContainer { VFIOContainerBase bcontainer; int fd; /* /dev/vfio/vfio, empowered by the attached groups */ unsigned iommu_type; + bool unmap_all_supported; QLIST_HEAD(, VFIOGroup) group_list; } VFIOContainer; diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index 0a863df0dc..24e48e3a07 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -76,7 +76,7 @@ int vfio_container_dma_map(VFIOContainerBase *bcontainer, void *vaddr, bool readonly, MemoryRegion *mrp); int vfio_container_dma_unmap(VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb); + IOMMUTLBEntry *iotlb, int flags); bool vfio_container_add_section_window(VFIOContainerBase *bcontainer, MemoryRegionSection *section, Error **errp); @@ -118,7 +118,7 @@ struct VFIOIOMMUClass { void *vaddr, bool readonly, MemoryRegion *mrp); int (*dma_unmap)(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, - IOMMUTLBEntry *iotlb); + IOMMUTLBEntry *iotlb, int flags); bool (*attach_device)(const char *name, VFIODevice *vbasedev, AddressSpace *as, Error **errp); void (*detach_device)(VFIODevice *vbasedev); From patchwork Wed Feb 19 14:48:34 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E8A80C021B0 for ; Wed, 19 Feb 2025 14:50:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOJ-0003HC-B3; Wed, 19 Feb 2025 09:49:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOH-0003GY-IQ; Wed, 19 Feb 2025 09:49:33 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOF-0007Aq-9Z; Wed, 19 Feb 2025 09:49:33 -0500 Received: from pps.filterd (m0127838.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBVcSa008853; Wed, 19 Feb 2025 06:49:28 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=UHAnUw18hvZ4CndV3ruk0AynqvJjp1s0gnGiSY7SI vg=; b=ExFF4vMYl9JTMlNFZXTvZZjvnMvNbkFKpnjUZuUEwvn5LtqxL7IWuPt9H zk710KiYVjIET6mXI++DLLdgF4nRxMDW3bYhPvxpqOnBEqcIF1UyRh21VU01lfBi Hnzprhq3VkQjgvyyLf+Ox6jc0k1JDjRv9EjHRiZpVq+D40ZRJ3DbCDWiwVWAAY4b 9GXbw5xUBIxqg2sGbQ5WgpXssoHsar8IVxBBM3HHTVmbQx5KaNRyIMuonfhPjUaW jMdPmJLtQPlnq8vN4nrwwvDuUzhIqKmSFKV5r8J2kFXjHASngwBv/ca0yK6ocpP6 CGo05intoyzrdJTc4sAqtqQbFVl2w== Received: from ch4pr04cu002.outbound.protection.outlook.com (mail-northcentralusazlp17013060.outbound.protection.outlook.com [40.93.20.60]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basy8w-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:27 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=kaiPKZpYXT+8stJ3m//FV0JSi2SOi4bEdFVumFlZqImvC0k6m2KVaH686eG9ckGd05RhXHRn96e2yBLSf8HA80YsoU826+TNydalZERLd1oWt2NUi2fxwCxBZmrCiPZWrbkM8G3A6AcbQnCPLW45uPdVVub1ry+pwONrqk1V/ftj88q8Fi2SpLgLA2qNIRHDG/BzjptHsQywoiZtNHI+4Hslq8/d+8YGmXZzPd18bX9W8oCWaZwOrC7n1uxrYW2uMlqOTgl/2+YniZAI/Xf2mhR6TAJe2brzPFEbPUeg3x1+TOaHMo9BRLNEqmLR7kh/qt69OQ9HcDCuIy3dEZsu3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=UHAnUw18hvZ4CndV3ruk0AynqvJjp1s0gnGiSY7SIvg=; b=ahAj8Z3+LafwiYAjwSJ+gPHP3KZ3g6L1fpYYQea0pipxgSvnmsgvAOGCfTFSPJCZd+udWjjLufDvjdLWjGG1rdpULtZIHcXq/SYcyRwQ18gxLb+YcSN7z4CKH8VkN2mZnBqrZDwXIlyBQ9p/WwPv6OfN2691TjVzs+1I8TyS/71Er2MscOvaw7HdnRMsBZvl2IjtrOmbhUXe83l/hUdoVBWnLVwarbsZ1NGx0+4Lm8xt+90xvF3XzoWmvg2zPeTTMNpUhUngoUXGvg0B3vTty8Ix+kyEm9RHdm18f6JHTqTRWT7C2i0c9DT4NTw1PPUf1oXHdXZ3BvwY9WbzExNe3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=UHAnUw18hvZ4CndV3ruk0AynqvJjp1s0gnGiSY7SIvg=; b=Yao/eFFiVqOpnnD4l/xb0rDs5Fvn2y6WUygIqyiIo3URUotBT5b67RaDZkoQ+bFyBkV9BlIBv1ME67rYLK2T1gl8p6a3I9tuBed2dsLD6ohkEmGn+2+lXqnqz49IHA5drU6j7qFqqHuBz3noF5y9b0qd9MGTbP0R0bpmDx0vzMNu3UAGpTMwPpcQurqBj2y6BXEGy6LI+wcB+u7EKg0DgspwHLOGuNGZ9cGvhW9Cd0DR6jCKb3G5KyE4jpYaGuml+xVGDIABqnFXx9yhkpK3iTe991QUxErWhjJuK/G/no6XnoqF7ok5QcMbec8c6sGWCHFX8NnxnuG6U5yHuM4NlQ== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:25 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:25 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" Subject: [PATCH v8 04/28] vfio: add vfio_attach_device_by_iommu_type() Date: Wed, 19 Feb 2025 15:48:34 +0100 Message-Id: <20250219144858.266455-5-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 49fac329-03c1-403c-e66b-08dd50f49a35 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: aziQnzZDnXBaNemCA2+LkDHB9CR0YRXlzw2fVCZhWKXsmNk8NgNBIjoNji+jw0MrO5GDakmEDL81SCOj+aVOoWKT1aYqLElv2tGu3w0+wUEhV2CUf7QP4uPYNpuhu7OZ8v8z+ivrqNwfvDkB7tPBs0d1S2soEB6qPr/X1N4YRG/TwnjP3Qodyjc18n65zetDxD8xF127T/Oqizcrqw7Pmyn8d2WueEWDH/EfB0kJN9aCn1mytt/OxQC/Pur/Vh4nNKg8BRnt/CG7mhFJ/DhnYI/LaxZhPtyV/6iylS/ebr+kpWbsmvafnBrnUyfVM74eWAfaHAJdyhSl9GQepBkHHTBBc3jyZB7QZv0YbxlarW8tw1ffLRPDTjRDaIJ74tclBWNzWNRCQFdbld1dXqzNv0GYdCwtWggC4Th4SsUxbB07xtSZ99IjYB3lW5q/v5Lapyrd5OqlKObkaWbJa0nLQwzWJM9RbRDXzTBIPeSkhYpbuGXTkQ68WhjsdlxJb12i6QfIGkLSWDY62oKgeTbs+iBoO6CK7EDDN7lF0HGNxqLksYWF5lhepe3Ql7ocP9ZEyUkLbx+9MULnOpcF/WLPU4+8Kz1X7Thtl92qRELmEISv8/rrVoa/7JnKVV1K3zw5f34rs+yZmsUol4kfOl23xxMr4/SY+9djCF8hc8mmEf1w5B7O3SQ6sNajZkNo4HqcI78rBImCJwCKw9yKZBGMel340CRqfQ0PD0ByQ3MIMKJkSDSS58aguJKC8IgOajX2Z5OT6G2pzcpoIW0JyeWdMsvJYURl97CEYkIgdkCBDjCmw53z0HMKrQ3kH2qkGLAT2COHxkv9rjgSRbPcK+WZP95FiUgjtovCgpYT3Bcr5+4sifXji4EvBnKxNeAICGdSmUNPgSCmek39x3S+XzI6NC95ztaoMmB0h0XzGbWnJJ8OPM4tCk4EkIUqbEUEMpsHMJnvQEyrGd3lr92MfX8e5pE1i3XJ72p7f5uHjHQcK1IWjCqP70fEXqkBtIKIXFrMAnZtlYKbznpPE2NJhOBHIyqbt5WADxhgBKZLkZMSZQEW6t86usw4JMRujHb25+kD1g6UZf/auJbvvKVCW3E11J5xNOg9cdhk4ZaRU0mWzvAKCtISqWuV5iNNqWXf3zZr0RLpMBbydFUeD0vD5dQ4Q+MUNLoEJs1tnjBh1D840styw+dL5wFwi3/1Me5wo4NGnpY/Oz+Pth7tnHCbwu3/yASRTXOsUjGiA9IRLPI2h9jkk5Ysw2yHz42V2VIRSuYjMUgWFV2E7X3XLQO7eMyaZWXXlnpwaOIHgVcZD5zCKXw8/HdJ0QAyiCQvQqzOBWB5LM8DmFyk+BiqOZYn2uFQGR7DOy+Hbwh3ErXc8PSoiMFYmSekeFPBBsxysWAz9bLD X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: cSGfoOJXLIoI8ltWJV3W7BlRfwyJOTwtB2dN5QEd9KLh3YS37ThWPdy+ECtUglaxqYeLgdQweS7U/di+5k+jhz3fblC0ITzu+w/ST6BkAAk9BNYgjKgon7/zyRI89BmhEP4Z9zVxLvxFVUWUTEBZo/lDeAAMDhxnwen+mFELhbPjN+Y8FR1dVxme/xRjG14rFa479UOqMm3gn0RDcnPvL4f2YcImIkmyYae+qwjE36A3rZs3KI8fSb8AMFOxAokbYiSUvf3QlwkCJSplVFVBz0B9kBtQd+NHmM5vTLT3qs1C1cTdsmLKmQAaaL7iU4NB09J9Pjgm6KtxKxfBMNfH1YYMGJ43SvmNAUM0OCpP1NQq15Ft95K+VSg+mQLyPP0kAOUN/b8H9qVHCnF6hUSOI5FCaC2paZAdpTR4eXlqBNy1kSLO48vCYCKcBHdN2aqTr1GZryH0V7Zab07RWgAFKfIBdvIbr+q1vMojwFaJD6bD5DI8V9b92aF8Z7eAUtWJe4BZEt71TrgszSxolX4S9OGnYNhR1hRHJ78AgcNDmiwPn8s4dt77n3nlkkc7JVBQjwvqqI3YFn9oKZwnqn692CVbpXq4KBaQ1RPilhOEs94lUqLBJEmhm3kbodIYQ/IdaLMB7NCVDtKOdGUXKez23LLtwpyU0o3RkBAzzRBzan7E7KciPTp9i3B7x8Zjjj8Tj+zPPY0lX1avplvj/9RK8pAP7OXM14qFC0xEjaof9GG+VZ1xoasHi8JLsVuof43fjyRH8BOUlA96nzBS30Y6H3mJO3ra+aAKP239+U72Av5b7Pse1IbaKXjpIR5N9gZ3ry5aZQLkz9o9Kxn4xmjY3GwWVxyxlWMvxVTGSnCINiIitfDq/XagSRWql68kvhKewtWiiJAvAyQ+Yr6XVrcAf1L64QLEYOqtqk66TJwl0XXK7YbPhSyMehZQdx2dA/5RiLdXG/VE8EcCR4/Kf7zyT/e8/6mW5itdxkMaEPARMQ+oPEG7QJJw/AIhSqPEto+IHwjjnZH0RHna3oCrRv/fMk4D2Zw2gZKbHgFOsuL3RaPjt49k8fQSNSkUiV7WkDRial5lHvpijmk43ltGd1KHRObOPeMp2MP4pJi2U+fLoqKMoiTt38ADtuVz14fqHbZOVPKXGdI3hQ5X+np1etaWujLZDQtVr/fNN8ELxIwYvz+schbSZ3vxuOqUIuaBE2gLG3AOGbQkIBhiIUDyAjdDTEBVvDL+lvu+1dDjvqAaiVlKOiZMRlUWc5gNwg7bXgopMEh0t7effsx7TmTGd+xDzh90ah+L0QCzwzM+FR7Lotp13jgLZbGysTXHzNGd8cAW2q7COJcN6BcdJo7X/wycpBMXbYJIbn3s/6HiAvV8Wd/ISI8VodUjT5yYRvbfNlFXhw1ppTGKn2lfYQZ5MAwjIc/X+4fwI8JxSAYNMU+QAVcY0egXxDTvpvRsqddTE/0/Rcsgdp2VO/WdGJfu+Smf0LT0P9Wjf/lUy8nGt+O+f1WX++24KJfLgMvk8m9/K+Y/UgRtN2dUrSOdP+U3OGLTPCM2513hxBUhCa+Gqx8YecVoGMZOl8df60TXkSFdUJ3j X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 49fac329-03c1-403c-e66b-08dd50f49a35 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:24.9180 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: S268tu0+I+q50bDpfxc7JLYRJ5RuD9WyNRyU57DupAfZvVRLtif5SD+GOa0WDUguVibWJu5uDq/AsBy2300wkQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: 9KB3nlOKFlMHb_RZWsuKSzSlGY3dvWfo X-Proofpoint-ORIG-GUID: 9KB3nlOKFlMHb_RZWsuKSzSlGY3dvWfo X-Authority-Analysis: v=2.4 cv=d58PyQjE c=1 sm=1 tr=0 ts=67b5ef78 cx=c_pps a=4HTA/yvmbQxnAJvhdENERA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=64Cc0HZtAAAA:8 a=1ktPcj0v_YJIm58XXvYA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Allow attachment by explicitly passing a TYPE_VFIO_IOMMU_* string; vfio-user will use this later. Signed-off-by: John Levon --- hw/vfio/common.c | 30 +++++++++++++++++++----------- include/hw/vfio/vfio-common.h | 3 +++ 2 files changed, 22 insertions(+), 11 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index b49aafc40c..eefd735bc6 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1569,25 +1569,20 @@ retry: return info; } -bool vfio_attach_device(char *name, VFIODevice *vbasedev, - AddressSpace *as, Error **errp) +bool vfio_attach_device_by_iommu_type(const char *iommu_type, char *name, + VFIODevice *vbasedev, AddressSpace *as, + Error **errp) { - const VFIOIOMMUClass *ops = - VFIO_IOMMU_CLASS(object_class_by_name(TYPE_VFIO_IOMMU_LEGACY)); HostIOMMUDevice *hiod = NULL; - - if (vbasedev->iommufd) { - ops = VFIO_IOMMU_CLASS(object_class_by_name(TYPE_VFIO_IOMMU_IOMMUFD)); - } - - assert(ops); - + const VFIOIOMMUClass *ops = + VFIO_IOMMU_CLASS(object_class_by_name(iommu_type)); if (!vbasedev->mdev) { hiod = HOST_IOMMU_DEVICE(object_new(ops->hiod_typename)); vbasedev->hiod = hiod; } + if (!ops->attach_device(name, vbasedev, as, errp)) { object_unref(hiod); vbasedev->hiod = NULL; @@ -1597,6 +1592,19 @@ bool vfio_attach_device(char *name, VFIODevice *vbasedev, return true; } +bool vfio_attach_device(char *name, VFIODevice *vbasedev, + AddressSpace *as, Error **errp) +{ + const char *iommu_type = TYPE_VFIO_IOMMU_LEGACY; + + if (vbasedev->iommufd) { + iommu_type = TYPE_VFIO_IOMMU_IOMMUFD; + } + + return vfio_attach_device_by_iommu_type(iommu_type, name, vbasedev, + as, errp); +} + void vfio_detach_device(VFIODevice *vbasedev) { if (!vbasedev->bcontainer) { diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index f4f08eb8a6..c40f8de6bc 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -252,6 +252,9 @@ bool vfio_device_is_mdev(VFIODevice *vbasedev); bool vfio_device_hiod_realize(VFIODevice *vbasedev, Error **errp); bool vfio_attach_device(char *name, VFIODevice *vbasedev, AddressSpace *as, Error **errp); +bool vfio_attach_device_by_iommu_type(const char *iommu_type, char *name, + VFIODevice *vbasedev, AddressSpace *as, + Error **errp); void vfio_detach_device(VFIODevice *vbasedev); VFIODevice *vfio_get_vfio_device(Object *obj); From patchwork Wed Feb 19 14:48:35 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982419 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 775F6C021AA for ; Wed, 19 Feb 2025 14:55:49 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOM-0003IJ-EK; Wed, 19 Feb 2025 09:49:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOJ-0003HJ-HH; Wed, 19 Feb 2025 09:49:35 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOH-0007BS-F2; Wed, 19 Feb 2025 09:49:35 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9AX1j005473; Wed, 19 Feb 2025 06:49:29 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=HQdSA7x38g8YEsjh3cyKdpWtPJB9DO8ho7wS3JbgW V0=; b=LGweDeW7voL2bp7exPHQjFgAEsUN98wd0TcjR/9sxhFjm0juwZh2/2ux/ cbuufqvYluOh3ca03LV4LY/qdphWyE3QBpoBDaNHH00d2vjHTkre2dKA2jsIFw89 EIrHUhQiNl1cgQPpNKTHyEDTXXvc9YGNRR69bQ5QvZjZSNGW9Gv0xQ0e5Whm1Rm0 +v7uqBADv4aYqFubK6eUlNLYNHYNpPpxvP6wjyFUBzvzfAK9L5t1u9C+rkhoWuYR VNhhWMKOKSpL5SujEN94aVbZEUGWRGnemQTGlHcr2QefRqt6vXJkSb32rLYhA7x0 dmtkImHdyCBuphNh+u6udhiub4/og== Received: from ch4pr04cu002.outbound.protection.outlook.com (mail-northcentralusazlp17013063.outbound.protection.outlook.com [40.93.20.63]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxr5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:29 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=mt8Nt6WVqi+3CjOsgT+LkTs2SLyN+wiM+JoWP4y72wQi9vUj57bnaYzw0lsdy+YWSLIGIFCjYE98V7fLjM2KqLREi2r4lFTyYAv8BgJUAcUZ/ePkE4CNLPWN6EYC4k9548sL1INlD4VBBWZjdDk8rYXqeJV7NH3uj2+NL3C4S4XQkLClgB6nldJbz/lSn2tVQ9/NGs906LOXJNXBNUlDsmVbT0H8PhbVttbzzfH7kDUpn6Ye8aU8UnzbGsdIfir0TXtJGAcwOw+He3neuK4NKDnTwg7BbAVe499G2dfk6QH9lBfPHLPAPc6ohqGHbAN8wwoJ/MMwiiN/9KEsGurD/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HQdSA7x38g8YEsjh3cyKdpWtPJB9DO8ho7wS3JbgWV0=; b=tu2P5qG4TXlYj1Y985T/gVO/J/88QgXUnGZ5QiYJNDDqDnK+ezAMlDby0T3Mml7IgAFxqhXrqVUCKOrbghAYo+GNR3CUWZ7jmxFel9kovscw2I5yycz2hbV4p6RwGyU9Pm/RGbpAMWVTxutl0UD4edyyCXm4ugeEhjzJ6/G5z4R8jyzzCAXN1sKv+XuKthJoSbMKDCAIbxQpPdlm+tfp7XBfM6M/9PqK2VEZ1B19N6fVkOhXStY2+XQLxiet1qCOYK6jW+XGJNHbR4pkAuXdmjCHs9Jrfq1h2Ijv1W9alS0qdHlXEd/8QjShkTzJiGHbLrCYI2qXYT+7WMJmT9wo0w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HQdSA7x38g8YEsjh3cyKdpWtPJB9DO8ho7wS3JbgWV0=; b=KjAGC4yJ7t6fROLhlPHWJJLF5XYm2FF3q0JZMgljfslaQq2Z2C4FesLO0IYup3mMNcDdyQX1jKl3dvde+K/gCyM6eOFtAFbccFA5INC5DxbI/VFXdrSqq8mGbZrn6HS59tvfnNgZIRLSm+9m7B56/0eYbU0XUvQiYvPEqDmM9SUDJKS9N+y048PtB1VfgDCcGVYWrnFCDIoWHL8qz2Z/2nlM/nSji5ctxBOhoRCS8eFitLrmNFE+qGlZblO0jkIzhecnk7L15qk0JjBbIaLXP6KvOEUQkW8rKOnPYazk2zcd1GGfGlHrfPHEFk4Ub7hpjhkKrerEpM02z6yy+CbIlg== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:28 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:27 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" Subject: [PATCH v8 05/28] vfio: add vfio_prepare_device() Date: Wed, 19 Feb 2025 15:48:35 +0100 Message-Id: <20250219144858.266455-6-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 92cfc5eb-cbdc-4ac1-7567-08dd50f49c18 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: eCPDiyTBFPu66P4VbXa8ABtn4Joxv0kVuEf01Q4oS6YW85kdNw7uPrLCEqRZNeGO/nbIpnLIiSKdaFncdq9TRrOK2vzevu4W064tW2WZMJyyTlmVmF4eZzKLeoDwrO1haSjyk7NM7VFUDf0yUWeDRmHXwQsF5TaMbsWFcBYxZd/Tp0SqeLV6oKXsmyy67TnJzYBG8jZs+Do6axhsJQdBQzBC6YqzlUkK4ZqOHUhP2lok678ER+raYmSLuwa/HWx3hyHJe7reXq2SnlblxEdtLBCKmN2A0yTB7enK2dx52JulpMcZd5+nnB4S/G1wcMDGe0VTia7RNv/KitF/LXMeL70irAk9hljgBkkwJGjU0ax670ujqsAtgtRUf9jWMSd4WkcR4uFy0x3+lyBDnsC1uCiu96xsQ8WLEP5qaZSQHxI7tXu6MWqaq9NxXmEcGIKwbRnPPOqNksO3ycEkFJt85ImaZrYL6jL9J88ydvjQahvvvbZHfxqqWzx2EUVwWlMEHa3CU4b8JuLruKTi87NWkvI0cJk/OPdv/uiQQ+CDEx0d3vEHfpA/LnbEBdR92JM0CpYkNGmqpqmtgKo46awL9vWVmRrM0yDFm2FUtr9//5Z4/q9EfN3xD7TRy/LrfPCa6gm/uFnE5ut6Y8Qh7VJmhg4qnL6XJO9YLKvHJhSoTpMU04tNktbNvNrtPcpSE87m/NtXLPZPMVnhQM3qPfu0Q+35DxO+LMT9VjObzFvUVMrW9XyAB7ZLdKLnvFcshdB3dhvyQIyM+anLm+VhnBwHuEcqbPWvPEUFLryBDF7IX54hgCTfZoJFKyZ0pDrAOQtBou2xDyhdslVssgwa/BgdeMJpe5dqYqojo2MGLS/MLzkOKBeBmZxLBhgu+vnZXfWSTZiCHiaSm+btR78cSsLnrVtdPtEV/+11dWsVdw1cDtb9KZoXr7+djrB/o1nDh663ogPUy2nQrFx1vtVImAQtmCNQyJYBTRIw2eqEdsF4ZSs4K6oPzSoTwrbHEjaq5OfDYHn8+n5Nc5yFADpkOIGFWOATDrv/ZpKYC2xb0q3bgVppQdBS+zRWjxku+5Wp/9qxAAubMkX18d89BPLaPXdO9fV+urUNs7QAHZdnRfT9ZBzXDLl8C64KxYjF4qnoCK0M0k2pCD/8qimHGUUbjNwQULQgnbpHj+s63XNeFetXSyvjqB4FIGiBLunNOniuAPwP8xgsMpYnlO2LzzYnp6L9tzc5fzRd1kTZkSkCgBJ4NUHjRxJ1MSwYpvQs5WWG0rKs9ReS/Zfi4RtDKaZM2TNlS3pmHB7MXpYivfb5axbfJMkfU3hfCX2Ng0IVhTlHUbHMhTLvIDTLDe0f6HnvoNU4BezZeriIQuDbfLuTE6TX7UwZdh6IOT740Bx+eqAhE79T X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: iN3ieMTC2RN3av4PfQQyQtoLYv2qwp6scf+NeSMTp8P4w5PjYRaUIyFXiz5bnnUgq4aBsoS6TCcAXrGLsrXYHpwMRn4AcrHJz3ReY55ISRvhliJGwHpDPVwWUGrwTCed9Ihhj729sFHtFMxcxOYdP5EAwyOrEQgPWqQaLr40e2HVPWv2+gO9m0gnFWzLxODguGbcl00yT7rgDdg7knPPNRiRXj5eKvHkLRezEn/IZg55byVDvshrGpLeKbn3uTlxIl/eFHK8I+vFneHkzyBdN/u0FMMEYQhtT0b9Gz+L18ywWcUKhg/rcvDLnEx/nixiOKb+WPKbLUppWg2XDBWo5HUNMGZC2VrWc9MqgHj0bP+lLdG0V2rL2FF3icpDRpwByWU944BYC4wMfpUtpWTynn+Bled16/Qf2ZrILxqbagE1CJuVILZMnWgMr2LDZ5PSuDyIungL8Izv9+KZuXOCIH1r6ivxQgYMO4OD22AN/SD99FJR+Bqs//LpEqzWqDoLu4ONiF2HvlyTToFqvCmUdXJQ39Tz1yZtB1dhDQs+zgw9TFNarmuP+A15B9ng9fUMJGa1h3QoLn9Z8RCluqfFeMdkAT3mRw0Pyi3GSHsWuJDBEoC/qIbKkSUtG9SkaucPPs+wxkiimCnSeLXAvaFyL578+mZQphqIGpbKP7CrFpIx239m8BY8S7sPcqE3ADKMUsmu/sQyJUXfaNBFkynz0NOKEnQIEDmk16Zkvv2FvwoQVmNZwNc3Piar0oTbwx2qGU3HKKwHLhu5kgU9l8oaxhhCdZr4B7U88r2iBUVNFGyNmFOyCujg7NFhDkQCIALwBbFGpqgSkW8h3bWAWwP7hCPOWalNANrZUe1nSQ+vqtaR+jrkaY6YTHr3tSvfQG22ybtj+lXnIvX68kMx28VPDWe0DHSkuCHT4jVw+GljmTZiTmtgG0AqfDW9eSCdEIsNCX1B/YB1WCm755mt+7yNGQQpqGHD8/8h3djcKIsG2vzVuVOmH5lNjsrJBZKe2AUpjmD62TrMeiFQqLZ06pqwV+mFvji0Hsi/utdJTayXu6QgVWGsU9dLHvVzCIdnue4BNjAI1LjB1p5d6QaFN5v96ZlJb3HG488g57M2SSHbO9BObgi9HoCBiXjCWfmQD6DiBD3qVqdQ4Vy3W2alAaj6H1Ga2MdBoyQtcAylI4XW2TitVIzUINOpNGeqhLDvTyBK0+U56zZIlP1n/Ik7HSKVfF19NTqwONE49POXKKQ1ulLIMO2YW0ls7f5NwHTf/QfR8NZp+xcgeoRlmpUX6kqYr6/pFPpPQDvvScwCej1/1Tp/AimUDFNSjiJZJjmo61/zhMTncM/Uc/ZZt/IJyHFOeHgeZr8D8T7BX33wyeNjOED43HmB/UmuKHH81QRcWgdeHMvHoXDrh1S+xdVfwnFKlZMSLKb8JEQCEMefqpGHPGrC25VGtvX+dXw8P3rp5tqMQldu1oSzw+3C4fFHTXpotVX3TJyuH3mkgoUXl3QX+8F7MAj9ozYLeCGBn1iIywsc8RTArf16Jz0GfWwsmyG6a0zhFkyWARNc1y2U7DaTXijpLkwXHB53eH5ruwAS55E2 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 92cfc5eb-cbdc-4ac1-7567-08dd50f49c18 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:27.8789 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zJ3QELh6jZyod5bxmvijoqIywWrdI8i0KBwVm+z7nyOXQ7rmuW+/OWV+54TS7uxiwIeTmHjhO1mew0vDHP8Rdw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: WRd8pl8y2CcjgXEDh_mkVUAWLrtZohUl X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5ef79 cx=c_pps a=FN31aPQdsEfnd7fT0GsFIA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=64Cc0HZtAAAA:8 a=nsq3sfgFcGYXhcUyotgA:9 a=YF8NTHzWcJO_nPwQaAZU:22 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: WRd8pl8y2CcjgXEDh_mkVUAWLrtZohUl X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Commonize some initialization code shared by the legacy and iommufd vfio implementations (and later by vfio-user). Signed-off-by: John Levon --- hw/vfio/common.c | 19 +++++++++++++++++++ hw/vfio/container.c | 14 +------------- hw/vfio/iommufd.c | 9 +-------- include/hw/vfio/vfio-common.h | 2 ++ 4 files changed, 23 insertions(+), 21 deletions(-) diff --git a/hw/vfio/common.c b/hw/vfio/common.c index eefd735bc6..4434e0a0a2 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1569,6 +1569,25 @@ retry: return info; } +void vfio_prepare_device(VFIODevice *vbasedev, VFIOContainerBase *bcontainer, + VFIOGroup *group, struct vfio_device_info *info) +{ + vbasedev->group = group; + + vbasedev->num_irqs = info->num_irqs; + vbasedev->num_regions = info->num_regions; + vbasedev->flags = info->flags; + vbasedev->reset_works = !!(info->flags & VFIO_DEVICE_FLAGS_RESET); + + vbasedev->bcontainer = bcontainer; + QLIST_INSERT_HEAD(&bcontainer->device_list, vbasedev, container_next); + if (group) { + QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); + } + + QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); +} + bool vfio_attach_device_by_iommu_type(const char *iommu_type, char *name, VFIODevice *vbasedev, AddressSpace *as, Error **errp) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 82987063e5..37a3befbc5 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -876,17 +876,11 @@ static bool vfio_get_device(VFIOGroup *group, const char *name, } vbasedev->fd = fd; - vbasedev->group = group; - QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); - vbasedev->num_irqs = info->num_irqs; - vbasedev->num_regions = info->num_regions; - vbasedev->flags = info->flags; + vfio_prepare_device(vbasedev, &group->container->bcontainer, group, info); trace_vfio_get_device(name, info->flags, info->num_regions, info->num_irqs); - vbasedev->reset_works = !!(info->flags & VFIO_DEVICE_FLAGS_RESET); - return true; } @@ -939,7 +933,6 @@ static bool vfio_legacy_attach_device(const char *name, VFIODevice *vbasedev, int groupid = vfio_device_groupid(vbasedev, errp); VFIODevice *vbasedev_iter; VFIOGroup *group; - VFIOContainerBase *bcontainer; if (groupid < 0) { return false; @@ -968,11 +961,6 @@ static bool vfio_legacy_attach_device(const char *name, VFIODevice *vbasedev, return false; } - bcontainer = &group->container->bcontainer; - vbasedev->bcontainer = bcontainer; - QLIST_INSERT_HEAD(&bcontainer->device_list, vbasedev, container_next); - QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); - return true; } diff --git a/hw/vfio/iommufd.c b/hw/vfio/iommufd.c index e295f251c0..85c70eae37 100644 --- a/hw/vfio/iommufd.c +++ b/hw/vfio/iommufd.c @@ -604,14 +604,7 @@ found_container: iommufd_cdev_ram_block_discard_disable(false); } - vbasedev->group = 0; - vbasedev->num_irqs = dev_info.num_irqs; - vbasedev->num_regions = dev_info.num_regions; - vbasedev->flags = dev_info.flags; - vbasedev->reset_works = !!(dev_info.flags & VFIO_DEVICE_FLAGS_RESET); - vbasedev->bcontainer = bcontainer; - QLIST_INSERT_HEAD(&bcontainer->device_list, vbasedev, container_next); - QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); + vfio_prepare_device(vbasedev, bcontainer, NULL, &dev_info); trace_iommufd_cdev_device_info(vbasedev->name, devfd, vbasedev->num_irqs, vbasedev->num_regions, vbasedev->flags); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index c40f8de6bc..ae3ecbd9f6 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -250,6 +250,8 @@ void vfio_reset_handler(void *opaque); struct vfio_device_info *vfio_get_device_info(int fd); bool vfio_device_is_mdev(VFIODevice *vbasedev); bool vfio_device_hiod_realize(VFIODevice *vbasedev, Error **errp); +void vfio_prepare_device(VFIODevice *vbasedev, VFIOContainerBase *bcontainer, + VFIOGroup *group, struct vfio_device_info *info); bool vfio_attach_device(char *name, VFIODevice *vbasedev, AddressSpace *as, Error **errp); bool vfio_attach_device_by_iommu_type(const char *iommu_type, char *name, From patchwork Wed Feb 19 14:48:36 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982384 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A05BDC021B1 for ; Wed, 19 Feb 2025 14:50:42 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOO-0003J8-8l; Wed, 19 Feb 2025 09:49:40 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOL-0003I5-H1; Wed, 19 Feb 2025 09:49:37 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOJ-0007CD-Oz; Wed, 19 Feb 2025 09:49:37 -0500 Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9Lst3011379; Wed, 19 Feb 2025 06:49:32 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=YNuD0PXGBECTDc5z781n3QjmM1jjnUtEiN7jRkmCD kM=; b=uZ+QglhlngsUS9bx84qz0gEDFDJDkw/H+9GTuc1Yt3orb2e4mgxUoq0lc DNizl4XYcCMX3FlzmJw5rn8+id4aO0FzK1XC3gPzVEANntxjEIMtOu1VoBGELNmp KPZkjov/wzI+pZX/EGusk0BCLv+3yoCtLJIYBEgAANw9sVIyTnS6dcZhXTsP6A5v 4X5LNkgfhZzK1i9iGi6tTBbmTuZAvzb9DkltuM/EmcxFDkRWejjl2XxdNiaFsmti R96xZLYAeYOREoH7IPoSE+P6SgpNNo9G8sm8HwJZr6lpwY6CepfwPD8o8tZrNpb/ CkKV0gKgMAs5A44Xde2Pt7nw3Zl2w== Received: from ch4pr04cu002.outbound.protection.outlook.com (mail-northcentralusazlp17013058.outbound.protection.outlook.com [40.93.20.58]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4bdhxw4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:32 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DXtdbFZS63kM4iNC0IypNRyAExWFuiIn/ma8T6SobtnTKJSJLUad07tyNnv67v8At1syBZoX7w/t4arE8UulUjEPvwQLUgj14+XSY3Iev3/3y0RKRQVly3nEerKPfQsSg9z5FVnFgUhpDdHyzDhqdefWmvJiyK89rRY1Xs9UoiaUBJwYUwLCOkIDP5LHFEfqZnJiA0nIeFBzkFayeGY9dK695vs4DH8gh0LTf0UP8evvun8J5kYSAqSYr4Yjrdh0wUuKwzS3pDF5sXa2mt+2aXx/VigN1kCA0nV6tkRapw/qje3wxXR9eY/cD+nqwhNWyWVkH9JizBxAwk2M27c+MQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=YNuD0PXGBECTDc5z781n3QjmM1jjnUtEiN7jRkmCDkM=; b=XD4hheFfsTmDIV1OmSdG+HrIc6t7Rw6RT3fmggCd56kV5Slsc+FUSdyQTYDbEuHw19M70ihQYO4lal3jvpSUiKUU0dYuP01WZbxO7quAKGmOTcCzQRMLlJtTEXaRMat/Ze6nflIpDBTRccC2C/KLuAnmCjL0TGUItU7hrZEI5F1S7kCkaSyAP9kMNsoakZrF4O23oO/XXkcJFPRTjrQniK0caNnzmNL62FjUc/xTm3nMcL6ChNSKMCs8lYPMrLyeaUocw8ykPtSl73puT8M/TQdPt8z4npIDNK/piZdjPvbaC7dxbXV76TKWTTGqMA3l8yKWhCQHzgffrCMK41hRQg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YNuD0PXGBECTDc5z781n3QjmM1jjnUtEiN7jRkmCDkM=; b=kIW0zSEtizFtEHBFp0lr8dQje9QAt2In/Xdkyn2/yTVm3VAll/hsgfpBY/d9xPbSC7+F2Bt6JTxvhG2spvHxABXQH51a+P1r10cV/A6KWBgxFmzC+ynI/c5Ehufcmi42KSpG5b+HGLE0EjkY4EhbbO8zWcT7TMOei64LNQludz8OHDe2sQma/XaFVM0r94oAwc8jsa44rqPvpWsx/Kp6mnQx+YWLVrUXaMCSJPBjtAHpt7OrZWGq69U8140PINSHbk1AzgJVkapcalT3NVl4BpW/6rw4GJqgkbpP1ny5/6NK8F4IYZBaVPKyT4UVJkzbLr17Twt/mvCCxYfxFzhofQ== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:31 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:30 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" Subject: [PATCH v8 06/28] vfio: refactor out vfio_interrupt_setup() Date: Wed, 19 Feb 2025 15:48:36 +0100 Message-Id: <20250219144858.266455-7-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 4d99be53-0033-48f1-d27a-08dd50f49dde x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: GtqoGq291ecKt4Ocu8S/DYQY/ZbTttsaVgYjz7F5wg1tWq8/aed993+wt1fpM6oA49k0ACSG/o+gOOv7m01YZdY8UU1Sk1IBxVFC6RjrfLZogUOJkZ4W528Ebtixy+vHRNSZxH/lC5UpNs5+pL8wJzpKR7qBr6ICIR3QlUo1Sa7MfNIUrMceHvbxv6W3Fwy+rd8TyP2BE/PfRDGLyWxs+Mo/BKP/CZ3VXJaBbjWywQKMqZFtBEOq7fKY2GzvlMTF3+RLUXI1oxC/3nxvONbgmGels+OgjwTQpR2YRb3KTDWWl/t8NLNJm4N6siyirGEzQfa4GymAwdsl4y0pt9cLXdpNspA6IEG32bZtZme2ZLc1Iw/YXaCwvEO/qHSXwJSkkaRAq2KF5zvTgBP4oPOH2Rf2FwrexKkfXwwr5cFsDAJTM+5IXz8RBijpYlphkC/06cX5UAu/npVn7mXDnc38KmADsF+JT+tvqITgPsaARmtP1/8YXeTjNwQvYcFg3acynx/VAqOn4AVRqOYRpHceFkSYtdtwOrsQmpqf7ggnw0jF08BXf0ehckayiv0ityAY/EgUfm4jrmrTib8dH7lKCZBCoW28EKn5f1kEmud+W+I6ua/KCzz8SlhMJ2USU19kxbwDEE8r6I/mqBl9GUjggPPE75El7rsbK1wva0N0RZuP/uimT4M4PVINO5b3yJrS8e4BnuGw46S5OtXnSUElfsY/mV4Ht9a5hcjOqXgCDJVlIP4ppAZybFNbZDObxbDLvC9znzRjZ/cSSif7x7yEuuLLzKikIpZpmqf/Lk4jRMLBH0RAGhpcZ/oTDvSvoMthNS9TPYfpeMzzTxFykQX/t7mOLroeZLmKv2UeGViVR0uJfsZz3dKTUsIQf2XQspVbj00bPN/72Bu/bCwyWjXU0PSPMK9l9ncV4dZfUfP/6PPIopba7JsO9pLXiid7Uswo5W+PR1sDCyQMy0vRDoO1vIfKGjjE2cbY7rmLLiHzYqvPd8aO0WOxdSfEs4QDcBfoZ51qsMSyNmiRjfOD8iixA0CkK4cGp3GCnJx3LNzzdJB7JAdaDAN2WAtHvXCV5YLR9fzrKYgoP7UmU5FjbgnEBpVSd7+f+Wva75FvN5tietuUJ/quTYpDIzdDok3vsxgOgpHQbMAtyceECAMcz++Y+b9SaWTmWDQJb3qrE9GOQ51Bzx04nwtNXPvWuQalvHK2GzonF2Xh7zOGRrLsV5hrh0TG44KxiAAsWSQXCb7MPf8nFHGRsEtG5l5UNzWP/pk1Zi9AWnJQMHDr08o46a158tPkYYjmkznFReLNyBjvplHAPYbcWJ3ZES1f76e4V6w4INVjfBP2T63+r7z4oLMz7u2vjrCOHn9jyJhepub9YfFg29jy5civlKaj94BFMQb8 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: UyjD1VW3rnMLke6eUe+ssZXfdfV+zjrIG9WD0jPnfoRr8pda18E9ing6AvjftVHVNbp8ouVEqQYfldvYmfUPXK9KfJf93L6sO5Y8fKdOYDvbInCYEAZFXWMNVfo0hvdop7I1Z0W7gQUkbjyEggTR2l544bEKfBhFE3+C1y3L2dPq/1jukXEFs92GUP//iwpD58Jxn6kS8FQ+5sZNQ4qiUdICxCqiBJGcRcjwgMJFjeEwURnR58NnOwFQxX2I3uN2f/Za2JXJPlauha+tNUnGDqEm2GdbiTaM1gny6yIU68yfL9yUV6fBAPyoiu/9e2ON2iH/tPH2Tl5guI9kl+e+ovKUu35JCkqzr/8o4CTU1X98e7KSUpL668syZ3UgR9kIoLGi3X02hiX4D02e+Zzq9jKTQshsexYarb9oxLdGAY4UFtk4q5YsILt50xgdrvQJku0EIGKrpESnrZt/hJEFv/56XKP96Vf5GhVo7Omeir6adQJ1DAI9GfK/ckbXJBr3mOkYPixekbDVUweG5Ea9Wbyon5Qhwq8wNQioor5fDLVk8bjmee7m8RjWwlQX1LVtimuUB3LIzhJujJeitZK/zOc5UFzP7HCHwT2yD1ETfABgRf6ZJux0g7+0tsH6dZGWnVEszxkvtpB4WHmj+2arYrkZL2sVrO0NvS23hqKKzO6k3kei5zpYcq+hhWIEbbmlgP9pOlvcM0qE5BCOsNvC7NQGptdr3gcTy8ghesZbovbvZpeBooMwmM7EUIWBoGbqqO86X98O6JuQuZ1bwT4BvY9bJnQVcrEOqLsddIxmOq2LzU+dzWUci2BEfUphKLjGtARh45Fi9z0of5O9guRdbCUbj2vr83LHxy+DBr5ReiFmQ4whdUOsmYSP/beOTkABTN/E7haUb0Pkj9PLdz9LWnR0hmb9cIX5arSpPZeNse6Wlq0khOIQUy9tFOzAgeMt7R9kVJdBTqqfSYvcAL/szVGqlT7GPXQmnaxqtARZMm8m6YL3c2l5YIHOewUnBTqtYiODsg/hXPkmBnYe+rT+c4GwLqi/kWxCdP7/lLMsdiF6NR4z5bQLdIEBUGpm1YGTNoK8aTsm3Vhr9DbbPtjBdzj5ITIdH1liIC15iX2wgnWPj9w1+a5NNmbPQ0eufjLj2MJwzVoXnd4zsb2sZYxcvfa+dAaZEiwEgwe0mYFC0icSrp3bW8kxI3JYmnQZf6W1Tsd9KfQ8S6u/0XSy0gm1qmF58PU7OslO1pD50AokpupyxFmIPI+kAUhfpe0VjYuNJQ+deUjTG6o5QtiKiz+TInnMSmOxnu57ygZ34n5CAqqgRYbGFh0vM3qcWJaa3em1Pa1p544YjvA16sYsplaEjqe8snm1uqPwc4pZeld3pxKRjelGSUOGyUbtGWaM1NGsLdiNmBIn3mJ6vxVEmRBoxzBXznq3G48SjbFjeCsKNeWU6dPJdOPtoCBiHhPVVh7pLpYhC9I2Jt6ybZHPGNzRDRTTLADIJXI9VrwhMg1mFRkvIeSdC2nRjrZU/0i111ndvRDh8DMKw3xRjsbI/Cu2kSRJL6pAqn02YeTEU1mA7/bTmsjQdwMONtaCPOlpmv7B X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4d99be53-0033-48f1-d27a-08dd50f49dde X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:30.8629 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8se9jg8cA6yxGTzf6Qc6AxA6JZdq0m1ARKmkSQ2aizsyZswYetstgF6fiSYcT6ZUdFY9NNqr9Ff4ikrMKdmBgQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-ORIG-GUID: SH-KHXSU54c47Hk7u8uaT5MM1htHN_G5 X-Proofpoint-GUID: SH-KHXSU54c47Hk7u8uaT5MM1htHN_G5 X-Authority-Analysis: v=2.4 cv=Fo7//3rq c=1 sm=1 tr=0 ts=67b5ef7c cx=c_pps a=UiiUhvOI59TtQsb/yF5oqw==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=64Cc0HZtAAAA:8 a=DOYYwuSLhGa-iB6POOMA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Refactor the interrupt setup code out of vfio_realize(), as we will later need this for vfio-user too. Signed-off-by: John Levon --- hw/vfio/pci.c | 54 +++++++++++++++++++++++++++++++-------------------- 1 file changed, 33 insertions(+), 21 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 89d900e9cf..5fb6c4c4c6 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2957,6 +2957,37 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } +static bool vfio_interrupt_setup(VFIOPCIDevice *vdev, Error **errp) +{ + PCIDevice *pdev = &vdev->pdev; + + /* QEMU emulates all of MSI & MSIX */ + if (pdev->cap_present & QEMU_PCI_CAP_MSIX) { + memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff, + MSIX_CAP_LENGTH); + } + + if (pdev->cap_present & QEMU_PCI_CAP_MSI) { + memset(vdev->emulated_config_bits + pdev->msi_cap, 0xff, + vdev->msi_cap_size); + } + + if (vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1)) { + vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, + vfio_intx_mmap_enable, vdev); + pci_device_set_intx_routing_notifier(&vdev->pdev, + vfio_intx_routing_notifier); + vdev->irqchip_change_notifier.notify = vfio_irqchip_change; + kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); + if (!vfio_intx_enable(vdev, errp)) { + pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); + kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); + return false; + } + } + return true; +} + static void vfio_realize(PCIDevice *pdev, Error **errp) { ERRP_GUARD(); @@ -3157,27 +3188,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } } - /* QEMU emulates all of MSI & MSIX */ - if (pdev->cap_present & QEMU_PCI_CAP_MSIX) { - memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff, - MSIX_CAP_LENGTH); - } - - if (pdev->cap_present & QEMU_PCI_CAP_MSI) { - memset(vdev->emulated_config_bits + pdev->msi_cap, 0xff, - vdev->msi_cap_size); - } - - if (vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1)) { - vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, - vfio_intx_mmap_enable, vdev); - pci_device_set_intx_routing_notifier(&vdev->pdev, - vfio_intx_routing_notifier); - vdev->irqchip_change_notifier.notify = vfio_irqchip_change; - kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); - if (!vfio_intx_enable(vdev, errp)) { - goto out_deregister; - } + if (!vfio_interrupt_setup(vdev, errp)) { + goto out_teardown; } if (vdev->display != ON_OFF_AUTO_OFF) { From patchwork Wed Feb 19 14:48:37 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982417 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9651DC021B0 for ; Wed, 19 Feb 2025 14:55:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOR-0003Jl-R6; Wed, 19 Feb 2025 09:49:43 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOQ-0003JI-2q; Wed, 19 Feb 2025 09:49:42 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOO-0007D1-1a; Wed, 19 Feb 2025 09:49:41 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9Lbhp026947; Wed, 19 Feb 2025 06:49:36 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=Pd+nhoCV4Dxd9tMdn0X+FSv/i/j1nvi4kOe9UsWGf A8=; b=YiQMOJjTArZVAL+9rBO/7zvyxd4n8JXnD0i0qIHOBWBT5jy4hw+sKyBcE udxRLqfOz6f9Fkt2jd3BZGzJTPeAdaYs1Qu42++IE13cOz5usLybRcfyFpnjIDlA seYMdoYxB6GTu1HEZZrRM4tu51DBr0trMKeUxooWxi9IkXPiGgbREY36PV6FHlcB 5DL+D73QDu+RYRqtsGlJRbhANqldWuTnaXmurUeKrk6l6UVtYmmOWVfb+22n8Xfo jJOn9b7PnOUQ0XwBDW3qoiKzoc7bD9Ta1ZAF3w2aFPOM1aZzg3p4+mbL9s+Q5is9 PGyuGCrnDuu69VQYK50ZgeBhoSbJw== Received: from ch4pr04cu002.outbound.protection.outlook.com (mail-northcentralusazlp17013058.outbound.protection.outlook.com [40.93.20.58]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxrd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:36 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=uHX30EZTB/ONnmfuz3vKMelYdIdyDatc4+3aXL6rSqps+NtJCpvvRW39LOKrbV04rMLmGRwk0gdme8tyaNaROdMqxIYt2bpCXpKDPiV0O7r9qL4unYitLYyp9wzP8WZcvH5JhgMZ5Ek2KJM98xbzVYtS/0lcpQDeGgzXu0AwqAziz3rdoD2nyFGGV+nkNyX/+eoCDQiQweCEBdfrYCHnEcAI5EpcFi985Kz7esadeghgEVLJ5BshWcvalUPval5As17jactJnLdaKBewwPxM8zRwQQiL6mi7rc4rJMb9F6PMo+0gagYJf+pK2DoOfAJnWt3oU4bb1ZrnQlXn+m+Zgg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Pd+nhoCV4Dxd9tMdn0X+FSv/i/j1nvi4kOe9UsWGfA8=; b=Uw6uk5rA28eG0cPel5dFrj2M7Pe9sWiwUsHCddGLO9APCYBFvQCA54tEY17e3Dux50wrLhsiiroh4bj/96LCVNtr4knRB+3bHIs/yj+PTPiny8MrM/k1adXbylROpPQ9/AE0pZKpdi0ZfCdRwiyi471k2isbCTjOyuXgNiXVLF8t6po2YHk45sLvrdDYvo2XR+8vyNaTI/m2+n1QUmyJOhrVRu/mtpmTf5EwumsqTPLupB18mOENiVvoo+wHptDgsRoKFJSJNFUwpX8xqXthJ/ONrGR6At7hHEvO6ORKbw9vaeIamyKHuW2AiJWuZ/4x8I9QA0aaoqpdqaCRawCoog== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Pd+nhoCV4Dxd9tMdn0X+FSv/i/j1nvi4kOe9UsWGfA8=; b=j1k9ZKb02OrenmD02OxFeXeKX2oacz5D10EmdupBb7Jzh0fJbIq3ONJowhtWC8n7001It2u9vzaQkrhXEVvuZ50ijRelmHDxROUdS5qaLITdXTIJF8MoVdpdl6gLr4oV949RmRLAUgV7+iqQCyVycw2HdYDtz9thP6Cro5ba6D2/exW6v7EW8h0hUk9VQHvsq8OeakBJwH/jEjN7cuJOuZQsh1R/nZBF9lhTZoJDRRyG53fEjsHlHVYB38wGvOKo3kvQZHHqHVZBGZvIYR12SGg3OuGVQPWmozPsniXPKYZ4SziajeFJHGT4fz4Qo4j/z1SeHAclL7cpZbG6ihFRlw== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:34 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:34 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" Subject: [PATCH v8 07/28] vfio: refactor out vfio_pci_config_setup() Date: Wed, 19 Feb 2025 15:48:37 +0100 Message-Id: <20250219144858.266455-8-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: e0583152-8f81-4db9-6c92-08dd50f49fad x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: 9GKgCVtWMlbf2lPemHQ3qNbLOiDF95NkTHW4TX2Y3g+JqGoWq/WMHzL8XZtlGSsyOiK2BinGgtJYj9iVN+4nmoiDDFegtIKeVFkdJY5bYx7dgKNekfSrvuK3LS1VTsBUnBDCua4frdjfth9mjRsVAIlQsij9SxKk1zP+1xH3dWQPpZkF4U2Tx+3IBlYRHKjmfQo2SZeoJfpYMl6s7eQvCzKx2oj/zA1Qg5YusZZr6AjBO3nJaDRmhutA91CMjjfzwKyZF3dEn1BMJMf6jixM+A0bTtp845QozNm0US2R7Z8eEJWzix7SnvF9UVXkoc7I8M6nFWUSmtwB4+KciIE2xVo9+usOpSjQuMufZrQHow/JPovhVxbuJ1jtyRgRSQdJm2SJNbz3QWL5wfdztEgwpBibKr0LufuUllZjaL+BbP9z0aWk+gdttdEKiXlbhDPjb2OgDe0Kjk7Ur8G5S0I/+l9fJinra7QnIfwyHbrKuHfVbUtoo63xwfCZqQEx8g04Lxn0PdRTbwi3Kw9cAOYrSvfGvMKxoy7ElszU3lTfqD6wydtprlKGcdIpz/KBccjx6UCZeUaq5DMkSpLZFFCu1D5CgahlB1OJyjW36meXMxpnOOZGn4QGs1gPbmbjPt9D8wed8HQjmXNYUIAW96HPkgirCHhgbraFCiTSMyemNsiKxRyjKt2SuaqfrIVZSBl08mPyT3DEqGJ5cHQY0GnjCZXWtZbxJo7iby1zWvioPEC9sOAE9uEsHSnDL+/kFRx/fDE0Y0qqHCKM9DqIgQREphO+PXtKTJ836ZwQKfOfN7da1M2V5Fbc7liEd8rPr0ADhl64pki0e770mEvwNe+YeRHhZSpup/OoTiOm9InkVKJaEyh77FDZ4PcU1OjMEpZk7xfYvCD3butELQ0Vvd8J1jlKgF5nDIVYaSeo2nIcvObY0Yc/QOaLZQr5k5LA4ybSH3uYVSShBnZPJJIZPG76V1RcRP6DXi0RmwbZeDwPbk1n3oLdLv1UuHnDFEW9k0bvLORQ5TsDEgjy8UB2Y/ztxQwxOvz4LdOJ+WzG74UFH1cdH6N8zNTK6Rj10uplqqygdIr6cIBJ5LiRLROrGGIB6pF07GTn5PhfJotAW/JzkyDFlz6ZCwUEtSA1ghplUSkiMMvyGQN77WKloU6DxFAvZFnU48P6bR3Wm+sAHhX8MglD2uFerSEUno0IzXCD6hvGdjXs2ZgwufxAdFRJiTRF8dZCasDOBUNPB8MSASZGHRJPbWtVryE2S0x/suS+loP3pG3Sb0i5yy5qBWM6cwq1c7OSyu55uM5ZTnzmMo1wyWyscWv+IUEj9bjeqbQ4zcEpoEzaQj9QJzOEwuaD/gspoT/wR3ZeFiK4u/qPEcDdOaCMSyn+VkkPNeAF7bwHft46 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 2cwEM+6cgF7kRXtkrBhp6xQ+qgLXPg6EUao1anC17UlOSWsVnsO4K4ZL+A+XDsRm/2MhKh1I61/fZHzRKbRDiu/BXL1IJdVa9BfVEPKd198c4NAJEs8pu2+diMmz+Du8mhPiCHdbSDP0RRINzZg3iEkbgwxVMaqNs/eDO/45GhLS5DByITryZM60fzfdiLiJEUM9DU2NBdHO+MRYIQpwxtyghx/WflSM3yBqmuaT8A03h3IFP8sUm+6AwMPWN+ccid0Nag4NSULlA0/nW3W8wEwCrgdnlGEAKQxFlXJQ0aWtKoFaXbZHQWH4MssCfWg4TeoXZRUQcHyduRiesG0e2Qs2zSyavaRHqeK0c0Wqglm/AgapTM/X00U1Zu1OSbYF/2ebWT2kaA/CpsD/OXeROfDO49VmHihLbaPDomnTZxpVks1Nejc810oeMZApDOicCeJ2iC9JDBrnhtbMqPosQPAoKnuTD446sxaDig93Cpsp8qGN1Zfb7W10gZqobWDUjTD22y0TBWqXTSnlo/bf2y+YeE/pzLJT7TjjqlcQfHXalcdWzAdj6jPmIX6xBydtLu+j6FKaQZ5ICH889G2fcWVuCRRe8rbDHW2fEbD9KLSlRvWDfxH+wlYMxnfQE9W5yyyqjpTxYMdTTmWaek/CauWQQdoRvdJ5/M0q6r0rur4h4SZmBpZ2MI8uKRFlg9IqThu6NdH+MHuNnWefJhgu5cj657qorEhQqIKd7loX69yWFY+mACUrgESHQKT4U53k41RGxolBS87uirIpOS8fMpZM4B4zdQgiPzZdefnnxV41mZ2JS2u6UGEWSGMPJt/OoaZfxCEozK0F7n426RfDeD36zmDndBTIgWGee1K+F/+s89ICzDPbeNnR6q+b4vmkgegCDcJp1JPzoeWSI6+mivNQ0TvK8E0IA24f7drJpwBAvDG96BU71z2az0ibiNrmjhlYk1CaBhZ1GTOVx1/pCs6fTqW/31WJKK3lgriogUZ50/xCN2lvEDkdnkMwXKphMQVohnd0+6kx2mZBOAgNA2mHfV8tOsGw6k4wBHUjyY0Ov1CxY6JWPVhwZi+5k9ID9Ymb2UFktj6BSasq4GiHnHrAx7a7Q1k/HGPvSbbsxLSna1krIu5Cx0BsX9fbH50h8OrlpZKwGQEKZbWFZrzT9D5uqjs+/pPArsWqbVOgTNaRXi2chTpY+AwMXjlrwZv0XY+w+1KM5D+0BQG7BvyW5m+UNfZXPbQt5Sfklj35LePDkYKCJKxXMTVV+iwQfKQBUKNSC+L0uxTSdVgyb2cSwuRmoUuWvdEuxAqknYkz8Vvr2UsvjnhrnTJAdNaPMjWYBYDGEWRDXJlBkQkuuzULPljk31R4hG8dc2Y+VT657xh1b0LHnS3FolE8+BX7Mv1wcAgPPfFbwC8Ctb+5+AmscCChCzMzzlJtAe6T9l7cFe8ovMIEzRuJOYNpf+D7cCpMqBwUnBP83HDUq5jV7eH2D3eGcE7NQEVf62v9GDg7xv9ufVSbj6M0UmIh/ECgnm0ymQcFJ64b+usg7nfjzxtwWUwAVm9wCgtEhtZKiGlwyTI1mh4zb0UTnQMTd4+5Y+0l X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: e0583152-8f81-4db9-6c92-08dd50f49fad X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:33.9019 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: i0GTklGKP5ferYMIz5A8XS7u34jJF58Um4PAI5NnHJ8WZ10eMqxGyodL1q+tjhkwHmF/8Mn/4Dg6/LXustMeNw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: RZ72BaqKvKDheUqYdy9hD6RBcM1tOEOF X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5ef80 cx=c_pps a=UiiUhvOI59TtQsb/yF5oqw==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=64Cc0HZtAAAA:8 a=q3r9P8AP8Ax0Fte6b8AA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: RZ72BaqKvKDheUqYdy9hD6RBcM1tOEOF X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Refactor the PCI config setup code out of vfio_realize(), as we will later need this for vfio-user too. Signed-off-by: John Levon --- hw/vfio/pci.c | 176 +++++++++++++++++++++++++++----------------------- 1 file changed, 94 insertions(+), 82 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5fb6c4c4c6..83fe329474 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2957,6 +2957,99 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } +static bool vfio_pci_config_setup(VFIOPCIDevice *vdev, Error **errp) +{ + PCIDevice *pdev = &vdev->pdev; + VFIODevice *vbasedev = &vdev->vbasedev; + + /* vfio emulates a lot for us, but some bits need extra love */ + vdev->emulated_config_bits = g_malloc0(vdev->config_size); + + /* QEMU can choose to expose the ROM or not */ + memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4); + /* QEMU can also add or extend BARs */ + memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); + + /* + * The PCI spec reserves vendor ID 0xffff as an invalid value. The + * device ID is managed by the vendor and need only be a 16-bit value. + * Allow any 16-bit value for subsystem so they can be hidden or changed. + */ + if (vdev->vendor_id != PCI_ANY_ID) { + if (vdev->vendor_id >= 0xffff) { + error_setg(errp, "invalid PCI vendor ID provided"); + return false; + } + vfio_add_emulated_word(vdev, PCI_VENDOR_ID, vdev->vendor_id, ~0); + trace_vfio_pci_emulated_vendor_id(vbasedev->name, vdev->vendor_id); + } else { + vdev->vendor_id = pci_get_word(pdev->config + PCI_VENDOR_ID); + } + + if (vdev->device_id != PCI_ANY_ID) { + if (vdev->device_id > 0xffff) { + error_setg(errp, "invalid PCI device ID provided"); + return false; + } + vfio_add_emulated_word(vdev, PCI_DEVICE_ID, vdev->device_id, ~0); + trace_vfio_pci_emulated_device_id(vbasedev->name, vdev->device_id); + } else { + vdev->device_id = pci_get_word(pdev->config + PCI_DEVICE_ID); + } + + if (vdev->sub_vendor_id != PCI_ANY_ID) { + if (vdev->sub_vendor_id > 0xffff) { + error_setg(errp, "invalid PCI subsystem vendor ID provided"); + return false; + } + vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, + vdev->sub_vendor_id, ~0); + trace_vfio_pci_emulated_sub_vendor_id(vbasedev->name, + vdev->sub_vendor_id); + } + + if (vdev->sub_device_id != PCI_ANY_ID) { + if (vdev->sub_device_id > 0xffff) { + error_setg(errp, "invalid PCI subsystem device ID provided"); + return false; + } + vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_ID, vdev->sub_device_id, ~0); + trace_vfio_pci_emulated_sub_device_id(vbasedev->name, + vdev->sub_device_id); + } + + /* QEMU can change multi-function devices to single function, or reverse */ + vdev->emulated_config_bits[PCI_HEADER_TYPE] = + PCI_HEADER_TYPE_MULTI_FUNCTION; + + /* Restore or clear multifunction, this is always controlled by QEMU */ + if (vdev->pdev.cap_present & QEMU_PCI_CAP_MULTIFUNCTION) { + vdev->pdev.config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION; + } else { + vdev->pdev.config[PCI_HEADER_TYPE] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION; + } + + /* + * Clear host resource mapping info. If we choose not to register a + * BAR, such as might be the case with the option ROM, we can get + * confusing, unwritable, residual addresses from the host here. + */ + memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24); + memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4); + + vfio_pci_size_rom(vdev); + + vfio_bars_prepare(vdev); + + if (!vfio_msix_early_setup(vdev, errp)) { + return false; + } + + vfio_bars_register(vdev); + + return true; +} + static bool vfio_interrupt_setup(VFIOPCIDevice *vdev, Error **errp) { PCIDevice *pdev = &vdev->pdev; @@ -3060,91 +3153,10 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) goto error; } - /* vfio emulates a lot for us, but some bits need extra love */ - vdev->emulated_config_bits = g_malloc0(vdev->config_size); - - /* QEMU can choose to expose the ROM or not */ - memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4); - /* QEMU can also add or extend BARs */ - memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); - - /* - * The PCI spec reserves vendor ID 0xffff as an invalid value. The - * device ID is managed by the vendor and need only be a 16-bit value. - * Allow any 16-bit value for subsystem so they can be hidden or changed. - */ - if (vdev->vendor_id != PCI_ANY_ID) { - if (vdev->vendor_id >= 0xffff) { - error_setg(errp, "invalid PCI vendor ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_VENDOR_ID, vdev->vendor_id, ~0); - trace_vfio_pci_emulated_vendor_id(vbasedev->name, vdev->vendor_id); - } else { - vdev->vendor_id = pci_get_word(pdev->config + PCI_VENDOR_ID); - } - - if (vdev->device_id != PCI_ANY_ID) { - if (vdev->device_id > 0xffff) { - error_setg(errp, "invalid PCI device ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_DEVICE_ID, vdev->device_id, ~0); - trace_vfio_pci_emulated_device_id(vbasedev->name, vdev->device_id); - } else { - vdev->device_id = pci_get_word(pdev->config + PCI_DEVICE_ID); - } - - if (vdev->sub_vendor_id != PCI_ANY_ID) { - if (vdev->sub_vendor_id > 0xffff) { - error_setg(errp, "invalid PCI subsystem vendor ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, - vdev->sub_vendor_id, ~0); - trace_vfio_pci_emulated_sub_vendor_id(vbasedev->name, - vdev->sub_vendor_id); - } - - if (vdev->sub_device_id != PCI_ANY_ID) { - if (vdev->sub_device_id > 0xffff) { - error_setg(errp, "invalid PCI subsystem device ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_ID, vdev->sub_device_id, ~0); - trace_vfio_pci_emulated_sub_device_id(vbasedev->name, - vdev->sub_device_id); - } - - /* QEMU can change multi-function devices to single function, or reverse */ - vdev->emulated_config_bits[PCI_HEADER_TYPE] = - PCI_HEADER_TYPE_MULTI_FUNCTION; - - /* Restore or clear multifunction, this is always controlled by QEMU */ - if (vdev->pdev.cap_present & QEMU_PCI_CAP_MULTIFUNCTION) { - vdev->pdev.config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION; - } else { - vdev->pdev.config[PCI_HEADER_TYPE] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION; - } - - /* - * Clear host resource mapping info. If we choose not to register a - * BAR, such as might be the case with the option ROM, we can get - * confusing, unwritable, residual addresses from the host here. - */ - memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24); - memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4); - - vfio_pci_size_rom(vdev); - - vfio_bars_prepare(vdev); - - if (!vfio_msix_early_setup(vdev, errp)) { + if (!vfio_pci_config_setup(vdev, errp)) { goto error; } - vfio_bars_register(vdev); - if (!vbasedev->mdev && !pci_device_set_iommu_device(pdev, vbasedev->hiod, errp)) { error_prepend(errp, "Failed to set vIOMMU: "); From patchwork Wed Feb 19 14:48:38 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 43B4FC021B0 for ; Wed, 19 Feb 2025 14:52:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOa-0003LE-Ci; Wed, 19 Feb 2025 09:49:52 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOY-0003Kg-Hz; Wed, 19 Feb 2025 09:49:50 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOW-0007E7-Hp; Wed, 19 Feb 2025 09:49:50 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9K12V024315; Wed, 19 Feb 2025 06:49:45 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=TO+bwFhGPGzXhvJDbbFkpLIKPKdPUyL7isPRaKiTw S8=; b=TflzW9VzxYXNC/Z7r9ZsHo6P547Hwd9Q5XAU+sGvnq6wkhoDMZbhBohxs sM/fl4hgCuJGp+mSuk2581eLpbnqsuOQX2tqTfIdnAfhP2EA/qse5Vaq2qSDC1DO Qk7xzMPULfAPKK/cWYbNBjxdxgG43syBj1QGK6qA0va2u7E39OuH4De4mEGTURrm YLMIXVt/Ow+odW8dF6QZEyOPC12Tp4vt+WTf5S/Aez6CtKfwX7bFexyLu3u1bqTL 5RWRnbWDAFcgfKt0bK5UJ5ROhh01OGkTq1BKbWToOVSgPrvWQix9wEnGho5i5g+a Eg0ZMUkCYWF8qP1jaZiQ7XE8GTo4Q== Received: from dm5pr21cu001.outbound.protection.outlook.com (mail-centralusazlp17011027.outbound.protection.outlook.com [40.93.13.27]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxrv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:45 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=iHtqqxSlZlWVQlTcOn0c/0G/orHo/PrdGEJFJjPHZ3VMogjq+HwJ3shxqMDtwD6KISTHvJjZ76bG03ZufbJ/622rW3v20dEQY80rilH4EIv45VSkqhPkN/0UCuKCGSCYmiEGPgGF5PkzprAw2mt/oC1hTIwqHmbbiwduoZ8TfqIQr0UnsWMBe77ixIckTuTiKKkabHrjCetqsNgRVEcI5YHzYapcc7YGi/riZ8HnQSfZyy9IBSjZfm4hSt8t7WkaV/lrXilKgl7CZ+QyzklLhBYJT4HVhzG2KuVvNVGX+krIXFGKSz5Xwji12ouHiTEDwqtQ8wm7p1DiGfpGnrbS4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TO+bwFhGPGzXhvJDbbFkpLIKPKdPUyL7isPRaKiTwS8=; b=EkxfiFAgerNAUycQRjMTIzE7CSJug0T0Bbfkz1JSw8hzu7PH/pY04USz9KfO71aOM93AezojjAYXG6RVqO3dUO1il6dwvwdTWdKXPg7UHWaMws7moydOvIxgiRCK/V2gwv5Whm89drysBNkLrDgTeTF4UUgzh0esAzsbcTC5MyQvYJxLrANm75h3pc1W2qKZAHEQuDM9XblTPD7uNXOvUlILCj5T+JQ6gL6dDW3ceiyG142EXTFROXlbs29SsPX8Tl0WUcdH7xc1ENgCLRKS/I5inR4QsfnASvYG5pPILLHLlBm3xI0e91VoI59bgUjIt/GBUJT9xBM3wHD5mYovMQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TO+bwFhGPGzXhvJDbbFkpLIKPKdPUyL7isPRaKiTwS8=; b=okO6ZmYgGBHvfnQMBXT9UBeWCUkSx3GpS9QDIVuW3G8ufLjnY9xCCfZ91FklRbmXqb8b34N/K+uLNOnZbALjEw+kb9Ru2PcFXmteXpfNwe/yZhXKVL3+fCPfgfIUPARXm0R0J6gxWWVT5sLuniR2t+oH8AOPjG7ZOk5lNZynsLCx7Bmqd8p4ahWEbLS6SmioLMvl4VIVnFwqZMYZMoUqBAH/6Fgk5y2vDpnrDbS59g/Vi6YfPydV1M60jG51t78J2Sav4muqgEKavscS3qTwMiOLWZsspanp5tOu7ZyWmxM8Focl2AI8gqoLG3AfTFQmdypwNp5jFnO3BCi9t5BOog== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:37 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:37 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 08/28] vfio: add region cache Date: Wed, 19 Feb 2025 15:48:38 +0100 Message-Id: <20250219144858.266455-9-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 4e14f789-4d37-4ad7-3d4a-08dd50f4a1b0 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: 4X+8w6RIkHdZ79J2+RihCzjaif9vqI0MJDROZlfk6y9C5mWeCUoUH2/6Lvs58t2LvtAHaBGq0PxL7WSf5v4BttRl6fTqy3OmRHzbcr74pBIMm1bwR8mvvgI2oz/v55J/sWRg2A5L0+Pi+2By9CMcMSwwzkCGiu4VsPldB/xFau7fb8fWaX06LHbAJyrBe2q+lEd9o1PbDtCvZQQZ7nGlzB2YZJcBRwJFo1osB0RTbVQGF08NH7cViwa76vJSuG176mRGs9nptiMNY5XKMQc7JZRWY3+TEK6px+nI4BSfjeX1HEtCyZZ114luLA0VYRRyTeSGuxDj8CpZbMoOk35P7n5t45hbACinLYPkalRYNDupdsnIbcxbiQllgr/E2Rbww25VCVZ2m2zOpEFzTYGAG3dWS9zqWN5QikaiAJdh+g794UQIWC/eQg9kw+S352WOdtjVuEM7o6FVQVOnRM2c20FLN4K8/7SxJwDj62l1/g/t8/bl3BorvyqrHFJKClpMrCIRBVKvGuSr+XvTfhQHxXztFJodxEq9HYUjW1YdK9WcVGyWJXr39M5nQGe5hC1Kh5YnbAwL9O/EGmZrMvkRu8E+SmVgEMCxdzYPOi+2aeApUfeNX7JfZhrqCqdJ5ZDqr9OLnlc2GALGUnMkwsWLWqzJRbOTSQ5sPZe2Cp4fMRLqX8uDbYosMQBv/MQ5qglox59HVSxfvCP6UqllLjCZRu39dSTsdq6nzywDzGWWyWTesDmwffxFdCnHeoKKpAujJU+2aNbXxAON4gKPaP3dcRTy96RYiyC1KNQeieqC7TzuTLjsceAEDS0KOIU8DhmZRO3v5sRXuEb3wrbgZk12R0uVHRYLsx+jlwpoN2nSXjbOhQgcToJJ3nIxnDtPCouhH3MmAVnmm/5mM0GROfaYgl3pqV9/YxuunB2VFmKYlpjXbyfNOhKr2Al2awDiozfHAOUt5WCQOMjfgw6POAVjwBaNyujcTNVD+OZZvnIGlMTSkguUB2a+ZBFxlsq4nPwK62eJJKXYW0wNhCK/sW6Wfc/SjckpWcgjWuHza/z0+SbC2dfQe0bM0CLg2JDJSjiKPL/tseNVFkTTERfwyNWYeJE4YkrfMHftESYtfB05pU4llJDZsmfA0fC3y6aEXSreLGg9fFBZMu0vG+S3r0ssqLbBmwvTZUKOtY9DZKo1sfkQ2rPEgaO6xdDKRPcrxLxQUnkH2T2KcvPzYt6FlsW6ppqrb7XqFQwwVFUK4CiZCl2gCHvnbWRMMFTxwpjTJT4KRXP/ybubgYylp4c49SnbuLJUX6DXKrmU8Rj6s/576W44chKym+pTTiNaeEeL/WPD09wKXb5pSOPEQDxUxvo+CXx4nIj9kOlMguq1vtJ5Zz7s2uNOs4lZyyZFyWuMPgYf X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Qa3WN+s8pZdwICHeLOdI0k4YWVS1V3zo/kNam+iKIHOSPGs7AClKYxxOhAaQmni6I9Bx/Dz2sbfR4lS/jV6uZY2xaiIKyUEBYfk4UY9jwzK+sDetqNd4a82uEw9Nz+UhjnyvTg7WQcPA6qNVOHAFtN3hzQtQJHak4EsgBsCdyXSVJVMB+PxK3UTgzaF6DSV0LBp2TlSHH8DhQdWx9aH8HiAF7Fqqw04V+c87ZhpdLe2rF4/vZ9ejKMd80Pz7nX3Rmx3dl8n71DYg6w5P7mJN1L3fl5oeK0u323wPtHn5RQnV0YhVjOuamU32oMSczMeKfZQa/at/q8vVsqAAua+vkdlB9+XVmQEvwL++SpV2rYoao8la5/1m5GrT6dbRKjzubaYUK7GKSiH33zy/nexZbaiAjSwHoNWH3OHLCLS0XJbWcmNC5s61ugjtw1RGom67Tj+Y595buZLhcJxys69+B/CN8WGULdpRSEUPbWP4mT6hLhOb+QI+ItgEWn47AfEMDQgngW2IH8cbkSYEWdxnG8HIEzmteNjn/10/Aiu+7jwlvC9Shi+X6kJ8ZNrjF4EmhrHO8ZwDYFk9spftXbL7RcHNNeVkkhAsT7Seac/qfeHb/16nATuGrW/OL4j/C4S7eVzFQjfu3r6SNxHtGJosPzlztpd6WRfD41YUCFZmnmYMFs2crpc2byiKdwb2VnB30QBaLQzexSON1Z6KLhE/lUgj+kA9hEnEdQ5hdY0zhcESyvcojDwpXwwovRYKRipvRDS0x6CVgnGGRoLI6PyoCwzGwAFsgSS1Pt5pip0u0/jwaNsgPx8HHvedOzZbDF7fKC+N6KqQd7uMqnKxjnsla1s/46Hx6H5S6loUrNi4qn/cfX3h3uqPMnBivdn0N9bpxeYHDLV714Ey2qfbFUrLt8OuO3I9/lLbfOG8fV0uGzJHSfvUDGDWorvxnyBFIofcXU635hd6Qwl5srq+iVEY8HfEilKXxg0weLo9Jp9UZhHF+QBMHTxWvxiKBeSaVVvLmtixK4x0Xf8NzKfWVtVnZbE3TlMi1FQUG+G2CknT412lHUsUcjKGPN79ognADgmMAya7Dc+DUCe/xTTkuzj2b5YBUQ41yFzcpfVBEBjYhonqPeJmwq499yWFE9pMohnGtlty3vXNEH99+AjezpvvmJ/CaoCrLX0yrJvWe2HdZEnYTN256SHgnMda87b2ek+0KeNXZ6fzs5zf9XrZz5YCRkcvmaB9YrJfwJ4l7i6uZ7LBKpOBcT09s1ojyZgm3BxAtybwlW/BZV0/jZ33cHGEG1m/fYI1fuomzXCpFcQ1hBlShQkrH/XnL0439CbSzk0gAi1sxsya57mfiF+knzbGRj7StACbR9JEMdDjmL3oHlWyErUhA0dKrk2kMwHm/gHNGNTgfrgODfQJu5Zqh4kEynG/FdVVORq/xbSMnbYpr8kNqhMAaDthL2VPQsbr023C53ua3sxIj0hNOcyzOwVzH4Lt1DNGeLC6YhM9Ithrz/odDlCSohW/o9h9BWdJ0b+W4D6fTDsMiQMiQTnnth7hQ8qkjuTpAv47udinivSivBT5qJn8WQAhkvjWIyjxinXB X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4e14f789-4d37-4ad7-3d4a-08dd50f4a1b0 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:37.2696 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: qP9ejPushGGuqVecEg8/2mV/IxPvHlAOiIxkECDsbyhzc+pVCpUeE5nDCRFDssnoIyPw8ytRD68DMp0brCkCUg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: HO9sYh2cjeqJnqFnVN2Wil5sUP0P0shU X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5ef89 cx=c_pps a=U0KzkmEawxegXmCr7eTojA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=yLJTwCVF-aHqTr61_xIA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: HO9sYh2cjeqJnqFnVN2Wil5sUP0P0shU X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Instead of requesting region information on demand with VFIO_DEVICE_GET_REGION_INFO, maintain a cache: this will become necessary for performance for vfio-user, where this call becomes a message over the control socket, so is of higher overhead than the traditional path. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio/ccw.c | 5 ----- hw/vfio/common.c | 12 ++++++++++++ hw/vfio/container.c | 10 ++++++++++ hw/vfio/helpers.c | 21 ++++++++++++++++----- hw/vfio/igd.c | 8 ++++---- hw/vfio/pci.c | 8 ++++---- include/hw/vfio/vfio-common.h | 1 + 7 files changed, 47 insertions(+), 18 deletions(-) diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 67bc137f9b..22378d50bc 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -510,7 +510,6 @@ static bool vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) vcdev->io_region_offset = info->offset; vcdev->io_region = g_malloc0(info->size); - g_free(info); /* check for the optional async command region */ ret = vfio_get_dev_region_info(vdev, VFIO_REGION_TYPE_CCW, @@ -523,7 +522,6 @@ static bool vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) } vcdev->async_cmd_region_offset = info->offset; vcdev->async_cmd_region = g_malloc0(info->size); - g_free(info); } ret = vfio_get_dev_region_info(vdev, VFIO_REGION_TYPE_CCW, @@ -536,7 +534,6 @@ static bool vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) } vcdev->schib_region_offset = info->offset; vcdev->schib_region = g_malloc(info->size); - g_free(info); } ret = vfio_get_dev_region_info(vdev, VFIO_REGION_TYPE_CCW, @@ -550,7 +547,6 @@ static bool vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) } vcdev->crw_region_offset = info->offset; vcdev->crw_region = g_malloc(info->size); - g_free(info); } return true; @@ -560,7 +556,6 @@ out_err: g_free(vcdev->schib_region); g_free(vcdev->async_cmd_region); g_free(vcdev->io_region); - g_free(info); return false; } diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 4434e0a0a2..1866b3d3c5 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1569,6 +1569,16 @@ retry: return info; } +static void vfio_get_all_regions(VFIODevice *vbasedev) +{ + struct vfio_region_info *info; + int i; + + for (i = 0; i < vbasedev->num_regions; i++) { + vfio_get_region_info(vbasedev, i, &info); + } +} + void vfio_prepare_device(VFIODevice *vbasedev, VFIOContainerBase *bcontainer, VFIOGroup *group, struct vfio_device_info *info) { @@ -1586,6 +1596,8 @@ void vfio_prepare_device(VFIODevice *vbasedev, VFIOContainerBase *bcontainer, } QLIST_INSERT_HEAD(&vfio_device_list, vbasedev, global_next); + + vfio_get_all_regions(vbasedev); } bool vfio_attach_device_by_iommu_type(const char *iommu_type, char *name, diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 37a3befbc5..36cd245c92 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -886,6 +886,16 @@ static bool vfio_get_device(VFIOGroup *group, const char *name, static void vfio_put_base_device(VFIODevice *vbasedev) { + if (vbasedev->regions != NULL) { + int i; + + for (i = 0; i < vbasedev->num_regions; i++) { + g_free(vbasedev->regions[i]); + } + g_free(vbasedev->regions); + vbasedev->regions = NULL; + } + if (!vbasedev->group) { return; } diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index 4b255d4f3a..3c923d23b9 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -345,7 +345,7 @@ static int vfio_setup_region_sparse_mmaps(VFIORegion *region, int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, int index, const char *name) { - g_autofree struct vfio_region_info *info = NULL; + struct vfio_region_info *info = NULL; int ret; ret = vfio_get_region_info(vbasedev, index, &info); @@ -562,6 +562,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, { size_t argsz = sizeof(struct vfio_region_info); + /* create region cache */ + if (vbasedev->regions == NULL) { + vbasedev->regions = g_new0(struct vfio_region_info *, + vbasedev->num_regions); + } + /* check cache */ + if (vbasedev->regions[index] != NULL) { + *info = vbasedev->regions[index]; + return 0; + } + *info = g_malloc0(argsz); (*info)->index = index; @@ -581,6 +592,9 @@ retry: goto retry; } + /* fill cache */ + vbasedev->regions[index] = *info; + return 0; } @@ -599,7 +613,6 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type, hdr = vfio_get_region_info_cap(*info, VFIO_REGION_INFO_CAP_TYPE); if (!hdr) { - g_free(*info); continue; } @@ -611,8 +624,6 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type, if (cap_type->type == type && cap_type->subtype == subtype) { return 0; } - - g_free(*info); } *info = NULL; @@ -621,7 +632,7 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type, bool vfio_has_region_cap(VFIODevice *vbasedev, int region, uint16_t cap_type) { - g_autofree struct vfio_region_info *info = NULL; + struct vfio_region_info *info = NULL; bool ret = false; if (!vfio_get_region_info(vbasedev, region, &info)) { diff --git a/hw/vfio/igd.c b/hw/vfio/igd.c index b1a237edd6..b5425ba9c0 100644 --- a/hw/vfio/igd.c +++ b/hw/vfio/igd.c @@ -490,10 +490,10 @@ void vfio_probe_igd_bar0_quirk(VFIOPCIDevice *vdev, int nr) void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) { - g_autofree struct vfio_region_info *rom = NULL; - g_autofree struct vfio_region_info *opregion = NULL; - g_autofree struct vfio_region_info *host = NULL; - g_autofree struct vfio_region_info *lpc = NULL; + struct vfio_region_info *rom = NULL; + struct vfio_region_info *opregion = NULL; + struct vfio_region_info *host = NULL; + struct vfio_region_info *lpc = NULL; VFIOQuirk *quirk; VFIOIGDQuirk *igd; PCIDevice *lpc_bridge; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 83fe329474..a4f99fc5e0 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -879,7 +879,7 @@ static void vfio_update_msi(VFIOPCIDevice *vdev) static void vfio_pci_load_rom(VFIOPCIDevice *vdev) { - g_autofree struct vfio_region_info *reg_info = NULL; + struct vfio_region_info *reg_info = NULL; uint64_t size; off_t off = 0; ssize_t bytes; @@ -2665,7 +2665,7 @@ static VFIODeviceOps vfio_pci_ops = { bool vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) { VFIODevice *vbasedev = &vdev->vbasedev; - g_autofree struct vfio_region_info *reg_info = NULL; + struct vfio_region_info *reg_info = NULL; int ret; ret = vfio_get_region_info(vbasedev, VFIO_PCI_VGA_REGION_INDEX, ®_info); @@ -2730,7 +2730,7 @@ bool vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) static bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) { VFIODevice *vbasedev = &vdev->vbasedev; - g_autofree struct vfio_region_info *reg_info = NULL; + struct vfio_region_info *reg_info = NULL; struct vfio_irq_info irq_info = { .argsz = sizeof(irq_info) }; int i, ret = -1; @@ -3177,7 +3177,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) if (!vdev->igd_opregion && vdev->features & VFIO_FEATURE_ENABLE_IGD_OPREGION) { - g_autofree struct vfio_region_info *opregion = NULL; + struct vfio_region_info *opregion = NULL; if (vdev->pdev.qdev.hotplugged) { error_setg(errp, diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ae3ecbd9f6..304030e71d 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -151,6 +151,7 @@ typedef struct VFIODevice { IOMMUFDBackend *iommufd; VFIOIOASHwpt *hwpt; QLIST_ENTRY(VFIODevice) hwpt_next; + struct vfio_region_info **regions; } VFIODevice; struct VFIODeviceOps { From patchwork Wed Feb 19 14:48:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982380 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EB9D6C021B1 for ; Wed, 19 Feb 2025 14:50:29 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOb-0003Ln-SU; Wed, 19 Feb 2025 09:49:53 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOa-0003LF-FM; Wed, 19 Feb 2025 09:49:52 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOY-0007EM-64; Wed, 19 Feb 2025 09:49:52 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9K12W024315; Wed, 19 Feb 2025 06:49:46 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=jEz2JX8x79y1IRqol+pvfhT7Ui8Oy5uDhNcOHBeJQ K4=; b=tZ7PUcnpvz0Dyur4GItorZYNvXUvO0+v2Jkb/VL8VaU81tYGrHzp+NqpD MNXzfx89Om0qS3S8xnaVpBEcHjIk4wcYoUEJsimRYaREHRfCvtrh0X7yTdg+shBK IJuiISjDkhihc5RjXdE3TNvvrj14N4X0uRQTYeit8IRBTpRBXwUf73Bt88R7mYJg 4KUFXPV82kcKPObu88i/wA0pVmJAmHge6M6vVjUWOY/T7V4ay2AF6IinZv8BSCS0 uPSnaU+iSfhqk7hKXE0fycjmwCNiV36+MnWEq/GX0ZEwgk6rsKhRAB08W+R07eCP q1K7llEkX0OMLbrPgEGLCdDGQar0A== Received: from dm5pr21cu001.outbound.protection.outlook.com (mail-centralusazlp17011027.outbound.protection.outlook.com [40.93.13.27]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxrv-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=SZXm4gbzwAEhh2LD7uilz+C7bpK8Ilq2Q4KWCXAsyNnDMNAApA8aqz7c4yCz3XPG+ZUjVjzhQ3Y9c7iZPNpDE/nA1QjkOhiyYaHiezml8nrl2dFmTwct1j8KrkjaD1v6M3FhrwDYNh0mjUKux1GWdHR5A4Q220aZAw/x543FaUCdUs1Hhe6k+AgRhEGyro02OBWcfJRoVduaQT1D1t6gw7bkX/H9QZV3JaMGmqwv+L6p05rhJpqS/l/HPxTAgZ/07y5QuJQ1drBdfaIt53pJvwpjt0cqYx//8gkXMYVSDQxnQ6CsHR+X2LDaRsU9BfybMYEADsWulFH30qyWvaHxWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=jEz2JX8x79y1IRqol+pvfhT7Ui8Oy5uDhNcOHBeJQK4=; b=wGSkX86Nr92T5TDo7iNnbkFx30Da39V2lnCR8wT3nm7IOClAxINitklS601g2BjXFUDzA+ass77grGFWuJV+VI4jrWRoJNqKVLBE280Qp7St9jpxaqy8215LejMJ4SyZ62FNew199QTSRZ8fUTg/sN28OzP0ipqX9QtPjXpdo21jiKTN2fVU0Bq3Hb/VY1adQah5RFOGM4LHH3qB20t1U5USfM9v3sREAE/I2wZXg3CIAYrAmwWiDlSS88FEeHtJhMpBpXkfznv+XhnUvV0DFFeXocfwbQogjnEevdyu4eXzZqeqGcOmHsGYsNLgXISkECgw0ZcQy6XQpE+n75D62w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jEz2JX8x79y1IRqol+pvfhT7Ui8Oy5uDhNcOHBeJQK4=; b=YifWNKVvjsXZfzItdH1tXi+C01of6OGnlUy0lVkONmY8lvlxfPjymLaaUXIGiryeOuURDXnmCyHcPdItX0qf6UJngb01uJ3CmM+CctZ66LGr2yR9M7ksZoCM7S90OF4ARqyOF0EhlMjWNUbPwmZ2kL0s53XuQLD2KduzEIBAaBg4QuM1U0pTJP7UmMqmyUt/vNLb7dDhfX1LbyZy7SQiODG0AbkId2d3PyphPyEz3qlfBluW7WzsOkOLWC81uGtp4RZMOdYAM1VwJyso0572CQkzuDP+6qbIY3u5WgolleVGZTwDRq2JcGyi4k8DvTDa+25LiOf9Rma43PzVOvzyLw== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:40 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:40 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 09/28] vfio: split out VFIOKernelPCIDevice Date: Wed, 19 Feb 2025 15:48:39 +0100 Message-Id: <20250219144858.266455-10-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 5195534f-0760-4d32-f2e3-08dd50f4a3b1 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: 8dGBbExDSMLelYgUws03h/5kOijffNeqEFqylS/K1Mpde5GA8ID/9+TPUKAG+f7/UQs//1RKZTKsx9I0cY6kREU0/qPCGyT+g8vqUFas4OdSs5wQSdfYlsSERRZhYaWRGWn86xpty30W3K1lJaFnzMIhu9+fJdFLOLChN6YrUcNf6DpyCeI6zvd/nKobBCOYIAfzBqL3AVHBu0xas4rO0C+IupNntvxOBNatJXit+otl6ydcpMOPNzQPHH0AUPT145Sl+d0sBNRXrvcLUfz34J3cohiAntmahGCMUspPlx0PXhZZzTv35XZ6AD80LkUaTDlJcs3yHOnxS7p7Fkzh/YVFu7rSm48ES1SUfp0upBfI7SSe5QRNvlybCbmfo9vGdjhT7NCm0I6jRIFHTI6/j90xOgpUnd9lTKW/dHdq28yQAU9Ka7LbSdcAKzra9foAdyPeE85HjR169XPlw28IMnh8gumvty7PZ8kV8c9iJWANJklkY+MXXIoYjspCItiRZpEggSduPMND/4vx0t4G7aA7gGZbNkogSE25c5GKTg+2PEOOFHdRPTrYP55K2rHAUkHCPaEvVDQbMUzcXrzzzAk/zIRGHmGSKAVEStt8Rxz/NLJKmlY6WeRVV17ZFNy12itRqqj4aZVhz1tWwsmPB6BHIV0a58xBguD1fPOiYATqo8dfpntexP9hvROovyxd2hNhbyaAxseDrA2yUPIGapg7hOib9sB/Q2ns7E+FuXBJfaxIV/qJgSHwKt78wL/d5NIzu64xYOTo0+FEa+V2zO4SbWpQkl99gsq/hH5/bl6CvcXovRODHzClWz20E1AENs/DuSPW8xKMU8+2DfJcsm5QiHzg2j2D3iVxLCsvQlitQJQ9BW/WEev8nLRGVCajNGXJQ/2EJ//1/ib3Cl+wCK6nxkpGzOKvCbM7xJPQT/yyKchSAL44gmLejlKOT1ERtWuOSb41VSb81DDSAimH0wJLVaE4f/NHL5HWdBPdJeElhSTAXvwSjzFY14Kz3your8wuZfAnYnQkTxOy/Nb2XLGsspamnv5c4wsu0Sk9qOBV/bLUuIQ0XzPCODRQm7+fvlvzo7Bl0/laeUjqAwE8vH6V8zflk8lOmU9HhRqZsQlr3+XUq1tGJcxa3alIBxhH1eneK5boGJ7ZpTwAlm2g58+Qewi1prCl/heK/1rZ7bBBv7N9Se44D0SkR8jeUPOIjqbnA/ZnFZQuBz+fBDyQYCVqVEYYBwE2QpsdKUaMFuZy0/CUx+oVWmPiAuOPPw/0jLqcAWvst+EFVHFZI1l+1eUr2EMUNKc8YAKP0mtNiIPhPlnD4GKLGOdzrtDLgO5A+c7v5YK2A4YJjAZsis/oTk0ePp3ggNc2XzyOn6C6xZQY2fbM+wVG6ydMx9rE+8PD X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +hmviUPo+fhGOtLqey1xwxjHa3lLvOlGVgAVvClIQK9xIazhLki6/sTCMf02iH8gZc/haA7nQBZBeUhlXd664I9FSFD87ox3UKwIrsaVmlgl/Ra63hzf0oQ2XXhITdGCPiOyOR+2tivdfSgHaremUNKkSvrGEGNtg/gxv5y4CRhVm9NZMGpf3WNAvi5sNFLbFyVZ4Ja1F7A8r6xuzWq66YX0WJzJNTU+ZWj38/5s7GuuOjqXH8Ezm5IaGF19FMpNqzHtfSDJTGy0ssCSlzhY+7J0QP9/lwBY/Gf5X/4WXLJFxligfWrDNi7VQAeoIu3Uu76T1rXrl1OOYaWQ7djxiOC7i/Cl5XZQL8YtCltVfkDrfWHRktV98FsSO8P88L8IuE/m2GPG+q5a/lMYUihaMSbL83vcPwIuxu2D3WZWr/WCIdJ74xoJmzn2TQRs7lew5K9Xf4RXEK30onr+23MH4Xg1+cKe89GGUNhi6Q95a9BuYJ3pzz67YqssiznbO4SuL5TYIjRA/XzhyjRmWvQNGO9S5ci1VuGTU/NPSc81mcqZMaG5MWXWmOtyDK3LIPiGCTVDGV/7JETp+CYMwdctYw4GogvlRnHTC8eSQvPXYJCH4KfKeZmYUZyxPDoeZziAHm/jSRgqF48MpKUl6sPFxtWD95jeW2Mc83dqLiRgCL4vQ1JbtgWC4bNUz8JCL0QgEBxoY025T+b5g6iUHiP4OitzXhpQKJPNbC5IukyaziGaYw2VvVwtvRHq4zoLYnhycsA6ndwwSX0el7iacv3Wq5ggjpeA0koOTyxyyCbJ0nJznZ6UXJMA4SE2zcSHkx/was34GzX1rmhkgp+SqVGE8sjrmwwf4ZLa8ugJGKDTwUsgogurVj/+BunXd9nL1IRTTD4YsJ195AkoGcpKEnqYCEGGfQ+hrTvj1gD/rDMP8caHHffJ13Ab0MenGxR5Yxe65i/65kfS1zQUsLnIFXpX5lKKTc1Kigo7xvn0x025pbj25yEsKVcfDX6ujEbFDj0l1l0OpRSoDiFFmP2FGEfngEbBaQhFusEWhn2ob3slOyhJeSF+Az6oe0apr2ZqWz5I4mdT2qbB3XHlzCd443geORVeMgzaJl6zV8PKttocGkp8bDzKQCv+bXfaCggsz8OrzZgb95lsOyPyGOrKuUwcsfOREMd6J3C5X6ujSsBubib/aHZlwkkizfAwy+9uz+FtVDGHkyH168L5RHQhR6EagXzjdmARxqdx3J88bHoZ/zqp/khtOOyJzIUHoTWl+W/iIwHP+hiFHNBU41eD45slTH1KqLIavNem5sHSBp2dFonpjcJAeZYc5LJYSwcD7SNdSSnab9br04+Hz237wAOeeR4PC3BNwXTGvoQSjAccplv4tQCNZuCH2VkOI/w2WR3vpwGgdtrzpgdqo2kbTDIKkXGe0f8nI2ORrCBZKpHddweBgF6ey+5lQu+4epv4BJBY9weV+uF4HR+apNEmHQErc0ijmidxFzDGj40crhQcE57ErLwPsUDog8Ok11fie2yANKG+G1nnSCF4LYMFsFKHqRqmEyOyZI1PtL3k0tNJyd86+DAeEoADyjUHmQq9P9CA X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5195534f-0760-4d32-f2e3-08dd50f4a3b1 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:40.7094 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tLJ64eSZFrAkGo5fKFdVlhv5cS7QrLQB3bAFiowqxi1QH/jfx/B68wA0Yr6dfnRokiIhFahUx10HFNVkYTjO1g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: NpCNxMONIiklx2Dn0QWsaSaD2M6Oqrpg X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5ef8a cx=c_pps a=U0KzkmEawxegXmCr7eTojA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=Wcf8dU1HRy81hTxyJVcA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: NpCNxMONIiklx2Dn0QWsaSaD2M6Oqrpg X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Split out code specific to the kernel-side vfio implementation from the VFIOPCIDevice class into a VFIOKernelPCIDevice. The forthcoming VFIOUserPCIDevice will share the base VFIOPCIDevice class. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio/helpers.c | 2 +- hw/vfio/pci.c | 107 ++++++++++++++++++++++++++++------------------ hw/vfio/pci.h | 16 ++++++- 3 files changed, 80 insertions(+), 45 deletions(-) diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index 3c923d23b9..94bbc5747c 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -744,7 +744,7 @@ bool vfio_device_hiod_realize(VFIODevice *vbasedev, Error **errp) VFIODevice *vfio_get_vfio_device(Object *obj) { if (object_dynamic_cast(obj, TYPE_VFIO_PCI)) { - return &VFIO_PCI(obj)->vbasedev; + return &VFIO_PCI(obj)->device.vbasedev; } else { return NULL; } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a4f99fc5e0..812743e9dd 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -239,7 +239,7 @@ static void vfio_intx_update(VFIOPCIDevice *vdev, PCIINTxRoute *route) static void vfio_intx_routing_notifier(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); PCIINTxRoute route; if (vdev->interrupt != VFIO_INT_INTx) { @@ -514,7 +514,7 @@ static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, MSIMessage *msg, IOHandler *handler) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector; int ret; bool resizing = !!(vdev->nr_vectors < nr + 1); @@ -619,7 +619,7 @@ static int vfio_msix_vector_use(PCIDevice *pdev, static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector = &vdev->msi_vectors[nr]; trace_vfio_msix_vector_release(vdev->vbasedev.name, nr); @@ -1167,7 +1167,7 @@ static const MemoryRegionOps vfio_vga_ops = { */ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIORegion *region = &vdev->bars[bar].region; MemoryRegion *mmap_mr, *region_mr, *base_mr; PCIIORegion *r; @@ -1213,7 +1213,7 @@ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) */ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t emu_bits = 0, emu_val = 0, phys_val = 0, val; memcpy(&emu_bits, vdev->emulated_config_bits + addr, len); @@ -1246,7 +1246,7 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t val_le = cpu_to_le32(val); trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); @@ -3084,7 +3084,7 @@ static bool vfio_interrupt_setup(VFIOPCIDevice *vdev, Error **errp) static void vfio_realize(PCIDevice *pdev, Error **errp) { ERRP_GUARD(); - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev = &vdev->vbasedev; int i, ret; char uuid[UUID_STR_LEN]; @@ -3274,7 +3274,7 @@ error: static void vfio_instance_finalize(Object *obj) { - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); vfio_display_finalize(vdev); vfio_bars_finalize(vdev); @@ -3292,7 +3292,7 @@ static void vfio_instance_finalize(Object *obj) static void vfio_exitfn(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev = &vdev->vbasedev; vfio_unregister_req_notifier(vdev); @@ -3316,7 +3316,7 @@ static void vfio_exitfn(PCIDevice *pdev) static void vfio_pci_reset(DeviceState *dev) { - VFIOPCIDevice *vdev = VFIO_PCI(dev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); trace_vfio_pci_reset(vdev->vbasedev.name); @@ -3356,7 +3356,7 @@ post_reset: static void vfio_instance_init(Object *obj) { PCIDevice *pci_dev = PCI_DEVICE(obj); - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIODevice *vbasedev = &vdev->vbasedev; device_add_bootindex_property(obj, &vdev->bootindex, @@ -3377,28 +3377,15 @@ static void vfio_instance_init(Object *obj) pci_dev->cap_present |= QEMU_PCI_CAP_EXPRESS; } -static const Property vfio_pci_dev_properties[] = { - DEFINE_PROP_PCI_HOST_DEVADDR("host", VFIOPCIDevice, host), - DEFINE_PROP_UUID_NODEFAULT("vf-token", VFIOPCIDevice, vf_token), - DEFINE_PROP_STRING("sysfsdev", VFIOPCIDevice, vbasedev.sysfsdev), +static const Property vfio_pci_base_dev_properties[] = { DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevice, vbasedev.pre_copy_dirty_page_tracking, ON_OFF_AUTO_ON), DEFINE_PROP_ON_OFF_AUTO("x-device-dirty-page-tracking", VFIOPCIDevice, vbasedev.device_dirty_page_tracking, ON_OFF_AUTO_ON), - DEFINE_PROP_ON_OFF_AUTO("display", VFIOPCIDevice, - display, ON_OFF_AUTO_OFF), - DEFINE_PROP_UINT32("xres", VFIOPCIDevice, display_xres, 0), - DEFINE_PROP_UINT32("yres", VFIOPCIDevice, display_yres, 0), DEFINE_PROP_UINT32("x-intx-mmap-timeout-ms", VFIOPCIDevice, intx.mmap_timeout, 1100), - DEFINE_PROP_BIT("x-vga", VFIOPCIDevice, features, - VFIO_FEATURE_ENABLE_VGA_BIT, false), - DEFINE_PROP_BIT("x-req", VFIOPCIDevice, features, - VFIO_FEATURE_ENABLE_REQ_BIT, true), - DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, - VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), DEFINE_PROP_ON_OFF_AUTO("enable-migration", VFIOPCIDevice, vbasedev.enable_migration, ON_OFF_AUTO_AUTO), DEFINE_PROP_BOOL("migration-events", VFIOPCIDevice, @@ -3409,8 +3396,6 @@ static const Property vfio_pci_dev_properties[] = { DEFINE_PROP_BOOL("x-no-kvm-intx", VFIOPCIDevice, no_kvm_intx, false), DEFINE_PROP_BOOL("x-no-kvm-msi", VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false), - DEFINE_PROP_BOOL("x-no-geforce-quirks", VFIOPCIDevice, - no_geforce_quirks, false), DEFINE_PROP_BOOL("x-no-kvm-ioeventfd", VFIOPCIDevice, no_kvm_ioeventfd, false), DEFINE_PROP_BOOL("x-no-vfio-ioeventfd", VFIOPCIDevice, no_vfio_ioeventfd, @@ -3421,12 +3406,57 @@ static const Property vfio_pci_dev_properties[] = { sub_vendor_id, PCI_ANY_ID), DEFINE_PROP_UINT32("x-pci-sub-device-id", VFIOPCIDevice, sub_device_id, PCI_ANY_ID), + DEFINE_PROP_OFF_AUTO_PCIBAR("x-msix-relocation", VFIOPCIDevice, msix_relo, + OFF_AUTO_PCIBAR_OFF), +}; + + +static void vfio_pci_base_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + device_class_set_props(dc, vfio_pci_base_dev_properties); + dc->desc = "VFIO PCI base device"; + set_bit(DEVICE_CATEGORY_MISC, dc->categories); + pdc->exit = vfio_exitfn; + pdc->config_read = vfio_pci_read_config; + pdc->config_write = vfio_pci_write_config; +} + +static const TypeInfo vfio_pci_base_dev_info = { + .name = TYPE_VFIO_PCI_BASE, + .parent = TYPE_PCI_DEVICE, + .instance_size = 0, + .abstract = true, + .class_init = vfio_pci_base_dev_class_init, + .interfaces = (InterfaceInfo[]) { + { INTERFACE_PCIE_DEVICE }, + { INTERFACE_CONVENTIONAL_PCI_DEVICE }, + { } + }, +}; + +static const Property vfio_pci_dev_properties[] = { + DEFINE_PROP_PCI_HOST_DEVADDR("host", VFIOPCIDevice, host), + DEFINE_PROP_UUID_NODEFAULT("vf-token", VFIOPCIDevice, vf_token), + DEFINE_PROP_STRING("sysfsdev", VFIOPCIDevice, vbasedev.sysfsdev), + DEFINE_PROP_ON_OFF_AUTO("display", VFIOPCIDevice, + display, ON_OFF_AUTO_OFF), + DEFINE_PROP_UINT32("xres", VFIOPCIDevice, display_xres, 0), + DEFINE_PROP_UINT32("yres", VFIOPCIDevice, display_yres, 0), + DEFINE_PROP_BIT("x-vga", VFIOPCIDevice, features, + VFIO_FEATURE_ENABLE_VGA_BIT, false), + DEFINE_PROP_BIT("x-req", VFIOPCIDevice, features, + VFIO_FEATURE_ENABLE_REQ_BIT, true), + DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, + VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), + DEFINE_PROP_BOOL("x-no-geforce-quirks", VFIOPCIDevice, + no_geforce_quirks, false), DEFINE_PROP_UINT32("x-igd-gms", VFIOPCIDevice, igd_gms, 0), DEFINE_PROP_UNSIGNED_NODEFAULT("x-nv-gpudirect-clique", VFIOPCIDevice, nv_gpudirect_clique, qdev_prop_nv_gpudirect_clique, uint8_t), - DEFINE_PROP_OFF_AUTO_PCIBAR("x-msix-relocation", VFIOPCIDevice, msix_relo, - OFF_AUTO_PCIBAR_OFF), #ifdef CONFIG_IOMMUFD DEFINE_PROP_LINK("iommufd", VFIOPCIDevice, vbasedev.iommufd, TYPE_IOMMUFD_BACKEND, IOMMUFDBackend *), @@ -3437,7 +3467,8 @@ static const Property vfio_pci_dev_properties[] = { #ifdef CONFIG_IOMMUFD static void vfio_pci_set_fd(Object *obj, const char *str, Error **errp) { - vfio_device_set_fd(&VFIO_PCI(obj)->vbasedev, str, errp); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + vfio_device_set_fd(&vdev->vbasedev, str, errp); } #endif @@ -3452,25 +3483,16 @@ static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) object_class_property_add_str(klass, "fd", NULL, vfio_pci_set_fd); #endif dc->desc = "VFIO-based PCI device assignment"; - set_bit(DEVICE_CATEGORY_MISC, dc->categories); pdc->realize = vfio_realize; - pdc->exit = vfio_exitfn; - pdc->config_read = vfio_pci_read_config; - pdc->config_write = vfio_pci_write_config; } static const TypeInfo vfio_pci_dev_info = { .name = TYPE_VFIO_PCI, - .parent = TYPE_PCI_DEVICE, - .instance_size = sizeof(VFIOPCIDevice), + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOKernelPCIDevice), .class_init = vfio_pci_dev_class_init, .instance_init = vfio_instance_init, .instance_finalize = vfio_instance_finalize, - .interfaces = (InterfaceInfo[]) { - { INTERFACE_PCIE_DEVICE }, - { INTERFACE_CONVENTIONAL_PCI_DEVICE }, - { } - }, }; static const Property vfio_pci_dev_nohotplug_properties[] = { @@ -3490,12 +3512,13 @@ static void vfio_pci_nohotplug_dev_class_init(ObjectClass *klass, void *data) static const TypeInfo vfio_pci_nohotplug_dev_info = { .name = TYPE_VFIO_PCI_NOHOTPLUG, .parent = TYPE_VFIO_PCI, - .instance_size = sizeof(VFIOPCIDevice), + .instance_size = sizeof(VFIOKernelPCIDevice), .class_init = vfio_pci_nohotplug_dev_class_init, }; static void register_vfio_pci_dev_type(void) { + type_register_static(&vfio_pci_base_dev_info); type_register_static(&vfio_pci_dev_info); type_register_static(&vfio_pci_nohotplug_dev_info); } diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 43c166680a..8e79740ddb 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -116,8 +116,13 @@ typedef struct VFIOMSIXInfo { bool noresize; } VFIOMSIXInfo; -#define TYPE_VFIO_PCI "vfio-pci" -OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI) +/* + * TYPE_VFIO_PCI_BASE is an abstract type used to share code + * between VFIO implementations that use a kernel driver + * with those that use user sockets. + */ +#define TYPE_VFIO_PCI_BASE "vfio-pci-base" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_BASE) struct VFIOPCIDevice { PCIDevice pdev; @@ -182,6 +187,13 @@ struct VFIOPCIDevice { Notifier irqchip_change_notifier; }; +#define TYPE_VFIO_PCI "vfio-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOKernelPCIDevice, VFIO_PCI) + +struct VFIOKernelPCIDevice { + VFIOPCIDevice device; +}; + /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device) { From patchwork Wed Feb 19 14:48:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982400 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A4865C021AA for ; Wed, 19 Feb 2025 14:52:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOd-0003Mm-TS; Wed, 19 Feb 2025 09:49:55 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOb-0003Lg-H5; Wed, 19 Feb 2025 09:49:53 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOY-0007EP-Go; Wed, 19 Feb 2025 09:49:53 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9K12X024315; Wed, 19 Feb 2025 06:49:47 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=Fo5T4wdttlCPCsIP7ttovjalrRTDCrL841CgZU45m bk=; b=k0XckMD//LVPgmr9cEAFCyTd/Z1/s7BK0TvIDEBVTtAod8NhjsTVEJPea XrY3DOdEY2yRb3SbFI27uTnsk8QqYMdvNufuEVKItpx2nvHAjAYqMp6ay3i4UyaT SffXifXakMWJjYZhxonvyUoZHO/nyxWMkXXp8ZKlOb8gJZb5CMoIjyGXEoGg/t54 INUE37xOXLAK4G03FR8B7C8zUsEn+INtSCf7JIRAcTUiCPuYPyAP1+QPZ10JqS+Y Q+lC8eWdVuoYd82Tb/R1u2HVKxSwAEZ/4+AfNdX1NMm1KApD+VRGpxOpU4M0v1Lr ujey/n5g4ylR6V+ogXUQlCTMuifjg== Received: from dm5pr21cu001.outbound.protection.outlook.com (mail-centralusazlp17011027.outbound.protection.outlook.com [40.93.13.27]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxrv-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:47 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=Lv0XmS/A3oG+WnQ+y+l1n007htJx+6MatK5Kv4yjB+UnxVlGwdzmA5dtbCsQuBt3ZtKU6U4zR99kpeBhMpFpQg9vRT5OXgh7G2HjMNo1RUzfFziQtmzaJ1OLXLTqBdzAEu06727b6aDnDfLMcEuZdd2VocuUMYEnoK8Yw45uUxAmm4k4PfywlD/j7J6N5bcv28LMTDVeQI70F1FVk+fYTG+rh+uaUXrpY0NlQOnLAPj1LPEky/HGe3lPe9TwOtJ4RRujIxqG3qLfqFIEbRzl1XWz2eLYVzGjSoe5Q6WT7FtRJZoErEiTAO68CGgKje11OCDemFiKasY+VaEiDGgHWw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fo5T4wdttlCPCsIP7ttovjalrRTDCrL841CgZU45mbk=; b=VYzC0U3ulEZ2h1doTDYXbB6h4RKpDxXf0025BSF7J5vvLP5QL9KzA3mA9QIH4GkKjKxsjASJfvfNnlkINFKnV96FYW1rELM+6euxxA9cuiNNqz1NSFaCVnxABGTWD+F2fcp6Cns99N1y1e3HrFO2a5UhS6zhqigMkmrjBiKGbGPmNiBz9pPUa522NAne4K7K+RzqSfR5bFtJ/v9H8lso2AJ6OFDRqVJ+ZScySjLx1DoIdEOfXUsEVDt9JUUxaeFqf9Fy+ObXS/OF2XD9ngHcZbCp6KMQ/BvTH2ACRAog5fCY0a+4NXAtNo6utQExXHPxKA5TeE2Zej9SDFhj/MQScw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fo5T4wdttlCPCsIP7ttovjalrRTDCrL841CgZU45mbk=; b=VeAfjniAMIxcejWqYzAm/nO0q7ZeIymvVHsvWey1nalDdxZ2c+UrM1XL58xtPcSca5QQ5f4iYYngLqNtuARzy53ym2cK610Ag4wbF5IGI86n1JwKMWQkIEglsH6SRa6Sqv3YVQ0Z9C9lN7RljHwmpnUv2Lqr5sdDMKtopFYo5iASxHc85dpOz7jFp8OcAUROMMCplK6zsrohXZ5kGyGskqzJCij1p7+XOepjL5STvfGkrGZYBfzBNMnZ2CrwZFopV6y5EKEufeYVdtxhbu4fuOKje1xicaAIIO50X+VTtzzc6X66L4Ywdgq48b5sywb+ANFrHhcOUw047FCn1L0gpQ== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:44 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:44 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 10/28] vfio: add device IO ops vector Date: Wed, 19 Feb 2025 15:48:40 +0100 Message-Id: <20250219144858.266455-11-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: c5f98aba-88a5-4c47-b64b-08dd50f4a5c3 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: P8+XPweW+iYp3IczoG0IQI3O3rpn0b4DgqBBNxQ0e7yWxgG55jHR0yrTLPi/wx0Vaa6art2Q06BR8x5H1a188S0L5Ejcz1eaWxcQWHROwMb9quUx/3nTidan6DO7NVxZKhr+5KM5VZDUZaaGF3FrHoeoVOgZbGRbF24DALw0t2cInuIw3jkVsrADoNgWMUAvRcWbMBdFBwZnNtXr7CN6tWAPuIsQ3mOXosLg1UbGVIZxzuWKhA7UYqi+SHKcNdb7AHTui0yWuKRd3LHA+yqqSx4+CFKeptkCGStvTZFkytbcCwyVLCbwyGXhO3ySw7/IOCZ9OECSMCioBQ9vdOLkI18eiNLxS1Fpfsr9F9cb5lOqq+WUdW1CMgaCpMSbIGFmNh6m7EGE7OZpK123HntL5NvQrrNKqHws7sutYekrNhv1Hw4bwEnWNN1NP9fy81AJ+6aMqQ2PN2tpR1huaAJVTU3BkheT2/WofjMEJ6husXpY4a1zu+Jtq+TNxqP1IwGDVhDdeARnK3wGUHIHgoJZBkS4BW41C2ndvBguVQCKFJmR/yqxszPq6HY7RWzvpTHivHFTrf/+XCN0Av7HgVfggTiU+GrlffwTSUyz7+ywlWDW68X5pQqKv9TCUTfQpdYD3564FXsQ22WdSNWYwLHKkojpGvc8nY8qS4nicWkgXF7gQdrRPVjoV72MU+UcNy+0nGC3tMrqpJZkv+vWQZvITG+5QisCkXeoSwwq/eC0/1le9iy2+eRmOvYX3pguftOukDkY5HFDQrsFSSIOX+ZMAul4qUrNcnuEDo97D+1bBn8LGE3gdYUbwCNaKYDBOGgCfv/bHS/JXo6cpJJlKALiqlkZA9n606N1mSfR6eOAfzHr2ltlkOux5wii54Ka6EV5wXYdkGh1GEAAZiXhFZvvD07y2F7yWQg58NlnP684gU+dUnYQ6LY02ggHiS7C2r+8DAzVEKHpIKjUmR+0i57HB5a9Vtdy3JiSL4Xw0tvFwEP+5k8Xp54wwx6Slx8V/VR3bHFTeoUHHC78gD/la8GwlsIEYQLN6yeBAdQy0UWovhRZgtZ+Q4G3IN4kZ/r27+Y/zp2zx76IeVpmbcWX5ohhVy+6bfjjKp/zbE138uFJISf3bN2SnCYvDLetp2ZWXZMFhuxBEZnhr4LMd9UUGvHPU90x3dCj7wrcgRNu98lM5UypEQho7ZJNVXR0/jDQTZjOtNyLkSczO/Bmr4RdvywIpdwDKAWQhS/sv2LGe84H3f/zumWwyyZVT7J1xmJSAI9LdATESW+7zIlL0puVsWj99NyeeKm03/RHev8VV1ZCW9WXtxxVLbQaUlzETLOBxXJQ4fMx4B4+rwzILnWBfieDImlQO2L7LRjWg1NTROUAnrD1pyvHNUUVe/dmdD5zNEJy X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: AcOzwbuXye3Z9H+43pepm5dnR06qqgWHF0BnR35XFV4VZVPYIeaq9U3P5sKI9o4P8wTSVYs63maZ61RIEpOP+pVuvKLYe8kgZ6wNdpK3ecJLUZlX02KsvzhA4b23TTjGmEqjOcxJDKSPoZ3rujSpzBQ/G+l0SSJPEwGkYI/XETLRjjI3fwTuyFw4VHHPYrvwVdcrk+twDdh7fTZIAmiyRXpWSOYPoRcvdty9Gqfufnwv9XVRm6NoypWl7mZ56xaJ8XR2XUUYcspbTSclZtSCMB7LE0qHv/Iv6xvLq9OyLV2cRQPlgKAAHkFcyu2N0AvrauGxTGsZbux4BUE+PtERgiED0TlGTBNDtlvsJBrbZOuKk62CM2ws+kdc16bAW4fSr17lntxzOCdOhDTbgZPgK8vI9axo3c2rGPxb2jLhCRk7OTlvYhA5h8NH2cL1vd8VCZRV/tMSPt4b2Ypz4Fe8NLEUH9Y3L3YgYfpAyU7F0akXxRhz2KOjF2IflPfViP0yOKc9BBAMZUKcb9o5gJsGhC9dpXgoUC4Kb9UBZrquZM2hpkaGgegZDPGVQtanlp17otedhYvNXBZrUyctdIjtCuwm07JOZgZdhAzI17yxA6d5d72bxAZSrDcjYwBPTKFGRfsFXOrvvCDK2VcNG81iU2y5Bm8KBna3N9dHVnzz8UEVWS7djWGg96BdtjOszd7iTL7cu1x3R/oUSfAlAlLlBeZCKfeAxPnMuA2RtXeNwEZOgqK5eKwfu8r28hAO4Nenl6/yQspVj7/pTLyzbHeBc09Gx67VnQCM/pv28WuaicmsELk1VMl/D747x3rDdukzBDtuvWE3hpK0FkdQe/ZR9kJ/Q8mx6MLEgVQ5H+aNjvNilHXeyXeC78Mb1133aLZDQ1mF+3/mkhs+v01FESPh957Mg4z+bdhhuIEgqkvzeMwebjYItFXpoQgA+9r4LPeZ87+jC6xRcoehhgCzceqTfAUpAzbyxvMfb5Pt/B/TBSxsMAv850PVpLV3igKAo7EN4lfoV9I4T1MUH7R/kDx1A9RkdwZ+RhoHBPozy6CaohEu//VV95z9MT8630RB8+pXDK5/BL8fjajGljdt1RlrqCkIkwS9d6b55GVBALMmHuH9tFZSeJG5aR6zqD1rUIxvPcfOs8eBklWelKrgpeShsbByezos9HsfeLrseOvYaqr4SFvszPAdQk6HsC+9mluAcI7NxJq79DyQoBJKgx4pofn6eQnqznuynkV94nB4qITE7d266L5SHd/22kmAWJ3FMrElrKFGr7vcCHO1QqC7Qc2BckBPy6mcg93jaP2KkarAJ1iX7zaYu4BI5G7ZtpEgBYQhwZXE6nuoT9BswEG9uVOwww3rn6z5Jpk3G4w56+7P7lNU+Ta2WtS3Ptj0cdb0KorEYvZmEfZdeQkgC/UJOpK6E0TvIfky39C7T3rP7A0Ir5q6N3QqRggIV8wRjLH2POrtD18TNyDv2w1MyPv3/TutnLN4zC3Sj4YzJKvRtULjNocIVQlxKlPYht6TMX8TjYRjE4oluy8rLNyGtOHq+tbiLbe4If83gWuGibhEw3B+QJ6tQjdxT/1tKcubuzNE X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: c5f98aba-88a5-4c47-b64b-08dd50f4a5c3 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:44.1742 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: eEo4P1l5SVj/fjkEfSt1XSWZxDs+ZyLwH83Wc28fQfgH3RbkQxzQXLKCTFx5Xcu143bpKYaB3HvQ2tuMT2dhvg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: C6Z3qZTNtpjrMK1HfLwvGtwYJc1FBcZB X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5ef8b cx=c_pps a=U0KzkmEawxegXmCr7eTojA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=cut_OxLeOoAalzpo9TUA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: C6Z3qZTNtpjrMK1HfLwvGtwYJc1FBcZB X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman For vfio-user, device operations such as IRQ handling and region read/writes are implemented in userspace over the control socket, not ioctl() or read()/write() to the vfio kernel driver; add an ops vector to generalize this, and implement vfio_dev_io_ioctl for interacting with the kernel vfio driver. The ops consistently use the "-errno" return style, as the vfio-user implementations get their errors from response messages not from the kernel; adjust the callers to handle this as necessary. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio/ap.c | 2 +- hw/vfio/ccw.c | 2 +- hw/vfio/common.c | 13 +-- hw/vfio/helpers.c | 110 ++++++++++++++++++++++--- hw/vfio/pci.c | 147 ++++++++++++++++++++++------------ hw/vfio/platform.c | 2 +- include/hw/vfio/vfio-common.h | 27 ++++++- 7 files changed, 227 insertions(+), 76 deletions(-) diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c index 30b08ad375..1adce1ab40 100644 --- a/hw/vfio/ap.c +++ b/hw/vfio/ap.c @@ -228,7 +228,7 @@ static void vfio_ap_instance_init(Object *obj) * handle ram_block_discard_disable(). */ vfio_device_init(vbasedev, VFIO_DEVICE_TYPE_AP, &vfio_ap_ops, - DEVICE(vapdev), true); + &vfio_dev_io_ioctl, DEVICE(vapdev), true); /* AP device is mdev type device */ vbasedev->mdev = true; diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 22378d50bc..8c16648819 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -682,7 +682,7 @@ static void vfio_ccw_instance_init(Object *obj) * ram_block_discard_disable(). */ vfio_device_init(vbasedev, VFIO_DEVICE_TYPE_CCW, &vfio_ccw_ops, - DEVICE(vcdev), true); + &vfio_dev_io_ioctl, DEVICE(vcdev), true); } #ifdef CONFIG_IOMMUFD diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 1866b3d3c5..cc0c0f7fc7 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -971,7 +971,7 @@ static void vfio_devices_dma_logging_stop(VFIOContainerBase *bcontainer) continue; } - if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { + if (vbasedev->io->device_feature(vbasedev, feature)) { warn_report("%s: Failed to stop DMA logging, err %d (%s)", vbasedev->name, -errno, strerror(errno)); } @@ -1074,10 +1074,9 @@ static bool vfio_devices_dma_logging_start(VFIOContainerBase *bcontainer, continue; } - ret = ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature); + ret = vbasedev->io->device_feature(vbasedev, feature); if (ret) { - ret = -errno; - error_setg_errno(errp, errno, "%s: Failed to start DMA logging", + error_setg_errno(errp, -ret, "%s: Failed to start DMA logging", vbasedev->name); goto out; } @@ -1145,6 +1144,7 @@ static int vfio_device_dma_logging_report(VFIODevice *vbasedev, hwaddr iova, struct vfio_device_feature *feature = (struct vfio_device_feature *)buf; struct vfio_device_feature_dma_logging_report *report = (struct vfio_device_feature_dma_logging_report *)feature->data; + int ret; report->iova = iova; report->length = size; @@ -1155,8 +1155,9 @@ static int vfio_device_dma_logging_report(VFIODevice *vbasedev, hwaddr iova, feature->flags = VFIO_DEVICE_FEATURE_GET | VFIO_DEVICE_FEATURE_DMA_LOGGING_REPORT; - if (ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature)) { - return -errno; + ret = vbasedev->io->device_feature(vbasedev, feature); + if (ret) { + return -ret; } return 0; diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index 94bbc5747c..bef1540295 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -44,7 +44,7 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + vbasedev->io->set_irqs(vbasedev, &irq_set); } void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) @@ -57,7 +57,7 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + vbasedev->io->set_irqs(vbasedev, &irq_set); } void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) @@ -70,7 +70,7 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + vbasedev->io->set_irqs(vbasedev, &irq_set); } static inline const char *action_to_str(int action) @@ -117,6 +117,7 @@ bool vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, int argsz; const char *name; int32_t *pfd; + int ret; argsz = sizeof(*irq_set) + sizeof(*pfd); @@ -129,7 +130,9 @@ bool vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; - if (!ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) { + ret = vbasedev->io->set_irqs(vbasedev, irq_set); + + if (!ret) { return true; } @@ -161,6 +164,7 @@ void vfio_region_write(void *opaque, hwaddr addr, uint32_t dword; uint64_t qword; } buf; + int ret; switch (size) { case 1: @@ -180,11 +184,12 @@ void vfio_region_write(void *opaque, hwaddr addr, break; } - if (pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { + ret = vbasedev->io->region_write(vbasedev, region->nr, addr, size, &buf); + if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64 - ",%d) failed: %m", + ",%d) failed: %s", __func__, vbasedev->name, region->nr, - addr, data, size); + addr, data, size, ret < 0 ? strerror(ret) : "short write"); } trace_vfio_region_write(vbasedev->name, region->nr, addr, data, size); @@ -212,11 +217,13 @@ uint64_t vfio_region_read(void *opaque, uint64_t qword; } buf; uint64_t data = 0; + int ret; - if (pread(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { - error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %m", + ret = vbasedev->io->region_read(vbasedev, region->nr, addr, size, &buf); + if (ret != size) { + error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %s", __func__, vbasedev->name, region->nr, - addr, size); + addr, size, ret < 0 ? strerror(ret) : "short read"); return (uint64_t)-1; } switch (size) { @@ -561,6 +568,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) { size_t argsz = sizeof(struct vfio_region_info); + int ret; /* create region cache */ if (vbasedev->regions == NULL) { @@ -579,10 +587,11 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, retry: (*info)->argsz = argsz; - if (ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info)) { + ret = vbasedev->io->get_region_info(vbasedev, *info); + if (ret != 0) { g_free(*info); *info = NULL; - return -errno; + return ret; } if ((*info)->argsz > argsz) { @@ -689,11 +698,12 @@ void vfio_device_set_fd(VFIODevice *vbasedev, const char *str, Error **errp) } void vfio_device_init(VFIODevice *vbasedev, int type, VFIODeviceOps *ops, - DeviceState *dev, bool ram_discard) + VFIODeviceIO *io, DeviceState *dev, bool ram_discard) { vbasedev->type = type; vbasedev->ops = ops; vbasedev->dev = dev; + vbasedev->io = io; vbasedev->fd = -1; vbasedev->ram_block_discard_allowed = ram_discard; @@ -749,3 +759,77 @@ VFIODevice *vfio_get_vfio_device(Object *obj) return NULL; } } + +/* + * Traditional ioctl() based io + */ + +static int vfio_io_device_feature(VFIODevice *vbasedev, + struct vfio_device_feature *feature) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_FEATURE, feature); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_get_region_info(VFIODevice *vbasedev, + struct vfio_region_info *info) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, info); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_get_irq_info(VFIODevice *vbasedev, + struct vfio_irq_info *info) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_IRQ_INFO, info); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irqs) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irqs); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_region_read(VFIODevice *vbasedev, uint8_t index, off_t off, + uint32_t size, void *data) +{ + struct vfio_region_info *info = vbasedev->regions[index]; + int ret; + + ret = pread(vbasedev->fd, data, size, info->offset + off); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_region_write(VFIODevice *vbasedev, uint8_t index, off_t off, + uint32_t size, void *data) +{ + struct vfio_region_info *info = vbasedev->regions[index]; + int ret; + + ret = pwrite(vbasedev->fd, data, size, info->offset + off); + + return ret < 0 ? -errno : ret; +} + +VFIODeviceIO vfio_dev_io_ioctl = { + .device_feature = vfio_io_device_feature, + .get_region_info = vfio_io_get_region_info, + .get_irq_info = vfio_io_get_irq_info, + .set_irqs = vfio_io_set_irqs, + .region_read = vfio_io_region_read, + .region_write = vfio_io_region_write, +}; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 812743e9dd..a9cc9366fb 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -45,6 +45,14 @@ #include "migration/qemu-file.h" #include "system/iommufd.h" +/* convenience macros for PCI config space */ +#define VDEV_CONFIG_READ(vbasedev, off, size, data) \ + ((vbasedev)->io->region_read((vbasedev), VFIO_PCI_CONFIG_REGION_INDEX, \ + (off), (size), (data))) +#define VDEV_CONFIG_WRITE(vbasedev, off, size, data) \ + ((vbasedev)->io->region_write((vbasedev), VFIO_PCI_CONFIG_REGION_INDEX, \ + (off), (size), (data))) + #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" /* Protected by BQL */ @@ -379,6 +387,7 @@ static void vfio_msi_interrupt(void *opaque) static int vfio_enable_msix_no_vec(VFIOPCIDevice *vdev) { g_autofree struct vfio_irq_set *irq_set = NULL; + VFIODevice *vbasedev = &vdev->vbasedev; int ret = 0, argsz; int32_t *fd; @@ -394,7 +403,7 @@ static int vfio_enable_msix_no_vec(VFIOPCIDevice *vdev) fd = (int32_t *)&irq_set->data; *fd = -1; - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + ret = vbasedev->io->set_irqs(vbasedev, irq_set); return ret; } @@ -453,7 +462,7 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + ret = vdev->vbasedev.io->set_irqs(&vdev->vbasedev, irq_set); g_free(irq_set); @@ -763,7 +772,8 @@ retry: ret = vfio_enable_vectors(vdev, false); if (ret) { if (ret < 0) { - error_report("vfio: Error: Failed to setup MSI fds: %m"); + error_report("vfio: Error: Failed to setup MSI fds: %s", + strerror(-ret)); } else { error_report("vfio: Error: Failed to enable %d " "MSI vectors, retry with %d", vdev->nr_vectors, ret); @@ -879,14 +889,17 @@ static void vfio_update_msi(VFIOPCIDevice *vdev) static void vfio_pci_load_rom(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; struct vfio_region_info *reg_info = NULL; uint64_t size; off_t off = 0; ssize_t bytes; + int ret; - if (vfio_get_region_info(&vdev->vbasedev, - VFIO_PCI_ROM_REGION_INDEX, ®_info)) { - error_report("vfio: Error getting ROM info: %m"); + ret = vfio_get_region_info(vbasedev, VFIO_PCI_ROM_REGION_INDEX, ®_info); + + if (ret != 0) { + error_report("vfio: Error getting ROM info: %s", strerror(-ret)); return; } @@ -911,18 +924,19 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev) memset(vdev->rom, 0xff, size); while (size) { - bytes = pread(vdev->vbasedev.fd, vdev->rom + off, - size, vdev->rom_offset + off); + bytes = vbasedev->io->region_read(vbasedev, VFIO_PCI_ROM_REGION_INDEX, + off, size, vdev->rom + off); if (bytes == 0) { break; } else if (bytes > 0) { off += bytes; size -= bytes; } else { - if (errno == EINTR || errno == EAGAIN) { + if (bytes == -EINTR || bytes == -EAGAIN) { continue; } - error_report("vfio: Error reading device ROM: %m"); + error_report("vfio: Error reading device ROM: %s", + strerror(-bytes)); break; } } @@ -1010,10 +1024,9 @@ static const MemoryRegionOps vfio_rom_ops = { static void vfio_pci_size_rom(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; uint32_t orig, size = cpu_to_le32((uint32_t)PCI_ROM_ADDRESS_MASK); - off_t offset = vdev->config_offset + PCI_ROM_ADDRESS; char *name; - int fd = vdev->vbasedev.fd; if (vdev->pdev.romfile || !vdev->pdev.rom_bar) { /* Since pci handles romfile, just print a message and return */ @@ -1030,11 +1043,12 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev) * Use the same size ROM BAR as the physical device. The contents * will get filled in later when the guest tries to read it. */ - if (pread(fd, &orig, 4, offset) != 4 || - pwrite(fd, &size, 4, offset) != 4 || - pread(fd, &size, 4, offset) != 4 || - pwrite(fd, &orig, 4, offset) != 4) { - error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); + if (VDEV_CONFIG_READ(vbasedev, PCI_ROM_ADDRESS, 4, &orig) != 4 || + VDEV_CONFIG_WRITE(vbasedev, PCI_ROM_ADDRESS, 4, &size) != 4 || + VDEV_CONFIG_READ(vbasedev, PCI_ROM_ADDRESS, 4, &size) != 4 || + VDEV_CONFIG_WRITE(vbasedev, PCI_ROM_ADDRESS, 4, &orig) != 4) { + + error_report("%s(%s) ROM access failed", __func__, vbasedev->name); return; } @@ -1214,6 +1228,7 @@ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; uint32_t emu_bits = 0, emu_val = 0, phys_val = 0, val; memcpy(&emu_bits, vdev->emulated_config_bits + addr, len); @@ -1226,12 +1241,13 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) if (~emu_bits & (0xffffffffU >> (32 - len * 8))) { ssize_t ret; - ret = pread(vdev->vbasedev.fd, &phys_val, len, - vdev->config_offset + addr); + ret = VDEV_CONFIG_READ(vbasedev, addr, len, &phys_val); if (ret != len) { - error_report("%s(%s, 0x%x, 0x%x) failed: %m", - __func__, vdev->vbasedev.name, addr, len); - return -errno; + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_report("%s(%s, 0x%x, 0x%x) failed: %s", + __func__, vbasedev->name, addr, len, err); + return -1; } phys_val = le32_to_cpu(phys_val); } @@ -1247,15 +1263,19 @@ void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; uint32_t val_le = cpu_to_le32(val); + int ret; trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); /* Write everything to VFIO, let it filter out what we can't write */ - if (pwrite(vdev->vbasedev.fd, &val_le, len, vdev->config_offset + addr) - != len) { - error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %m", - __func__, vdev->vbasedev.name, addr, val, len); + ret = VDEV_CONFIG_WRITE(vbasedev, addr, len, &val_le); + if (ret != len) { + const char *err = ret < 0 ? strerror(-ret) : "short write"; + + error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %s", + __func__, vbasedev->name, addr, val, len, err); } /* MSI/MSI-X Enabling/Disabling */ @@ -1343,9 +1363,12 @@ static bool vfio_msi_setup(VFIOPCIDevice *vdev, int pos, Error **errp) int ret, entries; Error *err = NULL; - if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS"); + ret = VDEV_CONFIG_READ(&vdev->vbasedev, pos + PCI_CAP_FLAGS, + sizeof(ctrl), &ctrl); + if (ret != sizeof(ctrl)) { + const char *errmsg = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed reading MSI PCI_CAP_FLAGS %s", errmsg); return false; } ctrl = le16_to_cpu(ctrl); @@ -1549,34 +1572,43 @@ static bool vfio_pci_relocate_msix(VFIOPCIDevice *vdev, Error **errp) */ static bool vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) { + VFIODevice *vbasedev = &vdev->vbasedev; uint8_t pos; uint16_t ctrl; uint32_t table, pba; - int ret, fd = vdev->vbasedev.fd; struct vfio_irq_info irq_info = { .argsz = sizeof(irq_info), .index = VFIO_PCI_MSIX_IRQ_INDEX }; VFIOMSIXInfo *msix; + int ret; pos = pci_find_capability(&vdev->pdev, PCI_CAP_ID_MSIX); if (!pos) { return true; } - if (pread(fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS"); + ret = VDEV_CONFIG_READ(vbasedev, pos + PCI_MSIX_FLAGS, + sizeof(ctrl), &ctrl); + if (ret != sizeof(ctrl)) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed to read PCI MSIX FLAGS: %s", err); return false; } - if (pread(fd, &table, sizeof(table), - vdev->config_offset + pos + PCI_MSIX_TABLE) != sizeof(table)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE"); + ret = VDEV_CONFIG_READ(vbasedev, pos + PCI_MSIX_TABLE, + sizeof(table), &table); + if (ret != sizeof(table)) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed to read PCI MSIX TABLE: %s", err); return false; } - if (pread(fd, &pba, sizeof(pba), - vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX PBA"); + ret = VDEV_CONFIG_READ(vbasedev, pos + PCI_MSIX_PBA, sizeof(pba), &pba); + if (ret != sizeof(pba)) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed to read PCI MSIX PBA: %s", err); return false; } @@ -1591,7 +1623,7 @@ static bool vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) msix->pba_offset = pba & ~PCI_MSIX_FLAGS_BIRMASK; msix->entries = (ctrl & PCI_MSIX_FLAGS_QSIZE) + 1; - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + ret = vdev->vbasedev.io->get_irq_info(&vdev->vbasedev, &irq_info); if (ret < 0) { error_setg_errno(errp, -ret, "failed to get MSI-X irq info"); g_free(msix); @@ -1735,10 +1767,12 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) } /* Determine what type of BAR this is for registration */ - ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar), - vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr)); + ret = VDEV_CONFIG_READ(&vdev->vbasedev, PCI_BASE_ADDRESS_0 + (4 * nr), + sizeof(pci_bar), &pci_bar); if (ret != sizeof(pci_bar)) { - error_report("vfio: Failed to read BAR %d (%m)", nr); + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_report("vfio: Failed to read BAR %d: %s", nr, err); return; } @@ -2438,21 +2472,25 @@ void vfio_pci_pre_reset(VFIOPCIDevice *vdev) void vfio_pci_post_reset(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; Error *err = NULL; - int nr; + int ret, nr; if (!vfio_intx_enable(vdev, &err)) { error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); } for (nr = 0; nr < PCI_NUM_REGIONS - 1; ++nr) { - off_t addr = vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr); + off_t addr = PCI_BASE_ADDRESS_0 + (4 * nr); uint32_t val = 0; uint32_t len = sizeof(val); - if (pwrite(vdev->vbasedev.fd, &val, len, addr) != len) { - error_report("%s(%s) reset bar %d failed: %m", __func__, - vdev->vbasedev.name, nr); + ret = VDEV_CONFIG_WRITE(vbasedev, addr, len, &val); + if (ret != len) { + const char *errmsg = ret < 0 ? strerror(-ret) : "short write"; + + error_report("%s(%s) reset bar %d failed: %s", __func__, + vbasedev->name, nr, errmsg); } } @@ -2794,10 +2832,10 @@ static bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index = VFIO_PCI_ERR_IRQ_INDEX; - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + ret = vbasedev->io->get_irq_info(vbasedev, &irq_info); if (ret) { /* This can fail for an old kernel or legacy PCI dev */ - trace_vfio_populate_device_get_irq_info_failure(strerror(errno)); + trace_vfio_populate_device_get_irq_info_failure(strerror(-ret)); } else if (irq_info.count == 1) { vdev->pci_aer = true; } else { @@ -2915,8 +2953,11 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) return; } - if (ioctl(vdev->vbasedev.fd, - VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) { + if (vdev->vbasedev.io->get_irq_info(&vdev->vbasedev, &irq_info) < 0) { + return; + } + + if (irq_info.count < 1) { return; } @@ -3368,7 +3409,7 @@ static void vfio_instance_init(Object *obj) vdev->host.function = ~0U; vfio_device_init(vbasedev, VFIO_DEVICE_TYPE_PCI, &vfio_pci_ops, - DEVICE(vdev), false); + &vfio_dev_io_ioctl, DEVICE(vdev), false); vdev->nv_gpudirect_clique = 0xFF; diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c index f491f4dc95..51534fd941 100644 --- a/hw/vfio/platform.c +++ b/hw/vfio/platform.c @@ -648,7 +648,7 @@ static void vfio_platform_instance_init(Object *obj) VFIODevice *vbasedev = &vdev->vbasedev; vfio_device_init(vbasedev, VFIO_DEVICE_TYPE_PLATFORM, &vfio_platform_ops, - DEVICE(vdev), false); + &vfio_dev_io_ioctl, DEVICE(vdev), false); } #ifdef CONFIG_IOMMUFD diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 304030e71d..3512556590 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -116,6 +116,7 @@ typedef struct VFIOIOMMUFDContainer { OBJECT_DECLARE_SIMPLE_TYPE(VFIOIOMMUFDContainer, VFIO_IOMMU_IOMMUFD); typedef struct VFIODeviceOps VFIODeviceOps; +typedef struct VFIODeviceIO VFIODeviceIO; typedef struct VFIODevice { QLIST_ENTRY(VFIODevice) next; @@ -136,6 +137,7 @@ typedef struct VFIODevice { OnOffAuto enable_migration; bool migration_events; VFIODeviceOps *ops; + VFIODeviceIO *io; unsigned int num_irqs; unsigned int num_regions; unsigned int flags; @@ -186,6 +188,29 @@ struct VFIODeviceOps { int (*vfio_load_config)(VFIODevice *vdev, QEMUFile *f); }; +#ifdef CONFIG_LINUX + +/* + * How devices communicate with the server. The default option is through + * ioctl() to the kernel VFIO driver, but vfio-user can use a socket to a remote + * process. + */ +struct VFIODeviceIO { + int (*device_feature)(VFIODevice *vdev, struct vfio_device_feature *); + int (*get_region_info)(VFIODevice *vdev, + struct vfio_region_info *info); + int (*get_irq_info)(VFIODevice *vdev, struct vfio_irq_info *irq); + int (*set_irqs)(VFIODevice *vdev, struct vfio_irq_set *irqs); + int (*region_read)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, + void *data); + int (*region_write)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, + void *data); +}; + +extern VFIODeviceIO vfio_dev_io_ioctl; + +#endif /* CONFIG_LINUX */ + typedef struct VFIOGroup { int fd; int groupid; @@ -317,6 +342,6 @@ int vfio_get_dirty_bitmap(const VFIOContainerBase *bcontainer, uint64_t iova, bool vfio_device_get_name(VFIODevice *vbasedev, Error **errp); void vfio_device_set_fd(VFIODevice *vbasedev, const char *str, Error **errp); void vfio_device_init(VFIODevice *vbasedev, int type, VFIODeviceOps *ops, - DeviceState *dev, bool ram_discard); + VFIODeviceIO *io, DeviceState *dev, bool ram_discard); int vfio_device_get_aw_bits(VFIODevice *vdev); #endif /* HW_VFIO_VFIO_COMMON_H */ From patchwork Wed Feb 19 14:48:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982415 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4503CC021B2 for ; Wed, 19 Feb 2025 14:55:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOh-0003NE-RK; Wed, 19 Feb 2025 09:49:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOg-0003N1-22; Wed, 19 Feb 2025 09:49:58 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOb-0007F4-1q; Wed, 19 Feb 2025 09:49:57 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JC0WDl014951; Wed, 19 Feb 2025 06:49:49 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=ZoRZlRVlHeatbh7T5NU3gk8yDktXfUE0gWqJQbzzV dI=; b=vAL+A/dxWsPyBgi4qIQOyb5Rrbxbp7+5LzsLuYupOr/d33lOF5gxxRtP0 iV3XM2XoHKnIf/olocQfbtDYe1Ak2ALH9tyLCwIctIoL8nVH4RLtWJ3gqKltckWd AZPSFT1N7SspbyRfIpBt/mt43dwpxae84BGgYg9Y2rdjuS80osNOa1J+O0mZNPJa 7yS6sJX/blYdSzB2zedMsjOwIo0QNa5bSpvjgFwwXZDJDd7iqoh5CgPQhcUZWeuS O7r1Da8UBx/QCGOTIX40kL20XqqDOcc/2jnf1kzogDSX5qgI4+rCGZbfs6fyx8bP X/002U1qlMqc+N3o+odl+c3wO8U3g== Received: from dm5pr21cu001.outbound.protection.outlook.com (mail-centralusazlp17011031.outbound.protection.outlook.com [40.93.13.31]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxs2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:49:49 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=o0WlFvnBgSNXO8voEbsLFkRJQ6x7aibpOZdEXm/QlhCeNwh0P6cjWP2imlyRadu0b7h+s59Kd1b/eC5Ma6eeTBgaosIaEj/8PXh8CHh1eCO66TH2byY+yY/KI0Pof/aFHaya3J7kPwfbebL3wEMHnBLm67JVh37774g2Ftq2nwQS7V6oqHhS7YzRtQUB0kL5IUZb3KSPEiOJy1WujNLUsmImPLeGAnKiOpxdcxS8aezJdEIqBOPFJVphfdKCrLhRr5lSWqZEpH0arUfmqvAPwuF1MZM+I9q7by8DJeJd1R82jt6QIYo/L2wKQmBY5Om076oLGdFOeauPD+5q9sZKTg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ZoRZlRVlHeatbh7T5NU3gk8yDktXfUE0gWqJQbzzVdI=; b=MfkxzZBaneqSk0LrnZvzsCStr5Y+dwSCtAZNyuRrYjXwozb/asZby34oyTe6dQtMsZF4WeDlzrDXgUYpy2nS9Oq9M74nC6+uHvgoUpIUKBfS0HsPqmPWps1OK5eaKojaA+6QeB9zXBqwuBwM6J4xDvx7axzMUJDubDoXKpbc+G4JmtAz3995/GX+3JLytAQhstqFoL8HEWNem8DXrGWWC7+aup3ZGNaqU5GRIRUI/X0WKUPmBdQHz7cNFs+Aqen5kiQ2/bKwWwIyKoJuu8W5xRh5SLLndpi1Ydhzp3plW31JKz1wx/ZFcK5bxAAwuAwVfO+wnbZGle/QgZ1Z8NuWQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ZoRZlRVlHeatbh7T5NU3gk8yDktXfUE0gWqJQbzzVdI=; b=pcBZHXZiZ/X3ZVbAlNaRQ2mMU4eYvrDuRGLLdZLIAT2nyGJc82Ibe28aCr6X8HRbwNxK7JdIoJd7bSDspJy+YVHfPasRgJl0pb8vZKpdOO5L+Eu5ldSF1ouwaf37Yy7gwkHbmZxicgaTPZqfkv0OE2Xne5lBl4sGIE+aFrdEtCSgjSMIAHt6XiW/1oT0XLNaANN9KdYeaLzw5EBHBhRaZei71+AubNHHeHvCCAawaEdwEiNirWn99OD8x+fk9MzrxkvBDzBWbhaJCs3PUYmsW+X1cMexKHNn5dY5yxgauZv4CIg43VkzWMGLmDedZxFNPbUY9B1B4x4yNmS89+iBLw== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:49:47 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:47 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" Subject: [PATCH v8 11/28] vfio-user: introduce vfio-user protocol specification Date: Wed, 19 Feb 2025 15:48:41 +0100 Message-Id: <20250219144858.266455-12-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: bb6990f2-a293-424a-7894-08dd50f4a7a3 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: nqBN/Xm5tfjEtRDOh+Hz+TtXTjl5/iz35Ae3d7sEMuE6tx9Gu0bBKInbYmNcKHQCVfK2SoxjfS7IC8LfwHaw5ATo0pI0qMKyA57azul9BgFAMxfezDA44nuSeOTcLESCN709BQe8m9SUwfqIGkD2srPnetGdKwy6hV3VND3HLDckesvSWOPjAFqgeQYx0CZe+rcBoF8IoIx1gcDZtECMu4+JzlZ4xDldBixIkxJIad/5lAMvBqvAbYXs1hKT+1TgxSykm8Jxh8/xRkCa0/aLW30ofo0ybAtPDbCuX2joexUmUc9XNoEU2Ym9wIssnrcF89Y4liMkEDTKmCSdgHIhKutCgEwiFU6C7npDjWYjuw3X9AOmnLOAhCCNQfj7N7rjSXuygafBVapBJHQk8COxmFI50cNjF8ybZAxi8Wt3h7Dy0xzjYh0FqFwFPietsSwIGk4gQLyA+u3sgUQ8uoahIg4EYHBzQXulP0tU/BuT0OX7gFBRWvHhM+37Qz1ncsGXholSJew2xVWGff/dX4y1OAg6M1dOC9EOFVxs0R6I47yu46jAp2HZ5vXhCPpfNzWrkDKGGaOWdkExFcycETTMYJx4Fn8AghQW0JXqBwDCVAtCq+Wdi4D0+yuJkE6J4cnhtY99hNnQ1BR/SavdfKkHsVVzyGhymaUCIYvAjKlJzFtnJeNsThucJP9+1kAUPIjw080pLX4pQYPrI3aUw73Z8memz9vzcfP6NQ/3DbTVivf1n2cvavrp/6Q3vbYBcmNc1Bl1zIDu06jJXyLwRJb0vmXubI8H2cNRjULnF/EGR5W+FCU1mbDhtrKFBcv4ACBTYX+d00PgGuPqa+hU4rokwgQ+/QxxYAhtzz7AAnZLcXQEz8dcRNE7Gu3E14BtFQGNPV2xxwrtxsdboR4cKvtQppOXE1OfOwdUuwj3yuNC9rbCJmuOZqf5EEhuoNdQK/0wfmUrA4oiSARca2LJXKIsWO30R5OMZZKnbGriqBTvM/eIaBC/bLC47Gcg7H+J3Fc+a7G0PwOhfPOBoVFlMzw2bA/YCPXJI3jlt9fWKG5WLPnuXAKF95gSlzmWjg9jCi1LmTg5qYxp0JC8eN1aMHzMZ5EonOTzGhDa/bjdzv4vi6wM1kpP/7g/JAfz9tVfq4sOBMdA6etNiMIlCK2BaW2ElKkj8hqqIUVEVDDAzdX8SAXmqSZj9+zMh5ajBbM42UmBMOII5eqMtaXa1ArYw8oHs/ZPXwyZo9TIk2TXAWg6wlQQ1Zlv/qKVOtlFkrS+R0pcgyaYYwicepV0V7LwLfyW6FQWYR+yF5cA5Gf65vGoEMyhALqD7NYwFgiB7MpvY88ZDoc6yo5f8af4vZXcIPAGfr6X30fQeDuj+wyZeD23CN1G1/e6zb+xOGVhG1kWYbxu X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: B4qe/MN9VSD7qQflSbBWC++Jaa/uDzM780ud/vw4TOFDDszvlDspvegf5/a4ye8+R8e8XNfFb1+hFnC3FSrgRCZlihSYFHjAnuZ+7chclr09Tejt2707f/WdkDV0lTkeMft2GB/EkFAC0HmrI2OiCd6C/Q0/wAhtzSFTXS4M1YpGIcFR90MfiKVg7MRuuOsYTfWkZMszip+TrF64dey7go2m1itYA6Dwe6tbWhCxiV1teUFdVqEB0Z1hgPcInrxc8YKsBl8TRJyZNPJ0dNEj6PZMU7n+5KtVTsmNJFiGGDnQosmm+npi9m+wngM0/EXWtDcBLSiO/tqY/XZoCwQVE77NRXnSkQamTT81plrUKm8AojFAQxp4GXQcEKGOlZznO/Jd6H4/8WpwtZ7hLPj80NtkJ8uV5CxnrywjXbHLPQWKJRMODpJ/g957+8ldafec4hktH6wpqOU5xFREIbr3DHo8GanveGULWAIt4pDS1NCNUxIU8oTycxfQNaY1Lx6pKNYGUTFaAGSQJROBZ+szCuo03ojxyeimjZJVCntdpSH3P2xD3zBnQbNrokinfK6d+6b5Oev4gH8ZH6hCtN+HOPWTOM9mV5oeGvdbKNWKhNqEgcZQ5NpHlO68bk9NFScs7SjnctVrEsNNDlIG4zWn4jrE17OthfFXklWpzQVSAp0CL3HVr0uBRtDvuHhpaE21GuTQzA5Sm3NY5IR0infpExwxcoKbsb6Nea0uM/mYcncVQYSXZq/EQ4TdQYpyU0N/2b4W4LguCjdjcWpmMoTiXi/XXBO53R7aPk5v/3w3GMl78HVmLlmrW+U6WelsK4dqQAnfakoxWXbYsjkYWxFLVINuq8mssnToKPl4jGgCs0y9IuPwd3Ib9gGx4UUlxt7eGiHDT7IAFlbpV2c1LGlz44sjjDLf8gWvoYryhjjGrhblrXZujqTOujaDa6338aN98i83lMlMQPDfUtjghIbj3k8ucObGtlDmnNW1ORzKZa/DWh4a2EnSOpJxwKVsY17QUg3L8+fD6ycveBCH2NzANcKxFIdTB8zSWn/9AGkDx3kbH42mUz0N+75Yuhz9AmuskR9+1lJDgMoDp8M+N5+Yjyrn3uT3nV9xiDLdGsCcDEbwuBVTXkG8x8pgxZPoYNlHEKfqtJ1vEoQ4JjtdIlUIyILHwqUzVMSgRZLrA9HUlrmVZJz/fGGS+KjbGSe2hIH6amly3dObMh7rapLLzqwrAoAjaLbmng7JA8KTu+FlK3QeUkM7C9VgZBpi/8SSHQcc0vJK9jSg+BqiK5kWPZYx3HSZwlVwEraJVM5A67Clv6cmNpHSFnjd9VCQ8sU2AaVJXtm4GZIf0A2j+5rQ/F48P7FBdnmfQQp06++jxeUbO7wZjSOICCwa/B0TeFZ7iip2spY0X9KiBoeWiGfI5KHZxOfnyTDEiuHs4EZEkU2ouLlqBVa5Mjk7NYemi1KxWN4ff5Ng4TOv45gEQzYWSyk/DlgdoAuj5aipKQxLMveX52YyZGcfDkSwhyKaxH9MpDUnX9d6trq7nB2gLt9PASiivmCXum9hvmtKhUTVc43Ws68m6cXEodKBtXC+kvSCoEWP X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: bb6990f2-a293-424a-7894-08dd50f4a7a3 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:47.4136 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bAsMZcNjRgS0PkmRcLpIBFox0JbMXWsSpllRqzvpSRaTD55WJmC/mXwFtGy27H/51/UkhPU80ri6/r+SZaXdMg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-GUID: wmnWl669aD8bGXX41e7MlqNrANkqO4lN X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5ef8d cx=c_pps a=9aUYF2eN3k5HGc45ncVHqA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=VwQbUJbxAAAA:8 a=pGLkceISAAAA:8 a=64Cc0HZtAAAA:8 a=20KFwNOVAAAA:8 a=z4glEzOvAAAA:8 a=h9SPtozqHe0Wh_k42joA:9 a=x5O-bmgOEKQMiMGR:21 a=N0yriQgkpQ4A:10 a=HplO-upLQ7EA:10 a=14NRyaPF5x3gF6G45PvQ:22 a=92dS5hN0c3Q7EetK7xW5:22 X-Proofpoint-ORIG-GUID: wmnWl669aD8bGXX41e7MlqNrANkqO4lN X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, WEIRD_QUOTING=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Thanos Makatos This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts. It has been earlier discussed as an RFC in: "RFC: use VFIO over a UNIX domain socket to implement device offloading" Signed-off-by: Thanos Makatos Signed-off-by: John Levon --- MAINTAINERS | 8 +- docs/devel/index-internals.rst | 1 + docs/devel/vfio-user.rst | 1522 ++++++++++++++++++++++++++++++++ 3 files changed, 1530 insertions(+), 1 deletion(-) create mode 100644 docs/devel/vfio-user.rst diff --git a/MAINTAINERS b/MAINTAINERS index 3848d37a38..3e7e6743cc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4148,12 +4148,18 @@ F: hw/remote/proxy-memory-listener.c F: include/hw/remote/proxy-memory-listener.h F: hw/remote/iohub.c F: include/hw/remote/iohub.h -F: subprojects/libvfio-user F: hw/remote/vfio-user-obj.c F: include/hw/remote/vfio-user-obj.h F: hw/remote/iommu.c F: include/hw/remote/iommu.h +VFIO-USER: +M: John Levon +M: Thanos Makatos +S: Supported +F: docs/devel/vfio-user.rst +F: subprojects/libvfio-user + EBPF: M: Jason Wang R: Andrew Melnychenko diff --git a/docs/devel/index-internals.rst b/docs/devel/index-internals.rst index bca597c658..0bc24f0e51 100644 --- a/docs/devel/index-internals.rst +++ b/docs/devel/index-internals.rst @@ -21,6 +21,7 @@ Details about QEMU's various subsystems including how to add features to them. s390-dasd-ipl tracing vfio-iommufd + vfio-user writing-monitor-commands virtio-backends crypto diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst new file mode 100644 index 0000000000..0d96477a68 --- /dev/null +++ b/docs/devel/vfio-user.rst @@ -0,0 +1,1522 @@ +.. include:: +******************************** +vfio-user Protocol Specification +******************************** + +-------------- +Version_ 0.9.1 +-------------- + +.. contents:: Table of Contents + +Introduction +============ +vfio-user is a protocol that allows a device to be emulated in a separate +process outside of a Virtual Machine Monitor (VMM). vfio-user devices consist +of a generic VFIO device type, living inside the VMM, which we call the client, +and the core device implementation, living outside the VMM, which we call the +server. + +The vfio-user specification is partly based on the +`Linux VFIO ioctl interface `_. + +VFIO is a mature and stable API, backed by an extensively used framework. The +existing VFIO client implementation in QEMU (``qemu/hw/vfio/``) can be largely +re-used, though there is nothing in this specification that requires that +particular implementation. None of the VFIO kernel modules are required for +supporting the protocol, on either the client or server side. Some source +definitions in VFIO are re-used for vfio-user. + +The main idea is to allow a virtual device to function in a separate process in +the same host over a UNIX domain socket. A UNIX domain socket (``AF_UNIX``) is +chosen because file descriptors can be trivially sent over it, which in turn +allows: + +* Sharing of client memory for DMA with the server. +* Sharing of server memory with the client for fast MMIO. +* Efficient sharing of eventfd's for triggering interrupts. + +Other socket types could be used which allow the server to run in a separate +guest in the same host (``AF_VSOCK``) or remotely (``AF_INET``). Theoretically +the underlying transport does not necessarily have to be a socket, however we do +not examine such alternatives. In this protocol version we focus on using a UNIX +domain socket and introduce basic support for the other two types of sockets +without considering performance implications. + +While passing of file descriptors is desirable for performance reasons, support +is not necessary for either the client or the server in order to implement the +protocol. There is always an in-band, message-passing fall back mechanism. + +Overview +======== + +VFIO is a framework that allows a physical device to be securely passed through +to a user space process; the device-specific kernel driver does not drive the +device at all. Typically, the user space process is a VMM and the device is +passed through to it in order to achieve high performance. VFIO provides an API +and the required functionality in the kernel. QEMU has adopted VFIO to allow a +guest to directly access physical devices, instead of emulating them in +software. + +vfio-user reuses the core VFIO concepts defined in its API, but implements them +as messages to be sent over a socket. It does not change the kernel-based VFIO +in any way, in fact none of the VFIO kernel modules need to be loaded to use +vfio-user. It is also possible for the client to concurrently use the current +kernel-based VFIO for one device, and vfio-user for another device. + +VFIO Device Model +----------------- + +A device under VFIO presents a standard interface to the user process. Many of +the VFIO operations in the existing interface use the ``ioctl()`` system call, and +references to the existing interface are called the ``ioctl()`` implementation in +this document. + +The following sections describe the set of messages that implement the vfio-user +interface over a socket. In many cases, the messages are analogous to data +structures used in the ``ioctl()`` implementation. Messages derived from the +``ioctl()`` will have a name derived from the ``ioctl()`` command name. E.g., the +``VFIO_DEVICE_GET_INFO`` ``ioctl()`` command becomes a +``VFIO_USER_DEVICE_GET_INFO`` message. The purpose of this reuse is to share as +much code as feasible with the ``ioctl()`` implementation``. + +Connection Initiation +^^^^^^^^^^^^^^^^^^^^^ + +After the client connects to the server, the initial client message is +``VFIO_USER_VERSION`` to propose a protocol version and set of capabilities to +apply to the session. The server replies with a compatible version and set of +capabilities it supports, or closes the connection if it cannot support the +advertised version. + +Device Information +^^^^^^^^^^^^^^^^^^ + +The client uses a ``VFIO_USER_DEVICE_GET_INFO`` message to query the server for +information about the device. This information includes: + +* The device type and whether it supports reset (``VFIO_DEVICE_FLAGS_``), +* the number of device regions, and +* the device presents to the client the number of interrupt types the device + supports. + +Region Information +^^^^^^^^^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_REGION_INFO`` messages to query the +server for information about the device's regions. This information describes: + +* Read and write permissions, whether it can be memory mapped, and whether it + supports additional capabilities (``VFIO_REGION_INFO_CAP_``). +* Region index, size, and offset. + +When a device region can be mapped by the client, the server provides a file +descriptor which the client can ``mmap()``. The server is responsible for +polling for client updates to memory mapped regions. + +Region Capabilities +""""""""""""""""""" + +Some regions have additional capabilities that cannot be described adequately +by the region info data structure. These capabilities are returned in the +region info reply in a list similar to PCI capabilities in a PCI device's +configuration space. + +Sparse Regions +"""""""""""""" +A region can be memory-mappable in whole or in part. When only a subset of a +region can be mapped by the client, a ``VFIO_REGION_INFO_CAP_SPARSE_MMAP`` +capability is included in the region info reply. This capability describes +which portions can be mapped by the client. + +.. Note:: + For example, in a virtual NVMe controller, sparse regions can be used so + that accesses to the NVMe registers (found in the beginning of BAR0) are + trapped (an infrequent event), while allowing direct access to the doorbells + (an extremely frequent event as every I/O submission requires a write to + BAR0), found in the next page after the NVMe registers in BAR0. + +Device-Specific Regions +""""""""""""""""""""""" + +A device can define regions additional to the standard ones (e.g. PCI indexes +0-8). This is achieved by including a ``VFIO_REGION_INFO_CAP_TYPE`` capability +in the region info reply of a device-specific region. Such regions are reflected +in ``struct vfio_user_device_info.num_regions``. Thus, for PCI devices this +value can be equal to, or higher than, ``VFIO_PCI_NUM_REGIONS``. + +Region I/O via file descriptors +------------------------------- + +For unmapped regions, region I/O from the client is done via +``VFIO_USER_REGION_READ/WRITE``. As an optimization, ioeventfds or ioregionfds +may be configured for sub-regions of some regions. A client may request +information on these sub-regions via ``VFIO_USER_DEVICE_GET_REGION_IO_FDS``; by +configuring the returned file descriptors as ioeventfds or ioregionfds, the +server can be directly notified of I/O (for example, by KVM) without taking a +trip through the client. + +Interrupts +^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_IRQ_INFO`` messages to query the server +for the device's interrupt types. The interrupt types are specific to the bus +the device is attached to, and the client is expected to know the capabilities +of each interrupt type. The server can signal an interrupt by directly injecting +interrupts into the guest via an event file descriptor. The client configures +how the server signals an interrupt with ``VFIO_USER_SET_IRQS`` messages. + +Device Read and Write +^^^^^^^^^^^^^^^^^^^^^ + +When the guest executes load or store operations to an unmapped device region, +the client forwards these operations to the server with +``VFIO_USER_REGION_READ`` or ``VFIO_USER_REGION_WRITE`` messages. The server +will reply with data from the device on read operations or an acknowledgement on +write operations. See `Read and Write Operations`_. + +Client memory access +-------------------- + +The client uses ``VFIO_USER_DMA_MAP`` and ``VFIO_USER_DMA_UNMAP`` messages to +inform the server of the valid DMA ranges that the server can access on behalf +of a device (typically, VM guest memory). DMA memory may be accessed by the +server via ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages over the +socket. In this case, the "DMA" part of the naming is a misnomer. + +Actual direct memory access of client memory from the server is possible if the +client provides file descriptors the server can ``mmap()``. Note that ``mmap()`` +privileges cannot be revoked by the client, therefore file descriptors should +only be exported in environments where the client trusts the server not to +corrupt guest memory. + +See `Read and Write Operations`_. + +Client/server interactions +========================== + +Socket +------ + +A server can serve: + +1) one or more clients, and/or +2) one or more virtual devices, belonging to one or more clients. + +The current protocol specification requires a dedicated socket per +client/server connection. It is a server-side implementation detail whether a +single server handles multiple virtual devices from the same or multiple +clients. The location of the socket is implementation-specific. Multiplexing +clients, devices, and servers over the same socket is not supported in this +version of the protocol. + +Authentication +-------------- + +For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files, +therefore it is up to the management layer to set up the socket as required. +Socket types that span guests or hosts will require a proper authentication +mechanism. Defining that mechanism is deferred to a future version of the +protocol. + +Command Concurrency +------------------- + +A client may pipeline multiple commands without waiting for previous command +replies. The server will process commands in the order they are received. A +consequence of this is if a client issues a command with the *No_reply* bit, +then subsequently issues a command without *No_reply*, the older command will +have been processed before the reply to the younger command is sent by the +server. The client must be aware of the device's capability to process +concurrent commands if pipelining is used. For example, pipelining allows +multiple client threads to concurrently access device regions; the client must +ensure these accesses obey device semantics. + +An example is a frame buffer device, where the device may allow concurrent +access to different areas of video memory, but may have indeterminate behavior +if concurrent accesses are performed to command or status registers. + +Note that unrelated messages sent from the server to the client can appear in +between a client to server request/reply and vice versa. + +Implementers should be prepared for certain commands to exhibit potentially +unbounded latencies. For example, ``VFIO_USER_DEVICE_RESET`` may take an +arbitrarily long time to complete; clients should take care not to block +unnecessarily. + +Socket Disconnection Behavior +----------------------------- +The server and the client can disconnect from each other, either intentionally +or unexpectedly. Both the client and the server need to know how to handle such +events. + +Server Disconnection +^^^^^^^^^^^^^^^^^^^^ +A server disconnecting from the client may indicate that: + +1) A virtual device has been restarted, either intentionally (e.g. because of a + device update) or unintentionally (e.g. because of a crash). +2) A virtual device has been shut down with no intention to be restarted. + +It is impossible for the client to know whether or not a failure is +intermittent or innocuous and should be retried, therefore the client should +reset the VFIO device when it detects the socket has been disconnected. +Error recovery will be driven by the guest's device error handling +behavior. + +Client Disconnection +^^^^^^^^^^^^^^^^^^^^ +The client disconnecting from the server primarily means that the client +has exited. Currently, this means that the guest is shut down so the device is +no longer needed therefore the server can automatically exit. However, there +can be cases where a client disconnection should not result in a server exit: + +1) A single server serving multiple clients. +2) A multi-process QEMU upgrading itself step by step, which is not yet + implemented. + +Therefore in order for the protocol to be forward compatible, the server should +respond to a client disconnection as follows: + + - all client memory regions are unmapped and cleaned up (including closing any + passed file descriptors) + - all IRQ file descriptors passed from the old client are closed + - the device state should otherwise be retained + +The expectation is that when a client reconnects, it will re-establish IRQ and +client memory mappings. + +If anything happens to the client (such as qemu really did exit), the control +stack will know about it and can clean up resources accordingly. + +Security Considerations +----------------------- + +Speaking generally, vfio-user clients should not trust servers, and vice versa. +Standard tools and mechanisms should be used on both sides to validate input and +prevent against denial of service scenarios, buffer overflow, etc. + +Request Retry and Response Timeout +---------------------------------- +A failed command is a command that has been successfully sent and has been +responded to with an error code. Failure to send the command in the first place +(e.g. because the socket is disconnected) is a different type of error examined +earlier in the disconnect section. + +.. Note:: + QEMU's VFIO retries certain operations if they fail. While this makes sense + for real HW, we don't know for sure whether it makes sense for virtual + devices. + +Defining a retry and timeout scheme is deferred to a future version of the +protocol. + +Message sizes +------------- + +Some requests have an ``argsz`` field. In a request, it defines the maximum +expected reply payload size, which should be at least the size of the fixed +reply payload headers defined here. The *request* payload size is defined by the +usual ``msg_size`` field in the header, not the ``argsz`` field. + +In a reply, the server sets ``argsz`` field to the size needed for a full +payload size. This may be less than the requested maximum size. This may be +larger than the requested maximum size: in that case, the full payload is not +included in the reply, but the ``argsz`` field in the reply indicates the needed +size, allowing a client to allocate a larger buffer for holding the reply before +trying again. + +In addition, during negotiation (see `Version`_), the client and server may +each specify a ``max_data_xfer_size`` value; this defines the maximum data that +may be read or written via one of the ``VFIO_USER_DMA/REGION_READ/WRITE`` +messages; see `Read and Write Operations`_. + +Protocol Specification +====================== + +To distinguish from the base VFIO symbols, all vfio-user symbols are prefixed +with ``vfio_user`` or ``VFIO_USER``. In this revision, all data is in the +endianness of the host system, although this may be relaxed in future +revisions in cases where the client and server run on different hosts +with different endianness. + +Unless otherwise specified, all sizes should be presumed to be in bytes. + +.. _Commands: + +Commands +-------- +The following table lists the VFIO message command IDs, and whether the +message command is sent from the client or the server. + +====================================== ========= ================= +Name Command Request Direction +====================================== ========= ================= +``VFIO_USER_VERSION`` 1 client -> server +``VFIO_USER_DMA_MAP`` 2 client -> server +``VFIO_USER_DMA_UNMAP`` 3 client -> server +``VFIO_USER_DEVICE_GET_INFO`` 4 client -> server +``VFIO_USER_DEVICE_GET_REGION_INFO`` 5 client -> server +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` 6 client -> server +``VFIO_USER_DEVICE_GET_IRQ_INFO`` 7 client -> server +``VFIO_USER_DEVICE_SET_IRQS`` 8 client -> server +``VFIO_USER_REGION_READ`` 9 client -> server +``VFIO_USER_REGION_WRITE`` 10 client -> server +``VFIO_USER_DMA_READ`` 11 server -> client +``VFIO_USER_DMA_WRITE`` 12 server -> client +``VFIO_USER_DEVICE_RESET`` 13 client -> server +``VFIO_USER_REGION_WRITE_MULTI`` 15 client -> server +====================================== ========= ================= + +Header +------ + +All messages, both command messages and reply messages, are preceded by a +16-byte header that contains basic information about the message. The header is +followed by message-specific data described in the sections below. + ++----------------+--------+-------------+ +| Name | Offset | Size | ++================+========+=============+ +| Message ID | 0 | 2 | ++----------------+--------+-------------+ +| Command | 2 | 2 | ++----------------+--------+-------------+ +| Message size | 4 | 4 | ++----------------+--------+-------------+ +| Flags | 8 | 4 | ++----------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0-3 | Type | | +| | +-----+------------+ | +| | | 4 | No_reply | | +| | +-----+------------+ | +| | | 5 | Error | | +| | +-----+------------+ | ++----------------+--------+-------------+ +| Error | 12 | 4 | ++----------------+--------+-------------+ +| | 16 | variable | ++----------------+--------+-------------+ + +* *Message ID* identifies the message, and is echoed in the command's reply + message. Message IDs belong entirely to the sender, can be re-used (even + concurrently) and the receiver must not make any assumptions about their + uniqueness. +* *Command* specifies the command to be executed, listed in Commands_. It is + also set in the reply header. +* *Message size* contains the size of the entire message, including the header. +* *Flags* contains attributes of the message: + + * The *Type* bits indicate the message type. + + * *Command* (value 0x0) indicates a command message. + * *Reply* (value 0x1) indicates a reply message acknowledging a previous + command with the same message ID. + * *No_reply* in a command message indicates that no reply is needed for this + command. This is commonly used when multiple commands are sent, and only + the last needs acknowledgement. + * *Error* in a reply message indicates the command being acknowledged had + an error. In this case, the *Error* field will be valid. + +* *Error* in a reply message is an optional UNIX errno value. It may be zero + even if the Error bit is set in Flags. It is reserved in a command message. + +Each command message in Commands_ must be replied to with a reply message, +unless the message sets the *No_Reply* bit. The reply consists of the header +with the *Reply* bit set, plus any additional data. + +If an error occurs, the reply message must only include the reply header. + +As the header is standard in both requests and replies, it is not included in +the command-specific specifications below; each message definition should be +appended to the standard header, and the offsets are given from the end of the +standard header. + +``VFIO_USER_VERSION`` +--------------------- + +.. _Version: + +This is the initial message sent by the client after the socket connection is +established; the same format is used for the server's reply. + +Upon establishing a connection, the client must send a ``VFIO_USER_VERSION`` +message proposing a protocol version and a set of capabilities. The server +compares these with the versions and capabilities it supports and sends a +``VFIO_USER_VERSION`` reply according to the following rules. + +* The major version in the reply must be the same as proposed. If the client + does not support the proposed major, it closes the connection. +* The minor version in the reply must be equal to or less than the minor + version proposed. +* The capability list must be a subset of those proposed. If the server + requires a capability the client did not include, it closes the connection. + +The protocol major version will only change when incompatible protocol changes +are made, such as changing the message format. The minor version may change +when compatible changes are made, such as adding new messages or capabilities, +Both the client and server must support all minor versions less than the +maximum minor version it supports. E.g., an implementation that supports +version 1.3 must also support 1.0 through 1.2. + +When making a change to this specification, the protocol version number must +be included in the form "added in version X.Y" + +Request +^^^^^^^ + +============== ====== ==== +Name Offset Size +============== ====== ==== +version major 0 2 +version minor 2 2 +version data 4 variable (including terminating NUL). Optional. +============== ====== ==== + +The version data is an optional UTF-8 encoded JSON byte array with the following +format: + ++--------------+--------+-----------------------------------+ +| Name | Type | Description | ++==============+========+===================================+ +| capabilities | object | Contains common capabilities that | +| | | the sender supports. Optional. | ++--------------+--------+-----------------------------------+ + +Capabilities: + ++--------------------+---------+------------------------------------------------+ +| Name | Type | Description | ++====================+=========+================================================+ +| max_msg_fds | number | Maximum number of file descriptors that can be | +| | | received by the sender in one message. | +| | | Optional. If not specified then the receiver | +| | | must assume a value of ``1``. | ++--------------------+---------+------------------------------------------------+ +| max_data_xfer_size | number | Maximum ``count`` for data transfer messages; | +| | | see `Read and Write Operations`_. Optional, | +| | | with a default value of 1048576 bytes. | ++--------------------+---------+------------------------------------------------+ +| pgsizes | number | Page sizes supported in DMA map operations | +| | | or'ed together. Optional, with a default value | +| | | of supporting only 4k pages. | ++--------------------+---------+------------------------------------------------+ +| max_dma_maps | number | Maximum number DMA map windows that can be | +| | | valid simultaneously. Optional, with a | +| | | value of 65535 (64k-1). | ++--------------------+---------+------------------------------------------------+ +| migration | object | Migration capability parameters. If missing | +| | | then migration is not supported by the sender. | ++--------------------+---------+------------------------------------------------+ +| write_multiple | boolean | ``VFIO_USER_REGION_WRITE_MULTI`` messages | +| | | are supported if the value is ``true``. | ++--------------------+---------+------------------------------------------------+ + +The migration capability contains the following name/value pairs: + ++-----------------+--------+--------------------------------------------------+ +| Name | Type | Description | ++=================+========+==================================================+ +| pgsize | number | Page size of dirty pages bitmap. The smallest | +| | | between the client and the server is used. | ++-----------------+--------+--------------------------------------------------+ +| max_bitmap_size | number | Maximum bitmap size in ``VFIO_USER_DIRTY_PAGES`` | +| | | and ``VFIO_DMA_UNMAP`` messages. Optional, | +| | | with a default value of 256MB. | ++-----------------+--------+--------------------------------------------------+ + +Reply +^^^^^ + +The same message format is used in the server's reply with the semantics +described above. + +``VFIO_USER_DMA_MAP`` +--------------------- + +This command message is sent by the client to the server to inform it of the +memory regions the server can access. It must be sent before the server can +perform any DMA to the client. It is normally sent directly after the version +handshake is completed, but may also occur when memory is added to the client, +or if the client uses a vIOMMU. + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++-------------+--------+-------------+ +| Name | Offset | Size | ++=============+========+=============+ +| argsz | 0 | 4 | ++-------------+--------+-------------+ +| flags | 4 | 4 | ++-------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0 | readable | | +| | +-----+------------+ | +| | | 1 | writeable | | +| | +-----+------------+ | ++-------------+--------+-------------+ +| offset | 8 | 8 | ++-------------+--------+-------------+ +| address | 16 | 8 | ++-------------+--------+-------------+ +| size | 24 | 8 | ++-------------+--------+-------------+ + +* *argsz* is the size of the above structure. Note there is no reply payload, + so this field differs from other message types. +* *flags* contains the following region attributes: + + * *readable* indicates that the region can be read from. + + * *writeable* indicates that the region can be written to. + +* *offset* is the file offset of the region with respect to the associated file + descriptor, or zero if the region is not mappable +* *address* is the base DMA address of the region. +* *size* is the size of the region. + +This structure is 32 bytes in size, so the message size is 16 + 32 bytes. + +If the DMA region being added can be directly mapped by the server, a file +descriptor must be sent as part of the message meta-data. The region can be +mapped via the mmap() system call. On ``AF_UNIX`` sockets, the file descriptor +must be passed as ``SCM_RIGHTS`` type ancillary data. Otherwise, if the DMA +region cannot be directly mapped by the server, no file descriptor must be sent +as part of the message meta-data and the DMA region can be accessed by the +server using ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages, +explained in `Read and Write Operations`_. A command to map over an existing +region must be failed by the server with ``EEXIST`` set in error field in the +reply. + +Reply +^^^^^ + +There is no payload in the reply message. + +``VFIO_USER_DMA_UNMAP`` +----------------------- + +This command message is sent by the client to the server to inform it that a +DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command +message, is no longer available for DMA. It typically occurs when memory is +subtracted from the client or if the client uses a vIOMMU. The DMA region is +described by the following structure: + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++--------------+--------+------------------------+ +| Name | Offset | Size | ++==============+========+========================+ +| argsz | 0 | 4 | ++--------------+--------+------------------------+ +| flags | 4 | 4 | ++--------------+--------+------------------------+ +| address | 8 | 8 | ++--------------+--------+------------------------+ +| size | 16 | 8 | ++--------------+--------+------------------------+ + +* *argsz* is the maximum size of the reply payload. +* *flags* is unused in this version. +* *address* is the base DMA address of the DMA region. +* *size* is the size of the DMA region. + +The address and size of the DMA region being unmapped must match exactly a +previous mapping. + +Reply +^^^^^ + +Upon receiving a ``VFIO_USER_DMA_UNMAP`` command, if the file descriptor is +mapped then the server must release all references to that DMA region before +replying, which potentially includes in-flight DMA transactions. + +The server responds with the original DMA entry in the request. + + +``VFIO_USER_DEVICE_GET_INFO`` +----------------------------- + +This command message is sent by the client to the server to query for basic +information about the device. + +Request +^^^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the maximum size of the reply payload +* all other fields must be zero. + +Reply +^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* contains the following device attributes. + + * ``VFIO_DEVICE_FLAGS_RESET`` indicates that the device supports the + ``VFIO_USER_DEVICE_RESET`` message. + * ``VFIO_DEVICE_FLAGS_PCI`` indicates that the device is a PCI device. + +* *num_regions* is the number of memory regions that the device exposes. +* *num_irqs* is the number of distinct interrupt types that the device supports. + +This version of the protocol only supports PCI devices. Additional devices may +be supported in future versions. + +``VFIO_USER_DEVICE_GET_REGION_INFO`` +------------------------------------ + +This command message is sent by the client to the server to query for +information about device regions. The VFIO region info structure is defined in +```` (``struct vfio_region_info``). + +Request +^^^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* all other fields must be zero. + +Reply +^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_REGION_INFO_FLAG_READ | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_REGION_INFO_FLAG_WRITE | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_REGION_INFO_FLAG_MMAP | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_REGION_INFO_FLAG_CAPS | | +| | +-----+-----------------------------+ | ++------------+--------+------------------------------+ ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* is the size required for the full reply payload (region info structure + plus the size of any region capabilities) +* *flags* are attributes of the region: + + * ``VFIO_REGION_INFO_FLAG_READ`` allows client read access to the region. + * ``VFIO_REGION_INFO_FLAG_WRITE`` allows client write access to the region. + * ``VFIO_REGION_INFO_FLAG_MMAP`` specifies the client can mmap() the region. + When this flag is set, the reply will include a file descriptor in its + meta-data. On ``AF_UNIX`` sockets, the file descriptors will be passed as + ``SCM_RIGHTS`` type ancillary data. + * ``VFIO_REGION_INFO_FLAG_CAPS`` indicates additional capabilities found in the + reply. + +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* *cap_offset* describes where additional region capabilities can be found. + cap_offset is relative to the beginning of the VFIO region info structure. + The data structure it points is a VFIO cap header defined in + ````. +* *size* is the size of the region. +* *offset* is the offset that should be given to the mmap() system call for + regions with the MMAP attribute. It is also used as the base offset when + mapping a VFIO sparse mmap area, described below. + +VFIO region capabilities +"""""""""""""""""""""""" + +The VFIO region information can also include a capabilities list. This list is +similar to a PCI capability list - each entry has a common header that +identifies a capability and where the next capability in the list can be found. +The VFIO capability header format is defined in ```` (``struct +vfio_info_cap_header``). + +VFIO cap header format +"""""""""""""""""""""" + ++---------+--------+------+ +| Name | Offset | Size | ++=========+========+======+ +| id | 0 | 2 | ++---------+--------+------+ +| version | 2 | 2 | ++---------+--------+------+ +| next | 4 | 4 | ++---------+--------+------+ + +* *id* is the capability identity. +* *version* is a capability-specific version number. +* *next* specifies the offset of the next capability in the capability list. It + is relative to the beginning of the VFIO region info structure. + +VFIO sparse mmap cap header +""""""""""""""""""""""""""" + ++------------------+----------------------------------+ +| Name | Value | ++==================+==================================+ +| id | VFIO_REGION_INFO_CAP_SPARSE_MMAP | ++------------------+----------------------------------+ +| version | 0x1 | ++------------------+----------------------------------+ +| next | | ++------------------+----------------------------------+ +| sparse mmap info | VFIO region info sparse mmap | ++------------------+----------------------------------+ + +This capability is defined when only a subrange of the region supports +direct access by the client via mmap(). The VFIO sparse mmap area is defined in +```` (``struct vfio_region_sparse_mmap_area`` and ``struct +vfio_region_info_cap_sparse_mmap``). + +VFIO region info cap sparse mmap +"""""""""""""""""""""""""""""""" + ++----------+--------+------+ +| Name | Offset | Size | ++==========+========+======+ +| nr_areas | 0 | 4 | ++----------+--------+------+ +| reserved | 4 | 4 | ++----------+--------+------+ +| offset | 8 | 8 | ++----------+--------+------+ +| size | 16 | 8 | ++----------+--------+------+ +| ... | | | ++----------+--------+------+ + +* *nr_areas* is the number of sparse mmap areas in the region. +* *offset* and size describe a single area that can be mapped by the client. + There will be *nr_areas* pairs of offset and size. The offset will be added to + the base offset given in the ``VFIO_USER_DEVICE_GET_REGION_INFO`` to form the + offset argument of the subsequent mmap() call. + +The VFIO sparse mmap area is defined in ```` (``struct +vfio_region_info_cap_sparse_mmap``). + + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` +-------------------------------------- + +Clients can access regions via ``VFIO_USER_REGION_READ/WRITE`` or, if provided, by +``mmap()`` of a file descriptor provided by the server. + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` provides an alternative access mechanism via +file descriptors. This is an optional feature intended for performance +improvements where an underlying sub-system (such as KVM) supports communication +across such file descriptors to the vfio-user server, without needing to +round-trip through the client. + +The server returns an array of sub-regions for the requested region. Each +sub-region describes a span (offset and size) of a region, along with the +requested file descriptor notification mechanism to use. Each sub-region in the +response message may choose to use a different method, as defined below. The +two mechanisms supported in this specification are ioeventfds and ioregionfds. + +The server in addition returns a file descriptor in the ancillary data; clients +are expected to configure each sub-region's file descriptor with the requested +notification method. For example, a client could configure KVM with the +requested ioeventfd via a ``KVM_IOEVENTFD`` ``ioctl()``. + +Request +^^^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried +* all other fields must be zero + +The client must set ``flags`` to zero and specify the region being queried in +the ``index``. + +Reply +^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ +| sub-regions | 16 | ... | ++-------------+--------+------+ + +* *argsz* is the size of the region IO FD info structure plus the + total size of the sub-region array. Thus, each array entry "i" is at offset + i * ((argsz - 32) / count). Note that currently this is 40 bytes for both IO + FD types, but this is not to be relied on. As elsewhere, this indicates the + full reply payload size needed. +* *flags* must be zero +* *index* is the index of memory region being queried +* *count* is the number of sub-regions in the array +* *sub-regions* is the array of Sub-Region IO FD info structures + +The reply message will additionally include at least one file descriptor in the +ancillary data. Note that more than one sub-region may share the same file +descriptor. + +Note that it is the client's responsibility to verify the requested values (for +example, that the requested offset does not exceed the region's bounds). + +Each sub-region given in the response has one of two possible structures, +depending whether *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` or +``VFIO_USER_IO_FD_TYPE_IOREGIONFD``: + +Sub-Region IO FD info format (ioeventfd) +"""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| datamatch | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations +* *fd_index* is the index in the ancillary data of the FD to use for ioeventfd + notification; it may be shared. +* *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` +* *flags* is any of: + + * ``KVM_IOEVENTFD_FLAG_DATAMATCH`` + * ``KVM_IOEVENTFD_FLAG_PIO`` + * ``KVM_IOEVENTFD_FLAG_VIRTIO_CCW_NOTIFY`` (FIXME: makes sense?) + +* *datamatch* is the datamatch value if needed + +See https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt, *4.59 +KVM_IOEVENTFD* for further context on the ioeventfd-specific fields. + +Sub-Region IO FD info format (ioregionfd) +""""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| user_data | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations; ``KVM_IOREGION_POSTED_WRITES`` + must be set in *flags* in this case +* *fd_index* is the index in the ancillary data of the FD to use for ioregionfd + messages; it may be shared +* *type* is ``VFIO_USER_IO_FD_TYPE_IOREGIONFD`` +* *flags* is any of: + + * ``KVM_IOREGION_PIO`` + * ``KVM_IOREGION_POSTED_WRITES`` + +* *user_data* is an opaque value passed back to the server via a message on the + file descriptor + +For further information on the ioregionfd-specific fields, see: +https://lore.kernel.org/kvm/cover.1613828726.git.eafanasova@gmail.com/ + +(FIXME: update with final API docs.) + +``VFIO_USER_DEVICE_GET_IRQ_INFO`` +--------------------------------- + +This command message is sent by the client to the server to query for +information about device interrupt types. The VFIO IRQ info structure is +defined in ```` (``struct vfio_irq_info``). + +Request +^^^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the maximum size of the reply payload (16 bytes today) +* index is the index of IRQ type being queried (e.g. ``VFIO_PCI_MSIX_IRQ_INDEX``) +* all other fields must be zero + +Reply +^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* defines IRQ attributes: + + * ``VFIO_IRQ_INFO_EVENTFD`` indicates the IRQ type can support server eventfd + signalling. + * ``VFIO_IRQ_INFO_MASKABLE`` indicates that the IRQ type supports the ``MASK`` + and ``UNMASK`` actions in a ``VFIO_USER_DEVICE_SET_IRQS`` message. + * ``VFIO_IRQ_INFO_AUTOMASKED`` indicates the IRQ type masks itself after being + triggered, and the client must send an ``UNMASK`` action to receive new + interrupts. + * ``VFIO_IRQ_INFO_NORESIZE`` indicates ``VFIO_USER_SET_IRQS`` operations setup + interrupts as a set, and new sub-indexes cannot be enabled without disabling + the entire type. +* index is the index of IRQ type being queried +* count describes the number of interrupts of the queried type. + +``VFIO_USER_DEVICE_SET_IRQS`` +----------------------------- + +This command message is sent by the client to the server to set actions for +device interrupt types. The VFIO IRQ set structure is defined in +```` (``struct vfio_irq_set``). + +Request +^^^^^^^ + ++-------+--------+------------------------------+ +| Name | Offset | Size | ++=======+========+==============================+ +| argsz | 0 | 4 | ++-------+--------+------------------------------+ +| flags | 4 | 4 | ++-------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_IRQ_SET_DATA_NONE | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_IRQ_SET_DATA_BOOL | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_IRQ_SET_DATA_EVENTFD | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_IRQ_SET_ACTION_MASK | | +| | +-----+-----------------------------+ | +| | | 4 | VFIO_IRQ_SET_ACTION_UNMASK | | +| | +-----+-----------------------------+ | +| | | 5 | VFIO_IRQ_SET_ACTION_TRIGGER | | +| | +-----+-----------------------------+ | ++-------+--------+------------------------------+ +| index | 8 | 4 | ++-------+--------+------------------------------+ +| start | 12 | 4 | ++-------+--------+------------------------------+ +| count | 16 | 4 | ++-------+--------+------------------------------+ +| data | 20 | variable | ++-------+--------+------------------------------+ + +* *argsz* is the size of the VFIO IRQ set request payload, including any *data* + field. Note there is no reply payload, so this field differs from other + message types. +* *flags* defines the action performed on the interrupt range. The ``DATA`` + flags describe the data field sent in the message; the ``ACTION`` flags + describe the action to be performed. The flags are mutually exclusive for + both sets. + + * ``VFIO_IRQ_SET_DATA_NONE`` indicates there is no data field in the command. + The action is performed unconditionally. + * ``VFIO_IRQ_SET_DATA_BOOL`` indicates the data field is an array of boolean + bytes. The action is performed if the corresponding boolean is true. + * ``VFIO_IRQ_SET_DATA_EVENTFD`` indicates an array of event file descriptors + was sent in the message meta-data. These descriptors will be signalled when + the action defined by the action flags occurs. In ``AF_UNIX`` sockets, the + descriptors are sent as ``SCM_RIGHTS`` type ancillary data. + If no file descriptors are provided, this de-assigns the specified + previously configured interrupts. + * ``VFIO_IRQ_SET_ACTION_MASK`` indicates a masking event. It can be used with + ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to mask an interrupt, + or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the guest masks + the interrupt. + * ``VFIO_IRQ_SET_ACTION_UNMASK`` indicates an unmasking event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to unmask an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + guest unmasks the interrupt. + * ``VFIO_IRQ_SET_ACTION_TRIGGER`` indicates a triggering event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to trigger an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + server triggers the interrupt. + +* *index* is the index of IRQ type being setup. +* *start* is the start of the sub-index being set. +* *count* describes the number of sub-indexes being set. As a special case, a + count (and start) of 0, with data flags of ``VFIO_IRQ_SET_DATA_NONE`` disables + all interrupts of the index. +* *data* is an optional field included when the + ``VFIO_IRQ_SET_DATA_BOOL`` flag is present. It contains an array of booleans + that specify whether the action is to be performed on the corresponding + index. It's used when the action is only performed on a subset of the range + specified. + +Not all interrupt types support every combination of data and action flags. +The client must know the capabilities of the device and IRQ index before it +sends a ``VFIO_USER_DEVICE_SET_IRQ`` message. + +In typical operation, a specific IRQ may operate as follows: + +1. The client sends a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_TRIGGER)`` along + with an eventfd. This associates the IRQ with a particular eventfd on the + server side. + +#. The client may send a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_MASK/UNMASK)`` along + with another eventfd. This associates the given eventfd with the + mask/unmask state on the server side. + +#. The server may trigger the IRQ by writing 1 to the eventfd. + +#. The server may mask/unmask an IRQ which will write 1 to the corresponding + mask/unmask eventfd, if there is one. + +5. A client may trigger a device IRQ itself, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_TRIGGER)``. + +6. A client may mask or unmask the IRQ, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_MASK/UNMASK)``. + +Reply +^^^^^ + +There is no payload in the reply. + +.. _Read and Write Operations: + +Note that all of these operations must be supported by the client and/or server, +even if the corresponding memory or device region has been shared as mappable. + +The ``count`` field must not exceed the value of ``max_data_xfer_size`` of the +peer, for both reads and writes. + +``VFIO_USER_REGION_READ`` +------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via ``mmap()`` of the underlying file descriptor. In this case, a client can +read from a device region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. +* *data* is the data that was read from the device region. + +``VFIO_USER_REGION_WRITE`` +-------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via mmap() of the underlying fd. In this case, a client can write to a device +region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DMA_READ`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +read from guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. +* *data* is the data read. + +``VFIO_USER_DMA_WRITE`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +write to guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 4 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DEVICE_RESET`` +-------------------------- + +This command message is sent from the client to the server to reset the device. +Neither the request or reply have a payload. + +``VFIO_USER_REGION_WRITE_MULTI`` +-------------------------------- + +This message can be used to coalesce multiple device write operations +into a single messgage. It is only used as an optimization when the +outgoing message queue is relatively full. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| wr_cnt | 0 | 8 | ++---------+--------+----------+ +| wrs | 8 | variable | ++---------+--------+----------+ + +* *wr_cnt* is the number of device writes coalesced in the message +* *wrs* is an array of device writes defined below + +Single Device Write Format +"""""""""""""""""""""""""" + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | 8 | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. This format can + only describe writes of 8 bytes or less. +* *data* is the data to write. + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| wr_cnt | 0 | 8 | ++---------+--------+----------+ + +* *wr_cnt* is the number of device writes completed. + + +Appendices +========== + +Unused VFIO ``ioctl()`` commands +-------------------------------- + +The following VFIO commands do not have an equivalent vfio-user command: + +* ``VFIO_GET_API_VERSION`` +* ``VFIO_CHECK_EXTENSION`` +* ``VFIO_SET_IOMMU`` +* ``VFIO_GROUP_GET_STATUS`` +* ``VFIO_GROUP_SET_CONTAINER`` +* ``VFIO_GROUP_UNSET_CONTAINER`` +* ``VFIO_GROUP_GET_DEVICE_FD`` +* ``VFIO_IOMMU_GET_INFO`` + +However, once support for live migration for VFIO devices is finalized some +of the above commands may have to be handled by the client in their +corresponding vfio-user form. This will be addressed in a future protocol +version. + +VFIO groups and containers +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The current VFIO implementation includes group and container idioms that +describe how a device relates to the host IOMMU. In the vfio-user +implementation, the IOMMU is implemented in SW by the client, and is not +visible to the server. The simplest idea would be that the client put each +device into its own group and container. + +Backend Program Conventions +--------------------------- + +vfio-user backend program conventions are based on the vhost-user ones. + +* The backend program must not daemonize itself. +* No assumptions must be made as to what access the backend program has on the + system. +* File descriptors 0, 1 and 2 must exist, must have regular + stdin/stdout/stderr semantics, and can be redirected. +* The backend program must honor the SIGTERM signal. +* The backend program must accept the following commands line options: + + * ``--socket-path=PATH``: path to UNIX domain socket, + * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible with + ``--socket-path`` +* The backend program must be accompanied with a JSON file stored under + ``/usr/share/vfio-user``. + +TODO add schema similar to docs/interop/vhost-user.json. From patchwork Wed Feb 19 14:48:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982383 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 3E454C021AA for ; Wed, 19 Feb 2025 14:50:41 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOv-0003Qa-Pf; Wed, 19 Feb 2025 09:50:13 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOu-0003PH-KF; Wed, 19 Feb 2025 09:50:12 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOr-0007UL-4U; Wed, 19 Feb 2025 09:50:12 -0500 Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9LnWq011251; Wed, 19 Feb 2025 06:50:06 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=Quprxgu3LSS9yb4vT50XBs4xUhFHKnzuYSyIGBjsC rM=; b=wp3FyABLlhxoTIy+yjWfE7tzEYcXj5M0Bf352lGQa9eQh+Z6ostx8kzaz OOSJA12RHxNrSUIO3L+o/QsQSkZiy2sNAXXm+gXVQtXwdw8R6eyekTEoMBNsSb6b PhyNcmu49lMMBV+OHih0k2/G3pc9AgPw7rG4cHo7bR8SkoNVfoYfSSZIMMR7oclj ODGFgw0rcIoAYbM5kK7RRT/D51eIHKgJE47uOHE90zK7D+BppeuUW5YdIC9YqQ35 /54wR7ywyIqJHeRFir/siwwS8G3f/2u65jfBNF9GK+JJVNV12eBs+ExwAh25bsHZ bMUjuLPMNq89gDPKc+XjxV+5XivdA== Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2044.outbound.protection.outlook.com [104.47.58.44]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4bdhxxr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:05 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=eEhWptoVp8XSICQDQlWhb9VXoZ2il7EYxNNve13K8epDOiCRMrI+H7magl+YOtdnQNuaAZNrcYhaj756VeI1uICOg7eECAsWpX/D7iwAOnD1/uRuAunFAKNLTaLzukkrIcD0xOVOR+XQzRFknXhCuaOEptCcuemvt3C19zMwZevctktujxLitfOjbxe3MpZV12aXrMvCHaRgFiOTypxIHfMTACD57eo6adTEdriFWQ35faaTIkCPekN6K08BXlm5T7tN4R3XTHG97IT4rWWNyVab8cUvbFnKasp7GIsV6x6sQaeDDXpWid1saduTdHdKehujJ/sbHwMGwZFEfvBGCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Quprxgu3LSS9yb4vT50XBs4xUhFHKnzuYSyIGBjsCrM=; b=OUa5EVwGU0UXFiW5wHq91QnHc+UhJNsfM6wRWLp/jrNjXhV/76NR+NpgHy4CLUEkHSgMQXs3OTvhsWzGCzOevnJCf4ESwSbB2ENtxD8c8IH/x/xpSQUdbxuCcwo+9IbKybQV4Pw+s5JJx75edfiVXAowBGhvGjd75aO18+ULnoeKQkrbrSb4p8iY6H1spUUWkKRmzc6fBw/6EVunuD4mkS6274yj1PL5p6fyOCiF629EIsMakiBeAAivCithyflUHd4F5CjlrzKqFIvlv6b1jVlIucm9havRW2lxiRccgfpXptDxBWRLH0p4JIV2RkV8756+kaqm1a42v6FUfuLXZA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Quprxgu3LSS9yb4vT50XBs4xUhFHKnzuYSyIGBjsCrM=; b=mOknv0u91tKyWq8eaKhVKFD2fedzfagTN8ak3jI9g6rCwB4BQuzFndBUsz2U/xN1OXU9QaXx2jVWBuEtYB5FCiq5PEvIBMPWykjlCB5hfFKmMZbncLuZUoJWboOA6FwkTkqJlA2RkPwolQLV5Npr7wxr90IHn94wpOHs0braKmtOcYxohO3oJk68eAyQf6bN3mnLMvhmmcNXUIzeNjWtoNt+rcJKDxzTugxnVUp7xalf6P9hQGLEoCD7tkYjTghJqH9QZLq6tLmEAc4ZfJ7N5rBXNsGYvqy9p83zOV9VN++NI7jpo2KVbGUxEGPgPQvjjLy7zTmaCNx5puUvjpbluA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:49:51 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:51 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 12/28] vfio-user: add vfio-user class and container Date: Wed, 19 Feb 2025 15:48:42 +0100 Message-Id: <20250219144858.266455-13-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 8991fc39-ff7c-4d1d-1e53-08dd50f4a9f7 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?eUXLudiLS0F/O+hAg4pABQ1HPuxmn3Z?= =?utf-8?q?GVwaSVoJTKSjQPo8xrgMQggBy8rH12MkC1w2igB5X3dSYT3uYsbdL1UTuOfW40LxC?= =?utf-8?q?otB7LLoBfZHpZ2Lp5VABbBSG3t+wGwnJPxEo/ipp4wgh77OrRWNx8GhvLB3M2usru?= =?utf-8?q?r++rWNeivkKA/BSZUe+eFglxiR6IUCF5BviJZDvFWU1zg1eG0hub1CZbOt38FBxyv?= =?utf-8?q?vEWQYnwQtKctd7e25IkEw6mPArt9UC6BBkpVFXjqZg/E7615WD79UbemDf/DTEFKQ?= =?utf-8?q?09dZPLi3B/BSLp/YU9C2D9YgEWBzC05GgZ4KG2geI2C1YqMyrjXWxwz5oBYVC5n+D?= =?utf-8?q?GPZI1NrTV114n1BvP+sWRBFCzkZJ7UZbac9tHf4iD+yvH+oWes+HCpRaFMHB3ytPx?= =?utf-8?q?z1c1jcMsUB6zl2y2Kdz3imnUgtcbv4PaD/XUPBykdgRgmkafcw3xOiFP2oJIQAo9l?= =?utf-8?q?QQNgYQ/C54ragulaEWJOYv9vUmL2BR7B7G7+R3OgJ2cm4lKqP0w04PxFnqwpUJ45R?= =?utf-8?q?p563S6NYVtx54kEVxJIFynvySbhmFaNLa94kiAYRdAhtNYNrC9mIo7kzSLfbRssXh?= =?utf-8?q?Vb+ro2D/LFeRxFyFs7Q1ZaWcKjANp0276aFzO9KdQiJr5k/bKF+86Qh1hXMAxwv/1?= =?utf-8?q?YD01xynvm3VN8lUo/oRUi9xDk4s+KSM7ul3Itqi1B9Qh/tLU+ldyudNErn1co+rHw?= =?utf-8?q?ufBLf4Jia57LXPfTI3v1ggKtPIvIjBxlnW6WLlag5+j/icaaHgkvlRpvOiaTsB3rY?= =?utf-8?q?YqdEZnxAO/mD5rjdCTpXCkk+DnyFawjQgbpOu7vDc31OMrfYJxOaKkiFeAUjZstCU?= =?utf-8?q?BZBDeM4ovKCsugvGK2rAHQzuxWrh1rHHfsItLbKUpdesV37k/gc+VtAjwWA5T5+Rl?= =?utf-8?q?EhcmN2+rNj0EHnoVcrixXg2GLOH56GudmeBE5GwNv9CysgDOd6aoEdNZVKb+dAcge?= =?utf-8?q?VP2G52BOLRdu6cI3gkgSBt99SjH8p2KFC1anQ6U519FmK0p3uW/e3ojyPHRdldpcZ?= =?utf-8?q?Gpd0qCimuw0z10sAxzk5IIQDiwir+HiDFyWxtujfJcTVXbAxjNdSqKZ1fod50yJtP?= =?utf-8?q?4dUD3ibHq1iHjpiF9DNDZFzFT1PrpYDxd0DoWJbYYuUkv+gnBOLvnHM4GM3JCABlS?= =?utf-8?q?HXpQwhFjY6220yiz3O7Z2pmTC3SfSKTDLfab46UPYw16UJQ9MqxsLq7lJVw2xsv/n?= =?utf-8?q?TtcQ01H5/KGUDqAOH4bhP7RAs3xu/pzm/9DUntJZoKPBLVGBjNSTBxLCmEOrMTzaM?= =?utf-8?q?Go6+HBf1aSJkZF7+IPWJYB/9yrL+A8ahHoxByL1TYYcTcRpSIYqHHeIdA8YY7C85a?= =?utf-8?q?pn8p/FYLe+Vk?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?K+CpXd4b/RB6RX/TrWmMcpQRn+TA?= =?utf-8?q?z/heltmV03O1+8PGLVVtW/Jk1rfXGW5cb7w8lYVdzJL4m/a97Fa79g7xuy8G2u8Zj?= =?utf-8?q?Cki9zHriE/pZLb+0pCb0vdw17nfaJ2gOcEW7C5OwtpXUF+zD8ccYuWP9U8YVKyK4s?= =?utf-8?q?9k5/OOZwlpsBjypdwmVVoc2Jy6xZwsunaW1ft0J85EuHzODBnvjhSI3MQrOpbkBtj?= =?utf-8?q?QA5zBBK6DAQdNvzOLV2wIR/jTasXI0SnkqRf6x6mVU/LBol00HDAnnAMsOjB7u52M?= =?utf-8?q?heNggboY07Xv4bx8c6TLP1DhvE9e+oG4eDqMSmLxbBmIsyLIIAzb27xMoWkTSimvV?= =?utf-8?q?i1dKkHuJfwRrZssLGBt4mnophh5p8b2L9N3i2YZV1a2O58aVBYFvJkLMBQyJ6FFGE?= =?utf-8?q?5o+00nnlg4JQw+jRDP0TQoukIEiV/D63coGCNS/3XhBU/ZGagEXXzlDm/ojRoqlYt?= =?utf-8?q?yNPEQUzEyNalps4o4wFtlm/bNmZ+cHBzGRADvzQzUtIvnV7ykDV5Zlgn3yg6ajTXK?= =?utf-8?q?wQgyftVEhWYzYo02bVrh3HBQWXxYbjmipVLzPGI/iSzvyWKQ/+jKkxCBIzI9NNuUY?= =?utf-8?q?ajmFaMbu10nk2S7Br9tMdFGf9GYe11ZgnGTIgqUzZnTmnlA6qHT6PHAZ5mOxVswEg?= =?utf-8?q?14EXG9aqJAtXzpH95UyIHVN169pwJ5ixoRi2obKBSwwAelkdzYhtb0G2smP4EDcvy?= =?utf-8?q?MKH/8Z9TDG2z4bzNN1y59CqGh6aDokS8JQ584fpDXlpL0NwJHYH/EtYQOOuw0gL0s?= =?utf-8?q?/DdqjH+dj+QRXXT5q9ggh1g3KuEZq/SVYCGusChF09qiHkwQQv3nU1/ZWnWSsOmWA?= =?utf-8?q?feiVxSsKzsMoXual8hlzlQgVLsV09loDM9OSv1yDaYMScHhQNpICYXD3zjHRYwVIw?= =?utf-8?q?ogUo0+K+f6Mk9qMHWqCuIL4tHnhH5Nc5Nfs2jCUBNBMb1hgeFyFx0uIinU68Sdb+m?= =?utf-8?q?r9kcyQRbyL/as3a0DtDxWM1r45vcJ5yXMjXF4++PwfU9+JI9M1Ycgh40/hcsrTOAr?= =?utf-8?q?D98z3hkwUppvyXMTky8p0lU9nEvV9OQrGTxdBeqHlKodlkn5Zx34D7S+MrLxQLKLQ?= =?utf-8?q?Nozxa2PZZH27f4sKXS4REctg86MxW2g6ASpnnLSbJWUVE7W1nw5fRV9Rk4u4hAtZP?= =?utf-8?q?lMGEdbv5JaHLNrdc9TnfOBCf6Sc5iCNz1OVsrG0GUn/4aDEmFTeVl3EsglEML4Ar/?= =?utf-8?q?RdY9HcmuzgYFHioBNE5EGm2AFP3P9LPCcJG7t5xn2IdGfyD2iNAC2TQSbnk23lrt4?= =?utf-8?q?OcoLVdD9108BZpCw1wfuiaWNFSWwV8DXa5jc6fdQOsxy++PjQx2lWd4v+iLOxTGFB?= =?utf-8?q?/PC6x61jMDX/L+PWjRn4inzrH/dx6Ymc0Z755S2wyuKZP56gykD+nDmi3IPqvONWc?= =?utf-8?q?nWfJRbFMic8C8cTN/XchNE7DXw74RnsGIz64uDMSsOWLyY271nFr66vwNobWzDNxt?= =?utf-8?q?4Iv6rMOUGcz3LJupDqPRA0yMxRdXy1d3BMa65O5vRZsVkAgWSJCQlRPDHkzkL+ZbU?= =?utf-8?q?h4j7c1c+KEYE?= X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8991fc39-ff7c-4d1d-1e53-08dd50f4a9f7 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:51.2679 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: OQIB9mKY+3hemLTvKmhTbB1yL3bGBJw3dKpiQoARw305r/U9AduDYZy/J0JTGQSt83cIpOUwTtcbrnqFGWR9nw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-ORIG-GUID: qW7vURIXtAZYPl_5bC-JZ-zBziikqm0C X-Proofpoint-GUID: qW7vURIXtAZYPl_5bC-JZ-zBziikqm0C X-Authority-Analysis: v=2.4 cv=Fo7//3rq c=1 sm=1 tr=0 ts=67b5ef9d cx=c_pps a=di3315gfm3qlniCp1Rh91A==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=IkcTkHD0fZMA:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=_S1ogib_BAg1t8ieMRMA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Introduce basic plumbing for vfio-user behind a new --enable-vfio-user-client option. We introduce VFIOUserContainer in hw/vfio-user/container.c, which is a container type for the "IOMMU" type "vfio-iommu-user", and share some common container code from hw/vfio/container.c. Add hw/vfio-user/pci.c for instantiating VFIOUserPCIDevice objects, sharing some common code from hw/vfio/pci.c. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- MAINTAINERS | 2 + hw/meson.build | 1 + hw/vfio-user/container.c | 222 ++++++++++++++++++++++++++ hw/vfio-user/container.h | 23 +++ hw/vfio-user/meson.build | 9 ++ hw/vfio-user/pci.c | 154 ++++++++++++++++++ hw/vfio/container.c | 2 +- hw/vfio/pci.c | 12 +- hw/vfio/pci.h | 7 + include/hw/vfio/vfio-common.h | 3 + include/hw/vfio/vfio-container-base.h | 1 + meson_options.txt | 2 + scripts/meson-buildoptions.sh | 4 + 13 files changed, 435 insertions(+), 7 deletions(-) create mode 100644 hw/vfio-user/container.c create mode 100644 hw/vfio-user/container.h create mode 100644 hw/vfio-user/meson.build create mode 100644 hw/vfio-user/pci.c diff --git a/MAINTAINERS b/MAINTAINERS index 3e7e6743cc..c403742c27 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4158,6 +4158,8 @@ M: John Levon M: Thanos Makatos S: Supported F: docs/devel/vfio-user.rst +F: hw/vfio-user/* +F: include/hw/vfio-user/* F: subprojects/libvfio-user EBPF: diff --git a/hw/meson.build b/hw/meson.build index b827c82c5d..91e8d2bdc0 100644 --- a/hw/meson.build +++ b/hw/meson.build @@ -38,6 +38,7 @@ subdir('tpm') subdir('ufs') subdir('usb') subdir('vfio') +subdir('vfio-user') subdir('virtio') subdir('watchdog') subdir('xen') diff --git a/hw/vfio-user/container.c b/hw/vfio-user/container.c new file mode 100644 index 0000000000..7b1c202517 --- /dev/null +++ b/hw/vfio-user/container.c @@ -0,0 +1,222 @@ +/* + * Container for vfio-user IOMMU type: rather than communicating with the kernel + * vfio driver, we communicate over a socket to a server using the vfio-user + * protocol. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#include +#include +#include "qemu/osdep.h" + +#include "exec/address-spaces.h" +#include "exec/memory.h" +#include "exec/ram_addr.h" +#include "hw/hw.h" +#include "hw/vfio/pci.h" +#include "hw/vfio-user/container.h" +#include "qemu/error-report.h" +#include "qemu/range.h" +#include "qapi/error.h" +#include "trace.h" + +static int vfio_user_dma_unmap(const VFIOContainerBase *bcontainer, + hwaddr iova, ram_addr_t size, + IOMMUTLBEntry *iotlb, int flags) +{ + return -ENOTSUP; +} + +static int vfio_user_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, + ram_addr_t size, void *vaddr, bool readonly, + MemoryRegion *mrp) +{ + return -ENOTSUP; +} + +static int +vfio_user_set_dirty_page_tracking(const VFIOContainerBase *bcontainer, + bool start, Error **errp) +{ + error_setg_errno(errp, ENOTSUP, "Not supported"); + return -ENOTSUP; +} + +static int vfio_user_query_dirty_bitmap(const VFIOContainerBase *bcontainer, + VFIOBitmap *vbmap, hwaddr iova, + hwaddr size, Error **errp) +{ + error_setg_errno(errp, ENOTSUP, "Not supported"); + return -ENOTSUP; +} + +static bool vfio_user_setup(VFIOContainerBase *bcontainer, Error **errp) +{ + error_setg_errno(errp, ENOTSUP, "Not supported"); + return -ENOTSUP; +} + +static VFIOUserContainer *vfio_create_user_container(Error **errp) +{ + VFIOUserContainer *container; + + container = VFIO_IOMMU_USER(object_new(TYPE_VFIO_IOMMU_USER)); + return container; +} + +/* + * Try to mirror vfio_connect_container() as much as possible. + */ +static VFIOUserContainer * +vfio_connect_user_container(AddressSpace *as, Error **errp) +{ + VFIOContainerBase *bcontainer; + VFIOUserContainer *container; + VFIOAddressSpace *space; + VFIOIOMMUClass *vioc; + + space = vfio_get_address_space(as); + + container = vfio_create_user_container(errp); + if (!container) { + goto put_space_exit; + } + + bcontainer = &container->bcontainer; + + if (!vfio_cpr_register_container(bcontainer, errp)) { + goto free_container_exit; + } + + vioc = VFIO_IOMMU_GET_CLASS(bcontainer); + assert(vioc->setup); + + if (!vioc->setup(bcontainer, errp)) { + goto unregister_container_exit; + } + + vfio_address_space_insert(space, bcontainer); + + bcontainer->listener = vfio_memory_listener; + memory_listener_register(&bcontainer->listener, bcontainer->space->as); + + if (bcontainer->error) { + errno = EINVAL; + error_propagate_prepend(errp, bcontainer->error, + "memory listener initialization failed: "); + goto listener_release_exit; + } + + bcontainer->initialized = true; + + return container; + +listener_release_exit: + memory_listener_unregister(&bcontainer->listener); + if (vioc->release) { + vioc->release(bcontainer); + } + +unregister_container_exit: + vfio_cpr_unregister_container(bcontainer); + +free_container_exit: + object_unref(container); + +put_space_exit: + vfio_put_address_space(space); + + return NULL; +} + +static void vfio_disconnect_user_container(VFIOUserContainer *container) +{ + VFIOContainerBase *bcontainer = &container->bcontainer; + VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); + + memory_listener_unregister(&bcontainer->listener); + if (vioc->release) { + vioc->release(bcontainer); + } + + VFIOAddressSpace *space = bcontainer->space; + + vfio_cpr_unregister_container(bcontainer); + object_unref(container); + + vfio_put_address_space(space); +} + +static bool vfio_user_get_device(VFIOUserContainer *container, + VFIODevice *vbasedev, Error **errp) +{ + struct vfio_device_info info = { 0 }; + + vbasedev->fd = -1; + + vfio_prepare_device(vbasedev, &container->bcontainer, NULL, &info); + + return true; +} + +/* + * vfio_user_attach_device: attach a device to a new container. + */ +static bool vfio_user_attach_device(const char *name, VFIODevice *vbasedev, + AddressSpace *as, Error **errp) +{ + VFIOUserContainer *container; + + container = vfio_connect_user_container(as, errp); + if (container == NULL) { + error_prepend(errp, "failed to connect proxy"); + return false; + } + + return vfio_user_get_device(container, vbasedev, errp); +} + +static void vfio_user_detach_device(VFIODevice *vbasedev) +{ + VFIOUserContainer *container = container_of(vbasedev->bcontainer, + VFIOUserContainer, bcontainer); + + QLIST_REMOVE(vbasedev, global_next); + QLIST_REMOVE(vbasedev, container_next); + vbasedev->bcontainer = NULL; + vfio_put_base_device(vbasedev); + vfio_disconnect_user_container(container); +} + +static int vfio_user_pci_hot_reset(VFIODevice *vbasedev, bool single) +{ + /* ->needs_reset is always false for vfio-user. */ + return 0; +} + +static void vfio_iommu_user_class_init(ObjectClass *klass, void *data) +{ + VFIOIOMMUClass *vioc = VFIO_IOMMU_CLASS(klass); + + vioc->setup = vfio_user_setup; + vioc->dma_map = vfio_user_dma_map; + vioc->dma_unmap = vfio_user_dma_unmap; + vioc->attach_device = vfio_user_attach_device; + vioc->detach_device = vfio_user_detach_device; + vioc->set_dirty_page_tracking = vfio_user_set_dirty_page_tracking; + vioc->query_dirty_bitmap = vfio_user_query_dirty_bitmap; + vioc->pci_hot_reset = vfio_user_pci_hot_reset; +}; + +static const TypeInfo types[] = { + { + .name = TYPE_VFIO_IOMMU_USER, + .parent = TYPE_VFIO_IOMMU, + .instance_size = sizeof(VFIOUserContainer), + .class_init = vfio_iommu_user_class_init, + }, +}; + +DEFINE_TYPES(types) diff --git a/hw/vfio-user/container.h b/hw/vfio-user/container.h new file mode 100644 index 0000000000..24ce13bc2d --- /dev/null +++ b/hw/vfio-user/container.h @@ -0,0 +1,23 @@ +/* + * vfio-user specific definitions. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + */ + +#ifndef HW_VFIO_USER_CONTAINER_H +#define HW_VFIO_USER_CONTAINER_H + +#include +#include + +#include "hw/vfio/vfio-common.h" + +/* MMU container sub-class for vfio-user. */ +typedef struct VFIOUserContainer { + VFIOContainerBase bcontainer; +} VFIOUserContainer; + +OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserContainer, VFIO_IOMMU_USER); + +#endif /* HW_VFIO_USER_CONTAINER_H */ diff --git a/hw/vfio-user/meson.build b/hw/vfio-user/meson.build new file mode 100644 index 0000000000..f1fee70c85 --- /dev/null +++ b/hw/vfio-user/meson.build @@ -0,0 +1,9 @@ +vfio_user_ss = ss.source_set() +vfio_user_ss.add(files( + 'container.c', + 'pci.c', +)) + +if get_option('vfio_user_client').enabled() + specific_ss.add_all(vfio_user_ss) +endif diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c new file mode 100644 index 0000000000..6c85c60179 --- /dev/null +++ b/hw/vfio-user/pci.c @@ -0,0 +1,154 @@ +/* + * vfio PCI device over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include +#include +#include "qemu/osdep.h" + +#include "hw/hw.h" +#include "hw/pci/msi.h" +#include "hw/pci/msix.h" +#include "hw/pci/pci_bridge.h" +#include "hw/qdev-properties.h" +#include "hw/qdev-properties-system.h" +#include "hw/vfio/pci.h" +#include "qapi/error.h" +#include "qobject/qdict.h" +#include "qemu/error-report.h" +#include "qemu/module.h" +#include "qemu/range.h" +#include "qemu/units.h" +#include "system/kvm.h" +#include "trace.h" + +#define TYPE_VFIO_USER_PCI "vfio-user-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) + +struct VFIOUserPCIDevice { + VFIOPCIDevice device; + char *sock_name; +}; + +/* + * Emulated devices don't use host hot reset + */ +static void vfio_user_compute_needs_reset(VFIODevice *vbasedev) +{ + vbasedev->needs_reset = false; +} + +static VFIODeviceOps vfio_user_pci_ops = { + .vfio_compute_needs_reset = vfio_user_compute_needs_reset, + .vfio_eoi = vfio_intx_eoi, + .vfio_get_object = vfio_pci_get_object, + .vfio_save_config = vfio_pci_save_config, + .vfio_load_config = vfio_pci_load_config, +}; + +static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) +{ + ERRP_GUARD(); + VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; + AddressSpace *as; + + /* + * TODO: make option parser understand SocketAddress + * and use that instead of having scalar options + * for each socket type. + */ + if (!udev->sock_name) { + error_setg(errp, "No socket specified"); + error_append_hint(errp, "Use -device vfio-user-pci,socket=\n"); + return; + } + + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); + vbasedev->ops = &vfio_user_pci_ops; + vbasedev->type = VFIO_DEVICE_TYPE_PCI; + vbasedev->dev = DEVICE(vdev); + + /* + * vfio-user devices are effectively mdevs (don't use a host iommu). + */ + vbasedev->mdev = true; + + as = pci_device_iommu_address_space(pdev); + if (!vfio_attach_device_by_iommu_type(TYPE_VFIO_IOMMU_USER, + vbasedev->name, vbasedev, + as, errp)) { + error_prepend(errp, VFIO_MSG_PREFIX, vbasedev->name); + return; + } +} + +static void vfio_user_instance_init(Object *obj) +{ + PCIDevice *pci_dev = PCI_DEVICE(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + VFIODevice *vbasedev = &vdev->vbasedev; + + device_add_bootindex_property(obj, &vdev->bootindex, + "bootindex", NULL, + &pci_dev->qdev); + vdev->host.domain = ~0U; + vdev->host.bus = ~0U; + vdev->host.slot = ~0U; + vdev->host.function = ~0U; + + vfio_device_init(vbasedev, VFIO_DEVICE_TYPE_PCI, &vfio_user_pci_ops, + &vfio_dev_io_ioctl, DEVICE(vdev), false); + + vdev->nv_gpudirect_clique = 0xFF; + + /* + * QEMU_PCI_CAP_EXPRESS initialization does not depend on QEMU command + * line, therefore, no need to wait to realize like other devices. + */ + pci_dev->cap_present |= QEMU_PCI_CAP_EXPRESS; +} + +static void vfio_user_instance_finalize(Object *obj) +{ + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + + vfio_pci_put_device(vdev); +} + +static const Property vfio_user_pci_dev_properties[] = { + DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), +}; + +static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + device_class_set_props(dc, vfio_user_pci_dev_properties); + dc->desc = "VFIO over socket PCI device assignment"; + pdc->realize = vfio_user_pci_realize; +} + +static const TypeInfo vfio_user_pci_dev_info = { + .name = TYPE_VFIO_USER_PCI, + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOUserPCIDevice), + .class_init = vfio_user_pci_dev_class_init, + .instance_init = vfio_user_instance_init, + .instance_finalize = vfio_user_instance_finalize, +}; + +static void register_vfio_user_dev_type(void) +{ + type_register_static(&vfio_user_pci_dev_info); +} + + type_init(register_vfio_user_dev_type) diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 36cd245c92..0e1af34ce4 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -884,7 +884,7 @@ static bool vfio_get_device(VFIOGroup *group, const char *name, return true; } -static void vfio_put_base_device(VFIODevice *vbasedev) +void vfio_put_base_device(VFIODevice *vbasedev) { if (vbasedev->regions != NULL) { int i; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a9cc9366fb..a7084a7690 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -109,7 +109,7 @@ static void vfio_intx_interrupt(void *opaque) } } -static void vfio_intx_eoi(VFIODevice *vbasedev) +void vfio_intx_eoi(VFIODevice *vbasedev) { VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); @@ -2587,7 +2587,7 @@ static void vfio_pci_compute_needs_reset(VFIODevice *vbasedev) } } -static Object *vfio_pci_get_object(VFIODevice *vbasedev) +Object *vfio_pci_get_object(VFIODevice *vbasedev) { VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); @@ -2643,7 +2643,7 @@ static const VMStateDescription vmstate_vfio_pci_config = { } }; -static int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp) +int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp) { VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); @@ -2651,7 +2651,7 @@ static int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp) errp); } -static int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) +int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f) { VFIOPCIDevice *vdev = container_of(vbasedev, VFIOPCIDevice, vbasedev); PCIDevice *pdev = &vdev->pdev; @@ -2847,7 +2847,7 @@ static bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) return true; } -static void vfio_pci_put_device(VFIOPCIDevice *vdev) +void vfio_pci_put_device(VFIOPCIDevice *vdev) { vfio_detach_device(&vdev->vbasedev); @@ -3394,7 +3394,7 @@ post_reset: vfio_pci_post_reset(vdev); } -static void vfio_instance_init(Object *obj) +void vfio_instance_init(Object *obj) { PCIDevice *pci_dev = PCI_DEVICE(obj); VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 8e79740ddb..c0f030f4db 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -213,6 +213,13 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len); void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len); +void vfio_intx_eoi(VFIODevice *vbasedev); +Object *vfio_pci_get_object(VFIODevice *vbasedev); +int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp); +int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f); +void vfio_pci_put_device(VFIOPCIDevice *vdev); +void vfio_instance_init(Object *obj); + uint64_t vfio_vga_read(void *opaque, hwaddr addr, unsigned size); void vfio_vga_write(void *opaque, hwaddr addr, uint64_t data, unsigned size); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 3512556590..ee8e7f7c0d 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -80,6 +80,7 @@ typedef struct VFIOMigration { struct VFIOGroup; +/* MMU container sub-class for legacy vfio implementation. */ typedef struct VFIOContainer { VFIOContainerBase bcontainer; int fd; /* /dev/vfio/vfio, empowered by the attached groups */ @@ -106,6 +107,7 @@ typedef struct VFIOIOASHwpt { QLIST_ENTRY(VFIOIOASHwpt) next; } VFIOIOASHwpt; +/* MMU container sub-class for vfio iommufd implementation. */ typedef struct VFIOIOMMUFDContainer { VFIOContainerBase bcontainer; IOMMUFDBackend *be; @@ -285,6 +287,7 @@ bool vfio_attach_device_by_iommu_type(const char *iommu_type, char *name, Error **errp); void vfio_detach_device(VFIODevice *vbasedev); VFIODevice *vfio_get_vfio_device(Object *obj); +void vfio_put_base_device(VFIODevice *vbasedev); int vfio_kvm_device_add_fd(int fd, Error **errp); int vfio_kvm_device_del_fd(int fd, Error **errp); diff --git a/include/hw/vfio/vfio-container-base.h b/include/hw/vfio/vfio-container-base.h index 24e48e3a07..1ce93c5b9b 100644 --- a/include/hw/vfio/vfio-container-base.h +++ b/include/hw/vfio/vfio-container-base.h @@ -100,6 +100,7 @@ vfio_container_get_page_size_mask(const VFIOContainerBase *bcontainer) #define TYPE_VFIO_IOMMU_LEGACY TYPE_VFIO_IOMMU "-legacy" #define TYPE_VFIO_IOMMU_SPAPR TYPE_VFIO_IOMMU "-spapr" #define TYPE_VFIO_IOMMU_IOMMUFD TYPE_VFIO_IOMMU "-iommufd" +#define TYPE_VFIO_IOMMU_USER TYPE_VFIO_IOMMU "-user" OBJECT_DECLARE_TYPE(VFIOContainerBase, VFIOIOMMUClass, VFIO_IOMMU) diff --git a/meson_options.txt b/meson_options.txt index 5eeaf3eee5..ba9bc07fcf 100644 --- a/meson_options.txt +++ b/meson_options.txt @@ -109,6 +109,8 @@ option('multiprocess', type: 'feature', value: 'auto', description: 'Out of process device emulation support') option('relocatable', type : 'boolean', value : true, description: 'toggle relocatable install') +option('vfio_user_client', type: 'feature', value: 'disabled', + description: 'vfio-user client support') option('vfio_user_server', type: 'feature', value: 'disabled', description: 'vfio-user server support') option('dbus_display', type: 'feature', value: 'auto', diff --git a/scripts/meson-buildoptions.sh b/scripts/meson-buildoptions.sh index a8066aab03..6ee381df8c 100644 --- a/scripts/meson-buildoptions.sh +++ b/scripts/meson-buildoptions.sh @@ -201,6 +201,8 @@ meson_options_help() { printf "%s\n" ' vdi vdi image format support' printf "%s\n" ' vduse-blk-export' printf "%s\n" ' VDUSE block export support' + printf "%s\n" ' vfio-user-client' + printf "%s\n" ' vfio-user client support' printf "%s\n" ' vfio-user-server' printf "%s\n" ' vfio-user server support' printf "%s\n" ' vhdx vhdx image format support' @@ -529,6 +531,8 @@ _meson_option_parse() { --disable-vdi) printf "%s" -Dvdi=disabled ;; --enable-vduse-blk-export) printf "%s" -Dvduse_blk_export=enabled ;; --disable-vduse-blk-export) printf "%s" -Dvduse_blk_export=disabled ;; + --enable-vfio-user-client) printf "%s" -Dvfio_user_client=enabled ;; + --disable-vfio-user-client) printf "%s" -Dvfio_user_client=disabled ;; --enable-vfio-user-server) printf "%s" -Dvfio_user_server=enabled ;; --disable-vfio-user-server) printf "%s" -Dvfio_user_server=disabled ;; --enable-vhdx) printf "%s" -Dvhdx=enabled ;; From patchwork Wed Feb 19 14:48:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982418 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E18EAC021B1 for ; Wed, 19 Feb 2025 14:55:48 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOx-0003Sh-GE; Wed, 19 Feb 2025 09:50:15 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOu-0003PS-VG; Wed, 19 Feb 2025 09:50:12 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOs-0007V3-2G; Wed, 19 Feb 2025 09:50:12 -0500 Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9LnWr011251; Wed, 19 Feb 2025 06:50:07 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=CbYCxq57zE19zKX5mUO4iVNHgd4VV/+dCMgQHeUY/ Js=; b=vJpQ5MxpHNRXB5yRinqVGlSGdtEN6urn7dYMklgF4x/ub/OThW87osTZ/ CSbds7aAaT3RpcXSQ+Ihrrxpym0gv8/0pWYr5RU3p+PSSeo2YNEAzO6uz+eksX2O 7QQPxP+Zn418UTPYg+ovBGLLyaSBlGjO0bwLqTO3yvTyn/nFxfnKg0I4/El4qNiB 0/nydOc9x8KBl/oSofGQhVFXaCe2wmTreG15vM0deax8NQBGwHaTpUXe88ULt2/6 v7s6eYfVn7cgu5hyK2MtmUXlRiDEUPWxWdw/7XDq63wI9HSOaMM3Lb1r2q4S9vVa SOzSmfjA/ZKKM2NdB4btl09R2VZnQ== Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2044.outbound.protection.outlook.com [104.47.58.44]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4bdhxxr-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:06 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=FrmeCOoEU0qiDdomy9mluVjMDjlIbFcBMjLeQBNlimRVifGazUIzI6tatzfyC4e+C1rc2us7iBVca5/EZnH+NGUUqxl+m0buNGw3ayo8K6dGkHhX9g+k4/HWOSYxDWfUFmucwQgFfCtVk7axOdnnnE3qCelynQqXLDY6A8Nr3Fzhd4bAGCWXYGBXhlCJ95fVeJz0qElAG1OAiSsCd9oX1wRaCwhrONPQG3FPtV5ARH2EpBNu6wWRI3xcmZKNlItlGPCn6MoymM80KcHHeFnywv07Yfvfk7gzGj3TIlGHKYA+ic8g4exvkshKW2bpn29KhBTSxYD1XChyjNS7oA3Ihw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=CbYCxq57zE19zKX5mUO4iVNHgd4VV/+dCMgQHeUY/Js=; b=EDee0+WGUDOoutrjxQ/0lDniQPzwEnKmGl3Nfz8Ptb71ZLBH+W65HDtXXEhev2djkhb9MfHv4JwWTgLzF6M1quY+yuV3mxIXd4crjXF09i7pSXaAjwsRC9zYd8w6NEbyFK8NlkwxM71rdTruSZWSz8NaiwXbki3/zGzqw9X2EvRqqrXGdAl/PhihpOohy899H0bKz5UuSbCj7hQJjXvdMQ2q5g2p7W6uef5j1G8NPpzbDHr3pNUE6vxoORnJ5RfZKaGFvwrioM3cDAYPTJparzIZuDnKd9h/iCx3K7+f6SXwEXkCOa72fYzvignTFUxJqaXM6MprrP/y4Ke6Zn4Bfg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=CbYCxq57zE19zKX5mUO4iVNHgd4VV/+dCMgQHeUY/Js=; b=cFDMuwhFeKzMVW5pRdF3EiPwrfW4OUJoLZmoH/UW7Bi6XO6K2t9CLKPxP76GGzAphvKnHz/CrHJEJJoEnzIKqtniezmv+QsRZ8ftNoX9f4yj5qCz99hzMTjSlh/eU9U1KE8GvfuGwjFZ6fd20raPzd/a/n6hoE4dD7qeo0qkOWXcFIHC+BahvXssZxcyaaObjkv4llgSbGKOd5yAKbZEJh3UPcYSpL5SdQy3tI39bkP/PfBfGH1le7N1TyktqfWjTlKaureesTQFCLmLID3cAy6uRYsJM35RmcG3wF8b/htCFW9ZFj4F4OasvXldWIl1obAC+SHszEM9M0sMQBK9EQ== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:49:54 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:54 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 13/28] vfio-user: connect vfio proxy to remote server Date: Wed, 19 Feb 2025 15:48:43 +0100 Message-Id: <20250219144858.266455-14-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 85357136-f640-4288-7f6e-08dd50f4ac06 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?6UHVrIQViVOCFj8x3COMERzugfd0ZwV?= =?utf-8?q?iWhyd3JA/gy1kQcL2PfeER/q2zXfi0+V5+LWxKs7AaYOcMzOdZf95sSlCnQgGlm6i?= =?utf-8?q?a9pBYHhdIFeV176t8scpynEnWijDzAzgNNnV/djGTUwf3WmbqA/2BiXBUc6yQwJyM?= =?utf-8?q?S0tO4lmYTY7zk4eTlf83EkSF+DthEaWDTNlMbzA2/JfBFzYsJUHfyl17Z/j9eJ32/?= =?utf-8?q?dsg+Ry32N5c2pFkLmpPlD/rpw3FR1uhmk/fTK8GbLn9Xt4r5/5MHmohYfcN1W1Ak4?= =?utf-8?q?OqTaTUQ+SMgar4O2nvkL8bHAPxR4ZaHedB3ukoLMFIHMr1GFH4wkdYuC3d0+m8esC?= =?utf-8?q?nIYfn0tyG3IukFQz1vIb4epJYbqR85YoiMskgh1kxvaYV2emGWQC2v2dYztIYu+zh?= =?utf-8?q?hboW9h8n6rVl9596IKegk3jfcqAzHocpMGtIxZI5aio3reaztKmwPi9fzOP+kLOcM?= =?utf-8?q?+Bm8R4TVwii6dJ9H0CIONWhlIZlyZVVFxjJ6G6o2Q2sylFMcNGAYQr6PuHnBJCmXP?= =?utf-8?q?u6hqsDHcwosGdm8Xv2oeV7RYnfkWa7sv50QOFnp0Z0I1/nJuBbhG/fOLoEYv6+ZZN?= =?utf-8?q?Y4vL1ZfN3B9remwTNYEAeKGgOo+hvtIn767M9nFacZL4mftj/eMZAEbAguOg20Qp9?= =?utf-8?q?OjN/sffPex6A9dcKVAsPg8LS/BTtKIEfHULJZMtqgsD7K8VBfTvWMfxmm//90eK8y?= =?utf-8?q?UaLrIhnNET372XcOi9Z6KaAEyDr5FLyPhcc2zjyNgET1L9q++pvvGDddzdNFf6BE5?= =?utf-8?q?QvZq4cUel8t5mJZE7UR8qTwSP126yuLvhp4cWeTTAu02tfo2Tet5d/YD7lFQNYl5Q?= =?utf-8?q?meVj8dN20p+10Upm7jC0RM8cXQQOgVWjCT0Aitpn492efXha286Vi9aN6DJNnoHP7?= =?utf-8?q?Qxv1BXk0FUY2j/XzpvOSBwUJTHH+/o+SmBo08XR51pF0QwhCPTYRo/H5ThxSc3Qv0?= =?utf-8?q?NSySjWqSxyC4A0P3TGYVYUC/lvr9+3+nItCW5vy9d5xMtD0cEQbVchA6/xTmta/aO?= =?utf-8?q?usbe/9QgLV4gUCvCoMO6LKwIjyYX92jiNEehaUt7jajJI7Yy8AVcjyK0SBK2HDHYg?= =?utf-8?q?XTWrSDCgXFgc62mEYDRlXr08tCoUGjF+i4sAMJrfIs1phyDlwYwi/9XkT87iLkMbC?= =?utf-8?q?LIe0CdnwsogK4q+Gyqd2WyfcSN58nYgIx86tGE1j2uIG8+s9znaK4XGqGe3x9X80y?= =?utf-8?q?gbnl97+TQzJPuJnuG+m0c2CvuSPTQBYLxl0+yQv3DUNVF3JL0UlcNMy3e+Hu4ve+2?= =?utf-8?q?htJ0ysEpTtTijc7EoUApFOmOpPwB6Ci0KsW4kett6XU8+pCcfTkjs3Xv3QZVPaUOX?= =?utf-8?q?8qw5IEaN+zgu?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?DDkNPdEfPnHi5Z8fbIKWxm7wHWal?= =?utf-8?q?V/jWt0FtuXSA+hkus1hH13UNAlJLlke/KyY0q2UT5xn57U8NGDkS3M0qbUvTUCQun?= =?utf-8?q?YvBTPRs5FRaDWhFAwEq6GZwQyEc6GVwcMngQi61ywfAeNX7BdX5nWSs+/wgiaBaGH?= =?utf-8?q?DaoMcG6DSb1pmGa+t/LcCLACzhdpPyHP5ZzQLl6bYpPGKlDeO1EY+VtFiZ57fLmMz?= =?utf-8?q?VKQ99Ta0sqXZfecY81TEn29NTnZv3twHZoXhGRD0JwrNnxZcSSG7CVnnFLFQea0dR?= =?utf-8?q?wKzmbfqaYFz2woEyNJ2xBb2K003s2HuomFUiS8GW/Fz2e3GjjvoLVe9mBebmNJ0zz?= =?utf-8?q?2GAVG2pbhRDSvpe3JV4vmOT4OFchea/9pDga0PxInRU2dGv15NDSjdybZzq53KtUc?= =?utf-8?q?R4VupXtVmtOWqN+x9RS90Jcmus9R3p1O85ZhiSMpMeNpj3xu9dCSgyLWbg7TQFGyt?= =?utf-8?q?3pLmuyg6uVLj4oa2MExkWGgU6BiRCRxE/nU5rCdQmQEZ+DJowrVRnrdel2V1vOp65?= =?utf-8?q?cTsQ951TADfUNBuOYWR37iIFi2DioWuuPZGkSDe3N/RiJNA6BoRi6bAglrmJWIil5?= =?utf-8?q?Qk4k7VxtHkZCpGsGEi3W1k2Ew9nUdzGFbSB7cvO62ERhS1gwjwAd71VcvQSf8iBt5?= =?utf-8?q?wHA7nbK4a7EAH+pu9EzBHeXq9XYPixBQQe2jQIzfEcPZOk7QUni6yvLDxms/U0l1H?= =?utf-8?q?WvfbAE63AMcZBoVZ0IqLEkax+OxhxqavDkILolYYHpqcfKzAeq2pVA709YEiKFD7f?= =?utf-8?q?P2bDuNptI8NiI4vJuOs0d6cx5IdTDZ/8YQCzQH9GwBdUyyIuEJ7D5NBzb/015cM4s?= =?utf-8?q?NOJ5TCVvYzpmEVLNL2u0DGVK6x97ZyreXYnOxibYDDt7p3qY2U8Z37nqZ/hVZ5K8W?= =?utf-8?q?DyNLZuXWcLERTBJhjKS8+L+Sh5imhvkIJpxK/cthQR3XfR5SSnpm5wnIhVa5M2BaV?= =?utf-8?q?i7gYsGceVD9RuU9NR/gTLmMaffuIrZQw/y8ofXksJJFJwOuEFXlu/NqCaYoqnLZ/w?= =?utf-8?q?wp40kXJ+d/yPxiYyRBD4cJb+FUogzLIl/Ct/d57NTe8sGFMS1u3s29CDZ9v2rinMc?= =?utf-8?q?6VbNFhEcp6DILbJf33+x6W73jYHV7ZbEZ4K/8B1MwbcNzRh14tTyKdnmfZay7EFD5?= =?utf-8?q?tPfLfKie2t4kGjJFV56NHwtZHidh4Ko0Uv8ZmGpekR9LKIvgweD/TQ4JD3itJCMPH?= =?utf-8?q?y3ryBGpX5pY1aY8lxFfdLotyspa5v9fn+nQLWWZ48tf8WdkSmAqLZCxxmL3Fuo3KY?= =?utf-8?q?Zr2yQI6COHvZ8ivwHJ8KReMaqPsfuDwFgcIwSkNjo3dNU/33t1Y7WVWclJtfYf3/o?= =?utf-8?q?NCCG1S4UVxX5uxZyJrlbBvCL0Vh5npFzC6AeDJkkqciB031aEdh6Ua6l7fASwdEbd?= =?utf-8?q?z/9Y5skd4DG0VA1MkObdTDARAghPf0yFMghWVBNeVMiMfW4KIIbYKTyOln2oTjJrk?= =?utf-8?q?WH6tGslTCueeU124MWcvhKB4f9zuK7sCS+/W4r1XYVbU/z0MwXZhDumT+TjC3aPBb?= =?utf-8?q?Z2oSYjSfMSaU?= X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 85357136-f640-4288-7f6e-08dd50f4ac06 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:54.6659 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: l7HVAsugf9UkMmXKgC2JYcIkBKAyy5384pVMWqtI8nvH1u7pbz5WhM0TLQnQlLczfwQqgnshr7rNX6hPXPmHTQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-ORIG-GUID: Eyxtzb7HSPh2SmtgFrvH8s-oR1ALwBCL X-Proofpoint-GUID: Eyxtzb7HSPh2SmtgFrvH8s-oR1ALwBCL X-Authority-Analysis: v=2.4 cv=Fo7//3rq c=1 sm=1 tr=0 ts=67b5ef9e cx=c_pps a=di3315gfm3qlniCp1Rh91A==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=IkcTkHD0fZMA:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=vhHEnE-nI2wpHcbFAcgA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Introduce the vfio-user "proxy": this is the client code responsible for sending and receiving vfio-user messages across the control socket. The new files hw/vfio-user/common.[ch] contain some basic plumbing for managing the proxy; initialize the proxy during realization of the VFIOUserPCIDevice instance. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 171 ++++++++++++++++++++++++++++++++++ hw/vfio-user/common.h | 78 ++++++++++++++++ hw/vfio-user/meson.build | 1 + hw/vfio-user/pci.c | 18 ++++ include/hw/vfio/vfio-common.h | 2 + 5 files changed, 270 insertions(+) create mode 100644 hw/vfio-user/common.c create mode 100644 hw/vfio-user/common.h diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c new file mode 100644 index 0000000000..e829abccec --- /dev/null +++ b/hw/vfio-user/common.c @@ -0,0 +1,171 @@ +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include +#include + +#include "hw/hw.h" +#include "hw/vfio/vfio-common.h" +#include "io/channel.h" +#include "io/channel-socket.h" +#include "io/channel-util.h" +#include "qapi/error.h" +#include "qemu/error-report.h" +#include "qemu/lockable.h" +#include "qemu/sockets.h" +#include "system/iothread.h" + +#include "common.h" + +static IOThread *vfio_user_iothread; + +static void vfio_user_shutdown(VFIOUserProxy *proxy); + + +/* + * Functions called by main, CPU, or iothread threads + */ + +static void vfio_user_shutdown(VFIOUserProxy *proxy) +{ + qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, NULL, + proxy->ctx, NULL, NULL); +} + +/* + * Functions only called by iothread + */ + +static void vfio_user_cb(void *opaque) +{ + VFIOUserProxy *proxy = opaque; + + QEMU_LOCK_GUARD(&proxy->lock); + + proxy->state = VFIO_PROXY_CLOSED; + qemu_cond_signal(&proxy->close_cv); +} + + +/* + * Functions called by main or CPU threads + */ + +static QLIST_HEAD(, VFIOUserProxy) vfio_user_sockets = + QLIST_HEAD_INITIALIZER(vfio_user_sockets); + +VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) +{ + VFIOUserProxy *proxy; + QIOChannelSocket *sioc; + QIOChannel *ioc; + char *sockname; + + if (addr->type != SOCKET_ADDRESS_TYPE_UNIX) { + error_setg(errp, "vfio_user_connect - bad address family"); + return NULL; + } + sockname = addr->u.q_unix.path; + + sioc = qio_channel_socket_new(); + ioc = QIO_CHANNEL(sioc); + if (qio_channel_socket_connect_sync(sioc, addr, errp)) { + object_unref(OBJECT(ioc)); + return NULL; + } + qio_channel_set_blocking(ioc, false, NULL); + + proxy = g_malloc0(sizeof(VFIOUserProxy)); + proxy->sockname = g_strdup_printf("unix:%s", sockname); + proxy->ioc = ioc; + proxy->flags = VFIO_PROXY_CLIENT; + proxy->state = VFIO_PROXY_CONNECTED; + + qemu_mutex_init(&proxy->lock); + qemu_cond_init(&proxy->close_cv); + + if (vfio_user_iothread == NULL) { + vfio_user_iothread = iothread_create("VFIO user", errp); + } + + proxy->ctx = iothread_get_aio_context(vfio_user_iothread); + + QTAILQ_INIT(&proxy->outgoing); + QTAILQ_INIT(&proxy->incoming); + QTAILQ_INIT(&proxy->free); + QTAILQ_INIT(&proxy->pending); + QLIST_INSERT_HEAD(&vfio_user_sockets, proxy, next); + + return proxy; +} + +void vfio_user_disconnect(VFIOUserProxy *proxy) +{ + VFIOUserMsg *r1, *r2; + + qemu_mutex_lock(&proxy->lock); + + /* our side is quitting */ + if (proxy->state == VFIO_PROXY_CONNECTED) { + vfio_user_shutdown(proxy); + if (!QTAILQ_EMPTY(&proxy->pending)) { + error_printf("vfio_user_disconnect: outstanding requests\n"); + } + } + object_unref(OBJECT(proxy->ioc)); + proxy->ioc = NULL; + + proxy->state = VFIO_PROXY_CLOSING; + QTAILQ_FOREACH_SAFE(r1, &proxy->outgoing, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->outgoing, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->incoming, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->incoming, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->pending, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->free, r1, next); + g_free(r1); + } + + /* + * Make sure the iothread isn't blocking anywhere + * with a ref to this proxy by waiting for a BH + * handler to run after the proxy fd handlers were + * deleted above. + */ + aio_bh_schedule_oneshot(proxy->ctx, vfio_user_cb, proxy); + qemu_cond_wait(&proxy->close_cv, &proxy->lock); + + /* we now hold the only ref to proxy */ + qemu_mutex_unlock(&proxy->lock); + qemu_cond_destroy(&proxy->close_cv); + qemu_mutex_destroy(&proxy->lock); + + QLIST_REMOVE(proxy, next); + if (QLIST_EMPTY(&vfio_user_sockets)) { + iothread_destroy(vfio_user_iothread); + vfio_user_iothread = NULL; + } + + g_free(proxy->sockname); + g_free(proxy); +} diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h new file mode 100644 index 0000000000..3529f03df9 --- /dev/null +++ b/hw/vfio-user/common.h @@ -0,0 +1,78 @@ +#ifndef VFIO_USER_COMMON_H +#define VFIO_USER_COMMON_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + */ + +typedef struct { + int send_fds; + int recv_fds; + int *fds; +} VFIOUserFDs; + +enum msg_type { + VFIO_MSG_NONE, + VFIO_MSG_ASYNC, + VFIO_MSG_WAIT, + VFIO_MSG_NOWAIT, + VFIO_MSG_REQ, +}; + +typedef struct VFIOUserMsg { + QTAILQ_ENTRY(VFIOUserMsg) next; + VFIOUserFDs *fds; + uint32_t rsize; + uint32_t id; + QemuCond cv; + bool complete; + enum msg_type type; +} VFIOUserMsg; + + +enum proxy_state { + VFIO_PROXY_CONNECTED = 1, + VFIO_PROXY_ERROR = 2, + VFIO_PROXY_CLOSING = 3, + VFIO_PROXY_CLOSED = 4, +}; + +typedef QTAILQ_HEAD(VFIOUserMsgQ, VFIOUserMsg) VFIOUserMsgQ; + +typedef struct VFIOUserProxy { + QLIST_ENTRY(VFIOUserProxy) next; + char *sockname; + struct QIOChannel *ioc; + void (*request)(void *opaque, VFIOUserMsg *msg); + void *req_arg; + int flags; + QemuCond close_cv; + AioContext *ctx; + QEMUBH *req_bh; + + /* + * above only changed when BQL is held + * below are protected by per-proxy lock + */ + QemuMutex lock; + VFIOUserMsgQ free; + VFIOUserMsgQ pending; + VFIOUserMsgQ incoming; + VFIOUserMsgQ outgoing; + VFIOUserMsg *last_nowait; + enum proxy_state state; +} VFIOUserProxy; + +/* VFIOProxy flags */ +#define VFIO_PROXY_CLIENT 0x1 + +VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); +void vfio_user_disconnect(VFIOUserProxy *proxy); + +#endif /* VFIO_USER_COMMON_H */ diff --git a/hw/vfio-user/meson.build b/hw/vfio-user/meson.build index f1fee70c85..b89985c641 100644 --- a/hw/vfio-user/meson.build +++ b/hw/vfio-user/meson.build @@ -1,5 +1,6 @@ vfio_user_ss = ss.source_set() vfio_user_ss.add(files( + 'common.c', 'container.c', 'pci.c', )) diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index 6c85c60179..bfd9051524 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -19,6 +19,8 @@ #include "hw/qdev-properties.h" #include "hw/qdev-properties-system.h" #include "hw/vfio/pci.h" +#include "hw/vfio-user/common.h" +#include "hw/vfio-user/container.h" #include "qapi/error.h" #include "qobject/qdict.h" #include "qemu/error-report.h" @@ -59,6 +61,8 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev = &vdev->vbasedev; AddressSpace *as; + SocketAddress addr; + VFIOUserProxy *proxy; /* * TODO: make option parser understand SocketAddress @@ -71,6 +75,15 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) return; } + memset(&addr, 0, sizeof(addr)); + addr.type = SOCKET_ADDRESS_TYPE_UNIX; + addr.u.q_unix.path = udev->sock_name; + proxy = vfio_user_connect_dev(&addr, errp); + if (!proxy) { + return; + } + vbasedev->proxy = proxy; + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->ops = &vfio_user_pci_ops; vbasedev->type = VFIO_DEVICE_TYPE_PCI; @@ -119,8 +132,13 @@ static void vfio_user_instance_init(Object *obj) static void vfio_user_instance_finalize(Object *obj) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + VFIODevice *vbasedev = &vdev->vbasedev; vfio_pci_put_device(vdev); + + if (vbasedev->proxy != NULL) { + vfio_user_disconnect(vbasedev->proxy); + } } static const Property vfio_user_pci_dev_properties[] = { diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ee8e7f7c0d..699e8a9376 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -79,6 +79,7 @@ typedef struct VFIOMigration { } VFIOMigration; struct VFIOGroup; +typedef struct VFIOUserProxy VFIOUserProxy; /* MMU container sub-class for legacy vfio implementation. */ typedef struct VFIOContainer { @@ -155,6 +156,7 @@ typedef struct VFIODevice { IOMMUFDBackend *iommufd; VFIOIOASHwpt *hwpt; QLIST_ENTRY(VFIODevice) hwpt_next; + VFIOUserProxy *proxy; struct vfio_region_info **regions; } VFIODevice; From patchwork Wed Feb 19 14:48:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982414 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1EB97C021B1 for ; Wed, 19 Feb 2025 14:55:26 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklP0-0003V9-5u; Wed, 19 Feb 2025 09:50:18 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOy-0003To-9v; Wed, 19 Feb 2025 09:50:16 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOv-0007WD-6K; Wed, 19 Feb 2025 09:50:16 -0500 Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9LnWs011251; Wed, 19 Feb 2025 06:50:08 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=nLfnGtfBTmnnpgLKtSGQN6h/7mlRWzPVdJOlBx539 eM=; b=p5cK+WhD0AKVF15IIl4PxWcxQtTkBfFkwE9k8a+Z7as6t/bDcYOQfUjh3 cG0JevZY4rapFxe7O+s2gCwBs/4ldzioaqzBkL2fbU1ILJseoBFzlS2iGPwyeUUZ xotXJtlTbQpZ64oc4m+zlF/F0u8j1RGiL2NHrfZzTyZm4S0+MCHiUlwBoOYweFOG y3GOHCK44x2Opr5pvljNgIee9aNCRxSrYmjDERgr/Uxwrm71AClUZaDHd90dvR8r cST4RpedFsJBjixoqvBdXD/vS99zXTjJQyyyleS2HipdixTRl2R3FyczAbZTvig8 1GNgVhYzZyGrmk6MKBSIMQfhFei4g== Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2044.outbound.protection.outlook.com [104.47.58.44]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4bdhxxr-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:07 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=nKzVlvp3OUABuiDxgQdzvQ5JjdTUstI7HOjCwLyZPS9xLvtLD97Zc1xVj28I+xhquExHJS7sakiVM34nUBk9tMFDeVz1qh8JpkITcI6Wb4/ZMX1m2Adfc4DQWvzoSQCW/mz7wrQVB7v1U+cLuyywE/E9q8E0QfvTu9nGBAil2aH1qUx/6Fsv5Xe5gUKM0DhpFQzF55JbngxJwYVLMfvSCGYmPM/OoCJpDfEcMcQetkNfNhyEUJ2UYogNdUcPAd+Lw99YUOfbYZF3XN7x7/O8ArVUAJ+3UIPBRcjRrZPTy3WZG7D/zvilL10SPk2xOHCqsGN8sMW4+q054be1LFnnVA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=nLfnGtfBTmnnpgLKtSGQN6h/7mlRWzPVdJOlBx539eM=; b=efz4eUA1ePyaR5WyzF6xhbGrfH41TGw1syKqtr3I45Skq1RYOgyhUgNSDnNtQdZHYP25/3CP9Xq17bDSrYPemqhux0e9WeJTJhjLy0CN5b4AB03AdX5e6h0nTAY7yMDk0ewiuTH/YJLc/4AXSfU62Z9/zDecYP/T5Rpy+l+Emcd+r+NyDIEHIu82zB3I7HGbecXfrLSx+U06L1QP9EFiOp1T/z27JT5ixu1SBSDmGnNeGqPojmZr8m39s9OO9d/V8A0WGJtDMVS7AsM/qLPVR5oEIle0n6UtBTQzdj1I6hmJVqafsBOen6sF4LEyyq/UCALlBUUy2+4D4uM/U6g+3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nLfnGtfBTmnnpgLKtSGQN6h/7mlRWzPVdJOlBx539eM=; b=v/zAgywWE8BB6quhJAjzqTIcLnzmfJI5CbOUveQOghriR7vWBBXrVX4yQ7UKd1MWcHeod2LFiviak514n00/Al2WBcd786kURwo/jR8WAcK9Q0dtd/vuLqUggoZxpwKY7dARx+rWiUX5Vv281QGSG2ZUDTFFPgzbl8uOCkED99BnQQxb4ljPTlGurplRE41UxVhTkk8HggTjDZ5yUXAY+Mbx+B/MVeRW9GQQnsS+5r/c/LpNDqFlonznUxt9ZiDrsulgbY+pcpYydSgZVvYBiij6KAbZ2IeaKzQdssWH3vohPWaSVD4kQCcGM8jOb3Np1TOnfNyY2HlO87bBWGP9wg== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:49:58 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:49:58 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 14/28] vfio-user: implement message receive infrastructure Date: Wed, 19 Feb 2025 15:48:44 +0100 Message-Id: <20250219144858.266455-15-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 9cefdfc2-0cda-4872-f913-08dd50f4ae36 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: =?utf-8?q?RKMAytDgVh18XprqWlwZItwshDcdiN6?= =?utf-8?q?TnhfTSW1M0vGevKzeEDQvJ5PUNInVmwMz4gOIi3ITb2wkAuA4xX9jWsoKFgzjjxnk?= =?utf-8?q?i8Vhb31DVqWpt6Ft8jjxPRMxIvull517EDxwnzhjKcAPYeTxCA5D/aaTzluj6kEf1?= =?utf-8?q?xIuAr7wf9ygPbEbEN8eylJTBBY5eW7++41QBgR8tPWiwyCF3ML+jjOD9XBIv5a/QH?= =?utf-8?q?eGs89nbZsXtS2PFNZ4RpJK9k/KyUnf1xGq5k+vfQMZ1ugyubPGYAb0Vo94oVEtYjb?= =?utf-8?q?HRe5V5ZHCVDNoBu76fRsAmY/T+S9811PZ0zCicrp0PROt3prNfXQUyFrzat1+D0m4?= =?utf-8?q?qbsTdliARvS/Y7ooajHPyLS2D+ldLdZdry3y2npNDfaC+1V0iLykVBrO8d7/QcpVa?= =?utf-8?q?oeQIMYyKBc7nrwMCdQx0ImCTnSyIr1sN11T6SVYW5Ruxmh97GES7BM7pu4el6g+K1?= =?utf-8?q?do704sJe1LUdap/bj2e5t2cXC4/GIkc9TRI4dTRXto8gv6h4R5QimiDUQ0xy5djH5?= =?utf-8?q?qNvQL6utBhleDTuEN178VZ+Q1qai8laGjYpLImHcHP4BLDAfZG0S9mJdFKXXkRnlO?= =?utf-8?q?quYEjV3vRn0Mgbqq9lH8rTx6TwLbfGX/HP0qL8anBzru+ij6zOgk4T/jfxIDAyWCC?= =?utf-8?q?a2AVdJZAnwlqgyLhlTnHvatLM+brvMH4ENaJz6EtdjDhZNg8T4HescCsHlx6XvSBK?= =?utf-8?q?eAjc/eXCNihjYBA1K0uoVOig01/zA08+Ka3YuzMupEttITuA70LovzmmrX8LFIr1i?= =?utf-8?q?+ya4s4D9rnORxE4vk1HQXaeJGwIcteIiQcRv0g6ZbckkLisjmQkCctl14kMj+gQVX?= =?utf-8?q?8PMFdiipokFb366EDgUnVjULucbk3AFD4qm/UQpRP4TsP3fumo9hO68ehD7TGyrfS?= =?utf-8?q?ww6zPXOffIizcGmL4eOnfRURaIGEgTkdLeqnWPunlCP9lY3tkPchq6u8KzxyH2QM4?= =?utf-8?q?UihRvhBWE/wAFjRo7Kl/chio5oUdQdgvtDhED+tOJS0MvlotmWjfMkbpjBf4f8IgB?= =?utf-8?q?JzJTl4rg4OkBKlYbQUXRESlZuHHI6jFmNd76I8v05M9QFFTaGRPTuG6OL1/kwteVy?= =?utf-8?q?xgcObdCCPZETKKvU+brwSz5NKqAEji5+PcB7Hx7LS9zt1h8WJIeeyoHqOhJd84KTX?= =?utf-8?q?vmp8uauZ4ZIdNjK3Qj1gRIW1uxTpbTGpdusWM4tkF1PyP3hppMbWvufEkx8k6Acai?= =?utf-8?q?4oKQeZn3ZLyB0qrZ4R8BYaHgbL9Bym7XHZbaJkqdyBjTaRdc7yOq3fAyhdOUgWJu4?= =?utf-8?q?d9yiFzWprJuT/zJ2NU6wZP01o5vUF6BOHgVadRXoSDtsqMMQsuxNyMiOVU2PJ5Ori?= =?utf-8?q?eaNo+q7NH0bL?= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?aNOPxynsa+trHGelt4eAkVq9Hnqt?= =?utf-8?q?GKMfMtZtgbhoY8FSXwKOEHNyLHS+u9vqW326326RkIjsR26wvxSZhu/7xdSU4eP1y?= =?utf-8?q?jyINLNevSNhsRF63kMxKK2WkY3BEua/Cx8U/QtsOCqm0p6eth/mI+3Bx96WdFuyLE?= =?utf-8?q?eW/UZ7QJAav+p65I8r/kS1P8GCm4ZrvZV/n+JiGQl9y0iWxDnNlXkWiJawHXltbJe?= =?utf-8?q?I7y7ddMu7QD9oOmjPQxcT7O9YZMlraFgH6MLDx8Br8UEhDmulxjqpVhjf3BTds0U0?= =?utf-8?q?sWx4jgBgMd3IQUQWqKUAV0vl4vU0w+f4MvEubisax0uKAR+rz2X2Ub4H+CrP7RCYA?= =?utf-8?q?ql6FnjNW/mkeEWY4FZcD6wDkhQLVKNkjMRk35OuyLeUOZvjetvedfzbGhZhiv59LQ?= =?utf-8?q?Slt8yPj72lBI6rfrU9E0cuGtJNI3vXHSL1SiN6O7MtApNFK6yy1y6kahaUJEDLQ3A?= =?utf-8?q?PsyLW1rqSokaYjS/0SS9Ug8a30X137Opf1KoN9LSwjnuK6l26IHmqoF/9huK0gtDH?= =?utf-8?q?ZbPjNSes7xfLb6Ah4kA+BlRbydDs6pvwL6XNghGwAwAWl8YCopeEgN9THVR8O9Iei?= =?utf-8?q?sR7xBQYH6vuraWP/AehSbvygzIpqHEyjbnV4w2c2kYnqhWiPRAnxrNoxBYsgbdn2M?= =?utf-8?q?49Dv3DhTMCAuOgfDJhXutih7zeeW0oOIddjYYkTtY/DNGQlX0fXdN5mYywo87n37r?= =?utf-8?q?a7C0HCHwPBQNTD/fGKhBepvcFuTMbYfjSOkFDyaDbMDs6hOTeydvkIDObSNTyzhVF?= =?utf-8?q?s7ebmtolepI4CgjeteRYCYauAsuqPkO6lGXqyA1P7tQbR4+gZbq10CdrF5qJ2TLZc?= =?utf-8?q?ya9Km8jhEONiZOrx6eHGuqiQimCaeMgKuZ7XOMBpnhUwcSkEFLsdpC741YJlJbJBt?= =?utf-8?q?pMdSl4CZbS50iPi2XBoOyfRIcc5xluY2TaS9zgCP6sQSCoSWXvzpHG1JYvZbZPbJN?= =?utf-8?q?2yFYUGMJyc6mc9qRvHVXp5B4DTzEGmIniOROMVKLOwKv0lyodyrUdOQoiJbHzaDMb?= =?utf-8?q?VwXeY4uimd2/JSuaL7cZKoDwy/7FkMSPQLT/THuAyZLBJcA2vwVyrDigJDAteK2U9?= =?utf-8?q?jFxYU6MnSb7mTL+8dZ1iRmufhINr1tn1SXdWHHx8j8s5WZRDmSRJEJeZFm7iDiJyt?= =?utf-8?q?tTPlVMxuwhPAtVOnAV2OVsXZ455ruoKScsJ3Z29XiCrNjwrcevnJMmqPXTko7Vfo7?= =?utf-8?q?S3kOcEMkYAtLWs7oaE4Aq6ZPl9QFTmrqAbWRUnlHXxeZ1DZ2LVXIONHhrDg7PoZPA?= =?utf-8?q?VuGuTOJUmXLa3OK+0JpHlscnDkSYQMVB7S8h11ChcqRYRh68or1Kucy2rOp0c1ilq?= =?utf-8?q?tMuxb4KGFcVz+ysaCpcQCccuuUMRaole4Pg+mJRnHnDsd5iiAi/fslanbGQ50p+Fw?= =?utf-8?q?C+BnUmz/0GUxLK+Heul0AeIF+mm0WZJV/9S1i/BEsDjqIL44h489HJD7Azs1YBrjY?= =?utf-8?q?/fyOhTtjVidTnxwvXRoUZ9c2q3vbi7N74HEPiuc19zC4QY4l2aH7QnZWt8j2YYC55?= =?utf-8?q?IlZx+lLjnVPX?= X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9cefdfc2-0cda-4872-f913-08dd50f4ae36 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:49:58.4087 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: H8rqG1VL0eY3FvG5PRzxQupsleYOmzA188e0b/mZEAf68nFpq6ibENrMb/yx/8V5oFX9JV0mUUi2WRPc5vvrHQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-ORIG-GUID: YcsxcfGHZQTIAioR5P2_HApIIts87iq6 X-Proofpoint-GUID: YcsxcfGHZQTIAioR5P2_HApIIts87iq6 X-Authority-Analysis: v=2.4 cv=Fo7//3rq c=1 sm=1 tr=0 ts=67b5ef9f cx=c_pps a=di3315gfm3qlniCp1Rh91A==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=IkcTkHD0fZMA:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=vXCtUE0a-urccVGwgxAA:9 a=3ZKOabzyN94A:10 a=QEXdDO2ut3YA:10 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Add the basic implementation for receiving vfio-user messages from the control socket. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 412 +++++++++++++++++++++++++++++++++++++- hw/vfio-user/common.h | 10 + hw/vfio-user/pci.c | 11 + hw/vfio-user/protocol.h | 54 +++++ hw/vfio-user/trace-events | 6 + hw/vfio-user/trace.h | 1 + meson.build | 1 + 7 files changed, 493 insertions(+), 2 deletions(-) create mode 100644 hw/vfio-user/protocol.h create mode 100644 hw/vfio-user/trace-events create mode 100644 hw/vfio-user/trace.h diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index e829abccec..2c4ee14ede 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -14,21 +14,37 @@ #include "hw/hw.h" #include "hw/vfio/vfio-common.h" +#include "hw/vfio-user/common.h" +#include "hw/vfio-user/trace.h" #include "io/channel.h" #include "io/channel-socket.h" #include "io/channel-util.h" #include "qapi/error.h" #include "qemu/error-report.h" #include "qemu/lockable.h" +#include "qemu/main-loop.h" #include "qemu/sockets.h" #include "system/iothread.h" -#include "common.h" - static IOThread *vfio_user_iothread; static void vfio_user_shutdown(VFIOUserProxy *proxy); +static VFIOUserMsg *vfio_user_getmsg(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds); +static VFIOUserFDs *vfio_user_getfds(int numfds); +static void vfio_user_recycle(VFIOUserProxy *proxy, VFIOUserMsg *msg); + +static void vfio_user_recv(void *opaque); +static int vfio_user_recv_one(VFIOUserProxy *proxy); +static void vfio_user_cb(void *opaque); + +static void vfio_user_request(void *opaque); +static inline void vfio_user_set_error(VFIOUserHdr *hdr, uint32_t err) +{ + hdr->flags |= VFIO_USER_ERROR; + hdr->error_reply = err; +} /* * Functions called by main, CPU, or iothread threads @@ -41,10 +57,340 @@ static void vfio_user_shutdown(VFIOUserProxy *proxy) proxy->ctx, NULL, NULL); } +static VFIOUserMsg *vfio_user_getmsg(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds) +{ + VFIOUserMsg *msg; + + msg = QTAILQ_FIRST(&proxy->free); + if (msg != NULL) { + QTAILQ_REMOVE(&proxy->free, msg, next); + } else { + msg = g_malloc0(sizeof(*msg)); + qemu_cond_init(&msg->cv); + } + + msg->hdr = hdr; + msg->fds = fds; + return msg; +} + +/* + * Recycle a message list entry to the free list. + */ +static void vfio_user_recycle(VFIOUserProxy *proxy, VFIOUserMsg *msg) +{ + if (msg->type == VFIO_MSG_NONE) { + error_printf("vfio_user_recycle - freeing free msg\n"); + return; + } + + /* free msg buffer if no one is waiting to consume the reply */ + if (msg->type == VFIO_MSG_NOWAIT || msg->type == VFIO_MSG_ASYNC) { + g_free(msg->hdr); + if (msg->fds != NULL) { + g_free(msg->fds); + } + } + + msg->type = VFIO_MSG_NONE; + msg->hdr = NULL; + msg->fds = NULL; + msg->complete = false; + QTAILQ_INSERT_HEAD(&proxy->free, msg, next); +} + +static VFIOUserFDs *vfio_user_getfds(int numfds) +{ + VFIOUserFDs *fds = g_malloc0(sizeof(*fds) + (numfds * sizeof(int))); + + fds->fds = (int *)((char *)fds + sizeof(*fds)); + + return fds; +} + /* * Functions only called by iothread */ +/* + * Process a received message. + */ +static void vfio_user_process(VFIOUserProxy *proxy, VFIOUserMsg *msg, + bool isreply) +{ + + /* + * Replies signal a waiter, if none just check for errors + * and free the message buffer. + * + * Requests get queued for the BH. + */ + if (isreply) { + msg->complete = true; + if (msg->type == VFIO_MSG_WAIT) { + qemu_cond_signal(&msg->cv); + } else { + if (msg->hdr->flags & VFIO_USER_ERROR) { + error_printf("vfio_user_process: error reply on async "); + error_printf("request command %x error %s\n", + msg->hdr->command, + strerror(msg->hdr->error_reply)); + } + /* youngest nowait msg has been ack'd */ + if (proxy->last_nowait == msg) { + proxy->last_nowait = NULL; + } + vfio_user_recycle(proxy, msg); + } + } else { + QTAILQ_INSERT_TAIL(&proxy->incoming, msg, next); + qemu_bh_schedule(proxy->req_bh); + } +} + +/* + * Complete a partial message read + */ +static int vfio_user_complete(VFIOUserProxy *proxy, Error **errp) +{ + VFIOUserMsg *msg = proxy->part_recv; + size_t msgleft = proxy->recv_left; + bool isreply; + char *data; + int ret; + + data = (char *)msg->hdr + (msg->hdr->size - msgleft); + while (msgleft > 0) { + ret = qio_channel_read(proxy->ioc, data, msgleft, errp); + + /* error or would block */ + if (ret <= 0) { + /* try for rest on next iternation */ + if (ret == QIO_CHANNEL_ERR_BLOCK) { + proxy->recv_left = msgleft; + } + return ret; + } + trace_vfio_user_recv_read(msg->hdr->id, ret); + + msgleft -= ret; + data += ret; + } + + /* + * Read complete message, process it. + */ + proxy->part_recv = NULL; + proxy->recv_left = 0; + isreply = (msg->hdr->flags & VFIO_USER_TYPE) == VFIO_USER_REPLY; + vfio_user_process(proxy, msg, isreply); + + /* return positive value */ + return 1; +} + +static void vfio_user_recv(void *opaque) +{ + VFIOUserProxy *proxy = opaque; + + QEMU_LOCK_GUARD(&proxy->lock); + + if (proxy->state == VFIO_PROXY_CONNECTED) { + while (vfio_user_recv_one(proxy) == 0) { + ; + } + } +} + +/* + * Receive and process one incoming message. + * + * For replies, find matching outgoing request and wake any waiters. + * For requests, queue in incoming list and run request BH. + */ +static int vfio_user_recv_one(VFIOUserProxy *proxy) +{ + VFIOUserMsg *msg = NULL; + g_autofree int *fdp = NULL; + VFIOUserFDs *reqfds; + VFIOUserHdr hdr; + struct iovec iov = { + .iov_base = &hdr, + .iov_len = sizeof(hdr), + }; + bool isreply = false; + int i, ret; + size_t msgleft, numfds = 0; + char *data = NULL; + char *buf = NULL; + Error *local_err = NULL; + + /* + * Complete any partial reads + */ + if (proxy->part_recv != NULL) { + ret = vfio_user_complete(proxy, &local_err); + + /* still not complete, try later */ + if (ret == QIO_CHANNEL_ERR_BLOCK) { + return ret; + } + + if (ret <= 0) { + goto fatal; + } + /* else fall into reading another msg */ + } + + /* + * Read header + */ + ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, 0, + &local_err); + if (ret == QIO_CHANNEL_ERR_BLOCK) { + return ret; + } + + /* read error or other side closed connection */ + if (ret <= 0) { + goto fatal; + } + + if (ret < sizeof(hdr)) { + error_setg(&local_err, "short read of header"); + goto fatal; + } + + /* + * Validate header + */ + if (hdr.size < sizeof(VFIOUserHdr)) { + error_setg(&local_err, "bad header size"); + goto fatal; + } + switch (hdr.flags & VFIO_USER_TYPE) { + case VFIO_USER_REQUEST: + isreply = false; + break; + case VFIO_USER_REPLY: + isreply = true; + break; + default: + error_setg(&local_err, "unknown message type"); + goto fatal; + } + trace_vfio_user_recv_hdr(proxy->sockname, hdr.id, hdr.command, hdr.size, + hdr.flags); + + /* + * For replies, find the matching pending request. + * For requests, reap incoming FDs. + */ + if (isreply) { + QTAILQ_FOREACH(msg, &proxy->pending, next) { + if (hdr.id == msg->id) { + break; + } + } + if (msg == NULL) { + error_setg(&local_err, "unexpected reply"); + goto err; + } + QTAILQ_REMOVE(&proxy->pending, msg, next); + + /* + * Process any received FDs + */ + if (numfds != 0) { + if (msg->fds == NULL || msg->fds->recv_fds < numfds) { + error_setg(&local_err, "unexpected FDs"); + goto err; + } + msg->fds->recv_fds = numfds; + memcpy(msg->fds->fds, fdp, numfds * sizeof(int)); + } + } else { + if (numfds != 0) { + reqfds = vfio_user_getfds(numfds); + memcpy(reqfds->fds, fdp, numfds * sizeof(int)); + } else { + reqfds = NULL; + } + } + + /* + * Put the whole message into a single buffer. + */ + if (isreply) { + if (hdr.size > msg->rsize) { + error_setg(&local_err, "reply larger than recv buffer"); + goto err; + } + *msg->hdr = hdr; + data = (char *)msg->hdr + sizeof(hdr); + } else { + buf = g_malloc0(hdr.size); + memcpy(buf, &hdr, sizeof(hdr)); + data = buf + sizeof(hdr); + msg = vfio_user_getmsg(proxy, (VFIOUserHdr *)buf, reqfds); + msg->type = VFIO_MSG_REQ; + } + + /* + * Read rest of message. + */ + msgleft = hdr.size - sizeof(hdr); + while (msgleft > 0) { + ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err); + + /* prepare to complete read on next iternation */ + if (ret == QIO_CHANNEL_ERR_BLOCK) { + proxy->part_recv = msg; + proxy->recv_left = msgleft; + return ret; + } + + if (ret <= 0) { + goto fatal; + } + trace_vfio_user_recv_read(hdr.id, ret); + + msgleft -= ret; + data += ret; + } + + vfio_user_process(proxy, msg, isreply); + return 0; + + /* + * fatal means the other side closed or we don't trust the stream + * err means this message is corrupt + */ +fatal: + vfio_user_shutdown(proxy); + proxy->state = VFIO_PROXY_ERROR; + + /* set error if server side closed */ + if (ret == 0) { + error_setg(&local_err, "server closed socket"); + } + +err: + for (i = 0; i < numfds; i++) { + close(fdp[i]); + } + if (isreply && msg != NULL) { + /* force an error to keep sending thread from hanging */ + vfio_user_set_error(msg->hdr, EINVAL); + msg->complete = true; + qemu_cond_signal(&msg->cv); + } + error_prepend(&local_err, "vfio_user_recv_one: "); + error_report_err(local_err); + return -1; +} + static void vfio_user_cb(void *opaque) { VFIOUserProxy *proxy = opaque; @@ -60,6 +406,53 @@ static void vfio_user_cb(void *opaque) * Functions called by main or CPU threads */ +/* + * Process incoming requests. + * + * The bus-specific callback has the form: + * request(opaque, msg) + * where 'opaque' was specified in vfio_user_set_handler + * and 'msg' is the inbound message. + * + * The callback is responsible for disposing of the message buffer, + * usually by re-using it when calling vfio_send_reply or vfio_send_error, + * both of which free their message buffer when the reply is sent. + * + * If the callback uses a new buffer, it needs to free the old one. + */ +static void vfio_user_request(void *opaque) +{ + VFIOUserProxy *proxy = opaque; + VFIOUserMsgQ new, free; + VFIOUserMsg *msg, *m1; + + /* reap all incoming */ + QTAILQ_INIT(&new); + WITH_QEMU_LOCK_GUARD(&proxy->lock) { + QTAILQ_FOREACH_SAFE(msg, &proxy->incoming, next, m1) { + QTAILQ_REMOVE(&proxy->incoming, msg, next); + QTAILQ_INSERT_TAIL(&new, msg, next); + } + } + + /* process list */ + QTAILQ_INIT(&free); + QTAILQ_FOREACH_SAFE(msg, &new, next, m1) { + QTAILQ_REMOVE(&new, msg, next); + trace_vfio_user_recv_request(msg->hdr->command); + proxy->request(proxy->req_arg, msg); + QTAILQ_INSERT_HEAD(&free, msg, next); + } + + /* free list */ + WITH_QEMU_LOCK_GUARD(&proxy->lock) { + QTAILQ_FOREACH_SAFE(msg, &free, next, m1) { + vfio_user_recycle(proxy, msg); + } + } +} + + static QLIST_HEAD(, VFIOUserProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -98,6 +491,7 @@ VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) } proxy->ctx = iothread_get_aio_context(vfio_user_iothread); + proxy->req_bh = qemu_bh_new(vfio_user_request, proxy); QTAILQ_INIT(&proxy->outgoing); QTAILQ_INIT(&proxy->incoming); @@ -108,6 +502,18 @@ VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) return proxy; } +void vfio_user_set_handler(VFIODevice *vbasedev, + void (*handler)(void *opaque, VFIOUserMsg *msg), + void *req_arg) +{ + VFIOUserProxy *proxy = vbasedev->proxy; + + proxy->request = handler; + proxy->req_arg = req_arg; + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, + vfio_user_recv, NULL, NULL, proxy); +} + void vfio_user_disconnect(VFIOUserProxy *proxy) { VFIOUserMsg *r1, *r2; @@ -123,6 +529,8 @@ void vfio_user_disconnect(VFIOUserProxy *proxy) } object_unref(OBJECT(proxy->ioc)); proxy->ioc = NULL; + qemu_bh_delete(proxy->req_bh); + proxy->req_bh = NULL; proxy->state = VFIO_PROXY_CLOSING; QTAILQ_FOREACH_SAFE(r1, &proxy->outgoing, next, r2) { diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 3529f03df9..71c0dea82d 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -11,6 +11,8 @@ * */ +#include "hw/vfio-user/protocol.h" + typedef struct { int send_fds; int recv_fds; @@ -27,6 +29,7 @@ enum msg_type { typedef struct VFIOUserMsg { QTAILQ_ENTRY(VFIOUserMsg) next; + VFIOUserHdr *hdr; VFIOUserFDs *fds; uint32_t rsize; uint32_t id; @@ -66,13 +69,20 @@ typedef struct VFIOUserProxy { VFIOUserMsgQ incoming; VFIOUserMsgQ outgoing; VFIOUserMsg *last_nowait; + VFIOUserMsg *part_recv; + size_t recv_left; enum proxy_state state; } VFIOUserProxy; /* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 +typedef struct VFIODevice VFIODevice; + VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); void vfio_user_disconnect(VFIOUserProxy *proxy); +void vfio_user_set_handler(VFIODevice *vbasedev, + void (*handler)(void *opaque, VFIOUserMsg *msg), + void *reqarg); #endif /* VFIO_USER_COMMON_H */ diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index bfd9051524..fb6c872768 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -38,6 +38,16 @@ struct VFIOUserPCIDevice { char *sock_name; }; +/* + * Incoming request message callback. + * + * Runs off main loop, so BQL held. + */ +static void vfio_user_pci_process_req(void *opaque, VFIOUserMsg *msg) +{ + +} + /* * Emulated devices don't use host hot reset */ @@ -83,6 +93,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) return; } vbasedev->proxy = proxy; + vfio_user_set_handler(vbasedev, vfio_user_pci_process_req, vdev); vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->ops = &vfio_user_pci_ops; diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h new file mode 100644 index 0000000000..d23877c958 --- /dev/null +++ b/hw/vfio-user/protocol.h @@ -0,0 +1,54 @@ +#ifndef VFIO_USER_PROTOCOL_H +#define VFIO_USER_PROTOCOL_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + * Each message has a standard header that describes the command + * being sent, which is almost always a VFIO ioctl(). + * + * The header may be followed by command-specific data, such as the + * region and offset info for read and write commands. + */ + +typedef struct { + uint16_t id; + uint16_t command; + uint32_t size; + uint32_t flags; + uint32_t error_reply; +} VFIOUserHdr; + +/* VFIOUserHdr commands */ +enum vfio_user_command { + VFIO_USER_VERSION = 1, + VFIO_USER_DMA_MAP = 2, + VFIO_USER_DMA_UNMAP = 3, + VFIO_USER_DEVICE_GET_INFO = 4, + VFIO_USER_DEVICE_GET_REGION_INFO = 5, + VFIO_USER_DEVICE_GET_REGION_IO_FDS = 6, + VFIO_USER_DEVICE_GET_IRQ_INFO = 7, + VFIO_USER_DEVICE_SET_IRQS = 8, + VFIO_USER_REGION_READ = 9, + VFIO_USER_REGION_WRITE = 10, + VFIO_USER_DMA_READ = 11, + VFIO_USER_DMA_WRITE = 12, + VFIO_USER_DEVICE_RESET = 13, + VFIO_USER_DIRTY_PAGES = 14, + VFIO_USER_MAX, +}; + +/* VFIOUserHdr flags */ +#define VFIO_USER_REQUEST 0x0 +#define VFIO_USER_REPLY 0x1 +#define VFIO_USER_TYPE 0xF + +#define VFIO_USER_NO_REPLY 0x10 +#define VFIO_USER_ERROR 0x20 + +#endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events new file mode 100644 index 0000000000..89d6c11c4c --- /dev/null +++ b/hw/vfio-user/trace-events @@ -0,0 +1,6 @@ +# See docs/devel/tracing.rst for syntax documentation. + +# common.c +vfio_user_recv_hdr(const char *name, uint16_t id, uint16_t cmd, uint32_t size, uint32_t flags) " (%s) id 0x%x cmd 0x%x size 0x%x flags 0x%x" +vfio_user_recv_read(uint16_t id, int read) " id 0x%x read 0x%x" +vfio_user_recv_request(uint16_t cmd) " command 0x%x" diff --git a/hw/vfio-user/trace.h b/hw/vfio-user/trace.h new file mode 100644 index 0000000000..574b59aa89 --- /dev/null +++ b/hw/vfio-user/trace.h @@ -0,0 +1 @@ +#include "trace/trace-hw_vfio_user.h" diff --git a/meson.build b/meson.build index 8ed10b6624..0651d0710b 100644 --- a/meson.build +++ b/meson.build @@ -3593,6 +3593,7 @@ if have_system 'hw/ufs', 'hw/usb', 'hw/vfio', + 'hw/vfio-user', 'hw/virtio', 'hw/watchdog', 'hw/xen', From patchwork Wed Feb 19 14:48:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982411 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 142F2C021AA for ; Wed, 19 Feb 2025 14:55:07 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklOz-0003Ug-A1; Wed, 19 Feb 2025 09:50:17 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOy-0003Tj-3Y; Wed, 19 Feb 2025 09:50:16 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklOu-0007W2-RK; Wed, 19 Feb 2025 09:50:15 -0500 Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9LnWt011251; Wed, 19 Feb 2025 06:50:09 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=wP9VKDoVdVMgajX1HZl3mwV4iHY8MTZ1OCowAKPBY 7c=; b=2K6elqscgQW2lmY7/TZfr0eX2oDuZCnGrM6puEjxtCGgpRlznZsyvAYQN bqcAdggYnSQJQpdC5BxNmLuJ9NztlFhG3IkUwHXoWMr1ptGI2lBBzg5nIrSKsOCt xZhpH+bjI2+4lCojONqM8JNzWWs1g1nTAX87K48y62G9IDwGQ9wqVm/sHdzugcDL p8BhRK7GU2wU39nS8AaH3/UY8r/bO+Ni9iVImKII89/zFtjocaaOd8lYVMwxCgGU jTRJqoApYD8/vwhnnxU3pStQHYdWM2z133ESPPrZ22sV6m/JYBD0/ADbzKJ4iZD/ ev+I0fVqLOaRoXxZ+ouzfR1psqMxQ== Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2044.outbound.protection.outlook.com [104.47.58.44]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4bdhxxr-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:08 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=taYq6wQpZ1dhXyK3aWqLP+FcGTW7mJPmirHpjz9wMQyWqG4KKb+g7EtGiCl+ENqzccQ51QrEsjbD7vWh/2pBO9m9Dhy+Vjm2Oxjqjcp5AKruW9SY3/Fao/jtEjQhHS+r0KK7GvDd24GcYfUd+JWbsgIYtRrbqzEvy8GbIfhjKJoPYfBbx8fTkUqpxPRl+8C5Ffzm16ZnXYuzXMUikOdIf5FYXgzKTVHBytyYGXtDJTFlTYf1lQcoKqd8LQeeY2NcHjWUJIrLvB+4h56dgtXpng41q+x3Dk/zjcJejSgT2JfgqbMVyYaNVwDHOpPMt4kxPBgaP9qT+Vi3AKQnGcVt0Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wP9VKDoVdVMgajX1HZl3mwV4iHY8MTZ1OCowAKPBY7c=; b=sNH5ZOdwpfce/qyujhC2u+DOlH5r+LP0qb/YIZUeeCmo3lTlK9L4pDAY4mlhpiL5tPt4orA2edH3wEax+H56B7mn2qdYBAVI6wiYGRmV96kqnRyDppd0FEZ0UVvEPTHMn7pYVKsxUKfW0uDr1fv1u6AAHzKukxJuZ37bPn8HropGx7p25nRGisKDy25rknlj/BLk8VfJ1ncYBOJiQiDV6VYzS6nfynG00IKmj+WIUBttd/IYhDZCZxco1hDrpbVwOFd11HEr2V4wwc7blYK38ISzfBKptC5LMFp2l249HHf7V/BecXr+CLfrB37Vy2rVj8SeVJ+l2BkWALke+tEo7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wP9VKDoVdVMgajX1HZl3mwV4iHY8MTZ1OCowAKPBY7c=; b=A1vskrQqf3lcSWmEDH7QKlT0KRTsZAG/ozwl689snLR/H/QuDWvqR6wqMUdOGfzvnzNCs73+r04ccJECg5idcug4ej752HcLGxmBvro9E4g4UgV4wid9vIrBi4k8iC+q6Av01NPK/RzvyFyN6JwYr602PP2C/VvXW1fliX9LwmqhGTYJ2mURVaw7XiEaROzxcBrc17FkHiBKU7qoSxmQO4BIPvvQX4EiPecngSd/SinhdRhr3/TmfTMmPj3r024ZiaGmm7q+iaQOPFdLDBsEDfr6pdHtSu71WX4WXZ7ASIGuLxwAsZK9w4u84+H6oqjORWac9XXXLx9O8nqid9Uq2g== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:02 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:01 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 15/28] vfio-user: implement message send infrastructure Date: Wed, 19 Feb 2025 15:48:45 +0100 Message-Id: <20250219144858.266455-16-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 21d5852b-7310-421c-b7c6-08dd50f4b050 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: pDVbP/lJBl7NvouFbqskF+Sc3+RD8+R67JuCIzQh4wnt+fLH5512TPuc1j0oeZacdbslmgR9bXpcQ1hvugPTDZ9j7bEIW7q36j2aHx1GcV1T8hvQdQRc0ek0UA1TbqwL4+S+NfR5Q83IoFCHYPvtlx9ecBoWa313GdkT9mfEWwaM60nvPR4ASKwFYp8D5HENIq5Js99kMFiwvfpEyk397anzEkWQnMg5BPdLS4S1Ju1UCGNZmc+ZbR0aD9rdNRw0JjMXdQEVLT419+kqnk2nCob/FkkXqOplAWlZe2ZoZ/HxoFPmZkCUvw7mNXJVN8dnojBW1V0BTcBzKP/YKZEHRDgs/sDwxPQWh1Uw49H+MvxUqghY0qjwIuJ/aYclVzEy691A66PYO/bGdyYpxg/M7W8D1v5kx+sD0vG+94tcEpgZSYoZaobs4/YdvocL2N2NUIGay9d4uIKJk9DxerF2KWwEgasBB5CNvPuMpKag5REXu4J+d/ljdQbi8T9u1cJBb+BauOvPg5T+XlHcJRzitiBXqs7SCu4Z1idZ0EHqnme1/+t5NvFHUa5Kn5rvgplZFd5HtsaOb0Gt5gIoOGRPJBbPLJ1RDB2b4Q+FqWENQS9mpVYEaOJslk7VDxGeGKXtO7eE4GhXCFAFgRJFt8hW6mE8BpCbNx9MvdfYYbfZcT2KtWUD9PgM0iU1iwAA5AC+hTvQqdlSSEi9u8e40hjYwHatgfgmUfhplkHhRILkg3Y8ntQPZBMBZuoER8wtqLEdiRqfJGvMUYx2NO7Qs/3lv2ujdCTbqhWjNCZkiugjaLdMbUhX3qOJWe4mis5nhwQQbjyE7Z7dVINsdKzptDffF8pl36w3wgiMBDr2N1uuAyrXaF9yMw6z7DJinHSlJolgsh/SnXRvP8fNYhRP2Z6T69uU4v23Wf29CXOGVFWE/upGtPjFA61b2BPn1xH8vYFMcXr2vOAinYuha0p6eIfJ4/mjolgc8W3p7KgGnTzH6y/hSVosW07qvQT7q2a0JYaVBDdIOUywq0CRW/JSX5mWFXmz7PPLV2UurasbbWvR6+car/J3D7lf6wRe1uTzjxmq9pzqbqcq8Byq7dTed/MmRmmeWFnFLPxZsbz7BHn3rIPfvizhyLv7hddOwXdBdo3EcC80nKhUCpftnSepyBJ9c1OaGmCIHlB+WXwCVh7YB4Hnyjx6z1gmqQCX4yJXEpQ2NLydLOsjr0iCxdYDb31Hf0/C1kVXWL0YSNNI2LwaRpetVkr+9Xf9o0osAU519FnnnRTIUn6ThmmyJcvYCo09dtoA3qxojCzzopSFyTksNasRA2dJyFh6cMgIVgcfuhx8S8Q+4cylBed2mcizrAyjGksIM7P8JPJ1AupQuI2fqKQSWgZTb+alzXCr4A3CSqCx X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 51gUi3ujjfUmZcOv/pM2udMctb5KrGRi1MFKThN2hYXzb8mXovswDip20dUVxZ4S7Hx35iv+8V+9vdOGo+ZFJiGWtanm7hnVsrgPvMsTa/tBqfrL0T9jPrFvmPWPrDCZe+kq/qAjaEz1/Fv5Y2SIkLXGQHFTW+GcrbqzGLz8cAljAS0N4o3DBES+DTTpWImucNA5kLgAwdXn3UwQ3dXqp19hSct0IjtoF7AUVB+g93XD6yOeVwBwOqlW7XhMfCi6/IJPtmbq6qRknvkssuCZlAO9Q1ttjbLA/rdaK0p4Zuy+GxBmhp9WAduRzisYsGST+3mxv3c2mSUW22Vl38F9ogRYqYo2Ycfmwo1SIME/oACAkWYCJetQR3ztt/Q8X08aW/LT1wbm6d4F3XdNBaABAhYVnM1u2xmuJI/eur7uaBUDf3MHNd0LDVZ9KoxFdpUjpT6/vzPI8jhGYZOgeJutQR0J7Z1xyP0tTbQhZRYXOel/Hs2VpNUt6Q7uCYkKxLBuO9ZBHfWHVx2PXVNCaxYuvvHSrLLHHvkewENWgEend9AIubyd0I2C2vE4HvtHx1Auoa+xAPe6ZwRn/gzpSHZ6Ss1pk8g6ld9LJRszPdrzbB+KX/+CYHKqTsiru41/SB4OIYy4aD83CLa3wZnq0fwuUt8yy8Ol8zvXh1euNrZY9sqgDFPZHGVjj8h54yrXl53gmr5K7/VoG3U9DPtFbbU5VD4E77ai6S9S5AUOvgpblyqgtXyAUPFA5koxSS8rh+4K8QkkCBhp4FGyYF3uFgC1FcD7XAd49bgK2v0dqqY8J1B4ofLsFbKI7K00Uvx45KDm9VSDqBPzyz3cYEK8eMlurbE2bZzF8AOxbbrgE2J2V8KM7B8zJJ/EO8DRz49rAwhyLwB2TUMw3YvEbMNbc951IsapoN/BlSt608rMmLDE6qp+669ex6KLuJkC1LId9lYcuOBC9zg7hRmJz0yf37y90zQf4Wbc6ccpasBZWlrKE0elPLXCKxw7OiQU1UQ7wf6xAw/oILOoKhr4Gz9YrkQi6UyTSqp3UugpGm798uNqCoZIdU3A9FDLlQVvZj4NPMa4HpGZcQglOtJ8r80A8b/8HVA5gl6IrRbX/Kc3L9hTR8tRMVT90yzfltn4CLASOKrl8X/qe7CBX+IQQzs0TU3B8Rj5/vN8G4ryAbeQjKFd57X7n0pEBnJyGIMDdF/GMlnrOFObMpai08n0sJve16bZd2uzD97iZie77FNZawh0pBPnyFfukg/8LvCDM2OMzE+DDIaBH+t/HPxxfQ+TlmbBU9Cq7EX6X3XiFjAIYKW9AxVrbvsuuqiONYIjmQJuIykst9/7ncKmgrYc9Cq0AL3FkoIBAIDMB7z0zxWEmyaoaxIk1YWmQwNAzK0T3S4+df3dbqU9LlyZ2adtFSE1XwYW8lDezs2Q75GiBXOwE29jrLYEkRfIZGLUFIxiONnFZzbtGxxrwcBQvWYHJmy53RX3Aa1/H3UMTY7auUP3HKk9Hb3+SEUu1FRuF1gGGLMgd7hG2bFLV8Ag6NcLG7Et2KhbKpmModQX8jLdY5dxZ8EClGJKe6odj6AmP1TrV2l9/Bmj X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 21d5852b-7310-421c-b7c6-08dd50f4b050 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:01.8751 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kMSRiadyoVeH7Zsn5N+W28jN7jYy4Ou/6YgwOEhH2y9UlrIL8/+7tnaCgnzSNWES5lryVqdZH3TNUQU9NzVomg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-ORIG-GUID: CqHtOcj3dmiGAHsheVtgMdn0h_hdlaiJ X-Proofpoint-GUID: CqHtOcj3dmiGAHsheVtgMdn0h_hdlaiJ X-Authority-Analysis: v=2.4 cv=Fo7//3rq c=1 sm=1 tr=0 ts=67b5efa1 cx=c_pps a=di3315gfm3qlniCp1Rh91A==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=UnscCpd6-9sbi9RN3AUA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Add plumbing for sending vfio-user messages on the control socket. Add initial version negotation on connection. Originally-by: John Johnson Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John Levon --- hw/vfio-user/common.c | 495 ++++++++++++++++++++++++++++++++++++++ hw/vfio-user/common.h | 9 + hw/vfio-user/pci.c | 18 +- hw/vfio-user/protocol.h | 62 +++++ hw/vfio-user/trace-events | 2 + 5 files changed, 584 insertions(+), 2 deletions(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 2c4ee14ede..634f95f2e3 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -20,15 +20,21 @@ #include "io/channel-socket.h" #include "io/channel-util.h" #include "qapi/error.h" +#include "qobject/qdict.h" +#include "qobject/qjson.h" +#include "qobject/qstring.h" +#include "qobject/qnum.h" #include "qemu/error-report.h" #include "qemu/lockable.h" #include "qemu/main-loop.h" #include "qemu/sockets.h" #include "system/iothread.h" +static int wait_time = 5000; /* wait up to 5 sec for busy servers */ static IOThread *vfio_user_iothread; static void vfio_user_shutdown(VFIOUserProxy *proxy); +static int vfio_user_send_qio(VFIOUserProxy *proxy, VFIOUserMsg *msg); static VFIOUserMsg *vfio_user_getmsg(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds); static VFIOUserFDs *vfio_user_getfds(int numfds); @@ -36,9 +42,16 @@ static void vfio_user_recycle(VFIOUserProxy *proxy, VFIOUserMsg *msg); static void vfio_user_recv(void *opaque); static int vfio_user_recv_one(VFIOUserProxy *proxy); +static void vfio_user_send(void *opaque); +static int vfio_user_send_one(VFIOUserProxy *proxy); static void vfio_user_cb(void *opaque); static void vfio_user_request(void *opaque); +static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg); +static void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize); +static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags); static inline void vfio_user_set_error(VFIOUserHdr *hdr, uint32_t err) { @@ -57,6 +70,35 @@ static void vfio_user_shutdown(VFIOUserProxy *proxy) proxy->ctx, NULL, NULL); } +static int vfio_user_send_qio(VFIOUserProxy *proxy, VFIOUserMsg *msg) +{ + VFIOUserFDs *fds = msg->fds; + struct iovec iov = { + .iov_base = msg->hdr, + .iov_len = msg->hdr->size, + }; + size_t numfds = 0; + int ret, *fdp = NULL; + Error *local_err = NULL; + + if (fds != NULL && fds->send_fds != 0) { + numfds = fds->send_fds; + fdp = fds->fds; + } + + ret = qio_channel_writev_full(proxy->ioc, &iov, 1, fdp, numfds, 0, + &local_err); + + if (ret == -1) { + vfio_user_set_error(msg->hdr, EIO); + vfio_user_shutdown(proxy); + error_report_err(local_err); + } + trace_vfio_user_send_write(msg->hdr->id, ret); + + return ret; +} + static VFIOUserMsg *vfio_user_getmsg(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds) { @@ -97,6 +139,7 @@ static void vfio_user_recycle(VFIOUserProxy *proxy, VFIOUserMsg *msg) msg->hdr = NULL; msg->fds = NULL; msg->complete = false; + msg->pending = false; QTAILQ_INSERT_HEAD(&proxy->free, msg, next); } @@ -391,6 +434,54 @@ err: return -1; } +/* + * Send messages from outgoing queue when the socket buffer has space. + * If we deplete 'outgoing', remove ourselves from the poll list. + */ +static void vfio_user_send(void *opaque) +{ + VFIOUserProxy *proxy = opaque; + + QEMU_LOCK_GUARD(&proxy->lock); + + if (proxy->state == VFIO_PROXY_CONNECTED) { + while (!QTAILQ_EMPTY(&proxy->outgoing)) { + if (vfio_user_send_one(proxy) < 0) { + return; + } + } + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, + vfio_user_recv, NULL, NULL, proxy); + } +} + +/* + * Send a single message. + * + * Sent async messages are freed, others are moved to pending queue. + */ +static int vfio_user_send_one(VFIOUserProxy *proxy) +{ + VFIOUserMsg *msg; + int ret; + + msg = QTAILQ_FIRST(&proxy->outgoing); + ret = vfio_user_send_qio(proxy, msg); + if (ret < 0) { + return ret; + } + + QTAILQ_REMOVE(&proxy->outgoing, msg, next); + if (msg->type == VFIO_MSG_ASYNC) { + vfio_user_recycle(proxy, msg); + } else { + QTAILQ_INSERT_TAIL(&proxy->pending, msg, next); + msg->pending = true; + } + + return 0; +} + static void vfio_user_cb(void *opaque) { VFIOUserProxy *proxy = opaque; @@ -452,6 +543,119 @@ static void vfio_user_request(void *opaque) } } +/* + * Messages are queued onto the proxy's outgoing list. + * + * It handles 3 types of messages: + * + * async messages - replies and posted writes + * + * There will be no reply from the server, so message + * buffers are freed after they're sent. + * + * nowait messages - map/unmap during address space transactions + * + * These are also sent async, but a reply is expected so that + * vfio_wait_reqs() can wait for the youngest nowait request. + * They transition from the outgoing list to the pending list + * when sent, and are freed when the reply is received. + * + * wait messages - all other requests + * + * The reply to these messages is waited for by their caller. + * They also transition from outgoing to pending when sent, but + * the message buffer is returned to the caller with the reply + * contents. The caller is responsible for freeing these messages. + * + * As an optimization, if the outgoing list and the socket send + * buffer are empty, the message is sent inline instead of being + * added to the outgoing list. The rest of the transitions are + * unchanged. + * + * returns 0 if the message was sent or queued + * returns -1 on send error + */ +static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg) +{ + int ret; + + /* + * Unsent outgoing msgs - add to tail + */ + if (!QTAILQ_EMPTY(&proxy->outgoing)) { + QTAILQ_INSERT_TAIL(&proxy->outgoing, msg, next); + return 0; + } + + /* + * Try inline - if blocked, queue it and kick send poller + */ + if (proxy->flags & VFIO_PROXY_FORCE_QUEUED) { + ret = QIO_CHANNEL_ERR_BLOCK; + } else { + ret = vfio_user_send_qio(proxy, msg); + } + if (ret == QIO_CHANNEL_ERR_BLOCK) { + QTAILQ_INSERT_HEAD(&proxy->outgoing, msg, next); + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, + vfio_user_recv, proxy->ctx, + vfio_user_send, proxy); + return 0; + } + if (ret == -1) { + return ret; + } + + /* + * Sent - free async, add others to pending + */ + if (msg->type == VFIO_MSG_ASYNC) { + vfio_user_recycle(proxy, msg); + } else { + QTAILQ_INSERT_TAIL(&proxy->pending, msg, next); + msg->pending = true; + } + + return 0; +} + +static void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize) +{ + VFIOUserMsg *msg; + int ret; + + if (hdr->flags & VFIO_USER_NO_REPLY) { + error_printf("vfio_user_send_wait on async message\n"); + vfio_user_set_error(hdr, EINVAL); + return; + } + + qemu_mutex_lock(&proxy->lock); + + msg = vfio_user_getmsg(proxy, hdr, fds); + msg->id = hdr->id; + msg->rsize = rsize ? rsize : hdr->size; + msg->type = VFIO_MSG_WAIT; + + ret = vfio_user_send_queued(proxy, msg); + + if (ret == 0) { + while (!msg->complete) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + VFIOUserMsgQ *list; + + list = msg->pending ? &proxy->pending : &proxy->outgoing; + QTAILQ_REMOVE(list, msg, next); + vfio_user_set_error(hdr, ETIMEDOUT); + break; + } + } + } + vfio_user_recycle(proxy, msg); + + qemu_mutex_unlock(&proxy->lock); +} static QLIST_HEAD(, VFIOUserProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -480,6 +684,15 @@ VFIOUserProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) proxy = g_malloc0(sizeof(VFIOUserProxy)); proxy->sockname = g_strdup_printf("unix:%s", sockname); proxy->ioc = ioc; + + /* init defaults */ + proxy->max_xfer_size = VFIO_USER_DEF_MAX_XFER; + proxy->max_send_fds = VFIO_USER_DEF_MAX_FDS; + proxy->max_dma = VFIO_USER_DEF_MAP_MAX; + proxy->dma_pgsizes = VFIO_USER_DEF_PGSIZE; + proxy->max_bitmap = VFIO_USER_DEF_MAX_BITMAP; + proxy->migr_pgsize = VFIO_USER_DEF_PGSIZE; + proxy->flags = VFIO_PROXY_CLIENT; proxy->state = VFIO_PROXY_CONNECTED; @@ -577,3 +790,285 @@ void vfio_user_disconnect(VFIOUserProxy *proxy) g_free(proxy->sockname); g_free(proxy); } + +static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags) +{ + static uint16_t next_id; + + hdr->id = qatomic_fetch_inc(&next_id); + hdr->command = cmd; + hdr->size = size; + hdr->flags = (flags & ~VFIO_USER_TYPE) | VFIO_USER_REQUEST; + hdr->error_reply = 0; +} + +struct cap_entry { + const char *name; + bool (*check)(VFIOUserProxy *proxy, QObject *qobj, Error **errp); +}; + +static bool caps_parse(VFIOUserProxy *proxy, QDict *qdict, + struct cap_entry caps[], Error **errp) +{ + QObject *qobj; + struct cap_entry *p; + + for (p = caps; p->name != NULL; p++) { + qobj = qdict_get(qdict, p->name); + if (qobj != NULL) { + if (!p->check(proxy, qobj, errp)) { + return false; + } + qdict_del(qdict, p->name); + } + } + + /* warning, for now */ + if (qdict_size(qdict) != 0) { + warn_report("spurious capabilities"); + } + return true; +} + +static bool check_migr_pgsize(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t pgsize; + + if (qn == NULL || !qnum_get_try_uint(qn, &pgsize)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_PGSIZE); + return false; + } + + /* must be larger than default */ + if (pgsize & (VFIO_USER_DEF_PGSIZE - 1)) { + error_setg(errp, "pgsize 0x%"PRIx64" too small", pgsize); + return false; + } + + proxy->migr_pgsize = pgsize; + return true; +} + +static bool check_bitmap(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t bitmap_size; + + if (qn == NULL || !qnum_get_try_uint(qn, &bitmap_size)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_BITMAP); + return false; + } + + /* can only lower it */ + if (bitmap_size > VFIO_USER_DEF_MAX_BITMAP) { + error_setg(errp, "%s too large", VFIO_USER_CAP_MAX_BITMAP); + return false; + } + + proxy->max_bitmap = bitmap_size; + return true; +} + +static struct cap_entry caps_migr[] = { + { VFIO_USER_CAP_PGSIZE, check_migr_pgsize }, + { VFIO_USER_CAP_MAX_BITMAP, check_bitmap }, + { NULL } +}; + +static bool check_max_fds(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t max_send_fds; + + if (qn == NULL || !qnum_get_try_uint(qn, &max_send_fds) || + max_send_fds > VFIO_USER_MAX_MAX_FDS) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return false; + } + proxy->max_send_fds = max_send_fds; + return true; +} + +static bool check_max_xfer(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t max_xfer_size; + + if (qn == NULL || !qnum_get_try_uint(qn, &max_xfer_size) || + max_xfer_size > VFIO_USER_MAX_MAX_XFER) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_XFER); + return false; + } + proxy->max_xfer_size = max_xfer_size; + return true; +} + +static bool check_pgsizes(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t pgsizes; + + if (qn == NULL || !qnum_get_try_uint(qn, &pgsizes)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_PGSIZES); + return false; + } + + /* must be larger than default */ + if (pgsizes & (VFIO_USER_DEF_PGSIZE - 1)) { + error_setg(errp, "pgsize 0x%"PRIx64" too small", pgsizes); + return false; + } + + proxy->dma_pgsizes = pgsizes; + return true; +} + +static bool check_max_dma(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t max_dma; + + if (qn == NULL || !qnum_get_try_uint(qn, &max_dma)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAP_MAX); + return false; + } + + /* can only lower it */ + if (max_dma > VFIO_USER_DEF_MAP_MAX) { + error_setg(errp, "%s too large", VFIO_USER_CAP_MAP_MAX); + return false; + } + + proxy->max_dma = max_dma; + return true; +} + +static bool check_migr(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return true; + } + return caps_parse(proxy, qdict, caps_migr, errp); +} + +static struct cap_entry caps_cap[] = { + { VFIO_USER_CAP_MAX_FDS, check_max_fds }, + { VFIO_USER_CAP_MAX_XFER, check_max_xfer }, + { VFIO_USER_CAP_PGSIZES, check_pgsizes }, + { VFIO_USER_CAP_MAP_MAX, check_max_dma }, + { VFIO_USER_CAP_MIGR, check_migr }, + { NULL } +}; + +static bool check_cap(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL) { + error_setg(errp, "malformed %s", VFIO_USER_CAP); + return false; + } + return caps_parse(proxy, qdict, caps_cap, errp); +} + +static struct cap_entry ver_0_0[] = { + { VFIO_USER_CAP, check_cap }, + { NULL } +}; + +static bool caps_check(VFIOUserProxy *proxy, int minor, const char *caps, + Error **errp) +{ + QObject *qobj; + QDict *qdict; + bool ret; + + qobj = qobject_from_json(caps, NULL); + if (qobj == NULL) { + error_setg(errp, "malformed capabilities %s", caps); + return false; + } + qdict = qobject_to(QDict, qobj); + if (qdict == NULL) { + error_setg(errp, "capabilities %s not an object", caps); + qobject_unref(qobj); + return false; + } + ret = caps_parse(proxy, qdict, ver_0_0, errp); + + qobject_unref(qobj); + return ret; +} + +static GString *caps_json(void) +{ + QDict *dict = qdict_new(); + QDict *capdict = qdict_new(); + QDict *migdict = qdict_new(); + GString *str; + + qdict_put_int(migdict, VFIO_USER_CAP_PGSIZE, VFIO_USER_DEF_PGSIZE); + qdict_put_int(migdict, VFIO_USER_CAP_MAX_BITMAP, VFIO_USER_DEF_MAX_BITMAP); + qdict_put_obj(capdict, VFIO_USER_CAP_MIGR, QOBJECT(migdict)); + + qdict_put_int(capdict, VFIO_USER_CAP_MAX_FDS, VFIO_USER_MAX_MAX_FDS); + qdict_put_int(capdict, VFIO_USER_CAP_MAX_XFER, VFIO_USER_DEF_MAX_XFER); + qdict_put_int(capdict, VFIO_USER_CAP_PGSIZES, VFIO_USER_DEF_PGSIZE); + qdict_put_int(capdict, VFIO_USER_CAP_MAP_MAX, VFIO_USER_DEF_MAP_MAX); + + qdict_put_obj(dict, VFIO_USER_CAP, QOBJECT(capdict)); + + str = qobject_to_json(QOBJECT(dict)); + qobject_unref(dict); + return str; +} + +bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp) +{ + g_autofree VFIOUserVersion *msgp = NULL; + GString *caps; + char *reply; + int size, caplen; + + caps = caps_json(); + caplen = caps->len + 1; + size = sizeof(*msgp) + caplen; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_VERSION, size, 0); + msgp->major = VFIO_USER_MAJOR_VER; + msgp->minor = VFIO_USER_MINOR_VER; + memcpy(&msgp->capabilities, caps->str, caplen); + g_string_free(caps, true); + trace_vfio_user_version(msgp->major, msgp->minor, msgp->capabilities); + + vfio_user_send_wait(proxy, &msgp->hdr, NULL, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + error_setg_errno(errp, msgp->hdr.error_reply, "version reply"); + return false; + } + + if (msgp->major != VFIO_USER_MAJOR_VER || + msgp->minor > VFIO_USER_MINOR_VER) { + error_setg(errp, "incompatible server version"); + return false; + } + + reply = msgp->capabilities; + if (reply[msgp->hdr.size - sizeof(*msgp) - 1] != '\0') { + error_setg(errp, "corrupt version reply"); + return false; + } + + if (!caps_check(proxy, msgp->minor, reply, errp)) { + return false; + } + + trace_vfio_user_version(msgp->major, msgp->minor, msgp->capabilities); + return true; +} diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 71c0dea82d..9f4243a67d 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -35,6 +35,7 @@ typedef struct VFIOUserMsg { uint32_t id; QemuCond cv; bool complete; + bool pending; enum msg_type type; } VFIOUserMsg; @@ -54,6 +55,12 @@ typedef struct VFIOUserProxy { struct QIOChannel *ioc; void (*request)(void *opaque, VFIOUserMsg *msg); void *req_arg; + uint64_t max_xfer_size; + uint64_t max_send_fds; + uint64_t max_dma; + uint64_t dma_pgsizes; + uint64_t max_bitmap; + uint64_t migr_pgsize; int flags; QemuCond close_cv; AioContext *ctx; @@ -76,6 +83,7 @@ typedef struct VFIOUserProxy { /* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_FORCE_QUEUED 0x4 typedef struct VFIODevice VFIODevice; @@ -84,5 +92,6 @@ void vfio_user_disconnect(VFIOUserProxy *proxy); void vfio_user_set_handler(VFIODevice *vbasedev, void (*handler)(void *opaque, VFIOUserMsg *msg), void *reqarg); +bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp); #endif /* VFIO_USER_COMMON_H */ diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index fb6c872768..69806eb863 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -36,6 +36,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) struct VFIOUserPCIDevice { VFIOPCIDevice device; char *sock_name; + bool send_queued; /* all sends are queued */ }; /* @@ -95,6 +96,14 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->proxy = proxy; vfio_user_set_handler(vbasedev, vfio_user_pci_process_req, vdev); + if (udev->send_queued) { + proxy->flags |= VFIO_PROXY_FORCE_QUEUED; + } + + if (!vfio_user_validate_version(proxy, errp)) { + goto error; + } + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->ops = &vfio_user_pci_ops; vbasedev->type = VFIO_DEVICE_TYPE_PCI; @@ -109,9 +118,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) if (!vfio_attach_device_by_iommu_type(TYPE_VFIO_IOMMU_USER, vbasedev->name, vbasedev, as, errp)) { - error_prepend(errp, VFIO_MSG_PREFIX, vbasedev->name); - return; + goto error; } + + return; + +error: + error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } static void vfio_user_instance_init(Object *obj) @@ -154,6 +167,7 @@ static void vfio_user_instance_finalize(Object *obj) static const Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), + DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), }; static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index d23877c958..5de5b2030c 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -51,4 +51,66 @@ enum vfio_user_command { #define VFIO_USER_NO_REPLY 0x10 #define VFIO_USER_ERROR 0x20 + +/* + * VFIO_USER_VERSION + */ +typedef struct { + VFIOUserHdr hdr; + uint16_t major; + uint16_t minor; + char capabilities[]; +} VFIOUserVersion; + +#define VFIO_USER_MAJOR_VER 0 +#define VFIO_USER_MINOR_VER 0 + +#define VFIO_USER_CAP "capabilities" + +/* "capabilities" members */ +#define VFIO_USER_CAP_MAX_FDS "max_msg_fds" +#define VFIO_USER_CAP_MAX_XFER "max_data_xfer_size" +#define VFIO_USER_CAP_PGSIZES "pgsizes" +#define VFIO_USER_CAP_MAP_MAX "max_dma_maps" +#define VFIO_USER_CAP_MIGR "migration" + +/* "migration" members */ +#define VFIO_USER_CAP_PGSIZE "pgsize" +#define VFIO_USER_CAP_MAX_BITMAP "max_bitmap_size" + +/* + * Max FDs mainly comes into play when a device supports multiple interrupts + * where each ones uses an eventfd to inject it into the guest. + * It is clamped by the the number of FDs the qio channel supports in a + * single message. + */ +#define VFIO_USER_DEF_MAX_FDS 8 +#define VFIO_USER_MAX_MAX_FDS 16 + +/* + * Max transfer limits the amount of data in region and DMA messages. + * Region R/W will be very small (limited by how much a single instruction + * can process) so just use a reasonable limit here. + */ +#define VFIO_USER_DEF_MAX_XFER (1024 * 1024) +#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) + +/* + * Default pagesizes supported is 4k. + */ +#define VFIO_USER_DEF_PGSIZE 4096 + +/* + * Default max number of DMA mappings is stolen from the + * linux kernel "dma_entry_limit" + */ +#define VFIO_USER_DEF_MAP_MAX 65535 + +/* + * Default max bitmap size is also take from the linux kernel, + * where usage of signed ints limits the VA range to 2^31 bytes. + * Dividing that by the number of bits per byte yields 256MB + */ +#define VFIO_USER_DEF_MAX_BITMAP (256 * 1024 * 1024) + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events index 89d6c11c4c..7a3645024f 100644 --- a/hw/vfio-user/trace-events +++ b/hw/vfio-user/trace-events @@ -4,3 +4,5 @@ vfio_user_recv_hdr(const char *name, uint16_t id, uint16_t cmd, uint32_t size, uint32_t flags) " (%s) id 0x%x cmd 0x%x size 0x%x flags 0x%x" vfio_user_recv_read(uint16_t id, int read) " id 0x%x read 0x%x" vfio_user_recv_request(uint16_t cmd) " command 0x%x" +vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" +vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s" From patchwork Wed Feb 19 14:48:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982407 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5A630C021AA for ; Wed, 19 Feb 2025 14:54:10 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPH-00046d-CZ; Wed, 19 Feb 2025 09:50:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklP9-0003fJ-Ro; Wed, 19 Feb 2025 09:50:28 -0500 Received: from mx0b-002c1b01.pphosted.com ([148.163.155.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklP7-0007Z7-GJ; Wed, 19 Feb 2025 09:50:27 -0500 Received: from pps.filterd (m0127841.ppops.net [127.0.0.1]) by mx0b-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBEuNi022547; Wed, 19 Feb 2025 06:50:22 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=ryOwBWkEr0cGzO0zYpHTYTJrOd2B666yTBCakbU1S uY=; b=Fqv+bDmsL6Op5sP2px5xDMVL0iKeaQFEamoKE+YYjDwCeotLFGep1VCVr yplE9xl6iIP897dUlGh+cUAdID9n/SN/WGexefBoSxI5S+uOWi45xe7DvSWAI4b4 gvS6jd3VC3M4EDhIMPwMWqFdVIhQy50sPP+WKNd4LMYf0sy5Ojv/NryjLEYGMSgp Y0HS07AE9lZ5f3YMJqhxh2BtWdFLmQji/PyodKW2r0aKt+OZTj9vIP/lt3FQZ4oB kyf1kk0319fChPh4sZ5F/MgG/Pc2dS751LLviRRigaFFXKoH1wSxIQRjgLRy5yDl apGv/H4G+fpOFmBt1PBB0Luw5XEAw== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2040.outbound.protection.outlook.com [104.47.70.40]) by mx0b-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4b9hy5d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:21 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=xSuICd9IavAThBCsk7ZdHpoN6Yf/lCyOyIZstxtuQ2xrG3mFwDBDqQ6AjbdmhOik6KpBDgdV2ro0ZTW1JcHPPAxASh+uV/QyDJGPUpSZh8G9t+JTMbUUDG1dutJw8esxOr7QXQOarx0OBx6klQ2NM93TyKKe2nembpuE94NOVaM+947TQJpIthnX/zoVjBllmOnZ6N4lW7n5JUm5SZZgLubFMAIttyO+khGlI+mcSBgY6gNCmyDX1CCqda/yB4mUNcXnSFMHvuT6Kb8C4KJEYm2/Vo0660sUf9x6reb/k7ZR6pLOXtKgfBd3HJpud+unrj/FLOTm5IdorECbWCQg5w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ryOwBWkEr0cGzO0zYpHTYTJrOd2B666yTBCakbU1SuY=; b=qJDAVHuwJZRqATF7fgfiic+KGNTlkNYreEFl5ZlNqL6sM7QRQDzslPUAUYyYr2wjTvI7pT3KuwPCScdEjpwK6HEVCIos9x9MFW1E5yIkwabNy4iHdxpcFCM3wCEzBjJD+H6O4GDXuSWeioQbKecY9pLZhIeT502MbmEYr+Hp9tU1oymhsh9rpybK4DkFueq/yfjciMbhpFM5F0mYDpg+GQg9yKpe0RYAKQr7pxJxRGFf2XZ3Y3Ma+WmBkpUgulZAf1DZ8XA0c7aTCB03Fe/OUrXauKxTQD6MmnnEhHqCobb+xVZgxzCsZfFa5O65lEHVbgDKpl0StBwjZKGrrV/4kA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ryOwBWkEr0cGzO0zYpHTYTJrOd2B666yTBCakbU1SuY=; b=lZFn5+0eabTLqeZC0dwPtBPGeYrasM522aO8v5ojBTbf2Bm4nn3CzaZrY11sxYXN6wYxGVCIgN3MzCpTn5lF8xZBmM0BhsC+AlimEkWIdtPj/1ZYzy+Mdv0ltInqa8Uvu0VU1uJuh2/dQV2VvXWjzZy0PEwPwnc5+t3Hi3nIBYNM7/km59/5/tJfhvOOcUtWWLRWZhWqZxeDIOS9r6smDqs7zq2xYBKPPEj0K0WimGV94JgtSc9JkzzAudH4ZiN9rbQ8tP93L0KNTWrjiBRLKqtHawFoAmJ27cupDJlTl90BIglyPvoIyzqEcGj0WRRjoygl3XZaKZLyOGo02UVneA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:05 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:05 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 16/28] vfio-user: implement VFIO_USER_DEVICE_GET_INFO Date: Wed, 19 Feb 2025 15:48:46 +0100 Message-Id: <20250219144858.266455-17-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 8b4e1796-7b8c-4dde-7402-08dd50f4b281 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: gIYqjAQRRMegzxxbIGFRWYeSFttVRM8IwTjXAwp5XP58T2mvxBp1sgEnvZol9nFq1kfZH4NTaKc89wCnAgQM0oj54FmK0DUFf8xc1KKyKHxoSPDElmBiz5SRIpFos/ZCzBXmnCNAwG7Poh5tz/YmwDwDLGndnnGxYQhV1MuX1oZymM0R65LZcnwQu0oonF//prEEw0QP+guPNBwwwhkMOYI2aZqFVBZCxM5acZ4epj2DxPnpJQWGbTvBxOV1B96kK+Ol4g4sq8Rib8ejXCuX4jElwrVVGNcGX2Gw/B5BlyLwKjsAivCYnwz7GyixPsvS0J3RE70RWT0V+0Pae0d7vRwQlElToEWF/iB3tDnPXI7g7BovaO1aSAuG1slSyhXUoKuZxMR+Rrr2hMorCASbvc4j36txGlg4X1vYATuBrSFA4pxsRwyayaE4HXnEHlucSIViqPNwazQ+L33vJeCCd8/VEHYxY1poOGsHneAsreB61kt5RUd14cVxcYAHQET/U3c1FTEqrGQ0C1d0jKo5v6Oc6ZFB48W2DswfTHBAqeiXoDGlaDjCb9p9Nd2rpbOefqthmQW/ZrbSUPy7j9T0+JDyAbf1pjBNRqxW+lIRgHMyo/5U5R8ev43ZXiUUD+6bMkEvDV12sX9k3iTOzWWpa2y4FQbT36smZWYP192dmZWLb32CQzKz+BIBuDv/TIHJQe01lV6n0W9eZtqh0bhjUs/2MzxtGThU/nQ7zkOQ56VC7CU5S2LS2tJ8hkGDL6gx0inR9smpAfTSxAYDlFRvvNrFnVFku97h4FciWReaFQgbxQzMmGuM8ZM7DoxGhViXodTzZ22UhOMhowySqkCq5cz+CfUlWv+xoSkh/VPlGHam8vYifAgJ+8FlP7AJvckiZ0pvlQRb1LUku5f73yp8Dnyz3UYrhontKYQlNQWUfatoZOQUAjOfF0kJLfirOScqVZXXPpgvqPXDimO5JiMkhwb7+2C3YKWWi6965QriAmWD2s8j2GV7qorOsvT2+IjJLp8YSHVXylRhKnMOfjlm6VTfyuenu8dyMIF83X2OPBsio1X8QKbx4xZPJMbfHSejmVNS1y+c6ycT4BW/MzqOHezLD2HG2ubfttk2SKRe+1I8f18EtXYRAn9usg5Xk4WQYJ7jTQfa5ji5G8tir9hLtdPrOmaRpTNO5kyjMOAJgJyhG422dEmtJKmtFIdtnE6tRF+kUbe5+ihZMZnSegI4A0jmVv1kRNNQbXE9spt5zu+03APAZ4f8txYoPvOAhxORnjknxiIKQ4gMQh/2cnmwciPQQfuhqvw/+28XeIh6nWNy3pqus3fY6Ga6IhApkREr/atmVVVjgndSIFycz5AEIpHlIKcMEdIUZd3IyeN9/H0NyWxb9z+uoF2N+HLn3wRk X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 1nt9gmwReRrjsYdO+sWMveCv/bYnpe5UVuOrhrlnuttZlbIOAia0NRG33R8fnRok1zCrcuqvyxxBv+dBYaKP/N78Oi0jXf8NMNCW/7HfAsCUZV4mJxsSJC6rby6IdagoY+4hC67esKsMikApfjgSUTTMm+zNEUm6h4XBB4FYttkUv4ipahoil7w3ftGr1hbswdRBPwoxhUzcUvlCz08ZCaPnfFVyGkvf8L1KGl0nEh9ImYI80fSDHxPRj1wAOVhmwoycMaHu8Sge5cyUgOs1KkRq1UXxOerU3rIxfuoHtI2okGOA/B16cJp1dx6Pm0QBujnwKzUNAWlCoyHA7Y5d01bubMpB9hG75V9tQsr91kgu+bvQUcx3/2rIZIcoca8r89EOTdEJx2DPLuJw/Mc4uD1zW68KdXnQXXqjshiNtp9gOqLvt2tNAEueoERbQvWMJ06yu867dLhadyK5rLNAklpzC5LRhX+qV2bi9NsUay67/OrFJULm/wbsNxh5s1L5xI4vhJNj1XHE5bNGUqOgxtEg/rplLn8CcxHSBT2pWf3NfGI/FcYURTYue8JVBYXZ5WDIMlVwXpBAGS3CJSYLeXc0QrltvJlcktkXWAySUOkmDjgXxWxDDHrOkKPEHBmy8ZEIUBEp7DDMKffTvzBo+lsYb18+EWQvusJbM0LLYkONW9/oxO20N3o+p4LOiQdSEC7VDMITcuZ6R7OKVayxgHgRIJsAC+u2UctEPyeavTIVYSu69VmJsi/cNNxICagm7VjVLexKmpkFZByKLa0qvxP5pN49wHNvaEHPS/249tlqeFQVd1n2o1umma/eCPXjhliFgIil9lmol1W5n2+x4ogpkBa5awzahfhPFcX1AP4XyqUFm2m1DcCE4PbZB1iYs4biqEfvW8Awix2IWFymwodjO3A7e6Z1jmab+hrDuQIlk4bRVG7GXP283uZjkvDlZpfEMcFUUtrbgtH6disBH2iO524O1GEjHZw56A7B6R0P72dD42sKF1wLOz5wrwGbuuW5WbFPnOVMk00nSP+/Xby3do4KOIyKzqL+nM3Nm1Sh8gBqPMmuYJGULrrw8NuBAfmo5/Ua/COPUFWnrV9ToItpogAQDJ/QyhJkpZy1qbdkOR8+hxH4Efvdq1XWOMCiLKi0urZxzhTrqM4DGqjVIdNtKCoRbsJ4/l6+GYfwpmk+a5Y0oL1r4xgjhIphWvA+X2nZUpAie8kiN4ujDTFLDJIJcp90MkPS+1Jh2AAAQQvh3kmknqwiEw51q88Y9N+mSkbkoOOdUcWgiULE5TrfQdRVTIxynfwCPEiES7QX0wFZWAyWxoZLMJH/7kxrHIxo6jOxtoc8wAlGeDwFIN4XfniPwdZ+240i5XiAiVgArakkE/MTu8YkfcKDEBLztpDFdvgNG7berC+9fClsnKtozgt6C142Ewz6g3WHvM9EQEJcFoyvyPaxcPsxnZ1Swa1lBsZdKdohZXsiBMzXSj1CdPg2s//zD3gW3/M15YnFzSiFYSfhS61QK2oB7NmBXMdRDHqBSd/3a6XQ5E3Dj0osRipHOk7f+v1VFUyjW7CiFH6DZz28g8IurD/OxXQMLCoX X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8b4e1796-7b8c-4dde-7402-08dd50f4b281 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:05.4794 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: w7d3l+cDBAWEIFb83KNDyZall0UtE7H2ThD5bY1+GMvLxztDqyVAzXPjAyupxkpqI+Xt08ucJeOOlYiZ/CHKYg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: Dc41D4D_I3yjh-4qinkryyGFf1mmHN2X X-Proofpoint-ORIG-GUID: Dc41D4D_I3yjh-4qinkryyGFf1mmHN2X X-Authority-Analysis: v=2.4 cv=HMQ5Fptv c=1 sm=1 tr=0 ts=67b5efad cx=c_pps a=bqH6H/OQt14Rv/FmpY1ebg==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=1OX1_22eDB2b9HH4iF4A:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.155.12; envelope-from=john.levon@nutanix.com; helo=mx0b-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Add support for getting basic device information. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 34 ++++++++++++++++++++++++++++++++++ hw/vfio-user/common.h | 1 + hw/vfio-user/container.c | 10 +++++++++- hw/vfio-user/protocol.h | 12 ++++++++++++ hw/vfio-user/trace-events | 1 + 5 files changed, 57 insertions(+), 1 deletion(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 634f95f2e3..55d1da4e51 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -30,6 +30,13 @@ #include "qemu/sockets.h" #include "system/iothread.h" +/* + * These are to defend against a malign server trying + * to force us to run out of memory. + */ +#define VFIO_USER_MAX_REGIONS 100 +#define VFIO_USER_MAX_IRQS 50 + static int wait_time = 5000; /* wait up to 5 sec for busy servers */ static IOThread *vfio_user_iothread; @@ -1072,3 +1079,30 @@ bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp) trace_vfio_user_version(msgp->major, msgp->minor, msgp->capabilities); return true; } + +int vfio_user_get_info(VFIOUserProxy *proxy, struct vfio_device_info *info) +{ + VFIOUserDeviceInfo msg; + uint32_t argsz = sizeof(msg) - sizeof(msg.hdr); + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0); + msg.argsz = argsz; + + vfio_user_send_wait(proxy, &msg.hdr, NULL, 0); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + trace_vfio_user_get_info(msg.num_regions, msg.num_irqs); + + memcpy(info, &msg.argsz, argsz); + + /* defend against a malicious server */ + if (info->num_regions > VFIO_USER_MAX_REGIONS || + info->num_irqs > VFIO_USER_MAX_IRQS) { + error_printf("%s: invalid reply\n", __func__); + return -EINVAL; + } + + return 0; +} diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 9f4243a67d..11a80d4e7c 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -93,5 +93,6 @@ void vfio_user_set_handler(VFIODevice *vbasedev, void (*handler)(void *opaque, VFIOUserMsg *msg), void *reqarg); bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp); +int vfio_user_get_info(VFIOUserProxy *proxy, struct vfio_device_info *info); #endif /* VFIO_USER_COMMON_H */ diff --git a/hw/vfio-user/container.c b/hw/vfio-user/container.c index 7b1c202517..c079d6f89b 100644 --- a/hw/vfio-user/container.c +++ b/hw/vfio-user/container.c @@ -16,6 +16,7 @@ #include "exec/ram_addr.h" #include "hw/hw.h" #include "hw/vfio/pci.h" +#include "hw/vfio-user/common.h" #include "hw/vfio-user/container.h" #include "qemu/error-report.h" #include "qemu/range.h" @@ -152,7 +153,14 @@ static void vfio_disconnect_user_container(VFIOUserContainer *container) static bool vfio_user_get_device(VFIOUserContainer *container, VFIODevice *vbasedev, Error **errp) { - struct vfio_device_info info = { 0 }; + struct vfio_device_info info = { .argsz = sizeof(info) }; + int ret; + + ret = vfio_user_get_info(vbasedev->proxy, &info); + if (ret) { + error_setg_errno(errp, -ret, "get info failure"); + return ret; + } vbasedev->fd = -1; diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index 5de5b2030c..5f9ef1768f 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -113,4 +113,16 @@ typedef struct { */ #define VFIO_USER_DEF_MAX_BITMAP (256 * 1024 * 1024) +/* + * VFIO_USER_DEVICE_GET_INFO + * imported from struct vfio_device_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t num_regions; + uint32_t num_irqs; +} VFIOUserDeviceInfo; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events index 7a3645024f..6b06a3ed82 100644 --- a/hw/vfio-user/trace-events +++ b/hw/vfio-user/trace-events @@ -6,3 +6,4 @@ vfio_user_recv_read(uint16_t id, int read) " id 0x%x read 0x%x" vfio_user_recv_request(uint16_t cmd) " command 0x%x" vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s" +vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d" From patchwork Wed Feb 19 14:48:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982403 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7B914C021AA for ; Wed, 19 Feb 2025 14:53:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPD-0003sM-Ra; Wed, 19 Feb 2025 09:50:31 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPA-0003fV-3P; Wed, 19 Feb 2025 09:50:28 -0500 Received: from mx0b-002c1b01.pphosted.com ([148.163.155.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklP7-0007Z0-Gy; Wed, 19 Feb 2025 09:50:27 -0500 Received: from pps.filterd (m0127841.ppops.net [127.0.0.1]) by mx0b-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBEuNj022547; Wed, 19 Feb 2025 06:50:22 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=DkbgFufOS5IH+aPw9Pa3u+L+Aso3ErcJHh7tPyw9n Vg=; b=Iz8XpC/no/9fiKhXAVElyuqZSWXL71RYP9dz5/usmCW1i4Kkjcbtp3iK3 leAUklqL9tqUPddtU3tn43cQ4DaV6IwKQpheTb0PuNfUV0d++9kL6GZ54Xx+L5Ho 2ucZXa6mLo/1E9+HIJpExSaR07YUeS6nRvXawn8hfNSxRB3TT8jbMwjtotv/dl1B YOdwXTyk/QBd5zaO6YSNGVhzUPDriC0zxik0BjL7lQ5ULy1iZM3WLdUQXh3WzPiz 807mmkLsw4j3mZnckTiZnkNleBe8TzrvjMrnZhh6WJClclr2EDrqIZUXu5Dq2KcS V0ps8jZTvIEAM75WFuvSxIzWwE5dQ== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2040.outbound.protection.outlook.com [104.47.70.40]) by mx0b-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4b9hy5d-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=HEoRTR1SbHcg72LDyCUpVjAmdyBMXZiyC9aLnEhcb0opRJDXetS0M8I6HluTCPE7+KaX1e1OcdvGfphgNw4+GaE7cIHWz5JyxYQJCRCfbx2yGWEoRp626XkiT7MLDhs4fFFV2Fgp/6Su7eCkvJBKUVFx6+lkeTBu+yLayMyXk7t/hY/S9OiA1XQ9S9lWyYaKvXhyGpnkhyVli8U3xgQ6YLiXMVVWDkBUZVjHC0NAFBYwj2130OCHLnG6Ht06hzFUK6vrE2A4ard/rqNSFotuLKYhcRE/l3HcNc1jh0KyhZbvFRU4VYLww05k/ZudCUHfz8eBkXavYW6x+Peh6Okvhw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DkbgFufOS5IH+aPw9Pa3u+L+Aso3ErcJHh7tPyw9nVg=; b=cphWG/UF1b4u4DZqob+DbsnqRv13rb4OgV6t4iNEpy0hhHK3EfmqTbZow6LVjiZU3cLBlwXKrFaEUbpIEJ6o/k+nlCqSZQtlsiuviFWDpP0xDgXcVCkGb5WqdSKqG7sy/obvjgH1Kd6RG9xGtjp6Ap1s3ocpbxC7dJSzMfMBoWxpCmAoFpjYirG/XTiuo04oyAvet0kNuPy0zFOOtIwW5bV2/8WXFlGPXEYLSFSD34rc3yJmSMjtka/YslmI2DIhMN4U/h5H//kxSHR8bIn7vXo7uNtxlT9kJ2eaE9RqfW+Rmnt0m+zHDi17icmW1h7hjLyIBrnn/OmbCQcb/oQQjQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DkbgFufOS5IH+aPw9Pa3u+L+Aso3ErcJHh7tPyw9nVg=; b=nIILOxO1idhdT2zmRuQnm16gfGVqfAw1FFALf4Un9L+h12+OqCZVMDzYz1vEbRRCINR7jNJ0KXn+U9F3YWL9fLUYERcyzCLoK3jf+83gEboFH4dPGQNzMIQ/PP0CiFjP5Y1JgCiQES6GqtpXZFpNQIoYnXdYUUJT9Fff/9lrVHM9Qc33G5KCB6GzJJPdOv0jsYmMc4tDn6UEK5/ntIyDbzTUc/kLvtPZWRSh+7lOBpZpP1Kma34Euo4zVLlZ/dqXOBhIocqCfdMMDNIIIRaz5lFccb8Q65E5McsJiZnr3VzGrr2rgC+xri5Tq4HukfCd96un0aMbq6DiLIqaVgpZqA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:09 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:09 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 17/28] vfio-user: implement VFIO_USER_DEVICE_GET_REGION_INFO Date: Wed, 19 Feb 2025 15:48:47 +0100 Message-Id: <20250219144858.266455-18-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 55383ad4-b038-492e-cbbe-08dd50f4b4ae x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: q1QQHZuNRmqQSrb0wpCwEyy4yhHe1G+szfFDWNF8dAsvhQo9nXU0R27+BAvhXyH5RpddJmU2CxIsCjo+8b2ikQLoxXQyQw/AHg6i10plWB0zvE0KiCSotEGDR5JWI/m1OFV8oU4DE45LEpLCDTusnYLqdyAwA+ajnvG5U++OmxsW5p5AsSmg7+GSWhxLMpw/04F1bcsS5fDY1/H1OM1xjOAbfmndNE21NmE8Z7bIIP+pRx+ManU592h7LTN65Lq+lWYZ65IHRV1BIextumk4wJcFo6PcjzutD8FlOk+9drZSOC7bqAbJNwWn4SV8bnWrx49FvKbupILrgBSAuSOBoqiFrEqthaxGiE17bxBGEKK0KSCuvOBa0BYGym67cWJ/r3+HcxzwUX9LmBeajDqDkNqd5vMe0AzSu6k5FfEkjXq1MPx2ePbuc4tDDVCYH1zeb7nafOJvn2WcBmlkNKDhMv80yPmkQtXJ147Iesz5d0c5+S9zMXP72N+CwfQtEw8NBMC7nK12Fa65ZthCyZQWuwIUAsL4Fuw9POXf+lPlvB0hWHoj5Lt8aoDRX/qEW9UttAoDXG+LnMfM5fkz1GiUItT1OGUbuwTLXdXEyoqGpIHnEc+Z6EddeiijrEdxmf1YWTC5yFIYB3JKKmg7GEifOQdQjAqvKqIyh8IiEKqYH0CTxHI97LVZlw21XZ6bw78sOVfyJhjI9JkvrNpgF6VPLZRvA8JIEooUGZXyjaPTNXRe61/MjXxxPSUjAhEC3lvHA3q7Q9DyXI0hLAot7e8+ZiRekFC+8FqYwSJbGC+pJ3OnkG3Divs2pUll6fh3XN5THw2Xl0CmzK8avyGJtpVixBBOzf1OIzbM6K2V1CN+HD3/70i2Ua8VOrgoYuBZuraPd/WehJ+eVhXdKy1qi/rmyHZFC8Udct9RYdkuUamJHN431mh2Ya8kT47z0HaP6nEGRRXZkNxcRTuzE9U7ffi4NyC6IvkeDyRmYGhPdnGe0yJitwO4K09ILF+2peg+AeOw3mchFt0HJj+9f4bq4Oax1/pYf0idFtZ5wbZTF+T/OGpVIEgFLA1YsT9wPyS6JjljgRaU5tAj+44dXr7GCb8blzCazk7Ub/V/L1GKycXZtj/+Onl6I2XDkIIkz9ASa87npTfkwU77yb3RyFitpHJKkqAACaxLrfMHPFocJr5cg2ZjcOgY292HsHWyvm3CtIIxHRa/D/uQSTplmH4olREHUF3R1W8PQaiyuJ4XXz6YZ+gY5SDx5ixKTCbkZgF+aFGuQuEBujNJJYZZaDuJmjOmxUO5Y9mpMEOyNxnA04U1Mz3HCV8HnvM3jflpzmyhdQy22y/jqNYgULwaeAWNWdPpJwfmX/V87Q6XxwGfign0MPR2NryaCqtKU6lDQ81HQ+j7 X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 0COByQiI4MBNzbOGYjo+1B3sGAHKsmJyPvzQZFXxSIY3nw7RbdGrVjHL22DKPJE0HLq98p63YfsVGJI0yKksDYLPmSQLhm3vpP6msYnlwmlZ5c8ByqA4DxJjSOdGdpeH2NVWVyt4eKQ2CKQ7twhNJZwy0TvlVQEQZ1y7FQM1Uh2cFt8/u8XCR0SK3wnSEznfEdwxygI2i/0OAJLkkbmGy4dA4WM8G44VHxJKFzbBLW1ZuNya4zJCJaE5MF3CxtLozvZYqRTDtZr6YFfwOqJfD2tDDh4FsPL0myrDcEojwwGZqCFkkziX/c4kulIF+zNGFCeNtTTphcuooRFoFSaE+b+uL9PbpUGHuF6oFJpzNbUUMBcqmgvM8Nyg7aG9fDdn4hHvMEaf0cp7ivkgyq5hEhlZgG3KmL0O486tqs5i+7lr0l4P4TM741Gy4o1GGPs1sy5LQuEakQJCCG37+HdyLUc1QyrWUwFCFn8qzVBL9NOE4Gr7GH0zoiMSodePU/5RQOdOLnjtB94/ZiXQzVYq0x7zXnjlara90o5zbDSLKANIOvqkKiroyhI2EEkyrLSNg8OGfvz3oGU3vASDGPu03SBf287bii8Po+5rMKig6n/WBXXegErJeu3G+FrC6CVKFdGL9l830pXhwfPw5HcZ3CKlv2Fw+0+UWJFQmcb9jrpf+Nm8KTBEDvNUeo5UZ80LhrXH0FAHFL94d8At4oc1Ar5/bsHJfnHE5/u58hr8OsL8sUSNk+JORpSpzOW/VhLKXcV5KSmKMmT250dvSdOeeHaBDw+qc3uWVMb6ZHKZQbAaTASL520p/PsVhnLn/ai4Ux8OA2dVxO14HamGAIOSiX7c9u2Ww/7/cs1HOGSMednJMYV9iVB4gvIDO28IhP4bJMptfvg8U1UOETVnEocP6bA12gK9UDxABoNDE+vK7Sox+lcaX1vQK0h/31H1EtDt5rb5DRwYLhD6JZ+GFouYCGj42saGoDpKrKyGizg9HfoaEddtXPl7FHGlastdcmogMUnXj/ZyE98mgyHR+dENNM9S9MQ0A5FkxfUNQ86BT8FWpZJ756PBtpZEutTGQO0mVVATCTJsIut/J0ofo7BSu7Paly8KVMn2ObYTjf8zgeutGfVOgwmbEo9oIPhAC/H1wbEp0kpeJaCmPGZk1VI2gf5TkbkpQdqjqfXcdBq9Nox8rMIWZzSEDHWrsx3VHARZH+zyMplJd9XVzMz59Acg/cc3bMgGPCy1k3kOaJeoL7pCEYfK4OXuhUlHUTjBXbTr5+Zlki4y9ybj5m9VaJUda6IwBAQhYmPb68ECJVBdqC2GGlGxq6tlrVLRwtg30kCJQDTOJWc0A9KRdPN5vAS6H9ZplV85rnpXs1HO1wjZqqH4b9Wgr/xGJiV/XToR7t9L91jHtcxHScBxAWbOo5drh9YQuMPlkHWJc9JiL4X1YJ+aZbkrtY5DE0AuQWTlASJdXfVZlg6SE/R6J509cIeq+1wsocgjEJLPvGKmk1qGfkxUE6ekMjmiBvob85tLgKaFs+qNud9D99Sq0yXRxirG3QBgNJWxkj7gUWjHGKRFJxNW5/pS13Ezr5L2+1m6+FS3 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 55383ad4-b038-492e-cbbe-08dd50f4b4ae X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:09.2302 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MQPEDYxqt/Je5DRc7GJ/rgvzGl4p3VTPzT+0zZRh80chosmNSIgC7qtIV52Zt+RlQxBMUSjdpmfSnmgfFOm07w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: 0RjrKGk_Rmfm7_9pHQv96GDnfqU4dl9W X-Proofpoint-ORIG-GUID: 0RjrKGk_Rmfm7_9pHQv96GDnfqU4dl9W X-Authority-Analysis: v=2.4 cv=HMQ5Fptv c=1 sm=1 tr=0 ts=67b5efae cx=c_pps a=bqH6H/OQt14Rv/FmpY1ebg==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=NH85Zxiy9fDlYZs8F5cA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.155.12; envelope-from=john.levon@nutanix.com; helo=mx0b-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Add support for per-region info fds. Unlike kernel vfio, vfio-user can have a separate fd to support mmap() of individual regions; add ->use_regfds as needed to support this difference. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 68 +++++++++++++++++++++++++++++++++++ hw/vfio-user/common.h | 2 ++ hw/vfio-user/pci.c | 2 ++ hw/vfio-user/protocol.h | 14 ++++++++ hw/vfio-user/trace-events | 1 + hw/vfio/ap.c | 2 ++ hw/vfio/ccw.c | 2 ++ hw/vfio/container.c | 7 ++++ hw/vfio/helpers.c | 28 +++++++++++++-- hw/vfio/pci.c | 2 ++ hw/vfio/platform.c | 2 ++ include/hw/vfio/vfio-common.h | 5 ++- 12 files changed, 131 insertions(+), 4 deletions(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 55d1da4e51..7b96d715cb 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -1106,3 +1106,71 @@ int vfio_user_get_info(VFIOUserProxy *proxy, struct vfio_device_info *info) return 0; } + +static int vfio_user_get_region_info(VFIOUserProxy *proxy, + struct vfio_region_info *info, + VFIOUserFDs *fds) +{ + g_autofree VFIOUserRegionInfo *msgp = NULL; + uint32_t size; + + /* data returned can be larger than vfio_region_info */ + if (info->argsz < sizeof(*info)) { + error_printf("vfio_user_get_region_info argsz too small\n"); + return -E2BIG; + } + if (fds != NULL && fds->send_fds != 0) { + error_printf("vfio_user_get_region_info can't send FDs\n"); + return -EINVAL; + } + + size = info->argsz + sizeof(VFIOUserHdr); + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_GET_REGION_INFO, + sizeof(*msgp), 0); + msgp->argsz = info->argsz; + msgp->index = info->index; + + vfio_user_send_wait(proxy, &msgp->hdr, fds, size); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + trace_vfio_user_get_region_info(msgp->index, msgp->flags, msgp->size); + + memcpy(info, &msgp->argsz, info->argsz); + return 0; +} + + +/* + * Socket-based io_ops + */ + +static int vfio_user_io_get_region_info(VFIODevice *vbasedev, + struct vfio_region_info *info, + int *fd) +{ + int ret; + VFIOUserFDs fds = { 0, 1, fd}; + + ret = vfio_user_get_region_info(vbasedev->proxy, info, &fds); + if (ret) { + return ret; + } + + if (info->index > vbasedev->num_regions) { + return -EINVAL; + } + /* cap_offset in valid area */ + if ((info->flags & VFIO_REGION_INFO_FLAG_CAPS) && + (info->cap_offset < sizeof(*info) || info->cap_offset > info->argsz)) { + return -EINVAL; + } + + return 0; +} + +VFIODeviceIO vfio_dev_io_sock = { + .get_region_info = vfio_user_io_get_region_info, +}; diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 11a80d4e7c..30a3125ea3 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -95,4 +95,6 @@ void vfio_user_set_handler(VFIODevice *vbasedev, bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp); int vfio_user_get_info(VFIOUserProxy *proxy, struct vfio_device_info *info); +extern VFIODeviceIO vfio_dev_io_sock; + #endif /* VFIO_USER_COMMON_H */ diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index 69806eb863..fdff6c99e6 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -108,6 +108,8 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->ops = &vfio_user_pci_ops; vbasedev->type = VFIO_DEVICE_TYPE_PCI; vbasedev->dev = DEVICE(vdev); + vbasedev->io = &vfio_dev_io_sock; + vbasedev->use_regfds = true; /* * vfio-user devices are effectively mdevs (don't use a host iommu). diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index 5f9ef1768f..6f70a48905 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -125,4 +125,18 @@ typedef struct { uint32_t num_irqs; } VFIOUserDeviceInfo; +/* + * VFIO_USER_DEVICE_GET_REGION_INFO + * imported from struct vfio_region_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t cap_offset; + uint64_t size; + uint64_t offset; +} VFIOUserRegionInfo; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events index 6b06a3ed82..1860430301 100644 --- a/hw/vfio-user/trace-events +++ b/hw/vfio-user/trace-events @@ -7,3 +7,4 @@ vfio_user_recv_request(uint16_t cmd) " command 0x%x" vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s" vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d" +vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index %d flags 0x%x size 0x%"PRIx64 diff --git a/hw/vfio/ap.c b/hw/vfio/ap.c index 1adce1ab40..54b1815f1d 100644 --- a/hw/vfio/ap.c +++ b/hw/vfio/ap.c @@ -162,6 +162,8 @@ static void vfio_ap_realize(DeviceState *dev, Error **errp) return; } + vbasedev->use_regfds = false; + if (!vfio_attach_device(vbasedev->name, vbasedev, &address_space_memory, errp)) { goto error; diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 8c16648819..085a3fc6e6 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -586,6 +586,8 @@ static void vfio_ccw_realize(DeviceState *dev, Error **errp) goto out_unrealize; } + vbasedev->use_regfds = false; + if (!vfio_attach_device(cdev->mdevid, vbasedev, &address_space_memory, errp)) { goto out_attach_dev_err; diff --git a/hw/vfio/container.c b/hw/vfio/container.c index 0e1af34ce4..ddb86edb65 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -890,10 +890,17 @@ void vfio_put_base_device(VFIODevice *vbasedev) int i; for (i = 0; i < vbasedev->num_regions; i++) { + if (vbasedev->regfds != NULL && vbasedev->regfds[i] != -1) { + close(vbasedev->regfds[i]); + } g_free(vbasedev->regions[i]); } g_free(vbasedev->regions); vbasedev->regions = NULL; + if (vbasedev->regfds != NULL) { + g_free(vbasedev->regfds); + vbasedev->regfds = NULL; + } } if (!vbasedev->group) { diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index bef1540295..4ed393420e 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -365,6 +365,12 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, region->size = info->size; region->fd_offset = info->offset; region->nr = index; + if (vbasedev->regfds != NULL) { + region->fd = vbasedev->regfds[index]; + } else { + region->fd = vbasedev->fd; + } + if (region->size) { region->mem = g_new0(MemoryRegion, 1); @@ -443,7 +449,7 @@ int vfio_region_mmap(VFIORegion *region) region->mmaps[i].mmap = mmap(map_align, region->mmaps[i].size, prot, MAP_SHARED | MAP_FIXED, - region->vbasedev->fd, + region->fd, region->fd_offset + region->mmaps[i].offset); if (region->mmaps[i].mmap == MAP_FAILED) { @@ -568,12 +574,16 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) { size_t argsz = sizeof(struct vfio_region_info); + int fd = -1; int ret; /* create region cache */ if (vbasedev->regions == NULL) { vbasedev->regions = g_new0(struct vfio_region_info *, vbasedev->num_regions); + if (vbasedev->use_regfds) { + vbasedev->regfds = g_new0(int, vbasedev->num_regions); + } } /* check cache */ if (vbasedev->regions[index] != NULL) { @@ -587,22 +597,33 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, retry: (*info)->argsz = argsz; - ret = vbasedev->io->get_region_info(vbasedev, *info); + ret = vbasedev->io->get_region_info(vbasedev, *info, &fd); if (ret != 0) { g_free(*info); *info = NULL; + if (vbasedev->regfds != NULL) { + vbasedev->regfds[index] = -1; + } + return ret; } if ((*info)->argsz > argsz) { argsz = (*info)->argsz; *info = g_realloc(*info, argsz); + if (fd != -1) { + close(fd); + fd = -1; + } goto retry; } /* fill cache */ vbasedev->regions[index] = *info; + if (vbasedev->regfds != NULL) { + vbasedev->regfds[index] = fd; + } return 0; } @@ -775,10 +796,11 @@ static int vfio_io_device_feature(VFIODevice *vbasedev, } static int vfio_io_get_region_info(VFIODevice *vbasedev, - struct vfio_region_info *info) + struct vfio_region_info *info, int *fd) { int ret; + *fd = -1; ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, info); return ret < 0 ? -errno : ret; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a7084a7690..4ab3bfa2fa 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3175,6 +3175,8 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) name = g_strdup(vbasedev->name); } + vbasedev->use_regfds = false; + if (!vfio_attach_device(name, vbasedev, pci_device_iommu_address_space(pdev), errp)) { goto error; diff --git a/hw/vfio/platform.c b/hw/vfio/platform.c index 51534fd941..a76cc6a194 100644 --- a/hw/vfio/platform.c +++ b/hw/vfio/platform.c @@ -575,6 +575,8 @@ static void vfio_platform_realize(DeviceState *dev, Error **errp) VFIODevice *vbasedev = &vdev->vbasedev; int i; + vbasedev->use_regfds = false; + qemu_mutex_init(&vdev->intp_mutex); trace_vfio_platform_realize(vbasedev->sysfsdev ? diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 699e8a9376..ab7fa7135a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -59,6 +59,7 @@ typedef struct VFIORegion { uint32_t nr_mmaps; VFIOMmap *mmaps; uint8_t nr; /* cache the region number for debug */ + int fd; /* fd to mmap() region */ } VFIORegion; typedef struct VFIOMigration { @@ -139,6 +140,7 @@ typedef struct VFIODevice { bool ram_block_discard_allowed; OnOffAuto enable_migration; bool migration_events; + bool use_regfds; VFIODeviceOps *ops; VFIODeviceIO *io; unsigned int num_irqs; @@ -158,6 +160,7 @@ typedef struct VFIODevice { QLIST_ENTRY(VFIODevice) hwpt_next; VFIOUserProxy *proxy; struct vfio_region_info **regions; + int *regfds; } VFIODevice; struct VFIODeviceOps { @@ -202,7 +205,7 @@ struct VFIODeviceOps { struct VFIODeviceIO { int (*device_feature)(VFIODevice *vdev, struct vfio_device_feature *); int (*get_region_info)(VFIODevice *vdev, - struct vfio_region_info *info); + struct vfio_region_info *info, int *fd); int (*get_irq_info)(VFIODevice *vdev, struct vfio_irq_info *irq); int (*set_irqs)(VFIODevice *vdev, struct vfio_irq_set *irqs); int (*region_read)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, From patchwork Wed Feb 19 14:48:48 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982404 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E8262C021B1 for ; Wed, 19 Feb 2025 14:53:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPG-00044K-Sy; Wed, 19 Feb 2025 09:50:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPC-0003o7-Bj; Wed, 19 Feb 2025 09:50:30 -0500 Received: from mx0b-002c1b01.pphosted.com ([148.163.155.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklP8-0007ZP-5U; Wed, 19 Feb 2025 09:50:29 -0500 Received: from pps.filterd (m0127841.ppops.net [127.0.0.1]) by mx0b-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBEuNk022547; Wed, 19 Feb 2025 06:50:23 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=DKDm8DZbRTnYTY+8HOnT3pbBPW7w1QFY++eASlZ37 iM=; b=FvvDX1YQLp2A2pCCdNsn7DKFjmw9vAaWEYCoZJizN97MYrtM15r7j2rx1 66bAUMdluS/1ovzGNa9HbEY09l6ZjYlgmvFHfaUnMSjAIXzJPX3O6oEi25WZiGeZ 9aFBhfGlJfvlLpi3mAYcnUCeWbg5a+McQrp7KLqYi5heyPwLYV2Rbmvm7k7kvBOv QUg0smtnM0yiJhRLILoJwCKJWjZ1wPta7Uqu/ZbYUEOMteQR/LY/IwFwPJhfwNuy 9X1XvArGDt4XthFeOlP4OM/gu7k+/zIrbkcYU/kZpUTpGEzMGxPYgApBZHySuIAV uVepPWZFmYq4b9CTunTmpuxFI6whg== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2040.outbound.protection.outlook.com [104.47.70.40]) by mx0b-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4b9hy5d-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:22 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=BS5fEsUpchrlRBLbfy77VI2FarX7mA79vm5aUuX7gy7V6vEv9LvkUdMH6DWadBglJsYWXwDMTzzvhp9CF8KEcMlAeCPNqieRaG/rqWXzzRzkeaVupoL6MW7+nScxtVVq9Jv49bX8pzQ+25xhOxdQvZW8wIUNnVqNc9aHuZGDe3aYq5qYNz8nBVFhVlrmUL3loeO6u+XBl1/sU9HLb4WsmdVHjej70hNRR9h1y6QVsI/hqMHrNnMAI9r8wEFFM/bSgyNwRa1wX/FOuBCV1GIf5+pUntshcBVj0gtlrnIEQejlWdYh45W5hpLmAWp0Bjsut3mZXkch7ItHcMrd1BT9jA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DKDm8DZbRTnYTY+8HOnT3pbBPW7w1QFY++eASlZ37iM=; b=ZiWlPEeXPAmV92uuyyHmZVgQC+7i7T0IAWLveT2GONuZnvBRclALSh9mcKoqjuNr6V4AmbBrQK4fTPKWnNEa7IOQEZuc8dO6HEduu+iyE4vezlf1m7bKI70+R/gB2+aZQxfAGIOZZ11QO1MkG9lAfNZszUnajR0FuNYjnJNlTTH5tK+nE/VK4xAZIhg91xklDj6fkX4vnM5lbU/MniiexbSijH/JAGrvTYgcotiZNGcbY2AG6RDonTY3UiP8fdD48JU9BCgz/WXwJrD/atQMV8fIQADK08+Apg7g36lfsM5Oh6F4bV8+SzzPFsIprNYok4Tntw+weeoMSL9dol0Gsg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DKDm8DZbRTnYTY+8HOnT3pbBPW7w1QFY++eASlZ37iM=; b=nwMt+hUEXGNTxGqH+v1250Om+ZLl6XEsZ0sD0ayXuNDhYPy56qixRZad9lW8QItgzYMECo+fmDaySwnv0FT15VfSwR8FKdB8JE4ZcppB38xT+tRNUwn2te8kVurj6AneprB5wXeo9rfDF7xfabpchBWEYU1qSUvFVQo244M09XU8O8ie3AClA/k7wVPYR9LrKvpak3BpYgx3AvvxpyuFUa7zLC29en+DunPJaLeVCtivjiEUgn/9hiFgRgm/IefXrNq/aWygj2mptZhTM7kluIUrPVbSARWPiQMV9nNzOPvpMpN54NRI4KNelLMKS4N4cMb/Zv+E0HXPQWMQv64JdQ== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:12 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:12 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 18/28] vfio-user: implement VFIO_USER_REGION_READ/WRITE Date: Wed, 19 Feb 2025 15:48:48 +0100 Message-Id: <20250219144858.266455-19-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: d04eb0d8-3a01-4f0b-92d1-08dd50f4b6cb x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: F3AC84czEaqYAo5z9XhLxUBNAwAQLG5XmocnTTO+7sE2oGKYcOc3gRpHGe83g3WGiIdVaGSVSfEvCUby153ImpKXpkydAmpi6eAjFAxLATm57sitB7yBwqEcpOSOvgTHFFQOb4mB85d6VMEhZ8zZKuZOcH4S5nPCvzdFWbXD3o01Nm99RGEdZJSqJuMkhjGtdK4HEyVY8MWYjKLNFTaCz9TBlPCfVOhYuha7FM2vRQ2mSvo8UaCkLn9ySJM5s7oX5r2ld7IuyMXwBc9YstZUYRPtoBfZBYHjci738ShQ3JSm4zgz/IAuRAjGXW+iK5E4TenGJAmkVp6EiTENmdql0RDapF5bPe29O9TealQpoUd+oNDS474IYnsKF9IO0Kbm+3t8F8GuZBVqewtkKUhPgMG93gCj0wpIN98gQwOlgcGPFf8EfOMMrJOvAZRpE5PzFBXKfcmpIxZXZhrigLxiLttZKCXs+wOijc6Xks1GBQdV8/QepY0O/hVzVqvO1PHEpH7S6c1wBpn2Fs4izc7D5US+KnNDR/x71WlEnDS/lSwYbrvGA8o2lJ1PByTR1ja9V3CURN+AmL6mlTY8Swq8g0lTAsAXqUeqljlG3mLfTGPsW44AxSdmovym/s3JnqZ8WxfZsqX7cMh/lbTKRE7ScAHupEA8qW+FIcbx+1V3qNuYoIsnHWUgdiDfqybAQA2DjuzZxM1+DgJFIpIdJHeIf/+qv2MYn29V2zz974NCcWaiF/plQV0yiI5C1JNv6BRwZEwjb94nYJsyOlz+a/KkrfGvYlYbqPCEl/tb0RCXNt8pBCthJYmiBQh3BXKZWna7BcZYzMQr52/6VUQ5ISl+JsMnridfxWEogkbUf0fflb1nnAr9ODL9dRvFQqSYSa3gK9gFaTYd4oU4RarTDH6tsfVfIizWJShFdECI71IfWZvRZE7ZOf2XNGh77V8uq9JKvzyfeJhVr+vY/ZZRNFx+EahB28iYgsrsQ1WOxT7zd2J9W+l0MMc4b0TGI5r2QUdNnQl41NfWSYuuXyzZ0jAjfDYRAKTZ7nKdT+9knDr1aPYZ50by+GvAa9jZ9k9wPh95dSVp+su/b6wOOagbWzlddJnQKnU525cIZtW6TVmuInnbqbfEkAIH0sEKlave/qx3P83VUbr49L3aTTydJYiAh+EpQuHJTtVarYOUPSMZjUQPjNlntCqXS/bvMm+V6uSuIJykDVhC5i93RGZqF+9ncEhKVyzcQ75jaNLSTzzjuorvdesVCoyBSt9594QPFV6eLlPQFVD/jrFvqQRazcj20nUjDSuMqVfB96r+gG4KZ4XThNo5HyQdmo2X7ASxxGz8zLajRgCpnKIWeYus6uSQALnN3CLSj729ShurQexbdHumj9Nm4ByIMgWTEOUdtbtJ X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QnTcncMio0ckd+hgmDYmDd7T+SzTFgdCCDn8XuFNMHSj9Kw2xgFiuMDTGN4GOW1DPBZKI7cfzq04LSPdH2xIbUUT7wkgYWknZw3RwzFpmvOU7io3rgbzi4ajwKUB8TyroNZ2mfuiOmPJEIe48snEIQ432IdqOXxGF5+Eu2KFpQZHiThw64zJvzIba8CONrTJKoinPG1Im+t/pPYtegjbNPgfytC+My3SNoSB6M+6XQsphnf7xcSKSoTEghULtDbM3tCLaF5W8qdRT0cq06UOWk9mMWqBbDBRqx4Epyfhpf9OZdGH5v9VEeawcXCcoIFYI+wbU8dMLL/PcePTn3coc5V4Xymf0Mr3vdLh9rWVLidZyitN0F1/9HfaUsjqWX/ULtVD8Vil8mw3cRHz9kBX97GtpLkcv0P20hj6XG5GYhT35DkHRB/+sN4MCxv2q/wO51CsUuyshQjx50hk7Q8lL9rPocDKigYzQ1g2/e3LXZF5PdkrnSJMlO74oypuowaXDNuPiqGgNF9x/ERmep0jJCizusZFn1HWw0HGaUqMaR1+Y/FLemKk29D3zLcN6m+lEKtEKPECpVM/sRajS8he6jTJ6ZcraRfE1Ddw/RdncQJypvISd2zIwhKhd/EK/nIKTi9Cx1H3F9kU7bURin4+zpQZkvnziiyAyerFTatlWB0Ev9seACrfcj5r13wsrOBNzUILSwV4osFEflTSp5T/Pcl0k7To8mDHhBRaADGr18i4fOfp3VjsGrxjTVJ+ueWZU+txIbH56xoyjQ3aZT6+Pvj04fbtySXxNzul784Aw6bgDl6/kUZWTxFwm6DfLtNbD/qExNyt7x62zKF35ik/vWBRWVLHolJg3nn1ENjtLi63loWzby5RGuHuP/jCFUYLpHt9FW9STFdySaKAoHgDRTuniKyEggrUX86woBtXbOU1HOHIqjcHImZOPq2mh/0jIdJlYzgiM2MvpLK4iRQvyiJywk8Uy2mN24N0tjJNF5gnu5yFX0+7/jzoruQ2OLBp7HdSD4OGdtK6ZnS/PA7cE6qdlLXa/S0BrTUKamkP30hnQ3eUPiBIvQQQcKh/Os/qFJHyPVe7xfBWpbydpJyQokBpjQcMvxT+igc2Gu2nY1/xb2VsXtu2DqUnELpNw16tDrQK33i6mso5473404ozAqA1Y9iKCBy5kt0ocUV/cvolIjRNHjIK2yRoeqH9nHoIgA8a/iVib4F1L/Ah1QvU0IeWD5ClgPCxEEutbPjzpQ0rYS40IwFXydLnJjf4Zc1ThNslTmspbl3yo10EecGrqbwP0DYJUpricCIyeOhINQf3palZb/sEGzqSnpf5VnZ4eFbomifH3MudHp2qOcB4KufGJcqR7JAsJkcpFrqD6ffZJHPekngDoKCsgcDUohwAickGJKS2sXOpYTCuYOWKiXw+wbMHAl5OTJrKdoKThQR619Mq7HDJmuZFyhvyo2cc6WZI4AMuKi/ZshjQBFQHdjFuuS+6l1mLGmSJ0butjp8Zk/RpLnIiy/RLxgZP+31N2I7gbsOGtfIVsD/PjDi8hNHBbcdP2CK8sjeAoOIzRXw4e4SsVjspcQgj6tTH9erj X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: d04eb0d8-3a01-4f0b-92d1-08dd50f4b6cb X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:12.7708 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ysHXaO+NiqAYX7wMLh/BpPBlVsrp1MoQupjCMiVt+yKfUbf0/AL4KDdwgR+D/V+MzXUbZwck3U4iZiYXcbhvhQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: rnF96_WwRPo185M73iD-EERV7nJqSiKy X-Proofpoint-ORIG-GUID: rnF96_WwRPo185M73iD-EERV7nJqSiKy X-Authority-Analysis: v=2.4 cv=HMQ5Fptv c=1 sm=1 tr=0 ts=67b5efae cx=c_pps a=bqH6H/OQt14Rv/FmpY1ebg==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=Qylz81XdT_8TPN73I5EA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.155.12; envelope-from=john.levon@nutanix.com; helo=mx0b-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Also add support for posted writes on remote devices. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 120 ++++++++++++++++++++++++++++++++++ hw/vfio-user/common.h | 1 + hw/vfio-user/pci.c | 5 ++ hw/vfio-user/protocol.h | 12 ++++ hw/vfio-user/trace-events | 1 + hw/vfio/helpers.c | 8 ++- hw/vfio/pci.c | 5 +- include/hw/vfio/vfio-common.h | 3 +- 8 files changed, 150 insertions(+), 5 deletions(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 7b96d715cb..1a9033af5c 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -55,6 +55,8 @@ static void vfio_user_cb(void *opaque); static void vfio_user_request(void *opaque); static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg); +static void vfio_user_send_async(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds); static void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize); static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, @@ -626,6 +628,33 @@ static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg) return 0; } +/* + * async send - msg can be queued, but will be freed when sent + */ +static void vfio_user_send_async(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds) +{ + VFIOUserMsg *msg; + int ret; + + if (!(hdr->flags & (VFIO_USER_NO_REPLY | VFIO_USER_REPLY))) { + error_printf("vfio_user_send_async on sync message\n"); + return; + } + + QEMU_LOCK_GUARD(&proxy->lock); + + msg = vfio_user_getmsg(proxy, hdr, fds); + msg->id = hdr->id; + msg->rsize = 0; + msg->type = VFIO_MSG_ASYNC; + + ret = vfio_user_send_queued(proxy, msg); + if (ret < 0) { + vfio_user_recycle(proxy, msg); + } +} + static void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize) { @@ -1139,9 +1168,84 @@ static int vfio_user_get_region_info(VFIOUserProxy *proxy, trace_vfio_user_get_region_info(msgp->index, msgp->flags, msgp->size); memcpy(info, &msgp->argsz, info->argsz); + + /* read-after-write hazard if guest can directly access region */ + if (info->flags & VFIO_REGION_INFO_FLAG_MMAP) { + WITH_QEMU_LOCK_GUARD(&proxy->lock) { + proxy->flags |= VFIO_PROXY_NO_POST; + } + } + return 0; } +static int vfio_user_region_read(VFIOUserProxy *proxy, uint8_t index, + off_t offset, uint32_t count, void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; + int size = sizeof(*msgp) + count; + + if (count > proxy->max_xfer_size) { + return -EINVAL; + } + + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_READ, sizeof(*msgp), 0); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + trace_vfio_user_region_rw(msgp->region, msgp->offset, msgp->count); + + vfio_user_send_wait(proxy, &msgp->hdr, NULL, size); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } else if (msgp->count > count) { + return -E2BIG; + } else { + memcpy(data, &msgp->data, msgp->count); + } + + return msgp->count; +} + +static int vfio_user_region_write(VFIOUserProxy *proxy, uint8_t index, + off_t offset, uint32_t count, void *data, + bool post) +{ + VFIOUserRegionRW *msgp = NULL; + int flags = post ? VFIO_USER_NO_REPLY : 0; + int size = sizeof(*msgp) + count; + int ret; + + if (count > proxy->max_xfer_size) { + return -EINVAL; + } + + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size, flags); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + memcpy(&msgp->data, data, count); + trace_vfio_user_region_rw(msgp->region, msgp->offset, msgp->count); + + /* async send will free msg after it's sent */ + if (post && !(proxy->flags & VFIO_PROXY_NO_POST)) { + vfio_user_send_async(proxy, &msgp->hdr, NULL); + return count; + } + + vfio_user_send_wait(proxy, &msgp->hdr, NULL, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + ret = -msgp->hdr.error_reply; + } else { + ret = count; + } + + g_free(msgp); + return ret; +} + /* * Socket-based io_ops @@ -1171,6 +1275,22 @@ static int vfio_user_io_get_region_info(VFIODevice *vbasedev, return 0; } +static int vfio_user_io_region_read(VFIODevice *vbasedev, uint8_t index, + off_t off, uint32_t size, void *data) +{ + return vfio_user_region_read(vbasedev->proxy, index, off, size, data); +} + +static int vfio_user_io_region_write(VFIODevice *vbasedev, uint8_t index, + off_t off, unsigned size, void *data, + bool post) +{ + return vfio_user_region_write(vbasedev->proxy, index, off, size, data, + post); +} + VFIODeviceIO vfio_dev_io_sock = { .get_region_info = vfio_user_io_get_region_info, + .region_read = vfio_user_io_region_read, + .region_write = vfio_user_io_region_write, }; diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 30a3125ea3..31b11ed614 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -84,6 +84,7 @@ typedef struct VFIOUserProxy { /* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 #define VFIO_PROXY_FORCE_QUEUED 0x4 +#define VFIO_PROXY_NO_POST 0x8 typedef struct VFIODevice VFIODevice; diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index fdff6c99e6..26d2960985 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -37,6 +37,7 @@ struct VFIOUserPCIDevice { VFIOPCIDevice device; char *sock_name; bool send_queued; /* all sends are queued */ + bool no_post; /* all regions write are sync */ }; /* @@ -99,6 +100,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) if (udev->send_queued) { proxy->flags |= VFIO_PROXY_FORCE_QUEUED; } + if (udev->no_post) { + proxy->flags |= VFIO_PROXY_NO_POST; + } if (!vfio_user_validate_version(proxy, errp)) { goto error; @@ -170,6 +174,7 @@ static void vfio_user_instance_finalize(Object *obj) static const Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), + DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false), }; static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index 6f70a48905..6987435e96 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -139,4 +139,16 @@ typedef struct { uint64_t offset; } VFIOUserRegionInfo; +/* + * VFIO_USER_REGION_READ + * VFIO_USER_REGION_WRITE + */ +typedef struct { + VFIOUserHdr hdr; + uint64_t offset; + uint32_t region; + uint32_t count; + char data[]; +} VFIOUserRegionRW; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events index 1860430301..3f5aebe7ac 100644 --- a/hw/vfio-user/trace-events +++ b/hw/vfio-user/trace-events @@ -8,3 +8,4 @@ vfio_user_send_write(uint16_t id, int wrote) " id 0x%x wrote 0x%x" vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d minor %d caps: %s" vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d" vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index %d flags 0x%x size 0x%"PRIx64 +vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d" diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index 4ed393420e..6bc7600ab1 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -184,7 +184,8 @@ void vfio_region_write(void *opaque, hwaddr addr, break; } - ret = vbasedev->io->region_write(vbasedev, region->nr, addr, size, &buf); + ret = vbasedev->io->region_write(vbasedev, region->nr, addr, size, &buf, + region->post_wr); if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64 ",%d) failed: %s", @@ -365,13 +366,14 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, region->size = info->size; region->fd_offset = info->offset; region->nr = index; + region->post_wr = false; + if (vbasedev->regfds != NULL) { region->fd = vbasedev->regfds[index]; } else { region->fd = vbasedev->fd; } - if (region->size) { region->mem = g_new0(MemoryRegion, 1); memory_region_init_io(region->mem, obj, &vfio_region_ops, @@ -837,7 +839,7 @@ static int vfio_io_region_read(VFIODevice *vbasedev, uint8_t index, off_t off, } static int vfio_io_region_write(VFIODevice *vbasedev, uint8_t index, off_t off, - uint32_t size, void *data) + uint32_t size, void *data, bool post) { struct vfio_region_info *info = vbasedev->regions[index]; int ret; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 4ab3bfa2fa..168e331607 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -51,7 +51,7 @@ (off), (size), (data))) #define VDEV_CONFIG_WRITE(vbasedev, off, size, data) \ ((vbasedev)->io->region_write((vbasedev), VFIO_PCI_CONFIG_REGION_INDEX, \ - (off), (size), (data))) + (off), (size), (data), false)) #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" @@ -1782,6 +1782,9 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) bar->type = pci_bar & (bar->ioport ? ~PCI_BASE_ADDRESS_IO_MASK : ~PCI_BASE_ADDRESS_MEM_MASK); bar->size = bar->region.size; + + /* IO regions are sync, memory can be async */ + bar->region.post_wr = (bar->ioport == 0); } static void vfio_bars_prepare(VFIOPCIDevice *vdev) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index ab7fa7135a..5adbff4fa7 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -60,6 +60,7 @@ typedef struct VFIORegion { VFIOMmap *mmaps; uint8_t nr; /* cache the region number for debug */ int fd; /* fd to mmap() region */ + bool post_wr; /* writes can be posted */ } VFIORegion; typedef struct VFIOMigration { @@ -211,7 +212,7 @@ struct VFIODeviceIO { int (*region_read)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, void *data); int (*region_write)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, - void *data); + void *data, bool post); }; extern VFIODeviceIO vfio_dev_io_ioctl; From patchwork Wed Feb 19 14:48:49 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982406 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0E60BC021B0 for ; Wed, 19 Feb 2025 14:54:08 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPG-00040H-4l; Wed, 19 Feb 2025 09:50:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPB-0003lr-Va; Wed, 19 Feb 2025 09:50:30 -0500 Received: from mx0b-002c1b01.pphosted.com ([148.163.155.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklP9-0007Zs-CN; Wed, 19 Feb 2025 09:50:29 -0500 Received: from pps.filterd (m0127841.ppops.net [127.0.0.1]) by mx0b-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBEuNl022547; Wed, 19 Feb 2025 06:50:23 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=DwxnAX2BHGq2NTNcQXfmH8RKvi9Q6E9jAgmxXCPtR xs=; b=YBz6f+H3C+vvYKrMyVbD2d6fYY2Ca/9RKr2TyeGLxyQNQDCCGskkeycM2 XolIeYunxIYuNDKdLwDaLzqlPJukqkhRNiZAHmJAAxjjCtZY2DM4CTzhXyNmWuyQ 0LwKHlPu4qyL04Vn6wYB1kFynnNPR3fvcLyjDcL9jnbSwhGqmIkkY/b/6BJZ94qN TP5XXtPX7f1qydvufRqofdmqEhFEIrHgkro9P0dxufBe8E69YbblWbt8lakLPYE4 xqA9bLW9HSQcSIGyAuQxmQMccpEhDBXNRkpjsUnRU6h947l7M8SW5ynuhbsTAI8q FG/Ict6B6r8aGv6Y6lo4MWYkEpgYQ== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2040.outbound.protection.outlook.com [104.47.70.40]) by mx0b-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4b9hy5d-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=EPVt2CbDqgIyDKG9klxPwkilkQlBNSskjN8uxH/WektBwssaXmz0gMwcpAou9QzMaN3ZLJERykrWq5prh2wPje6ruDWhqPVIimwhVubK/Tx8AyDZ2J1u4NYc/OVxjm+A/VdvEtckR3elNEbnOXq5a/tgfTI2jgDOCOYqqncXDLf0RkDg3kNQf8bzPQUpSHDuoc6DMzLTTrc2hWpQaBVTz/EvO/SpcLsZsZmZDCPp8uEgFnzmdOXO7SGk3C4beMKRvdFmE4Bv7YEaLR/jQgfG2dwBCr5828d/J5edeVfGqnWN9QU8EOXPIQRPLpHrKD65tUEH8vQq5YixfsWJOXd4ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=DwxnAX2BHGq2NTNcQXfmH8RKvi9Q6E9jAgmxXCPtRxs=; b=DIJSC7C2yN4ZnsP9mT1WSJf6p4pjqIqXZpptvC0PuX2KAwspxAIXNhvuDBrNajn4lSVw1RPr7iNkt2A36VCtrv3yRTz/0uH3VXovz0OQxbJBUqEFswKhZJLh3dANDnLbfivQIYrymZBdJGA1VhbzQDQfyZWFbMhkOK15AUk6CrhD+HLIoL93xZNh5c7Gb//lRsv6P13mS6FPVTUZxvBaCDZEzS9n56atmm7PjAjy/VRf5qetX7qbvK4KIiiSImgoak2lwjKEz+XIriHVZL/xGKuB3DHyt207QxAGtvg8equUYO4uWDlNbYX+9VoUeGhy4UcMFGsvnFQv4NBmoAbYxQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DwxnAX2BHGq2NTNcQXfmH8RKvi9Q6E9jAgmxXCPtRxs=; b=dpQJqtGB8PrjUso/ikcJfknDLQXkLqmFstLEF5suWLDhIh0rSM96gkGoWw4uRXqmr4CeuRp1PzGdHfFdQh/iLnpRX8b7VudysAXEz0RzeAdOHz5DPzfFIuqrH/rnEFo6lervjwkCd9BBQG/lUDrTMQE3JaN9HTNqyBPApYVg3da3GDjapTykGmBmGNOp3ZNZIzLxE/QXnP75tZF/snrBJueBVnGiCB5xalH7J5HqDPntc4ViQZzmf7e015Tnn4sa+dKEG8eXetWI7ZhRsFXZnm3cjRB1c8z2MK/Dl8WR6YYuNKJDYVRxGkqgMloYP/MZ64MpV8IvxCdQbfA6xqLRVA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:16 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:16 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 19/28] vfio-user: set up PCI in vfio_user_pci_realize() Date: Wed, 19 Feb 2025 15:48:49 +0100 Message-Id: <20250219144858.266455-20-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: a455c394-657c-425e-b89a-08dd50f4b8d7 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: uhp1EJtN673+6NoNTGUDrccCa6F8d3TfGW3kZPRjLJa2p/SFq2+2x/AwAFCntRuhbw4bInO6SBKL694W+ukTeTV8ozcO6wOQ7LptHz3LsneY0nD1Bjs6NF2hzWdNDGl3gIa39S6GXCoovQ0z7Xyz1tcbIFTZLrUfXybK0Omnxed1KZQNhz+UnlgffQhLZwpeD0rvk0d6ldWgobh50imonP7y/1iqb4pXcAyTaE+AryG3IURjRDe7cfEeMxKDYpaY8LxIY11DghcAdWZjFm24YjYKeLK645J6Q3xI+x6XPLXYKg+CU2nKHzZTKXiRUUuwOCkRkFK9248be6EGawEw/uLzGngXbXv9BA/8eMSKRDsGkaX4JGeJmEBDP9QJ17BxqQMYP9E2T3K9L5lRtqox/98DkiAz9QrTb7Ci+UY1xEYE80nOSwpITAUxB1b+v1DM96RKNcmq/+M2Jfw+3eWPZMK0LZ2uhZ5S7Dipj0kG+gioY+CFa36K3NFlUa0rgjOGyhUpyc2ezNpIoCWaShkHPV411c7+8e65WoR+T7i5/UufD2Rc+sic89Sr6NA0f62asKlbXi2NlxPuqbPLiJWwAVDDTHGiLnmjuxRhyE7GTYtW4Dv7Ye3S1JTXY6mMN7z1bnlGQiPtYtUfS3E1YbqgndMIKScRyg9LhoSEcmTFoxN/5QYMLqL7KewtNLyolXk8ndPANWzicRPMCwJEte4j+QIDay21yGgL5/tIKjWYZqXWm2wGi5RSUpmX2gJcSRsgLfiyQ6y/Ozm+L0PBnWnvYETYCKzqOsbjr4u/6OjevkS0hGxksohqTXdG7t8f/fRYrwOaE5t/sM4bnlCjPJ+H99956JfZ/gGPIGqwPZKYH/qA7H3l9G3QaDQNeNKoz+rbav8q5hf/pOPGkbokFbVjjzmKfFRcslXejATjeQmapLM8ik+ZbmDQlUOPGKTz/2mZyQDKdc9QssjSIi69q1LBncelE31HUBe4aJnMEJPfLswuA0hbayagdAZWunrp2HRQjDQfu1qFUmGR+cuoarFzAKg7v4LHudlR89KeLHnT5wSM75hWYc8BwzzL+F9URbSnJypee3CV5zlmmitxqsQg6m4Zv0MqvLUeOwJEvq7MTFd+OVrdjROJSTsKPg02A5g5M1mYima+1Dyhkq+W1pA/5QeF7g1z1frLj5FXBNgAhx6EUndrhYyVbp2PZt16TEJytZnqZMWnA+4MXzCtUfKvKi5wfTGHmjTNKd+HJuvrGI4hNoXsPAhSGS2L9K1FsDTFbuWScZipnFGZ+hLa1LcI2u0yT8JlrytGQkYgt6+9FmceStSmSOFUMpuaupDUM97Ac6Xl9cEQ2QcMMe1mG76E02UbzzgE1WpJB0SKavIRWsSzB0oNlPfWrOE7S4jQ37bc X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ASAHOudkRrs5cCx96vMd9hEK8RUatIqNqYmUXC4oZNhu8ZwXubDKXG63GY6MTMIuaSkHNu8397KdyI6heva1iUhNPW52Dv0GSgiSqEFKKrwvyqf+T35Q+6R3Pe1CRdCg000UjMDh2lSRuL8kTbak7Gb6GwLuhehgfR/RtoZgaup0eztywTrdUKizGkJHI4a5wemWPlcRbg9oqPJ1jvJ4JQ+gf+pHYE4ae29OxuQHco+BJd7pVB5jkIp2x5iFxlEGK/rnv3S8896BbIxHujudek8MrWjT6hXf/Hyo/ESBR6r7inltyVqbknBf8wgQTRDrMa0/8f+cFqXVazMuEIwA7wLGrfAE/NkttDBqjauiy3UFsFVp6vjNPo34IyOtzaOOsp0RGQNpJpGS4ez6zJ9fe1PcocGRDYS/VjSNJDgzOZsWifpyxOI8DS5dn5AtoX/BhvqaHukOjO2EBHiVLNv++pFxxZ4Jim8yrEe35EIfPOU9rmW5MDdYzNhBHNi652QqJHvA/q3+ObbxfcKMYrv1afp+aUaW/q9a1ArIphP5W1hYAKyvf1uwlFZaMiboW22RuQjpJLVKZd/VWyMZjql2xqjG1nDlMSsxiMEzhxEGcIX4qrQNXrfc8nhEoNT0YWOI54evm6Q6kJg0UGV6DVaNU+wDNqR6yjsRRY8tSXcs0IAGz1J9p7cEFQ3xB9x1GM+0J97aZRVFlkgGT+hILj3aGm4/TdZE/LHbul/xhnhLElnGKCIEsBMXcTNl9m2r9uLkMONOmo58rnV40aLMQtZpTPKZGdpm5K2wIvQ/x5cC/eok8b8zfxfu35FXI6iR7fFHYsuC4YQJsP2R2jFXOL2MqZob1WFpMLPzrUkQ/4mVs4joyreLo7eJ/Ad/eGgsW3MRpRBSOW1g0+pHzb/djd85p5QRXSJuqDq1z4Rk5A1BPtxpU5UAwb2BZ5EqMlFiGn2t40I7MBGDkXJEChh+Yi/hWuy/G4N0TwfgQQPhwH8v6KRuqdsK4C/hY++4iwSehhoRyjID2fZJJeKAgMPlNpTmrsXr7w+Fc/m7cQo0Zw/anGMygAYjRTebtmTDzVFTkEb+89YLAJZ3d+9Yuu2vWKCZJWCQ+BzrICdBgj8hzMnvzm31wCnBzCFXtU9d3loVhZiGLe39yr6qbSban3aAw0C7z65fEunDWiMG+/EC6gj8PXhv4z3/LAjKooA8b7PCslRytkjEy+0y1IY9ebQfv1A90KUkySRBt0+T8Y3d11+A3cDccUeWRqKnQRgVjEqd8+ZktixDQ5GMIJRBF6By9S5vb13DL9OFrPEllyhtbQ61ll6SDoF6OqgBK7l5iS4yuuzMkOeGbe2Aa4dGL/rTB1yrph81Gk0JLz9mplTqIz+BDfA1g/64eKalNO5SmQ23Pv2zFs3WjlEyTk3aCu6gp9LiNforKJj2lmC5aX1FmXWEEEUbMKSeU9IFx+5h7+F9UE5F8nTx3IRyxbDc8fGJe1rtSQ5igniu1J6h/2A3R/sY0CYgMmG26m7g9HVgNFhKOAKbKc9OTBtTUkyEZMYSMAyQumiGY1ilEITbl0L23KVezWwGqb1HPDamH36gjZUUK6/L X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: a455c394-657c-425e-b89a-08dd50f4b8d7 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:16.2058 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: x19yrGT8+NC/kd2hNMIDC/CtkHMoaBRiExsABRpi8hctlHLqJCebcoDI2FfhKDamlKVsAXteTbvc6vs7RVuGPA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: 0gRgtM4zmho0VErEccK5RnA-Mw88o_MZ X-Proofpoint-ORIG-GUID: 0gRgtM4zmho0VErEccK5RnA-Mw88o_MZ X-Authority-Analysis: v=2.4 cv=HMQ5Fptv c=1 sm=1 tr=0 ts=67b5efaf cx=c_pps a=bqH6H/OQt14Rv/FmpY1ebg==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=3F7eiexbnXk0NDyZYdgA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.155.12; envelope-from=john.levon@nutanix.com; helo=mx0b-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Re-use PCI setup functions from hw/vfio/pci.c to realize the vfio-user PCI device. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/pci.c | 42 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 18 +++++++++--------- hw/vfio/pci.h | 9 +++++++++ 3 files changed, 60 insertions(+), 9 deletions(-) diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index 26d2960985..b8c1cc34c2 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -75,6 +75,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) AddressSpace *as; SocketAddress addr; VFIOUserProxy *proxy; + int ret; /* * TODO: make option parser understand SocketAddress @@ -127,8 +128,45 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } + if (!vfio_populate_device(vdev, errp)) { + goto error; + } + + /* Get a copy of config space */ + ret = vbasedev->io->region_read(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, 0, + MIN(pci_config_size(pdev), vdev->config_size), + pdev->config); + if (ret < (int)MIN(pci_config_size(&vdev->pdev), vdev->config_size)) { + error_setg_errno(errp, -ret, "failed to read device config space"); + goto error; + } + + if (!vfio_pci_config_setup(vdev, errp)) { + goto error; + } + + /* + * vfio_pci_config_setup will have registered the device's BARs + * and setup any MSIX BARs, so errors after it succeeds must + * use out_teardown + */ + + if (!vfio_add_capabilities(vdev, errp)) { + goto out_teardown; + } + + if (!vfio_interrupt_setup(vdev, errp)) { + goto out_teardown; + } + + vfio_register_err_notifier(vdev); + vfio_register_req_notifier(vdev); + return; +out_teardown: + vfio_teardown_msi(vdev); + vfio_bars_exit(vdev); error: error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } @@ -164,6 +202,10 @@ static void vfio_user_instance_finalize(Object *obj) VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIODevice *vbasedev = &vdev->vbasedev; + vfio_bars_finalize(vdev); + g_free(vdev->emulated_config_bits); + g_free(vdev->rom); + vfio_pci_put_device(vdev); if (vbasedev->proxy != NULL) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 168e331607..f85215417d 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1730,7 +1730,7 @@ static bool vfio_msix_setup(VFIOPCIDevice *vdev, int pos, Error **errp) return true; } -static void vfio_teardown_msi(VFIOPCIDevice *vdev) +void vfio_teardown_msi(VFIOPCIDevice *vdev) { msi_uninit(&vdev->pdev); @@ -1831,7 +1831,7 @@ static void vfio_bars_register(VFIOPCIDevice *vdev) } } -static void vfio_bars_exit(VFIOPCIDevice *vdev) +void vfio_bars_exit(VFIOPCIDevice *vdev) { int i; @@ -1851,7 +1851,7 @@ static void vfio_bars_exit(VFIOPCIDevice *vdev) } } -static void vfio_bars_finalize(VFIOPCIDevice *vdev) +void vfio_bars_finalize(VFIOPCIDevice *vdev) { int i; @@ -2419,7 +2419,7 @@ static void vfio_add_ext_cap(VFIOPCIDevice *vdev) return; } -static bool vfio_add_capabilities(VFIOPCIDevice *vdev, Error **errp) +bool vfio_add_capabilities(VFIOPCIDevice *vdev, Error **errp) { PCIDevice *pdev = &vdev->pdev; @@ -2768,7 +2768,7 @@ bool vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) return true; } -static bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) +bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) { VFIODevice *vbasedev = &vdev->vbasedev; struct vfio_region_info *reg_info = NULL; @@ -2886,7 +2886,7 @@ static void vfio_err_notifier_handler(void *opaque) * and continue after disabling error recovery support for the * device. */ -static void vfio_register_err_notifier(VFIOPCIDevice *vdev) +void vfio_register_err_notifier(VFIOPCIDevice *vdev) { Error *err = NULL; int32_t fd; @@ -2945,7 +2945,7 @@ static void vfio_req_notifier_handler(void *opaque) } } -static void vfio_register_req_notifier(VFIOPCIDevice *vdev) +void vfio_register_req_notifier(VFIOPCIDevice *vdev) { struct vfio_irq_info irq_info = { .argsz = sizeof(irq_info), .index = VFIO_PCI_REQ_IRQ_INDEX }; @@ -3001,7 +3001,7 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } -static bool vfio_pci_config_setup(VFIOPCIDevice *vdev, Error **errp) +bool vfio_pci_config_setup(VFIOPCIDevice *vdev, Error **errp) { PCIDevice *pdev = &vdev->pdev; VFIODevice *vbasedev = &vdev->vbasedev; @@ -3094,7 +3094,7 @@ static bool vfio_pci_config_setup(VFIOPCIDevice *vdev, Error **errp) return true; } -static bool vfio_interrupt_setup(VFIOPCIDevice *vdev, Error **errp) +bool vfio_interrupt_setup(VFIOPCIDevice *vdev, Error **errp) { PCIDevice *pdev = &vdev->pdev; diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index c0f030f4db..7e126822f9 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -218,6 +218,15 @@ Object *vfio_pci_get_object(VFIODevice *vbasedev); int vfio_pci_save_config(VFIODevice *vbasedev, QEMUFile *f, Error **errp); int vfio_pci_load_config(VFIODevice *vbasedev, QEMUFile *f); void vfio_pci_put_device(VFIOPCIDevice *vdev); +bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp); +void vfio_teardown_msi(VFIOPCIDevice *vdev); +void vfio_bars_exit(VFIOPCIDevice *vdev); +void vfio_bars_finalize(VFIOPCIDevice *vdev); +bool vfio_add_capabilities(VFIOPCIDevice *vdev, Error **errp); +void vfio_register_err_notifier(VFIOPCIDevice *vdev); +void vfio_register_req_notifier(VFIOPCIDevice *vdev); +bool vfio_pci_config_setup(VFIOPCIDevice *vdev, Error **errp); +bool vfio_interrupt_setup(VFIOPCIDevice *vdev, Error **errp); void vfio_instance_init(Object *obj); uint64_t vfio_vga_read(void *opaque, hwaddr addr, unsigned size); From patchwork Wed Feb 19 14:48:50 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982405 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 68549C021B0 for ; Wed, 19 Feb 2025 14:53:44 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPG-00042W-KN; Wed, 19 Feb 2025 09:50:34 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPB-0003ls-VP; Wed, 19 Feb 2025 09:50:30 -0500 Received: from mx0b-002c1b01.pphosted.com ([148.163.155.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklP8-0007Ze-E9; Wed, 19 Feb 2025 09:50:29 -0500 Received: from pps.filterd (m0127841.ppops.net [127.0.0.1]) by mx0b-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBEuNm022547; Wed, 19 Feb 2025 06:50:24 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=Fi+vH5LkgI+HN9VSWmPbqx2cBNXjhtbbuBXvylDKr C8=; b=P2G8S5lTRCW6xoboERN2KeTRmCBuiORiRKvxpulGJpmIlUb4yEGhaFrYR ZeGtnq5Q4rkFSWsU045ExJBXhUjZKIPw1Wch93w0zMo7YJQZBSUzWSfiawSTE25m 07ulVso1xzXr8GQcCvkusu1A0/uwSEynUkj9/ycBCSY7ytsOSYa96AZg0rq+Fcgs xHu6t/fhcOCwymTxvvsuo7krTrb9YEYu3QuF8EI6YciN7BeYSW92QYYjjRs8OiUc Y20LAveadncikNk4J92yDEaE3QCUav8yxbaozEpngu/M1qWzmfQgIUPx9WzSdZ3O 8YaY2+uC1Ig0dRgrOrWKndKFG4Zkg== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2040.outbound.protection.outlook.com [104.47.70.40]) by mx0b-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4b9hy5d-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:23 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KUzAKexAdta52D07u5A9y8vOZdOdhNXIQ5lKVijfdoLSbL3G89oClMMuLxB/16KnGKNY4qdAxbV98MoTDOoVITeSvEhgQip3T3xuUXHgNuT32NWfvwlVFZoybMNL4/78rerrjrw0KOUp3D3Yeaa7wvX9m1y6hucnJBW2t/TLQ7o6dwy1UckKek6QewySKZoRLAtEtyGW0/iFSoK5dJWntxuCviTqWTC1IlKfuqyK7XpoK5aHujn1iJlpB65gQ1f/Y1qPX/u++RCADnRZlOEhsH1jUGrklvMkVWpAWQxcK8Mud7nzKhnoUfIJSZqzyQDKgmlFhhmQ1G9nDoK7+AmBIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Fi+vH5LkgI+HN9VSWmPbqx2cBNXjhtbbuBXvylDKrC8=; b=s0/Pk5dXbpMNpHuwbUZgYkRCJ9hl5WLYaVQhwq+bEgUz0MOPXh+zlY2gk/2F8Kfb8L9Wm9PypaCfnL/Oe4FkstGHCpnJZ0hIyhSheTM16+5WTRNcR7M8dC+/5E7m3g+qjBnskTQZLdYeYTmNqJzfPYIuA1ztF0nt4OwpCNz6VF0aPgfauG8Vvj/UrJAHkvgp/m+aPggGWFDyeM5vgtex5ofbrh4qEsV+IzCr7rARZ/QBtCOFq7vIdy7TWgyTMVbYXy+RGBFG/1yDKBOdsi5m02QR/D17v/X+u+asCIRR4Jy6Ryb4Ls1qiaiEydIojKT6AXWBgSR4w0p/MKYmzK5unA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Fi+vH5LkgI+HN9VSWmPbqx2cBNXjhtbbuBXvylDKrC8=; b=tyIxT5DQgl2ikD7Ai5QyFJ/yop/WbEoL0QjmyU9ss0MQdxdQpIQa/Y67GjGC/d/ck2WLjw4kJbIFsdSW6w/DtG1Zq51fSQB4tsE+k6htpAPR7SlhZhA672uePFf8oGnIk8mB7va2ocv6QzRdwUHWX0UyU7j63Js53eQmnKpDnbw4GPYDWenSyYLh/c5lzSA1b6Oan8Ov1VDiotlBP9pdJxGxjfBNk3I3Jkx0irY4U2ErTQAyaEE8RNAx1rMKE4o1YT2GrwvyHhZVWS1hljxjTe43FgXErKw9hRa0GB7cqaEUFVhMBfjvtItQe7NvskhhgPlH9yse8ue3fTVqMSzC+g== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:19 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:19 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 20/28] vfio-user: implement VFIO_USER_DEVICE_GET/SET_IRQ* Date: Wed, 19 Feb 2025 15:48:50 +0100 Message-Id: <20250219144858.266455-21-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 6ea53429-fa1a-4390-e6f8-08dd50f4bae6 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: Q1OtlZA75qcirFLlk15H4TsfFk4vc6ny86vNCGkGKzezZCHmUtepc3WcVyxkQp/ia2i4+sDTgvNeBuNmgRIqBlWnRe2U/jND2ooYcck8hI37VPev6PUyBOLFfAr7vYb2Fa7ogsFQARpm++WfRo1OzcWDBqfSjMEZv0YxZ2OtDNSRCx4h+HemGF76cnTo2+fvyYrxR3AX1a1cxFzz78RMaBz/2yaHB8rO79QSmaFLj+DwajTABEiprSFjpiKQ55C/l3jEOwkt+Ovj5XOkiMWBCFhhzh5nUuD27+ziqkNHhO2h8CZtA3Z6SEVDg5l+dwKVo3KTsq30+bZCQFFsZkQadLeWDMiTts4dDRa90KOzkMRobNOGfLSdvGOIvlsb6JHxxsW/v0rZ3DZYRq6Pr5YCO3EXvwczMdTw05pCApq7VKW9wxQp7MlLI1eRYgm4s4watFVVvNkjPljTFJAqrJmQ6CJyLZ3Jhw2/EZ45tKcxZlJ399kSuwAlbuf7g8xXuoKccqO5JBo4BD2qqaBtspL7KPDg5rd5MC3mnagCxIz0tPRS3/lS6fncXCFMjLLnEC6bz80H0JnewHOQ03CM2n4pG5MJ/o8P6nNVcaNRiOGZ6lgVja7qFtRCDKBdAQVOqJZxamBdutvHdac523BeqhhhAsJPxSLSRE6T7hltoIAUcjVca4ZCj6GOZdrL94RFkl57MtWuCQTlFqRLlb19bWTBK8fqau9ElhtDDewBiF2LkyWOS5z1eJPCE3C1CtnfwBt3JBVCVjnS7Jpu0ptN1d87deMpJs6medJhEpe5G8r5kaKMb8g2fbjI8j52gBqUywX6w0/0ji0GUtVH/gPo+W5yvO4mR2WULkcpZ4XKxsnizdbjE4U7+QzzMZFNnLAbvaK9a96L3Ao8WaX/zEmKePuZS7Zm89WITTGt9qq/2fjmm9cZ0tH+2uQ8zeomEKAW++OzdYsu7qK/aEZRMZfqFXetNpyLKQz6tPpJCw/MjirBPxWEJX0VGFM/WQ+vVk+yKWhCL7kqtHSr/m9eRoi2tkwSEhax2fWyRdlDGIPzSMZdS7W3O3/yRTASPlBUClbg96+3EDrQDwaXnZ50S5vVGtlZadaPT5za2iUxt7VHmRedEFhvp2zUlTD7EM4yVPbnf32IWxAwdjBMdehtGFwPuR4KKotiaeEH4p6pEQoXyxY2loqCflV2YyJJfHiHfI6moZkinIQY0qKbAFSiCkhg11+62UQiFUpBLrxUVrgiAwB8pjTSX9cjFLAkeCO9gSq0DDFZGByl8j0OTJ37io3i1W30RfTXY8z79mreSdlOo1suxhznqO1xuS54jaLI5NrLq2K/+AkTF43+/I6UpxcyvYihbNU3HcZ4KTmX+sphLNccK+4QzkfnWin6EdvxbD4N8zJD X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: faGR9j43ijc+Evj5o50dOjR2mroAi3MLxX/aZwnvS8c9+yMLXc+DYDJu40cUzOn0B4xDJui9oZyOH53wxIhIF+RmN5I7y94dS+FqjwPxpnszPg27dAhAziLmd8v6J34ComWQNPPMuZ+IyP0EHyxiUYilFW+dKsaVqod2Px+T66LUaZ3BkCuebN9ai9OStATG4swRUd/uJRMol5wXUZk0KhZtm/f34GdxQYnzh8N3aiBkJhOYIW+azZtAl3CU0s0ogH58NkeL+tOE22SAUvJEnDgZk6ct3HGfbh/GGslDwsAHaNgbv03ClBawaO/f3IK6dK36GgIJF7WI5w5qNtTWC2GRlsUspRMfghKaTn5DsB/rsjgik3JfhI8zdB71HqpLWDN9b385Nicy/pduxZkGd6DeI4i8adFtSiseQmvtHsm2evPfqT4KfKq+l9tlazu+yWSPjJuCl2qFs/0i4qdTVbg9sG0YKWHuGd017One49XNw14LhoeTxJ/shLtYH7zwKd4jSK70ZNVfmDlKqVJ5TJIOc59BSEs0/QD62nve5KCFfnPH8AzR676rDN+LAVl3LCXhz0Izj1ZOqpzeXoza3QKCOBRShWw8x/lM0h0XxxEtoIBYh7V4/wVLDIVkbvdQSdvxDxkXN9yZy7rstPIDGwxb5ZuSqsGu5m0k0NBPpv4kyUp3MJ/xyqYUJ46zVKYoVYEypK1GNCPH/cdJRIdt3D2kZS1eMIqufpOgFl+buyml7CA4fFXbNUpWprxAD5GOfywQv7WH+RgAXnJXmgOvEuLjWyot+ky+WhNcFcdZofchGOLC7KLNuQqTLSsCpy86ovkj31JOB29p5h9Z4Cc343q1KSUso5OAPezuXaGqnCcfRGw0CFYkamQyEXjBiwXPN06CcyvKClVK9uUb96iAxraSr3ygrWVxF+SU1StJP0ZOTsHlUA6khTSdkcybyYBTmw9cbSoG5vq3ynOZ5uM2hMM5V21YjdjURIJvxpTLXxwCHSym9IgTn/iv41EpBxSHE8qsL26C3kqxQwhVCLji/8xAFx86ymQF5yf8adjYHBgT9aBXGwZ2Ao3kax2hjgZEy5ZuF687TGoS8p86nWs4Pbn74uFiwuJjAceZx0/w7w/LRiu4Gx2gqt3tLFdVOd69jxLEcE2S3YbUD1UE2HezU2P+LtlHKmq14di2vzT38RCq28tiQkmpd7JAbz9Q82vJF4BBSePHMz8EUuVE4pV5ZS0G6pJX2b6dI3Tn/eC1waM1IT80jbvyKx9976gw1+i6KmpMqUbAIuKO6mUgXA6Sv4V/vHxK2xU9cIIAP5HdUrdvOA9q634pVIG4CSTcRyZlOVzp+XBXMclGv23/xONB6jRcz/T8ejonPYp9sZu6g/5zGvEJ705bpv8LDaWi9TEz9sPmaHb62DXW2gztrGdY7TIcYwjKKtQDbOBKDjT+wANZ9SMaezMTP+M6t6XqC5XnhsWwAUKJ8QpT7tjMEQcKYv9vd0HzfIgTjVymTGLJPmW6SzB0Cn/3Vf57hWqybLR4pwNsw35r0GTbIAAYQl3Jj8tkJnKJYBMsombWOaZoL+IzsCe7KPdmS6Orm67VT2Jm X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6ea53429-fa1a-4390-e6f8-08dd50f4bae6 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:19.6682 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: NHJEBrDh24mzm2RAoPL80s64IMn+IFfiK5DUEkKQPkZN4CPf22Jya0YF/3tuw3q1vBXEq0J1tqIWwEZ5pu91qg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: 1y8wkTbswxcA1klijLwUveemjvN00k8h X-Proofpoint-ORIG-GUID: 1y8wkTbswxcA1klijLwUveemjvN00k8h X-Authority-Analysis: v=2.4 cv=HMQ5Fptv c=1 sm=1 tr=0 ts=67b5efb0 cx=c_pps a=bqH6H/OQt14Rv/FmpY1ebg==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=eeJ3FIb8qCAeb1asaDIA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.155.12; envelope-from=john.levon@nutanix.com; helo=mx0b-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman IRQ setup uses the same semantics as the traditional vfio path, but we need to share the corresponding file descriptors with the server as necessary. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 140 ++++++++++++++++++++++++++++++++++++++ hw/vfio-user/protocol.h | 25 +++++++ hw/vfio-user/trace-events | 2 + 3 files changed, 167 insertions(+) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 1a9033af5c..182ef5ab8f 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -1179,6 +1179,122 @@ static int vfio_user_get_region_info(VFIOUserProxy *proxy, return 0; } +static int vfio_user_get_irq_info(VFIOUserProxy *proxy, + struct vfio_irq_info *info) +{ + VFIOUserIRQInfo msg; + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO, + sizeof(msg), 0); + msg.argsz = info->argsz; + msg.index = info->index; + + vfio_user_send_wait(proxy, &msg.hdr, NULL, 0); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + trace_vfio_user_get_irq_info(msg.index, msg.flags, msg.count); + + memcpy(info, &msg.argsz, sizeof(*info)); + return 0; +} + +static int irq_howmany(int *fdp, uint32_t cur, uint32_t max) +{ + int n = 0; + + if (fdp[cur] != -1) { + do { + n++; + } while (n < max && fdp[cur + n] != -1); + } else { + do { + n++; + } while (n < max && fdp[cur + n] == -1); + } + + return n; +} + +static int vfio_user_set_irqs(VFIOUserProxy *proxy, struct vfio_irq_set *irq) +{ + g_autofree VFIOUserIRQSet *msgp = NULL; + uint32_t size, nfds, send_fds, sent_fds, max; + + if (irq->argsz < sizeof(*irq)) { + error_printf("vfio_user_set_irqs argsz too small\n"); + return -EINVAL; + } + + /* + * Handle simple case + */ + if ((irq->flags & VFIO_IRQ_SET_DATA_EVENTFD) == 0) { + size = sizeof(VFIOUserHdr) + irq->argsz; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, size, 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start; + msgp->count = irq->count; + trace_vfio_user_set_irqs(msgp->index, msgp->start, msgp->count, + msgp->flags); + + vfio_user_send_wait(proxy, &msgp->hdr, NULL, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + return 0; + } + + /* + * Calculate the number of FDs to send + * and adjust argsz + */ + nfds = (irq->argsz - sizeof(*irq)) / sizeof(int); + irq->argsz = sizeof(*irq); + msgp = g_malloc0(sizeof(*msgp)); + /* + * Send in chunks if over max_send_fds + */ + for (sent_fds = 0; nfds > sent_fds; sent_fds += send_fds) { + VFIOUserFDs *arg_fds, loop_fds; + + /* must send all valid FDs or all invalid FDs in single msg */ + max = nfds - sent_fds; + if (max > proxy->max_send_fds) { + max = proxy->max_send_fds; + } + send_fds = irq_howmany((int *)irq->data, sent_fds, max); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, + sizeof(*msgp), 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start + sent_fds; + msgp->count = send_fds; + trace_vfio_user_set_irqs(msgp->index, msgp->start, msgp->count, + msgp->flags); + + loop_fds.send_fds = send_fds; + loop_fds.recv_fds = 0; + loop_fds.fds = (int *)irq->data + sent_fds; + arg_fds = loop_fds.fds[0] != -1 ? &loop_fds : NULL; + + vfio_user_send_wait(proxy, &msgp->hdr, arg_fds, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + } + + return 0; +} + static int vfio_user_region_read(VFIOUserProxy *proxy, uint8_t index, off_t offset, uint32_t count, void *data) { @@ -1275,6 +1391,28 @@ static int vfio_user_io_get_region_info(VFIODevice *vbasedev, return 0; } +static int vfio_user_io_get_irq_info(VFIODevice *vbasedev, + struct vfio_irq_info *irq) +{ + int ret; + + ret = vfio_user_get_irq_info(vbasedev->proxy, irq); + if (ret) { + return ret; + } + + if (irq->index > vbasedev->num_irqs) { + return -EINVAL; + } + return 0; +} + +static int vfio_user_io_set_irqs(VFIODevice *vbasedev, + struct vfio_irq_set *irqs) +{ + return vfio_user_set_irqs(vbasedev->proxy, irqs); +} + static int vfio_user_io_region_read(VFIODevice *vbasedev, uint8_t index, off_t off, uint32_t size, void *data) { @@ -1291,6 +1429,8 @@ static int vfio_user_io_region_write(VFIODevice *vbasedev, uint8_t index, VFIODeviceIO vfio_dev_io_sock = { .get_region_info = vfio_user_io_get_region_info, + .get_irq_info = vfio_user_io_get_irq_info, + .set_irqs = vfio_user_io_set_irqs, .region_read = vfio_user_io_region_read, .region_write = vfio_user_io_region_write, }; diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index 6987435e96..48dd475ab3 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -139,6 +139,31 @@ typedef struct { uint64_t offset; } VFIOUserRegionInfo; +/* + * VFIO_USER_DEVICE_GET_IRQ_INFO + * imported from struct vfio_irq_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t count; +} VFIOUserIRQInfo; + +/* + * VFIO_USER_DEVICE_SET_IRQS + * imported from struct vfio_irq_set + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t start; + uint32_t count; +} VFIOUserIRQSet; + /* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events index 3f5aebe7ac..053f5932eb 100644 --- a/hw/vfio-user/trace-events +++ b/hw/vfio-user/trace-events @@ -9,3 +9,5 @@ vfio_user_version(uint16_t major, uint16_t minor, const char *caps) " major %d m vfio_user_get_info(uint32_t nregions, uint32_t nirqs) " #regions %d #irqs %d" vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index %d flags 0x%x size 0x%"PRIx64 vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d" +vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %d flags 0x%x count %d" +vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x" From patchwork Wed Feb 19 14:48:51 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982385 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B6F2AC021AA for ; Wed, 19 Feb 2025 14:50:51 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPH-00047t-Uf; Wed, 19 Feb 2025 09:50:35 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPE-0003v0-57; Wed, 19 Feb 2025 09:50:32 -0500 Received: from mx0b-002c1b01.pphosted.com ([148.163.155.12]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklP9-0007a0-OU; Wed, 19 Feb 2025 09:50:31 -0500 Received: from pps.filterd (m0127844.ppops.net [127.0.0.1]) by mx0b-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBNXdE020678; Wed, 19 Feb 2025 06:50:25 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=iuKF4UJzddEc5h/Rz9H7FfhJIcYYfZ5IJCF3eIRC8 G8=; b=alHUR1j/ARXHw8TSnxfur2ay03gIhAfCTj6GA5yqsRB3We30eqcA9AKJ6 17kUbgomuz4OmZ7jb+QfICkwdsuf2HxZUm9XeX2WcB1aOuvVJ2l8V6kf63FJmDYx FxgDhvHrZnnLc6uvFdNoiAXHWkgHUceQMcCQaHNlugSG7t1YRu/JPYritXzWWd2V ROCEibSY4BtRdCzq4tevidh2gI6BOL9BTDxXLKHjn4/CGlpXA4p92Ua9PfWF8OIL 7tNX6eTG1P1ER5zrQM3UCxwoSrLsHn/iEuedyqShEm2XfLzchv0dKUnPdfn30ohO sW6SW8EsixEr504TPBOt9E18Pe89g== Received: from ch1pr05cu001.outbound.protection.outlook.com (mail-northcentralusazlp17010007.outbound.protection.outlook.com [40.93.20.7]) by mx0b-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4b9hyef-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:24 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OadSdvypRCHlHfDwPTRLmD9sDnNx9sK12nunA7wGfnvqK7vjHhFPswd5+NsKTkVzswnxHFVlbDr3GFB9L8II+SXW3RROOS73g9lV01WQW+0VCuaElE7PqoQJY3E7+fTOVJyvPBeal28xSGbTpm2jg3iSGKcX26TQ8Xqr4pZv+0X4Rba97t6ZjuFheaKIfiHOJ9V55YUgRDCN1+/CrdRtzxLFCFppz5HBhSXURiP5Iw8zkhEb362ve28ym/QH8o3I4dITYV8IwDiTYjm29palj7NAn29yFHuc5mqbV6Jxb9oY7dYWQL3d4/SPxCYOrnhmCrZM9cv14XByGpnsFGdOhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iuKF4UJzddEc5h/Rz9H7FfhJIcYYfZ5IJCF3eIRC8G8=; b=oILdIlqx9PylC2f3PslYegG++rLAkXK+fCQt8Ps09NXwBaNf7sdpz86GrpZlGVUFI/qJnButVCTwgwDqdGb4DHG7+BTPNwaW5yhqBrb32qBcGY/cvKnFCzqIl60i47E7kix1G7BjF9Pm9Krlw3HE4K44tlYinYcWgXp4WW1np5wIytJk05dit6IcknHDGY7/Jl2EqXCtw6jzAj6chNRd2fOAVNR6HqZyC+g3/2NBZ6Q62VDHucyQRY9K2dnD1Sqb565oNFjmXp54wkdzT8S6h3NKX8DjriWbPBuJMAoulD9i63IK10RPAdxc5ghXyS4L0XgKqjB9qhqbmz+iWt2bPg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iuKF4UJzddEc5h/Rz9H7FfhJIcYYfZ5IJCF3eIRC8G8=; b=sCuSxQjFTrrHMjw+Ec6fGd+l8Yd3Rk7X+v1b/0AFNcoSBLOECmGjY1uCzl4Hd02GHo6JiPbVBzUB49yfIBs/N2mjI1hLESp726N5k66KjKCTcJWJtiXPrKmCegvfHjvd69U373VKTzJVkshteAWZIT4NyXlLhpxWmk4pi+FGq7jyp6ius8K0x4rXV5y2MpRoVyg70zl+2cMf4WHELoICSESXSprpM+doGfQrqozbyfg9DMgmeuCpzAWiEU4R5ibVCUo0LXJyYiL3Nu913uVr5Kef592czAdXd4IP2iaJj14k+we78TTsvATwa5g763IsjpKyTSzcN5c2pTsq6bt2Eg== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:50:23 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:23 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 21/28] vfio-user: forward MSI-X PBA BAR accesses to server Date: Wed, 19 Feb 2025 15:48:51 +0100 Message-Id: <20250219144858.266455-22-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 2e59852e-f461-454e-c345-08dd50f4bcf1 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: o9fQmzlFNM25FtbRaS2sGlkn4iBKXOKNiFMmD5dAU6q4V7plBpxPS0lsBJTmg1NDoNtbGgcJrrRhw/BHFcviexpMLepkC3vmTQzv0lwuuZoMTM8Soog/SpdCGTNiX2Nmp1++UFaECkOBAtG/7DTCeBvJr9fThAZqhoW7KJJpXUYvEjFXxsZWZNTBmEumLFbgO6t1KaSjgGGC/exSE+1KwO9fubVW2Zy1GVoPRBVn7goFc2R4ultjdyJfUcu7iPbj/hfGdM3HP6sElJ1golDNNC+WnM42UcE4K2BwelNHY5dBkO7dh1RiNrcHLcOjOjrEVsurXdLI4a8rpIwHL0m6JNsynVFF9c+6jj71tEHKJDXGC1c7o8xppMA4P+77hbrzLbktvzmc7tfFaPxL9A6cBf8Ewd6PeBAIg7rGfiKpySI6fIh7vA8dy91hfO5j89Ey7JedufDaLviO86Wwb8RB9sw6DKo/cdz2INneQUyabD+Q5YdiqId1FJia8LjE6phE0uFgjlvOi4bE1iklaf4vZUdRooDEafzUipocB4TM6Pyo++SqKn1CC4C1olU+Jeozmro0gTc5u0ZTqPZacTB5KeChsR7Dts/HqeaL780jV+UFhAeT+wKSYP+TliCJ5miAeC5kEWxxIar0f7j0zaKxtdZZG2IzSJ7lI81MiM+XHxhZl3QfpmCRKSUDBk4t9lsIGCQ4KLo6b4PuDvcCrWZjbLQaoz1+8pCcJeoYEqAUxoHU/8lhphcTVAojDyGi2dyRdwJrpcjI9eCUTRQATiebRtSLNlp4vWFX5AXq1uNecEwJZzUYN4QkICOHxgFl/mjfKDLWbA0Guuo87/ewZyXMxjS2fE9aEetbqB1/nxp/1f7jqKhbw/K/Iyayt0WYkqk7Kp4vqlGapj0eBBIsMIgYrV8Itt0KWPS2QoizxAcjQBkpT62U4dp0FSyQPS5NISYBkXXex8+nHkt/aEfL2BYf6aVKVs9DeBTnl2DS97QcNgte74rsyrV73kKJuvwlM4L972F2EuvKwXTy+N/sZ0GRdd8TvAPP6vdiDvmFRQXyWRmONtkK9QGFn9sGRINT+BaBnd6jvcwph+23ADTR0qxlmIv/F5d2DFFQYz3iDU5tV5QE7BDAYvRQ1a97bw481W/D/7zZxm0cLO+SuwO4lMlL4EOxu86GUdpj3kxKf+U4N4nq0Lf6ot5wJD/BHeTJPqTGuY94kfvIKE5u70Aw3UaPRYlBwtoYehKsNA+b/U8B+GqDL4sLkT+J2Conl01X7xBBEQPTdnxwB7QlieGyUg8i2IhDHlf2tZX2e7LpwcwNrFq/w/hnobVDmFS0XvECP9yzzHI+TcLj0RF4GuGUflRYqSygBInFBE+FtkMwT/VSinVL0Ks7KBR1evS1oOBE7dMP X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: w3UHfyra16MDnIvIALmMwdQzsgIZSSGp3EMkUrZrH6kTuH9mAb6pmftE4PF7iDOXwvw6P8byKXPR9wC+gnm9fOGeBde+Bbzt6ACl6jS6gTR2QRpy8lrHZrrcN/mSlT+3ODZGcoKS3qwhJiWHPLsj4kZ+4NywNzzidldW0/pmjp2oX9+KVN7AhYR0SzBzRBj3zJnnUHX4vc62YrfjZHE/xtwdqLfQoI17ILos1A7/20p37jh/8GBNzml9fIgfckBntL2DxPh5weZuTJI7NmyGh+r+U7UXotm1aZ8e1wFgYFV49Mq0O4HSTWW8Buu48JVsnf9M/zXN4z/2B18vCsw1Yr+NQIiBohi+xFzOSxGb2+UO2+naec/Pmh8WTRd9PLPOcxBsLPByG4KqGxQT8K3amOgM29gHv8abuvJd9HnFEUyf3xpxPhPhKUNYM9jnpn8Jbf25oCBd8JW1InodBnhvZfd0zlNhd3OaelYgVYM4fKY+9K4zU60bdiUQh1otS1c9IvqmsVbXFzDiELr0Ilqrgw7wlYfKo3wNpvlSsBXjkRGlswVuhZg2zuU6yqzGlLmu3JXzl8IS/xGWwfdlyrjAshyTm4Mo6F5lAaiOXQOMj705VvklksrXwKoET8oyh1hO/AJDhyvSs+oNYEZTqxuHw5tU3q+gFGVMwuhZcgD4rKsItb/LSNxDd9RR3AqjnuEtdhZQkyrC11h7VsfPTihxQaLSMt1aa2a44UprdMujFfE0Lsx9JOKB4rR7M2EkYP0fHf7j2G65Y03sTCHOk3CQVwijPws3jeV5cgRL5BfoY5Hzk7YP8yzjoD0wF3FOzFcixN4lJi5+DJ5W6RLbQqjAa2hkYt+6j3CkaB4RsDAFH2r//1VQe9Q/x3NP9aiHx6X0+1SdCtMM43Mxqyi9jTGFlWkxnZzIrtwzjmGNmvVg+SgDXDJO1eHclssJkRSSNnl2ojENKakJCpUrDfOx55XBFCBZhzlrlmRaiXBJ2k8DoTJZ1TmRNRMH+gfrsXgY77+xsug9eaynlfcMYrDcevzOJ9sO+wx7eLNnguC8eWfjf05m4syC7XOwN1GvjJAyscuajRy6AcPT0QzPzsseNe9nIAiGDPhn0N8946fEt1kY7pcyrTn9H4QJ9I8pkxgfMnZWhnOHDFW08DKy2FK/k6nL4GVHaiALFnr/+Mh8plQLJUGF4CxEGst//Krk/ghzzZ6UsQRPz+XbASNHdx4Z5UGi1vVy/wyjVO7freh9EFnFHc/jIYvvmfdM6k2UmJSTCooGbRm+UYsYbkT5hZoEfHcyQGsxhuJy1OdHtmmifutLcCaql5pCrshAjZaq0H4yNNjrWM71Ja+W/Uf39vqTph68rAXTEqtq0TQcKA6jnppjbeCfFOAs258jHaTmwOLbTGZeNi8nJ5TC9agAM3aWae0IB3vD+b/iqZqckyO2NqeAkBHcu+a6ZJVKw8KL4/aUL/V44ahQWIc3C2EU+0xNd2j+2nD5TRrDADjaFUpQFgYk7I+a120PyJlELMhZ4IpGsK3OcyST1qC+9eFuIqFqGUb4y+K78+811kTUJq1TpTxm4TmVzu5JopEabXikkuKHLea9 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2e59852e-f461-454e-c345-08dd50f4bcf1 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:23.0169 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2Odg0gPqYUyX7iKB5oYFyI27lHhge4M3tHWAnwj2jV38hdxH5IqPxiHfdzW8OJLGv/cX829rLIaygqOOTVTbtQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Authority-Analysis: v=2.4 cv=e7j8Sbp/ c=1 sm=1 tr=0 ts=67b5efb0 cx=c_pps a=2D6/CIrCIWs5X5ruZf5FWQ==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=1d_4kwcE-g3AVD5dyDkA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-GUID: W-dsrnnRVv7eEiwIPZ0Tozsbgy52MpYr X-Proofpoint-ORIG-GUID: W-dsrnnRVv7eEiwIPZ0Tozsbgy52MpYr X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.155.12; envelope-from=john.levon@nutanix.com; helo=mx0b-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman For vfio-user, the server holds the pending IRQ state; set up an I/O region for the MSI-X PBA so we can ask the server for this state on a PBA read. If VFIO_IRQ_INFO_MASKABLE is set for VFIO_PCI_MSIX_IRQ_INDEX, record this in ->can_mask_msix, and use it to individually mask MSI-X interrupts as needed. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/pci.c | 63 +++++++++++++++++++++++++ hw/vfio/helpers.c | 26 +++++++++++ hw/vfio/pci.c | 86 +++++++++++++++++++++++++---------- hw/vfio/pci.h | 2 + include/hw/vfio/vfio-common.h | 2 + 5 files changed, 156 insertions(+), 23 deletions(-) diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index b8c1cc34c2..cf1e642399 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -40,6 +40,62 @@ struct VFIOUserPCIDevice { bool no_post; /* all regions write are sync */ }; +/* + * The server maintains the device's pending interrupts, + * via its MSIX table and PBA, so we treat these accesses + * like PCI config space and forward them. + */ +static uint64_t vfio_user_pba_read(void *opaque, hwaddr addr, + unsigned size) +{ + VFIOPCIDevice *vdev = opaque; + VFIORegion *region = &vdev->bars[vdev->msix->pba_bar].region; + uint64_t data; + + /* server copy is what matters */ + data = vfio_region_read(region, addr + vdev->msix->pba_offset, size); + return data; +} + +static void vfio_user_pba_write(void *opaque, hwaddr addr, + uint64_t data, unsigned size) +{ + /* dropped */ +} + +static const MemoryRegionOps vfio_user_pba_ops = { + .read = vfio_user_pba_read, + .write = vfio_user_pba_write, + .endianness = DEVICE_LITTLE_ENDIAN, +}; + +static void vfio_user_msix_setup(VFIOPCIDevice *vdev) +{ + MemoryRegion *vfio_reg, *msix_reg, *pba_reg; + + pba_reg = g_new0(MemoryRegion, 1); + vdev->msix->pba_region = pba_reg; + + vfio_reg = vdev->bars[vdev->msix->pba_bar].mr; + msix_reg = &vdev->pdev.msix_pba_mmio; + memory_region_init_io(pba_reg, OBJECT(vdev), &vfio_user_pba_ops, vdev, + "VFIO MSIX PBA", int128_get64(msix_reg->size)); + memory_region_add_subregion_overlap(vfio_reg, vdev->msix->pba_offset, + pba_reg, 1); +} + +static void vfio_user_msix_teardown(VFIOPCIDevice *vdev) +{ + MemoryRegion *mr, *sub; + + mr = vdev->bars[vdev->msix->pba_bar].mr; + sub = vdev->msix->pba_region; + memory_region_del_subregion(mr, sub); + + g_free(vdev->msix->pba_region); + vdev->msix->pba_region = NULL; +} + /* * Incoming request message callback. * @@ -154,6 +210,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) if (!vfio_add_capabilities(vdev, errp)) { goto out_teardown; } + if (vdev->msix != NULL) { + vfio_user_msix_setup(vdev); + } if (!vfio_interrupt_setup(vdev, errp)) { goto out_teardown; @@ -206,6 +265,10 @@ static void vfio_user_instance_finalize(Object *obj) g_free(vdev->emulated_config_bits); g_free(vdev->rom); + if (vdev->msix != NULL) { + vfio_user_msix_teardown(vdev); + } + vfio_pci_put_device(vdev); if (vbasedev->proxy != NULL) { diff --git a/hw/vfio/helpers.c b/hw/vfio/helpers.c index 6bc7600ab1..2ab30fa91b 100644 --- a/hw/vfio/helpers.c +++ b/hw/vfio/helpers.c @@ -73,6 +73,32 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) vbasedev->io->set_irqs(vbasedev, &irq_set); } +void vfio_mask_single_irq(VFIODevice *vbasedev, int index, int irq) +{ + struct vfio_irq_set irq_set = { + .argsz = sizeof(irq_set), + .flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_MASK, + .index = index, + .start = irq, + .count = 1, + }; + + vbasedev->io->set_irqs(vbasedev, &irq_set); +} + +void vfio_unmask_single_irq(VFIODevice *vbasedev, int index, int irq) +{ + struct vfio_irq_set irq_set = { + .argsz = sizeof(irq_set), + .flags = VFIO_IRQ_SET_DATA_NONE | VFIO_IRQ_SET_ACTION_UNMASK, + .index = index, + .start = irq, + .count = 1, + }; + + vbasedev->io->set_irqs(vbasedev, &irq_set); +} + static inline const char *action_to_str(int action) { switch (action) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index f85215417d..b9c7e13053 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -520,11 +520,30 @@ static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, kvm_irqchip_commit_routes(kvm_state); } +static void set_irq_signalling(VFIODevice *vbasedev, VFIOMSIVector *vector, + unsigned int nr) +{ + Error *err = NULL; + int32_t fd; + + if (vector->virq >= 0) { + fd = event_notifier_get_fd(&vector->kvm_interrupt); + } else { + fd = event_notifier_get_fd(&vector->interrupt); + } + + if (!vfio_set_irq_signaling(vbasedev, VFIO_PCI_MSIX_IRQ_INDEX, nr, + VFIO_IRQ_SET_ACTION_TRIGGER, fd, &err)) { + error_reportf_err(err, VFIO_MSG_PREFIX, vbasedev->name); + } +} + static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, MSIMessage *msg, IOHandler *handler) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector; + bool new_vec = false; int ret; bool resizing = !!(vdev->nr_vectors < nr + 1); @@ -539,6 +558,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, error_report("vfio: Error: event_notifier_init failed"); } vector->use = true; + new_vec = true; msix_vector_use(pdev, nr); } @@ -565,6 +585,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, kvm_irqchip_commit_route_changes(&vfio_route_change); vfio_connect_kvm_msi_virq(vector); } + new_vec = true; } } @@ -574,38 +595,35 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, * in use, so we shutdown and incrementally increase them as needed. * nr_vectors represents the total number of vectors allocated. * + * Otherwise, unmask the vector if the vector is already setup (and we can + * do so) or send the fd if not. + * * When dynamic allocation is supported, let the host only allocate * and enable a vector when it is in use in guest. nr_vectors represents * the upper bound of vectors being enabled (but not all of the ranges * is allocated or enabled). */ + if (resizing) { vdev->nr_vectors = nr + 1; } if (!vdev->defer_kvm_irq_routing) { - if (vdev->msix->noresize && resizing) { - vfio_disable_irqindex(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX); - ret = vfio_enable_vectors(vdev, true); - if (ret) { - error_report("vfio: failed to enable vectors, %d", ret); - } - } else { - Error *err = NULL; - int32_t fd; - - if (vector->virq >= 0) { - fd = event_notifier_get_fd(&vector->kvm_interrupt); + if (resizing) { + if (vdev->msix->noresize) { + vfio_disable_irqindex(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX); + ret = vfio_enable_vectors(vdev, true); + if (ret) { + error_report("vfio: failed to enable vectors, %d", ret); + } } else { - fd = event_notifier_get_fd(&vector->interrupt); - } - - if (!vfio_set_irq_signaling(&vdev->vbasedev, - VFIO_PCI_MSIX_IRQ_INDEX, nr, - VFIO_IRQ_SET_ACTION_TRIGGER, fd, - &err)) { - error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); + set_irq_signalling(&vdev->vbasedev, vector, nr); } + } else if (vdev->can_mask_msix && !new_vec) { + vfio_unmask_single_irq(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX, + nr); + } else { + set_irq_signalling(&vdev->vbasedev, vector, nr); } } @@ -633,6 +651,12 @@ static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) trace_vfio_msix_vector_release(vdev->vbasedev.name, nr); + /* just mask vector if peer supports it */ + if (vdev->can_mask_msix) { + vfio_mask_single_irq(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX, nr); + return; + } + /* * There are still old guests that mask and unmask vectors on every * interrupt. If we're using QEMU bypass with a KVM irqfd, leave all of @@ -704,7 +728,7 @@ static void vfio_msix_enable(VFIOPCIDevice *vdev) if (ret) { error_report("vfio: failed to enable vectors, %d", ret); } - } else { + } else if (!vdev->can_mask_msix) { /* * Some communication channels between VF & PF or PF & fw rely on the * physical state of the device and expect that enabling MSI-X from the @@ -721,6 +745,13 @@ static void vfio_msix_enable(VFIOPCIDevice *vdev) if (ret) { error_report("vfio: failed to enable MSI-X, %d", ret); } + } else { + /* + * If we can use irq masking, send an invalid fd on vector 0 + * to enable MSI-X without any vectors enabled. + */ + vfio_set_irq_signaling(&vdev->vbasedev, VFIO_PCI_MSIX_IRQ_INDEX, 0, + VFIO_IRQ_SET_ACTION_TRIGGER, -1, NULL); } trace_vfio_msix_enable(vdev->vbasedev.name); @@ -2772,7 +2803,7 @@ bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) { VFIODevice *vbasedev = &vdev->vbasedev; struct vfio_region_info *reg_info = NULL; - struct vfio_irq_info irq_info = { .argsz = sizeof(irq_info) }; + struct vfio_irq_info irq_info; int i, ret = -1; /* Sanity check device */ @@ -2833,8 +2864,17 @@ bool vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) } } - irq_info.index = VFIO_PCI_ERR_IRQ_INDEX; + irq_info.index = VFIO_PCI_MSIX_IRQ_INDEX; + irq_info.argsz = sizeof(irq_info); + ret = vbasedev->io->get_irq_info(vbasedev, &irq_info); + if (ret == 0 && (irq_info.flags & VFIO_IRQ_INFO_MASKABLE)) { + vdev->can_mask_msix = true; + } else { + vdev->can_mask_msix = false; + } + irq_info.index = VFIO_PCI_ERR_IRQ_INDEX; + irq_info.argsz = sizeof(irq_info); ret = vbasedev->io->get_irq_info(vbasedev, &irq_info); if (ret) { /* This can fail for an old kernel or legacy PCI dev */ diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 7e126822f9..db1b2010fe 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -114,6 +114,7 @@ typedef struct VFIOMSIXInfo { uint32_t pba_offset; unsigned long *pending; bool noresize; + MemoryRegion *pba_region; } VFIOMSIXInfo; /* @@ -183,6 +184,7 @@ struct VFIOPCIDevice { bool defer_kvm_irq_routing; bool clear_parent_atomics_on_exit; bool skip_vsc_check; + bool can_mask_msix; VFIODisplay *dpy; Notifier irqchip_change_notifier; }; diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 5adbff4fa7..cf2033d6c0 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -267,6 +267,8 @@ void vfio_address_space_insert(VFIOAddressSpace *space, void vfio_disable_irqindex(VFIODevice *vbasedev, int index); void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index); void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index); +void vfio_unmask_single_irq(VFIODevice *vbasedev, int index, int irq); +void vfio_mask_single_irq(VFIODevice *vbasedev, int index, int irq); bool vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, int action, int fd, Error **errp); void vfio_region_write(void *opaque, hwaddr addr, From patchwork Wed Feb 19 14:48:52 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982413 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5B7DEC021B0 for ; Wed, 19 Feb 2025 14:55:18 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPK-0004EQ-22; Wed, 19 Feb 2025 09:50:38 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPF-00041K-Oc; Wed, 19 Feb 2025 09:50:34 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPD-0007bL-Ga; Wed, 19 Feb 2025 09:50:33 -0500 Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBKnF3003929; Wed, 19 Feb 2025 06:50:28 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=95DOlyGBkcQkytLouJyPHU0w/0KpUKHO+DZafAk3B Y4=; b=WSlxmbICgOIfwDkR+8AFUBhFsxwWpieLFrVEvgxs1Qe6Q5cWAswrdrUS0 1yyZ5J5XWmuMl/y3q7k7NSfjh+bSG8GObC7PCpEXJJJAr724ghvgF4QSUUC/x/Qh 9cSX74JRt5gt7sHWVKvg6HMX7otnCAhrPhzuy4nW0ReaTpCEWorIeL+8+om2hXwY MQqCp53kkaJ04MM02ZWP566BjmMM6AlH7+X1YYsrnJdy+xPgUDHQLyiM5uZXKD35 0ZYp3Uhiarm9civIMCpHJgAjCvMgc9lLEhxYd97CZshCF7ShgNINQI3ZfJTAI3Ac /lBjLZHa8954sLxW2CFUw11JWCQGw== Received: from ch5pr02cu005.outbound.protection.outlook.com (mail-northcentralusazlp17012055.outbound.protection.outlook.com [40.93.20.55]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4bdhxyp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=NXq+JORqf9ni0x5HjJJpnZQwZjUAd3nfPEhFlBLwqQ7UYMSlqqYnMrONdOZahh8bGruI1YURK7irpdSz4fFBqFr8ZDOneFGhohDjB0lZ8DPJh9prMWCtETA3n6kqpSI/XCZzr17bNZaAJBl/ChPSlSuLDRDfJIBABz+YIdaXisjYgGH2RMq4+A9lM5aDaqxjisZe9u/5bArK7AR0OfhM9voyrcL8TkKVy6pvPCKgG5la0B4vd3oEcj5+F5guLpPhA1Pf9G9ru9eR2xw49aSMjlwqIaoh6MkvdRjwQCB0TB+yKe90G4+yNTDQv7Nz/MHQNuQEWVJ1oVzO6Bp7bY9+Ig== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=95DOlyGBkcQkytLouJyPHU0w/0KpUKHO+DZafAk3BY4=; b=seX+yk62dxkgEwHgwAHuRTg99SAJco/4z9n7+FJ8QZpSaaMOpNu49sKCLMlMmePvA5S+wN6L8oFBJnYdQIiDyKpCFaWZtxP4ZJC9OeZvaPp78l2oB/UwnlGrB8S1lCdmUxXyxHwXf/P+1qtKSAbXpJoSYZ+4ItRCONuXI5ZeftpkTQOZ2+KIByGk+31T3EAjE7uIYIVzzMi734QQoDmktS5Z+VqIi9YAa1DfrhCepv2qPvJ7LiIQ6CuD7tV6NX1tcIKMZS0oiMBUgxo2Ne2kTGNwxEY72dE1aa4ocVZmk9fx5NQphsq0Xs2n2nN4Z9N5xSM+S7JdYwH/V8fPUv8E/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=95DOlyGBkcQkytLouJyPHU0w/0KpUKHO+DZafAk3BY4=; b=RUz99GoaExPJTDAspNq0qfI4Q0PzK420M5x0H57DcvQNDTtl8VZ/SKzHhHnnIz6r3IufOKma4z5PV9rQfzRRq1mayY8eo7AvxXq+Md8FQSbj1ZYPPOhA9SsVRA/q41Re/g5fdCIsDiDCadQ3/WBeKJeT4+GWKepnj54nvlty/zQvfrgM0FrXMbDAH9GY2MsM3hVTjtWw3f2rIss6iIAS7xwZoNp7IN4zlgD+Ry6JyflAucZyN1BzzvzlSdAOMR78Y+2SaRbsS8TTCasYn3ljxrSb156UEZOWUt4TiqaQKVI3DL5b2xBpfmQaJWey+ezd41jLTDL89p1gvjkyP86v1Q== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by CH3PR02MB10559.namprd02.prod.outlook.com (2603:10b6:610:204::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8445.18; Wed, 19 Feb 2025 14:50:26 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:26 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 22/28] vfio-user: set up container access to the proxy Date: Wed, 19 Feb 2025 15:48:52 +0100 Message-Id: <20250219144858.266455-23-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|CH3PR02MB10559:EE_ X-MS-Office365-Filtering-Correlation-Id: 3dd78521-4f85-48d6-5753-08dd50f4bef9 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|1800799024|366016|376014|7416014; X-Microsoft-Antispam-Message-Info: fLsyA4gHbTcbn/kPUUbFNMdAhoieChIzsK/x/rS74iJQWRMrz0Cr9gaZv1Zmuz056CYbKiVpKzTvfoSepgV0faUqu3q8RHmaCpBYwnN2wnhmNTP/19DrYdwsTkbHG2QSln+xFB/fWoHszFZRMHWuBePes/YVDy5z+Sibuga0TNTD4zJR8NleUtqsFuJSx5dCbWwBb7PJDFKxme91abmVqlQ0nsnrtp2Uirxk3jGkSkXrPVsLTEh0djt/hfHicJP1gYaPOQFq8Kt1lPrShM0TGfIpQ8p2uUPWKvYhU4QvnJ/jzqa/yDfTja6kJWz8ph7rqIA9TDf42vmPYrKSC2XsIt22KiNPx3LKnosYsQcNN2pqPIB3sd4Vsud8XUjqlHyfBRro9Ku+Ne2PLc/9/b/8e9oQiY2IU2aHTgpCTpvlPPhLz2WANKtL5F9VzReM9fdyfrWpjUaEz5nW5AOVFKukRs7S7cOExvaAw9LMBuRPSq48zgkHplcWSVH5Ml5M6Pk6hnn7QR8jr94+ZFd3s6keKbMnpPyddq84bhftCwXWuwcafsNxiAZut8k7Ij8XUxx2OS0r7kxbHCGkRhXPJXi2aF1votOe5gWKw9BZt064kih24yMwHtTxcg86xF692B2MVKv/4sw/LcJJ9xyTsb0Li9YPY7/lyobpV5SSH60d7mF9VG0czNrF5q+yBDnnZxFcEK4IjF/3dfVG9TvM9OcCokcnozZ5oRitV43T2+8yioFXmLZkLspUqjNHH7wR0V2A+pZTdhu3TG6FW84x6mQdtgUlnAm4QKY8Gmh9xFl/cTOli40ExlpMOGtW2qrO864r6C1RqUhtNsjT784sr3qSRly8Y6oCMQt8dzibCsqqvMhuzGcETBnIn5d6N0CKgr2ixOMbli4dAkjylII89wSynhTwykSJBdRqIBXI9EeZRGG5zMvKuD8MGOnGWiePu9gOAwAPQHaAEbOhfeO7wd5qRBCXtce3bQP5K37tcIRCmRQX/Y2gz2ulum4mvcId9G/IH6if7UYAZbNNrxuzO4eWvKq3GsEhFrIjwfJbcBE1+Isl8M/3e63se+VezyS+IbOwkiMiIUzdI5DThnX4L8kjNOTNnGa2XfLs8bDNcb9JEgyqH33poFc8sviJxxLLjVXGMDVWSGYBSSv5/n4B1NvAUNbeQMWwy4Iha+WvQCTJ6fk+EZ4aukyeeikANvL875IHo+6xeqWb49+YYd/cSUL03ILy5W0elHtJpm37ZBx7za1kufYXhmK0VYMBsdskQPf0J9YyAgYNb13QcwP/P+VJUH5areN5xOJ9Ma08m/mY6faM+ES5+ozNFZ7RpTOD1zAr5iPTItLctW6Pe/de8uL2Q0bcmyaStMRSSpfafOFtUqpgTGySiZTEuf5xS914JaPu X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(1800799024)(366016)(376014)(7416014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sG3jOJtuaYnNj7Wh4g3rjc/0fNGdgTbTOcAhG6VFJA31qRBtPj7ojKzOUVSxSyOLV3eOlrmcbzgUikrBg6OMpIWw9ubjy905IAysJoOg0uLjjwG10UiubAoM0garGkoWkImnzsXBTq+79UGZCu/CbvWwMJz3yRoGk1UPoNs5oMwHx9v8BZtpIPntjs5+BxBuN6l6MZlrt1tVdbqkppJhhYaOvlWF6hp5plAnYM4OQrf+dPoyfkGGpj6eCyXBOf4mn98+uLGzTdXtVBTNimnFhp+vlwd/r5Z2QO3kHin+YROSP+uSSVTBaAh3Hhe2Co5JrczYYt8m+b5pm4+7ZHEg/tNDlgNYds1xX0Q+CCc/i1V9sU6Dcm/z9QUuSb9vfQ/sItn8MczF7Ha+MkFcxoroCkt3vYzmdOKCcNNrXPELvySDIeRldlpkky8sbtfFfLW13CDCeHZJY9MwLjBShFubxeoUYBRMhb0EozcRL4+yd+QV6c9PBRBuAq69+0rs7atvUJVNNOd3yft5PmMi3djbK2cY3zhm9qWaWRomGEQzMRS0DkhG00/s3zGOmSPPhSBoLPslIi36ZHB7SQ5tMYcNQfLzPeOPsFDAHFyuK2czTHW9KzNufmCTJ+ZBal5PMVpKToG3FOiYAiGy7xAi53jr3fNj3Tftl7y4MPbHcRTXPdFjmFZ0IrcBh1U3KfBD/7CEdyKMkEt2ElJ0ke3yZ7f+n0YrRcxVWKX8MumKq/3T2LljQxBwjMA92OnxJ9qOtFRLS0veN4OIUzdPpvB/8ltQUK2wnLb+gKvTNmfB385b3UnuJulaejOElpo0p8CT/ZzGddFaU/9mypfKWv0xyKSkpvhMjuUilhID5C2NBcFfILfi/9ufaUzFspzp7weDmJSieIQX6QJhrfIXC/S0Voq0o39ap2DWEH5OJax53OX0zzOoAj3Y2na+OYv9YiqBBHmEEo8I/lhx0vH8piD3DCBPyfjDYPtl5ymkuExxSJwTjzJtxiIGMcBPG6/Ybum/IrMlZr3dWMnCZpMd28cY6LF3m9ye8aEdgsDwgXJsrbBlFDdcDLR/FzRXRxZvwykqPRKGHNpmU8X87KYlh4FsHGGdxU7rRn0fzkDjpNfnPevz+V7q9O4rf+nG5GeeJrJP2b9UPSRdCiNb1CtuCfnAOdNO1J8Sk1euI85fCaBlRoapgtRGsKpMISxG1AIXbroaFhZ/ebN58LxQDiNMj5gq1Jboa1MD1ZRkD8WpEB3XoxdMELheJxjc3E4q4m9hOCeOWEbybn/bja3xACH9ofKkfLgKNR5gQc6DlqwQOwPjDtqOOyVE75/NX/gUsXH/pkPCUDKuUaVLwY/IG8Nvs7/9Gz229ZQhvnyZmoZWbqcYMrSKqdO9sNqCcFt1MNQrYrZMs+jwVCqZwrxOOB6RHQSVvkeBIE4M8M6fV5bpJI9RpfWeOuhq4GRRCE3cCJS0DizdqGuXxND7Uaz+hESelY4Vn5+utK4y5a8hCOTB3Se1W5BtdV+9v4QF9Vi32vyWKaAUUmZSHiFLHbD3+/2+fvWKlY5oTHEY5DtzVJe9/o8kAqzeO+5vEanMkq4hTlDBlcKcKKVC X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3dd78521-4f85-48d6-5753-08dd50f4bef9 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:26.4414 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: NVEUPDGti3PLOpWikE9sPxf6j2Dou/niS+y33VWvNyIXPjZkGWEYY+a5tF14x5ciuR+41o3rRKQd+Dt4nR59+w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR02MB10559 X-Proofpoint-ORIG-GUID: m_VuRw4OiAVLb7yJorMzZH4MGmazD0nF X-Proofpoint-GUID: m_VuRw4OiAVLb7yJorMzZH4MGmazD0nF X-Authority-Analysis: v=2.4 cv=Fo7//3rq c=1 sm=1 tr=0 ts=67b5efb4 cx=c_pps a=qdevHIbYfAzI1c5gbp2lUA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=z6ziuncRFgQoj_6Oz7YA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman The user container will shortly need access to the underlying vfio-user proxy; set this up. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/container.c | 43 +++++++++++++++++++++++++++++++--------- hw/vfio-user/container.h | 1 + hw/vfio/container.c | 4 +++- 3 files changed, 38 insertions(+), 10 deletions(-) diff --git a/hw/vfio-user/container.c b/hw/vfio-user/container.c index c079d6f89b..0c487dbb92 100644 --- a/hw/vfio-user/container.c +++ b/hw/vfio-user/container.c @@ -55,15 +55,28 @@ static int vfio_user_query_dirty_bitmap(const VFIOContainerBase *bcontainer, static bool vfio_user_setup(VFIOContainerBase *bcontainer, Error **errp) { - error_setg_errno(errp, ENOTSUP, "Not supported"); - return -ENOTSUP; + VFIOUserContainer *container = container_of(bcontainer, VFIOUserContainer, + bcontainer); + + assert(container->proxy->dma_pgsizes != 0); + bcontainer->pgsizes = container->proxy->dma_pgsizes; + bcontainer->dma_max_mappings = container->proxy->max_dma; + + /* No live migration support yet. */ + bcontainer->dirty_pages_supported = false; + bcontainer->max_dirty_bitmap_size = container->proxy->max_bitmap; + bcontainer->dirty_pgsizes = container->proxy->migr_pgsize; + + return true; } -static VFIOUserContainer *vfio_create_user_container(Error **errp) +static VFIOUserContainer *vfio_create_user_container(VFIODevice *vbasedev, + Error **errp) { VFIOUserContainer *container; container = VFIO_IOMMU_USER(object_new(TYPE_VFIO_IOMMU_USER)); + container->proxy = vbasedev->proxy; return container; } @@ -71,16 +84,18 @@ static VFIOUserContainer *vfio_create_user_container(Error **errp) * Try to mirror vfio_connect_container() as much as possible. */ static VFIOUserContainer * -vfio_connect_user_container(AddressSpace *as, Error **errp) +vfio_connect_user_container(AddressSpace *as, VFIODevice *vbasedev, + Error **errp) { VFIOContainerBase *bcontainer; VFIOUserContainer *container; VFIOAddressSpace *space; VFIOIOMMUClass *vioc; + int ret; space = vfio_get_address_space(as); - container = vfio_create_user_container(errp); + container = vfio_create_user_container(vbasedev, errp); if (!container) { goto put_space_exit; } @@ -91,11 +106,17 @@ vfio_connect_user_container(AddressSpace *as, Error **errp) goto free_container_exit; } + ret = ram_block_uncoordinated_discard_disable(true); + if (ret) { + error_setg_errno(errp, -ret, "Cannot set discarding of RAM broken"); + goto unregister_container_exit; + } + vioc = VFIO_IOMMU_GET_CLASS(bcontainer); assert(vioc->setup); if (!vioc->setup(bcontainer, errp)) { - goto unregister_container_exit; + goto enable_discards_exit; } vfio_address_space_insert(space, bcontainer); @@ -120,6 +141,9 @@ listener_release_exit: vioc->release(bcontainer); } +enable_discards_exit: + ram_block_uncoordinated_discard_disable(false); + unregister_container_exit: vfio_cpr_unregister_container(bcontainer); @@ -136,14 +160,15 @@ static void vfio_disconnect_user_container(VFIOUserContainer *container) { VFIOContainerBase *bcontainer = &container->bcontainer; VFIOIOMMUClass *vioc = VFIO_IOMMU_GET_CLASS(bcontainer); + VFIOAddressSpace *space = bcontainer->space; + + ram_block_uncoordinated_discard_disable(false); memory_listener_unregister(&bcontainer->listener); if (vioc->release) { vioc->release(bcontainer); } - VFIOAddressSpace *space = bcontainer->space; - vfio_cpr_unregister_container(bcontainer); object_unref(container); @@ -177,7 +202,7 @@ static bool vfio_user_attach_device(const char *name, VFIODevice *vbasedev, { VFIOUserContainer *container; - container = vfio_connect_user_container(as, errp); + container = vfio_connect_user_container(as, vbasedev, errp); if (container == NULL) { error_prepend(errp, "failed to connect proxy"); return false; diff --git a/hw/vfio-user/container.h b/hw/vfio-user/container.h index 24ce13bc2d..8a033d5598 100644 --- a/hw/vfio-user/container.h +++ b/hw/vfio-user/container.h @@ -16,6 +16,7 @@ /* MMU container sub-class for vfio-user. */ typedef struct VFIOUserContainer { VFIOContainerBase bcontainer; + VFIOUserProxy *proxy; } VFIOUserContainer; OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserContainer, VFIO_IOMMU_USER); diff --git a/hw/vfio/container.c b/hw/vfio/container.c index ddb86edb65..797707d0fd 100644 --- a/hw/vfio/container.c +++ b/hw/vfio/container.c @@ -909,7 +909,9 @@ void vfio_put_base_device(VFIODevice *vbasedev) QLIST_REMOVE(vbasedev, next); vbasedev->group = NULL; trace_vfio_put_base_device(vbasedev->fd); - close(vbasedev->fd); + if (vbasedev->fd != -1) { + close(vbasedev->fd); + } } static int vfio_device_groupid(VFIODevice *vbasedev, Error **errp) From patchwork Wed Feb 19 14:48:53 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8AE26C021AA for ; Wed, 19 Feb 2025 14:51:58 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPQ-0004dC-K6; Wed, 19 Feb 2025 09:50:44 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPK-0004Eg-4P; Wed, 19 Feb 2025 09:50:38 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPI-0007cb-0O; Wed, 19 Feb 2025 09:50:37 -0500 Received: from pps.filterd (m0127837.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JDJ7w4008185; Wed, 19 Feb 2025 06:50:32 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=wTeY5bgrk9IMtnV4kh7v6+m+mRcJTI0QQQetnoJiv f8=; b=17mOsGtqVZNRlfngUkMSlydM7Rlc1PFncLpP8K47++KM0T2oJwJWQhmhU 56OBiJNaqAL9SrZNDKqDYM+WfggjC0A8KpWsNc5wbiRodWOB6rQQd90bEIX16AJr SCJVTTJLM4dqXQ5VxtaSPDOoZEt6cGZ31AyZiFsWRTJsmMhpMOlX//ARuAu0rYdi yVXwOR3P+cMLbRhhA0us9cdxN9Jk1J0vYfmVzGWCahw6x332S2f3EXqKZRXT1zOF ThFgOH77HdPXAca5UTtVaqvh6hwntBQjWbBxo/zJtpu4dLMyPbftZjT0ifY5I9h6 9VRm2AW/4Qf6bN6z0tABYsqrOUTQQ== Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44wg2wg7a1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:31 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=DrsiLuEURDtlfX+4cAIQM72+WmKORFM/H30LzUZ3dHEv10tiVaBz014+/1iHWD83EesrHJizvr1C1g4xv9tD4SzAkSuVvEs7CIVk3Lb5zBFsRAY7JlbWqjJrbyHJKL6S5MzhsR+wRUlypT+EsxjQql+Zwk7R8nu9VqcBNc3s7U0ovXEvjWlSH/FAPVQ5iqI2Dh7e8O1je7dzeROMLbV9MwqONbKB5PXtIZSczM7TQX5MfjR445gpijhAce9VXJ4TPb4So8ARxWgBkdYeLFmetXqoVsusxyry5K7YKeHG86DATc2X24qcH24Cym0DXMJqUvrBDyR4W/dzIQrAh/O1sg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=wTeY5bgrk9IMtnV4kh7v6+m+mRcJTI0QQQetnoJivf8=; b=EGzXt0iFGghtNol08R3TnVHCML+9ZddG35jgV9wI3jl8xvhwN0T0e1omUOQhkYjX2VHynLhJJGi9X12KmIDieWIYmE3pVNcuFt/1mExbW6MPKtAZMD98Ffqg1kGOiRgRYFpxZ0FPAbLsr4LaW9RVYQrTIO6E+N1GEJCphO9qAj7OVpdgOTFT38VIMcQkCQakbkv6RAxVyLjYR6JZsHMJA87Kcvao4LQlNidkSEG4ePYXprLQqvLR0cEZiGdVfS/PZYBgyzR09Lfp5tTmXQN36BZou2Yl3HGlrUkZzNB8e4bcZOfElyJaTXfC+BcaGzwaEk3QNa3UpAnIv3e7Z/B8ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wTeY5bgrk9IMtnV4kh7v6+m+mRcJTI0QQQetnoJivf8=; b=lGRlDmK02YMv2ogMc1fG3qR7RK7/+XGyqG5+VszMXs1/H1imWiUsZMEo+Jx5J3veru1EK9kkgUOIersd69w5RE6P9sZv+owOXCteWx0o+MFYJ0oB6hN9YfWJDgNG704OUfcSn+WAOOJVJS7lVHBK3XXX/UExZXL9gJwC1WcaPdkb6tgsRCQmDoBzwlfhGKIx8CpHbWDRvVfRSjGnyjzNyTt2viCoaF+364D1xkOh9C6Di0rfi49ChZDAGK5xsN6WBB+iLBR7wRLuNOLq5rW2NSk8FLSco0kUQ05hX0/UB9fQJvnHHXtx5eUmTyvRiuaotB21PT1dBMRwBODrOCWTuA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:29 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:29 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 23/28] vfio-user: implement VFIO_USER_DEVICE_RESET Date: Wed, 19 Feb 2025 15:48:53 +0100 Message-Id: <20250219144858.266455-24-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: b3c62de3-9d84-4009-ea96-08dd50f4c103 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: eAiI+gd078B9fUNXBwo0bi+lBaq0VxAW/yB/dHuu4VL6jmXxvpC+ckMNCIESORB0pGg8EM30/Xnahp9qYO5VwMsdgzqKjakWaug7LJt1dv8IgxCSkxN45lswvvWDhMQG3+EUtw9IEHIe5C4g3+uKA5ns+ExwrSMU/KR/MNHQ5xXEG4lCsqGCflvrDHMI6ByKd1jRTCY2XyqqSZUUdubIMYit138+d12w/aQIbn3259GDLbk5CADSC8bojDXzrEmI/a/GsI5ixDLpj2Yc6Uss2bLMS0o/jkgvJ0VHTvV72Mc0MzDaVbZ4TZ2g8xcmWY7kNNfmnSyai/RgPa0jN65pIjVAd1AMNJutSoZ/eVOYvUJvVcqTBu6tqB2q04x4Zy3wA9bPBLL4+6+fpUJBaxLZ2lzWuuGFofyONAyxxzztaYlVqmI3tINiUzWx147t/vv3Ud1oCN3Id4tkV/CqlpGjfg2cF/fGyPvHv6beDJ7GYYkNdquO78coeV/2DDaP2QRpFHMcZSTVWyCLju7LQcTBKFTCFIGtXvWceXtTB9NhluAydizXx1fNaf3C4KIQqI9P0Yzb84xQdYJAYiKPWHgZwAyMxzcHz5XEEGtJw300Yykf28+bq/RldS6fKuIKE16YyGEL6FV1nITR7qT7WQoYxJxY3nWKsT2jtgQ+/NlzdUhh0uNDE4D5bd6tJWNlcP29y1bqSnnnlkWSmi+dttYauDBI7U/eIKP+o9VGj35MsrTwgcYaLz+CMWir/Pbs/ciFRoA2EXwo/j2VKd7rCDN7CUXxUDMs4u5HNr3YcEKVEADu27CaLUP+4FhYKqys2Uv2Vrh1na4dhaB1H1aisfiVL6ohHG3HEYLdGU8Mw1NJ/AzJH9V+iXORtSYM2z2y1EMo6qAvGFjT9/7gLzksjd9ZiuQdeFSfZa9QyKmcjXLut+PN/MdRSOrDXv/QJ6DDaoWCPy/sJG0cLz8mj5+PY6vY0Pc9sT9b1AGoQniW3rg+4TIw/IKuR33OU9xwPBRxBAaOgDj18wiask9x3BbHb7WZRg/pvkncHdISWrLf/tTMvXIp3jZNdXHZNTNI7TdYXS1txa3J6Xgp41jGt7IFM//B1lnJJ94wsYVKTygZvttQEdDdOOZcHLYwWFU4EWgPqx78di5yG4MeRO+7h8DklVyM1UPxoAuk10OpuBDFYeyTDzBhXmQTUuUxu2v+yBjkPEqx/SYVDrMSPfnY1gIJhWVXmWYzZVzLlvX3XpA+Vg/AynosrBir80lbxSoy8FI5PnPbvRHbHzSXPpaxDCnFyJd+1SYftkAaFV6TBCR9WLoBz+mG1cAt2xBn6bY/8cXsQbgdYdOSs8z26kt1I54YP3nxjv+ncK1wQe3u6w6YxSLAuWLNk+VVKjcONoeQl9WRZGyB X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: KA6q58ysfyjAguOVIpp/2FdYe9kzhVXivSes9N6TvPcrjuJaOalBIrjbELStIuz/JN6ZdQpmgHNmZgz2RdzeyoSqmJ9ylyuKI9AJc+KZ6VgufhGqIlKX97tG6PO6XkKzHahAQfJ7GGFZZWdKQPgheKDSrg0clJGWTezDv/r1BECMSx+JAKa0Poio84xtttbhdRDVQMQ5Oja84XORjTAMosW22X5dIENvKNzvUwBklMQKeZ8u8mZpvBFra9tNwg8MsYcYf/ruZdZ5YvdB7wrs6n6m34vi4uBsxTxKmWMhKb3MO6GZumr2o8kpYZ9mgwRTtaBm7P4caRyq1qS6kJdidr8iyxQGnfFBwYm13hNdX+Vq66dlOxsLsOjWUsQgxptpacUMIvET37+vvr34fO5TDK6Nq8WORv+uAi1ZOqqDweNhKL83lCnel2zQC3wx4Lw+XWT4WOcqCRlqstlDW7mZqmsUFYkzdhE5RJTyl+5kk4txvMvZD9UP20vwDHlTvl0RWTebu6zLhKJEoq/VSNa5bIbdhlDR6GQu1dtzwEfijZYGC8+WHCXLBx1Zkba9bMf8wdPPa2szMFa0n8ULXGH3D4q3nBjcRGrNRHUWDpfZtE3tpB8PWJfU7faHh1sk0NiF20eUEdXs4gQaIZT+K8FJ60IkpXhd3cLBWqVwywPyn3bc9irMMxJUeNGAHedOm9sCOwNr/IMYxh43EpuV8GkRxFsV7XAEKlaX5E2l9/vSR0GoBRHmrmIoCRQJ9YAjnjW7/u7Ze5ukPAOOrv5OxxPVTOvGcEHxGJAeIHtea2UishhcuUGqNX6pdsEBhvjk8POjcHiAjntFz9okwirTGkvzLma4w4eLTEPNDMwWUQJt4ZtZM8O8lUkJ7oZMwewpXirMJ3Q4q3M+nuk5x9y5lU6fm85Tm7UFy5NzFnDoNgBz8TgFt1yUK+ieg7oU2417VM4MlR329w9jJXdbxXcGherh2gHl2jlMOoxjVJeaIoqQww7SMz9KjUBAWp2TbTin9WPuUHUFjRuu+n18corev+IQxupB66P0SbZ1vt/cO38Afe1NHafSWjpZtoU037O5ffXzWrmoTPclpAEQskW6ZhhwEZzHmSnpeDw2R/CaGGAkZhFv9t/MOYjwMQ5eZMmgW1yBaxO3fdl5e4YC0KWThGvX73B9A4p8vtQQ4YOaPH/0rnG5tzhXjlAXmZPIesfaLk8VyF9x0ORvOlZrakX5cYf9NZY668TlNUskzHEFojh4jdJEyw4WdK5UXDhsTovrcbc4Y0Iuc0srAgXUjGHK91FGt5BFdQ+s+c61AvZUtRP8RASbtOkavmhdMAVL/v5ar0x5lg4tF5Sp4MH5wskobxCS14ekgOFgIQQHwAH9VQkh0YfvtaKulhBO3tbhQr/HgavnYIFN/OV+0u2+M5tWUNIEkbU3WFRIO3lAbTao02SXczvjlqb0SRJmJaN/1Bbui4xO4v6crg+J+Dv1jHP3k7e80BLdR5FAzybqXDWKoSIs3NN8k6AT1L/kA2TDXcjNZHAGdg3jxpjPscohuA6eHDeyzbYKdlgqqagckJB83IbC9cOcuMwpsLFTsjlYHDdvVYg2 X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: b3c62de3-9d84-4009-ea96-08dd50f4c103 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:29.8490 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nHTWmbjtLU55wkpEmIFTjg2BhKOUgDDUPFP90UXhWgnZqLlz33rHmjxNIa71yVbkYh3FDtcDBp2iqGWyLDTSDg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: yyyPqQluq5NhqP5i5CTWhPLr--P9N8dc X-Authority-Analysis: v=2.4 cv=Ps5pbxM3 c=1 sm=1 tr=0 ts=67b5efb7 cx=c_pps a=o9WQ8H7iXVZ6wSn1fOU0uA==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=Vo7S2oCIfNTITXvqP7UA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: yyyPqQluq5NhqP5i5CTWhPLr--P9N8dc X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Hook this call up to the legacy reset handler for vfio-user-pci. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 12 ++++++++++++ hw/vfio-user/common.h | 1 + hw/vfio-user/pci.c | 15 +++++++++++++++ 3 files changed, 28 insertions(+) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 182ef5ab8f..160a1f0536 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -1362,6 +1362,18 @@ static int vfio_user_region_write(VFIOUserProxy *proxy, uint8_t index, return ret; } +void vfio_user_reset(VFIOUserProxy *proxy) +{ + VFIOUserHdr msg; + + vfio_user_request_msg(&msg, VFIO_USER_DEVICE_RESET, sizeof(msg), 0); + + vfio_user_send_wait(proxy, &msg, NULL, 0); + if (msg.flags & VFIO_USER_ERROR) { + error_printf("reset reply error %d\n", msg.error_reply); + } +} + /* * Socket-based io_ops diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 31b11ed614..c09637dd48 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -95,6 +95,7 @@ void vfio_user_set_handler(VFIODevice *vbasedev, void *reqarg); bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp); int vfio_user_get_info(VFIOUserProxy *proxy, struct vfio_device_info *info); +void vfio_user_reset(VFIOUserProxy *proxy); extern VFIODeviceIO vfio_dev_io_sock; diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index cf1e642399..d520b7592b 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -276,6 +276,20 @@ static void vfio_user_instance_finalize(Object *obj) } } +static void vfio_user_pci_reset(DeviceState *dev) +{ + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); + VFIODevice *vbasedev = &vdev->vbasedev; + + vfio_pci_pre_reset(vdev); + + if (vbasedev->reset_works) { + vfio_user_reset(vbasedev->proxy); + } + + vfio_pci_post_reset(vdev); +} + static const Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), @@ -287,6 +301,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) DeviceClass *dc = DEVICE_CLASS(klass); PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + device_class_set_legacy_reset(dc, vfio_user_pci_reset); device_class_set_props(dc, vfio_user_pci_dev_properties); dc->desc = "VFIO over socket PCI device assignment"; pdc->realize = vfio_user_pci_realize; From patchwork Wed Feb 19 14:48:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982422 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7036AC021AA for ; Wed, 19 Feb 2025 14:56:24 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPV-000500-19; Wed, 19 Feb 2025 09:50:49 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPS-0004n5-BC; Wed, 19 Feb 2025 09:50:46 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPN-0007di-S3; Wed, 19 Feb 2025 09:50:46 -0500 Received: from pps.filterd (m0127839.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBKnF5003929; Wed, 19 Feb 2025 06:50:36 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=3UIxPx86gPTh5N+zlwkv3sY4JZRFdiQ7kvskGd/V+ eE=; b=u47DgXi907QqMs1tod4lJHzVxJQzZZ8y0+lIdmLlpTIGzZdWRAtUXDOqc ausAWY0kgMx+N1gqBokG1UjP4hnTrNUfh+zwCfy1A2Uzf3Oopr2W5KALWqmvaSkB 2OulGiNigCahXCZJVNGSVclFysF6sOt68b0WtsvichsrzZyBOYCBF60gzN34hpjV JaUkzeUPsPv9hmNc7CR4iTiEoO5SpLJrNgQF002nOpQJZ2ecYaSc/o/uOvGdILth UhSOWFIcSAJK7VYydzdNaEWGu2pIZPmmjC46E/QkwV/AaRg0wJBQOi/HRVERdY5T hUmCWnIH8jl9E1jxC6SVBgVFvy/ZA== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2045.outbound.protection.outlook.com [104.47.70.45]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4bdhy0m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:35 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=oD2r7wiYNgBIcUvC/7oD5OMQXfa9rGkKdClzxopVXCIvA6vkUEgRK9iQTU/m0BXzc/LdLHFvOGiNnl+XGsENl0p3Qgq/EV3KTE3tAseUDRIQCrJMLMzRQ3JNvkftgbeeWy0NqeyC1lP00BLLrzPk7Tfx99bJC8S4FuAg6/Hld9dNxpXdHrB/RgI8T7CYvIhkgDrfkvgcA4fybn81bv6G2sOHV5g+B+nJrPUx6L8wt/YV7py2yyqilG3iqIsSQ0PXaog0xCuRUBpxSIBZ1/48OrjvAxeEADkq/ui2i0SRLu+ntPpfIeYRDmWzOJyFKY//24NPBfISPZ6yLwL8juYlLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3UIxPx86gPTh5N+zlwkv3sY4JZRFdiQ7kvskGd/V+eE=; b=P7nRON6v+6cjjN1WD7XNTeJIj0AWdWsxCwofy1tht/VufN3wvXwF1txb2yP/Zz9d0L9jU77kQxy43bHZk9Lx9KXK4bkqCWh1zBkNPUFsMIDmfijOGO5NvvegItFAXzgOTXfcPI3lAe3ajsJbB7GU/vek6bATXkhkf9ykFwT09SSNmBGrzvX7QI+5a8tAnOAMVu22uGI1ZJ4AyHcMtXZ42AY1ipK1Rm2f/aAje4GPIPXHB2sEdSkh5azexePnT3EFTZc4PITyFlANyNxxBMOFnIFBKG3pQsGeROA7t02hyulN1WjxYZIdekFYIPp6XzsmcTjQiM1V3CywaHrGzMJzjw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3UIxPx86gPTh5N+zlwkv3sY4JZRFdiQ7kvskGd/V+eE=; b=rzkccA/Z7rYn6SEwjF+HZzsyqg0xNx25shEYnyGWBRzBsW4XJpa+Lyiw2elj395i5e374r4Pjb166zkodK9+n9ae4mE44lrRJHCTX++EMtw85HVTEKM2NJdQU5b71GfvpLnXeFfRmAKjCgizbJygZ0oSsfNMVVbxowFE4n0Ixgs6EAuqjZWPLnLA8ismPYRPdl/uyCYQpxDM1zr3i93G/TU8NjWxHgDQYPkGsyLBu86ZmpinK39nC3QJh9dU9pPiFFvJDOCiWNiDEwv0YDlXnSvWYRCPNHL0+d7y/wmHoq8SixtzwJ8ITsXaxFOiTkgnDYuAR23/keadCxMKjivPww== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:33 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:33 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , John Levon , John Johnson , Jagannathan Raman , Elena Ufimtseva Subject: [PATCH v8 24/28] vfio-user: implement VFIO_USER_DMA_MAP/UNMAP Date: Wed, 19 Feb 2025 15:48:54 +0100 Message-Id: <20250219144858.266455-25-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 9d629eec-00c3-4118-28c4-08dd50f4c321 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: HOawGSZJFvqIj7FEQdv8mdJeK3r0EDPa1foov2l2HleU1MJnNsXxq7d9jRI6k6i0Z1KaulNEWrtCtbOch6EUpmZ+motDOTn5jp+4HE+SQvsGeuA4DrATrn2mvCIR1baEkTizAUTZUh+PpmQ4+48p9ng3SO0Y6eEE2Y+a8kiBRHg7rl6p9IUWZEzoLwJf4GraYipoVGmQPubcVVSSV1xsvkCowcvnGDfNuipcnUvWQZnckedYwfpr/Nhl999GRRDj59kZCfbNFS1fx8CQTp3evW8XpJJk3Bf6CfB3Q/QklgNxThkBGY7SK9MdoXGqqLZHX0dmbx4kGRMNWtPsT4OO5FkEM6t/6H2hqUn+XcSy9QUWJDZF1Da2JBCQrMxocJfX3SxytWI3724T4JbbyDAbMox18XgIancfId9/W2R1qx2RNLvolSRvjO1+Rl3q3kV3jAfKZxNjLSweJ42CbgvCUHcCr3N2GY7AnOa7hKv4OGk6J32u2o5Z5KuoF0P7j5WcCk8YgWo/LvmqT7X5tMAp6F3IvdCowKsudixDXibKy07A1pmCIlfQUadxIvpQ6cmV0TDWwYE6VN7mdvTPRE/Y0hVgV+U3R8+vmwgwdcVmyIk35NJeSRuHEaUDydl9wt9QGj7eMnKq8F1FNZ5d7SqBowbw6uZkuJO4iqzKToWdTTpcigOtW76OovMvCUWxumFGTMsZ97uub9t9ufisoslfvGwH7zDkxmmT93A7cH0gV8Px4uhUMcNuGGECcpoeMbTLlaI0Xg+IUzo0EFFJXK123DqA/Y9gn3poaXNAL/Oh4M6mnVuyoe8ILLp5JYDE45eS0AjxO2XY6982RJIZ2WSwh5Q2+yMZI1dsA2+9GLZCcCedKlA09e8+7R2e4Q7HSY65iTeX0f0VRQzAIS58x2dSPn4x/ZSNeN2Es/Frjecx1/65+PoGOsNeQ0XAXqLc2B58TTRQhpE9Z+THA9F/v81t67xmuhDRFZQaEXa8tkAL9wMRUDVppUybN8hRR5lsg84Uy7FCSbLLjjuH4BVX3VPehIa0xUKH+asPa4UaGufJOv35yDsd1nibsfR69gJxPGoVCLBpEAZaNMJCcV0wZw1hKQgmV1qlUjs1gG7ZTyOt9L/w5xBIFcJgK+YAgDs12o949gq9zDRDpIFlRU7UqvVJ+lH471InCT7TPiJXJFW5zxN+uPmJnSnmcWtSbQvmXtqP5ICqhjbHhAYHSqrMdSePQElyoAJa6jQb1qSAtunf6M5JNuK4ayiQtjLhYHHuneaVyEtJE/L2nH6d78oUw7sa1dLsjQImZ5qmmZNISurgel7tCndZZOt2a1FBYQ2qwfekWM9zNGtqBqZil5G5FPsmLRDS2/WxLAK+ylXFXwPYXIk60Fb4i7uwYr1Wx8V1Yl1e X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: bKH7q5XHvb79yplWp7yXwFirZRRyX3R5pxeSbg99agYblXaaKvS6obcX2k1u3aTJnpQUTCXQ5CEhE84RuHSmAF2GpCsCqMr7BxqVqAcA1/m6S3FlmRwmwNaC/jkyPeLcsK4S3yT/iA/ieD2sTpyHSWduDJkBb3n0aN+k5u3gUQtWdXC6vmDL/36mCQd7jqfAxNi5EF9jajlOkZqlR+b2C/DZsp0E4PZAJSiZQNecZgc0hwRCUEUbKC/TOJVp4mr+ZvVK5OXDHdVkuxSN99ASC6paeFB8ezxjBoxMyLZCUkiUqLqoTJYkTpSpFFemjdPp/D5IQBSPNtrbzh2QvpUwjBMsR7RKpgJCiPx4E3RdrZbsUx+TR71U48UELy53NER+q9qFRqZoPrLva3JUg+8+61MBkSB8Vg9YpM2Rh1j8rzRjxTk5gYP1CGTEGAR0nV//YGpq3S3jL/NsibHmY4TNoL72/T8WNXtsy13g+Zmd4rrRtI8Puu1kuFFyHM1ZAvWM/3+VVJG3eMKubyHop642Zq+fYDUDCqHuW9v6RAoiuFn+3/PkzIGsr1LpyXtYsF7VlgAukSufdsRj/NFczVGZR6+N+3A52hoQWiwSBMEg5Ex3LMhq0VqEvlQMfY2zcWbn69LiDSZcFf3Us95XnB8U4fPanL2SS7r3oh4LTXLQHL/YRHF4ZA4If8mFVytgfzOWcz1jteWAzFaxLG2cWLuD26z7PtUXVU8TEsJM5P8RZQ81/1JNCEnRPw0G5wLGhVjP2K45nU+B2JJcYEEncpR+TwY2mjTypCedDaFJ/D2BszUCktG57VrZYm4pIHRnOsCnkOJTOmCxrH3FDaJvz0InQ8M2ICwQMorg12WJ07lMP95Y2uW0bHFK+EJVhHxcybeqC1+fP5ArfSKLNROB2bxOBlcte7EPEMkm8K48JHrs/BVgSrgM7TRiXMnAdchKSBpj65UIsLCzaxdTWuUZeje0Pd9cIBNcekuVeVcgNI31G2PDl0FO2qWTiLye6ntImBUrFOEtfNwK4iOPV4GxaK7RaerehcLqd7hBRsguv+ciY/AQ9d8w5847y0cTcJCBxaAf4CgPpGYmTc7JAhKIRNa/1zHoKU3b8JzdBoMM3UxVdRCstOg2BOhwYWmpdsa6keSBzFYQ+FbjusAvPq+S2mhdbilHFV3sW+oANWEo/W3NU5lXMTEuH9LpU7FicKt6hcPORtzrTCNVzOy0cCuAaJRYq1/xoLbo9yn3z4CTstfrg0Dfh4Dy2G1hGdV3Om3CR4Rgak85kR+xc6PppxPjfkTNlCG8+bwytZk6qC0K5g03akKX0R9PJFQU/SbIoUhDgpBJXnu5KrJHFc3m8sKBTxydUXaMb2tEB9PpGj7UvXLLwN3Bu7PBYNeNG6ZRpFwA80qmfIGwihDPk48Wbzz1Si1mHgdusZz2yVpq0m87VdawZLt2HMURd+4mbIGCt0iH43AcyKW5MBCL2zxauMWVHoRvh7qQyLo68spKFuBLZMfIDjC0a1cKLmhCwOPloWlwl97YNyiPpD7bj3NyeUzCvnzEFpjSkgE9lWU0z8kB4bfm8E226dqIQb+L3aSoZyVJSy/e X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9d629eec-00c3-4118-28c4-08dd50f4c321 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:33.3934 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Rh6qfAj1EXDnl58/8/DKiht0FjOenl1Mom1PosfYrWe8LkfTb5BQz5nOwV1Ybq3JFSXVvwFnk4ts/TPK6DL0cg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-ORIG-GUID: qg9YaZBJwDuQWLhOGS82Y4hF44JpfPiX X-Proofpoint-GUID: qg9YaZBJwDuQWLhOGS82Y4hF44JpfPiX X-Authority-Analysis: v=2.4 cv=Fo7//3rq c=1 sm=1 tr=0 ts=67b5efbb cx=c_pps a=WCFCujto17ieNoiWBJjljg==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=RQZ_2NmkAAAA:8 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=LQllIVlj0uuMV4zLHkkA:9 a=46pEW5UW3zrkaSsnLxuo:22 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: John Levon When the vfio-user container gets mapping updates, share them with the vfio-user by sending a message; this can include the region fd, allowing the server to directly mmap() the region as needed. For performance, we only wait for the message responses when we're doing with a series of updates via the listener_commit() callback. Originally-by: John Johnson Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John Levon --- hw/vfio-user/common.c | 89 +++++++++++++++++++++++++++---- hw/vfio-user/common.h | 9 ++++ hw/vfio-user/container.c | 107 +++++++++++++++++++++++++++++++++++++- hw/vfio-user/protocol.h | 32 ++++++++++++ hw/vfio-user/trace-events | 4 ++ 5 files changed, 229 insertions(+), 12 deletions(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 160a1f0536..b78b9e57e8 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -44,7 +44,6 @@ static void vfio_user_shutdown(VFIOUserProxy *proxy); static int vfio_user_send_qio(VFIOUserProxy *proxy, VFIOUserMsg *msg); static VFIOUserMsg *vfio_user_getmsg(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds); -static VFIOUserFDs *vfio_user_getfds(int numfds); static void vfio_user_recycle(VFIOUserProxy *proxy, VFIOUserMsg *msg); static void vfio_user_recv(void *opaque); @@ -57,10 +56,6 @@ static void vfio_user_request(void *opaque); static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg); static void vfio_user_send_async(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds); -static void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, - VFIOUserFDs *fds, int rsize); -static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, - uint32_t size, uint32_t flags); static inline void vfio_user_set_error(VFIOUserHdr *hdr, uint32_t err) { @@ -152,7 +147,7 @@ static void vfio_user_recycle(VFIOUserProxy *proxy, VFIOUserMsg *msg) QTAILQ_INSERT_HEAD(&proxy->free, msg, next); } -static VFIOUserFDs *vfio_user_getfds(int numfds) +VFIOUserFDs *vfio_user_getfds(int numfds) { VFIOUserFDs *fds = g_malloc0(sizeof(*fds) + (numfds * sizeof(int))); @@ -655,8 +650,38 @@ static void vfio_user_send_async(VFIOUserProxy *proxy, VFIOUserHdr *hdr, } } -static void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, - VFIOUserFDs *fds, int rsize) +/* + * nowait send - vfio_wait_reqs() can wait for it later + */ +void vfio_user_send_nowait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize) +{ + VFIOUserMsg *msg; + int ret; + + if (hdr->flags & VFIO_USER_NO_REPLY) { + error_printf("vfio_user_send_nowait on async message\n"); + return; + } + + QEMU_LOCK_GUARD(&proxy->lock); + + msg = vfio_user_getmsg(proxy, hdr, fds); + msg->id = hdr->id; + msg->rsize = rsize ? rsize : hdr->size; + msg->type = VFIO_MSG_NOWAIT; + + ret = vfio_user_send_queued(proxy, msg); + if (ret < 0) { + vfio_user_recycle(proxy, msg); + return; + } + + proxy->last_nowait = msg; +} + +void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize) { VFIOUserMsg *msg; int ret; @@ -693,6 +718,50 @@ static void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, qemu_mutex_unlock(&proxy->lock); } +void vfio_user_wait_reqs(VFIOUserProxy *proxy) +{ + VFIOUserMsg *msg; + + /* + * Any DMA map/unmap requests sent in the middle + * of a memory region transaction were sent nowait. + * Wait for them here. + */ + qemu_mutex_lock(&proxy->lock); + if (proxy->last_nowait != NULL) { + /* + * Change type to WAIT to wait for reply + */ + msg = proxy->last_nowait; + msg->type = VFIO_MSG_WAIT; + proxy->last_nowait = NULL; + while (!msg->complete) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + VFIOUserMsgQ *list; + + list = msg->pending ? &proxy->pending : &proxy->outgoing; + QTAILQ_REMOVE(list, msg, next); + error_printf("vfio_wait_reqs - timed out\n"); + break; + } + } + + if (msg->hdr->flags & VFIO_USER_ERROR) { + error_printf("vfio_user_wait_reqs - error reply on async "); + error_printf("request: command %x error %s\n", msg->hdr->command, + strerror(msg->hdr->error_reply)); + } + + /* + * Change type back to NOWAIT to free + */ + msg->type = VFIO_MSG_NOWAIT; + vfio_user_recycle(proxy, msg); + } + + qemu_mutex_unlock(&proxy->lock); +} + static QLIST_HEAD(, VFIOUserProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -827,8 +896,8 @@ void vfio_user_disconnect(VFIOUserProxy *proxy) g_free(proxy); } -static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, - uint32_t size, uint32_t flags) +void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags) { static uint16_t next_id; diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index c09637dd48..f7cc02d2e7 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -65,6 +65,7 @@ typedef struct VFIOUserProxy { QemuCond close_cv; AioContext *ctx; QEMUBH *req_bh; + bool async_ops; /* * above only changed when BQL is held @@ -96,6 +97,14 @@ void vfio_user_set_handler(VFIODevice *vbasedev, bool vfio_user_validate_version(VFIOUserProxy *proxy, Error **errp); int vfio_user_get_info(VFIOUserProxy *proxy, struct vfio_device_info *info); void vfio_user_reset(VFIOUserProxy *proxy); +VFIOUserFDs *vfio_user_getfds(int numfds); +void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags); +void vfio_user_wait_reqs(VFIOUserProxy *proxy); +void vfio_user_send_nowait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize); +void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize); extern VFIODeviceIO vfio_dev_io_sock; diff --git a/hw/vfio-user/container.c b/hw/vfio-user/container.c index 0c487dbb92..3974bc8a8c 100644 --- a/hw/vfio-user/container.c +++ b/hw/vfio-user/container.c @@ -23,18 +23,119 @@ #include "qapi/error.h" #include "trace.h" +/* + * When DMA space is the physical address space, the region add/del listeners + * will fire during memory update transactions. These depend on BQL being held, + * so do any resulting map/demap ops async while keeping BQL. + */ +static void vfio_user_listener_begin(VFIOContainerBase *bcontainer) +{ + VFIOUserContainer *container = container_of(bcontainer, VFIOUserContainer, + bcontainer); + + container->proxy->async_ops = true; +} + +static void vfio_user_listener_commit(VFIOContainerBase *bcontainer) +{ + VFIOUserContainer *container = container_of(bcontainer, VFIOUserContainer, + bcontainer); + + /* wait here for any async requests sent during the transaction */ + container->proxy->async_ops = false; + vfio_user_wait_reqs(container->proxy); +} + static int vfio_user_dma_unmap(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, IOMMUTLBEntry *iotlb, int flags) { - return -ENOTSUP; + VFIOUserContainer *container = container_of(bcontainer, VFIOUserContainer, + bcontainer); + + VFIOUserDMAUnmap *msgp = g_malloc(sizeof(*msgp)); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DMA_UNMAP, sizeof(*msgp), 0); + msgp->argsz = sizeof(struct vfio_iommu_type1_dma_unmap); + msgp->flags = flags; + msgp->iova = iova; + msgp->size = size; + trace_vfio_user_dma_unmap(msgp->iova, msgp->size, msgp->flags, + container->proxy->async_ops); + + if (container->proxy->async_ops) { + vfio_user_send_nowait(container->proxy, &msgp->hdr, NULL, 0); + return 0; + } + + vfio_user_send_wait(container->proxy, &msgp->hdr, NULL, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + g_free(msgp); + return 0; } static int vfio_user_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly, MemoryRegion *mrp) { - return -ENOTSUP; + VFIOUserContainer *container = container_of(bcontainer, VFIOUserContainer, + bcontainer); + + VFIOUserProxy *proxy = container->proxy; + int fd = memory_region_get_fd(mrp); + int ret; + + VFIOUserFDs *fds = NULL; + VFIOUserDMAMap *msgp = g_malloc0(sizeof(*msgp)); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DMA_MAP, sizeof(*msgp), 0); + msgp->argsz = sizeof(struct vfio_iommu_type1_dma_map); + msgp->flags = VFIO_DMA_MAP_FLAG_READ; + msgp->offset = 0; + msgp->iova = iova; + msgp->size = size; + + /* + * vaddr enters as a QEMU process address; make it either a file offset + * for mapped areas or leave as 0. + */ + if (fd != -1) { + msgp->offset = qemu_ram_block_host_offset(mrp->ram_block, vaddr); + } + + if (!readonly) { + msgp->flags |= VFIO_DMA_MAP_FLAG_WRITE; + } + + trace_vfio_user_dma_map(msgp->iova, msgp->size, msgp->offset, msgp->flags, + container->proxy->async_ops); + + /* + * The async_ops case sends without blocking or dropping BQL. + * They're later waited for in vfio_send_wait_reqs. + */ + if (container->proxy->async_ops) { + /* can't use auto variable since we don't block */ + if (fd != -1) { + fds = vfio_user_getfds(1); + fds->send_fds = 1; + fds->fds[0] = fd; + } + vfio_user_send_nowait(proxy, &msgp->hdr, fds, 0); + ret = 0; + } else { + VFIOUserFDs local_fds = { 1, 0, &fd }; + + fds = fd != -1 ? &local_fds : NULL; + vfio_user_send_wait(proxy, &msgp->hdr, fds, 0); + ret = (msgp->hdr.flags & VFIO_USER_ERROR) ? -msgp->hdr.error_reply : 0; + g_free(msgp); + } + + return ret; } static int @@ -234,6 +335,8 @@ static void vfio_iommu_user_class_init(ObjectClass *klass, void *data) VFIOIOMMUClass *vioc = VFIO_IOMMU_CLASS(klass); vioc->setup = vfio_user_setup; + vioc->listener_begin = vfio_user_listener_begin, + vioc->listener_commit = vfio_user_listener_commit, vioc->dma_map = vfio_user_dma_map; vioc->dma_unmap = vfio_user_dma_unmap; vioc->attach_device = vfio_user_attach_device; diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index 48dd475ab3..109076d820 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -113,6 +113,31 @@ typedef struct { */ #define VFIO_USER_DEF_MAX_BITMAP (256 * 1024 * 1024) +/* + * VFIO_USER_DMA_MAP + * imported from struct vfio_iommu_type1_dma_map + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint64_t offset; /* FD offset */ + uint64_t iova; + uint64_t size; +} VFIOUserDMAMap; + +/* + * VFIO_USER_DMA_UNMAP + * imported from struct vfio_iommu_type1_dma_unmap + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint64_t iova; + uint64_t size; +} VFIOUserDMAUnmap; + /* * VFIO_USER_DEVICE_GET_INFO * imported from struct vfio_device_info @@ -176,4 +201,11 @@ typedef struct { char data[]; } VFIOUserRegionRW; +/*imported from struct vfio_bitmap */ +typedef struct { + uint64_t pgsize; + uint64_t size; + char data[]; +} VFIOUserBitmap; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events index 053f5932eb..7ef98813b3 100644 --- a/hw/vfio-user/trace-events +++ b/hw/vfio-user/trace-events @@ -11,3 +11,7 @@ vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d" vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %d flags 0x%x count %d" vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x" + +# container.c +vfio_user_dma_map(uint64_t iova, uint64_t size, uint64_t off, uint32_t flags, bool async_ops) " iova 0x%"PRIx64" size 0x%"PRIx64" off 0x%"PRIx64" flags 0x%x async_ops %d" +vfio_user_dma_unmap(uint64_t iova, uint64_t size, uint32_t flags, bool async_ops) " iova 0x%"PRIx64" size 0x%"PRIx64" flags 0x%x async_ops %d" From patchwork Wed Feb 19 14:48:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982412 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 377E5C021AA for ; Wed, 19 Feb 2025 14:55:16 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPT-0004sN-B0; Wed, 19 Feb 2025 09:50:47 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPQ-0004dY-E1; Wed, 19 Feb 2025 09:50:44 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPO-0007do-29; Wed, 19 Feb 2025 09:50:44 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51JBqX12018818; Wed, 19 Feb 2025 06:50:38 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=TF6SxOBcL7NnUxy4Y7t98bPkndE6ynnFrD+nCSwzU Hc=; b=OeybuQtnYZEMIv4wrndPH3O48HU/2sClK5Eewm+LbkwC7MdMZIwgu0mlw gsIiV9CbbgIa7zUI73UZFhrSpWqFng+scxW4DIcdG+IIjvJsEI5za3L7E0vO5Zka VVJYpyzwpI+yle37iuEWsX2A7NmhLBZ7FXi4Aeq2gYf9zxJwKtHQZ71ilIB31jFH rZgCN5YnRPiXh4/c20LzUD/xOpV9Ycv4PLN+FlLua2TIm6YHRZSwfm2f8jK1/tzM RPECDzZVPzwTpirvTFUGW+eIMzWweCLpx+WLAbPZL1z/j1P6YWoHsZFqHVA6DN1R Q2w9JcSSdqN5W4c3BGasCFyjCaeTg== Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxu9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:38 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=q5BaJ8HM9f6r4fXQ6gE5r6cJICA4r1a+c+r/cfEy5hdn27+7VU0DhHSbIcH2orntuS+3blQpzqGiTK2PtKPQ3rsMRmrnhPWjU6QrWRpn6kqjHe76e6HOhm7YhzhlImuMeE3Oo7EJ1Kq8Dkbcy+gPZsmtaFaipr1APtTh+S39FRgJZ0vFK6HFkybuLfRBmDd+hXYZ2h4Ocb7Td/VCc819/B3Uqpx7pEJS5ZHE/sv3cB/9vIoYaetvD8pNhwigxYJVvYtR4th5/PZ+5ZnUr4iWgVfrTUGbneu1k95X6nnKtI2uYKDronVzd+LhwZtJV+9284V15UGXVyMGbdL06ewRZg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=TF6SxOBcL7NnUxy4Y7t98bPkndE6ynnFrD+nCSwzUHc=; b=bzScnLDqYEEE0pboc7VI3NJyooSpvFEXNiGqp/P8/EGmg7Bv03SNMUYTWPJP0ELKKW6Txcxo9nx95/pp8lBl6Z04uwdXO4loVx1r/EKokudsOEt0U+8oH+Bt9y4UoXjuOerSGto3JGFOuICp/2rQtojiFKwzi7k0GzYr8OhWUNfKnKCT9b6SkZXIecYcEeNFL0Ug9+MiXikAQn5bMaA+XL7I32USVXTxjoUocM0+E1q1uOcM+Arg6OzWzD9PTt4AzpSbqjObMN9wuJrjr7265tbMCNl/jRQ+/GN2kDrJ837LqFZSq3x8Pw9SvoJ1ZjQdaPQ4UQ/gDnkdDrb8oVAYIA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=TF6SxOBcL7NnUxy4Y7t98bPkndE6ynnFrD+nCSwzUHc=; b=pLe/+xJ97BXotDrJGc9Lh8lWFaxgojPq4ojJOWwnBTpGJiAA8Kryd5Dn7Hr4tpVUxFsp0RLeE6ag4nkkMwtOaDJ6Omrs5As4QwcWrZH8LOTUA/B0jK0kfqhiY+LadJBEHSFbz4RvFnXWBj3lqASNUS/+ojZ9wA+XllcCQ60+m2Cv7/UjmU3WxB6GPSFUVJRVXtN/2TyOqs5crV6lA0Al+T0BEpaXntZicaHtH6fc7rpcN+Bi2VMjReXkvSenaOTHhrKN5yCJfRuOzFxVv7WVF6AJyGp2Rxww7hV6Cb+s5N9aIDjXUcJj7BUCoAL1Xcx2fKVhd80/pBwsAfqsMyfoLA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:36 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:36 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 25/28] vfio-user: implement VFIO_USER_DMA_READ/WRITE Date: Wed, 19 Feb 2025 15:48:55 +0100 Message-Id: <20250219144858.266455-26-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: f9273137-43f3-417f-29a1-08dd50f4c51f x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: obaT+0mmNOSsLnPLUaHzQuwEapY/D0f0MIf22VLZtN4eUUwz39nCwYjiCqTtYPZ43uyNF85UtriF5BnvwCfGT9HyfWnG2fFUxe6jzIKMPL2Umn9k4IPdoWx+lwv1iwKeet76a11vD9t3ewItwQIbGs+zUOYCa4KCR2HuSR3KYP/CPU6iQKCmhuXg1Urv3YF3a3Pu2z56vgfcIUTkeAo4ZrQc18k/keQzbn2Al8CbFT7QVHzoDj5uKlz99a4eztmzvAnF2MVePDZFlQF8nK8p8vqzAB7pHlDB5NM6+y3UsHrDOUhEuKloOSYrrH3z8ye66P5sHerMHmnCKR/QF+9GI50hE0ck1PI50BIHvsfNlaHFPKhg7qn4G5ef0sPOtP7fSJcNOm4BgDL5+QpUCwMilr/xQr9iRHkZU6i5NjML4lZZYwdhcrlcibLZ2930ewnXDu9nyqedzyJUVHTo4Mgau9rn8rGsW7fxpmmBhMgTVqyEgvvi7oxd6vrXGwneWX0/Pa/7CmeLLbBpzNQo8QbQHyV1+KXMv2aXfqoPQx/sBh3MLB+2kkYL49l9dpqGJcTyffXRRKFDLrgraTSX2fvXELmyfec/zgc21sLCSM/ThC8B2e5VSOOwqW73BFcjVglEfh4qMwBzgk0ODQjsfdXfhSNsYo48qoV00o6Y8NaduwdN/Mt2V01yNvGr9YjDYrGWhpFK165frNs3dJE06s7Sgxqxe62epd+ylyWyDg3CSyaktnM+va3WvKVu86VEI5N2lrXd01koCYhnCRVdtiqx5QVVy40Wk6b9EBvVlwkNo7ciFgETU1mC0ZqqOStCuQC4BBwP3dmqrxDDp0RZExNt2EVJ0PX52gj0qWLcRm1WFC1AFAriHO9+sv3lIDvyWfNzBjKKIS6ESoeuApkR6Es6+3O3dhRGSRNakWzR9QNsXAmBJDOJB860qjp5K37PT8GCW1q8D7B91PmTh8Ur6am85zLCN9i+63+v64hsg/jEvOhio5zzWyD3mrfvEnQDa/JHfgqwzcVuiE9j1aQNdprtgEXh/GmppbXcYal39GrJmlmfuPvxx9sgN3aEUk6KLzCHm/xMQxWcN/tzmNwuYNWpduKSVQaHR8JxZgF9JfH5RhvHMYxYTlQBgnCmrJEEP/yHAUUCtBNx0ORNDgiy85mnONIHMBpJkoA6AWc14HHgsHsakUax30dH95aUxMMVCvKm8scgXMb4AgV4Rq35iR/WNE4lk70uSgoA52APVTfMfslzxidPAwAi7Dxpnm7G47GmZx5tA0slcW9iUWZ1l646WTPv6v/jFejHiao9ybBgzMVV+IHLNi+Jz/OzpczH+9cRPOEP3/psIqLoX7lNPsveJ5+CRaiIYQjjhrC/I3k4N6Pc+j12BVOigQXee+Ea2zBd X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 9UiahmsCrwKmzTHYqlhbGegJ92EhQRHS5PewzmD2/E/l2mqC+SJ+SypKqOpQB89MJPt3kpJvBT3ZMQdACzjcfqtHsAUfbl4WmAeGYGhPZa0xRBOT+YKIxRFNOQv0KIYZGJij5u6V1m3dGYW3bIkBMWTuq9DwbtXwtAiihxOjKqIXBt9S3PqfLa/DNPU48Kqs/+U04P1f8bQRaRMZV/RhrEnApe6vNA8X1K398w9MwV8StOkGGJuqix5pCdISRdg7eYarv9/mzjyuXICSk3sjIMYHogwzh0k7Kq5PliVvPjh7j7qM2srkG12HqZ0O7uQYT2kwTVUZ615f0d2WmsMFJ7BnFtxcwi5jdPIq/kCUVnWOIfAjbyBMwSuLwIpnRj8tNP/W8wPIoUkvOGDipGGWah5VgMFXaT316r/BLXir1KMMlSIh0B9GnAJZEygYJC3T4oZTu1aVVgcpQoC8QvW+WQW1x9YCWhFfZe5RygI/4W8vSVFMXt6L7y2loGSl0w6hsSikuYBWIb8oJLPHApvUcnO1IcPIc1cTjjtMung8oBxzzHB8Oqvz+8926pFoZJ6zjotMMdmaOJZSjE/JiNfGerJkclxFpHhEdKpDONFrUt0thE8Gfjae5yx7i6i5es8i9rdB5otVnL1SkRdv0rbbej7NFKnuoldlIUfrRcbpZ1h4aX4V/GvQ9MMATCeh24OVQvHLWW61c9YKinf7ttjS45dhh1YS7C5xLBJ1Bp5x+GSX77kq5/7MpuJ5LJViFLAKQWJ85YhAB45B3uhu83gR/C9jbcE/aCGbls+1TzCr4AWG2tcBXBtkGFWXlydC2JhCLIz9QSat9jy2R5L5gWivd/cgGlA7Sm9phiGyHHmdkJoVu45lQMls1de+qIaVQfODc1Ec9TGKcW/ieTjbFjM+BklESOyKiChTf2GjhzsUtdqwundQijjBR/DeZhc08DKs+05JNED4AmVZGCLwmKPoNpk1pcwWTeokwniVfFOcr27FFHs1lXajclLhpskaJyUKBs39SSP5PPJp/Mad4suCxTalPKNjHu4y4xrHDAuYP7AbQ762Is2WKxhZ63McjZL1zJDTCToto6B9QXR8BuPMKu0w7wiDr7QCNmHWhwr2AcJchWbVjuxs7fZ7UbUzyl5o7+gv67hb9cM5+K7xatspH9HY0AStCCdnYQaMkbKQSwQ6ejvYDt+nvWy562ZWWOrsWszPMzND92G5dcvs5WOWK7sT55+5Ppqogmi1YjwlorZEwEfdqKOIuYZFEnYJxgdfDaQ7R3rYUtKW5kpZJYAG9uM8EXClhrkuQ2pxrzP5VJvDf4p5+3PfV8fh2B7ALzZTLy/LXNWHjAqsCSt0+HQAgF1dzQ5Jpw0VzgZH3upkcJ60rwZAOwCTjgZOsbsQswx9R2qGRIuxz0aTdDZcdlZhChVoFzC1Vr8e+2CuHwIw2MYWrhdB9E8scSdEe8/LVzDE+qyYjtNd4rNxNImgEx9f56gmFqAHsC3XSxsL7d4uDHClMX6ZKwCia3T+HcCklWe6ghne18JnvoVfUBFl3q+fpNu8eJ57062PVpdUatjLhMRxs0TZisO+oxAIbr7AliUJ X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: f9273137-43f3-417f-29a1-08dd50f4c51f X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:36.7284 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JJ8pmv8Ww3zqAKZt48tuR68uVfa6jOaS7g1LZHzEX7pmdwLsuL97mFPAoziuu04U+GJ2tYRuS7QIyvgzNbVmgg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: pZ0vai2JnY8Vc6mSXxp73FwsyE2LkBGZ X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5efbe cx=c_pps a=CSNy8/ODUcREoDexjutt+g==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=sNJ477vccCyBWE1g380A:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: pZ0vai2JnY8Vc6mSXxp73FwsyE2LkBGZ X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Unlike most other messages, this is a server->client message, for when a server wants to do "DMA"; this is slow, so normally the server has memory directly mapped instead. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 57 +++++++++++++++++++++ hw/vfio-user/common.h | 3 ++ hw/vfio-user/pci.c | 110 ++++++++++++++++++++++++++++++++++++++++ hw/vfio-user/protocol.h | 13 ++++- 4 files changed, 182 insertions(+), 1 deletion(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index b78b9e57e8..38f8eef317 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -377,6 +377,10 @@ static int vfio_user_recv_one(VFIOUserProxy *proxy) *msg->hdr = hdr; data = (char *)msg->hdr + sizeof(hdr); } else { + if (hdr.size > proxy->max_xfer_size + sizeof(VFIOUserDMARW)) { + error_setg(&local_err, "vfio_user_recv request larger than max"); + goto err; + } buf = g_malloc0(hdr.size); memcpy(buf, &hdr, sizeof(hdr)); data = buf + sizeof(hdr); @@ -762,6 +766,59 @@ void vfio_user_wait_reqs(VFIOUserProxy *proxy) qemu_mutex_unlock(&proxy->lock); } +/* + * Reply to an incoming request. + */ +void vfio_user_send_reply(VFIOUserProxy *proxy, VFIOUserHdr *hdr, int size) +{ + + if (size < sizeof(VFIOUserHdr)) { + error_printf("vfio_user_send_reply - size too small\n"); + g_free(hdr); + return; + } + + /* + * convert header to associated reply + */ + hdr->flags = VFIO_USER_REPLY; + hdr->size = size; + + vfio_user_send_async(proxy, hdr, NULL); +} + +/* + * Send an error reply to an incoming request. + */ +void vfio_user_send_error(VFIOUserProxy *proxy, VFIOUserHdr *hdr, int error) +{ + + /* + * convert header to associated reply + */ + hdr->flags = VFIO_USER_REPLY; + hdr->flags |= VFIO_USER_ERROR; + hdr->error_reply = error; + hdr->size = sizeof(*hdr); + + vfio_user_send_async(proxy, hdr, NULL); +} + +/* + * Close FDs erroneously received in an incoming request. + */ +void vfio_user_putfds(VFIOUserMsg *msg) +{ + VFIOUserFDs *fds = msg->fds; + int i; + + for (i = 0; i < fds->recv_fds; i++) { + close(fds->fds[i]); + } + g_free(fds); + msg->fds = NULL; +} + static QLIST_HEAD(, VFIOUserProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index f7cc02d2e7..f8c61f2128 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -105,6 +105,9 @@ void vfio_user_send_nowait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize); void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize); +void vfio_user_send_reply(VFIOUserProxy *proxy, VFIOUserHdr *hdr, int size); +void vfio_user_send_error(VFIOUserProxy *proxy, VFIOUserHdr *hdr, int error); +void vfio_user_putfds(VFIOUserMsg *msg); extern VFIODeviceIO vfio_dev_io_sock; diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index d520b7592b..e65c7eaf02 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -96,6 +96,95 @@ static void vfio_user_msix_teardown(VFIOPCIDevice *vdev) vdev->msix->pba_region = NULL; } +static void vfio_user_dma_read(VFIOPCIDevice *vdev, VFIOUserDMARW *msg) +{ + PCIDevice *pdev = &vdev->pdev; + VFIOUserProxy *proxy = vdev->vbasedev.proxy; + VFIOUserDMARW *res; + MemTxResult r; + size_t size; + + if (msg->hdr.size < sizeof(*msg)) { + vfio_user_send_error(proxy, &msg->hdr, EINVAL); + return; + } + if (msg->count > proxy->max_xfer_size) { + vfio_user_send_error(proxy, &msg->hdr, E2BIG); + return; + } + + /* switch to our own message buffer */ + size = msg->count + sizeof(VFIOUserDMARW); + res = g_malloc0(size); + memcpy(res, msg, sizeof(*res)); + g_free(msg); + + r = pci_dma_read(pdev, res->offset, &res->data, res->count); + + switch (r) { + case MEMTX_OK: + if (res->hdr.flags & VFIO_USER_NO_REPLY) { + g_free(res); + return; + } + vfio_user_send_reply(proxy, &res->hdr, size); + break; + case MEMTX_ERROR: + vfio_user_send_error(proxy, &res->hdr, EFAULT); + break; + case MEMTX_DECODE_ERROR: + vfio_user_send_error(proxy, &res->hdr, ENODEV); + break; + case MEMTX_ACCESS_ERROR: + vfio_user_send_error(proxy, &res->hdr, EPERM); + break; + default: + error_printf("vfio_user_dma_read unknown error %d\n", r); + vfio_user_send_error(vdev->vbasedev.proxy, &res->hdr, EINVAL); + } +} + +static void vfio_user_dma_write(VFIOPCIDevice *vdev, VFIOUserDMARW *msg) +{ + PCIDevice *pdev = &vdev->pdev; + VFIOUserProxy *proxy = vdev->vbasedev.proxy; + MemTxResult r; + + if (msg->hdr.size < sizeof(*msg)) { + vfio_user_send_error(proxy, &msg->hdr, EINVAL); + return; + } + /* make sure transfer count isn't larger than the message data */ + if (msg->count > msg->hdr.size - sizeof(*msg)) { + vfio_user_send_error(proxy, &msg->hdr, E2BIG); + return; + } + + r = pci_dma_write(pdev, msg->offset, &msg->data, msg->count); + + switch (r) { + case MEMTX_OK: + if ((msg->hdr.flags & VFIO_USER_NO_REPLY) == 0) { + vfio_user_send_reply(proxy, &msg->hdr, sizeof(msg->hdr)); + } else { + g_free(msg); + } + break; + case MEMTX_ERROR: + vfio_user_send_error(proxy, &msg->hdr, EFAULT); + break; + case MEMTX_DECODE_ERROR: + vfio_user_send_error(proxy, &msg->hdr, ENODEV); + break; + case MEMTX_ACCESS_ERROR: + vfio_user_send_error(proxy, &msg->hdr, EPERM); + break; + default: + error_printf("vfio_user_dma_write unknown error %d\n", r); + vfio_user_send_error(vdev->vbasedev.proxy, &msg->hdr, EINVAL); + } +} + /* * Incoming request message callback. * @@ -103,7 +192,28 @@ static void vfio_user_msix_teardown(VFIOPCIDevice *vdev) */ static void vfio_user_pci_process_req(void *opaque, VFIOUserMsg *msg) { + VFIOPCIDevice *vdev = opaque; + VFIOUserHdr *hdr = msg->hdr; + + /* no incoming PCI requests pass FDs */ + if (msg->fds != NULL) { + vfio_user_send_error(vdev->vbasedev.proxy, hdr, EINVAL); + vfio_user_putfds(msg); + return; + } + switch (hdr->command) { + case VFIO_USER_DMA_READ: + vfio_user_dma_read(vdev, (VFIOUserDMARW *)hdr); + break; + case VFIO_USER_DMA_WRITE: + vfio_user_dma_write(vdev, (VFIOUserDMARW *)hdr); + break; + default: + error_printf("vfio_user_pci_process_req unknown cmd %d\n", + hdr->command); + vfio_user_send_error(vdev->vbasedev.proxy, hdr, ENOSYS); + } } /* diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index 109076d820..6bc5809cce 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -201,7 +201,18 @@ typedef struct { char data[]; } VFIOUserRegionRW; -/*imported from struct vfio_bitmap */ +/* + * VFIO_USER_DMA_READ + * VFIO_USER_DMA_WRITE + */ +typedef struct { + VFIOUserHdr hdr; + uint64_t offset; + uint32_t count; + char data[]; +} VFIOUserDMARW; + +/* imported from struct vfio_bitmap */ typedef struct { uint64_t pgsize; uint64_t size; From patchwork Wed Feb 19 14:48:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982416 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DE53DC021AA for ; Wed, 19 Feb 2025 14:55:46 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklPz-0005TZ-R1; Wed, 19 Feb 2025 09:51:23 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPW-000589-Pp; Wed, 19 Feb 2025 09:50:50 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPU-0007fC-6c; Wed, 19 Feb 2025 09:50:50 -0500 Received: from pps.filterd (m0127838.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9M77f023811; Wed, 19 Feb 2025 06:50:44 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=h10HlgpitvorHBaRD8agt8MJNtXj/9PBe26dcGJVU Zg=; b=OgpEOMcSQ/gJlppBv8bJQdE4Qxu9qZgFdgElDv8r2mEA56HWztLI4SQDI 34/FHdBZSCO9pEf6X9Q2quYtZkoQPP+P0BVlOI/ZLL5N78eR7UyxcAcXP5sajY60 i+UlzGkjSzAoJcV7miiHF7CmArEDV+97trSZafMYuSFFaOd8DhiE85l1f8jF3EIR QvMg2JJi6BqrMBbM3NMTMPO43YlCOc4Ga6b42rWYIoplmydOkQdA7GoYJsQ+T0S+ iXr8QkAe54CgRVvKY9iNVEE3+hClTlvY3IEM+M38iM1eIChBxzvtkehyM1wC1qTN 80SVUc9UUXEz8dEe551jeV1+mTIRg== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2045.outbound.protection.outlook.com [104.47.70.45]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basyc9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:44 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=mwbl9/dkczxmzVfgJOjIcn8h3OH4kCfxX2PU4pB8uXUsIyOSO1ikhvm04/tdvuePK8JiZ7+T9f3/cyUeA5u9KSv0ZqQIX9+LfKp/WexRIWNoYdKFq7S7b3R9X7QV/Lw/sjDR6dhkMSLvBywWuhFq/9pvlNVYeTjxXV1nn+12lHcJX88Ui/KVOrlYYdQpUEZRz0RuLOOKrxNPJnBSBClCBiTYH5xrbG+mVRAbKqYXplzm276ee+lCe7MggUnryn/lPwlHP9UnPkfxRwYAXHJlAjB+hA+Kwg4qzMMKbpcduhB1z8P3OPVRPbuNDWslwbv/XYdwGkfWmUfSRal+KRaBng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=h10HlgpitvorHBaRD8agt8MJNtXj/9PBe26dcGJVUZg=; b=ePtjnrtUvjTVtu9o9TMVECYOimv8Zbasu2Fh5dMLu2v/obBB5tRB1FOkaPc3dMvnI6ZvOOi0Dhf27JdJ7DgjowL2uZiQTQF7UKOOr6UPCGPLDFNPgF0WZULgRpPUNDBYNd/Ph0RqUryj07J9WmujlIMGb8Ye/HR0Wi0KuN2O/EiyKRvo2Xozj8ZIExwnVkicPC3yZuHY9WzFhR77iKvgDUF1fRwc52Wj9NrDucwNUVQhoa7VG8/XmySagPYPYWUxnKdhGeU8QBd5h03m2tMMaPSs43BCU3Q6Yxjq15BytfadOND6rmmcUgmapwmMF/XcYgBh+QQkixuJjq10OTsrgQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=h10HlgpitvorHBaRD8agt8MJNtXj/9PBe26dcGJVUZg=; b=ntT8i99WW+dBL4wIbsbu3oh0dr/8EgjHod/Y4Qf25ROhgKxaQmZ25vbe93WZ3aJm400ENvzsvS0uE+4Lk2FDZde4ReXwhrzIxNQMb4Hxupx9FpAhQJ/Qp5O/O3OU5XGXUu4xt3VMDlPWpDf/1e5Jqqbd3wcp0a+kmHDze+EUNh+AYQZtOez55iubGpXUHq+JyfzXikCQbzjwtXqqFxwG6V0mW97L8gxSoht9FrICcTh0bdqImp7ZNQVsLS7j3f+TPIJF8aieUrYeB1x6yjPr2TNH6eyWXXroX0t2CYUYH2PCcn9Mf6MPPkbZcLdteYGxIMMExCEyjSMPxJ9GAI7g5Q== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:40 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:40 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 26/28] vfio-user: add 'no-direct-dma' option Date: Wed, 19 Feb 2025 15:48:56 +0100 Message-Id: <20250219144858.266455-27-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: cf54e931-1f06-43df-8bfe-08dd50f4c787 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: Z5+P+hQi8fyTz2VB+8xVk73nds8Uj0gDhhQgQUQ2V0QwoLZarmjvEkzsviOEweL4b9QgIz6iTXufEJCITcFMWZLDNDABDjOM6Gx3CWfuD1QHGMKoTMMXKy3gRN1zQUrsouiE6HMwXZAifDyTUNLufzj22/qMv8vEHklKJ5lW9V1VqLo0DQ7vfedN8zdr/rWHrF9fd2xzjBhXguwDN8sfy7T5U4gwtUTTJgHgrzL/MfYbYC3kntQtI9McwtcfkKdkYog79UH7zimENM7aSqL8KnWWiHrzrLHay6u62nubS7JNg/uq9h3eVkJp0IuK+4ROYQC+4DcQbRKnfBRwLNNOlasqTSm7it9AbEInn/ze+3LGxVgie8nOsKU8HRIIZpRGDdCCP+Y/ofU8szAXZsCJgjeU3WUIZ9wbhFH2yGYuW1Acr3e954rvXqjW7rZrWbTn5U3XN/x0Mp9w0asQ6W7uehCsUNgiGp/akjWKRu+s5oe+ZZDpbtjtzo/xPCHX34socVUtBpUYXu6N2izWUtlvDqfFcep6BnXxYrRQtpTg+jOgQBPnxSxdajk3Xku1+4O8izdU75yv1QflknYy92l3UumvXqQTSFw/+VaBathIw6ryc6KN3aCtHzOUvkAxXJ2j1JWq9Z2w3+rjY29RclxOEtAVYLPSO9Il6Ci74JCmmqTiBpbDX1Zrj5oK3MH/Mzb9M5Ipm8MOdrhxWTVXfzlw9JOhoqJ/2WJJVXJxF2C9AphbD7+aiQTAMozyyLu1PTc02zRpFYnpPirworenZrrQJ7S6CrAlvTCsG8WPvHSTGG0A7HZY0/LlT1kL5uEyvCSH9drJfH983u47Z405AtPbMoNH6BTabzj+duf1jJjkCKihfSDT4CTIuO8Hof1lHJ0eG97ozGym4g9RBU0d1fh4SWQg8PdHZmVxrAczVTyh2sicQYHdvAvJIPeVC+YiKaP3LiQmAr5rvAsQRLY5nEpNPOE1u3dfoYwWyv1C2T4aWwXDwE6jrMdffEGMCoVWu/zVBUJxpEstOp88cZkNvh9bjlr3mdMIhfmqTmmiXcYtSQiODHc9Sxv8dgJbZNP3a2tc2dBWAzBObU41bGKTB5OA4GP9t3SzaEu1nrzeubaqMCoJLpQwNGHUfqsVjx9CjpJ1dkz87HklemwGGHawIb7LYi3/6uyPBkbHxSbenMt6KPp8lXJbhgnNNzmNXXA8DsIgwLOsOUcdyW92Y3pmrNewtI9tq8f5qbkgKTsFKK+vxsKAjO+EoxxsBIMTfwona3qpKYMGP8MmusOEjwlVKSC0fPQliSTSZlDJ80/zMmB1Gohaj69Wk8wLK2XlObDiNr76ES6gPtxiM50Rrq8LED/2U7Zic7SmC0dlVUYbm5RVryGjfOL4Vq1nGUMBMpiSB0tz X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ys/07sGmjlRlDdo+wqsA1eG8HgOHFKgRUUtDdgo+0T433c0p51/sXV/x1iZjxIKENU8ly5YSoo6LpGCvvNzZySFQywBID60WHqaILQc7a/euBnfyM3FAzogauev0gsC+S2VdNZ8GFOmMfmbgUZq/FvfhbtNzEarUni7dW3UXizOmn8kPOLwlOnzJQY1lv2oipO3mwcjRwdQfdqsLCIN+vD8HcDxPxFFkus9sqYspKizsjazqDVqLkXrzhHu8ocluE2tH12R2ZGDr0aJVhU/UXvW2YpLqPkvj0fBjn3BAjT0ndAV8XVqTsy46Qrj5/YI+YbeHchWFc0IU6IJ/drUvFfDdRsGptF50TMb1zivmssPwyNov61B2o74CEFsP8oXIKkgx72vvKSLCtJtd08W0fcUHN8zdQD1IPsK4FcgWO7pALt1ScVtxnKKLoPrb2jCrJCG7KJQLOyj7xraVoEQHAJBMVsMcesNg0b/Y25Kw1ZuVedq+R1hEHXrMRcyOyDo0dugv9dvPozskrKpnmi9pDQQnFkfEY8KdjEzeXqHsP2KJO5DWxev3MAvaRUzzD6km/dyeJoGtf0NrlAb7f62VtEn7gMv5fwe7DjGYlm1QjUH8UvolEHcaPC+LOFKBmtSQR6ylq3HBe66DAZmdIzxUSlfyZo51YEu9U62l+RV3FSaRbSrXrEtFIZLND3DkpXVQ7JrgU24mYL6Zhehgf3eURcZSYPjZUn3lSHeWSRwKuI9fQI+UFLBGMo1wMwwcc/eAwTGviaQK8OAurwwfWYF8r2681IlntqvaeYoy30zERlwXD6hEDFyZg6wqtijL2ibASgb+52uruontiySWXVy3fp/2deHXsm3Qjm8GteJQVk4mZ5v1qaDsQzJPV4I93OQsnVBu+1ew2iwd93qBTErel/ROeiLE+AJ8bGDKAED1sGOt70DS2peS5rxe4ZF2zCGP6g+SsK9kJA3agVWD2fRJjVQu5d6G92aoR0TND+QewA/iobv5OPDP6fmU72FqOc/TxAMcJI7KZ67NEYMKjcJ68lalmtA3cErRcBRI8c5V1lQ76YtKtD+8nDr+HpwJZ/O7Zbh2s5P6pd5gJ1koezDHdhKJ9/J3aUUvgQhlaHXjiPLzqQgEkw89ljI1q9HyS08GkWlJeLagze41o9WclEzLyNLhEPyCPL8Ch4qvMIPLFh8lRvRHhttxR+lhV4oSxnKHz6H0t6iZJgBOBfMZPb8JgxRmLl7LI2O9Kyr4CMIecAaEbeXV+gbx3u1LlyS8O0ZBEpZG1dmn3tV2+HKli4sKGo77DoOoa0d+WW0GJbfsEXnBtQQePger6e6RgeAE4056hf+0O+04Tqs6eHj9SMXg25SJNBu/jW2YL97dLhffQmlzTwKLBXoJWqP007B7+BHPtwUbWsxqc+3J4PW69kjIqOiZg9oTSX7ietTEbzda5RWn0hSL36ihCIpVmU+gBjU+2Mz2yVN06tqS7lTu3oJl85YyK3TjNJQVhlXDu24JEh2ZNygjESVm6JB8KeTD1/cdoOXzR6lkvZJU1UIgPES55azrig7FkZSh/Tn0ZradGcoj5rExoLhaKJH8m6M1Yk+C X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: cf54e931-1f06-43df-8bfe-08dd50f4c787 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:40.7847 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KiWOfVk38lrc+MhkSGivNYPiJ/rrUQ0gYCUF6X8aCbZsAlCqOv0BynzEbq4AH3MGWp8GXLAvKFzPYEnDt8BqjQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: 5WVOL5rPDXsSbDJW0w7rKPmOlm087BwJ X-Proofpoint-ORIG-GUID: 5WVOL5rPDXsSbDJW0w7rKPmOlm087BwJ X-Authority-Analysis: v=2.4 cv=d58PyQjE c=1 sm=1 tr=0 ts=67b5efc4 cx=c_pps a=WCFCujto17ieNoiWBJjljg==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=9g9m8Zzpkuj7DW7ovtoA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Normally, the vfio-user client will share a region's file descriptor with the server to allow it directly mmap() the region memory. Add an option to disable this, so the server must use VFIO_USER_REGION_READ/WRITE instead. FIXME: doesn't actually stop sending the fd?? Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.h | 1 + hw/vfio-user/container.c | 2 +- hw/vfio-user/pci.c | 5 +++++ 3 files changed, 7 insertions(+), 1 deletion(-) diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index f8c61f2128..72138220ba 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -84,6 +84,7 @@ typedef struct VFIOUserProxy { /* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_NO_MMAP 0x2 #define VFIO_PROXY_FORCE_QUEUED 0x4 #define VFIO_PROXY_NO_POST 0x8 diff --git a/hw/vfio-user/container.c b/hw/vfio-user/container.c index 3974bc8a8c..3880316238 100644 --- a/hw/vfio-user/container.c +++ b/hw/vfio-user/container.c @@ -102,7 +102,7 @@ static int vfio_user_dma_map(const VFIOContainerBase *bcontainer, hwaddr iova, * vaddr enters as a QEMU process address; make it either a file offset * for mapped areas or leave as 0. */ - if (fd != -1) { + if (fd != -1 && !(container->proxy->flags & VFIO_PROXY_NO_MMAP)) { msgp->offset = qemu_ram_block_host_offset(mrp->ram_block, vaddr); } diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index e65c7eaf02..8a05e69a46 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -36,6 +36,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) struct VFIOUserPCIDevice { VFIOPCIDevice device; char *sock_name; + bool no_direct_dma; /* disable shared mem for DMA */ bool send_queued; /* all sends are queued */ bool no_post; /* all regions write are sync */ }; @@ -264,6 +265,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->proxy = proxy; vfio_user_set_handler(vbasedev, vfio_user_pci_process_req, vdev); + if (udev->no_direct_dma) { + proxy->flags |= VFIO_PROXY_NO_MMAP; + } if (udev->send_queued) { proxy->flags |= VFIO_PROXY_FORCE_QUEUED; } @@ -402,6 +406,7 @@ static void vfio_user_pci_reset(DeviceState *dev) static const Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), + DEFINE_PROP_BOOL("no-direct-dma", VFIOUserPCIDevice, no_direct_dma, false), DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false), }; From patchwork Wed Feb 19 14:48:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982408 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EDF7DC021AA for ; Wed, 19 Feb 2025 14:54:14 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklQG-0006Kn-5S; Wed, 19 Feb 2025 09:51:36 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPt-0005cf-9A; Wed, 19 Feb 2025 09:51:15 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPr-0007fe-7n; Wed, 19 Feb 2025 09:51:13 -0500 Received: from pps.filterd (m0127840.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9K12f024315; Wed, 19 Feb 2025 06:50:47 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=HMJ+9iSjjqAl305TKYaYKWiCdY9JvKcSY6TVzyMbG Z8=; b=IPPJ4QdYTxPBuMsa5Zf8NujQbbzO+vMAQ7VRpaty7E9VoW/Hp6TifmWX/ VShirukT71Yjb1r74XA5ikQ4fvMAvCKF59NGlHwDRSMHe1YJVK9mTGd28Z6zNt0X GyRQEz+7AjMhpUG7VaLrQBorTzjefOIQ8HZA/5D2nOfLllJ88bn5Da01mvOF2SO5 GRYdT3tJd/utLFH7darmh/pQUbVxrsSiSwnuDvEJQMFfVjHl8/4VTqFcbV4leYv7 RGje9FSeIH3o0WWaLQrIsBBo9jdUtRihWy7zTeGQt5DvsdL2ar7HMPM3CIQcMf4p GFSkK5+q1r7RrnXAgGiLMco+bVmOQ== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2042.outbound.protection.outlook.com [104.47.70.42]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basxut-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:46 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=UmQlb+roUhivB7XOw10l4GNM+Y3U1oMQbpcy674EnDx340dMEBTbqkEoY/w59wpD0TvZp4HH1UUrhLrz1fTqgrfzRHFmAg2V/mLCi6pcwb+REvARkUsjKF7qgePTK5WO2dLKu+hxbg7XHRw/AsrITCsd57KCH0VHmanYtV+RGGX454qmoV+EVwwo6wmo/xWD4+hWxbgUCkV2sa1NvXvl5Eqg4ZDCL6FlpeItSOe0Prp6gyTF+9mMZb/oQrReOmGJkjePTDwWvkoYTQ4bEHtTXCc8u5PjmZojgMgxvxFhCJX44Rtxvf0K4cZxoTWH5msuehK+vrGn1T3ZbqjAMmL/uQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=HMJ+9iSjjqAl305TKYaYKWiCdY9JvKcSY6TVzyMbGZ8=; b=wSDhLVLsAs9goVpflcFksftehecA+W8qkg8baVkv5iQNfQt6EMVAEoMrTeoM9hHVAFmSeu2SKH2mwy66GeiMfxk3Pl+WjzVVaC+ODGKwCHuzhclCw523RNgfvBgLamVeUMDukmzbGhyO1aO2YT/IY8amfT5W49nor8mRny5vBKMm6rToKOwOPhHnyv/XnVPVFFzS4sT/JzTi7Dad7CysouyWIAHWRhB7hzmigNITqdkke3oyQnTl6BOPgh4DLvAbxUqmfG9pM7HcysHzWwUKcN+IHbVdYNkLpnTxFre/ZuGXE6bcus7jSQX7PswX06H1vxXaj5u+LY9AnBmn7bjMwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HMJ+9iSjjqAl305TKYaYKWiCdY9JvKcSY6TVzyMbGZ8=; b=lwxyZF+1rCj/tbVolFn7V/QlEjyYXmYvXbZjsg5MBWaZ77jt8W5q4duCCBcswM8skG+Nkju+XEmIvZkKSstEjo7er0h7N3bSzghCs3hoLLJ1K2EaaheXQLvz9yTAOVZhVSsSKeSC/GCmEguamrvWf7uTidN/1PI0J07BG59Kjl2C9K6iyce8kw+24lNVrwZQbFN+GiiNyZ5xMTy+mpBqaEMwKsV/mWq3TiRK/uF90HsH4G1sPLEryigZI12DtroUuUKsS8/AHWctac1IP3bWwtuxK5NEMyCmbhKFw9kmoyCG9W4PNSu60ozgXlf9vDbRh09WadylbSont7wylO2FIA== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:44 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:44 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 27/28] vfio-user: add 'x-msg-timeout' option Date: Wed, 19 Feb 2025 15:48:57 +0100 Message-Id: <20250219144858.266455-28-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: e7bc0996-0102-4459-1216-08dd50f4c987 x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: l3FWCjzL/G7r1cqTdpFC6z5jkDS4qY7sDdpHB56zuLm8Op0xFaTSlcEwVbWFvpuVTDQ0B7vmC18lIRMkHIu22AwmGB4uKHe/pwtwVFMOt/jkfOezzf+BOKlh+20xDO57g7mBsdMs3W+4174dCzJNkG6ZQYMEcts9CJXQKj6FmlVIo6nZ536yUXeyLPWNlVHcHjOh3mt0b7d+nznu+YskBw1nGMiOxt3KZxCxVuIOGyedSm1huFEThSV4jCXhO65az+1GvnL1/k3yDhI8guFpQS56gWz+bZs8t4Eju4LTiGwVjTChb7JX52/9WEUvOeN6Ui1TyuY52iReJ3vaAy5vVXdfFa27ODOhOGue8Z5VJ8N/ETsjZQ65m+eW7L/iUnje6VsGKFQ9JprIcQWsH8aymi4xx7LrqWbLu7dZUpEjEBUsGCspMeD8QhKRMKOmLR4SxQolmLPh/zgO71YmFHdkpuWIfvPMrsnV4V38EU2yX+kU+uwkIo/Yh/0bNOsXPdIK+96iBgCZTcIcQBqSC3coz3+uQYLERT5p7PKaiCuJ9tmQVeA3/1VLAmo+5NDV7j5IXUWQK/oZ3KCb9pW5dA2sNy+SdHSYsdAYFSoCE6C+TU0UIN+CvbwgSDsYO2O6H5mRRZEfl4Cz4p5gWmzXNIaVtH8Iw5bAFjJHSPz6EdY7xPsKyR7aFQOn9YHw2dwRtKhZ/J7+H1UH5Kn9I9Jrj9U4DJi+2aC6H33LBy7PvYg0DdLzc6b70wEJrQqByvm1MSI/Ug8cc2Im7X6cw3AUEgzw69OQ0/nVehzAjAivMdIpGlOkHU26b210qHFOwhvDfk2tcQkZ3nPIeqbwPPAxVKc7l9+qkwNvNGDIYz7RsfTD49llQFes69bU5oDvxJ4GTv4Qr3SSYrDf9i5llPVRpdz6aX0ug55TqTp0DadZtqKZYvE0TmPSm/1FPvNterV291O0HatKdoXdUwiK0Trcco+pd9jFEUF1kIwRDOtjFu1Bi5zzNYwnAuZWym5wjheqDK1rIRbfi0mInqa6RzqDMkz5vPwOuF6q98enqZ2UkxVrJ5QseLiYelXxYMaVTeJbc3PZ6n/NqYSgQlBindLBdR3U/wUZ12EVMLqN75ziCrlITL17vgt+hD5d916EJexe/rq21dNqM36dvEZ68zG64x513iIqzfkhStsJKghHFZ3KlIGKU58+zsO4+wVRKOiEhP9H3uuj9wSM0TIIhv9cGXOwqi5odGV3Ow1M4BXvxB+JxZKdN+iqXKMDiIYRevUE3l2OAhmrvIMOxLB7Wg7oO1dyQ3AmuhYO3LyBCXcfaPwoPqcUGQlaLMogGH45ntOEnb212piuILYf7suSUZTSfJ/X2kS9EFa6+erNFmfMJCXlTiAasvbROPR9jtdUqpXisYFi X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: RhEV4LCUPn+00Rhu6oHqJqSLkQ2S9ykzLM6UgIU/RKPYOIfma9vTZ+9EYjiPi19RJqorYbKbSRQ2SlS4TXw1mNSiHoGOScsJULR/Cw6MuvVmo2IAMkN60ApxO2Xv7GXw98mQxTxKRg1CkcwEAemsqwuQ1tR/haR/G7lPZas0kyDtAS3ufjd0Z06mJvzbWvT33bqaLSIlitR0swVZ1IawyJAorKbWu+2TrY/+uhEL2edEUHvblbQpunlLPn+bcXbVXU7Iq45zpeJsyrnFB7qtyow8OKp96Uh5gJ14U9yiO736Q/1pHV6DlhDSrCKb9xDR4vMMwVqF4q3GiYC/DQR5BeojvJdjItCP52iFBfMzUGoZaJJJSGgU5O69F613jt8zs/DSxyO+XUrVgwDhn5D7wrdWzEqlbdtl2pcwW0uRdn+bbQlNqiYx6iWuRev92Wj3ETRS5TcWDeD0NO1JFQI5F37UipHlxqHJhFsXtrOh3SU+Zvt7StpmGzD2n4OX15TvJRgRyM0AXlxpktvPMaNt3EDAEurvNxw9AwtpirfFrcXN1ik0McmTGfvpIuYeYWOyyu913OQmD+vGsh3SIcI3GcG316a1sjqP9vUwX/sLE0ABx/xkteLWjoBx+96g9TC11l55TcxhbfZlTzk5Z9EF4xW7ti5PJxmPEA0DmDW+gjdg6OvOOus5a6S0N2gD8LIl3LxLgjfRWNWMoQWthkSFi4455dLzfC7Q292g+HfWDZPLiNO60Ksj+RjLpn18U+Ehdedf2S2x/jsd//l1HXwmQtW9KC222xE+nTlLUkMMJEARZdiSRNEtLm6RVqDEFHFpuJ75eMhLl3s+M39iuMQFeKL74UPS8iZsktHPjX/JQyT8vniaLHcTnLTXmE9Xvt5PdFyL/o5BRz2uaSKfWDyAFnmtUbjmQmGX0sKXcmX5gEJ98p2hp6q2rIr9CPAnFeTxEpjhgDTlbZhhzvCoP/acX10fBImLX0D6i+Hyrbe9ARhgxqjYBB8LPAXakEJC447LDJXtX94VUi/84paMT1ptKY5PUMuJwNkNBvJFpSwi6s2rIzkm0YBDAupfb1ObUlKACKv5e6OhG78rZshHIkKzSx+ZN+Gp7Gh/ivR/qXph1g7rj4JJ30Fn1VhNU31KsohyYSdNUM2Vvp2XGayb4lIG6r3+Jul4NcgE9oF9EWMk1k4wmtOvH2RkEZG88uW3jhpuxkf4JFHt7cxLVQbIQqaenXCPl0pUqkx9/hqoEhNAQ5K2KALqdxyuS5PLrP9x6hImfZYYx6CwnhnJh5Qeh5UqVUwijIedTttemK7VPF7bnNm/4XoTWqOsDlFY3fF02IjBq3Ppov0g6nrw0f1d+NkxL4KJ0PlKnMTN8BI2hIPBYCEhsMeo6UbjYc/npAwufsFW0uU8zHpuFg/gOq6SqMWhJKXfyGu/fW9irnaGjci3GZxOfJ9wnUvJYgPpJ7Ayj5K4vzwujKKTlcfKTdJYS5dsAtK0rpqkXBaOlEfYztVSrV3JK2JnkXzomSyf14v12Rb8sHCzKg3GlSiSU18UVpeg47+Pd/fea1CdldthocNh1SIYwISV0Xd5oCgnP83bvWjq X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: e7bc0996-0102-4459-1216-08dd50f4c987 X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:44.1037 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: q2i1/IK8TKgaxrcQwxCU7H2vfw6HgfHBv7WmkiVtrvgBeYMujNwKLQ5fW+QV5fKdDZTMzUvfFWQ+qlDqj5lVsQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: oinUPCP5A6FnwUVWqPVKpc9ufvnTvFWH X-Authority-Analysis: v=2.4 cv=bfyRUPPB c=1 sm=1 tr=0 ts=67b5efc6 cx=c_pps a=Dwc0YCQp5x8Ajc78WMz93g==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=Ql6ANm6PklPDYutWWPYA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-ORIG-GUID: oinUPCP5A6FnwUVWqPVKpc9ufvnTvFWH X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman By default, the vfio-user subsystem will wait 5 seconds for a message reply from the server. Add an option to allow this to be configurable. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 7 ++++--- hw/vfio-user/common.h | 1 + hw/vfio-user/pci.c | 4 ++++ 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index 38f8eef317..e44c8a2568 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -37,7 +37,6 @@ #define VFIO_USER_MAX_REGIONS 100 #define VFIO_USER_MAX_IRQS 50 -static int wait_time = 5000; /* wait up to 5 sec for busy servers */ static IOThread *vfio_user_iothread; static void vfio_user_shutdown(VFIOUserProxy *proxy); @@ -707,7 +706,8 @@ void vfio_user_send_wait(VFIOUserProxy *proxy, VFIOUserHdr *hdr, if (ret == 0) { while (!msg->complete) { - if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, + proxy->wait_time)) { VFIOUserMsgQ *list; list = msg->pending ? &proxy->pending : &proxy->outgoing; @@ -740,7 +740,8 @@ void vfio_user_wait_reqs(VFIOUserProxy *proxy) msg->type = VFIO_MSG_WAIT; proxy->last_nowait = NULL; while (!msg->complete) { - if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, + proxy->wait_time)) { VFIOUserMsgQ *list; list = msg->pending ? &proxy->pending : &proxy->outgoing; diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 72138220ba..9acf634ca7 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -62,6 +62,7 @@ typedef struct VFIOUserProxy { uint64_t max_bitmap; uint64_t migr_pgsize; int flags; + uint32_t wait_time; QemuCond close_cv; AioContext *ctx; QEMUBH *req_bh; diff --git a/hw/vfio-user/pci.c b/hw/vfio-user/pci.c index 8a05e69a46..fe096cc7a2 100644 --- a/hw/vfio-user/pci.c +++ b/hw/vfio-user/pci.c @@ -39,6 +39,7 @@ struct VFIOUserPCIDevice { bool no_direct_dma; /* disable shared mem for DMA */ bool send_queued; /* all sends are queued */ bool no_post; /* all regions write are sync */ + uint32_t wait_time; /* timeout for message replies */ }; /* @@ -274,6 +275,8 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) if (udev->no_post) { proxy->flags |= VFIO_PROXY_NO_POST; } + /* user specified or 5 sec default */ + proxy->wait_time = udev->wait_time; if (!vfio_user_validate_version(proxy, errp)) { goto error; @@ -409,6 +412,7 @@ static const Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_BOOL("no-direct-dma", VFIOUserPCIDevice, no_direct_dma, false), DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false), + DEFINE_PROP_UINT32("x-msg-timeout", VFIOUserPCIDevice, wait_time, 5000), }; static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) From patchwork Wed Feb 19 14:48:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Levon X-Patchwork-Id: 13982402 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2F0A7C021AA for ; Wed, 19 Feb 2025 14:53:03 +0000 (UTC) Received: from localhost ([::1] helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1tklQI-0006Vc-SA; Wed, 19 Feb 2025 09:51:39 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPz-0005n2-2t; Wed, 19 Feb 2025 09:51:21 -0500 Received: from mx0a-002c1b01.pphosted.com ([148.163.151.68]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1tklPv-0007gP-Vg; Wed, 19 Feb 2025 09:51:18 -0500 Received: from pps.filterd (m0127838.ppops.net [127.0.0.1]) by mx0a-002c1b01.pphosted.com (8.18.1.2/8.18.1.2) with ESMTP id 51J9MNpB023984; Wed, 19 Feb 2025 06:50:52 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; h= cc:content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to; s= proofpoint20171006; bh=VVfuLbKvyGRDB+8NToYZ0v+0jzdF+wvxukjmVQLeH Gg=; b=w5qsa69YYXJjidBrzfLLWPwZWzZ5MUSkWBST9jJX4z5mzsoV5PABTidgu RAiFCFMg7mJyL9OHXYsWcBcCFWbvkRDwIBNhDcCBN0ViR5e2QmBrVod4B6ZyTKgg iE9ZQSoMfQADOWEUDG3ey4VH/mVmnH7dqsRaPkGNs4qOnHyiaOAi8pv/2wIaZcvG +lzqejEmSkG7ueEST7JUYSzP322llu09f+mqEKvMh6SEqRpI92bdjDnn3s3ADq92 d5x3yyDH3XLRDBlON/fCKG7GRPcZse5m7JmYGVL8c/q9mqGQQHhVKMIRDQArdvoZ sC2cVSPkdqwL17EHbbjVWs1awmkxQ== Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2049.outbound.protection.outlook.com [104.47.70.49]) by mx0a-002c1b01.pphosted.com (PPS) with ESMTPS id 44w4basycj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 19 Feb 2025 06:50:51 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=KPWnE3DEgpEkidBDq2bcGD/hKL1qvyyxdy0fgg6iA/FGSdW0XT79pYrvoO4S4nQicuV4Tw2xLyFd+5o3kt6SSmhHFZFwUl6ilTuvqN7HZtSypShzsHj0D2g+Nso5BQfEjUk4sl7xQKKx5daR85VDfeUnVffoYjSgO9U3CWZmJ0uhhsmMpAsAeL8QpXGTYNrdMjVv094pwtqJNS3yZv0HkCy5RcV9HaMcx8Y4k64kLIk/zUcqANFI33ITm90MTtoSLpfRcCmvYY3nxY8L9BIxXwALZab9YNEdnrnoPClBPuP6qkDhRkfsLPJZmknrICRNnsq/ufXdLbPOZm5iOgT/8Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VVfuLbKvyGRDB+8NToYZ0v+0jzdF+wvxukjmVQLeHGg=; b=LXRtE7CLSHrymhQfCtIBY8PJyBtUAfmI0X03elh2lhPLfYr9pajMJok1qjkqmKID90xkakNtRBH4k/ePOh3Zn7l7GTWRjsvRy8o1DuEwjV/zI4GsYDttFfoeEXZacqwGQFzpPqzpSnbqPUD7Lx0afVvjBTkhBdkFwJU6HwYldKXS3gdkcYMWIgJ9awHbysAjQAq+vv//VxCKRYitD9oDCVtT6+LIItOlq34NwxqXjWVBqmqYQcoKc9MwsS4JFxX7b/E6Qcj5DfkG1rn1kZtkK5I7IkW+zAEj5Y6A3qmSzskuQieflUE0Q908t8gDFaXdTawfRH0+BzR9lZZVKuWT3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nutanix.com; dmarc=pass action=none header.from=nutanix.com; dkim=pass header.d=nutanix.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nutanix.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VVfuLbKvyGRDB+8NToYZ0v+0jzdF+wvxukjmVQLeHGg=; b=icdWuITG+O8oYV9/AojU+8XL+mTX4WUvDz4zFh35+cR++atHBdIC149a9zwFiHaUMMyQffXiFmnkWyRSevTAjVB3Yd985iDPYxPIRfonCH4tomIYPuqYB7BPxedwBuPh/Opbk3CtfbKOq00OqAmSnLsd6cWogZz9rKfhz4jfQxHq0EtualJ4o8ryTZkSuT7mxd4IpF9UB4c+U1ZC9zzldL/fhHRt6zTpMMIMoOQ4CZlrd3kXHXPSOIlKiwLY3hW7A+tkzxmYFqxKflIbKgcoKv1UmHI3LGPmjjlQSPeNhSdXSp81zEWpMfOoDaq7RA3a3dYihVLbxiPZakoYi3xGVw== Received: from CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) by LV8PR02MB10096.namprd02.prod.outlook.com (2603:10b6:408:181::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8466.14; Wed, 19 Feb 2025 14:50:47 +0000 Received: from CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51]) by CH2PR02MB6760.namprd02.prod.outlook.com ([fe80::fd77:ea65:a159:ef51%7]) with mapi id 15.20.8466.013; Wed, 19 Feb 2025 14:50:47 +0000 From: John Levon To: qemu-devel@nongnu.org Cc: Jason Herne , Thanos Makatos , Halil Pasic , =?utf-8?q?Daniel_P=2E_Berrang=C3=A9?= , Eric Farman , Tony Krowiak , Thomas Huth , qemu-s390x@nongnu.org, Matthew Rosato , John Levon , Paolo Bonzini , =?utf-8?q?Marc-Andr=C3=A9_Lureau?= , Stefano Garzarella , Alex Williamson , David Hildenbrand , =?utf-8?q?C=C3=A9dric_Le_Goater?= , Peter Xu , =?utf-8?q?Philippe_Mathieu-Daud=C3=A9?= , "Michael S. Tsirkin" , Jagannathan Raman , John Johnson , Elena Ufimtseva Subject: [PATCH v8 28/28] vfio-user: add coalesced posted writes Date: Wed, 19 Feb 2025 15:48:58 +0100 Message-Id: <20250219144858.266455-29-john.levon@nutanix.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250219144858.266455-1-john.levon@nutanix.com> References: <20250219144858.266455-1-john.levon@nutanix.com> X-ClientProxiedBy: AM0PR02CA0154.eurprd02.prod.outlook.com (2603:10a6:20b:28d::21) To CH2PR02MB6760.namprd02.prod.outlook.com (2603:10b6:610:7f::9) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH2PR02MB6760:EE_|LV8PR02MB10096:EE_ X-MS-Office365-Filtering-Correlation-Id: 2f9d199d-f5e6-44e7-f656-08dd50f4cb8a x-proofpoint-crosstenant: true X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|366016|7416014|1800799024|376014; X-Microsoft-Antispam-Message-Info: AOBAh3nXUgKOhGtDNYMCpGBBKScJl0fCa4GLQzje5J0x6Q9nJXvEjr55j38Nh8+E8OmpXgS7BikUgJCZtxaGGMybRJGbngIu7BYn+gFkf+VHC05ga82wecwLqdI9vDYO1icsRS5F9G6TY9gcbuWp0mgqKb9cyFZaehYyyfC+l317FCIMedNYaqQaY4LG7xiywWEOPMAjHWSAN1CMbjbDqrzSa63y7118Hp6aTdkO7l1tFujzh5feMYvptgaxx0Tbm6cTlMBZNyrdOn7OjcxnOGllA09XAsPyvs31bKwEgvu0FqOzdG8R/LpFvX1VJpJJf6198cLc/1sWBLVZJHDqCPi89lWBYmU6Hv1xFx+kJtQfZBoiOS4iOwNw/2mf7yJ/JCLgnFtB86ZP3zU6kN3rd4zge7uAhfOaDHgQKIpA+p0R1zNYq7xVUnwSqeeTHz19S/QO/Qfby03A+a5DPPOdnvpZIOzdDCueNuHKg3UERlFrXEHgfdIaBkcHEbvfskQ35lmHK8vX0HqhXYgXWatPFc76FjTm+/k9NAUkJYDeriWldTsDQy4lmb8JPUgDM3690NKfEzkFxZ28xPXlUlmwvxxjZFJKwvcyyw/kpFqIdzGPRForVB4NTaongeLq7D2xnl94uPfU/hgX9ke5rgeGOrvBVCbl960HLmF0k0EnDerVG7seQgwGDDCdaGzi+cHKaL9zJxyGhbZs2DyqW+8h5LAyiEt2MASFuY61Sc/Icf3LfZaXHwW8H+UJN/Tma5hB6AHt8qNhsKTv1TkZYUiiw5bAaKyz9rRy6Qf3wSdidyGYNKxQhE4mUh23GyVaSk3fn+gUfnoq5pqwDVhMn7DRJyJtAGZvqD4KXDaNyJ8rbBzX8o65sUvxAhYvo6uW+gcaKvcTc+XWMSBcjTp6I4aDtaG4aw/gJC8jC+GefTYXxqVYe7aEm4rhGjsJNiVZv0HrYPPL4Mw/qy9uKYv4LDVUdayvkoyVqZoBPVu34wzaBMCPwZ2cq0RFlTMmM+Kwqu2bU8GZpnnrR8lfkEOUCMLsDkgvt2lSUIVm+NKavp4yxslrJ3X5Uq9eAGrD7Q4aCwDjHzWlJy+jLch5qyov80pesCuGE7q5FhTvzJDoVqOjSxMyWcanzqcj4bvKCDpbbgu1wYr9f0W8WSpyECrjYSroo4VpmLkmTvtwpsvnmGOfg7s0gW0K8+epM7Mh1pLkNL8noshHHQ5QzthzzlSglLEvmDn4kqxuWHrdm4S8buh5UFVssskYUHdAn2l3tkw7P48Jv97AfbfCqny4lZAH8f1ZkY+nALRcOPk7GGV3uCSZbNE5wMkC50qzbdR8p8cKFYxiQiCQbuI7Wt2ZHjQrDNG7mycFs9FAi7iuO7hDqVKbvwfdaY3gACneGshhrnLLa9rP X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:CH2PR02MB6760.namprd02.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230040)(366016)(7416014)(1800799024)(376014); DIR:OUT; SFP:1102; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QKjVhNqCY2lzv0Ci6YiKm6ALfad1p7hymQ0efXaVQUJVMUQI0ZLqirwtwYIC9yLVxhqLbm93bDIN+mNVWAOb41QQ1XVCRQaF22PEVTVDofVNDz6B7EqOXs09CYcA0PHNQYWul6eHys/+7kmuAuPifiLzNvZD0CP/wT8JgSTM2VXMVRdA47/5EK+PWc1cox9+zn8Sr5eEcTqxI++JlWpPLsdBy5AWY3ignbSKkBOFkeB3WOj9a53lQZNiXI3tkcW18KrRgBArK1wJyvxsLHQ05gkid580tvbGopgVV0l/FMk6rN7v8VcJbgQvGVo9eOIZtYNciO+jUdXzy85sWoG09X6LHomWmIOEBbiRg1r8UzK7zUYN3B4eOwzqPXU6rxmRCUm3EnfFK3+Cm7ffEn8rRA7SAa2NpLCimBrk9QyB6HRhO6tc+YSjTKb96KOCsDJ5Pcf2YBX8frWoRB3kmyFM0noQ1Sjue4xc5NVZPOf5XZ3Z6/3O96PBC02Eh8pmkG6fjU4YRr1ULXmlxFuMc5gEPJmpemDfS61F50uMAim+hvejiMWSYc4Q4dU5FJ32Se4GTt01AVQ3p5Z95W4EZl5V3gj5kGVEF5dctIlrtvRmR7Q8AmmqLrKN2p9+t1IIvuCsETv9UGiZZfJf/Bh1A+zIUlOizVk749sNFcRk0SMo0ETccdvq81c6uq/w61xgXwoCk9gCkw3UorYDkzwWrmOnCbyO6uxCIk9A5I2NPuPAcYrm5DWIDLr1uzo2G8NTklzU3K6QmFwJZJqgfu0amJB6h2QqJHrtwD0svIRkc5byS1sJD1t3iiBwtJIIkr6nqAfNWE2/5rCrJn45vNDBTcGHiA9tc5evzZjZE1GSv12XHX+Yvje1ctFVHngJfqWNvUfa0A5YkxgaMoXj8alai0z4Vh4PijLKSeTXjLvEZBZPBuvjpW4L8wilKy8AYBwIVSqUB5R+lNDSk2WBUmCdhD/fVIp3wOrVdPIQ1UBUDrKYZI6xzQmOJ0vLkYQuzXxGVGXKwaBgM9lH4ZU0eXyG/bd+A4amiUltCAn8gqIqfb/D3oR1Y4U6sv804Ih/cFXIiOfZsOlYgWI79uSHaOuq3hQSA2Y30lTqnnGfRP/cHGLNQbhbdAYOg5b1/Bt3WgstOW5VQmyyeiXrPRxCCnXjhacUXUZBe9SkViX1f2BA7Hq3EeuisN/PPuuA3OkkRwEMSJrOJ3VzRheqr8H3cbATg/KFT6zRaqdjE3aCHQeC4bRy0H5DgVEfOz9m2q6TV62ihitkV8A6dQsrGwXST37Htfkb2OW1c4vy1btcV4lN7T5NCq2sq1oD63n6Cdq8uZwllAzeR7fI4mv+5hhM9z+izwN6vkfjBjU96JdRDXDgf9T1+wDoHXyKlku4r6xdJbik3pZIIbI72FPMNlb761VSHXVEhy2c91Z8/p6DBfNQDVx09DHEj7O4f13lRiIZVxxECw3qn01yKE/IBHZdsWr3owJGwVgaES3rRjdjRpCJT9d+Gm7KemodvCWSqlHR0DA4mdTqvV0nvP7eXWkjApqWMaRYU6obDg1xkMLd3Ikj9sajaMhaoDQ9EkH33/W3nk5oi8+C X-OriginatorOrg: nutanix.com X-MS-Exchange-CrossTenant-Network-Message-Id: 2f9d199d-f5e6-44e7-f656-08dd50f4cb8a X-MS-Exchange-CrossTenant-AuthSource: CH2PR02MB6760.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Feb 2025 14:50:47.5216 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: bb047546-786f-4de1-bd75-24e5b6f79043 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: pvIawW2hYSrXjNfJkb9D7Tmlwm6q7nvfgbJvFdXHWeD1C0lKc7vgjTj5KwLbcAMCQadOKv+2pzJTUZrkz3ncpA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: LV8PR02MB10096 X-Proofpoint-GUID: RiRuC0fzvSg13kVYajf3ojKaZywH7RF6 X-Proofpoint-ORIG-GUID: RiRuC0fzvSg13kVYajf3ojKaZywH7RF6 X-Authority-Analysis: v=2.4 cv=d58PyQjE c=1 sm=1 tr=0 ts=67b5efcb cx=c_pps a=YkRwJB1Lcas13D3J5+BQ6w==:117 a=wKuvFiaSGQ0qltdbU6+NXLB8nM8=:19 a=Ol13hO9ccFRV9qXi2t6ftBPywas=:19 a=xqWC_Br6kY4A:10 a=T2h4t0Lz3GQA:10 a=0034W8JfsZAA:10 a=0kUYKlekyDsA:10 a=yPCof4ZbAAAA:8 a=64Cc0HZtAAAA:8 a=L0C8xICtdHRruC6hUNIA:9 a=14NRyaPF5x3gF6G45PvQ:22 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.293,Aquarius:18.0.1057,Hydra:6.0.680,FMLib:17.12.68.34 definitions=2025-02-19_06,2025-02-19_01,2024-11-22_01 X-Proofpoint-Spam-Reason: safe Received-SPF: pass client-ip=148.163.151.68; envelope-from=john.levon@nutanix.com; helo=mx0a-002c1b01.pphosted.com X-Spam_score_int: -29 X-Spam_score: -3.0 X-Spam_bar: --- X-Spam_report: (-3.0 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.191, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org From: Jagannathan Raman Add new message to send multiple writes to server in a single message. Prevents the outgoing queue from overflowing when a long latency operation is followed by a series of posted writes. Originally-by: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman Signed-off-by: John Levon --- hw/vfio-user/common.c | 131 +++++++++++++++++++++++++++++++++++++- hw/vfio-user/common.h | 7 ++ hw/vfio-user/protocol.h | 21 ++++++ hw/vfio-user/trace-events | 1 + 4 files changed, 158 insertions(+), 2 deletions(-) diff --git a/hw/vfio-user/common.c b/hw/vfio-user/common.c index e44c8a2568..809c8e6614 100644 --- a/hw/vfio-user/common.c +++ b/hw/vfio-user/common.c @@ -20,6 +20,7 @@ #include "io/channel-socket.h" #include "io/channel-util.h" #include "qapi/error.h" +#include "qobject/qbool.h" #include "qobject/qdict.h" #include "qobject/qjson.h" #include "qobject/qstring.h" @@ -55,6 +56,7 @@ static void vfio_user_request(void *opaque); static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg); static void vfio_user_send_async(VFIOUserProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds); +static void vfio_user_flush_multi(VFIOUserProxy *proxy); static inline void vfio_user_set_error(VFIOUserHdr *hdr, uint32_t err) { @@ -459,6 +461,11 @@ static void vfio_user_send(void *opaque) } qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, vfio_user_recv, NULL, NULL, proxy); + + /* queue empty - send any pending multi write msgs */ + if (proxy->wr_multi != NULL) { + vfio_user_flush_multi(proxy); + } } } @@ -479,6 +486,7 @@ static int vfio_user_send_one(VFIOUserProxy *proxy) } QTAILQ_REMOVE(&proxy->outgoing, msg, next); + proxy->num_outgoing--; if (msg->type == VFIO_MSG_ASYNC) { vfio_user_recycle(proxy, msg); } else { @@ -586,11 +594,18 @@ static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg) { int ret; + /* older coalesced writes go first */ + if (proxy->wr_multi != NULL && + ((msg->hdr->flags & VFIO_USER_TYPE) == VFIO_USER_REQUEST)) { + vfio_user_flush_multi(proxy); + } + /* * Unsent outgoing msgs - add to tail */ if (!QTAILQ_EMPTY(&proxy->outgoing)) { QTAILQ_INSERT_TAIL(&proxy->outgoing, msg, next); + proxy->num_outgoing++; return 0; } @@ -604,6 +619,7 @@ static int vfio_user_send_queued(VFIOUserProxy *proxy, VFIOUserMsg *msg) } if (ret == QIO_CHANNEL_ERR_BLOCK) { QTAILQ_INSERT_HEAD(&proxy->outgoing, msg, next); + proxy->num_outgoing = 1; qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, vfio_user_recv, proxy->ctx, vfio_user_send, proxy); @@ -1119,12 +1135,27 @@ static bool check_migr(VFIOUserProxy *proxy, QObject *qobj, Error **errp) return caps_parse(proxy, qdict, caps_migr, errp); } +static bool check_multi(VFIOUserProxy *proxy, QObject *qobj, Error **errp) +{ + QBool *qb = qobject_to(QBool, qobj); + + if (qb == NULL) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MULTI); + return false; + } + if (qbool_get_bool(qb)) { + proxy->flags |= VFIO_PROXY_USE_MULTI; + } + return true; +} + static struct cap_entry caps_cap[] = { { VFIO_USER_CAP_MAX_FDS, check_max_fds }, { VFIO_USER_CAP_MAX_XFER, check_max_xfer }, { VFIO_USER_CAP_PGSIZES, check_pgsizes }, { VFIO_USER_CAP_MAP_MAX, check_max_dma }, { VFIO_USER_CAP_MIGR, check_migr }, + { VFIO_USER_CAP_MULTI, check_multi }, { NULL } }; @@ -1183,6 +1214,7 @@ static GString *caps_json(void) qdict_put_int(capdict, VFIO_USER_CAP_MAX_XFER, VFIO_USER_DEF_MAX_XFER); qdict_put_int(capdict, VFIO_USER_CAP_PGSIZES, VFIO_USER_DEF_PGSIZE); qdict_put_int(capdict, VFIO_USER_CAP_MAP_MAX, VFIO_USER_DEF_MAP_MAX); + qdict_put_bool(capdict, VFIO_USER_CAP_MULTI, true); qdict_put_obj(dict, VFIO_USER_CAP, QOBJECT(capdict)); @@ -1451,19 +1483,114 @@ static int vfio_user_region_read(VFIOUserProxy *proxy, uint8_t index, return msgp->count; } +static void vfio_user_flush_multi(VFIOUserProxy *proxy) +{ + VFIOUserMsg *msg; + VFIOUserWRMulti *wm = proxy->wr_multi; + int ret; + + proxy->wr_multi = NULL; + + /* adjust size for actual # of writes */ + wm->hdr.size -= (VFIO_USER_MULTI_MAX - wm->wr_cnt) * sizeof(VFIOUserWROne); + + msg = vfio_user_getmsg(proxy, &wm->hdr, NULL); + msg->id = wm->hdr.id; + msg->rsize = 0; + msg->type = VFIO_MSG_ASYNC; + trace_vfio_user_wrmulti("flush", wm->wr_cnt); + + ret = vfio_user_send_queued(proxy, msg); + if (ret < 0) { + vfio_user_recycle(proxy, msg); + } +} + +static void vfio_user_create_multi(VFIOUserProxy *proxy) +{ + VFIOUserWRMulti *wm; + + wm = g_malloc0(sizeof(*wm)); + vfio_user_request_msg(&wm->hdr, VFIO_USER_REGION_WRITE_MULTI, + sizeof(*wm), VFIO_USER_NO_REPLY); + proxy->wr_multi = wm; +} + +static void vfio_user_add_multi(VFIOUserProxy *proxy, uint8_t index, + off_t offset, uint32_t count, void *data) +{ + VFIOUserWRMulti *wm = proxy->wr_multi; + VFIOUserWROne *w1 = &wm->wrs[wm->wr_cnt]; + + w1->offset = offset; + w1->region = index; + w1->count = count; + memcpy(&w1->data, data, count); + + wm->wr_cnt++; + trace_vfio_user_wrmulti("add", wm->wr_cnt); + if (wm->wr_cnt == VFIO_USER_MULTI_MAX || + proxy->num_outgoing < VFIO_USER_OUT_LOW) { + vfio_user_flush_multi(proxy); + } +} + static int vfio_user_region_write(VFIOUserProxy *proxy, uint8_t index, off_t offset, uint32_t count, void *data, bool post) { VFIOUserRegionRW *msgp = NULL; - int flags = post ? VFIO_USER_NO_REPLY : 0; + int flags; int size = sizeof(*msgp) + count; + bool can_multi; int ret; if (count > proxy->max_xfer_size) { return -EINVAL; } + if (proxy->flags & VFIO_PROXY_NO_POST) { + post = false; + } + + /* write eligible to be in a WRITE_MULTI msg ? */ + can_multi = (proxy->flags & VFIO_PROXY_USE_MULTI) && post && + count <= VFIO_USER_MULTI_DATA; + + /* + * This should be a rare case, so first check without the lock, + * if we're wrong, vfio_send_queued() will flush any posted writes + * we missed here + */ + if (proxy->wr_multi != NULL || + (proxy->num_outgoing > VFIO_USER_OUT_HIGH && can_multi)) { + + /* + * re-check with lock + * + * if already building a WRITE_MULTI msg, + * add this one if possible else flush pending before + * sending the current one + * + * else if outgoing queue is over the highwater, + * start a new WRITE_MULTI message + */ + WITH_QEMU_LOCK_GUARD(&proxy->lock) { + if (proxy->wr_multi != NULL) { + if (can_multi) { + vfio_user_add_multi(proxy, index, offset, count, data); + return count; + } + vfio_user_flush_multi(proxy); + } else if (proxy->num_outgoing > VFIO_USER_OUT_HIGH && can_multi) { + vfio_user_create_multi(proxy); + vfio_user_add_multi(proxy, index, offset, count, data); + return count; + } + } + } + + flags = post ? VFIO_USER_NO_REPLY : 0; msgp = g_malloc0(size); vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size, flags); msgp->offset = offset; @@ -1473,7 +1600,7 @@ static int vfio_user_region_write(VFIOUserProxy *proxy, uint8_t index, trace_vfio_user_region_rw(msgp->region, msgp->offset, msgp->count); /* async send will free msg after it's sent */ - if (post && !(proxy->flags & VFIO_PROXY_NO_POST)) { + if (post) { vfio_user_send_async(proxy, &msgp->hdr, NULL); return count; } diff --git a/hw/vfio-user/common.h b/hw/vfio-user/common.h index 9acf634ca7..96efc32bd7 100644 --- a/hw/vfio-user/common.h +++ b/hw/vfio-user/common.h @@ -80,6 +80,8 @@ typedef struct VFIOUserProxy { VFIOUserMsg *last_nowait; VFIOUserMsg *part_recv; size_t recv_left; + VFIOUserWRMulti *wr_multi; + int num_outgoing; enum proxy_state state; } VFIOUserProxy; @@ -88,6 +90,11 @@ typedef struct VFIOUserProxy { #define VFIO_PROXY_NO_MMAP 0x2 #define VFIO_PROXY_FORCE_QUEUED 0x4 #define VFIO_PROXY_NO_POST 0x8 +#define VFIO_PROXY_USE_MULTI 0x10 + +/* coalescing high and low water marks for VFIOProxy num_outgoing */ +#define VFIO_USER_OUT_HIGH 1024 +#define VFIO_USER_OUT_LOW 128 typedef struct VFIODevice VFIODevice; diff --git a/hw/vfio-user/protocol.h b/hw/vfio-user/protocol.h index 6bc5809cce..98cd0e1920 100644 --- a/hw/vfio-user/protocol.h +++ b/hw/vfio-user/protocol.h @@ -40,6 +40,7 @@ enum vfio_user_command { VFIO_USER_DMA_WRITE = 12, VFIO_USER_DEVICE_RESET = 13, VFIO_USER_DIRTY_PAGES = 14, + VFIO_USER_REGION_WRITE_MULTI = 15, VFIO_USER_MAX, }; @@ -73,6 +74,7 @@ typedef struct { #define VFIO_USER_CAP_PGSIZES "pgsizes" #define VFIO_USER_CAP_MAP_MAX "max_dma_maps" #define VFIO_USER_CAP_MIGR "migration" +#define VFIO_USER_CAP_MULTI "write_multiple" /* "migration" members */ #define VFIO_USER_CAP_PGSIZE "pgsize" @@ -219,4 +221,23 @@ typedef struct { char data[]; } VFIOUserBitmap; +/* + * VFIO_USER_REGION_WRITE_MULTI + */ +#define VFIO_USER_MULTI_DATA 8 +#define VFIO_USER_MULTI_MAX 200 + +typedef struct { + uint64_t offset; + uint32_t region; + uint32_t count; + char data[VFIO_USER_MULTI_DATA]; +} VFIOUserWROne; + +typedef struct { + VFIOUserHdr hdr; + uint64_t wr_cnt; + VFIOUserWROne wrs[VFIO_USER_MULTI_MAX]; +} VFIOUserWRMulti; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio-user/trace-events b/hw/vfio-user/trace-events index 7ef98813b3..64fac9137f 100644 --- a/hw/vfio-user/trace-events +++ b/hw/vfio-user/trace-events @@ -11,6 +11,7 @@ vfio_user_get_region_info(uint32_t index, uint32_t flags, uint64_t size) " index vfio_user_region_rw(uint32_t region, uint64_t off, uint32_t count) " region %d offset 0x%"PRIx64" count %d" vfio_user_get_irq_info(uint32_t index, uint32_t flags, uint32_t count) " index %d flags 0x%x count %d" vfio_user_set_irqs(uint32_t index, uint32_t start, uint32_t count, uint32_t flags) " index %d start %d count %d flags 0x%x" +vfio_user_wrmulti(const char *s, uint64_t wr_cnt) " %s count 0x%"PRIx64 # container.c vfio_user_dma_map(uint64_t iova, uint64_t size, uint64_t off, uint32_t flags, bool async_ops) " iova 0x%"PRIx64" size 0x%"PRIx64" off 0x%"PRIx64" flags 0x%x async_ops %d"