From patchwork Thu May 5 17:19:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 652F9C433F5 for ; Thu, 5 May 2022 17:15:49 +0000 (UTC) Received: from localhost ([::1]:38746 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmf4u-0001kh-2z for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:15:48 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52646) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0m-00063T-J0 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:44378) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0e-0002BM-Of for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:30 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245GRcVn019339 for ; Thu, 5 May 2022 17:11:20 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=bC8xB9P3KntV3snsDA8xANSEDcUUEDnSOp/26V7b94s=; b=GBSyWXP98fR8JRL7pxfV5kGmG/6RaGazNcWoYNXEJwCwBhss79AV7rJw9P+P1v5gIyWo RMz5V63eWvEEEgeSUfLSQwGnwh21eF9ZapAk6jRGe01v+VQjbaoDOOBELT7gW0agYhRX 4MqSRLGqculh3lFz0bHd6TWOP2OJnck6XKSCTOjhFNYQZdf4Ly/czeg+Nw4oaybXce78 hVK3LcuuXSvYqcYTOh+5hzqkAZGe/z8vh5jDnafIqb84rQnD/ckHkEYngIQjccCJX2M1 3I0II2c+GUVmZgVYQ3eYVyoXjQrN+8ANnflfLtMmR42GY06HhjbcvTotFdKRFFUYx4jh fg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frwntc3kc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:19 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0sqN013487 for ; Thu, 5 May 2022 17:11:19 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fruj4xq8q-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SI3ht45gnh1hXJRQDifza2xWa5GC3hjm4pgZRJFk1k+5ryc9f/YhDUmAYui8fQ9vSJpBTEGf7s0CylpQyeXQ6e1U8q/O1rqap4YtNVsaGSRhHH8729zDj6Lost8Y88iIrZLOLubdZxurqtpjts7zulWvWeWB4/Vc9cr2udP9+6cZx9YJdWIPVOjyybQfvvID4PJhxMRJWRevwTsuAZvVmfrabjfzmTgtQybUN4Es8btxBFjM4wiG3jf8/OnigCMm1QTcqLxPn2VR+BBeyAEnJwteQ6oCGz9AFFUD8T6TGIWuNzqXnQv3IMdqxjrbXJI2Xk4gYqvc6Z2OkiMDF80p0A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=bC8xB9P3KntV3snsDA8xANSEDcUUEDnSOp/26V7b94s=; b=HvWMeA8r8c5RBi0gUW5FjOap08SGQlqSQqrKXWg82LGs4ASgNoFOUuuHjzQRdLf7oPmoKl6H28IBtnvemGAOKzQpOZC6MiZpKBidftSDaXJCzUlM6szdjbwUr2gWX/+jgQ0HYGwJwdi/fzkTNBAnulYnX0nGiwNWh1TMgu0Wj+Ofzpr7z0mU+Q8IFVNTN4JPk0ozgulGQflJsmKQZ9Q7x+tOnZenEOIFykHA3nARdXVWuAItlaiydF8lR6KjGxKIeMRIEhtIo+Vx3p47OCzulUHaX8WFWBRaj01syZyrDMkzNmHrMjbxeqnR/QXbCIkY7ZGay+g/Y+dRsbYvODc8fg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bC8xB9P3KntV3snsDA8xANSEDcUUEDnSOp/26V7b94s=; b=OssSjxp6Gf+Kf3+AHHoZTPLSwUxboU4rzY71ynV/vq3CxhlpYnfq/p2NS0NENOjpbMsunwEGWftjQsqbyhl7xjDrNWfwEiajGi+xYfYW9t8K20nI5fM02qsZZb7Hn09ZR1C24InVWPkZIeKjGPGvLfKr2kmGMClvVcugZRZrzKY= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:15 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:15 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 01/23] vfio-user: introduce vfio-user protocol specification Date: Thu, 5 May 2022 10:19:44 -0700 Message-Id: <2ac62cd4c9fb6e85aca222aef670d288acc2d3e5.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1bce12f9-aae8-45ea-d627-08da2eba4350 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pv53PwTbJXQ8n4TU4aSJlAQ8oE1DDB0pqTdlNakq/pVrmlBAdjJUdtb9bov+w5WKQX1kxA19p0rA6tFG7lb/kQstsZY491MTGjQbLK5lj5J5KKtlCA6R1db8t7kIBZT9BcHLNfLbl4nhor+jIxKg/GQ8gLyS+/m77De/KLV2Hjri7WS289DUYSe+IU/fURmprDEtjtAC3XR0BhGfDo8xXPfchj0RtPxlZeMAQz9nfdPpkrWeFpab+GkebnDhb1e7fNcB6fp7J3dt7n8O69liDclPoCkX4QrWRCnDoMLjqKV2LlSd8IJ4UERJrmn5gONVKSj90Sfr7CckyFVldkjAxM9m4HKPHHMqd8ss5a+/GQrDjSXMlSyJ9jw0I47vHbTlju/NFp5fKFobBbTvtasNLvEUm1kqj44BlaBhedHdxyjUcfR3CFac0aH1Roi7ng9PM/18eZSPfZ/X7kgIAg1Epv6+H4zw2cjDdJ/JsrTgS8hVI1n4WrFXbLivfh8l9QbawE4fr/zGe7iVh5L/BFWqpK4soglBPYb47K+MyX4jVXNnzgwSvcHHE347G8swy+tTRKqmmiMnpTX8/0HTsMes/qNUhwWR0VusT2KFhYyfAWvzdkJ0Ud6KshuyCPGZN/5k232BJs5Lfrzk5IWSIeVqCi6SX3R3teDFFgiO7Snd0rEVAsspl7ncX7428G9h+uoLNJJyMJjfMsR8XNjmuKQSH1qv87NrsmAJ/aYaVlPiQe9Ti88gplKLt+Sxuq/LRT0MKSzl3PpCImJj2tazGRaQVInx7/9cqEG7xjVfGT7mjZ+IgOoDSANsZAfq9yEFlCY2NHmGI6+A3qqpM3YiAgt1Kg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(966005)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(30864003)(2616005)(2906002)(26005)(6512007)(6916009)(559001)(579004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: vHiut/ccKY+rgnEmBx9GGw1sMtAV1GWilCtneQf5bzMBws8zBNtOgzxsu7dEw3j6gYkatrBgkd9GL3FGePfWAKVEe5/NY+ETTbaLh5gb4B42wTaBU1toKHruEfIxvhebq6kkUE7FvTBSGdlVzxKC3gvnOH/kQvGVtCryvXxzKPFS7+rKk0NPYm7zwY6vxO+OARUH5tZK35q7gZeyTm8Sfjd1uIx00t6ZU2I5yabvUkB+phr61Q/22+Q5kpa851HLkm5vZwYZiXfp95VLt3L64PgrHnvyuItsPkWCbWVJGmSU/OBGNTarTytGi+hff/pX5vZPDcyaeY9DzpIIeWzvTs5x34Z7W32duni0gifaptwaDNkCjC13LRi4K74cet/9ifcJ7ZlrNgMAE5vhXPibEupVXPFIUlcuaLzp+9LoTvBKKveYIECS9sm0w9QPSzORIjhTdaGu8VNayzEdRYVT4NsThbPZSIvDCueGcZLsSblWpet1u9onMDfxG32goDeLJpfZYFX5hPEMwSwwvrCBlPc2kzxyvJSnY9OFQLMz82lSdUK95C3DrO3UvYE6iksYVoMLyhcKq4SRJDAWdqcNTA1LPjWW06P14O0PbzvInMq0unuciYyofRVhAar59L5s/RJSL8Q2Po9sxYGBbpC4t2qcqcdTXYF3GRXyab5DkVykjJlmLZObtLSeXR54BbCjsH1IIQuTT1/67Ta2y0JU/28kGrV0PFeXnVZTwUriHotDHap1c4mNFIft6bdifVTT6luy8JYF563nPwsOgzdW2IwjSjsSU9LvFUDk4nLxxU2Xfseeplz7CVbU15I7jZ5uNAFwmDZ4fzFoscFa4eMfeDBKYxpsEUDmPNDQCbS5njwH+c9ckoEKY6LkZmyRyeFOE8QiFHZrrLg012zdQVgRyusiKjLj+oyePo0+yoK4kmHgLE+e6Vca1vGxI9Cw1rm3NKuJ3Iwx5zau6yWfmPJHpzc0NQtnPZ3ZHzu6JSzk9pgZLBEBPRUpWB6fRvawqEHUH6G6QegfDyTrgIP2vSmpb7zBRvLDjNPHsB/puDg634XwnCgGK1/s3almmBqBt3vLEjF1fQKP5tSPS5lRU4SfmvB/zN9PGEB9ofvivW4wcT3BrdLfi0gCYlYI6dvMEUqZBKTnWHXWzMLrfrHXkMuEBm0BzlbVxjxfjMhmMF6iVD068OhHevj/eW+YJj+6Lh6RMY8F0BvLvRzXujXL8DftFgJh4K/s5XxXyQWNS8BA8v7IYP6M6Icf8ThWhCo9L2whV858sqrWuBL2waz77XsM/gtPXdDXK1AeL+xpyehHnBi0Pb74MbTm5ZyaxfR/m0rAWO1tmlBNTl26EBd9l30ap7vZAZvGdWhIdePgeidJsJaQc1zd3grojqPCrl1FDbGXfIet2OoSz+KjQ9xVD2ReBl0wc3jqpsLCOGgoGfMZjZSWonwzz0pNFVs6FxBqDZxWLNoDZsGokHu7JmVP0juZGOGxtbJ9P+T7RJtX1MyKMkBYOB1jwi9dipni684bsSJT2VbFmVsNy5ugVrySvq0rAQxRQH7c0u2q6A2okVMxXQYcicuXPChx6JMzJD8Q5RJeI1CZ33nzLUfsRtpMZXpZFx6UuwyZOEMH9RytHFW9TY2ut7wukpTy7WXFPyt2LLMFcUSG5tNhbmBaj0G/JBQfUFf4Kku54/VuzAxDHAmzWUrLGQOLf2/8+3q2LDR1sYqMwE88FlAoehmo0J6W0N+k+wnqzzSSpYqGmKf4n7zZfjM= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1bce12f9-aae8-45ea-d627-08da2eba4350 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:15.6387 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: KfgYtut/AgBQl0bAMxw9/o7uHPhPpQuCIkBYwSrB6t8weh/YJreX/rGYeZYIzSrBqs31QV5r2pFE0cZZVJs3Xjed8tVzeAZwb7Xw4idY/60= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: Y6uzrfoGY9uoq6ifcJYeWqADzmLrjjrt X-Proofpoint-GUID: Y6uzrfoGY9uoq6ifcJYeWqADzmLrjjrt Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01, WEIRD_QUOTING=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Thanos Makatos This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts. It has been earlier discussed as an RFC in: "RFC: use VFIO over a UNIX domain socket to implement device offloading" Signed-off-by: John G Johnson Signed-off-by: Thanos Makatos Signed-off-by: John Levon --- docs/devel/index-internals.rst | 1 + docs/devel/vfio-user.rst | 1453 ++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 6 + 3 files changed, 1460 insertions(+) create mode 100644 docs/devel/vfio-user.rst diff --git a/docs/devel/index-internals.rst b/docs/devel/index-internals.rst index a50889c..c71a79e 100644 --- a/docs/devel/index-internals.rst +++ b/docs/devel/index-internals.rst @@ -17,4 +17,5 @@ Details about QEMU's various subsystems including how to add features to them. s390-dasd-ipl tracing vfio-migration + vfio-user writing-monitor-commands diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst new file mode 100644 index 0000000..0fb1e33 --- /dev/null +++ b/docs/devel/vfio-user.rst @@ -0,0 +1,1453 @@ +.. include:: +******************************** +vfio-user Protocol Specification +******************************** + +-------------- +Version_ 0.9.1 +-------------- + +.. contents:: Table of Contents + +Introduction +============ +vfio-user is a protocol that allows a device to be emulated in a separate +process outside of a Virtual Machine Monitor (VMM). vfio-user devices consist +of a generic VFIO device type, living inside the VMM, which we call the client, +and the core device implementation, living outside the VMM, which we call the +server. + +The vfio-user specification is partly based on the +`Linux VFIO ioctl interface `_. + +VFIO is a mature and stable API, backed by an extensively used framework. The +existing VFIO client implementation in QEMU (``qemu/hw/vfio/``) can be largely +re-used, though there is nothing in this specification that requires that +particular implementation. None of the VFIO kernel modules are required for +supporting the protocol, on either the client or server side. Some source +definitions in VFIO are re-used for vfio-user. + +The main idea is to allow a virtual device to function in a separate process in +the same host over a UNIX domain socket. A UNIX domain socket (``AF_UNIX``) is +chosen because file descriptors can be trivially sent over it, which in turn +allows: + +* Sharing of client memory for DMA with the server. +* Sharing of server memory with the client for fast MMIO. +* Efficient sharing of eventfd's for triggering interrupts. + +Other socket types could be used which allow the server to run in a separate +guest in the same host (``AF_VSOCK``) or remotely (``AF_INET``). Theoretically +the underlying transport does not necessarily have to be a socket, however we do +not examine such alternatives. In this protocol version we focus on using a UNIX +domain socket and introduce basic support for the other two types of sockets +without considering performance implications. + +While passing of file descriptors is desirable for performance reasons, support +is not necessary for either the client or the server in order to implement the +protocol. There is always an in-band, message-passing fall back mechanism. + +Overview +======== + +VFIO is a framework that allows a physical device to be securely passed through +to a user space process; the device-specific kernel driver does not drive the +device at all. Typically, the user space process is a VMM and the device is +passed through to it in order to achieve high performance. VFIO provides an API +and the required functionality in the kernel. QEMU has adopted VFIO to allow a +guest to directly access physical devices, instead of emulating them in +software. + +vfio-user reuses the core VFIO concepts defined in its API, but implements them +as messages to be sent over a socket. It does not change the kernel-based VFIO +in any way, in fact none of the VFIO kernel modules need to be loaded to use +vfio-user. It is also possible for the client to concurrently use the current +kernel-based VFIO for one device, and vfio-user for another device. + +VFIO Device Model +----------------- + +A device under VFIO presents a standard interface to the user process. Many of +the VFIO operations in the existing interface use the ``ioctl()`` system call, and +references to the existing interface are called the ``ioctl()`` implementation in +this document. + +The following sections describe the set of messages that implement the vfio-user +interface over a socket. In many cases, the messages are analogous to data +structures used in the ``ioctl()`` implementation. Messages derived from the +``ioctl()`` will have a name derived from the ``ioctl()`` command name. E.g., the +``VFIO_DEVICE_GET_INFO`` ``ioctl()`` command becomes a +``VFIO_USER_DEVICE_GET_INFO`` message. The purpose of this reuse is to share as +much code as feasible with the ``ioctl()`` implementation``. + +Connection Initiation +^^^^^^^^^^^^^^^^^^^^^ + +After the client connects to the server, the initial client message is +``VFIO_USER_VERSION`` to propose a protocol version and set of capabilities to +apply to the session. The server replies with a compatible version and set of +capabilities it supports, or closes the connection if it cannot support the +advertised version. + +Device Information +^^^^^^^^^^^^^^^^^^ + +The client uses a ``VFIO_USER_DEVICE_GET_INFO`` message to query the server for +information about the device. This information includes: + +* The device type and whether it supports reset (``VFIO_DEVICE_FLAGS_``), +* the number of device regions, and +* the device presents to the client the number of interrupt types the device + supports. + +Region Information +^^^^^^^^^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_REGION_INFO`` messages to query the +server for information about the device's regions. This information describes: + +* Read and write permissions, whether it can be memory mapped, and whether it + supports additional capabilities (``VFIO_REGION_INFO_CAP_``). +* Region index, size, and offset. + +When a device region can be mapped by the client, the server provides a file +descriptor which the client can ``mmap()``. The server is responsible for +polling for client updates to memory mapped regions. + +Region Capabilities +""""""""""""""""""" + +Some regions have additional capabilities that cannot be described adequately +by the region info data structure. These capabilities are returned in the +region info reply in a list similar to PCI capabilities in a PCI device's +configuration space. + +Sparse Regions +"""""""""""""" +A region can be memory-mappable in whole or in part. When only a subset of a +region can be mapped by the client, a ``VFIO_REGION_INFO_CAP_SPARSE_MMAP`` +capability is included in the region info reply. This capability describes +which portions can be mapped by the client. + +.. Note:: + For example, in a virtual NVMe controller, sparse regions can be used so + that accesses to the NVMe registers (found in the beginning of BAR0) are + trapped (an infrequent event), while allowing direct access to the doorbells + (an extremely frequent event as every I/O submission requires a write to + BAR0), found in the next page after the NVMe registers in BAR0. + +Device-Specific Regions +""""""""""""""""""""""" + +A device can define regions additional to the standard ones (e.g. PCI indexes +0-8). This is achieved by including a ``VFIO_REGION_INFO_CAP_TYPE`` capability +in the region info reply of a device-specific region. Such regions are reflected +in ``struct vfio_user_device_info.num_regions``. Thus, for PCI devices this +value can be equal to, or higher than, ``VFIO_PCI_NUM_REGIONS``. + +Region I/O via file descriptors +------------------------------- + +For unmapped regions, region I/O from the client is done via +``VFIO_USER_REGION_READ/WRITE``. As an optimization, ioeventfds or ioregionfds +may be configured for sub-regions of some regions. A client may request +information on these sub-regions via ``VFIO_USER_DEVICE_GET_REGION_IO_FDS``; by +configuring the returned file descriptors as ioeventfds or ioregionfds, the +server can be directly notified of I/O (for example, by KVM) without taking a +trip through the client. + +Interrupts +^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_IRQ_INFO`` messages to query the server +for the device's interrupt types. The interrupt types are specific to the bus +the device is attached to, and the client is expected to know the capabilities +of each interrupt type. The server can signal an interrupt by directly injecting +interrupts into the guest via an event file descriptor. The client configures +how the server signals an interrupt with ``VFIO_USER_SET_IRQS`` messages. + +Device Read and Write +^^^^^^^^^^^^^^^^^^^^^ + +When the guest executes load or store operations to an unmapped device region, +the client forwards these operations to the server with +``VFIO_USER_REGION_READ`` or ``VFIO_USER_REGION_WRITE`` messages. The server +will reply with data from the device on read operations or an acknowledgement on +write operations. See `Read and Write Operations`_. + +Client memory access +-------------------- + +The client uses ``VFIO_USER_DMA_MAP`` and ``VFIO_USER_DMA_UNMAP`` messages to +inform the server of the valid DMA ranges that the server can access on behalf +of a device (typically, VM guest memory). DMA memory may be accessed by the +server via ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages over the +socket. In this case, the "DMA" part of the naming is a misnomer. + +Actual direct memory access of client memory from the server is possible if the +client provides file descriptors the server can ``mmap()``. Note that ``mmap()`` +privileges cannot be revoked by the client, therefore file descriptors should +only be exported in environments where the client trusts the server not to +corrupt guest memory. + +See `Read and Write Operations`_. + +Client/server interactions +========================== + +Socket +------ + +A server can serve: + +1) one or more clients, and/or +2) one or more virtual devices, belonging to one or more clients. + +The current protocol specification requires a dedicated socket per +client/server connection. It is a server-side implementation detail whether a +single server handles multiple virtual devices from the same or multiple +clients. The location of the socket is implementation-specific. Multiplexing +clients, devices, and servers over the same socket is not supported in this +version of the protocol. + +Authentication +-------------- + +For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files, +therefore it is up to the management layer to set up the socket as required. +Socket types that span guests or hosts will require a proper authentication +mechanism. Defining that mechanism is deferred to a future version of the +protocol. + +Command Concurrency +------------------- + +A client may pipeline multiple commands without waiting for previous command +replies. The server will process commands in the order they are received. A +consequence of this is if a client issues a command with the *No_reply* bit, +then subsequently issues a command without *No_reply*, the older command will +have been processed before the reply to the younger command is sent by the +server. The client must be aware of the device's capability to process +concurrent commands if pipelining is used. For example, pipelining allows +multiple client threads to concurrently access device regions; the client must +ensure these accesses obey device semantics. + +An example is a frame buffer device, where the device may allow concurrent +access to different areas of video memory, but may have indeterminate behavior +if concurrent accesses are performed to command or status registers. + +Note that unrelated messages sent from the server to the client can appear in +between a client to server request/reply and vice versa. + +Implementers should be prepared for certain commands to exhibit potentially +unbounded latencies. For example, ``VFIO_USER_DEVICE_RESET`` may take an +arbitrarily long time to complete; clients should take care not to block +unnecessarily. + +Socket Disconnection Behavior +----------------------------- +The server and the client can disconnect from each other, either intentionally +or unexpectedly. Both the client and the server need to know how to handle such +events. + +Server Disconnection +^^^^^^^^^^^^^^^^^^^^ +A server disconnecting from the client may indicate that: + +1) A virtual device has been restarted, either intentionally (e.g. because of a + device update) or unintentionally (e.g. because of a crash). +2) A virtual device has been shut down with no intention to be restarted. + +It is impossible for the client to know whether or not a failure is +intermittent or innocuous and should be retried, therefore the client should +reset the VFIO device when it detects the socket has been disconnected. +Error recovery will be driven by the guest's device error handling +behavior. + +Client Disconnection +^^^^^^^^^^^^^^^^^^^^ +The client disconnecting from the server primarily means that the client +has exited. Currently, this means that the guest is shut down so the device is +no longer needed therefore the server can automatically exit. However, there +can be cases where a client disconnection should not result in a server exit: + +1) A single server serving multiple clients. +2) A multi-process QEMU upgrading itself step by step, which is not yet + implemented. + +Therefore in order for the protocol to be forward compatible, the server should +respond to a client disconnection as follows: + + - all client memory regions are unmapped and cleaned up (including closing any + passed file descriptors) + - all IRQ file descriptors passed from the old client are closed + - the device state should otherwise be retained + +The expectation is that when a client reconnects, it will re-establish IRQ and +client memory mappings. + +If anything happens to the client (such as qemu really did exit), the control +stack will know about it and can clean up resources accordingly. + +Security Considerations +----------------------- + +Speaking generally, vfio-user clients should not trust servers, and vice versa. +Standard tools and mechanisms should be used on both sides to validate input and +prevent against denial of service scenarios, buffer overflow, etc. + +Request Retry and Response Timeout +---------------------------------- +A failed command is a command that has been successfully sent and has been +responded to with an error code. Failure to send the command in the first place +(e.g. because the socket is disconnected) is a different type of error examined +earlier in the disconnect section. + +.. Note:: + QEMU's VFIO retries certain operations if they fail. While this makes sense + for real HW, we don't know for sure whether it makes sense for virtual + devices. + +Defining a retry and timeout scheme is deferred to a future version of the +protocol. + +Message sizes +------------- + +Some requests have an ``argsz`` field. In a request, it defines the maximum +expected reply payload size, which should be at least the size of the fixed +reply payload headers defined here. The *request* payload size is defined by the +usual ``msg_size`` field in the header, not the ``argsz`` field. + +In a reply, the server sets ``argsz`` field to the size needed for a full +payload size. This may be less than the requested maximum size. This may be +larger than the requested maximum size: in that case, the full payload is not +included in the reply, but the ``argsz`` field in the reply indicates the needed +size, allowing a client to allocate a larger buffer for holding the reply before +trying again. + +In addition, during negotiation (see `Version`_), the client and server may +each specify a ``max_data_xfer_size`` value; this defines the maximum data that +may be read or written via one of the ``VFIO_USER_DMA/REGION_READ/WRITE`` +messages; see `Read and Write Operations`_. + +Protocol Specification +====================== + +To distinguish from the base VFIO symbols, all vfio-user symbols are prefixed +with ``vfio_user`` or ``VFIO_USER``. In this revision, all data is in the +endianness of the host system, although this may be relaxed in future +revisions in cases where the client and server run on different hosts +with different endianness. + +Unless otherwise specified, all sizes should be presumed to be in bytes. + +.. _Commands: + +Commands +-------- +The following table lists the VFIO message command IDs, and whether the +message command is sent from the client or the server. + +====================================== ========= ================= +Name Command Request Direction +====================================== ========= ================= +``VFIO_USER_VERSION`` 1 client -> server +``VFIO_USER_DMA_MAP`` 2 client -> server +``VFIO_USER_DMA_UNMAP`` 3 client -> server +``VFIO_USER_DEVICE_GET_INFO`` 4 client -> server +``VFIO_USER_DEVICE_GET_REGION_INFO`` 5 client -> server +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` 6 client -> server +``VFIO_USER_DEVICE_GET_IRQ_INFO`` 7 client -> server +``VFIO_USER_DEVICE_SET_IRQS`` 8 client -> server +``VFIO_USER_REGION_READ`` 9 client -> server +``VFIO_USER_REGION_WRITE`` 10 client -> server +``VFIO_USER_DMA_READ`` 11 server -> client +``VFIO_USER_DMA_WRITE`` 12 server -> client +``VFIO_USER_DEVICE_RESET`` 13 client -> server +====================================== ========= ================= + +Header +------ + +All messages, both command messages and reply messages, are preceded by a +16-byte header that contains basic information about the message. The header is +followed by message-specific data described in the sections below. + ++----------------+--------+-------------+ +| Name | Offset | Size | ++================+========+=============+ +| Message ID | 0 | 2 | ++----------------+--------+-------------+ +| Command | 2 | 2 | ++----------------+--------+-------------+ +| Message size | 4 | 4 | ++----------------+--------+-------------+ +| Flags | 8 | 4 | ++----------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0-3 | Type | | +| | +-----+------------+ | +| | | 4 | No_reply | | +| | +-----+------------+ | +| | | 5 | Error | | +| | +-----+------------+ | ++----------------+--------+-------------+ +| Error | 12 | 4 | ++----------------+--------+-------------+ +| | 16 | variable | ++----------------+--------+-------------+ + +* *Message ID* identifies the message, and is echoed in the command's reply + message. Message IDs belong entirely to the sender, can be re-used (even + concurrently) and the receiver must not make any assumptions about their + uniqueness. +* *Command* specifies the command to be executed, listed in Commands_. It is + also set in the reply header. +* *Message size* contains the size of the entire message, including the header. +* *Flags* contains attributes of the message: + + * The *Type* bits indicate the message type. + + * *Command* (value 0x0) indicates a command message. + * *Reply* (value 0x1) indicates a reply message acknowledging a previous + command with the same message ID. + * *No_reply* in a command message indicates that no reply is needed for this + command. This is commonly used when multiple commands are sent, and only + the last needs acknowledgement. + * *Error* in a reply message indicates the command being acknowledged had + an error. In this case, the *Error* field will be valid. + +* *Error* in a reply message is an optional UNIX errno value. It may be zero + even if the Error bit is set in Flags. It is reserved in a command message. + +Each command message in Commands_ must be replied to with a reply message, +unless the message sets the *No_Reply* bit. The reply consists of the header +with the *Reply* bit set, plus any additional data. + +If an error occurs, the reply message must only include the reply header. + +As the header is standard in both requests and replies, it is not included in +the command-specific specifications below; each message definition should be +appended to the standard header, and the offsets are given from the end of the +standard header. + +``VFIO_USER_VERSION`` +--------------------- + +.. _Version: + +This is the initial message sent by the client after the socket connection is +established; the same format is used for the server's reply. + +Upon establishing a connection, the client must send a ``VFIO_USER_VERSION`` +message proposing a protocol version and a set of capabilities. The server +compares these with the versions and capabilities it supports and sends a +``VFIO_USER_VERSION`` reply according to the following rules. + +* The major version in the reply must be the same as proposed. If the client + does not support the proposed major, it closes the connection. +* The minor version in the reply must be equal to or less than the minor + version proposed. +* The capability list must be a subset of those proposed. If the server + requires a capability the client did not include, it closes the connection. + +The protocol major version will only change when incompatible protocol changes +are made, such as changing the message format. The minor version may change +when compatible changes are made, such as adding new messages or capabilities, +Both the client and server must support all minor versions less than the +maximum minor version it supports. E.g., an implementation that supports +version 1.3 must also support 1.0 through 1.2. + +When making a change to this specification, the protocol version number must +be included in the form "added in version X.Y" + +Request +^^^^^^^ + +============== ====== ==== +Name Offset Size +============== ====== ==== +version major 0 2 +version minor 2 2 +version data 4 variable (including terminating NUL). Optional. +============== ====== ==== + +The version data is an optional UTF-8 encoded JSON byte array with the following +format: + ++--------------+--------+-----------------------------------+ +| Name | Type | Description | ++==============+========+===================================+ +| capabilities | object | Contains common capabilities that | +| | | the sender supports. Optional. | ++--------------+--------+-----------------------------------+ + +Capabilities: + ++--------------------+--------+------------------------------------------------+ +| Name | Type | Description | ++====================+========+================================================+ +| max_msg_fds | number | Maximum number of file descriptors that can be | +| | | received by the sender in one message. | +| | | Optional. If not specified then the receiver | +| | | must assume a value of ``1``. | ++--------------------+--------+------------------------------------------------+ +| max_data_xfer_size | number | Maximum ``count`` for data transfer messages; | +| | | see `Read and Write Operations`_. Optional, | +| | | with a default value of 1048576 bytes. | ++--------------------+--------+------------------------------------------------+ +| migration | object | Migration capability parameters. If missing | +| | | then migration is not supported by the sender. | ++--------------------+--------+------------------------------------------------+ + +The migration capability contains the following name/value pairs: + ++--------+--------+-----------------------------------------------+ +| Name | Type | Description | ++========+========+===============================================+ +| pgsize | number | Page size of dirty pages bitmap. The smallest | +| | | between the client and the server is used. | ++--------+--------+-----------------------------------------------+ + +Reply +^^^^^ + +The same message format is used in the server's reply with the semantics +described above. + +``VFIO_USER_DMA_MAP`` +--------------------- + +This command message is sent by the client to the server to inform it of the +memory regions the server can access. It must be sent before the server can +perform any DMA to the client. It is normally sent directly after the version +handshake is completed, but may also occur when memory is added to the client, +or if the client uses a vIOMMU. + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++-------------+--------+-------------+ +| Name | Offset | Size | ++=============+========+=============+ +| argsz | 0 | 4 | ++-------------+--------+-------------+ +| flags | 4 | 4 | ++-------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0 | readable | | +| | +-----+------------+ | +| | | 1 | writeable | | +| | +-----+------------+ | ++-------------+--------+-------------+ +| offset | 8 | 8 | ++-------------+--------+-------------+ +| address | 16 | 8 | ++-------------+--------+-------------+ +| size | 24 | 8 | ++-------------+--------+-------------+ + +* *argsz* is the size of the above structure. Note there is no reply payload, + so this field differs from other message types. +* *flags* contains the following region attributes: + + * *readable* indicates that the region can be read from. + + * *writeable* indicates that the region can be written to. + +* *offset* is the file offset of the region with respect to the associated file + descriptor, or zero if the region is not mappable +* *address* is the base DMA address of the region. +* *size* is the size of the region. + +This structure is 32 bytes in size, so the message size is 16 + 32 bytes. + +If the DMA region being added can be directly mapped by the server, a file +descriptor must be sent as part of the message meta-data. The region can be +mapped via the mmap() system call. On ``AF_UNIX`` sockets, the file descriptor +must be passed as ``SCM_RIGHTS`` type ancillary data. Otherwise, if the DMA +region cannot be directly mapped by the server, no file descriptor must be sent +as part of the message meta-data and the DMA region can be accessed by the +server using ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages, +explained in `Read and Write Operations`_. A command to map over an existing +region must be failed by the server with ``EEXIST`` set in error field in the +reply. + +Reply +^^^^^ + +There is no payload in the reply message. + +``VFIO_USER_DMA_UNMAP`` +----------------------- + +This command message is sent by the client to the server to inform it that a +DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command +message, is no longer available for DMA. It typically occurs when memory is +subtracted from the client or if the client uses a vIOMMU. The DMA region is +described by the following structure: + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++--------------+--------+------------------------+ +| Name | Offset | Size | ++==============+========+========================+ +| argsz | 0 | 4 | ++--------------+--------+------------------------+ +| flags | 4 | 4 | ++--------------+--------+------------------------+ +| address | 8 | 8 | ++--------------+--------+------------------------+ +| size | 16 | 8 | ++--------------+--------+------------------------+ + +* *argsz* is the maximum size of the reply payload. +* *flags* is unused in this version. +* *address* is the base DMA address of the DMA region. +* *size* is the size of the DMA region. + +The address and size of the DMA region being unmapped must match exactly a +previous mapping. + +Reply +^^^^^ + +Upon receiving a ``VFIO_USER_DMA_UNMAP`` command, if the file descriptor is +mapped then the server must release all references to that DMA region before +replying, which potentially includes in-flight DMA transactions. + +The server responds with the original DMA entry in the request. + + +``VFIO_USER_DEVICE_GET_INFO`` +----------------------------- + +This command message is sent by the client to the server to query for basic +information about the device. + +Request +^^^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the maximum size of the reply payload +* all other fields must be zero. + +Reply +^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* contains the following device attributes. + + * ``VFIO_DEVICE_FLAGS_RESET`` indicates that the device supports the + ``VFIO_USER_DEVICE_RESET`` message. + * ``VFIO_DEVICE_FLAGS_PCI`` indicates that the device is a PCI device. + +* *num_regions* is the number of memory regions that the device exposes. +* *num_irqs* is the number of distinct interrupt types that the device supports. + +This version of the protocol only supports PCI devices. Additional devices may +be supported in future versions. + +``VFIO_USER_DEVICE_GET_REGION_INFO`` +------------------------------------ + +This command message is sent by the client to the server to query for +information about device regions. The VFIO region info structure is defined in +```` (``struct vfio_region_info``). + +Request +^^^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* all other fields must be zero. + +Reply +^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_REGION_INFO_FLAG_READ | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_REGION_INFO_FLAG_WRITE | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_REGION_INFO_FLAG_MMAP | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_REGION_INFO_FLAG_CAPS | | +| | +-----+-----------------------------+ | ++------------+--------+------------------------------+ ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* is the size required for the full reply payload (region info structure + plus the size of any region capabilities) +* *flags* are attributes of the region: + + * ``VFIO_REGION_INFO_FLAG_READ`` allows client read access to the region. + * ``VFIO_REGION_INFO_FLAG_WRITE`` allows client write access to the region. + * ``VFIO_REGION_INFO_FLAG_MMAP`` specifies the client can mmap() the region. + When this flag is set, the reply will include a file descriptor in its + meta-data. On ``AF_UNIX`` sockets, the file descriptors will be passed as + ``SCM_RIGHTS`` type ancillary data. + * ``VFIO_REGION_INFO_FLAG_CAPS`` indicates additional capabilities found in the + reply. + +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* *cap_offset* describes where additional region capabilities can be found. + cap_offset is relative to the beginning of the VFIO region info structure. + The data structure it points is a VFIO cap header defined in + ````. +* *size* is the size of the region. +* *offset* is the offset that should be given to the mmap() system call for + regions with the MMAP attribute. It is also used as the base offset when + mapping a VFIO sparse mmap area, described below. + +VFIO region capabilities +"""""""""""""""""""""""" + +The VFIO region information can also include a capabilities list. This list is +similar to a PCI capability list - each entry has a common header that +identifies a capability and where the next capability in the list can be found. +The VFIO capability header format is defined in ```` (``struct +vfio_info_cap_header``). + +VFIO cap header format +"""""""""""""""""""""" + ++---------+--------+------+ +| Name | Offset | Size | ++=========+========+======+ +| id | 0 | 2 | ++---------+--------+------+ +| version | 2 | 2 | ++---------+--------+------+ +| next | 4 | 4 | ++---------+--------+------+ + +* *id* is the capability identity. +* *version* is a capability-specific version number. +* *next* specifies the offset of the next capability in the capability list. It + is relative to the beginning of the VFIO region info structure. + +VFIO sparse mmap cap header +""""""""""""""""""""""""""" + ++------------------+----------------------------------+ +| Name | Value | ++==================+==================================+ +| id | VFIO_REGION_INFO_CAP_SPARSE_MMAP | ++------------------+----------------------------------+ +| version | 0x1 | ++------------------+----------------------------------+ +| next | | ++------------------+----------------------------------+ +| sparse mmap info | VFIO region info sparse mmap | ++------------------+----------------------------------+ + +This capability is defined when only a subrange of the region supports +direct access by the client via mmap(). The VFIO sparse mmap area is defined in +```` (``struct vfio_region_sparse_mmap_area`` and ``struct +vfio_region_info_cap_sparse_mmap``). + +VFIO region info cap sparse mmap +"""""""""""""""""""""""""""""""" + ++----------+--------+------+ +| Name | Offset | Size | ++==========+========+======+ +| nr_areas | 0 | 4 | ++----------+--------+------+ +| reserved | 4 | 4 | ++----------+--------+------+ +| offset | 8 | 8 | ++----------+--------+------+ +| size | 16 | 8 | ++----------+--------+------+ +| ... | | | ++----------+--------+------+ + +* *nr_areas* is the number of sparse mmap areas in the region. +* *offset* and size describe a single area that can be mapped by the client. + There will be *nr_areas* pairs of offset and size. The offset will be added to + the base offset given in the ``VFIO_USER_DEVICE_GET_REGION_INFO`` to form the + offset argument of the subsequent mmap() call. + +The VFIO sparse mmap area is defined in ```` (``struct +vfio_region_info_cap_sparse_mmap``). + + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` +-------------------------------------- + +Clients can access regions via ``VFIO_USER_REGION_READ/WRITE`` or, if provided, by +``mmap()`` of a file descriptor provided by the server. + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` provides an alternative access mechanism via +file descriptors. This is an optional feature intended for performance +improvements where an underlying sub-system (such as KVM) supports communication +across such file descriptors to the vfio-user server, without needing to +round-trip through the client. + +The server returns an array of sub-regions for the requested region. Each +sub-region describes a span (offset and size) of a region, along with the +requested file descriptor notification mechanism to use. Each sub-region in the +response message may choose to use a different method, as defined below. The +two mechanisms supported in this specification are ioeventfds and ioregionfds. + +The server in addition returns a file descriptor in the ancillary data; clients +are expected to configure each sub-region's file descriptor with the requested +notification method. For example, a client could configure KVM with the +requested ioeventfd via a ``KVM_IOEVENTFD`` ``ioctl()``. + +Request +^^^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried +* all other fields must be zero + +The client must set ``flags`` to zero and specify the region being queried in +the ``index``. + +Reply +^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ +| sub-regions | 16 | ... | ++-------------+--------+------+ + +* *argsz* is the size of the region IO FD info structure plus the + total size of the sub-region array. Thus, each array entry "i" is at offset + i * ((argsz - 32) / count). Note that currently this is 40 bytes for both IO + FD types, but this is not to be relied on. As elsewhere, this indicates the + full reply payload size needed. +* *flags* must be zero +* *index* is the index of memory region being queried +* *count* is the number of sub-regions in the array +* *sub-regions* is the array of Sub-Region IO FD info structures + +The reply message will additionally include at least one file descriptor in the +ancillary data. Note that more than one sub-region may share the same file +descriptor. + +Note that it is the client's responsibility to verify the requested values (for +example, that the requested offset does not exceed the region's bounds). + +Each sub-region given in the response has one of two possible structures, +depending whether *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` or +``VFIO_USER_IO_FD_TYPE_IOREGIONFD``: + +Sub-Region IO FD info format (ioeventfd) +"""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| datamatch | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations +* *fd_index* is the index in the ancillary data of the FD to use for ioeventfd + notification; it may be shared. +* *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` +* *flags* is any of: + + * ``KVM_IOEVENTFD_FLAG_DATAMATCH`` + * ``KVM_IOEVENTFD_FLAG_PIO`` + * ``KVM_IOEVENTFD_FLAG_VIRTIO_CCW_NOTIFY`` (FIXME: makes sense?) + +* *datamatch* is the datamatch value if needed + +See https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt, *4.59 +KVM_IOEVENTFD* for further context on the ioeventfd-specific fields. + +Sub-Region IO FD info format (ioregionfd) +""""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| user_data | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations; ``KVM_IOREGION_POSTED_WRITES`` + must be set in *flags* in this case +* *fd_index* is the index in the ancillary data of the FD to use for ioregionfd + messages; it may be shared +* *type* is ``VFIO_USER_IO_FD_TYPE_IOREGIONFD`` +* *flags* is any of: + + * ``KVM_IOREGION_PIO`` + * ``KVM_IOREGION_POSTED_WRITES`` + +* *user_data* is an opaque value passed back to the server via a message on the + file descriptor + +For further information on the ioregionfd-specific fields, see: +https://lore.kernel.org/kvm/cover.1613828726.git.eafanasova@gmail.com/ + +(FIXME: update with final API docs.) + +``VFIO_USER_DEVICE_GET_IRQ_INFO`` +--------------------------------- + +This command message is sent by the client to the server to query for +information about device interrupt types. The VFIO IRQ info structure is +defined in ```` (``struct vfio_irq_info``). + +Request +^^^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the maximum size of the reply payload (16 bytes today) +* index is the index of IRQ type being queried (e.g. ``VFIO_PCI_MSIX_IRQ_INDEX``) +* all other fields must be zero + +Reply +^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* defines IRQ attributes: + + * ``VFIO_IRQ_INFO_EVENTFD`` indicates the IRQ type can support server eventfd + signalling. + * ``VFIO_IRQ_INFO_MASKABLE`` indicates that the IRQ type supports the ``MASK`` + and ``UNMASK`` actions in a ``VFIO_USER_DEVICE_SET_IRQS`` message. + * ``VFIO_IRQ_INFO_AUTOMASKED`` indicates the IRQ type masks itself after being + triggered, and the client must send an ``UNMASK`` action to receive new + interrupts. + * ``VFIO_IRQ_INFO_NORESIZE`` indicates ``VFIO_USER_SET_IRQS`` operations setup + interrupts as a set, and new sub-indexes cannot be enabled without disabling + the entire type. +* index is the index of IRQ type being queried +* count describes the number of interrupts of the queried type. + +``VFIO_USER_DEVICE_SET_IRQS`` +----------------------------- + +This command message is sent by the client to the server to set actions for +device interrupt types. The VFIO IRQ set structure is defined in +```` (``struct vfio_irq_set``). + +Request +^^^^^^^ + ++-------+--------+------------------------------+ +| Name | Offset | Size | ++=======+========+==============================+ +| argsz | 0 | 4 | ++-------+--------+------------------------------+ +| flags | 4 | 4 | ++-------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_IRQ_SET_DATA_NONE | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_IRQ_SET_DATA_BOOL | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_IRQ_SET_DATA_EVENTFD | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_IRQ_SET_ACTION_MASK | | +| | +-----+-----------------------------+ | +| | | 4 | VFIO_IRQ_SET_ACTION_UNMASK | | +| | +-----+-----------------------------+ | +| | | 5 | VFIO_IRQ_SET_ACTION_TRIGGER | | +| | +-----+-----------------------------+ | ++-------+--------+------------------------------+ +| index | 8 | 4 | ++-------+--------+------------------------------+ +| start | 12 | 4 | ++-------+--------+------------------------------+ +| count | 16 | 4 | ++-------+--------+------------------------------+ +| data | 20 | variable | ++-------+--------+------------------------------+ + +* *argsz* is the size of the VFIO IRQ set request payload, including any *data* + field. Note there is no reply payload, so this field differs from other + message types. +* *flags* defines the action performed on the interrupt range. The ``DATA`` + flags describe the data field sent in the message; the ``ACTION`` flags + describe the action to be performed. The flags are mutually exclusive for + both sets. + + * ``VFIO_IRQ_SET_DATA_NONE`` indicates there is no data field in the command. + The action is performed unconditionally. + * ``VFIO_IRQ_SET_DATA_BOOL`` indicates the data field is an array of boolean + bytes. The action is performed if the corresponding boolean is true. + * ``VFIO_IRQ_SET_DATA_EVENTFD`` indicates an array of event file descriptors + was sent in the message meta-data. These descriptors will be signalled when + the action defined by the action flags occurs. In ``AF_UNIX`` sockets, the + descriptors are sent as ``SCM_RIGHTS`` type ancillary data. + If no file descriptors are provided, this de-assigns the specified + previously configured interrupts. + * ``VFIO_IRQ_SET_ACTION_MASK`` indicates a masking event. It can be used with + ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to mask an interrupt, + or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the guest masks + the interrupt. + * ``VFIO_IRQ_SET_ACTION_UNMASK`` indicates an unmasking event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to unmask an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + guest unmasks the interrupt. + * ``VFIO_IRQ_SET_ACTION_TRIGGER`` indicates a triggering event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to trigger an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + server triggers the interrupt. + +* *index* is the index of IRQ type being setup. +* *start* is the start of the sub-index being set. +* *count* describes the number of sub-indexes being set. As a special case, a + count (and start) of 0, with data flags of ``VFIO_IRQ_SET_DATA_NONE`` disables + all interrupts of the index. +* *data* is an optional field included when the + ``VFIO_IRQ_SET_DATA_BOOL`` flag is present. It contains an array of booleans + that specify whether the action is to be performed on the corresponding + index. It's used when the action is only performed on a subset of the range + specified. + +Not all interrupt types support every combination of data and action flags. +The client must know the capabilities of the device and IRQ index before it +sends a ``VFIO_USER_DEVICE_SET_IRQ`` message. + +In typical operation, a specific IRQ may operate as follows: + +1. The client sends a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_TRIGGER)`` along + with an eventfd. This associates the IRQ with a particular eventfd on the + server side. + +#. The client may send a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_MASK/UNMASK)`` along + with another eventfd. This associates the given eventfd with the + mask/unmask state on the server side. + +#. The server may trigger the IRQ by writing 1 to the eventfd. + +#. The server may mask/unmask an IRQ which will write 1 to the corresponding + mask/unmask eventfd, if there is one. + +5. A client may trigger a device IRQ itself, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_TRIGGER)``. + +6. A client may mask or unmask the IRQ, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_MASK/UNMASK)``. + +Reply +^^^^^ + +There is no payload in the reply. + +.. _Read and Write Operations: + +Note that all of these operations must be supported by the client and/or server, +even if the corresponding memory or device region has been shared as mappable. + +The ``count`` field must not exceed the value of ``max_data_xfer_size`` of the +peer, for both reads and writes. + +``VFIO_USER_REGION_READ`` +------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via ``mmap()`` of the underlying file descriptor. In this case, a client can +read from a device region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. +* *data* is the data that was read from the device region. + +``VFIO_USER_REGION_WRITE`` +-------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via mmap() of the underlying fd. In this case, a client can write to a device +region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DMA_READ`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +read from guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. +* *data* is the data read. + +``VFIO_USER_DMA_WRITE`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +write to guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 4 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DEVICE_RESET`` +-------------------------- + +This command message is sent from the client to the server to reset the device. +Neither the request or reply have a payload. + + +Appendices +========== + +Unused VFIO ``ioctl()`` commands +-------------------------------- + +The following VFIO commands do not have an equivalent vfio-user command: + +* ``VFIO_GET_API_VERSION`` +* ``VFIO_CHECK_EXTENSION`` +* ``VFIO_SET_IOMMU`` +* ``VFIO_GROUP_GET_STATUS`` +* ``VFIO_GROUP_SET_CONTAINER`` +* ``VFIO_GROUP_UNSET_CONTAINER`` +* ``VFIO_GROUP_GET_DEVICE_FD`` +* ``VFIO_IOMMU_GET_INFO`` + +However, once support for live migration for VFIO devices is finalized some +of the above commands may have to be handled by the client in their +corresponding vfio-user form. This will be addressed in a future protocol +version. + +VFIO groups and containers +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The current VFIO implementation includes group and container idioms that +describe how a device relates to the host IOMMU. In the vfio-user +implementation, the IOMMU is implemented in SW by the client, and is not +visible to the server. The simplest idea would be that the client put each +device into its own group and container. + +Backend Program Conventions +--------------------------- + +vfio-user backend program conventions are based on the vhost-user ones. + +* The backend program must not daemonize itself. +* No assumptions must be made as to what access the backend program has on the + system. +* File descriptors 0, 1 and 2 must exist, must have regular + stdin/stdout/stderr semantics, and can be redirected. +* The backend program must honor the SIGTERM signal. +* The backend program must accept the following commands line options: + + * ``--socket-path=PATH``: path to UNIX domain socket, + * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible with + ``--socket-path`` +* The backend program must be accompanied with a JSON file stored under + ``/usr/share/vfio-user``. + +TODO add schema similar to docs/interop/vhost-user.json. diff --git a/MAINTAINERS b/MAINTAINERS index 294c88a..8117241 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1955,6 +1955,12 @@ F: hw/vfio/ap.c F: docs/system/s390x/vfio-ap.rst L: qemu-s390x@nongnu.org +vfio-user +M: John G Johnson +M: Thanos Makatos +S: Supported +F: docs/devel/vfio-user.rst + vhost M: Michael S. Tsirkin S: Supported From patchwork Thu May 5 17:19:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839856 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94EB2C433FE for ; Thu, 5 May 2022 17:50:01 +0000 (UTC) Received: from localhost ([::1]:34720 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfbz-0004qK-So for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:50:00 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52828) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0z-0006B8-VM for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:45 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:41964) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0s-0002BC-JT for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:45 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FTMUW030007 for ; Thu, 5 May 2022 17:11:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=WJrXH9w4WXzMwxBe2udP8+wGYiE0ysr7klSvd1YyjGg=; b=dG5ov3DylixNer4tOQJsJeUwU3IPn+6e1yKl1NVjijNRgr+Xz7rZmDkcoX8NjejoaUXn 6taxI6KDsjXGBOQZ9S8OOJQwnjxmk3ZwJ9CpvXge0kg48JMntycWWBccY5b8kW3oX5ae Z4EqCDmd36Zr+g8R5Hk55VSgp4K9cdQ/ryqjPF+2jy+MYWJTD2FJez8p5l8RwOBdhNyz OQB7IAo9Le7tDC0d9Aktzbv6ILormKUCnxrDD38cCs/fDs1GJtRzHemE4su0a6ioAO7B h2W9Oqx4IttrBWY5Io0Z5v9JMJ0WDzzejOoumfGsTDOp4/4P9zCnuTIotqu/uVPl3YNR Tg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0m36w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:18 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0sqK013487 for ; Thu, 5 May 2022 17:11:18 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fruj4xq8q-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EX0WM9+LeNsWUkS3ceT+yaiYYVq2vjh+40TpKsPKtoaZ5uZnPAt88lH8cbfsdsZKa4zMgbhFNEzevtzGZGlaioEASFA32IR6QaMrLd9oa6s9EuhPTRm+TtdlbM9IIO477ogy/Z3+egdvCnflHFthstxggk37sqbwcLyzFNU7oJnUC4TfdrmPBwzuVhndUv/2f11/CZqbqThfnvitP+R01PaFXh1c+6MDOKWy+TxyyYp2z6UYMkzyHO5h0sqHDRyL/PTIU4svfWSKrhPA136qcxhIMZ2ZDwzvxwiYi+jeKEcGZJLhfFhJ7/YNp9vs6/usnmJj2TxuqUc77sSwgREcJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WJrXH9w4WXzMwxBe2udP8+wGYiE0ysr7klSvd1YyjGg=; b=jKQjkYGMgdARwyFy4yhAjf7iDj9t1YYQudmeLFKuCkOmFqRuekH/exr8lNHxZI4PasD2AkN9WYP3e+1XHFFEvZtNaqnaaQEjEHneIafi/3/M9ZDbdOvJ6tQz5m3b7Zu4/iJp5flF99KLMZ4tj+xrbwSICwk3I21Bon3D6GzgQK8M3GssgTyCdc4H1YCDdEiEazUsurvjTjDQTshaFrknGWXdjB5z2+Xz2LZmipZDqP1bSy2oi/h+Fu5D+Zs0cZqEdg31cX7BjatPLyiVctsO0vjT97/d/wzXw5yz4KGsMASpBwDPkEhVKNjCb34oGRw7+o0BNo92JkcDxXaWb+3r+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WJrXH9w4WXzMwxBe2udP8+wGYiE0ysr7klSvd1YyjGg=; b=ztfcr19FFaanLFlZqWiV62avD55seMU32UjgoGjnxbcCw5jnJIqP4n5SqX+Or4MBIj+iHxFkYsN1M+gNYZbGD+L3wXWgM3ULkfdtfWBeHtJJzbYRv7K011JofOB9lGifZL77itaPkCyqglKL7NciXregTGtQfaIrv5oSFvtY3Ck= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:16 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:16 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 02/23] vfio-user: add VFIO base abstract class Date: Thu, 5 May 2022 10:19:45 -0700 Message-Id: <16e96a8cef34aba538588d8a4b2949523622b3d3.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 13a2a28d-8d46-41d9-0da5-08da2eba438a X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ar7WMpji79w2uA9yNPBRdY6XQVEVvwr2pfecUPhXf5HWTYkcrZWCTY2yi7WsZPc6CtfXM8aWb9bdJObPD3eKhVmeBGA/FgyEIDXQQs1wU0iY1o54C+dkmGAQ/LYhVXCbrWLpIDzVAxj3RBj6AJnc4C673cA4OWsj2pHjt5MhQnC3sW3RD79kNlI4jTKB5jd5OGX0zBGalETB+ow0GSljCbq+OflvB1IY61n1FVieES9ezUjJtkpDgKdVYSWmMTd2x+3yMAv0clpG7R2f3y/TmiY7PVRoo/BtYgRZO9XWcOY136+skzR5Ex7+WXn5bXoyo9mjrpesOLEhDQ9ZNHlmoyswTz2av91msvScIMMvPpGianSCR4EB5v4YxBiFgkhpXf5BheXHFhemZ1Q5yXCCOaSFVnH2CIWdm8DMHA8vviZjoqR05Ng4Tf5MvkQxabXi1/4kssnyP8gePDD1j562BsHxVEOlt6+3IqFhXB09N8bO5EutnkxSms+k/WO/DeUZGyHWW0QmMNSKJGhOvnMhssiZz5ZnU0AyC/7UUtImgkWpJCQ8d+0zIhxvvbSn0X4+567+8og7defwpxM/XL22NI5od2VZKzClEQIdVVnVfDRjJ+ufhx52ZR5PXc4Ku2TY/UAA+9ofQUJX62YtHOwmLbhnmxKgwrO34HhbfpL9bVl0VW9K5w8VbmxhR+mGGdIOfLbnkVBrm51HpXmZHVBxhA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(30864003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Mbos/j91dpuYZ6vejLFjQAKbxbNSKyEbEvFXjvtGlEo9eBJ5motlBPrO2iwbsyJLj7YkR2/N/cWjMWFWBsEQY0hfUN6sI50XICttt6j/UzAyNuqidp5xOzUs6iWQLO9FZXRhbYdgBAWGH/QMwIZd5XwaGr7xwiassJdn88ZKOgWIkrZWFWwnfe3Kn7mL4DRhfjrDNtqTr2VLRL+GhDnV7YcD8YaL+Y0bd36BiO8tm4DSe2hTJD2xwcP/Rx8yMvb8x/tUGrTgQt5n2bGRKNqJTsySzeXhZzRLa+zpHvbxDjCaH7J4NZGo/0Y/82+iB8de9KXpy0klFdUz87gy/r0suT1gwWdumxxGMqMrJCE96BQwIqIfFxgPNzY7YLuta4oslyc+so6IoYCiTQdPTr1NfTsOZnrdGwZyaVI6DsAS+PtPDVf8x1z2wmuIZrJHwd2m4MSoJeXqCNAwFZbMyUF7KnsqKKbLN3Cq6LA4DUu1XpdaihHSGqZr/iI4VSsUbYKZ9X1UDrKNlsf4hWZdwIXnxMfhNfQ1bpd5iy7eDMUdvWgQE6dhAyuHE8iOq1YPDjAUInQI2BVjkFF7/lX5XwEyk8NtwMd6CYu3Vg/n2K7tLTriqryR2r3SlBWmgB8gInzAqC0E6ypk4sL+5INUDl30XKWxIAv+/Bf8gY0B4/heZa95u55av4wxoPnKnV/Aam7yYMw8qLzdJqAOUJGvfic8ukdvHO3BxOY8BcD2146YCXG8TDxvu/CLuxuNhJZv2RGxuLQJrdyile8Jzm0JgZ6BMDX1LgNfCTx+4YvLufr9IpvICR0mB+THuJLPvvoJkID4OsZbquLy2PXL9ZSn/RMDsn6ykK0+v8HjCDLEywqr5IStgh5bn9zbRdNJ6ZP0Nk35LJoAfXlfygBc20MbKAp72xdp+bw2c8uGn9dc+9nt7nXf68Nse1NKSjYHrJGSh4kWg3QqXspw0VpAh9DZmfFJzlQTNNqp1IKng5woV9LnEXpImP61ktU+ze+wfGS+BImRIIrDGSaQOdzmKhLExV1wN7LXZJPMiUhzLYiyeAHUCwA00MPhqxRrWY/kZE56Au0eVDdZmDUM9PDZ9pzU4A5KLNQzSqWwXzmLbKg2JIZWVZ7z+V4dBKWpY/pbokOcsYs/yd/bOTtnVjRlFp/T/rs/hf0fMELiWWFqzr0v6Cci3ljWdq+Gf13Fno3DDyr3fmrNy2QvT4orpD+MthmxVsSp3tiiaNdDqrYcaYzW/9MoWGaJXibaSAk90tck2cJQAjdR6p5n38jBZsHghw5YjxCpdfE9BozhmIyD+0cVj7YPv/Q/oYZPMPaBidVMICw73h+Da5Lx8ep+RrUKab/J+/9P53HScPwredd+X57o1hv81GopR5oGyNh66mrlSmpJaRgevBqBuv9xMQrV6mH/ASU8y+viQo8bOfwoRl2G/yDFdtBGgWkmKlc8WOzRN8tgH7Ui53Fy7V/QU09L8c+9JXqxEq1BsqZcN1IrFFMWfg//G8GAKFccwVbyT/RNDemwE8aRLriDKWFoW2lb33x3/+ErBwpmyBpLo5HuCcUfzCcXpGMRGg7so9om43r4zmWAtahEysiUZQgHEf3cSZ0qa+gyMpuxWB1JI97AxgJsRkMzL2rTd8bJReS0K6ePSFt0SffMx1RdPzaInBxZNgNCLzAhWSOQ7EgqRdhkeP5LwlYEaBemBwWl4cwmlpMsUuj3XnOXlwsoo92Bh8VoVjJSlDoxaTEU+PXo8YKhQiN6lt/LBOE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 13a2a28d-8d46-41d9-0da5-08da2eba438a X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:15.9668 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uY73RMBMmpUDyCPnTet4QgLpLEBY26f9dh94jQcUROv61EiAq+T+kRwUel2wo2yuY69RZCfF5/dmEsj9Kwz/DvGa+5Bz5P7LmmgKW0aCyak= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: uoQCkJ_aMrzi_GY8DjbzTr1Tc3NNnSBx X-Proofpoint-GUID: uoQCkJ_aMrzi_GY8DjbzTr1Tc3NNnSBx Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add an abstract base class both the kernel driver and user socket implementations can use to share code. Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 16 +++++++-- hw/vfio/pci.c | 106 +++++++++++++++++++++++++++++++++++----------------------- 2 files changed, 78 insertions(+), 44 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 6477751..bbc78aa 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -114,8 +114,13 @@ typedef struct VFIOMSIXInfo { unsigned long *pending; } VFIOMSIXInfo; -#define TYPE_VFIO_PCI "vfio-pci" -OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI) +/* + * TYPE_VFIO_PCI_BASE is an abstract type used to share code + * between VFIO implementations that use a kernel driver + * with those that use user sockets. + */ +#define TYPE_VFIO_PCI_BASE "vfio-pci-base" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_BASE) struct VFIOPCIDevice { PCIDevice pdev; @@ -175,6 +180,13 @@ struct VFIOPCIDevice { Notifier irqchip_change_notifier; }; +#define TYPE_VFIO_PCI "vfio-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOKernPCIDevice, VFIO_PCI) + +struct VFIOKernPCIDevice { + VFIOPCIDevice device; +}; + /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 9fd9fae..4ee5215 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -231,7 +231,7 @@ static void vfio_intx_update(VFIOPCIDevice *vdev, PCIINTxRoute *route) static void vfio_intx_routing_notifier(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); PCIINTxRoute route; if (vdev->interrupt != VFIO_INT_INTx) { @@ -460,7 +460,7 @@ static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, MSIMessage *msg, IOHandler *handler) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector; int ret; @@ -545,7 +545,7 @@ static int vfio_msix_vector_use(PCIDevice *pdev, static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector = &vdev->msi_vectors[nr]; trace_vfio_msix_vector_release(vdev->vbasedev.name, nr); @@ -1066,7 +1066,7 @@ static const MemoryRegionOps vfio_vga_ops = { */ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIORegion *region = &vdev->bars[bar].region; MemoryRegion *mmap_mr, *region_mr, *base_mr; PCIIORegion *r; @@ -1112,7 +1112,7 @@ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) */ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t emu_bits = 0, emu_val = 0, phys_val = 0, val; memcpy(&emu_bits, vdev->emulated_config_bits + addr, len); @@ -1145,7 +1145,7 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t val_le = cpu_to_le32(val); trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); @@ -2802,7 +2802,7 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) static void vfio_realize(PCIDevice *pdev, Error **errp) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev_iter; VFIOGroup *group; char *tmp, *subsys, group_path[PATH_MAX], *group_name; @@ -3125,7 +3125,7 @@ error: static void vfio_instance_finalize(Object *obj) { - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIOGroup *group = vdev->vbasedev.group; vfio_display_finalize(vdev); @@ -3145,7 +3145,7 @@ static void vfio_instance_finalize(Object *obj) static void vfio_exitfn(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); @@ -3164,7 +3164,7 @@ static void vfio_exitfn(PCIDevice *pdev) static void vfio_pci_reset(DeviceState *dev) { - VFIOPCIDevice *vdev = VFIO_PCI(dev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); trace_vfio_pci_reset(vdev->vbasedev.name); @@ -3204,7 +3204,7 @@ post_reset: static void vfio_instance_init(Object *obj) { PCIDevice *pci_dev = PCI_DEVICE(obj); - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); device_add_bootindex_property(obj, &vdev->bootindex, "bootindex", NULL, @@ -3221,24 +3221,12 @@ static void vfio_instance_init(Object *obj) pci_dev->cap_present |= QEMU_PCI_CAP_EXPRESS; } -static Property vfio_pci_dev_properties[] = { - DEFINE_PROP_PCI_HOST_DEVADDR("host", VFIOPCIDevice, host), - DEFINE_PROP_STRING("sysfsdev", VFIOPCIDevice, vbasedev.sysfsdev), +static Property vfio_pci_base_dev_properties[] = { DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevice, vbasedev.pre_copy_dirty_page_tracking, ON_OFF_AUTO_ON), - DEFINE_PROP_ON_OFF_AUTO("display", VFIOPCIDevice, - display, ON_OFF_AUTO_OFF), - DEFINE_PROP_UINT32("xres", VFIOPCIDevice, display_xres, 0), - DEFINE_PROP_UINT32("yres", VFIOPCIDevice, display_yres, 0), DEFINE_PROP_UINT32("x-intx-mmap-timeout-ms", VFIOPCIDevice, intx.mmap_timeout, 1100), - DEFINE_PROP_BIT("x-vga", VFIOPCIDevice, features, - VFIO_FEATURE_ENABLE_VGA_BIT, false), - DEFINE_PROP_BIT("x-req", VFIOPCIDevice, features, - VFIO_FEATURE_ENABLE_REQ_BIT, true), - DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, - VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice, vbasedev.enable_migration, false), DEFINE_PROP_BOOL("x-no-mmap", VFIOPCIDevice, vbasedev.no_mmap, false), @@ -3247,8 +3235,6 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_BOOL("x-no-kvm-intx", VFIOPCIDevice, no_kvm_intx, false), DEFINE_PROP_BOOL("x-no-kvm-msi", VFIOPCIDevice, no_kvm_msi, false), DEFINE_PROP_BOOL("x-no-kvm-msix", VFIOPCIDevice, no_kvm_msix, false), - DEFINE_PROP_BOOL("x-no-geforce-quirks", VFIOPCIDevice, - no_geforce_quirks, false), DEFINE_PROP_BOOL("x-no-kvm-ioeventfd", VFIOPCIDevice, no_kvm_ioeventfd, false), DEFINE_PROP_BOOL("x-no-vfio-ioeventfd", VFIOPCIDevice, no_vfio_ioeventfd, @@ -3259,10 +3245,6 @@ static Property vfio_pci_dev_properties[] = { sub_vendor_id, PCI_ANY_ID), DEFINE_PROP_UINT32("x-pci-sub-device-id", VFIOPCIDevice, sub_device_id, PCI_ANY_ID), - DEFINE_PROP_UINT32("x-igd-gms", VFIOPCIDevice, igd_gms, 0), - DEFINE_PROP_UNSIGNED_NODEFAULT("x-nv-gpudirect-clique", VFIOPCIDevice, - nv_gpudirect_clique, - qdev_prop_nv_gpudirect_clique, uint8_t), DEFINE_PROP_OFF_AUTO_PCIBAR("x-msix-relocation", VFIOPCIDevice, msix_relo, OFF_AUTOPCIBAR_OFF), /* @@ -3273,28 +3255,25 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_END_OF_LIST(), }; -static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) +static void vfio_pci_base_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); - dc->reset = vfio_pci_reset; - device_class_set_props(dc, vfio_pci_dev_properties); - dc->desc = "VFIO-based PCI device assignment"; + device_class_set_props(dc, vfio_pci_base_dev_properties); + dc->desc = "VFIO PCI base device"; set_bit(DEVICE_CATEGORY_MISC, dc->categories); - pdc->realize = vfio_realize; pdc->exit = vfio_exitfn; pdc->config_read = vfio_pci_read_config; pdc->config_write = vfio_pci_write_config; } -static const TypeInfo vfio_pci_dev_info = { - .name = TYPE_VFIO_PCI, +static const TypeInfo vfio_pci_base_dev_info = { + .name = TYPE_VFIO_PCI_BASE, .parent = TYPE_PCI_DEVICE, - .instance_size = sizeof(VFIOPCIDevice), - .class_init = vfio_pci_dev_class_init, - .instance_init = vfio_instance_init, - .instance_finalize = vfio_instance_finalize, + .instance_size = 0, + .abstract = true, + .class_init = vfio_pci_base_dev_class_init, .interfaces = (InterfaceInfo[]) { { INTERFACE_PCIE_DEVICE }, { INTERFACE_CONVENTIONAL_PCI_DEVICE }, @@ -3302,6 +3281,48 @@ static const TypeInfo vfio_pci_dev_info = { }, }; +static Property vfio_pci_dev_properties[] = { + DEFINE_PROP_PCI_HOST_DEVADDR("host", VFIOPCIDevice, host), + DEFINE_PROP_STRING("sysfsdev", VFIOPCIDevice, vbasedev.sysfsdev), + DEFINE_PROP_ON_OFF_AUTO("display", VFIOPCIDevice, + display, ON_OFF_AUTO_OFF), + DEFINE_PROP_UINT32("xres", VFIOPCIDevice, display_xres, 0), + DEFINE_PROP_UINT32("yres", VFIOPCIDevice, display_yres, 0), + DEFINE_PROP_BIT("x-vga", VFIOPCIDevice, features, + VFIO_FEATURE_ENABLE_VGA_BIT, false), + DEFINE_PROP_BIT("x-req", VFIOPCIDevice, features, + VFIO_FEATURE_ENABLE_REQ_BIT, true), + DEFINE_PROP_BIT("x-igd-opregion", VFIOPCIDevice, features, + VFIO_FEATURE_ENABLE_IGD_OPREGION_BIT, false), + DEFINE_PROP_BOOL("x-no-geforce-quirks", VFIOPCIDevice, + no_geforce_quirks, false), + DEFINE_PROP_UINT32("x-igd-gms", VFIOPCIDevice, igd_gms, 0), + DEFINE_PROP_UNSIGNED_NODEFAULT("x-nv-gpudirect-clique", VFIOPCIDevice, + nv_gpudirect_clique, + qdev_prop_nv_gpudirect_clique, uint8_t), + DEFINE_PROP_END_OF_LIST(), +}; + +static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + dc->reset = vfio_pci_reset; + device_class_set_props(dc, vfio_pci_dev_properties); + dc->desc = "VFIO-based PCI device assignment"; + pdc->realize = vfio_realize; +} + +static const TypeInfo vfio_pci_dev_info = { + .name = TYPE_VFIO_PCI, + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOKernPCIDevice), + .class_init = vfio_pci_dev_class_init, + .instance_init = vfio_instance_init, + .instance_finalize = vfio_instance_finalize, +}; + static Property vfio_pci_dev_nohotplug_properties[] = { DEFINE_PROP_BOOL("ramfb", VFIOPCIDevice, enable_ramfb, false), DEFINE_PROP_END_OF_LIST(), @@ -3318,12 +3339,13 @@ static void vfio_pci_nohotplug_dev_class_init(ObjectClass *klass, void *data) static const TypeInfo vfio_pci_nohotplug_dev_info = { .name = TYPE_VFIO_PCI_NOHOTPLUG, .parent = TYPE_VFIO_PCI, - .instance_size = sizeof(VFIOPCIDevice), + .instance_size = sizeof(VFIOKernPCIDevice), .class_init = vfio_pci_nohotplug_dev_class_init, }; static void register_vfio_pci_dev_type(void) { + type_register_static(&vfio_pci_base_dev_info); type_register_static(&vfio_pci_dev_info); type_register_static(&vfio_pci_nohotplug_dev_info); } From patchwork Thu May 5 17:19:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D13F9C433EF for ; Thu, 5 May 2022 17:41:32 +0000 (UTC) Received: from localhost ([::1]:49588 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfTn-00046v-KE for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:41:31 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52930) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf17-0006R0-75 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:53 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:41966) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf14-0002BD-SZ for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:52 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FGvR9004338 for ; Thu, 5 May 2022 17:11:19 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=yoFJ9Qsq95Gs0eyOcaeO9MC/j6xqhhNZYswheoezAeE=; b=qBLCtUWUYyXY6p9IGlwl684kRgTSSwMmLICnBCikww36Zhr6EknBVtvabWiPt/aE5xTD hDApB+5v3584L53fxgV/dIPzQjoBTK2nz8halGo1V/oIGJ6ZzZE9hPTeK3WoTG1JoTJx oXQEmpg0GXooGdEFbnZqtFkvDeJ3uikNBjhsTNwA9egxX2Vru7p55PyJJrZBmZ3VfflC sOZ5XzWr09OJvlnKtZWd12+En655IPqrh1lCbFH74Sxt37sfpzzUi6NPtpXQh97dRV9W 2kr6I5QVkQeD9M/K+5zhtK3R1vcGvO3s8ncxijZNnsVamWqqi9qY5FQpFcKbjg53YDWc GA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0m36x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:19 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0sqL013487 for ; Thu, 5 May 2022 17:11:18 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fruj4xq8q-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:18 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=FwHyBri0MnMxaTEmBXGS0rQQolZfXMqR1ynqrNtfs4ua+5aC8sBUDV23i4SFH5UtFuN/vlym38pulNTHgL0Azy8MF67kY71uNUMakSiauohuamivwekxowzL8xLsDL5GjYqzhMSughawJXK71Q0t3L82vynY/N1HZOd60hAqteQCUwEXNz/HY9BKKKnU4qogJV+NcJW1jXReD3HyZFeY3D2uuUbe+aNxSfus6eYY+osC+jfkKcNJuQ1AK14POoTr/jxkqs27Bc4aWr0W19Bg4VbSD3flbAcyQTeoI42Fd79ec9sbq0MQ2tN4PJwhExV5cmmBplGHnjI6eWOyye3ePA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yoFJ9Qsq95Gs0eyOcaeO9MC/j6xqhhNZYswheoezAeE=; b=iPK8WAX+rtcoKrYUrD4XiSkH3qRJjweCAgs7CCrhyoxfB3NRsuzrxaqRWWOBKJtpduVi4U5NBfz3j1+TKFvE6BFvkd7jWMVvdtQMDWdgUtT+oL5Ap5KUi34Jmc9CWuXUsq++pY//iAS/gAtB0CMJr6jnSYCh8pQqobZis6AavHnPXlZQzcAJ0b7Ud5n26bOYNVuWqymK5kzGLY7GKyHCfuA/D99lAcME5IE48BkgzcBvCVrGu41ZjoiDPKkVyZ/ER3aT4atcxcn35b4TNPJJBt2fzFEjoVln6lEX0psmgO+/vMRApyybLZgjVcJWK7bvgMRyo27f2xtwEhx8lrgpvg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yoFJ9Qsq95Gs0eyOcaeO9MC/j6xqhhNZYswheoezAeE=; b=abeBMErUXj1IJLx5L5d7kvAOkKeYDnheseMF9l+PqcQmqoTYw16807ur+AGOeLEyfIFi8rK0wP7R0UvdIf4qFI+nMlMsCmntvMtG/jREzRJNZppgifHJvDNvVa8BrMHHfGJUhB00yZy77xXG2qsH2SzU6Pj2khZbAGaVvPYYqxM= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:17 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:16 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 03/23] vfio-user: add container IO ops vector Date: Thu, 5 May 2022 10:19:46 -0700 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7b6d5c3d-97b4-4cac-4522-08da2eba43c3 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: jhIiKJsQHlhNUofMPvZNhwpC2VfUZKOWLMHDhlOvT31HRCYUyZdT9QadXnUlLT/rUan20vUmw5dkeVobLpB7fB4OLxe+kMzbJI/bI5yf58KshVvYfYDfL8GTMF75PNF+IWFhROShRT7w2uL7GxKcZz0IwayPf3nb/p752r0qN52slxM/mLWRcIMk2BkN+02WF/NlZEvxYEmEP2XlWbffpJ/ot2atTlw6qxtv01AuTpuLK72m20Tl9CtsrQDSSPEQFqDgDpjlu8GBrkHptQzBjkiNSj+9IKNxXcbFXC+sRKSDoNtdHPtmD2oOCrnwOxjgGZSG64L0JksdQojWkK1kj6AFSojmmHl5teEWPiwy7DSF1/yKjDyXK4cIXfnfBl+88viysvQ4LCI9FnFMoNiGFngnD2dDj9pDJilUqfknELBtlY7Lr6DxSwpp0OGqDgUx8rHCWskYkfAmsnw7MoI1oGijz9132dLkjEoFOyKXJGsXEknI0fxZg18LjPpeFTDILYFRpdPReClI74QG5kaRsk8u4Q9L8zMnzKSS7QHZm1hIKQ0M+d/dQE0kAnNnWhK64wzuWv2aLfn/uJOn/RSG7yxpwwZ9X8q6AesMHa73Nqq8wqislzlCfoXxzrNqvPxK9qgjb/NjkjO0Lf48kHIukFm7AsSdDgS37gV7RVIJyLNDu8Wjj1QO26mWwBM4evwtkYWYkVylP/xaYnPzv1t9XQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /FQ2Na2A0Ja9AhgDBhetaauCK5TsB/71MTPTjBURbDl8pL0omJSAmGzoF9hYFromzzxLUJI9tqZiswcmpiBzKjwjbGB/+ITTswkFZdY/hXswCk4YmBcu8UZsN81G+Rts7/6L3Htz2WiJN+/zvTskB4CKUhkOz+KXoq5YRpE8jZvMCv3s17sKOZrsO6zb1bJaYaZfaHNdFZu+lzF7s8JruvYjaWLgYqy8EAH9exZ7cdUF/KuC6i/Ki4r4sfxUenL+BOV4GrX4E18PnTrVHOhIej9F+KygNhN3pQhE+d8KKDz+TxCsF4lhB4+GVToQdp05mfZf4zNHWW2v2heYx50TOxMFllnwZ40eaxtNOVsnLm7HCz8NMcyWtCxrQuMtljF56ERVK2ydrlDogwLmYFBl8EKc3wVYOXPPsd4Wu1dowlKyX6WGKCGbjAyxX9XMglpEXPiZsCpL9ccoKvcJo9IquIOTkaJNDkrbiIbuogany0QPFNMAsf65LwH9AKAECueFqVhoTeR7xztU8/nslBILG/2/HMZr30dexs0fKcnen7MVIcfyoTCajr/U3euWDMHzQULsICByucqnJCnqX8f+fWJcUlfg4rRS2sVE4yiYDn/Fg5vC927uU3me+e+/9bhx7KUuSZ7U1QcQ/V9SeCgBsMfoFp4e/Eol7VO35XQLODrOYvS28+fbA0nM3JTHEvZk13HI/tu1svTqY65p38e0RFpoyssasN5L4o5PcyMrufer2TX/6fJhuL34jF9WZKbpto8+wV8YUudNltw56o2IO6/vdLbWlNuu9uN9/GV8LWDKM5i8y2VWca4kBtsHjNdFhiQr3IRoS+mL07A+B0p8iMZ4Mfnk3rmG3UBGIZkhWCETZBe4hNt4NrT2+SH/mkFA6oMHvNUQII9qOwbQZlOCMyg3oEuzKVSGJ+Mwf7Ug15SEfy0MJXMMAwEdLdCrG1S43bKh75bOX1Z4kZ2WrUDPv16yGfjt7WWiLuMVWhtW6uJQCY3vP8mL4tmnMAzW4sTqk0shXPeaAnDY+oOSCkqGTHEBTmCqSA3baPtNB5o0Y91EUN/iOcJWfCStc0a/QW565qwIyEVrOBJjRZRa3crcVnAHoclIF9t3i3rDBeTfTj34+lRlxyff+xGpeCooMZIYg4RMTQ/iupv1/+NqsDiQDaNsFnHnowW4OREmgYhOcDwLerZt270BEMQBNaUstcsz1sTo3hMTFlju+T/OimtRvrX8/prlKiSzJtnpmH/1wi/cSYz4oKW9b7mGOmvmLSFAYCuAP449CQU+nmdpssRuCnLWSPPSunYiJfrTVaabcH1PRP/2F2QSV5rH0D/WB08Hh9dJG92ONMzrh0E32qeS8CsZ2CWWI57Brc/BpzIuo95AMN+nxFXgq5jHX/+czIgCPVscNBsvJ67koJTG0cg7cVvDKo8XppcXs1gKyPrb9ERqE6IlQtf6nkfHv4Gw5jf00n1SynbKrV+8OPAE9UYjzXR5xnEckr33EPH7x1MZXcFGeYuhgCVdH7+tgPNJ/wyp/pyqcvj7QG1MTeFuxWPCKYxNvvv9j5aRYPO7VV47gN6tOyM3BcUwiAgkl0Dm0DRBGyNHW8j9jdjw1AaTsO4rrHkLM9Ls5M1tD/Yga3GWRuLxKHe8WO61JXktFAoQgkDq0dZWZnfOBoLELW174Hp68UDRhcDDm6ZCjLg/ZX/QwKAG+NZHFGAWmU8rAZA1OWgWqA9Z7KXUCELiIyt7R80rPHyBSaOynWYUrMHN/vqGd9s= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7b6d5c3d-97b4-4cac-4522-08da2eba43c3 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:16.3418 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aWFqzmB2NrLgJ12sjzG1gwGD9t8C7XW2nMEUKzH2suzw9te7e3x189tTK2y3jS0O+cKKlyhcr7a/Mhdx1QvHAfLm8lTYRZ/qr7PJSXVmbYU= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: yl92ww0495SgQJ14ePlAiQfK5o78_Z6- X-Proofpoint-GUID: yl92ww0495SgQJ14ePlAiQfK5o78_Z6- Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Used for communication with VFIO driver (prep work for vfio-user, which will communicate over a socket) Signed-off-by: John G Johnson --- include/hw/vfio/vfio-common.h | 33 +++++++++++ hw/vfio/common.c | 126 ++++++++++++++++++++++++++++-------------- 2 files changed, 117 insertions(+), 42 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 8af11b0..2761a62 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace { } VFIOAddressSpace; struct VFIOGroup; +typedef struct VFIOContIO VFIOContIO; typedef struct VFIOContainer { VFIOAddressSpace *space; @@ -83,6 +84,7 @@ typedef struct VFIOContainer { MemoryListener prereg_listener; unsigned iommu_type; Error *error; + VFIOContIO *io_ops; bool initialized; bool dirty_pages_supported; uint64_t dirty_pgsizes; @@ -154,6 +156,37 @@ struct VFIODeviceOps { int (*vfio_load_config)(VFIODevice *vdev, QEMUFile *f); }; +#ifdef CONFIG_LINUX + +/* + * The next 2 ops vectors are how Devices and Containers + * communicate with the server. The default option is + * through ioctl() to the kernel VFIO driver, but vfio-user + * can use a socket to a remote process. + */ + +struct VFIOContIO { + int (*dma_map)(VFIOContainer *container, + struct vfio_iommu_type1_dma_map *map); + int (*dma_unmap)(VFIOContainer *container, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap); + int (*dirty_bitmap)(VFIOContainer *container, + struct vfio_iommu_type1_dirty_bitmap *bitmap, + struct vfio_iommu_type1_dirty_bitmap_get *range); +}; + +#define CONT_DMA_MAP(cont, map) \ + ((cont)->io_ops->dma_map((cont), (map))) +#define CONT_DMA_UNMAP(cont, unmap, bitmap) \ + ((cont)->io_ops->dma_unmap((cont), (unmap), (bitmap))) +#define CONT_DIRTY_BITMAP(cont, bitmap, range) \ + ((cont)->io_ops->dirty_bitmap((cont), (bitmap), (range))) + +extern VFIOContIO vfio_cont_io_ioctl; + +#endif /* CONFIG_LINUX */ + typedef struct VFIOGroup { int fd; int groupid; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 2b1f78f..917da0f 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -431,12 +431,12 @@ static int vfio_dma_unmap_bitmap(VFIOContainer *container, goto unmap_exit; } - ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + ret = CONT_DMA_UNMAP(container, unmap, bitmap); if (!ret) { cpu_physical_memory_set_dirty_lebitmap((unsigned long *)bitmap->data, iotlb->translated_addr, pages); } else { - error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %m"); + error_report("VFIO_UNMAP_DMA with DIRTY_BITMAP : %s", strerror(-ret)); } g_free(bitmap->data); @@ -464,30 +464,7 @@ static int vfio_dma_unmap(VFIOContainer *container, return vfio_dma_unmap_bitmap(container, iova, size, iotlb); } - while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { - /* - * The type1 backend has an off-by-one bug in the kernel (71a7d3d78e3c - * v4.15) where an overflow in its wrap-around check prevents us from - * unmapping the last page of the address space. Test for the error - * condition and re-try the unmap excluding the last page. The - * expectation is that we've never mapped the last page anyway and this - * unmap request comes via vIOMMU support which also makes it unlikely - * that this page is used. This bug was introduced well after type1 v2 - * support was introduced, so we shouldn't need to test for v1. A fix - * is queued for kernel v5.0 so this workaround can be removed once - * affected kernels are sufficiently deprecated. - */ - if (errno == EINVAL && unmap.size && !(unmap.iova + unmap.size) && - container->iommu_type == VFIO_TYPE1v2_IOMMU) { - trace_vfio_dma_unmap_overflow_workaround(); - unmap.size -= 1ULL << ctz64(container->pgsizes); - continue; - } - error_report("VFIO_UNMAP_DMA failed: %s", strerror(errno)); - return -errno; - } - - return 0; + return CONT_DMA_UNMAP(container, &unmap, NULL); } static int vfio_dma_map(VFIOContainer *container, hwaddr iova, @@ -500,24 +477,18 @@ static int vfio_dma_map(VFIOContainer *container, hwaddr iova, .iova = iova, .size = size, }; + int ret; if (!readonly) { map.flags |= VFIO_DMA_MAP_FLAG_WRITE; } - /* - * Try the mapping, if it fails with EBUSY, unmap the region and try - * again. This shouldn't be necessary, but we sometimes see it in - * the VGA ROM space. - */ - if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) == 0 || - (errno == EBUSY && vfio_dma_unmap(container, iova, size, NULL) == 0 && - ioctl(container->fd, VFIO_IOMMU_MAP_DMA, &map) == 0)) { - return 0; - } + ret = CONT_DMA_MAP(container, &map); - error_report("VFIO_MAP_DMA failed: %s", strerror(errno)); - return -errno; + if (ret < 0) { + error_report("VFIO_MAP_DMA failed: %s", strerror(-ret)); + } + return ret; } static void vfio_host_win_add(VFIOContainer *container, @@ -1230,10 +1201,10 @@ static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool start) dirty.flags = VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; } - ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); + ret = CONT_DIRTY_BITMAP(container, &dirty, NULL); if (ret) { error_report("Failed to set dirty tracking flag 0x%x errno: %d", - dirty.flags, errno); + dirty.flags, -ret); } } @@ -1283,11 +1254,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, goto err_out; } - ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + ret = CONT_DIRTY_BITMAP(container, dbitmap, range); if (ret) { error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, - (uint64_t)range->size, errno); + (uint64_t)range->size, -ret); goto err_out; } @@ -2058,6 +2029,7 @@ static int vfio_connect_container(VFIOGroup *group, AddressSpace *as, container->error = NULL; container->dirty_pages_supported = false; container->dma_max_mappings = 0; + container->io_ops = &vfio_cont_io_ioctl; QLIST_INIT(&container->giommu_list); QLIST_INIT(&container->hostwin_list); QLIST_INIT(&container->vrdl_list); @@ -2594,3 +2566,73 @@ int vfio_eeh_as_op(AddressSpace *as, uint32_t op) } return vfio_eeh_container_op(container, op); } + +/* + * Traditional ioctl() based io_ops + */ + +static int vfio_io_dma_map(VFIOContainer *container, + struct vfio_iommu_type1_dma_map *map) +{ + + /* + * Try the mapping, if it fails with EBUSY, unmap the region and try + * again. This shouldn't be necessary, but we sometimes see it in + * the VGA ROM space. + */ + if (ioctl(container->fd, VFIO_IOMMU_MAP_DMA, map) == 0 || + (errno == EBUSY && + vfio_dma_unmap(container, map->iova, map->size, NULL) == 0 && + ioctl(container->fd, VFIO_IOMMU_MAP_DMA, map) == 0)) { + return 0; + } + return -errno; +} + +static int vfio_io_dma_unmap(VFIOContainer *container, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap) +{ + + while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap)) { + /* + * The type1 backend has an off-by-one bug in the kernel (71a7d3d78e3c + * v4.15) where an overflow in its wrap-around check prevents us from + * unmapping the last page of the address space. Test for the error + * condition and re-try the unmap excluding the last page. The + * expectation is that we've never mapped the last page anyway and this + * unmap request comes via vIOMMU support which also makes it unlikely + * that this page is used. This bug was introduced well after type1 v2 + * support was introduced, so we shouldn't need to test for v1. A fix + * is queued for kernel v5.0 so this workaround can be removed once + * affected kernels are sufficiently deprecated. + */ + if (errno == EINVAL && unmap->size && !(unmap->iova + unmap->size) && + container->iommu_type == VFIO_TYPE1v2_IOMMU) { + trace_vfio_dma_unmap_overflow_workaround(); + unmap->size -= 1ULL << ctz64(container->pgsizes); + continue; + } + error_report("VFIO_UNMAP_DMA failed: %s", strerror(errno)); + return -errno; + } + + return 0; +} + +static int vfio_io_dirty_bitmap(VFIOContainer *container, + struct vfio_iommu_type1_dirty_bitmap *bitmap, + struct vfio_iommu_type1_dirty_bitmap_get *range) +{ + int ret; + + ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, bitmap); + + return ret < 0 ? -errno : ret; +} + +VFIOContIO vfio_cont_io_ioctl = { + .dma_map = vfio_io_dma_map, + .dma_unmap = vfio_io_dma_unmap, + .dirty_bitmap = vfio_io_dirty_bitmap, +}; From patchwork Thu May 5 17:19:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839823 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A99CBC433EF for ; Thu, 5 May 2022 17:27:47 +0000 (UTC) Received: from localhost ([::1]:57492 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfGU-0006k1-Mq for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:27:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52588) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0k-00063L-NH for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:47520) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0e-0002Bi-Ok for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:29 -0400 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245GgoMr019152 for ; Thu, 5 May 2022 17:11:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=19u8VelCgchRdim8fYLw4N1ux/9bG6io3C8mB630iDg=; b=Mmo3uk6wguGmIXsXPOBgiCdpDfLVWx//6Qg8ZdUBXUfMwRcMGFOcasPtGDD4TeqlvNjR Yan3fCHYBIVWXM6c1E+RBTjU1+yG0uhwenhturddZJVIXoYTa8AmN/PysIWkzMsR2MMj wLmwLJfiEORCL1q7r8j3Z8O30E+sEKh4CmBKc2k6GRZK7dTuBYlE+W5KiKSrlspuq7BU NsK4HHVRxQq1eOzFlZd010QrKvc85CdJ85ZYlWcLXsCd4gFx8wZR7MDPIn7iTwmwqNX2 ukxsIU8fI0ksiMXyjGLFquxSERRIJ/5c8XNYgp0sDeCmQ05f1OBoB/8lq/WKtB7PrCIX 7Q== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frw0aupam-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:20 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0sqO013487 for ; Thu, 5 May 2022 17:11:19 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fruj4xq8q-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WVvk3h2XHB1sxAPNYEziK+D5IYUXohE125TPf1s+JnLEIBEonMsGpzOF+YKQNmZ529sbw8AsllRGdWVoJT1hI9lXgTMgo5JBb1kycg4IIntDaBlKBtS0Urj9YMlOEQue6xH6JF/gl3EAUQy4y1iuS7HJ4rSJd3iR0MAi29pB0ORvh4VqYwiaDPXOmAYQF+w4I3KhJL0HBTRkbj9/axahkVF6fU9eKyRlJ14NJET568OSU3nffqXkFkBbhCT39CuLZ3wYjqlNOK+9li95AyzsUXbRIhHstTr4eIi+pyxTDKCNoxRDPS+gbO6mWIXcYmlI42dJlyMwLVkYTfkbk0wjyw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=19u8VelCgchRdim8fYLw4N1ux/9bG6io3C8mB630iDg=; b=QYWxJX2ZDvpprLKSY08GXFWCCyzORQq/8Wbjg7Asju541uRvOUflCJznXw2eyp5kg9w6YOaHUFvMAu86/ajhDlq28l5+X2z2h8xvvCm+h/rvk6DrE2hGJQUw+Mi85DAGq2ij5Vs4ADpMviQb3RvkiLloThXfhCbjZL7pw6Qh3OaUlz6IzKyNJrYS04dUcHMerV07WrQVm7+sd9pKHMG2whzvvPGsHQxuF2qcvPGMJ5VCZpqk7Ac+bQWZU3dUeOLV1NnrWnShbyy4ZgrhRnccCJOO1iF/Wnx540jO0uMJnNqsdnO4FecKx+92ytxpygzvB8b7P9FLw9f+jqgMMdnNNg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=19u8VelCgchRdim8fYLw4N1ux/9bG6io3C8mB630iDg=; b=KjRMXW6bCnkplSYldGXqNpZt78DYA8UWaZQXnTvuLkws5J3uBqnKTTOoODt1iUFADLwwqwMCwH2oyN0I8BIskedxxrvOOSD3WTs4jsC+qGMHa3xDdV3pEgqIBIBcBRGzGWrUSo5QV1ff8WMYida+8066bnj7ckpQY3jvPnAf3EY= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:17 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:17 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 04/23] vfio-user: add region cache Date: Thu, 5 May 2022 10:19:47 -0700 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: cb6ed96c-7bfe-4992-7ca3-08da2eba43fc X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: xwumnNw3mRzN3Vr7e6K5/QPYLd/YhrosjgWtHMZE0B9oCoejrIHcu+YL+b/U7kddtmrYkdmAVAcCp6Oyvmt7hD4/mb7MXqUbu/hVVEpNRvF0YlXxBzEYTFal07wDEvqtMQGWBFOivUP+Y35kSRb5xWuU5KFXKKn+aWox7Vpc4tQ7LBzOknZEcMw8xmCa0HZOUr5yBMMlfGIwe4xIDi53ogTjgwA+ETl0K8iCNt1pqyNpLf2wgWqTp6qy8/fX7AJpyokfzz1OIZMmTilOuXmXDQpp9tQSGSyIO6RyxsmZ9rZS1drNy4lIYZJpB2DoIOJo6hwxR/6CQS4PrhQSnCy62LfVWs5cJGonaMnqVyZTzHIv4Lwm1l6KpBZVYKACY3hcih8mJBqGvKSRWf41ljtdGpmAfYSSsm3kKqnzQTmmtHkHhmwdcloneJTlqcpJrV5xm/lLg8oSNzWWT23sU0Y6mbtJ4esCwXqxUZ0Ioot8qRqatFb4cKRc1mtYDCZJUiLnWent7YKJFaZYdvTNDSmAhMpFLaxhZfH/D4yNqul3YY91OmyztrFa7/C9aThKNxMnyo2cJ0UYhoFa8WMswzKXlnpVpEvwl7r0jdbLDeeffFml6Vofpl+zZVLU1+4C5GfcIAoAocogidJBxn+dwnyuQe+BaBYv9t+uhMe8hZ9GzsnoWUmNR3jUwowMXgEREpZD6qH4pab8rOaKVEJRyckq7g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(30864003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Pk3z0zgpjDeKhPatSJXoXNEIkU2+ufWuh3HniclxYWsIRylKQcuV9gazOspU9E78zztapYJjXS5MCPWOJmUBplBTZqI3tPfi3/9775Es2ZAEQ54yYfZ+EhecJv5UUvbIYEOeU0WejnNmzaChD84j+5u58YG8hN3w78rATrI+jJ+bZTFnTbr5QFh7pjBY1mfTgJYTYn90P6Z5RpYMkqSNzzAA54YU+hoO0H/vWz5XR2q82Vm0vWdAkVHi1Tzd41Dy0HSdYMvrcf53F8EHc2kPgiGCWa3FQoMPe4cEC0/2MbJUILo2i+aMG7GwDi1Edv5cpDk3/zfvNJmv1KT2SusVXYqSJa5U1HyMF51F29DVBr+4BGcIZSMSZfHVftMJWWfOkFY9KAv+DeG4FLcgaf1Je9/v3bGhtHBORDk0JuZLZKtigFdIbMda+n8+qij3Rjgaygke5+EuPexkCDtVrVudB+IvXNFyxpOzH/petrAjMaitOGJo2p/YEcz+hfZr8EQYBGjisR1oh5/ll3U394JfYW673GVZpcAyP4LK+H2qDW6rSlujrAaPlIW/+ssh8dhQ3eS+WufWieoT06bAzgStRbxI96+/qPAJp2PIBNX2giNezFaNrkWPRGwdDwNTjXgQd7QfiZ3tPE2sfkAJLU3eNcGsn+byk7dYZgTKbC2NJDKBFBLDhdTnzBXVX5J6LtbQypXt0DPs3W8A+2v7BU0BZ6lBM2n0JjHEWXRppQjsxearKyWEm3v8xp6I5bgnsP+mKePmClWU2b5S3K+xwiZoNOV4wfni/0IcaYJrWIu+DbsF0tBSWYIqk25oRmlGswuXpWVC68qVeDppyJD/ZSqC2mHBKO/SLPS3D85nSwC4mKV8/Vgv9bP2PRWTOdneR36YcS7da+I5krjpVJUukycXXANvem+VnLVjKwjf8g9xgV6Niv4Pu31dzK4tmCxt4BNqjxlapR/VI3YPDOBT/xGLEsBa/Gq47geMd3sQBhIKST/letgCZGnSRxfyUtvnOHV16PjZbuzPG401alVVoqaS+9SM79hdeg5DA5PggkT6mkulz2z7gkzc6X3Nz+j94OP1Zkjk50QgaymldWMKnZuFrD6Fjb7ooLRNJ2UnQCuQHSu1B+EcXD9R8p+YpY/UMZ52wEibtMyZvEthKUhyEAHVtb8rXLqBhxgMyQNYUbMQb0Qx8qTo/O1x0wY+Oq/BIzkYelxxO2R42BemI1Q0Bhdl/AkBGqtmxf8VL0emnfxOstnG3S63uzkmp4OKP0Lcvqdta/T/u2B2IItXHkFxXlTNgTZLO/dLfwlnKQdLLlcuXCrSU5ZYEQeBz3Q+5esuD7M6Zqe7zpieLJCu7uAE1dyLvfV1TNI/sARmWAxw9ZCcPPeoZpb/6TWJjVftJrj/Wk9aOI5nwRXxx4XV2PppL/4IrqF00O5vrACa7XDy2p1jrwleEGgjxUNvtpuO3NY0Qd8eeHjP+UUYDAyVRxsDQ5ueVYU1KKt921DqU1r5KgxeC0+F+f3kYc5XaiqD93WpcAUearshUGXBMsjoumS9nHeHBq3speLaQexabThZ1o0SVohGHLsQh/Y2FICrVYhnSZJGo8BMRHdr7G0KEgnIXRtpp4KMPPc2cp8D0pGTB3GgS/+vB7emVl1q3oiKKIQE343iYpLPEvIkVjyQHuIOhatFmVVXd+6MpB08cvduPffX/IjdcmQCj9pvwMsMuP+wZ0tvpajM9tCLZcDdNGjOofwibiYSR0i5RqVhSVL7V/6/T3c= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: cb6ed96c-7bfe-4992-7ca3-08da2eba43fc X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:16.6855 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: FBUbVwrq49CZhc3ML26H0vUu1TtTkwgLO/b3Tru498H7y3of7arjYZDEA8aNBjvK7lM4j3bIycs8/k1LVKeJ9/5azZb8EQE2q68km3DYr8A= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: UQ2Ls4Xjc4Hi9JW9XciUzXR3S_RIMqPR X-Proofpoint-ORIG-GUID: UQ2Ls4Xjc4Hi9JW9XciUzXR3S_RIMqPR Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" cache VFIO_DEVICE_GET_REGION_INFO results to reduce memory alloc/free cycles and as prep work for vfio-user Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- include/hw/vfio/vfio-common.h | 2 ++ hw/vfio/ccw.c | 5 ----- hw/vfio/common.c | 41 +++++++++++++++++++++++++++++++++++------ hw/vfio/igd.c | 23 +++++++++-------------- hw/vfio/migration.c | 2 -- hw/vfio/pci-quirks.c | 19 +++++-------------- hw/vfio/pci.c | 8 -------- 7 files changed, 51 insertions(+), 49 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 2761a62..1a032f4 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -145,6 +145,7 @@ typedef struct VFIODevice { VFIOMigration *migration; Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; + struct vfio_region_info **regions; } VFIODevice; struct VFIODeviceOps { @@ -258,6 +259,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info); int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type, uint32_t subtype, struct vfio_region_info **info); +void vfio_get_all_regions(VFIODevice *vbasedev); bool vfio_has_region_cap(VFIODevice *vbasedev, int region, uint16_t cap_type); struct vfio_info_cap_header * vfio_get_region_info_cap(struct vfio_region_info *info, uint16_t id); diff --git a/hw/vfio/ccw.c b/hw/vfio/ccw.c index 0354737..06b588c 100644 --- a/hw/vfio/ccw.c +++ b/hw/vfio/ccw.c @@ -517,7 +517,6 @@ static void vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) vcdev->io_region_offset = info->offset; vcdev->io_region = g_malloc0(info->size); - g_free(info); /* check for the optional async command region */ ret = vfio_get_dev_region_info(vdev, VFIO_REGION_TYPE_CCW, @@ -530,7 +529,6 @@ static void vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) } vcdev->async_cmd_region_offset = info->offset; vcdev->async_cmd_region = g_malloc0(info->size); - g_free(info); } ret = vfio_get_dev_region_info(vdev, VFIO_REGION_TYPE_CCW, @@ -543,7 +541,6 @@ static void vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) } vcdev->schib_region_offset = info->offset; vcdev->schib_region = g_malloc(info->size); - g_free(info); } ret = vfio_get_dev_region_info(vdev, VFIO_REGION_TYPE_CCW, @@ -557,7 +554,6 @@ static void vfio_ccw_get_region(VFIOCCWDevice *vcdev, Error **errp) } vcdev->crw_region_offset = info->offset; vcdev->crw_region = g_malloc(info->size); - g_free(info); } return; @@ -567,7 +563,6 @@ out_err: g_free(vcdev->schib_region); g_free(vcdev->async_cmd_region); g_free(vcdev->io_region); - g_free(info); return; } diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 917da0f..d9290f3 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1568,8 +1568,6 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, } } - g_free(info); - trace_vfio_region_setup(vbasedev->name, index, name, region->flags, region->fd_offset, region->size); return 0; @@ -2325,6 +2323,16 @@ void vfio_put_group(VFIOGroup *group) } } +void vfio_get_all_regions(VFIODevice *vbasedev) +{ + struct vfio_region_info *info; + int i; + + for (i = 0; i < vbasedev->num_regions; i++) { + vfio_get_region_info(vbasedev, i, &info); + } +} + int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp) { @@ -2380,12 +2388,23 @@ int vfio_get_device(VFIOGroup *group, const char *name, trace_vfio_get_device(name, dev_info.flags, dev_info.num_regions, dev_info.num_irqs); + vfio_get_all_regions(vbasedev); vbasedev->reset_works = !!(dev_info.flags & VFIO_DEVICE_FLAGS_RESET); return 0; } void vfio_put_base_device(VFIODevice *vbasedev) { + if (vbasedev->regions != NULL) { + int i; + + for (i = 0; i < vbasedev->num_regions; i++) { + g_free(vbasedev->regions[i]); + } + g_free(vbasedev->regions); + vbasedev->regions = NULL; + } + if (!vbasedev->group) { return; } @@ -2400,6 +2419,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, { size_t argsz = sizeof(struct vfio_region_info); + /* create region cache */ + if (vbasedev->regions == NULL) { + vbasedev->regions = g_new0(struct vfio_region_info *, + vbasedev->num_regions); + } + /* check cache */ + if (vbasedev->regions[index] != NULL) { + *info = vbasedev->regions[index]; + return 0; + } + *info = g_malloc0(argsz); (*info)->index = index; @@ -2419,6 +2449,9 @@ retry: goto retry; } + /* fill cache */ + vbasedev->regions[index] = *info; + return 0; } @@ -2437,7 +2470,6 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type, hdr = vfio_get_region_info_cap(*info, VFIO_REGION_INFO_CAP_TYPE); if (!hdr) { - g_free(*info); continue; } @@ -2449,8 +2481,6 @@ int vfio_get_dev_region_info(VFIODevice *vbasedev, uint32_t type, if (cap_type->type == type && cap_type->subtype == subtype) { return 0; } - - g_free(*info); } *info = NULL; @@ -2466,7 +2496,6 @@ bool vfio_has_region_cap(VFIODevice *vbasedev, int region, uint16_t cap_type) if (vfio_get_region_info_cap(info, cap_type)) { ret = true; } - g_free(info); } return ret; diff --git a/hw/vfio/igd.c b/hw/vfio/igd.c index afe3fe7..22efa1a 100644 --- a/hw/vfio/igd.c +++ b/hw/vfio/igd.c @@ -425,7 +425,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) if ((ret || !rom->size) && !vdev->pdev.romfile) { error_report("IGD device %s has no ROM, legacy mode disabled", vdev->vbasedev.name); - goto out; + return; } /* @@ -436,7 +436,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) error_report("IGD device %s hotplugged, ROM disabled, " "legacy mode disabled", vdev->vbasedev.name); vdev->rom_read_failed = true; - goto out; + return; } /* @@ -449,7 +449,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) if (ret) { error_report("IGD device %s does not support OpRegion access," "legacy mode disabled", vdev->vbasedev.name); - goto out; + return; } ret = vfio_get_dev_region_info(&vdev->vbasedev, @@ -458,7 +458,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) if (ret) { error_report("IGD device %s does not support host bridge access," "legacy mode disabled", vdev->vbasedev.name); - goto out; + return; } ret = vfio_get_dev_region_info(&vdev->vbasedev, @@ -467,7 +467,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) if (ret) { error_report("IGD device %s does not support LPC bridge access," "legacy mode disabled", vdev->vbasedev.name); - goto out; + return; } gmch = vfio_pci_read_config(&vdev->pdev, IGD_GMCH, 4); @@ -481,7 +481,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); error_report("IGD device %s failed to enable VGA access, " "legacy mode disabled", vdev->vbasedev.name); - goto out; + return; } /* Create our LPC/ISA bridge */ @@ -489,7 +489,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) if (ret) { error_report("IGD device %s failed to create LPC bridge, " "legacy mode disabled", vdev->vbasedev.name); - goto out; + return; } /* Stuff some host values into the VM PCI host bridge */ @@ -497,7 +497,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) if (ret) { error_report("IGD device %s failed to modify host bridge, " "legacy mode disabled", vdev->vbasedev.name); - goto out; + return; } /* Setup OpRegion access */ @@ -505,7 +505,7 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) if (ret) { error_append_hint(&err, "IGD legacy mode disabled\n"); error_reportf_err(err, VFIO_MSG_PREFIX, vdev->vbasedev.name); - goto out; + return; } /* Setup our quirk to munge GTT addresses to the VM allocated buffer */ @@ -608,9 +608,4 @@ void vfio_probe_igd_bar4_quirk(VFIOPCIDevice *vdev, int nr) trace_vfio_pci_igd_bdsm_enabled(vdev->vbasedev.name, ggms_mb + gms_mb); -out: - g_free(rom); - g_free(opregion); - g_free(host); - g_free(lpc); } diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index ff6b45d..04bfb3a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -876,13 +876,11 @@ int vfio_migration_probe(VFIODevice *vbasedev, Error **errp) } trace_vfio_migration_probe(vbasedev->name, info->index); - g_free(info); return 0; add_blocker: error_setg(&vbasedev->migration_blocker, "VFIO device doesn't support migration"); - g_free(info); ret = migrate_add_blocker(vbasedev->migration_blocker, errp); if (ret < 0) { diff --git a/hw/vfio/pci-quirks.c b/hw/vfio/pci-quirks.c index 0cf69a8..5cfd93d 100644 --- a/hw/vfio/pci-quirks.c +++ b/hw/vfio/pci-quirks.c @@ -1601,16 +1601,14 @@ int vfio_pci_nvidia_v100_ram_init(VFIOPCIDevice *vdev, Error **errp) hdr = vfio_get_region_info_cap(nv2reg, VFIO_REGION_INFO_CAP_NVLINK2_SSATGT); if (!hdr) { - ret = -ENODEV; - goto free_exit; + return -ENODEV; } cap = (void *) hdr; p = mmap(NULL, nv2reg->size, PROT_READ | PROT_WRITE, MAP_SHARED, vdev->vbasedev.fd, nv2reg->offset); if (p == MAP_FAILED) { - ret = -errno; - goto free_exit; + return -errno; } quirk = vfio_quirk_alloc(1); @@ -1623,8 +1621,6 @@ int vfio_pci_nvidia_v100_ram_init(VFIOPCIDevice *vdev, Error **errp) (void *) (uintptr_t) cap->tgt); trace_vfio_pci_nvidia_gpu_setup_quirk(vdev->vbasedev.name, cap->tgt, nv2reg->size); -free_exit: - g_free(nv2reg); return ret; } @@ -1651,16 +1647,14 @@ int vfio_pci_nvlink2_init(VFIOPCIDevice *vdev, Error **errp) hdr = vfio_get_region_info_cap(atsdreg, VFIO_REGION_INFO_CAP_NVLINK2_SSATGT); if (!hdr) { - ret = -ENODEV; - goto free_exit; + return -ENODEV; } captgt = (void *) hdr; hdr = vfio_get_region_info_cap(atsdreg, VFIO_REGION_INFO_CAP_NVLINK2_LNKSPD); if (!hdr) { - ret = -ENODEV; - goto free_exit; + return -ENODEV; } capspeed = (void *) hdr; @@ -1669,8 +1663,7 @@ int vfio_pci_nvlink2_init(VFIOPCIDevice *vdev, Error **errp) p = mmap(NULL, atsdreg->size, PROT_READ | PROT_WRITE, MAP_SHARED, vdev->vbasedev.fd, atsdreg->offset); if (p == MAP_FAILED) { - ret = -errno; - goto free_exit; + return -errno; } quirk = vfio_quirk_alloc(1); @@ -1690,8 +1683,6 @@ int vfio_pci_nvlink2_init(VFIOPCIDevice *vdev, Error **errp) (void *) (uintptr_t) capspeed->link_speed); trace_vfio_pci_nvlink2_setup_quirk_lnkspd(vdev->vbasedev.name, capspeed->link_speed); -free_exit: - g_free(atsdreg); return ret; } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 4ee5215..35b6551 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -793,8 +793,6 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev) vdev->rom_size = size = reg_info->size; vdev->rom_offset = reg_info->offset; - g_free(reg_info); - if (!vdev->rom_size) { vdev->rom_read_failed = true; error_report("vfio-pci: Cannot read device rom at " @@ -2521,7 +2519,6 @@ int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) error_setg(errp, "unexpected VGA info, flags 0x%lx, size 0x%lx", (unsigned long)reg_info->flags, (unsigned long)reg_info->size); - g_free(reg_info); return -EINVAL; } @@ -2530,8 +2527,6 @@ int vfio_populate_vga(VFIOPCIDevice *vdev, Error **errp) vdev->vga->fd_offset = reg_info->offset; vdev->vga->fd = vdev->vbasedev.fd; - g_free(reg_info); - vdev->vga->region[QEMU_PCI_VGA_MEM].offset = QEMU_PCI_VGA_MEM_BASE; vdev->vga->region[QEMU_PCI_VGA_MEM].nr = QEMU_PCI_VGA_MEM; QLIST_INIT(&vdev->vga->region[QEMU_PCI_VGA_MEM].quirks); @@ -2626,8 +2621,6 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) } vdev->config_offset = reg_info->offset; - g_free(reg_info); - if (vdev->features & VFIO_FEATURE_ENABLE_VGA) { ret = vfio_populate_vga(vdev, errp); if (ret) { @@ -3035,7 +3028,6 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } ret = vfio_pci_igd_opregion_init(vdev, opregion, errp); - g_free(opregion); if (ret) { goto out_teardown; } From patchwork Thu May 5 17:19:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id CCD14C433EF for ; Thu, 5 May 2022 17:34:13 +0000 (UTC) Received: from localhost ([::1]:37754 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfMi-0004NP-Ta for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:34:12 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52586) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0l-00063K-8z for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:45262) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0f-0002BR-8S for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:29 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FTMUY030007 for ; Thu, 5 May 2022 17:11:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=VtsDZ9Hg7Rrm/a6/XVUff29pbC98CuZpy222DxzXrSM=; b=RR4dvzcTN01MSquVrc74mpXQ3zV/ctFXkjJdOg2MF+HBMsani+FTBDLO81zlWIPGZn4L CISLo1n9wmab8mDqW+c2IBjisamk6KOMyjThF1e0KAfp30H87DooUc2nQ2QTcakI+26q AjahZEgD71M9rkqhPcYPC0JdMfD6LKWSVOAUbHRBJOSnT/6lo0d6lDDYfXy9k0jOsFUi lYgBJ0Fy/StvYHz5ZWVLyToNhhWMxKeE1mx0Xprt+hjxnwO7/ZD5N+DCGNUgNvrn0T9R jq90M4AT33MX03MV3x5qzJvtuxlOAUlJsaYROyJdz6oU2l6ry8GONMOrm5iZF4tIFLJ5 NA== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0m372-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:20 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0sqP013487 for ; Thu, 5 May 2022 17:11:20 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fruj4xq8q-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:19 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Cl8A5H0f06TVY2kP7VxRjfHyraPcZ3vA8cs2xf6TBLUyoYKUGkrPUrJlh3lcnHOKPx5Wt84NqlQ/ES4j964Uoufqf9ys5Rvk+8LIycLjocYMX16RPVvjsuLDxc0puIcHQCoxcW7Y+c6QZALzLBd91L/oEykXkgNR9JCnWApgXmD1hlieB6RIrY4z+QGd+UA6OyQ/7EarTIGgAuaaPHIpJaFXBmfmufWa1ddMfn8gcA785tAcNVlPdVZ9PT63wOrRqDIA+p5TvbQJlo63yb8//OBTlJAwh7XAycx7pAxC3redYK3u/D05KWgpjuMQ9mYT5EdhwE4emSVQ0hZldHby/Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=VtsDZ9Hg7Rrm/a6/XVUff29pbC98CuZpy222DxzXrSM=; b=aoc4fhhbIXaOlWOAbe5F6XXtudhnBsPvGhW0wwtTOocaJOGcJaa4L1KUQZAoDowpDcR3njPCwAdXOr7fAyIwCfdrkWi5EdYhEW3I5XILsX77utIEym/v66bODDMfDXrYPiXpSuRN0OALITG+q5DPCwWVWePPwYEk1NtHndvQPLVYJFjikjFmIrfKxy+1o1oXhx8m+f09+NEH3MKW/WGXe8Z+IAsQRCSbVplO4A9hdyK/8Ro7daXFI7cbCos36jGBxb98ubCOntqXALLusLImC0Pph5dv78RMTxuzEWGRxBhZg2zqwU/ZiUvptWobDuNCYzH0grxEE841doOcvQ69wQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VtsDZ9Hg7Rrm/a6/XVUff29pbC98CuZpy222DxzXrSM=; b=hsBS4lclsK/fLiVqCk9sTfsBGjy0tJy06k7TNdZ7/KU/tblvYwaG49n3M34wl9aaLDfUAO8cCJzWVUvRjYoOmhDty3ajmKQJUkmSAL7lLDen/TusvupEjVvdvNiHMRBL+jTNKZDVZZ4CeftgxsB2cT3rds9qVFcfK2Z2yAK2qOg= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:18 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:18 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 05/23] vfio-user: add device IO ops vector Date: Thu, 5 May 2022 10:19:48 -0700 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b3ebf06e-bd38-4717-8d41-08da2eba442e X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ZkjE4W4cSTk3AJ+rnPnmWaXq3urnom84U5JWbZHYQGGUKa+5+/JQCjPXyir0JdYKKg4mwgTOImRnSazURiZkdWK5hSOgLxSqo5KVQLW674oF4W8bfocFY4dzQ/2+VPrBmcGtLyR0Tv6LAeB9ql7LaXHbB/3dR1RXYHBT51Th2r9lMuA/Yt3Pz1tY8N5/sHjUEoXls9QT6ZpM/UKmVQX1+onhlKBSRdGFFyeSHtFs+qTIheaEnIIHCGwiga5RGdY9yYacvtWZu37g8UHWu2bTstTuFk6A2jesGwqimsN6MNnQZqeO0ym5Tuh4qECMwJJ0BOI9jUBR27efNw9r5Ui8HgjVKo9c3IvOJKcC2lHK3n/CzjiJyRdOK1X1Auzo4slHFB7vu42jtAALpf6O3EYKHOPEd8Bzcl3zordo8CN0i7mTS0HtfdsiO0JQ+zbqG+TeJ/CDhaFXqnKhprn6znoBTrxAqXnSacsKHIK4QeSzj+qDDFw8AINd5H6kMOCt+g3uesGC7a+Jyl9eAOICoO8/4ICdpjmMJYo4bx8tfPdUOc/3H85Ohel5mcR3LnIe5sBRcVl7M4We13qgzG+Sdw3w5JjuVgfMxcJfHSNfzrfc9ghMi2f16xtOkGimget0SxD30gGHmjCG1Mnn2vI4jImNoR7y8mJYGgavi2rM/U6bDCAKUngoqHA+lCzSYsYDH7BTKcK/bBHsf09zJuUVi+XSwQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(30864003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: qkFwkz20Ba54X5A0yDoTT1OHUdYCZG+w8pvHt0Pg71ByqLE06WTLtxHk4IPyPJYC+5yit7hY5FaUtvqhd7s/V38kRp9UWmRYmJ+Th7VUv7YlbF7aZPXOo+1j3fmNk1zBWRgK/o1zouk7lH5hsKtcW1HhKov9fX6rzuL+p6h+jv79oSLLYZb8ukGjpvMVhJVDeqSrILOETQLjyA8O3M0mXNQd82tVMc/2QDG5UAI1Ehu8aX1RM4rxRYMViaPovo3PcO/6ajRH7FpvBpt9cEBukuz6q/YnH9JAHe0SekJopODBamV63t+BMIhuaeYC19P8rRvOxnBVrr+0w9IiMypL4qQCEXcoQT8twF/Zq0mKs6VBPtcTTv3XE0zk8JQ5n7lXhtQLdutjvk/uJaDwBG9nlIQvIGv7R/Bh3Xrd+gLsumV/oHezRTDGGIk77dYZCkUa5ZB0rBHrnvwX9jTFVaXAQgnun4qFucLq1JLIXdS0rXMqPOa36jju0oDmw7AP04BDcFnXUFluPOikaaf/iC15PBpehOyNYLOfL9NiL12XkgGta/YULobHEmuEJZet6Zp6xbUXmyhDzeesUvagopXKK2thbTOV+MfBly/oHOeUJc1LPMyPHuElVCcCIi7GGhNi7gltGUx9m0YP2fb5quSz7At9teIcNiDJJevCpi+UpHZGInzKJrZWBjUBrOQK3cQgYhdLyPeTAcyQeni2y/9nLIlC26dEUoARVNegTNGIx/bO4QLSh1S5RDOQptecb3T4J9ch/IVhcxTY9IRvR5t4lIvwKYju+k9PTERKupnOSiQ9knGz2NaE4nCT8ZseCK7kiCUQRlWE1ncQlHq1TwUDqqmTS5S0wAXqa6YNZmAWj8ljG4NDFhLz/Mkg/m1jd3yZhRocKWUD+hZWbpY486UqRd/rSo1sLiPr7Lv94W2uOBqT/SLKs0wZBxYAxydylqztsE75832Mc3ZQsVTaab6gnHs6fgxjGxUT3vOSBZvJ6ngaEa5pmGC7MxFhhveQMDOj8nziKYW4X8dXVu0ZbxVzH4fPhoFfS5lIRlfBSae9xsUOO0L0Zs0dPmI2R7B3mZRCzDJ0MAV9NRvmdJOblFKIMjvXpzzGZlVKmWUE1zA9+pxc8OyVFqM3draNy9aLNO5tbl1sYU5GS9FRaAXTfXWx0Yvih4JkkFgyP1UwFFw/ZjqK5DswxdLmtdJxqUXj1jSOL2poDEd+2CknYB2GzP+tqJbOtCCxb8O9KbQpzIoMt3DXZ24TT4ykoaelv4at0U+5C9WDi+nthS2fKhqd/S5nQByNofdOkwb1KfYUUf+0/lZ0BX48q616Zya0SbXGpiOvIB1fxMot/zQt/6cd6Xr+D14WQK1z67asBSqlJ1YhF5dT6/dtZNamg/TbiYsilTi5nuMeqILV7aStQGXrRKP9maIGaa1PPvYcGOoKa3mADdMS9sYkyuRoSeYX6uz53UDLA/vD7LIUC6CruoWQTEUICNvPFdjHb77wwVg/wckaxvt46GE8UuF6wBKkuNJ2OrlhD6HVedPLShIPqbVOCLbJOOk5v77giRsy8i1y0Q3dhxUrbGdNzoNaCnq/ybNk7iSDzgAn1iOaRx6NUoPabOHGNWg03FPsbEM2CZcOGxe7uRvwZWg2bLZyZAlh+vg6W/KzPDTddCEjSY80B+IUeWykBkwyj8GzAIMMQNdN2zugytIX/PykIlNMQPThfnD3HRDi9Pk4sCNwJywnQhCury8sEzH3Taj5eu/x9c21bVsfUlE= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b3ebf06e-bd38-4717-8d41-08da2eba442e X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:17.0761 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: CQplmhRA0+N1mDPl+NiyfXtyPiVuq66JcO9u+gMfaY58zHZPIghDEx+y/r/f9i5bxr6c3qhOifD+7tds/nFIA3indjSyFpbr+jR/Vu+zznM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: pVB6ZEYZxYKo-VFZwRja1O5WYX4CJIKU X-Proofpoint-GUID: pVB6ZEYZxYKo-VFZwRja1O5WYX4CJIKU Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Used for communication with VFIO driver (prep work for vfio-user, which will communicate over a socket) Signed-off-by: John G Johnson --- include/hw/vfio/vfio-common.h | 27 ++++++++ hw/vfio/common.c | 107 +++++++++++++++++++++++++++----- hw/vfio/pci.c | 140 ++++++++++++++++++++++++++---------------- 3 files changed, 206 insertions(+), 68 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 1a032f4..826cd98 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -124,6 +124,7 @@ typedef struct VFIOHostDMAWindow { } VFIOHostDMAWindow; typedef struct VFIODeviceOps VFIODeviceOps; +typedef struct VFIODevIO VFIODevIO; typedef struct VFIODevice { QLIST_ENTRY(VFIODevice) next; @@ -139,6 +140,7 @@ typedef struct VFIODevice { bool ram_block_discard_allowed; bool enable_migration; VFIODeviceOps *ops; + VFIODevIO *io_ops; unsigned int num_irqs; unsigned int num_regions; unsigned int flags; @@ -165,6 +167,30 @@ struct VFIODeviceOps { * through ioctl() to the kernel VFIO driver, but vfio-user * can use a socket to a remote process. */ +struct VFIODevIO { + int (*get_info)(VFIODevice *vdev, struct vfio_device_info *info); + int (*get_region_info)(VFIODevice *vdev, + struct vfio_region_info *info); + int (*get_irq_info)(VFIODevice *vdev, struct vfio_irq_info *irq); + int (*set_irqs)(VFIODevice *vdev, struct vfio_irq_set *irqs); + int (*region_read)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, + void *data); + int (*region_write)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, + void *data); +}; + +#define VDEV_GET_INFO(vdev, info) \ + ((vdev)->io_ops->get_info((vdev), (info))) +#define VDEV_GET_REGION_INFO(vdev, info) \ + ((vdev)->io_ops->get_region_info((vdev), (info))) +#define VDEV_GET_IRQ_INFO(vdev, irq) \ + ((vdev)->io_ops->get_irq_info((vdev), (irq))) +#define VDEV_SET_IRQS(vdev, irqs) \ + ((vdev)->io_ops->set_irqs((vdev), (irqs))) +#define VDEV_REGION_READ(vdev, nr, off, size, data) \ + ((vdev)->io_ops->region_read((vdev), (nr), (off), (size), (data))) +#define VDEV_REGION_WRITE(vdev, nr, off, size, data) \ + ((vdev)->io_ops->region_write((vdev), (nr), (off), (size), (data))) struct VFIOContIO { int (*dma_map)(VFIOContainer *container, @@ -184,6 +210,7 @@ struct VFIOContIO { #define CONT_DIRTY_BITMAP(cont, bitmap, range) \ ((cont)->io_ops->dirty_bitmap((cont), (bitmap), (range))) +extern VFIODevIO vfio_dev_io_ioctl; extern VFIOContIO vfio_cont_io_ioctl; #endif /* CONFIG_LINUX */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index d9290f3..0616169 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -70,7 +70,7 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + VDEV_SET_IRQS(vbasedev, &irq_set); } void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) @@ -83,7 +83,7 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + VDEV_SET_IRQS(vbasedev, &irq_set); } void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) @@ -96,7 +96,7 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + VDEV_SET_IRQS(vbasedev, &irq_set); } static inline const char *action_to_str(int action) @@ -177,9 +177,7 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; - if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) { - ret = -errno; - } + ret = VDEV_SET_IRQS(vbasedev, irq_set); g_free(irq_set); if (!ret) { @@ -214,6 +212,7 @@ void vfio_region_write(void *opaque, hwaddr addr, uint32_t dword; uint64_t qword; } buf; + int ret; switch (size) { case 1: @@ -233,13 +232,15 @@ void vfio_region_write(void *opaque, hwaddr addr, break; } - if (pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { + ret = VDEV_REGION_WRITE(vbasedev, region->nr, addr, size, &buf); + if (ret != size) { + const char *err = ret < 0 ? strerror(-ret) : "short write"; + error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64 - ",%d) failed: %m", + ",%d) failed: %s", __func__, vbasedev->name, region->nr, - addr, data, size); + addr, data, size, err); } - trace_vfio_region_write(vbasedev->name, region->nr, addr, data, size); /* @@ -265,13 +266,18 @@ uint64_t vfio_region_read(void *opaque, uint64_t qword; } buf; uint64_t data = 0; + int ret; + + ret = VDEV_REGION_READ(vbasedev, region->nr, addr, size, &buf); + if (ret != size) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; - if (pread(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { - error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %m", + error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %s", __func__, vbasedev->name, region->nr, - addr, size); + addr, size, err); return (uint64_t)-1; } + switch (size) { case 1: data = buf.byte; @@ -2418,6 +2424,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) { size_t argsz = sizeof(struct vfio_region_info); + int ret; /* create region cache */ if (vbasedev->regions == NULL) { @@ -2436,10 +2443,11 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, retry: (*info)->argsz = argsz; - if (ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info)) { + ret = VDEV_GET_REGION_INFO(vbasedev, *info); + if (ret != 0) { g_free(*info); *info = NULL; - return -errno; + return ret; } if ((*info)->argsz > argsz) { @@ -2600,6 +2608,75 @@ int vfio_eeh_as_op(AddressSpace *as, uint32_t op) * Traditional ioctl() based io_ops */ +static int vfio_io_get_info(VFIODevice *vbasedev, struct vfio_device_info *info) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_INFO, info); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_get_region_info(VFIODevice *vbasedev, + struct vfio_region_info *info) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, info); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_get_irq_info(VFIODevice *vbasedev, + struct vfio_irq_info *info) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_IRQ_INFO, info); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irqs) +{ + int ret; + + ret = ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irqs); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_region_read(VFIODevice *vbasedev, uint8_t index, off_t off, + uint32_t size, void *data) +{ + struct vfio_region_info *info = vbasedev->regions[index]; + int ret; + + ret = pread(vbasedev->fd, data, size, info->offset + off); + + return ret < 0 ? -errno : ret; +} + +static int vfio_io_region_write(VFIODevice *vbasedev, uint8_t index, off_t off, + uint32_t size, void *data) +{ + struct vfio_region_info *info = vbasedev->regions[index]; + int ret; + + ret = pwrite(vbasedev->fd, data, size, info->offset + off); + + return ret < 0 ? -errno : ret; +} + +VFIODevIO vfio_dev_io_ioctl = { + .get_info = vfio_io_get_info, + .get_region_info = vfio_io_get_region_info, + .get_irq_info = vfio_io_get_irq_info, + .set_irqs = vfio_io_set_irqs, + .region_read = vfio_io_region_read, + .region_write = vfio_io_region_write, +}; + static int vfio_io_dma_map(VFIOContainer *container, struct vfio_iommu_type1_dma_map *map) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 35b6551..4524342 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -43,6 +43,14 @@ #include "migration/blocker.h" #include "migration/qemu-file.h" +/* convenience macros for PCI config space */ +#define VDEV_CONFIG_READ(vbasedev, off, size, data) \ + VDEV_REGION_READ((vbasedev), VFIO_PCI_CONFIG_REGION_INDEX, (off), \ + (size), (data)) +#define VDEV_CONFIG_WRITE(vbasedev, off, size, data) \ + VDEV_REGION_WRITE((vbasedev), VFIO_PCI_CONFIG_REGION_INDEX, (off), \ + (size), (data)) + #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" static void vfio_disable_interrupts(VFIOPCIDevice *vdev); @@ -402,7 +410,7 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + ret = VDEV_SET_IRQS(&vdev->vbasedev, irq_set); g_free(irq_set); @@ -775,14 +783,16 @@ static void vfio_update_msi(VFIOPCIDevice *vdev) static void vfio_pci_load_rom(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; struct vfio_region_info *reg_info; uint64_t size; off_t off = 0; ssize_t bytes; + int ret; - if (vfio_get_region_info(&vdev->vbasedev, - VFIO_PCI_ROM_REGION_INDEX, ®_info)) { - error_report("vfio: Error getting ROM info: %m"); + ret = vfio_get_region_info(vbasedev, VFIO_PCI_ROM_REGION_INDEX, ®_info); + if (ret < 0) { + error_report("vfio: Error getting ROM info: %s", strerror(-ret)); return; } @@ -807,18 +817,19 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev) memset(vdev->rom, 0xff, size); while (size) { - bytes = pread(vdev->vbasedev.fd, vdev->rom + off, - size, vdev->rom_offset + off); + bytes = VDEV_REGION_READ(vbasedev, VFIO_PCI_ROM_REGION_INDEX, off, + size, vdev->rom + off); if (bytes == 0) { break; } else if (bytes > 0) { off += bytes; size -= bytes; } else { - if (errno == EINTR || errno == EAGAIN) { + if (bytes == -EINTR || bytes == -EAGAIN) { continue; } - error_report("vfio: Error reading device ROM: %m"); + error_report("vfio: Error reading device ROM: %s", + strerror(-bytes)); break; } } @@ -906,11 +917,10 @@ static const MemoryRegionOps vfio_rom_ops = { static void vfio_pci_size_rom(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; uint32_t orig, size = cpu_to_le32((uint32_t)PCI_ROM_ADDRESS_MASK); - off_t offset = vdev->config_offset + PCI_ROM_ADDRESS; DeviceState *dev = DEVICE(vdev); char *name; - int fd = vdev->vbasedev.fd; if (vdev->pdev.romfile || !vdev->pdev.rom_bar) { /* Since pci handles romfile, just print a message and return */ @@ -927,11 +937,12 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev) * Use the same size ROM BAR as the physical device. The contents * will get filled in later when the guest tries to read it. */ - if (pread(fd, &orig, 4, offset) != 4 || - pwrite(fd, &size, 4, offset) != 4 || - pread(fd, &size, 4, offset) != 4 || - pwrite(fd, &orig, 4, offset) != 4) { - error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); + if (VDEV_CONFIG_READ(vbasedev, PCI_ROM_ADDRESS, 4, &orig) != 4 || + VDEV_CONFIG_WRITE(vbasedev, PCI_ROM_ADDRESS, 4, &size) != 4 || + VDEV_CONFIG_READ(vbasedev, PCI_ROM_ADDRESS, 4, &size) != 4 || + VDEV_CONFIG_WRITE(vbasedev, PCI_ROM_ADDRESS, 4, &orig) != 4) { + + error_report("%s(%s) ROM access failed", __func__, vbasedev->name); return; } @@ -1111,6 +1122,7 @@ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; uint32_t emu_bits = 0, emu_val = 0, phys_val = 0, val; memcpy(&emu_bits, vdev->emulated_config_bits + addr, len); @@ -1123,12 +1135,13 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) if (~emu_bits & (0xffffffffU >> (32 - len * 8))) { ssize_t ret; - ret = pread(vdev->vbasedev.fd, &phys_val, len, - vdev->config_offset + addr); + ret = VDEV_CONFIG_READ(vbasedev, addr, len, &phys_val); if (ret != len) { - error_report("%s(%s, 0x%x, 0x%x) failed: %m", - __func__, vdev->vbasedev.name, addr, len); - return -errno; + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_report("%s(%s, 0x%x, 0x%x) failed: %s", + __func__, vbasedev->name, addr, len, err); + return -1; } phys_val = le32_to_cpu(phys_val); } @@ -1144,15 +1157,19 @@ void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; uint32_t val_le = cpu_to_le32(val); + int ret; trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); /* Write everything to VFIO, let it filter out what we can't write */ - if (pwrite(vdev->vbasedev.fd, &val_le, len, vdev->config_offset + addr) - != len) { - error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %m", - __func__, vdev->vbasedev.name, addr, val, len); + ret = VDEV_CONFIG_WRITE(vbasedev, addr, len, &val_le); + if (ret != len) { + const char *err = ret < 0 ? strerror(-ret) : "short write"; + + error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %s", + __func__, vbasedev->name, addr, val, len, err); } /* MSI/MSI-X Enabling/Disabling */ @@ -1240,10 +1257,13 @@ static int vfio_msi_setup(VFIOPCIDevice *vdev, int pos, Error **errp) int ret, entries; Error *err = NULL; - if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS"); - return -errno; + ret = VDEV_CONFIG_READ(&vdev->vbasedev, pos + PCI_CAP_FLAGS, + sizeof(ctrl), &ctrl); + if (ret != sizeof(ctrl)) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed reading MSI PCI_CAP_FLAGS %s", err); + return ret; } ctrl = le16_to_cpu(ctrl); @@ -1445,33 +1465,39 @@ static void vfio_pci_relocate_msix(VFIOPCIDevice *vdev, Error **errp) */ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) { + VFIODevice *vbasedev = &vdev->vbasedev; uint8_t pos; uint16_t ctrl; uint32_t table, pba; - int fd = vdev->vbasedev.fd; VFIOMSIXInfo *msix; + int ret; pos = pci_find_capability(&vdev->pdev, PCI_CAP_ID_MSIX); if (!pos) { return; } - if (pread(fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS"); - return; + ret = VDEV_CONFIG_READ(vbasedev, pos + PCI_MSIX_FLAGS, + sizeof(ctrl), &ctrl); + if (ret != sizeof(ctrl)) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed to read PCI MSIX FLAGS %s", err); } - if (pread(fd, &table, sizeof(table), - vdev->config_offset + pos + PCI_MSIX_TABLE) != sizeof(table)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE"); - return; + ret = VDEV_CONFIG_READ(vbasedev, pos + PCI_MSIX_TABLE, + sizeof(table), &table); + if (ret != sizeof(table)) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed to read PCI MSIX TABLE %s", err); } - if (pread(fd, &pba, sizeof(pba), - vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX PBA"); - return; + ret = VDEV_CONFIG_READ(vbasedev, pos + PCI_MSIX_PBA, sizeof(pba), &pba); + if (ret != sizeof(pba)) { + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_setg(errp, "failed to read PCI MSIX PBA %s", err); } ctrl = le16_to_cpu(ctrl); @@ -1609,7 +1635,6 @@ static void vfio_mmap_set_enabled(VFIOPCIDevice *vdev, bool enabled) static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) { VFIOBAR *bar = &vdev->bars[nr]; - uint32_t pci_bar; int ret; @@ -1619,10 +1644,12 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) } /* Determine what type of BAR this is for registration */ - ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar), - vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr)); + ret = VDEV_CONFIG_READ(&vdev->vbasedev, PCI_BASE_ADDRESS_0 + (4 * nr), + sizeof(pci_bar), &pci_bar); if (ret != sizeof(pci_bar)) { - error_report("vfio: Failed to read BAR %d (%m)", nr); + const char *err = ret < 0 ? strerror(-ret) : "short read"; + + error_report("vfio: Failed to read BAR %d (%s)", nr, err); return; } @@ -2170,8 +2197,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev) static void vfio_pci_post_reset(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; Error *err = NULL; - int nr; + int ret, nr; vfio_intx_enable(vdev, &err); if (err) { @@ -2179,13 +2207,16 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev) } for (nr = 0; nr < PCI_NUM_REGIONS - 1; ++nr) { - off_t addr = vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr); + off_t addr = PCI_BASE_ADDRESS_0 + (4 * nr); uint32_t val = 0; uint32_t len = sizeof(val); - if (pwrite(vdev->vbasedev.fd, &val, len, addr) != len) { - error_report("%s(%s) reset bar %d failed: %m", __func__, - vdev->vbasedev.name, nr); + ret = VDEV_CONFIG_WRITE(vbasedev, addr, len, &val); + if (ret != len) { + const char *err = ret < 0 ? strerror(-ret) : "short write"; + + error_report("%s(%s) reset bar %d failed: %s", __func__, + vbasedev->name, nr, err); } } @@ -2632,7 +2663,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index = VFIO_PCI_ERR_IRQ_INDEX; - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + ret = VDEV_GET_IRQ_INFO(vbasedev, &irq_info); if (ret) { /* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure(strerror(errno)); @@ -2751,8 +2782,10 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) return; } - if (ioctl(vdev->vbasedev.fd, - VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) { + if (VDEV_GET_IRQ_INFO(&vdev->vbasedev, &irq_info) < 0) { + return; + } + if (irq_info.count < 1) { return; } @@ -2830,6 +2863,7 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) vdev->vbasedev.ops = &vfio_pci_ops; vdev->vbasedev.type = VFIO_DEVICE_TYPE_PCI; vdev->vbasedev.dev = DEVICE(vdev); + vdev->vbasedev.io_ops = &vfio_dev_io_ioctl; tmp = g_strdup_printf("%s/iommu_group", vdev->vbasedev.sysfsdev); len = readlink(tmp, group_path, sizeof(group_path)); From patchwork Thu May 5 17:19:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D26CC433F5 for ; Thu, 5 May 2022 17:58:54 +0000 (UTC) Received: from localhost ([::1]:53888 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfkb-00014r-7n for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:58:53 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53126) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf1f-0007Ma-LD for qemu-devel@nongnu.org; Thu, 05 May 2022 13:12:27 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:45814) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf1d-0002BV-89 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:12:27 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FpOfi029440 for ; Thu, 5 May 2022 17:11:21 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=z4YQmOEc2RDGScN538VMp7No3l4BG8BPB6iLj44+8Zc=; b=izqfE6BKKQcagiepMfjCEhavMlNrTOpuEmn9mNRfwHzIF+HkuegVPEXB7v278oqXXK8j V7u5JzF24LnE5X6LWsnmZCMtLKwfXEe0yLG7HzyPDId0S+Ye7/JRYOYRDnxR7NCGLaSV FDypuMc2U212JqtOcmzZaG/zCRofkTVZ4tyWB4cTHAV42l9nEVoQjFc/CzWFIMwo8bKS LUnWgEzQhGehtYbFJMyesn4e3xzBZREM7TZqs1mqn4my9jZLyKzsCY6LaQi0FRn7qrt3 tSMl1OjDNVVCCQ5e7gV9BkcnpQqG1qPZUJVb4fSE1JsKFL8e6z6g7/lVgDEPFWcZX4hu Yg== Received: from phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta01.appoci.oracle.com [138.1.114.2]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0m373-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:21 +0000 Received: from pps.filterd (phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0sqQ013487 for ; Thu, 5 May 2022 17:11:20 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by phxpaimrmta01.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fruj4xq8q-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:20 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=euxRIPtcX4mutgFrSTpyqds/fgilEZ7r154yk5rIUqd2KQtkI818+oGk9ElNacXnVvkIKWt0aPD6lK5QJkBEuzow37Yeq66/FnGHycLIPcwkmuk/LJIzdAHBpBVb9vuOSSdx7tdibuUjOEHjbiF/4m2839tpxr2CCFgTTwvk1lHJ/1J2bE1qRFfFe8k9W9PbmqjCuWffbGlZNGnM3rS5ILGhX7/6S524Da2OTniEktzAhik2h+b9SRPszHOGxIN5Rq7r4X51kIV2bvQwNORrx5ec1yDYtYvmgFjpX6gCkBjSboqxSUvGwxvI2Me8EimfrWn1icMDsb8Ha5SY4g7UcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=z4YQmOEc2RDGScN538VMp7No3l4BG8BPB6iLj44+8Zc=; b=ldTDMUH0Tw92mP/WBOMY+SYhiVzpd8XJfFlGWGUbRXQT3Am786UR6JxjBqtpmx8AiZ3SXGGaqy5BicjqxXPMO8qlY/BeCOMUxtNkr4HRLSj23dWaXzOZafdsJjgk+BE15kQyXryL/CMBYDwLPAWZ4/CT6XFXUmfAzoFfS/ytz1k6FVlj6SzB7hOpilNQolcs6XJlFxaPab6KQ9/5l/0nNmu+IZ19z+OIj1Jx4SKMp/AYRq8Hoj61pu6OabYQ6xO7qHIxu71GlnYe1J6s1uGbQ6LjBzSQlsQl8BbmK+e8ZcXwe0JlkOrRI9s+4qt9LT3ubwv8ikEuvNmFXtX/DaV56g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=z4YQmOEc2RDGScN538VMp7No3l4BG8BPB6iLj44+8Zc=; b=UwF5I9vG+aukumeXsdNEzSF86eb3LmgOrisjT7YvpY6ZVsmyA1Wa05bem1osFoTLTEV4DygowEkcDaKhOE/bI/WdxibfTU82g7dPDo6DfUWHCiyNqcEf/sTXhFy5vPvYY/wmU6XDnBK8zTUExjvTadDY3ntriDU2LmOqlyYocVI= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:18 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:18 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 06/23] vfio-user: Define type vfio_user_pci_dev_info Date: Thu, 5 May 2022 10:19:49 -0700 Message-Id: <0bbbb66ee30dd894d148cb325f221ab65d78d0c7.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d2bf6145-5165-4af1-626b-08da2eba4465 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0LTcj3BDWwZzge+mB+vgbeqNmFltcvnTh8qlws2deYw+QkxtZdXuLEzr1GfdmIBLp8flGF9xSigi2jQ5XcM9TbctFV8OkiRUhi1vl8QWJwdDxla5dL6tlrmvIVmLT2N1jGxCtLhjDnB045ZCcboIbTiGRU357IPeYueyNeQ02S8o8BSo5cl/9Xvt3rrGyEA6zuJY3IfcnhoT/+Syw631QkFx2ZuKuykA10HIUI0UZPaYT3UAofXEZp1QNbVO/V9DNnSwtDQ9Z0j4lZ3onRlW1ubQMMhohxKdHc/+ilZY4jupJV1rSM42EK+HYiQpy8XfB8MqrsiImAMH22reeia2ubEodoXEm4tBHkKWaVdHASuknpC9tBJ4EIiLl5lRCHxPsy791881/41d06pidTwalT3se2dSwxilpZPeldlLZZCcFYp6o5BtlC94rySnAaiJth0O9gFmiqyOR4JOmSrEyxhVqIzHsVXqDromYxo3/4B7wPZJmCrSlObBMc7AD4hDKJgtWuv2V2LkEHGADandec3XsDriMlI0rextVGc8S5pV5yyOQVrrD1mfeyiaPY+ppPReRBaw2+Fpp2TFnHlZtCqp16yeQI6LPJuNKRqw7nqCDm9BlyDwimcS4NgJLRdq9UVmy8Xtmn3FygsVCc+jMdg9C9eevjadrtqfYe9NUxSo49PE6al9TgtuvvlLgLix8AQdl06bmLkgtEG5zVMI8w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(36756003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sM5npD9NOmP9LTye7KjapEWrN3zxpGW24+JKTqy9NiGLaTjxdpTDdPgB1piSNpHZcl8XcB2oRwHDtqTWJtfIqnQ4II0lPa0mcEdH9a0YTpkOeP7LM0SrhsIsbLYu1JNwQnTseDA+OO2P6a+eLamXmxcVjMpd0N7Y/ywNbKxza85d5+/exmuMwoOXy72X0O4AWOOSTG0NDXFvzAc1hNuF18FCwoX5xCiTAt6LYJxL/GiH5A2l1vpuQhnenJb9vZfk2obXIho3DV8/CbQyVo3QNJzKGJh/QyOKzDdurxVhY0LAqoJ0LfkOaMaGa6u3g4S+RpK8c3o0VU+FhZNo0MhTOqlGdO+S4a7SzTnsJi+pWcVi5WkaESgOEryZ8Qt/UZRYmBsl97omI18ap6yqUAWO39PAxxOvOAoauULiqeHmZdSlvHRbWI4kmQQdVoM5P9uwwTO8G4EaoxFkHmWkOI24S5SstCPrySXAMzgHcz50YzTR2DH0SS+qtUC/s9WTArknBXIZluLuZoSKOKVAzidZoVATNCtMk0XO4x/oGJI5VyA8TNXPKWwna5ga5xZTUzMIDHPCLqOUnQDiTWt6+evRNEKSScw2F6dRmilhumFPNx5fraZPTBdjl7qPjIYthIjHJFM2XGVseLcWWD4eRN0QrkZlPNRNgswst4p/zAZ9jjwTp3Rr0+/Z2Uakdmouq2uGvYbzBxw6uOmrVpjRBgFhNExNTZNE4nw06RGmvoAn51/CWiq/hCyhozicUnM+N1BcJ1B/MbKLD3FpgKGTf5o9uRj42CNm2TkrzWqpL4gG8IVF6ew8FQNSlStA/FA2UfsioRlUCRn2+3UHy8pVgOnC+50ylHy6PLe82I915RFSWyKyYAI2DvQqGytsnaraQCX3hwtDMw8v/8gUt04yJ4+wpvKcMwfmkQHfKLORLzh6mcSgZaLybf4Kl/NposNNCvBKjlca0Yz74cySL6qFidjhdOdOQhG2KR9XoN+x38MH3ljNIsea/6Hcem+2examejxzmo6Yvrz0ehuBpaY7niOdqDHgqo70mjHoimtJBFd1nBom8EsWAfvbDkdvALqb9tktSZIgEsIpHxE7J01+182BquHpTU56L8njfcOjtC5keffAvMM8JFKXMi09awbCguWMCCpj+/Ks8DW2oX7lszRGFYlsyosGKhgmBLrwFFGjObAlJuoLCMi16h4GBv1OOCzL3AhwtR8bBzkg2p9dtPGuflZKQyJtDuiR92mCComa0mYOI6K5U68UHi6zvMDe6TdL9anbbuQY0sfpKtzvhsBPJlSMCJJyuOuURfRCJKJn5f5yjJImdp2LDMpUMmc7VWevW+fWXdpkWHNcJBy6MWNjBhqfkJMtOLy6tWeDUIyVwzlg+2XtFVZKyUSpRIUSbNOnw20SbqwdyiWUDMtX1U0cNKvFwe9FkypXsgfnKDNjsm9Z4bYXVnrmF6LFhBvTZ5pZrIsTmw1vCAQ6Qx9iS0fBvcKsiNnWaNEg6/Qk6iQb9Srwbr7k6mW84/PhxIZjaKYSqwlMmRt8k6QHrXLYlL6ig7I1ZQCyuevu/onhbHKiGh14I39zbooS6RfCNV8BiRJy2XnesJzycTWD1PsQbgaWh9CX9bKCw+tPsNN0h72u3fR72EDP/dPoc5YFi8u28vUhOfdygtGlKX88gDpgK3htvwf2hdj1+zjQKQyh1i/bPPPqd21xX0n6AncKepdoMkf2etXlVDF26JfcvtmqLyp0kLbF9kTOFECFzfHuDPoTAaY= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d2bf6145-5165-4af1-626b-08da2eba4465 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:17.4198 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1iOid6DzdFb7m+aI/KviOb+z7crr/PG00epXoXzcN5YmTq6COZziUP/LyL6MwSw/TkCGV+/JMgUfHukcGwu325pEABOLeZgx31jrP3Bmgeg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 phishscore=0 suspectscore=0 spamscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: zFeTcLC4qIZVELfVfth4LFDklaSzMhsX X-Proofpoint-GUID: zFeTcLC4qIZVELfVfth4LFDklaSzMhsX Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" New class for vfio-user with its class and instance constructors and destructors, and its pci ops. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 8 +++++ hw/vfio/common.c | 5 ++++ hw/vfio/pci.c | 90 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/Kconfig | 10 +++++++ 4 files changed, 113 insertions(+) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index bbc78aa..59e636c 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -187,6 +187,14 @@ struct VFIOKernPCIDevice { VFIOPCIDevice device; }; +#define TYPE_VFIO_USER_PCI "vfio-user-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) + +struct VFIOUserPCIDevice { + VFIOPCIDevice device; + char *sock_name; +}; + /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device) { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 0616169..da18fd5 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1742,6 +1742,11 @@ void vfio_reset_handler(void *opaque) QLIST_FOREACH(group, &vfio_group_list, next) { QLIST_FOREACH(vbasedev, &group->device_list, next) { if (vbasedev->dev->realized && vbasedev->needs_reset) { + if (vbasedev->ops->vfio_hot_reset_multi == NULL) { + error_printf("%s: No hot reset handler specified\n", + vbasedev->name); + continue; + } vbasedev->ops->vfio_hot_reset_multi(vbasedev); } } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 4524342..be8fe1d 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -19,6 +19,7 @@ */ #include "qemu/osdep.h" +#include CONFIG_DEVICES #include #include @@ -3377,3 +3378,92 @@ static void register_vfio_pci_dev_type(void) } type_init(register_vfio_pci_dev_type) + + +#ifdef CONFIG_VFIO_USER_PCI + +/* + * vfio-user routines. + */ + +/* + * Emulated devices don't use host hot reset + */ +static void vfio_user_compute_needs_reset(VFIODevice *vbasedev) +{ + vbasedev->needs_reset = false; +} + +static VFIODeviceOps vfio_user_pci_ops = { + .vfio_compute_needs_reset = vfio_user_compute_needs_reset, + .vfio_eoi = vfio_intx_eoi, + .vfio_get_object = vfio_pci_get_object, + .vfio_save_config = vfio_pci_save_config, + .vfio_load_config = vfio_pci_load_config, +}; + +static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) +{ + ERRP_GUARD(); + VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; + + /* + * TODO: make option parser understand SocketAddress + * and use that instead of having scalar options + * for each socket type. + */ + if (!udev->sock_name) { + error_setg(errp, "No socket specified"); + error_append_hint(errp, "Use -device vfio-user-pci,socket=\n"); + return; + } + + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); + vbasedev->dev = DEVICE(vdev); + vbasedev->fd = -1; + vbasedev->type = VFIO_DEVICE_TYPE_PCI; + vbasedev->ops = &vfio_user_pci_ops; + +} + +static void vfio_user_instance_finalize(Object *obj) +{ + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + + vfio_put_device(vdev); +} + +static Property vfio_user_pci_dev_properties[] = { + DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), + DEFINE_PROP_END_OF_LIST(), +}; + +static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + device_class_set_props(dc, vfio_user_pci_dev_properties); + dc->desc = "VFIO over socket PCI device assignment"; + pdc->realize = vfio_user_pci_realize; +} + +static const TypeInfo vfio_user_pci_dev_info = { + .name = TYPE_VFIO_USER_PCI, + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOUserPCIDevice), + .class_init = vfio_user_pci_dev_class_init, + .instance_init = vfio_instance_init, + .instance_finalize = vfio_user_instance_finalize, +}; + +static void register_vfio_user_dev_type(void) +{ + type_register_static(&vfio_user_pci_dev_info); +} + +type_init(register_vfio_user_dev_type) + +#endif /* VFIO_USER_PCI */ diff --git a/hw/vfio/Kconfig b/hw/vfio/Kconfig index 7cdba05..301894e 100644 --- a/hw/vfio/Kconfig +++ b/hw/vfio/Kconfig @@ -2,6 +2,10 @@ config VFIO bool depends on LINUX +config VFIO_USER + bool + depends on VFIO + config VFIO_PCI bool default y @@ -9,6 +13,12 @@ config VFIO_PCI select EDID depends on LINUX && PCI +config VFIO_USER_PCI + bool + default y + select VFIO_USER + depends on VFIO_PCI + config VFIO_CCW bool default y From patchwork Thu May 5 17:19:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839824 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 27996C433EF for ; Thu, 5 May 2022 17:29:46 +0000 (UTC) Received: from localhost ([::1]:60772 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfIP-0000bN-1g for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:29:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52592) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0l-00063N-Ce for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:6132) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0f-0002Bo-5Z for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:29 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245F0uXe026110 for ; Thu, 5 May 2022 17:11:22 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=ivSwuHYvoIcC7zPSkKbAaaoFSdZhWtUd40sxVSelbHo=; b=rweMy0y+nIo3l6RdP05U0oIu9YCUH3I4lUXprLqQ2RMFw0TY67Jg6IbvzYu2ZwuFHKxs uhyorFQAotOvifoPYL02Xdl9I9szNaSxfSWNvnbCDBmC5Cusi4+brquvDXIFidnvHRqS 9OkOtKSld80uvyGsDeyiHaBpOY/tJgI9iqnpOFpUuHr3yD8XmhyuzzHSVXzrVCD7KzSX CzOK/RkV7k1maKdhm+Ex5quU+XXn4H/AdZG40SV4gs+RepD0wQ14ibjJYmq0h/X8OnhW zaoL82SGnlRXo56vdQkkico3XmGrpy24BI3xGoLQ6QEsPfEbYTO5ZhqtNFjm3Mtn3rMJ 6w== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruhcbs64-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:22 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1PHX001942 for ; Thu, 5 May 2022 17:11:21 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a7amqq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:21 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=euOZ33V8JC05KzF7w8sImHBWY/OumeWXoeIil2YOJqLqC1J6NA1n2o8FIbeFPOX7uo28PVHoNIuwVm5Xyw0rIFDMkkYwJKcezVncD/Kj/UcKNjVOj/iXthe2PrQkFOKYyDhy4+eeJoXGPVdt3Pf9+g/Vi5KvZzp6GXEnYwj6NNqrzm3voGkGzHsJGTSgNgyvUcqHJlqHdnzHQxtthmPu9OXSfXi2fiUA/WwtFBuLkXZx74r29jxl1oWRAnLWTwaWoBovnYRq82VtVv2Xe8rQiNgictD8cf7NkUm28zdFIxhQLuOe1PQpUBhy+/eW/ZTrNQr8js6Mglv8yppcxBcAng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ivSwuHYvoIcC7zPSkKbAaaoFSdZhWtUd40sxVSelbHo=; b=V2uCOVG5yRV8tyiXn3cyDJ3xF5DZrxJLTHF0ybIVf2XPS/16D4AqPw0mgjeUVNISHq8PgwBlnzNldTj7UoV0c8hvcijsCjalqHvlAUUthpOOJzz732xbxJ6KNXz9+5BBdy/43BlOeAkO2HgWIhjBZYUMsyju+oMeUuk7aY4IqWpuy0E0NprCJ+putex8CPZwaAlDxs19Wse9mQYRk8tHjBtxD6DAQQx7SMEsuq27KLSFC1SmWl6mzV2kfhrOgBBPib8SBSYm0mdO9Vv70P/GEGlp7+BUJuElwiCMHWtjz6I1qkBLhub2FlQldmf3TbyyzsNade75duj90hVId1sWeQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ivSwuHYvoIcC7zPSkKbAaaoFSdZhWtUd40sxVSelbHo=; b=xBE74uQuwLqvb6YgC0ytfeCi0keR3RYwIra1RGPhgUl2cDiMPpM7mUbWRhjmphLVdOYuZ0fhjo6tsRfRPP7W2VWql0t44xaRYqCzzmxPB96LYSqVUsOlURgqSs7FKc4aB34T8wsX2r2Q+choQNyxCkHugld57vWrG5wnFyNiX74= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:19 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:19 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 07/23] vfio-user: connect vfio proxy to remote server Date: Thu, 5 May 2022 10:19:50 -0700 Message-Id: <90c382c8aa89c1ec14768102e8f2c5a96e63b039.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 798fa9b2-16b7-4286-99de-08da2eba449e X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +ZNre4xEAifsuZpu3NHmh34ImTVr26IGDaR96mvZLbCK23trTP3zVlswahn4nbglTLPophkzKPi8Lf+Nb+hg7vZlsHc8xz5oe2dbTwV2uVisKrbiKGubeKCTGwxwPdBqMS57M8OVtx6NyhDmAIYwotygoO8fmjNW/W8H/y8PNp/YerpN/XKdhpsPz5MvRlFCSgklDRHqnWoLS/+CQObyWbWBOMNChWjGqiAImnUutHNwVjxXMOHdYKlt2EBn4CJ3fWTjmmLI5ILVQTtZbD1+1BKWskqomcxZDGVaNT+3gchlSkK5OVJkWeBH+0cLOtuvbaTFyGSPRlXgVHpv9XmI5SjWDkC/cU1bxxl4Y/PMCDc39mUxJmjTDdi57ghSaBTM7NGswSnwfwiYiyDMf/36qGQb8xbDVNMT2JGRPLJUwt2rzZ3yaTsHAoe6W6ApdGE1rB4AJUOeKQrHdS0HQlP785Pa3KmsnwyJPRl3+yfTQ/A1b5nYZIH9+YyPLO7vUZ1EI9T8t18feE+OKhtdpheG82td00m/klO8t8Qg1GSJ1rZ9tjWu8kFVTLm49GQMUcFWUcECaqiZ5F87sBGiWcnXN1EZxFXfnoTilZPfbCZ9+fmvL3153xzxZVDkP7ywf++0dLjk5zSoXKCHBW7qKNT0StiPZ50fp8qjAYDNIL+vZU3cNhbzaYQIxXaxNeXLWpe+k/RTj2G/XTLRKiKFi8aedA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?ecQVeryy1Qf/73goCzX1lyjNcDkb?= =?utf-8?q?MhQkD575QpQ/hZLP9WBtJpSJbGr5YHl8gvGt2Xs0OpUzwFMHaJ7nnj7Mu1Ty+Zgom?= =?utf-8?q?ST+wPxs4YVNRogqd4827wHdVgN8ZaqhE5o5Q7/lnfNW3tRuzMm0+2MtbZMYCUdKvt?= =?utf-8?q?AT09/vDlcQVZ2G/xUqQzA99KVsVlrsMs96wKaIZvBWvEgE2z7D7CDP0lIfIv1qGyN?= =?utf-8?q?3Ba8ICdmnRbPcvXjvfwFWhVB+TNt0G9w7uxS5lS+jBUrJshxxDXfbTDNSBkjN4KEp?= =?utf-8?q?az07FrG8aKBHB+RqkS/yVay3/vwCs/NiT3LBUA2EQ452cTt5KGUQl101IP+ZOLgna?= =?utf-8?q?aBHwQkQeNhPjHL1htZ6yjOtHf1vcOnZ8ogihRFWiDeVqF3pGRtQnkj8q+QMqlJPXY?= =?utf-8?q?ZedtkbRhqPRYfKrXlFJ4GUTuggMkUFwyNvtFIgkQRnEY4Q/mrIVtUi0jBUiSm8vpu?= =?utf-8?q?qcEFUKO9V9w3UDaIxglKR+Fk0R4FdzFfAiF4zBIaT87SpqfY+LS70y+65IBfkC8cV?= =?utf-8?q?kqWCr/Jgm1O30MickxbWcyVCEKDepjLTRYwplx944SAS2zQgoZ/+Ij9ykk6ThvZc5?= =?utf-8?q?GEY9rEc8io5OPusU0P1/Vly3wWe7c+dlVrwRcPx9gSIeaV95O13/40jgmm4dgSS43?= =?utf-8?q?BMQOXUakFnejke2dBvnZ+GV2pz8L0NOj3gH4Y9D8+GgXHd1Zv/yNi28rjof92LNG9?= =?utf-8?q?B5XPmk5ofZyUlFvpHJpBVjTUAXmyDl+uM6TI7EI4tp0FBE98w4bhd5GeCCBR98tqm?= =?utf-8?q?G5p/kqNDuvyPRzbLdW1zY470AnxsGlPgPyPe37ptfTe7P926dme4crPkKk77raVaA?= =?utf-8?q?dL32k7d6gJAWAZMdDwV0tVKZGCg9vclRGWHe0df35DuyLTTr59G99TWaylrym2EgR?= =?utf-8?q?aW+xHyzuc5dRifgSbW+FUic3h1tNyn/e4BJNpH00UQFyCTa23vEvBtTYwwXfbqtiv?= =?utf-8?q?2+ZACmxDVqxtWBODdOHYW9TPfn7FqTle365PirwZYBOFn0iPEhrqal9YsHJDhfmBJ?= =?utf-8?q?PhJu8RdZUqZtK7gxNnYifkVo7Jd+qVDNgKwOurz6i+aCP9SCiQ1oZX7ki/EgDOpbJ?= =?utf-8?q?78hV3gbZI9zS6bovdLbfFMtYoRXe7fzihPnhcwh115qWiHwQPVza6hMcot5/aCQzY?= =?utf-8?q?feSvxWFNwIriu8vqhkHIu2MEG5dFUYZCmW9LA+ITo+NMjLUlY9W0DwVvIYsWa/6Ke?= =?utf-8?q?bcir2Yp8vIffWahDQZWoxgeUsFb6mFMtZzIsFUT376cJ4B3YzYRsXnlYkcZ3upX7l?= =?utf-8?q?hF0aQaCRU9tkyvdBJe1MK4+PCvxjdsVxA3jGSJYn2dZ5MlmwyGWCAyAfyzfbuDXHm?= =?utf-8?q?FIWewYr3fqF7yKuQue2xWZ4ZUxh6pvAKRhehxYcRGTOwkIIlv9QxkimHjVy39ztrM?= =?utf-8?q?ynll/vUrmNj5F5vK99/VtrW56ftkZXniAStBO3N2ifyD/qvfSXCLu4aePeXA29jF4?= =?utf-8?q?MbiBlenY+AnAEBu0hRdt/aLYsIK32po+CXHAEzVFRxWHmSk0LEwaMj3K45dGlE6zD?= =?utf-8?q?5WpWSEyDajDNdC03k1NeC2CIdiKc9Fzr13r9kxQwuVDeBHpC+fs2AgPhSLrUZGrOh?= =?utf-8?q?KnPeNlPUbNeS929AYPT/Jt2Bvq+mL6tO6CVf9wBVqTCr4V2r+wls/hVE2wZoHDiQQ?= =?utf-8?q?ct79313UETm9JNT4uVAFSZTVWp614fpS4C5MM5WAqiLXlrpyEiLDE=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 798fa9b2-16b7-4286-99de-08da2eba449e X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:17.7323 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wBEO00x74dZ8OmdlVIvz1UAIcML/rfDu3vZUMum1SYLinGFKVyeFpT87WA2hrR/Oy491E5thCqeJceRhNwjKMDD3pgWM+aL0tRBTX3pmdgY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: JOXO_afqcR93hY8Qxew8vjqy9KgguAHM X-Proofpoint-ORIG-GUID: JOXO_afqcR93hY8Qxew8vjqy9KgguAHM Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" add user.c & user.h files for vfio-user code add proxy struct to handle comms with remote server Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 78 +++++++++++++++++++ include/hw/vfio/vfio-common.h | 2 + hw/vfio/pci.c | 19 +++++ hw/vfio/user.c | 170 ++++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 4 + hw/vfio/meson.build | 1 + 6 files changed, 274 insertions(+) create mode 100644 hw/vfio/user.h create mode 100644 hw/vfio/user.c diff --git a/hw/vfio/user.h b/hw/vfio/user.h new file mode 100644 index 0000000..da92862 --- /dev/null +++ b/hw/vfio/user.h @@ -0,0 +1,78 @@ +#ifndef VFIO_USER_H +#define VFIO_USER_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + */ + +typedef struct { + int send_fds; + int recv_fds; + int *fds; +} VFIOUserFDs; + +enum msg_type { + VFIO_MSG_NONE, + VFIO_MSG_ASYNC, + VFIO_MSG_WAIT, + VFIO_MSG_NOWAIT, + VFIO_MSG_REQ, +}; + +typedef struct VFIOUserMsg { + QTAILQ_ENTRY(VFIOUserMsg) next; + VFIOUserFDs *fds; + uint32_t rsize; + uint32_t id; + QemuCond cv; + bool complete; + enum msg_type type; +} VFIOUserMsg; + + +enum proxy_state { + VFIO_PROXY_CONNECTED = 1, + VFIO_PROXY_ERROR = 2, + VFIO_PROXY_CLOSING = 3, + VFIO_PROXY_CLOSED = 4, +}; + +typedef QTAILQ_HEAD(VFIOUserMsgQ, VFIOUserMsg) VFIOUserMsgQ; + +typedef struct VFIOProxy { + QLIST_ENTRY(VFIOProxy) next; + char *sockname; + struct QIOChannel *ioc; + void (*request)(void *opaque, VFIOUserMsg *msg); + void *req_arg; + int flags; + QemuCond close_cv; + AioContext *ctx; + QEMUBH *req_bh; + + /* + * above only changed when BQL is held + * below are protected by per-proxy lock + */ + QemuMutex lock; + VFIOUserMsgQ free; + VFIOUserMsgQ pending; + VFIOUserMsgQ incoming; + VFIOUserMsgQ outgoing; + VFIOUserMsg *last_nowait; + enum proxy_state state; +} VFIOProxy; + +/* VFIOProxy flags */ +#define VFIO_PROXY_CLIENT 0x1 + +VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); +void vfio_user_disconnect(VFIOProxy *proxy); + +#endif /* VFIO_USER_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 826cd98..3eb0b19 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -76,6 +76,7 @@ typedef struct VFIOAddressSpace { struct VFIOGroup; typedef struct VFIOContIO VFIOContIO; +typedef struct VFIOProxy VFIOProxy; typedef struct VFIOContainer { VFIOAddressSpace *space; @@ -147,6 +148,7 @@ typedef struct VFIODevice { VFIOMigration *migration; Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; + VFIOProxy *proxy; struct vfio_region_info **regions; } VFIODevice; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index be8fe1d..8f65074 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -43,6 +43,7 @@ #include "qapi/error.h" #include "migration/blocker.h" #include "migration/qemu-file.h" +#include "hw/vfio/user.h" /* convenience macros for PCI config space */ #define VDEV_CONFIG_READ(vbasedev, off, size, data) \ @@ -3408,6 +3409,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev); VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev = &vdev->vbasedev; + SocketAddress addr; + VFIOProxy *proxy; + Error *err = NULL; /* * TODO: make option parser understand SocketAddress @@ -3420,6 +3424,16 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) return; } + memset(&addr, 0, sizeof(addr)); + addr.type = SOCKET_ADDRESS_TYPE_UNIX; + addr.u.q_unix.path = udev->sock_name; + proxy = vfio_user_connect_dev(&addr, &err); + if (!proxy) { + error_setg(errp, "Remote proxy not found"); + return; + } + vbasedev->proxy = proxy; + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->dev = DEVICE(vdev); vbasedev->fd = -1; @@ -3431,8 +3445,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) static void vfio_user_instance_finalize(Object *obj) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + VFIODevice *vbasedev = &vdev->vbasedev; vfio_put_device(vdev); + + if (vbasedev->proxy != NULL) { + vfio_user_disconnect(vbasedev->proxy); + } } static Property vfio_user_pci_dev_properties[] = { diff --git a/hw/vfio/user.c b/hw/vfio/user.c new file mode 100644 index 0000000..c843f90 --- /dev/null +++ b/hw/vfio/user.c @@ -0,0 +1,170 @@ +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include +#include + +#include "qemu/error-report.h" +#include "qapi/error.h" +#include "qemu/main-loop.h" +#include "hw/hw.h" +#include "hw/vfio/vfio-common.h" +#include "hw/vfio/vfio.h" +#include "qemu/sockets.h" +#include "io/channel.h" +#include "io/channel-socket.h" +#include "io/channel-util.h" +#include "sysemu/iothread.h" +#include "user.h" + +static IOThread *vfio_user_iothread; + +static void vfio_user_shutdown(VFIOProxy *proxy); + + +/* + * Functions called by main, CPU, or iothread threads + */ + +static void vfio_user_shutdown(VFIOProxy *proxy) +{ + qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, NULL, NULL, NULL); +} + +/* + * Functions only called by iothread + */ + +static void vfio_user_cb(void *opaque) +{ + VFIOProxy *proxy = opaque; + + QEMU_LOCK_GUARD(&proxy->lock); + + proxy->state = VFIO_PROXY_CLOSED; + qemu_cond_signal(&proxy->close_cv); +} + + +/* + * Functions called by main or CPU threads + */ + +static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = + QLIST_HEAD_INITIALIZER(vfio_user_sockets); + +VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) +{ + VFIOProxy *proxy; + QIOChannelSocket *sioc; + QIOChannel *ioc; + char *sockname; + + if (addr->type != SOCKET_ADDRESS_TYPE_UNIX) { + error_setg(errp, "vfio_user_connect - bad address family"); + return NULL; + } + sockname = addr->u.q_unix.path; + + sioc = qio_channel_socket_new(); + ioc = QIO_CHANNEL(sioc); + if (qio_channel_socket_connect_sync(sioc, addr, errp)) { + object_unref(OBJECT(ioc)); + return NULL; + } + qio_channel_set_blocking(ioc, false, NULL); + + proxy = g_malloc0(sizeof(VFIOProxy)); + proxy->sockname = g_strdup_printf("unix:%s", sockname); + proxy->ioc = ioc; + proxy->flags = VFIO_PROXY_CLIENT; + proxy->state = VFIO_PROXY_CONNECTED; + + qemu_mutex_init(&proxy->lock); + qemu_cond_init(&proxy->close_cv); + + if (vfio_user_iothread == NULL) { + vfio_user_iothread = iothread_create("VFIO user", errp); + } + + proxy->ctx = iothread_get_aio_context(vfio_user_iothread); + + QTAILQ_INIT(&proxy->outgoing); + QTAILQ_INIT(&proxy->incoming); + QTAILQ_INIT(&proxy->free); + QTAILQ_INIT(&proxy->pending); + QLIST_INSERT_HEAD(&vfio_user_sockets, proxy, next); + + return proxy; +} + +void vfio_user_disconnect(VFIOProxy *proxy) +{ + VFIOUserMsg *r1, *r2; + + qemu_mutex_lock(&proxy->lock); + + /* our side is quitting */ + if (proxy->state == VFIO_PROXY_CONNECTED) { + vfio_user_shutdown(proxy); + if (!QTAILQ_EMPTY(&proxy->pending)) { + error_printf("vfio_user_disconnect: outstanding requests\n"); + } + } + object_unref(OBJECT(proxy->ioc)); + proxy->ioc = NULL; + + proxy->state = VFIO_PROXY_CLOSING; + QTAILQ_FOREACH_SAFE(r1, &proxy->outgoing, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->pending, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->incoming, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->pending, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->pending, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->free, r1, next); + g_free(r1); + } + + /* + * Make sure the iothread isn't blocking anywhere + * with a ref to this proxy by waiting for a BH + * handler to run after the proxy fd handlers were + * deleted above. + */ + aio_bh_schedule_oneshot(proxy->ctx, vfio_user_cb, proxy); + qemu_cond_wait(&proxy->close_cv, &proxy->lock); + + /* we now hold the only ref to proxy */ + qemu_mutex_unlock(&proxy->lock); + qemu_cond_destroy(&proxy->close_cv); + qemu_mutex_destroy(&proxy->lock); + + QLIST_REMOVE(proxy, next); + if (QLIST_EMPTY(&vfio_user_sockets)) { + iothread_destroy(vfio_user_iothread); + vfio_user_iothread = NULL; + } + + g_free(proxy->sockname); + g_free(proxy); +} diff --git a/MAINTAINERS b/MAINTAINERS index 8117241..cd44f91 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1958,8 +1958,12 @@ L: qemu-s390x@nongnu.org vfio-user M: John G Johnson M: Thanos Makatos +M: Elena Ufimtseva +M: Jagannathan Raman S: Supported F: docs/devel/vfio-user.rst +F: hw/vfio/user.c +F: hw/vfio/user.h vhost M: Michael S. Tsirkin diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index da9af29..2f86f72 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -9,6 +9,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'pci-quirks.c', 'pci.c', )) +vfio_ss.add(when: 'CONFIG_VFIO_USER', if_true: files('user.c')) vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c')) vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c')) vfio_ss.add(when: 'CONFIG_VFIO_XGMAC', if_true: files('calxeda-xgmac.c')) From patchwork Thu May 5 17:19:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839820 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA6ACC433F5 for ; Thu, 5 May 2022 17:24:11 +0000 (UTC) Received: from localhost ([::1]:52018 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfD0-0002th-Sq for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:24:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52536) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0j-00063C-JT for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:7178) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0e-0002Br-RA for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:27 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FnNPo013665 for ; Thu, 5 May 2022 17:11:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=vgJTM+6/t7u+pXKc3NHsWJe1Vv8p8k3alDYFcnH/pDI=; b=o2exDFH+7E7TveSfSeBbbxSNcoX/EbzUKIHoNHEpkLxCDEutThq0cYvH73BB8UBhoF4C zpkEb+Dvhsy9njLybwjYCpL0eJ09wIcnF3iA0AEycR93NmfI7FmQNCpMphdoL5MGZodR hvshy+0j2sQOUi0qs1RDytI6/hxEM7e1CdtrYDU94eA/MN+HtbuQFWOdxoBp3spbpqaf 0sVQfZtnSHD6hf4aYzluCB3KmnSKv1GcR/7xprA7rwaAfQUv2hE2qX2eDy3+krFbs54C C93N9pxSv91W4fNO3l2mjkXoQEwDfomG5bssStn4NHAOq7oIvREPGLpjTwLfJc0xD1mb tA== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frvqsm1gd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:22 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1PHY001942 for ; Thu, 5 May 2022 17:11:22 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a7amqq-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QSx/afDmkzSxL/5AdFxNBG3Bo94QSa5Mbt2fbOz7waRQyELEetQWsdzO7rCjGvOsxAGP3162tqYsPTUqLwbG9t82mOKCfpSEK30cJYxSM0O7kAzktamqWsVIGXeFCktFxE6Th6cKTK5ZmA4YAn9xYZVLBFO4d/f2WTHcVhbZV1Ik5+0DIydkBbwHnqdh1GRLcumINyloQCHQy1Yzqba+gzzXgC9LnfHkno16amzOPQokZ/hunsfxIkyiEsP533aKC1GOuME8vqrPf+lbQpi9Dy5MAl7RitsLzT1g2mRzF8+eH9t/mX888RE338cVe6gw4T3bsfGkStYcnUiR9bJNZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vgJTM+6/t7u+pXKc3NHsWJe1Vv8p8k3alDYFcnH/pDI=; b=mZSB7Zegp+Ii9ptH8qgizBr3nlAKzCAaSuzJd0VJv6q+CX63mDiqJzoIMDXZUDYkErkPXGCuIjnpsoEi/RalEric5A/sm6f/I0+fc5T9n60mqcDl+KmGa9HbQL84KXZdNGkYCkmJ6o/P58riJgf47P5ENNMyWWD1hK2T7iV5lOMInBp5/H7cACFqhM+d6SiaDdJlEXmOGnosv5qJv/I03taShbaYplCjIfs7iCyYVJ07NvxQ5gixfaAXA/P/V5wQnhf/su9JaE1SLkuE1dwH7u3P6wEV0+Vefr+sFoYL+CBbV/NieB45+Rkbs8vp6MOPUZbjhDDpvfr6H7CYw6PtkA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vgJTM+6/t7u+pXKc3NHsWJe1Vv8p8k3alDYFcnH/pDI=; b=VQ/oqCIWrr8TIXTkgCYgqDq81wrkQNlgzZc32XXaFJMpQiINjCxCFRU0ZwdkTQkZNFP9bPFrvp2dpVWe54Ei/TUJnpPP7CX3U0lsMpWSsUPW1PunlkmpSwuZIoUyzRGmd42h+zDf8DwZ27pygP4WTp9m2N+nODvL+I4lcZu0bd8= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:19 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:19 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 08/23] vfio-user: define socket receive functions Date: Thu, 5 May 2022 10:19:51 -0700 Message-Id: <13284a1a153e6791dc6afba2f4edf2ddf18c8d69.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 061cfecf-c91c-4c59-0cb5-08da2eba44d3 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BcbmPST/M5oEg0nqP1efZ99k19G9eybXp101x0SyhYkeIseDlRgo50xxzPq3ZT/MepwLycYWpejPLArnmxwUtEc7vXkLubJJTlzD6I5u4PUmTdR3gX+6D9v9nvIPSwlR5Aerld9qwe7RPdTwDGZZ64rhGS4pK9CNzUQF9ZsmdRClfItqt6IWFAfz8ms6wY+Pq4UUlUGezEvTMRfWXA/8NbEcydpp3BQdC9lUlGSrkCXEXlslgyDtmy6MCFV4lXtZRow1UNmjARAqjZdGMZvxZYnV2h09YABfk+7rb6JifpVog522QTcQ4hPb0MRz35rec1cWn+NdXuxeZdT7oOV0XX+acM9d1F2gtMZ1PcjUrhBYmtE+q2uM5Hl/xQIM5oYX4F2e+tY8IvPw4LT9HpzTGC0C9ZvrhIXFm3peZ8OtPuUeeGSW6JDTYqrZlqrbpRVM2nfF6GjIJzxuvQPSOho4p06F6rkCgpr3louLQA2wZZQ15pKMpPozeCP9YL+ILlYZLdR7Z43VK7WCAjvnP68o1x76m+7PNNMyTv0ClATjDa/p1O/NQlzHohiOmdWTtG0nvC8DsUeuGHi9ShGpXNAsy/LPVJspKjpsqPicv2qOsTHntZKzWBZVw6wrfEBZnmx7VSDfPUUSnSM9TbZGqR8kURfJ1tfRPU5nYiQx4lUNT+frz8jPGSd+VRLrQ2i8BGtGdkABmOKO6NyLz1UMkZ/4uQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(30864003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?8prjLPD4YD9qm1C/KNZeEEXksVqw?= =?utf-8?q?IOQ3xcrszMRW8kozhd5vApI1yOTPWgiJkeENprwpluGfMo9p7EUzH1nlb2uluCAgb?= =?utf-8?q?VDM1TFv7HhP6O1fz0Imimh4FniXPpOcLhxpIacT0uK4RKyZ5KikqCNhHfqGp56EBU?= =?utf-8?q?UyZaKOhu7E1oErF/MXzengW4Z85WshKyhipFbB1jqpvXRCVLb1oLSfzFqmEBhKd7c?= =?utf-8?q?XtaMDHzbhC3v7cxB8bFxwe+mOg8D96OThWn6vbksDylhIqf3DGj/sproDQO/3Z7IE?= =?utf-8?q?31e9NOYSYjoE8LhaR1HJg6D47uiyV85sEdzEBjZvb/feKFba/BDYcJXCKMmqcdX3e?= =?utf-8?q?qToUHrtytm0xWhFi/rrojrzkxj3vIbiY4GSYMF+J4EjimDwP4aVgqODWplu6Mpxg1?= =?utf-8?q?1CVPWgcBtUipsURkInx9aEQFp9r+gkp5i9OXSlyzlXHwYO9DiNM/tuna7ZOZ3tWiu?= =?utf-8?q?6SPXFjtwf5boZwCD6eqPpH8Ln+5LJY1ZiA8hyPEsAu5PYaG9iLojYDxwZnnlyHgv+?= =?utf-8?q?QRBcHxkD3x2g0e0tN8LvoNn/GqPByiDOhKSBMgIJJLIYKDmHh1OX0goz1XwGEHKKW?= =?utf-8?q?lph2hMR9czcBKQi2hxIuiXDu5d+VqyKZLJkfEyse1hDA3OwYf4EfVtef5xETa6KrH?= =?utf-8?q?M3VCOpZz/fpCd7nS5EPlHWckR/3tm53qqS5quxDW8JCqchOi9vBmRDo/Y3LrGth4t?= =?utf-8?q?3i6U+BduszqySq69QWzHTOkjzsLR13U848gs7vrSOOKi+aeyUI+ApxhOTr8sHLcHg?= =?utf-8?q?FK/4m6GgxHl85mWRp6ILy7r5agzvhxJE4tG7XFvA/lRVGv9EzvkwPnvHJxS4NgPSz?= =?utf-8?q?jlM8ICAbk1YMAwNvGjPDcfpFdWuPiRfoolf1oatV7KaDDzMWMmNdtOAZXGKguk/00?= =?utf-8?q?vLooY02ElOyn5G3DAoZ4oWASmApL/RJP1pua76wfp1Dp92Lx0X/NevXNusMgkV58W?= =?utf-8?q?A/+i9dyq5IpLWAX7psjVbYltRmLfo4W9X1UJE76k44JTm5sv8g410EdZnm+BDtG1N?= =?utf-8?q?j0RVR11dBfUlkgalFRef4d6WNtgwk9Gc8lHDO7rixLCn5PyK/irn1bM++KlCboXu9?= =?utf-8?q?KxVBgp5IprCmapJ1+iNv3S1bv5BXaSKuise7JdnzeVgrzKYfXs5IrZxeCbmJ+gFP1?= =?utf-8?q?wcvr7IN/WIZGC3Kv17aIyp3y79uikeGIIdehtIkUXdPnG6Qxc6yzbh6DkXscezzfc?= =?utf-8?q?NWFWd83POZDB3BYpTjHRJxexyPwvRMMMIJ0hkwR2FVhdeui6Hph5xnlBDe+u2SiGE?= =?utf-8?q?bIyavpU7jCxsmivm5LMe15GlxU7GIc1QQH8LKfQngNtSL64zkHU1327fg99Wan9yv?= =?utf-8?q?Ox0K7qp0NVaf3ojrCXDLKFGRNfQoqPU84z6xfl/xtcaPmja0cn07aqzYZ5BREZZpu?= =?utf-8?q?0xaGw6bNuOeuj3EYH4PeypE8GDqJdwJjzzL++1pjZ/xRgm8oKRdkAZrKTobYkeoXm?= =?utf-8?q?0TypeRYunUSjE6HjFMD/ozUbyaaKuuIlQYqW8KU+ejIPZdvuQKRKRC8nNKszUsbKZ?= =?utf-8?q?coifIDNZtEYpm8diH19rUsjERTiQksD/7IJMXLLsWQTpC0eNr+r7wnDXZi1COT8YL?= =?utf-8?q?ahXsf5L9JY1rYWLArPZ/CcHMHIOsLtRgZLbY9jj6dAJuo4HcnUeryv9hijyTHMlo5?= =?utf-8?q?NdtJlsBzP9vnlT5AbJd4nn7eCpTgCT7D+KvaJYa7nsPAiI4vwp+/s=3D?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 061cfecf-c91c-4c59-0cb5-08da2eba44d3 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:18.0916 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: aX2HuHQ/yq1qUiK2rht0YzDyayMaw+gKJmJmG5H88/KZgzOxiQ6Zc7oGg9rylBZugm8w1fgQjifeRdIrMKa3kfqRvt1drb+79TuEwDFmuvQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: eo7M_APFv2pnP5LqbJq6iZClG2uLZmjF X-Proofpoint-ORIG-GUID: eo7M_APFv2pnP5LqbJq6iZClG2uLZmjF Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add infrastructure needed to receive incoming messages Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 54 +++++++ hw/vfio/user.h | 8 + hw/vfio/pci.c | 6 + hw/vfio/user.c | 404 ++++++++++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 5 files changed, 473 insertions(+) create mode 100644 hw/vfio/user-protocol.h diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h new file mode 100644 index 0000000..d23877c --- /dev/null +++ b/hw/vfio/user-protocol.h @@ -0,0 +1,54 @@ +#ifndef VFIO_USER_PROTOCOL_H +#define VFIO_USER_PROTOCOL_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + * Each message has a standard header that describes the command + * being sent, which is almost always a VFIO ioctl(). + * + * The header may be followed by command-specific data, such as the + * region and offset info for read and write commands. + */ + +typedef struct { + uint16_t id; + uint16_t command; + uint32_t size; + uint32_t flags; + uint32_t error_reply; +} VFIOUserHdr; + +/* VFIOUserHdr commands */ +enum vfio_user_command { + VFIO_USER_VERSION = 1, + VFIO_USER_DMA_MAP = 2, + VFIO_USER_DMA_UNMAP = 3, + VFIO_USER_DEVICE_GET_INFO = 4, + VFIO_USER_DEVICE_GET_REGION_INFO = 5, + VFIO_USER_DEVICE_GET_REGION_IO_FDS = 6, + VFIO_USER_DEVICE_GET_IRQ_INFO = 7, + VFIO_USER_DEVICE_SET_IRQS = 8, + VFIO_USER_REGION_READ = 9, + VFIO_USER_REGION_WRITE = 10, + VFIO_USER_DMA_READ = 11, + VFIO_USER_DMA_WRITE = 12, + VFIO_USER_DEVICE_RESET = 13, + VFIO_USER_DIRTY_PAGES = 14, + VFIO_USER_MAX, +}; + +/* VFIOUserHdr flags */ +#define VFIO_USER_REQUEST 0x0 +#define VFIO_USER_REPLY 0x1 +#define VFIO_USER_TYPE 0xF + +#define VFIO_USER_NO_REPLY 0x10 +#define VFIO_USER_ERROR 0x20 + +#endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index da92862..68a1080 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -11,6 +11,8 @@ * */ +#include "user-protocol.h" + typedef struct { int send_fds; int recv_fds; @@ -27,6 +29,7 @@ enum msg_type { typedef struct VFIOUserMsg { QTAILQ_ENTRY(VFIOUserMsg) next; + VFIOUserHdr *hdr; VFIOUserFDs *fds; uint32_t rsize; uint32_t id; @@ -66,6 +69,8 @@ typedef struct VFIOProxy { VFIOUserMsgQ incoming; VFIOUserMsgQ outgoing; VFIOUserMsg *last_nowait; + VFIOUserMsg *part_recv; + size_t recv_left; enum proxy_state state; } VFIOProxy; @@ -74,5 +79,8 @@ typedef struct VFIOProxy { VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); +void vfio_user_set_handler(VFIODevice *vbasedev, + void (*handler)(void *opaque, VFIOUserMsg *msg), + void *reqarg); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 8f65074..7ef11c0 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3387,6 +3387,11 @@ type_init(register_vfio_pci_dev_type) * vfio-user routines. */ +static void vfio_user_pci_process_req(void *opaque, VFIOUserMsg *msg) +{ + +} + /* * Emulated devices don't use host hot reset */ @@ -3433,6 +3438,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) return; } vbasedev->proxy = proxy; + vfio_user_set_handler(vbasedev, vfio_user_pci_process_req, vdev); vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->dev = DEVICE(vdev); diff --git a/hw/vfio/user.c b/hw/vfio/user.c index c843f90..16b37cb 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -25,10 +25,26 @@ #include "sysemu/iothread.h" #include "user.h" +static uint64_t max_xfer_size; static IOThread *vfio_user_iothread; static void vfio_user_shutdown(VFIOProxy *proxy); +static VFIOUserMsg *vfio_user_getmsg(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds); +static VFIOUserFDs *vfio_user_getfds(int numfds); +static void vfio_user_recycle(VFIOProxy *proxy, VFIOUserMsg *msg); +static void vfio_user_recv(void *opaque); +static int vfio_user_recv_one(VFIOProxy *proxy); +static void vfio_user_cb(void *opaque); + +static void vfio_user_request(void *opaque); + +static inline void vfio_user_set_error(VFIOUserHdr *hdr, uint32_t err) +{ + hdr->flags |= VFIO_USER_ERROR; + hdr->error_reply = err; +} /* * Functions called by main, CPU, or iothread threads @@ -40,10 +56,338 @@ static void vfio_user_shutdown(VFIOProxy *proxy) qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, NULL, NULL, NULL); } +static VFIOUserMsg *vfio_user_getmsg(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds) +{ + VFIOUserMsg *msg; + + msg = QTAILQ_FIRST(&proxy->free); + if (msg != NULL) { + QTAILQ_REMOVE(&proxy->free, msg, next); + } else { + msg = g_malloc0(sizeof(*msg)); + qemu_cond_init(&msg->cv); + } + + msg->hdr = hdr; + msg->fds = fds; + return msg; +} + +/* + * Recycle a message list entry to the free list. + */ +static void vfio_user_recycle(VFIOProxy *proxy, VFIOUserMsg *msg) +{ + if (msg->type == VFIO_MSG_NONE) { + error_printf("vfio_user_recycle - freeing free msg\n"); + return; + } + + /* free msg buffer if no one is waiting to consume the reply */ + if (msg->type == VFIO_MSG_NOWAIT || msg->type == VFIO_MSG_ASYNC) { + g_free(msg->hdr); + if (msg->fds != NULL) { + g_free(msg->fds); + } + } + + msg->type = VFIO_MSG_NONE; + msg->hdr = NULL; + msg->fds = NULL; + msg->complete = false; + QTAILQ_INSERT_HEAD(&proxy->free, msg, next); +} + +static VFIOUserFDs *vfio_user_getfds(int numfds) +{ + VFIOUserFDs *fds = g_malloc0(sizeof(*fds) + (numfds * sizeof(int))); + + fds->fds = (int *)((char *)fds + sizeof(*fds)); + + return fds; +} + /* * Functions only called by iothread */ +/* + * Process a received message. + */ +static void vfio_user_process(VFIOProxy *proxy, VFIOUserMsg *msg, bool isreply) +{ + + /* + * Replies signal a waiter, if none just check for errors + * and free the message buffer. + * + * Requests get queued for the BH. + */ + if (isreply) { + msg->complete = true; + if (msg->type == VFIO_MSG_WAIT) { + qemu_cond_signal(&msg->cv); + } else { + if (msg->hdr->flags & VFIO_USER_ERROR) { + error_printf("vfio_user_rcv: error reply on async request "); + error_printf("command %x error %s\n", msg->hdr->command, + strerror(msg->hdr->error_reply)); + } + /* youngest nowait msg has been ack'd */ + if (proxy->last_nowait == msg) { + proxy->last_nowait = NULL; + } + vfio_user_recycle(proxy, msg); + } + } else { + QTAILQ_INSERT_TAIL(&proxy->incoming, msg, next); + qemu_bh_schedule(proxy->req_bh); + } +} + +/* + * Complete a partial message read + */ +static int vfio_user_complete(VFIOProxy *proxy, Error **errp) +{ + VFIOUserMsg *msg = proxy->part_recv; + size_t msgleft = proxy->recv_left; + bool isreply; + char *data; + int ret; + + data = (char *)msg->hdr + (msg->hdr->size - msgleft); + while (msgleft > 0) { + ret = qio_channel_read(proxy->ioc, data, msgleft, errp); + + /* error or would block */ + if (ret <= 0) { + /* try for rest on next iternation */ + if (ret == QIO_CHANNEL_ERR_BLOCK) { + proxy->recv_left = msgleft; + } + return ret; + } + + msgleft -= ret; + data += ret; + } + + /* + * Read complete message, process it. + */ + proxy->part_recv = NULL; + proxy->recv_left = 0; + isreply = (msg->hdr->flags & VFIO_USER_TYPE) == VFIO_USER_REPLY; + vfio_user_process(proxy, msg, isreply); + + /* return positive value */ + return 1; +} + +static void vfio_user_recv(void *opaque) +{ + VFIOProxy *proxy = opaque; + + QEMU_LOCK_GUARD(&proxy->lock); + + if (proxy->state == VFIO_PROXY_CONNECTED) { + while (vfio_user_recv_one(proxy) == 0) { + ; + } + } +} + +/* + * Receive and process one incoming message. + * + * For replies, find matching outgoing request and wake any waiters. + * For requests, queue in incoming list and run request BH. + */ +static int vfio_user_recv_one(VFIOProxy *proxy) +{ + VFIOUserMsg *msg = NULL; + g_autofree int *fdp = NULL; + VFIOUserFDs *reqfds; + VFIOUserHdr hdr; + struct iovec iov = { + .iov_base = &hdr, + .iov_len = sizeof(hdr), + }; + bool isreply = false; + int i, ret; + size_t msgleft, numfds = 0; + char *data = NULL; + char *buf = NULL; + Error *local_err = NULL; + + /* + * Complete any partial reads + */ + if (proxy->part_recv != NULL) { + ret = vfio_user_complete(proxy, &local_err); + + /* still not complete, try later */ + if (ret == QIO_CHANNEL_ERR_BLOCK) { + return ret; + } + + if (ret <= 0) { + goto fatal; + } + /* else fall into reading another msg */ + } + + /* + * Read header + */ + ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, + &local_err); + if (ret == QIO_CHANNEL_ERR_BLOCK) { + return ret; + } + + /* read error or other side closed connection */ + if (ret <= 0) { + goto fatal; + } + + if (ret < sizeof(msg)) { + error_setg(&local_err, "short read of header"); + goto fatal; + } + + /* + * Validate header + */ + if (hdr.size < sizeof(VFIOUserHdr)) { + error_setg(&local_err, "bad header size"); + goto fatal; + } + switch (hdr.flags & VFIO_USER_TYPE) { + case VFIO_USER_REQUEST: + isreply = false; + break; + case VFIO_USER_REPLY: + isreply = true; + break; + default: + error_setg(&local_err, "unknown message type"); + goto fatal; + } + + /* + * For replies, find the matching pending request. + * For requests, reap incoming FDs. + */ + if (isreply) { + QTAILQ_FOREACH(msg, &proxy->pending, next) { + if (hdr.id == msg->id) { + break; + } + } + if (msg == NULL) { + error_setg(&local_err, "unexpected reply"); + goto err; + } + QTAILQ_REMOVE(&proxy->pending, msg, next); + + /* + * Process any received FDs + */ + if (numfds != 0) { + if (msg->fds == NULL || msg->fds->recv_fds < numfds) { + error_setg(&local_err, "unexpected FDs"); + goto err; + } + msg->fds->recv_fds = numfds; + memcpy(msg->fds->fds, fdp, numfds * sizeof(int)); + } + } else { + if (numfds != 0) { + reqfds = vfio_user_getfds(numfds); + memcpy(reqfds->fds, fdp, numfds * sizeof(int)); + } else { + reqfds = NULL; + } + } + + /* + * Put the whole message into a single buffer. + */ + if (isreply) { + if (hdr.size > msg->rsize) { + error_setg(&local_err, "reply larger than recv buffer"); + goto err; + } + *msg->hdr = hdr; + data = (char *)msg->hdr + sizeof(hdr); + } else { + if (hdr.size > max_xfer_size) { + error_setg(&local_err, "vfio_user_recv request larger than max"); + goto err; + } + buf = g_malloc0(hdr.size); + memcpy(buf, &hdr, sizeof(hdr)); + data = buf + sizeof(hdr); + msg = vfio_user_getmsg(proxy, (VFIOUserHdr *)buf, reqfds); + msg->type = VFIO_MSG_REQ; + } + + /* + * Read rest of message. + */ + msgleft = hdr.size - sizeof(hdr); + while (msgleft > 0) { + ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err); + + /* prepare to complete read on next iternation */ + if (ret == QIO_CHANNEL_ERR_BLOCK) { + proxy->part_recv = msg; + proxy->recv_left = msgleft; + return ret; + } + + if (ret <= 0) { + goto fatal; + } + + msgleft -= ret; + data += ret; + } + + vfio_user_process(proxy, msg, isreply); + return 0; + + /* + * fatal means the other side closed or we don't trust the stream + * err means this message is corrupt + */ +fatal: + vfio_user_shutdown(proxy); + proxy->state = VFIO_PROXY_ERROR; + + /* set error if server side closed */ + if (ret == 0) { + error_setg(&local_err, "server closed socket"); + } + +err: + for (i = 0; i < numfds; i++) { + close(fdp[i]); + } + if (isreply && msg != NULL) { + /* force an error to keep sending thread from hanging */ + vfio_user_set_error(msg->hdr, EINVAL); + msg->complete = true; + qemu_cond_signal(&msg->cv); + } + error_prepend(&local_err, "vfio_user_recv: "); + error_report_err(local_err); + return -1; +} + static void vfio_user_cb(void *opaque) { VFIOProxy *proxy = opaque; @@ -59,6 +403,51 @@ static void vfio_user_cb(void *opaque) * Functions called by main or CPU threads */ +/* + * Process incoming requests. + * + * The bus-specific callback has the form: + * request(opaque, msg) + * where 'opaque' was specified in vfio_user_set_handler + * and 'msg' is the inbound message. + * + * The callback is responsible for disposing of the message buffer, + * usually by re-using it when calling vfio_send_reply or vfio_send_error, + * both of which free their message buffer when the reply is sent. + * + * If the callback uses a new buffer, it needs to free the old one. + */ +static void vfio_user_request(void *opaque) +{ + VFIOProxy *proxy = opaque; + VFIOUserMsgQ new, free; + VFIOUserMsg *msg, *m1; + + /* reap all incoming */ + QTAILQ_INIT(&new); + WITH_QEMU_LOCK_GUARD(&proxy->lock) { + QTAILQ_FOREACH_SAFE(msg, &proxy->incoming, next, m1) { + QTAILQ_REMOVE(&proxy->pending, msg, next); + QTAILQ_INSERT_TAIL(&new, msg, next); + } + } + + /* process list */ + QTAILQ_INIT(&free); + QTAILQ_FOREACH_SAFE(msg, &new, next, m1) { + QTAILQ_REMOVE(&new, msg, next); + proxy->request(proxy->req_arg, msg); + QTAILQ_INSERT_HEAD(&free, msg, next); + } + + /* free list */ + WITH_QEMU_LOCK_GUARD(&proxy->lock) { + QTAILQ_FOREACH_SAFE(msg, &free, next, m1) { + vfio_user_recycle(proxy, msg); + } + } +} + static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -97,6 +486,7 @@ VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) } proxy->ctx = iothread_get_aio_context(vfio_user_iothread); + proxy->req_bh = qemu_bh_new(vfio_user_request, proxy); QTAILQ_INIT(&proxy->outgoing); QTAILQ_INIT(&proxy->incoming); @@ -107,6 +497,18 @@ VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) return proxy; } +void vfio_user_set_handler(VFIODevice *vbasedev, + void (*handler)(void *opaque, VFIOUserMsg *msg), + void *req_arg) +{ + VFIOProxy *proxy = vbasedev->proxy; + + proxy->request = handler; + proxy->req_arg = req_arg; + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, + vfio_user_recv, NULL, proxy); +} + void vfio_user_disconnect(VFIOProxy *proxy) { VFIOUserMsg *r1, *r2; @@ -122,6 +524,8 @@ void vfio_user_disconnect(VFIOProxy *proxy) } object_unref(OBJECT(proxy->ioc)); proxy->ioc = NULL; + qemu_bh_delete(proxy->req_bh); + proxy->req_bh = NULL; proxy->state = VFIO_PROXY_CLOSING; QTAILQ_FOREACH_SAFE(r1, &proxy->outgoing, next, r2) { diff --git a/MAINTAINERS b/MAINTAINERS index cd44f91..c81f8b6 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1964,6 +1964,7 @@ S: Supported F: docs/devel/vfio-user.rst F: hw/vfio/user.c F: hw/vfio/user.h +F: hw/vfio/user-protocol.h vhost M: Michael S. Tsirkin From patchwork Thu May 5 17:19:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id EC473C433EF for ; Thu, 5 May 2022 17:21:42 +0000 (UTC) Received: from localhost ([::1]:47760 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfAb-00088f-MN for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:21:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52590) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0l-00063M-A0 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:51714) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0e-0002C2-VT for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:29 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FpOfj029440 for ; Thu, 5 May 2022 17:11:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=ocdII6lbRdQ8YqAiUNKiuNn0ZsUcAik4BZfUSdlSYA0=; b=RQ3fCTDbcPvZE4ZXTxkPdJpg6Wkwdg2lAsjTf2OcXvg/8JWa3kn0TkeMvXyfTtwCbn4K QHrVgok70mU79jIXIdNUUnVK8iQQIa+Q34l7ABZK44tX8jSYLnErd/eyP1KnojPkrzWy t4Pw9KC/I8xZPzsPxNuT/WoRTUNToEBD3ahmt3YjEpq7or7BHqFBXnGUCGtnSimhecY1 kYjz3YqMti+rISxC3jtSZVGHBVheCqAbPVX2i9R3jxupQ8y7FSNVA9qeLV2SMIRc9Poe B2b139Ia9pANfjntPmRxaNyhAnLAbqj5f+t83U49M9TriYmbUF/pZTj+I+JbO/Wmm61w ig== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0m376-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:23 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1PHZ001942 for ; Thu, 5 May 2022 17:11:22 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2168.outbound.protection.outlook.com [104.47.56.168]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a7amqq-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:22 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Gmj4r5N0LWd2pYjqwpa9tCRl3PYzIPo2/JCymRW18tYdoTcLxrG9bDlXztZCBC/CfZF95v95RPX7sPvm4K1Gy3V1qJXijKRkk3NOL5UuUjeV9FfSeIGeZiY/P5VwtqZ0g/WzdbUGpoq4F+Ou3xUQsQIeE8NySz94zRWWw+b8yolWUnaCeRq6+JBfnUcK3rnI1L8qXN3/QDM7GneN9r37pZJCblsWxOHo8Dn0NxdndRzruWXrpi9Q8y45nmzunmjFz5S4JetUWuIkHkkeRKjW7qpu+HD1ir0FYSiFto1/v3KthYShyJhjeUrruvMEfF3DIRR2Ohz+Au6yWGtlBtw0aQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ocdII6lbRdQ8YqAiUNKiuNn0ZsUcAik4BZfUSdlSYA0=; b=NdWJLFIekv6nZc4q28ImwjTuNvLN24ZGFISCRdv5rR4mixzPVv99E1vTfkm51lFJoDDYDNvwDBCA8sevaLivxFRJ4x/54w/s9XXT022ZksUZVcnvdbg6LWnPCBCKK2sCmGzJM3yxwjtY3vp5LfjIiHKiUP6sC8p1u0VVfTbdMFOnQcMRPesXS0AmYmIITpgLac4uAI2BI17VK+6efMz6g1ywsS1P8HBmoZeTG7mvqyEI/Zsh6QLNSaovHTTFQZjSQTfW7eNzJTs0g6EEHunGm76OsOJgksARqAujsIVZ2VOTOHoAyb+X4sh1c6XtJKe61u2fOj/AkjEn3mvcGdir7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ocdII6lbRdQ8YqAiUNKiuNn0ZsUcAik4BZfUSdlSYA0=; b=V0bfpadRH3w6U9l9h9vtPx2aGbm633JX1CqCVnRsWU+y3vp4tpcGNzmgNSY7+O9tG6jsJg26mDfq9EX7LE6/Tq5dHrqDDBguttN1hlbXbdHcgmD+xtAFXF97QIOeHpIq4Z/JvxXKIWoDC/x2oc9GsZ1+u89gnrLxdL4h39SiczQ= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:20 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:20 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 09/23] vfio-user: define socket send functions Date: Thu, 5 May 2022 10:19:52 -0700 Message-Id: <455feebdac7fc42700bf5a870493007f7f354cfa.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 744c22e8-dbd5-4005-1d5b-08da2eba4505 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +fR2vlzMMzaB96CzsAtJkeKXiKFvLM2EIZW5n3XlIHz0OrP6fh+NrhjAkS60VyTAEAMC6Djhru6xCgDVgMk2K0wvsqwNYy9heZFgVyTphgq3iEAjUyZabaiWbof9UFecQabN0v1So57TEQgYT07NBY85Hxg7dX9cGReE6uM5Cwhot2fPz3gKQIiHZYGEOY+P6vBzVlNiox/up0FesjSYRgGTUjyfQITp0h+mzDUDhl3Qd7dlQL6qOFmunuqKZd+XjwllXSeKzi6M9TDB8ImJj8Zfd1qLzfzmqb9x9XDvH6ItBAgs2ePFNYJ1aIq3v+Bi98xFtUYkuMEnBZ4dRw4otXXNiomT2YYHdV/nSfrbmmNnvZ+O/KFOmmQu3qrT2M8Njj3mSIfBV3LvwejMJKTfg2FllUR7s1PSoo2abQU+ELbI2n+ZAtj57M7i6PSBgxH6iUmHbAhpje7jbjvgsrenYf/4QGLcys8C1csx5i6EmcVb4nQQVPYIf7vQSzYPMkNzY2nWBqHd/e/4qzRlzAsJDRqdMy/JTsNf4TylfBCYGi9WxEHB4ew9SGMx9ypCQzNERiiS2LTA4vaqeIqF1fcI0Rn9R9NiBfZEmLa2RE4xSrjDR5L0eisclHxXEwgq+ayWzPqlyi4HY4Bx49hIat6cNRx6+UKWZ/3T//QLQ0BSJO9i7c+Yx0RlR/qHKm+km/rSvjqq9khCSuuw8YaoADhHgQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(30864003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: CKt/aH1qfuUSJy39DwtVDGfiQYApu8yr72q6YTP6KwfFcYACrQieRl+Lps4vMNkMmEWUwZN/WX5223Zws8YnrPaxvdJxN9gKEr8CR2DymYxe1Wbqgd94f8av7rqIGShhKW7wuyIRa3ilGTOpcPvZ1rNCSRiAE7YVwBjxSf3Y22emLOULrhD7hBLCno1tp92NJ68C7wG6QD3Pm1RElfOTQ04VoweJVoOKPBdTRK3J9zfF3oILTVFJDMG3DoF3vmoD0pdMk4xYC2TxaZlcF/ikZ9J+kWWUGkXj+vc13sJOGwoKIq/fROLz26SJ2QFGQ95xdgCtPlIGoOIeSWGnMJjygDTTIv+n1LvC7wGUNayVL4U7uex5+MY04n3UU/51SdDril1aSuASebhzD2bB5zoTRwCeorMQI5TNjKAQUJgkYD3SZfEJLZoXyORDnOwTpek8VkAoUS04mSSVAeg0sGDEtltz8Ebn75uH0XVVUDEscIQnoUflV0SQF2hZdLPZNDa1qb6bUoZK90LBZTIAMurIpv2mFX35aO1rRWF76DdCx4+BZpxvX6ogSP7MWOCqlsyRASohFeF0jrLh6J82UwjdUMd0Vl7lyQgR2yhKfyjuuOiaSs84M5gxFmfoFbQqTDOJeDhj8DihPG1T7Aiw6Up4cVbG36uz9it2pc7Z8AoNOvtS6Nfptvcdn0D6VXlvUnmOj8wr6Es9mQ4fmkOUGd7jauSy6P4G3F4e0BS2QOILjQf0hDUNAq5kjZnsiZ1ArIhtA4PrPYdmadeypRoM9c1AjNUdWbV5HNEcYjngP2Dw+aUnKAI2wg3XgWhuNZdwGYumcIryKgLouq6cJSb0zYYLo095Cd+3lYLD656Pk/GtTNmygCj1DNEeBLE8QcZ6zoebb24mP3mYha9d7tAn7QwTvvwNKWJN0VsNCUOB4v8XL4ocKo0tHwpflgEJaX4cYea5MeiKmX+0ztoqbgAc64714A5x0mIu+bFbFOrr4iRVAhREAQOnx5kGTekBS8KFaRX8CbFZ1qifhR9qfC14Vsdh9D9LZs70/a5pDGX46NMnbZGNeB2JbyRKMSlw9HYSzA01Gxly+kEHIa7/yq+29GJ7suMs//C2RIZcySBMuISZxRdkmUBSp7v/rn0Ix+Hk+3/lnLbd9DXXnARpKGxUjUbZi+I9yVoturstgvMiYmXHhrEQifpqHSY83YyGEvc1lnolIDRpEiWyoE1qu/2eRzTgXCocRcdieOag3oxg836SmvR2BQImqxZQGLexF+rFCh1FgOFnHtU7zGkKUs7lhNr8Ylp926OlLs/4nBaXtyvhtCStHx+m1C00sb0hp9hWBAI2V/Rsr5cFLeSL/gzmyuxZuhfWX7gJUdtHJM5i83m3p9eWqsqFxKEMma1fh8APAIx0MUhAhQE4QxdqrVK8t2GZ42Noe+wM48+hlQV0w69t3JAX6lb8XpNmly4JjLTHVwPzQfJAngCl6o3loYV4Kni9+Ha/x+2/EbTHH9EdnK0a+v3WvaUl+luvL29H7JVOxkxxJAfCIXUs0e2elU1hqDYnBf9NxBj9VWAVUML3M7U8EjSQWK5PiFpy/mD/wudY2027tekVscwrI0l1PsPi+xAX0QPVSYfn/TB5zflVnrLcNECeJ+zCFLJG7YZyNIjQUbv5fLjOhgHri+aa8KKfSySJjmDyvin9ACkOumw+x/4DAUc2VJfolSfgh2OekTo/N/7xe489IFxcqsSWakiH4MAG5EfF2gGZ0YJOSG2he+F1z28= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 744c22e8-dbd5-4005-1d5b-08da2eba4505 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:18.4510 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jyKnHszmX2scXSzCtt1VFKgFA9YSmEunqC2zuHFt+zcK4asX1YkURJc9CKGvDOMOYFlnq7hiZmDVLfgSkz0mB7DPsuoWBzEQOcWrP4VbyRs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: 3Tw7pKPrialgvM10X0qVZztIgYhr4Zw3 X-Proofpoint-GUID: 3Tw7pKPrialgvM10X0qVZztIgYhr4Zw3 Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Also negotiate protocol version with remote server Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson --- hw/vfio/pci.h | 1 + hw/vfio/user-protocol.h | 41 +++++ hw/vfio/user.h | 2 + hw/vfio/pci.c | 16 ++ hw/vfio/user.c | 414 +++++++++++++++++++++++++++++++++++++++++++++++- 5 files changed, 473 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 59e636c..ec9f345 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -193,6 +193,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) struct VFIOUserPCIDevice { VFIOPCIDevice device; char *sock_name; + bool send_queued; /* all sends are queued */ }; /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index d23877c..a0889f6 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -51,4 +51,45 @@ enum vfio_user_command { #define VFIO_USER_NO_REPLY 0x10 #define VFIO_USER_ERROR 0x20 + +/* + * VFIO_USER_VERSION + */ +typedef struct { + VFIOUserHdr hdr; + uint16_t major; + uint16_t minor; + char capabilities[]; +} VFIOUserVersion; + +#define VFIO_USER_MAJOR_VER 0 +#define VFIO_USER_MINOR_VER 0 + +#define VFIO_USER_CAP "capabilities" + +/* "capabilities" members */ +#define VFIO_USER_CAP_MAX_FDS "max_msg_fds" +#define VFIO_USER_CAP_MAX_XFER "max_data_xfer_size" +#define VFIO_USER_CAP_MIGR "migration" + +/* "migration" member */ +#define VFIO_USER_CAP_PGSIZE "pgsize" + +/* + * Max FDs mainly comes into play when a device supports multiple interrupts + * where each ones uses an eventfd to inject it into the guest. + * It is clamped by the the number of FDs the qio channel supports in a + * single message. + */ +#define VFIO_USER_DEF_MAX_FDS 8 +#define VFIO_USER_MAX_MAX_FDS 16 + +/* + * Max transfer limits the amount of data in region and DMA messages. + * Region R/W will be very small (limited by how much a single instruction + * can process) so just use a reasonable limit here. + */ +#define VFIO_USER_DEF_MAX_XFER (1024 * 1024) +#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 68a1080..00d21bf 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -76,11 +76,13 @@ typedef struct VFIOProxy { /* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_FORCE_QUEUED 0x4 VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); void vfio_user_set_handler(VFIODevice *vbasedev, void (*handler)(void *opaque, VFIOUserMsg *msg), void *reqarg); +int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7ef11c0..7e5b910 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3440,12 +3440,27 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->proxy = proxy; vfio_user_set_handler(vbasedev, vfio_user_pci_process_req, vdev); + if (udev->send_queued) { + proxy->flags |= VFIO_PROXY_FORCE_QUEUED; + } + + vfio_user_validate_version(vbasedev, &err); + if (err != NULL) { + error_propagate(errp, err); + goto error; + } + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->dev = DEVICE(vdev); vbasedev->fd = -1; vbasedev->type = VFIO_DEVICE_TYPE_PCI; vbasedev->ops = &vfio_user_pci_ops; + return; + +error: + vfio_user_disconnect(proxy); + error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } static void vfio_user_instance_finalize(Object *obj) @@ -3462,6 +3477,7 @@ static void vfio_user_instance_finalize(Object *obj) static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), + DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 16b37cb..dc3f1a6 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -23,12 +23,20 @@ #include "io/channel-socket.h" #include "io/channel-util.h" #include "sysemu/iothread.h" +#include "qapi/qmp/qdict.h" +#include "qapi/qmp/qjson.h" +#include "qapi/qmp/qnull.h" +#include "qapi/qmp/qstring.h" +#include "qapi/qmp/qnum.h" #include "user.h" -static uint64_t max_xfer_size; +static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER; +static uint64_t max_send_fds = VFIO_USER_DEF_MAX_FDS; +static int wait_time = 1000; /* wait 1 sec for replies */ static IOThread *vfio_user_iothread; static void vfio_user_shutdown(VFIOProxy *proxy); +static int vfio_user_send_qio(VFIOProxy *proxy, VFIOUserMsg *msg); static VFIOUserMsg *vfio_user_getmsg(VFIOProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds); static VFIOUserFDs *vfio_user_getfds(int numfds); @@ -36,9 +44,16 @@ static void vfio_user_recycle(VFIOProxy *proxy, VFIOUserMsg *msg); static void vfio_user_recv(void *opaque); static int vfio_user_recv_one(VFIOProxy *proxy); +static void vfio_user_send(void *opaque); +static int vfio_user_send_one(VFIOProxy *proxy, VFIOUserMsg *msg); static void vfio_user_cb(void *opaque); static void vfio_user_request(void *opaque); +static int vfio_user_send_queued(VFIOProxy *proxy, VFIOUserMsg *msg); +static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize, bool nobql); +static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags); static inline void vfio_user_set_error(VFIOUserHdr *hdr, uint32_t err) { @@ -56,6 +71,32 @@ static void vfio_user_shutdown(VFIOProxy *proxy) qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, NULL, NULL, NULL); } +static int vfio_user_send_qio(VFIOProxy *proxy, VFIOUserMsg *msg) +{ + VFIOUserFDs *fds = msg->fds; + struct iovec iov = { + .iov_base = msg->hdr, + .iov_len = msg->hdr->size, + }; + size_t numfds = 0; + int ret, *fdp = NULL; + Error *local_err = NULL; + + if (fds != NULL && fds->send_fds != 0) { + numfds = fds->send_fds; + fdp = fds->fds; + } + + ret = qio_channel_writev_full(proxy->ioc, &iov, 1, fdp, numfds, &local_err); + + if (ret == -1) { + vfio_user_set_error(msg->hdr, EIO); + vfio_user_shutdown(proxy); + error_report_err(local_err); + } + return ret; +} + static VFIOUserMsg *vfio_user_getmsg(VFIOProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds) { @@ -388,6 +429,53 @@ err: return -1; } +/* + * Send messages from outgoing queue when the socket buffer has space. + * If we deplete 'outgoing', remove ourselves from the poll list. + */ +static void vfio_user_send(void *opaque) +{ + VFIOProxy *proxy = opaque; + VFIOUserMsg *msg; + + QEMU_LOCK_GUARD(&proxy->lock); + + if (proxy->state == VFIO_PROXY_CONNECTED) { + while (!QTAILQ_EMPTY(&proxy->outgoing)) { + msg = QTAILQ_FIRST(&proxy->outgoing); + if (vfio_user_send_one(proxy, msg) < 0) { + return; + } + } + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, + vfio_user_recv, NULL, proxy); + } +} + +/* + * Send a single message. + * + * Sent async messages are freed, others are moved to pending queue. + */ +static int vfio_user_send_one(VFIOProxy *proxy, VFIOUserMsg *msg) +{ + int ret; + + ret = vfio_user_send_qio(proxy, msg); + if (ret < 0) { + return ret; + } + + QTAILQ_REMOVE(&proxy->outgoing, msg, next); + if (msg->type == VFIO_MSG_ASYNC) { + vfio_user_recycle(proxy, msg); + } else { + QTAILQ_INSERT_TAIL(&proxy->pending, msg, next); + } + + return 0; +} + static void vfio_user_cb(void *opaque) { VFIOProxy *proxy = opaque; @@ -448,6 +536,130 @@ static void vfio_user_request(void *opaque) } } +/* + * Messages are queued onto the proxy's outgoing list. + * + * It handles 3 types of messages: + * + * async messages - replies and posted writes + * + * There will be no reply from the server, so message + * buffers are freed after they're sent. + * + * nowait messages - map/unmap during address space transactions + * + * These are also sent async, but a reply is expected so that + * vfio_wait_reqs() can wait for the youngest nowait request. + * They transition from the outgoing list to the pending list + * when sent, and are freed when the reply is received. + * + * wait messages - all other requests + * + * The reply to these messages is waited for by their caller. + * They also transition from outgoing to pending when sent, but + * the message buffer is returned to the caller with the reply + * contents. The caller is responsible for freeing these messages. + * + * As an optimization, if the outgoing list and the socket send + * buffer are empty, the message is sent inline instead of being + * added to the outgoing list. The rest of the transitions are + * unchanged. + * + * returns 0 if the message was sent or queued + * returns -1 on send error + */ +static int vfio_user_send_queued(VFIOProxy *proxy, VFIOUserMsg *msg) +{ + int ret; + + /* + * Unsent outgoing msgs - add to tail + */ + if (!QTAILQ_EMPTY(&proxy->outgoing)) { + QTAILQ_INSERT_TAIL(&proxy->outgoing, msg, next); + return 0; + } + + /* + * Try inline - if blocked, queue it and kick send poller + */ + if (proxy->flags & VFIO_PROXY_FORCE_QUEUED) { + ret = QIO_CHANNEL_ERR_BLOCK; + } else { + ret = vfio_user_send_qio(proxy, msg); + } + if (ret == QIO_CHANNEL_ERR_BLOCK) { + QTAILQ_INSERT_HEAD(&proxy->outgoing, msg, next); + qio_channel_set_aio_fd_handler(proxy->ioc, proxy->ctx, + vfio_user_recv, vfio_user_send, + proxy); + return 0; + } + if (ret == -1) { + return ret; + } + + /* + * Sent - free async, add others to pending + */ + if (msg->type == VFIO_MSG_ASYNC) { + vfio_user_recycle(proxy, msg); + } else { + QTAILQ_INSERT_TAIL(&proxy->pending, msg, next); + } + + return 0; +} + +static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize, bool nobql) +{ + VFIOUserMsg *msg; + bool iolock = false; + int ret; + + if (hdr->flags & VFIO_USER_NO_REPLY) { + error_printf("vfio_user_send_wait on async message\n"); + return; + } + + /* + * We may block later, so use a per-proxy lock and drop + * BQL while we sleep unless 'nobql' says not to. + */ + qemu_mutex_lock(&proxy->lock); + if (!nobql) { + iolock = qemu_mutex_iothread_locked(); + if (iolock) { + qemu_mutex_unlock_iothread(); + } + } + + msg = vfio_user_getmsg(proxy, hdr, fds); + msg->id = hdr->id; + msg->rsize = rsize ? rsize : hdr->size; + msg->type = VFIO_MSG_WAIT; + + ret = vfio_user_send_queued(proxy, msg); + + if (ret == 0) { + while (!msg->complete) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + QTAILQ_REMOVE(&proxy->pending, msg, next); + vfio_user_set_error(hdr, ETIMEDOUT); + break; + } + } + } + vfio_user_recycle(proxy, msg); + + /* lock order is BQL->proxy - don't hold proxy when getting BQL */ + qemu_mutex_unlock(&proxy->lock); + if (iolock) { + qemu_mutex_lock_iothread(); + } +} + static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -572,3 +784,203 @@ void vfio_user_disconnect(VFIOProxy *proxy) g_free(proxy->sockname); g_free(proxy); } + +static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags) +{ + static uint16_t next_id; + + hdr->id = qatomic_fetch_inc(&next_id); + hdr->command = cmd; + hdr->size = size; + hdr->flags = (flags & ~VFIO_USER_TYPE) | VFIO_USER_REQUEST; + hdr->error_reply = 0; +} + +struct cap_entry { + const char *name; + int (*check)(QObject *qobj, Error **errp); +}; + +static int caps_parse(QDict *qdict, struct cap_entry caps[], Error **errp) +{ + QObject *qobj; + struct cap_entry *p; + + for (p = caps; p->name != NULL; p++) { + qobj = qdict_get(qdict, p->name); + if (qobj != NULL) { + if (p->check(qobj, errp)) { + return -1; + } + qdict_del(qdict, p->name); + } + } + + /* warning, for now */ + if (qdict_size(qdict) != 0) { + error_printf("spurious capabilities\n"); + } + return 0; +} + +static int check_pgsize(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t pgsize; + + if (qn == NULL || !qnum_get_try_uint(qn, &pgsize)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_PGSIZE); + return -1; + } + return pgsize == 4096 ? 0 : -1; +} + +static struct cap_entry caps_migr[] = { + { VFIO_USER_CAP_PGSIZE, check_pgsize }, + { NULL } +}; + +static int check_max_fds(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + + if (qn == NULL || !qnum_get_try_uint(qn, &max_send_fds) || + max_send_fds > VFIO_USER_MAX_MAX_FDS) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return -1; + } + return 0; +} + +static int check_max_xfer(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + + if (qn == NULL || !qnum_get_try_uint(qn, &max_xfer_size) || + max_xfer_size > VFIO_USER_MAX_MAX_XFER) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_XFER); + return -1; + } + return 0; +} + +static int check_migr(QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return -1; + } + return caps_parse(qdict, caps_migr, errp); +} + +static struct cap_entry caps_cap[] = { + { VFIO_USER_CAP_MAX_FDS, check_max_fds }, + { VFIO_USER_CAP_MAX_XFER, check_max_xfer }, + { VFIO_USER_CAP_MIGR, check_migr }, + { NULL } +}; + +static int check_cap(QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL) { + error_setg(errp, "malformed %s", VFIO_USER_CAP); + return -1; + } + return caps_parse(qdict, caps_cap, errp); +} + +static struct cap_entry ver_0_0[] = { + { VFIO_USER_CAP, check_cap }, + { NULL } +}; + +static int caps_check(int minor, const char *caps, Error **errp) +{ + QObject *qobj; + QDict *qdict; + int ret; + + qobj = qobject_from_json(caps, NULL); + if (qobj == NULL) { + error_setg(errp, "malformed capabilities %s", caps); + return -1; + } + qdict = qobject_to(QDict, qobj); + if (qdict == NULL) { + error_setg(errp, "capabilities %s not an object", caps); + qobject_unref(qobj); + return -1; + } + ret = caps_parse(qdict, ver_0_0, errp); + + qobject_unref(qobj); + return ret; +} + +static GString *caps_json(void) +{ + QDict *dict = qdict_new(); + QDict *capdict = qdict_new(); + QDict *migdict = qdict_new(); + GString *str; + + qdict_put_int(migdict, VFIO_USER_CAP_PGSIZE, 4096); + qdict_put_obj(capdict, VFIO_USER_CAP_MIGR, QOBJECT(migdict)); + + qdict_put_int(capdict, VFIO_USER_CAP_MAX_FDS, VFIO_USER_MAX_MAX_FDS); + qdict_put_int(capdict, VFIO_USER_CAP_MAX_XFER, VFIO_USER_DEF_MAX_XFER); + + qdict_put_obj(dict, VFIO_USER_CAP, QOBJECT(capdict)); + + str = qobject_to_json(QOBJECT(dict)); + qobject_unref(dict); + return str; +} + +int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) +{ + g_autofree VFIOUserVersion *msgp; + GString *caps; + char *reply; + int size, caplen; + + caps = caps_json(); + caplen = caps->len + 1; + size = sizeof(*msgp) + caplen; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_VERSION, size, 0); + msgp->major = VFIO_USER_MAJOR_VER; + msgp->minor = VFIO_USER_MINOR_VER; + memcpy(&msgp->capabilities, caps->str, caplen); + g_string_free(caps, true); + + vfio_user_send_wait(vbasedev->proxy, &msgp->hdr, NULL, 0, false); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + error_setg_errno(errp, msgp->hdr.error_reply, "version reply"); + return -1; + } + + if (msgp->major != VFIO_USER_MAJOR_VER || + msgp->minor > VFIO_USER_MINOR_VER) { + error_setg(errp, "incompatible server version"); + return -1; + } + + reply = msgp->capabilities; + if (reply[msgp->hdr.size - sizeof(*msgp) - 1] != '\0') { + error_setg(errp, "corrupt version reply"); + return -1; + } + + if (caps_check(msgp->minor, reply, errp) != 0) { + return -1; + } + + return 0; +} From patchwork Thu May 5 17:19:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839836 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E5887C433F5 for ; Thu, 5 May 2022 17:41:42 +0000 (UTC) Received: from localhost ([::1]:49830 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfTx-0004GW-UC for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:41:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52708) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0q-00066c-91 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:40 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:16862) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0i-0002CN-NR for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:32 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245EeRAA026258 for ; Thu, 5 May 2022 17:11:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=Jesr9CV9S2p218sXK0m1lCBen6sVEv/udbrNHytm7+I=; b=nMB+yUPaVdvznTPwc59K/WPrqSTCz5WqxQQyQETyYkMT4mb15D77o1V51uF672cmXH19 n/DneVML94x1DyDKmqlUrTPEFnnSwWoTIYCngj+fKJpsWFSu69xvOaLa7ZjwzsT3aCAe GqTkKTcaFE6PqDTMgLoNSfBu90yK6vRLgG4ahmR549z3Hub5qWLdLM2zfpsImVM9fLXC Qqtexr3saSJLc9fanFZGsNPYnMAEotPwIT84RNVyr84RyaJI74eLufWjY8ExcDKJizw3 JseZXyStD4LYWi8UABLrkea7SneKD2FBm6h7GWALwyEBK1GvfdBbMHFdlKLfQQ1IMn+5 YA== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruhcbs68-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:26 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1bPJ018925 for ; Thu, 5 May 2022 17:11:26 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fusah2e9b-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:26 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=aTex0FV/NFFHGgIAuboRoI1tocMl9+mE2Nr22cO7kpGmsVdgbUi3it7ElTEgBQnrsUJUPk7G6s1uBWx9kjzVyHpDhN+wBgQpf6OWLcKLnp1LAvWiFS6B6zaiFKncV97HLBmQQOMA0ogSCOy76dC2XVeEzhrOq+F7VoGU9vSoIPRKIF7ZbwpAOaFqF+50r00IqkPRlA9FoYQZsr7UhUyC6/b6Q3vJZ7VWc1yeRONps2xh2NgD5L4FCzEmVB9nEV9aqy4lfvf0+IBZdB5b+YN0T+3Gk1+WkPY5MMuLi/lO7O4khXiLYr8mfTbJWp4pYwbz4IQNaXUqHpMdivRvkpzwMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Jesr9CV9S2p218sXK0m1lCBen6sVEv/udbrNHytm7+I=; b=X/gfMEuGRoY4YP7pNK4u3NtLSyT6b7eQnOggAEVqCtvR7bV2TKgnXwYFtwF80w4AA0b4d93jvfU1wHbfiBmnZQpvyRY66Sg/owD2SgvBG+DV6hhzFs+VlSvYFGIHK8/Ij99YAGmH+9G74BNFsfcOaisZwpjQJG2T0GI/yNfZYc1cADzFcYzoscTysZVH58uCptlyV8ZRnYYQBMDsw+MIMJR7O9pXPKHpCa7uVIRFNeufEX0P+bP5xBDX9SkqMt7aPtziRVzjpFoBINNoaP+R9IkNCZsT1ZuejBx0iQCUF/f+E33+6mB3GfQ0OxlfnjXW0aSHMCYREL0nMmlvPtdTug== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Jesr9CV9S2p218sXK0m1lCBen6sVEv/udbrNHytm7+I=; b=MCVAhlWW4Ai4yNO8zFe9NTzJabYbgbF5jQnL1kSDEV6J0AuyU7QaoIvXC7BT025FmG75qd2pbZ0gHGnAhkJCMOWJC4P8Da9LzFMSazdXLkFJR5qMfxgz8B4JGNGl+x0aI1HBCYaqT6Yu6bfxTcBRLAdtQyhL6YDS/FS3frzsz7A= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:20 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:20 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 10/23] vfio-user: get device info Date: Thu, 5 May 2022 10:19:53 -0700 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: a43ec9f7-1688-46c0-b70d-08da2eba4540 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: yJ4gnbiFBSjbYuDn2lYLGEFkZxg8ldjHKqQcL5i1QlcRGYvtBn4t1fRMD96GvbADxQ3bIdT3rGIOPmN7zmV89k1kFFnqNojHzU+iZVfvUMZu7uN2Zt+Pm0I85h9EbT+qtCeUIpk5u2nJteBVkua+MQOZbw+ruUTw7gXQZ0mKvZHpbYvVNuU9sjT6WisY/Q8rq/+Oio0s8zJguqXXCT8gYL01O4A6bwkRar4IX+IrbPGapON9Km9Wuq69juwZq5qeSeqcDiCDab4tugS8J/OZEAk+C0ZFP2P6IxFTK9PJsXUAimf6uY15BqXQX8yvIOsLO40I2+sifcvgtbrIpHbIwJ2lGGTvxNb/UhpfIZHizEYaje7AGEwU+AOqoThHw1V1+5v3Zqeik+7lXsR5RYNpuNdd35t5XIXl5gSLQ6PKtE02V4V2LIhNG91Xv+Efx+1igpHrArDRWha5ddUJdQq3kgOm1y9PygjLEH9jpgJNDjSsz1MJJGgAnTWVHBdz/XfKfBCaFXTtzRdrSzbZ0Ru2Ii8lgOLTXdXDyqagBR0j1S9XR4+rW7svBmOrHE0/gdN2DCqoDGSQtjQkAqqspO2CHC8eG9BwqU0yob3pQy6W/AQ/7BkzUL8tOfkYMBDgEyfWzW+0zSwwQz+K7tCBILezDNDrbjFkwqzW30KcwbIzJTXK4Mq5pb+Z5knAXCxNY6U58INKlQyBS/dVJviEWY+Lhg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: IuIaFnHVSl4PKC8Op37SWERlut1jLvn34+w1qQBSCgiWTJlZdDGjFTgFD0LD1Uk0nlu8XjY9jLPXN6y05EyXNoRcLBPE3PEPNw8kQ/NxWn9mHddjaVxl8My/MZzEX3xxa8CRS5c+J6DpW9ctaLuEA5Nj0x+VE+AQhwrc5Iw01XT8srkvvjQLq3ZjdSQ9fYbSP/em6nz8Fo1Fzq82E2iQsPRs1RKl5zs0LWMqY2ttg1ng0zxuo5+ecOQPfG6/EKbES4ght7ceF3dOi9LEHapyjP8vLa+q3mOYGrXDashlqkSZOujyS3bDtobj7gipW6Hulcy/1NXmhf9oLsF0li+mYRlmMKdpQKp++NaMgNCxMfx1QX8LXcoaz32UXxmNaoCWaxj4qYwk7AHddcU2qYOZZYhruw7W94teDGaneNfC4t7haS1FxzoDVA/veSCU4gpf/A9hhZvYco+5C/paY7uF9ddvEYzup44YMWN8IfnmE1rnqXnniqxNBoDC64t+lakScIbWitf5G4d9eJvKJBeM3uJRUAws1j55Z6fuGLJ5+hpTdLeAt5cO4eYI5KUb38MwJV9B2JOIDL/yndyJgWc4oTS3FyUnvaIvI8+BYBCTGEL114zlzetHz/02+2Sw/+jf+T0JkzgUJCOPDlQADPnvpaNckIWDt4k0QgXunkoz5m1mcVNZWj2wqOhLWf+oRzgoePgqYuzWnSS2yL142VCZjdea/p0iB20ataRJDpV8XsttKLQdso7RuTkwDtLxAUtC5ERu9GC5koXiLLELcocH7RjZQcL0ostmnKTceW/TJ0yh75OwBcOg0RFjK1i9lejWtRQBFar4fTMUeMZxAmJCzRHyApD7pNO+SZLpKMCrPqoigGJFTXuwNk8Gy/or+tJ9MovXLkMhU1GZZfOsDXu91s+GKPF880JycmZhdbyaSCSZm+9f1h0t4LYv5RhRSLvGEo4bMM1uj+6/mFu+cx0UShPm6nqOP5gTbTWm7qwiZuHtaYXDd5xlrD1FG7FX+knK4oSs7NqvouhYOXQ3a5/qCx9yztcJNc08JEUYSfwZHXucYHWD5xD6s1D4yFLl1fGekAKRSwtVt0PFGzWvyaOgzXqf4BKznQ4SKy6g8KgAG1Ndue5bNaDq9zCZvTd/OA6xRtmZoaS47TY6Iv+80AVNr4wNCbevoDLko9ioc5jH4Cp154og1kUKoOg6Vhkpp/dBKAo34Q298u6epEHri0FfOMhpt/A4ar1Lx5245KsvlLS+63oBG8v17UCDi3KSRc/wkdgL6RdEW5J6p7pDESdXLkONe2tWJTUEQV0iTza53t2RQroe/q1X7TxpMoX8QfNtlqjbGilKbwcct5lDfZdRaiap0B8WN997uXcvY4N3Zu4mSATIZCXbKmnpkau5veJU+oi8PIgJi/W7EBtv2r5/nWedYmyDx8IHg3WTv20YFJPSwkPruoprBRGsyXFDGzPnIND+To3bRCcTahVsyCjj8xSgPTUh8EqD+Xx2oWR4qQnH73B0K5B1ET0WZgUM099/uDLmDo/agcYPX8FAgAqoC7Rz3mn6QEhwtRdggKRwjP5NQFRq1YzawUTgomNNJEIY5QDI5DMtnBFKAnBzbTVCdhBom32CAhFWwzX2J9f4uUSUqbZw8SV3wKQtuQa7TiWtkV624n932Ki2RR2GyzX5lqz05BIr/yJHmRjtpeJeOroy2z3mead72BN+Cs9LdiP4yBeMcseH9LSEmn/nymKg0DMkeKCHuFSbwXLZjI9rtxM= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: a43ec9f7-1688-46c0-b70d-08da2eba4540 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:18.8572 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: N1xfvC5h5gu0w20bgH7oFkcFI/zJNIaQZqXeV/ddyT0W4jjne+b/womr3f/Vsn9BNiH6pAb7FoGkNDahv5pHAVRcLWJ22mK5Jy0PymLK9go= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: HLQF19FUBZp27k3vsyD5OcTEU-tZUt3R X-Proofpoint-ORIG-GUID: HLQF19FUBZp27k3vsyD5OcTEU-tZUt3R Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 14 +++++++++++++ hw/vfio/user.h | 2 ++ hw/vfio/pci.c | 26 ++++++++++++++++++++++++ hw/vfio/user.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 96 insertions(+) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index a0889f6..4ad8f45 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -92,4 +92,18 @@ typedef struct { #define VFIO_USER_DEF_MAX_XFER (1024 * 1024) #define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) + +/* + * VFIO_USER_DEVICE_GET_INFO + * imported from struct_device_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t num_regions; + uint32_t num_irqs; + uint32_t cap_offset; +} VFIOUserDeviceInfo; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 00d21bf..633b3ea 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -85,4 +85,6 @@ void vfio_user_set_handler(VFIODevice *vbasedev, void *reqarg); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); +extern VFIODevIO vfio_dev_io_sock; + #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7e5b910..68d6f0c 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3416,6 +3416,8 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIODevice *vbasedev = &vdev->vbasedev; SocketAddress addr; VFIOProxy *proxy; + struct vfio_device_info info; + int ret; Error *err = NULL; /* @@ -3455,6 +3457,30 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->fd = -1; vbasedev->type = VFIO_DEVICE_TYPE_PCI; vbasedev->ops = &vfio_user_pci_ops; + vbasedev->io_ops = &vfio_dev_io_sock; + + ret = VDEV_GET_INFO(vbasedev, &info); + if (ret) { + error_setg_errno(errp, -ret, "get info failure"); + goto error; + } + /* must be PCI */ + if ((info.flags & VFIO_DEVICE_FLAGS_PCI) == 0) { + error_setg(errp, "remote device not PCI"); + goto error; + } + + vbasedev->num_irqs = info.num_irqs; + vbasedev->num_regions = info.num_regions; + vbasedev->flags = info.flags; + vbasedev->reset_works = !!(info.flags & VFIO_DEVICE_FLAGS_RESET); + + vfio_get_all_regions(vbasedev); + vfio_populate_device(vdev, &err); + if (err) { + error_propagate(errp, err); + goto error; + } return; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index dc3f1a6..51e23dd 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -30,6 +30,13 @@ #include "qapi/qmp/qnum.h" #include "user.h" +/* + * These are to defend against a malign server trying + * to force us to run out of memory. + */ +#define VFIO_USER_MAX_REGIONS 100 +#define VFIO_USER_MAX_IRQS 50 + static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER; static uint64_t max_send_fds = VFIO_USER_DEF_MAX_FDS; static int wait_time = 1000; /* wait 1 sec for replies */ @@ -984,3 +991,50 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) return 0; } + +static int vfio_user_get_info(VFIOProxy *proxy, struct vfio_device_info *info) +{ + VFIOUserDeviceInfo msg; + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0); + msg.argsz = sizeof(struct vfio_device_info); + + vfio_user_send_wait(proxy, &msg.hdr, NULL, 0, false); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + + memcpy(info, &msg.argsz, sizeof(*info)); + return 0; +} + + +/* + * Socket-based io_ops + */ + +static int vfio_user_io_get_info(VFIODevice *vbasedev, + struct vfio_device_info *info) +{ + int ret; + + ret = vfio_user_get_info(vbasedev->proxy, info); + if (ret) { + return ret; + } + + /* defend against a malicious server */ + if (info->num_regions > VFIO_USER_MAX_REGIONS || + info->num_irqs > VFIO_USER_MAX_IRQS) { + error_printf("vfio_user_get_info: invalid reply\n"); + return -EINVAL; + } + + return 0; +} + +VFIODevIO vfio_dev_io_sock = { + .get_info = vfio_user_io_get_info, +}; + From patchwork Thu May 5 17:19:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9B83C433F5 for ; Thu, 5 May 2022 17:37:34 +0000 (UTC) Received: from localhost ([::1]:46622 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfPx-00024R-Tb for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:37:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52644) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0m-00063S-JF for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:35 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:12892) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0g-0002CA-9L for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:31 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245EeRA9026258 for ; Thu, 5 May 2022 17:11:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=WY6AW+r55TRFa5MuXT+MOgq4tx7hZp8j9+vRSaldZ7E=; b=mjoDVygEwwQ+hQb/OLg50kpuYD3/qGJiWj7l/afOxtfBWeoWoYQt2pp48ViAR+j5nXfh 5iSY8gIXEJRdj14DlWfLg+bBZN4benAL+o0EJ6H5AQSlLhTpoIWtk4zMS1iPKWYLRbzy L0+XMcXsd1+GkbjOPEj5WOcXde+ShLxh2PyYHTktv0jW3lZBBjcvyvDl7Fxr3azBNnpw kcdXjnf3E3oGntDMS6OB4bNsGmLlhxLa4N+8ZRibAbt/rr0EVZ7/n72gaBkQ898gM3nl k5oFREDzyrSBs/BYj2P0msON1/H7xKMI4wJ3l98EPAX1kvOS7Ddy7c+dt19ZkIj/1zZi yg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruhcbs66-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:25 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1bPG018925 for ; Thu, 5 May 2022 17:11:24 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fusah2e9b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Z70nF/X+iy68WH215FlD6q/D5RVZVnRIJqBP36j17+xuqXfElcu3yucenP+d98+697yxOnthAvRkrr8FYkJDJZr7so55w/3wTJGAZNd5Jj8gEx1lY28vZ55gfXfs27gvppN2dKZPSxmL4+ZtnfEpPJHlAqgSst+Swro72hVcy9Zsek8jxwORcr3PeFnHaj5YY/tNKy6iBGbcZsSdyNdW4zGrC30holgPmMUhDVHfZFN+RggbM3y0o3SORNBFveap+rmMTtxoBw/SZdzYH8IOfvFkEtUiB5/23mjq+QDIcdWrALPEiITYTr6xsMaNU22JQJrcpy11t6yyflQCLJ7iCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WY6AW+r55TRFa5MuXT+MOgq4tx7hZp8j9+vRSaldZ7E=; b=iA/+IWTBUkLAl1I1H8SgBFEZ9COYHvqBhLo7vKgvAuYyPbiMvQjyUqbTA75U6KYtI+dC8GKKhe5M+Le+atDlzK9N/BE7GBWzlQqadrSrmpb/Xj42pmCv4ivHvfMv4CyBiH6y6B2rkfVjfFQ8bg/io9zgek7UWvb+rfpwL+HWo5BYWpCtbzJvOnl2aolHH5SCh3FafpEgKTbbac62HYPJMnbhC0aOb+bSrJ5GzRlGdv/+iajwvbIF6HzS9duZtJwRPN/Fll42LJPCShJ3tefk8KD0t6hytk76T0gyoQHR8cPC3cTCXyY9xUm62V2kx0Xgx6O8x2v8SEAxPgsCJDGgVg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WY6AW+r55TRFa5MuXT+MOgq4tx7hZp8j9+vRSaldZ7E=; b=h+7YhTfXFERywomvKEGPwp5Vo2M0b+Ynj2toSJYZ3Kl2AOxzubCgS8c0CTq8ahaTnjvU+x6+6WWtJ5mmiFBBGu7m7e98kztDqomZn1UYJvnr5zbTjvHHmVOycUJjOACrMrup1Cld641DGXPtEclBEbMjKXaRXQoAOYOMzDGF4HU= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:21 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:21 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 11/23] vfio-user: get region info Date: Thu, 5 May 2022 10:19:54 -0700 Message-Id: <1f48fe5f10f9887f46ee8f929e42e3b50c440d2e.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b437795b-c8e2-4353-8c36-08da2eba4577 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: x38Wf/izQq+URWvkNXT4ZokeiuZUTANIO6oHGO8xEiwIsxrtE+uhsQWlqa5bh+y/udR6vBl5nHXFjLPsZ65PFydrsDmbYBBwohM21JGVkoNCtjOCkBP1bPW3UGO6gaowpJwl7absgamsOY6L5Nmgd8wlLnBK55jB8vjWS0THtFCBdeRLuWPH47Lc8x67Lbi4cjpDINPnccMBckVOHI0vqE2gRRSlSHIHyUCusYZQ/CeLugywmamdUz8fu4fen5DbIka6oFxGPPtx4WdWnnkEJFoQBW0kzZpj/9Aki53aPADfeApeIqTbzXjxrntUR7iYwoOpAG8sHYr+B6pQ26L6diF6Y6L1mg+EB8/ue9nULrhYgiHP+wG9dEYat3/KPsi9fxk/iJ9nfvyXhi9xio3gteHkL46R9sC179FaAanYAQtHerc86ShWHtAGhMUNYwHi1UzS1HGIjrW4ime0OWjga72ke5s36eXrMI6tA3fGCs1GNkgv5q7obAHD7ihOVlHwjy7SYzM0AvUiCmOw0eUqvn4lshf60falICtfxOfXC8fE2osNsA/cmBSQeNj2Zoofmk67G1mlJ7AzSKDPQjlFAU1IA+7Nz6qTn6WlQD5rs6qbwrBl2r6e1ItE2kkII2ZLhsZcXBPX6hRToHq/i2qjT2sBHnJ3iPKRZAMgO21imBeYmIUpIxgWIu2lWALM0SJhFzpfU8bUG7jdTTQsvvWSHw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sfWBWGAOgYd58P2e9UySzvhiE0s4i9we17YSAQK7/wd7gZ06pNbqVleH753OMaRBJyMq7SZdFk+Q4sItfPxwtKjupYfUVxEG0pg8jX8kC2lC9xYXbpN5AwxP2uWOfniJtJFIcoNWy713d6DtbMA1UIi61BOnCvyZ4/dQpkJ8LdJMn4dwnln+MLTpuUxLFS/o8xClgB2zDd0aOvj8l0ojGNTiBZfr2d/+/Km+1lF9iuWZDKKnoAyOS/MCFQ9D0xEpP8ZlYpL4qkIGKKZuo95hovzW9DR5VKqXgTHRcHrVgLqIuRMqsYVKUcFXZ0FUHUVFzlxdQ+scnlrNlLPHJrGD3E8oKAY15LNxuXSVwhTW73Df/shV7ayJdEusinILy6g9wChYUjm0gacdqdPCOZ+KBKLu/vDL0GtkVwlqtDJW8FsxuKmvMIMBo7VtmU5aYELiT2Xx8nGsoMGvI1h3pCLQ/dzXu9XFLu8tT3iQyjtSPMeHO2lzA1aCXNJKli50EupdwJiYbfoGP6Ify3D419eYFE4Kni+5fIQdayxpD83XmmsNiFlqIWNp5pDVxKxvuYr9GDSMfbC7Vswf+syMsW8mE/GZDVoYEKB07FCZs0EicapFKcnjoGE2Vy8yZhu6WPmLF60BrtLUoMbT1OKbIgUqAOPzHiC4ruXPOpritvFo17jSZBzDknWsHMR6QE9Y5O65yrrwLyFfNbxXokPkUY9CXuhTHu9CqdCoNcdd8pjmeEQAqMaZlJndmIohXDbc+buLudaKL+XHf4D+rf/ZD7fILzizjaM+o2cDHCH9ouC2HDYmDB8cwPBVjvGT3GJKX5SRwCLo8bhu3QI6pQaWBqMmbD5IV/TbvTCoQ02gWSZ6ANg2ffQNX29leDcZAGg2hgFyqGXRPoYSLN1r93WFo1AZEl5fLWI+/e6sI61oAL0mHvjhTR3TXF8XuAA9AZhDbztGb2E5YO4e5Au+oiaFprG1R+Op7MqB7FjxeMFk2cI4cYZXRlukOXCHETJfOQORlLkWe1GDkPgay9yusAaEXihjnyrtEb0Hgs0aU2xAUOODq0Zz1+mTM3N42/TkLIgllKgm1f8FFywSaysnFp4uzCiyB6VWbf9cJbGdgOBRF/oBAvYzA8T58qDUgp23vjsw/MipPOj2kElxwv4E0cz4IZatBzDPQsNiYuGzexYcD3tlWbVpVipVQTbQv182DfpyTifena5w+kEz5XZgJywYbXJVK5Mzv8+FDd88alxrPEVuIKwe/2EA3BMRPypvYiXFxHuzb/W1IcIxLKBJBvAYYFLAPlfpR4qi7VUdhJlV5K4dltKsgTsCZ+7DFFar2NdSoWlR85pdqA4aOolkNXLDUZziPgt/7VmzwiRWRABnfy1mwoSkan0dWbWFsWoOesACwGF4GaYEUpTGrb37SrLY2G+H4d8E6nZ6FPvWRvvwcoKwzHfTTpIFncQBclbn6j14WOYmmoDtX9mT8T55fMLWcG86MZ02mg504Z+WUPT3TazmbFDxKpHVGUoo5W1w9hoUnCMhi4HBJ7QrZm9HaJ/9/D9ZAxctXSUKRnBoY7312sis1aiucwnCafy2qTwAEyfBYBlv56bUvHf3/ebAJaevmp8C1vwrnNLJ9sV0iJ+q1dGetohSipC4hCVNX6tCbWEal0WF7pI5TRpPtuJFTfQNZUJTu6UTLuyzvmYgfGL4Cxqz4c0biWpFEn86IfDUNvq8hcFm6cB/xBsiFn9s21Qas337hlFnwHM/Ti1eonkuT97q+do= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b437795b-c8e2-4353-8c36-08da2eba4577 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:19.2010 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8C5wcx7HSzbmVjaDbjl3bKE5bSIT7fuogbGNwJR9L9p5jys90mR0t46p3+uzEO1WDPx3O6Zpet2EOwGoVjP3kEzLya6kc0e5gjBpmG5mEcM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: EMAKumZUNKRb_v2jIBVY0024gePeoCl4 X-Proofpoint-ORIG-GUID: EMAKumZUNKRb_v2jIBVY0024gePeoCl4 Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add per-region FD to support mmap() of remote device regions Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 14 ++++++++++ include/hw/vfio/vfio-common.h | 8 +++--- hw/vfio/common.c | 32 ++++++++++++++++++++--- hw/vfio/user.c | 59 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 107 insertions(+), 6 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 4ad8f45..caa523a 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -106,4 +106,18 @@ typedef struct { uint32_t cap_offset; } VFIOUserDeviceInfo; +/* + * VFIO_USER_DEVICE_GET_REGION_INFO + * imported from struct_vfio_region_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t cap_offset; + uint64_t size; + uint64_t offset; +} VFIOUserRegionInfo; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 3eb0b19..2552557 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -56,6 +56,7 @@ typedef struct VFIORegion { uint32_t nr_mmaps; VFIOMmap *mmaps; uint8_t nr; /* cache the region number for debug */ + int fd; /* fd to mmap() region */ } VFIORegion; typedef struct VFIOMigration { @@ -150,6 +151,7 @@ typedef struct VFIODevice { OnOffAuto pre_copy_dirty_page_tracking; VFIOProxy *proxy; struct vfio_region_info **regions; + int *regfds; } VFIODevice; struct VFIODeviceOps { @@ -172,7 +174,7 @@ struct VFIODeviceOps { struct VFIODevIO { int (*get_info)(VFIODevice *vdev, struct vfio_device_info *info); int (*get_region_info)(VFIODevice *vdev, - struct vfio_region_info *info); + struct vfio_region_info *info, int *fd); int (*get_irq_info)(VFIODevice *vdev, struct vfio_irq_info *irq); int (*set_irqs)(VFIODevice *vdev, struct vfio_irq_set *irqs); int (*region_read)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, @@ -183,8 +185,8 @@ struct VFIODevIO { #define VDEV_GET_INFO(vdev, info) \ ((vdev)->io_ops->get_info((vdev), (info))) -#define VDEV_GET_REGION_INFO(vdev, info) \ - ((vdev)->io_ops->get_region_info((vdev), (info))) +#define VDEV_GET_REGION_INFO(vdev, info, fd) \ + ((vdev)->io_ops->get_region_info((vdev), (info), (fd))) #define VDEV_GET_IRQ_INFO(vdev, irq) \ ((vdev)->io_ops->get_irq_info((vdev), (irq))) #define VDEV_SET_IRQS(vdev, irqs) \ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index da18fd5..c30da14 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -40,6 +40,7 @@ #include "trace.h" #include "qapi/error.h" #include "migration/migration.h" +#include "hw/vfio/user.h" VFIOGroupList vfio_group_list = QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -1554,6 +1555,11 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, region->size = info->size; region->fd_offset = info->offset; region->nr = index; + if (vbasedev->regfds != NULL) { + region->fd = vbasedev->regfds[index]; + } else { + region->fd = vbasedev->fd; + } if (region->size) { region->mem = g_new0(MemoryRegion, 1); @@ -1605,7 +1611,7 @@ int vfio_region_mmap(VFIORegion *region) for (i = 0; i < region->nr_mmaps; i++) { region->mmaps[i].mmap = mmap(NULL, region->mmaps[i].size, prot, - MAP_SHARED, region->vbasedev->fd, + MAP_SHARED, region->fd, region->fd_offset + region->mmaps[i].offset); if (region->mmaps[i].mmap == MAP_FAILED) { @@ -2410,10 +2416,17 @@ void vfio_put_base_device(VFIODevice *vbasedev) int i; for (i = 0; i < vbasedev->num_regions; i++) { + if (vbasedev->regfds != NULL && vbasedev->regfds[i] != -1) { + close(vbasedev->regfds[i]); + } g_free(vbasedev->regions[i]); } g_free(vbasedev->regions); vbasedev->regions = NULL; + if (vbasedev->regfds != NULL) { + g_free(vbasedev->regfds); + vbasedev->regfds = NULL; + } } if (!vbasedev->group) { @@ -2429,12 +2442,16 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) { size_t argsz = sizeof(struct vfio_region_info); + int fd = -1; int ret; /* create region cache */ if (vbasedev->regions == NULL) { vbasedev->regions = g_new0(struct vfio_region_info *, vbasedev->num_regions); + if (vbasedev->proxy != NULL) { + vbasedev->regfds = g_new0(int, vbasedev->num_regions); + } } /* check cache */ if (vbasedev->regions[index] != NULL) { @@ -2448,7 +2465,7 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, retry: (*info)->argsz = argsz; - ret = VDEV_GET_REGION_INFO(vbasedev, *info); + ret = VDEV_GET_REGION_INFO(vbasedev, *info, &fd); if (ret != 0) { g_free(*info); *info = NULL; @@ -2458,12 +2475,19 @@ retry: if ((*info)->argsz > argsz) { argsz = (*info)->argsz; *info = g_realloc(*info, argsz); + if (fd != -1) { + close(fd); + fd = -1; + } goto retry; } /* fill cache */ vbasedev->regions[index] = *info; + if (vbasedev->regfds != NULL) { + vbasedev->regfds[index] = fd; + } return 0; } @@ -2623,10 +2647,12 @@ static int vfio_io_get_info(VFIODevice *vbasedev, struct vfio_device_info *info) } static int vfio_io_get_region_info(VFIODevice *vbasedev, - struct vfio_region_info *info) + struct vfio_region_info *info, + int *fd) { int ret; + *fd = -1; ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, info); return ret < 0 ? -errno : ret; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 51e23dd..c87699a 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1009,6 +1009,40 @@ static int vfio_user_get_info(VFIOProxy *proxy, struct vfio_device_info *info) return 0; } +static int vfio_user_get_region_info(VFIOProxy *proxy, + struct vfio_region_info *info, + VFIOUserFDs *fds) +{ + g_autofree VFIOUserRegionInfo *msgp = NULL; + uint32_t size; + + /* data returned can be larger than vfio_region_info */ + if (info->argsz < sizeof(*info)) { + error_printf("vfio_user_get_region_info argsz too small\n"); + return -EINVAL; + } + if (fds != NULL && fds->send_fds != 0) { + error_printf("vfio_user_get_region_info can't send FDs\n"); + return -EINVAL; + } + + size = info->argsz + sizeof(VFIOUserHdr); + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_GET_REGION_INFO, + sizeof(*msgp), 0); + msgp->argsz = info->argsz; + msgp->index = info->index; + + vfio_user_send_wait(proxy, &msgp->hdr, fds, size, false); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + memcpy(info, &msgp->argsz, info->argsz); + return 0; +} + /* * Socket-based io_ops @@ -1034,7 +1068,32 @@ static int vfio_user_io_get_info(VFIODevice *vbasedev, return 0; } +static int vfio_user_io_get_region_info(VFIODevice *vbasedev, + struct vfio_region_info *info, + int *fd) +{ + int ret; + VFIOUserFDs fds = { 0, 1, fd}; + + ret = vfio_user_get_region_info(vbasedev->proxy, info, &fds); + if (ret) { + return ret; + } + + if (info->index > vbasedev->num_regions) { + return -EINVAL; + } + /* cap_offset in valid area */ + if ((info->flags & VFIO_REGION_INFO_FLAG_CAPS) && + (info->cap_offset < sizeof(*info) || info->cap_offset > info->argsz)) { + return -EINVAL; + } + + return 0; +} + VFIODevIO vfio_dev_io_sock = { .get_info = vfio_user_io_get_info, + .get_region_info = vfio_user_io_get_region_info, }; From patchwork Thu May 5 17:19:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839826 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 19E7EC433EF for ; Thu, 5 May 2022 17:33:23 +0000 (UTC) Received: from localhost ([::1]:36784 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfLu-0003bA-1n for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:33:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52638) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0m-00063P-HF for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:15002) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0h-0002CE-51 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:31 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245F4QNL026132 for ; Thu, 5 May 2022 17:11:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=5aBUYX5OmAUgvb/9DYxS76pyxsIGUeT6S3INotjWpk4=; b=zR1/06KWyLEw0Rly0q6gU5QpEF3wiXtdCwvjlhvf/aCgZ/PmrI7V7XprW1NkE0tmUeff zmh8mbb2zyIL4d7dHpQ50UCZzL4WAatMJ5R2YicYf/eAaOzf4v4i/OB0LgUdewGVoLQl QqD01m2TB3NDyLP1Oi1/QfuHJS+vdmwacAXeeSifPo8JpWjU1lj0vwFaKv1blaD3XFKq Ji6gjZW+oQquK+bArh64gBTFdbzMTpXkwjBIxV/v6iKFI6qaoNqp6qHJ/VEPJsx0wJXF +Cw0LvQPiB8SUI3HpI9qOBl2Dp/hXHzo97y3sc0VTpCWvmH/gdGBt44wUCuzeC+dqxdL +A== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruhcbs67-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:25 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1bPI018925 for ; Thu, 5 May 2022 17:11:25 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fusah2e9b-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lcG5zkUkCrtFIlDjM3ZP50Zaub02doSKTZuxEPOpFpDHaD0rPruVPaNYIBBcOJnTTaiC/S3dptS4rcEsd0/R2QDeBziAnxR+nm54HHPcHsSx35hUM92aDoxWqZCjDv1bbZsYKmYOkrYjkAuZA++L06ABwAoQrGhyYmom/BIoY16XW12ydy1ftL8YMvj2Y2JN/vJhXgxlLb/oOZU6cFK4VzFWt3G0R+KQ+YU8RwzdN4GjsFYqozw65umuqD8+ylaPg2Xonm7KTkgi04eeiHCw8B/mfMxhGRDDHP5Btnwk9DYOKUwe4RrXkA1G5GStTV2NiFu/JB4PQAQo4ZZupW56Aw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=5aBUYX5OmAUgvb/9DYxS76pyxsIGUeT6S3INotjWpk4=; b=IZgBVG0k+OTl1uB/Tlcid9Y0jpuZPIlsZgPvJzujAjujlqAFwmNaexx2fcMdRi+M/ijPZ1mQ9Nb7oqhSPT3zRbiiyzl5xHA2oSErbaCK/X696VsoirxZixHyNo+aIsthyQSjAnhR7jAzdnW9qlSdzsu1jqGunck+UqEY1nn99tpJ+F3eeOxlK75XIAa8SGfQgorB+Xi0RNeg+SjGEJ/3gvVHHYwVsrBoRWe+/OSyxMBL2cGcMerM2KmzcJy4rKc+70JdUSWI/2rRuLtomumxmQIZpQScssNc892YIP4/W78Xfd22AB5fCX1e9S0+IbWS8YEYZkpyjn7cZ47lpkr6MQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=5aBUYX5OmAUgvb/9DYxS76pyxsIGUeT6S3INotjWpk4=; b=FxvXxPPcedQ+reSjofAecdcON94MTbpZJpf1XqCmYb36//K+mIci6Cx6x1KaR/X7rxTiTtUOEuc8Uy0nIIh8aiHKGoQeXUDavGXhrxFLlnazmzzFyLFDgfernY0DsjY8NNLgxlooA0pQy28jP2GfwupcrwJ53i+YibQCqKWkdp8= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:21 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:21 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 12/23] vfio-user: region read/write Date: Thu, 5 May 2022 10:19:55 -0700 Message-Id: <8c4609c33a9f7d5289d0c3a86697b4b22bceb72d.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 553300cf-1663-4b06-6171-08da2eba45ae X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: QXXLUqMqnxp2bLkfVTFzDJEVYiQSnlI+sIzS9dtqLNBJYuxjVsWC1rjuGqkMjf/68fe8i7qjS3M0m0X/DtMKrGc10R04rQM06YAz5TXxgdb4ZHoX2rXIxfq/EL5qvjXPzGFvNI/Ea2UA10aZtn9+OleZwSsnCCU+ADFr5QVPg2xaD3GkNndk7AI4v0cG+3oU6+gr+NQkAEHUSxtC0vCI+uCeDqxRIW1b688ECnOBsbbWjLJm3hyNUHrluy5EECFejd0/od1cg8jXX/50HJUrQJ8BYcDqe4PXHphZ395fjcyRWmhmNpMnjn4VfW+qi3uDUfEyeZr+/7VoXDccsbJrwpsIIaOFeowFXnxb0w8WGYsrTlIcirglK1aIwA5B8IstQpcVeAXLl5CTU9uv6Ocp9d3Anc/I0kzTXyrMngAAI5ma/FPYAJzs1ecIZKSeiLdDHjS5RS2e0ngq3u/2RASc0urIisVXB4oPMxxQ0jOWRkP5D/wwa3aXhQc3zaoRP4UQGYxB1ny5iJ6787G4HOamIce9oWlrappFaODBAKEmEh/YDYexr3NVxCwZEjUjTFW+2Jo4lPiM7EhhTQCfTOwiKYiixWb8C6v3u75nrYSJUSBIpRi+OtBAQG0YRJjzYx2tyieSXdrYVFHAOvOgXGjqEu8q9ETCK3OjTM/LuutdCKleDd7/g3Av5i6nEXB/gaC9fwj02DbZx/aeqN89od5gFg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ucM7uVCz3pl+1QraGV0WW3rgG/H1IbgCGxB1w1zaWjA2jA0J9+dz852BCJGlMJxzKHeniclKkerUx7Eor2uVicjWKX2BH2yppwGPmolQGtVpZEvxKkTiXGz0Uwfyw2ApebPSBcYcmw7bA7GrUuhZDccjpac3dNA39RLH2Ye4BJdF0S3/CBHDoY6dGJpxvig6fXmXlkoxoF7sY2H0IzZTvDeXgF9zMUTqDl/7I/65Zm1v1iaWWr0EgCIgX/amhG9VbUWQ68wVrKQNJq9v8dvuogSWRvSk6ChlvdKAsZduWRTTDwBzG+vkkzyIkMsdHuA3WEx7mw3UkAu0ulLbINcJ/qkiYMVoKATvY/qbA1UBwgzwcGLPjzF8+R9b4sQZZejArGrm943f/HD4IXjeGYnu6ojFbCUcWqIJoHelUKc7VzzbLVNlF9wQoN1Bi+e4V+C9DLCbi7z7hpZ/68x4JsPgPAocE1QV2Av39aJngMsU7gGZjaooUn6g6hxslAJayAu4dQ5cF4nHLNSnpnPwwkX0ha4+XGI8PGLdUhmL2ZMm2drUagPm77IjUBkONyOEmRMoDD6JrOltIWN1jKcc/ZU2YB08QNOznRj5w+DzSMvfCMOK/QIZvT5XdLtcCXKoida44xCRw44nrYJFBLUiCANr/Wz4dp/dNYFN8T9/o6PHd/hP34fNojeQfH09NzTxiACVaGd+Gf1PunaPFi/qzi1jkaHxm+VJXAy6kVE3aquJtSSRdlESUT3hO+oNS/vwuCwlDieNgYYld900ytw1u5dA2SFW+XZexDGUDRUBdy0K1eqGllr4jqUn4YRW2jx8H0wrPjQjMHNqN3hyqBl3ImbL8NCBeGr/j0pAE6x42V+Et7jg4L1DtLoc7o+uCroZrL1hmwVcbGVgdEDP8gVRM3/xKVs3dPopcG8q2C8o+mmxn7nZ96vCHh972O+9GD6KG4QaHPRtVvFEhmdX+RTbAyAL+bfjkofgb7aGr18G6prlVVCLMwVSCQVzwazuc7aP6XmvGcRNFpTCsuDdC3aRq8mDlQDfcJQ5v/DvaBcsHVtTnaakWGdPBsZBKiUg2JiHLP0uhMQzIqi1J0bsDcAr+VvRz7atT+3ZV+djrOj2cdJL3zbGi22ogt4f9g+9hedcaQ/DpQp6df/QeywrTYyeW6/3zkWp2A7tpdSHiGZObsME87upsTXmSHBUbC8qLdMhrnu30lwt18cf21XrnGMFgMVTDcw0g5Is9TSdXjVpk3SRXMvy0RRLydjFmlGsT0+mMt2aV4FT/O0Tq3Yzr7tVwC+eCVZ/+DbmTlRLtuWnHHWdJgkQHMfLEnnZfQuhl8jblEvGwvaiUrNl3RKsFyNXb9Ezgamf0tNlW6EdYH7WKCHWfToUW91hRyRZk8cdLS9gklMOjtPQ8U2gE8j+34GjC4LuhKK1VUHLzNYjFddBImPPYrDQPbNSrCzIBDeHA7c+jRn/WNEt6vaicFCn7C6uQj6BW97lWKlzPPBXouy0nAVeCESqWEavdxsMptol57YRKSTuYUx6ymwKjYmXkheeT9RyjDILU095tucPTpZb+v27Y8Khq3BQ8ocoLBJEUc1sNo5Ra31H1k3px98BSFyboABri9VoqujU+M4wQBawvT1A7dXFUYxXGPPZQCXKHOoH03C8M+Xg1G2XlOJoY/wik63taFkj0Axk15b53WI1DIEss9JggHbZWeMU2T+YvN54kc9aGgoNWpw+HLHAp9crwD1hjQwkXWcD5k6bI1I9VUMbhXA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 553300cf-1663-4b06-6171-08da2eba45ae X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:19.5447 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ZlfAJK5vOey4HcWLKBIdGODzq6WpA29fFz4+I5NHo/dkdd9wpQj+whFvVdDso05jGZhaWNcB8m86H1ZzkVsw4roZsv5S4WJqARBeTMnF10k= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: -iWmQ9DRPtXVzUwaLTYQdxe-FTdlodtl X-Proofpoint-ORIG-GUID: -iWmQ9DRPtXVzUwaLTYQdxe-FTdlodtl Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add support for posted writes on remote devices Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 1 + hw/vfio/user-protocol.h | 12 +++++ hw/vfio/user.h | 1 + include/hw/vfio/vfio-common.h | 7 +-- hw/vfio/common.c | 10 +++- hw/vfio/pci.c | 9 +++- hw/vfio/user.c | 109 ++++++++++++++++++++++++++++++++++++++++++ 7 files changed, 143 insertions(+), 6 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index ec9f345..643ff75 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -194,6 +194,7 @@ struct VFIOUserPCIDevice { VFIOPCIDevice device; char *sock_name; bool send_queued; /* all sends are queued */ + bool no_post; /* all regions write are sync */ }; /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index caa523a..b1ea55f 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -120,4 +120,16 @@ typedef struct { uint64_t offset; } VFIOUserRegionInfo; +/* + * VFIO_USER_REGION_READ + * VFIO_USER_REGION_WRITE + */ +typedef struct { + VFIOUserHdr hdr; + uint64_t offset; + uint32_t region; + uint32_t count; + char data[]; +} VFIOUserRegionRW; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 633b3ea..a641351 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -77,6 +77,7 @@ typedef struct VFIOProxy { /* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 #define VFIO_PROXY_FORCE_QUEUED 0x4 +#define VFIO_PROXY_NO_POST 0x8 VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 2552557..4118b8a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -57,6 +57,7 @@ typedef struct VFIORegion { VFIOMmap *mmaps; uint8_t nr; /* cache the region number for debug */ int fd; /* fd to mmap() region */ + bool post_wr; /* writes can be posted */ } VFIORegion; typedef struct VFIOMigration { @@ -180,7 +181,7 @@ struct VFIODevIO { int (*region_read)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, void *data); int (*region_write)(VFIODevice *vdev, uint8_t nr, off_t off, uint32_t size, - void *data); + void *data, bool post); }; #define VDEV_GET_INFO(vdev, info) \ @@ -193,8 +194,8 @@ struct VFIODevIO { ((vdev)->io_ops->set_irqs((vdev), (irqs))) #define VDEV_REGION_READ(vdev, nr, off, size, data) \ ((vdev)->io_ops->region_read((vdev), (nr), (off), (size), (data))) -#define VDEV_REGION_WRITE(vdev, nr, off, size, data) \ - ((vdev)->io_ops->region_write((vdev), (nr), (off), (size), (data))) +#define VDEV_REGION_WRITE(vdev, nr, off, size, data, post) \ + ((vdev)->io_ops->region_write((vdev), (nr), (off), (size), (data), (post))) struct VFIOContIO { int (*dma_map)(VFIOContainer *container, diff --git a/hw/vfio/common.c b/hw/vfio/common.c index c30da14..351f727 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -213,6 +213,7 @@ void vfio_region_write(void *opaque, hwaddr addr, uint32_t dword; uint64_t qword; } buf; + bool post = region->post_wr; int ret; switch (size) { @@ -233,7 +234,11 @@ void vfio_region_write(void *opaque, hwaddr addr, break; } - ret = VDEV_REGION_WRITE(vbasedev, region->nr, addr, size, &buf); + /* read-after-write hazard if guest can directly access region */ + if (region->nr_mmaps) { + post = false; + } + ret = VDEV_REGION_WRITE(vbasedev, region->nr, addr, size, &buf, post); if (ret != size) { const char *err = ret < 0 ? strerror(-ret) : "short write"; @@ -1555,6 +1560,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, region->size = info->size; region->fd_offset = info->offset; region->nr = index; + region->post_wr = false; if (vbasedev->regfds != NULL) { region->fd = vbasedev->regfds[index]; } else { @@ -2689,7 +2695,7 @@ static int vfio_io_region_read(VFIODevice *vbasedev, uint8_t index, off_t off, } static int vfio_io_region_write(VFIODevice *vbasedev, uint8_t index, off_t off, - uint32_t size, void *data) + uint32_t size, void *data, bool post) { struct vfio_region_info *info = vbasedev->regions[index]; int ret; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 68d6f0c..98520dd 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -51,7 +51,7 @@ (size), (data)) #define VDEV_CONFIG_WRITE(vbasedev, off, size, data) \ VDEV_REGION_WRITE((vbasedev), VFIO_PCI_CONFIG_REGION_INDEX, (off), \ - (size), (data)) + (size), (data), false) #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" @@ -1661,6 +1661,9 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) bar->type = pci_bar & (bar->ioport ? ~PCI_BASE_ADDRESS_IO_MASK : ~PCI_BASE_ADDRESS_MEM_MASK); bar->size = bar->region.size; + + /* IO regions are sync, memory can be async */ + bar->region.post_wr = (bar->ioport == 0); } static void vfio_bars_prepare(VFIOPCIDevice *vdev) @@ -3445,6 +3448,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) if (udev->send_queued) { proxy->flags |= VFIO_PROXY_FORCE_QUEUED; } + if (udev->no_post) { + proxy->flags |= VFIO_PROXY_NO_POST; + } vfio_user_validate_version(vbasedev, &err); if (err != NULL) { @@ -3504,6 +3510,7 @@ static void vfio_user_instance_finalize(Object *obj) static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), + DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index c87699a..fb6851e 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -57,6 +57,8 @@ static void vfio_user_cb(void *opaque); static void vfio_user_request(void *opaque); static int vfio_user_send_queued(VFIOProxy *proxy, VFIOUserMsg *msg); +static void vfio_user_send_async(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds); static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize, bool nobql); static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, @@ -618,6 +620,33 @@ static int vfio_user_send_queued(VFIOProxy *proxy, VFIOUserMsg *msg) return 0; } +/* + * async send - msg can be queued, but will be freed when sent + */ +static void vfio_user_send_async(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds) +{ + VFIOUserMsg *msg; + int ret; + + if (!(hdr->flags & (VFIO_USER_NO_REPLY | VFIO_USER_REPLY))) { + error_printf("vfio_user_send_async on sync message\n"); + return; + } + + QEMU_LOCK_GUARD(&proxy->lock); + + msg = vfio_user_getmsg(proxy, hdr, fds); + msg->id = hdr->id; + msg->rsize = 0; + msg->type = VFIO_MSG_ASYNC; + + ret = vfio_user_send_queued(proxy, msg); + if (ret < 0) { + vfio_user_recycle(proxy, msg); + } +} + static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize, bool nobql) { @@ -1043,6 +1072,70 @@ static int vfio_user_get_region_info(VFIOProxy *proxy, return 0; } +static int vfio_user_region_read(VFIOProxy *proxy, uint8_t index, off_t offset, + uint32_t count, void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; + int size = sizeof(*msgp) + count; + + if (count > max_xfer_size) { + return -EINVAL; + } + + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_READ, sizeof(*msgp), 0); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + + vfio_user_send_wait(proxy, &msgp->hdr, NULL, size, false); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } else if (msgp->count > count) { + return -E2BIG; + } else { + memcpy(data, &msgp->data, msgp->count); + } + + return msgp->count; +} + +static int vfio_user_region_write(VFIOProxy *proxy, uint8_t index, off_t offset, + uint32_t count, void *data, bool post) +{ + VFIOUserRegionRW *msgp = NULL; + int flags = post ? VFIO_USER_NO_REPLY : 0; + int size = sizeof(*msgp) + count; + int ret; + + if (count > max_xfer_size) { + return -EINVAL; + } + + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size, flags); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + memcpy(&msgp->data, data, count); + + /* async send will free msg after it's sent */ + if (post && !(proxy->flags & VFIO_PROXY_NO_POST)) { + vfio_user_send_async(proxy, &msgp->hdr, NULL); + return count; + } + + vfio_user_send_wait(proxy, &msgp->hdr, NULL, 0, false); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + ret = -msgp->hdr.error_reply; + } else { + ret = count; + } + + g_free(msgp); + return ret; +} + /* * Socket-based io_ops @@ -1092,8 +1185,24 @@ static int vfio_user_io_get_region_info(VFIODevice *vbasedev, return 0; } +static int vfio_user_io_region_read(VFIODevice *vbasedev, uint8_t index, + off_t off, uint32_t size, void *data) +{ + return vfio_user_region_read(vbasedev->proxy, index, off, size, data); +} + +static int vfio_user_io_region_write(VFIODevice *vbasedev, uint8_t index, + off_t off, unsigned size, void *data, + bool post) +{ + return vfio_user_region_write(vbasedev->proxy, index, off, size, data, + post); +} + VFIODevIO vfio_dev_io_sock = { .get_info = vfio_user_io_get_info, .get_region_info = vfio_user_io_get_region_info, + .region_read = vfio_user_io_region_read, + .region_write = vfio_user_io_region_write, }; From patchwork Thu May 5 17:19:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839828 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4C51EC433F5 for ; Thu, 5 May 2022 17:34:16 +0000 (UTC) Received: from localhost ([::1]:37922 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfMl-0004To-5j for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:34:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52830) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0z-0006B9-W3 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:46 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:65260) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0u-0002Cv-Cv for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:45 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FTeZf018676 for ; Thu, 5 May 2022 17:11:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=G86HGXI4UWHM7J6/zVMsbvYOcRLeBqvbRmFLKXWAxJQ=; b=p3RW4B/4lYWkJBSbfpZg2WNRb2LuIeRXzhjIn/Cb3GqiGuU9wWERS7rg4qhh6T/Lr8Lt 4OwK78Y5Bgvspm1W1sVSDSvE9ea10ujExPxqiv2nYNr+cOsMQ4gxDO1KzOk0849myQ76 rXow8U0zgu4Ej8GK33DEVqXuasn2Y7i9Y8i4Y2RTaEl0XWUOUAdZHGkLvQ7EoENvsA+x od3kyXJZU6tu8Igm271jF2kQSQbxTjLW/C/++3+wGlqXuJerXNw1oEzzMsgWzY4A1iQx lWAH5UbykqQyK/MYVZmr59rU/0jKnOOs67u5vaa4xFk6kmE9IAUtrLsdPYk7ZxiVT0pM 7Q== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frwntc3ks-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:28 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1bPK018925 for ; Thu, 5 May 2022 17:11:27 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2170.outbound.protection.outlook.com [104.47.56.170]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fusah2e9b-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:26 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=OvNlB/+w6d0AnXpT6XSUUJsKQVIr7j9v7kFjkEzv86K9o1d7zh7Epn/L5hw6z9Rm/ScRZZS91m3R7V51Cpab1gWjLd5YYSF/aroh8hQd04CAh3T5oFdaLTerHEF2ccpAmPg4ihmPJZ+iYj4SqBCdzwr9tP/Aiw/NJVtDBQFTTMJ/6VWIyME6Gw86iF2PbgWSFOi4nypEKzW8EPL1TWcA0bFd9UmjyULUDtf+F/Iul91Z02VebMhveeKrjxKpYoSYW+2pvPIJT9QLHoRmfrjIZuG6bB3uk8QM6gdJ7AUjtAyL+xUs1duYwkBdAUZ8UKuITggRT8S9dwSd76evT2kWJA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=G86HGXI4UWHM7J6/zVMsbvYOcRLeBqvbRmFLKXWAxJQ=; b=bpDmCdUy+KtcUMnUBCAJHxWhv7/H6YiLNVwIK2Azspk25NavBrJy1DYhLSxPC2ijYmWzl2E7xG7bRqlV/2y5PxQ3363gjMk2W6comMis0scpZjDzYt/mMAPixnIPsB7plXAY+GzQgM2C4y56GM0x8UmGTQoQnOh/uJsJhzYEO4hRN03i+Z/icee/1WmpDengskpVNmzKNbtHIpfPvi2B3fGowSajbtWCf4W61kxkB6u5DuqbWqSbLnbilUGtS3+8PPNlsjkrhgjBggToEuGkhQh7i9VZ2dxbQ+6RtHo/WD99zvitqFpMTHJnqOlKix2fcpCY0+OEnkTED/1jbRlHwA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=G86HGXI4UWHM7J6/zVMsbvYOcRLeBqvbRmFLKXWAxJQ=; b=KfpOs0/LKxw+KlFGOJDw7guOR9n7N44wBfxnNVLNqr+nWusOo5BD/dZHQsuSE9QE1v5Wl3H3awE9SKXvTTDEGw+2rLmKkA7753ga0vWwrs0Jwlq8CMTmR+8ef0FXGeeozjr9VA+UCt/s5SNeSkyYW/2JO0JYx5/PRXDwotCDDoE= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by DM5PR10MB1867.namprd10.prod.outlook.com (2603:10b6:3:10b::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.24; Thu, 5 May 2022 17:11:22 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:22 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 13/23] vfio-user: pci_user_realize PCI setup Date: Thu, 5 May 2022 10:19:56 -0700 Message-Id: <674df55656c1e4b77b11997ae43f002b628c21f1.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 95aaf345-0404-4517-d461-08da2eba45e2 X-MS-TrafficTypeDiagnostic: DM5PR10MB1867:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ByslBNWGYHvPezJWrYv31rqGumZ9Nm/45L+1d08qGOahN7lsyYkMFZMNRy34hb8nkEhpGt6TypOjEitMSoj7RXMXR1C6DPmLFhxE0i9xeCXlhEbcRbzQz7/6hKsAgjoUmE7irlhw30Nqe/m8TD1EhZRFx6Zr6H3Sxs9YdlcfQJl5GyvnQpAWu6xV0x2udwXh1WkKBm4qshjcycNONceWZdarcU37tGOWdXsPpkB14sm9W2MGwaHBImttXTuxROOL2PghKGzGNZTUuDqA5TxDZJKdZnrVXKCRrEn1BrR68JDHEKxkNGtt9Vlg5WIn7W2PWpgq7R3r6cCdN93zvkYyjPx7N63yki5mCht00FKnSBJYF2kh708+F39XTqkQuWV2d45GlFjhF65/UMg5F1qjpHPxJ+4BUfYWePJQ0bFYwYoZNiQO6SsMNFbh/u5xbokYUri8L9WA8BbU0b13JKxhOX7sxbdKA2KoDh0okhMGmxdj3vQKrMuKfH8Kq0UOXM8vTjDPK+/0yncP1jR1+y3LG485Sp8PTcDDSgHQEz+CzBzWeXx9u4lxrNxiOYUGPbjvTMDjC+cMugZUwlm5vK4NnF654ye8h9FHiyd3S8sTPW9H+72GyA8Oq7tSRk4LMXHPgKkt8nkQBBng/YFWSGjdYoTSBblYGS/vcGOVTGO7tIHFsPsCeDSFvJWyZuQcAngXF3EK3V4u7wNskNRK3m6pxA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(52116002)(6666004)(8676002)(66556008)(66946007)(66476007)(8936002)(38100700002)(38350700002)(5660300002)(86362001)(508600001)(6486002)(316002)(6506007)(186003)(83380400001)(36756003)(30864003)(2616005)(2906002)(26005)(6512007)(6916009); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4f2KWq3Fw5p9MBGbdS3whJPfiwgFJE9iP4ambMENv0AMdeidrw5HLfWctM5uK4zO8QwkIErdfQcRskw8aFJon3OTo59wxojRY13+riG7ZsxW/1MnRYpsXQH8n8VviBYghKiBdAUimF+5gGJOAo7KK5bk5iipI0jNxEcWe8WgG40uf3D1JgtpSLrzRM3LbAGmxwNUTwyP0CJHskdHO9cnJtVsdo6UGnGD43LaPEVW8Bd7+nScmWxLL3oCrhk/q4h9m2GsV54CO2UzurZTRwrq95/6BF0aG+39FJ6ZHMhpZwepi/QxKUow0Ksp1casMQezFPdGzMlypYgNpMzKuk7E1TLqX6EJeYKeTro29IKq17XIYzf7B3tFqPHYdyeNUPFI42FRp+Axv2vVwsLC2o5YycOwQB3mZ4TxuvD3mEM4ucRYKJEnSyxGxR0Plj/pdmpOW8lmwN/DmCopWzJ4G5jcujbzTuFqCOXb5PnUhnFPpQsv22Q+jOkc+MxyOc7z5boWTkZUGMnaYTtQKxIT32NqjmOZ6LZ22y/mcAwCB5Onx8eImw2semYfgHIrZHyzaWMOcr1IULt4qj/7ho8Qou8E9MYhTkEekmG+NwCX6pnYo+DCfYa9NfcojF4Jd0wtgE6FzDuvyakEa4XKx/O6Ar+CeYluEeS/P0wWYJEd4xWQr/nLhQhrmB2xplCvGPARP6evaik3yYMvcirN+N6Q5KBud/41mUb4jFJ7xtBL4wbLJuyB9ZFiDl8VlhF1w4Q9h2/SQydk54O3cphTXQ7ioegM5DFdbU9drHU7lDBhWHaMErjbS6zq781nCP08NDSZM97CfHfD5Ed/cw7HBooazQuYgQyg4ja6R1wxpbWjuatA63dem9XrnVQX3CTVxaUNnXUJlYwK6VMXioBq3WAK9OLWgmnH2jrz0uIp62ZnnNAEM8vfKwMtzS1e88jQGqh8bQ7vYmkZ7ZfIq/WP/B/++ZKVc6aN5V+m/7UvZrniMUgU7u3w83txsqUP5/KQk1gL0bPLW2aDlVALZ5pD3gt8abDWuejhPv9/Vwd+rduYaofbtVBX84e5ZYcx59On+AhPpe1ZDwGfvlS2XZn6i0UL3haD7TG7oy1vLVcGByUFbTq+m+EZViHS0e8KPQEHsGFSSzziQqBrnpoWcm38S1TgLWCzakFJNEOm1/16QVyeVrLUOHY4XeTYmxQIvtDijattgVyP7X2pitddUpB5DPLCSqGdI6Y0Y/GFYegpDT3WX1y65lCzJ7JAwcIsGCvMPyOMguva3zJYhIDbr0stOfjMRDNN7bXcayaxHVQPJdhEjBtrO2vz/DCZh3NIMnBkFi45qct1OG4uGDnQM1DdC5gvCB9/JGruPjwT7csc/gVW5P+PjJU7AtgCjb+Xpyp22du7/Pj7TMYJpWcKSW7v7jh5auQNSpHNJbma8idGU710TlwQajTG5E++FgtjBwX6IpjNI5nDZbqzomRrSXd91P6Gf7IKHbz1VBgwBwJIet1zzmLs9WdD+XinEKdnTSyC0fBY9rmFzfeXFVxb2TA5cF8JiL3vnRK4NF5e8UlqcNpKE7p5c8K0K3GJnQFwEjJt9j8qKFm7osIxF4NuHq40dLk8XH0QW4jHkXC2Vc9VeU1j6+WRALhXwMKiMKc6IIqzyXBTRh29Z6L42V/4jIw0RO9qi3Z8ibnZLD+nWfU/AjnMuawA3MW0a4OGTCU1pxmE2wygCaB55UI+il9qwVLH/eVz178miMczgeaFcr6Tvs4XTcJ1AsA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 95aaf345-0404-4517-d461-08da2eba45e2 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:19.9197 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 622mHnbfyzoQiWcyaP/fRlXOgwJkTDfSA7qw4I5n6sSpnfmpiyQbGkt+1FzZXqSua68KurPOJ3NMlfjWxdbblCDmB131P8Zs19i5vtA+u8c= X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR10MB1867 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: Wz8tuVPEeH1ub4ueZg2L1VyZPycQc2o_ X-Proofpoint-GUID: Wz8tuVPEeH1ub4ueZg2L1VyZPycQc2o_ Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" PCI BARs read from remote device PCI config reads/writes sent to remote server Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.c | 275 ++++++++++++++++++++++++++++++++++++---------------------- 1 file changed, 172 insertions(+), 103 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 98520dd..1be6683 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2831,6 +2831,132 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) vdev->req_enabled = false; } +static void vfio_pci_config_setup(VFIOPCIDevice *vdev, Error **errp) +{ + PCIDevice *pdev = &vdev->pdev; + Error *err = NULL; + + /* vfio emulates a lot for us, but some bits need extra love */ + vdev->emulated_config_bits = g_malloc0(vdev->config_size); + + /* QEMU can choose to expose the ROM or not */ + memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4); + /* QEMU can also add or extend BARs */ + memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); + + /* + * The PCI spec reserves vendor ID 0xffff as an invalid value. The + * device ID is managed by the vendor and need only be a 16-bit value. + * Allow any 16-bit value for subsystem so they can be hidden or changed. + */ + if (vdev->vendor_id != PCI_ANY_ID) { + if (vdev->vendor_id >= 0xffff) { + error_setg(errp, "invalid PCI vendor ID provided"); + return; + } + vfio_add_emulated_word(vdev, PCI_VENDOR_ID, vdev->vendor_id, ~0); + trace_vfio_pci_emulated_vendor_id(vdev->vbasedev.name, vdev->vendor_id); + } else { + vdev->vendor_id = pci_get_word(pdev->config + PCI_VENDOR_ID); + } + + if (vdev->device_id != PCI_ANY_ID) { + if (vdev->device_id > 0xffff) { + error_setg(errp, "invalid PCI device ID provided"); + return; + } + vfio_add_emulated_word(vdev, PCI_DEVICE_ID, vdev->device_id, ~0); + trace_vfio_pci_emulated_device_id(vdev->vbasedev.name, vdev->device_id); + } else { + vdev->device_id = pci_get_word(pdev->config + PCI_DEVICE_ID); + } + + if (vdev->sub_vendor_id != PCI_ANY_ID) { + if (vdev->sub_vendor_id > 0xffff) { + error_setg(errp, "invalid PCI subsystem vendor ID provided"); + return; + } + vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, + vdev->sub_vendor_id, ~0); + trace_vfio_pci_emulated_sub_vendor_id(vdev->vbasedev.name, + vdev->sub_vendor_id); + } + + if (vdev->sub_device_id != PCI_ANY_ID) { + if (vdev->sub_device_id > 0xffff) { + error_setg(errp, "invalid PCI subsystem device ID provided"); + return; + } + vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_ID, vdev->sub_device_id, ~0); + trace_vfio_pci_emulated_sub_device_id(vdev->vbasedev.name, + vdev->sub_device_id); + } + + /* QEMU can change multi-function devices to single function, or reverse */ + vdev->emulated_config_bits[PCI_HEADER_TYPE] = + PCI_HEADER_TYPE_MULTI_FUNCTION; + + /* Restore or clear multifunction, this is always controlled by QEMU */ + if (vdev->pdev.cap_present & QEMU_PCI_CAP_MULTIFUNCTION) { + vdev->pdev.config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION; + } else { + vdev->pdev.config[PCI_HEADER_TYPE] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION; + } + + /* + * Clear host resource mapping info. If we choose not to register a + * BAR, such as might be the case with the option ROM, we can get + * confusing, unwritable, residual addresses from the host here. + */ + memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24); + memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4); + + vfio_pci_size_rom(vdev); + + vfio_bars_prepare(vdev); + + vfio_msix_early_setup(vdev, &err); + if (err) { + error_propagate(errp, err); + return; + } + + vfio_bars_register(vdev); +} + +static int vfio_interrupt_setup(VFIOPCIDevice *vdev, Error **errp) +{ + PCIDevice *pdev = &vdev->pdev; + int ret; + + /* QEMU emulates all of MSI & MSIX */ + if (pdev->cap_present & QEMU_PCI_CAP_MSIX) { + memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff, + MSIX_CAP_LENGTH); + } + + if (pdev->cap_present & QEMU_PCI_CAP_MSI) { + memset(vdev->emulated_config_bits + pdev->msi_cap, 0xff, + vdev->msi_cap_size); + } + + if (vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1)) { + vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, + vfio_intx_mmap_enable, vdev); + pci_device_set_intx_routing_notifier(&vdev->pdev, + vfio_intx_routing_notifier); + vdev->irqchip_change_notifier.notify = vfio_irqchip_change; + kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); + ret = vfio_intx_enable(vdev, errp); + if (ret) { + pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); + kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); + return ret; + } + } + return 0; +} + static void vfio_realize(PCIDevice *pdev, Error **errp) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); @@ -2946,92 +3072,16 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) goto error; } - /* vfio emulates a lot for us, but some bits need extra love */ - vdev->emulated_config_bits = g_malloc0(vdev->config_size); - - /* QEMU can choose to expose the ROM or not */ - memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4); - /* QEMU can also add or extend BARs */ - memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); - - /* - * The PCI spec reserves vendor ID 0xffff as an invalid value. The - * device ID is managed by the vendor and need only be a 16-bit value. - * Allow any 16-bit value for subsystem so they can be hidden or changed. - */ - if (vdev->vendor_id != PCI_ANY_ID) { - if (vdev->vendor_id >= 0xffff) { - error_setg(errp, "invalid PCI vendor ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_VENDOR_ID, vdev->vendor_id, ~0); - trace_vfio_pci_emulated_vendor_id(vdev->vbasedev.name, vdev->vendor_id); - } else { - vdev->vendor_id = pci_get_word(pdev->config + PCI_VENDOR_ID); - } - - if (vdev->device_id != PCI_ANY_ID) { - if (vdev->device_id > 0xffff) { - error_setg(errp, "invalid PCI device ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_DEVICE_ID, vdev->device_id, ~0); - trace_vfio_pci_emulated_device_id(vdev->vbasedev.name, vdev->device_id); - } else { - vdev->device_id = pci_get_word(pdev->config + PCI_DEVICE_ID); - } - - if (vdev->sub_vendor_id != PCI_ANY_ID) { - if (vdev->sub_vendor_id > 0xffff) { - error_setg(errp, "invalid PCI subsystem vendor ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_VENDOR_ID, - vdev->sub_vendor_id, ~0); - trace_vfio_pci_emulated_sub_vendor_id(vdev->vbasedev.name, - vdev->sub_vendor_id); - } - - if (vdev->sub_device_id != PCI_ANY_ID) { - if (vdev->sub_device_id > 0xffff) { - error_setg(errp, "invalid PCI subsystem device ID provided"); - goto error; - } - vfio_add_emulated_word(vdev, PCI_SUBSYSTEM_ID, vdev->sub_device_id, ~0); - trace_vfio_pci_emulated_sub_device_id(vdev->vbasedev.name, - vdev->sub_device_id); - } - - /* QEMU can change multi-function devices to single function, or reverse */ - vdev->emulated_config_bits[PCI_HEADER_TYPE] = - PCI_HEADER_TYPE_MULTI_FUNCTION; - - /* Restore or clear multifunction, this is always controlled by QEMU */ - if (vdev->pdev.cap_present & QEMU_PCI_CAP_MULTIFUNCTION) { - vdev->pdev.config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION; - } else { - vdev->pdev.config[PCI_HEADER_TYPE] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION; - } - - /* - * Clear host resource mapping info. If we choose not to register a - * BAR, such as might be the case with the option ROM, we can get - * confusing, unwritable, residual addresses from the host here. - */ - memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24); - memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4); - - vfio_pci_size_rom(vdev); - - vfio_bars_prepare(vdev); - - vfio_msix_early_setup(vdev, &err); + vfio_pci_config_setup(vdev, &err); if (err) { - error_propagate(errp, err); goto error; } - vfio_bars_register(vdev); + /* + * vfio_pci_config_setup will have registered the device's BARs + * and setup any MSIX BARs, so errors after it succeeds must + * use out_teardown + */ ret = vfio_add_capabilities(vdev, errp); if (ret) { @@ -3072,29 +3122,15 @@ static void vfio_realize(PCIDevice *pdev, Error **errp) } } - /* QEMU emulates all of MSI & MSIX */ - if (pdev->cap_present & QEMU_PCI_CAP_MSIX) { - memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff, - MSIX_CAP_LENGTH); - } - - if (pdev->cap_present & QEMU_PCI_CAP_MSI) { - memset(vdev->emulated_config_bits + pdev->msi_cap, 0xff, - vdev->msi_cap_size); + ret = vfio_interrupt_setup(vdev, errp); + if (ret) { + goto out_teardown; } - if (vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1)) { - vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, - vfio_intx_mmap_enable, vdev); - pci_device_set_intx_routing_notifier(&vdev->pdev, - vfio_intx_routing_notifier); - vdev->irqchip_change_notifier.notify = vfio_irqchip_change; - kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); - ret = vfio_intx_enable(vdev, errp); - if (ret) { - goto out_deregister; - } - } + /* + * vfio_interrupt_setup will have setup INTx's KVM routing + * so errors after it succeeds must use out_deregister + */ if (vdev->display != ON_OFF_AUTO_OFF) { ret = vfio_display_probe(vdev, errp); @@ -3488,8 +3524,41 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } + /* Get a copy of config space */ + ret = VDEV_REGION_READ(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, 0, + MIN(pci_config_size(pdev), vdev->config_size), + pdev->config); + if (ret < (int)MIN(pci_config_size(&vdev->pdev), vdev->config_size)) { + error_setg_errno(errp, -ret, "failed to read device config space"); + goto error; + } + + vfio_pci_config_setup(vdev, &err); + if (err) { + goto error; + } + + /* + * vfio_pci_config_setup will have registered the device's BARs + * and setup any MSIX BARs, so errors after it succeeds must + * use out_teardown + */ + + ret = vfio_add_capabilities(vdev, errp); + if (ret) { + goto out_teardown; + } + + ret = vfio_interrupt_setup(vdev, errp); + if (ret) { + goto out_teardown; + } + return; +out_teardown: + vfio_teardown_msi(vdev); + vfio_bars_exit(vdev); error: vfio_user_disconnect(proxy); error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); From patchwork Thu May 5 17:19:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839829 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D2004C433EF for ; Thu, 5 May 2022 17:35:41 +0000 (UTC) Received: from localhost ([::1]:41030 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfO8-0006fv-Sz for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:35:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52648) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0m-00063U-Jm for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:35 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:58112) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0i-0002CG-29 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:31 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FTMUZ030007 for ; Thu, 5 May 2022 17:11:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=ApAy4fxodgJ6qAhXdB6hzwjclmc8Xyt4FMplOhahbt0=; b=Va/3zP8QUVtH3GqDdJdfczknDiRkcvQDzDHnYb++M5F1t9i6cMTSD6+pDnR4arc3K60A cINm/MeCYHvWfls4f8HwhHNnp2MsNz1ixwa5TNTrkyfiqT5X4TrlM5bXa8mg1xRPi0XF sKWs2+JAtKPfEUU/fQuOhz6zvGPCQyDQ8kBnocGAj9u/bp7xs8ylWJPMw/WKzMTo0Bfv P5Jw8gUVa30//hjr7RVlKlC+KvD2mRHXsBiuabzefqU3LV4ILCTe1Osj2S909JEmdatG KzI48jsUyO9tI6kiAbS/Zg7Ypm9gMJShpwLoS1rgh61tQsBrutpucFkDYLKZ5AB1rh+9 /Q== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0m379-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:25 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0Zqd006282 for ; Thu, 5 May 2022 17:11:24 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2173.outbound.protection.outlook.com [104.47.55.173]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fus8ybftd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XIKJYvlYHGowuGbDuX3LDMo2IoyN1nuntseN4VaahgTIx3/A3nqxqQDjmjZp9Hym4mLttWz8LlKJzuNr0bs1pmJDxLTPSc8ibri/8NwvsUnmQ4wh3MtxI249TqnKXxUb403mcm8wGF5igPBOpW4OkfI+dIsRiQq2qA2ZcTvwu+WqhHQqtK7Zz/qKIhpYDqtGwJy6bmZOl9zW0Jj6ZG3ul0FgucYcMlhCBxGz80RtQVaExydYSuEyINYrkT1SeC3TUtPhsAYCM+g/sRPjIeqmK9DAGmwPdmBInNXu8hf1iIIg9w8pzS8Y+bHq83R0CvpeQfYIwGWKjcJl4ugignkTZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=ApAy4fxodgJ6qAhXdB6hzwjclmc8Xyt4FMplOhahbt0=; b=N2KJQmqmI3CkiGu+RONEJpSNW1bifkBvI4Xn28xkLN98aCKbPEqKEJtxpt1velb3o6XFdbu21s2m/jzu5zifDTxxBBIPrk1nt6YAsN2Y2Y7+z20S0K/rWQOj5loqbtUNVbbK6zbLCkjS/6lroXcjEta6KQN9uKi1IVTh2Hu9PZSNYbUHnTztJ3KcFotGjKwcekRDecQEq3CgiTOCwGV8d+uvBg+AEojyKrP2gDSmB9Fn1cE61vW+4DZo4de9aXugKmme3jBlkucowivWj1+D8sueOK4XH2Ulfjh1xmL94+5y072BYX0415bB5OgFT6CouL/CS36yJqZrNh2/YtC6Ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ApAy4fxodgJ6qAhXdB6hzwjclmc8Xyt4FMplOhahbt0=; b=VJorGDyhzOqJOYzBRCdIPl2+tslyPBQz7JrNsSe8UEIL7RSyeE6wn/y2mhVp8EKm/G6V51JFdg9iE/n5hsVub1yNwrWtgF0UFY8FpZ9iZiGhg5F5KEXDAJjCtU1cmYsAeVU3aEhpy8C/Kg8GIw8p+7JUDGyKTTdkhUHBKTgttVM= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:22 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:22 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 14/23] vfio-user: forward msix BAR accesses to server Date: Thu, 5 May 2022 10:19:57 -0700 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f39960fd-78a5-48ee-0bc7-08da2eba461c X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: i5Hux2jlFEtfXSTXQtjO+smyJ/NgK4ResYXUDtb9iGTD3XVOa90+FKTbjaBJbAtaGVvEqu0+wjXn4IyKYG7EjTIK7g352R2TIYVVo/HfHhXiBM0LbHVY9lhmxdy7QZTP1UmWiP335ghQ7pvwvqXYUerX952I6S//FUkw2CgDzKQwrIq38rqshVIpJ5da6YezS36S+vu33RSUnLn/os56H3T1/U4HetIcbP1qn4EOAzENDYAnoCc1V3Nmblws1dP3mGHxzklm8+wz0R3RbIamyYxoT2GnBqhpZPOP6FbjAJ0Mxdu/36gCyS6JIC/1lRE/9WkK/BcAHNAhALC6javAV+3wYinz87wXN69fWMOlK1n2zDizQVgB5VHzOmr0rsrtDGxD6tze8GnXkINpq37fiM9e4jsDWXLOnuLZtrpeD9tChzVBVkwnxmdutGaX4sdcX7wxV9YHJZDfB1/8+kKydJmCI6M8PXzXNz/jwNKEBTF+0WtpsGZOKTue0TecqA/jQK0sCIB4BG6D+txw3StN2V96LmZTOy5NcP/R5JBDT6pCwRrrILAvakRznwmC0nj3sHAK08NTBIuRCi+60p/ymCGWO90PMIrO5NWIDZORP490EIrWyYPJup3M2QARt+f4OAmg+gAJpQWSGs3sYrtjMpHSGHBgWHeP62n2bMz14EmaRzVoNLLNHusuxQjR5T58DMoBMwD8Z79U3Eyd9UY6UA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: wghMNcDKOZDN8prkl9jkXeSlIeTlfwPMStkgUtyivHkqyn6KYnDqEpi+Zu5TlRzTGEqqAm+uO3Ue6nnRBSUE5AoWK7IfMBX7JxRl7j4DExqbo9sAE8XQZsPmw9qhVeVWsw8o75fqF2497F+g/xAguOvcGt18mFRSb0UAPzjHTcCvvE43i4mFiN2DCXtW2M2gYppcbjZfUMsTK6kjdnx+WOaGMDFSVRC6cpb+WKH4GNeEhUuFWaeOpC1Qx2eflc3gG9MKQYqZt/5MkzFXbpIoftmZfjYstOCwv2jg/o86pKGzQTEEsOzMLlzKuZ5TwHojw9B1sl08jhsm2s3r/SaGReHYO2XSdNX8LjES8YkeJ5rs1NMSRBmMaunSIpehfJPnv+iEVyMRxL7cq3JipmGDKEP2aGDzhH+0QyX43PFFajIWanVj5Jk9pZbFJ3D6TtWwWh//EHWpzotMF2BPoThOqt12BYNReKCkgjB7miAL1O4aGjSlgTPIuDCLZIHNllWRTmDI6zhyvCZX7lw8aX0JHNVufT5TomA38sld/1GIk8NCkZLVdmL5HiFsJXibUGi72RakVphZrFE7lfiCv+uh4rAVFh93i/OW/8bpVO0FzdMn9IT/PzHYurvQNrNmcl8dci+vCmv1I1UPvdQ89YIIdmwTZBIfaTYX6yw4rQBQrRsqo8Mk42ssTohDs6btra1nXzNrBg9PZz12ZZ+S8k+SelqeyQi5UIbuaBaI6Wo2xNEajA1kFTjuHmP2wjDe7kl0NRRK/MUTdUNqMGspaoJcdCotaMLzO1/oc6kaOuhH4T+m/n2avxGvUV1BXyovYK12FpCENUw7lqhQQWi2q7JkbQ3kPuVzg9X9alq/jUvjVX8WWFkagvLP1R9F9sqi/l3uFMj9lzIAwNfkmks6GYp9BBl8Io2UyvujzQjMlO6/D9ib2f1ObdpxgDoeRg8EnyRb+Y7v91sSQ8xU/UGu1nk5t2rEa/QoZGzVP5aMjeoqgdTdFva1gDVTjLX8HXKr7+KKB90M3IduzzhtfQgMvRmYjGjJme+nmgoYdrQa/gbyr8dMFDepuRoLbIcwaXOnsy7xKXG4mjvmbgR8JxCWAjlTGG2/cL+MSDxYjTdpQFLtRGS8CGX6wslcw7UDDj6PcYzCqOEMz1HSjtQHgKjgUpXqxOda4AvcKhyTikcmDncWaC5cIFdEwlXxhs7vJFmtkQc/TlYZNRco7WlMGjhO1xeJ84FcOnkT5GzgCnd/bNnTfwP0pVxlIsD9rkRacmTTcIq0m+t/I0gbAOZ1WbzbrmUd4M4NQC2oqmt1iyb5dVkESJB9+/CzjQYvR/exvinDvqnTo+yd11AJRAeAayvo/E2ABm2wnj0WU124VTne32cetRXhpXTjW7u8ZQsxksxtxHoMvCVNgTjsAqPK30e+HFORwuagNXgrkF3sh9L6Lld8davxt1kIA0kAcP80FJflHCsv+C3yQkdn4hCJv6584nN8D7JhH8ngNiuz5qvFsR0kk8dvwY1NwIL9ckT7lRYsWwJe7ev8jSlQJ79sysaeyhcuzFFN3aedAEN1r+1HjwjBrX4mJ3gvjv5T6ekp0oIs96nY56SpQ2/pWA9/LYwBI6vSMhFJLMfbRMoxt4ZMQcfK5fA3Ru6JBPTHXM66pK0eCp1Ch5WL9Cno6wlJLSBJ9m8uIhk3uqgsysO7yLhvbQlmytKrD8Xh1p40c6p36pUEzM1lhQtRwxfSERB6kcGj6+q7rj+1I8LTXVs4wZQLKiLkXG0= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f39960fd-78a5-48ee-0bc7-08da2eba461c X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:20.2478 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3+xBpGgvAukg8G0XiqpCa3Lpf5tmNRoc217yePLHao6jTcuBNiS/dMzWaHa1z7YQEHS/LUEaxHzfoqsn8JLLaZ0TkX0XTMj45wKEMOG4AZQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: Dk1dJIaexZ4dTKeR0tg9ZdPK_5THjW4b X-Proofpoint-GUID: Dk1dJIaexZ4dTKeR0tg9ZdPK_5THjW4b Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Server holds device current device pending state Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 1 + hw/vfio/pci.c | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 113 insertions(+) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 643ff75..a4eb5b9 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -112,6 +112,7 @@ typedef struct VFIOMSIXInfo { uint32_t table_offset; uint32_t pba_offset; unsigned long *pending; + MemoryRegion *msix_regions; } VFIOMSIXInfo; /* diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 1be6683..bc70968 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3426,6 +3426,111 @@ type_init(register_vfio_pci_dev_type) * vfio-user routines. */ +/* + * The server maintains the device's pending interrupts, + * via its MSIX table and PBA, so we treat these acceses + * like PCI config space and forward them. + */ +static uint64_t vfio_user_table_read(void *opaque, hwaddr addr, + unsigned size) +{ + VFIOPCIDevice *vdev = opaque; + uint64_t data; + + /* server doesn't change these, so local copy is good */ + memory_region_dispatch_read(&vdev->pdev.msix_table_mmio, addr, + &data, size_memop(size) | MO_LE, + MEMTXATTRS_UNSPECIFIED); + return data; +} + +static void vfio_user_table_write(void *opaque, hwaddr addr, + uint64_t data, unsigned size) +{ + VFIOPCIDevice *vdev = opaque; + VFIORegion *region = &vdev->bars[vdev->msix->table_bar].region; + + /* forward, then perform locally */ + vfio_region_write(region, addr + vdev->msix->table_offset, data, size); + memory_region_dispatch_write(&vdev->pdev.msix_table_mmio, addr, + data, size_memop(size) | MO_LE, + MEMTXATTRS_UNSPECIFIED); +} + +static const MemoryRegionOps vfio_user_table_ops = { + .read = vfio_user_table_read, + .write = vfio_user_table_write, + .endianness = DEVICE_LITTLE_ENDIAN, +}; + +static uint64_t vfio_user_pba_read(void *opaque, hwaddr addr, + unsigned size) +{ + VFIOPCIDevice *vdev = opaque; + VFIORegion *region = &vdev->bars[vdev->msix->pba_bar].region; + uint64_t data; + + /* server copy is what matters */ + data = vfio_region_read(region, addr + vdev->msix->pba_offset, size); + return data; +} + +static void vfio_user_pba_write(void *opaque, hwaddr addr, + uint64_t data, unsigned size) +{ + /* dropped */ +} + +static const MemoryRegionOps vfio_user_pba_ops = { + .read = vfio_user_pba_read, + .write = vfio_user_pba_write, + .endianness = DEVICE_LITTLE_ENDIAN, +}; + +static void vfio_user_msix_setup(VFIOPCIDevice *vdev) +{ + MemoryRegion *vfio_reg, *msix_reg, *new_reg; + + vdev->msix->msix_regions = g_new0(MemoryRegion, 2); + + vfio_reg = vdev->bars[vdev->msix->table_bar].mr; + msix_reg = &vdev->pdev.msix_table_mmio; + new_reg = &vdev->msix->msix_regions[0]; + memory_region_init_io(new_reg, OBJECT(vdev), &vfio_user_table_ops, vdev, + "VFIO MSIX table", int128_get64(msix_reg->size)); + memory_region_add_subregion_overlap(vfio_reg, vdev->msix->table_offset, + new_reg, 1); + + vfio_reg = vdev->bars[vdev->msix->pba_bar].mr; + msix_reg = &vdev->pdev.msix_pba_mmio; + new_reg = &vdev->msix->msix_regions[1]; + memory_region_init_io(new_reg, OBJECT(vdev), &vfio_user_pba_ops, vdev, + "VFIO MSIX PBA", int128_get64(msix_reg->size)); + memory_region_add_subregion_overlap(vfio_reg, vdev->msix->pba_offset, + new_reg, 1); +} + +static void vfio_user_msix_teardown(VFIOPCIDevice *vdev) +{ + MemoryRegion *mr, *sub; + + mr = vdev->bars[vdev->msix->table_bar].mr; + sub = &vdev->msix->msix_regions[0]; + memory_region_del_subregion(mr, sub); + + mr = vdev->bars[vdev->msix->pba_bar].mr; + sub = &vdev->msix->msix_regions[1]; + memory_region_del_subregion(mr, sub); + + g_free(vdev->msix->msix_regions); + vdev->msix->msix_regions = NULL; +} + +/* + * Incoming request message callback. + * + * Runs off main loop, so BQL held. + */ static void vfio_user_pci_process_req(void *opaque, VFIOUserMsg *msg) { @@ -3548,6 +3653,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) if (ret) { goto out_teardown; } + if (vdev->msix != NULL) { + vfio_user_msix_setup(vdev); + } ret = vfio_interrupt_setup(vdev, errp); if (ret) { @@ -3569,6 +3677,10 @@ static void vfio_user_instance_finalize(Object *obj) VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIODevice *vbasedev = &vdev->vbasedev; + if (vdev->msix != NULL) { + vfio_user_msix_teardown(vdev); + } + vfio_put_device(vdev); if (vbasedev->proxy != NULL) { From patchwork Thu May 5 17:19:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839822 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E621C433F5 for ; Thu, 5 May 2022 17:27:24 +0000 (UTC) Received: from localhost ([::1]:56428 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfG6-00061B-T9 for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:27:22 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52642) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0m-00063R-IZ for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:35 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:58452) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0i-0002CI-IM for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:31 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FpPqY018740 for ; Thu, 5 May 2022 17:11:26 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=IItzUkMRhj29kZspoVTdwkT1Pk6L/mEgCGr8ZyZ2lUg=; b=UQVtf+NO6fQoX5LrqoydAdZEpbAiXbX+sVhXe434baoL8NnTH2j5bwC+xM+ZZan4xrVB AJyLac8VwHXZDUe/lYeuwTneqYQ4QF+kdx0blnHxmyeBTlVRbi5K7tPsHQUPMqFSRoNe bdNbowNQiL4YyzjEbl1kwsiLAFwt/bmqYDFBaCTLp7OJiDzGjQ9N3gRcebrDjXhkQqSw bceBGIIHvmmVaVCl5nCr/JVcpO1GB5WN+tGS5d6IwloDt7T7gkaDVtmz1myblRt3B7SO ei7GYkuQ1quB8P+PgQsvNLciykkv7RrMvijA7nD6KqC3acr3kExcsj4AkZ0STKkMWXOi UA== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frwntc3km-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:25 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0Zqe006282 for ; Thu, 5 May 2022 17:11:24 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2173.outbound.protection.outlook.com [104.47.55.173]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fus8ybftd-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:24 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XW0Y3PcDBxKXh6Qa6vcwN6FkNiPsMTwSS6GGgsrQ/CgjUNBoEiaf5ASpbovTvorKVsUScuBQP3YBK7VtQO4hVj6NqcJNl424QtU9s3Om/uHMkkvI/ieYRtmMRaV9Ez3Z0qW8rEF1kve5X6YqA7w2jLTrfxGKqudVqqkSYg4BxypscPYSoVpH5sn3r2+Gl5KZPAYHAORY+s567D+H0kW9DvrI/t6PF6bwp2+r42BJYKES9K+H6PDOnlv0/n+LH/8Nza4kn8l4Q0Ljbja44nBB/pfv3+4j8I3JulfMQbBabCSFWuo9b+1S3qNfxAE2jvM3goSVvJTI1GIEfxB/hJ8O2w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IItzUkMRhj29kZspoVTdwkT1Pk6L/mEgCGr8ZyZ2lUg=; b=Pbi92TydskvdlOEXId+aYr8jogf6RTyGNDJ7tdFvXzA9peq6/s5/F7bTixBDZ0M38SWoyAkF+i2+uVRIDwQMD/IZTTzYpl/8Prynq9JaATgPKiMSqjmP3j2owsxYaj0wWsuhK/rNTr6DVjxut2PzQIMMkjBsNU+wDNUBc2Q9h54mO4FOOYVLYtDdqO26ITyXZXJ7TIcys7KeZoevzGxx/FIirg8A7tbxlvwEOBePWsizMEHNYfR4ULyynX+b7tUlWqDJlVTfgljN6y0hFayL+wMika69vsaoovAti15A8EG56WHajNsht0LxpTOSGgqJr7O+Rdc1o09H6w2RVadV9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IItzUkMRhj29kZspoVTdwkT1Pk6L/mEgCGr8ZyZ2lUg=; b=TsCMb/zQVKYe0h0GEs9B0bQFiR4H5+qEhY8GEVorXT4CNhSkb9XWpQPXvFzw7OOy9/3nplaqAhk8IuBWTeDluJOsMxIuMPjPJ9tiC7r59WBEws6LA/So8MnMP+tp2sZwu3kW15MY667wJ9ijj7jhXNBDK6gZUP4B1H8N7Fh5Yz4= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:23 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:23 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 15/23] vfio-user: get and set IRQs Date: Thu, 5 May 2022 10:19:58 -0700 Message-Id: <788217d1a64b2b70097236ec8fec39a39a01aefb.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8a8a0670-6bbf-4d4c-4348-08da2eba4655 X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Fyct2qaHbtetHJsEC2bnx0kVbdzFYC/0a0TfgrWOMzT+5SeBw1RBVDcFJUFXCXCVdCo8Mz13TmW5H9p7kXR/zrPQoN9ef9fn3hmK0kbpOWUF5U1UspvFTBuq2S51xD9Wpldu2oCNF3VPY0IOMdnqwtyGx2JxfusXo/MNff6VMo1/JUCA2q5jRiu95WVNyoqlMnnj8MmFMUzmN5c3I2sX5uEWOhrdfwKGsTLYL+8o33GbGsIV73Bp8PYgbaOwqn75z8yTo3xzitNg/pROcvvcfLj68nYVcUfUne1A64w0WpP4d2YXtBAY0PlvFmx9FsaqbeJCUlwrPiRx680EiEGBYSRNXyzOCiZrXGamkme8BWldfR3GiFMZBSJNyuq451Hiw58Dx4h1FXNk2+91VHvoIaGOZyQXoI7/4whuYNiC0brYBcdoAj3x58CnhdFC6w5tMnc8QekKTw5lx4WMGxbOvZodXi/5vjVo6uv0iL1A/PF6RtQztj0tpSri083T34pqjX5bfMvF2XkXNOWWoaw1oONIimcR0PFizWnN0Pclk9nx1NGVLaoI8FvH6x+D2QkomN6Anxh7rZzdQEfSvbgvb+xZJ03EiHm01vH0ll5u5qBZyuFkuf7lSF4V9/SVghyLayGx9ZFfxuGWEGUxsO8Z7V6vm2hFE/MSaHyQ4bRN4hzab0BJK7bFH42BNdwDPR2Wl0JMTuxtMon4t8+tYS/+2Q== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: yoQNWuuTxn+0CxbUmzSIFszrT1hxZ+2DIQPI8XheCEXIkS5j4aeHi/+eFyPE6CITcM8bA6gQ1cRtAiW6TSP3nowvf/B83r+w+m8rHnLPBnBzzHdXHbvvtDJU74o53TAi5GsWRvixLp3IMKuWA8sGmg+/f7QOtptUwe7T5O0OM/sgJ5Yu05yhJGBbZCEmQw0krsItUkYhxh/UCTLC9jBFUkvozvrNpMHqIfQQSO2pQZM3txY820fp9hmm5Qr9GlYw4XQbDttqagKPWrepcw+T+Nf1PJE44SpLS4tOIiYXvJiB6tHws3GyDAXsfe/ry6pE1IKD0tTcaz6DyVexiKLMKva6bLeLoitpbQUMadUiAfFGj71mYKx1GshL9JFGYfujQsTiZ7jyqh3KHG3409Z5LdEdFDr3SD7NPxxSAOfVVBja9y7fwO6NL2LW/ArKwFJem0BbjYlTs5s+cm9EHv8xiZdztF33dyKfmISCf05n8Ydo1yrUwA0xCMMcX0wKZVJfpzht4Y2vfd2/rtHN0Dl41styY6fLzZgIvS7Yvigq2eLonmMPEaDz+QAOw+mA13iBtz+jP/2PDfEi/OsbogqKBxy7FHwNNr7iJQlKKmBD6O8eSp4FPgC2NGcWwnMX8g1eljAEPDNWP+W/gyAadOzW73Hbrsm3A+0F5Y5NwpJAVtC/P4/PgBnM/XdeUrIvOR2fvok46wtAPD/3pqr18TCEk9eoaOGH+jruSwtEg+PRdyG/BH0uDgZZqYnrl6u6ZUgzVhGwAkFTk0GVTuY6JKKTszJbyVKUNp/uAzd9cYZv6AgZpAjuMcXCjQmC2nSL8VZ07ehvqBMJQJ3SUdO3dqx5R+Wb896Uve40fmJfYnhIgcFT3/RFJIopWmNt4cX3e/ambQ5CihmcpeUWP6+Yaq1W+iAav2KOlFidD12p6/cyafeeD41MRgFN6sSL6JkXJ8XFQm/jS0iCpoPx7Z51izsgL0oMMTOmU34KZicvyfDM2hFcCy58f3YjjuuldF4wqUXJpd+8Wecwt/5sCNohPggkzvECAH9lKPqLixtWb/jXPk+vEUNCUveBJ1Fx66AwLlszzYDFbjgSycvUj7TSJepCBm3RAZnIx0fiZNq+QHQuhpmPvR4DatNX1/RV4jNShw6VilREwbUUP0IQDX5OwuGWxZ55+2nrOjh82nD31Pddf60JN/717B77tCg00+wPkBN97s1nmUjtIdlnRqNHj3f/I8qu/RWN9w0buBZ0EBoJX72Wnyn/NO+KKP0k1nuiIuFLidJgn8ZXjpTeaQ9ArNIKHjmiCPUGj8H6Tr3meyu44vLSaXCbfOCJqVX2kT2rLAjiHu+iswJ8eItJaGfLfR/PCWoIvIS0nY3LWxfsVzswg3N/zf/apy6IUgH3tYvA9aFW7Plk4SKTBNTTBnfKL5aOLr8WJRe2Zod3x9M283pxQqrzuTvh8tvpXS/8poN2JCt/+pcb5FN+CA8vQ2OhEgNvpc25tLRBXgm70jg5NpE00zGIo9EloycVzx9fLfBIjKN3DB5azoVaHv97VZL4/ZDlZdnQcRkZcrAXre7zhKDE9bTfRCpQZ4KvC8Ylm/ZQOfoLqwHrbRPS/u7acAR/O69O0pjjOZFUPje4WrawlNevELFxf7U/IM8uS1zbygzUMzaQKwyRSQcvH5sWvmw/UEtEYH922ee7XMMqpnMcX85rhA3xdXyyj+vont8V5WhOPCGNc099ED7ktgCRc9RjIWfeCcTeMlrufyGXlpPAu7bAK2Q= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8a8a0670-6bbf-4d4c-4348-08da2eba4655 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:20.6696 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: tTBDAbVswBvWCuYTiY/Hq1aOvDMrRA77ahu3tBjzdSVeyu3MgwDA7u3U/JZiuFQU9WrHQ0qlRxBcprYDSvPMuMlZiuu2olSMGTOAQTQaBWk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: E8O1PbP39BCLVBG-Ui2B1NogGWKXy6dG X-Proofpoint-GUID: E8O1PbP39BCLVBG-Ui2B1NogGWKXy6dG Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 25 +++++++++ hw/vfio/pci.c | 9 +++- hw/vfio/user.c | 131 ++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 163 insertions(+), 2 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index b1ea55f..4852882 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -121,6 +121,31 @@ typedef struct { } VFIOUserRegionInfo; /* + * VFIO_USER_DEVICE_GET_IRQ_INFO + * imported from struct vfio_irq_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t count; +} VFIOUserIRQInfo; + +/* + * VFIO_USER_DEVICE_SET_IRQS + * imported from struct vfio_irq_set + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t start; + uint32_t count; +} VFIOUserIRQSet; + +/* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bc70968..0a4208b 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -517,7 +517,7 @@ static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, vdev->nr_vectors = nr + 1; ret = vfio_enable_vectors(vdev, true); if (ret) { - error_report("vfio: failed to enable vectors, %d", ret); + error_report("vfio: failed to enable vectors, %s", strerror(-ret)); } } else { Error *err = NULL; @@ -662,7 +662,8 @@ retry: ret = vfio_enable_vectors(vdev, false); if (ret) { if (ret < 0) { - error_report("vfio: Error: Failed to setup MSI fds: %m"); + error_report("vfio: Error: Failed to setup MSI fds: %s", + strerror(-ret)); } else if (ret != vdev->nr_vectors) { error_report("vfio: Error: Failed to enable %d " "MSI vectors, retry with %d", vdev->nr_vectors, ret); @@ -2669,6 +2670,7 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index = VFIO_PCI_ERR_IRQ_INDEX; ret = VDEV_GET_IRQ_INFO(vbasedev, &irq_info); + if (ret) { /* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure(strerror(errno)); @@ -3662,6 +3664,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto out_teardown; } + vfio_register_err_notifier(vdev); + vfio_register_req_notifier(vdev); + return; out_teardown: diff --git a/hw/vfio/user.c b/hw/vfio/user.c index fb6851e..d0140d6 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1072,6 +1072,113 @@ static int vfio_user_get_region_info(VFIOProxy *proxy, return 0; } +static int vfio_user_get_irq_info(VFIOProxy *proxy, + struct vfio_irq_info *info) +{ + VFIOUserIRQInfo msg; + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO, + sizeof(msg), 0); + msg.argsz = info->argsz; + msg.index = info->index; + + vfio_user_send_wait(proxy, &msg.hdr, NULL, 0, false); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + + memcpy(info, &msg.argsz, sizeof(*info)); + return 0; +} + +static int irq_howmany(int *fdp, int cur, int max) +{ + int n = 0; + + if (fdp[cur] != -1) { + do { + n++; + } while (n < max && fdp[cur + n] != -1 && n < max_send_fds); + } else { + do { + n++; + } while (n < max && fdp[cur + n] == -1 && n < max_send_fds); + } + + return n; +} + +static int vfio_user_set_irqs(VFIOProxy *proxy, struct vfio_irq_set *irq) +{ + g_autofree VFIOUserIRQSet *msgp = NULL; + uint32_t size, nfds, send_fds, sent_fds; + + if (irq->argsz < sizeof(*irq)) { + error_printf("vfio_user_set_irqs argsz too small\n"); + return -EINVAL; + } + + /* + * Handle simple case + */ + if ((irq->flags & VFIO_IRQ_SET_DATA_EVENTFD) == 0) { + size = sizeof(VFIOUserHdr) + irq->argsz; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, size, 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start; + msgp->count = irq->count; + + vfio_user_send_wait(proxy, &msgp->hdr, NULL, 0, false); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + return 0; + } + + /* + * Calculate the number of FDs to send + * and adjust argsz + */ + nfds = (irq->argsz - sizeof(*irq)) / sizeof(int); + irq->argsz = sizeof(*irq); + msgp = g_malloc0(sizeof(*msgp)); + /* + * Send in chunks if over max_send_fds + */ + for (sent_fds = 0; nfds > sent_fds; sent_fds += send_fds) { + VFIOUserFDs *arg_fds, loop_fds; + + /* must send all valid FDs or all invalid FDs in single msg */ + send_fds = irq_howmany((int *)irq->data, sent_fds, nfds - sent_fds); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, + sizeof(*msgp), 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start + sent_fds; + msgp->count = send_fds; + + loop_fds.send_fds = send_fds; + loop_fds.recv_fds = 0; + loop_fds.fds = (int *)irq->data + sent_fds; + arg_fds = loop_fds.fds[0] != -1 ? &loop_fds : NULL; + + vfio_user_send_wait(proxy, &msgp->hdr, arg_fds, 0, false); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + } + + return 0; +} + static int vfio_user_region_read(VFIOProxy *proxy, uint8_t index, off_t offset, uint32_t count, void *data) { @@ -1185,6 +1292,28 @@ static int vfio_user_io_get_region_info(VFIODevice *vbasedev, return 0; } +static int vfio_user_io_get_irq_info(VFIODevice *vbasedev, + struct vfio_irq_info *irq) +{ + int ret; + + ret = vfio_user_get_irq_info(vbasedev->proxy, irq); + if (ret) { + return ret; + } + + if (irq->index > vbasedev->num_irqs) { + return -EINVAL; + } + return 0; +} + +static int vfio_user_io_set_irqs(VFIODevice *vbasedev, + struct vfio_irq_set *irqs) +{ + return vfio_user_set_irqs(vbasedev->proxy, irqs); +} + static int vfio_user_io_region_read(VFIODevice *vbasedev, uint8_t index, off_t off, uint32_t size, void *data) { @@ -1202,6 +1331,8 @@ static int vfio_user_io_region_write(VFIODevice *vbasedev, uint8_t index, VFIODevIO vfio_dev_io_sock = { .get_info = vfio_user_io_get_info, .get_region_info = vfio_user_io_get_region_info, + .get_irq_info = vfio_user_io_get_irq_info, + .set_irqs = vfio_user_io_set_irqs, .region_read = vfio_user_io_region_read, .region_write = vfio_user_io_region_write, }; From patchwork Thu May 5 17:19:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 62BDCC433F5 for ; Thu, 5 May 2022 17:54:08 +0000 (UTC) Received: from localhost ([::1]:44952 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmffz-0003NH-5D for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:54:07 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52824) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0z-0006B1-VL for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:45 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:61234) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0s-0002CZ-Np for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:43 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FTeZd018676 for ; Thu, 5 May 2022 17:11:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=JiPoLDm1MhM6XJC7ZtMToKKXD3MriMPOj96SvoxCsNo=; b=pHElXzxuUbwzNg7JRUhNY/9eNDJlZrDJSE46bDacgFPhAhX9zBPP4c3vYABXMKDrP09l PrptoexbuJ9FhUoETYDUw5paEbYsbNC95kX1DLQ5YwB4E5xni+8Npo8h8YDqSbRidA7S 07gOF1wfxpBT6GVHBDpTc28nX9807kOHhhHwM/d2M0/vL1k4G5+ay6x6Xt4+joE9jSYx FfRGJ9g3Odu7ilmo6qK8mFZA7OE0t1OtO4CU95Ne/ewxDXJFvHphyQH9bPCtQd6tyHUv X2J7yvtS5sveWV0Oa96hgQnzv7H4iKVGTPcIjblrWldT3T3kQDfCS90m5T48yFW0jNoa Bw== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frwntc3kp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:26 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1bbB018946 for ; Thu, 5 May 2022 17:11:25 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2169.outbound.protection.outlook.com [104.47.55.169]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fusah2ebg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:25 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Ql0x8nTuXADGRHdILlU5O/+QUtf5PKNvHzEQ/e/uvInufbYf4hdfLl6a9JUo6U8iq703OBWfwEcsMbWFAk0H0fkzS2lHQsKuDzqOIMVWDsifo359tVEqLzxDrHL+TuqganDKKGFXAul8FR9fQi+i9uW3vyacgNqlkzQcphUdNbi/5jRGpy45RL3tzyP0QV79/O4rUb3glZtGae5CUzC0Y6onGx+b6XVbwaiOJUI24na9vXSqaq1TWAAoRVCgJ8MkAcb1gkfLXm2Xi+uu5KFF8QgrSZqelwVAG46aIN2x1OypM91b2ltrHdmfpMZ9l/IzwwAzZFb3iC3kZUohHyRrEA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JiPoLDm1MhM6XJC7ZtMToKKXD3MriMPOj96SvoxCsNo=; b=Y81EnCXtfwTlsO+8QmTeBMh00CbkvQc/03vpMvF6brKdqB4ryLluIdS+g0E1JjHcmfmCZnVrwpd8/zDRW2yBlc9ZvyJq9jEybZaK2tRi1ymZT7VFN3Z+C+tvLY07MPEXN1k4+5aLMAH9hQdnxvShrNR+u6jtEASuqbDgsAFKk9WDTNbDH4wioI3oSKe+Dp7gF9FTz98J9usUQqflk/TRxCjZSjeklcSU0Tg26XA6ei0S4bcNYnjR9WhdlEzBCLx2e4MLnsHeH0IKCnEa2pZElSMoAgiXwmWJChoN4kcFQjfgxdq1V5jjGvLlS1iu65x4WWxdUVuXW9/N5qlvhI76ow== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JiPoLDm1MhM6XJC7ZtMToKKXD3MriMPOj96SvoxCsNo=; b=0IORLAr6RAaQpMsAxGzajt2OYwPMEhbEvSK1rYy9mYwy7C41QThTWyxXclSSBgKI6IQAhN0qUAp9Br0c+tTac/dKD7hK+WyDLoDVStNLZoXwWRKQWtX/UrgtljKaTHOGZp3cD+s4V8TVYkxoqz6/Pwtz+1lk3xv/eydYnnenEJM= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:24 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:24 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 16/23] vfio-user: proxy container connect/disconnect Date: Thu, 5 May 2022 10:19:59 -0700 Message-Id: <4f09120f9389cbaa6698f0d3351b7eb0bacbbc3d.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7467b5a6-8127-4e4d-dd1a-08da2eba4690 X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: lD4Qa+SsIOu75Emml1mKN8H6ou8/bFJCchvIDR30opZ60EkHhzulAysUEkMbKa1tzZnECgX8E/2gZWaY7iRYHwF1WjcQLosBBsEiZlK427yCCtl1XZMagTmL0gpDCwmMNiZceSznfxv9x9cSRJB3z1hUzRhhkmrhh2VYVxPFGncHKzpaE5zEbAQTNhyQzyHLlNbZt+KN+PT+D5//wQw/ac1QyGSAlIuBJbiufZYU/4SIUVKqHcXDwkPwL8PXeh8gkJaynVdyn1Lb6A9q/jHiJDz6gkS5zBNTB4Od37JmruHURuZYZTFr6Cnt0tcza4lnoRps6WCPmQrxoS2TCJg0PqFDvP+Fex5yHw3qzlUHLakDCDM/PfhTB4EFUzgMOmY/SkOjpGzce4X59CA/+WXZfUN1QulyAQ3ASJHnonvZ5CnXPrZdBMiv9ibzQjjJj0Vaf6pIkIJq610cjO1YOTUMiWCcSJwvw44NY6opKrVHmqIc4thEDMr3CVRL99jDQHl31IldOLHFMW/MJWbQYF7C+N+7c2En/MUCQVmiTULQq3F8kCshbvE72bkZYBPu9nCZK1xYFNzSQpNxC4mArstQrneJJgoy/fOIlw7kgfLTxU7sgeGe5VZ+5qLZO/EU/B14Cj0aIE/CSF4l9WOmCmz2+csIQOzZ1zRlmrtHusBWqvXnHnK+NYhRKvLbSCK6UgkIqn+DLtYrYs15Lb9I+82vZQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: sZZEkA/+PvNbbzc09Y1CuN5OMoc+ntw7efyLsLdxhT73r8tXx7hHIxFJvae32lvSEUI8mr+3cOhvSC14WOyCBQxXq01yMxhukPhoX+YzvdrlJyVqEvammuFO4ORMQhnbEjzp12dLMJW3XA/fdr8y5tMvrCjn9cMtu85BF8Z+/+M4zTdSpZQXGOW62mAZ1Lnvoq0InMR60Nih8SmVQ/gm5ZzVrhBdCk9jko7cooDJ29MxNE/cnyjVOjzHd3QLmeB2yJ28kFuy+m+/2yecJg8N6jVs77IklIFfBcwEdUmEvWEoG8YCuYQKUZ5ENQjJIIfjrvUQKMu+ap1UGn4x4UcXEt+oKOt6QhWXPxZwASKPOTI5qGVJ8UIFPzDg0CnfvZQ0+S/LwB0TORbG4oW4GIlvqyv1ynfm+p0AiU9chnV5bq0k6+XYIcIi+0zldKL57hN3N3KxzoG7wLLXByAuzXAA5MS7ESZuu3Pk9HHZZD5HUwKO+GFcgSIPC3Bg0J68ayBy62bTZB09lU85LcnYYguNXTduMWCLUyctbUfCa+pZGYwywuNgtKnA09PzYiJZjLEvx4Fgq2MULJaxjE99FMpZP6izu9rHXi/8ceDEzBzzP5NPqCB/43HCMFM8pkDwmZM0UkYs1eIWI4Cuc74bS/fJeMTpJQf1f5uXkpiEScMnDL9+Xvu+ZCCK6q0r/uD+yuWcAikgWZfYO1htAuTo7rFu5umU71uGBN7zxDkMpRWxq/8+XV4LEFT0IAtfJJ9rY++1UZbpzh5WkzJM43ESAvsr+YZgxL8lIzES0ZNK30AkK2gEUQ+F5PoYiQlMiGQjIoWA9jnqxLIBU/uxpuTUVmgBYOHOTG0f75MFDyWz+8dJpfqTxY3vyvMglcw1IycUo45haclExP0ph+uagem4UnywXCVqAV8+pQ9QaLnuGpYNMn/MyzGgAuIY9J7ZPCeLDIZeJVlK34lN1LCRSpUebwDovB333SGNca8LkXP9jMVwwiqxu0f/4Fd4ugnHQpXjtnzbc/AteQjC2oOAEYkLYnsux0tvQ1q99FZy/pNXYHfGSeX+7H8bRZBl5tXj/fJXRN4TPJYkGUdLn5YYYDOI4qL0Zzk8qGipzU4S5mWzEO+LoOtwdMKDyAwADKXlBHkf3Y6xSXWbxD1CJCpOy7qJsh7Z77dRoH90uAznLATcT17z3Es9fmNgHludLrNSkz6vIolNGlYyGM560MdS7Rd1zwhG2mBaFidu3zlljQxZeI7pJoZr75WV0XTAirzpCyYDBiyHCIprVh3XZlbGrAmcV/FUFn2+qRoFKnrLygjRc0uUgnfPYWGfMgHbw72QXpiBQ6/D7FDh9T4loZZx3kp6MsNimjTMftiXJkzapcpBm8cBQRIlhwAu2njOw7iXtcnS9vHKFrtB4w8vQ1PwxH+SImk3ybH812h0FaopkxTTajZeO/5Ycgz08/fesifxV5eoHHacswbyCFvZaqHJg0JdlGXf5nOTgQBKazfLDXI0IY1D2dhKTF+WZ+ghWliQDTVDKeKsTeQJi0b/cqBeR13Slp7HyQfnIXfYlwOzptxOiY+mYwVxkoWXqJf0LepMlFJQ2WDx8YWKsX+to3xY9J9aR4fmuCm/gFdkceAiD3mQMNnmm+kc0RoqPYh3OuOJ8AXJtbJeYezCUFsqdv2DWtX9Q08YGRoZJyDgqMr3Sa+Q6wA6r+MCT7viDwdF47/jT/g72kl062n5a3wz+dA3D7XOKh08629Av3ICQH7XFJphsWgEXM0= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7467b5a6-8127-4e4d-dd1a-08da2eba4690 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:21.0290 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cZdw+e6M6/rDPLxCgLveOWQZGk9Sz6RN0atUq5EXysOzd4hH0aQwyNrwcsUwWajcxHcaoHFB3NV6TOOM/ALw5149XVr+HH9eioR5lY4T8wA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: Mbn1vnbWwZ7T9SIcbb0L796smrWekuW0 X-Proofpoint-GUID: Mbn1vnbWwZ7T9SIcbb0L796smrWekuW0 Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 1 + include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 105 ++++++++++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 25 ++++++++++ hw/vfio/user.c | 3 ++ 5 files changed, 137 insertions(+) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index a641351..742e1a9 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -87,5 +87,6 @@ void vfio_user_set_handler(VFIODevice *vbasedev, int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); extern VFIODevIO vfio_dev_io_sock; +extern VFIOContIO vfio_cont_io_sock; #endif /* VFIO_USER_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 4118b8a..59a8299 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -94,6 +94,7 @@ typedef struct VFIOContainer { uint64_t max_dirty_bitmap_size; unsigned long pgsizes; unsigned int dma_max_mappings; + VFIOProxy *proxy; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; QLIST_HEAD(, VFIOGroup) group_list; @@ -278,6 +279,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp); void vfio_put_group(VFIOGroup *group); int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp); +void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as); +void vfio_disconnect_proxy(VFIOGroup *group); extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 351f727..beb5689 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -19,6 +19,7 @@ */ #include "qemu/osdep.h" +#include CONFIG_DEVICES #include #ifdef CONFIG_KVM #include @@ -2209,6 +2210,62 @@ put_space_exit: return ret; } + +#ifdef CONFIG_VFIO_USER + +void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as) +{ + VFIOAddressSpace *space; + VFIOContainer *container; + + if (QLIST_EMPTY(&vfio_group_list)) { + qemu_register_reset(vfio_reset_handler, NULL); + } + + QLIST_INSERT_HEAD(&vfio_group_list, group, next); + + /* + * try to mirror vfio_connect_container() + * as much as possible + */ + + space = vfio_get_address_space(as); + + container = g_malloc0(sizeof(*container)); + container->space = space; + container->fd = -1; + container->io_ops = &vfio_cont_io_sock; + QLIST_INIT(&container->giommu_list); + QLIST_INIT(&container->hostwin_list); + container->proxy = proxy; + + /* + * The proxy uses a SW IOMMU in lieu of the HW one + * used in the ioctl() version. Use TYPE1 with the + * target's page size for maximum capatibility + */ + container->iommu_type = VFIO_TYPE1_IOMMU; + vfio_host_win_add(container, 0, (hwaddr)-1, TARGET_PAGE_SIZE); + container->pgsizes = TARGET_PAGE_SIZE; + + container->dirty_pages_supported = true; + container->max_dirty_bitmap_size = VFIO_USER_DEF_MAX_XFER; + container->dirty_pgsizes = TARGET_PAGE_SIZE; + + QLIST_INIT(&container->group_list); + QLIST_INSERT_HEAD(&space->containers, container, next); + + group->container = container; + QLIST_INSERT_HEAD(&container->group_list, group, container_next); + + container->listener = vfio_memory_listener; + memory_listener_register(&container->listener, container->space->as); + container->initialized = true; +} + +#endif /* CONFIG_VFIO_USER */ + + static void vfio_disconnect_container(VFIOGroup *group) { VFIOContainer *container = group->container; @@ -2258,6 +2315,54 @@ static void vfio_disconnect_container(VFIOGroup *group) } } + +#ifdef CONFIG_VFIO_USER + +void vfio_disconnect_proxy(VFIOGroup *group) +{ + VFIOContainer *container = group->container; + VFIOAddressSpace *space = container->space; + VFIOGuestIOMMU *giommu, *tmp; + VFIOHostDMAWindow *hostwin, *next; + + /* + * try to mirror vfio_disconnect_container() + * as much as possible, knowing each device + * is in one group and one container + */ + + QLIST_REMOVE(group, container_next); + group->container = NULL; + + /* + * Explicitly release the listener first before unset container, + * since unset may destroy the backend container if it's the last + * group. + */ + memory_listener_unregister(&container->listener); + + QLIST_REMOVE(container, next); + + QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, tmp) { + memory_region_unregister_iommu_notifier( + MEMORY_REGION(giommu->iommu), &giommu->n); + QLIST_REMOVE(giommu, giommu_next); + g_free(giommu); + } + + QLIST_FOREACH_SAFE(hostwin, &container->hostwin_list, hostwin_next, + next) { + QLIST_REMOVE(hostwin, hostwin_next); + g_free(hostwin); + } + + g_free(container); + vfio_put_address_space(space); +} + +#endif /* CONFIG_VFIO_USER */ + + VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) { VFIOGroup *group; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 0a4208b..054a2bd 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3562,6 +3562,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIODevice *vbasedev = &vdev->vbasedev; SocketAddress addr; VFIOProxy *proxy; + VFIOGroup *group = NULL; struct vfio_device_info info; int ret; Error *err = NULL; @@ -3608,6 +3609,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->ops = &vfio_user_pci_ops; vbasedev->io_ops = &vfio_dev_io_sock; + /* + * each device gets its own group and container + * make them unrelated to any host IOMMU groupings + */ + group = g_malloc0(sizeof(*group)); + group->fd = -1; + group->groupid = -1; + QLIST_INIT(&group->device_list); + QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); + vbasedev->group = group; + + vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev)); + ret = VDEV_GET_INFO(vbasedev, &info); if (ret) { error_setg_errno(errp, -ret, "get info failure"); @@ -3673,6 +3687,10 @@ out_teardown: vfio_teardown_msi(vdev); vfio_bars_exit(vdev); error: + if (group != NULL) { + vfio_disconnect_proxy(group); + g_free(group); + } vfio_user_disconnect(proxy); error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } @@ -3681,6 +3699,13 @@ static void vfio_user_instance_finalize(Object *obj) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIODevice *vbasedev = &vdev->vbasedev; + VFIOGroup *group = vbasedev->group; + + if (group != NULL) { + vfio_disconnect_proxy(group); + g_free(group); + vbasedev->group = NULL; + } if (vdev->msix != NULL) { vfio_user_msix_teardown(vdev); diff --git a/hw/vfio/user.c b/hw/vfio/user.c index d0140d6..9906d81 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1337,3 +1337,6 @@ VFIODevIO vfio_dev_io_sock = { .region_write = vfio_user_io_region_write, }; + +VFIOContIO vfio_cont_io_sock = { +}; From patchwork Thu May 5 17:20:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 868F4C433F5 for ; Thu, 5 May 2022 17:47:34 +0000 (UTC) Received: from localhost ([::1]:58862 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfZd-00026X-O1 for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:47:33 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52826) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0z-0006B3-Uo for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:45 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:63340) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0t-0002Cj-NJ for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:43 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FTeZe018676 for ; Thu, 5 May 2022 17:11:28 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=0riSyXkpF6nuhVPb7+9TleWWgExFSznktzTyiNbDdNc=; b=AyqS3a9jn7aUJvdg4kVMqmKmXZcFs75SFhpOV39VSrsaRYVX9763eiB+e8wwIsNh7LU6 HFFNnlCzLyRqDTvPQuhcDPfF9HVfkHGQo2ubukcI/hyjC8G9cpT1mB8LrxtopBRwyjZM AMe0iSUPxFkRfwfHRKUMIa8TZc9P8fglSL0fOTEMmbGIXtlhnpTuLVmNizAFW4fED4Nu SQWa09vPziu+vUtubnEseY0NZdf9342QVNQ9Std/b2PjEEpd7S3xQ6dkgBERVsMbQZ+S x4bzdbZg57Ew09UhTR4IuHco8zH7JrqprsKpr+F7cGf622/+eIqBn1cZbvHhfmzrSH3i ng== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frwntc3kr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:27 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0Zqg006282 for ; Thu, 5 May 2022 17:11:26 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2174.outbound.protection.outlook.com [104.47.55.174]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fus8ybfv0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:26 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=e16bOWTU0+WTKq65YE6qZaZdeUcaFfmo3m0GM6E14H3eKUmsVzsQ+KUV6DcClnTG4MURpLuzL8o2+YlfTxa6k0HzOIXcjWe/PSfnhhZzxU1y0CoCj1HtRbgqjq0rLHf0gy1lGqeT6YKCIY63aQEO/ZvMakXfH9/OC6AvPcmgvr4TiofD18Djs7H8n5FBY84YcvzKtn+FgRhZuix9FqTuxOhXhkKxxucNNvsBFyXZBb2h2IJh0MS2UmIY8Qeba242hZ/HVWEO+7YeX/a0C9w9JbP2sA/Qh5foZxh/Y3bKRRXFYfVMuCVNe1lJJ2W9ASWgmIf/fSmOEGz79p6j8eR7oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=0riSyXkpF6nuhVPb7+9TleWWgExFSznktzTyiNbDdNc=; b=H6LK8ksMVPFujxuuDYTMg35lU+u5sgE9lSYjhA8fjvZW5xRH84hAAUfoxLPYAgFAQQLediWs+n6FoU1NSug6WHtFkHEtchxU/4QBNMgiWD6Y0IZIK9cIR0UwC1dM+fp7xIHvyQ7cODr+waAjDl81GnaXhLTZN4uk/WXP8Swqb1N/eDKoBTwIEn+ByHCUuPP3d/lpZzKyF3GK8HG13rARU9Yykjgcrxw6J7YW0qa50fcbL9/HcIe3La/C4AQTdczcKYfOCJlzh5utTckOvmt15XxJQ0+aa4xiZhRlht27zm35L41leBljD9CAwZ3m5Pj/ic8bYTP+ck3AXT8wgcK/zw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=0riSyXkpF6nuhVPb7+9TleWWgExFSznktzTyiNbDdNc=; b=QcurvDdTZ77dlKQUvHhkUwGtjF8nh+z+sUTzzyhzBVCBjss0XeNScU5Lleehn3q/GUjjzwsCwHIW7ZMM3eq+yDVo7uW8iHrokeAR7Mi6/dc2PNqpUBL4ppWEEgZbzWXIZe1rSh2+GQRFf0kLP8o1i2B1Qvqi+Br9IozjUeGMQCk= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:24 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:24 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 17/23] vfio-user: dma map/unmap operations Date: Thu, 5 May 2022 10:20:00 -0700 Message-Id: <8f76f28dfaaeb3a6237136ecb5603431f62c657a.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: d2f9e06a-8d69-46c9-ed9f-08da2eba46c2 X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Z78iIGCd1vUBpF+ra7OlVwxJZ3CeVzpibSFBwMYZrvjd6Gwr+LT2Xh+9mipypqmzP/hxi2Z0HmMdNNnnyx7eKxD0uzQU2XpdOzaa6AzlLIbVeHyzq5DLJifZHrdBDMQsMA4slw5q85vW3OdgxT/S3Ex+5eZ5YWuEuwX/34CfJ6p6ApnDQgt3ERxBm/v5aEr/RFKX5Sm5fpEpsbHGjrJA0VNCfyJA8ag3pn2C85dq37SEatjPAZGZxAErSOyNgrELOg+6ccn5c4RrOqwtEZQj1ms2glh1oWzYGdT04HIsK+9oAeKLG+cCoG8gCAJi8C41aaQ3At3OBhIJGZxbTGu4PBOoifqhzk/DtS/lrKtRQoNVwwdYhAP/1Q5AM+HmM5HcTGaPb8MUZdB4fYmzCmzzUuBtgA/nVDUOMuz+VMh4NbqRTt5PA8vaeMZcypphlDw/CTpxGKXBap4FltTDU2mE9dcPu5zudRWx/n+CiyJEuxHeonypPy91FNn63RJIJJwU4LwnJL5Jxfi/t/0807DCVhVSkoBt9bJ1gZ4w3lsdqlGSLjWs2nYlsoXyvJo5FVRCoExkXQLGliQ9g3LO0DRDan+WPb00+5O5Dl6iMTXlgzzXrqjRMd7iU3cCivbetjk/pZumH20rgFuNiAQk/G1wY84/N6ioqUnR+tpgjr0WyTxAGEo32QV308TZQQlO23LDeN8XPEwNd/FwWxDT0AzfXw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(30864003)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: B0WwIvPWuqu2W38UdHrDgbKi+4nNraKrtIcST1hMUw93Qba1SCsF1dsVyKb7N1qxaGDjDbX6wNonvwhqOK1dToWWnH9jCVXcQ8s+orkgzft0FXqn+52Kbp0PBK0nWezWFNyaaZ0b05U6a8tFsbC3mV7+fCnxC324VkVlaD+3Q+hW5rHuy6RJ7b88yUNTtb7CvvuCssu8Us1GxEeJme/asgIexK6dNk2zDvZg7GfpPUBEtfRZIi3hGdUPxjqmGVRH3Z3oz+aB3c4G/N3vAncBdQlpNUqNkHfFrmkX1YBB9xcZ7Vl3hljCTmnwrBVWwjUSVbkZRExNhBke+GmpIF+JAnXAhlkHo6w5+ajBrUDHiIdGg49mInEE231nwgGrqD+o5jbzy0zo+cC6rTdQl2qhQey5SKcMzW/ZmHldZRKLyuk/32dN20Wnot0OOvpO2HvNobusuetU3JYQ+GHKkBx5pxrS0BT/bMKmn5146UZOZ0unX/+eaHwV2+uck7aAQLM5Fag1wL5zGqsm/1tCgfk++eC/0FdIDfJ/HmjTnvVigtkL9O5tYkOL6IKQ2VM/1pu4lKvXRaL7YDs0TNCqO6+y6mG8y+XwkiO2tLh4rirQQG/2fTv5BUEGcTSOxftAowERJ66rfHzTl4iRPYFZj3cPBSz2HgvqdKBm/ZYQ1GoDiJO8j+HSxG/T1ZezzLpi17BJE391/Kw0HKv37Y6agUDkLeWm9KMKCv9nLWHnzfuQpdfBN5GhnoARvK4PeqzcfnBANZx0WkFZaKRp/7Lj6WrrEupyJEzE4Qf3peVi1rfUgT7htZP+uvoWbcG5bDyJHTN8KTpGJ8HDHcWHs4StAjGlbTMubg6uck5nqu0WQ5auvsgIbTRT3NV+Oo5w41ytiKmp40ciFUfyuOOwO8/CrPnJROd94c2Wdq1jlLYrzvRdhP9YFKUmuFtkuqMEtAzILvF5Wtz+ajzuHNKGICtE0rknIEOov1TaRV5bnDiku0YuZd1qfS9WkHCjgIHGSAebTwbS+MGTgzfl1I5MFPqeOac0EWp8DsuNMw0EGeEP/fPWqMDJsNh8INAn6Fk37sSTFLu6/EO+xS6Vht1ZV4ZBQfnkov8cESQuA6Xc6Bw9QevH7mwHboslQjU1UYcIUadIQ5ZKzt7QhbSYEIplpvM3Dcwwa3p8dHMbjnswXkC6LCi4nvaTe5In2euB83ZUJDJXFuVDFGncaEzxSjStnZIZ2qQTn7B1I+PcHXj+oXIPAVb5eag7mn+6uq8ElklN497LiZ2pObkz2RdwfcAxKjewmPC2KtZlSVtp2+piIj6YKfw8RRpef6mJo8XJsrevEci+/mCSCIT6V6aFcxqD4ROx0aUN6lUFn86psVzkgkd+i5w+kAiFj3zo/hkJobr2RL7UcoDmSF+tqC7oPCF5MepSBsA0VNoMQm6Mwl+SBpX+oL6zlwilGhWVjWYAvP6qvCWW6tm1HFaUpE1DFpbbznIR8UyecyFfK1ofWEKkiL9Ds+iOGC2VmPlWI2P7m4FyYj8eryeNIJhIm6Y1NchmVzu84kRC/cmnuwktIzkLtR5iAdFTs0t5d2BPFZtu6AS79Lqm58gqC5yod1+PnXZ993jF7BDvE2rZqoYMZUuqO9/ccXQbra56OxrmZyrkbGcplzvHR2NZPqUEYrQlnttZ2YETPfdxiHmWpkgp/RvO5F6DKnDfI71WkF1PnsJbZ7RSV1hC7guWEpntIs4WZw5/fZCucTYaFLH5rxz+tj91feV6u027ffA= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: d2f9e06a-8d69-46c9-ed9f-08da2eba46c2 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:21.3414 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1MXDGh5WzYtKj5egQUZcd+CqDCKLhuX+UMlkTL0GAEXEsStsU8jbU3jSUA7QzcRy7D3F1lksYt7cXYMpP4i7rMS/e1RBNAS2ltxd2PTRMGo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: XugVJftQq4NLcXcd1qfYGPGwWpNhMzUQ X-Proofpoint-GUID: XugVJftQq4NLcXcd1qfYGPGwWpNhMzUQ Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add ability to do async operations during memory transactions Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson --- hw/vfio/user-protocol.h | 32 +++++++ include/hw/vfio/vfio-common.h | 9 +- hw/vfio/common.c | 63 +++++++++--- hw/vfio/user.c | 217 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 305 insertions(+), 16 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 4852882..ad63f21 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -94,6 +94,31 @@ typedef struct { /* + * VFIO_USER_DMA_MAP + * imported from struct vfio_iommu_type1_dma_map + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint64_t offset; /* FD offset */ + uint64_t iova; + uint64_t size; +} VFIOUserDMAMap; + +/* + * VFIO_USER_DMA_UNMAP + * imported from struct vfio_iommu_type1_dma_unmap + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint64_t iova; + uint64_t size; +} VFIOUserDMAUnmap; + +/* * VFIO_USER_DEVICE_GET_INFO * imported from struct_device_info */ @@ -157,4 +182,11 @@ typedef struct { char data[]; } VFIOUserRegionRW; +/*imported from struct vfio_bitmap */ +typedef struct { + uint64_t pgsize; + uint64_t size; + char data[]; +} VFIOUserBitmap; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 59a8299..a84e10a 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -90,6 +90,7 @@ typedef struct VFIOContainer { VFIOContIO *io_ops; bool initialized; bool dirty_pages_supported; + bool async_ops; uint64_t dirty_pgsizes; uint64_t max_dirty_bitmap_size; unsigned long pgsizes; @@ -199,7 +200,7 @@ struct VFIODevIO { ((vdev)->io_ops->region_write((vdev), (nr), (off), (size), (data), (post))) struct VFIOContIO { - int (*dma_map)(VFIOContainer *container, + int (*dma_map)(VFIOContainer *container, MemoryRegion *mr, struct vfio_iommu_type1_dma_map *map); int (*dma_unmap)(VFIOContainer *container, struct vfio_iommu_type1_dma_unmap *unmap, @@ -207,14 +208,16 @@ struct VFIOContIO { int (*dirty_bitmap)(VFIOContainer *container, struct vfio_iommu_type1_dirty_bitmap *bitmap, struct vfio_iommu_type1_dirty_bitmap_get *range); + void (*wait_commit)(VFIOContainer *container); }; -#define CONT_DMA_MAP(cont, map) \ - ((cont)->io_ops->dma_map((cont), (map))) +#define CONT_DMA_MAP(cont, mr, map) \ + ((cont)->io_ops->dma_map((cont), (mr), (map))) #define CONT_DMA_UNMAP(cont, unmap, bitmap) \ ((cont)->io_ops->dma_unmap((cont), (unmap), (bitmap))) #define CONT_DIRTY_BITMAP(cont, bitmap, range) \ ((cont)->io_ops->dirty_bitmap((cont), (bitmap), (range))) +#define CONT_WAIT_COMMIT(cont) ((cont)->io_ops->wait_commit(cont)) extern VFIODevIO vfio_dev_io_ioctl; extern VFIOContIO vfio_cont_io_ioctl; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index beb5689..a9d9991 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -480,7 +480,7 @@ static int vfio_dma_unmap(VFIOContainer *container, return CONT_DMA_UNMAP(container, &unmap, NULL); } -static int vfio_dma_map(VFIOContainer *container, hwaddr iova, +static int vfio_dma_map(VFIOContainer *container, MemoryRegion *mr, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly) { struct vfio_iommu_type1_dma_map map = { @@ -496,7 +496,7 @@ static int vfio_dma_map(VFIOContainer *container, hwaddr iova, map.flags |= VFIO_DMA_MAP_FLAG_WRITE; } - ret = CONT_DMA_MAP(container, &map); + ret = CONT_DMA_MAP(container, mr, &map); if (ret < 0) { error_report("VFIO_MAP_DMA failed: %s", strerror(-ret)); @@ -559,7 +559,8 @@ static bool vfio_listener_skipped_section(MemoryRegionSection *section) /* Called with rcu_read_lock held. */ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, - ram_addr_t *ram_addr, bool *read_only) + ram_addr_t *ram_addr, bool *read_only, + MemoryRegion **mrp) { MemoryRegion *mr; hwaddr xlat; @@ -640,6 +641,10 @@ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, *read_only = !writable || mr->readonly; } + if (mrp != NULL) { + *mrp = mr; + } + return true; } @@ -647,6 +652,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) { VFIOGuestIOMMU *giommu = container_of(n, VFIOGuestIOMMU, n); VFIOContainer *container = giommu->container; + MemoryRegion *mr; hwaddr iova = iotlb->iova + giommu->iommu_offset; void *vaddr; int ret; @@ -665,7 +671,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) { bool read_only; - if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only)) { + if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, &mr)) { goto out; } /* @@ -675,14 +681,14 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) * of vaddr will always be there, even if the memory object is * destroyed and its backing memory munmap-ed. */ - ret = vfio_dma_map(container, iova, + ret = vfio_dma_map(container, mr, iova, iotlb->addr_mask + 1, vaddr, read_only); if (ret) { error_report("vfio_dma_map(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx", %p) = %d (%m)", + "0x%"HWADDR_PRIx", %p)", container, iova, - iotlb->addr_mask + 1, vaddr, ret); + iotlb->addr_mask + 1, vaddr); } } else { ret = vfio_dma_unmap(container, iova, iotlb->addr_mask + 1, iotlb); @@ -737,7 +743,7 @@ static int vfio_ram_discard_notify_populate(RamDiscardListener *rdl, section->offset_within_address_space; vaddr = memory_region_get_ram_ptr(section->mr) + start; - ret = vfio_dma_map(vrdl->container, iova, next - start, + ret = vfio_dma_map(vrdl->container, section->mr, iova, next - start, vaddr, section->readonly); if (ret) { /* Rollback */ @@ -845,6 +851,29 @@ static void vfio_unregister_ram_discard_listener(VFIOContainer *container, g_free(vrdl); } +static void vfio_listener_begin(MemoryListener *listener) +{ + VFIOContainer *container = container_of(listener, VFIOContainer, listener); + + /* + * When DMA space is the physical address space, + * the region add/del listeners will fire during + * memory update transactions. These depend on BQL + * being held, so do any resulting map/demap ops async + * while keeping BQL. + */ + container->async_ops = true; +} + +static void vfio_listener_commit(MemoryListener *listener) +{ + VFIOContainer *container = container_of(listener, VFIOContainer, listener); + + /* wait here for any async requests sent during the transaction */ + CONT_WAIT_COMMIT(container); + container->async_ops = false; +} + static void vfio_listener_region_add(MemoryListener *listener, MemoryRegionSection *section) { @@ -1044,12 +1073,12 @@ static void vfio_listener_region_add(MemoryListener *listener, } } - ret = vfio_dma_map(container, iova, int128_get64(llsize), + ret = vfio_dma_map(container, section->mr, iova, int128_get64(llsize), vaddr, section->readonly); if (ret) { error_setg(&err, "vfio_dma_map(%p, 0x%"HWADDR_PRIx", " - "0x%"HWADDR_PRIx", %p) = %d (%m)", - container, iova, int128_get64(llsize), vaddr, ret); + "0x%"HWADDR_PRIx", %p)", + container, iova, int128_get64(llsize), vaddr); if (memory_region_is_ram_device(section->mr)) { /* Allow unexpected mappings not to be fatal for RAM devices */ error_report_err(err); @@ -1310,7 +1339,7 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) } rcu_read_lock(); - if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL)) { + if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL, NULL)) { int ret; ret = vfio_get_dirty_bitmap(container, iova, iotlb->addr_mask + 1, @@ -1428,6 +1457,8 @@ static void vfio_listener_log_sync(MemoryListener *listener, static const MemoryListener vfio_memory_listener = { .name = "vfio", + .begin = vfio_listener_begin, + .commit = vfio_listener_commit, .region_add = vfio_listener_region_add, .region_del = vfio_listener_region_del, .log_global_start = vfio_listener_log_global_start, @@ -2819,7 +2850,7 @@ VFIODevIO vfio_dev_io_ioctl = { .region_write = vfio_io_region_write, }; -static int vfio_io_dma_map(VFIOContainer *container, +static int vfio_io_dma_map(VFIOContainer *container, MemoryRegion *mr, struct vfio_iommu_type1_dma_map *map) { @@ -2879,8 +2910,14 @@ static int vfio_io_dirty_bitmap(VFIOContainer *container, return ret < 0 ? -errno : ret; } +static void vfio_io_wait_commit(VFIOContainer *container) +{ + /* ioctl()s are synchronous */ +} + VFIOContIO vfio_cont_io_ioctl = { .dma_map = vfio_io_dma_map, .dma_unmap = vfio_io_dma_unmap, .dirty_bitmap = vfio_io_dirty_bitmap, + .wait_commit = vfio_io_wait_commit, }; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 9906d81..29eff8a 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -59,8 +59,11 @@ static void vfio_user_request(void *opaque); static int vfio_user_send_queued(VFIOProxy *proxy, VFIOUserMsg *msg); static void vfio_user_send_async(VFIOProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds); +static void vfio_user_send_nowait(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize); static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize, bool nobql); +static void vfio_user_wait_reqs(VFIOProxy *proxy); static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, uint32_t size, uint32_t flags); @@ -647,6 +650,36 @@ static void vfio_user_send_async(VFIOProxy *proxy, VFIOUserHdr *hdr, } } +/* + * nowait send - vfio_wait_reqs() can wait for it later + */ +static void vfio_user_send_nowait(VFIOProxy *proxy, VFIOUserHdr *hdr, + VFIOUserFDs *fds, int rsize) +{ + VFIOUserMsg *msg; + int ret; + + if (hdr->flags & VFIO_USER_NO_REPLY) { + error_printf("vfio_user_send_nowait on async message\n"); + return; + } + + QEMU_LOCK_GUARD(&proxy->lock); + + msg = vfio_user_getmsg(proxy, hdr, fds); + msg->id = hdr->id; + msg->rsize = rsize ? rsize : hdr->size; + msg->type = VFIO_MSG_NOWAIT; + + ret = vfio_user_send_queued(proxy, msg); + if (ret < 0) { + vfio_user_recycle(proxy, msg); + return; + } + + proxy->last_nowait = msg; +} + static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, VFIOUserFDs *fds, int rsize, bool nobql) { @@ -696,6 +729,57 @@ static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, } } +static void vfio_user_wait_reqs(VFIOProxy *proxy) +{ + VFIOUserMsg *msg; + bool iolock = false; + + /* + * Any DMA map/unmap requests sent in the middle + * of a memory region transaction were sent nowait. + * Wait for them here. + */ + qemu_mutex_lock(&proxy->lock); + if (proxy->last_nowait != NULL) { + iolock = qemu_mutex_iothread_locked(); + if (iolock) { + qemu_mutex_unlock_iothread(); + } + + /* + * Change type to WAIT to wait for reply + */ + msg = proxy->last_nowait; + msg->type = VFIO_MSG_WAIT; + while (!msg->complete) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + QTAILQ_REMOVE(&proxy->pending, msg, next); + error_printf("vfio_wait_reqs - timed out\n"); + break; + } + } + + if (msg->hdr->flags & VFIO_USER_ERROR) { + error_printf("vfio_user_wait_reqs - error reply on async request "); + error_printf("command %x error %s\n", msg->hdr->command, + strerror(msg->hdr->error_reply)); + } + + proxy->last_nowait = NULL; + /* + * Change type back to NOWAIT to free + */ + msg->type = VFIO_MSG_NOWAIT; + vfio_user_recycle(proxy, msg); + } + + /* lock order is BQL->proxy - don't hold proxy when getting BQL */ + qemu_mutex_unlock(&proxy->lock); + if (iolock) { + qemu_mutex_lock_iothread(); + } +} + static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -1021,6 +1105,103 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) return 0; } +static int vfio_user_dma_map(VFIOProxy *proxy, + struct vfio_iommu_type1_dma_map *map, + int fd, bool will_commit) +{ + VFIOUserFDs *fds = NULL; + VFIOUserDMAMap *msgp = g_malloc0(sizeof(*msgp)); + int ret; + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DMA_MAP, sizeof(*msgp), 0); + msgp->argsz = map->argsz; + msgp->flags = map->flags; + msgp->offset = map->vaddr; + msgp->iova = map->iova; + msgp->size = map->size; + + /* + * The will_commit case sends without blocking or dropping BQL. + * They're later waited for in vfio_send_wait_reqs. + */ + if (will_commit) { + /* can't use auto variable since we don't block */ + if (fd != -1) { + fds = vfio_user_getfds(1); + fds->send_fds = 1; + fds->fds[0] = fd; + } + vfio_user_send_nowait(proxy, &msgp->hdr, fds, 0); + ret = 0; + } else { + VFIOUserFDs local_fds = { 1, 0, &fd }; + + fds = fd != -1 ? &local_fds : NULL; + vfio_user_send_wait(proxy, &msgp->hdr, fds, 0, will_commit); + ret = (msgp->hdr.flags & VFIO_USER_ERROR) ? -msgp->hdr.error_reply : 0; + g_free(msgp); + } + + return ret; +} + +static int vfio_user_dma_unmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap, bool will_commit) +{ + struct { + VFIOUserDMAUnmap msg; + VFIOUserBitmap bitmap; + } *msgp = NULL; + int msize, rsize; + bool blocking = !will_commit; + + if (bitmap == NULL && + (unmap->flags & VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP)) { + error_printf("vfio_user_dma_unmap mismatched flags and bitmap\n"); + return -EINVAL; + } + + /* + * If a dirty bitmap is returned, allocate extra space for it + * and block for reply even in the will_commit case. + * Otherwise, can send the unmap request without waiting. + */ + if (bitmap != NULL) { + blocking = true; + msize = sizeof(*msgp); + rsize = msize + bitmap->size; + msgp = g_malloc0(rsize); + msgp->bitmap.pgsize = bitmap->pgsize; + msgp->bitmap.size = bitmap->size; + } else { + msize = rsize = sizeof(VFIOUserDMAUnmap); + msgp = g_malloc0(rsize); + } + + vfio_user_request_msg(&msgp->msg.hdr, VFIO_USER_DMA_UNMAP, msize, 0); + msgp->msg.argsz = rsize - sizeof(VFIOUserHdr); + msgp->msg.argsz = unmap->argsz; + msgp->msg.flags = unmap->flags; + msgp->msg.iova = unmap->iova; + msgp->msg.size = unmap->size; + + if (blocking) { + vfio_user_send_wait(proxy, &msgp->msg.hdr, NULL, rsize, will_commit); + if (msgp->msg.hdr.flags & VFIO_USER_ERROR) { + return -msgp->msg.hdr.error_reply; + } + if (bitmap != NULL) { + memcpy(bitmap->data, &msgp->bitmap.data, bitmap->size); + } + g_free(msgp); + } else { + vfio_user_send_nowait(proxy, &msgp->msg.hdr, NULL, rsize); + } + + return 0; +} + static int vfio_user_get_info(VFIOProxy *proxy, struct vfio_device_info *info) { VFIOUserDeviceInfo msg; @@ -1338,5 +1519,41 @@ VFIODevIO vfio_dev_io_sock = { }; +static int vfio_user_io_dma_map(VFIOContainer *container, MemoryRegion *mr, + struct vfio_iommu_type1_dma_map *map) +{ + int fd = memory_region_get_fd(mr); + + /* + * map->vaddr enters as a QEMU process address + * make it either a file offset for mapped areas or 0 + */ + if (fd != -1) { + void *addr = (void *)(uintptr_t)map->vaddr; + + map->vaddr = qemu_ram_block_host_offset(mr->ram_block, addr); + } else { + map->vaddr = 0; + } + + return vfio_user_dma_map(container->proxy, map, fd, container->async_ops); +} + +static int vfio_user_io_dma_unmap(VFIOContainer *container, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap) +{ + return vfio_user_dma_unmap(container->proxy, unmap, bitmap, + container->async_ops); +} + +static void vfio_user_io_wait_commit(VFIOContainer *container) +{ + vfio_user_wait_reqs(container->proxy); +} + VFIOContIO vfio_cont_io_sock = { + .dma_map = vfio_user_io_dma_map, + .dma_unmap = vfio_user_io_dma_unmap, + .wait_commit = vfio_user_io_wait_commit, }; From patchwork Thu May 5 17:20:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839819 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 563C8C433F5 for ; Thu, 5 May 2022 17:22:41 +0000 (UTC) Received: from localhost ([::1]:48876 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfBY-0000Tg-4w for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:22:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52640) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0m-00063Q-Hc for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:34 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:18670) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0i-0002Cb-NN for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:32 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245F4QNM026132 for ; Thu, 5 May 2022 17:11:27 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=BGLSCE8TMVyI2IuUbCHzxeNWmxLKzvzE4Tm0bcBtTvw=; b=B3eWrfbkj7WixJSzEtKhjuzvwwfLtJbdv2SorXWTAP86RdkduSX6Fn2MCs80ARyFbLKa OBodFlrAiyn40memttrCT3CTp1zdquuMH+R7JtVFB31kAE9ufjLt6yrvl5ROWzNqAFUu sAAh2lJwJgloUx8jc6KzN5gMAzsnfi0yc/pJa/oBSuUeNmcEJRuRFuWPnDTZXIAz3p2o 39794RF892oy6mMU7HKQ80wZuN6B8R9nc3OVHSlTKXdTCDNt5YkgrrK9NE+BJFRqOEAZ 0JsbwBQR3MMkHhWrm+4+DWhD/+JlvdmnXsYRl5Yux5pTTs47e6gO62Z2WV+KrEFBoMwz aw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruhcbs69-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:27 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0Zqh006282 for ; Thu, 5 May 2022 17:11:26 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2174.outbound.protection.outlook.com [104.47.55.174]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fus8ybfv0-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:26 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TjcF93vpQCAXNJqdJLuPwW30U2Rl46wy1SSkKgaDlpzEj0eF4nnagC+iA6gqvf2mr8Cq8n0UUczLXSiFG6LPMM2QhpDve73bkYw5UTLbBPhCSiIOOYeGL0z5PV+eSNZFfHWk3Y3aYi8XNtk+L/YQjxRnyRch97mI52+oc4USgu3h18mJt83uAKQ9T7k1qSBRYWTlmLpy29THOrjY80ywGKIXTikCVFhlTKPZeNqVdyC5VR5CpG056OddCOlPC3Ey0fAWalJHrP+I7rxs/jeu1XKtrSVko2Z5AJv5gV9oJqfyE5NF6VIw4nPKMu2ydSKEevvMgo+ZSozERY+G+Y8inQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=BGLSCE8TMVyI2IuUbCHzxeNWmxLKzvzE4Tm0bcBtTvw=; b=GvTws8X4mGeTWbUVtXSWzaoXl8zREeVZHdCjqL38gEp1MHc+xblYA4UQ/PXw2S/3kOA1NZtMTqCDUlalki2akrWdcO0UgLgZA+ftrN1pSmyPRFrtWk3btN1YiX3pCzJOf/ztXCZQJlWVZyFJOH/Wc0Fejwrn/cWXLn9J8NeCMCFOvsDciZgXTs+DT2ietOu9T5Y67dBOiVXHAaK9NNw7FcI8d4qzr9JY3ZUdaiUVN7c5Aa++aUzz8CjZrsrORUX52KSC0JSaVT26WZxuG9q3V3Yi+pRrBW1dHTlHi4xp+ksyC0ODPZ/j78Lz7ksLO5YLQZQLjd4V7y2eyMeHAMXbNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BGLSCE8TMVyI2IuUbCHzxeNWmxLKzvzE4Tm0bcBtTvw=; b=pWm0P/vNni313s8VnvGdtbpJQWKgyR/cPylj1qx523yYywYOEAOvoXuJ0GiCvkOfKZC8VpWrqyGZDOTv+we1ldIVNjv576YklzuyCZzys63J1nsS0Jfebr1USsI6SvN0HxyDZDDIIaC/5KrZAOPqxMgeawfv2oE1V/63gFVXLJw= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:25 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:25 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 18/23] vfio-user: secure DMA support Date: Thu, 5 May 2022 10:20:01 -0700 Message-Id: <009bd0af58106715c1b5be2454a1485909c213b5.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 104b1e7e-ab3e-4910-fe48-08da2eba46f4 X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IJNyh+7YIo9S7dAGSCJr5fZtKAifM79snhMBj/dvZC6bk8RgqZLksMY6tusr9+A7d/2YZ2jJAgNYTFbQPMLVxTK2jbUnbdBvRziwgS76+ovBJu5U9OlFipMXcJbMSYmOEgeKKix+7gI+wtAmPZyPZzt3b1uf10l/AqqoVfpyFqrXBIFZAEA3H95CRsKxU7Sui8IDJcqY/UPyxyjyEunAYXbHt/tUtbsQYSohAcMvorrEiq5cKs9olGKttURi6z5h1hSXpsDfyTlv/xf5WlUGHJWY8wNt2KbSTcLYoqEEJNutBXJkXoC7kdpuUz7thkw6pUPtt2KzRm8UlEAhz8dSGAjQYGC7G6YHAz4xKsB5xL/Mm7l9r902ozU/pXQB3oUf1m+sHtg4nHuPzGjt7wKIQPjYR14S7JdbR8o9TCJgQn/vzWly+7SC/cFWxiz0NpyyzflkO5dqR+quhEXzVrJ38qyijpNCYWG/s0+oMdpEzACYRpGpmk0TV9ztt70dNm/fKTTqGrAKTcVm4hQrFLYl5LOVnea4u+IHTIJD0otZFwWlOhFawFeV/nZ2tMu05/XNgeSKoekitx3/JJXp6kBNMI4g3XxCi+srcmm4n1rA6Kvrtv/MTV6HrGQPSkdimxD7UV5+cUSbfhY9IGmZd43IVtz14nj8lSkLal9/Hgh2DO60jvGYmtsB8PSGI/hB0e8DV9W9+xWD2qdFJhb0+tjK8g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: s37pWdYFNT9K+45RmUtjA/mEpyfDXz1CdM9ZMFYzOFgWu3lNK8bVN0YMdvMKXWGLeMwWpOZj9++Zic++buooMYj2Bs6GwZKz2uxOMZUyI2qlaKQupA43+hlcKIFpj0RF579r7MGlu2jaMlNflGJ64v2lstzpRfUGzxkZf/fzbB/WS30lGhyGMUiIQLqOJYyYKMT88CFITYlWztL5MszIg3pcOyGaGLBPRe+kwgN/cFiBE3Ics8ouKTFVBaIK2OMqqy/RwZ1mbcGYPW3ucFxE9+RNd5VewLmNiws+qfi20Khn7YU83kRP9aYeHSJ/uxJ3tZL09n8xNo2Y8/Fh6rL9A6/hT8e8YyI/F146k1Y0aqXl9+g3XOwkdsMd0Ur7D35R3QCXPpjvU4zfyH4W0RKQ+SelIiiKNyViHuNFtv9nasE21rGw1q88n0Q4EbNVE6ieLvbVLfTIc6NAGk5120/w+yv2knfg3Qy9w8k7yhGjoP3RU/pmfMvWpDZktu/+d/lSnggULr8II79lnd5UmbV5yv4s7GpKP9ZWTeDiMkVDjtqLOqwMEebutWMU3SHe2U2tGol1jWkPD+/BtGZKHGIrzHppmysKxMBzKcpVRWTs6gtQ0mceYZdt82xTQ3RfC7oPvcM/gs84LDyo2q0mYDscaSJbiFXHy5WKm4PrlK3EuoYiFjR1fGFSjcYKwCV0zsE0rebuCmfR0X0jUQvOTr8E2RjaqMwWXpDR/7Lprlx1S8WKyiv7L6K66lOZugp7hcpAnYvpkp8hb8LLCSRPOtBg0iq1Wu6MxEE7J/ylU3FPqsGkukbK/r1cfEYJpKZiBtUTSPdiM4mKPV7TI4+P+/wi4YQSuX6pg+D0F7DEWCmZZTbPP/3n6t0KusVLfK0zhqRKpN9j54534JqKJpR4M9mMbYah/mMm2Qr4E06m+qIobtls5lwhmQwyCIU82fnXb2vvAclAnADE01G2SLiWv8N5rt16ZVt2yE90wgovHaXDSXSr5xLWQRruvOo7yDPl3Ak0kMIPEOpE95j68BkmimAtygqySJjHvFKeVdwQzt4ytjaZV/S1WyFjiKVDRVo7zXalrBz7qO816RgueFvIyw7286lPbG5v/2yNTxnNHzoy8i44g4Y4CfxqfojNqZ2/gwsYTm+f2a7Tk0BWRwlDEocdw4OAwFHkNLCCzXsmWMHLMaR4kwOZZYQOUCZgk1+g0MBbfKT5+cyXwrq8KJwapNmgMQqg/fJoV1dvwO2WwSWq8jaWrxx1JrPO/RwCQxjhisyQN16myXKQa4dl82cPFRorbU/1XJ/lJEXUQXaPmGTZRHcAHYQ1V+OKLGEk40Qj5usaHVPd0c8it0gCMJc75lXdnzINOu4oJ56OsQVrfBkJbwJM2EpMKB/p7LXTespzohO4Jy6wRRfc+NX8SV8NPCKSV0QNOLbWI+3XgWThnxJZ7NMWMWDbnUSbwo8sqKEoBhlxUcNW/dJdDHoha+vgJJZaITKM3Rye9hgtWtjrxl6l2pNnP6uhnPCRDTkmIey404fgldaM5gy1uvBYjPbL14HoFTXyG34iZq9pXgzSUTzR8DSNCw857CvOsGTBFOIWaMmuZebzHBBUB1iOjBHw6CoHzQmXUE/7hIowxeEtogi26eY9I1+0vbPWzClNb+qRubAD7k8dBR109NVRdwbSC0Kau6MsszsmOb78BrpWMfSvTCqarGeCTCtN7RTF9RtNuwKZAgb0o8MxSAxN3y9Flszhj5OPraepQoWm4SIoJ9lw134= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 104b1e7e-ab3e-4910-fe48-08da2eba46f4 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:21.7008 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: r4fFC3AfIWPKPjhk/ju3mXmm3lT/7Hg3BKD5WTPX4cuZfcBAZDmQw9gZ8bi78OfQdsXbXAccW6LE2dXEbMaN8nQsZKbGUt0Co614EpYOEag= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: Fw-FnWEtTQW-ck9bnxeaPgzD0d8HJEfh X-Proofpoint-ORIG-GUID: Fw-FnWEtTQW-ck9bnxeaPgzD0d8HJEfh Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Secure DMA forces the remote process to use DMA r/w messages instead of directly mapping guest memeory. Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 1 + hw/vfio/user.h | 1 + hw/vfio/pci.c | 4 ++++ hw/vfio/user.c | 2 +- 4 files changed, 7 insertions(+), 1 deletion(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index a4eb5b9..c207847 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -194,6 +194,7 @@ OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) struct VFIOUserPCIDevice { VFIOPCIDevice device; char *sock_name; + bool secure_dma; /* disable shared mem for DMA */ bool send_queued; /* all sends are queued */ bool no_post; /* all regions write are sync */ }; diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 742e1a9..ec764d3 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -76,6 +76,7 @@ typedef struct VFIOProxy { /* VFIOProxy flags */ #define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_SECURE 0x2 #define VFIO_PROXY_FORCE_QUEUED 0x4 #define VFIO_PROXY_NO_POST 0x8 diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 054a2bd..2faf890 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3589,6 +3589,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->proxy = proxy; vfio_user_set_handler(vbasedev, vfio_user_pci_process_req, vdev); + if (udev->secure_dma) { + proxy->flags |= VFIO_PROXY_SECURE; + } if (udev->send_queued) { proxy->flags |= VFIO_PROXY_FORCE_QUEUED; } @@ -3720,6 +3723,7 @@ static void vfio_user_instance_finalize(Object *obj) static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), + DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false), DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false), DEFINE_PROP_END_OF_LIST(), diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 29eff8a..b08108c 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1528,7 +1528,7 @@ static int vfio_user_io_dma_map(VFIOContainer *container, MemoryRegion *mr, * map->vaddr enters as a QEMU process address * make it either a file offset for mapped areas or 0 */ - if (fd != -1) { + if (fd != -1 && (container->proxy->flags & VFIO_PROXY_SECURE) == 0) { void *addr = (void *)(uintptr_t)map->vaddr; map->vaddr = qemu_ram_block_host_offset(mr->ram_block, addr); From patchwork Thu May 5 17:20:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8B7D6C433F5 for ; Thu, 5 May 2022 17:51:19 +0000 (UTC) Received: from localhost ([::1]:35944 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfdG-0005fO-D7 for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:51:18 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52832) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0z-0006BA-Vy for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:46 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:65496) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0u-0002Cx-Jf for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:44 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245GRcVq019339 for ; Thu, 5 May 2022 17:11:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=RPAQ/mjEoZa9PMW/bcbliJUWhyDPBG0vUiZuMhPC2uk=; b=fB9ivsz3Q2p8w8WdPesdvglBFQtZNbsKfvUWaQVqwXspsq+Csr9ClBwf5RpSE6NH2dKo nnJ7JS0oNzZTkQqY3g8Qnryp7WyjadQbx70CYsqFaCiTXyfBR6OAyV0pFFvF0P6sL7Go yv8wmsbYysVN0KMJARzaYT+se5MhYAP4xvoHT1qSDIVW1CtJPUa9CQ/0wiBsM1+F1+vB 8ad9ymxrcY20nFskY/leKs4S9r1BznIi22QoidzOX5pcabcHfleyerd0S09Fi/SdXgZ0 kQWrGMJNb9dlrnQ4cM+ughjTrXB8qI9HALBICfItZ/C7NNbBRuX8ggXn8mZ5Rbt6/aLu Mg== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3frwntc3kt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:28 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1bxn018989 for ; Thu, 5 May 2022 17:11:27 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2169.outbound.protection.outlook.com [104.47.55.169]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fusah2ed7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H5DyzldwYWzCnymDplBFFpjOTgYs6mZLpR4XqMbTptwXGZ/MRYw+CUA0XBiGghw6DyqBP6J4PA1mf3Tjq3VAA54ux256nyvf7ZTw30cqsISQmeD5GWRKr5CfCq//9cGIDWPVsnAQVICwE+ok6b7RlVmChahGXbnvZKd72Ocj7tB62rxLI2RLAKQ4Sdfb9IbOH2W4Mvi5zXpzml9aarR9NCjMt+4x2oLzOx9JzY+vjDA0Gr8E4M9iSnI0h5FHFJ58FD9HShVZ0JFN3NJKe3/nzLIWfY7yUSESGbljsrb5l9IPe6KGa3wHD3x7aEx0Nn0aQgPHHa9xbPQsGL2g1txClA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=RPAQ/mjEoZa9PMW/bcbliJUWhyDPBG0vUiZuMhPC2uk=; b=Hov07hyH8AUB+oJPPQ8DQzY55r5J8SCA8t8S/dzGKlqkwryhsGCrDVsD/4enbNoboRyRQdVMdPxHfURv/rqRbKTkK0y9Hriq3q6VXWcYSIa56dstO3DC1IadeoTjPbhsxlLIMripcVXm3gdhMNkm5siCzWkNwVCP9Puprpq3eXc8ddLGvrNr/DRlvFy3CMCAW785xJ/s7HJtPYkOv8MWNKTCNV6cAPUk72ivEVMAleSTgI58mcKq2D8iHF2aZ5kf2K96cpf77KNS6dk23kk3YHLCTpF/ENudfcUUKspSYyRSVXQ4g2RginFKgp5dPwJPMjAVaQynJrpVkwzxGs4PBg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=RPAQ/mjEoZa9PMW/bcbliJUWhyDPBG0vUiZuMhPC2uk=; b=qxVBR9LPI2g2woUvg+evUKSPJFOrcW4OBG9vUtBHFBIhJGgsT2wN99/KxeDSQLEuK1OIuo+Y2SwiOIxepJFUobt5OcueRfaDhpi8DUyQU1/YpBFo3kI8BIxmLol8FtASoBLhKEuSXEHXY2vpECb15eHP5TnSjCCPC5194S3Yf+I= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:26 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:26 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 19/23] vfio-user: dma read/write operations Date: Thu, 5 May 2022 10:20:02 -0700 Message-Id: <34ca04ac5e8c7dc8fe3f00fa254065a8d5025e27.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bb2d2f96-0f5d-44dd-d35d-08da2eba472b X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qocaT400Ui4w+1CXOPM9MskD9U5XBQoxw4jDA0USGE8lquZ7c+cohADZi/aKx8zr775rTS8qbgZwT3P7XhLGN5t7mzGT4NHKT3riJpK110q48Df+1hjvoNQdCa1dQErW1UqMGigXSOj0c4I1uzBDxPYI3Ab8hZHM5eSiao/PP3nOal5BkHe18DXSS3k4Mi1GejP0Ze3w70nr/vY0E3YmsXuj7QrgpcAGu6yilKoTV4wC1v4APEes8LjQUiqe9Rk6LsrapEdJVPShbzrS/zer/WwwsTzw04LZF1zmGkMYHBKWFjFbTBzSZxc8IIpwstOy5MB39zaOUkuK0xbYFUNrMMTEMPvGAFn47Ry6NE1uDF27iQrRQlgl8PmcwoThkv0itpN2OcFaMSCAccXu31ynZ5JQ0HORLEvdvQhgPIdx2FP9M0JwFs8/4K8pRBQIprCEqBNJBg4IAonGs8gCefUryVY9CH2abkPXBd+1cgrszmFvZr5cfo4dNWT1F3pvsSO3KwJ2CJIt7lRcG4uv+7o1I51weARnwD5OW6Fz+ZuvpWcifDPqrlcmf9kfCpxhOR81xO48DID2a3Wf8snnREEPg8KpcGNgrkKYqDHp6nqQlWYBl3+hp1UTchenFEax0OmQuSWPIPhNMDaemyfM0esVJHvyFmaRcvODHP1f5XYV7dKqOO3n7IS6oh26wv8tOsEj X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 3RJ8bxFNYqSRc0i9w0F5hiCBqvmUP63FajClNk2EJtqqhpY2PkD787efKBcZZCkFcqdnByYA32nzuntY0U54sLGrYo7bgl0TAtzts7q4lhgGSR1MVn9xBv7oOrywJrmeaZusG20TTX1o5G2YYBJV9tN9fs76G1yOK2pzpZUb+VtNUWHR9Rx4iMGldg/omBVVWvj0/LgrTvBjVCZYP/fOJ9NZcANbKtOU2IjZ5qjXTDgvYWpfgQ3XUAQPSCiH8+nBSG9RTJUwghBcOWvFeFAAoj4Skz4W7E7EeRRLJX6juluL+TT4ypD8EEZ9mQObpkOF1v6s+JOTkQqo98pmjvOnUFWfqBh5U1TbvI/WMzflBkEu/+Zv3cwrya8tgryVuaDTEbq++EUvy20y6/FeXbZqRIehPPGv4bH6wM94eMA1AaC0QfdC+gaYNL9VqV12rVqUW+geJuf4vxTkDOUz7MvZefV9dKwkBHCgmvkYFU3LVtxwzGZwf8CL1rW/8wPoMPm+CKufIr3GxbzpixfAxmpA0y6gm10h8D9KBvJ185BANg170lYaGJaSrfcr39CCa2cckBgsoGLrs5C2xeK48LlVmV1vkW8PtUG2bos2ea44G6LuYOvLZZAomjbPyBccjWhric8TLr4DoxEfT76nsODvPUO06h9nOy42c2Q4ARNmVskRtSrw5M61+30PpgIEvLxaUme/lLSAMrP0Q90UBZlxPE67klo+lqCmH177pS0rDi+sU28i6MB0SmOM0QyUxeMHMfgItWUxf1aMbAhhqYtLc5hGU8N5OF4EGBAkYjSqOAoHuxnfNMFAcaEEBMNj7HXY+Ki59QFu2pIUUbKgJRBxC/MosMbGZv9/jOLg2qHaUgVIO8f/aEKi7NW5ccyVi42j0o6CvvmtlQfFpiRxfKA2QMieXKPjARGDaWpyWcUXvwdWN5XPd+/xjFLKhieKoumoWCS0iDgmGkX+pLyagBLYEZxTFjLlRuhU3j2w3iqRBhP2W1ZUVOnX5EzeWStAJXfodg1f8oXwaUiuXmCxuaN/nXYLQm0IRk5KdfsP6fs1q6jdNeHpIPP9x/6kwWUDRssIJwuR2JSn/5nxoif5Isha73xDNTlUg2krl3+DnY9oEOHQLCyG5d2t/j5fCA4+O7uR8b3JXwUH4wYFEb1YoiBTKrobopRfnSqtLIoF6IkOlbtKFiT18//6tTvAL7pgi82okiNgaUedhgZg8dVBvJahIKFGMIO+yIatp+2PGFFEt0wMpYGzYr2uGfZ6uYiGKRx8CQXiAd0hhSq8XlrjzO9qeGRBGJvHiOG+tr7DvZbq6e1cu30IYw0EDsmS2K3AenuxCNENOmAid+qYVKxwXkjPJalRESzVJSdZIjhMsv3dLYHspEOvUagNuIPWNS9rQejVr3vLRJBS22pMundrRUf2hJIE9/ZVXS0elJo+QmtvN1V4GTUXc4eOdRoKFMxm6VTr02i/ubgp9kyNpUhuU/cUzMAuszWRvf2cud58x9NAQTKlgjgr+ExJDP1xOW24m6tsqdcUpfCkS+nv1hJQXTILgxzbKlj2UyRzRPd0ML4zm8y755i7FgFysv71EEbSh/XGlK08dAChV5KohQIW4YQjwo30macz7sdDM45Sz80z3ywncVtMB76RWd2qtyXddZ+wsqIxK6Z98oz0WuApOkTxmIetK0tcKMsWT+AlKVYH7lCLm72oaNA0iVYU4D68P0ghK7Z+XfJ047dOuwPGY1nq4QC+Fq0CS5sS5I/pj83e9io= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: bb2d2f96-0f5d-44dd-d35d-08da2eba472b X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:22.0601 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: m7Pj+DohPJrPTHS8Ipq5UBaAPA7+h4g/PwL8GJA8SjKa6HqOvM428ChNtMoKkDUka18qEmCm7kwsfobUMfJjg1QVNJTbmAOa7JpASQguwWk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: KZpsBg1iIaW6QCbezd78gsRuNGUwXPBo X-Proofpoint-GUID: KZpsBg1iIaW6QCbezd78gsRuNGUwXPBo Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Messages from server to client that peform device DMA. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 11 ++++++ hw/vfio/user.h | 4 ++ hw/vfio/pci.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/user.c | 60 ++++++++++++++++++++++++++++- 4 files changed, 174 insertions(+), 1 deletion(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index ad63f21..8932311 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -182,6 +182,17 @@ typedef struct { char data[]; } VFIOUserRegionRW; +/* + * VFIO_USER_DMA_READ + * VFIO_USER_DMA_WRITE + */ +typedef struct { + VFIOUserHdr hdr; + uint64_t offset; + uint32_t count; + char data[]; +} VFIOUserDMARW; + /*imported from struct vfio_bitmap */ typedef struct { uint64_t pgsize; diff --git a/hw/vfio/user.h b/hw/vfio/user.h index ec764d3..412c77a 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -82,9 +82,13 @@ typedef struct VFIOProxy { VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); +uint64_t vfio_user_max_xfer(void); void vfio_user_set_handler(VFIODevice *vbasedev, void (*handler)(void *opaque, VFIOUserMsg *msg), void *reqarg); +void vfio_user_send_reply(VFIOProxy *proxy, VFIOUserHdr *hdr, int size); +void vfio_user_send_error(VFIOProxy *proxy, VFIOUserHdr *hdr, int error); +void vfio_user_putfds(VFIOUserMsg *msg); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); extern VFIODevIO vfio_dev_io_sock; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 2faf890..25b3ebb 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3528,6 +3528,85 @@ static void vfio_user_msix_teardown(VFIOPCIDevice *vdev) vdev->msix->msix_regions = NULL; } +static void vfio_user_dma_read(VFIOPCIDevice *vdev, VFIOUserDMARW *msg) +{ + PCIDevice *pdev = &vdev->pdev; + VFIOProxy *proxy = vdev->vbasedev.proxy; + VFIOUserDMARW *res; + MemTxResult r; + size_t size; + + if (msg->hdr.size < sizeof(*msg)) { + vfio_user_send_error(proxy, &msg->hdr, EINVAL); + return; + } + if (msg->count > vfio_user_max_xfer()) { + vfio_user_send_error(proxy, &msg->hdr, E2BIG); + return; + } + + /* switch to our own message buffer */ + size = msg->count + sizeof(VFIOUserDMARW); + res = g_malloc0(size); + memcpy(res, msg, sizeof(*res)); + g_free(msg); + + r = pci_dma_read(pdev, res->offset, &res->data, res->count); + + switch (r) { + case MEMTX_OK: + if (res->hdr.flags & VFIO_USER_NO_REPLY) { + g_free(res); + return; + } + vfio_user_send_reply(proxy, &res->hdr, size); + break; + case MEMTX_ERROR: + vfio_user_send_error(proxy, &res->hdr, EFAULT); + break; + case MEMTX_DECODE_ERROR: + vfio_user_send_error(proxy, &res->hdr, ENODEV); + break; + } +} + +static void vfio_user_dma_write(VFIOPCIDevice *vdev, VFIOUserDMARW *msg) +{ + PCIDevice *pdev = &vdev->pdev; + VFIOProxy *proxy = vdev->vbasedev.proxy; + MemTxResult r; + + if (msg->hdr.size < sizeof(*msg)) { + vfio_user_send_error(proxy, &msg->hdr, EINVAL); + return; + } + /* make sure transfer count isn't larger than the message data */ + if (msg->count > msg->hdr.size - sizeof(*msg)) { + vfio_user_send_error(proxy, &msg->hdr, E2BIG); + return; + } + + r = pci_dma_write(pdev, msg->offset, &msg->data, msg->count); + + switch (r) { + case MEMTX_OK: + if ((msg->hdr.flags & VFIO_USER_NO_REPLY) == 0) { + vfio_user_send_reply(proxy, &msg->hdr, sizeof(msg->hdr)); + } else { + g_free(msg); + } + break; + case MEMTX_ERROR: + vfio_user_send_error(proxy, &msg->hdr, EFAULT); + break; + case MEMTX_DECODE_ERROR: + vfio_user_send_error(proxy, &msg->hdr, ENODEV); + break; + } + + return; +} + /* * Incoming request message callback. * @@ -3535,9 +3614,30 @@ static void vfio_user_msix_teardown(VFIOPCIDevice *vdev) */ static void vfio_user_pci_process_req(void *opaque, VFIOUserMsg *msg) { + VFIOPCIDevice *vdev = opaque; + VFIOUserHdr *hdr = msg->hdr; + + /* no incoming PCI requests pass FDs */ + if (msg->fds != NULL) { + vfio_user_send_error(vdev->vbasedev.proxy, hdr, EINVAL); + vfio_user_putfds(msg); + return; + } + switch (hdr->command) { + case VFIO_USER_DMA_READ: + vfio_user_dma_read(vdev, (VFIOUserDMARW *)hdr); + break; + case VFIO_USER_DMA_WRITE: + vfio_user_dma_write(vdev, (VFIOUserDMARW *)hdr); + break; + default: + error_printf("vfio_user_process_req unknown cmd %d\n", hdr->command); + vfio_user_send_error(vdev->vbasedev.proxy, hdr, ENOSYS); + } } + /* * Emulated devices don't use host hot reset */ diff --git a/hw/vfio/user.c b/hw/vfio/user.c index b08108c..1a0d002 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -77,6 +77,11 @@ static inline void vfio_user_set_error(VFIOUserHdr *hdr, uint32_t err) * Functions called by main, CPU, or iothread threads */ +uint64_t vfio_user_max_xfer(void) +{ + return max_xfer_size; +} + static void vfio_user_shutdown(VFIOProxy *proxy) { qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); @@ -377,7 +382,7 @@ static int vfio_user_recv_one(VFIOProxy *proxy) *msg->hdr = hdr; data = (char *)msg->hdr + sizeof(hdr); } else { - if (hdr.size > max_xfer_size) { + if (hdr.size > max_xfer_size + sizeof(VFIOUserDMARW)) { error_setg(&local_err, "vfio_user_recv request larger than max"); goto err; } @@ -780,6 +785,59 @@ static void vfio_user_wait_reqs(VFIOProxy *proxy) } } +/* + * Reply to an incoming request. + */ +void vfio_user_send_reply(VFIOProxy *proxy, VFIOUserHdr *hdr, int size) +{ + + if (size < sizeof(VFIOUserHdr)) { + error_printf("vfio_user_send_reply - size too small\n"); + g_free(hdr); + return; + } + + /* + * convert header to associated reply + */ + hdr->flags = VFIO_USER_REPLY; + hdr->size = size; + + vfio_user_send_async(proxy, hdr, NULL); +} + +/* + * Send an error reply to an incoming request. + */ +void vfio_user_send_error(VFIOProxy *proxy, VFIOUserHdr *hdr, int error) +{ + + /* + * convert header to associated reply + */ + hdr->flags = VFIO_USER_REPLY; + hdr->flags |= VFIO_USER_ERROR; + hdr->error_reply = error; + hdr->size = sizeof(*hdr); + + vfio_user_send_async(proxy, hdr, NULL); +} + +/* + * Close FDs erroneously received in an incoming request. + */ +void vfio_user_putfds(VFIOUserMsg *msg) +{ + VFIOUserFDs *fds = msg->fds; + int i; + + for (i = 0; i < fds->recv_fds; i++) { + close(fds->fds[i]); + } + g_free(fds); + msg->fds = NULL; +} + static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); From patchwork Thu May 5 17:20:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 15F35C433EF for ; Thu, 5 May 2022 17:53:46 +0000 (UTC) Received: from localhost ([::1]:43680 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmffd-0002Y5-1f for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:53:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52864) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf12-0006Ge-D5 for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:48 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:14226) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf10-0002Ha-Cc for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:48 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245FtOQI030616 for ; Thu, 5 May 2022 17:11:35 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=b7OdG0hmnODjgrUuFXmmyj24v2fvWXCjGM6zfw2PkaA=; b=g9gBFQSJeFUjF8MOzAEq1Ucgx2dKqAicKGloiakBtMfHt5oZQuuPn7BvcVH6lJmEVMtP kGvj2kxX/TlJxe3BFlj4lPTOzfQioSNcXBGlFq9+b3mISAH8mxxr8U/6cs8h0kFLembO yK/zlzUdB5Bk8DBCx/A3tjNKZZkgvJEvmDRFxVL67xZeoH6LHSqOQmPLi4Es565Uj8Hb 9fIg3lUzQoe7QIK76X3PX00RdzKSZiYqNRZ1I1RDKxZju7ldvLvoyfBgK2XJWUIL0/Rj JXCshu5Vlb7ox4iabXVhl36byTJc+U8UFKgQU3FBeTqtqgJt/B6iWDYzRcQGpVDbga+t +A== Received: from iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta03.appoci.oracle.com [130.35.103.27]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruq0m37g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:34 +0000 Received: from pps.filterd (iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1bxo018989 for ; Thu, 5 May 2022 17:11:27 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2169.outbound.protection.outlook.com [104.47.55.169]) by iadpaimrmta03.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fusah2ed7-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:27 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nUs3zl1ZO4DDVj/xN64x7KbUFE6oQjSYNXICPyD4NotTSjhR5cZ0pDyYnoa1/Rg15bR/9sLYG3ZSEe5ow4HmB+ELiwE73pVYocNXty8Xf4dkZREOaClSr/ySWfwGwhGAZF4LFDGa+8qOm+secFoIZJ6nzhnLPmdGTqg61fV+R/G7uPxNDaKrpBFpxDc5p/BHLkkiEnaO48DUmifSVFiZRO4BOxpvnxL8QDm93CCDgdR/uyq4EcnqWdEn2jSVUr6ixbEI2dDPz+6UtW2MfckoYeWVrAeDJF4UHqxVTsOWHWJWyJv8tlShwkc755EwteoSjEW8eD3oQXBKMNX+2lcLJg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=b7OdG0hmnODjgrUuFXmmyj24v2fvWXCjGM6zfw2PkaA=; b=h7E75MDo63mc+QBrsfS2Ni6Dct04TZDb4YCyBpskHz7ryzUfuqKX5KDhzNXk+Fn+TRG39bXNxhIeAT8xrmTbC9clfuyWiPid/44lMkLxt2r7cUuOmlfLKoHhFj6JDrAs3XUdnKJ52ExJR8cWdJa0eOzkPqEzIeO+MMzazUHn4cawZjczEeMSqF5Dx1Cvn9ou4Ak5dKuK0JVv2hgRMlxP31u5oZZ8gQHV1N59uEUQkYnMvc5nD0/kkJ5Kz5Aw5+CJIfTIMn0lZzCXWJ/AJNqPzUTWHJ86L/IbssOnYFg5vehAo7tQ/TREsd5pzio55Gq6jEM2+HlzcKXH4I2syJ7HsQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=b7OdG0hmnODjgrUuFXmmyj24v2fvWXCjGM6zfw2PkaA=; b=gW7wjK3lWBo0eFJBepnhX33IkdfMYMPH/hwU3mqjnnzCfp/NjfXK98i9w6Pb/P/Vbr+/twQF4ahxGkJFqGpjv29yg4DBkTNjo8Aa6fR/COSqgHRDE5OS33APzS4GinSbs5nkqcFGYlg/HfU04TkTJQxCZr6niyNFHzsIcE3sy0E= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:26 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:26 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 20/23] vfio-user: pci reset Date: Thu, 5 May 2022 10:20:03 -0700 Message-Id: <89e5b8641ab7781a7704056bf793e80cefe73b27.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c9296a9d-e499-4cd2-58bc-08da2eba4773 X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ymml3Wc6zx10l2HPNpRWQJ2KeZp5if9MOoNRmffptis6VJfGDhfzShSIV+TlsYppRAXJhi6OY4TCHPUhAtrEKJpeDJIR/4mjP/B4LxVsEMCHfexlAzePmHB/B1uPfDJ1W3uHdE6itrPGrVy1HeFECzRgDmyZQy8W1WgdPpKe+RPGQ7D7p5Db2/9TSaS3DtAazq6v6amdn9kZWocIUDJbGhHdbhvVD7IcP8TMYDwQZn+x02l77f3A8ca0Vz9fxMJ60FmtJguhhgRUiP9U5IU4pi9mipdisJdha/lWX0cT4PB4krflDs9nXPiyn876lpRdwJkkm13XULVx7dTrSzQVmIFri3QcHFpK258hltCKX2vxn6P2ElANOv8BUG4YLvZUamavh7KNxi23yPhMUQMnAIHKuae7iO+MCeYVymJPxgM1bZeeJeghmn2EqZkPYjaRIkymG/1LqciGaD6uAqDGTsdLPb7/0xZMfce+5ljceDMz8htBWGcMDdJKkt+hQRTJRUmDFio5Vb98UJ6BLox71UVZnCO7WsNdqJTeiZjekOvukjNlmy/pi+veq1OU9HWUJ9OqVPPLvQ81QNBCl6AcZyfEZ6RzDa5RVkHvGBn2zyBl+ML9xNBfuBCBDLWEyqyelFtm7isWg+/bu76Hz9ErhDoVBcs82KKq6VEAazTdo9uhpOcq/0L3iz0Stus7FXtrj+26Nj5eOxJyVvK33D0RsA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: umQuP0KUULS9VP1KhCTaQdRO6zlrIDcPIw6MmBvHG740vKEflCk/54hhaSu9YC9087GvPjkqU1XrYm3MO6vc1aCw5tbgh1RQZNeZNapfnot2w1HvF8aSJCHQpjWMgjSTLzYo/yTangOrpq8jQSwWavvagKIJEkY4+L+XK457ARBKHVqgHE3mDwsuTq5cH78b6hNv+cMRdsNLRXJ51t+ou/WAt+C6cfUzk+zmdDfwbLGT1ITB1jXV3zCgKECcAWWPPcac1OX5uNQdGF1cKxlteE839L9ReEwHyRthOJI1HMeWGmnaXK+BczQEfUeXkvMqKr1GumkpUSsuGo4pHoOPWlFLL9txm996J7SmKdi+uRC/8lLtWTdiiMCC5csCN7D78xo91Z7lLUSxnKnfVKkamKcRx3AkjVeyck9V3/OxzsGoJFpRa6mqQcXDb3EhgxPbeEOJxDlRV4dFEkLDrGhe5+lX3C49tmuFHA2BqfzSJzEDmitr8biII6TwS5792zo2mdccwoz69gMcE0zpAkMlVhmzxIfpagbtS98I3Sn10DKj0h5dj7ZQoIbqKQuV18bRnAD1U+aqrWz7dltIVHIj8fEaxYfk9rBfVXUFk4psqG59UDy/PtLcVfEBE1VMTfiq0IHNVpbnk3A8njq6Zxisf/zLerV8/ILOf5ij3c8iPOaJFqiUzAOUYwfmLI8qsehU5ttXkHvfqyVLBjQes+HIUBswu4RmofYtAFzIQ+JvTSXEDcQRYhWJIr4Mjg1qbWujqCZHZSGYMI8sSdIiRijHvOmV8QqMcPD40p2LYlb82jbOx6KLBuiHENizb7rl1vu+aNryVjcSn402RslAwPLAOqGU9RNCQcw9f9EcP3CoiJIK/SdxiFfi8LoS5GEc9swgVbBR8dlU2p372hrt5fvxgW94LYMw1UoDOY82WzNPJ9CyyY/7T1/1zmHfjde94JBl9waI95XA+9Zqlirpq7/smNgq5YMpiAObYTw4q8MHFjxFLVR96xZJoVvicHWUwyjqvk16i/RkXQapNbcqUhOrlEJUWLLEeLcJlxJJzrqkrNW8Cl3pwS2PqVgGWpzwKVVs/AzT9Tz/ndsYotQ6k4iWIH/hHv6kySCWwVAhnXpe0/ljS8aZLAZVITUcZ/ZNTtPU77zJJVKDbsVWW6vILoUvfpRaGHCxcktGoK2sb8E+Vy4SqO7MIJVg6nhdbNLnxAPfZBSA4vsyZshVHqm0ExfJRlJb4OHrEfYikO8q6jC99jSZsew7d4n4W67378yWdUaKOs7NLhp4jytYYbMveeNO/3BDWWXmbmnAW+SHgH/RyaTUg036ybRRcoIDQfiuSHNTDaohO3Mu8nkMdR+6/2GDrHg6Tr55KZSuuWky8XiQE2P8SdtCBFWRypoOTJ9M1BpMIGfB5aHNpbHif+jP3al+o2USKWVXiaLhCwEyVTUsTzpxoDw5isKAQKBlOxYnJ0eMDvT83wDj1HWpe2nVG8mLtJ3ewumsaq6KRB55nVQ//zBZJxU2srbWKu7OCczZ5sXl8vNvi/Df1G3m3EhIse2mwfgMxZ02A1hpqBQ6EjACQ3VZq+EcGQntJLyyYXlkT+YyUQUhbkYW5d/ITEt2voa7TesW6JBSqzvy+ab9ou4pFk4Ax+n9OWNY+uXnli/K59ew3mvwjG9Uz7+TSZ6vIx13Z2aKXA8s8vMSLSdJ5TqZB3hzECHwxQCmXXXIgJ9yM+aT586wQO0Vm7TdFhYVFDmFDniVT+ma5wz+0AFqY468ZG4= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c9296a9d-e499-4cd2-58bc-08da2eba4773 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:22.5132 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PIuqAlDkyZChl6wO1O6Fkdra/pLGkg70XLxkSdJBIAVLnCnXBz/aBrmNSN0qoUEqhh8HET6C4zMZrgMSenzFsYq4xE+bMmEl1Ky5aqvPzWs= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 adultscore=0 bulkscore=0 phishscore=0 mlxlogscore=999 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-ORIG-GUID: JFgDuEf0LSBZ4c_UEnKo_bB0ZNUtVXBZ X-Proofpoint-GUID: JFgDuEf0LSBZ4c_UEnKo_bB0ZNUtVXBZ Received-SPF: pass client-ip=205.220.165.32; envelope-from=john.g.johnson@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Message to tell the server to reset the device. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 1 + hw/vfio/pci.c | 15 +++++++++++++++ hw/vfio/user.c | 12 ++++++++++++ 3 files changed, 28 insertions(+) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 412c77a..902facf 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -90,6 +90,7 @@ void vfio_user_send_reply(VFIOProxy *proxy, VFIOUserHdr *hdr, int size); void vfio_user_send_error(VFIOProxy *proxy, VFIOUserHdr *hdr, int error); void vfio_user_putfds(VFIOUserMsg *msg); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); +void vfio_user_reset(VFIOProxy *proxy); extern VFIODevIO vfio_dev_io_sock; extern VFIOContIO vfio_cont_io_sock; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 25b3ebb..f4b4a30 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3821,6 +3821,20 @@ static void vfio_user_instance_finalize(Object *obj) } } +static void vfio_user_pci_reset(DeviceState *dev) +{ + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); + VFIODevice *vbasedev = &vdev->vbasedev; + + vfio_pci_pre_reset(vdev); + + if (vbasedev->reset_works) { + vfio_user_reset(vbasedev->proxy); + } + + vfio_pci_post_reset(vdev); +} + static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false), @@ -3834,6 +3848,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) DeviceClass *dc = DEVICE_CLASS(klass); PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + dc->reset = vfio_user_pci_reset; device_class_set_props(dc, vfio_user_pci_dev_properties); dc->desc = "VFIO over socket PCI device assignment"; pdc->realize = vfio_user_pci_realize; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 1a0d002..262d1a7 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1482,6 +1482,18 @@ static int vfio_user_region_write(VFIOProxy *proxy, uint8_t index, off_t offset, return ret; } +void vfio_user_reset(VFIOProxy *proxy) +{ + VFIOUserHdr msg; + + vfio_user_request_msg(&msg, VFIO_USER_DEVICE_RESET, sizeof(msg), 0); + + vfio_user_send_wait(proxy, &msg, NULL, 0, false); + if (msg.flags & VFIO_USER_ERROR) { + error_printf("reset reply error %d\n", msg.error_reply); + } +} + /* * Socket-based io_ops From patchwork Thu May 5 17:20:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9B0BCC433EF for ; Thu, 5 May 2022 17:36:50 +0000 (UTC) Received: from localhost ([::1]:45280 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfPF-00017f-J6 for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:36:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52710) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0q-00066t-BL for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:41 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:24178) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0k-0002Cz-JT for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:33 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245EeRAB026258 for ; Thu, 5 May 2022 17:11:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=tMhPJ6I8x9XnOpLitiRP4wsXVpLiDj5kcfeRdrChY00=; b=O2B0ohZsD55aEuDfDXrOXX5Yv/tkO14244U7iikFgWH413MqLY/QHik0Z7EFO0QeLUU3 5X/u58qZPX8ch5I/HSfcVknttPSV26VaAcpGtAnKCHBgWIJbbM2bzmqg2/g6Lbm9Fls1 ULD2zb/viJgrMuAfEwbQ4MqC044PNk5RmeHBxizhtcIXASScXm65rcbs86MJAaY5qXAd j4pPtnoyoWRy95oHlk+6Zs8us1DoJFpb7v8nMkCwYlyBFjoHn0fHzrGWDwc3Jovtm3S1 exlaQ9Tj77Jh6SXI0rPV/nDBMuZqHz/UjaB97MTANWv8HPM2aQxYANZzKi/LbLwlIysU ZA== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruhcbs6b-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:29 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H0ZjO006371 for ; Thu, 5 May 2022 17:11:28 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2172.outbound.protection.outlook.com [104.47.55.172]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com with ESMTP id 3fus8ybfxd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:28 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eEaYzRFruKn1av2RjHaimMEBB3D2sD98h0vyrHwGrGH/x1SJLfdZisSh4QqEW9Aga222HBQ19MyhMEAHiAJg4G9tci5+sEUDQ9N0D18BwgiEI0UmL9+SiJkzNI3eBrAGHffApoKlhrYm7x916o+d0carXbhmTVXFu5WqgMrRE/kQCzpOLeKhCWK0dmlvgJ9LnVy0Uu1zuho/Ye/z/2SwFn6priZWtd9u9QP7+gOQ6OYKVfypxykOIPjeYyRFcyTO8W0b9vO2HX7zd2gkFsrFpgpRIF9vflcf7Z1eRmcxKiAPxo2zSyo8le6khUTG+zq/b2z5saQzhOHu7TSLaG4Qtw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=tMhPJ6I8x9XnOpLitiRP4wsXVpLiDj5kcfeRdrChY00=; b=X0+BOpWLt+VnE8OQg/ma8FMLFel+MaCcr9b8c5IO/0/Adh2UJV8xzfq7oMEWEaGiNPip3q0SVBKytzq7Iq5ZAfH/MoaDhcmWcE79qBWfLTmu0eL4RICa+A9bU4jVu8lbdIptHS+qgfj442BkgUtEx43vzMdw9eeYxMkcVfNmywhHxD7ucNR90+T+RcYe/YnXSHAWW+XKqmNp9k5J/V2b4SmamLFwdWwhPxEyfFoxtnC1GxDOkzuZ3KGTuiw55M5kXv91GqbAqftgmUtjMdAyu6gGAxmDVmF79MWS6FtkfZ0Wa7qL3qi/u+dLxvXyHe1NrZ8jR6/0Odp30dxckKGA3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tMhPJ6I8x9XnOpLitiRP4wsXVpLiDj5kcfeRdrChY00=; b=lkYHriqCNNOhKILS/WLAZXmVqaBw82tubcjgxk3GysAG4U4Bf7XRKP2snm74hCGcrtZsDkkqyNWfngYWCNKpajXxVY21JnJxxgCfv3426nvgoIG5C+ST1UECIWClvzDxMxPdATP/YdtT9ngKSYpT3Aq5zNC6egODyiLhzJ4tJqo= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:27 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:27 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 21/23] vfio-user: add 'x-msg-timeout' option that specifies msg wait times Date: Thu, 5 May 2022 10:20:04 -0700 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 21ff9bd3-6d16-4cf0-54d5-08da2eba47b1 X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: D0BGvWfpMmR9oZ1sLD0Falxl4l7Oqb3oxalQqjNDVOcDeLyafEDFl4XwFeMwKOy5YM604k6Dgyh+dpV02v61KmH3uWEcsgLYATzg3ZUSYHHV9kSOZsvxi5GAvk5WU6XBZPFasFURC1drXNw5wbbS7RC3rRlpgqTHO0POh648/GEVji55l3YisriXe5VZGSgAu8jY7g4r6QE6roEz+qMT0oKVcZ9ckF7sLexNLP18yfp3bniW8KeOMUlhSHe8Xfteo5h1A50BavMZgRNXgLo8sfBDIxHHkUr9XOpuaKnblWXlkhTrk87AaGZVaUZiTwkK00Md6xVeTDoqjMFY/Isn5P2MH1uhT56RrBtXlWTwAgZ5R/WjLNFRnShPcQLY6b6I+jMK11WvqCXLQLHL8zDTkML0EF6s7JDeS4WsfyF5KUZoRvckH0wHlhLt3x2kYkZxZFcVkJXcWt9rZhEWAnpErV7EQ+4Xp1CKJSydB+Z1jL7sCfTqGKbPm3nZ+a6o+zIFDk69TcIyYcOHsrDu71I9Vqxd9C/tUriI8IsYyHYUKRUkJKpPMwcsr+dW6uJY28s3BHfvCy1HNJJvUlmnjjEJdsx+VpG7OvRms+QUXBdT+1dbgH2INiTtPxUNF76hZg2N5tWPSRltVAyfJqz2xvndsR2abdl1JQuPOqKLgZFp5Bt9qDK99jJxerw0/IzexUQkq4sKrEEVp2B4QJYZbmss5w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: HT4pwvf9mFkB3qk6Yp5cedF41ukEG/je+pUnf5SgkMtHcdLawhNOBpOfM6AsQAaxDU9yu1PJTU7deLi4tCBFV3BtzqVKiEHZ15pbgkaBt8Ou3LXgTmDhrhdFQfyplOIC6uBi6D7bR3Ez28/1mUmx0X+CJ2pbnVkEFTRLKxaGstjzEKEJFK3ofADKSfx4o5tRX1vkhao86cWpcwCy3cIu1h+KIAyVQ7eDB0rIMj7Dr4t8z1Z8MlhdqjxKBOX2FCF9NPjprLcvRZu58q0HiIorP3UwaFgoM5BWnMfNrEQup8VY/Ec9VRfJrzKKzphv6EhSbrYXcW/dyG3wTNM6/WGZ3y4tsHYtXppBCyU63dZcSjRD3dQ9cI6hgxp9Q2kJHZ27YQnS7Anjn66GMhb81ENt0u1OFPZiqdA25jJjgXJL8Sg4RybECN0LJGsECF6dBm/FYumtMbtcsKSL7Gxb2gUYqVWqFGTCAo6R/w2EVImyv+nQBAROhTf9v8dr2ruqWH2d67ZiCVG2E6I3KqgNsOBBG9YzG4M3RxoNyX0aV8ppEGfLc5uL1/IWW8M/xk2lO2Y/wV04ghed+siJdXtUMAd5b74aNAiEnCZPOVZ117jrQW6kN5tysrMVaGR/5vP51qFx9QpezSEjgClvHQLABsirMrTbUPYpChgOGGrdRgB80Nm++59qNccK9rKCvzcaNEEoEyy/RTnPbazjWQ3my6JtYowv5lPu4yTlcHaklAVCcUiikcnpNX+cBGCwlxyxKieO6PZM2jYCMmIJ+vZ+jgyo/CdOSf+0PZ02HZJ1JHSI1NrzBjvV9x2B39jdAJeN6CiYM4OAV9197K9iB/84T23sNJJJ5woa4l2JJwe9rHjlAXyn21YerwnwLxkcGKM2H23YHJ9hjKbBqlbmKddj1mIGVPsxXxyq4yAUq5I9cWqr0ovBXbG2HrUbF9sMs1y7ZnJIlVst0mHVYubJc+4ysM7wdeFbhzFIYXFtMocCnCXXxvy7rlYw55nWBV41haFQRV+s3jQJchnH0LwGF8X8jpwZOSSf6Nhi+Akzd0wYK93aw45zO5QridWCahM/zAgk0WCSUBrif0FNyNX79YwPJU0XbTqCTPLeOTzSBkGPGMORLHk1LOvM8JSflEEkn7sIHhzGW2N/C8OWlNM2subZXHmMSkYQPcRqKmVhzjQktrWJKXiOnFTEGaAl1QxgKRw+oBL87d0VmR0B8oKs61rHPNlc44hMGOBlYKNDGXEF5uiWf8dnuVKD91sWx8D18yRmeMRlK/ZmNVEsdTFCVIO1w29NkuvFwknqZRtzJK7O5A+rZlIEsHtcg+Ap1GPth0bj9DSldzPHkyBdlsGYUretsQJIFcPn7CdwWpS5wDqFSr1q54M/meG3CIE87VNudX7FSpQxjit5nVzy1j0DJ7r1kLG9lTZxzwy+JURxEj5TTzUIyVniHBtm3/OqSZvlTUDPIt6iSeQYJbVq8tdFbU9ZmjMtTxPwPkS2U2tZeGzpfDcrh0EpnDitaYMjwQ4y5nbnhws09FISu4IIZJjQhP2/Wm3EzWuNujonBJo2zCwq+tSvdfO5FBFK+JjAH4tE+lgYWOzZ9dVG0wsP742wHfQlKJZLyZQgd26Te+6HvAz7QR2VSvYZ8XVnIxjsG4R8tm/Zt+msAiOJZ4ZfoGA2oDobb2fWglQ3tVaPL4QWfav37IGyvruoMfdGO+KlAtaE/0Y59xy8AiKd1H3goVOezxDXp1crSsDsKZLe7sLO0TrnQE54M6c= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 21ff9bd3-6d16-4cf0-54d5-08da2eba47b1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:22.9195 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YuEHC3IlC9feuxhfSjPzTSJ3DIR7eeeOynQHhiazkmXJFBnfNxWaqORa9uDc4/W/jllRjXiBxMlrNee6yK+MmklN3V375eaSWeBhES2iGZ4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 suspectscore=0 mlxlogscore=999 adultscore=0 mlxscore=0 spamscore=0 malwarescore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: q77cTvFFryE1GREbpt2QaRSRxIG6EaSU X-Proofpoint-ORIG-GUID: q77cTvFFryE1GREbpt2QaRSRxIG6EaSU Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 1 + hw/vfio/user.h | 1 + hw/vfio/pci.c | 4 ++++ hw/vfio/user.c | 7 ++++--- 4 files changed, 10 insertions(+), 3 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index c207847..ca50858 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -197,6 +197,7 @@ struct VFIOUserPCIDevice { bool secure_dma; /* disable shared mem for DMA */ bool send_queued; /* all sends are queued */ bool no_post; /* all regions write are sync */ + uint32_t wait_time; /* timeout for message replies */ }; /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 902facf..18c6404 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -55,6 +55,7 @@ typedef struct VFIOProxy { void (*request)(void *opaque, VFIOUserMsg *msg); void *req_arg; int flags; + uint32_t wait_time; QemuCond close_cv; AioContext *ctx; QEMUBH *req_bh; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index f4b4a30..b103d08 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3698,6 +3698,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) if (udev->no_post) { proxy->flags |= VFIO_PROXY_NO_POST; } + if (udev->wait_time) { + proxy->wait_time = udev->wait_time; + } vfio_user_validate_version(vbasedev, &err); if (err != NULL) { @@ -3840,6 +3843,7 @@ static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false), DEFINE_PROP_BOOL("x-send-queued", VFIOUserPCIDevice, send_queued, false), DEFINE_PROP_BOOL("x-no-posted-writes", VFIOUserPCIDevice, no_post, false), + DEFINE_PROP_UINT32("x-msg-timeout", VFIOUserPCIDevice, wait_time, 0), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 262d1a7..ec2d89b 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -39,7 +39,7 @@ static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER; static uint64_t max_send_fds = VFIO_USER_DEF_MAX_FDS; -static int wait_time = 1000; /* wait 1 sec for replies */ +static uint32_t wait_time = 1000; /* wait 1 sec for replies */ static IOThread *vfio_user_iothread; static void vfio_user_shutdown(VFIOProxy *proxy); @@ -718,7 +718,7 @@ static void vfio_user_send_wait(VFIOProxy *proxy, VFIOUserHdr *hdr, if (ret == 0) { while (!msg->complete) { - if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, proxy->wait_time)) { QTAILQ_REMOVE(&proxy->pending, msg, next); vfio_user_set_error(hdr, ETIMEDOUT); break; @@ -757,7 +757,7 @@ static void vfio_user_wait_reqs(VFIOProxy *proxy) msg = proxy->last_nowait; msg->type = VFIO_MSG_WAIT; while (!msg->complete) { - if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, wait_time)) { + if (!qemu_cond_timedwait(&msg->cv, &proxy->lock, proxy->wait_time)) { QTAILQ_REMOVE(&proxy->pending, msg, next); error_printf("vfio_wait_reqs - timed out\n"); break; @@ -867,6 +867,7 @@ VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) proxy->ioc = ioc; proxy->flags = VFIO_PROXY_CLIENT; proxy->state = VFIO_PROXY_CONNECTED; + proxy->wait_time = wait_time; qemu_mutex_init(&proxy->lock); qemu_cond_init(&proxy->close_cv); From patchwork Thu May 5 17:20:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6C74FC433F5 for ; Thu, 5 May 2022 17:44:33 +0000 (UTC) Received: from localhost ([::1]:54182 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfWi-0007JU-CB for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:44:32 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52744) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0t-00069B-NI for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:45 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:28874) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0m-0002Dh-Bz for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:36 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245G9xWB027626 for ; Thu, 5 May 2022 17:11:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=LSoWM2+bLwAdmEE//LrstuXMLj/duRf80GNLCbTNwIk=; b=J8DPCXZ3c1e+AlTBf18lZPzNGSFwhCA+grlw3NFuBmT1XZdSGAfWZxPMs0y2yKPGNIMf h+eNS45UAzLzAT4PL+1pgn+y6+ONP+VFjqn5oZHPB3fp3XxMAFZii/JNcD7/QXPEJp42 oG8P5/qSj6L9HDLfZnIRzAA8XxjU3crXTEqfp6RGUp7B688FiwrrndbE6UhE2vjej0zp As2jYzAk+HfLPK1F9snBAxykx/pxZngyk8ZmKOFAwikgS0PLBrsgDHXX2mJSAqUZWodv 6kfVDVKmybadsPxuYz8b/FwaT9PgJ8eQoMG8vDrBAJeF9FUxugQYr2dU+IhUo2kO7T8Z 5A== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruw2m0nc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:31 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1O7l001931 for ; Thu, 5 May 2022 17:11:30 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2172.outbound.protection.outlook.com [104.47.55.172]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a7amx9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TkdDw+5onwuQHfutP+VHdQ30LqDxF2EN+IRlfK9t/mS4YEXeP16KNf1GKeqYcwcoVSV0PD4yjjSXKSsHWPNax6qj4tNZ7jlXgxjyA9s2EWOJNef4qdkcARKelxFREn7rKX8DXVFfwo3aKlKZZamlOG/1rEdfeD4yRsyip23MlonpvZEx9KLA2gwA65vhgvzHKG0gaMsEYnUKJre9W07sbcFHq0RQByi/V3Oim06J6RlnhY/N6v5mzjpZPHpiXHwaVExfw16k0KKz7c2xo5D7xXlA7ZLVLUONgFEybaAntB5CuV4ENVoVfOMhpumyUUEqK/f0Jgg/4uLdi583hL/mJw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=LSoWM2+bLwAdmEE//LrstuXMLj/duRf80GNLCbTNwIk=; b=L8Ntz+Z/ItSTyCRl1ujbG+ojIcrlfgYUN+sGyhRvUykvRLbMmfDUR2RKzNxEfu4++TcrhIj1KwPBtXDVzvu7ZokKa5P6tcC/uP3vlsLnuuf+WRmbLJQg0D+N9gsvalYoQv3CbkeilUmciRCC+8Ia8k41QERdWXZX1Dhpny4Oxgmln+IzTS1XQw7n01PYa/HK/9ESdri7RT96hRgyn1UKn0LpSf3CuvqJDR+Dn70zGJGxOdt9V/BdUBNOh/7dt+Xe17FAlcbXZSASNGv9BcCWOuQYq2S+9a25T6sCjhNg67SnWNbjYZ7d9LE9NXJgQD8pglINRcMwCKi1xJZ4cN0Lzw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LSoWM2+bLwAdmEE//LrstuXMLj/duRf80GNLCbTNwIk=; b=sHvcaXqjdEpEVSLBUhXGIVQtdnmcb1IctxfapV1GEXuDuD20ZbEWOBCAUOxEsLpOYSFEy5YdaSZYk7eVcohccm1FZ8LaI+gSJCajj1n44TEesjgZcWQWpupMEGl+pPg76RGwviewaCg9Z62lqio7PBK0puGt5KHhPvfpcIlKcgI= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:28 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:28 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 22/23] vfio-user: add tracing to send/recv paths Date: Thu, 5 May 2022 10:20:05 -0700 Message-Id: <115153bd4c41c4b2e2d5b046e76fc0f06ea3bf78.1651709440.git.john.g.johnson@oracle.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 37260f08-27c2-4719-8c74-08da2eba47ea X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Y6GnIn2+ELi+U4G+LAG4v3qbc8QOEvS1cSIde7aamOWBIhp9IO3YR4BmG+9g9du8BpPKgV1iSuIidEk7zLd67Dj6DTetM45mr5r7PlmifznxzzT53DZGvBxxr/BIGAjbKHm6/Y5GzZxPWKJDQZRnheOdDRQ05Rr54gW6fbZg2sUNv9kCADQA1gVEa/c8URyvweJDRaJvHq/0l0fAKzD9jicJS3gxbTYL4EGtwKoM0pj8eu/Jlr9Q7yXE+loTvzpAiHUzGUj37yM/Ks9vcTjB1N7H+MMKZMGikvx6jfSEE0zEtmjK2vqcMai5NSgvOUKL0aL/gPdpNPCTSlt718s2/W2YfjNnzQIvEi8rlCGEFXdyQTNTApb9TaRGVgQG6TwpzLNWz/rnZ2WELt+/hm1N3HDjTvpHWLECD19RBTMu3tFvkC6JGH7BgUwewT3MKDp/W/t9/ab3ckHVhguegiLTYUlJFZe/pr4cP5/b3A5NwMg2FccSrrmc3L/hOFMV8aFTz2etCmrKgPffctK3Man+dFYCp+NII1JGyAKb0RIXTR8KsKOewlsG/40ZQTZ3GuUfn4yGmwWAPlmhEKIH2tEkGTiE1vFaxGrxDDHQ2OFVxC0THRTtUVn5F2xtVEBD5WXWhf4TVw0HJcZjQ1fk/yAAJiCjqD5pzo57fKq9IJt81UQTUKex8R8VME6Xwq0tQP3A0eLRTNYKrqZdFyF5vGqy7Q== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(83380400001)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 08ne6mxm/6Tc6mtUkrDGENaELbzEDTTV6OB1LRCyxUhX0sxsDcm86GpLc2BgaOsiaJd1lohuhdcXSVLFq8Oj69U0NUGI5E7BA/tyGSCM5SSe4/j8Uwr8UpVMLWc/5XThm02Y387WorDVuH2GsuRsG7HeUeGvy4BsGMOty1Moi6O3jDeiNvdoreEfDFbkX9ICThuakSOHHvL0nn0UEqPGFk+DHl6wQaiZ4NXDk9LjxIrOQM5g+biVX9kIAAEMmQ5XZZwrM0SGoxn0r4Ec/M5lPfKvuhuTlT2msniG/8skQtjQn1tLVaRQ5fUGVRN3nwOcRa32iY92LDuPylD/YFscqiWcNjoGy6SCkwYHwbznG9m1K2QhU+pmW70wU+xJ6QDV+jUkaaKAlz86njsWVbxdohj/kH4C23AfAozxasryZzFeO81C0GxoNN2ir6vSCoronT1GCDi1nu16VZJIdMwPmqNJsMdS3JZhUCpXUqfhlUCpI1asCxI+S3ZCF7x7cfYVTxbx/8R4SCBJa0W5LHCaBvwvpKuhKzerNXVHa7rLVRvXZA3ICL+T8zdqyFLfot3rOgik4wRuR8koxLYwtcwAaMMraYsLu7wVIYmUYGv4Y6rZPfmp9YWZCVz/T6LO6v+VynOBJK7md06d1snG6Nsr9WEJnjghtweNhhjBQDKfEodxhj+JXl+ShqJw5MVysLgBG31ilQmu4GeBARljGHJjmHKjpw3Hss/KWafxLcELSc6LbGSljcIjJC63fsANqp9NBiK0+6bEiu+umwjpPfz9Rw//MTA7sm6oqmbba8wT1A3Pw4IZUP/BC6rXAyNba4H3SEblF/MmFhkHYzpQerEtdGcHWojIjcffAJoWRuYTWIbLM2n0oOntPG7d6fb46QDxxI3PCucmYRxp6llH6tDFt65UZ0I0pX7qkCDT36rHahQP0Ayljdl3ecPivt15D/YDCKet6rwyV2iiV8rsN0ZBDDcVOaLb57cv5RB7RbL1XFkitS/G706dRVNGeMavUMbFP2i7XvAvP81ok64X+4fDssNrHLy28DuM4/2IZ8ns8vhJevwln++TxV+nlJPy1/YakjPDjbIzBUh1oh3b9VXA6avIwiuF6UEzervi+kRSYf4jsOCIPCw1jk6OCd1DOmTcjih5OUPRTd7kI2+8p3uI2Gur77baazcFcL/jxCdq3LswtAYcV63h8Co58gCnSHtII84LzIiAFJTg761ZZACxaQCXvjvLVQ9LxpFUKXzv7g1uziQsCgbn7r3s1q8jl1yF87SVjdANKjAF5Y8kNRr/F6VLqp6TqTq213TkfRBgoLSkijQ6yNfEeA6FqmufziEeOtJKrlqCPQZ0KKb8VObZV0UKYRkfOh3Ha9nGaYC3cZxknC33ISmr0IVB/P2hnY+GBP/RiiSKvLE9W6Enl2oB9rcPRxCZpXxlWgd6VaTR5yjVVD/LwQ3qJxHLSRs6sbrk52buJ3sMwTCSZHqHDAlTve7hLmfTZnS6fv3VREMB5duJL2qWdjtIMFJXskUlDL96MzXjfIbk41ZNsD6xehxN92qrA7uNSjw9GEKy7fbFX++pTPvQRzReQsmiWqkBQm8vCpenLB4NFbCXzQwR6NWZifng/CHYJ/UeMR5z3EK8nsJPula2NYgcML81p9NSdrMokqlTkZaLvK/mHs38PnCldH//HwT67lixrA0ZWbNJ+cSTvlzszoO7WmxBNg+9tUH5zEMS3rVQGKTLQZaDueNKN9scqiAQJ1msbNGbWHsyu2c= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 37260f08-27c2-4719-8c74-08da2eba47ea X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:23.2944 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GCVdYrxJdbzpcpNZfpTc8rKHcAT0BIfetgSu5kKr4OvujkBnPi8E6lQK6ViId7agWNePPGDEYySlBa+OmVKUjMN/L6Af5ibmRok3s5CK3SY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: kOoTNEWmFFzVcNTV5LLXYuoj3fT7KEfN X-Proofpoint-ORIG-GUID: kOoTNEWmFFzVcNTV5LLXYuoj3fT7KEfN Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user.c | 8 ++++++++ hw/vfio/trace-events | 5 +++++ 2 files changed, 13 insertions(+) diff --git a/hw/vfio/user.c b/hw/vfio/user.c index ec2d89b..a3e4dc8 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -29,6 +29,8 @@ #include "qapi/qmp/qstring.h" #include "qapi/qmp/qnum.h" #include "user.h" +#include "trace.h" + /* * These are to defend against a malign server trying @@ -111,6 +113,8 @@ static int vfio_user_send_qio(VFIOProxy *proxy, VFIOUserMsg *msg) vfio_user_shutdown(proxy); error_report_err(local_err); } + trace_vfio_user_send_write(msg->hdr->id, ret); + return ret; } @@ -227,6 +231,7 @@ static int vfio_user_complete(VFIOProxy *proxy, Error **errp) } return ret; } + trace_vfio_user_recv_read(msg->hdr->id, ret); msgleft -= ret; data += ret; @@ -334,6 +339,8 @@ static int vfio_user_recv_one(VFIOProxy *proxy) error_setg(&local_err, "unknown message type"); goto fatal; } + trace_vfio_user_recv_hdr(proxy->sockname, hdr.id, hdr.command, hdr.size, + hdr.flags); /* * For replies, find the matching pending request. @@ -410,6 +417,7 @@ static int vfio_user_recv_one(VFIOProxy *proxy) if (ret <= 0) { goto fatal; } + trace_vfio_user_recv_read(hdr.id, ret); msgleft -= ret; data += ret; diff --git a/hw/vfio/trace-events b/hw/vfio/trace-events index 0ef1b5f..ea4bd7e 100644 --- a/hw/vfio/trace-events +++ b/hw/vfio/trace-events @@ -165,3 +165,8 @@ vfio_load_state_device_data(const char *name, uint64_t data_offset, uint64_t dat vfio_load_cleanup(const char *name) " (%s)" vfio_get_dirty_bitmap(int fd, uint64_t iova, uint64_t size, uint64_t bitmap_size, uint64_t start) "container fd=%d, iova=0x%"PRIx64" size= 0x%"PRIx64" bitmap_size=0x%"PRIx64" start=0x%"PRIx64 vfio_iommu_map_dirty_notify(uint64_t iova_start, uint64_t iova_end) "iommu dirty @ 0x%"PRIx64" - 0x%"PRIx64 + +# user.c +vfio_user_recv_hdr(const char *name, uint16_t id, uint16_t cmd, uint32_t size, uint32_t flags) " (%s) id %x cmd %x size %x flags %x" +vfio_user_recv_read(uint16_t id, int read) " id %x read %x" +vfio_user_send_write(uint16_t id, int wrote) " id %x wrote %x" From patchwork Thu May 5 17:20:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Johnson X-Patchwork-Id: 12839838 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E4A96C433F5 for ; Thu, 5 May 2022 17:45:53 +0000 (UTC) Received: from localhost ([::1]:55524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nmfY0-0008CQ-R7 for qemu-devel@archiver.kernel.org; Thu, 05 May 2022 13:45:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:52746) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0u-00069W-Vh for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:45 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:29488) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nmf0n-0002Er-1f for qemu-devel@nongnu.org; Thu, 05 May 2022 13:11:37 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 245EeRAD026258 for ; Thu, 5 May 2022 17:11:31 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2021-07-09; bh=8ze8WgeNzH9Ve9OQvNlMe8Qo7pia3EkUB2Oi+2M6pEA=; b=J6gLghHlTt74WAwo/+oj7ZU3s65y6CE3O53oFdiTANZ0maqPguEYCMFegiFz0W//GLI5 9PWRh9/MjML9z41nb228ZZIs/Rw4m+QliWMBZAuau3ucWU4LFyCrsZyndzpffdhMo1ur gdUALxF/Uecj9+1CZzf7X1R5LYU/7sO3zXF7LYOS6dIhi9embvVtMWE/cwnF59D9dyNt A97BTU89/PA0NTEJbzEbakYUnkepDI+aJbm3CFG7J1zDE+pP1VYYQ9TaXZyGJ81LYtOx br8wywxTWgRDBYE0tyJ2HSapHFeqiafqdp50oWDS1qFkvjJbahlUeids08F0g+PQZPnV 3g== Received: from phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (phxpaimrmta02.appoci.oracle.com [147.154.114.232]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3fruhcbs6d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:31 +0000 Received: from pps.filterd (phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com [127.0.0.1]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com (8.16.1.2/8.16.1.2) with SMTP id 245H1O7m001931 for ; Thu, 5 May 2022 17:11:30 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2172.outbound.protection.outlook.com [104.47.55.172]) by phxpaimrmta02.imrmtpd1.prodappphxaev1.oraclevcn.com with ESMTP id 3fs1a7amx9-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Thu, 05 May 2022 17:11:30 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PRfGfMjaOVKKwMvyCIWHtMjHnY6jg83fOlYzhkKMGf81dk/QFxEw/7Iu0uXCUJa8DKw9K8/8m/fHa5Ey+CQYbASw9//uvx1BhROtkfLEiDfWsObONZlVR+YFcDrQE6/lGAZ3XrvQTLfi41tzes8VvYbyNOtjfijdQRyQkMYAHm5YvU6Q+JRjsHURg6xF5l5iYSl86G/tD8o7/FbCvoUrM5aPKPXa9tRwVAQeJ3BGCtilXgWgBffx8NQI6iMnsp6kOZ/EZifaKe/uaqoD/AU26Tq8yq6oTbEowQ2aHJtOW+ECFDR+lswlM1j8uXz+IrFt4GHu6+8+wX5PxQusad4Fhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=8ze8WgeNzH9Ve9OQvNlMe8Qo7pia3EkUB2Oi+2M6pEA=; b=KLVbm5vnE85kbD3LUbYyXf2nJWiVC0/xc7GaVUTu2Ot33NVpHACzJnRsY9fd6KhZM4vCFBZ7n4o/CS9lyyEo9Jdt8XSsl+1uqtQl4RERWtrDHOCU2HuMGCMnBvS2nWMRJMvclw1PDQIMBww+/M/ttP9ZT6lU5jQQStjERMaTw8Obdl0+d37LgrlfrMZxr6fiq++p7l75TsYOxFivB7zwIk2+PTheR41AdFNdGM939MWtfQw9ueFU40LAEbGcaS1bPqKNPVTVEjNxNIL2eUQQtAD3MC+/EoiBJ7HSR8A4jgaL5C5wljZMF0HKAM/qu2FS7tqC2G30Z/7bk+PGBQq8uA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8ze8WgeNzH9Ve9OQvNlMe8Qo7pia3EkUB2Oi+2M6pEA=; b=AmDgfeegvIxdGvqnsocN5r20ks5ZK2+uzqF2U/SWekvubty3CsNeE2DQ9knlY57JaLAdBSfSLBGKPJTQFUpfj8veCc0a4WJ2rsgDEoAT6fmK8GOrAUzbaEMumY4rbg27gz0I+QvhptH5fHnQUBNqplJpSILpUs5cvY/SiK24/Aw= Received: from BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) by BN7PR10MB2689.namprd10.prod.outlook.com (2603:10b6:406:c3::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5206.27; Thu, 5 May 2022 17:11:28 +0000 Received: from BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05]) by BYAPR10MB3255.namprd10.prod.outlook.com ([fe80::cce6:1c15:c6bd:dd05%7]) with mapi id 15.20.5206.027; Thu, 5 May 2022 17:11:28 +0000 From: John Johnson To: qemu-devel@nongnu.org Subject: [RFC v5 23/23] vfio-user: add dirty_bitmap stub until it support migration Date: Thu, 5 May 2022 10:20:06 -0700 Message-Id: X-Mailer: git-send-email 1.8.3.1 In-Reply-To: References: X-ClientProxiedBy: BYAPR06CA0048.namprd06.prod.outlook.com (2603:10b6:a03:14b::25) To BYAPR10MB3255.namprd10.prod.outlook.com (2603:10b6:a03:156::22) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e333a69a-8f5c-4cab-c3d5-08da2eba4823 X-MS-TrafficTypeDiagnostic: BN7PR10MB2689:EE_ X-Microsoft-Antispam-PRVS: X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MFtBX7sawlYFIonOjBDiQ5VfxGk9ULTYpH4vitIAKYVTm4/csRAgDFKf6sZ4V3aOolo87HhPJBx/RiajlVotRqr13D2vVZUQ2CAf9czlHQl/1E59ST2Tu6mnREb1vD/MUXq/Hpa5ekZHY+YJ2iFAfaauUBiLwZxWQYstUbGI0pycI4kQxVD8iptKFb0+qz1kero/BwPpCO12P/Reb3pKDwnzXltgAUfbAY+CE5zvrAIsTB6CRn99jNd7e0iAjpm+8x8onwBC7UxvOeEe8Yo2oSCSwL6KJowGiD1JpS57iDwIu7eUpZ+XPFLmGQ3F8PG7cbagrjMawTNwqHfo+LEmJPIS8u4xgyFmbAA1pSLKBEDGuuV+d/1PdHOG3k8FX+joxswqIRNyvZUieLQ+JL9bjYw3VCHHG+vnVi6mxXV/vl+5BGGoraa1zMcUpB0iPBGjfBixD8H4fbvN5RyIJsyjpaREDrYnK3jSmg+oeRUS+UqafCjy1cZJW68H7KBmuYKhfBt+2krx3bv+B6OwFlzPxZZO8HvJzqWiAlrws5nXhGymJ/z4ZypYfpeRsFSqQVi6RzZTDPI0e3K/zXaYzVtHke5jFo8TFOFnX/ihlMI4sDeBUDwkFnV/vJHm1fBos1j0NCcC/RNGnwuDUrh21blR5FSfdD7fdDwaVXY8irw/6BqV6KAyAdbz0JmFpaDDtmot3qyGm8inZJRzJ+VR4+GkUcMMYRY8ThnhXg2AOz1nkto= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB3255.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(13230001)(366004)(38100700002)(6916009)(6506007)(6512007)(38350700002)(316002)(36756003)(2906002)(2616005)(26005)(66476007)(8676002)(86362001)(186003)(66556008)(66946007)(6666004)(52116002)(6486002)(5660300002)(8936002)(508600001)(14143004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: QLdsGLWtGuEpzYB3cHBRdQC21+3aoZHTTbeCdNRqzcUMsniYUEQ81N6I8ajddCVJmFyN/kq+zaqcuzrzxGRDtPBtRxvXv7zxEwMuOTNoH3HGykISA4m/qeYdirfsGBgCLSNq1mere/6f5UEaDVc/RpGJcOAyLiE+K2BRTFbDFbyk+1upDnpdrJgQP2iMwJUZ9Ncoj5AaXZzwQIEe+WcsRVdlbgiSEaIJFaBKs0xzHuM1M4jvorJpPDr9jloLOHRucyhiT5DetQH/9vOdZDIlXRMy5n6D8i6xLD/h2tVT4m7HPCTTQpMS+kJZKx2UNZdHGo2FPNeXnFgxhFieDIW9YkwFmz+YbtQRHc4vKKLmNgxnxB7HRKih3iW8va0tHaQVFWTDHFUVyEE5gPwJa53K9TkaD9pXF4ByfCrngCQkCrqmJpS4A0+fOjkuRu7imAkUCtRnFf1qGmIENcky6ZKFTrpjlAIgeBSX7V8wTm1wGBzBwNXoDhIDJ05Xz5Vfo3fMsrxP7irCipumhGEVZVUl4ymni8o4d/JBoMEDaBvVq8bL/uIaXDg8huVca907orQuBh9ErcxS/zKqtiQjv0/9T+ofFR7CM95x2oWmgyM9XDnq0JYNxx9M9/2Q6ygc1I0RQYO53tOCN/9+9IW9jz0aYzqZcSS0EfkuFVUQQhao6y8h88PlsQJy966excTFi89t5KCValMnQSWL/Gd+l361Z84Emntzgq6MabDsmMuWjDljGcGZcx8cgx4o3rt3wRSHu8FWvne4PyPCkppx5natyFC/dHkrLXr7lfKWdHVOCSZbsjBphaRyEKb876db06oQ5kl01cmWfgfoykRI8rI3JE9rLFL7wRXJJN8agnFR6YZrV8r0/gFreulNClbXXRRAdsoPVD1VMCRhbL79t5rKxTLIr4UXCtCfgM5QK5KrQ0UUYq8GI01fuH0moPPAMIWdidxOgaFrDC9v52u7VgiLYziVo44o6pHWXaeizglUqRERG5O7IBbJlicZXBl/df/SCu4WFKceGUqUNGVgoyJ4KPQjSuA5J1i60ndaxOBaRWrvs1xhLfkkNinmmjd26RACKqAMkrg0HT+s4xkQb5EaGemsJr5U4w+fJYIm2MxnXepDthk6wg/adPWT3QqbdMoa/k2kZU1UJNChzHghG6lHXqql+tmEPeVBgb/XQfZdzh46vb6IVzd/FUtuFWczgxIHJ4icqKjsUanLA8HLjc30MQoEa8vbaqyFtuEUvrZlasuYn7pt7TpvnC25M+7FYCznSd/zRSrzfm5ZfZEYhPg8lvHzq0xT28wN7BI6KMzCxkfmYJPMqO8KpDb8e5/z7drba9YABfP4B4chSRQiWw7K0djthA0g2IY27mS/rS4xoOAzgp3srPCsHHRCOMgWyQ7yINZYgSdbYg1JXbY01Dy/s3zEB/3obsYG+misr1VJ894M+s9i6L243WbpS7BfKAWbUlgGCTYecgDgOhubMTUSO3VBuF4GqzVesWy5UV+isRJQRFkM3CA3KDk/ZNo3bf8c1RQdEw9CEmynC97/hE08Sl7XS6ZXE0i0qjWMtFr3djjs/Ws7Wl/Wsm/1du8i+N0I+6c4G9kSMBEI7q+oUFc04KsldB3vpt696RbX9d8BpXgX9tYlSfR1YA7Zsg6O8CSGkbN7W1370Rrkr1/wNfne28tr7gG3axBrDsQZzLRppUQC7F9ZLUdjPFdAAHfvrRkfz8oyLhXIOtLM2K0Rn072P+LyVF6JwKgYRwwCDO3F56M= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: e333a69a-8f5c-4cab-c3d5-08da2eba4823 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3255.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 May 2022 17:11:23.6694 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RE3pA+nb+niCo3lmJxGUHoRPAZf1CJ7bn9Ir2STYwzqRhTgRTOh+sKJP0qzBN9S11gfS3zf7cBQ+DcsiQQcwbWamM8pdS3qU2tF4foo6aPE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR10MB2689 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.486, 18.0.858 definitions=2022-05-05_06:2022-05-05, 2022-05-05 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 mlxscore=0 suspectscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2202240000 definitions=main-2205050118 X-Proofpoint-GUID: Y3yWEoJYKCnR1XWP_vVlMk4eXmwj5vvH X-Proofpoint-ORIG-GUID: Y3yWEoJYKCnR1XWP_vVlMk4eXmwj5vvH Received-SPF: pass client-ip=205.220.177.32; envelope-from=john.g.johnson@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/hw/vfio/user.c b/hw/vfio/user.c index a3e4dc8..eb79785 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1626,6 +1626,15 @@ static int vfio_user_io_dma_unmap(VFIOContainer *container, container->async_ops); } +static int vfio_user_io_dirty_bitmap(VFIOContainer *container, + struct vfio_iommu_type1_dirty_bitmap *bitmap, + struct vfio_iommu_type1_dirty_bitmap_get *range) +{ + + /* vfio-user doesn't support migration */ + return -EINVAL; +} + static void vfio_user_io_wait_commit(VFIOContainer *container) { vfio_user_wait_reqs(container->proxy); @@ -1634,5 +1643,6 @@ static void vfio_user_io_wait_commit(VFIOContainer *container) VFIOContIO vfio_cont_io_sock = { .dma_map = vfio_user_io_dma_map, .dma_unmap = vfio_user_io_dma_unmap, + .dirty_bitmap = vfio_user_io_dirty_bitmap, .wait_commit = vfio_user_io_wait_commit, };