From patchwork Mon Jul 19 06:27:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,WEIRD_QUOTING autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04837C636C9 for ; Mon, 19 Jul 2021 06:37:54 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 48F2461166 for ; Mon, 19 Jul 2021 06:37:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 48F2461166 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:55890 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5MuW-0007rs-Fo for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:37:52 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33448) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlH-00049I-A9 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:19 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:28942) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlA-0002EN-Jx for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:19 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6HiHS031108; Mon, 19 Jul 2021 06:28:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=Nl3/1mkfwIUlsl+43JbQyxcQkekx1Ug7U/YnW8tRucc=; b=mey3uhM4UoVWqFXPl8EqriOdNhEnZs9kC7YtYuqZlyAjCOyp2DoNS64/xeodPL6/vtME B02nI3zipCND/zPEXpNLhmtJoMMz1uOe7ztn5AB4XNGxrMsMF0GeoHr/co5/sjxl5no4 8OD0lL+c7fFMvAMagoLWe0Hz9rrr94Wwt9WEZKnWXYN4y7rrE6ExI/6XFTRSl8h0n7CG 4NFRMRXmhbQa55uPrDvTyJXNJ2Y9NCekd2pemQyeTa2dMbHmLt2XKKIdnHaKPe+hQHGt cGSsoIyfLPSwS2w6L43I5OnsOSHZ+w1fTB2yjWb53kFa/FgLwhYlnwCCZAxaoMgB+07N nQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=Nl3/1mkfwIUlsl+43JbQyxcQkekx1Ug7U/YnW8tRucc=; b=stCGKpRQwVfESG2gL9nWBGHJlvqv1aNyjPXXx3xbuvaTVdAQrg/WH5mtRs/xI0xESRTw L6OoygrWETTY54y2IqgUtu1FaJBhNa8pfGqeGxJYbnkiPJJYX0CGTGruwEjWhBoLV4ex rHHY7nBM67s96zyCNi/K9HmHN8oIcn4NgrP2Tk5gWpgo0QP3SkHLU2vNfkVvTlyEMbjW WOsxnPHXpu8qBdhpX1L6o506vPlHgiNEB0IcB0l3RMp5Q0+7+5C5EAHkpRMXVTc7c7fF V3Xkd/XiUpWxsAMlavHaTwc8ftbZJUIjN86jZvFsagAkOUNpWF2+GuwjNvfPKHbdXhLK gg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39up0327p0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:09 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T2086873; Mon, 19 Jul 2021 06:28:08 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hBfb/GOkrvoKkLNZx1SGZnwK0fdwFHrB++8Om9FwNTr8LcnIirodCB7rN3dpuvB6f072jOURVfj/gJCHJt4PYKn1cOA6WnOLpeDc5Fb8eL6JadivvWOWUNy89S1PDdMKmhu06+U/KLnaAP+7FiURAYVrhH74nByGtKywn96jru4NhTYXHel59aSEQkMDWG6xLh98XlUsADe/f+r8V9XnASZpwXeSONTdVTDmo4WmCkX6FuhgrZbhY2b6n8b1W0oNqzW4wGg2nbB3ugmQ8f6Zhpc83KXLbXidJ3GViPJldecPm2P/APLpjTZWMtFAzil8H4CEPOvN7vmi+2XvyuHjgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Nl3/1mkfwIUlsl+43JbQyxcQkekx1Ug7U/YnW8tRucc=; b=c7By2BZnEckjSOjQvh3+0xjnvXx5D8UE8YTXJ5jskBSH2RM9ZNlTn6Bp1AF9wHdNKfStlMjPIOGq/r7YVsN2J46VTYFXGNwYPrIXQOk1JVdt2/6bAI6wL4HLcm7iGAXDfA/ojFjoL4upFTI4UvWAv7poaEGRovTKkyI74wr9w74+N5Zpk6LRaKJ8ZTZhU4JqrJuUSfj9ibZUPY4plLt+UtgRFPlbswsK8f++NJON2jMkZ2rpYLiIc3yKex6n7tXMT3LD9aBk5BSqQ5gbB2N6qy76lqPCi2GnsIRohgPJFYOp7HT7BXoc+kSlpN+Ic/8S3LiEAfFFS+Cq4LCnAcGa6w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Nl3/1mkfwIUlsl+43JbQyxcQkekx1Ug7U/YnW8tRucc=; b=SHAEL6776NKqveDPR++kFXgiSmu7y7X4FizOtmfqD9DEheWraa8zvBAywRwlplhRW6SM2kOwHshJN/ZRfkrUgN1UuXzvgFOJ8R+/3V+1se/dKQMSa9WUizBoISmmHJn22OHQyE4CF0ThcVSmbgH6jg97vifRsWWnssRLbpEE/gM= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:04 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:04 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 01/19] vfio-user: introduce vfio-user protocol specification Date: Sun, 18 Jul 2021 23:27:40 -0700 Message-Id: <801d2c9e344dc36f64f406321d17cd2fdfbf0cb4.1626675354.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:04 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f26fc4b2-eb04-4a44-22c5-08d94a7e5d81 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: R6V7RkcsplEeIXgVHC7+K1uIha1H+v2YbSQU0PH5H2s+tfBjfyliVILgvDvrZit/2NWJOKNQIH2R+D05coO4SQ9Q9ghdVWfB+cbSqA9qKMgFsNhys09ZV9hkQiHVJRso3gEAoLMuXNt/jFa268iKUV6XYx0PvVB5qxDG6fzoovVNiIoHOE8TsxMebKgvxte319EBk0TX3NbK5DJOZn0hVitAlvehdnJPKanpCv80kgR5bwyjsw5vnNg/x4unRfQ7Ft2jJhp/wlgB9aC/rI1zc79pEo3QBwofzJVQT9Qo5RYuNdkglUjOPGAg6shkYl6egQNNK0SzAmpE7bSgwqIH+f9+sH7WD73WHBJKosOABIsHIaQP/nrsUoEIRq/D0dD8lQCVTdRopNOTEd0qcMwl2MfS1xt+RCh/nPPNDkkrsmdRnS4fcg7oysTvNv6eFBzdaKhKcMrqZKsR+WsoeYp+4TULBEoahy0wwSWUGZWN5sEmYjVTvKRcDdGUlfsN7oW/FT3tL+OG7e+OFQy+Gg/e5fzdPIIkH5K/cFhmo21A0zrZrfrBXZEJ4OSXYnt9md7haS7lds6tM2Yo54ieWuskhxn7Jo05fhMFv01KXSeyEKUP/iysTtVAPiJw2uxJY7pQ3vv/JqDJ70Pezpowp5t7d+bXP8d1+ke/BD+tFyCAcQblb/0AjXdZFXPcYTo8v+I4qUROso7sRptJ+g9nenLDeJ3tu9engKui7lVH6IDRs4cho4r/UbjpqeZdbbnVV0VIkXmwQxdzO7+JHT++P6qNqw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(30864003)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(966005)(83380400001)(66946007)(478600001)(36756003)(4326008)(579004)(559001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ZyHtgnJSrfmchPIGdoOXhLzOl9eCXgw5i2Mh2R0mHxOaqHikmkiO7pv7h6Ayd8N4pq5bdSQ5yIo5NoNuFieI8QLQHFJHK84YT5gb1zlMK5+CcdcWTo2i7s4nqH0sWlapxRjFjgvaQK27QvtFGjSctfYkHEbLsymZE0Hztg5uE9iBqST1OTtUzE0uxvnDAxJ4nUXLHNViW/VR7DrCYO5TyLUDDdrnJyV4dz2rbeAnwDKbN0rBxf9q91xENebtSeB36WewkDk+FThpwkCnEZ6fl1cxWdHH70Z9K4tuOzrl/6YUH9J8U98AP2KbPCfsGB/IJGr9MMA7yl5jB6U+9cBSCSbaTMj2Ja2YP1x6QUTPgUAKiG+DnNkxFjHTs2TDmQV4ZifNiwqluVgZ49GAcwHlz5JQJ6zWL5HIXXfrkU88M9GNvohRqGPuMi11xrvK+NituHSIlR2p41V8nz+5u/+dROUrvWhCaNQBLdq/lVpHk4CRtGLFTpXvSfXx0PCCIQThzyQPjWf4IaVzHYQbJyCVh63zNK8rjFnrAcQdwnpu7dlDNTMVwM2IgWkTq78kgZcTYGehzpt13iVWtFTs/qae8Qbw2VSTos6eOEw/7P7H1w5axFsYRJMFQa0m+3k63+/25/uPrRmaFW0nlxc6RKOBeW7/y4jPIT7lzqitjyvrQHS9eLGDt91tI/+gZmKGdnznVlZiKXKayaJJ+aiqbhusoi19f8L4XWwiGgYEbc6qky+MyNHVy+hzVt9t5zSVqf1q1k4AaHm+36J448emO6YldAupxyJPp5l7uO05fNHrYvjX54IdTbAZhCfrCjBVpXhlv2aQhjd3Ch0yA5zkruvU3vAo6Z4mVc4A/Gx6V6J9O3dvXTJza34vgglhTNUqsJUx1kkzgL3A9JZutdhQJaLLQRg5nveBFzAsYghSs+6HyCx61b2mnoXJUlwdEkbRTLDeGDiZd4CJ01Yu1n8Z66vfHxJ/x5cD7n/ZIL3wPactT+WIWmk/lMNsiaA7NoUnQkhoXmJOsMtTmWiUxeRRJQHMpeXMBw6nPJPptfxrj1IMcJch/wBb9lQIK6bo/TW/eKWhGndDMxYpIpJYxxPbkimSx6pPVvdXFUh02XYtuoqV5UCEaWXeDGOLfW4kSVmb7gNo5ujh+NyXelOxcnS2ftTmbn1Ux70ub9NBSVIVYy6c54lOAfesuTKDVhxmBIOfxh+gzQqMQ7o9pHwnrgnKf55jHxMdmB67capyNNPVHsmTNge5zt6p6BZJ6dDGGoWxKqRbZ4zRgn5LoCRGNHiH8yZR5fSzkedPMxSLRXsAxxI05ZpjjFyosJ8E2sO+RaWf9zWrgMhBmO8nezxE49kjgjUklXrQyPzB6slkW6TVgQj+SkfXsroUpXCpZYp8T7zuCoKy X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f26fc4b2-eb04-4a44-22c5-08d94a7e5d81 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:04.6523 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 02Lv4jlYhpcrs38nj+sKQGADeLCfvO+9lGs4XyBtqev33Ce1j6P0GW0oS/R8B9/AxSMLm3XojmQNFF+IMVJj/96NWrxrEO5ZcvAVq6WBaDY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-GUID: JgVIT0N20dtVXUvueSqQYfgaQhawfzq2 X-Proofpoint-ORIG-GUID: JgVIT0N20dtVXUvueSqQYfgaQhawfzq2 Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: 0 X-Spam_score: 0.0 X-Spam_bar: / X-Spam_report: (0.0 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, WEIRD_QUOTING=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Thanos Makatos This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts. This patch is sourced from: https://patchwork.kernel.org/project/qemu-devel/patch/20210614104608.212276-1-thanos.makatos@nutanix.com/ It has been earlier discussed as an RFC in: "RFC: use VFIO over a UNIX domain socket to implement device offloading" Signed-off-by: John G Johnson Signed-off-by: Thanos Makatos Signed-off-by: John Levon --- docs/devel/index.rst | 1 + docs/devel/vfio-user.rst | 1809 ++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 6 + 3 files changed, 1816 insertions(+) create mode 100644 docs/devel/vfio-user.rst diff --git a/docs/devel/index.rst b/docs/devel/index.rst index 153979caf4..215eb4ff7a 100644 --- a/docs/devel/index.rst +++ b/docs/devel/index.rst @@ -42,3 +42,4 @@ modifying QEMU's source code. multi-process ebpf_rss vfio-migration + vfio-user diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst new file mode 100644 index 0000000000..0b2acec101 --- /dev/null +++ b/docs/devel/vfio-user.rst @@ -0,0 +1,1809 @@ +.. include:: +******************************** +vfio-user Protocol Specification +******************************** + +-------------- +Version_ 0.9.1 +-------------- + +.. contents:: Table of Contents + +Introduction +============ +vfio-user is a protocol that allows a device to be emulated in a separate +process outside of a Virtual Machine Monitor (VMM). vfio-user devices consist +of a generic VFIO device type, living inside the VMM, which we call the client, +and the core device implementation, living outside the VMM, which we call the +server. + +The vfio-user specification is partly based on the +`Linux VFIO ioctl interface `_. + +VFIO is a mature and stable API, backed by an extensively used framework. The +existing VFIO client implementation in QEMU (``qemu/hw/vfio/``) can be largely +re-used, though there is nothing in this specification that requires that +particular implementation. None of the VFIO kernel modules are required for +supporting the protocol, on either the client or server side. Some source +definitions in VFIO are re-used for vfio-user. + +The main idea is to allow a virtual device to function in a separate process in +the same host over a UNIX domain socket. A UNIX domain socket (``AF_UNIX``) is +chosen because file descriptors can be trivially sent over it, which in turn +allows: + +* Sharing of client memory for DMA with the server. +* Sharing of server memory with the client for fast MMIO. +* Efficient sharing of eventfd's for triggering interrupts. + +Other socket types could be used which allow the server to run in a separate +guest in the same host (``AF_VSOCK``) or remotely (``AF_INET``). Theoretically +the underlying transport does not necessarily have to be a socket, however we do +not examine such alternatives. In this protocol version we focus on using a UNIX +domain socket and introduce basic support for the other two types of sockets +without considering performance implications. + +While passing of file descriptors is desirable for performance reasons, support +is not necessary for either the client or the server in order to implement the +protocol. There is always an in-band, message-passing fall back mechanism. + +Overview +======== + +VFIO is a framework that allows a physical device to be securely passed through +to a user space process; the device-specific kernel driver does not drive the +device at all. Typically, the user space process is a VMM and the device is +passed through to it in order to achieve high performance. VFIO provides an API +and the required functionality in the kernel. QEMU has adopted VFIO to allow a +guest to directly access physical devices, instead of emulating them in +software. + +vfio-user reuses the core VFIO concepts defined in its API, but implements them +as messages to be sent over a socket. It does not change the kernel-based VFIO +in any way, in fact none of the VFIO kernel modules need to be loaded to use +vfio-user. It is also possible for the client to concurrently use the current +kernel-based VFIO for one device, and vfio-user for another device. + +VFIO Device Model +----------------- + +A device under VFIO presents a standard interface to the user process. Many of +the VFIO operations in the existing interface use the ``ioctl()`` system call, and +references to the existing interface are called the ``ioctl()`` implementation in +this document. + +The following sections describe the set of messages that implement the vfio-user +interface over a socket. In many cases, the messages are analogous to data +structures used in the ``ioctl()`` implementation. Messages derived from the +``ioctl()`` will have a name derived from the ``ioctl()`` command name. E.g., the +``VFIO_DEVICE_GET_INFO`` ``ioctl()`` command becomes a +``VFIO_USER_DEVICE_GET_INFO`` message. The purpose of this reuse is to share as +much code as feasible with the ``ioctl()`` implementation``. + +Connection Initiation +^^^^^^^^^^^^^^^^^^^^^ + +After the client connects to the server, the initial client message is +``VFIO_USER_VERSION`` to propose a protocol version and set of capabilities to +apply to the session. The server replies with a compatible version and set of +capabilities it supports, or closes the connection if it cannot support the +advertised version. + +Device Information +^^^^^^^^^^^^^^^^^^ + +The client uses a ``VFIO_USER_DEVICE_GET_INFO`` message to query the server for +information about the device. This information includes: + +* The device type and whether it supports reset (``VFIO_DEVICE_FLAGS_``), +* the number of device regions, and +* the device presents to the client the number of interrupt types the device + supports. + +Region Information +^^^^^^^^^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_REGION_INFO`` messages to query the +server for information about the device's regions. This information describes: + +* Read and write permissions, whether it can be memory mapped, and whether it + supports additional capabilities (``VFIO_REGION_INFO_CAP_``). +* Region index, size, and offset. + +When a device region can be mapped by the client, the server provides a file +descriptor which the client can ``mmap()``. The server is responsible for +polling for client updates to memory mapped regions. + +Region Capabilities +""""""""""""""""""" + +Some regions have additional capabilities that cannot be described adequately +by the region info data structure. These capabilities are returned in the +region info reply in a list similar to PCI capabilities in a PCI device's +configuration space. + +Sparse Regions +"""""""""""""" +A region can be memory-mappable in whole or in part. When only a subset of a +region can be mapped by the client, a ``VFIO_REGION_INFO_CAP_SPARSE_MMAP`` +capability is included in the region info reply. This capability describes +which portions can be mapped by the client. + +.. Note:: + For example, in a virtual NVMe controller, sparse regions can be used so + that accesses to the NVMe registers (found in the beginning of BAR0) are + trapped (an infrequent event), while allowing direct access to the doorbells + (an extremely frequent event as every I/O submission requires a write to + BAR0), found in the next page after the NVMe registers in BAR0. + +Device-Specific Regions +""""""""""""""""""""""" + +A device can define regions additional to the standard ones (e.g. PCI indexes +0-8). This is achieved by including a ``VFIO_REGION_INFO_CAP_TYPE`` capability +in the region info reply of a device-specific region. Such regions are reflected +in ``struct vfio_user_device_info.num_regions``. Thus, for PCI devices this +value can be equal to, or higher than, ``VFIO_PCI_NUM_REGIONS``. + +Region I/O via file descriptors +------------------------------- + +For unmapped regions, region I/O from the client is done via +``VFIO_USER_REGION_READ/WRITE``. As an optimization, ioeventfds or ioregionfds +may be configured for sub-regions of some regions. A client may request +information on these sub-regions via ``VFIO_USER_DEVICE_GET_REGION_IO_FDS``; by +configuring the returned file descriptors as ioeventfds or ioregionfds, the +server can be directly notified of I/O (for example, by KVM) without taking a +trip through the client. + +Interrupts +^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_IRQ_INFO`` messages to query the server +for the device's interrupt types. The interrupt types are specific to the bus +the device is attached to, and the client is expected to know the capabilities +of each interrupt type. The server can signal an interrupt by directly injecting +interrupts into the guest via an event file descriptor. The client configures +how the server signals an interrupt with ``VFIO_USER_SET_IRQS`` messages. + +Device Read and Write +^^^^^^^^^^^^^^^^^^^^^ + +When the guest executes load or store operations to an unmapped device region, +the client forwards these operations to the server with +``VFIO_USER_REGION_READ`` or ``VFIO_USER_REGION_WRITE`` messages. The server +will reply with data from the device on read operations or an acknowledgement on +write operations. See `Read and Write Operations`_. + +Client memory access +-------------------- + +The client uses ``VFIO_USER_DMA_MAP`` and ``VFIO_USER_DMA_UNMAP`` messages to +inform the server of the valid DMA ranges that the server can access on behalf +of a device (typically, VM guest memory). DMA memory may be accessed by the +server via ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages over the +socket. In this case, the "DMA" part of the naming is a misnomer. + +Actual direct memory access of client memory from the server is possible if the +client provides file descriptors the server can ``mmap()``. Note that ``mmap()`` +privileges cannot be revoked by the client, therefore file descriptors should +only be exported in environments where the client trusts the server not to +corrupt guest memory. + +See `Read and Write Operations`_. + +Client/server interactions +========================== + +Socket +------ + +A server can serve: + +1) one or more clients, and/or +2) one or more virtual devices, belonging to one or more clients. + +The current protocol specification requires a dedicated socket per +client/server connection. It is a server-side implementation detail whether a +single server handles multiple virtual devices from the same or multiple +clients. The location of the socket is implementation-specific. Multiplexing +clients, devices, and servers over the same socket is not supported in this +version of the protocol. + +Authentication +-------------- + +For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files, +therefore it is up to the management layer to set up the socket as required. +Socket types than span guests or hosts will require a proper authentication +mechanism. Defining that mechanism is deferred to a future version of the +protocol. + +Command Concurrency +------------------- + +A client may pipeline multiple commands without waiting for previous command +replies. The server will process commands in the order they are received. A +consequence of this is if a client issues a command with the *No_reply* bit, +then subsequently issues a command without *No_reply*, the older command will +have been processed before the reply to the younger command is sent by the +server. The client must be aware of the device's capability to process +concurrent commands if pipelining is used. For example, pipelining allows +multiple client threads to concurrently access device regions; the client must +ensure these accesses obey device semantics. + +An example is a frame buffer device, where the device may allow concurrent +access to different areas of video memory, but may have indeterminate behavior +if concurrent accesses are performed to command or status registers. + +Note that unrelated messages sent from the server to the client can appear in +between a client to server request/reply and vice versa. + +Implementers should be prepared for certain commands to exhibit potentially +unbounded latencies. For example, ``VFIO_USER_DEVICE_RESET`` may take an +arbitrarily long time to complete; clients should take care not to block +unnecessarily. + +Socket Disconnection Behavior +----------------------------- +The server and the client can disconnect from each other, either intentionally +or unexpectedly. Both the client and the server need to know how to handle such +events. + +Server Disconnection +^^^^^^^^^^^^^^^^^^^^ +A server disconnecting from the client may indicate that: + +1) A virtual device has been restarted, either intentionally (e.g. because of a + device update) or unintentionally (e.g. because of a crash). +2) A virtual device has been shut down with no intention to be restarted. + +It is impossible for the client to know whether or not a failure is +intermittent or innocuous and should be retried, therefore the client should +reset the VFIO device when it detects the socket has been disconnected. +Error recovery will be driven by the guest's device error handling +behavior. + +Client Disconnection +^^^^^^^^^^^^^^^^^^^^ +The client disconnecting from the server primarily means that the client +has exited. Currently, this means that the guest is shut down so the device is +no longer needed therefore the server can automatically exit. However, there +can be cases where a client disconnection should not result in a server exit: + +1) A single server serving multiple clients. +2) A multi-process QEMU upgrading itself step by step, which is not yet + implemented. + +Therefore in order for the protocol to be forward compatible, the server should +respond to a client disconnection as follows: + + - all client memory regions are unmapped and cleaned up (including closing any + passed file descriptors) + - all IRQ file descriptors passed from the old client are closed + - the device state should otherwise be retained + +The expectation is that when a client reconnects, it will re-establish IRQ and +client memory mappings. + +If anything happens to the client (such as qemu really did exit), the control +stack will know about it and can clean up resources accordingly. + +Security Considerations +----------------------- + +Speaking generally, vfio-user clients should not trust servers, and vice versa. +Standard tools and mechanisms should be used on both sides to validate input and +prevent against denial of service scenarios, buffer overflow, etc. + +Request Retry and Response Timeout +---------------------------------- +A failed command is a command that has been successfully sent and has been +responded to with an error code. Failure to send the command in the first place +(e.g. because the socket is disconnected) is a different type of error examined +earlier in the disconnect section. + +.. Note:: + QEMU's VFIO retries certain operations if they fail. While this makes sense + for real HW, we don't know for sure whether it makes sense for virtual + devices. + +Defining a retry and timeout scheme is deferred to a future version of the +protocol. + +Message sizes +------------- + +Some requests have an ``argsz`` field. In a request, it defines the maximum +expected reply payload size, which should be at least the size of the fixed +reply payload headers defined here. The *request* payload size is defined by the +usual ``msg_size`` field in the header, not the ``argsz`` field. + +In a reply, the server sets ``argsz`` field to the size needed for a full +payload size. This may be less than the requested maximum size. This may be +larger than the requested maximum size: in that case, the full payload is not +included in the reply, but the ``argsz`` field in the reply indicates the needed +size, allowing a client to allocate a larger buffer for holding the reply before +trying again. + +In addition, during negotiation (see `Version`_), the client and server may +each specify a ``max_data_xfer_size`` value; this defines the maximum data that +may be read or written via one of the ``VFIO_USER_DMA/REGION_READ/WRITE`` +messages; see `Read and Write Operations`_. + +Protocol Specification +====================== + +To distinguish from the base VFIO symbols, all vfio-user symbols are prefixed +with ``vfio_user`` or ``VFIO_USER``. In this revision, all data is in the +little-endian format, although this may be relaxed in future revisions in cases +where the client and server are both big-endian. + +Unless otherwise specified, all sizes should be presumed to be in bytes. + +.. _Commands: + +Commands +-------- +The following table lists the VFIO message command IDs, and whether the +message command is sent from the client or the server. + +====================================== ========= ================= +Name Command Request Direction +====================================== ========= ================= +``VFIO_USER_VERSION`` 1 client -> server +``VFIO_USER_DMA_MAP`` 2 client -> server +``VFIO_USER_DMA_UNMAP`` 3 client -> server +``VFIO_USER_DEVICE_GET_INFO`` 4 client -> server +``VFIO_USER_DEVICE_GET_REGION_INFO`` 5 client -> server +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` 6 client -> server +``VFIO_USER_DEVICE_GET_IRQ_INFO`` 7 client -> server +``VFIO_USER_DEVICE_SET_IRQS`` 8 client -> server +``VFIO_USER_REGION_READ`` 9 client -> server +``VFIO_USER_REGION_WRITE`` 10 client -> server +``VFIO_USER_DMA_READ`` 11 server -> client +``VFIO_USER_DMA_WRITE`` 12 server -> client +``VFIO_USER_DEVICE_RESET`` 13 client -> server +``VFIO_USER_DIRTY_PAGES`` 14 client -> server +====================================== ========= ================= + +Header +------ + +All messages, both command messages and reply messages, are preceded by a +16-byte header that contains basic information about the message. The header is +followed by message-specific data described in the sections below. + ++----------------+--------+-------------+ +| Name | Offset | Size | ++================+========+=============+ +| Message ID | 0 | 2 | ++----------------+--------+-------------+ +| Command | 2 | 2 | ++----------------+--------+-------------+ +| Message size | 4 | 4 | ++----------------+--------+-------------+ +| Flags | 8 | 4 | ++----------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0-3 | Type | | +| | +-----+------------+ | +| | | 4 | No_reply | | +| | +-----+------------+ | +| | | 5 | Error | | +| | +-----+------------+ | ++----------------+--------+-------------+ +| Error | 12 | 4 | ++----------------+--------+-------------+ +| | 16 | variable | ++----------------+--------+-------------+ + +* *Message ID* identifies the message, and is echoed in the command's reply + message. Message IDs belong entirely to the sender, can be re-used (even + concurrently) and the receiver must not make any assumptions about their + uniqueness. +* *Command* specifies the command to be executed, listed in Commands_. It is + also set in the reply header. +* *Message size* contains the size of the entire message, including the header. +* *Flags* contains attributes of the message: + + * The *Type* bits indicate the message type. + + * *Command* (value 0x0) indicates a command message. + * *Reply* (value 0x1) indicates a reply message acknowledging a previous + command with the same message ID. + * *No_reply* in a command message indicates that no reply is needed for this + command. This is commonly used when multiple commands are sent, and only + the last needs acknowledgement. + * *Error* in a reply message indicates the command being acknowledged had + an error. In this case, the *Error* field will be valid. + +* *Error* in a reply message is an optional UNIX errno value. It may be zero + even if the Error bit is set in Flags. It is reserved in a command message. + +Each command message in Commands_ must be replied to with a reply message, +unless the message sets the *No_Reply* bit. The reply consists of the header +with the *Reply* bit set, plus any additional data. + +If an error occurs, the reply message must only include the reply header. + +As the header is standard in both requests and replies, it is not included in +the command-specific specifications below; each message definition should be +appended to the standard header, and the offsets are given from the end of the +standard header. + +``VFIO_USER_VERSION`` +--------------------- + +.. _Version: + +This is the initial message sent by the client after the socket connection is +established; the same format is used for the server's reply. + +Upon establishing a connection, the client must send a ``VFIO_USER_VERSION`` +message proposing a protocol version and a set of capabilities. The server +compares these with the versions and capabilities it supports and sends a +``VFIO_USER_VERSION`` reply according to the following rules. + +* The major version in the reply must be the same as proposed. If the client + does not support the proposed major, it closes the connection. +* The minor version in the reply must be equal to or less than the minor + version proposed. +* The capability list must be a subset of those proposed. If the server + requires a capability the client did not include, it closes the connection. + +The protocol major version will only change when incompatible protocol changes +are made, such as changing the message format. The minor version may change +when compatible changes are made, such as adding new messages or capabilities, +Both the client and server must support all minor versions less than the +maximum minor version it supports. E.g., an implementation that supports +version 1.3 must also support 1.0 through 1.2. + +When making a change to this specification, the protocol version number must +be included in the form "added in version X.Y" + +Request +^^^^^^^ + +============== ====== ==== +Name Offset Size +============== ====== ==== +version major 0 2 +version minor 2 2 +version data 4 variable (including terminating NUL). Optional. +============== ====== ==== + +The version data is an optional UTF-8 encoded JSON byte array with the following +format: + ++--------------+--------+-----------------------------------+ +| Name | Type | Description | ++==============+========+===================================+ +| capabilities | object | Contains common capabilities that | +| | | the sender supports. Optional. | ++--------------+--------+-----------------------------------+ + +Capabilities: + ++--------------------+--------+------------------------------------------------+ +| Name | Type | Description | ++====================+========+================================================+ +| max_msg_fds | number | Maximum number of file descriptors that can be | +| | | received by the sender in one message. | +| | | Optional. If not specified then the receiver | +| | | must assume a value of ``1``. | ++--------------------+--------+------------------------------------------------+ +| max_data_xfer_size | number | Maximum ``count`` for data transfer messages; | +| | | see `Read and Write Operations`_. Optional, | +| | | with a default value of 1048576 bytes. | ++--------------------+--------+------------------------------------------------+ +| migration | object | Migration capability parameters. If missing | +| | | then migration is not supported by the sender. | ++--------------------+--------+------------------------------------------------+ + +The migration capability contains the following name/value pairs: + ++--------+--------+-----------------------------------------------+ +| Name | Type | Description | ++========+========+===============================================+ +| pgsize | number | Page size of dirty pages bitmap. The smallest | +| | | between the client and the server is used. | ++--------+--------+-----------------------------------------------+ + +Reply +^^^^^ + +The same message format is used in the server's reply with the semantics +described above. + +``VFIO_USER_DMA_MAP`` +--------------------- + +This command message is sent by the client to the server to inform it of the +memory regions the server can access. It must be sent before the server can +perform any DMA to the client. It is normally sent directly after the version +handshake is completed, but may also occur when memory is added to the client, +or if the client uses a vIOMMU. + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++-------------+--------+-------------+ +| Name | Offset | Size | ++=============+========+=============+ +| argsz | 0 | 4 | ++-------------+--------+-------------+ +| flags | 4 | 4 | ++-------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0 | readable | | +| | +-----+------------+ | +| | | 1 | writeable | | +| | +-----+------------+ | ++-------------+--------+-------------+ +| offset | 8 | 8 | ++-------------+--------+-------------+ +| address | 16 | 8 | ++-------------+--------+-------------+ +| size | 24 | 8 | ++-------------+--------+-------------+ + +* *argsz* is the size of the above structure. Note there is no reply payload, + so this field differs from other message types. +* *flags* contains the following region attributes: + + * *readable* indicates that the region can be read from. + + * *writeable* indicates that the region can be written to. + +* *offset* is the file offset of the region with respect to the associated file + descriptor, or zero if the region is not mappable +* *address* is the base DMA address of the region. +* *size* is the size of the region. + +This structure is 32 bytes in size, so the message size is 16 + 32 bytes. + +If the DMA region being added can be directly mapped by the server, a file +descriptor must be sent as part of the message meta-data. The region can be +mapped via the mmap() system call. On ``AF_UNIX`` sockets, the file descriptor +must be passed as ``SCM_RIGHTS`` type ancillary data. Otherwise, if the DMA +region cannot be directly mapped by the server, no file descriptor must be sent +as part of the message meta-data and the DMA region can be accessed by the +server using ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages, +explained in `Read and Write Operations`_. A command to map over an existing +region must be failed by the server with ``EEXIST`` set in error field in the +reply. + +Reply +^^^^^ + +There is no payload in the reply message. + +``VFIO_USER_DMA_UNMAP`` +----------------------- + +This command message is sent by the client to the server to inform it that a +DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command +message, is no longer available for DMA. It typically occurs when memory is +subtracted from the client or if the client uses a vIOMMU. The DMA region is +described by the following structure: + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++--------------+--------+------------------------+ +| Name | Offset | Size | ++==============+========+========================+ +| argsz | 0 | 4 | ++--------------+--------+------------------------+ +| flags | 4 | 4 | ++--------------+--------+------------------------+ +| | +-----+-----------------------+ | +| | | Bit | Definition | | +| | +=====+=======================+ | +| | | 0 | get dirty page bitmap | | +| | +-----+-----------------------+ | ++--------------+--------+------------------------+ +| address | 8 | 8 | ++--------------+--------+------------------------+ +| size | 16 | 8 | ++--------------+--------+------------------------+ + +* *argsz* is the maximum size of the reply payload. +* *flags* contains the following DMA region attributes: + + * *get dirty page bitmap* indicates that a dirty page bitmap must be + populated before unmapping the DMA region. The client must provide a + `VFIO Bitmap`_ structure, explained below, immediately following this + entry. + +* *address* is the base DMA address of the DMA region. +* *size* is the size of the DMA region. + +The address and size of the DMA region being unmapped must match exactly a +previous mapping. The size of request message depends on whether or not the +*get dirty page bitmap* bit is set in Flags: + +* If not set, the size of the total request message is: 16 + 24. + +* If set, the size of the total request message is: 16 + 24 + 16. + +.. _VFIO Bitmap: + +VFIO Bitmap Format +"""""""""""""""""" + ++--------+--------+------+ +| Name | Offset | Size | ++========+========+======+ +| pgsize | 0 | 8 | ++--------+--------+------+ +| size | 8 | 8 | ++--------+--------+------+ + +* *pgsize* is the page size for the bitmap, in bytes. +* *size* is the size for the bitmap, in bytes, excluding the VFIO bitmap header. + +Reply +^^^^^ + +Upon receiving a ``VFIO_USER_DMA_UNMAP`` command, if the file descriptor is +mapped then the server must release all references to that DMA region before +replying, which potentially includes in-flight DMA transactions. + +The server responds with the original DMA entry in the request. If the +*get dirty page bitmap* bit is set in flags in the request, then +the server also includes the `VFIO Bitmap`_ structure sent in the request, +followed by the corresponding dirty page bitmap, where each bit represents +one page of size *pgsize* in `VFIO Bitmap`_ . + +The total size of the total reply message is: +16 + 24 + (16 + *size* in `VFIO Bitmap`_ if *get dirty page bitmap* is set). + +``VFIO_USER_DEVICE_GET_INFO`` +----------------------------- + +This command message is sent by the client to the server to query for basic +information about the device. + +Request +^^^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the maximum size of the reply payload +* all other fields must be zero. + +Reply +^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* contains the following device attributes. + + * ``VFIO_DEVICE_FLAGS_RESET`` indicates that the device supports the + ``VFIO_USER_DEVICE_RESET`` message. + * ``VFIO_DEVICE_FLAGS_PCI`` indicates that the device is a PCI device. + +* *num_regions* is the number of memory regions that the device exposes. +* *num_irqs* is the number of distinct interrupt types that the device supports. + +This version of the protocol only supports PCI devices. Additional devices may +be supported in future versions. + +``VFIO_USER_DEVICE_GET_REGION_INFO`` +------------------------------------ + +This command message is sent by the client to the server to query for +information about device regions. The VFIO region info structure is defined in +```` (``struct vfio_region_info``). + +Request +^^^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* all other fields must be zero. + +Reply +^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_REGION_INFO_FLAG_READ | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_REGION_INFO_FLAG_WRITE | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_REGION_INFO_FLAG_MMAP | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_REGION_INFO_FLAG_CAPS | | +| | +-----+-----------------------------+ | ++------------+--------+------------------------------+ ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* is the size required for the full reply payload (region info structure + plus the size of any region capabilities) +* *flags* are attributes of the region: + + * ``VFIO_REGION_INFO_FLAG_READ`` allows client read access to the region. + * ``VFIO_REGION_INFO_FLAG_WRITE`` allows client write access to the region. + * ``VFIO_REGION_INFO_FLAG_MMAP`` specifies the client can mmap() the region. + When this flag is set, the reply will include a file descriptor in its + meta-data. On ``AF_UNIX`` sockets, the file descriptors will be passed as + ``SCM_RIGHTS`` type ancillary data. + * ``VFIO_REGION_INFO_FLAG_CAPS`` indicates additional capabilities found in the + reply. + +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* *cap_offset* describes where additional region capabilities can be found. + cap_offset is relative to the beginning of the VFIO region info structure. + The data structure it points is a VFIO cap header defined in + ````. +* *size* is the size of the region. +* *offset* is the offset that should be given to the mmap() system call for + regions with the MMAP attribute. It is also used as the base offset when + mapping a VFIO sparse mmap area, described below. + +VFIO region capabilities +"""""""""""""""""""""""" + +The VFIO region information can also include a capabilities list. This list is +similar to a PCI capability list - each entry has a common header that +identifies a capability and where the next capability in the list can be found. +The VFIO capability header format is defined in ```` (``struct +vfio_info_cap_header``). + +VFIO cap header format +"""""""""""""""""""""" + ++---------+--------+------+ +| Name | Offset | Size | ++=========+========+======+ +| id | 0 | 2 | ++---------+--------+------+ +| version | 2 | 2 | ++---------+--------+------+ +| next | 4 | 4 | ++---------+--------+------+ + +* *id* is the capability identity. +* *version* is a capability-specific version number. +* *next* specifies the offset of the next capability in the capability list. It + is relative to the beginning of the VFIO region info structure. + +VFIO sparse mmap cap header +""""""""""""""""""""""""""" + ++------------------+----------------------------------+ +| Name | Value | ++==================+==================================+ +| id | VFIO_REGION_INFO_CAP_SPARSE_MMAP | ++------------------+----------------------------------+ +| version | 0x1 | ++------------------+----------------------------------+ +| next | | ++------------------+----------------------------------+ +| sparse mmap info | VFIO region info sparse mmap | ++------------------+----------------------------------+ + +This capability is defined when only a subrange of the region supports +direct access by the client via mmap(). The VFIO sparse mmap area is defined in +```` (``struct vfio_region_sparse_mmap_area`` and ``struct +vfio_region_info_cap_sparse_mmap``). + +VFIO region info cap sparse mmap +"""""""""""""""""""""""""""""""" + ++----------+--------+------+ +| Name | Offset | Size | ++==========+========+======+ +| nr_areas | 0 | 4 | ++----------+--------+------+ +| reserved | 4 | 4 | ++----------+--------+------+ +| offset | 8 | 8 | ++----------+--------+------+ +| size | 16 | 9 | ++----------+--------+------+ +| ... | | | ++----------+--------+------+ + +* *nr_areas* is the number of sparse mmap areas in the region. +* *offset* and size describe a single area that can be mapped by the client. + There will be *nr_areas* pairs of offset and size. The offset will be added to + the base offset given in the ``VFIO_USER_DEVICE_GET_REGION_INFO`` to form the + offset argument of the subsequent mmap() call. + +The VFIO sparse mmap area is defined in ```` (``struct +vfio_region_info_cap_sparse_mmap``). + +VFIO region type cap header +""""""""""""""""""""""""""" + ++------------------+---------------------------+ +| Name | Value | ++==================+===========================+ +| id | VFIO_REGION_INFO_CAP_TYPE | ++------------------+---------------------------+ +| version | 0x1 | ++------------------+---------------------------+ +| next | | ++------------------+---------------------------+ +| region info type | VFIO region info type | ++------------------+---------------------------+ + +This capability is defined when a region is specific to the device. + +VFIO region info type cap +""""""""""""""""""""""""" + +The VFIO region info type is defined in ```` +(``struct vfio_region_info_cap_type``). + ++---------+--------+------+ +| Name | Offset | Size | ++=========+========+======+ +| type | 0 | 4 | ++---------+--------+------+ +| subtype | 4 | 4 | ++---------+--------+------+ + +The only device-specific region type and subtype supported by vfio-user is +``VFIO_REGION_TYPE_MIGRATION`` (3) and ``VFIO_REGION_SUBTYPE_MIGRATION`` (1). + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` +-------------------------------------- + +Clients can access regions via ``VFIO_USER_REGION_READ/WRITE`` or, if provided, by +``mmap()`` of a file descriptor provided by the server. + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` provides an alternative access mechanism via +file descriptors. This is an optional feature intended for performance +improvements where an underlying sub-system (such as KVM) supports communication +across such file descriptors to the vfio-user server, without needing to +round-trip through the client. + +The server returns an array of sub-regions for the requested region. Each +sub-region describes a span (offset and size) of a region, along with the +requested file descriptor notification mechanism to use. Each sub-region in the +response message may choose to use a different method, as defined below. The +two mechanisms supported in this specification are ioeventfds and ioregionfds. + +The server in addition returns a file descriptor in the ancillary data; clients +are expected to configure each sub-region's file descriptor with the requested +notification method. For example, a client could configure KVM with the +requested ioeventfd via a ``KVM_IOEVENTFD`` ``ioctl()``. + +Request +^^^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried +* all other fields must be zero + +The client must set ``flags`` to zero and specify the region being queried in +the ``index``. + +Reply +^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ +| sub-regions | 16 | ... | ++-------------+--------+------+ + +* *argsz* is the size of the region IO FD info structure plus the + total size of the sub-region array. Thus, each array entry "i" is at offset + i * ((argsz - 32) / count). Note that currently this is 40 bytes for both IO + FD types, but this is not to be relied on. As elsewhere, this indicates the + full reply payload size needed. +* *flags* must be zero +* *index* is the index of memory region being queried +* *count* is the number of sub-regions in the array +* *sub-regions* is the array of Sub-Region IO FD info structures + +The reply message will additionally include at least one file descriptor in the +ancillary data. Note that more than one sub-region may share the same file +descriptor. + +Note that it is the client's responsibility to verify the requested values (for +example, that the requested offset does not exceed the region's bounds). + +Each sub-region given in the response has one of two possible structures, +depending whether *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` or +``VFIO_USER_IO_FD_TYPE_IOREGIONFD``: + +Sub-Region IO FD info format (ioeventfd) +"""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| datamatch | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations +* *fd_index* is the index in the ancillary data of the FD to use for ioeventfd + notification; it may be shared. +* *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` +* *flags* is any of: + + * ``KVM_IOEVENTFD_FLAG_DATAMATCH`` + * ``KVM_IOEVENTFD_FLAG_PIO`` + * ``KVM_IOEVENTFD_FLAG_VIRTIO_CCW_NOTIFY`` (FIXME: makes sense?) + +* *datamatch* is the datamatch value if needed + +See https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt, *4.59 +KVM_IOEVENTFD* for further context on the ioeventfd-specific fields. + +Sub-Region IO FD info format (ioregionfd) +""""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| user_data | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations; ``KVM_IOREGION_POSTED_WRITES`` + must be set in *flags* in this case +* *fd_index* is the index in the ancillary data of the FD to use for ioregionfd + messages; it may be shared +* *type* is ``VFIO_USER_IO_FD_TYPE_IOREGIONFD`` +* *flags* is any of: + + * ``KVM_IOREGION_PIO`` + * ``KVM_IOREGION_POSTED_WRITES`` + +* *user_data* is an opaque value passed back to the server via a message on the + file descriptor + +For further information on the ioregionfd-specific fields, see: +https://lore.kernel.org/kvm/cover.1613828726.git.eafanasova@gmail.com/ + +(FIXME: update with final API docs.) + +``VFIO_USER_DEVICE_GET_IRQ_INFO`` +--------------------------------- + +This command message is sent by the client to the server to query for +information about device interrupt types. The VFIO IRQ info structure is +defined in ```` (``struct vfio_irq_info``). + +Request +^^^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the maximum size of the reply payload (16 bytes today) +* index is the index of IRQ type being queried (e.g. ``VFIO_PCI_MSIX_IRQ_INDEX``) +* all other fields must be zero + +Reply +^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* defines IRQ attributes: + + * ``VFIO_IRQ_INFO_EVENTFD`` indicates the IRQ type can support server eventfd + signalling. + * ``VFIO_IRQ_INFO_MASKABLE`` indicates that the IRQ type supports the ``MASK`` + and ``UNMASK`` actions in a ``VFIO_USER_DEVICE_SET_IRQS`` message. + * ``VFIO_IRQ_INFO_AUTOMASKED`` indicates the IRQ type masks itself after being + triggered, and the client must send an ``UNMASK`` action to receive new + interrupts. + * ``VFIO_IRQ_INFO_NORESIZE`` indicates ``VFIO_USER_SET_IRQS`` operations setup + interrupts as a set, and new sub-indexes cannot be enabled without disabling + the entire type. +* index is the index of IRQ type being queried +* count describes the number of interrupts of the queried type. + +``VFIO_USER_DEVICE_SET_IRQS`` +----------------------------- + +This command message is sent by the client to the server to set actions for +device interrupt types. The VFIO IRQ set structure is defined in +```` (``struct vfio_irq_set``). + +Request +^^^^^^^ + ++-------+--------+------------------------------+ +| Name | Offset | Size | ++=======+========+==============================+ +| argsz | 0 | 4 | ++-------+--------+------------------------------+ +| flags | 4 | 4 | ++-------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_IRQ_SET_DATA_NONE | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_IRQ_SET_DATA_BOOL | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_IRQ_SET_DATA_EVENTFD | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_IRQ_SET_ACTION_MASK | | +| | +-----+-----------------------------+ | +| | | 4 | VFIO_IRQ_SET_ACTION_UNMASK | | +| | +-----+-----------------------------+ | +| | | 5 | VFIO_IRQ_SET_ACTION_TRIGGER | | +| | +-----+-----------------------------+ | ++-------+--------+------------------------------+ +| index | 8 | 4 | ++-------+--------+------------------------------+ +| start | 12 | 4 | ++-------+--------+------------------------------+ +| count | 16 | 4 | ++-------+--------+------------------------------+ +| data | 20 | variable | ++-------+--------+------------------------------+ + +* *argsz* is the size of the VFIO IRQ set request payload, including any *data* + field. Note there is no reply payload, so this field differs from other + message types. +* *flags* defines the action performed on the interrupt range. The ``DATA`` + flags describe the data field sent in the message; the ``ACTION`` flags + describe the action to be performed. The flags are mutually exclusive for + both sets. + + * ``VFIO_IRQ_SET_DATA_NONE`` indicates there is no data field in the command. + The action is performed unconditionally. + * ``VFIO_IRQ_SET_DATA_BOOL`` indicates the data field is an array of boolean + bytes. The action is performed if the corresponding boolean is true. + * ``VFIO_IRQ_SET_DATA_EVENTFD`` indicates an array of event file descriptors + was sent in the message meta-data. These descriptors will be signalled when + the action defined by the action flags occurs. In ``AF_UNIX`` sockets, the + descriptors are sent as ``SCM_RIGHTS`` type ancillary data. + If no file descriptors are provided, this de-assigns the specified + previously configured interrupts. + * ``VFIO_IRQ_SET_ACTION_MASK`` indicates a masking event. It can be used with + ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to mask an interrupt, + or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the guest masks + the interrupt. + * ``VFIO_IRQ_SET_ACTION_UNMASK`` indicates an unmasking event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to unmask an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + guest unmasks the interrupt. + * ``VFIO_IRQ_SET_ACTION_TRIGGER`` indicates a triggering event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to trigger an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + server triggers the interrupt. + +* *index* is the index of IRQ type being setup. +* *start* is the start of the sub-index being set. +* *count* describes the number of sub-indexes being set. As a special case, a + count (and start) of 0, with data flags of ``VFIO_IRQ_SET_DATA_NONE`` disables + all interrupts of the index. +* *data* is an optional field included when the + ``VFIO_IRQ_SET_DATA_BOOL`` flag is present. It contains an array of booleans + that specify whether the action is to be performed on the corresponding + index. It's used when the action is only performed on a subset of the range + specified. + +Not all interrupt types support every combination of data and action flags. +The client must know the capabilities of the device and IRQ index before it +sends a ``VFIO_USER_DEVICE_SET_IRQ`` message. + +In typical operation, a specific IRQ may operate as follows: + +1. The client sends a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_TRIGGER)`` along + with an eventfd. This associates the IRQ with a particular eventfd on the + server side. + +#. The client may send a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_MASK/UNMASK)`` along + with another eventfd. This associates the given eventfd with the + mask/unmask state on the server side. + +#. The server may trigger the IRQ by writing 1 to the eventfd. + +#. The server may mask/unmask an IRQ which will write 1 to the corresponding + mask/unmask eventfd, if there is one. + +5. A client may trigger a device IRQ itself, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_TRIGGER)``. + +6. A client may mask or unmask the IRQ, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_MASK/UNMASK)``. + +Reply +^^^^^ + +There is no payload in the reply. + +.. _Read and Write Operations: + +Note that all of these operations must be supported by the client and/or server, +even if the corresponding memory or device region has been shared as mappable. + +The ``count`` field must not exceed the value of ``max_data_xfer_size`` of the +peer, for both reads and writes. + +``VFIO_USER_REGION_READ`` +------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via ``mmap()`` of the underlying file descriptor. In this case, a client can +read from a device region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. +* *data* is the data that was read from the device region. + +``VFIO_USER_REGION_WRITE`` +-------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via mmap() of the underlying fd. In this case, a client can write to a device +region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DMA_READ`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +read from guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. +* *data* is the data read. + +``VFIO_USER_DMA_WRITE`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +write to guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 4 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DEVICE_RESET`` +-------------------------- + +This command message is sent from the client to the server to reset the device. +Neither the request or reply have a payload. + +``VFIO_USER_DIRTY_PAGES`` +------------------------- + +This command is analogous to ``VFIO_IOMMU_DIRTY_PAGES``. It is sent by the client +to the server in order to control logging of dirty pages, usually during a live +migration. + +Dirty page tracking is optional for server implementation; clients should not +rely on it. + +Request +^^^^^^^ + ++-------+--------+-----------------------------------------+ +| Name | Offset | Size | ++=======+========+=========================================+ +| argsz | 0 | 4 | ++-------+--------+-----------------------------------------+ +| flags | 4 | 4 | ++-------+--------+-----------------------------------------+ +| | +-----+----------------------------------------+ | +| | | Bit | Definition | | +| | +=====+========================================+ | +| | | 0 | VFIO_IOMMU_DIRTY_PAGES_FLAG_START | | +| | +-----+----------------------------------------+ | +| | | 1 | VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP | | +| | +-----+----------------------------------------+ | +| | | 2 | VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP | | +| | +-----+----------------------------------------+ | ++-------+--------+-----------------------------------------+ + +* *argsz* is the size of the VFIO dirty bitmap info structure for + ``START/STOP``; and for ``GET_BITMAP``, the maximum size of the reply payload + +* *flags* defines the action to be performed by the server: + + * ``VFIO_IOMMU_DIRTY_PAGES_FLAG_START`` instructs the server to start logging + pages it dirties. Logging continues until explicitly disabled by + ``VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP``. + + * ``VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP`` instructs the server to stop logging + dirty pages. + + * ``VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP`` requests the server to return + the dirty bitmap for a specific IOVA range. The IOVA range is specified by + a "VFIO Bitmap Range" structure, which must immediately follow this + "VFIO Dirty Pages" structure. See `VFIO Bitmap Range Format`_. + This operation is only valid if logging of dirty pages has been previously + started. + + These flags are mutually exclusive with each other. + +This part of the request is analogous to VFIO's ``struct +vfio_iommu_type1_dirty_bitmap``. + +.. _VFIO Bitmap Range Format: + +VFIO Bitmap Range Format +"""""""""""""""""""""""" + ++--------+--------+------+ +| Name | Offset | Size | ++========+========+======+ +| iova | 0 | 8 | ++--------+--------+------+ +| size | 8 | 8 | ++--------+--------+------+ +| bitmap | 16 | 24 | ++--------+--------+------+ + +* *iova* is the IOVA offset + +* *size* is the size of the IOVA region + +* *bitmap* is the VFIO Bitmap explained in `VFIO Bitmap`_. + +This part of the request is analogous to VFIO's ``struct +vfio_iommu_type1_dirty_bitmap_get``. + +Reply +^^^^^ + +For ``VFIO_IOMMU_DIRTY_PAGES_FLAG_START`` or +``VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP``, there is no reply payload. + +For ``VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP``, the reply payload is as follows: + ++--------------+--------+-----------------------------------------+ +| Name | Offset | Size | ++==============+========+=========================================+ +| argsz | 0 | 4 | ++--------------+--------+-----------------------------------------+ +| flags | 4 | 4 | ++--------------+--------+-----------------------------------------+ +| | +-----+----------------------------------------+ | +| | | Bit | Definition | | +| | +=====+========================================+ | +| | | 2 | VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP | | +| | +-----+----------------------------------------+ | ++--------------+--------+-----------------------------------------+ +| bitmap range | 8 | 40 | ++--------------+--------+-----------------------------------------+ +| bitmap | 48 | variable | ++--------------+--------+-----------------------------------------+ + +* *argsz* is the size required for the full reply payload (dirty pages structure + + bitmap range structure + actual bitmap) +* *flags* is ``VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP`` +* *bitmap range* is the same bitmap range struct provided in the request, as + defined in `VFIO Bitmap Range Format`_. +* *bitmap* is the actual dirty pages bitmap corresponding to the range request + +VFIO Device Migration Info +-------------------------- + +A device may contain a migration region (of type +``VFIO_REGION_TYPE_MIGRATION``). The beginning of the region must contain +``struct vfio_device_migration_info``, defined in ````. This +subregion is accessed like any other part of a standard vfio-user region +using ``VFIO_USER_REGION_READ``/``VFIO_USER_REGION_WRITE``. + ++---------------+--------+-----------------------------+ +| Name | Offset | Size | ++===============+========+=============================+ +| device_state | 0 | 4 | ++---------------+--------+-----------------------------+ +| | +-----+----------------------------+ | +| | | Bit | Definition | | +| | +=====+============================+ | +| | | 0 | VFIO_DEVICE_STATE_RUNNING | | +| | +-----+----------------------------+ | +| | | 1 | VFIO_DEVICE_STATE_SAVING | | +| | +-----+----------------------------+ | +| | | 2 | VFIO_DEVICE_STATE_RESUMING | | +| | +-----+----------------------------+ | ++---------------+--------+-----------------------------+ +| reserved | 4 | 4 | ++---------------+--------+-----------------------------+ +| pending_bytes | 8 | 8 | ++---------------+--------+-----------------------------+ +| data_offset | 16 | 8 | ++---------------+--------+-----------------------------+ +| data_size | 24 | 8 | ++---------------+--------+-----------------------------+ + +* *device_state* defines the state of the device: + + The client initiates device state transition by writing the intended state. + The server must respond only after it has successfully transitioned to the new + state. If an error occurs then the server must respond to the + ``VFIO_USER_REGION_WRITE`` operation with the Error field set accordingly and + must remain at the previous state, or in case of internal error it must + transition to the error state, defined as + ``VFIO_DEVICE_STATE_RESUMING | VFIO_DEVICE_STATE_SAVING``. The client must + re-read the device state in order to determine it afresh. + + The following device states are defined: + + +-----------+---------+----------+-----------------------------------+ + | _RESUMING | _SAVING | _RUNNING | Description | + +===========+=========+==========+===================================+ + | 0 | 0 | 0 | Device is stopped. | + +-----------+---------+----------+-----------------------------------+ + | 0 | 0 | 1 | Device is running, default state. | + +-----------+---------+----------+-----------------------------------+ + | 0 | 1 | 0 | Stop-and-copy state | + +-----------+---------+----------+-----------------------------------+ + | 0 | 1 | 1 | Pre-copy state | + +-----------+---------+----------+-----------------------------------+ + | 1 | 0 | 0 | Resuming | + +-----------+---------+----------+-----------------------------------+ + | 1 | 0 | 1 | Invalid state | + +-----------+---------+----------+-----------------------------------+ + | 1 | 1 | 0 | Error state | + +-----------+---------+----------+-----------------------------------+ + | 1 | 1 | 1 | Invalid state | + +-----------+---------+----------+-----------------------------------+ + + Valid state transitions are shown in the following table: + + +-------------------------+---------+---------+---------------+----------+----------+ + | |darr| From / To |rarr| | Stopped | Running | Stop-and-copy | Pre-copy | Resuming | + +=========================+=========+=========+===============+==========+==========+ + | Stopped | \- | 1 | 0 | 0 | 0 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Running | 1 | \- | 1 | 1 | 1 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Stop-and-copy | 1 | 1 | \- | 0 | 0 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Pre-copy | 0 | 0 | 1 | \- | 0 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Resuming | 0 | 1 | 0 | 0 | \- | + +-------------------------+---------+---------+---------------+----------+----------+ + + A device is migrated to the destination as follows: + + * The source client transitions the device state from the running state to + the pre-copy state. This transition is optional for the client but must be + supported by the server. The source server starts sending device state data + to the source client through the migration region while the device is + running. + + * The source client transitions the device state from the running state or the + pre-copy state to the stop-and-copy state. The source server stops the + device, saves device state and sends it to the source client through the + migration region. + + The source client is responsible for sending the migration data to the + destination client. + + A device is resumed on the destination as follows: + + * The destination client transitions the device state from the running state + to the resuming state. The destination server uses the device state data + received through the migration region to resume the device. + + * The destination client provides saved device state to the destination + server and then transitions the device to back to the running state. + +* *reserved* This field is reserved and any access to it must be ignored by the + server. + +* *pending_bytes* Remaining bytes to be migrated by the server. This field is + read only. + +* *data_offset* Offset in the migration region where the client must: + + * read from, during the pre-copy or stop-and-copy state, or + + * write to, during the resuming state. + + This field is read only. + +* *data_size* Contains the size, in bytes, of the amount of data copied to: + + * the source migration region by the source server during the pre-copy or + stop-and copy state, or + + * the destination migration region by the destination client during the + resuming state. + +Device-specific data must be stored at any position after +``struct vfio_device_migration_info``. Note that the migration region can be +memory mappable, even partially. In practise, only the migration data portion +can be memory mapped. + +The client processes device state data during the pre-copy and the +stop-and-copy state in the following iterative manner: + + 1. The client reads ``pending_bytes`` to mark a new iteration. Repeated reads + of this field is an idempotent operation. If there are no migration data + to be consumed then the next step depends on the current device state: + + * pre-copy: the client must try again. + + * stop-and-copy: this procedure can end and the device can now start + resuming on the destination. + + 2. The client reads ``data_offset``; at this point the server must make + available a portion of migration data at this offset to be read by the + client, which must happen *before* completing the read operation. The + amount of data to be read must be stored in the ``data_size`` field, which + the client reads next. + + 3. The client reads ``data_size`` to determine the amount of migration data + available. + + 4. The client reads and processes the migration data. + + 5. Go to step 1. + +Note that the client can transition the device from the pre-copy state to the +stop-and-copy state at any time; ``pending_bytes`` does not need to become zero. + +The client initializes the device state on the destination by setting the +device state in the resuming state and writing the migration data to the +destination migration region at ``data_offset`` offset. The client can write the +source migration data in an iterative manner and the server must consume this +data before completing each write operation, updating the ``data_offset`` field. +The server must apply the source migration data on the device resume state. The +client must write data on the same order and transaction size as read. + +If an error occurs then the server must fail the read or write operation. It is +an implementation detail of the client how to handle errors. + +Appendices +========== + +Unused VFIO ``ioctl()`` commands +-------------------------------- + +The following VFIO commands do not have an equivalent vfio-user command: + +* ``VFIO_GET_API_VERSION`` +* ``VFIO_CHECK_EXTENSION`` +* ``VFIO_SET_IOMMU`` +* ``VFIO_GROUP_GET_STATUS`` +* ``VFIO_GROUP_SET_CONTAINER`` +* ``VFIO_GROUP_UNSET_CONTAINER`` +* ``VFIO_GROUP_GET_DEVICE_FD`` +* ``VFIO_IOMMU_GET_INFO`` + +However, once support for live migration for VFIO devices is finalized some +of the above commands may have to be handled by the client in their +corresponding vfio-user form. This will be addressed in a future protocol +version. + +VFIO groups and containers +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The current VFIO implementation includes group and container idioms that +describe how a device relates to the host IOMMU. In the vfio-user +implementation, the IOMMU is implemented in SW by the client, and is not +visible to the server. The simplest idea would be that the client put each +device into its own group and container. + +Backend Program Conventions +--------------------------- + +vfio-user backend program conventions are based on the vhost-user ones. + +* The backend program must not daemonize itself. +* No assumptions must be made as to what access the backend program has on the + system. +* File descriptors 0, 1 and 2 must exist, must have regular + stdin/stdout/stderr semantics, and can be redirected. +* The backend program must honor the SIGTERM signal. +* The backend program must accept the following commands line options: + + * ``--socket-path=PATH``: path to UNIX domain socket, + * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible with + ``--socket-path`` +* The backend program must be accompanied with a JSON file stored under + ``/usr/share/vfio-user``. + +TODO add schema similar to docs/interop/vhost-user.json. diff --git a/MAINTAINERS b/MAINTAINERS index 4256ad1adb..12d69f3a45 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1880,6 +1880,12 @@ F: hw/vfio/ap.c F: docs/system/s390x/vfio-ap.rst L: qemu-s390x@nongnu.org +vfio-user +M: John G Johnson +M: Thanos Makatos +S: Supported +F: docs/devel/vfio-user.rst + vhost M: Michael S. Tsirkin S: Supported From patchwork Mon Jul 19 06:27:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3FA8C636CB for ; Mon, 19 Jul 2021 06:32:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 52B7660BD3 for ; Mon, 19 Jul 2021 06:32:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 52B7660BD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37728 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mp0-00049D-EN for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:32:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33382) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlF-000441-7v for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:17 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:9156) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlA-0002EH-Hl for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:16 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GRNK009287; Mon, 19 Jul 2021 06:28:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=xikYxdDBMphjgwUL3W8+0CRy9R1jKeEBwNoTvL20p28=; b=WpDZDGODUXgXtGvWy1JOOu3NDxzzoWVlnKupI8vsX7Bt4ZK/SlKhJydIe7F6mQjcjVDL ZmKOuKZQOCR5RqsS8cjlrnTbHB2t8B2Ke5zxYOeEwpb3kWv6lgqCC2YxfX7bsGX7EPua /eLPJauuM7B/WkoklhABd/o1AfKXOxkCVGteG8hdHYrlq/dV8Jj9J1osCxhAc7/zRqsY nGh2cQ0IKQC/a+dxdbm8pv1B0lnlrfw3QrSJ2W6eLikpAAu11vsHwEro6QbFE2uQ28I5 nIgcdS94pm2O0X7RozYd+iI5iADpVCfwG4s4ixGsee2MCQ6g9jsRBbVB8LdlZxnfY02D 2Q== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=xikYxdDBMphjgwUL3W8+0CRy9R1jKeEBwNoTvL20p28=; b=BlWRgcBOCDMBNNY2XKzhXWma6spKjupswYexvgnDdroRbZN3hE0nfwqft8m3m1kjreUv eNvXvFHtEh0ZpYClCkwgG5hOqnxLIyt477AvhmzLlcGH/94CWkZDhXREaMCX6Q+ZHq5V OL/jqp/8KCkbabqTA8cR8VBQ1Xm6QlJrKFcuZrp1ifOJNS3Tfwz86rACj/+eMg5eceJp KAN6Wb1suJUV+A4xc5HR9mRgXxt42KQ0tgzHJAqnGj4TScBHuRWu3qyFIrhbw310Jac8 /YLka1vJoUMNpmBqxBT4eI1E/3KuGElE+Mt5eADRQ3W1mQw/1/uaKJTnRQftmq2BTYVx sg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vrn5gkjd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:07 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T0086873; Mon, 19 Jul 2021 06:28:06 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eelXm0hLM0f7Mc6+OnkUUUDIwADeiVnq8PbRyVoirGfyarNow7ir1OiXW8gCUgN4MSPAzOFwIIz0n43LdrQNZKOLU5uFqJra1XSKvZuN1X0qL3bAGOcoB2Xmm5zYEjU/1+y0SKaPhoXzaR24G5zgO3I16DbKv2rG5htMqLbnlSafjxHSHxn0EcyUaSkftcVETvElYpriNxB9aoLn7dTjMTfv3sL/swmCYPMEdAb/O4oAlmbkhtfNqf8QhnnWByL7LNVO8Qa8rFMCaLMR9iovnMT4eAkH0UzRQMSnXCvK68eYe3E1amtsTvOZN8EFkAQroTwg8jGZG1K691AaIEf8vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xikYxdDBMphjgwUL3W8+0CRy9R1jKeEBwNoTvL20p28=; b=abqgY9AIagstfbL4JkEPBcaficF1GyWPs1W+lQbwVkI/IK+UU4U1YUCH04VEL9T2ZjVpuMXKChr4Dvm3J64O2R36YSBj5fTRCzQR7/Ce4b+TT/RNyMfqw4RY8ZOHTh6htOwEWGvoExzp4ptAY8mqPP/A2/IB+gNGxKiduGzYVx36vF1elpRVXVbRp/Uz4JRl3XVZs6OMP+M2QttKfxcDHHMzKzh7p6NYccEQtTFqQnNYP3vfdR/04og7IyjlfixCrLaNNj4oETi8h4lYTEnDQt8Dl1lxYxLsNIUE8aSdlgLh3O1gEzXLEzbpdVlHZYUScf2yAQkPLQKNKBGsO8dmXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xikYxdDBMphjgwUL3W8+0CRy9R1jKeEBwNoTvL20p28=; b=zLzyp7g8UQngK7yzFjpwszrj7SRjwFkKDH4DpHDf5elyxcw6mtSyBFPKD8pXif16pthhpAnn6LC8MGe/WXSpIMgIq+FSenv/ecgWTR/1OfVPCgBNY1b+eGVEDdGXa8jqIcQsF/A9ZYqA+6TNnJTPQvxY5hhFYipglXzS74H1R3I= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:05 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:05 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 02/19] vfio-user: add VFIO base abstract class Date: Sun, 18 Jul 2021 23:27:41 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:04 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c7e294cc-2b53-4931-55f9-08d94a7e5dd3 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2582; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7KJcxxHZ42wUMSJNXwnn0UQa9NkhKfnAZxT7++fv39q150KzxWKd5wlFZCOzBlQGXu+3JE60s3ExUx3VS5wx6IWxYBqyyqjJ0bTVh4nZFFHvjvlGTpE5LUeOd+CI3vFzogVmGNHsDsEkcGM8Q9ybeZ88FQ2Pokq+9m4WWRb+TWj2TmVjhLWFYqjJ8cY9d5tHEU0psS8ZZX5nexutMiKN1hdwSiZjYJev5RiVvkNeKle++BNZ3ODoQZL3L2MrCU73bLyLTMDqTqpPVYsQB8TAOFI8QOoYm7wUFUDnX72Yba4HODJY3yTjYKCj1/sabR5IW/SLNt485dmm8T+GfsmQgdPYkGJcgS1bh0rtMUlZpWofHtyAKSaP/FuVpUC0n0ec/KI1s0Qy5Jv8JGJibZq1+X5G79Mr1N+VpQdasCUiabB77T7ETZG9FiuTeX/GRuVnCs4ntLyRf6lr6QtE3t7Lvxl17slk2ieNEX/F4pf7wkjnEjR5bz9WWWQjTnUd24oB3v3gT0QUoFACVNjLHcFtHAG6MVlKn1rFU2M1ZCqgmecV3pRWkFFKsJiZvbdOs+LU5L2LdbEMxLZ9iG+j1Q/R/K0mAkyRZrh5wZv5R4eFMv3Lpu0Esf8hBmcBZmUUrW1Ayijif+gnkvzrnQPJTM1iAA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: /6aoCFs3VaScgOoWG5+RHTKDNRg5pznbdRPAApczWesVEK1lsPj7Jt5hBmGbIzrbIXEddDSAlGuHmYBkiIH8fhsSAEWbdRqfRR1b27s4nmttOGjaLDRmILDf6wMeVO2NYu0O3AipoAlSx8v0ETxSRZxl7xv2GDW9bLVfhFdSGwXxHkvPqPFcwJJbPsRejgkQ0kR/piYiMSX6YtzijUI4c6G0sUmMSe4InJIruxK+CP3j1o2J4waz2r/mVVxr0RgIyQWnRu3d9cPlAZT3C3LeZ+Wx8FibTSERjUlEcyk3eXVOhIygL99Te5enSrQRUSl6QO6vGlGbg+WD97tTlzmgxva92HAbmbEAuB8ev2C24ija7kQFOYToB4vQSThBWuzqRB9zZvfZUDhh4zNMYIvql/Vw5ZkLLpJHxrWJ/SiP2MDkrWyeGrfygrI+edc/XM3JFDFst0uHGjuxxpEzXdC9xM6q+/8RxeOkBvi+gjqfZYCxYhdDgg4lWvEOeyvWI6gewgRoD1Rxusc71rKl2OrI57WoZgsLLMcw5nxoT0Ji+7To+5Ks2Bom2K+7o7Bw3cotSjZQvzT4r+RWkwFcGQkt90nt/NT8z2m/nbfHPciv/HrbIsu1xNux1EDk7IZQTZK9xO9W4LVbNfYxicNlw/f1oIOgO8DCAmlcWlb9mO9Olzgg/XM+zExJ33na2A/RPqhdXP0ydjNiHyyZ0RRpzLe5LpyuAsTOGPshw+pDE8uCSlMsvHT4vy9O2/2befrffQCtt+HUV2pd/QFICST5eWOHxStvx1mfCAE7pW/9jeLloUJB/pJ9wBocoNFZFM4YiBoTfqzXlNLbcGX9t6FojLol0W3H6RT1U/DngFsFYXrfOdIK3JiCP2AbXOI3kKlnjafGvVJE3pvBg3lHuUDdd4butQnsoOZj1e/9j5hFcgdN1B43Q4BBa6G0U/C49RpbIcPXpiCr/9eTJZMMRxIpwQmmGn5/CMF+FP00JPrf2N10BNcFRw3qMmUJQoO/JZInoqiaiRaww1aaDkTruMdmPlj1iYz9jR0VZprVkqojcp+IaBAbaWDSO17J0YeCJWHaAha9456Q+qGLO6IVpCJXQav17IsiaXU/rtiN5wteMEnCXmTu1JFm7QavJaogMbtg2kx3vOqPRTMLHtyapGZR1M+XhDNbON4yYrDNqtAJr5rXRrOWTt/kW5gsxN75O+ZPUUMXtCyffgq+zIMIlcnmeW3WoJmTKX1ajj97mALw9WTe7a5/wv0pyPZ2geGH1md6/VvxwthAMICI8pCUs63K6OXsVvPepOo4/pcrWfrbImHEpxKD1oT0NQOzQONm5eUKcZsODENFZWIKJP3h9UTwk4CtZamYoa/868tGsdwti+8UFc65kQ/vXpgINCGwt5O33qvk X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c7e294cc-2b53-4931-55f9-08d94a7e5dd3 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:05.1121 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: etS1tMTIwGOiM/pKB0UPjTsYkdsXqjA1dyDqMCJ8XyyzJjjufHbmIc7soqINk+tfiSZfP00WfpZDAVmzw5gVhOiAEhdH3nPZzIdIN5bQFYA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-GUID: HZt7GIvIrfYIFqtCsjMWfQAlco5Z8ZtP X-Proofpoint-ORIG-GUID: HZt7GIvIrfYIFqtCsjMWfQAlco5Z8ZtP Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Add an abstract base class both the kernel driver and user socket implementations can use to share code. Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 25 ++++++++++++++++++-- hw/vfio/pci.c | 63 ++++++++++++++++++++++++++++++++------------------- 2 files changed, 63 insertions(+), 25 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 64777516d1..ba2f51d98f 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -114,8 +114,13 @@ typedef struct VFIOMSIXInfo { unsigned long *pending; } VFIOMSIXInfo; -#define TYPE_VFIO_PCI "vfio-pci" -OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI) +/* + * TYPE_VFIO_PCI_BASE is an abstract type used to share code + * between VFIO implementations that use a kernel driver + * with those that use user sockers. + */ +#define TYPE_VFIO_PCI_BASE "vfio-pci-base" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_BASE) struct VFIOPCIDevice { PCIDevice pdev; @@ -175,6 +180,22 @@ struct VFIOPCIDevice { Notifier irqchip_change_notifier; }; +#define TYPE_VFIO_PCI "vfio-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOKernPCIDevice, VFIO_PCI) + +struct VFIOKernPCIDevice { + VFIOPCIDevice device; +}; + +#define TYPE_VFIO_USER_PCI "vfio-user-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) + +struct VFIOUserPCIDevice { + VFIOPCIDevice device; + char *sock_name; + bool secure; +}; + /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index e1ea1d8a23..bea95efc33 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -231,7 +231,7 @@ static void vfio_intx_update(VFIOPCIDevice *vdev, PCIINTxRoute *route) static void vfio_intx_routing_notifier(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); PCIINTxRoute route; if (vdev->interrupt != VFIO_INT_INTx) { @@ -457,7 +457,7 @@ static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, MSIMessage *msg, IOHandler *handler) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector; int ret; @@ -542,7 +542,7 @@ static int vfio_msix_vector_use(PCIDevice *pdev, static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector = &vdev->msi_vectors[nr]; trace_vfio_msix_vector_release(vdev->vbasedev.name, nr); @@ -1063,7 +1063,7 @@ static const MemoryRegionOps vfio_vga_ops = { */ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIORegion *region = &vdev->bars[bar].region; MemoryRegion *mmap_mr, *region_mr, *base_mr; PCIIORegion *r; @@ -1109,7 +1109,7 @@ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) */ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t emu_bits = 0, emu_val = 0, phys_val = 0, val; memcpy(&emu_bits, vdev->emulated_config_bits + addr, len); @@ -1142,7 +1142,7 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t val_le = cpu_to_le32(val); trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); @@ -2782,7 +2782,7 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) static void vfio_realize(PCIDevice *pdev, Error **errp) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev_iter; VFIOGroup *group; char *tmp, *subsys, group_path[PATH_MAX], *group_name; @@ -3105,7 +3105,7 @@ error: static void vfio_instance_finalize(Object *obj) { - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIOGroup *group = vdev->vbasedev.group; vfio_display_finalize(vdev); @@ -3125,7 +3125,7 @@ static void vfio_instance_finalize(Object *obj) static void vfio_exitfn(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); @@ -3144,7 +3144,7 @@ static void vfio_exitfn(PCIDevice *pdev) static void vfio_pci_reset(DeviceState *dev) { - VFIOPCIDevice *vdev = VFIO_PCI(dev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); trace_vfio_pci_reset(vdev->vbasedev.name); @@ -3184,7 +3184,7 @@ post_reset: static void vfio_instance_init(Object *obj) { PCIDevice *pci_dev = PCI_DEVICE(obj); - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); device_add_bootindex_property(obj, &vdev->bootindex, "bootindex", NULL, @@ -3253,28 +3253,24 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_END_OF_LIST(), }; -static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) +static void vfio_pci_base_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); - dc->reset = vfio_pci_reset; - device_class_set_props(dc, vfio_pci_dev_properties); - dc->desc = "VFIO-based PCI device assignment"; + dc->desc = "VFIO PCI base device"; set_bit(DEVICE_CATEGORY_MISC, dc->categories); - pdc->realize = vfio_realize; pdc->exit = vfio_exitfn; pdc->config_read = vfio_pci_read_config; pdc->config_write = vfio_pci_write_config; } -static const TypeInfo vfio_pci_dev_info = { - .name = TYPE_VFIO_PCI, +static const TypeInfo vfio_pci_base_dev_info = { + .name = TYPE_VFIO_PCI_BASE, .parent = TYPE_PCI_DEVICE, - .instance_size = sizeof(VFIOPCIDevice), - .class_init = vfio_pci_dev_class_init, - .instance_init = vfio_instance_init, - .instance_finalize = vfio_instance_finalize, + .instance_size = 0, + .abstract = true, + .class_init = vfio_pci_base_dev_class_init, .interfaces = (InterfaceInfo[]) { { INTERFACE_PCIE_DEVICE }, { INTERFACE_CONVENTIONAL_PCI_DEVICE }, @@ -3282,6 +3278,26 @@ static const TypeInfo vfio_pci_dev_info = { }, }; +static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + dc->reset = vfio_pci_reset; + device_class_set_props(dc, vfio_pci_dev_properties); + dc->desc = "VFIO-based PCI device assignment"; + pdc->realize = vfio_realize; +} + +static const TypeInfo vfio_pci_dev_info = { + .name = TYPE_VFIO_PCI, + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOKernPCIDevice), + .class_init = vfio_pci_dev_class_init, + .instance_init = vfio_instance_init, + .instance_finalize = vfio_instance_finalize, +}; + static Property vfio_pci_dev_nohotplug_properties[] = { DEFINE_PROP_BOOL("ramfb", VFIOPCIDevice, enable_ramfb, false), DEFINE_PROP_END_OF_LIST(), @@ -3298,12 +3314,13 @@ static void vfio_pci_nohotplug_dev_class_init(ObjectClass *klass, void *data) static const TypeInfo vfio_pci_nohotplug_dev_info = { .name = TYPE_VFIO_PCI_NOHOTPLUG, .parent = TYPE_VFIO_PCI, - .instance_size = sizeof(VFIOPCIDevice), + .instance_size = sizeof(VFIOKernPCIDevice), .class_init = vfio_pci_nohotplug_dev_class_init, }; static void register_vfio_pci_dev_type(void) { + type_register_static(&vfio_pci_base_dev_info); type_register_static(&vfio_pci_dev_info); type_register_static(&vfio_pci_nohotplug_dev_info); } From patchwork Mon Jul 19 06:27:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 831F5C636C9 for ; Mon, 19 Jul 2021 06:29:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E7C6A61165 for ; Mon, 19 Jul 2021 06:29:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E7C6A61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57390 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mmb-0006lM-0Z for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:29:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33336) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlD-00042x-6a for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:15 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:28452) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlA-0002EF-4H for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:14 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GLnv029074; Mon, 19 Jul 2021 06:28:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=IdpYrI33Psaru+b+HcRdQyutB3piV2brraP7gYI/c8A=; b=wBLWP6sbzDIDpL/5xRKKU0fRNdTuk5WST6lAt1ZN6UHR80dGu7ZvTk3laW7IU14F9P5j bA9pGErx89BnXMjelPiXTzfuEBNdv4KX+8F5ZKjJMtscs08tdCKBtjvSeIf1vw9IDeqO 9537hY7z8Diwu+Z+vpKV0bCz/sv8LKZPO15vqO1E/kF0y8LtEt6IP+7G2AZWtPSYe3Bh Hqjuv9sJWGti0GZaF0wGvqU+ZDrcFBXTBUr3B6BK87Y80zuDsqeGz1jrPaXjoCQhmDiw SJiiLRWFPT668ix7vNzOb9h8Ujy46Eq8ynd9YsP1lPt8Ah28nOJzsIxeM0vsi+9vvlIF 3w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2020-01-29; bh=IdpYrI33Psaru+b+HcRdQyutB3piV2brraP7gYI/c8A=; b=B5GnfZIu7dntNZRWGAjPbDvBd4XgYF3ekCqpYAWJsi1ChSft/DIXq8vU3UH007Fz0tCj 8o6V5k08EL+ILTaqPVdm5o2F0Y3eAus9y7VmaabdhYC5SYaur4Qiw4/5GvtA1zOLxECQ m8AiHY6FeqqU/sKRbyy4wUhuMTdsPVoH8Q9qRrgawLsrutRGLg9Jz/6PhJYp8XSVNlkh OCxp8XHfKjZeKf7NE/Rb34aPxvpEa++D21E+K4d3P+tiyYaMbS0Rjzq4Hb1lRWPeGqkH ccNHFKDqxjUIeKygHULdPgcRIiX1DlJNcg5OKfgZSdP1n4uIfGFmFnPYICpv6BCJMm3R lw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vpkwrnj9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:08 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T1086873; Mon, 19 Jul 2021 06:28:07 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CbCWHEizAxx/LjNCCizf6FAWLXloCswE1FrM0ZX5ncxABQiHPysMRRODBDKeCYmYzpw/5bhPLZaBvGMdhSSrYIdmY8Vfa/146i6NfgiTHGMK9tOW7KbFsVHaNI0pcxbuZSosVslHwT1YtBw6qjgjWsn/CjEfToM6WR9HnCpoHbw8DJh1hVCbD+/kNbMPZ8HXGjNDOES1dVa9AV3rz0q8/TFnr38mrpcYykyTN8WSen1gf7fbtAOuddw9s6BlcsQKADf8O/EF1e2JPZAvMLX3nwdyAf6NmUymb713qev0+QsvPLPB+IDJv8s6OCmHO953SJsjIrnPkbuOZTnjwjavhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IdpYrI33Psaru+b+HcRdQyutB3piV2brraP7gYI/c8A=; b=Z21D0x9BN/QL4pUyPxSE1we4WifPYrjRv/lN91pOFxnvaYrI6VjF+A64gPgZxXI97ehe/9aUt4NF03o6A2/KA57eAQdWtOdScwlonxahMNhgZjVfrz+hSvZvcIhsyqJ109LpEkoi/c6+rxnJ1Sxx2d6z84U0/Ijr2Ao6E0hBuW7nKicNCeBPOB3UDuVdGNUDqcR48CZb/wn9ISV1xERKz86uFbeRBh5eUGzYNOv/LctPwPwl3AgiXKUlleQHF7Z74KKnHljq/U+vDfGMXTSnSzhWiwMSXzzxYXAiIxLc4C0ABQI7rGRmL4lJa/hZIVO019EDrU0WjgVHzVfD1QG4+Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IdpYrI33Psaru+b+HcRdQyutB3piV2brraP7gYI/c8A=; b=WA2ozn9kM9UHrzBXZHkmS1S+PjEjdRbQys43nCyuOA0OEwpLmVLGE6V3qfBv9BgEDlMutD+miuXipPZgDJGmz/CgQPoeIdtKEF05oJax8UD8mqMDlESXzch6r+kEuEx3o740T6SHrTuveM81GgiFg2tSwbv/IXuetlBc2hLl+ds= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:05 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:05 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 03/19] vfio-user: define VFIO Proxy and communication functions Date: Sun, 18 Jul 2021 23:27:42 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0ed30e42-8392-4291-959c-08d94a7e5e1f X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3631; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 46a7oJGeWfv8faCHW79MOk7PUzOZviM4u4OlSptsWERiIRziOjBhTMEBqZkl1YH/W7O4PgiV52tElyYMZgwso1LMk1BRYlkaLS5cbnGXSBWcKoaZl7H0vPQpULn4UPVPHXw3ReKinZynzCB+YXpyNimINxBc9vlI2DwMb+iQOE9nT2++rD+P9NW/6DqX+4AptuQMSUCXpjfJrpBE3bZcqaUOvhLdWlkXv73Mt7SeAGn56gAYBeHZfTVhrPGsZyinwVvwHLTudpoao1/GA0WRXnIYKK4MitS1s9rvMUpCsG797NdPVXZyMYWYVkkGcJN9JNBl5kK6mMD4U1BEG5Hlcyfhjs3bBZHztT8JUBFMeFLXpgxuu6HBpDxKeG7c2Fe0s01Em+dFAm/K7FPt5qO+FXE/7XpVWSbDfDGs5QXYbePUkVUEfYdnzS/ikRgoYfyjF1cyJe/2wCYdUM6ZVlcshMUV2q1p7i/FNsg30PwVYftSs1q1N8eiFaF4nuCjjBrTyS/Gq8GgAz+y3akApjcHzWMmBrxpBkV1FjQmTd5dXdaCWgVDALwZqnzBodh9VCQtqADUgdSkIWLLDoBvpCmWrel2MuhlqKKx0S+GS1iqH9orAkCkF0PlgQ0PzJj+jM9/lehYWsL6SRyBpzmQfMyukg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(30864003)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?kADoCvsmhFwc8lbEVHeNpXLIn/sn?= =?utf-8?q?wwyh2zzbBDbe9qVRSgw7WxrcI8DeMOKebBGVK0eFUKV1TGNgJAybCbt/uEqY0uFvl?= =?utf-8?q?yr7pQggC9ou+/+I7nlqwjk0hM5KXraT6zti10BuzW7SO7F1MoH3hmfLMSS35b1kDF?= =?utf-8?q?741+BGJmqw2ntpGc2xWFPKnAgMeOgPoKUiezSR8aIvX++kJajxLmAFnrcnHhxhtWe?= =?utf-8?q?bKOnFBmxKxQUkCwVwWnQLsSJSBxVHyBm6xTuX8nbmCGC4km4JezXA8Krf75Wfp5+s?= =?utf-8?q?bsAYngEabwKtVLyPSpT8kRc+sPrGPeIR9yZl4RqnSZDUSxPEmuGxjeVrhaecb6unZ?= =?utf-8?q?Lr8t9IOJX3cmeaWTqS0PDBPQh435slr3pcSc+v9hKkx+5RsN0HqigTo3uXIRUF28V?= =?utf-8?q?atmPxw1Cjc2OW/EeeO2MCVg6yZr8pMshrH5ioCkoPZDgCHJYBeLLQIpQ8iMM/vCLs?= =?utf-8?q?0de8WKxmABgIM9MqiSW4B0zzwwsaRvwElcrkAf7ATmOJNeacueF3CbEgFJf422CCM?= =?utf-8?q?xFb7I6RpaZlqSFvDZQkpB1MXO2FutyjJPtoeq+LGDJhElNueA+KbnzSgrIWCbwdN8?= =?utf-8?q?HnLyr8qvROTKNJOU1U9qjGMP7pFNGGuAL8jP5ZZ2NlCggu6rDVDvIeqeH7C9rwkGv?= =?utf-8?q?VVUU54JrmpFyAOGFtBcLd6s3z5o1ZwaYw6jofhIWwFpzVmCqre/gRyFhjK08jjmzW?= =?utf-8?q?HDe+qu8jOJyKQClBuL+J9SC4D1v31Bv5bsjXc2eNxZ0IjWDWbegXkoGQkPW73zd8I?= =?utf-8?q?lmUr+b6VkjzsQ6nqElT45oAFbgw+bri+UJog5Mr6XwRTXcsJC/7hbRuqyytOCuUfC?= =?utf-8?q?Tx15inPB9NDTNrz91BsdQrbKUycrVoGOPNPsFgsWuhgzv3CU3B65rtRXdYmD8Xq+c?= =?utf-8?q?l35l6Nje0vTqy62h3/Ll5ltBgH1thbuS8mOg2jxOs9g7z/HqYWvwgooMyJPues/AZ?= =?utf-8?q?h6OHj57/JDGqpScnJiduk04sYW80OMqZ49IAzdn7dOI5hR14hh0XR9w0BlxDFduRT?= =?utf-8?q?FQt7oY8IwY7kTtZr5SCvnsf1G7k8wZRcp7Kvep1hooQPH6nxgMxr4fKvKU7yXa2XV?= =?utf-8?q?BRt8Tv5I+WAKjJ7oVlqQmWYzWeQww6T6NvAxNccB0KPHqdhbMdu4AeqFDiLkJTBRS?= =?utf-8?q?ZJ1Su2Cxi/0prkY1Rl5EMdIo01chWoyY2xvGJjyvXRmm1JoP6KrIy1JhYza2HauZD?= =?utf-8?q?j6LhbM+LPoVg5xx1lHMjJ9E/35HP7/7j1Gox36BuFtZoumlLZUMJsN1x5FduPObiM?= =?utf-8?q?DjKjxVrzHQqt0SjoQN1KsZngGvFL13SfR90+8ylzCZVFHsrWCWCxQ/hOStP0NNEtl?= =?utf-8?q?skH2IdlEjuZphVD?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0ed30e42-8392-4291-959c-08d94a7e5e1f X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:05.6018 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Q+b+jz2jvGh26MYLDIF3TNI3K/rHiT1bEBPVIyc5LAecMIMHPqQvtTszLY70hwARnErcgaZxFrLGTriOuxq/Vtrm369gWCtSHkw76wjDhwE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: 3uRxCo-L4u69AryTL53hSUigq3UWbk32 X-Proofpoint-GUID: 3uRxCo-L4u69AryTL53hSUigq3UWbk32 Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Add user.c and user.h files for vfio-user with the basic send and receive functions. Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 120 ++++++++++++++ include/hw/vfio/vfio-common.h | 2 + hw/vfio/user.c | 286 ++++++++++++++++++++++++++++++++++ MAINTAINERS | 4 + hw/vfio/meson.build | 1 + 5 files changed, 413 insertions(+) create mode 100644 hw/vfio/user.h create mode 100644 hw/vfio/user.c diff --git a/hw/vfio/user.h b/hw/vfio/user.h new file mode 100644 index 0000000000..cdbc074579 --- /dev/null +++ b/hw/vfio/user.h @@ -0,0 +1,120 @@ +#ifndef VFIO_USER_H +#define VFIO_USER_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + * Each message has a standard header that describes the command + * being sent, which is almost always a VFIO ioctl(). + * + * The header may be followed by command-specfic data, such as the + * region and offset info for read and write commands. + */ + +/* commands */ +enum vfio_user_command { + VFIO_USER_VERSION = 1, + VFIO_USER_DMA_MAP = 2, + VFIO_USER_DMA_UNMAP = 3, + VFIO_USER_DEVICE_GET_INFO = 4, + VFIO_USER_DEVICE_GET_REGION_INFO = 5, + VFIO_USER_DEVICE_GET_REGION_IO_FDS = 6, + VFIO_USER_DEVICE_GET_IRQ_INFO = 7, + VFIO_USER_DEVICE_SET_IRQS = 8, + VFIO_USER_REGION_READ = 9, + VFIO_USER_REGION_WRITE = 10, + VFIO_USER_DMA_READ = 11, + VFIO_USER_DMA_WRITE = 12, + VFIO_USER_DEVICE_RESET = 13, + VFIO_USER_DIRTY_PAGES = 14, + VFIO_USER_MAX, +}; + +/* flags */ +#define VFIO_USER_REQUEST 0x0 +#define VFIO_USER_REPLY 0x1 +#define VFIO_USER_TYPE 0xF + +#define VFIO_USER_NO_REPLY 0x10 +#define VFIO_USER_ERROR 0x20 + +typedef struct vfio_user_hdr { + uint16_t id; + uint16_t command; + uint32_t size; + uint32_t flags; + uint32_t error_reply; +} vfio_user_hdr_t; + +/* + * VFIO_USER_VERSION + */ +#define VFIO_USER_MAJOR_VER 0 +#define VFIO_USER_MINOR_VER 0 + +struct vfio_user_version { + vfio_user_hdr_t hdr; + uint16_t major; + uint16_t minor; + char capabilities[]; +}; + +#define VFIO_USER_DEF_MAX_FDS 8 +#define VFIO_USER_MAX_MAX_FDS 16 + +#define VFIO_USER_DEF_MAX_XFER (1024 * 1024) +#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) + +typedef struct VFIOUserFDs { + int send_fds; + int recv_fds; + int *fds; +} VFIOUserFDs; + +typedef struct VFIOUserReply { + QTAILQ_ENTRY(VFIOUserReply) next; + vfio_user_hdr_t *msg; + VFIOUserFDs *fds; + int rsize; + uint32_t id; + QemuCond cv; + uint8_t complete; +} VFIOUserReply; + +enum proxy_state { + CONNECTED = 1, + RECV_ERROR = 2, + CLOSING = 3, + CLOSED = 4, +}; + +typedef struct VFIOProxy { + QLIST_ENTRY(VFIOProxy) next; + char *sockname; + struct QIOChannel *ioc; + int (*request)(void *opaque, char *buf, VFIOUserFDs *fds); + void *reqarg; + int flags; + QemuCond close_cv; + + /* + * above only changed when iolock is held + * below are protected by per-proxy lock + */ + QemuMutex lock; + QTAILQ_HEAD(, VFIOUserReply) free; + QTAILQ_HEAD(, VFIOUserReply) pending; + enum proxy_state state; + int close_wait; +} VFIOProxy; + +#define VFIO_PROXY_CLIENT 0x1 + +void vfio_user_recv(void *opaque); +void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); +#endif /* VFIO_USER_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 8af11b0a76..f43dc6e5d0 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace { } VFIOAddressSpace; struct VFIOGroup; +typedef struct VFIOProxy VFIOProxy; typedef struct VFIOContainer { VFIOAddressSpace *space; @@ -143,6 +144,7 @@ typedef struct VFIODevice { VFIOMigration *migration; Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; + VFIOProxy *proxy; } VFIODevice; struct VFIODeviceOps { diff --git a/hw/vfio/user.c b/hw/vfio/user.c new file mode 100644 index 0000000000..021d5540e0 --- /dev/null +++ b/hw/vfio/user.c @@ -0,0 +1,286 @@ +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include +#include + +#include "qemu/error-report.h" +#include "qapi/error.h" +#include "qemu/main-loop.h" +#include "hw/hw.h" +#include "hw/vfio/vfio-common.h" +#include "hw/vfio/vfio.h" +#include "qemu/sockets.h" +#include "io/channel.h" +#include "io/channel-util.h" +#include "sysemu/iothread.h" +#include "user.h" + +static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER; +static IOThread *vfio_user_iothread; +static void vfio_user_send_locked(VFIOProxy *proxy, vfio_user_hdr_t *msg, + VFIOUserFDs *fds); +static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg, + VFIOUserFDs *fds); +static void vfio_user_shutdown(VFIOProxy *proxy); + +static void vfio_user_shutdown(VFIOProxy *proxy) +{ + qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); + qio_channel_set_aio_fd_handler(proxy->ioc, + iothread_get_aio_context(vfio_user_iothread), + NULL, NULL, NULL); +} + +void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret) +{ + vfio_user_hdr_t *hdr = (vfio_user_hdr_t *)buf; + + /* + * convert header to associated reply + * positive ret is reply size, negative is error code + */ + hdr->flags = VFIO_USER_REPLY; + if (ret > 0) { + hdr->size = ret; + } else if (ret < 0) { + hdr->flags |= VFIO_USER_ERROR; + hdr->error_reply = -ret; + hdr->size = sizeof(*hdr); + } + vfio_user_send(proxy, hdr, NULL); +} + +void vfio_user_recv(void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOProxy *proxy = vbasedev->proxy; + VFIOUserReply *reply = NULL; + g_autofree int *fdp = NULL; + VFIOUserFDs reqfds = { 0, 0, fdp }; + vfio_user_hdr_t msg; + struct iovec iov = { + .iov_base = &msg, + .iov_len = sizeof(msg), + }; + int isreply, i, ret; + size_t msgleft, numfds = 0; + char *data = NULL; + g_autofree char *buf = NULL; + Error *local_err = NULL; + + qemu_mutex_lock(&proxy->lock); + if (proxy->state == CLOSING) { + qemu_mutex_unlock(&proxy->lock); + return; + } + + ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, + &local_err); + if (ret <= 0) { + /* read error or other side closed connection */ + error_setg_errno(&local_err, errno, "vfio_user_recv read error"); + goto fatal; + } + + if (ret < sizeof(msg)) { + error_setg(&local_err, "vfio_user_recv short read of header"); + goto err; + } + + /* + * For replies, find the matching pending request + */ + switch (msg.flags & VFIO_USER_TYPE) { + case VFIO_USER_REQUEST: + isreply = 0; + break; + case VFIO_USER_REPLY: + isreply = 1; + break; + default: + error_setg(&local_err, "vfio_user_recv unknown message type"); + goto err; + } + + if (isreply) { + QTAILQ_FOREACH(reply, &proxy->pending, next) { + if (msg.id == reply->id) { + break; + } + } + if (reply == NULL) { + error_setg(&local_err, "vfio_user_recv unexpected reply"); + goto err; + } + QTAILQ_REMOVE(&proxy->pending, reply, next); + + /* + * Process any received FDs + */ + if (numfds != 0) { + if (reply->fds == NULL || reply->fds->recv_fds < numfds) { + error_setg(&local_err, "vfio_user_recv unexpected FDs"); + goto err; + } + reply->fds->recv_fds = numfds; + memcpy(reply->fds->fds, fdp, numfds * sizeof(int)); + } + + } else { + /* + * The client doesn't expect any FDs in requests, but + * they will be expected on the server + */ + if (numfds != 0 && (proxy->flags & VFIO_PROXY_CLIENT)) { + error_setg(&local_err, "vfio_user_recv fd in client reply"); + goto err; + } + reqfds.recv_fds = numfds; + } + + /* + * put the whole message into a single buffer + */ + msgleft = msg.size - sizeof(msg); + if (isreply) { + if (msg.size > reply->rsize) { + error_setg(&local_err, + "vfio_user_recv reply larger than recv buffer"); + goto fatal; + } + *reply->msg = msg; + data = (char *)reply->msg + sizeof(msg); + } else { + if (msg.size > max_xfer_size) { + error_setg(&local_err, "vfio_user_recv request larger than max"); + goto fatal; + } + buf = g_malloc0(msg.size); + memcpy(buf, &msg, sizeof(msg)); + data = buf + sizeof(msg); + } + + if (msgleft != 0) { + ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err); + if (ret < 0) { + goto fatal; + } + if (ret != msgleft) { + error_setg(&local_err, "vfio_user_recv short read of msg body"); + goto err; + } + } + + /* + * Replies signal a waiter, requests get processed by vfio code + * that may assume the iothread lock is held. + */ + qemu_mutex_unlock(&proxy->lock); + if (isreply) { + reply->complete = 1; + qemu_cond_signal(&reply->cv); + } else { + qemu_mutex_lock_iothread(); + /* + * make sure proxy wasn't closed while we waited + * checking without holding the proxy lock is safe + * since state is only set to CLOSING when iolock is held + */ + if (proxy->state != CLOSING) { + ret = proxy->request(proxy->reqarg, buf, &reqfds); + if (ret < 0 && !(msg.flags & VFIO_USER_NO_REPLY)) { + vfio_user_send_reply(proxy, buf, ret); + } + } + qemu_mutex_unlock_iothread(); + } + + return; + fatal: + vfio_user_shutdown(proxy); + proxy->state = RECV_ERROR; + + err: + qemu_mutex_unlock(&proxy->lock); + for (i = 0; i < numfds; i++) { + close(fdp[i]); + } + if (reply != NULL) { + /* force an error to keep sending thread from hanging */ + reply->msg->flags |= VFIO_USER_ERROR; + reply->msg->error_reply = EINVAL; + reply->complete = 1; + qemu_cond_signal(&reply->cv); + } + error_report_err(local_err); +} + +static void vfio_user_send_locked(VFIOProxy *proxy, vfio_user_hdr_t *msg, + VFIOUserFDs *fds) +{ + struct iovec iov = { + .iov_base = msg, + .iov_len = msg->size, + }; + size_t numfds = 0; + int msgleft, ret, *fdp = NULL; + char *buf; + Error *local_err = NULL; + + if (proxy->state != CONNECTED) { + msg->flags |= VFIO_USER_ERROR; + msg->error_reply = ECONNRESET; + return; + } + + if (fds != NULL && fds->send_fds != 0) { + numfds = fds->send_fds; + fdp = fds->fds; + } + ret = qio_channel_writev_full(proxy->ioc, &iov, 1, fdp, numfds, &local_err); + if (ret < 0) { + goto err; + } + if (ret == msg->size) { + return; + } + + buf = iov.iov_base + ret; + msgleft = iov.iov_len - ret; + do { + ret = qio_channel_write(proxy->ioc, buf, msgleft, &local_err); + if (ret < 0) { + goto err; + } + buf += ret, msgleft -= ret; + } while (msgleft != 0); + return; + + err: + error_report_err(local_err); +} + +static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg, + VFIOUserFDs *fds) +{ + bool iolock = qemu_mutex_iothread_locked(); + + if (iolock) { + qemu_mutex_unlock_iothread(); + } + qemu_mutex_lock(&proxy->lock); + vfio_user_send_locked(proxy, msg, fds); + qemu_mutex_unlock(&proxy->lock); + if (iolock) { + qemu_mutex_lock_iothread(); + } +} diff --git a/MAINTAINERS b/MAINTAINERS index 12d69f3a45..aa4df6c418 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1883,8 +1883,12 @@ L: qemu-s390x@nongnu.org vfio-user M: John G Johnson M: Thanos Makatos +M: Elena Ufimtseva +M: Jagannathan Raman S: Supported F: docs/devel/vfio-user.rst +F: hw/vfio/user.c +F: hw/vfio/user.h vhost M: Michael S. Tsirkin diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index da9af297a0..739b30be73 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', 'pci-quirks.c', 'pci.c', + 'user.c', )) vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c')) vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c')) From patchwork Mon Jul 19 06:27:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 804AEC636CB for ; Mon, 19 Jul 2021 06:29:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EAD0760BD3 for ; Mon, 19 Jul 2021 06:29:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EAD0760BD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57434 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mma-0006n3-SO for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:29:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33346) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlD-000436-SK for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:15 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:28964) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlA-0002EO-Hl for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:15 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GOHM021843; Mon, 19 Jul 2021 06:28:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=hHnr9eIDXnnun8OjXcEjP7t0JygDmj9ZfCzR5QH8K90=; b=VWcBPXMB89XEfPOavij0bevjUVeGIdv8JmB5x82p3Gz0DjobCWWzRwVvT2Swbh4nCjj9 3l3lfnmTrCYDtA7lxHCm+R+y8InNzXSqYE7sUj+FAod3/OA2AEYxEluXLLvsShGQf1pt z4pRLLs8QKFrNWuChfPceSi19tgOMZptTi+f9/XGF0EI7wC8B2m6oogPvTKvqhfybQLJ LIVJw3cuKKc5JTSAB0zPm0susIWWt+IjKXRLk6w/luyvamu5Rm5BiTBc6wmK2yRkglg5 HnTPUWDsaKKLfjAbHX8OM8eg84Ayvrb4qyXkWaz3SE5T6hTkjyyTzt2OHwbkgKpZC86m Ag== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=hHnr9eIDXnnun8OjXcEjP7t0JygDmj9ZfCzR5QH8K90=; b=lRIMN3aesR/GLkxE+UWaCOFnUr6oAG0V2Ex1hlYhlA7T1DFL4y/soTaG1Q83uqb5/eaU CpQWQ87OzkWKLMFngI4A68F3BiDte7ZcCPFnMwmQnHYJYe/Q1tmzGpuSmfieygSjhbMv /IOBX22D8kh0NO5c38zlyPMmV1pWBZbJWI3XIjVYdYFYyCk/aCjXD/cG1JmzlmmSNa0P bfisp7pfqS4OLfd6Y1clAsqqb4HmVsnjbuKEu2GJc1TvBgv7stI2at8hmdMPeAC3Cl5j vBNvbW6bKoYhpjJMDSJ0kAPOts4dpLyHgsb+HBnN3JegvA+SMk/j434TxkB49cRgrEQ7 7w== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39uptrt6bv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:09 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T3086873; Mon, 19 Jul 2021 06:28:08 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a4hXN2ED7cFH4eA0oIYslkssAkyMB1BCP7WrtWZNdd3Qi513nIkpl0I2wook6wUuiTAzEL3BrZ1kr3OiuEvvleeAj6ScuS7J1h50EyR32TBEkcK8mkf10HDipKFPNTkcG2HKWkFfD588kozzl6ZvwokJgRf8idpoAMrlv2h2weLa3c1p5M00KEBlPvwIMxHUhv32fU2G+H9z8MkmOkb/rh37y1MT5MsPYdyCN4m43nRo8NTPo0JHpoXnhxvxrdt2K67B6vYGqeQIFgxpbEqWQU9+4uHoB4jrmtv5qxav/Fj22PU22rjqkqE/YMLBdXgI4k9xYIVTzOJ8+HFu1HTtLg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hHnr9eIDXnnun8OjXcEjP7t0JygDmj9ZfCzR5QH8K90=; b=Sx7QbnyWckQleBtRqOBwMoSDBAfWqpZa3z16sPEpbKi6BtMFJAqDsbxANbFUhgpobtdFlOvcOzOggxZLU6P1k1cf+tcL+whMTpmKrEhL6iBVBHsya0FBynHpLotq/P+Vx3ykLqcKYkxd30+7qULOfHentNW0XETm98+7m9uM5e+NA+G6BKhL46k48bBY4bm5NNYWAdyCF+3hBsfOzRDQ5I2lNHJpTMZ7PyBkQ2Zu30OoxGFLODSARcGGjyLq0p8nVVDwYGpNrtCRuse75eRzUq6pLPnX/9WZjY8iY36ZVyhhohrX3W+Rqqrk4DuFbWDrcJTFuxJL9Z/+NrWpgXu3Lg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=hHnr9eIDXnnun8OjXcEjP7t0JygDmj9ZfCzR5QH8K90=; b=gGCX4otY4QEjzuTpGvrc9xd/59hlC5VDxa1XEiSgbPLZxSNZV59siQSG8mzakrV+Xx0Ck2wlSlMt1zG6xH0H5rc1IxxLVr6xFeP7aCGdy3O9Ch8WHkkAry3iw5mAFEEohVxYaoYdWJgwixupjmwCk4u8+0H4N4dBOrw3hUuZPG0= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:06 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:06 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 04/19] vfio-user: Define type vfio_user_pci_dev_info Date: Sun, 18 Jul 2021 23:27:43 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8a1fe54e-337f-4f82-f338-08d94a7e5e78 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:556; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mfwdDQLHEDI9pwuES7EZcL97nK7hRMvOkxDod/Cl+dvk4EZZz1nykI8Xs1UBr/O3gZc61hcM/IcOJSG9UHOfPc4a84MuK5yZKLMgGKuCQvLYjJU+HPt7nkMf5sOYroFnob0X+LvqCGFZo1AIYhEnFYJ9Sz0fxjnhpoRck6RcW4xenLRMN8b3WUv6cSgWgAiT1J0u5gA71KAHNUoRMnt/1CQttNESIHJHI1bykXvx1sasliHzDtnrZAkGF7S7CF/szl7Jc2pt5Pf9Xi4Kcpc7BCx807bht32o3ZhNMKT3a3IRBaOn8FraAJGy3U/plfRy7i81fAAISqhxYKC7ONkjnUMAmeqE9cZaG1sO8IyBUxGnKByT2C2HxT6kNI/4jo3zy4mTh1IW0EK0ZIr3AeAGU5NPExonqK5V8Ho82uIOqRF7o9OoLbXHSGl5/7NRFrtbFjwlG81X40u2BqY/5QrwYFbyqI1PbGR1ndZta+WU9p0DPdsl8xwYcap69ei7/SqBu76UVUgiNU7t9SitJNBdHnazB4HroeYhBMvLoaPsqydhli0ERQnwnAMztsUmXThzM9q4WBY78PLEHBBPHUIb1Z9yIMTUx/kLR7Mq9du7Qe7sfY/bNTJfBvJspC8Yj/gxA2vWsGSQ72aNMbeqZ493Dg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: A55Rvi5/tsMs3pouuC/Kqj3Ei1RughhdDte6H9wNRG31t12yAx8gP0LIKZ1MCFIr7jf9hJ/71G4yFWXksVjWT9psq2zxq3YoQtyvUDxj1gzIujZ4vtweKltKoCq4TS20/SUNohN89Kk+BgLF7ILIN1LfjTtX7UkwLltRJIvSNgbFCtUmzTYk19T4G5EhvSRR3e9MHos6x70IPI5t2f5EFb9rljU3zKs4NvBCCpBb80PNfhPdoXT0X8hAvNQZWXzjV422R8wjOXzy9VpNqBkk6kCHjuWbzG8D6satqybMpmoO+sCP+LVafxs54ZVC4ZD0+IBfjIwBdG6rTEMsCXlYdz1Zcxbyp4z1F09nn+lRTnpqNzGqaxaaS6f9mp7ZNO806b05SaBXSfHpbcMKWoO8qRAhdhGyNinmGiDNvYuxPtwS2OhZxW4AYyttion/pF6gtU7yAsBpoHCQXRY996O2bNR2Zs9m+svEKEdXaAuEYjJcV3CZZAqAIR1Vy286KwRiugYKAX5Y4n+rkM3ubouyGU8bGHXNnIpkvGnttpenpSiiACpNjPDLNhn1bOzanVH/N67DFAFJs2reYzlQyDp3OXlycwR7U+HG6L/IQWTnuepwlMAoJKlqTqw/8Va6uB/jbm4OCVJL/H9rl/8j56W9qF0xZj78LYqWuxkTSA5d+r3DIs8HMAtVcM/Lh1C7x0DXWAqu67idh9MM5bUolzqoCwD+bsT7C0KHbb8R8TIAD8XL9Wu73g0wwj7irWWJtfpLBQ/qKZv3+mWe3IeOtHqUGC3atrwqM3R5QguhjSR2ICgSdoTpgo9mE3pnIO5nyJsgIiCwsBEpni8AUy/3jSSLrGNYm8jrr0rAADp+mgsoL8UYXJ+nsmbI9ovRHmH7emKUHxVBgi69NmLeCPYVHYPghSup2mffwF0HaAmFOwGe4TIxUuMObH8AIsyI4zpswf0ju9lavHCAEmAAxXEV+qjK9bEscBpd8nq7wod/Xymdfg0gNRElrkMrFdlX85ntJHZWKf6gI2gOJURAeoKMSgbwnernx4Luj2EfZsXVcwnStwDDLHCROliq/w7O5FWJNNFsFX+gZblsCDk4fBH+m0/opBnoFleVw8f8s6WDKBCnGoXEZTR1VF4+0NWJXEcNIMduR8ZffhnizxRLKArMKjalnWPTsx6YGaRa/gp86xGgFrDMlbUKNChDcPvV+hD78KBt5voWTBKpnTTAt3ghLK9fLOVIigoNCg+gq4M8f7b8idLhhE/3s7kXzl4+AuaqIAvEKS2pc88eWYdkXryXgzNPua0GAB0XfxSvPcEdrEWOXmia56cCIn0LD1y6myiRCxCiUUgds0nJzznVxj6u4BlI1oh4VIhXAklpqEovz9/FNXHiY6ZI2CTylmjMDl98x3dy X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8a1fe54e-337f-4f82-f338-08d94a7e5e78 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:06.2254 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: yeAczSbgsp67LpgPaG2K1e07pgSDFtg1QasjjIX9sLVI0SvAYDTNebo3soXnh2nE+pOklk/Y/RfM4CgY1uch1a1SZr36OenMMPgKJVxfgUA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: d-ubVUmEevvoFe6cXscfXtc5KJRI6ebJ X-Proofpoint-GUID: d-ubVUmEevvoFe6cXscfXtc5KJRI6ebJ Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson New class for vfio-user with its class and instance constructors and destructors. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bea95efc33..554b562769 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -42,6 +42,7 @@ #include "qapi/error.h" #include "migration/blocker.h" #include "migration/qemu-file.h" +#include "hw/vfio/user.h" #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" @@ -3326,3 +3327,51 @@ static void register_vfio_pci_dev_type(void) } type_init(register_vfio_pci_dev_type) + +static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) +{ + ERRP_GUARD(); + VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev); + + if (!udev->sock_name) { + error_setg(errp, "No socket specified"); + error_append_hint(errp, "Use -device vfio-user-pci,socket=\n"); + return; + } +} + +static void vfio_user_instance_finalize(Object *obj) +{ +} + +static Property vfio_user_pci_dev_properties[] = { + DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), + DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false), + DEFINE_PROP_END_OF_LIST(), +}; + +static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + device_class_set_props(dc, vfio_user_pci_dev_properties); + dc->desc = "VFIO over socket PCI device assignment"; + pdc->realize = vfio_user_pci_realize; +} + +static const TypeInfo vfio_user_pci_dev_info = { + .name = TYPE_VFIO_USER_PCI, + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOUserPCIDevice), + .class_init = vfio_user_pci_dev_class_init, + .instance_init = vfio_instance_init, + .instance_finalize = vfio_user_instance_finalize, +}; + +static void register_vfio_user_dev_type(void) +{ + type_register_static(&vfio_user_pci_dev_info); +} + +type_init(register_vfio_user_dev_type) From patchwork Mon Jul 19 06:27:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE122C636C9 for ; Mon, 19 Jul 2021 06:33:04 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7E8D060BD3 for ; Mon, 19 Jul 2021 06:33:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7E8D060BD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:40636 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mpr-00068E-KG for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:33:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33350) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlE-000438-2I for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:16 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:10432) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlA-0002Fj-HK for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:15 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GRg0009274; Mon, 19 Jul 2021 06:28:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=bGLK7zVgi4iGLxrUGVtYjpkeEwAsYIliWGVlWXJb0/c=; b=ShvpSD20egc41K1cn6djaVbb7+sETSEXXUbwB2Ce9AfuGDQY0NiOSxZ+2MUOEPrXrxhr FncHm34QA7e/aT1oFUoa+VetH5Vv2VZjoSbBMHcYRcKUyopcaupfXdxvjfA6rO0D8TAZ FKpNIUj0OrtahPy1Mn1Dhdp6rwKGmiWD0CmVS95WRrJYm7GPOpqAgptnJA4cbzlW45zd ZckkFQMSTZhSEQ8WsVu4V92pKamZGT0vrW7DPcN02lISZb4EMX8KOTw/m3CmPeFkRYQo SapPfU7HWV5kyvEcgeWdsoovFSlUvaaM+VjzU+DciFesz/d8yx99hdoVVXKZVAeXMcFp +w== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=bGLK7zVgi4iGLxrUGVtYjpkeEwAsYIliWGVlWXJb0/c=; b=B6wLf1TVqKcXrY336FQWq6CCrYT8A3q/ujKd3wkCKRbK3RdG7TIyoit6+MqJgCHRKw7k ivcDlep7L/XrpwT6WCWtbsORivqmi5bikzus1zBJNLs8EQs40ORPsDGUsPAKn26dfKfX lNjgNFst9/QkrgqIBPZrXqKS/+khAjMO69hwRw7QlBOjmorEkxsXhMmRZxThtiQ1O4UL tz0U5/cF6sqNL8BsRPzJvONwKXgueGGVu6j2GdXZEDEKPopvsXk5lirwGjYEt9kN7If+ 9U9qXFew/Ubh8+4hgD1lYKlqPiq8q+MLfT6tFJWvRvHyTGtGSW3sfA+0s31hY0DmFWaG 5g== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vrn5gkjg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:10 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T4086873; Mon, 19 Jul 2021 06:28:09 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XXS9hagRPhEb+AEx9Cv+iGvTLky6b190HkeDrEAvGBNCECcsiRb19NWZ72yHH7GXNUeUxagJ+s+JoelcmP7wdvfA33jRzMaIaWTwjjmTTacYs1Jmigo8H0adiJpFJqx8McDCm10PYi+EMgFt22sU3S7hlzcLKi6iBY5GymOnbgXD/xTCbRUZEgDGoiibmV2mhs2RNiYfAy9G8MOoQkORHvNiw7TWNoJ53nIWxAKVHqN+HWuvrdH0YdESh6MpABeH2VemFjxvk4t7O6qprd8Cgdi12LFzvt5fIqSCjjmpH9DLKCydXFntnZpg1kx7BlWI76W9Z5wC1UxshR/VVM1vCQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bGLK7zVgi4iGLxrUGVtYjpkeEwAsYIliWGVlWXJb0/c=; b=GTp2siHsKuxQ2bTFE12e4NmfB48y53pOkhcwaSklzi65v9jUyJ7NGLcD/d7Gq/V90S+r5Pb/w6d7U2ib2nn8SC6v5z6woknaP1p6x21quDx35MVFpGNGyKkKOWMj1IqcTWILrKr1SSbmBrU5BIvGx5bs7tcawGOPYP7KNZlkS3BBH7pQ2AnTgo8KS0aYUIq3l7DObmvBzNSZGqoN4lNRnTFke18RDFQH4MEGe/vpSkHTtFygQ9kKwYOVzbE9uZQJOLJDJbPjG0UdTLTfz59twsTf4spk1GTzSaUHRfUuvgic60KzPrdJxT3U1ZETO1aD30NP85HFoVMjc0vnSzqDSw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bGLK7zVgi4iGLxrUGVtYjpkeEwAsYIliWGVlWXJb0/c=; b=EUS05TllzbcccYHG7RSUV2tY9OTI/1KQuDY/42lBqWDkNzMiMEkus8ZOP3Xy+PpbvyCHC8h5TYgwegCYQnCPvttdI/ibz5L1Ym6ynygQMTXh5//T5A/BlmvUo54P3CfySWIEt9svR+xPdn3faZZfIH6tJ7fJxpHRizOMKF5vMIs= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:06 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:06 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 05/19] vfio-user: connect vfio proxy to remote server Date: Sun, 18 Jul 2021 23:27:44 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:06 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b5c082e8-823a-40c1-d873-08d94a7e5ec1 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:758; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6FaddS10L8qQWoGDtEiFcpbKrIEY2IGGeaY6efaLLlkdLwJgZZTcdPOh4yvnju8Wp9H4uZx9wfrZv6/rIKIwVk+TvfwYPj6z5GsTdW7a4OPsF48fzlaFjBGd1ujCZtghHKnc+G2V+ir8Zxwo+6jtaeE4c9ge/bIJzoOG7LpODuM4xYb+L+9r1vvBX5yGhXo+JQfybM6U+9ISio/2R1ehv4ONLVVF9oPuBwTg4PzaPeUCcqezcOCcr9cUZYqOVNDiceZk4HHkdR7ammQQmvsBzvD3xGTG5Um9nW3FEh8YLWGhiYCQVNoP3P+vPh6XmdHWVw1sQusN5LdrysQGi262XZPtOWnXkYSJZ2/t0MCYBA7VIWCbNWzZ2vQXBtJhLzlXJnk8ijafdX54TNwbm4xfdZoheyTXcScSkIAAl/xtB+UgN0eB4Ve4+BQ+DpqfC4OvGJsyKbtRfirMPGPM+fjJrz3FC6z9BRr0aDRsa6U4jo5LCQvxCmkWZILSzyeUNjMnFunvR20OSCk6/s2VdJrLlK4rRuj7FUYVFmmBeHjYzaCH5vlGA3mEBOBUM4o+IACVnq/KkaxD7YEwNXsG91BFOTKRBoxFaJvq19lIy1IuMajDG19aNl63oFrsf/yIUI4SBFobRO+N5/OJbh2ADmJvnw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4KHzh1bwQsIkhRlTMPb1wICtpHN+O9ivVytfM79FS52Xdz0a/bBKD3dp4zOnrqcnPYiYHyjbxN3mh5XW9+J4GYRZEktOqt9QkxiljYBi+FrRYV9mZpzgzYy8feYo37rtIh3myKMYLGOCJaySOyiFfmLTqJdRIEPrxticoddQMm7Zzd2U0lcnjvxTLwCCiagGJ1vQbbo1z/2ctkt2ngZHPP+3HhRSIyP+yQKCO6eRjqDrS6MfcfSgqpcNtwOmq9VxN6LY3jQaqUcl+rNkjih4/Di0VKlYka8zSz7rBNHct/cmc6g0Q4BP4be+uoNvftpaJdm8yunyl8aZ1XQBN6X8W+o7qYpmarh+rgHiu2afvFuzxti2feikm9T/p9blzZEIkxYKF8YxjvuaVtGzcMbwgiti2la8QMAB6dZsBnqmZr65WrkklDx3nE2mkk0BWkYpp3e94CZ24bBuYrcgiupfkv1sTLXAQIoFyKM8zdXIOUxVgZEoNyhlhaW3/LOtPWQ6HfrFm5qnosnsJF7bMtvmpm4u7RnTDetRB+CGJT78JbjGbrWlWqsIvkLa+XpHrFavcTC22MQLOxJEid+CwxPcMyUJfyelxDi/D9FvKYJsDb1WTezyHgrR51lWZe0urTj46W3PE2ZnjZ/BsPHNf89RKwBlcOklFWgvO16QJ/Odnq73lUcxXoziIQWLuRDI5I55/fmsm6alvu8SIGL7YsK+WRtkDBDd9mHUDL9kqoyxD50wpjZPIZRdpAwUxUNHRUoqLPCufomx+FgW3dkTHbMmiEY/WOOBYr7nbI+linrTT4dM+KimfcfJvLk9HqpC7O8r+FAeZHR6zwq9k1qsqQxSJx0NfX01aqHWvOfbid+wdtpls47pi332rApEiaCK8ekP9A8SqHi3hbHIkuRTSGAjE825OXNAblPLY03pUkYOAFJ0hijUNN8F8t68ukrvC6IL/ab3W1qRE1z3lelnR0cshdjHosmtXjDgPkfhAOh24Gu/BygjHH6PDpC+fUWY75O3zH0aCIsEvunrVYamFyjdMhdJ0wVEzjb2H1nUrnFxfVUxN6LLMRXPwnfBgOorK6nE4DM30RzTXOhSHb6OcK5hpr9l5V19F6QSA2QqDvp2k0CTlz2cAC2pxmkBwzQ2Ex2/W24WXXTnB8a49Hy7V/YwMuGP1cbXSx3jmWqsoLMaO0+M6Ke+7dRI5hjYjaBCglY5AklMlBVum8P+Jk1Wzw0hzMYeMQya1432ZgW+UB+H4x/cWC1wR/yYX9gw3OTNpFk22Xx8rOLKH+2aJ7AkPe6Wn5l78O3BbiBLOhwHtAexvp678R10dJLg+DUSJQqqNNc86xcOa76xBJGWDZqUbI+z7P5n4lJF01beL0gCQNsBD8w9hXMsLZ/Pw5yyMGSdvxyi X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b5c082e8-823a-40c1-d873-08d94a7e5ec1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:06.6952 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +UFDUDP+6/Wxlhjs9xlL/KHc4r9Hb1WWYfZiFHULBLuk6QZ6yFSOvXuFBy66M0D1260FeiaPzw/KVUTej7jQEMGM+IHDUfhUDhppOpzX+Wo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-GUID: t53pNtgqMkERLcBUikxcOF9h4ak9raOp X-Proofpoint-ORIG-GUID: t53pNtgqMkERLcBUikxcOF9h4ak9raOp Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 2 ++ hw/vfio/pci.c | 16 ++++++++++ hw/vfio/user.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 105 insertions(+) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index cdbc074579..12106ccb6a 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -117,4 +117,6 @@ typedef struct VFIOProxy { void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); +VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); +void vfio_user_disconnect(VFIOProxy *proxy); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 554b562769..1effdcd5c0 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3332,16 +3332,32 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) { ERRP_GUARD(); VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; + VFIOProxy *proxy; + Error *err = NULL; if (!udev->sock_name) { error_setg(errp, "No socket specified"); error_append_hint(errp, "Use -device vfio-user-pci,socket=\n"); return; } + proxy = vfio_user_connect_dev(udev->sock_name, &err); + if (!proxy) { + error_setg(errp, "Remote proxy not found"); + return; + } + vbasedev->proxy = proxy; } static void vfio_user_instance_finalize(Object *obj) { + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + VFIODevice *vbasedev = &vdev->vbasedev; + + vfio_put_device(vdev); + + vfio_user_disconnect(vbasedev->proxy); } static Property vfio_user_pci_dev_properties[] = { diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 021d5540e0..371ee9cd8b 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -284,3 +284,90 @@ static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg, qemu_mutex_lock_iothread(); } } + +static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = + QLIST_HEAD_INITIALIZER(vfio_user_sockets); + +VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp) +{ + VFIOProxy *proxy; + struct QIOChannel *ioc; + int sockfd; + + sockfd = unix_connect(sockname, errp); + if (sockfd == -1) { + return NULL; + } + + ioc = qio_channel_new_fd(sockfd, errp); + if (ioc == NULL) { + close(sockfd); + return NULL; + } + qio_channel_set_blocking(ioc, true, NULL); + + proxy = g_malloc0(sizeof(VFIOProxy)); + proxy->sockname = sockname; + proxy->ioc = ioc; + proxy->flags = VFIO_PROXY_CLIENT; + proxy->state = CONNECTED; + qemu_cond_init(&proxy->close_cv); + + if (vfio_user_iothread == NULL) { + vfio_user_iothread = iothread_create("VFIO user", errp); + } + + qemu_mutex_init(&proxy->lock); + QTAILQ_INIT(&proxy->free); + QTAILQ_INIT(&proxy->pending); + QLIST_INSERT_HEAD(&vfio_user_sockets, proxy, next); + + return proxy; +} + +void vfio_user_disconnect(VFIOProxy *proxy) +{ + VFIOUserReply *r1, *r2; + + qemu_mutex_lock(&proxy->lock); + + /* our side is quitting */ + if (proxy->state == CONNECTED) { + vfio_user_shutdown(proxy); + if (!QTAILQ_EMPTY(&proxy->pending)) { + error_printf("vfio_user_disconnect: outstanding requests\n"); + } + } + qio_channel_close(proxy->ioc, NULL); + proxy->state = CLOSING; + + QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->pending, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->free, r1, next); + g_free(r1); + } + + /* drop locks so the iothread can make progress */ + qemu_mutex_unlock_iothread(); + qemu_cond_wait(&proxy->close_cv, &proxy->lock); + + /* we now hold the only ref to proxy */ + qemu_mutex_unlock(&proxy->lock); + qemu_cond_destroy(&proxy->close_cv); + qemu_mutex_destroy(&proxy->lock); + + qemu_mutex_lock_iothread(); + + QLIST_REMOVE(proxy, next); + if (QLIST_EMPTY(&vfio_user_sockets)) { + iothread_destroy(vfio_user_iothread); + vfio_user_iothread = NULL; + } + + g_free(proxy); +} From patchwork Mon Jul 19 06:27:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38C8AC636CD for ; Mon, 19 Jul 2021 06:32:15 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AE66561165 for ; Mon, 19 Jul 2021 06:32:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE66561165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:38020 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mp3-0004KO-Tc for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:32:13 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33436) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlH-00048H-0F for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:19 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:11176) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlB-0002Go-5A for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:18 -0400 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GFh2024198; Mon, 19 Jul 2021 06:28:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=gQwXSGT1VJgRsGfyEUhNRTVltwlYSVVa2oUvkt+arFs=; b=eRX+qzupr3GexcRniNcUrLqSZGZCIZDNL6iCht3kr1JJtqZOnVgXbg2dfJeQD4nBH3Jz pObrH2KbVpHcoUnUTlTdrGMSQkcuEHetHclcjW40imI7Sz8Iq4SsHorrN1bDWHBRK1ni jPrRtXTwXgmWA/st8ss4aLSxKy8ODnFgPJ3Q30QeyBomoxGnVeIZxL7D3yB9wLnOVIjn KDyQ7l1YTwFc2utAnZCAaLfV2nAeWOzMxcPo+HW4mpo09V9ULe2SA4TsC88JWKDpu4v1 cr5OrvtPAlbB8bdf23dsXNJuAHf/mXArq594S8Jiais1F8Wi9Vm5k+fNG+ltEHn1W2k2 kQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=gQwXSGT1VJgRsGfyEUhNRTVltwlYSVVa2oUvkt+arFs=; b=tpQvSHf+vpTgKMvRZ7jwnfobnnliYr7uTSG9HYsrZ6sXL/FywAn8Aeqkmnsxon6M9qlW rK/IzkHvQNGRDpKUaRSzJ0m46Z8sqoVkEaLoceupeov+knA384v8loWKVSgug6tXp2YB U6EfzkKqL4MjQVUD7v/VR5b37kOdhdyVyVXS+CTJjWbn6macdknBcpfI2Yp7g4Lz4ZiW 1GvHp4Ha/9Ed1FTgJcqwIlltsBs9HhOIs97yE/PvAN0UDpUn+FFyfRkDVFAq17oL/ENl kr5h3JPRQVZ3gX2kZsaCaJU7eYd/CibvfNtyeoYBCidfAG8qPFWLXesYRCN/DxNgACAY dw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vqm98mmr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:10 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T5086873; Mon, 19 Jul 2021 06:28:09 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-7 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fJoj19h4SX3EUzE0ivsd0sEXnHFo0kl3Dc8s+0XyK78yaxLlyb4tNyc+gbav6MYRsgi3YEKyGtJa8TIfqGT+x6mnxCvtzIknyx3WLP5arqByCo3oyYDCI/DiysDR6igphKZbiaDIyb3N6M46YW3xOK/u16gq86W0Pc5XVpTJSqt2/2XE8JHJoAev7kC7yhXeZIqPckCJWr5isMlljnPi79w7PIsvka0bVpm/XoIdNrX6eRyC6h5vS5CEwofh4BQVIhQ/PBjpGY0TxuEddl7TOIp70v6QVC9x1DPCPiV3h34ZxxsjU3jVcepARSaTaWRoJLvaJRy3OdT6wHl0QchRYg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gQwXSGT1VJgRsGfyEUhNRTVltwlYSVVa2oUvkt+arFs=; b=XVH4mvSQxXshA11L7LLSE7NaBC+juzNHh4yvDMDLfGBZocf7ay10LM5swElFPYdDIXqMogRrUwWLxI1UgfYTmdJBeLjaZHGV98LbqPZTJdBSNGBk5hvdF4INJyOYIjYhMDFQOsZVtUxrM6l9AHn3SbzWylWmV/V+BrwZ9FV6JOsM/1wSY9G7NCiFwMWRZpKJgLn+DNIW8P6JVGtBML4SRMPMIf4xwfV+cPi5kDjKyZfflYcKQUQ/CCTX4TIc3RevbuCKcZk8TJh1bdsjQG5OpZ42v6lfip6Jy1CFHzvvTHiP7zXHvHJ09qBM78s5j6HMSmhyNooz8KtGR0J6ttLwgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gQwXSGT1VJgRsGfyEUhNRTVltwlYSVVa2oUvkt+arFs=; b=r6UoflilAW6Orec1EX0JqDgwOWgTnz3b6Ygfdgwf1H4gWvjOAC7ttH0d+C2jNlgdf2jcdpb5TyxmpLvBo+73GptrRDJcDruTpIOSHfI+89P3RkGAgm3n5x2rJnXm8PBTonraHaDOq3smTJB7exSnq7Lq4OEacie4IFT8R/6yQjg= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:07 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:07 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 06/19] vfio-user: negotiate protocol with remote server Date: Sun, 18 Jul 2021 23:27:45 -0700 Message-Id: <6113d0ffcca09a2d3c74abed63871a53ab583693.1626675354.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:06 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f31ec901-10e5-40e3-b41f-08d94a7e5f17 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:873; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1V1cfRlzeawja4Rd+7GNQ84+HeG8QPaz540duZoZXgKGoyf/LiEZCcwdsFULlFeDDnMkfFHccvBIcuzfSFViKgFYy7Eu5AoQGdj07/fv6ylcRNLgMWcN7mqNZgnpyJ2I/6w01mjrGmmy/kelDTPm2oGCnvCywoWfEh7rXSk/N4c9s4AcxpPTmyDpqOASfrZJNaLJLc6Hg8ZeRyuej83UNDRmvcEWSPdQ6DEVPsDzyM+yNTPoNZaR0+K3/VI7FKdxe+lOppxFpK59y1d81SQx5yH6VE6PALg4IV0LKRH4jYDt1me5h9NrwAWKVdteYvP541Txu4MLiM2y3EyvPcId1t+7rKM3iSDv1FadSDlpa5cOPdTkwWDgMFGmAIv8mZibXHjfVj487aDcgBO3Vi08VpOkL1QHZ6JO7Vy3YARNgKYSppVFj3FrC7IwBJIX7GtBLjLJJl5xMuvQx/e85ZmEbRmUXhIGf2qbBM4LFyUwgfK8BQ6waRO9IPdDG7s964ScqiAuNJ3sxURlAUIWlv6ymgSEKPCBWLpW7wmS8aflvhHkL0Pc054oayFZtX5+LZxIXHZ6qplvWEH6sXNMeDEaEWBjHqjYXfTUNJ2Z+8AFx6p5Atvx/uMtyc3/ETyfJFROdr3Al/QZa3ijdhDB4gekjg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: tSGLbtB9HwPdrFqP8wl+V2L0frrQxKZmQFE3AZQR7fDv3ks68pxIc4tSd24GRs+10SA7s2WO8F1wfV2csVQuW6GGC6m1tArpIgsEU07X+8KXuwSYiw4Scn5AUg6PlKtfDFb3XU1S+GWiKeyqGsAotRex6m9ZwLnIFo12fK6X81lCdjL/Lgy6Is0wQLgupa5nU7jCIdr6W09isbO1w9qbqBV+K4yxa7/QVUdciOXLSzyeJzhaYKl4e3qfGjI5JZ6jSyy3Bn7oydsTzaJEESkHbry2rnyIySmCrRdoEYdv/Q7Yf+jI0u2vRFg3tAb9RXN0eFX5+Gp3cZuKaiA6lA/5rBKaPnjfktUhtYUxxWXBmDtUPyJmKiQoQTJAv5RHq8jSdNCTayb4054H64h7MMqml/w3+OVnjuTMA3248DvynfxWtJTbr1VujKkZX17Lx2RAR8Dzwn/RMSHl/sNQvJVss2x4AtmaYFP0J1n2XxTqtDWA6cwNotLJKJn12IqBqbLVh5R4AE/3t6QsIdkb0sci8qZhhezBgBTAchJh+UbuxlhUGYI+qAx7SSg6uPjEi1Dm12ypx3qgRMDH9ABO+4rPFJD5LB+PdyrtSc8uG4UpBwA/31SmN5yka2RkPEn35fh0YQiYByxeXDiP1qO+4PZoESIRyWK0H2TgtBFoKgwPbT4abl8K1YMXc5ugmcTnQaBowznzYn/89j0AO3QSbDXgxR0YYsDL/vXovEo10pNtVUP1SiXaksSuoEkMUrljSE13w43T3BIN3ronbVMhyEUDNoVxtiJCmmMRBc/ezDylKSivXgyQPDS2wSHjVbVhSacOMBdRPkkdQoHRCgYbSW/btr+fhM7m3uTEb43Ckvixj4BI3NOHzR1QQqYdVWGub8bqSxdfwt40iWY3xB6YK7uQ4aVmAKOSLTg4g2zKeBZDtb9jwZwbUaMqXsF1sitrpVmDcoqCEphqqhtqxMCBdZlU9ID2X9Zq42JYpZw5LZypXFu8nPgsoPixxy81w9HMzSmUawyN9CFGL6R6EoHDlHqT0zEXdLJpaV810voHRjOgoKIlR7DgRl8HLV3sZsgd232HuMwCJ5NryzZRzUSBoidukLdjMwakNQH9Kawckc0+Ni/uCGB2OFMkvF2xuoTowt11iD/l5q6dJWtnFwFUe5zAComTfI6U0jp741grqTyHIE72FXuulcLqQN24vxyEiiDBWWPdHZDSm32vkHr7nVyZhlwQClmTm3zJrBusFBbNC/e+b8KZQlwmvkxDanw767AAzfB5fzw/mK4AraIqp7P+oAahI2P7nQIvOylE+y0zhyJ5iqRyQXRgKlzmliw04dpYdgAHKY35noc7szLHd8L0orWi+i9uB+yxA41IgrYd60bXhQiDJT5vrg9QuQ5SQFLh X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: f31ec901-10e5-40e3-b41f-08d94a7e5f17 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:07.2478 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: iv8AUS2T8EyJfFpu9ljDkp9tTTH1GT1R9dm6cmbAceaQ3R930Dxgyxnq5sJ35arATPu0BbR5YTC5kzHTE8/VVNlKlWpTAL5ze3BJ5IKemS8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-GUID: H169o8KRJ-dAOaucWeuVKGVmR1AhVtNd X-Proofpoint-ORIG-GUID: H169o8KRJ-dAOaucWeuVKGVmR1AhVtNd Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Send version and capabilities and validate reply. Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson --- hw/vfio/user.h | 8 ++ hw/vfio/pci.c | 10 +++ hw/vfio/user.c | 223 +++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 241 insertions(+) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 12106ccb6a..844496ef82 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -64,6 +64,13 @@ struct vfio_user_version { char capabilities[]; }; + +#define VFIO_USER_CAP "capabilities" + +/* "capabilities" members */ +#define VFIO_USER_CAP_MAX_FDS "max_msg_fds" +#define VFIO_USER_CAP_MAX_XFER "max_data_xfer_size" + #define VFIO_USER_DEF_MAX_FDS 8 #define VFIO_USER_MAX_MAX_FDS 16 @@ -119,4 +126,5 @@ void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); +int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 1effdcd5c0..8ca1431cca 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3348,6 +3348,16 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) return; } vbasedev->proxy = proxy; + + vfio_user_validate_version(vbasedev, &err); + if (err != NULL) { + error_propagate(errp, err); + goto error; + } + return; + + error: + error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } static void vfio_user_instance_finalize(Object *obj) diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 371ee9cd8b..24dd45b55d 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -22,15 +22,25 @@ #include "io/channel.h" #include "io/channel-util.h" #include "sysemu/iothread.h" +#include "qapi/qmp/qdict.h" +#include "qapi/qmp/qjson.h" +#include "qapi/qmp/qnull.h" +#include "qapi/qmp/qstring.h" +#include "qapi/qmp/qnum.h" #include "user.h" static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER; +static uint64_t max_send_fds = VFIO_USER_DEF_MAX_FDS; static IOThread *vfio_user_iothread; static void vfio_user_send_locked(VFIOProxy *proxy, vfio_user_hdr_t *msg, VFIOUserFDs *fds); static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg, VFIOUserFDs *fds); static void vfio_user_shutdown(VFIOProxy *proxy); +static void vfio_user_request_msg(vfio_user_hdr_t *hdr, uint16_t cmd, + uint32_t size, uint32_t flags); +static void vfio_user_send_recv(VFIOProxy *proxy, vfio_user_hdr_t *msg, + VFIOUserFDs *fds, int rsize); static void vfio_user_shutdown(VFIOProxy *proxy) { @@ -40,6 +50,72 @@ static void vfio_user_shutdown(VFIOProxy *proxy) NULL, NULL, NULL); } +static void vfio_user_request_msg(vfio_user_hdr_t *hdr, uint16_t cmd, + uint32_t size, uint32_t flags) +{ + static uint16_t next_id; + + hdr->id = qatomic_fetch_inc(&next_id); + hdr->command = cmd; + hdr->size = size; + hdr->flags = (flags & ~VFIO_USER_TYPE) | VFIO_USER_REQUEST; + hdr->error_reply = 0; +} + +static int wait_time = 1000; /* wait 1 sec for replies */ + +static void vfio_user_send_recv(VFIOProxy *proxy, vfio_user_hdr_t *msg, + VFIOUserFDs *fds, int rsize) +{ + VFIOUserReply *reply; + bool iolock = qemu_mutex_iothread_locked(); + + if (msg->flags & VFIO_USER_NO_REPLY) { + error_printf("vfio_user_send_recv on async message\n"); + return; + } + + /* + * We will block later, so use a per-proxy lock and let + * the iothreads run while we sleep. + */ + if (iolock) { + qemu_mutex_unlock_iothread(); + } + qemu_mutex_lock(&proxy->lock); + + reply = QTAILQ_FIRST(&proxy->free); + if (reply != NULL) { + QTAILQ_REMOVE(&proxy->free, reply, next); + reply->complete = 0; + } else { + reply = g_malloc0(sizeof(*reply)); + qemu_cond_init(&reply->cv); + } + reply->msg = msg; + reply->fds = fds; + reply->id = msg->id; + reply->rsize = rsize ? rsize : msg->size; + QTAILQ_INSERT_TAIL(&proxy->pending, reply, next); + + vfio_user_send_locked(proxy, msg, fds); + if ((msg->flags & VFIO_USER_ERROR) == 0) { + while (reply->complete == 0) { + if (!qemu_cond_timedwait(&reply->cv, &proxy->lock, wait_time)) { + msg->flags |= VFIO_USER_ERROR; + msg->error_reply = ETIMEDOUT; + break; + } + } + } + + QTAILQ_INSERT_HEAD(&proxy->free, reply, next); + qemu_mutex_unlock(&proxy->lock); + if (iolock) { + qemu_mutex_lock_iothread(); + } +} + void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret) { vfio_user_hdr_t *hdr = (vfio_user_hdr_t *)buf; @@ -285,6 +361,153 @@ static void vfio_user_send(VFIOProxy *proxy, vfio_user_hdr_t *msg, } } +struct cap_entry { + const char *name; + int (*check)(QObject *qobj, Error **errp); +}; + +static int caps_parse(QDict *qdict, struct cap_entry caps[], Error **errp) +{ + QObject *qobj; + struct cap_entry *p; + + for (p = caps; p->name != NULL; p++) { + qobj = qdict_get(qdict, p->name); + if (qobj != NULL) { + if (p->check(qobj, errp)) { + return -1; + } + qdict_del(qdict, p->name); + } + } + + /* warning, for now */ + if (qdict_size(qdict) != 0) { + error_printf("spurious capabilities\n"); + } + return 0; +} + +static int check_max_fds(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + + if (qn == NULL || !qnum_get_try_uint(qn, &max_send_fds) || + max_send_fds > VFIO_USER_MAX_MAX_FDS) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return -1; + } + return 0; +} + +static int check_max_xfer(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + + if (qn == NULL || !qnum_get_try_uint(qn, &max_xfer_size) || + max_xfer_size > VFIO_USER_MAX_MAX_XFER) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_XFER); + return -1; + } + return 0; +} + +static struct cap_entry caps_cap[] = { + { VFIO_USER_CAP_MAX_FDS, check_max_fds }, + { VFIO_USER_CAP_MAX_XFER, check_max_xfer }, + { NULL } +}; + +static int check_cap(QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL || caps_parse(qdict, caps_cap, errp)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP); + return -1; + } + return 0; +} + +static struct cap_entry ver_0_0[] = { + { VFIO_USER_CAP, check_cap }, + { NULL } +}; + +static int caps_check(int minor, const char *caps, Error **errp) +{ + QObject *qobj; + QDict *qdict; + int ret; + + qobj = qobject_from_json(caps, NULL); + if (qobj == NULL) { + error_setg(errp, "malformed capabilities %s", caps); + return -1; + } + qdict = qobject_to(QDict, qobj); + if (qdict == NULL) { + error_setg(errp, "capabilities %s not an object", caps); + qobject_unref(qobj); + return -1; + } + ret = caps_parse(qdict, ver_0_0, errp); + + qobject_unref(qobj); + return ret; +} + +static GString *caps_json(void) +{ + QDict *dict = qdict_new(); + QDict *capdict = qdict_new(); + GString *str; + + qdict_put_int(capdict, VFIO_USER_CAP_MAX_FDS, VFIO_USER_MAX_MAX_FDS); + qdict_put_int(capdict, VFIO_USER_CAP_MAX_XFER, VFIO_USER_DEF_MAX_XFER); + + qdict_put_obj(dict, VFIO_USER_CAP, QOBJECT(capdict)); + + str = qobject_to_json(QOBJECT(dict)); + qobject_unref(dict); + return str; +} + +int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) +{ + g_autofree struct vfio_user_version *msgp; + GString *caps; + int size, caplen; + + caps = caps_json(); + caplen = caps->len + 1; + size = sizeof(*msgp) + caplen; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_VERSION, size, 0); + msgp->major = VFIO_USER_MAJOR_VER; + msgp->minor = VFIO_USER_MINOR_VER; + memcpy(&msgp->capabilities, caps->str, caplen); + g_string_free(caps, true); + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + error_setg_errno(errp, msgp->hdr.error_reply, "version reply"); + return -1; + } + + if (msgp->major != VFIO_USER_MAJOR_VER || + msgp->minor > VFIO_USER_MINOR_VER) { + error_setg(errp, "incompatible server version"); + return -1; + } + if (caps_check(msgp->minor, (char *)msgp + sizeof(*msgp), errp) != 0) { + return -1; + } + + return 0; +} + static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); From patchwork Mon Jul 19 06:27:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384771 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BAE38C636CD for ; Mon, 19 Jul 2021 06:29:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4882D61182 for ; Mon, 19 Jul 2021 06:29:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4882D61182 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57448 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mmb-0006nN-Da for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:29:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33302) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlC-00042n-T4 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:30242) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlA-0002Fp-5N for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:13 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GLnw029074; Mon, 19 Jul 2021 06:28:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=OBr7xnOBLJAOEml4hrlZsz2y2nx1l/6aT2fGklc69/U=; b=JqheCGhXi0g+iBgXAdXsLpJxsqZ5d4rnLCjZQV9HhfLOchkxTtrrovvZ33Pg6p1PMjlu 4hkNcTFwOV5CawRdha/h9R+5LgGbu/Xu1N2hDRzdRbvDgWFnPP7ERcNlxrdQBLBCaW1w fRj5JTeJRfrlzc5d70Fl1DqFEc40DX/XQTjwc5TpnNKHrLli4LQTl5mnV6NOMG2RxnYH +XYkFz6CAqFiqdfswrnhgecPpcE/P5RpF59Von/1LVYOyd68267ZpaDuLt9Kn+MHNzrZ vbxWqojqqycwUUCchaFFRGCB8kS+gBx5hdLve1RDqu9qASTqarjTN+vd1KROSIyHvJIC Ig== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=OBr7xnOBLJAOEml4hrlZsz2y2nx1l/6aT2fGklc69/U=; b=BT06vkLraEfraw/8TGvIHcTcPZgHJpI7yVRevEesn4ICa+dmk3be4px1KvPFywsW+PSv J5Dqnz67KHgpc5j8TpPZh1Chf8jPX+56z7AMAWGeUX+kCNAcWltLwx/HPhuneNaa/CcR eAzz2trgbAECuRgpZ6aGjcVrve3qXdug92/TSuNM82dmT8IeHUvv9gNVJO5t1IKISsTI B0tfDGZFNfEUxRKi+Y/8gMghA94civCfBUAJVQ07kiOVaRbXHjGVT5MXslAlk8s8/wl8 nHiMHeD1vUeAEI+AkNVFVIKiigBpFldon1+vtVbDyJbMhCn6qSGcfbipXTVMlwrw0lGe uw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vpkwrnjb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:11 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T6086873; Mon, 19 Jul 2021 06:28:10 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:09 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IlSbDu8vd3hJxX9SSU06wSEh7cZO531s2GZof5VgnInFJaxyl5DHITa6gzvZf5c1VThbf2Il7713xyWrx8a0V8s1wQsujo19keru2H4cyb7engoo8nxnPopTEr/EDMnLr5MyVa8IL+XBCgNT97oK7X+DUeH0p6HoCq9UeUmczJzi1dKufivWNd4vY8MR2qW0JfAfd3VRrOTktSNyAX6JjSRxDuAH4F6JTGfWpsCLg+73hT/ky/Tg0QM4gjFWwIdkioRr3QMDXUu6Iw+1crs9gdBM13xsQYMzjrMZzXFTR2iloJGnxt8YATOnrC+bYem2bWw25C757fZVL8BqaeRugQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OBr7xnOBLJAOEml4hrlZsz2y2nx1l/6aT2fGklc69/U=; b=KBxCpk5c/GUt2sgGGLAJblefXa9FNyux5F4FvXZRoW+eQOtV3jc2yyoNeiZ4yGNmE5LV48wKZO9CXyePc7tYQFBawkq9eJ+YVQxqhttBDp96wp2mC2Pv5Y9o/MvnZxbS26h1gI9S70t+lyPPRcsZ5WXuGb15F9Kg4xcghRITQhcz3/AQtJMVxKUnQCwWjepcn9xQfw5LIZYDJS8Y+C66RDXCxhpoHtRhukX32RLi5cG42FhAUbL/3Bfq5rwDBK15R/uYnCGD6Ww4utBuD7655N4tO7pZLwrBKO6c0tJ+BPSumXnBWymmTlA9QPke5LMh1V4uNA15kJR5i57vXgzNXA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OBr7xnOBLJAOEml4hrlZsz2y2nx1l/6aT2fGklc69/U=; b=vcMLtEiq0kZx+HX8ddAoFvgmyG0LifsKQ9L935tJO0JAV9HCn/FZO5vIGLXjKchviQrjZzViqCfQd0crWjXy1tuHOu/4k4d3RZZ4Hlyfhn9hoElh8IeBR+MxCBUYURUU+ndDvHUiBt9IJOV6gYLE283sVriv3WxQbcVpZnt25Vs= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:07 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:07 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 07/19] vfio-user: define vfio-user pci ops Date: Sun, 18 Jul 2021 23:27:46 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3ff9fe14-78ed-4bee-b3a3-08d94a7e5f5d X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:546; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: GZMpin+DuvQqY/swoB0wfJ5IfRUvaEjWLPJrDwzdm0PVcNKwf0LT/a1Pctuwf/AdEymyesCvyzY3nRfHtUfI6nPE62m3hLn7Ahl+COhy0fMrl2Gmh0v7JlUEp0fvw9o0VOc2jquzJHD85WHkO4EgmMvONyzw29Mzr3Zr+nmgZW3tr+k9/PyqJocwo6UvePpdrKCTfvB2e6Q6BqszzJfEnYKQ6jfVeBo+H5Ua8xxp1vfjiR3WE1oWTi9rIyr0vah6OG74seOAQ6a0zEGMLboXpU5k4HCKQh1IUk240AOHIUaUgt+R9Dprz+0HZYHyiZjODf20b1Agg2G9vpLkUhm1kS56MaOcIy4YSiWQ4qz3FyBCct5DybVdKDkOXTUAFA8jBVMbHoaVFBaN51lqeQCJntZLXzkN50476Gp1pH/TWwKDN0p22kt/yMZI0nvM25L58zTVT9g28fYC7RV+S+PJvtqbc5w9luAL7H9JdMC1cX2ycvARFT32LE4e2CvQBBHw+UvTdbnsf2tjEoj4pxR7Yv04rFZFQ1hkDTYYtBp8YWCr/186FmOqxx+8haacSSDULB0oNXMWDGiCW/1RyNTY+Oyr2nl9aRcFDOd1YQdFg17/8QlzSnGAOLj37/YgLjvNAMvpv2OMSf/vJwYgB3ydGQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: x0poM7GmSBLhhZOnyPaO9KTajPpUcZL0e2jo4EUmGxEXmNQZHLUJgKO5hys8p2bl5NWx+7LWLlTWAMaY9bOIXUc3fEo0Jd3KmjTgm4hgyd4Kklm1ZHYIjYNSbIsaRCrnZ6Q774aea+6GuzP85GeHrnKT4km8QqthZz4kM7um00PDpQs3NOm4F2pN8NCDeDb1HVMVLZdpKSBI6x3HwMx2Id2FQBsnziqdaEVKd9XuQXEIVbXcZcprKgsCPJ2Xa3ycUP3YW9zG344SJVGzG4SE+LJlRtFyM43Q/22oSXMKWKg4mbFIXFlg9N194z1w07YHoj3CvC9cJ/X7iuqZmoRXuQ48GdyD3yrPlLd/CHQnAQ3HNB1G9r/MYk854mBCcLecCKDcQ0xFyd5AGwuK9HNnictbadSADcq4FHQ+/ocKss8dI7dv426+AfoK6oKnu0gBCOlBybVZ1x1Sg1dvx11TzLYnKuAmMqsmTZnV0D4U8rufTfioo5m0RTWwIgus5FvFNJ1HL0MAsmMlf3eNb+rg6hcBiP9wC2+qssos+lqcQ00iYWk9s+Yj1XwtUlvYkxICCSeEsNk0IPkdBNSMwhHJkJIUfFtHBHrXuoPUKPA0wpq7pFtXaloTXQ65aBrzPkwsH5k0DEfMiXlmia+1/SAPrL7liaMqd/X5ZSPu4JflwqS1xSzO84UC6kJRqRy0NUk7IY5uj6KGocmD/Y9uCyvDuEHP2YqxPX7U5JwOlc6HaN2tbR9NkL5HrFTvAh48KNpNTceo+DxTW7HsKi23/2VzIhFoVDj97wVlH2+2Sx6Mv8ey9FbXsk4Q4ViMOleI6hvJPjkHNJpBTzVKwkPvG8xPbIn/hc6i36Nl6auZXkUz8XdT9nMYLxmmZ3RJw6y492jXg/TsA/YhLquCCVmzb/A5Yq3y2V33kE9uXeTfQXHC5pxEQXMdmzTY+mbFxM59mitUErVPaNF4loYPBhLILDSdmkNSrglI43slQk3wcWOi/Zd4dAkJp0rkDaR88Q853JZPjCqOpe/tSDOWnmHbXxaGRtdu5O2ERGndErHTwYKVwwUh077u8oeVJMHi+ZmP3Zev8nSbXJ6/V/eq9P7d98wbFiZUxJEmJ8gBP8LxwleNmB/1bc/oW/S1z7Q+wdHWbljPzOYAPGZFKZ2cKYHB5yVHKrrr2r6PFA8ECuHJp3fIXHAq/6j2N/xfbc+9jZxNEo4MfRg0vDuA4YYKXq909IGM59Egf2ORx+EKziVWkokwr5VoRmJF7udzAdDe6VUwI9QlROW5ajKtx8CAFYjVHkUm88WG/PBFSFNaKVudZikL7xGjCGRbmmbsc9o1yZZwkjTYpEmH6JMNf/S/d+pckfySfyWpjCMf4R/Epxr4HwleO/wlvGo2eaV1xhUweZzIBG4q X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3ff9fe14-78ed-4bee-b3a3-08d94a7e5f5d X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:07.7026 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: DVdIWF9Z67bBY25Eu99S9Nxhs1I6YWQZhEfPglGKl3ojtebif4Taberv/+ePGfL2KSNHNDkyaypTalpu1Sp2S6C17YTmM/z3Lr3zaqBYLnk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: 3cNSO1iSwf8i-7_vgTLIcVcHPrSc7Z-m X-Proofpoint-GUID: 3cNSO1iSwf8i-7_vgTLIcVcHPrSc7Z-m Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 8ca1431cca..388b7d82d7 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3328,6 +3328,29 @@ static void register_vfio_pci_dev_type(void) type_init(register_vfio_pci_dev_type) +/* + * Emulated devices don't use host hot reset + */ +static int vfio_user_pci_no_reset(VFIODevice *vbasedev) +{ + error_printf("vfio-user - no hot reset\n"); + return 0; +} + +static void vfio_user_pci_not_needed(VFIODevice *vbasedev) +{ + vbasedev->needs_reset = false; +} + +static VFIODeviceOps vfio_user_pci_ops = { + .vfio_compute_needs_reset = vfio_user_pci_not_needed, + .vfio_hot_reset_multi = vfio_user_pci_no_reset, + .vfio_eoi = vfio_intx_eoi, + .vfio_get_object = vfio_pci_get_object, + .vfio_save_config = vfio_pci_save_config, + .vfio_load_config = vfio_pci_load_config, +}; + static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) { ERRP_GUARD(); @@ -3354,6 +3377,14 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) error_propagate(errp, err); goto error; } + + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); + vbasedev->dev = DEVICE(vdev); + vbasedev->fd = -1; + vbasedev->type = VFIO_DEVICE_TYPE_PCI; + vbasedev->no_mmap = false; + vbasedev->ops = &vfio_user_pci_ops; + return; error: From patchwork Mon Jul 19 06:27:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 519A2C12002 for ; Mon, 19 Jul 2021 06:29:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B470D61164 for ; Mon, 19 Jul 2021 06:29:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B470D61164 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57474 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mma-0006o9-KF for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:29:40 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33354) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlE-00043E-52 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:16 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:30574) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlA-0002GG-Jx for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:15 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GHn7029063; Mon, 19 Jul 2021 06:28:11 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=FFdyD+a/2ZQkDVqNWUbWAz5EoorT/d2BRjeGywpi544=; b=C2fCLmE/nGgZDcEIw3/dS7KtFuFFDcCgONvNnAcVBFXuGxWD6fkm58qlaCGT0bjYeRkL eS9e0hKtGekFb0cBKKi9N94IeiBPHMZTGJxoGFoCk1PgdR+x0b7vwQv6t1Jom7bzFw/8 a+/SSUo6fgaP1uPUsTY7IAJkk+dEOW0SV6OUynceCZm2nBsAX9h+zyS+pXFEN+2vMnXx ldgGNiG3c/PwxjA983YWugVjUv3WN3FZS29FRiNfpSj+vzPmhuheWAtqQB8SpiVml1wf VQwAc4ItxoTR/J8HHdXjNFsQaJtt81uncsdD6vOOVTHlJ2Qiy0W0BhEafkoFweQVTzOf yQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=FFdyD+a/2ZQkDVqNWUbWAz5EoorT/d2BRjeGywpi544=; b=xPdneKT7W8f02gyzJTIFww/MP32vZumrkZcLl9jRGCdNRhOqBkm7mWAfyg3uKYzyUDqL iwahPABO4PjkQAMQbnKTvyMEjrvcHXSa5me2Wy7c/3KNDzwEbyISZqKgMbDyit1EPrxs xn5q2UGO4bh2jAoQqp8y75Pj2AjfOmq4Tg5KxFHd+cjgWm+oREhndijFUYWx1HSL3EQi G71edVnbQlPOckfuptrvyOk3Ovre44qKwirH8/cctPAd0blRfs5jkWBgnDcdvEXb+b0A wC0CzAs00uWG9f1J+O1N8LlOLUS0vsdlDzcOof60Ssu2WlH3+oST+wTS5z6Ea7QMgME5 1w== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vpkwrnjc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:11 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T7086873; Mon, 19 Jul 2021 06:28:10 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-9 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=fg41+RypQRa1kDmOXj+omryvBqgVLD1VwA+ZugDRgWgY/bH0e5cbf//5YSrKqHUf1AezUxcLwsVlcIRYYJ/QVgYs+CEe5PfWuVUQuCZQg0R++8TtbRnhtH7SixOh44RPqtmenTX4ODI0+LBPy7Jbxw7xsnguAI0Iey3zY48qhcJ+0e6GcX5bye5CCqfZBq0/fbsRSvMVNkajyOU9lbPuuMnK0f6YANNx4ylNQA+Q1DrH/9UYujJT+YfNZR/WF/xMVXj703xjy3qq3AaYJ3OHqTfBJzGFJNhmDCwI96DpvOB47ZbBQlLOI3Etr8d5Cpis4Fv+CyYQU5hE3pJaUqGNUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FFdyD+a/2ZQkDVqNWUbWAz5EoorT/d2BRjeGywpi544=; b=FoZA/aQC0Y4QeIuXt8UczFkZxMXug1gmDOXzfK4T0YkACMOdB9fHcBH8MixfW1mKQ0JiRMHfjyVNeEUp+HQAoADivOymZAPsOitC9xb3Yse5q1x0hbpL6re4IDwViFUljwm01NBpWTOva54ww5yUzFoW3moC9PwLVQUSoj2MV/+9+ZVvaa02jocXOeEbWLLdvkbFPKlzS6jVyo5kME+ZkPZGgI7mWkG03NXyMLDrlXoawhd+iiRmge3EryFpQYbIWbkwUN6itX3Iby79olvNJWagl7JVb/UA560LH5QrbLfn9VpIlHQIHQqYTWQIw3B96VQ3CCUs7wTdKy+VbfCO5w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FFdyD+a/2ZQkDVqNWUbWAz5EoorT/d2BRjeGywpi544=; b=OxfcpOnY/P0BlYOfOEApWFFWQ6urgkwNVnWtHGn6w/V7QmyhapJoYH7IzrMsLe8dbxiIsmuzUlsiHlOBhQvsXKqhHMO4N4MbmtrSDQgcKB7WrbnYopxn5bxwIo7MTuguRlvl8eoYoxmT8kOkjJ2ioY314oOz6hnL9jcFd2oxfgg= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:08 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:08 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 08/19] vfio-user: VFIO container setup & teardown Date: Sun, 18 Jul 2021 23:27:47 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5bc51341-7de7-4e94-1278-08d94a7e5fa5 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1186; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SmLamdJq0b/wVAkkJ9q4Kj5g8CPGLL1gNvDHSZ9jyNDQ/RO5DQMGcu4TFGyuSQcX6UFS8765FYdQLzj8dHGajPVt2aYK2VtmkDTD5Hdv1gUONdBjTSxL6AB0S44C01EniY4bMSfL5qMlrEGyoZ8LEaJbewLzexzQ51YDaKSUOfTPFGu4vzLsirORNlDpfcieEyM7KyzUvhlWoAQwpYCzJUwdnrBi+Io8CcFS0HMackA4i0VrBmI2ptsxcnu4phFztrJc1r7XkfsdmwlOS25LnGN54zbp/s5jr+EOEB/wPlVco8J7nIwRWXu2P2ELgRFzaGAI/aZIKQZRj/rm62N8tatCbpv7aqr/PYZMgv16J6kT6NCGRK8gZz9CVKnRcjDTjB3otXg75vrQtUd1WpcQoZMmp7zv1uQVBNJ2gXD5pQqp4VNegMHUC+VVBCjfMzvLAov68gj2+9ysXpSZlOlTeAZslz8fnlQqt/cpPUkn0RLugCMTVC0Q/kQEsaXuh/oujTOHD4cT3n73sK8QzrhLLlKOfalDgP08D0mJSklfEM4xDhw9hleMs3xEIAJ/ZVBZQxRdlImjeZExWbYghnhg1naIYxBd5TTaups/ya0zkTknbMelX5YrFrDsDxytpAY55vJ8kojPTmRe2VP8ssw40g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: moXI3oVngmMn5vj8NCIvQKeYj9CRaptbaFYxVqCA4WO8Px1I8d2gn7da2Tv8hniOBWIDWg5r36RbV1dfjS0CrxVCOdj8rq32GLb5n6T0dNSBfcUWxhKCU7KbOjqM6ZM+OnwgqKCv13iCWvd1P98Bbjx1qCx/iVpH06mj9aHAVoWUMBkTN2rOOBVHGupwcNQssbG99ZOb2Yj+3DspPyAIhjQnsKuatV2wRTJvK29jytm4zWrtbIbjTPK2PSgEIR6lM9anJX3299/pMN50jJ7dEwRtTbWnPWprJekDcHi9OgstJEBrV5pNlI5vrAn05z9YqVUS+rzLJFU0xcj/nhD6bPTwXOVYjMH+X9pOoOP0BrmXfJA91IlDxnijal+aEhR9kyv3WSwLDqyTNZtqdmBHQizIfE9EsdRqwlSlcqsvdvzRCfk2CXU3zsZiqOQeAFYNENAsYlc54xBXp2kBOoGWQrpImEwuuwgDmy0Wb9dflliNmppttuIokB2Ej/HF8xkZwj0vjrMQEXUVhbAlpL7V3140mvr7XKp9RONYaeFbeFJaYYWrFwGbxqb956uphZc4sFCfxZ3mjYyIhLfcixWxhtLFRGwnQVKiIRzjM0TshFipzCPnumN+FLBKviHfHvMhU7fnLBQntbavNHlVhxFYKW7Xi7FQcRmxttvw+mcDgHgS8c1S3yp770vpwOfxcidPpW9HrACqpZsD34SJgKsIEidXGWUbs/6vsri1GqGmXAslhxJnB9dcBTHhf7nRxGzxC6jlS7elqTqSmP0eQiyRRZRe9t5sjHgTiqRUK049thCpF9U1r3JnfDo1LX9BMVOTlB84M0BGrEdC5g1m5XF1f2Z6Zq7FtSCi8WXMszjoYvzQuSWOaheemVdlmwFkEB5QlBXqEtdAMfHKonWm8W6xNyA0P2Y1toBpOJQjkWuSdEHrX3YZftpEkTWcZ7wE0PfCnZLrt4tjLif3Q5oaXSYWGwlVLhwfAStNWIf2hluOeUdCBr3AOXq7bJXWZC9t7fttvQv+wQ2arld+ULwElASJrawcUTV4HEVPJ4t1ICPr4xeFCtQloXSI9Z06yqU/R/ZbBb4cQqMGontWwdOUwtZ+KY6+G3tRvjBx30Ih+GJM887JuYvE2DGNjJNrCxQT7gjRPvNYKkS0rnx4d9jvyHriFPUUbVx5boTgaSQgyO1Z86tcnOJ5LzvfovGSmzVm9OagybzirG+k4mlHyLD2aoyO1IScuv7WI4Xt7Wfv+Pf9lPNmm4Crsox+E4PyhNXLbHMbXnbTgr+VteUweC/Pwz/Mjk91sitWnk21dXOfNkcTTU7DRookEmcPehjJbmw9AaC9UeRpVP6Dj5iP8anYolHTGyHCeCzc7s4igVAIYhQ1b4DGxNA8U4BlLpC43s1xp7jg X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5bc51341-7de7-4e94-1278-08d94a7e5fa5 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:08.1933 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MrQwoS9jcl2gx3hV2N+wtVGMbVqVdGoVBTmGZaLDVGcvdb4D+WQ/Z1DWRtlURTGKNSK5QnQvXkfMwG6AGr0BoAp45nzzaS1dcwdwj+HJWfE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: sA3UFmLFIu-aznlLwwQFcvrs3PQkISLA X-Proofpoint-GUID: sA3UFmLFIu-aznlLwwQFcvrs3PQkISLA Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: 0 X-Spam_score: 0.0 X-Spam_bar: / X-Spam_report: (0.0 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Create SW-emulated containers and groups for vfio-user in lieu of the host IOMMU based ones used by the kernel driver VFIO implementation. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 70 +++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 19 ++++++++++ 3 files changed, 92 insertions(+) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index f43dc6e5d0..491a92b4f5 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -90,6 +90,7 @@ typedef struct VFIOContainer { uint64_t max_dirty_bitmap_size; unsigned long pgsizes; unsigned int dma_max_mappings; + VFIOProxy *proxy; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; QLIST_HEAD(, VFIOGroup) group_list; @@ -214,6 +215,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp); void vfio_put_group(VFIOGroup *group); int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp); +void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as); +void vfio_disconnect_proxy(VFIOGroup *group); extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 8728d4d5c2..45acdeeb46 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -2206,6 +2206,41 @@ put_space_exit: return ret; } +void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as) +{ + VFIOAddressSpace *space; + VFIOContainer *container; + + /* + * try to mirror vfio_connect_container() + * as much as possible + */ + + space = vfio_get_address_space(as); + + container = g_malloc0(sizeof(*container)); + container->space = space; + container->fd = -1; + QLIST_INIT(&container->hostwin_list); + container->proxy = proxy; + + container->iommu_type = VFIO_TYPE1_IOMMU; + vfio_host_win_add(container, 0, (hwaddr)-1, 4096); + container->pgsizes = 4096; + + QLIST_INIT(&container->group_list); + QLIST_INSERT_HEAD(&space->containers, container, next); + + QLIST_INIT(&container->giommu_list); + + group->container = container; + QLIST_INSERT_HEAD(&container->group_list, group, container_next); + + container->listener = vfio_memory_listener; + memory_listener_register(&container->listener, container->space->as); + container->initialized = true; +} + static void vfio_disconnect_container(VFIOGroup *group) { VFIOContainer *container = group->container; @@ -2248,6 +2283,41 @@ static void vfio_disconnect_container(VFIOGroup *group) } } +void vfio_disconnect_proxy(VFIOGroup *group) +{ + VFIOContainer *container = group->container; + VFIOAddressSpace *space = container->space; + VFIOGuestIOMMU *giommu, *tmp; + + /* + * try to mirror vfio_disconnect_container() + * as much as possible, knowing each device + * is in one group and one container + */ + + QLIST_REMOVE(group, container_next); + group->container = NULL; + + /* + * Explicitly release the listener first before unset container, + * since unset may destroy the backend container if it's the last + * group. + */ + memory_listener_unregister(&container->listener); + + QLIST_REMOVE(container, next); + + QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, tmp) { + memory_region_unregister_iommu_notifier( + MEMORY_REGION(giommu->iommu), &giommu->n); + QLIST_REMOVE(giommu, giommu_next); + g_free(giommu); + } + + g_free(container); + vfio_put_address_space(space); +} + VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) { VFIOGroup *group; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 388b7d82d7..5ed42ad858 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3358,6 +3358,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev = &vdev->vbasedev; VFIOProxy *proxy; + VFIOGroup *group = NULL; Error *err = NULL; if (!udev->sock_name) { @@ -3385,6 +3386,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; + /* + * each device gets its own group and container + * make them unrelated to any host IOMMU groupings + */ + group = g_malloc0(sizeof(*group)); + group->fd = -1; + group->groupid = -1; + QLIST_INIT(&group->device_list); + QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); + vbasedev->group = group; + + vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev)); + return; error: @@ -3395,6 +3409,11 @@ static void vfio_user_instance_finalize(Object *obj) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIODevice *vbasedev = &vdev->vbasedev; + VFIOGroup *group = vbasedev->group; + + vfio_disconnect_proxy(group); + g_free(group); + vbasedev->group = NULL; vfio_put_device(vdev); From patchwork Mon Jul 19 06:27:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3038AC636C9 for ; Mon, 19 Jul 2021 06:35:13 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A4A2A61166 for ; Mon, 19 Jul 2021 06:35:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4A2A61166 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:49316 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mrv-0003Or-LY for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:35:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33396) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlF-00044h-N5 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:17 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:30916) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlB-0002HD-3d for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:17 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6Gf3b029140; Mon, 19 Jul 2021 06:28:12 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=C1cl8+/Z64RxlRrnft/e5lkanWrnkZHPQtG7ZHNBckQ=; b=KAAojAByOBTPmRQBpWvLWgKzIbs86sPXffxYBKjndr2tqdg199HDN0UI3DqiP0ahKHAY 9FtnecHM5gwsOfWQtxy1WztA8iKviI1gKDgIvEEJO+K0ozmeU2X26B/Kcz/fnyHxx5W1 eHlZqJlvg0AtzaUQ8m54kr+gMVpcH8YFrQMF+jo9a/1juw83ad96/na0dsrvQFIo1BH/ 0dShaL2KMXDZtigegleOJcor07RlbOCrgQAmN6DF32dQM1r+PzaeYTxNLoLqT80iliuR yIZiKdvQ+D+n90EmlGtv/SVmDT1iIVyspJ8bu4HSuubgrEeTObFyKyKek2sm3fNVQqhJ Kg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=C1cl8+/Z64RxlRrnft/e5lkanWrnkZHPQtG7ZHNBckQ=; b=bsxa4XK9adkO7zCGNDbe7AHOQjw9cIHbfC+8XfF7D+EzhAtW6JZpYvIiNv5bSFOhVFjo ZMXKtQO0dWJrtdZGdvi2cmutXZnDq9qU6IzZ75lZamIa86g+vDadS2C9lypD4OnpezUU qFT514qjeStdyyt0hX10l3I21k9Lfvx+i6Girt2E9swlKRNBuff1sv3/UjjMUrdouNz2 XYrpEwIgPctWUgR35qHRJwyGlQLjRcylfsGL60XLPHhBEwC42qXdMJUJ8qMIc88c5MGO xldAzOia+NyrWH369ZIYSmPKBACXZfM0uaflytrtnYSP85JyUnjBMd3M3vSUGzbYMTdL dA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vpkwrnjd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:12 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T8086873; Mon, 19 Jul 2021 06:28:11 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-10 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:10 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Dg5f9EykGaxyLvzmfmE2mWypqLdVdwVs0mdGtPG9UkfRx6tEgVlUELHFmI9ZZ6R9kRfBAheMe6mfcQ8qOK3cqUo+l7EVu4bgM4Dzfk+Njl9fQhj8dco7oxcPY8gcBxG9Gf4It1txVX9WdH+SS98bKQiLWnq+KIDcQvN5xZh2mwbeFX7F68MIeF4AbWk6kJOOMDlnq78Twe3JbCRUQv/rmriAT5xpfU+OiifneTNS4vSGcU+nvbdNOvQKbZ5eyxIwj78cUYTlYGwD21Fh3B96mHQxyODMLJot6db0wujI3f2o+SflOD7v4cHsEDtpuEuNVLtm+2NV53m6ObQghpZc6Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C1cl8+/Z64RxlRrnft/e5lkanWrnkZHPQtG7ZHNBckQ=; b=ivrjcOO6F+1RCFWwQ4UygtIXBUcxiop59kIYMx3qRoobolC86lQE9E3tt6mBZsIXspXtP8Q0QTzMHRGl5jPp3TqKmxvx4WzStRbKGneaC6gVDJuJhATcZA5lErywKyIP3mD0qx81z14+t//ixFWB+sLLAhLYv6b4ej/htBzUnxRM/H6QEcWfi/ZDUbINxERYLSOXqMKkAxsv9iW+Ccyj3dG8uKm6pLRZ58GJH0C4pAkV9vKFevYL0txBxtgZZnFofGKUNbVjuHdz8ZBQx3Ul4e9fMASAOeK/mtq9ZbeLG5wH0mQuu7Xzr+wzdPIvtGV6mBfOGsQZzDvy5AjT3TWXgA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C1cl8+/Z64RxlRrnft/e5lkanWrnkZHPQtG7ZHNBckQ=; b=y1AwQ1JChlRuiMZM6EaN5hLZWcz9dgz125GpZz+04FUACWZS4nULeqp5nQ9NQg0AeMklyzkclWx9lM/Pq9xLYQDXSVtR9jfZikfPJRXlXc+Tn8b1S7i2z7y7By14PVzkNQGPze+nC5/rlhzHuIz40HbxSYrvmOrjtI6/CaTQs8A= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:08 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:08 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 09/19] vfio-user: get device info and get irq info Date: Sun, 18 Jul 2021 23:27:48 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:08 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 7def8d77-fd8c-44ef-ba46-08d94a7e5ff7 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3513; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ryx7rZREAlxpvQN40puchhb35caYxOjqGuxDr6eGO4pdP9GezCM84TfbwOXSgJuGKHBzCnPuCiYquhUM7/0BqBWI7+k9K+iBguDa3l0uyl26ary9ZZWkGEpQA8wcW0N9EoL51i9tZQGWoy0UwRZDHyqtt52C2wiThVQ1TBeF8vGvVI1/K6j1mbxOh8fcgYPNApr4ciF31oa2MEp3OZLw5Cy1EPl8Svz+Z2DFawfsJ8u453fRCUqdu1u/8nDaW1BjQdDjeo3KTyNIv4n+9JDx7uKb2N3C17Goqn46VgGSOjlK1sLxI7tDESHbu0jsAyDTxwFOKI6nO1w6tVkV0DkqI5/w1ssno6w8ftPo+ZncEw0YNrNbEeCEXrLFeUrdVQRw1z3Kmle/oUywgMPXd24QVIez08K211Gzq6GRHjKe6kDuEohHMtMlY/+cc0yzqX7wfGGoSgmi9Uqr+nEo5mz6X/pdj6xt8PULQ6RKWn8QDJHf2dz+C32LFGMV9GCjChNd8eQrfNTWO6KjoSAcTgDzMGdyLqHLN3kb8Il8tPO06QB1SyNHs7lXLMBAyirrp4mauUszTHr82F3AwzjE2xv8HjMjnvVhJm8RVnXvrH8ufSWajIKvqJGV+FE0kNaYYfYGtxuO0p3frhmvDTmuB5b27Q== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: trpmdXfbNReZ/kxo3EI6iJLAmifzAA/qaAPS5eV5BwLROzLXqIt4GBP43fAoErlOvYBgR99JG7dncHfmXHFRszK1EsjiFZDIiXsjcGSATGColJEodLLmDyiPU6z0wSnPC5m8J/yKb0pkq2xVTXjzmG+QOqhaG1uAnmRicuIaNK9Pe6szj78MWhlOfrvuJoBeh9ZHkw7OX8iRm4t03yqAaNrzGtF8PVPGSDudznqn35d1pp9nqA0xL87ArmFjBlnH8qyXPnd9X+TOuNT5EOVF0svsKTrZ0NPoXW10kzHZF+qDHGiP9/1JUmajmVIBiw60WCV8hw7yMVjL4lKQ08yfu161gFVM55L9jnNMw70cSLQ7wyoxvC9iu8z9H4Kg+IORZE4keyfMQQ2wy6TR+pAbXoNe0Q5xsK8Xo355JTlfPPZOlW7U5EuZCsO+cFDDjDPZ2UuQeCFhAd1qLkvcFPgdOxtvEsQ3nG798gwQAQCKoGDc5odsyxwVWgp6YxZBAU1QHDE6X91JU7PTIJImP3a/HdsbV0YG4gJS0SVAntmf3wZupv95Oybjep3++GTIll4LwRolfhnKjGBvYV/P6XyURuh2x9NljJ5D+UTylgJB7lwVVbKJPNe/njsOkirHcoZ6qvqWzgg8W4G2jAQ39EyhUMUHmOrCNMZkIP+xBb/8kBMuT0Fp3jjEBmdxhV2J1K7iRXN92s+YFGjdNSlwDG3+2nbMEhve9b8qnCcPna4Nfhvdx7HZZX5/wAqzS3W5wAWJwYkk/kl6jz830W/hG+voY8qGGDo9uBeOOJ+/SkWLgFwL2dOeiwGnv2uQC/SxUi2MLtMYxfMas8pqWd3buEp4uMaefpz13NKSckhGGVN7COlJlYdfT2WYRgSbYEKSxY2EWFn3AsqU3LkSQweYn/SiEvugfx0nxhTX1IJsfKFVp0/ziKLD/Q2u8U80xPxIvh6+QgtVtNmTylrTepd/Qp/Xh4Y2S/+nlKS3DJGWE8cNkOLh1yqrnLJLm2++2MIC9g+CjOsqtR2qoIR8oYdy+lPN33leRhxfUOgznOLqWSnbo/6+HGX66Peogh6x5r9eX1uxx6SMHOqpUjSw6dBdmoz8EL89FPEExa13Qjvpp6PbMQxipqZi3TRW86ZL99yLrCePmQ9mEnwrXjAgEUALSnPmYPXnHsBgFNzlDJNqbcCbPtp8Y5zr7uFnYVN6QXJgL+wsNZ/aiyc5aSdvCn1me/Wa/QhOB3oJtcjNZhGltJ50kImkyYXAGoXtp1R45mFlEJ1honLAgobh2bChgVg8D956Z7bTvKrg+N+MMaTPxJuRlCIzr1dLvZUT3RiddK66CTWinciB4sYGKsGf8/Zdguehg97R0R1ChpCQm38nSitlFdwb69gQ/nXwSaOXzgqWjJP8 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7def8d77-fd8c-44ef-ba46-08d94a7e5ff7 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:08.7010 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kmg2wRBbzC98e9t8Fv5WZM5gajb09AekjCp5vJEH3IytF16HUXmOoc7OXm5CuOWn1I5Hk66+5g+RQ0bWLJTptwUxuBskRr1BCsvrKRNN3pE= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: s9Ej_hLG7WcGqDoWiAs6f6b58iRrzyWm X-Proofpoint-GUID: s9Ej_hLG7WcGqDoWiAs6f6b58iRrzyWm Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Send VFIO_USER_DEVICE_GET_INFO and VFIO_USER_DEVICE_GET_IRQ_INFO commands. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 27 +++++++++++++++++++++++++++ hw/vfio/pci.c | 32 +++++++++++++++++++++++++++++--- hw/vfio/user.c | 40 ++++++++++++++++++++++++++++++++++++++++ 3 files changed, 96 insertions(+), 3 deletions(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 844496ef82..9f51e14c7c 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -122,9 +122,36 @@ typedef struct VFIOProxy { #define VFIO_PROXY_CLIENT 0x1 +/* + * VFIO_USER_DEVICE_GET_INFO + * imported from struct_device_info + */ +struct vfio_user_device_info { + vfio_user_hdr_t hdr; + uint32_t argsz; + uint32_t flags; + uint32_t num_regions; + uint32_t num_irqs; + uint32_t cap_offset; +}; + +/* + * VFIO_USER_DEVICE_GET_IRQ_INFO + * imported from struct vfio_irq_info + */ +struct vfio_user_irq_info { + vfio_user_hdr_t hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t count; +}; + void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); +int vfio_user_get_info(VFIODevice *vbasedev); +int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 5ed42ad858..029a191bcb 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2620,7 +2620,12 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index = VFIO_PCI_ERR_IRQ_INDEX; - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + if (vbasedev->proxy != NULL) { + ret = vfio_user_get_irq_info(vbasedev, &irq_info); + } else { + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + } + if (ret) { /* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure(strerror(errno)); @@ -2739,8 +2744,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) return; } - if (ioctl(vdev->vbasedev.fd, - VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) { + if (vdev->vbasedev.proxy != NULL) { + if (vfio_user_get_irq_info(&vdev->vbasedev, &irq_info) < 0) { + return; + } + } else { + if (ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0) { + return; + } + } + if (irq_info.count < 1) { return; } @@ -3359,6 +3372,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIODevice *vbasedev = &vdev->vbasedev; VFIOProxy *proxy; VFIOGroup *group = NULL; + int ret; Error *err = NULL; if (!udev->sock_name) { @@ -3399,6 +3413,18 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev)); + ret = vfio_user_get_info(&vdev->vbasedev); + if (ret) { + error_setg_errno(errp, -ret, "get info failure"); + goto error; + } + + vfio_populate_device(vdev, &err); + if (err) { + error_propagate(errp, err); + goto error; + } + return; error: diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 24dd45b55d..a282b7b7b8 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -508,6 +508,27 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) return 0; } +int vfio_user_get_info(VFIODevice *vbasedev) +{ + struct vfio_user_device_info msg; + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0); + msg.argsz = sizeof(struct vfio_device_info); + + vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + + vbasedev->num_irqs = msg.num_irqs; + vbasedev->num_regions = msg.num_regions; + vbasedev->flags = msg.flags; + vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET); + return 0; + +} + static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -594,3 +615,22 @@ void vfio_user_disconnect(VFIOProxy *proxy) g_free(proxy); } + +int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info) +{ + struct vfio_user_irq_info msg; + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO, + sizeof(msg), 0); + msg.argsz = info->argsz; + msg.index = info->index; + + vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + + memcpy(info, &msg.argsz, sizeof(*info)); + return 0; +} From patchwork Mon Jul 19 06:27:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384775 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3F71C636C9 for ; Mon, 19 Jul 2021 06:32:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 67D7961165 for ; Mon, 19 Jul 2021 06:32:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67D7961165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37814 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mp0-0004Ca-FD for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:32:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33468) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlI-0004Ch-A7 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:23 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:12022) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlC-0002Hv-Ek for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:20 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6FxMW017887; Mon, 19 Jul 2021 06:28:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=JoSRHchFLZc7vB7+cNiCPubvaI91cmiZGXjF6WeqNpM=; b=vYKtnHkRaupj6zSdGCVTdstc4F6O+W0Wd0G+S2qpG0QqKagNG9yxE6Nlz95dYfeB6OsI Cd+Mx2ob1ATxtvLeOHzTW0eIOm0v8zipwAxMjNCRurNxk+n5UTJHkR7RIPM8Ot78wSIW wlorP9Ff7CLjEEshdQJ/Zzg3cJ1yyCC8poYkQBpe4P/fJ1wYxqj2SiTw0HWRSO7Hocs6 rQF5NjVkGgVSLNmfxnO7zYN8YRT6oF7O2voLF/ii6dHV0/eaeG1mZyq7YyxbimWIkNh7 BU3MHUX7rJaKzwyjs7a8hZk6/pxUPOE9N7S4zk1v81vOhBDTCHksCYzHfWQZY+W8zw9x Ow== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=JoSRHchFLZc7vB7+cNiCPubvaI91cmiZGXjF6WeqNpM=; b=X+V/ZOVZ1uExfzuBaNIdp3QsZLe9KSn26C2slvqjhgw3KyjHigXkRelCLvZ7PGHICvb/ dz0gs+GGIj3J7GlFWGHk0RDvCsUrB8nzlLxFCdMhKpV6noUQViRBO4elAWoQBnxFPi7C IT0jdmqWSwA7k+235sLPEd9QsBP8XqqNxjOZx0cJxF1eBV/Rzhz7PqydTVjZtZHuAZ9C UvTO/Kc6/gA7BFltTt+30wPLDhljszD4x/TacD+8IR67FtNGLN/kwJsZOAxOsk5+3AdH MrNOjNqMq/JcRmcN/O/YLORaPJ5IdSDdNnNokfH92qDKzNkvSD3gsyFwur2+ylOQdaHt oA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39uqrtj615-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:12 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9T9086873; Mon, 19 Jul 2021 06:28:11 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-11 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:11 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=QDy4/T8D/l49jYtgjn2/URET6iEmgC6WT0mxdsZfr0UkZVMqtFXVyveCznP/k7Q77czYqDNVBhYbUo9vVXDRCpgbdjhkblfTKEualVJOBbD6RyGGJKZEYnDjFQO/6u9etMUnGp+0HXgoaDCIK0NHD4+3IIuy3bAZl4WBCKEnW9MeHjApbzbDfR8agIE9iXtChnh67o7uUY6+sxQ9E6Yg/P4KCl707vstCDoicyxwbsOSnhQxwyVqs4bKD7w0bYXfGsBDIEMO3KlViWhWFWC2HTELcyfdhABwSIontu1DkEKhRlW6NuK3cMbHVPc5HkeEA2kEuNy8Ogq+qItFyXTBNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JoSRHchFLZc7vB7+cNiCPubvaI91cmiZGXjF6WeqNpM=; b=RpI7RSXqyH99XVyBuuITQE6JeSYuxO0X9E2ZSef4vmf4J/vxI0mkrsnYo0H/Sx/2zZ+oQ9xrSfwExH+TJ+cRl/GaSrbpiAq+HY2zDQzNWNljjHWtSQfJoH3ZKL+DBOSWX9epnOQjxJQZi04RXp99RdpMgJz/Un9ZoVdC8JSzzV4k9vYOIgiwO6Bq/rt5d1asJ2OSK7wwYDKLpCqH/UBbCIU+7VM4NLd4B3Zznk/asK3srI/ck2zroCBB90DlFKG9+Nqg+Mdkcvmhl9mZ+Ag4fW5iT85IFIAzyWsj2KXjYXVhixtrGJBmhBsKLDx109Sj9phItV+1bpPVNa26fleqzQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JoSRHchFLZc7vB7+cNiCPubvaI91cmiZGXjF6WeqNpM=; b=TfeWNWpNBVAEQUFKaZ15NzSPNtcUAA5SRYSC4NXVU0maaxxi5dOd4MjcI58TPyRSiMzckAaCo1yF/IgTB5P/szAuKW/+AIg7iueSp2GPwnaD/UoHH4U2KJjw4B6KhPv/RDJ+Cw0vu7xgPD7yJgh//R+d8PAdtzvNNLC3DAlu7ak= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:09 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:09 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 10/19] vfio-user: device region read/write Date: Sun, 18 Jul 2021 23:27:49 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:08 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ae1bbe3a-1805-47b9-cedf-08d94a7e6045 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:96; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: YQBqyiKm44m3cArzIDgM6a+xpEvWv+oerjE0uQrG9DR9/Kk4IiAnekZK7ZTrNWAaNrDSuysoRnygN7f7VDB3rNtEfZrfYtJPwJPgtiDYhn0AuFPwqHoOeIl+RhVPr2AwP/Dzm+4DE26mF0Pf1jXyd3Us+L8LMko8ef3hF7mYxKuZXaGuIg2D2epI1/I91yXAheF9akwVOcs1veAsBrmfYvrb8wbzHQxM2BENRjetZKr7axqiZeAQ1IpRenq74WRFBlGnDcRZaONo2Y1sPCUuhVEp08heFxvZtl+fLla955TG/4R5r7dn8dAmWJ0jLp85p5yUv0j9zxx1Fxxu5LdzuN5I/zT4QIWnTWRkXFE0jdnYlz+fZ2FO26FQ1wOqQfOmRa2pMIEO3SzcUANGtpBe5exzGmGNXYd8kPdaqQWJQ9Ki14JyOCQryQMgzuAX0LHck6SotLeaOUkKv7m2K1kBqfRPuvMvUtvG0BfMDun+q0JgNqCn+U0BeU3hFkp6aL5ZK6bZR31Ar+KBVMgexy1HfTjGiFn/JzHtJVJ+Y5WM6ZEQtvQgHTwUIMuYw0ChQTQhdL8NEOddZaxirgtXUlXD9IaLUIlZzVEvS+7YMVg8Ou85piCiBuxIIlM79oaWusFZZkAHFOKhviqmZuWSPcwptw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: b2i/G4Kgvjwb8Ay1CkXisetF2k8jV6KuuyIeb5T3SxWn4DRQ0XAP0pNhZURFIilkNkCPLkMkQq4hW1tpR66cdPIHBKhRNTKhuhX2PkY/ShCcOCC3gYWDkoxzk64KH6KN4ta7TJh99V0WRLizv8MUYATnYviLqu2QaqnR9Ve6cO2ZQ/AAxnPx+k0r8EMPQXG8QQARLVVyA0SAzdm3/FMI2sgkVKIJEehazj8icQ8UTRBSpEEqNdrT9pkvtj/7HoBid1cRDFzHBabc2TuD4GybOLWvV0ROmRKLlw3gK+2xZy/L2u9yrz6PnFqmEkBG/JS2d4hQWnbTJwjhigUZZmyLcATVKMfe/L/4sk3LTaKGlnLn1N/cqE8HycITC4Ay5Ew7yx2cfP5ZErhgJjql3V++Gabwg0WRvtJaQxNhNUKlQC/sM/oNGH9YJa2PFnwqNpH2sWCwxJSjAFr7j4TqxIIBXNqo7s8NVabZ67+Vn6hnc4VnHKIZ3XjehjIaDfumVy4a0CF2dO/h01WJJN2uYg5KjL+ejIfFGVySZhJOUvKD4sPMAM792YsghnHMgafFztID3n411Kut6XsTmuPu7rcJfPhYEPVCq/xAWym8UMgVQfIqOVywD05G794yMoopEECn7ZVZgE0GLrV8V1r0DtLapqmm4uJdHmQxNQovdnQUJ/8ZrWq8b0O6Q3G6XwFjZCamiSuYz34H9smiNbz6LV5OcHL0rWpUgbLiBNZffC71tXVWQUx0ZWWzteWZ3lBrEzEfs8YnwdDY9pV9YO+CkoUPYi2OfRusXPkuDX+YcZesjJajhWaAjV49SioRwWVEvTMk6iSr6CgAO5hbx4Auh1f17UcYDns/YxdRyBW47zu5GJwOk9BB8ORUKKeLZufennINYBYaWBbEwdZV335if1JOFcyFNTliFsn5ghwzBvQP9FJbvxxwy2YFiF/QSWy8QKNBdeQrNaUYPFXo2GVBhV5UV3FJVOaexP8SSVduPiy2m7Cr73uL+LYtZECjEOzTiBsPDi0B68hOfUKW5e4zNIOwlNBVNbR96LobqIjS2OTWC/30nT0XaqJ+VsDClC442K62+pYpAcdG3rRSGvdpD642KYSP7fVVq+80HI+k4rdaSnLLmbhJz9p85VI7RMhlsLSPFhcff4Yqw/+II8aOa4LPtY0zxOYNCJyh23FFCkcxo2ZBPOC5SH+K+yuJ8yaF5mD0KKq7ykK4b94E8v59PG6XrKtGdU4v811WXXW7Dx57sz4ipxiNBpXY1wHJLYgM932wvBRxWOw5m5QFEKLVmrTTc+HVtBfpaBzBgB8b2qeWn4TxRsswErBzmy6HLZ4eeXB7/0fHXIXYiKyeLOSD9QQdh3w4Ds/73S3v32Iaf/bTLtWP1L5cgEu06Rr3AbWcj94d X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ae1bbe3a-1805-47b9-cedf-08d94a7e6045 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:09.2197 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gJgzJTdwhbJOTNTlMrpYLqxEJelhB62Zz2AL2UB5TYXzBgRnuhejHvPDOvXXl7QC83l8dESRw0oxEZ5Y85Iv12akO+Uc6FqLoxd28NCQGNw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: W6RYi4LwDH4wuLOSH-Ic-GFJohrU_RvW X-Proofpoint-GUID: W6RYi4LwDH4wuLOSH-Ic-GFJohrU_RvW Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Send VFIO_REGION_READ and VFIO_REGION_WRITE commands. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 16 ++++++++++++++++ hw/vfio/common.c | 17 +++++++++++++++-- hw/vfio/pci.c | 13 +++++++++++++ hw/vfio/user.c | 45 +++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 89 insertions(+), 2 deletions(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 9f51e14c7c..17c4d90ef1 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -147,6 +147,18 @@ struct vfio_user_irq_info { uint32_t count; }; +/* + * VFIO_USER_REGION_READ + * VFIO_USER_REGION_WRITE + */ +struct vfio_user_region_rw { + vfio_user_hdr_t hdr; + uint64_t offset; + uint32_t region; + uint32_t count; + char data[]; +}; + void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); @@ -154,4 +166,8 @@ void vfio_user_disconnect(VFIOProxy *proxy); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); int vfio_user_get_info(VFIODevice *vbasedev); int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info); +int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, + uint32_t count, void *data); +int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, + uint64_t offset, uint32_t count, void *data); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 45acdeeb46..74041cc438 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -40,6 +40,7 @@ #include "trace.h" #include "qapi/error.h" #include "migration/migration.h" +#include "hw/vfio/user.h" VFIOGroupList vfio_group_list = QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -214,6 +215,7 @@ void vfio_region_write(void *opaque, hwaddr addr, uint32_t dword; uint64_t qword; } buf; + int ret; switch (size) { case 1: @@ -233,7 +235,12 @@ void vfio_region_write(void *opaque, hwaddr addr, break; } - if (pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { + if (vbasedev->proxy != NULL) { + ret = vfio_user_region_write(vbasedev, region->nr, addr, size, &data); + } else { + ret = pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr); + } + if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64 ",%d) failed: %m", __func__, vbasedev->name, region->nr, @@ -265,8 +272,14 @@ uint64_t vfio_region_read(void *opaque, uint64_t qword; } buf; uint64_t data = 0; + int ret; - if (pread(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { + if (vbasedev->proxy != NULL) { + ret = vfio_user_region_read(vbasedev, region->nr, addr, size, &buf); + } else { + ret = pread(vbasedev->fd, &buf, size, region->fd_offset + addr); + } + if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %m", __func__, vbasedev->name, region->nr, addr, size); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 029a191bcb..1054978e5e 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3424,6 +3424,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) error_propagate(errp, err); goto error; } + /* Get a copy of config space */ + ret = vfio_user_region_read(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, 0, + MIN(pci_config_size(pdev), vdev->config_size), + pdev->config); + if (ret < 0) { + goto error; + } + + /* vfio emulates a lot for us, but some bits need extra love */ + vdev->emulated_config_bits = g_malloc0(vdev->config_size); + + /* QEMU can also add or extend BARs */ + memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); return; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index a282b7b7b8..2bb6f8650e 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -634,3 +634,48 @@ int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info) memcpy(info, &msg.argsz, sizeof(*info)); return 0; } + +int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, + uint32_t count, void *data) +{ + g_autofree struct vfio_user_region_rw *msgp = NULL; + int size = sizeof(*msgp) + count; + + /* most reads are just registers, only allocate for larger ones */ + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_READ, sizeof(*msgp), 0); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, size); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } else if (msgp->count > count) { + return -E2BIG; + } else { + memcpy(data, &msgp->data, msgp->count); + } + + return msgp->count; +} + +int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, + uint64_t offset, uint32_t count, void *data) +{ + g_autofree struct vfio_user_region_rw *msgp = NULL; + int size = sizeof(*msgp) + count; + + /* most writes are just registers, only allocate for larger ones */ + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size, + VFIO_USER_NO_REPLY); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + memcpy(&msgp->data, data, count); + + vfio_user_send(vbasedev->proxy, &msgp->hdr, NULL); + + return count; +} From patchwork Mon Jul 19 06:27:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43CECC636C9 for ; Mon, 19 Jul 2021 06:38:41 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A20DF6024A for ; Mon, 19 Jul 2021 06:38:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A20DF6024A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57836 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5MvH-0000m4-R7 for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:38:39 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33452) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlH-0004BU-Rr for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:19 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:31964) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlC-0002I0-I5 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:19 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GLnx029074; Mon, 19 Jul 2021 06:28:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=C9ZLxtfUWYwDgZJWWzfKcWuF4VObv81neoVJim6Ryfk=; b=ZXQ6K+LDZg103tJe5tcOwP66/ySyyWxgjm/XxyFCd/jsWqHSkgMZKzkGQ59qZ5AinjAC 2LmOXKkM18t2bawTiItLdIeo2dCNIRUInrV2nzRbeB/jzanK8dmkiNbT+NdplJMvyroZ Ba5kGBUw0FispYi3zVWd3yByvkwDZvQfVweoOWeYrv6bN1gYHNXtGWO4dTa0CuP4Qsxj kf2rFbFYZxkS3v5vvsupkVUCzKrjGGY3iWsYHx2lZmhbi6PNps1ZKQeFoOcnvvdMJIZb XFnDNyuGFNEc9ahFo3wbWzbtEnujzienho+EHGOr5uJqGLsL+c1wglQERkYEC0MRMf41 zQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=C9ZLxtfUWYwDgZJWWzfKcWuF4VObv81neoVJim6Ryfk=; b=fqgLXU+p/6l9Q75I2MMGjgjwJ32Taq7RCIyDkJHV7pCUB7h6l8eCHV/3bs6y6LWVTiNZ rdJz5i0aWWCcp8LixhSNtnMeAJqw6kOiIuqFv4/AL7jIP5gld8WDqJ1oBGoinBd0F21k avFzg6MBwWuwQeBEAAWHk7XXK7ZXgIsvSSGgVCWspBQwwgKE3RrRVNWMojlTpwRWyURN XMr+aQ4CWTNsKoFLkMjAQZKM+/ZJ8NGYQPd0VMn8fb9910E4O3qLWnM4E6RGZbdoIhO8 U25wCYZdQ1vabRMhDbNmOGuwvC1ifDTfLZlmxZOfd0tE/10P41MkIcZxE1HbTajKThI0 sQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vpkwrnjf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:13 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9TA086873; Mon, 19 Jul 2021 06:28:12 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-12 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=L3aGPE4fQ9+KkTZlGVXVvPca57NF1T4Idnn/0uasSNmBftVXeate7uhhlt319N6YkVvLHGOM8ZAx7EMVM3tOMn+xL7WZpdXnFjbtVCPMO1PQ5PVfhzgU/Dpksx5C0Ll3qzJKDK+49W5QhheRDQnZafEmNJfbW2p6hwNmKqzTeNOCprtGAcNZjuQEtk6gnRe9kVs9jsSnrMW23sbxtvaMt0zX0w0jIQRVxKoWycfu6ViotxFmz3tRWHMc/e9luyqONmNi7hTHiukMj4du/w7OQoSMkff1D0Rj6+q9Txv6wqdBvnybVnBI+aMLCo+qr++Ahb5zgZSJca4LnE2Fa7UVHA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C9ZLxtfUWYwDgZJWWzfKcWuF4VObv81neoVJim6Ryfk=; b=bot0Z8xVSIuKcrrAbXwTSKHZSFakR4f3W+Hr3KCpLzmaObu8ZpXHn5Cm8AsI8qsk0u4AOr1xDZ2t9Q9V4aP6Gy16tUNPNV6YceId694WomZB8H4IfZoixfLDYpqrRPmfRGbw9+84fFpWRa4Hy1kyJAN0RByCaPHgcJcPMnSX8lECcVENlVof16V4wPVXrdJHCmC/PszydfS8/Wy6AE4WiFi9A1bsEIHZvzShXBlW+Ts1iT7TPMdHyxHiOoMUU8BgxjwSCOYjRUa8hQPusUS6NVisTkEoTCKzO6KfJz1a5Ao2uA2e2PbHPGhQu0PneyDkt/lSRcqPzPvF2HBSxri99g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=C9ZLxtfUWYwDgZJWWzfKcWuF4VObv81neoVJim6Ryfk=; b=hqgFAcKiKsCbsyaNtf4NxnAZLKudkLnZWs9o2yhSlr30kG3vwCTPBuaML9Cbo1T0B6j3/bUEVgfHQxvq9iFwAR2/T2HOp5tshM8rjHedw6+GAnmiogkPt/rNePMQNSAGQa5eYszautv2PX0C/UXB/xSJUlM5EPhc9w0hhN1/iFM= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:09 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:09 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 11/19] vfio-user: get region and DMA map/unmap operations Date: Sun, 18 Jul 2021 23:27:50 -0700 Message-Id: <7774798805f2a30ef47364976461fa1b1eabaae0.1626675354.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:09 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 97b7a7dd-2c6f-4441-2946-08d94a7e608f X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oG9BnZLBIKVUYxdLDEV3K4exTJjb947iB6SrLRDN5yiSw97pWFS066KgQGU/lte0Vbvy6SjWrhXhnsdBn7XXtYb0gFdgy/Xg0C1NFpZICHktSueS4I2b/CoE/pyzeyyVbqGa2aBeufD/rZ2IjVIInQ7qAu7IkHp715D1XU2g0FLBTTdKwi8lxeneJE+HsOmlsNcBFH+0knzUG8Rqf0NMGOSDmk4+Hq6UwdUGTslAgAA18oi9Yrefc/gW4S8p5LlMjzlx0QpTKVH/FfaHfNe3qmPOFxPeDi35Sgho4LZaI+Z7Bw4zZf7uV1imqBR9Zeo+W8ikFdeeWSUMFhaPcpu8ycKqIVxFbgbv/H3JrBOkGXV1y7GV5tVUI9JaFU7OGlBiPgTQj6IHoPNl5jO/0bi6/syZ/2B4EIFzBzHMHmtDirIEkMaw6bJL4aHnzd/E9/bfxoSbjjoc/DYe4qQ8Oct0Tz0FzEEmLlDPCuPStbz9rVRERwdSP8rqDFQco4EHAZDKnXL2cwHc6D1GUlbbh8W2aKwdy3hXqzr7mZ6To+KDw66DXn3OlVH4GhbI0hpNuUFYY14sA1qn79SwV3lWsRw2gvPmZmwpfrnjnjorzyuQKV6ntb0xDTWuaW0kS+L8F5Rg79NGRXxsAArdbU2WTuXjeg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(30864003)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: LNPEcTWHkNj4xQ3g+dmbooNmsdwnyETwyxv314rXy9E49bMiVgBSogfbL41/jmXayZ8q46QbrEKl/hGGbJwpt+xgL+PHdEYtV1e6Qm9dSeQ63A7NGXXEEDLrUD0GTLJKKFlrD5ssPvShQKHzoyzceJRz+3I9q3w1cRppRp65QTSKQNMJa1cFgPIOCzgTlGcbJU6Kbeamj9DxJTs2Uuy/V7CcYgVgFgYQqFTTMQL/NWq4uUQjoqTbch5lmeM138z2yiSAU8CQvo+7/Z8segduQeo7JqeMrtGfURUOI8ygdkWWbjzNXoHJ5/6fv6/NtWWsEGnX5+vKUd9SUyBK7H//UdxrXIyQ+egnB9wHKd7aTdNvGoNdb6gSStH6hpADO6fLBYzhMG2E15LdWobNTvjvdGOW819hJ/LNCHTn6ATtKw5cjsUIe67jvkmX7jvQahARz3MqyZ4MkFHU0z26A6IuGHG5QOM+I+Sti6yLTHB4V8ix9tCkKWNPjQxZAzKjIVANLTMnXBga1jWwebS0PRC1uOEzm9F4h/e6xvVLefdGBXBx7yvuO/wncUN+gC3Y7Oe7FFiohbqXGjjZj6e5bBTCIL8DbQzY/IrHT9tR3CucIaEGMFKgzG7/7W7RggEoJqIMlSKqGK8OkBW9qVUFR38rEQsgaXQeL5XT+Jc2F6tCx6x0BIuUXCefwpWZm04XtZnZijQfQXQmlZJ/Co1bACqX69MwqKuW+vuhS/lz81GDJN0ia0kHgWOEA/HwpUTFFA8Suf4wHHw2tRYnQxwYJGaOtFBlymAvtysVuAcJI38L7NbmUeadcgiH3NCMqgXWxoZYZb8IQk9bddBlawnNAeQkH6BHP9dohPZDkEDqjIxs/INdd2qDiSJzsRNqkyfmKipUSv6B3kkSsHhyajF8dyT/+WCHq1d8J6BrCmArYmzsemjHaJNvkPsiasY5AAOeXqrCWVawn4C5h6wVxqE7lDw1lwVmeGaxTx6/2NweyKj5Y1BM/ZY8e6s8keu4CXYHvDBGsh9D1y8s4W8KKBygkNHxqHqO76BZnKClIPpWdHTSfivfwGP2sjHxj/UmCnPedIbLmrmtBADJZxBQY0TagwwTPot50ooBZC8HTCuei1Oda3DzTYkYgBN/wNy3GWwSLZ3A0m8Pv8NiYQxSyKaAr9Lupx1jAWhE6ugyvi+dQuS6Ga87tCDgf0E8dRzt+MNYuvIf3bD29OYgcbS5+TdHE4HRzVc2Di67QWUfvIF0N4EEYYDCn/YAEMA/VrIaHQFOL/LqKUbCHAnLMN/PQ53rLb6pbhKtnosiPV8XxxC2bPZXuZW6S1b1WauDpU0wiPItVWc7Bk/FrSwOe6AalOUHjVB7qn0KWm4MKigUlA3IhaKOV37DMxBxF0LnniswhctIHwMp X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 97b7a7dd-2c6f-4441-2946-08d94a7e608f X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:09.7214 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: XsDi2ejFbuKlOBEAeudwnPyH/GVc5weQaMMECVU+hP5PhCZHxr16zLrAf3Z8IJtEtuABzkU834cI+uEnzF2EZkZQXqUc/QnBNndCOkuflmM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: bh_ZZimwEgeL0dKrRo4ZkTK8AF2kvFnK X-Proofpoint-GUID: bh_ZZimwEgeL0dKrRo4ZkTK8AF2kvFnK Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Send VFIO_USER_DEVICE_GET_REGION_INFO to get device regions and VFIO_USER_DMA_MAP/UNMAP to tell remote server the DMA addresses it can access. Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson --- hw/vfio/user.h | 54 ++++++++++++++++++ include/hw/vfio/vfio-common.h | 2 + hw/vfio/common.c | 84 +++++++++++++++++++++++++--- hw/vfio/pci.c | 4 ++ hw/vfio/user.c | 100 ++++++++++++++++++++++++++++++++++ 5 files changed, 236 insertions(+), 8 deletions(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 17c4d90ef1..351fdb3ee1 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -121,6 +121,7 @@ typedef struct VFIOProxy { } VFIOProxy; #define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_SECURE 0x2 /* * VFIO_USER_DEVICE_GET_INFO @@ -159,6 +160,52 @@ struct vfio_user_region_rw { char data[]; }; +/* + * VFIO_USER_DMA_MAP + * imported from struct vfio_iommu_type1_dma_map + */ +struct vfio_user_dma_map { + vfio_user_hdr_t hdr; + uint32_t argsz; + uint32_t flags; + uint64_t offset; /* FD offset */ + uint64_t iova; + uint64_t size; +}; + +/*imported from struct vfio_bitmap */ +struct vfio_user_bitmap { + uint64_t pgsize; + uint64_t size; + char data[]; +}; + +/* + * VFIO_USER_DMA_UNMAP + * imported from struct vfio_iommu_type1_dma_unmap + */ +struct vfio_user_dma_unmap { + vfio_user_hdr_t hdr; + uint32_t argsz; + uint32_t flags; + uint64_t iova; + uint64_t size; +}; + +/* + * VFIO_USER_DEVICE_GET_REGION_INFO + * imported from struct_vfio_region_info + */ +struct vfio_user_region_info { + vfio_user_hdr_t hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t cap_offset; + uint64_t size; + uint64_t offset; +}; + void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); @@ -170,4 +217,11 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); +int vfio_user_dma_map(VFIOProxy *proxy, struct vfio_iommu_type1_dma_map *map, + VFIOUserFDs *fds); +int vfio_user_dma_unmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap); +int vfio_user_get_region_info(VFIODevice *vbasedev, int index, + struct vfio_region_info *info, VFIOUserFDs *fds); #endif /* VFIO_USER_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 491a92b4f5..d7b717594b 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -146,6 +146,8 @@ typedef struct VFIODevice { Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; VFIOProxy *proxy; + struct vfio_region_info **regions; + int *regfds; } VFIODevice; struct VFIODeviceOps { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 74041cc438..52a092e168 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -477,6 +477,10 @@ static int vfio_dma_unmap(VFIOContainer *container, return vfio_dma_unmap_bitmap(container, iova, size, iotlb); } + if (container->proxy != NULL) { + return vfio_user_dma_unmap(container->proxy, &unmap, NULL); + } + while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { /* * The type1 backend has an off-by-one bug in the kernel (71a7d3d78e3c @@ -503,7 +507,7 @@ static int vfio_dma_unmap(VFIOContainer *container, return 0; } -static int vfio_dma_map(VFIOContainer *container, hwaddr iova, +static int vfio_dma_map(VFIOContainer *container, MemoryRegion *mr, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly) { struct vfio_iommu_type1_dma_map map = { @@ -518,6 +522,24 @@ static int vfio_dma_map(VFIOContainer *container, hwaddr iova, map.flags |= VFIO_DMA_MAP_FLAG_WRITE; } + if (container->proxy != NULL) { + VFIOUserFDs fds; + int fd; + + fd = memory_region_get_fd(mr); + if (fd != -1 && !(container->proxy->flags & VFIO_PROXY_SECURE)) { + fds.send_fds = 1; + fds.recv_fds = 0; + fds.fds = &fd; + map.vaddr = qemu_ram_block_host_offset(mr->ram_block, vaddr); + + return vfio_user_dma_map(container->proxy, &map, &fds); + } else { + map.vaddr = 0; + return vfio_user_dma_map(container->proxy, &map, NULL); + } + } + /* * Try the mapping, if it fails with EBUSY, unmap the region and try * again. This shouldn't be necessary, but we sometimes see it in @@ -586,7 +608,8 @@ static bool vfio_listener_skipped_section(MemoryRegionSection *section) /* Called with rcu_read_lock held. */ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, - ram_addr_t *ram_addr, bool *read_only) + ram_addr_t *ram_addr, bool *read_only, + MemoryRegion **mrp) { MemoryRegion *mr; hwaddr xlat; @@ -667,6 +690,10 @@ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, *read_only = !writable || mr->readonly; } + if (mrp != NULL) { + *mrp = mr; + } + return true; } @@ -674,6 +701,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) { VFIOGuestIOMMU *giommu = container_of(n, VFIOGuestIOMMU, n); VFIOContainer *container = giommu->container; + MemoryRegion *mr; hwaddr iova = iotlb->iova + giommu->iommu_offset; void *vaddr; int ret; @@ -692,7 +720,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) { bool read_only; - if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only)) { + if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, &mr)) { goto out; } /* @@ -702,7 +730,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) * of vaddr will always be there, even if the memory object is * destroyed and its backing memory munmap-ed. */ - ret = vfio_dma_map(container, iova, + ret = vfio_dma_map(container, mr, iova, iotlb->addr_mask + 1, vaddr, read_only); if (ret) { @@ -764,7 +792,7 @@ static int vfio_ram_discard_notify_populate(RamDiscardListener *rdl, section->offset_within_address_space; vaddr = memory_region_get_ram_ptr(section->mr) + start; - ret = vfio_dma_map(vrdl->container, iova, next - start, + ret = vfio_dma_map(vrdl->container, section->mr, iova, next - start, vaddr, section->readonly); if (ret) { /* Rollback */ @@ -1064,7 +1092,7 @@ static void vfio_listener_region_add(MemoryListener *listener, } } - ret = vfio_dma_map(container, iova, int128_get64(llsize), + ret = vfio_dma_map(container, section->mr, iova, int128_get64(llsize), vaddr, section->readonly); if (ret) { error_setg(&err, "vfio_dma_map(%p, 0x%"HWADDR_PRIx", " @@ -1330,7 +1358,7 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) } rcu_read_lock(); - if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL)) { + if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL, NULL)) { int ret; ret = vfio_get_dirty_bitmap(container, iova, iotlb->addr_mask + 1, @@ -2493,6 +2521,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) { size_t argsz = sizeof(struct vfio_region_info); + int fd = -1; + int ret; + + /* create region cache */ + if (vbasedev->regions == NULL) { + vbasedev->regions = g_new0(struct vfio_region_info *, + vbasedev->num_regions); + if (vbasedev->proxy != NULL) { + vbasedev->regfds = g_new0(int, vbasedev->num_regions); + } + } + /* check cache */ + if (vbasedev->regions[index] != NULL) { + *info = g_malloc0(vbasedev->regions[index]->argsz); + memcpy(*info, vbasedev->regions[index], + vbasedev->regions[index]->argsz); + return 0; + } *info = g_malloc0(argsz); @@ -2500,7 +2546,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, retry: (*info)->argsz = argsz; - if (ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info)) { + if (vbasedev->proxy != NULL) { + VFIOUserFDs fds = { 0, 1, &fd}; + + ret = vfio_user_get_region_info(vbasedev, index, *info, &fds); + } else { + ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info); + if (ret < 0) { + ret = -errno; + } + } + if (ret != 0) { g_free(*info); *info = NULL; return -errno; @@ -2509,10 +2565,22 @@ retry: if ((*info)->argsz > argsz) { argsz = (*info)->argsz; *info = g_realloc(*info, argsz); + if (fd != -1) { + close(fd); + fd = -1; + } goto retry; } + /* fill cache */ + vbasedev->regions[index] = g_malloc0(argsz); + memcpy(vbasedev->regions[index], *info, argsz); + *vbasedev->regions[index] = **info; + if (vbasedev->regfds != NULL) { + vbasedev->regfds[index] = fd; + } + return 0; } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 1054978e5e..054e673552 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3387,6 +3387,10 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } vbasedev->proxy = proxy; + if (udev->secure) { + proxy->flags |= VFIO_PROXY_SECURE; + } + vfio_user_validate_version(vbasedev, &err); if (err != NULL) { error_propagate(errp, err); diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 2bb6f8650e..eea8b9b402 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -679,3 +679,103 @@ int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, return count; } + +int vfio_user_dma_map(VFIOProxy *proxy, struct vfio_iommu_type1_dma_map *map, + VFIOUserFDs *fds) +{ + struct vfio_user_dma_map msg; + int ret; + + vfio_user_request_msg(&msg.hdr, VFIO_USER_DMA_MAP, sizeof(msg), 0); + msg.argsz = map->argsz; + msg.flags = map->flags; + msg.offset = map->vaddr; + msg.iova = map->iova; + msg.size = map->size; + + vfio_user_send_recv(proxy, &msg.hdr, fds, 0); + ret = (msg.hdr.flags & VFIO_USER_ERROR) ? -msg.hdr.error_reply : 0; + return ret; +} + +int vfio_user_dma_unmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap) +{ + g_autofree struct { + struct vfio_user_dma_unmap msg; + struct vfio_user_bitmap bitmap; + } *msgp = NULL; + int msize, rsize; + + if (bitmap == NULL && (unmap->flags & + VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP)) { + error_printf("vfio_user_dma_unmap mismatched flags and bitmap\n"); + return -EINVAL; + } + + /* + * If a dirty bitmap is returned, allocate extra space for it + * otherwise, just send the unmap request + */ + if (bitmap != NULL) { + msize = sizeof(*msgp); + rsize = msize + bitmap->size; + msgp = g_malloc0(rsize); + msgp->bitmap.pgsize = bitmap->pgsize; + msgp->bitmap.size = bitmap->size; + } else { + msize = rsize = sizeof(struct vfio_user_dma_unmap); + msgp = g_malloc0(rsize); + } + + vfio_user_request_msg(&msgp->msg.hdr, VFIO_USER_DMA_UNMAP, msize, 0); + msgp->msg.argsz = unmap->argsz; + msgp->msg.flags = unmap->flags; + msgp->msg.iova = unmap->iova; + msgp->msg.size = unmap->size; + + vfio_user_send_recv(proxy, &msgp->msg.hdr, NULL, rsize); + if (msgp->msg.hdr.flags & VFIO_USER_ERROR) { + return -msgp->msg.hdr.error_reply; + } + + if (bitmap != NULL) { + memcpy(bitmap->data, &msgp->bitmap.data, bitmap->size); + } + + return 0; +} + +int vfio_user_get_region_info(VFIODevice *vbasedev, int index, + struct vfio_region_info *info, VFIOUserFDs *fds) +{ + g_autofree struct vfio_user_region_info *msgp = NULL; + int size; + + /* data returned can be larger than vfio_region_info */ + if (info->argsz < sizeof(*info)) { + error_printf("vfio_user_get_region_info argsz too small\n"); + return -EINVAL; + } + if (fds != NULL && fds->send_fds != 0) { + error_printf("vfio_user_get_region_info can't send FDs\n"); + return -EINVAL; + } + + size = info->argsz + sizeof(vfio_user_hdr_t); + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_GET_REGION_INFO, + sizeof(*msgp), 0); + msgp->argsz = info->argsz; + msgp->index = info->index; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, fds, size); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + memcpy(info, &msgp->argsz, info->argsz); + return 0; +} From patchwork Mon Jul 19 06:27:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EFF28C636CB for ; Mon, 19 Jul 2021 06:34:18 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 970D560BD3 for ; Mon, 19 Jul 2021 06:34:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 970D560BD3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:46588 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mr3-0001ci-AR for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:34:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33450) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlH-00049Q-CY for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:19 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:32156) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlC-0002IJ-Qk for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:19 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GHn9029063; Mon, 19 Jul 2021 06:28:13 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=82EyQ5Xr4Rr03Me9GioAsIl3kdlCseeiWrWJ7DdI8oY=; b=uuOzxM6R1YYwOzIKp406nVeNQQuwAIxQkAXbXnas68LZqWOpA5PYjI9PKiwTlkYGs+Pr k7rxISKZPiOsPNf0pfMtZTtMp4nx23074QO6240GwGRSvd0igsm+0RI/ra7/wyP/TT/h v1dBZARva5IVWq4QtP5FloTgbQXgVgiJJ+ACgxtju/FZSWdtf5qC8vEfLi3dIgU3iQWI our+NYLt+5eJYslQrbZ5e6N42+sqr6TS8phJV0fEYKJlEUnCUx+pYKv/H4W3+g9VCpEV hVhMVct7lZv85rFEppKfvlNtSd07q3gvdKZWQeOBbw0331xjjHzx5UbT2cFl2l3Z9M3M WQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=82EyQ5Xr4Rr03Me9GioAsIl3kdlCseeiWrWJ7DdI8oY=; b=eqsU02O2YBXaDTScLEGTu49ahotpAuthplKEDl/KRbr5NeQHXNfKTRnCp+op672lpEtT 60kc4eGqo5zkU8qicQFbgswzk3+7erAoOepLJRzcyyLgvLdI1XjmR0i1YaJ3seWgtORg M90Cg6V1wNTdccVLJwapU5PQXT7tAhfHOLeFttNgvRfqIaFka5Z4DKSe1c4O/YpfcetE 6R2MA/BbiFgB63P+QAt02qE0qVrecld1y6BEIKQ4qayNGvTLAsZF83W2+mocT55tUC28 ozwsC1cWX5qh7cVhM/HUh0HTCdBWcldLIO+6Kdbq0gqIScDQz04ODHkNBna60ob+FwUK bg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39vpkwrnjg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:13 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6G9TB086873; Mon, 19 Jul 2021 06:28:12 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2174.outbound.protection.outlook.com [104.47.57.174]) by userp3020.oracle.com with ESMTP id 39v8ys1vt5-13 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:12 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xnbs7uLDIjRKJnzphjFjpSiTA4w7tHjClDDZdpDgYjU0o6i/8hYCDClnSlnbLrQ53GqudO0GVFW8GaPX11m2MbS3RMWd91gAjHOuzh8kP95Mf4p3bgwJ8wO/OhAKXBk7ZoTE8nxZot2MLp3LxhMHQLaCllvmpC7trdVeeD3cjWnM+mj1QWJt0yVrX2Yb6Aad5fM/kkucgMCuW4IfxDA0rKhdPpnfsWskWMAax3lDzzi54m7FXcdNGapIuA313ZoymEYbR9aE1PsDiEx3lRN8bk+wfKe8BqVi4tWm9MssyZPsdnuDu72+5euqFzgwbKfe5L+cSUKW6gzYS92wIPutAA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=82EyQ5Xr4Rr03Me9GioAsIl3kdlCseeiWrWJ7DdI8oY=; b=ULpEoFidy4XhEzKlmCDtYVORBKV9EzEvtkjI7Wye4Wlge/Ec2807daZYjaB6RDZO8ybqsQ2kdCD5rIoXQiusA4GEEn+SSTEAThYYnZa5mpL0uDCKgr0yDeaB2c0Pf78GOPTW3CTLyratKNEAzwMnqxOcQyPjmKZXq5ttTK4rbm6JjNPadyNEsqCytUM3Dtod1xz2vovsLVhqc4/cYs/wf6HQj21jZcEG76PxgjVOgDZAwUfz7sutevC54kMe9xvBrVRmAedD9d+g4UTWVzkweVBj48YWHjTBZAlMMllY1eEK1xPMltMlOyfSfyFSGKawOIpwnIyP+DaLi/GQ4MAwyg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=82EyQ5Xr4Rr03Me9GioAsIl3kdlCseeiWrWJ7DdI8oY=; b=rX+Mbn3P1ImJU+y/TeLuNVgVcWBi3jM9EAxFmEt10UnKWOg0m11ilqUGpHF87ycqTO6/O+LSuUUHmO3wAR6RTNPeHifBqEQRFvctsZ3spJkZBA0vPIwqB/B5mYKgdZTeeR5HkqQzLZc8t3yvgg2ND8ihLZK/MGzKwqsg3aRhtHs= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:10 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:10 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 12/19] vfio-user: probe remote device's BARs Date: Sun, 18 Jul 2021 23:27:51 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:09 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4289201f-0749-4f00-3112-08d94a7e60da X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rwjYowbexP+F9M6DDED0AZElEeUw7R3wYQ2SLsRZ4sqszjWrejVXfxGBC7GEIxrPpDuMs/nvEbbsCcqhdo44Qg0ixVViF2qUjnrjJGf8g81lFzFbJJa+Y60I9KJ08HZ75ShxjcT8le9jnDKT62YwixnzjRre6oosTKR9T21kNJcOJY9p9gNDBoNRVynDjuBCvwOv46iJVdCOhG5vN6SGZLY1rT2CVc/nYqXbDzqztl3fto4xiBu1KpktoN7duz0evgwNIGR57Ls1zlUhQUvpX+plwseeRkB1osy72K11TqZVG07TCM8m+Ty6ypERxC+gvjlm7oOAVEFYdSXZBb8ClOxNd+XC1yhSzy43CsELyC29Zt5Wb5CBHVvdDNz/0tGzeVeiodo3i/7SYotIpnJ7t44FNrkgx8nlWJY/ELwM5VA9kbzIjZfai1tdRBzTDzerhnE/6/FRQ/EhzSYyrfGWVY1aL+/oyLrkzarjr1RYiAJ4YPzXxdk6dqA/AtLHBVEFoFcSQDyhiJaSIjuNajbhtd66+VaFpE2rR+XGcyRhKrha8EKkOKECMgb9Gfez6CSx8tqIDqZlSud07fSxU/cDyahIilfsbDINTHmcw35REu2HkEeUr+At6eJspzSVZt2k0i9rv+3kyXYiz5XUck4tFQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Ly/zJSCV9PcV4DYxkM6a9kSHuEL/vVAeqUkjnrZGg5BPSxonfC0DuqvEvPN5H/vGOV/3NiGO9IT6ZflgYJD23liM1bFAg2yHbpQ5VhRk0l8RnjHhBjvwuMh6PCXSx5xZVCTUmTdhU0IjWDzbMynzT4fwGto0gHWWYpt015z6i3/yw7VneD92yE1ZGR+BJmOpBuyNwlH0aQqZgGa6agQOq9glkMtkOw28kxv8SVzH1FtwsfMoBADfIHykR+O25aAiE4avT1mrcWLKGzlGPQNwaVV0yKnu7hl4C9aJqxn7gaoginuN0XdFUJVrxJhcFkgv/qhAlm8mo3Wdx3MOP3PrmH1We54zjyjJNl40yXTChAqQNU73eK7T0EE/+xTCivKTk5a6P1jV99FEFT+II+c959fke8rIfhMcObTcGBTnY+Jqkbbzs1bv/mSfshWrT6M1I8lsC8mtWtkdtNEY1dLIt0xHZsVhFOg1rB55VP8r0qpZp6RUQ8Zu52iRn23YZHB7KtHKR3SKU1eyOCw8U/YuCnvGv8C1qRacKD8uOEjkOU7T2B2u01lsEvBEBNhF6K/Sae1zvKiiQwAUvo/otVLzQELa8wCE4Uh8CBhCZM19Gb5lJXHYf38stsIMqqkyrUSWPw47kiJ2ASb+ztLfm4dhAuxupludaxhp5Hfyv1evyGZoVVNVgA6L3LOoQBQ22pIGKrR53n342su3sZXzJSbuoJIRv4+ia6fHOLqBPOkzNmyBhz2rn+YOOli1byuImdzt5S3WnDfD5NSiLgHJl27EpV+B+OqeaMGMLCvJk5RHpVJ3GrlJL8ltZMCaYVoA1qwREdYHQJqEpMwMKtSyd2adQJY86pgq5OzeeJjS6ZrEz9l1UYxEFu/GSOUb9FTBFwLBVAYpWYcVqimp7pjlUPgfSySwtxz4UvjjR9dZfsOgn1giQ5YXxTUyTMFfgelb8GxYLj5ZhLxlxWnG5QkdyIAq9nlh3qHxgqGpLRs0Qh6rrs1YyU8kYJQqCypf8Vqv8cKHcGY0AirCYNBWLtxpGedR7MQyHexyQaUbjrRGpFh3neklZ0ggpLhjP7hP1OeQmpLyg1H8pk5f+QE+pXu3As5BSkfnJo1Qr+KmTeDKk+hqVmFpHZr1KVcxv7luPfrQf4erXVptaQEMXRU3yCOKEQszNql3XfX18aqwv/v4gjiMgVwk7J9PPkMUvri1kbVix20/rBXizDSzN2uG9eRYwuMONXcyCVmafXODopepvSulnIrcy7uVBfFSNebNX1raiEHFaKs4/GlpS51f6abSOub70leUiAII4S/T1JtwMlssW6HtLy5eKvtj3MDZDtI4IWPlZFzLAv/PuRbgrfFQS98JATVLWNraaikzoZs1l4aNEGxQm0G7V5KoUPnXCafopXSO X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4289201f-0749-4f00-3112-08d94a7e60da X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:10.1772 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kDF5Wl2DjMG2hXq6vpITFqxuG697UEHC0fJXApq5+isMJSeXQFGCnDPB10xj0shgHo/jtmHRZ4D6X3btR/uR49KNuaBS8yRGlp7cebDaBck= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: MFK7XhlMOJLt1DsFI0Sj6tfBIhXa3Wgb X-Proofpoint-GUID: MFK7XhlMOJLt1DsFI0Sj6tfBIhXa3Wgb Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: 0 X-Spam_score: 0.0 X-Spam_bar: / X-Spam_report: (0.0 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.c | 32 +++++++++++++++++++++++++++----- 1 file changed, 27 insertions(+), 5 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 054e673552..a8d2e59470 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -1619,11 +1619,17 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) } /* Determine what type of BAR this is for registration */ - ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar), - vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr)); - if (ret != sizeof(pci_bar)) { - error_report("vfio: Failed to read BAR %d (%m)", nr); - return; + if (vdev->vbasedev.proxy != NULL) { + /* during setup, config space was initialized from remote */ + memcpy(&pci_bar, vdev->pdev.config + PCI_BASE_ADDRESS_0 + (4 * nr), + sizeof(pci_bar)); + } else { + ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar), + vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr)); + if (ret != sizeof(pci_bar)) { + error_report("vfio: Failed to read BAR %d (%m)", nr); + return; + } } pci_bar = le32_to_cpu(pci_bar); @@ -3442,6 +3448,22 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) /* QEMU can also add or extend BARs */ memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); + /* + * Local QEMU overrides aren't allowed + * They must be done in the device process + */ + if (pdev->cap_present & QEMU_PCI_CAP_MULTIFUNCTION) { + error_setg(errp, "Multi-function must be specified by device process"); + goto error; + } + if (pdev->romfile) { + error_setg(errp, "Romfile must be specified by device process"); + goto error; + } + + vfio_bars_prepare(vdev); + + return; error: From patchwork Mon Jul 19 06:27:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42FD1C12002 for ; Mon, 19 Jul 2021 06:42:59 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B8AC261182 for ; Mon, 19 Jul 2021 06:42:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B8AC261182 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:39914 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5MzR-0007jG-V1 for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:42:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33574) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlR-0004Pv-SV for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:30 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:34690) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlP-0002LH-PK for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:29 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GHnA029063; Mon, 19 Jul 2021 06:28:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=DxoLnpKSJiheY0iFOgokE79+y4SVd9S5Gz8SOvfqQJg=; b=CATkaWCuUupXbGEvCouc7XgS3otZV1sZsg5akRQYQX4DVoupp1vS3jK9pHAJ0fS0w88A gorerFgnXmyhQ+nvEOYpALFE0yQ7+gNya+7moT2w0HhUKc5DFMCbm12MVhfuyu9iaT8E o1TxxKKiDRjkNfcqY1aVyBZ5yA8qNk4BJ67X3C9wg/UOcHgnouDkc3kX/TYgApvkkZLw 29umidayqGPFeKhOsMuj8gn5wnmgcpmO5RwKGbjjn4mileOQyJylEnU7mX8Ge1hyajXF lsbbLOxDBu0PIdTEUQ+K3VLxbOtyAmDXX49/bKO0lkcf9hY3UqLsjRRS7r7XHVH+5SzI AA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=DxoLnpKSJiheY0iFOgokE79+y4SVd9S5Gz8SOvfqQJg=; b=aMEP6LRJ4O3Gm8att6cVGm9HFy+rofFPu+zn9WE3nT7nsO50mvxRKzqQ8kjc0AV9lW+n rJ8NVTssN0r4NCTNnkZLrwcvomBAqhAKeLuGOEm229u+iYUDPBUMnwZSzV6BusnDo96l KEtYUdtV0C0UGMlsfOZhoMdBbiRMnB3CV9Ck7tBC7qdf7U9Os9V3jgu45/zXp+zNRSer RGmsaq7wu7X9KQJLQe2/yZlQyPtLa1RLFe6cB/2CqfBRJp6XPrnhUr1LTah/KOyGb0be 1TLuCAHE/Kf70CcWhzQNqBICI8PvWSqwsMQ+SL/vk7BogiLJRX+9KPtuWuxscXmOM4u8 5g== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39vpkwrnjn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:16 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6GMKR099799; Mon, 19 Jul 2021 06:28:15 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2169.outbound.protection.outlook.com [104.47.57.169]) by userp3030.oracle.com with ESMTP id 39umawam46-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KKmw+ta4VzwLaqPgBjwjXKq++8y68E7H3qDS0bozYr1Dr6zwfXM8UTD8Tdy55t5Wi74DD/Ct+ZbewNTJ500iQqh4e525cGAH6ZI9X9/cv+lmiZRt3HJRIQvzl/lGJQ0VTbyjduvi0UflYoYC7kfo9z5rjrR/prBI5FG1/KRc5XImpnOyc4RNxMGqMpSeYr/cTEEaMbfzxulm18gkqHTu/SCZCIpS/jVHzY4kvnL5vMvlZRgKTdL/SI6pgzverxfLO44upNy93aex0rkZmPxWEBGqmaY+EVvAhK2NnkturwROySuiZXqFz/peFh7JCJA/6RrMT/uvl5p9REqXytgsCA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DxoLnpKSJiheY0iFOgokE79+y4SVd9S5Gz8SOvfqQJg=; b=gVc/e8BhoTGsmIASyOYZraYM5o4grypE8nEx4Iv9Er17WR5EvPWGfZnnK6CnmyFxSdiaTG/eVpUyJl18Rs/5k9srSDWzupnGQHjKTypvPHVWEoY3E2u8XqbVmuzS0InPYx0J/g62h5nO/6lskfDVcrbiuWR++VdDjKJXVhA/bzVgp7yzhcMYNV3rxXX3mOOOUokJD2fbVpNkwEfW6oyjAhEJI9drL5jq/3tTyuxa+cTV0YeFDV035WrCEjAAvUi6SrRc5YnS7VayDLqneMLCiKyu/wUydm1UBogFOZKtDB55wMYj8MBj76Yfb+MbFuB+shI2xHBNC45+oxJdt2y2IA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DxoLnpKSJiheY0iFOgokE79+y4SVd9S5Gz8SOvfqQJg=; b=bG/Oiy8NPN6ouOsm/aezztSAJkPhAS1/Zi5n1YE7eW8irj9ub/DFRI6lH3ZQFdTUGrPxJpq5BDItii/HryHkc3OfsG7ceCXoVDuHez2WWKoWB9Y8sIBtogQLYI6Ra4DWhSED5Rds006GFghbVeNunHSXzYo1ts1V5sctVFs33QA= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:10 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:10 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 13/19] vfio-user: respond to remote DMA read/write requests Date: Sun, 18 Jul 2021 23:27:52 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:10 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: aac365c5-bd72-43c0-0466-08d94a7e612b X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:813; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Cv9GnLygtvJKA5k1BgtkQX/0588GMgm1rEvzcZZxjfIC9LC4+Kj/4FbaKsU7HGtNXojL1XxFICmeoD960zhBtVWMM0oYBaClt/tJf9aHtaPnDSBlqsdWuT83jrnrJYrznJdZYMrkETuMZlRXTSEiMpyxJFdN7QpDbsGVB56aNZuyZDTzPLJE7m4KwL85H0LbS8/0l+aGzTCEC41gLoHrIoTUuQXzc0dkIAMkZWSEUoZ5WBNLoPSeTWgAYcagIQWsWxPzqd7BFRFQBbTX5h0J4i0w8yAyq7hkY98qUNtCw59GTdiOV7rl1NkV9ZmRT+Hu2teRH1Yd00VIPIdf2GKFf26jg3p2+GeuGoBZQ37nha4evnKUMoNsiH9MGGCThrhlBrkQcaQK2qem4LP+TR66Q+tmMJvfBo0NzF7/2xXCVFpmPO7BBZcWUtAHghSnBBLOH0PPqLw4sznHPPZpOaB62jDy7Rnczp3Ah8W9OkzafDuwDDqF0iiDoZz7yU4vg/Gy2PNBNghqiuOEKZ4l8x7/m/rFgALuoLdSlbd3DGWybdclO0zsPXN1M6rDkckXj3wKr9EFOmOwB3xB6WUrxzFQgBYfcAJnVcTgtJ0k8fcwECcWpLO6/lju4tNzSbrix4aEKFu85NvTUmnkFVHCEhAWKA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: Qkfl34IV/xg4YTyOonoIJVu1sOrR54QFMaWvyMki0QiVMqRQw44gz1Sh01zTArRrop5rQB7LQT486xzq6Lu3F/tkUvCzbTy00wa0v1d/N17lgAPbF0OBsL5vb04YjyvFRyRfcxYGKmQyOgQxYw01R/mX/FBlk7SEKMtFLeWLBggvdRsOT8+Hk49Ftr8XrxO6LStwIaIja+9uwAFxJjsZeGlFTCmNYkbc3A1aVnNTK0q3D6+MH921HjcI79B181D19evyYuFSj8Vgm1hIfpxwwcHLb25kQPoYK6PS6huWSHTGdxWCnxMaDj4zKbuLgJBhFl65JEVgqwaKDwoIIQnS5sPlRs6G2g9b5NLk/Mv1GVysr8pnK2IKa5gPUOJ11mmlcV9depIDAK2zY8AfIDxqsL8NqwEHUdCaNKH4x5q33JiK0WfttIAIjBAiGsPp7JOiAW51bWBAakuyn1Kg6u+hg1K5+SWdeT2KiUtyZ+83Sg+GSN+han3kee7wUU5HOLXa3Ht6nrkNZdH7ry8dOcnFXI3lLzn177GpugxtvbUaE9EJ6Zhdxrk9VqOzZHQVoZBR3DGwn/A0nUrRxZCcVIq+I2B6H5nXJlXg5M1BScjCp4Q1A5P6VwVzPWR+7uz5amMa2S0RBahoQtb2GkSAIeY5bI7mMURuU/8yr4OhElmT3y/kLj+QQtpI2DLso2T/wJh7AorNXXP1k+0BIvX7DG5NXORfMmhSjOzZC3N00XsOmxHDt0ZNZygm/iWqgC+hpO6c2hkVWM9ZbD6bugB0zyU3FkMwEKfKqpelnGfUK1uHcwsCLVP5Oa5Cin3p2kyFiYR3aysCPg+8x8JGUm563kiNo7DtDC/4bv1B1GxQOUPJdxuzhZFvIxasxmJ/RUl6aRFVJEaB/eADEjZjCuT1cXxwhUamOMIr26CNeEkzlMu3YnoA+6MZJfxE/He/AVTaYjAY3EMTyyPNguuHIACFS+PXo687FD8Xs31LMzcYc05hqHTS7aUrp91rQv7RJ4ujTrB6Qm5gfTDqTVHn8aC5USbnPJvztRolFS4+PZhaQcbGKu7Z5wYCSFYOviZ1TZ9B80+JnpCmv488LcxHXdW9Qa+KuVklb7zc0Hs5Af9FNFgGF1iCUdHmEKnJPbpfHDrKaXAPJntd20DYjwUqJbz9HLFqlIqA2unBk+IG8n0lYpaqZgN2mV55N1wVqSRyq34cveGJQCLUubrmUIBwVdweoLs8gzuSLuKDK7RjuojPe95e9h7lFGdpM1HKfNzh1kEWaaaXjMXUHnHlCaDIXGPJLiTsMjY+XOASfO58nka9eBHh98aDBvFuhffCA8qaOMEkHv33Gofv16osRqpnNnFjzADgyX308sPpR8l6EQPQIlIIjHqQGF++73nOzNUvu3LZrC1B X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: aac365c5-bd72-43c0-0466-08d94a7e612b X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:10.7099 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: R3PNrNVhS8HwHag05b5cDB8XzUEP+8aJq2LB6AE6KjiuiD+B+n09Hs64dmN2IPDW3cYBOGJfIkBYHc7vLTjyPdJkhmOAWUNmkR2LiL4KOjk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: AQVrrLo0kGV9C0jlJnMCOw3y_HDnDmr1 X-Proofpoint-GUID: AQVrrLo0kGV9C0jlJnMCOw3y_HDnDmr1 Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 16 ++++++++++++ hw/vfio/pci.c | 67 ++++++++++++++++++++++++++++++++++++++++++++++++++ hw/vfio/user.c | 21 +++++++++++++++- 3 files changed, 103 insertions(+), 1 deletion(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 351fdb3ee1..d08d94ed92 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -206,6 +206,17 @@ struct vfio_user_region_info { uint64_t offset; }; +/* + * VFIO_USER_DMA_READ + * VFIO_USER_DMA_WRITE + */ +struct vfio_user_dma_rw { + vfio_user_hdr_t hdr; + uint64_t offset; + uint32_t count; + char data[]; +}; + void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); @@ -224,4 +235,9 @@ int vfio_user_dma_unmap(VFIOProxy *proxy, struct vfio_bitmap *bitmap); int vfio_user_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info *info, VFIOUserFDs *fds); +uint64_t vfio_user_max_xfer(void); +void vfio_user_set_reqhandler(VFIODevice *vbasdev, + int (*handler)(void *opaque, char *buf, + VFIOUserFDs *fds), + void *reqarg); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a8d2e59470..7042c178dd 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3347,6 +3347,72 @@ static void register_vfio_pci_dev_type(void) type_init(register_vfio_pci_dev_type) +static int vfio_user_dma_read(VFIOPCIDevice *vdev, struct vfio_user_dma_rw *msg) +{ + PCIDevice *pdev = &vdev->pdev; + char *buf; + int size = msg->count + sizeof(struct vfio_user_dma_rw); + + if (msg->hdr.flags & VFIO_USER_NO_REPLY) { + return -EINVAL; + } + if (msg->count > vfio_user_max_xfer()) { + return -E2BIG; + } + + buf = g_malloc0(size); + memcpy(buf, msg, sizeof(*msg)); + + pci_dma_read(pdev, msg->offset, buf + sizeof(*msg), msg->count); + + vfio_user_send_reply(vdev->vbasedev.proxy, buf, size); + g_free(buf); + return 0; +} + +static int vfio_user_dma_write(VFIOPCIDevice *vdev, + struct vfio_user_dma_rw *msg) +{ + PCIDevice *pdev = &vdev->pdev; + char *buf = (char *)msg + sizeof(*msg); + + /* make sure transfer count isn't larger than the message data */ + if (msg->count > msg->hdr.size - sizeof(*msg)) { + return -E2BIG; + } + + pci_dma_write(pdev, msg->offset, buf, msg->count); + + if ((msg->hdr.flags & VFIO_USER_NO_REPLY) == 0) { + vfio_user_send_reply(vdev->vbasedev.proxy, (char *)msg, + sizeof(msg->hdr)); + } + return 0; +} + +static int vfio_user_pci_process_req(void *opaque, char *buf, VFIOUserFDs *fds) +{ + VFIOPCIDevice *vdev = opaque; + vfio_user_hdr_t *hdr = (vfio_user_hdr_t *)buf; + int ret; + + if (fds->recv_fds != 0) { + return -EINVAL; + } + switch (hdr->command) { + case VFIO_USER_DMA_READ: + ret = vfio_user_dma_read(vdev, (struct vfio_user_dma_rw *)hdr); + break; + case VFIO_USER_DMA_WRITE: + ret = vfio_user_dma_write(vdev, (struct vfio_user_dma_rw *)hdr); + break; + default: + error_printf("vfio_user_process_req unknown cmd %d\n", hdr->command); + ret = -ENOSYS; + } + return ret; +} + /* * Emulated devices don't use host hot reset */ @@ -3392,6 +3458,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) return; } vbasedev->proxy = proxy; + vfio_user_set_reqhandler(vbasedev, vfio_user_pci_process_req, vdev); if (udev->secure) { proxy->flags |= VFIO_PROXY_SECURE; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index eea8b9b402..8bedbc19f3 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -42,6 +42,11 @@ static void vfio_user_request_msg(vfio_user_hdr_t *hdr, uint16_t cmd, static void vfio_user_send_recv(VFIOProxy *proxy, vfio_user_hdr_t *msg, VFIOUserFDs *fds, int rsize); +uint64_t vfio_user_max_xfer(void) +{ + return max_xfer_size; +} + static void vfio_user_shutdown(VFIOProxy *proxy) { qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); @@ -236,7 +241,7 @@ void vfio_user_recv(void *opaque) *reply->msg = msg; data = (char *)reply->msg + sizeof(msg); } else { - if (msg.size > max_xfer_size) { + if (msg.size > max_xfer_size + sizeof(struct vfio_user_dma_rw)) { error_setg(&local_err, "vfio_user_recv request larger than max"); goto fatal; } @@ -779,3 +784,17 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index, memcpy(info, &msgp->argsz, info->argsz); return 0; } + +void vfio_user_set_reqhandler(VFIODevice *vbasedev, + int (*handler)(void *opaque, char *buf, + VFIOUserFDs *fds), + void *reqarg) +{ + VFIOProxy *proxy = vbasedev->proxy; + + proxy->request = handler; + proxy->reqarg = reqarg; + qio_channel_set_aio_fd_handler(proxy->ioc, + iothread_get_aio_context(vfio_user_iothread), + vfio_user_recv, NULL, vbasedev); +} From patchwork Mon Jul 19 06:27:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE687C636C9 for ; Mon, 19 Jul 2021 06:34:16 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 16AFC61165 for ; Mon, 19 Jul 2021 06:34:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 16AFC61165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:46518 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mr0-0001Zu-Q5 for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:34:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33472) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlJ-0004Cn-Mw for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:23 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:33970) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlE-0002KX-Mh for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:21 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GQT1021865; Mon, 19 Jul 2021 06:28:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=OKiTuGzWvSN5SkZe9+4oARYCgcVyj2tZdnAtp/n8nYY=; b=hPe57ahM3zrMnYjBWK7fw3uUhkZCkhrcFK1FhofSTAtULoSF/eAm+gqnYRS3mcJXnUu5 T/HQddx5Cd4tO2lU+4/R2cTQUDuF8AdwoIFKQwDvHRUmm5DJQ6ZE5I2vB9hVm3VFUEqA 5BQvrg1+v0vFAx8aQJB44p7FiYNUvzZDSajLycbOWmReBcm/So4aI8K3KMZVzyCY6HGK hB1pgUxprlJwBXj7cMD8wqZBg0T5qbi3DJ9epgibfVqOwxsoyDE3uQ19lfz/FgQ5biUE 2zYwlNdrCxehCZWnUxgq9de5hbysKjXVndiCInxWIrIgi8usn7OtV1BTvIi4YQCoaZ6X fQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=OKiTuGzWvSN5SkZe9+4oARYCgcVyj2tZdnAtp/n8nYY=; b=Z35A4Mt48AK6eV/0MXemDw+PtkETYQfXgK5epGqZdfgY0O2lODxIeyft7E2+Twmr4AUP dFNUvif+hUPdbwDozkzOnnKWal4phlJoYWhN2ImUGarmrft9dbRoN/602KCWCMJdDAHn mxh8CYzw+kWSexhWNSCsAz7er7Gv2VEBtXEaWv8Lu3IiKlKXk06AdM5bL6dqh6i2/LJ/ hRDdGHPDVqDfxwby1tdIgVWYpipI9h4n6QudOg/r8ebn92YCz+x/iwX0glj1Pgwqwjfk OoDoHlJQohZOgNwvPkRLD1FC1qbx/qH6r1yh5HavjmtUhKqN42rRrkG8BQeFTGk//jRU hA== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39uptrt6c4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:15 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6GMKP099799; Mon, 19 Jul 2021 06:28:14 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2169.outbound.protection.outlook.com [104.47.57.169]) by userp3030.oracle.com with ESMTP id 39umawam46-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jpr2dT2keogzYsZxcE0kUiayYmgX7lEFZE1gDBNTk61hl5jBo1fex1vI9CLE2WnOAaH61wlIBkVplRLAvMtuEJKrRC86buuISJqtdL7SIXzh3oU8MoRAna+BhOmnuXjBQm0tQoSXqeI76Xjk3olas2K5IOASb0AfnmLhD7zfhhsODE7Oy1lEcjFt3G90596oYsCF9gk0U4+AOcQbLL4Gldgfu/9GC/wx2bbj/4kJ3iWYxKAAskj6X9bSjpZUpBk3HRnePRUKgK0Hf/Er+lMXpUzKMrJurfjvXr/0mnel0m0J4ho7gR2oFf/rGiuPQTE4VIdt3RYbsP9toeW/uXHaCw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OKiTuGzWvSN5SkZe9+4oARYCgcVyj2tZdnAtp/n8nYY=; b=YzGi9yNwLl7rzqaASlG4616aO9JXSDD5qo9y9I7CLodo1hocx2eTqio8hdAM24QlxoLNLOcZln/AXtKF8FjCkx9Rp/xcSTVC4xDFygi05O1hfW/GO19z/xteuzEVYIyn2sXnNx9hHmlE0qGg7xjxDpf+wM37Hg9KvjA15phzhMRx0xVsSRAW409LkEGSjBIHaFAiXUv5LYf68Bmjae0PzbL4c4AfLc5gYMCcpkPnuEzN8jmnDqzNntsHguc8geH3Hv0bDb4cQukBVtHAt3JbffXzxxihlCVUSrW87dDd9hhJd+8xywwQk2kjP7hYC96579hqSrkaRYRIE/Lygql8ew== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=OKiTuGzWvSN5SkZe9+4oARYCgcVyj2tZdnAtp/n8nYY=; b=f1C9e08FJw36FVZx6lmpXnWnpXsnmcxKzPVGFibDHeM/D0/2tK01oa89L36lQgnhL4tB2+9Cp00VctnNnrX92wKsep4qZ0O/aas0ZTy42Xr+ZoA/9CBtDSNbaNjft+ljhpH2SpR0+S81PZzaqoNlj+Sh2rlZy7L7NcZwNUxLUCc= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:11 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:11 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 14/19] vfio_user: setup MSI/X interrupts and PCI config operations Date: Sun, 18 Jul 2021 23:27:53 -0700 Message-Id: <89ac29a87ee532702079075ab5aee58744f9216a.1626675354.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:10 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ca2e4434-0143-4940-43ea-08d94a7e6180 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:843; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IgX8ryoe+kZ3j+kx4mdeP/uF8Im8a5ObfwARfBfXqZPN/xQBvbt6tWgmLTXISYwCPGZCJ2aOxmmJqmS3B0nKY8RuKMCr9cFN17y9ZIW3UYx5h0C1AriYBI/6xnYIRxLfaV5uI69dE/aMj0IucIXNfbxqGPVN3NmBnzuVGoKXeLVL9u4DHpawVv+42KsWu/eeM40u/2p0lYDdRv8GKUaHjkegamRpkSLgWq3j/G+BXQSL9A2LUgWVYZoTu7U5WVHyEtq7Rb7iOcV6B+h2HzXB+GlvqGEn/GzSX8D3lpLAAZuUjLczgyHgZcoVaDRBCis8E1DPat3529RrJOq++UUaas7krjQu1bQStwIIdQpPx3GTFbWunE5N3b4OeHSq1UZgkRptEwL4Kq7nfrh1D8pxulYj62MqM4tRkXgFxpfgRu19KiFAFSvS0Apzec4zDYl2WafebX6Drg54O/3ysNUJMB45zTw0GtNFAJePdDhPksNxHrXHtOSA13opd4mzceAqxCiU6H+22HPpnsBRkguNS+pD6+C/DbhbpmPgfx+dyIY2WbShJ7CnL60VjyhEQprs29s7aTtruwvakJ9Lqf2O4wXaUqVqmDmGPLfZjaWcX47XvfhOeNbLVZ+VzSir8pTT9cACeWxplVPJIzEEUYZhGA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: EVlrdfmhz8jjX7ET/6+BIxNUmbc1zxHs63mJT723SH997odq5gI6l4rXiDAp4YJbI5VS4eecAA03gJtRRo7Yc1Y/Ai/H7wgJPf/D4SZh38yRIxJcUOgnnj7wOUQE4MheAnp8Sm1D6EOKbOo3qZiInRHI2SacUmLcAnVfWqu1P6B6sFoDfMpjd39WN1jgLRcoGk+RFHqWgdieBx6pkuR/beqQzdSvq/Xml0jMsM2+eh47fZWrBoAWApAsPcuFwwaykhLSczDPMG581PJfAUs72pp8jh25nyb6aTV5ryXLPAVG1vmXpxbA//MSix4LZRVQJVpup8lM5tjifQj4RFy9sba6eTGS84CsUxrJdt1h1uZDdgs+BNBCEU7uEKIKu4b+aKlKcOWMREeD+bUr0y1bVzi5BZMJ6ui4I2RjbxkVDYns304SgcBO01XZ3biSwwybEuvHC4vK6rLOnTTEl4yQzSMXeMql6/oQf4Ll0M8R5rd4FjOp2DO9hM75URrmTIkpHQdWh2cy+RYqoqV6m1n2Ws0NoIoExgxgX3U2hGWp8g8V4Frsuf3b6dTOZ8oejPaeIq0AZisMvDg7MFStGzGNkW3Kv4SRn3XETvB9Np7fzUhDecX110lZ34QrPezoSG+2OO9JWDBSCd65MtTZvQ0vdFVjbJxXYhlfyfnuU965lUDqXdh48vRmXnSi1nn+gFVmQAgpT+9UEFD7Ba53ldViU+jaw9bw8jQo8ZSeDU/HXv1Q1RsB3mjzt45TP9FfhdXbPgV6oeCzUpDooRIOFOq+8YfYh9aLZWi+M9dzzeTC60VH8UUE11vPNOyiu48WKLP/KsSbqjRomLqjeF9WnpREEC3tePgWhps6MZBm8IA+EWoCwFUVpnYkoPX5Q9M6KlKfFOuzb4GgelEC9Jqfbvblr0cxmDq0vDkAajQ7ehpJcWi10om4VWZaXczY00jb1LmJfVuU7sa6gRqK0FLpoonrW+aYrHiHhBl8bE3euVDx4vtSR+Gvt+72VI5F6i8Ll9KTECKxeLiF5N4+y6f0vGbJ62aV/dG+JucALpcb2teZNHBv32WL9f2g7YKOoWGoWtSj+YHXxVNDe7za/qFZA3uK1Y9pSF3XTWrJ/FyeTIGZf6yuf3HQpr7XLlWapIQ2W6HxeYKm1Xq/adK+a9D4IBot5W7UsbOgoZn6sxZy3i+o6XF47/obKAkp85nrPXirYVsI7r65TB0GE2xGwePsR1nWxqZOuk/4QX7g0S8KcNppwPi6JDdPFv03xr//3UT0PxDfVVJOQ1QlCGmpk/U4UmJbQIRmLM2Jiej3PpNhrsEyccv4S9UodPDEfP65MdcddaXD7qt3gClUfYLna3v9PfUiYCnQ+h2SFpa8f/e7fk0ngMuYQ+6LZ1I7JMM9kIH9QWPS X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ca2e4434-0143-4940-43ea-08d94a7e6180 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:11.2895 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GAtFNQwZ3TttKcYIrUSDNdfAD9MVmD9ZGhWSteq6smpR/MheYV25mj7lBYJnFsIbw5Sa00oKPcykfksJ/9XLtxH9THThMVbwXH2Ez+zG3dw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: Li6I39BQa4Et2PTxlr0Hpd-nHbvPW17W X-Proofpoint-GUID: Li6I39BQa4Et2PTxlr0Hpd-nHbvPW17W Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Send VFIO_USER_DEVICE_SET_IRQS to setup interrup configuration. vfio_pci_write_config/vfio_pci_read_config iforms the remote server of PCI config space reads and writes. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 14 ++++++++ hw/vfio/common.c | 26 ++++++++++++--- hw/vfio/pci.c | 71 ++++++++++++++++++++++++++++----------- hw/vfio/user.c | 87 ++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 173 insertions(+), 25 deletions(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index d08d94ed92..afb85952da 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -217,6 +217,19 @@ struct vfio_user_dma_rw { char data[]; }; +/* + * VFIO_USER_DEVICE_SET_IRQS + * imported from struct vfio_irq_set + */ +struct vfio_user_irq_set { + vfio_user_hdr_t hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t start; + uint32_t count; +}; + void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); @@ -240,4 +253,5 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, int (*handler)(void *opaque, char *buf, VFIOUserFDs *fds), void *reqarg); +int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 52a092e168..9b68416599 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + if (vbasedev->proxy != NULL) { + vfio_user_set_irqs(vbasedev, &irq_set); + } else { + ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + } } void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) @@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + if (vbasedev->proxy != NULL) { + vfio_user_set_irqs(vbasedev, &irq_set); + } else { + ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + } } void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) @@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + if (vbasedev->proxy != NULL) { + vfio_user_set_irqs(vbasedev, &irq_set); + } else { + ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + } } static inline const char *action_to_str(int action) @@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; - if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) { - ret = -errno; + if (vbasedev->proxy != NULL) { + ret = vfio_user_set_irqs(vbasedev, irq_set); + } else { + if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) { + ret = -errno; + } } g_free(irq_set); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7042c178dd..3362e8f3f5 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + if (vdev->vbasedev.proxy != NULL) { + ret = vfio_user_set_irqs(&vdev->vbasedev, irq_set); + } else { + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + } g_free(irq_set); @@ -1123,8 +1127,14 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) if (~emu_bits & (0xffffffffU >> (32 - len * 8))) { ssize_t ret; - ret = pread(vdev->vbasedev.fd, &phys_val, len, - vdev->config_offset + addr); + if (vdev->vbasedev.proxy != NULL) { + ret = vfio_user_region_read(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + addr, len, &phys_val); + } else { + ret = pread(vdev->vbasedev.fd, &phys_val, len, + vdev->config_offset + addr); + } if (ret != len) { error_report("%s(%s, 0x%x, 0x%x) failed: %m", __func__, vdev->vbasedev.name, addr, len); @@ -1145,12 +1155,20 @@ void vfio_pci_write_config(PCIDevice *pdev, { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t val_le = cpu_to_le32(val); + int ret; trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); /* Write everything to VFIO, let it filter out what we can't write */ - if (pwrite(vdev->vbasedev.fd, &val_le, len, vdev->config_offset + addr) - != len) { + if (vdev->vbasedev.proxy != NULL) { + ret = vfio_user_region_write(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + addr, len, &val_le); + } else { + ret = pwrite(vdev->vbasedev.fd, &val_le, len, + vdev->config_offset + addr); + } + if (ret != len) { error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %m", __func__, vdev->vbasedev.name, addr, val, len); } @@ -1175,7 +1193,7 @@ void vfio_pci_write_config(PCIDevice *pdev, vfio_update_msi(vdev); } } - } else if (pdev->cap_present & QEMU_PCI_CAP_MSIX && + } else if (pdev->cap_present & QEMU_PCI_CAP_MSIX && ranges_overlap(addr, len, pdev->msix_cap, MSIX_CAP_LENGTH)) { int is_enabled, was_enabled = msix_enabled(pdev); @@ -1456,22 +1474,30 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) return; } - if (pread(fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS"); - return; - } + if (vdev->vbasedev.proxy != NULL) { + /* during setup, config space was initialized from remote */ + memcpy(&ctrl, vdev->pdev.config + pos + PCI_MSIX_FLAGS, sizeof(ctrl)); + memcpy(&table, vdev->pdev.config + pos + PCI_MSIX_TABLE, sizeof(table)); + memcpy(&pba, vdev->pdev.config + pos + PCI_MSIX_PBA, sizeof(pba)); + } else { + if (pread(fd, &ctrl, sizeof(ctrl), + vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) { + error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS"); + return; + } - if (pread(fd, &table, sizeof(table), - vdev->config_offset + pos + PCI_MSIX_TABLE) != sizeof(table)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE"); - return; - } + if (pread(fd, &table, sizeof(table), + vdev->config_offset + pos + + PCI_MSIX_TABLE) != sizeof(table)) { + error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE"); + return; + } - if (pread(fd, &pba, sizeof(pba), - vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX PBA"); - return; + if (pread(fd, &pba, sizeof(pba), + vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) { + error_setg_errno(errp, errno, "failed to read PCI MSIX PBA"); + return; + } } ctrl = le16_to_cpu(ctrl); @@ -3530,6 +3556,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vfio_bars_prepare(vdev); + vfio_msix_early_setup(vdev, &err); + if (err) { + error_propagate(errp, err); + goto error; + } return; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 8bedbc19f3..6afbde8ba8 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -798,3 +798,90 @@ void vfio_user_set_reqhandler(VFIODevice *vbasedev, iothread_get_aio_context(vfio_user_iothread), vfio_user_recv, NULL, vbasedev); } + +static int irq_howmany(int *fdp, int cur, int max) +{ + int n = 0; + + if (fdp[cur] != -1) { + do { + n++; + } while (n < max && fdp[cur + n] != -1 && n < max_send_fds); + } else { + do { + n++; + } while (n < max && fdp[cur + n] == -1 && n < max_send_fds); + } + + return n; +} + +int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq) +{ + g_autofree struct vfio_user_irq_set *msgp = NULL; + uint32_t size, nfds, send_fds, sent_fds; + + if (irq->argsz < sizeof(*irq)) { + error_printf("vfio_user_set_irqs argsz too small\n"); + return -EINVAL; + } + + /* + * Handle simple case + */ + if ((irq->flags & VFIO_IRQ_SET_DATA_EVENTFD) == 0) { + size = sizeof(vfio_user_hdr_t) + irq->argsz; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, size, 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start; + msgp->count = irq->count; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + return 0; + } + + /* + * Calculate the number of FDs to send + * and adjust argsz + */ + nfds = (irq->argsz - sizeof(*irq)) / sizeof(int); + irq->argsz = sizeof(*irq); + msgp = g_malloc0(sizeof(*msgp)); + /* + * Send in chunks if over max_send_fds + */ + for (sent_fds = 0; nfds > sent_fds; sent_fds += send_fds) { + VFIOUserFDs *arg_fds, loop_fds; + + /* must send all valid FDs or all invalid FDs in single msg */ + send_fds = irq_howmany((int *)irq->data, sent_fds, nfds - sent_fds); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, + sizeof(*msgp), 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start + sent_fds; + msgp->count = send_fds; + + loop_fds.send_fds = send_fds; + loop_fds.recv_fds = 0; + loop_fds.fds = (int *)irq->data + sent_fds; + arg_fds = loop_fds.fds[0] != -1 ? &loop_fds : NULL; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, arg_fds, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + } + + return 0; +} From patchwork Mon Jul 19 06:27:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0E9BC12002 for ; Mon, 19 Jul 2021 06:34:11 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5E0B561165 for ; Mon, 19 Jul 2021 06:34:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5E0B561165 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:46432 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mqv-0001X4-UD for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:34:10 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33474) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlK-0004Cy-GM for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:23 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:34346) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlF-0002Kz-6Y for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:22 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GPpo021851; Mon, 19 Jul 2021 06:28:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=+wkEwY71OjRAzKi+pA16TE5qyBm+GnCCN5j0NoNrIsA=; b=quwBINIyZNbBGHBMoBWajmwts3kOaQVe+lEdlbj/dngCzFTI6uDstVfp2Mq92FG6z3tn 4lEI2t8knyLbKbLMgQXi8vBLPvAUzJ+vpAb2ZqXcp9vPWF/u3zdavkiBDp1Prxp4HEbI 5Ogd9W6S2k2Hyh6pkDfc4IfZaExD91WM/dZwwJe+/LL3y/iGa9iBL9UZoOGBEqP+tzcj tULrNG0DuJyBR++hXT9hBdjjv/qqOWXG4Ah/LQs0SRyg4Bu+PKISQOSuPzJyGskvdlU1 9os7esBhwpLaBwHESZJx2EfPjSQkJg7xsS1HszkT4L3FVf/xCnFD5QADGnmvpXJ24dih 0A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=+wkEwY71OjRAzKi+pA16TE5qyBm+GnCCN5j0NoNrIsA=; b=VwD0YExCZW8HXPNvX4V9zO5W5S92oswZMORCUvu9p7UiyDsko5+4NuC7BEhNGemNzwvB jnw8WBEhEIwOXpZJLVJyO3x5FnaQBHNyJP6b+W05WSWdxNyf9rV2Go0EB7c778dUaxkD AC+Fw+3MwtcJMhjxKJxBcJkGDwSr7pFwbPVw8eVlFngOeXT85hZfKXPIb3JYtt1IBgxd RQXNDYI8hYD6awkPk/BNuySm1W5a0GEPuPcEBVBvltCbtsnKq90UnUbclDUKiInygBRq Mn6lZB6/XwLJgLirzyo/lRioFaFkaxMIR78BLTPTZ3j9mshVLQKzFqTBj+LVZKeXL+4y Gw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39uptrt6c5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:15 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6GMKQ099799; Mon, 19 Jul 2021 06:28:14 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2169.outbound.protection.outlook.com [104.47.57.169]) by userp3030.oracle.com with ESMTP id 39umawam46-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:14 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XnxrqJfW6DEajl0SWTDfWe85DZwB2qcqZL2jdB6VPoZ5sYnKSZWMo2qvdBBbk2nsR+XFWubS8WzAyJXHUOWT9qsjhpGFca7ZRUl0rwr3uk2XPu/IShqymrhndIhoCcgvTZhYOxtUpIA7NlUAqHrs4iWu1QtO4EY0dMC6Www6E+OxAnr7u6n7h7pQFjuzw+ZQQmf+7wicgRhJ6PvjuHLAzCa8XBgWmapheQwfwdAf+vKsVEMcgv7TXIf2BqJoELNhnw3NTytYGJoeACTZWuafgmLzNRT8LDIGIBrjUp2IRyRF53kvdvYYzen1jBSblBYaoT1Vxv8TwAm1Q1x4WJrrMA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+wkEwY71OjRAzKi+pA16TE5qyBm+GnCCN5j0NoNrIsA=; b=T1QH51TznEdzVy5x8AaWYzEiA17WlP1YygDEztwJJ8g7iQODpomJvsZMyE+1C0iC4jNFA5t5y4TavrXZrYVP0p/bFYI+CSvpwBtw3ec69tOiZZ6A/Y6Ll6ToadgnJ5aXhT8AE+Nz4VsHHgd/g4VwMfYZM8jFmsx1eZ78y/7aSgVu+LozyqLaoxBWbvKuVwZYiZ2HLLNzETVdYrhIGe7oZWawefwXXszr+Elk4zOEwEBjlyIE9oUtSJsCcht8BWZvL7FSYxfExdxB2RbacjP3ED8RC93vxZ6yMUHlFYD5aLmDENgmZ1SM9mY+GgryuyFnyYrsFKKl1blkOHHGh3PjNA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+wkEwY71OjRAzKi+pA16TE5qyBm+GnCCN5j0NoNrIsA=; b=LTXFnXKP1zldcOegW2hCH+68thhh2LGSVOb5SbD5gtYbZmLfihRV5k95A+8PmA3CwEiZ7G8rpkRi7Gv2UcVADY6xg0BVZ6mFv99y1ZxiiXT0N8E2x7pG5ei+gsGkJP92chfKVwtomHeVZcSQoVHLNfnu/1FG6Rvzmr8/F61jNM0= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:12 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:12 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 15/19] vfio-user: vfio user device realize Date: Sun, 18 Jul 2021 23:27:54 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:11 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4afd5445-8057-4264-8dab-08d94a7e61bf X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4LlFwuDV/Tpk6ZCnChvGpINfYB/IcjnxPnRo3jRxYE+NamY5j+EmN63+jT9KIqteSePIbhXXc5eExM7+CCZBdJuBDBHBH3M63/KYpoqRvrLv7c4D3OhPIIz0vdly44SCnLItE0yPCcRsF+D3dEaHF05aoeDqSHWJjrwhvw/nnRHTAAJyc3xuBeOuOOfEo7UcVardMATGBTxUiBuG48Mrec1M/uVERy5KqRg+Qw5VTujlEl4c66iIqf/Ga3WONoZHpqB2zPYG5J26HwrICNMNcmLkwaNLcE6QChHton3w/qCSoHuFcMYk/BEtP7PhcnzSqwReBu4QkkMq1bQcrxIJtrE5OdUC2ewCLRH/Q5iWWiiB3Uyl/5TCvTanu4YFmoDkCgYhcj+SVTzfqDIHi9ndHC5ePsLEvCffkHb6E7CWXlYxF+5OqL6ivOC/k+cjPUkJFmE694cOugUsgMQeqmjVjbo3+uM6k8ITxlCm27pQortjoOOjbPAjIbrYFBnNiZ+L0OWfDs1a+QKsDx5jzSruRg9px6yYQgxTeEE7V3bSIKBFZHueSpif7+vunCRj0pziCVj+qpYjxwTxbOGduFctwUPlBT/po35bEvDima/9izhIbaMJPrWX4cqgZQi9iu0UERS/+5u47czHbTjWqnZT/w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: q5hpRdy0ufbpH36EtnNYQCWA4/3lR7cdsTNyKyPPfiQ1bnxd8f48h94Fy1vJS1I9lm3gXfwVCqszthojZWQ+6iaN1BVH3hY0GG+w5nqkYMwGkU81UtiZUt2XheURiSaFBF2bykWlriVRGFYDPwEClTDeYbAUIiPVdkNBxV1oX/2sA37mUFXuyeb6tK8BLLFdr3iCc5fTMC4ps1861wMaZdWJzWhc1bD+eFbFVl8km/g/vR41/F4ZmH1U6pEdopWd7WxSKgy8zq7+DZoUivSxRLwZ7iKSlB53IeTwnideh5m0M5kzxogVVHU6IjbS1ZqXRCYWlh9x001UxoJPTy2GjLYuhTNeihnqAxFIQzw9136unOQSga6DHD/GJCeLTkfq4ZOCaHq8OqWz62xCAKGs7f8Uz69UJalaN6aiEEhTbhbdNJd21oR9/RzxAY1vc9E8aSSuduJhCQDX3XOuOkZSLGc+kfg91IuHqcwEA4My7tBK2bup+IjBmw1CQg4596ADRaUgg2lBaRNjqFp47jpDxw8p+diowEora0DnOXV6yC5zOQQ4VDIiVyGe6ArPZ2ImzWQ1gHGRWnCrEhjh/g4mNHpcKWxNHiPzLZFCpCJ9OKBukTMvSmQuUIox0xSikWKh59U7vFQbESSG5smi7E2dOKOPrZX6LE0hFJf9n9rS1rVIpsk/zX++D5ucBcQ9WetzizuJf0atjOjLIXY4mEyDRbkPC/1uc1NLb5Rt5AwR35aJ/SyARugLFIkZh3w6la1TPPyQrmhPkPnPiHIICFabgTGz4ABKjnZBEzi47oMuUlrNnm9otzWMUdonMLqCPxIftTNS26liiSpZb02QRbR0ieDP3cUdvsLgqMiT63SoDgF+HlKr+paMFjKIcjEUn2p4LRYhnH7+z6Z1C6zI84GVUe5Mg39UilC/rvCwNwY4ZVTYylX9CCHIe2bJPvbwfwJr+e80Pja1937FGVTeSjJ0pHjO2MCrTEwRGRRCrsAyqsTeGd5j2/yZZ2zBSFEL7k69k9lzawEOiANWsbi+5eT3ASY3a63Uxf9GE/pZC6hgurtq6O0CK9dovaPdpji9SqAtx1Tc4FAM11P8s/8/E33GpCI1xeWJEUcUImdU4pDNkEaHoR8Ivd9xRvJcPdLjD6u7UZmpSBl+51kvqpmGOUu7FOqITINlk5CJJNERcsQcqsucUX8lg+4Zvu+MdrP4wMx0bpRSNxHG5BoxaHl19wUavbqUsUzU3Bb1UVWrAz21CZKum9fm2EZGc0NttToKiK/B72HIgqdMXIKqoEDoKmk43EAx5MMSuaUWGtTEwG4gtu8/297ZVA7Y3Agg0vvDlu0Lx4zb7gAciyA247e47W8RuWej02k5/Pd4kw/xyuN6hwbsX7g4DhmjhLmVBfricxeg X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4afd5445-8057-4264-8dab-08d94a7e61bf X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:11.7163 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /DtQBzcLSlTJlvo8fXHoORhkzC/1UFQq6M89AD6Q1fAnCAegtkDmBin+CHe0ZXNmeTL898gfU+BiqR7+P/ha1hs70uAQ8h+i3OAM1gKsYj8= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: N8B1VijTO6yk1AHp7H8z0UNSLJ6t3ctd X-Proofpoint-GUID: N8B1VijTO6yk1AHp7H8z0UNSLJ6t3ctd Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Setup INTx interrupts and a device region info cache for remote device info. Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva --- include/hw/vfio/vfio-common.h | 1 + hw/vfio/common.c | 33 ++++++++++++++++++- hw/vfio/pci.c | 61 ++++++++++++++++++++++++++++++++--- hw/vfio/user.c | 20 ++++++++++++ 4 files changed, 109 insertions(+), 6 deletions(-) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index d7b717594b..688660c28d 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -56,6 +56,7 @@ typedef struct VFIORegion { uint32_t nr_mmaps; VFIOMmap *mmaps; uint8_t nr; /* cache the region number for debug */ + int remfd; /* fd if exported from remote process */ } VFIORegion; typedef struct VFIOMigration { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 9b68416599..953d9e7b55 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1571,6 +1571,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, return true; } +static int vfio_get_region_info_remfd(VFIODevice *vbasedev, int index) +{ + struct vfio_region_info *info; + + if (vbasedev->regions == NULL || vbasedev->regions[index] == NULL) { + vfio_get_region_info(vbasedev, index, &info); + } + return vbasedev->regfds != NULL ? vbasedev->regfds[index] : -1; +} + static int vfio_setup_region_sparse_mmaps(VFIORegion *region, struct vfio_region_info *info) { @@ -1624,6 +1634,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, region->size = info->size; region->fd_offset = info->offset; region->nr = index; + region->remfd = vfio_get_region_info_remfd(vbasedev, index); if (region->size) { region->mem = g_new0(MemoryRegion, 1); @@ -1667,6 +1678,7 @@ int vfio_region_mmap(VFIORegion *region) { int i, prot = 0; char *name; + int fd; if (!region->mem) { return 0; @@ -1675,9 +1687,11 @@ int vfio_region_mmap(VFIORegion *region) prot |= region->flags & VFIO_REGION_INFO_FLAG_READ ? PROT_READ : 0; prot |= region->flags & VFIO_REGION_INFO_FLAG_WRITE ? PROT_WRITE : 0; + fd = region->remfd != -1 ? region->remfd : region->vbasedev->fd; + for (i = 0; i < region->nr_mmaps; i++) { region->mmaps[i].mmap = mmap(NULL, region->mmaps[i].size, prot, - MAP_SHARED, region->vbasedev->fd, + MAP_SHARED, fd, region->fd_offset + region->mmaps[i].offset); if (region->mmaps[i].mmap == MAP_FAILED) { @@ -2524,6 +2538,23 @@ int vfio_get_device(VFIOGroup *group, const char *name, void vfio_put_base_device(VFIODevice *vbasedev) { + if (vbasedev->regions != NULL) { + int i; + + for (i = 0; i < vbasedev->num_regions; i++) { + if (vbasedev->regfds != NULL && vbasedev->regfds[i] != -1) { + close(vbasedev->regfds[i]); + } + g_free(vbasedev->regions[i]); + } + g_free(vbasedev->regions); + vbasedev->regions = NULL; + if (vbasedev->regfds != NULL) { + g_free(vbasedev->regfds); + vbasedev->regfds = NULL; + } + } + if (!vbasedev->group) { return; } diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 3362e8f3f5..52af5a1061 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -256,11 +256,16 @@ static void vfio_irqchip_change(Notifier *notify, void *data) static int vfio_intx_enable(VFIOPCIDevice *vdev, Error **errp) { - uint8_t pin = vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1); + uint8_t pin; Error *err = NULL; int32_t fd; int ret; + if (vdev->vbasedev.proxy != NULL) { + pin = vdev->pdev.config[PCI_INTERRUPT_PIN]; + } else { + pin = vfio_pci_read_config(&vdev->pdev, PCI_INTERRUPT_PIN, 1); + } if (!pin) { return 0; @@ -1258,10 +1263,15 @@ static int vfio_msi_setup(VFIOPCIDevice *vdev, int pos, Error **errp) int ret, entries; Error *err = NULL; - if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS"); - return -errno; + if (vdev->vbasedev.proxy != NULL) { + /* during setup, config space was initialized from remote */ + memcpy(&ctrl, vdev->pdev.config + pos + PCI_CAP_FLAGS, sizeof(ctrl)); + } else { + if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl), + vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) { + error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS"); + return -errno; + } } ctrl = le16_to_cpu(ctrl); @@ -3562,9 +3572,50 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } + vfio_bars_register(vdev); + + ret = vfio_add_capabilities(vdev, errp); + if (ret) { + goto out_teardown; + } + + /* QEMU emulates all of MSI & MSIX */ + if (pdev->cap_present & QEMU_PCI_CAP_MSIX) { + memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff, + MSIX_CAP_LENGTH); + } + + if (pdev->cap_present & QEMU_PCI_CAP_MSI) { + memset(vdev->emulated_config_bits + pdev->msi_cap, 0xff, + vdev->msi_cap_size); + } + + if (vdev->pdev.config[PCI_INTERRUPT_PIN] != 0) { + vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, + vfio_intx_mmap_enable, vdev); + pci_device_set_intx_routing_notifier(&vdev->pdev, + vfio_intx_routing_notifier); + vdev->irqchip_change_notifier.notify = vfio_irqchip_change; + kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); + ret = vfio_intx_enable(vdev, errp); + if (ret) { + goto out_deregister; + } + } + + vfio_register_err_notifier(vdev); + vfio_register_req_notifier(vdev); + return; +out_deregister: + pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); + kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); +out_teardown: + vfio_teardown_msi(vdev); + vfio_bars_exit(vdev); error: + vfio_user_disconnect(proxy); error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 6afbde8ba8..0fd7e01986 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -574,6 +574,16 @@ VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp) return proxy; } +static void vfio_user_cb(void *opaque) +{ + VFIOProxy *proxy = opaque; + + qemu_mutex_lock(&proxy->lock); + proxy->state = CLOSED; + qemu_mutex_unlock(&proxy->lock); + qemu_cond_signal(&proxy->close_cv); +} + void vfio_user_disconnect(VFIOProxy *proxy) { VFIOUserReply *r1, *r2; @@ -601,6 +611,16 @@ void vfio_user_disconnect(VFIOProxy *proxy) g_free(r1); } + /* + * Make sure the iothread isn't blocking anywhere + * with a ref to this proxy by waiting for a BH + * handler to run after the proxy fd handlers were + * deleted above. + */ + proxy->close_wait = 1; + aio_bh_schedule_oneshot(iothread_get_aio_context(vfio_user_iothread), + vfio_user_cb, proxy); + /* drop locks so the iothread can make progress */ qemu_mutex_unlock_iothread(); qemu_cond_wait(&proxy->close_cv, &proxy->lock); From patchwork Mon Jul 19 06:27:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02CEFC12002 for ; Mon, 19 Jul 2021 06:41:16 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4ABC960725 for ; Mon, 19 Jul 2021 06:41:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4ABC960725 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:35398 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Mxm-0004hJ-BW for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:41:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33604) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlT-0004UW-HT for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:31 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:35424) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlQ-0002MA-Fq for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:30 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GP0N021860; Mon, 19 Jul 2021 06:28:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=/B8XuM3rPOQeqQvMmQ0k8eBJT7GOVt553ydoCrjoy4I=; b=F9rErFyrTifjTpywl8JeAXRrhxA4BVLaYl7/8gwn2HyAo3PpMrb+pAFCvuiHNiocD0DA IAbR/LQ8+nqxYT0JrcwKLbu4TKOB9PqrzOkswBAgCs93McjBkRuvTvKtp61rZowVaAfp NpQsAx3RzODLefq42bEo2xPeUKisL1B+2o24mayHOwO3vX6H3PZ6glrBKqqwGUwJHJMd XAZ9J2b4JWKIWxNhsGSxjcmR0NHV1vG8pcZasLiAEKxWm1B33iITpecsk+sbZkE+e/m5 Ni73/ZKpqdpDzYo1Z3wxqLGSBoOjpQ7fpDKdKgCGUZAAXKvUn1jskZ6ygn0bKCtu/NpQ 8g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=/B8XuM3rPOQeqQvMmQ0k8eBJT7GOVt553ydoCrjoy4I=; b=tpbVHlZi3XkVgRzTiBD4c3lgPdkT8uE6Jiv6A5C4E0e4kJyNEYjM5YZUt3/RLzQw1rou 74YQQF+f8KkgJofFvPGWqHgj9RIYKIKQuS1N1Tiv7g3onBCM+zLSzTQ/ShLjWfNyMECo YwdC2hXcRRA6rAmVE0AR1a86+Kt81c5AqwRAH1Weghr5nuWutGjUgZwGZUaetdciw6rf p7pfPA6nO1gObY5aB4XAbbECBTlySX9ZymRlG8xUVuyqwZUobXPEJC+6hqHKktUJQEBS pAs6mzltqxCvgqN034W9on+TTY3AXHu/dIzhSzGpCFA6N7SRSg9cxbeAwZJYluuFchWK Fg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39uptrt6c8-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:17 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6GMKS099799; Mon, 19 Jul 2021 06:28:16 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2169.outbound.protection.outlook.com [104.47.57.169]) by userp3030.oracle.com with ESMTP id 39umawam46-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KYRouIikVzUKFS0egQpfsD9XJWUiuC8mZ3fKWYiPFiydV6+b+VriVmbV7H+O/+/tBQEJVG0Xnc5Ari704ovHaKg0plmQJKRQ3Bo3UItnL82wFa+9VuHYqyzClnEpES2lLk+eWZSHqFGtxzIzyEBs5A7YDmpuLGlwh8Dbl45mGk+5bkxuhLHWyDnvUDUamM+vDI5Loij1Tog1SgBQ5LrhrhWOM56MWwIHZTIAjW9WlcWA81wcXWBQq6UTVeir5PVJ9Y6Z4pMvXmwI6o0bLteU/XZzocxAWtqZC0x9B21ZYZwHkjkTJJUK/pt/Ev6R99FWZECeqYpmbHH0zcC+lP2azA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/B8XuM3rPOQeqQvMmQ0k8eBJT7GOVt553ydoCrjoy4I=; b=PZDoNZjj0Pwx3Nosjbr2kBtUjejFSDQNu6mKB4JUQSI0Jc/xcISCUDWEtJbDdPqjCSqpEEG/4EOwnqm9NbzzWE8vXgbQ7mJLvFmh1QVlJyiuADzZhMF7rienp1M1Gk1/fKPSUpX6BcywOqAcSnjzPGAM4EDlxBFF64f7Dhm5pOZSXNAVs4j7CDTuWQlBVbopVw8gzC+WCa5BWU5klKFM1V88WWALb/VQafT9S2eaLR5Bdm3RAL7jYHpDRFvHCB9fCcz8ig/wXO5CEUvFtcdertHiplGauTUyN6WKDZ1OfjQgcNH5YrbkNQzQ/FNK3crfGKvJBAS7zvbcfu92zuie3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/B8XuM3rPOQeqQvMmQ0k8eBJT7GOVt553ydoCrjoy4I=; b=QBA3BeUv9LxFQUoU8AgyYW23Hy/bTeuwctgyyb5RwzXOc94Szag5sCM2SKrUK0DWDfyYAuyIh7e5iLX0EX9XuJB4xLUjDWROIGR1DRdUmO6aiG6tPIGzmsNYx9TxDIxc4avwOYIyDEWfSamuWCUMz/ukDfuJ59PtShO/NAAYg74= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:12 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:12 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 16/19] vfio-user: pci reset Date: Sun, 18 Jul 2021 23:27:55 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:11 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 1923b104-2328-4b84-98ab-08d94a7e621a X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:220; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 5Bn6yNk0wrG4p5YPB5v9mYOaNGjn5uaYoS23jhiKj9ygHFMpqHy9klOYvt7nMRAMmy1BT80BfEfEPbpKfJuwr4IU/fPiJBZXpGQzL4Sz2WVL8A0brGvGdrVMQ01LBXpFgaV1jYtBbjFs/mzssN3iEYmRnE5oxA4Y4Rd2pJTvnZnkBuO4mxlIVQqb7tr0PD28IqZxfpDERUIVeXI0bK8P1bhgOOVaK20Obv2i0/4lUXQMgW7eQcl4Wmx3poVRdhU6zCCJOSS8/J6h9/onyoCnPQOmGTodOStrN84TQRQPdCcg1+SeAr46YmkkDaat9LY1tODnMceUv9zXSrbbwg2Qo8c1JL39YhXjt/A0OSl42HWBbVVx+nhcngPhnToCXovqfISI2FraKca4pHbW0+7qPFa0+rCCJtQmuRL/+e11ONNct1btcSVBpkOTWzmHDuNGzI4a16QjTdhPnOR+YWWWZ2pwFCY5cqKDICqhHPEOhjRu2ZiYATUU6ZRGPxMkE1cc2SaMpUGWQz0lNhQrTvmKKyORAxGvVkxFbjJlR8NdMkmmm6rxoHMo1G+hpmzKvVUK2YawdXs5bSSITZi1FwOeIzsYtbCvx/YQZSsTCHzB10VAW7pcHmbALpPC3a7iUCVb2/MtXJ5ZQOOcEuZdBqNSoA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: JhhaYppWG5JeXh1C8t5mG3yTXlQUCwUeIsSBh++Gm7VKKhQvXUxoyX/4yPvgfa3qNM4Y0gLJ8Z0IshGVZlKApMPHOtWpKOMCn+lit+OnZ3BjToiEaHfzOjV6S9RpsYzDQGB0gC+y4yCimxSMbTIyXIGSNVdlq0ur+UQJ0hwdWttlMotATPril3Gy03dGTfoyz27oV+pqyzbPxShhRECS6gjR3pX8duwTyxH6LmqkIAt98MpdN7vgoS562KdSLQp3Dd4cnRB0KOSd7NEhsheWp2755HpvwvDmMoGWW+oDCyOuWzjG36bAsiuaJH0YXb3crazvKif32tYjmg3JXsUWmuFbbPa6Y+3nx1PGj0hPtkt2VJRwwYgIYHXjgYpQWdJJiNej01X5Pund5UQtmB0O2K+N+PpWo/8aF6xZzr5nttbTvUlqCUAFtzzOGS73vyhAUWnEg+8kksgRKGhZ9LM4BWnFWsXBBGQpOJ9qwuN8sH2KO53uhe/O0DebPhA9MFljRhlGpWWqVHchIlc/CwoqsgE23ihQsddSv29ba9XHM5u8bg2mW8XW23XujAVJ6RL3dyiJ3cbWaUvVwItAjBclYd1qWOmLVY2VirHyoeYq07sKghkjkJbxrkJiI93KxEhywohOicNXbJA8oQFacv/3BP7xJ9aQO50zfjBhJFVrJ57Iu4+k7swzDCYPUceeUOZ82caMTuhCmqwm/+tVhlr3Tv2pEM+fxuxE4TqV3CgXnHaFTNbFuG/mpPo+3cktisLj8R2VGVuVfNjHCL57X96HU7CkF9dcA+qid6bEZdB0ZmIfoPCvP/wvetO8Dw4lIk/I0AMwFmbXihV/tcVB07ADMMk8ZBmXhMn7bX6tYhR6tvxRv4xcKmF2FVENrGtFCkaujJTN4qdGroM22Q0SFQR5X5CX+biPkP5923v+AKc7PqtaFnUWGt3RRQnS4Jqf0WVDLyS952cCzeMV5hfdgTa09m0nvT3K/CTYISUDze/leA68o6s9YqtoKYEA7iOcqB+tsPocF8900YTWsBI0XzjD1Lw66rhSXwGOck+5fpvFOGHfXl4OUgS9cQ64AEnMUmcW+UkHxF0zaJ/ekiDgHm2Qx6EajfbZKvhQT8y/ZNaidv2ATw7cysfMJJ3BUpIeAzJZ+wrRhNApE2ATHTB+4e/Qswbzz0UgP2FLdZXVUVyMkisbUw12qmqonnxiCJqhjJddwrpmMrvoEvatHtFoxE518HhEml7EjXfG27iI35e4ju4xVPfykcR88WpLHw+lKFS3VBfZTjkq/o5t1HRUOR9YA6FiiQXnvqLcCUWcF3cYYXtfD2ALDOb7eRG1cQAp3s4VkUWgykIKH8bfI92P1UNRiHV/D5g416kyvQ6wHex+1rDkGA38WU9zmw3ZEZ16eOXD X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1923b104-2328-4b84-98ab-08d94a7e621a X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:12.2940 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5eyMJBryvc+aMOWbRVvnG9MWf2faDipBB5Q5d1RP/BPi2pRmHHwzqadM3N9y+9XBMuKT2sLpUJ4dLBv8U5L5Gz2xUsI1VH/UHcZcMuUzxMI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: 0hCUuSk3EmVt0uKjRhihhEHg_N5nv5kr X-Proofpoint-GUID: 0hCUuSk3EmVt0uKjRhihhEHg_N5nv5kr Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Send VFIO_USER_DEVICE_RESET to reset remote device. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 1 + hw/vfio/pci.c | 29 ++++++++++++++++++++++++++--- hw/vfio/user.c | 12 ++++++++++++ 3 files changed, 39 insertions(+), 3 deletions(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index afb85952da..95c2fb1707 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -254,4 +254,5 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, VFIOUserFDs *fds), void *reqarg); int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq); +void vfio_user_reset(VFIODevice *vbasedev); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 52af5a1061..a6c28dac03 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2212,8 +2212,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev) static void vfio_pci_post_reset(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; Error *err = NULL; - int nr; + int ret, nr; vfio_intx_enable(vdev, &err); if (err) { @@ -2221,11 +2222,18 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev) } for (nr = 0; nr < PCI_NUM_REGIONS - 1; ++nr) { - off_t addr = vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr); + off_t addr = PCI_BASE_ADDRESS_0 + (4 * nr); uint32_t val = 0; uint32_t len = sizeof(val); - if (pwrite(vdev->vbasedev.fd, &val, len, addr) != len) { + if (vbasedev->proxy != NULL) { + ret = vfio_user_region_write(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, + addr, len, &val); + } else { + ret = pwrite(vdev->vbasedev.fd, &val, len, + vdev->config_offset + addr); + } + if (ret != len) { error_report("%s(%s) reset bar %d failed: %m", __func__, vdev->vbasedev.name, nr); } @@ -3634,6 +3642,20 @@ static void vfio_user_instance_finalize(Object *obj) vfio_user_disconnect(vbasedev->proxy); } +static void vfio_user_pci_reset(DeviceState *dev) +{ + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); + VFIODevice *vbasedev = &vdev->vbasedev; + + vfio_pci_pre_reset(vdev); + + if (vbasedev->reset_works) { + vfio_user_reset(vbasedev); + } + + vfio_pci_post_reset(vdev); +} + static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false), @@ -3645,6 +3667,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) DeviceClass *dc = DEVICE_CLASS(klass); PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + dc->reset = vfio_user_pci_reset; device_class_set_props(dc, vfio_user_pci_dev_properties); dc->desc = "VFIO over socket PCI device assignment"; pdc->realize = vfio_user_pci_realize; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 0fd7e01986..8917596a2f 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -905,3 +905,15 @@ int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq) return 0; } + +void vfio_user_reset(VFIODevice *vbasedev) +{ + vfio_user_hdr_t msg; + + vfio_user_request_msg(&msg, VFIO_USER_DEVICE_RESET, sizeof(msg), 0); + + vfio_user_send_recv(vbasedev->proxy, &msg, NULL, 0); + if (msg.flags & VFIO_USER_ERROR) { + error_printf("reset reply error %d\n", msg.error_reply); + } +} From patchwork Mon Jul 19 06:27:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 645CAC12002 for ; Mon, 19 Jul 2021 06:38:39 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F1BA761166 for ; Mon, 19 Jul 2021 06:38:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F1BA761166 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:57744 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5MvG-0000ip-53 for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:38:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33606) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlT-0004UY-G9 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:31 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:35570) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlQ-0002MO-OL for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:31 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6GPpp021851; Mon, 19 Jul 2021 06:28:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=AkAaN0Qv6D/iLNNVB+WB4JFriA2FY/v21hNPm4hasW8=; b=VRIGF281rBoohh8b3bT32kLkWJwgNVdO/UOGEhAsff99tBHAFalksZ/rfYrorFsfM+9o kqtwyCxE9Ve3i5npRDwSjVBDzgQHKBx+xBrFHP2QVSRM8ePL4JZJQX2IBAGNxWld3KYM /oQtUG2NrJ3uy6bJdyRDlo/aCSF/fAIW4hzQXbZ8cl8Sd5bQBXsFLQNLvmCkl4/A2GqQ O80Rrvq8iLTVQ5ZqaQHDWjBY4HflHXH2igh4NAaUNZiG6Gm07eW28EH7BEedJ2FBnwoD NeF5aX+t4S1hzD8S9hTJKM8TRfQnZlLeFv4aEaSGCqxiKaxcdCMfibWHhP5VnK8QbeRL wg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=AkAaN0Qv6D/iLNNVB+WB4JFriA2FY/v21hNPm4hasW8=; b=riNQgXaBrQnfnLatWVI2xVwQL30+TxD8iNPZSBHhL5ciR8hwsNNwrZRjuheXkRbHJOPV AZWxI3ISG4K9CRwW52eJuroiZ2A//kifR+GcCcX4hyjJFshf2qLqex93FQ36ZnN2MqFY zYCrNtbjB/AbS1VicSunZvRkoPwAJlkCUGJDXhtfxOo02iMql2t4/YxmkDqXJjW+8NtI NcYTonpTHBT7WCDw8WgJhFi5s2TvenCLuPSrmxptX0gWmuTbAFnqc6H4I5DRjk/e7c6C 7QYHk04JY6B0huGjwmTnkgWRaWAKMj7G58wJ8SjbNfMPobe6vdWuvlJlhR2qs/A/syO4 GQ== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 39uptrt6ca-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:17 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6GMKT099799; Mon, 19 Jul 2021 06:28:16 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2169.outbound.protection.outlook.com [104.47.57.169]) by userp3030.oracle.com with ESMTP id 39umawam46-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:16 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U0OoioCrXyCHbO+GsJukcrPBKLrKAf4cj0s+KkEBhSjjmIjz25U8DqfLyCV2RhHZWdDUDGZmXUXeww5sTCaYu9zEw2n2olMrkg0iJ4NBhkN0WoEA1f52295ttB5ibf2oxrs3jpf3G36vX3fgZpgx/GqlV/xNWzx5sX66ne8hb67rctWGz//lf16efC6R9zACh2Z4SwqHEWrg83VZ+/R8Ymf08fejpJzYo5It2RbukNdjVdcLzLp2nFkRTvfIBJj+NkEUQzFPEPw6wYSafIHbANEr8GUek+WRiKrIJnJ0MgzasTy/6RNNSBWZOVBV/W4OY1yfwJIuD2u4ZGKxT3ApHQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AkAaN0Qv6D/iLNNVB+WB4JFriA2FY/v21hNPm4hasW8=; b=BjCwIAhurtC7yTpUI68gttUEvdpkwThBweg7hQSiF7SEJRy5hbI0fahK0AuQ54pBiYW6Cxu6D77sEmEZX2yvN3H0jKjY1jeh4cj+QITm1EuWHdFWKAXqv+QuQAikWXBGBsboU+3wAvL1JJ8Vv26F7ha4AR/1/Hkg0xHGWWTVhybi8t124GfjhmbiO89+IL2W8wE7tXKLVoXYOvVZh7+Pk2z3OvsKAUVsuTvE1HwTYmMhAUhfYETm/7nYmL26F+KaJuvX2X8vawL6Yjx1vKANv8v8atTSZiiBfue8x+ctyqarwAMNvddWZhU88ofbvfrm/v9uavh1MaXsqN/WGqSGLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AkAaN0Qv6D/iLNNVB+WB4JFriA2FY/v21hNPm4hasW8=; b=E9bjRA3X5i5uPO5ivA6ZGKgG3ObFec9HGYD4D3ucxRyUHi2ICMN9ZlvSkEnGARQR7pGpQZkcV6GXnXoV+YjpvzhH1uLFblITDkwNxmPyUpVkY8XZFbJJBoqa/kWIMF6XrHQOVwNGwiozYub78S/VXM9Ui9rHVcoX+sbVkSO7fU8= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4814.namprd10.prod.outlook.com (2603:10b6:a03:2d5::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.22; Mon, 19 Jul 2021 06:28:13 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:13 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 17/19] vfio-user: probe remote device ROM BAR Date: Sun, 18 Jul 2021 23:27:56 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:12 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ff1f3c9f-9f7d-45fe-e974-08d94a7e6270 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4814: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:854; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ihDirogRrBALtOZN/4IusoAOWGXIAdCHLViuPccx60DbgrBYhP0qn/OBM7jUMI2IhwGpe8ySfSvXXgfT9c+PGoxO8Kkq19R36DtTqJXLwfxPPeu02pFFPLBQXpkSwrPT5GvipfuX7ab4+6obyfDLpC9KLqujKjoAFVLoyV4U2efavSSv52JPCcZrbcA9Ta2qucRJMNPlmy7TfFWZm+ZEDrn3m9SRza1EtuSX5sVOF0NyAs+Jp949EEsNfv1U0C/vN5ocySabu9Nm3xc8u/D3F2DgVJLIC6NddFgnhlmstxagcVRH4Q1mdq9gZtiO5DfmpdQdPvY99OsGz8xUnmRpvJUhIc+GCW2JoCJh8ojAg1t+07vHMKw1AP7Eb7szb+VI73ijZAsAIBLsfdexf62XPw6JHg7wGAr9hGgDfDfN7dotAuFjTvVJO1gF9MzMEhrLl4gjhwVhz07y/+gk/uCDFx7qgtehILZdWxfwZ6z8YcI1e/QDB5DeUcuUh23a/EMYCzdiyIFGaOXu3yqhfVrMdHsf03jqq3FA9eKO4XGdt6Y50DgHsIM2BrAWPdFeZTXco8TLloBPXVGP81e5n0rnhO1FFaWKTxrsmKrV2s4F4DCg8t+Tf8hZC+q8bQCSnaiE79Wl1vHtmY9W5IKkqn+dZw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(396003)(39860400002)(366004)(376002)(346002)(136003)(2906002)(6486002)(5660300002)(316002)(6512007)(186003)(66476007)(44832011)(66556008)(2616005)(6506007)(38100700002)(6916009)(52116002)(6666004)(8936002)(86362001)(107886003)(8676002)(83380400001)(66946007)(478600001)(36756003)(4326008); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +9q1J36TiD+GnzhTBeJT3SSPto5J76SDLo52LdsGZVg+S4TmzNMPsZheEt3A9N6qIVW8kK5KAADDpnV5AFgAPlOJ68wKSpZ2wvh8Mckn0CuE2FASsw9oUPey3XlxnwYsZls0DflsGZ+nFwOWnxKu+UBA5sAl3UM8ciqqU/nAiuQ60E9+3IoI4rYq9v1Y+9KoJQbta7kZ3hGptKZkckGiEAFE1KhRyYX0dsWAUahKnbAL3d7nGPEtHZ7NLOPaXFAhVTZ00kDX45ujO60j881WYKLQZTZyqyhokUKBmsa59cFKlj5lEF3awT6JvfkkqKUSTc9q66dim3a1Kw6POkd5PE/viCSjB+FuRLmRH2FFgut9yIfU57KjOuvGwLBe/i1sausUN9nEvZSM9Ko5HdsThk3Ik3SkiA0asYwPyatqvtBiIX/Pd37IWZbihv2hLfa55s1/2sKTaEZl4JSSS8KGavYnFASlW8ukT4/gwQilmp14/ZKG7/Ubxt/rB5e3sVPjYcYsagy6vRegMAegJi3tpGhNbvE6gZ/rBq0eLV0wzeM3B6Z2YpWMOaFv6GUVs4CpIRV40zgRyNrMeblkqQGzGjaz5ONjMsilyimC9DhEBfLWl1tHBbWLerGsZWR6PU8CAco9JIb0UPFwRZnl1Wetn4jyLh83syHwe89eTn2npZ6rujmp1J6gRH0d2Zb3eUcfYpQgDEVvCPUEbf7B5KtmBHzXDhYWo2P72snYwosORCjhBKvBfwv05Gl2AO/QjSTsh9lj5JbBEuUWqsdwEjsNF+gMU7XPZPvnmPim+ke16XAGGtW6R/F6B8WLyS6A6YsVyrw7XKYj59L2VNCBZ/esM6Oj6wEhAkpvyZq28YUf3tWu4iHSSAn58B+aPN111gLdN5YCpjoYCv3L33zoXCw6UHXKrjslx2aeADef+g/wrtDphUfQG/EkB4SoIWo/rWf6ehXp7QhBV69LFQTqbAyOF8m/O4/9BmCyc/6VZNyEtb2KsJatgYkKweE4MnBhL64eVfwDuE/PUo4DCuG8SwCS/dgRty2HzarIytpazNc7tB8GR3woG0Fzxh1kzetXD1KHvBlpMTDgr+XmjTWcBKbv64rbfwzSeTjauiqj+SlGNvMmRHs4+jI0QLvng1MHFOhuH0OrlSSIosNYu+YMNgyvXsa+TzZ4jzgiwuDhDiDm5QkfTKiPzUKiRdY4+Vld8mfdCR9HcOflzAfOwIpXtA/H6WZSeF+BHiO8tVlgHIRdl1QsYdSOQ05RaTBAqczQQ3d5HWrJyWBnikSyX1n4TRTAJjkmdbGmIA10AjZ+Tl5LLRZcOS9GuAoK/ZGH1RhjKucocDQmTPTbMybwbi/2g+9NyPEOwL27TOHQBPqCCeBCWlhKy/8/eS6VD2K1gGL/jkSr X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ff1f3c9f-9f7d-45fe-e974-08d94a7e6270 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:12.8776 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: kOBx0fkAKR3M7jfgaBIs4/Sxdi8WRMXoxV2n3PKc+Kn9BSf/HBaM8jd3diw6/ENBV+7lYVUaJFkBvPkAo02Qq9TZNQzeTmGy6laCC/G8epg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4814 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 mlxlogscore=999 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-ORIG-GUID: av7F1J77MPmxfMm9urC_a1R0j2WLpNNC X-Proofpoint-GUID: av7F1J77MPmxfMm9urC_a1R0j2WLpNNC Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: 0 X-Spam_score: 0.0 X-Spam_bar: / X-Spam_report: (0.0 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.c | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index a6c28dac03..bed8eaa4c2 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -816,8 +816,14 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev) memset(vdev->rom, 0xff, size); while (size) { - bytes = pread(vdev->vbasedev.fd, vdev->rom + off, - size, vdev->rom_offset + off); + if (vdev->vbasedev.proxy != NULL) { + bytes = vfio_user_region_read(&vdev->vbasedev, + VFIO_PCI_ROM_REGION_INDEX, + off, size, vdev->rom + off); + } else { + bytes = pread(vdev->vbasedev.fd, vdev->rom + off, + size, vdev->rom_offset + off); + } if (bytes == 0) { break; } else if (bytes > 0) { @@ -936,12 +942,28 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev) * Use the same size ROM BAR as the physical device. The contents * will get filled in later when the guest tries to read it. */ - if (pread(fd, &orig, 4, offset) != 4 || - pwrite(fd, &size, 4, offset) != 4 || - pread(fd, &size, 4, offset) != 4 || - pwrite(fd, &orig, 4, offset) != 4) { - error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); - return; + if (vdev->vbasedev.proxy != NULL) { + if (vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &orig) != 4 || + vfio_user_region_write(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &size) != 4 || + vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &size) != 4 || + vfio_user_region_write(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &orig) != 4) { + error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); + return; + } + } else { + if (pread(fd, &orig, 4, offset) != 4 || + pwrite(fd, &size, 4, offset) != 4 || + pread(fd, &size, 4, offset) != 4 || + pwrite(fd, &orig, 4, offset) != 4) { + error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); + return; + } } size = ~(le32_to_cpu(size) & PCI_ROM_ADDRESS_MASK) + 1; From patchwork Mon Jul 19 06:27:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FE8AC12002 for ; Mon, 19 Jul 2021 06:45:40 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BE63A61175 for ; Mon, 19 Jul 2021 06:45:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BE63A61175 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:44876 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5N22-0002cx-Qw for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:45:38 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33584) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlS-0004Rs-D3 for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:30 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:34766) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlP-0002LR-Qv for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:30 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6HgWt031091; Mon, 19 Jul 2021 06:28:16 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=fSz+4zFsi1bH47MrTWLJVuf2Yo02EDh5t6GoiFdDNis=; b=Nn5WFb7TbS1WQt3QRfTfhw7wnnnlrJbS4g5BNE5aMaS18uhH8y/msIVot4AvgnRSHdsC fAY24iUW9EQMDwnnWHp/o+TbjIadMggOqrVbgjZTqAKUyKMYb7G5fYC9fqLwhbuh7ISb H6NXPR7xHCD1nKRo8FrVPLyeoj0TBySxB5Yxy/vi/FSigz+QFEHNQ2JAUagdh/ZJz0Nl 0jvOHKQR7y2OCUuz7iz2yWQ0KicnO6uFsiGBJm+q+7B9RGU1J0uXukjNpRn/kdEZNb9L LyHsosXff5qf6p6QrxL1x00zCj8KThGy8Bd8L013UVyA/5V6uUhSYVpxKq6uc0pkszbJ pg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=fSz+4zFsi1bH47MrTWLJVuf2Yo02EDh5t6GoiFdDNis=; b=uj8ONkHHSKE4sDh9xeG1YmhwTg0BU70TDB3UZRcRw2H2FGEGJ4cvY+kto/w2wW30uOnb wR7BZchvoNjlxtYoR2nARKdv3WojHCZW2OOqm2GQBae0UHGHoES8a0GUKwIncMWmTdUp +uiK0gKi1Rk8JqPClQM8/8Kgc2SBrp6ps4vIm5OnhvR5fkUWvT277tTtsBLTU7znUqx9 Ywo1nWH5YBYYUAcp+BHo05Og2l/Fx67deFCjmPxaM95wKZ+x2uJPaIPGr44hsb+0y9Yi 9CkI7YqK9XMjq/zi139YOxwq6RDKVVYDZu7CmX6riJZpWksYU/+rvYa4UuMg4RuS/aLU xA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39up0327pb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:16 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6GAHR086933; Mon, 19 Jul 2021 06:28:15 GMT Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1anam02lp2048.outbound.protection.outlook.com [104.47.57.48]) by userp3020.oracle.com with ESMTP id 39v8ys1vy3-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jJ0VTfcqo4jfHiFlepfJ4TvFGFMG+Yk+CVjDLVGqWEMNm95+Qae6EF46oVi2976q7dTmVPU8Nvv7qPuxfmTy8xEUsxnptXfkpriLDpSeaX/rmQopfFTIQKwYDH6RaPH3YRO1HRbcJiQs5VIpg9YZuvyv497V6WAGGxU5zR4gCcB57onvFnhktVolGM2HG+6cxbW2xgUm/c1mxOgJRaUy5Svgti5mxQtiW4vCJLxJUMHvvH3tJhVb0JQdKOqdxG/jsMhOwlemCLXrOetA3htFQu2ymHwobqXHBz/lbUmLK5lDiEW5KpKrqWOqNVN6VpA0LzrM9ecfPSKe0r3NfBE5sw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fSz+4zFsi1bH47MrTWLJVuf2Yo02EDh5t6GoiFdDNis=; b=LcT2/WvzcDXPw7cM4MUM7wxgkahThke3V82XQblYPVz7olU+6eCpqyYhzctE9dziOCHH4MmEcWzizn+k6piTdhwBokh8+RNsv60Utp19Mj4qn77Fi5YdHMplncNxeHj1DVsOxgbnIzZDfD0eNXqhxhFfAmTZBEtifrj3CN/cxu5xIKVPGZx9f6+4NtJWtDm9vWNW8JVCqqRYyHjLvJTOvf+ZuDk7+QSixis/0nx83lPidO1XQFOX6SeCFJ0a85U45bTajzxGuZ2mPL4j933lj08ZnkUjlCxQOTAzBVfzEFrMbVL+W3TNLBOPZIUy+2AWPpiMasl9b+vZyXM3zdBKfg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fSz+4zFsi1bH47MrTWLJVuf2Yo02EDh5t6GoiFdDNis=; b=lKZSvqRKcgVjBZtiSzT2GdEKtYZ3AW0hSVPK+CppSTEyYYN8l/N4taYRnH9WPy8mlwHEeIrMqQGEW2U2rR+F30hZTwY17xImBvoqhR+SHubuFe1OczDh3GoBe8TJ8ouL0qyDBjHPCbr86mLtq55gO4YgcJ0viGJCePNpf3McBgc= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BY5PR10MB4033.namprd10.prod.outlook.com (2603:10b6:a03:1fe::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Mon, 19 Jul 2021 06:28:13 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:13 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 18/19] vfio-user: migration support Date: Sun, 18 Jul 2021 23:27:57 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:13 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: bd3e8b3a-973c-4e22-d705-08d94a7e62be X-MS-TrafficTypeDiagnostic: BY5PR10MB4033: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:130; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Wi44g+vhNQm6oW4RCqpMXHos1WZhfBvVLp63pp+LNMhPxSjUgSfp52UCsiEMmTOqqe3vz2vYPAs5Btny+faa+MRyW7c9BgVeXfac3DgDcnAtwZYeEpifJ2Y1CbC8VTOSOJhUmP61fxY7WMymUsx2cc9HSg8F3K2ykczmhSg6jGtZ7uAsknZLdBYz64PIPBNLOMVWlARJ0tLU6SrcAtYPIBryXMNGhqAsDpCXxZ0BEQv1Mhal0PAJfq1+1vg3FLiVxZ52uPXliik2RNFqGDO3T8b2F4wsyRKTReV6hR8p55sHaJ3bkHoUeY2BLllGWtnR7m0zaOnRuhW3LILfJQtC0akc5CijKtfjX+uZ0PKG6wSA/V+lnSaCBfWkh+AWp4AlY7T5pzVF5Se/7ur421GrxlgW421XNuOJB7vledRdz+1w0vfiLe5AXjaB2B8qvWw4gDJ38xbwE05xLSqv0X8ze/Mz7lHrZO6aYsEfywzk/QI/cRLYjw0djUVLUQBk84luWtseg6HoX4JzbA/dqYIOnfZ/5/z+KFA28u5s1bFKzPMryUBTQmI1HHpE07XWSMTeUX/L2qTxPIQbqyyvwOFaicCJYAiL9gkhFkOaL2b+Opb+tuazgcdtY5JLtoCUtN3O+TK3I4kUUf6NSFvKJ2R8RQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(136003)(396003)(39860400002)(376002)(346002)(366004)(38100700002)(6506007)(30864003)(83380400001)(186003)(52116002)(6512007)(2906002)(6666004)(8936002)(478600001)(2616005)(44832011)(36756003)(5660300002)(86362001)(107886003)(6486002)(6916009)(8676002)(4326008)(66946007)(66556008)(66476007)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xtsL8WopPu1PDQV0nFmSEkjDWHFhJf0PiruLSdEYAIala9TO/2hui3qKCs/zw5ewLOvaKdZH8aYskIcfxMaU5lHiFJWqRjIFr6IHFD0xTNL4A+YHIRGnROtBM1jvHFfAGhpFCEajggFoNuKO3npEizzd19Y95WrKv+PNL+DE8ykcbjDU6eoOH3UjeYz+8Azh9My5XfFivY+vCBWAefackORwp27iFXbqIGpSrfb4ycE2lQPpBxBeij0HQ3kHtvp+rMGQ2OCzQRF7Ub382n+rrhQNW13DevALXFK3xv4g5KglvqlOCQxskEOrosfCr7aHlhaCY4dwbKhDJyoUncsAU3M32hw47/94tn178ipIwLEPh8yopOxhMH/evRL8+uUCkb8cw+NRgIjhvLmYI+lWxhGGCnD5Tx4XOi8FUxIDiOyqhuVjYyjXhAaYvv1oPdWD97UevLeze3JBQN8Js7Q0WrHt7c1MQkAr+GDNsOxOiZmD6oqkcQKoCp4zWzU0N/zYUJth6XzXA8PPrvcwKm+T3vo7RtmcAKgrEa2ZbAw5FFMm3iHlnRQGRHNGKxGoN4vYb8ELKAKaJE5eFdXcCoSDdSYP9ITPiOYYDkQrMIPVshTZIFIaC6TerquJxe/TEXK74OCgGl0hmLeGMxmb4YnajUt9Jo4JM1UeQD0eYHTByGjMqvMhovCfwPNCX+7zNFpv2gW3EUgr7DSCRJQ+Q2RXxO7Lmk1WyfoC81jmnRFalfoU0YExSkQo8irmPGIe48jsa2SnIPTmQJYntR8WiGdykocQdMHm+dBfT1au7bEQCa+oiWze83FjnV7QRbSXXEeoPYbnYnn+EdKq149wDR0EWn4LEZIjsqGhozK8OcTZC2O40lUJyC8bDX0U7a3MQWES4c5OJ47DLuAnf+LRn4TkitKGXg4slvtNJYwUi3P+Dnloio2liI+pO7oSAt4YGD50FRTUzrlnQDrfvY/P7ISHiBQwZNwO9unc9beGGr4GT/po4QNwH4AwB6ZW4DWIl706m8an8po8w9VxeXxi6RrB+m7uoAtIc+OUmLEwRxynKW1eQ7svLvf4fqJKSfv5Mais317+BbgqeAQRwUf8BPlzHZKIBn42upvfhDXzi7unkbhkHmJYiE5xoI6sXUA4/v5W4pxdXN+t2I26HbqcGrM44tuQlWK+jAnA1Pi9isNM0TQdEAOES13vwoydpxKyxwdyKjAydZ4NzSCDyTwi8G4ukZaOXswgqX6QygliuEB9tpUhdqmvNDT29WTSlDeByE9+Cpwhf0tns1rHY5fCP+bmvympmU4flACmlbA3cxdnhmHH34MOOSxcCA5lp/4w3dmCSRg2ZG/7I//g7CQyiB6HSsflDyjJBMBImv9xw7wrdhMSXrhBh2kvbDsAbapiEemG X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: bd3e8b3a-973c-4e22-d705-08d94a7e62be X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:13.3863 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RcAcKezU1nq2yvOAEZ5TTBh9NdtfSMcI1qRmAM/5Y3HlXxRU00nwXL+ocKTRzDLy6pc76ZSm/u3nq768CzF0aMbxj6oZDL8VDnMsRNw1VnI= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4033 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-GUID: qK5rEcSl19WJtJ7ZkydGPcjDJV6RRyui X-Proofpoint-ORIG-GUID: qK5rEcSl19WJtJ7ZkydGPcjDJV6RRyui Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: 0 X-Spam_score: 0.0 X-Spam_bar: / X-Spam_report: (0.0 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Send migration region operations to remote server. Send VFIO_USER_USER_DIRTY_PAGES to get remote dirty bitmap. Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 17 +++++++++++++++ hw/vfio/common.c | 51 ++++++++++++++++++++++++++++++++++++--------- hw/vfio/migration.c | 35 ++++++++++++++++++------------- hw/vfio/pci.c | 7 +++++++ hw/vfio/user.c | 45 +++++++++++++++++++++++++++++++++++++++ 5 files changed, 130 insertions(+), 25 deletions(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 95c2fb1707..eeb328c0a9 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -230,6 +230,20 @@ struct vfio_user_irq_set { uint32_t count; }; +/* imported from struct vfio_iommu_type1_dirty_bitmap_get */ +struct vfio_user_bitmap_range { + uint64_t iova; + uint64_t size; + struct vfio_user_bitmap bitmap; +}; + +/* imported from struct vfio_iommu_type1_dirty_bitmap */ +struct vfio_user_dirty_pages { + vfio_user_hdr_t hdr; + uint32_t argsz; + uint32_t flags; +}; + void vfio_user_recv(void *opaque); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); VFIOProxy *vfio_user_connect_dev(char *sockname, Error **errp); @@ -255,4 +269,7 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, void *reqarg); int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq); void vfio_user_reset(VFIODevice *vbasedev); +int vfio_user_dirty_bitmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dirty_bitmap *bitmap, + struct vfio_iommu_type1_dirty_bitmap_get *range); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 953d9e7b55..bd31731c0f 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -460,7 +460,11 @@ static int vfio_dma_unmap_bitmap(VFIOContainer *container, goto unmap_exit; } - ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + if (container->proxy != NULL) { + ret = vfio_user_dma_unmap(container->proxy, unmap, bitmap); + } else { + ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + } if (!ret) { cpu_physical_memory_set_dirty_lebitmap((unsigned long *)bitmap->data, iotlb->translated_addr, pages); @@ -1278,10 +1282,19 @@ static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool start) dirty.flags = VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; } - ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); - if (ret) { - error_report("Failed to set dirty tracking flag 0x%x errno: %d", - dirty.flags, errno); + if (container->proxy != NULL) { + ret = vfio_user_dirty_bitmap(container->proxy, &dirty, NULL); + if (ret) { + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, -ret); + } + } else { + ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); + if (ret) { + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, errno); + ret = -errno; + } } } @@ -1331,7 +1344,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, goto err_out; } - ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + if (container->proxy != NULL) { + ret = vfio_user_dirty_bitmap(container->proxy, dbitmap, range); + } else { + ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + } if (ret) { error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, @@ -2282,6 +2299,12 @@ void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as) VFIOAddressSpace *space; VFIOContainer *container; + if (QLIST_EMPTY(&vfio_group_list)) { + qemu_register_reset(vfio_reset_handler, NULL); + } + + QLIST_INSERT_HEAD(&vfio_group_list, group, next); + /* * try to mirror vfio_connect_container() * as much as possible @@ -2292,18 +2315,26 @@ void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as) container = g_malloc0(sizeof(*container)); container->space = space; container->fd = -1; + QLIST_INIT(&container->giommu_list); QLIST_INIT(&container->hostwin_list); container->proxy = proxy; + /* + * The proxy uses a SW IOMMU in lieu of the HW one + * used in the ioctl() version. Use TYPE1 with the + * target's page size for maximum capatibility + */ container->iommu_type = VFIO_TYPE1_IOMMU; - vfio_host_win_add(container, 0, (hwaddr)-1, 4096); - container->pgsizes = 4096; + vfio_host_win_add(container, 0, (hwaddr)-1, TARGET_PAGE_SIZE); + container->pgsizes = TARGET_PAGE_SIZE; + + container->dirty_pages_supported = true; + container->max_dirty_bitmap_size = VFIO_USER_DEF_MAX_XFER; + container->dirty_pgsizes = TARGET_PAGE_SIZE; QLIST_INIT(&container->group_list); QLIST_INSERT_HEAD(&space->containers, container, next); - QLIST_INIT(&container->giommu_list); - group->container = container; QLIST_INSERT_HEAD(&container->group_list, group, container_next); diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 82f654afb6..8005b1171a 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -27,6 +27,7 @@ #include "pci.h" #include "trace.h" #include "hw/hw.h" +#include "user.h" /* * Flags to be used as unique delimiters for VFIO devices in the migration @@ -49,10 +50,18 @@ static int64_t bytes_transferred; static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count, off_t off, bool iswrite) { + VFIORegion *region = &vbasedev->migration->region; int ret; - ret = iswrite ? pwrite(vbasedev->fd, val, count, off) : - pread(vbasedev->fd, val, count, off); + if (vbasedev->proxy != NULL) { + ret = iswrite ? + vfio_user_region_write(vbasedev, region->nr, off, count, val) : + vfio_user_region_read(vbasedev, region->nr, off, count, val); + } else { + off += region->fd_offset; + ret = iswrite ? pwrite(vbasedev->fd, val, count, off) : + pread(vbasedev->fd, val, count, off); + } if (ret < count) { error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count, @@ -111,9 +120,7 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask, uint32_t value) { VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - off_t dev_state_off = region->fd_offset + - VFIO_MIG_STRUCT_OFFSET(device_state); + off_t dev_state_off = VFIO_MIG_STRUCT_OFFSET(device_state); uint32_t device_state; int ret; @@ -201,13 +208,13 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) int ret; ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); + VFIO_MIG_STRUCT_OFFSET(data_offset)); if (ret < 0) { return ret; } ret = vfio_mig_read(vbasedev, &data_size, sizeof(data_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); + VFIO_MIG_STRUCT_OFFSET(data_size)); if (ret < 0) { return ret; } @@ -233,8 +240,7 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) } buf_allocated = true; - ret = vfio_mig_read(vbasedev, buf, sec_size, - region->fd_offset + data_offset); + ret = vfio_mig_read(vbasedev, buf, sec_size, data_offset); if (ret < 0) { g_free(buf); return ret; @@ -269,7 +275,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, do { ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); + VFIO_MIG_STRUCT_OFFSET(data_offset)); if (ret < 0) { return ret; } @@ -309,8 +315,8 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, qemu_get_buffer(f, buf, sec_size); if (buf_alloc) { - ret = vfio_mig_write(vbasedev, buf, sec_size, - region->fd_offset + data_offset); + + ret = vfio_mig_write(vbasedev, buf, sec_size, data_offset); g_free(buf); if (ret < 0) { @@ -322,7 +328,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, } ret = vfio_mig_write(vbasedev, &report_size, sizeof(report_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); + VFIO_MIG_STRUCT_OFFSET(data_size)); if (ret < 0) { return ret; } @@ -334,12 +340,11 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, static int vfio_update_pending(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; uint64_t pending_bytes = 0; int ret; ret = vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_bytes)); + VFIO_MIG_STRUCT_OFFSET(pending_bytes)); if (ret < 0) { migration->pending_bytes = 0; return ret; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bed8eaa4c2..36f8524e7c 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3633,6 +3633,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } + if (!pdev->failover_pair_id) { + ret = vfio_migration_probe(&vdev->vbasedev, errp); + if (ret) { + error_report("%s: Migration disabled", vdev->vbasedev.name); + } + } + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 8917596a2f..eceaeeccea 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -917,3 +917,48 @@ void vfio_user_reset(VFIODevice *vbasedev) error_printf("reset reply error %d\n", msg.error_reply); } } + +int vfio_user_dirty_bitmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dirty_bitmap *cmd, + struct vfio_iommu_type1_dirty_bitmap_get *dbitmap) +{ + g_autofree struct { + struct vfio_user_dirty_pages msg; + struct vfio_user_bitmap_range range; + } *msgp = NULL; + int msize, rsize; + + /* + * If just the command is sent, the returned bitmap isn't needed. + * The bitmap structs are different from the ioctl() versions, + * ioctl() returns the bitmap in a local VA + */ + if (dbitmap != NULL) { + msize = sizeof(*msgp); + rsize = msize + dbitmap->bitmap.size; + msgp = g_malloc0(rsize); + msgp->range.iova = dbitmap->iova; + msgp->range.size = dbitmap->size; + msgp->range.bitmap.pgsize = dbitmap->bitmap.pgsize; + msgp->range.bitmap.size = dbitmap->bitmap.size; + } else { + msize = rsize = sizeof(struct vfio_user_dirty_pages); + msgp = g_malloc0(rsize); + } + + vfio_user_request_msg(&msgp->msg.hdr, VFIO_USER_DIRTY_PAGES, msize, 0); + msgp->msg.argsz = msize - sizeof(msgp->msg.hdr); + msgp->msg.flags = cmd->flags; + + vfio_user_send_recv(proxy, &msgp->msg.hdr, NULL, rsize); + if (msgp->msg.hdr.flags & VFIO_USER_ERROR) { + return -msgp->msg.hdr.error_reply; + } + + if (dbitmap != NULL) { + memcpy(dbitmap->bitmap.data, &msgp->range.bitmap.data, + dbitmap->bitmap.size); + } + + return 0; +} From patchwork Mon Jul 19 06:27:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12384807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5D51C12002 for ; Mon, 19 Jul 2021 06:42:06 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5B42461166 for ; Mon, 19 Jul 2021 06:42:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5B42461166 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:37728 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1m5Myb-0006Db-Hh for qemu-devel@archiver.kernel.org; Mon, 19 Jul 2021 02:42:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:33614) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlV-0004Xj-1Z for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:33 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:35630) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1m5MlQ-0002MZ-QU for qemu-devel@nongnu.org; Mon, 19 Jul 2021 02:28:32 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 16J6HiHU031108; Mon, 19 Jul 2021 06:28:17 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=irPRvaBDaEQrHwfm0kPhFL3GIsCF0Jf2WOsculUoluU=; b=kJbqn8R6g6P07drwjlREeedWPiFN6lMmp58Sq+Vw2xMhjqOgjDn3sZ5p4+ets0efuMhj G4lgpi24OmiEZYruIJM49TBVmXOU6IW8wd5fvXa7v+JhRC2hgRWbAVn30JoDvTmA0W4C FvHfguyt0tRQY92RPU6pARmSHwE9WIV0AVgWXRKK/Voxek1eNDmyj/gX0UXZvEb1rhj5 vMPbEZCp6JFIEApTAF0CbOfdma+mrmoGe69RrpEDpTIsNiv5ReR8wQccP/EJ/K6J8IpY TEPOMJ9k9Qpyt13B0DLnfcG3+fB9HZSX4Lm4SV/j+elbUNzpuNvN5uQozNu1UWHmFTn3 VA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=irPRvaBDaEQrHwfm0kPhFL3GIsCF0Jf2WOsculUoluU=; b=vdHR1hp7cGP86/8jj2iuHJLDAcgCU/qUQnRp34+BjvO3kpihTg1mzCQqyJ28+q5yd+Ep en92ZG/5u34FjH//g5uCSnS+6o6IjFMuoOC6DcKAPBD2vSLJ41qeABTdGK7MdgPQz621 4oq0W198+Z1l12PQxDfn8KDEnd7kbGG5S+Zs2nKcHLAMrZek+2Y/hi9quPni/RRix+Un Y0CVsecMwiPYEUV/kfGkgoxJHsmEmPucBLpR15YZZVCX7AD3yHu50BVoRkXvGL/nYykc OnQ8QNs4Dj5YJvp4z8s45HuIRXpsJ5WbbJzJv3n81jFp3EYRUP//qCGmqYpOvIC+bAT3 SA== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 39up0327pd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:17 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 16J6GAHS086933; Mon, 19 Jul 2021 06:28:16 GMT Received: from nam02-sn1-obe.outbound.protection.outlook.com (mail-sn1anam02lp2048.outbound.protection.outlook.com [104.47.57.48]) by userp3020.oracle.com with ESMTP id 39v8ys1vy3-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 19 Jul 2021 06:28:15 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y4keiYiFcH9CBVV3g3MadV3IR/DFl0CYVZMPVp7/dAuxgxkf6/WAfIWguex6wKfqqPveyMTjIoa0dEJROvEUqKNYLELpiQ72q90NjCeQeUMde0aIgaSE5HiW0rdDdZj3pE6eFDfZY71WA+SHALHctJ/hf/iRww8Di0VOITu9FGh+Q24nBD9tDI0G1x97nUP91KvvYE8kuI1T/wQoYb6kmMPU83o1XYeM+6pMMk9F0FpUWpQlOQq9QHBze+wqyABNS+p/DbrddVzXCOEMqp2wqNepfW/SAxxaCCePB9LwlVEG5cZ8U+vU0hkb3wLYYF8nRZPUVAAR7//ad/1Y+vCubw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=irPRvaBDaEQrHwfm0kPhFL3GIsCF0Jf2WOsculUoluU=; b=gmdQ64XjukuGSOqzvbR/j24Lj0wSsltM7dB2fHzj/V1BS+2mK+WRXV5QzDwx2pNdy8E3NI4RFCRrta7LRal4DIQ4BRHKKc2gF38Hi3US71sNY8HNFEoneLoS7kXrt8b/Jp0ww2IarXVkpES22N/qh4j+IXhxiXbCKMVXkTrp5kVHrMg/ND7//h3RdyF59CL+Lxr7E4rEB5RdnIUHzzazmIHWQdC9S950JGJnQ27ZAiyJAvwUqRvePNP4qgyAYX5AQ7FQCXNKIpR/SpQUJ7z6UUwHOI7QN8qqb6ng/UAe17hppY1h+Dk7rLDNsjHFJdtAjsDQzcmoPOv3jRwaIjBbhA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=irPRvaBDaEQrHwfm0kPhFL3GIsCF0Jf2WOsculUoluU=; b=NFYnXtiC5ejR8qvTmWw2lh6RfISo6Cloz5uJRaKhqMT1DeiumMUb9hVl5bwN0+K+dlJhQHoTURG7pPt40ybK541qAd9zzNFcmbGLbZr4eodwDvdUouLVYwpAq9jQFZ+wtRxF4HU2l6O4/cta94DxhfaCdvhH7k6ZfWNPS5OODh0= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BY5PR10MB4033.namprd10.prod.outlook.com (2603:10b6:a03:1fe::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.23; Mon, 19 Jul 2021 06:28:14 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4331.032; Mon, 19 Jul 2021 06:28:14 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC 19/19] vfio-user: add migration cli options and version negotiation Date: Sun, 18 Jul 2021 23:27:58 -0700 Message-Id: <95efca08f3160ce4ac2fb49fe9c9bc1183ca187d.1626675354.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:3c73:d99e:ec8c:2487) by BY5PR16CA0022.namprd16.prod.outlook.com (2603:10b6:a03:1a0::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4331.21 via Frontend Transport; Mon, 19 Jul 2021 06:28:13 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: ffd1afcb-ade0-4e93-f3f7-08d94a7e6311 X-MS-TrafficTypeDiagnostic: BY5PR10MB4033: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qexwhun7hirq+MBFuLn0bx2uuzwPXey3gILXmrNY3fzmhyh8sG9nSYtyQg7TOa/pcceNM7L5UoyofmFXfE5l6oBnUwK0b/tbAN1E9C+eQ1/7DUiVW1gmmhzOo0Af6KFL/gy4ftfH+zGuGIN1Uy3DgZw4F1ar7CU+UwVLetuYuK8GhqbdYHGw2/3IkLdz5x/Rt3iZBE8mqC3M/4vlPO+xlr/2Bdq392mGrkNC/aGKg7lDrOjNbEnRHf4CEPO6CKWgkzHqfr2TdUuUGAj9vCA80/RTqP9LMPCoykQu2tvaMJfFts8k35iG4K2bEAxjmNMtFYQzKi26FlaT16hf99q7rx0/sqA8sQlqFXLJmkidAODwAMBfu4Aeu6lZv9HuQJU9xfDer5mF4gs7q8jOS7KEd21QT8/3+gD54OlyFPFTKsJbUT4U+OOcgJzsnebYwwZPpkjuXiCt4cvNyj66ehwwM6W7OqADo6Fn1JPEsYRH5vt4lJLU0esRgoPfEBa6MKBP/+kDXe8FsVdzwlGObxJC3GNzKgUzAGBGVZt+TRE0xy1P3dn8XkcEImSVo2m/QSP50gGBISq7yw6M44P2Tw2Px3LbhG2eHA56NW4RiYIe27Fe/O8p/oDPeDJQSCYHxHBcMTDfrdtJ2Jj7nGTPCcCKSA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(136003)(396003)(39860400002)(376002)(346002)(366004)(38100700002)(6506007)(186003)(52116002)(6512007)(2906002)(6666004)(8936002)(478600001)(2616005)(44832011)(36756003)(5660300002)(86362001)(107886003)(6486002)(6916009)(8676002)(4326008)(66946007)(66556008)(66476007)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4C3bZzKeLRCWpDPOsJHA8ULVlpN3kEIA+wvGlnuY8iF4k5j4VyVnEnxpjNUI/ZNFdo/aPUp6dfBpZjfcq/q44ZV2bGhrP0jQSk/GKX3NPS2dppoOk2NJtIIbr4Zd298f51EOSu0lfDV3GudZri9SIh6BkvwNshvJREXuUB0Ag0pwUjOOaLXsRU7JOONUszYYgp51XVJwuM+IyJF7BZ/jJDwaC1p6zKB++GZWWc4bv038pg3kQ+E2fFtASuh+lRidj59glETMTaWAFgnqQ+MqGzYl+U1PrARbAoFYIVtT8/SWuOTVdyjfuqVIhdUYTUqVbWHaqvuTDJ940ymC2g64p96sl3GrIDr5twJaAiEWU+w/p/8mVAgqkZiic4C5TUXsLWd8Gn+5QKc67hd7MK4GrK181QjhzfOs3TZg58+frYdSoODfh+EDC7oZ2aBgcvwDoppKItLAByem7x1VlgM7EaTPJ27jea/71wO/Dgd9KepQCGW5ywn1HOfuw9Es1yJdEV1qSQ9m4XWGiKU6IRGaOtUivup97wrDSueLM0mWff+vuMnZobfMXlVNwP09Zoel+ZwglIZZCXe5RkSWn5Ds3qj7hnDJVWv8mPiUZLsQ1hK8OR4aW9sELv2+ilM5BtPChVg3RFGoo+t1tArZ/Oe8/CFTsuWbUCTGSOMxKEVJInnC5gNSnL3CCRXzQW1VUBlQFO5gwJMD8yDdWDkjhLvYNPOvEzOBoy7a27GaOkeH5M/O4221Sa1pIjiIzpAkP5+FyH9bUc3IU0yifvk8m1bvs9AjmnCI4s5trZ45x27Umc6g3cJVKfG39gd11Og+exGMlE5rpN6g6+i/LoLsns7Rotc9rTClzUVPjiFPapalH0dw4+1H9da50o6RjAelpr3Qt43jwKihvzrRWWLLGWAD4waH0U2xnwHnvHNr19Ux4u0hForLXmd32SvmbdIByIQMbF0WsdImVG4gcRASRsBOG5U7mmgimB6P/wCMlFiTr/TWqzigDAPKvOPVfpX1dBUKLSp3QYqyEeS2VOwXrMLbI0oTm9v2nR/C5H1SkmcSM/K6Tl9rrIHmNziu8+gQMprXlKH9Uh6n0gGWT7196/PXerbyxXtqXY89vep4Xmord5n2shTNkggxde0tAiqi7PoYw72xjMogPgSdYSkbW0Ji8EjMrZl9Sb23ZBzbUPdRmLgdAQoPPLoW/FoEtPRfNsf/yP2sUAJNBmBubIoV6nLgko2wxzPQ1MZG4GWkRRhqxQdVl8NsD6RhUKha/bMwDWV/1zq0hmKwHKa0vR3Q58S+V177lJ3IJ6VumY27dzrOpK8Ow2yVL1/6+qag3ii1kU6M3IwJum/X9Hig6gtvr3NimhvntByu95S8IYRLwZ6S7ZN6UBhI7HZYlkFXqIZLkiSk X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: ffd1afcb-ade0-4e93-f3f7-08d94a7e6311 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 19 Jul 2021 06:28:13.9230 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 0WdqTsB0FOqmMRuVzgEDmEeHOMvlX6fhBbaD+rnRRO910kchtZWWca+U0XyhrZ184DHHgTJgqqeOXZN2IqeYjcxDOM+z8JlZAa48tsqlMuA= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB4033 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10049 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 mlxscore=0 malwarescore=0 adultscore=0 phishscore=0 suspectscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2104190000 definitions=main-2107190035 X-Proofpoint-GUID: DbWyipfUtA7ErF9umCOB0JIlLL6sbAmF X-Proofpoint-ORIG-GUID: DbWyipfUtA7ErF9umCOB0JIlLL6sbAmF Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -18 X-Spam_score: -1.9 X-Spam_bar: - X-Spam_report: (-1.9 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.01, RCVD_IN_MSPIKE_L3=0.9, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John G Johnson Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva --- hw/vfio/user.h | 4 ++++ hw/vfio/pci.c | 5 +++++ hw/vfio/user.c | 33 +++++++++++++++++++++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index eeb328c0a9..5542aa1932 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -70,6 +70,10 @@ struct vfio_user_version { /* "capabilities" members */ #define VFIO_USER_CAP_MAX_FDS "max_msg_fds" #define VFIO_USER_CAP_MAX_XFER "max_data_xfer_size" +#define VFIO_USER_CAP_MIGR "migration" + +/* "migration" member */ +#define VFIO_USER_CAP_PGSIZE "pgsize" #define VFIO_USER_DEF_MAX_FDS 8 #define VFIO_USER_MAX_MAX_FDS 16 diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 36f8524e7c..2f97160147 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3688,6 +3688,11 @@ static void vfio_user_pci_reset(DeviceState *dev) static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure, false), + DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice, + vbasedev.enable_migration, false), + DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevice, + vbasedev.pre_copy_dirty_page_tracking, + ON_OFF_AUTO_ON), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index eceaeeccea..23ace82bbb 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -393,6 +393,23 @@ static int caps_parse(QDict *qdict, struct cap_entry caps[], Error **errp) return 0; } +static int check_pgsize(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t pgsize; + + if (qn == NULL || !qnum_get_try_uint(qn, &pgsize)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_PGSIZE); + return -1; + } + return pgsize == 4096 ? 0 : -1; +} + +static struct cap_entry caps_migr[] = { + { VFIO_USER_CAP_PGSIZE, check_pgsize }, + { NULL } +}; + static int check_max_fds(QObject *qobj, Error **errp) { QNum *qn = qobject_to(QNum, qobj); @@ -417,9 +434,21 @@ static int check_max_xfer(QObject *qobj, Error **errp) return 0; } +static int check_migr(QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL || caps_parse(qdict, caps_migr, errp)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return -1; + } + return 0; +} + static struct cap_entry caps_cap[] = { { VFIO_USER_CAP_MAX_FDS, check_max_fds }, { VFIO_USER_CAP_MAX_XFER, check_max_xfer }, + { VFIO_USER_CAP_MIGR, check_migr }, { NULL } }; @@ -466,8 +495,12 @@ static GString *caps_json(void) { QDict *dict = qdict_new(); QDict *capdict = qdict_new(); + QDict *migdict = qdict_new(); GString *str; + qdict_put_int(migdict, VFIO_USER_CAP_PGSIZE, 4096); + qdict_put_obj(capdict, VFIO_USER_CAP_MIGR, QOBJECT(migdict)); + qdict_put_int(capdict, VFIO_USER_CAP_MAX_FDS, VFIO_USER_MAX_MAX_FDS); qdict_put_int(capdict, VFIO_USER_CAP_MAX_XFER, VFIO_USER_DEF_MAX_XFER);