From patchwork Mon Aug 16 16:42:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT,WEIRD_QUOTING autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D638C4338F for ; Mon, 16 Aug 2021 16:54:23 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 982B160EFE for ; Mon, 16 Aug 2021 16:54:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 982B160EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:53336 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfsT-0003zN-Gy for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:54:21 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37224) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhi-00010q-BS for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:14 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:49738) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhY-0007y3-Vn for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:14 -0400 Received: from pps.filterd (m0246617.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaeJP011290; Mon, 16 Aug 2021 16:43:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=VK4HpqJzexgufdeGvvyFkzlbHQd0SjE/hKmlvmuY7Vw=; b=tawX31y1KVxis0h1yGTnIOTHfWy+cv+w5szBxG7hkumWKFw18/5FHdNhGILPVlZ3xbhW /YQ1Grho605+JzFO0XpQHk89TfUT8RTKPKBVIktDmcdPof9e8ouLm7Huu7Ec3Bvva0UL G9EnFZsCkQSqUgbvH6LD51cIMSOfw40NDdWr6odtLWfykO8bDG1f7UTI6h9fMnHOSVjB 4vxJKBf+MaTNILx7W5LhroQBzkw5uurU2GI+aQors14TLDQRHJQh+wDIVF6xxQwJEwUb DcT2d8fNYsIUDFah24ILa3C+80AfAeTSFYIICMhnxeW6jDb9Uk71unCdtSBLzyag7F3/ fg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=VK4HpqJzexgufdeGvvyFkzlbHQd0SjE/hKmlvmuY7Vw=; b=yIZoFJ/WeNnZtGzuLaiaGeX2wie3H6TeJ+LFYxLig9LQQxaZxxDMVkp73z+RJdokd06O Br7FzdE9qgKYimCcvfe74LTw3JKDjseMLubdNwqoJHBYKJVV7sH3Jf1BPHuJ0Df1xXj0 bhHGLI2CH6pGlcF2TTGu+qm8EihI2+97q5GJtZEtYcFhKcoqoxxMlwRe/OQJeLeGjBhh JQ3PRVitxPvflGnsENPQ9XkwZjTymhSoxVYdNaT6MTZ7tSJ9vzvg6zwFsmqDsBhCwQgx EUicqs4f9eg6cLoG1pxGh5EIlgnzg4lywd4vg8JZ9j/GByW5/1xBG2P1o0ttTdSA3XJI ug== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by mx0b-00069f02.pphosted.com with ESMTP id 3af83029gn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:03 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeu1k137465; Mon, 16 Aug 2021 16:43:01 GMT Received: from nam12-bn8-obe.outbound.protection.outlook.com (mail-bn8nam12lp2168.outbound.protection.outlook.com [104.47.55.168]) by userp3020.oracle.com with ESMTP id 3aeqksfrwu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iUeQH9O08eZWVH5Emwn2RjnZfjywa54vdLKH71ypbg0RANkueBQSUHmjUV7lzFyp2jnky66K8S4QUOdOuXEVYYq+ifxYpi97Cmf60dpTSc2hFMhVk5TDU49paK4XG+CJSzydATf2WFJskzShFXYQt0poMrufBt4DFV3Vau7kE8j9oWutLvdJjBknLoYdjrbU0SgwpmJ0ZlbQqhbEk2T4knDETbST418/Y8h4gtJNd55i4N+OOGmMc685ewFrdyAoBcx/gvYN3ECM5Top6P91QIF4tsuBIvp5PzNHEW/GSA5vL5h2taqI7Wb3TJ9BKLi+fVl1JRB/J6qwwowKpzzDNQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VK4HpqJzexgufdeGvvyFkzlbHQd0SjE/hKmlvmuY7Vw=; b=nKsxGOYS6ii1ZJPrK/1+EcAcTHvdC+PJBM/mHQAkAVqS4U18zpZrl6ENeEQfNcc6cLVs/jlaLFeWAFf+vyDNA4KzHR7qX65nUWJdS1wp84dHjZiE2O4Rxj6dd4CuCqSBD8nR29sa4wIzcCx9oGcCdHPPIPTj1uoDKHbnbO8pUwIfcdnw+CyJOOE+nNMHUYvVObbuJ3ttonmCOrrJYBmbPo4OmJtFR710YEe83dwY89Ac/2mOt4dhHwKxfM4lQeuNkHNyApUPfuYSIe01aCWhf8gQ8XqEw6TP4JAZQBpoiwFq+NU7Kzmu6VYKzt6BrVA8GozRFZlatjoywwPqmpI9IQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=VK4HpqJzexgufdeGvvyFkzlbHQd0SjE/hKmlvmuY7Vw=; b=Ru18iiu6SjzjWOrVY2SCIsVJ6bJVJxZ379jt1R9WrSq4bXCsfub+TTKzqO0GQLEdCevfLdyH2tIPJXiN5vkpzrlV2xBD14ykUHrnE4/J2/IhnywP2nrPaNdpSAy1KsSm75MkGnvgBexe/C/18GxXnY/K3rdciaeYuusyaee4H4U= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BYAPR10MB2565.namprd10.prod.outlook.com (2603:10b6:a02:b2::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Mon, 16 Aug 2021 16:42:55 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:55 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 01/16] vfio-user: introduce vfio-user protocol specification Date: Mon, 16 Aug 2021 09:42:34 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:55 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 254ed045-3fe1-497a-aed3-08d960d4e5d4 X-MS-TrafficTypeDiagnostic: BYAPR10MB2565: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ilXB+GKI0tbqyUDJxsyXz4kSo678jNlkwGGjwRmyxKp/2ZjSIxeJpm7/yzySEGuVho3iSjQUG5wGtSEKjTPtW28NITJ9IR5+1Tk7747cetrTbIR7oA+26YsG3z3Z+Wl/xVolShNC44EBEDHXYv+lFR/8Hhj/7Ez2pJspH63LVSnKOwtnzXqq3AobRFR7I44bV0ZcEAB8ZCG29Phd22bMV7uu9rwIRg4X0E8yhHsm6L7k9oX/zgUZD0vzdWNZvNbiKLiwVBi7SPoNeq9O099IPnw9EIxelTdyzy8w3f77KXWP2J7xRYcBOKz3W8aUvj+NSX5xJlVL3mB0vIPq98RbHsCawwFNlAsJPrAPoHUeawGdiP/39Jl8tfuo0Y0imWBO4Jsr5T6iET7bgCYQWpKzOFjNxoxb/nIExr03fRAYs9yAxFlyrZrFqNyOCDOOYdOWXKxdgr9ixwgV/CyfN0cs+g4/Xt4A7COjpYIyDlozjpwEhDxheAcw7uVdrLtKV/lVNUmO/KYKM547dyRLuvPTQeh0Hj4SLssYP8PIZ+HFIsjjfhbwyFCK1T/e7DfHmyUVTAoOpqUFYcxEqzS8S65onUMZ/G4F2rie7D0Bul5v1UbhmpbaceA09B5SW4+HCTyudU5ZFwAJzJka+BMy9nLfMEleIBPyJeIDN3IkdpQuXR6Y3O7xID2eOumTBOpf5AetDtUq8QEc+lfI3RWotCIU62gKsm2dPm8M27q1GhoBb6TWtTcPSHXYUw/qNOImj645CW1QOowenVYIk+a7v0Pwhg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(6486002)(316002)(38100700002)(44832011)(2616005)(6512007)(2906002)(6916009)(52116002)(5660300002)(966005)(508600001)(6506007)(66946007)(4326008)(66476007)(66556008)(6666004)(30864003)(186003)(86362001)(8676002)(8936002)(36756003)(83380400001)(579004)(559001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: rPyVYY9Wx7W4BBRwbebk4fRmaBYnk0Wt5Z7VyJLDHvCRjc7FpOGYJi7zspg3VcI7130m00EAr79sBjqNHqu+0SV1H8J1skJxVOppJi1GTD/tgbq6kONo2x7xqXBROJvCxZrxcZ3W5No7Uv9mr4vhGPGgA0FQiTCzjmnh4B0Q6eVVoTLTOReo1ji/oQgfdem+imJMwjFAIWqAQpRItiSyYuWlsZOFe65G5DGBggMIH2XK53njKsEecoOj36HpmZ8Dcv0dUw2F9fvV0bKGoxglm2kGtKZHWVDW7RvrxCTJ5FTWUPrmEXO2NtIltFrLhI9Jc5WM6bDJ1pU7mClXeZKKPOIJG0iFKYVznw71eABUv16NMjqMtG76CK+ALENqELeVo8EjDHrFnUQ4hUww9Ry0D77skx85gI23hCCSQ1gnCupH4CbeBhYvf7HlbeuZlfkZs9E+eoSKfnFcv/i515xFWU9H7cjrVcIXI5ujLjbkeN4x2GYfi4wAv16jFCcKv0jd4gmV5I+REonlMI4r8dLk863cBmbOMj89/f5XbKvnSYjF+8PF/arLJeZEW+Tu+bkDcxcEVpOAVhYGVYSl9fs/27hcFzqyLJkJXZRZ3qp32MpkAMgQDf2xbD7aoCiKrrjxeB1AoePTKFi5OJvn1JoYRFciTdxFnFDRCKjOG7DDe9C4AHAojbSZFTyZSE66V18+Lml+bvKO5fsjylGH5WUM+aM8JVWR766+UJNPgfDyc2/gcZb0PLodtFtDSttONbdym9FojPwveGki/7Hm56Twv4VhmXXJ+rQvLRMP+NdlExJYtNFG5zpNa88YnVSSmupZUkqc1wLzgVCnu8P7QupN2EIVGkqa9LtgQtiScus2bXCRJXUnGaI2jaJMRt+06LdIJnU9ipTiwBDCMxKJzGoDGppbihrVDczhlY0oRMQ/F5L5MMcjCexdDz0ezVNXLFjC7T5ltwdepEZ8a+ZTNL1QfE2XqgPvjxUm16j1BGQnGq7rHa+FxgXFy3nZmP+R/jhFxGq4U4x5pHd3olpjYZYukw7DY+Te0aWGzAckoSjxpCPW3nzrjF9AEljvWicUX27ALI/FUfXkibZIj6PWKMeJbE+7GIIaE0t//YLaOd5EiH3J2xli/bLoYttb9u/owZR4Yay3Ma9Wa7Mgq8zEoc8W90VgjA9Ld0voAybAhsWv03KmfCaiGdiWkx173/MCfRtQgFNSiheaZCrJ3Ods/hao5S/ZphV13IVKJah0x2xabhdPz8RaIwpFX3ZMZsKxsLzwnFS0GONSLQb0pLrjNtxXvEVmAHJwmhAx/qNsYPDbHMudJMoI16ioWYuSEpKO7QXmeVh8XSbUqdfbkAwUGlJWFeW5dGwtJ229UkVR1Gq/yk7i7FxdDDTEcE84syMOvuKu X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 254ed045-3fe1-497a-aed3-08d960d4e5d4 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:55.7708 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 9lWDrznxZlwkB7nLuqqt5bG5tC/r5iaCXRN4k9p16683dGGczm0vJg/rAtJKdRZReqEJQS0vnJWGGRX7PlbJRtg6tvDdXRS/qh8NkIbWyD4= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2565 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 bulkscore=0 malwarescore=0 mlxscore=0 spamscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-ORIG-GUID: b3CZXGB2kNzqAF6ztMS0yQ1kxSOXlY8b X-Proofpoint-GUID: b3CZXGB2kNzqAF6ztMS0yQ1kxSOXlY8b Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, WEIRD_QUOTING=0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: Thanos Makatos This patch introduces the vfio-user protocol specification (formerly known as VFIO-over-socket), which is designed to allow devices to be emulated outside QEMU, in a separate process. vfio-user reuses the existing VFIO defines, structs and concepts. This patch is sourced from: https://patchwork.kernel.org/project/qemu-devel/patch/20210614104608.212276-1-thanos.makatos@nutanix.com/ It has been earlier discussed as an RFC in: "RFC: use VFIO over a UNIX domain socket to implement device offloading" Signed-off-by: John G Johnson Signed-off-by: Thanos Makatos Signed-off-by: John Levon --- docs/devel/index.rst | 1 + docs/devel/vfio-user.rst | 1809 ++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 6 + 3 files changed, 1816 insertions(+) create mode 100644 docs/devel/vfio-user.rst diff --git a/docs/devel/index.rst b/docs/devel/index.rst index 5522db7241..304ca1c12f 100644 --- a/docs/devel/index.rst +++ b/docs/devel/index.rst @@ -44,3 +44,4 @@ modifying QEMU's source code. vfio-migration qapi-code-gen writing-qmp-commands + vfio-user diff --git a/docs/devel/vfio-user.rst b/docs/devel/vfio-user.rst new file mode 100644 index 0000000000..0b2acec101 --- /dev/null +++ b/docs/devel/vfio-user.rst @@ -0,0 +1,1809 @@ +.. include:: +******************************** +vfio-user Protocol Specification +******************************** + +-------------- +Version_ 0.9.1 +-------------- + +.. contents:: Table of Contents + +Introduction +============ +vfio-user is a protocol that allows a device to be emulated in a separate +process outside of a Virtual Machine Monitor (VMM). vfio-user devices consist +of a generic VFIO device type, living inside the VMM, which we call the client, +and the core device implementation, living outside the VMM, which we call the +server. + +The vfio-user specification is partly based on the +`Linux VFIO ioctl interface `_. + +VFIO is a mature and stable API, backed by an extensively used framework. The +existing VFIO client implementation in QEMU (``qemu/hw/vfio/``) can be largely +re-used, though there is nothing in this specification that requires that +particular implementation. None of the VFIO kernel modules are required for +supporting the protocol, on either the client or server side. Some source +definitions in VFIO are re-used for vfio-user. + +The main idea is to allow a virtual device to function in a separate process in +the same host over a UNIX domain socket. A UNIX domain socket (``AF_UNIX``) is +chosen because file descriptors can be trivially sent over it, which in turn +allows: + +* Sharing of client memory for DMA with the server. +* Sharing of server memory with the client for fast MMIO. +* Efficient sharing of eventfd's for triggering interrupts. + +Other socket types could be used which allow the server to run in a separate +guest in the same host (``AF_VSOCK``) or remotely (``AF_INET``). Theoretically +the underlying transport does not necessarily have to be a socket, however we do +not examine such alternatives. In this protocol version we focus on using a UNIX +domain socket and introduce basic support for the other two types of sockets +without considering performance implications. + +While passing of file descriptors is desirable for performance reasons, support +is not necessary for either the client or the server in order to implement the +protocol. There is always an in-band, message-passing fall back mechanism. + +Overview +======== + +VFIO is a framework that allows a physical device to be securely passed through +to a user space process; the device-specific kernel driver does not drive the +device at all. Typically, the user space process is a VMM and the device is +passed through to it in order to achieve high performance. VFIO provides an API +and the required functionality in the kernel. QEMU has adopted VFIO to allow a +guest to directly access physical devices, instead of emulating them in +software. + +vfio-user reuses the core VFIO concepts defined in its API, but implements them +as messages to be sent over a socket. It does not change the kernel-based VFIO +in any way, in fact none of the VFIO kernel modules need to be loaded to use +vfio-user. It is also possible for the client to concurrently use the current +kernel-based VFIO for one device, and vfio-user for another device. + +VFIO Device Model +----------------- + +A device under VFIO presents a standard interface to the user process. Many of +the VFIO operations in the existing interface use the ``ioctl()`` system call, and +references to the existing interface are called the ``ioctl()`` implementation in +this document. + +The following sections describe the set of messages that implement the vfio-user +interface over a socket. In many cases, the messages are analogous to data +structures used in the ``ioctl()`` implementation. Messages derived from the +``ioctl()`` will have a name derived from the ``ioctl()`` command name. E.g., the +``VFIO_DEVICE_GET_INFO`` ``ioctl()`` command becomes a +``VFIO_USER_DEVICE_GET_INFO`` message. The purpose of this reuse is to share as +much code as feasible with the ``ioctl()`` implementation``. + +Connection Initiation +^^^^^^^^^^^^^^^^^^^^^ + +After the client connects to the server, the initial client message is +``VFIO_USER_VERSION`` to propose a protocol version and set of capabilities to +apply to the session. The server replies with a compatible version and set of +capabilities it supports, or closes the connection if it cannot support the +advertised version. + +Device Information +^^^^^^^^^^^^^^^^^^ + +The client uses a ``VFIO_USER_DEVICE_GET_INFO`` message to query the server for +information about the device. This information includes: + +* The device type and whether it supports reset (``VFIO_DEVICE_FLAGS_``), +* the number of device regions, and +* the device presents to the client the number of interrupt types the device + supports. + +Region Information +^^^^^^^^^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_REGION_INFO`` messages to query the +server for information about the device's regions. This information describes: + +* Read and write permissions, whether it can be memory mapped, and whether it + supports additional capabilities (``VFIO_REGION_INFO_CAP_``). +* Region index, size, and offset. + +When a device region can be mapped by the client, the server provides a file +descriptor which the client can ``mmap()``. The server is responsible for +polling for client updates to memory mapped regions. + +Region Capabilities +""""""""""""""""""" + +Some regions have additional capabilities that cannot be described adequately +by the region info data structure. These capabilities are returned in the +region info reply in a list similar to PCI capabilities in a PCI device's +configuration space. + +Sparse Regions +"""""""""""""" +A region can be memory-mappable in whole or in part. When only a subset of a +region can be mapped by the client, a ``VFIO_REGION_INFO_CAP_SPARSE_MMAP`` +capability is included in the region info reply. This capability describes +which portions can be mapped by the client. + +.. Note:: + For example, in a virtual NVMe controller, sparse regions can be used so + that accesses to the NVMe registers (found in the beginning of BAR0) are + trapped (an infrequent event), while allowing direct access to the doorbells + (an extremely frequent event as every I/O submission requires a write to + BAR0), found in the next page after the NVMe registers in BAR0. + +Device-Specific Regions +""""""""""""""""""""""" + +A device can define regions additional to the standard ones (e.g. PCI indexes +0-8). This is achieved by including a ``VFIO_REGION_INFO_CAP_TYPE`` capability +in the region info reply of a device-specific region. Such regions are reflected +in ``struct vfio_user_device_info.num_regions``. Thus, for PCI devices this +value can be equal to, or higher than, ``VFIO_PCI_NUM_REGIONS``. + +Region I/O via file descriptors +------------------------------- + +For unmapped regions, region I/O from the client is done via +``VFIO_USER_REGION_READ/WRITE``. As an optimization, ioeventfds or ioregionfds +may be configured for sub-regions of some regions. A client may request +information on these sub-regions via ``VFIO_USER_DEVICE_GET_REGION_IO_FDS``; by +configuring the returned file descriptors as ioeventfds or ioregionfds, the +server can be directly notified of I/O (for example, by KVM) without taking a +trip through the client. + +Interrupts +^^^^^^^^^^ + +The client uses ``VFIO_USER_DEVICE_GET_IRQ_INFO`` messages to query the server +for the device's interrupt types. The interrupt types are specific to the bus +the device is attached to, and the client is expected to know the capabilities +of each interrupt type. The server can signal an interrupt by directly injecting +interrupts into the guest via an event file descriptor. The client configures +how the server signals an interrupt with ``VFIO_USER_SET_IRQS`` messages. + +Device Read and Write +^^^^^^^^^^^^^^^^^^^^^ + +When the guest executes load or store operations to an unmapped device region, +the client forwards these operations to the server with +``VFIO_USER_REGION_READ`` or ``VFIO_USER_REGION_WRITE`` messages. The server +will reply with data from the device on read operations or an acknowledgement on +write operations. See `Read and Write Operations`_. + +Client memory access +-------------------- + +The client uses ``VFIO_USER_DMA_MAP`` and ``VFIO_USER_DMA_UNMAP`` messages to +inform the server of the valid DMA ranges that the server can access on behalf +of a device (typically, VM guest memory). DMA memory may be accessed by the +server via ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages over the +socket. In this case, the "DMA" part of the naming is a misnomer. + +Actual direct memory access of client memory from the server is possible if the +client provides file descriptors the server can ``mmap()``. Note that ``mmap()`` +privileges cannot be revoked by the client, therefore file descriptors should +only be exported in environments where the client trusts the server not to +corrupt guest memory. + +See `Read and Write Operations`_. + +Client/server interactions +========================== + +Socket +------ + +A server can serve: + +1) one or more clients, and/or +2) one or more virtual devices, belonging to one or more clients. + +The current protocol specification requires a dedicated socket per +client/server connection. It is a server-side implementation detail whether a +single server handles multiple virtual devices from the same or multiple +clients. The location of the socket is implementation-specific. Multiplexing +clients, devices, and servers over the same socket is not supported in this +version of the protocol. + +Authentication +-------------- + +For ``AF_UNIX``, we rely on OS mandatory access controls on the socket files, +therefore it is up to the management layer to set up the socket as required. +Socket types than span guests or hosts will require a proper authentication +mechanism. Defining that mechanism is deferred to a future version of the +protocol. + +Command Concurrency +------------------- + +A client may pipeline multiple commands without waiting for previous command +replies. The server will process commands in the order they are received. A +consequence of this is if a client issues a command with the *No_reply* bit, +then subsequently issues a command without *No_reply*, the older command will +have been processed before the reply to the younger command is sent by the +server. The client must be aware of the device's capability to process +concurrent commands if pipelining is used. For example, pipelining allows +multiple client threads to concurrently access device regions; the client must +ensure these accesses obey device semantics. + +An example is a frame buffer device, where the device may allow concurrent +access to different areas of video memory, but may have indeterminate behavior +if concurrent accesses are performed to command or status registers. + +Note that unrelated messages sent from the server to the client can appear in +between a client to server request/reply and vice versa. + +Implementers should be prepared for certain commands to exhibit potentially +unbounded latencies. For example, ``VFIO_USER_DEVICE_RESET`` may take an +arbitrarily long time to complete; clients should take care not to block +unnecessarily. + +Socket Disconnection Behavior +----------------------------- +The server and the client can disconnect from each other, either intentionally +or unexpectedly. Both the client and the server need to know how to handle such +events. + +Server Disconnection +^^^^^^^^^^^^^^^^^^^^ +A server disconnecting from the client may indicate that: + +1) A virtual device has been restarted, either intentionally (e.g. because of a + device update) or unintentionally (e.g. because of a crash). +2) A virtual device has been shut down with no intention to be restarted. + +It is impossible for the client to know whether or not a failure is +intermittent or innocuous and should be retried, therefore the client should +reset the VFIO device when it detects the socket has been disconnected. +Error recovery will be driven by the guest's device error handling +behavior. + +Client Disconnection +^^^^^^^^^^^^^^^^^^^^ +The client disconnecting from the server primarily means that the client +has exited. Currently, this means that the guest is shut down so the device is +no longer needed therefore the server can automatically exit. However, there +can be cases where a client disconnection should not result in a server exit: + +1) A single server serving multiple clients. +2) A multi-process QEMU upgrading itself step by step, which is not yet + implemented. + +Therefore in order for the protocol to be forward compatible, the server should +respond to a client disconnection as follows: + + - all client memory regions are unmapped and cleaned up (including closing any + passed file descriptors) + - all IRQ file descriptors passed from the old client are closed + - the device state should otherwise be retained + +The expectation is that when a client reconnects, it will re-establish IRQ and +client memory mappings. + +If anything happens to the client (such as qemu really did exit), the control +stack will know about it and can clean up resources accordingly. + +Security Considerations +----------------------- + +Speaking generally, vfio-user clients should not trust servers, and vice versa. +Standard tools and mechanisms should be used on both sides to validate input and +prevent against denial of service scenarios, buffer overflow, etc. + +Request Retry and Response Timeout +---------------------------------- +A failed command is a command that has been successfully sent and has been +responded to with an error code. Failure to send the command in the first place +(e.g. because the socket is disconnected) is a different type of error examined +earlier in the disconnect section. + +.. Note:: + QEMU's VFIO retries certain operations if they fail. While this makes sense + for real HW, we don't know for sure whether it makes sense for virtual + devices. + +Defining a retry and timeout scheme is deferred to a future version of the +protocol. + +Message sizes +------------- + +Some requests have an ``argsz`` field. In a request, it defines the maximum +expected reply payload size, which should be at least the size of the fixed +reply payload headers defined here. The *request* payload size is defined by the +usual ``msg_size`` field in the header, not the ``argsz`` field. + +In a reply, the server sets ``argsz`` field to the size needed for a full +payload size. This may be less than the requested maximum size. This may be +larger than the requested maximum size: in that case, the full payload is not +included in the reply, but the ``argsz`` field in the reply indicates the needed +size, allowing a client to allocate a larger buffer for holding the reply before +trying again. + +In addition, during negotiation (see `Version`_), the client and server may +each specify a ``max_data_xfer_size`` value; this defines the maximum data that +may be read or written via one of the ``VFIO_USER_DMA/REGION_READ/WRITE`` +messages; see `Read and Write Operations`_. + +Protocol Specification +====================== + +To distinguish from the base VFIO symbols, all vfio-user symbols are prefixed +with ``vfio_user`` or ``VFIO_USER``. In this revision, all data is in the +little-endian format, although this may be relaxed in future revisions in cases +where the client and server are both big-endian. + +Unless otherwise specified, all sizes should be presumed to be in bytes. + +.. _Commands: + +Commands +-------- +The following table lists the VFIO message command IDs, and whether the +message command is sent from the client or the server. + +====================================== ========= ================= +Name Command Request Direction +====================================== ========= ================= +``VFIO_USER_VERSION`` 1 client -> server +``VFIO_USER_DMA_MAP`` 2 client -> server +``VFIO_USER_DMA_UNMAP`` 3 client -> server +``VFIO_USER_DEVICE_GET_INFO`` 4 client -> server +``VFIO_USER_DEVICE_GET_REGION_INFO`` 5 client -> server +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` 6 client -> server +``VFIO_USER_DEVICE_GET_IRQ_INFO`` 7 client -> server +``VFIO_USER_DEVICE_SET_IRQS`` 8 client -> server +``VFIO_USER_REGION_READ`` 9 client -> server +``VFIO_USER_REGION_WRITE`` 10 client -> server +``VFIO_USER_DMA_READ`` 11 server -> client +``VFIO_USER_DMA_WRITE`` 12 server -> client +``VFIO_USER_DEVICE_RESET`` 13 client -> server +``VFIO_USER_DIRTY_PAGES`` 14 client -> server +====================================== ========= ================= + +Header +------ + +All messages, both command messages and reply messages, are preceded by a +16-byte header that contains basic information about the message. The header is +followed by message-specific data described in the sections below. + ++----------------+--------+-------------+ +| Name | Offset | Size | ++================+========+=============+ +| Message ID | 0 | 2 | ++----------------+--------+-------------+ +| Command | 2 | 2 | ++----------------+--------+-------------+ +| Message size | 4 | 4 | ++----------------+--------+-------------+ +| Flags | 8 | 4 | ++----------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0-3 | Type | | +| | +-----+------------+ | +| | | 4 | No_reply | | +| | +-----+------------+ | +| | | 5 | Error | | +| | +-----+------------+ | ++----------------+--------+-------------+ +| Error | 12 | 4 | ++----------------+--------+-------------+ +| | 16 | variable | ++----------------+--------+-------------+ + +* *Message ID* identifies the message, and is echoed in the command's reply + message. Message IDs belong entirely to the sender, can be re-used (even + concurrently) and the receiver must not make any assumptions about their + uniqueness. +* *Command* specifies the command to be executed, listed in Commands_. It is + also set in the reply header. +* *Message size* contains the size of the entire message, including the header. +* *Flags* contains attributes of the message: + + * The *Type* bits indicate the message type. + + * *Command* (value 0x0) indicates a command message. + * *Reply* (value 0x1) indicates a reply message acknowledging a previous + command with the same message ID. + * *No_reply* in a command message indicates that no reply is needed for this + command. This is commonly used when multiple commands are sent, and only + the last needs acknowledgement. + * *Error* in a reply message indicates the command being acknowledged had + an error. In this case, the *Error* field will be valid. + +* *Error* in a reply message is an optional UNIX errno value. It may be zero + even if the Error bit is set in Flags. It is reserved in a command message. + +Each command message in Commands_ must be replied to with a reply message, +unless the message sets the *No_Reply* bit. The reply consists of the header +with the *Reply* bit set, plus any additional data. + +If an error occurs, the reply message must only include the reply header. + +As the header is standard in both requests and replies, it is not included in +the command-specific specifications below; each message definition should be +appended to the standard header, and the offsets are given from the end of the +standard header. + +``VFIO_USER_VERSION`` +--------------------- + +.. _Version: + +This is the initial message sent by the client after the socket connection is +established; the same format is used for the server's reply. + +Upon establishing a connection, the client must send a ``VFIO_USER_VERSION`` +message proposing a protocol version and a set of capabilities. The server +compares these with the versions and capabilities it supports and sends a +``VFIO_USER_VERSION`` reply according to the following rules. + +* The major version in the reply must be the same as proposed. If the client + does not support the proposed major, it closes the connection. +* The minor version in the reply must be equal to or less than the minor + version proposed. +* The capability list must be a subset of those proposed. If the server + requires a capability the client did not include, it closes the connection. + +The protocol major version will only change when incompatible protocol changes +are made, such as changing the message format. The minor version may change +when compatible changes are made, such as adding new messages or capabilities, +Both the client and server must support all minor versions less than the +maximum minor version it supports. E.g., an implementation that supports +version 1.3 must also support 1.0 through 1.2. + +When making a change to this specification, the protocol version number must +be included in the form "added in version X.Y" + +Request +^^^^^^^ + +============== ====== ==== +Name Offset Size +============== ====== ==== +version major 0 2 +version minor 2 2 +version data 4 variable (including terminating NUL). Optional. +============== ====== ==== + +The version data is an optional UTF-8 encoded JSON byte array with the following +format: + ++--------------+--------+-----------------------------------+ +| Name | Type | Description | ++==============+========+===================================+ +| capabilities | object | Contains common capabilities that | +| | | the sender supports. Optional. | ++--------------+--------+-----------------------------------+ + +Capabilities: + ++--------------------+--------+------------------------------------------------+ +| Name | Type | Description | ++====================+========+================================================+ +| max_msg_fds | number | Maximum number of file descriptors that can be | +| | | received by the sender in one message. | +| | | Optional. If not specified then the receiver | +| | | must assume a value of ``1``. | ++--------------------+--------+------------------------------------------------+ +| max_data_xfer_size | number | Maximum ``count`` for data transfer messages; | +| | | see `Read and Write Operations`_. Optional, | +| | | with a default value of 1048576 bytes. | ++--------------------+--------+------------------------------------------------+ +| migration | object | Migration capability parameters. If missing | +| | | then migration is not supported by the sender. | ++--------------------+--------+------------------------------------------------+ + +The migration capability contains the following name/value pairs: + ++--------+--------+-----------------------------------------------+ +| Name | Type | Description | ++========+========+===============================================+ +| pgsize | number | Page size of dirty pages bitmap. The smallest | +| | | between the client and the server is used. | ++--------+--------+-----------------------------------------------+ + +Reply +^^^^^ + +The same message format is used in the server's reply with the semantics +described above. + +``VFIO_USER_DMA_MAP`` +--------------------- + +This command message is sent by the client to the server to inform it of the +memory regions the server can access. It must be sent before the server can +perform any DMA to the client. It is normally sent directly after the version +handshake is completed, but may also occur when memory is added to the client, +or if the client uses a vIOMMU. + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++-------------+--------+-------------+ +| Name | Offset | Size | ++=============+========+=============+ +| argsz | 0 | 4 | ++-------------+--------+-------------+ +| flags | 4 | 4 | ++-------------+--------+-------------+ +| | +-----+------------+ | +| | | Bit | Definition | | +| | +=====+============+ | +| | | 0 | readable | | +| | +-----+------------+ | +| | | 1 | writeable | | +| | +-----+------------+ | ++-------------+--------+-------------+ +| offset | 8 | 8 | ++-------------+--------+-------------+ +| address | 16 | 8 | ++-------------+--------+-------------+ +| size | 24 | 8 | ++-------------+--------+-------------+ + +* *argsz* is the size of the above structure. Note there is no reply payload, + so this field differs from other message types. +* *flags* contains the following region attributes: + + * *readable* indicates that the region can be read from. + + * *writeable* indicates that the region can be written to. + +* *offset* is the file offset of the region with respect to the associated file + descriptor, or zero if the region is not mappable +* *address* is the base DMA address of the region. +* *size* is the size of the region. + +This structure is 32 bytes in size, so the message size is 16 + 32 bytes. + +If the DMA region being added can be directly mapped by the server, a file +descriptor must be sent as part of the message meta-data. The region can be +mapped via the mmap() system call. On ``AF_UNIX`` sockets, the file descriptor +must be passed as ``SCM_RIGHTS`` type ancillary data. Otherwise, if the DMA +region cannot be directly mapped by the server, no file descriptor must be sent +as part of the message meta-data and the DMA region can be accessed by the +server using ``VFIO_USER_DMA_READ`` and ``VFIO_USER_DMA_WRITE`` messages, +explained in `Read and Write Operations`_. A command to map over an existing +region must be failed by the server with ``EEXIST`` set in error field in the +reply. + +Reply +^^^^^ + +There is no payload in the reply message. + +``VFIO_USER_DMA_UNMAP`` +----------------------- + +This command message is sent by the client to the server to inform it that a +DMA region, previously made available via a ``VFIO_USER_DMA_MAP`` command +message, is no longer available for DMA. It typically occurs when memory is +subtracted from the client or if the client uses a vIOMMU. The DMA region is +described by the following structure: + +Request +^^^^^^^ + +The request payload for this message is a structure of the following format: + ++--------------+--------+------------------------+ +| Name | Offset | Size | ++==============+========+========================+ +| argsz | 0 | 4 | ++--------------+--------+------------------------+ +| flags | 4 | 4 | ++--------------+--------+------------------------+ +| | +-----+-----------------------+ | +| | | Bit | Definition | | +| | +=====+=======================+ | +| | | 0 | get dirty page bitmap | | +| | +-----+-----------------------+ | ++--------------+--------+------------------------+ +| address | 8 | 8 | ++--------------+--------+------------------------+ +| size | 16 | 8 | ++--------------+--------+------------------------+ + +* *argsz* is the maximum size of the reply payload. +* *flags* contains the following DMA region attributes: + + * *get dirty page bitmap* indicates that a dirty page bitmap must be + populated before unmapping the DMA region. The client must provide a + `VFIO Bitmap`_ structure, explained below, immediately following this + entry. + +* *address* is the base DMA address of the DMA region. +* *size* is the size of the DMA region. + +The address and size of the DMA region being unmapped must match exactly a +previous mapping. The size of request message depends on whether or not the +*get dirty page bitmap* bit is set in Flags: + +* If not set, the size of the total request message is: 16 + 24. + +* If set, the size of the total request message is: 16 + 24 + 16. + +.. _VFIO Bitmap: + +VFIO Bitmap Format +"""""""""""""""""" + ++--------+--------+------+ +| Name | Offset | Size | ++========+========+======+ +| pgsize | 0 | 8 | ++--------+--------+------+ +| size | 8 | 8 | ++--------+--------+------+ + +* *pgsize* is the page size for the bitmap, in bytes. +* *size* is the size for the bitmap, in bytes, excluding the VFIO bitmap header. + +Reply +^^^^^ + +Upon receiving a ``VFIO_USER_DMA_UNMAP`` command, if the file descriptor is +mapped then the server must release all references to that DMA region before +replying, which potentially includes in-flight DMA transactions. + +The server responds with the original DMA entry in the request. If the +*get dirty page bitmap* bit is set in flags in the request, then +the server also includes the `VFIO Bitmap`_ structure sent in the request, +followed by the corresponding dirty page bitmap, where each bit represents +one page of size *pgsize* in `VFIO Bitmap`_ . + +The total size of the total reply message is: +16 + 24 + (16 + *size* in `VFIO Bitmap`_ if *get dirty page bitmap* is set). + +``VFIO_USER_DEVICE_GET_INFO`` +----------------------------- + +This command message is sent by the client to the server to query for basic +information about the device. + +Request +^^^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the maximum size of the reply payload +* all other fields must be zero. + +Reply +^^^^^ + ++-------------+--------+--------------------------+ +| Name | Offset | Size | ++=============+========+==========================+ +| argsz | 0 | 4 | ++-------------+--------+--------------------------+ +| flags | 4 | 4 | ++-------------+--------+--------------------------+ +| | +-----+-------------------------+ | +| | | Bit | Definition | | +| | +=====+=========================+ | +| | | 0 | VFIO_DEVICE_FLAGS_RESET | | +| | +-----+-------------------------+ | +| | | 1 | VFIO_DEVICE_FLAGS_PCI | | +| | +-----+-------------------------+ | ++-------------+--------+--------------------------+ +| num_regions | 8 | 4 | ++-------------+--------+--------------------------+ +| num_irqs | 12 | 4 | ++-------------+--------+--------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* contains the following device attributes. + + * ``VFIO_DEVICE_FLAGS_RESET`` indicates that the device supports the + ``VFIO_USER_DEVICE_RESET`` message. + * ``VFIO_DEVICE_FLAGS_PCI`` indicates that the device is a PCI device. + +* *num_regions* is the number of memory regions that the device exposes. +* *num_irqs* is the number of distinct interrupt types that the device supports. + +This version of the protocol only supports PCI devices. Additional devices may +be supported in future versions. + +``VFIO_USER_DEVICE_GET_REGION_INFO`` +------------------------------------ + +This command message is sent by the client to the server to query for +information about device regions. The VFIO region info structure is defined in +```` (``struct vfio_region_info``). + +Request +^^^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* all other fields must be zero. + +Reply +^^^^^ + ++------------+--------+------------------------------+ +| Name | Offset | Size | ++============+========+==============================+ +| argsz | 0 | 4 | ++------------+--------+------------------------------+ +| flags | 4 | 4 | ++------------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_REGION_INFO_FLAG_READ | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_REGION_INFO_FLAG_WRITE | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_REGION_INFO_FLAG_MMAP | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_REGION_INFO_FLAG_CAPS | | +| | +-----+-----------------------------+ | ++------------+--------+------------------------------+ ++------------+--------+------------------------------+ +| index | 8 | 4 | ++------------+--------+------------------------------+ +| cap_offset | 12 | 4 | ++------------+--------+------------------------------+ +| size | 16 | 8 | ++------------+--------+------------------------------+ +| offset | 24 | 8 | ++------------+--------+------------------------------+ + +* *argsz* is the size required for the full reply payload (region info structure + plus the size of any region capabilities) +* *flags* are attributes of the region: + + * ``VFIO_REGION_INFO_FLAG_READ`` allows client read access to the region. + * ``VFIO_REGION_INFO_FLAG_WRITE`` allows client write access to the region. + * ``VFIO_REGION_INFO_FLAG_MMAP`` specifies the client can mmap() the region. + When this flag is set, the reply will include a file descriptor in its + meta-data. On ``AF_UNIX`` sockets, the file descriptors will be passed as + ``SCM_RIGHTS`` type ancillary data. + * ``VFIO_REGION_INFO_FLAG_CAPS`` indicates additional capabilities found in the + reply. + +* *index* is the index of memory region being queried, it is the only field + that is required to be set in the command message. +* *cap_offset* describes where additional region capabilities can be found. + cap_offset is relative to the beginning of the VFIO region info structure. + The data structure it points is a VFIO cap header defined in + ````. +* *size* is the size of the region. +* *offset* is the offset that should be given to the mmap() system call for + regions with the MMAP attribute. It is also used as the base offset when + mapping a VFIO sparse mmap area, described below. + +VFIO region capabilities +"""""""""""""""""""""""" + +The VFIO region information can also include a capabilities list. This list is +similar to a PCI capability list - each entry has a common header that +identifies a capability and where the next capability in the list can be found. +The VFIO capability header format is defined in ```` (``struct +vfio_info_cap_header``). + +VFIO cap header format +"""""""""""""""""""""" + ++---------+--------+------+ +| Name | Offset | Size | ++=========+========+======+ +| id | 0 | 2 | ++---------+--------+------+ +| version | 2 | 2 | ++---------+--------+------+ +| next | 4 | 4 | ++---------+--------+------+ + +* *id* is the capability identity. +* *version* is a capability-specific version number. +* *next* specifies the offset of the next capability in the capability list. It + is relative to the beginning of the VFIO region info structure. + +VFIO sparse mmap cap header +""""""""""""""""""""""""""" + ++------------------+----------------------------------+ +| Name | Value | ++==================+==================================+ +| id | VFIO_REGION_INFO_CAP_SPARSE_MMAP | ++------------------+----------------------------------+ +| version | 0x1 | ++------------------+----------------------------------+ +| next | | ++------------------+----------------------------------+ +| sparse mmap info | VFIO region info sparse mmap | ++------------------+----------------------------------+ + +This capability is defined when only a subrange of the region supports +direct access by the client via mmap(). The VFIO sparse mmap area is defined in +```` (``struct vfio_region_sparse_mmap_area`` and ``struct +vfio_region_info_cap_sparse_mmap``). + +VFIO region info cap sparse mmap +"""""""""""""""""""""""""""""""" + ++----------+--------+------+ +| Name | Offset | Size | ++==========+========+======+ +| nr_areas | 0 | 4 | ++----------+--------+------+ +| reserved | 4 | 4 | ++----------+--------+------+ +| offset | 8 | 8 | ++----------+--------+------+ +| size | 16 | 9 | ++----------+--------+------+ +| ... | | | ++----------+--------+------+ + +* *nr_areas* is the number of sparse mmap areas in the region. +* *offset* and size describe a single area that can be mapped by the client. + There will be *nr_areas* pairs of offset and size. The offset will be added to + the base offset given in the ``VFIO_USER_DEVICE_GET_REGION_INFO`` to form the + offset argument of the subsequent mmap() call. + +The VFIO sparse mmap area is defined in ```` (``struct +vfio_region_info_cap_sparse_mmap``). + +VFIO region type cap header +""""""""""""""""""""""""""" + ++------------------+---------------------------+ +| Name | Value | ++==================+===========================+ +| id | VFIO_REGION_INFO_CAP_TYPE | ++------------------+---------------------------+ +| version | 0x1 | ++------------------+---------------------------+ +| next | | ++------------------+---------------------------+ +| region info type | VFIO region info type | ++------------------+---------------------------+ + +This capability is defined when a region is specific to the device. + +VFIO region info type cap +""""""""""""""""""""""""" + +The VFIO region info type is defined in ```` +(``struct vfio_region_info_cap_type``). + ++---------+--------+------+ +| Name | Offset | Size | ++=========+========+======+ +| type | 0 | 4 | ++---------+--------+------+ +| subtype | 4 | 4 | ++---------+--------+------+ + +The only device-specific region type and subtype supported by vfio-user is +``VFIO_REGION_TYPE_MIGRATION`` (3) and ``VFIO_REGION_SUBTYPE_MIGRATION`` (1). + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` +-------------------------------------- + +Clients can access regions via ``VFIO_USER_REGION_READ/WRITE`` or, if provided, by +``mmap()`` of a file descriptor provided by the server. + +``VFIO_USER_DEVICE_GET_REGION_IO_FDS`` provides an alternative access mechanism via +file descriptors. This is an optional feature intended for performance +improvements where an underlying sub-system (such as KVM) supports communication +across such file descriptors to the vfio-user server, without needing to +round-trip through the client. + +The server returns an array of sub-regions for the requested region. Each +sub-region describes a span (offset and size) of a region, along with the +requested file descriptor notification mechanism to use. Each sub-region in the +response message may choose to use a different method, as defined below. The +two mechanisms supported in this specification are ioeventfds and ioregionfds. + +The server in addition returns a file descriptor in the ancillary data; clients +are expected to configure each sub-region's file descriptor with the requested +notification method. For example, a client could configure KVM with the +requested ioeventfd via a ``KVM_IOEVENTFD`` ``ioctl()``. + +Request +^^^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ + +* *argsz* the maximum size of the reply payload +* *index* is the index of memory region being queried +* all other fields must be zero + +The client must set ``flags`` to zero and specify the region being queried in +the ``index``. + +Reply +^^^^^ + ++-------------+--------+------+ +| Name | Offset | Size | ++=============+========+======+ +| argsz | 0 | 4 | ++-------------+--------+------+ +| flags | 4 | 4 | ++-------------+--------+------+ +| index | 8 | 4 | ++-------------+--------+------+ +| count | 12 | 4 | ++-------------+--------+------+ +| sub-regions | 16 | ... | ++-------------+--------+------+ + +* *argsz* is the size of the region IO FD info structure plus the + total size of the sub-region array. Thus, each array entry "i" is at offset + i * ((argsz - 32) / count). Note that currently this is 40 bytes for both IO + FD types, but this is not to be relied on. As elsewhere, this indicates the + full reply payload size needed. +* *flags* must be zero +* *index* is the index of memory region being queried +* *count* is the number of sub-regions in the array +* *sub-regions* is the array of Sub-Region IO FD info structures + +The reply message will additionally include at least one file descriptor in the +ancillary data. Note that more than one sub-region may share the same file +descriptor. + +Note that it is the client's responsibility to verify the requested values (for +example, that the requested offset does not exceed the region's bounds). + +Each sub-region given in the response has one of two possible structures, +depending whether *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` or +``VFIO_USER_IO_FD_TYPE_IOREGIONFD``: + +Sub-Region IO FD info format (ioeventfd) +"""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| datamatch | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations +* *fd_index* is the index in the ancillary data of the FD to use for ioeventfd + notification; it may be shared. +* *type* is ``VFIO_USER_IO_FD_TYPE_IOEVENTFD`` +* *flags* is any of: + + * ``KVM_IOEVENTFD_FLAG_DATAMATCH`` + * ``KVM_IOEVENTFD_FLAG_PIO`` + * ``KVM_IOEVENTFD_FLAG_VIRTIO_CCW_NOTIFY`` (FIXME: makes sense?) + +* *datamatch* is the datamatch value if needed + +See https://www.kernel.org/doc/Documentation/virtual/kvm/api.txt, *4.59 +KVM_IOEVENTFD* for further context on the ioeventfd-specific fields. + +Sub-Region IO FD info format (ioregionfd) +""""""""""""""""""""""""""""""""""""""""" + ++-----------+--------+------+ +| Name | Offset | Size | ++===========+========+======+ +| offset | 0 | 8 | ++-----------+--------+------+ +| size | 8 | 8 | ++-----------+--------+------+ +| fd_index | 16 | 4 | ++-----------+--------+------+ +| type | 20 | 4 | ++-----------+--------+------+ +| flags | 24 | 4 | ++-----------+--------+------+ +| padding | 28 | 4 | ++-----------+--------+------+ +| user_data | 32 | 8 | ++-----------+--------+------+ + +* *offset* is the offset of the start of the sub-region within the region + requested ("physical address offset" for the region) +* *size* is the length of the sub-region. This may be zero if the access size is + not relevant, which may allow for optimizations; ``KVM_IOREGION_POSTED_WRITES`` + must be set in *flags* in this case +* *fd_index* is the index in the ancillary data of the FD to use for ioregionfd + messages; it may be shared +* *type* is ``VFIO_USER_IO_FD_TYPE_IOREGIONFD`` +* *flags* is any of: + + * ``KVM_IOREGION_PIO`` + * ``KVM_IOREGION_POSTED_WRITES`` + +* *user_data* is an opaque value passed back to the server via a message on the + file descriptor + +For further information on the ioregionfd-specific fields, see: +https://lore.kernel.org/kvm/cover.1613828726.git.eafanasova@gmail.com/ + +(FIXME: update with final API docs.) + +``VFIO_USER_DEVICE_GET_IRQ_INFO`` +--------------------------------- + +This command message is sent by the client to the server to query for +information about device interrupt types. The VFIO IRQ info structure is +defined in ```` (``struct vfio_irq_info``). + +Request +^^^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the maximum size of the reply payload (16 bytes today) +* index is the index of IRQ type being queried (e.g. ``VFIO_PCI_MSIX_IRQ_INDEX``) +* all other fields must be zero + +Reply +^^^^^ + ++-------+--------+---------------------------+ +| Name | Offset | Size | ++=======+========+===========================+ +| argsz | 0 | 4 | ++-------+--------+---------------------------+ +| flags | 4 | 4 | ++-------+--------+---------------------------+ +| | +-----+--------------------------+ | +| | | Bit | Definition | | +| | +=====+==========================+ | +| | | 0 | VFIO_IRQ_INFO_EVENTFD | | +| | +-----+--------------------------+ | +| | | 1 | VFIO_IRQ_INFO_MASKABLE | | +| | +-----+--------------------------+ | +| | | 2 | VFIO_IRQ_INFO_AUTOMASKED | | +| | +-----+--------------------------+ | +| | | 3 | VFIO_IRQ_INFO_NORESIZE | | +| | +-----+--------------------------+ | ++-------+--------+---------------------------+ +| index | 8 | 4 | ++-------+--------+---------------------------+ +| count | 12 | 4 | ++-------+--------+---------------------------+ + +* *argsz* is the size required for the full reply payload (16 bytes today) +* *flags* defines IRQ attributes: + + * ``VFIO_IRQ_INFO_EVENTFD`` indicates the IRQ type can support server eventfd + signalling. + * ``VFIO_IRQ_INFO_MASKABLE`` indicates that the IRQ type supports the ``MASK`` + and ``UNMASK`` actions in a ``VFIO_USER_DEVICE_SET_IRQS`` message. + * ``VFIO_IRQ_INFO_AUTOMASKED`` indicates the IRQ type masks itself after being + triggered, and the client must send an ``UNMASK`` action to receive new + interrupts. + * ``VFIO_IRQ_INFO_NORESIZE`` indicates ``VFIO_USER_SET_IRQS`` operations setup + interrupts as a set, and new sub-indexes cannot be enabled without disabling + the entire type. +* index is the index of IRQ type being queried +* count describes the number of interrupts of the queried type. + +``VFIO_USER_DEVICE_SET_IRQS`` +----------------------------- + +This command message is sent by the client to the server to set actions for +device interrupt types. The VFIO IRQ set structure is defined in +```` (``struct vfio_irq_set``). + +Request +^^^^^^^ + ++-------+--------+------------------------------+ +| Name | Offset | Size | ++=======+========+==============================+ +| argsz | 0 | 4 | ++-------+--------+------------------------------+ +| flags | 4 | 4 | ++-------+--------+------------------------------+ +| | +-----+-----------------------------+ | +| | | Bit | Definition | | +| | +=====+=============================+ | +| | | 0 | VFIO_IRQ_SET_DATA_NONE | | +| | +-----+-----------------------------+ | +| | | 1 | VFIO_IRQ_SET_DATA_BOOL | | +| | +-----+-----------------------------+ | +| | | 2 | VFIO_IRQ_SET_DATA_EVENTFD | | +| | +-----+-----------------------------+ | +| | | 3 | VFIO_IRQ_SET_ACTION_MASK | | +| | +-----+-----------------------------+ | +| | | 4 | VFIO_IRQ_SET_ACTION_UNMASK | | +| | +-----+-----------------------------+ | +| | | 5 | VFIO_IRQ_SET_ACTION_TRIGGER | | +| | +-----+-----------------------------+ | ++-------+--------+------------------------------+ +| index | 8 | 4 | ++-------+--------+------------------------------+ +| start | 12 | 4 | ++-------+--------+------------------------------+ +| count | 16 | 4 | ++-------+--------+------------------------------+ +| data | 20 | variable | ++-------+--------+------------------------------+ + +* *argsz* is the size of the VFIO IRQ set request payload, including any *data* + field. Note there is no reply payload, so this field differs from other + message types. +* *flags* defines the action performed on the interrupt range. The ``DATA`` + flags describe the data field sent in the message; the ``ACTION`` flags + describe the action to be performed. The flags are mutually exclusive for + both sets. + + * ``VFIO_IRQ_SET_DATA_NONE`` indicates there is no data field in the command. + The action is performed unconditionally. + * ``VFIO_IRQ_SET_DATA_BOOL`` indicates the data field is an array of boolean + bytes. The action is performed if the corresponding boolean is true. + * ``VFIO_IRQ_SET_DATA_EVENTFD`` indicates an array of event file descriptors + was sent in the message meta-data. These descriptors will be signalled when + the action defined by the action flags occurs. In ``AF_UNIX`` sockets, the + descriptors are sent as ``SCM_RIGHTS`` type ancillary data. + If no file descriptors are provided, this de-assigns the specified + previously configured interrupts. + * ``VFIO_IRQ_SET_ACTION_MASK`` indicates a masking event. It can be used with + ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to mask an interrupt, + or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the guest masks + the interrupt. + * ``VFIO_IRQ_SET_ACTION_UNMASK`` indicates an unmasking event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to unmask an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + guest unmasks the interrupt. + * ``VFIO_IRQ_SET_ACTION_TRIGGER`` indicates a triggering event. It can be used + with ``VFIO_IRQ_SET_DATA_BOOL`` or ``VFIO_IRQ_SET_DATA_NONE`` to trigger an + interrupt, or with ``VFIO_IRQ_SET_DATA_EVENTFD`` to generate an event when the + server triggers the interrupt. + +* *index* is the index of IRQ type being setup. +* *start* is the start of the sub-index being set. +* *count* describes the number of sub-indexes being set. As a special case, a + count (and start) of 0, with data flags of ``VFIO_IRQ_SET_DATA_NONE`` disables + all interrupts of the index. +* *data* is an optional field included when the + ``VFIO_IRQ_SET_DATA_BOOL`` flag is present. It contains an array of booleans + that specify whether the action is to be performed on the corresponding + index. It's used when the action is only performed on a subset of the range + specified. + +Not all interrupt types support every combination of data and action flags. +The client must know the capabilities of the device and IRQ index before it +sends a ``VFIO_USER_DEVICE_SET_IRQ`` message. + +In typical operation, a specific IRQ may operate as follows: + +1. The client sends a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_TRIGGER)`` along + with an eventfd. This associates the IRQ with a particular eventfd on the + server side. + +#. The client may send a ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_EVENTFD|VFIO_IRQ_SET_ACTION_MASK/UNMASK)`` along + with another eventfd. This associates the given eventfd with the + mask/unmask state on the server side. + +#. The server may trigger the IRQ by writing 1 to the eventfd. + +#. The server may mask/unmask an IRQ which will write 1 to the corresponding + mask/unmask eventfd, if there is one. + +5. A client may trigger a device IRQ itself, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_TRIGGER)``. + +6. A client may mask or unmask the IRQ, by sending a + ``VFIO_USER_DEVICE_SET_IRQ`` message with + ``flags=(VFIO_IRQ_SET_DATA_NONE/BOOL|VFIO_IRQ_SET_ACTION_MASK/UNMASK)``. + +Reply +^^^^^ + +There is no payload in the reply. + +.. _Read and Write Operations: + +Note that all of these operations must be supported by the client and/or server, +even if the corresponding memory or device region has been shared as mappable. + +The ``count`` field must not exceed the value of ``max_data_xfer_size`` of the +peer, for both reads and writes. + +``VFIO_USER_REGION_READ`` +------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via ``mmap()`` of the underlying file descriptor. In this case, a client can +read from a device region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. +* *data* is the data that was read from the device region. + +``VFIO_USER_REGION_WRITE`` +-------------------------- + +If a device region is not mappable, it's not directly accessible by the client +via mmap() of the underlying fd. In this case, a client can write to a device +region with this message. + +Request +^^^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ +| data | 16 | variable | ++--------+--------+----------+ + +* *offset* into the region being accessed. +* *region* is the index of the region being accessed. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++--------+--------+----------+ +| Name | Offset | Size | ++========+========+==========+ +| offset | 0 | 8 | ++--------+--------+----------+ +| region | 8 | 4 | ++--------+--------+----------+ +| count | 12 | 4 | ++--------+--------+----------+ + +* *offset* into the region accessed. +* *region* is the index of the region accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DMA_READ`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +read from guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. +* *data* is the data read. + +``VFIO_USER_DMA_WRITE`` +----------------------- + +If the client has not shared mappable memory, the server can use this message to +write to guest memory. + +Request +^^^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 8 | ++---------+--------+----------+ +| data | 16 | variable | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. This address must have + been previously exported to the server with a ``VFIO_USER_DMA_MAP`` message. +* *count* is the size of the data to be transferred. +* *data* is the data to write + +Reply +^^^^^ + ++---------+--------+----------+ +| Name | Offset | Size | ++=========+========+==========+ +| address | 0 | 8 | ++---------+--------+----------+ +| count | 8 | 4 | ++---------+--------+----------+ + +* *address* is the client DMA memory address being accessed. +* *count* is the size of the data transferred. + +``VFIO_USER_DEVICE_RESET`` +-------------------------- + +This command message is sent from the client to the server to reset the device. +Neither the request or reply have a payload. + +``VFIO_USER_DIRTY_PAGES`` +------------------------- + +This command is analogous to ``VFIO_IOMMU_DIRTY_PAGES``. It is sent by the client +to the server in order to control logging of dirty pages, usually during a live +migration. + +Dirty page tracking is optional for server implementation; clients should not +rely on it. + +Request +^^^^^^^ + ++-------+--------+-----------------------------------------+ +| Name | Offset | Size | ++=======+========+=========================================+ +| argsz | 0 | 4 | ++-------+--------+-----------------------------------------+ +| flags | 4 | 4 | ++-------+--------+-----------------------------------------+ +| | +-----+----------------------------------------+ | +| | | Bit | Definition | | +| | +=====+========================================+ | +| | | 0 | VFIO_IOMMU_DIRTY_PAGES_FLAG_START | | +| | +-----+----------------------------------------+ | +| | | 1 | VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP | | +| | +-----+----------------------------------------+ | +| | | 2 | VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP | | +| | +-----+----------------------------------------+ | ++-------+--------+-----------------------------------------+ + +* *argsz* is the size of the VFIO dirty bitmap info structure for + ``START/STOP``; and for ``GET_BITMAP``, the maximum size of the reply payload + +* *flags* defines the action to be performed by the server: + + * ``VFIO_IOMMU_DIRTY_PAGES_FLAG_START`` instructs the server to start logging + pages it dirties. Logging continues until explicitly disabled by + ``VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP``. + + * ``VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP`` instructs the server to stop logging + dirty pages. + + * ``VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP`` requests the server to return + the dirty bitmap for a specific IOVA range. The IOVA range is specified by + a "VFIO Bitmap Range" structure, which must immediately follow this + "VFIO Dirty Pages" structure. See `VFIO Bitmap Range Format`_. + This operation is only valid if logging of dirty pages has been previously + started. + + These flags are mutually exclusive with each other. + +This part of the request is analogous to VFIO's ``struct +vfio_iommu_type1_dirty_bitmap``. + +.. _VFIO Bitmap Range Format: + +VFIO Bitmap Range Format +"""""""""""""""""""""""" + ++--------+--------+------+ +| Name | Offset | Size | ++========+========+======+ +| iova | 0 | 8 | ++--------+--------+------+ +| size | 8 | 8 | ++--------+--------+------+ +| bitmap | 16 | 24 | ++--------+--------+------+ + +* *iova* is the IOVA offset + +* *size* is the size of the IOVA region + +* *bitmap* is the VFIO Bitmap explained in `VFIO Bitmap`_. + +This part of the request is analogous to VFIO's ``struct +vfio_iommu_type1_dirty_bitmap_get``. + +Reply +^^^^^ + +For ``VFIO_IOMMU_DIRTY_PAGES_FLAG_START`` or +``VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP``, there is no reply payload. + +For ``VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP``, the reply payload is as follows: + ++--------------+--------+-----------------------------------------+ +| Name | Offset | Size | ++==============+========+=========================================+ +| argsz | 0 | 4 | ++--------------+--------+-----------------------------------------+ +| flags | 4 | 4 | ++--------------+--------+-----------------------------------------+ +| | +-----+----------------------------------------+ | +| | | Bit | Definition | | +| | +=====+========================================+ | +| | | 2 | VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP | | +| | +-----+----------------------------------------+ | ++--------------+--------+-----------------------------------------+ +| bitmap range | 8 | 40 | ++--------------+--------+-----------------------------------------+ +| bitmap | 48 | variable | ++--------------+--------+-----------------------------------------+ + +* *argsz* is the size required for the full reply payload (dirty pages structure + + bitmap range structure + actual bitmap) +* *flags* is ``VFIO_IOMMU_DIRTY_PAGES_FLAG_GET_BITMAP`` +* *bitmap range* is the same bitmap range struct provided in the request, as + defined in `VFIO Bitmap Range Format`_. +* *bitmap* is the actual dirty pages bitmap corresponding to the range request + +VFIO Device Migration Info +-------------------------- + +A device may contain a migration region (of type +``VFIO_REGION_TYPE_MIGRATION``). The beginning of the region must contain +``struct vfio_device_migration_info``, defined in ````. This +subregion is accessed like any other part of a standard vfio-user region +using ``VFIO_USER_REGION_READ``/``VFIO_USER_REGION_WRITE``. + ++---------------+--------+-----------------------------+ +| Name | Offset | Size | ++===============+========+=============================+ +| device_state | 0 | 4 | ++---------------+--------+-----------------------------+ +| | +-----+----------------------------+ | +| | | Bit | Definition | | +| | +=====+============================+ | +| | | 0 | VFIO_DEVICE_STATE_RUNNING | | +| | +-----+----------------------------+ | +| | | 1 | VFIO_DEVICE_STATE_SAVING | | +| | +-----+----------------------------+ | +| | | 2 | VFIO_DEVICE_STATE_RESUMING | | +| | +-----+----------------------------+ | ++---------------+--------+-----------------------------+ +| reserved | 4 | 4 | ++---------------+--------+-----------------------------+ +| pending_bytes | 8 | 8 | ++---------------+--------+-----------------------------+ +| data_offset | 16 | 8 | ++---------------+--------+-----------------------------+ +| data_size | 24 | 8 | ++---------------+--------+-----------------------------+ + +* *device_state* defines the state of the device: + + The client initiates device state transition by writing the intended state. + The server must respond only after it has successfully transitioned to the new + state. If an error occurs then the server must respond to the + ``VFIO_USER_REGION_WRITE`` operation with the Error field set accordingly and + must remain at the previous state, or in case of internal error it must + transition to the error state, defined as + ``VFIO_DEVICE_STATE_RESUMING | VFIO_DEVICE_STATE_SAVING``. The client must + re-read the device state in order to determine it afresh. + + The following device states are defined: + + +-----------+---------+----------+-----------------------------------+ + | _RESUMING | _SAVING | _RUNNING | Description | + +===========+=========+==========+===================================+ + | 0 | 0 | 0 | Device is stopped. | + +-----------+---------+----------+-----------------------------------+ + | 0 | 0 | 1 | Device is running, default state. | + +-----------+---------+----------+-----------------------------------+ + | 0 | 1 | 0 | Stop-and-copy state | + +-----------+---------+----------+-----------------------------------+ + | 0 | 1 | 1 | Pre-copy state | + +-----------+---------+----------+-----------------------------------+ + | 1 | 0 | 0 | Resuming | + +-----------+---------+----------+-----------------------------------+ + | 1 | 0 | 1 | Invalid state | + +-----------+---------+----------+-----------------------------------+ + | 1 | 1 | 0 | Error state | + +-----------+---------+----------+-----------------------------------+ + | 1 | 1 | 1 | Invalid state | + +-----------+---------+----------+-----------------------------------+ + + Valid state transitions are shown in the following table: + + +-------------------------+---------+---------+---------------+----------+----------+ + | |darr| From / To |rarr| | Stopped | Running | Stop-and-copy | Pre-copy | Resuming | + +=========================+=========+=========+===============+==========+==========+ + | Stopped | \- | 1 | 0 | 0 | 0 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Running | 1 | \- | 1 | 1 | 1 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Stop-and-copy | 1 | 1 | \- | 0 | 0 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Pre-copy | 0 | 0 | 1 | \- | 0 | + +-------------------------+---------+---------+---------------+----------+----------+ + | Resuming | 0 | 1 | 0 | 0 | \- | + +-------------------------+---------+---------+---------------+----------+----------+ + + A device is migrated to the destination as follows: + + * The source client transitions the device state from the running state to + the pre-copy state. This transition is optional for the client but must be + supported by the server. The source server starts sending device state data + to the source client through the migration region while the device is + running. + + * The source client transitions the device state from the running state or the + pre-copy state to the stop-and-copy state. The source server stops the + device, saves device state and sends it to the source client through the + migration region. + + The source client is responsible for sending the migration data to the + destination client. + + A device is resumed on the destination as follows: + + * The destination client transitions the device state from the running state + to the resuming state. The destination server uses the device state data + received through the migration region to resume the device. + + * The destination client provides saved device state to the destination + server and then transitions the device to back to the running state. + +* *reserved* This field is reserved and any access to it must be ignored by the + server. + +* *pending_bytes* Remaining bytes to be migrated by the server. This field is + read only. + +* *data_offset* Offset in the migration region where the client must: + + * read from, during the pre-copy or stop-and-copy state, or + + * write to, during the resuming state. + + This field is read only. + +* *data_size* Contains the size, in bytes, of the amount of data copied to: + + * the source migration region by the source server during the pre-copy or + stop-and copy state, or + + * the destination migration region by the destination client during the + resuming state. + +Device-specific data must be stored at any position after +``struct vfio_device_migration_info``. Note that the migration region can be +memory mappable, even partially. In practise, only the migration data portion +can be memory mapped. + +The client processes device state data during the pre-copy and the +stop-and-copy state in the following iterative manner: + + 1. The client reads ``pending_bytes`` to mark a new iteration. Repeated reads + of this field is an idempotent operation. If there are no migration data + to be consumed then the next step depends on the current device state: + + * pre-copy: the client must try again. + + * stop-and-copy: this procedure can end and the device can now start + resuming on the destination. + + 2. The client reads ``data_offset``; at this point the server must make + available a portion of migration data at this offset to be read by the + client, which must happen *before* completing the read operation. The + amount of data to be read must be stored in the ``data_size`` field, which + the client reads next. + + 3. The client reads ``data_size`` to determine the amount of migration data + available. + + 4. The client reads and processes the migration data. + + 5. Go to step 1. + +Note that the client can transition the device from the pre-copy state to the +stop-and-copy state at any time; ``pending_bytes`` does not need to become zero. + +The client initializes the device state on the destination by setting the +device state in the resuming state and writing the migration data to the +destination migration region at ``data_offset`` offset. The client can write the +source migration data in an iterative manner and the server must consume this +data before completing each write operation, updating the ``data_offset`` field. +The server must apply the source migration data on the device resume state. The +client must write data on the same order and transaction size as read. + +If an error occurs then the server must fail the read or write operation. It is +an implementation detail of the client how to handle errors. + +Appendices +========== + +Unused VFIO ``ioctl()`` commands +-------------------------------- + +The following VFIO commands do not have an equivalent vfio-user command: + +* ``VFIO_GET_API_VERSION`` +* ``VFIO_CHECK_EXTENSION`` +* ``VFIO_SET_IOMMU`` +* ``VFIO_GROUP_GET_STATUS`` +* ``VFIO_GROUP_SET_CONTAINER`` +* ``VFIO_GROUP_UNSET_CONTAINER`` +* ``VFIO_GROUP_GET_DEVICE_FD`` +* ``VFIO_IOMMU_GET_INFO`` + +However, once support for live migration for VFIO devices is finalized some +of the above commands may have to be handled by the client in their +corresponding vfio-user form. This will be addressed in a future protocol +version. + +VFIO groups and containers +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The current VFIO implementation includes group and container idioms that +describe how a device relates to the host IOMMU. In the vfio-user +implementation, the IOMMU is implemented in SW by the client, and is not +visible to the server. The simplest idea would be that the client put each +device into its own group and container. + +Backend Program Conventions +--------------------------- + +vfio-user backend program conventions are based on the vhost-user ones. + +* The backend program must not daemonize itself. +* No assumptions must be made as to what access the backend program has on the + system. +* File descriptors 0, 1 and 2 must exist, must have regular + stdin/stdout/stderr semantics, and can be redirected. +* The backend program must honor the SIGTERM signal. +* The backend program must accept the following commands line options: + + * ``--socket-path=PATH``: path to UNIX domain socket, + * ``--fd=FDNUM``: file descriptor for UNIX domain socket, incompatible with + ``--socket-path`` +* The backend program must be accompanied with a JSON file stored under + ``/usr/share/vfio-user``. + +TODO add schema similar to docs/interop/vhost-user.json. diff --git a/MAINTAINERS b/MAINTAINERS index 694973ed23..d838b9e3f2 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1879,6 +1879,12 @@ F: hw/vfio/ap.c F: docs/system/s390x/vfio-ap.rst L: qemu-s390x@nongnu.org +vfio-user +M: John G Johnson +M: Thanos Makatos +S: Supported +F: docs/devel/vfio-user.rst + vhost M: Michael S. Tsirkin S: Supported From patchwork Mon Aug 16 16:42:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1020C3F6A3 for ; Mon, 16 Aug 2021 16:52:09 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 71E09604DC for ; Mon, 16 Aug 2021 16:52:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 71E09604DC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:47284 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfqK-0008SI-Jt for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:52:08 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37134) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhe-0000oX-HM for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:10 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:35300) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhY-0007vK-PS for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:10 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaE6R000648; Mon, 16 Aug 2021 16:43:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=KcjsSD+Wq2I/FtUzkAJgb2ONemFI/sZTtoqwRT3vav4=; b=dPGpK6XIokMzcf0R9LwRAKU13q+OIMxwgtmdxWmWunZcVbKgyf2FuSvvSJU/H8/g0QHj esXX2mrBsESo5gN9JpJQ9zMAoKfen7ka4zsR5bZT8efd9kOsVWyAQYXT4jPBFoD+Se2U HQCkk1SAQp8aA0w8P7eF1RuUmvLkiVhM3Pnb23RxE3Zlf0gYLIM7grOEt00CEcxh9KsI Tro/Ms4GayIgBYfXe6rS4glbAIMCd69Pb0OQxBBEPVsr6yIv/TJEYSEXt6IAidlsUag4 twv+Qv7nWmCFm6ymVauaUR2k2dcg7mgbdaXYwKG47o4CjwY0YxTX6Ual+DQdw9bKtBw8 YQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=KcjsSD+Wq2I/FtUzkAJgb2ONemFI/sZTtoqwRT3vav4=; b=nsZaWmXHTLpzRIwBQxqofngbdtBEZGLpnq6O8erG2q8WS/qE3gZoDjaorl0+w5k9vUgm Kx/cvwCTeXih/pSuScKTBvougMGTrlo3b/6IB5qsOJ72Ns3s23ZdZIboc88rQ1nIDgdk KDvaZ7dHzJxJStS+sEhAx+47bjQqZRY4p4VSun1NyYX33zBrwSaEJ1KkP/OIZqkD8Ny6 koc+fltVD4/HiQtcSLDIvvktyN71hXJUQDduNVrqmRTLkgCvdc8gT9FRXQR/hab7nb1z tQVRTTCaTH08Ejwye7GcuU4d0sxqUCk2GEa2HeDIrnXLplz5TYhtr2CMNRBdjE1EHxfh Fg== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by mx0b-00069f02.pphosted.com with ESMTP id 3af1q9anur-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:00 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeuOL025726; Mon, 16 Aug 2021 16:42:59 GMT Received: from nam10-mw2-obe.outbound.protection.outlook.com (mail-mw2nam10lp2102.outbound.protection.outlook.com [104.47.55.102]) by userp3030.oracle.com with ESMTP id 3ae2xxgrw2-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:42:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=V9r9krC7R9rJV/fzdW7KWs89mH0Te+T/GEoP3DL2VYEQnsBDqhNhnPHl8k4YQXdURRM2CjmYs0ie8QIb7oABKG85DQhiVOd278/bXOt7vBRzGG5y+0jLRXI07gA7nmp0t+n09hdH9RvZ2oE6ngNtngGLdNyX7BZhDwkGKfXmDcceylkd2pCTgwNzZ8FRn3caoV3aPKP/lyzAtqW33rsxS7D/UqVxRjvCIxHcSwQKCe9Gm38LAJN/FdNHnr7SKFcHCFJp/ysFvbbaUq/ysupv8c7kkxWs+BzQH/L/2WQp5LUiezLpqCctdTFTuxlV7RQ3ld2QPCaWi+WvjWdJB06dGQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KcjsSD+Wq2I/FtUzkAJgb2ONemFI/sZTtoqwRT3vav4=; b=g5qHaViMWXmVjm8vzu9nEYhtDtiTDgqNcKTvVnsmoKYzGu/j6angUTE/L7uKU9Uh1KCEKphHvXhs0eTG4YYNszErbuLOZSzPbHVORg2c1xztu9dvmDBDhSEghwFlSmeuMBNRqgkX3OiG77p5vkL+5vIltgR1nOO3gTtgYhXu6wCAp9HCbs87pPsA4E9yd8H6Ipb522ff11A2nYZqpeAj/HlAIxBb2++OfmuPk3uKHT56J8bvrmhYLXDb20KZDmaWwG63lWLq1z6kH3xKZBz/sFwSb5Rix0DB/q0H8Kat43IfaIY7JoMvO8NN1lswzn3jfJjdscuQFMUW957xDWWU+w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=KcjsSD+Wq2I/FtUzkAJgb2ONemFI/sZTtoqwRT3vav4=; b=Oc5/HdCVzsA15t/jtN52XapI4IDSZRkWl3la7TbMiVq/rU1HNNbuC3wGxmbT1lFTIE6YX+GMx5QdhziyTo0pTv7J5Ix4VBvGtDW3tsSVQo8a0iqWh5lG8RmQSiPdLkTI8o1VQuBdnmAx7lzv7PI2fl+PIdG38yvNYlRelPeV42g= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BYAPR10MB2565.namprd10.prod.outlook.com (2603:10b6:a02:b2::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Mon, 16 Aug 2021 16:42:56 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:56 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 02/16] vfio-user: add VFIO base abstract class Date: Mon, 16 Aug 2021 09:42:35 -0700 Message-Id: <3610c39c2e12132e01a23752d161e7209be12f38.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:56 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 55adc678-9e03-4070-29e9-08d960d4e63b X-MS-TrafficTypeDiagnostic: BYAPR10MB2565: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2582; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KlpfkIvGMDuobs4JPzmk0BI0gflBMBXcMl+8qGDzXMr1ybiw/yYCcLlVCnmSdg6SWUdZWubD0Qcnl2ct4qX6wU5PpzMjZ8RK3xSom+zXkd3RT9bVRqyAIogYtpodS15wLgkxmp6DsGpAv8Ox8SF1ahuMEh2T7qvnvtMFdpOEUxQV3/7bbykOGzmeKZWf5PT0K0vJ9CYPYPBkF9dOgVCevj1RZReHTBy8Fn3CUmcSkUyr/bcBnh8L2uQMreuUEA822A4BVlHhDygu4/rN6d/ESKfBgZVYlSjmJAAjRug77zr2GuKJnx+78V01IfM+ivBQUF7wKa8KwH2Dt6cwuvpmWs2GW7vFQf7wfE/o6uWZUUs62AeWNNVG5mmgWRZUKHCWk3XIzjhyuE1tmqeYQ/jB5lRSA68NdniUMUr7ALd+N6hF1/FLPZJCh8aBo23SgrynVvpO2KAEwrsYHq4Sgm9uj58JOpzeSEF/lj6Ze5NN5uG04vc/+ETpLDk7JyVLfiGPasacUYgi0xmApvUxfZ5HTbjOHEKqIuNvGTnL9IM32PLBoif+yqSzjwYwk0LQXA/xoWwE6nguEULw4n6gzmYX/XdsmH423c3ZWaYkXa4MOJWJZv05zKNzFHgSPfQIE7UmZToueNtJvJoWvVablhGuUA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(6486002)(316002)(38100700002)(44832011)(2616005)(6512007)(2906002)(6916009)(52116002)(5660300002)(508600001)(6506007)(66946007)(4326008)(66476007)(66556008)(6666004)(186003)(86362001)(8676002)(8936002)(36756003)(83380400001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xNhU9V3yqzyrpVb1nWSqZ6jDxw5WuQWkdLIOZN85g0PfdMB1CBprqKo3WaT28MDZGOS6kDSOLaq94H6Zry5VMyT0suVgnJgbONZo5e3NYz54aHwnLyjiQ0zT2+VVTySbZhiJsNAZFXF1r2ufpGLpejhNY/mWSY0NfpRTFoN/+pLq8GFr4m5ojFpOfI1W7SdGRX1RYKeV6oZ48/9H2bKn8oC+ajJaD98AdwDi2PTqaR2DWHMtilW6gwVyswd9rPWojYPoX1DwPudduO2k1ZFL/KKaqvLEAHZWZfvkxDt/zkpBwTJ/LPKk3R1NyfYvVkUYCUUYgYwplBV1ANtOs6wE0nqTT3Z6GVuQlTFhyue7uZdxMFomoGCJ2bhpQy3bJcUVG7lLAfCF5EmLg+aBV3d42/uwr2yJ31b4Exmq3qktM7JVFUJ+Dw2P7NTEWoxGD6fz/8akSYbRkUxTbTNq02XP0l4CZd+lspQrHxLtjZZpgPH1erus6hKZN0b5TnStZbNdoNHnFi7hA6OqHY1OWgxDxPfn225sCGj/mv3eXIWFvfJgNEnkKwSd1XoP2a1Kf/vS788wQyf1vdn56VL+QfeCnsOgFyjLilXCjxZMx9OBiyAszwY1r+ww44DJfQ9DVYmR0wMhLJgwjuqNjdis9TTBA99ceSyxHeSRYyDZ04utkOO0yih7QessphwmE2ixohCW59ZBv5u6VrXDVTA/inn/Idc/GiqYY8zM6tZ216QQh04uPJVmLSb2xr4NTgdeBF1gEGgf2ZBS2Et720sySWv1kkF2WOT0eWt/GroQ/eHWVXdTy6OtSwyBH+nTR7xiPSdtWQ7298XVsHOUlQ+78AT9rb2EcVMlS8k66L8HBSSzPTB06/YNmUuak0xeComHs/fLC641T0F9vv7feKEZzqFmmJCqbQ9PVWto9F5hv+yxO/eCBh8DCgzuB4uH70vC/aOrz7X9OZy8LQ70WhDIr9zXkibsBUCzr2jvd2aPqaU9e5laM61/Io8400WUXcuPCnx0CAvMHVw2tO1ovEAKi3Bc7YPF54D6Q2ey4z2AgKHhXnRt6x5LRXcRpLBiQi5o+geXlSatEOSezT/tfDLylZX7XE2HVvJWY4USSBB9ne/VCsof0wxlZfAz9KJLadnJ/8yI1hiQdt9oWm3U5vTh5td6N9WNQ9sAyhjv4H1XDzubT4rLg4sNVgUENND24oQOVzJbZOml3BdGT62HeDzPDE1SJGzNmox7RJgwAw9bM3AuuJFy8bmX+CHfGGN72Lif+0VurwAWUgS/P0rLa4Pg29s7/rr2uJkhkY3rEbNP6O77CgPqII34mcxrpT2ffUUMcZyLDIsm8dtoIuwQ1WWYvv+eyKQxVfnn7CaXTatLnhqTmmwi3E8YBlRYckHLiIbI7xZr X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 55adc678-9e03-4070-29e9-08d960d4e63b X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:56.2965 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ijze4Or8Mxo1wwhhnLr4rrCUWqSce8kFcahQ4dJ5Xm/rCjwx1+dbGFC5gr9d75KQVVpfQ1fBKv7DTXmRkT8w5r4nqC66GZi592CCg2QGsPw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB2565 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 spamscore=0 adultscore=0 suspectscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: DKMMYzoQrwgKi5jFKEfBfJv1BKRz83l3 X-Proofpoint-ORIG-GUID: DKMMYzoQrwgKi5jFKEfBfJv1BKRz83l3 Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Add an abstract base class both the kernel driver and user socket implementations can use to share code. Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 16 +++++++++++-- hw/vfio/pci.c | 63 ++++++++++++++++++++++++++++++++------------------- 2 files changed, 54 insertions(+), 25 deletions(-) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index 64777516d1..bbc78aaeb3 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -114,8 +114,13 @@ typedef struct VFIOMSIXInfo { unsigned long *pending; } VFIOMSIXInfo; -#define TYPE_VFIO_PCI "vfio-pci" -OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI) +/* + * TYPE_VFIO_PCI_BASE is an abstract type used to share code + * between VFIO implementations that use a kernel driver + * with those that use user sockets. + */ +#define TYPE_VFIO_PCI_BASE "vfio-pci-base" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOPCIDevice, VFIO_PCI_BASE) struct VFIOPCIDevice { PCIDevice pdev; @@ -175,6 +180,13 @@ struct VFIOPCIDevice { Notifier irqchip_change_notifier; }; +#define TYPE_VFIO_PCI "vfio-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOKernPCIDevice, VFIO_PCI) + +struct VFIOKernPCIDevice { + VFIOPCIDevice device; +}; + /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index e1ea1d8a23..bea95efc33 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -231,7 +231,7 @@ static void vfio_intx_update(VFIOPCIDevice *vdev, PCIINTxRoute *route) static void vfio_intx_routing_notifier(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); PCIINTxRoute route; if (vdev->interrupt != VFIO_INT_INTx) { @@ -457,7 +457,7 @@ static void vfio_update_kvm_msi_virq(VFIOMSIVector *vector, MSIMessage msg, static int vfio_msix_vector_do_use(PCIDevice *pdev, unsigned int nr, MSIMessage *msg, IOHandler *handler) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector; int ret; @@ -542,7 +542,7 @@ static int vfio_msix_vector_use(PCIDevice *pdev, static void vfio_msix_vector_release(PCIDevice *pdev, unsigned int nr) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIOMSIVector *vector = &vdev->msi_vectors[nr]; trace_vfio_msix_vector_release(vdev->vbasedev.name, nr); @@ -1063,7 +1063,7 @@ static const MemoryRegionOps vfio_vga_ops = { */ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIORegion *region = &vdev->bars[bar].region; MemoryRegion *mmap_mr, *region_mr, *base_mr; PCIIORegion *r; @@ -1109,7 +1109,7 @@ static void vfio_sub_page_bar_update_mapping(PCIDevice *pdev, int bar) */ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t emu_bits = 0, emu_val = 0, phys_val = 0, val; memcpy(&emu_bits, vdev->emulated_config_bits + addr, len); @@ -1142,7 +1142,7 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) void vfio_pci_write_config(PCIDevice *pdev, uint32_t addr, uint32_t val, int len) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t val_le = cpu_to_le32(val); trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); @@ -2782,7 +2782,7 @@ static void vfio_unregister_req_notifier(VFIOPCIDevice *vdev) static void vfio_realize(PCIDevice *pdev, Error **errp) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev_iter; VFIOGroup *group; char *tmp, *subsys, group_path[PATH_MAX], *group_name; @@ -3105,7 +3105,7 @@ error: static void vfio_instance_finalize(Object *obj) { - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIOGroup *group = vdev->vbasedev.group; vfio_display_finalize(vdev); @@ -3125,7 +3125,7 @@ static void vfio_instance_finalize(Object *obj) static void vfio_exitfn(PCIDevice *pdev) { - VFIOPCIDevice *vdev = VFIO_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); vfio_unregister_req_notifier(vdev); vfio_unregister_err_notifier(vdev); @@ -3144,7 +3144,7 @@ static void vfio_exitfn(PCIDevice *pdev) static void vfio_pci_reset(DeviceState *dev) { - VFIOPCIDevice *vdev = VFIO_PCI(dev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); trace_vfio_pci_reset(vdev->vbasedev.name); @@ -3184,7 +3184,7 @@ post_reset: static void vfio_instance_init(Object *obj) { PCIDevice *pci_dev = PCI_DEVICE(obj); - VFIOPCIDevice *vdev = VFIO_PCI(obj); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); device_add_bootindex_property(obj, &vdev->bootindex, "bootindex", NULL, @@ -3253,28 +3253,24 @@ static Property vfio_pci_dev_properties[] = { DEFINE_PROP_END_OF_LIST(), }; -static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) +static void vfio_pci_base_dev_class_init(ObjectClass *klass, void *data) { DeviceClass *dc = DEVICE_CLASS(klass); PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); - dc->reset = vfio_pci_reset; - device_class_set_props(dc, vfio_pci_dev_properties); - dc->desc = "VFIO-based PCI device assignment"; + dc->desc = "VFIO PCI base device"; set_bit(DEVICE_CATEGORY_MISC, dc->categories); - pdc->realize = vfio_realize; pdc->exit = vfio_exitfn; pdc->config_read = vfio_pci_read_config; pdc->config_write = vfio_pci_write_config; } -static const TypeInfo vfio_pci_dev_info = { - .name = TYPE_VFIO_PCI, +static const TypeInfo vfio_pci_base_dev_info = { + .name = TYPE_VFIO_PCI_BASE, .parent = TYPE_PCI_DEVICE, - .instance_size = sizeof(VFIOPCIDevice), - .class_init = vfio_pci_dev_class_init, - .instance_init = vfio_instance_init, - .instance_finalize = vfio_instance_finalize, + .instance_size = 0, + .abstract = true, + .class_init = vfio_pci_base_dev_class_init, .interfaces = (InterfaceInfo[]) { { INTERFACE_PCIE_DEVICE }, { INTERFACE_CONVENTIONAL_PCI_DEVICE }, @@ -3282,6 +3278,26 @@ static const TypeInfo vfio_pci_dev_info = { }, }; +static void vfio_pci_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + dc->reset = vfio_pci_reset; + device_class_set_props(dc, vfio_pci_dev_properties); + dc->desc = "VFIO-based PCI device assignment"; + pdc->realize = vfio_realize; +} + +static const TypeInfo vfio_pci_dev_info = { + .name = TYPE_VFIO_PCI, + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOKernPCIDevice), + .class_init = vfio_pci_dev_class_init, + .instance_init = vfio_instance_init, + .instance_finalize = vfio_instance_finalize, +}; + static Property vfio_pci_dev_nohotplug_properties[] = { DEFINE_PROP_BOOL("ramfb", VFIOPCIDevice, enable_ramfb, false), DEFINE_PROP_END_OF_LIST(), @@ -3298,12 +3314,13 @@ static void vfio_pci_nohotplug_dev_class_init(ObjectClass *klass, void *data) static const TypeInfo vfio_pci_nohotplug_dev_info = { .name = TYPE_VFIO_PCI_NOHOTPLUG, .parent = TYPE_VFIO_PCI, - .instance_size = sizeof(VFIOPCIDevice), + .instance_size = sizeof(VFIOKernPCIDevice), .class_init = vfio_pci_nohotplug_dev_class_init, }; static void register_vfio_pci_dev_type(void) { + type_register_static(&vfio_pci_base_dev_info); type_register_static(&vfio_pci_dev_info); type_register_static(&vfio_pci_nohotplug_dev_info); } From patchwork Mon Aug 16 16:42:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F37D6C4338F for ; Mon, 16 Aug 2021 16:44:42 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5D96760F55 for ; Mon, 16 Aug 2021 16:44:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5D96760F55 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:55206 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfj7-0003LT-6E for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:44:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37010) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhZ-0000dk-Uy for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:06 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:32166) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhX-0007uU-FY for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:05 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaAhs021058; Mon, 16 Aug 2021 16:43:00 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=HU+WVZn20enFNMZWdVa5bfgLHRj1V4MI+wkyDyMte0Y=; b=yCNoOFoHUrEDmZmMKgVPVyRYB7bJQbrP9vPA1wimH5vaDAKglVD+XS3sllmkPbc4kQqL c0nnBx3u3NcS+HQeJ8qsQJLMn2tAHsAjOBUfKj+IEKO3RuuVXjoj1qLG3rVJJcNVg7jQ xbM6PXjtKiukIbRziJYxJmfq3VEpfZKovygCZLo9jTWRtPO3hzi9dj11wkzR2hB69VsW G6ByizbphwVVuY+/h1Ga47/cKs2gw79rOjmeReLVVwy1BwIdmn9uJyZOTUsF+PgeITjv qNJvkZ/pKiYEIkFR6RrxmOZoYFhEL/TX1RUrB3ntN8NVTdBdgmkGEeeQlnJso3NjH/sb Jg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=HU+WVZn20enFNMZWdVa5bfgLHRj1V4MI+wkyDyMte0Y=; b=bHAesTW/SIunEFvYfZFZdBhmTqHBVlqSczVw6CD5kP3kzpX9g7dueF/B2TEZMPXxaqy2 zVl5SsnS4eutCb5o7Lffb+oqkHwzm9BflNZqw8Bapim1ix7t0bzZR0X7VDRpFG0BylV7 LlhMqLzPyUm/zfN8m5QmtqvPosZtCzeK8g6CtUllf+8p0WrfMyTyN8rMwor2G2RQ2Qha lH+HRF/OnSzAo+XmJPpyAg2GDGR0DKc3VDQzS9Zs7aYj+XhG++EUuPJ8/DmFC8IWvbdU l9OlfUqSvj4kgFnssEWAGTm+KsMxuqKJjYY3BovJcw4zEq0kDRGqjTnv12wJhgBhH557 +g== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgmb9jyr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:00 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeijA113651; Mon, 16 Aug 2021 16:42:59 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2103.outbound.protection.outlook.com [104.47.58.103]) by aserp3020.oracle.com with ESMTP id 3ae5n609gk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:42:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h6HQETKFi+hK0bFK6FSJMDqPJgfNwhdWvTfWtwbTNMBXW4m+3+sNBtS6pgneFFzVVgh2s1zNPzPm9nN5GgeWtIVN4PbP0DZBlcqU5rE+e3nNUkW1KiVVdM8p/Xu+X90DLYV91ZDnYeqND6GQUuqJ0ok7h/+8XG63ukkNlciV+xK+KYvTZ97RNf4QqMRCOs+xJ2FDAQRnwQnJZBlUjwzsFBwerkMvuzy5YDpsGzffadgDFkLAl4ADS73dpKXZdRpAa3BMe4rzxINvKUHgUVZlWpheCcJz+3ala2ohap69tAfoFh0SV/BtnjKpuKwwxh73X5WdziqD2o/iTvCHrhtiYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HU+WVZn20enFNMZWdVa5bfgLHRj1V4MI+wkyDyMte0Y=; b=SiZRLKjQKHo95wv2cjhdM1Q7fkR4q8TIaJDz1q7RcKT4iKTFPVReFb4UxLoGXZSNonQKKxuQvQ3q8H9GJbDpzaoZyTD2oon5wYRsKI/8SlDLDtldwHJ+aoB/FV70IkIU2hS81qXKNhaP8qGHh6XSj8bsmD5B89XYWagWzkOJaYCTX3CquirJ3oYYjKNYP8VdryO4MlXZEXE/VMC1bGT/wCt++zLzdre/TXk0qhaZhvTG5aalx9VCR7EI7CK1Zj+SqGtCw41nHHKjkL94oVyzWdJZmdSlyeP90K0AThvA0Ojb7RwYxsMuO9h7PMfz0khQlidi+1JHnZWINWDXjP9LTQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HU+WVZn20enFNMZWdVa5bfgLHRj1V4MI+wkyDyMte0Y=; b=epXmIH/ONiqfxCY8uH0RoM3jxsUGBXdzvWM2GWvNHYsc1o5JysY+Yo47nQ1J0lazdITxN2h3EBqXj+k/HzzS8///B1SUi0jP9s6Yg1x2LrrglWm3/PH9OaLsu4uq4qf4mdfdnXTmgWEtg88J7evkNTa7VHK5C6TKHTFj1UeMNNs= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BYAPR10MB3221.namprd10.prod.outlook.com (2603:10b6:a03:14f::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Mon, 16 Aug 2021 16:42:57 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:57 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 03/16] vfio-user: Define type vfio_user_pci_dev_info Date: Mon, 16 Aug 2021 09:42:36 -0700 Message-Id: <4153fb93ac5c353911a1321efc0a02f9f402acd5.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:56 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8a8465e7-6639-43ec-0e86-08d960d4e68d X-MS-TrafficTypeDiagnostic: BYAPR10MB3221: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1186; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bSu5LiiNnUFhnoeE8UIHZXNP4EhFo9rvdcy+jYKA1At4/YGlf2xWnWKMliwXH9A1Nl5XdVIJja3wKRKo+/x0jC2z/xlC4W5d1kh+Be/t0hhENyFmhOg+frhf9xNtAUvM9Lf6MvrXsaBBm7lFs7D78V8rBjPIaDncbnuDUzgoZGkRmXD4MUarUvpmKyaEz9Dwn+0ASGCTnJtgF7h4ThL/B04X8jgoMJjdsCGKaZ4OB9aWC00cbuJOKdhxBOZvPVz2i8PoiPq62KlSnB/Q/yRN8WPOGYQc+NaZ08JIey0obaYyF/LnDtZsP3uUAKsimNBl7imLvWQtPGiOrOV5GaKYvMdhWd4DayYXHfUCDS+fpXOwEVP4MbDN8F3WkB8hbk8bdF+uOZvVjC+ywBibN/79Qi9Zah+Nidyahr/fnIWyzrjfA5oPwGLZ9UIU7XJA5hpMYJ7cdLEjx2pv4MZGDPeoLMM7nFaHzq4wIhoztkRVAyRUikV2GnyN5WWzRW25y8HHOfEuGAO4vIrkZvrFnd13FifmDCMxLKNXqBvzhT7cCI+VvsUSamjk+knVZP8jB/dShndthRx9LSOZOFrSOPuMSUd1oUUV/bnhGGGEG065xTufCYVLD2Lnc7ia0py7J0JnPxxobsekjH8aHfp7QvWbSA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(39860400002)(376002)(346002)(366004)(396003)(136003)(2616005)(6916009)(66556008)(66476007)(66946007)(52116002)(478600001)(36756003)(5660300002)(6506007)(6666004)(6512007)(186003)(316002)(6486002)(86362001)(44832011)(8936002)(8676002)(38100700002)(4326008)(2906002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: I13HZtuLLRZzi89PHSCxCjtRdsE/gRzrrKKxXCaKLPpV7nlACyNts0Y2FKym8gS0FfSOs50O/I58ADcqUfqguMi+90SqpfR3G5y85KoRQ72tNYgULSypNKmamxGtVaXKZB9Vjc9q4aavROaQmntI4XCMp8VV+wN6IG2E+p2ev/ye2VA2Y4UXUtbFJE3kMK8XNUntIiASURBmmKxPjJAoRiLpbQBPXqUeqkgOpA8e4k3JKrmDF+waRKqgs5052vaTdBwTH3u+kD5j5WPzaf85Y5m23DQ60GMr6DEie000nAhYGFJJKQi7BUjdIzfADeWlijnkd5XyIjKKJKzSJG9Gt+JGnUQVs4gbTSG8tGqXsSY5VB0jNRXXBjUG7IgO8NPzS/3vkmvhlonPHDoIq4jxapnE2nWV0tnlJxkcJbI0W8Vj2Y2GF2d/F28IdY24Dp+YE9+cUncALDWg8/CfHZDuCKblyaC4C+NHuGAoB6OKI/4pu3NbSHfbr8iCA5r4N8ADzxcQV0Z1fxcI555+xPfiWvz+FPjTxUsV8CRuGcW3brGvuFSrRk7gAZawexZtTSXDpYX9SrE5G1/Fyb3xLeQSngNNhOfWlHEsnWyhQ6mduQf/UQPmvChLTQZUEMNgdAkS+z8rFvTKiI2Qv4eryZobTA4FK5/VYsHYhhbpmEuT+wxWiZQkjDO3/ECVVHD4HfyHErwExXnggAOEC0OssdpHILPzlNAihySZ+fIqsqqIl6cw0J6Sp+l85esQYSI+ABPM3eBbv5PaVMeoh07knqPSOnFj69DuvddMBUN3EggGvS+l4BlykaPsmJuMtLKOZRXgXwDCvdtZrjCWlrypQ/UFDwnuYb8NVAh+eZC6IJ59QxUhaymIRFcidNxkjlBG+szMZtNCzc97wEJQ9gpFXx+xThiX1Mi2G5I7zdTleSdRO5/y6v4G+Qw6WmxzupJ5CQEZfBtON5qUrFjc9YOsBiyhl1jqvofOAxIi0mAL/jCTv656A7+KVdSxKYuobiQcdu2WoEKlSxf25C0DDE/yoQDmg9X3hqS7rSLrNYfqpuTJ6ez+A9oRfcBPhgloTm2EaSDxI6vUb6vtqIWgKX9f86rdqOkUZXDMboZ+CALOhHmZNAH2LXDJHeXM6rn8U2y+ajXQZFmXoHhqlDq/VLYIwTHv9i0JdXI+NpijVZ8P0zh0DdCrOC+MndACPElstC/AJhIP7xX9QejSVdd0C7xo3Za44sCqStKsADxvOKQ/fJvtoZG9nLqdQsLft2HJqVB+pjTl4pNU/TBb3GVYGRsedQ/EyZo3sg5X3zzTQ4ubSbG6LJFDONt5X3kT5tqsS56CUFAPWB6DDjOyOAVLP/dQ+zH8bZTNjINbs9PlXutUBnZnqRLUKG/s01Dn8j6f+rAs3Yjy X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8a8465e7-6639-43ec-0e86-08d960d4e68d X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:56.8292 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1ZvnPMdum1+EnrJXOHbPYsiSt+sF8+o1MaNC9H4Zau3y/M8lji49pfJ0OlIfh2y9lvNuM6j63tIdaD/km4edzdOsm3SmCKHU6+OY2oF/wCc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3221 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-ORIG-GUID: YJicBTfLWVRwpKWVZL02A1fwAXrmG_d9 X-Proofpoint-GUID: YJicBTfLWVRwpKWVZL02A1fwAXrmG_d9 Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson New class for vfio-user with its class and instance constructors and destructors, and its pci ops. Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/pci.h | 9 ++++++ hw/vfio/pci.c | 86 +++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 95 insertions(+) diff --git a/hw/vfio/pci.h b/hw/vfio/pci.h index bbc78aaeb3..08ac6475a4 100644 --- a/hw/vfio/pci.h +++ b/hw/vfio/pci.h @@ -187,6 +187,15 @@ struct VFIOKernPCIDevice { VFIOPCIDevice device; }; +#define TYPE_VFIO_USER_PCI "vfio-user-pci" +OBJECT_DECLARE_SIMPLE_TYPE(VFIOUserPCIDevice, VFIO_USER_PCI) + +struct VFIOUserPCIDevice { + VFIOPCIDevice device; + char *sock_name; + bool secure_dma; /* disable shared mem for DMA */ +}; + /* Use uin32_t for vendor & device so PCI_ANY_ID expands and cannot match hw */ static inline bool vfio_pci_is(VFIOPCIDevice *vdev, uint32_t vendor, uint32_t device) { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index bea95efc33..d642aafb7f 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3326,3 +3326,89 @@ static void register_vfio_pci_dev_type(void) } type_init(register_vfio_pci_dev_type) + + +/* + * vfio-user routines. + */ + +/* + * Emulated devices don't use host hot reset + */ +static int vfio_user_pci_no_reset(VFIODevice *vbasedev) +{ + error_printf("vfio-user - no hot reset\n"); + return 0; +} + +static void vfio_user_pci_not_needed(VFIODevice *vbasedev) +{ + vbasedev->needs_reset = false; +} + +static VFIODeviceOps vfio_user_pci_ops = { + .vfio_compute_needs_reset = vfio_user_pci_not_needed, + .vfio_hot_reset_multi = vfio_user_pci_no_reset, + .vfio_eoi = vfio_intx_eoi, + .vfio_get_object = vfio_pci_get_object, + .vfio_save_config = vfio_pci_save_config, + .vfio_load_config = vfio_pci_load_config, +}; + +static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) +{ + ERRP_GUARD(); + VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev); + VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); + VFIODevice *vbasedev = &vdev->vbasedev; + + if (!udev->sock_name) { + error_setg(errp, "No socket specified"); + error_append_hint(errp, "Use -device vfio-user-pci,socket=\n"); + return; + } + + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); + vbasedev->dev = DEVICE(vdev); + vbasedev->fd = -1; + vbasedev->type = VFIO_DEVICE_TYPE_PCI; + vbasedev->no_mmap = false; + vbasedev->ops = &vfio_user_pci_ops; + +} + +static void vfio_user_instance_finalize(Object *obj) +{ +} + +static Property vfio_user_pci_dev_properties[] = { + DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), + DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false), + DEFINE_PROP_END_OF_LIST(), +}; + +static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) +{ + DeviceClass *dc = DEVICE_CLASS(klass); + PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + + device_class_set_props(dc, vfio_user_pci_dev_properties); + dc->desc = "VFIO over socket PCI device assignment"; + pdc->realize = vfio_user_pci_realize; +} + +static const TypeInfo vfio_user_pci_dev_info = { + .name = TYPE_VFIO_USER_PCI, + .parent = TYPE_VFIO_PCI_BASE, + .instance_size = sizeof(VFIOUserPCIDevice), + .class_init = vfio_user_pci_dev_class_init, + .instance_init = vfio_instance_init, + .instance_finalize = vfio_user_instance_finalize, +}; + +static void register_vfio_user_dev_type(void) +{ + type_register_static(&vfio_user_pci_dev_info); +} + +type_init(register_vfio_user_dev_type) From patchwork Mon Aug 16 16:42:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D391C4338F for ; Mon, 16 Aug 2021 16:45:01 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8135860EFE for ; Mon, 16 Aug 2021 16:45:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 8135860EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:56752 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfjP-0004Le-Mm for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:44:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37012) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfha-0000e9-6A for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:06 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:33086) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhX-0007uY-9M for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:05 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGZhsd008102; Mon, 16 Aug 2021 16:43:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=bHVoFr/SonDv8WbdTf1Kb6rfL+P7v0JCZowJuVtb15A=; b=gtiAxBX69Qnv4giWZxCS6aaWmiLr7T6vYivggc8IZtCfIKxwiQDcdDU+EWeGkQRiWX7Z gg8BIwIJZSbSKxjnBPrjneA1xI0poO9FnDatfwGDYo11kZKZaV1P8/klDwzXbZtsTKSi HAFMqS9eIG9hlnHpGdZkJuU/0QeJPOz8fYfhftQeXSJ1XlJrvW2a5AeGBsim1NfvGxEF XUxeGcEReC4Urt3O/bG3K8VuFE5ifNtGZCVr7uk0yGTOBACEtmeHfdoZQVQVIuGcW6+/ adQZVaP8nWlW/iOF5SZpvfLlRtIJiABrxjwrDR/wNEeNJgAHiwQwwedQGsVKKC4SCk/e Ng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2020-01-29; bh=bHVoFr/SonDv8WbdTf1Kb6rfL+P7v0JCZowJuVtb15A=; b=mO2koF+1IP0aYiGSYpMiFZGz952RczCF0B5NtBJA0Ckm/4gPG/BWN3azHq7oTR4/lVAq 0Xo/dXqm6uGkYj7HdyrlN4AeiLIKGErlyjDXpuFTo6FJrTFXgevoP/2VPmoMzqOjivXW 4GUR6PD/TFKxKXKc+aEiY2jDpqcMMvaWivxqmQcOWsSysITRqSei8DaIzPsBpBSqNGN+ eLRrOHEF29Sx99J7OMflDzDkhIeAS4WpWNe/qY5pOhDXbb9Y1Xi1Gi2uMbgYYgtCUNKn wgOOyQHFEGQ6nMCFpl7hchTu39zBv/h+alYtHdur2ouVKlynzSfeG2I/X7b3FiUi5PX2 ow== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgpghjpk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:00 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeijB113651; Mon, 16 Aug 2021 16:42:59 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2103.outbound.protection.outlook.com [104.47.58.103]) by aserp3020.oracle.com with ESMTP id 3ae5n609gk-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:42:59 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=T+HjycCWk8RntAqr2Mp1v+Ji6iSqcYdL+7NKUol8Ype6J/1jZMDxVp3iGA0mel7pKK5LZW01xKSSLK9dtmdL1aZm8L8ucnrBoZzW0wkW520X2BP875x5QEUOZoi9hvZZ/npQHj6CuK9ChuAqSho1j7BlRnxQfaWedbOzKcuUv8WLXX2BwR2hNN982D7b/Ps/2UkZkuiyftMRvjpKkAWSVu27/8AF+xNNlr1BmFQG+JYze36a/3zQxg8ws7pYEVNvhG/We+euA9xWLvWjxYD7Ue5cTo/tylb30tC11PXp3ZYmKryFuCR+Lp7WrIBNLbdbE+WenpsnjO+s86gH22F3RQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bHVoFr/SonDv8WbdTf1Kb6rfL+P7v0JCZowJuVtb15A=; b=Y2nIDJzOVS1N99blE6LMxVL1WbVsfIArd/glBRbDVZrbmSVEwaN4lm+/lOj1AKggo9/JGwUiFPsFFCA49c2rETNPsitd4Hh6MIP9L0oTzqBWpdexNjoL2ydun0WqfTgYNmvJ+QoPo3FyfIww/8+0iWhQK8wokN+5kzG2bnhMqhUyg5Jf7gufvnqNKGSJh2qU6Did+vaSp+P1MilAhsT0MLodIbeTch5M29FFHUTUV8NQmBAX+Jj59IsI+U2KuqVMXsgPFg78UiSI2zJ7CC6gDB5IdnPyTdFPAaos4+OdxVUvJQzp4Uo1jRb8SBZkhZ2zwj2MfUF0/fZ6P+QaHGE7bw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bHVoFr/SonDv8WbdTf1Kb6rfL+P7v0JCZowJuVtb15A=; b=upVZ1GvXrRG8CaeRaTqRz/ATXP+yxK/PQDYqR3jq3fMJXjqRg8UHHHiD7gszOvKP8pHqpAtweErlBJpXAgxhri8u9+TZbSaKgWL597p9TVq6vLThvSmXImMrHqcdhhRs4vjW4vhIvF3iTP4/HVfnn70f7D0dpdUiENiriVYDOiA= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BYAPR10MB3221.namprd10.prod.outlook.com (2603:10b6:a03:14f::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Mon, 16 Aug 2021 16:42:57 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:57 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 04/16] vfio-user: connect vfio proxy to remote server Date: Mon, 16 Aug 2021 09:42:37 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:56 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 062b21a8-2c59-4fbd-98ed-08d960d4e6d2 X-MS-TrafficTypeDiagnostic: BYAPR10MB3221: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:792; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: kd/DSYq1O/rdNA3fAF3eTogp5v5GwSuTO1Ro8DZlA8z+SpvsviuEm+S373J4pSEWNYlgcI1krSpNjqAZ/iqjV59qRqgpAWUgtmfFsLMzGL4KUYSc3CBY7KRGQEC7W3C3SRkJlkET5BnEaGZNd1t9S3qa+7zQk/AlVCYu0CUpPwk1MKu5apXI89eNt8ed6CBIlybgaXa751MnyxMiHpa863+a8L/xwgq8vbisZ2o9OW8xtQd5amEj9oEx+xrVTmD3G164cFK9fYb0Cfr1tcwYyIK98AV3dxjA7uUXpJZAbuZLvhoetSD0bsfIG/B8+aqBPvSz7vzbN0B0a6h1sSAeGF4DcyXX7c1LQUs6NEs6nrPpyWuVh5IHB9+FyGZIlXuvmBgZqyfOxuucM5ZmVlBfL1mCCa3Lo9o7LLYXApAHJN8oBOsXXNNqBGfRukhY8PFSJl8/J9iJeJHgjpA94gbe7h7JLuIl56/bm2qL1YaH5W4oyAmNPVOq50QVPsPsQyVTnoTIdhT+s2XVW9g67LGPCnvwJKeEDYkefn1IMnVpY+q1EjdGR7s+89S8kPgUbbqxSQQeRaKX53Qygfsb07hL/OArBk74rMjW9UKSfjwzj4s0IWqw+/eum5A3ecbnt0kMKKqtbpvoim+S4e+IQdIwoQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(39860400002)(376002)(346002)(366004)(396003)(136003)(2616005)(6916009)(66556008)(66476007)(66946007)(52116002)(83380400001)(478600001)(36756003)(5660300002)(6506007)(6666004)(6512007)(186003)(316002)(6486002)(86362001)(44832011)(8936002)(8676002)(38100700002)(4326008)(2906002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?JPVu31y6mgAPo8IqShcAQJZF6JxC?= =?utf-8?q?ph/014T2km+JGcoJXEBknDMznO0Kh71eXZTKuIRfY8jQthF+Q+UP81Mh7W6oJHr8F?= =?utf-8?q?WcJ8+OYgdjs7h2oiAvw21tm072igNBPt8jDs6WigXhveSFZOMrhkJlsyrgJ0HlpKf?= =?utf-8?q?dnqorid3TCc8/pm9MrBq4k8E69a1HVVNU3YZJwxJoLrcknKKJnE/bgI//UlrFZn2q?= =?utf-8?q?D7ZFK8wpMfhTFRZ3PhHEplPFozBEL3ulyQM3/3uCs2RX9G+B3QTqjN8Ja+q4yyBbD?= =?utf-8?q?Fa/CR3bX6QKzHYrs38rvvj0HyK2D3ImRVTW5fqb7ygrFQvaS3lh3tUbVSPnToxr4l?= =?utf-8?q?3R7wC4vSfSqnGKD8ajFVch3J8uY4a8GtbxlL5xvvQyjwJYnUCQ+XtSigq34UyMZoJ?= =?utf-8?q?Aa6Ym+ghEGxeLz9jKndYl8OepMIOZMNwDXg6j6+FLfe8KzJ8hLjU/OCDKus+6nX5/?= =?utf-8?q?u9sTeWBvonU5M12oPJF50XuzydBJ4muYI4LBb+11XO4J4c1EQ62a2zsBx3WefFtJu?= =?utf-8?q?/i2F0tIERGdfPE9oSPf2SCHaWJWIrsiu38plhWO5eZcht44rybe9Hfb88La1okcUs?= =?utf-8?q?3zA4bYStjms/KpDobNJupAD0XJRPuixyBMXfwsYy0yssAG1OJYUtG5raaX3Y0ZKS5?= =?utf-8?q?IzMpQCsQOLvktDA2tw30EK6YsML9ar0on17HjGMOVSdAsawredxZmJmNMJZ/DoxTP?= =?utf-8?q?Q2yYD8FBMjMHDKq0f63o0Hxt0OF8W5u5LFaOBeaxDJv/BKONcVzwDS+AJpwXkfzo1?= =?utf-8?q?4w9zdD5QKBBIK3GMcqtFdFW7EfPozhsFqjYydQY7qzWkLwjOWw+P2coimAqd8AvZu?= =?utf-8?q?IxJ1+eQFJCH53OJfMKWj1xZmcrhabCeJVopYJRIoORIXCknikuy2iyNV3N1KEuQZ5?= =?utf-8?q?yIbyxag2LvadWYS+5/ATyJ0Ji07T1T8KButSrvWOKb5sW8PYnOqy3ON2JvsAZNG5u?= =?utf-8?q?G9yHHpsVBMoY8HDZ+fpdclpvmhfuQXnTD3vO3OQLlRZZdYxTeubBdhZ4blLt1G5rj?= =?utf-8?q?zzuFIBUcuPa2hVqiLssiOByezlqoC2Xg6+AwdKnESOcnADfHTd65g8AqvHU4/2HKU?= =?utf-8?q?nIbaZaDuTUO4gkQ3NJminFfWW15cXMn3RtECKqpjoEa5SEeIlKKzZQd7dASqOaUSn?= =?utf-8?q?HBXkwR6eZ/Q+OY1AhYX9M6rWuosAoCgybBN4BsTM9GZ3uwMqkldEtZta/B/zJln3O?= =?utf-8?q?GDH/kVWk6ueQW7uN+gt9PZFJLfYPxfGlXJ+LGTOXJyzLS/caYkw82UA6RS6LzKmPo?= =?utf-8?q?HbmWcBGiWWhu9VLAjdGHaM02JfaN2Ys2VRsNR4l/XlpKQcP55C8FqBwsS2xQxlx3b?= =?utf-8?q?IYNcGiYF4MPf02u?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 062b21a8-2c59-4fbd-98ed-08d960d4e6d2 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:57.2769 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: BGMtzRAw/I8joIGmNr9COxHgHFmXAOCrPVRr9KJnNiU4z9f2DraZyjCUtmNfp0rx3Y9YgOlcbh0hyKCEElUoAYE3KVhR6x/OJiQCy1mcpyw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3221 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: oIFuVEKSr94_acTkya3rXAa8bHXfNF4M X-Proofpoint-ORIG-GUID: oIFuVEKSr94_acTkya3rXAa8bHXfNF4M Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 66 ++++++++++++++ include/hw/vfio/vfio-common.h | 2 + hw/vfio/pci.c | 29 ++++++ hw/vfio/user.c | 160 ++++++++++++++++++++++++++++++++++ MAINTAINERS | 4 + hw/vfio/meson.build | 1 + 6 files changed, 262 insertions(+) create mode 100644 hw/vfio/user.h create mode 100644 hw/vfio/user.c diff --git a/hw/vfio/user.h b/hw/vfio/user.h new file mode 100644 index 0000000000..62b2d03d56 --- /dev/null +++ b/hw/vfio/user.h @@ -0,0 +1,66 @@ +#ifndef VFIO_USER_H +#define VFIO_USER_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + */ + +typedef struct { + int send_fds; + int recv_fds; + int *fds; +} VFIOUserFDs; + +typedef struct VFIOUserReply { + QTAILQ_ENTRY(VFIOUserReply) next; + VFIOUserFDs *fds; + uint32_t rsize; + uint32_t id; + QemuCond cv; + bool complete; + bool nowait; +} VFIOUserReply; + + +enum proxy_state { + VFIO_PROXY_CONNECTED = 1, + VFIO_PROXY_RECV_ERROR = 2, + VFIO_PROXY_CLOSING = 3, + VFIO_PROXY_CLOSED = 4, +}; + +typedef struct VFIOProxy { + QLIST_ENTRY(VFIOProxy) next; + char *sockname; + struct QIOChannel *ioc; + int (*request)(void *opaque, char *buf, VFIOUserFDs *fds); + void *reqarg; + int flags; + QemuCond close_cv; + + /* + * above only changed when BQL is held + * below are protected by per-proxy lock + */ + QemuMutex lock; + QTAILQ_HEAD(, VFIOUserReply) free; + QTAILQ_HEAD(, VFIOUserReply) pending; + VFIOUserReply *last_nowait; + enum proxy_state state; + bool close_wait; +} VFIOProxy; + +/* VFIOProxy flags */ +#define VFIO_PROXY_CLIENT 0x1 +#define VFIO_PROXY_SECURE 0x2 + +VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); +void vfio_user_disconnect(VFIOProxy *proxy); + +#endif /* VFIO_USER_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 8af11b0a76..f43dc6e5d0 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -75,6 +75,7 @@ typedef struct VFIOAddressSpace { } VFIOAddressSpace; struct VFIOGroup; +typedef struct VFIOProxy VFIOProxy; typedef struct VFIOContainer { VFIOAddressSpace *space; @@ -143,6 +144,7 @@ typedef struct VFIODevice { VFIOMigration *migration; Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; + VFIOProxy *proxy; } VFIODevice; struct VFIODeviceOps { diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index d642aafb7f..7c2d245ca5 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -42,6 +42,7 @@ #include "qapi/error.h" #include "migration/blocker.h" #include "migration/qemu-file.h" +#include "hw/vfio/user.h" #define TYPE_VFIO_PCI_NOHOTPLUG "vfio-pci-nohotplug" @@ -3361,13 +3362,35 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIOUserPCIDevice *udev = VFIO_USER_PCI(pdev); VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); VFIODevice *vbasedev = &vdev->vbasedev; + SocketAddress addr; + VFIOProxy *proxy; + Error *err = NULL; + /* + * TODO: make option parser understand SocketAddress + * and use that instead of having scaler options + * for each socket type. + */ if (!udev->sock_name) { error_setg(errp, "No socket specified"); error_append_hint(errp, "Use -device vfio-user-pci,socket=\n"); return; } + memset(&addr, 0, sizeof(addr)); + addr.type = SOCKET_ADDRESS_TYPE_UNIX; + addr.u.q_unix.path = udev->sock_name; + proxy = vfio_user_connect_dev(&addr, &err); + if (!proxy) { + error_setg(errp, "Remote proxy not found"); + return; + } + vbasedev->proxy = proxy; + + if (udev->secure_dma) { + proxy->flags |= VFIO_PROXY_SECURE; + } + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->dev = DEVICE(vdev); vbasedev->fd = -1; @@ -3379,6 +3402,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) static void vfio_user_instance_finalize(Object *obj) { + VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); + VFIODevice *vbasedev = &vdev->vbasedev; + + vfio_put_device(vdev); + + vfio_user_disconnect(vbasedev->proxy); } static Property vfio_user_pci_dev_properties[] = { diff --git a/hw/vfio/user.c b/hw/vfio/user.c new file mode 100644 index 0000000000..3bd304e036 --- /dev/null +++ b/hw/vfio/user.c @@ -0,0 +1,160 @@ +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2 or later. + * See the COPYING file in the top-level directory. + * + */ + +#include "qemu/osdep.h" +#include +#include + +#include "qemu/error-report.h" +#include "qapi/error.h" +#include "qemu/main-loop.h" +#include "hw/hw.h" +#include "hw/vfio/vfio-common.h" +#include "hw/vfio/vfio.h" +#include "qemu/sockets.h" +#include "io/channel.h" +#include "io/channel-socket.h" +#include "io/channel-util.h" +#include "sysemu/iothread.h" +#include "user.h" + +static IOThread *vfio_user_iothread; +static void vfio_user_shutdown(VFIOProxy *proxy); + + +/* + * Functions called by main, CPU, or iothread threads + */ + +static void vfio_user_shutdown(VFIOProxy *proxy) +{ + qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); +} + + +/* + * Functions only called by iothread + */ + +static void vfio_user_cb(void *opaque) +{ + VFIOProxy *proxy = opaque; + + qemu_mutex_lock(&proxy->lock); + proxy->state = VFIO_PROXY_CLOSED; + qemu_mutex_unlock(&proxy->lock); + qemu_cond_signal(&proxy->close_cv); +} + + +/* + * Functions called by main or CPU threads + */ + +static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = + QLIST_HEAD_INITIALIZER(vfio_user_sockets); + +VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) +{ + VFIOProxy *proxy; + QIOChannelSocket *sioc; + QIOChannel *ioc; + char *sockname; + + if (addr->type != SOCKET_ADDRESS_TYPE_UNIX) { + error_setg(errp, "vfio_user_connect - bad address family"); + return NULL; + } + sockname = addr->u.q_unix.path; + + sioc = qio_channel_socket_new(); + ioc = QIO_CHANNEL(sioc); + if (qio_channel_socket_connect_sync(sioc, addr, errp)) { + object_unref(OBJECT(ioc)); + return NULL; + } + qio_channel_set_blocking(ioc, true, NULL); + + proxy = g_malloc0(sizeof(VFIOProxy)); + proxy->sockname = sockname; + proxy->ioc = ioc; + proxy->flags = VFIO_PROXY_CLIENT; + proxy->state = VFIO_PROXY_CONNECTED; + qemu_cond_init(&proxy->close_cv); + + if (vfio_user_iothread == NULL) { + vfio_user_iothread = iothread_create("VFIO user", errp); + } + + qemu_mutex_init(&proxy->lock); + QTAILQ_INIT(&proxy->free); + QTAILQ_INIT(&proxy->pending); + QLIST_INSERT_HEAD(&vfio_user_sockets, proxy, next); + + return proxy; +} + +void vfio_user_disconnect(VFIOProxy *proxy) +{ + VFIOUserReply *r1, *r2; + + qemu_mutex_lock(&proxy->lock); + + /* our side is quitting */ + if (proxy->state == VFIO_PROXY_CONNECTED) { + vfio_user_shutdown(proxy); + if (!QTAILQ_EMPTY(&proxy->pending)) { + error_printf("vfio_user_disconnect: outstanding requests\n"); + } + } + object_unref(OBJECT(proxy->ioc)); + proxy->ioc = NULL; + + proxy->state = VFIO_PROXY_CLOSING; + QTAILQ_FOREACH_SAFE(r1, &proxy->pending, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->pending, r1, next); + g_free(r1); + } + QTAILQ_FOREACH_SAFE(r1, &proxy->free, next, r2) { + qemu_cond_destroy(&r1->cv); + QTAILQ_REMOVE(&proxy->free, r1, next); + g_free(r1); + } + + /* + * Make sure the iothread isn't blocking anywhere + * with a ref to this proxy by waiting for a BH + * handler to run after the proxy fd handlers were + * deleted above. + */ + proxy->close_wait = 1; + aio_bh_schedule_oneshot(iothread_get_aio_context(vfio_user_iothread), + vfio_user_cb, proxy); + + /* drop locks so the iothread can make progress */ + qemu_mutex_unlock_iothread(); + qemu_cond_wait(&proxy->close_cv, &proxy->lock); + + /* we now hold the only ref to proxy */ + qemu_mutex_unlock(&proxy->lock); + qemu_cond_destroy(&proxy->close_cv); + qemu_mutex_destroy(&proxy->lock); + + qemu_mutex_lock_iothread(); + + QLIST_REMOVE(proxy, next); + if (QLIST_EMPTY(&vfio_user_sockets)) { + iothread_destroy(vfio_user_iothread); + vfio_user_iothread = NULL; + } + + g_free(proxy); +} diff --git a/MAINTAINERS b/MAINTAINERS index d838b9e3f2..f429bab391 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1882,8 +1882,12 @@ L: qemu-s390x@nongnu.org vfio-user M: John G Johnson M: Thanos Makatos +M: Elena Ufimtseva +M: Jagannathan Raman S: Supported F: docs/devel/vfio-user.rst +F: hw/vfio/user.c +F: hw/vfio/user.h vhost M: Michael S. Tsirkin diff --git a/hw/vfio/meson.build b/hw/vfio/meson.build index da9af297a0..739b30be73 100644 --- a/hw/vfio/meson.build +++ b/hw/vfio/meson.build @@ -8,6 +8,7 @@ vfio_ss.add(when: 'CONFIG_VFIO_PCI', if_true: files( 'display.c', 'pci-quirks.c', 'pci.c', + 'user.c', )) vfio_ss.add(when: 'CONFIG_VFIO_CCW', if_true: files('ccw.c')) vfio_ss.add(when: 'CONFIG_VFIO_PLATFORM', if_true: files('platform.c')) From patchwork Mon Aug 16 16:42:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3A11C4338F for ; Mon, 16 Aug 2021 16:48:12 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 63BA960EFE for ; Mon, 16 Aug 2021 16:48:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 63BA960EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:37300 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfmV-0001lT-JY for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:48:11 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37078) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhc-0000jF-Je for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:08 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:34316) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhY-0007uh-3V for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:08 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaA9n021054; Mon, 16 Aug 2021 16:43:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2021-07-09; bh=ujoH7bCSKiw2S++WAYDzYW+rsvzgBNv0frY7sMGl0U0=; b=KJEbxjiEjovko16lh+hxCDUiOYtkzFcT4hyQbmL35qq53C+STQOCQyDlA3UhVbdzLPAG pp4sabPzZRQQVpwwH2Syg5PI8GkwwBm0b/Cij5fBDcSVRhx5xArz0EehGmzU9+ap6kgw hoSO6+7TzM1NhyrD6RFs3iXgMu6g93s2yNQJIs5s6z7vWTnNG/39HPvNW6bXibkj8w2H 1LbavsWQDBcEnGWsm6oRODYHxNHKHuK5AsW95jemfj62tyrvrRO9NMsLzp8XmG7ibYRW mDiyLJvDGvLe4BDHxW1msouVVNAHzxElh2JP0cWvGO0OXNB8uenKLT6S+RMvs1jTPeKj dA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : content-transfer-encoding : mime-version; s=corp-2020-01-29; bh=ujoH7bCSKiw2S++WAYDzYW+rsvzgBNv0frY7sMGl0U0=; b=HJEXkH2PeXCYKiwPY0hVQ18Xs8I6b7HW40Km0mVXG8HaSByrIcP/6eg5AMW4AMK2h8xQ UkSKAchVLVVkE/1SffzcvDGvN/CNoNgZrrTpuI4zufwJRgZPg8Z2c6edlV6zdE31J3Nx KSHvJdk9QoFsYWqWzxCdXF8ALAnPyFt80dxrX8i4UOxA0Jq+iNI0hj0xVoenQv88WK6u BpHTNka7Nw+hrxb6A5SvBCvuVuXBXkdW7e2wIH7iE2TG4EOfDpPZmt8uzcpYboJbiYS1 zdRxVCB7phomrZeRlBaO20DrpfEhltbwUZjWZP5HS6O12ZAupry2S1m4T9An/RxiXAfH IA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgmb9jyx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:01 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeijC113651; Mon, 16 Aug 2021 16:43:00 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2103.outbound.protection.outlook.com [104.47.58.103]) by aserp3020.oracle.com with ESMTP id 3ae5n609gk-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=MI+fXm4lLL2F86LNXPAH9q/mbGzdOEtbcf+v11XzYGuD1tegnsZHoS+J57nB/0/PH2rShakAtReT3on8kXnNKcUrH4dbEIZUSTUxBzUvBfJBlF+zPfk9V6CbwzlNPF1AviOW0v390N20YaQnZRSKOkSzv+TTm23Cub3SCFHdvdIDpFd4R/hvN8TG2aaOIvOqqlKO/oRaAUW8vldgK0ZT9fb9APy9Jd76eVutwO0MSxglnbwqtLi7lJ4cqOpOHpm3axOTQ8lxz32+86d1/XwreteupUGjVR1NAe3XWmYTTR9F0qKCXzyM69NpY8mBXvT/dIvEUGkEAD4k0giTTRHmHg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ujoH7bCSKiw2S++WAYDzYW+rsvzgBNv0frY7sMGl0U0=; b=nlNx6trd/9Udl/840rEtK9rNR2+KDb96KZIpzFJy4Tkg21PS5Z4BmmPRZqO8PEYUz9Tz/xDY6fEkMTM9Klw1uSNqF6eI3IT7Jt5mf2crnrERKCxS45vPT8cWEiagoXUjPY/nhIqRXps8rlp4Cb8mJAZxF9eSP0SyI/HZDAhFpw99xYwpjZ2CW2mg4go8sHExE+aTCKmdmFlnXbhtrOu80GBIcmDSdtRMMyc6vIQNnxF4MhAeo5pU95vzGnu+cJORJD1GU3wJmlTP/ppw1mRJwgXSaVVBpgUNkdM3RPvUP3EbCOTsVJEKRLLhY9ps383Ru3DFerQ97Zlo+IaDgETvTA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ujoH7bCSKiw2S++WAYDzYW+rsvzgBNv0frY7sMGl0U0=; b=B4kg3BVppp9EtbKOtq/wMwD6MybBpppSdWQ7fVJnA8RBQRV9SGYBlJ2RbFdQ9Yf11rEw7EqmTycahbH2t6FAsQSwmoJGa3sHINfv1VLIHGvflcTSQorB6uboNWxMrBbzw+wDHobFoYUhoJgQguzFvMUl31vn1WXJBR21NWWGKkA= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BYAPR10MB3221.namprd10.prod.outlook.com (2603:10b6:a03:14f::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Mon, 16 Aug 2021 16:42:58 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:57 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 05/16] vfio-user: define VFIO Proxy and communication functions Date: Mon, 16 Aug 2021 09:42:38 -0700 Message-Id: <3d3befd308bacb7ec9f3ccd8f99e5184261279b2.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:57 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8ec3c874-e9e4-4ef3-1202-08d960d4e717 X-MS-TrafficTypeDiagnostic: BYAPR10MB3221: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0zsCA0XJL5UwntX+fMsaz84GhgJF4A/ZF5A5kXm7P4mO4Ssx1pf+ykZ+9DTOVOw0KK7v05wns5atpzTXITDWB/RDSH/tb/8M6b8GlIC9SAfgbZHCKTpGaKF6K37gXxZm0/VcQEfCQ1dRJMcJ1/1Lzq2RH/S+1Udss8VyroUkd6cfZV5Y26VzlpbC5LNgxG+eE8IaRg5xFYu+5SwxplNagjrjUN3NJoRMdu78b4adv5hkhQv71eBBwBrsGopn5a1XofrqjsQRytVsCUyDJ0UTkYFk6h/Ds0+lJs8UFfrxOa1vkMM9Bx3iTU7nyFbokKp3WSpV+IGGAR+8WPp/S0V8AzPHw+6OOpL60BK9m08BFwiDKUOrt/lVczC32cglJbuOoJNgHuR0TfBUWl4WE6zlbCd3LVuBNOxZ3w+KBFFTwkJNCy0c5EvG1VYlEZ4HKY2y0GD/8C1Ba/7uQumogbVadM9UPVI1vpQVlE8X2Z9cjDNgR2Yo7btM1pbxJUhAaft9zE+zoQa/r/M7NJaHP2htMSBe2CcNRZb4K6RQfAttFcLFSBmvte82BQO/MCOzANzHdLd8LQbqBRKQOnbIS3bWvTkk993r+oNs9CuvyNf28mdUxH/rPwywFNcmVmuynyPU13129C4Rv6CEEaPkaQYSfA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(39860400002)(376002)(346002)(366004)(396003)(136003)(2616005)(6916009)(66556008)(66476007)(66946007)(52116002)(30864003)(83380400001)(478600001)(36756003)(5660300002)(6506007)(6666004)(6512007)(186003)(316002)(6486002)(86362001)(44832011)(8936002)(8676002)(38100700002)(4326008)(2906002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?ve3HL9vAQUR36Kp7faDnIbpS/BQo?= =?utf-8?q?MtJtqbs8lZQ5dUVBtoudreIe1dcS3XvqP068wVdTlLfy8SWdvjRqYksHTxB5KRrRL?= =?utf-8?q?IHvVfI3OF2WSh+7+TsHCdcu0Z1K+bgmIdnULbgQpzTYAUk8vgwPtftWEQIZWwA+ln?= =?utf-8?q?CN53ZXsBmvq8kBv0BdLiMZzdGPZmcJLwuv8VLfZIrtGRBQHeslFcXM86wIQCchWvm?= =?utf-8?q?gy+BM5KWQFjid/58NZL3hpbGuOZdx25TfpgAHBV6Bmbsls7FNLtpVfv1pnOg2n3Qn?= =?utf-8?q?guKKfJ8eFDEMLWt8yXCvJYmwnlsL0zLlnBLYa6BRz1yW0j/uzaUqYtJeOr3vvFkbo?= =?utf-8?q?YKNCCN6u9YVx+3Wiqgcoer8f4C6O9irA4kxFiknfYcTFKSucA2KCgxgmNU2ad9mFc?= =?utf-8?q?2TBV1fGNIgPnH2tkZr43Es9eJ8JOGXpHBJIsViV7wFsRIparkvKBurhW8sgzmrqyX?= =?utf-8?q?ItQLYU+cYzn5nKuZwjXtGOUrtYIcPssNMiJg6KUPV8NauiIvmFjGpQUibolh1MPbE?= =?utf-8?q?gcb7xtxwQfMye8xPkAIHARCbC3te4PRiRl4BO7Km78OSpiuyTU3rH+xK0hBKk9Nqo?= =?utf-8?q?O1+eMPdvwE89xSXK09yacdkYRHwzo3rE0B98CBAhS2IquR0brUgXeKdkbg/sq31NY?= =?utf-8?q?WzNqtVhFWq5iZwRoVyJ2215Dv7jJWITDcmnaFw6TUR9s/Yfakq+vrHqEOoGgBPcD4?= =?utf-8?q?31pYzbpRAXoDbYAN/w4P4n8FaiUEKQvtZFyCvzXLXK8BnIH+mKsUHtcDY/VahRGqQ?= =?utf-8?q?4cGz8AMN+3q4Fc1ujP2hzzDDj9rjYGL8fp+al44tBzVei324hDzbtZaGV4g96sV96?= =?utf-8?q?l015piNIZkvrRbgpokddnzKnQaBsgwepqgVlFmFicwgDBdzzujicXOiI0ISzafl+o?= =?utf-8?q?ybtlNQEDUvforcXDgUlUqVh4qXVryQ5Mq511EJe56ODDMD+3ASyBVQD7bTx+9397b?= =?utf-8?q?ZVvEwhbbdTntmUZ7xcKM5RD2Y/xVpE1xujZgCBAAPy+UhKZ6PjGSSM5C0JP2jdxok?= =?utf-8?q?di4IDiCRMxbhHLjAZrRcGFcHykJDURPgN4kuGDp58Ek8qd7U0XSOiWFB93NxKftEt?= =?utf-8?q?cFM08vo95JPM5FqhqA2CY1CzWDWSJQeyDaywfWOnrWpTZVSsqQJm/zmfspDURZffa?= =?utf-8?q?a//6Y3HGNSIbGVadHjpdRBTK/qfQN0CFEYjlqBpFmwcV0E79TM+/6tdz5elmXvdKP?= =?utf-8?q?RLNCdcbHYE9oJRzabbmN423i4W6Rn8C+duKMSWEJUy1cf7HHIP6yH+t8WFSa2VcDV?= =?utf-8?q?dBd2xHjp34uLqZ4KyMvNegzMqqRiaWuEx3uA1WowUrySNpzdXB4ohyRNzoDDH6OTD?= =?utf-8?q?hw8UEiF7nuDxNc1?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8ec3c874-e9e4-4ef3-1202-08d960d4e717 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:57.7597 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: j95TseykZUMVeKCmFNyGIGKv6Hsmaq2/Ls8jlHzE8/cJHO083BP60mddnLa5CrLUPWhmha3OqX80WFyl6G7Fi/W33EozQfGq7WiKK8TYXHM= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3221 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-ORIG-GUID: 3EqtAwqddRKT-csdXOJdSZEoa9o1xvIb X-Proofpoint-GUID: 3EqtAwqddRKT-csdXOJdSZEoa9o1xvIb Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 62 +++++++++ hw/vfio/user.h | 8 ++ hw/vfio/pci.c | 6 + hw/vfio/user.c | 289 ++++++++++++++++++++++++++++++++++++++++ MAINTAINERS | 1 + 5 files changed, 366 insertions(+) create mode 100644 hw/vfio/user-protocol.h diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h new file mode 100644 index 0000000000..27062cb910 --- /dev/null +++ b/hw/vfio/user-protocol.h @@ -0,0 +1,62 @@ +#ifndef VFIO_USER_PROTOCOL_H +#define VFIO_USER_PROTOCOL_H + +/* + * vfio protocol over a UNIX socket. + * + * Copyright © 2018, 2021 Oracle and/or its affiliates. + * + * This work is licensed under the terms of the GNU GPL, version 2. See + * the COPYING file in the top-level directory. + * + * Each message has a standard header that describes the command + * being sent, which is almost always a VFIO ioctl(). + * + * The header may be followed by command-specific data, such as the + * region and offset info for read and write commands. + */ + +typedef struct { + uint16_t id; + uint16_t command; + uint32_t size; + uint32_t flags; + uint32_t error_reply; +} VFIOUserHdr; + +/* VFIOUserHdr commands */ +enum vfio_user_command { + VFIO_USER_VERSION = 1, + VFIO_USER_DMA_MAP = 2, + VFIO_USER_DMA_UNMAP = 3, + VFIO_USER_DEVICE_GET_INFO = 4, + VFIO_USER_DEVICE_GET_REGION_INFO = 5, + VFIO_USER_DEVICE_GET_REGION_IO_FDS = 6, + VFIO_USER_DEVICE_GET_IRQ_INFO = 7, + VFIO_USER_DEVICE_SET_IRQS = 8, + VFIO_USER_REGION_READ = 9, + VFIO_USER_REGION_WRITE = 10, + VFIO_USER_DMA_READ = 11, + VFIO_USER_DMA_WRITE = 12, + VFIO_USER_DEVICE_RESET = 13, + VFIO_USER_DIRTY_PAGES = 14, + VFIO_USER_MAX, +}; + +/* VFIOUserHdr flags */ +#define VFIO_USER_REQUEST 0x0 +#define VFIO_USER_REPLY 0x1 +#define VFIO_USER_TYPE 0xF + +#define VFIO_USER_NO_REPLY 0x10 +#define VFIO_USER_ERROR 0x20 + + +#define VFIO_USER_DEF_MAX_FDS 8 +#define VFIO_USER_MAX_MAX_FDS 16 + +#define VFIO_USER_DEF_MAX_XFER (1024 * 1024) +#define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) + + +#endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 62b2d03d56..905e374e12 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -11,6 +11,8 @@ * */ +#include "user-protocol.h" + typedef struct { int send_fds; int recv_fds; @@ -19,6 +21,7 @@ typedef struct { typedef struct VFIOUserReply { QTAILQ_ENTRY(VFIOUserReply) next; + VFIOUserHdr *msg; VFIOUserFDs *fds; uint32_t rsize; uint32_t id; @@ -62,5 +65,10 @@ typedef struct VFIOProxy { VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); +void vfio_user_set_reqhandler(VFIODevice *vbasdev, + int (*handler)(void *opaque, char *buf, + VFIOUserFDs *fds), + void *reqarg); +void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7c2d245ca5..7005d9f891 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3333,6 +3333,11 @@ type_init(register_vfio_pci_dev_type) * vfio-user routines. */ +static int vfio_user_pci_process_req(void *opaque, char *buf, VFIOUserFDs *fds) +{ + return 0; +} + /* * Emulated devices don't use host hot reset */ @@ -3386,6 +3391,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) return; } vbasedev->proxy = proxy; + vfio_user_set_reqhandler(vbasedev, vfio_user_pci_process_req, vdev); if (udev->secure_dma) { proxy->flags |= VFIO_PROXY_SECURE; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 3bd304e036..2fcc77d997 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -25,8 +25,15 @@ #include "sysemu/iothread.h" #include "user.h" +static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER; static IOThread *vfio_user_iothread; + static void vfio_user_shutdown(VFIOProxy *proxy); +static void vfio_user_recv(void *opaque); +static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, + VFIOUserFDs *fds); +static void vfio_user_send(VFIOProxy *proxy, VFIOUserHdr *msg, + VFIOUserFDs *fds); /* @@ -36,6 +43,67 @@ static void vfio_user_shutdown(VFIOProxy *proxy); static void vfio_user_shutdown(VFIOProxy *proxy) { qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); + qio_channel_set_aio_fd_handler(proxy->ioc, + iothread_get_aio_context(vfio_user_iothread), + NULL, NULL, NULL); +} + +static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, + VFIOUserFDs *fds) +{ + struct iovec iov = { + .iov_base = msg, + .iov_len = msg->size, + }; + size_t numfds = 0; + int msgleft, ret, *fdp = NULL; + char *buf; + Error *local_err = NULL; + + if (proxy->state != VFIO_PROXY_CONNECTED) { + msg->flags |= VFIO_USER_ERROR; + msg->error_reply = ECONNRESET; + return; + } + + if (fds != NULL && fds->send_fds != 0) { + numfds = fds->send_fds; + fdp = fds->fds; + } + + ret = qio_channel_writev_full(proxy->ioc, &iov, 1, fdp, numfds, &local_err); + if (ret < 0) { + goto err; + } + if (ret == msg->size) { + return; + } + + buf = iov.iov_base + ret; + msgleft = iov.iov_len - ret; + do { + ret = qio_channel_write(proxy->ioc, buf, msgleft, &local_err); + if (ret < 0) { + goto err; + } + buf += ret; + msgleft -= ret; + } while (msgleft != 0); + return; + +err: + msg->flags |= VFIO_USER_ERROR; + msg->error_reply = EIO; + error_report_err(local_err); +} + +static void vfio_user_send(VFIOProxy *proxy, VFIOUserHdr *msg, + VFIOUserFDs *fds) +{ + + qemu_mutex_lock(&proxy->lock); + vfio_user_send_locked(proxy, msg, fds); + qemu_mutex_unlock(&proxy->lock); } @@ -43,6 +111,213 @@ static void vfio_user_shutdown(VFIOProxy *proxy) * Functions only called by iothread */ +void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret) +{ + VFIOUserHdr *hdr = (VFIOUserHdr *)buf; + + /* + * convert header to associated reply + * positive ret is reply size, negative is error code + */ + hdr->flags = VFIO_USER_REPLY; + if (ret >= sizeof(VFIOUserHdr)) { + hdr->size = ret; + } else if (ret < 0) { + hdr->flags |= VFIO_USER_ERROR; + hdr->error_reply = -ret; + hdr->size = sizeof(*hdr); + } else { + error_printf("vfio_user_send_reply - size too small\n"); + return; + } + vfio_user_send(proxy, hdr, NULL); +} + +void vfio_user_recv(void *opaque) +{ + VFIODevice *vbasedev = opaque; + VFIOProxy *proxy = vbasedev->proxy; + VFIOUserReply *reply = NULL; + g_autofree int *fdp = NULL; + VFIOUserFDs reqfds = { 0, 0, fdp }; + VFIOUserHdr msg; + struct iovec iov = { + .iov_base = &msg, + .iov_len = sizeof(msg), + }; + bool isreply; + int i, ret; + size_t msgleft, numfds = 0; + char *data = NULL; + g_autofree char *buf = NULL; + Error *local_err = NULL; + + qemu_mutex_lock(&proxy->lock); + if (proxy->state == VFIO_PROXY_CLOSING) { + qemu_mutex_unlock(&proxy->lock); + return; + } + + ret = qio_channel_readv_full(proxy->ioc, &iov, 1, &fdp, &numfds, + &local_err); + if (ret <= 0) { + /* read error or other side closed connection */ + goto fatal; + } + + if (ret < sizeof(msg)) { + error_setg(&local_err, "vfio_user_recv short read of header"); + goto err; + } + if (msg.size < sizeof(VFIOUserHdr)) { + error_setg(&local_err, "vfio_user_recv bad header size"); + goto err; + } + + /* + * For replies, find the matching pending request + */ + switch (msg.flags & VFIO_USER_TYPE) { + case VFIO_USER_REQUEST: + isreply = 0; + break; + case VFIO_USER_REPLY: + isreply = 1; + break; + default: + error_setg(&local_err, "vfio_user_recv unknown message type"); + goto err; + } + + if (isreply) { + QTAILQ_FOREACH(reply, &proxy->pending, next) { + if (msg.id == reply->id) { + break; + } + } + if (reply == NULL) { + error_setg(&local_err, "vfio_user_recv unexpected reply"); + goto err; + } + QTAILQ_REMOVE(&proxy->pending, reply, next); + + /* + * Process any received FDs + */ + if (numfds != 0) { + if (reply->fds == NULL || reply->fds->recv_fds < numfds) { + error_setg(&local_err, "vfio_user_recv unexpected FDs"); + goto err; + } + reply->fds->recv_fds = numfds; + memcpy(reply->fds->fds, fdp, numfds * sizeof(int)); + } + + } else { + /* + * The client doesn't expect any FDs in requests, but + * they will be expected on the server + */ + if (numfds != 0 && (proxy->flags & VFIO_PROXY_CLIENT)) { + error_setg(&local_err, "vfio_user_recv fd in client reply"); + goto err; + } + reqfds.recv_fds = numfds; + } + + /* + * put the whole message into a single buffer + */ + if (isreply) { + if (msg.size > reply->rsize) { + error_setg(&local_err, + "vfio_user_recv reply larger than recv buffer"); + goto fatal; + } + *reply->msg = msg; + data = (char *)reply->msg + sizeof(msg); + } else { + if (msg.size > max_xfer_size) { + error_setg(&local_err, "vfio_user_recv request larger than max"); + goto fatal; + } + buf = g_malloc0(msg.size); + memcpy(buf, &msg, sizeof(msg)); + data = buf + sizeof(msg); + } + + msgleft = msg.size - sizeof(msg); + if (msgleft != 0) { + ret = qio_channel_read(proxy->ioc, data, msgleft, &local_err); + if (ret < 0) { + goto fatal; + } + if (ret != msgleft) { + error_setg(&local_err, "vfio_user_recv short read of msg body"); + goto err; + } + } + + /* + * Replies signal a waiter, requests get processed by vfio code + * that may assume the iothread lock is held. + */ + if (isreply) { + reply->complete = 1; + if (!reply->nowait) { + qemu_cond_signal(&reply->cv); + } else { + if (msg.flags & VFIO_USER_ERROR) { + error_printf("vfio_user_rcv error reply on async request "); + error_printf("command %x error %s\n", msg.command, + strerror(msg.error_reply)); + } + /* just free it if no one is waiting */ + reply->nowait = 0; + if (proxy->last_nowait == reply) { + proxy->last_nowait = NULL; + } + g_free(reply->msg); + QTAILQ_INSERT_HEAD(&proxy->free, reply, next); + } + qemu_mutex_unlock(&proxy->lock); + } else { + qemu_mutex_unlock(&proxy->lock); + qemu_mutex_lock_iothread(); + /* + * make sure proxy wasn't closed while we waited + * checking state without holding the proxy lock is safe + * since it's only set to CLOSING when BQL is held + */ + if (proxy->state != VFIO_PROXY_CLOSING) { + ret = proxy->request(proxy->reqarg, buf, &reqfds); + if (ret < 0 && !(msg.flags & VFIO_USER_NO_REPLY)) { + vfio_user_send_reply(proxy, buf, ret); + } + } + qemu_mutex_unlock_iothread(); + } + return; + +fatal: + vfio_user_shutdown(proxy); + proxy->state = VFIO_PROXY_RECV_ERROR; + +err: + for (i = 0; i < numfds; i++) { + close(fdp[i]); + } + if (reply != NULL) { + /* force an error to keep sending thread from hanging */ + reply->msg->flags |= VFIO_USER_ERROR; + reply->msg->error_reply = EINVAL; + reply->complete = 1; + qemu_cond_signal(&reply->cv); + } + qemu_mutex_unlock(&proxy->lock); + error_report_err(local_err); +} + static void vfio_user_cb(void *opaque) { VFIOProxy *proxy = opaque; @@ -101,6 +376,20 @@ VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp) return proxy; } +void vfio_user_set_reqhandler(VFIODevice *vbasedev, + int (*handler)(void *opaque, char *buf, + VFIOUserFDs *fds), + void *reqarg) +{ + VFIOProxy *proxy = vbasedev->proxy; + + proxy->request = handler; + proxy->reqarg = reqarg; + qio_channel_set_aio_fd_handler(proxy->ioc, + iothread_get_aio_context(vfio_user_iothread), + vfio_user_recv, NULL, vbasedev); +} + void vfio_user_disconnect(VFIOProxy *proxy) { VFIOUserReply *r1, *r2; diff --git a/MAINTAINERS b/MAINTAINERS index f429bab391..52d37dd088 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -1888,6 +1888,7 @@ S: Supported F: docs/devel/vfio-user.rst F: hw/vfio/user.c F: hw/vfio/user.h +F: hw/vfio/user-protocol.h vhost M: Michael S. Tsirkin From patchwork Mon Aug 16 16:42:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A1F0C4338F for ; Mon, 16 Aug 2021 16:47:50 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 059F760EFE for ; Mon, 16 Aug 2021 16:47:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 059F760EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:35726 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfm9-0000kv-1T for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:47:49 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37060) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhc-0000iA-6M for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:08 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:36808) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhX-0007xB-Lf for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:07 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaDUi000575; Mon, 16 Aug 2021 16:43:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=wurk93t3EgNHShjLwZYYsyPz+MrafcioJ7ppfdsGyhk=; b=yFE8kNoS7OZl8IAfyGUcvABGJMs0B8txERGOuzJR9WzKrMUFtX/OYAj6lXPqaIF4rBgd G7svVm9m4G1TgbtmqbGek82B3npQHfmy43ULW51CwiLmqkHkGHQ1Y5TDVE+lbgul9ifQ WRlIFDYFJr1A0LdJHXDBSQf5i/qpyfZAFQ/T7TjLROwKUMAKSoGj2JEZFXrHt/6ZF81Q DdIiuSgY7njgvo3uZdKRFFGrrjdcyrU616N9L4RjU+Gu2PZOm0+19rrdVTVsQe8ydaF+ ngWLGfxhuI2UhsLEIFaTaY3wC1hze2nKpFQoWILDjn1jqGj4Fw/vaErqA9NeBVxg3hme ZA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=wurk93t3EgNHShjLwZYYsyPz+MrafcioJ7ppfdsGyhk=; b=SgXhP8i6LSMKUZz68bqKmkV22m0eLhprEpzYd6NIpqenOzHZsSIMmSwrIAGBpnPkeWNn Gbdtrq2iQ21z4YSGPnfgLq+XtPEw6cbnIIlTeQI9sdUEgY8XPiriD5FZQi37H0fD78bp yxe3ryvlKLZZsMUxYJjo5yeAoWm+dnh1dOuUkBwG2RJZnXr6bzx79+q8eym+4epU/EWj TF7HooYPnyNu6VLvbfeIW1TpWmVJvABFpw9CziHw8I1XpGbEC/EUNGHs8yLWbjVENiXq gULrbabNW+6yJDLw6AumT1hbIPDE3ZeOFVco7s41ftyaLChEYnCF3Lne3dyC3iFCU+4H jg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3af1q9anuw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:01 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeijD113651; Mon, 16 Aug 2021 16:43:01 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2103.outbound.protection.outlook.com [104.47.58.103]) by aserp3020.oracle.com with ESMTP id 3ae5n609gk-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=G5RYhsXAZiUJ7PJ08QtSW1vu1JtjV9OIdEdNM1U+G9m1INYYZPZAzqubEuNYtT5qOMU0lYZ51jTwLPxnozZ1oez0fepC7pm4ynf0p7q9suK0aSwehxgDrDJqnAGol4LwE8srN1NvTKMaAqpaMmi2DsNJwhujdqaJHdRZ/G0VTT3QeSkKQkXfChRAEScoj7DEdQUe2n9wdTrejit90mPT0Jl3r8m/RgGBQ05OjwPvUSb/ezXtVcvo1oBF/lu3gXH+LNE2WMVKEwoAb+FhZAhvUgBV9IV3/SwsC1VXHoKf/lWcznFi1V6HAIJEhR9MHzBCSDLORCYcem+siR8puEi85w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wurk93t3EgNHShjLwZYYsyPz+MrafcioJ7ppfdsGyhk=; b=K/9+Upc5XkqDH9adaXFCVqEcxcku8sJFxL91RRV9cYDXS+JcXoRjxvb511Govn79I1j6vOdlsyiH4MDjnJfCNIA3OTUtltM4A/aj6IX9rmZ0nCRYlnb3HcaLIhcEjRdP8FUV4yAOWCuIy+Yb5pJ7jamml1SRRmcnpCnMl7avUFScr9+x/QU/6gKBY7yi5QJdg6Lw5bGkTRgWuUfUrjc8EyP1M2rkYnN0YfDShCf72AK444DynPold5+0bPtpCHyXyIgjJM2m/fmpb4eaKEswuqFbvyd4DZszYv/90FRb1qVM1/osngLCseSkEM1AxJBlGVePTnBlHZG8zkYawM53Cw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=wurk93t3EgNHShjLwZYYsyPz+MrafcioJ7ppfdsGyhk=; b=J0QmalywJKNTDM4/WF12whivR1NkfV7K7oUJYy8F7CSEodQxXDd0NwpMusEslys0qHpB8PG6ecIH04W3ZQ6M8yBp8EKp3gn029+yE7/NN+UxcBlpiHjcPh3kaLBn1Mvkp6asOd0f+mU9Y6aDOamnoDPUtxn/Pzys95nLEzWucuc= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by BYAPR10MB3221.namprd10.prod.outlook.com (2603:10b6:a03:14f::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.19; Mon, 16 Aug 2021 16:42:58 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:58 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 06/16] vfio-user: negotiate version with remote server Date: Mon, 16 Aug 2021 09:42:39 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:57 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 939681f2-6134-4cb6-8220-08d960d4e75f X-MS-TrafficTypeDiagnostic: BYAPR10MB3221: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:873; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HeK6ZdnVt6Pnd4buy7pEm2C6454sl1eMbOwXXfmZGILSeob5hPwKcR6ijWE6zQIu5Xq1THlHKmH/HJ9IdAK7pDZGx5Zc+SEJq2ljYpfD2mIAm4us+AmD3sobzrNF0c70s/lsHtfeSvEeEcUfT8/9x9bjjyscsHGUV0OHzM4a8mc8KCqGEj/QoEzNoLS9WUsURF1WSmS7aYIBwuumoluBQHliXlVG57m4BZlxTmdg/nPqpDkEKcSJbFn0O2qiF5dSgG6AUoDx4GfuYq3Hff3SiCD9T0s0vj7C6v8U41goIHktzkxcWTfbQG8FLb3WTFklQlewZLlOKyC9x46hiqReddTk9d79HovyQx0BLjqYQ/M3+soB8JqWZ2qDFK7TF6gLzOd3fjRbBRgVXbO0MO83AXVSgURu4HxFnciwSY3RFKuuLjTkqqjG4OcV1JOmKO9uBhGKDYOA5xndFVz+DqNk14HKr7z89b34h0W0kn0UPLw2PrfCTx1enjsyeSEkFg9aWu9H66WOYi4Z8povrg2gHoZjnb0AxPD5rdAw2Ay2SFZ2jMZzUltqnO5gDnSE2twVBzFJCjua/u8jFO9dIWEctZD32Z6hpJhOpPfeOvcSHDxOg6pV67hlGfO/6hU02FmoRvfoGcYA9Fx88n81EAw43Q== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(39860400002)(376002)(346002)(366004)(396003)(136003)(2616005)(6916009)(66556008)(66476007)(66946007)(52116002)(83380400001)(478600001)(36756003)(5660300002)(6506007)(6666004)(6512007)(186003)(316002)(6486002)(86362001)(44832011)(8936002)(8676002)(38100700002)(4326008)(2906002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: WhpJ0+8QuK7pvrGIgm5mO2gk8x49uPB7cCcpCNbc7EdirI5uZLIhv6Y121yzBs7BOzgP7b9gyHOaFE3ydoWqirlv9JPdkDBiaJB7lEeBMaXjzm5R5VFFIz83IfbeAML7q683uDmJhC/tMdigW0ZiVcUwfinAQCWyGYkyU2v4DNQGgyTG2rTS5keamV21AslmmUKwZDARTXgWVG0Gdea9O35NzdzolTF6MHtWxBh/Rq6wHO7OWXI0VaVXVitcT5pDKIQIYQc7TnXlngZ5ZTO/lYYfwGg8nH40EYx2I7A+D/VkqWKcy2zHERlsD3FW+6j3gESJXU7tWfkiQByUj3r+6nV6HbEIVIUOA+FtNqYR9y5f4zWd+o/0oXvVuBIi9YhD3hr8tDSMP+C8tKJD1VnjLifxegsWLHrWrKpfr2NMjqF/zYl/S2DNiwxAFT3le5hnMo4oib42wRfVwhGVKD/CnBi5znR/2uV3va1bGDQxD9A/WXCPL64XVRoK3UgltF1ApFYXmGILqsNf9MlXKtqVXASnu4jSq/gDrcQwlxgg533IBBnFgDrx9RomCpOyJVCuRw25SHWowhjzQohFW77SbGS0iM198FB/YXFQ7+QzdMQEoLhpEC47rYP67+dnvO5o+hznbQEOSdwAr3PCvHmxBccdMXlfufGGHV/nqAFEHWsIP9Eq95a56kcqBgROE3zXiBZWYAXC/ijzQLK3QlCipgeOu/CaeErR5IeetmBBAvJOpmNFAkC1muI1hNBiG/RalhocU/MweJTc+3KZ1d/twf4ocsg/+aoan9qKlfUJATSWWB225LQBixExozdle+9mIcPqobO5xA7tHRchUEy60O5lQqZLa4AygSA3h/DnWeKrZ/Ol1HXZ3YlFKX2qkesi3RrvebChE8envNj9hbDlD2BQPWrwRhYMg3IrOPZi8ie7Me78PVqymX5L60i1x4gOX07dEhAbRq7kSWmTswO6+KknT3un7BChCxp19RXRyH/9cqpISXZx2KuO8PbofpzoMCZeEF208wHnf1McBQsj6gc4txHInxCPoa5w2XdKf8YMC1vT4OpQNp1n1+44nhzK21JYpMf89zMMxXWS8hEO2OCsx4kN8Ho3PuPNNZ48lC0zuiG7QBnVpUvFqBjqrJLQS9w5kKCB8srNMoHacAjsWsW7d0RDwNZ6v49oC3DB31Kc7F4iyeRCMYjyiMYEmbFy/Iloort/JsDaHVyT3nZrZvSEdpcoTd9nq9g2fg/w7SEbQS1mSUGlSFOkPbyIX0ybUbe3/5yD7D+WvCU8exJ5uFqecwN8v7W+4r7KigPRDyD8dMc/IVoKhJmldvjoeUcpJ3yB3NZTBJL6n8QU1XTZlE+5HnxSW9R7K+RURai0FP69JOtqwNRfTFBjhXzOCGiI X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 939681f2-6134-4cb6-8220-08d960d4e75f X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:58.6541 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: y5+PNTGiUgwTyLXxUePO9cVlxzFhAbXThzQOIbcNYuINaa0m/41FsRT+Nb/oLPJgDUFBI/7VksWHLvQDUhZ1SXAgYuusn9HoXpkLPcpmD7w= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3221 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: MLR0wU_GX5YYoJf4hzscZrCrhsnCo7BD X-Proofpoint-ORIG-GUID: MLR0wU_GX5YYoJf4hzscZrCrhsnCo7BD Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -8 X-Spam_score: -0.9 X-Spam_bar: / X-Spam_report: (-0.9 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson --- hw/vfio/user-protocol.h | 23 ++++ hw/vfio/user.h | 1 + hw/vfio/pci.c | 9 ++ hw/vfio/user.c | 267 ++++++++++++++++++++++++++++++++++++++++ 4 files changed, 300 insertions(+) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 27062cb910..14b762d1ad 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -52,6 +52,29 @@ enum vfio_user_command { #define VFIO_USER_ERROR 0x20 +/* + * VFIO_USER_VERSION + */ +typedef struct { + VFIOUserHdr hdr; + uint16_t major; + uint16_t minor; + char capabilities[]; +} VFIOUserVersion; + +#define VFIO_USER_MAJOR_VER 0 +#define VFIO_USER_MINOR_VER 0 + +#define VFIO_USER_CAP "capabilities" + +/* "capabilities" members */ +#define VFIO_USER_CAP_MAX_FDS "max_msg_fds" +#define VFIO_USER_CAP_MAX_XFER "max_data_xfer_size" +#define VFIO_USER_CAP_MIGR "migration" + +/* "migration" member */ +#define VFIO_USER_CAP_PGSIZE "pgsize" + #define VFIO_USER_DEF_MAX_FDS 8 #define VFIO_USER_MAX_MAX_FDS 16 diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 905e374e12..cab957ba7a 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -70,5 +70,6 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, VFIOUserFDs *fds), void *reqarg); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); +int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 7005d9f891..eae33e746f 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3397,6 +3397,12 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) proxy->flags |= VFIO_PROXY_SECURE; } + vfio_user_validate_version(vbasedev, &err); + if (err != NULL) { + error_propagate(errp, err); + goto error; + } + vbasedev->name = g_strdup_printf("VFIO user <%s>", udev->sock_name); vbasedev->dev = DEVICE(vdev); vbasedev->fd = -1; @@ -3404,6 +3410,9 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; +error: + vfio_user_disconnect(proxy); + error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } static void vfio_user_instance_finalize(Object *obj) diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 2fcc77d997..e89464a571 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -23,9 +23,16 @@ #include "io/channel-socket.h" #include "io/channel-util.h" #include "sysemu/iothread.h" +#include "qapi/qmp/qdict.h" +#include "qapi/qmp/qjson.h" +#include "qapi/qmp/qnull.h" +#include "qapi/qmp/qstring.h" +#include "qapi/qmp/qnum.h" #include "user.h" static uint64_t max_xfer_size = VFIO_USER_DEF_MAX_XFER; +static uint64_t max_send_fds = VFIO_USER_DEF_MAX_FDS; +static int wait_time = 1000; /* wait 1 sec for replies */ static IOThread *vfio_user_iothread; static void vfio_user_shutdown(VFIOProxy *proxy); @@ -34,7 +41,14 @@ static void vfio_user_send_locked(VFIOProxy *proxy, VFIOUserHdr *msg, VFIOUserFDs *fds); static void vfio_user_send(VFIOProxy *proxy, VFIOUserHdr *msg, VFIOUserFDs *fds); +static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags); +static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg, + VFIOUserFDs *fds, int rsize, int flags); +/* vfio_user_send_recv flags */ +#define NOWAIT 0x1 /* do not wait for reply */ +#define NOIOLOCK 0x2 /* do not drop iolock */ /* * Functions called by main, CPU, or iothread threads @@ -333,6 +347,79 @@ static void vfio_user_cb(void *opaque) * Functions called by main or CPU threads */ +static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg, + VFIOUserFDs *fds, int rsize, int flags) +{ + VFIOUserReply *reply; + bool iolock = 0; + + if (msg->flags & VFIO_USER_NO_REPLY) { + error_printf("vfio_user_send_recv on async message\n"); + return; + } + + /* + * We may block later, so use a per-proxy lock and let + * the iothreads run while we sleep unless told no to + */ + QEMU_LOCK_GUARD(&proxy->lock); + if (!(flags & NOIOLOCK)) { + iolock = qemu_mutex_iothread_locked(); + if (iolock) { + qemu_mutex_unlock_iothread(); + } + } + + reply = QTAILQ_FIRST(&proxy->free); + if (reply != NULL) { + QTAILQ_REMOVE(&proxy->free, reply, next); + reply->complete = 0; + } else { + reply = g_malloc0(sizeof(*reply)); + qemu_cond_init(&reply->cv); + } + reply->msg = msg; + reply->fds = fds; + reply->id = msg->id; + reply->rsize = rsize ? rsize : msg->size; + QTAILQ_INSERT_TAIL(&proxy->pending, reply, next); + + vfio_user_send_locked(proxy, msg, fds); + if (!(msg->flags & VFIO_USER_ERROR)) { + if (!(flags & NOWAIT)) { + while (reply->complete == 0) { + if (!qemu_cond_timedwait(&reply->cv, &proxy->lock, wait_time)) { + msg->flags |= VFIO_USER_ERROR; + msg->error_reply = ETIMEDOUT; + break; + } + } + QTAILQ_INSERT_HEAD(&proxy->free, reply, next); + } else { + reply->nowait = 1; + proxy->last_nowait = reply; + } + } else { + QTAILQ_INSERT_HEAD(&proxy->free, reply, next); + } + + if (iolock) { + qemu_mutex_lock_iothread(); + } +} + +static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, + uint32_t size, uint32_t flags) +{ + static uint16_t next_id; + + hdr->id = qatomic_fetch_inc(&next_id); + hdr->command = cmd; + hdr->size = size; + hdr->flags = (flags & ~VFIO_USER_TYPE) | VFIO_USER_REQUEST; + hdr->error_reply = 0; +} + static QLIST_HEAD(, VFIOProxy) vfio_user_sockets = QLIST_HEAD_INITIALIZER(vfio_user_sockets); @@ -447,3 +534,183 @@ void vfio_user_disconnect(VFIOProxy *proxy) g_free(proxy); } + +struct cap_entry { + const char *name; + int (*check)(QObject *qobj, Error **errp); +}; + +static int caps_parse(QDict *qdict, struct cap_entry caps[], Error **errp) +{ + QObject *qobj; + struct cap_entry *p; + + for (p = caps; p->name != NULL; p++) { + qobj = qdict_get(qdict, p->name); + if (qobj != NULL) { + if (p->check(qobj, errp)) { + return -1; + } + qdict_del(qdict, p->name); + } + } + + /* warning, for now */ + if (qdict_size(qdict) != 0) { + error_printf("spurious capabilities\n"); + } + return 0; +} + +static int check_pgsize(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + uint64_t pgsize; + + if (qn == NULL || !qnum_get_try_uint(qn, &pgsize)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_PGSIZE); + return -1; + } + return pgsize == 4096 ? 0 : -1; +} + +static struct cap_entry caps_migr[] = { + { VFIO_USER_CAP_PGSIZE, check_pgsize }, + { NULL } +}; + +static int check_max_fds(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + + if (qn == NULL || !qnum_get_try_uint(qn, &max_send_fds) || + max_send_fds > VFIO_USER_MAX_MAX_FDS) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return -1; + } + return 0; +} + +static int check_max_xfer(QObject *qobj, Error **errp) +{ + QNum *qn = qobject_to(QNum, qobj); + + if (qn == NULL || !qnum_get_try_uint(qn, &max_xfer_size) || + max_xfer_size > VFIO_USER_MAX_MAX_XFER) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_XFER); + return -1; + } + return 0; +} + +static int check_migr(QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL || caps_parse(qdict, caps_migr, errp)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP_MAX_FDS); + return -1; + } + return 0; +} + +static struct cap_entry caps_cap[] = { + { VFIO_USER_CAP_MAX_FDS, check_max_fds }, + { VFIO_USER_CAP_MAX_XFER, check_max_xfer }, + { VFIO_USER_CAP_MIGR, check_migr }, + { NULL } +}; + +static int check_cap(QObject *qobj, Error **errp) +{ + QDict *qdict = qobject_to(QDict, qobj); + + if (qdict == NULL || caps_parse(qdict, caps_cap, errp)) { + error_setg(errp, "malformed %s", VFIO_USER_CAP); + return -1; + } + return 0; +} + +static struct cap_entry ver_0_0[] = { + { VFIO_USER_CAP, check_cap }, + { NULL } +}; + +static int caps_check(int minor, const char *caps, Error **errp) +{ + QObject *qobj; + QDict *qdict; + int ret; + + qobj = qobject_from_json(caps, NULL); + if (qobj == NULL) { + error_setg(errp, "malformed capabilities %s", caps); + return -1; + } + qdict = qobject_to(QDict, qobj); + if (qdict == NULL) { + error_setg(errp, "capabilities %s not an object", caps); + qobject_unref(qobj); + return -1; + } + ret = caps_parse(qdict, ver_0_0, errp); + + qobject_unref(qobj); + return ret; +} + +static GString *caps_json(void) +{ + QDict *dict = qdict_new(); + QDict *capdict = qdict_new(); + QDict *migdict = qdict_new(); + GString *str; + + qdict_put_int(migdict, VFIO_USER_CAP_PGSIZE, 4096); + qdict_put_obj(capdict, VFIO_USER_CAP_MIGR, QOBJECT(migdict)); + + qdict_put_int(capdict, VFIO_USER_CAP_MAX_FDS, VFIO_USER_MAX_MAX_FDS); + qdict_put_int(capdict, VFIO_USER_CAP_MAX_XFER, VFIO_USER_DEF_MAX_XFER); + + qdict_put_obj(dict, VFIO_USER_CAP, QOBJECT(capdict)); + + str = qobject_to_json(QOBJECT(dict)); + qobject_unref(dict); + return str; +} + +int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) +{ + g_autofree VFIOUserVersion *msgp; + GString *caps; + int size, caplen; + + caps = caps_json(); + caplen = caps->len + 1; + size = sizeof(*msgp) + caplen; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_VERSION, size, 0); + msgp->major = VFIO_USER_MAJOR_VER; + msgp->minor = VFIO_USER_MINOR_VER; + memcpy(&msgp->capabilities, caps->str, caplen); + g_string_free(caps, true); + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, 0, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + error_setg_errno(errp, msgp->hdr.error_reply, "version reply"); + return -1; + } + + if (msgp->major != VFIO_USER_MAJOR_VER || + msgp->minor > VFIO_USER_MINOR_VER) { + error_setg(errp, "incompatible server version"); + return -1; + } + if (caps_check(msgp->minor, (char *)msgp + sizeof(*msgp), errp) != 0) { + return -1; + } + + return 0; +} From patchwork Mon Aug 16 16:42:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 314C8C4338F for ; Mon, 16 Aug 2021 16:47:48 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id B670560EFE for ; Mon, 16 Aug 2021 16:47:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org B670560EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:35634 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfm6-0000hm-PP for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:47:46 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37048) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhb-0000gj-Oa for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:07 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:47136) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhX-0007w9-M1 for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:07 -0400 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGbX1d029459; Mon, 16 Aug 2021 16:43:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=W4lpY0dj94sm3O/uZtUx8TQVK8x1Yaa9DO9CP6I9Zno=; b=rOEfmXBrKFRL2uR+LtmrtLsqJkAwmzNRbcvS2rx5TEtmWq2Lim3XXel0H6ViWxxNoGRU 7u5mFIRYaf5B+ky/iqMLQBZGmJL1qc3nkR0Nbrz4+guwt5D/N1F9caXGQ8fyliar4VzK i356gjkgeVCZOc0+xeEBb0w5XLa8DN+pptfOxuzdrDSdcSBE72mWkC3ukTfIdUGwQIrA UVGera3pmfm2ZQTCEMY4zpE0qGIZKO8aBxTQ0lz+VN9DNBm/nUsbnCSgrunQdHPWtrRc OemD7sxe2qRpNXrnpcX0D6ptxtKdeJEaOk28E+OhsrrEWAJCeycy5mW3hmuNocUQb6jl ng== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=W4lpY0dj94sm3O/uZtUx8TQVK8x1Yaa9DO9CP6I9Zno=; b=fg4ekc9a9Kxh50Z/WWI55wQwZBP67v7OWLkuEbFOzKlZfEtRyM6DTiSidoJ7zvrAZL8H 7IUSxRq+zP4rez+gZXmWruyVmhMXUxkkG9VqoO+moyzQH/XSldjEJ1YWOxVqU/2W9Gnd rbYIqScdQ6ldp65JpTQyPSHEs4hK2if0pGeiuzBXy5JOVr/HhOTVZesKhUT0Ly9puolR 3kBgOJO1FogNBDUK27uvP8AbpR11K/MQVc2lJKWUrHQXRJehk/c1eKkr99UInt2chvkF XuQ0lZDSqmdhh1DdaWCJkM8p8ZcWKecjYhQul9cM5EKIoFk15IaKVlfYm0ny1Lut1yOJ fg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3af3kxtg0s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:01 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeo5s159237; Mon, 16 Aug 2021 16:43:00 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by aserp3030.oracle.com with ESMTP id 3ae3ve34y4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:00 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=DopNxm+FCOJnSi6HRPFSdqVLFseRKDt3CoIC17kjXkx5MHOW7AJWk6OPukeqiwZNTv8sYaPazOXmcWcn/dXAgX5BrtaOMfWy3bX+0uxKnSD6GleCHqPhmovhrK6AabtvLUDQkBNDKBBVNSnw1J1fW3HCimXWscEgpj30tgZ2KIf1HuXT/BP/JnEFPCO3OJGIls6rreIEkiKMPgyVaOs1CCgj5BAeEB6bs7ofEs6Samzr59TY7mjSi8YthRpHZOLv7albjZqZage0SLGvHdjo6th/LKN8osDtuztA534vFtHgCr3p/0NtU8V4o0tIcqooherZMHnAkYytyVATwrHgXg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W4lpY0dj94sm3O/uZtUx8TQVK8x1Yaa9DO9CP6I9Zno=; b=NNd8Du8MyoAc0a+zPXlb+9aYvY8TTGac+hq98uxlTI/E3mB1yl3+XWGdr5RKzMtE6AW22V7iecx3Y984YTljR06pL5JWm47NEQiq0XWQcmsaow3D/egR0cCcfPlIvtlYY0i+kcMLwALKOnOXVh86nMhaQeyIxYCC61X3ldL+sDjaigqvUsNSIQoRDcQOdMPVgFUmQigTpcmDUKszcoZeAkJXxyI+UrNCZykofY7m8VK6GjijBb4ZP8MqJovn65gqhU5VS4Jrdw7FVxFF6j66tux90jlHIuk4aCcfPfRqKo7aJsVKAHSc4SsTnHhVXytfL4Vi1zN9OCP9//FMjdGJCg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=W4lpY0dj94sm3O/uZtUx8TQVK8x1Yaa9DO9CP6I9Zno=; b=OlY3CYU2w5h+KxDhg06fp/DV1p0vV4PNTg37fXLsWfSh/ue4aEn8Mb5RVOHKF0tTOzXZW+Hgm2rjQ2W4fpLm6ldgZ3SD84oYpcJM0QBCKzvVgpKupOOHHDZ4m5oakPgQdYozbTWSQHTUekJ2aFbiJWi1Wf+8xtDeXdEAcLrvTs4= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:42:59 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:59 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 07/16] vfio-user: get device info Date: Mon, 16 Aug 2021 09:42:40 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:58 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 620908f6-de42-4fed-f59c-08d960d4e7f4 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:669; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: VrleL+Y+SkdABFTZqjAgI8vXtgjWZRbBS/nYW29S3oo8+sal4gyLT53wrmIV1QAkbs54VOGxNqRenbkpBkC3zyGe1mjOd6puTgP1MgO+U6hSzv5qQkPWEUbEjJc6MKxIuRtznnl+4XS66f0x0grHGNfb3/cDRcYv0zrbzKgu6zBqjevk729jmRviBU4yOb5lin0HoiDjEw1kpNrhLnSfiIkeOxrebNP/GZYKJYkgf7x0gfeaYetVLyRY44q7vEDIfbSm7L4wlHKL1a4ChYzwfATBVYTk5IgBaYJSjFK7G555SLqBWMlnf1Lh9xesx8ZtYCdMRBpqxM8F03KCBO5vmf95/XNapmhb/47EV9Ne7leI/M2Sqp9cTKrOOf+LNrNbSfAvbjcqaXQSgkwYAIgBLBM/gpQKtpO5iRLR9OhsTJYmLGptI0dLpfrcQsqjQGTJUeqdxu8k6iQDOHS7TIHiP4Evs3HXeOI7AXQsWn3PKdMjS5eNd1l0+DbrlL/ypdXwSXlQaT9QE1lRINVcDBLoF4F4taX2VRSNtPQoYcHtVxlu1U9y1L4th8cb60aaCUNkpiW8Zi9h5EBmvkey+LtuLLuoke3JJHIx4z4xe5W95xH4xMCPtPxbh79WX4e0tVN9cDVD+i4exeoWQmwfNoVIrA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: +1mzodtN4htY4PE4CWwzi8JmQTXuWGaI1qeuTskF8g2XI9NeTTeDe3TiR8SqkoUO/ttbwAqI5lhJeM80hvFsmzcKNBkamHkCVP+40nn8Yvwat9SsNcz40MVe20g7RdDj91CKhzdIHwCcW8De8nYSEtTS8oW5IgnCDFnfbUtolXMXAfL0rKgthk/tgLjT+SWmIAB8S7a1F0U3EENUGm1B9Qfmo8V/SOjdEi6nxc+Maow0VDpreuKCaKV95SaWXWJimuy5r3EmbRbnYZnfAABBxfhNKlbuAp7IVV47JWx03KRGRvG8P20UjNNKoT1zI442I1eM+7I6Bd5miymDmRmI1KxCoJkPkC/9M2z9WsUlvnjbcbW/DAEWmN4LwHNtsUwEceBB+bT2ZmvegoXa3S5h7FEed4rIFYmqIl20DKxgf6kj4olZoDFqpUv2IFS1MgV+EBcHr7dnCBSvsE5gkbkWQg2CHtTJDPJ/cJaYWPg6JqP3eWoe6zSfFXCm4kE19kP8HLTRA7TmXgcCex/jl/yZ82PXjEW6+Ey8aEnUxmiJycSOi0FVeU0GihFjYuImsrGGKHYon995DmiPzxnQCHGAfjiN79xIViACRTRY3NGm3JMU7OsV2WkOIQrcpTAOlNl4QST9Zbw1cQNdZ2K5Erv+NbcfgI3G0KT+XzkdVfW3K6vVipAZzJ5uJEG0X6pbL+tlMREv2vOS/SWXlUH+zLBeScwx26iSya8s6o16aI+A5SW64wrU2d1uUQm3Z5FuNS6uoAjfya3PB7W0T3fi+FSWN1QGaYPKr6aOOu5O3EGRCAEOvXGQ4ssoWqNK4q4zetTwFarhiw7QpovWh7Mrd5tOAgCR63Ft3AqYDk8Izc6b0fmOBzZP0bFK1l2KetMSC5vVEAUXG0k1xYHVeyQVfLkKVWW8QOA3aA9qA6V7VsDM8vk333zzCDFuPyXza+sXImwpx+EuM6aOSgbboESg4J/m+fbX3ZyvhaRjBncw6wsuju+enp+NO1KfhbB2bsOsupwITkxJy07CHZM2zdVNUA/mL82tjgiGo+YCO2HlfDCLi871tKWUI08J0lt1WtPPlF1GA2t5LxGQCIk5Iy77Y/sPT4iEhZSuQZeYemN6EjvMDumY6V+K/W/t30ztsS44EYBts+cOdnHSd/jgTjNm6M0GBWmcO/+mhqLApvD48UFVzGGLLzuKVd78EBj3kYats2uNYXxXCIUhn7EdvEqokG5xowl2LS83fVu6Wxgf5NdJwnxdS+aPjdInw232xOEj48DoP1bNIn7A28eV97df7KAp725XG23HFMS0Sjvm8xbdgsQLuclSNkFPuFplmoxVKpz10X+kEqyl9mUE/QgrDNeMkhF2dQIr5b8PS7CxIRYJmcaFGcUWgcA5xD/8ScUsRdrL X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 620908f6-de42-4fed-f59c-08d960d4e7f4 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:59.2008 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: xx4DGMYG9ilVo/+o0PKhIyy09Eqhl7GeBJOmj/Th4vHWNavBgMpnFypqoVsi2JQ0nu1qh6XeYl8Sw1KWc0D2eum+szpu7tEwKr6w6DFmygk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: ZhRxo4LBAlc5gDS3M7WOofluhnF4Cmqk X-Proofpoint-ORIG-GUID: ZhRxo4LBAlc5gDS3M7WOofluhnF4Cmqk Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 13 +++++++++++++ hw/vfio/user.h | 1 + hw/vfio/pci.c | 13 +++++++++++++ hw/vfio/user.c | 20 ++++++++++++++++++++ 4 files changed, 47 insertions(+) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 14b762d1ad..13e44ebf1c 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -82,4 +82,17 @@ typedef struct { #define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) +/* + * VFIO_USER_DEVICE_GET_INFO + * imported from struct_device_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t num_regions; + uint32_t num_irqs; + uint32_t cap_offset; +} VFIOUserDeviceInfo; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index cab957ba7a..82044e7e78 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -71,5 +71,6 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, void *reqarg); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); +int vfio_user_get_info(VFIODevice *vbasedev); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index eae33e746f..63aa2441f0 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3369,6 +3369,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIODevice *vbasedev = &vdev->vbasedev; SocketAddress addr; VFIOProxy *proxy; + int ret; Error *err = NULL; /* @@ -3410,6 +3411,18 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; + ret = vfio_user_get_info(&vdev->vbasedev); + if (ret) { + error_setg_errno(errp, -ret, "get info failure"); + goto error; + } + + vfio_populate_device(vdev, &err); + if (err) { + error_propagate(errp, err); + goto error; + } + error: vfio_user_disconnect(proxy); error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); diff --git a/hw/vfio/user.c b/hw/vfio/user.c index e89464a571..b584b8e0f2 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -714,3 +714,23 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) return 0; } + +int vfio_user_get_info(VFIODevice *vbasedev) +{ + VFIOUserDeviceInfo msg; + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_INFO, sizeof(msg), 0); + msg.argsz = sizeof(struct vfio_device_info); + + vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + + vbasedev->num_irqs = msg.num_irqs; + vbasedev->num_regions = msg.num_regions; + vbasedev->flags = msg.flags; + vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET); + return 0; +} From patchwork Mon Aug 16 16:42:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C737FC4338F for ; Mon, 16 Aug 2021 16:51:44 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4A132604DC for ; Mon, 16 Aug 2021 16:51:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 4A132604DC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:46164 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfpv-0007j0-EH for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:51:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37108) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhd-0000l1-9E for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:09 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:47936) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhX-0007x8-U7 for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:09 -0400 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGbZUE029485; Mon, 16 Aug 2021 16:43:02 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=Z+HYnZBTg8/iVO2ADTDXw+BbrtUybeoKYSv9IN97OaE=; b=An1LXz2+MAiGckgOlh/4RWEusKrzS3FrQhgkZc87qCen8WRpKdQ5kECu0oJiT8EA+2AR 1WX/T8IPqEtK+7S5BQufCKI0DlQTOPJjpuzjK56d9VclbFELVRDZ1ueiB56t+A7udLwZ GI1wnnEdXRAxoNlOjcK889/wh8KkEzm/mSrbcsoOqAk+rQyB4l08cGmMcXsgQTSmp2ME Fy8e3xPqvtvyg6SasGig/JksxfLSyTmKxqcq8BKIXMIhKJb29Ngd/AtubH3Q/DoS766x puisixNrgg0jjHtIEZPd/cGCYFzb4ubCTZR86IxKWEvGpSJ+R1znUBLLGTkFGNx2/CqL TQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=Z+HYnZBTg8/iVO2ADTDXw+BbrtUybeoKYSv9IN97OaE=; b=KMqLPWRGEGCKd9GkW5uz+/4cNOZZ95bE2SpZXMk9sQXcQ6kEK13sjsIET6cVxFksN9h5 n6cFdTJiLv4OyUgiR34hT373DV5WMR0AqeU5XMI3B0KmvYXK6I+sRxMNFarDZOsqYsaV QENKmuboY6Y690b+xgwniFSw58uPmt5T9GiqbMxsz17KT7/imFzJGg/FzCIuQn5pqUNy Djz4oj/YWcujmjPnkYI4MZnzx+eOThQhmRCV7vEx0FYApOmaVu7YyU8Dzt01yrvt1U9Z m2MvfN9W9iRYx+TBFZX795VUA3MrZHDKUvjbgdOsCOL4GAS14GSiMGz0spw4UP4Gkgrb hA== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3af3kxtg0t-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:02 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeo5t159237; Mon, 16 Aug 2021 16:43:01 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by aserp3030.oracle.com with ESMTP id 3ae3ve34y4-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AXz7yXqZ+T3fFrewj3I2eFbsnwuHY1DugQ6iiqCgzIE0fJHmt896QmWxk8TMtsI7e4ijEJPotbczkD/wNJWHpfN5YpMdQq1nfkxgYFFGbVwnFGrH4+Jb0uv1/gCllfNO8ZZBMEgs4J5Wz84iWAXHrWcAfCY+y9ix0nWj0qiT4Ojte3dzNVA3/zGhQ0aQzJSj5da0/wVHxEQ9BN8Phb0mMxy4jq2X2MjXxJK9PuQUxya2ROcxb6Uzm7IWNnutmWYlMq8iBoZNA/dBUNBvY4SlBS0k2gMtQjilQyAaadkrCfs2sKqk14n+5Uxdq/0MeHTdn2Zx2Lwwuopr/fhFYAeYWA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z+HYnZBTg8/iVO2ADTDXw+BbrtUybeoKYSv9IN97OaE=; b=anROrCoSKPhxFnbyxpIfMM2aHxhqbYM1vpH4YEFxeaYbLeraizAKLaDB/6YtgWadPC8u9MWFL9IyPkpd6/mO2wz1NuUMnzZjOcldtOzaPSPe2ums/WThu0evWGNIy84dJGfiiJNLUnPPtVAUcYqUGz3+rq6ptK8dvqlfHEpQICsyG/JnpFm6QkgjPicbC/+nXSePKDPNsR5O+I3hzUfDgcn8XGSfGLZu3T1u57lGp9KGiqqnMEPEEb/lhclms2lPbzeXakYrSoqI/JY87Hf2wRpD5qvtMw06ep0Q5bdQmfq0J7ZFpUuSQ8+nAv7dFlY4G5z8ppXIS8AObfO2ue0wxw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Z+HYnZBTg8/iVO2ADTDXw+BbrtUybeoKYSv9IN97OaE=; b=FS1Jr5A1v99ZmEHv2KHmgEa5oQYAKMNCLrsF4Rz/gVjUPoTcqnRW/J5ab9miw5vnOgCmpMQQwKyQNasUiYNKeDaDvQHsQa3hnDvaNZa0dGnh57o2nzBsB9Tc/9//lBm4NTYQpJ5tiskePb5yhICiJycC43eNXSPQt6WOnPLUCUc= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:42:59 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:42:59 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 08/16] vfio-user: get region info Date: Mon, 16 Aug 2021 09:42:41 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:42:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 0448f41a-ba4e-4662-170d-08d960d4e846 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:324; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JEZoHPer/nkBxQJisYyA42A1gmyEN/4ZTZNC9ZlWZXxIKagQjjSnx8EXrq8NPiJyHsAv4LiyHWM72xIVRs/lDkm/stOGOuZpTt9Gg5E9CbKWld4oARZqBZbk+lNTxxFPj1ad4oDfb+wBVxRq7WMk4BKje899SzgqMKDiler80eH8uLayPNysfnW+dSwjCKOsl15Wqkutp/duf5/O0IiXu+bY+UaLOF/yBijNF6AMiZ8HIRQT13B3He3BvuH4I11ywCUAKKrKaWp22k4icMgMOLX2HDzzeHRjo/avmrP73ogvud6gbIGbhqatRk86FEOuOlgVWRpZsGOF9uZpsvMZPurmn4W4xI/vWfWeWDvw+kXldCGKgy/nGZzutDPMxwmCbFN4iImzgCvuJYtiPwOeOda+1sARVX2hmbP2oqvG3sUwDERBM+WJczv1Zg8kHIZOOxyVqvB50x+QG3mT1N5SeZye56mOKtBRjwyGI7ESyODM/FGsvv11Ujuj8SGQV7wWflGAx2c95bOmVkj5eE7Y5+Y51UZ7uYbtKlog6iDLT1x5oPu/tZLWgATMQjboNQ1xc0G4bZhMoXVW1bQbcWnXgnHL/WwuMaeGqm7eJK8ggA+erLNgkXSrVYMMSbr7j47SH14iZVwOgZbvYFKFqLu4HQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: ds4ImlLk4inMKv3nEv/3rd9sb777gf54GN8VDzbO9KRJXtFoMXDnIo/OdaL4ko/JwK4lwrl94QZKxgK8zXKSgCDdZjcDLE1dggoAlnh4b/suug2n7kxZj0ogccihSdxjhgc6iZnBmBAULu/4dLJO++53BQfUd46fXNEyTBU4mMSPg0RI47VwB0a/YMi8yrvbci3zLT43EU2I++df8zeO73UgvD+LBEeJ0BbTmY30gcMXgQ0s3TuJuoE/ncQ4D3BG7bjx41r64ci+dRxYUhlt07bYCjly43w8NBCcYt//knQjc8KlEuQx+BKOy633Ewk7bBV+Cd50RtZoizhD+R5WkKyep1yG3mlGoPE5AhWro0k9t0p2OLBL6eRWqHIkhInozHxZfbKhXe3/FPkJLd0ho+XHJP+aaLFxoK7Go7p2mL7D/zFHIb1lAKyAGFUnHchNKR0bE2IXyCaEKBC0mQ88CWmmGHEm6eMCX9ny+YL8DUnIh7tVWgRh2F1o5Mub3VdT5+SptYT8g408JHy3P8cKcYwnNUNhNmNtuhpyB08XBm1yBz2Sm6iOwjMBb3/uiWjS0PtkBNhFla9Ec9MEgy4cuxDOUxX1DP9NFKffp7oKK4ITRGE9/7IQO7lFMPAtVryrRyLP8h9OUUCZaQmAbYZwxiUKwFc7TXR/BwssPWOuAh6MxVyY8FKgSjtoo4bUEbkaFn5Yhxxcff0tumTwnI81j31QdCholNjh48lHdd9JxmfLa/yLzDK5VO6hkwBlU4WePAgn4X/LDQctPWeeciAC+Tkrx+9eoBseZv10iI/7LoKnklrSsm+iabBuQ0sW9AN8Au1258F6Im7iIPFKuqpSmPa2qTc7KS/uDZdwFexIQLkUWmfrJa10TrziE1eR20dpTTlqF15iD+Sr1O2WtK2lORbVbisSyzl26nmVLBHg3qr9XN4rs/+bziNK1K8gAO8PT8dEIbslbLo9xY+JUmx9EdmqUvOxw74/dJ9yTmHM227tYD0qpbrEClOnCVhyAi2E8LJqH103LHwrBDm+Lj7IyC4vaAnciJLKNMr+cE1DAY84RdUQWLxmaLwVlEc/J60slnd8tMsiTWngHaQTeP76QloKqZsTC3DNLo+x6BBzoVveiaGgq6qi7yss6/CUo6cZmfzB1My8Boc6ry+W2uOrPTQFYnTDa5Ls91hOxKT4BYMgr90h4ey5ZolJ8t69FKaKNsC0NmeAFFxfJ24pfH3N5VkoSmscLDK/uhP9QiQ20mhgoKfvIzXZ01bPYLhEDEFZiFK8a+i0PHm+Z2UdddmZyAYvVuwSlBljENs3PgYijtCjgdvu7mnrIwMR8G/kBBXz8zdnpAaD67drUa1Elkzy0tyz9KskxQ1JGQpMiy/gLKa2Xfh1QzZPRkG++5wAne+F X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0448f41a-ba4e-4662-170d-08d960d4e846 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:42:59.7055 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: PfCVIzAyztpbmOgLjV8helwtS4QD1Xn0IFxDRQPeJo3fDNyiJKTZMcqCr+BH3oUVnT7TnyXY4FY104gz1MC6XxkruzPC8PgvOifmeUyUq64= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: wsVNoaoUIxlHJvBjLZHiSh8-jXPNQxFi X-Proofpoint-ORIG-GUID: wsVNoaoUIxlHJvBjLZHiSh8-jXPNQxFi Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 14 +++++++ hw/vfio/user.h | 2 + include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 76 ++++++++++++++++++++++++++++++++++- hw/vfio/user.c | 33 +++++++++++++++ 5 files changed, 126 insertions(+), 2 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 13e44ebf1c..104bf4ff31 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -95,4 +95,18 @@ typedef struct { uint32_t cap_offset; } VFIOUserDeviceInfo; +/* + * VFIO_USER_DEVICE_GET_REGION_INFO + * imported from struct_vfio_region_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t cap_offset; + uint64_t size; + uint64_t offset; +} VFIOUserRegionInfo; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 82044e7e78..f0122539ba 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -72,5 +72,7 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); int vfio_user_get_info(VFIODevice *vbasedev); +int vfio_user_get_region_info(VFIODevice *vbasedev, int index, + struct vfio_region_info *info, VFIOUserFDs *fds); #endif /* VFIO_USER_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index f43dc6e5d0..bdd25a546c 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -56,6 +56,7 @@ typedef struct VFIORegion { uint32_t nr_mmaps; VFIOMmap *mmaps; uint8_t nr; /* cache the region number for debug */ + int remfd; /* fd if exported from remote process */ } VFIORegion; typedef struct VFIOMigration { @@ -145,6 +146,8 @@ typedef struct VFIODevice { Error *migration_blocker; OnOffAuto pre_copy_dirty_page_tracking; VFIOProxy *proxy; + struct vfio_region_info **regions; + int *regfds; } VFIODevice; struct VFIODeviceOps { diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 8728d4d5c2..7d667b0533 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -40,6 +40,7 @@ #include "trace.h" #include "qapi/error.h" #include "migration/migration.h" +#include "hw/vfio/user.h" VFIOGroupList vfio_group_list = QLIST_HEAD_INITIALIZER(vfio_group_list); @@ -1514,6 +1515,16 @@ bool vfio_get_info_dma_avail(struct vfio_iommu_type1_info *info, return true; } +static int vfio_get_region_info_remfd(VFIODevice *vbasedev, int index) +{ + struct vfio_region_info *info; + + if (vbasedev->regions == NULL || vbasedev->regions[index] == NULL) { + vfio_get_region_info(vbasedev, index, &info); + } + return vbasedev->regfds != NULL ? vbasedev->regfds[index] : -1; +} + static int vfio_setup_region_sparse_mmaps(VFIORegion *region, struct vfio_region_info *info) { @@ -1567,6 +1578,7 @@ int vfio_region_setup(Object *obj, VFIODevice *vbasedev, VFIORegion *region, region->size = info->size; region->fd_offset = info->offset; region->nr = index; + region->remfd = vfio_get_region_info_remfd(vbasedev, index); if (region->size) { region->mem = g_new0(MemoryRegion, 1); @@ -1610,6 +1622,7 @@ int vfio_region_mmap(VFIORegion *region) { int i, prot = 0; char *name; + int fd; if (!region->mem) { return 0; @@ -1618,9 +1631,11 @@ int vfio_region_mmap(VFIORegion *region) prot |= region->flags & VFIO_REGION_INFO_FLAG_READ ? PROT_READ : 0; prot |= region->flags & VFIO_REGION_INFO_FLAG_WRITE ? PROT_WRITE : 0; + fd = region->remfd != -1 ? region->remfd : region->vbasedev->fd; + for (i = 0; i < region->nr_mmaps; i++) { region->mmaps[i].mmap = mmap(NULL, region->mmaps[i].size, prot, - MAP_SHARED, region->vbasedev->fd, + MAP_SHARED, fd, region->fd_offset + region->mmaps[i].offset); if (region->mmaps[i].mmap == MAP_FAILED) { @@ -2397,6 +2412,23 @@ int vfio_get_device(VFIOGroup *group, const char *name, void vfio_put_base_device(VFIODevice *vbasedev) { + if (vbasedev->regions != NULL) { + int i; + + for (i = 0; i < vbasedev->num_regions; i++) { + if (vbasedev->regfds != NULL && vbasedev->regfds[i] != -1) { + close(vbasedev->regfds[i]); + } + g_free(vbasedev->regions[i]); + } + g_free(vbasedev->regions); + vbasedev->regions = NULL; + if (vbasedev->regfds != NULL) { + g_free(vbasedev->regfds); + vbasedev->regfds = NULL; + } + } + if (!vbasedev->group) { return; } @@ -2410,6 +2442,24 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info **info) { size_t argsz = sizeof(struct vfio_region_info); + int fd = -1; + int ret; + + /* create region cache */ + if (vbasedev->regions == NULL) { + vbasedev->regions = g_new0(struct vfio_region_info *, + vbasedev->num_regions); + if (vbasedev->proxy != NULL) { + vbasedev->regfds = g_new0(int, vbasedev->num_regions); + } + } + /* check cache */ + if (vbasedev->regions[index] != NULL) { + *info = g_malloc0(vbasedev->regions[index]->argsz); + memcpy(*info, vbasedev->regions[index], + vbasedev->regions[index]->argsz); + return 0; + } *info = g_malloc0(argsz); @@ -2417,7 +2467,17 @@ int vfio_get_region_info(VFIODevice *vbasedev, int index, retry: (*info)->argsz = argsz; - if (ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info)) { + if (vbasedev->proxy != NULL) { + VFIOUserFDs fds = { 0, 1, &fd}; + + ret = vfio_user_get_region_info(vbasedev, index, *info, &fds); + } else { + ret = ioctl(vbasedev->fd, VFIO_DEVICE_GET_REGION_INFO, *info); + if (ret < 0) { + ret = -errno; + } + } + if (ret != 0) { g_free(*info); *info = NULL; return -errno; @@ -2426,10 +2486,22 @@ retry: if ((*info)->argsz > argsz) { argsz = (*info)->argsz; *info = g_realloc(*info, argsz); + if (fd != -1) { + close(fd); + fd = -1; + } goto retry; } + /* fill cache */ + vbasedev->regions[index] = g_malloc0(argsz); + memcpy(vbasedev->regions[index], *info, argsz); + *vbasedev->regions[index] = **info; + if (vbasedev->regfds != NULL) { + vbasedev->regfds[index] = fd; + } + return 0; } diff --git a/hw/vfio/user.c b/hw/vfio/user.c index b584b8e0f2..91b51f37df 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -734,3 +734,36 @@ int vfio_user_get_info(VFIODevice *vbasedev) vbasedev->reset_works = !!(msg.flags & VFIO_DEVICE_FLAGS_RESET); return 0; } + +int vfio_user_get_region_info(VFIODevice *vbasedev, int index, + struct vfio_region_info *info, VFIOUserFDs *fds) +{ + g_autofree VFIOUserRegionInfo *msgp = NULL; + int size; + + /* data returned can be larger than vfio_region_info */ + if (info->argsz < sizeof(*info)) { + error_printf("vfio_user_get_region_info argsz too small\n"); + return -EINVAL; + } + if (fds != NULL && fds->send_fds != 0) { + error_printf("vfio_user_get_region_info can't send FDs\n"); + return -EINVAL; + } + + size = info->argsz + sizeof(VFIOUserHdr); + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_GET_REGION_INFO, + sizeof(*msgp), 0); + msgp->argsz = info->argsz; + msgp->index = info->index; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, fds, size, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + memcpy(info, &msgp->argsz, info->argsz); + return 0; +} From patchwork Mon Aug 16 16:42:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B007BC4338F for ; Mon, 16 Aug 2021 16:44:46 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 25E1460EFE for ; Mon, 16 Aug 2021 16:44:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 25E1460EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:55616 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfjB-0003bS-AG for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:44:45 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37128) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhd-0000lw-O8 for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:09 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:48514) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhY-0007xJ-AY for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:09 -0400 Received: from pps.filterd (m0246627.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGbwOP029362; Mon, 16 Aug 2021 16:43:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=grNf/V8qZcO6D7heJPv97ZzLoNAJ/jpkLVPA5jc+de4=; b=Ezc3Bnk9jFKsgzi4fhgZadx1A/Gxf3tVoyqxN7ZhmH1khtn0FIktobxcImiistCSov7P 1m1lqfzhwjbLMBvyiqSZVmzLlXxr5gGg0Yuv5N+BwCNfEOgnsXLoteTi3kgReaB0KDiq 1Q8+RpMF/+t105Borh4ZLCy1SoGyLbNTLXhqPlhUmaaOl/DzCsRF6pY+nsX/cJaGeYlk Nha8KgYVEVwOv3Ymj0YaTPA9zRKninsNFRGCcip+amLeLWpbFWo5lMirLHCJzgZhvvoi kQuaW6xC7bPIYZGQBGxG362y5LktW9NF7eVD0SEpPTHWHlnIMsWaWWKSqnT9YfdRFV7c tg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=grNf/V8qZcO6D7heJPv97ZzLoNAJ/jpkLVPA5jc+de4=; b=UFoYG92JwuxOVEeAHAg3WvzceEta279i+yxdXh8XS2Vjpu6ViQSb9q+X+9sJdqAikFmL S7nqFhs5hW3CRbDLdAIgS54+nFWVd9Qei9jN2xnKQENLloBUHGUu6zj+gOpeAH38GY1n +hHajiIBvmQpQ0DF5KpYuunPbXOrEnyhIyCceRJw13l8C8lqjXevimQLdFA7MsOYGnA+ 0Qa4VOHOsbCb06/GJSz34WmWBS1ECTKdQ2ZiNqF9Ti74qF+mjB+SDdzzOyBO/Faz3NjP v+hF+2ctdJUUYSS0Lvwbzu1+YRicl6+n7mCsu3uzIyEtK0Aco0P1KsrtZhY0rAHEXduW oQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3afdbd1wrb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:02 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeo5u159237; Mon, 16 Aug 2021 16:43:01 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by aserp3030.oracle.com with ESMTP id 3ae3ve34y4-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:01 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CpKORUEjqDFoRNNo8cRksLU45jvTKYwrGbjRex2sz93+JJ62GFtBVvwWw1FOk3Cizqk92dIoYaX/Bd9eomRZFZD8TSi7alutYYKbSCftikbM9qMi60jtw5Hf2DuUqaTKVu4A1uSpBCpGyBXRePd80dCyzvfF0GgXzqcrfonV01f14vez+YaLsuC55BMYoR48PPWQwDNWXETuHaU4zEEO973uSk6apfJ17HYiuMptIFx+KcPMld6fJmsLHSk+VZuSRCzGPrNWhy54XUbHcfPttc+kyhDO/0f5APqhjkv8GdGvpLkfmc4b5FqkMgN0fffT0gyhjUpNCEqYAgT4Aecryw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=grNf/V8qZcO6D7heJPv97ZzLoNAJ/jpkLVPA5jc+de4=; b=UjS1JxM/NajHMLvL9jptt9wzr8I6BkeXPRl0CuDsmKyZLf2+wmkaysXNj5S0QCLiCpmcqDaxUAlvbb2cHcSSGTxrUYTdtXoqxSU1+AFJYBfTMDSUvPBdiCzTeW33YslKuc780rITAGCNhj3boZgeaqle6Jfrv6DRNWluHvKiBOQKR1H4U0MJe72M0yutQ8Xeh6U8Gw87ekzDs597O70NX24HaiECcbXMmFH5j/tcZyUJHycLaJsL097iYUEnhUCX9rY8UxUCtYIOWKrghcoeSldP2ScnFqGz7uh9+S/y3Z0umdNOW8Lt9n1sAPjOELKxAZrOExInH7B+qi7C2c5VNw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=grNf/V8qZcO6D7heJPv97ZzLoNAJ/jpkLVPA5jc+de4=; b=piLV5m2GwP75nKOLEEIIhfAdj0YI3pfPKduKl8UWbOMxgOT7PvArG+WxQoyywYUnD0jnBgaK1sbBzFvghABew1XARRNJeKcy9UuZ+t3qpSpr+8RpucifYqEQgMbBts17zh2Rf/tM+GXhixK3ou4InmZfR/g/K3ak8x1kXtQAYuw= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:00 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:00 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 09/16] vfio-user: region read/write Date: Mon, 16 Aug 2021 09:42:42 -0700 Message-Id: <92fb16181e71a1c4049f9995294dbd7ff4270627.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:00 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 92af5206-e312-4b99-702c-08d960d4e8a1 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:78; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: AeAs5MlZgHgobQGGj5eXIWj3a94NW5KU03vl596IRFPi3Z2aU3nlOYmuhuWEZaLZab+5Cm8wqHZNjuI7E5Ri2IvnA52TZVdUeY8DdPoDQuoh8IRIhC6HlDx9INhuUvAJpIy9x+hhSzpffiSKnZtq29f41Kf0dQNsofIQQ0Xi3veIELIWOoVwkBIVXb0EkNWwd0CL/8I2sLbavz1cSK3p4CSJztsWV1IrOWwozO7WzfczEbauSWNl9CD/YuCyYUESEss+s4NA6b1jl6JBRIBiD8MAHIy/JB0WuPJE8ZQtHA30QxSxfyc1wCbEE4/DCUlugbdO38XxLzHMr6xD9ApRsPpkd7yNq27jUHrfPkrQkJwaI/3aXYtOV69+IYZKP6wZ/vUDoOEYwTcBXTC0fyd9djP/Ed78+LSzkYXeFOHfZHiwpkaFEaOWUfyRirZEK552tAFk8WpPxwWqQE10WyK9EvFaugCdh1KX8SqwMr8SnD0QZhE4b7sgha5/3HrH2VUYoYpTzWRjDrIPlZQzgmIy6wnTzQyA07fVmwAekm1HKR/NJ1CgAYwIUlu9b6G06Q8AgLjHFwUPsLMuIxwzNOr3hFVL+xqI7qwFX1eOcknNotiwP8OGCtbBoAIqFEZJavU2kVgWFAIeTNDSdVL23K532g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: YZtnBy2DDANasd37hsdW1D6Sut8F88gd9g47pFNuLNR2rhRxZrl//uiIm3NsLyjxOoJf2Tv46y9I/bgId6i07tq7HvDf9JUeWE95RJGh5oeuQ+x2seWfDOOHAmrGLFRKNULxgKu0ct0WiK9WW5R+i8ErIEURbfevGlli8jJbzGOU0yZSTDOdgL5XtixYUbI+stU2v0InPQat4wF/ToJlF2F/GAzfuyiDTy9e/mp30jQPy2lF1+ELge8FAl2ZSMmG/Hm78VlOW2it7+gWYmzxD2/cxhCFytmEszJ1UQs2zIXhX5xdDiwLiuCuP5q4Wx5oiL+RKUcFg9nvr/KL2lpB4rNx0dY7heCYuVvhK/dLBS5YUdnNQfVwI+y1cn6s+UrtQ5BqtUlH7o6G4g6PZ33DGAdFGO6ErF1NQ1BVK6Ue4FrQmqTOHrY2IDIcxhVyPwkcCYAEEnmCBKK0EQ3O4FvCOJWcAmOB0wUwuMNF4jYcGbeoR2sHrCs3/BnpeqHyJ1lhQwZJ2c4DIDrpP6babwrCnPY+vRZ6C/UUUjCLdRy6YhZGasFAlAwqJzWwLE+xPYf6/pP2TaOBY7qC0y9u9kzXME4R4wnYjKBmYWOaxqk42tY4D7tFSN1Ezinriie4BssGtMfhWoKT1eYadFyQy5RMCqLlPkqyxuSwp6/SV9R7U+YJksSe0+QPqlw0jKh8PwIATV3aefII4N2COs7JJQqqAyJoEiZgwiK2t+8sHnR7J2C4U2Xx4ergkJAyUDsXInMImBCSdLTKVm71cJkag9nHc5/oo6sJWwHbrvBlj0FwPYaGNos3pKGbYv0IxFrU4SBKwNLqRy6BduQjTyWHdPwof1u2kJcIOrhhzxiYuOkTcQP0laNIRK905EY0sIxyiYbfsJo8crdXPcmbY7zySpLQsjgGSmHN5WjvMI50Ni5tjSJEsgsV79RP5eFCHKBzgIlc01V75XDGzZ3zrKY3XAGD+UP+ZTeR1PSyK01Wte8HBQ/Uty+QZdo7Fbae1/FQMJfN5ivtnLrTgO3NxbQ9nXLMnYx2vzlCAMdihmjscL9umCjWUlT/I6mJstkUJExvo7oVPGqNm0Yw77bJSWcvDBwC1ZkMZ/AXAcl2vDZUlXYmsKgbuShgFHxhWG78JyRNOILKohBOb7I73k4Fq4+CpH0CiFgXqN7iLD7qAk7hQPT7wqTXnXju3szYHMJBTYhATxAVVUOrhx5qI2++ukC8+z4nCkT699r7YgcnX+QMmct5P+fhYYrcoh5vGbpqaxSUpT/RERdi4O9cIk+1KftuFPZLns1vn/9kbgqi+zzbn10jIhFg2/5pEQvK6XAmEx3QGjg0K0RWxoGngKGrfTeKLPaxhW8P95Kwak9X8AonX2MT/ARsZtXz2XB/PEMwscN+8deg X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 92af5206-e312-4b99-702c-08d960d4e8a1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:00.3182 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: RgdWpWy558yQyAoc41wRK+AVonTsXymTjusJ2Lbdio6M75oDT5xXJJkxjl6o/ulAdq/mSGsKlrmVktIblj6VnEyMBBc11IshDdTFn/rvOlg= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-ORIG-GUID: ymYekk42WEl1Z8yo9NPhEw2tR022hJBt X-Proofpoint-GUID: ymYekk42WEl1Z8yo9NPhEw2tR022hJBt Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 12 ++++++++++++ hw/vfio/user.h | 4 ++++ hw/vfio/common.c | 16 +++++++++++++-- hw/vfio/user.c | 43 +++++++++++++++++++++++++++++++++++++++++ 4 files changed, 73 insertions(+), 2 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 104bf4ff31..56904cf872 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -109,4 +109,16 @@ typedef struct { uint64_t offset; } VFIOUserRegionInfo; +/* + * VFIO_USER_REGION_READ + * VFIO_USER_REGION_WRITE + */ +typedef struct { + VFIOUserHdr hdr; + uint64_t offset; + uint32_t region; + uint32_t count; + char data[]; +} VFIOUserRegionRW; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index f0122539ba..02f832a173 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -74,5 +74,9 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); int vfio_user_get_info(VFIODevice *vbasedev); int vfio_user_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info *info, VFIOUserFDs *fds); +int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, + uint32_t count, void *data); +int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, + uint64_t offset, uint32_t count, void *data); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 7d667b0533..a8b1ea9358 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -215,6 +215,7 @@ void vfio_region_write(void *opaque, hwaddr addr, uint32_t dword; uint64_t qword; } buf; + int ret; switch (size) { case 1: @@ -234,7 +235,12 @@ void vfio_region_write(void *opaque, hwaddr addr, break; } - if (pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { + if (vbasedev->proxy != NULL) { + ret = vfio_user_region_write(vbasedev, region->nr, addr, size, &data); + } else { + ret = pwrite(vbasedev->fd, &buf, size, region->fd_offset + addr); + } + if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", 0x%"PRIx64 ",%d) failed: %m", __func__, vbasedev->name, region->nr, @@ -266,8 +272,14 @@ uint64_t vfio_region_read(void *opaque, uint64_t qword; } buf; uint64_t data = 0; + int ret; - if (pread(vbasedev->fd, &buf, size, region->fd_offset + addr) != size) { + if (vbasedev->proxy != NULL) { + ret = vfio_user_region_read(vbasedev, region->nr, addr, size, &buf); + } else { + ret = pread(vbasedev->fd, &buf, size, region->fd_offset + addr); + } + if (ret != size) { error_report("%s(%s:region%d+0x%"HWADDR_PRIx", %d) failed: %m", __func__, vbasedev->name, region->nr, addr, size); diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 91b51f37df..83235b2411 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -767,3 +767,46 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index, memcpy(info, &msgp->argsz, info->argsz); return 0; } + +int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, + uint32_t count, void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; + int size = sizeof(*msgp) + count; + + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_READ, sizeof(*msgp), 0); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, size, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } else if (msgp->count > count) { + return -E2BIG; + } else { + memcpy(data, &msgp->data, msgp->count); + } + + return msgp->count; +} + +int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, + uint64_t offset, uint32_t count, void *data) +{ + g_autofree VFIOUserRegionRW *msgp = NULL; + int size = sizeof(*msgp) + count; + + msgp = g_malloc0(size); + vfio_user_request_msg(&msgp->hdr, VFIO_USER_REGION_WRITE, size, + VFIO_USER_NO_REPLY); + msgp->offset = offset; + msgp->region = index; + msgp->count = count; + memcpy(&msgp->data, data, count); + + vfio_user_send(vbasedev->proxy, &msgp->hdr, NULL); + + return count; +} From patchwork Mon Aug 16 16:42:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2BCC3C4338F for ; Mon, 16 Aug 2021 16:54:45 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 908DF604DC for ; Mon, 16 Aug 2021 16:54:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 908DF604DC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:54032 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfsp-0004RZ-QO for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:54:43 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37152) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhf-0000qg-A9 for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:11 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:37958) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhY-0007xg-9l for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:10 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGZhse008102; Mon, 16 Aug 2021 16:43:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=D4ZqAyq4SE/ICj/ozzOgWBRxaqFlE2xHi8oF3tQZWe8=; b=vbwLr3kFKbTRlDuI64RscayhLij23ulaHPWrFiU3SPDQkJM2hb32IQhKq3NeJvFC4W7G vu2uGnbwHJs0kZAfwLa5ARhqXSDry+o7CxicBRgAC2OyCHhp/rV9EvLr9rN9sKn+2gau qYDbtjUpYXRdLAHeisz+9JqsftRUaSwueKTGO0iweEDpnsJgE20NdOmjFu7zD09QBUQM HiXZULaiuQNZgJgeVqEa1szfNhoa+rSrfLRdTbDSmyp58sE0tA8AkKRZdBDGfaVFFhM6 FE+6vuB9ljCseurNGCF7t210G4RejY3mw7Vgcg6zYwSzr67YqDp83gLHtUCo/nAb3HVI nA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=D4ZqAyq4SE/ICj/ozzOgWBRxaqFlE2xHi8oF3tQZWe8=; b=zpqoUqy9n/uAa+btmsD0g9Bp+eFVFHYTJVOBfwZuPG4G9rH0fX/7Ia5dL3AhRTAXVIzh bhPwFvLo+wbI8l6kbVPiyvp1hVXef0im2C9SBThTkqS9tbNMdljr70vEGufALNWgHELs HeGSDqaOiY/n6OTeBxCMH3uhTCE19q4flnN01TI2IVOwglPxEB0DzBqKXmlK+kXS/qPM NYhJahOIWQ/fdb41p+f66iFd/QYJ8wlpRBQ9CUvhKbGPmJ1aod/deGbWHcoUk+T/36Qr GJFQlxATWvK6vYv+4PcUU9Jg9qeJdqjLH238QqXwfT5nZS5YSX/1UDq7eKMwZB+ey2lN pQ== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgpghjpt-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:03 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeo5v159237; Mon, 16 Aug 2021 16:43:02 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by aserp3030.oracle.com with ESMTP id 3ae3ve34y4-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LfULiFQ/6ehwivAjs+yoCAZFXKU7w37DwWrKSUJPPMyBDG8UqeLY9I4cg38hwWhaDhUJLq7aZzzG4CC+gA4sXSpz9NH/edw0ic0LtKD4szG2l9vOppZZuB/uk6Eqrr65JWOCUiIqDGoVt545J7SZ0BkT/wqP4DkSMMxTdMBmkKzqR/ynXef2AcIziZftuHhMAgIF7DftvR0kxvzK+3UhMDLawiY+3XOks7ba4zQ5Wdp48pUDhvb7Iwh3PVK88OmWcfEEGW5rVzY4DgAAjMR1EGmJEPMa1cZjO41bGT6AxRzGEgaApyafVDyIPgEAdfmjHZLWEdDbXEAvAEhyjLgCpw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D4ZqAyq4SE/ICj/ozzOgWBRxaqFlE2xHi8oF3tQZWe8=; b=eyLH7X59JGqijT8elRKhJD/1BHd8MrB6zSAIxxvWmnaQqoHmUsdSviBjcrrGvfIkM6JYDXJVrfLvRYJaVKkMw5SXYpYfWwcYb3fCI6/JYYfJYnS8mhfDbN/Qs5DZ2qCnbzf/sP7+bT2RVtsfBZ+o1S9L4wFljmrqYaDTeGdjMKFu6akuZswRslxXxzeQNw3ErWxET4vnisyB6+Xp9Xbq38k6ml++LxgGvD+PesumpYHj9ERZa4jjAju2sa6IOACoO9eUXE5Uw6m9JHxKgzEQs/BlnLjLT8zVeHDRX/Rf1w2kMIekU+RLq3vP6q010LryhAo5NcaNPtqoMDwTVzwoPg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=D4ZqAyq4SE/ICj/ozzOgWBRxaqFlE2xHi8oF3tQZWe8=; b=zNv/VQtiwRemo0ZBRZ+6RXJHHSGyVfxvXFC9qiA4isB0pk7yBXoFnQpQiu/H+V4eZiT6fMG3chA30RE5fSIdTnmDhh7OjgpprY6at2gvqV+kpw45SAI7ar0RAo6DpvQPcerOu6reFJUZABhO7Yzv+FzFpq3bmemFbmCGtN5S5dc= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:01 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:01 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 10/16] vfio-user: pci_user_realize PCI setup Date: Mon, 16 Aug 2021 09:42:43 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:00 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b888b83c-7149-45b9-a241-08d960d4e8fc X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1201; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4IKBKuJt4QWdpYBA5zdfB3EbGHlR91ziNefq2Uj61+FVAuAQObL5071IEBMlrEDGfhxkWBLFxWJW3C0mDO+7uYUtVuW5f+2LxxNzan3JDcKNJjoxe2rWLzmNHYQNsx9BUpjmXOtv6eVl+0bD0D3MEqeYQjZP+E2J4MsT+cmc1m0UogVNFlohZEk1AHaZom4Q2Pu0xQNTL7nK1MSNy0h2XbnwsRNY52j6ONcVN79zaGTEvxaarEp2e3a/iTDF5AZegkRt8qwSWH8h7YVz1h2lwBpW9TD55CN2LM+alJ3RY1y62psufQYX4GuW/5Wyj5s+2ScaYLSovlJQJfVQ3IFeLl1z6UEVWChdMzmK5Oe2FYX0L2ThAL/OOdMiHLNA73SP6RBRzIE14xYsEoQerpxGg2+lbIl16EMTWu2H3N67vXOajmKC04yExuCpzKwisc/aHShwLesEMgA6xvJeqDjGVREQFasFUP/gqiOxaqL58qo8X9iHLggs17A0m2iBu6N9nwOvK25va0FNR6m2nDE+FHnZ83RJS3h2iHF8TMaQv8dYXwayoyv0x4CFDxx0VKvOMKnT7UT56eiFtMB2wxFZKRjjNbRAxTt1HC1tD95Gf59HGYzdrWaAOm3js2XOxnDu7lwzkZf09Kj1PJtoaxu2Og== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: 4XOSCZZQpgO9V4bUKxzxndPMZlyRjr2VW0aFopvlMvf6wl7rEtUlq03+CyAYk6XMloUqZJwBduh9/1tMWqb8IF1evlQKieSNZHMcnC8Rl9KLF7uarsoROh34Q1CerEl+a4jRJNDj/RDNtVz5sQ4uDC3s0LC25fgjgzU5UlGeTDCtLvfSkXRQZQgiyFEVawy+S+5IDEKvNIDyZx9ShM2UZElaX29V/GkslHzXo+v6tm4fmcsWZCc7yq+fKRDUvZhcGw7eYBjUdQKwLJRgjr/IRyA1B0mH4rT3ql8Q3FmhTfjnIHoDnSbMhO6q4mRRJG+PneKVAu4NvuY8eZZ0lU2MImLzQNb8BCDzfykvT4eabdcXtH9YumZOYgpiEi3JRu1613bomhGa81EeR8JS3qNb737UsedFJxqKWKOoIdLseqD+4JbOBCLzjVWM+OdDwNsk+sIU8j/rweiBELLIxcEUnkHRLmCyLM98Fm88mX92Eo94mwKnVNxUgKre+BsfwJDnw8fOtvqryT2fU6AAv1h6AXMTcls+h8jdipIjwmKQlDzi8sv0xeMU3tSJqbmiiFX6LAgqmepZvGv2s5bJyjYn+bzHv9PVfugXNZlX97yA7o7gk47lbN/IWXhpwY312Z3BpJI5E0ZvqUxlrstL+lyOW0auCJa7MPaQdMBYfQxWyDDA70uz+vSOPW4Tn/eUfL0XHtQoKTfSRWutk2sY57nnKjg7aJWlwPOkDiWq3b9XdQN6ozNLs33q2At2ucLPI14iQPjmxTEf2Z8z0ML1EoQ6kcLYgEjDfUApEQVySnPjh40uxkd9f+KLVBhH5L+SPQStr2eKrENJcQWF3ycdh+b3RL8Wxqz0uYcbbpYrFVURjy/ACKbzVw+NfmfUxsquAWksj/pxF2wREfpIiXvzVPB6SVMcYblbI8XmDSk2v1A5EinEtW8dgRN5Vq43NhjUTG188H1yEGBbTjuwUvnrO48S/InL0FU/XoZmvVcup8UuZvd2C5c36qIGzoYJ8oxIlcF9HcwjN/jzrhKkHTeDi0JbN7kGHQoYwDvKFOZ2x3OoOWBLJEZSAqwznLxsLO6YrKifnUA58WZoQcYah1gGK6biiUPNvn+zzjfr0EtORrJ5p+Y4arKnuV9hb+u3AyrbV2im4JQhvVlSjD2pKEJQ5h5EL2LFkcjTJbDV9bL7OKvi52lzV1InK8LOPC4+H8IIAzZFZsWPVABf0Y4NSoe86c0aUqZfC074i0/M3XB9DREGjBtMnAtJ02C0cPmzD8eicq+T8R0OoPAl+lz+g3VfvwWT2hXPzYVIBs7r7H7EuxSlZ1JLmUpBnklZ/8SY+3EkUxgbMW5XzUXawVRz2zIhoHudO73Xq0nZcmpXuIUtolt8XavCDjxDIlDK58EvItUqSH5f X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: b888b83c-7149-45b9-a241-08d960d4e8fc X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:00.9298 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: zSOn4p5ocsv1DHd/YO2apmDElhdqMX15BaM/pDkbukK6mp14r9IUtdYS86mXji7ZbnKnUdX+9EOR1+uU2XM5XB729wtwh71a3JW1v51vxoo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: mW2HA9vHsKV2Rp7bDGmaU9vQle4qp64t X-Proofpoint-ORIG-GUID: mW2HA9vHsKV2Rp7bDGmaU9vQle4qp64t Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson PCI BARs read from remote device PCI config reads/writes sent to remote server Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman Reviewed-by: Stefan Hajnoczi --- hw/vfio/pci.c | 210 +++++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 175 insertions(+), 35 deletions(-) diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 63aa2441f0..ea0df8be65 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -807,8 +807,14 @@ static void vfio_pci_load_rom(VFIOPCIDevice *vdev) memset(vdev->rom, 0xff, size); while (size) { - bytes = pread(vdev->vbasedev.fd, vdev->rom + off, - size, vdev->rom_offset + off); + if (vdev->vbasedev.proxy != NULL) { + bytes = vfio_user_region_read(&vdev->vbasedev, + VFIO_PCI_ROM_REGION_INDEX, + off, size, vdev->rom + off); + } else { + bytes = pread(vdev->vbasedev.fd, vdev->rom + off, + size, vdev->rom_offset + off); + } if (bytes == 0) { break; } else if (bytes > 0) { @@ -927,12 +933,28 @@ static void vfio_pci_size_rom(VFIOPCIDevice *vdev) * Use the same size ROM BAR as the physical device. The contents * will get filled in later when the guest tries to read it. */ - if (pread(fd, &orig, 4, offset) != 4 || - pwrite(fd, &size, 4, offset) != 4 || - pread(fd, &size, 4, offset) != 4 || - pwrite(fd, &orig, 4, offset) != 4) { - error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); - return; + if (vdev->vbasedev.proxy != NULL) { + if (vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &orig) != 4 || + vfio_user_region_write(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &size) != 4 || + vfio_user_region_read(&vdev->vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &size) != 4 || + vfio_user_region_write(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + PCI_ROM_ADDRESS, 4, &orig) != 4) { + error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); + return; + } + } else { + if (pread(fd, &orig, 4, offset) != 4 || + pwrite(fd, &size, 4, offset) != 4 || + pread(fd, &size, 4, offset) != 4 || + pwrite(fd, &orig, 4, offset) != 4) { + error_report("%s(%s) failed: %m", __func__, vdev->vbasedev.name); + return; + } } size = ~(le32_to_cpu(size) & PCI_ROM_ADDRESS_MASK) + 1; @@ -1123,8 +1145,14 @@ uint32_t vfio_pci_read_config(PCIDevice *pdev, uint32_t addr, int len) if (~emu_bits & (0xffffffffU >> (32 - len * 8))) { ssize_t ret; - ret = pread(vdev->vbasedev.fd, &phys_val, len, - vdev->config_offset + addr); + if (vdev->vbasedev.proxy != NULL) { + ret = vfio_user_region_read(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + addr, len, &phys_val); + } else { + ret = pread(vdev->vbasedev.fd, &phys_val, len, + vdev->config_offset + addr); + } if (ret != len) { error_report("%s(%s, 0x%x, 0x%x) failed: %m", __func__, vdev->vbasedev.name, addr, len); @@ -1145,12 +1173,20 @@ void vfio_pci_write_config(PCIDevice *pdev, { VFIOPCIDevice *vdev = VFIO_PCI_BASE(pdev); uint32_t val_le = cpu_to_le32(val); + int ret; trace_vfio_pci_write_config(vdev->vbasedev.name, addr, val, len); /* Write everything to VFIO, let it filter out what we can't write */ - if (pwrite(vdev->vbasedev.fd, &val_le, len, vdev->config_offset + addr) - != len) { + if (vdev->vbasedev.proxy != NULL) { + ret = vfio_user_region_write(&vdev->vbasedev, + VFIO_PCI_CONFIG_REGION_INDEX, + addr, len, &val_le); + } else { + ret = pwrite(vdev->vbasedev.fd, &val_le, len, + vdev->config_offset + addr); + } + if (ret != len) { error_report("%s(%s, 0x%x, 0x%x, 0x%x) failed: %m", __func__, vdev->vbasedev.name, addr, val, len); } @@ -1240,10 +1276,15 @@ static int vfio_msi_setup(VFIOPCIDevice *vdev, int pos, Error **errp) int ret, entries; Error *err = NULL; - if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS"); - return -errno; + if (vdev->vbasedev.proxy != NULL) { + /* during setup, config space was initialized from remote */ + memcpy(&ctrl, vdev->pdev.config + pos + PCI_CAP_FLAGS, sizeof(ctrl)); + } else { + if (pread(vdev->vbasedev.fd, &ctrl, sizeof(ctrl), + vdev->config_offset + pos + PCI_CAP_FLAGS) != sizeof(ctrl)) { + error_setg_errno(errp, errno, "failed reading MSI PCI_CAP_FLAGS"); + return -errno; + } } ctrl = le16_to_cpu(ctrl); @@ -1456,22 +1497,30 @@ static void vfio_msix_early_setup(VFIOPCIDevice *vdev, Error **errp) return; } - if (pread(fd, &ctrl, sizeof(ctrl), - vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS"); - return; - } + if (vdev->vbasedev.proxy != NULL) { + /* during setup, config space was initialized from remote */ + memcpy(&ctrl, vdev->pdev.config + pos + PCI_MSIX_FLAGS, sizeof(ctrl)); + memcpy(&table, vdev->pdev.config + pos + PCI_MSIX_TABLE, sizeof(table)); + memcpy(&pba, vdev->pdev.config + pos + PCI_MSIX_PBA, sizeof(pba)); + } else { + if (pread(fd, &ctrl, sizeof(ctrl), + vdev->config_offset + pos + PCI_MSIX_FLAGS) != sizeof(ctrl)) { + error_setg_errno(errp, errno, "failed to read PCI MSIX FLAGS"); + return; + } - if (pread(fd, &table, sizeof(table), - vdev->config_offset + pos + PCI_MSIX_TABLE) != sizeof(table)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE"); - return; - } + if (pread(fd, &table, sizeof(table), + vdev->config_offset + pos + + PCI_MSIX_TABLE) != sizeof(table)) { + error_setg_errno(errp, errno, "failed to read PCI MSIX TABLE"); + return; + } - if (pread(fd, &pba, sizeof(pba), - vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) { - error_setg_errno(errp, errno, "failed to read PCI MSIX PBA"); - return; + if (pread(fd, &pba, sizeof(pba), + vdev->config_offset + pos + PCI_MSIX_PBA) != sizeof(pba)) { + error_setg_errno(errp, errno, "failed to read PCI MSIX PBA"); + return; + } } ctrl = le16_to_cpu(ctrl); @@ -1619,11 +1668,17 @@ static void vfio_bar_prepare(VFIOPCIDevice *vdev, int nr) } /* Determine what type of BAR this is for registration */ - ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar), - vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr)); - if (ret != sizeof(pci_bar)) { - error_report("vfio: Failed to read BAR %d (%m)", nr); - return; + if (vdev->vbasedev.proxy != NULL) { + /* during setup, config space was initialized from remote */ + memcpy(&pci_bar, vdev->pdev.config + PCI_BASE_ADDRESS_0 + (4 * nr), + sizeof(pci_bar)); + } else { + ret = pread(vdev->vbasedev.fd, &pci_bar, sizeof(pci_bar), + vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr)); + if (ret != sizeof(pci_bar)) { + error_report("vfio: Failed to read BAR %d (%m)", nr); + return; + } } pci_bar = le32_to_cpu(pci_bar); @@ -3423,6 +3478,91 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) goto error; } + /* Get a copy of config space */ + ret = vfio_user_region_read(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, 0, + MIN(pci_config_size(pdev), vdev->config_size), + pdev->config); + if (ret < (int)MIN(pci_config_size(&vdev->pdev), vdev->config_size)) { + error_setg_errno(errp, -ret, "failed to read device config space"); + goto error; + } + + /* vfio emulates a lot for us, but some bits need extra love */ + vdev->emulated_config_bits = g_malloc0(vdev->config_size); + + /* QEMU can choose to expose the ROM or not */ + memset(vdev->emulated_config_bits + PCI_ROM_ADDRESS, 0xff, 4); + /* QEMU can also add or extend BARs */ + memset(vdev->emulated_config_bits + PCI_BASE_ADDRESS_0, 0xff, 6 * 4); + vdev->vendor_id = pci_get_word(pdev->config + PCI_VENDOR_ID); + vdev->device_id = pci_get_word(pdev->config + PCI_DEVICE_ID); + + /* QEMU can change multi-function devices to single function, or reverse */ + vdev->emulated_config_bits[PCI_HEADER_TYPE] = + PCI_HEADER_TYPE_MULTI_FUNCTION; + + /* Restore or clear multifunction, this is always controlled by QEMU */ + if (vdev->pdev.cap_present & QEMU_PCI_CAP_MULTIFUNCTION) { + vdev->pdev.config[PCI_HEADER_TYPE] |= PCI_HEADER_TYPE_MULTI_FUNCTION; + } else { + vdev->pdev.config[PCI_HEADER_TYPE] &= ~PCI_HEADER_TYPE_MULTI_FUNCTION; + } + + /* + * Clear host resource mapping info. If we choose not to register a + * BAR, such as might be the case with the option ROM, we can get + * confusing, unwritable, residual addresses from the host here. + */ + memset(&vdev->pdev.config[PCI_BASE_ADDRESS_0], 0, 24); + memset(&vdev->pdev.config[PCI_ROM_ADDRESS], 0, 4); + + vfio_pci_size_rom(vdev); + + vfio_bars_prepare(vdev); + + vfio_msix_early_setup(vdev, &err); + if (err) { + error_propagate(errp, err); + goto error; + } + + vfio_bars_register(vdev); + + ret = vfio_add_capabilities(vdev, errp); + if (ret) { + goto out_teardown; + } + + /* QEMU emulates all of MSI & MSIX */ + if (pdev->cap_present & QEMU_PCI_CAP_MSIX) { + memset(vdev->emulated_config_bits + pdev->msix_cap, 0xff, + MSIX_CAP_LENGTH); + } + + if (pdev->cap_present & QEMU_PCI_CAP_MSI) { + memset(vdev->emulated_config_bits + pdev->msi_cap, 0xff, + vdev->msi_cap_size); + } + + if (vdev->pdev.config[PCI_INTERRUPT_PIN] != 0) { + vdev->intx.mmap_timer = timer_new_ms(QEMU_CLOCK_VIRTUAL, + vfio_intx_mmap_enable, vdev); + pci_device_set_intx_routing_notifier(&vdev->pdev, + vfio_intx_routing_notifier); + vdev->irqchip_change_notifier.notify = vfio_irqchip_change; + kvm_irqchip_add_change_notifier(&vdev->irqchip_change_notifier); + ret = vfio_intx_enable(vdev, errp); + if (ret) { + goto out_deregister; + } + } + +out_deregister: + pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); + kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); +out_teardown: + vfio_teardown_msi(vdev); + vfio_bars_exit(vdev); error: vfio_user_disconnect(proxy); error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); From patchwork Mon Aug 16 16:42:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A227FC43214 for ; Mon, 16 Aug 2021 16:47:52 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 27E1360F39 for ; Mon, 16 Aug 2021 16:47:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 27E1360F39 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:35982 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfmB-0000uY-9S for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:47:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37130) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhe-0000nX-7l for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:10 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:38904) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhY-0007y8-Lw for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:09 -0400 Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaA9q021054; Mon, 16 Aug 2021 16:43:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=nUV3Y0uIESmLz7Zkqh1tpA3nnyQpn+3Fz/PyO5rgOHU=; b=gnFaoNCCsUe62fPezMnI+Rt7lr7nA8k8wgGq8iB4/NOWWHVWwaDe9BKE9604Z3HXwPpH b2yKLuZGZQ0su16qpPTrCkhZ4esTdf34VVnPhwlY10Lw0cb1B+ZC9mn8yEs1pmBeAzX5 aI24w/M8cxrI6gxEtf1AnkqvvfRFRzfnajAGQEqiR+MPPy2N6Bd3/72EhF9TDwNAPFh8 mx4Hw/U5KNAyoQlL1ln3lgvTbF4kHiR5//ARa59dYxAp2plMB6eu3hd3E3QVRnXcU6xc qcsmoa7Eue8Xw2GA/wPRuwbtNbW6aXP5D471EdXlPA0Ffr/xgASsF6rUxVKANm9tWb6H cA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=nUV3Y0uIESmLz7Zkqh1tpA3nnyQpn+3Fz/PyO5rgOHU=; b=Z+RLqUyUj4EekElBmqBRd9A5pbSf5jO5jjT+nkUo8/T0IXnWgPX6BJcf2QZrgovyJ2rp dh2zz1cg9JXF1rhkmuqu8UfwpP2iKsjVKFfh0caAZOHIDUMMJIDpKI3rbVOe6Ga5RPDO 7lG4tQvD8xsohqqr4JoFnBUunMrMpl/P3dCz3yv22oHFbe2EWlTE6lZ5w+dKcHLbnWOj rcU6OQF7mmtwjKYcSX7m4Ggc49Ov5iGPt4YKOHWFW+Mwi1Enw3kFpmX6mGnSS5YeDbSs ypG/rmGPU7hdx2XhSgdyQMFk4tDqa3DC/bIoPtDazSrq5B7aqi7gelnplBjjSN8IMqD+ gg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgmb9k04-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:03 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeo5w159237; Mon, 16 Aug 2021 16:43:02 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by aserp3030.oracle.com with ESMTP id 3ae3ve34y4-5 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:02 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F8Drh/648U9JmUfzCD4PNHZs6T1SVGjA1jIx2dzW6Wyyj4NLFPulB8Fv33AHvBshQVPIA6sc2qfaawRd/KhXGiYICtVddGnac80sUJpmqsXLcCHaUDTwPZDf3qnYE/xfGPitxY2ji++uDPSxeJSSyNinHifDNaMnI9raHvJejPUhwzeBeyQLC7HCBWE2sVVpmMtzJanTChKYxoGe2VjugRVuSUZRspV0qZjwVr+eFaICBTO2ySzGE7I4GbIcarb+jUOph4ndBZGVjMw1rkL+DXkbKdHOcobi/nQDQ4V4+DEJbehXN+JUNpkXEJVNLHbClwRfhXcJ8xlxNKyqsjAR9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nUV3Y0uIESmLz7Zkqh1tpA3nnyQpn+3Fz/PyO5rgOHU=; b=H0EWHP/YN9CWEsYgTjXoFeAdGcxr1Jk4NIMa02jzon+16U42WOvHPzPMfB+jtTP7StBNGVheCY/HrFu5BAvTQxvKYT8gT7E0OzoINh5pq/DyVoCBkcbpJ78hv8Hvg+HzEXteCvosbwPJ54BoCVb8+tdhvFs9wa6gXIV/T/m2yQYxEl9aelg48buMFw+uvh29mRjcHypGO4n4Cx1PAnTFieW7Iu+ptTnHCzgfBtI96onFy5108DRUlVYY0iSpCYz31eYNuHx8yo7z4Gi0Rj96ZqNFYLUVcsxwLIxhysozH5snTeFS8iAklYFQBgdwW5fWXwtYpqcoWQuMhbFn5/nK8g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nUV3Y0uIESmLz7Zkqh1tpA3nnyQpn+3Fz/PyO5rgOHU=; b=O0X69LWGS9s7BWZk7bz27oLtVfqH19qtPrZ02K+R0IwcqquZtzQKCM+PxvAI23MTvz97Mz9vteKZJL33iNpeacjv+0jh+qr1b61keD3GJmfTKDPM/CykRr12yYw6ErVw5UveCzKT9Rq4nAoRr9ncK3tGyfroxsLXlrQUiWX7sP4= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:01 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:01 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 11/16] vfio-user: get and set IRQs Date: Mon, 16 Aug 2021 09:42:44 -0700 Message-Id: <4d292fab645b18b8a20f0f5a83356ae7fc0efa28.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 669b6871-2c9a-4f4b-3d45-08d960d4e94c X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6430; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: S6lD3zrhA9HFLZRhecQfWEhWwzY92bKIiyI3mtt1U1+VefYwrCNFkYPtY729Uz/1VMbrERKCoRnUnpe8TU1TpfLLRN+dsjGZjLffqs38MUDosLtVU0YMvSS4opCkyPGdzV/txGswoJ0/IQk9MPRCjc0CrpxroiPWMiRbTk2xjrYUrqKMsx2nExN1F2jBZS8JirUX4zSxc2rq4cE/szf9002TX9S1QZcRWByhScRnXOySFBhkZJVeyag+gjU5BdGZS8AQcHC2uHXfd9d47IIriAD45no8i/kQ94fuO7FeKje1jCDBDDBtr/gU8oUBacsX4wXA3IqAtY/kWnlsd/N1VUtyDb5OvqTl2z1Sqvk7aXUrVt30uy0Uuvyx08gdlxTG4y2mvr9sdOOrTFzypSIp5qLmYJEs+peG260H+Z51C40kSe4N67kXkjPArFPi3au5S14SNyBFDVcSvQiHwnJ2xumu/872JSK0B4FHfu7c8RFgCVcSXvnC3l1FiUV/U20RodlitywdsWq1eEJlhfWIRCNKAHFhpCQBkRqQejnSNYMOFqW9/rdaTfj2bTwP80eqwt6lYHPAphkXtZskWkxk4BK21+xYJ1x59WLJdqD2PCdHU2DgVeBtl+R4j3EE6MwW3hGcVXnODLcnQprZ1MubmA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: LwqJ8yNtBvqrUNMsBupJSH2DZELQkfTqCXSTypPO9fmmpcbMlCSMROMchafr4U/GquNJfAo51Sn/g/GOp9Obx2X4kxFNE0eOtUTtp0fjCAitdGuNx6sIl/x7+AiCi2xvK30Osfnd5zyOrXDWoK2f1Iviq+/Jy84BXdPG6+Ge5watkM6+haxX4qt8S5GFnK2fp9GZZGS45h1OF4/IWYB3fpjcTqlMK6SAWnKmjGNcra1OKLHM1laS7LrkCir+fufySDqoFHBDeylT47uEZMgNC+ncJn8g73iowZImpE7OUfOZnOJPxVtyCGmuneo9iFMosXxupcexYncZClaKqMa4XcSdlp/SEnVg6rnKz04r6E3uFSZ6WFdzOLvjYXe5lckd6cCgCVArBQqHZow61GBTFG9WmCzPep+BEYQOmB58CqVezyrNQ+yTpgHFgx7G6pBDTiVHbWsLW0PmvGVj8x2Z/s3WrPxKRhiPEEK3KIcdyrcCMaBbBJIpL7g52VseEylpr8y5o4BqZXF+zxw7YgVJwy3N3mGb1Jug83mVhXoZSzrwKpgyEHjoptQO3Kzzf4u8nTskGUlME3hUSGiS/9QVgn6A8pO1z0Sw2W9DRTG+WIMsbVRrVpjC1h7KxVGnzOrhUXPS/jeE4SizFdw0IGD2qg2GjorvobtB+g4ucgUgGMmKlcisn7+bDo8cAb6Xbl2mOWx+jmyo+rEEG8AOHgcZzoWL0jexocbWCCC9I+DFFe/h58wmEGC0maMfeDQ9pXcx+RgQI4MSjkfQLWUrJ4ZoigNzPl/KMtcxpmgdciZwjKR2t834YNgibLmLyKFTPQvazeZxyTYSKjOIwuzaMPK3Jt059iDi8bRddmCD292t+gPJ1xTv4v++qRdC+UVmKiokUhFvLBP9tIgr2Ua2dBHAQj5oC3khgmfoVuI6OtOQ4f/SHaqTDwYW6Q9SXr07Cj6Um6wkexL/zYuqu4o1v9rQZQVgYr/Tc2a32relBDvgiwLsSkttBs1FUM27ZCXZkakHEmnZiVFx8289q4Z1esFw2aZ6sTiQfpovYWlLQybpGLA1jo1Jxl4P0lpbyz0nHA7IbZkMo/H8XWUaQN3l5nOmPPV+JKwuTVkvi19Md7ZHa3MHnKeHHpiYJSLQmYZx8kt6UHOgRMbDUo1UeDRNnZqmAy1z4vaZ30lKmOmyqbvwJQ/BMy8m3sFjLw7c2ZCgS6Fq0das/3yCeJn8OO4oQpKTsDhdAtFTkpbMYc0ihgT8ff60jhVnlx8WUGE7muwpshxhZ0R1G/Ayz0nxU2NzsMMOidt//5LHWrX0eow7DArO1EuaNqEnL7ioBIYSPF41t4l32Naai16nthJRxBAN05Ztw0Pic7MOWq96jdFuwzEOabgkRNfsbEsYCdYLRkN8tMxZ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 669b6871-2c9a-4f4b-3d45-08d960d4e94c X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:01.4695 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 10fU2+Ux1N7evxwTktAFVlHoFpEYcurBe66Q29AE4pn6+9Ok/CNQIb72ilh0AOfuS2TzfX5JrAp2pY8QS6W+TLpkwUtjrAB6mdlOZpiV+lo= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-ORIG-GUID: U3V5AmsFVs7sHS8ti7MBGc1sU_s0Q2PV X-Proofpoint-GUID: U3V5AmsFVs7sHS8ti7MBGc1sU_s0Q2PV Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -8 X-Spam_score: -0.9 X-Spam_bar: / X-Spam_report: (-0.9 / 5.0 requ) DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 25 ++++++++++ hw/vfio/user.h | 2 + hw/vfio/common.c | 26 ++++++++-- hw/vfio/pci.c | 31 ++++++++++-- hw/vfio/user.c | 106 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 181 insertions(+), 9 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 56904cf872..5614efa0a4 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -109,6 +109,31 @@ typedef struct { uint64_t offset; } VFIOUserRegionInfo; +/* + * VFIO_USER_DEVICE_GET_IRQ_INFO + * imported from struct vfio_irq_info + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t count; +} VFIOUserIRQInfo; + +/* + * VFIO_USER_DEVICE_SET_IRQS + * imported from struct vfio_irq_set + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint32_t index; + uint32_t start; + uint32_t count; +} VFIOUserIRQSet; + /* * VFIO_USER_REGION_READ * VFIO_USER_REGION_WRITE diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 02f832a173..248ad80943 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -74,6 +74,8 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); int vfio_user_get_info(VFIODevice *vbasedev); int vfio_user_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info *info, VFIOUserFDs *fds); +int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info); +int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq); int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, diff --git a/hw/vfio/common.c b/hw/vfio/common.c index a8b1ea9358..9fe3e05dc6 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -71,7 +71,11 @@ void vfio_disable_irqindex(VFIODevice *vbasedev, int index) .count = 0, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + if (vbasedev->proxy != NULL) { + vfio_user_set_irqs(vbasedev, &irq_set); + } else { + ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + } } void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) @@ -84,7 +88,11 @@ void vfio_unmask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + if (vbasedev->proxy != NULL) { + vfio_user_set_irqs(vbasedev, &irq_set); + } else { + ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + } } void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) @@ -97,7 +105,11 @@ void vfio_mask_single_irqindex(VFIODevice *vbasedev, int index) .count = 1, }; - ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + if (vbasedev->proxy != NULL) { + vfio_user_set_irqs(vbasedev, &irq_set); + } else { + ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, &irq_set); + } } static inline const char *action_to_str(int action) @@ -178,8 +190,12 @@ int vfio_set_irq_signaling(VFIODevice *vbasedev, int index, int subindex, pfd = (int32_t *)&irq_set->data; *pfd = fd; - if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) { - ret = -errno; + if (vbasedev->proxy != NULL) { + ret = vfio_user_set_irqs(vbasedev, irq_set); + } else { + if (ioctl(vbasedev->fd, VFIO_DEVICE_SET_IRQS, irq_set)) { + ret = -errno; + } } g_free(irq_set); diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index ea0df8be65..282de6a30b 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -403,7 +403,11 @@ static int vfio_enable_vectors(VFIOPCIDevice *vdev, bool msix) fds[i] = fd; } - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + if (vdev->vbasedev.proxy != NULL) { + ret = vfio_user_set_irqs(&vdev->vbasedev, irq_set); + } else { + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_SET_IRQS, irq_set); + } g_free(irq_set); @@ -2675,7 +2679,13 @@ static void vfio_populate_device(VFIOPCIDevice *vdev, Error **errp) irq_info.index = VFIO_PCI_ERR_IRQ_INDEX; - ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + if (vbasedev->proxy != NULL) { + ret = vfio_user_get_irq_info(vbasedev, &irq_info); + } else { + ret = ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info); + } + + if (ret) { /* This can fail for an old kernel or legacy PCI dev */ trace_vfio_populate_device_get_irq_info_failure(strerror(errno)); @@ -2794,8 +2804,16 @@ static void vfio_register_req_notifier(VFIOPCIDevice *vdev) return; } - if (ioctl(vdev->vbasedev.fd, - VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0 || irq_info.count < 1) { + if (vdev->vbasedev.proxy != NULL) { + if (vfio_user_get_irq_info(&vdev->vbasedev, &irq_info) < 0) { + return; + } + } else { + if (ioctl(vdev->vbasedev.fd, VFIO_DEVICE_GET_IRQ_INFO, &irq_info) < 0) { + return; + } + } + if (irq_info.count < 1) { return; } @@ -3557,6 +3575,11 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } + vfio_register_err_notifier(vdev); + vfio_register_req_notifier(vdev); + + return; + out_deregister: pci_device_set_intx_routing_notifier(&vdev->pdev, NULL); kvm_irqchip_remove_change_notifier(&vdev->irqchip_change_notifier); diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 83235b2411..b68ca1279d 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -768,6 +768,112 @@ int vfio_user_get_region_info(VFIODevice *vbasedev, int index, return 0; } +int vfio_user_get_irq_info(VFIODevice *vbasedev, struct vfio_irq_info *info) +{ + VFIOUserIRQInfo msg; + + memset(&msg, 0, sizeof(msg)); + vfio_user_request_msg(&msg.hdr, VFIO_USER_DEVICE_GET_IRQ_INFO, + sizeof(msg), 0); + msg.argsz = info->argsz; + msg.index = info->index; + + vfio_user_send_recv(vbasedev->proxy, &msg.hdr, NULL, 0, 0); + if (msg.hdr.flags & VFIO_USER_ERROR) { + return -msg.hdr.error_reply; + } + + memcpy(info, &msg.argsz, sizeof(*info)); + return 0; +} + +static int irq_howmany(int *fdp, int cur, int max) +{ + int n = 0; + + if (fdp[cur] != -1) { + do { + n++; + } while (n < max && fdp[cur + n] != -1 && n < max_send_fds); + } else { + do { + n++; + } while (n < max && fdp[cur + n] == -1 && n < max_send_fds); + } + + return n; +} + +int vfio_user_set_irqs(VFIODevice *vbasedev, struct vfio_irq_set *irq) +{ + g_autofree VFIOUserIRQSet *msgp = NULL; + uint32_t size, nfds, send_fds, sent_fds; + + if (irq->argsz < sizeof(*irq)) { + error_printf("vfio_user_set_irqs argsz too small\n"); + return -EINVAL; + } + + /* + * Handle simple case + */ + if ((irq->flags & VFIO_IRQ_SET_DATA_EVENTFD) == 0) { + size = sizeof(VFIOUserHdr) + irq->argsz; + msgp = g_malloc0(size); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, size, 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start; + msgp->count = irq->count; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, NULL, 0, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + + return 0; + } + + /* + * Calculate the number of FDs to send + * and adjust argsz + */ + nfds = (irq->argsz - sizeof(*irq)) / sizeof(int); + irq->argsz = sizeof(*irq); + msgp = g_malloc0(sizeof(*msgp)); + /* + * Send in chunks if over max_send_fds + */ + for (sent_fds = 0; nfds > sent_fds; sent_fds += send_fds) { + VFIOUserFDs *arg_fds, loop_fds; + + /* must send all valid FDs or all invalid FDs in single msg */ + send_fds = irq_howmany((int *)irq->data, sent_fds, nfds - sent_fds); + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DEVICE_SET_IRQS, + sizeof(*msgp), 0); + msgp->argsz = irq->argsz; + msgp->flags = irq->flags; + msgp->index = irq->index; + msgp->start = irq->start + sent_fds; + msgp->count = send_fds; + + loop_fds.send_fds = send_fds; + loop_fds.recv_fds = 0; + loop_fds.fds = (int *)irq->data + sent_fds; + arg_fds = loop_fds.fds[0] != -1 ? &loop_fds : NULL; + + vfio_user_send_recv(vbasedev->proxy, &msgp->hdr, arg_fds, 0, 0); + if (msgp->hdr.flags & VFIO_USER_ERROR) { + return -msgp->hdr.error_reply; + } + } + + return 0; +} + int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data) { From patchwork Mon Aug 16 16:42:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438967 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85A16C4338F for ; Mon, 16 Aug 2021 16:57:17 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D40B060F35 for ; Mon, 16 Aug 2021 16:57:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D40B060F35 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:60524 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfvH-0000Kb-U8 for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:57:15 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37166) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhf-0000rt-N4 for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:11 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:40068) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhZ-0007yP-DC for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:11 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaFJO000687; Mon, 16 Aug 2021 16:43:04 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=qGePs2QFNSfTvuBH7+YB/8qpJE2m7WykWxFo6Uuiu3o=; b=gc6MdU+eOfiCF28+mSky+2BJstuPqxrCtuXOhpYgLvrGylnWmFwhEgBKtxtP+AiZ5YAd ArPYZRJi/OpiKwJ5k8sOpFBtff8llsRX2ep38JNYVY3PQsnuOVtke71Q0Im/QniNtXVF 5DUnq/iF23h1m8VnOt+r616Nb3OaA5krlo2IE7ooIjUBEzDZXoUlsphFSRMZWBnVvb38 QGX6dWWpWOl/pxNTu5xIQ9TotyelMqE7ftEs1BO8rRrELcvzctHLNGQNHVDNSPOLBvdE KQKgAzzAYhIPYRA1xK2xorPErJRsM7l6LRxhlWtgbq3hFSAw+8ITNK69KpLnjk+NcOnl tA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=qGePs2QFNSfTvuBH7+YB/8qpJE2m7WykWxFo6Uuiu3o=; b=G32KNQz1z/+BohJ5NZE5d97LAJWDlDxBgl06Egbmo+QyU7+KKPg+y1yOwxp/mqEz5+6e aP16StKvznq7lOd79B22BkZ+dRVYuqn8J3s9eY/8ihRi+tvPVcuTFGjLuyBFNYK8k+Gl zJgS7LqyaSP8r3ULtQ5mpqTZfhskdo/EL1GjZSVflCCJjG0qkxLayFbdkPxf/blqwltT xez+1FOpcdj87EqLNDoUQ+J6XK356HetzTSvSAyGe72IArltIbrxB7gjG2x4x2IRANfk f5SsJKMkv6Sav+UPWTGgGBR5c/sRhlgag9tSaDxPLbPHRvcHqUWRkVPNvQGr83b1nnnL og== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by mx0b-00069f02.pphosted.com with ESMTP id 3af1q9anv0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:04 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeo5x159237; Mon, 16 Aug 2021 16:43:03 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2106.outbound.protection.outlook.com [104.47.58.106]) by aserp3030.oracle.com with ESMTP id 3ae3ve34y4-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:03 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=F50STqsKGHZCa4qnxX0k/hoMzwKhqjv8EZkgx2iQnzTfF27Yim3WYktAq8NWkmmj8IpqDGH/9rNAZC43hInvYpuwBIdEKc4sxVnk2lBo+liGB6UuGCaOoYUfmMFqqzWNlLZYrKulVcgtiiJATYJXpkSAVIuOGzqZIGPp8yQlDZjNh8kqnXuSGXOT1ExwRrv7mfQiMqT629pinB9gZ14hIhuqkAGJGWB2870u0m8w+TLh1A8wzu8tob++hGTQ6Wlwcabl3oZFGSlGvQbSgFHA6NDRQ3IDqE9pX9GMvgg31/po89a1z8IXZsIw+o0utHQPoMbfnpEj+uuBSSgBdVzEgA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qGePs2QFNSfTvuBH7+YB/8qpJE2m7WykWxFo6Uuiu3o=; b=XvxacVjcb+MgL6TOiSCMoVqrvBNJ57SSSbSB18T9Jo8pg7LmZ9iShBtZh21tkvjriAuaVZNj772kFi8DM1AdoKNlHatnGhZRmAAlqXekgkwdB7fUd0hwDfaN0+XSWQlgB2zbfIdWTjQY1yN+PBttLR5Gisz4Vgzs+0zBFA9YmsZDtn432BbeyIy2S4+1t6p9sZaOzSYjHdADm2BSztXiX4v8zNlT28gbpeQk5dSaClEb9T9UPdVDHEqZU8/MD9zsljyE/eytRKGVD/IcNiw+JdHQc4GEfYxh1I67NMXtRM3hVKGOBEpvUXqRFmCKgoQVuxtCR5e1PYEDUObTdk9qyA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qGePs2QFNSfTvuBH7+YB/8qpJE2m7WykWxFo6Uuiu3o=; b=rJtFRQgl2gVA47f9Jc2KQ4GHRtEYJe/W8URT8pjWUXPV/z1PJ9oGqFFOyccM+xKAxN5Q5CWQFYjDY8w8ot9LSI7m8qIk9/CR1mdvJ2cHCYAe3kTabQH7wzG+mP9v4Q7z4VXvTTedVZPASqG8di5gIIT98TeV45lzYLFvz3ZidZg= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:02 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:02 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 12/16] vfio-user: proxy container connect/disconnect Date: Mon, 16 Aug 2021 09:42:45 -0700 Message-Id: <7f6595dcd88bfc0b459d3befd615a569635bae7d.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:01 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fd93c160-5cd9-4990-03cf-08d960d4e99a X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1751; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mbFngX9tuTfBFS2HKkr7hZGalHDeFBjZzaAzbSLYcAwMOn3rSJV8L8R/MhGqM7ktLN25lNTZMNW+QMPQna+cfUeioTf74jzxQi22ra/nSpaM4LwEENyq0qX7DXI59Y5aRyLnhr7HiPm5wB5/5vuAnkb6w/t3j5A8z/k6zsPojkWqXY/xBddzeBm4MvJFUuPkC7W1Cl0GYY6BEu6uOywbNFtGmrllqKfxmfXurnK8tRfWbqMwv9+4+iysuc82+MG8fIpLYl7QkDJ09twuKglsrSGro/SDSzZmmOs0FcNl+iEiFZQIO84sE1wP/Jp9ilmwMKJgTss5T55HBfuoJJIeE9ym8l8mNW4+Ota9QGLFtJ8ZeJqyYMC3blZd2O697Ex6C9BJmw1Ivny3camU/PsUpt/Zq3zue4W/DwD/s0T0XB+2OTrRKAhB2HnyZY6WOb4HZsnNJXSVMUtmxqI6yW1diWUIlB6IFJMDw1yuWMbLWSJ6k47Z4OtWpG7JlAxYLW5LFGZ0IOvdB9Du9jUGRTWxFOMvMu82G7NDd9rF/DGbFd+MV4ORHmJ1/cQXS8s1nn3d/rigIOGUXou0RgRQ3v7DogtE73f2e8GLFOsVG9VRP9wa0ZYaKrMD/m2FosUK8+mmlJWTirY3Iq+Ypxr4hmi9ww== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: cx10qUFyuC1D66I15ksf+IU802yCUiELQxciIunvOq0kslz33cupOJtw/BLrsnUTi50RAtdn+UycPyfCrJT/+DaIAjszZr+bkHcpO8tPEX3kb4xOCZOFmf+OKv7gXtsOcQ0Nt20vnEJ+GTbBaXKxNxcAReT+g5qYBnWeEdd6zQsmai2xSsjfv6LGDwcpe0LOov2dhH8HusdaPCDJx9/6WIvYAU9FEF/ZNWieO0OaWlpJpc9r1ZD9pTrbamhsWWEho7qKacZrKeT6VYeFaxyJ7RlyQDV4cbksAYkphz2ezRAvwZk41LW7Btbmaj73O3yB1K+zyF6NigF7QpgUqN9df+QkOrCx00z5/DSx9dQBjfr+rITuCj1KoR9ySwKM4OzxVxUsFQhHy0pgc4hmKTDGjWa85QLB6CrQ4gkJAh+NZKn66zhkRWD0F4/vlpGT1O9x1JYpS/xaVpebwLUwxptLi/qmyn0ydnfbZZuI5+bGd4csawboaKZM4MF9/eCiSRKji2o87yc0nMAfg3pZBZEuknYpE1p+N9oKXlwYZqeq+rVIkB8GWta+u1xtL++qkT3Xmak4AvL41OY6MOh23HAI/s/1mAeZGm9pn3CwTxWefVRXTAUJIsaEK9OH5qclc8Cr458WXFdy8p7tDc4hELqZgKI6ohwRJiZhMnyA+ojBsnziJqLGNp65rPyQRhFdIrQOMVlmso42pC1cPN9uBn6raaXwBckpi1nCgeSe6RxNBeBdxq1TMmYn6z5axTtATatLErBu0st6METg5njO2hl7xw5WybvIqczm4Xy3L368SjROA/CVzeUBual328OW2HY/8d6RCQskGverQlyxceswZDHikGTwPANz68B1+G7SU1LuPKjRUf0MS2dav5z8ALC6buWQFsKl3EMclb0faG/J2lY0yVPK1FWp+se+r6Cb1G4e1Vjon+0aJpMjwZBnPpQ33+pxoxL+rJIXwjUkKvPRX1xsymnt+ND6XH7C9chtGY3CMNssAgNiV5f2hhfGvC5knTvg35b1nfo5XshjuqJx82IVp6/nf2rHLWpINfvG1UFZZEmrEEFmR14mHBJf3tJ/Oq0Sc+/0UfbsHnNHLmeXo/2vIa2rgfTOkRE+s5Y6mgvwx0io2tYXM0HqvtjU/e7XkCBqtfIbfrlvmE1gw0a5I1n5iL3w0wzRAsxmV3UqX9KssutwnnhYi0M4Tqm7xxbTk9JDnv+pAyKdALIRxaaygAW5ABWLt30QMF/L0TeI5A5P12ptV7EPnTHhKOlCi2ow/2UIoCnWhIpzPhwFjpO9aDTpTTYpQ2h8BZyT8cBzi1RBdu/cxuH4H/rF6ngaCxN28koyZn3MlMDTklLJvto6LDhbPT3Kct7JsrW3Lpa/H+AWFdGYVRlz4oN5P1e8+v/+ X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: fd93c160-5cd9-4990-03cf-08d960d4e99a X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:01.9463 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lQ3sPeC7R80UvcvlceQeg/QpmPtFaDGtgc1JZO/U8ZD8OGsKvPTm66CdxBP8pvziPIU8jJFvwABKgPN2PVvGtp2hB/FQs5iV2Iq8DgWoVLc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 spamscore=0 bulkscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: LeNAUWSeXRzR7ZqcyHZfFvjnHvi-TK6z X-Proofpoint-ORIG-GUID: LeNAUWSeXRzR7ZqcyHZfFvjnHvi-TK6z Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- include/hw/vfio/vfio-common.h | 3 ++ hw/vfio/common.c | 84 +++++++++++++++++++++++++++++++++++ hw/vfio/pci.c | 22 +++++++++ 3 files changed, 109 insertions(+) diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index bdd25a546c..688660c28d 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -91,6 +91,7 @@ typedef struct VFIOContainer { uint64_t max_dirty_bitmap_size; unsigned long pgsizes; unsigned int dma_max_mappings; + VFIOProxy *proxy; QLIST_HEAD(, VFIOGuestIOMMU) giommu_list; QLIST_HEAD(, VFIOHostDMAWindow) hostwin_list; QLIST_HEAD(, VFIOGroup) group_list; @@ -217,6 +218,8 @@ VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp); void vfio_put_group(VFIOGroup *group); int vfio_get_device(VFIOGroup *group, const char *name, VFIODevice *vbasedev, Error **errp); +void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as); +void vfio_disconnect_proxy(VFIOGroup *group); extern const MemoryRegionOps vfio_region_ops; typedef QLIST_HEAD(VFIOGroupList, VFIOGroup) VFIOGroupList; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 9fe3e05dc6..57b9e111e6 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -2249,6 +2249,55 @@ put_space_exit: return ret; } +void vfio_connect_proxy(VFIOProxy *proxy, VFIOGroup *group, AddressSpace *as) +{ + VFIOAddressSpace *space; + VFIOContainer *container; + + if (QLIST_EMPTY(&vfio_group_list)) { + qemu_register_reset(vfio_reset_handler, NULL); + } + + QLIST_INSERT_HEAD(&vfio_group_list, group, next); + + /* + * try to mirror vfio_connect_container() + * as much as possible + */ + + space = vfio_get_address_space(as); + + container = g_malloc0(sizeof(*container)); + container->space = space; + container->fd = -1; + QLIST_INIT(&container->giommu_list); + QLIST_INIT(&container->hostwin_list); + container->proxy = proxy; + + /* + * The proxy uses a SW IOMMU in lieu of the HW one + * used in the ioctl() version. Use TYPE1 with the + * target's page size for maximum capatibility + */ + container->iommu_type = VFIO_TYPE1_IOMMU; + vfio_host_win_add(container, 0, (hwaddr)-1, TARGET_PAGE_SIZE); + container->pgsizes = TARGET_PAGE_SIZE; + + container->dirty_pages_supported = true; + container->max_dirty_bitmap_size = VFIO_USER_DEF_MAX_XFER; + container->dirty_pgsizes = TARGET_PAGE_SIZE; + + QLIST_INIT(&container->group_list); + QLIST_INSERT_HEAD(&space->containers, container, next); + + group->container = container; + QLIST_INSERT_HEAD(&container->group_list, group, container_next); + + container->listener = vfio_memory_listener; + memory_listener_register(&container->listener, container->space->as); + container->initialized = true; +} + static void vfio_disconnect_container(VFIOGroup *group) { VFIOContainer *container = group->container; @@ -2291,6 +2340,41 @@ static void vfio_disconnect_container(VFIOGroup *group) } } +void vfio_disconnect_proxy(VFIOGroup *group) +{ + VFIOContainer *container = group->container; + VFIOAddressSpace *space = container->space; + VFIOGuestIOMMU *giommu, *tmp; + + /* + * try to mirror vfio_disconnect_container() + * as much as possible, knowing each device + * is in one group and one container + */ + + QLIST_REMOVE(group, container_next); + group->container = NULL; + + /* + * Explicitly release the listener first before unset container, + * since unset may destroy the backend container if it's the last + * group. + */ + memory_listener_unregister(&container->listener); + + QLIST_REMOVE(container, next); + + QLIST_FOREACH_SAFE(giommu, &container->giommu_list, giommu_next, tmp) { + memory_region_unregister_iommu_notifier( + MEMORY_REGION(giommu->iommu), &giommu->n); + QLIST_REMOVE(giommu, giommu_next); + g_free(giommu); + } + + g_free(container); + vfio_put_address_space(space); +} + VFIOGroup *vfio_get_group(int groupid, AddressSpace *as, Error **errp) { VFIOGroup *group; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 282de6a30b..2c9fcb2fa9 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3442,6 +3442,7 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) VFIODevice *vbasedev = &vdev->vbasedev; SocketAddress addr; VFIOProxy *proxy; + VFIOGroup *group = NULL; int ret; Error *err = NULL; @@ -3484,6 +3485,19 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) vbasedev->no_mmap = false; vbasedev->ops = &vfio_user_pci_ops; + /* + * each device gets its own group and container + * make them unrelated to any host IOMMU groupings + */ + group = g_malloc0(sizeof(*group)); + group->fd = -1; + group->groupid = -1; + QLIST_INIT(&group->device_list); + QLIST_INSERT_HEAD(&group->device_list, vbasedev, next); + vbasedev->group = group; + + vfio_connect_proxy(proxy, group, pci_device_iommu_address_space(pdev)); + ret = vfio_user_get_info(&vdev->vbasedev); if (ret) { error_setg_errno(errp, -ret, "get info failure"); @@ -3587,6 +3601,9 @@ out_teardown: vfio_teardown_msi(vdev); vfio_bars_exit(vdev); error: + if (group != NULL) { + vfio_disconnect_proxy(group); + } vfio_user_disconnect(proxy); error_prepend(errp, VFIO_MSG_PREFIX, vdev->vbasedev.name); } @@ -3595,6 +3612,11 @@ static void vfio_user_instance_finalize(Object *obj) { VFIOPCIDevice *vdev = VFIO_PCI_BASE(obj); VFIODevice *vbasedev = &vdev->vbasedev; + VFIOGroup *group = vbasedev->group; + + vfio_disconnect_proxy(group); + g_free(group); + vbasedev->group = NULL; vfio_put_device(vdev); From patchwork Mon Aug 16 16:42:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 275CEC4338F for ; Mon, 16 Aug 2021 16:55:07 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A93E0604DC for ; Mon, 16 Aug 2021 16:55:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A93E0604DC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:54850 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFftB-0004z0-Ry for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:55:05 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37226) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhi-00011D-Db for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:14 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:46090) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhc-000812-49 for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:14 -0400 Received: from pps.filterd (m0246630.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGaE6T000648; Mon, 16 Aug 2021 16:43:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=R+/8Xn6UCRdNqx3X44X9ogFDXli8hym8wKlPB0YaaRw=; b=gZ44t6Zcv54un4jlD4l6Q0u8M07ACkppMosZ9kTN1RCQff7pE75hXfV7+kjd3bWhEEAc KshH+ORqoq5oplKvPMJqH6hBWliYgi24ker3Du3Vuq1OBET4Kw0zfJ9D3GPB//kGE8cv psstx2E2r5PSYkw7oYrG0G56GVh8FCBUJm2J8grlrUJ6vFmhi7HAj09FL6ZNG5oUBl8C rCxJvoVI6NYyjIGkryPMV/XhB6J2CXwRBCTqSgaCRER+/MSXrftjU7+E3vYiGrXczcZY O7D4gzzU0G4NZ0BGrChHJUwTAyu2+T7rrwGT5RAHJuaJJu9NVVtXYyWJQC68SZYO/yUM TQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=R+/8Xn6UCRdNqx3X44X9ogFDXli8hym8wKlPB0YaaRw=; b=tB0wOVf8T7qqJzsrWEuVuFti6WIUY5xch6Di/suWBEIqyfEh5muQNKCCcHcAp7DYZ7Ps MhEXh86rbrDcR2iU3AGRwz4cY0AMU8y1Y7yO6OFoNL5qErSC7QxinUvLDT0L7EEyiWyw 4MZYQXljcJJCgkHQG4H9zad2PZEiIeZSkFmiHDpsVGRWJFsW2f2Bqz6GD8j5xRTl2zXL LW/LmkbxztWOMlbEOkfACP2R/JiRRrPaOhQvHWvB2hWPihjobw0QotHWYL+AQDELtTNc cml79GZ9oMQgj+6TqNGKYozlyffWlpljNQHGdnIcZK45fTBRX8w34Zxr6YEYw/AbFQwa Ag== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3af1q9anv2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:06 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeiuQ113652; Mon, 16 Aug 2021 16:43:04 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2108.outbound.protection.outlook.com [104.47.58.108]) by aserp3020.oracle.com with ESMTP id 3ae5n609mu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:04 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=G/tcG7o7XWVKhFctcQj9Wvd57M30H+0aAvfZuTlZ+hBzCLRooFuyF+hN6sS376uC5oBOarVQOg84png0FJSAezlK8pVUXKKcL+kFdrWWN40lKgS7Po86p/hxdJI387DrImuNjOPFPSg8Vun8w02DSwFHn3gb7MXs6EzlWnVnnTQsN0/ncn+Ya3SvkrtaDh9/TVVwoIbiCg4YhKWTZUna5xenY3wsYBxZzsxRI4X2nU/NqwCQ65/k6XzuLJq2SH8/epNKA1+dfmlSF0/IGQT6FeTDRXKMLpG3ucuy8+rBYnQSaOk7xohQ7lgfHIn/XTsIPS7ZohmpukpmXrXsAtii9w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R+/8Xn6UCRdNqx3X44X9ogFDXli8hym8wKlPB0YaaRw=; b=diNz6X+ZzuLQ0vHfFnnX7caKY8hhmCQiej7B8dKJMI175vGZ//b5OtmFd9K5GaRRBOlamOldQPQLWt2DESdRx5i8pyIgGJbqaAsvhPdnLKO0BsCU6++iEvWFR4chHIyTxa2CR4YHixJic02dPM4YqrpCin4lCNJ/CfckD9cT58rUTTWzBRgwCgTDjA4ULXJZe4bJ7F/Vr3b2+VZQMIXGoLXtPXOmapUFrWLsYgbkuMgn92LpFS5KVt1554af9hlXnaB9s10DG0nMjpJZJ3o+oVMPb+J7w8FGIq5A5WMSF3EFiRtAhlfEfmgws5iTQNC1+Zug1oYvQERbT/SIBa3Pkg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=R+/8Xn6UCRdNqx3X44X9ogFDXli8hym8wKlPB0YaaRw=; b=cX4418KNxnBWESQOfGaPz7y0fMvy1Wz3s0EYZaab49HGVgOVCdaIUCqj5uxrle/hpA+Pb8N45FGCES41qjGVJIswvaqIsvYH0dXdHyOOmdTKl2V7m2w3272sjXGGTqNXi0ErxdH/SGYEeKv9zkj2Vl6Hykjx3i0nWuu9XADzbjA= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:02 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:02 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 13/16] vfio-user: dma map/unmap operations Date: Mon, 16 Aug 2021 09:42:46 -0700 Message-Id: <8960370c1ce02089099702878ed6fb6c3d42ad06.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:02 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 17d240b4-106a-4a5b-bc4f-08d960d4e9f6 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1332; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ec2VJpIvPTbMvh7QaHJ3h1S57cAHAqtd7ZxGFECqlLKzJtTpQq0sJXCkAr2cyRDzEkZmrrFPuz1i40khqBU7td3m6vwE08Dzx5oB0/M0do84f5D4oFN+PubMdA1TCfagK2H5zz7c8DztNl3CNEG54qS3tPia/s9miIRwiUpIWmb1ZLtE2IjBF1OYrhuecTDgFpGH5zTf8z98BZ7aF3svhiv+kH77aboTCbXPjy2FER+RGHyIwTj81xtWHTE0NWoqgzvGJjuieHIe1TIC/FOGeD8AbNCY1CHMnqU/PpHP7e7eS8j4zl5dDnXD7DpJI40oKcNu89mp3IoDKBIWhACyxsNxlrx83lgXmBTvF630wxgOyegkVWHdq9w9Z10T/v1ueboYxD14Lgi8y1+peV8USWAkFWlfwBNn3OStoT6leph7Hz8Izw35NO9WZsuc3GM8CY5xct9/RTOJgB7jp9AfPlPg2MK/7C9rbe1vWGuCFhu+ii8dps5mM1ShftT4aZSV+kYUojA34KQWmY0k8eqVAK5225bKEpMAQMwA7TlSEBXci+7Fsu9Ci1xKTl9/BsjbrB4AvatXGjBloIhukFBKNfKo4aamiSAfZAHqkqlOO7gbaWlF+hqgscUSf4GWJWFwYy+45BBMwjkPwgOzjYgA1g== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(30864003)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: AWnQHpEXy6OCoe91mOoAvAnL1GYu9qG914LZSb+cc8ZuEnF3J5W4ULm5HZ3jAW0MwSNWtMjalw3/8cuJtyyu2tTrorbFkVpRTVmF2VlaVgW6L9xhIkD3LjPrW2YJyqRqQREbTjdiOQkAZDuwbaj8+kqWQN/aS5em/W8fzrmIDaDl832z2N7gTyfmGkVd9E06N53mdhzZ5TbypYa9hH3MczMsLPbbtEcj1/hrl9FclzQMpvCIAgUWNysdgEP8LxKVQvbpGIGNsQunel8XapDec7naPbJ0kM5700q/5EiSFuVyD9NriCbx3Aax8DUSY2w/UOCM76hsZb3WWU2d2v3/14PX3mjLui6lInp7ClfbSNAH0pP44wO+7hFgXIn7i0ZnBmQSE4VhKTVIFsmzSgf8Rw9FGLyKozj9CKOJ/QO7kdNoIyeyQuPuZ4n4FMfEC5u6voQo45QRKq8/trSkKsAEQasE9yby+uW3GQBkf8gHIPY6v9c1X2DaCK6zH3k/BxH0Iyww/HeWM/DpdBnqxD5O8T7LX8Mug5XqSGVYAzAGlAfWyiBZYgZok2LCwcldmVrzDWhC5X+TmyTbiI88v3iSP/ytscnrzYyNUoLDpm3oHBiXT2DHzne/elEVldmlmrp+qbX7P8d/y1ayQfGjJScEFqvVfC+abI4dTLwA/mBDSOp6sabl2J/1gxlrAHscqMZokvSWm0omxHC0BDKY+WwKjTON0pUutpa/wjbEbFjNK0vdFPyvi5bNR1kNvrpRQEJ+qLX2zD/AoRIHjiPs/cX76Em2RBSXo4d6ricrN1W4XMKIxuUMq41U8YQMMWR5oIR+ZFECpVTgHk/iQhV9ri3pYYiZHmtt1Nl2FgBsdjndL1LsNd7uVhmHhy7WucmlW7vTwWFXG66qP+tgK8g1lXJjetU5XzHrxN7sEeQ9zhdu9AFcSmovQ3l8NM5X7K1VVvLxctlj9CnVDVCvgOIL10ymIcKwN3N7HmmFf5Baf04EPrgEj3ENPiI9TrmnkitNjfFMgDyl5PuNmk/zAgQmdw7FduTSZDpG9LwubqW9qtnb645QZL7ndNCV0RZXIK0uRP+DTvGVl5jkdfRkb/+Dx0z/sJ8Gi0A282cCqqdkjfAR7532cd1SkAG2JNkm7cVhIjEDtauVwfhcnI82e98DqPv2rRSFjX63/6wtYggGIeqVljQPeP3brDr8V7s+M7F1OlVZJa8dyhMZApY3hGEbwrqKlQbSDlOm63uqCz4NAh5dpWteN/aT22OO1ERTftxYp1MDw66lsH1ahQoAOs1G8ULS78xTnJU1vMwmoENJWRVJspjt4fZFr/Y5iRQMzPuYbLp4ttJyqOuqIyEA6sdi/1vw+qZgNh2UDvwc5zvycTjjR0SrqvzjRzcmNr0lXkNmK+5q X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 17d240b4-106a-4a5b-bc4f-08d960d4e9f6 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:02.5579 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: SSTYhRKcRw3t2m8Zn/ZsgERbSczrkxIWFX/c5kVeRAku/QSZpkqrao6t0C/zC7eExQ3ghYz54mb78uj1Fm4b20xX9pKVA1K7LXWcOSk3EFQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: AXjR6khvQVIOmDGrzgaIG8VVwaQ_L6pt X-Proofpoint-ORIG-GUID: AXjR6khvQVIOmDGrzgaIG8VVwaQ_L6pt Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Jagannathan Raman Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson --- hw/vfio/user-protocol.h | 32 +++++++++ hw/vfio/user.h | 6 ++ include/hw/vfio/vfio-common.h | 1 + hw/vfio/common.c | 71 ++++++++++++++++--- hw/vfio/user.c | 124 ++++++++++++++++++++++++++++++++++ 5 files changed, 226 insertions(+), 8 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index 5614efa0a4..ca53fce5f4 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -82,6 +82,31 @@ typedef struct { #define VFIO_USER_MAX_MAX_XFER (64 * 1024 * 1024) +/* + * VFIO_USER_DMA_MAP + * imported from struct vfio_iommu_type1_dma_map + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint64_t offset; /* FD offset */ + uint64_t iova; + uint64_t size; +} VFIOUserDMAMap; + +/* + * VFIO_USER_DMA_UNMAP + * imported from struct vfio_iommu_type1_dma_unmap + */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; + uint64_t iova; + uint64_t size; +} VFIOUserDMAUnmap; + /* * VFIO_USER_DEVICE_GET_INFO * imported from struct_device_info @@ -146,4 +171,11 @@ typedef struct { char data[]; } VFIOUserRegionRW; +/*imported from struct vfio_bitmap */ +typedef struct { + uint64_t pgsize; + uint64_t size; + char data[]; +} VFIOUserBitmap; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 248ad80943..7786ab57c5 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -71,6 +71,11 @@ void vfio_user_set_reqhandler(VFIODevice *vbasdev, void *reqarg); void vfio_user_send_reply(VFIOProxy *proxy, char *buf, int ret); int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp); +int vfio_user_dma_map(VFIOProxy *proxy, struct vfio_iommu_type1_dma_map *map, + VFIOUserFDs *fds, bool will_commit); +int vfio_user_dma_unmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap, bool will_commit); int vfio_user_get_info(VFIODevice *vbasedev); int vfio_user_get_region_info(VFIODevice *vbasedev, int index, struct vfio_region_info *info, VFIOUserFDs *fds); @@ -80,5 +85,6 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); +void vfio_user_drain_reqs(VFIOProxy *proxy); #endif /* VFIO_USER_H */ diff --git a/include/hw/vfio/vfio-common.h b/include/hw/vfio/vfio-common.h index 688660c28d..13d1d14c3b 100644 --- a/include/hw/vfio/vfio-common.h +++ b/include/hw/vfio/vfio-common.h @@ -87,6 +87,7 @@ typedef struct VFIOContainer { Error *error; bool initialized; bool dirty_pages_supported; + bool will_commit; uint64_t dirty_pgsizes; uint64_t max_dirty_bitmap_size; unsigned long pgsizes; diff --git a/hw/vfio/common.c b/hw/vfio/common.c index 57b9e111e6..a532e52bcf 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -427,6 +427,7 @@ static int vfio_dma_unmap_bitmap(VFIOContainer *container, struct vfio_iommu_type1_dma_unmap *unmap; struct vfio_bitmap *bitmap; uint64_t pages = REAL_HOST_PAGE_ALIGN(size) / qemu_real_host_page_size; + bool will_commit = container->will_commit; int ret; unmap = g_malloc0(sizeof(*unmap) + sizeof(*bitmap)); @@ -460,7 +461,11 @@ static int vfio_dma_unmap_bitmap(VFIOContainer *container, goto unmap_exit; } - ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + if (container->proxy != NULL) { + ret = vfio_user_dma_unmap(container->proxy, unmap, bitmap, will_commit); + } else { + ret = ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, unmap); + } if (!ret) { cpu_physical_memory_set_dirty_lebitmap((unsigned long *)bitmap->data, iotlb->translated_addr, pages); @@ -487,12 +492,17 @@ static int vfio_dma_unmap(VFIOContainer *container, .iova = iova, .size = size, }; + bool will_commit = container->will_commit; if (iotlb && container->dirty_pages_supported && vfio_devices_all_running_and_saving(container)) { return vfio_dma_unmap_bitmap(container, iova, size, iotlb); } + if (container->proxy != NULL) { + return vfio_user_dma_unmap(container->proxy, &unmap, NULL, will_commit); + } + while (ioctl(container->fd, VFIO_IOMMU_UNMAP_DMA, &unmap)) { /* * The type1 backend has an off-by-one bug in the kernel (71a7d3d78e3c @@ -519,7 +529,7 @@ static int vfio_dma_unmap(VFIOContainer *container, return 0; } -static int vfio_dma_map(VFIOContainer *container, hwaddr iova, +static int vfio_dma_map(VFIOContainer *container, MemoryRegion *mr, hwaddr iova, ram_addr_t size, void *vaddr, bool readonly) { struct vfio_iommu_type1_dma_map map = { @@ -529,11 +539,30 @@ static int vfio_dma_map(VFIOContainer *container, hwaddr iova, .iova = iova, .size = size, }; + bool will_commit = container->will_commit; if (!readonly) { map.flags |= VFIO_DMA_MAP_FLAG_WRITE; } + if (container->proxy != NULL) { + VFIOUserFDs fds; + int fd; + + fd = memory_region_get_fd(mr); + if (fd != -1 && !(container->proxy->flags & VFIO_PROXY_SECURE)) { + fds.send_fds = 1; + fds.recv_fds = 0; + fds.fds = &fd; + map.vaddr = qemu_ram_block_host_offset(mr->ram_block, vaddr); + + return vfio_user_dma_map(container->proxy, &map, &fds, will_commit); + } else { + map.vaddr = 0; + return vfio_user_dma_map(container->proxy, &map, NULL, will_commit); + } + } + /* * Try the mapping, if it fails with EBUSY, unmap the region and try * again. This shouldn't be necessary, but we sometimes see it in @@ -602,7 +631,8 @@ static bool vfio_listener_skipped_section(MemoryRegionSection *section) /* Called with rcu_read_lock held. */ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, - ram_addr_t *ram_addr, bool *read_only) + ram_addr_t *ram_addr, bool *read_only, + MemoryRegion **mrp) { MemoryRegion *mr; hwaddr xlat; @@ -683,6 +713,10 @@ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr, *read_only = !writable || mr->readonly; } + if (mrp != NULL) { + *mrp = mr; + } + return true; } @@ -690,6 +724,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) { VFIOGuestIOMMU *giommu = container_of(n, VFIOGuestIOMMU, n); VFIOContainer *container = giommu->container; + MemoryRegion *mr; hwaddr iova = iotlb->iova + giommu->iommu_offset; void *vaddr; int ret; @@ -708,7 +743,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) { bool read_only; - if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only)) { + if (!vfio_get_xlat_addr(iotlb, &vaddr, NULL, &read_only, &mr)) { goto out; } /* @@ -718,7 +753,7 @@ static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) * of vaddr will always be there, even if the memory object is * destroyed and its backing memory munmap-ed. */ - ret = vfio_dma_map(container, iova, + ret = vfio_dma_map(container, mr, iova, iotlb->addr_mask + 1, vaddr, read_only); if (ret) { @@ -780,7 +815,7 @@ static int vfio_ram_discard_notify_populate(RamDiscardListener *rdl, section->offset_within_address_space; vaddr = memory_region_get_ram_ptr(section->mr) + start; - ret = vfio_dma_map(vrdl->container, iova, next - start, + ret = vfio_dma_map(vrdl->container, section->mr, iova, next - start, vaddr, section->readonly); if (ret) { /* Rollback */ @@ -888,6 +923,24 @@ static void vfio_unregister_ram_discard_listener(VFIOContainer *container, g_free(vrdl); } +static void vfio_listener_begin(MemoryListener *listener) +{ + VFIOContainer *container = container_of(listener, VFIOContainer, listener); + + container->will_commit = 1; +} + +static void vfio_listener_commit(MemoryListener *listener) +{ + VFIOContainer *container = container_of(listener, VFIOContainer, listener); + + /* wait for any async requests sent during the transaction */ + if (container->proxy != NULL) { + vfio_user_drain_reqs(container->proxy); + } + container->will_commit = 0; +} + static void vfio_listener_region_add(MemoryListener *listener, MemoryRegionSection *section) { @@ -1080,7 +1133,7 @@ static void vfio_listener_region_add(MemoryListener *listener, } } - ret = vfio_dma_map(container, iova, int128_get64(llsize), + ret = vfio_dma_map(container, section->mr, iova, int128_get64(llsize), vaddr, section->readonly); if (ret) { error_setg(&err, "vfio_dma_map(%p, 0x%"HWADDR_PRIx", " @@ -1346,7 +1399,7 @@ static void vfio_iommu_map_dirty_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb) } rcu_read_lock(); - if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL)) { + if (vfio_get_xlat_addr(iotlb, NULL, &translated_addr, NULL, NULL)) { int ret; ret = vfio_get_dirty_bitmap(container, iova, iotlb->addr_mask + 1, @@ -1463,6 +1516,8 @@ static void vfio_listener_log_sync(MemoryListener *listener, } static const MemoryListener vfio_memory_listener = { + .begin = vfio_listener_begin, + .commit = vfio_listener_commit, .region_add = vfio_listener_region_add, .region_del = vfio_listener_region_del, .log_global_start = vfio_listener_log_global_start, diff --git a/hw/vfio/user.c b/hw/vfio/user.c index b68ca1279d..06bcd46e60 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -408,6 +408,47 @@ static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg, } } +void vfio_user_drain_reqs(VFIOProxy *proxy) +{ + VFIOUserReply *reply; + bool iolock = 0; + + /* + * Any DMA map/unmap requests sent in the middle + * of a memory region transaction were sent async. + * Wait for them here. + */ + QEMU_LOCK_GUARD(&proxy->lock); + if (proxy->last_nowait != NULL) { + iolock = qemu_mutex_iothread_locked(); + if (iolock) { + qemu_mutex_unlock_iothread(); + } + + reply = proxy->last_nowait; + reply->nowait = 0; + while (reply->complete == 0) { + if (!qemu_cond_timedwait(&reply->cv, &proxy->lock, wait_time)) { + error_printf("vfio_drain_reqs - timed out\n"); + break; + } + } + + if (reply->msg->flags & VFIO_USER_ERROR) { + error_printf("vfio_user_rcv error reply on async request "); + error_printf("command %x error %s\n", reply->msg->command, + strerror(reply->msg->error_reply)); + } + proxy->last_nowait = NULL; + g_free(reply->msg); + QTAILQ_INSERT_HEAD(&proxy->free, reply, next); + } + + if (iolock) { + qemu_mutex_lock_iothread(); + } +} + static void vfio_user_request_msg(VFIOUserHdr *hdr, uint16_t cmd, uint32_t size, uint32_t flags) { @@ -715,6 +756,89 @@ int vfio_user_validate_version(VFIODevice *vbasedev, Error **errp) return 0; } +int vfio_user_dma_map(VFIOProxy *proxy, struct vfio_iommu_type1_dma_map *map, + VFIOUserFDs *fds, bool will_commit) +{ + VFIOUserDMAMap *msgp = g_malloc(sizeof(*msgp)); + int ret, flags; + + /* commit will wait, so send async without dropping BQL */ + flags = will_commit ? (NOIOLOCK | NOWAIT) : 0; + + vfio_user_request_msg(&msgp->hdr, VFIO_USER_DMA_MAP, sizeof(*msgp), 0); + msgp->argsz = map->argsz; + msgp->flags = map->flags; + msgp->offset = map->vaddr; + msgp->iova = map->iova; + msgp->size = map->size; + + vfio_user_send_recv(proxy, &msgp->hdr, fds, 0, flags); + ret = (msgp->hdr.flags & VFIO_USER_ERROR) ? -msgp->hdr.error_reply : 0; + + if (!(flags & NOWAIT)) { + g_free(msgp); + } + return ret; +} + +int vfio_user_dma_unmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dma_unmap *unmap, + struct vfio_bitmap *bitmap, bool will_commit) +{ + struct { + VFIOUserDMAUnmap msg; + VFIOUserBitmap bitmap; + } *msgp = NULL; + int msize, rsize, flags; + + if (bitmap == NULL && (unmap->flags & + VFIO_DMA_UNMAP_FLAG_GET_DIRTY_BITMAP)) { + error_printf("vfio_user_dma_unmap mismatched flags and bitmap\n"); + return -EINVAL; + } + + /* can't drop BQL until commit */ + flags = will_commit ? NOIOLOCK : 0; + + /* + * If a dirty bitmap is returned, allocate extra space for it + * otherwise, just send the unmap request + */ + if (bitmap != NULL) { + msize = sizeof(*msgp); + rsize = msize + bitmap->size; + msgp = g_malloc0(rsize); + msgp->bitmap.pgsize = bitmap->pgsize; + msgp->bitmap.size = bitmap->size; + } else { + /* can only send async if no bitmap returned */ + flags |= will_commit ? NOWAIT : 0; + msize = rsize = sizeof(VFIOUserDMAUnmap); + msgp = g_malloc0(rsize); + } + + vfio_user_request_msg(&msgp->msg.hdr, VFIO_USER_DMA_UNMAP, msize, flags); + msgp->msg.argsz = unmap->argsz; + msgp->msg.flags = unmap->flags; + msgp->msg.iova = unmap->iova; + msgp->msg.size = unmap->size; + + vfio_user_send_recv(proxy, &msgp->msg.hdr, NULL, rsize, flags); + if (msgp->msg.hdr.flags & VFIO_USER_ERROR) { + g_free(msgp); + return -msgp->msg.hdr.error_reply; + } + + if (bitmap != NULL) { + memcpy(bitmap->data, &msgp->bitmap.data, bitmap->size); + } + if (!(flags & NOWAIT)) { + g_free(msgp); + } + + return 0; +} + int vfio_user_get_info(VFIODevice *vbasedev) { VFIOUserDeviceInfo msg; From patchwork Mon Aug 16 16:42:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438935 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70B6EC4338F for ; Mon, 16 Aug 2021 16:51:16 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CF8FB60EFE for ; Mon, 16 Aug 2021 16:51:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org CF8FB60EFE Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44692 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfpS-0006fT-UI for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:51:14 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37214) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhh-0000yp-JW for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:13 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:55508) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhc-00080w-As for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:13 -0400 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGbX1f029459; Mon, 16 Aug 2021 16:43:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=nmiwh8p0OoAJzqrtdn4jXK/dOs6qH6Bv7KKpzspOa/M=; b=PSk1f6/CLsx1HgpvgJAQpYRz59RlcLV4S4f6cltJiJEZgjLt1j93Sb/erPSfQUP4hbcH bQFLF7yf5d9GtpYuVTO2EeiwhwxJv8p17Lw+yYKKK5GMbZh+2eKEPFYzh+mKuQCx3z0t jSu33eZEA28S+akFDXwASGB3xLXfI9NoMySMScHQJzXVlJbYnFn6owkxiAOwu90NR/kX jOSGkCzfRv5S1hzxchKSpHdv1GSiZajrbL4gLUAQG3Kx2OB/jQA303CrfO8+Jw6T6EKB UYSz2SJ2BS54oNFYRUvle9XcY5iEhXwbQHzEXX29S16XzFC4IRrcLqiU4le7sv2iaU98 XQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=nmiwh8p0OoAJzqrtdn4jXK/dOs6qH6Bv7KKpzspOa/M=; b=GCWaF3au+n/ojY+lKyDmwJtODOhRRh5FxzRLPSwqYgXPIL46hddQYDW10CDHmyIotx78 FNKppfnONxJbGAXsVO0BDiEBpYe1Db4zRFw3svQg94OTla+B3T/4XYcYeyGTsP0r9KtH LPR2rX6juNAIHJ3BuA/LRMs+j2MNmvPgvObAvxdhxmOWktVwy+xdDcRpf9ohyGdS9kqA 1rIH3OEcfg4T6Le0V/GcWoTZfCYj2o5KLQYKxA4Dz1qCTavLlTP764fVDQT3oUrsC9Hp 3zD/BvZYZt/Mb8w1KIh/Te9F7FNJiHTepK1tIUSd6KGkE2Sey/r8D1mWEkSwFhGNtiAO 0A== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3af3kxtg10-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:06 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeiuR113652; Mon, 16 Aug 2021 16:43:05 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2108.outbound.protection.outlook.com [104.47.58.108]) by aserp3020.oracle.com with ESMTP id 3ae5n609mu-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mONJUyB1nRBK/X7KQnfKF8Bp1vrTfbu9wmBeJCDFcds8QfUZbfOJQXcy/BWu36D/SBxWO816wh7Jkw6bYfdS1ax59FUJLHrLYNQXLIuk4XoUHK3dfIav9Fxd9O9d9JAGd1OWI4HACDX7UZcXBThBRyLZ9iiP9rZabIDsR0MWlCMKGpr/2kaACnAPiDm2EkfyQCupMXHKCFn4n4bbWPBmhw3mW/h8RwhGdHm9m5kJjliR8125NQgcmz97wwbzBQ/SHtuVSC26Ys1JrMDnHScI1Y9Q0cUSMkcS4ktfd3wGEl6mmuXX8zrflxFHabgpY35JIuWJ9ntCf37oWfzPHKUhzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nmiwh8p0OoAJzqrtdn4jXK/dOs6qH6Bv7KKpzspOa/M=; b=Xbwpnuu549emb2MxaDOIUCjn7w4MLXPorQmvM/X7kM72DdHJIONSh/2oFelKmodcaioIEOkmZzaD1ECDbjGA+2L68B7tdb+wLa7vfhBxNHE/W6OzbRZKrQ/DNGcEwTtU/+o5416cSbtSwLAsfzUWmtfqTGSKJfnXwVlDeOvOs7uvWRlQwiVFG50sFSggER3bUv3zbagoWdy9ZiblluR/JZc4rApkWrPb8ANGMj0Fq0VoFvnkSjcqHicWea0xZwBOOwaxpsNBqZ3rVOD7DDofu8zjsroW5JVTZSerMeOgiNzdcjdhpCXlV4vhejNbEdUQ3mhDWoCkunPiBVPLCYQG9A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nmiwh8p0OoAJzqrtdn4jXK/dOs6qH6Bv7KKpzspOa/M=; b=u8NKmG1RC66ISucnQRrWWq4/ShTwdbiLW+9Z5VNTt1RnXNWCSwC5SHl74R3+wDE3Xkx+w10qLgMiZHHtbGq2OABHZRd8tKTYHRhffSAScirJ8lPUv4I72peK2V2J5H7CqPRxSm4hu5xgLBEH+1hlekkn3eXW2UAUcVX+SoHOf4g= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:03 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:03 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 14/16] vfio-user: dma read/write operations Date: Mon, 16 Aug 2021 09:42:47 -0700 Message-Id: <7b21118256af2cb3d0dfe45b1e4ef9683fabccb5.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:02 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 321be7c0-0342-4b19-6423-08d960d4ea49 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:962; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ynqDGncm38DR+tjBZEW7cEQtxSzNwsWYRVhG2tktWOKz4hMHMsLOzIXZZysG6cc5IxwD+fdVL/UdvCCQ7SYPl+wQUSR4IGAFLFsPUYbZ6TWFVQF/9qDQqxo9Erx8CHcIM/u+7DWPAxh+hS4UutsMhPWYEfUV3W3m1X4DTYGzLGCCX1WsY0HOfApVfPdRkhY9suFbG+a7IGpimrx9PTab9ZHMDGLQGd9hlCnmrcqibRQDwP6lBYYQWQI0Z1yB6pE5wxAx+3xWq5Dlm0xcYt/Y5eECyfONnXlsIe49ong5KKaoTVWyB1gKgtAummPoBblVHuH5FEAfvXwv3tM+QNBQH9tkaY58MmLs9CB+J3I+6VUNYkz+bahYLF9skmz0ucRPO+uKyjNFRKtS2lV4boJvmSBYqYomHw58ni0U+BEaArO+Jg9bAdoqXocbgTu6f1e2AjApA+7Q99EyETSkWuPx4BiMlPIWa0ysLmZrCZ07HTgWFEwpFhrZnCBGrfuR3iVn9p5w9CpPvxjs5RCEwwq7C+/nEHNsK+cMUbPPVcNXGlx0wI/gavCmci/0BgCGAI1roeC5nuKQA0plMAapwZAcEOHLxxFz5rILq4CTlDkTFfOQ38pSsJSi5VEePFVoUP8x+UHC8bG0+CsjZY807vi3pg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: mWCH18PknI++ABn1aurzv0Gj0sqzJBftp8akXBsBnmzsEN21HnY66oGRRctv+MRPkt4DQJ168ExueN4TdQdnUD8WTN32FfatwLIOJGKh3s36C7utPbWVGNIt2NfjhRDYl+WKwcA7rzHA8iKX0R08Lab1SI1Jv+eVHvRfPMc/glnE60N2qvgwQnYrQ2HhYosfN6scRGRdUzf2KXHcBDbvy9hxg4MKy5Ppt/pPShv5toD/YXxKhUyBSwK5Slsy0O8aSmW8hbxpsJyKchU8M24aIjecjlsqJa4xgDmBVqfjU532v5iD7ldwe8bk5mViz+zVn8x9ihMdjj78qLTMRaux17P/DVXUMRF5jDJ56Y0OJyItZiLjWGH1FFnvrI+f7K1W0mUPgt8JUxTzgoZgcbbMd1XZUrGUcuuN6m99JNjZWY0nc4bzJzwpV3Wj0XIarwTXhjMdkK77yb+7Hxm8nZqaZrB1+mYKLjlyD7qhqOciB+gEFp7n7Uw8aNipqf9tAjYBBICfl4rpzrLaowfhZyQ5Ki7R17+zRl+4t9IIUcntBhBwLZ27rBFXV/WjdvsKzY1sJLkbntje46R6uX/8tiQvg+8sy7CKeOOvKWnv+D8TzHJtdRR8o+fwHy2v+oIzAmhK3HneuwryTpzn5SD3y2PbtsReJdK5OESbCeQuTx29fpvk3jV8GOtnMLUupXVcLFN/eOXqGZLVHjqS55LdYof1HVwOLmThCrTDJ7VvTp87p+nyB6VZvRCvd9KtfKONFl/Mvq0HPIplPoxqVzc/ZiFkUBOSWzlB27zYXUaYoMqv0aKJDSm8cVXxSHgbi02ew/y5kWoAgdddlwJtdknXj9VkP+FaFCuu1mofwSAhOY88kQoFSPPDYn8I6lWklPeWEi9s3/NNbV7c+3N0+GlVZhKvHF/pRukac3083dLVvmw3oSyZxEMsm2yugZY3BezxkqazCPZFKOFCKK2ldwZf6+5SvcjTHtldTCWUBiZf0gDj1chvPRnc7v4FhfHoREDhFscrW5OxGOTXlLWOBWSMHKZxuz1omvufvjAsNtCaT1A5N483aV+uYvAexCmCvRkOrpM6JDh194W4i6v9JQrVX1B0HDpJmg75LBe16KHz3YAjeNFaySX12h5fXfTCnIm07rgjzvfQDS5MNz10haLJ41xgxGAQBjoioCXYyyjhxMX/MlvSrsfuOBuOwDSZDHWJiKMhThIkGg/RHn+bKRSVn0ax+RyfHBTXj4OGk45YHnwRqSXH5pna/q4xuakVV4uxTT2sCprf6cPdP5D2IHBalEDfsIaC+Qv3RC/2WfLo2RPrzbX+4oZeSdcP+k3ew0feLN5zPdNdd0vo8gByRBHkrMtoEx4im6Z1VLn8QjPE8V/ORLesYU0lw3OriVdvyJcpWcN1 X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 321be7c0-0342-4b19-6423-08d960d4ea49 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:03.0876 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 3TwkrLXeAn0/D7mhHBsSZ+w6+P29Wch9/zYC9DgMtsFS1CRI7CCxVpAZcivsCntndLbUugFT5tyk7oYPrYNQoNiZ5L6MKhvcTHWwxC6IzfY= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: 1BRbLtOUmRBkw1CHLxE7Rjz3WMV4-ent X-Proofpoint-ORIG-GUID: 1BRbLtOUmRBkw1CHLxE7Rjz3WMV4-ent Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 11 +++++++ hw/vfio/user.h | 1 + hw/vfio/pci.c | 63 ++++++++++++++++++++++++++++++++++++++++- hw/vfio/user.c | 7 ++++- 4 files changed, 80 insertions(+), 2 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index ca53fce5f4..c5d9473f8f 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -171,6 +171,17 @@ typedef struct { char data[]; } VFIOUserRegionRW; +/* + * VFIO_USER_DMA_READ + * VFIO_USER_DMA_WRITE + */ +typedef struct { + VFIOUserHdr hdr; + uint64_t offset; + uint32_t count; + char data[]; +} VFIOUserDMARW; + /*imported from struct vfio_bitmap */ typedef struct { uint64_t pgsize; diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 7786ab57c5..32e8b70d28 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -65,6 +65,7 @@ typedef struct VFIOProxy { VFIOProxy *vfio_user_connect_dev(SocketAddress *addr, Error **errp); void vfio_user_disconnect(VFIOProxy *proxy); +uint64_t vfio_user_max_xfer(void); void vfio_user_set_reqhandler(VFIODevice *vbasdev, int (*handler)(void *opaque, char *buf, VFIOUserFDs *fds), diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 2c9fcb2fa9..29a874c066 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3406,11 +3406,72 @@ type_init(register_vfio_pci_dev_type) * vfio-user routines. */ -static int vfio_user_pci_process_req(void *opaque, char *buf, VFIOUserFDs *fds) +static int vfio_user_dma_read(VFIOPCIDevice *vdev, VFIOUserDMARW *msg) { + PCIDevice *pdev = &vdev->pdev; + char *buf; + int size = msg->count + sizeof(VFIOUserDMARW); + + if (msg->hdr.flags & VFIO_USER_NO_REPLY) { + return -EINVAL; + } + if (msg->count > vfio_user_max_xfer()) { + return -E2BIG; + } + + buf = g_malloc0(size); + memcpy(buf, msg, sizeof(*msg)); + + pci_dma_read(pdev, msg->offset, buf + sizeof(*msg), msg->count); + + vfio_user_send_reply(vdev->vbasedev.proxy, buf, size); + g_free(buf); return 0; } +static int vfio_user_dma_write(VFIOPCIDevice *vdev, + VFIOUserDMARW *msg) +{ + PCIDevice *pdev = &vdev->pdev; + char *buf = (char *)msg + sizeof(*msg); + + /* make sure transfer count isn't larger than the message data */ + if (msg->count > msg->hdr.size - sizeof(*msg)) { + return -E2BIG; + } + + pci_dma_write(pdev, msg->offset, buf, msg->count); + + if ((msg->hdr.flags & VFIO_USER_NO_REPLY) == 0) { + vfio_user_send_reply(vdev->vbasedev.proxy, (char *)msg, + sizeof(msg->hdr)); + } + return 0; +} + +static int vfio_user_pci_process_req(void *opaque, char *buf, VFIOUserFDs *fds) +{ + VFIOPCIDevice *vdev = opaque; + VFIOUserHdr *hdr = (VFIOUserHdr *)buf; + int ret; + + if (fds->recv_fds != 0) { + return -EINVAL; + } + switch (hdr->command) { + case VFIO_USER_DMA_READ: + ret = vfio_user_dma_read(vdev, (VFIOUserDMARW *)hdr); + break; + case VFIO_USER_DMA_WRITE: + ret = vfio_user_dma_write(vdev, (VFIOUserDMARW *)hdr); + break; + default: + error_printf("vfio_user_process_req unknown cmd %d\n", hdr->command); + ret = -ENOSYS; + } + return ret; +} + /* * Emulated devices don't use host hot reset */ diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 06bcd46e60..fcc041959c 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -54,6 +54,11 @@ static void vfio_user_send_recv(VFIOProxy *proxy, VFIOUserHdr *msg, * Functions called by main, CPU, or iothread threads */ +uint64_t vfio_user_max_xfer(void) +{ + return max_xfer_size; +} + static void vfio_user_shutdown(VFIOProxy *proxy) { qio_channel_shutdown(proxy->ioc, QIO_CHANNEL_SHUTDOWN_READ, NULL); @@ -251,7 +256,7 @@ void vfio_user_recv(void *opaque) *reply->msg = msg; data = (char *)reply->msg + sizeof(msg); } else { - if (msg.size > max_xfer_size) { + if (msg.size > max_xfer_size + sizeof(VFIOUserDMARW)) { error_setg(&local_err, "vfio_user_recv request larger than max"); goto fatal; } From patchwork Mon Aug 16 16:42:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51BB8C4338F for ; Mon, 16 Aug 2021 16:50:57 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D027F60EB2 for ; Mon, 16 Aug 2021 16:50:56 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org D027F60EB2 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:44254 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfp9-0006OB-R7 for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:50:55 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37176) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhg-0000vD-Ib for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:12 -0400 Received: from mx0b-00069f02.pphosted.com ([205.220.177.32]:45702) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhb-000810-Pw for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:12 -0400 Received: from pps.filterd (m0246631.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGZfZ6008090; Mon, 16 Aug 2021 16:43:06 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=gK+Q0/rA5s16W+k1rf3tURI3UrxB7LD4G/Aog3Zoeyk=; b=MKNTClW7qizuI8SW16g1By3eofYdoZmuzg/eMd+HXURfYRfYHHyjpPZqoZw9CQL1vTxm zED8psAEm8f9FzVTmUCKZWr+RPOAwZhPYt2AgHF/4W/XR9rKNZgkAqddFBjDj8wQSNVe M+RLqTKoLOGVZgzzHTHF4NHcMqH/KrqVtyGqX2Vvph5kPDUiaRbzWjqAPMdmBD1bn2KO cqRb8O8w4xhk0ZpFU7JqveSCjbRFye3IGf7W/J7zKwdp01FJ7h7U6y6hcOAHPreTeOgZ 9EBdNL2kbWW6q9+Iyy72HfYjc3pA9sJFSHJ9dtU9eFbmPlOgmsRIifhpWqGOp6eBZXqm CQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=gK+Q0/rA5s16W+k1rf3tURI3UrxB7LD4G/Aog3Zoeyk=; b=oC2GmxKr7MtYe4tyE15V07HfaNwAPj0jyWEX+kNfJH2aL2zafQ4/YG6iX14cuDvmkJOh 6jcXB7nVPQp+B5jvHTrUg2j/SaUtvJGWljmDOeQpVr1XjqXhgB4I0X4QvYukGDzT8PXn r3JlEUHpjij5U7hbyeJFS6j2rRm14HtMw6sDMWaxgsNekp3An+qO5W/V5l8JZbsSP9ho 3XQkJ4+LkjSUIuWpWI7lCBz9wMuRZ0j3f1S5AObmyQDKQhAJvNKfUr1nMWt5T3Z2fqkh WKHk43Y6yQY9r339lPNBidGQ7F4gronWiLlclbIpuOSCsmrgLpIIzADjW770le1JiLF0 Qg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3afgpghjq2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:06 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeiuS113652; Mon, 16 Aug 2021 16:43:05 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2108.outbound.protection.outlook.com [104.47.58.108]) by aserp3020.oracle.com with ESMTP id 3ae5n609mu-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:05 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Hc5hcOl6Fv7y2iJPiQjBIFoFphWKJeBt2KDsL1XIKJVrfMkd96MmSYjQkYv0ZNuyq51AC7T5MqyVXupogba15KoIxZnsJADd/FpgVpk5OtUTEPwAMUCezp5/dHb9w3xtpZh/k0o7EMN1HwCURVbsJD9vBlqvEWX1YwfFuTBl/RZ1yYlvQNlZZNxa0WTT994wqX5yJBXGrmvLh2UBlKcvEFTiq7X2KVSbJBgjwmjs9L2CiwGQM+lFOyg/xbQMhOhx80k9gxFxQkjWQTz6iZha1MjIUe2Nxdbp0RtOVpDEmx0/N2lFWsS9kmkPneaZUXrxPEJ/ofe/CyBXdQs2SEFAhA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gK+Q0/rA5s16W+k1rf3tURI3UrxB7LD4G/Aog3Zoeyk=; b=Jq/i33fjCRnQza2cO9Jvz4kR2ph03wytaT1vhOdaWeTSel+oa8LuKMGJ5HMxwwtLg4r8dZdDX4x7KY4hhenG6MzGfHw+Pkaf5m7HvckTXu2sdYxu2SCN/5b96CMnoS5gc+//TcYV8CV1ESOnZXp/cSD4u8RPPK6hWh9q5fqS1FYXZkOZACHCI24HgkVCIhrQUqS6yjFLv2hLzaKfCh7IweXcUkjuozK/Z6o9AoCI1ym3H9YV9QQStqfEbuIGxW5sWWj/VDpGw/ntJfIU3fwN7gz2FN/V+my35Eao/nigP8sg8J8aWgfh2elNYyo3rMt6BHRNSC4CmuMLC9I8zZ23ig== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gK+Q0/rA5s16W+k1rf3tURI3UrxB7LD4G/Aog3Zoeyk=; b=oiyHiytyDCe4diT+Fg5GE8fvu6ExLCJQK5OKBCh+Drs6LNw9/df2ney/o00jMmtxAWOCBLGoKELNBOccn1mbFAABzrWU+GMZE3eGCVQpabnvV2cmM77UIKhJB4JVUXoNXwPrsdbc0HAeFdfcDWQaLZyaogahXSywSNGZYTKzXS8= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:03 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:03 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 15/16] vfio-user: pci reset Date: Mon, 16 Aug 2021 09:42:48 -0700 Message-Id: X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:03 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8aed286d-24d3-4e40-2ab4-08d960d4ea97 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:183; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: E0U5mPtOVIU/v5M/IPeb953MiTn/2s+XkFwWAktW9wCsXiFAxXYFhTwTo6jdDyH6bprUjpVGWv9zs6KX0uJNLWTbvbfy159ArFvQZ7Gsc4DXScAjjlaE180n/yU/5sUxm5Lj+D4iMeVjJJsS4+WK7AJFOX9Ys0ijWqtM9bSrCc/44rxj0ijnrhCRZgUJ92T7XFmReIt1x8+1h1kztlvjU/xXuP7cuA8D5ys9CruHJs4AebMlMn0m/JPV1ChdpTlUyYpOFzas6bqgeFLEFFh82IurfwmfpSEqOujbBajqCvLJkopCX0JongMQ61VJaYtruJnNTmKzah42RA/mA2mYGR2vcTv7RJQIcelWr0pCgbDmdIbN4wqLsHAkcZEBQ4O+hk49ntPM7ZURsMNKi1J4OBZqXCT36lYJ5MMePcuCOSo/RAg6PusPdZu3VZpzAt8faLEk7gdGaNC8Y4SZbahp36wcx8bz67HafTrYVeLZxyR4A2YOpfY8gLs7vldjyz7G8dgaCou9ZYh38JsSe5AMUhADyMOs9tSJnV63luUllNMxXlxCcBQ82GC/9qIm/L7YyeD36BnFSAB3JMID21PXzwMYhR66P9O5NPACIgMF4lDLJ78bNdaSzYls4C3QevBSILCE37FtdwSN7X6yNGc3/A== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: xGwKZQqyRzSCxLG7+EHJ43cz8k5VaIpsmD4MLXouwmopVbsc0rSQmk90lMJwtec3Jqf50b/5M5S55iH6IurJX6eJ0ZID1t9AHs/toWtKjwpeVUoeJpzbXwkIKokXF3ftZXq675IRZOQcYRIwWzNye7212CVc6rj8DGACGj1n67lJH3GTV0Qtfk2owyIidrhCa1lB0ET50m6nrMcQ/icbx1DQ5Kd+oH/uPyl+PwJSAts7AZYnsKHo/xArOXUIadgtAYrxY/TwL67NXt6YxfPMqQS7JkBT3wOdzVu3+3ZQI5sFAXcU8A2MnXyXJK0w3ekb0BP0fH4JjpcbinXTTuuTJt+C6brmR37pywFH7mXCiEktnfNZ407CVa4fQFRTfSzCJ1wZrH9JPRaAhD/k2PMoxFeTFMPFy0M9JyTPKR27ewUQPC2H2NaLiaE14FbleqflaubmTsNy3b/zvkHCH//TalNDXXjSAh6Kb17sBX5xDepIOg6TnO5qCkCPael6B2FkxwfuzSj0lI/NcXNZ4TbR/oS95HS2urCgtAzG/2fbYugbGehgo8YjLg54tbnWMaHuA0bDeKmApHxIOixlUjRI7NCQjq12nIG/Z5YU0L3HgMdxWzd+LahHcuRR07pYh6Ad+aHoki7YS0m+75/tkCKkG/qj9xRFcdTIpITA7TcqRjs4lxN4LSxqbzGYFjXxzLiF2vg0Q4rpFqtrOtwZBjUHMGtCl21es9V0JY7OrOIz4LTvTeQVwJz2wRpYf4Jz/fDfraLpLIlPdfjAtcp3LmtZlX88s1LXM5rZ17YtdaDHzpSUsJMqidoXWeLpnjfSmIRWM1S6ahyHsnOUlhkU/g0lmtLyh40HSM0XIpw9x086kbqJW5EL13Xy/Jr+qFSJ7V7t38y7OgPOYK+tt2DfSoTee7ps3R/dDScUPp/xXys/tJokdHinnA16fIdra7gGuYM6RieYx4lFfwUhNuyoC3+x5dOdXbfqfGlL2GId3cPbZeLRFiS8x7vQq9xcsn8Bmg22prSbdlpGouMthRjJNwTC0WlCszVD4w4G/2G17CcexyKhbN/WROayLqEVDkfGbU+2e0oZH7tGd4d2Ran27Al2Ba7pIVofAl7nY/KCq0evDFeuO7gU1MikoLrzBjZPa+rT075nIVANPJ9ZAGcczBtIFbqsN2W36GaMkXaVihr2LznahSgavOvXwFFILYr3ho34UREbODvs0PSfnhD4ExgQNOIkZHwykiDDPlq+iPYjxyndklMDyk7cqxlw2LygawgocMA2k6rFn5geXZB4PaGBt5fkkOfL2Z7PDGcndrr9miG5vgYNZIjtYarAViSiD2ISY+z0r99hQLU3mXQyyOJpzRaQgcZmERNFUA4AzaTFNN548ZagNlWTtgt6MkU9vfKP X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8aed286d-24d3-4e40-2ab4-08d960d4ea97 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:03.6013 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YvdoONj2+iKXTj1voFcxQfI1GGKR/jdlEXhLeWCEuLS+uOCcpsr8+gfiMm7gLc/PlFa/UL1zRaug+98IfzHA7nSVKKtV126uqMWTf3+6I2A= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: iI5wjs3fZpTrd9P67XQLNCynDIJL7nSk X-Proofpoint-ORIG-GUID: iI5wjs3fZpTrd9P67XQLNCynDIJL7nSk Received-SPF: pass client-ip=205.220.177.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0b-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: John G Johnson Signed-off-by: Jagannathan Raman --- hw/vfio/user.h | 1 + hw/vfio/pci.c | 29 ++++++++++++++++++++++++++--- hw/vfio/user.c | 12 ++++++++++++ 3 files changed, 39 insertions(+), 3 deletions(-) diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 32e8b70d28..5d4d0a43ba 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -86,6 +86,7 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); +void vfio_user_reset(VFIODevice *vbasedev); void vfio_user_drain_reqs(VFIOProxy *proxy); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 29a874c066..4b933ed10f 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -2229,8 +2229,9 @@ static void vfio_pci_pre_reset(VFIOPCIDevice *vdev) static void vfio_pci_post_reset(VFIOPCIDevice *vdev) { + VFIODevice *vbasedev = &vdev->vbasedev; Error *err = NULL; - int nr; + int ret, nr; vfio_intx_enable(vdev, &err); if (err) { @@ -2238,11 +2239,18 @@ static void vfio_pci_post_reset(VFIOPCIDevice *vdev) } for (nr = 0; nr < PCI_NUM_REGIONS - 1; ++nr) { - off_t addr = vdev->config_offset + PCI_BASE_ADDRESS_0 + (4 * nr); + off_t addr = PCI_BASE_ADDRESS_0 + (4 * nr); uint32_t val = 0; uint32_t len = sizeof(val); - if (pwrite(vdev->vbasedev.fd, &val, len, addr) != len) { + if (vbasedev->proxy != NULL) { + ret = vfio_user_region_write(vbasedev, VFIO_PCI_CONFIG_REGION_INDEX, + addr, len, &val); + } else { + ret = pwrite(vdev->vbasedev.fd, &val, len, + vdev->config_offset + addr); + } + if (ret != len) { error_report("%s(%s) reset bar %d failed: %m", __func__, vdev->vbasedev.name, nr); } @@ -3684,6 +3692,20 @@ static void vfio_user_instance_finalize(Object *obj) vfio_user_disconnect(vbasedev->proxy); } +static void vfio_user_pci_reset(DeviceState *dev) +{ + VFIOPCIDevice *vdev = VFIO_PCI_BASE(dev); + VFIODevice *vbasedev = &vdev->vbasedev; + + vfio_pci_pre_reset(vdev); + + if (vbasedev->reset_works) { + vfio_user_reset(vbasedev); + } + + vfio_pci_post_reset(vdev); +} + static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false), @@ -3695,6 +3717,7 @@ static void vfio_user_pci_dev_class_init(ObjectClass *klass, void *data) DeviceClass *dc = DEVICE_CLASS(klass); PCIDeviceClass *pdc = PCI_DEVICE_CLASS(klass); + dc->reset = vfio_user_pci_reset; device_class_set_props(dc, vfio_user_pci_dev_properties); dc->desc = "VFIO over socket PCI device assignment"; pdc->realize = vfio_user_pci_realize; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index fcc041959c..7de2125346 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1045,3 +1045,15 @@ int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, return count; } + +void vfio_user_reset(VFIODevice *vbasedev) +{ + VFIOUserHdr msg; + + vfio_user_request_msg(&msg, VFIO_USER_DEVICE_RESET, sizeof(msg), 0); + + vfio_user_send_recv(vbasedev->proxy, &msg, NULL, 0, 0); + if (msg.flags & VFIO_USER_ERROR) { + error_printf("reset reply error %d\n", msg.error_reply); + } +} From patchwork Mon Aug 16 16:42:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Elena Ufimtseva X-Patchwork-Id: 12438969 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CEFFC4338F for ; Mon, 16 Aug 2021 16:59:05 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7C47F60F11 for ; Mon, 16 Aug 2021 16:59:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7C47F60F11 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:34630 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mFfx1-0001tv-NH for qemu-devel@archiver.kernel.org; Mon, 16 Aug 2021 12:59:03 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:37230) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhi-00011t-N3 for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:14 -0400 Received: from mx0a-00069f02.pphosted.com ([205.220.165.32]:57618) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mFfhd-00081V-5a for qemu-devel@nongnu.org; Mon, 16 Aug 2021 12:43:14 -0400 Received: from pps.filterd (m0246629.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 17GGbZUF029485; Mon, 16 Aug 2021 16:43:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2021-07-09; bh=dEIQoWq1svlzZmaE3BBG0U9ChO0Ig3r7lzdRFu1zmXw=; b=OKDYLn5jlPLLKJw0oYntG/Vss9HE+NyQ8eODHM9n6tq6/5c6RfZjKvd9xlHhE/Vrwg7E Q/bdqrzKAbbHC66RhBNojxR8Z2FZJlCt/98DkRTzt9M8bEoHNQfDueajU6bLnHBD22rY iSMqel09Ps1E1xPvUoAf4+tqd0C/ta+m2Vls5NXRT0Fg8dorDAE3lpdWAnT9dnZTYr8C gOUiiQIDOfSCfhEhU1N0K+U3J5o/6Dn3eMnZSmjzGXwXt0M71DqdY+1fXzMoN9DR60aL mor4UvQlcLWYNtKBhd8LzJN5dmsN+BczaD68YlMlQHnW/mZzOjjUvCPs0P6y2G1oqWEk FA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=dEIQoWq1svlzZmaE3BBG0U9ChO0Ig3r7lzdRFu1zmXw=; b=hlG96/HiLKSUUcNKBHlXx2XrOAJu9O3TkzmzdKakGALAdMFpd2yNaifLVh8GxvPK/yRJ 7lSg6tG5FxGVVWdeTWPh1f8w6Quf4EvOBtRU6aDx7jwD6IdxGTPiwE5+SiA24iZzooMc BcZJoe+mxN33VND3LEFXkwZeCj2aHiNI2XVls2xBO37OUQSBjQu3v3Qddkyqb9ksUl5k Hn4auw5p5fUpw0crhuWgBGKm/6QIjY8T+oVDu1BHJHat6RIcgEv4+tw1Q3p+mShOJr9H yMnMzZ9exTHeTwQm3ptclluUvAMv004JX+D7F+FDVaUDVNCN8jkrCUcU8x4pjyWWU/7j 6w== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by mx0b-00069f02.pphosted.com with ESMTP id 3af3kxtg13-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:07 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 17GGeiuT113652; Mon, 16 Aug 2021 16:43:06 GMT Received: from nam10-dm6-obe.outbound.protection.outlook.com (mail-dm6nam10lp2108.outbound.protection.outlook.com [104.47.58.108]) by aserp3020.oracle.com with ESMTP id 3ae5n609mu-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 16 Aug 2021 16:43:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=WNQyAwaxvOwS0hknTBmwjhzYQwwIBuQPe0EDuCD+fdRdsDf36gPUD2Xz6EndfLAwknpkrbKbjevyjNSmdnjUf4PUv1Axb1dvkvUsIaoV5P5ZdAcBBRJI+W8MTuBdoey1gRICveSjl4q4nKAxouDV3hxBAiH+F16VnTz/kxpWGmr59OCGKeX57pCppDQg+1Ez5my9qLCWiENlurZfAD7e+VTAKnKfVapZ6uroSJQLZiMg3crC7Ib57urr/qZ0OiWwUDdtuxo0Z5y8lKm5PqPTh0GhuiFsWcC6MY807vMK6WvXL1nX9e8AaJvKPmmCwwvVFWpN79Xwz8R5UE+jUoA21A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dEIQoWq1svlzZmaE3BBG0U9ChO0Ig3r7lzdRFu1zmXw=; b=aAp/X6AX+V2iTJsxQbG1xLOwOv2xAzg/r7FKfwmqBHJnmieQc2v79TmQbvFKCu2Hz7uu4DDyczglgpOA0jip7nxiSEu7UKWfRRXRseaTDroIzfbDrpNVRiVdXZPEbIioT6Belk5CXbXR4wA/+QLqoCH6JKE1YvXKp0P9OpAr6z9qxop0ET6YsePNmtsYYK/2rlhSPoPmz3372HTiMMtc6i5ftx3S23rAxaeyKWrj0+5KomPB2CvGo0S/sleq60m++wUrMQUIToUJDaymO42oZVxuIPLucDsQnHiMzkOJTUYquENN7Khh1CZvi90m/tOjzCwaNH5GvcNyVLHKmO1FeA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=dEIQoWq1svlzZmaE3BBG0U9ChO0Ig3r7lzdRFu1zmXw=; b=aNTr/5y7I/m6XzXRDp2sZ65wzHZICmZLmc8uH9XtR/ehwOlyfPe9ZWzeSBN6aSjz3alzVeCTugKPWHXfW7bSL+gOaJdx04SDu+WzSQeth6mRWzdeZ6t5K9+PpmmqTZ3vl0cdKsGkohiGby0UcKFQYMMHChIgN5Wo4uH75F4Kd7g= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none;nongnu.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) by SJ0PR10MB4765.namprd10.prod.outlook.com (2603:10b6:a03:2af::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Mon, 16 Aug 2021 16:43:04 +0000 Received: from BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539]) by BYAPR10MB2869.namprd10.prod.outlook.com ([fe80::5891:a7dc:5c2b:5539%7]) with mapi id 15.20.4415.023; Mon, 16 Aug 2021 16:43:04 +0000 From: Elena Ufimtseva To: qemu-devel@nongnu.org Subject: [PATCH RFC v2 16/16] vfio-user: migration support Date: Mon, 16 Aug 2021 09:42:49 -0700 Message-Id: <42758d8b56b1036d86cf2252c175071c1094a8b0.1629131628.git.elena.ufimtseva@oracle.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: References: X-ClientProxiedBy: BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) To BYAPR10MB2869.namprd10.prod.outlook.com (2603:10b6:a03:85::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from nuker.hsd1.ca.comcast.net (2601:642:4000:a09:154a:3cd:532c:f36d) by BY3PR03CA0026.namprd03.prod.outlook.com (2603:10b6:a03:39a::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.16 via Frontend Transport; Mon, 16 Aug 2021 16:43:03 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 97624a51-8ec2-42bd-f8ee-08d960d4eaf2 X-MS-TrafficTypeDiagnostic: SJ0PR10MB4765: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:31; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pJPeXVDf5qfRZjqVXwEEnVMKvbGmKOn3qwwKkklYYbhOQFJseg3vdiViVA9GIE9f7vwHJKLXCpJdO0D8j0yxZxTDW2JeygDSiZdns6yKKVudlCRCw119dOyHEm+LQbMHsMWX2hP3RsLioq7NZu+fvY1tr3ZXQ9pEzSnwj/oYaJfEscvR+m1zseI+foBjwvu4qatcWDvKmRbttWO1GO+GWZUgXFu75Y5GnooXylyCFsSn4wR7V7UrpIYf6FzyJZ5bE1A5C4ftairpmGeJ2XZqlFb6Eg8ErdENseUp4DjrS+gBuJQ/PdRyGg5e0L1A7egx3TRqnIvzLZMrrJMiZrcQHYIeSG6oD8ieC2XqJM4R+Uzbrv5aN1sPh9L3jy5sah9sFLTNy7Tntj8Pg1ajKk448Ut5kBqlcOETq2aZXdkaR8R8ijIEKJopqVNEub97PcCrJSchJpNBOTpprGvC4F/0W4ZRi/fUK0i9eYOrY7VvqfCwxzLKbR05crp+pGecdsr8vlwW/Uv5lVTZfg239obeQ+N9NcyqizKcZt2aPbCr8Ba46e5HN0rrcUugcvNtWmPoCNyVQArcYT6oM5X/POTSmm6YRhM7SEYL9TmsjOKy0KHdqN0WX1G5ZCVflUoaeCpLG9kwpjbRaekKPNSJVu5qGA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BYAPR10MB2869.namprd10.prod.outlook.com; PTR:; CAT:NONE; SFS:(366004)(186003)(2906002)(86362001)(66556008)(4326008)(38100700002)(8676002)(508600001)(66476007)(6486002)(66946007)(6506007)(5660300002)(44832011)(2616005)(6512007)(6666004)(36756003)(6916009)(316002)(52116002)(83380400001)(8936002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: s+eeu+JresGXjtcgOvyAW6TR9fN/xf7yIhwa67hJM+9FUhjxQfSoeQ02REFZVf0XRuBeoR3KlMwtZjOLAB1PScIRTvgQH0XycRjQ5Iw0XqSg1zsRyb/2Jt+KmhcuMHAtjtbuUgbnTJRxXXbNeNWRjX/B9mQSI8zY/VgQQPe7Gq0lhNuxHX9tTShL54XUUyestTUGaHKJcp2q30uJndtBtX+5q8iOx/Baq6/wsPrznRsUwRx1rSteIeOV/xGyF2RmVTk6geCZK+kqWky1TJjKBW3iS+KYoOvHRmihFkPlyCoO1ZNIZ465FqanI9Sv+g7+0MIfxwt4NBNbEuq6gVqG97WTzpu+/yI+qI73BhN+VSwxgFLwhLz/04Iq07zYU/R1sg2gasAErxujzZw6CLqbPzAdYCRnc4BrR6Z+f7HO5sLulITARLNWnSo6Dg6gBn/Xnf/qg8gRZvsrsHNVtPHjaXmTP9k3NQKYvEnlybUQySx32A1PRQAIKvCVmMEhGtwwr8yhx4Vl85hqS0mfjPlRwyuX9OWppCn3fqy2gYTWqz7/Sn5JmdJ0ehCsepWhHOAqxxfDdB+efTr3XFR5QuLuRO4hb4cH4kZi1agvIRB0Q5md4c20oWiJ94eoTd3x1E07a8XC2PW0gQQGYEmOBaS1tfle6thA78N4J/QVmtgmwjgWrE2nc8PxDyZdcX/EFX7hsq4X2MGjjtzpQLCtJr0zMuQznh2Qgh450KP85/1JxL0/fqUtcmhLyLA2vU5W9WTGrYMxbGVn28wyzizgbtb830Dp9D3AtjGpNa6W8SV5sdBXDwxOQYyd1riUarLaXKK54LkoZ1HZkgFabUrWU4hBhDLq9tpZ8ys1xKw30puBg4WFgQGUOzy61+YX5ejUWtqQJaPWrMWcFXCybk636vpCZcgiyNdg3O9qM6+iSXDVSy+NaHTDIs+lyRpezOdJnjr6BQG0jfGH3rt9ZXFluTo81FerLEgRuYjClNNuOkQp3KACXLYq+sHGKbSD4EBn/9ScjbY5S7Ga5plB0UyhSFQshfi+wRP0AYOb0/L/HPU08hkHfACRka1GZOHcIx0fTiQe7vJ7bhoClJfRMoVnZGQGoO+o6Qu3eRpFz+1n2b+tGc/msM4LYsVBCP67e9SFJdJv8nWWm4c2jaC/xptImVs63HqJxbzZ8zXD86k9LSJM07qLgwXxWMzARGkbrYnOHrO/Gmfoy9S4UD7dCUXncEy2Tzp0HFI2DKlWzdCwZMC8KCYeeOwYkIYMQhIkNXKwRB3ccz2f72Yi4jv+nMz7x1TunRArtHMA5n2+VMgEbruWzXClBJSkEAO3/L15Ft0soPNk7OlQPBVY88e6bDFVUExRaBTTv+eh2dpQV7FWkCdROrKtnNmDONc3ETmlTNne3WZo X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 97624a51-8ec2-42bd-f8ee-08d960d4eaf2 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB2869.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 16 Aug 2021 16:43:04.2320 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: efrAWQHbgL6kA5Zpf9yvfTfYVcoat4IHOpvd5HXIsOj22enzWU4ILHi7LAIdaYBDGRW9bwEH/etUbPWwoy43wSf03Ti7Upo7EUua14VTANQ= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR10MB4765 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=10078 signatures=668682 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 adultscore=0 bulkscore=0 suspectscore=0 mlxlogscore=999 phishscore=0 mlxscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2107140000 definitions=main-2108160105 X-Proofpoint-GUID: AHfl2hyeULaEDUMwpNRoxHwtbzZBwaKq X-Proofpoint-ORIG-GUID: AHfl2hyeULaEDUMwpNRoxHwtbzZBwaKq Received-SPF: pass client-ip=205.220.165.32; envelope-from=elena.ufimtseva@oracle.com; helo=mx0a-00069f02.pphosted.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, MSGID_FROM_MTA_HEADER=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_BL=0.001, RCVD_IN_MSPIKE_L3=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: elena.ufimtseva@oracle.com, john.g.johnson@oracle.com, jag.raman@oracle.com, swapnil.ingle@nutanix.com, john.levon@nutanix.com, alex.williamson@redhat.com, stefanha@redhat.com, thanos.makatos@nutanix.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Johnson Signed-off-by: John G Johnson Signed-off-by: Elena Ufimtseva Signed-off-by: Jagannathan Raman --- hw/vfio/user-protocol.h | 18 +++++++++++++++++ hw/vfio/user.h | 3 +++ hw/vfio/common.c | 23 ++++++++++++++++----- hw/vfio/migration.c | 34 +++++++++++++++++-------------- hw/vfio/pci.c | 12 +++++++++++ hw/vfio/user.c | 45 +++++++++++++++++++++++++++++++++++++++++ 6 files changed, 115 insertions(+), 20 deletions(-) diff --git a/hw/vfio/user-protocol.h b/hw/vfio/user-protocol.h index c5d9473f8f..bad067a570 100644 --- a/hw/vfio/user-protocol.h +++ b/hw/vfio/user-protocol.h @@ -182,6 +182,10 @@ typedef struct { char data[]; } VFIOUserDMARW; +/* + * VFIO_USER_DIRTY_PAGES + */ + /*imported from struct vfio_bitmap */ typedef struct { uint64_t pgsize; @@ -189,4 +193,18 @@ typedef struct { char data[]; } VFIOUserBitmap; +/* imported from struct vfio_iommu_type1_dirty_bitmap_get */ +typedef struct { + uint64_t iova; + uint64_t size; + VFIOUserBitmap bitmap; +} VFIOUserBitmapRange; + +/* imported from struct vfio_iommu_type1_dirty_bitmap */ +typedef struct { + VFIOUserHdr hdr; + uint32_t argsz; + uint32_t flags; +} VFIOUserDirtyPages; + #endif /* VFIO_USER_PROTOCOL_H */ diff --git a/hw/vfio/user.h b/hw/vfio/user.h index 5d4d0a43ba..905e0ee28d 100644 --- a/hw/vfio/user.h +++ b/hw/vfio/user.h @@ -87,6 +87,9 @@ int vfio_user_region_read(VFIODevice *vbasedev, uint32_t index, uint64_t offset, int vfio_user_region_write(VFIODevice *vbasedev, uint32_t index, uint64_t offset, uint32_t count, void *data); void vfio_user_reset(VFIODevice *vbasedev); +int vfio_user_dirty_bitmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dirty_bitmap *bitmap, + struct vfio_iommu_type1_dirty_bitmap_get *range); void vfio_user_drain_reqs(VFIOProxy *proxy); #endif /* VFIO_USER_H */ diff --git a/hw/vfio/common.c b/hw/vfio/common.c index a532e52bcf..09d0147df2 100644 --- a/hw/vfio/common.c +++ b/hw/vfio/common.c @@ -1303,10 +1303,19 @@ static void vfio_set_dirty_page_tracking(VFIOContainer *container, bool start) dirty.flags = VFIO_IOMMU_DIRTY_PAGES_FLAG_STOP; } - ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); - if (ret) { - error_report("Failed to set dirty tracking flag 0x%x errno: %d", - dirty.flags, errno); + if (container->proxy != NULL) { + ret = vfio_user_dirty_bitmap(container->proxy, &dirty, NULL); + if (ret) { + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, -ret); + } + } else { + ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, &dirty); + if (ret) { + error_report("Failed to set dirty tracking flag 0x%x errno: %d", + dirty.flags, errno); + ret = -errno; + } } } @@ -1356,7 +1365,11 @@ static int vfio_get_dirty_bitmap(VFIOContainer *container, uint64_t iova, goto err_out; } - ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + if (container->proxy != NULL) { + ret = vfio_user_dirty_bitmap(container->proxy, dbitmap, range); + } else { + ret = ioctl(container->fd, VFIO_IOMMU_DIRTY_PAGES, dbitmap); + } if (ret) { error_report("Failed to get dirty bitmap for iova: 0x%"PRIx64 " size: 0x%"PRIx64" err: %d", (uint64_t)range->iova, diff --git a/hw/vfio/migration.c b/hw/vfio/migration.c index 82f654afb6..89926a3b01 100644 --- a/hw/vfio/migration.c +++ b/hw/vfio/migration.c @@ -27,6 +27,7 @@ #include "pci.h" #include "trace.h" #include "hw/hw.h" +#include "user.h" /* * Flags to be used as unique delimiters for VFIO devices in the migration @@ -49,10 +50,18 @@ static int64_t bytes_transferred; static inline int vfio_mig_access(VFIODevice *vbasedev, void *val, int count, off_t off, bool iswrite) { + VFIORegion *region = &vbasedev->migration->region; int ret; - ret = iswrite ? pwrite(vbasedev->fd, val, count, off) : - pread(vbasedev->fd, val, count, off); + if (vbasedev->proxy != NULL) { + ret = iswrite ? + vfio_user_region_write(vbasedev, region->nr, off, count, val) : + vfio_user_region_read(vbasedev, region->nr, off, count, val); + } else { + off += region->fd_offset; + ret = iswrite ? pwrite(vbasedev->fd, val, count, off) : + pread(vbasedev->fd, val, count, off); + } if (ret < count) { error_report("vfio_mig_%s %d byte %s: failed at offset 0x%" HWADDR_PRIx", err: %s", iswrite ? "write" : "read", count, @@ -111,9 +120,7 @@ static int vfio_migration_set_state(VFIODevice *vbasedev, uint32_t mask, uint32_t value) { VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; - off_t dev_state_off = region->fd_offset + - VFIO_MIG_STRUCT_OFFSET(device_state); + off_t dev_state_off = VFIO_MIG_STRUCT_OFFSET(device_state); uint32_t device_state; int ret; @@ -201,13 +208,13 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) int ret; ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); + VFIO_MIG_STRUCT_OFFSET(data_offset)); if (ret < 0) { return ret; } ret = vfio_mig_read(vbasedev, &data_size, sizeof(data_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); + VFIO_MIG_STRUCT_OFFSET(data_size)); if (ret < 0) { return ret; } @@ -233,8 +240,7 @@ static int vfio_save_buffer(QEMUFile *f, VFIODevice *vbasedev, uint64_t *size) } buf_allocated = true; - ret = vfio_mig_read(vbasedev, buf, sec_size, - region->fd_offset + data_offset); + ret = vfio_mig_read(vbasedev, buf, sec_size, data_offset); if (ret < 0) { g_free(buf); return ret; @@ -269,7 +275,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, do { ret = vfio_mig_read(vbasedev, &data_offset, sizeof(data_offset), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_offset)); + VFIO_MIG_STRUCT_OFFSET(data_offset)); if (ret < 0) { return ret; } @@ -309,8 +315,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, qemu_get_buffer(f, buf, sec_size); if (buf_alloc) { - ret = vfio_mig_write(vbasedev, buf, sec_size, - region->fd_offset + data_offset); + ret = vfio_mig_write(vbasedev, buf, sec_size, data_offset); g_free(buf); if (ret < 0) { @@ -322,7 +327,7 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, } ret = vfio_mig_write(vbasedev, &report_size, sizeof(report_size), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(data_size)); + VFIO_MIG_STRUCT_OFFSET(data_size)); if (ret < 0) { return ret; } @@ -334,12 +339,11 @@ static int vfio_load_buffer(QEMUFile *f, VFIODevice *vbasedev, static int vfio_update_pending(VFIODevice *vbasedev) { VFIOMigration *migration = vbasedev->migration; - VFIORegion *region = &migration->region; uint64_t pending_bytes = 0; int ret; ret = vfio_mig_read(vbasedev, &pending_bytes, sizeof(pending_bytes), - region->fd_offset + VFIO_MIG_STRUCT_OFFSET(pending_bytes)); + VFIO_MIG_STRUCT_OFFSET(pending_bytes)); if (ret < 0) { migration->pending_bytes = 0; return ret; diff --git a/hw/vfio/pci.c b/hw/vfio/pci.c index 4b933ed10f..976fb89786 100644 --- a/hw/vfio/pci.c +++ b/hw/vfio/pci.c @@ -3658,6 +3658,13 @@ static void vfio_user_pci_realize(PCIDevice *pdev, Error **errp) } } + if (!pdev->failover_pair_id) { + ret = vfio_migration_probe(&vdev->vbasedev, errp); + if (ret) { + error_report("%s: Migration disabled", vdev->vbasedev.name); + } + } + vfio_register_err_notifier(vdev); vfio_register_req_notifier(vdev); @@ -3709,6 +3716,11 @@ static void vfio_user_pci_reset(DeviceState *dev) static Property vfio_user_pci_dev_properties[] = { DEFINE_PROP_STRING("socket", VFIOUserPCIDevice, sock_name), DEFINE_PROP_BOOL("secure-dma", VFIOUserPCIDevice, secure_dma, false), + DEFINE_PROP_BOOL("x-enable-migration", VFIOPCIDevice, + vbasedev.enable_migration, false), + DEFINE_PROP_ON_OFF_AUTO("x-pre-copy-dirty-page-tracking", VFIOPCIDevice, + vbasedev.pre_copy_dirty_page_tracking, + ON_OFF_AUTO_ON), DEFINE_PROP_END_OF_LIST(), }; diff --git a/hw/vfio/user.c b/hw/vfio/user.c index 7de2125346..486f7c0fe7 100644 --- a/hw/vfio/user.c +++ b/hw/vfio/user.c @@ -1057,3 +1057,48 @@ void vfio_user_reset(VFIODevice *vbasedev) error_printf("reset reply error %d\n", msg.error_reply); } } + +int vfio_user_dirty_bitmap(VFIOProxy *proxy, + struct vfio_iommu_type1_dirty_bitmap *cmd, + struct vfio_iommu_type1_dirty_bitmap_get *dbitmap) +{ + g_autofree struct { + VFIOUserDirtyPages msg; + VFIOUserBitmapRange range; + } *msgp = NULL; + int msize, rsize; + + /* + * If just the command is sent, the returned bitmap isn't needed. + * The bitmap structs are different from the ioctl() versions, + * ioctl() returns the bitmap in a local VA + */ + if (dbitmap != NULL) { + msize = sizeof(*msgp); + rsize = msize + dbitmap->bitmap.size; + msgp = g_malloc0(rsize); + msgp->range.iova = dbitmap->iova; + msgp->range.size = dbitmap->size; + msgp->range.bitmap.pgsize = dbitmap->bitmap.pgsize; + msgp->range.bitmap.size = dbitmap->bitmap.size; + } else { + msize = rsize = sizeof(VFIOUserDirtyPages); + msgp = g_malloc0(rsize); + } + + vfio_user_request_msg(&msgp->msg.hdr, VFIO_USER_DIRTY_PAGES, msize, 0); + msgp->msg.argsz = msize - sizeof(msgp->msg.hdr); + msgp->msg.flags = cmd->flags; + + vfio_user_send_recv(proxy, &msgp->msg.hdr, NULL, rsize, 0); + if (msgp->msg.hdr.flags & VFIO_USER_ERROR) { + return -msgp->msg.hdr.error_reply; + } + + if (dbitmap != NULL) { + memcpy(dbitmap->bitmap.data, &msgp->range.bitmap.data, + dbitmap->bitmap.size); + } + + return 0; +}