From patchwork Wed Aug 31 23:12:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 12961530 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EB40C6FA83 for ; Wed, 31 Aug 2022 23:13:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232199AbiHaXNC (ORCPT ); Wed, 31 Aug 2022 19:13:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33942 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231873AbiHaXM5 (ORCPT ); Wed, 31 Aug 2022 19:12:57 -0400 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (mail-dm6nam10on2076.outbound.protection.outlook.com [40.107.93.76]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 072EAE68C0; Wed, 31 Aug 2022 16:12:54 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lMuCRCMuT6p/FKulXWO8gbB38MYXhfsuOmjGiuepKm6nUWGqmRpuoN5NVk0W3vcR93JHIDuikx8N1GLflkKz29Wg5nMq8tV19/im69xfxrbRHaS955fiVL/V5hcGRuvLzIydc6Umi9aobBPHSAPxehysrg1lchsskSsvYHflEMzy4CmySgn3XLdnomMKT3jz/GBmovD7drzvBiUD7hr9ZDO0y5KaLa2WicA6J31BuCsXm8LwV1KWn6jyl98Q+dX+enxJsk6dHe3aZ3w9nbWPK3wcwS8KOJkfyb1Phi5VSjHxzuM3gnlRtfWudHrOR1vpM9DupNRc/s6IYWew0CHbpQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=XZhg0vFUSkiUQYS7H1fY0vREXOHo5YKAPQOLbz46IC8=; b=dO/sQJCsWuf/vCofk9HB9IU2sdDJ1GfJrErXOvDN76YlIL27TOXVAzRYGdBvT5bmJFsBzrJp810Bx5X84xlyKTJVVOuhRLzIycgf2d3Is0qeaiIXaINrKBdPAIg9HPP1nZsCiiK0+h45sOv4Lhm6fkQDgO2+oyEbmZNYqYv92aJn97SBiVgO06HBjYHTErY/zpMSqyLDxVzmr70xJ7K2VE7pHRvwMBEYzQFkWw4h1/6uGICboJAHgm3X52ftLEtibWD8h4F9qzCgPWNDglwnRZu6VHNmyz5YtXMGMuk/9Y+vGMvD+U2Ojkh/YmYUtfVSXsnjO73ADMue3STtatYgXg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XZhg0vFUSkiUQYS7H1fY0vREXOHo5YKAPQOLbz46IC8=; b=jvch4Shf9Gg2FulnRc0PvVqeJ6rQxRvYfKRazQhYlyU7jLH31Hkg2yCkqkkJRzJi8F641RY20BVQt3tQBEVTZI8wgMsS7+M1bGmb9qHxlHcYi3bTDqTHGDG7rGg14N1Idr6At2uaQQ13XritM6u4iMBQ/gVUdcAG8qJgcCKuWnU6kZtnVDL2uhyTVzvSzPvUX6mmA/FZLPxAz3Moc/3ct1S6URBQLsdZTcphT+tANjl3/hyeTpbqkuO/0CE60nPFTwihGRnepMzmPP6ezFU37zWeriT6yeEq0MICfSURlNKQlAAB70U/Lkw+N35Z3KzgIumj4aupOgmkz3+5srguCg== Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; Received: from MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) by MW5PR12MB5650.namprd12.prod.outlook.com (2603:10b6:303:19e::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5588.10; Wed, 31 Aug 2022 23:12:47 +0000 Received: from MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5]) by MN2PR12MB4192.namprd12.prod.outlook.com ([fe80::462:7fe:f04f:d0d5%7]) with mapi id 15.20.5588.010; Wed, 31 Aug 2022 23:12:47 +0000 From: Jason Gunthorpe To: Alex Williamson , =?utf-8?q?Christian_K?= =?utf-8?q?=C3=B6nig?= , Cornelia Huck , dri-devel@lists.freedesktop.org, kvm@vger.kernel.org, linaro-mm-sig@lists.linaro.org, linux-media@vger.kernel.org, Sumit Semwal Cc: Daniel Vetter , Leon Romanovsky , linux-rdma@vger.kernel.org, Maor Gottlieb , Oded Gabbay Subject: [PATCH v2 0/4] Allow MMIO regions to be exported through dma-buf Date: Wed, 31 Aug 2022 20:12:40 -0300 Message-Id: <0-v2-472615b3877e+28f7-vfio_dma_buf_jgg@nvidia.com> X-ClientProxiedBy: BLAPR05CA0004.namprd05.prod.outlook.com (2603:10b6:208:36e::8) To MN2PR12MB4192.namprd12.prod.outlook.com (2603:10b6:208:1d5::15) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 3c41f172-9517-4621-990b-08da8ba650b0 X-MS-TrafficTypeDiagnostic: MW5PR12MB5650:EE_ X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: EdA9knTyDyxcoVGacDHJ6fj74BWMGqUm5GCli3jb5dsHvnABIlMhNhVXwoBI1NQTHy5RpgF8G+dMdtmlR8LfK2P/A9nAkc+rQM3dTLRwQAKJzvo67oWqeh3AcfeIXFk+rY4+LKDg7Y3diIFv9hcuDIUU+DsqMyobk7r3TQZ3V4JWgM6eakeH/+izVenkmgGfn75fGlHx4ibSUy4h31kE8SXU8caiWQQdsQ0mBK2eq5sl6LXAAZXwjIsvVuRlhdB/ukcT2/XIA9E23bYKbJvtVHQTfFfKVgzdGLv8lJ2JF0beLNmzsgTLweiQaf3DlAudNr6FWAnsTXqRpG5ngSzQg3IdXp8pYLyX8xhKR6CCTb64PYzlSG1zA2q2GCI5kM30wLiDEsnb0Feh2EY/q1m8ONykSTh0RyD6b2Qj2gBbytHqca58DrCem2hj2IfZm0pHeI+vnXsbcnWrx2V9U+9JOOp9w9otm+HWb1R/ht8fR7wotYvnEccRlGGgffohR8c4W7T/T0M0IfNDb92KeIYJcHOGtE/TaXfy3Irc5j60qAf9b80grdOKegsa8ep22ZAoBZex0N0grS4X44UYbzOMoODwR8hN+31w7ZwUMVnJR+M107Ke5EJRyFzUdCClEeaquIpJRh3p0NCM5FYosxNtsvCrOWfj2C7CojQ6iB5fKL2GKXxswqSKEZ7zZn0fjPO9BvjAD6fNi34ZqkTJFAuOF+iO3ctyfhPli/F/rx5/TiA15U4SMnM33ITt4Z0svZHCLjXUSybRcQaqXNEOuUlswBw4TvC0kfIAuqh6fvjBejLaJcUXWJK0KQWvkwb43TrwAmVAhA6XIc7gUW+S+bMrU6pw8jakOFrNDWlwE1jhLOQPJmiepkHSaZbDsJDmzgBV X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:MN2PR12MB4192.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230016)(4636009)(366004)(376002)(346002)(136003)(396003)(39860400002)(6506007)(66476007)(66556008)(8676002)(4326008)(316002)(8936002)(6512007)(26005)(6666004)(86362001)(110136005)(478600001)(6486002)(966005)(38100700002)(54906003)(41300700001)(186003)(2906002)(66946007)(7416002)(2616005)(36756003)(83380400001)(66574015)(5660300002)(4216001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?q?O723hNEb+1llvlkCdcG3/c8fXKmv?= =?utf-8?q?i7xJ66JEQxpIcQE2caQTTpPue54wp6eZNpcXySkbtnJRB3D8uzb+L7YSnEfNB28AG?= =?utf-8?q?x5Z6lxPqp2iz5648axgkgqS5IH0NwgLBmac8Wcqe0m181fynJe82VwTY7Nkr4znT6?= =?utf-8?q?ZRDdIJzhJJD/Y2qOOCUP1qW9GaeFg3ii/79F3idc3UIW7N6D5ZdAzyXcnxC+6qyz7?= =?utf-8?q?GGWhjk+mg+erHbRymNsJYsiaPsb7N7/HTlEeWkpjDHL7NA6LDVyQzOh6P7F/qeL1I?= =?utf-8?q?wymfklrJLZTcRAm8e8s4YwN58t/2bpZ4oVaDTPWj9k5LF7aKhZ4/gybQlhN8fyDFg?= =?utf-8?q?q1/fHjrIjQDWsKQI3RER7ObuY0pr5E+vebduuWLpnt//U2AI6ELAMExdPm0/qbOEx?= =?utf-8?q?HakGoR9M4RUsFsq3EbNnue0utFk6B9Fk7mluSsUduFxqzVzO9kOu1gymK5qsXvZrQ?= =?utf-8?q?91u53HD07H1I/kowfEjVfIyWpCe0nXA+yQYpOrM9qtAD2ce8IfhGX1TasRKGg3CsY?= =?utf-8?q?n31Jz5XJapLYpL0BhLx1Q0E354mhOD11wc/tzFP9BNlyr4EuMxrLv6JKXsBHXCNVn?= =?utf-8?q?48pW/eIQsl3una4JhMFC9VYAoDx2ZxWN5r68JXnTYnrNlLgFUOr1K9pK3LY1Ywt9Z?= =?utf-8?q?kLJtKVfeD7KlHUSVN/Ka4t02PxkzTCwo/3Krqkvj5sJZGAQfmLMxGYVl/tJuyweIi?= =?utf-8?q?ChsF7a93UPCb52FhqoQgYiN1AEJbR9x4JWt/WpALAKENjPMeLusE5qqwsLroKF81A?= =?utf-8?q?FLFRyi709AFNNL11nAE4ZLxK1hR2jzZ8K4lc2fWFtmvr7dXP2zik2ZBO7iygFu/Tn?= =?utf-8?q?W7KefGXUOyRxFX0LER9gWs4P5jSl24/rJn+9zbj/fhRR97qDUn3/cnuX2M85u3JTj?= =?utf-8?q?yKBT1ksKT9uV3bQO2sDaET438iRAZIFMJ68VC2SqpZ0cSPqsoF8GU5NelNqlN3/bp?= =?utf-8?q?VdHLeipshGcIFB/u01q63G/A1HBYPmgb78FkTFNIXCthQu6F9m8rG098PUuQVajPp?= =?utf-8?q?nyM2TRRRk+6zSGxV8UBfK0NO+3cOnbvSEKmqVz24alI2aWDggI3Y+1oYeyH4olWaL?= =?utf-8?q?1SrA9nbGK66Z07OKh1wzy3o0JiZUPb79GhIOxcxKm25rZeOavOeacCxMpTsltb4UH?= =?utf-8?q?vvYdvNpdQh8Fr2fJOcHS/2Qc+rOEnEKKw8d27RzeZneiZn/jY782fMcGsMYjH1teI?= =?utf-8?q?Vn295o6QPqdRPdDvSt1vwdMLihzxj6AsJyrAZbeMOCDgciykHuxE6mI+P9DbWqdDz?= =?utf-8?q?FYG2K2KnwcReGd4I2S9PdChyKQLQRBL3ttqnYnEgb9IdHlsrJJ4HA8p2WmmHAkgvX?= =?utf-8?q?ZsSP1i3i0RngSEYY9BDzFOZrma3+0RUYR3hy2DO4D9Mx7gJTnWru7BnwwmOoR6Wqz?= =?utf-8?q?Ae/jruBqQ4F64mvKR1C2/zfGhjiCbD/DHiACpqMzdZexHYlBKIHg2m2e+fAbmBqz5?= =?utf-8?q?s+DjaFCYz8XoaqY4zR4MVy69bLjsJpHKnI0kuStrK8v2X6p0WMQcorkgMcw55w/u6?= =?utf-8?q?iY077VuJ1Xdy?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3c41f172-9517-4621-990b-08da8ba650b0 X-MS-Exchange-CrossTenant-AuthSource: MN2PR12MB4192.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2022 23:12:46.1879 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Mab9YO5WahAspTBsnF1oePGcZCJIffhNF6kcJjnNb0GjJl9OHMaHV1lI7B9Rmtlh X-MS-Exchange-Transport-CrossTenantHeadersStamped: MW5PR12MB5650 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org dma-buf has become a way to safely acquire a handle to non-struct page memory that can still have lifetime controlled by the exporter. Notably RDMA can now import dma-buf FDs and build them into MRs which allows for PCI P2P operations. Extend this to allow vfio-pci to export MMIO memory from PCI device BARs. This series supports a use case for SPDK where a NVMe device will be owned by SPDK through VFIO but interacting with a RDMA device. The RDMA device may directly access the NVMe CMB or directly manipulate the NVMe device's doorbell using PCI P2P. However, as a general mechanism, it can support many other scenarios with VFIO. I imagine this dmabuf approach to be usable by iommufd as well for generic and safe P2P mappings. This series goes after the "Break up ioctl dispatch functions to one function per ioctl" series. This is on github: https://github.com/jgunthorpe/linux/commits/vfio_dma_buf v2: - Name the new file dma_buf.c - Restore orig_nents before freeing - Fix reversed logic around priv->revoked - Set priv->index - Rebased on v2 "Break up ioctl dispatch functions" v1: https://lore.kernel.org/r/0-v1-9e6e1739ed95+5fa-vfio_dma_buf_jgg@nvidia.com Cc: linux-rdma@vger.kernel.org Cc: Oded Gabbay Cc: Christian König Cc: Daniel Vetter Cc: Leon Romanovsky Cc: Maor Gottlieb Cc: dri-devel@lists.freedesktop.org Signed-off-by: Jason Gunthorpe Jason Gunthorpe (4): dma-buf: Add dma_buf_try_get() vfio: Add vfio_device_get() vfio_pci: Do not open code pci_try_reset_function() vfio/pci: Allow MMIO regions to be exported through dma-buf drivers/vfio/pci/Makefile | 1 + drivers/vfio/pci/dma_buf.c | 269 +++++++++++++++++++++++++++++ drivers/vfio/pci/vfio_pci_config.c | 22 ++- drivers/vfio/pci/vfio_pci_core.c | 33 +++- drivers/vfio/pci/vfio_pci_priv.h | 24 +++ drivers/vfio/vfio_main.c | 3 +- include/linux/dma-buf.h | 13 ++ include/linux/vfio.h | 6 + include/linux/vfio_pci_core.h | 1 + include/uapi/linux/vfio.h | 18 ++ 10 files changed, 368 insertions(+), 22 deletions(-) create mode 100644 drivers/vfio/pci/dma_buf.c base-commit: 285fef0ff7f1a97d8acd380971c061985d8dafb5