From patchwork Wed Dec 2 20:57:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Xiong, Jianxin" X-Patchwork-Id: 11947095 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F34E5C83014 for ; Wed, 2 Dec 2020 20:44:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BD71E221FA for ; Wed, 2 Dec 2020 20:44:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2389505AbgLBUoW (ORCPT ); Wed, 2 Dec 2020 15:44:22 -0500 Received: from mga11.intel.com ([192.55.52.93]:58903 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2389502AbgLBUoV (ORCPT ); Wed, 2 Dec 2020 15:44:21 -0500 IronPort-SDR: Kn/9tW2XsLELJ3pYeb0COtGg8nwwj64qBMuYnjXoI4J9tBjPjc4G0Ha1oxgL1qeV894IB5hpD8 S4805HxcUxUw== X-IronPort-AV: E=McAfee;i="6000,8403,9823"; a="169579856" X-IronPort-AV: E=Sophos;i="5.78,387,1599548400"; d="scan'208";a="169579856" X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Dec 2020 12:43:24 -0800 IronPort-SDR: +4BAfJ5mg0Q0U8uw0mGcVOI6HrXJKQzkd9JiBEky2XEuWal1KlX7A8vjFQ2oabgsw+aYZHOgmo Ugd0AG8ufzqA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.78,387,1599548400"; d="scan'208";a="365457414" Received: from cst-dev.jf.intel.com ([10.23.221.69]) by fmsmga004.fm.intel.com with ESMTP; 02 Dec 2020 12:43:23 -0800 From: Jianxin Xiong To: linux-rdma@vger.kernel.org, dri-devel@lists.freedesktop.org Cc: Jianxin Xiong , Doug Ledford , Jason Gunthorpe , Leon Romanovsky , Sumit Semwal , Christian Koenig , Daniel Vetter , Edward Srouji Subject: [PATCH rdma-core v4 0/6] Add user space dma-buf support Date: Wed, 2 Dec 2020 12:57:15 -0800 Message-Id: <1606942641-47234-1-git-send-email-jianxin.xiong@intel.com> X-Mailer: git-send-email 1.8.3.1 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This is the fourth version of the patch series. Change log: v4: * Rework the cmake funciton rdma_cython_module to support both single source (.pyx) and multiple source (.pyx + [.c]*) scenarios instead of using two separate functions * Rename 'dri_*' to 'drm_*' for the dmabuf allocation interface * Add option to dmabuf allocation routine to allow allocation from GTT instead of VRAM * Add proper CPU access flags when allocating dmabufs * Remove 'radeon' driver support from the dmabuf allocation routines * Add comand line arguments to the tests for selecting GPU unit and setting the option for allocating from GTT v3: https://www.spinics.net/lists/linux-rdma/msg98059.html * Add parameter 'iova' to the new ibv_reg_dmabuf_mr() API * Change the way of allocating dma-buf object - use /dev/dri/renderD* instead of /dev/dri/card* and use GEM object instead of dumb buffer * Add cmake function to allow building modules with mixed cython and C source files * Add new tests that use dma-buf MRs for send/recv and rdma traffic * Skip dma-buf tests on unsupported systems * Remove some use of random values in the new tests * Add dealloc() and close() methods to the new classes * Replace string.format with f-string in python code * Fix some coding style issues: spacing, indentation, typo, comments v2: https://www.spinics.net/lists/linux-rdma/msg97936.html * Put the kernel header updates into a separate commit * Add comments for the data structure used in python ioctl calls * Fix issues related to symbol versioning * Fix styling issues: extra spaces, unncecessary variable, typo * Fix an inproper error code usage * Put the new op into ibv_context_ops instead if verbs_context v1: https://www.spinics.net/lists/linux-rdma/msg97865.html * Add user space API for registering dma-buf based memory regions * Update pyverbs with the new API * Add new tests This is the user space counter-part of the kernel patch set to add dma-buf support to the RDMA subsystem. This series consists of six patches. The first patch updates the kernel headers for dma-buf support. Patch 2 adds the new API function and updates the man pages. Patch 3 implements the new API in the mlx5 provider. Patch 4 adds new class definitions to pyverbs for the new API. Patch 5 adds a set of new tests for the new API. Patch 6 fixes bug in the utility code of the tests. Pull request at github: https://github.com/linux-rdma/rdma-core/pull/895 Jianxin Xiong (6): Update kernel headers verbs: Support dma-buf based memory region mlx5: Support dma-buf based memory region pyverbs: Add dma-buf based MR support tests: Add tests for dma-buf based memory regions tests: Bug fix for get_access_flags() buildlib/pyverbs_functions.cmake | 78 ++++++--- debian/libibverbs1.symbols | 2 + kernel-headers/rdma/ib_user_ioctl_cmds.h | 14 ++ kernel-headers/rdma/ib_user_verbs.h | 14 -- libibverbs/CMakeLists.txt | 2 +- libibverbs/cmd_mr.c | 38 +++++ libibverbs/driver.h | 7 + libibverbs/dummy_ops.c | 11 ++ libibverbs/libibverbs.map.in | 6 + libibverbs/man/ibv_reg_mr.3 | 27 ++- libibverbs/verbs.c | 18 ++ libibverbs/verbs.h | 11 ++ providers/mlx5/mlx5.c | 2 + providers/mlx5/mlx5.h | 3 + providers/mlx5/verbs.c | 22 +++ pyverbs/CMakeLists.txt | 11 +- pyverbs/dmabuf.pxd | 15 ++ pyverbs/dmabuf.pyx | 73 ++++++++ pyverbs/dmabuf_alloc.c | 278 +++++++++++++++++++++++++++++++ pyverbs/dmabuf_alloc.h | 19 +++ pyverbs/libibverbs.pxd | 2 + pyverbs/mr.pxd | 6 + pyverbs/mr.pyx | 105 +++++++++++- tests/args_parser.py | 4 + tests/test_mr.py | 264 ++++++++++++++++++++++++++++- tests/utils.py | 30 +++- 26 files changed, 1012 insertions(+), 50 deletions(-) create mode 100644 pyverbs/dmabuf.pxd create mode 100644 pyverbs/dmabuf.pyx create mode 100644 pyverbs/dmabuf_alloc.c create mode 100644 pyverbs/dmabuf_alloc.h