From patchwork Sun Jul 5 08:19:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yishai Hadas X-Patchwork-Id: 11644201 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A88E718C2 for ; Sun, 5 Jul 2020 08:20:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 99807206B6 for ; Sun, 5 Jul 2020 08:20:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726495AbgGEIUM (ORCPT ); Sun, 5 Jul 2020 04:20:12 -0400 Received: from mail-il-dmz.mellanox.com ([193.47.165.129]:33120 "EHLO mellanox.co.il" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726494AbgGEIUL (ORCPT ); Sun, 5 Jul 2020 04:20:11 -0400 Received: from Internal Mail-Server by MTLPINE1 (envelope-from yishaih@mellanox.com) with SMTP; 5 Jul 2020 11:20:06 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [10.7.2.17]) by labmailer.mlnx (8.13.8/8.13.8) with ESMTP id 0658K5Gt001732; Sun, 5 Jul 2020 11:20:05 +0300 Received: from vnc17.mtl.labs.mlnx (vnc17.mtl.labs.mlnx [127.0.0.1]) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8) with ESMTP id 0658K5aw009331; Sun, 5 Jul 2020 11:20:05 +0300 Received: (from yishaih@localhost) by vnc17.mtl.labs.mlnx (8.13.8/8.13.8/Submit) id 0658K5gj009330; Sun, 5 Jul 2020 11:20:05 +0300 From: Yishai Hadas To: linux-rdma@vger.kernel.org Cc: jgg@mellanox.com, yishaih@mellanox.com, maorg@mellanox.com, Edward Srouji Subject: [PATCH V1 rdma-core 12/13] Documentation: Add usage example for verbs import Date: Sun, 5 Jul 2020 11:19:48 +0300 Message-Id: <1593937189-8744-13-git-send-email-yishaih@mellanox.com> X-Mailer: git-send-email 1.8.2.3 In-Reply-To: <1593937189-8744-1-git-send-email-yishaih@mellanox.com> References: <1593937189-8744-1-git-send-email-yishaih@mellanox.com> Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org From: Edward Srouji Add a documentation for shared PD and verbs import usage in Pyverbs. This includes a code snippet to demonstrate a usage example. Reviewed-by: Ido Kalir Signed-off-by: Edward Srouji --- Documentation/pyverbs.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/Documentation/pyverbs.md b/Documentation/pyverbs.md index 77a5406..2516b10 100755 --- a/Documentation/pyverbs.md +++ b/Documentation/pyverbs.md @@ -626,3 +626,43 @@ ctx = Context(name='rocep0s8f0') uar = Mlx5UAR(ctx) uar.close() ``` + +##### Import device, PD and MR +Importing a device, PD and MR enables processes to share their context and then +share PDs and MRs that is associated with. +A process creates a device and then uses some of the Linux systems calls to dup +its 'cmd_fd' member which lets other process to obtain ownership. +Once other process obtains the 'cmd_fd' it can import the device, then PD(s) and +MR(s) to share these objects. +Like in C, Pyverbs users are responsible for unimporting the imported objects +(which will also close the Pyverbs instance in our case) after they finish using +them, and they have to sync between the different processes in order to +coordinate the closure of the objects. +Unlike in C, closing the underlying objects is currently supported only via the +"original" object (meaning only by the process that creates them) and not via +the imported object. This limitation is made because currently there's no +reference or relation between different Pyverbs objects in different processes. +But it's doable and might be added in the future. +Here is a demonstration of importing a device, PD and MR in one process. +```python +from pyverbs.device import Context +from pyverbs.pd import PD +from pyverbs.mr import MR +import pyverbs.enums as e +import os + +ctx = Context(name='ibp0s8f0') +pd = PD(ctx) +mr = MR(pd, 100, e.IBV_ACCESS_LOCAL_WRITE) +cmd_fd_dup = os.dup(ctx.cmd_fd) +improted_ctx = Context(cmd_fd=cmd_fd_dup) +imported_pd = PD(improted_ctx, handle=pd.handle) +imported_mr = MR(imported_pd, handle=mr.handle) +# MRs can be created as usual on the imported PD +secondary_mr = MR(imported_pd, 100, e.IBV_ACCESS_REMOTE_READ) +# Must manually unimport the imported objects (which close the object and frees +# other resources that use them) before closing the "original" objects. +# This prevents unexpected behaviours caused by the GC. +imported_mr.unimport() +imported_pd.unimport() +```