From patchwork Thu May 16 18:24:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Adit Ranadive X-Patchwork-Id: 10946985 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD9EF924 for ; Thu, 16 May 2019 18:25:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9C29D2891A for ; Thu, 16 May 2019 18:25:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8D90428957; Thu, 16 May 2019 18:25:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 65C7D2891A for ; Thu, 16 May 2019 18:25:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726979AbfEPSZC (ORCPT ); Thu, 16 May 2019 14:25:02 -0400 Received: from mail-eopbgr680056.outbound.protection.outlook.com ([40.107.68.56]:33243 "EHLO NAM04-BN3-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726357AbfEPSZC (ORCPT ); Thu, 16 May 2019 14:25:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=vmware.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pVCihJkF6fThVlqlb15yRpgMSMhuPS9v5sWTve9f+Ho=; b=Oyhd19RtE4Wkd1MvJ7CvWDMYldoovpDRCytl6k8etkfmpwQjDeQ8iBSTineqXQqPIRUPStVJMJJIQPyc6HkdjFi0v3unmmHP6cV7UxkZUof4DvYWi6ePpha9p3dvUmfQIwOeaaQsSncOiZyyh1IW+PCPRJotmnGUjhudVXWc+AQ= Received: from BYAPR05MB5511.namprd05.prod.outlook.com (20.177.186.28) by BYAPR05MB6245.namprd05.prod.outlook.com (20.178.196.14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1900.13; Thu, 16 May 2019 18:24:48 +0000 Received: from BYAPR05MB5511.namprd05.prod.outlook.com ([fe80::d99b:a85f:758a:f04b]) by BYAPR05MB5511.namprd05.prod.outlook.com ([fe80::d99b:a85f:758a:f04b%7]) with mapi id 15.20.1922.002; Thu, 16 May 2019 18:24:48 +0000 From: Adit Ranadive To: "jgg@mellanox.com" , "dledford@redhat.com" CC: Bryan Tan , "linux-rdma@vger.kernel.org" , Pv-drivers , Adit Ranadive Subject: [PATCH for-next] RDMA/vmw_pvrdma: Use resource ids from physical device if available Thread-Topic: [PATCH for-next] RDMA/vmw_pvrdma: Use resource ids from physical device if available Thread-Index: AQHVDBSlkap1QWBkwEm57ccCex0bMg== Date: Thu, 16 May 2019 18:24:48 +0000 Message-ID: <1558031071-14110-1-git-send-email-aditr@vmware.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: BYAPR07CA0076.namprd07.prod.outlook.com (2603:10b6:a03:12b::17) To BYAPR05MB5511.namprd05.prod.outlook.com (2603:10b6:a03:1a::28) x-mailer: git-send-email 1.8.3.1 authentication-results: spf=none (sender IP is ) smtp.mailfrom=aditr@vmware.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [66.170.99.2] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 096ed262-c9f1-484f-a1dc-08d6da2bc767 x-microsoft-antispam: BCL:0;PCL:0;RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600141)(711020)(4605104)(2017052603328)(7193020);SRVR:BYAPR05MB6245; x-ms-traffictypediagnostic: BYAPR05MB6245: x-ms-exchange-purlcount: 1 x-ld-processed: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0,ExtAddr x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:422; x-forefront-prvs: 0039C6E5C5 x-forefront-antispam-report: SFV:NSPM;SFS:(10009020)(136003)(346002)(39860400002)(396003)(366004)(376002)(199004)(189003)(66066001)(6306002)(6512007)(14454004)(52116002)(966005)(6506007)(102836004)(386003)(478600001)(4720700003)(26005)(186003)(110136005)(2501003)(54906003)(6436002)(316002)(6486002)(53936002)(99286004)(68736007)(2906002)(71190400001)(71200400001)(86362001)(73956011)(66946007)(25786009)(66556008)(66476007)(107886003)(14444005)(64756008)(256004)(5660300002)(81166006)(81156014)(50226002)(66446008)(8676002)(8936002)(305945005)(2616005)(476003)(3846002)(6116002)(36756003)(7736002)(4326008)(486006)(309714004);DIR:OUT;SFP:1101;SCL:1;SRVR:BYAPR05MB6245;H:BYAPR05MB5511.namprd05.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; received-spf: None (protection.outlook.com: vmware.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: Fgu4YNK5ecFzva6s6V7synU4bAvrVWD39W3Nk3c0BiCwdGwj75NhGSdjuhUYJaEHNB2EVZ6U/hIjh1rGVvTqLHK54vi7jkGvzM+Xfmb6U598gI0zoRszSm+1S0PYJl6E8YLIvLe0SWtQ+pHh8+t1IUHekIr/oPrWV71C/9AQNWyrh9jfJxv7YXAgY/6hWCFmVTP9tT8Pra9DdQI/f1jGtYOFkuEh/I/B0Tx8uiGs1GKCm02jTUt9ncvETq1SuNr/tgiq7c0YtZq2Wwn2iyxxH980fWLS6NKfUYXrbnhbxAgpFkA5n2Oem/Qg+M66yvmAVSIOLKVPw2L68Hd6GLsAu6/21SJCYBsd0Ne6gvvO/a7ATEndFkHw8wrY2RPzVB4EhN/jfVbYfEN93H50MSFddTnd+6imkpB4XOsG/Afvz6A= MIME-Version: 1.0 X-OriginatorOrg: vmware.com X-MS-Exchange-CrossTenant-Network-Message-Id: 096ed262-c9f1-484f-a1dc-08d6da2bc767 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 May 2019 18:24:48.6758 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b39138ca-3cee-4b4a-a4d6-cd83d9dd62f0 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR05MB6245 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP From: Bryan Tan This change allows the RDMA stack to use physical resource numbers if they are passed up from the device. Doing so allows communication with physical non-ESX endpoints (such as a bare-metal Linux machine or a SR-IOV-enabled VM). This is accomplished by separating the concept of the QP number from the QP handle. Previously, the two were the same, as the QP number was exposed to the guest and also used to reference a virtual QP in the device backend. With physical resource numbers exposed, the QP number given to the guest is the QP number assigned to the physical HCA's QP, while the QP handle is still the internal handle used to reference a virtual QP. Regardless of whether the device is exposing physical ids, the driver will still try to pick up the QP handle from the backend if possible. The MR keys exposed to the guest will also be the MR keys created by the physical HCA, instead of virtual MR keys. A new version of the create QP response has been added to the device API. The device backend will pass the QP handle up to the driver, if both the device and driver are at the appriopriate version, and the ABI has also been updated to allow the return of the QP handle to the guest library. The PVRDMA version and ABI version have been bumped up because of these non-compatible changes. Reviewed-by: Jorgen Hansen Signed-off-by: Adit Ranadive Signed-off-by: Bryan Tan --- drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h | 15 +++++++++++++- drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c | 8 +++++++- drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c | 24 +++++++++++++++++++++-- include/uapi/rdma/vmw_pvrdma-abi.h | 9 ++++++++- 4 files changed, 51 insertions(+), 5 deletions(-) --- The PR for userspace was sent: https://github.com/linux-rdma/rdma-core/pull/531 --- diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h index 8f9749d54688..86a6c054ea26 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_dev_api.h @@ -58,7 +58,8 @@ #define PVRDMA_ROCEV1_VERSION 17 #define PVRDMA_ROCEV2_VERSION 18 #define PVRDMA_PPN64_VERSION 19 -#define PVRDMA_VERSION PVRDMA_PPN64_VERSION +#define PVRDMA_QPHANDLE_VERSION 20 +#define PVRDMA_VERSION PVRDMA_QPHANDLE_VERSION #define PVRDMA_BOARD_ID 1 #define PVRDMA_REV_ID 1 @@ -581,6 +582,17 @@ struct pvrdma_cmd_create_qp_resp { u32 max_inline_data; }; +struct pvrdma_cmd_create_qp_resp_v2 { + struct pvrdma_cmd_resp_hdr hdr; + u32 qpn; + u32 qp_handle; + u32 max_send_wr; + u32 max_recv_wr; + u32 max_send_sge; + u32 max_recv_sge; + u32 max_inline_data; +}; + struct pvrdma_cmd_modify_qp { struct pvrdma_cmd_hdr hdr; u32 qp_handle; @@ -663,6 +675,7 @@ struct pvrdma_cmd_destroy_bind { struct pvrdma_cmd_create_cq_resp create_cq_resp; struct pvrdma_cmd_resize_cq_resp resize_cq_resp; struct pvrdma_cmd_create_qp_resp create_qp_resp; + struct pvrdma_cmd_create_qp_resp_v2 create_qp_resp_v2; struct pvrdma_cmd_query_qp_resp query_qp_resp; struct pvrdma_cmd_destroy_qp_resp destroy_qp_resp; struct pvrdma_cmd_create_srq_resp create_srq_resp; diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c index 40182297f87f..02e337837a2e 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_main.c @@ -201,7 +201,13 @@ static int pvrdma_register_device(struct pvrdma_dev *dev) dev->ib_dev.owner = THIS_MODULE; dev->ib_dev.num_comp_vectors = 1; dev->ib_dev.dev.parent = &dev->pdev->dev; - dev->ib_dev.uverbs_abi_ver = PVRDMA_UVERBS_ABI_VERSION; + + if (dev->dsr_version >= PVRDMA_QPHANDLE_VERSION) + dev->ib_dev.uverbs_abi_ver = PVRDMA_UVERBS_ABI_VERSION; + else + dev->ib_dev.uverbs_abi_ver = + PVRDMA_UVERBS_NO_QP_HANDLE_ABI_VERSION; + dev->ib_dev.uverbs_cmd_mask = (1ull << IB_USER_VERBS_CMD_GET_CONTEXT) | (1ull << IB_USER_VERBS_CMD_QUERY_DEVICE) | diff --git a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c index 0eaaead5baec..8cba7623f379 100644 --- a/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c +++ b/drivers/infiniband/hw/vmw_pvrdma/pvrdma_qp.c @@ -195,7 +195,9 @@ struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, union pvrdma_cmd_resp rsp; struct pvrdma_cmd_create_qp *cmd = &req.create_qp; struct pvrdma_cmd_create_qp_resp *resp = &rsp.create_qp_resp; + struct pvrdma_cmd_create_qp_resp_v2 *resp_v2 = &rsp.create_qp_resp_v2; struct pvrdma_create_qp ucmd; + struct pvrdma_create_qp_resp qp_resp = {}; unsigned long flags; int ret; bool is_srq = !!init_attr->srq; @@ -379,13 +381,31 @@ struct ib_qp *pvrdma_create_qp(struct ib_pd *pd, } /* max_send_wr/_recv_wr/_send_sge/_recv_sge/_inline_data */ - qp->qp_handle = resp->qpn; qp->port = init_attr->port_num; - qp->ibqp.qp_num = resp->qpn; + if (dev->dsr_version >= PVRDMA_QPHANDLE_VERSION) { + qp->ibqp.qp_num = resp_v2->qpn; + qp->qp_handle = resp_v2->qp_handle; + } else { + qp->ibqp.qp_num = resp->qpn; + qp->qp_handle = resp->qpn; + } + spin_lock_irqsave(&dev->qp_tbl_lock, flags); dev->qp_tbl[qp->qp_handle % dev->dsr->caps.max_qp] = qp; spin_unlock_irqrestore(&dev->qp_tbl_lock, flags); + if (!qp->is_kernel) { + /* Copy udata back. */ + qp_resp.qpn = qp->ibqp.qp_num; + qp_resp.qp_handle = qp->qp_handle; + if (ib_copy_to_udata(udata, &qp_resp, sizeof(qp_resp))) { + dev_warn(&dev->pdev->dev, + "failed to copy back udata\n"); + pvrdma_destroy_qp(&qp->ibqp, udata); + return ERR_PTR(-EINVAL); + } + } + return &qp->ibqp; err_pdir: diff --git a/include/uapi/rdma/vmw_pvrdma-abi.h b/include/uapi/rdma/vmw_pvrdma-abi.h index 6e73f0274e41..8ebab11dadcb 100644 --- a/include/uapi/rdma/vmw_pvrdma-abi.h +++ b/include/uapi/rdma/vmw_pvrdma-abi.h @@ -49,7 +49,9 @@ #include -#define PVRDMA_UVERBS_ABI_VERSION 3 /* ABI Version. */ +#define PVRDMA_UVERBS_NO_QP_HANDLE_ABI_VERSION 3 +#define PVRDMA_UVERBS_ABI_VERSION 4 /* ABI Version. */ + #define PVRDMA_UAR_HANDLE_MASK 0x00FFFFFF /* Bottom 24 bits. */ #define PVRDMA_UAR_QP_OFFSET 0 /* QP doorbell. */ #define PVRDMA_UAR_QP_SEND (1 << 30) /* Send bit. */ @@ -179,6 +181,11 @@ struct pvrdma_create_qp { __aligned_u64 qp_addr; }; +struct pvrdma_create_qp_resp { + __u32 qpn; + __u32 qp_handle; +}; + /* PVRDMA masked atomic compare and swap */ struct pvrdma_ex_cmp_swap { __aligned_u64 swap_val;