From patchwork Fri Aug 19 02:47:12 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Hefty, Sean" X-Patchwork-Id: 1078652 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter2.kernel.org (8.14.4/8.14.4) with ESMTP id p7J2l9JI014620 for ; Fri, 19 Aug 2011 02:47:18 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753584Ab1HSCrR (ORCPT ); Thu, 18 Aug 2011 22:47:17 -0400 Received: from mga14.intel.com ([143.182.124.37]:17516 "EHLO mga14.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753400Ab1HSCrO convert rfc822-to-8bit (ORCPT ); Thu, 18 Aug 2011 22:47:14 -0400 Received: from azsmga002.ch.intel.com ([10.2.17.35]) by azsmga102.ch.intel.com with ESMTP; 18 Aug 2011 19:47:14 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.68,248,1312182000"; d="scan'208";a="8786152" Received: from azsmsx603.amr.corp.intel.com ([10.2.161.23]) by AZSMGA002.ch.intel.com with ESMTP; 18 Aug 2011 19:47:14 -0700 Received: from azsmsx604.amr.corp.intel.com (10.2.161.34) by azsmsx603.amr.corp.intel.com (10.2.161.23) with Microsoft SMTP Server (TLS) id 8.2.255.0; Thu, 18 Aug 2011 19:47:14 -0700 Received: from fmsmsx106.amr.corp.intel.com (10.19.9.37) by azsmsx604.amr.corp.intel.com (10.2.161.34) with Microsoft SMTP Server (TLS) id 8.2.255.0; Thu, 18 Aug 2011 19:47:13 -0700 Received: from fmsmsx151.amr.corp.intel.com ([169.254.6.155]) by FMSMSX106.amr.corp.intel.com ([169.254.6.78]) with mapi id 14.01.0323.003; Thu, 18 Aug 2011 19:47:13 -0700 From: "Hefty, Sean" To: "linux-rdma (linux-rdma@vger.kernel.org)" CC: "Hefty, Sean" Subject: [PATCH 5/6 v2] libibverbs: Add ibv_open_qp() Thread-Topic: [PATCH 5/6 v2] libibverbs: Add ibv_open_qp() Thread-Index: AcxeGc+V0ODhOg9PREaz9o6TJf/Ykg== Date: Fri, 19 Aug 2011 02:47:12 +0000 Message-ID: <1828884A29C6694DAF28B7E6B8A8237316E41B7A@FMSMSX151.amr.corp.intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.22.254.140] MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter2.kernel.org [140.211.167.43]); Fri, 19 Aug 2011 02:47:18 +0000 (UTC) XRC receive QPs are shareable across multiple processes. Allow any process with access to the xrc domain to open an existing QP. After opening the QP, the process will receive events related to the QP and be able to modify the QP. Signed-off-by: Sean Hefty --- Submitting this change separate from xrc support, since the corresponding kernel changes are new and have not been discussed. include/infiniband/driver.h | 4 ++++ include/infiniband/kern-abi.h | 18 +++++++++++++++++- include/infiniband/verbs.h | 17 ++++++++++++++++- src/cmd.c | 22 ++++++++++++++++++++++ src/libibverbs.map | 2 ++ src/verbs.c | 29 +++++++++++++++++++++++++++++ 6 files changed, 90 insertions(+), 2 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/infiniband/driver.h b/include/infiniband/driver.h index 9aea854..7fad2a2 100644 --- a/include/infiniband/driver.h +++ b/include/infiniband/driver.h @@ -122,6 +122,10 @@ int ibv_cmd_create_qp(struct ibv_pd *pd, struct ibv_qp *qp, struct ibv_qp_init_attr *attr, struct ibv_create_qp *cmd, size_t cmd_size, struct ibv_create_qp_resp *resp, size_t resp_size); +int ibv_cmd_open_qp(struct ibv_xrcd *xrcd, + struct ibv_qp *qp, struct ibv_qp_open_attr *attr, + struct ibv_open_qp *cmd, size_t cmd_size, + struct ibv_create_qp_resp *resp, size_t resp_size); int ibv_cmd_query_qp(struct ibv_qp *qp, struct ibv_qp_attr *qp_attr, int attr_mask, struct ibv_qp_init_attr *qp_init_attr, diff --git a/include/infiniband/kern-abi.h b/include/infiniband/kern-abi.h index 36cd133..32a14c7 100644 --- a/include/infiniband/kern-abi.h +++ b/include/infiniband/kern-abi.h @@ -88,7 +88,8 @@ enum { IB_USER_VERBS_CMD_POST_SRQ_RECV, IB_USER_VERBS_CMD_OPEN_XRCD, IB_USER_VERBS_CMD_CLOSE_XRCD, - IB_USER_VERBS_CMD_CREATE_XSRQ + IB_USER_VERBS_CMD_CREATE_XSRQ, + IB_USER_VERBS_CMD_OPEN_QP }; /* @@ -475,6 +476,20 @@ struct ibv_create_qp { __u64 driver_data[0]; }; +struct ibv_open_qp { + __u32 command; + __u16 in_words; + __u16 out_words; + __u64 response; + __u64 user_handle; + __u32 pd_handle; + __u32 qpn; + __u8 qp_type; + __u8 reserved[7]; + __u64 driver_data[0]; +}; + +/* also used for open response */ struct ibv_create_qp_resp { __u32 qp_handle; __u32 qpn; @@ -852,6 +867,7 @@ enum { IB_USER_VERBS_CMD_OPEN_XRCD_V2 = -1, IB_USER_VERBS_CMD_CLOSE_XRCD_V2 = -1, IB_USER_VERBS_CMD_CREATE_XSRQ_V2 = -1, + IB_USER_VERBS_CMD_OPEN_QP_V2 = -1, }; struct ibv_destroy_cq_v1 { diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index 183d171..81077d8 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -450,6 +450,12 @@ struct ibv_qp_init_attr { } ext; }; +struct ibv_qp_open_attr { + void *qp_context; + uint32_t qp_num; + enum ibv_qp_type qp_type; +}; + enum ibv_qp_attr_mask { IBV_QP_STATE = 1 << 0, IBV_QP_CUR_STATE = 1 << 1, @@ -481,7 +487,8 @@ enum ibv_qp_state { IBV_QPS_RTS, IBV_QPS_SQD, IBV_QPS_SQE, - IBV_QPS_ERR + IBV_QPS_ERR, + IBV_QPS_UNKNOWN }; enum ibv_mig_state { @@ -757,6 +764,8 @@ struct ibv_xrc_ops { struct ibv_xrcd * (*open_xrcd)(struct ibv_context *context, int fd, int oflags); int (*close_xrcd)(struct ibv_xrcd *xrcd); + struct ibv_qp * (*open_qp)(struct ibv_xrcd *xrcd, + struct ibv_qp_open_attr *attr); }; struct ibv_context { @@ -1072,6 +1081,12 @@ struct ibv_qp *ibv_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr *qp_init_attr); /** + * ibv_open_qp - Open a shareable queue pair. + */ +struct ibv_qp *ibv_open_qp(struct ibv_xrcd *xrcd, + struct ibv_qp_open_attr *qp_open_attr); + +/** * ibv_modify_qp - Modify a queue pair. */ int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, diff --git a/src/cmd.c b/src/cmd.c index 6d82d94..582d8b2 100644 --- a/src/cmd.c +++ b/src/cmd.c @@ -734,6 +734,28 @@ int ibv_cmd_create_qp(struct ibv_pd *pd, return 0; } +int ibv_cmd_open_qp(struct ibv_xrcd *xrcd, struct ibv_qp *qp, + struct ibv_qp_open_attr *attr, + struct ibv_open_qp *cmd, size_t cmd_size, + struct ibv_create_qp_resp *resp, size_t resp_size) +{ + IBV_INIT_CMD_RESP(cmd, cmd_size, OPEN_QP, resp, resp_size); + + cmd->user_handle = (uintptr_t) qp; + cmd->pd_handle = xrcd->handle; + cmd->qpn = attr->qp_num; + cmd->qp_type = attr->qp_type; + + if (write(xrcd->context->cmd_fd, cmd, cmd_size) != cmd_size) + return errno; + + VALGRIND_MAKE_MEM_DEFINED(resp, resp_size); + + qp->handle = resp->qp_handle; + + return 0; +} + int ibv_cmd_query_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, int attr_mask, struct ibv_qp_init_attr *init_attr, diff --git a/src/libibverbs.map b/src/libibverbs.map index c2f7bb4..efd6712 100644 --- a/src/libibverbs.map +++ b/src/libibverbs.map @@ -108,4 +108,6 @@ IBVERBS_1.1 { ibv_open_xrcd; ibv_close_xrcd; ibv_create_xsrq; + ibv_cmd_open_qp; + ibv_open_qp; } IBVERBS_1.0; diff --git a/src/verbs.c b/src/verbs.c index 40dbef5..1631610 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -502,6 +502,35 @@ struct ibv_qp *__ibv_create_qp(struct ibv_pd *pd, } default_symver(__ibv_create_qp, ibv_create_qp); +struct ibv_qp *__ibv_open_qp(struct ibv_xrcd *xrcd, + struct ibv_qp_open_attr *qp_open_attr) +{ + struct ibv_xrc_ops *ops; + struct ibv_qp *qp; + + ops = ibv_get_ext_ops(xrcd->context, IBV_XRC_OPS); + qp = ops->open_qp(xrcd, qp_open_attr); + if (qp) { + qp->context = xrcd->context; + qp->qp_context = qp_open_attr->qp_context; + + qp->pd = NULL; + qp->send_cq = qp->recv_cq = NULL; + qp->srq = NULL; + qp->ext.xrc_recv.xrcd = xrcd; + + qp->qp_num = qp_open_attr->qp_num; + qp->qp_type = qp_open_attr->qp_type; + qp->state = IBV_QPS_UNKNOWN; + qp->events_completed = 0; + pthread_mutex_init(&qp->mutex, NULL); + pthread_cond_init(&qp->cond, NULL); + } + + return qp; +} +default_symver(__ibv_open_qp, ibv_open_qp); + int __ibv_query_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, int attr_mask, struct ibv_qp_init_attr *init_attr)