From patchwork Thu May 19 05:29:36 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Hefty, Sean" X-Patchwork-Id: 796202 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter1.kernel.org (8.14.4/8.14.3) with ESMTP id p4J5TdP9018890 for ; Thu, 19 May 2011 05:29:39 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754105Ab1ESF3j (ORCPT ); Thu, 19 May 2011 01:29:39 -0400 Received: from mga11.intel.com ([192.55.52.93]:1074 "EHLO mga11.intel.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754049Ab1ESF3i convert rfc822-to-8bit (ORCPT ); Thu, 19 May 2011 01:29:38 -0400 Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga102.fm.intel.com with ESMTP; 18 May 2011 22:29:38 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="4.65,235,1304319600"; d="scan'208";a="3797230" Received: from orsmsx602.amr.corp.intel.com ([10.22.226.211]) by fmsmga002.fm.intel.com with ESMTP; 18 May 2011 22:29:38 -0700 Received: from orsmsx104.amr.corp.intel.com (10.22.225.131) by orsmsx602.amr.corp.intel.com (10.22.226.211) with Microsoft SMTP Server (TLS) id 8.2.255.0; Wed, 18 May 2011 22:29:38 -0700 Received: from orsmsx101.amr.corp.intel.com ([169.254.8.114]) by ORSMSX104.amr.corp.intel.com ([169.254.3.212]) with mapi id 14.01.0289.001; Wed, 18 May 2011 22:29:37 -0700 From: "Hefty, Sean" To: Roland Dreier CC: Jason Gunthorpe , "linux-rdma (linux-rdma@vger.kernel.org)" Subject: RE: [RFC] XRC upstream merge reboot Thread-Topic: [RFC] XRC upstream merge reboot Thread-Index: AcwUDiP/aTrMGg7SSlKpFkhkExJnRwBp1KaAAACkqQAADjcq0P//tYMA///Su5A= Date: Thu, 19 May 2011 05:29:36 +0000 Message-ID: <1828884A29C6694DAF28B7E6B8A82373FDDE@ORSMSX101.amr.corp.intel.com> References: <1828884A29C6694DAF28B7E6B8A82373F7AB@ORSMSX101.amr.corp.intel.com> <20110518170226.GA2595@obsidianresearch.com> <1828884A29C6694DAF28B7E6B8A82373FBC7@ORSMSX101.amr.corp.intel.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.9.131.214] MIME-Version: 1.0 Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter1.kernel.org [140.211.167.41]); Thu, 19 May 2011 05:29:40 +0000 (UTC) What about something along the lines of the following? This is 2 incomplete patches squashed together, lacking any serious documentation. I believe this will support existing apps and driver libraries, as either binary or re-compiling unmodified source code. A driver library calls ibv_register_driver_ext() if it implements the extended operations. (Roland, you mentioned changing the ibv_register_driver init_func parameter, but I didn't follow your thought.) Extended ops are defined at both the ibv_device and ibv_context levels. An app may also query a device to see if it supports a specific extension. There is an assumption that extension names reflect the availability of the extension. For example, there may be "ibv_foo", "ofed_foo", or "mlx_foo" APIs. The "ibv" extensions would be those defined by the libibverbs package. --- include/infiniband/verbs.h | 32 ++++++++++++++++++++++++++++++++ src/device.c | 18 ++++++++++++++++++ src/init.c | 12 +++++++++++- src/libibverbs.map | 5 +++++ src/verbs.c | 9 +++++++++ 5 files changed, 75 insertions(+), 1 deletions(-) -- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h index 0f1cb2e..fad875f 100644 --- a/include/infiniband/verbs.h +++ b/include/infiniband/verbs.h @@ -617,12 +617,19 @@ struct ibv_device { enum ibv_transport_type transport_type; /* Name of underlying kernel IB device, eg "mthca0" */ char name[IBV_SYSFS_NAME_MAX]; + /* + * Support for extended operations is recorded at the end of + * the name character array. + */ +#define ext_ops_supported name[IBV_SYSFS_NAME_MAX - 1] /* Name of uverbs device, eg "uverbs0" */ char dev_name[IBV_SYSFS_NAME_MAX]; /* Path to infiniband_verbs class device in sysfs */ char dev_path[IBV_SYSFS_PATH_MAX]; /* Path to infiniband class device in sysfs */ char ibdev_path[IBV_SYSFS_PATH_MAX]; + int (*have_ext_ops)(char *ext_name); + void * (*get_ext_device_ops)(char *ext_name); }; struct ibv_context_ops { @@ -683,6 +690,14 @@ struct ibv_context_ops { void (*async_event)(struct ibv_async_event *event); }; +#define IBV_XRC_OPS "ibv_xrc" + +struct ibv_xrc_ops { + struct ibv_xrcd (*open_xrcd)(); + int (*close_xrcd)(); + struct ibv_src (*create_srq)(); +}; + struct ibv_context { struct ibv_device *device; struct ibv_context_ops ops; @@ -691,6 +706,7 @@ struct ibv_context { int num_comp_vectors; pthread_mutex_t mutex; void *abi_compat; + void * (*get_ext_ops)(char *ext_name); }; /** @@ -724,6 +740,17 @@ const char *ibv_get_device_name(struct ibv_device *device); uint64_t ibv_get_device_guid(struct ibv_device *device); /** + * ibv_have_ext_ops - Return true if device supports the requested + * extended operations. + */ +int ibv_have_ext_ops(struct ibv_device *device, const char *name); + +/** + * ibv_get_device_ext_ops - Return extended operations. + */ +void *ibv_get_device_ext_ops(struct ibv_device *device, const char *name); + +/** * ibv_open_device - Initialize device for use */ struct ibv_context *ibv_open_device(struct ibv_device *device); @@ -734,6 +761,11 @@ struct ibv_context *ibv_open_device(struct ibv_device *device); int ibv_close_device(struct ibv_context *context); /** + * ibv_get_ext_ops - Return extended operations. + */ +void *ibv_get_ext_ops(struct ibv_context *context, const char *name); + +/** * ibv_get_async_event - Get next async event * @event: Pointer to use to return async event * diff --git a/src/device.c b/src/device.c index 185f4a6..da0eca4 100644 --- a/src/device.c +++ b/src/device.c @@ -181,6 +181,24 @@ int __ibv_close_device(struct ibv_context *context) } default_symver(__ibv_close_device, ibv_close_device); +int __ibv_have_ext_ops(struct ibv_device *device, const char *name) +{ + if (!device->ext_ops_supported) + return ENOSYS; + + return device->have_ext_ops(device, name); +} +default_symver(__ibv_have_ext_ops, ibv_have_ext_ops); + +void *__ibv_get_device_ext_ops(struct ibv_device *device, const char *name) +{ + if (!device->ext_ops_supported) + return NULL; + + return device->get_device_ext_ops(device, name); +} +default_symver(__ibv_get_device_ext_ops, ibv_get_device_ext_ops); + int __ibv_get_async_event(struct ibv_context *context, struct ibv_async_event *event) { diff --git a/src/init.c b/src/init.c index 4f0130e..0fdc89c 100644 --- a/src/init.c +++ b/src/init.c @@ -71,6 +71,7 @@ struct ibv_driver { const char *name; ibv_driver_init_func init_func; struct ibv_driver *next; + int ext_ops_supported; }; static struct ibv_sysfs_dev *sysfs_dev_list; @@ -153,7 +154,8 @@ static int find_sysfs_devs(void) return ret; } -void ibv_register_driver(const char *name, ibv_driver_init_func init_func) +void ibv_register_driver_ext(const char *name, ibv_driver_init_func init_func, + int ext_ops_supported) { struct ibv_driver *driver; @@ -174,6 +176,11 @@ void ibv_register_driver(const char *name, ibv_driver_init_func init_func) tail_driver = driver; } +void ibv_register_driver(const char *name, ibv_driver_init_func init_func) +{ + ibv_register_driver_ext(name, init_func, 0); +} + static void load_driver(const char *name) { char *so_name; @@ -368,6 +375,9 @@ static struct ibv_device *try_driver(struct ibv_driver *driver, strcpy(dev->name, sysfs_dev->ibdev_name); strcpy(dev->ibdev_path, sysfs_dev->ibdev_path); + if (strlen(dev->name) < sizeof(dev->name)) + dev->ext_ops_supported = driver->ext_ops_supported; + return dev; } diff --git a/src/libibverbs.map b/src/libibverbs.map index 1827da0..422e07f 100644 --- a/src/libibverbs.map +++ b/src/libibverbs.map @@ -96,4 +96,9 @@ IBVERBS_1.1 { ibv_port_state_str; ibv_event_type_str; ibv_wc_status_str; + + ibv_register_driver_ext; + ibv_have_ext_ops; + ibv_get_device_ext_ops; + ibv_get_ext_ops; } IBVERBS_1.0; diff --git a/src/verbs.c b/src/verbs.c index ba3c0a4..0be9ac2 100644 --- a/src/verbs.c +++ b/src/verbs.c @@ -76,6 +76,15 @@ enum ibv_rate mult_to_ibv_rate(int mult) } } +void *__ibv_get_ext_ops(struct ibv_context *context, const char *name) +{ + if (!context->device->ext_ops_supported) + return NULL; + + return context->get_ext_ops(context, name); +} +default_symver(__ibv_get_ext_ops, ibv_get_ext_ops); + int __ibv_query_device(struct ibv_context *context, struct ibv_device_attr *device_attr) {