From patchwork Thu Sep 13 19:16:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Wise X-Patchwork-Id: 10614659 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 45C9F1390 for ; Tue, 25 Sep 2018 17:41:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 368132AAE3 for ; Tue, 25 Sep 2018 17:41:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2A8572AAEE; Tue, 25 Sep 2018 17:41:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_96_XX, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9F1A62AAE3 for ; Tue, 25 Sep 2018 17:41:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726456AbeIYXt7 (ORCPT ); Tue, 25 Sep 2018 19:49:59 -0400 Received: from 72-48-214-68.dyn.grandenetworks.net ([72.48.214.68]:57552 "EHLO smtp.opengridcomputing.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726163AbeIYXt7 (ORCPT ); Tue, 25 Sep 2018 19:49:59 -0400 Received: by smtp.opengridcomputing.com (Postfix, from userid 503) id 1728F22786; Tue, 25 Sep 2018 12:41:25 -0500 (CDT) Message-Id: <5753829890367ccdd54a6a80d2420930922dd4cd.1537896647.git.swise@opengridcomputing.com> In-Reply-To: References: From: Steve Wise Date: Thu, 13 Sep 2018 12:16:20 -0700 Subject: [RFC WIP 1/2] RDMA/Core: add RDMA_NLDEV_CMD_NEWLINK/DELLLINK support To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, BMT@zurich.ibm.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add support for new LINK messages to allow adding and deleting rdma interfaces. This will be used initially for soft rdma drivers which instantiate device instances dynamically by the admin specifying a netdev device to use. The rdma_rxe module will be the first user of these messages. The design is modeled after RTNL_NEWLINK/DELLINK: rdma drivers register with the rdma core if they provide link add/delete functions. Each driver registers with a unique "type" string, that is used to dispatch messages coming from user space. A new RDMA_NLDEV_ATTR is defined for the "type" string. User mode will pass 3 attributes in a NEWLINK message: RDMA_NLDEV_ATTR_IBDEV_NAME for the desired rdma device name to be created, RDMA_NLDEV_ATTR_LINK_TYPE for the "type" of link being added, and RDMA_NLDEV_ATTR_NDEV_NAME for the net_device interface to use for this link. The DELLINK message will contain the IBDEV_NAME and LINK_TYPE attributes. Signed-off-by: Steve Wise --- drivers/infiniband/core/nldev.c | 113 +++++++++++++++++++++++++++++++++++++++ include/rdma/rdma_netlink.h | 11 ++++ include/uapi/rdma/rdma_netlink.h | 9 ++++ 3 files changed, 133 insertions(+) diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c index 0385ab438320..d107b982c210 100644 --- a/drivers/infiniband/core/nldev.c +++ b/drivers/infiniband/core/nldev.c @@ -33,6 +33,7 @@ #include #include #include +#include #include #include #include @@ -107,6 +108,8 @@ [RDMA_NLDEV_ATTR_DRIVER_U32] = { .type = NLA_U32 }, [RDMA_NLDEV_ATTR_DRIVER_S64] = { .type = NLA_S64 }, [RDMA_NLDEV_ATTR_DRIVER_U64] = { .type = NLA_U64 }, + [RDMA_NLDEV_ATTR_LINK_TYPE] = { .type = NLA_NUL_STRING, + .len = IFNAMSIZ }, }; static int put_driver_name_print_type(struct sk_buff *msg, const char *name, @@ -1072,6 +1075,110 @@ static int nldev_res_get_pd_dumpit(struct sk_buff *skb, return res_get_common_dumpit(skb, cb, RDMA_RESTRACK_PD); } +static LIST_HEAD(link_ops); +static DEFINE_MUTEX(link_ops_mutex); + +void rdma_link_register(struct rdma_link_ops *ops) +{ + mutex_lock(&link_ops_mutex); + list_add(&ops->list, &link_ops); + mutex_unlock(&link_ops_mutex); +} +EXPORT_SYMBOL(rdma_link_register); + +void rdma_link_unregister(struct rdma_link_ops *ops) +{ + mutex_lock(&link_ops_mutex); + list_del_init(&ops->list); + mutex_unlock(&link_ops_mutex); +} +EXPORT_SYMBOL(rdma_link_unregister); + +static const struct rdma_link_ops *link_ops_get(const char *type) +{ + const struct rdma_link_ops *ops; + + mutex_lock(&link_ops_mutex); + list_for_each_entry(ops, &link_ops, list) { + if (!strncmp(ops->type, type, IFNAMSIZ)) + goto out; + } + ops = NULL; +out: + mutex_unlock(&link_ops_mutex); + return ops; +} + +static int nldev_newlink(struct sk_buff *skb, struct nlmsghdr *nlh, + struct netlink_ext_ack *extack) +{ + struct nlattr *tb[RDMA_NLDEV_ATTR_MAX]; + char ibdev_name[IB_DEVICE_NAME_MAX]; + const struct rdma_link_ops *ops; + char ndev_name[IFNAMSIZ]; + char type[IFNAMSIZ]; + int err; + + err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); + if (err || !tb[RDMA_NLDEV_ATTR_DEV_NAME] || + !tb[RDMA_NLDEV_ATTR_LINK_TYPE] || !tb[RDMA_NLDEV_ATTR_NDEV_NAME]) { + err = -EINVAL; + goto err_out; + } + + nla_strlcpy(ibdev_name, tb[RDMA_NLDEV_ATTR_DEV_NAME], + sizeof(ibdev_name)); + nla_strlcpy(type, tb[RDMA_NLDEV_ATTR_LINK_TYPE], sizeof(type)); + nla_strlcpy(ndev_name, tb[RDMA_NLDEV_ATTR_NDEV_NAME], + sizeof(ndev_name)); + pr_debug("ibdev_name |%s| type |%s| ndev_name |%s|\n", ibdev_name, + type, ndev_name); + + ops = link_ops_get(type); + if (!ops) { + err = -ENODEV; + goto err_out; + } + + err = ops->newlink(ibdev_name, ndev_name); +err_out: + return err; +} + +static int nldev_dellink(struct sk_buff *skb, struct nlmsghdr *nlh, + struct netlink_ext_ack *extack) +{ + struct nlattr *tb[RDMA_NLDEV_ATTR_MAX]; + char ibdev_name[IB_DEVICE_NAME_MAX]; + const struct rdma_link_ops *ops; + char type[IFNAMSIZ]; + int err; + + err = nlmsg_parse(nlh, 0, tb, RDMA_NLDEV_ATTR_MAX - 1, + nldev_policy, extack); + if (err || !tb[RDMA_NLDEV_ATTR_DEV_NAME] || + !tb[RDMA_NLDEV_ATTR_LINK_TYPE]) { + err = !err ? -EINVAL : err; + goto err_out; + } + + nla_strlcpy(ibdev_name, tb[RDMA_NLDEV_ATTR_DEV_NAME], + sizeof(ibdev_name)); + nla_strlcpy(type, tb[RDMA_NLDEV_ATTR_LINK_TYPE], sizeof(type)); + pr_debug("ibdev_name |%s| type |%s|\n", ibdev_name, type); + + ops = link_ops_get(type); + if (!ops) { + err = -ENODEV; + goto err_out; + } + + err = ops->dellink(ibdev_name); +err_out: + return err; +} + static const struct rdma_nl_cbs nldev_cb_table[RDMA_NLDEV_NUM_OPS] = { [RDMA_NLDEV_CMD_GET] = { .doit = nldev_get_doit, @@ -1110,6 +1217,12 @@ static int nldev_res_get_pd_dumpit(struct sk_buff *skb, [RDMA_NLDEV_CMD_RES_PD_GET] = { .dump = nldev_res_get_pd_dumpit, }, + [RDMA_NLDEV_CMD_NEWLINK] = { + .doit = nldev_newlink, + }, + [RDMA_NLDEV_CMD_DELLINK] = { + .doit = nldev_dellink, + }, }; void __init nldev_init(void) diff --git a/include/rdma/rdma_netlink.h b/include/rdma/rdma_netlink.h index c369703fcd69..b5a5a1a7cddc 100644 --- a/include/rdma/rdma_netlink.h +++ b/include/rdma/rdma_netlink.h @@ -99,4 +99,15 @@ int ibnl_put_attr(struct sk_buff *skb, struct nlmsghdr *nlh, * Returns 0 on success or a negative for no listeners. */ int rdma_nl_chk_listeners(unsigned int group); + +struct rdma_link_ops { + struct list_head list; + const char *type; + int (*newlink)(char *ibdev_name, char *ndev_name); + int (*dellink)(char *ibdev_name); +}; + +void rdma_link_register(struct rdma_link_ops *ops); +void rdma_link_unregister(struct rdma_link_ops *ops); + #endif /* _RDMA_NETLINK_H */ diff --git a/include/uapi/rdma/rdma_netlink.h b/include/uapi/rdma/rdma_netlink.h index edba6351ac13..025172d41fa7 100644 --- a/include/uapi/rdma/rdma_netlink.h +++ b/include/uapi/rdma/rdma_netlink.h @@ -246,6 +246,10 @@ enum rdma_nldev_command { RDMA_NLDEV_CMD_RES_PD_GET, /* can dump */ + RDMA_NLDEV_CMD_NEWLINK, + + RDMA_NLDEV_CMD_DELLINK, + RDMA_NLDEV_NUM_OPS }; @@ -427,6 +431,11 @@ enum rdma_nldev_attr { RDMA_NLDEV_ATTR_DRIVER_U64, /* u64 */ /* + * Identifies the rdma driver. eg: "rxe" or "siw" + */ + RDMA_NLDEV_ATTR_LINK_TYPE, /* string */ + + /* * Always the end */ RDMA_NLDEV_ATTR_MAX From patchwork Fri Sep 14 18:43:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steve Wise X-Patchwork-Id: 10614661 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D2981390 for ; Tue, 25 Sep 2018 17:41:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D43C2AAE3 for ; Tue, 25 Sep 2018 17:41:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 710822AAEE; Tue, 25 Sep 2018 17:41:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.5 required=2.0 tests=BAYES_00,DATE_IN_PAST_96_XX, MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F1BBE2AAE3 for ; Tue, 25 Sep 2018 17:41:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726460AbeIYXuF (ORCPT ); Tue, 25 Sep 2018 19:50:05 -0400 Received: from 72-48-214-68.dyn.grandenetworks.net ([72.48.214.68]:57576 "EHLO smtp.opengridcomputing.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726163AbeIYXuF (ORCPT ); Tue, 25 Sep 2018 19:50:05 -0400 Received: by smtp.opengridcomputing.com (Postfix, from userid 503) id 1F4A422786; Tue, 25 Sep 2018 12:41:30 -0500 (CDT) Message-Id: <4df04403a9c875e30fdd96be95072e8b7b857f50.1537896647.git.swise@opengridcomputing.com> In-Reply-To: References: From: Steve Wise Date: Fri, 14 Sep 2018 11:43:55 -0700 Subject: [RFC WIP 2/2] rdma_rxe: use netlink messages to add/delete links To: linux-rdma@vger.kernel.org Cc: leon@kernel.org, jgg@ziepe.ca, BMT@zurich.ibm.com Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Steve Wise --- drivers/infiniband/sw/rxe/rxe.c | 61 +++++++++++++++++++++++++++++++++++ drivers/infiniband/sw/rxe/rxe_net.c | 3 +- drivers/infiniband/sw/rxe/rxe_net.h | 2 +- drivers/infiniband/sw/rxe/rxe_sysfs.c | 6 ++-- drivers/infiniband/sw/rxe/rxe_verbs.c | 1 - 5 files changed, 67 insertions(+), 6 deletions(-) diff --git a/drivers/infiniband/sw/rxe/rxe.c b/drivers/infiniband/sw/rxe/rxe.c index 10999fa69281..83f52954c300 100644 --- a/drivers/infiniband/sw/rxe/rxe.c +++ b/drivers/infiniband/sw/rxe/rxe.c @@ -31,6 +31,7 @@ * SOFTWARE. */ +#include #include #include "rxe.h" #include "rxe_loc.h" @@ -340,6 +341,64 @@ void rxe_remove(struct rxe_dev *rxe) rxe_dev_put(rxe); } +static int rxe_newlink(char *ibdev_name, char *ndev_name) +{ + struct net_device *ndev = NULL; + struct rxe_dev *rxe; + int err = 0; + + ndev = dev_get_by_name(&init_net, ndev_name); + if (!ndev) { + pr_err("interface %s not found\n", ndev_name); + err = -ENODEV; + goto err; + } + + if (net_to_rxe(ndev)) { + pr_err("already configured on %s\n", ndev_name); + err = -EEXIST; + goto err; + } + + rxe = rxe_net_add(ibdev_name, ndev); + if (!rxe) { + pr_err("failed to add %s\n", ndev_name); + err = -EINVAL; + goto err; + } + + if (netif_running(ndev) && netif_carrier_ok(ndev)) + rxe_port_up(rxe); + else + rxe_port_down(rxe); + pr_info("added %s to %s\n", rxe->ib_dev.name, ndev->name); +err: + if (ndev) + dev_put(ndev); + return err; +} + +static int rxe_dellink(char *ibdev_name) +{ + struct rxe_dev *rxe; + + rxe = get_rxe_by_name(ibdev_name); + if (!rxe) { + pr_err("not configured on %s\n", ibdev_name); + return -ENODEV; + } + + list_del(&rxe->list); + rxe_remove(rxe); + return 0; +} + +static struct rdma_link_ops rxe_link_ops = { + .type = "rxe", + .newlink = rxe_newlink, + .dellink = rxe_dellink, +}; + static int __init rxe_module_init(void) { int err; @@ -355,12 +414,14 @@ static int __init rxe_module_init(void) if (err) return err; + rdma_link_register(&rxe_link_ops); pr_info("loaded\n"); return 0; } static void __exit rxe_module_exit(void) { + rdma_link_unregister(&rxe_link_ops); rxe_remove_all(); rxe_net_exit(); rxe_cache_exit(); diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index 8094cbaa54a9..0c520999c8d4 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -565,7 +565,7 @@ enum rdma_link_layer rxe_link_layer(struct rxe_dev *rxe, unsigned int port_num) return IB_LINK_LAYER_ETHERNET; } -struct rxe_dev *rxe_net_add(struct net_device *ndev) +struct rxe_dev *rxe_net_add(char *ibdev_name, struct net_device *ndev) { int err; struct rxe_dev *rxe = NULL; @@ -575,6 +575,7 @@ struct rxe_dev *rxe_net_add(struct net_device *ndev) return NULL; rxe->ndev = ndev; + strlcpy(rxe->ib_dev.name, ibdev_name, IB_DEVICE_NAME_MAX); err = rxe_add(rxe, ndev->mtu); if (err) { diff --git a/drivers/infiniband/sw/rxe/rxe_net.h b/drivers/infiniband/sw/rxe/rxe_net.h index 106c586dbb26..5e646039d393 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.h +++ b/drivers/infiniband/sw/rxe/rxe_net.h @@ -43,7 +43,7 @@ struct rxe_recv_sockets { struct socket *sk6; }; -struct rxe_dev *rxe_net_add(struct net_device *ndev); +struct rxe_dev *rxe_net_add(char *ibdev_name, struct net_device *ndev); int rxe_net_init(void); void rxe_net_exit(void); diff --git a/drivers/infiniband/sw/rxe/rxe_sysfs.c b/drivers/infiniband/sw/rxe/rxe_sysfs.c index d5ed7571128f..1f82019ff8f3 100644 --- a/drivers/infiniband/sw/rxe/rxe_sysfs.c +++ b/drivers/infiniband/sw/rxe/rxe_sysfs.c @@ -97,7 +97,7 @@ static int rxe_param_set_add(const char *val, const struct kernel_param *kp) goto err; } - rxe = rxe_net_add(ndev); + rxe = rxe_net_add(NULL, ndev); if (!rxe) { pr_err("failed to add %s\n", intf); err = -EINVAL; @@ -152,6 +152,6 @@ static int rxe_param_set_remove(const char *val, const struct kernel_param *kp) }; module_param_cb(add, &rxe_add_ops, NULL, 0200); -MODULE_PARM_DESC(add, "Create RXE device over network interface"); +MODULE_PARM_DESC(add, "DEPRECATED. Create RXE device over network interface"); module_param_cb(remove, &rxe_remove_ops, NULL, 0200); -MODULE_PARM_DESC(remove, "Remove RXE device over network interface"); +MODULE_PARM_DESC(remove, "DEPRECATED. Remove RXE device over network interface"); diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index f5b1e0ad6142..d7e39408c3af 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -1159,7 +1159,6 @@ int rxe_register_device(struct rxe_dev *rxe) struct ib_device *dev = &rxe->ib_dev; struct crypto_shash *tfm; - strlcpy(dev->name, "rxe%d", IB_DEVICE_NAME_MAX); strlcpy(dev->node_desc, "rxe", sizeof(dev->node_desc)); dev->owner = THIS_MODULE;