From patchwork Tue Jan 30 03:34:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 10191289 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id D0950601A0 for ; Tue, 30 Jan 2018 03:34:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B0DBC288E7 for ; Tue, 30 Jan 2018 03:34:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A497E28918; Tue, 30 Jan 2018 03:34:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A9A37288E7 for ; Tue, 30 Jan 2018 03:34:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752479AbeA3Dek (ORCPT ); Mon, 29 Jan 2018 22:34:40 -0500 Received: from mail-it0-f68.google.com ([209.85.214.68]:36517 "EHLO mail-it0-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752131AbeA3Dej (ORCPT ); Mon, 29 Jan 2018 22:34:39 -0500 Received: by mail-it0-f68.google.com with SMTP id n206so9333796itg.1 for ; Mon, 29 Jan 2018 19:34:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=zIFJktzTxxJ5v0aBE468KNV2qjQ/cWJzJlKH3Kktv/4=; b=YoaiYSBKa6Rk95FsUmJN2WeSbUfcUal9esXfZZ27CZEGugU7SaLWjUNCT5EwH1QQ0Y lXcOV0ClvV8Pu7hlxsC+mDtoOTeS6JgwabWbjCHSTnuy98ngHfd5niPIDzgG/sHQdZVT G0tW+e83M+ouwWzH2+Jfwxdzf0j3i3CAlxXfBMB+b3n2V/7nvqcl2Y5QtKk8JO+e+u1X qeT5+7NTvckDjAqpCf0CrOuNI3p80WPBUjMjWl7gssdqtvIVSn52tKvjQpkHu4W09LTG QKW++MoLo43HQ6dCMompsmckSTP1xvuD7OkXaGmuN3FEE/Bm1ek4MSPaMR6XSnoQM2Gc Qj+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=zIFJktzTxxJ5v0aBE468KNV2qjQ/cWJzJlKH3Kktv/4=; b=f94cW4mVMVBuDSLKvn4bm0yU2rSbA9gpa+v8d46TgGqcpjmBdJJGY7rhzREFQhBPtn ZlSV9Z7W9oPOl4EabiuRGNk3hj2kfWwDggoO7m8gWHV3nsL1nX/nojkGdGUnA4exjArq 1Mp8nWSyycnlcs8JnpLRtlTHSo/PLONLaowd85yBppBk1I4JLSDujezLPQJT/ygOqzNM /AC5/Mts+xiW+6l50FKqnmWJj2zHtmcdJ9QOt9JGEkmGLRglmeyKJFD25cR/5gSIbcKg rC8DR2NL2wx4wJtdE8oy47DfgfxnDmqgTQ5Xj/IRgTaKhZ9yx/58+V00rOAp8mBGyf1R 5bYA== X-Gm-Message-State: AKwxytec4TumlQvaHQK0GvYPBPd9EiyxWGiyYXk5XpzetLUEdlQY5pUQ zvEx5Yr8CbkQi4pf3jUjHaHXZw== X-Google-Smtp-Source: AH8x227ZWeS8fYkrrU+M+fEeV9atXtdFol7BG0G5XfWNkCbXudEWsw8yb+QxH4X4rHGQi6M92kOG8g== X-Received: by 10.36.91.210 with SMTP id g201mr29849041itb.50.1517283278999; Mon, 29 Jan 2018 19:34:38 -0800 (PST) Received: from ziepe.ca (S010614cc2056d97f.ed.shawcable.net. [70.74.179.152]) by smtp.gmail.com with ESMTPSA id i22sm5099793ioo.39.2018.01.29.19.34.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 29 Jan 2018 19:34:37 -0800 (PST) Received: from jgg by mlx.ziepe.ca with local (Exim 4.86_2) (envelope-from ) id 1egMgu-0002e8-Oq; Mon, 29 Jan 2018 20:34:36 -0700 Date: Mon, 29 Jan 2018 20:34:36 -0700 From: Jason Gunthorpe To: Doug Ledford Cc: Leon Romanovsky , RDMA mailing list , Mark Bloch , Steve Wise Subject: Re: [PATCH rdma-next v7 0/8] RDMA resource tracking Message-ID: <20180130033436.GA17053@ziepe.ca> References: <20180128091725.13103-1-leon@kernel.org> <20180128210520.GK23869@ziepe.ca> <1517256713.27592.241.camel@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <1517256713.27592.241.camel@redhat.com> User-Agent: Mutt/1.5.24 (2015-08-30) Sender: linux-rdma-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Mon, Jan 29, 2018 at 03:11:53PM -0500, Doug Ledford wrote: > On Sun, 2018-01-28 at 14:05 -0700, Jason Gunthorpe wrote: > > On Sun, Jan 28, 2018 at 11:17:17AM +0200, Leon Romanovsky wrote: > > > > > The original goal of this series was to allow ability to view connection > > > (QP) information about running processes, however I used this opportunity and > > > created common infrastructure to track and report various resources. The report > > > part is implemented in netlink (nldev), but smart ULPs can now create > > > advanced usage models based on device utilization. > > > > > > The current implementation relies on one lock per-object per-device, so > > > creation/destroying of various objects (CQ, PD, e.t.c) on various or the > > > same devices doesn't interfere each with another. > > > > > > The data protection is performed with SRCU and its reader-writer model > > > ensures that resource won't be destroyed till readers will finish their > > > work. > > > > Well, this cover letter isn't quite right anymore.. but no matter. > > > > My small comments aside it looks OK to me. > > Likewise. I'm happy with it at this point. Okay, I fixed up the small things and applied the patches to for-next Leon: Please validate I didn't screw it up. Here is the diff against what you sent: - Success path on the main execution flow, not under an if - constify static structure - Remove confusing comment about locking, ib_enum_all_devs obtains locks to iterate its list and rdma_restrack_count holds res->rwsem so everything is accounted for directly without trickyness - Speeling - Remove extra lock in rdma_restrack_del - Restore pd = NULL in ib_create_xrc_qp. This scraed me a bit, xrc is wonky. But ib_create_xrc_q is only called in cases where rdma_restrack_add is not added, so keeping things as-they-are should not impact restrack. If restrack needs the pd for a XRC someday it should get it from qp->real_qp - Remove SET/NEW/DEL cargo cult, please send a patch for rest? Branch is here: https://git.kernel.org/pub/scm/linux/kernel/git/rdma/rdma.git/log/?h=wip/jgg-for-next Still unhappy with the kref-as-not-a-kref. --- To unsubscribe from this list: send the line "unsubscribe linux-rdma" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/drivers/infiniband/core/core_priv.h b/drivers/infiniband/core/core_priv.h index 3dcacf220e5e7e..c4560d84dfaebd 100644 --- a/drivers/infiniband/core/core_priv.h +++ b/drivers/infiniband/core/core_priv.h @@ -310,21 +310,21 @@ static inline struct ib_qp *_ib_create_qp(struct ib_device *dev, struct ib_qp *qp; qp = dev->create_qp(pd, attr, udata); - if (!IS_ERR(qp)) { - qp->device = dev; - qp->pd = pd; - /* - * We don't track XRC QPs for now, because they don't have PD - * and more importantly they are created internaly by driver, - * see mlx5 create_dev_resources() as an example. - */ - if (attr->qp_type < IB_QPT_XRC_INI) { - qp->res.type = RDMA_RESTRACK_QP; - rdma_restrack_add(&qp->res); - } else { - qp->res.valid = false; - } - } + if (IS_ERR(qp)) + return qp; + + qp->device = dev; + qp->pd = pd; + /* + * We don't track XRC QPs for now, because they don't have PD + * and more importantly they are created internaly by driver, + * see mlx5 create_dev_resources() as an example. + */ + if (attr->qp_type < IB_QPT_XRC_INI) { + qp->res.type = RDMA_RESTRACK_QP; + rdma_restrack_add(&qp->res); + } else + qp->res.valid = false; return qp; } diff --git a/drivers/infiniband/core/nldev.c b/drivers/infiniband/core/nldev.c index 421d5fa6a81731..fa8655e3b3edfe 100644 --- a/drivers/infiniband/core/nldev.c +++ b/drivers/infiniband/core/nldev.c @@ -178,7 +178,7 @@ static int fill_res_info_entry(struct sk_buff *msg, static int fill_res_info(struct sk_buff *msg, struct ib_device *device) { - static const char *names[RDMA_RESTRACK_MAX] = { + static const char * const names[RDMA_RESTRACK_MAX] = { [RDMA_RESTRACK_PD] = "pd", [RDMA_RESTRACK_CQ] = "cq", [RDMA_RESTRACK_QP] = "qp", @@ -553,10 +553,6 @@ static int _nldev_res_get_dumpit(struct ib_device *device, static int nldev_res_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { - /* - * There is no need to take lock, because - * we are relying on ib_core's lists_rwsem - */ return ib_enum_all_devs(_nldev_res_get_dumpit, skb, cb); } @@ -627,8 +623,8 @@ static int nldev_res_get_qp_dumpit(struct sk_buff *skb, (!rdma_is_kernel_res(res) && task_active_pid_ns(current) != task_active_pid_ns(res->task))) /* - * 1. Kernel QPs should be visible in init namsapce only - * 2. Preent only QPs visible in the current namespace + * 1. Kernel QPs should be visible in init namspace only + * 2. Present only QPs visible in the current namespace */ goto next; diff --git a/drivers/infiniband/core/restrack.c b/drivers/infiniband/core/restrack.c index 351b6940f6dc17..857637bf46da27 100644 --- a/drivers/infiniband/core/restrack.c +++ b/drivers/infiniband/core/restrack.c @@ -150,9 +150,7 @@ void rdma_restrack_del(struct rdma_restrack_entry *res) if (!dev) return; - down_read(&dev->res.rwsem); rdma_restrack_put(res); - up_read(&dev->res.rwsem); wait_for_completion(&res->comp); diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index a98a3e8412f810..16ebc6372c31ab 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -849,6 +849,7 @@ static struct ib_qp *ib_create_xrc_qp(struct ib_qp *qp, qp->event_handler = __ib_shared_qp_event_handler; qp->qp_context = qp; + qp->pd = NULL; qp->send_cq = qp->recv_cq = NULL; qp->srq = NULL; qp->xrcd = qp_init_attr->xrcd; diff --git a/include/uapi/rdma/rdma_netlink.h b/include/uapi/rdma/rdma_netlink.h index 23bef401598208..17e59bec169ec0 100644 --- a/include/uapi/rdma/rdma_netlink.h +++ b/include/uapi/rdma/rdma_netlink.h @@ -237,14 +237,8 @@ enum rdma_nldev_command { RDMA_NLDEV_CMD_PORT_DEL, RDMA_NLDEV_CMD_RES_GET, /* can dump */ - RDMA_NLDEV_CMD_RES_SET, - RDMA_NLDEV_CMD_RES_NEW, - RDMA_NLDEV_CMD_RES_DEL, RDMA_NLDEV_CMD_RES_QP_GET, /* can dump */ - RDMA_NLDEV_CMD_RES_QP_SET, - RDMA_NLDEV_CMD_RES_QP_NEW, - RDMA_NLDEV_CMD_RES_QP_DEL, RDMA_NLDEV_NUM_OPS };