From patchwork Tue Jul 14 20:29:43 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jason Gunthorpe X-Patchwork-Id: 6789131 Return-Path: X-Original-To: patchwork-linux-nfs@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork1.web.kernel.org (Postfix) with ESMTP id B7EC69F3E6 for ; Tue, 14 Jul 2015 20:30:07 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id A33712061E for ; Tue, 14 Jul 2015 20:30:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 80A6120658 for ; Tue, 14 Jul 2015 20:30:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752752AbbGNUaD (ORCPT ); Tue, 14 Jul 2015 16:30:03 -0400 Received: from quartz.orcorp.ca ([184.70.90.242]:46355 "EHLO quartz.orcorp.ca" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752781AbbGNUaA (ORCPT ); Tue, 14 Jul 2015 16:30:00 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=obsidianresearch.com; s=rsa1; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=ZNd/Adhl20c/38bpQY5SN6uA6jMUUW6g5kI8LyM8WP4=; b=MTRElXSC0gA+DWA4FpzMOZiv7NoVwJpt86AMPTAc+vZ3y/qRnRqTOw9G8GJq0u5saL5eJ1AqkbnU8LRiFbIipy1CV7Qqsf+gFDx6avU79CsLbk1arqiSfX7vhoIErWuG7PrlOjrU7GcoIp0BijLbSegy1e6Gkoej88swlAVg3GM=; Received: from [10.0.0.192] (helo=jggl.edm.orcorp.ca) by quartz.orcorp.ca with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84) (envelope-from ) id 1ZF6pk-0005vx-1p; Tue, 14 Jul 2015 14:29:44 -0600 Received: from jgg by jggl.edm.orcorp.ca with local (Exim 4.84) (envelope-from ) id 1ZF6pj-0005RV-TO; Tue, 14 Jul 2015 14:29:43 -0600 Date: Tue, 14 Jul 2015 14:29:43 -0600 From: Jason Gunthorpe To: 'Christoph Hellwig' Cc: Steve Wise , 'Sagi Grimberg' , 'Steve Wise' , 'Tom Talpey' , 'Doug Ledford' , sagig@mellanox.com, ogerlitz@mellanox.com, roid@mellanox.com, linux-rdma@vger.kernel.org, eli@mellanox.com, target-devel@vger.kernel.org, linux-nfs@vger.kernel.org, trond.myklebust@primarydata.com, bfields@fieldses.org, 'Oren Duer' Subject: Re: [PATCH V3 1/5] RDMA/core: Transport-independent access flags Message-ID: <20150714202943.GB26927@obsidianresearch.com> References: <559EF332.7060103@redhat.com> <20150709225306.GA30741@obsidianresearch.com> <559FC710.1050307@talpey.com> <20150710161108.GA19042@obsidianresearch.com> <55A24571.60902@dev.mellanox.co.il> <00e201d0be6a$e49bc910$add35b30$@opengridcomputing.com> <20150714192941.GA26292@obsidianresearch.com> <00e401d0be6b$d3952750$7abf75f0$@opengridcomputing.com> <20150714195511.GB7716@infradead.org> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20150714195511.GB7716@infradead.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Broken-Reverse-DNS: no host name found for IP address 10.0.0.192 Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Spam-Status: No, score=-8.2 required=5.0 tests=BAYES_00,DKIM_SIGNED, RCVD_IN_DNSWL_HI,RP_MATCHES_RCVD,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP On Tue, Jul 14, 2015 at 12:55:11PM -0700, 'Christoph Hellwig' wrote: > On Tue, Jul 14, 2015 at 02:32:31PM -0500, Steve Wise wrote: > > You mean "should not", yea? > > > > Ok. I'll check for iWARP. But don't tell me to remove the transport-specific hacks in this series when I post it! ;) > > Just curious if there are any holes in this little scheme to deal with > the lkey mess: > > (1) make sure all drivers that currently do not set > IB_DEVICE_LOCAL_DMA_LKEY but which can safely use ib_get_dma_mr > call it underneath at device setup time, and tear it down before > removal. Yes, I'd like this. local_dma_lkey appears to be global, it works with any PD. ib_get_dma_mr is tied to a PD, so it cannot replace local_dma_lkey at the struct device level. ib_alloc_pd is the in-kernel entry point, the UAPI calls device->alloc_pd directly.. So how about the below patch as a starting >point? (Steve the goal of step #1 would be to remove ib_get_dma_mr from ULPs Follow on patches would be to convert all ULPs to use this API change.) In the long run this also makes it easier to develop helper wrappers around posting because a local_dma_lkey is now always accessible to the core code. > (2) now ULD can check for IB_DEVICE_LOCAL_DMA_LKEY and use local_dma_lkey > in that case, or will have to perform a per-IO MR with LOCAL and > REMOTE flags if not If we do the above all ULPs simply use pd->local_dma_lkey and we drop correct uses of ib_get_dma_mr completely ? > (3) remove insecure remote uses of ib_get_dma_mr from ULDs > (4) remove ib_get_dma_mr from the public API Sure would be nice! > This should help to sort out the lkey side of the memory registrations, > and isn't too hard. For rkeys I'd love to go with something like Sagis > proposal as a first step, and then we can see if we can refine it in > another iteration. Agree. I'd love to see FMR fit under that, but even if we cannot, it is appears to be a sane way to advance indirect MR.. Jason From 5a9799bf487d822daae5a8b8f3b197f1dda1db45 Mon Sep 17 00:00:00 2001 From: Jason Gunthorpe Date: Tue, 14 Jul 2015 14:27:37 -0600 Subject: [PATCH] IB/core: Guarantee that a local_dma_lkey is available Every single ULP requires a local_dma_lkey to do anything with a QP, so lets us ensure one exists for every PD created. If the driver can supply a global local_dma_lkey then use that, otherwise ask the driver to create a local use all physical memory MR associated with the new PD. Signed-off-by: Jason Gunthorpe Acked-by: Christoph Hellwig --- drivers/infiniband/core/verbs.c | 40 +++++++++++++++++++++++++++++++++++----- include/rdma/ib_verbs.h | 2 ++ 2 files changed, 37 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/core/verbs.c b/drivers/infiniband/core/verbs.c index bac3fb406a74..1ddf06314f36 100644 --- a/drivers/infiniband/core/verbs.c +++ b/drivers/infiniband/core/verbs.c @@ -213,24 +213,54 @@ EXPORT_SYMBOL(rdma_port_get_link_layer); /* Protection domains */ +/* Return a pd for in-kernel use that has a local_dma_lkey which provides + local access to all physical memory. */ struct ib_pd *ib_alloc_pd(struct ib_device *device) { struct ib_pd *pd; + struct ib_device_attr devattr; + int rc; + + rc = ib_query_device(device, &devattr); + if (rc) + return ERR_PTR(rc); pd = device->alloc_pd(device, NULL, NULL); + if (IS_ERR(pd)) + return pd; + + pd->device = device; + pd->uobject = NULL; + pd->local_mr = NULL; + atomic_set(&pd->usecnt, 0); + + if (devattr.device_cap_flags & IB_DEVICE_LOCAL_DMA_LKEY) + pd->local_dma_lkey = device->local_dma_lkey; + else { + struct ib_mr *mr; + + mr = ib_get_dma_mr(pd, IB_ACCESS_LOCAL_WRITE); + if (IS_ERR(mr)) { + ib_dealloc_pd(pd); + return (struct ib_pd *)mr; + } - if (!IS_ERR(pd)) { - pd->device = device; - pd->uobject = NULL; - atomic_set(&pd->usecnt, 0); + pd->local_mr = mr; + pd->local_dma_lkey = pd->local_mr->lkey; } - return pd; } + EXPORT_SYMBOL(ib_alloc_pd); int ib_dealloc_pd(struct ib_pd *pd) { + if (pd->local_mr) { + if (ib_dereg_mr(pd->local_mr)) + return -EBUSY; + pd->local_mr = NULL; + } + if (atomic_read(&pd->usecnt)) return -EBUSY; diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h index 986fddb08579..cfda95d7b067 100644 --- a/include/rdma/ib_verbs.h +++ b/include/rdma/ib_verbs.h @@ -1255,6 +1255,8 @@ struct ib_pd { struct ib_device *device; struct ib_uobject *uobject; atomic_t usecnt; /* count all resources */ + struct ib_mr *local_mr; + u32 local_dma_lkey; }; struct ib_xrcd {