From patchwork Mon Apr 29 15:25:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13647269 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 91E6682899; Mon, 29 Apr 2024 15:25:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404344; cv=none; b=miOxgg5jOqjsWDzFzeqx3cpuoCv9zqPM1W9R2eziKF0fU+dWGsZ+4i0IisHJDx+0aDBbizN5G5SfDTFkECYvy1r5tCZR1KKBYEKK3Fj17hU71L2JnqQSh/c8auFahtg9/xjqJ7ou02K0drJoDx6WGlsBTVAQDxKI8SkgdBrt3h0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404344; c=relaxed/simple; bh=SBTI5/HIEyy11LCUlDCzM8NfKdiPa85kJMeTkvD+bHA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=gZ2XwiR9zo6PMJxX+Fu6eh6VDba3hGRyvnFStxfMs1NJNomh+CXNbfxq6/ACtmhIgxrpTyG4NxvF9emyzYgb3mFJ4jKc5Jc07DdKNdP3RXJoht9SrfLktJk3t+r6YewCJSXt85y99PCWQqkn1O9wQ3ZGb+cwT0Df4Mfs93vJFSQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=qc0SD1Vt; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="qc0SD1Vt" Received: by smtp.kernel.org (Postfix) with ESMTPSA id D247FC116B1; Mon, 29 Apr 2024 15:25:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714404344; bh=SBTI5/HIEyy11LCUlDCzM8NfKdiPa85kJMeTkvD+bHA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=qc0SD1VtRGa6pDnFuq0SBfXZ3cMgmEPMqgl6xhcRaYbQR6t9Le9xEHl5LsXG/0+HJ XHTbrhlPHmaVVUdGkGAQ3QTdqzxfViJZOQJlxG1aSg4+TYRACX+sx5TqvhENGoAnce hh1QTULR359HJ6gGcuN0SskxA7muDaDDFGDgJF1Npg20Du3IXq+U6BDtEjrhk+MXb2 b9DoU6DLQMLFiV8hy0ibWilv/ObAnUSoSGbkBzc1WfryeADwSSJE/q1gY4T8KXymFE Hr9PAmCkod5eazis8d7uPIcdpl/bCvoGaajOA7/mFbpUel2cqmhty+mD6lQGi/gfKB tyHhqTq6rp/9A== From: cel@kernel.org To: , Cc: Chuck Lever Subject: [RFC PATCH 1/4] xprtrdma: Remove temp allocation of rpcrdma_rep objects Date: Mon, 29 Apr 2024 11:25:39 -0400 Message-ID: <20240429152537.212958-7-cel@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240429152537.212958-6-cel@kernel.org> References: <20240429152537.212958-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8024; i=chuck.lever@oracle.com; h=from:subject; bh=WT9JnHTpv+JT0VdzN6n3GQ5CG6EnAUnEZK9i4sSYjWE=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmL7vxZkNcn7DWsYB4s+m0yfgWtAvqqIpxe8fa/ P4qZJEqE06JAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZi+78QAKCRAzarMzb2Z/ l1qiEACZuqE/x5goHCMz7Z4V5sa/tpkJ5pE4k8uwYQheDI20aR/i8hE3nzF4RxH7oXUyTqau5n3 ClfMSfixhn82ROo6UaOT/e+ksmTfCqSCamohnWIm5/YmpyEIg5cIVKy1gW9m/sBzchI3J4W0ic8 YMj+DJSO+dBN206Bx2rs7F3iWlRRuPrlF6Z5YuoaOF3/NFSldeZhF+jG67kxJ2LB4qyokNuDnYD 0mW9KVxtfJejGVVEeST8/eQ6FVeDkZVZJoGhmnZXfYRH6tjU5eDY4KjQIOWjTX2cY2jQ10Ku4Fg rfpK///O4KUGLddckkjnMgZ/uJiO0vi/6sJ2OCzk6hy/zyT12Nfot3rUHB+TnkjmSHlG59xnWF4 VgsmXaZBXZZL4EI4o0WMICsgHAXSCZBhmaBLYELWoqHLOAndF1wM9GhZSICL2s9mzYFtP1sFbFO uIyf5IS/NanSarmWc/NIhA6jTpND/buEEVbpnkDaFqs1DoAfS0vtvP3BgaXjSk7lv9o3X7g98Z4 cEKs62zeYCDPn6S2nfUtpg2yi96Cnsc9ee15BGGUKcgId5XisElsitqui7/CVS8rfnIYkhfgaTv X23OK/6TwWKZrzMj2FEq5pwXzIA+hLas4sRF4eCFNzu062y/O83L6nr6ZSUBjolIVWMs2b/hg5W 1pjgRUMdNOroGaw== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever The original code was designed so that most calls to rpcrdma_rep_create() would occur on the NUMA node that the device preferred. There are a few cases where that's not possible, so those reps are marked as temporary. However, we have the device (and its preferred node) already in rpcrdma_rep_create(), so let's use that. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/rpc_rdma.c | 3 +- net/sunrpc/xprtrdma/verbs.c | 57 ++++++++++++++------------------- net/sunrpc/xprtrdma/xprt_rdma.h | 3 +- 3 files changed, 26 insertions(+), 37 deletions(-) diff --git a/net/sunrpc/xprtrdma/rpc_rdma.c b/net/sunrpc/xprtrdma/rpc_rdma.c index 190a4de239c8..1478c41c7e9d 100644 --- a/net/sunrpc/xprtrdma/rpc_rdma.c +++ b/net/sunrpc/xprtrdma/rpc_rdma.c @@ -1471,8 +1471,7 @@ void rpcrdma_reply_handler(struct rpcrdma_rep *rep) credits = 1; /* don't deadlock */ else if (credits > r_xprt->rx_ep->re_max_requests) credits = r_xprt->rx_ep->re_max_requests; - rpcrdma_post_recvs(r_xprt, credits + (buf->rb_bc_srv_max_requests << 1), - false); + rpcrdma_post_recvs(r_xprt, credits + (buf->rb_bc_srv_max_requests << 1)); if (buf->rb_credits != credits) rpcrdma_update_cwnd(r_xprt, credits); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 4f8d7efa469f..c6d9d94c28ba 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -69,13 +69,15 @@ static void rpcrdma_sendctx_put_locked(struct rpcrdma_xprt *r_xprt, struct rpcrdma_sendctx *sc); static int rpcrdma_reqs_setup(struct rpcrdma_xprt *r_xprt); static void rpcrdma_reqs_reset(struct rpcrdma_xprt *r_xprt); -static void rpcrdma_rep_destroy(struct rpcrdma_rep *rep); static void rpcrdma_reps_unmap(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_destroy(struct rpcrdma_xprt *r_xprt); static void rpcrdma_ep_get(struct rpcrdma_ep *ep); static int rpcrdma_ep_put(struct rpcrdma_ep *ep); static struct rpcrdma_regbuf * +rpcrdma_regbuf_alloc_node(size_t size, enum dma_data_direction direction, + int node); +static struct rpcrdma_regbuf * rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction); static void rpcrdma_regbuf_dma_unmap(struct rpcrdma_regbuf *rb); static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb); @@ -501,7 +503,7 @@ int rpcrdma_xprt_connect(struct rpcrdma_xprt *r_xprt) * outstanding Receives. */ rpcrdma_ep_get(ep); - rpcrdma_post_recvs(r_xprt, 1, true); + rpcrdma_post_recvs(r_xprt, 1); rc = rdma_connect(ep->re_id, &ep->re_remote_cma); if (rc) @@ -920,18 +922,20 @@ static void rpcrdma_reqs_reset(struct rpcrdma_xprt *r_xprt) } static noinline -struct rpcrdma_rep *rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, - bool temp) +struct rpcrdma_rep *rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt) { struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_ep *ep = r_xprt->rx_ep; + struct ib_device *device = ep->re_id->device; struct rpcrdma_rep *rep; rep = kzalloc(sizeof(*rep), XPRTRDMA_GFP_FLAGS); if (rep == NULL) goto out; - rep->rr_rdmabuf = rpcrdma_regbuf_alloc(r_xprt->rx_ep->re_inline_recv, - DMA_FROM_DEVICE); + rep->rr_rdmabuf = rpcrdma_regbuf_alloc_node(ep->re_inline_recv, + DMA_FROM_DEVICE, + ibdev_to_node(device)); if (!rep->rr_rdmabuf) goto out_free; @@ -946,7 +950,6 @@ struct rpcrdma_rep *rpcrdma_rep_create(struct rpcrdma_xprt *r_xprt, rep->rr_recv_wr.wr_cqe = &rep->rr_cqe; rep->rr_recv_wr.sg_list = &rep->rr_rdmabuf->rg_iov; rep->rr_recv_wr.num_sge = 1; - rep->rr_temp = temp; spin_lock(&buf->rb_lock); list_add(&rep->rr_all, &buf->rb_all_reps); @@ -965,17 +968,6 @@ static void rpcrdma_rep_free(struct rpcrdma_rep *rep) kfree(rep); } -static void rpcrdma_rep_destroy(struct rpcrdma_rep *rep) -{ - struct rpcrdma_buffer *buf = &rep->rr_rxprt->rx_buf; - - spin_lock(&buf->rb_lock); - list_del(&rep->rr_all); - spin_unlock(&buf->rb_lock); - - rpcrdma_rep_free(rep); -} - static struct rpcrdma_rep *rpcrdma_rep_get_locked(struct rpcrdma_buffer *buf) { struct llist_node *node; @@ -1007,10 +999,8 @@ static void rpcrdma_reps_unmap(struct rpcrdma_xprt *r_xprt) struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_rep *rep; - list_for_each_entry(rep, &buf->rb_all_reps, rr_all) { + list_for_each_entry(rep, &buf->rb_all_reps, rr_all) rpcrdma_regbuf_dma_unmap(rep->rr_rdmabuf); - rep->rr_temp = true; /* Mark this rep for destruction */ - } } static void rpcrdma_reps_destroy(struct rpcrdma_buffer *buf) @@ -1227,14 +1217,15 @@ void rpcrdma_buffer_put(struct rpcrdma_buffer *buffers, struct rpcrdma_req *req) * or Replies they may be registered externally via frwr_map. */ static struct rpcrdma_regbuf * -rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction) +rpcrdma_regbuf_alloc_node(size_t size, enum dma_data_direction direction, + int node) { struct rpcrdma_regbuf *rb; - rb = kmalloc(sizeof(*rb), XPRTRDMA_GFP_FLAGS); + rb = kmalloc_node(sizeof(*rb), XPRTRDMA_GFP_FLAGS, node); if (!rb) return NULL; - rb->rg_data = kmalloc(size, XPRTRDMA_GFP_FLAGS); + rb->rg_data = kmalloc_node(size, XPRTRDMA_GFP_FLAGS, node); if (!rb->rg_data) { kfree(rb); return NULL; @@ -1246,6 +1237,12 @@ rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction) return rb; } +static struct rpcrdma_regbuf * +rpcrdma_regbuf_alloc(size_t size, enum dma_data_direction direction) +{ + return rpcrdma_regbuf_alloc_node(size, direction, NUMA_NO_NODE); +} + /** * rpcrdma_regbuf_realloc - re-allocate a SEND/RECV buffer * @rb: regbuf to reallocate @@ -1323,10 +1320,9 @@ static void rpcrdma_regbuf_free(struct rpcrdma_regbuf *rb) * rpcrdma_post_recvs - Refill the Receive Queue * @r_xprt: controlling transport instance * @needed: current credit grant - * @temp: mark Receive buffers to be deleted after one use * */ -void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed, bool temp) +void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed) { struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_ep *ep = r_xprt->rx_ep; @@ -1340,8 +1336,7 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed, bool temp) if (likely(ep->re_receive_count > needed)) goto out; needed -= ep->re_receive_count; - if (!temp) - needed += RPCRDMA_MAX_RECV_BATCH; + needed += RPCRDMA_MAX_RECV_BATCH; if (atomic_inc_return(&ep->re_receiving) > 1) goto out; @@ -1350,12 +1345,8 @@ void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed, bool temp) wr = NULL; while (needed) { rep = rpcrdma_rep_get_locked(buf); - if (rep && rep->rr_temp) { - rpcrdma_rep_destroy(rep); - continue; - } if (!rep) - rep = rpcrdma_rep_create(r_xprt, temp); + rep = rpcrdma_rep_create(r_xprt); if (!rep) break; if (!rpcrdma_regbuf_dma_map(r_xprt, rep->rr_rdmabuf)) { diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index da409450dfc0..08bda29ed953 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -198,7 +198,6 @@ struct rpcrdma_rep { __be32 rr_proc; int rr_wc_flags; u32 rr_inv_rkey; - bool rr_temp; struct rpcrdma_regbuf *rr_rdmabuf; struct rpcrdma_xprt *rr_rxprt; struct rpc_rqst *rr_rqst; @@ -466,7 +465,7 @@ void rpcrdma_flush_disconnect(struct rpcrdma_xprt *r_xprt, struct ib_wc *wc); int rpcrdma_xprt_connect(struct rpcrdma_xprt *r_xprt); void rpcrdma_xprt_disconnect(struct rpcrdma_xprt *r_xprt); -void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed, bool temp); +void rpcrdma_post_recvs(struct rpcrdma_xprt *r_xprt, int needed); /* * Buffer calls - xprtrdma/verbs.c From patchwork Mon Apr 29 15:25:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13647270 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A1EA582899; Mon, 29 Apr 2024 15:25:45 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404345; cv=none; b=PvmUQXYQrKHiNS/Cv2buld+zFqLboXCDd3DCs2DahaSgFqjuaIs05369lUOADLQXhCXD2wnIesuoxqrwsdCD7sexIQTPy6cHlkSiouEK2tmwyfiITGf8mYlaT8iuZLQKxlsGHCm1bgh247+DzcORLcTEVVGJZzkxqrn6YWPGSMk= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404345; c=relaxed/simple; bh=3chjRlWFA+CIIEOGgMd3ndkEz7Spm+a36WUbDUMnrAY=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=WKJLqfqTIQ/J08u13q8DXOBZQ34FrXDyDh8UeJ1lwm1qCsMRv1zXOZi5zRLJ0a3tZctLV9eUVEulYQFAw49gWst0MfiBNBqSNZbUw6H9/LSzh0bu80j71/fm58z+TFDYBelDc7C4BOrIxaa5K6AfZwO+iit19Z5xRR/XYq00G9s= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=FDi/an3A; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="FDi/an3A" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3380CC113CD; Mon, 29 Apr 2024 15:25:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714404345; bh=3chjRlWFA+CIIEOGgMd3ndkEz7Spm+a36WUbDUMnrAY=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=FDi/an3ADW45fwNGBk1mLLU5oEgUXzD6Nwar9hxj75+6V4BVK88/w8StPhm6gXgqW br1h3ti99140VWpKxFSptM4YJO1qXu+w23Am39qXBTjLZ5fXBZTFMlArHWzylvid6I eED2FinnNwXCcWVG6g8iP6k5gyLo4vmLj7XkLxhrWCjnWAxYBwJdATwQACE7ltu2Mr Ttl3SGuyMuX2sejRW5Q3SO7PkF7TJCEd/2GwJ8k3zbr6dJd1ozC6whUQ9PQ5K6GDGf 5DjpMSFqODTdQOegD0sCLSRk7/JpxKa6YzNwPyiz0yXu3oVVncpQPsYyg0ydz2l1xz w42AWhL6xLTbg== From: cel@kernel.org To: , Cc: Chuck Lever Subject: [RFC PATCH 2/4] xprtrdma: Clean up synopsis of frwr_mr_unmap() Date: Mon, 29 Apr 2024 11:25:40 -0400 Message-ID: <20240429152537.212958-8-cel@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240429152537.212958-6-cel@kernel.org> References: <20240429152537.212958-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=1342; i=chuck.lever@oracle.com; h=from:subject; bh=mrzVot53dktixzRfAgrCLvQUOjngZn1HCXDlSGimSQ4=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmL7vxRhWS9cWJKwy3Nm4QTXfiKvaSXSrkDsPpo 83hTBWlFryJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZi+78QAKCRAzarMzb2Z/ lwarEACxMF8Ans33wi8UMWdclPVAgQqib8TPr8ILTA+W9LUHVjUY5Pq4LhdDUMKzXsFu3t29CrK Kno+wH1MkTQhFoKt9TgEfRp8Rg5lBqwu3BM+clwfaB7N3dk7EJMDrTI5uZm9Kvcf0FjtOol1Tmo bNT0yXL8oMHFsUd18ZaNRV25IPOilz+oMfAKRUhWDaN9gwbWGdUL+/sp0LqZei1by/XRAtj99dt HJuaCk12gQqGLR/Fy5yJAV/n2zjIKoF2cNeFOdjjiBcQTkIiaZ8B7uv2yn5Qrm1yOk0p472tuJL a36PcHljwEBCwi/Dx39wjzARuP699qoO3IfLt0jKeHyKz1U4PxD+j3tzXbDsLOKnzN9w0NoIitd fPfSq8qFEcScvv/RLlIEMcmTCKD7Ff9Y0mE2BeqU0QhNaYzdMIXhS6s8xWtBAlR7EdVDykRN1nI OW0ICzTMb2kGsTM0gAIaecxeq9M7bfmr8gJxbce7tdm8/mGZHcYskKf/3iw034VhSGzki2+Rf6V a/NWsOBgfNDcq2+3wDqfgZz2+Wxf7t5rCGirpLU8uc+0YK3FtEt0Y9820MWLEbdbzI50CVg6eWn pUsF907I3WvYD1RJRTNcPp4w2R14nN/mnUnxI769cloQI3ZVB9znj+5qb7aQeky3KQCbsBlk4dO Sja8gido/RYMFMQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever Commit 7a03aeb66c41 ("xprtrdma: Micro-optimize MR DMA-unmapping") removed the last use of the @r_xprt parameter in this function, but neglected to remove the parameter itself. Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index ffbf99894970..6e508708d06d 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -54,7 +54,7 @@ static void frwr_cid_init(struct rpcrdma_ep *ep, cid->ci_completion_id = mr->mr_ibmr->res.id; } -static void frwr_mr_unmap(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr) +static void frwr_mr_unmap(struct rpcrdma_mr *mr) { if (mr->mr_device) { trace_xprtrdma_mr_unmap(mr); @@ -73,7 +73,7 @@ void frwr_mr_release(struct rpcrdma_mr *mr) { int rc; - frwr_mr_unmap(mr->mr_xprt, mr); + frwr_mr_unmap(mr); rc = ib_dereg_mr(mr->mr_ibmr); if (rc) @@ -84,7 +84,7 @@ void frwr_mr_release(struct rpcrdma_mr *mr) static void frwr_mr_put(struct rpcrdma_mr *mr) { - frwr_mr_unmap(mr->mr_xprt, mr); + frwr_mr_unmap(mr); /* The MR is returned to the req's MR free list instead * of to the xprt's MR free list. No spinlock is needed. From patchwork Mon Apr 29 15:25:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13647271 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 2460382D66; Mon, 29 Apr 2024 15:25:46 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404347; cv=none; b=UjoH/ZrQqT9tkd4GeO/lbdRengJV1kcFyf+Jx1OIXFT6LJg6b0sGTDY2BwaW1XcDZVBTIiJwW0GvoJVXlPmcFuIHT5Mcg+LS5CmLc/Mh1zc8iTmxjr0jy+alEPb4RnCBZFCs/gKwPuFwDzprbNTfNGLS33VGv8pGiISIV7+HWj8= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404347; c=relaxed/simple; bh=wb5J0O2tSSIZAlZWpMSKSpqG+aU95kjnpuduXitZFSM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=G82S7ScPkKDmJ512bG6NNKeABKnKalSshgVDxhvzzAMI+ktNv9vx/z3rkntCmHP2bAwwqTQWs8oud5SGkGsnqy+jhctMbpaDJwwJ8J/0rEo9QpX3OJILzWKwC/46B/NSFgKPd5aHqhRm29QKTMaN0saDDEEm5zRool8XFAu7imM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=ci3FB8rc; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="ci3FB8rc" Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6F294C4AF14; Mon, 29 Apr 2024 15:25:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714404346; bh=wb5J0O2tSSIZAlZWpMSKSpqG+aU95kjnpuduXitZFSM=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ci3FB8rcIrFqkXrgWbekAOWtOw6ebmuD0CmtR1MN0ZnI79hQHVs9mY7ZY6lB3gEkq HUuTgvo2Tx7JbshJhD06Y9qQIKz3uzcgEndNlzJcM6vj+ZMRA7XOth6EHovJ7ma6QG HpW5XGV/wuD9tjKBQ5vsocgBeYCJSa8JS08CZkJv40e+8ZyBM9nUGtQjRAmEvfBIOG AhIsrNKKARB+mVJuxGRpOFdkn+a1XI7DiE2xcJLZuDhrpjpcphVtuTuiegih8rICRo 2fWLqYND5hp3PVy/nViG7DjKxWqCiBgMmLq1mtMdx+Kd4ZSQT5m5gmj/ns3yJacVyj ePxm62v9Cassg== From: cel@kernel.org To: , Cc: Chuck Lever Subject: [RFC PATCH 3/4] xprtrdma: Delay releasing connection hardware resources Date: Mon, 29 Apr 2024 11:25:41 -0400 Message-ID: <20240429152537.212958-9-cel@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240429152537.212958-6-cel@kernel.org> References: <20240429152537.212958-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=8400; i=chuck.lever@oracle.com; h=from:subject; bh=OYkQ/NF3NTAWlR/nAswSjjfmIkUb8wLKSPlbH5UhdGE=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmL7vyOIJ2Wsp4jV5PQ2ghK0k+jBRjwRrVJRA9M RDafhOFfJ6JAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZi+78gAKCRAzarMzb2Z/ lzxqD/wMiRmygt9tXm2WggHHoMX/lWwDOwp4NPZ6QmutNiQNrDV22h/nXBl5kM75cBTr290QBb3 T6nE5eV+Ptex1Z7IHSL/IRDAkCpK+jgDf05TPOuzBQmItslHRTKTUfQJHOf+xehC5khpK3Naxp0 QZzKg4R8/pLyuL4d8dlUzdW4AVZBBnlZQonJZ7QOYWP2YgcUqrUZAmI9UCkm041LZ2gLuFhF7im 6nr1/lXCY2PctXe0m+HvruBi2kh/Hhh+yvthP2XuXX6BE46hDoG8dnEGkFkPvm82SVPmPdPmlSJ 4Yx7xCMMRX4i4DVFs58HptcCJZzVoZTAhUb3v/le+zRPCi4FDXVrvKHN/Wy3QEAafzcKvlsPSWj T7zmF5TehqwDgd/q+p0BHWIULmTC5RdXvWh/ygK12XZ6TrHTho82D7695M/ZlgL49EAZCatNysX FJGDBV4ndfSx69u/KYu7Ar6W6zD6bswCPugN5Ga5cmsIR56Fi3iLDnpF+YKHjDtLQ3V/r1qkMBM /b5ikyqZV6o0O/HvpL9CCfKOY+oB/HYOLmQiGOGDDNPMqNgAzlR6zCDwZpqpjvNUzOtuJdNmU3I rqHin2dtD/yIo8wTarpGLbEKXbqgApMTg00an2aVamMCVauMMoL2SD+ODNP8yfQojC0mDSQmhm2 ScaO0JwtK3YP3yQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever xprtiod_workqueue is a MEM_RECLAIM-enabled workqueue. However, the RDMA core API functions are not memory reclaim-safe. This was partially accomplished by commit 6b1eb3b22272 ("SUNRPC: Replace the use of the xprtiod WQ in rpcrdma"). This commit addressed the issue in the connect path, but not in the disconnect path. Thus sometimes a transport disconnect results in this splat: workqueue: WQ_MEM_RECLAIM xprtiod:xprt_autoclose [sunrpc] is flushing !WQ_MEM_RECLAIM events_highpri:rpcrdma_mr_refresh_worker [rpcrdma] WARNING: CPU: 1 PID: 20378 at kernel/workqueue.c:3728 check_flush_dependency+0x101/0x120 ? check_flush_dependency+0x101/0x120 ? report_bug+0x175/0x1a0 ? handle_bug+0x44/0x90 ? exc_invalid_op+0x1c/0x70 ? asm_exc_invalid_op+0x1f/0x30 ? __pfx_rpcrdma_mr_refresh_worker+0x10/0x10 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac] ? check_flush_dependency+0x101/0x120 __flush_work.isra.0+0x20a/0x290 __cancel_work_sync+0x129/0x1c0 cancel_work_sync+0x14/0x20 rpcrdma_xprt_disconnect+0x229/0x3f0 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac] xprt_rdma_close+0x16/0x40 [rpcrdma aefd3d1b298311368fa14fa93ae5fb3818c3aeac] xprt_autoclose+0x63/0x110 [sunrpc a04d701bce94b5a8fb541cafbe1a489d6b1ab5b3] process_one_work+0x19e/0x3f0 worker_thread+0x340/0x510 ? __pfx_worker_thread+0x10/0x10 kthread+0xf7/0x130 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x41/0x60 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 Create a context in which it is safe to manage resources that are not memory reclaim-safe that can be invoked during transport disconnect. Essentially this means that releasing an rpcrdma_ep is now done completely asynchronously. Subsequent patches will move the release of transport resources into this new context. Link: https://bugzilla.kernel.org/show_bug.cgi?id=218704 Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/transport.c | 20 +++++++++++++- net/sunrpc/xprtrdma/verbs.c | 46 ++++++++++++++++++++------------- net/sunrpc/xprtrdma/xprt_rdma.h | 5 +++- 3 files changed, 51 insertions(+), 20 deletions(-) diff --git a/net/sunrpc/xprtrdma/transport.c b/net/sunrpc/xprtrdma/transport.c index 29b0562d62e7..237d78c1ec54 100644 --- a/net/sunrpc/xprtrdma/transport.c +++ b/net/sunrpc/xprtrdma/transport.c @@ -761,8 +761,12 @@ static struct xprt_class xprt_rdma = { .netid = { "rdma", "rdma6", "" }, }; +struct workqueue_struct *rpcrdma_release_wq __read_mostly; + void xprt_rdma_cleanup(void) { + struct workqueue_struct *wq; + #if IS_ENABLED(CONFIG_SUNRPC_DEBUG) if (sunrpc_table_header) { unregister_sysctl_table(sunrpc_table_header); @@ -772,18 +776,32 @@ void xprt_rdma_cleanup(void) xprt_unregister_transport(&xprt_rdma); xprt_unregister_transport(&xprt_rdma_bc); + + wq = rpcrdma_release_wq; + rpcrdma_release_wq = NULL; + destroy_workqueue(wq); } int xprt_rdma_init(void) { + struct workqueue_struct *wq; int rc; + /* provision a WQ that is always unbound and !mem_reclaim */ + wq = alloc_workqueue("rpcrdma_release", WQ_UNBOUND, 0); + if (!wq) + return -ENOMEM; + rpcrdma_release_wq = wq; + rc = xprt_register_transport(&xprt_rdma); - if (rc) + if (rc) { + destroy_workqueue(wq); return rc; + } rc = xprt_register_transport(&xprt_rdma_bc); if (rc) { + destroy_workqueue(wq); xprt_unregister_transport(&xprt_rdma); return rc; } diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index c6d9d94c28ba..f1e4a28325fa 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -73,7 +73,7 @@ static void rpcrdma_reps_unmap(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_destroy(struct rpcrdma_xprt *r_xprt); static void rpcrdma_ep_get(struct rpcrdma_ep *ep); -static int rpcrdma_ep_put(struct rpcrdma_ep *ep); +static void rpcrdma_ep_put(struct rpcrdma_ep *ep); static struct rpcrdma_regbuf * rpcrdma_regbuf_alloc_node(size_t size, enum dma_data_direction direction, int node); @@ -234,15 +234,15 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) case RDMA_CM_EVENT_ROUTE_RESOLVED: ep->re_async_rc = 0; complete(&ep->re_done); - return 0; + break; case RDMA_CM_EVENT_ADDR_ERROR: ep->re_async_rc = -EPROTO; complete(&ep->re_done); - return 0; + break; case RDMA_CM_EVENT_ROUTE_ERROR: ep->re_async_rc = -ENETUNREACH; complete(&ep->re_done); - return 0; + break; case RDMA_CM_EVENT_DEVICE_REMOVAL: pr_info("rpcrdma: removing device %s for %pISpc\n", ep->re_id->device->name, sap); @@ -269,12 +269,13 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) ep->re_connect_status = -ENOTCONN; wake_connect_worker: wake_up_all(&ep->re_connect_wait); - return 0; + break; case RDMA_CM_EVENT_DISCONNECTED: ep->re_connect_status = -ECONNABORTED; disconnected: rpcrdma_force_disconnect(ep); - return rpcrdma_ep_put(ep); + rpcrdma_ep_put(ep); + fallthrough; default: break; } @@ -328,9 +329,13 @@ static struct rdma_cm_id *rpcrdma_create_id(struct rpcrdma_xprt *r_xprt, return ERR_PTR(rc); } -static void rpcrdma_ep_destroy(struct kref *kref) +/* Delayed release of a connection's hardware resources. Releasing + * RDMA hardware resources is done in a !MEM_RECLAIM context because + * the RDMA core API functions are generally not reclaim-safe. + */ +static void rpcrdma_ep_destroy(struct work_struct *work) { - struct rpcrdma_ep *ep = container_of(kref, struct rpcrdma_ep, re_kref); + struct rpcrdma_ep *ep = container_of(work, struct rpcrdma_ep, re_worker); if (ep->re_id->qp) { rdma_destroy_qp(ep->re_id); @@ -348,22 +353,30 @@ static void rpcrdma_ep_destroy(struct kref *kref) ib_dealloc_pd(ep->re_pd); ep->re_pd = NULL; + if (ep->re_id) + rdma_destroy_id(ep->re_id); + ep->re_id = NULL; + kfree(ep); module_put(THIS_MODULE); } +static void rpcrdma_ep_release(struct kref *kref) +{ + struct rpcrdma_ep *ep = container_of(kref, struct rpcrdma_ep, re_kref); + + INIT_WORK(&ep->re_worker, rpcrdma_ep_destroy); + queue_work(rpcrdma_release_wq, &ep->re_worker); +} + static noinline void rpcrdma_ep_get(struct rpcrdma_ep *ep) { kref_get(&ep->re_kref); } -/* Returns: - * %0 if @ep still has a positive kref count, or - * %1 if @ep was destroyed successfully. - */ -static noinline int rpcrdma_ep_put(struct rpcrdma_ep *ep) +static noinline void rpcrdma_ep_put(struct rpcrdma_ep *ep) { - return kref_put(&ep->re_kref, rpcrdma_ep_destroy); + kref_put(&ep->re_kref, rpcrdma_ep_release); } static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt) @@ -475,7 +488,6 @@ static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt) out_destroy: rpcrdma_ep_put(ep); - rdma_destroy_id(id); return rc; } @@ -566,10 +578,8 @@ void rpcrdma_xprt_disconnect(struct rpcrdma_xprt *r_xprt) rpcrdma_mrs_destroy(r_xprt); rpcrdma_sendctxs_destroy(r_xprt); - if (rpcrdma_ep_put(ep)) - rdma_destroy_id(id); - r_xprt->rx_ep = NULL; + rpcrdma_ep_put(ep); } /* Fixed-size circular FIFO queue. This implementation is wait-free and diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 08bda29ed953..048d2e329384 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -70,7 +70,6 @@ */ struct rpcrdma_mr; struct rpcrdma_ep { - struct kref re_kref; struct rdma_cm_id *re_id; struct ib_pd *re_pd; unsigned int re_max_rdma_segs; @@ -100,6 +99,9 @@ struct rpcrdma_ep { atomic_t re_completion_ids; char re_write_pad[XDR_UNIT]; + + struct kref re_kref; + struct work_struct re_worker; }; /* Pre-allocate extra Work Requests for handling reverse-direction @@ -583,6 +585,7 @@ void xprt_rdma_format_addresses(struct rpc_xprt *xprt, struct sockaddr *sap); void xprt_rdma_free_addresses(struct rpc_xprt *xprt); void xprt_rdma_close(struct rpc_xprt *xprt); void xprt_rdma_print_stats(struct rpc_xprt *xprt, struct seq_file *seq); +extern struct workqueue_struct *rpcrdma_release_wq; int xprt_rdma_init(void); void xprt_rdma_cleanup(void); From patchwork Mon Apr 29 15:25:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chuck Lever X-Patchwork-Id: 13647272 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 9031A127E21; Mon, 29 Apr 2024 15:25:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=10.30.226.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404348; cv=none; b=PsdMiG4uzOUB3qladiR423+BdaKKstC44Pa6EieIGXMjXtz3Rq6yjjRR84z2RHwbiBMFmK3qsbq8JrmaaCoVZhSDW+tDSEXn/VrI3rc02efmbDPxDUkyl+v++7A+JarBjVhZrM/9i4dufRqvuxiOQtrMzv4U8VRtbPTyZeX2iHI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1714404348; c=relaxed/simple; bh=CQQfWet9gRYvytRYkhPPwpkxKK1mvYgGDcAwgOS5qq8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lkdsNykc4mYT+n0rdCfX47psPaQtUz6gok68tl5xu5FSylT+xEoCl4xEkPGJWU0Z6941jrwy/J6cxg8SNeO/6LoHtUPPFZggo18U6OKburZLRSt4+4Z7N/7XCytaX+zY7Gn4zVSO1IUuKbz4ZCbO7oxAzolX7cXXsipWIORD5io= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b=kq8U/A47; arc=none smtp.client-ip=10.30.226.201 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel.org header.i=@kernel.org header.b="kq8U/A47" Received: by smtp.kernel.org (Postfix) with ESMTPSA id B5738C4AF14; Mon, 29 Apr 2024 15:25:47 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1714404348; bh=CQQfWet9gRYvytRYkhPPwpkxKK1mvYgGDcAwgOS5qq8=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=kq8U/A47lB/1NODC/Kh6VcY/ruMTF9Qv7513eb5dGgAr18xX/z2tqkCgOPQXFHhfx zGTCVFeNlsPhD65M/Vqlml6hR3OGTJJkAct15k9IKgtXX8sU7MmkkuanDflbnt8m+J hepoEqHOkGkhoA9yELEKxzftSMtLl/TYfWNzS/Tuj4qZwSX1/R0FEJXQYx7aFw1VRn nDtHh7+d9KRNrnwwkh8VNNfZTuaKw1ipyi5MFXOnd/l5MWVIsE4esP1rstdQXR5+XH mm2cU8vl4u5v19NxQEG9Lf9lDGDlKgVsveQZHEm6zfufPPPHX1uCQQc3VpYIWOvpsy XoMX/PokCIYSw== From: cel@kernel.org To: , Cc: Chuck Lever Subject: [RFC PATCH 4/4] xprtrdma: Move MRs to struct rpcrdma_ep Date: Mon, 29 Apr 2024 11:25:42 -0400 Message-ID: <20240429152537.212958-10-cel@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240429152537.212958-6-cel@kernel.org> References: <20240429152537.212958-6-cel@kernel.org> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Developer-Signature: v=1; a=openpgp-sha256; l=9678; i=chuck.lever@oracle.com; h=from:subject; bh=l6rX/ZC8L4NkMCKe75WVMTLvRYW9ylOWHZBIeAcPFs8=; b=owEBbQKS/ZANAwAIATNqszNvZn+XAcsmYgBmL7vy2WObc4SeGY0QS08ZGftCF6Z/O1ZwLJPYe 5AUr2Aqi9OJAjMEAAEIAB0WIQQosuWwEobfJDzyPv4zarMzb2Z/lwUCZi+78gAKCRAzarMzb2Z/ lw3dD/9MwY+wnUScEPuB+AiOwPYc+kKmXYJaiQnYr6eAUy0aSrLTxHUcOxjJDIf5HFoHNHHJR7w Qf+kCvjiTtyU0Escy6w8hC0KtZGciu9XdSVsYkAZxqNbExhjftv5I2Q9b6H1EkG2F0ObUjP2zXv DVneX39ysB/5li9PjCuVUDktvh+wNJzZPaHxYwiR+X4agVsqE4/huj3eu4UpCsbLtEXwTFnkqJ6 y2L4MVdKxfkuyrXCYQJjo7Mtcn9lz4TNGPu7M54oyyTjIndxFHkW9BghrzOtsUWYKs+EzmVWMk0 OSZKsRgtYPNlHe+gK68SYsjXr6Sof3woBo08CgIsv1cb+BnTPSR/osfRIOA4JwXtU5a+S6m2nei gFKfjY8Ljaody4Mwh/VDhhoZ8B4u1MJemrUT1UakxFP5Qb1iTgMTW8rSRmWPwYWGrwvi8qX6tCZ A1nS6AhAbHagz0/LgtgNIAGkI4A3MJSraqS/xDXNbgpZYSESHb6lfBJObz/vmxPPv50DgH/wlex 792ApRdoA5PtowzCD/Rrc178ofadFN2kht0dHYxxqO79NUH4nkA0Lf5xtD8Exn7zjfKgyGGQvQ5 VcKOCG9snPnvvT74+ehOj+QP/59L+78s81rlEcPPeWv7nqfD3cq+wDgbNgQTKeqrh1JUbzKRvWA eLBe0IGnB9AglFQ== X-Developer-Key: i=chuck.lever@oracle.com; a=openpgp; fpr=28B2E5B01286DF243CF23EFE336AB3336F667F97 From: Chuck Lever MRs are a connection-specific hardware resource, thus they should be anchored in the EP and released with the other hardware resources. Closes: Link: https://bugzilla.kernel.org/show_bug.cgi?id=218704 Signed-off-by: Chuck Lever --- net/sunrpc/xprtrdma/frwr_ops.c | 7 ++-- net/sunrpc/xprtrdma/verbs.c | 70 ++++++++++++++++----------------- net/sunrpc/xprtrdma/xprt_rdma.h | 13 +++--- 3 files changed, 45 insertions(+), 45 deletions(-) diff --git a/net/sunrpc/xprtrdma/frwr_ops.c b/net/sunrpc/xprtrdma/frwr_ops.c index 6e508708d06d..7e918753eec4 100644 --- a/net/sunrpc/xprtrdma/frwr_ops.c +++ b/net/sunrpc/xprtrdma/frwr_ops.c @@ -112,15 +112,14 @@ void frwr_reset(struct rpcrdma_req *req) /** * frwr_mr_init - Initialize one MR - * @r_xprt: controlling transport instance + * @ep: controlling transport instance * @mr: generic MR to prepare for FRWR * * Returns zero if successful. Otherwise a negative errno * is returned. */ -int frwr_mr_init(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr) +int frwr_mr_init(struct rpcrdma_ep *ep, struct rpcrdma_mr *mr) { - struct rpcrdma_ep *ep = r_xprt->rx_ep; unsigned int depth = ep->re_max_fr_depth; struct scatterlist *sg; struct ib_mr *frmr; @@ -134,7 +133,7 @@ int frwr_mr_init(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr) if (IS_ERR(frmr)) goto out_mr_err; - mr->mr_xprt = r_xprt; + mr->mr_ep = ep; mr->mr_ibmr = frmr; mr->mr_device = NULL; INIT_LIST_HEAD(&mr->mr_list); diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index f1e4a28325fa..2578d9e77056 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -71,7 +71,8 @@ static int rpcrdma_reqs_setup(struct rpcrdma_xprt *r_xprt); static void rpcrdma_reqs_reset(struct rpcrdma_xprt *r_xprt); static void rpcrdma_reps_unmap(struct rpcrdma_xprt *r_xprt); static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt); -static void rpcrdma_mrs_destroy(struct rpcrdma_xprt *r_xprt); +static void rpcrdma_mrs_destroy(struct rpcrdma_ep *ep); +static void rpcrdma_mr_refresh_worker(struct work_struct *work); static void rpcrdma_ep_get(struct rpcrdma_ep *ep); static void rpcrdma_ep_put(struct rpcrdma_ep *ep); static struct rpcrdma_regbuf * @@ -337,6 +338,8 @@ static void rpcrdma_ep_destroy(struct work_struct *work) { struct rpcrdma_ep *ep = container_of(work, struct rpcrdma_ep, re_worker); + rpcrdma_mrs_destroy(ep); + if (ep->re_id->qp) { rdma_destroy_qp(ep->re_id); ep->re_id->qp = NULL; @@ -393,6 +396,11 @@ static int rpcrdma_ep_create(struct rpcrdma_xprt *r_xprt) ep->re_xprt = &r_xprt->rx_xprt; kref_init(&ep->re_kref); + spin_lock_init(&ep->re_mr_lock); + INIT_WORK(&ep->re_refresh_worker, rpcrdma_mr_refresh_worker); + INIT_LIST_HEAD(&ep->re_mrs); + INIT_LIST_HEAD(&ep->re_all_mrs); + id = rpcrdma_create_id(r_xprt, ep); if (IS_ERR(id)) { kfree(ep); @@ -575,7 +583,6 @@ void rpcrdma_xprt_disconnect(struct rpcrdma_xprt *r_xprt) rpcrdma_xprt_drain(r_xprt); rpcrdma_reps_unmap(r_xprt); rpcrdma_reqs_reset(r_xprt); - rpcrdma_mrs_destroy(r_xprt); rpcrdma_sendctxs_destroy(r_xprt); r_xprt->rx_ep = NULL; @@ -749,7 +756,6 @@ static void rpcrdma_sendctx_put_locked(struct rpcrdma_xprt *r_xprt, static void rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt) { - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_ep *ep = r_xprt->rx_ep; struct ib_device *device = ep->re_id->device; unsigned int count; @@ -764,16 +770,16 @@ rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt) if (!mr) break; - rc = frwr_mr_init(r_xprt, mr); + rc = frwr_mr_init(ep, mr); if (rc) { kfree(mr); break; } - spin_lock(&buf->rb_lock); - rpcrdma_mr_push(mr, &buf->rb_mrs); - list_add(&mr->mr_all, &buf->rb_all_mrs); - spin_unlock(&buf->rb_lock); + spin_lock(&ep->re_mr_lock); + rpcrdma_mr_push(mr, &ep->re_mrs); + list_add(&mr->mr_all, &ep->re_all_mrs); + spin_unlock(&ep->re_mr_lock); } r_xprt->rx_stats.mrs_allocated += count; @@ -783,10 +789,11 @@ rpcrdma_mrs_create(struct rpcrdma_xprt *r_xprt) static void rpcrdma_mr_refresh_worker(struct work_struct *work) { - struct rpcrdma_buffer *buf = container_of(work, struct rpcrdma_buffer, - rb_refresh_worker); - struct rpcrdma_xprt *r_xprt = container_of(buf, struct rpcrdma_xprt, - rx_buf); + struct rpcrdma_ep *ep = container_of(work, struct rpcrdma_ep, + re_refresh_worker); + struct rpcrdma_xprt *r_xprt = container_of(ep->re_xprt, + struct rpcrdma_xprt, + rx_xprt); rpcrdma_mrs_create(r_xprt); xprt_write_space(&r_xprt->rx_xprt); @@ -799,7 +806,6 @@ rpcrdma_mr_refresh_worker(struct work_struct *work) */ void rpcrdma_mrs_refresh(struct rpcrdma_xprt *r_xprt) { - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_ep *ep = r_xprt->rx_ep; /* If there is no underlying connection, it's no use @@ -807,7 +813,7 @@ void rpcrdma_mrs_refresh(struct rpcrdma_xprt *r_xprt) */ if (ep->re_connect_status != 1) return; - queue_work(system_highpri_wq, &buf->rb_refresh_worker); + queue_work(system_highpri_wq, &ep->re_refresh_worker); } /** @@ -1044,9 +1050,6 @@ int rpcrdma_buffer_create(struct rpcrdma_xprt *r_xprt) buf->rb_bc_srv_max_requests = 0; spin_lock_init(&buf->rb_lock); - INIT_LIST_HEAD(&buf->rb_mrs); - INIT_LIST_HEAD(&buf->rb_all_mrs); - INIT_WORK(&buf->rb_refresh_worker, rpcrdma_mr_refresh_worker); INIT_LIST_HEAD(&buf->rb_send_bufs); INIT_LIST_HEAD(&buf->rb_allreqs); @@ -1085,11 +1088,11 @@ void rpcrdma_req_destroy(struct rpcrdma_req *req) list_del(&req->rl_all); while ((mr = rpcrdma_mr_pop(&req->rl_free_mrs))) { - struct rpcrdma_buffer *buf = &mr->mr_xprt->rx_buf; + struct rpcrdma_ep *ep = mr->mr_ep; - spin_lock(&buf->rb_lock); + spin_lock(&ep->re_mr_lock); list_del(&mr->mr_all); - spin_unlock(&buf->rb_lock); + spin_unlock(&ep->re_mr_lock); frwr_mr_release(mr); } @@ -1102,31 +1105,28 @@ void rpcrdma_req_destroy(struct rpcrdma_req *req) /** * rpcrdma_mrs_destroy - Release all of a transport's MRs - * @r_xprt: controlling transport instance + * @ep: controlling transport instance * - * Relies on caller holding the transport send lock to protect - * removing mr->mr_list from req->rl_free_mrs safely. */ -static void rpcrdma_mrs_destroy(struct rpcrdma_xprt *r_xprt) +static void rpcrdma_mrs_destroy(struct rpcrdma_ep *ep) { - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; struct rpcrdma_mr *mr; - cancel_work_sync(&buf->rb_refresh_worker); + cancel_work_sync(&ep->re_refresh_worker); - spin_lock(&buf->rb_lock); - while ((mr = list_first_entry_or_null(&buf->rb_all_mrs, + spin_lock(&ep->re_mr_lock); + while ((mr = list_first_entry_or_null(&ep->re_all_mrs, struct rpcrdma_mr, mr_all)) != NULL) { list_del(&mr->mr_list); list_del(&mr->mr_all); - spin_unlock(&buf->rb_lock); + spin_unlock(&ep->re_mr_lock); frwr_mr_release(mr); - spin_lock(&buf->rb_lock); + spin_lock(&ep->re_mr_lock); } - spin_unlock(&buf->rb_lock); + spin_unlock(&ep->re_mr_lock); } /** @@ -1162,12 +1162,12 @@ rpcrdma_buffer_destroy(struct rpcrdma_buffer *buf) struct rpcrdma_mr * rpcrdma_mr_get(struct rpcrdma_xprt *r_xprt) { - struct rpcrdma_buffer *buf = &r_xprt->rx_buf; + struct rpcrdma_ep *ep = r_xprt->rx_ep; struct rpcrdma_mr *mr; - spin_lock(&buf->rb_lock); - mr = rpcrdma_mr_pop(&buf->rb_mrs); - spin_unlock(&buf->rb_lock); + spin_lock(&ep->re_mr_lock); + mr = rpcrdma_mr_pop(&ep->re_mrs); + spin_unlock(&ep->re_mr_lock); return mr; } diff --git a/net/sunrpc/xprtrdma/xprt_rdma.h b/net/sunrpc/xprtrdma/xprt_rdma.h index 048d2e329384..ce703b6e3b86 100644 --- a/net/sunrpc/xprtrdma/xprt_rdma.h +++ b/net/sunrpc/xprtrdma/xprt_rdma.h @@ -96,6 +96,11 @@ struct rpcrdma_ep { unsigned int re_inline_send; /* negotiated */ unsigned int re_inline_recv; /* negotiated */ + spinlock_t re_mr_lock; + struct list_head re_mrs; + struct list_head re_all_mrs; + struct work_struct re_refresh_worker; + atomic_t re_completion_ids; char re_write_pad[XDR_UNIT]; @@ -253,7 +258,7 @@ struct rpcrdma_mr { struct ib_reg_wr mr_regwr; struct ib_send_wr mr_invwr; }; - struct rpcrdma_xprt *mr_xprt; + struct rpcrdma_ep *mr_ep; u32 mr_handle; u32 mr_length; u64 mr_offset; @@ -365,7 +370,6 @@ rpcrdma_mr_pop(struct list_head *list) struct rpcrdma_buffer { spinlock_t rb_lock; struct list_head rb_send_bufs; - struct list_head rb_mrs; unsigned long rb_sc_head; unsigned long rb_sc_tail; @@ -373,7 +377,6 @@ struct rpcrdma_buffer { struct rpcrdma_sendctx **rb_sc_ctxs; struct list_head rb_allreqs; - struct list_head rb_all_mrs; struct list_head rb_all_reps; struct llist_head rb_free_reps; @@ -383,8 +386,6 @@ struct rpcrdma_buffer { u32 rb_bc_srv_max_requests; u32 rb_bc_max_requests; - - struct work_struct rb_refresh_worker; }; /* @@ -533,7 +534,7 @@ rpcrdma_data_dir(bool writing) */ void frwr_reset(struct rpcrdma_req *req); int frwr_query_device(struct rpcrdma_ep *ep, const struct ib_device *device); -int frwr_mr_init(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr *mr); +int frwr_mr_init(struct rpcrdma_ep *ep, struct rpcrdma_mr *mr); void frwr_mr_release(struct rpcrdma_mr *mr); struct rpcrdma_mr_seg *frwr_map(struct rpcrdma_xprt *r_xprt, struct rpcrdma_mr_seg *seg,