From patchwork Tue Nov 28 18:28:07 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Potnuri Bharat Teja X-Patchwork-Id: 10080865 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 9BC2460353 for ; Tue, 28 Nov 2017 18:28:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9522229419 for ; Tue, 28 Nov 2017 18:28:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 89DDA2954D; Tue, 28 Nov 2017 18:28:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=2.0 tests=BAYES_00,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D8B5E29419 for ; Tue, 28 Nov 2017 18:28:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752283AbdK1S2j (ORCPT ); Tue, 28 Nov 2017 13:28:39 -0500 Received: from stargate.chelsio.com ([12.32.117.8]:46205 "EHLO stargate.chelsio.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751796AbdK1S2i (ORCPT ); Tue, 28 Nov 2017 13:28:38 -0500 Received: from localhost ([10.193.186.242]) by stargate.chelsio.com (8.13.8/8.13.8) with ESMTP id vASISOmg020712; Tue, 28 Nov 2017 10:28:25 -0800 From: Potnuri Bharat Teja To: nab@linux-iscsi.org, sagi@grimberg.me Cc: target-devel@vger.kernel.org, linux-rdma@vger.kernel.org, bharat@chelsio.com Subject: [PATCH RFC] iser-target: avoid reinitializing rdma contexts for isert commands Date: Tue, 28 Nov 2017 23:58:07 +0530 Message-Id: <1511893687-29298-1-git-send-email-bharat@chelsio.com> X-Mailer: git-send-email 2.5.3 Sender: target-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: target-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP isert commands that failed during isert_rdma_rw_ctx_post() are queued to Queue-Full(QF) queue and are scheduled to be reposted during queue-full queue processing. During this reposting, the rdma contexts are initialised again in isert_rdma_rw_ctx_post(), which is leaking significant memory. unreferenced object 0xffff8830201d9640 (size 64): comm "kworker/0:2", pid 195, jiffies 4295374851 (age 4528.436s) hex dump (first 32 bytes): 00 60 8b cb 2e 00 00 00 00 10 00 00 00 00 00 00 .`.............. 00 90 e3 cb 2e 00 00 00 00 10 00 00 00 00 00 00 ................ backtrace: [] kmemleak_alloc+0x4e/0xb0 [] __kmalloc+0x125/0x2b0 [] rdma_rw_ctx_init+0x15f/0x6f0 [ib_core] [] isert_rdma_rw_ctx_post+0xc4/0x3c0 [ib_isert] [] isert_put_datain+0x112/0x1c0 [ib_isert] [] lio_queue_data_in+0x2e/0x30 [iscsi_target_mod] [] target_qf_do_work+0x2b2/0x4b0 [target_core_mod] [] process_one_work+0x1db/0x5d0 [] worker_thread+0x4d/0x3e0 [] kthread+0x117/0x150 [] ret_from_fork+0x27/0x40 [] 0xffffffffffffffff Here is patch to use the older rdma contexts while reposting the isert commands intead of reinitialising them. Signed-off-by: Potnuri Bharat Teja Reviewed-by: Sagi Grimberg --- drivers/infiniband/ulp/isert/ib_isert.c | 7 +++++++ drivers/infiniband/ulp/isert/ib_isert.h | 1 + 2 files changed, 8 insertions(+) diff --git a/drivers/infiniband/ulp/isert/ib_isert.c b/drivers/infiniband/ulp/isert/ib_isert.c index 720dfb3a1ac2..fd55163801a3 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.c +++ b/drivers/infiniband/ulp/isert/ib_isert.c @@ -2123,6 +2123,9 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, u32 rkey, offset; int ret; + if (cmd->ctx_init_done) + goto rdma_ctx_post; + if (dir == DMA_FROM_DEVICE) { addr = cmd->write_va; rkey = cmd->write_stag; @@ -2150,11 +2153,15 @@ isert_rdma_rw_ctx_post(struct isert_cmd *cmd, struct isert_conn *conn, se_cmd->t_data_sg, se_cmd->t_data_nents, offset, addr, rkey, dir); } + if (ret < 0) { isert_err("Cmd: %p failed to prepare RDMA res\n", cmd); return ret; } + cmd->ctx_init_done = true; + +rdma_ctx_post: ret = rdma_rw_ctx_post(&cmd->rw, conn->qp, port_num, cqe, chain_wr); if (ret < 0) isert_err("Cmd: %p failed to post RDMA res\n", cmd); diff --git a/drivers/infiniband/ulp/isert/ib_isert.h b/drivers/infiniband/ulp/isert/ib_isert.h index d6fd248320ae..3b296bac4f60 100644 --- a/drivers/infiniband/ulp/isert/ib_isert.h +++ b/drivers/infiniband/ulp/isert/ib_isert.h @@ -126,6 +126,7 @@ struct isert_cmd { struct rdma_rw_ctx rw; struct work_struct comp_work; struct scatterlist sg; + bool ctx_init_done; }; static inline struct isert_cmd *tx_desc_to_cmd(struct iser_tx_desc *desc)