From patchwork Tue Jun 29 06:53:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12349361 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45283C11F67 for ; Tue, 29 Jun 2021 06:53:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2644761DC6 for ; Tue, 29 Jun 2021 06:53:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232116AbhF2Gzw (ORCPT ); Tue, 29 Jun 2021 02:55:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232041AbhF2Gzw (ORCPT ); Tue, 29 Jun 2021 02:55:52 -0400 Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77377C061574 for ; Mon, 28 Jun 2021 23:53:25 -0700 (PDT) Received: by mail-ed1-x531.google.com with SMTP id i24so29765288edx.4 for ; Mon, 28 Jun 2021 23:53:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KuD5r33pAc3SDjyN/D8crf5rIUn7LGhRSs0oQPcggq0=; b=JerIfoIPhUfQp7emday/7dd5CXZE3mTVDy8LKvKuHZDzftzbnbuesXkjRKVJW2MF1b rhbs92phUBuXWaxrQMSBBnJjkyxUuHqa35gDdTiprhLkfFzYzmkA/ZQfDAG1cWm75zRt jmj9MYlQ8UeLKLI9Q/VSqTjWeiy9ZZXJT7VepAMLWX9zDhngUbEkf2xLVC0l8qVR1/6A 8KQPoNMwEVPvVB/7V33/1BSF0w2HZGLXM1WugYE2z776nIbviPiCsdEmjKo+YerKyHg+ NlhOwR2mx3kTB1F1uWLWqRGHU64EqO7UICKmegY8n57bFBeI6/bJStYLV+09y2kavoby lAAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KuD5r33pAc3SDjyN/D8crf5rIUn7LGhRSs0oQPcggq0=; b=Pj6K1QcWyKHV0h6k67/6rVmoyDr9oBx5QQwIl8FrFpCXyZlRo2tJHmoxVoB1jLVWa0 5uQMJT/A0aByYXqHizoxN9HayBt8htJAI/rsKKVMoaFRiLMn0hXcW7iHDQqsnHCai0K9 +O+GuSRWVlKYa8W8+yFoTJ/xb5GW+e0pyKM6+/Y2GtrHKMZiNUFq9iy59iaYph3biajZ w6RTGgTvfyM5QfMyeE+hPP7S/ao2SmUy2Sojf38o/9MBAxcwgl21b7cJC2/EQ/AgGt2n UmfYL0tG0eyt8Lnezhph+bsTrUc4fn1g2GLcm/sOzg4WQ93TaFmj27XdCsVyeRD2YCGO 57Bw== X-Gm-Message-State: AOAM530ZJ1MP2SZ+cz4U1zxfwNDvJD0RSEdeS5CepHoYdd1JA7wQ607Y LGPpq+E43nK466R3i1XisiFv4OTpTVIL8g== X-Google-Smtp-Source: ABdhPJyPAjOkOepu6cq6HT3bRBNWSrbDJfAL8XgShwfihZeXcwFiyLcrNWSUHZuMVlA4YDc0sgQeaQ== X-Received: by 2002:aa7:c997:: with SMTP id c23mr38319812edt.42.1624949603951; Mon, 28 Jun 2021 23:53:23 -0700 (PDT) Received: from jwang-Latitude-5491.fritz.box ([2001:16b8:49c9:3e00:293f:8e14:7de3:8980]) by smtp.gmail.com with ESMTPSA id t27sm7717853eje.86.2021.06.28.23.53.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 23:53:23 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, axboe@kernel.dk, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCH for-next 1/6] RDMA/rtrs: Add error messages for failed operations. Date: Tue, 29 Jun 2021 08:53:16 +0200 Message-Id: <20210629065321.12600-2-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210629065321.12600-1-jinpu.wang@ionos.com> References: <20210629065321.12600-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org It could help debugging in case of error happens. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 61919ebd92b2..870b21f551a4 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -316,6 +316,7 @@ void rtrs_send_hb_ack(struct rtrs_sess *sess) err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, 0, NULL); if (err) { + rtrs_err(sess, "send HB ACK failed, errno: %d\n", err); sess->hb_err_handler(usr_con); return; } @@ -333,6 +334,7 @@ static void hb_work(struct work_struct *work) usr_con = sess->con[0]; if (sess->hb_missed_cnt > sess->hb_missed_max) { + rtrs_err(sess, "HB missed max reached.\n"); sess->hb_err_handler(usr_con); return; } @@ -348,6 +350,7 @@ static void hb_work(struct work_struct *work) err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, 0, NULL); if (err) { + rtrs_err(sess, "HB send failed, errno: %d\n", err); sess->hb_err_handler(usr_con); return; } From patchwork Tue Jun 29 06:53:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12349363 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0C76C11F68 for ; Tue, 29 Jun 2021 06:53:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8DAC161DC8 for ; Tue, 29 Jun 2021 06:53:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232126AbhF2Gzx (ORCPT ); Tue, 29 Jun 2021 02:55:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232041AbhF2Gzx (ORCPT ); Tue, 29 Jun 2021 02:55:53 -0400 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3EF93C061574 for ; Mon, 28 Jun 2021 23:53:26 -0700 (PDT) Received: by mail-ej1-x62b.google.com with SMTP id c17so8557619ejk.13 for ; Mon, 28 Jun 2021 23:53:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=I6cH6Z7wdQumqLb7ZmTrwNtl31imikISuzvWrMPXUmQ=; b=Kjb1gPpV5sGLqfz2Y2Yi8Flz7HIQb8Rh3cBXpHUu7UJyaoBPD6hjy+T7fXxP7W0xkq fG6CsZuIluCZcp6H9O7vnvoXesY7lqJPfqo4f1tDvrNnfS8/T3qo/NFq/bnj8HbQg9nz XlnklYiUMjCIh5G8DK3HOsHJVst5o/Q/NV2WJNRahVPm83hNGRdBcDQwAo/1scp27wTe 9G68ec0tts/CYjtm8HhbBpeP5MGmSObdoloby6MdN/lukOcmkdTveSUFVc9Xi3Ez5cc6 oUEUHfBtRb0GgRUzeXL6XAMUIdQJ5Hg0c9qocxh9eO8sMm/a6L0eWFnz2I/Bmi4YTaCq Mj6g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=I6cH6Z7wdQumqLb7ZmTrwNtl31imikISuzvWrMPXUmQ=; b=IK0LS57JVT+ppHII5mgdM6AaDckjtH1HDBzmRdboHnXxF6+t8+ygpzyTprRhP+VJlt iqvZRbH77qsyPl4tcUi3U0tdEQr1u4tw/SCvXU34lHsV9ky+9L0aaV8qPrGzXcbhqSm1 MW5z841fQEGErNgWfiYG15BMhzC/1hetuOUrozEcFr3BdHAAk5a2E/NIvmoo3qc6Hl4M PRh5V797BjS5giFv/Ui9QJaVJqsgfAOTdZS75W6EhQtXxAmbLFl7e7E3LHu68n4umvKY X32rT1x0fSx2RS7uHgV9VyAF2DgY+zsCLKqBXhV9YtkhLhF1NwB7mB8oi+BXgq+GFCB2 2IHw== X-Gm-Message-State: AOAM532dvFaBmKYS0WN5Nlbcd6sttxjlkYQUvZWxZPXKG+Y9t4SLDuLR 5ILbOQQA0a3/344mN/RTvQMEFp+3Pp5kHw== X-Google-Smtp-Source: ABdhPJwz+i2uPISsAYydGt0Dn/LZzC1lKeCLEdhLzRH71jCjIAc5gCfPgpceqfwC6Ndzq5PcePx4sQ== X-Received: by 2002:a17:906:c2ca:: with SMTP id ch10mr28355575ejb.305.1624949604710; Mon, 28 Jun 2021 23:53:24 -0700 (PDT) Received: from jwang-Latitude-5491.fritz.box ([2001:16b8:49c9:3e00:293f:8e14:7de3:8980]) by smtp.gmail.com with ESMTPSA id t27sm7717853eje.86.2021.06.28.23.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 23:53:24 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, axboe@kernel.dk, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCH for-next 2/6] RDMA/rtrs: move wr_cnt from rtrs_srv_con to rtrs_con Date: Tue, 29 Jun 2021 08:53:17 +0200 Message-Id: <20210629065321.12600-3-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210629065321.12600-1-jinpu.wang@ionos.com> References: <20210629065321.12600-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org We need to track also the wr used for heatbeat. This is a preparation for that, will be used in later patch. The io_cnt in rtrs_clt is removed, use wr_cnt instead. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 7 ++++--- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 - drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 6 +++--- drivers/infiniband/ulp/rtrs/rtrs-srv.h | 1 - 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index f2c40e50f25e..5cb00ea08919 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -478,7 +478,7 @@ static int rtrs_post_send_rdma(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->io_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -1043,7 +1043,7 @@ static int rtrs_post_rdma_write_sg(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->io_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -1601,7 +1601,8 @@ static int create_con(struct rtrs_clt_sess *sess, unsigned int cid) con->cpu = (cid ? cid - 1 : 0) % nr_cpu_ids; con->c.cid = cid; con->c.sess = &sess->s; - atomic_set(&con->io_cnt, 0); + /* Align with srv, init as 1 */ + atomic_set(&con->c.wr_cnt, 1); mutex_init(&con->con_mutex); sess->s.con[cid] = &con->c; diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.h b/drivers/infiniband/ulp/rtrs/rtrs-clt.h index e276a2dfcf7c..3c3ff094588c 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.h @@ -74,7 +74,6 @@ struct rtrs_clt_con { u32 queue_num; unsigned int cpu; struct mutex con_mutex; - atomic_t io_cnt; int cm_err; }; diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index 36f184a3b676..a44a4fb1b515 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -96,6 +96,7 @@ struct rtrs_con { struct rdma_cm_id *cm_id; unsigned int cid; int nr_cqe; + atomic_t wr_cnt; }; struct rtrs_sess { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 3df290086169..31b846ca0c5e 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -269,7 +269,7 @@ static int rdma_write_sg(struct rtrs_srv_op *id) * From time to time we have to post signaled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&id->con->wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&id->con->c.wr_cnt) % srv->queue_depth) ? 0 : IB_SEND_SIGNALED; if (need_inval) { @@ -396,7 +396,7 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&con->wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&con->c.wr_cnt) % srv->queue_depth) ? 0 : IB_SEND_SIGNALED; imm = rtrs_to_io_rsp_imm(id->msg_id, errno, need_inval); imm_wr.wr.next = NULL; @@ -1648,7 +1648,7 @@ static int create_con(struct rtrs_srv_sess *sess, con->c.cm_id = cm_id; con->c.sess = &sess->s; con->c.cid = cid; - atomic_set(&con->wr_cnt, 1); + atomic_set(&con->c.wr_cnt, 1); wr_limit = sess->s.dev->ib_dev->attrs.max_qp_wr; if (con->c.cid == 0) { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h index f8da2e3f0bda..6785c3b6363e 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h @@ -42,7 +42,6 @@ struct rtrs_srv_stats { struct rtrs_srv_con { struct rtrs_con c; - atomic_t wr_cnt; atomic_t sq_wr_avail; struct list_head rsp_wr_wait_list; spinlock_t rsp_wr_wait_lock; From patchwork Tue Jun 29 06:53:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12349365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CFEBC11F66 for ; Tue, 29 Jun 2021 06:53:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DC8EC61DC9 for ; Tue, 29 Jun 2021 06:53:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232154AbhF2Gzy (ORCPT ); Tue, 29 Jun 2021 02:55:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56812 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232041AbhF2Gzy (ORCPT ); Tue, 29 Jun 2021 02:55:54 -0400 Received: from mail-ed1-x534.google.com (mail-ed1-x534.google.com [IPv6:2a00:1450:4864:20::534]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1F6A5C061574 for ; Mon, 28 Jun 2021 23:53:27 -0700 (PDT) Received: by mail-ed1-x534.google.com with SMTP id q14so29753435eds.5 for ; Mon, 28 Jun 2021 23:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aBRxRRZoYMjSvEWvj5G/ZD0nol7Dpgz8ughADCODhMk=; b=Y+2UQHrUGXMnqgdF4RNCs2mYH2YATNgLhYiKLBS40c7v96+x7JgkpCFvydMteJ98pM cekAdNgPoNlfxXSigdl3T+np/DKzlm0qek2lzMyFXGwp6Phgzg9SUgz1lEIQtf5b9A9t +fcyePSVeX76iJM5OSLK86VZGtuY/Ti48aZyvZOYaa0yi58mEUkYQNASfc/FXzdSNi2W ZAUjoOF4VfY9BGzJPKZUAifB0Vn9GaFTfOrKaLNnvNBOXrgb/D1Q9vDA35ECtshfhQQJ 6msp2s9p/scxJzWVp63zL1dly198DpMkyPcphuswRrbNStmicxu2kdLxQMEdc/d6X6qB LaGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aBRxRRZoYMjSvEWvj5G/ZD0nol7Dpgz8ughADCODhMk=; b=WBYoaPGwGPvhjYMv81Cxk5PXmoRfwnvaoaT2UmLK42PhKufjXooT7legzznStu9KN8 e9NbKo44NWYoXEdo48RDBgDznmb8K7M5XrVyR8zxrB7Jf0K0/pWD5uQeYJJpHgWZLZTV Y5ye5tdP2g4NJzEo5hI0OpCqxNp4qoIueLWUehZ3Tge2E6zP0KcPmQmdIp65WQFR9H1w dptNe57r5mykekbtPhsAJFzg6iL26WeP8dCs26deUbTVdEmCB+NIlb918IDgNzi0ylKZ XjqAE6De4XICzvELY59PwBqTBEO4Bl4TGX1zd5w2r5vFtJYTLPGfJnCfPK+fcVCc27Vb +3rA== X-Gm-Message-State: AOAM531ut9hxuZSM+aqNUlBkH0c9hgPFV3N9bYC7Hs9txK4x+fHOoWkt zYacetNzjKywWONYJXVYlhQIXS2EXfK1BQ== X-Google-Smtp-Source: ABdhPJx1T6e4FO3KGqJ3iRuDhbkdNDjpBn5pWu50qWqB1pIz1J1YOWxw5N8SuPQhhW0ZZbMzKoB2VQ== X-Received: by 2002:a05:6402:b9e:: with SMTP id cf30mr38186559edb.47.1624949605512; Mon, 28 Jun 2021 23:53:25 -0700 (PDT) Received: from jwang-Latitude-5491.fritz.box ([2001:16b8:49c9:3e00:293f:8e14:7de3:8980]) by smtp.gmail.com with ESMTPSA id t27sm7717853eje.86.2021.06.28.23.53.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 23:53:25 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, axboe@kernel.dk, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCH for-next 3/6] RDMA/rtrs: Enable the same selective signal for heartbeat and IO Date: Tue, 29 Jun 2021 08:53:18 +0200 Message-Id: <20210629065321.12600-4-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210629065321.12600-1-jinpu.wang@ionos.com> References: <20210629065321.12600-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On idle session, because we do not do signal for heartbeat, it will overflow the send queue after sometime. To avoid that, we need to enable the signal for heartbeat. To do that, add a new member signal_interval in rtrs_path, which will set min of queue_depth and SERVICE_CON_QUEUE_DEPTH, and track it for both heartbeat and IO, so the sq queue full accounting is correct. Fixes: b38041d50add ("RDMA/rtrs: Do not signal for heatbeat") Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 7 +++++-- drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 11 ++++++----- drivers/infiniband/ulp/rtrs/rtrs.c | 7 ++++++- 4 files changed, 18 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 5cb00ea08919..f023676e05e4 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -478,7 +478,7 @@ static int rtrs_post_send_rdma(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->s.signal_interval ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -680,6 +680,7 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc) case IB_WC_RDMA_WRITE: /* * post_send() RDMA write completions of IO reqs (read/write) + * and hb. */ break; @@ -1043,7 +1044,7 @@ static int rtrs_post_rdma_write_sg(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->s.signal_interval ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -1849,6 +1850,8 @@ static int rtrs_rdma_conn_established(struct rtrs_clt_con *con, return -ENOMEM; } sess->queue_depth = queue_depth; + sess->s.signal_interval = min_not_zero(queue_depth, + (unsigned short) SERVICE_CON_QUEUE_DEPTH); sess->max_hdr_size = le32_to_cpu(msg->max_hdr_size); sess->max_io_size = le32_to_cpu(msg->max_io_size); sess->flags = le32_to_cpu(msg->flags); diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index a44a4fb1b515..b88a4944cb30 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -109,6 +109,7 @@ struct rtrs_sess { unsigned int con_num; unsigned int irq_con_num; unsigned int recon_cnt; + unsigned int signal_interval; struct rtrs_ib_dev *dev; int dev_ref; struct ib_cqe *hb_cqe; diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 31b846ca0c5e..44ed15f38896 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -201,7 +201,6 @@ static int rdma_write_sg(struct rtrs_srv_op *id) struct rtrs_srv_sess *sess = to_srv_sess(s); dma_addr_t dma_addr = sess->dma_addr[id->msg_id]; struct rtrs_srv_mr *srv_mr; - struct rtrs_srv *srv = sess->srv; struct ib_send_wr inv_wr; struct ib_rdma_wr imm_wr; struct ib_rdma_wr *wr = NULL; @@ -269,7 +268,7 @@ static int rdma_write_sg(struct rtrs_srv_op *id) * From time to time we have to post signaled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&id->con->c.wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&id->con->c.wr_cnt) % s->signal_interval) ? 0 : IB_SEND_SIGNALED; if (need_inval) { @@ -347,7 +346,6 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id, struct ib_send_wr inv_wr, *wr = NULL; struct ib_rdma_wr imm_wr; struct ib_reg_wr rwr; - struct rtrs_srv *srv = sess->srv; struct rtrs_srv_mr *srv_mr; bool need_inval = false; enum ib_send_flags flags; @@ -396,7 +394,7 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&con->c.wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&con->c.wr_cnt) % s->signal_interval) ? 0 : IB_SEND_SIGNALED; imm = rtrs_to_io_rsp_imm(id->msg_id, errno, need_inval); imm_wr.wr.next = NULL; @@ -1268,8 +1266,9 @@ static void rtrs_srv_rdma_done(struct ib_cq *cq, struct ib_wc *wc) case IB_WC_SEND: /* * post_send() RDMA write completions of IO reqs (read/write) + * and hb. */ - atomic_add(srv->queue_depth, &con->sq_wr_avail); + atomic_add(s->signal_interval, &con->sq_wr_avail); if (unlikely(!list_empty_careful(&con->rsp_wr_wait_list))) rtrs_rdma_process_wr_wait_list(con); @@ -1659,6 +1658,8 @@ static int create_con(struct rtrs_srv_sess *sess, max_send_wr = min_t(int, wr_limit, SERVICE_CON_QUEUE_DEPTH * 2 + 2); max_recv_wr = max_send_wr; + s->signal_interval = min_not_zero(srv->queue_depth, + (size_t)SERVICE_CON_QUEUE_DEPTH); } else { /* when always_invlaidate enalbed, we need linv+rinv+mr+imm */ if (always_invalidate) diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 870b21f551a4..7f2dfb9d11fc 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -187,10 +187,15 @@ int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, struct ib_send_wr *head) { struct ib_rdma_wr wr; + struct rtrs_sess *sess = con->sess; + enum ib_send_flags sflags; + + sflags = (atomic_inc_return(&con->wr_cnt) % sess->signal_interval) ? + 0 : IB_SEND_SIGNALED; wr = (struct ib_rdma_wr) { .wr.wr_cqe = cqe, - .wr.send_flags = flags, + .wr.send_flags = sflags, .wr.opcode = IB_WR_RDMA_WRITE_WITH_IMM, .wr.ex.imm_data = cpu_to_be32(imm_data), }; From patchwork Tue Jun 29 06:53:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12349367 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70133C11F67 for ; Tue, 29 Jun 2021 06:53:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5B3BC61DC8 for ; Tue, 29 Jun 2021 06:53:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232164AbhF2Gzz (ORCPT ); Tue, 29 Jun 2021 02:55:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232142AbhF2Gzy (ORCPT ); Tue, 29 Jun 2021 02:55:54 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C886BC061760 for ; Mon, 28 Jun 2021 23:53:27 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id hc16so34350548ejc.12 for ; Mon, 28 Jun 2021 23:53:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BLAc5qb7iSWM1BvhJGNoOPseoj8qwLBemACiNj21/oI=; b=LssMiIYHe4O5W+znX2LPhO6vOGOVykovUo9JT9aX+CBIyNuAMLtNLBEmy7VCs9oj+W quax65wQCf0oxQknUZ73oNHhcfwLzUV4XER7ihhiQYTUv1gWYtyoJA3W63Dyfg2gvNwj BGu9bjKG/xvJu48Y5uZNSMminyjyUm+2YqchdUxdEqby9XcQX322eeR13n7YQAForKqH JSnlJCtABb+djLG6ovr6y42riyvHtnAJ+Jz6zoyePqf4ZBh3T68Kmwm4BoUly0Om+doD nX3vR9QPKahTefktWnjMFJYfMTIDLI6Tkj7luD2YqJ00TbD1uxItLcNKEX/hrNrw5Lz2 e/EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BLAc5qb7iSWM1BvhJGNoOPseoj8qwLBemACiNj21/oI=; b=EN1geggNLU4YXtrxfNX5h0HXWFLnGlNOXgJ1f1yOxToNHWRALnDAMBKHKL1thbzLT6 TpCO0BhiKCN0jd7HOWxSPf1mM6lmOSWHM5/ZW+Uug1cRt1dTfksqR9Gqz7uPRnoltUg+ z59/rjwn4nQuNpPuzUATVvznDsDptqSVv3/mjADRrpL9K0VZW3+HGTvQoYp8XGiR76/H 0Dx2E03L94coi2qrqA7DZinka3Cah+oDscdN2IwjhBxGBdr5DaoCX1IrvP0b1GsrLnuv S5j/xfrOUu7/BxXzlJUYyyZEar3ucvDRJffoledaVu4CLvVmpaEbA61rXTkqj6vS+bKs bSXQ== X-Gm-Message-State: AOAM530fc5ZTRFuwdoRps2e+rTWcZO8Ze11cHEgTMWocvrVm8APZ2/6h VzYt2q/aIV26J8tSdbdJeef7Y5Vn9DFFVA== X-Google-Smtp-Source: ABdhPJxtmaiI/82sspg6H8B9jz5X1ArPilPeMXZcR4ezEW3PlBvY0lkGNSNUehoWG5YK/KIqxBhwFw== X-Received: by 2002:a17:907:d9f:: with SMTP id go31mr25813632ejc.165.1624949606351; Mon, 28 Jun 2021 23:53:26 -0700 (PDT) Received: from jwang-Latitude-5491.fritz.box ([2001:16b8:49c9:3e00:293f:8e14:7de3:8980]) by smtp.gmail.com with ESMTPSA id t27sm7717853eje.86.2021.06.28.23.53.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 23:53:26 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, axboe@kernel.dk, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCH for-next 4/6] RDMA/rtrs: Make rtrs_post_rdma_write_imm_empty static Date: Tue, 29 Jun 2021 08:53:19 +0200 Message-Id: <20210629065321.12600-5-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210629065321.12600-1-jinpu.wang@ionos.com> References: <20210629065321.12600-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org It's only used in rtrs.c, so no need to export. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-pri.h | 3 --- drivers/infiniband/ulp/rtrs/rtrs.c | 9 +++++---- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index b88a4944cb30..76581ebaed1d 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -311,9 +311,6 @@ int rtrs_iu_post_rdma_write_imm(struct rtrs_con *con, struct rtrs_iu *iu, struct ib_send_wr *tail); int rtrs_post_recv_empty(struct rtrs_con *con, struct ib_cqe *cqe); -int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, - u32 imm_data, enum ib_send_flags flags, - struct ib_send_wr *head); int rtrs_cq_qp_create(struct rtrs_sess *sess, struct rtrs_con *con, u32 max_send_sge, int cq_vector, int nr_cqe, diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 7f2dfb9d11fc..528d6a57c9b6 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -182,9 +182,11 @@ int rtrs_iu_post_rdma_write_imm(struct rtrs_con *con, struct rtrs_iu *iu, } EXPORT_SYMBOL_GPL(rtrs_iu_post_rdma_write_imm); -int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, - u32 imm_data, enum ib_send_flags flags, - struct ib_send_wr *head) +static int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, + struct ib_cqe *cqe, + u32 imm_data, + enum ib_send_flags flags, + struct ib_send_wr *head) { struct ib_rdma_wr wr; struct rtrs_sess *sess = con->sess; @@ -202,7 +204,6 @@ int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, return rtrs_post_send(con->qp, head, &wr.wr, NULL); } -EXPORT_SYMBOL_GPL(rtrs_post_rdma_write_imm_empty); static void qp_event_handler(struct ib_event *ev, void *ctx) { From patchwork Tue Jun 29 06:53:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12349369 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 216F4C11F69 for ; Tue, 29 Jun 2021 06:53:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 05D3661DC8 for ; Tue, 29 Jun 2021 06:53:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232041AbhF2Gzz (ORCPT ); Tue, 29 Jun 2021 02:55:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232134AbhF2Gzz (ORCPT ); Tue, 29 Jun 2021 02:55:55 -0400 Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 82377C061574 for ; Mon, 28 Jun 2021 23:53:28 -0700 (PDT) Received: by mail-ed1-x529.google.com with SMTP id w13so23816252edc.0 for ; Mon, 28 Jun 2021 23:53:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bMOlkJ/7n+lh5liKE0v3uj39M6w6vAHGDdB9+X7SX2U=; b=fSEY8lw0rCYXbUM8yWhyRTlfoo2cL+f5ex/IGqeKle9Z51seaGoHz65Zx1IkTusK55 Ni18/26dACzlEBHWfeGDYECaX906RL5UMbSvCS4daLT7T7HLQaMs611TOXhai8MF1J5a aaTmIDCNB16MiRaxbtjDMk58YAzFIktij/1Pndw+WYGi4HtZNsYA6tpvHZIM+u9vXv8R tZYbOgQg+xsUM6h31al5yFQaY76FdlnXbIYKxXqnjGiOuqD9Ew0wwT2VWIhFvkvmQXRa bB35Erlm8RVi+iQ4c4PD2Pz6l3PKPEWnrAI5XDxYpdo50RIGtVlUzDG6nW46oprBhJC7 5nyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bMOlkJ/7n+lh5liKE0v3uj39M6w6vAHGDdB9+X7SX2U=; b=cM7Cos2e1DzyWPZjTcU0+UE9CXMc0O9xiIgxJosPKAI4Yg9/HIoQQlNCsNjWUFMdWJ HYr3VUyi7CILzzDN0nWRK6ld/ez7ijy+5WB/WMPkYZKnZCOqYVrcWsH2tFx0Tf3B1wKk EsgVuuOHSbg0/dM6Vghk1lE4ScAI90vZSRkMPsNFeg/nltoL7SrTDfvHZbRpMf2v3x3o CMESSlP+YEXqK6Pi57v7cxiZ66JynTBG50sbcozAskVN6nbwpH6J8mksDd3F/aIB7k2d oBqa0byBDeEtM3O9a68STXSVB9V9lWfEfmn867gYvvxqArMDjSgD8BvdHSv0bnl/VJJI DYQw== X-Gm-Message-State: AOAM5317WSrNj5BBH8mzWlfryjBK2cIsEXJWmUGdRwoj1bH7YTSRd2vU tDudV1kBUb29R4U/p41A2o5OFFSe4wrOLw== X-Google-Smtp-Source: ABdhPJxvvbG1Ha2lrH0Tx2aas1O+nqqfECaam48jichM+oeLpMZQ5f6JeYNM4Zp0ViWEbjedMPdyMA== X-Received: by 2002:a05:6402:290b:: with SMTP id ee11mr37676155edb.325.1624949607068; Mon, 28 Jun 2021 23:53:27 -0700 (PDT) Received: from jwang-Latitude-5491.fritz.box ([2001:16b8:49c9:3e00:293f:8e14:7de3:8980]) by smtp.gmail.com with ESMTPSA id t27sm7717853eje.86.2021.06.28.23.53.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 23:53:26 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, axboe@kernel.dk, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCH for-next 5/6] RDMA/rtrs: Remove unused flags parameter Date: Tue, 29 Jun 2021 08:53:20 +0200 Message-Id: <20210629065321.12600-6-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210629065321.12600-1-jinpu.wang@ionos.com> References: <20210629065321.12600-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org flags is not used, so remove it from rtrs_post_rdma_write_imm_empty. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 528d6a57c9b6..b56dc5b82db0 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -185,7 +185,6 @@ EXPORT_SYMBOL_GPL(rtrs_iu_post_rdma_write_imm); static int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, u32 imm_data, - enum ib_send_flags flags, struct ib_send_wr *head) { struct ib_rdma_wr wr; @@ -320,7 +319,7 @@ void rtrs_send_hb_ack(struct rtrs_sess *sess) imm = rtrs_to_imm(RTRS_HB_ACK_IMM, 0); err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, - 0, NULL); + NULL); if (err) { rtrs_err(sess, "send HB ACK failed, errno: %d\n", err); sess->hb_err_handler(usr_con); @@ -354,7 +353,7 @@ static void hb_work(struct work_struct *work) imm = rtrs_to_imm(RTRS_HB_MSG_IMM, 0); err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, - 0, NULL); + NULL); if (err) { rtrs_err(sess, "HB send failed, errno: %d\n", err); sess->hb_err_handler(usr_con); From patchwork Tue Jun 29 06:53:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12349371 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DF5D3C11F6A for ; Tue, 29 Jun 2021 06:53:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C821A61DC6 for ; Tue, 29 Jun 2021 06:53:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232161AbhF2Gz4 (ORCPT ); Tue, 29 Jun 2021 02:55:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232134AbhF2Gz4 (ORCPT ); Tue, 29 Jun 2021 02:55:56 -0400 Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D79EC061574 for ; Mon, 28 Jun 2021 23:53:29 -0700 (PDT) Received: by mail-ej1-x631.google.com with SMTP id c17so8557859ejk.13 for ; Mon, 28 Jun 2021 23:53:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VInWHDB2M54SAf2pi/ohFQ/fjdS46jXhiC35qkGExsU=; b=JuvnF+YlcFip2sz1zpc31VjAzrQimerZtxnnP9rhmn+nVbmgopd2yNwiFTiOtub/d2 1QGTx+MWlYbz5IejntMrd++RXea7SHr7VATdnbsuMJKmBZ3JuQjUYzZOSSlbqKnG5atz gMIn9UnUzoKtWmVZqp2Gz67QHiaEND4Og3SPg58HP4QJe15FQAF7XTxYEueQEQmacium T95Gy8zbMQ5cHZvmW/ZRIY9v5mdw0nEdmn2TnTVBbZD/D0GDQIzAeAwPKxxkjjyiXGym O/s8KnvfRh3WY9/lCkvO9wSTgCz5pbHEN1z6llWUK+AxojHTAReMY0xVkdGOJ9O/7A09 HkDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VInWHDB2M54SAf2pi/ohFQ/fjdS46jXhiC35qkGExsU=; b=IFw2AUEY1Xe9twAFxqFIL0U0EVzTcChzfvPEylKPWOqHdwicigzpVqPg/R9zoKCeQw 4viNgb1m7v/gnAYOWKff7La5hcBnsHYrjx9SxKxsgKyPvJ7wUMEHm9lG5VOvbzQX1mJH bjU3DspwTpgyOPst/Un6UZyKH+ka2IQLY5FXd6f3PQX1Qlrlii71hC4HyDSYx6M6wBG4 Pbta4rAjGShAsJDiXb88NH4eV1O1M07FdAoOQtJxRIaQD9aSVkGOwKykLXTNpET3zput GyB6jnGY2o1bVdWHc6khrI1AlPt2Ar+tROo5e14ZR+NjwZxjLW1tLUwD42omVPXZNGZr uAJw== X-Gm-Message-State: AOAM5302iY5XY8R373rE93ycAhrtJR8w2XLLPP59EsKQMOjXGBOOAcAf rOyoiginj8c9g+WyhQfABVtmlY+uJT2n1A== X-Google-Smtp-Source: ABdhPJzvotEUD2R120BLxtl+E0mWcAx/3nMuBXZvmpVEHPjLmHkmNI11a0/KryaUj6jCBFk7CLmFRQ== X-Received: by 2002:a17:906:2306:: with SMTP id l6mr27996542eja.362.1624949607768; Mon, 28 Jun 2021 23:53:27 -0700 (PDT) Received: from jwang-Latitude-5491.fritz.box ([2001:16b8:49c9:3e00:293f:8e14:7de3:8980]) by smtp.gmail.com with ESMTPSA id t27sm7717853eje.86.2021.06.28.23.53.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 23:53:27 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, axboe@kernel.dk, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCH for-next 6/6] RDMA/rtrs: Move sq_wr_avail to rtrs_con Date: Tue, 29 Jun 2021 08:53:21 +0200 Message-Id: <20210629065321.12600-7-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210629065321.12600-1-jinpu.wang@ionos.com> References: <20210629065321.12600-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In order to account HB for sq_wr_avail properly, move sq_wr_avail from rtrs_srv_con to rtrs_con. Although rtrs-clt do not care sq_wr_avail, but still init it to max_send_wr. Fixes: b38041d50add ("RDMA/rtrs: Do not signal for heatbeat") Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 1 + drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 8 ++++---- drivers/infiniband/ulp/rtrs/rtrs-srv.h | 1 - drivers/infiniband/ulp/rtrs/rtrs.c | 1 + 5 files changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index f023676e05e4..ece3205531b8 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -1680,6 +1680,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con) sess->queue_depth * 3 + 1); max_send_sge = 2; } + atomic_set(&con->c.sq_wr_avail, max_send_wr); cq_num = max_send_wr + max_recv_wr; /* alloc iu to recv new rkey reply when server reports flags set */ if (sess->flags & RTRS_MSG_NEW_RKEY_F || con->c.cid == 0) { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index 76581ebaed1d..d12ddfa50747 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -97,6 +97,7 @@ struct rtrs_con { unsigned int cid; int nr_cqe; atomic_t wr_cnt; + atomic_t sq_wr_avail; }; struct rtrs_sess { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 44ed15f38896..cd9a4ccf4c28 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -507,11 +507,11 @@ bool rtrs_srv_resp_rdma(struct rtrs_srv_op *id, int status) ib_update_fast_reg_key(mr->mr, ib_inc_rkey(mr->mr->rkey)); } if (unlikely(atomic_sub_return(1, - &con->sq_wr_avail) < 0)) { + &con->c.sq_wr_avail) < 0)) { rtrs_err(s, "IB send queue full: sess=%s cid=%d\n", kobject_name(&sess->kobj), con->c.cid); - atomic_add(1, &con->sq_wr_avail); + atomic_add(1, &con->c.sq_wr_avail); spin_lock(&con->rsp_wr_wait_lock); list_add_tail(&id->wait_list, &con->rsp_wr_wait_list); spin_unlock(&con->rsp_wr_wait_lock); @@ -1268,7 +1268,7 @@ static void rtrs_srv_rdma_done(struct ib_cq *cq, struct ib_wc *wc) * post_send() RDMA write completions of IO reqs (read/write) * and hb. */ - atomic_add(s->signal_interval, &con->sq_wr_avail); + atomic_add(s->signal_interval, &con->c.sq_wr_avail); if (unlikely(!list_empty_careful(&con->rsp_wr_wait_list))) rtrs_rdma_process_wr_wait_list(con); @@ -1680,7 +1680,7 @@ static int create_con(struct rtrs_srv_sess *sess, */ } cq_num = max_send_wr + max_recv_wr; - atomic_set(&con->sq_wr_avail, max_send_wr); + atomic_set(&con->c.sq_wr_avail, max_send_wr); cq_vector = rtrs_srv_get_next_cq_vector(sess); /* TODO: SOFTIRQ can be faster, but be careful with softirq context */ diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h index 6785c3b6363e..e81774f5acd3 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h @@ -42,7 +42,6 @@ struct rtrs_srv_stats { struct rtrs_srv_con { struct rtrs_con c; - atomic_t sq_wr_avail; struct list_head rsp_wr_wait_list; spinlock_t rsp_wr_wait_lock; }; diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index b56dc5b82db0..ca542e477d38 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -191,6 +191,7 @@ static int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct rtrs_sess *sess = con->sess; enum ib_send_flags sflags; + atomic_dec_if_positive(&con->sq_wr_avail); sflags = (atomic_inc_return(&con->wr_cnt) % sess->signal_interval) ? 0 : IB_SEND_SIGNALED;