From patchwork Mon Jul 12 06:07:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12370349 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A154C07E9B for ; Mon, 12 Jul 2021 06:07:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1C73B61042 for ; Mon, 12 Jul 2021 06:07:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230419AbhGLGKn (ORCPT ); Mon, 12 Jul 2021 02:10:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230215AbhGLGKm (ORCPT ); Mon, 12 Jul 2021 02:10:42 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F70CC0613DD for ; Sun, 11 Jul 2021 23:07:54 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id i2-20020a05600c3542b02902058529ea07so10650454wmq.3 for ; Sun, 11 Jul 2021 23:07:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=KuD5r33pAc3SDjyN/D8crf5rIUn7LGhRSs0oQPcggq0=; b=GpIn/Cab0kMhkyTAg8Rts3nKeIAjqaYhOBDzCD7QarLzEgy4N6syl8TV7KdxqJoJ2M gQYUeEavY8HfuUNdasdj1TQ9JacNOX87x3/i+0EQcSbNtMQN65dtypAQI8IaoI9Y6uWx d5F5T8Ua8GFsyTH+YeI5V/gI9HXICEHMkAZ5xAwr8lLCCnC8EhHJzOhcxnfkXnAlTDKM abwpu4+ZFUO08nNz665hNsHlCTRbPcac4LBQE0ZkH9TIcfebU9390N4YiyBLMVF74tmY i5yjiRP1PAdKsxjBgFglGlyZla2V56vsaEu25yE6JCtdEFtdXNdBmEqQnOXIVd1FnZRO kfmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=KuD5r33pAc3SDjyN/D8crf5rIUn7LGhRSs0oQPcggq0=; b=iIhRMoPefEep2SKC2QDX3GqjzTxg4F1wS8oY4NyjK/rxA5iKRjbNOMfZ7bjfkUBHYc bkUORd6Wvs79pe2PC+w6xcwv2L1VwxwLIIIoUZLGpzx7Q814eWDBza3TtzdLJ1nWVwGk 6CpokslaMpWd00lNCxA/UpTLhHP7jK5D21wY48c0HOp7SJ2rckZ3nnBOk5ryRBfToePw VRjmSHKrCNzX3qDo6BEK0zUQUw3kaxB+azw77ch7DAFXom22hvkDO9P+mE7ot4bADjzf 2w89i2bfDXc9JjJ62iCZNcZlXhiGLXrETtP0GRBbivZxxVPYPs476jnCdPpgcc/m2Tgw TWSw== X-Gm-Message-State: AOAM532hARzMBiyjoehc1PoCZgOmDx0wyVyxQ27XwTqzLUttkL2r8809 b+RG+70TuaNmpd+OY1QkideYS0xENNNUgg== X-Google-Smtp-Source: ABdhPJz3D570pvvsqIcjnBl+NmUwUyHNOnb78y4jQNT7FBBohyNEUzuTG/lrA6AJbBCxbe7yY63kxg== X-Received: by 2002:a1c:143:: with SMTP id 64mr14646772wmb.187.1626070072685; Sun, 11 Jul 2021 23:07:52 -0700 (PDT) Received: from jwang-Latitude-5491.fkb.profitbricks.net ([2001:16b8:49bc:8600:895c:e529:e1b8:7823]) by smtp.gmail.com with ESMTPSA id s17sm13344245wrv.2.2021.07.11.23.07.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Jul 2021 23:07:52 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCHv2 for-rc 1/6] RDMA/rtrs: Add error messages for failed operations. Date: Mon, 12 Jul 2021 08:07:45 +0200 Message-Id: <20210712060750.16494-2-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210712060750.16494-1-jinpu.wang@ionos.com> References: <20210712060750.16494-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org It could help debugging in case of error happens. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 61919ebd92b2..870b21f551a4 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -316,6 +316,7 @@ void rtrs_send_hb_ack(struct rtrs_sess *sess) err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, 0, NULL); if (err) { + rtrs_err(sess, "send HB ACK failed, errno: %d\n", err); sess->hb_err_handler(usr_con); return; } @@ -333,6 +334,7 @@ static void hb_work(struct work_struct *work) usr_con = sess->con[0]; if (sess->hb_missed_cnt > sess->hb_missed_max) { + rtrs_err(sess, "HB missed max reached.\n"); sess->hb_err_handler(usr_con); return; } @@ -348,6 +350,7 @@ static void hb_work(struct work_struct *work) err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, 0, NULL); if (err) { + rtrs_err(sess, "HB send failed, errno: %d\n", err); sess->hb_err_handler(usr_con); return; } From patchwork Mon Jul 12 06:07:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12370351 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58668C07E9C for ; Mon, 12 Jul 2021 06:07:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D7536101D for ; Mon, 12 Jul 2021 06:07:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230215AbhGLGKn (ORCPT ); Mon, 12 Jul 2021 02:10:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51436 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230397AbhGLGKm (ORCPT ); Mon, 12 Jul 2021 02:10:42 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2F81C0613E5 for ; Sun, 11 Jul 2021 23:07:54 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id w13so10572792wmc.3 for ; Sun, 11 Jul 2021 23:07:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=I6cH6Z7wdQumqLb7ZmTrwNtl31imikISuzvWrMPXUmQ=; b=BVerEYq8OQfjANGW0tgiVL7PVX/AbkaJn9pd3yYaYTw4T4OaVf9eJGJScVcvHzz72T jfklMiEgbeO13Rikk0T+qiAoCKHDyG5ZBEKKA5me/e2YzKoXYVygTWe97u5efabmIbaV b786C+3hRUMmq+sHrBXvwUBByIcuTusZDqgV9GnR5KdhOKkr18BzwTkyWOWZU2bO17hA /Gy3oQGgaIx6LUWzEmz9B/i406Pbfi8RtdZovGFxG6ZN9p48MukdxpGikdtFJZkALgev Wb0cMxWfZZnVMI34fZsAIn/MWFzyjHVDhw28ShG9+KELXKpE9XgfHhHdsn2zJcP5ypv+ 8c1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=I6cH6Z7wdQumqLb7ZmTrwNtl31imikISuzvWrMPXUmQ=; b=k7seDK6fJ38ZSLzBKbYXbHGdFx7ASMJjbnJ0NlUggVj5c41AZptT+hKJU4FhysgwMo /q6AM36KkQx0NQGGnmyTmgVH/bA3NziGBu73KS3Hfp564n3hmllPgDRSdJQTno4yOuJJ 1jUST3Pz/I7iV6vwqJJGJyNgSXHZkCv33rU+R54S4WpG25OQwn24gWml7blkJ492hFSo iK+M3u4lUPHfk0/QdkrgE18TyuJKKx4CvsC1nTbfDcdOi0D7Kv0kgpS8v+Xr69s5LsrT kUZPiwnrAOaxpxhQ5k8JosLqdGXdg7i3IRvqP5g1I+Ko9b28393rJurrT1KXHspCIqW6 CWYw== X-Gm-Message-State: AOAM5337HHiy79eOe7/gmbap8TTkbSEfnftKbxJ7skcpmWthRtCxmWki ECYRzTSC/yVDZWcj13PEIf7BrQ1yArzLQA== X-Google-Smtp-Source: ABdhPJwu3LKx9SbGsQhU9S2cJm33hDo7RWZT0ReSyaAeDPzKK0I/bK+5lmnsXNQx3n1Dm8QaypWOsw== X-Received: by 2002:a05:600c:2907:: with SMTP id i7mr12561746wmd.184.1626070073376; Sun, 11 Jul 2021 23:07:53 -0700 (PDT) Received: from jwang-Latitude-5491.fkb.profitbricks.net ([2001:16b8:49bc:8600:895c:e529:e1b8:7823]) by smtp.gmail.com with ESMTPSA id s17sm13344245wrv.2.2021.07.11.23.07.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Jul 2021 23:07:53 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCHv2 for-rc 2/6] RDMA/rtrs: move wr_cnt from rtrs_srv_con to rtrs_con Date: Mon, 12 Jul 2021 08:07:46 +0200 Message-Id: <20210712060750.16494-3-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210712060750.16494-1-jinpu.wang@ionos.com> References: <20210712060750.16494-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org We need to track also the wr used for heatbeat. This is a preparation for that, will be used in later patch. The io_cnt in rtrs_clt is removed, use wr_cnt instead. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 7 ++++--- drivers/infiniband/ulp/rtrs/rtrs-clt.h | 1 - drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 6 +++--- drivers/infiniband/ulp/rtrs/rtrs-srv.h | 1 - 5 files changed, 8 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index f2c40e50f25e..5cb00ea08919 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -478,7 +478,7 @@ static int rtrs_post_send_rdma(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->io_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -1043,7 +1043,7 @@ static int rtrs_post_rdma_write_sg(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->io_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -1601,7 +1601,8 @@ static int create_con(struct rtrs_clt_sess *sess, unsigned int cid) con->cpu = (cid ? cid - 1 : 0) % nr_cpu_ids; con->c.cid = cid; con->c.sess = &sess->s; - atomic_set(&con->io_cnt, 0); + /* Align with srv, init as 1 */ + atomic_set(&con->c.wr_cnt, 1); mutex_init(&con->con_mutex); sess->s.con[cid] = &con->c; diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.h b/drivers/infiniband/ulp/rtrs/rtrs-clt.h index e276a2dfcf7c..3c3ff094588c 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.h @@ -74,7 +74,6 @@ struct rtrs_clt_con { u32 queue_num; unsigned int cpu; struct mutex con_mutex; - atomic_t io_cnt; int cm_err; }; diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index 36f184a3b676..a44a4fb1b515 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -96,6 +96,7 @@ struct rtrs_con { struct rdma_cm_id *cm_id; unsigned int cid; int nr_cqe; + atomic_t wr_cnt; }; struct rtrs_sess { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 3df290086169..31b846ca0c5e 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -269,7 +269,7 @@ static int rdma_write_sg(struct rtrs_srv_op *id) * From time to time we have to post signaled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&id->con->wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&id->con->c.wr_cnt) % srv->queue_depth) ? 0 : IB_SEND_SIGNALED; if (need_inval) { @@ -396,7 +396,7 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&con->wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&con->c.wr_cnt) % srv->queue_depth) ? 0 : IB_SEND_SIGNALED; imm = rtrs_to_io_rsp_imm(id->msg_id, errno, need_inval); imm_wr.wr.next = NULL; @@ -1648,7 +1648,7 @@ static int create_con(struct rtrs_srv_sess *sess, con->c.cm_id = cm_id; con->c.sess = &sess->s; con->c.cid = cid; - atomic_set(&con->wr_cnt, 1); + atomic_set(&con->c.wr_cnt, 1); wr_limit = sess->s.dev->ib_dev->attrs.max_qp_wr; if (con->c.cid == 0) { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h index f8da2e3f0bda..6785c3b6363e 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h @@ -42,7 +42,6 @@ struct rtrs_srv_stats { struct rtrs_srv_con { struct rtrs_con c; - atomic_t wr_cnt; atomic_t sq_wr_avail; struct list_head rsp_wr_wait_list; spinlock_t rsp_wr_wait_lock; From patchwork Mon Jul 12 06:07:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12370353 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1474DC07E9E for ; Mon, 12 Jul 2021 06:07:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ECBBB60FF3 for ; Mon, 12 Jul 2021 06:07:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230455AbhGLGKo (ORCPT ); Mon, 12 Jul 2021 02:10:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51438 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230397AbhGLGKn (ORCPT ); Mon, 12 Jul 2021 02:10:43 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C0C85C0613DD for ; Sun, 11 Jul 2021 23:07:55 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id i2-20020a05600c3542b02902058529ea07so10650487wmq.3 for ; Sun, 11 Jul 2021 23:07:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aBRxRRZoYMjSvEWvj5G/ZD0nol7Dpgz8ughADCODhMk=; b=ZUqWPaW9IythuwVVvAvrnlBZdGDuAWGlpLU3d6h5gUoqCZJjzfnuHHo/PccDx3YcQW fB9Yk+VHMLlPCRMLAw5xB7PUizidtD6iLLRTFh294ITH26c6craEbLUKdbLD4Do/2HSQ bLTtthBS2MFFFZWavsXHutXiGAjmodlAc4MXZvXMIBYbHfaizcem0yszlkfOSx+tlslq z5O8l5z4GjsN5ubiJKwq3B6QWXlDESNxNNLzNHvR0fqitunle1LcBUPVDYX8hIvcSeQM ATMX8ncwBjUuCP7mys9+VD0Gb1A2nFma2YO92KmJfQNWJEuIUsE2cbGOgAtGwaVCFgMj XjNw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aBRxRRZoYMjSvEWvj5G/ZD0nol7Dpgz8ughADCODhMk=; b=CjL3QAoV9vOE9+KZUmt0HioEj4afmBKmuhzFqdj50DONUI5caZBfjQ+7tpXvZD58pG tSD/HfaU9ApbDuQfCsytaH+dItVz66mkGasgp3umOJiFxAXB6qB5zOmC/OEz7UdXZW49 If1kh8aLAnD984EbZvSZeXd89X/cddwkc1tvFsverzG6fDqT2QwQfvXj8YMTB1c7Kp0U NAapclhfJFmhszqA7EPlKADs4wxBoGpA9yTtENVLDYGNdNWGhQwNPjhOQYWWsLRZOjbA 0w0oETzYywBIYtp21WIjztpKFp+Yu/Si1M6DFnBgNGwpER4d25aApseq2TZo5R1pA9qN pdUQ== X-Gm-Message-State: AOAM530veyq+7OEUm2rlsptBmOUuCAkgu0FiPa09AhP3zO0bkqvlGXyn 1JKwoaLWuUyT2PZBo9GDiICcIEVUSuUhfg== X-Google-Smtp-Source: ABdhPJzsU99Lv4lqhLuqnxqSXmX+mtuRzCMDrS8XVh55IkWvpqhl3hhdW1F1ZkXJrzRVUcueYD3boA== X-Received: by 2002:a1c:7411:: with SMTP id p17mr12478916wmc.116.1626070074108; Sun, 11 Jul 2021 23:07:54 -0700 (PDT) Received: from jwang-Latitude-5491.fkb.profitbricks.net ([2001:16b8:49bc:8600:895c:e529:e1b8:7823]) by smtp.gmail.com with ESMTPSA id s17sm13344245wrv.2.2021.07.11.23.07.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Jul 2021 23:07:53 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCHv2 for-rc 3/6] RDMA/rtrs: Enable the same selective signal for heartbeat and IO Date: Mon, 12 Jul 2021 08:07:47 +0200 Message-Id: <20210712060750.16494-4-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210712060750.16494-1-jinpu.wang@ionos.com> References: <20210712060750.16494-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org On idle session, because we do not do signal for heartbeat, it will overflow the send queue after sometime. To avoid that, we need to enable the signal for heartbeat. To do that, add a new member signal_interval in rtrs_path, which will set min of queue_depth and SERVICE_CON_QUEUE_DEPTH, and track it for both heartbeat and IO, so the sq queue full accounting is correct. Fixes: b38041d50add ("RDMA/rtrs: Do not signal for heatbeat") Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 7 +++++-- drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 11 ++++++----- drivers/infiniband/ulp/rtrs/rtrs.c | 7 ++++++- 4 files changed, 18 insertions(+), 8 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index 5cb00ea08919..f023676e05e4 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -478,7 +478,7 @@ static int rtrs_post_send_rdma(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->s.signal_interval ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -680,6 +680,7 @@ static void rtrs_clt_rdma_done(struct ib_cq *cq, struct ib_wc *wc) case IB_WC_RDMA_WRITE: /* * post_send() RDMA write completions of IO reqs (read/write) + * and hb. */ break; @@ -1043,7 +1044,7 @@ static int rtrs_post_rdma_write_sg(struct rtrs_clt_con *con, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = atomic_inc_return(&con->c.wr_cnt) % sess->queue_depth ? + flags = atomic_inc_return(&con->c.wr_cnt) % sess->s.signal_interval ? 0 : IB_SEND_SIGNALED; ib_dma_sync_single_for_device(sess->s.dev->ib_dev, req->iu->dma_addr, @@ -1849,6 +1850,8 @@ static int rtrs_rdma_conn_established(struct rtrs_clt_con *con, return -ENOMEM; } sess->queue_depth = queue_depth; + sess->s.signal_interval = min_not_zero(queue_depth, + (unsigned short) SERVICE_CON_QUEUE_DEPTH); sess->max_hdr_size = le32_to_cpu(msg->max_hdr_size); sess->max_io_size = le32_to_cpu(msg->max_io_size); sess->flags = le32_to_cpu(msg->flags); diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index a44a4fb1b515..b88a4944cb30 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -109,6 +109,7 @@ struct rtrs_sess { unsigned int con_num; unsigned int irq_con_num; unsigned int recon_cnt; + unsigned int signal_interval; struct rtrs_ib_dev *dev; int dev_ref; struct ib_cqe *hb_cqe; diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 31b846ca0c5e..44ed15f38896 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -201,7 +201,6 @@ static int rdma_write_sg(struct rtrs_srv_op *id) struct rtrs_srv_sess *sess = to_srv_sess(s); dma_addr_t dma_addr = sess->dma_addr[id->msg_id]; struct rtrs_srv_mr *srv_mr; - struct rtrs_srv *srv = sess->srv; struct ib_send_wr inv_wr; struct ib_rdma_wr imm_wr; struct ib_rdma_wr *wr = NULL; @@ -269,7 +268,7 @@ static int rdma_write_sg(struct rtrs_srv_op *id) * From time to time we have to post signaled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&id->con->c.wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&id->con->c.wr_cnt) % s->signal_interval) ? 0 : IB_SEND_SIGNALED; if (need_inval) { @@ -347,7 +346,6 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id, struct ib_send_wr inv_wr, *wr = NULL; struct ib_rdma_wr imm_wr; struct ib_reg_wr rwr; - struct rtrs_srv *srv = sess->srv; struct rtrs_srv_mr *srv_mr; bool need_inval = false; enum ib_send_flags flags; @@ -396,7 +394,7 @@ static int send_io_resp_imm(struct rtrs_srv_con *con, struct rtrs_srv_op *id, * From time to time we have to post signalled sends, * or send queue will fill up and only QP reset can help. */ - flags = (atomic_inc_return(&con->c.wr_cnt) % srv->queue_depth) ? + flags = (atomic_inc_return(&con->c.wr_cnt) % s->signal_interval) ? 0 : IB_SEND_SIGNALED; imm = rtrs_to_io_rsp_imm(id->msg_id, errno, need_inval); imm_wr.wr.next = NULL; @@ -1268,8 +1266,9 @@ static void rtrs_srv_rdma_done(struct ib_cq *cq, struct ib_wc *wc) case IB_WC_SEND: /* * post_send() RDMA write completions of IO reqs (read/write) + * and hb. */ - atomic_add(srv->queue_depth, &con->sq_wr_avail); + atomic_add(s->signal_interval, &con->sq_wr_avail); if (unlikely(!list_empty_careful(&con->rsp_wr_wait_list))) rtrs_rdma_process_wr_wait_list(con); @@ -1659,6 +1658,8 @@ static int create_con(struct rtrs_srv_sess *sess, max_send_wr = min_t(int, wr_limit, SERVICE_CON_QUEUE_DEPTH * 2 + 2); max_recv_wr = max_send_wr; + s->signal_interval = min_not_zero(srv->queue_depth, + (size_t)SERVICE_CON_QUEUE_DEPTH); } else { /* when always_invlaidate enalbed, we need linv+rinv+mr+imm */ if (always_invalidate) diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 870b21f551a4..7f2dfb9d11fc 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -187,10 +187,15 @@ int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, struct ib_send_wr *head) { struct ib_rdma_wr wr; + struct rtrs_sess *sess = con->sess; + enum ib_send_flags sflags; + + sflags = (atomic_inc_return(&con->wr_cnt) % sess->signal_interval) ? + 0 : IB_SEND_SIGNALED; wr = (struct ib_rdma_wr) { .wr.wr_cqe = cqe, - .wr.send_flags = flags, + .wr.send_flags = sflags, .wr.opcode = IB_WR_RDMA_WRITE_WITH_IMM, .wr.ex.imm_data = cpu_to_be32(imm_data), }; From patchwork Mon Jul 12 06:07:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12370355 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88405C11F66 for ; Mon, 12 Jul 2021 06:07:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 732E26101D for ; Mon, 12 Jul 2021 06:07:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230503AbhGLGKo (ORCPT ); Mon, 12 Jul 2021 02:10:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51444 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230397AbhGLGKo (ORCPT ); Mon, 12 Jul 2021 02:10:44 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 666CCC0613DD for ; Sun, 11 Jul 2021 23:07:56 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id g8-20020a1c9d080000b02901f13dd1672aso9160454wme.0 for ; Sun, 11 Jul 2021 23:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BLAc5qb7iSWM1BvhJGNoOPseoj8qwLBemACiNj21/oI=; b=D64HFzDlvxfE9e8LaDhyF4tEWJfIlpVIo6mvrM/C7Wuu3LaleeIWjXWI1UQle7duMq knL+X5qwKELKMML9wLFv7B+TMlT4ye8vuD3vGMkwALqPQm11l0dfJAtIhWkgLbp7umEY ESH4gFBq8rUpS28zbCKQ4uV01bLjd6lkEbJS68U+1NC1s5ATcX/K7xVepz4BgVdZ+Qt8 YOPTLa+mnTrzdyt0p5RDrkHWJbsdUSruS8WPygodRAxZJ7Rajh62u9ya4Yu6HN2ntFq1 U4L651Ukpqv2EG20JhTgWBnq+c3Sfx/17HDDzjJRQNE8s9GMTfLLlnCfziMNN+Maihy0 Mx5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BLAc5qb7iSWM1BvhJGNoOPseoj8qwLBemACiNj21/oI=; b=W0LsWELRM5q+ADOOepuM4l4m5VXQjRsNuooOgVCD1GYenwCFuyZ1+xf0ZC+pS/AcQP WdmXdOkjjTciXIXVY3Acl/06dWWko4kZn+oDdrbMBqzNi7PE5rxl05chFAxvgcBbR4Im gboxu5mybbAX/e7kj5oknGRXkcQuijw6SHNmD3fUaf00KnTUheBjyevgA0R9inIuduSA ubVM0Eij6tOAJ+6czZOqZtMGcF+hJzKX77a1dTd2lBnudGwqzmJk/VWsZuMsIngysDFn f1Ft3MSUIV4x9MZy3LOKeXacoO4opa3K214l5AOupmi66vrYEsZakjgdXycqDSgwwu2M D23A== X-Gm-Message-State: AOAM531c1X5IKnBVTQrnOOIqTPZrXSpVy208KZ7vPNpv/RjwrzUXLO34 67Fq6WFoT3RIXpvD9vTKhvyFvTHBZ4r2gw== X-Google-Smtp-Source: ABdhPJwMqRpQbIw9BeAYgOG/cPdnO2MqH19HJyEJeSaApza+itTOiSZHwYgStbayGNZZc4N6MUEKuQ== X-Received: by 2002:a7b:ce98:: with SMTP id q24mr22785097wmj.73.1626070074977; Sun, 11 Jul 2021 23:07:54 -0700 (PDT) Received: from jwang-Latitude-5491.fkb.profitbricks.net ([2001:16b8:49bc:8600:895c:e529:e1b8:7823]) by smtp.gmail.com with ESMTPSA id s17sm13344245wrv.2.2021.07.11.23.07.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Jul 2021 23:07:54 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCHv2 for-rc 4/6] RDMA/rtrs: Make rtrs_post_rdma_write_imm_empty static Date: Mon, 12 Jul 2021 08:07:48 +0200 Message-Id: <20210712060750.16494-5-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210712060750.16494-1-jinpu.wang@ionos.com> References: <20210712060750.16494-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org It's only used in rtrs.c, so no need to export. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-pri.h | 3 --- drivers/infiniband/ulp/rtrs/rtrs.c | 9 +++++---- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index b88a4944cb30..76581ebaed1d 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -311,9 +311,6 @@ int rtrs_iu_post_rdma_write_imm(struct rtrs_con *con, struct rtrs_iu *iu, struct ib_send_wr *tail); int rtrs_post_recv_empty(struct rtrs_con *con, struct ib_cqe *cqe); -int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, - u32 imm_data, enum ib_send_flags flags, - struct ib_send_wr *head); int rtrs_cq_qp_create(struct rtrs_sess *sess, struct rtrs_con *con, u32 max_send_sge, int cq_vector, int nr_cqe, diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 7f2dfb9d11fc..528d6a57c9b6 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -182,9 +182,11 @@ int rtrs_iu_post_rdma_write_imm(struct rtrs_con *con, struct rtrs_iu *iu, } EXPORT_SYMBOL_GPL(rtrs_iu_post_rdma_write_imm); -int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, - u32 imm_data, enum ib_send_flags flags, - struct ib_send_wr *head) +static int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, + struct ib_cqe *cqe, + u32 imm_data, + enum ib_send_flags flags, + struct ib_send_wr *head) { struct ib_rdma_wr wr; struct rtrs_sess *sess = con->sess; @@ -202,7 +204,6 @@ int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, return rtrs_post_send(con->qp, head, &wr.wr, NULL); } -EXPORT_SYMBOL_GPL(rtrs_post_rdma_write_imm_empty); static void qp_event_handler(struct ib_event *ev, void *ctx) { From patchwork Mon Jul 12 06:07:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12370357 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2F33C07E99 for ; Mon, 12 Jul 2021 06:07:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA0E260FF3 for ; Mon, 12 Jul 2021 06:07:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230397AbhGLGKp (ORCPT ); Mon, 12 Jul 2021 02:10:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231349AbhGLGKp (ORCPT ); Mon, 12 Jul 2021 02:10:45 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37265C0613E5 for ; Sun, 11 Jul 2021 23:07:57 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id i2-20020a05600c3542b02902058529ea07so10650517wmq.3 for ; Sun, 11 Jul 2021 23:07:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bMOlkJ/7n+lh5liKE0v3uj39M6w6vAHGDdB9+X7SX2U=; b=bEcgNo0GP+KMGh5HmGbYzd8pACjrzaWt3oTyAxrzVCY5mYRKJk7X1tkCH5ADJOWUju E/de2G3758DxOgxef8Kcxw86Jqn58Vh6Ae6Uqzp2sAVKJ+1W/iwyUjfgGiVckdzBc72J V+tYC4kYIfzMv841uUoTfjPjMLd8QLdsJKEQP5YdarSFYaibKmEK1adJiuyXI64FtMi7 JhkM8yQcWjpFO8uDB83/6x2rcjKGrK2uDKcSeVUGrbtJKN8O++XJvCdIT+hG5pAsL+U3 7zj7r6EtCjpxVOJus3Nw/rvym8c0vquCw7KbX/mVnwAkJ4uIQ+sXh3uuFL2WgGxMFvl4 M5cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bMOlkJ/7n+lh5liKE0v3uj39M6w6vAHGDdB9+X7SX2U=; b=mJ8ilpmZ7D5vwXVSbdgdeVSUCH3b9ZGrPeKi/0pfyQ+oYLaJEWQSu0x3i16MQB/LBi dlrGLjKtSJenLRd/pHTe58PdGWFRxDZKB/uP0zMgW3WbOEbltTUC/wluqmnnZaEvmir8 GzS4b0o1mzjFCXyV5qd/P+BFjyyBLp+0o6ZgfgWSkKz32a+N/+Z5pVdEc7ChIIufSLS0 jDyElMWjI/KzzIs19NxC1GY2H4vbe8RK0TEsAkW/JPwleh76zMpXZACfTt2rgJ6j0TXa F5yWdYjm16jZMMXFcEV5i6yEYOX8FB04Wg3Hm3zMK8u7ZYnuiYgYBvp50w/BA3SDC6rz 84Sw== X-Gm-Message-State: AOAM53340zOL2BAZo9TdsjVT1kzC94EMgEhUGHeyig5p2zonAWfr2njr HBxEzCLObM95e84m1YiXWnXoHmGA+DVQEQ== X-Google-Smtp-Source: ABdhPJwXQqYDLDJ2jVXl/T22XS8215EcynfIq8YSDuFQw6leBJoovcTy97tiWx2utqEscgAXH94xbA== X-Received: by 2002:a7b:cc15:: with SMTP id f21mr12641255wmh.5.1626070075700; Sun, 11 Jul 2021 23:07:55 -0700 (PDT) Received: from jwang-Latitude-5491.fkb.profitbricks.net ([2001:16b8:49bc:8600:895c:e529:e1b8:7823]) by smtp.gmail.com with ESMTPSA id s17sm13344245wrv.2.2021.07.11.23.07.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Jul 2021 23:07:55 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCHv2 for-rc 5/6] RDMA/rtrs: Remove unused flags parameter Date: Mon, 12 Jul 2021 08:07:49 +0200 Message-Id: <20210712060750.16494-6-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210712060750.16494-1-jinpu.wang@ionos.com> References: <20210712060750.16494-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org flags is not used, so remove it from rtrs_post_rdma_write_imm_empty. Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index 528d6a57c9b6..b56dc5b82db0 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -185,7 +185,6 @@ EXPORT_SYMBOL_GPL(rtrs_iu_post_rdma_write_imm); static int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct ib_cqe *cqe, u32 imm_data, - enum ib_send_flags flags, struct ib_send_wr *head) { struct ib_rdma_wr wr; @@ -320,7 +319,7 @@ void rtrs_send_hb_ack(struct rtrs_sess *sess) imm = rtrs_to_imm(RTRS_HB_ACK_IMM, 0); err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, - 0, NULL); + NULL); if (err) { rtrs_err(sess, "send HB ACK failed, errno: %d\n", err); sess->hb_err_handler(usr_con); @@ -354,7 +353,7 @@ static void hb_work(struct work_struct *work) imm = rtrs_to_imm(RTRS_HB_MSG_IMM, 0); err = rtrs_post_rdma_write_imm_empty(usr_con, sess->hb_cqe, imm, - 0, NULL); + NULL); if (err) { rtrs_err(sess, "HB send failed, errno: %d\n", err); sess->hb_err_handler(usr_con); From patchwork Mon Jul 12 06:07:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jinpu Wang X-Patchwork-Id: 12370359 X-Patchwork-Delegate: jgg@ziepe.ca Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C879C07E9B for ; Mon, 12 Jul 2021 06:07:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 87E8960FF3 for ; Mon, 12 Jul 2021 06:07:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231349AbhGLGKq (ORCPT ); Mon, 12 Jul 2021 02:10:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51466 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231835AbhGLGKq (ORCPT ); Mon, 12 Jul 2021 02:10:46 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03439C0613E8 for ; Sun, 11 Jul 2021 23:07:58 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id h18-20020a05600c3512b029020e4ceb9588so13603308wmq.5 for ; Sun, 11 Jul 2021 23:07:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ionos.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VInWHDB2M54SAf2pi/ohFQ/fjdS46jXhiC35qkGExsU=; b=SCOCwk1BoDZevRcBQGb8oTeXcN4KFAhYFcUWZX1sCejSWdc/GrXTnTOeaxcfgsL/ao 6Ikau/SCURt0VlMTdg6jH5Roa41O7hFy1bkAOR2UkZCv5h0yWLHJaIfIExmxTAfXjIqr ZNMKyJyS+cz4oV8bzvhfJXo/tVykL/71HpSUioXdcpo/nvdND6IqI3tuEdYFFJmt+qZt mlA5qK9u9EcGhD74qCDepCdQ0Cg7fttgJ3GjADjL+vtKzGqhllYzNc5F2gNvB5N19ipE 4Wd2/dw2FIxgDQY0AXFID7PZNdyFMIgrt9mktPfED33YDU9h48CY/k/uNRU2kSsbVgiD fm/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VInWHDB2M54SAf2pi/ohFQ/fjdS46jXhiC35qkGExsU=; b=R6rovbkdQC7ndcR+44Quxmrt77nU3+MKZHj0hLv+tlcPW+oBPdMx/WBlkf9Rfg3K9g 047xOJ5FfiYoFAt7Tu4dT502YzGyxFbKY9nknlghcMmGwkj6n9hiDSfHlhB3P47VMW2E a/+cogo/pfDn3Ml1cZkkjxfOgE5mjgYj0Lkav9kIh8vaeIIc1bWW4ZKvX9VN/xcFBzOb hMWMOto5AAIO8yT/a+TCC5sidZg0d4vfvGwO2SXrmZFpNB/Kzusry0dDIptmdqbXGDzv /2K5IgTWpsosMdz7L4KA5mtJwExpbXQLbQdFdeacjphGJTcxvMcpeLfwTClrGFhcNCG+ 0v8g== X-Gm-Message-State: AOAM532XCYG2kb2fZYpA9M+uIjRz38VZVmIZ2ZFKC4R13TzeYAIWf7ck M4iA+KVuuJE59rpJlZpsviG8qv6xtyPcmA== X-Google-Smtp-Source: ABdhPJz3ikkHnnMztBw4qqixovFnev5cFOZvNOqHbhHr7F1cWGN60i659MHma5FR84Sbpi6sPrG0bw== X-Received: by 2002:a05:600c:1c1f:: with SMTP id j31mr600321wms.132.1626070076477; Sun, 11 Jul 2021 23:07:56 -0700 (PDT) Received: from jwang-Latitude-5491.fkb.profitbricks.net ([2001:16b8:49bc:8600:895c:e529:e1b8:7823]) by smtp.gmail.com with ESMTPSA id s17sm13344245wrv.2.2021.07.11.23.07.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 11 Jul 2021 23:07:56 -0700 (PDT) From: Jack Wang To: linux-rdma@vger.kernel.org Cc: bvanassche@acm.org, leon@kernel.org, dledford@redhat.com, jgg@ziepe.ca, haris.iqbal@ionos.com, jinpu.wang@ionos.com, Aleksei Marov , Gioh Kim Subject: [PATCHv2 for-rc 6/6] RDMA/rtrs: Move sq_wr_avail to rtrs_con Date: Mon, 12 Jul 2021 08:07:50 +0200 Message-Id: <20210712060750.16494-7-jinpu.wang@ionos.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210712060750.16494-1-jinpu.wang@ionos.com> References: <20210712060750.16494-1-jinpu.wang@ionos.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org In order to account HB for sq_wr_avail properly, move sq_wr_avail from rtrs_srv_con to rtrs_con. Although rtrs-clt do not care sq_wr_avail, but still init it to max_send_wr. Fixes: b38041d50add ("RDMA/rtrs: Do not signal for heatbeat") Signed-off-by: Jack Wang Reviewed-by: Aleksei Marov Reviewed-by: Gioh Kim Reviewed-by: Md Haris Iqbal --- drivers/infiniband/ulp/rtrs/rtrs-clt.c | 1 + drivers/infiniband/ulp/rtrs/rtrs-pri.h | 1 + drivers/infiniband/ulp/rtrs/rtrs-srv.c | 8 ++++---- drivers/infiniband/ulp/rtrs/rtrs-srv.h | 1 - drivers/infiniband/ulp/rtrs/rtrs.c | 1 + 5 files changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/ulp/rtrs/rtrs-clt.c b/drivers/infiniband/ulp/rtrs/rtrs-clt.c index f023676e05e4..ece3205531b8 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-clt.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-clt.c @@ -1680,6 +1680,7 @@ static int create_con_cq_qp(struct rtrs_clt_con *con) sess->queue_depth * 3 + 1); max_send_sge = 2; } + atomic_set(&con->c.sq_wr_avail, max_send_wr); cq_num = max_send_wr + max_recv_wr; /* alloc iu to recv new rkey reply when server reports flags set */ if (sess->flags & RTRS_MSG_NEW_RKEY_F || con->c.cid == 0) { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-pri.h b/drivers/infiniband/ulp/rtrs/rtrs-pri.h index 76581ebaed1d..d12ddfa50747 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-pri.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-pri.h @@ -97,6 +97,7 @@ struct rtrs_con { unsigned int cid; int nr_cqe; atomic_t wr_cnt; + atomic_t sq_wr_avail; }; struct rtrs_sess { diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.c b/drivers/infiniband/ulp/rtrs/rtrs-srv.c index 44ed15f38896..cd9a4ccf4c28 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.c +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.c @@ -507,11 +507,11 @@ bool rtrs_srv_resp_rdma(struct rtrs_srv_op *id, int status) ib_update_fast_reg_key(mr->mr, ib_inc_rkey(mr->mr->rkey)); } if (unlikely(atomic_sub_return(1, - &con->sq_wr_avail) < 0)) { + &con->c.sq_wr_avail) < 0)) { rtrs_err(s, "IB send queue full: sess=%s cid=%d\n", kobject_name(&sess->kobj), con->c.cid); - atomic_add(1, &con->sq_wr_avail); + atomic_add(1, &con->c.sq_wr_avail); spin_lock(&con->rsp_wr_wait_lock); list_add_tail(&id->wait_list, &con->rsp_wr_wait_list); spin_unlock(&con->rsp_wr_wait_lock); @@ -1268,7 +1268,7 @@ static void rtrs_srv_rdma_done(struct ib_cq *cq, struct ib_wc *wc) * post_send() RDMA write completions of IO reqs (read/write) * and hb. */ - atomic_add(s->signal_interval, &con->sq_wr_avail); + atomic_add(s->signal_interval, &con->c.sq_wr_avail); if (unlikely(!list_empty_careful(&con->rsp_wr_wait_list))) rtrs_rdma_process_wr_wait_list(con); @@ -1680,7 +1680,7 @@ static int create_con(struct rtrs_srv_sess *sess, */ } cq_num = max_send_wr + max_recv_wr; - atomic_set(&con->sq_wr_avail, max_send_wr); + atomic_set(&con->c.sq_wr_avail, max_send_wr); cq_vector = rtrs_srv_get_next_cq_vector(sess); /* TODO: SOFTIRQ can be faster, but be careful with softirq context */ diff --git a/drivers/infiniband/ulp/rtrs/rtrs-srv.h b/drivers/infiniband/ulp/rtrs/rtrs-srv.h index 6785c3b6363e..e81774f5acd3 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs-srv.h +++ b/drivers/infiniband/ulp/rtrs/rtrs-srv.h @@ -42,7 +42,6 @@ struct rtrs_srv_stats { struct rtrs_srv_con { struct rtrs_con c; - atomic_t sq_wr_avail; struct list_head rsp_wr_wait_list; spinlock_t rsp_wr_wait_lock; }; diff --git a/drivers/infiniband/ulp/rtrs/rtrs.c b/drivers/infiniband/ulp/rtrs/rtrs.c index b56dc5b82db0..ca542e477d38 100644 --- a/drivers/infiniband/ulp/rtrs/rtrs.c +++ b/drivers/infiniband/ulp/rtrs/rtrs.c @@ -191,6 +191,7 @@ static int rtrs_post_rdma_write_imm_empty(struct rtrs_con *con, struct rtrs_sess *sess = con->sess; enum ib_send_flags sflags; + atomic_dec_if_positive(&con->sq_wr_avail); sflags = (atomic_inc_return(&con->wr_cnt) % sess->signal_interval) ? 0 : IB_SEND_SIGNALED;