From patchwork Wed Nov 29 03:24:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13472167 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="Oi6Q2c3V" Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A0AC1AD for ; Tue, 28 Nov 2023 19:24:51 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701228290; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jUNm+K3QdC8Sh7qsblPNC2rl7ScFDfmW+mqWIBh5fr0=; b=Oi6Q2c3VhpU7+lRXh9zKzuvOXkIfEaI6s2vFEd7qA4DJWGSgDB4NIGXOAIEEv/40VlCp9Q gmtToajLDXuEgz2NI4EqOOFDUIpGad62/jouzJmZpv7alFywHgGOJrLrh0SEEUVruvd+J4 NdHXRmz04JukkdwX9/jZDu/DaXGEEqc= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH 1/4] RDMA/siw: Move tx_cpu ahead Date: Wed, 29 Nov 2023 11:24:15 +0800 Message-Id: <20231129032418.26705-2-guoqing.jiang@linux.dev> In-Reply-To: <20231129032418.26705-1-guoqing.jiang@linux.dev> References: <20231129032418.26705-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT We can reduce one cacheline for the usage of struct siw_qp. Before, /* size: 1928, cachelines: 31, members: 38 */ /* sum members: 1920, holes: 2, sum holes: 8 */ /* paddings: 4, sum paddings: 13 */ /* forced alignments: 3 */ after /* size: 1920, cachelines: 30, members: 38 */ /* paddings: 4, sum paddings: 13 */ /* forced alignments: 3 */ Signed-off-by: Guoqing Jiang Acked-by: Bernard Metzler --- drivers/infiniband/sw/siw/siw.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index b36d1ec25327..d14bb965af75 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -417,10 +417,10 @@ struct siw_iwarp_tx { struct siw_qp { struct ib_qp base_qp; struct siw_device *sdev; + int tx_cpu; struct kref ref; struct completion qp_free; struct list_head devq; - int tx_cpu; struct siw_qp_attrs attrs; struct siw_cep *cep; From patchwork Wed Nov 29 03:24:16 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13472168 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="A/X40f/O" Received: from out-171.mta0.migadu.com (out-171.mta0.migadu.com [91.218.175.171]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9A0B51AD for ; Tue, 28 Nov 2023 19:25:01 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701228300; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J9Ozm7DPrsE3luK3/lo3v/CromQpLb5Eg4/mogLhpmA=; b=A/X40f/OmqVz+CVUvO5pc1WMRZh3QpBWzC/z/5WHExjy7wzsSCRybM7rXMEvNa53H+oxgM OlJkYY5nuOwxqseAhvK6sB0U45WSXsmQupzHJYC5bNkKkUw2nz6av+oZ6Zdlg1zWkBiJfw FxEcqtj+YVLpsrUUtyEztCie1dNyCYQ= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH 2/4] RDMA/siw: Reduce memory usage of struct siw_rx_stream Date: Wed, 29 Nov 2023 11:24:16 +0800 Message-Id: <20231129032418.26705-3-guoqing.jiang@linux.dev> In-Reply-To: <20231129032418.26705-1-guoqing.jiang@linux.dev> References: <20231129032418.26705-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT We can reduce the memory of the struct by move some of it's member. Before, /* size: 144, cachelines: 3, members: 17 */ /* sum members: 124, holes: 3, sum holes: 12 */ /* sum bitfield members: 7 bits (0 bytes) */ /* padding: 7 */ /* bit_padding: 1 bits */ After /* size: 128, cachelines: 2, members: 17 */ /* padding: 3 */ /* bit_padding: 1 bits */ Signed-off-by: Guoqing Jiang Acked-by: Bernard Metzler --- drivers/infiniband/sw/siw/siw.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index d14bb965af75..2edba2a864bb 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -288,10 +288,11 @@ struct siw_rx_stream { int skb_offset; /* offset in skb */ int skb_copied; /* processed bytes in skb */ + enum siw_rx_state state; + union iwarp_hdr hdr; struct mpa_trailer trailer; - - enum siw_rx_state state; + struct shash_desc *mpa_crc_hd; /* * For each FPDU, main RX loop runs through 3 stages: @@ -313,7 +314,6 @@ struct siw_rx_stream { u64 ddp_to; u32 inval_stag; /* Stag to be invalidated */ - struct shash_desc *mpa_crc_hd; u8 rx_suspend : 1; u8 pad : 2; /* # of pad bytes expected */ u8 rdmap_op : 4; /* opcode of current frame */ From patchwork Wed Nov 29 03:24:17 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13472169 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="DYB6aCMd" Received: from out-173.mta0.migadu.com (out-173.mta0.migadu.com [91.218.175.173]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF7A21A3 for ; Tue, 28 Nov 2023 19:25:11 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701228310; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Nlmr0KT1zVJ15PyQIOLV7DYKZMz3r76Fa+xYktsyAA0=; b=DYB6aCMdP+zvna24hb9D4oA/JM05THshf1evI/FbNnOGxjM5f3Rg6xSXC84XT6cXzwKB0p nnGv41+HkTV/2F51oLtIuDDA6PCu34jTeB/Lr0XEC10Gy96qCcmZPoZwJs/XUnzC4QzfOK tqM8NxwYIvG7TevZVzRUaW9opefMv6Y= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH 3/4] RDMA/siw: Set qp_state in siw_query_qp Date: Wed, 29 Nov 2023 11:24:17 +0800 Message-Id: <20231129032418.26705-4-guoqing.jiang@linux.dev> In-Reply-To: <20231129032418.26705-1-guoqing.jiang@linux.dev> References: <20231129032418.26705-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Run test_query_rc_qp against siw failed since siw didn't set qp_state accordingly. To address it, introduce siw_qp_state_to_ib_qp_state which convert SIW_QP_STATE_IDLE to IB_QPS_INIT which is similar as in cxgb4. rdma-core# ./build/bin/run_tests.py --dev siw0 tests.test_qp.QPTest.test_query_rc_qp -v test_query_rc_qp (tests.test_qp.QPTest) Queries an RC QP after creation. Verifies that its properties are as ... FAIL ====================================================================== FAIL: test_query_rc_qp (tests.test_qp.QPTest) Queries an RC QP after creation. Verifies that its properties are as ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gjiang/rdma-core/tests/test_qp.py", line 284, in test_query_rc_qp self.query_qp_common_test(e.IBV_QPT_RC) File "/home/gjiang/rdma-core/tests/test_qp.py", line 265, in query_qp_common_test self.verify_qp_attrs(caps, e.IBV_QPS_INIT, qp_init_attr, qp_attr) File "/home/gjiang/rdma-core/tests/test_qp.py", line 239, in verify_qp_attrs self.assertEqual(state, attr.qp_state) AssertionError: != 0 ---------------------------------------------------------------------- Ran 1 test in 0.057s FAILED (failures=1) Signed-off-by: Guoqing Jiang --- drivers/infiniband/sw/siw/siw_verbs.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c index dca6a155523d..233985434cfd 100644 --- a/drivers/infiniband/sw/siw/siw_verbs.c +++ b/drivers/infiniband/sw/siw/siw_verbs.c @@ -19,6 +19,15 @@ #include "siw_verbs.h" #include "siw_mem.h" +static int siw_qp_state_to_ib_qp_state[IB_QPS_ERR] = { + [SIW_QP_STATE_IDLE] = IB_QPS_INIT, + [SIW_QP_STATE_RTR] = IB_QPS_RTR, + [SIW_QP_STATE_RTS] = IB_QPS_RTS, + [SIW_QP_STATE_CLOSING] = IB_QPS_SQD, + [SIW_QP_STATE_TERMINATE] = IB_QPS_SQE, + [SIW_QP_STATE_ERROR] = IB_QPS_ERR +}; + static int ib_qp_state_to_siw_qp_state[IB_QPS_ERR + 1] = { [IB_QPS_RESET] = SIW_QP_STATE_IDLE, [IB_QPS_INIT] = SIW_QP_STATE_IDLE, @@ -504,6 +513,7 @@ int siw_query_qp(struct ib_qp *base_qp, struct ib_qp_attr *qp_attr, } else { return -EINVAL; } + qp_attr->qp_state = siw_qp_state_to_ib_qp_state[qp->attrs.state]; qp_attr->cap.max_inline_data = SIW_MAX_INLINE; qp_attr->cap.max_send_wr = qp->attrs.sq_size; qp_attr->cap.max_send_sge = qp->attrs.sq_max_sges; From patchwork Wed Nov 29 03:24:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13472170 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="fXv/l5me" Received: from out-175.mta0.migadu.com (out-175.mta0.migadu.com [91.218.175.175]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6098C1AD for ; Tue, 28 Nov 2023 19:25:20 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701228318; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0ZNnDj3gZH8Ja7vN1CAquuuBpN9vbVWfJ2FqrFhLWGc=; b=fXv/l5me5xpZAVGzgVhZ8SLPM62JoKdD2WyjS7OJsBgJzv3F7K6YZYZONQtBy2RhX6F5W8 3eSnJEU3e0NEOsxAQTjK8FEowNub6At4vF2j88C1aOFMTTV5gtlKyOgjA/BzTtDgWEv3dF uxa2cwegwWqqZsPgoE0tqaIidNwkvqc= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH 4/4] RDMA/siw: Call orq_get_current if possible Date: Wed, 29 Nov 2023 11:24:18 +0800 Message-Id: <20231129032418.26705-5-guoqing.jiang@linux.dev> In-Reply-To: <20231129032418.26705-1-guoqing.jiang@linux.dev> References: <20231129032418.26705-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT We can call it in siw_orq_empty. Signed-off-by: Guoqing Jiang Acked-by: Bernard Metzler --- drivers/infiniband/sw/siw/siw.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index 2edba2a864bb..75253f2b3e3d 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -657,7 +657,7 @@ static inline struct siw_sqe *orq_get_free(struct siw_qp *qp) static inline int siw_orq_empty(struct siw_qp *qp) { - return qp->orq[qp->orq_get % qp->attrs.orq_size].flags == 0 ? 1 : 0; + return orq_get_current(qp)->flags == 0 ? 1 : 0; } static inline struct siw_sqe *irq_alloc_free(struct siw_qp *qp)