From patchwork Sun Dec 3 09:26:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13477218 X-Patchwork-Delegate: jgg@ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="mJw0/6AV" Received: from out-180.mta1.migadu.com (out-180.mta1.migadu.com [95.215.58.180]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 721EBE8 for ; Sun, 3 Dec 2023 01:27:38 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701595655; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HTxQflORM0ute0ovnCJ1V14VzEpqdJ0ViqQLbjbbRJ8=; b=mJw0/6AV4JUEvJBfYNoRMaZX8cFY3dV+VHfs/ZoNRbo167SrKjPEg1ylTrqDOT1aI76xp8 5JgD1SvOBO1X5MP7vcjwhXmS3YVdacskDpGduIjnnpzI24UsUAae3nwL/bEkVjvn0zl50K Q0ouzNQyT+EFNGlyKP0+TqkFYIW1N/o= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH V2 1/4] RDMA/siw: Move tx_cpu ahead Date: Sun, 3 Dec 2023 17:26:52 +0800 Message-Id: <20231203092655.28102-2-guoqing.jiang@linux.dev> In-Reply-To: <20231203092655.28102-1-guoqing.jiang@linux.dev> References: <20231203092655.28102-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT We can reduce one cacheline for the usage of struct siw_qp. Before, /* size: 1928, cachelines: 31, members: 38 */ /* sum members: 1920, holes: 2, sum holes: 8 */ /* paddings: 4, sum paddings: 13 */ /* forced alignments: 3 */ after /* size: 1920, cachelines: 30, members: 38 */ /* paddings: 4, sum paddings: 13 */ /* forced alignments: 3 */ Acked-by: Bernard Metzler Signed-off-by: Guoqing Jiang --- drivers/infiniband/sw/siw/siw.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index b36d1ec25327..d14bb965af75 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -417,10 +417,10 @@ struct siw_iwarp_tx { struct siw_qp { struct ib_qp base_qp; struct siw_device *sdev; + int tx_cpu; struct kref ref; struct completion qp_free; struct list_head devq; - int tx_cpu; struct siw_qp_attrs attrs; struct siw_cep *cep; From patchwork Sun Dec 3 09:26:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13477219 X-Patchwork-Delegate: jgg@ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="qOggcZwp" Received: from out-177.mta1.migadu.com (out-177.mta1.migadu.com [95.215.58.177]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 58AD6E7 for ; Sun, 3 Dec 2023 01:27:40 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701595658; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1jyFTmfTcRc/N3YNW3p9yTrNvCItvRn4ZVJp4GPZO6c=; b=qOggcZwpsK1iE2lGZ3yqmzweaTcaK2/mqKAfqXS7v/t4hhlTfEPBoeRLmcO8tkhzUU+8M8 jmEIepORH9nVxhGxxHyXujMVqjqws5MX6c2JoVQeyJNLbtjzIDiDiuXkmdroXeAhWgV5KM NaU+MoFz/xxU2t10SjO6Ef7zZpErxpg= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH V2 2/4] RDMA/siw: Reduce memory usage of struct siw_rx_stream Date: Sun, 3 Dec 2023 17:26:53 +0800 Message-Id: <20231203092655.28102-3-guoqing.jiang@linux.dev> In-Reply-To: <20231203092655.28102-1-guoqing.jiang@linux.dev> References: <20231203092655.28102-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT We can reduce the memory of the struct by move some of it's member. Before, /* size: 144, cachelines: 3, members: 17 */ /* sum members: 124, holes: 3, sum holes: 12 */ /* sum bitfield members: 7 bits (0 bytes) */ /* padding: 7 */ /* bit_padding: 1 bits */ After /* size: 128, cachelines: 2, members: 17 */ /* padding: 3 */ /* bit_padding: 1 bits */ Acked-by: Bernard Metzler Signed-off-by: Guoqing Jiang --- drivers/infiniband/sw/siw/siw.h | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index d14bb965af75..2edba2a864bb 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -288,10 +288,11 @@ struct siw_rx_stream { int skb_offset; /* offset in skb */ int skb_copied; /* processed bytes in skb */ + enum siw_rx_state state; + union iwarp_hdr hdr; struct mpa_trailer trailer; - - enum siw_rx_state state; + struct shash_desc *mpa_crc_hd; /* * For each FPDU, main RX loop runs through 3 stages: @@ -313,7 +314,6 @@ struct siw_rx_stream { u64 ddp_to; u32 inval_stag; /* Stag to be invalidated */ - struct shash_desc *mpa_crc_hd; u8 rx_suspend : 1; u8 pad : 2; /* # of pad bytes expected */ u8 rdmap_op : 4; /* opcode of current frame */ From patchwork Sun Dec 3 09:26:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13477220 X-Patchwork-Delegate: jgg@ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="v7iClf8C" Received: from out-186.mta1.migadu.com (out-186.mta1.migadu.com [95.215.58.186]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28D28E8 for ; Sun, 3 Dec 2023 01:27:43 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701595661; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NEESUjwfy67jdK+XIt8mpUL8qc37vyoJFBg+LpK5w8w=; b=v7iClf8CMbYcEk14rJ7QWqmcWsWGiYKA4vTZsP0UU3oaVjdD18ceHI3DNPCAZy3USGWAxN OwRKGm0P2KTFM+NY+sjqBuRi1bJvpRpaFEH7Hkw4B8CkS2ZVkrxghOPcu1rW7M/Viw7mHG EH6Z8ELX4oWkREPMISTG6mzbVJL8MTc= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH V2 3/4] RDMA/siw: Set qp_state in siw_query_qp Date: Sun, 3 Dec 2023 17:26:54 +0800 Message-Id: <20231203092655.28102-4-guoqing.jiang@linux.dev> In-Reply-To: <20231203092655.28102-1-guoqing.jiang@linux.dev> References: <20231203092655.28102-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Run test_query_rc_qp against siw failed since siw didn't set qp_state accordingly. To address it, introduce siw_qp_state_to_ib_qp_state which convert SIW_QP_STATE_IDLE to IB_QPS_INIT which is similar as in cxgb4. rdma-core# ./build/bin/run_tests.py --dev siw0 tests.test_qp.QPTest.test_query_rc_qp -v test_query_rc_qp (tests.test_qp.QPTest) Queries an RC QP after creation. Verifies that its properties are as ... FAIL ====================================================================== FAIL: test_query_rc_qp (tests.test_qp.QPTest) Queries an RC QP after creation. Verifies that its properties are as ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/gjiang/rdma-core/tests/test_qp.py", line 284, in test_query_rc_qp self.query_qp_common_test(e.IBV_QPT_RC) File "/home/gjiang/rdma-core/tests/test_qp.py", line 265, in query_qp_common_test self.verify_qp_attrs(caps, e.IBV_QPS_INIT, qp_init_attr, qp_attr) File "/home/gjiang/rdma-core/tests/test_qp.py", line 239, in verify_qp_attrs self.assertEqual(state, attr.qp_state) AssertionError: != 0 ---------------------------------------------------------------------- Ran 1 test in 0.057s FAILED (failures=1) Signed-off-by: Guoqing Jiang Acked-by: Bernard Metzler --- drivers/infiniband/sw/siw/siw_verbs.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c index dca6a155523d..ecf0444666b4 100644 --- a/drivers/infiniband/sw/siw/siw_verbs.c +++ b/drivers/infiniband/sw/siw/siw_verbs.c @@ -19,6 +19,15 @@ #include "siw_verbs.h" #include "siw_mem.h" +static int siw_qp_state_to_ib_qp_state[SIW_QP_STATE_COUNT] = { + [SIW_QP_STATE_IDLE] = IB_QPS_INIT, + [SIW_QP_STATE_RTR] = IB_QPS_RTR, + [SIW_QP_STATE_RTS] = IB_QPS_RTS, + [SIW_QP_STATE_CLOSING] = IB_QPS_SQD, + [SIW_QP_STATE_TERMINATE] = IB_QPS_SQE, + [SIW_QP_STATE_ERROR] = IB_QPS_ERR +}; + static int ib_qp_state_to_siw_qp_state[IB_QPS_ERR + 1] = { [IB_QPS_RESET] = SIW_QP_STATE_IDLE, [IB_QPS_INIT] = SIW_QP_STATE_IDLE, @@ -504,6 +513,7 @@ int siw_query_qp(struct ib_qp *base_qp, struct ib_qp_attr *qp_attr, } else { return -EINVAL; } + qp_attr->qp_state = siw_qp_state_to_ib_qp_state[qp->attrs.state]; qp_attr->cap.max_inline_data = SIW_MAX_INLINE; qp_attr->cap.max_send_wr = qp->attrs.sq_size; qp_attr->cap.max_send_sge = qp->attrs.sq_max_sges; From patchwork Sun Dec 3 09:26:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Guoqing Jiang X-Patchwork-Id: 13477221 X-Patchwork-Delegate: jgg@ziepe.ca Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux.dev header.i=@linux.dev header.b="j2wqyJJ5" Received: from out-174.mta1.migadu.com (out-174.mta1.migadu.com [95.215.58.174]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0B96E7 for ; Sun, 3 Dec 2023 01:27:45 -0800 (PST) X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1701595664; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=twbMlmA50cyTWn+OjyRmB+rWRJ+rRbrrWItsr3z3XcM=; b=j2wqyJJ5l4Dyfa9szJfwFbzywtarsL95DK3mKq2/CCtFm/RR41khPoMn1Tuo6d/kMdAU2e DeBWW1T6pJnAQ5CgAz8Qj3VRaLeX2QjOosRVmFZ1n4EARSDKQY00b0yHCFKuHvw6l1dOIA r+HSYSLXH+1IRtNbS9lg0bdyoWYNsdI= From: Guoqing Jiang To: bmt@zurich.ibm.com, jgg@ziepe.ca, leon@kernel.org Cc: linux-rdma@vger.kernel.org, guoqing.jiang@linux.dev Subject: [PATCH V2 4/4] RDMA/siw: Call orq_get_current if possible Date: Sun, 3 Dec 2023 17:26:55 +0800 Message-Id: <20231203092655.28102-5-guoqing.jiang@linux.dev> In-Reply-To: <20231203092655.28102-1-guoqing.jiang@linux.dev> References: <20231203092655.28102-1-guoqing.jiang@linux.dev> Precedence: bulk X-Mailing-List: linux-rdma@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Use orq_get_current() in siw_orq_empty(). Acked-by: Bernard Metzler Signed-off-by: Guoqing Jiang --- drivers/infiniband/sw/siw/siw.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index 2edba2a864bb..75253f2b3e3d 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -657,7 +657,7 @@ static inline struct siw_sqe *orq_get_free(struct siw_qp *qp) static inline int siw_orq_empty(struct siw_qp *qp) { - return qp->orq[qp->orq_get % qp->attrs.orq_size].flags == 0 ? 1 : 0; + return orq_get_current(qp)->flags == 0 ? 1 : 0; } static inline struct siw_sqe *irq_alloc_free(struct siw_qp *qp)