From patchwork Wed Jun 15 08:40:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Metzmacher X-Patchwork-Id: 12881927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30D01C43334 for ; Wed, 15 Jun 2022 08:41:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243682AbiFOIld (ORCPT ); Wed, 15 Jun 2022 04:41:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44020 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346460AbiFOIlN (ORCPT ); Wed, 15 Jun 2022 04:41:13 -0400 Received: from hr2.samba.org (hr2.samba.org [IPv6:2a01:4f8:192:486::2:0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6A1C4AE10 for ; Wed, 15 Jun 2022 01:41:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=Message-Id:Date:Cc:To:From; bh=25p7925ITEYPPnsTBDZLySHrB0vJm02meGuExfPpW+M=; b=KIiPDFdUlcHRPeUQiijKcli1YV ZsOwLGjV7iibEb3OJRuy8e52zxHMECj8f25l9PLMiBYdA2SvtjUpoFsL5PhPXUu0hRB/tjNSj7A3x K06zwgruEb/2ZJUwudiuAglxns3RXU4H6kTSdAWf6d4VTGFLfY8YrgmU8PqYLyrDcWxfG9kCnWfyP t11UhKz3Hjbld2IoxQUKN7y/isT6p/hx+N4aF+pfd3lbGzHRtkJohYpcLBVEHIqLLo+WCG+iVTxXY pPzmk/BqS1yxQGQMGhJ9XJBo+GNWWiHlADLREcFdy1V7lCfo2JP59wq49yeVZEB0op6F/0ec8I1R4 2by/d1pr2ZqJ3bYzcH21VSPsY54l+ydOe9XvMsfsLhxr/aNALMWe2GNWVWJV24qZ4Paouqpv6m89B mPavPWzC0ahShXAchjSbLLF9ggD7GO2SCNxkEnWfbDcOgFXv6MgbgHFZd+Fz1nbKc6L7Z50KVCzqn hDyk1DzdAzhGzmzoQnQm1htF; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1o1OaM-005p9H-Id; Wed, 15 Jun 2022 08:41:10 +0000 From: Stefan Metzmacher To: Bernard Metzler , linux-rdma@vger.kernel.org Cc: Stefan Metzmacher Subject: [PATCH 1/7] rdma/siw: make use of kernel_{bind,connect,listen}() Date: Wed, 15 Jun 2022 10:40:01 +0200 Message-Id: <883fa5d941c9fa7d131abe9b9424438076f05a9a.1655248086.git.metze@samba.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org That's nicer than dereferencing socket structures. This prepares making rdma_connect()/siw_connect() non-blocking in order to avoid deadlocks in the callers. Fixes: 6c52fdc244b5 ("rdma/siw: connection management") Signed-off-by: Stefan Metzmacher Cc: Bernard Metzler Cc: linux-rdma@vger.kernel.org --- drivers/infiniband/sw/siw/siw_cm.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 17f34d584cd9..644c7d50e991 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -1318,11 +1318,11 @@ static int kernel_bindconnect(struct socket *s, struct sockaddr *laddr, return rv; } - rv = s->ops->bind(s, laddr, size); + rv = kernel_bind(s, laddr, size); if (rv < 0) return rv; - rv = s->ops->connect(s, raddr, size, flags); + rv = kernel_connect(s, raddr, size, flags); return rv < 0 ? rv : 0; } @@ -1788,8 +1788,8 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) if (ipv4_is_zeronet(laddr->sin_addr.s_addr)) s->sk->sk_bound_dev_if = sdev->netdev->ifindex; - rv = s->ops->bind(s, (struct sockaddr *)laddr, - sizeof(struct sockaddr_in)); + rv = kernel_bind(s, (struct sockaddr *)laddr, + sizeof(struct sockaddr_in)); } else { struct sockaddr_in6 *laddr = &to_sockaddr_in6(id->local_addr); @@ -1806,8 +1806,8 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) if (ipv6_addr_any(&laddr->sin6_addr)) s->sk->sk_bound_dev_if = sdev->netdev->ifindex; - rv = s->ops->bind(s, (struct sockaddr *)laddr, - sizeof(struct sockaddr_in6)); + rv = kernel_bind(s, (struct sockaddr *)laddr, + sizeof(struct sockaddr_in6)); } if (rv) { siw_dbg(id->device, "socket bind error: %d\n", rv); @@ -1827,7 +1827,7 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) rv, backlog); goto error; } - rv = s->ops->listen(s, backlog); + rv = kernel_listen(s, backlog); if (rv) { siw_dbg(id->device, "listen error %d\n", rv); goto error; From patchwork Wed Jun 15 08:40:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Metzmacher X-Patchwork-Id: 12881928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEC26CCA47B for ; Wed, 15 Jun 2022 08:41:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346297AbiFOIle (ORCPT ); Wed, 15 Jun 2022 04:41:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346532AbiFOIlV (ORCPT ); Wed, 15 Jun 2022 04:41:21 -0400 Received: from hr2.samba.org (hr2.samba.org [IPv6:2a01:4f8:192:486::2:0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFDE44AE15 for ; Wed, 15 Jun 2022 01:41:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=Message-Id:Date:Cc:To:From; bh=8lmA+8ryXLmsg83Rufbtg4k6StXa61BzR9Q+RfHAEIk=; b=eFdnVhYbZP4B2ZuuX1vfiy9JR8 qw5eppitylKPtuDbcjM2Vh/0EcRYFZBLhGxeTbwUBPI6NRoUooDxfwQOb0pAgZ/Bz0CN1UGRN5oWE gDij2uWq5Idxf73HXHWcww1nAEYDiuzqKZVNz3O6LXZsJ8UCu/FN/1fWXuc/AxtL4bb4oIJGe5nIg OoYaE5r7CYEtBomSkejfJGPQu1Y/UhjFDvkH2MdRVmsL0SMunqIlpzVs+0ySt6Rv1gQBYnTtLLoya IdVP4IORGfvURI3CZ5fbkxT9UApU2CrYueu4rZMR54R03hCeZpnxGLT0XKPl9nZfAk63W6MwwKTTO gHz+2afQKz6aY5OwbI1qomhxsdtULFfjUIA7AEz/fBfUOi5B/N0BXVcjgo2eDOJwBSV8Hj0EuOoia RJUS38x+XsurxojgU7jAk++kRk6+LGhRL5teBRilGOCW6qvJDBvtMp2vjyZyWC08qvvrbkzBALAYQ 4cTQwVXGs4kkk81J7n0pEvfF; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1o1OaU-005p9Q-Pj; Wed, 15 Jun 2022 08:41:18 +0000 From: Stefan Metzmacher To: Bernard Metzler , linux-rdma@vger.kernel.org Cc: Stefan Metzmacher Subject: [PATCH 2/7] rdma/siw: let siw_connect() set AWAIT_MPAREP before siw_send_mpareqrep() Date: Wed, 15 Jun 2022 10:40:02 +0200 Message-Id: X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org There's no real change made in this commit, but it makes the follwing commits easier to review. The idea is that we stay in SIW_EPSTATE_CONNECTING as long as we only deal with tcp and not started the MPA negotiation. Fixes: 6c52fdc244b5 ("rdma/siw: connection management") Signed-off-by: Stefan Metzmacher Cc: Bernard Metzler Cc: linux-rdma@vger.kernel.org --- drivers/infiniband/sw/siw/siw_cm.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 644c7d50e991..8d1e7f497cf9 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -1425,8 +1425,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) */ siw_cep_socket_assoc(cep, s); - cep->state = SIW_EPSTATE_AWAIT_MPAREP; - /* * Set MPA Request bits: CRC if required, no MPA Markers, * MPA Rev. according to module parameter 'mpa_version', Key 'Request'. @@ -1469,6 +1467,8 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) } memcpy(cep->mpa.hdr.key, MPA_KEY_REQ, 16); + cep->state = SIW_EPSTATE_AWAIT_MPAREP; + rv = siw_send_mpareqrep(cep, params->private_data, pd_len); /* * Reset private data. From patchwork Wed Jun 15 08:40:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Metzmacher X-Patchwork-Id: 12881929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6BFCCCA481 for ; Wed, 15 Jun 2022 08:41:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346460AbiFOIle (ORCPT ); Wed, 15 Jun 2022 04:41:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346596AbiFOIl3 (ORCPT ); Wed, 15 Jun 2022 04:41:29 -0400 Received: from hr2.samba.org (hr2.samba.org [IPv6:2a01:4f8:192:486::2:0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F012D4AE10 for ; Wed, 15 Jun 2022 01:41:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=Message-Id:Date:Cc:To:From; bh=NdCH8KXK+QOpawcMPiDpjErRzx6oy04IxS+cwW7QQVo=; b=gfqT/BYdMGHi+vc+g7zOkU77lh rAte9UDQ0Rrn1mrDltAjCAULYsK1tr7LyJJRR1kN/1+rZBP/xplbXk0hwsxM+6eD2t+lCYyAXdSGR nPltTQzkU12SiiX6E0ilElMpyfleoov2XiC3qZ+CZTuNNCm43g5uqFHNiq2T5jWw+DKo5sKE7O2S1 sxgGJIEafINDWxupESGPN9Sf9N+hI2X4ENd4L4VXQKsRc4fJbUFWKfPhbUdNnDx2kVyBuJM1WYuYs mJp/2nA22Xpm/VXrGGuzKU9wVX87r7R/wA6DLqCrwH1W4ApaHy8HWaPtCYwd03dyr2OS++jDUj0Nh B9T/w1Ys5qgvrJAl1O+SQMb40/+fUoshTaBuY5dM7Avic/B8MXRk/SAYE0SkoPtEiN3pZ5zwgD38e kXrmd6MeLggYCvMYwBHLTSZY2NqFgXHWXrmLDHtRRXxRQslIquURNfUEEdm9LfGTBN47anH5TcT06 e/nVVNKf/wi7CaY+Ft0aFa2k; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1o1Oab-005p9Z-Uf; Wed, 15 Jun 2022 08:41:26 +0000 From: Stefan Metzmacher To: Bernard Metzler , linux-rdma@vger.kernel.org Cc: Stefan Metzmacher Subject: [PATCH 3/7] rdma/siw: create a temporary copy of private data Date: Wed, 15 Jun 2022 10:40:03 +0200 Message-Id: <128fce3f256c46feb2e4d31e27da572ff7ed1336.1655248086.git.metze@samba.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The final patch will implement a non-blocking connect, which means that siw_connect() will be split into siw_connect() and siw_connected(). kernel_bindconnect() will be the last action in siw_connect(), while the MPA negotiation is deferred to siw_connected(). We should not rely on the callers private data pointers to be still valid when siw_connected() is called, so we better create a copy. Also note that __siw_cep_dealloc() already calls kfree(cep->mpa.pdata), so we already have the required cleanup when we'll split out siw_connected() and an error will prevent siw_connected() being called at all. Fixes: 6c52fdc244b5 ("rdma/siw: connection management") Signed-off-by: Stefan Metzmacher Cc: Bernard Metzler Cc: linux-rdma@vger.kernel.org --- drivers/infiniband/sw/siw/siw_cm.c | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 8d1e7f497cf9..0e53219d29de 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -1467,13 +1467,27 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) } memcpy(cep->mpa.hdr.key, MPA_KEY_REQ, 16); + if (pd_len > 0) { + cep->mpa.pdata = kmemdup(params->private_data, pd_len, GFP_KERNEL); + if (IS_ERR_OR_NULL(cep->mpa.pdata)) { + rv = -ENOMEM; + goto error; + } + cep->mpa.hdr.params.pd_len = pd_len; + } + cep->state = SIW_EPSTATE_AWAIT_MPAREP; - rv = siw_send_mpareqrep(cep, params->private_data, pd_len); + rv = siw_send_mpareqrep(cep, cep->mpa.pdata, + cep->mpa.hdr.params.pd_len); /* * Reset private data. */ - cep->mpa.hdr.params.pd_len = 0; + if (cep->mpa.hdr.params.pd_len) { + cep->mpa.hdr.params.pd_len = 0; + kfree(cep->mpa.pdata); + cep->mpa.pdata = NULL; + } if (rv >= 0) { rv = siw_cm_queue_work(cep, SIW_CM_WORK_MPATIMEOUT); From patchwork Wed Jun 15 08:40:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Metzmacher X-Patchwork-Id: 12881930 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BDABC43334 for ; Wed, 15 Jun 2022 08:41:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346570AbiFOIlh (ORCPT ); Wed, 15 Jun 2022 04:41:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44176 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346459AbiFOIlf (ORCPT ); Wed, 15 Jun 2022 04:41:35 -0400 Received: from hr2.samba.org (hr2.samba.org [IPv6:2a01:4f8:192:486::2:0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7046F4AE17 for ; Wed, 15 Jun 2022 01:41:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=Message-Id:Date:Cc:To:From; bh=s5HWzCEtMjmjkB+xRG1YT2W4LPFIUXExDZNg/wHwxPk=; b=R1EMRO7fyVRIgDz4IJSV7F2UNA I3sTLmvgtHAZF2gonYh2lL+oO609vP527mlzTiYlCoghs5XxlYdUwJk5n3ibfdtotqEYfZEoKW50P VYEN0zn3G+N1/y7z0pA94GHmWHAWwzLM5ry+TPco2PiCmr29qQc7MLzXiVim02gtvbEIcl2vyW5bJ GgQlCAh3IpRiZJVQxIAndKJT86pp/1RU8TAo5eF9DGo3k4jkhpXm/CPnYILuInhQr8jwOzv/e/BLf 1+3hHU7XmooSEjaJa/uGLU/zBQdaLb1QB9wHHGuPeDBmDFUDswLh03J1YxUCQBWkjZ9icBkcd4M8Q /VaERwWfqNpi5Uw2uCg/PrwNwjCMAIEJEluQyq3Y+Mu0Y8AxV9CkB8UZVXptknbO6l4qMqKc0/t9j cKUBEpq2kVIQHCVnR/qp02c/M4IZbfCCCf3yf2SySyIqtAWLIkaJaX6Q+ddJdYeAaQFjJOx5UQ/OI XjJm7WdoegXo773RhY5pzJsj; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1o1Oai-005p9t-Ev; Wed, 15 Jun 2022 08:41:32 +0000 From: Stefan Metzmacher To: Bernard Metzler , linux-rdma@vger.kernel.org Cc: Stefan Metzmacher Subject: [PATCH 4/7] rdma/siw: use error and out logic at the end of siw_connect() Date: Wed, 15 Jun 2022 10:40:04 +0200 Message-Id: <954c86484412d79ce9116e67fdf0efb94add67ba.1655248086.git.metze@samba.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This will make the following changes easier. Fixes: 6c52fdc244b5 ("rdma/siw: connection management") Signed-off-by: Stefan Metzmacher Cc: Bernard Metzler Cc: linux-rdma@vger.kernel.org --- drivers/infiniband/sw/siw/siw_cm.c | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 0e53219d29de..b19a2b777814 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -1489,14 +1489,19 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) cep->mpa.pdata = NULL; } - if (rv >= 0) { - rv = siw_cm_queue_work(cep, SIW_CM_WORK_MPATIMEOUT); - if (!rv) { - siw_dbg_cep(cep, "[QP %u]: exit\n", qp_id(qp)); - siw_cep_set_free(cep); - return 0; - } + if (rv < 0) { + goto error; + } + + rv = siw_cm_queue_work(cep, SIW_CM_WORK_MPATIMEOUT); + if (rv != 0) { + goto error; } + + siw_dbg_cep(cep, "[QP %u]: exit\n", qp_id(qp)); + siw_cep_set_free(cep); + return 0; + error: siw_dbg(id->device, "failed: %d\n", rv); From patchwork Wed Jun 15 08:40:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Metzmacher X-Patchwork-Id: 12881931 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0C60C43334 for ; Wed, 15 Jun 2022 08:41:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346459AbiFOIlp (ORCPT ); Wed, 15 Jun 2022 04:41:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44458 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346684AbiFOIln (ORCPT ); Wed, 15 Jun 2022 04:41:43 -0400 Received: from hr2.samba.org (hr2.samba.org [IPv6:2a01:4f8:192:486::2:0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB9624B403 for ; Wed, 15 Jun 2022 01:41:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=Message-Id:Date:Cc:To:From; bh=2bnt6S+jT4KW1xKlfXKsfg8/BcScuy9VEIFYJAE2HlI=; b=e+ZTFD8TCiZTDF9Ga0NuCX9O2r s1hgiXmZk4WcJ9Fyfg5Zn6DAXcNKqtrHF8uyruiEKQgwQJ0YBboR184R0w4YXeT99od2pl0aZ7cWJ F0KVxOvxhfFxapK4tQxPVB8lKhI9fipmdljjtqG+ye07TjJ3WMrGdccseHbV1gq0B//we0o1Tu38q 5bID0ogLTQxRzJzPqRhDIBqjZSJMUfR1XVJVBFRabRzbCxpx+jctwqTkkf87ez6jtQdsIbAvyeEMy 8u8YhTG0ZLocGexIbWZ38TTX6LyQkrdfNCcb9ZKVtdWmMKyGdeCxNgl8pjYisxyTcFeUdtzTxs2Jm YhqGbGIJdcfXU7v2BfZf06occfuMV2uznv0uzcGH5BZj4p857YUNzbETZIwf25BA/mKv2Lg6ehAzM 7DahDAiJMXrikOuACTfLcgDUAXmS2Z8Bkt6r4gxYlBc/2JcQ3M2dF2UIdYSCkMtTGzMq0scFvOezC XX7b39InrDAfrjDQNWCFwc3z; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1o1Oao-005pA2-UC; Wed, 15 Jun 2022 08:41:39 +0000 From: Stefan Metzmacher To: Bernard Metzler , linux-rdma@vger.kernel.org Cc: Stefan Metzmacher Subject: [PATCH 5/7] rdma/siw: start mpa timer before calling siw_send_mpareqrep() Date: Wed, 15 Jun 2022 10:40:05 +0200 Message-Id: <505a18c83a283963174c9e567ce73d055e26ec7b.1655248086.git.metze@samba.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org The mpa timer will also span the non-blocking connect in the final patch. Fixes: 6c52fdc244b5 ("rdma/siw: connection management") Signed-off-by: Stefan Metzmacher Cc: Bernard Metzler Cc: linux-rdma@vger.kernel.org --- drivers/infiniband/sw/siw/siw_cm.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index b19a2b777814..3fee1d4ef252 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -1476,6 +1476,11 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) cep->mpa.hdr.params.pd_len = pd_len; } + rv = siw_cm_queue_work(cep, SIW_CM_WORK_MPATIMEOUT); + if (rv != 0) { + goto error; + } + cep->state = SIW_EPSTATE_AWAIT_MPAREP; rv = siw_send_mpareqrep(cep, cep->mpa.pdata, @@ -1493,11 +1498,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) goto error; } - rv = siw_cm_queue_work(cep, SIW_CM_WORK_MPATIMEOUT); - if (rv != 0) { - goto error; - } - siw_dbg_cep(cep, "[QP %u]: exit\n", qp_id(qp)); siw_cep_set_free(cep); return 0; @@ -1506,6 +1506,8 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) siw_dbg(id->device, "failed: %d\n", rv); if (cep) { + siw_cancel_mpatimer(cep); + siw_socket_disassoc(s); sock_release(s); cep->sock = NULL; From patchwork Wed Jun 15 08:40:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Metzmacher X-Patchwork-Id: 12881932 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8768C43334 for ; Wed, 15 Jun 2022 08:41:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346652AbiFOIlt (ORCPT ); Wed, 15 Jun 2022 04:41:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346642AbiFOIls (ORCPT ); Wed, 15 Jun 2022 04:41:48 -0400 Received: from hr2.samba.org (hr2.samba.org [IPv6:2a01:4f8:192:486::2:0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 857034B1FE for ; Wed, 15 Jun 2022 01:41:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=Message-Id:Date:Cc:To:From; bh=YgADCWgD7M2HHL2ab+yTnWybxXOMGoyAlvlMkl4oa1s=; b=nvbW0FsPDSvJC3tZQIajrkprLA +ywA1rSFIeia7Wb4RaRD59b0dzR7HVt1qxngNnP9li9VLzKNo7B1AdLkD2Qj4YbqyAtdKB0OWvkYS q8o+Hr64nZpwQciI4CckTRi6BDwZzTrRYAKPAMohsAdbOeRN2RCupgtR90BPj2zt18elJzZgV/3dW QweEsvRBQunJD1LCJLZ0Tv7gLNJcr1UAlEveJZN08Chc5KflCa/h5vJ4g2T3A6qHzuQmpXN4vC9am Ux/HNfGAse+VVSvTGfOWAX+UcJz2t//fSgeio8o5q7mghJoM+d5ewPrWGVq8GGprE6JACTt6gGb5f kudysy8v/ZxldgIOjxHjzeeM6oSjjhPevIoRCM0xcF4yvunV0VKkBxUMDpA5vYn1GURUxKliEBBSK Pgamhfvhj7h/k9+IiYcz2NdYUa5Nui2b+e2/2kvvblD/I780PO4cr7WOtz1m0iAuTkT2dvCXY0dGl 4fnme/yHI4NfQO7bl3zqgfEj; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1o1Oav-005pAD-G9; Wed, 15 Jun 2022 08:41:45 +0000 From: Stefan Metzmacher To: Bernard Metzler , linux-rdma@vger.kernel.org Cc: Stefan Metzmacher Subject: [PATCH 6/7] rdma/siw: call the blocking kernel_bindconnect() just before siw_send_mpareqrep() Date: Wed, 15 Jun 2022 10:40:06 +0200 Message-Id: <8f97aa28bad93d95b12eb8393ede721340ea0ba2.1655248086.git.metze@samba.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org We should build all state before calling kernel_bindconnect(). This will allow us to go async in the final patch. Fixes: 6c52fdc244b5 ("rdma/siw: connection management") Signed-off-by: Stefan Metzmacher Cc: Bernard Metzler Cc: linux-rdma@vger.kernel.org --- drivers/infiniband/sw/siw/siw_cm.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 3fee1d4ef252..9c5276d08538 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -1373,18 +1373,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) if (rv < 0) goto error; - /* - * NOTE: For simplification, connect() is called in blocking - * mode. Might be reconsidered for async connection setup at - * TCP level. - */ - rv = kernel_bindconnect(s, laddr, raddr, id->afonly); - if (rv != 0) { - siw_dbg_qp(qp, "kernel_bindconnect: error %d\n", rv); - goto error; - } - if (siw_tcp_nagle == false) - tcp_sock_set_nodelay(s->sk); cep = siw_cep_alloc(sdev); if (!cep) { rv = -ENOMEM; @@ -1481,6 +1469,19 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) goto error; } + /* + * NOTE: For simplification, connect() is called in blocking + * mode. Might be reconsidered for async connection setup at + * TCP level. + */ + rv = kernel_bindconnect(s, laddr, raddr, id->afonly); + if (rv != 0) { + siw_dbg_qp(qp, "kernel_bindconnect: error %d\n", rv); + goto error; + } + if (siw_tcp_nagle == false) + tcp_sock_set_nodelay(s->sk); + cep->state = SIW_EPSTATE_AWAIT_MPAREP; rv = siw_send_mpareqrep(cep, cep->mpa.pdata, From patchwork Wed Jun 15 08:40:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Metzmacher X-Patchwork-Id: 12881933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10C83C43334 for ; Wed, 15 Jun 2022 08:41:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1346642AbiFOIl5 (ORCPT ); Wed, 15 Jun 2022 04:41:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1346676AbiFOIl4 (ORCPT ); Wed, 15 Jun 2022 04:41:56 -0400 Received: from hr2.samba.org (hr2.samba.org [IPv6:2a01:4f8:192:486::2:0]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3AF004BB9F for ; Wed, 15 Jun 2022 01:41:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=samba.org; s=42; h=Message-Id:Date:Cc:To:From; bh=edULU+ezUxEWy9UkiTGJmVAkFx2pVj88syU9zDLkdqI=; b=FJVwvdKVkBkmPMRxBUcWiUQZog ruZyeajNQMyNp8rBbZrhaoMPjE5NcrTPvfFZeqyf7wkLQJN84YrjPZAHslgs+DF9xJzBkDTPvXIKd bcdPpCgCcAo+zdMcbdZGbi5lqZ0TPTr0kl2uOcCmbsfRWILwuIvuikbS9wUJ1ixE04LPPzgSpX9Tn 5p3ZmYdI1cCrtL2Xt+0bINQ84QBjQmC0nO2NY/AmO3Fq/KHtBPJ7/o4zVLmlQH4lB8VEzahIANilR Mkep7/aJXoURQMWH2yk07HtM0BINHRb9Hvc59Eiq6aes/+P6Tm11fzc9wr/N2kgGLPKa9ejz5o8dm mkJd70xu/FgrYm1c1mTcA553z/x149/f0hOpIzgIByzkCnXagQdyJeMqCp6WwiOlZghqLrVgKVuM9 J3XQBpTOhIm4MDtGBMv/IrXd4ahUauLdpHnIoZR+z5E8lTOtOEkgv3fOCEZNMb9Nk530ez8wVbFyK AoIW8JKHIQpi/cfWHHeZB6XS; Received: from [127.0.0.2] (localhost [127.0.0.1]) by hr2.samba.org with esmtpsa (TLS1.3:ECDHE_SECP256R1__ECDSA_SECP256R1_SHA256__CHACHA20_POLY1305:256) (Exim) id 1o1Ob2-005pAN-7S; Wed, 15 Jun 2022 08:41:52 +0000 From: Stefan Metzmacher To: Bernard Metzler , linux-rdma@vger.kernel.org Cc: Stefan Metzmacher Subject: [PATCH 7/7] rdma/siw: implement non-blocking connect. Date: Wed, 15 Jun 2022 10:40:07 +0200 Message-Id: <56c6768ccff38b64feb4dbbded754da3138dbd50.1655248086.git.metze@samba.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: linux-rdma@vger.kernel.org This is very important in order to prevent deadlocks. The RDMA application layer expects rdma_connect() to be non-blocking as the completion is handled via RDMA_CM_EVENT_ESTABLISHED and other async events. It's not unlikely to hold a lock during the rdma_connect() call. Without out this a connection attempt to a non-existing/reachable server block until the very long tcp timeout hits. The application layer had no chance to have its own timeout handler as that would just deadlock with the already blocking rdma_connect(). First rdma_connect() holds id_priv->handler_mutex and deadlocks rdma_destroy_id(). And iw_cm_connect() called from within rdma_connect() sets IWCM_F_CONNECT_WAIT during the call to cm_id->device->ops.iw_connect(), siw_connect() in this case. It means that iw_cm_disconnect() and iw_destroy_cm_id() will both deadlock waiting for IWCM_F_CONNECT_WAIT being cleared. Fixes: 6c52fdc244b5 ("rdma/siw: connection management") Signed-off-by: Stefan Metzmacher Cc: Bernard Metzler Cc: linux-rdma@vger.kernel.org --- drivers/infiniband/sw/siw/siw_cm.c | 125 ++++++++++++++++++++++------- drivers/infiniband/sw/siw/siw_cm.h | 1 + 2 files changed, 96 insertions(+), 30 deletions(-) diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 9c5276d08538..e06375018472 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -37,6 +37,7 @@ static void siw_cm_llp_write_space(struct sock *s); static void siw_cm_llp_error_report(struct sock *s); static int siw_cm_upcall(struct siw_cep *cep, enum iw_cm_event_type reason, int status); +static void siw_connected(struct siw_cep *cep); static void siw_sk_assign_cm_upcalls(struct sock *sk) { @@ -1012,6 +1013,10 @@ static void siw_cm_work_handler(struct work_struct *w) siw_accept_newconn(cep); break; + case SIW_CM_WORK_CONNECTED: + siw_connected(cep); + break; + case SIW_CM_WORK_READ_MPAHDR: if (cep->state == SIW_EPSTATE_AWAIT_MPAREQ) { if (cep->listen_cep) { @@ -1226,6 +1231,7 @@ static void siw_cm_llp_data_ready(struct sock *sk) switch (cep->state) { case SIW_EPSTATE_RDMA_MODE: case SIW_EPSTATE_LISTENING: + case SIW_EPSTATE_CONNECTING: break; case SIW_EPSTATE_AWAIT_MPAREQ: @@ -1279,12 +1285,26 @@ static void siw_cm_llp_state_change(struct sock *sk) switch (sk->sk_state) { case TCP_ESTABLISHED: - /* - * handle accepting socket as special case where only - * new connection is possible - */ - siw_cm_queue_work(cep, SIW_CM_WORK_ACCEPT); - break; + if (cep->state == SIW_EPSTATE_CONNECTING) { + /* + * handle accepting socket as special case where only + * new connection is possible + */ + siw_cm_queue_work(cep, SIW_CM_WORK_CONNECTED); + break; + + } else if (cep->state == SIW_EPSTATE_LISTENING) { + /* + * handle accepting socket as special case where only + * new connection is possible + */ + siw_cm_queue_work(cep, SIW_CM_WORK_ACCEPT); + break; + } + siw_dbg_cep(cep, + "unexpected socket state %d with cep state %d\n", + sk->sk_state, cep->state); + fallthrough; case TCP_CLOSE: case TCP_CLOSE_WAIT: @@ -1303,7 +1323,7 @@ static void siw_cm_llp_state_change(struct sock *sk) static int kernel_bindconnect(struct socket *s, struct sockaddr *laddr, struct sockaddr *raddr, bool afonly) { - int rv, flags = 0; + int rv; size_t size = laddr->sa_family == AF_INET ? sizeof(struct sockaddr_in) : sizeof(struct sockaddr_in6); @@ -1322,7 +1342,18 @@ static int kernel_bindconnect(struct socket *s, struct sockaddr *laddr, if (rv < 0) return rv; - rv = kernel_connect(s, raddr, size, flags); + /* + * Yes, this is really O_NONBLOCK instead of + * SOCK_NONBLOCK. + * + * __sys_connect_file() passes + * sock->file->f_flags | file_flags to + * sock->ops->connect(). + * + * Also io_connect() from io_uring forces + * file_flags=O_NONBLOCK to __sys_connect_file(). + */ + rv = kernel_connect(s, raddr, size, O_NONBLOCK); return rv < 0 ? rv : 0; } @@ -1469,36 +1500,27 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) goto error; } - /* - * NOTE: For simplification, connect() is called in blocking - * mode. Might be reconsidered for async connection setup at - * TCP level. - */ rv = kernel_bindconnect(s, laddr, raddr, id->afonly); + if (rv == -EINPROGRESS) { + siw_dbg_qp(qp, "kernel_bindconnect: EINPROGRESS\n"); + rv = 0; + } if (rv != 0) { siw_dbg_qp(qp, "kernel_bindconnect: error %d\n", rv); goto error; } - if (siw_tcp_nagle == false) - tcp_sock_set_nodelay(s->sk); - - cep->state = SIW_EPSTATE_AWAIT_MPAREP; - rv = siw_send_mpareqrep(cep, cep->mpa.pdata, - cep->mpa.hdr.params.pd_len); /* - * Reset private data. + * The rest will be done by siw_connected() + * + * siw_cm_llp_state_change() will detect + * TCP_ESTABLISHED and schedules SIW_CM_WORK_CONNECTED, + * which will finally call siw_connected(). + * + * As siw_cm_llp_state_change() handles everything + * siw_cm_llp_data_ready() can be a noop for + * SIW_EPSTATE_CONNECTING. */ - if (cep->mpa.hdr.params.pd_len) { - cep->mpa.hdr.params.pd_len = 0; - kfree(cep->mpa.pdata); - cep->mpa.pdata = NULL; - } - - if (rv < 0) { - goto error; - } - siw_dbg_cep(cep, "[QP %u]: exit\n", qp_id(qp)); siw_cep_set_free(cep); return 0; @@ -1537,6 +1559,49 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) return rv; } +static void siw_connected(struct siw_cep *cep) +{ + struct siw_qp *qp = cep->qp; + struct socket *s = cep->sock; + int rv = -ECONNABORTED; + + /* + * already called with + * siw_cep_set_inuse(cep); + */ + + if (cep->state != SIW_EPSTATE_CONNECTING) + goto error; + + if (siw_tcp_nagle == false) + tcp_sock_set_nodelay(s->sk); + + cep->state = SIW_EPSTATE_AWAIT_MPAREP; + + rv = siw_send_mpareqrep(cep, cep->mpa.pdata, + cep->mpa.hdr.params.pd_len); + /* + * Reset private data. + */ + if (cep->mpa.hdr.params.pd_len) { + cep->mpa.hdr.params.pd_len = 0; + kfree(cep->mpa.pdata); + cep->mpa.pdata = NULL; + } + + if (rv < 0) { + goto error; + } + + siw_dbg_cep(cep, "[QP %u]: exit\n", qp_id(qp)); + return; + +error: + siw_dbg_cep(cep, "[QP %u]: exit, error %d\n", qp_id(qp), rv); + siw_qp_cm_drop(qp, 1); + return; +} + /* * siw_accept - Let SoftiWARP accept an RDMA connection request * diff --git a/drivers/infiniband/sw/siw/siw_cm.h b/drivers/infiniband/sw/siw/siw_cm.h index 8c59cb3e2868..c01bdc8e64ee 100644 --- a/drivers/infiniband/sw/siw/siw_cm.h +++ b/drivers/infiniband/sw/siw/siw_cm.h @@ -76,6 +76,7 @@ struct siw_cep { enum siw_work_type { SIW_CM_WORK_ACCEPT = 1, + SIW_CM_WORK_CONNECTED, SIW_CM_WORK_READ_MPAHDR, SIW_CM_WORK_CLOSE_LLP, /* close socket */ SIW_CM_WORK_PEER_CLOSE, /* socket indicated peer close */