From patchwork Mon Nov 16 00:59:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11907009 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0A22B14B3 for ; Mon, 16 Nov 2020 01:00:33 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D24E82223C for ; Mon, 16 Nov 2020 01:00:32 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D24E82223C Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lustre-devel-bounces@lists.lustre.org Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 71BA721F9C9; Sun, 15 Nov 2020 17:00:32 -0800 (PST) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp3.ccs.ornl.gov (smtp3.ccs.ornl.gov [160.91.203.39]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id ACE3C21F8DB for ; Sun, 15 Nov 2020 17:00:09 -0800 (PST) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp3.ccs.ornl.gov (Postfix) with ESMTP id 57773223A; Sun, 15 Nov 2020 20:00:06 -0500 (EST) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 4F3668AEB; Sun, 15 Nov 2020 20:00:06 -0500 (EST) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Sun, 15 Nov 2020 19:59:40 -0500 Message-Id: <1605488401-981-8-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1605488401-981-1-git-send-email-jsimmons@infradead.org> References: <1605488401-981-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 07/28] lnet: o2iblnd: Don't retry indefinitely X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Amir Shehata , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Amir Shehata If peer is down don't retry indefinitely. Use the retry_count parameter to restrict the number of retries. After which the connection fails and error is propagated up. This prevents long timeouts when mounting a file system with nodes which might have their NIDs configured in the FS, but the nodes have been taken offline. WC-bug-id: https://jira.whamcloud.com/browse/LU-13972 Lustre-commit: 7c8ad11ef08f0f ("LU-13972 o2iblnd: Don't retry indefinitely") Signed-off-by: Amir Shehata Reviewed-on: https://review.whamcloud.com/39981 Reviewed-by: Serguei Smirnov Reviewed-by: Chris Horn Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/klnds/o2iblnd/o2iblnd.h | 2 ++ net/lnet/klnds/o2iblnd/o2iblnd_cb.c | 9 +++++++++ net/lnet/klnds/o2iblnd/o2iblnd_modparams.c | 2 +- 3 files changed, 12 insertions(+), 1 deletion(-) diff --git a/net/lnet/klnds/o2iblnd/o2iblnd.h b/net/lnet/klnds/o2iblnd/o2iblnd.h index 9a2fb42..9b2f043 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd.h +++ b/net/lnet/klnds/o2iblnd/o2iblnd.h @@ -626,6 +626,8 @@ struct kib_peer_ni { unsigned char ibp_races; /* # consecutive reconnection attempts to this peer_ni */ unsigned int ibp_reconnected; + /* number of total active retries */ + unsigned int ibp_retries; /* errno on closing this peer_ni */ int ibp_error; /* max map_on_demand */ diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c index b642162..9de733d 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_cb.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_cb.c @@ -2181,6 +2181,9 @@ static int kiblnd_map_tx(struct lnet_ni *ni, struct kib_tx *tx, /* connection established */ write_lock_irqsave(&kiblnd_data.kib_global_lock, flags); + /* reset retry count */ + peer_ni->ibp_retries = 0; + conn->ibc_last_send = ktime_get(); kiblnd_set_conn_state(conn, IBLND_CONN_ESTABLISHED); kiblnd_peer_alive(peer_ni); @@ -2631,6 +2634,11 @@ static int kiblnd_map_tx(struct lnet_ni *ni, struct kib_tx *tx, goto out; } + if (peer_ni->ibp_retries > *kiblnd_tunables.kib_retry_count) { + reason = "retry count exceeded due to no listener"; + goto out; + } + switch (why) { default: reason = "Unknown"; @@ -2688,6 +2696,7 @@ static int kiblnd_map_tx(struct lnet_ni *ni, struct kib_tx *tx, break; case IBLND_REJECT_INVALID_SRV_ID: + peer_ni->ibp_retries++; reason = "invalid service id"; break; } diff --git a/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c b/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c index 73ad22d..029c9fb 100644 --- a/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c +++ b/net/lnet/klnds/o2iblnd/o2iblnd_modparams.c @@ -93,7 +93,7 @@ static int retry_count = 5; module_param(retry_count, int, 0644); -MODULE_PARM_DESC(retry_count, "Retransmissions when no ACK received"); +MODULE_PARM_DESC(retry_count, "Number of times to retry connection operations"); static int rnr_retry_count = 6; module_param(rnr_retry_count, int, 0644);