From patchwork Wed Jun 3 00:59:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 11584747 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7A68C912 for ; Wed, 3 Jun 2020 01:00:43 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 62A572072F for ; Wed, 3 Jun 2020 01:00:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 62A572072F Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lustre-devel-bounces@lists.lustre.org Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 7B7CE21FB46; Tue, 2 Jun 2020 18:00:24 -0700 (PDT) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp3.ccs.ornl.gov (smtp3.ccs.ornl.gov [160.91.203.39]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 82B3D21F563 for ; Tue, 2 Jun 2020 18:00:09 -0700 (PDT) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp3.ccs.ornl.gov (Postfix) with ESMTP id 7FB506BB; Tue, 2 Jun 2020 21:00:02 -0400 (EDT) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 7E9F32C6; Tue, 2 Jun 2020 21:00:02 -0400 (EDT) From: James Simmons To: Andreas Dilger , Oleg Drokin , NeilBrown Date: Tue, 2 Jun 2020 20:59:53 -0400 Message-Id: <1591146001-27171-15-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1591146001-27171-1-git-send-email-jsimmons@infradead.org> References: <1591146001-27171-1-git-send-email-jsimmons@infradead.org> Subject: [lustre-devel] [PATCH 14/22] lnet: Force full discovery cycle X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Amir Shehata , Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" From: Amir Shehata There are scenarios where there could be a discrepancy between cached peer information and reality. In these cases what could end-up happening is incomplete interface information might be cached because one side determined that the peer didn't require a PUSH. This will lead to undesired MR behavior, where not all the interfaces are used for a period of time. Therefore, it is safer to always force a full discovery cycle: GET/PUSH to ensure both sides are up-to-date. In the NMR case, when discovery is turned off, make sure to flag discovery as complete to avoid stalling the state machine. WC-bug-id: https://jira.whamcloud.com/browse/LU-13477 Lustre-commit: 9bafd530d7858 ("LU-13477 lnet: Force full discovery cycle") Signed-off-by: Amir Shehata Reviewed-on: https://review.whamcloud.com/38322 Reviewed-by: Chris Horn Reviewed-by: Serguei Smirnov Reviewed-by: Oleg Drokin Signed-off-by: James Simmons --- net/lnet/lnet/peer.c | 40 ++++++++++++++++++++++++---------------- net/lnet/lnet/router.c | 5 +++-- 2 files changed, 27 insertions(+), 18 deletions(-) diff --git a/net/lnet/lnet/peer.c b/net/lnet/lnet/peer.c index 60749da..5869a20 100644 --- a/net/lnet/lnet/peer.c +++ b/net/lnet/lnet/peer.c @@ -1179,6 +1179,11 @@ struct lnet_peer_ni * lp = lpni->lpni_peer_net->lpn_peer; while (!lnet_peer_is_uptodate(lp)) { + spin_lock(&lp->lp_lock); + /* force a full discovery cycle */ + lp->lp_state |= LNET_PEER_FORCE_PING | LNET_PEER_FORCE_PUSH; + spin_unlock(&lp->lp_lock); + rc = lnet_discover_peer_locked(lpni, cpt, true); if (rc) goto out_decref; @@ -3043,6 +3048,19 @@ static int lnet_peer_push_failed(struct lnet_peer *lp) return rc ? rc : LNET_REDISCOVER_PEER; } +/* Mark the peer as discovered. */ +static int lnet_peer_discovered(struct lnet_peer *lp) +__must_hold(&lp->lp_lock) +{ + lp->lp_state |= LNET_PEER_DISCOVERED; + lp->lp_state &= ~(LNET_PEER_DISCOVERING | + LNET_PEER_REDISCOVER); + + CDEBUG(D_NET, "peer %s\n", libcfs_nid2str(lp->lp_primary_nid)); + + return 0; +} + /* Active side of push. */ static int lnet_peer_send_push(struct lnet_peer *lp) __must_hold(&lp->lp_lock) @@ -3056,6 +3074,12 @@ static int lnet_peer_send_push(struct lnet_peer *lp) /* Don't push to a non-multi-rail peer. */ if (!(lp->lp_state & LNET_PEER_MULTI_RAIL)) { lp->lp_state &= ~LNET_PEER_FORCE_PUSH; + /* if peer's NIDs are uptodate then peer is discovered */ + if (lp->lp_state & LNET_PEER_NIDS_UPTODATE) { + rc = lnet_peer_discovered(lp); + return rc; + } + return 0; } @@ -3147,22 +3171,6 @@ static void lnet_peer_discovery_error(struct lnet_peer *lp, int error) } /* - * Mark the peer as discovered. - */ -static int lnet_peer_discovered(struct lnet_peer *lp) -__must_hold(&lp->lp_lock) -{ - lp->lp_state |= LNET_PEER_DISCOVERED; - lp->lp_state &= ~(LNET_PEER_DISCOVERING | - LNET_PEER_REDISCOVER); - - CDEBUG(D_NET, "peer %s\n", libcfs_nid2str(lp->lp_primary_nid)); - - return 0; -} - - -/* * Discovering this peer is taking too long. Cancel any Ping or Push * that discovery is waiting on by unlinking the relevant MDs. The * lnet_discovery_event_handler() will proceed from here and complete diff --git a/net/lnet/lnet/router.c b/net/lnet/lnet/router.c index af50e51..903d027 100644 --- a/net/lnet/lnet/router.c +++ b/net/lnet/lnet/router.c @@ -1154,9 +1154,10 @@ bool lnet_router_checker_active(void) spin_unlock(&rtr->lp_lock); continue; } - /* make sure we actively discover the router */ + /* make sure we fully discover the router */ rtr->lp_state &= ~LNET_PEER_NIDS_UPTODATE; - rtr->lp_state |= LNET_PEER_RTR_DISCOVERY; + rtr->lp_state |= LNET_PEER_FORCE_PING | LNET_PEER_FORCE_PUSH | + LNET_PEER_RTR_DISCOVERY; spin_unlock(&rtr->lp_lock); /* find the peer_ni associated with the primary NID */