From patchwork Wed Nov 20 08:29:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11253365 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 07BDB186D for ; Wed, 20 Nov 2019 08:29:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DD45422312 for ; Wed, 20 Nov 2019 08:29:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="eb0rTtu/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727736AbfKTI3S (ORCPT ); Wed, 20 Nov 2019 03:29:18 -0500 Received: from us-smtp-1.mimecast.com ([207.211.31.81]:55477 "EHLO us-smtp-delivery-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726038AbfKTI3S (ORCPT ); Wed, 20 Nov 2019 03:29:18 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1574238557; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0434QUXjPvIaHT1EaIkqlZLBAS2xooOlYe242h7NpmM=; b=eb0rTtu/9wlUN77jUTQ0vHrsLqdiwIexVtoxCOpE1ydwMEEm7+roPbPExz5XdqAFaZ6Jzg jE9IXrGNU8uEat3boyecPbk32W7SZ/rDRfkoo9l03WKJIDiX7gtXbjoRSCwftClnDeONOn TgCbLqc5b3ERV+XcUx//xeO9AqU3u5E= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-413-4CqXei4_O6yhjlcEhd37yg-1; Wed, 20 Nov 2019 03:29:16 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2FD36100550E; Wed, 20 Nov 2019 08:29:15 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-58.pek2.redhat.com [10.72.12.58]) by smtp.corp.redhat.com (Postfix) with ESMTP id 8B41367E52; Wed, 20 Nov 2019 08:29:12 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org Cc: sage@redhat.com, idryomov@gmail.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH 1/3] mdsmap: add more debug info when decoding Date: Wed, 20 Nov 2019 03:29:00 -0500 Message-Id: <20191120082902.38666-2-xiubli@redhat.com> In-Reply-To: <20191120082902.38666-1-xiubli@redhat.com> References: <20191120082902.38666-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-MC-Unique: 4CqXei4_O6yhjlcEhd37yg-1 X-Mimecast-Spam-Score: 0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li Show the laggy state. Signed-off-by: Xiubo Li --- fs/ceph/mdsmap.c | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/fs/ceph/mdsmap.c b/fs/ceph/mdsmap.c index aeec1d6e3769..471bac335fae 100644 --- a/fs/ceph/mdsmap.c +++ b/fs/ceph/mdsmap.c @@ -158,6 +158,7 @@ struct ceph_mdsmap *ceph_mdsmap_decode(void **p, void *end) void *pexport_targets = NULL; struct ceph_timespec laggy_since; struct ceph_mds_info *info; + bool laggy; ceph_decode_need(p, end, sizeof(u64) + 1, bad); global_id = ceph_decode_64(p); @@ -190,6 +191,7 @@ struct ceph_mdsmap *ceph_mdsmap_decode(void **p, void *end) if (err) goto corrupt; ceph_decode_copy(p, &laggy_since, sizeof(laggy_since)); + laggy = laggy_since.tv_sec != 0 || laggy_since.tv_nsec != 0; *p += sizeof(u32); ceph_decode_32_safe(p, end, namelen, bad); *p += namelen; @@ -207,10 +209,11 @@ struct ceph_mdsmap *ceph_mdsmap_decode(void **p, void *end) *p = info_end; } - dout("mdsmap_decode %d/%d %lld mds%d.%d %s %s\n", + dout("mdsmap_decode %d/%d %lld mds%d.%d %s %s%s\n", i+1, n, global_id, mds, inc, ceph_pr_addr(&addr), - ceph_mds_state_name(state)); + ceph_mds_state_name(state), + laggy ? "(laggy)" : ""); if (mds < 0 || state <= 0) continue; @@ -230,8 +233,7 @@ struct ceph_mdsmap *ceph_mdsmap_decode(void **p, void *end) info->global_id = global_id; info->state = state; info->addr = addr; - info->laggy = (laggy_since.tv_sec != 0 || - laggy_since.tv_nsec != 0); + info->laggy = laggy; info->num_export_targets = num_export_targets; if (num_export_targets) { info->export_targets = kcalloc(num_export_targets, @@ -355,6 +357,8 @@ struct ceph_mdsmap *ceph_mdsmap_decode(void **p, void *end) m->m_damaged = false; } bad_ext: + dout("mdsmap_decode m_enabled: %d, m_damaged: %d, m_num_laggy: %d\n", + !!m->m_enabled, !!m->m_damaged, m->m_num_laggy); *p = end; dout("mdsmap_decode success epoch %u\n", m->m_epoch); return m; From patchwork Wed Nov 20 08:29:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11253367 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 38EC41390 for ; Wed, 20 Nov 2019 08:29:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1082822312 for ; Wed, 20 Nov 2019 08:29:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="iJFnLXBB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727809AbfKTI3X (ORCPT ); Wed, 20 Nov 2019 03:29:23 -0500 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:55227 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726038AbfKTI3W (ORCPT ); Wed, 20 Nov 2019 03:29:22 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1574238561; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Dyxk4x5q2SY+ocukli6qC9/tLSscgKWf/oX8Rxgu6x4=; b=iJFnLXBBjAjZpSKWX7bSeCr2pzoGnA9L4xryjNJeTIiKM7JK4dNzVs0OqqlySMEvMxocK8 OuROrogQYPy4I0jI14LfMng/8IdKIaezJINqgNUsJMKg8MyEw9pJf2yqgAYK34Eydwb3k1 +AQ3SivW8oPPIkfVNAjAqa1L6l2/4aM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-31-K5HvO55MPkKvodHE_88qhw-1; Wed, 20 Nov 2019 03:29:19 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 541E4107ACC4; Wed, 20 Nov 2019 08:29:18 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-58.pek2.redhat.com [10.72.12.58]) by smtp.corp.redhat.com (Postfix) with ESMTP id B1BAF67E52; Wed, 20 Nov 2019 08:29:15 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org Cc: sage@redhat.com, idryomov@gmail.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH 2/3] mdsmap: fix mdsmap cluster available check based on laggy number Date: Wed, 20 Nov 2019 03:29:01 -0500 Message-Id: <20191120082902.38666-3-xiubli@redhat.com> In-Reply-To: <20191120082902.38666-1-xiubli@redhat.com> References: <20191120082902.38666-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-MC-Unique: K5HvO55MPkKvodHE_88qhw-1 X-Mimecast-Spam-Score: 0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li In case the max_mds > 1 in MDS cluster and there is no any standby MDS and all the max_mds MDSs are in up:active state, if one of the up:active MDSs is dead, the m->m_num_laggy in kclient will be 1. Then the mount will fail without considering other healthy MDSs. Only when all the MDSs in the cluster are laggy will treat the cluster as not be available. Signed-off-by: Xiubo Li --- fs/ceph/mdsmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/fs/ceph/mdsmap.c b/fs/ceph/mdsmap.c index 471bac335fae..8b4f93e5b468 100644 --- a/fs/ceph/mdsmap.c +++ b/fs/ceph/mdsmap.c @@ -396,7 +396,7 @@ bool ceph_mdsmap_is_cluster_available(struct ceph_mdsmap *m) return false; if (m->m_damaged) return false; - if (m->m_num_laggy > 0) + if (m->m_num_laggy == m->m_num_mds) return false; for (i = 0; i < m->m_num_mds; i++) { if (m->m_info[i].state == CEPH_MDS_STATE_ACTIVE) From patchwork Wed Nov 20 08:29:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xiubo Li X-Patchwork-Id: 11253369 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 601E0188B for ; Wed, 20 Nov 2019 08:29:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 41A6322312 for ; Wed, 20 Nov 2019 08:29:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="LGdutJvR" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727868AbfKTI3Z (ORCPT ); Wed, 20 Nov 2019 03:29:25 -0500 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120]:34852 "EHLO us-smtp-1.mimecast.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1726038AbfKTI3Z (ORCPT ); Wed, 20 Nov 2019 03:29:25 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1574238563; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GJEPnd+ByM4bOAId6/vd9Lxq6m1pDU3luZ/aPT7Lp80=; b=LGdutJvRZV5zz8uDwmW4L8Y2dNh/A1KR/Qi0PAGJvGSy7WNHO8IrjYv+/VgY0dJHP06SRW kn2zOR4DTJ9ZyabPF4aaCaXzmdsd17+qnnBDzLs84tl6JgRmc63DyZcrrtJNk++M3jBLJX GVc4bt2lrWnzkaL9Uryw0R09JYURugU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-297-Z6k2klcpMtKed3GAkX85CA-1; Wed, 20 Nov 2019 03:29:22 -0500 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 7D24D1005512; Wed, 20 Nov 2019 08:29:21 +0000 (UTC) Received: from localhost.localdomain (ovpn-12-58.pek2.redhat.com [10.72.12.58]) by smtp.corp.redhat.com (Postfix) with ESMTP id D914367E45; Wed, 20 Nov 2019 08:29:18 +0000 (UTC) From: xiubli@redhat.com To: jlayton@kernel.org Cc: sage@redhat.com, idryomov@gmail.com, zyan@redhat.com, pdonnell@redhat.com, ceph-devel@vger.kernel.org, Xiubo Li Subject: [PATCH 3/3] mdsmap: only choose one MDS who is in up:active state without laggy Date: Wed, 20 Nov 2019 03:29:02 -0500 Message-Id: <20191120082902.38666-4-xiubli@redhat.com> In-Reply-To: <20191120082902.38666-1-xiubli@redhat.com> References: <20191120082902.38666-1-xiubli@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-MC-Unique: Z6k2klcpMtKed3GAkX85CA-1 X-Mimecast-Spam-Score: 0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org From: Xiubo Li Even the MDS is in up:active state, but it also maybe laggy. Here will skip the laggy MDSs. Signed-off-by: Xiubo Li --- fs/ceph/mds_client.c | 6 ++++-- fs/ceph/mdsmap.c | 13 +++++++++---- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index 82a929084671..a4e7026aaec9 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -972,7 +972,8 @@ static int __choose_mds(struct ceph_mds_client *mdsc, frag.frag, mds, (int)r, frag.ndist); if (ceph_mdsmap_get_state(mdsc->mdsmap, mds) >= - CEPH_MDS_STATE_ACTIVE) + CEPH_MDS_STATE_ACTIVE && + !ceph_mdsmap_is_laggy(mdsc->mdsmap, mds)) goto out; } @@ -987,7 +988,8 @@ static int __choose_mds(struct ceph_mds_client *mdsc, "frag %u mds%d (auth)\n", inode, ceph_vinop(inode), frag.frag, mds); if (ceph_mdsmap_get_state(mdsc->mdsmap, mds) >= - CEPH_MDS_STATE_ACTIVE) + CEPH_MDS_STATE_ACTIVE && + !ceph_mdsmap_is_laggy(mdsc->mdsmap, mds)) goto out; } } diff --git a/fs/ceph/mdsmap.c b/fs/ceph/mdsmap.c index 8b4f93e5b468..098669e6f1e4 100644 --- a/fs/ceph/mdsmap.c +++ b/fs/ceph/mdsmap.c @@ -13,6 +13,7 @@ #include "super.h" +#define CEPH_MDS_IS_READY(i) (m->m_info[i].state > 0 && !m->m_info[i].laggy) /* * choose a random mds that is "up" (i.e. has a state > 0), or -1. @@ -23,12 +24,16 @@ int ceph_mdsmap_get_random_mds(struct ceph_mdsmap *m) int i, j; /* special case for one mds */ - if (1 == m->m_num_mds && m->m_info[0].state > 0) - return 0; + if (1 == m->m_num_mds && m->m_info[0].state > 0) { + if (m->m_info[0].laggy) + return -1; + else + return 0; + } /* count */ for (i = 0; i < m->m_num_mds; i++) - if (m->m_info[i].state > 0) + if (CEPH_MDS_IS_READY(i)) n++; if (n == 0) return -1; @@ -36,7 +41,7 @@ int ceph_mdsmap_get_random_mds(struct ceph_mdsmap *m) /* pick */ n = prandom_u32() % n; for (j = 0, i = 0; i < m->m_num_mds; i++) { - if (m->m_info[i].state > 0) + if (CEPH_MDS_IS_READY(i)) j++; if (j > n) break;