From patchwork Thu Dec 12 11:01:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287921 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8049930 for ; Thu, 12 Dec 2019 11:02:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C5CB62077B for ; Thu, 12 Dec 2019 11:02:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DUksATwb" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728887AbfLLLCO (ORCPT ); Thu, 12 Dec 2019 06:02:14 -0500 Received: from mail-pf1-f194.google.com ([209.85.210.194]:45944 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCN (ORCPT ); Thu, 12 Dec 2019 06:02:13 -0500 Received: by mail-pf1-f194.google.com with SMTP id 2so563071pfg.12 for ; Thu, 12 Dec 2019 03:02:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Fktl9a826M71BPZIK+iiAqg6eAaRJhlG243Z84C4xoc=; b=DUksATwbwuOFMaPs5Bo7ckD6bwY85Ev+3JD5rUUKo7t83+yTRYO9/mHHT8ANCr6bdR nFs9yr9GdXlFbddHY5Ng56dX7YVCsqSNAx56LDxHKCuuxEkLPxH3A3yySC3qcBM0FbQT umUf8L5Z8Qb1VItdniBDlc3h1PeaHRrb223+46kcR4Za81ifWb9EkUCJcg4p/LawkMlX t4tm10uXNCzDTcc9YBxJqBrfnU/tyngIdTaeCh1U3oNirspsYJ6qjnj2Wc9TLtKCBV7f a2t+nXD3haCSbIpplLw2F9isMRMXBrRfYgmTnOIgJEvTvtoLMgN7HcK7pZBRxisk8goi rVZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Fktl9a826M71BPZIK+iiAqg6eAaRJhlG243Z84C4xoc=; b=H9guqZDhlZP+XhhMuODLiJUjxDMr81hyVRZDuruY4wsjHEZ9rUif+eo+30o2xolBf2 ugryU6ok4hfFUz4rACQPG4noFimlo0d24f72st8Tof6tiW87R27mGMk3MaIj8lAxSHo6 ziKW5/K3l+SG6JUOotvVAE4ZrB9BX+Xj8QIL8vznOwQkiRmf7XIDNjQgGq5WRWoYBiWm rBbcbWX3HK70Y92JY4hab7mdYJe2JjAq51wkLdBbYDm0vhRVWAINLOhJ5/pKThy03M7R i9vy9LMUDggA9LAgZGe1IDzgE6ItEEh2Z4UEGTtyEjWk5F3wVOlaIdMxETo0o1GWuqyB lOrg== X-Gm-Message-State: APjAAAX0ZxJI7jdMYZpjd/5t8D4yiduBVU5BpbfLDMLLYIiSiCL+SUkP GvbUh9jz1hziSK/YeC0VcfqtP3Y6gzs= X-Google-Smtp-Source: APXvYqwJGLJSiaS9gH10+RUThOtsGHATjBDw/uAf+MUIplTI1tiOqKfZMJ3pYmviTKQZSAHkgl/7GQ== X-Received: by 2002:a62:f842:: with SMTP id c2mr9131517pfm.104.1576148532749; Thu, 12 Dec 2019 03:02:12 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.11 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:12 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 01/11] btrfs-progs: misc-tests/034: reload btrfs module before running failure_recovery Date: Thu, 12 Dec 2019 19:01:54 +0800 Message-Id: <20191212110204.11128-2-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue One reload_btrfs is lost, add it. Fixes: 0de2e22ad226 ("btrfs-progs: tests: Add tests for changing fsid feature") Signed-off-by: Su Yue Reviewed-by: Nikolay Borisov --- tests/misc-tests/034-metadata-uuid/test.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/misc-tests/034-metadata-uuid/test.sh b/tests/misc-tests/034-metadata-uuid/test.sh index 6ac55b1cacfa..ff51bf22fadf 100755 --- a/tests/misc-tests/034-metadata-uuid/test.sh +++ b/tests/misc-tests/034-metadata-uuid/test.sh @@ -211,11 +211,10 @@ failure_recovery "./disk1.raw.xz" "./disk2.raw.xz" check_inprogress_flag reload_btrfs failure_recovery "./disk2.raw.xz" "./disk1.raw.xz" check_inprogress_flag -reload_btrfs - # disk4 contains an image in with the in-progress flag set and disk 3 is part # of the same filesystem but has both METADATA_UUID incompat and a new # metadata uuid set. So disk 3 must always take precedence +reload_btrfs failure_recovery "./disk3.raw.xz" "./disk4.raw.xz" check_completed reload_btrfs failure_recovery "./disk4.raw.xz" "./disk3.raw.xz" check_completed @@ -224,6 +223,7 @@ failure_recovery "./disk4.raw.xz" "./disk3.raw.xz" check_completed # than once, disk6 on the other hand is member of the same filesystem but # hasn't completed its last change. Thus it has both the FSID_CHANGING flag set # and METADATA_UUID flag set. +reload_btrfs failure_recovery "./disk5.raw.xz" "./disk6.raw.xz" check_multi_fsid_change reload_btrfs failure_recovery "./disk6.raw.xz" "./disk5.raw.xz" check_multi_fsid_change From patchwork Thu Dec 12 11:01:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287923 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 047A5930 for ; Thu, 12 Dec 2019 11:02:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D7419214D8 for ; Thu, 12 Dec 2019 11:02:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hKLjfovp" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728912AbfLLLCP (ORCPT ); Thu, 12 Dec 2019 06:02:15 -0500 Received: from mail-pf1-f196.google.com ([209.85.210.196]:36304 "EHLO mail-pf1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCP (ORCPT ); Thu, 12 Dec 2019 06:02:15 -0500 Received: by mail-pf1-f196.google.com with SMTP id x184so580080pfb.3 for ; Thu, 12 Dec 2019 03:02:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iploGeyQw0n2X+UztIXgwkw6IT4Eye3P4U1NgSFdGHQ=; b=hKLjfovpphuCMNEDbgmT/W1gJJUOq/Ef+ChVUWIyFnsbggF20SV70bGNsmd7Jg0JA5 ikaopSs6TGaCB4Zi6pFiYQXx6iLIhq50MC0ZzEEKbVKcGQou5iXnS4N2tSRrMYjMUJ+6 lppvlyLD9aIbYSLe44aoODu74cOLwx5xkGjmVv8GBAF+qr/9anCthSYTfSUGJ5jgtxxB ZgrJH0N34Fg1Sw7b/zKvLqvLkYmWLCVNsbeYGELvqX6WViM5AHeATHbvfNck2B5eEAx4 QdgdxZQt0ACrDs2kiY/HLXpRwrtduvgruDaFNArnGA+BBK3TvhNSqRaP7KJY2qWDlbl7 Xtww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iploGeyQw0n2X+UztIXgwkw6IT4Eye3P4U1NgSFdGHQ=; b=fj7+1b3yxB6iqJR+0SrYaFKsaYx4ixBYh1WUOq9+GDCWBmMoTBA3f6Wq+8YorE6u+F grm0bgeSdA86AT9l5lNPDxpHGPyXpUbgHtFIT+Vwb3NgiN28vQCTEPSq7TeOp1C9+a9D JXz9Uow1k/CJN57gtN300tBesrMjE1B/WwE9TwS6tCnlqEVs1Qthg5qhWm8z/JB8fnKD hh3KRcjGbDBnjZWt4VWN457jL9rVDO5/jjoEvfxR1hVDb1c8gUXspC0R+3CqXX/eHeth QSfvoKj0PLHf4RL98ckcyyZrc/jVe7ne0AHqPp9/bw7eJQpnE8+hu8TP/YrEN0W7k4zf 1/8Q== X-Gm-Message-State: APjAAAVZ/eDAvl82OWnkqMklxHsoMzn3L8OWSVjOKqlSJQot9qMtv07w Z000imG+HYDMM1RA9uUjJiKdybcnO9c= X-Google-Smtp-Source: APXvYqxM4vWmLIRHeG+D+uxk06L0hMmodDnb+H/X6++zxv+/4dMdwjOkIVzs8jRECnBRr8lmKq9z4A== X-Received: by 2002:a62:ac03:: with SMTP id v3mr8972010pfe.17.1576148534458; Thu, 12 Dec 2019 03:02:14 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.12 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:14 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 02/11] btrfs-progs: misc-tests/034: mount the second device if first device mount failed Date: Thu, 12 Dec 2019 19:01:55 +0800 Message-Id: <20191212110204.11128-3-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue The 034 test may fail to mount, and dmesg says open_ctree() failed due to device missing. The partly work flow is step1 loop1 = losetup image1 step2 loop2 = losetup image2 setp3 mount loop1 The dmesg says the loop2 device is missing. It's possible and known that while step3 is in open_ctree() and fs_devices->opened is nonzero, loop2 device has not been added into the fs_devces. Then read_one_chunk() reports that loop2 is missing. The solution for this test is try to mount loop2 if loop mount failed. Fixes: 0de2e22ad226 ("btrfs-progs: tests: Add tests for changing fsid feature") Signed-off-by: Su Yue --- tests/misc-tests/034-metadata-uuid/test.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/misc-tests/034-metadata-uuid/test.sh b/tests/misc-tests/034-metadata-uuid/test.sh index ff51bf22fadf..5fe553705fcf 100755 --- a/tests/misc-tests/034-metadata-uuid/test.sh +++ b/tests/misc-tests/034-metadata-uuid/test.sh @@ -173,7 +173,9 @@ failure_recovery() { loop2=$(run_check_stdout $SUDO_HELPER losetup --find --show "$image2") # Mount and unmount, on trans commit all disks should be consistent - run_check $SUDO_HELPER mount "$loop1" "$TEST_MNT" + run_mayfail $SUDO_HELPER mount "$loop1" "$TEST_MNT" + [ $? -ne 0 ] && run_check $SUDO_HELPER mount "$loop2" "$TEST_MNT" + run_check $SUDO_HELPER umount "$TEST_MNT" # perform any specific check From patchwork Thu Dec 12 11:01:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287925 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1E45930 for ; Thu, 12 Dec 2019 11:02:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A139C214D8 for ; Thu, 12 Dec 2019 11:02:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Ic8+0gbx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728961AbfLLLCR (ORCPT ); Thu, 12 Dec 2019 06:02:17 -0500 Received: from mail-pj1-f65.google.com ([209.85.216.65]:46670 "EHLO mail-pj1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCR (ORCPT ); Thu, 12 Dec 2019 06:02:17 -0500 Received: by mail-pj1-f65.google.com with SMTP id z21so873711pjq.13 for ; Thu, 12 Dec 2019 03:02:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WZkU7GiVRo1EbBkkcon+70ZfebtsOT9k8iNA7sNJh8c=; b=Ic8+0gbxOO8oKbJ7TM34N8OlJKykGZxiDxVCKvhubtncRU+AH6izNvXS1Yvq38avoc xe72TxsGIsClJ5G08aveejg3JNPJUK3vVvQBUnGz5oZVEiIJM3oO+513nu2KUiUffuoZ SSPWvBj+9/kvNzLtVB2Av7Hqr8bVx3h9oBGA758+yA4/RXArGjjreXV2FvB8aRL6n7Ye ei/Hd4klj8mU/em0Z++DEu/eWlR3gYrHG9IC/loAdxn9zGPdEHbPqtoa6BudCWTPKRDr LBPyhZWhuavpCdPIzbaoaEeUWSwteSsZZORAM6GIMOdlTtBzFYgXvo2JfyDF73c0Oj/s wcoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WZkU7GiVRo1EbBkkcon+70ZfebtsOT9k8iNA7sNJh8c=; b=piTTAlnPmAKkg1W0Vm81pgDBCMYoQY90SekDjccb4yF860U414mC7MQpKyLUvZT9vc TD7jy/NW9FenZkRMwEKjtI4VFWACnPfHpXQpOPufsOmGdb/6FkSdT9ZBXMf1aXqARNVq nOLqT4LPwDrWPle/w8tN6dJL/aUX6fPovIROUknNQ4sRUi4iMUy4HZSNJf0+x4Yc1guh LzrvKqoP1DsxxGxFH6ztHji1DLP5ZS8DS4Vf2mAt8K05fZ8ER3l6mY6MhfWAPX2NaCxe zlAqKw/55jYfBaW0DzKSGgNNOQstHW/GY05m8P96OJIfvFLLYyF3ILuHtHRMl/0Gb21z 1NwA== X-Gm-Message-State: APjAAAXAwolgP2jaSjRRPmNjKlCUkaCfUcp3+af5ixvU3oquAyzLxpWG u32weqgpp4CKVm9rqgLKZyG9BBT2rM8= X-Google-Smtp-Source: APXvYqwrbTV23CSeBWJzfBcyEmgtpEAR6PD9vOYfCtyjXtxRxKN7QUWARIselUtNwJH9cK7M4hOENg== X-Received: by 2002:a17:902:aa04:: with SMTP id be4mr8584673plb.311.1576148536356; Thu, 12 Dec 2019 03:02:16 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:15 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 03/11] btrfs-progs: metadata_uuid: add new member btrfs_fs_devices::fsid_change Date: Thu, 12 Dec 2019 19:01:56 +0800 Message-Id: <20191212110204.11128-4-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue Like what is done in kernel, we need extra information for split-brain cases. The new member btrfs_fs_devices::fsid_change records whether the devices are with FSID_CHANGING_V2 flag. The existing btrfs_fs_devices::last_transid can be used like the last_generation field in kernel. It records the highest generation of the fs_devices. Signed-off-by: Su Yue --- volumes.c | 4 +++- volumes.h | 1 + 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/volumes.c b/volumes.c index 143164f02ac0..88e926e98d5b 100644 --- a/volumes.c +++ b/volumes.c @@ -214,7 +214,8 @@ static int device_list_add(const char *path, u64 found_transid = btrfs_super_generation(disk_super); bool metadata_uuid = (btrfs_super_incompat_flags(disk_super) & BTRFS_FEATURE_INCOMPAT_METADATA_UUID); - + bool fsid_change_in_progress = (btrfs_super_flags(disk_super) & + BTRFS_SUPER_FLAG_CHANGING_FSID_V2); if (metadata_uuid) fs_devices = find_fsid(disk_super->fsid, disk_super->metadata_uuid); @@ -238,6 +239,7 @@ static int device_list_add(const char *path, fs_devices->latest_devid = devid; fs_devices->latest_trans = found_transid; fs_devices->lowest_devid = (u64)-1; + fs_devices->fsid_change = fsid_change_in_progress; device = NULL; } else { device = find_device(fs_devices, devid, diff --git a/volumes.h b/volumes.h index 41574f21dd23..1c734b515346 100644 --- a/volumes.h +++ b/volumes.h @@ -72,6 +72,7 @@ struct btrfs_device { struct btrfs_fs_devices { u8 fsid[BTRFS_FSID_SIZE]; /* FS specific uuid */ u8 metadata_uuid[BTRFS_FSID_SIZE]; /* FS specific uuid */ + bool fsid_change; /* the device with this id has the most recent copy of the super */ u64 latest_devid; From patchwork Thu Dec 12 11:01:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287927 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B51F7138D for ; Thu, 12 Dec 2019 11:02:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 91E752077B for ; Thu, 12 Dec 2019 11:02:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Q3gA68Ga" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728964AbfLLLCT (ORCPT ); Thu, 12 Dec 2019 06:02:19 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:44245 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCT (ORCPT ); Thu, 12 Dec 2019 06:02:19 -0500 Received: by mail-pl1-f194.google.com with SMTP id az3so432953plb.11 for ; Thu, 12 Dec 2019 03:02:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8AIHIwYWl7e/QhoTB8ocdp1XdiU5kIZiWIADqz3lybQ=; b=Q3gA68GaL0sgrRvGq+C9kI/SwJbx5zMSO+tiEHKZ9NyMf246+K3O4hrrj8qkbqbYpn puzAB+5go948C0IWZ210EP4w0Vz2lwbzfvnwKf/Su4YWwnKlomuBlQKccnrqLay6zsGv S98W4adtqgQYH5bNsgYZ/jXZG3OHSWgelMK+vxQOC/eJLlbmpkt+UE4xVdSSaSIERsO+ UJE1tBI4pU7LyX1uiW7KxcxWjpE9kflvJasHyMQuDgZjo6WKhGTaSW0wJAW2gU61IFYC c5G34phwB0S8bL+DYmVctp6gqOYGm5e20qCYM+b060x0VFy8xnWeCu1VKa1xxGuTcLG8 mwjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8AIHIwYWl7e/QhoTB8ocdp1XdiU5kIZiWIADqz3lybQ=; b=op0xazDQ/OmNZt/IYqAv33yGN8+pTb6KVLR3muVAwI2uEeXCUU9vCTtLA+XQKE7i2i BRrTX4C40SEeXjXMXm9ABkJGkaz9ootR73kSOmWLXuKum8IZHFFbNFuBYaRrQLos9jwy geP1j0elMAA68t2CsVRwVBeYGPZBCsIx+clmPfOlmgDuWXXsyuXJbmQX+FXTDnxCIfHF T/NsGLByCIWX6Lw31xJgyfxR2B3mKXnXBhsHMPbqfZKAOpJdgFJBJoZ/GkW3/y/I2lj2 pkGoWC/+ZYgG5JAexqSmd0P/SNhqSnojC3wNXSjGC8QGfSh4nOFc3HHWju2o+ciLLnGN QXuw== X-Gm-Message-State: APjAAAUjKhJwKioHMHvFS5ck1BWdpO0wtPAT5KORQZmO1SK35SBdoYmc nXpzjPj0jmXQ/FBdTt6c1yK/zQ+vwXo= X-Google-Smtp-Source: APXvYqzNXwfY3uyTTxAjE+sHz65hA0c/4k67iEZAbRXkblzG0Rts2jRxivdHeAvpH4Krk2QV1N620Q== X-Received: by 2002:a17:902:67:: with SMTP id 94mr8964696pla.241.1576148538514; Thu, 12 Dec 2019 03:02:18 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.16 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:18 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 04/11] btrfs-progs: handle split-brain scenario for scanned changing device without INCOMPAT_METADATA_UUID Date: Thu, 12 Dec 2019 19:01:57 +0800 Message-Id: <20191212110204.11128-5-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue For a scanned changing device without INCOMPAT_METADATA_UUID, the situation may be a) The scanned failed to be pull into disk, and there are successful synced devices which is already in fs_devices->devices. So their fsid and metadata_uuid must differ. Also since the scanned device is without INCOMPAT_METADATA_UUID, so fs_device->metadata_uuid must equals disk_super->fs_id. b) The scanned device is in the newest state. There are some old devices failed to be with CHANGING_FSID_V2 or some devices just like the device. Signed-off-by: Su Yue --- volumes.c | 32 ++++++++++++++++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/volumes.c b/volumes.c index 88e926e98d5b..bc1726975ce4 100644 --- a/volumes.c +++ b/volumes.c @@ -205,18 +205,46 @@ static struct btrfs_fs_devices *find_fsid(u8 *fsid, u8 *metadata_uuid) return NULL; } +/* + * Handle scanned device which has CHANGING_FSID_V2 set, it might belong to + * either a filesystem which has disks with completed fsid change or it might + * belong to fs with no UUID changes in effect, handle both. + */ +static struct btrfs_fs_devices *find_fsid_inprogress( + struct btrfs_super_block *disk_super) +{ + struct btrfs_fs_devices *fs_devices; + + list_for_each_entry(fs_devices, &fs_uuids, list) { + if (memcmp(fs_devices->metadata_uuid, fs_devices->fsid, + BTRFS_FSID_SIZE) != 0 && + memcmp(fs_devices->metadata_uuid, disk_super->fsid, + BTRFS_FSID_SIZE) == 0 && !fs_devices->fsid_change) { + return fs_devices; + } + } + + return find_fsid(disk_super->fsid, NULL); +} + static int device_list_add(const char *path, struct btrfs_super_block *disk_super, u64 devid, struct btrfs_fs_devices **fs_devices_ret) { struct btrfs_device *device; - struct btrfs_fs_devices *fs_devices; + struct btrfs_fs_devices *fs_devices = NULL; u64 found_transid = btrfs_super_generation(disk_super); bool metadata_uuid = (btrfs_super_incompat_flags(disk_super) & BTRFS_FEATURE_INCOMPAT_METADATA_UUID); bool fsid_change_in_progress = (btrfs_super_flags(disk_super) & BTRFS_SUPER_FLAG_CHANGING_FSID_V2); - if (metadata_uuid) + + if (fsid_change_in_progress) { + if (!metadata_uuid) + fs_devices = find_fsid_inprogress(disk_super); + } + + if (metadata_uuid && !fs_devices) fs_devices = find_fsid(disk_super->fsid, disk_super->metadata_uuid); else From patchwork Thu Dec 12 11:01:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287929 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 749B3138D for ; Thu, 12 Dec 2019 11:02:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5262A2077B for ; Thu, 12 Dec 2019 11:02:23 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jD2mpEzN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728967AbfLLLCW (ORCPT ); Thu, 12 Dec 2019 06:02:22 -0500 Received: from mail-pg1-f195.google.com ([209.85.215.195]:37415 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCW (ORCPT ); Thu, 12 Dec 2019 06:02:22 -0500 Received: by mail-pg1-f195.google.com with SMTP id q127so978358pga.4 for ; Thu, 12 Dec 2019 03:02:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=gRbLyKi3H9G0lv79P5y0hQtzSz2uUw0jMqN2LdizlCY=; b=jD2mpEzN14MLzulB4ar0SPobl1qIyT5JyNb+mb/yam+dTnskF0GF/YsA39wIeIWMWn Wka8EFB7yxaJvcdhcO3V9yDbqIC5FgdxWTp7we/SKIXl4+luncP2cZsGq2EOAymvDt41 zMiksfIqOK15RCqM64teAgDkFiQXNdq2xW0XkvFx+h6JobkfYyEkg9R6TZWcoU1dPQVZ roeCGHLhuktjw7brvn4yIAWFK30AMySB1rSidUI28tUdcpgeuCfDtnCSGBEO0blpjTJY o+QJ4Tn/OFw1LV1/nRTpr1blDOfb2sfX2WSSaZgpbbid1ZiZY9BWOywZcFZ8bZGtjykg mGTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gRbLyKi3H9G0lv79P5y0hQtzSz2uUw0jMqN2LdizlCY=; b=g3skNEHxbOxOu+lsChXlJCaQNjIKr/XO/WZxVIOSZptFMg+mynsPrtoZ71/kaKSnTF /gUxxTdHrcp3NrBYtk1Xt/3EV+Fizx8+/8kBzXo49aex7C1ReYlcTC2YfAXcQgqt8fMZ +PW0EyzgOii2Mafy1ztaZS7hGCboXePDE9IOIpLiyKZ+f08XnC/b3QFFx9xR9X1lnKr3 kiurNzQfu3m9vnURWOLb0kW/RCfsTpCw1EVMrMp2KRL4GAgR6DQNzmklkxOzxBo4GK+0 yDaHTR7nq9ecN+su3h2wVMrSBP4jxg979rJlzXgwjkZc5ucJ66ZfRrQOPBDcsNFId/vo TbjA== X-Gm-Message-State: APjAAAVeweUYgKh465628wRYyXyKyhOoNb4ddlfeT+yTL8E/LuVtYwWy vh+WiFTwc+P6J29pBknqcGg/lhUV508= X-Google-Smtp-Source: APXvYqwWPsvCuqAr9Egt5TW0gXvbJFStm/xrgKESI8t2qTvOEMelyv9s1P13AxycIBFgKQDqoqaLUQ== X-Received: by 2002:a65:644b:: with SMTP id s11mr9456243pgv.332.1576148541037; Thu, 12 Dec 2019 03:02:21 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:20 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 05/11] btrfs-progs: handle split-brain scenario for scanned changing device with INCOMPAT_METADATA_UUID Date: Thu, 12 Dec 2019 19:01:58 +0800 Message-Id: <20191212110204.11128-6-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue For a scanned changing device with INCOMPAT_METADATA_UUID, which means its fsid and metadata_uuid differs, then situation can be a) The final changed devices are still own the INCOMPAT_METADATA_UUID feature. So their metadata_uuid is same as the device's, but fsids must differ. b) The new fsid to be changing is same as the scanned device's metadata_uuid. So the successful synced devices are like some never changed. fs_devices's fsid is same with their metadata_uuid in memory. In disk, those synced device's fsid differ from their metadata_uuid, because the later member is full zero bytes. c) The device is the newest device, there are old devices with INCOMPAT_METADATA_UUID or scanned changing devices. Signed-off-by: Su Yue --- volumes.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/volumes.c b/volumes.c index bc1726975ce4..0fd41186c54a 100644 --- a/volumes.c +++ b/volumes.c @@ -227,6 +227,54 @@ static struct btrfs_fs_devices *find_fsid_inprogress( return find_fsid(disk_super->fsid, NULL); } +static struct btrfs_fs_devices *find_fsid_changed( + struct btrfs_super_block *disk_super) +{ + struct btrfs_fs_devices *fs_devices; + + /* + * Handles the case where scanned device is part of an fs that had + * multiple successful changes of FSID but currently device didn't + * observe it. + * + * Case 1: the devices already changed still owns the feature, their + * fsid must differ from the disk_super->fsid. + */ + list_for_each_entry(fs_devices, &fs_uuids, list) { + if (fs_devices->fsid_change) + continue; + if (memcmp(fs_devices->metadata_uuid, fs_devices->fsid, + BTRFS_FSID_SIZE) != 0 && + memcmp(fs_devices->metadata_uuid, disk_super->metadata_uuid, + BTRFS_FSID_SIZE) == 0 && + memcmp(fs_devices->fsid, disk_super->fsid, + BTRFS_FSID_SIZE) != 0) { + return fs_devices; + } + } + + /* + * Case 2: the synced devices doesn't have the metadata_uuid feature. + * NOTE: the fs_devices has same metadata_uuid and fsid in memory, but + * they differs in disk, because fs_id is copied to + * fs_devices->metadata_id while alloc_fs_devices if no metadata + * feature. + */ + list_for_each_entry(fs_devices, &fs_uuids, list) { + if (memcmp(fs_devices->metadata_uuid, fs_devices->fsid, + BTRFS_FSID_SIZE) == 0 && + memcmp(fs_devices->fsid, disk_super->metadata_uuid, + BTRFS_FSID_SIZE) == 0 && !fs_devices->fsid_change) + return fs_devices; + } + + /* + * Okay, can't found any fs_devices already synced, back to + * search devices unchanged or changing like the device. + */ + return find_fsid(disk_super->fsid, disk_super->metadata_uuid); +} + static int device_list_add(const char *path, struct btrfs_super_block *disk_super, u64 devid, struct btrfs_fs_devices **fs_devices_ret) @@ -242,6 +290,8 @@ static int device_list_add(const char *path, if (fsid_change_in_progress) { if (!metadata_uuid) fs_devices = find_fsid_inprogress(disk_super); + else + fs_devices = find_fsid_changed(disk_super); } if (metadata_uuid && !fs_devices) From patchwork Thu Dec 12 11:01:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287931 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 80DFB930 for ; Thu, 12 Dec 2019 11:02:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5F2D320637 for ; Thu, 12 Dec 2019 11:02:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Pxpoer3+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728969AbfLLLCY (ORCPT ); Thu, 12 Dec 2019 06:02:24 -0500 Received: from mail-pg1-f193.google.com ([209.85.215.193]:39154 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCY (ORCPT ); Thu, 12 Dec 2019 06:02:24 -0500 Received: by mail-pg1-f193.google.com with SMTP id b137so971860pga.6 for ; Thu, 12 Dec 2019 03:02:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=s6RfplAOYpdeeS/qL+EYY/6ygvavVHjtqB/z4E+QHdk=; b=Pxpoer3+kae/lgjrafAI0RDu6KgBuZhZFxXcPwadPSDrip+ZAWpOQfAssl8oJGl71c vkYbPSV+GE52cW/eo+HrX8ATK/ruqK/d4YLbVNRotYka+u8sRLT+fJCpvlnPw74y+diz VQCa6PBe+ADiwP5TcnfRCTI5GZLo+vCLMOk4os5EieJ3QjjsKLPiEy8ttXwQaUZfL4Cg JlHKf4tGoFIF+8lDzX+OBCXT5z+ohRco0o9jUbk9xDHUqp5lEux7RfRSL2lD9MrEO/ew PJwVlKvoTk2UJelqPzTaAedQY5vneT+IFj28pmG4D/5r4KPxerYR0fOvmS09atpR1k3G NIqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=s6RfplAOYpdeeS/qL+EYY/6ygvavVHjtqB/z4E+QHdk=; b=lD1Hg5JruV7qGW4EjepVLcZSDpvZYOeQ+jlri+EEMzR/sZzNiuGEEbf3V4VH1EfbQF 7O1k+5Dqz4WmNnfxsq0bPvqMA85plcbXJTmocLkk/wec7UWtJpeEJ0Rjd1hhOG7SwTO4 PTP8eMNoPhp3aqnU/7L2tHQ2WPY3TuqfYUK9e56dlsgi0V9udOZQJHJaz64NtAdKI79H t8up4w3TWi7JurueQcE5YbCiUTiDDlpbSBWIqlIrkuUhapfM6srHMAC5os667DNwaHYn mYm8FuGC1HJoMp9qo2aI5tjMIUBFQnIXMxMY4zi8BNN/aJGUalQMbQcDfau3vC+pVBFF iIqA== X-Gm-Message-State: APjAAAWOaZSM0bKXErI4bIQtKSc8XouAF/GQhUJdiLnt4TKZhkngur1S m1wG/r+5qw7rrr/bEiWCR+y2gdD60pg= X-Google-Smtp-Source: APXvYqz3XM7k1sFKSk7aEgobdigRjLI7D8MYQda/2pQaQ/yhWtWFuQLONSjgOcfNTus2xOe2xdgHcg== X-Received: by 2002:a63:d94b:: with SMTP id e11mr9712807pgj.79.1576148543189; Thu, 12 Dec 2019 03:02:23 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:22 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 06/11] btrfs-progs: handle split-brain scenario for scanned changed/unchanged device with INCOMPAT_METADATA_UUID Date: Thu, 12 Dec 2019 19:01:59 +0800 Message-Id: <20191212110204.11128-7-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue For the unchanged/changed device, just need to fs_devices in changing state, or devices with same status: a) The scanned device succeeded to sync into disk. The fs_devices can be with INCOMPAT_METADATA_UUID. So their fsid and metadata_uuid differs, and metadata_uuid is same as its metada_uuid. b) The scanned device succeeded to sync into disk. The fs_devices can be without INCOMPAT_METADATA_UUID. So their fsid and metadata_uuid be same, and fsid is same as its metadata_uuid. c) The scanned device failed to be into changing state. There are some devices whose fsids and metadata_uuids are same as the device's. d) The above cases all are missed, only unchanged devices are same as the device. Case c and d can be merged into one that both fsid and metadata requirements are meet. Signed-off-by: Su Yue --- volumes.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/volumes.c b/volumes.c index 0fd41186c54a..d094f85999d4 100644 --- a/volumes.c +++ b/volumes.c @@ -275,6 +275,50 @@ static struct btrfs_fs_devices *find_fsid_changed( return find_fsid(disk_super->fsid, disk_super->metadata_uuid); } +static struct btrfs_fs_devices *find_fsid_changing_metadata_uuid( + struct btrfs_super_block *disk_super) +{ + struct btrfs_fs_devices *fs_devices; + + /* + * Handle scanned device having completed its fsid change but + * belonging to a fs_devices that was created by first scanning + * a device which didn't have its fsid/metadata_uuid changed + * at all and the CHANGING_FSID_V2 flag set. + */ + list_for_each_entry(fs_devices, &fs_uuids, list) { + if (fs_devices->fsid_change && + memcmp(disk_super->metadata_uuid, fs_devices->fsid, + BTRFS_FSID_SIZE) == 0 && + memcmp(fs_devices->fsid, fs_devices->metadata_uuid, + BTRFS_FSID_SIZE) == 0) { + return fs_devices; + } + } + /* + * Handle scanned device having completed its fsid change but + * belonging to a fs_devices that was created by a device that + * has an outdated pair of fsid/metadata_uuid and + * CHANGING_FSID_V2 flag set. + */ + list_for_each_entry(fs_devices, &fs_uuids, list) { + if (fs_devices->fsid_change && + memcmp(fs_devices->metadata_uuid, + fs_devices->fsid, BTRFS_FSID_SIZE) != 0 && + memcmp(disk_super->metadata_uuid, + fs_devices->metadata_uuid, BTRFS_FSID_SIZE) == 0) { + return fs_devices; + } + } + + /* + * The scanned device is unchanged. Try to find devices which are + * successful in changing stage. Or old devices failed to be + * changeing liked current device. + */ + return find_fsid(disk_super->fsid, disk_super->metadata_uuid); +} + static int device_list_add(const char *path, struct btrfs_super_block *disk_super, u64 devid, struct btrfs_fs_devices **fs_devices_ret) @@ -292,6 +336,8 @@ static int device_list_add(const char *path, fs_devices = find_fsid_inprogress(disk_super); else fs_devices = find_fsid_changed(disk_super); + } else if (metadata_uuid) { + fs_devices = find_fsid_changing_metadata_uuid(disk_super); } if (metadata_uuid && !fs_devices) From patchwork Thu Dec 12 11:02:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287933 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7BF79930 for ; Thu, 12 Dec 2019 11:02:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5AAAD2077B for ; Thu, 12 Dec 2019 11:02:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ojF5EoYw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728972AbfLLLC0 (ORCPT ); Thu, 12 Dec 2019 06:02:26 -0500 Received: from mail-pf1-f194.google.com ([209.85.210.194]:45967 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCZ (ORCPT ); Thu, 12 Dec 2019 06:02:25 -0500 Received: by mail-pf1-f194.google.com with SMTP id 2so563419pfg.12 for ; Thu, 12 Dec 2019 03:02:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rLvlEok35MafMPzYVV8dLWPKxrqP2gkK5+o2+VtTio0=; b=ojF5EoYwn4/kOfpt55diLmfXR5EAAqoexOhFHM+jTx0C56s9aMyVvo92L/8Rd6mGSo p/Kit3oMU5yDo1HvjNyT18xjFSJ7HbgQI83j6BZvyLWq9DMvdVvS9q7k+gfrANmuhqno UHsbvfd8I67CZEjiRvQcbuwXiAn4fyyL22UtLw9NvG0NrneDYZY43WUArUBsG7To9Kf/ AZkmmTYn1CpeImajmZ8rzfUkfuRarCMi2loEcHpj01QcWo7HDbLWi50UWBL+qwhkRQ8g xBIJkWUKRSUPw5agbxHlfRAEyg9/6v9I4qk83jv7x6PLjTDcWCPuemdxkB78sY1sqlf0 eCLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rLvlEok35MafMPzYVV8dLWPKxrqP2gkK5+o2+VtTio0=; b=V/TrrHxPnubtem0Z8RIHUoB7UJ/l4vy/nrnEcTBzZd1XSlxhYlKs9CuR38reZPR7iM gP2/30SJu1tlA3YiODG1cFSDH4ERl7k+/6BGfH5CpUS4z35sB1Fz3hf8xv/zWIiQRiIt 11aGXe7yEQhPwrs6sKboJK3KE/hsIFMG4FJDdTD5kycRMJS4BkGSgDTh27/p1KdqnW25 fQz+p0JHPySpV67sckEk8Z56UK+FPSqxYNoUDIEXE6DWiU2XlSBPDNyva6gDOzLy6tRJ UW8O8uY4u8aCHR0V3yz4SZ3VtugRPlHEO/orL5rAswTH2zdJZWoju7+efeRK2wDoSni5 z0RA== X-Gm-Message-State: APjAAAW+Khknr7m43CohGpaCsPPdfvIJPLWo0KtVFQWvufMZFlH7xyAz G0mgCkNQj9Hm4dLOMOscnFDdv8Sv260= X-Google-Smtp-Source: APXvYqw4+aTmbIW5vdbTLo9Z5ZhxnGJl60ffCJWIIHiHDK03IKAvNIMPyIXeP03vwNDZqMbr6SCdww== X-Received: by 2002:a63:5718:: with SMTP id l24mr9575984pgb.136.1576148544874; Thu, 12 Dec 2019 03:02:24 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:24 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 07/11] btrfs-progs: handle split-brain scenario for scanned changed/unchanged device without INCOMPAT_METADATA_UUID Date: Thu, 12 Dec 2019 19:02:00 +0800 Message-Id: <20191212110204.11128-8-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue For the unchanged/changed device, just need to fs_devices in changing state, or devices with same status: a) The scanned device succeeded to sync into disk. The fs_devices must be with INCOMPAT_METADATA_UUID. So their fsid and metadata_uuid differs, and metadata_uuid is same as its fsid. b) The scanned device failed to be into changing state. There are some devices whose fsids are same as the device's. c) The above cases all are missed, only unchanged devices are same as the device. Case b and c can be merged into one that only same fsid requirement is meet. Signed-off-by: Su Yue --- volumes.c | 31 +++++++++++++++++++++++++++++++ 1 file changed, 31 insertions(+) diff --git a/volumes.c b/volumes.c index d094f85999d4..94940dd82d0f 100644 --- a/volumes.c +++ b/volumes.c @@ -319,6 +319,35 @@ static struct btrfs_fs_devices *find_fsid_changing_metadata_uuid( return find_fsid(disk_super->fsid, disk_super->metadata_uuid); } +static struct btrfs_fs_devices *find_fsid_changing( + struct btrfs_super_block *disk_super) +{ + struct btrfs_fs_devices *fs_devices; + + /* + * Handles the case where scanned device is part of an fs that had + * multiple successful changes of FSID but currently device didn't + * observe it. + * Since the scanned devices does not own the metadata_uuid feature, + * fsid and metadata_uuid of the changing devices must differ, and + * their metadata_uuid must be same as disk_super->fsid. + */ + list_for_each_entry(fs_devices, &fs_uuids, list) { + if (!fs_devices->fsid_change) + continue; + if (memcmp(fs_devices->fsid, fs_devices->metadata_uuid, + BTRFS_FSID_SIZE) != 0 && + memcmp(fs_devices->metadata_uuid, disk_super->fsid, + BTRFS_FSID_SIZE) == 0) + return fs_devices; + } + + /* + * Back to find newer fs_devices is changeing or some in same stage. + */ + return find_fsid(disk_super->fsid, NULL); +} + static int device_list_add(const char *path, struct btrfs_super_block *disk_super, u64 devid, struct btrfs_fs_devices **fs_devices_ret) @@ -338,6 +367,8 @@ static int device_list_add(const char *path, fs_devices = find_fsid_changed(disk_super); } else if (metadata_uuid) { fs_devices = find_fsid_changing_metadata_uuid(disk_super); + } else { + fs_devices = find_fsid_changing(disk_super); } if (metadata_uuid && !fs_devices) From patchwork Thu Dec 12 11:02:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287935 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 16C59138D for ; Thu, 12 Dec 2019 11:02:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E90D42077B for ; Thu, 12 Dec 2019 11:02:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tj2ANHhC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728975AbfLLLC2 (ORCPT ); Thu, 12 Dec 2019 06:02:28 -0500 Received: from mail-pg1-f195.google.com ([209.85.215.195]:42464 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLC1 (ORCPT ); Thu, 12 Dec 2019 06:02:27 -0500 Received: by mail-pg1-f195.google.com with SMTP id s64so965846pgb.9 for ; Thu, 12 Dec 2019 03:02:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bE0+OeuKSDTmUIEJuVVKKRAOFsbafmVqNIWRa4Y/oaU=; b=tj2ANHhCanbjAXJ1n7d5/iWHpMo7sVvh9k35rYuSj5Xao9m2D79EkC1axLsMgxI1zL kDL9qMYWQB/e9Ti+GJ26f3DYf7L1JHTxFqu6gdNmjWVFZ+WBgi3DXnRN2KS6dKQnorp0 Skr0GObPyf/nktRGZRiPMPqbuFEcVEnPPdR1byLv8dI4yK32kumLT8cZjgbkTZdQzQP5 0FOfX5UH2tceqEO37ZFJs1kFpKJVTxL5RUjVpH3Aj/nnKYXBVZVzITI2t51WqjHu9fEJ wTFyPbU7ulTFkGPNRsrqEC+kd3jKs854BZaxE/vLTGNMBHmFotHSAririkPoyEZhJfpy DsEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bE0+OeuKSDTmUIEJuVVKKRAOFsbafmVqNIWRa4Y/oaU=; b=hVp8/spwYHc5yFaiJCnrsBrRBXPWxipQzf98CBBCXdbpYusDlc/ZMS0dfdfEbPOjQS v0KxNL2baRabjNfckUu3pcocjy3wkHWqwsamV2gEln2s/S+lK2qjeqk/RRK6PmiYmlb3 Y4FlDgXUpM3uiOyQFO1vh3++ABS9lw22wNYrwHt1zZxQxVMN3q+oAChYdR22ItKLbMD6 +RXvNlzKGOBVmZW24thfnwo0pdi7LuZZzuwTDik3xmaAOl+//sMJOD2FCvDtr8QSiJIj 0fXS8gQYhq+Fz1tEGRTk2AF66xpX5qVVOrFS+5ngDj4Gs70ycw9rzOiDTcKfz9SzfIOz O+OQ== X-Gm-Message-State: APjAAAUzu3YJ0MbYaWzWcBwsvWO7mbgS1juAHPUZiqXewXuslC0Uq0CK DU3uVX2SqyTfp/p2Gr1zcsPDkM1Ly/0= X-Google-Smtp-Source: APXvYqz8kpJeZIYt3fGE23x0E0+u/4UkEw7bxM817owWNQmX6xl2cL5CYAHFjjlcLqnhEn908lXRGQ== X-Received: by 2002:a65:640e:: with SMTP id a14mr8974355pgv.402.1576148546580; Thu, 12 Dec 2019 03:02:26 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:26 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 08/11] btrfs-progs: metadata_uuid: remove old logic to find fs_devices Date: Thu, 12 Dec 2019 19:02:01 +0800 Message-Id: <20191212110204.11128-9-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue Now, the new code for finding fs_devices should work, remove those old logic. Signed-off-by: Su Yue --- volumes.c | 6 ------ 1 file changed, 6 deletions(-) diff --git a/volumes.c b/volumes.c index 94940dd82d0f..9d76f9a102b2 100644 --- a/volumes.c +++ b/volumes.c @@ -371,12 +371,6 @@ static int device_list_add(const char *path, fs_devices = find_fsid_changing(disk_super); } - if (metadata_uuid && !fs_devices) - fs_devices = find_fsid(disk_super->fsid, - disk_super->metadata_uuid); - else - fs_devices = find_fsid(disk_super->fsid, NULL); - if (!fs_devices) { fs_devices = kzalloc(sizeof(*fs_devices), GFP_NOFS); if (!fs_devices) From patchwork Thu Dec 12 11:02:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287937 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F00AD138D for ; Thu, 12 Dec 2019 11:02:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CE5E32077B for ; Thu, 12 Dec 2019 11:02:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bGU/A+ua" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728978AbfLLLCa (ORCPT ); Thu, 12 Dec 2019 06:02:30 -0500 Received: from mail-pf1-f193.google.com ([209.85.210.193]:42356 "EHLO mail-pf1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLC3 (ORCPT ); Thu, 12 Dec 2019 06:02:29 -0500 Received: by mail-pf1-f193.google.com with SMTP id 4so570328pfz.9 for ; Thu, 12 Dec 2019 03:02:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5q+lLSympSS8Qq1ZgfsoRfjAA0HVFX/einZgRwXV9fU=; b=bGU/A+ua/XbUqV1Rm+Zfbrl9kyBOK5bVhJp7LRGNZmRHT9zTJvdGVswp2vQ8h9XhGB VEN6hWOdr2agdO57TBEwBu+FnVMOt8SbcgFe73roQg5zpqlMROv9Mayf1BUcvFEEXhye WTox1aTEUpvCTfnXhdvqvLD2I/uxgr8t6Z4JyAams6aQ7Bc90zoIqMpuvk+Gc8ikRBnT xLNeHRMEHmVM5MbSJGuP5gFsaW2ByRWhB4uAk8eXbdiHHFbc0GnHjLS7bmSD6dmjbFyr LgNhoXFmS/jAs+VYp1GHJvFutJCDJU9KJ2bbSDoOpOKfof1QCYRMWx8AQKJpS5Vq9RLx n35A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5q+lLSympSS8Qq1ZgfsoRfjAA0HVFX/einZgRwXV9fU=; b=DMU4zfhwdF+hfy2BWzLgX02ZdvDO5d6HWc2AkuSPbctqP51bAHuHY/1uZMZdO1CQLb OwY+34YcH2Nx0/UuN5Zi37IHWm5aF1FP2+Ln3N6hfmwNAn6fBcvu0g1lTlCLL4Jh3JQ1 Nf7bQbc/wZhpqtIqT9RNJV9bRgj61kqj27awFjykbCIlwY/x222rRp1iIYIDw7JmD4Ru BIKlZA1CyDTsQPixb+sHkt30rKuPgF0GpOyyGTwEkmR4l3JLViPagijtQ8Lr3r0hbC2A 9WIzkN9PG/PmvZQIZVl2zhrcEjwy4hX9NpLIqKCnjMW9KIBySIbzrgpVfjqAdv2yYkLI 08RA== X-Gm-Message-State: APjAAAUp2DNamJjK1ukN+gxZg+cgTevgrAP5xzwt7noMQFdV5+qEeL4Q FbmNJCpAIhvWSsZ83dlJKJ30oiIBglc= X-Google-Smtp-Source: APXvYqyY3K5PBx0bZdFJkOm4YsG+Ac1W1zOI5Q8bODEjaxzePnKv26mvbsKDallMdt8ByoFglnrlVw== X-Received: by 2002:a63:a4b:: with SMTP id z11mr9345709pgk.97.1576148548741; Thu, 12 Dec 2019 03:02:28 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:28 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 09/11] btrfs-progs: metadata_uuid: rewrite fs_devices fsid and metadata_uuid if it's changing Date: Thu, 12 Dec 2019 19:02:02 +0800 Message-Id: <20191212110204.11128-10-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue As what kernel does, changing fs_devices's fsid and metadata uuid should be changed to the successful changed device's. Signed-off-by: Su Yue --- volumes.c | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/volumes.c b/volumes.c index 9d76f9a102b2..6decc699ba2c 100644 --- a/volumes.c +++ b/volumes.c @@ -359,6 +359,8 @@ static int device_list_add(const char *path, BTRFS_FEATURE_INCOMPAT_METADATA_UUID); bool fsid_change_in_progress = (btrfs_super_flags(disk_super) & BTRFS_SUPER_FLAG_CHANGING_FSID_V2); + bool new_device_added = false; + bool fs_devices_found = false; if (fsid_change_in_progress) { if (!metadata_uuid) @@ -391,9 +393,11 @@ static int device_list_add(const char *path, fs_devices->fsid_change = fsid_change_in_progress; device = NULL; } else { + fs_devices_found = true; device = find_device(fs_devices, devid, disk_super->dev_item.uuid); } + if (!device) { device = kzalloc(sizeof(*device), GFP_NOFS); if (!device) { @@ -424,6 +428,7 @@ static int device_list_add(const char *path, btrfs_stack_device_bytes_used(&disk_super->dev_item); list_add(&device->dev_list, &fs_devices->devices); device->fs_devices = fs_devices; + new_device_added = true; } else if (!device->name || strcmp(device->name, path)) { char *name; @@ -446,6 +451,23 @@ static int device_list_add(const char *path, device->name = name; } + /* + * If this disk has been pulled into an fs devices created by + * a device which had the CHANGING_FSID_V2 flag then replace the + * metadata_uuid/fsid values of the fs_devices. + */ + if (new_device_added && fs_devices_found && fs_devices->fsid_change && + found_transid > fs_devices->latest_trans) { + memcpy(fs_devices->fsid, disk_super->fsid, + BTRFS_FSID_SIZE); + if (metadata_uuid) + memcpy(fs_devices->metadata_uuid, + disk_super->metadata_uuid, BTRFS_FSID_SIZE); + else + memcpy(fs_devices->metadata_uuid, + disk_super->fsid, BTRFS_FSID_SIZE); + fs_devices->fsid_change = false; + } if (found_transid > fs_devices->latest_trans) { fs_devices->latest_devid = devid; From patchwork Thu Dec 12 11:02:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287939 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F2B89930 for ; Thu, 12 Dec 2019 11:02:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id D12E52077B for ; Thu, 12 Dec 2019 11:02:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="WZ467oJg" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728980AbfLLLCc (ORCPT ); Thu, 12 Dec 2019 06:02:32 -0500 Received: from mail-pf1-f194.google.com ([209.85.210.194]:37523 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCb (ORCPT ); Thu, 12 Dec 2019 06:02:31 -0500 Received: by mail-pf1-f194.google.com with SMTP id p14so579371pfn.4 for ; Thu, 12 Dec 2019 03:02:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FCzqjF7wPpLXtJSukHdDkjlpAl6qG8iexlnaXNo3H+w=; b=WZ467oJgc2Q5olAcOnYkVvLyO70W4/gQesTa+pY6xMgpXgooXT5+0WZ4Avdx+C9DKE +0SV+NZIo9k4Zq4I+BrktBog9MTsp1jCE8VcMlhLHDdV/F1Ph55Bj5vKjCGlM9a/kaky BL3TFEvhbvG59Li0lXIYnfV8wJ2b22nQdTRMcgklToiOTKb3+w42sZZdOmOGwxfFpd5M yfWhBgGfX72Vj407Mz1nXdUlcIJNeHbfr3fTFoWge87l2KT/o8Gliw/03YdzFr+6RVxV i8rTq3PgkrR5Dq2JLBIUS8ZBmvCSlzrbHBMwHNJDbjAssCK3f/13i2aEcMcCnOystNtO 2n1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FCzqjF7wPpLXtJSukHdDkjlpAl6qG8iexlnaXNo3H+w=; b=nDX0sF+JkSxATmo+/A7CwlbpLWtAokst5+jWA5ZuM+oDQCN76sDREzuAZ7EyDjtCRY 3dx4pOVSAB14NnehYJe3nzBHtZyH9hBIwocaLsp2Azc1flOq3wNZu3T+qjy0RaQQTWnP isv56I8k2+AvkSlfd13zik87o0kCjJ0R/firr0zpcizugA3UorrKp6kVyiyUn9I5k6sf L+sHrkTXaU9FajGIOEIo5WoG1lw7Vdm0KGz+yuBxOhqe38mk/Ak16DtsXxVmAsW3piaf ZFt060X1aUcC5eowiLYp2QoeA088tEBFNqwSN0MhGXyLqJR8PbrduRR73fyO/9pYqIOI SZZw== X-Gm-Message-State: APjAAAUre8FAW+ZpEfekdpOSYw7W4doAMsPdLfZxvjCRfIFTqk68sYwv UpXFitZgj69kdH13QHnyYuB7TzZRoSY= X-Google-Smtp-Source: APXvYqzVaNTbPfkQBCSUpX5zVZjJhJQAnzS4XygWmYluMJqzFr5quRhm3PiI3QasFV6RgW3IioJGxw== X-Received: by 2002:a63:d642:: with SMTP id d2mr9427863pgj.205.1576148550535; Thu, 12 Dec 2019 03:02:30 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.29 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:30 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 10/11] btrfs-progs: metadata_uuid: clear FSID_CHANGING_V2 while open_ctree() Date: Thu, 12 Dec 2019 19:02:03 +0800 Message-Id: <20191212110204.11128-11-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue Progs now is able to handle the metadata_uuid feature without kernel help, clear the FSID_CHANGING_V2 flag if latest device is marked as changing. Signed-off-by: Su Yue --- disk-io.c | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/disk-io.c b/disk-io.c index 659f8b93a7ca..09dacbc83e06 100644 --- a/disk-io.c +++ b/disk-io.c @@ -1177,6 +1177,7 @@ static struct btrfs_fs_info *__open_ctree_fd(int fp, const char *path, int ret; int oflags; unsigned sbflags = SBREAD_DEFAULT; + u64 sflags; /* records btrfs_super_flags */ if (sb_bytenr == 0) sb_bytenr = BTRFS_SUPER_INFO_OFFSET; @@ -1242,12 +1243,19 @@ static struct btrfs_fs_info *__open_ctree_fd(int fp, const char *path, goto out_devices; } - if (btrfs_super_flags(disk_super) & BTRFS_SUPER_FLAG_CHANGING_FSID && + sflags = btrfs_super_flags(disk_super); + if (sflags & BTRFS_SUPER_FLAG_CHANGING_FSID && !fs_info->ignore_fsid_mismatch) { fprintf(stderr, "ERROR: Filesystem UUID change in progress\n"); goto out_devices; } + if (sflags & BTRFS_SUPER_FLAG_CHANGING_FSID_V2) { + sflags &= ~BTRFS_SUPER_FLAG_CHANGING_FSID_V2; + btrfs_set_super_flags(disk_super, sflags); + printf("Found metadata UUID change in progress flag, clearing\n"); + } + ASSERT(!memcmp(disk_super->fsid, fs_devices->fsid, BTRFS_FSID_SIZE)); ASSERT(!memcmp(disk_super->fsid, fs_devices->fsid, BTRFS_FSID_SIZE)); if (btrfs_fs_incompat(fs_info, METADATA_UUID)) From patchwork Thu Dec 12 11:02:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Su Yue X-Patchwork-Id: 11287941 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7C9CE138D for ; Thu, 12 Dec 2019 11:02:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5068020637 for ; Thu, 12 Dec 2019 11:02:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PtyvT6DK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728983AbfLLLCf (ORCPT ); Thu, 12 Dec 2019 06:02:35 -0500 Received: from mail-pg1-f193.google.com ([209.85.215.193]:42473 "EHLO mail-pg1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728613AbfLLLCf (ORCPT ); Thu, 12 Dec 2019 06:02:35 -0500 Received: by mail-pg1-f193.google.com with SMTP id s64so965982pgb.9 for ; Thu, 12 Dec 2019 03:02:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=iYGYU8TVlH7JLHVdrEUiUgTO+QULmqjGrDAacT6ITX8=; b=PtyvT6DKKMyf6Zp9fFPkVWDGkwQHEqLekNTLWdrXbBg7axhk8XTBgu25/0+wo+Q1c6 ftoCMjChf5Kguk6x3XfBb9x6MEHl1Qa68IBUi/B0ZlsG4c/kMGilD0Mp0xoicXaJwW3A /1vmO6mpE/uT5dkUwfTXk11TKcfQc00XhOEkhQh6FEYin+V6L8GkoOqEAx/Oeb7ghKVY +j1dIq+U5JRIVqR0Dfgq02WQtoKCdpaVv4g85MvgylLhgRJ3JNJqkCJLyEs5qPnPZmrW k/5orLGA0BgOLQpwRGjgui8zKF+V5kdsq/nWpCzFnr00e5Eo45cdcCMLfMh1ZXl2hwS0 Inig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iYGYU8TVlH7JLHVdrEUiUgTO+QULmqjGrDAacT6ITX8=; b=hivKTmXyI+624/tu7lmvU1Mc5bvycJBtRYMlhV4F2yNJNdNpP41ft18wOIo2x+eFRX Fi22BMM7Nx8FHh5cocdYBqUC9Y25etKP7a6dMI2cSXAKUcQXx3ffvqQO3slXsqefprbO 3i40MrYWr8jld2ETWiuwBrtfp2oC4+tjp9xMJ4T94lG5rIqLFi5yS7X2HJ1jmG/8a3qe zaOYMVVosze+NGuFgxhuDK5DKp1l/GiNy6ZAvYwlz0t3ZsWfyp5AYAwtP30UW5n3AEse S2Ac1n9ouHu+G4dEUUos4JB7eIhTV/gDaEBXbdAukU3qAg1qajCa3/gNZ/Lb3CM/u9YS 68aA== X-Gm-Message-State: APjAAAXsUCJWxQxnvMQEyCwz91/SpIDiSZNzs50JpZxLV3oYgMIbf6TZ K4ZZpEBnk1Uyde8vPpKCIiF3ztGMGGs= X-Google-Smtp-Source: APXvYqxQiENNwKzfUAeLGfd1GvuLW/k3wI4cFjTf4u95qcj1BDNneAeMozlbqThkBOgyqCkQj3dqHQ== X-Received: by 2002:a63:4b48:: with SMTP id k8mr9722226pgl.362.1576148553180; Thu, 12 Dec 2019 03:02:33 -0800 (PST) Received: from p.lan ([45.58.38.246]) by smtp.gmail.com with ESMTPSA id e20sm6587857pff.96.2019.12.12.03.02.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 12 Dec 2019 03:02:32 -0800 (PST) From: damenly.su@gmail.com X-Google-Original-From: Damenly_Su@gmx.com To: linux-btrfs@vger.kernel.org Cc: Su Yue Subject: [PATCH 11/11] btrfs-progs: misc-tests/034: add new test images and modify the script Date: Thu, 12 Dec 2019 19:02:04 +0800 Message-Id: <20191212110204.11128-12-Damenly_Su@gmx.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122.2) In-Reply-To: <20191212110204.11128-1-Damenly_Su@gmx.com> References: <20191212110204.11128-1-Damenly_Su@gmx.com> MIME-Version: 1.0 Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org From: Su Yue Disk7 contains an image which has undergone a successful fsid change more than once with FSID_CHANGING_v2 flag. disk8 on the other hand is member of the same filesystem but has completed its last change without METADATA_UUID flag set. The expected recovery result is both image don't have FSID_CHANGING_v2 and INCOMPAT_METADATA_UUID. Change the test script to test it. NOTE: It needs kernel fixes for metadata_uuid feature, otherwise will fail. Signed-off-by: Su Yue --- .../misc-tests/034-metadata-uuid/disk7.raw.xz | Bin 0 -> 48388 bytes .../misc-tests/034-metadata-uuid/disk8.raw.xz | Bin 0 -> 47084 bytes tests/misc-tests/034-metadata-uuid/test.sh | 38 +++++++++++++++--- 3 files changed, 32 insertions(+), 6 deletions(-) create mode 100644 tests/misc-tests/034-metadata-uuid/disk7.raw.xz create mode 100644 tests/misc-tests/034-metadata-uuid/disk8.raw.xz diff --git a/tests/misc-tests/034-metadata-uuid/disk7.raw.xz b/tests/misc-tests/034-metadata-uuid/disk7.raw.xz new file mode 100644 index 0000000000000000000000000000000000000000..157815fd1f87039f81d0f7fbfe6888b8bac96b85 GIT binary patch literal 48388 zcmeHQc{tSl`kt{9F&L7_QiyCJ*uX@qfludZ|buJdy`Z*$(}T>kj@dtBeS?zx})d7k@uKA-QpVj#jA0-<@|b6W5iJ`VduWg+ArQ~?jg9<=8@hk!!M&9nj2~gd-seZtuPlu7hrg9>R91x^ zE5H0ICdNQIpQS&fXe>?`W;$ZEqtP#haDiBv9K!5YGvur9WSd8S$X6#8m85UU@O*oI z2aEBC`33l_bFPH~f0)lxp9$##s#aH-8NCWEtfE<1s2CKL{C;{h#zrIk!_mdAvb4f) zBo5U>HRwcqSc*^w!T+}OAlsXO#K^Y#TNN~WL?Bg^to=yiRCJ;+!uv(3bZ5H=?;U= z_D>-6BU01yqi2mXd-BgyyYw|g9#jo`RvQJ$Nb6R9Ct8fVImuCtU#<&~$qHXIwW^5q zds*)Z^WNdeK$yU zXU@ZTS&#GC#lm)TV$D%+u47{^i=#h@`hDf8p~D(r*ZWGl6ImPfni9XD56aROmMwVJ zo;T2CeuIYH#@X{dX)71qVQUpV?PT~cCeRwMHDzFP>l@m@FtzsbyZGU!{l(^wlvh>X zXwB#Dp=V5C9?w#r9V9ZdO%`hOTif|JnX+Ktu9Vti^uI)O*N?@(U^>Uet)AF-F)BC1 zv5L2R9iO~ID>9ZTmke~lO+zYtT9J*+4xO~9GTD2V#`i@i*%};{(77|Y@lZ(Qg>G%1wtA!rtSoz857Us1jI6dT1?$rf1SwVjGP#@`EFm*I_w z;?-mf)EeeyUkhenlG_j^7Ji$ ze_$AW!&q*ficK7R?=~1K4hNm;aYh>Ul3^u~}mH>^osJR(CIw@!nc**MnH3A&{U1M&xuhbv|0Sz@B@{_hAJt$+pK>a zJENJ%v3-LriRg9Finy(H#bFcat|bDgAFPHne#C%UjLn3|7~A#Eof*pT}H~H z=33O}h_D*xzYa4BKH8X7y&`9$g_?dV0@gBDU;D}4;%03r*RW@t11>_||M00vwQ8=m z8?d~h7yf&m4_x+f;ax0%X0YFo9{j5Q^geQ_ofw;&UFjIAaxs8j)Aeijt0}q65sAP| zzm1Umb{w%>NbAWyjr0P?q*qKYsSEWD>8mbc^AhYaulV{o{0h8`V5WKZWzNNl98u&B z*1bP^^!AKJ)F3SGG~Qh+n)NA8d#{Mp6h=w|3OnLleL0sfBZy%LU-+2kbme_s2O?Bs zcbCL2gAY3ePMy_$#feB@E56DwW<7r@moc0*567_$o5g4TM|*XoV^h^hh{I}vD9dBlG=}JDtE>9 z7)?#?tFp>)63WNHXhno{Gp!S!=!-nn`~Dw~ymC$msb?T#`%6y0t}I^0eApZ97uH^O z9BCdPDO2+KQ9J3(Va$%QD}R_n@0;~9f5B&{M(w)=8x+YYjxeZll%Iszrj^&oUyGVj zpHDkTWzhEQVyA^#r2pebD%6Y}CUhza?R5Ivx^%JDHsVT^L6!+|#|k@hA2d$hVn888 z1gv=z&)n9(8+4zytyGHk6t`?ykJLeu@oG+&m|h>6rX;_}Y?WUR-~Q=R#LoT)*fhP% zyK-*z_XS89z~@=%vImEi*RqZTb)TG#*5pugxuH!|l(Zk5^{yD#SvFfl`R$i^S9ih; zEtE<}ZTITD3TMp$MQkA)S~!)@C_Z<4!&>ByH!~NexLWDjo>4M0oSw3BJW!9Hs_SOq z`k@{`Zhw-F0AOV1ZvYqo?B6yT6t94Qh5-%xJw3?JH0%YGuoY|U)=KfL!&1x)Z^c$~U)#ZuZe$Cw7;msmmlUQe zWFg2l6nEQNIQXQz^mYu=*S9)^iU99&BrZL?SG^R1W|xHZdRbLxXTML_-+rpc(VdlD z#q8w;G&A1!t~ht~M_W$%qanFm&!&t6p|Gj|64JjzK(Z-hBAYHJ>8f57r2e7 zA6waKUr{u|1*LG=8q%De_pOF5^a(JQ?6{q!(ll40yFtRyI<%@IWNy4~wl(1Gvh;B; z8f_v42tK$eckOg!u(k1NjIX|JgL?&L@{0u-gYprSkH5>!_?eP`Tt!~4A`z+8wxsm-zINIkxybfN(sIe_VXx?G6 zYqO|4p}go7^rAx+EI4X+_(ks1r#y8KHQtved{|7YAv~_m0T&V)7gC^SE7# zGxdUm1Vr##OKB{R>WbxhvOJ;wS?qM!{*6f`XzN4Y8u-Q8E>H?u5Cnbw$!6x`Yjb-twT!4v7E`4C9 zDJv0m<+>-wn3b&yCvW+(im1k-US_=*v$F+@b0NAqA)R{o~) zS$Z}s6VFZTuyZLgBh@br`f`iVPGIHI>4+J)@I8*>q@@GjzRnTWSU6oA;;8D}FOw|A zdg~0VDlvVZ`FVHdbm~7Au(=iqUDbHjwbL_g$f1aYk@3h4gwO=d(z+tAy}gwUOFkev z#9DKjeXV&igYR-lq>+kn2>0jWJpL|w&T&4Xgurh3!6BK%S@&e=hjF`jmrI2>e^|0Y z(k3S|S#Rm=SQ7Oj{}C>ud}giY<7X-up#@dJ9?1q3<$eb1_xkwFT>IGsSeVYsiHfGB z&fkH)L+(_qnVAi2Xb<}AQCgtndE|vFhvQCIbnOqPbF()5l~@1Yme9~_A_`OvRKBIJIF_GBB6pT( zoYg^}JlXi^sJlMq`U@!)NhUq_2M9ueqAPovjM_Z2H*R_$)Jrv%w!77OSFqemrcPI_ z&imKeW2%y;%a*X>V}mL_l9NZBb`o`JIdr8@D|#GDyIOQ^3>W+QOK7<5K<_tpb_PYh z;V;J1Jp9PB^s~DfbZLgB?9FTbNT}sGIQB`}8P^fovO2MQ-B}F*kGF8-HRSkCKH)Qj`DC4h6r5 zuEMAtPvFs6sNM$q#iiT?mFVY z4PRFt1WkT zqn!&jxWL;AZED6%8vC`gxvq^IuE=?FXf>f{3160&{534WTugF#q56nj!&B!6oa2n= z9w9C~ZavwLM<>jnTf^Dny(5-fydg_;I(g!1P@WwTW>ouw`d$hknd-d{oZvHD$z?)S zZjakhsmB}dd1}`cD@ieEZ!2v}x?ED zN_A@6R~KCHh)PyKa`~7v-m$XRs&W45!OTV}f>+cI<`S>!4}wDn?SIGUnXGiCMV+z& zau~>AAct+S9L5ISzZE;PHi!vNgH}LRc6SU%muD3QZ}F<7s(&g5mnyj-HJko_3ovA< z(0|%o_=S2teo*rg5HTQPK*YA3h;7Y6{D18fY+A?KENWn3;6LuK0l>ig8_d5q?{#jD z3re}2K*Rp-4({gP>WAn@LRBEM*}__n#PXarY}pC~Jv`re(t5-nfPj=;xh01Nk=CYL( zUeW%5MVK~;aL1JT`WJcY$R|<-^Df?pDOcJ5s$2@%F^mAFk;%it zvL|2|U>Ifh69K~j!vMp!)UiAek{~2ONP>_g%Pj!I5i$;pZ!WigwNw7vXVo`9`k%v* z?;~+vB=>M_y;0N`9!=``q}*3Z-$d51Pq-8Pqub8p_B4Q zWB=u#0|h*;|Bkhz5jkDJt^jrg<&3j`#w3{V{@V*TelB7FFaX%^xY7kM42;CVNF0pB zDNt{3fl?o2NsuK$mfS)&p#jSkSgycw1(qwYT>sxyuYa{b2LJFtDh>nVRo+u~SeD0K)*o0K+yJfw_ z2uToj?CcWU2uG3;;%1`2-9D3%)!!vMoT z^9`EsUwD1=<`NLJMr6t-01N;|*?a>S1{ekywmDUSN}tTQ0Du9&C@Um@VSr(PVW9a2 z%{OSipR|=z{Mszg8j+cA0AK(x$|h*QFu*XtFyK~z-HQM9C0?NUCR2FIN+U>I;*f#V7sSKzn;#}zoPvb2R|3!cDn1&-^_ z)ZkBhC-kZ)gb6a|V_Lix0ke%AfqegKvN=w7SiPMV0_|U+3JncSfkGfI=F!8P+$u8E Ke|~_*%K9H3vMksD literal 0 HcmV?d00001 diff --git a/tests/misc-tests/034-metadata-uuid/disk8.raw.xz b/tests/misc-tests/034-metadata-uuid/disk8.raw.xz new file mode 100644 index 0000000000000000000000000000000000000000..ec1fa6709348b713368d3afb6ba8b741e3eec582 GIT binary patch literal 47084 zcmeI5_g7PE7KTqKSE`{Z(o|5Xp@A&4`b!AQtv?9C#BitNg&O6T z+jpRTedPC=K-jOGz}w*e;L|l}exr80$V(5xN6+!6xBPidQCi(~ZA|0+uVj6SJE$kZ z=!tzQN%e_;m#~-SF;;AFh@YfEq*|j(mNhsO#$t4}Q(DaLrD`{BcJg-DciSDNXe>dn z`81fXB;$u^9gtFJU?=w}ObcQ41qJs$ptAAiXt(IgB_)XxEKaPq;^61&Pio^@`^6^Y za&Lm&l@KC!O4rQm6vNnub@Zxnfl$D2wK1af)v9ZnJC;`}p1f9Dl_bU{cqt+d_jjJb5fT&#ARA@-Cq%O@$4JS}?v7gfGe-JF_%AUW3^ z4td^Pg(vUSIXj!8Lo_~d zL|J3n^pv53d!^*@HQEy`MjGRCHk{aia(TzLEbM4WzLG^UTJu*s?J&XR48EJ=XMV$v zb5}AVWO3+3MzfihveL)*UZ%Zo(>0Pz1}_a|`Z!b>h7F}k*6D5cBI?^}CYp)WS3X#s z@yh6UKyPuKWd_ZYW#)JGGn}P@V%@{}JRDQGvi4=x{6^aLrwsbtS><+{92xS~QlsgW z6xe@09o?2?zk!w#Q=BJ<%T!Vy3K1|g%aCqL8jA96{%j-?m$9xqIm@ItQlGKE>~)ux zql3%@{}nZw6m2R#U%v9y>eZwJVP`JrN(}3T7R*fT_#5@y#YvvX`e%X_gAW3$ZzYlF z+%vf~wu3n<&XYgrMjzC&c289nKt>Bnt}64V#J20uiHgzNh;g^ECs*a@-shBPI^=RI zGdAo)k2G(HDb6*z5z$QBKv|pzkdT(@702L1DIkMxMr3-yLmySqxa<&BCl?}F+G0?) zJn(Hq6v0nE{ELjow^fv1qV`A8>(gt)&?W1Q#%bYORM|-x!_Gx6`}gs@4d+izvz-4N zNt*U)or+OWGNyzE{&Dd<#dcBu+5o-tF$8S3^UUN~LBRf-0L~h)9M<+Fc_j514~8pu zmfcU!#J=yFjz@gk-gn*nQA^NvG1_c`Nq_jvgY^1gGe5J^{!*F=JL9lcapH0 zEpGB^;g?s6nNozC`PQ4CYq0ev^qfgCm7Fgh>e*^5%9jghGwov zzLGDj*W!=ouAUs}$f>fJTzd~EaQyeN3QB48knpa2XSq^`UHum&r(}FFYdNQOwU`}d z&nT%^x1~Ds6>hz+LXAy)zu`r$s*m|Z_GSyh-B88IM$)reoa>PV>pHkteBFE3BGLKc7UtE&qJs}&s$L4_ML*vlG0I59j zS6{CWE&u4aR@f-GVT-(LeRbbalQN=e*sZ{z<$aD)yqUPYM7bN!-S=Gg;Ji;SuL@ER z{8HGdZ~0ZlsIIh%??q>+g08Hy<}&ugiKzYxl#X=|!E+zKV6VK5b@p>4P$ih>%@dP1 zhUv??IL2^DRFMU@$R>0Tm+I=Ycw2)A%HhG+#qD)9xoDQ?4~?5n3pVZ9;+w0Lg}%=D zlpz$sCD7$HRD+qZiu{Gu;P_I3A53N(+mPzYv*HH1Ks|I_$Gnur z?KQbjM`q>1S!EGOg*iw=z~=bo0Re-6{q*-1VBsHPVGw-~{duvQK=eWMLG=IAhGlXt z7SY{sw$zajjF*O)6#K4vQqb= z(p`Gp`hQPX{_GbPx2Ky}s;HcvaVs-_(iFuUx&q?F9wivugTcLdnp|dbwm{f^NZ6pe z3*Fs0==kg(6Tp##?k;q9p}YHkf0e{UNWfK`V{M0kLBJN)M}Q2241){9o^7BK`7 z0}%reTXG@>+ZLHE1O2Or{cF)9IAY+4fg=Wv7&v0!h=C&pj@Xav7zeDFKMN7S6~J{K z*8anhgd+))`Y@>vllpL(23XW!QG-Pd7WMaE8pVVj7o*Px7WH>r@Wtd=fu|3iK6v`z z>4T>ao<4Z`;OT>>zc`m@0*3*I0fzyH0fzyH0fzyH0fzyHp_lZR4dA$f<0_FyeIz4d z3g8Oh3g8Oh3g8Oh3g8Oh3gBw8P&>3?wqlMxF$4?(wy^IfkYSKvkYTX;7FOTF>Ra%` zW_tsMDhX8*sw7lNsFF}6p-MuPoF|_cumZ3GumZ3GumZ3GumZ3GuwrrA&%j~8VZdR) zVZdR)VZdR)VZdR)VLqy&aoHg-6g-<6z&GD>WM&B14+JcLKtZ_vyliZG$f=4niXdD; l5h)Ck{U!ueJi-zX5O5oXAo>2)Dncu