From patchwork Mon Jun 7 12:33:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F283C47082 for ; Mon, 7 Jun 2021 12:34:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0957361057 for ; Mon, 7 Jun 2021 12:34:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230478AbhFGMgW (ORCPT ); Mon, 7 Jun 2021 08:36:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230420AbhFGMgR (ORCPT ); Mon, 7 Jun 2021 08:36:17 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AED38C06178B for ; Mon, 7 Jun 2021 05:34:19 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id l2so17447009wrw.6 for ; Mon, 07 Jun 2021 05:34:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tnv8/RvaEXbX/IGlvdgT6jdK85+oA3TpgnT3UjIuqEE=; b=eXAOwdVD3Swv79TuExL5blzwx7v+4hyj3n2ZiRmtra5fpSuSOK2rMM6ZNbaWd0fPfu v6mTQvfZcr17M40/2PN/WKcRMNyPuWQQm671hmovBs8tnN/Lqy2KMIpJgERb2YZJeaKD h0P4OtWEX3UYezzK3pjGDsi++T1YqWmBlI6EHyeg8GOj/sdBI8VI5y5rhwQOx/vkxujB zpvK8b4+15mT9/IS4JCUYWK4g5j0PDTFjw2lhuRIdR2jw0uxpnkF7KftlbCKwCaK6A/R t3Il9Si7XiOFqRxPsfxbvaiImHqxVC1xmZGDwoVqGHsu1EewS2nUkl7a6+iauNFW/PJc NzDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tnv8/RvaEXbX/IGlvdgT6jdK85+oA3TpgnT3UjIuqEE=; b=aGCTQ+LgzHvL69XT43cb/7L9s+XX1wSDyAggxnyIrvkX4hAJ4wxDQAo+J4mV/BFXOZ 76qVwpOgGgwcPRz7rWNO15019BZL0wyd25qk/f5yS0LzQkhxtiPy57FYO/ipRxITlM7A YoT5GgnhFNnKwSr5TGL95kmj6VcQVW8tt13SFI+AX874BxONPzytH1GMZZmc64zpM85b j/63a+iErXx/DUTEi3OiRvSU9jJonZR8lggid6ckjPVZXQk/QzVI6Q0NdcoIQo/IMXFM w+fYTK0JDjfq9FhzrFS0IjCiw36O2fj/EEArR54ll0VHEOfjhHPiMvume2tKeH/vmQzh W9HA== X-Gm-Message-State: AOAM530G1yb7xtSWKhrjR3zbT8sIfzNWHcCTIedNSsNJKS+wcBRfrdf3 aE0Og3tj27TqvRLEi0HReK2DOTJ9iyA= X-Google-Smtp-Source: ABdhPJyCijNyu2RCApB2Is0ka39sHiiwes3CJ9Cr9g79i3SPKHvICamO+vuVzLUk50MigggNj6nfng== X-Received: by 2002:a5d:67d2:: with SMTP id n18mr16716293wrw.158.1623069255052; Mon, 07 Jun 2021 05:34:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w23sm14140208wmi.0.2021.06.07.05.34.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:14 -0700 (PDT) Message-Id: <5a2ed3d1d701b11c241bf3855db3bb35f5e93994.1623069252.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:33:59 +0000 Subject: [PATCH v5 01/14] sparse-index: skip indexes with unmerged entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse-index format is designed to be compatible with merge conflicts, even those outside the sparse-checkout definition. The reason is that when converting a full index to a sparse one, a cache entry with nonzero stage will not be collapsed into a sparse directory entry. However, this behavior was not tested, and a different behavior within convert_to_sparse() fails in this scenario. Specifically, cache_tree_update() will fail when unmerged entries exist. convert_to_sparse_rec() uses the cache-tree data to recursively walk the tree structure, but also to compute the OIDs used in the sparse-directory entries. Add an index scan to convert_to_sparse() that will detect if these merge conflict entries exist and skip the conversion before trying to update the cache-tree. This is marked as NEEDSWORK because this can be removed with a suitable update to cache_tree_update() or a similar method that can construct a cache-tree with invalid nodes, but still allow creating the nodes necessary for creating sparse directory entries. It is possible that in the future we will not need to make such an update, since if we do not expand a sparse-index into a full one, this conversion does not need to happen. Thus, this can be deferred until the merge machinery is made to integrate with the sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 18 ++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index 6f21397e2ee0..1b49898d0cb7 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -125,6 +125,17 @@ int set_sparse_index_config(struct repository *repo, int enable) return res; } +static int index_has_unmerged_entries(struct index_state *istate) +{ + int i; + for (i = 0; i < istate->cache_nr; i++) { + if (ce_stage(istate->cache[i])) + return 1; + } + + return 0; +} + int convert_to_sparse(struct index_state *istate) { int test_env; @@ -161,6 +172,13 @@ int convert_to_sparse(struct index_state *istate) return -1; } + /* + * NEEDSWORK: If we have unmerged entries, then stay full. + * Unmerged entries prevent the cache-tree extension from working. + */ + if (index_has_unmerged_entries(istate)) + return 0; + if (cache_tree_update(istate, 0)) { warning(_("unable to update cache-tree, staying full")); return -1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 12e6c453024f..4f2f09b53a32 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' ' done ' +# Sparse-index fails to convert the index in the +# final 'git cherry-pick' command. +test_expect_success 'cherry-pick with conflicts' ' + init_repos && + + write_script edit-conflict <<-\EOF && + echo $1 >conflict + EOF + + test_all_match git checkout -b to-cherry-pick && + run_on_all ../edit-conflict ABC && + test_all_match git add conflict && + test_all_match git commit -m "conflict to pick" && + + test_all_match git checkout -B base HEAD~1 && + run_on_all ../edit-conflict DEF && + test_all_match git add conflict && + test_all_match git commit -m "conflict in base" && + + test_all_match test_must_fail git cherry-pick to-cherry-pick +' + test_expect_success 'clean' ' init_repos && From patchwork Mon Jun 7 12:34:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 246ECC47082 for ; Mon, 7 Jun 2021 12:35:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F2D66611C0 for ; Mon, 7 Jun 2021 12:35:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231202AbhFGMhZ (ORCPT ); Mon, 7 Jun 2021 08:37:25 -0400 Received: from mail-wr1-f45.google.com ([209.85.221.45]:34516 "EHLO mail-wr1-f45.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230323AbhFGMhX (ORCPT ); Mon, 7 Jun 2021 08:37:23 -0400 Received: by mail-wr1-f45.google.com with SMTP id q5so17437792wrm.1 for ; Mon, 07 Jun 2021 05:35:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QTzuOgoXGKpn026RiRL8qt8jaloUIGodP1E75w97/mM=; b=O2KDXci/wgZz6XxyyQYq8BfbihKUyrZQnGjZFIKrMAkybko1mxLb36NWKnZDO6e64t mXCquc04OvOcqNwTBJJvgMsi+R0/3bjBOX3biJ92y+3S9CrW2eTzjha2ePGS7HXAbuwQ wZojLKw1VffQKnRga4/KBQtM6MME7RjanOg8dw7GJDKJwusR2UnhL6jzpiZ2jyyFfawU FRbBQjvIQTLt+8lzk2/x7zoZDOC03LPNAIvvW6A4W310c33wLxjMRwmuf+yxwh+Ru4Ul WNRRTRitrrWDDAUd9ITX+c79f1N2/91id2Wq1pdgZmMZVggaM2IZY60UPb9zCrSYOMfA OtwA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QTzuOgoXGKpn026RiRL8qt8jaloUIGodP1E75w97/mM=; b=cpym2lOYpUw7j58AoQnNzQmdigIMig8GKVJQj4QJZo3mDdtrRiqivzM5jZCRvmDRO2 zXJuRCcaS6xWIMzM8gfb+0BhZ/koU3MUYE0/I/KrQRBuQXHQLj6LTSg+FB+cDONajG+C RMO5I5KGKZ1KkDber7pC3p5JRHwbzIldT4Fe/XxoDau++wEXuD0zwlGw3avAb5y+l3im Le6uoakJY0Ujbr2U2taznCG1RY3YVQ+8dnLqsTS5ZhyF8U1E+gJfZPKmx1OnXmxpYviP 5GQ1rU1Id6gq1J+z0b4NzH3stxzvmYDBvJz094QGRXTy3zg+NG4+tupEV9I/DFdGc1RP faSQ== X-Gm-Message-State: AOAM5322eQdKI3tT5KAR3+AB5tUDHEmH9lOFks+Id0IOv+wkkim2SEs1 9PLUSSMbfWTf8V6NsiWTaiWQ24ZUAGE= X-Google-Smtp-Source: ABdhPJzFY0K8dMTkAzYRZCWj8dfirSOQHC3JDBdLXMMn4v1AXN2lPm+o05xOk7FNE7JZPsLS3fmPNA== X-Received: by 2002:adf:fe4f:: with SMTP id m15mr17530307wrs.361.1623069255596; Mon, 07 Jun 2021 05:34:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t14sm15423394wra.60.2021.06.07.05.34.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:15 -0700 (PDT) Message-Id: <8aa41e749471df3bd9d593b8f55db6506eafea12.1623069252.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:00 +0000 Subject: [PATCH v5 02/14] sparse-index: include EXTENDED flag when expanding Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When creating a full index from a sparse one, we create cache entries for every blob within a given sparse directory entry. These are correctly marked with the CE_SKIP_WORKTREE flag, but they must also be marked with the CE_EXTENDED flag to ensure that the skip-worktree bit is correctly written to disk in the case that the index is not converted back down to a sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index 1b49898d0cb7..b2b3fbd75050 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -222,7 +222,7 @@ static int add_path_to_index(const struct object_id *oid, strbuf_addstr(base, path); ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); - ce->ce_flags |= CE_SKIP_WORKTREE; + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED; set_index_entry(istate, istate->cache_nr++, ce); strbuf_setlen(base, len); From patchwork Mon Jun 7 12:34:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88FDDC47082 for ; Mon, 7 Jun 2021 12:35:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 725D461057 for ; Mon, 7 Jun 2021 12:35:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231199AbhFGMhJ (ORCPT ); Mon, 7 Jun 2021 08:37:09 -0400 Received: from mail-wm1-f51.google.com ([209.85.128.51]:50801 "EHLO mail-wm1-f51.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230261AbhFGMhH (ORCPT ); Mon, 7 Jun 2021 08:37:07 -0400 Received: by mail-wm1-f51.google.com with SMTP id d184so150073wmd.0 for ; Mon, 07 Jun 2021 05:35:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=y4mvuHoWzj7XrrowHJV5XjYD/DABbA0iLonUwaUawQ8=; b=P7UzSDxwpbJIsNFqw0SlPxPv+n6U9n2EBK0Kp6+wtyDBpc5jvAszi7H/QvBF4I/qQP CJFWr+//EntOO3MwoTq7mTUDKc6Mhha+UTS6lGvGuOJg8AjRIWeRUiAGEqYPQkLHCoqc JmdQiaSWEvodhHp0l5AtRyIrmi5p13cnGRypETW28eWCcJvcUuXaBsVGCPhzXqDWf+h3 bN9XtIP4oGkTYcRGns1uOJziRy9Jr32EOHPASbaPLO0eHk4cXCu8A3EAFq7d4ALTkZga HMnGBrlU62q5Cc6oS2bkqJskxOzGNxDbe9twdjuAybk3FkYi6cvXTBr25qt8kZX8OXMw 1+ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=y4mvuHoWzj7XrrowHJV5XjYD/DABbA0iLonUwaUawQ8=; b=LcpMGZ84HCj4uFnxA6fgbOLKHiD357G6Vj9zY+gNU/4rtu4y5OmoNYkXc20MLFwZjX Mr2SqSL6XRPNPigYPM5ERynP6THNTBZjOXeujVskoSsAxJ+3WCKu28YTLP9LoQfAPuDR xpb0WIZrzCl1TZlobmZ2Rx9I6jhsBQlBvpOguU+tTFEBiOLPxByryBPMs/fTmdU7VKqt +qQpj1f3MWdrQ9wxMPt0OlVsVbx/Gg+5jtYVAi91Su9pa8GwFPVWVRx68M9CFKQNBlK8 YPeaepSxHMZo7AYVH07N88cXVFP8ILog+98FPJNTB2DCWKcQ13vu7P+DlV5ge3WMmJe6 vSZA== X-Gm-Message-State: AOAM530tZlqRT5JwCFTJSsfqZnyHHM1YXMZ62sVTGifX9dyiZj7TujU3 Tahd8d+aSDZltEETv2k33cSt97U4KKg= X-Google-Smtp-Source: ABdhPJzkgItBygvXHSAKoRMPa77IkoUd66KtVMJvdKGd0LkIHEqSFqZla6Eb4bKjgoKimavLeqC1hA== X-Received: by 2002:a05:600c:1546:: with SMTP id f6mr4479756wmg.47.1623069256057; Mon, 07 Jun 2021 05:34:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y2sm20493770wmq.45.2021.06.07.05.34.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:01 +0000 Subject: [PATCH v5 03/14] t1092: replace incorrect 'echo' with 'cat' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This fixes the test data shape to be as expected, allowing rename detection to work properly now that the 'larger-conent' file actually has meaningful lines. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 4f2f09b53a32..d55478a1902b 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -40,7 +40,7 @@ test_expect_success 'setup' ' done && git checkout -b rename-base base && - echo >folder1/larger-content <<-\EOF && + cat >folder1/larger-content <<-\EOF && matching lines help From patchwork Mon Jun 7 12:34:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303473 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0A9EDC47082 for ; Mon, 7 Jun 2021 12:35:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E7D28610E7 for ; Mon, 7 Jun 2021 12:35:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231222AbhFGMh3 (ORCPT ); Mon, 7 Jun 2021 08:37:29 -0400 Received: from mail-wr1-f46.google.com ([209.85.221.46]:40925 "EHLO mail-wr1-f46.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230359AbhFGMhZ (ORCPT ); Mon, 7 Jun 2021 08:37:25 -0400 Received: by mail-wr1-f46.google.com with SMTP id y7so12779024wrh.7 for ; Mon, 07 Jun 2021 05:35:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aWHAF5+qY9QLeWxd+QVRUKURcvZl9MIay7iwE4iRI00=; b=EBHZ1trVBheRmkNeBgLfxPCzCUgqVXhfsHZkDVNTpPuWZun/pKOAc0lc3JOLKsOpAY Af90rBg05Ck/SQCkhnyqaB32igY/rtiVbC6kEeQ9jmY5fgKIqJheArm0dkNdeaEfc5DV Lw5ogExvurwE7HbdHCYzBkl08RMIxO+rfHdpyz/r1y/AVCcdaWfEYcdazE8Axxqbdi9g kpWMiBkVu4EPh0woR9mL+yhIQOUMobPoL0EG+NIdvq+t6ULF5OWwWQwoFamz/9VGYe9H zL7D6UvR8+FZfkNXFEVAX/yXRodPFx0rjIpDHhtQvx3tszbtO4irNzKxrge2YjwtK2Cu D0LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aWHAF5+qY9QLeWxd+QVRUKURcvZl9MIay7iwE4iRI00=; b=IF7Fwoxpe5jPWxYMkw3dIcFVSF1/UwwUVeBipNyB4CykmHHygmGUN7ieavpWHt6nTd hbBzpm9wA9PgnEpkOdBA4Xjw47fq9RBaSW/40pGQDHGz0rqe3enXGumnfkm8A3PGHr9r INyee695cA6aoJ/T8HFISgau6w1Ry/wNslmEPk4+KhgFq9bzL9Kb5W5pmTZyyp4hmPwW saDvI1v9gI1Cug+DC19ZIohHsi7Dqb5ZTTvwmNxDJjwbDspWWmjMg8VOGcUikoc4QHoe vSGdl3+VaCwlWeGK17N4WyFN6rIoUdWAHaur/8UBIvq05O2Y2s+66W+YNMwFLkXwBxuA YI+A== X-Gm-Message-State: AOAM533jWvrBreFpfJoaWKRGb/yfHNu07s6oCda9tFH7yspa+ubMviJR IawWP50YEbhmb3grcJB4MVD3EI8wR2o= X-Google-Smtp-Source: ABdhPJxrk9OKt39Vv/kgzbKQPzUIWkvHVnHPrMAmr8HEqTBbPXtiIpt42WPo0uTEV+KE/Vyda06FKg== X-Received: by 2002:adf:bc07:: with SMTP id s7mr17055184wrg.301.1623069256715; Mon, 07 Jun 2021 05:34:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f18sm14194706wmj.13.2021.06.07.05.34.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:16 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:02 +0000 Subject: [PATCH v5 04/14] t1092: expand repository data shape Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As more features integrate with the sparse-index feature, more and more special cases arise that require different data shapes within the tree structure of the repository in order to demonstrate those cases. Add several interesting special cases all at once instead of sprinkling them across several commits. The interesting cases being added here are: * Add sparse-directory entries on both sides of directories within the sparse-checkout definition. * Add directories outside the sparse-checkout definition who have only one entry and are the first entry of a directory with multiple entries. Later tests will take advantage of these shapes, but they also deepen the tests that already exist. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 39 ++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index d55478a1902b..014a507d8b06 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -17,7 +17,7 @@ test_expect_success 'setup' ' echo "after folder1" >g && echo "after x" >z && mkdir folder1 folder2 deep x && - mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1 deep/deeper2 deep/before deep/later && mkdir deep/deeper1/deepest && echo "after deeper1" >deep/e && echo "after deepest" >deep/deeper1/e && @@ -25,10 +25,20 @@ test_expect_success 'setup' ' cp a folder2 && cp a x && cp a deep && + cp a deep/before && cp a deep/deeper1 && cp a deep/deeper2 && + cp a deep/later && cp a deep/deeper1/deepest && cp -r deep/deeper1/deepest deep/deeper2 && + mkdir deep/deeper1/0 && + mkdir deep/deeper1/0/0 && + touch deep/deeper1/0/1 && + touch deep/deeper1/0/0/0 && + cp -r deep/deeper1/0 folder1 && + cp -r deep/deeper1/0 folder2 && + echo >>folder1/0/0/0 && + echo >>folder2/0/1 && git add . && git commit -m "initial commit" && git checkout -b base && @@ -56,11 +66,17 @@ test_expect_success 'setup' ' mv folder1/a folder2/b && mv folder1/larger-content folder2/edited-content && echo >>folder2/edited-content && + echo >>folder2/0/1 && + echo stuff >>deep/deeper1/a && git add . && git commit -m "rename folder1/... to folder2/..." && git checkout -b rename-out-to-in rename-base && mv folder1/a deep/deeper1/b && + echo more stuff >>deep/deeper1/a && + rm folder2/0/1 && + mkdir folder2/0/1 && + echo >>folder2/0/1/1 && mv folder1/larger-content deep/deeper1/edited-content && echo >>deep/deeper1/edited-content && git add . && @@ -68,6 +84,9 @@ test_expect_success 'setup' ' git checkout -b rename-in-to-out rename-base && mv deep/deeper1/a folder1/b && + echo >>folder2/0/1 && + rm -rf folder1/0/0 && + echo >>folder1/0/0 && mv deep/deeper1/larger-content folder1/edited-content && echo >>folder1/edited-content && git add . && @@ -262,13 +281,29 @@ test_expect_success 'diff --staged' ' test_all_match git diff --staged ' -test_expect_success 'diff with renames' ' +test_expect_success 'diff with renames and conflicts' ' init_repos && for branch in rename-out-to-out rename-out-to-in rename-in-to-out do test_all_match git checkout rename-base && test_all_match git checkout $branch -- .&& + test_all_match git status --porcelain=v2 && + test_all_match git diff --staged --no-renames && + test_all_match git diff --staged --find-renames || return 1 + done +' + +test_expect_success 'diff with directory/file conflicts' ' + init_repos && + + for branch in rename-out-to-out rename-out-to-in rename-in-to-out + do + git -C full-checkout reset --hard && + test_sparse_match git reset --hard && + test_all_match git checkout $branch && + test_all_match git checkout rename-base -- . && + test_all_match git status --porcelain=v2 && test_all_match git diff --staged --no-renames && test_all_match git diff --staged --find-renames || return 1 done From patchwork Mon Jun 7 12:34:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9344EC47094 for ; Mon, 7 Jun 2021 12:34:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7F414610E7 for ; Mon, 7 Jun 2021 12:34:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231140AbhFGMga (ORCPT ); Mon, 7 Jun 2021 08:36:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230322AbhFGMg0 (ORCPT ); Mon, 7 Jun 2021 08:36:26 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A4246C061787 for ; Mon, 7 Jun 2021 05:34:18 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id 3-20020a05600c0243b029019f2f9b2b8aso10060951wmj.2 for ; Mon, 07 Jun 2021 05:34:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+ogwDcxCEiMKWorMKe+xR/DrERNlwwf/QahOIWiecqg=; b=QSjv1EubhE26f5fDtYxN+e/O8iWIUVRELRZZ4rO35v5Azzy3ErSnI2WTD9HAp3rwNs 7dQam7kYV4t3uvwK8qVId07dx+EBW4uDkib781fAvLv71elUOjIO7CSyuQbJr+QpLozP EUji080RYa8zxYd6TN/dJhV9b7RjSEhwTkS3J8PqojRkiYWhmmrJbCB8FE3aGmui2mWV 2mAl0MU5A7kFv3WIoTMkCdfxAAgZ+5QMJtV6bnUKl44Schsr5yPizeq6LwC2XYwfF30o VddHeDfTCFUIXAtaW3LVQRi2rqV0X2V478+W4qzX/EFsJ64cKoRM5NXJw/xGuT+UxRNB YqFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+ogwDcxCEiMKWorMKe+xR/DrERNlwwf/QahOIWiecqg=; b=YpIqdjqCYbJlFxtUwStjJd4Fm9W+3qboOijwpqRpMN8K+rCqL+Gs/infcpMINYWeah y4g1cQwY3MGQfs3puWgTBCU+5OdMTsuiMX7h2qMB2p5Pet30tuuLaM8Vra0AaNYQMRK8 pOkVvIcdoKxMMTMb479543TnifqzQNi8TmjsvCaKB+HhFeCQbt35b2TKOXrc+dSaSDMi CPRMMgNCpe8Nq6lFwhQv9OfCQgfdqiTRfeygJOeD7SpQDkmIqU475sMinFNKgp8US5xH PM/J9RFN7HkTfO0cA4BZZEfUZpyEV1Capic8FQeZ8srAW3Se4zVNgAMSMPGcgOPzHii/ tDYA== X-Gm-Message-State: AOAM530RpjVn8rw8T1b2MS2fTeIF/KQg2X7P4LzxkAPOiX6ub+3AGPBQ ct1DLjgqGuJgUe5iHRKsQKIgxmYzdhY= X-Google-Smtp-Source: ABdhPJyKVGJ3Ch+8mVxmQX6S8hLAVHz1Pdb+hQj+tikVaqJtxtW28Wq6a1sqcBR6kk02WFox92emmA== X-Received: by 2002:a7b:c394:: with SMTP id s20mr12182505wmj.33.1623069257292; Mon, 07 Jun 2021 05:34:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i9sm10808467wrn.54.2021.06.07.05.34.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:17 -0700 (PDT) Message-Id: <856346b72f7988eed6fe41cc7f1221a221442c98.1623069252.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:03 +0000 Subject: [PATCH v5 05/14] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file is not shown as modified and adding it will fail. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 014a507d8b06..851a83388e4b 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -251,6 +251,44 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # adding a "missing" file outside the cone should fail + test_sparse_match test_must_fail git add folder1/a && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_sparse ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" fails with a warning + # in the sparse repos, differing from the full + # repo. This is intentional. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Mon Jun 7 12:34:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CE78C47094 for ; Mon, 7 Jun 2021 12:35:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 387E9611C0 for ; Mon, 7 Jun 2021 12:35:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231196AbhFGMhM (ORCPT ); Mon, 7 Jun 2021 08:37:12 -0400 Received: from mail-wm1-f41.google.com ([209.85.128.41]:42596 "EHLO mail-wm1-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231200AbhFGMhJ (ORCPT ); Mon, 7 Jun 2021 08:37:09 -0400 Received: by mail-wm1-f41.google.com with SMTP id l7-20020a05600c1d07b02901b0e2ebd6deso1821310wms.1 for ; Mon, 07 Jun 2021 05:35:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=c2E6qwF0Mtz3Ow363vUpgOZzTlsJJKP160gjU697heeeEFyZlByQ/SxTj6+iiw8/LO jUE+P/O4hoQYeONSwV2XRVNRM1jILt8BTfbt7NdFxJ51tfmkUZ44xTmWIiPIK6YQup3D xqHga8FchcUC771UvyVMsdc/K/RybXcfaf6ac/8g8kjsGdp4rEpnVElUKic5MtdPaNP2 FyztqwMDONEBD6V01RNvAsp7uKsv37FhAMeDqZBbUGQe8VNXw5cV9N3KpR5vXq0wY/TE RD8nBS4NQVmKhIJTtkJ5RgpROsupV4eyZjmKr+UWhzzEp7n9P3BUoFUMMbWNdcLViNkN R95Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=YtA6uDY7vIw7mJCcxra71Jo6nFAZLPbkCI0bgyKgmQgdhFA/vJmcgfQdzt/MebIuLq ImttB9gvnQBLc48vu15FOSVuoDyYqaTHCY2WVpOjBivUmBYfTAm+QlzgGtxXGk8a/cER MnqJ3O93MoeUCV4HrvdYQSpMW/NxmWVNETQxBgW6IBx9P3T5E76t5QLKwZ51iov2KrwT VNB7X/WwsHaKAmtt7bDxrftwI4o1PGrnPShjIOCapmG8jfOUevpFlREMCFjLWLY2bHN3 WjMlFVFMfpZ1TNGXc+2Qba4y52zxMrJaqWsPuDZ4rEY939azv7AJs+2eQCuyZH/t6fjs YJ3Q== X-Gm-Message-State: AOAM533cpkQ9NWo4onPkAwVqwFjwVQhWLVA+zMY8Y9PMscDU0xrU7vBd n8VSMzviEoIEKjOgk5ADjRcJK3f+CZc= X-Google-Smtp-Source: ABdhPJxg14GR2Q01t53Eh0eZv3jeg9Zcg5MkzQeIeaA8LMY9UQUoTuqC983o/IAeRAsqQ+p+DmsCJQ== X-Received: by 2002:a05:600c:4f90:: with SMTP id n16mr16644166wmq.139.1623069257772; Mon, 07 Jun 2021 05:34:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k8sm13292661wrp.3.2021.06.07.05.34.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:17 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:04 +0000 Subject: [PATCH v5 06/14] unpack-trees: preserve cache_bottom Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_bottom member of 'struct unpack_trees_options' is used to track the range of index entries corresponding to a node of the cache tree. While recursing with traverse_by_cache_tree(), this value is preserved on the call stack using a local and then restored as that method returns. The mark_ce_used() method normally modifies the cache_bottom member when it refers to the marked cache entry. However, sparse directory entries are stored as nodes in the cache-tree data structure as of 2de37c53 (cache-tree: integrate with sparse directory entries, 2021-03-30). Thus, the cache_bottom will be modified as the cache-tree walk advances. Do not update it as well within mark_ce_used(). Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index dddf106d5bd4..1067db19c9d2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; From patchwork Mon Jun 7 12:34:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1443C4743F for ; Mon, 7 Jun 2021 12:34:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C8DC7610E7 for ; Mon, 7 Jun 2021 12:34:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230503AbhFGMgY (ORCPT ); Mon, 7 Jun 2021 08:36:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230420AbhFGMgX (ORCPT ); Mon, 7 Jun 2021 08:36:23 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A239CC061789 for ; Mon, 7 Jun 2021 05:34:19 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id a20so17471164wrc.0 for ; Mon, 07 Jun 2021 05:34:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8BiWBmJ/VC9KkVb4hI/TM5y8+2cX+/ZhlTUXVP76k9Q=; b=LYVOnQL7PXDTM7S8YjCaYUoJhvZGWg4EQ/2df+lNtZ+arX+rTVuUUrZZjHMls6VrKq zpExnk+tj2S1wlqeZUEVUNJCWpGL0wq91Ckmt4Y+ExtNdJ0ISA2eZKxjxExtYL+odIOO P1Iz3hKf2BHwe5Kmu0i9ERNqG2O66e/HCzjP8r1TC9x1TUqM9Gpsgjyd1CLtL8/aQwRv lWa6kbziXBbk2dLbrUTvk3uFcrsPEUv2tEGod/vawl37nrEmBzqnkLSATypYctMWAnBJ QtdGyYX8BiAEQVNmlf1mTvsnTI4H4QAhWV8w8neJhfrLAszLnnDrXXvAzFM7d5b/QJaJ e+bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8BiWBmJ/VC9KkVb4hI/TM5y8+2cX+/ZhlTUXVP76k9Q=; b=X+IWmwmxc5WrYd1v8zc7C2sONgVd0neuHKDkE+9dTraaIqNU4K1Lvg4pwuPcJAxxVy U3nqh8SN57gfBel42H2PN8u9FkqKXdIEyrKIU6bY7PDBMn7mhbZEJ6aBSvehuF2cMFKo mPiqbgHcCGBgeB0xStNK23jYEI/RHCxJeSOSjg4+zmN82zik6WM6Xi7squEpMcahON6R yMQbLXaeMQqK/mJ5cd98whAYhHRf2ufpx6MND4mHuDqLQKKj0nXNrvndqmS0sDcsYSFQ DymziXEiWA1og2ihEj6tOlq9QS5Yop5IXzYeAC3mmnYjQ2PmUI9qKTZQRU5W4zXMhoyY stvQ== X-Gm-Message-State: AOAM532PO0Rbq7z37kF57YoWmG6mluPtUVn9EYE96k5chmzNkbLESIJB DtwZRvho64CUc47kw258y/YapiiaeBQ= X-Google-Smtp-Source: ABdhPJzoOBHddqQpI//K4CZzwnMXthXYR5rQl1ULWETtb2fe2jc6Y4L0a3h/lfBm0DWB7KkyQSATpQ== X-Received: by 2002:a5d:5987:: with SMTP id n7mr16808643wri.293.1623069258315; Mon, 07 Jun 2021 05:34:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w13sm17243491wrc.31.2021.06.07.05.34.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:18 -0700 (PDT) Message-Id: <45ae96adf28546c586cbf4e67f3ec59090ef2377.1623069252.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:05 +0000 Subject: [PATCH v5 07/14] unpack-trees: compare sparse directories correctly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we further integrate the sparse-index into unpack-trees, we need to ensure that we compare sparse directory entries correctly with other entries. This affects searching for an exact path as well as sorting index entries. Sparse directory entries contain the trailing directory separator. This is important for the sorting, in particular. Thus, within do_compare_entry() we stop using S_IFREG in all cases, since sparse directories should use S_IFDIR to indicate that the comparison should treat the entry name as a dirctory. Within compare_entry(), it first calls do_compare_entry() to check the leading portion of the name. When the input path is a directory name, we could match exactly already. Thus, we should return 0 if we have an exact string match on a sparse directory entry. Signed-off-by: Derrick Stolee --- unpack-trees.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 1067db19c9d2..ef6a2b1c951c 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce, int pathlen, ce_len; const char *ce_name; int cmp; + unsigned ce_mode; /* * If we have not precomputed the traverse path, it is quicker @@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG; + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode); } static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n) @@ -1000,6 +1002,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* + * At this point, we know that we have a prefix match. If ce + * is a sparse directory, then allow an exact match. This only + * works when the input name is a directory, since ce->name + * ends in a directory separator. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! From patchwork Mon Jun 7 12:34:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A83D3C47082 for ; Mon, 7 Jun 2021 12:34:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 95689610E7 for ; Mon, 7 Jun 2021 12:34:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230507AbhFGMgZ (ORCPT ); Mon, 7 Jun 2021 08:36:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55694 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230483AbhFGMgX (ORCPT ); Mon, 7 Jun 2021 08:36:23 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77B49C061766 for ; Mon, 7 Jun 2021 05:34:20 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id n17-20020a7bc5d10000b0290169edfadac9so12592920wmk.1 for ; Mon, 07 Jun 2021 05:34:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aKwPLFNBrCYiHtXh8ZOrMRw+NjjP+1mq7q+qzlZYbgs=; b=YWF+LgzZqsNX4HpEnwRHG5OiIAVoyz3mowVbJcRCUtf4MSN7wzqMa9YpoajvJbyDyd +5coHroyIyWtISCNO5F8H00HWT7JdEstKAhlm1GIGPdDXlt45l0/EH6CMn4uPCZEHHKD Y+wVuj7ssS0fDzW2C/sNlnctMWOqBjb0Yqp1w8VEnoZ/qd2g2zGy4HnbcV8hNelENFlS 1wKCe/oTVYVg3vLlrYE+Wi74gZaXtcx2FAQATLZPZRzF3DpJUlYz3F9xrDNovdQb8DQm f+m1RXcTtc2pjE3rrcaiA4mKGkW/4z376Rf4trit9Ga/bNTzjskyoI8GmZ5L1WBh8aSF PrrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aKwPLFNBrCYiHtXh8ZOrMRw+NjjP+1mq7q+qzlZYbgs=; b=ILkWlC+vvbNkxs1XWMTAT8SBVLN9xdgkG6bm1cUATuNYW0YJ+xl+kCYzce2H4Pq9AG OVnVwX77YgvDuuww7ai8TsM5GjkugZI1bZ/oCjY07lv5KCoprg1zz/a6g2IvzlPn25pi wumQQwnueIHSVLzcfPnhg7srcGGSzhKc/pB7KU2lrs/sx/J8ZLkmj5EaeOsdyLyanSvJ xBteV80cji5re1YfPanme7c/nSZwbz0AiSYHBhKW6+IYwDLT81siTlXDpjJwIvRu3P7H VE3idc8IQ6+/Yvwx0yu1ZTTJ5sPbvCSCQLvikOyVuCcMG245GeO+PEPQYJ/l8ol9QQzH iADw== X-Gm-Message-State: AOAM531dhvQX8amqpyCLEQKuc2209oPBijJzbkaZqljPRaLsKtIpttci /s6KjUya2ssmDNgvCLhOUGMcxroVfe4= X-Google-Smtp-Source: ABdhPJwhM1y+MjHhqq108j1/D4wkpEVs48cg8dyXMLj1Jwza37gPxAVZrX2WtALvJ7nWVk5hIFTN5Q== X-Received: by 2002:a1c:e91a:: with SMTP id q26mr16945505wmc.170.1623069259010; Mon, 07 Jun 2021 05:34:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z3sm16878478wrl.13.2021.06.07.05.34.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:18 -0700 (PDT) Message-Id: <724194eef9f6d5514bcd7761daf0eb2bb87c0619.1623069252.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:06 +0000 Subject: [PATCH v5 08/14] unpack-trees: unpack sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee During unpack_callback(), index entries are compared against tree entries. These are matched according to names and types. One goal is to decide if we should recurse into subtrees or simply operate on one index entry. In the case of a sparse-directory entry, we do not want to recurse into that subtree and instead simply compare the trees. In some cases, we might want to perform a merge operation on the entry, such as during 'git checkout ' which wants to replace a sparse tree entry with the tree for that path at the target commit. We extend the logic within unpack_nondirectories() to create a sparse-directory entry in this case, and then that is sent to call_unpack_fn(). There are some subtleties in this process. For instance, we need to update find_cache_entry() to allow finding a sparse-directory entry that exactly matches a given path. Signed-off-by: Derrick Stolee --- unpack-trees.c | 101 ++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 91 insertions(+), 10 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index ef6a2b1c951c..ff448ee8424e 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1037,13 +1037,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage, struct index_state *istate, - int is_transient) + int is_transient, + int is_sparse_directory) { size_t len = traverse_path_len(info, tree_entry_len(n)); + size_t alloc_len = is_sparse_directory ? len + 1 : len; struct cache_entry *ce = is_transient ? - make_empty_transient_cache_entry(len) : - make_empty_cache_entry(istate, len); + make_empty_transient_cache_entry(alloc_len) : + make_empty_cache_entry(istate, alloc_len); ce->ce_mode = create_ce_mode(n->mode); ce->ce_flags = create_ce_flags(stage); @@ -1052,6 +1054,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, /* len+1 because the cache_entry allocates space for NUL */ make_traverse_path(ce->name, len + 1, info, n->path, n->pathlen); + if (is_sparse_directory) { + ce->name[len] = '/'; + ce->name[len + 1] = 0; + ce->ce_namelen++; + ce->ce_flags |= CE_SKIP_WORKTREE; + } + return ce; } @@ -1064,16 +1073,24 @@ static int unpack_nondirectories(int n, unsigned long mask, unsigned long dirmask, struct cache_entry **src, const struct name_entry *names, - const struct traverse_info *info) + const struct traverse_info *info, + int sparse_directory) { int i; struct unpack_trees_options *o = info->data; unsigned long conflicts = info->df_conflicts | dirmask; - /* Do we have *only* directories? Nothing to do */ if (mask == dirmask && !src[0]) return 0; + /* no-op if our cache entry doesn't match the expectations. */ + if (sparse_directory) { + if (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode)) + BUG("expected sparse directory entry"); + } else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) { + return 0; + } + /* * Ok, we've filled in up to any potential index entry in src[0], * now do the rest. @@ -1103,7 +1120,9 @@ static int unpack_nondirectories(int n, unsigned long mask, * not stored in the index. otherwise construct the * cache entry from the index aware logic. */ - src[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge); + src[i + o->merge] = create_ce_entry(info, names + i, stage, + &o->result, o->merge, + sparse_directory); } if (o->merge) { @@ -1210,13 +1229,44 @@ static int find_cache_pos(struct traverse_info *info, static struct cache_entry *find_cache_entry(struct traverse_info *info, const struct name_entry *p) { + struct cache_entry *ce; int pos = find_cache_pos(info, p->path, p->pathlen); struct unpack_trees_options *o = info->data; if (0 <= pos) return o->src_index->cache[pos]; - else + + /* + * Check for a sparse-directory entry named "path/". + * Due to the input p->path not having a trailing + * slash, the negative 'pos' value overshoots the + * expected position by one, hence "-2" here. + */ + pos = -pos - 2; + + if (pos < 0 || pos >= o->src_index->cache_nr) + return NULL; + + ce = o->src_index->cache[pos]; + + if (!S_ISSPARSEDIR(ce->ce_mode)) return NULL; + + /* + * Compare ce->name to info->name + '/' + p->path + '/' + * if info->name is non-empty. Compare ce->name to + * p-.path + '/' otherwise. + */ + if (info->namelen) { + if (ce->ce_namelen == info->namelen + p->pathlen + 2 && + ce->name[info->namelen] == '/' && + !strncmp(ce->name, info->name, info->namelen) && + !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen)) + return ce; + } else if (ce->ce_namelen == p->pathlen + 1 && + !strncmp(ce->name, p->path, p->pathlen)) + return ce; + return NULL; } static void debug_path(struct traverse_info *info) @@ -1251,6 +1301,32 @@ static void debug_unpack_callback(int n, debug_name_entry(i, names + i); } +/* + * Returns true if and only if the given cache_entry is a + * sparse-directory entry that matches the given name_entry + * from the tree walk at the given traverse_info. + */ +static int is_sparse_directory_entry(struct cache_entry *ce, struct name_entry *name, struct traverse_info *info) +{ + size_t expected_len, name_start; + + if (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode)) + return 0; + + if (info->namelen) + name_start = info->namelen + 1; + else + name_start = 0; + expected_len = name->pathlen + 1 + name_start; + + if (ce->ce_namelen != expected_len || + strncmp(ce->name, info->name, info->namelen) || + strncmp(ce->name + name_start, name->path, name->pathlen)) + return 0; + + return 1; +} + /* * Note that traverse_by_cache_tree() duplicates some logic in this function * without actually calling it. If you change the logic here you may need to @@ -1307,7 +1383,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (unpack_nondirectories(n, mask, dirmask, src, names, info, 0) < 0) return -1; if (o->merge && src[0]) { @@ -1337,9 +1413,14 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, - names, info) < 0) + if (is_sparse_directory_entry(src[0], names, info)) { + if (unpack_nondirectories(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0) + return -1; + } else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, + names, info) < 0) { return -1; + } + return mask; } From patchwork Mon Jun 7 12:34:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CB5C5C47082 for ; Mon, 7 Jun 2021 12:34:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B0D3B61057 for ; Mon, 7 Jun 2021 12:34:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230264AbhFGMgP (ORCPT ); Mon, 7 Jun 2021 08:36:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230207AbhFGMgM (ORCPT ); Mon, 7 Jun 2021 08:36:12 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D9EF7C061795 for ; Mon, 7 Jun 2021 05:34:20 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id q5so17438034wrm.1 for ; Mon, 07 Jun 2021 05:34:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=enLsZo1kPstXaNr5J3W8tk/S6xFiq7UjxYz5mUCWeres9J5xjM8x7hf98TokOswxCG WjXxrIHIx3KkAKcvsCqt5KU33Aydm2h6EYEYWpvyxFe1UDUZDgooGpWro3L5+Ezx96Po oAbsERCtuenjTghFr+EKJpOdWi9HCF/OBpVtC6PpK2S4mj6xBZqGtMG9S+9nXZ1rJKlB myPAdngHphCky8egWDpSrnmplIDXG1o8/ihtwZe9wok0ot1pEjfX7vDEoxLF67ft9/SK YYocUgN9i1WA5iex7GEEFrSmos7JLRFyfGDlBpKbLZoPZUPP60fTRdWGYqR5BlKjP8jj ktuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=O8lT8GRVr4neOxm9sWtFH/ojdnB7QynNP46PShCbCZjA6xKbCTKhg9mbyLGsbuM1OL UkWD1FhamDX4mImw+MW425u+TlY31P+9473XytqZxlJKsQb3de9Ry3jpK7IWVQGherN4 VLGWZ2T/1xmh5Pucs4sCwYUBct4FBbaWJbZv7Poi3ODcinPEt+krB2EPo9aLmM77vTe+ MToYKmmgvwECJVV5SM4dYAD0FdYVE0iqQdSTU6HKjpPeYQLz1heMprbrf6uC3tVOSGC3 C4CK+7vnXRPmbD6XkA7FGXhbtvIwuyI1nPNO+rOKjuZ1w5HyuHjHB8XWd0/lICbqkXvp RtMA== X-Gm-Message-State: AOAM533UNGdVbFeNIUAAJ9DjB52pyVi0dHcOEmdBfR96RsBPCDi0Uqp0 4xAJ5kU+fkUYJ8M/dRTraN4skBeIN1U= X-Google-Smtp-Source: ABdhPJx8j1NMlcK2a48QKt5X0EJ+ujmG70b7WpJ/wsFhHRWSXXNSDS0rRWFwxGzy+v+XhlJTI3KUVA== X-Received: by 2002:a5d:4681:: with SMTP id u1mr9741087wrq.268.1623069259545; Mon, 07 Jun 2021 05:34:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k8sm13292770wrp.3.2021.06.07.05.34.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:19 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:07 +0000 Subject: [PATCH v5 09/14] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dir.c b/dir.c index 166238e79f52..ab76ef286495 100644 --- a/dir.c +++ b/dir.c @@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* + * Directory entries are matched if and only if a file + * contained immediately within them is matched. For the + * case of a directory entry, modify the path to create + * a fake filename within this directory, allowing us to + * use the file-base matching logic in an equivalent way. + */ + if (parent_pathname.len > 0 && + parent_pathname.buf[parent_pathname.len - 1] == '/') + strbuf_add(&parent_pathname, "-", 1); + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; From patchwork Mon Jun 7 12:34:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06412C47094 for ; Mon, 7 Jun 2021 12:35:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E6693610E7 for ; Mon, 7 Jun 2021 12:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231236AbhFGMhb (ORCPT ); Mon, 7 Jun 2021 08:37:31 -0400 Received: from mail-wm1-f41.google.com ([209.85.128.41]:46686 "EHLO mail-wm1-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230288AbhFGMhZ (ORCPT ); Mon, 7 Jun 2021 08:37:25 -0400 Received: by mail-wm1-f41.google.com with SMTP id h22-20020a05600c3516b02901a826f84095so5088473wmq.5 for ; Mon, 07 Jun 2021 05:35:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Vmwu6W4wC2XHe1k0Gpqxbz7pS0vrf/3ndg5LD7VPSvM=; b=ab3d0aSOhj/LnRuuAE1pHtJWrBYzC0fiKFn0ZbW7lxxBcr+c+WVGjDipv9fQIvOlBM 9/F9ZQvBadi0BGugBgrd5U4Ht9RsIGcphpi0VLVJ7TXTW0ARA9K7Nj17/ZavNQZxnffz xcHwJdfykrZvTvVikXRNueYOEyJDszK8afpy9/Rp4JiAN39D2P1M52cyxJmO46ZEIUSz oXcY4aIf8GvBbS1v8r4hrMrlxmGb1Dsnye4l36qMFA2dA/MzKmLLR0YWtuasbkj+/445 9dMJd0HVIAapJt+qZVs5PUpj/7z0PySDdACZoyWIz0gjwEbxiYn3bGKC8/e6GdKaXyTm B65Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Vmwu6W4wC2XHe1k0Gpqxbz7pS0vrf/3ndg5LD7VPSvM=; b=fNQCWeMPI4zVZTU1FhZyCWzqn3w1cTeBakN24NgGVNgUhjeoSevD2e+a7Pf2Ge10Id PHZKkzBbmmx+2JGRWt1oenEt2W3t50tKye7zDoDNZ/DNGCDtWQe3VNDOkv2En3XCX+YD E+7UJvvzBKxZMvnUBpXWV50ezxbGD161+Pe17DK7A+hgbHN/yQE43wAyrojVFR1OHV9g rbTR4okmiqoEs5PJyF9ipjupOrcWurk/ZA1eUdwpX2zdIaG+KjAO4JjeHZ/sj/98LcpR WPoGaylY2uKD2QkfgPnFztn4CKkun0szEv3hWoUcZqWJOcjjJQM7DQoAwxsnbD8cZCqz ixvg== X-Gm-Message-State: AOAM533wm7KqR4hh505V8Y2mdTn4cDvfAxp+O+PEhw2JV24RZNOANgEr 6A3m7HJ298phQ1bBdBRuiMbiuho/PHg= X-Google-Smtp-Source: ABdhPJyP1Z+kIaDNLBcVNoivub87xxE4U3pTbLMRRmqJg0vBsqMDFC86XZwptdLratnX0emVcVAlpg== X-Received: by 2002:a1c:3dc2:: with SMTP id k185mr5078066wma.15.1623069260116; Mon, 07 Jun 2021 05:34:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 6sm14431982wmg.17.2021.06.07.05.34.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:19 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:08 +0000 Subject: [PATCH v5 10/14] diff-lib: handle index diffs with sparse dirs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee While comparing an index to a tree, we may see a sparse directory entry. In this case, we should compare that portion of the tree to the tree represented by that entry. This could include a new tree which needs to be expanded to a full list of added files. It could also include an existing tree, in which case all of the changes inside are important to describe, including the modifications, additions, and deletions. Note that the case where the tree has a path and the index does not remains identical to before: the lack of a cache entry is the same with a sparse index. In the case where a tree is modified, we need to expand the tree recursively, and start comparing each contained entry as either an addition, deletion, or modification. This causes an interesting recursion that did not exist before. Signed-off-by: Derrick Stolee --- diff-lib.c | 188 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 188 insertions(+) diff --git a/diff-lib.c b/diff-lib.c index b73cc1859a49..ba4c683d4bc4 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -314,6 +314,48 @@ static int get_stat_data(const struct cache_entry *ce, return 0; } +struct show_new_tree_context { + struct rev_info *revs; + unsigned added:1; +}; + +static int show_new_file_from_tree(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct show_new_tree_context *ctx = context; + struct cache_entry *new_file = make_transient_cache_entry(mode, oid, path, /* stage */ 0); + + diff_index_show_file(ctx->revs, ctx->added ? "+" : "-", new_file, oid, !is_null_oid(oid), mode, 0); + discard_cache_entry(new_file); + return 0; +} + +static void show_directory(struct rev_info *revs, + const struct cache_entry *new_dir, + int added) +{ + /* + * new_dir is a sparse directory entry, so we want to collect all + * of the new files within the tree. This requires recursively + * expanding the trees. + */ + struct show_new_tree_context ctx = { revs, added }; + struct repository *r = revs->repo; + struct strbuf base = STRBUF_INIT; + struct pathspec ps; + struct tree *tree = lookup_tree(r, &new_dir->oid); + + memset(&ps, 0, sizeof(ps)); + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_add(&base, new_dir->name, new_dir->ce_namelen); + read_tree_at(r, tree, &base, &ps, + show_new_file_from_tree, &ctx); +} + static void show_new_file(struct rev_info *revs, const struct cache_entry *new_file, int cached, int match_missing) @@ -322,6 +364,11 @@ static void show_new_file(struct rev_info *revs, unsigned int mode; unsigned dirty_submodule = 0; + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) { + show_directory(revs, new_file, /*added */ 1); + return; + } + /* * New file in the index: it might actually be different in * the working tree. @@ -333,6 +380,136 @@ static void show_new_file(struct rev_info *revs, diff_index_show_file(revs, "+", new_file, oid, !is_null_oid(oid), mode, dirty_submodule); } +static int show_modified(struct rev_info *revs, + const struct cache_entry *old_entry, + const struct cache_entry *new_entry, + int report_missing, + int cached, int match_missing); + +static int compare_within_sparse_dir(int n, unsigned long mask, + unsigned long dirmask, struct name_entry *entry, + struct traverse_info *info) +{ + struct rev_info *revs = info->data; + struct object_id *oid0 = &entry[0].oid; + struct object_id *oid1 = &entry[1].oid; + + if (oideq(oid0, oid1)) + return mask; + + /* Directory/file conflicts are handled earlier. */ + if (S_ISDIR(entry[0].mode) && S_ISDIR(entry[1].mode)) { + struct tree_desc t[2]; + void *buf[2]; + struct traverse_info info_r = { NULL, }; + + info_r.name = xstrfmt("%s%s", info->traverse_path, entry[0].path); + info_r.namelen = strlen(info_r.name); + info_r.traverse_path = xstrfmt("%s/", info_r.name); + info_r.fn = compare_within_sparse_dir; + info_r.prev = info; + info_r.mode = entry[0].mode; + info_r.pathlen = entry[0].pathlen; + info_r.df_conflicts = 0; + info_r.data = revs; + + buf[0] = fill_tree_descriptor(revs->repo, &t[0], oid0); + buf[1] = fill_tree_descriptor(revs->repo, &t[1], oid1); + + traverse_trees(NULL, 2, t, &info_r); + + free((char *)info_r.name); + free((char *)info_r.traverse_path); + free(buf[0]); + free(buf[1]); + } else { + char *old_path = NULL, *new_path = NULL; + struct cache_entry *old_entry = NULL, *new_entry = NULL; + + if (entry[0].path) { + old_path = xstrfmt("%s%s", info->traverse_path, entry[0].path); + old_entry = make_transient_cache_entry( + entry[0].mode, &entry[0].oid, + old_path, /* stage */ 0); + old_entry->ce_flags |= CE_SKIP_WORKTREE; + } + if (entry[1].path) { + new_path = xstrfmt("%s%s", info->traverse_path, entry[1].path); + new_entry = make_transient_cache_entry( + entry[1].mode, &entry[1].oid, + new_path, /* stage */ 0); + new_entry->ce_flags |= CE_SKIP_WORKTREE; + } + + if (entry[0].path && entry[1].path) + show_modified(revs, old_entry, new_entry, 0, 1, 0); + else if (entry[0].path) + diff_index_show_file(revs, revs->prefix, + old_entry, &entry[0].oid, + 0, entry[0].mode, 0); + else if (entry[1].path) + show_new_file(revs, new_entry, 1, 0); + + discard_cache_entry(old_entry); + discard_cache_entry(new_entry); + free(old_path); + free(new_path); + } + + return mask; +} + +static void show_modified_sparse_directory(struct rev_info *revs, + const struct cache_entry *old_entry, + const struct cache_entry *new_entry, + int report_missing, + int cached, int match_missing) +{ + struct tree_desc t[2]; + void *buf[2]; + struct traverse_info info = { NULL }; + struct strbuf name = STRBUF_INIT; + struct strbuf parent_path = STRBUF_INIT; + char *last_dir_sep; + + if (oideq(&old_entry->oid, &new_entry->oid)) + return; + + info.fn = compare_within_sparse_dir; + info.prev = &info; + + strbuf_add(&name, new_entry->name, new_entry->ce_namelen - 1); + info.name = name.buf; + info.namelen = name.len; + + strbuf_add(&parent_path, new_entry->name, new_entry->ce_namelen - 1); + if ((last_dir_sep = find_last_dir_sep(parent_path.buf)) > parent_path.buf) + strbuf_setlen(&parent_path, (last_dir_sep - parent_path.buf) - 1); + else + strbuf_setlen(&parent_path, 0); + + info.pathlen = parent_path.len; + + if (parent_path.len) + info.traverse_path = parent_path.buf; + else + info.traverse_path = ""; + + info.mode = new_entry->ce_mode; + info.df_conflicts = 0; + info.data = revs; + + buf[0] = fill_tree_descriptor(revs->repo, &t[0], &old_entry->oid); + buf[1] = fill_tree_descriptor(revs->repo, &t[1], &new_entry->oid); + + traverse_trees(NULL, 2, t, &info); + + free(buf[0]); + free(buf[1]); + strbuf_release(&name); + strbuf_release(&parent_path); +} + static int show_modified(struct rev_info *revs, const struct cache_entry *old_entry, const struct cache_entry *new_entry, @@ -343,6 +520,17 @@ static int show_modified(struct rev_info *revs, const struct object_id *oid; unsigned dirty_submodule = 0; + /* + * If both are sparse directory entries, then expand the + * modifications to the file level. + */ + if (old_entry && new_entry && + S_ISSPARSEDIR(old_entry->ce_mode) && + S_ISSPARSEDIR(new_entry->ce_mode)) { + show_modified_sparse_directory(revs, old_entry, new_entry, report_missing, cached, match_missing); + return 0; + } + if (get_stat_data(new_entry, &oid, &mode, cached, match_missing, &dirty_submodule, &revs->diffopt) < 0) { if (report_missing) From patchwork Mon Jun 7 12:34:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6734FC4743F for ; Mon, 7 Jun 2021 12:34:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 51F99610E7 for ; Mon, 7 Jun 2021 12:34:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230516AbhFGMg3 (ORCPT ); Mon, 7 Jun 2021 08:36:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230483AbhFGMg0 (ORCPT ); Mon, 7 Jun 2021 08:36:26 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E77AC061795 for ; Mon, 7 Jun 2021 05:34:22 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id l9so934278wms.1 for ; Mon, 07 Jun 2021 05:34:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=FKiHIw3ebWQz9cNbabJCmOn9U0CL4gxPTfpaj57hnfY=; b=iWP3zmEQcBDkIkTQIqWzD7iKNWe2SSQTX/ITOSq+lcRGLFDeOBt05jloc81MrPuo4d 8thMXP3woRnBIBUK0TEjAFdmQKbL3CTPdRaibjvR8GTL07mIsFa0keB69rSbYAH/y9LR nfIvUX5ovJukO5/kUNFwtBzSS8U9slZYmG4+GlVuI1X9rLKb7FecmHnoakqaIPuVICxZ bVtwpXlTgnJlXP4Es02E4n381J9YpAL6BbKFABywY5y2cjLoX3RYGGudUY6LLhynrUQb s2aRAtGNMlEoY1s2GHDMWMn0U+c5ZKZBdwKrgdlPCAgaB/O6YDlGjXGJ5KcZHRVpWnGq Z1Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=FKiHIw3ebWQz9cNbabJCmOn9U0CL4gxPTfpaj57hnfY=; b=dL5+nzQ71vAx7I5sAa909vAjYR7axQAF8UoB7G7D0WA+dHOGrmS9uzigmTft4h5Q7d FeWeaR2BaKsOZp2m0pgGj6UiHropYV4Zua9PS2uKk5bvlxdfJGO3BVkaM7Z56i5re5pv krrZ56MlXUJ0bdk5gjWKwK8PF3yye56Dm/YNYdgh1PhdPA6DzDCaGAcrRge1Ga51JakK lw3lDKlEFrBRVn/ySnXsJcUGLMYiNk0KOajx2L0Rz2lKZ5QSuwJ+Rfj6iEEWKt2GGB+y dRztbWP7HbHM0Pe/MENfFETT8KjbqHld2Exi4zH8fOkDmhtZhhs6YQTXdsjx0TM1CpT8 Rkag== X-Gm-Message-State: AOAM531oHFgTONZNoI4Ea1xnHfOlWSvxnNfYSmu7xAJEif6tMkratDsk 2cfaUsOQLmcRvOjmAaWmSX/CqSftuV4= X-Google-Smtp-Source: ABdhPJz6GiMMBpz6Yoho0Nv7Ac+7SWglynGjjYQhzsl0UEG4r0efy5AcD7EUp2zYSE2TuI1i/evjwA== X-Received: by 2002:a1c:7912:: with SMTP id l18mr16619339wme.135.1623069260693; Mon, 07 Jun 2021 05:34:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v132sm17944482wmb.14.2021.06.07.05.34.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:20 -0700 (PDT) Message-Id: <611b9f61fb2c834d01ace24b82e1d9791a21c939.1623069252.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:09 +0000 Subject: [PATCH v5 11/14] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 851a83388e4b..f6b124e0500f 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -215,6 +215,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 0c8287a023e4..0425169c1895 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e1..ab9cc9d8f032 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Mon Jun 7 12:34:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E84ABC47094 for ; Mon, 7 Jun 2021 12:34:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CAFB76121D for ; Mon, 7 Jun 2021 12:34:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231157AbhFGMgb (ORCPT ); Mon, 7 Jun 2021 08:36:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230324AbhFGMg1 (ORCPT ); Mon, 7 Jun 2021 08:36:27 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 373FBC0617A6 for ; Mon, 7 Jun 2021 05:34:24 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id v206-20020a1cded70000b02901a586d3fa23so5781927wmg.4 for ; Mon, 07 Jun 2021 05:34:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=FVckP3j8w+mKO74ehu59pvxVbH22LOdegvSLJiu2gdA=; b=LdNtViATQxrBu59oTwZjDXjmhzDesc8cvTqEf5n85CigvtOAfjBPGgxVxKyZHDLILe fJKISqWxNVU0rXvyo0Wg/i3uVpLmnHG8uLufA/06/3jVyHutLP8dJJ6P+00A843fZO7T 7tS/+y0AXckYWHbN3fv5XJ9JSGTy3o6bmt/gNipM9AXBJ68mKwa3pOSUfj9FBtwxyL3x Gmb18adZFPdnF/QAzvBbj8fY3j4RPb3QCWlleEHoXG5TviyXNdaDbLbRzDThX9l1NIDq jJIrcgtlFzbPoU3W/o61NKUmMZ0B9batEzUNMd+fASDpHcDfut0PeAZ81m/qMHRRbuG6 27kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=FVckP3j8w+mKO74ehu59pvxVbH22LOdegvSLJiu2gdA=; b=OS0N5ycBnVveBG33VteaCBhl7a4q76P+fTvsZVwD1sRvOMwMvI0gTy/zgx3nOy5pjz jrxmBftWJhkLAUjqPaP31WP2OzOaB+YaEC+5jZKqy2XbO+mYzCFw41Uv8i+Ja7iI25sy p9AuadcJTqCWPz8CV0SxNv5RpotYhTOsvGk7HoRu30jOR0CsMJBf359YWoWcqLiYhuNx YA9PNCDAa4TorpyiXltglyoTCTE9KEIgxPxqd39/YBj3fG0F/T7m3hMivFnTnk/FMxjP 8AUFiArAMtRYlOn5LYoCkDSOC9NoWTgoRTGXB/h1CVHTDb0htKUw2p67P2rO+grJnhOH P1fQ== X-Gm-Message-State: AOAM531AtXgrCxguYSVFZOwhIKPzX6TR4n8MqRto8u154pGxwqjE8Hmk kd/JqlJBHMtrQzDg6mahMZNJHmyZLSg= X-Google-Smtp-Source: ABdhPJybEI0hXLaraqbkQrHmixv+nku7Uioa/epRgBnYYc7Lj9v+nTh7npERRUPtwblXcbo3b2YRoQ== X-Received: by 2002:a05:600c:1d1a:: with SMTP id l26mr9800930wms.189.1623069261262; Mon, 07 Jun 2021 05:34:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c7sm15925764wrs.23.2021.06.07.05.34.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:20 -0700 (PDT) Message-Id: <0c0a765dde80c75499c661d518a40a250b9809c5.1623069253.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:10 +0000 Subject: [PATCH v5 12/14] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index cf0c36d1dcb2..e529da7beadd 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 29ffa9ac5db9..f80e26831b36 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index f6b124e0500f..099dc2bf440f 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -508,12 +508,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Mon Jun 7 12:34:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5601BC47094 for ; Mon, 7 Jun 2021 12:35:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A75E61057 for ; Mon, 7 Jun 2021 12:35:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231243AbhFGMhb (ORCPT ); Mon, 7 Jun 2021 08:37:31 -0400 Received: from mail-wr1-f41.google.com ([209.85.221.41]:36711 "EHLO mail-wr1-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231211AbhFGMh1 (ORCPT ); Mon, 7 Jun 2021 08:37:27 -0400 Received: by mail-wr1-f41.google.com with SMTP id e11so7207299wrg.3 for ; Mon, 07 Jun 2021 05:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=smXFtkXovHCA1EwmRIxXBky+6I9XveV7+FJ/L5SZCoA=; b=B0qBX+SwaT3QwgkMcHoHJJaTi4wJnnh98nD2qWhHe9PcSNUbL6i54NhterlKPmPZU2 vAYl3Gs2QV9TAdKskErs7txesfdi3CVYSuhY3XnXYF6UlQCdta+7FBYxwlecDTih6oUZ Rnmkzpi3dATN/w8GcVM1a4mWkHFTGV2C5ccLESSX0BGb6jqKinoIVk8h7nTvxN631HcM mvdTnRen/e2l7tukY1mpm7d6hKb+/a6XA1YBPpsekLjRRZN1BYFyJvBiWzZja8ySNb/2 uE2zIjQu7vesMs3M+yAyiED//ief3xlh9PDeeod2piwKdT57i7gRJ5AiQrb9FT64iBvR Jvlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=smXFtkXovHCA1EwmRIxXBky+6I9XveV7+FJ/L5SZCoA=; b=tj3FQ67eurM1oeSL9oS7tOamIoJti76D/bxyVX/YEpusJsDZtNmxcXcu5iZO7N+Zdf bLK5M7GDeAA333UVnykdMzWtItHUWsCXXBQYRBArw758xia8KSWgPcOIFXP2Ovum7OtF ceEwQlpfohNHyiIGcFyDaWv//M7mpczDolxKp2toI/VMEEPXWLLa/rV+eqMll3ZC4qAQ WAkONepSNREgD4qgevedDo4fJJbOphQ08Fi9wwRlsuYM1EDHpyNnH9BqKVccE4/1puJR degBsXajQDBQELARrlnoaw50NtH/jAcMsHWBxg0iWuCQo6GibrU4KrKYcDtb4o8y0y5L YwQQ== X-Gm-Message-State: AOAM532KUJeH5joSYr+zT/+cwQ/hjOV15z2mLZgCXYLbJJXiaHFmClbX mYmU7LAReOClMJWbMc0BSLbYqAMnAGM= X-Google-Smtp-Source: ABdhPJxgX82AIdCUTO2sGjpW4lWUd1o7zMbpRmgtRH1K2AXj7Y8CqCuUqRkb8wrAAfLEssgxoW9myw== X-Received: by 2002:a5d:64a4:: with SMTP id m4mr6327797wrp.185.1623069261784; Mon, 07 Jun 2021 05:34:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v17sm7062915wrp.36.2021.06.07.05.34.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:21 -0700 (PDT) Message-Id: <02f2c7b63982cfbafc300e1cd901473d5b9b7297.1623069253.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:11 +0000 Subject: [PATCH v5 13/14] wt-status: expand added sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It is difficult, but possible, to get into a state where we intend to add a directory that is outside of the sparse-checkout definition. Add a test to t1092-sparse-checkout-compatibility.sh that demonstrates this using a combination of 'git reset --mixed' and 'git checkout --orphan'. This test failed before because the output of 'git status --porcelain=v2' would not match on the lines for folder1/: * The sparse-checkout repo (with a full index) would output each path name that is intended to be added. * The sparse-index repo would only output that "folder1/" is staged for addition. The status should report the full list of files to be added, and so this sparse-directory entry should be expanded to a full list when reaching it inside the wt_status_collect_changes_initial() method. Use read_tree_at() to assist. Somehow, this loop over the cache entries was not guarded by ensure_full_index() as intended. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 36 +++++++++++++++++ wt-status.c | 50 ++++++++++++++++++++++++ 2 files changed, 86 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 099dc2bf440f..39b86fbe2be6 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -521,4 +521,40 @@ test_expect_success 'sparse-index is not expanded' ' test_region ! index ensure_full_index trace2.txt ' +test_expect_success 'reset mixed and checkout orphan' ' + init_repos && + + test_all_match git checkout rename-out-to-in && + + # Sparse checkouts do not agree with full checkouts about + # how to report a directory/file conflict during a reset. + # This command would fail with test_all_match because the + # full checkout reports "T folder1/0/1" while a sparse + # checkout reports "D folder1/0/1". This matches because + # the sparse checkouts skip "adding" the other side of + # the conflict. + test_sparse_match git reset --mixed HEAD~1 && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 && + test_sparse_match git status --porcelain=v2 && + + # At this point, sparse-checkouts behave differently + # from the full-checkout. + test_sparse_match git checkout --orphan new-branch && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 && + test_sparse_match git status --porcelain=v2 +' + +test_expect_success 'add everything with deep new file' ' + init_repos && + + run_on_sparse git sparse-checkout set deep/deeper1/deepest && + + run_on_all touch deep/deeper1/x && + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git status --porcelain=v2 +' + test_done diff --git a/wt-status.c b/wt-status.c index 0425169c1895..90db8bd659fa 100644 --- a/wt-status.c +++ b/wt-status.c @@ -654,6 +654,34 @@ static void wt_status_collect_changes_index(struct wt_status *s) run_diff_index(&rev, 1); } +static int add_file_to_list(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct string_list_item *it; + struct wt_status_change_data *d; + struct wt_status *s = context; + char *full_name; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + full_name = xstrfmt("%s%s", base->buf, path); + it = string_list_insert(&s->change, full_name); + d = it->util; + if (!d) { + CALLOC_ARRAY(d, 1); + it->util = d; + } + + d->index_status = DIFF_STATUS_ADDED; + /* Leave {mode,oid}_head zero for adds. */ + d->mode_index = mode; + oidcpy(&d->oid_index, oid); + s->committable = 1; + return 0; +} + static void wt_status_collect_changes_initial(struct wt_status *s) { struct index_state *istate = s->repo->index; @@ -668,6 +696,28 @@ static void wt_status_collect_changes_initial(struct wt_status *s) continue; if (ce_intent_to_add(ce)) continue; + if (S_ISSPARSEDIR(ce->ce_mode)) { + /* + * This is a sparse directory entry, so we want to collect all + * of the added files within the tree. This requires recursively + * expanding the trees to find the elements that are new in this + * tree and marking them with DIFF_STATUS_ADDED. + */ + struct strbuf base = STRBUF_INIT; + struct pathspec ps; + struct tree *tree = lookup_tree(istate->repo, &ce->oid); + + memset(&ps, 0, sizeof(ps)); + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_add(&base, ce->name, ce->ce_namelen); + read_tree_at(istate->repo, tree, &base, &ps, + add_file_to_list, s); + continue; + } + it = string_list_insert(&s->change, ce->name); d = it->util; if (!d) { From patchwork Mon Jun 7 12:34:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12303469 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E5684C47094 for ; Mon, 7 Jun 2021 12:35:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CD29561208 for ; Mon, 7 Jun 2021 12:35:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231218AbhFGMhR (ORCPT ); Mon, 7 Jun 2021 08:37:17 -0400 Received: from mail-wr1-f50.google.com ([209.85.221.50]:47045 "EHLO mail-wr1-f50.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231214AbhFGMhO (ORCPT ); Mon, 7 Jun 2021 08:37:14 -0400 Received: by mail-wr1-f50.google.com with SMTP id a11so15541402wrt.13 for ; Mon, 07 Jun 2021 05:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HVK7x9ytblDj8iy4lmiMNtwNDpdpLC/uAkky1UzqoLw=; b=aSInP3ilRr1tDZqtgIQMi80zEIcR85QGbYpblwuc6Yh8LnCzeY17dd1e9Z9KnvE1Rg l5bvX+0PIwUOM1AI8bQRHTW7F0e7vVZUrpG0/JAFvbq6eS47TkbtCLYMMeKxNGIU7lXr EtA4DcvbhHPeo94yWcYAnFdvGEdrR7qMZvGbq8o2W5HsYke61U08HyWZMPnuaiy9fXDJ GjckFJMHXbi3ypZf6GaNiDjl3D4IL5TAiN4FVoxNvghoOE9k988TSy0h86KOgjaCdRtn DqXc9A+VCGPT/iFJhoNSjqZYyhxet2xBpHiV2X+q3+plJnRA1fkJZvKq569Dx4B6ojka R4JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HVK7x9ytblDj8iy4lmiMNtwNDpdpLC/uAkky1UzqoLw=; b=PE2QcDebtbJIGfy6lzUI33pZso1wRCrCUzXFkd8egP5iwp7S8UMKCeO7RmAMp3ixZw GL9TeD4G65jx96TLwdnaZgBikO6+8RRqhmON9Vs9IxiKL/dyGSfqz/VkotJpQ5AM7eDm kqyAWx9mpMKWhJP8hZAm/b1jCDHopi/1F7RWIPrA9YTsE3dRlJmPuFkes4JNixwd8BmH eO11vU+dRmb5PiDgvf6OU9TEJ5HZpSCmpv/d5Ice4seNF3B7ozMvnIThh09XeXmUYK+9 0NYmSDFSNT0p74c73USQG/A+PiIVrJ29T2CPr6kjDwGjbLLq6GvzmF06IAx6xp5qZs+B ouBg== X-Gm-Message-State: AOAM533TEF611Pk0jdECDzS0sRX4Y1zy0r/sUM9iDHOQuKFVyQvPe90O tzdqKNmsKcSvAbvhZLyb2r+gA0GlegE= X-Google-Smtp-Source: ABdhPJwSNLUTumNDVl9ssY08bFXrHLT8R3WYlSPYmLsGAoC5qJ3pbt6bt+g0nHSYZCAfJHtQccq8Ig== X-Received: by 2002:a05:6000:50d:: with SMTP id a13mr16509196wrf.130.1623069262434; Mon, 07 Jun 2021 05:34:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l2sm15535437wrp.21.2021.06.07.05.34.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 07 Jun 2021 05:34:22 -0700 (PDT) Message-Id: <46ca150c35486f6d760f03a5e1f896ac9d4e93d7.1623069253.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 07 Jun 2021 12:34:12 +0000 Subject: [PATCH v5 14/14] fsmonitor: integrate with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we need to expand a sparse-index into a full one, then the FS Monitor bitmap is going to be incorrect. Ensure that we start fresh at such an event. While this is currently a performance drawback, the eventual hope of the sparse-index feature is that these expansions will be rare and hence we will be able to keep the FS Monitor data accurate across multiple Git commands. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- sparse-index.c | 7 ++++++ t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index b2b3fbd75050..32ba0d17ef7c 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -195,6 +195,10 @@ int convert_to_sparse(struct index_state *istate) cache_tree_free(&istate->cache_tree); cache_tree_update(istate, 0); + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -291,6 +295,9 @@ void ensure_full_index(struct index_state *istate) istate->cache = full->cache; istate->cache_nr = full->cache_nr; istate->cache_alloc = full->cache_alloc; + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); strbuf_release(&base); free(full); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 45d025f96010..f70fe961902e 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -73,6 +73,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Usage: +# check_sparse_index_behavior [!] +# If "!" is supplied, then we verify that we do not call ensure_full_index +# during a call to 'git status'. Otherwise, we verify that we _do_ call it. +check_sparse_index_behavior () { + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set dir1 dir2 && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region $1 index ensure_full_index trace2.txt && + test_cmp expect actual && + rm trace2.txt && + git sparse-checkout disable +} + +test_expect_success 'status succeeds with sparse index' ' + git reset --hard && + + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + check_sparse_index_behavior ! && + + cp -r dir1 dir1a && + git add dir1a && + git commit -m "add dir1a" && + + # This one modifies outside the sparse-checkout definition + # and hence we expect to expand the sparse-index. + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + check_sparse_index_behavior +' + test_done