From patchwork Tue Jun 29 01:51:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 537F1C11F66 for ; Tue, 29 Jun 2021 01:51:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3CAE861CD7 for ; Tue, 29 Jun 2021 01:51:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231133AbhF2ByP (ORCPT ); Mon, 28 Jun 2021 21:54:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46934 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229936AbhF2ByN (ORCPT ); Mon, 28 Jun 2021 21:54:13 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CACDC061760 for ; Mon, 28 Jun 2021 18:51:46 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id u11so23618187wrw.11 for ; Mon, 28 Jun 2021 18:51:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cO98IPJ5A+rusUCsSXexJohE9wZWdkoHGqzHKeEQ398=; b=YcBzHoRBsFr3k6chpG+8uP5GuuhrZlagFipIRZvJu/4q/PJG1cYVIPtnGJQvIxH0DO 6RI7jfUYaFLzVnHqXi0RV9mq9p0pz5qy+WSlJBdhvyDqk58e5ZRFm3moY2JAjGJI9MUQ nc7dlgPIyPZWO82DtJMCYXllxnPNrqwEiFpY8vnQYaU3yCXjg/vmnPRm2KNH3dX9cB9o eBeRgY+z6nGH9yhOefknL9SEPxTQjjN4Wr6D2nGl8RVsi1eqgMgBel9NtYtTm2303zK1 s2ecb61cOVeMjwWxMz5AjnBIk+BiPaQSiufyjizHASyosGuSo/lYQ4eIf78IGAudOkyI UC5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cO98IPJ5A+rusUCsSXexJohE9wZWdkoHGqzHKeEQ398=; b=XMKK+wjKXxalbaqN6ZUyNQMa6fJko5j31XpZUbhjGRP2VT4zNKkcLJSwXh+efI/tt7 GiUMYYvyMiZwjn6HK4zfO/dqTfNcm3FvUU7ZM7Ot4n1FClZoutC91vNvtnZjaITJRy9+ FezdrXS+Mgyzv5HGwzgoE5GJWVXNuPRt0kzjy0Zy2SiqXGiObpnfzwOUa3HETLAUigqd NZFNAll25W33rguKjUvJiG0phdvRdiR5xx/wCkaUt/JilcgauqUAbHLTaZtFIYcTlTt/ wpyNqL3oA0/+JxQaSlX3dEN+d8Pm6qQi4JLwoKVVXHhhJ1/WNOwesLOhtrE3z3tCpv2h p9DQ== X-Gm-Message-State: AOAM530SgEDdejB4jnSop8FYECciBRzOp2NQx+qosT4f2z+0cPAxUnPY 52hMEBdgwO9h3IRFEjHkRJ6jjrtdU8Q= X-Google-Smtp-Source: ABdhPJzOG4ZO7XVNAaun0se3YjUyI0bGRtx4xWKMFvUb7PrErenBdkGhT/3gcPPNi34l76DKDwlX+g== X-Received: by 2002:a5d:6a51:: with SMTP id t17mr5550912wrw.255.1624931504621; Mon, 28 Jun 2021 18:51:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t9sm15479143wmq.14.2021.06.28.18.51.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:44 -0700 (PDT) Message-Id: <2a4a7256304c44f168baf54ff1bdc191be67408f.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:29 +0000 Subject: [PATCH v6 01/14] sparse-index: skip indexes with unmerged entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse-index format is designed to be compatible with merge conflicts, even those outside the sparse-checkout definition. The reason is that when converting a full index to a sparse one, a cache entry with nonzero stage will not be collapsed into a sparse directory entry. However, this behavior was not tested, and a different behavior within convert_to_sparse() fails in this scenario. Specifically, cache_tree_update() will fail when unmerged entries exist. convert_to_sparse_rec() uses the cache-tree data to recursively walk the tree structure, but also to compute the OIDs used in the sparse-directory entries. Add an index scan to convert_to_sparse() that will detect if these merge conflict entries exist and skip the conversion before trying to update the cache-tree. This is marked as NEEDSWORK because this can be removed with a suitable update to cache_tree_update() or a similar method that can construct a cache-tree with invalid nodes, but still allow creating the nodes necessary for creating sparse directory entries. It is possible that in the future we will not need to make such an update, since if we do not expand a sparse-index into a full one, this conversion does not need to happen. Thus, this can be deferred until the merge machinery is made to integrate with the sparse-index. Signed-off-by: Derrick Stolee --- sparse-index.c | 18 ++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 22 ++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index affc4048f27..2c695930275 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -116,6 +116,17 @@ int set_sparse_index_config(struct repository *repo, int enable) return res; } +static int index_has_unmerged_entries(struct index_state *istate) +{ + int i; + for (i = 0; i < istate->cache_nr; i++) { + if (ce_stage(istate->cache[i])) + return 1; + } + + return 0; +} + int convert_to_sparse(struct index_state *istate) { int test_env; @@ -152,6 +163,13 @@ int convert_to_sparse(struct index_state *istate) return -1; } + /* + * NEEDSWORK: If we have unmerged entries, then stay full. + * Unmerged entries prevent the cache-tree extension from working. + */ + if (index_has_unmerged_entries(istate)) + return 0; + if (cache_tree_update(istate, 0)) { warning(_("unable to update cache-tree, staying full")); return -1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index e9a815ca7aa..ba2fd94adaf 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -352,6 +352,28 @@ test_expect_success 'merge with outside renames' ' done ' +# Sparse-index fails to convert the index in the +# final 'git cherry-pick' command. +test_expect_success 'cherry-pick with conflicts' ' + init_repos && + + write_script edit-conflict <<-\EOF && + echo $1 >conflict + EOF + + test_all_match git checkout -b to-cherry-pick && + run_on_all ../edit-conflict ABC && + test_all_match git add conflict && + test_all_match git commit -m "conflict to pick" && + + test_all_match git checkout -B base HEAD~1 && + run_on_all ../edit-conflict DEF && + test_all_match git add conflict && + test_all_match git commit -m "conflict in base" && + + test_all_match test_must_fail git cherry-pick to-cherry-pick +' + test_expect_success 'clean' ' init_repos && From patchwork Tue Jun 29 01:51:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348735 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07B72C11F64 for ; Tue, 29 Jun 2021 01:51:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D96ED61CD5 for ; Tue, 29 Jun 2021 01:51:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231225AbhF2ByS (ORCPT ); Mon, 28 Jun 2021 21:54:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46938 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230341AbhF2ByO (ORCPT ); Mon, 28 Jun 2021 21:54:14 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83D1CC061766 for ; Mon, 28 Jun 2021 18:51:46 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id y3so4253580wrq.3 for ; Mon, 28 Jun 2021 18:51:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=iy9XiTv1ciKF1QkjuiSZgTtadS+h+8sdv46VtmcVxm4=; b=XK5hNi0BsPsxedEsu/3uUI6HlE1MKGQy5QyjM5qjPirT5B5IVyOtUjUaEaiCxmPBXA HDKm7pR+ncxDM1tbn/nYj9T2lIZ99vszA30r6U+MrDx1mwcAljPDg6KdHfdkyB0gYuIc eBV/yhptpwlNcdW+pbo6RIMtxGBnuDdnh8iyss2N/tiT4wsiBc5TTw0CzYcJbvB4ESmW 8y0b6N1jKyB4rg7lD68KxbREX+0AP80lAtd+LqfH2t0eZhL72M/T8HMhoE3qMJtO/PLj +PxZzxGLkC+PFUSzTtOOJ+M4NC7EkbEre7TJFXDa35LruDLZ6ftarI6LJ+EiidvyNs4/ 0eSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=iy9XiTv1ciKF1QkjuiSZgTtadS+h+8sdv46VtmcVxm4=; b=XJS6MtTy4BXFT+llErqt42rD57jMMGahEzJXDobHRRuY32xMWO06lKFXkJe5t8kA9/ /AO+S17w3JRb56+q+16SkTJYzKIsAd8BFwNhbNzZ0zqjGvBk+DYiIbdNr36aYQjivv32 sxep5sWV2rMtd6YuYJ5qHpn+/CpAld1fKj/EmNuGwuyvqmZ4opX8qXnRUGZZWRdLd/RJ zZh5LMiqgIjeZqGLG+t3sqDSZ5pkMTDd1boDoruJ1GrodU8LbWyW1Uj5G6dEKJoeeP+X 1/EpQTZ4x0eWfiKqu2miY2R2Z4bUISzXDRnTXTfUGB1iDpvWIPhNFNdfxOlyeKz2/R6M ThSQ== X-Gm-Message-State: AOAM533wZyokDb/Fm1vuccXya9jz0GlXbTbyPWM1h9jwxxg6WAa/B96y trL6IktE7LJfMfIqyIYKd1e+5CGaxVM= X-Google-Smtp-Source: ABdhPJxxBLQTJ/doqIpncTiVoxZhGlDhHEOcpn+n84fjN0bDrAbmRoysiDVNSVTkImxVw7EDWxvfNQ== X-Received: by 2002:a5d:5685:: with SMTP id f5mr22680017wrv.101.1624931505176; Mon, 28 Jun 2021 18:51:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j35sm1046431wms.7.2021.06.28.18.51.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:44 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:30 +0000 Subject: [PATCH v6 02/14] sparse-index: include EXTENDED flag when expanding Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When creating a full index from a sparse one, we create cache entries for every blob within a given sparse directory entry. These are correctly marked with the CE_SKIP_WORKTREE flag, but the CE_EXTENDED flag is not included. The CE_EXTENDED flag would exist if we loaded a full index from disk with these entries marked with CE_SKIP_WORKTREE, so we can add the flag here to be consistent. This allows us to directly compare the flags present in cache entries when testing the sparse-index feature, but has no significance to its correctness in the user-facing functionality. Signed-off-by: Derrick Stolee --- sparse-index.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index 2c695930275..ef53bd2198b 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -213,7 +213,7 @@ static int add_path_to_index(const struct object_id *oid, strbuf_addstr(base, path); ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); - ce->ce_flags |= CE_SKIP_WORKTREE; + ce->ce_flags |= CE_SKIP_WORKTREE | CE_EXTENDED; set_index_entry(istate, istate->cache_nr++, ce); strbuf_setlen(base, len); From patchwork Tue Jun 29 01:51:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348737 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83BE1C11F67 for ; Tue, 29 Jun 2021 01:51:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 69EAA61CD7 for ; Tue, 29 Jun 2021 01:51:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231317AbhF2ByU (ORCPT ); Mon, 28 Jun 2021 21:54:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230386AbhF2ByO (ORCPT ); Mon, 28 Jun 2021 21:54:14 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 192F5C061767 for ; Mon, 28 Jun 2021 18:51:47 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id y3so4253618wrq.3 for ; Mon, 28 Jun 2021 18:51:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=M5y7V+S57JkqDlEiYCsuFlFYkP2WRg1BWKCFbReDZCs=; b=SQbbAw2BI1W2Mrli/0l+ZwhzppYYdfgjRS1NdB2o5a9tb7M7bVJUGkuqj5f7zfuToM PhE6ISQAaF8hYBHaitIymHpilxQNaEF7MSDCNHtuiSX2U6f8HQWvDoknybPKWzqvN1oZ v+t57oHJRFYO9S2J71+Utrz351UCYNuhPWqUvAiTOZ02AcEkpjjlEuqZI77wGvGo3xFw 5kXwgV22WJx5c41qtTv96E/Xoz8dWi48e9OmQbPo78QPXR9IKGA5WsJy04zfWenwDUQb 0xdxlMKHh6D1VfcI6I9I2SLhl8CFCYuTRw25iijWchATRah7o4qgtqusQRR5mKZDU0AA iwNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=M5y7V+S57JkqDlEiYCsuFlFYkP2WRg1BWKCFbReDZCs=; b=SaCVq1IPlyYztadyPJjH8NQ24shLMiHnYF0I3DJWgTBSXBy9RILI9Ed4KBZAUSSNQs 8CxP2jkpCD9m7vyHxZl9h9GzslzjZaSG/X2DCBV7KHKQxeJa5Jvxze0/kakx664qPatq M/bE9uwwL4TkSX+xkOcvYwUjzQgyzRqVYFywKHzrHjoSS7Z8Xc4JWWFlAnEXi3q5oI0v SEcc03mp5lTydLgyrnxk5ORPcwy3JfMYkJybRpevHI9IS7WHejBopD3YOHZXTR/GvH4p uENkg7742gIm7ktf1CuOk8fwhoq6nHVUnfYPtgT1nLWyhGblIoiQon478f0MAQIB+aOt c6AA== X-Gm-Message-State: AOAM533hfWbBSs/N9j6s9BCyLLbaEXd2r38Jv4ykVDAZ5OITtxC3tWDL Q6UuLboSX5tcPXI6aSROHwrX5i8s4xA= X-Google-Smtp-Source: ABdhPJwcACBok9UoygOYoiFcTNKz/Rgy/3ve1PQzdowwT/Qq9MRCBoik7bOa7Ai8MEYqk8OBrOVoXQ== X-Received: by 2002:a5d:58c1:: with SMTP id o1mr29836140wrf.367.1624931505748; Mon, 28 Jun 2021 18:51:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w9sm16340522wru.3.2021.06.28.18.51.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:45 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:31 +0000 Subject: [PATCH v6 03/14] t1092: replace incorrect 'echo' with 'cat' Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This fixes the test data shape to be as expected, allowing rename detection to work properly now that the 'larger-content' file actually has meaningful lines. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index ba2fd94adaf..ebbba044f77 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -40,7 +40,7 @@ test_expect_success 'setup' ' done && git checkout -b rename-base base && - echo >folder1/larger-content <<-\EOF && + cat >folder1/larger-content <<-\EOF && matching lines help From patchwork Tue Jun 29 01:51:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348739 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54BD8C11F64 for ; Tue, 29 Jun 2021 01:51:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3159C61CD5 for ; Tue, 29 Jun 2021 01:51:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231559AbhF2ByY (ORCPT ); Mon, 28 Jun 2021 21:54:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230451AbhF2ByP (ORCPT ); Mon, 28 Jun 2021 21:54:15 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A3879C061574 for ; Mon, 28 Jun 2021 18:51:47 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id p10-20020a05600c430ab02901df57d735f7so686653wme.3 for ; Mon, 28 Jun 2021 18:51:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Yb2PnTjyl04QScNrCyVgl4dORDLdN86qbe9qCleSmYI=; b=dseMkZtjQ0fRuIJUEDWZeizu+ZfsZYyb18KRzujCu16TEROWX0GU0r7yO62zj4Gk1+ GvVTm/UpVvwXy3yge5EAjVffo0p9AiQ6hft2xK+nQaq78sJrLMuxUDK3V+CjTph2+DRm 1elYJO/TuuIEBYd1IQDxIF+qtS1MFFce3tsnxY9CUw0DZ+oRfmWD4zwZfBbOx9VKhY8w SnkV6g1J+8EVgR4vTadMxsr3A+AYAF8YZsKQvs8Q/IWA3y1hyBKr8MT3D9RRhanax5fm bCw0K+J8lefH08bLe6JqBdM8yGkmHla9zH+T5nZOJZD2PbKX5IWr2v206F+X7WJDmrx8 dDKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Yb2PnTjyl04QScNrCyVgl4dORDLdN86qbe9qCleSmYI=; b=TdrqjT5h642lNyFXS2EiUKtn0ho81aDykBYNo0+hOw1RnsNnauk4G+dZ4h75fvJx7B q8xjVxUJAYuSLBQnGB5OWVGx+EBwoF7cM9wS/1K0HQTpY+VLtMiw9qxSYiwq3HQWnFAv ktEmETKsn1Q5E63N3Ozr7z7aKMaKbGtvwEqbLPL2zoM1hjifgWoiFTYzwWaSy13dO7YD V1K/oxKwq64lFFqKR57ILIwQonf8OS5MuRutjTCTDV7lRl0Kg1KivRYwQ1DAEQyESJLb S6QS2Sk0fc7dOX8lu2MRKhJIYcVNtB/+5OmeExJpFwzi4uEjeMFehKV0FnxsXcUZWAjd j42A== X-Gm-Message-State: AOAM5329AwXcEeMYs+F94Lw/DeZXdC4N6Q025Ru1mNgdzl1YLbZu/fVz Zd21xjh61FK4g6zrXNmhLyY784kP87M= X-Google-Smtp-Source: ABdhPJxg7J8GGoFcOMUUe4jiSmRXYQ4xAEmLPeOpC3BgZ6nhgClO9MxdYzYdT7L2wd284Qnn55H/0Q== X-Received: by 2002:a05:600c:2482:: with SMTP id 2mr29274474wms.67.1624931506284; Mon, 28 Jun 2021 18:51:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p7sm4115605wrr.68.2021.06.28.18.51.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:46 -0700 (PDT) Message-Id: <44a940211b2fedc4a65b6dc2379e99c29cc5b64f.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:32 +0000 Subject: [PATCH v6 04/14] t1092: expand repository data shape Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As more features integrate with the sparse-index feature, more and more special cases arise that require different data shapes within the tree structure of the repository in order to demonstrate those cases. Add several interesting special cases all at once instead of sprinkling them across several commits. The interesting cases being added here are: * Add sparse-directory entries on both sides of directories within the sparse-checkout definition. * Add directories outside the sparse-checkout definition who have only one entry and are the first entry of a directory with multiple entries. Later tests will take advantage of these shapes, but they also deepen the tests that already exist. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 39 ++++++++++++++++++++++-- 1 file changed, 37 insertions(+), 2 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index ebbba044f77..4a7b48e8d3f 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -17,7 +17,7 @@ test_expect_success 'setup' ' echo "after folder1" >g && echo "after x" >z && mkdir folder1 folder2 deep x && - mkdir deep/deeper1 deep/deeper2 && + mkdir deep/deeper1 deep/deeper2 deep/before deep/later && mkdir deep/deeper1/deepest && echo "after deeper1" >deep/e && echo "after deepest" >deep/deeper1/e && @@ -25,10 +25,20 @@ test_expect_success 'setup' ' cp a folder2 && cp a x && cp a deep && + cp a deep/before && cp a deep/deeper1 && cp a deep/deeper2 && + cp a deep/later && cp a deep/deeper1/deepest && cp -r deep/deeper1/deepest deep/deeper2 && + mkdir deep/deeper1/0 && + mkdir deep/deeper1/0/0 && + touch deep/deeper1/0/1 && + touch deep/deeper1/0/0/0 && + cp -r deep/deeper1/0 folder1 && + cp -r deep/deeper1/0 folder2 && + echo >>folder1/0/0/0 && + echo >>folder2/0/1 && git add . && git commit -m "initial commit" && git checkout -b base && @@ -56,11 +66,17 @@ test_expect_success 'setup' ' mv folder1/a folder2/b && mv folder1/larger-content folder2/edited-content && echo >>folder2/edited-content && + echo >>folder2/0/1 && + echo stuff >>deep/deeper1/a && git add . && git commit -m "rename folder1/... to folder2/..." && git checkout -b rename-out-to-in rename-base && mv folder1/a deep/deeper1/b && + echo more stuff >>deep/deeper1/a && + rm folder2/0/1 && + mkdir folder2/0/1 && + echo >>folder2/0/1/1 && mv folder1/larger-content deep/deeper1/edited-content && echo >>deep/deeper1/edited-content && git add . && @@ -68,6 +84,9 @@ test_expect_success 'setup' ' git checkout -b rename-in-to-out rename-base && mv deep/deeper1/a folder1/b && + echo >>folder2/0/1 && + rm -rf folder1/0/0 && + echo >>folder1/0/0 && mv deep/deeper1/larger-content folder1/edited-content && echo >>folder1/edited-content && git add . && @@ -262,13 +281,29 @@ test_expect_success 'diff --staged' ' test_all_match git diff --staged ' -test_expect_success 'diff with renames' ' +test_expect_success 'diff with renames and conflicts' ' init_repos && for branch in rename-out-to-out rename-out-to-in rename-in-to-out do test_all_match git checkout rename-base && test_all_match git checkout $branch -- .&& + test_all_match git status --porcelain=v2 && + test_all_match git diff --staged --no-renames && + test_all_match git diff --staged --find-renames || return 1 + done +' + +test_expect_success 'diff with directory/file conflicts' ' + init_repos && + + for branch in rename-out-to-out rename-out-to-in rename-in-to-out + do + git -C full-checkout reset --hard && + test_sparse_match git reset --hard && + test_all_match git checkout $branch && + test_all_match git checkout rename-base -- . && + test_all_match git status --porcelain=v2 && test_all_match git diff --staged --no-renames && test_all_match git diff --staged --find-renames || return 1 done From patchwork Tue Jun 29 01:51:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348741 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5061EC11F64 for ; Tue, 29 Jun 2021 01:52:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A37061CCE for ; Tue, 29 Jun 2021 01:52:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231347AbhF2ByZ (ORCPT ); Mon, 28 Jun 2021 21:54:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231139AbhF2ByQ (ORCPT ); Mon, 28 Jun 2021 21:54:16 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45C83C061760 for ; Mon, 28 Jun 2021 18:51:48 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id j1so23627856wrn.9 for ; Mon, 28 Jun 2021 18:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7JZLdoqa+U8o4yiwGgoo7PP+W0UjuNrjA0Zez9LhlL0=; b=UKlX1EVg+Amzc0bavReHXEq/CoGdPXtM96g21GHuvN08CPXmqz4ielAymdTz/qJSl6 oVU7V+3cfhAHVo6dtWAAC888DBjj8mME0DFTOiP5XBxJvf1HuVHjVjRYbQbOTIWWAvr0 /soqNbEeVnZcEthrLXCWqGoHavpE+oU8sx1sXV/qIP5q21pTYs5mxhW1HxPVnqrQfLAz lg8kE08FLzP98qDoyUUYQv8qBjHeX15n5iRxgzqw2UisGLHGg44SBbwZYJifavjOf6XV jm7cnP4G/r4sSU5ZDE9aNucWJE5coklFFyw84VTJhwJsZfy/UkQc0NLojhYtolM352g9 xRKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7JZLdoqa+U8o4yiwGgoo7PP+W0UjuNrjA0Zez9LhlL0=; b=sqCB2pMeJMwI+9vLup2h95laa91Nsm/vaPX711xqmBss3pY49ot5RT3sWTe6Dnry4A aaQ/e1AUs9rRgtiF2v7HwbboQVdbQhJS/FEmZljYDdRSvxKVtg3yDubGR1TsFUoIyTkQ LvPvVHuGdOT4QVHVK4sNb15cMWxQA+qYiAW3rfeHkp6Qxym+C3xTluxLwvcMIBqsjUbN ZbIvel0mfOvuNjiFBM9JVSJDfPkXnwlC+OUPi1/tj9HLES2+b74gKUMMV9MEdGZt05gB Jwrv1zeWi6ZIuPIOuV6x13bmRWKwUoPddOPg8uLmOcpMS3hkzZRM1YRmZJMyOsX1VToC WFDA== X-Gm-Message-State: AOAM531FYgZumPxjXQ/28/atSuO9PiG3T47Y88Jk1vlfxjs02peZBs0d xXvoQ6tHepp/RjnQ5HUlpfRWXlfZnWI= X-Google-Smtp-Source: ABdhPJzQNrofL23pa5lWLRVFYwaFB6PvWMrApQ52FfK6MvGBcguCoRTy5dXK18qv7RylBPEAyFkrcg== X-Received: by 2002:adf:ce07:: with SMTP id p7mr7109560wrn.114.1624931506965; Mon, 28 Jun 2021 18:51:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s18sm16862439wrw.33.2021.06.28.18.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:46 -0700 (PDT) Message-Id: <701ac0e8ff6331284e63047bceed98d1219daa47.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:33 +0000 Subject: [PATCH v6 05/14] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file is not shown as modified and adding it will fail. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 38 ++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 4a7b48e8d3f..7c78e40b861 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -251,6 +251,44 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # adding a "missing" file outside the cone should fail + test_sparse_match test_must_fail git add folder1/a && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_sparse ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" fails with a warning + # in the sparse repos, differing from the full + # repo. This is intentional. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Tue Jun 29 01:51:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348745 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31766C11F64 for ; Tue, 29 Jun 2021 01:52:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1A48C61CCE for ; Tue, 29 Jun 2021 01:52:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231682AbhF2Bya (ORCPT ); Mon, 28 Jun 2021 21:54:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231168AbhF2ByQ (ORCPT ); Mon, 28 Jun 2021 21:54:16 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CD2F6C061766 for ; Mon, 28 Jun 2021 18:51:48 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id u8so10383296wrq.8 for ; Mon, 28 Jun 2021 18:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=20VyvcZukH9mBP0oxKGuPMDI0uq7we8hBvlg4nNNWng=; b=d0NbtkvqLn+4u8ddmOxSg/RcpqPdfdf4sGL4n6jTkn9hu+ficL5vTxGg1yNwRAo6xd IB6jskBmPWKGWj0ESoaWuHkcZAZQmY41O1sqgi1P2pWae2oltbN8bi4I22fq7pMB8N9a oj9N3m7nSgvpAtA0LlUkX72EqfPvTm7K9ifJIJbjQnBwMxeMb1Nw6UkFqZAQJITH/lcX K9VteRjmW9tiFT32ruLf3FVKR7E7Gg1RP5rccMutE24YDVdc6D9pDj+2ARlR73fM80zn NgH6N7lQzwy23heqP2BkrtMJgXqMIzQkdGS1zfITPlC1NWGq+n4DI+txkb+aYkprcJbB W6qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=20VyvcZukH9mBP0oxKGuPMDI0uq7we8hBvlg4nNNWng=; b=Ctl8XDQ9wcKlBlQHTb4Ep0YS1P+zESDpb+bdK26Q8vMR5/Gr88GEmJm5NCtV6gBpNt lQwbf0+0hZsk3Tlg4y21ffHv+IhBHvA3FXcdJ5jqGAvGEshQwAGz/gOyk+DtSNH7KBWR kJVYelk0EluUQjVK/cGLXJVz7l+EcqGwmxNkStLcb2108VHQMIIkFNsRwLXC7fCQ/jIg ReXUe5CZV2AQWLMzkZ9rtZkBATgs53SBx6RJd8P+GCF0T5BnASf1TkJD2p/qVvthxe5b kxXRtUx6cOgMoSXN5W4CkJ87U34GYg0mVC8MNZBd32GXVNdAvgrerm0CyWbyyHmv+pWz FjLg== X-Gm-Message-State: AOAM530Lpslxz66V4txX02UmmFmxcHp+uY2eWCuH2J7Y3+JilGRejO3Z d8fLwlXYdrZdzk4ZBqbCUW6lXucw7lE= X-Google-Smtp-Source: ABdhPJz0it1XoUXp2Uewtvakthi0WvZ/EFYl8YcsV1S/25q/BHCsuuYbH+d1vTqAefvuYUZolRrffA== X-Received: by 2002:adf:e38c:: with SMTP id e12mr31034469wrm.404.1624931507488; Mon, 28 Jun 2021 18:51:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h46sm18152705wrh.44.2021.06.28.18.51.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:47 -0700 (PDT) Message-Id: <587333f7c618f8eb65f7a25d1a4ab02f9ace92fd.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:34 +0000 Subject: [PATCH v6 06/14] unpack-trees: preserve cache_bottom Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_bottom member of 'struct unpack_trees_options' is used to track the range of index entries corresponding to a node of the cache tree. While recursing with traverse_by_cache_tree(), this value is preserved on the call stack using a local and then restored as that method returns. The mark_ce_used() method normally modifies the cache_bottom member when it refers to the marked cache entry. However, sparse directory entries are stored as nodes in the cache-tree data structure as of 2de37c53 (cache-tree: integrate with sparse directory entries, 2021-03-30). Thus, the cache_bottom will be modified as the cache-tree walk advances. Do not update it as well within mark_ce_used(). Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index f88a69f8e71..87c1ed204c8 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -600,6 +600,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; From patchwork Tue Jun 29 01:51:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348743 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82C48C11F65 for ; Tue, 29 Jun 2021 01:52:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6982661CCE for ; Tue, 29 Jun 2021 01:52:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231392AbhF2By2 (ORCPT ); Mon, 28 Jun 2021 21:54:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231216AbhF2ByR (ORCPT ); Mon, 28 Jun 2021 21:54:17 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 616E5C061767 for ; Mon, 28 Jun 2021 18:51:49 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id l8so14891343wry.13 for ; Mon, 28 Jun 2021 18:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=H3+gKjSmAWwdDpD6Q8E+pdKb8nC0l5EqTOcc9rWsUbM=; b=ckNrCAeZbIcJBEza7MtFilLRpIu6WRSVo9bivcr0F6CXjoDdixuDLjgsFS6HR9aCtc yzsiyEoaDrZuK3tMG57fYAqTSziAx6k4dvRrrJXD+XbRlaRLimr+uMsqHWU1lSEPDSIt s66PfTCHSpN+ZA2bOe+qLHcnLdccee+IO3ir9Q6py3WpwjLLGj48M2uqixLUB8wFaq56 6krSxOv/SEpAJtR390psaZ9et4iFpcq9qdbp26M+qlWyvSZWtbdYGQzLy1ndl0QdVK2H SHu8u5Wa2Fr7pQXcfPofu5BICIKokrG8hDvvP99YoX+DiM3KAqanetamMpRudMSOx4rZ 3X5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=H3+gKjSmAWwdDpD6Q8E+pdKb8nC0l5EqTOcc9rWsUbM=; b=VDFeXzTW052W2swDGWpmYiUC1QJThjVlZf5wGmTTuXLDrLyByOfWaDlKhZq46/FJoC 3AfuOf3PIyLzuC47gXK58nR6VDm7QpIMpH9F4KNX1ymt2bA5I2hXlI0UxoUt+CXXZV/9 TJ/lE2MQynXAwI9+uQ1ptzLRUuQ1t2GCVeyL3IrtYD7OjjgROWAV/ejlPBCQoTNSni9o bmnhxiQYxnWeTv1AnVIwAHNiHVUuMTTbyadJx+QhrKLCBj7napDa8lRJRKWlSm7dF2TK 37cnLzY06JQlNnDYBH+Lnk4MI+zWHyElfcITlLEjA6KlttsdFkRJvZUwvX8zgODv8A1N LbQA== X-Gm-Message-State: AOAM532GSp/axiFF7KxwdMBL6BDSUKQvrn8RUCps5QBSS0yGknOLDRoi ARnCtzWV7oW4sKzUFigYbBTat91bAzM= X-Google-Smtp-Source: ABdhPJyI0EGTTbOc6CyD8q92O2WBUWBQ+avXPx935uX6iU1sC1rOcWbSi4ep08htyFeZHh1Ddmhheg== X-Received: by 2002:a5d:6d81:: with SMTP id l1mr29941168wrs.282.1624931508071; Mon, 28 Jun 2021 18:51:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 22sm14936776wmi.4.2021.06.28.18.51.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:47 -0700 (PDT) Message-Id: <6fc898ac23e244b3f9c1406badb89441945c6e18.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:35 +0000 Subject: [PATCH v6 07/14] unpack-trees: compare sparse directories correctly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we further integrate the sparse-index into unpack-trees, we need to ensure that we compare sparse directory entries correctly with other entries. This affects searching for an exact path as well as sorting index entries. Sparse directory entries contain the trailing directory separator. This is important for the sorting, in particular. Thus, within do_compare_entry() we stop using S_IFREG in all cases, since sparse directories should use S_IFDIR to indicate that the comparison should treat the entry name as a dirctory. Within compare_entry(), it first calls do_compare_entry() to check the leading portion of the name. When the input path is a directory name, we could match exactly already. Thus, we should return 0 if we have an exact string match on a sparse directory entry. Signed-off-by: Derrick Stolee --- unpack-trees.c | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 87c1ed204c8..7a507ddfe05 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -983,6 +983,7 @@ static int do_compare_entry(const struct cache_entry *ce, int pathlen, ce_len; const char *ce_name; int cmp; + unsigned ce_mode; /* * If we have not precomputed the traverse path, it is quicker @@ -1005,7 +1006,8 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG; + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode); } static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n) @@ -1014,6 +1016,15 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* + * At this point, we know that we have a prefix match. If ce + * is a sparse directory, then allow an exact match. This only + * works when the input name is a directory, since ce->name + * ends in a directory separator. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! From patchwork Tue Jun 29 01:51:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348747 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23096C11F65 for ; Tue, 29 Jun 2021 01:52:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 067C861CCE for ; Tue, 29 Jun 2021 01:52:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231724AbhF2Byc (ORCPT ); Mon, 28 Jun 2021 21:54:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231218AbhF2ByR (ORCPT ); Mon, 28 Jun 2021 21:54:17 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B494C061768 for ; Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id g198so4750827wme.5 for ; Mon, 28 Jun 2021 18:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CUntNiF3tA3T57fQ1h5xuY8xT0tmjHBndAZRrek1+qw=; b=jhERgwvGwmRinaIjUl46z3uPdqNDEL/eoxsh3ax7ue4YVt/q/OJ2wK5REcj+p+1yH8 4rN/tDGwrxYU4wTOJfc3iFNfUgmjhvnz9jcVpGzsaOprUx015WbtQsvWaXVXgnd7TYZC 5+uQuiw4yA9Hku/EVeByrUPmRi5bhMiYteBnyi1Kss7ocp5M17LaVxeZM/YBWMwxSo0T VSIru3mvfL9U5+w/yikBM3Gb77GPvFQS0dmxOnE0N7KCQh/Csh0lbHNeUyfXmv+89u87 Svg/FlUDeKdVnfqDz4+VzZP1G6fUrCXtkZx0ajtsNhXRnthIhiS7F9+xQqss7g/7056x JbzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CUntNiF3tA3T57fQ1h5xuY8xT0tmjHBndAZRrek1+qw=; b=FDSh4wBZYt51XCj8IW35z7fve7FaXtyt2b5eElGL+FQTF3WwDd9Lnk3vO1PBcqg4xD kiazuAxhh8YnR/iRXsaJlgMcudp+xLTtVv9Tgy8+Sngo/dZR6kP3/eaKN1128FGI8RYA IjvwtJA5X2bK6QdSBh+Gw0X4P5gIWT8lgCAUScVVL47lU7YK9/VvUItUZqC9lHBubTdA YNxVuLCmKQoWJTusaJ2jXD1QDItqyC8LUfW1RwAfKtGRxD2QH1QYt1zCn1j+oaV84Vlx 9UFg45GLoQ52qv1DPZtiZXWmKJb6TAR+HRWstuPXu397sX4WZ96axXu8mvAl4Q8+9qCw S4Mg== X-Gm-Message-State: AOAM532RVVigkx7YokOF511eaIyMshHYQMJIWRlv5RZ7zCSvnGlp1joz DOSq0Oc+jLUxsDvt1jbAes3zPYSso8I= X-Google-Smtp-Source: ABdhPJxmQ6vzbQT1FzirAwTneqVgQpG+YCWGetxSADSvXiVBmqOQZd/dZ76iXxSxDVPx9XtTHPsC4w== X-Received: by 2002:a05:600c:3650:: with SMTP id y16mr28743543wmq.25.1624931508638; Mon, 28 Jun 2021 18:51:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n29sm1017137wms.34.2021.06.28.18.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:48 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:36 +0000 Subject: [PATCH v6 08/14] unpack-trees: unpack sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee During unpack_callback(), index entries are compared against tree entries. These are matched according to names and types. One goal is to decide if we should recurse into subtrees or simply operate on one index entry. In the case of a sparse-directory entry, we do not want to recurse into that subtree and instead simply compare the trees. In some cases, we might want to perform a merge operation on the entry, such as during 'git checkout ' which wants to replace a sparse tree entry with the tree for that path at the target commit. We extend the logic within unpack_nondirectories() to create a sparse-directory entry in this case, and then that is sent to call_unpack_fn(). Since the name becomes confusing by handling directories, rename it to unpack_single_entry() since it handles a blob entry or a sparse directory entry without using traverse_trees_recursive(). There are some subtleties in this process. For instance, we need to update find_cache_entry() to allow finding a sparse-directory entry that exactly matches a given path. Use the new helper method sparse_dir_matches_path() for this. Signed-off-by: Derrick Stolee --- unpack-trees.c | 110 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 96 insertions(+), 14 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index 7a507ddfe05..d205a15d61f 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1051,13 +1051,15 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, const struct name_entry *n, int stage, struct index_state *istate, - int is_transient) + int is_transient, + int is_sparse_directory) { size_t len = traverse_path_len(info, tree_entry_len(n)); + size_t alloc_len = is_sparse_directory ? len + 1 : len; struct cache_entry *ce = is_transient ? - make_empty_transient_cache_entry(len, NULL) : - make_empty_cache_entry(istate, len); + make_empty_transient_cache_entry(alloc_len, NULL) : + make_empty_cache_entry(istate, alloc_len); ce->ce_mode = create_ce_mode(n->mode); ce->ce_flags = create_ce_flags(stage); @@ -1066,6 +1068,13 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, /* len+1 because the cache_entry allocates space for NUL */ make_traverse_path(ce->name, len + 1, info, n->path, n->pathlen); + if (is_sparse_directory) { + ce->name[len] = '/'; + ce->name[len + 1] = '\0'; + ce->ce_namelen++; + ce->ce_flags |= CE_SKIP_WORKTREE; + } + return ce; } @@ -1074,20 +1083,28 @@ static struct cache_entry *create_ce_entry(const struct traverse_info *info, * without actually calling it. If you change the logic here you may need to * check and change there as well. */ -static int unpack_nondirectories(int n, unsigned long mask, - unsigned long dirmask, - struct cache_entry **src, - const struct name_entry *names, - const struct traverse_info *info) +static int unpack_single_entry(int n, unsigned long mask, + unsigned long dirmask, + struct cache_entry **src, + const struct name_entry *names, + const struct traverse_info *info, + int sparse_directory) { int i; struct unpack_trees_options *o = info->data; unsigned long conflicts = info->df_conflicts | dirmask; - /* Do we have *only* directories? Nothing to do */ if (mask == dirmask && !src[0]) return 0; + /* defer work if our cache entry doesn't match the expectations. */ + if (sparse_directory) { + if (src[0] && !S_ISSPARSEDIR(src[0]->ce_mode)) + BUG("expected sparse directory entry"); + } else if (src[0] && S_ISSPARSEDIR(src[0]->ce_mode)) { + return 0; + } + /* * Ok, we've filled in up to any potential index entry in src[0], * now do the rest. @@ -1117,7 +1134,9 @@ static int unpack_nondirectories(int n, unsigned long mask, * not stored in the index. otherwise construct the * cache entry from the index aware logic. */ - src[i + o->merge] = create_ce_entry(info, names + i, stage, &o->result, o->merge); + src[i + o->merge] = create_ce_entry(info, names + i, stage, + &o->result, o->merge, + sparse_directory); } if (o->merge) { @@ -1221,16 +1240,59 @@ static int find_cache_pos(struct traverse_info *info, return -1; } +/* + * Given a sparse directory entry 'ce', compare ce->name to + * info->name + '/' + p->path + '/' if info->name is non-empty. + * Compare ce->name to p->path + '/' otherwise. Note that + * ce->name must end in a trailing '/' because it is a sparse + * directory entry. + */ +static int sparse_dir_matches_path(const struct cache_entry *ce, + struct traverse_info *info, + const struct name_entry *p) +{ + assert(S_ISSPARSEDIR(ce->ce_mode)); + assert(ce->name[ce->ce_namelen - 1] == '/'); + + if (info->namelen) + return ce->ce_namelen == info->namelen + p->pathlen + 2 && + ce->name[info->namelen] == '/' && + !strncmp(ce->name, info->name, info->namelen) && + !strncmp(ce->name + info->namelen + 1, p->path, p->pathlen); + return ce->ce_namelen == p->pathlen + 1 && + !strncmp(ce->name, p->path, p->pathlen); +} + static struct cache_entry *find_cache_entry(struct traverse_info *info, const struct name_entry *p) { + struct cache_entry *ce; int pos = find_cache_pos(info, p->path, p->pathlen); struct unpack_trees_options *o = info->data; if (0 <= pos) return o->src_index->cache[pos]; - else + + /* + * Check for a sparse-directory entry named "path/". + * Due to the input p->path not having a trailing + * slash, the negative 'pos' value overshoots the + * expected position by one, hence "-2" here. + */ + pos = -pos - 2; + + if (pos < 0 || pos >= o->src_index->cache_nr) return NULL; + + ce = o->src_index->cache[pos]; + + if (!S_ISSPARSEDIR(ce->ce_mode)) + return NULL; + + if (sparse_dir_matches_path(ce, info, p)) + return ce; + + return NULL; } static void debug_path(struct traverse_info *info) @@ -1265,6 +1327,21 @@ static void debug_unpack_callback(int n, debug_name_entry(i, names + i); } +/* + * Returns true if and only if the given cache_entry is a + * sparse-directory entry that matches the given name_entry + * from the tree walk at the given traverse_info. + */ +static int is_sparse_directory_entry(struct cache_entry *ce, + struct name_entry *name, + struct traverse_info *info) +{ + if (!ce || !name || !S_ISSPARSEDIR(ce->ce_mode)) + return 0; + + return sparse_dir_matches_path(ce, info, name); +} + /* * Note that traverse_by_cache_tree() duplicates some logic in this function * without actually calling it. If you change the logic here you may need to @@ -1321,7 +1398,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (unpack_single_entry(n, mask, dirmask, src, names, info, 0) < 0) return -1; if (o->merge && src[0]) { @@ -1351,9 +1428,14 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, - names, info) < 0) + if (is_sparse_directory_entry(src[0], names, info)) { + if (unpack_single_entry(n, dirmask, mask & ~dirmask, src, names, info, 1) < 0) + return -1; + } else if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, + names, info) < 0) { return -1; + } + return mask; } From patchwork Tue Jun 29 01:51:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348749 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83C07C11F64 for ; Tue, 29 Jun 2021 01:52:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6C6B961CCE for ; Tue, 29 Jun 2021 01:52:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231740AbhF2Byd (ORCPT ); Mon, 28 Jun 2021 21:54:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46964 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231237AbhF2ByT (ORCPT ); Mon, 28 Jun 2021 21:54:19 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 70909C061787 for ; Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id u11so23618395wrw.11 for ; Mon, 28 Jun 2021 18:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9FJe++5tdMAw1p1s1FtpPZqddDVWa7mla9SPDPEisaI=; b=LMAJJdBd3ynb49qRlv+UUcRcGmPHyU3op1l0ZhyaxBj26X76bDwpCMuqXb2jJC3oMz O7kLQz825CD8mBqb01H/Ecz1+tpgDnN/sSNKAVUSlIPJyTBzipKf/wp5FajUGsRSkgbY oTMYhWhjhko8QlSPe9pDwfyJtOEALBMk+pCjdps+/hpPddxBRoMokaCMEB3gF11cJVF1 NmcVfn3NSvh/sZaC1cqF/o90PPbNkxdYPQpswWRvSWdU2tKTlAlRr/EdjlaYMo/QsleE c86jKNNLncTGy3IYQkFCRw1MKbsF9BODYzmvs89pYYhugF/M/ANEMCYox3urBIY+9F54 befg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9FJe++5tdMAw1p1s1FtpPZqddDVWa7mla9SPDPEisaI=; b=uVZukQ63vu0KDtDwa8ZuJj1lHs6auFb8t8KKjr7jeURxo4Ka23dET69CrZ0s0M/qEF BFIYOcP2G2ZZiG6i2Z/4Ll6AtJhIJivrbCaYzCm5qAYhyBtb6eRYN22UzHYHhictc8rS vkUSweks4CaPwYZOsQJPSmEcDtOH5C8mweBuQZND200oED5oDp6pJK5HJpuNbfI8BaN0 SuVU+NQ2NEZOkoyTfPAVwIcCF6PfQqJhFGo0OhA0ugrt+2zNPjKsM07ib+12wDSaVxX7 HhlJchXmkk/ck5Lc5VRTbVKuU7tUmhjogaZa0bXfY+FRc1yba0aWFm6jE8fg5bNnusnQ 4zYQ== X-Gm-Message-State: AOAM531I2NNoe2pq79Jrzyh8d0UaAnrkzfqi8HK33oL6DXt0Y5vNlea0 tNfYC42t9BxkCmGHQjFtLjHOY6ZBAA8= X-Google-Smtp-Source: ABdhPJwVJYD/xyEl85lsbIMdQmasP2/Dmg+SObcts5+7JaM/st0kDINusya6ml9zJeyhwOZ/fwZdmg== X-Received: by 2002:a5d:6d87:: with SMTP id l7mr1175210wrs.222.1624931509161; Mon, 28 Jun 2021 18:51:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z3sm1077049wmi.29.2021.06.28.18.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:48 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:37 +0000 Subject: [PATCH v6 09/14] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dir.c b/dir.c index ebe5ec046e0..2155107b1d7 100644 --- a/dir.c +++ b/dir.c @@ -1397,6 +1397,17 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* + * Directory entries are matched if and only if a file + * contained immediately within them is matched. For the + * case of a directory entry, modify the path to create + * a fake filename within this directory, allowing us to + * use the file-base matching logic in an equivalent way. + */ + if (parent_pathname.len > 0 && + parent_pathname.buf[parent_pathname.len - 1] == '/') + strbuf_add(&parent_pathname, "-", 1); + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; From patchwork Tue Jun 29 01:51:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25A98C11F64 for ; Tue, 29 Jun 2021 01:52:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 01AFE61CA3 for ; Tue, 29 Jun 2021 01:52:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231761AbhF2Byf (ORCPT ); Mon, 28 Jun 2021 21:54:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231263AbhF2ByT (ORCPT ); Mon, 28 Jun 2021 21:54:19 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BDA1C0617A6 for ; Mon, 28 Jun 2021 18:51:51 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id a13so23645151wrf.10 for ; Mon, 28 Jun 2021 18:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cf9nIryTcanflZXsZH2iY6xWOpEH8H2fFRcItvZOPQc=; b=KlfMHPf9dtASy5ggCKaaJ/rJWMiVvhkDjGwrISLYm1IKyfztrRK52IvgMPG9l/hrbL OirRrkpD9M/1r7BCW/QboTZdlcfYK+URifE1rAlH0Xio4G8pEurMmZ/kMwcq2BnQxT5d 7P3gR99Wpoz09SqJ0tpqksVPUVc87S0XDLXuwEX58fYU7Vx3k97bxdk+jXmn3lYcL6F+ 9IutEitRVbS84Iihi/J9z9kueTTmYZj72Kw6Fy1AESkdHelZDfNyUPKHOdADob3oUucH cRHZYYnvOSZC2wix3+Y6++CsGTK15xB/N2nAS+BwQ/7Z8Q+Djzfw+KAMy6mnM4iUMvM1 zWkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cf9nIryTcanflZXsZH2iY6xWOpEH8H2fFRcItvZOPQc=; b=iRbxehrtZuxX8co6jusjJGnfTvb9F+zM5O0X0ruXSi2L/S3cBVLchJuzCzzitc7bN2 CG6Tr8pIwoh5px1g9Q+PkQfCNA1Nk4pNSp1Ss6nW83cGeLOVBdJWFR/jYJXhV4R7VR4J 4ZhaQYs2hreFsWGRtFWrSdgqu/VJzMcCbaBYHZZBjIa3rwvypM0Dx/xUFk2QGWazvQna 7nQ2pgVUrHO6uQmVwc+pE+7GzBdORGI9sthQswCx+0UINtt6B8Jvu723myZrZJRfGPj+ +YWz1T/X8XJj3rDWUsdEyBATJsv7rqMufcFsdX2VdsJKGnGUbKTwfGSlxbmI8kAX85wI ZRuw== X-Gm-Message-State: AOAM531FoaDR2qJFp8FjJNICa78FcA0weEh5+h4f5D2jcfh0EHBCaRqw MxUs6llywh6pDV/gOYFCEL3anYwpjhY= X-Google-Smtp-Source: ABdhPJw4bVnbY8dsU/Dp9l60m5yHwQLTfRlgQweCFBX/nBXTrYodX9TK+Df2uc8cAKS4c8iJnZFYEg== X-Received: by 2002:a5d:5182:: with SMTP id k2mr29856431wrv.126.1624931509738; Mon, 28 Jun 2021 18:51:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a11sm1220355wrt.71.2021.06.28.18.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:49 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:38 +0000 Subject: [PATCH v6 10/14] diff-lib: handle index diffs with sparse dirs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee While comparing an index to a tree, we may see a sparse directory entry. In this case, we should compare that portion of the tree to the tree represented by that entry. This could include a new tree which needs to be expanded to a full list of added files. It could also include an existing tree, in which case all of the changes inside are important to describe, including the modifications, additions, and deletions. Note that the case where the tree has a path and the index does not remains identical to before: the lack of a cache entry is the same with a sparse index. Use diff_tree_oid() appropriately to appropriately compute the diff. Signed-off-by: Derrick Stolee --- diff-lib.c | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/diff-lib.c b/diff-lib.c index c2ac9250fe9..3f32f038371 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -325,6 +325,11 @@ static void show_new_file(struct rev_info *revs, unsigned dirty_submodule = 0; struct index_state *istate = revs->diffopt.repo->index; + if (new_file && S_ISSPARSEDIR(new_file->ce_mode)) { + diff_tree_oid(NULL, &new_file->oid, new_file->name, &revs->diffopt); + return; + } + /* * New file in the index: it might actually be different in * the working tree. @@ -347,6 +352,17 @@ static int show_modified(struct rev_info *revs, unsigned dirty_submodule = 0; struct index_state *istate = revs->diffopt.repo->index; + /* + * If both are sparse directory entries, then expand the + * modifications to the file level. + */ + if (old_entry && new_entry && + S_ISSPARSEDIR(old_entry->ce_mode) && + S_ISSPARSEDIR(new_entry->ce_mode)) { + diff_tree_oid(&old_entry->oid, &new_entry->oid, new_entry->name, &revs->diffopt); + return 0; + } + if (get_stat_data(istate, new_entry, &oid, &mode, cached, match_missing, &dirty_submodule, &revs->diffopt) < 0) { if (report_missing) From patchwork Tue Jun 29 01:51:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3EF0C11F64 for ; Tue, 29 Jun 2021 01:52:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 877AD61CCE for ; Tue, 29 Jun 2021 01:52:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231512AbhF2Byj (ORCPT ); Mon, 28 Jun 2021 21:54:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46948 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231405AbhF2ByX (ORCPT ); Mon, 28 Jun 2021 21:54:23 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9ED59C0617A8 for ; Mon, 28 Jun 2021 18:51:51 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id m18so23658549wrv.2 for ; Mon, 28 Jun 2021 18:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sxTXciz4KAY7fU0l8Y3HSNjtGXPSTPrbuXtexuD+41c=; b=EIF9j3pvE8z9useOqoCnka6SO8sDTkOyMfyhw99wbIPgkzzM3mhbAR/LV/dGFzzKss dYNvb7dofOuWtB6Nrc4K51yOkdo0yXnOLDMKLKvjybGvKUSs3Rru9DCli6KD3CTSWGuA HTOxT+3Gz6h+30EWqWbWKm52JsN9RXR012AWjW72d80nbY8pKlx99G9i0WtHGJg5vMoN 0e5VzYquAJeY4ZXIBevKX9p70F2KehalnJGANOPRUpoQnPhI4wSb3wokzw7buwmgzkNG WLT/723EJFPJrcxvBPQJ57q4LEoEpqnkujAd/0xHnF3xeHSd6K71KbSX5fHy7V42f7Pl HWYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sxTXciz4KAY7fU0l8Y3HSNjtGXPSTPrbuXtexuD+41c=; b=qfztsRDW5ARY7TI5UHf70bakJAfYsjaXJA9160Ya0OtO7XFTVSBqdjNFCOBo4eJ45N dgTZTM8/NsJVWCFKezamDZ3VNcMOhqsTtA/3qZFdvtMkrsGXHopFmWFgOWICZMJg4k+S NCcOjoJ6MZR+fDRjDaJG1ubnT/0zdlIaxsfZcMfooXqA12vCMDxL6+81Vpa2i8vPHZJ6 js6Yy7Uo5AkqRYUOS/HXpawIw0vFhU8rn7kA6sd21GD2YvA3SCDIv551XmGQ06nkbvfp sLU22IFRhFnH7bEB9dyyz/QbLGQeFaxe3iT5JwEKN+1GD/iRjjpeZab+xI8PF/FmxRnM H3ag== X-Gm-Message-State: AOAM532O2B1zhoJIgYV/8PBGHrf+ZHxaHtTvEvu5bIzIui63lPuEt9zB PFFHVB3H8xckcGMRY+HymzynEtCS7nU= X-Google-Smtp-Source: ABdhPJydHdzLKYkFouObaDvhNlJcu13I3X6nVyypya5pbShEtIuctUH3I10E2PKa9ih5ijY7p5qcWw== X-Received: by 2002:a05:6000:1245:: with SMTP id j5mr29831814wrx.371.1624931510272; Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k13sm16259010wrp.73.2021.06.28.18.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Message-Id: <48fd25aacbeb1baeca637aec40f3f4cba0da0f52.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:39 +0000 Subject: [PATCH v6 11/14] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 7c78e40b861..9035adcb7db 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -215,6 +215,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 42b67357169..96db3e74962 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1493,9 +1493,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1653,6 +1656,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e..ab9cc9d8f03 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Tue Jun 29 01:51:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5843FC11F64 for ; Tue, 29 Jun 2021 01:52:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3EBAE61CCE for ; Tue, 29 Jun 2021 01:52:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231849AbhF2Byp (ORCPT ); Mon, 28 Jun 2021 21:54:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231429AbhF2ByX (ORCPT ); Mon, 28 Jun 2021 21:54:23 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43708C0617AD for ; Mon, 28 Jun 2021 18:51:52 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id b3so23641493wrm.6 for ; Mon, 28 Jun 2021 18:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ECbNMnswesiYtXXqYlcprEFedHSup0mr+SkRjMRoLXE=; b=vTe+RLbBeRGKatg0dqwDAHHmyIY39gWQm4S9cerUrM/mg5y415Ph5U9xb5yJV7ZHSk BiHglw4urHkQdSw7tJnhUyVZvdbBMeHxeGPM9/Vyb6Hr+WznMEWcTYD0JetMFwiLVniT ggI4okTPbDJ74IW5V2pwDFx0WQN9cxGRY7rB2ilEPryoSwmkB8KNcOLVjBo/jrxCCHoe NSfi6v/N0mGOQZ1qws4dy8lZVeSL5V3mRS9Gnb+JLmmp2t/l5XUj3qIEmjL6tpGPAd7g zmXIlyFmSQEpVMcvykfInXA/ZniMFlQctueS8LLw7PLBWiZrbr9xsz+KImiGTaFIG5Cg cJnw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ECbNMnswesiYtXXqYlcprEFedHSup0mr+SkRjMRoLXE=; b=h0nvXqcF0QBwcb3dR+AWflfs9uDyTXXRPayEO5LpSMeJJC4n1kA80XG9Fk5yPz5VXH GMmzwme7nVhL91XptdK7PpUa1JxpIpV38nOwfwmfAS+rg9B4s4LNT3igDr76sIkb+XAX P2PrOl0/2s7kDRlW+VLLAMx1i8/6i5GMY3l8DyFDFBwLR4qIkNzLObH4H7a6gQKtmEYy XTAdMzsq6ZCXc5m2ujbccYaGmbvqPDqRmzJYB/IMfdZnNlCSAIsYeojHJa68iB4IPo2s lnKD6osWPeLA9OHdUkmVBwW0iN4tu0zG2dd4624UkJsDfwd72Mv71yGeJCU/GtmVUke2 ILgQ== X-Gm-Message-State: AOAM5313mZCtklGPe7XVCz0+uRu/GZXU0QkqZfr1Ikijiysoj3nz9OdR pB0UcZwsPBq3/IS6VoBYoWD+bZKKKtU= X-Google-Smtp-Source: ABdhPJzBHjg2Kh0oeMhFVSAIYu7FA2v1PFZxQ7tGiybpCWVoBbCh+giC6eGB3W9RH0gue79RSpZWBg== X-Received: by 2002:a5d:6da3:: with SMTP id u3mr15388034wrs.394.1624931510884; Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 9sm1631096wmf.3.2021.06.28.18.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:50 -0700 (PDT) Message-Id: <3499105eb67f8514589b5839c0661b4049714fb7.1624931502.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:40 +0000 Subject: [PATCH v6 12/14] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index 190d215d43b..12f51db158a 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1510,6 +1510,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 1b3c2eb408b..277c2970a03 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1584,8 +1584,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1600,6 +1599,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 9035adcb7db..1e9737cb4b7 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -508,12 +508,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Tue Jun 29 01:51:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 003C6C11F64 for ; Tue, 29 Jun 2021 01:52:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DAFF361CA3 for ; Tue, 29 Jun 2021 01:52:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230461AbhF2Byn (ORCPT ); Mon, 28 Jun 2021 21:54:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231438AbhF2ByX (ORCPT ); Mon, 28 Jun 2021 21:54:23 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D0680C0617AE for ; Mon, 28 Jun 2021 18:51:52 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id u11so23618455wrw.11 for ; Mon, 28 Jun 2021 18:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WX6vm+pB/oRDYTBvwe01sJEK6/dnRyi7uGURBFpaEWs=; b=cv7etHr/6u/Ight8gZuM5enqj1FzPY7guJYcYUMwL+R4Ahndq6izVfJ6I2PamHcOxJ UZ3lDhHe0dF1j2yes8WivneDafOFGH93IOL5HpC0NEXNaaPEqcZXOoRCnsJEQCWAEaJX xrjC1noqv5XsswB5mao3v77COq6o6RslvhcB4BCcOr2UPxL79UgyF5ANRvxQXA29KrHu JeVoUBLWvc5rsAo3x2UP/fh4FtW0OKAh1ESASSRPVLBpjvs6mrt5O11AJsypSoiY/moI OfRCI5F2jcjhiu2ZVlyPM65EMAPAw3cjPrA2pFW8Gu9comDdv9Psaz1KyFaIX+C5081P Efng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WX6vm+pB/oRDYTBvwe01sJEK6/dnRyi7uGURBFpaEWs=; b=WbHq+QFFkkOWhi5CnQuaOOyjcEaV5u91BIDf17flRJYd6lNe3RP5S6donNoAahjWFf ZMbpJrAT1gi6jQYS7vSYTIUekvbZc4nr6lNAmbEE2GOcPLbRCO/AKboAj+Cq5+WdPBCg afs7GO4QQddXsiPZyjQESX2L+XOaqf7ST8iRtdvI4UjyYZ94zWXWkfD4Gv5drZkFHb+q Y/xFvIWqbII4nzO0sa3xth/u0CYzpZzSg5c6spGNFnIht5DmU/FRxwb5jAkWTL55SI2B oHb9Bw+yI3G+ETqH8vG1Hd4Gfa73y+xM/wFcxIXhDrVYHGZRSfPp9gmucnGXX4UdEzes VPWA== X-Gm-Message-State: AOAM531a26ra3gUtbZoCEt884UWRyT9mEjznuOpKE7DDBxKK4OfUi1xp hJ4JFuOHJOvfj0PdtW+hzvCtuGNSuA0= X-Google-Smtp-Source: ABdhPJw7OsJtV0m9mLyWp3QPn5aNzZ5wsZbXyq0OeLEkNB9Qv7RQwC5Zd8i12hmxFAfR+nMgfnbjkA== X-Received: by 2002:a5d:46cc:: with SMTP id g12mr13888164wrs.136.1624931511523; Mon, 28 Jun 2021 18:51:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o11sm1531108wmc.2.2021.06.28.18.51.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:51 -0700 (PDT) Message-Id: <08225483d699514426d4ddaf268338e09ea4e8bb.1624931503.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:41 +0000 Subject: [PATCH v6 13/14] wt-status: expand added sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It is difficult, but possible, to get into a state where we intend to add a directory that is outside of the sparse-checkout definition. Add a test to t1092-sparse-checkout-compatibility.sh that demonstrates this using a combination of 'git reset --mixed' and 'git checkout --orphan'. This test failed before because the output of 'git status --porcelain=v2' would not match on the lines for folder1/: * The sparse-checkout repo (with a full index) would output each path name that is intended to be added. * The sparse-index repo would only output that "folder1/" is staged for addition. The status should report the full list of files to be added, and so this sparse-directory entry should be expanded to a full list when reaching it inside the wt_status_collect_changes_initial() method. Use read_tree_at() to assist. Somehow, this loop over the cache entries was not guarded by ensure_full_index() as intended. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 33 ++++++++++++++++ wt-status.c | 49 ++++++++++++++++++++++++ 2 files changed, 82 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 1e9737cb4b7..ef918437908 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -521,4 +521,37 @@ test_expect_success 'sparse-index is not expanded' ' test_region ! index ensure_full_index trace2.txt ' +test_expect_success 'reset mixed and checkout orphan' ' + init_repos && + + test_all_match git checkout rename-out-to-in && + + # Sparse checkouts do not agree with full checkouts about + # how to report a directory/file conflict during a reset. + # This command would fail with test_all_match because the + # full checkout reports "T folder1/0/1" while a sparse + # checkout reports "D folder1/0/1". This matches because + # the sparse checkouts skip "adding" the other side of + # the conflict. + test_sparse_match git reset --mixed HEAD~1 && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 && + + # At this point, sparse-checkouts behave differently + # from the full-checkout. + test_sparse_match git checkout --orphan new-branch && + test_sparse_match test-tool read-cache --table --expand && + test_sparse_match git status --porcelain=v2 +' + +test_expect_success 'add everything with deep new file' ' + init_repos && + + run_on_sparse git sparse-checkout set deep/deeper1/deepest && + + run_on_all touch deep/deeper1/x && + test_all_match git add . && + test_all_match git status --porcelain=v2 +' + test_done diff --git a/wt-status.c b/wt-status.c index 96db3e74962..a90c7b6aa8a 100644 --- a/wt-status.c +++ b/wt-status.c @@ -657,6 +657,34 @@ static void wt_status_collect_changes_index(struct wt_status *s) clear_pathspec(&rev.prune_data); } +static int add_file_to_list(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct string_list_item *it; + struct wt_status_change_data *d; + struct wt_status *s = context; + char *full_name; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + full_name = xstrfmt("%s%s", base->buf, path); + it = string_list_insert(&s->change, full_name); + d = it->util; + if (!d) { + CALLOC_ARRAY(d, 1); + it->util = d; + } + + d->index_status = DIFF_STATUS_ADDED; + /* Leave {mode,oid}_head zero for adds. */ + d->mode_index = mode; + oidcpy(&d->oid_index, oid); + s->committable = 1; + return 0; +} + static void wt_status_collect_changes_initial(struct wt_status *s) { struct index_state *istate = s->repo->index; @@ -671,6 +699,27 @@ static void wt_status_collect_changes_initial(struct wt_status *s) continue; if (ce_intent_to_add(ce)) continue; + if (S_ISSPARSEDIR(ce->ce_mode)) { + /* + * This is a sparse directory entry, so we want to collect all + * of the added files within the tree. This requires recursively + * expanding the trees to find the elements that are new in this + * tree and marking them with DIFF_STATUS_ADDED. + */ + struct strbuf base = STRBUF_INIT; + struct pathspec ps = { 0 }; + struct tree *tree = lookup_tree(istate->repo, &ce->oid); + + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_add(&base, ce->name, ce->ce_namelen); + read_tree_at(istate->repo, tree, &base, &ps, + add_file_to_list, s); + continue; + } + it = string_list_insert(&s->change, ce->name); d = it->util; if (!d) { From patchwork Tue Jun 29 01:51:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8A616C11F64 for ; Tue, 29 Jun 2021 01:52:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7574861CA3 for ; Tue, 29 Jun 2021 01:52:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230008AbhF2Byz (ORCPT ); Mon, 28 Jun 2021 21:54:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231472AbhF2ByX (ORCPT ); Mon, 28 Jun 2021 21:54:23 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 663EFC0613A2 for ; Mon, 28 Jun 2021 18:51:53 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id j1so23628028wrn.9 for ; Mon, 28 Jun 2021 18:51:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2NnBKce/32GyuSe6kI1cYosfQIE9q6lvos67gcH5Bfo=; b=PtBx5Xr1eLxbfo0U4+MOyKiq/NTWXgF/J9xjMD87wZTgJLYHoeIfEfjXRY6fv0xdzw WZvYGyCg+fgqsb2RRFkJAtrpluwWjDD/uXrICZX9+nKbGH0ZFjlwJNbqRWqDUMXfQUOp 3b+HyF9vlVIEbloGzb/PvmlE617JKkDLfT5ajI52V8UfiLHJ7ZnFa9dUPGdndIrg88vu C13qa8ajHyYn3lAJj9V8/nr2cPbhd0w+Yop1zgnwt5B7d6rNl3bKEgOjs+RFDQbsTue8 U0qlrW3OhQpCFRf3sj2Pi7IpR5TfQqu2JDzuWvKOvdOmtq97byIEsNXN8Rbz/0D8GfMj JL3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2NnBKce/32GyuSe6kI1cYosfQIE9q6lvos67gcH5Bfo=; b=dhBxTCxPnfnSTjRs64q2vKWrieaopsb1pYjaL3tfVO5Ek+h8nxYwIJ2K8/Kl0RWcEf qhF5lS2ZYn0QNipSmKQTXz5wJynmoKv6SAJ9SVjl2PcYOZOunFjtQJHD3bFijgKfx39Y piGx5BEiPiX6dF7lm/5D/Pv+rHp6LA/vDkQGmxkOkDftglqnTegGaIIHQIM60MnMe3GA B/TL8+ahav749alj/IDYVuqxDBKNBeB4zmggg0RV9BrPOx3OJOGoZ4PJnriKCgH0xyoU YkRIJuqJrZ61rIwQR7GVeZCb2/RyoHMU63L6sSLn8wYOXTpXMLpKzhWRt+NyWfSGtoxy MFLQ== X-Gm-Message-State: AOAM533UisG+k1W1E+H+eBnucjCrGFRRr4XZc8b7sjm+P1x10G2kW5RW iDgwchyA3v6mMMDChrl/RbIZlPgPC6g= X-Google-Smtp-Source: ABdhPJylLzM5bxSiFBVC5LaV8Z9BgGhIFAZ5WFckn9OOfhkESSwMudj7UHsqXJJ64zR1ZvBslViiyA== X-Received: by 2002:a5d:4fd0:: with SMTP id h16mr31097172wrw.302.1624931512064; Mon, 28 Jun 2021 18:51:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l10sm16613293wrv.82.2021.06.28.18.51.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 18:51:51 -0700 (PDT) Message-Id: <711e403a63a7815d0581b469a9f0d52d0c8054ca.1624931503.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 01:51:42 +0000 Subject: [PATCH v6 14/14] fsmonitor: integrate with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , git@jeffhostetler.com, johannes.schindelin@gmx.de, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we need to expand a sparse-index into a full one, then the FS Monitor bitmap is going to be incorrect. Ensure that we start fresh at such an event. While this is currently a performance drawback, the eventual hope of the sparse-index feature is that these expansions will be rare and hence we will be able to keep the FS Monitor data accurate across multiple Git commands. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- sparse-index.c | 7 ++++++ t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++ 2 files changed, 55 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index ef53bd2198b..53c8f711ccc 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -186,6 +186,10 @@ int convert_to_sparse(struct index_state *istate) cache_tree_free(&istate->cache_tree); cache_tree_update(istate, 0); + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -282,6 +286,9 @@ void ensure_full_index(struct index_state *istate) istate->cache = full->cache; istate->cache_nr = full->cache_nr; istate->cache_alloc = full->cache_alloc; + istate->fsmonitor_has_run_once = 0; + FREE_AND_NULL(istate->fsmonitor_dirty); + FREE_AND_NULL(istate->fsmonitor_last_update); strbuf_release(&base); free(full); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 637391c6ce4..8f9240b9b1a 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -73,6 +73,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Usage: +# check_sparse_index_behavior [!] +# If "!" is supplied, then we verify that we do not call ensure_full_index +# during a call to 'git status'. Otherwise, we verify that we _do_ call it. +check_sparse_index_behavior () { + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set dir1 dir2 && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region $1 index ensure_full_index trace2.txt && + test_cmp expect actual && + rm trace2.txt && + git sparse-checkout disable +} + +test_expect_success 'status succeeds with sparse index' ' + git reset --hard && + + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + check_sparse_index_behavior ! && + + cp -r dir1 dir1a && + git add dir1a && + git commit -m "add dir1a" && + + # This one modifies outside the sparse-checkout definition + # and hence we expect to expand the sparse-index. + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + check_sparse_index_behavior +' + test_done