From patchwork Fri Apr 23 21:34:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 635DFC433B4 for ; Fri, 23 Apr 2021 21:34:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 36719611F1 for ; Fri, 23 Apr 2021 21:34:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244036AbhDWVfM (ORCPT ); Fri, 23 Apr 2021 17:35:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231881AbhDWVfH (ORCPT ); Fri, 23 Apr 2021 17:35:07 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0BF5C06174A for ; Fri, 23 Apr 2021 14:34:28 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id t14-20020a05600c198eb029012eeb3edfaeso1917063wmq.2 for ; Fri, 23 Apr 2021 14:34:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Gng4CVH/clzaTPsSsOU7C34qQU1voJ+7k7GV9EFJ3C8=; b=cnkyodKP+XwuKRX2KnBn30DAr2D0K1UZdnAg3FbGO1ZWukZ+4vYft9m9KB/kT5VTlH lrg0kqasupA12brg+zw1Eou7LNCQvC3PEsJEIKTqigDt5yVKOc3S4XhfUTD4F+lXsJ7g wnP18f/nxMBAF7v3ELQm3TacdRJz3okcyrPQjErFh8hK88mbEMrnP0ADvtcdEMH/tGia A1XCiB4r6JSpWTHHFxsiTyVC6+j615+1ZC5pVIdwIC1iCdGPrRbcS3wZXa49Cdv6P/LW FeBpi8/ZfgJUWFipKn7ReTw8WsoPk17YVPMUAuWjMob7/AZ0BBFsesx5jkgspG1NJ8O8 2BeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Gng4CVH/clzaTPsSsOU7C34qQU1voJ+7k7GV9EFJ3C8=; b=uh25RXIhYUHygm2VYSKxZRfS/DVOBv2ZObfPgZvKCgjb80YWNXJZ4nk0u1NdPr5SWH Q1SeuvDB94O5Vu+h279rk0Z2m9635CTrTnf7tEeMNNtmua1Lgu1t/x8qLULvH1meLUmh cHjg/cs4H4mC/tfXO35Em11Kk7YN4hJIDtRXZTqs/A8cTciyAWZe3uyfrHxvLO7Ip9iV YE/aJBAMZj4XxzFuTn40t0ofuO4kJeE1YE7VbzGGz/9BZ34Ilij8uu4zr6GXkmjqINYC NXVd9s/PEkIIuX71rMdJfg/IgWJPGZ+OWqVW2rfuVcilAzlX20EyHsWNKJMKHlO7WnEn hHTA== X-Gm-Message-State: AOAM530uGMQIJyzvJMU30hedAn0ds4ULO2dBFBNXawLtcknsvfOj/Nif +7YdYHlTqPdH1e4CYK/UdoHwfu4hwW8= X-Google-Smtp-Source: ABdhPJy110TiOmdxDQfRg1/6CyUAPTBtJoypE1BVIazQ7NZXlYSb7YhdNJsPasBcR045OqutFxUrHg== X-Received: by 2002:a7b:cb4a:: with SMTP id v10mr6254851wmj.53.1619213667471; Fri, 23 Apr 2021 14:34:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u17sm10842051wmq.30.2021.04.23.14.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:27 -0700 (PDT) Message-Id: <3bac9edae7d82ef9fdabbe2f3959e574f79f1dd0.1619213665.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:18 +0000 Subject: [PATCH v2 1/8] t1092: add tests for status/add and sparse files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before moving to update 'git status' and 'git add' to work with sparse indexes, add an explicit test that ensures the sparse-index works the same as a normal sparse-checkout when the worktree contains directories and files outside of the sparse cone. Specifically, 'folder1/a' is a file in our test repo, but 'folder1' is not in the sparse cone. When 'folder1/a' is modified, the file 'folder1/a' is shown as modified, but adding it fails. This is new behavior as of a20f704 (add: warn when asked to update SKIP_WORKTREE entries, 2021-04-08). Before that change, these adds would be silently ignored. Untracked files are fine: adding new files both with 'git add .' and 'git add folder1/' works just as in a full checkout. This may not be entirely desirable, but we are not intending to change behavior at the moment, only document it. A future change could alter the behavior to be more sensible, and this test could be modified to satisfy the new expected behavior. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 40 ++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 12e6c453024f..0ec487acd283 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -232,6 +232,46 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'status/add: outside sparse cone' ' + init_repos && + + # folder1 is at HEAD, but outside the sparse cone + run_on_sparse mkdir folder1 && + cp initial-repo/folder1/a sparse-checkout/folder1/a && + cp initial-repo/folder1/a sparse-index/folder1/a && + + test_sparse_match git status && + + write_script edit-contents <<-\EOF && + echo text >>$1 + EOF + run_on_all ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + test_sparse_match git status --porcelain=v2 && + + # This "git add folder1/a" is completely ignored + # by the sparse-checkout repos. It causes the + # full repo to have a different staged environment. + # + # This is not a desirable behavior, but this test + # ensures that the sparse-index is not the cause + # of a behavior change. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + git -C full-checkout checkout HEAD -- folder1/a && + test_all_match git status --porcelain=v2 && + + test_all_match git add . && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/new && + + run_on_all ../edit-contents folder1/newer && + test_all_match git add folder1/ && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m folder1/newer +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Fri Apr 23 21:34:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4ABBDC433ED for ; Fri, 23 Apr 2021 21:34:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 23CD861468 for ; Fri, 23 Apr 2021 21:34:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244043AbhDWVfN (ORCPT ); Fri, 23 Apr 2021 17:35:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232438AbhDWVfI (ORCPT ); Fri, 23 Apr 2021 17:35:08 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A1E27C061756 for ; Fri, 23 Apr 2021 14:34:29 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id u187so7236821wmb.0 for ; Fri, 23 Apr 2021 14:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=a3arl8e4gwZWtFR/D9sPebc5HEpiWeVeBENuq2ciId1z0OO1rfC6H7Ok3k51JXK4PC AVel0cBgulAmsHnzIEy16EJsNdl1leM3XU+Y45uw+qoFPcxxltOL5QOkeKlWI0iDB8zf maJOxuhFu8QX3NNelwL8KvO9CBLN5Jue/TrlEuZsnmecAF7Wwh9NzwUGk1bAdPd8/ApB nVNIap+eavrtlZ+MWTDWqeQ+zfNsBCPiyp69eAzS8VoRUKabkMkGaHF3tcogmbsinYg0 50j4OwpCrqnM8/C3CTlJvnyQ6+05D6sNFiWuMn2oWEUU/kROnsP8oGUKDywfwQ6ZUeNY WLYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wFfpok35DynSJzLw2iHIzdshoI7yibKc8zfxCLq0Mn8=; b=cYDqBI0sWgZPODAwUMICELCyXFR6tSYyQ5BLz6V2HCEjYDf+r+K7s4xPV4ZJzsnAG4 bR1pjs2dCpBF9pu+OAImf6Nr85TJPfRcEPd9KLO0+a/xpCv+dp2yDkQpWGw4WZg4dA+k j8i0E1X9ZnHAHTE1QZgjzhgMuABplKeMCZlDSVAZZKmZp0Da6GXcq9v5FTu94s7KSN/U NQlttgS5KgnnkS5pQMixtCCZmKa11WeRklnDgXTAjemA8OEegUFD8S93tO1/17ndT35W kn7R6BElThkom22ou3oVui4csnbX3/RM7DVL//8xSGufOOAbOAIJxu6waMQlS0Z474r4 k3xQ== X-Gm-Message-State: AOAM531HDdm2tR2d7Sb6rvl8CjPvnwHsW0SyD/beItNndim1C36z4Vjj EiUDsl6aR99lx4c4Yx5gjh+Fpvk8btQ= X-Google-Smtp-Source: ABdhPJy7hha6oKmOx9QpRGkbUG4//OCjrciLvfyF1mX+C/ISuZRtqnltKjnrRCpkKhpAsldIpQPJtA== X-Received: by 2002:a1c:f407:: with SMTP id z7mr6123063wma.20.1619213668165; Fri, 23 Apr 2021 14:34:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q18sm10438123wrs.25.2021.04.23.14.34.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:27 -0700 (PDT) Message-Id: <19344394379dd1ac84f21ae5d72747af5e554668.1619213665.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:19 +0000 Subject: [PATCH v2 2/8] unpack-trees: preserve cache_bottom Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_bottom member of 'struct unpack_trees_options' is used to track the range of index entries corresponding to a node of the cache tree. While recursing with traverse_by_cache_tree(), this value is preserved on the call stack using a local and then restored as that method returns. The mark_ce_used() method normally modifies the cache_bottom member when it refers to the marked cache entry. However, sparse directory entries are stored as nodes in the cache-tree data structure as of 2de37c53 (cache-tree: integrate with sparse directory entries, 2021-03-30). Thus, the cache_bottom will be modified as the cache-tree walk advances. Do not update it as well within mark_ce_used(). Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index dddf106d5bd4..1067db19c9d2 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -586,6 +586,13 @@ static void mark_ce_used(struct cache_entry *ce, struct unpack_trees_options *o) { ce->ce_flags |= CE_UNPACKED; + /* + * If this is a sparse directory, don't advance cache_bottom. + * That will be advanced later using the cache-tree data. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return; + if (o->cache_bottom < o->src_index->cache_nr && o->src_index->cache[o->cache_bottom] == ce) { int bottom = o->cache_bottom; From patchwork Fri Apr 23 21:34:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04750C433B4 for ; Fri, 23 Apr 2021 21:34:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CEB716146B for ; Fri, 23 Apr 2021 21:34:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244054AbhDWVfR (ORCPT ); Fri, 23 Apr 2021 17:35:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232686AbhDWVfI (ORCPT ); Fri, 23 Apr 2021 17:35:08 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D359AC06175F for ; Fri, 23 Apr 2021 14:34:29 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id c15so40732280wro.13 for ; Fri, 23 Apr 2021 14:34:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0pOxUo1RuCQro+WwKXvgCKN+ys4nNfKLvPYvW8DNlHs=; b=HyZToWcaQguUd2rHnPIf17k+/1mYxwAqeo4ybAnR4v822P6gP8ZW9UQnHCyhir7Wsi Hh7HX5ApsYDRTmDdYpuxlFLBuwWcGB8xXdeZdVWiKRdi4fX3C/17FEYpd3cAd7Gp8RQ9 edWvvjG4BnrrFHaoOuAHtIcFzcGEP02d98Y4FCYmJxBVWIQLqSwCjujz1+KxQrw8vMxK 8RpnNYx1ds7ajiJEoUhUprhuQc6ZNz81J3dfM5i0J3HA/lKMDRgiFJfxJcPXNjnH6zyR TY9V6ci3a6TXmPaiTzUpRRp7nsAih0/HX1iUZSJXqmQ+R9nv+G5+NLGALUG5Q4Pdu1gu MHeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0pOxUo1RuCQro+WwKXvgCKN+ys4nNfKLvPYvW8DNlHs=; b=gzi9zi5IVbC3iiSUdIADWBQePpIQ69bg85ths0CepG3yAe8XStE3j5ZDOMDotc8ldT wrauTTmIBkavZfDT1wQB4/ZNJ9Uz+dIz/UHpHyVBMVJyKO4oxzAIrL5n6z/TmO1wRIFb K6617XVUfOI/FzsuTBju6PqdN5A/iaLqYZDrLGRCXrdUQk2vToi5yc+Lp42j9TPgReWM efdNyxL+MVcFYnieM/K546DQ7c3H8acsOgqVv4O4oYMG0kiKPQF4ybUSGnm/9yRZYMlx 3Sp0NSSoCWTa2PH+gQ2Fk5OpB6ftnVOffxWgAM9rP6pxaZakJmpZHx1+Px15N1QtHnIb T4aQ== X-Gm-Message-State: AOAM531YcntXyzcwPe5i/+njO1dOoKC8yrxOjqjlppyyDAEfmy1QOuIJ Tg9gzL/ciQ8P4NM8Vyy9VoP+kFYIm0U= X-Google-Smtp-Source: ABdhPJz3tZmEhHNUcPv5kxbTYLjPYxsOHoxCQBwfbfeQaLueW0jesgVGIAvBW28AxNwdux1oExQxwg== X-Received: by 2002:a5d:590a:: with SMTP id v10mr6998287wrd.306.1619213668689; Fri, 23 Apr 2021 14:34:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c77sm8822149wme.37.2021.04.23.14.34.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:28 -0700 (PDT) Message-Id: <24e71d8c062239be9b995bcc31dd12edf84106e3.1619213665.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:20 +0000 Subject: [PATCH v2 3/8] unpack-trees: compare sparse directories correctly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we further integrate the sparse-index into unpack-trees, we need to ensure that we compare sparse directory entries correctly with other entries. This affects searching for an exact path as well as sorting index entries. Sparse directory entries contain the trailing directory separator. This is important for the sorting, in particular. Thus, within do_compare_entry() we stop using S_IFREG in all cases, since sparse directories should use S_IFDIR to indicate that the comparison should treat the entry name as a dirctory. Within compare_entry(), it first calls do_compare_entry() to check the leading portion of the name. When the input path is a directory name, we could match exactly already. Thus, we should return 0 if we have an exact string match on a sparse directory entry. Signed-off-by: Derrick Stolee --- unpack-trees.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/unpack-trees.c b/unpack-trees.c index 1067db19c9d2..3af797093095 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -969,6 +969,7 @@ static int do_compare_entry(const struct cache_entry *ce, int pathlen, ce_len; const char *ce_name; int cmp; + unsigned ce_mode; /* * If we have not precomputed the traverse path, it is quicker @@ -991,7 +992,8 @@ static int do_compare_entry(const struct cache_entry *ce, ce_len -= pathlen; ce_name = ce->name + pathlen; - return df_name_compare(ce_name, ce_len, S_IFREG, name, namelen, mode); + ce_mode = S_ISSPARSEDIR(ce->ce_mode) ? S_IFDIR : S_IFREG; + return df_name_compare(ce_name, ce_len, ce_mode, name, namelen, mode); } static int compare_entry(const struct cache_entry *ce, const struct traverse_info *info, const struct name_entry *n) @@ -1000,6 +1002,10 @@ static int compare_entry(const struct cache_entry *ce, const struct traverse_inf if (cmp) return cmp; + /* If ce is a sparse directory, then allow an exact match. */ + if (S_ISSPARSEDIR(ce->ce_mode)) + return 0; + /* * Even if the beginning compared identically, the ce should * compare as bigger than a directory leading up to it! From patchwork Fri Apr 23 21:34:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 058FBC433B4 for ; Fri, 23 Apr 2021 21:34:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D86A661467 for ; Fri, 23 Apr 2021 21:34:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244075AbhDWVfS (ORCPT ); Fri, 23 Apr 2021 17:35:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51586 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243971AbhDWVfJ (ORCPT ); Fri, 23 Apr 2021 17:35:09 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78E55C06138B for ; Fri, 23 Apr 2021 14:34:30 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id o9-20020a1c41090000b029012c8dac9d47so1913579wma.1 for ; Fri, 23 Apr 2021 14:34:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=PWgdKnHxpR77nD6z+mgPXsfVi8En6vUQNKR8GlQu2hQ=; b=jcUXh5flR1FNglLcz5RMf+NXnJtDA0/8wQxhxyHWAyjNppkfMVbflQyWYtWcTJDduL Yb9qkm+a6z+KNGLcLGqCgmm+ML4VKDgJKbKvwKVrGJGED6cjmDJ4ppwO56oYfRH9guOa 5QmFkatzz3ji5ephGMxgkZy2BMsyF9d86PCbTkKEauaW6CQJREDmBWbZ4x2pyWV8OHue FCOicJNYSAPFCrV3OEAmobJqxML/MK7gHCGOZFGdaXeny+dxr55r3yvSCGh8TbaA65pc I9QjTktJBrRYEXXXFgXVlnVaxQhjco7oWsJ85uQNG2RCmGBcDz6zb2eRr9iQVx/4BGM/ Gtjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=PWgdKnHxpR77nD6z+mgPXsfVi8En6vUQNKR8GlQu2hQ=; b=BMB2WsQ/ndVBQOBbvGxljhWO6U36dSmGFZzXLV/r8DzOeFIFUpdF+hAH1JDxVS+But o6FMnFbocAQFdUAcVVl14kUByzhaJ8k1pA4bZCVj/k8ECwij6tN8v3qcPzLYEKxABpYI JP2shmGoYEyglQhioH3B/jCIgtK+X+cqnm1IWlWtOt+IDMoX1bUJgazacXGbcrdcrNR1 3K4riJ893tIZMx5yBV1tg0kBG1zgDpv1P37QjJ54wnkp9q2KRtaS0oIoLs2dtHFJs1yF 4bwgNRnEI2dkhpjVQDhQFfRp9JAk6paGB8AEuY/HlEImyyGlzheqGOWNOzvxOPOGQPYz N4bw== X-Gm-Message-State: AOAM5338f7VqlKXEpXkvIddHyjRczmXWcAem/VxmgLzibd75q/pgeEsZ bJ7Mx51uPlecOKI6u0YDEg8v1Wyh7ec= X-Google-Smtp-Source: ABdhPJzQm01Xtt0+d9x5mcy3wZr6Bnvf1ZXulXRa/je0AbTV+XS4iO+1Aa5pEJ9/2zyusVEDUp5dww== X-Received: by 2002:a1c:bd46:: with SMTP id n67mr7647441wmf.171.1619213669289; Fri, 23 Apr 2021 14:34:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r11sm10603186wrx.22.2021.04.23.14.34.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:29 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:21 +0000 Subject: [PATCH v2 4/8] unpack-trees: stop recursing into sparse directories Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When walking trees using traverse_trees_recursive() and unpack_callback(), we must not attempt to walk into a sparse directory entry. There are no index entries within that directory to compare to the tree object at that position, so skip over the entries of that tree. This code is used in many places, so the only way to test it is to start removing the command_requres_full_index option from one builtin at a time and carefully test that its use of unpack_trees() behaves correctly with a sparse-index. Such tests will be added by later changes. Signed-off-by: Derrick Stolee --- unpack-trees.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index 3af797093095..67777570f829 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1256,6 +1256,7 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str struct cache_entry *src[MAX_UNPACK_TREES + 1] = { NULL, }; struct unpack_trees_options *o = info->data; const struct name_entry *p = names; + unsigned unpack_tree = 1; /* Find first entry with a real name (we could use "mask" too) */ while (!p->mode) @@ -1297,12 +1298,16 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } src[0] = ce; + + if (S_ISSPARSEDIR(ce->ce_mode)) + unpack_tree = 0; } break; } } - if (unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) + if (unpack_tree && + unpack_nondirectories(n, mask, dirmask, src, names, info) < 0) return -1; if (o->merge && src[0]) { @@ -1332,7 +1337,8 @@ static int unpack_callback(int n, unsigned long mask, unsigned long dirmask, str } } - if (traverse_trees_recursive(n, dirmask, mask & ~dirmask, + if (unpack_tree && + traverse_trees_recursive(n, dirmask, mask & ~dirmask, names, info) < 0) return -1; return mask; From patchwork Fri Apr 23 21:34:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221765 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02714C433ED for ; Fri, 23 Apr 2021 21:34:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D462961468 for ; Fri, 23 Apr 2021 21:34:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244080AbhDWVfU (ORCPT ); Fri, 23 Apr 2021 17:35:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243974AbhDWVfK (ORCPT ); Fri, 23 Apr 2021 17:35:10 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2E10AC06138C for ; Fri, 23 Apr 2021 14:34:31 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id 26-20020a05600c22dab029013efd7879b8so34438wmg.0 for ; Fri, 23 Apr 2021 14:34:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=RZDgNae2n49g/nvAEhmQ+mdun7q/i5/+f5zRA+RZ/VTD6SukZxqf3CF4/fsHKD+/n0 CAdFHNZ4sxZQ/HhLN0fzdFhF8z84DQql/zdO2cjMf6y/TWjCvuItZEdqUmoji44/XfHn UhsAUxmxDj+AEG0W2D8YhAk6lSTL1BzkmofzObGYN8fcF4/xogRQdcauEePhtVaYIuIM ZXLCxHqSLTCXsKD2DDWIls6OHatHKSYwC4RFuU83ICQBx17W1M5lE8C9qYYxwrUDQ2Ta 3xAMxaR6Gecm3k2lYyB5KDu/yuwUqDzknSWKFT9HQHB6jgSKldN9CAho1tTu4EkWnxvI KAFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IKPjTUqki+7odL+EABJRxmdzP4v6AWl9RXlIatOMDLM=; b=btDKRVdrnS+s2CGzH+faoh1iJNBFOWNO8F4JKAwnWxnhnxm1gGXpEdCPlaOaEO14ER KqvF6NTLLc7HaCJ2ycms1nVfE8TqrDluutg2yeescrg03s+gcE8av63qTRVDA7LG4t8c IdxlPUlikArk3QtjLbAWfdEHBgClsjQe+IltbSE3MU5ItxvEeVwO1GX3h2oc9W4TZ0ry 3TrMVweRCmxhhUUA2QmDeGZ44l8gwSWbfXV5ILILgFwpH2hjO/m972Sz1OHAuhaPP8Bh Hc42Vtwj2D8aAKFdaZ986sIU47SDy3yS5C55wQyvaMHZwMX39ehgMjNCsLB7kC/LM2AF mU2A== X-Gm-Message-State: AOAM532ySTDDcTIJ24e6MgOdGnxNBH899y5LPjKpaRa8WaNQyl6ScbDG 6AyWAKyIq1ADzjzPk8n2l6PaxfB+EhU= X-Google-Smtp-Source: ABdhPJwcR6ILhJ96ySEvXXzVJeE9eEe8IWr/V/s5bFvu88DJdAMCmHxNLPFU7XZaSxwjY54xpwyhsA== X-Received: by 2002:a05:600c:35cf:: with SMTP id r15mr7490748wmq.183.1619213669887; Fri, 23 Apr 2021 14:34:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x9sm10737015wrt.13.2021.04.23.14.34.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:29 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:22 +0000 Subject: [PATCH v2 5/8] dir.c: accept a directory as part of cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When we have sparse directory entries in the index, we want to compare that directory against sparse-checkout patterns. Those pattern matching algorithms are built expecting a file path, not a directory path. This is especially important in the "cone mode" patterns which will match files that exist within the "parent directories" as well as the recursive directory matches. If path_matches_pattern_list() is given a directory, we can add a fake filename ("-") to the directory and get the same results as before, assuming we are in cone mode. Since sparse index requires cone mode patterns, this is an acceptable assumption. Signed-off-by: Derrick Stolee --- dir.c | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/dir.c b/dir.c index 166238e79f52..ab76ef286495 100644 --- a/dir.c +++ b/dir.c @@ -1378,6 +1378,17 @@ enum pattern_match_result path_matches_pattern_list( strbuf_addch(&parent_pathname, '/'); strbuf_add(&parent_pathname, pathname, pathlen); + /* + * Directory entries are matched if and only if a file + * contained immediately within them is matched. For the + * case of a directory entry, modify the path to create + * a fake filename within this directory, allowing us to + * use the file-base matching logic in an equivalent way. + */ + if (parent_pathname.len > 0 && + parent_pathname.buf[parent_pathname.len - 1] == '/') + strbuf_add(&parent_pathname, "-", 1); + if (hashmap_contains_path(&pl->recursive_hashmap, &parent_pathname)) { result = MATCHED_RECURSIVE; From patchwork Fri Apr 23 21:34:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1EDCC43460 for ; Fri, 23 Apr 2021 21:34:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8ACD161468 for ; Fri, 23 Apr 2021 21:34:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244046AbhDWVfR (ORCPT ); Fri, 23 Apr 2021 17:35:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51590 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243960AbhDWVfI (ORCPT ); Fri, 23 Apr 2021 17:35:08 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A38F7C06138D for ; Fri, 23 Apr 2021 14:34:31 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id p6so43002653wrn.9 for ; Fri, 23 Apr 2021 14:34:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=AiG42CSDvtiEqVNCos1umX9Q+yDZCmxbKcD4xVEEkZc=; b=RpBTxgtpKDs9nnhuxLm775qGBjEzQOVnPfuHy1Oem/um4bqBSufLLOIxmQd/5W5TTv hiffsFnSwmEWj0SJ0WoQLwrkUDWaSwWraTZt3Fx7AfqVVHTcOqUNS3yY6bklsi6e/+iA kgUWWqaV+jSvyaY1HTUZKHahmSpXXTEm1uEBzUI/TKIOYp5U6//ws+wk+fgmeaOY7gdw nV65D18gOx6oXBeNqzdB5zZn9UouhsY0Z2IgVC/y6vdGLej2Fxxa+nR0NApl/ginalfb AIa0hYUR5JPVQ1tIm2qMQ2da4q/6mc0IlCMlgpLZdZSz+KQiHBfvdDZ3+R7Lgykry5Iv AzaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=AiG42CSDvtiEqVNCos1umX9Q+yDZCmxbKcD4xVEEkZc=; b=qEmTWiWqhuJUDStcGiak7DDHtPp2ek5FHnXCnQ3ahEsXyM4T7hNo9GFsyXhvpPdATt J8tav/WN1pV/CiH4dKdcIO5E/Iguz8zvVVPMA5ogs9CIAfm7ST8Ojc6sT8ukK6218Bzt bKZKWM4cDnBB8f5QaHW1NRUpQOPAw29kkS3begNpLnM0f/VZgERj2wXEiwiiw8g/pjLr kxfDrJgmknNTuBEHaeaAOURudo4dtgKhwOgDp7u1nfqbiF5WluUtEjKQ4ERfJKj9KnW0 8a/sKvSMSppQr988eSIfzTsxI8cyqowZRPxC9Cikmabc2x8TLF82vkRVCVJkZVms+6X1 AkZA== X-Gm-Message-State: AOAM5306UZmtiLaJ8UzL/0XcWhTU42+Z7Xh85nBVUSnYuyMHI2cwfnfA ii/kAdVJxDnGzJjXLt20Rg9Vp5QfiSk= X-Google-Smtp-Source: ABdhPJxPg4qJ7HDUR6Du5P6WobJu2rFsP1fSJ6KELUe7Ox1xu95AQV9dNykGzTijzBUjeRty+RNrtg== X-Received: by 2002:a5d:650d:: with SMTP id x13mr7069632wru.264.1619213670417; Fri, 23 Apr 2021 14:34:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z16sm10516053wrq.21.2021.04.23.14.34.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:30 -0700 (PDT) Message-Id: <1f4ba56e741632be5d55effc4ed8b6d40e3863d2.1619213665.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:23 +0000 Subject: [PATCH v2 6/8] status: skip sparse-checkout percentage with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee 'git status' began reporting a percentage of populated paths when sparse-checkout is enabled in 051df3cf (wt-status: show sparse checkout status as well, 2020-07-18). This percentage is incorrect when the index has sparse directories. It would also be expensive to calculate as we would need to parse trees to count the total number of possible paths. Avoid the expensive computation by simplifying the output to only report that a sparse checkout exists, without the percentage. This change is the reason we use 'git status --porcelain=v2' in t1092-sparse-checkout-compatibility.sh. We don't want to ensure that this message is equal across both modes, but instead just the important information about staged, modified, and untracked files are compared. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 8 ++++++++ wt-status.c | 14 +++++++++++--- wt-status.h | 1 + 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 0ec487acd283..0dc551b25f67 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -196,6 +196,14 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 -uno ' +test_expect_success 'status reports sparse-checkout' ' + init_repos && + git -C sparse-checkout status >full && + git -C sparse-index status >sparse && + test_i18ngrep "You are in a sparse checkout with " full && + test_i18ngrep "You are in a sparse checkout." sparse +' + test_expect_success 'add, commit, checkout' ' init_repos && diff --git a/wt-status.c b/wt-status.c index 0c8287a023e4..0425169c1895 100644 --- a/wt-status.c +++ b/wt-status.c @@ -1490,9 +1490,12 @@ static void show_sparse_checkout_in_use(struct wt_status *s, if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_DISABLED) return; - status_printf_ln(s, color, - _("You are in a sparse checkout with %d%% of tracked files present."), - s->state.sparse_checkout_percentage); + if (s->state.sparse_checkout_percentage == SPARSE_CHECKOUT_SPARSE_INDEX) + status_printf_ln(s, color, _("You are in a sparse checkout.")); + else + status_printf_ln(s, color, + _("You are in a sparse checkout with %d%% of tracked files present."), + s->state.sparse_checkout_percentage); wt_longstatus_print_trailer(s); } @@ -1650,6 +1653,11 @@ static void wt_status_check_sparse_checkout(struct repository *r, return; } + if (r->index->sparse_index) { + state->sparse_checkout_percentage = SPARSE_CHECKOUT_SPARSE_INDEX; + return; + } + for (i = 0; i < r->index->cache_nr; i++) { struct cache_entry *ce = r->index->cache[i]; if (ce_skip_worktree(ce)) diff --git a/wt-status.h b/wt-status.h index 0d32799b28e1..ab9cc9d8f032 100644 --- a/wt-status.h +++ b/wt-status.h @@ -78,6 +78,7 @@ enum wt_status_format { }; #define SPARSE_CHECKOUT_DISABLED -1 +#define SPARSE_CHECKOUT_SPARSE_INDEX -2 struct wt_status_state { int merge_in_progress; From patchwork Fri Apr 23 21:34:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221767 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81B3FC433B4 for ; Fri, 23 Apr 2021 21:34:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5ABA361468 for ; Fri, 23 Apr 2021 21:34:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244106AbhDWVfW (ORCPT ); Fri, 23 Apr 2021 17:35:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51592 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244006AbhDWVfK (ORCPT ); Fri, 23 Apr 2021 17:35:10 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D187C061574 for ; Fri, 23 Apr 2021 14:34:32 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id a4so49692785wrr.2 for ; Fri, 23 Apr 2021 14:34:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JqWGd6Ui3AV4GFjFuc5ijzoQ+zhZRiNmDwuCLIAoU+U=; b=mF930G2lNaXxxrrkHbDB/Kto54VAb9ml9Uf0pkT2pJTxbXLyJrVEJDnPjSX3km0yC/ XG5MBG5hvTfZundyo874Gz2qp9ggYYF1HopYIHF+molG4arrrAB90MU0qiTYpeehdKU2 b7EgT6ZpgOLRQ0snaIxaW0u+Q3Z4bTYVazgGNrSC3WH0oe8m8FLQPUe5/5I+AoAxC5dp taFD88qUorlg/glPSVqW9QI1lGaIk/IVyG3h6TWsFmI4HPIMymWG619Iz+zbK+8neKmY Yzyx5PaHsJAjGQFF4nebde0zIkbqaxjqaJUKvQ75DRzvA5/t7duLMC1O8gZC5+4R8No8 jlMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JqWGd6Ui3AV4GFjFuc5ijzoQ+zhZRiNmDwuCLIAoU+U=; b=M61lfCCIl0s0B+GwGSb0D4R36d2ncxddnVbaKwWVqZKn2wmLiHxyPWnex++488pX/V MAi7TSIJVGuDLVRBs65MKbcZGmpuL6aj2KvMXmQlH1FrITiagzGzWm03AC4Q80pmkjNZ B0MMS2lL8hmZ7lg+NA7f3GI3/nnUWtZadffnOJrSsQ5ait68n7jgDnXnJOT7bB/XYFl+ ALDmyhXNSerCF18AS7fCUmTln8DNB9xNklMkrOgmx3WgJfEewb53nMnmZVakN+0ZYz2R Z7Wd9f/uhvP7s50dq6cVb3VpbJ80+95Hb66qUj28fQxaH0HGc6NFOBnhdYOJMVMB7SLz YxNQ== X-Gm-Message-State: AOAM530zR7GRg5MLq+XwQ5iJrPZKQwYd0MxGCJs0pZDVxgFj4cKHpPQY P6LBl1uET31N6XpusrtDEW59+UJoMuY= X-Google-Smtp-Source: ABdhPJz4zgiH3Vqdwp8TauxxU3zmakmUio1soaiD72f6yDzv0g7QjpurILKCgonb1aeeQ+RV2DUU7w== X-Received: by 2002:a5d:46d2:: with SMTP id g18mr6966689wrs.134.1619213671008; Fri, 23 Apr 2021 14:34:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x9sm10737044wrt.13.2021.04.23.14.34.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:30 -0700 (PDT) Message-Id: <3d09368c0541523d92ba272f3401b350f3ba3c0b.1619213666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:24 +0000 Subject: [PATCH v2 7/8] status: use sparse-index throughout Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By testing 'git -c core.fsmonitor= status -uno', we can check for the simplest index operations that can be made sparse-aware. The necessary implementation details are already integrated with sparse-checkout, so modify command_requires_full_index to be zero for cmd_status(). In refresh_index(), we loop through the index entries to refresh their stat() information. However, sparse directories have no stat() information to populate. Ignore these entries. This allows 'git status' to no longer expand a sparse index to a full one. This is further tested by dropping the "-uno" option and adding an untracked file into the worktree. The performance test p2000-sparse-checkout-operations.sh demonstrates these improvements: Test HEAD~1 HEAD ----------------------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.31(0.30+0.05) 0.31(0.29+0.06) +0.0% 2000.3: git status (full-index-v4) 0.31(0.29+0.07) 0.34(0.30+0.08) +9.7% 2000.4: git status (sparse-index-v3) 2.35(2.28+0.10) 0.04(0.04+0.05) -98.3% 2000.5: git status (sparse-index-v4) 2.35(2.24+0.15) 0.05(0.04+0.06) -97.9% Note that since HEAD~1 was expanding the sparse index by parsing trees, it was artificially slower than the full index case. Thus, the 98% improvement is misleading, and instead we should celebrate the 0.34s to 0.05s improvement of 85%. This is more indicative of the peformance gains we are expecting by using a sparse index. Note: we are dropping the assignment of core.fsmonitor here. This is not necessary for the test script as we are not altering the config any other way. Correct integration with FS Monitor will be validated in later changes. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 +++ read-cache.c | 10 ++++++++-- t/t1092-sparse-checkout-compatibility.sh | 13 +++++++++---- 3 files changed, 20 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index cf0c36d1dcb2..e529da7beadd 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1404,6 +1404,9 @@ int cmd_status(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_status_usage, builtin_status_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_status_config); argc = parse_options(argc, argv, prefix, builtin_status_options, diff --git a/read-cache.c b/read-cache.c index 29ffa9ac5db9..f80e26831b36 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1578,8 +1578,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, */ preload_index(istate, pathspec, 0); trace2_region_enter("index", "refresh", NULL); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); + for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; @@ -1594,6 +1593,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (ignore_skip_worktree && ce_skip_worktree(ce)) continue; + /* + * If this entry is a sparse directory, then there isn't + * any stat() information to update. Ignore the entry. + */ + if (S_ISSPARSEDIR(ce->ce_mode)) + continue; + if (pathspec && !ce_path_match(istate, ce, pathspec, seen)) filtered = 1; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 0dc551b25f67..5a8fe88dc894 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -453,12 +453,17 @@ test_expect_success 'sparse-index is expanded and converted back' ' GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ git -C sparse-index -c core.fsmonitor="" reset --hard && test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && + test_region index ensure_full_index trace2.txt +' - rm trace2.txt && +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + rm -f trace2.txt && + echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt + git -C sparse-index status && + test_region ! index ensure_full_index trace2.txt ' test_done From patchwork Fri Apr 23 21:34:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12221769 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7375EC433ED for ; Fri, 23 Apr 2021 21:34:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 59BBB6146B for ; Fri, 23 Apr 2021 21:34:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244116AbhDWVfZ (ORCPT ); Fri, 23 Apr 2021 17:35:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51600 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244030AbhDWVfL (ORCPT ); Fri, 23 Apr 2021 17:35:11 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C47B8C06174A for ; Fri, 23 Apr 2021 14:34:32 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id k4-20020a7bc4040000b02901331d89fb83so1893833wmi.5 for ; Fri, 23 Apr 2021 14:34:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jft80CHL8Z17lSl9Y7GRz9Mo3mBcOwwYstAOc67pnXo=; b=SIAn0+wT2qVtVtc6/hchH3asEqhuSDTH8vdhN2UXR0CkfPq3BS7CHwgKsA60XhSk08 HJJBxxPA0I6NZNg3vPy/rgroN294WhyhkdW3uO2OWOCw6i3uZjFYyVcZuEUe5ledgZnR 8G/TBi7a/uVu6H+wWMM33tzxmlTeI/MRoi0UIoyInUclkq85yjfIXZ7DTy8sD7wHv7+a wOlyuD9bVyY5pRV3brXoIEY++vuz1jYYXNLLZYbf53yNJtPPrI6HI0b2XOVyvT6HB6Zc X2g1ymGCUDTJz2a5SPj9tun9N4gL9L51NM93Cr0mgIKm8poH7FpnIfKTT+6D1LTvkKTZ mqyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jft80CHL8Z17lSl9Y7GRz9Mo3mBcOwwYstAOc67pnXo=; b=PnlL8AmJ//hKiIkARGlVoWZ7MRbwuZPet21CvHMM1VsS9T3FEghBYESeLzguDn3AbZ mfoEI+WBOj+dx4+NKSHQXMvwni9eHTH3uVk4e4BfVayhDBWhkGHc6P5/xF3sgsAB6/ZJ 626Av9hT5OqUTrD3+XZPBivWtmwmHK1J5vMotFVXpyOFGg3t6hHswEsffu4Bo7ups6qn 05aeHmaLUKYo+afXeu+QtW3wYUFo/M/UntXxPiUN/vmR/XSsjX5CU90cQj+edLxQRwGR Pd32lKoMhOAjvFKWxpL30a6hPrW15bMLci83OImo22BOc0NYpQwAiEkrllW66sSyK0kk qAMQ== X-Gm-Message-State: AOAM533g3NnFd43muymGrmgTOUw6ofd+AnjErF4JZ5t9zvnA2xvUAI5l cdIidGUEQF3N2Bz7L0EF47/sfJsnRw4= X-Google-Smtp-Source: ABdhPJzEOHaZS/8A7n0qqB2bB6iFNQ4IjPj4zhnoAGF+bXde9G/6OnzQVSCI/rBzZ87EPiMVqU3KuQ== X-Received: by 2002:a1c:64c4:: with SMTP id y187mr7562530wmb.162.1619213671564; Fri, 23 Apr 2021 14:34:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q7sm10323661wrr.62.2021.04.23.14.34.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Apr 2021 14:34:31 -0700 (PDT) Message-Id: <1fd033a6ebb2eebc85667920e93a4eeeab1b4fe0.1619213666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 23 Apr 2021 21:34:25 +0000 Subject: [PATCH v2 8/8] fsmonitor: test with sparse index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, Matheus Tavares Bernardino , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee During the effort to protect uses of the index to operate on a full index, we did not modify fsmonitor.c. This is because it already works effectively with only the change to index_name_stage_pos(). The only thing left to do is to test that it works correctly. These tests are added to demonstrate that the behavior is the same across a full index and a sparse index, but also that file modifications to a tracked directory outside of the sparse cone will trigger ensure_full_index(). Signed-off-by: Derrick Stolee --- t/t7519-status-fsmonitor.sh | 48 +++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 45d025f96010..f70fe961902e 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -73,6 +73,7 @@ test_expect_success 'setup' ' expect* actual* marker* + trace2* EOF ' @@ -383,4 +384,51 @@ test_expect_success 'status succeeds after staging/unstaging' ' ) ' +# Usage: +# check_sparse_index_behavior [!] +# If "!" is supplied, then we verify that we do not call ensure_full_index +# during a call to 'git status'. Otherwise, we verify that we _do_ call it. +check_sparse_index_behavior () { + git status --porcelain=v2 >expect && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set dir1 dir2 && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git status --porcelain=v2 >actual && + test_region $1 index ensure_full_index trace2.txt && + test_cmp expect actual && + rm trace2.txt && + git sparse-checkout disable +} + +test_expect_success 'status succeeds with sparse index' ' + git reset --hard && + + test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + EOF + git config core.fsmonitor .git/hooks/fsmonitor-test && + check_sparse_index_behavior ! && + + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1/modified\0" + EOF + check_sparse_index_behavior ! && + + cp -r dir1 dir1a && + git add dir1a && + git commit -m "add dir1a" && + + # This one modifies outside the sparse-checkout definition + # and hence we expect to expand the sparse-index. + write_script .git/hooks/fsmonitor-test<<-\EOF && + printf "last_update_token\0" + printf "dir1a/modified\0" + EOF + check_sparse_index_behavior +' + test_done