From patchwork Tue Mar 30 13:10:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D12C433E5 for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8106A619C5 for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232217AbhC3NLk (ORCPT ); Tue, 30 Mar 2021 09:11:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56648 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231986AbhC3NLJ (ORCPT ); Tue, 30 Mar 2021 09:11:09 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BC609C061762 for ; Tue, 30 Mar 2021 06:11:08 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id e18so16190178wrt.6 for ; Tue, 30 Mar 2021 06:11:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9j3j0x+FLRUsg8FjLEUdWVt79uLnrBjbjE/dVOIQKNs=; b=WoazZ3ITRQVUt8nWoCzKi0xKm30ekdMWiKU0CaT0lW65oFMaD12VgJb2miFjonUDuC 03nIQsJxZJV7SBYIEwZfqFclUaCgZuNH9+m9tlnszT7hhPI85E29yejtst2rWysnCbpn YkFvVXsZ20uNylvR67C4gKGLCtdjaQpr7jq20WPyR/8mhInNPHAScTUWNILSKIJWGu0a lXS56EAjoF+vno3B0mJWYgI/kxEX/tEFoXyt3FPC7erpYXU2fOe6qZduTcaRrX4a6kKR YPWc2jlTjOvqIDzgiRczup+2fderCRsy2XetAqEYd9EtoW1zxX5Rf73tnCkQzXurlpEi 7/lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9j3j0x+FLRUsg8FjLEUdWVt79uLnrBjbjE/dVOIQKNs=; b=d3eXBEuca+h4bIjbOtCvtnpyh07DE7XZNnwqIrOM+DMWVAL51zt/iomSYz1h02Rn7c QlC+BAsWS0/jwlRUCHOznR1Ht6/lYIUMGw/ImNxmK2srxRIzAvs2XAvN5wylG45yWgXO 1UaQDwc/OhaAw+6zFUt7oeHsnWcbfVPICTlXNUkKVmbUof00DAOIQj6Mb5taVD95sUSs xK3srWvCGxQ61SwrSWXNSMo5EfxvxvRifQ8aLysGaNlTgYVrw0fu8mwR6mGpI5fN7bNr JovMsnmvQGoLkoAiaOEAJ672ghqBI7SFyxy3VMaeO6EXndmvvAOubhJ+6ehDA8bXNiKR rXjw== X-Gm-Message-State: AOAM533mrY8soeWILYviB2lnmZzRqTQe1uXHdbCqrq5BIuIUBpjJNKow DquvDbWeZLx86MIarEy1vzqf3xIJh3g= X-Google-Smtp-Source: ABdhPJyPy1fMZh4oXSssvccz1LaxX0VpcYlR9xkGWv1XaDmdG0rEXsWGlWe2ynGXWIYWucuoAOYG/w== X-Received: by 2002:a5d:5904:: with SMTP id v4mr34214677wrd.261.1617109867314; Tue, 30 Mar 2021 06:11:07 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v189sm3623699wme.39.2021.03.30.06.11.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:06 -0700 (PDT) Message-Id: <7b600d536c6e7d06d204001c844d2fa1bb84e2db.1617109864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:44 +0000 Subject: [PATCH v5 01/21] sparse-index: design doc and format update Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This begins a long effort to update the index format to allow sparse directory entries. This should result in a significant improvement to Git commands when HEAD contains millions of files, but the user has selected many fewer files to keep in their sparse-checkout definition. Currently, the index format is only updated in the presence of extensions.sparseIndex instead of increasing a file format version number. This is temporary, and index v5 is part of the plan for future work in this area. The design document details many of the reasons for embarking on this work, and also the plan for completing it safely. Signed-off-by: Derrick Stolee --- Documentation/technical/index-format.txt | 7 + Documentation/technical/sparse-index.txt | 175 +++++++++++++++++++++++ 2 files changed, 182 insertions(+) create mode 100644 Documentation/technical/sparse-index.txt diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt index d363a71c37ec..3b74c05647db 100644 --- a/Documentation/technical/index-format.txt +++ b/Documentation/technical/index-format.txt @@ -44,6 +44,13 @@ Git index format localization, no special casing of directory separator '/'). Entries with the same name are sorted by their stage field. + An index entry typically represents a file. However, if sparse-checkout + is enabled in cone mode (`core.sparseCheckoutCone` is enabled) and the + `extensions.sparseIndex` extension is enabled, then the index may + contain entries for directories outside of the sparse-checkout definition. + These entries have mode `040000`, include the `SKIP_WORKTREE` bit, and + the path ends in a directory separator. + 32-bit ctime seconds, the last time a file's metadata changed this is stat(2) data diff --git a/Documentation/technical/sparse-index.txt b/Documentation/technical/sparse-index.txt new file mode 100644 index 000000000000..8d3d80804604 --- /dev/null +++ b/Documentation/technical/sparse-index.txt @@ -0,0 +1,175 @@ +Git Sparse-Index Design Document +================================ + +The sparse-checkout feature allows users to focus a working directory on +a subset of the files at HEAD. The cone mode patterns, enabled by +`core.sparseCheckoutCone`, allow for very fast pattern matching to +discover which files at HEAD belong in the sparse-checkout cone. + +Three important scale dimensions for a Git working directory are: + +* `HEAD`: How many files are present at `HEAD`? + +* Populated: How many files are within the sparse-checkout cone. + +* Modified: How many files has the user modified in the working directory? + +We will use big-O notation -- O(X) -- to denote how expensive certain +operations are in terms of these dimensions. + +These dimensions are ordered by their magnitude: users (typically) modify +fewer files than are populated, and we can only populate files at `HEAD`. + +Problems occur if there is an extreme imbalance in these dimensions. For +example, if `HEAD` contains millions of paths but the populated set has +only tens of thousands, then commands like `git status` and `git add` can +be dominated by operations that require O(`HEAD`) operations instead of +O(Populated). Primarily, the cost is in parsing and rewriting the index, +which is filled primarily with files at `HEAD` that are marked with the +`SKIP_WORKTREE` bit. + +The sparse-index intends to take these commands that read and modify the +index from O(`HEAD`) to O(Populated). To do this, we need to modify the +index format in a significant way: add "sparse directory" entries. + +With cone mode patterns, it is possible to detect when an entire +directory will have its contents outside of the sparse-checkout definition. +Instead of listing all of the files it contains as individual entries, a +sparse-index contains an entry with the directory name, referencing the +object ID of the tree at `HEAD` and marked with the `SKIP_WORKTREE` bit. +If we need to discover the details for paths within that directory, we +can parse trees to find that list. + +At time of writing, sparse-directory entries violate expectations about the +index format and its in-memory data structure. There are many consumers in +the codebase that expect to iterate through all of the index entries and +see only files. In fact, these loops expect to see a reference to every +staged file. One way to handle this is to parse trees to replace a +sparse-directory entry with all of the files within that tree as the index +is loaded. However, parsing trees is slower than parsing the index format, +so that is a slower operation than if we left the index alone. The plan is +to make all of these integrations "sparse aware" so this expansion through +tree parsing is unnecessary and they use fewer resources than when using a +full index. + +The implementation plan below follows four phases to slowly integrate with +the sparse-index. The intention is to incrementally update Git commands to +interact safely with the sparse-index without significant slowdowns. This +may not always be possible, but the hope is that the primary commands that +users need in their daily work are dramatically improved. + +Phase I: Format and initial speedups +------------------------------------ + +During this phase, Git learns to enable the sparse-index and safely parse +one. Protections are put in place so that every consumer of the in-memory +data structure can operate with its current assumption of every file at +`HEAD`. + +At first, every index parse will call a helper method, +`ensure_full_index()`, which scans the index for sparse-directory entries +(pointing to trees) and replaces them with the full list of paths (with +blob contents) by parsing tree objects. This will be slower in all cases. +The only noticeable change in behavior will be that the serialized index +file contains sparse-directory entries. + +To start, we use a new required index extension, `sdir`, to allow +inserting sparse-directory entries into indexes with file format +versions 2, 3, and 4. This prevents Git versions that do not understand +the sparse-index from operating on one, while allowing tools that do not +understand the sparse-index to operate on repositories as long as they do +not interact with the index. A new format, index v5, will be introduced +that includes sparse-directory entries by default. It might also +introduce other features that have been considered for improving the +index, as well. + +Next, consumers of the index will be guarded against operating on a +sparse-index by inserting calls to `ensure_full_index()` or +`expand_index_to_path()`. After these guards are in place, we can begin +leaving sparse-directory entries in the in-memory index structure. + +Even after inserting these guards, we will keep expanding sparse-indexes +for most Git commands using the `command_requires_full_index` repository +setting. This setting will be on by default and disabled one builtin at a +time until we have sufficient confidence that all of the index operations +are properly guarded. + +To complete this phase, the commands `git status` and `git add` will be +integrated with the sparse-index so that they operate with O(Populated) +performance. They will be carefully tested for operations within and +outside the sparse-checkout definition. + +Phase II: Careful integrations +------------------------------ + +This phase focuses on ensuring that all index extensions and APIs work +well with a sparse-index. This requires significant increases to our test +coverage, especially for operations that interact with the working +directory outside of the sparse-checkout definition. Some of these +behaviors may not be the desirable ones, such as some tests already +marked for failure in `t1092-sparse-checkout-compatibility.sh`. + +The index extensions that may require special integrations are: + +* FS Monitor +* Untracked cache + +While integrating with these features, we should look for patterns that +might lead to better APIs for interacting with the index. Coalescing +common usage patterns into an API call can reduce the number of places +where sparse-directories need to be handled carefully. + +Phase III: Important command speedups +------------------------------------- + +At this point, the patterns for testing and implementing sparse-directory +logic should be relatively stable. This phase focuses on updating some of +the most common builtins that use the index to operate as O(Populated). +Here is a potential list of commands that could be valuable to integrate +at this point: + +* `git commit` +* `git checkout` +* `git merge` +* `git rebase` + +Hopefully, commands such as `git merge` and `git rebase` can benefit +instead from merge algorithms that do not use the index as a data +structure, such as the merge-ORT strategy. As these topics mature, we +may enable the ORT strategy by default for repositories using the +sparse-index feature. + +Along with `git status` and `git add`, these commands cover the majority +of users' interactions with the working directory. In addition, we can +integrate with these commands: + +* `git grep` +* `git rm` + +These have been proposed as some whose behavior could change when in a +repo with a sparse-checkout definition. It would be good to include this +behavior automatically when using a sparse-index. Some clarity is needed +to make the behavior switch clear to the user. + +This phase is the first where parallel work might be possible without too +much conflicts between topics. + +Phase IV: The long tail +----------------------- + +This last phase is less a "phase" and more "the new normal" after all of +the previous work. + +To start, the `command_requires_full_index` option could be removed in +favor of expanding only when hitting an API guard. + +There are many Git commands that could use special attention to operate as +O(Populated), while some might be so rare that it is acceptable to leave +them with additional overhead when a sparse-index is present. + +Here are some commands that might be useful to update: + +* `git sparse-checkout set` +* `git am` +* `git clean` +* `git stash` From patchwork Tue Mar 30 13:10:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6DCEFC433DB for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 55968619CB for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232263AbhC3NLs (ORCPT ); Tue, 30 Mar 2021 09:11:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56650 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231968AbhC3NLK (ORCPT ); Tue, 30 Mar 2021 09:11:10 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6EC25C061764 for ; Tue, 30 Mar 2021 06:11:09 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id c8so16173514wrq.11 for ; Tue, 30 Mar 2021 06:11:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3Sib70KHcBBpG3tPfAQlzsWFLwlZKFCZNLjdNCTGm0w=; b=ooYIbTIgFNvl6okFUlqhzmmAoI7IZOtd05kY/10I7u7ppdkeCgxcaJjyDeV2gPE2Rj 73s6oDb2WKQJrsmFggfB0ocPLByjqmnkCzTqfc+Wu/0ZdmwMKSA2lR3AU3m5QXsyE1LN suXkDcdsZ7FsEeBYXHs+BSYVFo0HgPXFdLz58zWfanPuRi622RhS6WEQv/EbhV8pp1Sl K8DwpNyan+XMmRnT/eFEW1e2/lZNwf6O6s5Sx6gdDAPh/rtnQc2/KoB/jlu1jSuStkk8 hV12/WGhrc5cxZGVeUXBV8o00egoMgS8N9zmPBOZeIpo8qFMwwg3uX2hiv7V1grBUQPK VuSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3Sib70KHcBBpG3tPfAQlzsWFLwlZKFCZNLjdNCTGm0w=; b=W+d49bozU7egdfXh63hEkzX+ieWppzfHb+gntVSeZpU17YHcIwUqly8tuLdTSaMJOp JOoBiKrFAZoRn0EuRiEY77bU8ObDPNIrrObmQXozCuYxYUwydtszJIXbB/LpzND83iNu LS4YQqRzLtSOsYV2JOJ6cxY6qJnb40e0+WAzbFNEV87iymtkqTwq1dgWq9utBFB5R6l3 U1FdAATaHRe9hkpp2Mr4Db5TE1yV1zLtDxwRGtQItls4PSlPBjdx3FWL2WM36awvS6LC JnccOBS+qzskSJWUAAIUrscbJImnwZnQvE24pJP4pyul03C/ghSZS3rj7jLRPQ8+IDQz SgzA== X-Gm-Message-State: AOAM5302uj7liMuezjceHLVZf4ISbFPI5UOib6n8m7w8y8416lAKl7U6 wonDZ+bY7BocVpzZDlv7c6hrZfuRMzE= X-Google-Smtp-Source: ABdhPJxPlyG3nOMyftihHHmGYLA27ScW/J4Sh/c2VO49CGoSE9TNDzB9y2C+sorxPLIRHOETcxy/dg== X-Received: by 2002:a5d:6104:: with SMTP id v4mr5079698wrt.275.1617109868080; Tue, 30 Mar 2021 06:11:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 64sm3485625wmz.7.2021.03.30.06.11.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:07 -0700 (PDT) Message-Id: <202253ec82f34d1302ee7360e4d458ac4af89e82.1617109864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:45 +0000 Subject: [PATCH v5 02/21] t/perf: add performance test for sparse operations Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Create a test script that takes the default performance test (the Git codebase) and multiplies it by 256 using four layers of duplicated trees of width four. This results in nearly one million blob entries in the index. Then, we can clone this repository with sparse-checkout patterns that demonstrate four copies of the initial repository. Each clone will use a different index format or mode so peformance can be tested across the different options. Note that the initial repo is stripped of submodules before doing the copies. This preserves the expected data shape of the sparse index, because directories containing submodules are not collapsed to a sparse directory entry. Run a few Git commands on these clones, especially those that use the index (status, add, commit). Here are the results on my Linux machine: Test -------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.37(0.30+0.09) 2000.3: git status (full-index-v4) 0.39(0.32+0.10) 2000.4: git add -A (full-index-v3) 1.42(1.06+0.20) 2000.5: git add -A (full-index-v4) 1.26(0.98+0.16) 2000.6: git add . (full-index-v3) 1.40(1.04+0.18) 2000.7: git add . (full-index-v4) 1.26(0.98+0.17) 2000.8: git commit -a -m A (full-index-v3) 1.42(1.11+0.16) 2000.9: git commit -a -m A (full-index-v4) 1.33(1.08+0.16) It is perhaps noteworthy that there is an improvement when using index version 4. This is because the v3 index uses 108 MiB while the v4 index uses 80 MiB. Since the repeated portions of the directories are very short (f3/f1/f2, for example) this ratio is less pronounced than in similarly-sized real repositories. Signed-off-by: Derrick Stolee --- t/perf/p2000-sparse-operations.sh | 84 +++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100755 t/perf/p2000-sparse-operations.sh diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh new file mode 100755 index 000000000000..dddd527b6330 --- /dev/null +++ b/t/perf/p2000-sparse-operations.sh @@ -0,0 +1,84 @@ +#!/bin/sh + +test_description="test performance of Git operations using the index" + +. ./perf-lib.sh + +test_perf_default_repo + +SPARSE_CONE=f2/f4/f1 + +test_expect_success 'setup repo and indexes' ' + git reset --hard HEAD && + + # Remove submodules from the example repo, because our + # duplication of the entire repo creates an unlikely data shape. + if git config --file .gitmodules --get-regexp "submodule.*.path" >modules + then + git rm $(awk "{print \$2}" modules) && + git commit -m "remove submodules" || return 1 + fi && + + echo bogus >a && + cp a b && + git add a b && + git commit -m "level 0" && + BLOB=$(git rev-parse HEAD:a) && + OLD_COMMIT=$(git rev-parse HEAD) && + OLD_TREE=$(git rev-parse HEAD^{tree}) && + + for i in $(test_seq 1 4) + do + cat >in <<-EOF && + 100755 blob $BLOB a + 040000 tree $OLD_TREE f1 + 040000 tree $OLD_TREE f2 + 040000 tree $OLD_TREE f3 + 040000 tree $OLD_TREE f4 + EOF + NEW_TREE=$(git mktree >$SPARSE_CONE/a && + $command + ) + " + done +} + +test_perf_on_all git status +test_perf_on_all git add -A +test_perf_on_all git add . +test_perf_on_all git commit -a -m A + +test_done From patchwork Tue Mar 30 13:10:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59556C433F2 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D8E5619C8 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232258AbhC3NLr (ORCPT ); Tue, 30 Mar 2021 09:11:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232101AbhC3NLL (ORCPT ); Tue, 30 Mar 2021 09:11:11 -0400 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B84CC061765 for ; Tue, 30 Mar 2021 06:11:10 -0700 (PDT) Received: by mail-wr1-x42c.google.com with SMTP id j7so16228530wrd.1 for ; Tue, 30 Mar 2021 06:11:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=zT/KLSMEIY5jvVQ9z2xbPhRiUryKVuorFSgT+6Od1bg=; b=lbeHtc1/XMa2y3e0mqj3gkqn/XHLITwQJicXv2HYUgYlb6dYQiwIYihm5AnqSkARIb bYqi2aBqj2hPhSqtWAvfisZXSB87VV/i+Cb255GTzieKA9VZXxZuuixMTXIMsdw9Y56B QGhHC3S6zeIHoAb7adTBduSkJ1eLnJIpepf5vNPNQTIAOwIvblwcq9tEi2Guy3M5o5vt YwfQwrD16/jHpQoDvMIaksQR1WPeOe6ZL7+RbqHOuW+mUVpOOGCoXRjJDKidy+hL+1xN zNALj1tKBsLcQe3E+gEtoQNbbCcHxZ97hMguAxOMd7VWLXVUKC4Q+wCIcfKHRViv+H+z fneg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=zT/KLSMEIY5jvVQ9z2xbPhRiUryKVuorFSgT+6Od1bg=; b=Dfr5MRQJRFBTDnL3ya5dw9iy+86HmaNCIpSObhydXpOfLT4N1ffl7d4WsDhhqys2Nh yrFW8pEIxwfJt34/rXm5n3XxXCmzLp0OvzgE2+NA3C9DcLPuGWtizmUxtUj91ACnxnBu MGe7pRQnAe9d2NkAdqpKH/730KBGTzvcOLWUHT5rZpJWMQrXzOQIu0RAjG+3wF5x7nU4 0fDIFGwYWk3rXd8C7wtApcpr0luI5T9GkpapCrV2Tl3bGmgYDIJpjWiArnzqyd6oIhm8 tz3AZO76QZT31fDrjXukG8JjKtXbKFTcab1HNEVnKS36xysyd2agiPgyQOtU7KIkWZ3v daoA== X-Gm-Message-State: AOAM530vgj2JL0/vMnwO51bkA/rP94mUrttBLh8ssBdqDLrDxEnW7IeO tJuhUwwZNphGIlqujN6Lds/V+ggB20A= X-Google-Smtp-Source: ABdhPJw5D2FNPAaPY9whDDZsl/K3UvDromw4u+yDlg764+QM+jQGEMsPoCy+U5bMGDppQ/51lqBL/A== X-Received: by 2002:adf:e5cd:: with SMTP id a13mr33115463wrn.65.1617109868882; Tue, 30 Mar 2021 06:11:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j20sm3363396wmp.30.2021.03.30.06.11.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:08 -0700 (PDT) Message-Id: <437a0f144e57cd4d0811ff652e760cb961b1db1e.1617109864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:46 +0000 Subject: [PATCH v5 03/21] t1092: clean up script quoting Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This test was introduced in 19a0acc83e4 (t1092: test interesting sparse-checkout scenarios, 2021-01-23), but it contains issues with quoting that were not noticed until starting this follow-up series. The old mechanism would drop quoting such as in test_all_match git commit -m "touch README.md" The above happened to work because README.md is a file in the repository, so 'git commit -m touch REAMDE.md' would succeed by accident. Other cases included quoting for no good reason, so clean that up now. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 8cd3e5a8d227..3725d3997e70 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -96,20 +96,20 @@ init_repos () { run_on_sparse () { ( cd sparse-checkout && - $* >../sparse-checkout-out 2>../sparse-checkout-err + "$@" >../sparse-checkout-out 2>../sparse-checkout-err ) } run_on_all () { ( cd full-checkout && - $* >../full-checkout-out 2>../full-checkout-err + "$@" >../full-checkout-out 2>../full-checkout-err ) && - run_on_sparse $* + run_on_sparse "$@" } test_all_match () { - run_on_all $* && + run_on_all "$@" && test_cmp full-checkout-out sparse-checkout-out && test_cmp full-checkout-err sparse-checkout-err } @@ -119,7 +119,7 @@ test_expect_success 'status with options' ' test_all_match git status --porcelain=v2 && test_all_match git status --porcelain=v2 -z -u && test_all_match git status --porcelain=v2 -uno && - run_on_all "touch README.md" && + run_on_all touch README.md && test_all_match git status --porcelain=v2 && test_all_match git status --porcelain=v2 -z -u && test_all_match git status --porcelain=v2 -uno && @@ -135,7 +135,7 @@ test_expect_success 'add, commit, checkout' ' write_script edit-contents <<-\EOF && echo text >>$1 EOF - run_on_all "../edit-contents README.md" && + run_on_all ../edit-contents README.md && test_all_match git add README.md && test_all_match git status --porcelain=v2 && @@ -144,7 +144,7 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout HEAD~1 && test_all_match git checkout - && - run_on_all "../edit-contents README.md" && + run_on_all ../edit-contents README.md && test_all_match git add -A && test_all_match git status --porcelain=v2 && @@ -153,7 +153,7 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout HEAD~1 && test_all_match git checkout - && - run_on_all "../edit-contents deep/newfile" && + run_on_all ../edit-contents deep/newfile && test_all_match git status --porcelain=v2 -uno && test_all_match git status --porcelain=v2 && @@ -186,7 +186,7 @@ test_expect_success 'diff --staged' ' write_script edit-contents <<-\EOF && echo text >>README.md EOF - run_on_all "../edit-contents" && + run_on_all ../edit-contents && test_all_match git diff && test_all_match git diff --staged && @@ -280,7 +280,7 @@ test_expect_success 'clean' ' echo bogus >>.gitignore && run_on_all cp ../.gitignore . && test_all_match git add .gitignore && - test_all_match git commit -m ignore-bogus-files && + test_all_match git commit -m "ignore bogus files" && run_on_sparse mkdir folder1 && run_on_all touch folder1/bogus && From patchwork Tue Mar 30 13:10:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F272C433ED for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2A433619BB for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232185AbhC3NLq (ORCPT ); Tue, 30 Mar 2021 09:11:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56664 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232110AbhC3NLM (ORCPT ); Tue, 30 Mar 2021 09:11:12 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18F79C0613D8 for ; Tue, 30 Mar 2021 06:11:11 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id j9so14476702wrx.12 for ; Tue, 30 Mar 2021 06:11:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tG7+YZdwAKze8gBAJEZBRIJJgY5B1vbHrx8V3JKD0do=; b=STyRK7JcfvDzI43+YVi8PkBudg60XtQo3AHW7wl9ysAB1BvS+0w839Yk7E8cWKwN3+ WBnN6vUHD4TXoboJZ+F/Hrn0RwfRd0J3app98I3fhwpVX1KFlUhSm7PqDJVC4i/2y9BN oBaaJbIYrX6aHL+c9UHAQfYjW/sTf8YJpi5ViDud/EMlDYw/yaD09/bIHRkaKGni4256 HsGkJ/G4Zu3RTf/Zar0HBnGdqYfSQniCtD4czhPYxg7qr11uk0aqJspdeZuP5Xtj6R8G UrXYb1pTIM1zXmBGuzCrIU6UrA6HAPGUl/4rI69c2VO1ZfVQNOBx2JoYzLwl8JnMtMoE AIDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tG7+YZdwAKze8gBAJEZBRIJJgY5B1vbHrx8V3JKD0do=; b=TzAWsYqkoI19RfGlqv+gMYNvSkBVoycZtye9AQP+9BJH6K+Jt4gZCsx5KGQ+RlX1No CQ1W7Tp6vvJ8I0juwxylOsLrcMh452qua1bwbJNBpiFdn3NhkdLXjWRpQjSVm2WT5m+1 vrjle4aLkqjC4JfQzuWbpkbWfJiZPzSWBbWaKh+8qC/C5SBf4asgwfVsrXZSlRRTdUd1 QMxFTgB24m5IK/0ombPZmOwdMC0ggWvizxpDJBbSJ4f3GlvrELxe7H9piXMdzNaaTLUM NQnMULxqGVDpmgcFyZ7rOHDsJ5nx0qRNAmBuRzDKTspg64CvKb4Yn415AC9F4D4PAF+u ayGA== X-Gm-Message-State: AOAM532LZz9Sb24x5LqK3BEETX3vOXpoWsAebz+oOoGb/ALaKIO6Wnhh MHb9JhtmAcGIGt8rtbsUpoc3o9KFKto= X-Google-Smtp-Source: ABdhPJwL6me0X9QdLJz/4PTEp8v+eK/jkI/3AHGypV0worzsNaer5wWyVMffJllkIkelZbsR6GNlAA== X-Received: by 2002:adf:ee4f:: with SMTP id w15mr34249894wro.199.1617109869693; Tue, 30 Mar 2021 06:11:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j13sm34366286wrt.29.2021.03.30.06.11.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:09 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:47 +0000 Subject: [PATCH v5 04/21] sparse-index: add guard to ensure full index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Upcoming changes will introduce modifications to the index format that allow sparse directories. It will be useful to have a mechanism for converting those sparse index files into full indexes by walking the tree at those sparse directories. Name this method ensure_full_index() as it will guarantee that the index is fully expanded. This method is not implemented yet, and instead we focus on the scaffolding to declare it and call it at the appropriate time. Add a 'command_requires_full_index' member to struct repo_settings. This will be an indicator that we need the index in full mode to do certain index operations. This starts as being true for every command, then we will set it to false as some commands integrate with sparse indexes. If 'command_requires_full_index' is true, then we will immediately expand a sparse index to a full one upon reading from disk. This suffices for now, but we will want to add more callers to ensure_full_index() later. Signed-off-by: Derrick Stolee --- Makefile | 1 + repo-settings.c | 8 ++++++++ repository.c | 11 ++++++++++- repository.h | 2 ++ sparse-index.c | 8 ++++++++ sparse-index.h | 7 +++++++ 6 files changed, 36 insertions(+), 1 deletion(-) create mode 100644 sparse-index.c create mode 100644 sparse-index.h diff --git a/Makefile b/Makefile index dfb0f1000fa3..89b1d5374107 100644 --- a/Makefile +++ b/Makefile @@ -985,6 +985,7 @@ LIB_OBJS += setup.o LIB_OBJS += shallow.o LIB_OBJS += sideband.o LIB_OBJS += sigchain.o +LIB_OBJS += sparse-index.o LIB_OBJS += split-index.o LIB_OBJS += stable-qsort.o LIB_OBJS += strbuf.o diff --git a/repo-settings.c b/repo-settings.c index f7fff0f5ab83..d63569e4041e 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -77,4 +77,12 @@ void prepare_repo_settings(struct repository *r) UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_KEEP); UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_DEFAULT); + + /* + * This setting guards all index reads to require a full index + * over a sparse index. After suitable guards are placed in the + * codebase around uses of the index, this setting will be + * removed. + */ + r->settings.command_requires_full_index = 1; } diff --git a/repository.c b/repository.c index c98298acd017..a8acae002f71 100644 --- a/repository.c +++ b/repository.c @@ -10,6 +10,7 @@ #include "object.h" #include "lockfile.h" #include "submodule-config.h" +#include "sparse-index.h" /* The main repository */ static struct repository the_repo; @@ -261,6 +262,8 @@ void repo_clear(struct repository *repo) int repo_read_index(struct repository *repo) { + int res; + if (!repo->index) repo->index = xcalloc(1, sizeof(*repo->index)); @@ -270,7 +273,13 @@ int repo_read_index(struct repository *repo) else if (repo->index->repo != repo) BUG("repo's index should point back at itself"); - return read_index_from(repo->index, repo->index_file, repo->gitdir); + res = read_index_from(repo->index, repo->index_file, repo->gitdir); + + prepare_repo_settings(repo); + if (repo->settings.command_requires_full_index) + ensure_full_index(repo->index); + + return res; } int repo_hold_locked_index(struct repository *repo, diff --git a/repository.h b/repository.h index b385ca3c94b6..e06a23015697 100644 --- a/repository.h +++ b/repository.h @@ -41,6 +41,8 @@ struct repo_settings { enum fetch_negotiation_setting fetch_negotiation_algorithm; int core_multi_pack_index; + + unsigned command_requires_full_index:1; }; struct repository { diff --git a/sparse-index.c b/sparse-index.c new file mode 100644 index 000000000000..82183ead563b --- /dev/null +++ b/sparse-index.c @@ -0,0 +1,8 @@ +#include "cache.h" +#include "repository.h" +#include "sparse-index.h" + +void ensure_full_index(struct index_state *istate) +{ + /* intentionally left blank */ +} diff --git a/sparse-index.h b/sparse-index.h new file mode 100644 index 000000000000..09a20d036c46 --- /dev/null +++ b/sparse-index.h @@ -0,0 +1,7 @@ +#ifndef SPARSE_INDEX_H__ +#define SPARSE_INDEX_H__ + +struct index_state; +void ensure_full_index(struct index_state *istate); + +#endif From patchwork Tue Mar 30 13:10:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF52BC433EA for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A2B5A6190A for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232165AbhC3NLq (ORCPT ); Tue, 30 Mar 2021 09:11:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56666 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232115AbhC3NLM (ORCPT ); Tue, 30 Mar 2021 09:11:12 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CAF7FC0613D9 for ; Tue, 30 Mar 2021 06:11:11 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id v11so16161326wro.7 for ; Tue, 30 Mar 2021 06:11:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pOHqh+Af5P0jXkTT9g9iWoHBkxvTMOM1AiMZUVBuQro=; b=KknN4iFQdTaQog8lNpgogkZut8UjVKbJwEb4N4sGEmfkoCnePBfAsRjI3OvYarQ6LF 90K+OGLfLa4Ownm1KWkutf/N5PNnR2VUS04ZBl1v2nTWdXkMNu8hrI2UTqwHLXwKg9fh SI0Ln6X+btQtNQiw9Tb5oxX7UiRx065Chm9fWjnxDnFawYD8xdeO8S8saGlK4hs6hKYn be0grW//XSao2YuVurSOWlB1DqO4Zvs0nxlbM9uNjkwE8TE3s1ctonBgwciKQ/BZWeqc mtJsDPXMSkbqlrGvldNDH6eBzXjgiN4azcSDNtMcOWCUHdrPFjJyEvRlYlWyRnlQx402 6LxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pOHqh+Af5P0jXkTT9g9iWoHBkxvTMOM1AiMZUVBuQro=; b=OZJD6WuR1bI2zBlMV1Vi31b9s1Vgi+ti/kFpLGN04TQ5UZA2UaLR+J9eY/eYM26+mN ZKt0pGO9yOYuQj45VJhB5wA/toI80/VmnxWXg4lxDyFcQTjRDweJIVlhXyEZ76LehPwT wMrsF6R6SrGp5TbuMXk56PK3BfldNaDPaYtFBcKwj16mOlTtX74Im2kloGuExTGV4rgA VEHseK52XlUgrT8hCDQkb68/bjVZ64syMDDq96rOKNZAtohCdbcv8HWXepIPdP0sLtgP a9aaaDB3uJRuPS7WkKiyFg+PMviqWroGvtWcdrZlfZRsHWzRwo9LDpC3wQYBRUTt46W8 fsZQ== X-Gm-Message-State: AOAM531MvhP+oH4MifAwBTXy0aAF1PCRKDhnhzqq36VebhRs5opHNKpj dJvHU0x5rDuPcTq+NViasYLJ5BqWl3Q= X-Google-Smtp-Source: ABdhPJwRUrpRDTq4u3rrX5qgUF8dUTH6EFkgvktJ9i2ng+MbjgnozMZUuScC0A7DmA4rEIgj8ZuMCA== X-Received: by 2002:a5d:6c6f:: with SMTP id r15mr34417616wrz.77.1617109870555; Tue, 30 Mar 2021 06:11:10 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g16sm35192702wrs.76.2021.03.30.06.11.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:10 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:48 +0000 Subject: [PATCH v5 05/21] sparse-index: implement ensure_full_index() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee We will mark an in-memory index_state as having sparse directory entries with the sparse_index bit. These currently cannot exist, but we will add a mechanism for collapsing a full index to a sparse one in a later change. That will happen at write time, so we must first allow parsing the format before writing it. Commands or methods that require a full index in order to operate can call ensure_full_index() to expand that index in-memory. This requires parsing trees using that index's repository. Sparse directory entries have a specific 'ce_mode' value. The macro S_ISSPARSEDIR(ce->ce_mode) can check if a cache_entry 'ce' has this type. This ce_mode is not possible with the existing index formats, so we don't also verify all properties of a sparse-directory entry, which are: 1. ce->ce_mode == 0040000 2. ce->flags & CE_SKIP_WORKTREE is true 3. ce->name[ce->namelen - 1] == '/' (ends in dir separator) 4. ce->oid references a tree object. These are all semi-enforced in ensure_full_index() to some extent. Any deviation will cause a warning at minimum or a failure in the worst case. Signed-off-by: Derrick Stolee --- cache.h | 13 ++++++- read-cache.c | 9 +++++ sparse-index.c | 98 +++++++++++++++++++++++++++++++++++++++++++++++++- 3 files changed, 118 insertions(+), 2 deletions(-) diff --git a/cache.h b/cache.h index bb317abc91fb..136dd496c95d 100644 --- a/cache.h +++ b/cache.h @@ -204,6 +204,8 @@ struct cache_entry { #error "CE_EXTENDED_FLAGS out of range" #endif +#define S_ISSPARSEDIR(m) ((m) == S_IFDIR) + /* Forward structure decls */ struct pathspec; struct child_process; @@ -319,7 +321,14 @@ struct index_state { drop_cache_tree : 1, updated_workdir : 1, updated_skipworktree : 1, - fsmonitor_has_run_once : 1; + fsmonitor_has_run_once : 1, + + /* + * sparse_index == 1 when sparse-directory + * entries exist. Requires sparse-checkout + * in cone mode. + */ + sparse_index : 1; struct hashmap name_hash; struct hashmap dir_hash; struct object_id oid; @@ -722,6 +731,8 @@ int read_index_from(struct index_state *, const char *path, const char *gitdir); int is_index_unborn(struct index_state *); +void ensure_full_index(struct index_state *istate); + /* For use with `write_locked_index()`. */ #define COMMIT_LOCK (1 << 0) #define SKIP_IF_UNCHANGED (1 << 1) diff --git a/read-cache.c b/read-cache.c index 1e9a50c6c734..dd3980c12b53 100644 --- a/read-cache.c +++ b/read-cache.c @@ -101,6 +101,9 @@ static const char *alternate_index_output; static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce) { + if (S_ISSPARSEDIR(ce->ce_mode)) + istate->sparse_index = 1; + istate->cache[nr] = ce; add_name_hash(istate, ce); } @@ -2273,6 +2276,12 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist) trace2_data_intmax("index", the_repository, "read/cache_nr", istate->cache_nr); + if (!istate->repo) + istate->repo = the_repository; + prepare_repo_settings(istate->repo); + if (istate->repo->settings.command_requires_full_index) + ensure_full_index(istate); + return istate->cache_nr; unmap: diff --git a/sparse-index.c b/sparse-index.c index 82183ead563b..7095378a1b28 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -1,8 +1,104 @@ #include "cache.h" #include "repository.h" #include "sparse-index.h" +#include "tree.h" +#include "pathspec.h" +#include "trace2.h" + +static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce) +{ + ALLOC_GROW(istate->cache, nr + 1, istate->cache_alloc); + + istate->cache[nr] = ce; + add_name_hash(istate, ce); +} + +static int add_path_to_index(const struct object_id *oid, + struct strbuf *base, const char *path, + unsigned int mode, void *context) +{ + struct index_state *istate = (struct index_state *)context; + struct cache_entry *ce; + size_t len = base->len; + + if (S_ISDIR(mode)) + return READ_TREE_RECURSIVE; + + strbuf_addstr(base, path); + + ce = make_cache_entry(istate, mode, oid, base->buf, 0, 0); + ce->ce_flags |= CE_SKIP_WORKTREE; + set_index_entry(istate, istate->cache_nr++, ce); + + strbuf_setlen(base, len); + return 0; +} void ensure_full_index(struct index_state *istate) { - /* intentionally left blank */ + int i; + struct index_state *full; + struct strbuf base = STRBUF_INIT; + + if (!istate || !istate->sparse_index) + return; + + if (!istate->repo) + istate->repo = the_repository; + + trace2_region_enter("index", "ensure_full_index", istate->repo); + + /* initialize basics of new index */ + full = xcalloc(1, sizeof(struct index_state)); + memcpy(full, istate, sizeof(struct index_state)); + + /* then change the necessary things */ + full->sparse_index = 0; + full->cache_alloc = (3 * istate->cache_alloc) / 2; + full->cache_nr = 0; + ALLOC_ARRAY(full->cache, full->cache_alloc); + + for (i = 0; i < istate->cache_nr; i++) { + struct cache_entry *ce = istate->cache[i]; + struct tree *tree; + struct pathspec ps; + + if (!S_ISSPARSEDIR(ce->ce_mode)) { + set_index_entry(full, full->cache_nr++, ce); + continue; + } + if (!(ce->ce_flags & CE_SKIP_WORKTREE)) + warning(_("index entry is a directory, but not sparse (%08x)"), + ce->ce_flags); + + /* recursively walk into cd->name */ + tree = lookup_tree(istate->repo, &ce->oid); + + memset(&ps, 0, sizeof(ps)); + ps.recursive = 1; + ps.has_wildcard = 1; + ps.max_depth = -1; + + strbuf_setlen(&base, 0); + strbuf_add(&base, ce->name, strlen(ce->name)); + + read_tree_at(istate->repo, tree, &base, &ps, + add_path_to_index, full); + + /* free directory entries. full entries are re-used */ + discard_cache_entry(ce); + } + + /* Copy back into original index. */ + memcpy(&istate->name_hash, &full->name_hash, sizeof(full->name_hash)); + istate->sparse_index = 0; + free(istate->cache); + istate->cache = full->cache; + istate->cache_nr = full->cache_nr; + istate->cache_alloc = full->cache_alloc; + + strbuf_release(&base); + free(full); + + trace2_region_leave("index", "ensure_full_index", istate->repo); } From patchwork Tue Mar 30 13:10:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C3B5C433E9 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CFA9E619DA for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232084AbhC3NLq (ORCPT ); Tue, 30 Mar 2021 09:11:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232117AbhC3NLN (ORCPT ); Tue, 30 Mar 2021 09:11:13 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B96FAC061574 for ; Tue, 30 Mar 2021 06:11:12 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id n11-20020a05600c4f8bb029010e5cf86347so986260wmq.1 for ; Tue, 30 Mar 2021 06:11:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3UfR33i2EmD24Ig4R+PabyKQOjZ/JHxeIKOaa94XMX0=; b=m1HJ90YpNhBpO3AGhvuWbwGKF5uh7Ud7/4D7TjBOcuVMgucubeeVqTcBE+GbZWuueH S6OVlyDdWfW0f4ei+zt9xovCIfMEfswk6DmPM1TWkxr7a68ELv4qis0VMx83+DNEQdUB /d2i3kRwENph1lSqKhfHToTf0ITIdeVFKqqa+Gt+elmIp4IKXsxq06M1LvYz1/kipggL rkF37hjR829jRH6TTVQ3ENvTsQUblcZMpiV6mgEDZx+pRsz6ahfImzsg2KJ7Vk0YkN3F 4ziZw3xcBNmH0s4lZ1Btsfd5ZAKKCGiJNpH/GDyttlJOWzN695rf6lalX1icVdmZmP8s RxbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3UfR33i2EmD24Ig4R+PabyKQOjZ/JHxeIKOaa94XMX0=; b=oG5+q7Xehl9SYaPRNHnQSKfd1txU4Nem46mD2o8NDY7r7jPfM3t79aM40AXndogVhh aboSm/f5bCuOaPkol+yv8N2JSsEZZsgxkDMSiF4Rr+JDVsEgUR+45WsbFVbHLNxn5NrV O9/i4fB2Yj36NzHaGsCeNm1aBVW+xFeYTUneKp379VilixRCUI2j17GIk0r3Zh1O/+Ak B+CfVdL1NxpvjNH3MfLJDJlYBp35M1oqjja7/9u4OiWc27kOpg02Ndv1OHIO9zhNShjj jlgRNjm9WYYyERDHKmOFVIuDQSBDCEDunexn40t9emgg5moJPcPpUYRM9XlGKZi12s4x QPgg== X-Gm-Message-State: AOAM532wmrpIo5TKvVbfT3xiYbXKhRjxOvy1OQd8ZsYFjHTNBMg44QEo bdzKl8KPljDx3+o5tKC4NUYmxWOwTiM= X-Google-Smtp-Source: ABdhPJyYl1iJWwZVxCgLklSVi45frxHUjaZOf9mmrnyOM+3i88VifGheFaVQDn+YRICUAGCNwuQ66Q== X-Received: by 2002:a7b:cd8f:: with SMTP id y15mr4086058wmj.185.1617109871233; Tue, 30 Mar 2021 06:11:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o2sm3644424wme.19.2021.03.30.06.11.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:10 -0700 (PDT) Message-Id: <7bfbfbd17321942f8f3a9b799eeb09bf1c58a947.1617109864.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:49 +0000 Subject: [PATCH v5 06/21] t1092: compare sparse-checkout to sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Add a new 'sparse-index' repo alongside the 'full-checkout' and 'sparse-checkout' repos in t1092-sparse-checkout-compatibility.sh. Also add run_on_sparse and test_sparse_match helpers. These helpers will be used when the sparse index is implemented. Add the GIT_TEST_SPARSE_INDEX environment variable to enable the sparse-index by default. This can be enabled across all tests, but that will only affect cases where the sparse-checkout feature is enabled. Signed-off-by: Derrick Stolee --- t/README | 3 +++ t/t1092-sparse-checkout-compatibility.sh | 24 ++++++++++++++++++++---- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/t/README b/t/README index 593d4a4e270c..b98bc563aab5 100644 --- a/t/README +++ b/t/README @@ -439,6 +439,9 @@ and "sha256". GIT_TEST_WRITE_REV_INDEX=, when true enables the 'pack.writeReverseIndex' setting. +GIT_TEST_SPARSE_INDEX=, when true enables index writes to use the +sparse-index format by default. + Naming Tests ------------ diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 3725d3997e70..de5d8461c993 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -7,6 +7,7 @@ test_description='compare full workdir to sparse workdir' test_expect_success 'setup' ' git init initial-repo && ( + GIT_TEST_SPARSE_INDEX=0 && cd initial-repo && echo a >a && echo "after deep" >e && @@ -87,23 +88,32 @@ init_repos () { cp -r initial-repo sparse-checkout && git -C sparse-checkout reset --hard && - git -C sparse-checkout sparse-checkout init --cone && + + cp -r initial-repo sparse-index && + git -C sparse-index reset --hard && # initialize sparse-checkout definitions - git -C sparse-checkout sparse-checkout set deep + git -C sparse-checkout sparse-checkout init --cone && + git -C sparse-checkout sparse-checkout set deep && + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout init --cone && + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep } run_on_sparse () { ( cd sparse-checkout && - "$@" >../sparse-checkout-out 2>../sparse-checkout-err + GIT_TEST_SPARSE_INDEX=0 "$@" >../sparse-checkout-out 2>../sparse-checkout-err + ) && + ( + cd sparse-index && + GIT_TEST_SPARSE_INDEX=1 "$@" >../sparse-index-out 2>../sparse-index-err ) } run_on_all () { ( cd full-checkout && - "$@" >../full-checkout-out 2>../full-checkout-err + GIT_TEST_SPARSE_INDEX=0 "$@" >../full-checkout-out 2>../full-checkout-err ) && run_on_sparse "$@" } @@ -114,6 +124,12 @@ test_all_match () { test_cmp full-checkout-err sparse-checkout-err } +test_sparse_match () { + run_on_sparse "$@" && + test_cmp sparse-checkout-out sparse-index-out && + test_cmp sparse-checkout-err sparse-index-err +} + test_expect_success 'status with options' ' init_repos && test_all_match git status --porcelain=v2 && From patchwork Tue Mar 30 13:10:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1CBB0C433EC for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E1FDE619BB for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232257AbhC3NLp (ORCPT ); Tue, 30 Mar 2021 09:11:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232128AbhC3NLP (ORCPT ); Tue, 30 Mar 2021 09:11:15 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5FDF8C0613DA for ; Tue, 30 Mar 2021 06:11:13 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id j4-20020a05600c4104b029010c62bc1e20so8380430wmi.3 for ; Tue, 30 Mar 2021 06:11:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=b0SPiO5pLHpc3Yqb6Wr4cuzcLpirEqK37PQkp7wDU/s=; b=kjvmCFez32VGpDwS+04dZydBpQQj6yQmaawPl+0s4s7jB66Yx24VVmlmXfQ7jLgjcB 2zV98TOmIEB4x41YX5xNN1fIH/5JBrbyRR7rBafvanJix0ebEds1XQ41d0knCzP+loES jNjU+utySBk986fll0NaDXUfupZ02XBJ3qTP3o1jsWrfLBoIp+QVYSk2BH+AujeBuaGH PXOGJbfUn6Bb6S5kmZZlAKXuwDOer8dDAvINV+5Xmc5/jFxuV6d1Ekj3hsxg2NwzQYWc w767NPtHgRWL0TkeSMSPC0Ef0FlVat1k8ohjYu6v/3tgv7/bMkbghiQzv5XobWZzLv9i YBeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=b0SPiO5pLHpc3Yqb6Wr4cuzcLpirEqK37PQkp7wDU/s=; b=DA0Iw2LRO1m3ZV/Pf4/EUhYG0tYmZ3YEEkXYEK67j/3EebsrqpoCA8fTnU1GQlIkua dQk1Dtyhzom4Os+Ft+tXQM8ox2So0M2wIkNJrCQVIzfX/8Jr/fulHRpoyGGVWvaFAcIq wu9f+TOC6YSdM9XhiiDr577Tkjc+x+7FoMLuaTesoHJKi4SboCnVfElDOXijJrIk4tpy ru19gHOdxxf6zOzTSZOPL9Fwvp+U4YkOfFBbOO8u5iKSiApo5Gep8NgX1ENLhHWmN5yb pBZQWuwUL34SsaU180SZiCAodWruZfd8zvtWGszBLkdzCfpB53cAwtZvqTbVMz1JjRul j8hA== X-Gm-Message-State: AOAM532KguKtJfn9yhBSIPSnDM/dl3lcTLTKOe+MM1Zg3+hDPIA29MEV q//5gPTy6uW8Y1jHfhi/Orsk+gxYd90= X-Google-Smtp-Source: ABdhPJy6GK6mgyrcaErvHD4+pxOEU37myqFALIjN/uF079CSattT4/fk6zsjRedyydthAO4FtrLESg== X-Received: by 2002:a1c:f20e:: with SMTP id s14mr4106389wmc.100.1617109872191; Tue, 30 Mar 2021 06:11:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t14sm35214211wru.64.2021.03.30.06.11.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:11 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:50 +0000 Subject: [PATCH v5 07/21] test-read-cache: print cache entries with --table Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee This table is helpful for discovering data in the index to ensure it is being written correctly, especially as we build and test the sparse-index. This table includes an output format similar to 'git ls-tree', but should not be compared to that directly. The biggest reasons are that 'git ls-tree' includes a tree entry for every subdirectory, even those that would not appear as a sparse directory in a sparse-index. Further, 'git ls-tree' does not use a trailing directory separator for its tree rows. This does not print the stat() information for the blobs. That will be added in a future change with another option. The tests that are added in the next few changes care only about the object types and IDs. However, this future need for full index information justifies the need for this test helper over extending a user-facing feature, such as 'git ls-files'. To make the option parsing slightly more robust, wrap the string comparisons in a loop adapted from test-dir-iterator.c. Care must be taken with the final check for the 'cnt' variable. We continue the expectation that the numerical value is the final argument. Signed-off-by: Derrick Stolee --- t/helper/test-read-cache.c | 55 +++++++++++++++++++++++++++++++------- 1 file changed, 45 insertions(+), 10 deletions(-) diff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c index 244977a29bdf..6cfd8f2de71c 100644 --- a/t/helper/test-read-cache.c +++ b/t/helper/test-read-cache.c @@ -1,36 +1,71 @@ #include "test-tool.h" #include "cache.h" #include "config.h" +#include "blob.h" +#include "commit.h" +#include "tree.h" + +static void print_cache_entry(struct cache_entry *ce) +{ + const char *type; + printf("%06o ", ce->ce_mode & 0177777); + + if (S_ISSPARSEDIR(ce->ce_mode)) + type = tree_type; + else if (S_ISGITLINK(ce->ce_mode)) + type = commit_type; + else + type = blob_type; + + printf("%s %s\t%s\n", + type, + oid_to_hex(&ce->oid), + ce->name); +} + +static void print_cache(struct index_state *istate) +{ + int i; + for (i = 0; i < istate->cache_nr; i++) + print_cache_entry(istate->cache[i]); +} int cmd__read_cache(int argc, const char **argv) { + struct repository *r = the_repository; int i, cnt = 1; const char *name = NULL; + int table = 0; - if (argc > 1 && skip_prefix(argv[1], "--print-and-refresh=", &name)) { - argc--; - argv++; + for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) { + if (skip_prefix(*argv, "--print-and-refresh=", &name)) + continue; + if (!strcmp(*argv, "--table")) + table = 1; } - if (argc == 2) - cnt = strtol(argv[1], NULL, 0); + if (argc == 1) + cnt = strtol(argv[0], NULL, 0); setup_git_directory(); git_config(git_default_config, NULL); + for (i = 0; i < cnt; i++) { - read_cache(); + repo_read_index(r); if (name) { int pos; - refresh_index(&the_index, REFRESH_QUIET, + refresh_index(r->index, REFRESH_QUIET, NULL, NULL, NULL); - pos = index_name_pos(&the_index, name, strlen(name)); + pos = index_name_pos(r->index, name, strlen(name)); if (pos < 0) die("%s not in index", name); printf("%s is%s up to date\n", name, - ce_uptodate(the_index.cache[pos]) ? "" : " not"); + ce_uptodate(r->index->cache[pos]) ? "" : " not"); write_file(name, "%d\n", i); } - discard_cache(); + if (table) + print_cache(r->index); + discard_index(r->index); } return 0; } From patchwork Tue Mar 30 13:10:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF25DC433E8 for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 929BD61994 for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232246AbhC3NLo (ORCPT ); Tue, 30 Mar 2021 09:11:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56686 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232135AbhC3NLP (ORCPT ); Tue, 30 Mar 2021 09:11:15 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A457C0613DB for ; Tue, 30 Mar 2021 06:11:14 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id v11so16161486wro.7 for ; Tue, 30 Mar 2021 06:11:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3o+t5/dOGp5xpMAAQd/XUzGtlCJPY6SPrX9Z+uQ/DZ4=; b=lltXkex0f+N5vdT0LbJu8Bol/t1IXEI7EA5wSjwp1wF8KCTobhPkci0sJbKWO3F9y/ 0jD5aqfnETuNiNHcD0sGsfMbBad39gUGYEG6spGEff9rNx24m+9f9Qq1CVgKH8eIfNvc Xdv7q7EG4iaRXEUI3SUGijwNEtvyTZW35gkYdz5a7+GThkwQb/1fyJoDsOqndyMTP+XG xG+S4BRw6keaaANTMinWZKsdVkxTLIDvl3nGvvnXJ5yQDe5IAsGvy/egmSmxFJvszKsl wqvhJU02INOy1NVM+Mu/ftLkVKSlplAdycKj0sLcy3MwnFhH8k6k1iicym5iPOfGcVst UKyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3o+t5/dOGp5xpMAAQd/XUzGtlCJPY6SPrX9Z+uQ/DZ4=; b=mR+9ViWxjWJ7ZoZUS2NECAakOEjpImdwWjACYt/JJ1/tzdBYX4m2Y971iQANMGsU/Z 2RwejjW/mitaTqwWdUPj+RZOT/TYh7I76x79tUgXUVNrIQCZ+gwPGM9GABULnjdRqJ6f oYvjqYD18HurncBjqUEMHKvmVK4wq+Y85prjuEu5RQPj/aiFEiF5gxfL1Ei5sMvCf3XS 91/m5akesvarNy2sq5uWi7goVRLEz2bENufGJiC5J/lhwyma+Zis0a3sT+u6fGiVIyLv wFRva4aStpTKHIVaHfVgNUo+JaWJqa63qiB6wfTvrhibkzreLmfaBDQ+IhxDqwcyQZGi Xijg== X-Gm-Message-State: AOAM530ssquhUnheRLcqzlqv3mLTmNHhbuxn/16K0BG4ODUkBGNWjfGX R0lQpmiXf5F1PgQR0aIi8CT3BVvCv08= X-Google-Smtp-Source: ABdhPJx2BfdrzJn1iRDtoTtmccPHBizHgtfS32zSlUNLUWwWKvRgqRXVBo0OgNOOtGxlKeZTBcok1Q== X-Received: by 2002:adf:d1e3:: with SMTP id g3mr34240078wrd.195.1617109873034; Tue, 30 Mar 2021 06:11:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o62sm5307691wmo.3.2021.03.30.06.11.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:12 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:51 +0000 Subject: [PATCH v5 08/21] test-tool: don't force full index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee We will use 'test-tool read-cache --table' to check that a sparse index is written as part of init_repos. Since we will no longer always expand a sparse index into a full index, add an '--expand' parameter that adds a call to ensure_full_index() so we can compare a sparse index directly against a full index, or at least what the in-memory index looks like when expanded in this way. Signed-off-by: Derrick Stolee --- t/helper/test-read-cache.c | 13 ++++++++++++- t/t1092-sparse-checkout-compatibility.sh | 5 +++++ 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/t/helper/test-read-cache.c b/t/helper/test-read-cache.c index 6cfd8f2de71c..b52c174acc7a 100644 --- a/t/helper/test-read-cache.c +++ b/t/helper/test-read-cache.c @@ -4,6 +4,7 @@ #include "blob.h" #include "commit.h" #include "tree.h" +#include "sparse-index.h" static void print_cache_entry(struct cache_entry *ce) { @@ -35,13 +36,19 @@ int cmd__read_cache(int argc, const char **argv) struct repository *r = the_repository; int i, cnt = 1; const char *name = NULL; - int table = 0; + int table = 0, expand = 0; + + initialize_the_repository(); + prepare_repo_settings(r); + r->settings.command_requires_full_index = 0; for (++argv, --argc; *argv && starts_with(*argv, "--"); ++argv, --argc) { if (skip_prefix(*argv, "--print-and-refresh=", &name)) continue; if (!strcmp(*argv, "--table")) table = 1; + else if (!strcmp(*argv, "--expand")) + expand = 1; } if (argc == 1) @@ -51,6 +58,10 @@ int cmd__read_cache(int argc, const char **argv) for (i = 0; i < cnt; i++) { repo_read_index(r); + + if (expand) + ensure_full_index(r->index); + if (name) { int pos; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index de5d8461c993..a1aea141c62c 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -130,6 +130,11 @@ test_sparse_match () { test_cmp sparse-checkout-err sparse-index-err } +test_expect_success 'expanded in-memory index matches full index' ' + init_repos && + test_sparse_match test-tool read-cache --expand --table +' + test_expect_success 'status with options' ' init_repos && test_all_match git status --porcelain=v2 && From patchwork Tue Mar 30 13:10:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 731BEC433E4 for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5402161994 for ; Tue, 30 Mar 2021 13:12:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232239AbhC3NLm (ORCPT ); Tue, 30 Mar 2021 09:11:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56688 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232141AbhC3NLP (ORCPT ); Tue, 30 Mar 2021 09:11:15 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32969C0613DC for ; Tue, 30 Mar 2021 06:11:15 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id n11-20020a05600c4f8bb029010e5cf86347so986337wmq.1 for ; Tue, 30 Mar 2021 06:11:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=i+gKe8EgHfQlDyjY32XCjj4qcNKDY7Jfl7pEV2IN20w=; b=XyWDIo9q9dYyhxQN4WHKj1TmJOlEVow8IhFQv9eXmLFuHi+rbXyPML5bIpp+0Y3ctF 2d5jR+ULDtTO++43F4UVbqPbFdofUj7IxYmFPs1u63qfjpRNPjq5M3O585JEQaYtfm5A b7FU9aq8ATqyayMel6gCGn9gaAvQa5t+DdpDHZHv6TdT853OfbvdKkNseRi2n0BY8xqA gkcA3nDvxODWnbMKo4QnEqgO0YAI5eZHRbT2X+oe6RQGe+q4JVXr/DKZd8+NEyEqrDbS wqKmx7aGasbd/yt+aBYh+l8VM4+OwPVXu0iH0vUgETg5iIj0x3cKjkhF/LehHR/ygxl7 XaPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=i+gKe8EgHfQlDyjY32XCjj4qcNKDY7Jfl7pEV2IN20w=; b=qQOiQAceTu9w/MnJgxYCsZMy/j04Ci7GFS/krli56mXycHGNLfUb1SSw5QN6QFwT2W nvjxAxvZGUB7zAp3MCjEGOgba3xhYMUdQDaK9G4/y46eIOsfPn/urCh442IABd82bSm6 +ZTZ+8UAxvOGXRTpXaWBJB+M1c7amrqgWtS9RI8Tr0mPPAtcLE5/xMisEZNLCBkMHvr7 POBr6FyqMz7nuno7LPfc0xTSnjMufhKlBrvSM3FdofzLdQsC8cG4JUhGXqyzMYx/2c9P +6lp2jYcRDFi9M/OAc564vKiXfxzIjtSsF2ygB97DrIA7i/Wlz29J4OoQLkaCp7rAbmv 0gHA== X-Gm-Message-State: AOAM531CK36yOx7DbsVrV+Yl0jZqeLhTewCFFsgOWhyMGgIfwHRN+cf+ PRInE6KWLHI9BEaUHq/HaKfC6zxCO8k= X-Google-Smtp-Source: ABdhPJzig4XeMzNQ+Mp/Sm/XBy0GemyWfyvZfATBkGkTrX2YaeiUZuikvylfGxI/aMLqD4RWGXOfWw== X-Received: by 2002:a05:600c:2f08:: with SMTP id r8mr4054004wmn.95.1617109873955; Tue, 30 Mar 2021 06:11:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 7sm3494919wmk.8.2021.03.30.06.11.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:13 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:52 +0000 Subject: [PATCH v5 09/21] unpack-trees: ensure full index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The next change will translate full indexes into sparse indexes at write time. The existing logic provides a way for every sparse index to be expanded to a full index at read time. However, there are cases where an index is written and then continues to be used in-memory to perform further updates. unpack_trees() is frequently called after such a write. In particular, commands like 'git reset' do this double-update of the index. Ensure that we have a full index when entering unpack_trees(), but only when command_requires_full_index is true. This is always true at the moment, but we will later relax that after unpack_trees() is updated to handle sparse directory entries. Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/unpack-trees.c b/unpack-trees.c index f5f668f532d8..4dd99219073a 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1567,6 +1567,7 @@ static int verify_absent(const struct cache_entry *, */ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options *o) { + struct repository *repo = the_repository; int i, ret; static struct cache_entry *dfc; struct pattern_list pl; @@ -1578,6 +1579,12 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options trace_performance_enter(); trace2_region_enter("unpack_trees", "unpack_trees", the_repository); + prepare_repo_settings(repo); + if (repo->settings.command_requires_full_index) { + ensure_full_index(o->src_index); + ensure_full_index(o->dst_index); + } + if (!core_apply_sparse_checkout || !o->update) o->skip_sparse_checkout = 1; if (!o->skip_sparse_checkout && !o->pl) { From patchwork Tue Mar 30 13:10:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0C0DC433F1 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D0DC619C8 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232272AbhC3NLu (ORCPT ); Tue, 30 Mar 2021 09:11:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232149AbhC3NLQ (ORCPT ); Tue, 30 Mar 2021 09:11:16 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00252C061762 for ; Tue, 30 Mar 2021 06:11:15 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id j9so14476998wrx.12 for ; Tue, 30 Mar 2021 06:11:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=AkJvT4gY7lKLChaTErd6DbcmGyVraiVliS0HbBhNfCE=; b=UbprqlNdSt0S4FKnFsL0jH6vnCFiI5MtJjoRbq27+yBk/CC3bQzPCAdgbzWn1rfu+g 8puvKHIk9Kbk9GKLWzJ3EensbaqPcTGvLx/hjZMmsa18otA00taeZ8sD3MnoiMMhHw9O 230ys6RBdKXY4cZTjztYeFFW2l1P2iQT0nUKXbsSs/MUtdbwePiqRAsaxsYcBJxJX4Jj GZKSdb6CsAqB5RMG+J6tmORwpPnKe8S7KHFSw/eW35NSbonKEAfuVpYJ3FQ0/fAiEkmB OiQygcQqKyPqjhysntgE22Fs2Gl1WE5GsEMPmqKB6GjyGNT4g5hv7s9ZcC04LfwteEti 9aKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=AkJvT4gY7lKLChaTErd6DbcmGyVraiVliS0HbBhNfCE=; b=ClYjuct8W4KolGzuLZPJ0eUYBfaCSGeNjSr2I1vjWLuzq46ljC9pmdLsJxIIkJII7V sfiwmJef9b8K9JyKHo/TqQitHI1xswkyb22/CGAsFSDgsPRMlB1U/BCuTKDwlukPGrV2 Cig5r+hT3n2/jGvkC0kcZmO2EhHE5nUmPZPos6TBMKcqEk5zr6KaH+P6N3u4aiiXQ9Jm ECnKYdieJSosMt0+QNCBgFyk5bn9UOoZZ+jJUoj8HV5hXSOstfiAbAwRxUO7hCFzkEZ/ p46KIbnsB2nzGY9B2lG2Vc5gHpaeu2q3yZKRMvT2gOZ2XcC2GU/EIFUHxzecXPh9DVaz Id/A== X-Gm-Message-State: AOAM531o4VVtX3K+KOQvPIZtfsNAYXZ49GjI4yJWWMdBpeTlkNgQbPrC VSuvodddTRSul/oL8IsuyShYFz0Lfkw= X-Google-Smtp-Source: ABdhPJxf46Yapf3hk2mUJQZDkUyQx9qHXk0RUqVnrGsGiW7s+hkVVy78A/X0+WBBUSgchblWj2t2KQ== X-Received: by 2002:a5d:5270:: with SMTP id l16mr28941841wrc.317.1617109874778; Tue, 30 Mar 2021 06:11:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e17sm38818960wra.65.2021.03.30.06.11.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:14 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:53 +0000 Subject: [PATCH v5 10/21] sparse-checkout: hold pattern list in index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we modify the sparse-checkout definition, we perform index operations on a pattern_list that only exists in-memory. This allows easy backing out in case the index update fails. However, if the index write itself cares about the sparse-checkout pattern set, we need access to that in-memory copy. Place a pointer to a 'struct pattern_list' in the index so we can access this on-demand. This will be used in the next change which uses the sparse-checkout definition to filter out directories that are outside the sparse cone. Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 17 ++++++++++------- cache.h | 2 ++ 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 2306a9ad98e0..e00b82af727b 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -110,6 +110,8 @@ static int update_working_directory(struct pattern_list *pl) if (is_index_unborn(r->index)) return UPDATE_SPARSITY_SUCCESS; + r->index->sparse_checkout_patterns = pl; + memset(&o, 0, sizeof(o)); o.verbose_update = isatty(2); o.update = 1; @@ -138,6 +140,7 @@ static int update_working_directory(struct pattern_list *pl) else rollback_lock_file(&lock_file); + r->index->sparse_checkout_patterns = NULL; return result; } @@ -517,19 +520,18 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m) { int result; int changed_config = 0; - struct pattern_list pl; - memset(&pl, 0, sizeof(pl)); + struct pattern_list *pl = xcalloc(1, sizeof(*pl)); switch (m) { case ADD: if (core_sparse_checkout_cone) - add_patterns_cone_mode(argc, argv, &pl); + add_patterns_cone_mode(argc, argv, pl); else - add_patterns_literal(argc, argv, &pl); + add_patterns_literal(argc, argv, pl); break; case REPLACE: - add_patterns_from_input(&pl, argc, argv); + add_patterns_from_input(pl, argc, argv); break; } @@ -539,12 +541,13 @@ static int modify_pattern_list(int argc, const char **argv, enum modify_type m) changed_config = 1; } - result = write_patterns_and_update(&pl); + result = write_patterns_and_update(pl); if (result && changed_config) set_config(MODE_NO_PATTERNS); - clear_pattern_list(&pl); + clear_pattern_list(pl); + free(pl); return result; } diff --git a/cache.h b/cache.h index 136dd496c95d..8c4464420d0a 100644 --- a/cache.h +++ b/cache.h @@ -307,6 +307,7 @@ static inline unsigned int canon_mode(unsigned int mode) struct split_index; struct untracked_cache; struct progress; +struct pattern_list; struct index_state { struct cache_entry **cache; @@ -338,6 +339,7 @@ struct index_state { struct mem_pool *ce_mem_pool; struct progress *progress; struct repository *repo; + struct pattern_list *sparse_checkout_patterns; }; /* Name hashing */ From patchwork Tue Mar 30 13:10:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B7A6C433F4 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 68598619D4 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232270AbhC3NLt (ORCPT ); Tue, 30 Mar 2021 09:11:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56698 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232164AbhC3NLR (ORCPT ); Tue, 30 Mar 2021 09:11:17 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B818EC061574 for ; Tue, 30 Mar 2021 06:11:16 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id c8so16173972wrq.11 for ; Tue, 30 Mar 2021 06:11:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/OlR0ax3xndSWGKKDFthuKiuhYT+sjq9OOppgrZbaSQ=; b=IFsklqqtyKWhVgjvdGbyewfHXtO7SJ0ozoMkFbe2P5t8s7eR3T9rz2y64gE/KeeP8E 4c5MMiudMZWcX2jUFSSyeZT4OttNkklJxrdB6q9BHcM4AA/MrtT/3RVPTnGou+5SAB0S MBRRYheIhDfWDpO9iV+QX6dvtvvNrMGsaiRmrutM3kwhj6bYTXE7oEjwrHZY0d/X3vSK kscIRcS41GBHkyfQ89dQEy/O5pIvkjY8RcsxwUglzoxaV48xs4fIHiLC6I37Av/yX0K1 2g9QROUZ30zgZDL2FihyB3QRtvic83ek3KSOrHW2l35UeJhFx2p5BYWvvFtz5ATmxBVz fl1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/OlR0ax3xndSWGKKDFthuKiuhYT+sjq9OOppgrZbaSQ=; b=h13zhcZLrH79jrVMrLSPEXQeLcdGTKJiL+CAg61Ge+HBOuB0K1Vf9yXGZmrKmgOIaF jIDkty+MxcypkkjA+N6ILbIz5cRQsPgOG5q2BhspREpizHSMK4WanCU5Tqlsz2U6qziy MtRuA+AczxQNz576U2EYCwT5SOw+CF6TeX7J6w8s0COro3yOgM6Dplm4medeAYlxM3VV DnAxXUTD2EHoVvSEv21G2Cx1ZrpGCsBLpgdhlkIQRG+Ax5+Xi3atBUvgL8m6BIO/aQZj CuccM2WIBrKhhKNWccLmessOTP4co0Wc4i8fVkEBvfiilEFmwsvG4OWqsuOBjsu2nPzG 9fWw== X-Gm-Message-State: AOAM533LSmBnCm6JF/fmKMo+qwwPQx94Z0+33gbqr5sVsjqM4NC2y6Kk SLySzO+ZFTH6nsCVoYw7UBbg3seH4GQ= X-Google-Smtp-Source: ABdhPJxOfY0EsTR8vfowD2HpTmMxqbHX5l/mTI21XhnY87XKwxvs1ZBu63ZdwqIiyVa+85NCt08ZTA== X-Received: by 2002:a5d:63c8:: with SMTP id c8mr33708983wrw.299.1617109875569; Tue, 30 Mar 2021 06:11:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 81sm3682171wmc.11.2021.03.30.06.11.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:54 +0000 Subject: [PATCH v5 11/21] sparse-index: add 'sdir' index extension Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The index format does not currently allow for sparse directory entries. This violates some expectations that older versions of Git or third-party tools might not understand. We need an indicator inside the index file to warn these tools to not interact with a sparse index unless they are aware of sparse directory entries. Add a new _required_ index extension, 'sdir', that indicates that the index may contain sparse directory entries. This allows us to continue to use the differences in index formats 2, 3, and 4 before we create a new index version 5 in a later change. Signed-off-by: Derrick Stolee --- Documentation/technical/index-format.txt | 12 ++++++++++++ read-cache.c | 9 +++++++++ 2 files changed, 21 insertions(+) diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt index 3b74c05647db..65da0daaa563 100644 --- a/Documentation/technical/index-format.txt +++ b/Documentation/technical/index-format.txt @@ -392,3 +392,15 @@ The remaining data of each directory block is grouped by type: in this block of entries. - 32-bit count of cache entries in this block + +== Sparse Directory Entries + + When using sparse-checkout in cone mode, some entire directories within + the index can be summarized by pointing to a tree object instead of the + entire expanded list of paths within that tree. An index containing such + entries is a "sparse index". Index format versions 4 and less were not + implemented with such entries in mind. Thus, for these versions, an + index containing sparse directory entries will include this extension + with signature { 's', 'd', 'i', 'r' }. Like the split-index extension, + tools should avoid interacting with a sparse index unless they understand + this extension. diff --git a/read-cache.c b/read-cache.c index dd3980c12b53..b8f092d1b7eb 100644 --- a/read-cache.c +++ b/read-cache.c @@ -47,6 +47,7 @@ #define CACHE_EXT_FSMONITOR 0x46534D4E /* "FSMN" */ #define CACHE_EXT_ENDOFINDEXENTRIES 0x454F4945 /* "EOIE" */ #define CACHE_EXT_INDEXENTRYOFFSETTABLE 0x49454F54 /* "IEOT" */ +#define CACHE_EXT_SPARSE_DIRECTORIES 0x73646972 /* "sdir" */ /* changes that can be kept in $GIT_DIR/index (basically all extensions) */ #define EXTMASK (RESOLVE_UNDO_CHANGED | CACHE_TREE_CHANGED | \ @@ -1763,6 +1764,10 @@ static int read_index_extension(struct index_state *istate, case CACHE_EXT_INDEXENTRYOFFSETTABLE: /* already handled in do_read_index() */ break; + case CACHE_EXT_SPARSE_DIRECTORIES: + /* no content, only an indicator */ + istate->sparse_index = 1; + break; default: if (*ext < 'A' || 'Z' < *ext) return error(_("index uses %.4s extension, which we do not understand"), @@ -3020,6 +3025,10 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, if (err) return -1; } + if (istate->sparse_index) { + if (write_index_ext_header(&c, &eoie_c, newfd, CACHE_EXT_SPARSE_DIRECTORIES, 0) < 0) + return -1; + } /* * CACHE_EXT_ENDOFINDEXENTRIES must be written as the last entry before the SHA1 From patchwork Tue Mar 30 13:10:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B638C43381 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EE6E2619F2 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232056AbhC3NLx (ORCPT ); Tue, 30 Mar 2021 09:11:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56700 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232168AbhC3NLU (ORCPT ); Tue, 30 Mar 2021 09:11:20 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DEDBC061762 for ; Tue, 30 Mar 2021 06:11:17 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id r10-20020a05600c35cab029010c946c95easo8363250wmq.4 for ; Tue, 30 Mar 2021 06:11:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=X+sqD9rF7xMn6Nj1uKyhucikYnDcy5BZgVE3Tz6MYGQ=; b=qW7nXUNUE/Qx33CIqHqIldKvdQ1qVtD6/jZrcKt/WEz03Wyx0TnwGgrztNHLlBcSQn WfvBaIYZDgtHYPyGzkkVHn87lgfi/97Ypc4oax80nmGcQTzQrS4dLVKglxdn86Qv1WZ9 rHNBxCVKaeQ9J3t9z7B8Z3LRrsgjKwIIb5NtU5gwZeyD3nomrVDv3XzcKWMIwvCvmNRx Z9+aEHrtECXAMYYr/BXnmJaSfYhmoBgVp1oQoe76HHm1XEVdtW8gKLWk3X8AWOUSbERU 6T+o73hNimxEu74N10qZ3n8awZvkLbVzKwfaWGI21xFX8/qWI+R0zpZZtwJqbjubPvkq tzcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=X+sqD9rF7xMn6Nj1uKyhucikYnDcy5BZgVE3Tz6MYGQ=; b=dvI2CjfZcuAf6od21+KwEOHyCPKmBdd3vdcHpkxNj6fNwRou0CSWvGm5WGGxg4wTND LhcHd+r4JUl9+JiQoUDrnv7FFJ91ceTzmoViKIc2ffbk48HJn+8nIClmmMMSoypoEyTn aCHIq+t2ILIEkE26SDzr4GrBVNmbSV06WK3jVVUWvXNXspC+oGEnQC20fGELvPuFHRXB tGpZ64MZevkqqN6fru6pb+kG7u2QndyLwumkuQs3pBdd1HKNVHwPMNeKJ2it8q+DfKp3 ARsnu9Q4clXxukIgX8qfAt/6+lmUvPffxUO4X3BTCgk+LQVGjIE7wHtGHuqUobpBfSkt 4qDA== X-Gm-Message-State: AOAM530ur3uMrqaQubLmN6xdBPh3mz4LSiR6IfQ/AW/pyMfGQ9w+3HS1 erGkcdgfc0O+LQNuPt79+dtm6534CbI= X-Google-Smtp-Source: ABdhPJwpHvY+Mn7aI3UTSbr0Q7UY9bfR3xPhwwvO2bmKW/nn853MeX5C49wKcYkHtX2AadncI977kA== X-Received: by 2002:a7b:c214:: with SMTP id x20mr4078796wmi.186.1617109876354; Tue, 30 Mar 2021 06:11:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n7sm29412479wrv.71.2021.03.30.06.11.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:55 +0000 Subject: [PATCH v5 12/21] sparse-index: convert from full to sparse Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If we have a full index, then we can convert it to a sparse index by replacing directories outside of the sparse cone with sparse directory entries. The convert_to_sparse() method does this, when the situation is appropriate. For now, we avoid converting the index to a sparse index if: 1. the index is split. 2. the index is already sparse. 3. sparse-checkout is disabled. 4. sparse-checkout does not use cone mode. Finally, we currently limit the conversion to when the GIT_TEST_SPARSE_INDEX environment variable is enabled. A mode using Git config will be added in a later change. The trickiest thing about this conversion is that we might not be able to mark a directory as a sparse directory just because it is outside the sparse cone. There might be unmerged files within that directory, so we need to look for those. Also, if there is some strange reason why a file is not marked with CE_SKIP_WORKTREE, then we should give up on converting that directory. There is still hope that some of its subdirectories might be able to convert to sparse, so we keep looking deeper. The conversion process is assisted by the cache-tree extension. This is calculated from the full index if it does not already exist. We then abandon the cache-tree as it no longer applies to the newly-sparse index. Thus, this cache-tree will be recalculated in every sparse-full-sparse round-trip until we integrate the cache-tree extension with the sparse index. Some Git commands use the index after writing it. For example, 'git add' will update the index, then write it to disk, then read its entries to report information. To keep the in-memory index in a full state after writing, we re-expand it to a full one after the write. This is wasteful for commands that only write the index and do not read from it again, but that is only the case until we make those commands "sparse aware." We can compare the behavior of the sparse-index in t1092-sparse-checkout-compability.sh by using GIT_TEST_SPARSE_INDEX=1 when operating on the 'sparse-index' repo. We can also compare the two sparse repos directly, such as comparing their indexes (when expanded to full in the case of the 'sparse-index' repo). We also verify that the index is actually populated with sparse directory entries. The 'checkout and reset (mixed)' test is marked for failure when comparing a sparse repo to a full repo, but we can compare the two sparse-checkout cases directly to ensure that we are not changing the behavior when using a sparse index. Signed-off-by: Derrick Stolee --- cache-tree.c | 3 + cache.h | 2 + read-cache.c | 26 ++++- sparse-index.c | 139 +++++++++++++++++++++++ sparse-index.h | 1 + t/t1092-sparse-checkout-compatibility.sh | 61 +++++++++- 6 files changed, 228 insertions(+), 4 deletions(-) diff --git a/cache-tree.c b/cache-tree.c index 2fb483d3c083..5f07a39e501e 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -6,6 +6,7 @@ #include "object-store.h" #include "replace-object.h" #include "promisor-remote.h" +#include "sparse-index.h" #ifndef DEBUG_CACHE_TREE #define DEBUG_CACHE_TREE 0 @@ -442,6 +443,8 @@ int cache_tree_update(struct index_state *istate, int flags) if (i) return i; + ensure_full_index(istate); + if (!istate->cache_tree) istate->cache_tree = cache_tree(); diff --git a/cache.h b/cache.h index 8c4464420d0a..74b43aaa2bd1 100644 --- a/cache.h +++ b/cache.h @@ -251,6 +251,8 @@ static inline unsigned int create_ce_mode(unsigned int mode) { if (S_ISLNK(mode)) return S_IFLNK; + if (S_ISSPARSEDIR(mode)) + return S_IFDIR; if (S_ISDIR(mode) || S_ISGITLINK(mode)) return S_IFGITLINK; return S_IFREG | ce_permissions(mode); diff --git a/read-cache.c b/read-cache.c index b8f092d1b7eb..2410e6e0df13 100644 --- a/read-cache.c +++ b/read-cache.c @@ -25,6 +25,7 @@ #include "fsmonitor.h" #include "thread-utils.h" #include "progress.h" +#include "sparse-index.h" /* Mask for the name length in ce_flags in the on-disk index */ @@ -1003,8 +1004,14 @@ int verify_path(const char *path, unsigned mode) c = *path++; if ((c == '.' && !verify_dotfile(path, mode)) || - is_dir_sep(c) || c == '\0') + is_dir_sep(c)) return 0; + /* + * allow terminating directory separators for + * sparse directory entries. + */ + if (c == '\0') + return S_ISDIR(mode); } else if (c == '\\' && protect_ntfs) { if (is_ntfs_dotgit(path)) return 0; @@ -3088,6 +3095,14 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l unsigned flags) { int ret; + int was_full = !istate->sparse_index; + + ret = convert_to_sparse(istate); + + if (ret) { + warning(_("failed to convert to a sparse-index")); + return ret; + } /* * TODO trace2: replace "the_repository" with the actual repo instance @@ -3099,6 +3114,9 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l trace2_region_leave_printf("index", "do_write_index", the_repository, "%s", get_lock_file_path(lock)); + if (was_full) + ensure_full_index(istate); + if (ret) return ret; if (flags & COMMIT_LOCK) @@ -3189,9 +3207,10 @@ static int write_shared_index(struct index_state *istate, struct tempfile **temp) { struct split_index *si = istate->split_index; - int ret; + int ret, was_full = !istate->sparse_index; move_cache_to_base_index(istate); + convert_to_sparse(istate); trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); @@ -3199,6 +3218,9 @@ static int write_shared_index(struct index_state *istate, trace2_region_leave_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); + if (was_full) + ensure_full_index(istate); + if (ret) return ret; ret = adjust_shared_perm(get_tempfile_path(*temp)); diff --git a/sparse-index.c b/sparse-index.c index 7095378a1b28..619ff7c2e217 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -4,6 +4,145 @@ #include "tree.h" #include "pathspec.h" #include "trace2.h" +#include "cache-tree.h" +#include "config.h" +#include "dir.h" +#include "fsmonitor.h" + +static struct cache_entry *construct_sparse_dir_entry( + struct index_state *istate, + const char *sparse_dir, + struct cache_tree *tree) +{ + struct cache_entry *de; + + de = make_cache_entry(istate, S_IFDIR, &tree->oid, sparse_dir, 0, 0); + + de->ce_flags |= CE_SKIP_WORKTREE; + return de; +} + +/* + * Returns the number of entries "inserted" into the index. + */ +static int convert_to_sparse_rec(struct index_state *istate, + int num_converted, + int start, int end, + const char *ct_path, size_t ct_pathlen, + struct cache_tree *ct) +{ + int i, can_convert = 1; + int start_converted = num_converted; + enum pattern_match_result match; + int dtype; + struct strbuf child_path = STRBUF_INIT; + struct pattern_list *pl = istate->sparse_checkout_patterns; + + /* + * Is the current path outside of the sparse cone? + * Then check if the region can be replaced by a sparse + * directory entry (everything is sparse and merged). + */ + match = path_matches_pattern_list(ct_path, ct_pathlen, + NULL, &dtype, pl, istate); + if (match != NOT_MATCHED) + can_convert = 0; + + for (i = start; can_convert && i < end; i++) { + struct cache_entry *ce = istate->cache[i]; + + if (ce_stage(ce) || + !(ce->ce_flags & CE_SKIP_WORKTREE)) + can_convert = 0; + } + + if (can_convert) { + struct cache_entry *se; + se = construct_sparse_dir_entry(istate, ct_path, ct); + + istate->cache[num_converted++] = se; + return 1; + } + + for (i = start; i < end; ) { + int count, span, pos = -1; + const char *base, *slash; + struct cache_entry *ce = istate->cache[i]; + + /* + * Detect if this is a normal entry outside of any subtree + * entry. + */ + base = ce->name + ct_pathlen; + slash = strchr(base, '/'); + + if (slash) + pos = cache_tree_subtree_pos(ct, base, slash - base); + + if (pos < 0) { + istate->cache[num_converted++] = ce; + i++; + continue; + } + + strbuf_setlen(&child_path, 0); + strbuf_add(&child_path, ce->name, slash - ce->name + 1); + + span = ct->down[pos]->cache_tree->entry_count; + count = convert_to_sparse_rec(istate, + num_converted, i, i + span, + child_path.buf, child_path.len, + ct->down[pos]->cache_tree); + num_converted += count; + i += span; + } + + strbuf_release(&child_path); + return num_converted - start_converted; +} + +int convert_to_sparse(struct index_state *istate) +{ + if (istate->split_index || istate->sparse_index || + !core_apply_sparse_checkout || !core_sparse_checkout_cone) + return 0; + + /* + * For now, only create a sparse index with the + * GIT_TEST_SPARSE_INDEX environment variable. We will relax + * this once we have a proper way to opt-in (and later still, + * opt-out). + */ + if (!git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) + return 0; + + if (!istate->sparse_checkout_patterns) { + istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list)); + if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) + return 0; + } + + if (!istate->sparse_checkout_patterns->use_cone_patterns) { + warning(_("attempting to use sparse-index without cone mode")); + return -1; + } + + if (cache_tree_update(istate, 0)) { + warning(_("unable to update cache-tree, staying full")); + return -1; + } + + remove_fsmonitor(istate); + + trace2_region_enter("index", "convert_to_sparse", istate->repo); + istate->cache_nr = convert_to_sparse_rec(istate, + 0, 0, istate->cache_nr, + "", 0, istate->cache_tree); + istate->drop_cache_tree = 1; + istate->sparse_index = 1; + trace2_region_leave("index", "convert_to_sparse", istate->repo); + return 0; +} static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce) { diff --git a/sparse-index.h b/sparse-index.h index 09a20d036c46..64380e121d80 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -3,5 +3,6 @@ struct index_state; void ensure_full_index(struct index_state *istate); +int convert_to_sparse(struct index_state *istate); #endif diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index a1aea141c62c..1e888d195122 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -2,6 +2,11 @@ test_description='compare full workdir to sparse workdir' +# The verify_cache_tree() check is not sparse-aware (yet). +# So, disable the check until that integration is complete. +GIT_TEST_CHECK_CACHE_TREE=0 +GIT_TEST_SPLIT_INDEX=0 + . ./test-lib.sh test_expect_success 'setup' ' @@ -121,7 +126,9 @@ run_on_all () { test_all_match () { run_on_all "$@" && test_cmp full-checkout-out sparse-checkout-out && - test_cmp full-checkout-err sparse-checkout-err + test_cmp full-checkout-out sparse-index-out && + test_cmp full-checkout-err sparse-checkout-err && + test_cmp full-checkout-err sparse-index-err } test_sparse_match () { @@ -130,6 +137,38 @@ test_sparse_match () { test_cmp sparse-checkout-err sparse-index-err } +test_expect_success 'sparse-index contents' ' + init_repos && + + test-tool -C sparse-index read-cache --table >cache && + for dir in folder1 folder2 x + do + TREE=$(git -C sparse-index rev-parse HEAD:$dir) && + grep "040000 tree $TREE $dir/" cache \ + || return 1 + done && + + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set folder1 && + + test-tool -C sparse-index read-cache --table >cache && + for dir in deep folder2 x + do + TREE=$(git -C sparse-index rev-parse HEAD:$dir) && + grep "040000 tree $TREE $dir/" cache \ + || return 1 + done && + + GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep/deeper1 && + + test-tool -C sparse-index read-cache --table >cache && + for dir in deep/deeper2 folder1 folder2 x + do + TREE=$(git -C sparse-index rev-parse HEAD:$dir) && + grep "040000 tree $TREE $dir/" cache \ + || return 1 + done +' + test_expect_success 'expanded in-memory index matches full index' ' init_repos && test_sparse_match test-tool read-cache --expand --table @@ -137,6 +176,7 @@ test_expect_success 'expanded in-memory index matches full index' ' test_expect_success 'status with options' ' init_repos && + test_sparse_match ls && test_all_match git status --porcelain=v2 && test_all_match git status --porcelain=v2 -z -u && test_all_match git status --porcelain=v2 -uno && @@ -273,6 +313,17 @@ test_expect_failure 'checkout and reset (mixed)' ' test_all_match git reset update-folder2 ' +# Ensure that sparse-index behaves identically to +# sparse-checkout with a full index. +test_expect_success 'checkout and reset (mixed) [sparse]' ' + init_repos && + + test_sparse_match git checkout -b reset-test update-deep && + test_sparse_match git reset deepest && + test_sparse_match git reset update-folder1 && + test_sparse_match git reset update-folder2 +' + test_expect_success 'merge' ' init_repos && @@ -309,14 +360,20 @@ test_expect_success 'clean' ' test_all_match git status --porcelain=v2 && test_all_match git clean -f && test_all_match git status --porcelain=v2 && + test_sparse_match ls && + test_sparse_match ls folder1 && test_all_match git clean -xf && test_all_match git status --porcelain=v2 && + test_sparse_match ls && + test_sparse_match ls folder1 && test_all_match git clean -xdf && test_all_match git status --porcelain=v2 && + test_sparse_match ls && + test_sparse_match ls folder1 && - test_path_is_dir sparse-checkout/folder1 + test_sparse_match test_path_is_dir folder1 ' test_done From patchwork Tue Mar 30 13:10:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F80AC433F7 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 782B9619CB for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232274AbhC3NLv (ORCPT ); Tue, 30 Mar 2021 09:11:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56702 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231803AbhC3NLS (ORCPT ); Tue, 30 Mar 2021 09:11:18 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39A9EC061574 for ; Tue, 30 Mar 2021 06:11:18 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id j20-20020a05600c1914b029010f31e15a7fso10290410wmq.1 for ; Tue, 30 Mar 2021 06:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=d57sN1iwbvl34esKzbNWg9awcmZQjE9yfjMflF8q+sQ=; b=Dh5OZV/tps3Cr+D7KHiA3ccp/PfTWNpmrB5YKUdJ4XGXgjlGwDtrG+bytdEclbpSmK ZtWHUSHg68jOQVaa1S2RWuxJdNinJPsgL7D8x6yTRhLn0rpM02peVwlFCg6UvVMdoXlN hiTiuGfO2PKB0j2gk5dDZe9ZQRFgVHOylbYhVAQO8+dnhJ9xWl0d6Q3eo+nj/YI06ICN Vji3WXcgicthn6aLzllDIvwD8BZVxIMsr13bfJ1jERZFvGylPXVIg+5hf0ShsCqXZtCV P0kfxYfQQIYx0s7/8TuoUIDqBFwzx97GYEOr/sK6YF3cIKo3k1wjgW2/ZKf6HK85aU2P SK4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=d57sN1iwbvl34esKzbNWg9awcmZQjE9yfjMflF8q+sQ=; b=uoSotIRaSjXuO9J3oRy5Rmm5/+rp9llEdDxQ1a7SwenEZ8vE1/80JwZi8ycTd6u07H ObNfb2FAVNoOt3SxYFxCHPR3BWhD6HSDSZeQKUjmijLXo4PHWqdMkcBPE3aoezCyhHLp ICWsvm1Au1ie4SrLAuCNV0LxaYgApKkPoGECR6dCLKfp0HklJO+jKh39wxYXOwmplrOs ezIGumDBuIgX5ZLaYvAn0EUgeVdMf2qET3XHrbvF4bEd6qppFhGQ5n/pXYQeuTnAggKz ss1fWkpDMXghFhlqDAQDS37G1NhCUrICQSw1jXa+4A62iw2SKFpJBnp8X5Qc/zoaZRP8 zVlg== X-Gm-Message-State: AOAM530pzzj6MwbljHwe7UQVZt6r6GG0yJkDRkMHhCEV5DZHEYE/659v T3mi/FASkU6S9rejaElRitx2uxfr5ws= X-Google-Smtp-Source: ABdhPJy+719aGD1zR0Sp7pTEWTZhcsOfWkelAEusgnYNUv2iQVP+7f0PLs1RAB2jFyestlxwIXJKSw== X-Received: by 2002:a1c:23c2:: with SMTP id j185mr3991076wmj.54.1617109876996; Tue, 30 Mar 2021 06:11:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w22sm3690197wmi.22.2021.03.30.06.11.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:16 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:56 +0000 Subject: [PATCH v5 13/21] submodule: sparse-index should not collapse links Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A submodule is stored as a "Git link" that actually points to a commit within a submodule. Submodules are populated or not depending on submodule configuration, not sparse-checkout. To ensure that the sparse-index feature integrates correctly with submodules, we should not collapse a directory if there is a Git link within its range. Signed-off-by: Derrick Stolee --- sparse-index.c | 1 + t/t1092-sparse-checkout-compatibility.sh | 17 +++++++++++++++++ 2 files changed, 18 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index 619ff7c2e217..7631f7bd00b7 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -52,6 +52,7 @@ static int convert_to_sparse_rec(struct index_state *istate, struct cache_entry *ce = istate->cache[i]; if (ce_stage(ce) || + S_ISGITLINK(ce->ce_mode) || !(ce->ce_flags & CE_SKIP_WORKTREE)) can_convert = 0; } diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 1e888d195122..cba5f89b1e96 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -376,4 +376,21 @@ test_expect_success 'clean' ' test_sparse_match test_path_is_dir folder1 ' +test_expect_success 'submodule handling' ' + init_repos && + + test_all_match mkdir modules && + test_all_match touch modules/a && + test_all_match git add modules && + test_all_match git commit -m "add modules directory" && + + run_on_all git submodule add "$(pwd)/initial-repo" modules/sub && + test_all_match git commit -m "add submodule" && + + # having a submodule prevents "modules" from collapse + test-tool -C sparse-index read-cache --table >cache && + grep "100644 blob .* modules/a" cache && + grep "160000 commit $(git -C initial-repo rev-parse HEAD) modules/sub" cache +' + test_done From patchwork Tue Mar 30 13:10:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 127E5C43446 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DE49E619ED for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232285AbhC3NLy (ORCPT ); Tue, 30 Mar 2021 09:11:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56712 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232166AbhC3NLU (ORCPT ); Tue, 30 Mar 2021 09:11:20 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2040C061764 for ; Tue, 30 Mar 2021 06:11:18 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id e18so16190801wrt.6 for ; Tue, 30 Mar 2021 06:11:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=harCcaW6krA58Ri1udByTg/8REVVoWQHOitg/KDNLxQ=; b=fTy2SHqSPxGUTE2i8cZLFA5yikd/NQMkfp9GgFHMRUa3Wg6C+66Lu357qA8QXeODGY 6S+E/Dexd4JlC0hrqD3FKGpLfpRMPjW7FBxUnytzgnf4uklZWFCZ1NrBRYfkjZjh/0ak 4o4HgkXqqiYAlD50J4ZAB2Pcrdstn6ZHCHkpjLakjBfhRn20MAs4lQuNRdZIUd2hZP3I 2SEP5MdxVD9Iseken6cf84EHDK8b7CQlSx1+yvxGWMR6idWJ6lTiTTAl0rMCl5n/Ao5x R1grHV1NFCjUW3Ctyt62NGEFDbIBUwPSSk5/TO238mPiKMh6knVO38Jybx4TXhhofv7k 62Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=harCcaW6krA58Ri1udByTg/8REVVoWQHOitg/KDNLxQ=; b=ICFsBnQ0chEr2E0OG/xLwjNfdh2MWN2Me6p7JAANATpRyghVlLg1sipDa9lNazK5x3 y1HDZjJf8B6Olpd0MuvU63EuijajMZMJ7AMrTQFRdnCORsAoXFgcOzaaZtPaUNuzTcC4 lrGLt6XW98Tb9gXQKzGk2Bpzxo8u7o0TQ8HMuR5ZMNa6Vk73aRyGeil/mBNPVY+HU5cW ddZtz+uonAjFzY+mUxneeiVgnzf5zd0XGrfF2EjonKJ1IRsNnaEMCuR43Q37S742UY6S MH9GdxhiRm4pLYweGG4tzLWQJhdkr5rnyprmL08EA/BlEaRbQNPlGxrGqXR3gllZPiHk hgOA== X-Gm-Message-State: AOAM530LlRBYOPLJ7ElXRaninKA6zq1Bx2ruwk61BjfduZXgXMKOamTc G7KUnoz/Bbf4I/JZB0KhDRN7OV63nEQ= X-Google-Smtp-Source: ABdhPJxighXUeRef74PrxDZ3bUcPctapw4bXkNIcmP2CIatD1DyjMOzKy+l0cEwBxP1qZ5k84DkrCw== X-Received: by 2002:adf:f4c2:: with SMTP id h2mr1926803wrp.347.1617109877700; Tue, 30 Mar 2021 06:11:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f2sm3619693wmp.20.2021.03.30.06.11.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:17 -0700 (PDT) Message-Id: <7191b48237dee37c095e7e995913bd37b003dd45.1617109865.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:57 +0000 Subject: [PATCH v5 14/21] unpack-trees: allow sparse directories Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The index_pos_by_traverse_info() currently throws a BUG() when a directory entry exists exactly in the index. We need to consider that it is possible to have a directory in a sparse index as long as that entry is itself marked with the skip-worktree bit. The 'pos' variable is assigned a negative value if an exact match is not found. Since a directory name can be an exact match, it is no longer an error to have a nonnegative 'pos' value. Signed-off-by: Derrick Stolee --- unpack-trees.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index 4dd99219073a..0b888dab2246 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -746,9 +746,13 @@ static int index_pos_by_traverse_info(struct name_entry *names, strbuf_make_traverse_path(&name, info, names->path, names->pathlen); strbuf_addch(&name, '/'); pos = index_name_pos(o->src_index, name.buf, name.len); - if (pos >= 0) - BUG("This is a directory and should not exist in index"); - pos = -pos - 1; + if (pos >= 0) { + if (!o->src_index->sparse_index || + !(o->src_index->cache[pos]->ce_flags & CE_SKIP_WORKTREE)) + BUG("This is a directory and should not exist in index"); + } else { + pos = -pos - 1; + } if (pos >= o->src_index->cache_nr || !starts_with(o->src_index->cache[pos]->name, name.buf) || (pos > 0 && starts_with(o->src_index->cache[pos-1]->name, name.buf))) From patchwork Tue Mar 30 13:10:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED22DC433FB for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CAA72619C7 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232281AbhC3NLx (ORCPT ); Tue, 30 Mar 2021 09:11:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232063AbhC3NLU (ORCPT ); Tue, 30 Mar 2021 09:11:20 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 99940C061765 for ; Tue, 30 Mar 2021 06:11:19 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id o16so16251963wrn.0 for ; Tue, 30 Mar 2021 06:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=HXXyCb3008cEuxUb0eFiCeUe/gRR165/1lETbl5mS4A=; b=dAoYrdzuJv9tphGjjQdH3rhj79cI4b1WzxoA3VhwPoiwc5SvVyE3EEGYmxCN9333tL dxHbOwFcuchDC1O37st1nhvtM5QihFGk/SkbglNg4u4Ipn+OaAuYgQA7nWjZZ1Drjw/i L152b2o9OZBJM/DtxT7W7r6ZxXjI9Pv7YkmlQR/n/exN/tlk0MUAZ5wYSY2MGUeoKOYy 0itJv4Ud0d56foRc943oP+HDPq+XLZvo1SZDP1aRSQ2MZd5t3Pgq7LrUIwZ6pmX460ig cHx41p7owWsRqHlqslJDfE0P0fHCi6RbldJQ35C0wn6u1cRtGljUyRAhaS1USfPqkeEn Rdkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=HXXyCb3008cEuxUb0eFiCeUe/gRR165/1lETbl5mS4A=; b=mvvaisOlqnh3K+VRfPY8KGfQod5HsXZfkBn5Fc37Jdelkmoi0u+vNbd82WKFMDi5Y4 kvwW9U74rKHHdELUqvxTI5NsPlJUHRIKw4QYW7oqc3YXY/uoavJzY5bfTg5IqRaQsTyb iJVI9+/IE+u/t++sBsx+JzRvCuxk6BTMuEEWHWrCKQcArqm/0W1tNQhjEK2gf5+G62fA 8p7Dv+i6TU+3PbzKgiEFDWhQpe1WOwx9IV4HZLapsZd8XpKYQse8Hk+iHp/hLkd2gU/6 hrnK8agZHrGLbJ7JNaOJttvPVg2hh6E2StJ0CK/gEEU5hUh+2naJhxvVXRcjalbOHjIs vUvg== X-Gm-Message-State: AOAM531RU/N2P08/7QYf9T4gczqnrymJmCxn+m36DmCtP/TZJ11LuRdo TRbja/ByqmvV2obLQhEDXwIMOOqugWY= X-Google-Smtp-Source: ABdhPJymsnrA4SNnV6NSDx9eRVCKV20rOOUtTA6RvRQprxHqrKLQU/PBTS5p+Bs1zhNjsQKslorOLQ== X-Received: by 2002:a05:6000:5:: with SMTP id h5mr34748939wrx.97.1617109878367; Tue, 30 Mar 2021 06:11:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g15sm3421388wmq.31.2021.03.30.06.11.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:17 -0700 (PDT) Message-Id: <57be9b4a728b2bb16c841d3bad3c9c876a8839e0.1617109865.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:58 +0000 Subject: [PATCH v5 15/21] sparse-index: check index conversion happens Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Add a test case that uses test_region to ensure that we are truly expanding a sparse index to a full one, then converting back to sparse when writing the index. As we integrate more Git commands with the sparse index, we will convert these commands to check that we do _not_ convert the sparse index to a full index and instead stay sparse the entire time. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index cba5f89b1e96..47f983217852 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -393,4 +393,22 @@ test_expect_success 'submodule handling' ' grep "160000 commit $(git -C initial-repo rev-parse HEAD) modules/sub" cache ' +test_expect_success 'sparse-index is expanded and converted back' ' + init_repos && + + ( + GIT_TEST_SPARSE_INDEX=1 && + export GIT_TEST_SPARSE_INDEX && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" reset --hard && + test_region index convert_to_sparse trace2.txt && + test_region index ensure_full_index trace2.txt && + + rm trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" status -uno && + test_region index ensure_full_index trace2.txt + ) +' + test_done From patchwork Tue Mar 30 13:10:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B17E6C433F8 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A138A619D0 for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232280AbhC3NLw (ORCPT ); Tue, 30 Mar 2021 09:11:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56722 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232038AbhC3NLV (ORCPT ); Tue, 30 Mar 2021 09:11:21 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E7D1C061574 for ; Tue, 30 Mar 2021 06:11:20 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id a132-20020a1c668a0000b029010f141fe7c2so10305631wmc.0 for ; Tue, 30 Mar 2021 06:11:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ik/sbiJcgAItjz8wBWyJm4savRMM9zWovCm7yugKB3U=; b=mBLlzaFyqgXQ81HHHoolXqEv1kHeTIMGr6gfDm2M8xjw3ef2OfgiBvIpJEgRCmGsMh pnAzjL9TZWDDuX8jdrsGgn2fFWDED4IKDOAV6fl3MY0mzU+LUAlCv4sL4cBwXpG8i87c Uje5C63x29kpIrtEimPIHOd+Gi0B9b/ujRH+vDgC2aDt9m7QpY7JDHyXfo7qCQdFTVx2 oI9j/QSACL2V7grUqcFJRWL4v9uZaNu5jRKaBJ8CEZZBpE1+GxcWNdH1RPolB+grY2fB k1J+3ciaJviDuw5bs0OzaNihPBVe1zJMI131kurdyjEShEcp2dZHUqNXSUdsuHiygR5e FnEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ik/sbiJcgAItjz8wBWyJm4savRMM9zWovCm7yugKB3U=; b=cyhvyeufoohZE7t6T4mNt1IX76NZp943yC7CZZ+mYCM/LuxvSCxonkTYWxFTULmzmX LOVHxiqzTToqsP7UQRgAyFd8Mi63OU8ZiRBLX3sCHRMX86WsfS9BY3CtkmlyazSuTm2n gtsA0ajmZfxCo4hL/yzFXg2GxZVl+rRXFXeKbwE/5out1q/5JO5Jabz8xQg9Fk+KZmPy jO6NRaH2s3X2+0YAlqS86WrSNDctKp4HeS91xy1Pnwrb6dKJoaMrwp57jB35OOkd/SDM xMawbos/nDUZ5JG28akPkvrX4Bt0ukzvHXrMODhtkPXrhQNn0B8rRMYuNqmRpYh/2OHN HbGg== X-Gm-Message-State: AOAM532OYABzVK2QcgC4xQ5CaYzoLs/3kwqq1PRQN5c2kB5r38oXWEjV udlg/V/JrEvQVbeqlZrqwW5d2Qil49s= X-Google-Smtp-Source: ABdhPJwfX3KRtrzmRUNFYayMPDClSFij1ckJlxG8br1uzBKzrBBvWcgVgePNo9yFjKEic8RlMN4M5Q== X-Received: by 2002:a05:600c:198c:: with SMTP id t12mr3986129wmq.183.1617109879072; Tue, 30 Mar 2021 06:11:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h13sm3220872wmq.29.2021.03.30.06.11.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:18 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 30 Mar 2021 13:10:59 +0000 Subject: [PATCH v5 16/21] sparse-index: add index.sparse config option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When enabled, this config option signals that index writes should attempt to use sparse-directory entries. Signed-off-by: Derrick Stolee --- Documentation/config/index.txt | 5 +++++ cache.h | 1 + repo-settings.c | 7 +++++++ repository.h | 3 ++- sparse-index.c | 34 +++++++++++++++++++++++++++++----- 5 files changed, 44 insertions(+), 6 deletions(-) diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt index 7cb50b37e98d..75f3a2d10541 100644 --- a/Documentation/config/index.txt +++ b/Documentation/config/index.txt @@ -14,6 +14,11 @@ index.recordOffsetTable:: Defaults to 'true' if index.threads has been explicitly enabled, 'false' otherwise. +index.sparse:: + When enabled, write the index using sparse-directory entries. This + has no effect unless `core.sparseCheckout` and + `core.sparseCheckoutCone` are both enabled. Defaults to 'false'. + index.threads:: Specifies the number of threads to spawn when loading the index. This is meant to reduce index load time on multiprocessor machines. diff --git a/cache.h b/cache.h index 74b43aaa2bd1..8aede373aeb3 100644 --- a/cache.h +++ b/cache.h @@ -1059,6 +1059,7 @@ struct repository_format { int worktree_config; int is_bare; int hash_algo; + int sparse_index; char *work_tree; struct string_list unknown_extensions; struct string_list v1_only_extensions; diff --git a/repo-settings.c b/repo-settings.c index d63569e4041e..0cfe8b787db2 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -85,4 +85,11 @@ void prepare_repo_settings(struct repository *r) * removed. */ r->settings.command_requires_full_index = 1; + + /* + * Initialize this as off. + */ + r->settings.sparse_index = 0; + if (!repo_config_get_bool(r, "index.sparse", &value) && value) + r->settings.sparse_index = 1; } diff --git a/repository.h b/repository.h index e06a23015697..a45f7520fd9e 100644 --- a/repository.h +++ b/repository.h @@ -42,7 +42,8 @@ struct repo_settings { int core_multi_pack_index; - unsigned command_requires_full_index:1; + unsigned command_requires_full_index:1, + sparse_index:1; }; struct repository { diff --git a/sparse-index.c b/sparse-index.c index 7631f7bd00b7..6f4d95d35b1e 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -102,19 +102,43 @@ static int convert_to_sparse_rec(struct index_state *istate, return num_converted - start_converted; } +static int enable_sparse_index(struct repository *repo) +{ + const char *config_path = repo_git_path(repo, "config.worktree"); + + git_config_set_in_file_gently(config_path, + "index.sparse", + "true"); + + prepare_repo_settings(repo); + repo->settings.sparse_index = 1; + return 0; +} + int convert_to_sparse(struct index_state *istate) { if (istate->split_index || istate->sparse_index || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; + if (!istate->repo) + istate->repo = the_repository; + + /* + * The GIT_TEST_SPARSE_INDEX environment variable triggers the + * index.sparse config variable to be on. + */ + if (git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) { + int err = enable_sparse_index(istate->repo); + if (err < 0) + return err; + } + /* - * For now, only create a sparse index with the - * GIT_TEST_SPARSE_INDEX environment variable. We will relax - * this once we have a proper way to opt-in (and later still, - * opt-out). + * Only convert to sparse if index.sparse is set. */ - if (!git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) + prepare_repo_settings(istate->repo); + if (!istate->repo->settings.sparse_index) return 0; if (!istate->sparse_checkout_patterns) { From patchwork Tue Mar 30 13:11:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C50F0C433FC for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B1031619BB for ; Tue, 30 Mar 2021 13:12:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232277AbhC3NLv (ORCPT ); Tue, 30 Mar 2021 09:11:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232169AbhC3NLV (ORCPT ); Tue, 30 Mar 2021 09:11:21 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1AAA0C061762 for ; Tue, 30 Mar 2021 06:11:21 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id m20-20020a7bcb940000b029010cab7e5a9fso10282553wmi.3 for ; Tue, 30 Mar 2021 06:11:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3Fbxq3Je8mAhdAFjVgg4DCBTU+/byGfGpXkZoGbrpDk=; b=R9shOWDEak0us4lAryCZWTlNTBnvEm+tr8Zes0/1el9YpFYAYBiikIz0xHkoUQusfG nt1cX6Iwqu+A2V/oZwdhFoSmkmeUQnuvyrfDQp7fpPU515Maa3CHA8YG7FhguGNdi2nn WkJOAOHGE3EhaNpk1x/G7oJvDAI8DB/IgsOJb5bdQG1LRaMp/CVHOciWTpUI0gmcEfLZ GEvJCiO3C8Ip878XYSw7/O1QoIYoX9ha2x1w3zAKl8dJwPlb47r3QgvMw44la8cJqUuA Isih2pCgUGrFmKU9sKTPx6UrKlekO0wNLJZHKmaack6D6oFcCu5L6qEJ8j3PirztoPyw RSSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3Fbxq3Je8mAhdAFjVgg4DCBTU+/byGfGpXkZoGbrpDk=; b=N6tMA2HXmQe575Z5tmXywYaSs/dziLeSkzEUm/QZfLKRHWYo15CSJy4JGpXPR7HK4D xhSkOOqp5tfsBqiNy2uJNC9rpjsrBEPI8eIE9dhkmqxlIdiZJE8E+CcnenJSdG1z+LbY qEWifv7hwQ8NIyTXF9XT34ibzQurTymgbYge0Gt2rE/7D3lWHdss0PwmLX0V5fe5YSGy +xwufIXqCB56bT83o/7VkORSrKR1RZ3heRPS0B1Zv4KB8qxySSnFzLRGQRJWXPK4DK12 eCrRGqBPVV1xeIung7I9wZ9CKY7bdOgZ9bY9AlcISKoIpXbmeo66kCYEADdZpNoENeJP i/sA== X-Gm-Message-State: AOAM530QOSB4RF2R6amqgBnpLSshCwISmbJC/bqQkem8Ma72PzWXz1A7 aMRvNyU6IYocm417LjFFBPeYFzwbM0Y= X-Google-Smtp-Source: ABdhPJziyzYGcRWByufjfz1WamRHKAaXQabbFcK73HOsIHDpD2j/PG8xjnRTuY3+fauzE23QdLHyuQ== X-Received: by 2002:a05:600c:214d:: with SMTP id v13mr4087847wml.7.1617109879768; Tue, 30 Mar 2021 06:11:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x25sm5116118wmj.14.2021.03.30.06.11.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:19 -0700 (PDT) Message-Id: <75fe9b0f57da2cfc152a0e48f7573d381b4c44d2.1617109865.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:11:00 +0000 Subject: [PATCH v5 17/21] sparse-checkout: toggle sparse index from builtin Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse index extension is used to signal that index writes should be in sparse mode. This was only updated using GIT_TEST_SPARSE_INDEX=1. Add a '--[no-]sparse-index' option to 'git sparse-checkout init' that specifies if the sparse index should be used. It also updates the index to use the correct format, either way. Add a warning in the documentation that the use of a repository extension might reduce compatibility with third-party tools. 'git sparse-checkout init' already sets extension.worktreeConfig, which places most sparse-checkout users outside of the scope of most third-party tools. Update t1092-sparse-checkout-compatibility.sh to use this CLI instead of GIT_TEST_SPARSE_INDEX=1. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 14 +++++++ builtin/sparse-checkout.c | 17 ++++++++- sparse-index.c | 33 +++++++++++------ sparse-index.h | 3 ++ t/t1092-sparse-checkout-compatibility.sh | 47 +++++++++++++----------- 5 files changed, 80 insertions(+), 34 deletions(-) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index a0eeaeb02ee3..fdcf43f87cb3 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -45,6 +45,20 @@ To avoid interfering with other worktrees, it first enables the When `--cone` is provided, the `core.sparseCheckoutCone` setting is also set, allowing for better performance with a limited set of patterns (see 'CONE PATTERN SET' below). ++ +Use the `--[no-]sparse-index` option to toggle the use of the sparse +index format. This reduces the size of the index to be more closely +aligned with your sparse-checkout definition. This can have significant +performance advantages for commands such as `git status` or `git add`. +This feature is still experimental. Some commands might be slower with +a sparse index until they are properly integrated with the feature. ++ +**WARNING:** Using a sparse index requires modifying the index in a way +that is not completely understood by external tools. If you have trouble +with this compatibility, then run `git sparse-checkout init --no-sparse-index` +to rewrite your index to not be sparse. Older versions of Git will not +understand the sparse directory entries index extension and may fail to +interact with your repository until it is disabled. 'set':: Write a set of patterns to the sparse-checkout file, as given as diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index e00b82af727b..ca63e2c64e95 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -14,6 +14,7 @@ #include "unpack-trees.h" #include "wt-status.h" #include "quote.h" +#include "sparse-index.h" static const char *empty_base = ""; @@ -283,12 +284,13 @@ static int set_config(enum sparse_checkout_mode mode) } static char const * const builtin_sparse_checkout_init_usage[] = { - N_("git sparse-checkout init [--cone]"), + N_("git sparse-checkout init [--cone] [--[no-]sparse-index]"), NULL }; static struct sparse_checkout_init_opts { int cone_mode; + int sparse_index; } init_opts; static int sparse_checkout_init(int argc, const char **argv) @@ -303,11 +305,15 @@ static int sparse_checkout_init(int argc, const char **argv) static struct option builtin_sparse_checkout_init_options[] = { OPT_BOOL(0, "cone", &init_opts.cone_mode, N_("initialize the sparse-checkout in cone mode")), + OPT_BOOL(0, "sparse-index", &init_opts.sparse_index, + N_("toggle the use of a sparse index")), OPT_END(), }; repo_read_index(the_repository); + init_opts.sparse_index = -1; + argc = parse_options(argc, argv, NULL, builtin_sparse_checkout_init_options, builtin_sparse_checkout_init_usage, 0); @@ -326,6 +332,15 @@ static int sparse_checkout_init(int argc, const char **argv) sparse_filename = get_sparse_checkout_filename(); res = add_patterns_from_file_to_list(sparse_filename, "", 0, &pl, NULL); + if (init_opts.sparse_index >= 0) { + if (set_sparse_index_config(the_repository, init_opts.sparse_index) < 0) + die(_("failed to modify sparse-index config")); + + /* force an index rewrite */ + repo_read_index(the_repository); + the_repository->index->updated_workdir = 1; + } + /* If we already have a sparse-checkout file, use it. */ if (res >= 0) { free(sparse_filename); diff --git a/sparse-index.c b/sparse-index.c index 6f4d95d35b1e..4c73772c6d6c 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -102,21 +102,32 @@ static int convert_to_sparse_rec(struct index_state *istate, return num_converted - start_converted; } -static int enable_sparse_index(struct repository *repo) +static int set_index_sparse_config(struct repository *repo, int enable) { - const char *config_path = repo_git_path(repo, "config.worktree"); - - git_config_set_in_file_gently(config_path, - "index.sparse", - "true"); + int res; + char *config_path = repo_git_path(repo, "config.worktree"); + res = git_config_set_in_file_gently(config_path, + "index.sparse", + enable ? "true" : NULL); + free(config_path); prepare_repo_settings(repo); repo->settings.sparse_index = 1; - return 0; + return res; +} + +int set_sparse_index_config(struct repository *repo, int enable) +{ + int res = set_index_sparse_config(repo, enable); + + prepare_repo_settings(repo); + repo->settings.sparse_index = enable; + return res; } int convert_to_sparse(struct index_state *istate) { + int test_env; if (istate->split_index || istate->sparse_index || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; @@ -128,11 +139,9 @@ int convert_to_sparse(struct index_state *istate) * The GIT_TEST_SPARSE_INDEX environment variable triggers the * index.sparse config variable to be on. */ - if (git_env_bool("GIT_TEST_SPARSE_INDEX", 0)) { - int err = enable_sparse_index(istate->repo); - if (err < 0) - return err; - } + test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1); + if (test_env >= 0) + set_sparse_index_config(istate->repo, test_env); /* * Only convert to sparse if index.sparse is set. diff --git a/sparse-index.h b/sparse-index.h index 64380e121d80..39dcc859735e 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -5,4 +5,7 @@ struct index_state; void ensure_full_index(struct index_state *istate); int convert_to_sparse(struct index_state *istate); +struct repository; +int set_sparse_index_config(struct repository *repo, int enable); + #endif diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 47f983217852..472c5337de1b 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -6,6 +6,7 @@ test_description='compare full workdir to sparse workdir' # So, disable the check until that integration is complete. GIT_TEST_CHECK_CACHE_TREE=0 GIT_TEST_SPLIT_INDEX=0 +GIT_TEST_SPARSE_INDEX= . ./test-lib.sh @@ -100,25 +101,26 @@ init_repos () { # initialize sparse-checkout definitions git -C sparse-checkout sparse-checkout init --cone && git -C sparse-checkout sparse-checkout set deep && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout init --cone && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep + git -C sparse-index sparse-checkout init --cone --sparse-index && + test_cmp_config -C sparse-index true index.sparse && + git -C sparse-index sparse-checkout set deep } run_on_sparse () { ( cd sparse-checkout && - GIT_TEST_SPARSE_INDEX=0 "$@" >../sparse-checkout-out 2>../sparse-checkout-err + "$@" >../sparse-checkout-out 2>../sparse-checkout-err ) && ( cd sparse-index && - GIT_TEST_SPARSE_INDEX=1 "$@" >../sparse-index-out 2>../sparse-index-err + "$@" >../sparse-index-out 2>../sparse-index-err ) } run_on_all () { ( cd full-checkout && - GIT_TEST_SPARSE_INDEX=0 "$@" >../full-checkout-out 2>../full-checkout-err + "$@" >../full-checkout-out 2>../full-checkout-err ) && run_on_sparse "$@" } @@ -148,7 +150,7 @@ test_expect_success 'sparse-index contents' ' || return 1 done && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set folder1 && + git -C sparse-index sparse-checkout set folder1 && test-tool -C sparse-index read-cache --table >cache && for dir in deep folder2 x @@ -158,7 +160,7 @@ test_expect_success 'sparse-index contents' ' || return 1 done && - GIT_TEST_SPARSE_INDEX=1 git -C sparse-index sparse-checkout set deep/deeper1 && + git -C sparse-index sparse-checkout set deep/deeper1 && test-tool -C sparse-index read-cache --table >cache && for dir in deep/deeper2 folder1 folder2 x @@ -166,7 +168,14 @@ test_expect_success 'sparse-index contents' ' TREE=$(git -C sparse-index rev-parse HEAD:$dir) && grep "040000 tree $TREE $dir/" cache \ || return 1 - done + done && + + # Disabling the sparse-index removes tree entries with full ones + git -C sparse-index sparse-checkout init --no-sparse-index && + + test-tool -C sparse-index read-cache --table >cache && + ! grep "040000 tree" cache && + test_sparse_match test-tool read-cache --table ' test_expect_success 'expanded in-memory index matches full index' ' @@ -396,19 +405,15 @@ test_expect_success 'submodule handling' ' test_expect_success 'sparse-index is expanded and converted back' ' init_repos && - ( - GIT_TEST_SPARSE_INDEX=1 && - export GIT_TEST_SPARSE_INDEX && - GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" reset --hard && - test_region index convert_to_sparse trace2.txt && - test_region index ensure_full_index trace2.txt && - - rm trace2.txt && - GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index -c core.fsmonitor="" status -uno && - test_region index ensure_full_index trace2.txt - ) + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" reset --hard && + test_region index convert_to_sparse trace2.txt && + test_region index ensure_full_index trace2.txt && + + rm trace2.txt && + GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ + git -C sparse-index -c core.fsmonitor="" status -uno && + test_region index ensure_full_index trace2.txt ' test_done From patchwork Tue Mar 30 13:11:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1DDCBC43445 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0A4E1619F4 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232187AbhC3NLz (ORCPT ); Tue, 30 Mar 2021 09:11:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56726 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232170AbhC3NLW (ORCPT ); Tue, 30 Mar 2021 09:11:22 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9C7F5C061574 for ; Tue, 30 Mar 2021 06:11:21 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id v11so16161920wro.7 for ; Tue, 30 Mar 2021 06:11:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NjgXkWn7l+CyODn0S0i+uIkYbiIeO7IGgnPTtD2hg6A=; b=Bmjn3jGOqaMIqp1nhUkwIX3d6BhaaLncttynQumjsuUJEP/U8dCDpFGuYN10hF6Zkl d206XUmHq0ZVp3rIwouvgXdyhgwngAElPgR5aJSRS8w+/kLRLbzd9h/sCFxIim0X6lZu qguMaelJ/FACKBHlCMRLHdlgGi/C0etZaw3MQgfURLJNGWpsIKA7DvMcV0PaFTj6tNA8 qiGfx9u2oAqQZ2rxXSzBwm5xULe1geZAt0qkxIvknqOeEoOx9tzZBidIEEphzJa5n6Cb 7IrXhN+w7Y4/eQzxq8VNQhnGBV0TjuknOVr/ZlZ/rj9RhHc92zsdUqxDuagQSlJFWIWi LEyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NjgXkWn7l+CyODn0S0i+uIkYbiIeO7IGgnPTtD2hg6A=; b=Kq0ELyxbppNm3pk8g5tUvyojL732/NdAGJMcczJ1Wy3SndWbZtE5Kx3Rxss0iwVVV2 dg3fxnMqoBh/PvffexttCrdg3Tnqv53DDra4B/8vP8ZIsqoVNIxn8mOCgvogmdEu22JI KkU6I+qt6Z1RK6BS1/27kniC+uyPB8iiFqQZ5Ptez7nbOKtBSR3IzUXgPXknmPrl9a3g GVkl0bEixVOOi4zNyJGufu7mK79kQ0SD+y9UesC1UbRs0UFaJqvuQoI3e4ola59R3NRd i4KWRR2EQlSsPZBhg6JjosTy1jDS6oHbKTjjJk+H3GAOyOZhAZQGfL2f9z/Plm8N52kG HufA== X-Gm-Message-State: AOAM532j9zsbbSVHJzgl/oyX2qA+nb7bga5gsiCCdfDIn1GxPDkn56pd N6edPI2EnTBz4JAQbdETIORSh4orMKQ= X-Google-Smtp-Source: ABdhPJzYjkhDfbgux0Wy2wg/4vPsitOz84HSrAQJY4GMLIIfx+IbSgQudli/OB1fWyIIVeImQm3xmA== X-Received: by 2002:a05:6000:18d2:: with SMTP id w18mr34147286wrq.88.1617109880459; Tue, 30 Mar 2021 06:11:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m3sm3884205wme.40.2021.03.30.06.11.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:20 -0700 (PDT) Message-Id: <7f55a232e6479b62efc7b97d04cdf3ad11a0623d.1617109865.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:11:01 +0000 Subject: [PATCH v5 18/21] sparse-checkout: disable sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee We use 'git sparse-checkout init --cone --sparse-index' to toggle the sparse-index feature. It makes sense to also disable it when running 'git sparse-checkout disable'. This is particularly important because it removes the extensions.sparseIndex config option, allowing other tools to use this Git repository again. This does mean that 'git sparse-checkout init' will not re-enable the sparse-index feature, even if it was previously enabled. While testing this feature, I noticed that the sparse-index was not being written on the first run, but by a second. This was caught by the call to 'test-tool read-cache --table'. This requires adjusting some assignments to core_apply_sparse_checkout and pl.use_cone_patterns in the sparse_checkout_init() logic. Signed-off-by: Derrick Stolee --- builtin/sparse-checkout.c | 10 +++++++++- t/t1091-sparse-checkout-builtin.sh | 13 +++++++++++++ 2 files changed, 22 insertions(+), 1 deletion(-) diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index ca63e2c64e95..585343fa1972 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -280,6 +280,9 @@ static int set_config(enum sparse_checkout_mode mode) "core.sparseCheckoutCone", mode == MODE_CONE_PATTERNS ? "true" : NULL); + if (mode == MODE_NO_PATTERNS) + set_sparse_index_config(the_repository, 0); + return 0; } @@ -341,10 +344,11 @@ static int sparse_checkout_init(int argc, const char **argv) the_repository->index->updated_workdir = 1; } + core_apply_sparse_checkout = 1; + /* If we already have a sparse-checkout file, use it. */ if (res >= 0) { free(sparse_filename); - core_apply_sparse_checkout = 1; return update_working_directory(NULL); } @@ -366,6 +370,7 @@ static int sparse_checkout_init(int argc, const char **argv) add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0); strbuf_addstr(&pattern, "!/*/"); add_pattern(strbuf_detach(&pattern, NULL), empty_base, 0, &pl, 0); + pl.use_cone_patterns = init_opts.cone_mode; return write_patterns_and_update(&pl); } @@ -632,6 +637,9 @@ static int sparse_checkout_disable(int argc, const char **argv) strbuf_addstr(&match_all, "/*"); add_pattern(strbuf_detach(&match_all, NULL), empty_base, 0, &pl, 0); + prepare_repo_settings(the_repository); + the_repository->settings.sparse_index = 0; + if (update_working_directory(&pl)) die(_("error while refreshing working directory")); diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index fc64e9ed99f4..38fc8340f5c9 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -205,6 +205,19 @@ test_expect_success 'sparse-checkout disable' ' check_files repo a deep folder1 folder2 ' +test_expect_success 'sparse-index enabled and disabled' ' + git -C repo sparse-checkout init --cone --sparse-index && + test_cmp_config -C repo true index.sparse && + test-tool -C repo read-cache --table >cache && + grep " tree " cache && + + git -C repo sparse-checkout disable && + test-tool -C repo read-cache --table >cache && + ! grep " tree " cache && + git -C repo config --list >config && + ! grep index.sparse config +' + test_expect_success 'cone mode: init and set' ' git -C repo sparse-checkout init --cone && git -C repo config --list >config && From patchwork Tue Mar 30 13:11:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CE18C43459 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 28A40619F6 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232296AbhC3NL6 (ORCPT ); Tue, 30 Mar 2021 09:11:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56774 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232136AbhC3NLd (ORCPT ); Tue, 30 Mar 2021 09:11:33 -0400 Received: from mail-ej1-x62f.google.com (mail-ej1-x62f.google.com [IPv6:2a00:1450:4864:20::62f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E442C061574 for ; Tue, 30 Mar 2021 06:11:32 -0700 (PDT) Received: by mail-ej1-x62f.google.com with SMTP id ap14so11613723ejc.0 for ; Tue, 30 Mar 2021 06:11:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XMbMRlWDrV8aVVqgf/9ul8+2o5nutHV7Qcnd8W2n8Vg=; b=m/1irrU56wjoQIc2X11D1GwZpNuBg9BdZBaufdg62uMJcolqeEuuUpmzzneUhN82kP 4psVLT8fPCtia/23J9tRKUQZ9pVxNkZYuYh3u2Fo5nFeMEqFLD/0hc4yLCMR+Jsit22T tfS5rVWTvOgploM+rOAHkN6xna5pc/q8M5tVKSAGXRlZZa91H0MbSFfVVS4zVGJCJZZ1 dPvQR+VT1/PJ1HffB7wEzIeHfCzZmhlLfFKgLf8Q1Vy7JahDnDewjIfRtgB5pAVdsovh mcbipmTjGbRbdfvd/m8Ldq+ZVIggZYIUB+c1+V4FrF42yrn1x8vcHidnIdnJFZcUe1BI hE4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XMbMRlWDrV8aVVqgf/9ul8+2o5nutHV7Qcnd8W2n8Vg=; b=QFRcWblpTsE+15y3G9ULj103wIF6fMz/9hXbJgWxk33s+p/If7a4LFYcwpMbyMqshA Wn9SjZ6w2BbUY0gBJcAxhtfVFJ56hvj3IZ7AMj1iLOcVDAUwtoAJZHI64ziMBxPuBduI mAY+Kjc7pq3SEzjM00im48QgDk18IEx8BrCRUEOjQO4eFdtNhYYtxyd9biZxOxYI/7NM axI6wJx7AfUZ1PUl8TVATONAHdbLrDeXZCO8KmYAS7APcPTOSHXS8xocBQ3n27F2O3uV ufILU4uadlZtTOVSnQ0J1vaxqkbE2y0ddFKC5LZvdiB+ptLQYc1Nc7NFUMamesKtKEO/ t6Ug== X-Gm-Message-State: AOAM531WxmQtd+0z+0DrlwuqJD6esLvZnyJsjGh/7fieloLmICHhvgxH zipeCR4GTfYUSaO6Fzm2kUUimTkefTw= X-Google-Smtp-Source: ABdhPJw4AXbwhMnyt2lMcIs33hXWDMPA40TixaebTjG85ZvPc/NdSu6UJESKhq4R3pMBvygL3zVpPA== X-Received: by 2002:adf:f143:: with SMTP id y3mr26969245wro.198.1617109881153; Tue, 30 Mar 2021 06:11:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s83sm3649712wmf.26.2021.03.30.06.11.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:20 -0700 (PDT) Message-Id: <365901809d9d43f1229ac3e81990853b06b8fb35.1617109865.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:11:02 +0000 Subject: [PATCH v5 19/21] cache-tree: integrate with sparse directory entries Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache-tree extension was previously disabled with sparse indexes. However, the cache-tree is an important performance feature for commands like 'git status' and 'git add'. Integrate it with sparse directory entries. When writing a sparse index, completely clear and recalculate the cache tree. By starting from scratch, the only integration necessary is to check if we hit a sparse directory entry and create a leaf of the cache-tree that has an entry_count of one and no subtrees. Signed-off-by: Derrick Stolee --- cache-tree.c | 18 ++++++++++++++++++ sparse-index.c | 10 +++++++++- 2 files changed, 27 insertions(+), 1 deletion(-) diff --git a/cache-tree.c b/cache-tree.c index 5f07a39e501e..950a9615db8f 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -256,6 +256,24 @@ static int update_one(struct cache_tree *it, *skip_count = 0; + /* + * If the first entry of this region is a sparse directory + * entry corresponding exactly to 'base', then this cache_tree + * struct is a "leaf" in the data structure, pointing to the + * tree OID specified in the entry. + */ + if (entries > 0) { + const struct cache_entry *ce = cache[0]; + + if (S_ISSPARSEDIR(ce->ce_mode) && + ce->ce_namelen == baselen && + !strncmp(ce->name, base, baselen)) { + it->entry_count = 1; + oidcpy(&it->oid, &ce->oid); + return 1; + } + } + if (0 <= it->entry_count && has_object_file(&it->oid)) return it->entry_count; diff --git a/sparse-index.c b/sparse-index.c index 4c73772c6d6c..95ea17174da3 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -172,7 +172,11 @@ int convert_to_sparse(struct index_state *istate) istate->cache_nr = convert_to_sparse_rec(istate, 0, 0, istate->cache_nr, "", 0, istate->cache_tree); - istate->drop_cache_tree = 1; + + /* Clear and recompute the cache-tree */ + cache_tree_free(&istate->cache_tree); + cache_tree_update(istate, 0); + istate->sparse_index = 1; trace2_region_leave("index", "convert_to_sparse", istate->repo); return 0; @@ -273,5 +277,9 @@ void ensure_full_index(struct index_state *istate) strbuf_release(&base); free(full); + /* Clear and recompute the cache-tree */ + cache_tree_free(&istate->cache_tree); + cache_tree_update(istate, 0); + trace2_region_leave("index", "ensure_full_index", istate->repo); } From patchwork Tue Mar 30 13:11:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C6C8C4345A for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1A815619F2 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232288AbhC3NLz (ORCPT ); Tue, 30 Mar 2021 09:11:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232003AbhC3NLX (ORCPT ); Tue, 30 Mar 2021 09:11:23 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1D37AC061574 for ; Tue, 30 Mar 2021 06:11:23 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id b2-20020a7bc2420000b029010be1081172so8390943wmj.1 for ; Tue, 30 Mar 2021 06:11:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cRKY7qS/L3cDx5v3IYkxLha7hUdGoCFJcKbxvG3W5xM=; b=NyBLzCy2e4XQ2KUDg1SzwZR511ADEUlhW98nStixmywzeeWQ5svd3jbdlUvMqhsX2V NQeA1aH7DyFKZyrY1d2mWfKPgBqQ8SVRRyFtGLdiy9ilcrn8Tp7pqSLB09S6xYJXpycA PeqhPjKm/tfJykDMkaky7JsgmAdhm0ySdjoZzVLEVUKK/BbGsgVE8KVC6cLMgfBbur6Z ylT37e8KeKboAW6YLP5OD5H7iiriTnuqp9t9KLF46Po95n+vTUowB8PyOWjSdunEruru cz9wdX/PPbkBEEMQnpNXQmHv+7TAQT3rmkxPMqJXsgN3h9nZJFiQsnSxYzj4M83daBj5 GBOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cRKY7qS/L3cDx5v3IYkxLha7hUdGoCFJcKbxvG3W5xM=; b=mdTHAmzpIpTfsUXMVKsaKgSl4SY7/BRDXH+tENXr+sB68rj9fSOjYoobpZPatKT7Qs NJTxUeC+PYEVH6kj+cmLuEjHRlJbFZC74NOGk61BLMgkt3PhCFirWItOUCUwQYn/y9Ys /3toeZCcuwMC3fcdmDEJYeAjUtS7PrvzgF1uc8+rAQpZNjs9AjHWqF7/52u+TMFWWv9v QVlBOiUDxenMW47wUxcubSqRX+/pzUJ7MIqQSAw7utaKPqJndBbt71VbnKIetFS06dUd 9Xz+wk2v8Y/DzgCCzAYVuDv5oond2uyHyLMEj821p9fGPrFjffxE2ZhwpyY8LsfpZKmX NvVg== X-Gm-Message-State: AOAM530y9UoqNRkVfBMMo5NlWJFr4q7VVOVwda15NZUpGsJ5W4feB6CU /3gS9xXYma/aOVj1/TDie67CZdofFDk= X-Google-Smtp-Source: ABdhPJzweIbqv9Fvlr3SFqJxlJ7I8JNAIjL0CpH2naobai00YtrxITwjrNlNXhtUTQwZYxnyuKQnpA== X-Received: by 2002:a1c:bb0b:: with SMTP id l11mr4178650wmf.150.1617109881862; Tue, 30 Mar 2021 06:11:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c131sm3614305wma.37.2021.03.30.06.11.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:21 -0700 (PDT) Message-Id: <9b068c458898edfc5267c06e37563f66fac148cf.1617109865.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:11:03 +0000 Subject: [PATCH v5 20/21] sparse-index: loose integration with cache_tree_verify() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The cache_tree_verify() method is run when GIT_TEST_CHECK_CACHE_TREE is enabled, which it is by default in the test suite. The logic must be adjusted for the presence of these directory entries. For now, leave the test as a simple check for whether the directory entry is sparse. Do not go any further until needed. This allows us to re-enable GIT_TEST_CHECK_CACHE_TREE in t1092-sparse-checkout-compatibility.sh. Further, p2000-sparse-operations.sh uses the test suite and hence this is enabled for all tests. We need to integrate with it before we run our performance tests with a sparse-index. Signed-off-by: Derrick Stolee --- cache-tree.c | 19 +++++++++++++++++++ t/t1092-sparse-checkout-compatibility.sh | 3 --- 2 files changed, 19 insertions(+), 3 deletions(-) diff --git a/cache-tree.c b/cache-tree.c index 950a9615db8f..11bf1fcae6e1 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -808,6 +808,19 @@ int cache_tree_matches_traversal(struct cache_tree *root, return 0; } +static void verify_one_sparse(struct repository *r, + struct index_state *istate, + struct cache_tree *it, + struct strbuf *path, + int pos) +{ + struct cache_entry *ce = istate->cache[pos]; + + if (!S_ISSPARSEDIR(ce->ce_mode)) + BUG("directory '%s' is present in index, but not sparse", + path->buf); +} + static void verify_one(struct repository *r, struct index_state *istate, struct cache_tree *it, @@ -830,6 +843,12 @@ static void verify_one(struct repository *r, if (path->len) { pos = index_name_pos(istate, path->buf, path->len); + + if (pos >= 0) { + verify_one_sparse(r, istate, it, path, pos); + return; + } + pos = -pos - 1; } else { pos = 0; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 472c5337de1b..12e6c453024f 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -2,9 +2,6 @@ test_description='compare full workdir to sparse workdir' -# The verify_cache_tree() check is not sparse-aware (yet). -# So, disable the check until that integration is complete. -GIT_TEST_CHECK_CACHE_TREE=0 GIT_TEST_SPLIT_INDEX=0 GIT_TEST_SPARSE_INDEX= From patchwork Tue Mar 30 13:11:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12172521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BAACC4345C for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3A245619F7 for ; Tue, 30 Mar 2021 13:12:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232292AbhC3NL5 (ORCPT ); Tue, 30 Mar 2021 09:11:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56738 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232181AbhC3NLY (ORCPT ); Tue, 30 Mar 2021 09:11:24 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D2F20C061574 for ; Tue, 30 Mar 2021 06:11:23 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id z6-20020a1c4c060000b029010f13694ba2so8371488wmf.5 for ; Tue, 30 Mar 2021 06:11:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=W9y5dD+98QXMbGi6i1SxBzC8O8jQQKZVAIqARbUBgiE=; b=aJX8coU/RbTAwvk64xy5MlWufPYYMI3W4XOJ/IE6SyPojLpSK9BW+onb7A6oy1VJCr 3oUfuDPZsrZz2xW44VX8fbRxIHI+a4bamlRFOktCR4mdMLbkOM/92iBkTMVBsl9ERp3l V2Ah2jplytWhN5u9XiI0drfj3Jf3MidVxSX8FCdPOBewEPsiQhk9pq0lQSsdHmp88LbD wQyLDlsYOGhNaYXqyUotXDZt60h4BSO8rBNW8+hf0l6/jKN/IP317uve6o/4QxifiuFy XV+rnnlFz+8gV+ZPc+qwo1JhHPFu1Nl/LNShhG+Bbw6uYIUPyzRsRC2aQ74RwzfYw/SR 8CFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=W9y5dD+98QXMbGi6i1SxBzC8O8jQQKZVAIqARbUBgiE=; b=lLKiSU4bUBzANEybruPOrBQ0QxqQw/tWBzTpbCVRPL/pOfZI7Kh6Vo3raDnl2/Xm38 XWMKr+smEuZLAOp1yCF1qWn1xoCuk3NZidXdb7SHM3hlgzXXJfS1yQRg2a2MCpnHB9Eo Xpftq3ThDJp3+Vrp/qjfQ33WubNcxJjiW11aNszjypCoPPREkOZdA8ZAqqpQehN3to64 LINx0ygBacnpdvR/JTx410Nug18rX16qmuNGOvds6ASifUhVjnMXs3wHQWt3BBpSH7OI P53phGst1CHcRlGqQGKUixFu0WPFGHxjIzpw/vlxXPHRsKecCcjSnoJk1KvQRjcsOqIj S4TQ== X-Gm-Message-State: AOAM533fzIC+nd0QcPufwuNryeFbHJwSZYYyTvr6gIvTgHb9m3WwxaqL Hw7BMvg3waFQQ0aBFzv8Rgttyk/B1Wc= X-Google-Smtp-Source: ABdhPJzMX8WplzqlINGiYjGLCI4/pWc7cJJe9vzFC205J4XFfkHpFtae2pY7UnXSyoNuiXkvI8qDYw== X-Received: by 2002:a7b:c931:: with SMTP id h17mr4111382wml.4.1617109882580; Tue, 30 Mar 2021 06:11:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s13sm2544056wrv.80.2021.03.30.06.11.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Mar 2021 06:11:22 -0700 (PDT) Message-Id: <66602733cc95d9a53594520cd8b28d3338e258ea.1617109865.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 30 Mar 2021 13:11:04 +0000 Subject: [PATCH v5 21/21] p2000: add sparse-index repos Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: newren@gmail.com, gitster@pobox.com, pclouds@gmail.com, jrnieder@gmail.com, Martin =?utf-8?b?w4VncmVu?= , Derrick Stolee , SZEDER =?utf-8?b?R8OhYm9y?= , =?utf-8?b?w4Z2YXIgQXJu?= =?utf-8?b?ZmrDtnLDsA==?= Bjarmason , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee p2000-sparse-operations.sh compares different Git commands in repositories with many files at HEAD but using sparse-checkout to focus on a small portion of those files. Add extra copies of the repository that use the sparse-index format so we can track how that affects the performance of different commands. At this point in time, the sparse-index is 100% overhead from the CPU front, and this is measurable in these tests: Test --------------------------------------------------------------- 2000.2: git status (full-index-v3) 0.59(0.51+0.12) 2000.3: git status (full-index-v4) 0.59(0.52+0.11) 2000.4: git status (sparse-index-v3) 1.40(1.32+0.12) 2000.5: git status (sparse-index-v4) 1.41(1.36+0.08) 2000.6: git add -A (full-index-v3) 2.32(1.97+0.19) 2000.7: git add -A (full-index-v4) 2.17(1.92+0.14) 2000.8: git add -A (sparse-index-v3) 2.31(2.21+0.15) 2000.9: git add -A (sparse-index-v4) 2.30(2.20+0.13) 2000.10: git add . (full-index-v3) 2.39(2.02+0.20) 2000.11: git add . (full-index-v4) 2.20(1.94+0.16) 2000.12: git add . (sparse-index-v3) 2.36(2.27+0.12) 2000.13: git add . (sparse-index-v4) 2.33(2.21+0.16) 2000.14: git commit -a -m A (full-index-v3) 2.47(2.12+0.20) 2000.15: git commit -a -m A (full-index-v4) 2.26(2.00+0.17) 2000.16: git commit -a -m A (sparse-index-v3) 3.01(2.92+0.16) 2000.17: git commit -a -m A (sparse-index-v4) 3.01(2.94+0.15) Note that there is very little difference between the v3 and v4 index formats when the sparse-index is enabled. This is primarily due to the fact that the relative file sizes are the same, and the command time is mostly taken up by parsing tree objects to expand the sparse index into a full one. With the current file layout, the index file sizes are given by this table: | full index | sparse index | +-------------+--------------+ v3 | 108 MiB | 1.6 MiB | v4 | 80 MiB | 1.2 MiB | Future updates will improve the performance of Git commands when the index is sparse. Signed-off-by: Derrick Stolee --- t/perf/p2000-sparse-operations.sh | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh index dddd527b6330..94513c977489 100755 --- a/t/perf/p2000-sparse-operations.sh +++ b/t/perf/p2000-sparse-operations.sh @@ -59,12 +59,29 @@ test_expect_success 'setup repo and indexes' ' git sparse-checkout set $SPARSE_CONE && git config index.version 4 && git update-index --index-version=4 + ) && + git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v3 && + ( + cd sparse-index-v3 && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set $SPARSE_CONE && + git config index.version 3 && + git update-index --index-version=3 + ) && + git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v4 && + ( + cd sparse-index-v4 && + git sparse-checkout init --cone --sparse-index && + git sparse-checkout set $SPARSE_CONE && + git config index.version 4 && + git update-index --index-version=4 ) ' test_perf_on_all () { command="$@" - for repo in full-index-v3 full-index-v4 + for repo in full-index-v3 full-index-v4 \ + sparse-index-v3 sparse-index-v4 do test_perf "$command ($repo)" " (