| Message ID | pull.999.git.1626901619.gitgitgadget@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Sparse Index: Integrate with 'git add' | expand |
Hi, I haven't read the series yet (I'm going to try to read them all today), but wanted to comment on a couple things in your cover letter... On Wed, Jul 21, 2021 at 2:07 PM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote: > > This patch series re-submits the 'git add' integration with sparse-index. > The performance gains are the same as before. > > It is based on ds/commit-and-checkout-with-sparse-index. > > This series was delayed from its initial submission for a couple reasons. > > The first was because it was colliding with some changes in > mt/add-rm-in-sparse-checkout, so now we are far enough along that that > branch is in our history and we can work forwards. > > The other concern was about how 'git add ' should respond when a path > outside of the sparse-checkout cone exists. One recommendation (that I am > failing to find a link to the message, sorry) was to disallow adding files > that would become index entries with SKIP_WORKTREE on. However, as I worked I think the recommendation was: * Permitted: running git add on tracked files with the SKIP_WORKTREE bit *clear* * Disallowed: running git add on tracked files with the SKIP_WORKTREE bit *set* * Disallowed: running git add on untracked files that would become index entries with the SKIP_WORKTREE bit set where the latter two exit with an error message that suggests changing the sparsity specification first. I think this is what has existed for some time, other than Matheus adding some error messages to help the user when their add command doesn't match anything otherwise. > towards that goal I found that change would cause problems for a realistic > scenario: merge conflicts outside of the sparse-checkout cone. ...which wouldn't be a problem because these are tracked files whose SKIP_WORKTREE bit was cleared by the merge machinery (when it marked them as conflicted and wrote them to the working tree). > The first patch of this series adds tests that create merge conflicts > outside of the sparse cone and then presents different ways a user could > resolve the situation. We want all of them to be feasible, and this > includes: > > 1. Reverting the file to a known version in history. > 2. Adding the file with its contents on disk. > 3. Moving the file to a new location in the sparse directory. > > Without maintaining the behavior of adding files outside of the > sparse-checkout cone, we risk confusing users who get into this state. The > only workaround they would have is to modify their sparse-checkout > definition which might lead to expanding much more data than they need to > resolve the conflicts. > > For that reason, I stopped trying to limit 'git add' to be within the cone. > I'm open to suggestions here, but we need an approach that works for > out-of-cone conflicts. I believe my above suggestion works for out-of-cone conflicts. Some important other details to keep in mind in regards to how we make add behave: * We don't want "git add -A [GLOB_OR_DIRECTORY]" to accidentally be treated as a directive to remove files from the repository (and naively noticing that SKIP_WORKTREE files are missing but attempting to operate on them anyway would give this problematic result). * We don't want "git rm [GLOB_OR_DIRECTORY]" to nuke SKIP_WORKTREE files; it should only operate on files that are present. * We need add and rm to be consistent with each other in terms of how SKIP_WORKTREE files and the sparsity cone are treated and more generally about not-SKIP_WORKTREE-despite-not-matching-sparsity-paths: * These files that aren't SKIP_WORKTREE but normally would be are prone to "disappear" at some random later time after they are made to match the index. The disappearing can happen either with an explicit "git sparse-checkout reapply" (which is fine since it was explicit) or as a side-effect of various other commands that run through unpack_trees() since it attempts to heed the sparsity rules. Users tend to get confused by the latter case; they'll understand at some point that it was because the file was outside the sparsity paths, but the randomness in when it's pulled out as a side-effect of other commands can be slightly jarring. So, I'd like to avoid that where we can easily do so, which I think the recommendation above does. (As a side note to these kinds of cases, maybe we want to consider having the sparsification logic in unpack_trees() first check whether paths being removed from the working tree and having their SKIP_WORKTREE bit set not only match the index but also match HEAD? That'd be a bit more expensive to check in the sparsification paths, and I'm not sure they all have the relevant information, but it's an idea...) > The one place I did continue to update is 'git add --refresh ' to match the > behavior added by mt/add-rm-in-sparse-checkout which outputs an error > message. This happens even when the file exists in the working directory, > but that seems appropriate enough. > > Thanks, -Stolee > > Derrick Stolee (5): > t1092: test merge conflicts outside cone > add: allow operating on a sparse-only index > pathspec: stop calling ensure_full_index > t1092: 'git add --refresh' difference with sparse-index > add: ignore outside the sparse-checkout in refresh() > > builtin/add.c | 13 ++++- > pathspec.c | 2 - > t/t1092-sparse-checkout-compatibility.sh | 71 ++++++++++++++++++++---- > 3 files changed, 72 insertions(+), 14 deletions(-) > > > base-commit: 71e301501c88399711a1bf8515d1747e92cfbb9b > Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-999%2Fderrickstolee%2Fsparse-index%2Fadd-v1 > Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-999/derrickstolee/sparse-index/add-v1 > Pull-Request: https://github.com/gitgitgadget/git/pull/999 > -- > gitgitgadget
On Fri, Jul 23, 2021 at 5:51 AM Elijah Newren <newren@gmail.com> wrote: > > Hi, > > I haven't read the series yet (I'm going to try to read them all > today), but wanted to comment on a couple things in your cover > letter... > > On Wed, Jul 21, 2021 at 2:07 PM Derrick Stolee via GitGitGadget > <gitgitgadget@gmail.com> wrote: > > > > This patch series re-submits the 'git add' integration with sparse-index. > > The performance gains are the same as before. > > > > It is based on ds/commit-and-checkout-with-sparse-index. > > > > This series was delayed from its initial submission for a couple reasons. > > > > The first was because it was colliding with some changes in > > mt/add-rm-in-sparse-checkout, so now we are far enough along that that > > branch is in our history and we can work forwards. > > > > The other concern was about how 'git add ' should respond when a path > > outside of the sparse-checkout cone exists. One recommendation (that I am > > failing to find a link to the message, sorry) was to disallow adding files > > that would become index entries with SKIP_WORKTREE on. However, as I worked > > I think the recommendation was: > * Permitted: running git add on tracked files with the SKIP_WORKTREE > bit *clear* > * Disallowed: running git add on tracked files with the > SKIP_WORKTREE bit *set* > * Disallowed: running git add on untracked files that would become > index entries with the SKIP_WORKTREE bit set > > where the latter two exit with an error message that suggests changing > the sparsity specification first. I think this is what has existed > for some time, other than Matheus adding some error messages to help > the user when their add command doesn't match anything otherwise. > > > towards that goal I found that change would cause problems for a realistic > > scenario: merge conflicts outside of the sparse-checkout cone. > > ...which wouldn't be a problem because these are tracked files whose > SKIP_WORKTREE bit was cleared by the merge machinery (when it marked > them as conflicted and wrote them to the working tree). > > > The first patch of this series adds tests that create merge conflicts > > outside of the sparse cone and then presents different ways a user could > > resolve the situation. We want all of them to be feasible, and this > > includes: > > > > 1. Reverting the file to a known version in history. > > 2. Adding the file with its contents on disk. > > 3. Moving the file to a new location in the sparse directory. > > > > Without maintaining the behavior of adding files outside of the > > sparse-checkout cone, we risk confusing users who get into this state. The > > only workaround they would have is to modify their sparse-checkout > > definition which might lead to expanding much more data than they need to > > resolve the conflicts. > > > > For that reason, I stopped trying to limit 'git add' to be within the cone. > > I'm open to suggestions here, but we need an approach that works for > > out-of-cone conflicts. > > I believe my above suggestion works for out-of-cone conflicts. Some > important other details to keep in mind in regards to how we make add > behave: > > * We don't want "git add -A [GLOB_OR_DIRECTORY]" to accidentally be > treated as a directive to remove files from the repository (and > naively noticing that SKIP_WORKTREE files are missing but attempting > to operate on them anyway would give this problematic result). > * We don't want "git rm [GLOB_OR_DIRECTORY]" to nuke SKIP_WORKTREE > files; it should only operate on files that are present. > * We need add and rm to be consistent with each other in terms of how > SKIP_WORKTREE files and the sparsity cone are treated > > and more generally about not-SKIP_WORKTREE-despite-not-matching-sparsity-paths: > > * These files that aren't SKIP_WORKTREE but normally would be are > prone to "disappear" at some random later time after they are made to > match the index. The disappearing can happen either with an explicit > "git sparse-checkout reapply" (which is fine since it was explicit) or > as a side-effect of various other commands that run through > unpack_trees() since it attempts to heed the sparsity rules. Users > tend to get confused by the latter case; they'll understand at some > point that it was because the file was outside the sparsity paths, but > the randomness in when it's pulled out as a side-effect of other > commands can be slightly jarring. So, I'd like to avoid that where we > can easily do so, which I think the recommendation above does. (As a > side note to these kinds of cases, maybe we want to consider having > the sparsification logic in unpack_trees() first check whether paths > being removed from the working tree and having their SKIP_WORKTREE bit > set not only match the index but also match HEAD? That'd be a bit > more expensive to check in the sparsification paths, and I'm not sure > they all have the relevant information, but it's an idea...) > > > The one place I did continue to update is 'git add --refresh ' to match the > > behavior added by mt/add-rm-in-sparse-checkout which outputs an error > > message. This happens even when the file exists in the working directory, > > but that seems appropriate enough. I've read through the patches now. They look pretty good, and the performance improvements are definitely appealing. My comments on patch 1/5 probably reflect the differences in proposed paths between your cover letter and my proposal. My comments on patch 3/5 point out that the current behavior matches neither your description in your cover letter (or my understanding thereof), nor mine in my earlier response. However, it also suggests that the differences between my proposal and the code isn't due to your series but a pre-existing issue, and thus wouldn't necessarily need to hold your series up. My comments on patch 2/5, asking about why you removed a hunk are thus the biggest outstanding item for the series to move forward. It would be nice for us to figure out and agree on the end goal for add/rm/mv behavior, though.