[v2] midx: disable replace objects

Commit Message

Xing Xin April 8, 2024, 5:26 a.m. UTC

From: Xing Xin <[e-mail]>

We observed a series of clone failures arose in a specific set of
repositories after we fully enabled the MIDX bitmap feature within our
Codebase service. These failures were accompanied with error messages
such as:

    Cloning into bare repository 'clone.git'...
    remote: Enumerating objects: 8, done.
    remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)
    Receiving objects: 100% (8/8), done.
    fatal: did not receive expected object ...
    fatal: fetch-pack: invalid index-pack output

Temporarily disabling the MIDX feature eliminated the reported issues.
After some investigation we found that all repositories experiencing
failures contain replace references, which seem to be improperly
acknowledged by the MIDX bitmap generation logic.

A more thorough explanation about the root cause from Taylor Blau says:

Indeed, the pack-bitmap-write machinery does not itself call
disable_replace_refs(). So when it generates a reachability bitmap, it
is doing so with the replace refs in mind. You can see that this is
indeed the cause of the problem by looking at the output of an
instrumented version of Git that indicates what bits are being set
during the bitmap generation phase.

With replace refs (incorrectly) enabled, we get:

    [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]

and doing the same after calling disable_replace_refs(), we instead get:

    [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]

Single pack bitmaps are unaffected by this issue because we generate
them from within pack-objects, which does call disable_replace_refs().

This patch updates the MIDX logic to disable replace objects within the
multi-pack-index builtin, and a test showing a clone (which would fail
with MIDX bitmap) is added to demonstrate the bug.

Helped-by: Taylor Blau <[e-mail]>
Signed-off-by: Xing Xin <[e-mail]>
---
    midx: disable replace objects
    
    cc: Taylor Blau [e-mail]

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1711%2Fblanet%2Fxx%2Fmidx-ignore-replace-objects-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1711/blanet/xx/midx-ignore-replace-objects-v2
Pull-Request: https://github.com/gitgitgadget/git/pull/1711

Range-diff vs v1:

 1:  b1c838965ab ! 1:  1be25b55c5a midx: disable replace objects
     @@ Commit message
          Codebase service. These failures were accompanied with error messages
          such as:
      
     -      fatal: did not receive expected object ...
     -      fatal: fetch-pack: invalid index-pack output
     +        Cloning into bare repository 'clone.git'...
     +        remote: Enumerating objects: 8, done.
     +        remote: Total 8 (delta 0), reused 0 (delta 0), pack-reused 8 (from 1)
     +        Receiving objects: 100% (8/8), done.
     +        fatal: did not receive expected object ...
     +        fatal: fetch-pack: invalid index-pack output
      
          Temporarily disabling the MIDX feature eliminated the reported issues.
          After some investigation we found that all repositories experiencing
          failures contain replace references, which seem to be improperly
     -    acknowledged by the MIDX bitmap generation logic. During cloning or
     -    fetching, git-pack-objects, which may make use of MIDX bitmap to find
     -    objects to pack, would give wrong objects even if we explicitly
     -    specified not to enable replace refs by GIT_NO_REPLACE_OBJECTS=1.
     -    Indeed, this issue appears to have persisted since the introduction of
     -    MIDX.
     +    acknowledged by the MIDX bitmap generation logic.
      
     -    This patch updates the MIDX logic to disable replace objects during
     -    operations, mirroring the handling seen in single pack index scenarios,
     -    i.e. git-index-pack and git-pack-objects. The added test uses
     -    git-rev-list to give a more intuitive check.
     +    A more thorough explanation about the root cause from Taylor Blau says:
      
     +    Indeed, the pack-bitmap-write machinery does not itself call
     +    disable_replace_refs(). So when it generates a reachability bitmap, it
     +    is doing so with the replace refs in mind. You can see that this is
     +    indeed the cause of the problem by looking at the output of an
     +    instrumented version of Git that indicates what bits are being set
     +    during the bitmap generation phase.
     +
     +    With replace refs (incorrectly) enabled, we get:
     +
     +        [2, 4, 6, 8, 13, 3, 6, 7, 3, 4, 6, 8]
     +
     +    and doing the same after calling disable_replace_refs(), we instead get:
     +
     +        [2, 5, 6, 13, 3, 6, 7, 3, 4, 6, 8]
     +
     +    Single pack bitmaps are unaffected by this issue because we generate
     +    them from within pack-objects, which does call disable_replace_refs().
     +
     +    This patch updates the MIDX logic to disable replace objects within the
     +    multi-pack-index builtin, and a test showing a clone (which would fail
     +    with MIDX bitmap) is added to demonstrate the bug.
     +
     +    Helped-by: Taylor Blau <[e-mail]>
          Signed-off-by: Xing Xin <[e-mail]>
      
       ## builtin/multi-pack-index.c ##
     @@ t/t5326-multi-pack-bitmaps.sh: test_expect_success 'tagged commits are selected
      +		cd repo &&
      +
      +		test_commit A &&
     -+		A=$(git rev-parse HEAD) &&
      +		test_commit B &&
     -+		B=$(git rev-parse HEAD) &&
     -+		git checkout --orphan=orphan $A &&
     ++		git checkout --orphan=orphan A &&
      +		test_commit orphan &&
     -+		C=$(git rev-parse HEAD) &&
     -+		git rev-list --objects --no-object-names $B |sort >expected &&
      +
     -+		git replace $A $C &&
     -+		git repack -ad &&
     -+		git multi-pack-index write --bitmap &&
     -+		git rev-list --objects --no-object-names --use-bitmap-index $B |sort >actual &&
     -+		test_cmp expected actual
     ++		git replace A HEAD &&
     ++		git repack -ad --write-midx --write-bitmap-index &&
     ++
     ++		# generating reachability bitmaps with replace refs
     ++		# enabled will result in broken clones
     ++		git clone --no-local --bare . clone.git
      +	)
      +'
      +


 builtin/multi-pack-index.c    |  3 +++
 t/t5326-multi-pack-bitmaps.sh | 21 +++++++++++++++++++++
 2 files changed, 24 insertions(+)


base-commit: 3c2a3fdc388747b9eaf4a4a4f2035c1c9ddb26d0

Comments

Junio C Hamano April 17, 2024, 7:34 p.m. UTC | #1

"blanet via GitGitGadget" <[e-mail]> writes:

> From: Xing Xin <[e-mail]>
> ...
> Helped-by: Taylor Blau <[e-mail]>
> Signed-off-by: Xing Xin <[e-mail]>
> ---

I think this took the review in

  https://lore.kernel.org/git/[e-mail]/

into account and is in good shape?

Thanks, both.

Taylor Blau April 18, 2024, 1:22 p.m. UTC | #2

On Wed, Apr 17, 2024 at 12:34:27PM -0700, Junio C Hamano wrote:
> "blanet via GitGitGadget" <[e-mail]> writes:
>
> > From: Xing Xin <[e-mail]>
> > ...
> > Helped-by: Taylor Blau <[e-mail]>
> > Signed-off-by: Xing Xin <[e-mail]>
> > ---
>
> I think this took the review in
>
>   https://lore.kernel.org/git/[e-mail]/
>
> into account and is in good shape?

Yes, sorry for not explicitly ack-ing, this version looks good to me.

Thanks,
Taylor

Junio C Hamano April 18, 2024, 4:06 p.m. UTC | #3

Taylor Blau <[e-mail]> writes:

> On Wed, Apr 17, 2024 at 12:34:27PM -0700, Junio C Hamano wrote:
>> "blanet via GitGitGadget" <[e-mail]> writes:
>>
>> > From: Xing Xin <[e-mail]>
>> > ...
>> > Helped-by: Taylor Blau <[e-mail]>
>> > Signed-off-by: Xing Xin <[e-mail]>
>> > ---
>>
>> I think this took the review in
>>
>>   https://lore.kernel.org/git/[e-mail]/
>>
>> into account and is in good shape?
>
> Yes, sorry for not explicitly ack-ing, this version looks good to me.

Thanks.

Patch

diff --git a/builtin/multi-pack-index.c b/builtin/multi-pack-index.c
index a72aebecaa2..8360932d2e7 100644
--- a/builtin/multi-pack-index.c
+++ b/builtin/multi-pack-index.c
@@ -8,6 +8,7 @@ 
 #include "strbuf.h"
 #include "trace2.h"
 #include "object-store-ll.h"
+#include "replace-object.h"
 
 #define BUILTIN_MIDX_WRITE_USAGE \
 	N_("git multi-pack-index [<options>] write [--preferred-pack=<pack>]" \
@@ -273,6 +274,8 @@  int cmd_multi_pack_index(int argc, const char **argv,
 	};
 	struct option *options = parse_options_concat(builtin_multi_pack_index_options, common_opts);
 
+	disable_replace_refs();
+
 	git_config(git_default_config, NULL);
 
 	if (the_repository &&
diff --git a/t/t5326-multi-pack-bitmaps.sh b/t/t5326-multi-pack-bitmaps.sh
index 70d1b58709a..1fb3b0f9d7a 100755
--- a/t/t5326-multi-pack-bitmaps.sh
+++ b/t/t5326-multi-pack-bitmaps.sh
@@ -434,6 +434,27 @@  test_expect_success 'tagged commits are selected for bitmapping' '
 	)
 '
 
+test_expect_success 'do not follow replace objects for MIDX bitmap' '
+	rm -fr repo &&
+	git init repo &&
+	test_when_finished "rm -fr repo" &&
+	(
+		cd repo &&
+
+		test_commit A &&
+		test_commit B &&
+		git checkout --orphan=orphan A &&
+		test_commit orphan &&
+
+		git replace A HEAD &&
+		git repack -ad --write-midx --write-bitmap-index &&
+
+		# generating reachability bitmaps with replace refs
+		# enabled will result in broken clones
+		git clone --no-local --bare . clone.git
+	)
+'
+
 corrupt_file () {
 	chmod a+w "$1" &&
 	printf "bogus" | dd of="$1" bs=1 seek="12" conv=notrunc