| Message ID | 675d58ecea2f315bd05d2a21f6a473e9de1105a0.1544221121.git.steadmon@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Add commit-graph fuzzer and fix buffer overflow | expand |
Josh Steadmon <steadmon@google.com> writes: > diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh > index 5fe21db99f..5b6b44b78e 100755 > --- a/t/t5318-commit-graph.sh > +++ b/t/t5318-commit-graph.sh > @@ -366,24 +366,30 @@ GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \ > GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4)) > GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES)) > > -# usage: corrupt_graph_and_verify <position> <data> <string> > +# usage: corrupt_graph_and_verify <position> <data> <string> [<zero_pos>] > # Manipulates the commit-graph file at the position > -# by inserting the data, then runs 'git commit-graph verify' > +# by inserting the data, optionally zeroing the file > +# starting at <zero_pos>, then runs 'git commit-graph verify' > # and places the output in the file 'err'. Test 'err' for > # the given string. > corrupt_graph_and_verify() { > pos=$1 > data="${2:-\0}" > grepstr=$3 > + orig_size=$(stat --format=%s $objdir/info/commit-graph) "stat(1)" is not so portable, so you'll get complaints from minority platform users later. So is "truncate(1)". > + zero_pos=${4:-${orig_size}} > cd "$TRASH_DIRECTORY/full" && > test_when_finished mv commit-graph-backup $objdir/info/commit-graph && > cp $objdir/info/commit-graph commit-graph-backup && > printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc && > + truncate --size=$zero_pos $objdir/info/commit-graph && > + truncate --size=$orig_size $objdir/info/commit-graph && > test_must_fail git commit-graph verify 2>test_err && > grep -v "^+" test_err >err > test_i18ngrep "$grepstr" err > } > > + > test_expect_success 'detect bad signature' ' > corrupt_graph_and_verify 0 "\0" \ > "graph signature" > @@ -484,6 +490,11 @@ test_expect_success 'detect invalid checksum hash' ' > "incorrect checksum" > ' > > +test_expect_success 'detect incorrect chunk count' ' > + corrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT "\xff" \ Implementations of printf(1) may not grok "\xff" as a valid representation of "\377". The shell built-in of dash(1) for example would not work with this. > + "chunk lookup table entry missing" $GRAPH_CHUNK_LOOKUP_OFFSET > +' > + > test_expect_success 'git fsck (checks commit-graph)' ' > cd "$TRASH_DIRECTORY/full" && > git fsck &&
On Sun, Dec 09, 2018 at 01:01:29PM +0900, Junio C Hamano wrote: > Josh Steadmon <steadmon@google.com> writes: > > > diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh > > index 5fe21db99f..5b6b44b78e 100755 > > --- a/t/t5318-commit-graph.sh > > +++ b/t/t5318-commit-graph.sh > > @@ -366,24 +366,30 @@ GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \ > > GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4)) > > GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES)) > > > > -# usage: corrupt_graph_and_verify <position> <data> <string> > > +# usage: corrupt_graph_and_verify <position> <data> <string> [<zero_pos>] > > # Manipulates the commit-graph file at the position > > -# by inserting the data, then runs 'git commit-graph verify' > > +# by inserting the data, optionally zeroing the file > > +# starting at <zero_pos>, then runs 'git commit-graph verify' > > # and places the output in the file 'err'. Test 'err' for > > # the given string. > > corrupt_graph_and_verify() { > > pos=$1 > > data="${2:-\0}" > > grepstr=$3 > > + orig_size=$(stat --format=%s $objdir/info/commit-graph) > > "stat(1)" is not so portable, so you'll get complaints from minority > platform users later. So is "truncate(1)". I complain: this patch breaks on macOS (on Travis CI), but in a curious way. First, 'stat' in the above line errors out with: +++stat --format=%s .git/objects/info/commit-graph stat: illegal option -- - usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...] Alas, this doesn't immediately fail the test, because it's not part of the &&-chain. > > + zero_pos=${4:-${orig_size}} No && here, either. > > cd "$TRASH_DIRECTORY/full" && > > test_when_finished mv commit-graph-backup $objdir/info/commit-graph && > > cp $objdir/info/commit-graph commit-graph-backup && > > printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc && > > + truncate --size=$zero_pos $objdir/info/commit-graph && ++truncate --size= .git/objects/info/commit-graph t5318-commit-graph.sh: line 385: truncate: command not found Note that even if 'truncate' were available, it would most likely complain about the empty '--size=' argument resulting from the 'stat' error above. Alas, this doesn't fail the test, either, because ... > > + truncate --size=$orig_size $objdir/info/commit-graph && > > test_must_fail git commit-graph verify 2>test_err && > > grep -v "^+" test_err >err ... here the &&-chain was broken already before this patch. However, since this above command was not executed due to the missing 'truncate', it didn't have a chance to create the 'err' file, ... > > test_i18ngrep "$grepstr" err ... so 'test_i18ngrep' can't find the file, which triggers its linting error, finally aborting the whole test script. > > } > > > > + Stray newline. > > test_expect_success 'detect bad signature' ' > > corrupt_graph_and_verify 0 "\0" \ > > "graph signature" > > @@ -484,6 +490,11 @@ test_expect_success 'detect invalid checksum hash' ' > > "incorrect checksum" > > ' > > > > +test_expect_success 'detect incorrect chunk count' ' > > + corrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT "\xff" \ > > Implementations of printf(1) may not grok "\xff" as a valid > representation of "\377". The shell built-in of dash(1) for example > would not work with this. > > > + "chunk lookup table entry missing" $GRAPH_CHUNK_LOOKUP_OFFSET > > +' > > + > > test_expect_success 'git fsck (checks commit-graph)' ' > > cd "$TRASH_DIRECTORY/full" && > > git fsck &&
On 2018.12.09 13:01, Junio C Hamano wrote: > Josh Steadmon <steadmon@google.com> writes: > > > diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh > > index 5fe21db99f..5b6b44b78e 100755 > > --- a/t/t5318-commit-graph.sh > > +++ b/t/t5318-commit-graph.sh > > @@ -366,24 +366,30 @@ GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \ > > GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4)) > > GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES)) > > > > -# usage: corrupt_graph_and_verify <position> <data> <string> > > +# usage: corrupt_graph_and_verify <position> <data> <string> [<zero_pos>] > > # Manipulates the commit-graph file at the position > > -# by inserting the data, then runs 'git commit-graph verify' > > +# by inserting the data, optionally zeroing the file > > +# starting at <zero_pos>, then runs 'git commit-graph verify' > > # and places the output in the file 'err'. Test 'err' for > > # the given string. > > corrupt_graph_and_verify() { > > pos=$1 > > data="${2:-\0}" > > grepstr=$3 > > + orig_size=$(stat --format=%s $objdir/info/commit-graph) > > "stat(1)" is not so portable, so you'll get complaints from minority > platform users later. So is "truncate(1)". Ack, thanks for the catch. I have a workaround for stat in the form of "wc -c", and for truncate with a combination of dd and /dev/zero. However, I'm finding conflicting information about whether or not /dev/zero exists on macOS. At the least, it sounds like it might not work on very old versions. Would this be acceptable, or should I add a new test function to do this? > > + zero_pos=${4:-${orig_size}} > > cd "$TRASH_DIRECTORY/full" && > > test_when_finished mv commit-graph-backup $objdir/info/commit-graph && > > cp $objdir/info/commit-graph commit-graph-backup && > > printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc && > > + truncate --size=$zero_pos $objdir/info/commit-graph && > > + truncate --size=$orig_size $objdir/info/commit-graph && > > test_must_fail git commit-graph verify 2>test_err && > > grep -v "^+" test_err >err > > test_i18ngrep "$grepstr" err > > } > > > > + > > test_expect_success 'detect bad signature' ' > > corrupt_graph_and_verify 0 "\0" \ > > "graph signature" > > @@ -484,6 +490,11 @@ test_expect_success 'detect invalid checksum hash' ' > > "incorrect checksum" > > ' > > > > +test_expect_success 'detect incorrect chunk count' ' > > + corrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT "\xff" \ > > Implementations of printf(1) may not grok "\xff" as a valid > representation of "\377". The shell built-in of dash(1) for example > would not work with this. Ack, will fix in V4. Thanks. > > + "chunk lookup table entry missing" $GRAPH_CHUNK_LOOKUP_OFFSET > > +' > > + > > test_expect_success 'git fsck (checks commit-graph)' ' > > cd "$TRASH_DIRECTORY/full" && > > git fsck &&
On 2018.12.10 05:28, SZEDER Gábor wrote: > On Sun, Dec 09, 2018 at 01:01:29PM +0900, Junio C Hamano wrote: > > Josh Steadmon <steadmon@google.com> writes: > > > > > diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh > > > index 5fe21db99f..5b6b44b78e 100755 > > > --- a/t/t5318-commit-graph.sh > > > +++ b/t/t5318-commit-graph.sh > > > @@ -366,24 +366,30 @@ GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \ > > > GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4)) > > > GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES)) > > > > > > -# usage: corrupt_graph_and_verify <position> <data> <string> > > > +# usage: corrupt_graph_and_verify <position> <data> <string> [<zero_pos>] > > > # Manipulates the commit-graph file at the position > > > -# by inserting the data, then runs 'git commit-graph verify' > > > +# by inserting the data, optionally zeroing the file > > > +# starting at <zero_pos>, then runs 'git commit-graph verify' > > > # and places the output in the file 'err'. Test 'err' for > > > # the given string. > > > corrupt_graph_and_verify() { > > > pos=$1 > > > data="${2:-\0}" > > > grepstr=$3 > > > + orig_size=$(stat --format=%s $objdir/info/commit-graph) > > > > "stat(1)" is not so portable, so you'll get complaints from minority > > platform users later. So is "truncate(1)". > > I complain: this patch breaks on macOS (on Travis CI), but in a > curious way. First, 'stat' in the above line errors out with: > > +++stat --format=%s .git/objects/info/commit-graph > stat: illegal option -- - > usage: stat [-FlLnqrsx] [-f format] [-t timefmt] [file ...] > > Alas, this doesn't immediately fail the test, because it's not part of > the &&-chain. > > > > + zero_pos=${4:-${orig_size}} > > No && here, either. > > > > cd "$TRASH_DIRECTORY/full" && > > > test_when_finished mv commit-graph-backup $objdir/info/commit-graph && > > > cp $objdir/info/commit-graph commit-graph-backup && > > > printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc && > > > + truncate --size=$zero_pos $objdir/info/commit-graph && > > ++truncate --size= .git/objects/info/commit-graph > t5318-commit-graph.sh: line 385: truncate: command not found > > Note that even if 'truncate' were available, it would most likely > complain about the empty '--size=' argument resulting from the 'stat' > error above. > > Alas, this doesn't fail the test, either, because ... > > > > + truncate --size=$orig_size $objdir/info/commit-graph && > > > test_must_fail git commit-graph verify 2>test_err && > > > grep -v "^+" test_err >err > > ... here the &&-chain was broken already before this patch. However, > since this above command was not executed due to the missing > 'truncate', it didn't have a chance to create the 'err' file, ... > > > > test_i18ngrep "$grepstr" err > > ... so 'test_i18ngrep' can't find the file, which triggers its linting > error, finally aborting the whole test script. > > > > } > > > > > > + > > Stray newline. > > > > test_expect_success 'detect bad signature' ' > > > corrupt_graph_and_verify 0 "\0" \ > > > "graph signature" > > > @@ -484,6 +490,11 @@ test_expect_success 'detect invalid checksum hash' ' > > > "incorrect checksum" > > > ' > > > > > > +test_expect_success 'detect incorrect chunk count' ' > > > + corrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT "\xff" \ > > > > Implementations of printf(1) may not grok "\xff" as a valid > > representation of "\377". The shell built-in of dash(1) for example > > would not work with this. > > > > > + "chunk lookup table entry missing" $GRAPH_CHUNK_LOOKUP_OFFSET > > > +' > > > + > > > test_expect_success 'git fsck (checks commit-graph)' ' > > > cd "$TRASH_DIRECTORY/full" && > > > git fsck && Thanks for the catch. All these will be fixed in V4.
On Mon, Dec 10, 2018 at 01:56:49PM -0800, Josh Steadmon wrote: > > "stat(1)" is not so portable, so you'll get complaints from minority > > platform users later. So is "truncate(1)". > > Ack, thanks for the catch. I have a workaround for stat in the form of > "wc -c", and for truncate with a combination of dd and /dev/zero. > However, I'm finding conflicting information about whether or not > /dev/zero exists on macOS. At the least, it sounds like it might not > work on very old versions. Would this be acceptable, or should I add a > new test function to do this? If you're just interested in truncation (and not a bunch of zero bytes), you can dd from /dev/null. Another way to truncate is to move the file elsewhere, and then copy only some of the bytes back (see t1450's "fsck detects truncated loose object"). -Peff
diff --git a/commit-graph.c b/commit-graph.c index 07dd410f3c..836d65a1d3 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -165,10 +165,20 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd, last_chunk_offset = 8; chunk_lookup = data + 8; for (i = 0; i < graph->num_chunks; i++) { - uint32_t chunk_id = get_be32(chunk_lookup + 0); - uint64_t chunk_offset = get_be64(chunk_lookup + 4); + uint32_t chunk_id; + uint64_t chunk_offset; int chunk_repeated = 0; + if (data + graph_size - chunk_lookup < + GRAPH_CHUNKLOOKUP_WIDTH) { + error(_("chunk lookup table entry missing; graph file may be incomplete")); + free(graph); + return NULL; + } + + chunk_id = get_be32(chunk_lookup + 0); + chunk_offset = get_be64(chunk_lookup + 4); + chunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH; if (chunk_offset > graph_size - GIT_MAX_RAWSZ) { diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 5fe21db99f..5b6b44b78e 100755 --- a/t/t5318-commit-graph.sh +++ b/t/t5318-commit-graph.sh @@ -366,24 +366,30 @@ GRAPH_OCTOPUS_DATA_OFFSET=$(($GRAPH_COMMIT_DATA_OFFSET + \ GRAPH_BYTE_OCTOPUS=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4)) GRAPH_BYTE_FOOTER=$(($GRAPH_OCTOPUS_DATA_OFFSET + 4 * $NUM_OCTOPUS_EDGES)) -# usage: corrupt_graph_and_verify <position> <data> <string> +# usage: corrupt_graph_and_verify <position> <data> <string> [<zero_pos>] # Manipulates the commit-graph file at the position -# by inserting the data, then runs 'git commit-graph verify' +# by inserting the data, optionally zeroing the file +# starting at <zero_pos>, then runs 'git commit-graph verify' # and places the output in the file 'err'. Test 'err' for # the given string. corrupt_graph_and_verify() { pos=$1 data="${2:-\0}" grepstr=$3 + orig_size=$(stat --format=%s $objdir/info/commit-graph) + zero_pos=${4:-${orig_size}} cd "$TRASH_DIRECTORY/full" && test_when_finished mv commit-graph-backup $objdir/info/commit-graph && cp $objdir/info/commit-graph commit-graph-backup && printf "$data" | dd of="$objdir/info/commit-graph" bs=1 seek="$pos" conv=notrunc && + truncate --size=$zero_pos $objdir/info/commit-graph && + truncate --size=$orig_size $objdir/info/commit-graph && test_must_fail git commit-graph verify 2>test_err && grep -v "^+" test_err >err test_i18ngrep "$grepstr" err } + test_expect_success 'detect bad signature' ' corrupt_graph_and_verify 0 "\0" \ "graph signature" @@ -484,6 +490,11 @@ test_expect_success 'detect invalid checksum hash' ' "incorrect checksum" ' +test_expect_success 'detect incorrect chunk count' ' + corrupt_graph_and_verify $GRAPH_BYTE_CHUNK_COUNT "\xff" \ + "chunk lookup table entry missing" $GRAPH_CHUNK_LOOKUP_OFFSET +' + test_expect_success 'git fsck (checks commit-graph)' ' cd "$TRASH_DIRECTORY/full" && git fsck &&
fuzz-commit-graph identified a case where Git will read past the end of a buffer containing a commit graph if the graph's header has an incorrect chunk count. A simple bounds check in parse_commit_graph() prevents this. Signed-off-by: Josh Steadmon <steadmon@google.com> --- commit-graph.c | 14 ++++++++++++-- t/t5318-commit-graph.sh | 15 +++++++++++++-- 2 files changed, 25 insertions(+), 4 deletions(-)