[v3,0/3] batch blob diff generation

Message

Justin Tobler Feb. 25, 2025, 11:39 p.m. UTC

Through git-diff(1) it is possible to generate a diff directly between
two blobs. This is particularly useful when the pre-image and post-image
blobs are known and we only care about the diff between them.
Unfortunately, if a user has a batch of known blob pairs to compute
diffs for, there is currently not a way to do so via a single Git
process.

To enable support for batch diffs of multiple blob pairs, this series
introduces a new diff plumbing command git-diff-pairs(1) based on a
previous patch series submitted by Peff[1]. This command uses
NUL-delimited raw diffs as its source of input to control exactly which
filepairs are diffed. The advantage of using the raw diff format is that
it already has diff status type and object context information embedded
in each line making it more efficient to generate diffs with as we can
avoid having to peel revisions to get some the same info.

For example:

    git diff-tree -r -z -M $old $new |
    git diff-pairs -p -z

Here the output of git-diff-tree(1) is fed to git-diff-pairs(1) to
generate the same output that would be expected from `git diff-tree -p
-M`. While by itself not particularly useful, this means it is possible
to split git-diff-tree(1) output across multiple git-diff-pairs(1)
processes. Such a feature is useful on the server-side where diffs
bewteen a large set of changes may not be feasible all at once due to
timeout concerns.

This command can be viewed as a backend tool that exposes Git's diff
machinery. In its current form, the frontend that generates the raw diff
lines used as input is expected to most of the heavy lifting (ie.
pathspec limiting, tree object expansion).

This series is structured as follows:

    - Patch 1 adds some new helper functions to get access to the queued
      `diff_filepair` after `diff_queue()` is invoked.

    - Patch 2 introduces the new git-diff-pairs(1) plumbing command.

    - Patch 3 allows git-diff-pairs(1) to immediately compute diffs
      queued on stdin when a NUL-byte is written after a raw input line
      instead of waiting for stdin to close.

Changes since V2:

    - Pathspecs are not supported and thus rejected when provided as
      arguments. It should be possible in a future series to add support
      though.

    - Tree objects present in `diff-pairs` input are rejected. Support
      for tree objects could be added in the future, but for now they
      are rejected to enable to future support in a backwards compatible
      manner.

    - The -z option is required by git-diff-pairs(1). The NUL-delimited
      raw diff format is the only accepted form of input. Consequently,
      NUL-delimited output is the only option in the `--raw` mode.

    - git-diff-pairs(1) defaults to patch output instead of raw output.
      This better fits the intended usecase of the command.

    - A NUL-byte is now always used as the delimiter between batches of
      file pair diffs when queued diffs are explicitly computed by
      writing a NUL-byte on stdin.

    - Several other small cleanups and fixes along with documentation
      changes.

Changes since V1:

    - Changed from git-diff-blob(1) to git-diff-pairs(1) based on a
      previously submitted series.

    - Instead of each line containing a pair of blob revisions, the raw
      diff format is used as input which already has diff status and
      object context embedded.

-Justin

[1]: <[e-mail]>

Justin Tobler (3):
  diff: return diff_filepair from diff queue helpers
  builtin: introduce diff-pairs command
  builtin/diff-pairs: allow explicit diff queue flush

 .gitignore                        |   1 +
 Documentation/git-diff-pairs.adoc |  60 +++++++++
 Documentation/meson.build         |   1 +
 Makefile                          |   1 +
 builtin.h                         |   1 +
 builtin/diff-pairs.c              | 206 ++++++++++++++++++++++++++++++
 command-list.txt                  |   1 +
 diff.c                            |  70 +++++++---
 diff.h                            |  25 ++++
 git.c                             |   1 +
 meson.build                       |   1 +
 t/meson.build                     |   1 +
 t/t4070-diff-pairs.sh             |  83 ++++++++++++
 13 files changed, 432 insertions(+), 20 deletions(-)
 create mode 100644 Documentation/git-diff-pairs.adoc
 create mode 100644 builtin/diff-pairs.c
 create mode 100755 t/t4070-diff-pairs.sh

Comments

Phillip Wood Feb. 26, 2025, 2:58 p.m. UTC | #1

Hi Justin

On 25/02/2025 23:39, Justin Tobler wrote:
> 
> Changes since V2:
> 
>      - Pathspecs are not supported and thus rejected when provided as
>        arguments. It should be possible in a future series to add support
>        though.
> 
>      - Tree objects present in `diff-pairs` input are rejected. Support
>        for tree objects could be added in the future, but for now they
>        are rejected to enable to future support in a backwards compatible
>        manner.
> 
>      - The -z option is required by git-diff-pairs(1). The NUL-delimited
>        raw diff format is the only accepted form of input. Consequently,
>        NUL-delimited output is the only option in the `--raw` mode.
> 
>      - git-diff-pairs(1) defaults to patch output instead of raw output.
>        This better fits the intended usecase of the command.
> 
>      - A NUL-byte is now always used as the delimiter between batches of
>        file pair diffs when queued diffs are explicitly computed by
>        writing a NUL-byte on stdin.
> 
>      - Several other small cleanups and fixes along with documentation
>        changes.

This addresses all my comments on the previous version, thank you. I do 
wonder if tying the input line termination to the output line 
termination is a good idea for a program that aims to to transform one 
diff format into another. Having said that this series is aimed at 
machine consumption of the output so it probably isn't a big problem.

I also think we might want to massage the output in the tests so that 
we're not running test_cmp on files containing NUL bytes. Using

     git diff-tree -z ... | tr '\0' Q >actual

would get rid of the NULs but does not improve the readability of the 
raw diffs that much as everything is still on a single line. Using '\n' 
instead of 'Q' would give us mulit-line output but we would lose 
confidence that the original output was actually NUL terminated.

Best Wishes

Phillip


> Changes since V1:
> 
>      - Changed from git-diff-blob(1) to git-diff-pairs(1) based on a
>        previously submitted series.
> 
>      - Instead of each line containing a pair of blob revisions, the raw
>        diff format is used as input which already has diff status and
>        object context embedded.
> 
> -Justin
> 
> [1]: <[e-mail]>
> 
> Justin Tobler (3):
>    diff: return diff_filepair from diff queue helpers
>    builtin: introduce diff-pairs command
>    builtin/diff-pairs: allow explicit diff queue flush
> 
>   .gitignore                        |   1 +
>   Documentation/git-diff-pairs.adoc |  60 +++++++++
>   Documentation/meson.build         |   1 +
>   Makefile                          |   1 +
>   builtin.h                         |   1 +
>   builtin/diff-pairs.c              | 206 ++++++++++++++++++++++++++++++
>   command-list.txt                  |   1 +
>   diff.c                            |  70 +++++++---
>   diff.h                            |  25 ++++
>   git.c                             |   1 +
>   meson.build                       |   1 +
>   t/meson.build                     |   1 +
>   t/t4070-diff-pairs.sh             |  83 ++++++++++++
>   13 files changed, 432 insertions(+), 20 deletions(-)
>   create mode 100644 Documentation/git-diff-pairs.adoc
>   create mode 100644 builtin/diff-pairs.c
>   create mode 100755 t/t4070-diff-pairs.sh
>