mbox series

[v4,0/5] submodule: parallelize diff

Message ID 20221108184200.2813458-1-calvinwan@google.com (mailing list archive)
Headers show
Series submodule: parallelize diff | expand

Message

Calvin Wan Nov. 8, 2022, 6:41 p.m. UTC
Original cover letter for context:
https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/

Changes since v3

Renamed pipe_output_fn to duplicate_output_fn and now passes an
additional "strbuf* out" parameter. Output is directly duplicated
to that function rather than held in a separate variable.
Slightly rewrote the tests to more accurately capture the expected
output of duplicate_output_fn.

Removed a patch that added an option to hide child process output.
Child process output is now reset in status_duplicate_output.

More style changes as suggested by Avar

Calvin Wan (5):
  run-command: add duplicate_output_fn to run_processes_parallel_opts
  submodule: strbuf variable rename
  submodule: move status parsing into function
  diff-lib: refactor match_stat_with_submodule
  diff-lib: parallelize run_diff_files for submodules

 Documentation/config/submodule.txt |  12 ++
 diff-lib.c                         | 103 +++++++++++--
 run-command.c                      |  13 +-
 run-command.h                      |  24 +++
 submodule.c                        | 229 +++++++++++++++++++++++++----
 submodule.h                        |   9 ++
 t/helper/test-run-command.c        |  21 +++
 t/t0061-run-command.sh             |  39 +++++
 t/t4027-diff-submodule.sh          |  19 +++
 t/t7506-status-submodule.sh        |  19 +++
 10 files changed, 441 insertions(+), 47 deletions(-)

Comments

Glen Choo Nov. 23, 2022, 5:49 p.m. UTC | #1
Calvin Wan <calvinwan@google.com> writes:

> Original cover letter for context:
> https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/
>
> Changes since v3
>
> Renamed pipe_output_fn to duplicate_output_fn and now passes an
> additional "strbuf* out" parameter. Output is directly duplicated
> to that function rather than held in a separate variable.
> Slightly rewrote the tests to more accurately capture the expected
> output of duplicate_output_fn.
>
> Removed a patch that added an option to hide child process output.
> Child process output is now reset in status_duplicate_output.
>
> More style changes as suggested by Avar

For ease of navigation, here are the previous versions:

v3: https://lore.kernel.org/git/20221020232532.1128326-1-calvinwan@google.com/
v2: https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/
v1: https://lore.kernel.org/git/20220922232947.631309-1-calvinwan@google.com/
Junio C Hamano Jan. 15, 2023, 9:31 a.m. UTC | #2
Calvin Wan <calvinwan@google.com> writes:

> Original cover letter for context:
> https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/
> ...
> Calvin Wan (5):
>   run-command: add duplicate_output_fn to run_processes_parallel_opts
>   submodule: strbuf variable rename
>   submodule: move status parsing into function
>   diff-lib: refactor match_stat_with_submodule
>   diff-lib: parallelize run_diff_files for submodules
>
>  Documentation/config/submodule.txt |  12 ++
>  diff-lib.c                         | 103 +++++++++++--
>  run-command.c                      |  13 +-
>  run-command.h                      |  24 +++
>  submodule.c                        | 229 +++++++++++++++++++++++++----
>  submodule.h                        |   9 ++
>  t/helper/test-run-command.c        |  21 +++
>  t/t0061-run-command.sh             |  39 +++++
>  t/t4027-diff-submodule.sh          |  19 +++
>  t/t7506-status-submodule.sh        |  19 +++
>  10 files changed, 441 insertions(+), 47 deletions(-)

While the topic is marked as "Needs review" in the recent "What's
cooking" reports, merging this topic also breaks the "linux-leaks"
job by causing many tests fail:

    t3040-subprojects-basic.sh
    t4010-diff-pathspec.sh
    t4015-diff-whitespace.sh
    t4027-diff-submodule.sh
    t7403-submodule-sync.sh
    t7409-submodule-detached-work-tree.sh
    t7416-submodule-dash-url.sh
    t7450-bad-git-dotfiles.sh
    t7506-status-submodule.sh

Two of the test scripts are touched by this topic, and their
breakage could be caused by newly using other git subcommands that
were known to be leaking (iow, not because this series introduced
new leaks). It also is possible that they fail because this series
added new leaks to the commands these two test scripts use.  In
either case, other tests that haven't been touched by this topic
were definitely broken by new leaks introduced by the changes made
by this series.

Anybody interested should be able to see the breakage themselves by
checking out 'seen' and running

    SANTIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true \
    make test

to see the tree with all in-flight topics are clean, and then by
running the same test after merging this topic to 'seen'.

Thanks.
Calvin Wan Jan. 17, 2023, 7:31 p.m. UTC | #3
Hi Junio

I've sent out a reroll to fix this. Thanks!

Passing leaks CI at:
https://github.com/calvin-wan-google/git/actions/runs/3942292098
(linux-musl technically failed, but it looks like for other reasons)


On Sun, Jan 15, 2023 at 1:31 AM Junio C Hamano <gitster@pobox.com> wrote:
>
> Calvin Wan <calvinwan@google.com> writes:
>
> > Original cover letter for context:
> > https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/
> > ...
> > Calvin Wan (5):
> >   run-command: add duplicate_output_fn to run_processes_parallel_opts
> >   submodule: strbuf variable rename
> >   submodule: move status parsing into function
> >   diff-lib: refactor match_stat_with_submodule
> >   diff-lib: parallelize run_diff_files for submodules
> >
> >  Documentation/config/submodule.txt |  12 ++
> >  diff-lib.c                         | 103 +++++++++++--
> >  run-command.c                      |  13 +-
> >  run-command.h                      |  24 +++
> >  submodule.c                        | 229 +++++++++++++++++++++++++----
> >  submodule.h                        |   9 ++
> >  t/helper/test-run-command.c        |  21 +++
> >  t/t0061-run-command.sh             |  39 +++++
> >  t/t4027-diff-submodule.sh          |  19 +++
> >  t/t7506-status-submodule.sh        |  19 +++
> >  10 files changed, 441 insertions(+), 47 deletions(-)
>
> While the topic is marked as "Needs review" in the recent "What's
> cooking" reports, merging this topic also breaks the "linux-leaks"
> job by causing many tests fail:
>
>     t3040-subprojects-basic.sh
>     t4010-diff-pathspec.sh
>     t4015-diff-whitespace.sh
>     t4027-diff-submodule.sh
>     t7403-submodule-sync.sh
>     t7409-submodule-detached-work-tree.sh
>     t7416-submodule-dash-url.sh
>     t7450-bad-git-dotfiles.sh
>     t7506-status-submodule.sh
>
> Two of the test scripts are touched by this topic, and their
> breakage could be caused by newly using other git subcommands that
> were known to be leaking (iow, not because this series introduced
> new leaks). It also is possible that they fail because this series
> added new leaks to the commands these two test scripts use.  In
> either case, other tests that haven't been touched by this topic
> were definitely broken by new leaks introduced by the changes made
> by this series.
>
> Anybody interested should be able to see the breakage themselves by
> checking out 'seen' and running
>
>     SANTIZE=leak GIT_TEST_PASSING_SANITIZE_LEAK=true \
>     make test
>
> to see the tree with all in-flight topics are clean, and then by
> running the same test after merging this topic to 'seen'.
>
> Thanks.