mbox series

[00/23] Builtin FSMonitor Part 3

Message ID pull.1143.git.1644940773.gitgitgadget@gmail.com (mailing list archive)
Headers show
Series Builtin FSMonitor Part 3 | expand

Message

Philippe Blain via GitGitGadget Feb. 15, 2022, 3:59 p.m. UTC
Here is part 3 of my builtin FSMonitor series.

Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
"seen", so this series should be considered a preview until part 2 moves to
"next". (And part 2 should not graduate to "master" without this part.)

Part 2 established the client code (used by commands like git status) and an
MVP implementation of the FSMonitor daemon. This was sufficient to test the
concepts and basic functionality.

Part 3 finishes the daemon and adds additional tests. This includes:

 * On Windows, handle short- and long-name aliasing.
 * On Mac, handle Unicode aliasing.
 * Mark bare, virtual, and remote working directories incompatible with
   FSMonitor.
 * On Mac, ignore xattr change FSEvents.
 * On Windows, cd out of the working directory root.
 * Create a thread to monitor the health and shutdown the daemon if
   necessary.
 * Speed up handling of directory notification events.
 * Test directory move and rename events.
 * Add performance test.

Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
the untracked cache (uc) and FSMonitor (fsm) on git status.

$ ./p7527-builtin-fsmonitor.sh 
# passed all 67 test(s)
1..67
Test                                                                 this tree         
---------------------------------------------------------------------------------------
7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12) 
7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24) 
7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47) 
7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05) 
7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31) 
7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75) 
7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36) 

7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84) 
7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86) 
7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70) 
7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10) 
7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91) 
7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89) 
7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71) 
7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78) 

7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11) 
7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96) 
7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72) 
7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25) 
7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68) 
7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99) 
7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19) 

7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)   
7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64) 
7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92) 
7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)  
7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)   
7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49) 
7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)   
7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)   


Here is performance data from t/perf/p7519-fsmonitor.sh on the same
synthetic repo containing 1M files on a Macbook Pro. This test performs the
same series of commands on all three FSMonitor options: (1) using the hook
interface to talk to Watchman, (2) no FSMonitor, and (3) the builtin
FSMonitor daemon.

$ ./p7519-fsmonitor.sh 
# passed all 42 test(s)
1..42
Test                                                          this tree        
-------------------------------------------------------------------------------
7519.4: status (fsmonitor=fsmonitor-watchman)                  0.89(0.63+0.17)  
7519.5: status -uno (fsmonitor=fsmonitor-watchman)             0.56(0.43+0.13)  
7519.6: status -uall (fsmonitor=fsmonitor-watchman)           24.52(1.82+9.90) 
7519.7: status (dirty) (fsmonitor=fsmonitor-watchman)         32.40(6.54+9.94) 
7519.8: diff (fsmonitor=fsmonitor-watchman)                    0.66(0.38+0.42)  
7519.9: diff HEAD (fsmonitor=fsmonitor-watchman)               1.60(0.69+0.80)  
7519.10: diff -- 0_files (fsmonitor=fsmonitor-watchman)        0.37(0.29+0.08)  
7519.11: diff -- 10_files (fsmonitor=fsmonitor-watchman)       0.42(0.32+0.09)  
7519.12: diff -- 100_files (fsmonitor=fsmonitor-watchman)      0.42(0.32+0.10)  
7519.13: diff -- 1000_files (fsmonitor=fsmonitor-watchman)     0.41(0.31+0.09)  
7519.14: diff -- 10000_files (fsmonitor=fsmonitor-watchman)    0.44(0.32+0.12)  
7519.15: add (fsmonitor=fsmonitor-watchman)                   24.81(2.44+10.06)

7519.18: status (fsmonitor=disabled)                          11.18(1.80+74.68)
7519.19: status -uno (fsmonitor=disabled)                      7.11(1.50+73.95) 
7519.20: status -uall (fsmonitor=disabled)                    30.09(3.01+91.11)
7519.21: status (dirty) (fsmonitor=disabled)                  31.46(5.03+77.75)
7519.22: diff (fsmonitor=disabled)                             5.79(1.27+62.97) 
7519.23: diff HEAD (fsmonitor=disabled)                        7.29(1.57+76.03) 
7519.24: diff -- 0_files (fsmonitor=disabled)                  0.68(0.30+0.09)  
7519.25: diff -- 10_files (fsmonitor=disabled)                 0.32(0.29+0.07)  
7519.26: diff -- 100_files (fsmonitor=disabled)                0.32(0.29+0.07)  
7519.27: diff -- 1000_files (fsmonitor=disabled)               0.35(0.29+0.09)  
7519.28: diff -- 10000_files (fsmonitor=disabled)              0.56(0.29+0.21)  
7519.29: add (fsmonitor=disabled)                             29.49(3.52+85.88)

7519.31: status (builtin fsmonitor--daemon)                    0.67(0.54+0.09)  
7519.32: status -uno (builtin fsmonitor--daemon)               0.42(0.35+0.08)  
7519.33: status -uall (builtin fsmonitor--daemon)             22.77(1.71+10.35)
7519.34: status (dirty) (builtin fsmonitor--daemon)           21.79(4.18+9.97) 
7519.35: diff (builtin fsmonitor--daemon)                      0.52(0.34+0.36)  
7519.36: diff HEAD (builtin fsmonitor--daemon)                 1.55(0.66+0.80)  
7519.37: diff -- 0_files (builtin fsmonitor--daemon)           0.37(0.31+0.08)  
7519.38: diff -- 10_files (builtin fsmonitor--daemon)          0.35(0.29+0.07)  
7519.39: diff -- 100_files (builtin fsmonitor--daemon)         0.35(0.29+0.07)  
7519.40: diff -- 1000_files (builtin fsmonitor--daemon)        0.35(0.29+0.07)  
7519.41: diff -- 10000_files (builtin fsmonitor--daemon)       0.37(0.29+0.09)  
7519.42: add (builtin fsmonitor--daemon)                      24.64(2.40+10.04)


Jeff Hostetler (23):
  fsm-listen-win32: handle shortnames
  t7527: test FS event reporing on macOS WRT case and Unicode
  t7527: test builtin FSMonitor watching repos with unicode paths
  t/helper/fsmonitor-client: create stress test
  fsmonitor-settings: bare repos are incompatible with FSMonitor
  fsmonitor-settings: stub in platform-specific incompatibility checking
  fsmonitor-settings: virtual repos are incompatible with FSMonitor
  fsmonitor-settings: stub in macOS-specific incompatibility checking
  fsmonitor-settings: remote repos on macOS are incompatible with
    FSMonitor
  fsmonitor-settings: remote repos on Windows are incompatible with
    FSMonitor
  unpack-trees: initialize fsmonitor_has_run_once in o->result
  fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS
  fsmonitor--daemon: print start message only if
    fsmonitor.announceStartup
  fsmonitor--daemon: cd out of worktree root
  fsmonitor--daemon: prepare for adding health thread
  fsmonitor--daemon: rename listener thread related variables
  fsmonitor--daemon: stub in health thread
  fsm-health-win32: add framework to monitor daemon health
  fsm-health-win32: force shutdown daemon if worktree root moves
  fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed
  fsmonitor: optimize processing of directory events
  t7527: FSMonitor tests for directory moves
  t/perf/p7527: add perf test for builtin FSMonitor

 Makefile                               |  19 +-
 builtin/fsmonitor--daemon.c            | 138 ++++++++-
 builtin/update-index.c                 |   9 +
 compat/fsmonitor/fsm-health-darwin.c   |  24 ++
 compat/fsmonitor/fsm-health-win32.c    | 257 ++++++++++++++++
 compat/fsmonitor/fsm-health.h          |  47 +++
 compat/fsmonitor/fsm-listen-darwin.c   |  89 +++++-
 compat/fsmonitor/fsm-listen-win32.c    | 409 ++++++++++++++++++++-----
 compat/fsmonitor/fsm-listen.h          |   2 +-
 compat/fsmonitor/fsm-settings-darwin.c |  75 +++++
 compat/fsmonitor/fsm-settings-win32.c  | 137 +++++++++
 config.mak.uname                       |   5 +
 contrib/buildsystems/CMakeLists.txt    |   8 +
 fsmonitor--daemon.h                    |  11 +-
 fsmonitor-settings.c                   |  84 +++++
 fsmonitor-settings.h                   |  26 ++
 fsmonitor.c                            |  71 ++++-
 t/helper/test-fsmonitor-client.c       | 105 +++++++
 t/perf/p7527-builtin-fsmonitor.sh      | 257 ++++++++++++++++
 t/t7519-status-fsmonitor.sh            |  35 +++
 t/t7527-builtin-fsmonitor.sh           | 157 ++++++++++
 unpack-trees.c                         |   1 +
 22 files changed, 1838 insertions(+), 128 deletions(-)
 create mode 100644 compat/fsmonitor/fsm-health-darwin.c
 create mode 100644 compat/fsmonitor/fsm-health-win32.c
 create mode 100644 compat/fsmonitor/fsm-health.h
 create mode 100644 compat/fsmonitor/fsm-settings-darwin.c
 create mode 100644 compat/fsmonitor/fsm-settings-win32.c
 create mode 100755 t/perf/p7527-builtin-fsmonitor.sh


base-commit: ce8aa175364c7a02115c4b4f91d5674f4d6d8ae8
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-1143%2Fjeffhostetler%2Fbuiltin-fsmonitor-part3-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1143/jeffhostetler/builtin-fsmonitor-part3-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/1143

Comments

Junio C Hamano Feb. 16, 2022, 1 a.m. UTC | #1
"Jeff Hostetler via GitGitGadget" <gitgitgadget@gmail.com> writes:

> Here is part 3 of my builtin FSMonitor series.
>
> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
> "seen", so this series should be considered a preview until part 2 moves to
> "next". (And part 2 should not graduate to "master" without this part.)
>
> Part 2 established the client code (used by commands like git status) and an
> MVP implementation of the FSMonitor daemon. This was sufficient to test the
> concepts and basic functionality.

Sounds like part 2 is sufficiently done to be eligible for being in
'master', waiting for an improved daemon, no?  

Have people been reviewing the patches in part 2?  I haven't had a
chance to take a deeper look myself.

Thanks.
Jeff Hostetler Feb. 16, 2022, 2:04 p.m. UTC | #2
On 2/15/22 8:00 PM, Junio C Hamano wrote:
> "Jeff Hostetler via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> Here is part 3 of my builtin FSMonitor series.
>>
>> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
>> "seen", so this series should be considered a preview until part 2 moves to
>> "next". (And part 2 should not graduate to "master" without this part.)
>>
>> Part 2 established the client code (used by commands like git status) and an
>> MVP implementation of the FSMonitor daemon. This was sufficient to test the
>> concepts and basic functionality.
> 
> Sounds like part 2 is sufficiently done to be eligible for being in
> 'master', waiting for an improved daemon, no?
> 
> Have people been reviewing the patches in part 2?  I haven't had a
> chance to take a deeper look myself.
> 
> Thanks.
> 

Yes, I think Part 2 could advance if we wanted to.  I just didn't
want to presume that, so I was being conservative.  I've always
treated it and Part 3 as a unit.

On the other hand, we've been distributing Part 2 V4 and an
un-submitted version of Part 3 experimentally in git-for-windows
and microsoft/git.git (gvfs) since last summer, so I think we're
good.

It would be good to get some eyes on the Part 2 V5 changes having
to do with removing `core.useBuiltinFSMonitor` and overloading
the existing `core.fsmonitor` to take either a hook path or bool.
That was the only major change between V4 and V5.  And that V5
change has not yet been shipped in GFW nor GVFS.

Thanks
Jeff
Derrick Stolee Feb. 24, 2022, 4:21 p.m. UTC | #3
On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
> Here is part 3 of my builtin FSMonitor series.
> 
> Part 3 builds upon part 2 (jh/builtin-fsmonitor-part2) which is currently in
> "seen", so this series should be considered a preview until part 2 moves to
> "next". (And part 2 should not graduate to "master" without this part.)
> 
> Part 2 established the client code (used by commands like git status) and an
> MVP implementation of the FSMonitor daemon. This was sufficient to test the
> concepts and basic functionality.
> 
> Part 3 finishes the daemon and adds additional tests. This includes:
> 
>  * On Windows, handle short- and long-name aliasing.
>  * On Mac, handle Unicode aliasing.
>  * Mark bare, virtual, and remote working directories incompatible with
>    FSMonitor.
>  * On Mac, ignore xattr change FSEvents.
>  * On Windows, cd out of the working directory root.
>  * Create a thread to monitor the health and shutdown the daemon if
>    necessary.
>  * Speed up handling of directory notification events.
>  * Test directory move and rename events.
>  * Add performance test.

I had been more involved in early versions of parts 1 and 2, but was
not paying attention much as these updates for part 3 were coming into
git-for-windows/git and microsoft/git. (I do recall some of the support
issues that motivated them.)

Since these have been battle-tested in the wild for a while, I do not
doubt that they are solving the user needs they intend to solve.

Instead, I focused on looking at the patches for anything that might
be abnormal. The layout of the code is clear enough that I can easily
see how information from the platform-specific code is being handled,
so I do not worry about corner cases being an issue from the Git side
of the code. As for the platform-specific stuff, maybe there are cases
that are missed that require deep expertise in those platform calls. I
cannot speak to that until we get a user report of something even
stranger happening than the special cases targeted here.

With that said, I mostly found superficial things to improve the
series. Some typos, some rename possibilities, things like that.
 
> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
> the untracked cache (uc) and FSMonitor (fsm) on git status.
> 
> $ ./p7527-builtin-fsmonitor.sh 
> # passed all 67 test(s)
> 1..67
> Test                                                                 this tree         
> ---------------------------------------------------------------------------------------
> 7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12) 
> 7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24) 
> 7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47) 
> 7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05) 
> 7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
> 7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31) 
> 7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75) 
> 7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36) 
> 
> 7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84) 
> 7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86) 
> 7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70) 
> 7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10) 
> 7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91) 
> 7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89) 
> 7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71) 
> 7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78) 
> 
> 7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11) 
> 7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
> 7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96) 
> 7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72) 
> 7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25) 
> 7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68) 
> 7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99) 
> 7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19) 
> 
> 7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)   
> 7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64) 
> 7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92) 
> 7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)  
> 7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)   
> 7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49) 
> 7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)   
> 7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)   

The other stylistic thing is this performance test. It would be nice if
these tests were grouped by the operation (like "status after checkout")
so it is easier to compare the same operation across the matrix definitions.

This would require reordering the test definition as well as allowing the
different cases to simultaneously live in different repositories. The
p2000-sparse-operations.sh has this kind of organization, but you'll need
more involved test cases than "run this command".

Thanks,
-Stolee
Jeff Hostetler March 7, 2022, 9:23 p.m. UTC | #4
On 2/24/22 11:21 AM, Derrick Stolee wrote:
> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>> Here is part 3 of my builtin FSMonitor series.
[...]
>> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
>> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
>> the untracked cache (uc) and FSMonitor (fsm) on git status.
>>
>> $ ./p7527-builtin-fsmonitor.sh
>> # passed all 67 test(s)
>> 1..67
>> Test                                                                 this tree
>> ---------------------------------------------------------------------------------------
>> 7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12)
>> 7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24)
>> 7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47)
>> 7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05)
>> 7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
>> 7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31)
>> 7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75)
>> 7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36)
>>
>> 7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84)
>> 7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86)
>> 7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70)
>> 7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10)
>> 7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91)
>> 7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89)
>> 7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71)
>> 7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78)
>>
>> 7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11)
>> 7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
>> 7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96)
>> 7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72)
>> 7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25)
>> 7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68)
>> 7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99)
>> 7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19)
>>
>> 7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)
>> 7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64)
>> 7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92)
>> 7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)
>> 7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)
>> 7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49)
>> 7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)
>> 7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)
> 
> The other stylistic thing is this performance test. It would be nice if
> these tests were grouped by the operation (like "status after checkout")
> so it is easier to compare the same operation across the matrix definitions.
> 
> This would require reordering the test definition as well as allowing the
> different cases to simultaneously live in different repositories. The
> p2000-sparse-operations.sh has this kind of organization, but you'll need
> more involved test cases than "run this command".

Yeah, it would be nice to turn this test inside-out so that
could group the outputs by test case rather than by (uc,fsm)
combination.  That would certainly make it easier to see how
the two terms affect things.

The problem is I'd either need 4 parallel repos that I could
setup with each (uc,fsm) pair or I'd need to start/stop the
daemon and swap out the {.git/index, .git/config} between
each step.  The former is a problem for monorepos.  The latter
is doable, but I'm not sure it is worth the effort right now.

Jeff
Derrick Stolee March 9, 2022, 3:34 p.m. UTC | #5
On 3/7/2022 4:23 PM, Jeff Hostetler wrote:
> 
> 
> On 2/24/22 11:21 AM, Derrick Stolee wrote:
>> On 2/15/2022 10:59 AM, Jeff Hostetler via GitGitGadget wrote:
>>> Here is part 3 of my builtin FSMonitor series.
> [...]
>>> Here is performance data from t/perf/p7527-builtin-fsmonitor.sh on a
>>> synthetic repo containing 1M files on a Macbook Pro. It shows the effects of
>>> the untracked cache (uc) and FSMonitor (fsm) on git status.
>>>
>>> $ ./p7527-builtin-fsmonitor.sh
>>> # passed all 67 test(s)
>>> 1..67
>>> Test                                                                 this tree
>>> ---------------------------------------------------------------------------------------
>>> 7527.4: [uc false][fsm false] status after checkout                  29.99(3.14+80.12)
>>> 7527.6: [uc false][fsm false] status after big change                73.32(5.11+97.24)
>>> 7527.8: [uc false][fsm false] status after add all                   47.80(5.12+90.47)
>>> 7527.10: [uc false][fsm false] status after add dot                  49.22(5.16+92.05)
>>> 7527.12: [uc false][fsm false] status after commit                   51.53(3.35+100.74)
>>> 7527.14: [uc false][fsm false] status after reset hard               33.74(3.03+85.31)
>>> 7527.16: [uc false][fsm false] status after create untracked files   41.71(3.24+89.75)
>>> 7527.18: [uc false][fsm false] status after clean                    34.33(3.07+89.36)
>>>
>>> 7527.20: [uc false][fsm true] status after checkout                  29.23(1.94+10.84)
>>> 7527.22: [uc false][fsm true] status after big change                64.23(4.66+24.86)
>>> 7527.24: [uc false][fsm true] status after add all                   45.45(4.37+18.70)
>>> 7527.26: [uc false][fsm true] status after add dot                   44.42(4.02+17.10)
>>> 7527.28: [uc false][fsm true] status after commit                    30.52(1.95+10.91)
>>> 7527.30: [uc false][fsm true] status after reset hard                28.70(2.70+13.89)
>>> 7527.32: [uc false][fsm true] status after create untracked files    28.63(2.59+10.71)
>>> 7527.34: [uc false][fsm true] status after clean                     28.97(2.59+10.78)
>>>
>>> 7527.36: [uc true][fsm false] status after checkout                  35.06(3.17+86.11)
>>> 7527.38: [uc true][fsm false] status after big change                74.65(5.14+101.50)
>>> 7527.40: [uc true][fsm false] status after add all                   49.96(5.22+90.96)
>>> 7527.42: [uc true][fsm false] status after add dot                   49.77(5.24+91.72)
>>> 7527.44: [uc true][fsm false] status after commit                    36.95(3.27+92.25)
>>> 7527.46: [uc true][fsm false] status after reset hard                33.89(3.18+85.68)
>>> 7527.48: [uc true][fsm false] status after create untracked files    41.44(3.40+92.99)
>>> 7527.50: [uc true][fsm false] status after clean                     34.60(3.26+90.19)
>>>
>>> 7527.52: [uc true][fsm true] status after checkout                    0.58(0.45+0.10)
>>> 7527.54: [uc true][fsm true] status after big change                 65.16(4.91+25.64)
>>> 7527.56: [uc true][fsm true] status after add all                    45.43(4.45+18.92)
>>> 7527.58: [uc true][fsm true] status after add dot                    15.56(2.57+6.32)
>>> 7527.60: [uc true][fsm true] status after commit                      0.98(0.46+0.11)
>>> 7527.62: [uc true][fsm true] status after reset hard                 30.30(2.96+14.49)
>>> 7527.64: [uc true][fsm true] status after create untracked files      2.15(1.73+0.40)
>>> 7527.66: [uc true][fsm true] status after clean                       1.68(1.56+0.32)
>>
>> The other stylistic thing is this performance test. It would be nice if
>> these tests were grouped by the operation (like "status after checkout")
>> so it is easier to compare the same operation across the matrix definitions.
>>
>> This would require reordering the test definition as well as allowing the
>> different cases to simultaneously live in different repositories. The
>> p2000-sparse-operations.sh has this kind of organization, but you'll need
>> more involved test cases than "run this command".
> 
> Yeah, it would be nice to turn this test inside-out so that
> could group the outputs by test case rather than by (uc,fsm)
> combination.  That would certainly make it easier to see how
> the two terms affect things.
> 
> The problem is I'd either need 4 parallel repos that I could
> setup with each (uc,fsm) pair or I'd need to start/stop the
> daemon and swap out the {.git/index, .git/config} between
> each step.  The former is a problem for monorepos.  The latter
> is doable, but I'm not sure it is worth the effort right now.

Yeah, that extra data is not something to take lightly.

Reworking this test is not critical right now, so I'll leave it
for another time, if we need it. But, perhaps we could set up
the test to create four _worktrees_ so at least the only
duplication is the working tree and not the entire repository.

(The working tree is still a huge amount of data in some cases,
so this doesn't actually solve the problem, just reduces it.)

Thanks,
-Stolee