| Message ID | 56cabf3be3b9e228bc948da372db4c9d11fd3926.1661962145.git.gitgitgadget@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | fsmonitor: option to allow fsmonitor to run against network-mounted repos | expand |
On Wed, Aug 31 2022, Eric DeCosta via GitGitGadget wrote: > From: Eric DeCosta <edecosta@mathworks.com> > > Consider the following network working directory that is mounted under > /System/Volumes/Data: > > /network/working/directory > > The git working directory path is: > > /System/Volumes/Data/network/working/directory > > The paths reported by FSEvents always start with /network. fsmonitor > expects paths to be under the working directory; therefore it > fails to match /network/... and ignores the change. > > Change things such that if fsmonitor.allowRemote is true that the > paths reported via FSEevents are normalized to the real path. > > Signed-off-by: Eric DeCosta <edecosta@mathworks.com> > --- > compat/fsmonitor/fsm-listen-darwin.c | 11 ++++++++--- > 1 file changed, 8 insertions(+), 3 deletions(-) > > diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c > index 8e208e8289e..2ed828649ff 100644 > --- a/compat/fsmonitor/fsm-listen-darwin.c > +++ b/compat/fsmonitor/fsm-listen-darwin.c > @@ -26,6 +26,7 @@ > #include "fsmonitor.h" > #include "fsm-listen.h" > #include "fsmonitor--daemon.h" > +#include "fsmonitor-settings.h" > > struct fsm_listen_data > { > @@ -183,7 +184,6 @@ static void my_add_path(struct fsmonitor_batch *batch, const char *path) > free(composed); > } > > - Stray whitespace change, any one isn't much, but they add up (I saw another one earlier in this topic). > static void fsevent_callback(ConstFSEventStreamRef streamRef, > void *ctx, > size_t num_of_events, > @@ -209,7 +209,12 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, > /* > * On Mac, we receive an array of absolute paths. > */ > - path_k = paths[k]; > + if (fsm_settings__get_allow_remote(the_repository) > 0) { > + strbuf_reset(&tmp); > + strbuf_realpath_forgiving(&tmp, paths[k], 0); > + path_k = tmp.buf; > + } else > + path_k = paths[k]; Style: This else should have braces if any if/else arm does, see CodingGuidelines. > [...] > @@ -313,7 +319,6 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, > > case IS_WORKDIR_PATH: > /* try to queue normal pathnames */ > - More stray whitespace
On 8/31/22 12:09 PM, Eric DeCosta via GitGitGadget wrote: > From: Eric DeCosta <edecosta@mathworks.com> > > Consider the following network working directory that is mounted under > /System/Volumes/Data: > > /network/working/directory > > The git working directory path is: > > /System/Volumes/Data/network/working/directory > > The paths reported by FSEvents always start with /network. fsmonitor > expects paths to be under the working directory; therefore it > fails to match /network/... and ignores the change. I'm not sure I understand what's going on here. Are you saying that FSEvents reports network mounted directories with a path relative to the mount-point rather than an absolute path? In your example, is "/network/working/directory" a valid path on your system (that also happens to be the same directory as "/System/Volumes/...") That is, do you have some aliasing going on where both paths are valid -- like a pair of hard-linked directories? Or, is there something special about a network mount point? Did you have to "cd /System/Volumes/..." to get Git to have the absolute path be this? Or were you doing a "cd /network/..."? (Sorry to ask so many questions but I don't have a pair of systems to test any of this on right now.) > > Change things such that if fsmonitor.allowRemote is true that the > paths reported via FSEevents are normalized to the real path. > > Signed-off-by: Eric DeCosta <edecosta@mathworks.com> > --- > compat/fsmonitor/fsm-listen-darwin.c | 11 ++++++++--- > 1 file changed, 8 insertions(+), 3 deletions(-) > > diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c > index 8e208e8289e..2ed828649ff 100644 > --- a/compat/fsmonitor/fsm-listen-darwin.c > +++ b/compat/fsmonitor/fsm-listen-darwin.c > @@ -26,6 +26,7 @@ > #include "fsmonitor.h" > #include "fsm-listen.h" > #include "fsmonitor--daemon.h" > +#include "fsmonitor-settings.h" > > struct fsm_listen_data > { > @@ -183,7 +184,6 @@ static void my_add_path(struct fsmonitor_batch *batch, const char *path) > free(composed); > } > > - > static void fsevent_callback(ConstFSEventStreamRef streamRef, > void *ctx, > size_t num_of_events, > @@ -209,7 +209,12 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, > /* > * On Mac, we receive an array of absolute paths. > */ > - path_k = paths[k]; > + if (fsm_settings__get_allow_remote(the_repository) > 0) { > + strbuf_reset(&tmp); > + strbuf_realpath_forgiving(&tmp, paths[k], 0); > + path_k = tmp.buf; > + } else > + path_k = paths[k]; This feels wrong. It is not that fsmonitor.allowRemote is true, but whether or not this particular file system *IS* remote, right? > > /* > * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR. > @@ -237,6 +242,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, > > fsmonitor_force_resync(state); > fsmonitor_batch__free_list(batch); > + batch = NULL; > string_list_clear(&cookie_list, 0); > > /* > @@ -313,7 +319,6 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, > > case IS_WORKDIR_PATH: > /* try to queue normal pathnames */ > - > if (trace_pass_fl(&trace_fsmonitor)) > log_flags_set(path_k, event_flags[k]); > >
> -----Original Message----- > From: Jeff Hostetler <git@jeffhostetler.com> > Sent: Thursday, September 1, 2022 4:06 PM > To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>; > git@vger.kernel.org > Cc: Eric DeCosta <edecosta@mathworks.com> > Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to the > real path > > > > On 8/31/22 12:09 PM, Eric DeCosta via GitGitGadget wrote: > > From: Eric DeCosta <edecosta@mathworks.com> > > > > Consider the following network working directory that is mounted under > > /System/Volumes/Data: > > > > /network/working/directory > > > > The git working directory path is: > > > > /System/Volumes/Data/network/working/directory > > > > The paths reported by FSEvents always start with /network. fsmonitor > > expects paths to be under the working directory; therefore it fails to > > match /network/... and ignores the change. > > I'm not sure I understand what's going on here. > Are you saying that FSEvents reports network mounted directories with a > path relative to the mount-point rather than an absolute path? > Yes > In your example, is "/network/working/directory" a valid path on your > system (that also happens to be the same directory as > "/System/Volumes/...") > > That is, do you have some aliasing going on where both paths are valid -- like > a pair of hard-linked directories? > Or, is there something special about a network mount point? > > > Did you have to "cd /System/Volumes/..." to get Git to have the absolute > path be this? Or were you doing a "cd /network/..."? > (Sorry to ask so many questions but I don't have a pair of systems to test any > of this on right now.) > "/network/working/directory" is mounted under "/System/Volumes/Data/network/working/directory" There is also a symlink: "/network" -> "/System/Volumes/Data/network" No matter if I "cd /System/Volumes/Data/network/working/directory" or "cd /network/working/directory" the paths reported by FSEvents always start with "/network/working/directory" If I do a similar symlink with local directories, I do not get this unexpected behavior. I need to remove the symlink and try it that way, but I need to coordinate with the machine's owner. > > > > Change things such that if fsmonitor.allowRemote is true that the > > paths reported via FSEevents are normalized to the real path. > > > > Signed-off-by: Eric DeCosta <edecosta@mathworks.com> > > --- > > compat/fsmonitor/fsm-listen-darwin.c | 11 ++++++++--- > > 1 file changed, 8 insertions(+), 3 deletions(-) > > > > diff --git a/compat/fsmonitor/fsm-listen-darwin.c > > b/compat/fsmonitor/fsm-listen-darwin.c > > index 8e208e8289e..2ed828649ff 100644 > > --- a/compat/fsmonitor/fsm-listen-darwin.c > > +++ b/compat/fsmonitor/fsm-listen-darwin.c > > @@ -26,6 +26,7 @@ > > #include "fsmonitor.h" > > #include "fsm-listen.h" > > #include "fsmonitor--daemon.h" > > +#include "fsmonitor-settings.h" > > > > struct fsm_listen_data > > { > > @@ -183,7 +184,6 @@ static void my_add_path(struct fsmonitor_batch > *batch, const char *path) > > free(composed); > > } > > > > - > > static void fsevent_callback(ConstFSEventStreamRef streamRef, > > void *ctx, > > size_t num_of_events, > > @@ -209,7 +209,12 @@ static void > fsevent_callback(ConstFSEventStreamRef streamRef, > > /* > > * On Mac, we receive an array of absolute paths. > > */ > > - path_k = paths[k]; > > + if (fsm_settings__get_allow_remote(the_repository) > 0) { > > + strbuf_reset(&tmp); > > + strbuf_realpath_forgiving(&tmp, paths[k], 0); > > + path_k = tmp.buf; > > + } else > > + path_k = paths[k]; > > This feels wrong. > > It is not that fsmonitor.allowRemote is true, but whether or not this > particular file system *IS* remote, right? True. I suppose each path could be checked, or some bit could be set if the working directory is remote, perhaps along the lines of fsmonitor_ipc__get_path. Then the determination of the IPC path and whether it is remote would be in one place. fsm-settings-* would then just get that bit and check it against allowRemote. Thoughts? > > > > /* > > * If you want to debug FSEvents, log them to > GIT_TRACE_FSMONITOR. > > @@ -237,6 +242,7 @@ static void > fsevent_callback(ConstFSEventStreamRef > > streamRef, > > > > fsmonitor_force_resync(state); > > fsmonitor_batch__free_list(batch); > > + batch = NULL; > > string_list_clear(&cookie_list, 0); > > > > /* > > @@ -313,7 +319,6 @@ static void > fsevent_callback(ConstFSEventStreamRef > > streamRef, > > > > case IS_WORKDIR_PATH: > > /* try to queue normal pathnames */ > > - > > if (trace_pass_fl(&trace_fsmonitor)) > > log_flags_set(path_k, event_flags[k]); > > > >
On 9/2/22 12:35 PM, Eric DeCosta wrote: > > >> -----Original Message----- >> From: Jeff Hostetler <git@jeffhostetler.com> >> Sent: Thursday, September 1, 2022 4:06 PM >> To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>; >> git@vger.kernel.org >> Cc: Eric DeCosta <edecosta@mathworks.com> >> Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to the >> real path >> >> >> >> On 8/31/22 12:09 PM, Eric DeCosta via GitGitGadget wrote: >>> From: Eric DeCosta <edecosta@mathworks.com> >>> >>> Consider the following network working directory that is mounted under >>> /System/Volumes/Data: >>> >>> /network/working/directory >>> >>> The git working directory path is: >>> >>> /System/Volumes/Data/network/working/directory >>> >>> The paths reported by FSEvents always start with /network. fsmonitor >>> expects paths to be under the working directory; therefore it fails to >>> match /network/... and ignores the change. >> >> I'm not sure I understand what's going on here. >> Are you saying that FSEvents reports network mounted directories with a >> path relative to the mount-point rather than an absolute path? >> > > Yes > >> In your example, is "/network/working/directory" a valid path on your >> system (that also happens to be the same directory as >> "/System/Volumes/...") >> >> That is, do you have some aliasing going on where both paths are valid -- like >> a pair of hard-linked directories? >> Or, is there something special about a network mount point? >> >> >> Did you have to "cd /System/Volumes/..." to get Git to have the absolute >> path be this? Or were you doing a "cd /network/..."? >> (Sorry to ask so many questions but I don't have a pair of systems to test any >> of this on right now.) >> > > "/network/working/directory" is mounted under > "/System/Volumes/Data/network/working/directory" > > There is also a symlink: > > "/network" -> "/System/Volumes/Data/network" > > No matter if I "cd /System/Volumes/Data/network/working/directory" > or "cd /network/working/directory" the paths reported by FSEvents > always start with "/network/working/directory" > > If I do a similar symlink with local directories, I do not get this > unexpected behavior. > > I need to remove the symlink and try it that way, but I need to > coordinate with the machine's owner. I think you have stumbled upon a recent MacOS feature called "firmlinks". I'm just reading up on it myself, so I shouldn't speculate here (yet), but maybe you too could do some reading on the subject. This makes me wonder if the symlink is historical. The real magic is in the firmlinks. For example, if I do: $ (cd / ; ls -l | grep Users) drwxr-xr-x 6 root admin 192 Apr 6 2020 Users $ (cd /Users ; df .) Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted on /dev/disk1s1 976490576 608954344 338850488 65% 4181323 4878271557 0% /System/Volumes/Data we can see that /Users is on /System/Volumes/Data (and there is a /System/Volumes/Data/Users directory with the same metadata), but it is not a symlink. See [1] for more info. [1] http://www.swiftforensics.com/2019/10/macos-1015-volumes-firmlink-magic.html [...] >>> @@ -209,7 +209,12 @@ static void >> fsevent_callback(ConstFSEventStreamRef streamRef, >>> /* >>> * On Mac, we receive an array of absolute paths. >>> */ >>> - path_k = paths[k]; >>> + if (fsm_settings__get_allow_remote(the_repository) > 0) { >>> + strbuf_reset(&tmp); >>> + strbuf_realpath_forgiving(&tmp, paths[k], 0); >>> + path_k = tmp.buf; >>> + } else >>> + path_k = paths[k]; >> >> This feels wrong. >> >> It is not that fsmonitor.allowRemote is true, but whether or not this >> particular file system *IS* remote, right? > > True. I suppose each path could be checked, or some bit could be set > if the working directory is remote, perhaps along the lines of > fsmonitor_ipc__get_path. Then the determination of the IPC path > and whether it is remote would be in one place. fsm-settings-* > would then just get that bit and check it against allowRemote. > > Thoughts? I'll do some digging here. There ought to be a way to tell if a component directory in a pathname has a firmlink peer. (But I'm traveling for GitMerge, so I might not have time to look at this until afterwards.) This would indicate that a "bi-directional wormhole" (their words) alias is available (and hopefully, a way to computer the other peer....) I'm thinking that the "/network/..." path in the FSEvents is because FSEventD is using a particular peer spelling (that we weren't expecting). If we can compute the spellings of the peers once when the daemon starts up (and maybe make a little dictionary), then we can do prefix tricks on the absolute path after the path_k = paths[k]; step in fsevent_callback() to extract a relative path rather than an absolute path. Then call fsmonitor_classify_path_relative() instead of _absolute(). Watch out though, there are several places that do rel = path+k + ...state->path_worktree_watch.len... that would need to be adjusted. Hope this helps, Jeff
> -----Original Message----- > From: Jeff Hostetler <git@jeffhostetler.com> > Sent: Tuesday, September 6, 2022 1:14 PM > To: Eric DeCosta <edecosta@mathworks.com>; Eric DeCosta via GitGitGadget > <gitgitgadget@gmail.com>; git@vger.kernel.org > Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to > the real path > > > > On 9/2/22 12:35 PM, Eric DeCosta wrote: > > > > > >> -----Original Message----- > >> From: Jeff Hostetler <git@jeffhostetler.com> > >> Sent: Thursday, September 1, 2022 4:06 PM > >> To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>; > >> git@vger.kernel.org > >> Cc: Eric DeCosta <edecosta@mathworks.com> > >> Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths > >> to the real path > >> > >> > >> > >> On 8/31/22 12:09 PM, Eric DeCosta via GitGitGadget wrote: > >>> From: Eric DeCosta <edecosta@mathworks.com> > >>> > >>> Consider the following network working directory that is mounted > >>> under > >>> /System/Volumes/Data: > >>> > >>> /network/working/directory > >>> > >>> The git working directory path is: > >>> > >>> /System/Volumes/Data/network/working/directory > >>> > >>> The paths reported by FSEvents always start with /network. fsmonitor > >>> expects paths to be under the working directory; therefore it fails > >>> to match /network/... and ignores the change. > >> > >> I'm not sure I understand what's going on here. > >> Are you saying that FSEvents reports network mounted directories with > >> a path relative to the mount-point rather than an absolute path? > >> > > > > Yes > > > >> In your example, is "/network/working/directory" a valid path on your > >> system (that also happens to be the same directory as > >> "/System/Volumes/...") > >> > >> That is, do you have some aliasing going on where both paths are > >> valid -- like a pair of hard-linked directories? > >> Or, is there something special about a network mount point? > >> > >> > >> Did you have to "cd /System/Volumes/..." to get Git to have the > >> absolute path be this? Or were you doing a "cd /network/..."? > >> (Sorry to ask so many questions but I don't have a pair of systems to > >> test any of this on right now.) > >> > > > > "/network/working/directory" is mounted under > > "/System/Volumes/Data/network/working/directory" > > > > There is also a symlink: > > > > "/network" -> "/System/Volumes/Data/network" > > > > No matter if I "cd /System/Volumes/Data/network/working/directory" > > or "cd /network/working/directory" the paths reported by FSEvents > > always start with "/network/working/directory" > > > > If I do a similar symlink with local directories, I do not get this > > unexpected behavior. > > > > I need to remove the symlink and try it that way, but I need to > > coordinate with the machine's owner. > > > I think you have stumbled upon a recent MacOS feature called "firmlinks". > I'm just reading up on it myself, so I shouldn't speculate here (yet), but > maybe you too could do some reading on the subject. > > This makes me wonder if the symlink is historical. The real magic is in the > firmlinks. For example, if I do: > > $ (cd / ; ls -l | grep Users) > drwxr-xr-x 6 root admin 192 Apr 6 2020 Users > > $ (cd /Users ; df .) > Filesystem 512-blocks Used Available Capacity iused ifree %iused Mounted > on > /dev/disk1s1 976490576 608954344 338850488 65% 4181323 4878271557 0% > /System/Volumes/Data > > we can see that /Users is on /System/Volumes/Data (and there is a > /System/Volumes/Data/Users directory with the same metadata), but it is > not a symlink. > > > See [1] for more info. > > [1] > http://www.swiftforensics.com/2019/10/macos-1015-volumes-firmlink- > magic.html <https://protect- > us.mimecast.com/s/QN4qCYEZB2h1p7jWcG1Cbw?domain=swiftforensics.co > m> > > Hmm, I don't see anything that suggests a firmlink is involved but this is new to me so maybe I just don't see it. There is an automount in addition to the symlink though. So, dispensing with the abstract "/network" path, and getting to the details of my environment: % mount | grep /mathworks map auto.sfs.sol2.mathworks on /System/Volumes/Data/mathworks (autofs, automounted, nobrowse) triggered map auto.sfs.sol2.devel on /System/Volumes/Data/mathworks/devel (autofs, automounted, nobrowse) triggered map sfs.worldwide.US-Natick.sol2.devel.sbs on /System/Volumes/Data/mathworks/devel/sbs (autofs, automounted, nobrowse) triggered map sfs.worldwide.US-Natick.devel.sbs.29 on /System/Volumes/Data/mathworks/devel/sbs/29 (autofs, automounted, nobrowse) batfs-sb29-nfs:/vmgr/sbs29/edecosta.Bbashprod1.j1928560.2 on /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j1928560.2 (nfs, nodev, automounted, noatime, nobrowse) ... % ls -l / | grep mathworks lrwxr-xr-x 1 root wheel 35 Aug 29 10:25 home@ -> /System/Volumes/Data/mathworks/home lrwxr-xr-x 1 root wheel 30 Aug 29 10:25 mathworks@ -> /System/Volumes/Data/mathworks My worktree is: /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j1928560.2 or, equivalently via the symlink just: /mathworks/devel/sbs/29/edecosta.Bbashprod1.j1928560.2 I have sudo'ers access now; so I can try mounting things manually, mess about with symlinks and see what I get. > [...] > >>> @@ -209,7 +209,12 @@ static void > >> fsevent_callback(ConstFSEventStreamRef streamRef, > >>> /* > >>> * On Mac, we receive an array of absolute paths. > >>> */ > >>> - path_k = paths[k]; > >>> + if (fsm_settings__get_allow_remote(the_repository) > 0) { > >>> + strbuf_reset(&tmp); > >>> + strbuf_realpath_forgiving(&tmp, paths[k], 0); > >>> + path_k = tmp.buf; > >>> + } else > >>> + path_k = paths[k]; > >> > >> This feels wrong. > >> > >> It is not that fsmonitor.allowRemote is true, but whether or not this > >> particular file system *IS* remote, right? > > > > True. I suppose each path could be checked, or some bit could be set > > if the working directory is remote, perhaps along the lines of > > fsmonitor_ipc__get_path. Then the determination of the IPC path > > and whether it is remote would be in one place. fsm-settings-* > > would then just get that bit and check it against allowRemote. > > > > Thoughts? > > I'll do some digging here. There ought to be a way to tell if a > component directory in a pathname has a firmlink peer. (But I'm > traveling for GitMerge, so I might not have time to look at this > until afterwards.) > > This would indicate that a "bi-directional wormhole" (their words) > alias is available (and hopefully, a way to computer the other peer....) > > I'm thinking that the "/network/..." path in the FSEvents is because > FSEventD is using a particular peer spelling (that we weren't > expecting). > > If we can compute the spellings of the peers once when the daemon > starts up (and maybe make a little dictionary), then we can do > prefix tricks on the absolute path after the > path_k = paths[k]; > step in fsevent_callback() to extract a relative path rather than > an absolute path. > > Then call fsmonitor_classify_path_relative() instead of _absolute(). > > Watch out though, there are several places that do > rel = path+k + ...state->path_worktree_watch.len... > that would need to be adjusted. > > Hope this helps, > Jeff > > Thanks for the insights, I'll dig around and test things out some more. -Eric
> -----Original Message----- > From: Eric DeCosta > Sent: Tuesday, September 6, 2022 3:02 PM > To: Jeff Hostetler <git@jeffhostetler.com>; Eric DeCosta via GitGitGadget > <gitgitgadget@gmail.com>; git@vger.kernel.org > Subject: RE: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to the > real path > > > > > -----Original Message----- > > From: Jeff Hostetler <git@jeffhostetler.com> > > Sent: Tuesday, September 6, 2022 1:14 PM > > To: Eric DeCosta <edecosta@mathworks.com>; Eric DeCosta via > > GitGitGadget <gitgitgadget@gmail.com>; git@vger.kernel.org > > Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths > > to the real path > > > > > > > > On 9/2/22 12:35 PM, Eric DeCosta wrote: > > > > > > > > >> -----Original Message----- > > >> From: Jeff Hostetler <git@jeffhostetler.com> > > >> Sent: Thursday, September 1, 2022 4:06 PM > > >> To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>; > > >> git@vger.kernel.org > > >> Cc: Eric DeCosta <edecosta@mathworks.com> > > >> Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event > > >> paths to the real path > > >> > > >> > > >> > > >> On 8/31/22 12:09 PM, Eric DeCosta via GitGitGadget wrote: > > >>> From: Eric DeCosta <edecosta@mathworks.com> > > >>> > > >>> Consider the following network working directory that is mounted > > >>> under > > >>> /System/Volumes/Data: > > >>> > > >>> /network/working/directory > > >>> > > >>> The git working directory path is: > > >>> > > >>> /System/Volumes/Data/network/working/directory > > >>> > > >>> The paths reported by FSEvents always start with /network. > > >>> fsmonitor expects paths to be under the working directory; > > >>> therefore it fails to match /network/... and ignores the change. > > >> > > >> I'm not sure I understand what's going on here. > > >> Are you saying that FSEvents reports network mounted directories > > >> with a path relative to the mount-point rather than an absolute path? > > >> > > > > > > Yes > > > > > >> In your example, is "/network/working/directory" a valid path on > > >> your system (that also happens to be the same directory as > > >> "/System/Volumes/...") > > >> > > >> That is, do you have some aliasing going on where both paths are > > >> valid -- like a pair of hard-linked directories? > > >> Or, is there something special about a network mount point? > > >> > > >> > > >> Did you have to "cd /System/Volumes/..." to get Git to have the > > >> absolute path be this? Or were you doing a "cd /network/..."? > > >> (Sorry to ask so many questions but I don't have a pair of systems > > >> to test any of this on right now.) > > >> > > > > > > "/network/working/directory" is mounted under > > > "/System/Volumes/Data/network/working/directory" > > > > > > There is also a symlink: > > > > > > "/network" -> "/System/Volumes/Data/network" > > > > > > No matter if I "cd /System/Volumes/Data/network/working/directory" > > > or "cd /network/working/directory" the paths reported by FSEvents > > > always start with "/network/working/directory" > > > > > > If I do a similar symlink with local directories, I do not get this > > > unexpected behavior. > > > > > > I need to remove the symlink and try it that way, but I need to > > > coordinate with the machine's owner. > > > > > > I think you have stumbled upon a recent MacOS feature called "firmlinks". > > I'm just reading up on it myself, so I shouldn't speculate here (yet), > > but maybe you too could do some reading on the subject. > > > > This makes me wonder if the symlink is historical. The real magic is > > in the firmlinks. For example, if I do: > > > > $ (cd / ; ls -l | grep Users) > > drwxr-xr-x 6 root admin 192 Apr 6 2020 Users > > > > $ (cd /Users ; df .) > > Filesystem 512-blocks Used Available Capacity iused ifree %iused > > Mounted on > > /dev/disk1s1 976490576 608954344 338850488 65% 4181323 4878271557 0% > > /System/Volumes/Data > > > > we can see that /Users is on /System/Volumes/Data (and there is a > > /System/Volumes/Data/Users directory with the same metadata), but it > > is not a symlink. > > > > > > See [1] for more info. > > > > [1] > > http://www.swiftforensics.com/2019/10/macos-1015-volumes-firmlink- > > magic.html <https://protect- > > > us.mimecast.com/s/QN4qCYEZB2h1p7jWcG1Cbw?domain=swiftforensics.co > > m> > > > > > > Hmm, I don't see anything that suggests a firmlink is involved but this is new > to me so maybe I just don't see it. There is an automount in addition to the > symlink though. > > So, dispensing with the abstract "/network" path, and getting to the details > of my environment: > > % mount | grep /mathworks > map auto.sfs.sol2.mathworks on /System/Volumes/Data/mathworks > (autofs, automounted, nobrowse) triggered map auto.sfs.sol2.devel on > /System/Volumes/Data/mathworks/devel (autofs, automounted, > nobrowse) triggered map sfs.worldwide.US-Natick.sol2.devel.sbs on > /System/Volumes/Data/mathworks/devel/sbs (autofs, automounted, > nobrowse) triggered map sfs.worldwide.US-Natick.devel.sbs.29 on > /System/Volumes/Data/mathworks/devel/sbs/29 (autofs, automounted, > nobrowse) > batfs-sb29-nfs:/vmgr/sbs29/edecosta.Bbashprod1.j1928560.2 on > /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j19 > 28560.2 (nfs, nodev, automounted, noatime, nobrowse) ... > > % ls -l / | grep mathworks > lrwxr-xr-x 1 root wheel 35 Aug 29 10:25 home@ -> > /System/Volumes/Data/mathworks/home > lrwxr-xr-x 1 root wheel 30 Aug 29 10:25 mathworks@ -> > /System/Volumes/Data/mathworks > > My worktree is: > /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j19 > 28560.2 > or, equivalently via the symlink just: > /mathworks/devel/sbs/29/edecosta.Bbashprod1.j1928560.2 > > I have sudo'ers access now; so I can try mounting things manually, mess > about with symlinks and see what I get. > > > [...] > > >>> @@ -209,7 +209,12 @@ static void > > >> fsevent_callback(ConstFSEventStreamRef streamRef, > > >>> /* > > >>> * On Mac, we receive an array of absolute paths. > > >>> */ > > >>> - path_k = paths[k]; > > >>> + if (fsm_settings__get_allow_remote(the_repository) > 0) { > > >>> + strbuf_reset(&tmp); strbuf_realpath_forgiving(&tmp, paths[k], > > >>> + 0); path_k = tmp.buf; } else path_k = paths[k]; > > >> > > >> This feels wrong. > > >> > > >> It is not that fsmonitor.allowRemote is true, but whether or not > > >> this particular file system *IS* remote, right? > > > > > > True. I suppose each path could be checked, or some bit could be set > > > if the working directory is remote, perhaps along the lines of > > > fsmonitor_ipc__get_path. Then the determination of the IPC path and > > > whether it is remote would be in one place. fsm-settings-* would > > > then just get that bit and check it against allowRemote. > > > > > > Thoughts? > > > > I'll do some digging here. There ought to be a way to tell if a > > component directory in a pathname has a firmlink peer. (But I'm > > traveling for GitMerge, so I might not have time to look at this until > > afterwards.) > > > > This would indicate that a "bi-directional wormhole" (their words) > > alias is available (and hopefully, a way to computer the other > > peer....) > > > > I'm thinking that the "/network/..." path in the FSEvents is because > > FSEventD is using a particular peer spelling (that we weren't > > expecting). > > > > If we can compute the spellings of the peers once when the daemon > > starts up (and maybe make a little dictionary), then we can do prefix > > tricks on the absolute path after the path_k = paths[k]; step in > > fsevent_callback() to extract a relative path rather than an absolute > > path. > > > > Then call fsmonitor_classify_path_relative() instead of _absolute(). > > > > Watch out though, there are several places that do rel = path+k + > > ...state->path_worktree_watch.len... > > that would need to be adjusted. > > > > Hope this helps, > > Jeff > > > > > Thanks for the insights, I'll dig around and test things out some more. > > -Eric This is informative: https://developer.apple.com/forums/thread/120665 -Eric
> -----Original Message----- > From: Eric DeCosta > Sent: Tuesday, September 6, 2022 3:34 PM > To: 'Jeff Hostetler' <git@jeffhostetler.com>; 'Eric DeCosta via GitGitGadget' > <gitgitgadget@gmail.com>; 'git@vger.kernel.org' <git@vger.kernel.org> > Subject: RE: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to the > real path > > > > > -----Original Message----- > > From: Eric DeCosta > > Sent: Tuesday, September 6, 2022 3:02 PM > > To: Jeff Hostetler <git@jeffhostetler.com>; Eric DeCosta via > > GitGitGadget <gitgitgadget@gmail.com>; git@vger.kernel.org > > Subject: RE: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths > > to the real path > > > > > > > > > -----Original Message----- > > > From: Jeff Hostetler <git@jeffhostetler.com> > > > Sent: Tuesday, September 6, 2022 1:14 PM > > > To: Eric DeCosta <edecosta@mathworks.com>; Eric DeCosta via > > > GitGitGadget <gitgitgadget@gmail.com>; git@vger.kernel.org > > > Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event > > > paths to the real path > > > > > > > > > > > > On 9/2/22 12:35 PM, Eric DeCosta wrote: > > > > > > > > > > > >> -----Original Message----- > > > >> From: Jeff Hostetler <git@jeffhostetler.com> > > > >> Sent: Thursday, September 1, 2022 4:06 PM > > > >> To: Eric DeCosta via GitGitGadget <gitgitgadget@gmail.com>; > > > >> git@vger.kernel.org > > > >> Cc: Eric DeCosta <edecosta@mathworks.com> > > > >> Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event > > > >> paths to the real path > > > >> > > > >> > > > >> > > > >> On 8/31/22 12:09 PM, Eric DeCosta via GitGitGadget wrote: > > > >>> From: Eric DeCosta <edecosta@mathworks.com> > > > >>> > > > >>> Consider the following network working directory that is mounted > > > >>> under > > > >>> /System/Volumes/Data: > > > >>> > > > >>> /network/working/directory > > > >>> > > > >>> The git working directory path is: > > > >>> > > > >>> /System/Volumes/Data/network/working/directory > > > >>> > > > >>> The paths reported by FSEvents always start with /network. > > > >>> fsmonitor expects paths to be under the working directory; > > > >>> therefore it fails to match /network/... and ignores the change. > > > >> > > > >> I'm not sure I understand what's going on here. > > > >> Are you saying that FSEvents reports network mounted directories > > > >> with a path relative to the mount-point rather than an absolute path? > > > >> > > > > > > > > Yes > > > > > > > >> In your example, is "/network/working/directory" a valid path on > > > >> your system (that also happens to be the same directory as > > > >> "/System/Volumes/...") > > > >> > > > >> That is, do you have some aliasing going on where both paths are > > > >> valid -- like a pair of hard-linked directories? > > > >> Or, is there something special about a network mount point? > > > >> > > > >> > > > >> Did you have to "cd /System/Volumes/..." to get Git to have the > > > >> absolute path be this? Or were you doing a "cd /network/..."? > > > >> (Sorry to ask so many questions but I don't have a pair of > > > >> systems to test any of this on right now.) > > > >> > > > > > > > > "/network/working/directory" is mounted under > > > > "/System/Volumes/Data/network/working/directory" > > > > > > > > There is also a symlink: > > > > > > > > "/network" -> "/System/Volumes/Data/network" > > > > > > > > No matter if I "cd /System/Volumes/Data/network/working/directory" > > > > or "cd /network/working/directory" the paths reported by FSEvents > > > > always start with "/network/working/directory" > > > > > > > > If I do a similar symlink with local directories, I do not get > > > > this unexpected behavior. > > > > > > > > I need to remove the symlink and try it that way, but I need to > > > > coordinate with the machine's owner. > > > > > > > > > I think you have stumbled upon a recent MacOS feature called > "firmlinks". > > > I'm just reading up on it myself, so I shouldn't speculate here > > > (yet), but maybe you too could do some reading on the subject. > > > > > > This makes me wonder if the symlink is historical. The real magic is > > > in the firmlinks. For example, if I do: > > > > > > $ (cd / ; ls -l | grep Users) > > > drwxr-xr-x 6 root admin 192 Apr 6 2020 Users > > > > > > $ (cd /Users ; df .) > > > Filesystem 512-blocks Used Available Capacity iused ifree %iused > > > Mounted on > > > /dev/disk1s1 976490576 608954344 338850488 65% 4181323 4878271557 > 0% > > > /System/Volumes/Data > > > > > > we can see that /Users is on /System/Volumes/Data (and there is a > > > /System/Volumes/Data/Users directory with the same metadata), but it > > > is not a symlink. > > > > > > > > > See [1] for more info. > > > > > > [1] > > > http://www.swiftforensics.com/2019/10/macos-1015-volumes-firmlink- > > > magic.html <https://protect- > > > > > > us.mimecast.com/s/QN4qCYEZB2h1p7jWcG1Cbw?domain=swiftforensics.co > > > m> > > > > > > > > > > Hmm, I don't see anything that suggests a firmlink is involved but > > this is new to me so maybe I just don't see it. There is an automount > > in addition to the symlink though. > > > > So, dispensing with the abstract "/network" path, and getting to the > > details of my environment: > > > > % mount | grep /mathworks > > map auto.sfs.sol2.mathworks on /System/Volumes/Data/mathworks > (autofs, > > automounted, nobrowse) triggered map auto.sfs.sol2.devel on > > /System/Volumes/Data/mathworks/devel (autofs, automounted, > > nobrowse) triggered map sfs.worldwide.US-Natick.sol2.devel.sbs on > > /System/Volumes/Data/mathworks/devel/sbs (autofs, automounted, > > nobrowse) triggered map sfs.worldwide.US-Natick.devel.sbs.29 on > > /System/Volumes/Data/mathworks/devel/sbs/29 (autofs, automounted, > > nobrowse) > > batfs-sb29-nfs:/vmgr/sbs29/edecosta.Bbashprod1.j1928560.2 on > > > /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j19 > > 28560.2 (nfs, nodev, automounted, noatime, nobrowse) ... > > > > % ls -l / | grep mathworks > > lrwxr-xr-x 1 root wheel 35 Aug 29 10:25 home@ -> > > /System/Volumes/Data/mathworks/home > > lrwxr-xr-x 1 root wheel 30 Aug 29 10:25 mathworks@ -> > > /System/Volumes/Data/mathworks > > > > My worktree is: > > > /System/Volumes/Data/mathworks/devel/sbs/29/edecosta.Bbashprod1.j19 > > 28560.2 > > or, equivalently via the symlink just: > > /mathworks/devel/sbs/29/edecosta.Bbashprod1.j1928560.2 > > > > I have sudo'ers access now; so I can try mounting things manually, > > mess about with symlinks and see what I get. > > > > > [...] > > > >>> @@ -209,7 +209,12 @@ static void > > > >> fsevent_callback(ConstFSEventStreamRef streamRef, > > > >>> /* > > > >>> * On Mac, we receive an array of absolute paths. > > > >>> */ > > > >>> - path_k = paths[k]; > > > >>> + if (fsm_settings__get_allow_remote(the_repository) > 0) { > > > >>> + strbuf_reset(&tmp); strbuf_realpath_forgiving(&tmp, paths[k], > > > >>> + 0); path_k = tmp.buf; } else path_k = paths[k]; > > > >> > > > >> This feels wrong. > > > >> > > > >> It is not that fsmonitor.allowRemote is true, but whether or not > > > >> this particular file system *IS* remote, right? > > > > > > > > True. I suppose each path could be checked, or some bit could be > > > > set if the working directory is remote, perhaps along the lines of > > > > fsmonitor_ipc__get_path. Then the determination of the IPC path > > > > and whether it is remote would be in one place. fsm-settings-* > > > > would then just get that bit and check it against allowRemote. > > > > > > > > Thoughts? > > > > > > I'll do some digging here. There ought to be a way to tell if a > > > component directory in a pathname has a firmlink peer. (But I'm > > > traveling for GitMerge, so I might not have time to look at this > > > until > > > afterwards.) > > > > > > This would indicate that a "bi-directional wormhole" (their words) > > > alias is available (and hopefully, a way to computer the other > > > peer....) > > > > > > I'm thinking that the "/network/..." path in the FSEvents is because > > > FSEventD is using a particular peer spelling (that we weren't > > > expecting). > > > > > > If we can compute the spellings of the peers once when the daemon > > > starts up (and maybe make a little dictionary), then we can do > > > prefix tricks on the absolute path after the path_k = paths[k]; step > > > in > > > fsevent_callback() to extract a relative path rather than an > > > absolute path. > > > > > > Then call fsmonitor_classify_path_relative() instead of _absolute(). > > > > > > Watch out though, there are several places that do rel = path+k + > > > ...state->path_worktree_watch.len... > > > that would need to be adjusted. > > > > > > Hope this helps, > > > Jeff > > > > > > > > Thanks for the insights, I'll dig around and test things out some more. > > > > -Eric > > This is informative: > > https://developer.apple.com/forums/thread/120665 > > -Eric > Conversation with one of our macOS admins: Q: Is /mathworks a symbolic link or some sort of firmlink or synthetic firmlink? A: It’s a “synthetic” link, per apple. Config is managed in /etc/synthetic.conf The auto mounter will also automatically create it based on the contents of /etc/auto_master even without the synthetic.conf file. We went back and forth with apple for a while on the “right” way to do it and now I’m not sure what we settled on, off the top of my head I don’t know if they’re the same under the hood, but it seemed to be able to create root level links even though you can’t with ln anymore I suspect apple wants users to use the synthetic.conf file in all cases... ---- It appears that the automounter is creating "synthetic firmlinks", a la https://derflounder.wordpress.com/2020/01/18/creating-root-level-directories-and-symbolic-links-on-macos-catalina/ When I look in /etc/auto_master I see several entries that appear as symlinks in the output of "ls -l /" but in reality they are synthetic firmlinks. -Eric
On 9/6/22 3:33 PM, Eric DeCosta wrote: > > [...] > > This is informative: > > https://developer.apple.com/forums/thread/120665 > > -Eric > > How strange........ Have you tried the: fcntl(F_GETPATH) vs fcntl(F_GETPATH_NOFIRMLINK) vs realpath() comparison suggested in the comments and does it return anything sensical? Such that we could record both spellings when the daemon starts up. I think we'd have to do it on an open fd on the worktree root directory. Jeff https://developer.apple.com/forums/thread/120665
> -----Original Message----- > From: Jeff Hostetler <git@jeffhostetler.com> > Sent: Wednesday, September 7, 2022 3:15 PM > To: Eric DeCosta <edecosta@mathworks.com>; Eric DeCosta via GitGitGadget > <gitgitgadget@gmail.com>; git@vger.kernel.org > Subject: Re: [PATCH v4 4/4] fsmonitor: normalize FSEvents event paths to the > real path > > > > On 9/6/22 3:33 PM, Eric DeCosta wrote: > > > > > [...] > > > > This is informative: > > > > https://developer.apple.com/forums/thread/120665 > > <https://protect- > us.mimecast.com/s/Vj1mCL9GlLUjmyJjfqLBqG?domain=devel > > oper.apple.com> > > > > -Eric > > > > > > How strange........ > > Have you tried the: > fcntl(F_GETPATH) > vs fcntl(F_GETPATH_NOFIRMLINK) > vs realpath() > > comparison suggested in the comments and does it return anything sensical? > Such that we could record both spellings when the daemon starts up. I think > we'd have to do it on an open fd on the worktree root directory. > > Jeff > > > https://developer.apple.com/forums/thread/120665 <https://protect- > us.mimecast.com/s/Vj1mCL9GlLUjmyJjfqLBqG?domain=developer.apple.com > > I'm not getting the same result: Worktree is: '/System/Volumes/Data/mathworks/devel/sandbox/edecosta/git' F_GETPATH of worktree is '/System/Volumes/Data/mathworks/devel/sandbox/edecosta/git' F_GETPATH_NOFIRMLINK of worktree is '/System/Volumes/Data/mathworks/devel/sandbox/edecosta/git' realpath() of worktree is '/System/Volumes/Data/mathworks/devel/sandbox/edecosta/git' Trying /mathworks/devel/sandbox/edecosta/git F_GETPATH of /mathworks/devel/sandbox/edecosta/git is '/System/Volumes/Data/mathworks/devel/sandbox/edecosta/git' F_GETPATH_NOFIRMLINK of /mathworks/devel/sandbox/edecosta/git is '/System/Volumes/Data/mathworks/devel/sandbox/edecosta/git' realpath() of /mathworks/devel/sandbox/edecosta/git is '/System/Volumes/Data/mathworks/devel/sandbox/edecosta/git' Either something has changed or the fact that "/mathworks" is a "synthetic firmlink" has something to do with it. Maybe look for synthetic firmlinks in the root directory and use that information to figure out what aliases there might be to the worktree. lrwxr-xr-x 1 root wheel 30 Sep 2 11:52 mathworks -> /System/Volumes/Data/mathworks If what the link resolves to is a prefix of the worktree, then it is a firmlink to the worktree. Something like that. -Eric
diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 8e208e8289e..2ed828649ff 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -26,6 +26,7 @@ #include "fsmonitor.h" #include "fsm-listen.h" #include "fsmonitor--daemon.h" +#include "fsmonitor-settings.h" struct fsm_listen_data { @@ -183,7 +184,6 @@ static void my_add_path(struct fsmonitor_batch *batch, const char *path) free(composed); } - static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -209,7 +209,12 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, /* * On Mac, we receive an array of absolute paths. */ - path_k = paths[k]; + if (fsm_settings__get_allow_remote(the_repository) > 0) { + strbuf_reset(&tmp); + strbuf_realpath_forgiving(&tmp, paths[k], 0); + path_k = tmp.buf; + } else + path_k = paths[k]; /* * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR. @@ -237,6 +242,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, fsmonitor_force_resync(state); fsmonitor_batch__free_list(batch); + batch = NULL; string_list_clear(&cookie_list, 0); /* @@ -313,7 +319,6 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, case IS_WORKDIR_PATH: /* try to queue normal pathnames */ - if (trace_pass_fl(&trace_fsmonitor)) log_flags_set(path_k, event_flags[k]);