[v2,14/16] fsmonitor: support case-insensitive events

Commit Message

Jeff Hostetler Feb. 23, 2024, 3:18 a.m. UTC

From: Jeff Hostetler <[e-mail]>

Teach fsmonitor_refresh_callback() to handle case-insensitive
lookups if case-sensitive lookups fail on case-insensitive systems.
This can cause 'git status' to report stale status for files if there
are case issues/errors in the worktree.

The FSMonitor daemon sends FSEvents using the observed spelling
of each pathname.  On case-insensitive file systems this may be
different than the expected case spelling.

The existing code uses index_name_pos() to find the cache-entry for
the pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so
that the worktree scan/index refresh will revisit and revalidate the
path.

On a case-insensitive file system, the exact match lookup may fail
to find the associated cache-entry. This causes status to think that
the cached CE flags are correct and skip over the file.

Update event handling to optionally use the name-hash and dir-name-hash
if necessary.

Signed-off-by: Jeff Hostetler <[e-mail]>
---
 fsmonitor.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 110 insertions(+)

Comments

Junio C Hamano Feb. 23, 2024, 6:14 p.m. UTC | #1

"Jeff Hostetler via GitGitGadget" <[e-mail]> writes:

> +/*
> + * Use the name-hash to do a case-insensitive cache-entry lookup with
> + * the pathname and invalidate the cache-entry.
> + *
> + * Returns the number of cache-entries that we invalidated.
> + */
> +static size_t handle_using_name_hash_icase(
> +	struct index_state *istate, const char *name)
> +{
> +	struct cache_entry *ce = NULL;
> +
> +	ce = index_file_exists(istate, name, strlen(name), 1);
> +	if (!ce)
> +		return 0;
> +
> +	/*
> +	 * A case-insensitive search in the name-hash using the
> +	 * observed pathname found a cache-entry, so the observed path
> +	 * is case-incorrect.  Invalidate the cache-entry and use the
> +	 * correct spelling from the cache-entry to invalidate the
> +	 * untracked-cache.  Since we now have sparse-directories in
> +	 * the index, the observed pathname may represent a regular
> +	 * file or a sparse-index directory.
> +	 *
> +	 * Note that we should not have seen FSEvents for a
> +	 * sparse-index directory, but we handle it just in case.
> +	 *
> +	 * Either way, we know that there are not any cache-entries for
> +	 * children inside the cone of the directory, so we don't need to
> +	 * do the usual scan.
> +	 */
> +	trace_printf_key(&trace_fsmonitor,
> +			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
> +			 name, ce->name);
> +
> +	untracked_cache_invalidate_trimmed_path(istate, ce->name, 0);
> +	ce->ce_flags &= ~CE_FSMONITOR_VALID;
> +	return 1;
> +}

You first ask the name-hash to turn the incoming "name" into the
case variant that we know about, i.e. ce->name, and use that to
access the untracked cache.  Clever and makes sense.  But if we have
ce->name, doesn't it mean the name is tracked?  Do we find anything
useful to do in the untracked cache invalidation codepath in that
case?

An FSmonitor event with case-incorrect pathname for a directory may
not be this trivial, I presume, and I expect that is what the
remainder of this patch is about.

> +
> +/*
> + * Use the dir-name-hash to find the correct-case spelling of the
> + * directory.  Use the canonical spelling to invalidate all of the
> + * cache-entries within the matching cone.
> + *
> + * Returns the number of cache-entries that we invalidated.
> + */
> +static size_t handle_using_dir_name_hash_icase(
> +	struct index_state *istate, const char *name)

It is a bit unfortunate that here on the name-hash side we contrast
the two helper function variants as "dir-name" vs "name", while the
original handle_path side use "without_slash" vs "with_slash".

If I understand correctly, it is not like there are two distinct
hashes, "name-hash" vs "dir-name-hash".  Both of these helpers use
the same "name-hash" mechanism, and this function differs from the
previous one in that it is about a directory, which is why it has
"dir" in its name.  I wonder if we renamed the other one with
"nondir" in its name, and the other without_slash and with_slash
pair to match, e.g., handle_nondir_path() vs handle_dir_path(), or
something like that, the resulting names for these four functions
become easier to contrast and understand?

> +{
> +	struct strbuf canonical_path = STRBUF_INIT;
> +	int pos;
> +	size_t len = strlen(name);
> +	size_t nr_in_cone;
> +
> +	if (name[len - 1] == '/')
> +		len--;
> +
> +	if (!index_dir_find(istate, name, len, &canonical_path))
> +		return 0; /* name is untracked */
> +
> +	if (!memcmp(name, canonical_path.buf, canonical_path.len)) {
> +		strbuf_release(&canonical_path);
> +		/*
> +		 * NEEDSWORK: Our caller already tried an exact match
> +		 * and failed to find one.  They called us to do an
> +		 * ICASE match, so we should never get an exact match,
> +		 * so we could promote this to a BUG() here if we
> +		 * wanted to.  It doesn't hurt anything to just return
> +		 * 0 and go on becaus we should never get here.  Or we
> +		 * could just get rid of the memcmp() and this "if"
> +		 * clause completely.
> +		 */
> +		return 0; /* should not happen */
> +	}

"becaus" -> "because".

If we should never get here, having BUG("we should never get here")
would not hurt anything, either.  On the other hand, silently
returning 0 will hide the bug under the carpet, and I am not sure it
is fair to call it "doesn't hurt anything".

> +
> +	trace_printf_key(&trace_fsmonitor,
> +			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
> +			 name, canonical_path.buf);
> +
> +	/*
> +	 * The dir-name-hash only tells us the corrected spelling of
> +	 * the prefix.  We have to use this canonical path to do a
> +	 * lookup in the cache-entry array so that we repeat the
> +	 * original search using the case-corrected spelling.
> +	 */
> +	strbuf_addch(&canonical_path, '/');
> +	pos = index_name_pos(istate, canonical_path.buf,
> +			     canonical_path.len);
> +	nr_in_cone = handle_path_with_trailing_slash(
> +		istate, canonical_path.buf, pos);
> +	strbuf_release(&canonical_path);
> +	return nr_in_cone;
> +}

Nice.  Do we need to give this corrected name to help untracked
cache invalidation from the caller that called us?

> @@ -319,6 +416,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
>  	else
>  		nr_in_cone = handle_path_without_trailing_slash(istate, name, pos);
>  
> +	/*
> +	 * If we did not find an exact match for this pathname or any
> +	 * cache-entries with this directory prefix and we're on a
> +	 * case-insensitive file system, try again using the name-hash
> +	 * and dir-name-hash.
> +	 */
> +	if (!nr_in_cone && ignore_case) {
> +		nr_in_cone = handle_using_name_hash_icase(istate, name);
> +		if (!nr_in_cone)
> +			nr_in_cone = handle_using_dir_name_hash_icase(
> +				istate, name);
> +	}

It might be interesting to learn how often we go through these
"fallback" code paths by tracing.  Maybe it will become too noisy?
I dunno.

>  	if (nr_in_cone)
>  		trace_printf_key(&trace_fsmonitor,
>  				 "fsmonitor_refresh_callback CNT: %d",

Torsten Bögershausen Feb. 25, 2024, 1:10 p.m. UTC | #2

On Fri, Feb 23, 2024 at 03:18:18AM +0000, Jeff Hostetler via GitGitGadget wrote:
> From: Jeff Hostetler <[e-mail]>
>
> Teach fsmonitor_refresh_callback() to handle case-insensitive
> lookups if case-sensitive lookups fail on case-insensitive systems.
> This can cause 'git status' to report stale status for files if there
> are case issues/errors in the worktree.
>
> The FSMonitor daemon sends FSEvents using the observed spelling
> of each pathname.  On case-insensitive file systems this may be
> different than the expected case spelling.
>
> The existing code uses index_name_pos() to find the cache-entry for
> the pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so
> that the worktree scan/index refresh will revisit and revalidate the
> path.
>
> On a case-insensitive file system, the exact match lookup may fail
> to find the associated cache-entry. This causes status to think that
> the cached CE flags are correct and skip over the file.
>
> Update event handling to optionally use the name-hash and dir-name-hash
> if necessary.
>
> Signed-off-by: Jeff Hostetler <[e-mail]>
> ---
>  fsmonitor.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 110 insertions(+)
>
> diff --git a/fsmonitor.c b/fsmonitor.c
> index 739ddbf7aca..ac638a61c00 100644
> --- a/fsmonitor.c
> +++ b/fsmonitor.c
> @@ -5,6 +5,7 @@
>  #include "ewah/ewok.h"
>  #include "fsmonitor.h"
>  #include "fsmonitor-ipc.h"
> +#include "name-hash.h"
>  #include "run-command.h"
>  #include "strbuf.h"
>  #include "trace2.h"
> @@ -186,6 +187,102 @@ static int query_fsmonitor_hook(struct repository *r,
>  static size_t handle_path_with_trailing_slash(
>  	struct index_state *istate, const char *name, int pos);
>
> +/*
> + * Use the name-hash to do a case-insensitive cache-entry lookup with
> + * the pathname and invalidate the cache-entry.
> + *
> + * Returns the number of cache-entries that we invalidated.
> + */
> +static size_t handle_using_name_hash_icase(
> +	struct index_state *istate, const char *name)
> +{
> +	struct cache_entry *ce = NULL;
> +
> +	ce = index_file_exists(istate, name, strlen(name), 1);
> +	if (!ce)
> +		return 0;
> +
> +	/*
> +	 * A case-insensitive search in the name-hash using the
> +	 * observed pathname found a cache-entry, so the observed path
> +	 * is case-incorrect.  Invalidate the cache-entry and use the
> +	 * correct spelling from the cache-entry to invalidate the
> +	 * untracked-cache.  Since we now have sparse-directories in
> +	 * the index, the observed pathname may represent a regular
> +	 * file or a sparse-index directory.
> +	 *
> +	 * Note that we should not have seen FSEvents for a
> +	 * sparse-index directory, but we handle it just in case.
> +	 *
> +	 * Either way, we know that there are not any cache-entries for
> +	 * children inside the cone of the directory, so we don't need to
> +	 * do the usual scan.
> +	 */
> +	trace_printf_key(&trace_fsmonitor,
> +			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
> +			 name, ce->name);
> +
> +	untracked_cache_invalidate_trimmed_path(istate, ce->name, 0);
> +
> +	ce->ce_flags &= ~CE_FSMONITOR_VALID;
> +	return 1;
> +}
> +
> +/*
> + * Use the dir-name-hash to find the correct-case spelling of the
> + * directory.  Use the canonical spelling to invalidate all of the
> + * cache-entries within the matching cone.
> + *
> + * Returns the number of cache-entries that we invalidated.
> + */
> +static size_t handle_using_dir_name_hash_icase(
> +	struct index_state *istate, const char *name)
> +{
> +	struct strbuf canonical_path = STRBUF_INIT;
> +	int pos;
> +	size_t len = strlen(name);
> +	size_t nr_in_cone;
> +
> +	if (name[len - 1] == '/')
> +		len--;
> +
> +	if (!index_dir_find(istate, name, len, &canonical_path))
> +		return 0; /* name is untracked */
> +
> +	if (!memcmp(name, canonical_path.buf, canonical_path.len)) {
> +		strbuf_release(&canonical_path);
> +		/*
> +		 * NEEDSWORK: Our caller already tried an exact match
> +		 * and failed to find one.  They called us to do an
> +		 * ICASE match, so we should never get an exact match,
> +		 * so we could promote this to a BUG() here if we
> +		 * wanted to.  It doesn't hurt anything to just return
> +		 * 0 and go on becaus we should never get here.  Or we
> +		 * could just get rid of the memcmp() and this "if"
> +		 * clause completely.
> +		 */
> +		return 0; /* should not happen */

In very very theory, there may be a race-condition,
when a directory is renamed very fast, more than once.
I don't think, that the "it did not match exactly, but
now it matches" is a problem.
Question: Does it make sense to just remove this ?
And, may be, find out that the "corrected spelling (tm)"
of "DIR1" is not "dir1", neither "Dir1", but, exactly, "DIR1" ?
Would that be a problem ?


> +	}
> +
> +	trace_printf_key(&trace_fsmonitor,
> +			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
> +			 name, canonical_path.buf);
> +
> +	/*
> +	 * The dir-name-hash only tells us the corrected spelling of
> +	 * the prefix.  We have to use this canonical path to do a
> +	 * lookup in the cache-entry array so that we repeat the
> +	 * original search using the case-corrected spelling.
> +	 */
> +	strbuf_addch(&canonical_path, '/');
> +	pos = index_name_pos(istate, canonical_path.buf,
> +			     canonical_path.len);
> +	nr_in_cone = handle_path_with_trailing_slash(
> +		istate, canonical_path.buf, pos);
> +	strbuf_release(&canonical_path);
> +	return nr_in_cone;
> +}
> +
>  /*
>   * The daemon sent an observed pathname without a trailing slash.
>   * (This is the normal case.)  We do not know if it is a tracked or
> @@ -319,6 +416,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
>  	else
>  		nr_in_cone = handle_path_without_trailing_slash(istate, name, pos);
>
> +	/*
> +	 * If we did not find an exact match for this pathname or any
> +	 * cache-entries with this directory prefix and we're on a
> +	 * case-insensitive file system, try again using the name-hash
> +	 * and dir-name-hash.
> +	 */
> +	if (!nr_in_cone && ignore_case) {
> +		nr_in_cone = handle_using_name_hash_icase(istate, name);
> +		if (!nr_in_cone)
> +			nr_in_cone = handle_using_dir_name_hash_icase(
> +				istate, name);
> +	}
> +
>  	if (nr_in_cone)
>  		trace_printf_key(&trace_fsmonitor,
>  				 "fsmonitor_refresh_callback CNT: %d",
> --
> gitgitgadget
>
>

Jeff Hostetler Feb. 26, 2024, 8:41 p.m. UTC | #3

On 2/23/24 1:14 PM, Junio C Hamano wrote:
> "Jeff Hostetler via GitGitGadget" <[e-mail]> writes:
> 
>> +/*
>> + * Use the name-hash to do a case-insensitive cache-entry lookup with
>> + * the pathname and invalidate the cache-entry.
>> + *
>> + * Returns the number of cache-entries that we invalidated.
>> + */
>> +static size_t handle_using_name_hash_icase(
>> +	struct index_state *istate, const char *name)
>> +{
>> +	struct cache_entry *ce = NULL;
>> +
>> +	ce = index_file_exists(istate, name, strlen(name), 1);
>> +	if (!ce)
>> +		return 0;
>> +
>> +	/*
>> +	 * A case-insensitive search in the name-hash using the
>> +	 * observed pathname found a cache-entry, so the observed path
>> +	 * is case-incorrect.  Invalidate the cache-entry and use the
>> +	 * correct spelling from the cache-entry to invalidate the
>> +	 * untracked-cache.  Since we now have sparse-directories in
>> +	 * the index, the observed pathname may represent a regular
>> +	 * file or a sparse-index directory.
>> +	 *
>> +	 * Note that we should not have seen FSEvents for a
>> +	 * sparse-index directory, but we handle it just in case.
>> +	 *
>> +	 * Either way, we know that there are not any cache-entries for
>> +	 * children inside the cone of the directory, so we don't need to
>> +	 * do the usual scan.
>> +	 */
>> +	trace_printf_key(&trace_fsmonitor,
>> +			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
>> +			 name, ce->name);
>> +
>> +	untracked_cache_invalidate_trimmed_path(istate, ce->name, 0);
>> +	ce->ce_flags &= ~CE_FSMONITOR_VALID;
>> +	return 1;
>> +}
> 
> You first ask the name-hash to turn the incoming "name" into the
> case variant that we know about, i.e. ce->name, and use that to
> access the untracked cache.  Clever and makes sense.  But if we have
> ce->name, doesn't it mean the name is tracked?  Do we find anything
> useful to do in the untracked cache invalidation codepath in that
> case?
> 
> An FSmonitor event with case-incorrect pathname for a directory may
> not be this trivial, I presume, and I expect that is what the
> remainder of this patch is about.

We're going to use "handle_using_name_hash_icase()" to lookup both
qualified (with trailing slash provided by the daemon) paths and
unqualified paths (either a file or a directory on a platform that
can't tell), so there are 3 cases to worry about.

If we fail to find a cache-entry in the name-hash, we know nothing
about the path and we still have the three cases to worry about and
we let the caller deal with that.

If we DO find a matching cache-entry, then it is either a tracked
file or one of the new sparse-directories cache-entries.  We now know
the correct case-spelling.  I don't think it is possible for the UC
to have an entry for this spelling, so you're right, we may not need
to explicitly invalidate the UC here.  I'll add a comment to the code
about this.


> 
>> +
>> +/*
>> + * Use the dir-name-hash to find the correct-case spelling of the
>> + * directory.  Use the canonical spelling to invalidate all of the
>> + * cache-entries within the matching cone.
>> + *
>> + * Returns the number of cache-entries that we invalidated.
>> + */
>> +static size_t handle_using_dir_name_hash_icase(
>> +	struct index_state *istate, const char *name)
> 
> It is a bit unfortunate that here on the name-hash side we contrast
> the two helper function variants as "dir-name" vs "name", while the
> original handle_path side use "without_slash" vs "with_slash".
> 
> If I understand correctly, it is not like there are two distinct
> hashes, "name-hash" vs "dir-name-hash".  Both of these helpers use
> the same "name-hash" mechanism, and this function differs from the
> previous one in that it is about a directory, which is why it has
> "dir" in its name.  I wonder if we renamed the other one with
> "nondir" in its name, and the other without_slash and with_slash
> pair to match, e.g., handle_nondir_path() vs handle_dir_path(), or
> something like that, the resulting names for these four functions
> become easier to contrast and understand?

name-hash.[ch] has 2 distinct hash-maps inside it. The "name-hash"
that we typically think about. And a well-hidden "dir-name-hash"
in the same source file.

The "name-hash" maps each cache-entry's pathname to its ce* in
the cache-entry[] (case-insensitively).

The "dir-name-hash" maps each unique directory prefix over all
of the cache-entries to the case-correct prefix.  That is, if the
index contains "dir1/Dir2/DIR3/file1" and "dir1/dir4/file2", the
dir hash will have 4 entries
     { "dir1", "dir1/Dir2", "dir1/Dir2/DIR3", "dir1/dir4" }.
This lets us do lookups without having to do a linear search on
the entire cache-entry[] every time.

These 2 hashes are demand-loaded only when needed (and usually only
when ignore_case is set IIRC).

When "handle_using_dir_name_hash_icase()" is called we still don't
know if the pathname is actually a file or directory, all we know
is that we did not find a case-sensitive exact match nor a
case-insensitive match against the cache-entry[] using the name-hash.
The pathname could be a (unqualified) directory or just a plain
untracked file.  So here, if we find it in the dir-name-hash, we now
know that it is a directory and that there was a case-error and we now
know the directory's correct case-spelling.

So we use that discovered case-correct spelling to invalidate the
untracked-cache.

> 
>> +{
>> +	struct strbuf canonical_path = STRBUF_INIT;
>> +	int pos;
>> +	size_t len = strlen(name);
>> +	size_t nr_in_cone;
>> +
>> +	if (name[len - 1] == '/')
>> +		len--;
>> +
>> +	if (!index_dir_find(istate, name, len, &canonical_path))
>> +		return 0; /* name is untracked */
>> +
>> +	if (!memcmp(name, canonical_path.buf, canonical_path.len)) {
>> +		strbuf_release(&canonical_path);
>> +		/*
>> +		 * NEEDSWORK: Our caller already tried an exact match
>> +		 * and failed to find one.  They called us to do an
>> +		 * ICASE match, so we should never get an exact match,
>> +		 * so we could promote this to a BUG() here if we
>> +		 * wanted to.  It doesn't hurt anything to just return
>> +		 * 0 and go on becaus we should never get here.  Or we
>> +		 * could just get rid of the memcmp() and this "if"
>> +		 * clause completely.
>> +		 */
>> +		return 0; /* should not happen */
>> +	}
> 
> "becaus" -> "because".
> 
> If we should never get here, having BUG("we should never get here")
> would not hurt anything, either.  On the other hand, silently
> returning 0 will hide the bug under the carpet, and I am not sure it
> is fair to call it "doesn't hurt anything".

I'll make it a BUG().

> 
>> +
>> +	trace_printf_key(&trace_fsmonitor,
>> +			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
>> +			 name, canonical_path.buf);
>> +
>> +	/*
>> +	 * The dir-name-hash only tells us the corrected spelling of
>> +	 * the prefix.  We have to use this canonical path to do a
>> +	 * lookup in the cache-entry array so that we repeat the
>> +	 * original search using the case-corrected spelling.
>> +	 */
>> +	strbuf_addch(&canonical_path, '/');
>> +	pos = index_name_pos(istate, canonical_path.buf,
>> +			     canonical_path.len);
>> +	nr_in_cone = handle_path_with_trailing_slash(
>> +		istate, canonical_path.buf, pos);
>> +	strbuf_release(&canonical_path);
>> +	return nr_in_cone;
>> +}
> 
> Nice.  Do we need to give this corrected name to help untracked
> cache invalidation from the caller that called us?

In an earlier commit, I moved the call to invalidate the untracked-cache
into the two handle_path_with[out]_trailing_slash() functions so that
we wouldn't have to worry about it here.

> 
>> @@ -319,6 +416,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
>>   	else
>>   		nr_in_cone = handle_path_without_trailing_slash(istate, name, pos);
>>   
>> +	/*
>> +	 * If we did not find an exact match for this pathname or any
>> +	 * cache-entries with this directory prefix and we're on a
>> +	 * case-insensitive file system, try again using the name-hash
>> +	 * and dir-name-hash.
>> +	 */
>> +	if (!nr_in_cone && ignore_case) {
>> +		nr_in_cone = handle_using_name_hash_icase(istate, name);
>> +		if (!nr_in_cone)
>> +			nr_in_cone = handle_using_dir_name_hash_icase(
>> +				istate, name);
>> +	}
> 
> It might be interesting to learn how often we go through these
> "fallback" code paths by tracing.  Maybe it will become too noisy?
> I dunno.

I'm afraid it will be very noisy. On Windows and Mac we'll probably end
up falling back for anything that is untracked, unfortunately.  That is,
if there is no cache-entry for "foo.obj", then we'll look for a
case-error (maybe there is a tracked "FOO.OBJ" file in the index or a
"Foo.Obj" directory), before we can say it is untracked.

I'm not happy about this (and no, I haven't had time to measure the
perf hit we'll take), but right now I'm just worried about the
correctness -- I've had several reports of stale/incomplete status
when IDE tools change file/directory case in unexpected ways....

Thanks
Jeff

Jeff Hostetler Feb. 26, 2024, 8:47 p.m. UTC | #4

On 2/25/24 8:10 AM, Torsten Bögershausen wrote:
>> +	if (!memcmp(name, canonical_path.buf, canonical_path.len)) {
>> +		strbuf_release(&canonical_path);
>> +		/*
>> +		 * NEEDSWORK: Our caller already tried an exact match
>> +		 * and failed to find one.  They called us to do an
>> +		 * ICASE match, so we should never get an exact match,
>> +		 * so we could promote this to a BUG() here if we
>> +		 * wanted to.  It doesn't hurt anything to just return
>> +		 * 0 and go on becaus we should never get here.  Or we
>> +		 * could just get rid of the memcmp() and this "if"
>> +		 * clause completely.
>> +		 */
>> +		return 0; /* should not happen */
> 
> In very very theory, there may be a race-condition,
> when a directory is renamed very fast, more than once.
> I don't think, that the "it did not match exactly, but
> now it matches" is a problem.
> Question: Does it make sense to just remove this ?
> And, may be, find out that the "corrected spelling (tm)"
> of "DIR1" is not "dir1", neither "Dir1", but, exactly, "DIR1" ?
> Would that be a problem ?
> 

I just meant that the dir-name-hash that we computed when
we loaded the index found an exact-case match here that
wasn't found when called index_name_pos() and the negative
"pos" didn't point to this exact-case prefix.  This should
not happen.

Yeah I didn't think it should be a fatal condition, but
since it shouldn't happen, we can make it a BUG() and see.

Jeff

Junio C Hamano Feb. 26, 2024, 9:18 p.m. UTC | #5

Jeff Hostetler <[e-mail]> writes:

> I'm not happy about this (and no, I haven't had time to measure the
> perf hit we'll take), but right now I'm just worried about the
> correctness -- I've had several reports of stale/incomplete status
> when IDE tools change file/directory case in unexpected ways....

Of course, it is a good discipline, and I fully support the
direction, to focus on the correctness first.

Patch

diff --git a/fsmonitor.c b/fsmonitor.c
index 739ddbf7aca..ac638a61c00 100644
--- a/fsmonitor.c
+++ b/fsmonitor.c
@@ -5,6 +5,7 @@ 
 #include "ewah/ewok.h"
 #include "fsmonitor.h"
 #include "fsmonitor-ipc.h"
+#include "name-hash.h"
 #include "run-command.h"
 #include "strbuf.h"
 #include "trace2.h"
@@ -186,6 +187,102 @@  static int query_fsmonitor_hook(struct repository *r,
 static size_t handle_path_with_trailing_slash(
 	struct index_state *istate, const char *name, int pos);
 
+/*
+ * Use the name-hash to do a case-insensitive cache-entry lookup with
+ * the pathname and invalidate the cache-entry.
+ *
+ * Returns the number of cache-entries that we invalidated.
+ */
+static size_t handle_using_name_hash_icase(
+	struct index_state *istate, const char *name)
+{
+	struct cache_entry *ce = NULL;
+
+	ce = index_file_exists(istate, name, strlen(name), 1);
+	if (!ce)
+		return 0;
+
+	/*
+	 * A case-insensitive search in the name-hash using the
+	 * observed pathname found a cache-entry, so the observed path
+	 * is case-incorrect.  Invalidate the cache-entry and use the
+	 * correct spelling from the cache-entry to invalidate the
+	 * untracked-cache.  Since we now have sparse-directories in
+	 * the index, the observed pathname may represent a regular
+	 * file or a sparse-index directory.
+	 *
+	 * Note that we should not have seen FSEvents for a
+	 * sparse-index directory, but we handle it just in case.
+	 *
+	 * Either way, we know that there are not any cache-entries for
+	 * children inside the cone of the directory, so we don't need to
+	 * do the usual scan.
+	 */
+	trace_printf_key(&trace_fsmonitor,
+			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
+			 name, ce->name);
+
+	untracked_cache_invalidate_trimmed_path(istate, ce->name, 0);
+
+	ce->ce_flags &= ~CE_FSMONITOR_VALID;
+	return 1;
+}
+
+/*
+ * Use the dir-name-hash to find the correct-case spelling of the
+ * directory.  Use the canonical spelling to invalidate all of the
+ * cache-entries within the matching cone.
+ *
+ * Returns the number of cache-entries that we invalidated.
+ */
+static size_t handle_using_dir_name_hash_icase(
+	struct index_state *istate, const char *name)
+{
+	struct strbuf canonical_path = STRBUF_INIT;
+	int pos;
+	size_t len = strlen(name);
+	size_t nr_in_cone;
+
+	if (name[len - 1] == '/')
+		len--;
+
+	if (!index_dir_find(istate, name, len, &canonical_path))
+		return 0; /* name is untracked */
+
+	if (!memcmp(name, canonical_path.buf, canonical_path.len)) {
+		strbuf_release(&canonical_path);
+		/*
+		 * NEEDSWORK: Our caller already tried an exact match
+		 * and failed to find one.  They called us to do an
+		 * ICASE match, so we should never get an exact match,
+		 * so we could promote this to a BUG() here if we
+		 * wanted to.  It doesn't hurt anything to just return
+		 * 0 and go on becaus we should never get here.  Or we
+		 * could just get rid of the memcmp() and this "if"
+		 * clause completely.
+		 */
+		return 0; /* should not happen */
+	}
+
+	trace_printf_key(&trace_fsmonitor,
+			 "fsmonitor_refresh_callback MAP: '%s' '%s'",
+			 name, canonical_path.buf);
+
+	/*
+	 * The dir-name-hash only tells us the corrected spelling of
+	 * the prefix.  We have to use this canonical path to do a
+	 * lookup in the cache-entry array so that we repeat the
+	 * original search using the case-corrected spelling.
+	 */
+	strbuf_addch(&canonical_path, '/');
+	pos = index_name_pos(istate, canonical_path.buf,
+			     canonical_path.len);
+	nr_in_cone = handle_path_with_trailing_slash(
+		istate, canonical_path.buf, pos);
+	strbuf_release(&canonical_path);
+	return nr_in_cone;
+}
+
 /*
  * The daemon sent an observed pathname without a trailing slash.
  * (This is the normal case.)  We do not know if it is a tracked or
@@ -319,6 +416,19 @@  static void fsmonitor_refresh_callback(struct index_state *istate, char *name)
 	else
 		nr_in_cone = handle_path_without_trailing_slash(istate, name, pos);
 
+	/*
+	 * If we did not find an exact match for this pathname or any
+	 * cache-entries with this directory prefix and we're on a
+	 * case-insensitive file system, try again using the name-hash
+	 * and dir-name-hash.
+	 */
+	if (!nr_in_cone && ignore_case) {
+		nr_in_cone = handle_using_name_hash_icase(istate, name);
+		if (!nr_in_cone)
+			nr_in_cone = handle_using_dir_name_hash_icase(
+				istate, name);
+	}
+
 	if (nr_in_cone)
 		trace_printf_key(&trace_fsmonitor,
 				 "fsmonitor_refresh_callback CNT: %d",