From patchwork Wed Apr 20 20:42:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 28D70C433EF for ; Wed, 20 Apr 2022 20:43:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382129AbiDTUqR (ORCPT ); Wed, 20 Apr 2022 16:46:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382123AbiDTUqN (ORCPT ); Wed, 20 Apr 2022 16:46:13 -0400 Received: from mail-wm1-x333.google.com (mail-wm1-x333.google.com [IPv6:2a00:1450:4864:20::333]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B553138D8D for ; Wed, 20 Apr 2022 13:43:24 -0700 (PDT) Received: by mail-wm1-x333.google.com with SMTP id c190-20020a1c35c7000000b0038e37907b5bso4541917wma.0 for ; Wed, 20 Apr 2022 13:43:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aIMdZpuEsxrK+U6imb1bJeDjYFcR6uo/4kszjMrgtyw=; b=KVOMAx5Z5m2pvpikZLUdzDgzV09IqviE1Q3RTk9KYGe0JJj19tyxlPtl/fAVz1ciBy 486Y1cr1PhlftJ8l3hCq8bC0iPJMLqDQloY53MhR23X6+GAoH1ak0IC5apu1U7Qr0WIZ Qc6mRJ9k3khq4QULqtyTZr8TtJYA0qQeCr3JfCOvgJGiQneM1YbZsu2F0VO5bV3LemtE ZXx6GD+H9fLY/nV1slQw9mhWWm9Ulv2ZEBzNY+FQafujZpBOxE6XKnFBI0E3BznhUhn2 +QhS/shgsp+wKPNCDpn9CqpstVEMHdu5TEZ7TadwvPu+kARMzoXS5CV7QMc0B48CFnXZ SS6Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aIMdZpuEsxrK+U6imb1bJeDjYFcR6uo/4kszjMrgtyw=; b=ffBmYMQMyLQw8D589I1XF+xX0mR4scV/9GOvKG9ikkPfwsie2vhJD0yLdTUg8MmRq0 rmierEVXCf5ngiTReOXflf48J9LstdIYRVpzJqclBh9kPJ2yFMMu8k/Q1otgawJpjdAO itWMMgoCFrHRYI6MzmGImOV1hZnezpR+oe37XFYHuANrdKmvunpFS8wxbIYh6s80xpyv mk1qlPqsXvblmt5L9QZ5Z9rSt1fjuxErsxtFctV1coKAq3OG+eZ6lv3cs6S05OJZkAef FSncg/rSuMpFWRrOOyCugqENTKjub8RXjWQKEt6qHBRLCQUpBAmbXomPeqYgdvl3kYcx PrEQ== X-Gm-Message-State: AOAM532Seny2dIavK9JwwLjeqhhTNCwVHL/i1Skp71ZIqR8UvCOufvB9 dy3WwskPmu/wbC5dks4vQeu6s+bcUo4= X-Google-Smtp-Source: ABdhPJzaWJ2+c1rV0oSNryKzE1XVF7xDLWzEeXwvn8tIx/UY760YJkTcVyQHQkoAO9n1RlGfH7CGQQ== X-Received: by 2002:a05:600c:20f:b0:38e:b5a6:7b01 with SMTP id 15-20020a05600c020f00b0038eb5a67b01mr5309607wmi.168.1650487402624; Wed, 20 Apr 2022 13:43:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x4-20020adfdd84000000b00207b60ed68esm611337wrl.100.2022.04.20.13.43.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:21 -0700 (PDT) Message-Id: <8b7c5f4e234e5b139b640652fcfdafb2e24e9db8.1650487398.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:51 +0000 Subject: [PATCH v5 01/28] fsm-listen-win32: handle shortnames Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach FSMonitor daemon on Windows to recognize shortname paths as aliases of normal longname paths. FSMonitor clients, such as `git status`, should receive the longname spelling of changed files (when possible). Sometimes we receive FS events using the shortname, such as when a CMD shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1". The FS notification arrives using whatever combination of long and shortnames were used by the other process. (Shortnames do seem to be case normalized, however.) Use Windows GetLongPathNameW() to try to map the pathname spelling in the notification event into the normalized longname spelling. (This can fail if the file/directory is deleted, moved, or renamed, because we are asking the FS for the mapping in response to the event and after it has already happened, but we try.) Special case the shortname spelling of ".git" to avoid under-reporting these events. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++----- t/t7527-builtin-fsmonitor.sh | 65 +++++ 2 files changed, 374 insertions(+), 54 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 5b928ab66e5..3f1b68267bd 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -25,6 +25,9 @@ struct one_watch DWORD count; struct strbuf path; + wchar_t wpath_longname[MAX_PATH + 1]; + DWORD wpath_longname_len; + HANDLE hDir; HANDLE hEvent; OVERLAPPED overlapped; @@ -34,6 +37,21 @@ struct one_watch * need to later call GetOverlappedResult() and possibly CancelIoEx(). */ BOOL is_active; + + /* + * Are shortnames enabled on the containing drive? This is + * always true for "C:/" drives and usually never true for + * other drives. + * + * We only set this for the worktree because we only need to + * convert shortname paths to longname paths for items we send + * to clients. (We don't care about shortname expansion for + * paths inside a GITDIR because we never send them to + * clients.) + */ + BOOL has_shortnames; + BOOL has_tilda; + wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; struct fsmonitor_daemon_backend_data @@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data }; /* - * Convert the WCHAR path from the notification into UTF8 and - * then normalize it. + * Convert the WCHAR path from the event into UTF8 and normalize it. + * + * `wpath_len` is in WCHARS not bytes. */ -static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, +static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len, struct strbuf *normalized_path) { int reserve; int len = 0; strbuf_reset(normalized_path); - if (!info->FileNameLength) + if (!wpath_len) goto normalize; /* @@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, * sequence of 2 UTF8 characters. That should let us * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. */ - reserve = info->FileNameLength + 1; + reserve = 2 * wpath_len + 1; strbuf_grow(normalized_path, reserve); for (;;) { - len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, - info->FileNameLength / sizeof(WCHAR), + len = WideCharToMultiByte(CP_UTF8, 0, + wpath, wpath_len, normalized_path->buf, strbuf_avail(normalized_path) - 1, NULL, NULL); @@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, goto normalize; if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { error(_("[GLE %ld] could not convert path to UTF-8: '%.*ls'"), - GetLastError(), - (int)(info->FileNameLength / sizeof(WCHAR)), - info->FileName); + GetLastError(), (int)wpath_len, wpath); return -1; } @@ -98,6 +115,152 @@ normalize: return strbuf_normalize_path(normalized_path); } +/* + * See if the worktree root directory has shortnames enabled. + * This will help us decide if we need to do an expensive shortname + * to longname conversion on every notification event. + * + * We do not want to create a file to test this, so we assume that the + * root directory contains a ".git" file or directory. (Our caller + * only calls us for the worktree root, so this should be fine.) + * + * Remember the spelling of the shortname for ".git" if it exists. + */ +static void check_for_shortnames(struct one_watch *watch) +{ + wchar_t buf_in[MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + wchar_t *last_slash = NULL; + wchar_t *last_bslash = NULL; + wchar_t *last; + + /* build L"/.git" */ + wcscpy(buf_in, watch->wpath_longname); + wcscpy(buf_in + watch->wpath_longname_len, L".git"); + + if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH)) + return; + + last_slash = wcsrchr(buf_out, L'/'); + last_bslash = wcsrchr(buf_out, L'\\'); + if (last_slash > last_bslash) + last = last_slash + 1; + else if (last_bslash) + last = last_bslash + 1; + else + last = buf_out; + + if (!wcscmp(last, L".git")) + return; + + watch->has_shortnames = 1; + wcsncpy(watch->dotgit_shortname, last, + ARRAY_SIZE(watch->dotgit_shortname)); + + /* + * The shortname for ".git" is usually of the form "GIT~1", so + * we should be able to avoid shortname to longname mapping on + * every notification event if the source string does not + * contain a "~". + * + * However, the documentation for GetLongPathNameW() says + * that there are filesystems that don't follow that pattern + * and warns against this optimization. + * + * Lets test this. + */ + if (wcschr(watch->dotgit_shortname, L'~')) + watch->has_tilda = 1; +} + +enum get_relative_result { + GRR_NO_CONVERSION_NEEDED, + GRR_HAVE_CONVERSION, + GRR_SHUTDOWN, +}; + +/* + * Info notification paths are relative to the root of the watch. + * If our CWD is still at the root, then we can use relative paths + * to convert from shortnames to longnames. If our process has a + * different CWD, then we need to construct an absolute path, do + * the conversion, and then return the root-relative portion. + * + * We use the longname form of the root as our basis and assume that + * it already has a trailing slash. + * + * `wpath_len` is in WCHARS not bytes. + */ +static enum get_relative_result get_relative_longname( + struct one_watch *watch, + const wchar_t *wpath, DWORD wpath_len, + wchar_t *wpath_longname) +{ + wchar_t buf_in[2 * MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + DWORD root_len; + + /* Build L"/" */ + root_len = watch->wpath_longname_len; + wcsncpy(buf_in, watch->wpath_longname, root_len); + wcsncpy(buf_in + root_len, wpath, wpath_len); + buf_in[root_len + wpath_len] = 0; + + /* + * We don't actually know if the source pathname is a + * shortname or a longname. This routine allows either to be + * given as input. + */ + if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) { + /* + * The shortname to longname conversion can fail for + * various reasons, for example if the file has been + * deleted. (That is, if we just received a + * delete-file notification event and the file is + * already gone, we can't ask the file system to + * lookup the longname for it. Likewise, for moves + * and renames where we are given the old name.) + * + * Since deleting or moving a file or directory by its + * shortname is rather obscure, I'm going ignore the + * failure and ask the caller to report the original + * relative path. This seems kinder than failing here + * and forcing a resync. Besides, forcing a resync on + * every file/directory delete would effectively + * cripple monitoring. + * + * We might revisit this in the future. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (!wcscmp(buf_in, buf_out)) { + /* + * The path does not have a shortname alias. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (wcsncmp(buf_in, buf_out, root_len)) { + /* + * The spelling of the root directory portion of the computed + * longname has changed. This should not happen. Basically, + * it means that we don't know where (without recomputing the + * longname of just the root directory) to split out the + * relative path. Since this should not happen, I'm just + * going to let this fail and force a shutdown (because all + * subsequent events are probably going to see the same + * mismatch). + */ + return GRR_SHUTDOWN; + } + + /* Return the worktree root-relative portion of the longname. */ + + wcscpy(wpath_longname, buf_out + root_len); + return GRR_HAVE_CONVERSION; +} + void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); @@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, DWORD share_mode = FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; HANDLE hDir; - wchar_t wpath[MAX_PATH]; + DWORD len_longname; + wchar_t wpath[MAX_PATH + 1]; + wchar_t wpath_longname[MAX_PATH + 1]; if (xutftowcs_path(wpath, path) < 0) { error(_("could not convert to wide characters: '%s'"), path); @@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, return NULL; } + if (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) { + error(_("[GLE %ld] could not get longname of '%s'"), + GetLastError(), path); + CloseHandle(hDir); + return NULL; + } + + len_longname = wcslen(wpath_longname); + if (wpath_longname[len_longname - 1] != L'/' && + wpath_longname[len_longname - 1] != L'\\') { + wpath_longname[len_longname++] = L'/'; + wpath_longname[len_longname] = 0; + } + CALLOC_ARRAY(watch, 1); watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ @@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, strbuf_init(&watch->path, 0); strbuf_addstr(&watch->path, path); + wcscpy(watch->wpath_longname, wpath_longname); + watch->wpath_longname_len = len_longname; + watch->hDir = hDir; watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); @@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch) watch->is_active = FALSE; } +/* + * Process a single relative pathname event. + * Return 1 if we should shutdown. + */ +static int process_1_worktree_event( + struct string_list *cookie_list, + struct fsmonitor_batch **batch, + const struct strbuf *path, + enum fsmonitor_path_type t, + DWORD info_action) +{ + const char *slash; + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path->buf); + string_list_append(cookie_list, + slash ? slash + 1 : path->buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info_action == FILE_ACTION_REMOVED) || + (info_action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + return 1; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!*batch) + *batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(*batch, path->buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path->buf); + } + + return 0; +} + /* * Process filesystem events that happen anywhere (recursively) under the * root directory. For a normal working directory, this includes @@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) struct string_list cookie_list = STRING_LIST_INIT_DUP; struct fsmonitor_batch *batch = NULL; const char *p = watch->buffer; + wchar_t wpath_longname[MAX_PATH + 1]; /* * If the kernel gets more events than will fit in the kernel @@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) */ for (;;) { FILE_NOTIFY_INFORMATION *info = (void *)p; - const char *slash; + wchar_t *wpath = info->FileName; + DWORD wpath_len = info->FileNameLength / sizeof(WCHAR); enum fsmonitor_path_type t; + enum get_relative_result grr; + + if (watch->has_shortnames) { + if (!wcscmp(wpath, watch->dotgit_shortname)) { + /* + * This event exactly matches the + * spelling of the shortname of + * ".git", so we can skip some steps. + * + * (This case is odd because the user + * can "rm -rf GIT~1" and we cannot + * use the filesystem to map it back + * to ".git".) + */ + strbuf_reset(&path); + strbuf_addstr(&path, ".git"); + t = IS_DOT_GIT; + goto process_it; + } - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) - goto skip_this_path; - - t = fsmonitor_classify_path_workdir_relative(path.buf); - - switch (t) { - case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: - /* special case cookie files within .git */ - - /* Use just the filename of the cookie file. */ - slash = find_last_dir_sep(path.buf); - string_list_append(&cookie_list, - slash ? slash + 1 : path.buf); - break; - - case IS_INSIDE_DOT_GIT: - /* ignore everything inside of "/.git/" */ - break; + if (watch->has_tilda && !wcschr(wpath, L'~')) { + /* + * Shortnames on this filesystem have tildas + * and the notification path does not have + * one, so we assume that it is a longname. + */ + goto normalize_it; + } - case IS_DOT_GIT: - /* "/.git" was deleted (or renamed away) */ - if ((info->Action == FILE_ACTION_REMOVED) || - (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { - trace2_data_string("fsmonitor", NULL, - "fsm-listen/dotgit", - "removed"); + grr = get_relative_longname(watch, wpath, wpath_len, + wpath_longname); + switch (grr) { + case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */ + break; + case GRR_HAVE_CONVERSION: + wpath = wpath_longname; + wpath_len = wcslen(wpath); + break; + default: + case GRR_SHUTDOWN: goto force_shutdown; } - break; + } - case IS_WORKDIR_PATH: - /* queue normal pathname */ - if (!batch) - batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, path.buf); - break; +normalize_it: + if (normalize_path_in_utf8(wpath, wpath_len, &path) == -1) + goto skip_this_path; - case IS_GITDIR: - case IS_INSIDE_GITDIR: - case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: - default: - BUG("unexpected path classification '%d' for '%s'", - t, path.buf); - } + t = fsmonitor_classify_path_workdir_relative(path.buf); + +process_it: + if (process_1_worktree_event(&cookie_list, &batch, &path, t, + info->Action)) + goto force_shutdown; skip_this_path: if (!info->NextEntryOffset) @@ -382,6 +630,9 @@ force_shutdown: * Note that we DO NOT get filesystem events on the external * itself (it is not inside something that we are watching). In particular, * we do not get an event if the external is deleted. + * + * Also, we do not care about shortnames within the external , since + * we never send these paths to clients. */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { @@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state) const char *slash; enum fsmonitor_path_type t; - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) + if (normalize_path_in_utf8( + info->FileName, + info->FileNameLength / sizeof(WCHAR), + &path) == -1) goto skip_this_path; t = fsmonitor_classify_path_gitdir_relative(path.buf); @@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) if (!data->watch_worktree) goto failed; + check_for_shortnames(data->watch_worktree); + if (state->nr_paths_watching > 1) { data->watch_gitdir = create_watch(state, state->path_gitdir_watch.buf); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index bd0c952a116..1be21785162 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' ' test_must_fail git -C test_implicit_2 fsmonitor--daemon status ' +# File systems on Windows may or may not have shortnames. +# This is a volume-specific setting on modern systems. +# "C:/" drives are required to have them enabled. Other +# hard drives default to disabled. +# +# This is a crude test to see if shortnames are enabled +# on the volume containing the test directory. It is +# crude, but it does not require elevation like `fsutil`. +# +test_lazy_prereq SHORTNAMES ' + mkdir .foo && + test -d "FOO~1" +' + +# Here we assume that the shortname of ".git" is "GIT~1". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s" && + + git init test_implicit_1s && + + start_daemon -C test_implicit_1s && + + # renaming the .git directory will implicitly stop the daemon. + # this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + # this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}. + mv test_implicit_1s/.gitxyz test_implicit_1s/.git && + + test_must_fail git -C test_implicit_1s fsmonitor--daemon status +' + +# Here we first create a file with LONGNAME of "GIT~1" before +# we create the repo. This will cause the shortname of ".git" +# to be "GIT~2". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s2" && + + mkdir test_implicit_1s2 && + echo HELLO >test_implicit_1s2/GIT~1 && + git init test_implicit_1s2 && + + test_path_is_file test_implicit_1s2/GIT~1 && + test_path_is_dir test_implicit_1s2/GIT~2 && + + start_daemon -C test_implicit_1s2 && + + # renaming the .git directory will implicitly stop the daemon. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git && + + test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status +' + test_expect_success 'cannot start multiple daemons' ' test_when_finished "stop_daemon_delete_repo test_multiple" && From patchwork Wed Apr 20 20:42:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A53EC433FE for ; Wed, 20 Apr 2022 20:43:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382134AbiDTUqS (ORCPT ); Wed, 20 Apr 2022 16:46:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382122AbiDTUqN (ORCPT ); Wed, 20 Apr 2022 16:46:13 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6214393CD for ; Wed, 20 Apr 2022 13:43:25 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id l62-20020a1c2541000000b0038e4570af2fso2027111wml.5 for ; Wed, 20 Apr 2022 13:43:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=FYejdevIv/sNJ+Pm7nNh32hrd+w9z8t9/s1drm2KRzs=; b=eO3tXYjk5rpa5iFn2MkndjEykwzpvoVRKzD2dRI31gFJIE12oM8aMi4CbTIO6STZst fmZvPaDYJzaANZWnti8FEqLRtSarOUC3QjiuifeKqbQC6BbycUzuUvH2umfdE6jTzNkk sLwFUQaJ/UCnHjBSX6erl/wQOiABF1UgEcloTHa7NPrBycEK3qsvf25oqayivMfQ+lId uRtZ+Mb1fmCu+KXqrazgetNd0Aw8N0i7e1zmiatrhIszg0a+NNYKga8SGBlyZgIDZ1EJ AkNCcz+SsSYqxo9Cel1mlxUEBNUXB8TmG0KCewG0Eex0WLy+jhxcCZpZW6Kg9lAQ1lGE Ndyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=FYejdevIv/sNJ+Pm7nNh32hrd+w9z8t9/s1drm2KRzs=; b=NGHAhVJcUCb4TBP9h7z5KcZIkXvY1pojT1XWcSs54opHHCrILtdE194EHgdxMyN2vH zZUc9V6OidsNUnlOBLTa+9l8Tc6N4DpqQGVVyDrjSV12SdFA0IiHRdYSZ6hK6gSFjNjj 3/V5xbLE8Hbqr/QPArql6fQKuODcFK3wu+r5lxz+D3eMua8WD1CSoP4OrVIMSc/I3IPB gEcpIRLUa9R2V3XHh5Op8C+NwYhFvDa8EuS6xUUVuGF0se2MHlly9eVrq4GCtB1zcS+V RUZsQNYiO5T6DPG9OaxlBFAaMc8EKjCglTJEF+OcxT8z3cWhecqTDZ2pL3+umYP6Mj0h aW2A== X-Gm-Message-State: AOAM533SQpG4J9Fxto+nPCyUL51D+K8Ad6OGfQjrMvPZUvSDNSH+Hr52 hOEPDvU/Yr7kaJBzQW5dyk/Vgpqu8po= X-Google-Smtp-Source: ABdhPJwXrUheSyWgR3uKftzYw8LRNfTNsWfm/yJ59ob07QkmZpFEUmPoryTSiAvGL5pYuMOeNzU7Xw== X-Received: by 2002:a05:600c:3b91:b0:38f:f29a:202f with SMTP id n17-20020a05600c3b9100b0038ff29a202fmr5368955wms.35.1650487404104; Wed, 20 Apr 2022 13:43:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f7-20020a1c3807000000b0038ffac6f752sm308599wma.45.2022.04.20.13.43.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:23 -0700 (PDT) Message-Id: <5b246bec2470431898ddc74013f6fb325c2bc281.1650487398.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:52 +0000 Subject: [PATCH v5 02/28] t7527: test FSMonitor on repos with Unicode root paths Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create some test repos with UTF8 characters in the pathname of the root directory and verify that the builtin FSMonitor can watch them. This test is mainly for Windows where we need to avoid `*A()` routines. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 1be21785162..c9c7dd77e3c 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -671,4 +671,27 @@ do done done +# Test Unicode UTF-8 characters in the pathname of the working +# directory root. Use of "*A()" routines rather than "*W()" routines +# on Windows can sometimes lead to odd failures. +# +u1=$(printf "u_c3_a6__\xC3\xA6") +u2=$(printf "u_e2_99_ab__\xE2\x99\xAB") +u_values="$u1 $u2" +for u in $u_values +do + test_expect_success "Unicode in repo root path: $u" ' + test_when_finished "stop_daemon_delete_repo $u" && + + git init "$u" && + echo 1 >"$u"/file1 && + git -C "$u" add file1 && + git -C "$u" config core.fsmonitor true && + + start_daemon -C "$u" && + git -C "$u" status >actual && + grep "new file: file1" actual + ' +done + test_done From patchwork Wed Apr 20 20:42:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820793 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8EC15C433EF for ; Wed, 20 Apr 2022 20:43:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382151AbiDTUqV (ORCPT ); Wed, 20 Apr 2022 16:46:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58366 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382124AbiDTUqO (ORCPT ); Wed, 20 Apr 2022 16:46:14 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A93D3AA71 for ; Wed, 20 Apr 2022 13:43:27 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id ay36-20020a05600c1e2400b0038ebc885115so2768135wmb.1 for ; Wed, 20 Apr 2022 13:43:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8V1FoXCb86xqZ2XOaLWrGr4avTXqor6TX1WNeRZ9avw=; b=n7Fff8+FNYPUx1hpECFIH8aXbPgYVboN4WAbBlFFXR95H1nTTDzny0ebzMuIecUtwH gYRWgolg9iFYdehjsIqzVaXxnft7QWe1pyGZ9DmGe2jXMv8Kcmzn9v1OT4j4EY5oxEdx 3mlhtRKNeos5ddVhBpL5qXPyUZPDZ9d3Rw1KisB4FKJsHRjiNja93Cqg4Zjl7pHZiAep 0+uDMNR9HDX5bGuq3KWH02zI9DIM/5WF6LGmZ/IJNu5XtnxL5vCeEu/Z3HXNS1fAV0os u/6ys369rDvRk90cWZha9ZhZ7Mr7TQPd/GegAK7nhIBcXLcdrU955XicCXKyGNN8Sp2j iJiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8V1FoXCb86xqZ2XOaLWrGr4avTXqor6TX1WNeRZ9avw=; b=nkdgt4mJgkYh+sfrRPoW4An8v1rV8Px3YxQaBdq6jyXlRPL8dRNo2TCqrHwqsPdCsj nl+MT6af/r7AJvtIeQEp59OSCw84HH3+3AkjXRTg8u4st2jCkn5iZzL0hvabhjZlRCxV RnCkFBxIriy7E39IRFPssrKT00RrjTb+UKVbLYKVt4Vof4nTnpVpd/zdibM+STwHJFa/ rQ/ybCm3ptmkifynjcWH95O4sxgMsEXvT/3ChVkPrOgHsry8E7ayx0PtIULw+0GY+VVr tikGKmyUWR4V69heWOrCr76AHzQFeswSjZx3LTII/Pw6rKqWbZOkQTmlI3mbbnvn2s06 BjSw== X-Gm-Message-State: AOAM530pl89B+ggio/i4loDFsCWfsW0d1GDlU4frd/7m/Jcz0qGNxaz2 8pmYDlb8M8/eIpw+YodInVqB9ObVsnU= X-Google-Smtp-Source: ABdhPJzA1l8TlWV8YKig0vMXQYl1vs14wtcjf6gYo+Kuglxn7ybaBdbCWE0GrMGGHAHtfJ/viTs17w== X-Received: by 2002:a05:600c:a4b:b0:37b:ea2b:5583 with SMTP id c11-20020a05600c0a4b00b0037bea2b5583mr5394483wmq.139.1650487405711; Wed, 20 Apr 2022 13:43:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d5-20020a056000186500b0020a8688963bsm862285wri.89.2022.04.20.13.43.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:24 -0700 (PDT) Message-Id: <8a474d69999514b343ff576ca1d96180c22b403d.1650487398.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:53 +0000 Subject: [PATCH v5 03/28] t/helper/fsmonitor-client: create stress test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a stress test to hammer on the fsmonitor daemon. Create a client-side thread pool of n threads and have each of them make m requests as fast as they can. We do not currently inspect the contents of the response. We're only interested in placing a heavy request load on the daemon. This test is useful for interactive testing and various experimentation. For example, to place additional load on the daemon while another test is running. We currently do not have a test script that actually uses this helper. We might add such a test in the future. Signed-off-by: Jeff Hostetler --- t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c index 3062c8a3c2b..54a4856c48c 100644 --- a/t/helper/test-fsmonitor-client.c +++ b/t/helper/test-fsmonitor-client.c @@ -7,6 +7,8 @@ #include "cache.h" #include "parse-options.h" #include "fsmonitor-ipc.h" +#include "thread-utils.h" +#include "trace2.h" #ifndef HAVE_FSMONITOR_DAEMON_BACKEND int cmd__fsmonitor_client(int argc, const char **argv) @@ -79,20 +81,121 @@ static int do_send_flush(void) return 0; } +struct hammer_thread_data +{ + pthread_t pthread_id; + int thread_nr; + + int nr_requests; + const char *token; + + int sum_successful; + int sum_errors; +}; + +static void *hammer_thread_proc(void *_hammer_thread_data) +{ + struct hammer_thread_data *data = _hammer_thread_data; + struct strbuf answer = STRBUF_INIT; + int k; + int ret; + + trace2_thread_start("hammer"); + + for (k = 0; k < data->nr_requests; k++) { + strbuf_reset(&answer); + + ret = fsmonitor_ipc__send_query(data->token, &answer); + if (ret < 0) + data->sum_errors++; + else + data->sum_successful++; + } + + strbuf_release(&answer); + trace2_thread_exit(); + return NULL; +} + +/* + * Start a pool of client threads that will each send a series of + * commands to the daemon. + * + * The goal is to overload the daemon with a sustained series of + * concurrent requests. + */ +static int do_hammer(const char *token, int nr_threads, int nr_requests) +{ + struct hammer_thread_data *data = NULL; + int k; + int sum_join_errors = 0; + int sum_commands = 0; + int sum_errors = 0; + + if (!token || !*token) + token = get_token_from_index(); + if (nr_threads < 1) + nr_threads = 1; + if (nr_requests < 1) + nr_requests = 1; + + CALLOC_ARRAY(data, nr_threads); + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + p->thread_nr = k; + p->nr_requests = nr_requests; + p->token = token; + + if (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) { + warning("failed to create thread[%d] skipping remainder", k); + nr_threads = k; + break; + } + } + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + + if (pthread_join(p->pthread_id, NULL)) + sum_join_errors++; + sum_commands += p->sum_successful; + sum_errors += p->sum_errors; + } + + fprintf(stderr, "HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\n", + nr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors); + + free(data); + + /* + * Return an error if any of the _send_query requests failed. + * We don't care about thread create/join errors. + */ + return sum_errors > 0; +} + int cmd__fsmonitor_client(int argc, const char **argv) { const char *subcmd; const char *token = NULL; + int nr_threads = 1; + int nr_requests = 1; const char * const fsmonitor_client_usage[] = { "test-tool fsmonitor-client query []", "test-tool fsmonitor-client flush", + "test-tool fsmonitor-client hammer [] [] []", NULL, }; struct option options[] = { OPT_STRING(0, "token", &token, "token", "command token to send to the server"), + + OPT_INTEGER(0, "threads", &nr_threads, "number of client threads"), + OPT_INTEGER(0, "requests", &nr_requests, "number of requests per thread"), + OPT_END() }; @@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv) if (!strcmp(subcmd, "flush")) return !!do_send_flush(); + if (!strcmp(subcmd, "hammer")) + return !!do_hammer(token, nr_threads, nr_requests); + die("Unhandled subcommand: '%s'", subcmd); } #endif From patchwork Wed Apr 20 20:42:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820794 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89E65C433FE for ; Wed, 20 Apr 2022 20:43:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382153AbiDTUqW (ORCPT ); Wed, 20 Apr 2022 16:46:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58378 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382127AbiDTUqP (ORCPT ); Wed, 20 Apr 2022 16:46:15 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 886A938D8D for ; Wed, 20 Apr 2022 13:43:28 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id w4so3826445wrg.12 for ; Wed, 20 Apr 2022 13:43:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XEOJPlnBxdyHowBZdh3y4jZnw2C/J1d1GlvxDT+qV1c=; b=Yu8SNyL6RmGKK5hZjWU2AtOrvkeudmNaSOr1vSGy1yDsnHWdQx5FgU7c3bfL2hYSuQ +XE2A9X/pPY6N99qWoqXGrAhmSQPfbomrXucS+/Gy3NMm+OMt7w/lpdTyEaf/FJqdLzj 4n4NZxKr21vDTV38WwIYylh1UKwhdy57B9jHG0fys5zRierXSSQfWOnUzcD9pxW7UomD A3Oo+wYHQcQ5uNA8eTlRCHTJ+zBGwVXjeePXzXknMLprk0RIduF6I8xNxrCtUDDUARn3 TVG+rprO0TNOrR36x+lGM59uSj6osZ09pPmhBej4N2caqawfQNyMbHKVxL1kWf8YAeC4 f8Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XEOJPlnBxdyHowBZdh3y4jZnw2C/J1d1GlvxDT+qV1c=; b=Eeeq0SSStmygZNv2nXux0yg2wfZ6pp0XMcv3xL1vL+gk8I/DQ5rvOewq1EluHJSKxO iqtFx510nGnGq3p6Kb4NQx30V6rQSzF/qHg6bdJ58VjuNg05ZjEQ8m7N+xctvMhxjjYa 0H3iD4oNOclbV+3lNvAPQX5xvRhjlLWQvQHtpqeMKbFHf6nkJJ4gfS58o0X6IXXmeTZM 2wCKVchkww/0rUMuB8hUuDfYLNUwb3+cYmYn4oYaHbBmdP9yRN+QwkPT/LfbSEOWKnvQ 5t2hiYpOCbGxQWbExvqB4pOil1Fa88RImGKM/aWoNlpG84CcUeGUCNiVrShs/r8IXgpv u0SA== X-Gm-Message-State: AOAM533jh1mG7FGEgRQqVySBpPmHUmnHDUdFrk0my2OS9H0JWJTwO+M7 rHVJrBj+rxaskVo96GGClbgvZH1rJ9U= X-Google-Smtp-Source: ABdhPJzs3Syf64KliF9NpAZM1fb2gRIDrMDWtWZGjr9YpL4XDqvTzr6OSi/1aNUC+JyKO8OdaM1I5g== X-Received: by 2002:adf:d235:0:b0:207:9c11:c8c7 with SMTP id k21-20020adfd235000000b002079c11c8c7mr16662267wrh.443.1650487406719; Wed, 20 Apr 2022 13:43:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u7-20020a05600c19c700b003928959f92dsm383615wmq.2.2022.04.20.13.43.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:26 -0700 (PDT) Message-Id: <004b67b62e3717c9daeb6dcda1e9d84d44fa4984.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:54 +0000 Subject: [PATCH v5 04/28] fsmonitor-settings: bare repos are incompatible with FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Bare repos do not have a worktree, so there is nothing for the daemon watch. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 6 ++++ builtin/update-index.c | 4 +++ fsmonitor-settings.c | 60 +++++++++++++++++++++++++++++++++++++ fsmonitor-settings.h | 12 ++++++++ t/t7519-status-fsmonitor.sh | 23 ++++++++++++++ 5 files changed, 105 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 46be55a4618..50ae3cca575 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1449,6 +1449,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) die(_("invalid 'ipc-threads' value (%d)"), fsmonitor__ipc_threads); + prepare_repo_settings(the_repository); + fsm_settings__set_ipc(the_repository); + + if (fsm_settings__error_if_incompatible(the_repository)) + return 1; + if (!strcmp(subcmd, "start")) return !!try_to_start_background_daemon(); diff --git a/builtin/update-index.c b/builtin/update-index.c index 876112abb21..d29048f16f2 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1237,6 +1237,10 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) if (fsmonitor > 0) { enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r); + + if (fsm_settings__error_if_incompatible(the_repository)) + return 1; + if (fsm_mode == FSMONITOR_MODE_DISABLED) { warning(_("core.fsmonitor is unset; " "set it if you really want to " diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 757d230d538..86c09bd35fe 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -9,9 +9,33 @@ */ struct fsmonitor_settings { enum fsmonitor_mode mode; + enum fsmonitor_reason reason; char *hook_path; }; +static void set_incompatible(struct repository *r, + enum fsmonitor_reason reason) +{ + struct fsmonitor_settings *s = r->settings.fsmonitor; + + s->mode = FSMONITOR_MODE_INCOMPATIBLE; + s->reason = reason; +} + +static int check_for_incompatible(struct repository *r) +{ + if (!r->worktree) { + /* + * Bare repositories don't have a working directory and + * therefore have nothing to watch. + */ + set_incompatible(r, FSMONITOR_REASON_BARE); + return 1; + } + + return 0; +} + static void lookup_fsmonitor_settings(struct repository *r) { struct fsmonitor_settings *s; @@ -23,6 +47,7 @@ static void lookup_fsmonitor_settings(struct repository *r) CALLOC_ARRAY(s, 1); s->mode = FSMONITOR_MODE_DISABLED; + s->reason = FSMONITOR_REASON_OK; r->settings.fsmonitor = s; @@ -86,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r) lookup_fsmonitor_settings(r); + if (check_for_incompatible(r)) + return; + r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } @@ -97,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path) lookup_fsmonitor_settings(r); + if (check_for_incompatible(r)) + return; + r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK; FREE_AND_NULL(r->settings.fsmonitor->hook_path); r->settings.fsmonitor->hook_path = strdup(path); @@ -110,5 +141,34 @@ void fsm_settings__set_disabled(struct repository *r) lookup_fsmonitor_settings(r); r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED; + r->settings.fsmonitor->reason = FSMONITOR_REASON_OK; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } + +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + return r->settings.fsmonitor->reason; +} + +int fsm_settings__error_if_incompatible(struct repository *r) +{ + enum fsmonitor_reason reason = fsm_settings__get_reason(r); + + switch (reason) { + case FSMONITOR_REASON_OK: + return 0; + + case FSMONITOR_REASON_BARE: + error(_("bare repository '%s' is incompatible with fsmonitor"), + xgetcwd()); + return 1; + } + + BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'", + reason); +} diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index a4c5d7b4889..8654edf33d8 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -4,11 +4,20 @@ struct repository; enum fsmonitor_mode { + FSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */ FSMONITOR_MODE_DISABLED = 0, FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */ FSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */ }; +/* + * Incompatibility reasons. + */ +enum fsmonitor_reason { + FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ + FSMONITOR_REASON_BARE, +}; + void fsm_settings__set_ipc(struct repository *r); void fsm_settings__set_hook(struct repository *r, const char *path); void fsm_settings__set_disabled(struct repository *r); @@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r); enum fsmonitor_mode fsm_settings__get_mode(struct repository *r); const char *fsm_settings__get_hook_path(struct repository *r); +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r); +int fsm_settings__error_if_incompatible(struct repository *r); + struct fsmonitor_settings; #endif /* FSMONITOR_SETTINGS_H */ diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index a6308acf006..9a8e21c5608 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE ' test $ret -ne 1 ' +# Test that we detect and disallow repos that are incompatible with FSMonitor. +test_expect_success 'incompatible bare repo' ' + test_when_finished "rm -rf ./bare-clone actual expect" && + git init --bare bare-clone && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=foo \ + update-index --fsmonitor 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=true \ + update-index --fsmonitor 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual +' + +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' + test_when_finished "rm -rf ./bare-clone actual" && + git init --bare bare-clone && + test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Wed Apr 20 20:42:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820795 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97228C433F5 for ; Wed, 20 Apr 2022 20:43:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382162AbiDTUq1 (ORCPT ); Wed, 20 Apr 2022 16:46:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382131AbiDTUqR (ORCPT ); Wed, 20 Apr 2022 16:46:17 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C4D81393CD for ; Wed, 20 Apr 2022 13:43:29 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id r187-20020a1c44c4000000b0038ccb70e239so4510453wma.3 for ; Wed, 20 Apr 2022 13:43:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=77SaOZDqR5GCe8rGTE1gWxph/6mABa7uqFiGsLfjuL0=; b=T1BnLDxjQmuWVvRwReyQ410PxIg0fhRUgrTFn3ogtJWhnwws8glMFrAK88rTHGavA9 W4rsJTh40zVovKM2DzpQ6XCE5BJanNdhTcdK+tcBr3jprEllxVudb26V8mYZJsDFQA8e y6ScdiURp3WdAX+5JfVVHlp/ESoO1TqTwsVsJodce8+uVWSEWORekdZRaDwHPKw8s86Q uciNzq4Glhi9WWic2dlXOIKLv0tXf+yKmVJ26Yi3cuxIt9AKXYo8AEYJSTDhfRS+wReG RAEMHMtRHq8A+HfM5xH0mlRnUsf+CoFYx/ZDV0hZQ2PPpeuUFM6/qOld651HoehM4nmj kn0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=77SaOZDqR5GCe8rGTE1gWxph/6mABa7uqFiGsLfjuL0=; b=yjfT8RsVTSQiKfOX01Sql/DbyJxz0Hn99/wY6pF9YF1eBiaYaUgpzbKT0ZMbqFJtAy 19E6wjg5Bz6T4+yMIH032xYbPT6fB/MI8xP5Q+mNttvT1Mm1aWn3429NBhyByx7jWCMz QREWMyHtw1ul+VXJRrjuhE5ScKKCYlk/JMXIjko98BFEEBUjDhkGR3Sbu6AQ+cjankJR R7FFh0QQfusEHuperVejlP0/mGTl8znE5hlOf+5Pc3pB/1V0IlDICY5tG/EFSD080mYk CYecY4+gRmldfBj/ZNkR1kr21HEZ5BOQXxOfWnJm2EeLtSDvMF134FmToj0cvNvlJVKw YFCw== X-Gm-Message-State: AOAM533DKf0A9FAWI1vfRTMSVh1XSCyPXzPZ9HEz8TIVSlY2w7DosLB0 QHf0zZleBVwwoqwJIICk6C29JV6Byig= X-Google-Smtp-Source: ABdhPJy2G8Pgjn5OKI+mNL7KgSJOp+KJIsrARv2YPoyPlBBFSIiZHqd/Kf7hoQjlXFIpwLkU/yLM3Q== X-Received: by 2002:a05:600c:34c4:b0:38e:b628:da95 with SMTP id d4-20020a05600c34c400b0038eb628da95mr5355210wmq.150.1650487407946; Wed, 20 Apr 2022 13:43:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z18-20020a5d6552000000b00207b65f745bsm645629wrv.83.2022.04.20.13.43.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:27 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:55 +0000 Subject: [PATCH v5 05/28] fsmonitor-settings: stub in Win32-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend generic incompatibility checkout with platform-specific mechanism. Stub in Win32 version. In the existing fsmonitor-settings code we have a way to mark types of repos as incompatible with fsmonitor (whether via the hook and IPC APIs). For example, we do this for bare repos, since there are no files to watch. Extend this exclusion mechanism for platform-specific reasons. This commit just creates the framework and adds a stub for Win32. Signed-off-by: Jeff Hostetler --- Makefile | 13 +++++++++++++ compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++ config.mak.uname | 4 ++++ contrib/buildsystems/CMakeLists.txt | 3 +++ fsmonitor-settings.c | 12 ++++++++++++ fsmonitor-settings.h | 13 +++++++++++++ 6 files changed, 54 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-win32.c diff --git a/Makefile b/Makefile index daa21bed6c3..93604fe8ef7 100644 --- a/Makefile +++ b/Makefile @@ -475,6 +475,11 @@ all:: # `compat/fsmonitor/fsm-listen-.c` that implements the # `fsm_listen__*()` routines. # +# If your platform has OS-specific ways to tell if a repo is incompatible with +# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS +# to the "" of the corresponding `compat/fsmonitor/fsm-settings-.c` +# that implements the `fsm_os_settings__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o endif +ifdef FSMONITOR_OS_SETTINGS + COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE ifdef FSMONITOR_DAEMON_BACKEND @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ endif +ifdef FSMONITOR_OS_SETTINGS + @echo FSMONITOR_OS_SETTINGS=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_OS_SETTINGS)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index 501970902da..cf224768ad6 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index ee0d7257b77..16705da2000 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 86c09bd35fe..8ff55f8c3fd 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r) return 1; } +#ifdef HAVE_FSMONITOR_OS_SETTINGS + { + enum fsmonitor_reason reason; + + reason = fsm_os__incompatible(r); + if (reason != FSMONITOR_REASON_OK) { + set_incompatible(r, reason); + return 1; + } + } +#endif + return 0; } diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 8654edf33d8..4b35f051fb1 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -30,4 +30,17 @@ int fsm_settings__error_if_incompatible(struct repository *r); struct fsmonitor_settings; +#ifdef HAVE_FSMONITOR_OS_SETTINGS +/* + * Ask platform-specific code whether the repository is incompatible + * with fsmonitor (both hook and ipc modes). For example, if the working + * directory is on a remote volume and mounted via a technology that does + * not support notification events, then we should not pretend to watch it. + * + * fsm_os__* routines should considered private to fsm_settings__ + * routines. + */ +enum fsmonitor_reason fsm_os__incompatible(struct repository *r); +#endif /* HAVE_FSMONITOR_OS_SETTINGS */ + #endif /* FSMONITOR_SETTINGS_H */ From patchwork Wed Apr 20 20:42:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820797 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA6A5C433FE for ; Wed, 20 Apr 2022 20:43:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382170AbiDTUqa (ORCPT ); Wed, 20 Apr 2022 16:46:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382135AbiDTUqS (ORCPT ); Wed, 20 Apr 2022 16:46:18 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31F0338BEB for ; Wed, 20 Apr 2022 13:43:31 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id i20so3832523wrb.13 for ; Wed, 20 Apr 2022 13:43:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ekJrkqFg1yJzdNh+egoYJeP3EUXrY0TIhPEKqSMHHf8=; b=cuBSG0PH5STso8lS0e4nGX3ySg9ch393hik1HyzAWyZmZHCMj+YEPyftwZGDaSWzPE rP7VxfvMn1XWttEwDQbiibuXfd2cMq2hFdpjnuLijvc/MeaZYKOgJNcXlmtciMmrFv6G 2j2I9eZ8KKN/2rf+cOEBs3Np96hD6VGWaQwV4hFEYbmZPB01eCtfPhw64LjiyFkEISyd QJCWxVajnCK9dJlmLbRmOBGFitxJiWSgmKUMH73GXrlocChHIZ2etNytJZ94ogDxbvk7 pUZDtng3+C0Oaku+SHchoVfQLjAqFEALCqhkTfjr9UAsotSYsoFTJQqoakDAB67LY9tf TehQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ekJrkqFg1yJzdNh+egoYJeP3EUXrY0TIhPEKqSMHHf8=; b=5AyQj7EGqRRcXqapZJ2TXzFm+dtGlCWsiqVrUiNfoLbXaO0gxY8dShDyHwsRirtqaV 04giuABt+n1tEJdR50qwk8diWT5GuoEm6DVdQ9p4pDOkLZO1bWjkpDQu8b3JmDxdyusG HjopUa/rvhqvqJdzVWcPppF0CrIFkKXMTiIj3OIcOyJ8WluZhD8c26xUWC8l8gpNL+K4 cjdTBzaWT/sFejR74dDfNNoDPm+nzZjjfkadet/n7s2uRGCaZH1nOEGVOFYoSD2+SY+F H8qxojVyZ5UoRTnrmcXj672SANr8of6T83FSk1j6/Lt94lP92lJbEDOQmRQ9bKgg1C3x DG0Q== X-Gm-Message-State: AOAM533leUxEmhirIUYQURVApKrCs2P1/0GaLCwXetyRUD4WxGUIkzWH ZhOEQPCmAxnLOGz3J86kMZLBZGdO9Ek= X-Google-Smtp-Source: ABdhPJzexeC7oYhhZ489+IVP+htbwTFnZHBcqsNE9aF5F053h8TjgFIBJ/SqR5/Xga4s46xAIAsXuQ== X-Received: by 2002:adf:fb48:0:b0:203:f986:874a with SMTP id c8-20020adffb48000000b00203f986874amr16815067wrs.614.1650487409412; Wed, 20 Apr 2022 13:43:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r9-20020a05600c320900b0038f0894d80csm329872wmp.7.2022.04.20.13.43.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:28 -0700 (PDT) Message-Id: <2d68fc9a46ad66177d686e889f191dd252f1c964.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:56 +0000 Subject: [PATCH v5 06/28] fsmonitor-settings: VFS for Git virtual repos are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler VFS for Git virtual repositories are incompatible with FSMonitor. VFS for Git is a downstream fork of Git. It contains its own custom file system watcher that is aware of the virtualization. If a working directory is being managed by VFS for Git, we should not try to watch it because we may get incomplete results. We do not know anything about how VFS for Git works, but we do know that VFS for Git working directories contain a well-defined config setting. If it is set, mark the working directory as incompatible. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++ fsmonitor-settings.c | 5 +++++ fsmonitor-settings.h | 1 + t/t7519-status-fsmonitor.sh | 9 +++++++++ 4 files changed, 41 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index 7fce32a3c5b..ee78bba38e3 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -3,7 +3,33 @@ #include "repository.h" #include "fsmonitor-settings.h" +/* + * VFS for Git is incompatible with FSMonitor. + * + * Granted, core Git does not know anything about VFS for Git and we + * shouldn't make assumptions about a downstream feature, but users + * can install both versions. And this can lead to incorrect results + * from core Git commands. So, without bringing in any of the VFS for + * Git code, do a simple config test for a published config setting. + * (We do not look at the various *_TEST_* environment variables.) + */ +static enum fsmonitor_reason check_vfs4git(struct repository *r) +{ + const char *const_str; + + if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str)) + return FSMONITOR_REASON_VFS4GIT; + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_vfs4git(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 8ff55f8c3fd..1efb6e17a20 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -179,6 +179,11 @@ int fsm_settings__error_if_incompatible(struct repository *r) error(_("bare repository '%s' is incompatible with fsmonitor"), xgetcwd()); return 1; + + case FSMONITOR_REASON_VFS4GIT: + error(_("virtual repository '%s' is incompatible with fsmonitor"), + r->worktree); + return 1; } BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 4b35f051fb1..6361fcbf6b0 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -16,6 +16,7 @@ enum fsmonitor_mode { enum fsmonitor_reason { FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; void fsm_settings__set_ipc(struct repository *r); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 9a8e21c5608..156895f9437 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' grep "bare repository .* is incompatible with fsmonitor" actual ' +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' ' + test_when_finished "rm -rf ./fake-virtual-clone actual" && + git init fake-virtual-clone && + test_must_fail git -C ./fake-virtual-clone \ + -c core.virtualfilesystem=true \ + fsmonitor--daemon run 2>actual && + grep "virtual repository .* is incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Wed Apr 20 20:42:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820796 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83F68C433F5 for ; Wed, 20 Apr 2022 20:43:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382166AbiDTUq2 (ORCPT ); Wed, 20 Apr 2022 16:46:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382137AbiDTUqT (ORCPT ); Wed, 20 Apr 2022 16:46:19 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2C03E38D8D for ; Wed, 20 Apr 2022 13:43:32 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id x18so3896749wrc.0 for ; Wed, 20 Apr 2022 13:43:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=Xd3T1wtfrkOobolx0CZMbgpl6r/AExFm+3TfWFlIWLetI/bVBWr9ga85j5gVYwAr46 T5vos5qt6TA/QoHwSvOl+MmZgWZgZEa3M1QDI7b/em8rHqRygxCqOPsL3vGm6e1smMQ1 F3JFOdF6fRrKUE1PlzVsdI17Do46G0UZkRjkt1+LqAgWvFU38aoo+aUKf3YIaQcqPQrE PNqK5e74eWSoUS782X3FlEg+Yw5Qv9Zm2SMoKL0bJrtMO+Cf+4Zu0PCLbtnaWbo0gnxl nvGbFNRTyJkgaU86QwQyhgkkEG9gGpdgGtR0tbmHN6otwaM1G5rqNyOzynV7jkq9yWlN 18eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=dQBLh0BsNFRpTMKO806aYfELt3aWRZIPvFwqyT6II2H66ZE5Bbn+5Cu11ud/FWXF+d LrMpQmg6/CQNBHmfEmzoBQghrPr/1bzZqFHXUqWI3hADGyZJKomNUn7iwER643XwmXBQ 5U87c7syMnaqqsPlMA2a0+8L3csRH2+SMyS1+XsMEFBTeXuZjDf83SmdUhdFfPcPSfGk fmN/1jHtDDmjcnE2pUEDW4z4WErWAyg541SzOS1F3lmrPDppAqdU6fycSqUhkTlnPYI/ 5qYNv4UXiu+5SoliNq3pZ8avBcN29KPlZ3IY66OOO78bZWgHFEdx9kduiYZyIEDoIDhm 8CFg== X-Gm-Message-State: AOAM532le0ViVWnz48LDwlGocG5EA/XZqNcVy2bZ1EScnOrkWrqTosBt yhP7UL5h2a0qpCxZ7kYYLsUaiTy5RRY= X-Google-Smtp-Source: ABdhPJwpEdgoYm+ZlgBdDSYSSbJ9Q2kP9IhMAmwwVxbq31o36b1ZUYqJr4TgMyt731qvJCusQ9YjwA== X-Received: by 2002:a5d:4ec1:0:b0:207:b1c0:a417 with SMTP id s1-20020a5d4ec1000000b00207b1c0a417mr16390232wrv.561.1650487410501; Wed, 20 Apr 2022 13:43:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w17-20020a5d6091000000b00207a89b952asm648193wrt.77.2022.04.20.13.43.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:30 -0700 (PDT) Message-Id: <94ae2e424f183344a19c76cc5bb481b64be3f31f.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:57 +0000 Subject: [PATCH v5 07/28] fsmonitor-settings: stub in macOS-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++ config.mak.uname | 1 + contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 13 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-darwin.c diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index cf224768ad6..cf911d141f2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin + FSMONITOR_OS_SETTINGS = darwin endif endif diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 16705da2000..b8f9f7a0388 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) endif() endif() From patchwork Wed Apr 20 20:42:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820798 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BFAFC433EF for ; Wed, 20 Apr 2022 20:43:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382172AbiDTUqc (ORCPT ); Wed, 20 Apr 2022 16:46:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382148AbiDTUqU (ORCPT ); Wed, 20 Apr 2022 16:46:20 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 454573AA71 for ; Wed, 20 Apr 2022 13:43:33 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id n40-20020a05600c3ba800b0038ff1939b16so2037662wms.2 for ; Wed, 20 Apr 2022 13:43:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=k5ZREbqCqHvt+JSRQlVPZDSjI/JECTPQcuz7v45GQYU=; b=gdy3AStzP/Gu8DGxGjM9hIWp6/MRh3CYQvcSlFFkE8+cXL39QXJ/oRmlalOEEbXz8I L//Ox90ngch0vUcUHFvqg0w4VBQyLQ1lwCHCi0wMsiorqO7LOp1hTZ01UmO0gyY0br90 s8QltV1bmF7EHkdYmXRE08Qi1Pc6A8i5TzydiKOb8spf+zoTTe13snWo/X9KKXLoYcL9 1OMIV3eqQGoDVbeLJI8bQ4DNMN+pWeMg8c6udtmyN+U6O8LfkPn2Qe03oEM8RruOVrmO o4kf4Wj0N5B3RpKONbrJjyv4EGjOYN39xh31X4xlHoKgZpTtL7K/y/eHA6pWjulLitqY 0uIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=k5ZREbqCqHvt+JSRQlVPZDSjI/JECTPQcuz7v45GQYU=; b=eE+0l9s9wNXEunZyJkKOode7tzwZrF2ups8HQQlxRX4KNDGtt4tyB5q4QF8De2eONA 2paykcW0HekQ853GXOxqjbqidOAxWYn+u2jzANl1txLKe/jQ8qwuMXYhDojihRD0zQFU YcXmShHzdI5vaZUqQgNDC905VoUy2VkBmSuLQpKVVn1r0CxwddJpqnCQIdU0Wq3XZirb X30iVQiB5FJhJFlG5jRb8wFNiphShfhIFH9TN3O8Y9xP/azqGL8rZdE10uXsY9ogpR7b HROWfvO0v11o5AuapgFQyO97WbLOY+u0Hq9hus9KsYeAcBfrtaiHdy7g/48OnmCpP1Ef dvzA== X-Gm-Message-State: AOAM530c02Z6Qdv2n1TYQPNeTlEi/t1NoemuKvfpx6ixC0x3MJXj6pwG 5FcPwJnrDS2KsR09TggJrik5LeNrJu0= X-Google-Smtp-Source: ABdhPJy9p0HIhyZiJ+UIUZNMiBtf/aJ7vnXmV3UgnEtgz+oXE5ZRYCWGTz3sdO5pcBCDfJWRnuFTlw== X-Received: by 2002:a05:600c:1c1f:b0:38e:c425:5b1a with SMTP id j31-20020a05600c1c1f00b0038ec4255b1amr5343870wms.69.1650487411437; Wed, 20 Apr 2022 13:43:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b1-20020a5d40c1000000b00207b49d3023sm640616wrq.44.2022.04.20.13.43.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:31 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:58 +0000 Subject: [PATCH v5 08/28] fsmonitor-settings: remote repos on macOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on macOS and mark them as incompatible with FSMonitor. With this, `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, like `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++ fsmonitor-settings.c | 10 ++++ fsmonitor-settings.h | 2 + 3 files changed, 78 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index 7fce32a3c5b..fdd762bf79d 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -2,8 +2,74 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" +#include +#include + +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type (NFS, SAMBA, etc.) dictates whether notification events + * are available at all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * For the builtin FSMonitor, we create the Unix domain socket for the + * IPC in the .git directory. If the working directory is remote, + * then the socket will be created on the remote file system. This + * can fail if the remote file system does not support UDS file types + * (e.g. smbfs to a Windows server) or if the remote kernel does not + * allow a non-local process to bind() the socket. (These problems + * could be fixed by moving the UDS out of the .git directory and to a + * well-known local directory on the client machine, but care should + * be taken to ensure that $HOME is actually local and not a managed + * file share.) + * + * So (for now at least), mark remote working directories as + * incompatible. + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + struct statfs fs; + + if (statfs(r->worktree, &fs) == -1) { + int saved_errno = errno; + trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s", + r->worktree, strerror(saved_errno)); + errno = saved_errno; + return FSMONITOR_REASON_ERROR; + } + + trace_printf_key(&trace_fsmonitor, + "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'", + r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename); + + if (!(fs.f_flags & MNT_LOCAL)) + return FSMONITOR_REASON_REMOTE; + + return FSMONITOR_REASON_OK; +} enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 1efb6e17a20..0a1811ff004 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -180,6 +180,16 @@ int fsm_settings__error_if_incompatible(struct repository *r) xgetcwd()); return 1; + case FSMONITOR_REASON_ERROR: + error(_("repository '%s' is incompatible with fsmonitor due to errors"), + r->worktree); + return 1; + + case FSMONITOR_REASON_REMOTE: + error(_("remote repository '%s' is incompatible with fsmonitor"), + r->worktree); + return 1; + case FSMONITOR_REASON_VFS4GIT: error(_("virtual repository '%s' is incompatible with fsmonitor"), r->worktree); diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 6361fcbf6b0..34391b583b3 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -16,6 +16,8 @@ enum fsmonitor_mode { enum fsmonitor_reason { FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ + FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; From patchwork Wed Apr 20 20:42:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820799 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF890C433F5 for ; Wed, 20 Apr 2022 20:43:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382174AbiDTUqf (ORCPT ); Wed, 20 Apr 2022 16:46:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382127AbiDTUqZ (ORCPT ); Wed, 20 Apr 2022 16:46:25 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B80F3BFB0 for ; Wed, 20 Apr 2022 13:43:34 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id m14so3860506wrb.6 for ; Wed, 20 Apr 2022 13:43:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=CNWW6UCxIwoeBVVJFQxyjOqdpmsTHUdQ6F0G8WfYYGdvMlrAjyyi0Dqa9osKXrWfIS YRn9fZbyQVAmzJCC42b1mh97KRiRXc45dk7BjFbEorQTWLidJ/QNREleL6+gUSLmf4tK wUcB7vdxeQCRqNhXrR3Pw8qW+JKVr9krlfXgcjr8r0qCX/AqNX/m/aM78FwwhViMGcDx 4CObwiBQcVo1XvvLumku/BBaVvUgQnMv3wpASQzhMYFLQAcUc71FOuEW4A2ly+cxPpEk CZjayexPbZKYoipw8sGnyBVHZYBIUEyZx2dM8hPU+w8oi1JXAafojwgG6SjQgmqfE5Pm 169Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=H/KSOWdI37j3VkB6BSYBSUq+zmKB5xDhoOFCNdn1j0q7RMpOprJ4bpm9wFqOl/M33U d5xmvZRPpu9kJms6cRLSRhjIDbgvkE176g98Zm97X90qf5PpfF15y4InhuF7w1ctDpCc vc0hJgUVg2MvHTfQvmKeQNUvAo7bYAFNwNyeDEV3nHumwguoqh6gretnrli+Pqae2WwB w9k+rxhK5HDYZny/3PqNC2k8bzyDjdSUt0XGO/E5A20kwysFiN/vopcahPtFmRhDdj8S 1NSfBeAmJI/VP329BD7PviVB+1vp5sBxam+YAJT3tCr2gTjjeA3a9mwSE4SM0MoVftI0 RTvw== X-Gm-Message-State: AOAM532iMoYXUFoUb5UGbNRrP/ZJjTr99UQeWUPC0fjqf8jCZzJEYWzX JQzNT5EqLZTh4pxHBRShL5ntYo9kNTU= X-Google-Smtp-Source: ABdhPJyESJ+j4xWr6d02aitY2Zulkjtd35WPvbG3PWFiFANdlGcJCBBtngjG13K+fg7852aPByl6Tg== X-Received: by 2002:adf:f2cb:0:b0:20a:77c2:3958 with SMTP id d11-20020adff2cb000000b0020a77c23958mr16440071wrp.589.1650487412811; Wed, 20 Apr 2022 13:43:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o10-20020a5d47ca000000b0020a992ce354sm725576wrc.76.2022.04.20.13.43.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:32 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Apr 2022 20:42:59 +0000 Subject: [PATCH v5 09/28] fsmonitor-settings: remote repos on Windows are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on Windows and mark them as incompatible with FSMonitor. With this `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, such as `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index ee78bba38e3..907655720bb 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -2,6 +2,7 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" /* * VFS for Git is incompatible with FSMonitor. @@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r) return FSMONITOR_REASON_OK; } +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type dictates whether notification events are available at + * all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * So (for now at least), mark remote working directories as + * incompatible. + * + * Notes for testing: + * + * (a) Windows allows a network share to be mapped to a drive letter. + * (This is the normal method to access it.) + * + * $ NET USE Z: \\server\share + * $ git -C Z:/repo status + * + * (b) Windows allows a network share to be referenced WITHOUT mapping + * it to drive letter. + * + * $ NET USE \\server\share\dir + * $ git -C //server/share/repo status + * + * (c) Windows allows "SUBST" to create a fake drive mapping to an + * arbitrary path (which may be remote) + * + * $ SUBST Q: Z:\repo + * $ git -C Q:/ status + * + * (d) Windows allows a directory symlink to be created on a local + * file system that points to a remote repo. + * + * $ mklink /d ./link //server/share/repo + * $ git -C ./link status + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + wchar_t wpath[MAX_PATH]; + wchar_t wfullpath[MAX_PATH]; + size_t wlen; + UINT driveType; + + /* + * Do everything in wide chars because the drive letter might be + * a multi-byte sequence. See win32_has_dos_drive_prefix(). + */ + if (xutftowcs_path(wpath, r->worktree) < 0) + return FSMONITOR_REASON_ERROR; + + /* + * GetDriveTypeW() requires a final slash. We assume that the + * worktree pathname points to an actual directory. + */ + wlen = wcslen(wpath); + if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') { + wpath[wlen++] = L'\\'; + wpath[wlen] = 0; + } + + /* + * Normalize the path. If nothing else, this converts forward + * slashes to backslashes. This is essential to get GetDriveTypeW() + * correctly handle some UNC "\\server\share\..." paths. + */ + if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL)) + return FSMONITOR_REASON_ERROR; + + driveType = GetDriveTypeW(wfullpath); + trace_printf_key(&trace_fsmonitor, + "DriveType '%s' L'%ls' (%u)", + r->worktree, wfullpath, driveType); + + if (driveType == DRIVE_REMOTE) { + trace_printf_key(&trace_fsmonitor, + "check_remote('%s') true", + r->worktree); + return FSMONITOR_REASON_REMOTE; + } + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; @@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) if (reason != FSMONITOR_REASON_OK) return reason; + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } From patchwork Wed Apr 20 20:43:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820800 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD019C433F5 for ; Wed, 20 Apr 2022 20:44:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382185AbiDTUqr (ORCPT ); Wed, 20 Apr 2022 16:46:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382154AbiDTUqZ (ORCPT ); Wed, 20 Apr 2022 16:46:25 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD03F3CA63 for ; Wed, 20 Apr 2022 13:43:35 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id x18so3896929wrc.0 for ; Wed, 20 Apr 2022 13:43:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KKDUtqfcz4RAKt3TKC0Ge43bGODedopwH2p3XKXOALk=; b=Dm7uMXHOkqinmHTSSv8nALsR7tJdpjxbQsdCOXL4cd/8RxS5JLaoeXrOALnln1Ya4S Y3HukD3508zK8EhY+tojkJn4lrPKxWh5OluXJ4qXx7A5Irf402e/0vGSWcwnOFPwiIE0 KUejlgV6AKNDNMb0mQJ3cbXfjL3/DsQBH/PJZ3yHMjIhDkcvJaAp56zKX51NP97I0dzT DO33vNVXAFcb/qB/lJD8+RSPxYOVkMdb8FwX9nGg36YV6cNqE1eAIXwl+YBmOE7s1CLD FyuSq8BsCCzHEOVhZeXGEazwGox1v1gr9LSMc6gUhWb9fvxDEO52GQJSXuY29nIX9x8d WbtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KKDUtqfcz4RAKt3TKC0Ge43bGODedopwH2p3XKXOALk=; b=mYnm6WH+pDzzIZHcdJT6RW3cVRGULCnfHeIf6nvGPoc7CunPF8RitLl/84tXbY2FxX 9jxxKCUi5uOXsPyk8D5E9GuJiuJ2Ucc7BjzyguYOxnrwALpSyiGuqs+lC0chbhmL80Rz /GzoehyePaV6yljePmTxntu8tIP0G5JOAsZj6YfGhD95QlBGe+HjQEHqVqJafxfvo+Nb u5Hoavf8Lmnj5jXO+ecLiIR2Dyhm7mC6HvqM/pC4KCmIRsbqRQfBhXp3Q2xZOKgaCtRg OBcZoI2bf51kefHLt18C2fK11HHf4DQqPGltV38wIPniyE9KFtWWjUQ9dL2dIOqFCpsr 4tpw== X-Gm-Message-State: AOAM5333VOYNc/uy4Dja4opoCx/KikqCn3oygw4O22/Aa1iYn8VK64TQ ezHAdoA28C2OHTIq2SFhQxcZCdFQyBc= X-Google-Smtp-Source: ABdhPJx+y1+QGWXrqOUWzNlWNvPtrooi2aywjkyR6P8jk71FeamW/kteFXZQ87Ekjeh3Nz8uGDGDKQ== X-Received: by 2002:adf:8122:0:b0:1e7:b111:3b92 with SMTP id 31-20020adf8122000000b001e7b1113b92mr17029290wrm.695.1650487414179; Wed, 20 Apr 2022 13:43:34 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v11-20020a056000144b00b0020a9c02f60dsm634166wrx.50.2022.04.20.13.43.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:33 -0700 (PDT) Message-Id: <8f1f4840751892708de7f2e4ad14c13649943e25.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:00 +0000 Subject: [PATCH v5 10/28] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler On MacOS mark repos on NTFS or FAT32 volumes as incompatible. The builtin FSMonitor used Unix domain sockets on MacOS for IPC with clients. These sockets are kept in the .git directory. Unix sockets are not supported by NTFS and FAT32, so the daemon cannot start up. Test for this during our compatibility checking so that client commands do not keep trying to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++--- fsmonitor-settings.c | 5 +++++ fsmonitor-settings.h | 1 + 3 files changed, 23 insertions(+), 3 deletions(-) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index fdd762bf79d..efc732c0f31 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -7,7 +7,7 @@ #include /* - * Remote working directories are problematic for FSMonitor. + * [1] Remote working directories are problematic for FSMonitor. * * The underlying file system on the server machine and/or the remote * mount type (NFS, SAMBA, etc.) dictates whether notification events @@ -40,8 +40,16 @@ * * So (for now at least), mark remote working directories as * incompatible. + * + * + * [2] FAT32 and NTFS working directories are problematic too. + * + * The builtin FSMonitor uses a Unix domain socket in the .git + * directory for IPC. These Windows drive formats do not support + * Unix domain sockets, so mark them as incompatible for the daemon. + * */ -static enum fsmonitor_reason check_remote(struct repository *r) +static enum fsmonitor_reason check_volume(struct repository *r) { struct statfs fs; @@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r) if (!(fs.f_flags & MNT_LOCAL)) return FSMONITOR_REASON_REMOTE; + if (!strcmp(fs.f_fstypename, "msdos")) /* aka FAT32 */ + return FSMONITOR_REASON_NOSOCKETS; + + if (!strcmp(fs.f_fstypename, "ntfs")) + return FSMONITOR_REASON_NOSOCKETS; + return FSMONITOR_REASON_OK; } @@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; - reason = check_remote(r); + reason = check_volume(r); if (reason != FSMONITOR_REASON_OK) return reason; diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 0a1811ff004..60d5eaee497 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -194,6 +194,11 @@ int fsm_settings__error_if_incompatible(struct repository *r) error(_("virtual repository '%s' is incompatible with fsmonitor"), r->worktree); return 1; + + case FSMONITOR_REASON_NOSOCKETS: + error(_("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets"), + r->worktree); + return 1; } BUG("Unhandled case in fsm_settings__error_if_incompatible: '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 34391b583b3..23d5676c8c8 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -19,6 +19,7 @@ enum fsmonitor_reason { FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ + FSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */ }; void fsm_settings__set_ipc(struct repository *r); From patchwork Wed Apr 20 20:43:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820801 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 49134C433EF for ; Wed, 20 Apr 2022 20:44:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382163AbiDTUqs (ORCPT ); Wed, 20 Apr 2022 16:46:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382157AbiDTUq0 (ORCPT ); Wed, 20 Apr 2022 16:46:26 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E37EA3DA46 for ; Wed, 20 Apr 2022 13:43:36 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id v64-20020a1cac43000000b0038cfd1b3a6dso4504456wme.5 for ; Wed, 20 Apr 2022 13:43:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=E/aeU+tyQeU64OOCduUCtqneaVwXgKVG0aEJ6Tle/oGlKs1K2NbX/SZJWbfBCNRpxp LtJz3PyHlCIXh3oQUtLXSCz4c05IimMohpGxorpKAco9qZwxf0aXLmDRv+/spJ7Xdwe5 LSVy24gTi0eWgzPkdPXR5YNLpXSCYu2rngGt/L8u/IprVixbVCPdv2sDtNxncAHCgjCD tSBTEzKU/hxPSJx/bD8EMfp1GPaAtpe+QuD1cS1M7qQtx4I29JWVtfHL2yhVZd/Dsclr 0xYBWFCUyeQmhUXJctNKMDVqxm4zOdZdZdb/K9fFRxVsa1a9p71/PqKgcSPytQaquzDG 99kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=ptWB5lQcmYs4e9IIcntAFGv1oJW2fbP2WXaeeJMtSS0e4DqBPYh9DIcI6I7qcxPVw+ vwppzPMAnMcPlxJWrD9r8quYD19LCMWK9RafWJUh+d9GgAqFQZUYmDUfagCLvBmYAJZp H9ZBHSfk7st7Lkmhsl6sF99V2Z+lsZtrP/S+XShHCSwuoquo6yIFLYD3shWPzwtjdXJQ uWvdeAGB1ciL3iyb+w3FMmJMLr12+esFZ0VbRGsQVN0fnctoSGIA9Rx/gd8FLKTb9Z8m w0Vepokw+3ogvWu8mQQbzfwwlZpwzBVRXrwM2B/Tr1D+1RNKvVsJc25mlK5ODOAAP1KJ c48A== X-Gm-Message-State: AOAM531kEHc4HRNaMWEQsbQOVIz5lipJqkntgEYUqO5K5PAGjb7ZxbZA f9E6kPZzDE0qBkIEK3Qt9xt4+kC+h9s= X-Google-Smtp-Source: ABdhPJwWeD3yrk5JGRVjlv6P/uicmvnzMqg+9PjKliYXyafp6EoqPbkrT37JIlsJJjTMld05jFF+wQ== X-Received: by 2002:a05:600c:1d8b:b0:392:ab4f:365d with SMTP id p11-20020a05600c1d8b00b00392ab4f365dmr4770106wms.113.1650487415220; Wed, 20 Apr 2022 13:43:35 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m188-20020a1c26c5000000b0039187bb7e9asm316128wmm.6.2022.04.20.13.43.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:34 -0700 (PDT) Message-Id: <8d48d9c562369a29c1488dfcf7bd51a58158236a.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:01 +0000 Subject: [PATCH v5 11/28] unpack-trees: initialize fsmonitor_has_run_once in o->result Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Initialize `o->result.fsmonitor_has_run_once` based upon value in `o->src_index->fsmonitor_has_run_once` to prevent a second fsmonitor query during the tree traversal and possibly getting a skewed view of the working directory. The checkout code has already talked to the fsmonitor and the traversal is updating the index as it traverses, so there is no need to query the fsmonitor. Signed-off-by: Jeff Hostetler --- unpack-trees.c | 1 + 1 file changed, 1 insertion(+) diff --git a/unpack-trees.c b/unpack-trees.c index 360844bda3a..888cff81f9c 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options o->result.fsmonitor_last_update = xstrdup_or_null(o->src_index->fsmonitor_last_update); + o->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once; /* * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries From patchwork Wed Apr 20 20:43:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820802 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A00FC433FE for ; Wed, 20 Apr 2022 20:44:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382194AbiDTUqt (ORCPT ); Wed, 20 Apr 2022 16:46:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382158AbiDTUq0 (ORCPT ); Wed, 20 Apr 2022 16:46:26 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A7813EB94 for ; Wed, 20 Apr 2022 13:43:38 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id x3so1959613wmj.5 for ; Wed, 20 Apr 2022 13:43:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5W2x8t68NoDAVIFeMr0BVtQdXxzEzVw8wanSKIA/+K4=; b=SOj+zjBvBhMGGNdaGo/b+v8qD7SNB0a3h9lauVK2Q1fWNxzLsUEjkVtHAjWvzOO9m1 dCphtGoLQ5Mr7yGTS0P9TsgBOliQuy6JCXDAYy/6xidSLFPwYPnTcgUPsnl7J/RKd5d8 8WAESZmzlIXvnfKNW7cAJPE3Wo9i2NSgde79t7/hxZ8Z6Dcqn3w180/cdQaJcDVpX03v z3Vog+qX+bEzzpTxCYo/zleDeUaBuecOxmE1xZrDklnbIspfQEpAMOOAKTCxnLGygDN0 fmAttQtyUlB5HhQziqYE1eVJ+KiBcMKEGnsO1dDf+DENw7PlBvyYgNSVxUC5CJAbCmkN B74g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5W2x8t68NoDAVIFeMr0BVtQdXxzEzVw8wanSKIA/+K4=; b=SLGwV40Jfr6X9drgwIursaDxg4okuMfv8Jl4RgxklRImBNDYQEsVxpR/pJLIuEWYtg RISJQJJqPY9W8wLZes1Vtr3f+g627/UUHyeTI5lUvgGSyfxotZVpsNg8nd9s0Zdi70Jb TB/ObiiSEtcswZ1+r6PjS0x/VSGudfekKENFG3bFFV3S8hiox2ummFX2lFlokRMlrL4m wPwZqzv5zteeUgSEARles5MB2Dga70qQE+xEzuivJMtVhaMm56/CNVfpZn0RhV73dX5x ePTdIFQJrr8oG7Zw4WpNXVxvuLsI4Vwbca/6HLKEqUWJGtavr++xLwAN5BHSKEuaAjXK yjNw== X-Gm-Message-State: AOAM5307kl4jvZ2OhQUmFDkemYaPjEE6VOS1opPjBKyVvWKyU3yv0kGL czuDaUnZUAOKpfajInkY8tuuqjK/Ci4= X-Google-Smtp-Source: ABdhPJz1Sq5zEGC2caOn3aZZX7eaDXHFzej8szRrx71bTgz+Ir/npipQKt4crPcD5KMlbCDeUOYtPw== X-Received: by 2002:a1c:35c2:0:b0:38e:c75d:90a3 with SMTP id c185-20020a1c35c2000000b0038ec75d90a3mr5320831wma.98.1650487416582; Wed, 20 Apr 2022 13:43:36 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i6-20020a0560001ac600b0020a93f75030sm694539wry.48.2022.04.20.13.43.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:35 -0700 (PDT) Message-Id: <088c7b3334c79679d605dd16dc90211c08320428.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:02 +0000 Subject: [PATCH v5 12/28] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Ignore FSEvents resulting from `xattr` changes. Git does not care about xattr's or changes to xattr's, so don't waste time collecting these events in the daemon nor transmitting them to clients. Various security tools add xattrs to files and/or directories, such as to mark them as having been downloaded. We should ignore these events since it doesn't affect the content of the file/directory or the normal meta-data that Git cares about. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 0741fe834c3..14105f45c18 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) if (flag & kFSEventStreamEventFlagItemCloned) strbuf_addstr(&msg, "ItemCloned|"); - trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s", path, flag, msg.buf); strbuf_release(&msg); @@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef) ef & kFSEventStreamEventFlagUserDropped); } +/* + * If an `xattr` change is the only reason we received this event, + * then silently ignore it. Git doesn't care about xattr's. We + * have to be careful here because the kernel can combine multiple + * events for a single path. And because events always have certain + * bits set, such as `ItemIsFile` or `ItemIsDir`. + * + * Return 1 if we should ignore it. + */ +static int ef_ignore_xattr(const FSEventStreamEventFlags ef) +{ + static const FSEventStreamEventFlags mask = + kFSEventStreamEventFlagItemChangeOwner | + kFSEventStreamEventFlagItemCreated | + kFSEventStreamEventFlagItemFinderInfoMod | + kFSEventStreamEventFlagItemInodeMetaMod | + kFSEventStreamEventFlagItemModified | + kFSEventStreamEventFlagItemRemoved | + kFSEventStreamEventFlagItemRenamed | + kFSEventStreamEventFlagItemXattrMod | + kFSEventStreamEventFlagItemCloned; + + return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); +} + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_ignore_xattr(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "ignore-xattr: '%s', flags=0x%x", + path_k, event_flags[k]); + continue; + } + switch (fsmonitor_classify_path_absolute(state, path_k)) { case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: From patchwork Wed Apr 20 20:43:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820803 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FF29C433F5 for ; Wed, 20 Apr 2022 20:44:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382196AbiDTUqw (ORCPT ); Wed, 20 Apr 2022 16:46:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382160AbiDTUq0 (ORCPT ); Wed, 20 Apr 2022 16:46:26 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B1C140E7E for ; Wed, 20 Apr 2022 13:43:39 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id t6so427592wra.4 for ; Wed, 20 Apr 2022 13:43:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7rcq2e2OM4nGjZ+ZqucdxU8VZqK8SlixUVA4duCOPBc=; b=JA4+HxvbB5JqAHpBHe0fEEk3UuoMmVUp9fSjGD4AdxwoELXQno1X1qyzYvUq+S07Qq O0k07tuPHk1P/PyktlhPINVaGw9V49AFBfeSH1hB+2I8RZpBHMLKii2raUOhb2dGgUDd DSY2bq99OfVeBoeUO16rsvyvMXFncFHmx0qg5EQFl0Q7vHT3GFooiykIqyd+PLQPVkWr ZqBgEAUkPSWf3eZk008m7cuZ/jplwLHSAQmGjAl1sRx+eXrQmYTuH+EdK+o3sMgX2sDi SdPWy9yz84q4oY6ak9dfgAwfpg/bvXUeCkupnv6NKT1b+r++PlgixcnJRT29RZsIXZZE Ev8A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7rcq2e2OM4nGjZ+ZqucdxU8VZqK8SlixUVA4duCOPBc=; b=Xis+jbMqtS69xSY5ZGHf2sWS0lymRXtTQBzi1E9Y76Be1m3A3k8ax13VslyhprSmzZ k5Tl0oJRrkGbSR+E82raNQnFYoODr7Zymy+4uunk6t7O6tnOmOaYxL7vGqgrwhiWteWQ HsA3YIIEYGmDTZr8SQfqgobDmRrDdn3qttdmk8lkuLrbICfIgqROqHx627mxOl7mbG3x 8jpuUsAX8+F9yUcQ3Y5gP5y7gzUHu38JLzBD4ZcbpgktIDEPIAwRSXeueshot8oFBr29 KWEQUnfTooUCFgTtc2tHh/TSMjKWmbql2qo6RCEK/Gd2aqTmgeND0yZpCx2S2bAVKsQc 8rtg== X-Gm-Message-State: AOAM531q0W0v/KnFUdp+2TtIfAsfMHr9AFdUEYufqR7rWRYr0Y4sY2vE fElGsIwM62y7eNTBplwv93r66dvJBbY= X-Google-Smtp-Source: ABdhPJy4ALMvzgTvKpdj5Rd4GKaWu40euYsFyC5JRkC+X62uuJ3Mh95EVZmDQxc5E3YD67DtBP2v6Q== X-Received: by 2002:a5d:510a:0:b0:209:16c7:d8 with SMTP id s10-20020a5d510a000000b0020916c700d8mr16814444wrt.684.1650487417711; Wed, 20 Apr 2022 13:43:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o13-20020a05600c4fcd00b00392951086efsm343242wmq.34.2022.04.20.13.43.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:37 -0700 (PDT) Message-Id: <00fab626663292f8c82233a3a72b4f9e885de17b.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:03 +0000 Subject: [PATCH v5 13/28] fsmonitor--daemon: cd out of worktree root Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the fsmonitor--daemon to CD outside of the worktree before starting up. The common Git startup mechanism causes the CWD of the daemon process to be in the root of the worktree. On Windows, this causes the daemon process to hold a locked handle on the CWD and prevents other processes from moving or deleting the worktree while the daemon is running. CD to HOME before entering main event loops. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++-- compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------ fsmonitor--daemon.h | 1 + 3 files changed, 47 insertions(+), 8 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 50ae3cca575..1b7c757f583 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * before we need it. */ if (ipc_server_run_async(&state->ipc_server_data, - fsmonitor_ipc__get_path(), &ipc_opts, + state->path_ipc.buf, &ipc_opts, handle_client, state)) return error_errno( _("could not start IPC thread pool on '%s'"), - fsmonitor_ipc__get_path()); + state->path_ipc.buf); /* * Start the fsmonitor listener thread to collect filesystem @@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) static int fsmonitor_run_daemon(void) { struct fsmonitor_daemon_state state; + const char *home; int err; memset(&state, 0, sizeof(state)); @@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void) strbuf_addch(&state.path_cookie_prefix, '/'); + /* + * We create a named-pipe or unix domain socket inside of the + * ".git" directory. (Well, on Windows, we base our named + * pipe in the NPFS on the absolute path of the git + * directory.) + */ + strbuf_init(&state.path_ipc, 0); + strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path())); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void) goto done; } + /* + * CD out of the worktree root directory. + * + * The common Git startup mechanism causes our CWD to be the + * root of the worktree. On Windows, this causes our process + * to hold a locked handle on the CWD. This prevents the + * worktree from being moved or deleted while the daemon is + * running. + * + * We assume that our FS and IPC listener threads have either + * opened all of the handles that they need or will do + * everything using absolute paths. + */ + home = getenv("HOME"); + if (home && *home && chdir(home)) + die_errno(_("could not cd home '%s'"), home); + err = fsmonitor_run_daemon_1(&state); done: @@ -1310,6 +1337,7 @@ done: strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); strbuf_release(&state.path_cookie_prefix); + strbuf_release(&state.path_ipc); return err; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 3f1b68267bd..c43d92b9620 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch) } /* - * NEEDSWORK: If an external is deleted, the above - * returns an error. I'm not sure that there's anything that - * we can do here other than failing -- the /.git - * link file would be broken anyway. We might try to check - * for that and return a better error message, but I'm not - * sure it is worth it. + * GetOverlappedResult() fails if the watched directory is + * deleted while we were waiting for an overlapped IO to + * complete. The documentation did not list specific errors, + * but I observed ERROR_ACCESS_DENIED (0x05) errors during + * testing. + * + * Note that we only get notificaiton events for events + * *within* the directory, not *on* the directory itself. + * (These might be properies of the parent directory, for + * example). + * + * NEEDSWORK: We might try to check for the deleted directory + * case and return a better error message, but I'm not sure it + * is worth it. + * + * Shutdown if we get any error. */ error(_("GetOverlappedResult failed on '%s' [GLE %ld]"), diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index bd09fffc176..223c2131b58 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -54,6 +54,7 @@ struct fsmonitor_daemon_state { struct fsmonitor_daemon_backend_data *backend_data; struct ipc_server_data *ipc_server_data; + struct strbuf path_ipc; }; /* From patchwork Wed Apr 20 20:43:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820804 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1BEB3C433EF for ; Wed, 20 Apr 2022 20:44:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382198AbiDTUqy (ORCPT ); Wed, 20 Apr 2022 16:46:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58552 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382165AbiDTUq2 (ORCPT ); Wed, 20 Apr 2022 16:46:28 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CCEB6340F6 for ; Wed, 20 Apr 2022 13:43:40 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id p18so3863851wru.5 for ; Wed, 20 Apr 2022 13:43:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jLD5/kKpi8DMO5qU1ej4+CHxgwvkoymtmnde3SN53k4=; b=iRIFIJlAdVwBUkc/hj/6Na3Yt7pLGuKarnjA2Vpcm4RCzB7kk5+62XtpYNsPQpzIyE beMVlcQuulTnKkSFMf7CXP+3wk29cI1CPWurMV0UplOM8WrQIfDNREcnC3OAV/paGeMZ Z7MoNkz+ORNz92DP2XQtH6bVfnNVvxZiLkxGC/aUO57xmCH8/IrRz+ffVjO+fZ9Xp6Zl 2ACiRXY/ZowZXdFxlQuGLxpaYPcaJH2OiGmy47s9goscg45fKdMV6gCwsOcuniRvXT33 ru9FRi7htAQH5qfT44h+Eax2NhBlyvXkAsOaSGhh6OzLhrR0uYTi9cboTRYaQownnOUM M51Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jLD5/kKpi8DMO5qU1ej4+CHxgwvkoymtmnde3SN53k4=; b=M52V2QhTKFiy7J03Pmk3/Hi09thfe4lQmRLNocIcNUkG0Qq+nELwCSMpUo4nBk4E24 MyAZZ1+2Fk4OrGRHwFV/inp3Rwa5hflXvwsrpDLOXF7pzO7P3V+5hkD+6tIixDxiVlXk mbvn2J43slYru26IGpPskaytFo6LQ7ltAfyasc7N5QfO97cZkwUK5IvGqYRj61XMvBoh eYGctPVKAPLHA7BaFKwyIqVWrZAq61dXSzAM6sLyjIanYxoj+dXIXP/6IOnlYNA1/R8M usLrGeS5I+iNd04frUhdHZ2lnU+k5cAOsQF/xJNHx48aNIyy0OFka0EyirI/zr81KYCx yYQA== X-Gm-Message-State: AOAM532TDxkXQBaMkvC1Q+eLQnBX+buPb5lS2qKvY8HjPuu4Ql+Cb7of 6hYM9uH3P4vbwLkRglHU7/jr6YM8SyM= X-Google-Smtp-Source: ABdhPJwVfJvCRRy9T2ovMTVGpDKXeKktZR2RHqY92/gj1uRuKJM/H4eGK/I+DHY4PZMLIOaeoezUFg== X-Received: by 2002:a5d:4563:0:b0:207:bc6f:e345 with SMTP id a3-20020a5d4563000000b00207bc6fe345mr16798068wrc.372.1650487419141; Wed, 20 Apr 2022 13:43:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n15-20020a5d6b8f000000b00207ab69284csm626268wrx.23.2022.04.20.13.43.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:38 -0700 (PDT) Message-Id: <6552f51802ba383f4c82866ce20d989749d47be3.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:04 +0000 Subject: [PATCH v5 14/28] fsmonitor--daemon: prepare for adding health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Refactor daemon thread startup to make it easier to start a third thread class to monitor the health of the daemon. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 1b7c757f583..14cd2d5eb52 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int listener_started = 0; + int err = 0; /* * Start the IPC thread pool before the we've started the file @@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) if (pthread_create(&state->listener_thread, NULL, fsm_listen__thread_proc, state) < 0) { ipc_server_stop_async(state->ipc_server_data); - ipc_server_await(state->ipc_server_data); - - return error(_("could not start fsmonitor listener thread")); + err = error(_("could not start fsmonitor listener thread")); + goto cleanup; } + listener_started = 1; /* * The daemon is now fully functional in background threads. + * Our primary thread should now just wait while the threads + * do all the work. + */ +cleanup: + /* * Wait for the IPC thread pool to shutdown (whether by client - * request or from filesystem activity). + * request, from filesystem activity, or an error). */ ipc_server_await(state->ipc_server_data); @@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * event from the IPC thread pool, but it doesn't hurt to tell * it again. And wait for it to shutdown. */ - fsm_listen__stop_async(state); - pthread_join(state->listener_thread, NULL); + if (listener_started) { + fsm_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + } - return state->error_code; + if (err) + return err; + if (state->error_code) + return state->error_code; + return 0; } static int fsmonitor_run_daemon(void) From patchwork Wed Apr 20 20:43:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820805 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A976C433F5 for ; Wed, 20 Apr 2022 20:44:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382216AbiDTUrL (ORCPT ); Wed, 20 Apr 2022 16:47:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382167AbiDTUq3 (ORCPT ); Wed, 20 Apr 2022 16:46:29 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2CB2038D8D for ; Wed, 20 Apr 2022 13:43:42 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id n126-20020a1c2784000000b0038e8af3e788so2037589wmn.1 for ; Wed, 20 Apr 2022 13:43:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RHDySr29nbivzTxmvRnigE4gmiYhmYbPGYTJmKipC9k=; b=q5+8eDYTq43oJO0TuaAFTelNdtRsck2k6btxOoSIEUbe0597KfebruxwkHHW0AOuWW m19pFyFTQBCNWjIQ+Qh84uZv416zkYDCdPuljNH8Ox7haXcMoiwC4zItcjcg8oE8V/IX ZwpSJEkibz+Y+54q3xo2MxYgf+CR28uNFeCeWwbU8pXQhoNGAfG0RQmamW/JeLo6reFz rIalris4A72KIIjjsileysTH1mTjrWzwXavwD1ZDdnsQxSz/aGhWmwO9c4AMPu3mEoBz bGKsg6XebRvqg0s8bIV5CYqDOBromAOc0CYCVMTuEujLFitIGY0T+485EhWfQpYslPYy FBDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RHDySr29nbivzTxmvRnigE4gmiYhmYbPGYTJmKipC9k=; b=Q5PzpKP7i3NXiL2vyiEdQcK6XrxlvuLWtVGUlZDqsf59TM1SgePDRqQ1NPK8F3xmFK I1J8rgrvU8CqfnInwxrBNIz41/DoxLTfZVsDtJXnMKaeo+OFy3hImdETJrZFqVVeZziL DarM3l1IS1rnIjmCJllOgd1PgI0Mxvs1lOrtzFb9mBEzmhZ6Cd8GolHXNxOt62stpNQf i8PcjHlZnGD7IoS3oyR3sccAYFmR2cppO8cNmeZtM9ykx0KJMn9vZ66qkMT4QFiem+r1 IiPH7PQWGncCOCepftVdULeq1mVL0pdKGp1zdl4yZ+2ZJfOT5JK3sYf7JgsvNVLgxxj3 U7ng== X-Gm-Message-State: AOAM5309Npnm7cMBFfVPajLwjgneDclcJv6B8hrQxDiuQlgRtVAemXEF egWtIf1eNphhLmoTnkCUvaHOCwPAbng= X-Google-Smtp-Source: ABdhPJxG9ruDWlrG9l1jeoGjlQX0+5FmIbilgzyrA8PFEQ4ZGIkEzFSZYLdTXv/ti0BjMytqUaXquw== X-Received: by 2002:a05:600c:1d9f:b0:38e:d527:33fc with SMTP id p31-20020a05600c1d9f00b0038ed52733fcmr5319585wms.186.1650487420346; Wed, 20 Apr 2022 13:43:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b6-20020adfd1c6000000b0020aac8a9946sm827044wrd.47.2022.04.20.13.43.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:39 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:05 +0000 Subject: [PATCH v5 15/28] fsmonitor--daemon: rename listener thread related variables Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Rename platform-specific listener thread related variables and data types as we prepare to add another backend thread type. [] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data` [] `state->backend_data` becomes `state->listen_data` [] `state->error_code` becomes `state->listen_error_code` Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 6 +++--- compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++-------------- compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++------------- compat/fsmonitor/fsm-listen.h | 2 +- fsmonitor--daemon.h | 6 +++--- 5 files changed, 36 insertions(+), 36 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 14cd2d5eb52..d5def8faf90 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1225,8 +1225,8 @@ cleanup: if (err) return err; - if (state->error_code) - return state->error_code; + if (state->listen_error_code) + return state->listen_error_code; return 0; } @@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void) hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); - state.error_code = 0; + state.listen_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 14105f45c18..07113205a61 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -27,7 +27,7 @@ #include "fsm-listen.h" #include "fsmonitor--daemon.h" -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { CFStringRef cfsr_worktree_path; CFStringRef cfsr_gitdir_path; @@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, const FSEventStreamEventId event_ids[]) { struct fsmonitor_daemon_state *state = ctx; - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; char **paths = (char **)event_paths; struct fsmonitor_batch *batch = NULL; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) NULL, NULL }; - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; const void *dir_array[2]; CALLOC_ARRAY(data, 1); - state->backend_data = data; + state->listen_data = data; data->cfsr_worktree_path = CFStringCreateWithCString( NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); @@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) failed: error(_("Unable to create FSEventStream.")); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); return -1; } void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; if (data->stream) { if (data->stream_started) @@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state) FSEventStreamRelease(data->stream); } - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->shutdown_style = SHUTDOWN_EVENT; CFRunLoopStop(data->rl); @@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->rl = CFRunLoopGetCurrent(); @@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) switch (data->shutdown_style) { case FORCE_ERROR_STOP: - state->error_code = -1; + state->listen_error_code = -1; /* fall thru */ case FORCE_SHUTDOWN: ipc_server_stop_async(state->ipc_server_data); @@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) return; force_error_stop_without_loop: - state->error_code = -1; + state->listen_error_code = -1; ipc_server_stop_async(state->ipc_server_data); return; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index c43d92b9620..be2d93f47b2 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -54,7 +54,7 @@ struct one_watch wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { struct one_watch *watch_worktree; struct one_watch *watch_gitdir; @@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname( void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); + SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]); } static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, @@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch) free(watch); } -static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, +static int start_rdcw_watch(struct fsm_listen_data *data, struct one_watch *watch) { DWORD dwNotifyFilter = @@ -516,7 +516,7 @@ static int process_1_worktree_event( */ static int process_worktree_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_worktree; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -646,7 +646,7 @@ force_shutdown: */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_gitdir; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -704,11 +704,11 @@ skip_this_path: void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; DWORD dwWait; int result; - state->error_code = 0; + state->listen_error_code = 0; if (start_rdcw_watch(data, data->watch_worktree) == -1) goto force_error_stop; @@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) } force_error_stop: - state->error_code = -1; + state->listen_error_code = -1; force_shutdown: /* @@ -790,7 +790,7 @@ clean_shutdown: int fsm_listen__ctor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; CALLOC_ARRAY(data, 1); @@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) data->nr_listener_handles++; } - state->backend_data = data; + state->listen_data = data; return 0; failed: @@ -836,16 +836,16 @@ failed: void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; CloseHandle(data->hEventShutdown); destroy_watch(data->watch_worktree); destroy_watch(data->watch_gitdir); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h index f0539349baf..41650bf8972 100644 --- a/compat/fsmonitor/fsm-listen.h +++ b/compat/fsmonitor/fsm-listen.h @@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state); * do so if the listener thread receives a normal shutdown signal from * the IPC layer.) * - * It should set `state->error_code` to -1 if the daemon should exit + * It should set `state->listen_error_code` to -1 if the daemon should exit * with an error. */ void fsm_listen__loop(struct fsmonitor_daemon_state *state); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 223c2131b58..2c6fa1a5d91 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); */ void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); -struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ +struct fsm_listen_data; /* opaque platform-specific data for listener thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; @@ -50,8 +50,8 @@ struct fsmonitor_daemon_state { int cookie_seq; struct hashmap cookies; - int error_code; - struct fsmonitor_daemon_backend_data *backend_data; + int listen_error_code; + struct fsm_listen_data *listen_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Wed Apr 20 20:43:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820806 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD006C433EF for ; Wed, 20 Apr 2022 20:44:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382154AbiDTUrO (ORCPT ); Wed, 20 Apr 2022 16:47:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58512 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382127AbiDTUqm (ORCPT ); Wed, 20 Apr 2022 16:46:42 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6503E4162C for ; Wed, 20 Apr 2022 13:43:43 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id bv16so3839530wrb.9 for ; Wed, 20 Apr 2022 13:43:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=oo0AetpE73jEsBAv+/tKCahFPPNqYumxsjVcEzUnxQs=; b=ehP96wh/MbW/l9z0o6hxGG2KYgAWJ1oO+AuiZei54YueglETluUOPc56JwJKMh1+h/ WgvUFA4O7YHUYzUQaN9mQzrgtDqaUzqW8W7gvZ+vutA+jmQQRXUl9EXUUBk+ajLVKRPn gqJPCrUDf21RaWmwavMEWsAFT6td8a3zxmvtni8x8mMC1gROtFeN4YmIbfIrZxrYs4DI bVgij298cAn9yf3bSnxOw/dSuia31P9fqNVrwwCugwPauBBJ4d86W+5Aa20Yd+dFkO7N jVsQ7AgUd/Ki40HIEPC2vdV1acDQxYAxVOX94fceUYuWw6IdL1BIdemj1QJwpJlmx/5G S15Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=oo0AetpE73jEsBAv+/tKCahFPPNqYumxsjVcEzUnxQs=; b=UpRw87Cd6j1u7DHSJSH+/CWGEuck6QReZtYyBbC7TqxSXub+rXi/ynK7Q2V6xK3jeP QW4fWVLE7iyoROuK13/Uf5VQASe4L1VmoIhIgSP8/USPhEp1wDkCGNCNb2ZiWuo2nDgj JPZkdPbrRbCJGauboGUIRdQS1hEE88Pt7jNfifE0E05XQ/Jc3BGcSkXLBPx70bL59lmB ezFnm2qEwq9xyaaEFsMQx06mnJUBboHKFFEJC2iSQo1E33vnzaHZ6OwP/XGiBSSA8skB zP50txfQIR4ncySKlbqr2Gg5RzBpy699CcR5Ukr4XpX1+QW5t3hnXpuB5KmC3Qdhnt8D kvWA== X-Gm-Message-State: AOAM532Z4goJjPZE+mA++vQh29tUgNxviyFpK7LN4+boQfNlqJx/gjt+ 8+gRWJkclva37f6LGjPhA1SqO1sF5Vk= X-Google-Smtp-Source: ABdhPJzInRalR7qEwdjQ48SQWZKuMSxU98yJia0JBmHVyP7LuhVJItw80ZxADoRTgfv4tOi9tfiRlg== X-Received: by 2002:adf:a1c4:0:b0:20a:92c3:abfd with SMTP id v4-20020adfa1c4000000b0020a92c3abfdmr13738886wrv.551.1650487421517; Wed, 20 Apr 2022 13:43:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r3-20020a5d6c63000000b0020a94e82a3dsm687557wrz.64.2022.04.20.13.43.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:41 -0700 (PDT) Message-Id: <2a44f2eded1eba45dc240ee743ff4e178aa72f4d.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:06 +0000 Subject: [PATCH v5 16/28] fsmonitor--daemon: stub in health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create another thread to watch over the daemon process and automatically shut it down if necessary. This commit creates the basic framework for a "health" thread to monitor the daemon and/or the file system. Later commits will add platform-specific code to do the actual work. The "health" thread is intended to monitor conditions that would be difficult to track inside the IPC thread pool and/or the file system listener threads. For example, when there are file system events outside of the watched worktree root or if we want to have an idle-timeout auto-shutdown feature. This commit creates the health thread itself, defines the thread-proc and sets up the thread's event loop. It integrates this new thread into the existing IPC and Listener thread models. This commit defines the API to the platform-specific code where all of the monitoring will actually happen. The platform-specific code for MacOS is just stubs. Meaning that the health thread will immediately exit on MacOS, but that is OK and expected. Future work can define MacOS-specific monitoring. The platform-specific code for Windows sets up enough of the WaitForMultipleObjects() machinery to watch for system and/or custom events. Currently, the set of wait handles only includes our custom shutdown event (sent from our other theads). Later commits in this series will extend the set of wait handles to monitor other conditions. Signed-off-by: Jeff Hostetler --- Makefile | 6 ++- builtin/fsmonitor--daemon.c | 39 +++++++++++++++ compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++ compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++ compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++ contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor--daemon.h | 4 ++ 7 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-darwin.c create mode 100644 compat/fsmonitor/fsm-health-win32.c create mode 100644 compat/fsmonitor/fsm-health.h diff --git a/Makefile b/Makefile index 93604fe8ef7..5f1623baadd 100644 --- a/Makefile +++ b/Makefile @@ -472,8 +472,9 @@ all:: # # If your platform supports a built-in fsmonitor backend, set # FSMONITOR_DAEMON_BACKEND to the "" of the corresponding -# `compat/fsmonitor/fsm-listen-.c` that implements the -# `fsm_listen__*()` routines. +# `compat/fsmonitor/fsm-listen-.c` and +# `compat/fsmonitor/fsm-health-.c` files +# that implement the `fsm_listen__*()` and `fsm_health__*()` routines. # # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS @@ -1982,6 +1983,7 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index d5def8faf90..e22f53026e9 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,6 +3,7 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsm-health.h" #include "compat/fsmonitor/fsm-listen.h" #include "fsmonitor--daemon.h" #include "simple-ipc.h" @@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, pthread_mutex_unlock(&state->main_lock); } +static void *fsm_health__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-health"); + + fsm_health__loop(state); + + trace2_thread_exit(); + return NULL; +} + static void *fsm_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int health_started = 0; int listener_started = 0; int err = 0; @@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) } listener_started = 1; + /* + * Start the health thread to watch over our process. + */ + if (pthread_create(&state->health_thread, NULL, + fsm_health__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + err = error(_("could not start fsmonitor health thread")); + goto cleanup; + } + health_started = 1; + /* * The daemon is now fully functional in background threads. * Our primary thread should now just wait while the threads @@ -1223,10 +1248,17 @@ cleanup: pthread_join(state->listener_thread, NULL); } + if (health_started) { + fsm_health__stop_async(state); + pthread_join(state->health_thread, NULL); + } + if (err) return err; if (state->listen_error_code) return state->listen_error_code; + if (state->health_error_code) + return state->health_error_code; return 0; } @@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); state.listen_error_code = 0; + state.health_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ @@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void) goto done; } + if (fsm_health__ctor(&state)) { + err = error(_("could not initialize health thread")); + goto done; + } + /* * CD out of the worktree root directory. * @@ -1344,6 +1382,7 @@ done: pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsm_listen__dtor(&state); + fsm_health__dtor(&state); ipc_server_free(state.ipc_server_data); diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c new file mode 100644 index 00000000000..b9f709e8548 --- /dev/null +++ b/compat/fsmonitor/fsm-health-darwin.c @@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c new file mode 100644 index 00000000000..94b1d020f25 --- /dev/null +++ b/compat/fsmonitor/fsm-health-win32.c @@ -0,0 +1,72 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +struct fsm_health_data +{ + HANDLE hEventShutdown; + + HANDLE hHandles[1]; /* the array does not own these handles */ +#define HEALTH_SHUTDOWN 0 + int nr_handles; /* number of active event handles */ +}; + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + CALLOC_ARRAY(data, 1); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown; + data->nr_handles++; + + state->health_data = data; + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + if (!state || !state->health_data) + return; + + data = state->health_data; + + CloseHandle(data->hEventShutdown); + + FREE_AND_NULL(state->health_data); +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data = state->health_data; + + for (;;) { + DWORD dwWait = WaitForMultipleObjects(data->nr_handles, + data->hHandles, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) + goto clean_shutdown; + + error(_("health thread wait failed [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->health_error_code = -1; + ipc_server_stop_async(state->ipc_server_data); +clean_shutdown: + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ + SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]); +} diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h new file mode 100644 index 00000000000..45547ba9380 --- /dev/null +++ b/compat/fsmonitor/fsm-health.h @@ -0,0 +1,47 @@ +#ifndef FSM_HEALTH_H +#define FSM_HEALTH_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor health thread. + * This will be called from the main thread PRIOR to staring the + * thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsm_health__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the health thread. + * This will be called from the main thread AFTER joining the thread. + */ +void fsm_health__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to monitor the + * health of the daemon process. This will run in the health thread. + * + * The health thread should call `ipc_server_stop_async()` if it needs + * to cause a shutdown. (It should NOT do so if it receives a shutdown + * shutdown signal.) + * + * It should set `state->health_error_code` to -1 if the daemon should exit + * with an error. + */ +void fsm_health__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the health thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsm_health__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSM_HEALTH_H */ diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index b8f9f7a0388..16ace43d1c7 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 2c6fa1a5d91..2102a5c9ff5 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); struct fsm_listen_data; /* opaque platform-specific data for listener thread */ +struct fsm_health_data; /* opaque platform-specific data for health thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; + pthread_t health_thread; pthread_mutex_t main_lock; struct strbuf path_worktree_watch; @@ -51,7 +53,9 @@ struct fsmonitor_daemon_state { struct hashmap cookies; int listen_error_code; + int health_error_code; struct fsm_listen_data *listen_data; + struct fsm_health_data *health_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Wed Apr 20 20:43:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820807 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98042C433F5 for ; Wed, 20 Apr 2022 20:44:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382175AbiDTUrR (ORCPT ); Wed, 20 Apr 2022 16:47:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382176AbiDTUqn (ORCPT ); Wed, 20 Apr 2022 16:46:43 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 78D55424BF for ; Wed, 20 Apr 2022 13:43:44 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id b19so3824543wrh.11 for ; Wed, 20 Apr 2022 13:43:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=Yu7L3KWzFCamvDNOlKM9W9rzPBFCKsl2tkhTtwdkkkfz81ZbX6VOj5HHtNT12rir7s 22C93dX28TBYxcvHD7a28u6xcgVVgtKB24MAGbBGm/kTnmi2/0VITBKimBGNfAySrSY5 fl1C3BwL30xnkuuTZToABC3U/WeBUU9HNp/4bGzYdGbEE7u+TtANCgAAetyPsahuVq7Y EflJLjAV7BvefgLIsmLXO1Exl7H76qaK5VVrHTVyXhTucp3zh0nWTqKAII0tOzKeeZVs x3fjAyio7br+2sXuxKJrmaT2sEQl0bZ0DARSF1leOITbWT9iSE5zZxcgY9O2FHxHPHMn VZuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=ZmKx2XXxNpqngNIJIDA6TmKnUJz3qbFtL7VIRh0nGS0ZVaQwHjgPqhD1QWhMh9JwKJ cAQZ36u/mOjZ2zSKt9IiNgxgm3H2Mx4PINE6VXnjgiT1zMj46MEoK1X+hxaQe+IjXr+N oJz+MsrwCnOVnNIUCNHGSIs6bkzcq9GT3kw4u1ghbeKWHX0KF4IhXunKglWjqx5EoYiP 3pBcAxlYveCSICFgEpl1KBelcNRbCPJHo28f2SS0oc0J4jDIFCDGvVnsOAv9LvLO09ow oIdxWAuDffWl/eCmjal2EAH3/jRAty6SNzd5Q6MEpRPeA+NhzMZZc/rIh4sxZ+YXTbi1 2/bw== X-Gm-Message-State: AOAM530+LxWhupotPnRzcvVjXCnDtQBhFNNUKAwNzOmnEd4NvW7VNr5H rg1YYH3xKAneCzRlFpmohy14uwI7uoI= X-Google-Smtp-Source: ABdhPJyV3+hwIqH9EG/ZeKeSEPuahNNyR84i+6YpLcbos6lMYkwq7lQHv25zEnqnYTXzXdJuvkkHaA== X-Received: by 2002:a5d:6205:0:b0:1e4:b3fd:9ba8 with SMTP id y5-20020a5d6205000000b001e4b3fd9ba8mr16655104wru.426.1650487422779; Wed, 20 Apr 2022 13:43:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z6-20020a056000110600b00207aa9eec98sm647892wrw.30.2022.04.20.13.43.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:42 -0700 (PDT) Message-Id: <854fb5e3658156f93f76915aaf3d4a194145b3fb.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:07 +0000 Subject: [PATCH v5 17/28] fsm-health-win32: add polling framework to monitor daemon health Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend the Windows version of the "health" thread to periodically inspect the system and shutdown if warranted. This commit updates the thread's wait loop to use a timeout and defines a (currently empty) table of functions to poll the system. A later commit will add functions to the table to actually inspect the system. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 94b1d020f25..24fc612bf02 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -4,6 +4,24 @@ #include "fsm-health.h" #include "fsmonitor--daemon.h" +/* + * Every minute wake up and test our health. + */ +#define WAIT_FREQ_MS (60 * 1000) + +/* + * State machine states for each of the interval functions + * used for polling our health. + */ +enum interval_fn_ctx { + CTX_INIT = 0, + CTX_TERM, + CTX_TIMER +}; + +typedef int (interval_fn)(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx); + struct fsm_health_data { HANDLE hEventShutdown; @@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) FREE_AND_NULL(state->health_data); } +/* + * A table of the polling functions. + */ +static interval_fn *table[] = { + NULL, /* must be last */ +}; + +/* + * Call all of the polling functions in the table. + * Shortcut and return first error. + * + * Return 0 if all succeeded. + */ +static int call_all(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + int k; + + for (k = 0; table[k]; k++) { + int r = table[k](state, ctx); + if (r) + return r; + } + + return 0; +} + void fsm_health__loop(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data = state->health_data; + int r; + + r = call_all(state, CTX_INIT); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; for (;;) { DWORD dwWait = WaitForMultipleObjects(data->nr_handles, data->hHandles, - FALSE, INFINITE); + FALSE, WAIT_FREQ_MS); if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) goto clean_shutdown; + if (dwWait == WAIT_TIMEOUT) { + r = call_all(state, CTX_TIMER); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; + continue; + } + error(_("health thread wait failed [GLE %ld]"), GetLastError()); goto force_error_stop; @@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state) force_error_stop: state->health_error_code = -1; +force_shutdown: ipc_server_stop_async(state->ipc_server_data); clean_shutdown: + call_all(state, CTX_TERM); return; } From patchwork Wed Apr 20 20:43:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820817 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7888DC433F5 for ; Wed, 20 Apr 2022 20:44:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382225AbiDTUri (ORCPT ); Wed, 20 Apr 2022 16:47:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382179AbiDTUqo (ORCPT ); Wed, 20 Apr 2022 16:46:44 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECDDF427F4 for ; Wed, 20 Apr 2022 13:43:45 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id l3-20020a05600c1d0300b0038ff89c938bso2927053wms.0 for ; Wed, 20 Apr 2022 13:43:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pg+pIfJCewJfl3Uu7fv/1CMTNHr8lfgpM21Z9LQ/57U=; b=kwIil0ZwKL2QNCI0xrm9TAZ0POcJqg0RyX2McDPD9Wpxue3EbCams5hIpHrjtyVCuZ 0QMFyt2xq4kE7GvETSZDp9RYmE+fdIwLGILimGZPsCSYtrWOdSbAygorExmwaB5hl9zq yZCwQobu3A7vZraG158xM08jutsAvomTXUKMMMNDpg4j/y8dkA3UHxUQOBLA59Nb0acE 7pXNJs/BWhuhi9P08nwIO74/ei43fNKsMCT5jlJTMBYM+QxfXH8eOFgGzho7ohnsnvJZ +mhcUBizt93eCFn+bV5YPD1D/SXwFlC8K7O1fKESKIc3bAV/Y76SnviBkyPJlVeUAaPZ T1ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pg+pIfJCewJfl3Uu7fv/1CMTNHr8lfgpM21Z9LQ/57U=; b=Lpv+d12XtD2H+Sm0r3pjIMKFkOxfDEcZJC9/LZsiBXRSuovoQh5YQEa9aZlRk+UrTQ 6pVH9w96ZXosMT1GYcQt74ke8LrZdRWpjDl7c1sBdO3sGMLPzBQo5ssJd6txqbILy8P/ 5saTIYiBejSasxzYGMAf4ukpaot5KDmJYIsrNzQ3rC1ayYZ2/ph6p6rlM+PMyQB1zbxF i34ibW2dA1HtND9LvKI/vB2HSbfXdHbND/x3p+soUAh0OLqgt3ft3NMTpXRwl5P2j7FM kpxhgbNmX4+CfHNbH3VUZESv0prs3l6JfsjpgztWVEPmQavreBmoYvY/70kHhiBHgOfW PUdw== X-Gm-Message-State: AOAM53353ORLqndinjOsmWi+w7dzcqHfflgOEiu5GTCk6aSJQQfG6cI7 p08vHwIeh50EbqXNeOiKM0JqJmGyGzU= X-Google-Smtp-Source: ABdhPJw3inRIDiwO8fqjyjV3Y5XCWdSU2NJRoYPOleE0fMMXJU7M9uI2nv2sk1onf7bWhOPcHDdX+Q== X-Received: by 2002:a05:600c:4e11:b0:38f:fbc7:2bc4 with SMTP id b17-20020a05600c4e1100b0038ffbc72bc4mr5404007wmq.141.1650487424149; Wed, 20 Apr 2022 13:43:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a18-20020adffb92000000b00207a257d47asm658499wrr.95.2022.04.20.13.43.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:43 -0700 (PDT) Message-Id: <3af1fe0d61d711b32ffa23a1816b4f0117244878.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:08 +0000 Subject: [PATCH v5 18/28] fsm-health-win32: force shutdown daemon if worktree root moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Force shutdown fsmonitor daemon if the worktree root directory is moved, renamed, or deleted. Use Windows low-level GetFileInformationByHandle() to get and compare the Windows system unique ID for the directory with a cached version when we started up. This lets us detect the case where someone renames the directory that we are watching and then creates a new directory with the original pathname. This is important because we are listening to a named pipe for requests and they are stored in the Named Pipe File System (NPFS) which a kernel-resident pseudo filesystem not associated with the actual NTFS directory. For example, if the daemon was watching "~/foo/", it would have a directory-watch handle on that directory and a named-pipe handle for "//./pipe/...foo". Moving the directory to "~/bar/" does not invalidate the directory handle. (So the daemon would actually be watching "~/bar" but listening on "//./pipe/...foo". If the user then does "git init ~/foo" and causes another daemon to start, the first daemon will still have ownership of the pipe and the second daemon instance will fail to start. "git status" clients in "~/foo" will ask "//./pipe/...foo" about changes and the first daemon instance will tell them about "~/bar". This commit causes the first daemon to shutdown if the system unique ID for "~/foo" changes (changes from what it was when the daemon started). Shutdown occurs after a periodic poll. After the first daemon exits and releases the lock on the named pipe, subsequent Git commands may cause another daemon to be started on "~/foo". Similarly, a subsequent Git command may cause another daemon to be started on "~/bar". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 24fc612bf02..2ea08c1d4e8 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -29,8 +29,150 @@ struct fsm_health_data HANDLE hHandles[1]; /* the array does not own these handles */ #define HEALTH_SHUTDOWN 0 int nr_handles; /* number of active event handles */ + + struct wt_moved + { + wchar_t wpath[MAX_PATH + 1]; + BY_HANDLE_FILE_INFORMATION bhfi; + } wt_moved; }; +/* + * Lookup the system unique ID for the path. This is as close as + * we get to an inode number, but this also contains volume info, + * so it is a little stronger. + */ +static int lookup_bhfi(wchar_t *wpath, + BY_HANDLE_FILE_INFORMATION *bhfi) +{ + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + + hDir = CreateFileW(wpath, desired_access, share_mode, NULL, + OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] health thread could not open '%ls'"), + GetLastError(), wpath); + return -1; + } + + if (!GetFileInformationByHandle(hDir, bhfi)) { + error(_("[GLE %ld] health thread getting BHFI for '%ls'"), + GetLastError(), wpath); + CloseHandle(hDir); + return -1; + } + + CloseHandle(hDir); + return 0; +} + +/* + * Compare the relevant fields from two system unique IDs. + * We use this to see if two different handles to the same + * path actually refer to the same *instance* of the file + * or directory. + */ +static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1, + const BY_HANDLE_FILE_INFORMATION *bhfi_2) +{ + return (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber && + bhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh && + bhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow); +} + +/* + * Shutdown if the original worktree root directory been deleted, + * moved, or renamed? + * + * Since the main thread did a "chdir(getenv($HOME))" and our CWD + * is not in the worktree root directory and because the listener + * thread added FILE_SHARE_DELETE to the watch handle, it is possible + * for the root directory to be moved or deleted while we are still + * watching it. We want to detect that here and force a shutdown. + * + * Granted, a delete MAY cause some operations to fail, such as + * GetOverlappedResult(), but it is not guaranteed. And because + * ReadDirectoryChangesW() only reports on changes *WITHIN* the + * directory, not changes *ON* the directory, our watch will not + * receive a delete event for it. + * + * A move/rename of the worktree root will also not generate an event. + * And since the listener thread already has an open handle, it may + * continue to receive events for events within the directory. + * However, the pathname of the named-pipe was constructed using the + * original location of the worktree root. (Remember named-pipes are + * stored in the NPFS and not in the actual file system.) Clients + * trying to talk to the worktree after the move/rename will not + * reach our daemon process, since we're still listening on the + * pipe with original path. + * + * Furthermore, if the user does something like: + * + * $ mv repo repo.old + * $ git init repo + * + * A new daemon cannot be started in the new instance of "repo" + * because the named-pipe is still being used by the daemon on + * the original instance. + * + * So, detect move/rename/delete and shutdown. This should also + * handle unsafe drive removal. + * + * We use the file system unique ID to distinguish the original + * directory instance from a new instance and force a shutdown + * if the unique ID changes. + * + * Since a worktree move/rename/delete/unmount doesn't happen + * that often (and we can't get an immediate event anyway), we + * use a timeout and periodically poll it. + */ +static int has_worktree_moved(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + struct fsm_health_data *data = state->health_data; + BY_HANDLE_FILE_INFORMATION bhfi; + int r; + + switch (ctx) { + case CTX_TERM: + return 0; + + case CTX_INIT: + if (xutftowcs_path(data->wt_moved.wpath, + state->path_worktree_watch.buf) < 0) { + error(_("could not convert to wide characters: '%s'"), + state->path_worktree_watch.buf); + return -1; + } + + /* + * On the first call we lookup the unique sequence ID for + * the worktree root directory. + */ + return lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi); + + case CTX_TIMER: + r = lookup_bhfi(data->wt_moved.wpath, &bhfi); + if (r) + return r; + if (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) { + error(_("BHFI changed '%ls'"), data->wt_moved.wpath); + return -1; + } + return 0; + + default: + die(_("unhandled case in 'has_worktree_moved': %d"), + (int)ctx); + } + + return 0; +} + + int fsm_health__ctor(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data; @@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) * A table of the polling functions. */ static interval_fn *table[] = { + has_worktree_moved, NULL, /* must be last */ }; From patchwork Wed Apr 20 20:43:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820808 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FFF3C433EF for ; Wed, 20 Apr 2022 20:44:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382183AbiDTUrU (ORCPT ); Wed, 20 Apr 2022 16:47:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58534 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382155AbiDTUqq (ORCPT ); Wed, 20 Apr 2022 16:46:46 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3BE22443C8 for ; Wed, 20 Apr 2022 13:43:47 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id l62-20020a1c2541000000b0038e4570af2fso2027769wml.5 for ; Wed, 20 Apr 2022 13:43:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=m/7nmjzQrPWZOZRTUqvWzqRTYtxoxcBSCt/eY/PHK+w=; b=mMJXyEUuS00nmY1CyshN87S7duWHDwwpL0R1m1nhDHrnicveQZYw1hYHDvzFQ9zdXY LzeG4yLAeLEBaIXzJTDzAVMDY48nWbMKZZ+L0idB0pUMB2s5WKjwzN0UxyzdW8UHECjp h3qZ/v+aX+R7jv4SWFi6FQwAyjqPujQwAtV3Zzr3dAkXrAk0zOeCGyUjtBDpe3nRc8ev c+bNZdIlYvKort9qlIE+PoIB1pwoPtWeViaDeFd9UErvIlSoQ5q6CiRCBXhmpRz8FOSX Zsvxt7DE6TWEGCfLTOai6glGceCcm587Y9CSvxKYGDHmgWSLDYB3PINJI1puMVnhHgwx pjFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=m/7nmjzQrPWZOZRTUqvWzqRTYtxoxcBSCt/eY/PHK+w=; b=ffaSZhhlCBeH3jvZldCHcyytymJ1Aa75wM9W69wfN6jr8OxmKvIhMcJ2pgrgjDN2xd B736GsiqKZLUdir8zGib8FkxpQ0SZTDNy+uQD8yYoD9ZfxV13DtyF9aUWnuHbmygSsGn wfJ3SnaIKbSTjY0Cg9o1NnZVWi71vomcTtG3/RJ8xtwfLg5cvA0lHAyO3mnvTfFN1o1E 7FJIsj6LZtQ4stMfsW1e5OUvmUNVZEf1rwdg9zzVs1iz7PZhnMTRyEzN4Rdw9S74mvb9 QZYnyE0QgGZYHMMetWChDcOPE5AW0JYNMhz1KpndkrKVPo+w34wIdl1tXug9I9sl4eYb /H7Q== X-Gm-Message-State: AOAM530R/r4tCGu3w6HssKpjTQtilGMUGVahoMQqLsvthjKczlYPeAnd mV+s0Z3Bv0F3sTXBC2qR+dQABYg0Mps= X-Google-Smtp-Source: ABdhPJzjRwYfWAEE4VXGH007qIhQvaYPIuVNvTHrVoCqnytBOHhIfdRbK+ZlKzicQajnpWeS0hXjZw== X-Received: by 2002:a1c:f705:0:b0:37d:f2e5:d8ec with SMTP id v5-20020a1cf705000000b0037df2e5d8ecmr5455425wmh.21.1650487425435; Wed, 20 Apr 2022 13:43:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k23-20020adfb357000000b0020ab724ceb8sm747133wrd.54.2022.04.20.13.43.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:44 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:09 +0000 Subject: [PATCH v5 19/28] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the listener thread to shutdown the daemon if the spelling of the worktree root directory changes. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 07113205a61..83d38e8ac6c 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) strbuf_release(&msg); } +static int ef_is_root_changed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagRootChanged); +} + static int ef_is_root_delete(const FSEventStreamEventFlags ef) { return (ef & kFSEventStreamEventFlagItemIsDir && @@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_is_root_changed(event_flags[k])) { + /* + * The spelling of the pathname of the root directory + * has changed. This includes the name of the root + * directory itself or of any parent directory in the + * path. + * + * (There may be other conditions that throw this, + * but I couldn't find any information on it.) + * + * Force a shutdown now and avoid things getting + * out of sync. The Unix domain socket is inside + * the .git directory and a spelling change will make + * it hard for clients to rendezvous with us. + */ + trace_printf_key(&trace_fsmonitor, + "event: root changed"); + goto force_shutdown; + } + if (ef_ignore_xattr(event_flags[k])) { trace_printf_key(&trace_fsmonitor, "ignore-xattr: '%s', flags=0x%x", From patchwork Wed Apr 20 20:43:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820814 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6A25C433EF for ; Wed, 20 Apr 2022 20:44:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1355124AbiDTUre (ORCPT ); Wed, 20 Apr 2022 16:47:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58796 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382184AbiDTUqr (ORCPT ); Wed, 20 Apr 2022 16:46:47 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 64BDC44772 for ; Wed, 20 Apr 2022 13:43:48 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id s29so777171wrb.8 for ; Wed, 20 Apr 2022 13:43:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=XvgPKeUchtTZBF1tUwNmvMTJvMUZsAj3qQMssKjnUYZyxHGfLIKk0JV8HIAgWY69Gf oOxQ1p9T5f/UKQmDPaxBOtj2/OA80hO+wsu4yRm1Gl7eZYWd8yinWq4COlCQFdIFM4ix 8MX48HK/j/YB8o3P2opNT3iuOpX9r+bCDXVzblfPhZl1/8e0QUEwVj6ztJF3AnPs5gST D3h3SSTb34Dx3gFwkSb2TDrdPbRofVK8bkxRTtKf6bQEqO3dgNqjL1Xh3Inkzy8tuHNZ dT/pzqY8LD9toCLpfZllb0pAp3zd1EvZCXu/bXlXqzGxP43flT+tVjJgqTPjkzXJe7Yw wrVg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=i0GvIocbPEwlIBgNI6NGDUxbm2uWXrSc8SugnYPqR/fEDfsqT7IHGRFh7ppDcJER3k k72HNkvWW+1X9cN6axq23txIYBs4t1tOLNppUiQt2Nn1M41O4XaWFYzzjrDEOK23BRqV KSFGqAPA5Sxz7VjdRGQozxY23p1oXy1rydYEbwPQ9R+7xFBvSn2AaIsCcDDnZAyOZTvC 5SSdfc40h5FHsFBWcdL0Y88J9YX25sc02ezw1uExZ5tWpIFPoVawYKs/MfuuxJE2ATPL 8N+2KL6C3Rz9+udRsyMHWFKU98JcVj9aQJiW7HBmEmb0wwGaNStXCoLEpeoit8HnXCDa 7/og== X-Gm-Message-State: AOAM531JT/GjO4XJeoDCX3S+HsrqE++9Jjxwm9roTniwxKtJVi37xnFd yI9BjXDajS5p37TxCAW4t5iPacoefcA= X-Google-Smtp-Source: ABdhPJwUahu4BfFqB/FIxb+tWDDfQ3uAf/i07+UBhCbx5Y1VAMQh5mGkbhTGGPQ+RPg69V0jR+9CFg== X-Received: by 2002:a5d:4842:0:b0:207:c6ad:4a48 with SMTP id n2-20020a5d4842000000b00207c6ad4a48mr16533050wrs.149.1650487426639; Wed, 20 Apr 2022 13:43:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o11-20020a05600c4fcb00b00391447f7fd4sm345198wmq.24.2022.04.20.13.43.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:46 -0700 (PDT) Message-Id: <15698d64edd0e65e5486a97024818a55c9e4295c.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:10 +0000 Subject: [PATCH v5 20/28] fsmonitor: optimize processing of directory events Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to perform binary search over the cache-entries for a directory notification and then linearly scan forward to find the immediate children. Previously, when the FSMonitor reported a modified directory Git would perform a linear search on the entire cache-entry array for all entries matching that directory prefix and invalidate them. Since the cache-entry array is already sorted, we can use a binary search to find the first matching entry and then only linearly walk forward and invalidate entries until the prefix changes. Also, the original code would invalidate anything having the same directory prefix. Since a directory event should only be received for items that are immediately within the directory (and not within sub-directories of it), only invalidate those entries and not the whole subtree. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 54 insertions(+), 17 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 292a6742b4f..e1229c289cf 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r, static void fsmonitor_refresh_callback(struct index_state *istate, char *name) { int i, len = strlen(name); - if (name[len - 1] == '/') { + int pos = index_name_pos(istate, name, len); + + trace_printf_key(&trace_fsmonitor, + "fsmonitor_refresh_callback '%s' (pos %d)", + name, pos); + if (name[len - 1] == '/') { /* - * TODO We should binary search to find the first path with - * TODO this directory prefix. Then linearly update entries - * TODO while the prefix matches. Taking care to search without - * TODO the trailing slash -- because '/' sorts after a few - * TODO interesting special chars, like '.' and ' '. + * The daemon can decorate directory events, such as + * moves or renames, with a trailing slash if the OS + * FS Event contains sufficient information, such as + * MacOS. + * + * Use this to invalidate the entire cone under that + * directory. + * + * We do not expect an exact match because the index + * does not normally contain directory entries, so we + * start at the insertion point and scan. */ + if (pos < 0) + pos = -pos - 1; /* Mark all entries for the folder invalid */ - for (i = 0; i < istate->cache_nr; i++) { - if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID && - starts_with(istate->cache[i]->name, name)) - istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } - /* Need to remove the / from the path for the untracked cache */ + + /* + * We need to remove the traling "/" from the path + * for the untracked cache. + */ name[len - 1] = '\0'; + } else if (pos >= 0) { + /* + * We have an exact match for this path and can just + * invalidate it. + */ + istate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID; } else { - int pos = index_name_pos(istate, name, strlen(name)); - - if (pos >= 0) { - struct cache_entry *ce = istate->cache[pos]; - ce->ce_flags &= ~CE_FSMONITOR_VALID; + /* + * The path is not a tracked file -or- it is a + * directory event on a platform that cannot + * distinguish between file and directory events in + * the event handler, such as Windows. + * + * Scan as if it is a directory and invalidate the + * cone under it. (But remember to ignore items + * between "name" and "name/", such as "name-" and + * "name.". + */ + pos = -pos - 1; + + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + if ((unsigned char)istate->cache[i]->name[len] > '/') + break; + if (istate->cache[i]->name[len] == '/') + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } } @@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) * Mark the untracked cache dirty even if it wasn't found in the index * as it could be a new untracked file. */ - trace_printf_key(&trace_fsmonitor, "fsmonitor_refresh_callback '%s'", name); untracked_cache_invalidate_path(istate, name, 0); } From patchwork Wed Apr 20 20:43:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820816 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22730C433EF for ; Wed, 20 Apr 2022 20:44:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382222AbiDTUrg (ORCPT ); Wed, 20 Apr 2022 16:47:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58538 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382159AbiDTUqr (ORCPT ); Wed, 20 Apr 2022 16:46:47 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E21E44A35 for ; Wed, 20 Apr 2022 13:43:49 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id bg25so1263119wmb.4 for ; Wed, 20 Apr 2022 13:43:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=TYM9RAesGeaqNz2jBk6tvQuIkZPINQDoc+UjrbhzKIs=; b=e0fa1mrxfkDv5SvACTxZE03x2/4BC1MWDHF1y55gkhRbgIMaTUQDzjeUEdTIaOLGqj FYkXuJbDY+MH+i92CsitfRQCjvI3eX+qMPgFvDqlB6JxxQdpB8JwEM/r6x9LTUaLVJXA aqPY4nrfvRc77I7PEkeqDbGSSN7cXtCv8MrDumPEwa5jPb48oRGvHugGEO3uBe88jfku HAodY5x1kEXNTRoTS0F1DL0souIYR8FCJWas4eDK/uXEnoOAMCNI80n0FXL8d0z+/Xgn 46Gqf5vFJ2ZfjKmaNxccuZMt9e54ka27Gq1BaK3d++XjUvzxEJ4/UvgeXA60ZFmPGmXE j9BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=TYM9RAesGeaqNz2jBk6tvQuIkZPINQDoc+UjrbhzKIs=; b=t/eoG14wxyph5/d7dhUtWPGNaHKPnKuuOq6oSLncEP4R2t/eQFB/01p8mlmFOhtyQq y+/lGHt6HUK/q+7uEncLBfBu1d8NDSd6eHC0duIEdSTuU0Y4Rgg+G/wnrLvOM4Zz3MxL klatiV8ybNJszLcB2TWQwGShEXDSSh7Ao8OOEB480HBw7z8gxxnSiPW2qtHI6SoiZT3F t3VitewrRcg3FcnJHMMuXYebRrDsjX1r0mV4bnNh14J7bDSdrwF6mRJY9mQCSvhKwSqV zVSmOK/WnmFH3GoNfTye50wUb0Afud1Ekiz/SUPMxozPiyFttCb2yZ9GMEZS3RLk9ygj OSJw== X-Gm-Message-State: AOAM531ricnDMeAdoj425ycUY5uACcdP/HzntY0IBnQewhuwux7uw022 Bdj+44TRNYsisrRZtmC577lT0V4b1j8= X-Google-Smtp-Source: ABdhPJym7NJE5m4J6LEsICukTwmzUFsxjL3wEwH9B0ce/M2Ia4TVDNL7prKqIiUS6x1CMe8kGhuBKA== X-Received: by 2002:a05:600c:4f46:b0:391:8d69:b336 with SMTP id m6-20020a05600c4f4600b003918d69b336mr5424883wmq.83.1650487427731; Wed, 20 Apr 2022 13:43:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f4-20020a7bc8c4000000b0038ebbe10c5esm336137wml.25.2022.04.20.13.43.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:47 -0700 (PDT) Message-Id: <9d0da8fc22bd713344da11bee41e6e8c2c644b7b.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:11 +0000 Subject: [PATCH v5 21/28] t7527: FSMonitor tests for directory moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create unit tests to move a directory. Verify that `git status` gives the same result with and without FSMonitor enabled. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index c9c7dd77e3c..d0e681d008f 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -274,6 +274,16 @@ test_expect_success 'setup' ' trace* EOF + mkdir -p T1/T2/T3/T4 && + echo 1 >T1/F1 && + echo 1 >T1/T2/F1 && + echo 1 >T1/T2/T3/F1 && + echo 1 >T1/T2/T3/T4/F1 && + echo 2 >T1/F2 && + echo 2 >T1/T2/F2 && + echo 2 >T1/T2/T3/F2 && + echo 2 >T1/T2/T3/T4/F2 && + git -c core.fsmonitor=false add . && test_tick && git -c core.fsmonitor=false commit -m initial && @@ -356,6 +366,19 @@ directory_to_file () { echo 1 >dir1 } +move_directory_contents_deeper() { + mkdir T1/_new_ && + mv T1/[A-Z]* T1/_new_ +} + +move_directory_up() { + mv T1/T2/T3 T1 +} + +move_directory() { + mv T1/T2/T3 T1/T2/NewT3 +} + # The next few test cases confirm that our fsmonitor daemon sees each type # of OS filesystem notification that we care about. At this layer we just # ensure we are getting the OS notifications and do not try to confirm what @@ -660,6 +683,10 @@ do matrix_try $uc_val $fsm_val file_to_directory matrix_try $uc_val $fsm_val directory_to_file + matrix_try $uc_val $fsm_val move_directory_contents_deeper + matrix_try $uc_val $fsm_val move_directory_up + matrix_try $uc_val $fsm_val move_directory + if test $fsm_val = true then test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" ' From patchwork Wed Apr 20 20:43:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820810 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5778C433F5 for ; Wed, 20 Apr 2022 20:44:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382188AbiDTUr0 (ORCPT ); Wed, 20 Apr 2022 16:47:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58510 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382187AbiDTUqr (ORCPT ); Wed, 20 Apr 2022 16:46:47 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0BA6D45501 for ; Wed, 20 Apr 2022 13:43:51 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id y21so1972204wmi.2 for ; Wed, 20 Apr 2022 13:43:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=Db3U35E/FpnencFzAkSvlXIV3JsPLSFXbY7bgazeeef6oE1JzbzhWjdGmHc0F/u9ld M9HTdC+z79O6rfdoSbCCCU9nQwbVXoS3gji91gaQn7LaqUMllxvXN8RtB2FT8d7bP9oi XAnn33/cyf2drqOvoIWyCsc4DwfugSl5bVx1E5se+sZyypqZRNn13jVLEJiwVzdd78vX S6zRwesU13AVcxLOpvHQGs4gFvtJZOV4dqfx4Tc/JDKy397G3/0h2qvePMM+gEUw8d9c wjnoIkpxj/77eSjjNDiV6s0aGeHd3NkxpIs/rHEvTc/6UZaspCxhL9SHrFTyv9DfsyuX MXfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=8CAx0acsfDgQoafjCsKATlf9JCdAAE55NqlfNWLMZCB2QCVwf4D+iUp7RyhWJX2Iga j7nNir9Jglh/TeYopfcIDT165FsCSuWI+1UmFAkn/E1H0bluXjZ8YkzDswMr6j89xOyR G03yhWesn6QTjb3mh6KB2QY8Eg+Fy1ndfu5rMUWJkWxvxT9oVR58Njsa3EyHugCRHbj9 aUzyWmygn5wAnJidbeqZCz/EIXYqK+e+juS19PsZKsrdspF4MiVjmUIpUrLVNGGGXQ6G guHDc8PPLOMDG8L/6vU0KvOW7qkXgwAGTsoEx3lVGnBxPxMma3vzUO7jWnG1YlEf7493 CRhg== X-Gm-Message-State: AOAM533ykLKdkKlTeXEgyd+SL06AltuBNzVu+I1+WK4E6VzcYHXPCYb8 cMkd5nJwkzQYvxBBdZnV0WNp4fJwEG0= X-Google-Smtp-Source: ABdhPJzCaY/2BLFqbxdRtD9Y6qs81YgaucwiWbJ0VYOgRwPqVtaZ1JAcwO8DYgpyMcj6I3oKTtO16g== X-Received: by 2002:a05:600c:4ec6:b0:392:8b89:fce9 with SMTP id g6-20020a05600c4ec600b003928b89fce9mr5343199wmq.25.1650487429179; Wed, 20 Apr 2022 13:43:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az30-20020a05600c601e00b0038ebd950caesm324313wmb.30.2022.04.20.13.43.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:48 -0700 (PDT) Message-Id: <040c00cfd6f427cd686f9f64cec458dff1c26555.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:12 +0000 Subject: [PATCH v5 22/28] t/perf/p7527: add perf test for builtin FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100755 t/perf/p7527-builtin-fsmonitor.sh diff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh new file mode 100755 index 00000000000..9338b9ea008 --- /dev/null +++ b/t/perf/p7527-builtin-fsmonitor.sh @@ -0,0 +1,257 @@ +#!/bin/sh + +test_description="Perf test for the builtin FSMonitor" + +. ./perf-lib.sh + +if ! test_have_prereq FSMONITOR_DAEMON +then + skip_all="fsmonitor--daemon is not supported on this platform" + test_done +fi + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +# Lie to perf-lib and ask for a new empty repo and avoid +# the complaints about GIT_PERF_REPO not being big enough +# the perf hit when GIT_PERF_LARGE_REPO is copied into +# the trash directory. +# +# NEEDSWORK: It would be nice if perf-lib had an option to +# "borrow" an existing large repo (especially for gigantic +# monorepos) and use it in-place. For now, fake it here. +# +test_perf_fresh_repo + + +# Use a generated synthetic monorepo. If it doesn't exist, we will +# generate it. If it does exist, we will put it in a known state +# before we start our timings. +# +PARAM_D=5 +PARAM_W=10 +PARAM_F=9 + +PARAMS="$PARAM_D"."$PARAM_W"."$PARAM_F" + +BALLAST_BR=p0006-ballast +export BALLAST_BR + +TMP_BR=tmp_br +export TMP_BR + +REPO=../repos/gen-many-files-"$PARAMS".git +export REPO + +if ! test -d $REPO +then + (cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F) +fi + + +enable_uc () { + git -C $REPO config core.untrackedcache true + git -C $REPO update-index --untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +disable_uc () { + git -C $REPO config core.untrackedcache false + git -C $REPO update-index --no-untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +start_fsm () { + git -C $REPO fsmonitor--daemon start + git -C $REPO fsmonitor--daemon status + git -C $REPO config core.fsmonitor true + git -C $REPO update-index --fsmonitor + git -C $REPO status >/dev/null 2>&1 +} + +stop_fsm () { + git -C $REPO config --unset core.fsmonitor + git -C $REPO update-index --no-fsmonitor + test_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null + git -C $REPO status >/dev/null 2>&1 +} + + +# Ensure that FSMonitor is turned off on the borrowed repo. +# +test_expect_success "Setup borrowed repo (fsm+uc)" " + stop_fsm && + disable_uc +" + +# Also ensure that it starts in a known state. +# +# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time +# the ballast checkout, since only the first invocation does any work and the +# subsequent ones just print "already on branch" and quit, so the reported +# time is not useful. +# +# Create a temp branch and do all work relative to it so that we don't +# accidentially alter the real ballast branch. +# +test_expect_success "Setup borrowed repo (temp ballast branch)" " + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO reset --hard && + git -C $REPO clean -d -f && + test_might_fail git -C $REPO branch -D $TMP_BR && + git -C $REPO branch $TMP_BR $BALLAST_BR && + git -C $REPO checkout $TMP_BR +" + + +echo Data >data.txt + +# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With +# FSMonitor enabled, we can get a skewed view of status times, since +# the index MAY (or may not) be updated after the first invocation +# which will update the FSMonitor Token, so the subsequent invocations +# may get a smaller response from the daemon. +# +do_status () { + msg=$1 + + test_perf "$msg" " + git -C $REPO status >/dev/null 2>&1 + " +} + +do_matrix () { + uc=$1 + fsm=$2 + + t="[uc $uc][fsm $fsm]" + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_expect_success "$t Setup matrix branch" " + git -C $REPO clean -d -f && + git -C $REPO checkout $TMP_BR && + test_might_fail git -C $REPO branch -D $MATRIX_BR && + git -C $REPO branch $MATRIX_BR $TMP_BR && + git -C $REPO checkout $MATRIX_BR + " + + if test $uc = true + then + enable_uc + else + disable_uc + fi + + if test $fsm = true + then + start_fsm + else + stop_fsm + fi + + do_status "$t status after checkout" + + # Modify many files in the matrix branch. + # Stage them. + # Commit them. + # Rollback. + # + test_expect_success "$t modify tracked files" " + find $REPO -name file1 -exec cp data.txt {} \\; + " + + do_status "$t status after big change" + + # Don't bother timing the "add" because _REPEAT_COUNT + # issue described above. + # + test_expect_success "$t add all" " + git -C $REPO add -A + " + + do_status "$t status after add all" + + test_expect_success "$t add dot" " + git -C $REPO add . + " + + do_status "$t status after add dot" + + test_expect_success "$t commit staged" " + git -C $REPO commit -a -m data + " + + do_status "$t status after commit" + + test_expect_success "$t reset HEAD~1 hard" " + git -C $REPO reset --hard HEAD~1 >/dev/null 2>&1 + " + + do_status "$t status after reset hard" + + # Create some untracked files. + # + test_expect_success "$t create untracked files" " + cp -R $REPO/ballast/dir1 $REPO/ballast/xxx1 + " + + do_status "$t status after create untracked files" + + # Remove the new untracked files. + # + test_expect_success "$t clean -df" " + git -C $REPO clean -d -f + " + + do_status "$t status after clean" + + if test $fsm = true + then + stop_fsm + fi +} + +# Begin testing each case in the matrix that we care about. +# +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" + +fsm_values="false true" + +for uc_val in $uc_values +do + for fsm_val in $fsm_values + do + do_matrix $uc_val $fsm_val + done +done + +cleanup () { + uc=$1 + fsm=$2 + + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_might_fail git -C $REPO branch -D $MATRIX_BR +} + + +# We're borrowing this repo. We should leave it in a clean state. +# +test_expect_success "Cleanup temp and matrix branches" " + git -C $REPO clean -d -f && + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO branch -D $TMP_BR && + for uc_val in $uc_values + do + for fsm_val in $fsm_values + do + cleanup $uc_val $fsm_val + done + done +" + +test_done From patchwork Wed Apr 20 20:43:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820809 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0AE0C433EF for ; Wed, 20 Apr 2022 20:44:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382190AbiDTUrY (ORCPT ); Wed, 20 Apr 2022 16:47:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382189AbiDTUqr (ORCPT ); Wed, 20 Apr 2022 16:46:47 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31AE34551A for ; Wed, 20 Apr 2022 13:43:52 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id u17-20020a05600c211100b0038eaf4cdaaeso4526124wml.1 for ; Wed, 20 Apr 2022 13:43:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CHXUh21TwGifOMZAMmF5DZKmuMch+8bTNMzfHHNtpjM=; b=ktqETLvEYPHqjEvoVyqoWEUYc/MLXn/figIDxuzUmSm+XkjKSJFWVN94YwG4NELtyi Rerssg/WgeNYGFwlx81NhhMyXYKd+FvjHwII9obwD3E62M8uMpD9jeeEVnZDAyxSfV/r l1OiILO5/aZRPihHRsOhntRzP2sjFWEYV+g1LVvLIk5pJ3gQmMga20oty86NZO/EAiRg X8QhBJLK38lgvJY54NTkLB0Kv18j6bRgXw2J9SmHTXFvRR8fUpuO4MD64L1JY3XruV/5 +Iihg3r9aefQHHRnKj4H26be5Ca9jGOjIeGNPcAYhgwrOdv9nxSW20tYyBxoHZBXzLOy LDOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CHXUh21TwGifOMZAMmF5DZKmuMch+8bTNMzfHHNtpjM=; b=m5j8xVGfkNiBewn89Zv0lyW6QO6LHUGm78SvVKo3kvQjo1q+dMnpJ76JE3ZcmWs8b3 bGYEzhlmFYSBtg5CGarLAfdz+RwHSx5nqxhBgj4d9A0ztzthU3JyLR+u792IhInfqIlM arJKhpNwpz20QYTb88Q8R40E7J0Ha6WJceOKrF4W3NoMDPwxFNM3Tnm8Tv8SNqI+u5yF YkRKjOtVjQ9hEBSCX5XcZScOojfPNj/AvqkqiXabPVPewFmY47i4QUbHo2r9/UUNx5Ev QLLH3DEYs3Eb817larMgCzK1nzqf3EI0MxgdIk9rtI9HvgC9X/BQAPVcv9FmMe7f5NmH KPcw== X-Gm-Message-State: AOAM532cpwUO6dDp4CPAuQBtuEwUeOYVaDID89m+mVzWbGtQ9sVGqd88 buIncfwyoJRhsGN2I04P/+j6jwRlBv8= X-Google-Smtp-Source: ABdhPJy0EQhtuZ57DKS7Ae4M9MzkzqAFCeEyXUG95mcAY3+En/oh8uaYs6G309FgOrx7N1ZTUxY/jg== X-Received: by 2002:a7b:c0c8:0:b0:392:ad9a:5d91 with SMTP id s8-20020a7bc0c8000000b00392ad9a5d91mr3531472wmh.37.1650487430302; Wed, 20 Apr 2022 13:43:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e1-20020a05600c4e4100b00392910b276esm345254wmq.9.2022.04.20.13.43.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:49 -0700 (PDT) Message-Id: <5db241f7d2f654d46d4858ce9e9413032705abf3.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:13 +0000 Subject: [PATCH v5 23/28] fsmonitor: never set CE_FSMONITOR_VALID on submodules Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Never set CE_FSMONITOR_VALID on the cache-entry of submodule directories. During a client command like 'git status', we may need to recurse into each submodule to compute a status summary for the submodule. Since the purpose of the ce_flag is to let Git avoid scanning a cache-entry, setting the flag causes the recursive call to be avoided and we report incorrect (no status) for the submodule. We created an OS watch on the root directory of our working directory and we receive events for everything in the cone under it. When submodules are present inside our working directory, we receive events for both our repo (the super) and any subs within it. Since our index doesn't have any information for items within the submodules, we can't use those events. We could try to truncate the paths of those events back to the submodule boundary and mark the GITLINK as dirty, but that feels expensive since we would have to prefix compare every FS event that we receive against a list of submodule roots. And it still wouldn't be sufficient to correctly report status on the submodule, since we don't have any space in the cache-entry to cache the submodule's status (the 'SCMU' bits in porcelain V2 speak). That is, the CE_FSMONITOR_VALID bit just says that we don't need to scan/inspect it because we already know the answer -- it doesn't say that the item is clean -- and we don't have space in the cache-entry to store those answers. So we should always do the recursive scan. Therefore, we should never set the flag on GITLINK cache-entries. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 2 + fsmonitor.h | 11 ++++ t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+) diff --git a/fsmonitor.c b/fsmonitor.c index e1229c289cf..57d6a483bee 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate) if (fsmonitor_enabled) { /* Mark all entries valid */ for (i = 0; i < istate->cache_nr; i++) { + if (S_ISGITLINK(istate->cache[i]->ce_mode)) + continue; istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; } diff --git a/fsmonitor.h b/fsmonitor.h index 3f41f653691..edf7ce5203b 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate) * Set the given cache entries CE_FSMONITOR_VALID bit. This should be * called any time the cache entry has been updated to reflect the * current state of the file on disk. + * + * However, never mark submodules as valid. When commands like "git + * status" run they might need to recurse into the submodule (using a + * child process) to get a summary of the submodule state. We don't + * have (and don't want to create) the facility to translate every + * FS event that we receive and that happens to be deep inside of a + * submodule back to the submodule root, so we cannot correctly keep + * track of this bit on the gitlink directory. Therefore, we never + * set it on submodules. */ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce) { @@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache if (fsm_mode > FSMONITOR_MODE_DISABLED && !(ce->ce_flags & CE_FSMONITOR_VALID)) { + if (S_ISGITLINK(ce->ce_mode)) + return; istate->cache_changed = 1; ce->ce_flags |= CE_FSMONITOR_VALID; trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index d0e681d008f..1cf0ffa5676 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -721,4 +721,115 @@ do ' done +# Test fsmonitor interaction with submodules. +# +# If we start the daemon in the super, it will see FS events for +# everything in the working directory cone and this includes any +# files/directories contained *within* the submodules. +# +# A `git status` at top level will get events for items within the +# submodule and ignore them, since they aren't named in the index +# of the super repo. This makes the fsmonitor response a little +# noisy, but it doesn't alter the correctness of the state of the +# super-proper. +# +# When we have submodules, `git status` normally does a recursive +# status on each of the submodules and adds a summary row for any +# dirty submodules. (See the "S..." bits in porcelain V2 output.) +# +# It is therefore important that the top level status not be tricked +# by the FSMonitor response to skip those recursive calls. That is, +# even if FSMonitor says that the mtime of the submodule directory +# hasn't changed and it could be implicitly marked valid, we must +# not take that shortcut. We need to force the recusion into the +# submodule so that we get a summary of the status *within* the +# submodule. + +create_super () { + super=$1 && + + git init "${super}" && + echo x >${super}/file_1 && + echo y >${super}/file_2 && + echo z >${super}/file_3 && + mkdir ${super}/dir_1 && + echo a >${super}/dir_1/file_11 && + echo b >${super}/dir_1/file_12 && + mkdir ${super}/dir_1/dir_2 && + echo a >${super}/dir_1/dir_2/file_21 && + echo b >${super}/dir_1/dir_2/file_22 && + git -C ${super} add . && + git -C ${super} commit -m "initial ${super} commit" +} + +create_sub () { + sub=$1 && + + git init "${sub}" && + echo x >${sub}/file_x && + echo y >${sub}/file_y && + echo z >${sub}/file_z && + mkdir ${sub}/dir_x && + echo a >${sub}/dir_x/file_a && + echo b >${sub}/dir_x/file_b && + mkdir ${sub}/dir_x/dir_y && + echo a >${sub}/dir_x/dir_y/file_a && + echo b >${sub}/dir_x/dir_y/file_b && + git -C ${sub} add . && + git -C ${sub} commit -m "initial ${sub} commit" +} + +my_match_and_clean () { + git -C super --no-optional-locks status --porcelain=v2 >actual.with && + git -C super --no-optional-locks -c core.fsmonitor=false \ + status --porcelain=v2 >actual.without && + test_cmp actual.with actual.without && + + git -C super/dir_1/dir_2/sub reset --hard && + git -C super/dir_1/dir_2/sub clean -d -f +} + +test_expect_success "Submodule always visited" ' + test_when_finished "git -C super fsmonitor--daemon stop; \ + rm -rf super; \ + rm -rf sub" && + + create_super "super" && + create_sub "sub" && + + git -C super submodule add ../sub ./dir_1/dir_2/sub && + git -C super commit -m "add sub" && + + start_daemon -C super && + git -C super config core.fsmonitor true && + git -C super update-index --fsmonitor && + git -C super status && + + # Now run pairs of commands w/ and w/o FSMonitor while we make + # some dirt in the submodule and confirm matching output. + + # Completely clean status. + my_match_and_clean && + + # .M S..U + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u && + my_match_and_clean && + + # .M S.M. + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M S.M. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M SC.. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + git -C super/dir_1/dir_2/sub commit -m "SC.." && + my_match_and_clean +' + test_done From patchwork Wed Apr 20 20:43:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820815 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92563C433FE for ; Wed, 20 Apr 2022 20:44:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382215AbiDTUrg (ORCPT ); Wed, 20 Apr 2022 16:47:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382191AbiDTUqr (ORCPT ); Wed, 20 Apr 2022 16:46:47 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 656384578A for ; Wed, 20 Apr 2022 13:43:53 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id u3so3873539wrg.3 for ; Wed, 20 Apr 2022 13:43:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WbCyBB3b+JrlO6bcxV7sHoIjfAZBpv4bnjaEWZ+VEMc=; b=NiiunuM1lTI7oD3wiWQOfYDeFQqBNSs7Qm11Ren3UPSu0p2/8/1uCx0U4u4mVmRLhQ Dfvh3v5QGf9YyLI7Cq4J4rZ3Nd+btoOIvVqPdqmMeoXIesstuQQ6/BtFKWHam5+87rW5 dzRjOhvfE8QklqUBV1T4Sn1n093f/h0BxjGsfbuFQo6ZkEkxI2vYJtKTrJVaNp5oRNj9 8H8MKNc837T59z+qRQV5IxBePwwryoZIULCYnQS2LnonwMlNCFXdAW80TUJmUgFkfEYT ffWxAQWseaTply1O9wu2YHXY4TaKYFLAv9hpRIPYLwxq3fHpCqHepT3x4qHws2Vq8JXI gnHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WbCyBB3b+JrlO6bcxV7sHoIjfAZBpv4bnjaEWZ+VEMc=; b=Ibq+GJdZTRf4qZm36ZWDqi4bd0TRoD6L5T00cD8ai7nGjIphXx1Eld3pBGcOX2bZzo WAxeHhaE4bipZmJzxVcejqhkfLgFrOdr7DQyoFLxiVeiBmz4TZSwAB+U/Y4dGV7TLzyE gJKnNQZPXeMVVKdOvyeU2dgs+lEnRql04Y5OOZiAtzWZEo4jq5i3Um6Rqt4fIwgvDeEo nzhg1j+muq1jWh9Pdtp8BDoFzgsmGizj7UQThk1PfWIEiQa7VauSZgjIcbitDGSnGbj/ jqN4vftYEoUHv5RQOCpRx0r7mGWmxmBXbxVu2XWw2mZ9TrbclQNdqD2iKbV3FYPVNa/n M58Q== X-Gm-Message-State: AOAM53179nfxxrug2MWCvUTDSqS+VW5mXJ5qHGMVzPM3K7ZPaRrzY8NF p9CRzPBtMbP/4Ee/dJZCW8fOW7BFd1E= X-Google-Smtp-Source: ABdhPJxjszH7wfgW1aWo0ML62eeBBCXclAi1VuBcERAsI2krr+DLkQSOo+9vGaIuR+iQNzWrgPJGUA== X-Received: by 2002:adf:e48c:0:b0:20a:7e29:e16d with SMTP id i12-20020adfe48c000000b0020a7e29e16dmr17220796wrm.595.1650487431622; Wed, 20 Apr 2022 13:43:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s7-20020a5d6a87000000b00207aad420c4sm645718wru.101.2022.04.20.13.43.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:51 -0700 (PDT) Message-Id: <93de3707d26e1e4485a192b57cc4d2a9526318d5.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:14 +0000 Subject: [PATCH v5 24/28] t7527: test FSMonitor on case insensitive+preserving file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Test that FS events from the OS are received using the preserved, on-disk spelling of files/directories rather than spelling used to make the change. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 1cf0ffa5676..a4f8008fea7 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -832,4 +832,40 @@ test_expect_success "Submodule always visited" ' my_match_and_clean ' +# On a case-insensitive file system, confirm that the daemon +# notices when the .git directory is moved/renamed/deleted +# regardless of how it is spelled in the the FS event. +# That is, does the FS event receive the spelling of the +# operation or does it receive the spelling preserved with +# the file/directory. +# +test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' +# test_when_finished "stop_daemon_delete_repo test_insensitive" && + + git init test_insensitive && + + start_daemon -C test_insensitive --tf "$PWD/insensitive.trace" && + + mkdir -p test_insensitive/abc/def && + echo xyz >test_insensitive/ABC/DEF/xyz && + + test_path_is_dir test_insensitive/.git && + test_path_is_dir test_insensitive/.GIT && + + # Rename .git using an alternate spelling to verify that that + # daemon detects it and automatically shuts down. + mv test_insensitive/.GIT test_insensitive/.FOO && + sleep 1 && + mv test_insensitive/.FOO test_insensitive/.git && + test_must_fail git -C test_insensitive fsmonitor--daemon status && + + # Verify that events were reported using on-disk spellings of the + # directories and files that we touched. We may or may not get a + # trailing slash on modified directories. + # + egrep "^event: abc/?$" ./insensitive.trace && + egrep "^event: abc/def/?$" ./insensitive.trace && + egrep "^event: abc/def/xyz$" ./insensitive.trace +' + test_done From patchwork Wed Apr 20 20:43:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820812 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CA26C43217 for ; Wed, 20 Apr 2022 20:44:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382201AbiDTUr3 (ORCPT ); Wed, 20 Apr 2022 16:47:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58966 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382165AbiDTUrK (ORCPT ); Wed, 20 Apr 2022 16:47:10 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7842E45AE2 for ; Wed, 20 Apr 2022 13:43:54 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id o20-20020a05600c511400b0038ebbbb2ad8so2054067wms.0 for ; Wed, 20 Apr 2022 13:43:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bIorACD9MvGhf+EOOe9S7KEedrit1duOzRZ5v8bUeZQ=; b=aDZm//1/KlbHB3RTSchqdsIIZImLw/3+wjUb/0CxQmBEeFO5gyOHpbjH1GRw7bhNF3 fhpzm7mezgbHL023CvbgrYaf4XyYbsmshvjv7oiujpqTEvVrMgoOFJsKoofVEuyxzoCt iHvUNUhb57c0nQweVoRi00iAJ8SSJIhO9H+vYW1lImzRDkDlCxZVi+QSRDrSsUVyXO0m K2L70IYme7O0tUBrn3KRQjFJzuj8jfGjYvuC/g4uuPJ+3Pjhi5s4J6vJRQIPYRW18s67 55u15NT26IrtQSKR5EcGtHegF4XSNws0g8NCpjzqdDMkrjJPaX4BLLzz/iRkrdAIthsZ 4l8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bIorACD9MvGhf+EOOe9S7KEedrit1duOzRZ5v8bUeZQ=; b=V9U8NwQiiRfd8viVja9YCLVZVvV6Q8f4BbMQbm9HsC6NbHsG7sXNDCBqlfyBw9JA2h ZjXzllef6ppzRxdsd999wRALG8oTJv8qx8dBXOSMIwGcJaL+oLI00Ri0z+MWHA9cxj6i GkpeXYv14flCzlOXgjibthup3VcHx6HQgOEojR/elQjoE7dDA3ZUzzrEBjB8JvJYwivX FbcRWSHNyAK7O5MREROtuLNfi0sNHeiu5X8Z7/Co/i28le67M/EuwTw8RGqIJ8wGRG7J GReuy0Q9xAU6D29Oo6akASc+i+5XyqHV9CPbCUFILVTnX42GjxCptVxAGrHSM8Cuy3vl Ww3g== X-Gm-Message-State: AOAM533mTSbiqGk9y7G7ko9/sE8ePBwdDp2SKvgw7ouOOyPQjX13MuBf wJEGCBj25k5Parw/9IPl3JH8mhdum5k= X-Google-Smtp-Source: ABdhPJyUAIZXh46fdZln0Kn1eqprnV7e4zoldw/jh3hvxnpb46xd46OjIa1gF1sINjEp4I6cgYZacw== X-Received: by 2002:a7b:cd04:0:b0:38e:d7a4:1548 with SMTP id f4-20020a7bcd04000000b0038ed7a41548mr5277065wmj.73.1650487432699; Wed, 20 Apr 2022 13:43:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p1-20020a1c7401000000b0038ed3bb00c9sm342825wmc.6.2022.04.20.13.43.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:52 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:15 +0000 Subject: [PATCH v5 25/28] fsmonitor: on macOS also emit NFC spelling for NFD pathname Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Emit NFC or NFC and NFD spellings of pathnames on macOS. MacOS is Unicode composition insensitive, so NFC and NFD spellings are treated as aliases and collide. While the spelling of pathnames in filesystem events depends upon the underlying filesystem, such as APFS, HFS+ or FAT32, the OS enforces such collisions regardless of filesystem. Teach the daemon to always report the NFC spelling and to report the NFD spelling when stored in that format on the disk. This is slightly more general than "core.precomposeUnicode". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 83d38e8ac6c..823cf63999e 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef) return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); } +/* + * On MacOS we have to adjust for Unicode composition insensitivity + * (where NFC and NFD spellings are not respected). The different + * spellings are essentially aliases regardless of how the path is + * actually stored on the disk. + * + * This is related to "core.precomposeUnicode" (which wants to try + * to hide NFD completely and treat everything as NFC). Here, we + * don't know what the value the client has (or will have) for this + * config setting when they make a query, so assume the worst and + * emit both when the OS gives us an NFD path. + */ +static void my_add_path(struct fsmonitor_batch *batch, const char *path) +{ + char *composed; + + /* add the NFC or NFD path as received from the OS */ + fsmonitor_batch__add_path(batch, path); + + /* if NFD, also add the corresponding NFC spelling */ + composed = (char *)precompose_string_if_needed(path); + if (!composed || composed == path) + return; + + fsmonitor_batch__add_path(batch, composed); + free(composed); +} + + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, rel); + my_add_path(batch, rel); } if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { @@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, tmp.buf); + my_add_path(batch, tmp.buf); } break; From patchwork Wed Apr 20 20:43:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820811 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CC5CCC433FE for ; Wed, 20 Apr 2022 20:44:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382202AbiDTUr2 (ORCPT ); Wed, 20 Apr 2022 16:47:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382201AbiDTUrK (ORCPT ); Wed, 20 Apr 2022 16:47:10 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 027C445AE5 for ; Wed, 20 Apr 2022 13:43:56 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id v64-20020a1cac43000000b0038cfd1b3a6dso4504821wme.5 for ; Wed, 20 Apr 2022 13:43:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=E8xmQ7dYip7SaSRf9IXmrYL6K0sMohTDrDq1DYcKE54=; b=nHORp8dBkd60lRabKxW59VAp5ZdS8KHJFQDdiyIdIlMnEKTwVX4C3Ac+0Ma7aHeaeq Y5XiD3qiuBYbXDKnZblSsTYgH+yh2ARy76XyfTNYUgiqXOhasbaLg1hIdvojDH0RhZmF NELMLmnTZs60H9j8idXFs9ji/qJHcybKgNoE/hkakRgISfmKEyXrBy6L9yuAQOfaDYjm tjEdDqC2Kv78Okm295BibmACrlwJHzJ802nqYHLZSETmTVokImDkYqICXceFbyROMuL7 scJGThSE1waAxP52pPtzKtFlhRhcihjSxTRH4GSpW7zX1H4SiMDoib3SLWBsVlkJ2j4p UQEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=E8xmQ7dYip7SaSRf9IXmrYL6K0sMohTDrDq1DYcKE54=; b=VIldVbEVRd+5K9A2K33/bSj5af712xMPS7WvEKS1oxLT8YJ6Dm6QEqGXS1nb/L/tHB w5h4cObmXyXneEqTcLBtK9vhwj853/uN1+TGpsMojusFJblKRPp8pYHrjoJTpvIV4jXQ 47jSONXKbxKS9DqZJbgQAPhV1vVCWQxqzJfWohDRPptm9NNvmFAtGlSMAyutgcN1CEYr 59pFFL4lDgY0zJev8Tyb5TzFb+HSbtdqfFU8Ln+QNenTvK4Y2QINkGSHhGEw5yZV8UeO Egjq3vXJ4QrCioWP/zfH83T4bIl7vANPBBR6NbJ0FMb+egM7p1fJdLBgPytg9bqHQScF h6Jg== X-Gm-Message-State: AOAM532od/bflP8XLTVkDi8A4RXN2RDvdQMQE0M0CuWhPQ8MixjtJq/5 k1Sog0d2CnKkHCfmJqOxcpbYAj/aBH8= X-Google-Smtp-Source: ABdhPJxiVK/6H0TG0aEiiE2KvSMcnN+z30LE45t6QpUWt7lt+G4N9hZRj+EiA5emUrwlyxXvtD205Q== X-Received: by 2002:a7b:ce99:0:b0:38e:b72a:382c with SMTP id q25-20020a7bce99000000b0038eb72a382cmr5431578wmj.128.1650487434225; Wed, 20 Apr 2022 13:43:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b5-20020adff905000000b0020a8781be70sm680646wrr.12.2022.04.20.13.43.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:53 -0700 (PDT) Message-Id: <7c60623555770d6c9b2834262718fb183f483af1.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:16 +0000 Subject: [PATCH v5 26/28] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a set of prereqs to help understand how file names are handled by the filesystem when they contain NFC and NFD Unicode characters. Signed-off-by: Jeff Hostetler --- t/lib-unicode-nfc-nfd.sh | 167 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+) create mode 100755 t/lib-unicode-nfc-nfd.sh diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh new file mode 100755 index 00000000000..cf9c26d1e22 --- /dev/null +++ b/t/lib-unicode-nfc-nfd.sh @@ -0,0 +1,167 @@ +# Help detect how Unicode NFC and NFD are handled on the filesystem. + +# A simple character that has a NFD form. +# +# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE +# UTF8(NFC): \xc3 \xa9 +# +# NFD: U+0065 LATIN SMALL LETTER E +# U+0301 COMBINING ACUTE ACCENT +# UTF8(NFD): \x65 + \xcc \x81 +# +utf8_nfc=$(printf "\xc3\xa9") +utf8_nfd=$(printf "\x65\xcc\x81") + +# Is the OS or the filesystem "Unicode composition sensitive"? +# +# That is, does the OS or the filesystem allow files to exist with +# both the NFC and NFD spellings? Or, does the OS/FS lie to us and +# tell us that the NFC and NFD forms are equivalent. +# +# This is or may be independent of what type of filesystem we have, +# since it might be handled by the OS at a layer above the FS. +# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a +# collision, for example. +# +# This does not tell us how the Unicode pathname will be spelled +# on disk, but rather only that the two spelling "collide". We +# will examine the actual on disk spelling in a later prereq. +# +test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE ' + mkdir trial_${utf8_nfc} && + mkdir trial_${utf8_nfd} +' + +# Is the spelling of an NFC pathname preserved on disk? +# +# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD +# and on APFS, NFC paths are preserved. As we have established +# above, this is independent of "composition sensitivity". +# +# 0000000 63 5f c3 a9 +# +# (/usr/bin/od output contains different amount of whitespace +# on different platforms, so we need the wildcards here.) +# +test_lazy_prereq UNICODE_NFC_PRESERVED ' + mkdir c_${utf8_nfc} && + ls | od -t x1 | grep "63 *5f *c3 *a9" +' + +# Is the spelling of an NFD pathname preserved on disk? +# +# 0000000 64 5f 65 cc 81 +# +test_lazy_prereq UNICODE_NFD_PRESERVED ' + mkdir d_${utf8_nfd} && + ls | od -t x1 | grep "64 *5f *65 *cc *81" +' + mkdir c_${utf8_nfc} && + mkdir d_${utf8_nfd} && + +# The following _DOUBLE_ forms are more for my curiosity, +# but there may be quirks lurking when there are multiple +# combining characters in non-canonical order. + +# Unicode also allows multiple combining characters +# that can be decomposed in pieces. +# +# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI +# UTF8(NFC): \xe1 \xbd \xa7 +# +# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA +# U+0342 COMBINING GREEK PERISPOMENI +# UTF8(NFD1): \xe1 \xbd \xa1 + \xcd \x82 +# +# But U+1f61 decomposes into +# NFD2: U+03c9 GREEK SMALL LETTER OMEGA +# U+0314 COMBINING REVERSED COMMA ABOVE +# UTF8(NFD2): \xcf \x89 + \xcc \x94 +# +# Yielding: \xcf \x89 + \xcc \x94 + \xcd \x82 +# +# Note that I've used the canonical ordering of the +# combinining characters. It is also possible to +# swap them. My testing shows that that non-standard +# ordering also causes a collision in mkdir. However, +# the resulting names don't draw correctly on the +# terminal (implying that the on-disk format also has +# them out of order). +# +greek_nfc=$(printf "\xe1\xbd\xa7") +greek_nfd1=$(printf "\xe1\xbd\xa1\xcd\x82") +greek_nfd2=$(printf "\xcf\x89\xcc\x94\xcd\x82") + +# See if a double decomposition also collides. +# +test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE ' + mkdir trial_${greek_nfc} && + mkdir trial_${greek_nfd2} +' + +# See if the NFC spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED ' + mkdir c_${greek_nfc} && + ls | od -t x1 | grep "63 *5f *e1 *bd *a7" +' + +# See if the NFD spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED ' + mkdir d_${greek_nfd2} && + ls | od -t x1 | grep "64 *5f *cf *89 *cc *94 *cd *82" +' + +# The following is for debugging. I found it useful when +# trying to understand the various (OS, FS) quirks WRT +# Unicode and how composition/decomposition is handled. +# For example, when trying to understand how (macOS, APFS) +# and (macOS, HFS) and (macOS, FAT32) compare. +# +# It is rather noisy, so it is disabled by default. +# +if test "$unicode_debug" = "true" +then + if test_have_prereq UNICODE_COMPOSITION_SENSITIVE + then + echo NFC and NFD are distinct on this OS/filesystem. + else + echo NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_NFC_PRESERVED + then + echo NFC maintains original spelling. + else + echo NFC is modified. + fi + + if test_have_prereq UNICODE_NFD_PRESERVED + then + echo NFD maintains original spelling. + else + echo NFD is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE + then + echo DOUBLE NFC and NFD are distinct on this OS/filesystem. + else + echo DOUBLE NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED + then + echo Double NFC maintains original spelling. + else + echo Double NFC is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED + then + echo Double NFD maintains original spelling. + else + echo Double NFD is modified. + fi +fi From patchwork Wed Apr 20 20:43:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09DB6C433F5 for ; Wed, 20 Apr 2022 20:44:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382210AbiDTUrb (ORCPT ); Wed, 20 Apr 2022 16:47:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58978 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382203AbiDTUrK (ORCPT ); Wed, 20 Apr 2022 16:47:10 -0400 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 275083CA63 for ; Wed, 20 Apr 2022 13:43:57 -0700 (PDT) Received: by mail-wr1-x42b.google.com with SMTP id c10so3880680wrb.1 for ; Wed, 20 Apr 2022 13:43:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=49gznKgXXcaypBPlROyAWAIsdUble1wg/o3vHy/Znr4=; b=OQfBjneq8+g1B9khs3NnBTBaqkcadQvSKD+AXh8Q5YzB3/UNRfIinlhgJOJsd/oqhj KX5048/PU9BVdSAIyk8i6OlYoiDREfcxLPIOe9kXsRhrOuVNzbLEMZTYYea6hJ7H2fma +6Yc2oq8hOQ6nPYFhhcGnRIxGltJEH1SgnofAM556G4rKWSJHXoEC/dQjNiJtN/AYf2j sSsPoSCiwsbXAgeIShmqevDbTfpKUDSG2TQxIk0oauQC6JHm3AsHIYhKc5Hcl5BW+5wH xOsxAaRUNpOkQmoFGx/uDgajVYoIMLr+ykJF/Ifl6Uc8fcWWIrr+0bvVSx+G4hew0tVJ gkyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=49gznKgXXcaypBPlROyAWAIsdUble1wg/o3vHy/Znr4=; b=wbNrW23+h9Fhd6UyjFhI9UEwVeorrjmX5xo6F6TW5hwNjljs5tWvpytEGrIfyYSylD 6XEPiupQhWa2HHqAwruz2yPmgjOkliTyte8l8O/SSueXnjK7QCFi3YebKLO77Gx5RJMg 0w3c36aZ8N6NCFCvivqxLy1mW8jo9kHPmPQawPgdBxOeGjV0Qhh9Ts6epUCE75hnju66 jdFIir+adOtakt2CdCnVd2ycJ+eqiA70+sRvB1oQUM1Xo3zRWm0ERKSLUzk8HS1T5so3 E/tg4pqi6+m34gOnn7ST7C39M2CfH4BGlP6G/XRmSgRX1XSTLIXRdlAuHTtVlGvHq//P 9ZtQ== X-Gm-Message-State: AOAM531M7kekOQ4kqdhw5E72HzZKT6+7Us0v+2864w2ByAksIaeIMr+w XGpkqdmIOTdlCPoN8xE3wpnlL069JOs= X-Google-Smtp-Source: ABdhPJxbFeYLt5V7qsjhVAnM1WNbsWRwDcqkmJFpLP31xpUPn5ugojsQEN5MbX1BcMcRGa2pOs0ChA== X-Received: by 2002:adf:8b10:0:b0:20a:b1af:5597 with SMTP id n16-20020adf8b10000000b0020ab1af5597mr3135381wra.677.1650487435331; Wed, 20 Apr 2022 13:43:55 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d18-20020a05600c34d200b0038ed14b7ac3sm319180wmq.40.2022.04.20.13.43.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:54 -0700 (PDT) Message-Id: <9724c41d18d4429f8993c9cf83e7622eca079c46.1650487399.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:17 +0000 Subject: [PATCH v5 27/28] t7527: test Unicode NFC/NFD handling on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Confirm that the daemon reports events using the on-disk spelling for Unicode NFC/NFD characters. On APFS we still have Unicode aliasing, so we cannot create two files that only differ by NFC/NFD, but the on-disk format preserves the spelling used to create the file. On HFS+ we also have aliasing, but the path is always stored on disk in NFD. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index a4f8008fea7..5476a01cbff 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -868,4 +868,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' egrep "^event: abc/def/xyz$" ./insensitive.trace ' +# The variable "unicode_debug" is defined in the following library +# script to dump information about how the (OS, FS) handles Unicode +# composition. Uncomment the following line if you want to enable it. +# +# unicode_debug=true + +. "$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh" + +# See if the OS or filesystem does NFC/NFD aliasing/munging. +# +# The daemon should err on the side of caution and send BOTH the +# NFC and NFD forms. It does not know the original spelling of +# the pathname (how the user thinks it should be spelled), so +# emit both and let the client decide (when necessary). This is +# similar to "core.precomposeUnicode". +# +test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' ' + test_when_finished "stop_daemon_delete_repo test_unicode" && + + git init test_unicode && + + start_daemon -C test_unicode --tf "$PWD/unicode.trace" && + + # Create a directory using an NFC spelling. + # + mkdir test_unicode/nfc && + mkdir test_unicode/nfc/c_${utf8_nfc} && + + # Create a directory using an NFD spelling. + # + mkdir test_unicode/nfd && + mkdir test_unicode/nfd/d_${utf8_nfd} && + + git -C test_unicode fsmonitor--daemon stop && + + if test_have_prereq UNICODE_NFC_PRESERVED + then + # We should have seen NFC event from OS. + # We should not have synthesized an NFD event. + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace && + egrep -v "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace + else + # We should have seen NFD event from OS. + # We should have synthesized an NFC event. + egrep "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace + fi && + + # We assume UNICODE_NFD_PRESERVED. + # We should have seen explicit NFD from OS. + # We should have synthesized an NFC event. + egrep "^event: nfd/d_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfd/d_${utf8_nfc}/?$" ./unicode.trace +' + test_done From patchwork Wed Apr 20 20:43:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12820818 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D028EC433FE for ; Wed, 20 Apr 2022 20:44:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1382227AbiDTUrl (ORCPT ); Wed, 20 Apr 2022 16:47:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58980 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1382204AbiDTUrK (ORCPT ); Wed, 20 Apr 2022 16:47:10 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BFA63CA79 for ; Wed, 20 Apr 2022 13:43:58 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id i20so3833840wrb.13 for ; Wed, 20 Apr 2022 13:43:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QQfkCROKa0jO56Rlj7mP24f41yEspKClpJ9PGHlvIJM=; b=JuSf5DxL/9riqWIbFBPXJMcHh3VFrOHwZQcQK35DBbNku3kJwPcHO0R53ufpdYhk7m moJkc6tvJ1dsa7SOQzMxLc3UGYJUhhcCBDLbcxZlXrk4sUpfVAnht0UgzFXnDKULuGcC vWcVV9xyYB/o4+rq5GiGoijHbbmOI0orMfDnEW5v49L6Fei/wrieP3GcfE8iFnCq8BGi ydl+ArBdbroE/M6XnnHhn++qUzQEjY0RAYM5jo3CYJxuwEpbRIrWHeI/qx2sKnKyGGyY 6vDbx2YvTFmn7Dkg1Hu2zliOH0ki3FSwrXYlmQUbTlvrWlGmBPPffAVGtWRBKmqkrfLa kvHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QQfkCROKa0jO56Rlj7mP24f41yEspKClpJ9PGHlvIJM=; b=SYp2E3xfOWapx9jgamvMoYKqLeZbYoV+28+nVEqykPsmmRBZ8EkTjkiThb1Xpo9o0y /cW5JY3ju+4x/12mEzQRt/b3rK2hcEvErRkSZyiQvHOwoh2Di+tWpeImB72QTFJ5zqQf 4QJtJ5y/kfao5Edzwztn8bB2hIn7GCmI3Q90E9H5lH2PvLNEIFfhb2D1z8JEAS3cNfkG AI/5KWKYiR6o+x+0AB5g5/ClPUoBvTWKSOFcBVfaMIS6C6lu+yIgEkBzXGKSzPU/K1ty rbVa07LHzh9/JrBfQ5Ekw0lBibHQL/8nXcwiK5uGhZOuFEXx8mwLErDmlnNn5ew/ag+K zuow== X-Gm-Message-State: AOAM530msb2KJ0ELKX6CHfbvCsDZSsH7h5L4t//w05jXkrvSAA8YLDPm DJscMKVdJaaiccT4hX5jn77sMLzxh/A= X-Google-Smtp-Source: ABdhPJw7amANwpkB7ZYGU0RlMEJtCx99pnhdy4OjawqF0zbW13yCSX1cVqVOCzEfvaxIV88G02T67Q== X-Received: by 2002:a05:6000:2a1:b0:20a:7ae5:70ff with SMTP id l1-20020a05600002a100b0020a7ae570ffmr17414578wry.717.1650487436735; Wed, 20 Apr 2022 13:43:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v2-20020adf8b42000000b0020aa790a258sm712364wra.8.2022.04.20.13.43.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 Apr 2022 13:43:56 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 20 Apr 2022 20:43:18 +0000 Subject: [PATCH v5 28/28] fsmonitor--daemon: allow --super-prefix argument Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a test in t7527 to verify that we get a stray warning from `git fsmonitor--daemon start` when indirectly called from `git submodule absorbgitdirs`. Update `git fsmonitor--daemon` to take (and ignore) the `--super-prefix` argument to suppress the warning. When we have: 1. a submodule with a `sub/.git/` directory (rather than a `sub/.git` file). 2. `core.fsmonitor` is turned on in the submodule, but the daemon is not yet started in the submodule. 3. and someone does a `git submodule absorbgitdirs` in the super. Git will recursively invoke `git submodule--helper absorb-git-dirs` in the submodule. This will read the index and may attempt to start the fsmonitor--daemon with the `--super-prefix` argument. `git fsmonitor--daemon start` does not accept the `--super-prefix` argument and causes a warning to be issued. This does not cause a problem because the `refresh_index()` code assumes a trivial response if the daemon does not start. The net-net is a harmelss, but stray warning. Lets eliminate the warning. Signed-off-by: Jeff Hostetler --- git.c | 2 +- t/t7527-builtin-fsmonitor.sh | 50 ++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/git.c b/git.c index 3d8e48cf555..e6fdac1f8e3 100644 --- a/git.c +++ b/git.c @@ -537,7 +537,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, - { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY }, diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 5476a01cbff..237df6f3c4f 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -832,6 +832,56 @@ test_expect_success "Submodule always visited" ' my_match_and_clean ' +# If a submodule has a `sub/.git/` directory (rather than a file +# pointing to the super's `.git/modules/sub`) and `core.fsmonitor` +# turned on in the submodule and the daemon is not yet started in +# the submodule, and someone does a `git submodule absorbgitdirs` +# in the super, Git will recursively invoke `git submodule--helper` +# to do the work and this may try to read the index. This will +# try to start the daemon in the submodule *and* pass (either +# directly or via inheritance) the `--super-prefix` arg to the +# `git fsmonitor--daemon start` command inside the submodule. +# This causes a warning because fsmonitor--daemon does take that +# global arg (see the table in git.c) +# +# This causes a warning when trying to start the daemon that is +# somewhat confusing. It does not seem to hurt anything because +# the fsmonitor code maps the query failure into a trivial response +# and does the work anyway. +# +# It would be nice to silence the warning, however. + +have_t2_error_event () { + log=$1 + msg="fsmonitor--daemon doesnQt support --super-prefix" && + + tr '\047' Q <$1 | grep -e "$msg" +} + +test_expect_success "Stray Submodule super-prefix warning" ' + test_when_finished "rm -rf super; \ + rm -rf sub; \ + rm super-sub.trace" && + + create_super "super" && + create_sub "sub" && + + # Copy rather than submodule add so that we get a .git dir. + cp -R ./sub ./super/dir_1/dir_2/sub && + + git -C super/dir_1/dir_2/sub config core.fsmonitor true && + + git -C super submodule add ../sub ./dir_1/dir_2/sub && + git -C super commit -m "add sub" && + + test_path_is_dir super/dir_1/dir_2/sub/.git && + + GIT_TRACE2_EVENT="$PWD/super-sub.trace" \ + git -C super submodule absorbgitdirs && + + ! have_t2_error_event super-sub.trace +' + # On a case-insensitive file system, confirm that the daemon # notices when the .git directory is moved/renamed/deleted # regardless of how it is spelled in the the FS event.