From patchwork Wed May 25 15:00:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C902BC433EF for ; Wed, 25 May 2022 15:01:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242616AbiEYPBF (ORCPT ); Wed, 25 May 2022 11:01:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239651AbiEYPBB (ORCPT ); Wed, 25 May 2022 11:01:01 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 936BBAEE29 for ; Wed, 25 May 2022 08:00:58 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id r6-20020a1c2b06000000b00396fee5ebc9so3522553wmr.1 for ; Wed, 25 May 2022 08:00:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=LRnczsdNsb187exmfIDi1yo3KZ/kC2XNTqY5Ck4J+5s=; b=Gm336n8vxDUNZYFOgHW0ToIrDYNAD7OEk20WqLQS5t4JVC0anHKsBvf2H3UXPFbTwa 7d8UFne07Smn2lwV3qugDbiHLPz88443wkp3M7TIinft0r1q4sGHhKWlHEy7Jn8Rsr2n 8QfJJF+esvGD2KDwbm/0wpweZAFZjg62NoZnQ9hk1ovJa46d7lO1y9ScI9uG1gS1VpQA mYzYeqeIk7UJ4sIm1fkX133Wgd15nb1DJ0bjbsHfZrlCKUNgj6+/npy8zRClMW3YV8zr jHFgDfZWZLKCv1S9nOIqAFCY63WAcbTuf1I8qPnXvg9x/fIMyWAV6dUyi+JoNy2KsXUK Q68w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=LRnczsdNsb187exmfIDi1yo3KZ/kC2XNTqY5Ck4J+5s=; b=gXwIh4IpGqpj+iqK5HiGEY8IKBeu4UMSBjtDmNSF++N072xFClZpI/lKcY80p7WIZ4 p9b5gIoVO9FU5YucEWelJV85rM//hw1pFrtsT7zmnM+celGrCPmgUkLzLZrghcPHNW9L weTXwInN6p8MlnMXkOL/Rmzvj0MaXGDkFfv5aJvFZrD1HCbKAoaKIm3Fvqsy3bTQ5KNy 20th2LFYEByxOrhbYvZ+sRQldpYn/+xEmzrZRNX83y3mvFsucSIT3+1iGt1/j7W4ngCZ ZPktnIQG2JG3brpJ3SbT75v7OpS2Y5UQcWCj9hbLnndpwxMyMQ7b9K0Oc0ndd0dMJQ9u ecow== X-Gm-Message-State: AOAM532nqBfDMGvuPNSbi8DyMj1yTCRovWiWJiOsQWD5GNiYaV1GFE4a 9jh1aCFVraJ5qshUrW9yFNeuBlkZeeI= X-Google-Smtp-Source: ABdhPJzO5K1BuFBOkPLxbFvhXpzhXHxY7GE+3o+nu42XI3eNbs4A3aMFsKSMSOUh73S0vi1j856plw== X-Received: by 2002:a7b:c5d1:0:b0:37f:a8a3:9e17 with SMTP id n17-20020a7bc5d1000000b0037fa8a39e17mr8431360wmk.109.1653490856401; Wed, 25 May 2022 08:00:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t17-20020a0560001a5100b0020c5253d914sm2285578wry.96.2022.05.25.08.00.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:00:55 -0700 (PDT) Message-Id: <26144c586599a17c05f27192c3453e6f44c56d3a.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:23 +0000 Subject: [PATCH v8 01/30] fsm-listen-win32: handle shortnames Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach FSMonitor daemon on Windows to recognize shortname paths as aliases of normal longname paths. FSMonitor clients, such as `git status`, should receive the longname spelling of changed files (when possible). Sometimes we receive FS events using the shortname, such as when a CMD shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1". The FS notification arrives using whatever combination of long and shortnames were used by the other process. (Shortnames do seem to be case normalized, however.) Use Windows GetLongPathNameW() to try to map the pathname spelling in the notification event into the normalized longname spelling. (This can fail if the file/directory is deleted, moved, or renamed, because we are asking the FS for the mapping in response to the event and after it has already happened, but we try.) Special case the shortname spelling of ".git" to avoid under-reporting these events. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-win32.c | 386 ++++++++++++++++++++++++---- t/t7527-builtin-fsmonitor.sh | 65 +++++ 2 files changed, 397 insertions(+), 54 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 5b928ab66e5..0a86aea3f7e 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -25,6 +25,9 @@ struct one_watch DWORD count; struct strbuf path; + wchar_t wpath_longname[MAX_PATH + 1]; + DWORD wpath_longname_len; + HANDLE hDir; HANDLE hEvent; OVERLAPPED overlapped; @@ -34,6 +37,21 @@ struct one_watch * need to later call GetOverlappedResult() and possibly CancelIoEx(). */ BOOL is_active; + + /* + * Are shortnames enabled on the containing drive? This is + * always true for "C:/" drives and usually never true for + * other drives. + * + * We only set this for the worktree because we only need to + * convert shortname paths to longname paths for items we send + * to clients. (We don't care about shortname expansion for + * paths inside a GITDIR because we never send them to + * clients.) + */ + BOOL has_shortnames; + BOOL has_tilde; + wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; struct fsmonitor_daemon_backend_data @@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data }; /* - * Convert the WCHAR path from the notification into UTF8 and - * then normalize it. + * Convert the WCHAR path from the event into UTF8 and normalize it. + * + * `wpath_len` is in WCHARS not bytes. */ -static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, +static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len, struct strbuf *normalized_path) { int reserve; int len = 0; strbuf_reset(normalized_path); - if (!info->FileNameLength) + if (!wpath_len) goto normalize; /* @@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, * sequence of 2 UTF8 characters. That should let us * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. */ - reserve = info->FileNameLength + 1; + reserve = 2 * wpath_len + 1; strbuf_grow(normalized_path, reserve); for (;;) { - len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, - info->FileNameLength / sizeof(WCHAR), + len = WideCharToMultiByte(CP_UTF8, 0, + wpath, wpath_len, normalized_path->buf, strbuf_avail(normalized_path) - 1, NULL, NULL); @@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, goto normalize; if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { error(_("[GLE %ld] could not convert path to UTF-8: '%.*ls'"), - GetLastError(), - (int)(info->FileNameLength / sizeof(WCHAR)), - info->FileName); + GetLastError(), (int)wpath_len, wpath); return -1; } @@ -98,6 +115,173 @@ normalize: return strbuf_normalize_path(normalized_path); } +/* + * See if the worktree root directory has shortnames enabled. + * This will help us decide if we need to do an expensive shortname + * to longname conversion on every notification event. + * + * We do not want to create a file to test this, so we assume that the + * root directory contains a ".git" file or directory. (Our caller + * only calls us for the worktree root, so this should be fine.) + * + * Remember the spelling of the shortname for ".git" if it exists. + */ +static void check_for_shortnames(struct one_watch *watch) +{ + wchar_t buf_in[MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + wchar_t *last; + wchar_t *p; + + /* build L"/.git" */ + swprintf(buf_in, ARRAY_SIZE(buf_in) - 1, L"%s.git", + watch->wpath_longname); + + if (!GetShortPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out))) + return; + + /* + * Get the final filename component of the shortpath. + * We know that the path does not have a final slash. + */ + for (last = p = buf_out; *p; p++) + if (*p == L'/' || *p == '\\') + last = p + 1; + + if (!wcscmp(last, L".git")) + return; + + watch->has_shortnames = 1; + wcsncpy(watch->dotgit_shortname, last, + ARRAY_SIZE(watch->dotgit_shortname)); + + /* + * The shortname for ".git" is usually of the form "GIT~1", so + * we should be able to avoid shortname to longname mapping on + * every notification event if the source string does not + * contain a "~". + * + * However, the documentation for GetLongPathNameW() says + * that there are filesystems that don't follow that pattern + * and warns against this optimization. + * + * Lets test this. + */ + if (wcschr(watch->dotgit_shortname, L'~')) + watch->has_tilde = 1; +} + +enum get_relative_result { + GRR_NO_CONVERSION_NEEDED, + GRR_HAVE_CONVERSION, + GRR_SHUTDOWN, +}; + +/* + * Info notification paths are relative to the root of the watch. + * If our CWD is still at the root, then we can use relative paths + * to convert from shortnames to longnames. If our process has a + * different CWD, then we need to construct an absolute path, do + * the conversion, and then return the root-relative portion. + * + * We use the longname form of the root as our basis and assume that + * it already has a trailing slash. + * + * `wpath_len` is in WCHARS not bytes. + */ +static enum get_relative_result get_relative_longname( + struct one_watch *watch, + const wchar_t *wpath, DWORD wpath_len, + wchar_t *wpath_longname, size_t bufsize_wpath_longname) +{ + wchar_t buf_in[2 * MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + DWORD root_len; + DWORD out_len; + + /* + * Build L"/" + * Note that the might not be null terminated + * so we avoid swprintf() constructions. + */ + root_len = watch->wpath_longname_len; + if (root_len + wpath_len >= ARRAY_SIZE(buf_in)) { + /* + * This should not happen. We cannot append the observed + * relative path onto the end of the worktree root path + * without overflowing the buffer. Just give up. + */ + return GRR_SHUTDOWN; + } + wcsncpy(buf_in, watch->wpath_longname, root_len); + wcsncpy(buf_in + root_len, wpath, wpath_len); + buf_in[root_len + wpath_len] = 0; + + /* + * We don't actually know if the source pathname is a + * shortname or a longname. This Windows routine allows + * either to be given as input. + */ + out_len = GetLongPathNameW(buf_in, buf_out, ARRAY_SIZE(buf_out)); + if (!out_len) { + /* + * The shortname to longname conversion can fail for + * various reasons, for example if the file has been + * deleted. (That is, if we just received a + * delete-file notification event and the file is + * already gone, we can't ask the file system to + * lookup the longname for it. Likewise, for moves + * and renames where we are given the old name.) + * + * Since deleting or moving a file or directory by its + * shortname is rather obscure, I'm going ignore the + * failure and ask the caller to report the original + * relative path. This seems kinder than failing here + * and forcing a resync. Besides, forcing a resync on + * every file/directory delete would effectively + * cripple monitoring. + * + * We might revisit this in the future. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (!wcscmp(buf_in, buf_out)) { + /* + * The path does not have a shortname alias. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (wcsncmp(buf_in, buf_out, root_len)) { + /* + * The spelling of the root directory portion of the computed + * longname has changed. This should not happen. Basically, + * it means that we don't know where (without recomputing the + * longname of just the root directory) to split out the + * relative path. Since this should not happen, I'm just + * going to let this fail and force a shutdown (because all + * subsequent events are probably going to see the same + * mismatch). + */ + return GRR_SHUTDOWN; + } + + if (out_len - root_len >= bufsize_wpath_longname) { + /* + * This should not happen. We cannot copy the root-relative + * portion of the path into the provided buffer without an + * overrun. Just give up. + */ + return GRR_SHUTDOWN; + } + + /* Return the worktree root-relative portion of the longname. */ + + wcscpy(wpath_longname, buf_out + root_len); + return GRR_HAVE_CONVERSION; +} + void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); @@ -111,7 +295,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, DWORD share_mode = FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; HANDLE hDir; - wchar_t wpath[MAX_PATH]; + DWORD len_longname; + wchar_t wpath[MAX_PATH + 1]; + wchar_t wpath_longname[MAX_PATH + 1]; if (xutftowcs_path(wpath, path) < 0) { error(_("could not convert to wide characters: '%s'"), path); @@ -128,6 +314,21 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, return NULL; } + len_longname = GetLongPathNameW(wpath, wpath_longname, + ARRAY_SIZE(wpath_longname)); + if (!len_longname) { + error(_("[GLE %ld] could not get longname of '%s'"), + GetLastError(), path); + CloseHandle(hDir); + return NULL; + } + + if (wpath_longname[len_longname - 1] != L'/' && + wpath_longname[len_longname - 1] != L'\\') { + wpath_longname[len_longname++] = L'/'; + wpath_longname[len_longname] = 0; + } + CALLOC_ARRAY(watch, 1); watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ @@ -135,6 +336,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, strbuf_init(&watch->path, 0); strbuf_addstr(&watch->path, path); + wcscpy(watch->wpath_longname, wpath_longname); + watch->wpath_longname_len = len_longname; + watch->hDir = hDir; watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); @@ -258,6 +462,62 @@ static void cancel_rdcw_watch(struct one_watch *watch) watch->is_active = FALSE; } +/* + * Process a single relative pathname event. + * Return 1 if we should shutdown. + */ +static int process_1_worktree_event( + struct string_list *cookie_list, + struct fsmonitor_batch **batch, + const struct strbuf *path, + enum fsmonitor_path_type t, + DWORD info_action) +{ + const char *slash; + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path->buf); + string_list_append(cookie_list, + slash ? slash + 1 : path->buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info_action == FILE_ACTION_REMOVED) || + (info_action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + return 1; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!*batch) + *batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(*batch, path->buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path->buf); + } + + return 0; +} + /* * Process filesystem events that happen anywhere (recursively) under the * root directory. For a normal working directory, this includes @@ -274,6 +534,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) struct string_list cookie_list = STRING_LIST_INIT_DUP; struct fsmonitor_batch *batch = NULL; const char *p = watch->buffer; + wchar_t wpath_longname[MAX_PATH + 1]; /* * If the kernel gets more events than will fit in the kernel @@ -306,54 +567,64 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) */ for (;;) { FILE_NOTIFY_INFORMATION *info = (void *)p; - const char *slash; + wchar_t *wpath = info->FileName; + DWORD wpath_len = info->FileNameLength / sizeof(WCHAR); enum fsmonitor_path_type t; + enum get_relative_result grr; + + if (watch->has_shortnames) { + if (!wcscmp(wpath, watch->dotgit_shortname)) { + /* + * This event exactly matches the + * spelling of the shortname of + * ".git", so we can skip some steps. + * + * (This case is odd because the user + * can "rm -rf GIT~1" and we cannot + * use the filesystem to map it back + * to ".git".) + */ + strbuf_reset(&path); + strbuf_addstr(&path, ".git"); + t = IS_DOT_GIT; + goto process_it; + } - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) - goto skip_this_path; - - t = fsmonitor_classify_path_workdir_relative(path.buf); - - switch (t) { - case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: - /* special case cookie files within .git */ - - /* Use just the filename of the cookie file. */ - slash = find_last_dir_sep(path.buf); - string_list_append(&cookie_list, - slash ? slash + 1 : path.buf); - break; - - case IS_INSIDE_DOT_GIT: - /* ignore everything inside of "/.git/" */ - break; + if (watch->has_tilde && !wcschr(wpath, L'~')) { + /* + * Shortnames on this filesystem have tildes + * and the notification path does not have + * one, so we assume that it is a longname. + */ + goto normalize_it; + } - case IS_DOT_GIT: - /* "/.git" was deleted (or renamed away) */ - if ((info->Action == FILE_ACTION_REMOVED) || - (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { - trace2_data_string("fsmonitor", NULL, - "fsm-listen/dotgit", - "removed"); + grr = get_relative_longname(watch, wpath, wpath_len, + wpath_longname, + ARRAY_SIZE(wpath_longname)); + switch (grr) { + case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */ + break; + case GRR_HAVE_CONVERSION: + wpath = wpath_longname; + wpath_len = wcslen(wpath); + break; + default: + case GRR_SHUTDOWN: goto force_shutdown; } - break; + } - case IS_WORKDIR_PATH: - /* queue normal pathname */ - if (!batch) - batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, path.buf); - break; +normalize_it: + if (normalize_path_in_utf8(wpath, wpath_len, &path) == -1) + goto skip_this_path; - case IS_GITDIR: - case IS_INSIDE_GITDIR: - case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: - default: - BUG("unexpected path classification '%d' for '%s'", - t, path.buf); - } + t = fsmonitor_classify_path_workdir_relative(path.buf); + +process_it: + if (process_1_worktree_event(&cookie_list, &batch, &path, t, + info->Action)) + goto force_shutdown; skip_this_path: if (!info->NextEntryOffset) @@ -382,6 +653,9 @@ force_shutdown: * Note that we DO NOT get filesystem events on the external * itself (it is not inside something that we are watching). In particular, * we do not get an event if the external is deleted. + * + * Also, we do not care about shortnames within the external , since + * we never send these paths to clients. */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { @@ -403,8 +677,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state) const char *slash; enum fsmonitor_path_type t; - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) + if (normalize_path_in_utf8( + info->FileName, + info->FileNameLength / sizeof(WCHAR), + &path) == -1) goto skip_this_path; t = fsmonitor_classify_path_gitdir_relative(path.buf); @@ -538,6 +814,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) if (!data->watch_worktree) goto failed; + check_for_shortnames(data->watch_worktree); + if (state->nr_paths_watching > 1) { data->watch_gitdir = create_watch(state, state->path_gitdir_watch.buf); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index bd0c952a116..1be21785162 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -166,6 +166,71 @@ test_expect_success 'implicit daemon stop (rename .git)' ' test_must_fail git -C test_implicit_2 fsmonitor--daemon status ' +# File systems on Windows may or may not have shortnames. +# This is a volume-specific setting on modern systems. +# "C:/" drives are required to have them enabled. Other +# hard drives default to disabled. +# +# This is a crude test to see if shortnames are enabled +# on the volume containing the test directory. It is +# crude, but it does not require elevation like `fsutil`. +# +test_lazy_prereq SHORTNAMES ' + mkdir .foo && + test -d "FOO~1" +' + +# Here we assume that the shortname of ".git" is "GIT~1". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s" && + + git init test_implicit_1s && + + start_daemon -C test_implicit_1s && + + # renaming the .git directory will implicitly stop the daemon. + # this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + # this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}. + mv test_implicit_1s/.gitxyz test_implicit_1s/.git && + + test_must_fail git -C test_implicit_1s fsmonitor--daemon status +' + +# Here we first create a file with LONGNAME of "GIT~1" before +# we create the repo. This will cause the shortname of ".git" +# to be "GIT~2". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s2" && + + mkdir test_implicit_1s2 && + echo HELLO >test_implicit_1s2/GIT~1 && + git init test_implicit_1s2 && + + test_path_is_file test_implicit_1s2/GIT~1 && + test_path_is_dir test_implicit_1s2/GIT~2 && + + start_daemon -C test_implicit_1s2 && + + # renaming the .git directory will implicitly stop the daemon. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git && + + test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status +' + test_expect_success 'cannot start multiple daemons' ' test_when_finished "stop_daemon_delete_repo test_multiple" && From patchwork Wed May 25 15:00:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861314 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4EF15C433EF for ; Wed, 25 May 2022 15:01:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244931AbiEYPBK (ORCPT ); Wed, 25 May 2022 11:01:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39292 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239692AbiEYPBB (ORCPT ); Wed, 25 May 2022 11:01:01 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F17CAAEE2D for ; Wed, 25 May 2022 08:00:59 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id f23-20020a7bcc17000000b003972dda143eso1264656wmh.3 for ; Wed, 25 May 2022 08:00:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=LKUw9/jQ5pKOTmRH7NqnSRqnbMLjrfCSfR++z8h3F08=; b=iILse8iZ9G/8iiQemdlPrlgjiePD3Uq8mMtI8HnhzhWwrsXyukNswHkV2LLknm1aPg Y6IzsZafZEA7mKD2wS/Nfspkgmke2ej70TXKmYuQewLV1tBDd4/v2q9k2+3pMIkfyXo7 1S6NTSqJSiZXhoWif0PSHpdHDJDNGPrNw8XN7KSam6wyC3S5el+twry7GjVeKj2Lu2zp rsaRadJzMw9rmOVpZvbWycSgWHjxS/qtFAmR6406EKSp2gOG5xwbZs+q99XSno/5Njru bFSQiKMRT5zPY5XzBgIj8/asG9xGRljoI2SyHdvo27s0nq1d203QYh0fKvAdzZzXaXfY gOvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=LKUw9/jQ5pKOTmRH7NqnSRqnbMLjrfCSfR++z8h3F08=; b=vqLLkhRSKOkelaxScz2Kr2h/kDvGTxEAjhgsD7ECC6V2BS2viFeKI2TcmNkdj68onw 29rgHkLvep0UgGriNvUj+S379u2vK0b+15h3LqkiQ8//+3Jdb53Fd7A+O9xtAcYTUwen egD111UPdyeDnNuHFjnqNWv2PWOamrLLfzyJbs5MZy59ZiF/NLxIbiRytekQKa74i3SA SMjL1gGfymt6Lj9scM/leiCkxkdkVtk2WbC0dM+onFjXbaMIO6cCzs1mrV8AoRIIuIPF t5VEoSe14UDUAsIgYtk3DHEaot9amvQUKuzSEuF6Uu6bZEe7OAaZ+3g2xojy2su7RcWE QcJQ== X-Gm-Message-State: AOAM532ADXlp2V5hNt/+/2yflL7gTF4xnxDfhDkNUXrXXlIOsG59dcfn o7azZHPFpYzmoZT97f1DL1tX62cBTBg= X-Google-Smtp-Source: ABdhPJwKzkHA2JWCUQqLhuX2Fn5LbB30qa16FYs4F8Gez8xfxrGNkKKNGT/v2+kLsOkjSW5ixgMs/A== X-Received: by 2002:a7b:c183:0:b0:397:4924:7cfe with SMTP id y3-20020a7bc183000000b0039749247cfemr8690661wmi.128.1653490858048; Wed, 25 May 2022 08:00:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k5-20020a5d5245000000b0020ff7246934sm2245202wrc.95.2022.05.25.08.00.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:00:56 -0700 (PDT) Message-Id: <1bf2e36b6adb127dd1654bba0b6ba84f626535a0.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:24 +0000 Subject: [PATCH v8 02/30] t7527: test FSMonitor on repos with Unicode root paths Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create some test repos with UTF8 characters in the pathname of the root directory and verify that the builtin FSMonitor can watch them. This test is mainly for Windows where we need to avoid `*A()` routines. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 1be21785162..12655958e71 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -671,4 +671,27 @@ do done done +# Test Unicode UTF-8 characters in the pathname of the working +# directory root. Use of "*A()" routines rather than "*W()" routines +# on Windows can sometimes lead to odd failures. +# +u1=$(printf "u_c3_a6__\xC3\xA6") +u2=$(printf "u_e2_99_ab__\xE2\x99\xAB") +u_values="$u1 $u2" +for u in $u_values +do + test_expect_success "unicode in repo root path: $u" ' + test_when_finished "stop_daemon_delete_repo $u" && + + git init "$u" && + echo 1 >"$u"/file1 && + git -C "$u" add file1 && + git -C "$u" config core.fsmonitor true && + + start_daemon -C "$u" && + git -C "$u" status >actual && + grep "new file: file1" actual + ' +done + test_done From patchwork Wed May 25 15:00:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861315 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47E72C433EF for ; Wed, 25 May 2022 15:01:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240200AbiEYPBM (ORCPT ); Wed, 25 May 2022 11:01:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39310 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S242575AbiEYPBC (ORCPT ); Wed, 25 May 2022 11:01:02 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 76D8BAEE37 for ; Wed, 25 May 2022 08:01:01 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id 67-20020a1c1946000000b00397382b44f4so3519628wmz.2 for ; Wed, 25 May 2022 08:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8V1FoXCb86xqZ2XOaLWrGr4avTXqor6TX1WNeRZ9avw=; b=BLchjQjw4Z71g7sehYdFQETuJoxcWAWS+RbHxaHy7FHGgb5IGAhUvIgXMU4IZEOOA4 pOS7MDPAFB2TjaCpNoeuKhCEKTeOljpw8MnKzYoEx5DcawlvbQdMjjGnHspotLOX1pzU WjlaewJV9y25lgJaw+9reSy1t5m4QcBpu6O8Em0bu38MrQr3IJAtVdEUwwWy8Wlr2LXx OB5qD3cfgJatnnuAvIJuq5SvfxcYVpSwv+QkqWisQ1i5/DtBsIbFV9HZJ0+Pk9i5K83w GRwpQR8tMlCcpOmNPCB6bSvaqeZXHTAu/s+D5EDkozv6u8uIWu7zobX7BCypfhyt2QM0 Dd4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8V1FoXCb86xqZ2XOaLWrGr4avTXqor6TX1WNeRZ9avw=; b=vyXFNrd8neI0sx5Twfv1NK83F6TKMdgULH2ZlWSiZau6MO6ZcHTt64YPQ4rUWPvYbH lhsqmho5xogRHnjCWML+uwaPHWCkr0lo32By+tRYmkoFuCbOWSivYPJFB3zbqOht1bwS cNCRdIbVwXq6+jCnaSab4Zn2JJWbXF1KW8am2PpwyxvcnTy7RbLhV6vQFP9nI1QFUyCF T52/PXISEisTbX1Np2dNL6h8YwgBkkhopcIpB1wEN6RvGo8PMrTfz7qeVsBTHXFs7WwC TpbpVP6LeHg2trpiT5GOOsLrJ3SWgrB/PU+9YGqWrw8gcksGgITRcBM0MvFt5im+LtxR oAQQ== X-Gm-Message-State: AOAM532EZmGPiF9a6VEUOKyeExxkyuqu7oxbqI2U9JSqz+cJrEshL/K4 5byook7miktIxZdcfh9H6zxBtGc/iZ0= X-Google-Smtp-Source: ABdhPJwkvHuQAZNY0g8liD7Xxy7q9ld1KdU5Wnn1UObXTaAvfH+L9HZszs6xkjc4KxTuLZCdrOzI0w== X-Received: by 2002:a1c:7912:0:b0:397:41e9:872d with SMTP id l18-20020a1c7912000000b0039741e9872dmr8574504wme.184.1653490859543; Wed, 25 May 2022 08:00:59 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h3-20020a05600c350300b0039456fb80b3sm266832wmq.43.2022.05.25.08.00.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:00:58 -0700 (PDT) Message-Id: <4bca494bb22365fd9ccc59f6470c5124f74caf84.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:25 +0000 Subject: [PATCH v8 03/30] t/helper/fsmonitor-client: create stress test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a stress test to hammer on the fsmonitor daemon. Create a client-side thread pool of n threads and have each of them make m requests as fast as they can. We do not currently inspect the contents of the response. We're only interested in placing a heavy request load on the daemon. This test is useful for interactive testing and various experimentation. For example, to place additional load on the daemon while another test is running. We currently do not have a test script that actually uses this helper. We might add such a test in the future. Signed-off-by: Jeff Hostetler --- t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c index 3062c8a3c2b..54a4856c48c 100644 --- a/t/helper/test-fsmonitor-client.c +++ b/t/helper/test-fsmonitor-client.c @@ -7,6 +7,8 @@ #include "cache.h" #include "parse-options.h" #include "fsmonitor-ipc.h" +#include "thread-utils.h" +#include "trace2.h" #ifndef HAVE_FSMONITOR_DAEMON_BACKEND int cmd__fsmonitor_client(int argc, const char **argv) @@ -79,20 +81,121 @@ static int do_send_flush(void) return 0; } +struct hammer_thread_data +{ + pthread_t pthread_id; + int thread_nr; + + int nr_requests; + const char *token; + + int sum_successful; + int sum_errors; +}; + +static void *hammer_thread_proc(void *_hammer_thread_data) +{ + struct hammer_thread_data *data = _hammer_thread_data; + struct strbuf answer = STRBUF_INIT; + int k; + int ret; + + trace2_thread_start("hammer"); + + for (k = 0; k < data->nr_requests; k++) { + strbuf_reset(&answer); + + ret = fsmonitor_ipc__send_query(data->token, &answer); + if (ret < 0) + data->sum_errors++; + else + data->sum_successful++; + } + + strbuf_release(&answer); + trace2_thread_exit(); + return NULL; +} + +/* + * Start a pool of client threads that will each send a series of + * commands to the daemon. + * + * The goal is to overload the daemon with a sustained series of + * concurrent requests. + */ +static int do_hammer(const char *token, int nr_threads, int nr_requests) +{ + struct hammer_thread_data *data = NULL; + int k; + int sum_join_errors = 0; + int sum_commands = 0; + int sum_errors = 0; + + if (!token || !*token) + token = get_token_from_index(); + if (nr_threads < 1) + nr_threads = 1; + if (nr_requests < 1) + nr_requests = 1; + + CALLOC_ARRAY(data, nr_threads); + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + p->thread_nr = k; + p->nr_requests = nr_requests; + p->token = token; + + if (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) { + warning("failed to create thread[%d] skipping remainder", k); + nr_threads = k; + break; + } + } + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + + if (pthread_join(p->pthread_id, NULL)) + sum_join_errors++; + sum_commands += p->sum_successful; + sum_errors += p->sum_errors; + } + + fprintf(stderr, "HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\n", + nr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors); + + free(data); + + /* + * Return an error if any of the _send_query requests failed. + * We don't care about thread create/join errors. + */ + return sum_errors > 0; +} + int cmd__fsmonitor_client(int argc, const char **argv) { const char *subcmd; const char *token = NULL; + int nr_threads = 1; + int nr_requests = 1; const char * const fsmonitor_client_usage[] = { "test-tool fsmonitor-client query []", "test-tool fsmonitor-client flush", + "test-tool fsmonitor-client hammer [] [] []", NULL, }; struct option options[] = { OPT_STRING(0, "token", &token, "token", "command token to send to the server"), + + OPT_INTEGER(0, "threads", &nr_threads, "number of client threads"), + OPT_INTEGER(0, "requests", &nr_requests, "number of requests per thread"), + OPT_END() }; @@ -111,6 +214,9 @@ int cmd__fsmonitor_client(int argc, const char **argv) if (!strcmp(subcmd, "flush")) return !!do_send_flush(); + if (!strcmp(subcmd, "hammer")) + return !!do_hammer(token, nr_threads, nr_requests); + die("Unhandled subcommand: '%s'", subcmd); } #endif From patchwork Wed May 25 15:00:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72C69C433F5 for ; Wed, 25 May 2022 15:01:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244947AbiEYPBZ (ORCPT ); Wed, 25 May 2022 11:01:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243547AbiEYPBG (ORCPT ); Wed, 25 May 2022 11:01:06 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84F47AEE0E for ; Wed, 25 May 2022 08:01:03 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id y24so4894665wmq.5 for ; Wed, 25 May 2022 08:01:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/Rt8jZtS+yiXhlKyKszzHqUFdx87AzLiQnNRSQBgwxQ=; b=p7Q/+vId14lkhxyl9ox2QrSB0vBAFSUU74462InxXIUyDke4ToRHpVv5ozaqJaK7fJ 4Z8j7UivYNEySo6elpe6ATaXVa2zSdvwXYMvVyReV974EXOzAT2g1vzQ6Kzm7l8rDBcl cRu5diQjkwOoW8mVm7daz2bgAC08SppI92a1OARUv4/4pBZLOhh13rZ+iJU01gWx/wMV BKo3a0pOLMbMpT9gTPiypcQk8qZTxKE9+hSTNydsNx0etJ9ejS2cFd9tRhKdsHAXrZfb IqB7r5OdMEYsoxffJOdtCd/CMtqiBzkU0Ab3VPMBQc9qyMWIj8H6+2me3rKuQhxmwLc8 9BZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/Rt8jZtS+yiXhlKyKszzHqUFdx87AzLiQnNRSQBgwxQ=; b=HArHmfY8fx9mtex2D1zTfVKjEqIIg7rUpjyjFNkh/Fo3NLAkqOVPekXyqzjDPQRDeU ZmWhuBLKKzlmlSwUX9PvixYizurf7SiCZpgGesnTLafzZm9V6eBL0+1R7Wz63YtswfF1 J4t7vAgTSKpvpyZkkegeCujPnBnWKjoMGqk2i3c1wlrld32STkA5Wvy7bfgbO3TJfXao b5Q5YjKD15lj+ivTW+Y90wWYN4muczhBqFHBen4HspF65GMaNqSWcItTNaLixqNZEQne 7m0ArrdZOTyQ9ymP65QeGYcminG2rAPQFsSh9jp7vSDNkhbZihfkcK19WhJVZCUsFDd5 xrJw== X-Gm-Message-State: AOAM532dbfQF9neJB+1vZxeHbr3CKvZXy+7fnHGLtCc9IEF9pOPPy4ow NT/s/NM15H9J/q4Mhz2jIBQ/TWICKSk= X-Google-Smtp-Source: ABdhPJwS8C+oq3LFj98xszevF3tAp2cGV4zYdwzshR3XGrW2UbIJ3HXt20rxO0D77rMegl1yCsIp9g== X-Received: by 2002:a7b:c00e:0:b0:395:b7f2:103b with SMTP id c14-20020a7bc00e000000b00395b7f2103bmr8649221wmb.206.1653490861442; Wed, 25 May 2022 08:01:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bi19-20020a05600c3d9300b003942a244f39sm2231168wmb.18.2022.05.25.08.00.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:00 -0700 (PDT) Message-Id: <663deabc3f67c25fb6888871b779b0c8f25ac75b.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:26 +0000 Subject: [PATCH v8 04/30] fsmonitor-settings: bare repos are incompatible with FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Bare repos do not have a worktree, so there is nothing for the daemon watch. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 18 +++++ builtin/update-index.c | 16 +++++ fsmonitor-settings.c | 133 ++++++++++++++++++++++++++++++------ fsmonitor-settings.h | 16 +++++ t/t7519-status-fsmonitor.sh | 23 +++++++ 5 files changed, 186 insertions(+), 20 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 46be55a4618..66b78a0353f 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1423,6 +1423,7 @@ static int try_to_start_background_daemon(void) int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { const char *subcmd; + enum fsmonitor_reason reason; int detach_console = 0; struct option options[] = { @@ -1449,6 +1450,23 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) die(_("invalid 'ipc-threads' value (%d)"), fsmonitor__ipc_threads); + prepare_repo_settings(the_repository); + /* + * If the repo is fsmonitor-compatible, explicitly set IPC-mode + * (without bothering to load the `core.fsmonitor` config settings). + * + * If the repo is not compatible, the repo-settings will be set to + * incompatible rather than IPC, so we can use one of the __get + * routines to detect the discrepancy. + */ + fsm_settings__set_ipc(the_repository); + + reason = fsm_settings__get_reason(the_repository); + if (reason > FSMONITOR_REASON_OK) + die("%s", + fsm_settings__get_incompatible_msg(the_repository, + reason)); + if (!strcmp(subcmd, "start")) return !!try_to_start_background_daemon(); diff --git a/builtin/update-index.c b/builtin/update-index.c index 876112abb21..01ed4c4976b 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1237,6 +1237,22 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) if (fsmonitor > 0) { enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r); + enum fsmonitor_reason reason = fsm_settings__get_reason(r); + + /* + * The user wants to turn on FSMonitor using the command + * line argument. (We don't know (or care) whether that + * is the IPC or HOOK version.) + * + * Use one of the __get routines to force load the FSMonitor + * config settings into the repo-settings. That will detect + * whether the file system is compatible so that we can stop + * here with a nice error message. + */ + if (reason > FSMONITOR_REASON_OK) + die("%s", + fsm_settings__get_incompatible_msg(r, reason)); + if (fsm_mode == FSMONITOR_MODE_DISABLED) { warning(_("core.fsmonitor is unset; " "set it if you really want to " diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 757d230d538..7d3177d441a 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -9,23 +9,42 @@ */ struct fsmonitor_settings { enum fsmonitor_mode mode; + enum fsmonitor_reason reason; char *hook_path; }; -static void lookup_fsmonitor_settings(struct repository *r) +static enum fsmonitor_reason check_for_incompatible(struct repository *r) +{ + if (!r->worktree) { + /* + * Bare repositories don't have a working directory and + * therefore have nothing to watch. + */ + return FSMONITOR_REASON_BARE; + } + + return FSMONITOR_REASON_OK; +} + +static struct fsmonitor_settings *alloc_settings(void) { struct fsmonitor_settings *s; + + CALLOC_ARRAY(s, 1); + s->mode = FSMONITOR_MODE_DISABLED; + s->reason = FSMONITOR_REASON_UNTESTED; + + return s; +} + +static void lookup_fsmonitor_settings(struct repository *r) +{ const char *const_str; int bool_value; if (r->settings.fsmonitor) return; - CALLOC_ARRAY(s, 1); - s->mode = FSMONITOR_MODE_DISABLED; - - r->settings.fsmonitor = s; - /* * Overload the existing "core.fsmonitor" config setting (which * has historically been either unset or a hook pathname) to @@ -38,6 +57,8 @@ static void lookup_fsmonitor_settings(struct repository *r) case 0: /* config value was set to */ if (bool_value) fsm_settings__set_ipc(r); + else + fsm_settings__set_disabled(r); return; case 1: /* config value was unset */ @@ -53,18 +74,18 @@ static void lookup_fsmonitor_settings(struct repository *r) return; } - if (!const_str || !*const_str) - return; - - fsm_settings__set_hook(r, const_str); + if (const_str && *const_str) + fsm_settings__set_hook(r, const_str); + else + fsm_settings__set_disabled(r); } enum fsmonitor_mode fsm_settings__get_mode(struct repository *r) { if (!r) r = the_repository; - - lookup_fsmonitor_settings(r); + if (!r->settings.fsmonitor) + lookup_fsmonitor_settings(r); return r->settings.fsmonitor->mode; } @@ -73,31 +94,55 @@ const char *fsm_settings__get_hook_path(struct repository *r) { if (!r) r = the_repository; - - lookup_fsmonitor_settings(r); + if (!r->settings.fsmonitor) + lookup_fsmonitor_settings(r); return r->settings.fsmonitor->hook_path; } void fsm_settings__set_ipc(struct repository *r) { + enum fsmonitor_reason reason = check_for_incompatible(r); + + if (reason != FSMONITOR_REASON_OK) { + fsm_settings__set_incompatible(r, reason); + return; + } + + /* + * Caller requested IPC explicitly, so avoid (possibly + * recursive) config lookup. + */ if (!r) r = the_repository; - - lookup_fsmonitor_settings(r); + if (!r->settings.fsmonitor) + r->settings.fsmonitor = alloc_settings(); r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC; + r->settings.fsmonitor->reason = reason; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } void fsm_settings__set_hook(struct repository *r, const char *path) { + enum fsmonitor_reason reason = check_for_incompatible(r); + + if (reason != FSMONITOR_REASON_OK) { + fsm_settings__set_incompatible(r, reason); + return; + } + + /* + * Caller requested hook explicitly, so avoid (possibly + * recursive) config lookup. + */ if (!r) r = the_repository; - - lookup_fsmonitor_settings(r); + if (!r->settings.fsmonitor) + r->settings.fsmonitor = alloc_settings(); r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK; + r->settings.fsmonitor->reason = reason; FREE_AND_NULL(r->settings.fsmonitor->hook_path); r->settings.fsmonitor->hook_path = strdup(path); } @@ -106,9 +151,57 @@ void fsm_settings__set_disabled(struct repository *r) { if (!r) r = the_repository; - - lookup_fsmonitor_settings(r); + if (!r->settings.fsmonitor) + r->settings.fsmonitor = alloc_settings(); r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED; + r->settings.fsmonitor->reason = FSMONITOR_REASON_OK; + FREE_AND_NULL(r->settings.fsmonitor->hook_path); +} + +void fsm_settings__set_incompatible(struct repository *r, + enum fsmonitor_reason reason) +{ + if (!r) + r = the_repository; + if (!r->settings.fsmonitor) + r->settings.fsmonitor = alloc_settings(); + + r->settings.fsmonitor->mode = FSMONITOR_MODE_INCOMPATIBLE; + r->settings.fsmonitor->reason = reason; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } + +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r) +{ + if (!r) + r = the_repository; + if (!r->settings.fsmonitor) + lookup_fsmonitor_settings(r); + + return r->settings.fsmonitor->reason; +} + +char *fsm_settings__get_incompatible_msg(const struct repository *r, + enum fsmonitor_reason reason) +{ + struct strbuf msg = STRBUF_INIT; + + switch (reason) { + case FSMONITOR_REASON_UNTESTED: + case FSMONITOR_REASON_OK: + goto done; + + case FSMONITOR_REASON_BARE: + strbuf_addf(&msg, + _("bare repository '%s' is incompatible with fsmonitor"), + xgetcwd()); + goto done; + } + + BUG("Unhandled case in fsm_settings__get_incompatible_msg: '%d'", + reason); + +done: + return strbuf_detach(&msg, NULL); +} diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index a4c5d7b4889..8d9331c0c0a 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -4,18 +4,34 @@ struct repository; enum fsmonitor_mode { + FSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */ FSMONITOR_MODE_DISABLED = 0, FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */ FSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */ }; +/* + * Incompatibility reasons. + */ +enum fsmonitor_reason { + FSMONITOR_REASON_UNTESTED = 0, + FSMONITOR_REASON_OK, /* no incompatibility or when disabled */ + FSMONITOR_REASON_BARE, +}; + void fsm_settings__set_ipc(struct repository *r); void fsm_settings__set_hook(struct repository *r, const char *path); void fsm_settings__set_disabled(struct repository *r); +void fsm_settings__set_incompatible(struct repository *r, + enum fsmonitor_reason reason); enum fsmonitor_mode fsm_settings__get_mode(struct repository *r); const char *fsm_settings__get_hook_path(struct repository *r); +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r); +char *fsm_settings__get_incompatible_msg(const struct repository *r, + enum fsmonitor_reason reason); + struct fsmonitor_settings; #endif /* FSMONITOR_SETTINGS_H */ diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index a6308acf006..9a8e21c5608 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE ' test $ret -ne 1 ' +# Test that we detect and disallow repos that are incompatible with FSMonitor. +test_expect_success 'incompatible bare repo' ' + test_when_finished "rm -rf ./bare-clone actual expect" && + git init --bare bare-clone && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=foo \ + update-index --fsmonitor 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=true \ + update-index --fsmonitor 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual +' + +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' + test_when_finished "rm -rf ./bare-clone actual" && + git init --bare bare-clone && + test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual && + grep "bare repository .* is incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Wed May 25 15:00:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BBD8C433EF for ; Wed, 25 May 2022 15:01:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244979AbiEYPBa (ORCPT ); Wed, 25 May 2022 11:01:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244159AbiEYPBG (ORCPT ); Wed, 25 May 2022 11:01:06 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22EE3AEE3E for ; Wed, 25 May 2022 08:01:05 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id v4-20020a1cac04000000b00397001398c0so1256931wme.5 for ; Wed, 25 May 2022 08:01:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=V2qiF7m8aSYTil3cDpJgcVX4oQxYDeEfetoj3qel4cE=; b=RCl+ie3uB/wxB9tc3Z2xxviaGnMoqLWVy0PiU1aOicU7MFFUTl4s+Yz04HC6Xd9xgM f2Khz6ig0DV3PVPIrEjDN8Zk4IZ+xbQrofcIzfshL25utS1Xm55OBm+CAIsVcnrl7PWq CGnnbmRNmDP7cDh3hWcvvLmEkHp1plF5yD/GTOYzjAI/Od62rN9152xBjlR9zU2j4i8U 0vxvZy5V6pb0Uo6YUpgotbsvukejeX1+CX9JadtQyf6Px06NSwrq+mP7oEFfm1BMJ6sF t90HbOM5+t/mFut3KWdQvwqjz5XI2qXdnBy6e8hOtjscbIdO35fm2vIMlFfZ+bavCKuu +XvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=V2qiF7m8aSYTil3cDpJgcVX4oQxYDeEfetoj3qel4cE=; b=trxiFHW+7tbxTRuCA92YWs5Ky4HpsUqh3d/2ThrnY+4lU1HMqJPIE855vrCw+VHwOq KxTfZDC7lX4kBhm35Y97PpRRkPXzvZFxeDz7xQJzDuRpVnHpXCtRY03JqeW4M5y05yaL tsTUDVOLSmyb0GxxCTPD1WSLV7FtITm1Mo6tfgFYLInMuXMPRO70O8USJlAx4xGYSPSP d7AO5a9cTdYxE6edXpVOEwxNRNsqkMdTxPXAyM24XOCLqyqYGM9mVjOHf4VB7Mi61jDi tfBJx1xWgrogsL9KnZ64SquVY9sVBNZLUyr/ndHkqoEqAKlo5FAcgfuus/yoLHs0n2xU +ZMA== X-Gm-Message-State: AOAM532pjVCdRZi7y6HBjo4Yx2tniWyI+oTi/jfidiTqxl6GV5gVY/8V qbCVnkGfrm55+l7L/4Fj5/fOPkt/uyc= X-Google-Smtp-Source: ABdhPJzwXCbkpZibU1OfMg1Aokmo04UCPP/OPpHTX1wIAeOtsxcITtPWfctDUVS8nv+P2ok5aCbWWw== X-Received: by 2002:a7b:c451:0:b0:397:4c13:1873 with SMTP id l17-20020a7bc451000000b003974c131873mr8554954wmi.151.1653490863033; Wed, 25 May 2022 08:01:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m6-20020a7bce06000000b003944821105esm2385853wmc.2.2022.05.25.08.01.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:02 -0700 (PDT) Message-Id: <7cb0180a1ed95f9ebecc605cd1d968bced46d375.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:27 +0000 Subject: [PATCH v8 05/30] fsmonitor-settings: stub in Win32-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend generic incompatibility checkout with platform-specific mechanism. Stub in Win32 version. In the existing fsmonitor-settings code we have a way to mark types of repos as incompatible with fsmonitor (whether via the hook and IPC APIs). For example, we do this for bare repos, since there are no files to watch. Extend this exclusion mechanism for platform-specific reasons. This commit just creates the framework and adds a stub for Win32. Signed-off-by: Jeff Hostetler --- Makefile | 13 +++++++++++++ compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++ config.mak.uname | 4 ++++ contrib/buildsystems/CMakeLists.txt | 3 +++ fsmonitor-settings.c | 10 ++++++++++ fsmonitor-settings.h | 13 +++++++++++++ 6 files changed, 52 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-win32.c diff --git a/Makefile b/Makefile index daa21bed6c3..93604fe8ef7 100644 --- a/Makefile +++ b/Makefile @@ -475,6 +475,11 @@ all:: # `compat/fsmonitor/fsm-listen-.c` that implements the # `fsm_listen__*()` routines. # +# If your platform has OS-specific ways to tell if a repo is incompatible with +# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS +# to the "" of the corresponding `compat/fsmonitor/fsm-settings-.c` +# that implements the `fsm_os_settings__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o endif +ifdef FSMONITOR_OS_SETTINGS + COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE ifdef FSMONITOR_DAEMON_BACKEND @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ endif +ifdef FSMONITOR_OS_SETTINGS + @echo FSMONITOR_OS_SETTINGS=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_OS_SETTINGS)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index 501970902da..cf224768ad6 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index ee0d7257b77..16705da2000 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 7d3177d441a..f67db913f57 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -23,6 +23,16 @@ static enum fsmonitor_reason check_for_incompatible(struct repository *r) return FSMONITOR_REASON_BARE; } +#ifdef HAVE_FSMONITOR_OS_SETTINGS + { + enum fsmonitor_reason reason; + + reason = fsm_os__incompatible(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + } +#endif + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 8d9331c0c0a..6cb0d8e7d9f 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -34,4 +34,17 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r, struct fsmonitor_settings; +#ifdef HAVE_FSMONITOR_OS_SETTINGS +/* + * Ask platform-specific code whether the repository is incompatible + * with fsmonitor (both hook and ipc modes). For example, if the working + * directory is on a remote volume and mounted via a technology that does + * not support notification events, then we should not pretend to watch it. + * + * fsm_os__* routines should considered private to fsm_settings__ + * routines. + */ +enum fsmonitor_reason fsm_os__incompatible(struct repository *r); +#endif /* HAVE_FSMONITOR_OS_SETTINGS */ + #endif /* FSMONITOR_SETTINGS_H */ From patchwork Wed May 25 15:00:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6957C433F5 for ; Wed, 25 May 2022 15:01:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245021AbiEYPBh (ORCPT ); Wed, 25 May 2022 11:01:37 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244924AbiEYPBI (ORCPT ); Wed, 25 May 2022 11:01:08 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 846622E4 for ; Wed, 25 May 2022 08:01:06 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id n124-20020a1c2782000000b003972dfca96cso3513233wmn.4 for ; Wed, 25 May 2022 08:01:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=KflqYF0+S7RB9kOFxgFWU4lLQFA5eX5O/Scn3B7vbzU=; b=n08b//KAOZrttEBN+2A7dqXh6nbWbBaBcKA4z/1zviqTWQ8/dZGsjPIFprdNwQhoZx 4hQ1xi0W08QSV45VPK3I47VGt+g3xOKX4AsSHM4+AdWjO9KkOuThvlPTcoFTdGMUT0Yk FIMHd4HlBd8e7gn53Wr6SJRn5pLHco9MJA7JhyoaJiTq6T1LdMcdqBVevWJ0x9GAovFc ibkM64rRNsWMUVdp+7+gyvA1z24jGJmEEYGQMsRJzRh55ZlxpSmdjWWKdvM6Z7a9WGqV d9NESC9bOVYByyr/L9j0fFdE3ST/fADuFMfDhEQANNiw9WcKrXkJlF6Ho+6wUZMT55Uy sLsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=KflqYF0+S7RB9kOFxgFWU4lLQFA5eX5O/Scn3B7vbzU=; b=qU8yBVelPwV60KiL/gkjAJ4mxvtQJCm8TitspBP8j6rIPpJ+1vB/yRINBWXfGjSiUs AYCUyz4ioCwkCRTNKNTdvjFlkE5uc2neVQ4WBNu9/m7NDSMfY7h5Ff9Jmi2dCiD9XcM3 azXJXHn3ed/UhUsVC8r72vZBf5rkLDfjyHZUE9+4qLPIz2LFrJp6+OeY7fojiomiFoK8 ovEnQEV+OzEeM2v/TkOJHjES1o+WAe5ILe9SalphHipZxDy+jMt/TJlvcpJnAtVfqZOW xq/sM9D74KMUnRqHXrmtmuUvKFFDqxh8TY+mC71sFFKSm17dBJu+7empmp4xF8n1eDXn E9Bw== X-Gm-Message-State: AOAM532n3c0uhF/httkqunyRDgbLgMut5J8gGghe2n2Ja33aXfJUWlE+ wZuA0KMlTz3pvpMSVNF7y4n6hv1UUGQ= X-Google-Smtp-Source: ABdhPJzB6aWkpwbUdD/8Nq1sXnIOKYWIp94WlSLWKBQ6VIN352BlHHdKqJN8tSaFtCUnvaGyKWxlJQ== X-Received: by 2002:a05:600c:3ba4:b0:397:49b8:262 with SMTP id n36-20020a05600c3ba400b0039749b80262mr8582179wms.77.1653490864660; Wed, 25 May 2022 08:01:04 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t22-20020a1c7716000000b00397381a7ae8sm1960053wmi.30.2022.05.25.08.01.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:04 -0700 (PDT) Message-Id: <9774faddc45cd33362aeb1e3ea80125bc496cbc9.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:28 +0000 Subject: [PATCH v8 06/30] fsmonitor-settings: VFS for Git virtual repos are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler VFS for Git virtual repositories are incompatible with FSMonitor. VFS for Git is a downstream fork of Git. It contains its own custom file system watcher that is aware of the virtualization. If a working directory is being managed by VFS for Git, we should not try to watch it because we may get incomplete results. We do not know anything about how VFS for Git works, but we do know that VFS for Git working directories contain a well-defined config setting. If it is set, mark the working directory as incompatible. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++ fsmonitor-settings.c | 6 ++++++ fsmonitor-settings.h | 1 + t/t7519-status-fsmonitor.sh | 9 +++++++++ 4 files changed, 42 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index 7fce32a3c5b..ee78bba38e3 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -3,7 +3,33 @@ #include "repository.h" #include "fsmonitor-settings.h" +/* + * VFS for Git is incompatible with FSMonitor. + * + * Granted, core Git does not know anything about VFS for Git and we + * shouldn't make assumptions about a downstream feature, but users + * can install both versions. And this can lead to incorrect results + * from core Git commands. So, without bringing in any of the VFS for + * Git code, do a simple config test for a published config setting. + * (We do not look at the various *_TEST_* environment variables.) + */ +static enum fsmonitor_reason check_vfs4git(struct repository *r) +{ + const char *const_str; + + if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str)) + return FSMONITOR_REASON_VFS4GIT; + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_vfs4git(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index f67db913f57..600ae165ab1 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -207,6 +207,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r, _("bare repository '%s' is incompatible with fsmonitor"), xgetcwd()); goto done; + + case FSMONITOR_REASON_VFS4GIT: + strbuf_addf(&msg, + _("virtual repository '%s' is incompatible with fsmonitor"), + r->worktree); + goto done; } BUG("Unhandled case in fsm_settings__get_incompatible_msg: '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 6cb0d8e7d9f..a48802cde9c 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -17,6 +17,7 @@ enum fsmonitor_reason { FSMONITOR_REASON_UNTESTED = 0, FSMONITOR_REASON_OK, /* no incompatibility or when disabled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; void fsm_settings__set_ipc(struct repository *r); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 9a8e21c5608..156895f9437 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' grep "bare repository .* is incompatible with fsmonitor" actual ' +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' ' + test_when_finished "rm -rf ./fake-virtual-clone actual" && + git init fake-virtual-clone && + test_must_fail git -C ./fake-virtual-clone \ + -c core.virtualfilesystem=true \ + fsmonitor--daemon run 2>actual && + grep "virtual repository .* is incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Wed May 25 15:00:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861329 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 04D86C433EF for ; Wed, 25 May 2022 15:02:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231436AbiEYPCF (ORCPT ); Wed, 25 May 2022 11:02:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244948AbiEYPBY (ORCPT ); Wed, 25 May 2022 11:01:24 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B7E426EE for ; Wed, 25 May 2022 08:01:06 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id s28so30367568wrb.7 for ; Wed, 25 May 2022 08:01:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=PRw8FPAivf1H20GHcN+EjZvkU8FPOATe/G+l6/KMwT73zmNiBQz3pAWPKlFZbsFg7h dIuUwnfWLRF5/AispLRMYYReBvSbRU8n9k/dLWayKzpd7wYoRSfbvso9PWCb5nwLEf/1 qvp43xmZYe3fQgyX/G4uNRV79meJROkcYEMJ2aiwBBtPqc+E9sdl7E67VKTk3Diuk9/k kKQ82/YLyfTBcWAVCIW79NWF/dyd6cdNYLK00Wp5mvq4Ky9KonUs/0/yPyHa9+XWPUyX uJYwGNLcxf3zlVhRKkQyF4Qb7gQ333sjepOQoBovo0VyQhvtskd7M9lDJJtoLU6m/IrD J9dQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=lVmI7jmFpHFRzXYNvsjedrZOs2T5bmQWsJR0sSyCIStXdmXUD/+bKlrK7iFy5sYcKW dudQ/jwEScG4PIbEa7y6pDw2ANciAT+27OlQHoUd9gBOsi3jJe0ynE5YfMKxzQwGbVle LHerlgym17w+sFUyUidOwrzLRD9VQl9VVBixco7dfbsWGUaxrUmyXd6myRlSl0nU3VlZ R6TL/4osxYYVtAHSTD6WKqr+PL5og4k1A33ynWQFi9VHb2ouyh7NYYB46MjcqHpJRkjG sMet4uyiGF3EDdRMgfj/SPoV+XR4dM5tquKl3V/aTpTesrTknp2Nrt4B+bTYl/EStg/G YSxA== X-Gm-Message-State: AOAM531uGLx7E6XgEBeu8JLk0zOD66o4zqUIuJLBa2J06XaoTASrmTrc MrMBmt1IA/3zLU/eRpClSaqOZ6OFMfU= X-Google-Smtp-Source: ABdhPJxgTXEqxwlskRL1ezfQiysne7q+kvSz70GopOw2SVgkMKlji7fjn2Zm/WJ0w8ZDp4LW1ccoHg== X-Received: by 2002:a5d:5955:0:b0:20d:4b4:9879 with SMTP id e21-20020a5d5955000000b0020d04b49879mr26800300wri.550.1653490866158; Wed, 25 May 2022 08:01:06 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i7-20020a5d55c7000000b0020c5253d8d3sm2278543wrw.31.2022.05.25.08.01.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:05 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:29 +0000 Subject: [PATCH v8 07/30] fsmonitor-settings: stub in macOS-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++ config.mak.uname | 1 + contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 13 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-darwin.c diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index cf224768ad6..cf911d141f2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin + FSMONITOR_OS_SETTINGS = darwin endif endif diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 16705da2000..b8f9f7a0388 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) endif() endif() From patchwork Wed May 25 15:00:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861320 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F362EC433F5 for ; Wed, 25 May 2022 15:01:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245008AbiEYPBn (ORCPT ); Wed, 25 May 2022 11:01:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244960AbiEYPBY (ORCPT ); Wed, 25 May 2022 11:01:24 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C6230AF304 for ; Wed, 25 May 2022 08:01:09 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id z17so5788055wmf.1 for ; Wed, 25 May 2022 08:01:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0Mr17QihlnIGjvFpHWbrLWD1QkouTALYpOtyVyZM0ig=; b=L3dMoKHVsmU3UjzcQ0qkFXRN+0pFc4Lqmj2ZYHCg/yuwCMOEGjebxkb8QWOP68eVyB TlT3NIDCCj4ySncbKBCOx5PM6x9h0Qq2H9vFvEdKws197IYcd5BcGWGmHa1hW5e/i8UO xtfCs7wtjVwqglDGV1zaaV2sEhCqtLXrozxKUBmteAjbTlrFVWRS8Q6yjqlFpfBPlJ87 H4ear2Xktksxw2RjKSeEeY3jogi2JFDnyuWFyNhRrOMHCgi+O+XujAMajFvq3NRolK5v frPdnpPB1fKoAaGZUZUe86wuxt0DDSEqX1jmVJy+s7unDMVuC1vI1MvDb5BMHp0ReLkl jhnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0Mr17QihlnIGjvFpHWbrLWD1QkouTALYpOtyVyZM0ig=; b=eOwIJHUs1RM0nknWhRnfSsUdWmvQIxZg1HayQC4TTYSnoppE0kWA8YOfcEG5XDAZOW +ebU7R3PuJLm8Q4zKI2oxn9aecuNGI63mcuFzH2CVUOEN5Sl+DLbLYpBl7lubI//VPju 8PFVUThcoCeSV2ond8c5EiTVebc+yNzTOjaXqExLv/rKJ++tZxvqoCZaPHD8Pn3yAGzr 8p3gOKWf5YbJ35NmZg16JRH1feIhXY1NEgIEDfAIT5K07PPvz25UMWZAJuNPW99QD/T2 M3QaZveICBvZ/fYevIQUf0km12s3A/J3+XW4wy2s2P7cUcWhcVlSXdCZq1kYO+nrbWaK un+A== X-Gm-Message-State: AOAM532Ht01ve8+omdBVLmnZhu5MHMvVDluWVZUCn9kkMwTyYjG9Ai8I ERP56ZpCC0HOQ7vXK0VTVHhJ4GhG/Zg= X-Google-Smtp-Source: ABdhPJywjnCsdKwQGYgcFH19lRbeO4O7YWWjnVMA03HkgAz9e0vS8zTfUuTeeu/ws5uOFE8vSS2DnA== X-Received: by 2002:a05:600c:19ca:b0:394:8dc0:b5c2 with SMTP id u10-20020a05600c19ca00b003948dc0b5c2mr8604739wmq.167.1653490867677; Wed, 25 May 2022 08:01:07 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i9-20020a5d5849000000b0020fe4c5e94csm2475855wrf.19.2022.05.25.08.01.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:06 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:30 +0000 Subject: [PATCH v8 08/30] fsmonitor-settings: remote repos on macOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on macOS and mark them as incompatible with FSMonitor. With this, `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, like `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++ fsmonitor-settings.c | 12 +++++ fsmonitor-settings.h | 2 + 3 files changed, 80 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index 7fce32a3c5b..fdd762bf79d 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -2,8 +2,74 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" +#include +#include + +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type (NFS, SAMBA, etc.) dictates whether notification events + * are available at all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * For the builtin FSMonitor, we create the Unix domain socket for the + * IPC in the .git directory. If the working directory is remote, + * then the socket will be created on the remote file system. This + * can fail if the remote file system does not support UDS file types + * (e.g. smbfs to a Windows server) or if the remote kernel does not + * allow a non-local process to bind() the socket. (These problems + * could be fixed by moving the UDS out of the .git directory and to a + * well-known local directory on the client machine, but care should + * be taken to ensure that $HOME is actually local and not a managed + * file share.) + * + * So (for now at least), mark remote working directories as + * incompatible. + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + struct statfs fs; + + if (statfs(r->worktree, &fs) == -1) { + int saved_errno = errno; + trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s", + r->worktree, strerror(saved_errno)); + errno = saved_errno; + return FSMONITOR_REASON_ERROR; + } + + trace_printf_key(&trace_fsmonitor, + "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'", + r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename); + + if (!(fs.f_flags & MNT_LOCAL)) + return FSMONITOR_REASON_REMOTE; + + return FSMONITOR_REASON_OK; +} enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 600ae165ab1..d2fb0141f8e 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -208,6 +208,18 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r, xgetcwd()); goto done; + case FSMONITOR_REASON_ERROR: + strbuf_addf(&msg, + _("repository '%s' is incompatible with fsmonitor due to errors"), + r->worktree); + goto done; + + case FSMONITOR_REASON_REMOTE: + strbuf_addf(&msg, + _("remote repository '%s' is incompatible with fsmonitor"), + r->worktree); + goto done; + case FSMONITOR_REASON_VFS4GIT: strbuf_addf(&msg, _("virtual repository '%s' is incompatible with fsmonitor"), diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index a48802cde9c..afd1b3874ac 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -17,6 +17,8 @@ enum fsmonitor_reason { FSMONITOR_REASON_UNTESTED = 0, FSMONITOR_REASON_OK, /* no incompatibility or when disabled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ + FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; From patchwork Wed May 25 15:00:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0179FC433FE for ; Wed, 25 May 2022 15:01:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242575AbiEYPBk (ORCPT ); Wed, 25 May 2022 11:01:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39838 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244963AbiEYPBY (ORCPT ); Wed, 25 May 2022 11:01:24 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6779AF318 for ; Wed, 25 May 2022 08:01:11 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id z17so5788109wmf.1 for ; Wed, 25 May 2022 08:01:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=Pb/F8B2QFRhEbeU7X591uC50+IXpaHo1XWQwDOpvplPSya8zLVl7QP+YOhcH2fxnMJ YwJ312/NhjCMCAtIwwaXt/8tpgXav/3jRM50pPOEJKi0+pW1oU+NRHFvjOb7Et40V/nx y7Z8w7KcqqiIh8lExA6HA5Yr21c3Y6J3sAF2kmq0amVG/Pas3Xg9wC9z5c4zVVyLHfvd EV7u2uTR1uHJReVkW+Rnd/t2WKbPvQiVVHVsYchJ0I0m/gBtA5qWSRRdeRQBjoZj4XfX MvlwyTit3K5wPJfESMADFKsA13LCQ4D5XRBY+TU5RCbrAtTNs8IR2mnh4cf1TzMmmsDo z9lA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=fmBQ0NSJARfItN3m7tE+Pp5gLHcN9qdnv7cSlHsvdv3ITaAqZ/DNP2CdkszizjY3C5 /szfPwlbE9jPIDcndxRvZoBJQZYMdrbcABskuWjbi9YG8dSxWPJGpjOSxf02koHL6tzc rvAk8AuC35dJetGfTeG51SW1HRt5yKbptBHfeMabUnQIgsJg27j1N8jzqVSE472m57ub fjCE2tdt2ihNemad28WZFQ7GeruTpON9W+oKmpzbwt51VmHpMCEb55ia1G2ZZ2NS2IeY Qfa844yKzfY2TDsE+lwF74slcdRUD7/TotCXRcSwJCOPRg7/Tjcu3YHXd86xv0ss0CYA O5Ig== X-Gm-Message-State: AOAM53100/OIzR+zK1I+RBHqC0K6e3q6oqVhSiKifJPxt+pgaDx4n28S RJGD7mJd6jSHS2vxwQDXvpdLAahCe5w= X-Google-Smtp-Source: ABdhPJyXohCt07A9qdZN9OnJPThBcaEEBYcgTprjbgqaCZWNkSuehgMbZwXvrSiCJrN8Pn2yLpoSmg== X-Received: by 2002:a05:600c:4f0c:b0:397:6a3a:d3f9 with SMTP id l12-20020a05600c4f0c00b003976a3ad3f9mr3711336wmq.103.1653490869650; Wed, 25 May 2022 08:01:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u7-20020a5d6da7000000b0020fcc655e4asm2653397wrs.5.2022.05.25.08.01.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:08 -0700 (PDT) Message-Id: <5627038aaa3e21c9d8607accfd3b2a46fa60e7fb.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:31 +0000 Subject: [PATCH v8 09/30] fsmonitor-settings: remote repos on Windows are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on Windows and mark them as incompatible with FSMonitor. With this `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, such as `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index ee78bba38e3..907655720bb 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -2,6 +2,7 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" /* * VFS for Git is incompatible with FSMonitor. @@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r) return FSMONITOR_REASON_OK; } +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type dictates whether notification events are available at + * all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * So (for now at least), mark remote working directories as + * incompatible. + * + * Notes for testing: + * + * (a) Windows allows a network share to be mapped to a drive letter. + * (This is the normal method to access it.) + * + * $ NET USE Z: \\server\share + * $ git -C Z:/repo status + * + * (b) Windows allows a network share to be referenced WITHOUT mapping + * it to drive letter. + * + * $ NET USE \\server\share\dir + * $ git -C //server/share/repo status + * + * (c) Windows allows "SUBST" to create a fake drive mapping to an + * arbitrary path (which may be remote) + * + * $ SUBST Q: Z:\repo + * $ git -C Q:/ status + * + * (d) Windows allows a directory symlink to be created on a local + * file system that points to a remote repo. + * + * $ mklink /d ./link //server/share/repo + * $ git -C ./link status + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + wchar_t wpath[MAX_PATH]; + wchar_t wfullpath[MAX_PATH]; + size_t wlen; + UINT driveType; + + /* + * Do everything in wide chars because the drive letter might be + * a multi-byte sequence. See win32_has_dos_drive_prefix(). + */ + if (xutftowcs_path(wpath, r->worktree) < 0) + return FSMONITOR_REASON_ERROR; + + /* + * GetDriveTypeW() requires a final slash. We assume that the + * worktree pathname points to an actual directory. + */ + wlen = wcslen(wpath); + if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') { + wpath[wlen++] = L'\\'; + wpath[wlen] = 0; + } + + /* + * Normalize the path. If nothing else, this converts forward + * slashes to backslashes. This is essential to get GetDriveTypeW() + * correctly handle some UNC "\\server\share\..." paths. + */ + if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL)) + return FSMONITOR_REASON_ERROR; + + driveType = GetDriveTypeW(wfullpath); + trace_printf_key(&trace_fsmonitor, + "DriveType '%s' L'%ls' (%u)", + r->worktree, wfullpath, driveType); + + if (driveType == DRIVE_REMOTE) { + trace_printf_key(&trace_fsmonitor, + "check_remote('%s') true", + r->worktree); + return FSMONITOR_REASON_REMOTE; + } + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; @@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) if (reason != FSMONITOR_REASON_OK) return reason; + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } From patchwork Wed May 25 15:00:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2A37C433EF for ; Wed, 25 May 2022 15:01:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245025AbiEYPBq (ORCPT ); Wed, 25 May 2022 11:01:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39852 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244986AbiEYPBZ (ORCPT ); Wed, 25 May 2022 11:01:25 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F135AF333 for ; Wed, 25 May 2022 08:01:13 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id t13so11327304wrg.9 for ; Wed, 25 May 2022 08:01:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=gagmQdnkju4spnOwYIMQ8SdkJsvAj1+kX3+WK6gVByM=; b=AOUDZZwEGjYQ8JJI/xjSQX23V4Zh/1qWk7EiQQFS42/qVHsM0JHuzmyhDn406tg3N0 vIusrc3kYKiySa4t8kz/3xUMVaCkxiO3y3U3r9JJx2x5mJhHvzhm8lb1IhjINnO64ZR7 1fOn1eKkqPwBIygyIOxVk56s5cT7pK6SK8QAWSZ4J1g+6TtIYvCzJ1E6e3MlCf6vRK2A iOMisf8DZXPQXjzKwZZobWAmy4XWkkAIdlmEWPpEC8CfK/uIHzXfrGCNvertyMRoqOBs 6xLfGZLa8JO0aZPnfLt0ZJG9dFjtt6xgsOdbKi9fsDd9VkD/LGGrHlckwODwEgVfUDVI Mm/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=gagmQdnkju4spnOwYIMQ8SdkJsvAj1+kX3+WK6gVByM=; b=OSkUL2ddd89EsULfTLpkFK7SUJ9Khou0eD6XFx8aC6QZ0owRT/oxb0BiNztyn2gYAK JN/3lZBAPyq6ENswTz0UUsDZi0GpUrSS7HRlTQNTnA10lLL9x4H4xEFDdz+gCubTS6Hm rY/2uRl+v+mxOkY9hGIuQtvsHYFz1o9l4mt+Skw61W6P3+MJKNX1xLDfRL5ketdsfZOF i2ypjwFTNukl27iGcRv05i6D3IMpo8RIZB02Tu6th5c6x+Za6LOfDZ51ueu4YrtScl/7 uJmLK6gnZhvg0lTxx+0V5XWcdRoPsd1zJQqs6cHSXfu9QrkQKzjH3oGwLmVFb+YMEjlu d64w== X-Gm-Message-State: AOAM533QmtI48IBv3kEiOxfliK69+CPwu+UP092vl4JLqjGp3WTa1P8T IiIAAIsGVYcDmx/GNmEKU/lFMdm4g8I= X-Google-Smtp-Source: ABdhPJxfEqsJ+8k6lyInVTAd7X8YuCNsI5tqGw+7DhHX5l9IvSem+xX34PZ6CeWeDmMQf4jpnttgaQ== X-Received: by 2002:adf:ebc6:0:b0:20d:e8e:4df5 with SMTP id v6-20020adfebc6000000b0020d0e8e4df5mr26609360wrn.117.1653490871569; Wed, 25 May 2022 08:01:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a8-20020a5d5088000000b0020e68218048sm2256722wrt.93.2022.05.25.08.01.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:10 -0700 (PDT) Message-Id: <9a12cc78b5d16a05d4ed76cdb2a1f634ed32ef8a.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:32 +0000 Subject: [PATCH v8 10/30] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler On MacOS mark repos on NTFS or FAT32 volumes as incompatible. The builtin FSMonitor used Unix domain sockets on MacOS for IPC with clients. These sockets are kept in the .git directory. Unix sockets are not supported by NTFS and FAT32, so the daemon cannot start up. Test for this during our compatibility checking so that client commands do not keep trying to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++--- fsmonitor-settings.c | 6 ++++++ fsmonitor-settings.h | 1 + 3 files changed, 24 insertions(+), 3 deletions(-) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index fdd762bf79d..efc732c0f31 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -7,7 +7,7 @@ #include /* - * Remote working directories are problematic for FSMonitor. + * [1] Remote working directories are problematic for FSMonitor. * * The underlying file system on the server machine and/or the remote * mount type (NFS, SAMBA, etc.) dictates whether notification events @@ -40,8 +40,16 @@ * * So (for now at least), mark remote working directories as * incompatible. + * + * + * [2] FAT32 and NTFS working directories are problematic too. + * + * The builtin FSMonitor uses a Unix domain socket in the .git + * directory for IPC. These Windows drive formats do not support + * Unix domain sockets, so mark them as incompatible for the daemon. + * */ -static enum fsmonitor_reason check_remote(struct repository *r) +static enum fsmonitor_reason check_volume(struct repository *r) { struct statfs fs; @@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r) if (!(fs.f_flags & MNT_LOCAL)) return FSMONITOR_REASON_REMOTE; + if (!strcmp(fs.f_fstypename, "msdos")) /* aka FAT32 */ + return FSMONITOR_REASON_NOSOCKETS; + + if (!strcmp(fs.f_fstypename, "ntfs")) + return FSMONITOR_REASON_NOSOCKETS; + return FSMONITOR_REASON_OK; } @@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; - reason = check_remote(r); + reason = check_volume(r); if (reason != FSMONITOR_REASON_OK) return reason; diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index d2fb0141f8e..658cb79da01 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -225,6 +225,12 @@ char *fsm_settings__get_incompatible_msg(const struct repository *r, _("virtual repository '%s' is incompatible with fsmonitor"), r->worktree); goto done; + + case FSMONITOR_REASON_NOSOCKETS: + strbuf_addf(&msg, + _("repository '%s' is incompatible with fsmonitor due to lack of Unix sockets"), + r->worktree); + goto done; } BUG("Unhandled case in fsm_settings__get_incompatible_msg: '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index afd1b3874ac..d9c2605197f 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -20,6 +20,7 @@ enum fsmonitor_reason { FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ + FSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */ }; void fsm_settings__set_ipc(struct repository *r); From patchwork Wed May 25 15:00:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9EAA2C433EF for ; Wed, 25 May 2022 15:01:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245023AbiEYPBo (ORCPT ); Wed, 25 May 2022 11:01:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244993AbiEYPBZ (ORCPT ); Wed, 25 May 2022 11:01:25 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 333E9AFACE for ; Wed, 25 May 2022 08:01:15 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id m32-20020a05600c3b2000b0039756bb41f2so3518245wms.3 for ; Wed, 25 May 2022 08:01:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=V9HPbAw08ZIvkWIykZ/MJKGMTPCOfduOc+rXXAWC7bgH7G3PDtogiz+pbo886Dtsdr rFwAY1BcAHpoajvFo/8BDUkhpPdjKP8kFpTFBZRNePR6wJWiX2ll9rMtcg+G3FCa0/MT sBnG3jLtJkONFoX65bCvXvvFXwCjpfkmMyVtVJ85Z6FG3LRFFg3yq1ghRzZAwvEXJaqr v2yfOPZPS3OVXne4WDUwQpJNcXsv9XgcYVSyUoVGijziyzgKy7zm7AaZkIr5z2swPK37 yUT/OR+ECHMcVrkkdrBq+Gyx0pk8zLKapijP7dvaJ8ANaMtD7MIzsohs8lpxSnKb1egI iOTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=oCje+JmdqdolaesOUZq4p9NfgBbO2GBE2bANmZfb0b4XPtKO5GFYmorI8FfcDyurfR WfiXpEyZdf6+fQ7MXT5yM4UKe0C4Zbf+AFbeC0ykGAqzg6dzm5Zufpjkop+A0GT0Ta9u sQ9mXvJGY7eh08tc56b94T6sGZUuK1zoSQ0cTWS+m0PO9GouAT1hgVFwcf0eOZUAGx4X wZGwyG51X52L+PRkX7dQMJ1vJzJC8atMro48mqqZ9F5/amikmSfjneYr1iKT8sn87OmW DqMWhwnYMOKWHMt1eSMrZN4zDjLYMk5349mx0UrxpovPLVBlE0PE/vDVsn0NYsJy9mGm N1bQ== X-Gm-Message-State: AOAM533w6TIXnSnCxH8mJIFQEJW+TrFjx96NEPr6xjlXoeODlcok7/Ds yHHlLxwb0QeI3zR9zS6Qw9E4CsudYvE= X-Google-Smtp-Source: ABdhPJyPPGVDOQTt80d1jV+heUkJxx3GAAruppEf7Dj75pBrhrklRvkpevhaq0yKbZW8V0/hkfVfPg== X-Received: by 2002:a05:600c:3b29:b0:397:4f6d:5119 with SMTP id m41-20020a05600c3b2900b003974f6d5119mr8704244wms.129.1653490873163; Wed, 25 May 2022 08:01:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i8-20020adfdec8000000b0020c5253d8e5sm2287917wrn.49.2022.05.25.08.01.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:12 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:33 +0000 Subject: [PATCH v8 11/30] unpack-trees: initialize fsmonitor_has_run_once in o->result Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Initialize `o->result.fsmonitor_has_run_once` based upon value in `o->src_index->fsmonitor_has_run_once` to prevent a second fsmonitor query during the tree traversal and possibly getting a skewed view of the working directory. The checkout code has already talked to the fsmonitor and the traversal is updating the index as it traverses, so there is no need to query the fsmonitor. Signed-off-by: Jeff Hostetler --- unpack-trees.c | 1 + 1 file changed, 1 insertion(+) diff --git a/unpack-trees.c b/unpack-trees.c index 360844bda3a..888cff81f9c 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options o->result.fsmonitor_last_update = xstrdup_or_null(o->src_index->fsmonitor_last_update); + o->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once; /* * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries From patchwork Wed May 25 15:00:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA99AC433F5 for ; Wed, 25 May 2022 15:01:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245028AbiEYPBt (ORCPT ); Wed, 25 May 2022 11:01:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244998AbiEYPBZ (ORCPT ); Wed, 25 May 2022 11:01:25 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 361E9AFAD2 for ; Wed, 25 May 2022 08:01:15 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id v4-20020a1cac04000000b00397001398c0so1256931wme.5 for ; Wed, 25 May 2022 08:01:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5W2x8t68NoDAVIFeMr0BVtQdXxzEzVw8wanSKIA/+K4=; b=owW9lhpzojtIW6IA41gTlSNAHl9jkYIjraTAfVilvUwih9fDyMBRXbtETrBhc/90KR ELs7wbWr4RmQYGIULCAqNgqvRu+iRS9QxIm+PBLSPU2/6/bxD7DaUsykR2Nr1nCDvCfQ jfzia9dMpj09vq0iefPktZmOBpNU3jVtAuVt1gSHCPFj+OxPGUKmw9VtA9DhcFfi+QF6 BAS1CZEGd81IEjbDT0ks5c6ZpE9fkGP+BBc3SJf/tQmLM2nOdyFUsGZMce+KsFVNHJzJ gJjCg6Ty4/FM2PHBriEH3Aso78H8PT5TgIsLvTfolkJ1qzfL9poTiz1sk6Qn5bw/+kQG OxAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5W2x8t68NoDAVIFeMr0BVtQdXxzEzVw8wanSKIA/+K4=; b=tGY0xQFYIeHyIL6ADAwbpubU+LdA4mY/ecNhoLAHOYk5R8KEIVp5gFT+eSiZqW3Ixj ilW4DDJc8RrtEsEkE7Z3ifAD/JbN5YXKqSyxd2pnRHXm+3FvEtVhv9pIYjSfcRYGw7yj KFOMgH0sAHaghcTWM97CmeCfLXUxQHo3D1R0TGLjKUop+rJF0Q9LkQ1dWfyQXXoRsG+Z VomDnFJ0asJymJFltZfdnDt7RR0yJEg+Gvt+dJDZZ+14Ee/amJNdcK6xDBRyqDeKwlm8 sW5pmMGnFm3k33pErPLxwCnJLoHSes81sKUgaOohIdHxEnT/LYdNLA8kB/70Rt5nqzJb nk+g== X-Gm-Message-State: AOAM530kqLr42/Pl1IyLgTSVewMBmim5T7ofW5PY+kZchJ17hN/w7Rxj 8V/pgPk9WBZHdai4ZLQ7I+P9hBI1rUE= X-Google-Smtp-Source: ABdhPJzwbN8Iu+VHnOm/GSPXWpsGxVERYIqzBGLCXT0F64W6uxOVRlEmtcdGV0Ks/SIjpAauMg6GtA== X-Received: by 2002:a7b:c4d2:0:b0:397:6403:702c with SMTP id g18-20020a7bc4d2000000b003976403702cmr5917640wmk.58.1653490874398; Wed, 25 May 2022 08:01:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q21-20020a1ce915000000b0039768532c79sm2362183wmc.6.2022.05.25.08.01.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:13 -0700 (PDT) Message-Id: <4f2b15d3d1f10eaec8e485b84281eb4cca207a69.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:34 +0000 Subject: [PATCH v8 12/30] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Ignore FSEvents resulting from `xattr` changes. Git does not care about xattr's or changes to xattr's, so don't waste time collecting these events in the daemon nor transmitting them to clients. Various security tools add xattrs to files and/or directories, such as to mark them as having been downloaded. We should ignore these events since it doesn't affect the content of the file/directory or the normal meta-data that Git cares about. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 0741fe834c3..14105f45c18 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -100,7 +100,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) if (flag & kFSEventStreamEventFlagItemCloned) strbuf_addstr(&msg, "ItemCloned|"); - trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s", path, flag, msg.buf); strbuf_release(&msg); @@ -125,6 +125,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef) ef & kFSEventStreamEventFlagUserDropped); } +/* + * If an `xattr` change is the only reason we received this event, + * then silently ignore it. Git doesn't care about xattr's. We + * have to be careful here because the kernel can combine multiple + * events for a single path. And because events always have certain + * bits set, such as `ItemIsFile` or `ItemIsDir`. + * + * Return 1 if we should ignore it. + */ +static int ef_ignore_xattr(const FSEventStreamEventFlags ef) +{ + static const FSEventStreamEventFlags mask = + kFSEventStreamEventFlagItemChangeOwner | + kFSEventStreamEventFlagItemCreated | + kFSEventStreamEventFlagItemFinderInfoMod | + kFSEventStreamEventFlagItemInodeMetaMod | + kFSEventStreamEventFlagItemModified | + kFSEventStreamEventFlagItemRemoved | + kFSEventStreamEventFlagItemRenamed | + kFSEventStreamEventFlagItemXattrMod | + kFSEventStreamEventFlagItemCloned; + + return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); +} + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -190,6 +215,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_ignore_xattr(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "ignore-xattr: '%s', flags=0x%x", + path_k, event_flags[k]); + continue; + } + switch (fsmonitor_classify_path_absolute(state, path_k)) { case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: From patchwork Wed May 25 15:00:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861328 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3963EC433F5 for ; Wed, 25 May 2022 15:02:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244854AbiEYPCA (ORCPT ); Wed, 25 May 2022 11:02:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245007AbiEYPBZ (ORCPT ); Wed, 25 May 2022 11:01:25 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 895D3AF338 for ; Wed, 25 May 2022 08:01:17 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id t6so30521671wra.4 for ; Wed, 25 May 2022 08:01:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fniMHDevgS+sy0VW9Irk7dpFWcke4CVDxXVC7jIjUKc=; b=a5dL8AyZifYhFA94jPAzaFI+mj76+2U6xSr4dYQzMPci+71c99h88AfB3ijpoRGUo/ IZc7c952hm0N0JNIVS1Y4yPTwq3LtdFO0iTAmVtYg0RcQ6gS4bDrotkaJsEI7XBnUvFM T0qOErIUEhbEMcckeR4BLOsWg6LHtrh03V6/TVG/4MWn4bjKd5w06Eud8NyHY/TmO+9A 5cOidOZW4nJGgQ+v6AbWYBa4LWdHuQNDTc4Td9n1QCXUjQGmi4MzAEu6b55zM7uDIsSr yxARWyYgYZXrDOPc9fVxqmm64dwhD/3Zifonf6g8wJ/c1MaDvS/MC9ihisOQ5CUWr+fi lCfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fniMHDevgS+sy0VW9Irk7dpFWcke4CVDxXVC7jIjUKc=; b=r5HzXIO5wGN1s6Otz4srzjechxROBE6T4vy/udZke7endM+L7M3V6dPzOBVRmssO26 bzqRYfmtUA+67XwPoyf6NML3f5gvA4gZTsi1fKPRn4pUhyRiYexMwH95SP/6bzFR1e1Q U311Gh8n1YQiWcNTXQkLap6yO8jr9ssTooFL8Pw2dgcVqY+HbF1CmKwRtLSB3yYFuGns GwCZaGIz/dG4TmvcWCbLX53ekeqFhll0+4KxcU5cftVJW7njKZvqH2iX58abmW12pen5 ICpF1yHJT2uHb4E/k7sf+bwPC2UrhbSGsjN+jkYvzf7Ius6ILfWSgymfqpQ022un19Z0 SoVg== X-Gm-Message-State: AOAM531tRV7t7wcr3ks8Ltkrg09BZYiqY8kfKkbyXGFJ2k4mzf5yVhBh zj63t4T6F8Pq0q5sCeQ0FcPg/isWAHM= X-Google-Smtp-Source: ABdhPJwpKawi76+W0OvRXYWHN+/Q5upP3fNPc+SFRE49ahVEHo9r8anucjtgZDCg1zBzK93bP3Ma/w== X-Received: by 2002:a05:6000:1b8d:b0:20c:d372:f07c with SMTP id r13-20020a0560001b8d00b0020cd372f07cmr27837202wru.607.1653490875555; Wed, 25 May 2022 08:01:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m13-20020a05600c3b0d00b003976fbfbf00sm1980805wms.30.2022.05.25.08.01.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:15 -0700 (PDT) Message-Id: <427dec412a559bffab789126ff8fe34eaef346fd.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:35 +0000 Subject: [PATCH v8 13/30] fsmonitor--daemon: cd out of worktree root Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the fsmonitor--daemon to CD outside of the worktree before starting up. The common Git startup mechanism causes the CWD of the daemon process to be in the root of the worktree. On Windows, this causes the daemon process to hold a locked handle on the CWD and prevents other processes from moving or deleting the worktree while the daemon is running. CD to HOME before entering main event loops. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++-- compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------ fsmonitor--daemon.h | 1 + 3 files changed, 47 insertions(+), 8 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 66b78a0353f..db297649daf 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1181,11 +1181,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * before we need it. */ if (ipc_server_run_async(&state->ipc_server_data, - fsmonitor_ipc__get_path(), &ipc_opts, + state->path_ipc.buf, &ipc_opts, handle_client, state)) return error_errno( _("could not start IPC thread pool on '%s'"), - fsmonitor_ipc__get_path()); + state->path_ipc.buf); /* * Start the fsmonitor listener thread to collect filesystem @@ -1220,6 +1220,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) static int fsmonitor_run_daemon(void) { struct fsmonitor_daemon_state state; + const char *home; int err; memset(&state, 0, sizeof(state)); @@ -1289,6 +1290,15 @@ static int fsmonitor_run_daemon(void) strbuf_addch(&state.path_cookie_prefix, '/'); + /* + * We create a named-pipe or unix domain socket inside of the + * ".git" directory. (Well, on Windows, we base our named + * pipe in the NPFS on the absolute path of the git + * directory.) + */ + strbuf_init(&state.path_ipc, 0); + strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path())); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1298,6 +1308,23 @@ static int fsmonitor_run_daemon(void) goto done; } + /* + * CD out of the worktree root directory. + * + * The common Git startup mechanism causes our CWD to be the + * root of the worktree. On Windows, this causes our process + * to hold a locked handle on the CWD. This prevents the + * worktree from being moved or deleted while the daemon is + * running. + * + * We assume that our FS and IPC listener threads have either + * opened all of the handles that they need or will do + * everything using absolute paths. + */ + home = getenv("HOME"); + if (home && *home && chdir(home)) + die_errno(_("could not cd home '%s'"), home); + err = fsmonitor_run_daemon_1(&state); done: @@ -1310,6 +1337,7 @@ done: strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); strbuf_release(&state.path_cookie_prefix); + strbuf_release(&state.path_ipc); return err; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 0a86aea3f7e..7f48306df53 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -424,12 +424,22 @@ static int recv_rdcw_watch(struct one_watch *watch) } /* - * NEEDSWORK: If an external is deleted, the above - * returns an error. I'm not sure that there's anything that - * we can do here other than failing -- the /.git - * link file would be broken anyway. We might try to check - * for that and return a better error message, but I'm not - * sure it is worth it. + * GetOverlappedResult() fails if the watched directory is + * deleted while we were waiting for an overlapped IO to + * complete. The documentation did not list specific errors, + * but I observed ERROR_ACCESS_DENIED (0x05) errors during + * testing. + * + * Note that we only get notificaiton events for events + * *within* the directory, not *on* the directory itself. + * (These might be properies of the parent directory, for + * example). + * + * NEEDSWORK: We might try to check for the deleted directory + * case and return a better error message, but I'm not sure it + * is worth it. + * + * Shutdown if we get any error. */ error(_("GetOverlappedResult failed on '%s' [GLE %ld]"), diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index bd09fffc176..223c2131b58 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -54,6 +54,7 @@ struct fsmonitor_daemon_state { struct fsmonitor_daemon_backend_data *backend_data; struct ipc_server_data *ipc_server_data; + struct strbuf path_ipc; }; /* From patchwork Wed May 25 15:00:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DD025C433F5 for ; Wed, 25 May 2022 15:01:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S244302AbiEYPBz (ORCPT ); Wed, 25 May 2022 11:01:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39866 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244946AbiEYPBZ (ORCPT ); Wed, 25 May 2022 11:01:25 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9B962AFAE1 for ; Wed, 25 May 2022 08:01:18 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id x12so6254117wrg.2 for ; Wed, 25 May 2022 08:01:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9pd6ial0nEKbdJ3K7GTF9YvYqNRXvCcTnKwMvn0bh6Y=; b=PHO7zm8vGB1/UuLRwUdMKPBwMgVW64ivQ4svTT7nc0D/lb+A6+wK7yb/UrIjGv3JLl 3WbCiD8HnXFQZN66oRLP0blQ6BWcZN0WinllNDcwSdn61Ocgwhltwmzy8Mq5gfhUyUQ1 8IDSLuetNVF1iHm00LjbPiHo81cdi753SJFE4ASle9skBbCz1dBkQNn1BraerRc9T24h NVYin+OWldm1CwGGnN99mqvUA68G7kgq5AisxnGVirKtfTAzvYBFtvKE1tnxTZGT5Lt8 jHyybh0pWGmNQHx4+GovHSS3fARb+dArFiKgg6PNd1WNXoU46AgF0CrV3KI8/a46/bej PgiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9pd6ial0nEKbdJ3K7GTF9YvYqNRXvCcTnKwMvn0bh6Y=; b=EEpdWGpBAv4X7LfzcS88JiSsjjQ37CmL31OzSbOaa7SdIlnZ7sP3LbshrgZqxyIw4Q rpLPS9/WjPvoGtGdPNtIH67ODL05/NIf6wSbmtXVgko3Xxn/I2YeChlz5TsP6eA1Zj4g PjOzh5k3mDPzbM8sesn9epSQtbnsDSJhxbV7UgPusoKBs9OvT44kuPKdMcdU0EnXyZP/ 5qwxXCqu5ysthOuelKhYXPbnlSa92Gn1Jv4rPVZmM5S58oyBSdAOBxCxT2p3eqKxHyap GWRLr4VgleIc2uOcx+xGst0XsO7yaRCQXZ3nJkbBAedYrkBEyjtk14Glz0aHIKBubqRy LD/w== X-Gm-Message-State: AOAM531AWuzU8I4eKDoZPVerB17BWYYEixHhaTGgHzZyrq6d2bvrdZYU u9eTjUQv5VaTZpMXHdrE9TkoWiAEZ7k= X-Google-Smtp-Source: ABdhPJxd9DKUGoe9/RLYRo3kypEOCjt92PZF77xtouwF3e9Eb5x8LwaIxLFVNdZgxlsSEOKLFAmZjQ== X-Received: by 2002:adf:e10d:0:b0:20c:dc8f:e5a5 with SMTP id t13-20020adfe10d000000b0020cdc8fe5a5mr27630875wrz.265.1653490876817; Wed, 25 May 2022 08:01:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c10-20020a056000184a00b0020e615bab7bsm2654871wri.7.2022.05.25.08.01.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:16 -0700 (PDT) Message-Id: <51b266b06e1739f09d785787c93fc3e346f47332.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:36 +0000 Subject: [PATCH v8 14/30] fsmonitor--daemon: prepare for adding health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Refactor daemon thread startup to make it easier to start a third thread class to monitor the health of the daemon. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index db297649daf..90fa9d09efb 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1174,6 +1174,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int listener_started = 0; + int err = 0; /* * Start the IPC thread pool before the we've started the file @@ -1194,15 +1196,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) if (pthread_create(&state->listener_thread, NULL, fsm_listen__thread_proc, state) < 0) { ipc_server_stop_async(state->ipc_server_data); - ipc_server_await(state->ipc_server_data); - - return error(_("could not start fsmonitor listener thread")); + err = error(_("could not start fsmonitor listener thread")); + goto cleanup; } + listener_started = 1; /* * The daemon is now fully functional in background threads. + * Our primary thread should now just wait while the threads + * do all the work. + */ +cleanup: + /* * Wait for the IPC thread pool to shutdown (whether by client - * request or from filesystem activity). + * request, from filesystem activity, or an error). */ ipc_server_await(state->ipc_server_data); @@ -1211,10 +1218,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * event from the IPC thread pool, but it doesn't hurt to tell * it again. And wait for it to shutdown. */ - fsm_listen__stop_async(state); - pthread_join(state->listener_thread, NULL); + if (listener_started) { + fsm_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + } - return state->error_code; + if (err) + return err; + if (state->error_code) + return state->error_code; + return 0; } static int fsmonitor_run_daemon(void) From patchwork Wed May 25 15:00:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861327 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90A1AC433F5 for ; Wed, 25 May 2022 15:01:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245054AbiEYPB5 (ORCPT ); Wed, 25 May 2022 11:01:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244957AbiEYPBZ (ORCPT ); Wed, 25 May 2022 11:01:25 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03CBDAFAF3 for ; Wed, 25 May 2022 08:01:20 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id k30so30497972wrd.5 for ; Wed, 25 May 2022 08:01:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=s9t9Q4XW0lz3tr2ylEFcAfk51tlgxAPR8T+Lhple8cM=; b=JAPUpCLPBqt2auBv6VAUnPMr7AS7scweo6M6XC+rlSdWn4UrmYpsf7JAk2BkzIpxd/ RC1A+8JeNFJ8cfH1MXzEmgKDB6HHaxJtabpnui4R0NvLsY5HtSClSceP+0YrG6ymyvbz sL/wa7XPQmcz9rvyttf0GnKO7NC9Rj0J14Q/1rzfksjAzFL8V0WWrCqIZVqHI9EdspNW JmyyTj+ZelGeCFEzrL9bs9I9VtoEOsuAWD+GruJvnktMwM0dWHLsPdNe7zPEWNNl3a5s KAcAH/odIy76DxakQKXnJYax6B0kEWVhYWjdTTh+dx+sUsIhHxnECWzgv4gilLynUTuV oxvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=s9t9Q4XW0lz3tr2ylEFcAfk51tlgxAPR8T+Lhple8cM=; b=Pyz0ZlCwhRNtsmIqeQEWUk2/WUoCEHN/FuHgmzrXZzY8Pt3/2qUC56X05sNpJBNrLL MY5VrIdK16RcARE1YnYobnEK4YeCbfDRq79aYxSYpSetFhUaczrRHFykT9Y0pHt7zaR1 HLqSmSVz/BsoYcAAYbswb2o+ZS+9HNevrrZ4YsvSYuoihiUKVmZdWm5S3mhUPqv0n0xY A1uRRfxgUDvoJv6aNHHZumCbkhBTNHkR0ZxQS/X7mYlzu3WaNQ9yVKrcFzG3QhhA8Y2V 5H/Jfv6m/I766s+igAb3jK4AEXdhgySbd+eBRsX3YbpZzJEs4fnwGiDbt5c2h231WAdJ T8oQ== X-Gm-Message-State: AOAM5322n+a4mSDRC7nA+mq+ZMwNjSLw/e/SbJpQAW+4oRSnsjm8G6Or Rio0VnfFktc6p6l5ih6rLN90kXG99fA= X-Google-Smtp-Source: ABdhPJzLGmobocR7bW7hf+mUYMSO1a6xYSxU1Q/Sl7HI4zALVKjtKRh5Tb2gaPDYMpYKfg6lNRH7Tg== X-Received: by 2002:adf:9bd9:0:b0:210:53f:c945 with SMTP id e25-20020adf9bd9000000b00210053fc945mr513710wrc.240.1653490878145; Wed, 25 May 2022 08:01:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h8-20020a05600c414800b0039740903c39sm2476229wmm.7.2022.05.25.08.01.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:17 -0700 (PDT) Message-Id: <594e0ae243de868ff18f94aafc4358d5801b88c5.1653490852.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:37 +0000 Subject: [PATCH v8 15/30] fsmonitor--daemon: rename listener thread related variables Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Rename platform-specific listener thread related variables and data types as we prepare to add another backend thread type. [] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data` [] `state->backend_data` becomes `state->listen_data` [] `state->error_code` becomes `state->listen_error_code` Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 6 +++--- compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++-------------- compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++------------- compat/fsmonitor/fsm-listen.h | 2 +- fsmonitor--daemon.h | 6 +++--- 5 files changed, 36 insertions(+), 36 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 90fa9d09efb..b2f578b239a 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1225,8 +1225,8 @@ cleanup: if (err) return err; - if (state->error_code) - return state->error_code; + if (state->listen_error_code) + return state->listen_error_code; return 0; } @@ -1241,7 +1241,7 @@ static int fsmonitor_run_daemon(void) hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); - state.error_code = 0; + state.listen_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 14105f45c18..07113205a61 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -27,7 +27,7 @@ #include "fsm-listen.h" #include "fsmonitor--daemon.h" -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { CFStringRef cfsr_worktree_path; CFStringRef cfsr_gitdir_path; @@ -158,7 +158,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, const FSEventStreamEventId event_ids[]) { struct fsmonitor_daemon_state *state = ctx; - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; char **paths = (char **)event_paths; struct fsmonitor_batch *batch = NULL; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -350,11 +350,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) NULL, NULL }; - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; const void *dir_array[2]; CALLOC_ARRAY(data, 1); - state->backend_data = data; + state->listen_data = data; data->cfsr_worktree_path = CFStringCreateWithCString( NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); @@ -386,18 +386,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) failed: error(_("Unable to create FSEventStream.")); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); return -1; } void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; if (data->stream) { if (data->stream_started) @@ -407,14 +407,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state) FSEventStreamRelease(data->stream); } - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->shutdown_style = SHUTDOWN_EVENT; CFRunLoopStop(data->rl); @@ -422,9 +422,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->rl = CFRunLoopGetCurrent(); @@ -441,7 +441,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) switch (data->shutdown_style) { case FORCE_ERROR_STOP: - state->error_code = -1; + state->listen_error_code = -1; /* fall thru */ case FORCE_SHUTDOWN: ipc_server_stop_async(state->ipc_server_data); @@ -453,7 +453,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) return; force_error_stop_without_loop: - state->error_code = -1; + state->listen_error_code = -1; ipc_server_stop_async(state->ipc_server_data); return; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 7f48306df53..2943632c771 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -54,7 +54,7 @@ struct one_watch wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { struct one_watch *watch_worktree; struct one_watch *watch_gitdir; @@ -284,7 +284,7 @@ static enum get_relative_result get_relative_longname( void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); + SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]); } static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, @@ -359,7 +359,7 @@ static void destroy_watch(struct one_watch *watch) free(watch); } -static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, +static int start_rdcw_watch(struct fsm_listen_data *data, struct one_watch *watch) { DWORD dwNotifyFilter = @@ -538,7 +538,7 @@ static int process_1_worktree_event( */ static int process_worktree_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_worktree; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -669,7 +669,7 @@ force_shutdown: */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_gitdir; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -727,11 +727,11 @@ skip_this_path: void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; DWORD dwWait; int result; - state->error_code = 0; + state->listen_error_code = 0; if (start_rdcw_watch(data, data->watch_worktree) == -1) goto force_error_stop; @@ -796,7 +796,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) } force_error_stop: - state->error_code = -1; + state->listen_error_code = -1; force_shutdown: /* @@ -813,7 +813,7 @@ clean_shutdown: int fsm_listen__ctor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; CALLOC_ARRAY(data, 1); @@ -846,7 +846,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) data->nr_listener_handles++; } - state->backend_data = data; + state->listen_data = data; return 0; failed: @@ -859,16 +859,16 @@ failed: void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; CloseHandle(data->hEventShutdown); destroy_watch(data->watch_worktree); destroy_watch(data->watch_gitdir); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h index f0539349baf..41650bf8972 100644 --- a/compat/fsmonitor/fsm-listen.h +++ b/compat/fsmonitor/fsm-listen.h @@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state); * do so if the listener thread receives a normal shutdown signal from * the IPC layer.) * - * It should set `state->error_code` to -1 if the daemon should exit + * It should set `state->listen_error_code` to -1 if the daemon should exit * with an error. */ void fsm_listen__loop(struct fsmonitor_daemon_state *state); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 223c2131b58..2c6fa1a5d91 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); */ void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); -struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ +struct fsm_listen_data; /* opaque platform-specific data for listener thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; @@ -50,8 +50,8 @@ struct fsmonitor_daemon_state { int cookie_seq; struct hashmap cookies; - int error_code; - struct fsmonitor_daemon_backend_data *backend_data; + int listen_error_code; + struct fsm_listen_data *listen_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Wed May 25 15:00:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 808F4C433EF for ; Wed, 25 May 2022 15:01:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245031AbiEYPBt (ORCPT ); Wed, 25 May 2022 11:01:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245011AbiEYPB0 (ORCPT ); Wed, 25 May 2022 11:01:26 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9F679AFAFA for ; Wed, 25 May 2022 08:01:21 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id k30so30498097wrd.5 for ; Wed, 25 May 2022 08:01:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=OTLYUOFI+cGlsvNY4S9OqwrezTVUzg9mxH05B+h7KQc=; b=ShybcsEm+L7SQlajfi63CVrCWARI8FE6QysF9AeXcC+v8HkhXgPQfeVy7udg8Bd7xZ zW4h09zdZRoSRMdAgd4M6eaD/UUrXdJWYba6V8ZU2dMm9yxZLKDkBJG/ZhO3fTzK53QN Stlwf2fRkaAG+kt/pBEiqBbL0JKFj+lDi1vD39ZmF9XFY3uvFOQ4kUwCsKJb/YvBQjxj VPt2rJ5OwxMiYrT1OJJkXwDez+PanO5tZwrD53rbfw/lt9HgEuZEUXvSTDCd8t5wY6OL 6fhzrJDYsseZ71EpUphBAgq+3BzRfoqlgFb4M55mh200uQiycs38F57wiJQAK9NDCLvh 09KQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=OTLYUOFI+cGlsvNY4S9OqwrezTVUzg9mxH05B+h7KQc=; b=1CuZ24Dr68o0Uvv3sXkiWFCslLCxOM+uCr/IPlidxl6Y7ZUski50o1lh88y6Y6n0wM oN+zolfIxXTTIZW+a7qkGrrSANTDU8UVvb5xEt1o9OE9bUtI7kqGZBRFQVBvjsO0xxwg imrbmkgfHgrqnS4wwLTlNegYtYw+sYJua32kf36sdy502grYLXbL7a/HB12O5vCbDuYk SWIBNCZxvJ0ZK5YVWNQXIcVM9kUWyH/xgtUG+5g4wiO4uBF9M/VJsmde227K3U4ZmQsG fl+Hxh1UZ6TbEbJsWI1w/xe4+qRhSW27NKTkHyuAsB11KRXnLlaMe861a709jnAw+F8G MM6Q== X-Gm-Message-State: AOAM533kReXT9m2ZTXFd1sd1gK05PHrqcFaSDoBM41vZ7ybIBB25CcEW ZFejPryWPJXj6C8Y4DnBaPETfdJ3jcU= X-Google-Smtp-Source: ABdhPJwtP1KIzFCB1HOOUB4pPt7rL0ztJzsgW9cSu7iDlzrJlsjL4XB2vdjOGaLMp/66Eq8+khBfgg== X-Received: by 2002:a5d:5003:0:b0:20f:fe68:1dc4 with SMTP id e3-20020a5d5003000000b0020ffe681dc4mr3024911wrt.405.1653490879224; Wed, 25 May 2022 08:01:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i8-20020adfdec8000000b0020c5253d8e5sm2288144wrn.49.2022.05.25.08.01.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:18 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:38 +0000 Subject: [PATCH v8 16/30] fsmonitor--daemon: stub in health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create another thread to watch over the daemon process and automatically shut it down if necessary. This commit creates the basic framework for a "health" thread to monitor the daemon and/or the file system. Later commits will add platform-specific code to do the actual work. The "health" thread is intended to monitor conditions that would be difficult to track inside the IPC thread pool and/or the file system listener threads. For example, when there are file system events outside of the watched worktree root or if we want to have an idle-timeout auto-shutdown feature. This commit creates the health thread itself, defines the thread-proc and sets up the thread's event loop. It integrates this new thread into the existing IPC and Listener thread models. This commit defines the API to the platform-specific code where all of the monitoring will actually happen. The platform-specific code for MacOS is just stubs. Meaning that the health thread will immediately exit on MacOS, but that is OK and expected. Future work can define MacOS-specific monitoring. The platform-specific code for Windows sets up enough of the WaitForMultipleObjects() machinery to watch for system and/or custom events. Currently, the set of wait handles only includes our custom shutdown event (sent from our other theads). Later commits in this series will extend the set of wait handles to monitor other conditions. Signed-off-by: Jeff Hostetler --- Makefile | 6 ++- builtin/fsmonitor--daemon.c | 39 +++++++++++++++ compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++ compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++ compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++ contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor--daemon.h | 4 ++ 7 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-darwin.c create mode 100644 compat/fsmonitor/fsm-health-win32.c create mode 100644 compat/fsmonitor/fsm-health.h diff --git a/Makefile b/Makefile index 93604fe8ef7..5f1623baadd 100644 --- a/Makefile +++ b/Makefile @@ -472,8 +472,9 @@ all:: # # If your platform supports a built-in fsmonitor backend, set # FSMONITOR_DAEMON_BACKEND to the "" of the corresponding -# `compat/fsmonitor/fsm-listen-.c` that implements the -# `fsm_listen__*()` routines. +# `compat/fsmonitor/fsm-listen-.c` and +# `compat/fsmonitor/fsm-health-.c` files +# that implement the `fsm_listen__*()` and `fsm_health__*()` routines. # # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS @@ -1982,6 +1983,7 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index b2f578b239a..2c109cf8b37 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,6 +3,7 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsm-health.h" #include "compat/fsmonitor/fsm-listen.h" #include "fsmonitor--daemon.h" #include "simple-ipc.h" @@ -1136,6 +1137,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, pthread_mutex_unlock(&state->main_lock); } +static void *fsm_health__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-health"); + + fsm_health__loop(state); + + trace2_thread_exit(); + return NULL; +} + static void *fsm_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -1174,6 +1187,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int health_started = 0; int listener_started = 0; int err = 0; @@ -1201,6 +1215,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) } listener_started = 1; + /* + * Start the health thread to watch over our process. + */ + if (pthread_create(&state->health_thread, NULL, + fsm_health__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + err = error(_("could not start fsmonitor health thread")); + goto cleanup; + } + health_started = 1; + /* * The daemon is now fully functional in background threads. * Our primary thread should now just wait while the threads @@ -1223,10 +1248,17 @@ cleanup: pthread_join(state->listener_thread, NULL); } + if (health_started) { + fsm_health__stop_async(state); + pthread_join(state->health_thread, NULL); + } + if (err) return err; if (state->listen_error_code) return state->listen_error_code; + if (state->health_error_code) + return state->health_error_code; return 0; } @@ -1242,6 +1274,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); state.listen_error_code = 0; + state.health_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ @@ -1321,6 +1354,11 @@ static int fsmonitor_run_daemon(void) goto done; } + if (fsm_health__ctor(&state)) { + err = error(_("could not initialize health thread")); + goto done; + } + /* * CD out of the worktree root directory. * @@ -1344,6 +1382,7 @@ done: pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsm_listen__dtor(&state); + fsm_health__dtor(&state); ipc_server_free(state.ipc_server_data); diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c new file mode 100644 index 00000000000..b9f709e8548 --- /dev/null +++ b/compat/fsmonitor/fsm-health-darwin.c @@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c new file mode 100644 index 00000000000..94b1d020f25 --- /dev/null +++ b/compat/fsmonitor/fsm-health-win32.c @@ -0,0 +1,72 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +struct fsm_health_data +{ + HANDLE hEventShutdown; + + HANDLE hHandles[1]; /* the array does not own these handles */ +#define HEALTH_SHUTDOWN 0 + int nr_handles; /* number of active event handles */ +}; + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + CALLOC_ARRAY(data, 1); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown; + data->nr_handles++; + + state->health_data = data; + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + if (!state || !state->health_data) + return; + + data = state->health_data; + + CloseHandle(data->hEventShutdown); + + FREE_AND_NULL(state->health_data); +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data = state->health_data; + + for (;;) { + DWORD dwWait = WaitForMultipleObjects(data->nr_handles, + data->hHandles, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) + goto clean_shutdown; + + error(_("health thread wait failed [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->health_error_code = -1; + ipc_server_stop_async(state->ipc_server_data); +clean_shutdown: + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ + SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]); +} diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h new file mode 100644 index 00000000000..45547ba9380 --- /dev/null +++ b/compat/fsmonitor/fsm-health.h @@ -0,0 +1,47 @@ +#ifndef FSM_HEALTH_H +#define FSM_HEALTH_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor health thread. + * This will be called from the main thread PRIOR to staring the + * thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsm_health__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the health thread. + * This will be called from the main thread AFTER joining the thread. + */ +void fsm_health__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to monitor the + * health of the daemon process. This will run in the health thread. + * + * The health thread should call `ipc_server_stop_async()` if it needs + * to cause a shutdown. (It should NOT do so if it receives a shutdown + * shutdown signal.) + * + * It should set `state->health_error_code` to -1 if the daemon should exit + * with an error. + */ +void fsm_health__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the health thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsm_health__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSM_HEALTH_H */ diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index b8f9f7a0388..16ace43d1c7 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 2c6fa1a5d91..2102a5c9ff5 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); struct fsm_listen_data; /* opaque platform-specific data for listener thread */ +struct fsm_health_data; /* opaque platform-specific data for health thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; + pthread_t health_thread; pthread_mutex_t main_lock; struct strbuf path_worktree_watch; @@ -51,7 +53,9 @@ struct fsmonitor_daemon_state { struct hashmap cookies; int listen_error_code; + int health_error_code; struct fsm_listen_data *listen_data; + struct fsm_health_data *health_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Wed May 25 15:00:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80A67C433FE for ; Wed, 25 May 2022 15:01:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245011AbiEYPBv (ORCPT ); Wed, 25 May 2022 11:01:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39816 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245014AbiEYPB2 (ORCPT ); Wed, 25 May 2022 11:01:28 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF35DAFB07 for ; Wed, 25 May 2022 08:01:22 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id e28so29891036wra.10 for ; Wed, 25 May 2022 08:01:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=dmKeQ0WOQSZ+p3/TvtWLN+y0SszKau9M4RSxVVMYhvPRfinuaYJVCuEbiSn2hS4YLO snBMhbycWwBVgfqJS0ZmLgDbSvut2cVPJT5OHyXuqUKEc6gW+3CKx7iqlemH/MzehxPC ERYSy23JtchwQI2kXyP3KxbTC0zh52zAuhPDkyPl4GomP5QEUFPfi2X6SMfZySL5bWFe V1Xn0ukkHL2ovRgVmqy/JNtN5uPzbbfazKcWhLOpIjNCJLpJt2erQi+hStT7a63+1QTq tvI53BFRfINXwWfEc1LauCXQ73y/6YyK3XcCd9GXZLLrL6Io9X1gboivDkc3lHZxovd3 C/eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=Y2x/GQbPtfBuPFbS5a8KVqmFvJpdkL5NUG278qIC+AEB37EMOoUYFACIFmYAfmtmp/ 7mjZ+6pH14H0KhWv7gV8iNftMxYud7M1qfm8qc4w7FJmUiyANnmThgMTaYP4Hrmw2aRn O3Rp0NPeaUEcv4F2CTZt99oDAZxyLLbWNK7KxykJ1is4yE+pOUuHGrW00jiWl9E0XLOL Forr/3ddjhxTekiMJl62/0Nt+55OdRVlEBCnfVubI6SRUEqfWNhYHOLi2lQuglhv2PF7 Bd8FPXA55bEBuw9VLUPqOFZww6vcBvRlumMzfUIty1mqF4/J0uX3d5YTDAyFrvm7u384 baig== X-Gm-Message-State: AOAM533hQVrPX8j5JXuRizRGKFdoAs3mNmbaAdpScueEpdnBenVzrL1t jPL24D3z2jA/3t3u/Ea0Qy7N7ancQ/A= X-Google-Smtp-Source: ABdhPJxMyy2ir8RRudc/kCP2OOa/X0Fku0hSxUMm3AbZlhL66sDKUWsF2k2wPxYBLyqxHSzDUqq13g== X-Received: by 2002:a5d:5888:0:b0:20d:270f:6b61 with SMTP id n8-20020a5d5888000000b0020d270f6b61mr26705808wrf.211.1653490880992; Wed, 25 May 2022 08:01:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l16-20020a5d5270000000b0020c5253d8d2sm2409543wrc.30.2022.05.25.08.01.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:19 -0700 (PDT) Message-Id: <46a5ae2a6353e1da3c3ba84c9a2b5e03d6eeae44.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:39 +0000 Subject: [PATCH v8 17/30] fsm-health-win32: add polling framework to monitor daemon health Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend the Windows version of the "health" thread to periodically inspect the system and shutdown if warranted. This commit updates the thread's wait loop to use a timeout and defines a (currently empty) table of functions to poll the system. A later commit will add functions to the table to actually inspect the system. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 94b1d020f25..24fc612bf02 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -4,6 +4,24 @@ #include "fsm-health.h" #include "fsmonitor--daemon.h" +/* + * Every minute wake up and test our health. + */ +#define WAIT_FREQ_MS (60 * 1000) + +/* + * State machine states for each of the interval functions + * used for polling our health. + */ +enum interval_fn_ctx { + CTX_INIT = 0, + CTX_TERM, + CTX_TIMER +}; + +typedef int (interval_fn)(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx); + struct fsm_health_data { HANDLE hEventShutdown; @@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) FREE_AND_NULL(state->health_data); } +/* + * A table of the polling functions. + */ +static interval_fn *table[] = { + NULL, /* must be last */ +}; + +/* + * Call all of the polling functions in the table. + * Shortcut and return first error. + * + * Return 0 if all succeeded. + */ +static int call_all(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + int k; + + for (k = 0; table[k]; k++) { + int r = table[k](state, ctx); + if (r) + return r; + } + + return 0; +} + void fsm_health__loop(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data = state->health_data; + int r; + + r = call_all(state, CTX_INIT); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; for (;;) { DWORD dwWait = WaitForMultipleObjects(data->nr_handles, data->hHandles, - FALSE, INFINITE); + FALSE, WAIT_FREQ_MS); if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) goto clean_shutdown; + if (dwWait == WAIT_TIMEOUT) { + r = call_all(state, CTX_TIMER); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; + continue; + } + error(_("health thread wait failed [GLE %ld]"), GetLastError()); goto force_error_stop; @@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state) force_error_stop: state->health_error_code = -1; +force_shutdown: ipc_server_stop_async(state->ipc_server_data); clean_shutdown: + call_all(state, CTX_TERM); return; } From patchwork Wed May 25 15:00:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861330 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 779AFC433F5 for ; Wed, 25 May 2022 15:02:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234064AbiEYPCJ (ORCPT ); Wed, 25 May 2022 11:02:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244959AbiEYPB2 (ORCPT ); Wed, 25 May 2022 11:01:28 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56960AEE0E for ; Wed, 25 May 2022 08:01:24 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id p10so11874924wrg.12 for ; Wed, 25 May 2022 08:01:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pg+pIfJCewJfl3Uu7fv/1CMTNHr8lfgpM21Z9LQ/57U=; b=nXtd1+c5LGlcv3ilEAuHJPhomP96RzovGDM0kIVJgZlO84dpKUKkkE95YGkJvaTyOf WZvOkdRz8U+/O3rgLFyHlRVx4AcqaneM7XNYLw+uY1PpdApIjiQcFcYqI3Gt7HeVwJq2 Kirjw+M4QpBw3BCL23zDxQctQO3qsgZSzTZhBFMgasE9hPIKgSqnB0zXLKiIBdDbkhNx O0kjt37cFXWENiM0sgoY2C/OzRfbi0AYWiv849gZ2Zvi9a2KfCi+vIvRD4d5yJ9N8PUP 60qAZmyVrJOz11S0jDuER4vAbrZJ1/4nKd6fGqX5v9V3lwzaNiOUXTsx6PgqXCXmk6e9 gR0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pg+pIfJCewJfl3Uu7fv/1CMTNHr8lfgpM21Z9LQ/57U=; b=BQEjzM7QswoJLprHtzMJEpudjN3lu6bz1H6lOxPLd5JpVR9bg6zXVs9QIv2OgiDAR3 dJSRNl5MRqOyL6ulzPHD2JGCWnxNm7BpyKBctAGfOu+CAKIfkEd60N5IPtOEK4D0QypQ FtMo7lOufWutiCEs1c0PtqUXwhCQnDhUwK2bNRlrgAmStZSN2ZZIArnjL7eyvHcjwAwc wwT058oubDvXQAYF3mZA9wO7AR/bkU8AE/dYCF0/krOnPNPxx6Sp6VfsvyVEjafLagRr Xp4aVAF024cmOTBKB4MagypL14hAKB1d78Q6qG/iHmhqNlSz2Bn7ZVQpneKgOvyuRMH9 IKXw== X-Gm-Message-State: AOAM533A7FYz3BycK5FWbQjJzFaZ9nv9c8O2rp5hAfxzBrmBOEgnyWJE Mpg4t7OyWMi0RAFFvX5CW/JAS6TLdqs= X-Google-Smtp-Source: ABdhPJzbKrEoLlaVq7dDhK/YosBqoSM6ICNeme2vv/k3mlNBP8aQzYFFuj8SG4KieVcsQtPGIZ1Jng== X-Received: by 2002:a5d:584c:0:b0:20f:de2e:bd1a with SMTP id i12-20020a5d584c000000b0020fde2ebd1amr12996170wrf.670.1653490882423; Wed, 25 May 2022 08:01:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f21-20020a7bcc15000000b003942a244f57sm1883147wmh.48.2022.05.25.08.01.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:21 -0700 (PDT) Message-Id: <7cf1be5f8e2c644a5bce9eb4484de2f6c82af359.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:40 +0000 Subject: [PATCH v8 18/30] fsm-health-win32: force shutdown daemon if worktree root moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Force shutdown fsmonitor daemon if the worktree root directory is moved, renamed, or deleted. Use Windows low-level GetFileInformationByHandle() to get and compare the Windows system unique ID for the directory with a cached version when we started up. This lets us detect the case where someone renames the directory that we are watching and then creates a new directory with the original pathname. This is important because we are listening to a named pipe for requests and they are stored in the Named Pipe File System (NPFS) which a kernel-resident pseudo filesystem not associated with the actual NTFS directory. For example, if the daemon was watching "~/foo/", it would have a directory-watch handle on that directory and a named-pipe handle for "//./pipe/...foo". Moving the directory to "~/bar/" does not invalidate the directory handle. (So the daemon would actually be watching "~/bar" but listening on "//./pipe/...foo". If the user then does "git init ~/foo" and causes another daemon to start, the first daemon will still have ownership of the pipe and the second daemon instance will fail to start. "git status" clients in "~/foo" will ask "//./pipe/...foo" about changes and the first daemon instance will tell them about "~/bar". This commit causes the first daemon to shutdown if the system unique ID for "~/foo" changes (changes from what it was when the daemon started). Shutdown occurs after a periodic poll. After the first daemon exits and releases the lock on the named pipe, subsequent Git commands may cause another daemon to be started on "~/foo". Similarly, a subsequent Git command may cause another daemon to be started on "~/bar". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 24fc612bf02..2ea08c1d4e8 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -29,8 +29,150 @@ struct fsm_health_data HANDLE hHandles[1]; /* the array does not own these handles */ #define HEALTH_SHUTDOWN 0 int nr_handles; /* number of active event handles */ + + struct wt_moved + { + wchar_t wpath[MAX_PATH + 1]; + BY_HANDLE_FILE_INFORMATION bhfi; + } wt_moved; }; +/* + * Lookup the system unique ID for the path. This is as close as + * we get to an inode number, but this also contains volume info, + * so it is a little stronger. + */ +static int lookup_bhfi(wchar_t *wpath, + BY_HANDLE_FILE_INFORMATION *bhfi) +{ + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + + hDir = CreateFileW(wpath, desired_access, share_mode, NULL, + OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] health thread could not open '%ls'"), + GetLastError(), wpath); + return -1; + } + + if (!GetFileInformationByHandle(hDir, bhfi)) { + error(_("[GLE %ld] health thread getting BHFI for '%ls'"), + GetLastError(), wpath); + CloseHandle(hDir); + return -1; + } + + CloseHandle(hDir); + return 0; +} + +/* + * Compare the relevant fields from two system unique IDs. + * We use this to see if two different handles to the same + * path actually refer to the same *instance* of the file + * or directory. + */ +static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1, + const BY_HANDLE_FILE_INFORMATION *bhfi_2) +{ + return (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber && + bhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh && + bhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow); +} + +/* + * Shutdown if the original worktree root directory been deleted, + * moved, or renamed? + * + * Since the main thread did a "chdir(getenv($HOME))" and our CWD + * is not in the worktree root directory and because the listener + * thread added FILE_SHARE_DELETE to the watch handle, it is possible + * for the root directory to be moved or deleted while we are still + * watching it. We want to detect that here and force a shutdown. + * + * Granted, a delete MAY cause some operations to fail, such as + * GetOverlappedResult(), but it is not guaranteed. And because + * ReadDirectoryChangesW() only reports on changes *WITHIN* the + * directory, not changes *ON* the directory, our watch will not + * receive a delete event for it. + * + * A move/rename of the worktree root will also not generate an event. + * And since the listener thread already has an open handle, it may + * continue to receive events for events within the directory. + * However, the pathname of the named-pipe was constructed using the + * original location of the worktree root. (Remember named-pipes are + * stored in the NPFS and not in the actual file system.) Clients + * trying to talk to the worktree after the move/rename will not + * reach our daemon process, since we're still listening on the + * pipe with original path. + * + * Furthermore, if the user does something like: + * + * $ mv repo repo.old + * $ git init repo + * + * A new daemon cannot be started in the new instance of "repo" + * because the named-pipe is still being used by the daemon on + * the original instance. + * + * So, detect move/rename/delete and shutdown. This should also + * handle unsafe drive removal. + * + * We use the file system unique ID to distinguish the original + * directory instance from a new instance and force a shutdown + * if the unique ID changes. + * + * Since a worktree move/rename/delete/unmount doesn't happen + * that often (and we can't get an immediate event anyway), we + * use a timeout and periodically poll it. + */ +static int has_worktree_moved(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + struct fsm_health_data *data = state->health_data; + BY_HANDLE_FILE_INFORMATION bhfi; + int r; + + switch (ctx) { + case CTX_TERM: + return 0; + + case CTX_INIT: + if (xutftowcs_path(data->wt_moved.wpath, + state->path_worktree_watch.buf) < 0) { + error(_("could not convert to wide characters: '%s'"), + state->path_worktree_watch.buf); + return -1; + } + + /* + * On the first call we lookup the unique sequence ID for + * the worktree root directory. + */ + return lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi); + + case CTX_TIMER: + r = lookup_bhfi(data->wt_moved.wpath, &bhfi); + if (r) + return r; + if (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) { + error(_("BHFI changed '%ls'"), data->wt_moved.wpath); + return -1; + } + return 0; + + default: + die(_("unhandled case in 'has_worktree_moved': %d"), + (int)ctx); + } + + return 0; +} + + int fsm_health__ctor(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data; @@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) * A table of the polling functions. */ static interval_fn *table[] = { + has_worktree_moved, NULL, /* must be last */ }; From patchwork Wed May 25 15:00:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861333 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4939FC433EF for ; Wed, 25 May 2022 15:02:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245009AbiEYPCU (ORCPT ); Wed, 25 May 2022 11:02:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39800 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244956AbiEYPB3 (ORCPT ); Wed, 25 May 2022 11:01:29 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8C6992E4 for ; Wed, 25 May 2022 08:01:25 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id p10so11875017wrg.12 for ; Wed, 25 May 2022 08:01:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=m/7nmjzQrPWZOZRTUqvWzqRTYtxoxcBSCt/eY/PHK+w=; b=BosjjlA3Gv+q6+td9cgqxOEAe5yrdqAUvyGOYBeAbElPZ/VzY2yX3bDd4g9pLQzkQu +t+1VzG5h/pFGAnjHVkFq7Q3XiSB0cNrZArCN/GJfEKDW+Aab0awiJp0R5v7CQyzPlQz ddxZXjGRzyXOo35kAbDCy9ulsSmlmNyfck9pyE/VxviOFobZQlRfZ5a6zeVuLOLShMEc cPXHsET6dpcA1/nTtzb9y2fJ5PNBOu/jhlbZ9IqdrFUdu0jwXXmvFcNFldNwIvNN6e/e CPmWI+5r8T+Z3tm7aSwXkKM3Wn4MbsRx9r6VzHWLdthBUoVO15S09U02C6Oe/RMw65HB ytjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=m/7nmjzQrPWZOZRTUqvWzqRTYtxoxcBSCt/eY/PHK+w=; b=KYIafoi1CMxXc0XLHwoxOupv+WY4CHqXcI9m3wARIkhHHyASn/jlaLtEuiMkhUk+7u mYX4wSZ/N3yOCZXTyI6Pgn/h9Hz/s3TN9bi8A2nh850lkuad0VVzngvcLhS9enBctI60 5Yd48lRfYNd2OTaFGgo+iBPY4c6CNZdmLim9SvqNsu2jLRqQ14AtyhPXX7OG4+Yx67aZ cxsSvhsN3mTLBgihUbgrxOzo9/vVA2wmVV6dLuFnZfE2tfYQivpqiPg6HcWFAA+cdXTA PKIVtccVI/7Bq/wkoa31CiQtVBHKP93d27j97f02HR8sQ4C2xCfpuij+39YORHrWdMU7 Jcjw== X-Gm-Message-State: AOAM5305jhb2xQZu8x/xEWwdSsW1/Lu2PRZUe7p4h+c0dMYCkltYZWIF 21Ny9npcX5K6z0BPG4pKt46H/l/DDEM= X-Google-Smtp-Source: ABdhPJxVfUTXr7wy/82Zl7k4UXigV7RWhSOjZCXj3UNys9Bv9c62JNvGy+dahut5kdy74vo1J7wpSw== X-Received: by 2002:adf:d1cf:0:b0:20f:d7fd:4d68 with SMTP id b15-20020adfd1cf000000b0020fd7fd4d68mr13979585wrd.13.1653490883694; Wed, 25 May 2022 08:01:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bi27-20020a05600c3d9b00b003942a244ee7sm2265512wmb.44.2022.05.25.08.01.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:23 -0700 (PDT) Message-Id: <95cf1299d445696630d8c0850a69dc2c0ee040ce.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:41 +0000 Subject: [PATCH v8 19/30] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the listener thread to shutdown the daemon if the spelling of the worktree root directory changes. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 07113205a61..83d38e8ac6c 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -106,6 +106,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) strbuf_release(&msg); } +static int ef_is_root_changed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagRootChanged); +} + static int ef_is_root_delete(const FSEventStreamEventFlags ef) { return (ef & kFSEventStreamEventFlagItemIsDir && @@ -215,6 +220,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_is_root_changed(event_flags[k])) { + /* + * The spelling of the pathname of the root directory + * has changed. This includes the name of the root + * directory itself or of any parent directory in the + * path. + * + * (There may be other conditions that throw this, + * but I couldn't find any information on it.) + * + * Force a shutdown now and avoid things getting + * out of sync. The Unix domain socket is inside + * the .git directory and a spelling change will make + * it hard for clients to rendezvous with us. + */ + trace_printf_key(&trace_fsmonitor, + "event: root changed"); + goto force_shutdown; + } + if (ef_ignore_xattr(event_flags[k])) { trace_printf_key(&trace_fsmonitor, "ignore-xattr: '%s', flags=0x%x", From patchwork Wed May 25 15:00:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861332 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02797C433EF for ; Wed, 25 May 2022 15:02:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235839AbiEYPCS (ORCPT ); Wed, 25 May 2022 11:02:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39914 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244975AbiEYPB3 (ORCPT ); Wed, 25 May 2022 11:01:29 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 34560BDD for ; Wed, 25 May 2022 08:01:26 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id s28so30367568wrb.7 for ; Wed, 25 May 2022 08:01:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=TW+FRcuF8R+vXmxi3RMJG78zBpKWaCuJnXLd8tEwLd0oOpqRg7ewvNl8+ce9/RDnQZ LyFznEfxWtE11dKDJum9P2FhDLzb5kHUqa7XqE2mxzRMXPNDjkmMXYs7luHBSxmX0cZW tcwVusHwi4CuyZXZLxZezhWqmgmyFHuG+K0Vra/lj3gtcaS68EpFAV9Da2QteQaV4h8c 2P/HlRJDCSlQEwhcJ96ggUFWVVCn0jVFWen3tMLQ7hvvQTIw+4MU0WKqAIAew82MkKqj WqGPmv2078cR+++ZOenX5kToe0sQE0tiQPI/+yLRIdN8j31Kh2FHY0vyzP04UhJ+Gi3n jYFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=Gy9cTG2/bgy2gezfJMLyZR237B4iJUSGij4PLpB2jQoUj7+C2h7H5OHA06mSlHiTTj XSCNw46kxUAh3OSptby03K2E6pTDraNs+KQ3UiaVJrXFOqijbNGJePoL37f+gFv+c5dE 3E10QLTrcWNEbfMoBJrdiN8PXhGtjVT9Wj7wMKCnmoMWcCZ6Qn+jUIS8y7khCmfSyxOz TnqANvOSUn0Zt4m1mhTZoewmLAdUT9fuw+7ZcdbbHJwV0rPy8J/rNnzaUk2jtq4syIhY 9S2apSknAquNtT6dP8SP9N6MjzXGrfKsecUV6vIfAjwtLkDan8SdJXzTzIFQ/yWdiJ/X 2wQw== X-Gm-Message-State: AOAM5337HhLlQuX4indp7Mx+R8hTcU5qqW2KDvzQTqI8aRu5I+Z1Id/V xEKJY7bfQhOXNrcr6aTiACrY9QjB1G0= X-Google-Smtp-Source: ABdhPJwI0B6s3k2YMD40j44Zc0cPX7QJbDSCTf2IEx8zWNxdEgspwCfiDIoS+aoMWXvi3Mj0DuD/0w== X-Received: by 2002:a05:6000:1562:b0:20f:c999:fda7 with SMTP id 2-20020a056000156200b0020fc999fda7mr17855634wrz.65.1653490885246; Wed, 25 May 2022 08:01:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r3-20020adfdc83000000b0020fc3e24041sm2267256wrj.106.2022.05.25.08.01.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:24 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:42 +0000 Subject: [PATCH v8 20/30] fsmonitor: optimize processing of directory events Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to perform binary search over the cache-entries for a directory notification and then linearly scan forward to find the immediate children. Previously, when the FSMonitor reported a modified directory Git would perform a linear search on the entire cache-entry array for all entries matching that directory prefix and invalidate them. Since the cache-entry array is already sorted, we can use a binary search to find the first matching entry and then only linearly walk forward and invalidate entries until the prefix changes. Also, the original code would invalidate anything having the same directory prefix. Since a directory event should only be received for items that are immediately within the directory (and not within sub-directories of it), only invalidate those entries and not the whole subtree. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 54 insertions(+), 17 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 292a6742b4f..e1229c289cf 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r, static void fsmonitor_refresh_callback(struct index_state *istate, char *name) { int i, len = strlen(name); - if (name[len - 1] == '/') { + int pos = index_name_pos(istate, name, len); + + trace_printf_key(&trace_fsmonitor, + "fsmonitor_refresh_callback '%s' (pos %d)", + name, pos); + if (name[len - 1] == '/') { /* - * TODO We should binary search to find the first path with - * TODO this directory prefix. Then linearly update entries - * TODO while the prefix matches. Taking care to search without - * TODO the trailing slash -- because '/' sorts after a few - * TODO interesting special chars, like '.' and ' '. + * The daemon can decorate directory events, such as + * moves or renames, with a trailing slash if the OS + * FS Event contains sufficient information, such as + * MacOS. + * + * Use this to invalidate the entire cone under that + * directory. + * + * We do not expect an exact match because the index + * does not normally contain directory entries, so we + * start at the insertion point and scan. */ + if (pos < 0) + pos = -pos - 1; /* Mark all entries for the folder invalid */ - for (i = 0; i < istate->cache_nr; i++) { - if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID && - starts_with(istate->cache[i]->name, name)) - istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } - /* Need to remove the / from the path for the untracked cache */ + + /* + * We need to remove the traling "/" from the path + * for the untracked cache. + */ name[len - 1] = '\0'; + } else if (pos >= 0) { + /* + * We have an exact match for this path and can just + * invalidate it. + */ + istate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID; } else { - int pos = index_name_pos(istate, name, strlen(name)); - - if (pos >= 0) { - struct cache_entry *ce = istate->cache[pos]; - ce->ce_flags &= ~CE_FSMONITOR_VALID; + /* + * The path is not a tracked file -or- it is a + * directory event on a platform that cannot + * distinguish between file and directory events in + * the event handler, such as Windows. + * + * Scan as if it is a directory and invalidate the + * cone under it. (But remember to ignore items + * between "name" and "name/", such as "name-" and + * "name.". + */ + pos = -pos - 1; + + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + if ((unsigned char)istate->cache[i]->name[len] > '/') + break; + if (istate->cache[i]->name[len] == '/') + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } } @@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) * Mark the untracked cache dirty even if it wasn't found in the index * as it could be a new untracked file. */ - trace_printf_key(&trace_fsmonitor, "fsmonitor_refresh_callback '%s'", name); untracked_cache_invalidate_path(istate, name, 0); } From patchwork Wed May 25 15:00:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861331 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6F9FC433EF for ; Wed, 25 May 2022 15:02:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235492AbiEYPCN (ORCPT ); Wed, 25 May 2022 11:02:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39808 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244977AbiEYPB3 (ORCPT ); Wed, 25 May 2022 11:01:29 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97472E8A for ; Wed, 25 May 2022 08:01:27 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id s28so30367568wrb.7 for ; Wed, 25 May 2022 08:01:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=kUTb3iU8XWgR798P6YdKsfHBU+ci261+gHJ9yA28Wks=; b=MUJ7bWWFu1xQyVasmGFhRRuv4hG06B75KgwWkWUXEoXTkIdnXcRpkmcUUGQzNLY7Cd R/onSmEUF8LcIqWeG4mN2sYtGUbL5o3miJ6Eqzrk8lkHYKRmMUX2eabo3RiwaXVAAvqb iXg16DSJQwBGJxUncHmsGV4w8CRQj+L8cky302weQTqeyprSCTjWb0JULpEisXr+HnLe k12eOfjXN5yh2cflo+fgpaIeIlq990b+tLcg3U6BtCGxPF1cK0cJ9FVfcWf317G+6hpi /WA1eRmU7L+Nk4dHlNvBU7ny/WM6g3bzj+IgXjG3WetVpYl0pWQi5rrhRQJ11PXYMtrX EO1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=kUTb3iU8XWgR798P6YdKsfHBU+ci261+gHJ9yA28Wks=; b=KxLEAeQ1IMoDebQW+ICX+AkLr1bataRpqeOasU3YXBtER2UnTiofv8QFFVe2ENX6ml 1bPcVVTyDwGVbPyVeGUKq3MaxNhxDWKVXWA35G09UR+7Z0u0fZWQ2rsA3NstNcrfhxn3 UDaKKObbPS7ZebvwHnr0S7pGBLO5d0SAi8B6vt3Nmb9SGORSdVrLw2lsFL8htORcK+9h CSxMp9P6WB5fU66VXS2xH59KGvcK1BW05G9RQnFvsdEd8YGq4TcLZxuglwhjly0XPyZn MYwVqbSv4aSURqVe34qh5GebKlE3NZ9s92kKPmG9W5uXjTIH+CMKFrQq8Od7HaGScbwU yv8w== X-Gm-Message-State: AOAM530IgXVObpI+eltJXkR6TOLjtFPJwed32QasVwk+v4H2sct78/IL sVtcjBg591adlS7OW4xNrsJ4R7J4vdQ= X-Google-Smtp-Source: ABdhPJy1JnRsYKSnWBF67i/nsMGwfUyCCAiJg9oEjfaJ0nzIv/8FRpmXEu8B+1dxboS6Lnpv1c2iDA== X-Received: by 2002:adf:ef01:0:b0:20a:8068:ca5e with SMTP id e1-20020adfef01000000b0020a8068ca5emr27256320wro.661.1653490886665; Wed, 25 May 2022 08:01:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f9-20020a5d64c9000000b0020e5b4ebaecsm2656373wri.4.2022.05.25.08.01.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:26 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:43 +0000 Subject: [PATCH v8 21/30] t7527: FSMonitor tests for directory moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create unit tests to move a directory. Verify that `git status` gives the same result with and without FSMonitor enabled. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 12655958e71..3bc335b891d 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -274,6 +274,16 @@ test_expect_success 'setup' ' trace* EOF + mkdir -p T1/T2/T3/T4 && + echo 1 >T1/F1 && + echo 1 >T1/T2/F1 && + echo 1 >T1/T2/T3/F1 && + echo 1 >T1/T2/T3/T4/F1 && + echo 2 >T1/F2 && + echo 2 >T1/T2/F2 && + echo 2 >T1/T2/T3/F2 && + echo 2 >T1/T2/T3/T4/F2 && + git -c core.fsmonitor=false add . && test_tick && git -c core.fsmonitor=false commit -m initial && @@ -356,6 +366,19 @@ directory_to_file () { echo 1 >dir1 } +move_directory_contents_deeper() { + mkdir T1/_new_ && + mv T1/[A-Z]* T1/_new_ +} + +move_directory_up() { + mv T1/T2/T3 T1 +} + +move_directory() { + mv T1/T2/T3 T1/T2/NewT3 +} + # The next few test cases confirm that our fsmonitor daemon sees each type # of OS filesystem notification that we care about. At this layer we just # ensure we are getting the OS notifications and do not try to confirm what @@ -660,6 +683,10 @@ do matrix_try $uc_val $fsm_val file_to_directory matrix_try $uc_val $fsm_val directory_to_file + matrix_try $uc_val $fsm_val move_directory_contents_deeper + matrix_try $uc_val $fsm_val move_directory_up + matrix_try $uc_val $fsm_val move_directory + if test $fsm_val = true then test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" ' From patchwork Wed May 25 15:00:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861334 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 803FAC433FE for ; Wed, 25 May 2022 15:02:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243034AbiEYPCX (ORCPT ); Wed, 25 May 2022 11:02:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S244999AbiEYPBb (ORCPT ); Wed, 25 May 2022 11:01:31 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11A94AF318 for ; Wed, 25 May 2022 08:01:30 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id 206-20020a1c02d7000000b00397345f2c6fso837629wmc.4 for ; Wed, 25 May 2022 08:01:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=aOWSf7fUwb/p7Rs1hSVUTNg/6ck4Nk5UeuE8sDGfYrORsHHp7R/US4pJ4fSuCmrOut uVoUUnXgT7o9FilbNeLgwFW61oVSrGRk0ufglkA1BFM6dDykTmhnwjCxpZeql2ZUDD5y mnaYaA+/mYisJwsw3xHDIihe6r0ugrI+GbDGo94UkrWbomKZYYqWGO75HDivM4UdGjw8 /joJlGfxubUAOL/ENFL6kcTDdTpYLndmVks7pwKvQOiHcB+4C3uDkuPujD5fFxlyWQSf sWUeHhZWnDRvBDh9xGKdXtURSBPmTzgRu2R2U3FaCer6XQnrNAVrf3VA0ksmOYAQuliC vaVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=zqiN8erDEoR1fQyF1Pw+CedCv3rhEF8gNcFMMrdKIen0K+0QeRGeWykrEBgFp8qsZc 093R2I5dAKWKO8w25Vt3YX+oFI6U3PtxC3Q+PMOZQggyvct/S1OE1eGXKk3ceoQzrTkQ 80mnNsbVa8yL6KLpPR30HSaRuiBUxBlv9vR40DE0lT3Y/+bxzvsmENgP9rOIef+V+JQC /+wG9I6mwkoA0ab8v0ypRoTngvKZ5PeGvGMqWpVuIpgsqZAo1kCj/T0pxNUl+spgyIwK WgmtvEfEJKyJ8ZjXpbYUXhaOoX6jFzxi045US5eZvDYAFvGRpWyDHesxowYJUWVEVv+3 YVjQ== X-Gm-Message-State: AOAM532QI9dwDumILEZRJEujce4I24krgJol5/7IWKBqBFD/i+qQ+Fa4 UEmd1tbcYanNPDPmYw0LZaxfoaysYb0= X-Google-Smtp-Source: ABdhPJxOqJ3wtqCdVip6+iZJ8dJsdpnzbK4UGp6abkDmz3hRvVnCUOYuJThQHHImv6PEMlA0nUJ3Kg== X-Received: by 2002:a05:600c:1e17:b0:397:3a41:dbb0 with SMTP id ay23-20020a05600c1e1700b003973a41dbb0mr8370930wmb.7.1653490888047; Wed, 25 May 2022 08:01:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z26-20020a05600c221a00b003974a3af623sm2270066wml.17.2022.05.25.08.01.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:27 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:44 +0000 Subject: [PATCH v8 22/30] t/perf/p7527: add perf test for builtin FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100755 t/perf/p7527-builtin-fsmonitor.sh diff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh new file mode 100755 index 00000000000..9338b9ea008 --- /dev/null +++ b/t/perf/p7527-builtin-fsmonitor.sh @@ -0,0 +1,257 @@ +#!/bin/sh + +test_description="Perf test for the builtin FSMonitor" + +. ./perf-lib.sh + +if ! test_have_prereq FSMONITOR_DAEMON +then + skip_all="fsmonitor--daemon is not supported on this platform" + test_done +fi + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +# Lie to perf-lib and ask for a new empty repo and avoid +# the complaints about GIT_PERF_REPO not being big enough +# the perf hit when GIT_PERF_LARGE_REPO is copied into +# the trash directory. +# +# NEEDSWORK: It would be nice if perf-lib had an option to +# "borrow" an existing large repo (especially for gigantic +# monorepos) and use it in-place. For now, fake it here. +# +test_perf_fresh_repo + + +# Use a generated synthetic monorepo. If it doesn't exist, we will +# generate it. If it does exist, we will put it in a known state +# before we start our timings. +# +PARAM_D=5 +PARAM_W=10 +PARAM_F=9 + +PARAMS="$PARAM_D"."$PARAM_W"."$PARAM_F" + +BALLAST_BR=p0006-ballast +export BALLAST_BR + +TMP_BR=tmp_br +export TMP_BR + +REPO=../repos/gen-many-files-"$PARAMS".git +export REPO + +if ! test -d $REPO +then + (cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F) +fi + + +enable_uc () { + git -C $REPO config core.untrackedcache true + git -C $REPO update-index --untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +disable_uc () { + git -C $REPO config core.untrackedcache false + git -C $REPO update-index --no-untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +start_fsm () { + git -C $REPO fsmonitor--daemon start + git -C $REPO fsmonitor--daemon status + git -C $REPO config core.fsmonitor true + git -C $REPO update-index --fsmonitor + git -C $REPO status >/dev/null 2>&1 +} + +stop_fsm () { + git -C $REPO config --unset core.fsmonitor + git -C $REPO update-index --no-fsmonitor + test_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null + git -C $REPO status >/dev/null 2>&1 +} + + +# Ensure that FSMonitor is turned off on the borrowed repo. +# +test_expect_success "Setup borrowed repo (fsm+uc)" " + stop_fsm && + disable_uc +" + +# Also ensure that it starts in a known state. +# +# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time +# the ballast checkout, since only the first invocation does any work and the +# subsequent ones just print "already on branch" and quit, so the reported +# time is not useful. +# +# Create a temp branch and do all work relative to it so that we don't +# accidentially alter the real ballast branch. +# +test_expect_success "Setup borrowed repo (temp ballast branch)" " + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO reset --hard && + git -C $REPO clean -d -f && + test_might_fail git -C $REPO branch -D $TMP_BR && + git -C $REPO branch $TMP_BR $BALLAST_BR && + git -C $REPO checkout $TMP_BR +" + + +echo Data >data.txt + +# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With +# FSMonitor enabled, we can get a skewed view of status times, since +# the index MAY (or may not) be updated after the first invocation +# which will update the FSMonitor Token, so the subsequent invocations +# may get a smaller response from the daemon. +# +do_status () { + msg=$1 + + test_perf "$msg" " + git -C $REPO status >/dev/null 2>&1 + " +} + +do_matrix () { + uc=$1 + fsm=$2 + + t="[uc $uc][fsm $fsm]" + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_expect_success "$t Setup matrix branch" " + git -C $REPO clean -d -f && + git -C $REPO checkout $TMP_BR && + test_might_fail git -C $REPO branch -D $MATRIX_BR && + git -C $REPO branch $MATRIX_BR $TMP_BR && + git -C $REPO checkout $MATRIX_BR + " + + if test $uc = true + then + enable_uc + else + disable_uc + fi + + if test $fsm = true + then + start_fsm + else + stop_fsm + fi + + do_status "$t status after checkout" + + # Modify many files in the matrix branch. + # Stage them. + # Commit them. + # Rollback. + # + test_expect_success "$t modify tracked files" " + find $REPO -name file1 -exec cp data.txt {} \\; + " + + do_status "$t status after big change" + + # Don't bother timing the "add" because _REPEAT_COUNT + # issue described above. + # + test_expect_success "$t add all" " + git -C $REPO add -A + " + + do_status "$t status after add all" + + test_expect_success "$t add dot" " + git -C $REPO add . + " + + do_status "$t status after add dot" + + test_expect_success "$t commit staged" " + git -C $REPO commit -a -m data + " + + do_status "$t status after commit" + + test_expect_success "$t reset HEAD~1 hard" " + git -C $REPO reset --hard HEAD~1 >/dev/null 2>&1 + " + + do_status "$t status after reset hard" + + # Create some untracked files. + # + test_expect_success "$t create untracked files" " + cp -R $REPO/ballast/dir1 $REPO/ballast/xxx1 + " + + do_status "$t status after create untracked files" + + # Remove the new untracked files. + # + test_expect_success "$t clean -df" " + git -C $REPO clean -d -f + " + + do_status "$t status after clean" + + if test $fsm = true + then + stop_fsm + fi +} + +# Begin testing each case in the matrix that we care about. +# +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" + +fsm_values="false true" + +for uc_val in $uc_values +do + for fsm_val in $fsm_values + do + do_matrix $uc_val $fsm_val + done +done + +cleanup () { + uc=$1 + fsm=$2 + + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_might_fail git -C $REPO branch -D $MATRIX_BR +} + + +# We're borrowing this repo. We should leave it in a clean state. +# +test_expect_success "Cleanup temp and matrix branches" " + git -C $REPO clean -d -f && + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO branch -D $TMP_BR && + for uc_val in $uc_values + do + for fsm_val in $fsm_values + do + cleanup $uc_val $fsm_val + done + done +" + +test_done From patchwork Wed May 25 15:00:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861335 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E91AC433EF for ; Wed, 25 May 2022 15:02:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229468AbiEYPCZ (ORCPT ); Wed, 25 May 2022 11:02:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39868 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237514AbiEYPBb (ORCPT ); Wed, 25 May 2022 11:01:31 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 967BCAF31B for ; Wed, 25 May 2022 08:01:30 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id p10so11874924wrg.12 for ; Wed, 25 May 2022 08:01:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yAYOyrQ7lGddulrPeFKr0D/52H8sH3I2rvMs5wugd2A=; b=hAExHOMPMEp8S/ZX98vqf/GkoeMHfZdbcJByy0yNQuRry5mHaONs9cwIxrFgyVcDb9 hoQsdeDy911iB1KBC+seJxXXq/It0+t9AVEDuyDWMCHt2ZgVFmk5PkxCN+KylMmQc7Wh qbIKyX0QqEAqwLkMhzv1MZRNUyVT+dvRsOyZcvxVMUBbT5UH1XAs0ZP0V5O4LNrreerF /VOhudhpINufcmy/iNdVR/3AC5v8wOQ7Mi/YyV2XHuEb++xL1myBUEmEy1qd8b8sMhtN 0xMn4kegr6hX2b53Xlqdtsos2P7/R0hoUbi+DEPiIcS5nF4FQA4hEnvt/h7WXCtk4hut 0QNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yAYOyrQ7lGddulrPeFKr0D/52H8sH3I2rvMs5wugd2A=; b=lYXbyskLMagphYZ/W1nGLRiLbCqz4mMt+EyEKb3miXd3/bnU7IsT5Jmmfs2JlATuwD zvT7Z3+UlNs587T29ZV4ABono5Vy2xztZijlzJJBfk70kahVzzu8c99bx7tyZ0diLYsA DhS3MeaOab4106oJ+WJHjGPxfvsRvqnluf3aDi1dLW3dVQCpHAOqtISO9/ddV8eYXnxp c1lhKt3HqWjpqh7MChoWd2uKGBH5WIFGtjCqdap5bE8LIXZ5HaYNYSD0R1uwF+rNUBVi mPPfVoYiqZyKQsp7c9T/WjUV3pekFEg9bXvZoSauVmCWq2Oi2HG5/cbYPOr1FgYEz96k ZQRQ== X-Gm-Message-State: AOAM530wC+5LkLKzyFS3CpuN0+EwIvaIUQnMLzwo3/kgjK9XVNWV38KX 0e2vVxLdRizbW3LZ/sC0xlnWrrDMQvE= X-Google-Smtp-Source: ABdhPJzouX8eII1r0fsLXMAnAQypEYGUHu7nJXusXd//Faa1+YybUuvsf/4+TIACBlP/qGTr01gUow== X-Received: by 2002:adf:e710:0:b0:210:347:8dd4 with SMTP id c16-20020adfe710000000b0021003478dd4mr1804923wrm.295.1653490889471; Wed, 25 May 2022 08:01:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j20-20020a5d6e54000000b0020e5d8dbbb8sm2412755wrz.56.2022.05.25.08.01.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:28 -0700 (PDT) Message-Id: <92f5c0d2c8b1a69aec4325ab35193fe257d481f1.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:45 +0000 Subject: [PATCH v8 23/30] fsmonitor: never set CE_FSMONITOR_VALID on submodules Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Never set CE_FSMONITOR_VALID on the cache-entry of submodule directories. During a client command like 'git status', we may need to recurse into each submodule to compute a status summary for the submodule. Since the purpose of the ce_flag is to let Git avoid scanning a cache-entry, setting the flag causes the recursive call to be avoided and we report incorrect (no status) for the submodule. We created an OS watch on the root directory of our working directory and we receive events for everything in the cone under it. When submodules are present inside our working directory, we receive events for both our repo (the super) and any subs within it. Since our index doesn't have any information for items within the submodules, we can't use those events. We could try to truncate the paths of those events back to the submodule boundary and mark the GITLINK as dirty, but that feels expensive since we would have to prefix compare every FS event that we receive against a list of submodule roots. And it still wouldn't be sufficient to correctly report status on the submodule, since we don't have any space in the cache-entry to cache the submodule's status (the 'SCMU' bits in porcelain V2 speak). That is, the CE_FSMONITOR_VALID bit just says that we don't need to scan/inspect it because we already know the answer -- it doesn't say that the item is clean -- and we don't have space in the cache-entry to store those answers. So we should always do the recursive scan. Therefore, we should never set the flag on GITLINK cache-entries. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 2 + fsmonitor.h | 11 ++++ t/t7527-builtin-fsmonitor.sh | 111 +++++++++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+) diff --git a/fsmonitor.c b/fsmonitor.c index e1229c289cf..57d6a483bee 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate) if (fsmonitor_enabled) { /* Mark all entries valid */ for (i = 0; i < istate->cache_nr; i++) { + if (S_ISGITLINK(istate->cache[i]->ce_mode)) + continue; istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; } diff --git a/fsmonitor.h b/fsmonitor.h index 3f41f653691..edf7ce5203b 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate) * Set the given cache entries CE_FSMONITOR_VALID bit. This should be * called any time the cache entry has been updated to reflect the * current state of the file on disk. + * + * However, never mark submodules as valid. When commands like "git + * status" run they might need to recurse into the submodule (using a + * child process) to get a summary of the submodule state. We don't + * have (and don't want to create) the facility to translate every + * FS event that we receive and that happens to be deep inside of a + * submodule back to the submodule root, so we cannot correctly keep + * track of this bit on the gitlink directory. Therefore, we never + * set it on submodules. */ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce) { @@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache if (fsm_mode > FSMONITOR_MODE_DISABLED && !(ce->ce_flags & CE_FSMONITOR_VALID)) { + if (S_ISGITLINK(ce->ce_mode)) + return; istate->cache_changed = 1; ce->ce_flags |= CE_FSMONITOR_VALID; trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 3bc335b891d..cf4fb72c3f0 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -721,4 +721,115 @@ do ' done +# Test fsmonitor interaction with submodules. +# +# If we start the daemon in the super, it will see FS events for +# everything in the working directory cone and this includes any +# files/directories contained *within* the submodules. +# +# A `git status` at top level will get events for items within the +# submodule and ignore them, since they aren't named in the index +# of the super repo. This makes the fsmonitor response a little +# noisy, but it doesn't alter the correctness of the state of the +# super-proper. +# +# When we have submodules, `git status` normally does a recursive +# status on each of the submodules and adds a summary row for any +# dirty submodules. (See the "S..." bits in porcelain V2 output.) +# +# It is therefore important that the top level status not be tricked +# by the FSMonitor response to skip those recursive calls. That is, +# even if FSMonitor says that the mtime of the submodule directory +# hasn't changed and it could be implicitly marked valid, we must +# not take that shortcut. We need to force the recusion into the +# submodule so that we get a summary of the status *within* the +# submodule. + +create_super () { + super="$1" && + + git init "$super" && + echo x >"$super/file_1" && + echo y >"$super/file_2" && + echo z >"$super/file_3" && + mkdir "$super/dir_1" && + echo a >"$super/dir_1/file_11" && + echo b >"$super/dir_1/file_12" && + mkdir "$super/dir_1/dir_2" && + echo a >"$super/dir_1/dir_2/file_21" && + echo b >"$super/dir_1/dir_2/file_22" && + git -C "$super" add . && + git -C "$super" commit -m "initial $super commit" +} + +create_sub () { + sub="$1" && + + git init "$sub" && + echo x >"$sub/file_x" && + echo y >"$sub/file_y" && + echo z >"$sub/file_z" && + mkdir "$sub/dir_x" && + echo a >"$sub/dir_x/file_a" && + echo b >"$sub/dir_x/file_b" && + mkdir "$sub/dir_x/dir_y" && + echo a >"$sub/dir_x/dir_y/file_a" && + echo b >"$sub/dir_x/dir_y/file_b" && + git -C "$sub" add . && + git -C "$sub" commit -m "initial $sub commit" +} + +my_match_and_clean () { + git -C super --no-optional-locks status --porcelain=v2 >actual.with && + git -C super --no-optional-locks -c core.fsmonitor=false \ + status --porcelain=v2 >actual.without && + test_cmp actual.with actual.without && + + git -C super/dir_1/dir_2/sub reset --hard && + git -C super/dir_1/dir_2/sub clean -d -f +} + +test_expect_success 'submodule always visited' ' + test_when_finished "git -C super fsmonitor--daemon stop; \ + rm -rf super; \ + rm -rf sub" && + + create_super super && + create_sub sub && + + git -C super submodule add ../sub ./dir_1/dir_2/sub && + git -C super commit -m "add sub" && + + start_daemon -C super && + git -C super config core.fsmonitor true && + git -C super update-index --fsmonitor && + git -C super status && + + # Now run pairs of commands w/ and w/o FSMonitor while we make + # some dirt in the submodule and confirm matching output. + + # Completely clean status. + my_match_and_clean && + + # .M S..U + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u && + my_match_and_clean && + + # .M S.M. + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M S.M. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M SC.. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + git -C super/dir_1/dir_2/sub commit -m "SC.." && + my_match_and_clean +' + test_done From patchwork Wed May 25 15:00:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9499AC433F5 for ; Wed, 25 May 2022 15:02:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240140AbiEYPCa (ORCPT ); Wed, 25 May 2022 11:02:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243068AbiEYPBf (ORCPT ); Wed, 25 May 2022 11:01:35 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B28D0AFAC5 for ; Wed, 25 May 2022 08:01:32 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id k30so30497972wrd.5 for ; Wed, 25 May 2022 08:01:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4Q8NCwKBR4h76mmPvMVi3XVoVst203o7US4feb91EME=; b=WjdZXm8D5OR3dPMDEbFWZO3W4dDb5o1HwztXVjPcp6y147WqhXVzc00tg0Z+y4SCZH VoYQ6xj5ukenOc4tPFZSLH+kNZr8FkiePFZFrPV6FTGoHiD7Fr2T9vEEj2V7ZdAiji6P 5k/f+uG+Iw0Kye4J0MNFW4fArFAHcd4QBATzGRVXvNfUqSlxQG7WTt8xa6z3PJknvGf7 0E0mkj/+mcNP9TAfnI5BkEwF2MOk9HuMEtqaguAnLfgm36LSmgpmq1uUOHiFuJR7H0Rg IkgD7TfctXG9Rjbm0rsHtwuYfsL24maFxiBQEFnL6/lAQbr6N+Hu4uL9uQbJlyq3DFf5 3+YQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4Q8NCwKBR4h76mmPvMVi3XVoVst203o7US4feb91EME=; b=DYhNDTuUyjwjHq8dqGEa8n8I1nTpA8zrluIa13fySANSB26NK3kskl9qJp+B6tgI/e cRpJhHGDeoCsxlF14wIRQvr0TzwhtpDzDoJpUSgeAUSKr9vOAtxMavVRfTaOCmLl000z T7czsKEbHvp7NhwtR2Mj5diCzdFNztw2s6XUpNZQeo8Hg5SPg2Qnlm3h8xCZLzyYqzXH N+/dumDb+M11bp4edYOZzoMU0XWSFbdOv7z0Yi+Xa6e4oqwE34bEK3bkbJ21x4J5aKEb jLTRlYinaeZoDrFd1qpg+wlwB7P8nmjFdThLwZxv5d92ZLHWafBbL68gavBLd1wVJEo9 wNeQ== X-Gm-Message-State: AOAM530YcyfXhfp2qfGejLurTChU3NqzujrJeHUuZwe2T4KJDK4QFIol d/wrqtdmIkYKadA+Gmih/ATkRYOTsOk= X-Google-Smtp-Source: ABdhPJyi77Gm8QNyKb3DnLUHyFMVEZ4blanXBWfTa4IXPt3JMxyc3wrL5glXFjlSpEH5cnUVv1khtw== X-Received: by 2002:a05:6000:91:b0:20f:fb45:ab0a with SMTP id m17-20020a056000009100b0020ffb45ab0amr4303060wrx.34.1653490891467; Wed, 25 May 2022 08:01:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 206-20020a1c02d7000000b0039771fbffcasm1398678wmc.21.2022.05.25.08.01.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:30 -0700 (PDT) Message-Id: <40b80adbb31e866afb2b74b681695d3dadffc8df.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:46 +0000 Subject: [PATCH v8 24/30] t7527: test FSMonitor on case insensitive+preserving file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Test that FS events from the OS are received using the preserved, on-disk spelling of files/directories rather than spelling used to make the change. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 36 ++++++++++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index cf4fb72c3f0..fbb7d6aef6e 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -832,4 +832,40 @@ test_expect_success 'submodule always visited' ' my_match_and_clean ' +# On a case-insensitive file system, confirm that the daemon +# notices when the .git directory is moved/renamed/deleted +# regardless of how it is spelled in the the FS event. +# That is, does the FS event receive the spelling of the +# operation or does it receive the spelling preserved with +# the file/directory. +# +test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' +# test_when_finished "stop_daemon_delete_repo test_insensitive" && + + git init test_insensitive && + + start_daemon -C test_insensitive --tf "$PWD/insensitive.trace" && + + mkdir -p test_insensitive/abc/def && + echo xyz >test_insensitive/ABC/DEF/xyz && + + test_path_is_dir test_insensitive/.git && + test_path_is_dir test_insensitive/.GIT && + + # Rename .git using an alternate spelling to verify that that + # daemon detects it and automatically shuts down. + mv test_insensitive/.GIT test_insensitive/.FOO && + sleep 1 && + mv test_insensitive/.FOO test_insensitive/.git && + test_must_fail git -C test_insensitive fsmonitor--daemon status && + + # Verify that events were reported using on-disk spellings of the + # directories and files that we touched. We may or may not get a + # trailing slash on modified directories. + # + egrep "^event: abc/?$" ./insensitive.trace && + egrep "^event: abc/def/?$" ./insensitive.trace && + egrep "^event: abc/def/xyz$" ./insensitive.trace +' + test_done From patchwork Wed May 25 15:00:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72882C433F5 for ; Wed, 25 May 2022 15:02:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237514AbiEYPC2 (ORCPT ); Wed, 25 May 2022 11:02:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245019AbiEYPBg (ORCPT ); Wed, 25 May 2022 11:01:36 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 977DBAFACE for ; Wed, 25 May 2022 08:01:34 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id t6so30521671wra.4 for ; Wed, 25 May 2022 08:01:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=bIorACD9MvGhf+EOOe9S7KEedrit1duOzRZ5v8bUeZQ=; b=nv3AM7+miUpGMmChd2BIiaEpxyIu6rURBs+ylLKYy1vamv2wzhar4B402UD/n3atFR N3XKuAaRg7sFwgHSPx0i+rR8PuA3E9xALAbUMe+LVoEHkxpdYxpJP9ulaTh91z3GOXQG ApyJ8nlkuQg4tm2ofJrPcAg14DWuckISQb23Ay7ukXi9w9qZ3oSlxawgltCFBTZ4QLSA WJvbiPAfnOE2HMvsiR/uCRluazWj07Toohf4PTh6tajT4gVjzjwOUpLyFY223NG1RCeR iVZPAuvAAQxT7m+RSDJCEOVw5M/8GMMlBQJ9FphsbkF2eQhHPHwTxUAbD/Ul1gsj+wpd CqlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=bIorACD9MvGhf+EOOe9S7KEedrit1duOzRZ5v8bUeZQ=; b=fPeHMKeiTdwXruQTyM7waJBDRVYIxBoWYtdaHz0J6mfa6XyghpgVCfqVCzsLvgD3/A E9aPvCettZEcXzrVCtk/S825NGq3e0OCoTnajDp3VBsJheEASvhMOn2JHIE56SNgHall RaWmR8s3p5gL/Y7UmZ6bHxNXtc7OjXCamF6ntAWjRhrOPdm4dOj7eCvGk06MCSKE1xeg IW9m7In+gq0bZzVUGcPgYyd46/ow+ZWF4Rm7c+jOcVcZLVGRStAZMyGXChr+gH9zBSMc xiRztOG1jzaOhSpEeLHdyXFyaOMwBMy4QKBoXw8668juvmZqHFwqb9cjZJI7WV3LRLLP XEjw== X-Gm-Message-State: AOAM532lHEcJbhCFQD94vWg+EO+gODZRyklqkfsrIcFDDxDCOiyBr/LR Z3dUtm3kaFGtJndzLldjJjzrHj/154U= X-Google-Smtp-Source: ABdhPJwXEOUTpkI6P3nJWkAHML09TT9JdguFzC6A8RMKRIA47jqY7jTB0GsEpOMv3BWqZMLMPVYEqA== X-Received: by 2002:a5d:5262:0:b0:20f:fcaf:7d83 with SMTP id l2-20020a5d5262000000b0020ffcaf7d83mr3591113wrc.532.1653490893336; Wed, 25 May 2022 08:01:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h1-20020a1ccc01000000b00397342e3830sm5672594wmb.0.2022.05.25.08.01.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:32 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:47 +0000 Subject: [PATCH v8 25/30] fsmonitor: on macOS also emit NFC spelling for NFD pathname Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Emit NFC or NFC and NFD spellings of pathnames on macOS. MacOS is Unicode composition insensitive, so NFC and NFD spellings are treated as aliases and collide. While the spelling of pathnames in filesystem events depends upon the underlying filesystem, such as APFS, HFS+ or FAT32, the OS enforces such collisions regardless of filesystem. Teach the daemon to always report the NFC spelling and to report the NFD spelling when stored in that format on the disk. This is slightly more general than "core.precomposeUnicode". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 83d38e8ac6c..823cf63999e 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -155,6 +155,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef) return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); } +/* + * On MacOS we have to adjust for Unicode composition insensitivity + * (where NFC and NFD spellings are not respected). The different + * spellings are essentially aliases regardless of how the path is + * actually stored on the disk. + * + * This is related to "core.precomposeUnicode" (which wants to try + * to hide NFD completely and treat everything as NFC). Here, we + * don't know what the value the client has (or will have) for this + * config setting when they make a query, so assume the worst and + * emit both when the OS gives us an NFD path. + */ +static void my_add_path(struct fsmonitor_batch *batch, const char *path) +{ + char *composed; + + /* add the NFC or NFD path as received from the OS */ + fsmonitor_batch__add_path(batch, path); + + /* if NFD, also add the corresponding NFC spelling */ + composed = (char *)precompose_string_if_needed(path); + if (!composed || composed == path) + return; + + fsmonitor_batch__add_path(batch, composed); + free(composed); +} + + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -305,7 +334,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, rel); + my_add_path(batch, rel); } if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { @@ -318,7 +347,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, tmp.buf); + my_add_path(batch, tmp.buf); } break; From patchwork Wed May 25 15:00:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A08D3C433F5 for ; Wed, 25 May 2022 15:02:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245106AbiEYPCo (ORCPT ); Wed, 25 May 2022 11:02:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40708 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245041AbiEYPBz (ORCPT ); Wed, 25 May 2022 11:01:55 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6B53EAFB0E for ; Wed, 25 May 2022 08:01:41 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id t13so11327304wrg.9 for ; Wed, 25 May 2022 08:01:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=U8ZqUfuZDEUFYGsmHGxhn6kU6MMzXdTmjRg0f11rgdE=; b=Au3wWMdASe8FUYvbkRRxdAZphsu2fnCo1wICNkZ8H8Pt0crUWHUwBzQaPNYggE13ab xpX+C4CLxCMmG2OlcWWle9lQ9yrf+Y0UM/iA6MMW2ubxgsGY1FgBYiMnnhUrO5lhcyN9 z6CPvEjkl5cEOQ1Er67buSyifr1Pg7igL8vh+PcvvgDa5tn74YAwX8FzDXE2Un+JspMm KmSchKI9Fl+on0RntQKa6rYO+hYgiGD7yBTQseDfjiPI8sS8bdcTu59/lku/5tsn54np mFPIRbauKFdz1QweIc13asI02YD8EMJ7BIFsBz8xi5lUHIQGNR9PJVvS2WLGb1csyPdu XnFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=U8ZqUfuZDEUFYGsmHGxhn6kU6MMzXdTmjRg0f11rgdE=; b=FwA0+kYOEYUGbqh5/OUG6q63Zp+LRY/R8d+zDDWvtAsvY6luvk0wAeAQlT4WIBbwRM 0mNs7UBCfiMByC7lWET5OmeUcGat9JX//RjHbPss+CnImLGdR0pye6EGHjAy5u5lL2y4 3sABkDG+8Ge2ppKKtLORCT9H+i9HLFz1y2+cbDhRcEO3rPPeXjbBO8Jq07loDkyEakgg nOxHAbtNCtuwtfeSHjimEK6aurelAOf5ZwIRSTLQ1Vnt4YNqN23nJWfCMjQe1P1vu6jC jHReKzPLsfnyL7inho8CePaKLhpPwnNLkG42KPTiQk1zWi2S4rAE0Vy03x6uYnmPztJy /Oog== X-Gm-Message-State: AOAM530+k7oSS8ghKMnadwAypc9vgRP+B2nFI2BJKCJLB4DCp5DeMtRe 4ncpgvYNxrDl+bvxmdr5jg6Kk5J8aec= X-Google-Smtp-Source: ABdhPJwf6+3bRr9CEPwfaTJ+VWSyzK9FQW2xjt5q9j35EpfhEWVbsS/zf8DtcpcjDwSZfu3I1Kg9Ig== X-Received: by 2002:a05:6000:1867:b0:20f:ca21:7813 with SMTP id d7-20020a056000186700b0020fca217813mr17880242wri.100.1653490895681; Wed, 25 May 2022 08:01:35 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c4-20020adfc6c4000000b0020e5d2a9d0bsm2799442wrh.54.2022.05.25.08.01.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:33 -0700 (PDT) Message-Id: <66a01db47395a86cce6c0e427779c75d02c37757.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:48 +0000 Subject: [PATCH v8 26/30] t/helper/hexdump: add helper to print hexdump of stdin Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Co-authored-by: Johannes Schindelin Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- Makefile | 1 + t/helper/test-hexdump.c | 30 ++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 33 insertions(+) create mode 100644 t/helper/test-hexdump.c diff --git a/Makefile b/Makefile index 5f1623baadd..5afa194aac6 100644 --- a/Makefile +++ b/Makefile @@ -729,6 +729,7 @@ TEST_BUILTINS_OBJS += test-getcwd.o TEST_BUILTINS_OBJS += test-hash-speed.o TEST_BUILTINS_OBJS += test-hash.o TEST_BUILTINS_OBJS += test-hashmap.o +TEST_BUILTINS_OBJS += test-hexdump.o TEST_BUILTINS_OBJS += test-index-version.o TEST_BUILTINS_OBJS += test-json-writer.o TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o diff --git a/t/helper/test-hexdump.c b/t/helper/test-hexdump.c new file mode 100644 index 00000000000..811e89c1bcb --- /dev/null +++ b/t/helper/test-hexdump.c @@ -0,0 +1,30 @@ +#include "test-tool.h" +#include "git-compat-util.h" + +/* + * Read stdin and print a hexdump to stdout. + */ +int cmd__hexdump(int argc, const char **argv) +{ + char buf[1024]; + ssize_t i, len; + int have_data = 0; + + for (;;) { + len = xread(0, buf, sizeof(buf)); + if (len < 0) + die_errno("failure reading stdin"); + if (!len) + break; + + have_data = 1; + + for (i = 0; i < len; i++) + printf("%02x ", (unsigned char)buf[i]); + } + + if (have_data) + putchar('\n'); + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 0424f7adf5d..88c4b28cdfa 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -38,6 +38,7 @@ static struct test_cmd cmds[] = { { "getcwd", cmd__getcwd }, { "hashmap", cmd__hashmap }, { "hash-speed", cmd__hash_speed }, + { "hexdump", cmd__hexdump }, { "index-version", cmd__index_version }, { "json-writer", cmd__json_writer }, { "lazy-init-name-hash", cmd__lazy_init_name_hash }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index c876e8246fb..511f6251bf5 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -29,6 +29,7 @@ int cmd__genzeros(int argc, const char **argv); int cmd__getcwd(int argc, const char **argv); int cmd__hashmap(int argc, const char **argv); int cmd__hash_speed(int argc, const char **argv); +int cmd__hexdump(int argc, const char **argv); int cmd__index_version(int argc, const char **argv); int cmd__json_writer(int argc, const char **argv); int cmd__lazy_init_name_hash(int argc, const char **argv); From patchwork Wed May 25 15:00:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 227E6C433EF for ; Wed, 25 May 2022 15:02:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S245042AbiEYPCm (ORCPT ); Wed, 25 May 2022 11:02:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39896 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239593AbiEYPBz (ORCPT ); Wed, 25 May 2022 11:01:55 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9BF40AFAF5 for ; Wed, 25 May 2022 08:01:39 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id c5-20020a1c3505000000b0038e37907b5bso1291056wma.0 for ; Wed, 25 May 2022 08:01:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cTk9CmwWw4iGOz0CS3tNgQG+VGNLJ2QH91BcLkMnKNg=; b=Y98LTZv//k2BcgOifNdzpbjF8yV1FAn5+PziKkhoZ4nxJJFFVkfVrxcyOLyRDVk4Si e2IYWhA1o+cXQHwn3JCKnVe175koP9VbSqP1DHh18j/sSnCYDdNBM6ZjY5s8WQNoPH2s AdJJR6NS/8tr4vfoAcJZwC9pK9r0x3U6pSVI9Md/lOwwYaf8mraSz1dtSFD9U55Z+otb njkEH962W/fhG8iEEArZZs5O8g7N2zVG6d43kj19e/dZaN6Jr3J7S27L9Vndbv3vX3LS 4xaFpcj3TOYpI5hH2rX1oLKiPoiMMf+q3rhXr42QeewtVzfKDoazXPPmHDEViEDba4bv TU0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cTk9CmwWw4iGOz0CS3tNgQG+VGNLJ2QH91BcLkMnKNg=; b=YZ26i212SBj8tVz5/b/iCTdf75FdgAux/gAFlXYygc8SJwZRhosO238VjPARMAo1tV bLUlJu1CNR2jUFeXWWFbdFqu6Z1JjIfpJLKBy0joVvkAHGuDQWEKkZPRtwUju1ITzc7S iHQXEECgm2Qrp4ntiwVbtMWoptJ1qwsgjNQXSK5BNVC7BbZ+gi6tNEU1Y+mgt2hwSsda GSm6m7VEkRuhVc70zeSwqrnAhdZzSCWQFOzobOszxqQah4JHWXaQRO9TER2NHPfgvJdx 1By8NkKEDQCgHOQg0E1dYpIPCT2TEFKlY0rpUeXXgZv5LCexqMHPpnQLsS0ISQiwaY5i mCnw== X-Gm-Message-State: AOAM532/RCdmFwJ7s7XGc2QSw+k5YZBLpoNIaTDmmzjh7Z6t66LyzBtg xh4nN3O11Z/+pEBIPHp2bvR1zp9VGNI= X-Google-Smtp-Source: ABdhPJwLNY7ZYEoTjya/hBKolT7Penq9Xhf4xRcHAktq4y8pV0mOSVbijgcigLFOwH12mSPGHj0Zzg== X-Received: by 2002:a05:600c:b53:b0:397:335b:546e with SMTP id k19-20020a05600c0b5300b00397335b546emr8798154wmr.177.1653490897654; Wed, 25 May 2022 08:01:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x8-20020a7bc208000000b0039765a7add4sm1996432wmi.29.2022.05.25.08.01.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:36 -0700 (PDT) Message-Id: <25c6066eddc725c7d4c233620fca82d3638b394e.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:49 +0000 Subject: [PATCH v8 27/30] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a set of prereqs to help understand how file names are handled by the filesystem when they contain NFC and NFD Unicode characters. Signed-off-by: Jeff Hostetler --- t/lib-unicode-nfc-nfd.sh | 162 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100755 t/lib-unicode-nfc-nfd.sh diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh new file mode 100755 index 00000000000..22232247efc --- /dev/null +++ b/t/lib-unicode-nfc-nfd.sh @@ -0,0 +1,162 @@ +# Help detect how Unicode NFC and NFD are handled on the filesystem. + +# A simple character that has a NFD form. +# +# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE +# UTF8(NFC): \xc3 \xa9 +# +# NFD: U+0065 LATIN SMALL LETTER E +# U+0301 COMBINING ACUTE ACCENT +# UTF8(NFD): \x65 + \xcc \x81 +# +utf8_nfc=$(printf "\xc3\xa9") +utf8_nfd=$(printf "\x65\xcc\x81") + +# Is the OS or the filesystem "Unicode composition sensitive"? +# +# That is, does the OS or the filesystem allow files to exist with +# both the NFC and NFD spellings? Or, does the OS/FS lie to us and +# tell us that the NFC and NFD forms are equivalent. +# +# This is or may be independent of what type of filesystem we have, +# since it might be handled by the OS at a layer above the FS. +# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a +# collision, for example. +# +# This does not tell us how the Unicode pathname will be spelled +# on disk, but rather only that the two spelling "collide". We +# will examine the actual on disk spelling in a later prereq. +# +test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE ' + mkdir trial_${utf8_nfc} && + mkdir trial_${utf8_nfd} +' + +# Is the spelling of an NFC pathname preserved on disk? +# +# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD +# and on APFS, NFC paths are preserved. As we have established +# above, this is independent of "composition sensitivity". +# +test_lazy_prereq UNICODE_NFC_PRESERVED ' + mkdir c_${utf8_nfc} && + ls | test-tool hexdump >dump && + grep "63 5f c3 a9" dump +' + +# Is the spelling of an NFD pathname preserved on disk? +# +test_lazy_prereq UNICODE_NFD_PRESERVED ' + mkdir d_${utf8_nfd} && + ls | test-tool hexdump >dump && + grep "64 5f 65 cc 81" dump +' + +# The following _DOUBLE_ forms are more for my curiosity, +# but there may be quirks lurking when there are multiple +# combining characters in non-canonical order. + +# Unicode also allows multiple combining characters +# that can be decomposed in pieces. +# +# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI +# UTF8(NFC): \xe1 \xbd \xa7 +# +# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA +# U+0342 COMBINING GREEK PERISPOMENI +# UTF8(NFD1): \xe1 \xbd \xa1 + \xcd \x82 +# +# But U+1f61 decomposes into +# NFD2: U+03c9 GREEK SMALL LETTER OMEGA +# U+0314 COMBINING REVERSED COMMA ABOVE +# UTF8(NFD2): \xcf \x89 + \xcc \x94 +# +# Yielding: \xcf \x89 + \xcc \x94 + \xcd \x82 +# +# Note that I've used the canonical ordering of the +# combinining characters. It is also possible to +# swap them. My testing shows that that non-standard +# ordering also causes a collision in mkdir. However, +# the resulting names don't draw correctly on the +# terminal (implying that the on-disk format also has +# them out of order). +# +greek_nfc=$(printf "\xe1\xbd\xa7") +greek_nfd1=$(printf "\xe1\xbd\xa1\xcd\x82") +greek_nfd2=$(printf "\xcf\x89\xcc\x94\xcd\x82") + +# See if a double decomposition also collides. +# +test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE ' + mkdir trial_${greek_nfc} && + mkdir trial_${greek_nfd2} +' + +# See if the NFC spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED ' + mkdir c_${greek_nfc} && + ls | test-tool hexdump >dump && + grep "63 5f e1 bd a7" dump +' + +# See if the NFD spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED ' + mkdir d_${greek_nfd2} && + ls | test-tool hexdump >dump && + grep "64 5f cf 89 cc 94 cd 82" dump +' + +# The following is for debugging. I found it useful when +# trying to understand the various (OS, FS) quirks WRT +# Unicode and how composition/decomposition is handled. +# For example, when trying to understand how (macOS, APFS) +# and (macOS, HFS) and (macOS, FAT32) compare. +# +# It is rather noisy, so it is disabled by default. +# +if test "$unicode_debug" = "true" +then + if test_have_prereq UNICODE_COMPOSITION_SENSITIVE + then + echo NFC and NFD are distinct on this OS/filesystem. + else + echo NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_NFC_PRESERVED + then + echo NFC maintains original spelling. + else + echo NFC is modified. + fi + + if test_have_prereq UNICODE_NFD_PRESERVED + then + echo NFD maintains original spelling. + else + echo NFD is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE + then + echo DOUBLE NFC and NFD are distinct on this OS/filesystem. + else + echo DOUBLE NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED + then + echo Double NFC maintains original spelling. + else + echo Double NFC is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED + then + echo Double NFD maintains original spelling. + else + echo Double NFD is modified. + fi +fi From patchwork Wed May 25 15:00:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9E5AC433F5 for ; Wed, 25 May 2022 15:02:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S242046AbiEYPCq (ORCPT ); Wed, 25 May 2022 11:02:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40706 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245038AbiEYPBz (ORCPT ); Wed, 25 May 2022 11:01:55 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0292AFB13 for ; Wed, 25 May 2022 08:01:41 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id c5-20020a1c3505000000b0038e37907b5bso1291156wma.0 for ; Wed, 25 May 2022 08:01:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Z9IxtYpTOmmUNIoUdSuKzuSelfyJ5z/hHgvnSuipErQ=; b=BMNR8kkfMlfgCNbjTMXyGV75udLGan5CVrW51XrCFSm1HeDk5DIPouWS0vUNHTo0L3 HyS2VJg3EM9SeWbv9qeT2l8oHQjTezzuVIC5vC6hjhfqK5n/9oHDdCxoe9Vgyj2rkf1i 4/wSD7C9TD5D//tmMmZYgWiJw5tZxh3lIWrD/egEHy8K1QAVxpOeO8j3AoR0ol6OyNUK Iw7n+VO9RxZ5gt/6L1oC5Xdoe9Yhajtj+JrVVfSuJdi8B7P7Wi2XRrzu9IyzRF3+7B+E nkNpFjOM8ZzNBn7c+noa74LcKgYmWQLtrk+zz5ONdfJW06Mspk54Ij/Frq0rlgnUO/GU gjGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Z9IxtYpTOmmUNIoUdSuKzuSelfyJ5z/hHgvnSuipErQ=; b=u8hH57aRHPmB+gIEMnGdFvkOROJtgR6GDZcx73FFReKBLRtS1cTMTJogUav/gICVWF rJy6J/Wy2gvH3mcpK6LBOBRAU3rQu3Akw9t9IlegyP2stGTvq1Q3Nej0R7r56fDzuY+b ruWxCs6nTfvKGxw/jVLZHEKFzhOChI/OewD4/F/WiKlw3vrnekRzL0Hzj3GhkRcDD5aG gZhZAo2X43n3EwZtwRJg0m6JNVlhQ058vk4Wkk83u3p1MARgmSN8FH3UTlU1CJqsDNt2 01a80lc7ekVvrWwLLgjnES88HgvfSfN9hnExxvfqjYcEfzWMd9tBluGS9emdcAzcjD7G NLlg== X-Gm-Message-State: AOAM531v6YVy0BnkYt7Wz7uEt2Upi2SuB3u1wGW7tlegKTQ+DRiDXUUd gftDIZATylBEYeLruOKBUnGEtveC6Yw= X-Google-Smtp-Source: ABdhPJyPOahcmm3VLrD0VSqwD1ZmVxZWtV4kw5AXj/TykGIHjyTabGbffuE6Z7HL01dKdS23fqywAw== X-Received: by 2002:a7b:c5cd:0:b0:38c:8b1b:d220 with SMTP id n13-20020a7bc5cd000000b0038c8b1bd220mr8477182wmk.118.1653490900051; Wed, 25 May 2022 08:01:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r12-20020adff70c000000b0020c5253d925sm2193418wrp.113.2022.05.25.08.01.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:38 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:50 +0000 Subject: [PATCH v8 28/30] t7527: test Unicode NFC/NFD handling on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Confirm that the daemon reports events using the on-disk spelling for Unicode NFC/NFD characters. On APFS we still have Unicode aliasing, so we cannot create two files that only differ by NFC/NFD, but the on-disk format preserves the spelling used to create the file. On HFS+ we also have aliasing, but the path is always stored on disk in NFD. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 55 ++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index fbb7d6aef6e..9edae3ed830 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -868,4 +868,59 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' egrep "^event: abc/def/xyz$" ./insensitive.trace ' +# The variable "unicode_debug" is defined in the following library +# script to dump information about how the (OS, FS) handles Unicode +# composition. Uncomment the following line if you want to enable it. +# +# unicode_debug=true + +. "$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh" + +# See if the OS or filesystem does NFC/NFD aliasing/munging. +# +# The daemon should err on the side of caution and send BOTH the +# NFC and NFD forms. It does not know the original spelling of +# the pathname (how the user thinks it should be spelled), so +# emit both and let the client decide (when necessary). This is +# similar to "core.precomposeUnicode". +# +test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' ' + test_when_finished "stop_daemon_delete_repo test_unicode" && + + git init test_unicode && + + start_daemon -C test_unicode --tf "$PWD/unicode.trace" && + + # Create a directory using an NFC spelling. + # + mkdir test_unicode/nfc && + mkdir test_unicode/nfc/c_${utf8_nfc} && + + # Create a directory using an NFD spelling. + # + mkdir test_unicode/nfd && + mkdir test_unicode/nfd/d_${utf8_nfd} && + + git -C test_unicode fsmonitor--daemon stop && + + if test_have_prereq UNICODE_NFC_PRESERVED + then + # We should have seen NFC event from OS. + # We should not have synthesized an NFD event. + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace && + egrep -v "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace + else + # We should have seen NFD event from OS. + # We should have synthesized an NFC event. + egrep "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace + fi && + + # We assume UNICODE_NFD_PRESERVED. + # We should have seen explicit NFD from OS. + # We should have synthesized an NFC event. + egrep "^event: nfd/d_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfd/d_${utf8_nfc}/?$" ./unicode.trace +' + test_done From patchwork Wed May 25 15:00:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C2AAC433F5 for ; Wed, 25 May 2022 15:02:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236042AbiEYPCw (ORCPT ); Wed, 25 May 2022 11:02:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39834 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239987AbiEYPB4 (ORCPT ); Wed, 25 May 2022 11:01:56 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27B4BAFB35 for ; Wed, 25 May 2022 08:01:44 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id bg25so12631266wmb.4 for ; Wed, 25 May 2022 08:01:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uU2/c+aPwrNZLDw5bIR3T3N2u2DdFYRmZnPeQS1ZxNA=; b=PN7zz//7vtGO8uHoiUVcldNxos/iH1OzBNr58CgNferJ04sWITLusyb1FphSlyefz2 TJPt/YKHaxttBCC7b0oo/KRMsve19wN87E/QXwhlH1N6z1b8ZQO0MkFJpaav2qXgjEPU lGNZWcTZqsQwvYYSSJ0jKmUxmMlCUn+y3dhM/kFnj1bPRE7Wy3quIM5PMka3bqqzEG2l pBOHy6+7iH39aCsqMqGOq5wE4XI5N0BKdjYMJIcKeSNqo3uncf5uCxqis/j5jEun2aUs 7nFfThBG17f+wm7ucrAln9DeSNVtoBgiKdo37aPob3SzHKVfUOHFvaxjISsV3jLPI4dD b7tw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uU2/c+aPwrNZLDw5bIR3T3N2u2DdFYRmZnPeQS1ZxNA=; b=6ywvWkDbDwA4gtr75OKM71/tjECAlenmuglLtkOapvBat7PXfnXkyph0fXSg13IcET 0bfXMllHve23yJSX4c0LIiNNm5AaVxlMb9HEAhjdI7ew5HCaKFmEXMGCIWWLI8SEZmfd j91kcIXNZ6xhJNZGFx7BVG58D2KTdCFn/V6klXzRALbbSyWlYeVXUbAZWRw77hL+3LVI Z9SS2revxR2/1sLNgGxiFQkEgXl6dHkFZ8ry6xCqenDKvNjF8Wy0wvp4Mt9a93hNlzL6 5AY4dU4TNHUcvnmpM71ZS4GTprseLZ1BYFChe19KMy3YkpeFH5Su1DYb7luRiF1nCZhU 5Wsg== X-Gm-Message-State: AOAM530yvpm7u4exAcxOWJhWlsvona0oQnPX5mteMY2QgD6WmQO+c5Ld cuZUuD2B0hNQFA2tbNNvkCiSy6ozHJs= X-Google-Smtp-Source: ABdhPJx1S9itMT9ntMqL9VBfDtp1HEhAqz6nEI+ZxG5saRxiQLm7ylf8QuroRdw0e7Oj0OdNgV/DHA== X-Received: by 2002:a05:600c:600b:b0:395:b668:b3fa with SMTP id az11-20020a05600c600b00b00395b668b3famr8477848wmb.46.1653490902077; Wed, 25 May 2022 08:01:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t17-20020a0560001a5100b0020c5253d914sm2287449wry.96.2022.05.25.08.01.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:41 -0700 (PDT) Message-Id: <25676ca4ec2bdaa6fdaf265dcdfe39b06d4d7bfe.1653490853.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 25 May 2022 15:00:51 +0000 Subject: [PATCH v8 29/30] fsmonitor--daemon: allow --super-prefix argument Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a test in t7527 to verify that we get a stray warning from `git fsmonitor--daemon start` when indirectly called from `git submodule absorbgitdirs`. Update `git fsmonitor--daemon` to take (and ignore) the `--super-prefix` argument to suppress the warning. When we have: 1. a submodule with a `sub/.git/` directory (rather than a `sub/.git` file). 2. `core.fsmonitor` is turned on in the submodule, but the daemon is not yet started in the submodule. 3. and someone does a `git submodule absorbgitdirs` in the super. Git will recursively invoke `git submodule--helper absorb-git-dirs` in the submodule. This will read the index and may attempt to start the fsmonitor--daemon with the `--super-prefix` argument. `git fsmonitor--daemon start` does not accept the `--super-prefix` argument and causes a warning to be issued. This does not cause a problem because the `refresh_index()` code assumes a trivial response if the daemon does not start. The net-net is a harmelss, but stray warning. Lets eliminate the warning. Signed-off-by: Jeff Hostetler --- git.c | 2 +- t/t7527-builtin-fsmonitor.sh | 50 ++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/git.c b/git.c index 3d8e48cf555..e6fdac1f8e3 100644 --- a/git.c +++ b/git.c @@ -537,7 +537,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, - { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, SUPPORT_SUPER_PREFIX | RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY }, diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 9edae3ed830..19edc96fd4d 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -832,6 +832,56 @@ test_expect_success 'submodule always visited' ' my_match_and_clean ' +# If a submodule has a `sub/.git/` directory (rather than a file +# pointing to the super's `.git/modules/sub`) and `core.fsmonitor` +# turned on in the submodule and the daemon is not yet started in +# the submodule, and someone does a `git submodule absorbgitdirs` +# in the super, Git will recursively invoke `git submodule--helper` +# to do the work and this may try to read the index. This will +# try to start the daemon in the submodule *and* pass (either +# directly or via inheritance) the `--super-prefix` arg to the +# `git fsmonitor--daemon start` command inside the submodule. +# This causes a warning because fsmonitor--daemon does take that +# global arg (see the table in git.c) +# +# This causes a warning when trying to start the daemon that is +# somewhat confusing. It does not seem to hurt anything because +# the fsmonitor code maps the query failure into a trivial response +# and does the work anyway. +# +# It would be nice to silence the warning, however. + +have_t2_error_event () { + log=$1 + msg="fsmonitor--daemon doesnQt support --super-prefix" && + + tr '\047' Q <$1 | grep -e "$msg" +} + +test_expect_success "stray submodule super-prefix warning" ' + test_when_finished "rm -rf super; \ + rm -rf sub; \ + rm super-sub.trace" && + + create_super super && + create_sub sub && + + # Copy rather than submodule add so that we get a .git dir. + cp -R ./sub ./super/dir_1/dir_2/sub && + + git -C super/dir_1/dir_2/sub config core.fsmonitor true && + + git -C super submodule add ../sub ./dir_1/dir_2/sub && + git -C super commit -m "add sub" && + + test_path_is_dir super/dir_1/dir_2/sub/.git && + + GIT_TRACE2_EVENT="$PWD/super-sub.trace" \ + git -C super submodule absorbgitdirs && + + ! have_t2_error_event super-sub.trace +' + # On a case-insensitive file system, confirm that the daemon # notices when the .git directory is moved/renamed/deleted # regardless of how it is spelled in the the FS event. From patchwork Wed May 25 15:00:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12861355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82861C433F5 for ; Wed, 25 May 2022 15:02:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231556AbiEYPCu (ORCPT ); Wed, 25 May 2022 11:02:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39862 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S245057AbiEYPB5 (ORCPT ); Wed, 25 May 2022 11:01:57 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6B01B0414 for ; Wed, 25 May 2022 08:01:45 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id p19so4186902wmg.2 for ; Wed, 25 May 2022 08:01:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=btuy/6nsewzbtzT+ydmOLWgKtgnAfHH3ecrs/ObWIdg=; b=knQ3ljRqldvTgMWI9dsIAUkDmsGyxsnfNDY+WvSFshcEIoXarKU3NF40/ZDhUpEqio 1RU/19w7xg+Z13MD3+a2voCZdvHuhg8vI7Ef6L6V1Bf//NhkVaxSa4NOoA/jSf0plOGT wN0paQpmZa3Nq+fh2Ya9b4GX7jwqFc9Z5odmTsu2XehoFKsYYiL0eHIvY5lzh7V6BWyC htg96xja/6YVVSXVYnzlOfy6lCK5B5I+xV8BclybVVHRCKhyl2PMHf0vyN/14Nok/lX4 0SpPyKcw5ukGW1p1YwVU0vdHVE/lxottACtPPqMfALiLKsJ+kwxryB1t8e8+t+DWIHeW 69Uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=btuy/6nsewzbtzT+ydmOLWgKtgnAfHH3ecrs/ObWIdg=; b=idXvh0yXlwxdXGKz6o2gPjXypVz6MhfnSY3/JY2K9uOXPyMl5gRZbhHus+sN7qLOyv ngwxNF5HiCW3Gro3PoNRMGsE3aVZu4VW2LnlcI3G0e4KyljM5yg/W1O1nFWx/uxWOsPY C9S23FZEtbqysSyKskI6Ox5A80yRsl1hNaGDcK2OWaOZaTu0At46ibxM1uHcm28oA/r+ xo2HjS7aRq60j5eN+BMI1283OjgQyn9rYHRIJt5V0ivi5YdsH8vKubGvMug/6HzUiB0W 4blKa60+IXNKrJfAhBWu4mZcL99fKnwrVsm20L4gGNKIJySVkxJq+zfmo8dyJaUIvEPd qLYQ== X-Gm-Message-State: AOAM531h59jnJ/LQaNMphz0rTOrBPuVwSY7kEhO1lW3oe2eQtPsPKtcI rEpXZVFQj9cumKOMn8h/2eh37g86OEE= X-Google-Smtp-Source: ABdhPJx4r9ff8gC4yhX/ewHglbWdTgiBkelRQCBkGlXHLk+cEWYN2ZfAuRU0RtSlvYV9Key7UAaqRQ== X-Received: by 2002:a05:600c:a03:b0:395:bb63:b457 with SMTP id z3-20020a05600c0a0300b00395bb63b457mr8647790wmp.192.1653490903683; Wed, 25 May 2022 08:01:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az19-20020a05600c601300b003973c54bd69sm2093056wmb.1.2022.05.25.08.01.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 May 2022 08:01:43 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 25 May 2022 15:00:52 +0000 Subject: [PATCH v8 30/30] t7527: improve implicit shutdown testing in fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , rsbecker@nexbridge.com, Bagas Sanjaya , Johannes Schindelin , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Refactor the tests that exercise implicit shutdown cases to make them more robust and less racy. The fsmonitor--daemon will implicitly shutdown in a variety of situations, such as when the ".git" directory is deleted or renamed. The existing tests would delete or rename the directory, sleep for one second, and then check the status of the daemon. This is racy, since the client/status command has no way to sync with the daemon. This was noticed occasionally on very slow CI build machines where it would cause a random test to fail. Replace the simple sleep with a sleep-and-retry loop. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 54 ++++++++++++++++++++++++++---------- 1 file changed, 40 insertions(+), 14 deletions(-) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 19edc96fd4d..56c0dfffea6 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -124,6 +124,36 @@ test_expect_success 'implicit daemon start' ' test_must_fail git -C test_implicit fsmonitor--daemon status ' +# Verify that the daemon has shutdown. Spin a few seconds to +# make the test a little more robust during CI testing. +# +# We're looking for an implicit shutdown, such as when we delete or +# rename the ".git" directory. Our delete/rename will cause a file +# system event that the daemon will see and the daemon will +# auto-shutdown as soon as it sees it. But this is racy with our `git +# fsmonitor--daemon status` commands (and we cannot use a cookie file +# here to help us). So spin a little and give the daemon a chance to +# see the event. (This is primarily for underpowered CI build/test +# machines (where it might take a moment to wake and reschedule the +# daemon process) to avoid false alarms during test runs.) +# +IMPLICIT_TIMEOUT=5 + +verify_implicit_shutdown () { + r=$1 && + + k=0 && + while test "$k" -lt $IMPLICIT_TIMEOUT + do + git -C $r fsmonitor--daemon status || return 0 + + sleep 1 + k=$(( $k + 1 )) + done && + + return 1 +} + test_expect_success 'implicit daemon stop (delete .git)' ' test_when_finished "stop_daemon_delete_repo test_implicit_1" && @@ -142,10 +172,9 @@ test_expect_success 'implicit daemon stop (delete .git)' ' # This would make the test result dependent upon whether we # were using fsmonitor on our development worktree. # - sleep 1 && mkdir test_implicit_1/.git && - test_must_fail git -C test_implicit_1 fsmonitor--daemon status + verify_implicit_shutdown test_implicit_1 ' test_expect_success 'implicit daemon stop (rename .git)' ' @@ -160,10 +189,9 @@ test_expect_success 'implicit daemon stop (rename .git)' ' # See [1] above. # - sleep 1 && mkdir test_implicit_2/.git && - test_must_fail git -C test_implicit_2 fsmonitor--daemon status + verify_implicit_shutdown test_implicit_2 ' # File systems on Windows may or may not have shortnames. @@ -194,13 +222,11 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' ' # mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz && - sleep 1 && - # put it back so that our status will not crawl out to our - # parent directory. + # See [1] above. # this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}. mv test_implicit_1s/.gitxyz test_implicit_1s/.git && - test_must_fail git -C test_implicit_1s fsmonitor--daemon status + verify_implicit_shutdown test_implicit_1s ' # Here we first create a file with LONGNAME of "GIT~1" before @@ -223,12 +249,10 @@ test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' ' # mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz && - sleep 1 && - # put it back so that our status will not crawl out to our - # parent directory. + # See [1] above. mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git && - test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status + verify_implicit_shutdown test_implicit_1s2 ' test_expect_success 'cannot start multiple daemons' ' @@ -905,9 +929,11 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' # Rename .git using an alternate spelling to verify that that # daemon detects it and automatically shuts down. mv test_insensitive/.GIT test_insensitive/.FOO && - sleep 1 && + + # See [1] above. mv test_insensitive/.FOO test_insensitive/.git && - test_must_fail git -C test_insensitive fsmonitor--daemon status && + + verify_implicit_shutdown test_insensitive && # Verify that events were reported using on-disk spellings of the # directories and files that we touched. We may or may not get a