From patchwork Tue Mar 8 22:15:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774441 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CABA6C433EF for ; Tue, 8 Mar 2022 22:15:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350639AbiCHWQd (ORCPT ); Tue, 8 Mar 2022 17:16:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51290 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233079AbiCHWQb (ORCPT ); Tue, 8 Mar 2022 17:16:31 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F29D56220 for ; Tue, 8 Mar 2022 14:15:33 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id i8so240204wrr.8 for ; Tue, 08 Mar 2022 14:15:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5WxH5crprXI7/W5RjfU0fhLCaeINShHV5FcEKRxgqNs=; b=BRq4EMN3F4cPg0qUL4WOu1ulBiPYQrYbw0FJjfzmkNjGlY2XPyijl3etoCyIdLaz9n jLXmQKDP1EbkZsrb7tmlTt7wHibzgoCzJhouQhD7pwhMR/jCCttZbHAgh4sxDKKD49vI +hB7x+JYB7Yzk8NBC+QeXyXNThmokrWJTPy+sTF8WonCPnSnXXfLt3Rd6UgWXpzmb+di lOVYcV/s2IpV/2bSo/BX2wrFL2NbjHB3BIwqoo41CHSBgk+82qTCRUzPHoz2Ue7y8Cqj JMJvlsw+1Q8vV/hcl7FGNtHmSEgOOR8Zk5kk0xFkjDQArDD6G8n+sdZmo7QUogQcdbc1 +ieA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5WxH5crprXI7/W5RjfU0fhLCaeINShHV5FcEKRxgqNs=; b=4gjHdRVetsoPRYfvZWLEkIBh92O1KlaJPtL2kfs2uWN0ibjFbufbbvh2Jds+sx5roh tbVRbgHcE0gTfdW8j2VUrQV53vWmo2PPVgWEwvpeUlXDeLvlk1ePRWq9bUB3O3/Ogct1 1nCBjw/pqPOlFp6LyvRooooLDMdGnR6a8vnqcT0Y86fxGgDQ/fw5nIzHuFfboG/QncjA hizFwElzSrmGbTwk9GUrx/RvFvd3RJVr8hOpG0Y5FW27X/WFmIeamlgnuo80BU+gpbQB PKwcuRwD4vLzsB+ZIjiGNBjBJ1G7jRoqyx3+FVJOdQRlRl6CCtJKldNlsY26u0KesQC1 8bdg== X-Gm-Message-State: AOAM533/ia/yuOGxlJWfjC3q8qsQzCk+txgXwDhJS1Ghr0sQ8Btwvctd lObY7hr2euXbNtWQdTJsWors7MmLmCg= X-Google-Smtp-Source: ABdhPJyr/mYrDSPDQE8VBjfEXmOzELznGnPZO4Wfx6JmhRKS1AR/KvioGsHYT7peLlZaas/PLO0LDQ== X-Received: by 2002:adf:8003:0:b0:1f1:d81b:d752 with SMTP id 3-20020adf8003000000b001f1d81bd752mr13589114wrk.429.1646777731434; Tue, 08 Mar 2022 14:15:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v20-20020a7bcb54000000b0037fa63db8aasm3567162wmj.5.2022.03.08.14.15.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:31 -0800 (PST) Message-Id: <34619e0652b4723de40d9b8972ec45aaac9009e6.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:01 +0000 Subject: [PATCH v2 01/27] fsm-listen-win32: handle shortnames Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach FSMonitor daemon on Windows to recognize shortname paths as aliases of normal longname paths. FSMonitor clients, such as `git status`, should receive the longname spelling of changed files (when possible). Sometimes we receive FS events using the shortname, such as when a CMD shell runs "RENAME GIT~1 FOO" or "RMDIR GIT~1". The FS notification arrives using whatever combination of long and shortnames were used by the other process. (Shortnames do seem to be case normalized, however.) Use Windows GetLongPathNameW() to try to map the pathname spelling in the notification event into the normalized longname spelling. (This can fail if the file/directory is deleted, moved, or renamed, because we are asking the FS for the mapping in response to the event and after it has already happened, but we try.) Special case the shortname spelling of ".git" to avoid under-reporting these events. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-win32.c | 363 +++++++++++++++++++++++----- t/t7527-builtin-fsmonitor.sh | 65 +++++ 2 files changed, 374 insertions(+), 54 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index c2d11acbc1e..f4673d7d8b1 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -25,6 +25,9 @@ struct one_watch DWORD count; struct strbuf path; + wchar_t wpath_longname[MAX_PATH + 1]; + DWORD wpath_longname_len; + HANDLE hDir; HANDLE hEvent; OVERLAPPED overlapped; @@ -34,6 +37,21 @@ struct one_watch * need to later call GetOverlappedResult() and possibly CancelIoEx(). */ BOOL is_active; + + /* + * Are shortnames enabled on the containing drive? This is + * always true for "C:/" drives and usually never true for + * other drives. + * + * We only set this for the worktree because we only need to + * convert shortname paths to longname paths for items we send + * to clients. (We don't care about shortname expansion for + * paths inside a GITDIR because we never send them to + * clients.) + */ + BOOL has_shortnames; + BOOL has_tilda; + wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; struct fsmonitor_daemon_backend_data @@ -51,17 +69,18 @@ struct fsmonitor_daemon_backend_data }; /* - * Convert the WCHAR path from the notification into UTF8 and - * then normalize it. + * Convert the WCHAR path from the event into UTF8 and normalize it. + * + * `wpath_len` is in WCHARS not bytes. */ -static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, +static int normalize_path_in_utf8(wchar_t *wpath, DWORD wpath_len, struct strbuf *normalized_path) { int reserve; int len = 0; strbuf_reset(normalized_path); - if (!info->FileNameLength) + if (!wpath_len) goto normalize; /* @@ -70,12 +89,12 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, * sequence of 2 UTF8 characters. That should let us * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. */ - reserve = info->FileNameLength + 1; + reserve = 2 * wpath_len + 1; strbuf_grow(normalized_path, reserve); for (;;) { - len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, - info->FileNameLength / sizeof(WCHAR), + len = WideCharToMultiByte(CP_UTF8, 0, + wpath, wpath_len, normalized_path->buf, strbuf_avail(normalized_path) - 1, NULL, NULL); @@ -83,9 +102,7 @@ static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, goto normalize; if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { error("[GLE %ld] could not convert path to UTF-8: '%.*ls'", - GetLastError(), - (int)(info->FileNameLength / sizeof(WCHAR)), - info->FileName); + GetLastError(), (int)wpath_len, wpath); return -1; } @@ -98,6 +115,152 @@ normalize: return strbuf_normalize_path(normalized_path); } +/* + * See if the worktree root directory has shortnames enabled. + * This will help us decide if we need to do an expensive shortname + * to longname conversion on every notification event. + * + * We do not want to create a file to test this, so we assume that the + * root directory contains a ".git" file or directory. (Our caller + * only calls us for the worktree root, so this should be fine.) + * + * Remember the spelling of the shortname for ".git" if it exists. + */ +static void check_for_shortnames(struct one_watch *watch) +{ + wchar_t buf_in[MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + wchar_t *last_slash = NULL; + wchar_t *last_bslash = NULL; + wchar_t *last; + + /* build L"/.git" */ + wcscpy(buf_in, watch->wpath_longname); + wcscpy(buf_in + watch->wpath_longname_len, L".git"); + + if (!GetShortPathNameW(buf_in, buf_out, MAX_PATH)) + return; + + last_slash = wcsrchr(buf_out, L'/'); + last_bslash = wcsrchr(buf_out, L'\\'); + if (last_slash > last_bslash) + last = last_slash + 1; + else if (last_bslash) + last = last_bslash + 1; + else + last = buf_out; + + if (!wcscmp(last, L".git")) + return; + + watch->has_shortnames = 1; + wcsncpy(watch->dotgit_shortname, last, + ARRAY_SIZE(watch->dotgit_shortname)); + + /* + * The shortname for ".git" is usually of the form "GIT~1", so + * we should be able to avoid shortname to longname mapping on + * every notification event if the source string does not + * contain a "~". + * + * However, the documentation for GetLongPathNameW() says + * that there are filesystems that don't follow that pattern + * and warns against this optimization. + * + * Lets test this. + */ + if (wcschr(watch->dotgit_shortname, L'~')) + watch->has_tilda = 1; +} + +enum get_relative_result { + GRR_NO_CONVERSION_NEEDED, + GRR_HAVE_CONVERSION, + GRR_SHUTDOWN, +}; + +/* + * Info notification paths are relative to the root of the watch. + * If our CWD is still at the root, then we can use relative paths + * to convert from shortnames to longnames. If our process has a + * different CWD, then we need to construct an absolute path, do + * the conversion, and then return the root-relative portion. + * + * We use the longname form of the root as our basis and assume that + * it already has a trailing slash. + * + * `wpath_len` is in WCHARS not bytes. + */ +static enum get_relative_result get_relative_longname( + struct one_watch *watch, + const wchar_t *wpath, DWORD wpath_len, + wchar_t *wpath_longname) +{ + wchar_t buf_in[2 * MAX_PATH + 1]; + wchar_t buf_out[MAX_PATH + 1]; + DWORD root_len; + + /* Build L"/" */ + root_len = watch->wpath_longname_len; + wcsncpy(buf_in, watch->wpath_longname, root_len); + wcsncpy(buf_in + root_len, wpath, wpath_len); + buf_in[root_len + wpath_len] = 0; + + /* + * We don't actually know if the source pathname is a + * shortname or a longname. This routine allows either to be + * given as input. + */ + if (!GetLongPathNameW(buf_in, buf_out, MAX_PATH)) { + /* + * The shortname to longname conversion can fail for + * various reasons, for example if the file has been + * deleted. (That is, if we just received a + * delete-file notification event and the file is + * already gone, we can't ask the file system to + * lookup the longname for it. Likewise, for moves + * and renames where we are given the old name.) + * + * Since deleting or moving a file or directory by its + * shortname is rather obscure, I'm going ignore the + * failure and ask the caller to report the original + * relative path. This seems kinder than failing here + * and forcing a resync. Besides, forcing a resync on + * every file/directory delete would effectively + * cripple monitoring. + * + * We might revisit this in the future. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (!wcscmp(buf_in, buf_out)) { + /* + * The path does not have a shortname alias. + */ + return GRR_NO_CONVERSION_NEEDED; + } + + if (wcsncmp(buf_in, buf_out, root_len)) { + /* + * The spelling of the root directory portion of the computed + * longname has changed. This should not happen. Basically, + * it means that we don't know where (without recomputing the + * longname of just the root directory) to split out the + * relative path. Since this should not happen, I'm just + * going to let this fail and force a shutdown (because all + * subsequent events are probably going to see the same + * mismatch). + */ + return GRR_SHUTDOWN; + } + + /* Return the worktree root-relative portion of the longname. */ + + wcscpy(wpath_longname, buf_out + root_len); + return GRR_HAVE_CONVERSION; +} + void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); @@ -111,7 +274,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, DWORD share_mode = FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; HANDLE hDir; - wchar_t wpath[MAX_PATH]; + DWORD len_longname; + wchar_t wpath[MAX_PATH + 1]; + wchar_t wpath_longname[MAX_PATH + 1]; if (xutftowcs_path(wpath, path) < 0) { error(_("could not convert to wide characters: '%s'"), path); @@ -128,6 +293,20 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, return NULL; } + if (!GetLongPathNameW(wpath, wpath_longname, MAX_PATH)) { + error(_("[GLE %ld] could not get longname of '%s'"), + GetLastError(), path); + CloseHandle(hDir); + return NULL; + } + + len_longname = wcslen(wpath_longname); + if (wpath_longname[len_longname - 1] != L'/' && + wpath_longname[len_longname - 1] != L'\\') { + wpath_longname[len_longname++] = L'/'; + wpath_longname[len_longname] = 0; + } + CALLOC_ARRAY(watch, 1); watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ @@ -135,6 +314,9 @@ static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, strbuf_init(&watch->path, 0); strbuf_addstr(&watch->path, path); + wcscpy(watch->wpath_longname, wpath_longname); + watch->wpath_longname_len = len_longname; + watch->hDir = hDir; watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); @@ -258,6 +440,62 @@ static void cancel_rdcw_watch(struct one_watch *watch) watch->is_active = FALSE; } +/* + * Process a single relative pathname event. + * Return 1 if we should shutdown. + */ +static int process_1_worktree_event( + struct string_list *cookie_list, + struct fsmonitor_batch **batch, + const struct strbuf *path, + enum fsmonitor_path_type t, + DWORD info_action) +{ + const char *slash; + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path->buf); + string_list_append(cookie_list, + slash ? slash + 1 : path->buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info_action == FILE_ACTION_REMOVED) || + (info_action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + return 1; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!*batch) + *batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(*batch, path->buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path->buf); + } + + return 0; +} + /* * Process filesystem events that happen anywhere (recursively) under the * root directory. For a normal working directory, this includes @@ -274,6 +512,7 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) struct string_list cookie_list = STRING_LIST_INIT_DUP; struct fsmonitor_batch *batch = NULL; const char *p = watch->buffer; + wchar_t wpath_longname[MAX_PATH + 1]; /* * If the kernel gets more events than will fit in the kernel @@ -306,54 +545,63 @@ static int process_worktree_events(struct fsmonitor_daemon_state *state) */ for (;;) { FILE_NOTIFY_INFORMATION *info = (void *)p; - const char *slash; + wchar_t *wpath = info->FileName; + DWORD wpath_len = info->FileNameLength / sizeof(WCHAR); enum fsmonitor_path_type t; + enum get_relative_result grr; + + if (watch->has_shortnames) { + if (!wcscmp(wpath, watch->dotgit_shortname)) { + /* + * This event exactly matches the + * spelling of the shortname of + * ".git", so we can skip some steps. + * + * (This case is odd because the user + * can "rm -rf GIT~1" and we cannot + * use the filesystem to map it back + * to ".git".) + */ + strbuf_reset(&path); + strbuf_addstr(&path, ".git"); + t = IS_DOT_GIT; + goto process_it; + } - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) - goto skip_this_path; - - t = fsmonitor_classify_path_workdir_relative(path.buf); - - switch (t) { - case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: - /* special case cookie files within .git */ - - /* Use just the filename of the cookie file. */ - slash = find_last_dir_sep(path.buf); - string_list_append(&cookie_list, - slash ? slash + 1 : path.buf); - break; - - case IS_INSIDE_DOT_GIT: - /* ignore everything inside of "/.git/" */ - break; + if (watch->has_tilda && !wcschr(wpath, L'~')) { + /* + * Shortnames on this filesystem have tildas + * and the notification path does not have + * one, so we assume that it is a longname. + */ + goto normalize_it; + } - case IS_DOT_GIT: - /* "/.git" was deleted (or renamed away) */ - if ((info->Action == FILE_ACTION_REMOVED) || - (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { - trace2_data_string("fsmonitor", NULL, - "fsm-listen/dotgit", - "removed"); + grr = get_relative_longname(watch, wpath, wpath_len, + wpath_longname); + switch (grr) { + case GRR_NO_CONVERSION_NEEDED: /* use info buffer as is */ + break; + case GRR_HAVE_CONVERSION: + wpath = wpath_longname; + wpath_len = wcslen(wpath); + break; + default: + case GRR_SHUTDOWN: goto force_shutdown; } - break; + } - case IS_WORKDIR_PATH: - /* queue normal pathname */ - if (!batch) - batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, path.buf); - break; +normalize_it: + if (normalize_path_in_utf8(wpath, wpath_len, &path) == -1) + goto skip_this_path; - case IS_GITDIR: - case IS_INSIDE_GITDIR: - case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: - default: - BUG("unexpected path classification '%d' for '%s'", - t, path.buf); - } + t = fsmonitor_classify_path_workdir_relative(path.buf); + +process_it: + if (process_1_worktree_event(&cookie_list, &batch, &path, t, + info->Action)) + goto force_shutdown; skip_this_path: if (!info->NextEntryOffset) @@ -382,6 +630,9 @@ force_shutdown: * Note that we DO NOT get filesystem events on the external * itself (it is not inside something that we are watching). In particular, * we do not get an event if the external is deleted. + * + * Also, we do not care about shortnames within the external , since + * we never send these paths to clients. */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { @@ -403,8 +654,10 @@ static int process_gitdir_events(struct fsmonitor_daemon_state *state) const char *slash; enum fsmonitor_path_type t; - strbuf_reset(&path); - if (normalize_path_in_utf8(info, &path) == -1) + if (normalize_path_in_utf8( + info->FileName, + info->FileNameLength / sizeof(WCHAR), + &path) == -1) goto skip_this_path; t = fsmonitor_classify_path_gitdir_relative(path.buf); @@ -538,6 +791,8 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) if (!data->watch_worktree) goto failed; + check_for_shortnames(data->watch_worktree); + if (state->nr_paths_watching > 1) { data->watch_gitdir = create_watch(state, state->path_gitdir_watch.buf); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 0ccbfb9616f..dbca7f835eb 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -123,6 +123,71 @@ test_expect_success 'implicit daemon stop (rename .git)' ' test_must_fail git -C test_implicit_2 fsmonitor--daemon status ' +# File systems on Windows may or may not have shortnames. +# This is a volume-specific setting on modern systems. +# "C:/" drives are required to have them enabled. Other +# hard drives default to disabled. +# +# This is a crude test to see if shortnames are enabled +# on the volume containing the test directory. It is +# crude, but it does not require elevation like `fsutil`. +# +test_lazy_prereq SHORTNAMES ' + mkdir .foo && + test -d "FOO~1" +' + +# Here we assume that the shortname of ".git" is "GIT~1". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~1)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s" && + + git init test_implicit_1s && + + start_daemon test_implicit_1s && + + # renaming the .git directory will implicitly stop the daemon. + # this moves {.git, GIT~1} to {.gitxyz, GITXYZ~1}. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s/GIT~1 test_implicit_1s/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + # this moves {.gitxyz, GITXYZ~1} to {.git, GIT~1}. + mv test_implicit_1s/.gitxyz test_implicit_1s/.git && + + test_must_fail git -C test_implicit_1s fsmonitor--daemon status +' + +# Here we first create a file with LONGNAME of "GIT~1" before +# we create the repo. This will cause the shortname of ".git" +# to be "GIT~2". +test_expect_success MINGW,SHORTNAMES 'implicit daemon stop (rename GIT~2)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1s2" && + + mkdir test_implicit_1s2 && + echo HELLO >test_implicit_1s2/GIT~1 && + git init test_implicit_1s2 && + + test_path_is_file test_implicit_1s2/GIT~1 && + test_path_is_dir test_implicit_1s2/GIT~2 && + + start_daemon test_implicit_1s2 && + + # renaming the .git directory will implicitly stop the daemon. + # the rename-from FS Event will contain the shortname. + # + mv test_implicit_1s2/GIT~2 test_implicit_1s2/.gitxyz && + + sleep 1 && + # put it back so that our status will not crawl out to our + # parent directory. + mv test_implicit_1s2/.gitxyz test_implicit_1s2/.git && + + test_must_fail git -C test_implicit_1s2 fsmonitor--daemon status +' + test_expect_success 'cannot start multiple daemons' ' test_when_finished "stop_daemon_delete_repo test_multiple" && From patchwork Tue Mar 8 22:15:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774442 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84E24C433F5 for ; Tue, 8 Mar 2022 22:15:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350645AbiCHWQf (ORCPT ); Tue, 8 Mar 2022 17:16:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243250AbiCHWQc (ORCPT ); Tue, 8 Mar 2022 17:16:32 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2061554BCE for ; Tue, 8 Mar 2022 14:15:34 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id i8so240262wrr.8 for ; Tue, 08 Mar 2022 14:15:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=jbOjNU2UPDnti2uDmzPj9ZkhrM+9Lbl5QUOXunuauhU=; b=IU8U/qAdl/c/eCLlBYRq6h3FAoDFvtYbywVfNboAWlrIDA9+OtFQbBVofhI0Lee7uk 0X3TzZlnEDzrirjCoDVnVh65pXMOe1hzDmx8sNdcOuqS8yVQx6LknsIa5n+C4AvGaTI2 9bIA46NT47UmRg9Z3DxAQCvrac/0XSDctnbgdXwJzIpe+2Lhboh9Tq73rZcwbjRdAYzq rAqrOBCm6OAHgJ8keUVa/Vz3r8tHdMlCx5VjUFjA2mPl4Ro95NL5mJ2+702+LmH/o+bq GybzIwjyrLKawXXWFOz+NNiTymUefaQjoMp25Jg87+ZkH8fPIwX6kWTNt2A/jp42l134 oI6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=jbOjNU2UPDnti2uDmzPj9ZkhrM+9Lbl5QUOXunuauhU=; b=gwAs1RPaBFfxYu/pR+JwB2LUt6Jl6dTo3eCoAdDumKhXRWzZfFFXORrc5lTvDSs+Hu D5quQWrvyd5bZmXPpgkRHQ6ghQS0FUGUq4xHDAAkU1Gvm/bZlEnEuEp2oRBzNq1UR8LP WFpSl03Y4oWelfbUxITz1+JKahriCzwgNE2AEtibHOC+VhnD0sOAaqDEeh5zSihSPDx4 o4+zwJbBt549Rz3yylsQWYDL1OpqZc70ii6r/xei2/JiSp4RSO5K0o3hhijvtP8YHmkx 5v3iLX41X3AaCZsXov4EsHWTvau0s+vsSNJkmqt54OCFfbrP3xubeaiit1jvzM5aemKJ WGuw== X-Gm-Message-State: AOAM532MZD4b+ecrmnZ0hSLaBCV6PLs/8ZkmtkWhgiG7rLkUbjD0v/ls 8nPc1GrpBMxzUocvGuNqX06IeSr6xGk= X-Google-Smtp-Source: ABdhPJxaI0tSTGaVwkOHbwGBBm6uP+z67GQDMPLLodAg3nRyLll0XhHHPpVmp+eTz/Z7BkSqNlG6kw== X-Received: by 2002:adf:eb86:0:b0:1e6:8c92:af6b with SMTP id t6-20020adfeb86000000b001e68c92af6bmr13769161wrn.116.1646777732511; Tue, 08 Mar 2022 14:15:32 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r2-20020a5d6942000000b001f0485057c4sm95477wrw.74.2022.03.08.14.15.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:31 -0800 (PST) Message-Id: <3a0f30b849a85ee0f2737a08fcf6a746020efdcf.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:02 +0000 Subject: [PATCH v2 02/27] t7527: test FSMonitor on repos with Unicode root paths Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create some test repos with UTF8 characters in the pathname of the root directory and verify that the builtin FSMonitor can watch them. This test is mainly for Windows where we need to avoid `*A()` routines. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index dbca7f835eb..c2627f28865 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -666,4 +666,27 @@ do done done +# Test Unicode UTF-8 characters in the pathname of the working +# directory root. Use of "*A()" routines rather than "*W()" routines +# on Windows can sometimes lead to odd failures. +# +u1=$(printf "u_c3_a6__\xC3\xA6") +u2=$(printf "u_e2_99_ab__\xE2\x99\xAB") +u_values="$u1 $u2" +for u in $u_values +do + test_expect_success "Unicode in repo root path: $u" ' + test_when_finished "stop_daemon_delete_repo $u" && + + git init "$u" && + echo 1 >"$u"/file1 && + git -C "$u" add file1 && + git -C "$u" config core.fsmonitor true && + + start_daemon "$u" && + git -C "$u" status >actual && + grep "new file: file1" actual + ' +done + test_done From patchwork Tue Mar 8 22:15:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774443 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 42988C433EF for ; Tue, 8 Mar 2022 22:15:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350652AbiCHWQi (ORCPT ); Tue, 8 Mar 2022 17:16:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51308 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350600AbiCHWQc (ORCPT ); Tue, 8 Mar 2022 17:16:32 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 57C005717F for ; Tue, 8 Mar 2022 14:15:35 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id u10so233673wra.9 for ; Tue, 08 Mar 2022 14:15:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=6frvxGvKQ2KGBYzAbNuLBDFBvm1OqC1ZTc7LHozpsok=; b=R5B5WxXZbd8DNfWkD//2EfmFKFSQCpKdnJWRDzFI0kEYKOUs/YOMeeNIRKedset+mb XCelLZWuYcMPxRJduE/d+UK36w6Y5ZvUYdzCKbKkaNP2kFgyYUd1Jg54FnKO3sZGwybL cZlqzybMWkchUNdlN1ckE1ZmB9YzEldjRgIYEELtbCrY0w1iYv2G8SvMS7ttoNZgraBv Ap9M5UULk2Dn27HT7N3N6NVNwtj893EJxRFlO82KA6Kez5TgB5mOurkiuvXITB+Uv8mu DfB03TAaWqoeQYX0yJ5EeDjaAILBC3/6JCcaaCoP1qF6AZP92NL2wiXLNTjhViV0UE3k q9qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=6frvxGvKQ2KGBYzAbNuLBDFBvm1OqC1ZTc7LHozpsok=; b=ONwwtmYgNAkIsMXSQZ2u3ZmRWkSLPaupsfFfkfj4ElV3s9D0//1ZW4/2XnGHYLyudC zi5adwGev5Wnk5YTUwUs87xRzqqY5A13v5SJQKAt+ajhlw6EXfgmp0j0a+S/Iggj8A72 qRjd5QZZfwOHdeG/FKkaTgrT7XnS584Vmv2hzt0ML3ffXnnBdMtykCv59q90423YVjBO fuk0O/mY5D+vXvihAyLI0ZEgW3b80OLI1RK1PPxZIchi4UAS6DS7DnSFaJKp+GiRbvLx m6gJi2neLq7wMT+AjVrVMY9jLKpa1b1iNG+YzAFdsvs/fRmmP7x9itrIyb49c6Gd9Lrx I8hA== X-Gm-Message-State: AOAM530eDA9M9V7mRRd/mj0MkWhY3btrJdF63fMQBkPAe3zGBn0BADJ+ bVufXZmkpbZ3ZbP6J8Dm7eXXkhQvJDM= X-Google-Smtp-Source: ABdhPJwfajwDViLpLd+dWyLOakqjJdhWgBdnLAlt22McYWNzAHbYPDeOf192UzQqyFl4zCv04HmCvg== X-Received: by 2002:a05:6000:170d:b0:1f1:fefa:db85 with SMTP id n13-20020a056000170d00b001f1fefadb85mr7611775wrc.148.1646777733698; Tue, 08 Mar 2022 14:15:33 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e4-20020adffc44000000b001f1dba38a65sm83309wrs.105.2022.03.08.14.15.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:32 -0800 (PST) Message-Id: <87d1c0b6f2afda53bc8a8fc5949af6caca5a0329.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:03 +0000 Subject: [PATCH v2 03/27] t/helper/fsmonitor-client: create stress test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a stress test to hammer on the fsmonitor daemon. Create a client-side thread pool of n threads and have each of them make m requests as fast as they can. We do not currently inspect the contents of the response. We're only interested in placing a heavy request load on the daemon. This test is useful for interactive testing and various experimentation. For example, to place additional load on the daemon while another test is running. We currently do not have a test script that actually uses this helper. We might add such a test in the future. Signed-off-by: Jeff Hostetler --- t/helper/test-fsmonitor-client.c | 106 +++++++++++++++++++++++++++++++ 1 file changed, 106 insertions(+) diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c index f7a5b3a32fa..985340ba719 100644 --- a/t/helper/test-fsmonitor-client.c +++ b/t/helper/test-fsmonitor-client.c @@ -7,6 +7,8 @@ #include "cache.h" #include "parse-options.h" #include "fsmonitor-ipc.h" +#include "thread-utils.h" +#include "trace2.h" #ifndef HAVE_FSMONITOR_DAEMON_BACKEND int cmd__fsmonitor_client(int argc, const char **argv) @@ -79,20 +81,121 @@ static int do_send_flush(void) return 0; } +struct hammer_thread_data +{ + pthread_t pthread_id; + int thread_nr; + + int nr_requests; + const char *token; + + int sum_successful; + int sum_errors; +}; + +static void *hammer_thread_proc(void *_hammer_thread_data) +{ + struct hammer_thread_data *data = _hammer_thread_data; + struct strbuf answer = STRBUF_INIT; + int k; + int ret; + + trace2_thread_start("hammer"); + + for (k = 0; k < data->nr_requests; k++) { + strbuf_reset(&answer); + + ret = fsmonitor_ipc__send_query(data->token, &answer); + if (ret < 0) + data->sum_errors++; + else + data->sum_successful++; + } + + strbuf_release(&answer); + trace2_thread_exit(); + return NULL; +} + +/* + * Start a pool of client threads that will each send a series of + * commands to the daemon. + * + * The goal is to overload the daemon with a sustained series of + * concurrent requests. + */ +static int do_hammer(const char *token, int nr_threads, int nr_requests) +{ + struct hammer_thread_data *data = NULL; + int k; + int sum_join_errors = 0; + int sum_commands = 0; + int sum_errors = 0; + + if (!token || !*token) + token = get_token_from_index(); + if (nr_threads < 1) + nr_threads = 1; + if (nr_requests < 1) + nr_requests = 1; + + CALLOC_ARRAY(data, nr_threads); + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + p->thread_nr = k; + p->nr_requests = nr_requests; + p->token = token; + + if (pthread_create(&p->pthread_id, NULL, hammer_thread_proc, p)) { + warning("failed to create thread[%d] skipping remainder", k); + nr_threads = k; + break; + } + } + + for (k = 0; k < nr_threads; k++) { + struct hammer_thread_data *p = &data[k]; + + if (pthread_join(p->pthread_id, NULL)) + sum_join_errors++; + sum_commands += p->sum_successful; + sum_errors += p->sum_errors; + } + + fprintf(stderr, "HAMMER: [threads %d][requests %d] [ok %d][err %d][join %d]\n", + nr_threads, nr_requests, sum_commands, sum_errors, sum_join_errors); + + free(data); + + /* + * Return an error if any of the _send_query requests failed. + * We don't care about thread create/join errors. + */ + return sum_errors > 0; +} + int cmd__fsmonitor_client(int argc, const char **argv) { const char *subcmd; const char *token = NULL; + int nr_threads = 1; + int nr_requests = 1; const char * const fsmonitor_client_usage[] = { N_("test-helper fsmonitor-client query []"), N_("test-helper fsmonitor-client flush"), + N_("test-helper fsmonitor-client hammer [] [] []"), NULL, }; struct option options[] = { OPT_STRING(0, "token", &token, N_("token"), N_("command token to send to the server")), + + OPT_INTEGER(0, "threads", &nr_threads, N_("number of client threads")), + OPT_INTEGER(0, "requests", &nr_requests, N_("number of requests per thread")), + OPT_END() }; @@ -116,6 +219,9 @@ int cmd__fsmonitor_client(int argc, const char **argv) if (!strcmp(subcmd, "flush")) return !!do_send_flush(); + if (!strcmp(subcmd, "hammer")) + return !!do_hammer(token, nr_threads, nr_requests); + die("Unhandled subcommand: '%s'", subcmd); } #endif From patchwork Tue Mar 8 22:15:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774445 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1BDCC433F5 for ; Tue, 8 Mar 2022 22:15:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350662AbiCHWQm (ORCPT ); Tue, 8 Mar 2022 17:16:42 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51314 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350643AbiCHWQd (ORCPT ); Tue, 8 Mar 2022 17:16:33 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62B6C56220 for ; Tue, 8 Mar 2022 14:15:36 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id h15so249007wrc.6 for ; Tue, 08 Mar 2022 14:15:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tYMNxNOQocQ5cxp9AH3AWwQ4szZPjiifBMV3x37TQAg=; b=Ia/De43UM+wILTZoUl8LBsC3Q20B9/qp2kTJ9c0G0I+8yi+QJ7mLUV741w6pmj/ahQ ByQvpNyf8zRHiy5UatqO/aBqe7zO/Urw/v5pyG21loENY9ayRMZIPGyaNfB/qzrs9JGe m1AN05dDLPCwAVfTOq3imCCVnnaUiCKpYiJWADesVks9B/0DcSWVAkAZ7Hoyb8KgRqxH WQLP1OeBpQynon5SXfzYmN8Yjs439JiLaLdJRB7C/93xldTGhZZRwybkuda6R9GHi6bk tzrfIAX8hisOFvqyyVQW1o7Uqh1Vq6CVpYzcqoqK2Lun4ILghVlAIu+yrusAVmO1Ml7z yADA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tYMNxNOQocQ5cxp9AH3AWwQ4szZPjiifBMV3x37TQAg=; b=BGvahRL0qeigGxtksaTE4YWms6TzdTN7KrBlVuqDGbQUNR0EhKq9tHTTBnh9yz4Zl7 Qfkjhw/8XB6HFu4f0h0IZQWXgbVnDoWETIdbR0k+iyQt1Xiqe/hwfKyssxj+qvuzk93F W7/w1PurUsX+U4xVIPOLGBWLikSCCRL6JvX0tezLjIJj30GjWqdBYVAHAPyC8b69s00k VrBlQghFu3RO8L7SchMW8HQF3tgXVAKFrc70NqKfwMm3SZ31o7PkbGb8+Xgo1jZXZ9De eXRIovj9WQmWxR3PJCRSrocomccFoiRpvELVQ41a0Hg5dXux/kzmFwVX3D88UTbOTD0L ELfw== X-Gm-Message-State: AOAM530rak3S4xtOgeXgKABLKxZHKwcdb8H/PWwK08TLz7wEjxRCvIkE f6pzuCK7UoSYzaoW9JX8+mxqa0Ax3Is= X-Google-Smtp-Source: ABdhPJwhRmKIz39e+uQtdDVwpkvOp7xuB66Wo7QUCcGzVSlibw6DLPgkqH6+x1YBgkvpPFoSzkvD/w== X-Received: by 2002:a05:6000:1204:b0:1f1:e689:4c88 with SMTP id e4-20020a056000120400b001f1e6894c88mr11422530wrx.140.1646777734716; Tue, 08 Mar 2022 14:15:34 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b15-20020adfc74f000000b001e888b871a0sm89780wrh.87.2022.03.08.14.15.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:34 -0800 (PST) Message-Id: <8c4f90ae4fd5d9fbac9acb9307ee82ceffc7df08.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:04 +0000 Subject: [PATCH v2 04/27] fsmonitor-settings: bare repos are incompatible with FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Bare repos do not have a worktree, so there is nothing for the daemon watch. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 9 ++++++ builtin/update-index.c | 7 +++++ fsmonitor-settings.c | 57 +++++++++++++++++++++++++++++++++++++ fsmonitor-settings.h | 12 ++++++++ t/t7519-status-fsmonitor.sh | 23 +++++++++++++++ 5 files changed, 108 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 97ca2a356e5..00eaffbb945 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1443,6 +1443,15 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) die(_("invalid 'ipc-threads' value (%d)"), fsmonitor__ipc_threads); + prepare_repo_settings(the_repository); + fsm_settings__set_ipc(the_repository); + + if (fsm_settings__get_mode(the_repository) == FSMONITOR_MODE_INCOMPATIBLE) { + const char *msg = fsm_settings__get_reason_msg(the_repository); + + return error("%s '%s'", msg ? msg : "???", xgetcwd()); + } + if (!strcmp(subcmd, "start")) return !!try_to_start_background_daemon(); diff --git a/builtin/update-index.c b/builtin/update-index.c index d335f1ac72a..8f460e7195f 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1237,6 +1237,13 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) if (fsmonitor > 0) { enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r); + + if (fsm_mode == FSMONITOR_MODE_INCOMPATIBLE) { + const char *msg = fsm_settings__get_reason_msg(r); + + return error("%s '%s'", msg ? msg : "???", xgetcwd()); + } + if (fsm_mode == FSMONITOR_MODE_DISABLED) { advise(_("core.fsmonitor is unset; " "set it if you really want to " diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 3b54e7a51f6..8410cc73404 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -9,9 +9,33 @@ */ struct fsmonitor_settings { enum fsmonitor_mode mode; + enum fsmonitor_reason reason; char *hook_path; }; +static void set_incompatible(struct repository *r, + enum fsmonitor_reason reason) +{ + struct fsmonitor_settings *s = r->settings.fsmonitor; + + s->mode = FSMONITOR_MODE_INCOMPATIBLE; + s->reason = reason; +} + +static int check_for_incompatible(struct repository *r) +{ + if (!r->worktree) { + /* + * Bare repositories don't have a working directory and + * therefore have nothing to watch. + */ + set_incompatible(r, FSMONITOR_REASON_BARE); + return 1; + } + + return 0; +} + static void lookup_fsmonitor_settings(struct repository *r) { struct fsmonitor_settings *s; @@ -87,6 +111,9 @@ void fsm_settings__set_ipc(struct repository *r) lookup_fsmonitor_settings(r); + if (check_for_incompatible(r)) + return; + r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } @@ -98,6 +125,9 @@ void fsm_settings__set_hook(struct repository *r, const char *path) lookup_fsmonitor_settings(r); + if (check_for_incompatible(r)) + return; + r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK; FREE_AND_NULL(r->settings.fsmonitor->hook_path); r->settings.fsmonitor->hook_path = strdup(path); @@ -111,5 +141,32 @@ void fsm_settings__set_disabled(struct repository *r) lookup_fsmonitor_settings(r); r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED; + r->settings.fsmonitor->reason = FSMONITOR_REASON_OK; FREE_AND_NULL(r->settings.fsmonitor->hook_path); } + +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + return r->settings.fsmonitor->reason; +} + +const char *fsm_settings__get_reason_msg(struct repository *r) +{ + enum fsmonitor_reason reason = fsm_settings__get_reason(r); + + switch (reason) { + case FSMONITOR_REASON_OK: + return NULL; + + case FSMONITOR_REASON_BARE: + return _("bare repos are incompatible with fsmonitor"); + } + + BUG("Unhandled case in fsm_settings__get_reason_msg '%d'", + reason); +} diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index a4c5d7b4889..a1af058a287 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -4,11 +4,20 @@ struct repository; enum fsmonitor_mode { + FSMONITOR_MODE_INCOMPATIBLE = -1, /* see _reason */ FSMONITOR_MODE_DISABLED = 0, FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */ FSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */ }; +/* + * Incompatibility reasons. + */ +enum fsmonitor_reason { + FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ + FSMONITOR_REASON_BARE, +}; + void fsm_settings__set_ipc(struct repository *r); void fsm_settings__set_hook(struct repository *r, const char *path); void fsm_settings__set_disabled(struct repository *r); @@ -16,6 +25,9 @@ void fsm_settings__set_disabled(struct repository *r); enum fsmonitor_mode fsm_settings__get_mode(struct repository *r); const char *fsm_settings__get_hook_path(struct repository *r); +enum fsmonitor_reason fsm_settings__get_reason(struct repository *r); +const char *fsm_settings__get_reason_msg(struct repository *r); + struct fsmonitor_settings; #endif /* FSMONITOR_SETTINGS_H */ diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index a6308acf006..cf258d88f04 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -55,6 +55,29 @@ test_lazy_prereq UNTRACKED_CACHE ' test $ret -ne 1 ' +# Test that we detect and disallow repos that are incompatible with FSMonitor. +test_expect_success 'incompatible bare repo' ' + test_when_finished "rm -rf ./bare-clone actual expect" && + git init --bare bare-clone && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=foo \ + update-index --fsmonitor 2>actual && + grep "bare repos are incompatible with fsmonitor" actual && + + test_must_fail \ + git -C ./bare-clone -c core.fsmonitor=true \ + update-index --fsmonitor 2>actual && + grep "bare repos are incompatible with fsmonitor" actual +' + +test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' + test_when_finished "rm -rf ./bare-clone actual" && + git init --bare bare-clone && + test_must_fail git -C ./bare-clone fsmonitor--daemon run 2>actual && + grep "bare repos are incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Tue Mar 8 22:15:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 128E0C433FE for ; Tue, 8 Mar 2022 22:15:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350664AbiCHWQn (ORCPT ); Tue, 8 Mar 2022 17:16:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51372 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350646AbiCHWQg (ORCPT ); Tue, 8 Mar 2022 17:16:36 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1295154BCE for ; Tue, 8 Mar 2022 14:15:37 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id l1-20020a05600c4f0100b00389645443d2so409070wmq.2 for ; Tue, 08 Mar 2022 14:15:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5A+fsXXhpmpUKMBtcYUi70L1TOFe1J3MTYQocvv9ov0=; b=PKcTTn0Kn17HGDAXH94iuM07YJJtUT+5CjNToX/0VmSNPdUOswCR4+eudKzMe7B1wS fTXgopmobe09/t1HcF6QNTu2hymlG2PZS1lDN3X52qsjOuM6YOznlV/lWUosHcAMD+Ft 22/6eX7T/4daYy+chuqZ4D0h09LPCnFZ3mdCtVIz6t08TSq/VZrE6H9H6EkiBloHek9V WiKDktj7qjzChw532Upeky8egjBlycoZD8/gaq9TjifUYsJoJ1vhI0GoSp8DcinDs92R mwMXOi3b9vH+gaxIrnn+BG1ARvquv1TVo8A8hdTV7t7UlWHDCqkGQ+h9CATC12dJloCw ZX4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5A+fsXXhpmpUKMBtcYUi70L1TOFe1J3MTYQocvv9ov0=; b=TtGavN3nSy8rKT5AlrzaqkIAa5NuIVPZkMrmPx8BblZSdUJfuToqW3wgki9+INvUwY b/23eeSgbuF/VQNfAUm9SP5J4TfQLJhJqvOcpYFGAk/OMQqPiVYf7OwT1j4k6UrGHJxX 95NHauSaPZYIAR8aevQjnUNtFqjLDb3PRhRLtr0RpLdALPmLiqX5Vb9ypuSGx5JPFJcm ghhOP4PZP8MvUCqzMPsOzN8yfmMm/oQJqrQAyONT+WZzMTFSvuEvzSgpBOjtypYM4vRN hAj8gDRg0d95F2X9h+w+cHzmt2UlnAY3nEd5utfP35HDM5um8YJTaTTL3Qisomev72bH KCnQ== X-Gm-Message-State: AOAM530/PgOuhhE1+NNKPwIA0F35S/GMaHlkq3TNJ0wZlFr7fNw4z5Qz D6cnZ/gO1MGhTa4PzbNQ5fbCqTdFOHg= X-Google-Smtp-Source: ABdhPJxxZg9ZyJ/9huhVaL10vjUewah1IFnyy3+8uiNjfF5yN0C9R4iAzju70CPLPTyD84pLaYSA3A== X-Received: by 2002:a05:600c:1990:b0:381:1806:1d9 with SMTP id t16-20020a05600c199000b00381180601d9mr5238618wmq.93.1646777735821; Tue, 08 Mar 2022 14:15:35 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y12-20020adff14c000000b001f04d1959easm101811wro.13.2022.03.08.14.15.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:35 -0800 (PST) Message-Id: <6329328d1859bc360dc6305a1d7be6734c9c84ea.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:05 +0000 Subject: [PATCH v2 05/27] fsmonitor-settings: stub in platform-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend generic incompatibility checkout with platform-specific mechanism. Stub in Win32 version. In the existing fsmonitor-settings code we have a way to mark types of repos as incompatible with fsmonitor (whether via the hook and IPC APIs). For example, we do this for bare repos, since there are no files to watch. Extend this exclusion mechanism for platform-specific reasons. This commit just creates the framework and adds a stub for Win32. Signed-off-by: Jeff Hostetler --- Makefile | 13 +++++++++++++ compat/fsmonitor/fsm-settings-win32.c | 9 +++++++++ config.mak.uname | 4 ++++ contrib/buildsystems/CMakeLists.txt | 3 +++ fsmonitor-settings.c | 12 ++++++++++++ fsmonitor-settings.h | 13 +++++++++++++ 6 files changed, 54 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-win32.c diff --git a/Makefile b/Makefile index daa21bed6c3..93604fe8ef7 100644 --- a/Makefile +++ b/Makefile @@ -475,6 +475,11 @@ all:: # `compat/fsmonitor/fsm-listen-.c` that implements the # `fsm_listen__*()` routines. # +# If your platform has OS-specific ways to tell if a repo is incompatible with +# fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS +# to the "" of the corresponding `compat/fsmonitor/fsm-settings-.c` +# that implements the `fsm_os_settings__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1979,6 +1984,11 @@ ifdef FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o endif +ifdef FSMONITOR_OS_SETTINGS + COMPAT_CFLAGS += -DHAVE_FSMONITOR_OS_SETTINGS + COMPAT_OBJS += compat/fsmonitor/fsm-settings-$(FSMONITOR_OS_SETTINGS).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2901,6 +2911,9 @@ GIT-BUILD-OPTIONS: FORCE ifdef FSMONITOR_DAEMON_BACKEND @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ endif +ifdef FSMONITOR_OS_SETTINGS + @echo FSMONITOR_OS_SETTINGS=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_OS_SETTINGS)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index 501970902da..cf224768ad6 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -450,6 +450,8 @@ ifeq ($(uname_S),Windows) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -639,6 +641,8 @@ ifeq ($(uname_S),MINGW) # These are always available, so we do not have to conditionally # support it. FSMONITOR_DAEMON_BACKEND = win32 + FSMONITOR_OS_SETTINGS = win32 + RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index ee0d7257b77..16705da2000 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 8410cc73404..1b16b551d87 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -33,6 +33,18 @@ static int check_for_incompatible(struct repository *r) return 1; } +#ifdef HAVE_FSMONITOR_OS_SETTINGS + { + enum fsmonitor_reason reason; + + reason = fsm_os__incompatible(r); + if (reason != FSMONITOR_REASON_OK) { + set_incompatible(r, reason); + return 1; + } + } +#endif + return 0; } diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index a1af058a287..be25272c012 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -30,4 +30,17 @@ const char *fsm_settings__get_reason_msg(struct repository *r); struct fsmonitor_settings; +#ifdef HAVE_FSMONITOR_OS_SETTINGS +/* + * Ask platform-specific code whether the repository is incompatible + * with fsmonitor (both hook and ipc modes). For example, if the working + * directory is on a remote volume and mounted via a technology that does + * not support notification events, then we should not pretend to watch it. + * + * fsm_os__* routines should considered private to fsm_settings__ + * routines. + */ +enum fsmonitor_reason fsm_os__incompatible(struct repository *r); +#endif /* HAVE_FSMONITOR_OS_SETTINGS */ + #endif /* FSMONITOR_SETTINGS_H */ From patchwork Tue Mar 8 22:15:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774448 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 213B7C433EF for ; Tue, 8 Mar 2022 22:15:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350667AbiCHWQr (ORCPT ); Tue, 8 Mar 2022 17:16:47 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51374 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350647AbiCHWQg (ORCPT ); Tue, 8 Mar 2022 17:16:36 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 67E7D56220 for ; Tue, 8 Mar 2022 14:15:38 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id b5so275812wrr.2 for ; Tue, 08 Mar 2022 14:15:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=mfnUzrBPZiTqZr1H3mJX0p6D1Lxzn9AnXn9ve4rZMkY=; b=HhI5gut4RGIU7caKM/itonwCuW7CDksSEYh7aO+HDTkWQV/M8voUukDtd/SHZ87qEA VqIOOmhLr7rocI8E4BFEo5ajDfrTUuDKK/5dPHDzCBP9WEN3kKPXKsR4aaBJGsXNP3r7 3gMLxW1R2+kpdzJB37IxKo4GPl8Xnlv8mxb/bIxJt3L9QSj4ljrH+DFo9C2ZA2PlJwsO TsswaO3JQlSBjNX5hC0i7QAIUT8kNYi9MrVj6JRgXlurnx5g9LCQ7UKzSAmRQOeAswPr FhqWhYeuJbQjW+eu2/aI6eO8Uruu4Vgu3Qtgmo16T1v6+G14t/4BSuL77SuRSdKik3ZR r1Xw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=mfnUzrBPZiTqZr1H3mJX0p6D1Lxzn9AnXn9ve4rZMkY=; b=sthDA/KcE2aQX35bLUujhrmuAjQ+YmWX4hQI3U3LsXUZHE+4K6VfXN15hgnK2Rr9Z9 8m2zXJKabwVI6fhkH5CocjQf6B74WKTkN+CSDWU/r6vJCpQT7hJibpZphIm8E5xMFknX zf63pGjKvzaB10cW3/waumtyUj66U8H4nksqt60yeMDH7igRPOYFE32PYOjKjDNcYdo0 8/XeidNjLgvDz9zIc2P1fizA6RRAHLhNvqxgHFEnCwOvn3w64KupdK9BqpuLHXUhVDqh gkGEn63XpkL54ZhrSbYXwl9mwxRmoUq5dbQXBzRvPzONaaAkyi/p4JQ7U7iu3od7y1hq tEIQ== X-Gm-Message-State: AOAM533/vkiFLpdgS80vgqTZ1o4In0rO3Z94oM/wZpfT75P5/S87SvD3 nPT6ahyzP1IEoJNI/kmrTSADxUM0GM8= X-Google-Smtp-Source: ABdhPJyPOiASidmYMcNCqL/FRqCGj9alDXn8MSlRAebiOXcwEC95125Mai9ZhpGTqbFHYqf2DfqNCg== X-Received: by 2002:a5d:648b:0:b0:1f1:fc40:91e0 with SMTP id o11-20020a5d648b000000b001f1fc4091e0mr7532433wri.712.1646777736798; Tue, 08 Mar 2022 14:15:36 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f11-20020a7bcc0b000000b0037e0c362b6dsm61668wmh.31.2022.03.08.14.15.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:36 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:06 +0000 Subject: [PATCH v2 06/27] fsmonitor-settings: VFS for Git virtual repos are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler VFS for Git virtual repositories are incompatible with FSMonitor. VFS for Git is a downstream fork of Git. It contains its own custom file system watcher that is aware of the virtualization. If a working directory is being managed by VFS for Git, we should not try to watch it because we may get incomplete results. We do not know anything about how VFS for Git works, but we do know that VFS for Git working directories contain a well-defined config setting. If it is set, mark the working directory as incompatible. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 26 ++++++++++++++++++++++++++ fsmonitor-settings.c | 3 +++ fsmonitor-settings.h | 1 + t/t7519-status-fsmonitor.sh | 9 +++++++++ 4 files changed, 39 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index 7fce32a3c5b..ee78bba38e3 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -3,7 +3,33 @@ #include "repository.h" #include "fsmonitor-settings.h" +/* + * VFS for Git is incompatible with FSMonitor. + * + * Granted, core Git does not know anything about VFS for Git and we + * shouldn't make assumptions about a downstream feature, but users + * can install both versions. And this can lead to incorrect results + * from core Git commands. So, without bringing in any of the VFS for + * Git code, do a simple config test for a published config setting. + * (We do not look at the various *_TEST_* environment variables.) + */ +static enum fsmonitor_reason check_vfs4git(struct repository *r) +{ + const char *const_str; + + if (!repo_config_get_value(r, "core.virtualfilesystem", &const_str)) + return FSMONITOR_REASON_VFS4GIT; + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_vfs4git(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 1b16b551d87..ea3e365dfc4 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -177,6 +177,9 @@ const char *fsm_settings__get_reason_msg(struct repository *r) case FSMONITOR_REASON_BARE: return _("bare repos are incompatible with fsmonitor"); + + case FSMONITOR_REASON_VFS4GIT: + return _("virtual repos are incompatible with fsmonitor"); } BUG("Unhandled case in fsm_settings__get_reason_msg '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index be25272c012..7950529611b 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -16,6 +16,7 @@ enum fsmonitor_mode { enum fsmonitor_reason { FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; void fsm_settings__set_ipc(struct repository *r); diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index cf258d88f04..285508fb67e 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -78,6 +78,15 @@ test_expect_success FSMONITOR_DAEMON 'run fsmonitor-daemon in bare repo' ' grep "bare repos are incompatible with fsmonitor" actual ' +test_expect_success MINGW,FSMONITOR_DAEMON 'run fsmonitor-daemon in virtual repo' ' + test_when_finished "rm -rf ./fake-virtual-clone actual" && + git init fake-virtual-clone && + test_must_fail git -C ./fake-virtual-clone \ + -c core.virtualfilesystem=true \ + fsmonitor--daemon run 2>actual && + grep "virtual repos are incompatible with fsmonitor" actual +' + test_expect_success 'setup' ' mkdir -p .git/hooks && : >tracked && From patchwork Tue Mar 8 22:15:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774447 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4319EC433EF for ; Tue, 8 Mar 2022 22:15:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350666AbiCHWQp (ORCPT ); Tue, 8 Mar 2022 17:16:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350648AbiCHWQg (ORCPT ); Tue, 8 Mar 2022 17:16:36 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4D44B57485 for ; Tue, 8 Mar 2022 14:15:39 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id k24so244570wrd.7 for ; Tue, 08 Mar 2022 14:15:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=X7T/uH16BNRrhwq1p9PPN+g62AYM7KW/USnueA3FV6MgzlYBG+pTyEN5SKuzJO8T/4 CaBoLbWJuJB06sIs97CxGnXyjknnFDNCaL16i8Ujyi1CZyCMsM1EkOsqq3lkbiBEp41t LDLv5cls07zeuBDPbmR7lv4tCK6xnnyk7r7WHKtorjddyUBTWZ9uycjZmyX6qBqfFRK9 FiXbPUZPesdYyBcQj1eiOkdg4PjnuSfcCAmuhSq3hjzM86Rk7+Mo5XcunHnO86czxs2t rqAuQpwUYyCWxHTz4zUMQY/8rClPuvkQPPdUlQKUwc5GZ9XEv9cxjC3x6W0RzGo2JhHQ kHpg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9iru4da4j21im3N2Gig/H0t8AHc+fLOds26HUAcaQ6U=; b=lOMW9ApCLq+KWLGBg1LVbrmVagoCRlTH9Vc5Orp9hJ3Ig2sYRjIFA3LbbC42cKh6an YHgJTHOAX1w8ejYJWTM7Vvvx8VhkFAFx5KpIJ64Uu89CaIv8UhfKSSIldWkOCjdyQiFA SpgfSDLE6LuHKymZAUbq0E4B9SRAXO/EvYKKK3Tbey0QefktiGj9jzK065iz2E+31sZo +MHFIhzKE+36pW+dXBSfXqovkkthSNVRJ7uhQub8+19wHk1VIxVSTA5+ScDT5ONOx+Mh LbnEIcW/mp/68yMIODcdo64I+K5nMoef8/1pWdQE0kiJ5ZVcHzoGp0fAKHyCTHjq+g9i kKGA== X-Gm-Message-State: AOAM533QiBqWNpK0Ln5xqv7YyEaiHa0PWlTLHyNNlE9emOvf/HsBRbTT C0YTs9UITPQyyBdJNy03Vxc5z9mijgk= X-Google-Smtp-Source: ABdhPJxujSQJfLLUhgLULefnd2D/nuO3yrcwfsA0XeejcD3l9MhYptwoh24K4hfNvF+ggZ0pCiHm5Q== X-Received: by 2002:a05:6000:186d:b0:1e8:49fc:69ce with SMTP id d13-20020a056000186d00b001e849fc69cemr13801476wri.80.1646777737792; Tue, 08 Mar 2022 14:15:37 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g6-20020a5d5406000000b001f049726044sm89942wrv.79.2022.03.08.14.15.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:37 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:07 +0000 Subject: [PATCH v2 07/27] fsmonitor-settings: stub in macOS-specific incompatibility checking Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 9 +++++++++ config.mak.uname | 1 + contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 13 insertions(+) create mode 100644 compat/fsmonitor/fsm-settings-darwin.c diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c new file mode 100644 index 00000000000..7fce32a3c5b --- /dev/null +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -0,0 +1,9 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +enum fsmonitor_reason fsm_os__incompatible(struct repository *r) +{ + return FSMONITOR_REASON_OK; +} diff --git a/config.mak.uname b/config.mak.uname index cf224768ad6..cf911d141f2 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -163,6 +163,7 @@ ifeq ($(uname_S),Darwin) ifndef NO_PTHREADS ifndef NO_UNIX_SOCKETS FSMONITOR_DAEMON_BACKEND = darwin + FSMONITOR_OS_SETTINGS = darwin endif endif diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 16705da2000..b8f9f7a0388 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -295,6 +295,9 @@ if(SUPPORTS_SIMPLE_IPC) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + + add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) endif() endif() From patchwork Tue Mar 8 22:15:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774449 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7018BC433F5 for ; Tue, 8 Mar 2022 22:16:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350696AbiCHWQ7 (ORCPT ); Tue, 8 Mar 2022 17:16:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350651AbiCHWQh (ORCPT ); Tue, 8 Mar 2022 17:16:37 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43EF35748F for ; Tue, 8 Mar 2022 14:15:40 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id r81-20020a1c4454000000b00389d0a5c511so54099wma.5 for ; Tue, 08 Mar 2022 14:15:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=9jAtXhyIq9laZUe81g2r33zWzWajDnpxHDHbIsehyl4=; b=Sxio4bVpe/abD8t0xrix8Hn8VTgqCvDT3WPXBBWG9NA1kHVCNUmu1m3ZIGgV8qKEgk 2W1TtaFSbPQiQ2117X/8J9ivdii/2Ld6dkHO+Qum20FeNZ/EeO4YddTJKtk+RocQa7VR AhxpoIZIciWBaYWrPXsOLITJaiq0ZlAgvTY7M8uep7BmNqmVuWE7PjMQSo5E8/4klh10 xAWXay2zNIy++W5dSewJQn3267JdvnmA0lm7QlAsYtV8hc3PqtiML8UBe1C2G94SgcUd qST9oQzo0S7GPZWLs6+GPED3rQBND4Pf5Ev9edQZX/pEfp13GgWmmnoUi+oIpsOGiIvH J38w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=9jAtXhyIq9laZUe81g2r33zWzWajDnpxHDHbIsehyl4=; b=GqQU7i4Lf9TOeFRJWUWwZk6jnp0PGzApCidOK2LRIWKoYbAiqdaZdHkiPY59XB7t6n E22HEVtTpMpVevOsqbR+CtxcqtQ4+b+hTUG8swCmSlkUiNjtiTvdmV+JjvOAvxoOJY0U 87yvkn4dJO0YuaypPeVP4O/XwXTEyc57ND28v1KmqFYHgnxt6qvS2ChKl2CF5E6nwG2s iPTIV5pNcejFRFnJWboG8uE/G6oSoDzJSLb/aWpg1BCafRtKMyE6x/xH3pKsExz8UJWs b42OFyRL0cZvf513GJdNNGVQe9bkh76iNPu6UhdZEzXTLOIzBTdq/Qu1To2WyWUjBhjT dJnw== X-Gm-Message-State: AOAM5316yVMzJzEQNTd2WNbc8Lf2hMOKIsBYuOWv97F8LUM9+7ZKL4Ak BgpErUdU9/UXT0xpjd0WBajQBt5/I1c= X-Google-Smtp-Source: ABdhPJxamfBvMlA9tMR1ywqpVZOuO247EfpLzWZN/oPgU46XduE8ahFRLq3aMinouIARjuvNQs17jA== X-Received: by 2002:a1c:7409:0:b0:389:9408:b3fa with SMTP id p9-20020a1c7409000000b003899408b3famr1056216wmc.122.1646777738626; Tue, 08 Mar 2022 14:15:38 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v4-20020a5d6784000000b001f1f7b814d7sm94713wru.69.2022.03.08.14.15.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:38 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:08 +0000 Subject: [PATCH v2 08/27] fsmonitor-settings: remote repos on macOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on macOS and mark them as incompatible with FSMonitor. With this, `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, like `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 66 ++++++++++++++++++++++++++ fsmonitor-settings.c | 6 +++ fsmonitor-settings.h | 2 + 3 files changed, 74 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index 7fce32a3c5b..fdd762bf79d 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -2,8 +2,74 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" +#include +#include + +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type (NFS, SAMBA, etc.) dictates whether notification events + * are available at all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * For the builtin FSMonitor, we create the Unix domain socket for the + * IPC in the .git directory. If the working directory is remote, + * then the socket will be created on the remote file system. This + * can fail if the remote file system does not support UDS file types + * (e.g. smbfs to a Windows server) or if the remote kernel does not + * allow a non-local process to bind() the socket. (These problems + * could be fixed by moving the UDS out of the .git directory and to a + * well-known local directory on the client machine, but care should + * be taken to ensure that $HOME is actually local and not a managed + * file share.) + * + * So (for now at least), mark remote working directories as + * incompatible. + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + struct statfs fs; + + if (statfs(r->worktree, &fs) == -1) { + int saved_errno = errno; + trace_printf_key(&trace_fsmonitor, "statfs('%s') failed: %s", + r->worktree, strerror(saved_errno)); + errno = saved_errno; + return FSMONITOR_REASON_ERROR; + } + + trace_printf_key(&trace_fsmonitor, + "statfs('%s') [type 0x%08x][flags 0x%08x] '%s'", + r->worktree, fs.f_type, fs.f_flags, fs.f_fstypename); + + if (!(fs.f_flags & MNT_LOCAL)) + return FSMONITOR_REASON_REMOTE; + + return FSMONITOR_REASON_OK; +} enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { + enum fsmonitor_reason reason; + + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index ea3e365dfc4..7ff3f98964d 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -178,6 +178,12 @@ const char *fsm_settings__get_reason_msg(struct repository *r) case FSMONITOR_REASON_BARE: return _("bare repos are incompatible with fsmonitor"); + case FSMONITOR_REASON_ERROR: + return _("repo incompatible with fsmonitor due to errors"); + + case FSMONITOR_REASON_REMOTE: + return _("remote repos are incompatible with fsmonitor"); + case FSMONITOR_REASON_VFS4GIT: return _("virtual repos are incompatible with fsmonitor"); } diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 7950529611b..6aa9a00379b 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -16,6 +16,8 @@ enum fsmonitor_mode { enum fsmonitor_reason { FSMONITOR_REASON_OK = 0, /* no incompatibility or when disbled */ FSMONITOR_REASON_BARE, + FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ + FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ }; From patchwork Tue Mar 8 22:15:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774451 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0643FC433EF for ; Tue, 8 Mar 2022 22:16:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350698AbiCHWRC (ORCPT ); Tue, 8 Mar 2022 17:17:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350660AbiCHWQl (ORCPT ); Tue, 8 Mar 2022 17:16:41 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6459F5749B for ; Tue, 8 Mar 2022 14:15:41 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id l1-20020a05600c4f0100b00389645443d2so409170wmq.2 for ; Tue, 08 Mar 2022 14:15:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=LFpXr4G90Vyrm+x5T33AGdhCZ/3XXJ7/vXfCp/KMlLlK9pvy+S7bKliFqzFa8ojbi0 44086TM3we5PpB5N6n6vuQz4gqJYCW63/xWnNGFoUcGZ8z73vQRdIcDFUnO1AJt+0EY4 EBmNcIf4ezCh3o6t40Id7/71XneB4YlKDQf5rKaJFwY7nqzsJOri+Ny0f5iBWY//QdNn QPmEs08jN2ygtzuI7Em3sm8RBMPuNdsSHTXUjFS8/kxtGiKaiNfSRtiIv3QS2GEtdMsD zMEWNi4nOpDqEA7y0+81JSrJ2lF5tznN4v71tLExpb1sfsMMQd8y6D8fYE0c6bHxw68Q j7pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Sjfx7SFJhNxZLXokVMmn8yqRtBoQjZzh8JIwTJB3LAY=; b=FVvu4cyFWW/gOkrKdyfN2XkHrOIiw1HlA/SnJXySSFTxtBRgUHF8Kzpe3t86nKGmrc VEpfeYxr6o2quj/zqSgDLbu/74CDRhTBDfFazs08xKy2ICUSrzZeZQ2Mxtt8pWK6zE98 +cjtkdBQqit2JBUy/eFXB6Jt+Dea0J0luTtjs4l4qQaMG0+sFVJ30flLifrP6foyU/dK zJagDbYrKRWizIm7hBnrp9zq1KI8sbYQHizWU2R6ypLtfsM4fvlTTwtYESAk1KyZM2B8 seiQaQgHld+PP9WyNZvPh+qHwDHeEFL0mssun4tQeLXqpDIxE1gFbJdz1kmFxA3OZiz8 jKKw== X-Gm-Message-State: AOAM533FT9Z3huvXVCjypzJYA2xqJatfCfOoKJISh4GekvcRhb0jC4bH TvEzqDWZuIwE0lcHsZPf5A9o7fPZxuo= X-Google-Smtp-Source: ABdhPJxbedPAW0rSfyj5StRBowJm8BAcN3397KsVBu3iWHimNNdDHE5IoF/PRUgwLSAOqPpnGlH3bg== X-Received: by 2002:a1c:acc6:0:b0:380:e35f:ff1f with SMTP id v189-20020a1cacc6000000b00380e35fff1fmr5331355wme.52.1646777739771; Tue, 08 Mar 2022 14:15:39 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m13-20020a05600c4f4d00b00386744cb31bsm3551412wmq.35.2022.03.08.14.15.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:39 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:09 +0000 Subject: [PATCH v2 09/27] fsmonitor-settings: remote repos on Windows are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to detect remote working directories on Windows and mark them as incompatible with FSMonitor. With this `git fsmonitor--daemon run` will error out with a message like it does for bare repos. Client commands, such as `git status`, will not attempt to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-win32.c | 102 ++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) diff --git a/compat/fsmonitor/fsm-settings-win32.c b/compat/fsmonitor/fsm-settings-win32.c index ee78bba38e3..907655720bb 100644 --- a/compat/fsmonitor/fsm-settings-win32.c +++ b/compat/fsmonitor/fsm-settings-win32.c @@ -2,6 +2,7 @@ #include "config.h" #include "repository.h" #include "fsmonitor-settings.h" +#include "fsmonitor.h" /* * VFS for Git is incompatible with FSMonitor. @@ -23,6 +24,103 @@ static enum fsmonitor_reason check_vfs4git(struct repository *r) return FSMONITOR_REASON_OK; } +/* + * Remote working directories are problematic for FSMonitor. + * + * The underlying file system on the server machine and/or the remote + * mount type dictates whether notification events are available at + * all to remote client machines. + * + * Kernel differences between the server and client machines also + * dictate the how (buffering, frequency, de-dup) the events are + * delivered to client machine processes. + * + * A client machine (such as a laptop) may choose to suspend/resume + * and it is unclear (without lots of testing) whether the watcher can + * resync after a resume. We might be able to treat this as a normal + * "events were dropped by the kernel" event and do our normal "flush + * and resync" --or-- we might need to close the existing (zombie?) + * notification fd and create a new one. + * + * In theory, the above issues need to be addressed whether we are + * using the Hook or IPC API. + * + * So (for now at least), mark remote working directories as + * incompatible. + * + * Notes for testing: + * + * (a) Windows allows a network share to be mapped to a drive letter. + * (This is the normal method to access it.) + * + * $ NET USE Z: \\server\share + * $ git -C Z:/repo status + * + * (b) Windows allows a network share to be referenced WITHOUT mapping + * it to drive letter. + * + * $ NET USE \\server\share\dir + * $ git -C //server/share/repo status + * + * (c) Windows allows "SUBST" to create a fake drive mapping to an + * arbitrary path (which may be remote) + * + * $ SUBST Q: Z:\repo + * $ git -C Q:/ status + * + * (d) Windows allows a directory symlink to be created on a local + * file system that points to a remote repo. + * + * $ mklink /d ./link //server/share/repo + * $ git -C ./link status + */ +static enum fsmonitor_reason check_remote(struct repository *r) +{ + wchar_t wpath[MAX_PATH]; + wchar_t wfullpath[MAX_PATH]; + size_t wlen; + UINT driveType; + + /* + * Do everything in wide chars because the drive letter might be + * a multi-byte sequence. See win32_has_dos_drive_prefix(). + */ + if (xutftowcs_path(wpath, r->worktree) < 0) + return FSMONITOR_REASON_ERROR; + + /* + * GetDriveTypeW() requires a final slash. We assume that the + * worktree pathname points to an actual directory. + */ + wlen = wcslen(wpath); + if (wpath[wlen - 1] != L'\\' && wpath[wlen - 1] != L'/') { + wpath[wlen++] = L'\\'; + wpath[wlen] = 0; + } + + /* + * Normalize the path. If nothing else, this converts forward + * slashes to backslashes. This is essential to get GetDriveTypeW() + * correctly handle some UNC "\\server\share\..." paths. + */ + if (!GetFullPathNameW(wpath, MAX_PATH, wfullpath, NULL)) + return FSMONITOR_REASON_ERROR; + + driveType = GetDriveTypeW(wfullpath); + trace_printf_key(&trace_fsmonitor, + "DriveType '%s' L'%ls' (%u)", + r->worktree, wfullpath, driveType); + + if (driveType == DRIVE_REMOTE) { + trace_printf_key(&trace_fsmonitor, + "check_remote('%s') true", + r->worktree); + return FSMONITOR_REASON_REMOTE; + } + + return FSMONITOR_REASON_OK; +} + enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; @@ -31,5 +129,9 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) if (reason != FSMONITOR_REASON_OK) return reason; + reason = check_remote(r); + if (reason != FSMONITOR_REASON_OK) + return reason; + return FSMONITOR_REASON_OK; } From patchwork Tue Mar 8 22:15:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774453 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A17BC433EF for ; Tue, 8 Mar 2022 22:16:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350657AbiCHWRH (ORCPT ); Tue, 8 Mar 2022 17:17:07 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350659AbiCHWQl (ORCPT ); Tue, 8 Mar 2022 17:16:41 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CC405749F for ; Tue, 8 Mar 2022 14:15:42 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id j17so318712wrc.0 for ; Tue, 08 Mar 2022 14:15:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=PpmqAXj8rAPFRs45YHRX5SR7b1knHUKXRMphy4bIx96lhWS978Ofrg9A5eDtTBaSOo fE5UJyvNzADGxDPYisMVKnsQx/qov+VUpzLbwhYMcdqkhkmS1HT9TeBHPsbBNH2PFH1U mG4pc1vOo1RbwElre2zVmjUoUz3e2ZFNYCIX8dWFCo5PEpTbyjSbKiNAHnv7xKZbGrdp duzkqD56+4ohmazZvQpOetUHnsP3XFG/VnNHcOKq3KIhlLE4KcYCMD4DxsdPtr1UIpY2 FltXM2SXqBY65Ru+jqRDq7MD700PpJNpCOuASMu4hdKZPiXw87vc7sOwk9bcBGhwSQoe RtUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JW5/RT2bZpq4NBqwKIrDN6AyPlqs0iGhc4xOjJM6ksI=; b=ZcNk9ZBvArcYhNbJkegCz6nrEdyAvSJuu0fFRs1J2x6vYvLgm0mKhuJl89yQpbcfRW mIp3V4GIqkJu4cz1xkabk4P5og+unVCzzo81vEJyp/FIsrYWXEZDZRRsVTGJeMIpsSDb ea2PpZplm07hYyqd19F+GyRR+oef7A1GJksbzmRAcK6koScW2CA4rnpS1I6qlm8NhxFi 6a2k2GMoHl7YD0rV+/0FrOi0Mwy5odlAl4Up/Lkb0v7m25JFSkcecIRXQHvBCqSn7Wtz ulkL4d3aZpRw5efVt8eubvCoB+26ekTZOlOO67E1aVhpdyopArNOaSydhs1f8Qgh3bEE CMdg== X-Gm-Message-State: AOAM531TN5+eGEPcg0I0q7AMJnttTwu7aPsEEf7gysMAzSA32KxxRFNi 3PoDvpf/8fWNYdQfaNvRYtIJeqenLOc= X-Google-Smtp-Source: ABdhPJzF/uar0L+ZdFrCbwCNz/3bcJIOd9LI/YCrWcQqySYiwVdFEo0UGYl7uqP2eDnDXsONsnZK5w== X-Received: by 2002:adf:e0ce:0:b0:1ef:706d:d6b9 with SMTP id m14-20020adfe0ce000000b001ef706dd6b9mr13782564wri.71.1646777740800; Tue, 08 Mar 2022 14:15:40 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v15-20020a5d590f000000b001f04b96f9a9sm96379wrd.42.2022.03.08.14.15.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:40 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:10 +0000 Subject: [PATCH v2 10/27] unpack-trees: initialize fsmonitor_has_run_once in o->result Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Initialize `o->result.fsmonitor_has_run_once` based upon value in `o->src_index->fsmonitor_has_run_once` to prevent a second fsmonitor query during the tree traversal and possibly getting a skewed view of the working directory. The checkout code has already talked to the fsmonitor and the traversal is updating the index as it traverses, so there is no need to query the fsmonitor. Signed-off-by: Jeff Hostetler --- unpack-trees.c | 1 + 1 file changed, 1 insertion(+) diff --git a/unpack-trees.c b/unpack-trees.c index 360844bda3a..888cff81f9c 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1772,6 +1772,7 @@ int unpack_trees(unsigned len, struct tree_desc *t, struct unpack_trees_options o->result.fsmonitor_last_update = xstrdup_or_null(o->src_index->fsmonitor_last_update); + o->result.fsmonitor_has_run_once = o->src_index->fsmonitor_has_run_once; /* * Sparse checkout loop #1: set NEW_SKIP_WORKTREE on existing entries From patchwork Tue Mar 8 22:15:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774450 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1A41C433EF for ; Tue, 8 Mar 2022 22:16:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350684AbiCHWRB (ORCPT ); Tue, 8 Mar 2022 17:17:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52076 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350647AbiCHWQ5 (ORCPT ); Tue, 8 Mar 2022 17:16:57 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4A4D3574A3 for ; Tue, 8 Mar 2022 14:15:43 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id u10so234043wra.9 for ; Tue, 08 Mar 2022 14:15:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MqGV7+a23hsjgWwJRoNvuI9BBxP97TPrXX3hxK6Gjec=; b=Kk7YOmdyh2PQA09KJleubyH3nMfPqHIOUCn3ZlqB0QQsxI6EsfBAYgo0qjG9vuZR8E BHFfUXU2FEYqIhBIeCttdDtVNiqm55fKQ9DvJqCGVnn2EUmJCXO9G9hiezU+44Xkkq6l EqCfLEeCT+Hcu2Mbt/MvUN9xjBQObehzKdR6bzBZVytqbnyAjvnEkpferkA0F0qlp/PT 3aXpk1V0GTrYF+EuSO/ndb2Ay2Rl23HsAlnpJGjdjHNO60HiCWQPOmtlRYpQIi1CFWgU IOEZ4ojbSafQU/RBeAMqHuhCoV8IM48P22+eZYSm9AuPQZdyZ+1F3CcKE2euLJ1mG/zM 2UZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MqGV7+a23hsjgWwJRoNvuI9BBxP97TPrXX3hxK6Gjec=; b=FSa4LR/Q8lmBimZEHdZxmUr5D09ec/STFa5wn6jIncODtraLCecZiyNAxEKQ3ooKbY 8xM3tsUp+PACRLZepuaWwlikAek+sDy80uqaD+agucvut6CuklDwuJBjnAzJ2/BdwFWa aIwLJoQcnUkXQ9mBYm2sw9zp7oHJTpitAfNq5eoIQRtPvS36L6OHXpzYbT/waEJIoCzN mjweWeiF6sLrO4Uew642GeQTRuf/lba66Kvtk1e+WVg6VpBnv5M+zyS0QQqxCOMl/wQg 70kWhRsoSuwo1BjKjFjeLkbPmLeJROFR0i+Wxg1qx4+3tp63hyiQlOW1DtsNET73Az+B SbqA== X-Gm-Message-State: AOAM533btTjQgHNeYWLZLWCRWft0alid7BqI4lXlXpcks6MuU0fpwj6p KDv+11Tavm1aRsi2O+ppsfnI3LrpZ8c= X-Google-Smtp-Source: ABdhPJwvC7WGyYGETV5aGsupJw3+3dRUoUsSpq9q54U0AMWQA8o0h3N8ZD69CyOCcnmq8yNoNNF7aA== X-Received: by 2002:a5d:4dc4:0:b0:1ef:7aa9:5a8e with SMTP id f4-20020a5d4dc4000000b001ef7aa95a8emr13770221wru.168.1646777741694; Tue, 08 Mar 2022 14:15:41 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w7-20020a1cf607000000b00389a5390180sm65009wmc.25.2022.03.08.14.15.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:41 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:11 +0000 Subject: [PATCH v2 11/27] fsm-listen-darwin: ignore FSEvents caused by xattr changes on macOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Ignore FSEvents resulting from `xattr` changes. Git does not care about xattr's or changes to xattr's, so don't waste time collecting these events in the daemon nor transmitting them to clients. Various security tools add xattrs to files and/or directories, such as to mark them as having been downloaded. We should ignore these events since it doesn't affect the content of the file/directory or the normal meta-data that Git cares about. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 34 +++++++++++++++++++++++++++- 1 file changed, 33 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 5c5de1ae702..f9b61b7b1c0 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -172,7 +172,7 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) if (flag & kFSEventStreamEventFlagItemCloned) strbuf_addstr(&msg, "ItemCloned|"); - trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=0x%x %s", path, flag, msg.buf); strbuf_release(&msg); @@ -197,6 +197,31 @@ static int ef_is_dropped(const FSEventStreamEventFlags ef) ef & kFSEventStreamEventFlagUserDropped); } +/* + * If an `xattr` change is the only reason we received this event, + * then silently ignore it. Git doesn't care about xattr's. We + * have to be careful here because the kernel can combine multiple + * events for a single path. And because events always have certain + * bits set, such as `ItemIsFile` or `ItemIsDir`. + * + * Return 1 if we should ignore it. + */ +static int ef_ignore_xattr(const FSEventStreamEventFlags ef) +{ + static const FSEventStreamEventFlags mask = + kFSEventStreamEventFlagItemChangeOwner | + kFSEventStreamEventFlagItemCreated | + kFSEventStreamEventFlagItemFinderInfoMod | + kFSEventStreamEventFlagItemInodeMetaMod | + kFSEventStreamEventFlagItemModified | + kFSEventStreamEventFlagItemRemoved | + kFSEventStreamEventFlagItemRenamed | + kFSEventStreamEventFlagItemXattrMod | + kFSEventStreamEventFlagItemCloned; + + return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); +} + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -262,6 +287,13 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_ignore_xattr(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "ignore-xattr: '%s', flags=0x%x", + path_k, event_flags[k]); + continue; + } + switch (fsmonitor_classify_path_absolute(state, path_k)) { case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: From patchwork Tue Mar 8 22:15:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774452 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B9FCC433F5 for ; Tue, 8 Mar 2022 22:16:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350392AbiCHWRF (ORCPT ); Tue, 8 Mar 2022 17:17:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350668AbiCHWQ5 (ORCPT ); Tue, 8 Mar 2022 17:16:57 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 753A6574A6 for ; Tue, 8 Mar 2022 14:15:44 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id j26so282366wrb.1 for ; Tue, 08 Mar 2022 14:15:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EcFYnwkzP4+zTTHaSF78Zw6De5rCM8prxq8TifZp5Yg=; b=bIH90xI2eKB+1JhsHVV6y8Fm9Uzqnr6mfHJV/ao1/istAwsj7LW9YmNKNK2V/GjGeb R4KbZwjcmkTr2YpWhnx3/vleGi8JEyl4tIIJJ9LJaCPhgrhTgrtSmIilqR1ihDyQBeYG bo7+fB4LTA4UDTJBIiqq3cE/iGTQZ/CI1OvyItx5UtowHZmG3CVXI0LTFV3bFPN2GzKJ blQmuG+qU61smvT9cmB6MJ0ods3/ii6jIun9Dcy9YBp9cPjL3Xn2o1KlOOHUp/Nrx3ST BBxtbIe9hAYbIN+RlVKYcSI3lb+1TKNPRIoyCHh5sTGh4K9skMKRG+nyfgR29l+K5KFb 9xFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EcFYnwkzP4+zTTHaSF78Zw6De5rCM8prxq8TifZp5Yg=; b=6W3yCNnwtwzA1EAP3aFv4QvzPiW4ScHuleN+7kuN8nMCPrutA2GvkZrPnx9hcifVK/ /nKgAH5zHQ93hYiiKd08ByyCDxxIQTRwefCd7+A9W5JMQGhKOpBQCuvqQkM5teRYecyS +OH7+/KXtzx5jqcZSVVWTEuottVR0kooWiLOSxz69EVHuyLz8Q0jFOX/zPOyYLTr9/gb Bgtqvq01i5ORX+b4+KRu1Ib6Ws0PmlfGKBCHagYx6sFfcJE1uMt8H3eNuqUkgEyjQlem z4Uaq4aQHQTn2quIdvIxqIvUCGLTgyO+q1ywooAqSG5k7yUbV/jl2aVTz3kGTfdnsE3/ cpeA== X-Gm-Message-State: AOAM532JYrvZHXJx5ZufWl15wflPFXMUiJiwv01sbitq7MJlKSOTeyr2 8uV0xvbDpTV4gcGPm6BGXX+86bKoJR0= X-Google-Smtp-Source: ABdhPJzf0rgpnU7rRvkxtJ7em7+3kce7frboh2WhqS0WcYd5RHD6xJAQY6d4Y1k0wb0Df0zgp51eFg== X-Received: by 2002:adf:f348:0:b0:1f0:6849:2b7d with SMTP id e8-20020adff348000000b001f068492b7dmr13606163wrp.14.1646777742836; Tue, 08 Mar 2022 14:15:42 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r1-20020a5d4941000000b001ed89dcacbbsm104006wrs.23.2022.03.08.14.15.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:42 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:12 +0000 Subject: [PATCH v2 12/27] fsmonitor--daemon: cd out of worktree root Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the fsmonitor--daemon to CD outside of the worktree before starting up. The common Git startup mechanism causes the CWD of the daemon process to be in the root of the worktree. On Windows, this causes the daemon process to hold a locked handle on the CWD and prevents other processes from moving or deleting the worktree while the daemon is running. CD to HOME before entering main event loops. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 32 +++++++++++++++++++++++++++-- compat/fsmonitor/fsm-listen-win32.c | 22 ++++++++++++++------ fsmonitor--daemon.h | 1 + 3 files changed, 47 insertions(+), 8 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 00eaffbb945..dbd37a2b3b8 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1175,11 +1175,11 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * before we need it. */ if (ipc_server_run_async(&state->ipc_server_data, - fsmonitor_ipc__get_path(), &ipc_opts, + state->path_ipc.buf, &ipc_opts, handle_client, state)) return error_errno( _("could not start IPC thread pool on '%s'"), - fsmonitor_ipc__get_path()); + state->path_ipc.buf); /* * Start the fsmonitor listener thread to collect filesystem @@ -1214,6 +1214,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) static int fsmonitor_run_daemon(void) { struct fsmonitor_daemon_state state; + const char *home; int err; memset(&state, 0, sizeof(state)); @@ -1283,6 +1284,15 @@ static int fsmonitor_run_daemon(void) strbuf_addch(&state.path_cookie_prefix, '/'); + /* + * We create a named-pipe or unix domain socket inside of the + * ".git" directory. (Well, on Windows, we base our named + * pipe in the NPFS on the absolute path of the git + * directory.) + */ + strbuf_init(&state.path_ipc, 0); + strbuf_addstr(&state.path_ipc, absolute_path(fsmonitor_ipc__get_path())); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1292,6 +1302,23 @@ static int fsmonitor_run_daemon(void) goto done; } + /* + * CD out of the worktree root directory. + * + * The common Git startup mechanism causes our CWD to be the + * root of the worktree. On Windows, this causes our process + * to hold a locked handle on the CWD. This prevents the + * worktree from being moved or deleted while the daemon is + * running. + * + * We assume that our FS and IPC listener threads have either + * opened all of the handles that they need or will do + * everything using absolute paths. + */ + home = getenv("HOME"); + if (home && *home && chdir(home)) + die_errno("could not cd home '%s'", home); + err = fsmonitor_run_daemon_1(&state); done: @@ -1304,6 +1331,7 @@ done: strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); strbuf_release(&state.path_cookie_prefix); + strbuf_release(&state.path_ipc); return err; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index f4673d7d8b1..948f0e63915 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -402,12 +402,22 @@ static int recv_rdcw_watch(struct one_watch *watch) } /* - * NEEDSWORK: If an external is deleted, the above - * returns an error. I'm not sure that there's anything that - * we can do here other than failing -- the /.git - * link file would be broken anyway. We might try to check - * for that and return a better error message, but I'm not - * sure it is worth it. + * GetOverlappedResult() fails if the watched directory is + * deleted while we were waiting for an overlapped IO to + * complete. The documentation did not list specific errors, + * but I observed ERROR_ACCESS_DENIED (0x05) errors during + * testing. + * + * Note that we only get notificaiton events for events + * *within* the directory, not *on* the directory itself. + * (These might be properies of the parent directory, for + * example). + * + * NEEDSWORK: We might try to check for the deleted directory + * case and return a better error message, but I'm not sure it + * is worth it. + * + * Shutdown if we get any error. */ error("GetOverlappedResult failed on '%s' [GLE %ld]", diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index bd09fffc176..223c2131b58 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -54,6 +54,7 @@ struct fsmonitor_daemon_state { struct fsmonitor_daemon_backend_data *backend_data; struct ipc_server_data *ipc_server_data; + struct strbuf path_ipc; }; /* From patchwork Tue Mar 8 22:15:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774464 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 350B1C433F5 for ; Tue, 8 Mar 2022 22:16:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350699AbiCHWRa (ORCPT ); Tue, 8 Mar 2022 17:17:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52084 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350675AbiCHWQ5 (ORCPT ); Tue, 8 Mar 2022 17:16:57 -0500 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5A55A574B0 for ; Tue, 8 Mar 2022 14:15:45 -0800 (PST) Received: by mail-wm1-x334.google.com with SMTP id r65so317305wma.2 for ; Tue, 08 Mar 2022 14:15:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=nSWrqMziCbYTA8ezg6FoHNOt/kcUNK4llsbMjZRGMhw=; b=dWVrE1Z5pYAu6GDMVhlTIbd54nRXvcp6CX2gXSI0jC9GpiuVjgxv6pzFOeakWIv7t4 nYmgqwNVt8HujuTijPkqmDaIW9NyqcFYxkLiy4fNCEXLK0M21e1BmnhzpRIONcP9LCKS ch8/vGdhjy7o+rzjvaQPMX85/Tf3L8Jnry+KzWFMrtjUgHbcU9kmcY8YCyy6+zkBP0DO 2+H12wDQE369Q6cQ9Z5tnoJ+ILxEPj377NEvP/x9BBj7RjhYEXX0CKorPeWZ+BWEFM93 F0sL7AEObqWtvLa2N3L6r3ZA0nC48WX/KGHCFJdDFB1cSIwYy9pABfR7EEqOxz1lvXE4 SqUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=nSWrqMziCbYTA8ezg6FoHNOt/kcUNK4llsbMjZRGMhw=; b=Zla6FQsvMBFOLgJDGn3PWjR5dGKIIyy6/mJPhpuLjkOqFeJmdhY9I/qh2rwYKzy6t8 vXi8JJOu8jFv0IiYHV7R40KFUb6fYfkK3u/+BZUwNPDgCpJw0nRBTrmAXdYln2ZBZhhW X7OpPkAtsiGIfTeEF0lrAyJR+BV2KQikn3EIeFlXHr4pAWVmnIKb/skOjR8seARDfZhF WMZRcGGFNFjnA+vKOXJ35lhHWz8qUYqvkpXGeakeLan7pTP5zDHweo9ke2k3dab3N1at UP7Bpt65uA3fWBcCEfyrQeymi/ZliQUoqg2DzA0LNoOCFNQyggz3SIvMpcoYTyn/Cq0t lHfA== X-Gm-Message-State: AOAM532b6MzHfHDXfHGPCPcQpLI3/DvkHkztSy1xcXyHgUolSU6W6ToD 64A1t8nwpH6h/CeZPe0dFhBMa8qExgw= X-Google-Smtp-Source: ABdhPJw8DPVMPyLaZRboDUXTH+jV9n/ZpH2NG+CEefcJQMl0gc5yM+1UTDPuILsiiinYsXZ/Kf221g== X-Received: by 2002:a05:600c:4f09:b0:389:cf43:eaf8 with SMTP id l9-20020a05600c4f0900b00389cf43eaf8mr1035315wmq.201.1646777743767; Tue, 08 Mar 2022 14:15:43 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g2-20020a5d46c2000000b001f079ba0158sm91500wrs.60.2022.03.08.14.15.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:43 -0800 (PST) Message-Id: <71673be2da5e87163ec95f5d99f0a22b698276e6.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:13 +0000 Subject: [PATCH v2 13/27] fsmonitor--daemon: prepare for adding health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Refactor daemon thread startup to make it easier to start a third thread class to monitor the health of the daemon. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index dbd37a2b3b8..d5a8fc2ddc2 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1168,6 +1168,8 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int listener_started = 0; + int err = 0; /* * Start the IPC thread pool before the we've started the file @@ -1188,15 +1190,20 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) if (pthread_create(&state->listener_thread, NULL, fsm_listen__thread_proc, state) < 0) { ipc_server_stop_async(state->ipc_server_data); - ipc_server_await(state->ipc_server_data); - - return error(_("could not start fsmonitor listener thread")); + err = error(_("could not start fsmonitor listener thread")); + goto cleanup; } + listener_started = 1; /* * The daemon is now fully functional in background threads. + * Our primary thread should now just wait while the threads + * do all the work. + */ +cleanup: + /* * Wait for the IPC thread pool to shutdown (whether by client - * request or from filesystem activity). + * request, from filesystem activity, or an error). */ ipc_server_await(state->ipc_server_data); @@ -1205,10 +1212,16 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) * event from the IPC thread pool, but it doesn't hurt to tell * it again. And wait for it to shutdown. */ - fsm_listen__stop_async(state); - pthread_join(state->listener_thread, NULL); + if (listener_started) { + fsm_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + } - return state->error_code; + if (err) + return err; + if (state->error_code) + return state->error_code; + return 0; } static int fsmonitor_run_daemon(void) From patchwork Tue Mar 8 22:15:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774454 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6C96C433FE for ; Tue, 8 Mar 2022 22:16:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1344983AbiCHWRK (ORCPT ); Tue, 8 Mar 2022 17:17:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350677AbiCHWQ6 (ORCPT ); Tue, 8 Mar 2022 17:16:58 -0500 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C520C57B00 for ; Tue, 8 Mar 2022 14:15:46 -0800 (PST) Received: by mail-wm1-x336.google.com with SMTP id r65so317324wma.2 for ; Tue, 08 Mar 2022 14:15:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=oR97VX81YJBvKrgvIt5LNRhlScrjAiYa/2tKkCcD+ig=; b=BU2z4B/vnOuoxJmUYL8Tiwxec5/TTSuI9LILv+QOOdHyHC0F7dzPcsdyVFtutAsNS/ +oZgo2QEZx3yguckZ5CcNT0cZiyK++jjIB44nXkCFcz/zCupXRyGBwNLecdSc/rckvkg aAd1awDG1yZp8UFu1djsOPrg5Hzk0qmjn/DwgB+ZSC9e4XFhZgsml1xDe8iqxo/kPtA7 0iEMhry9LJaAQ+lxyLKfwYmuursr9lg5FGnrNio8CkLEFUuhJMyTmCgoh15u/8mlTAxV i6Kl5vHl/2ktY/2umj4bFrj3MhbgpLXMsxhEmQ3tSGkp5TAmY3rXRmRU8JTHFZVIN3zR FSnA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=oR97VX81YJBvKrgvIt5LNRhlScrjAiYa/2tKkCcD+ig=; b=6O29pvXdL/nuTkXlnBoxXlwGhUmFW3AdsM6ubQz07EsHxiRktHknFyQVRrYprpJu1A Fv6dQUoqhnPYmfdrTxWlfiBDoVlvP2B3sj48Rt3hCKs57KFlFz+Bls3VEswoIcW+wQA/ kyajDUh7Ozm1CTtHVBBfjfJ3nVfQB0IjLN7hZBOpnnKyz+PVpm9nxZrJsNtfqcnXFZM6 jS7++VUkhgeSljjj0giLG/OgCs9fBvVepYFpZqWbVyRfnZFwttAWrJNuKlj5mA5AnmJa zhIStcGpOzYnehcotuJuS2IcRaO1GnbukL2vgYl7C1RohlVEH/SWc144bAglX6JobeJ1 KWjA== X-Gm-Message-State: AOAM530ALu3SKWJTg6AP41EeYytyDwVi/WbvS6138AQrjlyYG8ZwBx++ DNrF5alzMK+Ph60xCyQXxe4Tpm143+o= X-Google-Smtp-Source: ABdhPJybwCpVnbdlAPofbxjRgw8XhfGT2iyN2PeWG/bHyB/ys/DZ87SE2yy+3X+nuod701ygKvNxLg== X-Received: by 2002:a05:600c:154e:b0:381:81d1:8e78 with SMTP id f14-20020a05600c154e00b0038181d18e78mr5394383wmg.51.1646777745114; Tue, 08 Mar 2022 14:15:45 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t14-20020a5d49ce000000b001f036a29f42sm75611wrs.116.2022.03.08.14.15.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:44 -0800 (PST) Message-Id: <5387baaf5d78244ff079bdd92b99604d33e957bc.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:14 +0000 Subject: [PATCH v2 14/27] fsmonitor--daemon: rename listener thread related variables Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Rename platform-specific listener thread related variables and data types as we prepare to add another backend thread type. [] `struct fsmonitor_daemon_backend_data` becomes `struct fsm_listen_data` [] `state->backend_data` becomes `state->listen_data` [] `state->error_code` becomes `state->listen_error_code` Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 6 +++--- compat/fsmonitor/fsm-listen-darwin.c | 30 ++++++++++++++-------------- compat/fsmonitor/fsm-listen-win32.c | 28 +++++++++++++------------- compat/fsmonitor/fsm-listen.h | 2 +- fsmonitor--daemon.h | 6 +++--- 5 files changed, 36 insertions(+), 36 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index d5a8fc2ddc2..92c61d5b94d 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -1219,8 +1219,8 @@ cleanup: if (err) return err; - if (state->error_code) - return state->error_code; + if (state->listen_error_code) + return state->listen_error_code; return 0; } @@ -1235,7 +1235,7 @@ static int fsmonitor_run_daemon(void) hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); - state.error_code = 0; + state.listen_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index f9b61b7b1c0..7c81bc7e5bf 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -99,7 +99,7 @@ void FSEventStreamRelease(FSEventStreamRef stream); #include "fsm-listen.h" #include "fsmonitor--daemon.h" -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { CFStringRef cfsr_worktree_path; CFStringRef cfsr_gitdir_path; @@ -230,7 +230,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, const FSEventStreamEventId event_ids[]) { struct fsmonitor_daemon_state *state = ctx; - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; char **paths = (char **)event_paths; struct fsmonitor_batch *batch = NULL; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -422,11 +422,11 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) NULL, NULL }; - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; const void *dir_array[2]; CALLOC_ARRAY(data, 1); - state->backend_data = data; + state->listen_data = data; data->cfsr_worktree_path = CFStringCreateWithCString( NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); @@ -458,18 +458,18 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) failed: error("Unable to create FSEventStream."); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); return -1; } void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; if (data->stream) { if (data->stream_started) @@ -479,14 +479,14 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state) FSEventStreamRelease(data->stream); } - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->shutdown_style = SHUTDOWN_EVENT; CFRunLoopStop(data->rl); @@ -494,9 +494,9 @@ void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - data = state->backend_data; + data = state->listen_data; data->rl = CFRunLoopGetCurrent(); @@ -513,7 +513,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) switch (data->shutdown_style) { case FORCE_ERROR_STOP: - state->error_code = -1; + state->listen_error_code = -1; /* fall thru */ case FORCE_SHUTDOWN: ipc_server_stop_async(state->ipc_server_data); @@ -525,7 +525,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) return; force_error_stop_without_loop: - state->error_code = -1; + state->listen_error_code = -1; ipc_server_stop_async(state->ipc_server_data); return; } diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 948f0e63915..a7c5d263940 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -54,7 +54,7 @@ struct one_watch wchar_t dotgit_shortname[16]; /* for 8.3 name */ }; -struct fsmonitor_daemon_backend_data +struct fsm_listen_data { struct one_watch *watch_worktree; struct one_watch *watch_gitdir; @@ -263,7 +263,7 @@ static enum get_relative_result get_relative_longname( void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { - SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); + SetEvent(state->listen_data->hListener[LISTENER_SHUTDOWN]); } static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, @@ -337,7 +337,7 @@ static void destroy_watch(struct one_watch *watch) free(watch); } -static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, +static int start_rdcw_watch(struct fsm_listen_data *data, struct one_watch *watch) { DWORD dwNotifyFilter = @@ -516,7 +516,7 @@ static int process_1_worktree_event( */ static int process_worktree_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_worktree; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -646,7 +646,7 @@ force_shutdown: */ static int process_gitdir_events(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; struct one_watch *watch = data->watch_gitdir; struct strbuf path = STRBUF_INIT; struct string_list cookie_list = STRING_LIST_INIT_DUP; @@ -704,11 +704,11 @@ skip_this_path: void fsm_listen__loop(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct fsm_listen_data *data = state->listen_data; DWORD dwWait; int result; - state->error_code = 0; + state->listen_error_code = 0; if (start_rdcw_watch(data, data->watch_worktree) == -1) goto force_error_stop; @@ -773,7 +773,7 @@ void fsm_listen__loop(struct fsmonitor_daemon_state *state) } force_error_stop: - state->error_code = -1; + state->listen_error_code = -1; force_shutdown: /* @@ -790,7 +790,7 @@ clean_shutdown: int fsm_listen__ctor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; CALLOC_ARRAY(data, 1); @@ -823,7 +823,7 @@ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) data->nr_listener_handles++; } - state->backend_data = data; + state->listen_data = data; return 0; failed: @@ -836,16 +836,16 @@ failed: void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { - struct fsmonitor_daemon_backend_data *data; + struct fsm_listen_data *data; - if (!state || !state->backend_data) + if (!state || !state->listen_data) return; - data = state->backend_data; + data = state->listen_data; CloseHandle(data->hEventShutdown); destroy_watch(data->watch_worktree); destroy_watch(data->watch_gitdir); - FREE_AND_NULL(state->backend_data); + FREE_AND_NULL(state->listen_data); } diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h index f0539349baf..41650bf8972 100644 --- a/compat/fsmonitor/fsm-listen.h +++ b/compat/fsmonitor/fsm-listen.h @@ -33,7 +33,7 @@ void fsm_listen__dtor(struct fsmonitor_daemon_state *state); * do so if the listener thread receives a normal shutdown signal from * the IPC layer.) * - * It should set `state->error_code` to -1 if the daemon should exit + * It should set `state->listen_error_code` to -1 if the daemon should exit * with an error. */ void fsm_listen__loop(struct fsmonitor_daemon_state *state); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 223c2131b58..2c6fa1a5d91 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -33,7 +33,7 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); */ void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); -struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ +struct fsm_listen_data; /* opaque platform-specific data for listener thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; @@ -50,8 +50,8 @@ struct fsmonitor_daemon_state { int cookie_seq; struct hashmap cookies; - int error_code; - struct fsmonitor_daemon_backend_data *backend_data; + int listen_error_code; + struct fsm_listen_data *listen_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Tue Mar 8 22:15:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774455 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BFFADC433F5 for ; Tue, 8 Mar 2022 22:16:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350673AbiCHWRL (ORCPT ); Tue, 8 Mar 2022 17:17:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350682AbiCHWQ6 (ORCPT ); Tue, 8 Mar 2022 17:16:58 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A00D57B04 for ; Tue, 8 Mar 2022 14:15:47 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id h15so249582wrc.6 for ; Tue, 08 Mar 2022 14:15:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fs9iPJ+0XRv7tWAgTyO8PFsvCHwRp8Z89GqJBryeRrg=; b=FPmFQJMrZbLjK1eD3vC7MPhUzCccfjwOsTRZndOFegmk7q2g1QoAk/FFsrp1ITCitl OeObNiHUvkXPVRilohOKtR6oted1DfZYzYTz914crjI10k/g6siQ4GnTLH4TU3hoPa9A IfQYsiW4zJjU628DehGznJYymVc0x7zrd4zIY124C6SMYIojOWNPamUlQHJxRioROPY+ geXvyq1NC9roTZK2e025uL7sZ2BzbsqYGKFNe026yNtolFL1cx/vYrTYxLD2um9TSHCC qU8g+H1SVWQE4Kv5QKvzWZcCu5tNmFu4oCw5EG7mRlVqvtUEUgyajYgpvOfPUKLnXO0r 4N/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fs9iPJ+0XRv7tWAgTyO8PFsvCHwRp8Z89GqJBryeRrg=; b=45qfwvi40vkj/bWsw7gAoBoRG38jCXKZqmsUTknXhZkT/PGYqD9bJ6DEfNUVtiH63Y KNpsW7AgDyiiulwP35w7/5Qlu7hOLFVau6NvnKJNss9EH5CfYZD3H6u7MRXeyqJlfn38 8naAp8rnHj4+kvBqLBVKtVa7PwDPavyurJJMZbFnEQtorsn3fOUvFqwqyOrXAQniPG0O 2iBc5d3wa5hdnfmppjSyvUCNqLUxz722c3ZmqRgN3a9kJqbz1uWD3CJYjg02ZgMKiRLp nte6rn5ZQ1unJjrT/eLdE1hreboXbjPDIIcIoHXO4kZgty7ZwlAt9SB4pV6eN7BSNrgR vb4Q== X-Gm-Message-State: AOAM532EMk+4Zfb6SmOlVVQHT0nmii8fHRYUKHPqDcUESd1eIALhf1R/ 31dOLDB99Z54XWZqIxIW6h17WnKaoe4= X-Google-Smtp-Source: ABdhPJz3AycHPSQGXm7yuy1VNDc6zv8m5juzytGF/182AacvmERDFgHbwZNF3tlTf/Ys3A8uK5Zqzw== X-Received: by 2002:adf:82aa:0:b0:1f0:21ec:6c46 with SMTP id 39-20020adf82aa000000b001f021ec6c46mr13537746wrc.286.1646777746200; Tue, 08 Mar 2022 14:15:46 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l25-20020a1c7919000000b0038999b380e9sm57480wme.38.2022.03.08.14.15.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:45 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:15 +0000 Subject: [PATCH v2 15/27] fsmonitor--daemon: stub in health thread Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create another thread to watch over the daemon process and automatically shut it down if necessary. This commit creates the basic framework for a "health" thread to monitor the daemon and/or the file system. Later commits will add platform-specific code to do the actual work. The "health" thread is intended to monitor conditions that would be difficult to track inside the IPC thread pool and/or the file system listener threads. For example, when there are file system events outside of the watched worktree root or if we want to have an idle-timeout auto-shutdown feature. This commit creates the health thread itself, defines the thread-proc and sets up the thread's event loop. It integrates this new thread into the existing IPC and Listener thread models. This commit defines the API to the platform-specific code where all of the monitoring will actually happen. The platform-specific code for MacOS is just stubs. Meaning that the health thread will immediately exit on MacOS, but that is OK and expected. Future work can define MacOS-specific monitoring. The platform-specific code for Windows sets up enough of the WaitForMultipleObjects() machinery to watch for system and/or custom events. Currently, the set of wait handles only includes our custom shutdown event (sent from our other theads). Later commits in this series will extend the set of wait handles to monitor other conditions. Signed-off-by: Jeff Hostetler --- Makefile | 6 ++- builtin/fsmonitor--daemon.c | 39 +++++++++++++++ compat/fsmonitor/fsm-health-darwin.c | 24 ++++++++++ compat/fsmonitor/fsm-health-win32.c | 72 ++++++++++++++++++++++++++++ compat/fsmonitor/fsm-health.h | 47 ++++++++++++++++++ contrib/buildsystems/CMakeLists.txt | 2 + fsmonitor--daemon.h | 4 ++ 7 files changed, 192 insertions(+), 2 deletions(-) create mode 100644 compat/fsmonitor/fsm-health-darwin.c create mode 100644 compat/fsmonitor/fsm-health-win32.c create mode 100644 compat/fsmonitor/fsm-health.h diff --git a/Makefile b/Makefile index 93604fe8ef7..5f1623baadd 100644 --- a/Makefile +++ b/Makefile @@ -472,8 +472,9 @@ all:: # # If your platform supports a built-in fsmonitor backend, set # FSMONITOR_DAEMON_BACKEND to the "" of the corresponding -# `compat/fsmonitor/fsm-listen-.c` that implements the -# `fsm_listen__*()` routines. +# `compat/fsmonitor/fsm-listen-.c` and +# `compat/fsmonitor/fsm-health-.c` files +# that implement the `fsm_listen__*()` and `fsm_health__*()` routines. # # If your platform has OS-specific ways to tell if a repo is incompatible with # fsmonitor (whether the hook or IPC daemon version), set FSMONITOR_OS_SETTINGS @@ -1982,6 +1983,7 @@ endif ifdef FSMONITOR_DAEMON_BACKEND COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o + COMPAT_OBJS += compat/fsmonitor/fsm-health-$(FSMONITOR_DAEMON_BACKEND).o endif ifdef FSMONITOR_OS_SETTINGS diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 92c61d5b94d..bebb3a292e2 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,6 +3,7 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsm-health.h" #include "compat/fsmonitor/fsm-listen.h" #include "fsmonitor--daemon.h" #include "simple-ipc.h" @@ -1130,6 +1131,18 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, pthread_mutex_unlock(&state->main_lock); } +static void *fsm_health__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-health"); + + fsm_health__loop(state); + + trace2_thread_exit(); + return NULL; +} + static void *fsm_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -1168,6 +1181,7 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) */ .uds_disallow_chdir = 0 }; + int health_started = 0; int listener_started = 0; int err = 0; @@ -1195,6 +1209,17 @@ static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) } listener_started = 1; + /* + * Start the health thread to watch over our process. + */ + if (pthread_create(&state->health_thread, NULL, + fsm_health__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + err = error(_("could not start fsmonitor health thread")); + goto cleanup; + } + health_started = 1; + /* * The daemon is now fully functional in background threads. * Our primary thread should now just wait while the threads @@ -1217,10 +1242,17 @@ cleanup: pthread_join(state->listener_thread, NULL); } + if (health_started) { + fsm_health__stop_async(state); + pthread_join(state->health_thread, NULL); + } + if (err) return err; if (state->listen_error_code) return state->listen_error_code; + if (state->health_error_code) + return state->health_error_code; return 0; } @@ -1236,6 +1268,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); pthread_cond_init(&state.cookies_cond, NULL); state.listen_error_code = 0; + state.health_error_code = 0; state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ @@ -1315,6 +1348,11 @@ static int fsmonitor_run_daemon(void) goto done; } + if (fsm_health__ctor(&state)) { + err = error(_("could not initialize health thread")); + goto done; + } + /* * CD out of the worktree root directory. * @@ -1338,6 +1376,7 @@ done: pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsm_listen__dtor(&state); + fsm_health__dtor(&state); ipc_server_free(state.ipc_server_data); diff --git a/compat/fsmonitor/fsm-health-darwin.c b/compat/fsmonitor/fsm-health-darwin.c new file mode 100644 index 00000000000..b9f709e8548 --- /dev/null +++ b/compat/fsmonitor/fsm-health-darwin.c @@ -0,0 +1,24 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c new file mode 100644 index 00000000000..94b1d020f25 --- /dev/null +++ b/compat/fsmonitor/fsm-health-win32.c @@ -0,0 +1,72 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-health.h" +#include "fsmonitor--daemon.h" + +struct fsm_health_data +{ + HANDLE hEventShutdown; + + HANDLE hHandles[1]; /* the array does not own these handles */ +#define HEALTH_SHUTDOWN 0 + int nr_handles; /* number of active event handles */ +}; + +int fsm_health__ctor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + CALLOC_ARRAY(data, 1); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->hHandles[HEALTH_SHUTDOWN] = data->hEventShutdown; + data->nr_handles++; + + state->health_data = data; + return 0; +} + +void fsm_health__dtor(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data; + + if (!state || !state->health_data) + return; + + data = state->health_data; + + CloseHandle(data->hEventShutdown); + + FREE_AND_NULL(state->health_data); +} + +void fsm_health__loop(struct fsmonitor_daemon_state *state) +{ + struct fsm_health_data *data = state->health_data; + + for (;;) { + DWORD dwWait = WaitForMultipleObjects(data->nr_handles, + data->hHandles, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) + goto clean_shutdown; + + error(_("health thread wait failed [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->health_error_code = -1; + ipc_server_stop_async(state->ipc_server_data); +clean_shutdown: + return; +} + +void fsm_health__stop_async(struct fsmonitor_daemon_state *state) +{ + SetEvent(state->health_data->hHandles[HEALTH_SHUTDOWN]); +} diff --git a/compat/fsmonitor/fsm-health.h b/compat/fsmonitor/fsm-health.h new file mode 100644 index 00000000000..45547ba9380 --- /dev/null +++ b/compat/fsmonitor/fsm-health.h @@ -0,0 +1,47 @@ +#ifndef FSM_HEALTH_H +#define FSM_HEALTH_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor health thread. + * This will be called from the main thread PRIOR to staring the + * thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsm_health__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the health thread. + * This will be called from the main thread AFTER joining the thread. + */ +void fsm_health__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to monitor the + * health of the daemon process. This will run in the health thread. + * + * The health thread should call `ipc_server_stop_async()` if it needs + * to cause a shutdown. (It should NOT do so if it receives a shutdown + * shutdown signal.) + * + * It should set `state->health_error_code` to -1 if the daemon should exit + * with an error. + */ +void fsm_health__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the health thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsm_health__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSM_HEALTH_H */ diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index b8f9f7a0388..16ace43d1c7 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,12 +289,14 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-win32.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-win32.c) elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-health-darwin.c) add_compile_definitions(HAVE_FSMONITOR_OS_SETTINGS) list(APPEND compat_SOURCES compat/fsmonitor/fsm-settings-darwin.c) diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 2c6fa1a5d91..2102a5c9ff5 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -34,9 +34,11 @@ void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); struct fsm_listen_data; /* opaque platform-specific data for listener thread */ +struct fsm_health_data; /* opaque platform-specific data for health thread */ struct fsmonitor_daemon_state { pthread_t listener_thread; + pthread_t health_thread; pthread_mutex_t main_lock; struct strbuf path_worktree_watch; @@ -51,7 +53,9 @@ struct fsmonitor_daemon_state { struct hashmap cookies; int listen_error_code; + int health_error_code; struct fsm_listen_data *listen_data; + struct fsm_health_data *health_data; struct ipc_server_data *ipc_server_data; struct strbuf path_ipc; From patchwork Tue Mar 8 22:15:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89839C433F5 for ; Tue, 8 Mar 2022 22:16:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350685AbiCHWR2 (ORCPT ); Tue, 8 Mar 2022 17:17:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51504 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350683AbiCHWQ6 (ORCPT ); Tue, 8 Mar 2022 17:16:58 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B8BCE57B0D for ; Tue, 8 Mar 2022 14:15:48 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id r81-20020a1c4454000000b00389d0a5c511so54243wma.5 for ; Tue, 08 Mar 2022 14:15:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=eIviaQSAxrTfIh3Aq5XMvNIDPt8STTcza4PYxm8fvT+Lw2PgzgOKh83XymuGrQ6i5M Lh1rXuyLIWX6BdD9YbvwTVjE+mXClZSC51VtIQT4ANV61VfZDiiN92la1Og/HFWerkGX SlMUbrJ+Jg2lRwvYxTCcdgYm7t5Yjx2KbylirbxXJTa8C29TJHqx87GW9PYjh/yoZM1g 4tnTm/a5XhUYy8rSVyV7gzxDb7FBiy/TRZdOinMExA84zrDquuYjsTKyJU1WERH+9Rf9 qEmTMWifvxHFi6bBsqwW/IAPFNCb6UNaM4QPseIJE0sSSLCySn8R61kcCpyosYPMyk4v 1rUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Mv6D24bMSysj3hV8m09u6C4BPD9U2eW4vus0bJo6UBo=; b=1TBOy0iHREdjJYdE537PhmDBy1Zjx56EQXJK4zMKMdxUcG3181bypOU+QmA8EPRlQJ niSiqyPMj9ZlvMlvSuw5gXlzSOdCX1OUeAGBA2VP32Sff0bAO1xyBH3Z2QcIBWkynIZv pf/+CrsbCBgQgKVkzD0u1ObctfGRiWGuCCL+oq1QUF95rM2FGKHSR0dDAf/Ohp8X1Qw5 Kyvfr7h8YlNo3mrsPFCE8TjjmguvMjUrELWJ/C9gPVOshC6mBc+uhbSdOzLxyveozGk+ 5MY3FR+/CMWvarAoMqkeM1/ogshD0I7yyPltF0L0by8E7GmRL4dItAONuIl5gNMDINt2 Og3Q== X-Gm-Message-State: AOAM532ZgXxfIIW7ZYG8FNadFQzyJfiQy9IeEO7fQqvLBaF0gxXYTPhV iVB1cnQQUMDFqwVZ17ljqVSRVsC2Ihk= X-Google-Smtp-Source: ABdhPJxCmGCKOrM+g22P8TgXFu6pijG7EaKUfOvyS3aBVa5NyyrRQzzjPgkiLnf+zoM8VOSVBu8EUQ== X-Received: by 2002:a1c:f003:0:b0:37b:d5fc:5c9e with SMTP id a3-20020a1cf003000000b0037bd5fc5c9emr1041878wmb.154.1646777747147; Tue, 08 Mar 2022 14:15:47 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bl20-20020adfe254000000b001f1fa450a3asm130829wrb.72.2022.03.08.14.15.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:46 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:16 +0000 Subject: [PATCH v2 16/27] fsm-health-win32: add polling framework to monitor daemon health Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Extend the Windows version of the "health" thread to periodically inspect the system and shutdown if warranted. This commit updates the thread's wait loop to use a timeout and defines a (currently empty) table of functions to poll the system. A later commit will add functions to the table to actually inspect the system. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 65 ++++++++++++++++++++++++++++- 1 file changed, 64 insertions(+), 1 deletion(-) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 94b1d020f25..24fc612bf02 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -4,6 +4,24 @@ #include "fsm-health.h" #include "fsmonitor--daemon.h" +/* + * Every minute wake up and test our health. + */ +#define WAIT_FREQ_MS (60 * 1000) + +/* + * State machine states for each of the interval functions + * used for polling our health. + */ +enum interval_fn_ctx { + CTX_INIT = 0, + CTX_TERM, + CTX_TIMER +}; + +typedef int (interval_fn)(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx); + struct fsm_health_data { HANDLE hEventShutdown; @@ -42,18 +60,61 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) FREE_AND_NULL(state->health_data); } +/* + * A table of the polling functions. + */ +static interval_fn *table[] = { + NULL, /* must be last */ +}; + +/* + * Call all of the polling functions in the table. + * Shortcut and return first error. + * + * Return 0 if all succeeded. + */ +static int call_all(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + int k; + + for (k = 0; table[k]; k++) { + int r = table[k](state, ctx); + if (r) + return r; + } + + return 0; +} + void fsm_health__loop(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data = state->health_data; + int r; + + r = call_all(state, CTX_INIT); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; for (;;) { DWORD dwWait = WaitForMultipleObjects(data->nr_handles, data->hHandles, - FALSE, INFINITE); + FALSE, WAIT_FREQ_MS); if (dwWait == WAIT_OBJECT_0 + HEALTH_SHUTDOWN) goto clean_shutdown; + if (dwWait == WAIT_TIMEOUT) { + r = call_all(state, CTX_TIMER); + if (r < 0) + goto force_error_stop; + if (r > 0) + goto force_shutdown; + continue; + } + error(_("health thread wait failed [GLE %ld]"), GetLastError()); goto force_error_stop; @@ -61,8 +122,10 @@ void fsm_health__loop(struct fsmonitor_daemon_state *state) force_error_stop: state->health_error_code = -1; +force_shutdown: ipc_server_stop_async(state->ipc_server_data); clean_shutdown: + call_all(state, CTX_TERM); return; } From patchwork Tue Mar 8 22:15:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774456 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA2F1C433EF for ; Tue, 8 Mar 2022 22:16:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350682AbiCHWRN (ORCPT ); Tue, 8 Mar 2022 17:17:13 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350687AbiCHWQ6 (ORCPT ); Tue, 8 Mar 2022 17:16:58 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D7E2257B27 for ; Tue, 8 Mar 2022 14:15:49 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id i8so240971wrr.8 for ; Tue, 08 Mar 2022 14:15:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=I0DEwYAS/m99+SGTZO7U7pFq7MF2RJupiNoTd3v948U=; b=GTJcwX8DUTx+FBq7SeE4laWCDdc7fHRuCXJ4PdSUL0aWOmEzsDbqTVuVk7GwspqRBf 8KhxhpT0XiHmwh92YNnkV2IxnzM/2+kpRMhugLNgxdPbu/uuhHOnhjstG1hf1VcBNJpP RyGpRgrv4F6s8ONu9K5PCoqMOTzUP7pKhZzLTyJJe/bJrNpb43U1z3iV6br+Oev+HgsX J9LXnfDSb3okkUz4H0YT64OLmw9U8BufphJc+ERmxOXnYo37u1OhtJK0r8aC3P4P8Oq+ aV/DYzaPFsWLoZFfv3eEvkljwnyfmpBZHe/wgks6raNCZiCYVB992nzULzSajCM+7V1J aBnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=I0DEwYAS/m99+SGTZO7U7pFq7MF2RJupiNoTd3v948U=; b=5bde1FUyub/AUFj8oPed8UtCrej9roIVJocg2rwvt8XlcEP1zs2/n63W90Beb/R36U nJuXpSU87wlMmOplc5IntKPcxIomwyO3dKmMVEREn/2ieKGIwm+lPGSu70YQgnNxMHTV ewMl76ZndQ/r0X+t7WB4O68ly6o3FZltsohbk4eC/hcTSFSkgYXk95qUB+SoRZkGnhQD 3Qes5yoF63rYGIAAFEdK+47MccFdXbAZJ6Ms+VIFQKG16LywilnJi4alJhyyZvvu0WTs 1WUXJA5xxDBVSYA1NT4b0WnmIr96+bAnawRVrCSGZ7lD4de86YXsnaNNie9Ax8uwQvCO maxg== X-Gm-Message-State: AOAM5317kXKSNdZyuLfZLtLvmldXmspZJDPqMoB3AjcYB7gCKy+4juCG Kz9XyU17sF2dFDHGmpXR1ux+IGOIO08= X-Google-Smtp-Source: ABdhPJwhdcT12VqJw1L1o7O991Mt3fjZP1oV7CtVbigifnWjjg0UhfUT7TGaN8k3DZdYVqozSYEcEw== X-Received: by 2002:a5d:404b:0:b0:1f1:f880:7aca with SMTP id w11-20020a5d404b000000b001f1f8807acamr9300698wrp.179.1646777748179; Tue, 08 Mar 2022 14:15:48 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m14-20020a05600c4f4e00b0038181486018sm64001wmq.40.2022.03.08.14.15.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:47 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:17 +0000 Subject: [PATCH v2 17/27] fsm-health-win32: force shutdown daemon if worktree root moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Force shutdown fsmonitor daemon if the worktree root directory is moved, renamed, or deleted. Use Windows low-level GetFileInformationByHandle() to get and compare the Windows system unique ID for the directory with a cached version when we started up. This lets us detect the case where someone renames the directory that we are watching and then creates a new directory with the original pathname. This is important because we are listening to a named pipe for requests and they are stored in the Named Pipe File System (NPFS) which a kernel-resident pseudo filesystem not associated with the actual NTFS directory. For example, if the daemon was watching "~/foo/", it would have a directory-watch handle on that directory and a named-pipe handle for "//./pipe/...foo". Moving the directory to "~/bar/" does not invalidate the directory handle. (So the daemon would actually be watching "~/bar" but listening on "//./pipe/...foo". If the user then does "git init ~/foo" and causes another daemon to start, the first daemon will still have ownership of the pipe and the second daemon instance will fail to start. "git status" clients in "~/foo" will ask "//./pipe/...foo" about changes and the first daemon instance will tell them about "~/bar". This commit causes the first daemon to shutdown if the system unique ID for "~/foo" changes (changes from what it was when the daemon started). Shutdown occurs after a periodic poll. After the first daemon exits and releases the lock on the named pipe, subsequent Git commands may cause another daemon to be started on "~/foo". Similarly, a subsequent Git command may cause another daemon to be started on "~/bar". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-health-win32.c | 143 ++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) diff --git a/compat/fsmonitor/fsm-health-win32.c b/compat/fsmonitor/fsm-health-win32.c index 24fc612bf02..0132ca79305 100644 --- a/compat/fsmonitor/fsm-health-win32.c +++ b/compat/fsmonitor/fsm-health-win32.c @@ -29,8 +29,150 @@ struct fsm_health_data HANDLE hHandles[1]; /* the array does not own these handles */ #define HEALTH_SHUTDOWN 0 int nr_handles; /* number of active event handles */ + + struct wt_moved + { + wchar_t wpath[MAX_PATH + 1]; + BY_HANDLE_FILE_INFORMATION bhfi; + } wt_moved; }; +/* + * Lookup the system unique ID for the path. This is as close as + * we get to an inode number, but this also contains volume info, + * so it is a little stronger. + */ +static int lookup_bhfi(wchar_t *wpath, + BY_HANDLE_FILE_INFORMATION *bhfi) +{ + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + + hDir = CreateFileW(wpath, desired_access, share_mode, NULL, + OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS, NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] health thread could not open '%ls'"), + GetLastError(), wpath); + return -1; + } + + if (!GetFileInformationByHandle(hDir, bhfi)) { + error(_("[GLE %ld] health thread getting BHFI for '%ls'"), + GetLastError(), wpath); + CloseHandle(hDir); + return -1; + } + + CloseHandle(hDir); + return 0; +} + +/* + * Compare the relevant fields from two system unique IDs. + * We use this to see if two different handles to the same + * path actually refer to the same *instance* of the file + * or directory. + */ +static int bhfi_eq(const BY_HANDLE_FILE_INFORMATION *bhfi_1, + const BY_HANDLE_FILE_INFORMATION *bhfi_2) +{ + return (bhfi_1->dwVolumeSerialNumber == bhfi_2->dwVolumeSerialNumber && + bhfi_1->nFileIndexHigh == bhfi_2->nFileIndexHigh && + bhfi_1->nFileIndexLow == bhfi_2->nFileIndexLow); +} + +/* + * Shutdown if the original worktree root directory been deleted, + * moved, or renamed? + * + * Since the main thread did a "chdir(getenv($HOME))" and our CWD + * is not in the worktree root directory and because the listener + * thread added FILE_SHARE_DELETE to the watch handle, it is possible + * for the root directory to be moved or deleted while we are still + * watching it. We want to detect that here and force a shutdown. + * + * Granted, a delete MAY cause some operations to fail, such as + * GetOverlappedResult(), but it is not guaranteed. And because + * ReadDirectoryChangesW() only reports on changes *WITHIN* the + * directory, not changes *ON* the directory, our watch will not + * receive a delete event for it. + * + * A move/rename of the worktree root will also not generate an event. + * And since the listener thread already has an open handle, it may + * continue to receive events for events within the directory. + * However, the pathname of the named-pipe was constructed using the + * original location of the worktree root. (Remember named-pipes are + * stored in the NPFS and not in the actual file system.) Clients + * trying to talk to the worktree after the move/rename will not + * reach our daemon process, since we're still listening on the + * pipe with original path. + * + * Furthermore, if the user does something like: + * + * $ mv repo repo.old + * $ git init repo + * + * A new daemon cannot be started in the new instance of "repo" + * because the named-pipe is still being used by the daemon on + * the original instance. + * + * So, detect move/rename/delete and shutdown. This should also + * handle unsafe drive removal. + * + * We use the file system unique ID to distinguish the original + * directory instance from a new instance and force a shutdown + * if the unique ID changes. + * + * Since a worktree move/rename/delete/unmount doesn't happen + * that often (and we can't get an immediate event anyway), we + * use a timeout and periodically poll it. + */ +static int has_worktree_moved(struct fsmonitor_daemon_state *state, + enum interval_fn_ctx ctx) +{ + struct fsm_health_data *data = state->health_data; + BY_HANDLE_FILE_INFORMATION bhfi; + int r; + + switch (ctx) { + case CTX_TERM: + return 0; + + case CTX_INIT: + if (xutftowcs_path(data->wt_moved.wpath, + state->path_worktree_watch.buf) < 0) { + error(_("could not convert to wide characters: '%s'"), + state->path_worktree_watch.buf); + return -1; + } + + /* + * On the first call we lookup the unique sequence ID for + * the worktree root directory. + */ + return lookup_bhfi(data->wt_moved.wpath, &data->wt_moved.bhfi); + + case CTX_TIMER: + r = lookup_bhfi(data->wt_moved.wpath, &bhfi); + if (r) + return r; + if (!bhfi_eq(&data->wt_moved.bhfi, &bhfi)) { + error(_("BHFI changed '%ls'"), data->wt_moved.wpath); + return -1; + } + return 0; + + default: + die("unhandled case in 'has_worktree_moved': %d", + (int)ctx); + } + + return 0; +} + + int fsm_health__ctor(struct fsmonitor_daemon_state *state) { struct fsm_health_data *data; @@ -64,6 +206,7 @@ void fsm_health__dtor(struct fsmonitor_daemon_state *state) * A table of the polling functions. */ static interval_fn *table[] = { + has_worktree_moved, NULL, /* must be last */ }; From patchwork Tue Mar 8 22:15:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774458 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 163BDC433F5 for ; Tue, 8 Mar 2022 22:16:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350688AbiCHWRR (ORCPT ); Tue, 8 Mar 2022 17:17:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350692AbiCHWQ6 (ORCPT ); Tue, 8 Mar 2022 17:16:58 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C440E57B34 for ; Tue, 8 Mar 2022 14:15:50 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id p9so216763wra.12 for ; Tue, 08 Mar 2022 14:15:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=IORzkfN9FX4mvaJUW1/JQvql7wO5s6l+Di6stClhuZA=; b=PC84uA7XD8T3CS8QFApb8Mr76iz04a3EdIuVxAEGs2q4R3kCCTw6EquwgTu1sjo5+m xmHs5O7M1cTSPBpWDziwqFkm822FfTH9RMy1uuwBURd+oyaw0QzxwGiYhIAoiwchIoy8 ztRNkH8owf+28KOh7rG/tr3MXobe7RVT5JQ9w9AkzEeBaGxq5DZtia9/w/McL602v3+c F8MDBgxPgdiMS/TigUs0OKOqfLrOdmQbCNblwYHPK4pr7HXBadzjFiHzR5LR14l3VzEk mqV64YPC62kD+ueQAmjuETfD3BriXdWhnRU84syMFQi7QDR6+N2+fDMcwO6P4ZvIoHoH G+3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=IORzkfN9FX4mvaJUW1/JQvql7wO5s6l+Di6stClhuZA=; b=nt9WMC+g8GoejKp0WmmGpK4QgmbW143AHzyUraB4BIVi3JaUbhmG0uFYGC4NpQ8264 Ff/9FirLZUKtBLsvJDcomUkv3di8VKTkcfKpELuksOXi9OJquZiKA3rdb3+GJE7/eLOz efkhpr//f2L7+plOcIGzryNBlc7GLND/c2vCjWKVjFobO2dl/LFHidsqtqIavPdGgeGZ N1uBDfA1BD8gxDgL0W9PFbuB3onxCalGtNMCoIbPPTEBQlXrN5MipOuh+9EFBiKa3UTf XopEyqcD9TV1PFvUiuVd8GXBoJqtAhpl2mstfBt3uWnA7/YVcJ5uLvZN7uKYvTvWElwk qIDQ== X-Gm-Message-State: AOAM5326fW3iBiJLExnPQIcwM0KiKQun7S5FizaF83Wki2LO8sY+Cb4s g+TVpvtE0Y+xonIOTGzDfsVwrDcS3cE= X-Google-Smtp-Source: ABdhPJwMFIZLDmE3tbnokNY31W0tZcZLddCzKiT3x828llYKxShGT5daBxOMOkhCSqHwRbvrkBCrUg== X-Received: by 2002:a05:6000:1a89:b0:203:7089:ea88 with SMTP id f9-20020a0560001a8900b002037089ea88mr3194403wry.651.1646777749125; Tue, 08 Mar 2022 14:15:49 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q14-20020a1cf30e000000b0038986a18ec8sm53492wmq.46.2022.03.08.14.15.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:48 -0800 (PST) Message-Id: <796b65913939e4de5ff745981e3d36fe4926ae08.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:18 +0000 Subject: [PATCH v2 18/27] fsm-listen-darwin: shutdown daemon if worktree root is moved/renamed Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the listener thread to shutdown the daemon if the spelling of the worktree root directory changes. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index 7c81bc7e5bf..b6c33f2cf3b 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -178,6 +178,11 @@ static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) strbuf_release(&msg); } +static int ef_is_root_changed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagRootChanged); +} + static int ef_is_root_delete(const FSEventStreamEventFlags ef) { return (ef & kFSEventStreamEventFlagItemIsDir && @@ -287,6 +292,26 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, continue; } + if (ef_is_root_changed(event_flags[k])) { + /* + * The spelling of the pathname of the root directory + * has changed. This includes the name of the root + * directory itself or of any parent directory in the + * path. + * + * (There may be other conditions that throw this, + * but I couldn't find any information on it.) + * + * Force a shutdown now and avoid things getting + * out of sync. The Unix domain socket is inside + * the .git directory and a spelling change will make + * it hard for clients to rendezvous with us. + */ + trace_printf_key(&trace_fsmonitor, + "event: root changed"); + goto force_shutdown; + } + if (ef_ignore_xattr(event_flags[k])) { trace_printf_key(&trace_fsmonitor, "ignore-xattr: '%s', flags=0x%x", From patchwork Tue Mar 8 22:15:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774457 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60766C433EF for ; Tue, 8 Mar 2022 22:16:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350676AbiCHWRP (ORCPT ); Tue, 8 Mar 2022 17:17:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350694AbiCHWQ6 (ORCPT ); Tue, 8 Mar 2022 17:16:58 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A033E57B3F for ; Tue, 8 Mar 2022 14:15:51 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id m42-20020a05600c3b2a00b00382ab337e14so2258985wms.3 for ; Tue, 08 Mar 2022 14:15:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=hAZBNC14DWoFoAtoCMmJip/ZRfCyYVrH43PTa6cCvxqu1mOgTCe1GYpdBwbclPADGg 1/qXqp2XlN8awB0aBzYXPUbt/gPB/wsR+emWDDnr4TZAC6knd765fGTnE/q54q/p9BCa w1I/XYZsHHdo/5GgUJTbcT/i9wjQK1Lp5+k9x5kid1+sR/FPMkgtJZH9V+HIZvWbBAmT xsqet9MX+4DDCwN6dVxjXunqUz+bxYjJvx7zd8MRw5EmscX0S6GStyVTik3QzZ6AXJun 16JkFrqQ8MvM9WI/1jz/MZrhyCqKUemaVhp08/mUucBpp/xYPPVciVBJHffesytsi0cP SEkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=aGhX76dz325gkGkIrL9UlyaHr3ziBUV0IbljHMDHg+g=; b=1JpivRomEdEC25cXgZZk8DPi1uRO+KlbXtOmiJ89WslGHQ7LKc/31QtxrasWCmCgCp veYES4QvVIakT0w+a5qWOd33cTq1Ftc7P7NJGLVMYxneH2U5dG3tp0mNknuA2JyNmxd1 hyO5JAtCRtHfhfCj2iKhioQYGcNSPrtho6k5N29b5rP4KTL0vOAZRjdq1Xq93kvdkBZ1 TVXmNqy92xdc5zZrXgygWQFI1jM4UGomn1Ja8LEW1ElAEq8yhRoVbNgMR/ObuXDqNAux DkHTMCM9HLXo8FV0D6BUpSM0dCM+e16rtqlGY+V1ECMdTLAl8lr5VCiwfKWbSBDtJPkF 7/eg== X-Gm-Message-State: AOAM531mXE/NvMxL6hnxDQ8vuVn+qGtoO4j/gr5V3ydririBx8R4WB8O 57j+e9naGRuXc0pxFCKHv5q+20dNJaQ= X-Google-Smtp-Source: ABdhPJyIux1N0dFgBCIah+lAeQxhiWR08g1ZA7Cf2SRdu5zOfNolsJzZ/8Rrl9LTlPC5l5rCEY3q5w== X-Received: by 2002:a05:600c:a06:b0:37b:fdd8:4f8 with SMTP id z6-20020a05600c0a0600b0037bfdd804f8mr5360564wmp.41.1646777750063; Tue, 08 Mar 2022 14:15:50 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 11-20020a05600c26cb00b0037ff53511f2sm3062788wmv.31.2022.03.08.14.15.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:49 -0800 (PST) Message-Id: <24591920878c37b0d0b9d91e8b2d36b2eb81b24e.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:19 +0000 Subject: [PATCH v2 19/27] fsmonitor: optimize processing of directory events Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach Git to perform binary search over the cache-entries for a directory notification and then linearly scan forward to find the immediate children. Previously, when the FSMonitor reported a modified directory Git would perform a linear search on the entire cache-entry array for all entries matching that directory prefix and invalidate them. Since the cache-entry array is already sorted, we can use a binary search to find the first matching entry and then only linearly walk forward and invalidate entries until the prefix changes. Also, the original code would invalidate anything having the same directory prefix. Since a directory event should only be received for items that are immediately within the directory (and not within sub-directories of it), only invalidate those entries and not the whole subtree. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 71 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 54 insertions(+), 17 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 292a6742b4f..e1229c289cf 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -184,30 +184,68 @@ static int query_fsmonitor_hook(struct repository *r, static void fsmonitor_refresh_callback(struct index_state *istate, char *name) { int i, len = strlen(name); - if (name[len - 1] == '/') { + int pos = index_name_pos(istate, name, len); + + trace_printf_key(&trace_fsmonitor, + "fsmonitor_refresh_callback '%s' (pos %d)", + name, pos); + if (name[len - 1] == '/') { /* - * TODO We should binary search to find the first path with - * TODO this directory prefix. Then linearly update entries - * TODO while the prefix matches. Taking care to search without - * TODO the trailing slash -- because '/' sorts after a few - * TODO interesting special chars, like '.' and ' '. + * The daemon can decorate directory events, such as + * moves or renames, with a trailing slash if the OS + * FS Event contains sufficient information, such as + * MacOS. + * + * Use this to invalidate the entire cone under that + * directory. + * + * We do not expect an exact match because the index + * does not normally contain directory entries, so we + * start at the insertion point and scan. */ + if (pos < 0) + pos = -pos - 1; /* Mark all entries for the folder invalid */ - for (i = 0; i < istate->cache_nr; i++) { - if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID && - starts_with(istate->cache[i]->name, name)) - istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } - /* Need to remove the / from the path for the untracked cache */ + + /* + * We need to remove the traling "/" from the path + * for the untracked cache. + */ name[len - 1] = '\0'; + } else if (pos >= 0) { + /* + * We have an exact match for this path and can just + * invalidate it. + */ + istate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID; } else { - int pos = index_name_pos(istate, name, strlen(name)); - - if (pos >= 0) { - struct cache_entry *ce = istate->cache[pos]; - ce->ce_flags &= ~CE_FSMONITOR_VALID; + /* + * The path is not a tracked file -or- it is a + * directory event on a platform that cannot + * distinguish between file and directory events in + * the event handler, such as Windows. + * + * Scan as if it is a directory and invalidate the + * cone under it. (But remember to ignore items + * between "name" and "name/", such as "name-" and + * "name.". + */ + pos = -pos - 1; + + for (i = pos; i < istate->cache_nr; i++) { + if (!starts_with(istate->cache[i]->name, name)) + break; + if ((unsigned char)istate->cache[i]->name[len] > '/') + break; + if (istate->cache[i]->name[len] == '/') + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; } } @@ -215,7 +253,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) * Mark the untracked cache dirty even if it wasn't found in the index * as it could be a new untracked file. */ - trace_printf_key(&trace_fsmonitor, "fsmonitor_refresh_callback '%s'", name); untracked_cache_invalidate_path(istate, name, 0); } From patchwork Tue Mar 8 22:15:20 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774462 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59DB0C433F5 for ; Tue, 8 Mar 2022 22:16:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350671AbiCHWR0 (ORCPT ); Tue, 8 Mar 2022 17:17:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350699AbiCHWQ7 (ORCPT ); Tue, 8 Mar 2022 17:16:59 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A041580C9 for ; Tue, 8 Mar 2022 14:15:52 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id u10so234490wra.9 for ; Tue, 08 Mar 2022 14:15:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=alHS+bRntRrjIrbh+LYXIH/iRVyMrR0OzALWZ2LvUN8=; b=MGoM7STdI4Z4yjmaxPZP/sMcnPyg4T21BDkkktbjjoRkBm38dv2OO4IqF4db9CwP82 tplX3LOyn/idBDGBDVrnOj2cbF+WDXuYK18+U2wnU7zuBTcuts85AN9m4tlUqDL6erxD RDCeMoVxD//gm0X6uWEqyT9eQH9gTrRVK4Srv8a6QVl7l5xr1QjQCln2QQaB/y8b5Cjn NXTp5/F8ShVkPquMIoQV4Tn30eju74Qoj/yLfuGxYg+uqeqIIid+er6QJVNCNeD4TiuF X2Yh/+C5biBCuVt94B9nWubKGNUub/lLrNWb5i9Cbe6MpOmvUzVIU+40zqBFL2wqSytQ FTDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=alHS+bRntRrjIrbh+LYXIH/iRVyMrR0OzALWZ2LvUN8=; b=eaF3zSPu+7qraKKjgQ+WaQgl+oV+Sjng+b5cUm0uBmNVKquk4vklrrcwkgAeBovFpq b680bbA2UiCVUp7JNKcXDk8Yk/XIcdGX9PzImeiQQAabvgET+uUFwiVv36cRvL408s6i TIvUjx4WRHX644zKo3PeHeSyOuby/3pLDaJotWMaIbj2CQ9TlfO7t888nQXiCA4xrvpq DxPaCFkWMPh0MxsAi7H71IBKxntu4cRkil7UzXNrGg3h/skcBfl/ES2pqlt9Iur0w6y7 VMX5MczjU9rmJn4MnFjBQQwjGO0BU/SxntqkERebWfIgY4GVAUS1Eogei4bE4VjvS8i2 llCw== X-Gm-Message-State: AOAM531RnCNmNeJrzyGBFAjFvOZLfxTY6NO6G+P+a+iZjm/DfclCxgFx igAJ+qbaye+sMdBJqxXjLrDAVQFVqhY= X-Google-Smtp-Source: ABdhPJwS7AbnVyTofQLhXxEsQWRG1p/S8hEGcBwlhadxFSrSc+nBGFqeNKVJyWLj1Q3LZEn+xsklBQ== X-Received: by 2002:adf:bc09:0:b0:1f0:2483:48b with SMTP id s9-20020adfbc09000000b001f02483048bmr14351788wrg.118.1646777750805; Tue, 08 Mar 2022 14:15:50 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h12-20020a5d548c000000b001f1f99e7792sm78101wrv.111.2022.03.08.14.15.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:50 -0800 (PST) Message-Id: <06a32413854c2b35f289da3b42861fc051e1d4cc.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:20 +0000 Subject: [PATCH v2 20/27] t7527: FSMonitor tests for directory moves Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create unit tests to move a directory. Verify that `git status` gives the same result with and without FSMonitor enabled. NEEDSWORK: This test exposes a bug in the untracked-cache on Windows when FSMonitor is disabled. These are commented out for the moment. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 39 ++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index c2627f28865..8b63067cbf9 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -231,6 +231,16 @@ test_expect_success 'setup' ' trace* EOF + mkdir -p T1/T2/T3/T4 && + echo 1 >T1/F1 && + echo 1 >T1/T2/F1 && + echo 1 >T1/T2/T3/F1 && + echo 1 >T1/T2/T3/T4/F1 && + echo 2 >T1/F2 && + echo 2 >T1/T2/F2 && + echo 2 >T1/T2/T3/F2 && + echo 2 >T1/T2/T3/T4/F2 && + git -c core.fsmonitor=false add . && test_tick && git -c core.fsmonitor=false commit -m initial && @@ -324,6 +334,19 @@ verify_status () { echo HELLO AFTER } +move_directory_contents_deeper() { + mkdir T1/_new_ + mv T1/[A-Z]* T1/_new_ +} + +move_directory_up() { + mv T1/T2/T3 T1 +} + +move_directory() { + mv T1/T2/T3 T1/T2/NewT3 +} + # The next few test cases confirm that our fsmonitor daemon sees each type # of OS filesystem notification that we care about. At this layer we just # ensure we are getting the OS notifications and do not try to confirm what @@ -655,6 +678,22 @@ do matrix_try $uc_val $fsm_val file_to_directory matrix_try $uc_val $fsm_val directory_to_file + # NEEDSWORK: On Windows the untracked-cache is buggy when FSMonitor + # is DISABLED. Turn off a few test that cause it problems until + # we can debug it. + # + try_moves="true" + test_have_prereq UNTRACKED_CACHE,WINDOWS && \ + test $uc_val = true && \ + test $fsm_val = false && \ + try_moves="false" + if test $try_moves = true + then + matrix_try $uc_val $fsm_val move_directory_contents_deeper + matrix_try $uc_val $fsm_val move_directory_up + matrix_try $uc_val $fsm_val move_directory + fi + if test $fsm_val = true then test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" ' From patchwork Tue Mar 8 22:15:21 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774460 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BD64C433F5 for ; Tue, 8 Mar 2022 22:16:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350694AbiCHWRV (ORCPT ); Tue, 8 Mar 2022 17:17:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52078 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350709AbiCHWQ7 (ORCPT ); Tue, 8 Mar 2022 17:16:59 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9829C580DC for ; Tue, 8 Mar 2022 14:15:53 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id n33-20020a05600c3ba100b003832caf7f3aso2094170wms.0 for ; Tue, 08 Mar 2022 14:15:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=AWHDxp71FooubSOu6NyoomkByJm5Fqm+U3/G0wbVPg3VMxIK0snCmF+rSBDVcEQRqz QeJMhNWGitiQlH5GYZ+lueIXBu5j0RpyMGGWJSicifW3atgLy5ZdMdDqpCbW+/K/zMEf ++FCNK0b+mrkMEf7H3PIhOg4ZGDjniNrO43X3GYaoCmgsVUKS4O/slDObUe37HvVRsiX ImR+TbGAgq5gB/wBjyuUiVLws+rcnahtHjGN6NjrZvC3GylPczcYsCUJmTdn/mv4MdT8 IZ6251MthyiP1F+jZQ2uUPQRz63L8/sKRSRSghZnXBpbeQZvR/bhhhc9ePdk5Nyh/C9p jrzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3IAiMHgRwhfbvkUdsr17fwV8M0Qk+EnCGfvQEEa+RAk=; b=PhJzE35BMiU3juLt3KrRh9zdZo0NbVB01YLQ7YXN7IWUzqMtvJKq4vH2Mnufw7Rxbn CP2IF0QjbcMCrYqPJKBkgi4vq1uZs1hpyTXAa7GI9XkHD7KVGrqkhzks7+H3Nav+Uq+u H8CFhd+/R0/0HFdRx6nNiHSnKm5QEcUxL6DpnqvF5ZZqy/v8NORkTedbjFMeMtMgYDVg OQUnm8y2GuSqL0biXYIOu5Fa4XNE0ueuz+CbExI466WwO/l4YqTcnAzRbAO2RY+F45Uv cI6OGMs29rJe4ZAAt32hQOiFrfn5mXCPIZWKcjTSl3hwx/74GJhZntxM6p14b1Hqx6B/ 8Hrw== X-Gm-Message-State: AOAM5315GS+7QRfMO4uvjcBbDDGFGP9a86DnGnsTPJQeOoeYV9HmBjjU YUP0CnbufK3psHBBBVBaDBpuUW+iInc= X-Google-Smtp-Source: ABdhPJyNWu4cEW2He/7Umo/s75S1gmX6+muu7f6wVIQ9tbG+cnVaVaScBytH+jR5ZSa/qaAtUBTw7w== X-Received: by 2002:a1c:7212:0:b0:389:9f29:4e28 with SMTP id n18-20020a1c7212000000b003899f294e28mr5303390wmc.165.1646777751837; Tue, 08 Mar 2022 14:15:51 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u10-20020adfa18a000000b001f04c24afe7sm99708wru.41.2022.03.08.14.15.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:51 -0800 (PST) Message-Id: <4b59013caddc0d8ea6bdc2e4b78166c31b6e0027.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:21 +0000 Subject: [PATCH v2 21/27] t/perf/p7527: add perf test for builtin FSMonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/perf/p7527-builtin-fsmonitor.sh | 257 ++++++++++++++++++++++++++++++ 1 file changed, 257 insertions(+) create mode 100755 t/perf/p7527-builtin-fsmonitor.sh diff --git a/t/perf/p7527-builtin-fsmonitor.sh b/t/perf/p7527-builtin-fsmonitor.sh new file mode 100755 index 00000000000..9338b9ea008 --- /dev/null +++ b/t/perf/p7527-builtin-fsmonitor.sh @@ -0,0 +1,257 @@ +#!/bin/sh + +test_description="Perf test for the builtin FSMonitor" + +. ./perf-lib.sh + +if ! test_have_prereq FSMONITOR_DAEMON +then + skip_all="fsmonitor--daemon is not supported on this platform" + test_done +fi + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +# Lie to perf-lib and ask for a new empty repo and avoid +# the complaints about GIT_PERF_REPO not being big enough +# the perf hit when GIT_PERF_LARGE_REPO is copied into +# the trash directory. +# +# NEEDSWORK: It would be nice if perf-lib had an option to +# "borrow" an existing large repo (especially for gigantic +# monorepos) and use it in-place. For now, fake it here. +# +test_perf_fresh_repo + + +# Use a generated synthetic monorepo. If it doesn't exist, we will +# generate it. If it does exist, we will put it in a known state +# before we start our timings. +# +PARAM_D=5 +PARAM_W=10 +PARAM_F=9 + +PARAMS="$PARAM_D"."$PARAM_W"."$PARAM_F" + +BALLAST_BR=p0006-ballast +export BALLAST_BR + +TMP_BR=tmp_br +export TMP_BR + +REPO=../repos/gen-many-files-"$PARAMS".git +export REPO + +if ! test -d $REPO +then + (cd ../repos; ./many-files.sh -d $PARAM_D -w $PARAM_W -f $PARAM_F) +fi + + +enable_uc () { + git -C $REPO config core.untrackedcache true + git -C $REPO update-index --untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +disable_uc () { + git -C $REPO config core.untrackedcache false + git -C $REPO update-index --no-untracked-cache + git -C $REPO status >/dev/null 2>&1 +} + +start_fsm () { + git -C $REPO fsmonitor--daemon start + git -C $REPO fsmonitor--daemon status + git -C $REPO config core.fsmonitor true + git -C $REPO update-index --fsmonitor + git -C $REPO status >/dev/null 2>&1 +} + +stop_fsm () { + git -C $REPO config --unset core.fsmonitor + git -C $REPO update-index --no-fsmonitor + test_might_fail git -C $REPO fsmonitor--daemon stop 2>/dev/null + git -C $REPO status >/dev/null 2>&1 +} + + +# Ensure that FSMonitor is turned off on the borrowed repo. +# +test_expect_success "Setup borrowed repo (fsm+uc)" " + stop_fsm && + disable_uc +" + +# Also ensure that it starts in a known state. +# +# Because we assume that $GIT_PERF_REPEAT_COUNT > 1, we are not going to time +# the ballast checkout, since only the first invocation does any work and the +# subsequent ones just print "already on branch" and quit, so the reported +# time is not useful. +# +# Create a temp branch and do all work relative to it so that we don't +# accidentially alter the real ballast branch. +# +test_expect_success "Setup borrowed repo (temp ballast branch)" " + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO reset --hard && + git -C $REPO clean -d -f && + test_might_fail git -C $REPO branch -D $TMP_BR && + git -C $REPO branch $TMP_BR $BALLAST_BR && + git -C $REPO checkout $TMP_BR +" + + +echo Data >data.txt + +# NEEDSWORK: We assume that $GIT_PERF_REPEAT_COUNT > 1. With +# FSMonitor enabled, we can get a skewed view of status times, since +# the index MAY (or may not) be updated after the first invocation +# which will update the FSMonitor Token, so the subsequent invocations +# may get a smaller response from the daemon. +# +do_status () { + msg=$1 + + test_perf "$msg" " + git -C $REPO status >/dev/null 2>&1 + " +} + +do_matrix () { + uc=$1 + fsm=$2 + + t="[uc $uc][fsm $fsm]" + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_expect_success "$t Setup matrix branch" " + git -C $REPO clean -d -f && + git -C $REPO checkout $TMP_BR && + test_might_fail git -C $REPO branch -D $MATRIX_BR && + git -C $REPO branch $MATRIX_BR $TMP_BR && + git -C $REPO checkout $MATRIX_BR + " + + if test $uc = true + then + enable_uc + else + disable_uc + fi + + if test $fsm = true + then + start_fsm + else + stop_fsm + fi + + do_status "$t status after checkout" + + # Modify many files in the matrix branch. + # Stage them. + # Commit them. + # Rollback. + # + test_expect_success "$t modify tracked files" " + find $REPO -name file1 -exec cp data.txt {} \\; + " + + do_status "$t status after big change" + + # Don't bother timing the "add" because _REPEAT_COUNT + # issue described above. + # + test_expect_success "$t add all" " + git -C $REPO add -A + " + + do_status "$t status after add all" + + test_expect_success "$t add dot" " + git -C $REPO add . + " + + do_status "$t status after add dot" + + test_expect_success "$t commit staged" " + git -C $REPO commit -a -m data + " + + do_status "$t status after commit" + + test_expect_success "$t reset HEAD~1 hard" " + git -C $REPO reset --hard HEAD~1 >/dev/null 2>&1 + " + + do_status "$t status after reset hard" + + # Create some untracked files. + # + test_expect_success "$t create untracked files" " + cp -R $REPO/ballast/dir1 $REPO/ballast/xxx1 + " + + do_status "$t status after create untracked files" + + # Remove the new untracked files. + # + test_expect_success "$t clean -df" " + git -C $REPO clean -d -f + " + + do_status "$t status after clean" + + if test $fsm = true + then + stop_fsm + fi +} + +# Begin testing each case in the matrix that we care about. +# +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" + +fsm_values="false true" + +for uc_val in $uc_values +do + for fsm_val in $fsm_values + do + do_matrix $uc_val $fsm_val + done +done + +cleanup () { + uc=$1 + fsm=$2 + + MATRIX_BR="$TMP_BR-$uc-$fsm" + + test_might_fail git -C $REPO branch -D $MATRIX_BR +} + + +# We're borrowing this repo. We should leave it in a clean state. +# +test_expect_success "Cleanup temp and matrix branches" " + git -C $REPO clean -d -f && + test_might_fail git -C $REPO checkout $BALLAST_BR && + test_might_fail git -C $REPO branch -D $TMP_BR && + for uc_val in $uc_values + do + for fsm_val in $fsm_values + do + cleanup $uc_val $fsm_val + done + done +" + +test_done From patchwork Tue Mar 8 22:15:22 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774459 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3A36C433EF for ; Tue, 8 Mar 2022 22:16:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350690AbiCHWRS (ORCPT ); Tue, 8 Mar 2022 17:17:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52170 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350710AbiCHWQ7 (ORCPT ); Tue, 8 Mar 2022 17:16:59 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50F39580F5 for ; Tue, 8 Mar 2022 14:15:54 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id u10so234599wra.9 for ; Tue, 08 Mar 2022 14:15:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=iX4OPQfbBmRdMhDwqhFXHVMEwlidoGBXjSK/As5RDHE=; b=LlSPixr3TrwHXCqtCYws6qj/SjafXaeeSym3kc+WoNTMCiSuW/HUeCtDin2EkRREeN FjehwDFY1Js0LmPEL8O1EgAWvQg9U+okr/QLabMElIc8rL9ALLVopEmdD9Tnuvlh2zvb R87HShr7K1Y0tNBX2RgAe5QmuNydfKkxPlbUq3KA2wzAJXAYV5SEoc2+UMjEXCmTvRxg gFLWPmOlMEgBsFBx2inij1VI2zu0ljSq9Jq3i2KJ1OUleMSYTENQ8pghU9WZhS86OFgR IfSNyhrONSfrKTQETU1uCNy9OO3eEIvIAjcVngJbg4QSFK50maQeNziVKYrAMQFP4A6R i3zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=iX4OPQfbBmRdMhDwqhFXHVMEwlidoGBXjSK/As5RDHE=; b=Zou69GbNJ7kRKEvVj4xLTum7LQQtO2Bs3avFyiq3yyHp+Setlh5m75Ioy0/cei2fZw 60myZTfAyejLswtx/7N1H7QrWzDocNee4wlCFF9S0Eh4hrTR+GxFaYy1y92fwMFC/Dk3 vdyfIXKP5PPHgdcHjOhaZ3EbMwafxe1ODlLd9GbdlvpxVmh7d0nEo/GqAIvZK36z5/WM OPX3svKslBG3RZCwkh7FucKcMfZ8Ozssif1Dia3kxLHmEIcgtk7UrRGOQ7UHJpqt44hR Ptp3T1o758X3W4sA+2v257UbJVIIBOgJK3bGldi7xZNq9e6ZoER9zS2k2VUK5Y08hpy+ XjDg== X-Gm-Message-State: AOAM531c1dpkwLiOMUngDt9t7vW4bMaXWvWSML3AObivo64S383WMIXC f0g3Srrd86UGpXkzNZtH2KE/fJALTQ8= X-Google-Smtp-Source: ABdhPJzjBtsO0/JTBlPO1xYXdzCcIzLTYGUoZ3VzIFkPZTQFKRlc4IvrgphVKB3fFVArfK0KiwwcZA== X-Received: by 2002:adf:e2cb:0:b0:203:7564:1076 with SMTP id d11-20020adfe2cb000000b0020375641076mr800372wrj.199.1646777752676; Tue, 08 Mar 2022 14:15:52 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w10-20020a5d680a000000b001f1dabec837sm77362wru.113.2022.03.08.14.15.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:52 -0800 (PST) Message-Id: <524d449ed64fab482cbe8eb1f325490ca3860440.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:22 +0000 Subject: [PATCH v2 22/27] fsmonitor: never set CE_FSMONITOR_VALID on submodules Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Never set CE_FSMONITOR_VALID on the cache-entry of submodule directories. During a client command like 'git status', we may need to recurse into each submodule to compute a status summary for the submodule. Since the purpose of the ce_flag is to let Git avoid scanning a cache-entry, setting the flag causes the recursive call to be avoided and we report incorrect (no status) for the submodule. We created an OS watch on the root directory of our working directory and we receive events for everything in the cone under it. When submodules are present inside our working directory, we receive events for both our repo (the super) and any subs within it. Since our index doesn't have any information for items within the submodules, we can't use those events. We could try to truncate the paths of those events back to the submodule boundary and mark the GITLINK as dirty, but that feels expensive since we would have to prefix compare every FS event that we receive against a list of submodule roots. And it still wouldn't be sufficient to correctly report status on the submodule, since we don't have any space in the cache-entry to cache the submodule's status (the 'SCMU' bits in porcelain V2 speak). That is, the CE_FSMONITOR_VALID bit just says that we don't need to scan/inspect it because we already know the answer -- it doesn't say that the item is clean -- and we don't have space in the cache-entry to store those answers. So we should always do the recursive scan. Therefore, we should never set the flag on GITLINK cache-entries. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 2 + fsmonitor.h | 11 +++++ t/t7527-builtin-fsmonitor.sh | 93 ++++++++++++++++++++++++++++++++++++ 3 files changed, 106 insertions(+) diff --git a/fsmonitor.c b/fsmonitor.c index e1229c289cf..57d6a483bee 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -580,6 +580,8 @@ void tweak_fsmonitor(struct index_state *istate) if (fsmonitor_enabled) { /* Mark all entries valid */ for (i = 0; i < istate->cache_nr; i++) { + if (S_ISGITLINK(istate->cache[i]->ce_mode)) + continue; istate->cache[i]->ce_flags |= CE_FSMONITOR_VALID; } diff --git a/fsmonitor.h b/fsmonitor.h index 3f41f653691..edf7ce5203b 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -68,6 +68,15 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate) * Set the given cache entries CE_FSMONITOR_VALID bit. This should be * called any time the cache entry has been updated to reflect the * current state of the file on disk. + * + * However, never mark submodules as valid. When commands like "git + * status" run they might need to recurse into the submodule (using a + * child process) to get a summary of the submodule state. We don't + * have (and don't want to create) the facility to translate every + * FS event that we receive and that happens to be deep inside of a + * submodule back to the submodule root, so we cannot correctly keep + * track of this bit on the gitlink directory. Therefore, we never + * set it on submodules. */ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce) { @@ -75,6 +84,8 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache if (fsm_mode > FSMONITOR_MODE_DISABLED && !(ce->ce_flags & CE_FSMONITOR_VALID)) { + if (S_ISGITLINK(ce->ce_mode)) + return; istate->cache_changed = 1; ce->ce_flags |= CE_FSMONITOR_VALID; trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name); diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 8b63067cbf9..a18e077d375 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -728,4 +728,97 @@ do ' done +# Test fsmonitor interaction with submodules. +# +# If we start the daemon in the super, it will see FS events for +# everything in the working directory cone and this includes any +# files/directories contained *within* the submodules. +# +# A `git status` at top level will get events for items within the +# submodule and ignore them, since they aren't named in the index +# of the super repo. This makes the fsmonitor response a little +# noisy, but it doesn't alter the correctness of the state of the +# super-proper. +# +# When we have submodules, `git status` normally does a recursive +# status on each of the submodules and adds a summary row for any +# dirty submodules. (See the "S..." bits in porcelain V2 output.) +# +# It is therefore important that the top level status not be tricked +# by the FSMonitor response to skip those recursive calls. + +my_match_and_clean () { + git -C super --no-optional-locks status --porcelain=v2 >actual.with && + git -C super --no-optional-locks -c core.fsmonitor=false \ + status --porcelain=v2 >actual.without && + test_cmp actual.with actual.without && + + git -C super/dir_1/dir_2/sub reset --hard && + git -C super/dir_1/dir_2/sub clean -d -f +} + +test_expect_success "Submodule" ' + test_when_finished "git -C super fsmonitor--daemon stop" && + + git init "super" && + echo x >super/file_1 && + echo y >super/file_2 && + echo z >super/file_3 && + mkdir super/dir_1 && + echo a >super/dir_1/file_11 && + echo b >super/dir_1/file_12 && + mkdir super/dir_1/dir_2 && + echo a >super/dir_1/dir_2/file_21 && + echo b >super/dir_1/dir_2/file_22 && + git -C super add . && + git -C super commit -m "initial super commit" && + + git init "sub" && + echo x >sub/file_x && + echo y >sub/file_y && + echo z >sub/file_z && + mkdir sub/dir_x && + echo a >sub/dir_x/file_a && + echo b >sub/dir_x/file_b && + mkdir sub/dir_x/dir_y && + echo a >sub/dir_x/dir_y/file_a && + echo b >sub/dir_x/dir_y/file_b && + git -C sub add . && + git -C sub commit -m "initial sub commit" && + + git -C super submodule add ../sub ./dir_1/dir_2/sub && + git -C super commit -m "add sub" && + + start_daemon super && + git -C super config core.fsmonitor true && + git -C super update-index --fsmonitor && + git -C super status && + + # Now run pairs of commands w/ and w/o FSMonitor while we make + # some dirt in the submodule and confirm matching output. + + # Completely clean status. + my_match_and_clean && + + # .M S..U + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_u && + my_match_and_clean && + + # .M S.M. + echo z >super/dir_1/dir_2/sub/dir_x/dir_y/foobar_m && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M S.M. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + my_match_and_clean && + + # .M SC.. + echo z >>super/dir_1/dir_2/sub/dir_x/dir_y/file_a && + git -C super/dir_1/dir_2/sub add . && + git -C super/dir_1/dir_2/sub commit -m "SC.." && + my_match_and_clean +' + test_done From patchwork Tue Mar 8 22:15:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774465 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2F9DC433EF for ; Tue, 8 Mar 2022 22:16:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350716AbiCHWRd (ORCPT ); Tue, 8 Mar 2022 17:17:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52114 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350714AbiCHWQ7 (ORCPT ); Tue, 8 Mar 2022 17:16:59 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3FA78580FD for ; Tue, 8 Mar 2022 14:15:55 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id r10so271093wrp.3 for ; Tue, 08 Mar 2022 14:15:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4+KRgNH1UC8Hq1Q1521pi3Q8bCSKs8SjV1oI5ePqdS4=; b=ANd79n6FDu04bhyUPcD2cBU6yG6M8lbMG2cXX87NNXUwlrnP/WudNHKmbUrt2rQp4e 5joz0C7CxyMcYEetyDACp5HNC2BZpOOu6XLfmgmf3fhQAnlrLddQF1qxI0fKvEpSaKcU wVxbgpeR9ttz13jQDI6BQ2gD2x91B9i7REE3HwBhxPGLmk/lVCXcDnOX5UWVh+5MXk8f 3UkixrjDz+eAhoHtFTgxQ/nvcGzgNoA0PN64iLL/Bc9Yck+OibPFpdmjxjlKWyEuLILt D9BBdbnN7TpspfyIh1R0sr7uTqTvDRZ6bnIjKJgcJ4Zqn7UMz+lamj6wXtyf6qENt6ot zlOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4+KRgNH1UC8Hq1Q1521pi3Q8bCSKs8SjV1oI5ePqdS4=; b=cn9EQjn6NgOYHlYigE5E+IAlxcV0aZDiuG7cp0KJvs16snljcPQQxSzunDGzfwGR/k f2Q7R7dsUUtv1soQ1Ky7X8SKte5g0bUduHvGt04t4ZSmdjFB0YEeUBIcfUTyIVrVibmC bUK6NR5OjJrIK/VE0mcc1SGTr0/SD/a8Ojyi9sdmi3sy0Xvwxbnu3K8pCTcghKEDLBJ7 KncnYCIoLX9cm0LgqmazCen9ByHaPOIY9LZ822wqwVWZ7lL725oeBRD1gZNzhlmLUd2m ZI1CqcPOy1kgi39smLrs8pIAC5MF7vgMyz3dk1oeYvj/qqlFPS3jnkST66UqRpSAtcL8 +rHA== X-Gm-Message-State: AOAM532nM1iw0C195AUvsNkatXIEy2/B+nvWpxtGVLQPRVkrf1WC6Vz+ cgf4Qm7Zf82B2VEmEiFTFKV+FyJ5fjA= X-Google-Smtp-Source: ABdhPJw/0VsWL/w9c4FrJaTbi/q0yTXIj8qKWfc1KnCP3Bk4sCZkbIulYG5yI6uHYMkWB4E0sQPmRw== X-Received: by 2002:a05:6000:1ac5:b0:1ea:7870:d7eb with SMTP id i5-20020a0560001ac500b001ea7870d7ebmr14091157wry.171.1646777753604; Tue, 08 Mar 2022 14:15:53 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a10-20020a7bc1ca000000b00389bc87db45sm76187wmj.7.2022.03.08.14.15.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:53 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:23 +0000 Subject: [PATCH v2 23/27] t7527: test FSMonitor on case insensitive+preserving file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Test that FS events from the OS are received using the preserved, on-disk spelling of files/directories rather than spelling used to make the change. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 40 ++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index a18e077d375..a41e37236b5 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -821,4 +821,44 @@ test_expect_success "Submodule" ' my_match_and_clean ' +# On a case-insensitive file system, confirm that the daemon +# notices when the .git directory is moved/renamed/deleted +# regardless of how it is spelled in the the FS event. +# That is, does the FS event receive the spelling of the +# operation or does it receive the spelling preserved with +# the file/directory. +# +test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' +# test_when_finished "stop_daemon_delete_repo test_insensitive" && + + git init test_insensitive && + ( + GIT_TRACE_FSMONITOR="$(pwd)/insensitive.trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon test_insensitive + ) && + + mkdir -p test_insensitive/abc/def && + echo xyz >test_insensitive/ABC/DEF/xyz && + + test_path_is_dir test_insensitive/.git && + test_path_is_dir test_insensitive/.GIT && + + # Rename .git using an alternate spelling to verify that that + # daemon detects it and automatically shuts down. + mv test_insensitive/.GIT test_insensitive/.FOO && + sleep 1 && + mv test_insensitive/.FOO test_insensitive/.git && + test_must_fail git -C test_insensitive fsmonitor--daemon status && + + # Verify that events were reported using on-disk spellings of the + # directories and files that we touched. We may or may not get a + # trailing slash on modified directories. + # + egrep "^event: abc/?$" ./insensitive.trace && + egrep "^event: abc/def/?$" ./insensitive.trace && + egrep "^event: abc/def/xyz$" ./insensitive.trace +' + test_done From patchwork Tue Mar 8 22:15:24 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774461 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D68B2C433EF for ; Tue, 8 Mar 2022 22:16:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350700AbiCHWRZ (ORCPT ); Tue, 8 Mar 2022 17:17:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52184 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350716AbiCHWQ7 (ORCPT ); Tue, 8 Mar 2022 17:16:59 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0A2AB58388 for ; Tue, 8 Mar 2022 14:15:56 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id q7-20020a7bce87000000b00382255f4ca9so2281055wmj.2 for ; Tue, 08 Mar 2022 14:15:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fZaRUHCUQGI6ja6oRWsed7mGTlu+pk0Joi+yuifJOlM=; b=G9tkxh9qviPRoPYfGE7Opg4k1JlR/qzqDDtLIwszMhyOwgZUvjufQ7O3JgrvO7GMGP 94paTR/8Xy8YPhab52aHswYHd6X4Wiz0bXuBqJVbHbPBSc9cWjow7uDesaRS3C/ycfC1 vA5dX7mQPzC4T4RzG7KY9WF+MY0MIFg2zK2yMFG3PD2Ig2GF36fSlfAMZp8mYAk9N6iR YRUqJlKY2ycdKS8xL0bE4hYNMlVdsyV5OiVHFjaaXa23ZW8jsdXPy9EB8QeaJkZGCUKL wCDg3EBf6YCc1LYDCWacK5HhlZUhZERklPh52QBaw5+VcP+1fjBdB9TxFy/E7QPaZtxN HMJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fZaRUHCUQGI6ja6oRWsed7mGTlu+pk0Joi+yuifJOlM=; b=FOZcwz96NSFElZ3pjOQBBDpEXranuoLJvTsMctDGLQwx8iRKemkzn1t3t54qMk6Mt8 JUTgdPfKc/HExACwQL88/SM65Uauj6d78e7u8MItZjNZ50j2sH0nQ0FU57i+ex1wu3+a HI3qWbJWtUKjSm+W+sxjpc2MHU24ZvlOWg4b2T1Tuu3O3c3BGhYjOvr2Jqdxr3PPxzAS 45BPpvU6B8sIDxrx+Inz4RoKQ58YOInNM01jBJE4z2wdsDqLZL/px1gzdhy4dP/ztYAn 1m29YOjhzd0OHxDhdxmtO9XgvyO+VdqLdCYFftmFaeXUutd05XHucjUQsenl15Xu4qyU XPQw== X-Gm-Message-State: AOAM5321vqFZIs4W6j/2mcaEmx38Z/6k0XEid96LJI5JhkOhrmpaeE7g OGUA7m7v9ORQBymoCmFSENG36DtL4nQ= X-Google-Smtp-Source: ABdhPJwR2dgTIKkMhAWdunLCbT4XbfZBmYfBhes5irldDW0x/WmtVIau4g2dyIQImztDjISzPoPCoQ== X-Received: by 2002:a7b:cd82:0:b0:389:77ef:66d7 with SMTP id y2-20020a7bcd82000000b0038977ef66d7mr1020221wmj.171.1646777754402; Tue, 08 Mar 2022 14:15:54 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bi9-20020a05600c3d8900b00389a623c80esm73868wmb.1.2022.03.08.14.15.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:54 -0800 (PST) Message-Id: <95b9d4210d2c888689d2d6ffd60adb4c43d03bcc.1646777727.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:24 +0000 Subject: [PATCH v2 24/27] fsmonitor: on macOS also emit NFC spelling for NFD pathname Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Emit NFC or NFC and NFD spellings of pathnames on macOS. MacOS is Unicode composition insensitive, so NFC and NFD spellings are treated as aliases and collide. While the spelling of pathnames in filesystem events depends upon the underlying filesystem, such as APFS, HFS+ or FAT32, the OS enforces such collisions regardless of filesystem. Teach the daemon to always report the NFC spelling and to report the NFD spelling when stored in that format on the disk. This is slightly more general than "core.precomposeUnicode". Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-listen-darwin.c | 33 ++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index b6c33f2cf3b..3332d3b7792 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -227,6 +227,35 @@ static int ef_ignore_xattr(const FSEventStreamEventFlags ef) return ((ef & mask) == kFSEventStreamEventFlagItemXattrMod); } +/* + * On MacOS we have to adjust for Unicode composition insensitivity + * (where NFC and NFD spellings are not respected). The different + * spellings are essentially aliases regardless of how the path is + * actually stored on the disk. + * + * This is related to "core.precomposeUnicode" (which wants to try + * to hide NFD completely and treat everything as NFC). Here, we + * don't know what the value the client has (or will have) for this + * config setting when they make a query, so assume the worst and + * emit both when the OS gives us an NFD path. + */ +static void my_add_path(struct fsmonitor_batch *batch, const char *path) +{ + char *composed; + + /* add the NFC or NFD path as received from the OS */ + fsmonitor_batch__add_path(batch, path); + + /* if NFD, also add the corresponding NFC spelling */ + composed = (char *)precompose_string_if_needed(path); + if (!composed || composed == path) + return; + + fsmonitor_batch__add_path(batch, composed); + free(composed); +} + + static void fsevent_callback(ConstFSEventStreamRef streamRef, void *ctx, size_t num_of_events, @@ -377,7 +406,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, rel); + my_add_path(batch, rel); } if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { @@ -390,7 +419,7 @@ static void fsevent_callback(ConstFSEventStreamRef streamRef, if (!batch) batch = fsmonitor_batch__new(); - fsmonitor_batch__add_path(batch, tmp.buf); + my_add_path(batch, tmp.buf); } break; From patchwork Tue Mar 8 22:15:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774466 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC1C4C433FE for ; Tue, 8 Mar 2022 22:16:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350706AbiCHWRf (ORCPT ); Tue, 8 Mar 2022 17:17:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51500 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350724AbiCHWRA (ORCPT ); Tue, 8 Mar 2022 17:17:00 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EDCC758390 for ; Tue, 8 Mar 2022 14:15:56 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id r10so271170wrp.3 for ; Tue, 08 Mar 2022 14:15:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=mvnJNku/ci6QCsp+2rYRjooUfFEeT0Zt7IFMOmtffic=; b=KiK6Xnrx5tcrRCkxoWdx9nvv/sRLIJMeuYHZ3GNqQuNSwDsKwLl0NGgvYccPZhqj1c 15CNQWuldzJk0HK8Y3yoeaYSPae2Aa4j9jJqyc0AOudmO8Ksc3RsMi2VC3PoMrikJyzp MGJKa5PmMMULbIPXB41DnVU+BHzPJZbcJQnOP+NHC/eyjm+I2Ls8F3CEVW+HzBkH8gOx s9Ekf4gQGEWDu1CpxB2+U3nQf50BZkVugq+VKrsTfzFV/mBM7qDe6h+gJsqfLPVRIrcy DwAFcxBzearMeVOxWwmMOXc+m9IJqyES0PxbYVjrvAKSHKQKMwRjChjd4L87Tn1SlxsS /TXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=mvnJNku/ci6QCsp+2rYRjooUfFEeT0Zt7IFMOmtffic=; b=qB+Vg1glYUEEiDF/4eY/EfLeykf1f8g+u+QeUzemsiEFPD4zXtm4zV8OQisxfQNz8J mjKguZZfQSqXGtPp8igbzS6B+NWbEHPiist5GIutRthyFTnG7L/FU0OoFeu67anwghI2 Lgd7bK4OPQIkrJ+coVIr4TSJk51aFoi4XTBc8D//75EEacDt3Y41iXc01SHBCX9afqz/ x+QN/VzWilw0fmiw9DIaEf0HXNeDvxAvep/y612+hMfHwxcbEb9AwWm4vw6TUSAKGS1C E54ho/ZiHEMrpneEVuutbTNpL1DvNOrmxk/NWnZp7Xkk99tlhZoSjgg8bIQ6F+djm1t6 i9ng== X-Gm-Message-State: AOAM531S5qxHeP1FNrE87BLI9gwU47JQM6fbL+DmKsYQOeieeUFAS9Xj IsLMd7m6HUcfxy3BIwAjZM/WqGxYLM4= X-Google-Smtp-Source: ABdhPJxW8xyLCO4cr97JqU8eTI/Z/MQKlTPClWvayiz9UtDx2h8iAtxCh0DcW9hLohQVha18zGznVg== X-Received: by 2002:a5d:6c61:0:b0:203:70ec:e77c with SMTP id r1-20020a5d6c61000000b0020370ece77cmr3129349wrz.692.1646777755353; Tue, 08 Mar 2022 14:15:55 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f11-20020a7bcc0b000000b0037e0c362b6dsm62079wmh.31.2022.03.08.14.15.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:54 -0800 (PST) Message-Id: <5a0c1b7a2873accc6db4b34493962378819eacd4.1646777728.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:25 +0000 Subject: [PATCH v2 25/27] t/lib-unicode-nfc-nfd: helper prereqs for testing unicode nfc/nfd Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a set of prereqs to help understand how file names are handled by the filesystem when they contain NFC and NFD Unicode characters. Signed-off-by: Jeff Hostetler --- t/lib-unicode-nfc-nfd.sh | 159 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100755 t/lib-unicode-nfc-nfd.sh diff --git a/t/lib-unicode-nfc-nfd.sh b/t/lib-unicode-nfc-nfd.sh new file mode 100755 index 00000000000..a09e910c302 --- /dev/null +++ b/t/lib-unicode-nfc-nfd.sh @@ -0,0 +1,159 @@ +# Help detect how Unicode NFC and NFD are handled on the filesystem. + +# A simple character that has a NFD form. +# +# NFC: U+00e9 LATIN SMALL LETTER E WITH ACUTE +# UTF8(NFC): \xc3 \xa9 +# +# NFD: U+0065 LATIN SMALL LETTER E +# U+0301 COMBINING ACUTE ACCENT +# UTF8(NFD): \x65 + \xcc \x81 +# +utf8_nfc=$(printf "\xc3\xa9") +utf8_nfd=$(printf "\x65\xcc\x81") + +# Is the OS or the filesystem "Unicode composition sensitive"? +# +# That is, does the OS or the filesystem allow files to exist with +# both the NFC and NFD spellings? Or, does the OS/FS lie to us and +# tell us that the NFC and NFD forms are equivalent. +# +# This is or may be independent of what type of filesystem we have, +# since it might be handled by the OS at a layer above the FS. +# Testing shows on MacOS using APFS, HFS+, and FAT32 reports a +# collision, for example. +# +# This does not tell us how the Unicode pathname will be spelled +# on disk, but rather only that the two spelling "collide". We +# will examine the actual on disk spelling in a later prereq. +# +test_lazy_prereq UNICODE_COMPOSITION_SENSITIVE ' + mkdir trial_${utf8_nfc} && + mkdir trial_${utf8_nfd} +' + +# Is the spelling of an NFC pathname preserved on disk? +# +# On MacOS with HFS+ and FAT32, NFC paths are converted into NFD +# and on APFS, NFC paths are preserved. As we have established +# above, this is independent of "composition sensitivity". +# +# 0000000 63 5f c3 a9 +# +# (/usr/bin/od output contains different amount of whitespace +# on different platforms, so we need the wildcards here.) +# +test_lazy_prereq UNICODE_NFC_PRESERVED ' + mkdir c_${utf8_nfc} && + ls | od -t x1 | grep "63 *5f *c3 *a9" +' + +# Is the spelling of an NFD pathname preserved on disk? +# +# 0000000 64 5f 65 cc 81 +# +test_lazy_prereq UNICODE_NFD_PRESERVED ' + mkdir d_${utf8_nfd} && + ls | od -t x1 | grep "64 *5f *65 *cc *81" +' + mkdir c_${utf8_nfc} && + mkdir d_${utf8_nfd} && + +# The following _DOUBLE_ forms are more for my curiosity, +# but there may be quirks lurking when there are multiple +# combining characters in non-canonical order. + +# Unicode also allows multiple combining characters +# that can be decomposed in pieces. +# +# NFC: U+1f67 GREEK SMALL LETTER OMEGA WITH DASIA AND PERISPOMENI +# UTF8(NFC): \xe1 \xbd \xa7 +# +# NFD1: U+1f61 GREEK SMALL LETTER OMEGA WITH DASIA +# U+0342 COMBINING GREEK PERISPOMENI +# UTF8(NFD1): \xe1 \xbd \xa1 + \xcd \x82 +# +# But U+1f61 decomposes into +# NFD2: U+03c9 GREEK SMALL LETTER OMEGA +# U+0314 COMBINING REVERSED COMMA ABOVE +# UTF8(NFD2): \xcf \x89 + \xcc \x94 +# +# Yielding: \xcf \x89 + \xcc \x94 + \xcd \x82 +# +# Note that I've used the canonical ordering of the +# combinining characters. It is also possible to +# swap them. My testing shows that that non-standard +# ordering also causes a collision in mkdir. However, +# the resulting names don't draw correctly on the +# terminal (implying that the on-disk format also has +# them out of order). +# +greek_nfc=$(printf "\xe1\xbd\xa7") +greek_nfd1=$(printf "\xe1\xbd\xa1\xcd\x82") +greek_nfd2=$(printf "\xcf\x89\xcc\x94\xcd\x82") + +# See if a double decomposition also collides. +# +test_lazy_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE ' + mkdir trial_${greek_nfc} && + mkdir trial_${greek_nfd2} +' + +# See if the NFC spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFC_PRESERVED ' + mkdir c_${greek_nfc} && + ls | od -t x1 | grep "63 *5f *e1 *bd *a7" +' + +# See if the NFD spelling appears on the disk. +# +test_lazy_prereq UNICODE_DOUBLE_NFD_PRESERVED ' + mkdir d_${greek_nfd2} && + ls | od -t x1 | grep "64 *5f *cf *89 *cc *94 *cd *82" +' + +if test $unicode_debug = 1 +then + if test_have_prereq UNICODE_COMPOSITION_SENSITIVE + then + echo NFC and NFD are distinct on this OS/filesystem. + else + echo NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_NFC_PRESERVED + then + echo NFC maintains original spelling. + else + echo NFC is modified. + fi + + if test_have_prereq UNICODE_NFD_PRESERVED + then + echo NFD maintains original spelling. + else + echo NFD is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_COMPOSITION_SENSITIVE + then + echo DOUBLE NFC and NFD are distinct on this OS/filesystem. + else + echo DOUBLE NFC and NFD are aliases on this OS/filesystem. + fi + + if test_have_prereq UNICODE_DOUBLE_NFC_PRESERVED + then + echo Double NFC maintains original spelling. + else + echo Double NFC is modified. + fi + + if test_have_prereq UNICODE_DOUBLE_NFD_PRESERVED + then + echo Double NFD maintains original spelling. + else + echo Double NFD is modified. + fi +fi From patchwork Tue Mar 8 22:15:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774468 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DA30C433EF for ; Tue, 8 Mar 2022 22:16:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350730AbiCHWRj (ORCPT ); Tue, 8 Mar 2022 17:17:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52090 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350723AbiCHWRA (ORCPT ); Tue, 8 Mar 2022 17:17:00 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A924658394 for ; Tue, 8 Mar 2022 14:15:57 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id t11so256853wrm.5 for ; Tue, 08 Mar 2022 14:15:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RuvnNt5XyOl7a++pYSyJu9F102IfAfVCdDUqPCvNqSk=; b=Ccs6giRyHYyFfeX2r5DCMK01PDbMSGODWcpQE15oGYLflI5c5icrW+vy+9XtvuZET5 f9wAPARl34AZUizsAs0MEHiQEFcwwaTZ0GLn6qesy7XFyeIIFIimXQWmiKzA6DVNFtrC SpnH7QoB2PYFes2K9Nv/37wguBtsg8638O2peXHZ//FISKlMfnZBgnCnJXckCM/tMQg7 c0JyXLQqA3fcg1A8Q1CWaWbWCpPNQa8HHOFoEgbWkWR9ntnunLzGYztdQbRabCQvp49c rpao5QVcghhxza0CgnlOr1dQ2kbAEee5AYj1z4Q2+lVi9WHUV4XkKEk/LN3P37QxTFWo sSBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RuvnNt5XyOl7a++pYSyJu9F102IfAfVCdDUqPCvNqSk=; b=VQ8BM8ELBoncCepyheEuQj0HEMutdZoVT2eiE9FVnJr4Y9AYdHUpIQXSQiInymfztQ KekfYLIdVlibBy8SLhIY9fR8hfFSFuR25BqkAPlZSFRkOhd8w/xXdxHs5adt1PF2zfKN WvmrIIZTcgjUWlaFiU2J/iTOtXDxiHa/7tdX0IJggGcYwX8CmZU8hyqWssIaHkuSmu0e 9Njne6Z2JagEiygekGJH8hv1oRCXSUHAq4eLPTSdfQXSLQEbs/wL8MwJ5fQu7DtvbNPi H1LBCxoCkWh514qz/tG1s+hSIAFxddhc1DYa31qULriKkzikMRsDwxFqEJVnEQy+tsEg 3ooA== X-Gm-Message-State: AOAM531TutEZvn8YpbBGQWTalqJwP8j7KnPIrBWraBIO2Ngg/4ABLqPD f5ZRxMeG+85pKylX06SwbzXqtoeucDA= X-Google-Smtp-Source: ABdhPJwRyM2mu0LPDeo/747efWHhGyPN2hqs9QGQ5Ki5Rq5uk7osC/vta73rled2lmwW8Xgcl4CJXw== X-Received: by 2002:adf:dfc2:0:b0:1f0:262a:d831 with SMTP id q2-20020adfdfc2000000b001f0262ad831mr13036443wrn.589.1646777756151; Tue, 08 Mar 2022 14:15:56 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e20-20020adfa454000000b001f01a14dce8sm86147wra.97.2022.03.08.14.15.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:55 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:26 +0000 Subject: [PATCH v2 26/27] t7527: test Unicode NFC/NFD handling on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Confirm that the daemon reports events using the on-disk spelling for Unicode NFC/NFD characters. On APFS we still have Unicode aliasing, so we cannot create two files that only differ by NFC/NFD, but the on-disk format preserves the spelling used to create the file. On HFS+ we also have aliasing, but the path is always stored on disk in NFD. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 54 ++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index a41e37236b5..48c9125d8da 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -861,4 +861,58 @@ test_expect_success CASE_INSENSITIVE_FS 'case insensitive+preserving' ' egrep "^event: abc/def/xyz$" ./insensitive.trace ' +unicode_debug=0 +. "$TEST_DIRECTORY/lib-unicode-nfc-nfd.sh" + +# See if the OS or filesystem does NFC/NFD aliasing/munging. +# +# The daemon should err on the side of caution and send BOTH the +# NFC and NFD forms. It does not know the original spelling of +# the pathname (how the user thinks it should be spelled), so +# emit both and let the client decide (when necessary). This is +# similar to "core.precomposeUnicode". +# +test_expect_success !UNICODE_COMPOSITION_SENSITIVE 'Unicode nfc/nfd' ' + test_when_finished "stop_daemon_delete_repo test_unicode" && + + git init test_unicode && + ( + GIT_TRACE_FSMONITOR="$(pwd)/unicode.trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon test_unicode + ) && + + # Create a directory using an NFC spelling. + # + mkdir test_unicode/nfc && + mkdir test_unicode/nfc/c_${utf8_nfc} && + + # Create a directory using an NFD spelling. + # + mkdir test_unicode/nfd && + mkdir test_unicode/nfd/d_${utf8_nfd} && + + git -C test_unicode fsmonitor--daemon stop && + + if test_have_prereq UNICODE_NFC_PRESERVED + then + # We should have seen NFC event from OS. + # We should not have synthesized an NFD event. + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace && + egrep -v "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace + else + # We should have seen NFD event from OS. + # We should have synthesized an NFC event. + egrep "^event: nfc/c_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfc/c_${utf8_nfc}/?$" ./unicode.trace + fi && + + # We assume UNICODE_NFD_PRESERVED. + # We should have seen explicit NFD from OS. + # We should have synthesized an NFC event. + egrep "^event: nfd/d_${utf8_nfd}/?$" ./unicode.trace && + egrep "^event: nfd/d_${utf8_nfc}/?$" ./unicode.trace +' + test_done From patchwork Tue Mar 8 22:15:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12774467 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CBD96C433F5 for ; Tue, 8 Mar 2022 22:16:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350707AbiCHWRg (ORCPT ); Tue, 8 Mar 2022 17:17:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350727AbiCHWRA (ORCPT ); Tue, 8 Mar 2022 17:17:00 -0500 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD2D4583A8 for ; Tue, 8 Mar 2022 14:15:58 -0800 (PST) Received: by mail-wr1-x42f.google.com with SMTP id u1so221511wrg.11 for ; Tue, 08 Mar 2022 14:15:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ckbNLnLtLBb+cunwvS4Wd9AD0GNuyCgcxOjizu4gyUo=; b=iksnshXPPmFLAp5HwfoUJCF3YhYBS7BuVPIpgChUiKEBRspjFzuT9E0PVSJzGEgJQT NgVLVqmw3+XwPBNAq+TAzhZRDU2C/7HOweO6NWqsHZd44LX7LRxwkZEvFy/6OjacbEEg V4wonuxRQrGiJ2Zm9zSU7TFZlJuFGz3Q+vQcDkPtm+paEsQNupT3WWnbfOwg+59x5uS0 +FiNz1RS5N/vaiA0h8yl1CjvdXGdt4QLAC4mF2srUJ5tXchwEFrqMdvl2wSicAvaw8yY ZqItOLxUOhe/U33DyTXFJ+Kf4en/Zvd9Ey5hLx8aZOaY/GqCbPjl7sa1SlnPugk+zExi YcGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ckbNLnLtLBb+cunwvS4Wd9AD0GNuyCgcxOjizu4gyUo=; b=k9LWxNFw0T173vjxSuIQupt2xN1UFcjbGkgKEOhKgdlkpbUpySdB3npt+dW1XcF9sN CGMwkbFBFJjUZJQK4LlKQ60vg0h3CR1S+etzYauZwpK2fZ/afoBM3zYXUiZNint+FXZ3 lFTviSSYAi+1Ie5Fm7qe8WOSu0flAFOAFEhslqF5ydvsRPLb9jtCxXewFZvtSEJRe5DB 0/iIXHK5R13kjyteWYFn3gKL35CMndLXuSklfMCKPVFy5JfQUJth8zrTy73TLXCmStqL yOET+xCtgaQf93rBFzsSB+98DVS6rjZ0vzHypZWuiEwJWazR9DRcDZCXtA/ma1RJ3Ylq lSaQ== X-Gm-Message-State: AOAM532waS8uQmbjUOzxZBER050htHbaWaqkuL9YkQWeRZcGxVixyTHR MEd+jNLbuZVrCmGIDjgVGsprxrMTqng= X-Google-Smtp-Source: ABdhPJwyrK80nxTLx/H6EOiElp1zJhlmOol8noo1NokbrVLfylFqMdWf3DudD4H+FuTxr+YiZE8RoA== X-Received: by 2002:a5d:66c5:0:b0:1f1:d7e3:7068 with SMTP id k5-20020a5d66c5000000b001f1d7e37068mr13459825wrw.410.1646777757266; Tue, 08 Mar 2022 14:15:57 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p11-20020adf9d8b000000b001f063deef80sm84709wre.86.2022.03.08.14.15.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Mar 2022 14:15:56 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 08 Mar 2022 22:15:27 +0000 Subject: [PATCH v2 27/27] fsmonitor-settings: NTFS and FAT32 on MacOS are incompatible Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7A=?= Bjarmason , Torsten =?unknown-8bit?q?B?= =?unknown-8bit?q?=C3=B6gershausen?= , Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler On MacOS mark repos on NTFS or FAT32 volumes as incompatible. The builtin FSMonitor used Unix domain sockets on MacOS for IPC with clients. These sockets are kept in the .git directory. Unix sockets are not supported by NTFS and FAT32, so the daemon cannot start up. Test for this during our compatibility checking so that client commands do not keep trying to start the daemon. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsm-settings-darwin.c | 20 +++++++++++++++++--- fsmonitor-settings.c | 3 +++ fsmonitor-settings.h | 1 + 3 files changed, 21 insertions(+), 3 deletions(-) diff --git a/compat/fsmonitor/fsm-settings-darwin.c b/compat/fsmonitor/fsm-settings-darwin.c index fdd762bf79d..efc732c0f31 100644 --- a/compat/fsmonitor/fsm-settings-darwin.c +++ b/compat/fsmonitor/fsm-settings-darwin.c @@ -7,7 +7,7 @@ #include /* - * Remote working directories are problematic for FSMonitor. + * [1] Remote working directories are problematic for FSMonitor. * * The underlying file system on the server machine and/or the remote * mount type (NFS, SAMBA, etc.) dictates whether notification events @@ -40,8 +40,16 @@ * * So (for now at least), mark remote working directories as * incompatible. + * + * + * [2] FAT32 and NTFS working directories are problematic too. + * + * The builtin FSMonitor uses a Unix domain socket in the .git + * directory for IPC. These Windows drive formats do not support + * Unix domain sockets, so mark them as incompatible for the daemon. + * */ -static enum fsmonitor_reason check_remote(struct repository *r) +static enum fsmonitor_reason check_volume(struct repository *r) { struct statfs fs; @@ -60,6 +68,12 @@ static enum fsmonitor_reason check_remote(struct repository *r) if (!(fs.f_flags & MNT_LOCAL)) return FSMONITOR_REASON_REMOTE; + if (!strcmp(fs.f_fstypename, "msdos")) /* aka FAT32 */ + return FSMONITOR_REASON_NOSOCKETS; + + if (!strcmp(fs.f_fstypename, "ntfs")) + return FSMONITOR_REASON_NOSOCKETS; + return FSMONITOR_REASON_OK; } @@ -67,7 +81,7 @@ enum fsmonitor_reason fsm_os__incompatible(struct repository *r) { enum fsmonitor_reason reason; - reason = check_remote(r); + reason = check_volume(r); if (reason != FSMONITOR_REASON_OK) return reason; diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c index 7ff3f98964d..5734c93baf9 100644 --- a/fsmonitor-settings.c +++ b/fsmonitor-settings.c @@ -186,6 +186,9 @@ const char *fsm_settings__get_reason_msg(struct repository *r) case FSMONITOR_REASON_VFS4GIT: return _("virtual repos are incompatible with fsmonitor"); + + case FSMONITOR_REASON_NOSOCKETS: + return _("repo filesystem does not support Unix sockets"); } BUG("Unhandled case in fsm_settings__get_reason_msg '%d'", diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h index 6aa9a00379b..af792313413 100644 --- a/fsmonitor-settings.h +++ b/fsmonitor-settings.h @@ -19,6 +19,7 @@ enum fsmonitor_reason { FSMONITOR_REASON_ERROR, /* FS error probing for compatibility */ FSMONITOR_REASON_REMOTE, FSMONITOR_REASON_VFS4GIT, /* VFS for Git virtualization */ + FSMONITOR_REASON_NOSOCKETS, /* NTFS,FAT32 do not support Unix sockets */ }; void fsm_settings__set_ipc(struct repository *r);