From patchwork Thu Apr 1 15:40:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12179205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25958C433B4 for ; Thu, 1 Apr 2021 18:33:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 009C260FD9 for ; Thu, 1 Apr 2021 18:33:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237218AbhDASdC (ORCPT ); Thu, 1 Apr 2021 14:33:02 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37872 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235531AbhDASVK (ORCPT ); Thu, 1 Apr 2021 14:21:10 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F2823C0225B7 for ; Thu, 1 Apr 2021 08:41:09 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id z2so2273685wrl.5 for ; Thu, 01 Apr 2021 08:41:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XkpOGspoFjNNxmzEoEsBAlqtHAH0QW/MpCd/rP67Qng=; b=SgYwbTjShPntm0S6SejfPLVotaclpplcgKFEAmnBIzuiNTSy+Fcv+caVeGxtfkSJLI 9nKqZtlGhxRmhcn4WOkgL5Fmetrox48Os7c0kKfGnRN70j90v4JwHW6f4QlxNjom0hyF 5a3pmWQrZvLifvA2OUGv3bzEwU3PkDSBqrLsRIdzJpb52ue/38Deg+DgaPIP26YUL2tk Cdfq7FxMYniCNMaqP2PoLMm+CXrdDmzsq1TFoZQhy7zWRvhYSpfD+1aIhcv1vwtH/TXA NEj/hDDJ48nVEJCWeVFKAAxVuwXOHybwTSP1xm+pmhmCsGIR/yuDIovvaPcCR2u71YIT 6mcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XkpOGspoFjNNxmzEoEsBAlqtHAH0QW/MpCd/rP67Qng=; b=OqhYmQH2T5MHx6NCdKkTITdtMD65ZhCWQRndjFysJM2mlfgRosaICgGervXpMFPO9X xE5vSIR4mqHGdgsqngoVJKBkC81NKJal3Y43IgEME+KLLoA8XqJpR0U/i2XU3MXJFvVL qs7v4VAgFXyXM90qIgE71P+m5/9srmCXpfF8IMi4+EiY9x+tLwfZ4IHQnc001fn5mdXH yHQSzFzXgCNJux5V+4Lm03sHPxNdHhTh+awwXZCZ9hVrR1vGOqS5vOqB3f6iaPV18B/0 CHHZj4CqDkn5vtGkXkg7lJSjhHiMG4UF46odaAwdJgi6sqsmA1mYNkucOprI0AtQnxQp D1Pw== X-Gm-Message-State: AOAM531iq2fiiRTIfzBCNcwgYZzMJI8/DjWtUvVftCvPzUowYxVkWYZd cin4g6+5LYpZUE53sjOL0kln5uEuC6g= X-Google-Smtp-Source: ABdhPJzD1/lVzVi+Bta1xcGNim+FoI4Avto6w/s81CKCODZH+NUzZ0GXAZv97MWfGVZjZk8RehvxtA== X-Received: by 2002:a5d:4b06:: with SMTP id v6mr10706387wrq.41.1617291668673; Thu, 01 Apr 2021 08:41:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i4sm7976592wmq.12.2021.04.01.08.41.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:08 -0700 (PDT) Message-Id: <074273330f8d6c656dfec7c8778fad20314c6ad1.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:43 +0000 Subject: [PATCH 01/23] fsmonitor--daemon: man page and documentation Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a manual page describing the `git fsmonitor--daemon` feature. Update references to `core.fsmonitor`, `core.fsmonitorHookVersion` and pointers to `watchman` to mention the built-in FSMonitor. Signed-off-by: Jeff Hostetler --- Documentation/config/core.txt | 45 +++++++--- Documentation/git-fsmonitor--daemon.txt | 104 ++++++++++++++++++++++++ Documentation/git-update-index.txt | 4 +- Documentation/githooks.txt | 3 +- 4 files changed, 144 insertions(+), 12 deletions(-) create mode 100644 Documentation/git-fsmonitor--daemon.txt diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a15..d6e2f01966cb 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -66,18 +66,43 @@ core.fsmonitor:: will identify all files that may have changed since the requested date/time. This information is used to speed up git by avoiding unnecessary processing of files that have not changed. - See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +Note: FSMonitor hooks (and this config setting) are ignored if the +built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`). core.fsmonitorHookVersion:: - Sets the version of hook that is to be used when calling fsmonitor. - There are currently versions 1 and 2. When this is not set, - version 2 will be tried first and if it fails then version 1 - will be tried. Version 1 uses a timestamp as input to determine - which files have changes since that time but some monitors - like watchman have race conditions when used with a timestamp. - Version 2 uses an opaque string so that the monitor can return - something that can be used to determine what files have changed - without race conditions. + Sets the version of hook that is to be used when calling the + FSMonitor hook (as configured via `core.fsmonitor`). ++ +There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors +like watchman have race conditions when used with a timestamp. +Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions. ++ +Note: FSMonitor hooks (and this config setting) are ignored if the +built-in FSMonitor is enabled (see `core.useBuiltinFSMonitor`). + +core.useBuiltinFSMonitor:: + If set to true, enable the built-in filesystem event watcher (for + technical details, see linkgit:git-fsmonitor--daemon[1]). ++ +Like external (hook-based) FSMonitors, the built-in FSMonitor can speed up +Git commands that need to refresh the Git index (e.g. `git status`) in a +worktree with many files. The built-in FSMonitor facility eliminates the +need to install and maintain an external third-party monitoring tool. ++ +The built-in FSMonitor is currently available only on a limited set of +supported platforms. ++ +Note: if this config setting is set to `true`, any FSMonitor hook +configured via `core.fsmonitor` (and possibly `core.fsmonitorHookVersion`) +is ignored. core.trustctime:: If false, the ctime differences between the index and the diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt new file mode 100644 index 000000000000..b94f57c97fe4 --- /dev/null +++ b/Documentation/git-fsmonitor--daemon.txt @@ -0,0 +1,104 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - Builtin file system monitor daemon + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' --start +'git fsmonitor--daemon' --run +'git fsmonitor--daemon' --stop +'git fsmonitor--daemon' --is-running +'git fsmonitor--daemon' --is-supported +'git fsmonitor--daemon' --query +'git fsmonitor--daemon' --query-index +'git fsmonitor--daemon' --flush + +DESCRIPTION +----------- + +Monitors files and directories in the working directory for changes using +platform-specific file system notification facilities. + +It communicates directly with commands like `git status` using the +link:technical/api-simple-ipc.html[simple IPC] interface instead of +the slower linkgit:githooks[5] interface. + +OPTIONS +------- + +--start:: + Starts the fsmonitor daemon in the background. + +--run:: + Runs the fsmonitor daemon in the foreground. + +--stop:: + Stops the fsmonitor daemon running for the current working + directory, if present. + +--is-running:: + Exits with zero status if the fsmonitor daemon is watching the + current working directory. + +--is-supported:: + Exits with zero status if the fsmonitor daemon feature is supported + on this platform. + +--query :: + Connects to the fsmonitor daemon (starting it if necessary) and + requests the list of changed files and directories since the + given token. + This is intended for testing purposes. + +--query-index:: + Read the current `` from the File System Monitor index + extension (if present) and use it to query the fsmonitor daemon. + This is intended for testing purposes. + +--flush:: + Force the fsmonitor daemon to flush its in-memory cache and + re-sync with the file system. + This is intended for testing purposes. + +REMARKS +------- +The fsmonitor daemon is a long running process that will watch a single +working directory. Commands, such as `git status`, should automatically +start it (if necessary) when `core.useBuiltinFSMonitor` is set to `true` +(see linkgit:git-config[1]). + +Configure the built-in FSMonitor via `core.useBuiltinFSMonitor` in each +working directory separately, or globally via `git config --global +core.useBuiltinFSMonitor true`. + +Tokens are opaque strings. They are used by the fsmonitor daemon to +mark a point in time and the associated internal state. Callers should +make no assumptions about the content of the token. In particular, +the should not assume that it is a timestamp. + +Query commands send a request-token to the daemon and it responds with +a summary of the changes that have occurred since that token was +created. The daemon also returns a response-token that the client can +use in a future query. + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and does +not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +should properly ignore these extra events, so performance may be affected +but it should not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt index 2853f168d976..8169aad7ee9f 100644 --- a/Documentation/git-update-index.txt +++ b/Documentation/git-update-index.txt @@ -498,7 +498,9 @@ FILE SYSTEM MONITOR This feature is intended to speed up git operations for repos that have large working directories. -It enables git to work together with a file system monitor (see the +It enables git to work together with a file system monitor (see +linkgit:git-fsmonitor--daemon[1] +and the "fsmonitor-watchman" section of linkgit:githooks[5]) that can inform it as to what files have been modified. This enables git to avoid having to lstat() every file to find modified files. diff --git a/Documentation/githooks.txt b/Documentation/githooks.txt index b51959ff9418..b7d5e926f7b0 100644 --- a/Documentation/githooks.txt +++ b/Documentation/githooks.txt @@ -593,7 +593,8 @@ fsmonitor-watchman This hook is invoked when the configuration option `core.fsmonitor` is set to `.git/hooks/fsmonitor-watchman` or `.git/hooks/fsmonitor-watchmanv2` -depending on the version of the hook to use. +depending on the version of the hook to use, unless overridden via +`core.useBuiltinFSMonitor` (see linkgit:git-config[1]). Version 1 takes two arguments, a version (1) and the time in elapsed nanoseconds since midnight, January 1, 1970. From patchwork Thu Apr 1 15:40:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CE84C4363E for ; Thu, 1 Apr 2021 17:44:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2968761383 for ; Thu, 1 Apr 2021 17:44:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235720AbhDARnI (ORCPT ); Thu, 1 Apr 2021 13:43:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57200 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234536AbhDARh4 (ORCPT ); Thu, 1 Apr 2021 13:37:56 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FBAFC0225B8 for ; Thu, 1 Apr 2021 08:41:10 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id x13so2256179wrs.9 for ; Thu, 01 Apr 2021 08:41:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=nXaGyCwufbowQyx7IWi+icYC2uFsuA2bq1r9jyL4u28=; b=jqHEzr0eNE0jErZPk0nCnCxSdo5hRdyfDqHWudoVsC3nMZK5vNjwWRTw9ar8s77H3c n85fGCrd4w8nvNpF8JB6LtywVElUVsqpVWbv7v+2oiD0Xxi6ItN9Q9XBGpiHSvC+HF/h 5Vv9hrAZ4Pb+WbEDKC1lXh17LfZuHIgez+FSbav6NCpDbj1hA88GJWmD5thm4t4GPwWE P62IqB/GtjdlFWwkZ9FjHr5cL/QB3IoqRRFmUW0HnGm1IY8J25J9Vf+dDdL4Z8oQCGKQ KJIVILeE1vKC9nRntnl5ETVJRV9toLFzk2KG6SfsZHGFGMXePNSuuSFs/EDezD5bfnM0 albg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=nXaGyCwufbowQyx7IWi+icYC2uFsuA2bq1r9jyL4u28=; b=SIrXQnt6R4fiHkBm0zD8eSLB8CMxGWPpxOSvoAWP+lJlwm3YkxE1X9LOPu9QA6OrsB MWr1Au9EXZGSvmgP884/Ho2k4Np9mCjxTTg0E6xCfQtZNnE6QJAOY3H05HnOVT6ynVEh VeW9qqLYGQ9iCEz7hxGT9LYzSBv/L0qv+WX2dSXELkqbngmSaQNYlR8BUP8vokcMdQQ9 43m4+uWj7XLVnKKOQhPC344Cz2fy+vYG/ZZbRxtD66NCzasf9MKkxffQTmWx2//injzr /cr8oKlu55xOTh2xFDUBAKy9SF14iUdD8R5Lg6G9c8O7dvjBncycoT0GFPE8JG5SnDkF 1ptA== X-Gm-Message-State: AOAM533pdFyC5ZnB7pfjlCqSTdAuBogOYQqoJTNVnCoF1Evm3rQ8xjkW Vs/rxT82OqWfNM0O11ZY8feWdLhzzrY= X-Google-Smtp-Source: ABdhPJzt+MHmEtuTXM6rzczAM50dvv6wd9ThgGxcRN4OvlusEgqsR1lZabZl2I41PqG+kud0WmULNQ== X-Received: by 2002:adf:fcc7:: with SMTP id f7mr10533138wrs.400.1617291669339; Thu, 01 Apr 2021 08:41:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w22sm9500129wmi.22.2021.04.01.08.41.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:09 -0700 (PDT) Message-Id: <3dac63eae201e6d0b949680e682238625cad59bd.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:44 +0000 Subject: [PATCH 02/23] fsmonitor-ipc: create client routines for git-fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create client routines to spawn a fsmonitor daemon and send it an IPC request using `simple-ipc`. Signed-off-by: Jeff Hostetler --- Makefile | 1 + fsmonitor-ipc.c | 153 ++++++++++++++++++++++++++++++++++++++++++++++++ fsmonitor-ipc.h | 48 +++++++++++++++ help.c | 4 ++ 4 files changed, 206 insertions(+) create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h diff --git a/Makefile b/Makefile index a6a73c574191..50977911d41a 100644 --- a/Makefile +++ b/Makefile @@ -891,6 +891,7 @@ LIB_OBJS += fetch-pack.o LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o +LIB_OBJS += fsmonitor-ipc.o LIB_OBJS += gettext.o LIB_OBJS += gpg-interface.o LIB_OBJS += graph.o diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c new file mode 100644 index 000000000000..b0dc334ff02d --- /dev/null +++ b/fsmonitor-ipc.c @@ -0,0 +1,153 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +#include "run-command.h" +#include "strbuf.h" +#include "trace2.h" + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND +#define FSMONITOR_DAEMON_IS_SUPPORTED 1 +#else +#define FSMONITOR_DAEMON_IS_SUPPORTED 0 +#endif + +/* + * A trivial function so that this source file always defines at least + * one symbol even when the feature is not supported. This quiets an + * annoying compiler error. + */ +int fsmonitor_ipc__is_supported(void) +{ + return FSMONITOR_DAEMON_IS_SUPPORTED; +} + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor") + +enum ipc_active_state fsmonitor_ipc__get_state(void) +{ + return ipc_get_active_state(fsmonitor_ipc__get_path()); +} + +static int spawn_daemon(void) +{ + const char *args[] = { "fsmonitor--daemon", "--start", NULL }; + + return run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD, + "fsmonitor"); +} + +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer) +{ + int ret = -1; + int tried_to_spawn = 0; + enum ipc_active_state state = IPC_STATE__OTHER_ERROR; + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + trace2_region_enter("fsm_client", "query", NULL); + + trace2_data_string("fsm_client", NULL, "query/command", + since_token); + +try_again: + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + + switch (state) { + case IPC_STATE__LISTENING: + ret = ipc_client_send_command_to_connection( + connection, since_token, answer); + ipc_client_close_connection(connection); + + trace2_data_intmax("fsm_client", NULL, + "query/response-length", answer->len); + + if (fsmonitor_is_trivial_response(answer)) + trace2_data_intmax("fsm_client", NULL, + "query/trivial-response", 1); + + goto done; + + case IPC_STATE__NOT_LISTENING: + ret = error(_("fsmonitor_ipc__send_query: daemon not available")); + goto done; + + case IPC_STATE__PATH_NOT_FOUND: + if (tried_to_spawn) + goto done; + + tried_to_spawn++; + if (spawn_daemon()) + goto done; + + /* + * Try again, but this time give the daemon a chance to + * actually create the pipe/socket. + * + * Granted, the daemon just started so it can't possibly have + * any FS cached yet, so we'll always get a trivial answer. + * BUT the answer should include a new token that can serve + * as the basis for subsequent requests. + */ + options.wait_if_not_found = 1; + goto try_again; + + case IPC_STATE__INVALID_PATH: + ret = error(_("fsmonitor_ipc__send_query: invalid path '%s'"), + fsmonitor_ipc__get_path()); + goto done; + + case IPC_STATE__OTHER_ERROR: + default: + ret = error(_("fsmonitor_ipc__send_query: unspecified error on '%s'"), + fsmonitor_ipc__get_path()); + goto done; + } + +done: + trace2_region_leave("fsm_client", "query", NULL); + + return ret; +} + +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer) +{ + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + int ret; + enum ipc_active_state state; + + strbuf_reset(answer); + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + if (state != IPC_STATE__LISTENING) { + die("fsmonitor--daemon is not running"); + return -1; + } + + ret = ipc_client_send_command_to_connection(connection, command, answer); + ipc_client_close_connection(connection); + + if (ret == -1) { + die("could not send '%s' command to fsmonitor--daemon", + command); + return -1; + } + + return 0; +} + +#endif diff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h new file mode 100644 index 000000000000..7d21c1260151 --- /dev/null +++ b/fsmonitor-ipc.h @@ -0,0 +1,48 @@ +#ifndef FSMONITOR_IPC_H +#define FSMONITOR_IPC_H + +/* + * Returns true if a filesystem notification backend is defined + * for this platform. This symbol must always be visible and + * outside of the HAVE_ ifdef. + */ +int fsmonitor_ipc__is_supported(void); + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND +#include "run-command.h" +#include "simple-ipc.h" + +/* + * Returns the pathname to the IPC named pipe or Unix domain socket + * where a `git-fsmonitor--daemon` process will listen. This is a + * per-worktree value. + */ +const char *fsmonitor_ipc__get_path(void); + +/* + * Try to determine whether there is a `git-fsmonitor--daemon` process + * listening on the IPC pipe/socket. + */ +enum ipc_active_state fsmonitor_ipc__get_state(void); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc + * and ask for the set of changed files since the given token. + * + * This DOES NOT use the hook interface. + * + * Spawn a daemon process in the background if necessary. + */ +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc and + * send a command verb. If no daemon is available, we DO NOT try to + * start one. + */ +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_IPC_H */ diff --git a/help.c b/help.c index 3c3bdec21356..e22ba1d246a5 100644 --- a/help.c +++ b/help.c @@ -11,6 +11,7 @@ #include "version.h" #include "refs.h" #include "parse-options.h" +#include "fsmonitor-ipc.h" struct category_description { uint32_t category; @@ -664,6 +665,9 @@ void get_version_info(struct strbuf *buf, int show_build_options) strbuf_addf(buf, "sizeof-size_t: %d\n", (int)sizeof(size_t)); strbuf_addf(buf, "shell-path: %s\n", SHELL_PATH); /* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */ + + if (fsmonitor_ipc__is_supported()) + strbuf_addstr(buf, "feature: fsmonitor--daemon\n"); } } From patchwork Thu Apr 1 15:40:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12179135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72E10C433B4 for ; Thu, 1 Apr 2021 18:22:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4B06D60551 for ; Thu, 1 Apr 2021 18:22:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234835AbhDASWM (ORCPT ); Thu, 1 Apr 2021 14:22:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236745AbhDASNz (ORCPT ); Thu, 1 Apr 2021 14:13:55 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1FAAFC0225B9 for ; Thu, 1 Apr 2021 08:41:11 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id x16so2273658wrn.4 for ; Thu, 01 Apr 2021 08:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=y2ml9Lzj9xQznH9AZ+zVhuRkUtNwq1khjP+NPlMkncA=; b=XoQgyQIeMosi0Lja/PgYdjvKemiftIOIxqRTblP2nCGoRQsWjpY6V/3PMSPwA79W/o 40c6jAj7s/ekbhI/zn7Us/B4pe8ulIChI1KQi0Fm1ANdLnZRivznC9kpXlI9sasaCEC+ lvSu4MHzCh8NoVH8ehwKHC87ZwlwCDLxN1vVeVYV9CYpnFQH24dde9+cB+Cf7QJYPF80 1+T9spTRRb/wymHn1VqYLw0TkAMAaGcyXGP4bZSYfIZiIulBQDzp3QlsHXWloMQhMGgI PETBoremLx4OSOxgXBZ3UN98Nzu0lcoBm33AjLTEKhoWgTzn3V1X6zPfODP1DsIn3iHe wUcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=y2ml9Lzj9xQznH9AZ+zVhuRkUtNwq1khjP+NPlMkncA=; b=rozU3gBeljvjyhlkgT4rnbzUQnRRNP4TGsOilfISAjR4mi1J9NHwHRkyoMyJukPIAX WI3l/Fj+CZk1w/L5PTSrNoi5QuMGMmk0bWpzEcBxfaG/BGc8EjYI1EVkrBpDRLwY4E8O ZHHO00/uxsdr5+hahkhWGzO55zGF/iP+48nwgLK5bsNDOnjSI8sLg54/Hl/SKr1DIJMT xc2R42tUvSF7iNPV2rDl6HeZXt8/8Mkci7kuaTqCUjx+CkiMWYsZQ7o4fRxcDLc4CskO ohQzXp03Kfu7qOeBZL/YOYmDjKFOaavHkgQKfe/zqmN//dd4zmepcFtbvVUiRc5ujynI //vA== X-Gm-Message-State: AOAM531SFv6VDIsttMIeiYj2OYeL1AmhyDay5kgRAHlhk+Qbm3+16AuZ C/siyxuroSx42yQrP1sqarjdP8wGWNc= X-Google-Smtp-Source: ABdhPJw4UYUdoozRV6fCV01s03ZKXH4Et7e9QthE8zCBfN2rWq6VTXeVSysCI2Vfq6NlY6j0ym32wQ== X-Received: by 2002:a05:6000:221:: with SMTP id l1mr10437688wrz.370.1617291669964; Thu, 01 Apr 2021 08:41:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u63sm9016862wmg.24.2021.04.01.08.41.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:09 -0700 (PDT) Message-Id: <18c125ec73dc9c23725d7c4688fdc1e753e14780.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:45 +0000 Subject: [PATCH 03/23] config: FSMonitor is repository-specific Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin This commit refactors `git_config_get_fsmonitor()` into the `repo_*()` form that takes a parameter `struct repository *r`. That change prepares for the upcoming `core.useFSMonitorDaemon` flag which will be stored in the `repo_settings` struct. Signed-off-by: Johannes Schindelin --- builtin/update-index.c | 4 ++-- config.c | 4 ++-- config.h | 2 +- fsmonitor.c | 2 +- 4 files changed, 6 insertions(+), 6 deletions(-) diff --git a/builtin/update-index.c b/builtin/update-index.c index 79087bccea4b..84793df8b2b6 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1214,14 +1214,14 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) } if (fsmonitor > 0) { - if (git_config_get_fsmonitor() == 0) + if (repo_config_get_fsmonitor(r) == 0) warning(_("core.fsmonitor is unset; " "set it if you really want to " "enable fsmonitor")); add_fsmonitor(&the_index); report(_("fsmonitor enabled")); } else if (!fsmonitor) { - if (git_config_get_fsmonitor() == 1) + if (repo_config_get_fsmonitor(r) == 1) warning(_("core.fsmonitor is set; " "remove it if you really want to " "disable fsmonitor")); diff --git a/config.c b/config.c index 6428393a4143..955ff4f9461d 100644 --- a/config.c +++ b/config.c @@ -2513,9 +2513,9 @@ int git_config_get_max_percent_split_change(void) return -1; /* default value */ } -int git_config_get_fsmonitor(void) +int repo_config_get_fsmonitor(struct repository *r) { - if (git_config_get_pathname("core.fsmonitor", &core_fsmonitor)) + if (repo_config_get_pathname(r, "core.fsmonitor", &core_fsmonitor)) core_fsmonitor = getenv("GIT_TEST_FSMONITOR"); if (core_fsmonitor && !*core_fsmonitor) diff --git a/config.h b/config.h index 19a9adbaa9a3..3139de81d986 100644 --- a/config.h +++ b/config.h @@ -607,7 +607,7 @@ int git_config_get_index_threads(int *dest); int git_config_get_untracked_cache(void); int git_config_get_split_index(void); int git_config_get_max_percent_split_change(void); -int git_config_get_fsmonitor(void); +int repo_config_get_fsmonitor(struct repository *r); /* This dies if the configured or default date is in the future */ int git_config_get_expiry(const char *key, const char **output); diff --git a/fsmonitor.c b/fsmonitor.c index ab9bfc60b34e..9c9b2abc9414 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -411,7 +411,7 @@ void remove_fsmonitor(struct index_state *istate) void tweak_fsmonitor(struct index_state *istate) { unsigned int i; - int fsmonitor_enabled = git_config_get_fsmonitor(); + int fsmonitor_enabled = repo_config_get_fsmonitor(istate->repo ? istate->repo : the_repository); if (istate->fsmonitor_dirty) { if (fsmonitor_enabled) { From patchwork Thu Apr 1 15:40:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Schindelin X-Patchwork-Id: 12178543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A013AC433ED for ; Thu, 1 Apr 2021 17:47:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7423661155 for ; Thu, 1 Apr 2021 17:47:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236861AbhDARrn (ORCPT ); Thu, 1 Apr 2021 13:47:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235067AbhDARmK (ORCPT ); Thu, 1 Apr 2021 13:42:10 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C360DC0225BA for ; Thu, 1 Apr 2021 08:41:11 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id r10-20020a05600c35cab029010c946c95easo1125188wmq.4 for ; Thu, 01 Apr 2021 08:41:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wJTM7pZEfyegDI5HT6aAfVtznByrjaix/R2aXA7PfEI=; b=VM+MEVogEK/7dWBh+SJwmZH7O3qBZSsOJDgaB2qLk1iIn2hteqD2MSo2VBvLZ2oQxJ bAqbskQRwZ9zUP2DVO06MITP+aJRgrscR+/JnYLyUVARSl04jyyTf1MhOMPW7xkin65R /KZ33+j7XNFoM0plHUhppvwV8nQHpvbNRYvHP8ragA28guhOOgAHSSDBUC27MB49WdFe 8rBEigaT/79aJDCfFMCcidCEhbzCTFsqga1caAwF5WU8wdQPT8LLmaRwi6jmEHXEGmq3 Xo/vCL+nNCDAE8z2dgHTJ7HPAPknWtU37nVSyVyDm22BAL5s86DWITxaBWTz5oUj0PEs HSRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wJTM7pZEfyegDI5HT6aAfVtznByrjaix/R2aXA7PfEI=; b=k7kZfPkhsiaYHEsPDpV59K1rRHNJChqPQseI88HiaQ7ybBVCEPAikdyQHga+jPFpJS FqVIdv5xSPH4D39IDaQvVdCBKp7AQIDWWJjRAZo089JuaPAMw94JLHQSnonroR52G+zb WBqY/A0ebbMaCy91mYP8+2TI9UpGdoYnpYOKroh55YnRXQfEE9sTmUdx0K7uaB960BD6 AAZX1E07/0w1MiuJca/GCSTB3mw224ATr528gHdLb1q71cuMhxExzO33+OYF/JNuhSdQ FnM5yRBxHjy10e3Soix0UlUYX8HZ2LN+Lnk67F88vbpx6qvY5hnGDkQu+o5xS7Kt3ASJ Mplw== X-Gm-Message-State: AOAM532yd4zFpDmlL0RtBoVcpr6c92BhqZNJHNPYjWpfRToxZJQ3vVoP 3VP6p7EoT/1vDfGN2BoZYUxsapdTIyw= X-Google-Smtp-Source: ABdhPJxUG195CvbQFMz48ijEl0F30KuR4VMRBZqYoaifYwIpt9fzUbXctHVCaAeNw/yEDkbjftxKcw== X-Received: by 2002:a05:600c:214d:: with SMTP id v13mr8782361wml.162.1617291670529; Thu, 01 Apr 2021 08:41:10 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a131sm8962097wmc.48.2021.04.01.08.41.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:10 -0700 (PDT) Message-Id: <7082528d8f7c1afa33e1146e3d274e044735f6a1.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:46 +0000 Subject: [PATCH 04/23] fsmonitor: introduce `core.useBuiltinFSMonitor` to call the daemon via IPC Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Johannes Schindelin Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Johannes Schindelin From: Johannes Schindelin The `core.fsmonitor` setting is supposed to be set to a path pointing to a script or executable that (via the Hook API) queries an fsmonitor process such as watchman. We are about to implement our own fsmonitor backend, and do not want to spawn hook processes just to query it. Let's use `Simple IPC` to directly communicate with the daemon (and start it if necessary), guarded by the brand-new `core.useBuiltinFSMonitor` toggle. Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- config.c | 5 +++++ fsmonitor.c | 20 +++++++++++++++++--- repo-settings.c | 3 +++ repository.h | 2 ++ 4 files changed, 27 insertions(+), 3 deletions(-) diff --git a/config.c b/config.c index 955ff4f9461d..31f2cbaf6dfb 100644 --- a/config.c +++ b/config.c @@ -2515,6 +2515,11 @@ int git_config_get_max_percent_split_change(void) int repo_config_get_fsmonitor(struct repository *r) { + if (r->settings.use_builtin_fsmonitor > 0) { + core_fsmonitor = "(built-in daemon)"; + return 1; + } + if (repo_config_get_pathname(r, "core.fsmonitor", &core_fsmonitor)) core_fsmonitor = getenv("GIT_TEST_FSMONITOR"); diff --git a/fsmonitor.c b/fsmonitor.c index 9c9b2abc9414..d7e18fc8cd47 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -3,6 +3,7 @@ #include "dir.h" #include "ewah/ewok.h" #include "fsmonitor.h" +#include "fsmonitor-ipc.h" #include "run-command.h" #include "strbuf.h" @@ -148,14 +149,27 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate) /* * Call the query-fsmonitor hook passing the last update token of the saved results. */ -static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result) +static int query_fsmonitor(int version, struct index_state *istate, struct strbuf *query_result) { + struct repository *r = istate->repo ? istate->repo : the_repository; + const char *last_update = istate->fsmonitor_last_update; struct child_process cp = CHILD_PROCESS_INIT; int result; if (!core_fsmonitor) return -1; + if (r->settings.use_builtin_fsmonitor > 0) { +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + return fsmonitor_ipc__send_query(last_update, query_result); +#else + /* Fake a trivial response. */ + warning(_("fsmonitor--daemon unavailable; falling back")); + strbuf_add(query_result, "/", 2); + return 0; +#endif + } + strvec_push(&cp.args, core_fsmonitor); strvec_pushf(&cp.args, "%d", version); strvec_pushf(&cp.args, "%s", last_update); @@ -263,7 +277,7 @@ void refresh_fsmonitor(struct index_state *istate) if (istate->fsmonitor_last_update) { if (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) { query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2, - istate->fsmonitor_last_update, &query_result); + istate, &query_result); if (query_success) { if (hook_version < 0) @@ -293,7 +307,7 @@ void refresh_fsmonitor(struct index_state *istate) if (hook_version == HOOK_INTERFACE_VERSION1) { query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1, - istate->fsmonitor_last_update, &query_result); + istate, &query_result); } trace_performance_since(last_update, "fsmonitor process '%s'", core_fsmonitor); diff --git a/repo-settings.c b/repo-settings.c index f7fff0f5ab83..93aab92ff164 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -58,6 +58,9 @@ void prepare_repo_settings(struct repository *r) r->settings.core_multi_pack_index = value; UPDATE_DEFAULT_BOOL(r->settings.core_multi_pack_index, 1); + if (!repo_config_get_bool(r, "core.usebuiltinfsmonitor", &value) && value) + r->settings.use_builtin_fsmonitor = 1; + if (!repo_config_get_bool(r, "feature.manyfiles", &value) && value) { UPDATE_DEFAULT_BOOL(r->settings.index_version, 4); UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE); diff --git a/repository.h b/repository.h index b385ca3c94b6..7eeab871ac3e 100644 --- a/repository.h +++ b/repository.h @@ -41,6 +41,8 @@ struct repo_settings { enum fetch_negotiation_setting fetch_negotiation_algorithm; int core_multi_pack_index; + + int use_builtin_fsmonitor; }; struct repository { From patchwork Thu Apr 1 15:40:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178909 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90CE9C433ED for ; Thu, 1 Apr 2021 18:03:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 545996112E for ; Thu, 1 Apr 2021 18:03:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234768AbhDASDu (ORCPT ); Thu, 1 Apr 2021 14:03:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33002 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237013AbhDAR7U (ORCPT ); Thu, 1 Apr 2021 13:59:20 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 749FFC0225BB for ; Thu, 1 Apr 2021 08:41:12 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id g20so1264704wmk.3 for ; Thu, 01 Apr 2021 08:41:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=L6cQcSSTtAe/Y0CatYDGYID+YGP5BAtSb7fqSpCmaC4=; b=SOsk5unzQOWZK+bNvQuxJEQP6FeGq/G1uufzJwLK83s5Y0AuhDoV0PU0xkuIefhI/X t3CbBJKXpCTgv8UnxV4ZnMmPkBwmhUo43H76WXA4snOwaMYr6mgv2Fhtwar8MlH24nhc 9G5ip4Mh2FbmqxIE59YEiXttar/kd+hM24rBOeSa/MwKUQo4hgtHyjF1qpCTmmO2l5Zc PFOWbZEp0sJJwOBXqHZ3SQmgFdVS6Z/g9g0oZ7xRX4t4cVgU6KBHmH2OWXvWYRy1ccB/ mUdMK5Aq5+nyrOvK/Iwr13LMSXNEHpqxGnMt2f54p9R4zSzTyTochsD8j1va8WEvXvcX I2oA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=L6cQcSSTtAe/Y0CatYDGYID+YGP5BAtSb7fqSpCmaC4=; b=kNwzba4i9wQNJU6bwhUPL5f51p/wU08YT1ZQ3CdTVmmLQlD5x7V+TzIcqGt/AC1uIY XG3Gxln5I/Zs1mD2Tq1ednGrv7EcH6aNI0aQNb/qw7s55K9BGqp5bEhuHpRNVu50quHh rcxr/w9TfB2bG8IDylllRH/mAHgqLryry71QxQa7n+Uh5RguZ3a6QhYFZ02WY2XbuMtg KwiYQUsgnju05D6cCmciI3b/lG4t3uSM5Rvck9+/cvEX+BGH3K1rB7cqF5/oMyYVhF02 f6zN8VeAu6ZPWnpl0vEKFcZCvDSKHy7uk1vCtrSfQhjCzlloOrMF2fFN0B4Sho3r02vV XsOg== X-Gm-Message-State: AOAM531szMq7uGzVMdprmxUV5T94vcULksNsZyJ7Ll9CtiCBmyqpEq1a AwR45P4Q+xqlER/W2cyBHc8V2cIAnl0= X-Google-Smtp-Source: ABdhPJwUDo6c6LR/73irWyQcKd/QMRX4QnDrxFYlNLFt+dGZltU5jUH366ARQiWAGVB3n7uf4rjf+Q== X-Received: by 2002:a7b:cdf7:: with SMTP id p23mr8728391wmj.26.1617291671235; Thu, 01 Apr 2021 08:41:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m14sm8675560wmi.27.2021.04.01.08.41.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:10 -0700 (PDT) Message-Id: <95d511d83b1211f24aeb17edbd4918750f406ece.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:47 +0000 Subject: [PATCH 05/23] fsmonitor--daemon: add a built-in fsmonitor daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a built-in file system monitoring daemon that can be used by the existing `fsmonitor` feature (protocol API and index extension) to improve the performance of various Git commands, such as `status`. The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and provides an alternative to hook access to existing fsmonitors such as `watchman`. This commit merely adds the new command without any functionality. Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- .gitignore | 1 + Makefile | 1 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 52 +++++++++++++++++++++++++++++++++++++ git.c | 1 + 5 files changed, 56 insertions(+) create mode 100644 builtin/fsmonitor--daemon.c diff --git a/.gitignore b/.gitignore index 3dcdb6bb5ab8..beccf34abe9e 100644 --- a/.gitignore +++ b/.gitignore @@ -71,6 +71,7 @@ /git-format-patch /git-fsck /git-fsck-objects +/git-fsmonitor--daemon /git-gc /git-get-tar-commit-id /git-grep diff --git a/Makefile b/Makefile index 50977911d41a..d792631d4250 100644 --- a/Makefile +++ b/Makefile @@ -1091,6 +1091,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o +BUILTIN_OBJS += builtin/fsmonitor--daemon.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o BUILTIN_OBJS += builtin/grep.o diff --git a/builtin.h b/builtin.h index b6ce981b7377..7554476f90a4 100644 --- a/builtin.h +++ b/builtin.h @@ -158,6 +158,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix); int cmd_for_each_repo(int argc, const char **argv, const char *prefix); int cmd_format_patch(int argc, const char **argv, const char *prefix); int cmd_fsck(int argc, const char **argv, const char *prefix); +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix); int cmd_gc(int argc, const char **argv, const char *prefix); int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix); int cmd_grep(int argc, const char **argv, const char *prefix); diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c new file mode 100644 index 000000000000..6700bac92c7d --- /dev/null +++ b/builtin/fsmonitor--daemon.c @@ -0,0 +1,52 @@ +#include "builtin.h" +#include "config.h" +#include "parse-options.h" +#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +#include "simple-ipc.h" +#include "khash.h" + +static const char * const builtin_fsmonitor__daemon_usage[] = { + NULL +}; + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + enum daemon_mode { + UNDEFINED_MODE, + } mode = UNDEFINED_MODE; + + struct option options[] = { + OPT_END() + }; + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + git_config(git_default_config, NULL); + + argc = parse_options(argc, argv, prefix, options, + builtin_fsmonitor__daemon_usage, 0); + + switch (mode) { + case UNDEFINED_MODE: + default: + die(_("Unhandled command mode %d"), mode); + } +} + +#else +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + struct option options[] = { + OPT_END() + }; + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + die(_("fsmonitor--daemon not supported on this platform")); +} +#endif diff --git a/git.c b/git.c index 9bc077a025cb..239deb9823fc 100644 --- a/git.c +++ b/git.c @@ -523,6 +523,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY }, From patchwork Thu Apr 1 15:40:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25A8AC41538 for ; Thu, 1 Apr 2021 17:43:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 138AC61383 for ; Thu, 1 Apr 2021 17:43:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235863AbhDARnR (ORCPT ); Thu, 1 Apr 2021 13:43:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234747AbhDARjm (ORCPT ); Thu, 1 Apr 2021 13:39:42 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2BB9CC0225BC for ; Thu, 1 Apr 2021 08:41:13 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id j9so2254773wrx.12 for ; Thu, 01 Apr 2021 08:41:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=daAB+2L2OPG1QBkdGoK+7+40yncSFaJ1sOuvJFAzK/A=; b=rpDTXyuHw4OSlz6ebbkMfUt84a4+dwJoIxm2rl5iERoUkhX8r389O5DUGooXIOxWEJ bRRVbMRb/0aAYEzUQbkgz4zfHNtjmeNIUsbZ96LPuO5H3Lbylpdt057nVb4GldupBpsn 0ofNI+NJAPVe6EjftODs4JKHmTZjOqbaTuRAnILyYyy8e6o4eV7uiGBnAb48DIRCaJC6 IkKruOXWb6TrH7s1o4+XDh0XGOvy9o8iG+azfypnNQGU5cZdMojQtSlDnpQpCCmSp424 4hYVQYBlyOS+YImkXpidBsngSv7hbM4wRaeQyrqTNeo0n6/2n6CWRLxE5hrt1UFtaDuG 0pCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=daAB+2L2OPG1QBkdGoK+7+40yncSFaJ1sOuvJFAzK/A=; b=gypgLdCc5fVYW9Yb0aBh9Ujm+FeyVtsC5d4X2OG/lBjAHJeYyVsJZ84oh2D45fVRHE xs6YCNuUbnmAdn1g+LKKnNKaoDThrsrOxkjobyLtXAqCQOYZ+TApXoQHZKzH0NvxGxjo 5It5vBzYvXEDI+baowaDc/hfq2l/9K0/5ITMaIzH4XIhCxmWhWN6UXaFP6lAx4/vn/gI GcvqV3j8rGqQ7Gv233R0U96QS/Jz5f7Szq0R0Y8Kf9NPg7YvTxkV2BCnu74Tqu4y8+eT 5sAvofzjPAh6ljK7mAqHbAuPlTu5WZb3ECINVKDvhRIZ1kaTvcPCzpubJNQwn2ucZQfx X5SQ== X-Gm-Message-State: AOAM531+Rw+mQKPEBC0WXQ40C1qnajtUmq/i8OSEG3HcMpDiXxO6Vqow tU+xnwl25UuL8HBk1vBaVJ7QKpY3PZw= X-Google-Smtp-Source: ABdhPJxrp+ZRcjrDjh9C2eNSCJQvZi2oA1NaQB20ka9TUlzEqeYy66iwhnfluROrw6ba+dU6ll9kOg== X-Received: by 2002:a5d:5051:: with SMTP id h17mr10272629wrt.80.1617291671910; Thu, 01 Apr 2021 08:41:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p14sm8533221wmc.30.2021.04.01.08.41.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:11 -0700 (PDT) Message-Id: <77170e521f67ce92587d833334f6951a9f375d4d.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:48 +0000 Subject: [PATCH 06/23] fsmonitor--daemon: implement client command options Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement command options `--stop`, `--is-running`, `--query`, `--query-index`, and `--flush` to control and query the status of a `fsmonitor--daemon` server process (and implicitly start a server process if necessary). Later commits will implement the actual server and monitor the file system. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 144 ++++++++++++++++++++++++++++++++++++ 1 file changed, 144 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 6700bac92c7d..10434bce4b64 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,18 +7,144 @@ #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon --stop"), + N_("git fsmonitor--daemon --is-running"), + N_("git fsmonitor--daemon --query "), + N_("git fsmonitor--daemon --query-index"), + N_("git fsmonitor--daemon --flush"), NULL }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Acting as a CLIENT. + * + * Send an IPC query to a `git-fsmonitor--daemon` SERVER process and + * ask for the changes since the given token. This will implicitly + * start a daemon process if necessary. The daemon process will + * persist after we exit. + * + * This feature is primarily used by the test suite. + */ +static int do_as_client__query_token(const char *token) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_query(token, &answer); + if (ret < 0) + die(_("could not query fsmonitor--daemon")); + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +/* + * Acting as a CLIENT. + * + * Read the `.git/index` to get the last token written to the FSMonitor index + * extension and use that to make a query. + * + * This feature is primarily used by the test suite. + */ +static int do_as_client__query_from_index(void) +{ + struct index_state *istate = the_repository->index; + + setup_git_directory(); + if (do_read_index(istate, the_repository->index_file, 0) < 0) + die("unable to read index file"); + if (!istate->fsmonitor_last_update) + die("index file does not have fsmonitor extension"); + + return do_as_client__query_token(istate->fsmonitor_last_update); +} + +/* + * Acting as a CLIENT. + * + * Send a "quit" command to the `git-fsmonitor--daemon` (if running) + * and wait for it to shutdown. + */ +static int do_as_client__send_stop(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("quit", &answer); + + /* The quit command does not return any response data. */ + strbuf_release(&answer); + + if (ret) + return ret; + + trace2_region_enter("fsm_client", "polling-for-daemon-exit", NULL); + while (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + sleep_millisec(50); + trace2_region_leave("fsm_client", "polling-for-daemon-exit", NULL); + + return 0; +} + +/* + * Acting as a CLIENT. + * + * Send a "flush" command to the `git-fsmonitor--daemon` (if running) + * and tell it to flush its cache. + * + * This feature is primarily used by the test suite to simulate a loss of + * sync with the filesystem where we miss kernel events. + */ +static int do_as_client__send_flush(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("flush", &answer); + if (ret) + return ret; + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +static int is_ipc_daemon_listening(void) +{ + return fsmonitor_ipc__get_state() == IPC_STATE__LISTENING; +} int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { enum daemon_mode { UNDEFINED_MODE, + STOP, + IS_RUNNING, + QUERY, + QUERY_INDEX, + FLUSH, } mode = UNDEFINED_MODE; struct option options[] = { + OPT_CMDMODE(0, "stop", &mode, N_("stop the running daemon"), + STOP), + + OPT_CMDMODE(0, "is-running", &mode, + N_("test whether the daemon is running"), + IS_RUNNING), + + OPT_CMDMODE(0, "query", &mode, + N_("query the daemon (starting if necessary)"), + QUERY), + OPT_CMDMODE(0, "query-index", &mode, + N_("query the daemon (starting if necessary) using token from index"), + QUERY_INDEX), + OPT_CMDMODE(0, "flush", &mode, N_("flush cached filesystem events"), + FLUSH), OPT_END() }; @@ -31,6 +157,24 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) builtin_fsmonitor__daemon_usage, 0); switch (mode) { + case STOP: + return !!do_as_client__send_stop(); + + case IS_RUNNING: + return !is_ipc_daemon_listening(); + + case QUERY: + if (argc != 1) + usage_with_options(builtin_fsmonitor__daemon_usage, + options); + return !!do_as_client__query_token(argv[0]); + + case QUERY_INDEX: + return !!do_as_client__query_from_index(); + + case FLUSH: + return !!do_as_client__send_flush(); + case UNDEFINED_MODE: default: die(_("Unhandled command mode %d"), mode); From patchwork Thu Apr 1 15:40:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12179137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 987A6C433ED for ; Thu, 1 Apr 2021 18:22:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 75B056023C for ; Thu, 1 Apr 2021 18:22:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236940AbhDASWS (ORCPT ); Thu, 1 Apr 2021 14:22:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38178 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236777AbhDASSd (ORCPT ); Thu, 1 Apr 2021 14:18:33 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2523C0225BD for ; Thu, 1 Apr 2021 08:41:13 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id g20so1264726wmk.3 for ; Thu, 01 Apr 2021 08:41:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3ZUBOkVH8ZSHppSsA61ocw2rjIoSNjZaCX8fV/A/TWM=; b=S4GtjvqclApOAuHc+dFDw1SmKDGzUxok+6cbOxH01q3uD/ekoEZXRjOOkSUVl+ELqS NWE4vJb3yo8+43yEeYqR8gaTnTsgxA2cu15O6CK7lY8jkTWXqSxCdkjW4ieANyWYGzlL aFvuLRxhY+QG3/JpvYKfsC3B0DARrHQuLNaFnlRXTHKp12Y1YA1eHjUqbIb7ypBqnPV5 EKFesPvLDlyr/lYXsIk2nqL2ulagzir5HJLrDiwti42t7HyPq0PoPJWBl6cbGdOqqQLp b3eNcF8Ho6LFv1Q/UV7sxLPJtXTAsNIfDuxU+yw43zmUMXq58AmILMz2ttsId/+WxyiB ilYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3ZUBOkVH8ZSHppSsA61ocw2rjIoSNjZaCX8fV/A/TWM=; b=AshlC3Tds62aB8UPUs0u2PueTok/PBFPjQ7pNMKZgb09UPXobB2nRHBqFATK3c513b 850NBknYlBqsrbNcdCYnzOmWeuoCLsJ7uSV9GSrvgZRLRV+2NoLn8kzvEEXI2tHZJNam Y+4DjieFUujs+KviAAwre+yXXcfM2ii0wv5wBbPWhuWFxP9oUDPNt3+VRv6Nkkf8TeeL KRKakyokEvKHQkAZgTST8Y4C/j1ktIIZYb5C9i90OZLrszBDsTBtfr1LwCPOFhCyuaph hdBcUIMYP4MQz7Dn4ul8UZtzucUOG9px7O+SpugNmxughciS9Wrz2fdKD8nBo6z2nRLK BWcw== X-Gm-Message-State: AOAM533gEQRUqL+TbqvhiiXsDzINcLH6e/jq7jbuDffbUBe4tnkFGLuF AsN5miTxNeqxozBAVn1Ey627wsUXeQ8= X-Google-Smtp-Source: ABdhPJw4kvHx9vSb7g9ECJqqYKOO+dN3xp65yufVTskY88UUHt1rd61KwgKobumb3ekPBTlA4TfwKQ== X-Received: by 2002:a1c:c918:: with SMTP id f24mr8888013wmb.12.1617291672485; Thu, 01 Apr 2021 08:41:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w6sm10321114wrl.49.2021.04.01.08.41.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:12 -0700 (PDT) Message-Id: <27f47dfbd9cf1103fcc5ffff9da6239c1a441987.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:49 +0000 Subject: [PATCH 07/23] fsmonitor-fs-listen-win32: stub in backend for Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty backend for fsmonitor--daemon on Windows. Signed-off-by: Jeff Hostetler --- Makefile | 13 ++++++ compat/fsmonitor/fsmonitor-fs-listen-win32.c | 21 +++++++++ compat/fsmonitor/fsmonitor-fs-listen.h | 49 ++++++++++++++++++++ config.mak.uname | 2 + contrib/buildsystems/CMakeLists.txt | 5 ++ 5 files changed, 90 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-win32.c create mode 100644 compat/fsmonitor/fsmonitor-fs-listen.h diff --git a/Makefile b/Makefile index d792631d4250..014bc1baa03a 100644 --- a/Makefile +++ b/Makefile @@ -467,6 +467,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports an built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the name of the corresponding +# `compat/fsmonitor/fsmonitor-fs-listen-.c` that implements the +# `fsmonitor_fs_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1904,6 +1909,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsmonitor-fs-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2761,6 +2771,9 @@ GIT-BUILD-OPTIONS: FORCE @echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+ @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c new file mode 100644 index 000000000000..880446b49e35 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c @@ -0,0 +1,21 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) +{ +} + +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsmonitor-fs-listen.h b/compat/fsmonitor/fsmonitor-fs-listen.h new file mode 100644 index 000000000000..c7b5776b3b60 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen.h @@ -0,0 +1,49 @@ +#ifndef FSMONITOR_FS_LISTEN_H +#define FSMONITOR_FS_LISTEN_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread PRIOR to staring the + * fsmonitor_fs_listener thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread AFTER joining the listener. + */ +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to watch for + * filesystem events. This will run in the fsmonitor_fs_listen thread. + * + * It should call `ipc_server_stop_async()` if the listener thread + * prematurely terminates (because of a filesystem error or if it + * detects that the .git directory has been deleted). (It should NOT + * do so if the listener thread receives a normal shutdown signal from + * the IPC layer.) + * + * It should set `state->error_code` to -1 if the daemon should exit + * with an error. + */ +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the fsmonitor listener thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_FS_LISTEN_H */ diff --git a/config.mak.uname b/config.mak.uname index cb443b4e023a..fcd88b60b14a 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -420,6 +420,7 @@ ifeq ($(uname_S),Windows) # so we don't need this: # # SNPRINTF_RETURNS_BOGUS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -598,6 +599,7 @@ ifneq (,$(findstring MINGW,$(uname_S))) NO_STRTOUMAX = YesPlease NO_MKDTEMP = YesPlease NO_SVN_TESTS = YesPlease + FSMONITOR_DAEMON_BACKEND = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 9897fcc8ea2a..727cfd561169 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -252,6 +252,11 @@ else() list(APPEND compat_SOURCES compat/simple-ipc/ipc-shared.c compat/simple-ipc/ipc-unix-socket.c) endif() +if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +endif() + set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) #header checks From patchwork Thu Apr 1 15:40:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178911 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3E68C43461 for ; Thu, 1 Apr 2021 18:04:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A83F06112E for ; Thu, 1 Apr 2021 18:04:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237575AbhDASD5 (ORCPT ); Thu, 1 Apr 2021 14:03:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33036 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237017AbhDAR7U (ORCPT ); Thu, 1 Apr 2021 13:59:20 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 77024C0225BE for ; Thu, 1 Apr 2021 08:41:14 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id j7so2288683wrd.1 for ; Thu, 01 Apr 2021 08:41:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=j8mRvxrP6D8rCyINcmWGLVKTxfMVEXKp6wmZZ3hVF0I=; b=HYaMKjXWdJ2lvC/zfMxYmyw9tFa7hK/9DKNY+IBQgcrA/XO97kSISyo8G+gxrQNzYH fHBR+pnGyEtfKVM7XiGwG8ayS+RtsFIfxbSH2aFlmSJryiUmaQieD6sJWZzRdvGApO/G 0ytukgUHQ/3xgoQnVZsszsOCJWozYQXGM+AAWE0TJWQt0nEXDbiG3y9l0ElrtanomIPl cz6bSgDzj/X3BzUfvgiAbUyFTbMtaTdE3Qlog/iodsUTdeXO/WVDH56FTMrYpczGajc9 HYMc3nhbK40qbk+kMx+6RdwWsp4uU6WXRZ0R7sxaM+Jc5WtVIjR/6vWruJ3k3fDjQVGc XLEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=j8mRvxrP6D8rCyINcmWGLVKTxfMVEXKp6wmZZ3hVF0I=; b=RbcX+l9oZVCtrk1YfhMepADHCOF1+ALEA+Euis8AeFPJXfgpyFQkNJXDmqU8pxytIl e2wi7h7j8pvzss30X7pUesngYxjIG28kfw9UUC2QVoKrk8JC+iVwJ0ZPzD29QuPlvPS6 LQPwG5N9b5CcPq7XrIp2I0PIPEZ0+BdthuxyzrsnjVPueDyXfuS1lCja7mVIebuqtRG7 RmohdVxeuAmEWzuJPbkhoEl+ILkE4yN+ZFndfAQ9O3AXlU4hy1GBOB0LXtge9LLXLpDX f1u/dJehgchKvg+H8AuDCs9VGcLB5MhWmLwf/8LfTR+1t0wZXbTabtbQsrcF8IFoigf4 anHg== X-Gm-Message-State: AOAM533/69E8yuT7T3b+5kxJiyFeEe8M45g619j2UJdL9EF3cqkLVb3S 69VTi6RQ4qwc3fcpod9xCZElKmxrXGQ= X-Google-Smtp-Source: ABdhPJzXBTv90f/nvbEI/Ix9OsAoz5/Wm3O5NyOd/ZnhAmPVw70cgGi644ge4SLNkJzCuvHLpMeYFA== X-Received: by 2002:adf:e4c7:: with SMTP id v7mr10527138wrm.245.1617291673224; Thu, 01 Apr 2021 08:41:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e13sm12546714wrg.72.2021.04.01.08.41.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:12 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:50 +0000 Subject: [PATCH 08/23] fsmonitor-fs-listen-macos: stub in backend for MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty implementation of fsmonitor--daemon backend for MacOS. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 20 ++++++++++++++++++++ config.mak.uname | 2 ++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 25 insertions(+) create mode 100644 compat/fsmonitor/fsmonitor-fs-listen-macos.c diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c new file mode 100644 index 000000000000..b91058d1c4f8 --- /dev/null +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsmonitor-fs-listen.h" + +int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) +{ +} diff --git a/config.mak.uname b/config.mak.uname index fcd88b60b14a..394355463e1e 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -147,6 +147,8 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + FSMONITOR_DAEMON_BACKEND = macos + BASIC_LDFLAGS += -framework CoreServices endif ifeq ($(uname_S),SunOS) NEEDS_SOCKET = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 727cfd561169..341a85e7bfc9 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -255,6 +255,9 @@ endif() if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-win32.c) +elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsmonitor-fs-listen-macos.c) endif() set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) From patchwork Thu Apr 1 15:40:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1A87C433B4 for ; Thu, 1 Apr 2021 17:48:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9A13160234 for ; Thu, 1 Apr 2021 17:48:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235939AbhDARsj (ORCPT ); Thu, 1 Apr 2021 13:48:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58348 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234526AbhDARq0 (ORCPT ); Thu, 1 Apr 2021 13:46:26 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4EDCBC0225BF for ; Thu, 1 Apr 2021 08:41:15 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id j7so2288730wrd.1 for ; Thu, 01 Apr 2021 08:41:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=G7IN06+H8IDLJ3y+IVxrDpCh8W6OR6uoamaJxA/C9xI=; b=fBXEVlSDLYscvFKuDaWgyjuVPGmAP94NYob4G3fskgUqhQxDgwn9xNWDOs1b2OlK5N pncKtct0q7RhcQUuGhwKjYde4jCzRYHu/uwgK2DTJY0V8zrZXXTIfhDgY+DbYaxVV89X 780SQbjMpz7SS8pgKublCtbkjwuyeMsZ344QZULYkPdi2nYOJNrFoNaTBVa/scsAklI4 PUeLnWV3j+CVW6Z1oS1wWj4nSWvphTeODWez6zV7upqKviqViHQqy45KuTA4Vf1RhLnx LWv8oXMmq0yp20NiNskujoTiHR3Wl2AWQ/f5gzYEUwg6rFp19FSpI+Qw6jeTUlSKQc1t bSdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=G7IN06+H8IDLJ3y+IVxrDpCh8W6OR6uoamaJxA/C9xI=; b=QZQKc4/R/mFRM66gQgSgixW0gEmqQQ254qaaOY2inJcoqZTsJfU1w8U3Zo7QK9khup ltH1eSXyFoJxylozNvQVlxd3u4Ccw4YgLiVWKK4OghxnNJypdPsC6v4VwYl/gLcISgch CllrQhjBWFLziQA2PfAg7H5xBJMH2h48wgEFZ/HnTBplnAC88TQcSSVmsZR7x6fXZ6yt 2UGp2CMtGcmeAN4NTJ+IUs7LpvaGnroAEcugddNj5WxGT6r0BcjdaCAXS5p0fYVQFAso xIUJjuBC2kfEtHQPgVega5Gb60VNNZDcjazIxYIo4VXe7fhYTAQpV+wHGU2Uc+1JYsqx ay5g== X-Gm-Message-State: AOAM532f6b4fgwwn+jJwQZ2HgtsK2obHvXD6F7iagdodiByRvP8ap+X3 wpxBStPobjWeieGF+FzGUCm795B7tXo= X-Google-Smtp-Source: ABdhPJxIEimZOZ9I1r2VFGomGa16htNV2uMv6IC6PbeJd8at0SlTYBEKaj8KZapHv0y34nb//4dZdw== X-Received: by 2002:a05:6000:1a8a:: with SMTP id f10mr10221066wry.232.1617291673952; Thu, 01 Apr 2021 08:41:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d204sm9270009wmc.17.2021.04.01.08.41.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:13 -0700 (PDT) Message-Id: <2b291d805d59b54203d939e048bb568782d5e17b.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:51 +0000 Subject: [PATCH 09/23] fsmonitor--daemon: implement daemon command options Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement command options `--run` and `--start` to try to begin listening for file system events. This version defines the thread structure with a single fsmonitor_fs_listen thread to watch for file system events and a simple IPC thread pool to wait for connections from Git clients over a well-known named pipe or Unix domain socket. This version does not actually do anything yet because the backends are still just stubs. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 395 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 36 ++++ 2 files changed, 430 insertions(+), 1 deletion(-) create mode 100644 fsmonitor--daemon.h diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 10434bce4b64..23a063707972 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,10 +3,14 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon --start []"), + N_("git fsmonitor--daemon --run []"), N_("git fsmonitor--daemon --stop"), N_("git fsmonitor--daemon --is-running"), N_("git fsmonitor--daemon --query "), @@ -16,6 +20,38 @@ static const char * const builtin_fsmonitor__daemon_usage[] = { }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Global state loaded from config. + */ +#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" +static int fsmonitor__ipc_threads = 8; + +#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout" +static int fsmonitor__start_timeout_sec = 60; + +static int fsmonitor_config(const char *var, const char *value, void *cb) +{ + if (!strcmp(var, FSMONITOR__IPC_THREADS)) { + int i = git_config_int(var, value); + if (i < 1) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__IPC_THREADS, i); + fsmonitor__ipc_threads = i; + return 0; + } + + if (!strcmp(var, FSMONITOR__START_TIMEOUT)) { + int i = git_config_int(var, value); + if (i < 0) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__START_TIMEOUT, i); + fsmonitor__start_timeout_sec = i; + return 0; + } + + return git_default_config(var, value, cb); +} + /* * Acting as a CLIENT. * @@ -113,15 +149,350 @@ static int do_as_client__send_flush(void) return 0; } +static ipc_server_application_cb handle_client; + +static int handle_client(void *data, const char *command, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + /* struct fsmonitor_daemon_state *state = data; */ + int result; + + trace2_region_enter("fsmonitor", "handle_client", the_repository); + trace2_data_string("fsmonitor", the_repository, "request", command); + + result = 0; /* TODO Do something here. */ + + trace2_region_leave("fsmonitor", "handle_client", the_repository); + + return result; +} + +static void *fsmonitor_fs_listen__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-listen"); + + trace_printf_key(&trace_fsmonitor, "Watching: worktree '%s'", + state->path_worktree_watch.buf); + if (state->nr_paths_watching > 1) + trace_printf_key(&trace_fsmonitor, "Watching: gitdir '%s'", + state->path_gitdir_watch.buf); + + fsmonitor_fs_listen__loop(state); + + trace2_thread_exit(); + return NULL; +} + +static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) +{ + struct ipc_server_opts ipc_opts = { + .nr_threads = fsmonitor__ipc_threads, + + /* + * We know that there are no other active threads yet, + * so we can let the IPC layer temporarily chdir() if + * it needs to when creating the server side of the + * Unix domain socket. + */ + .uds_disallow_chdir = 0 + }; + + /* + * Start the IPC thread pool before the we've started the file + * system event listener thread so that we have the IPC handle + * before we need it. + */ + if (ipc_server_run_async(&state->ipc_server_data, + fsmonitor_ipc__get_path(), &ipc_opts, + handle_client, state)) + return error(_("could not start IPC thread pool")); + + /* + * Start the fsmonitor listener thread to collect filesystem + * events. + */ + if (pthread_create(&state->listener_thread, NULL, + fsmonitor_fs_listen__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + ipc_server_await(state->ipc_server_data); + + return error(_("could not start fsmonitor listener thread")); + } + + /* + * The daemon is now fully functional in background threads. + * Wait for the IPC thread pool to shutdown (whether by client + * request or from filesystem activity). + */ + ipc_server_await(state->ipc_server_data); + + /* + * The fsmonitor listener thread may have received a shutdown + * event from the IPC thread pool, but it doesn't hurt to tell + * it again. And wait for it to shutdown. + */ + fsmonitor_fs_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + + return state->error_code; +} + +static int fsmonitor_run_daemon(void) +{ + struct fsmonitor_daemon_state state; + int err; + + memset(&state, 0, sizeof(state)); + + pthread_mutex_init(&state.main_lock, NULL); + state.error_code = 0; + state.current_token_data = NULL; + state.test_client_delay_ms = 0; + + /* Prepare to (recursively) watch the directory. */ + strbuf_init(&state.path_worktree_watch, 0); + strbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree())); + state.nr_paths_watching = 1; + + /* + * If ".git" is not a directory, then is not inside the + * cone of , so set up a second watch for it. + */ + strbuf_init(&state.path_gitdir_watch, 0); + strbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch); + strbuf_addstr(&state.path_gitdir_watch, "/.git"); + if (!is_directory(state.path_gitdir_watch.buf)) { + strbuf_reset(&state.path_gitdir_watch); + strbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir())); + state.nr_paths_watching = 2; + } + + /* + * Confirm that we can create platform-specific resources for the + * filesystem listener before we bother starting all the threads. + */ + if (fsmonitor_fs_listen__ctor(&state)) { + err = error(_("could not initialize listener thread")); + goto done; + } + + err = fsmonitor_run_daemon_1(&state); + +done: + pthread_mutex_destroy(&state.main_lock); + fsmonitor_fs_listen__dtor(&state); + + ipc_server_free(state.ipc_server_data); + + strbuf_release(&state.path_worktree_watch); + strbuf_release(&state.path_gitdir_watch); + + return err; +} + static int is_ipc_daemon_listening(void) { return fsmonitor_ipc__get_state() == IPC_STATE__LISTENING; } +static int try_to_run_foreground_daemon(void) +{ + /* + * Technically, we don't need to probe for an existing daemon + * process, since we could just call `fsmonitor_run_daemon()` + * and let it fail if the pipe/socket is busy. + * + * However, this method gives us a nicer error message for a + * common error case. + */ + if (is_ipc_daemon_listening()) + die("fsmonitor--daemon is already running."); + + return !!fsmonitor_run_daemon(); +} + +#ifndef GIT_WINDOWS_NATIVE +/* + * This is adapted from `daemonize()`. Use `fork()` to directly create + * and run the daemon in a child process. The fork-parent returns the + * child PID so that we can wait for the child to startup before exiting. + */ +static int spawn_background_fsmonitor_daemon(pid_t *pid) +{ + *pid = fork(); + + switch (*pid) { + case 0: + if (setsid() == -1) + error_errno(_("setsid failed")); + close(0); + close(1); + close(2); + sanitize_stdfds(); + + return !!fsmonitor_run_daemon(); + + case -1: + return error_errno(_("could not spawn fsmonitor--daemon in the background")); + + default: + return 0; + } +} +#else +/* + * Conceptually like `daemonize()` but different because Windows does not + * have `fork(2)`. Spawn a normal Windows child process but without the + * limitations of `start_command()` and `finish_command()`. + */ +static int spawn_background_fsmonitor_daemon(pid_t *pid) +{ + char git_exe[MAX_PATH]; + struct strvec args = STRVEC_INIT; + int in, out; + + GetModuleFileNameA(NULL, git_exe, MAX_PATH); + + in = open("/dev/null", O_RDONLY); + out = open("/dev/null", O_WRONLY); + + strvec_push(&args, git_exe); + strvec_push(&args, "fsmonitor--daemon"); + strvec_push(&args, "--run"); + + *pid = mingw_spawnvpe(args.v[0], args.v, NULL, NULL, in, out, out); + close(in); + close(out); + + strvec_clear(&args); + + if (*pid < 0) + return error(_("could not spawn fsmonitor--daemon in the background")); + + return 0; +} +#endif + +/* + * This is adapted from `wait_or_whine()`. Watch the child process and + * let it get started and begin listening for requests on the socket + * before reporting our success. + */ +static int wait_for_background_startup(pid_t pid_child) +{ + int status; + pid_t pid_seen; + enum ipc_active_state s; + time_t time_limit, now; + + time(&time_limit); + time_limit += fsmonitor__start_timeout_sec; + + for (;;) { + pid_seen = waitpid(pid_child, &status, WNOHANG); + + if (pid_seen == -1) + return error_errno(_("waitpid failed")); + + else if (pid_seen == 0) { + /* + * The child is still running (this should be + * the normal case). Try to connect to it on + * the socket and see if it is ready for + * business. + * + * If there is another daemon already running, + * our child will fail to start (possibly + * after a timeout on the lock), but we don't + * care (who responds) if the socket is live. + */ + s = fsmonitor_ipc__get_state(); + if (s == IPC_STATE__LISTENING) + return 0; + + time(&now); + if (now > time_limit) + return error(_("fsmonitor--daemon not online yet")); + + continue; + } + + else if (pid_seen == pid_child) { + /* + * The new child daemon process shutdown while + * it was starting up, so it is not listening + * on the socket. + * + * Try to ping the socket in the odd chance + * that another daemon started (or was already + * running) while our child was starting. + * + * Again, we don't care who services the socket. + */ + s = fsmonitor_ipc__get_state(); + if (s == IPC_STATE__LISTENING) + return 0; + + /* + * We don't care about the WEXITSTATUS() nor + * any of the WIF*(status) values because + * `cmd_fsmonitor__daemon()` does the `!!result` + * trick on all function return values. + * + * So it is sufficient to just report the + * early shutdown as an error. + */ + return error(_("fsmonitor--daemon failed to start")); + } + + else + return error(_("waitpid is confused")); + } +} + +static int try_to_start_background_daemon(void) +{ + pid_t pid_child; + int ret; + + /* + * Before we try to create a background daemon process, see + * if a daemon process is already listening. This makes it + * easier for us to report an already-listening error to the + * console, since our spawn/daemon can only report the success + * of creating the background process (and not whether it + * immediately exited). + */ + if (is_ipc_daemon_listening()) + die("fsmonitor--daemon is already running."); + + /* + * Run the actual daemon in a background process. + */ + ret = spawn_background_fsmonitor_daemon(&pid_child); + if (pid_child <= 0) + return ret; + + /* + * Wait (with timeout) for the background child process get + * started and begin listening on the socket/pipe. This makes + * the "start" command more synchronous and more reliable in + * tests. + */ + ret = wait_for_background_startup(pid_child); + + return ret; +} + int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { enum daemon_mode { UNDEFINED_MODE, + START, + RUN, STOP, IS_RUNNING, QUERY, @@ -130,6 +501,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) } mode = UNDEFINED_MODE; struct option options[] = { + OPT_CMDMODE(0, "start", &mode, + N_("run the daemon in the background"), + START), + OPT_CMDMODE(0, "run", &mode, + N_("run the daemon in the foreground"), RUN), OPT_CMDMODE(0, "stop", &mode, N_("stop the running daemon"), STOP), @@ -145,18 +521,35 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) QUERY_INDEX), OPT_CMDMODE(0, "flush", &mode, N_("flush cached filesystem events"), FLUSH), + + OPT_GROUP(N_("Daemon options")), + OPT_INTEGER(0, "ipc-threads", + &fsmonitor__ipc_threads, + N_("use ipc worker threads")), + OPT_INTEGER(0, "start-timeout", + &fsmonitor__start_timeout_sec, + N_("Max seconds to wait for background daemon startup")), OPT_END() }; if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_fsmonitor__daemon_usage, options); - git_config(git_default_config, NULL); + git_config(fsmonitor_config, NULL); argc = parse_options(argc, argv, prefix, options, builtin_fsmonitor__daemon_usage, 0); + if (fsmonitor__ipc_threads < 1) + die(_("invalid 'ipc-threads' value (%d)"), + fsmonitor__ipc_threads); switch (mode) { + case START: + return !!try_to_start_background_daemon(); + + case RUN: + return !!try_to_run_foreground_daemon(); + case STOP: return !!do_as_client__send_stop(); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h new file mode 100644 index 000000000000..09e4a6fb6675 --- /dev/null +++ b/fsmonitor--daemon.h @@ -0,0 +1,36 @@ +#ifndef FSMONITOR_DAEMON_H +#define FSMONITOR_DAEMON_H + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +#include "cache.h" +#include "dir.h" +#include "run-command.h" +#include "simple-ipc.h" +#include "thread-utils.h" + +struct fsmonitor_batch; +struct fsmonitor_token_data; + +struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ + +struct fsmonitor_daemon_state { + pthread_t listener_thread; + pthread_mutex_t main_lock; + + struct strbuf path_worktree_watch; + struct strbuf path_gitdir_watch; + int nr_paths_watching; + + struct fsmonitor_token_data *current_token_data; + + int error_code; + struct fsmonitor_daemon_backend_data *backend_data; + + struct ipc_server_data *ipc_server_data; + + int test_client_delay_ms; +}; + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_DAEMON_H */ From patchwork Thu Apr 1 15:40:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12179037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12240C433B4 for ; Thu, 1 Apr 2021 18:10:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D62576044F for ; Thu, 1 Apr 2021 18:10:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238052AbhDASJQ (ORCPT ); Thu, 1 Apr 2021 14:09:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33960 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237299AbhDASDi (ORCPT ); Thu, 1 Apr 2021 14:03:38 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A91B4C02D540 for ; Thu, 1 Apr 2021 08:41:15 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id e18so2266691wrt.6 for ; Thu, 01 Apr 2021 08:41:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=v+4ydFp20gfehTB/QwfCWmMDLcbizE3XcA7eYRrUGT0=; b=Ry1NS8AoUVnj5UHjSG3YyHi8Pdty8F92LUO0nQNiNaKoI5vcTfMvjdB7ZuDLDTF0oY 6YQ2kt9lcNBA4A1GuG8rEiCJaRztFwIYqx493RtYdueqwBYQL017w+9UX4edd7PULQJO b9iYHHNLDJVYHQBf/oaBXgV1hvLTMN5+FFwlBEG3lg6VDCGMXb/vCGcDWcZptKxPq0st +vI1iIEy36EuwElJ/bkSxq5Y6dhICOzyEzv42YufBheuxpaArM20L7A65AeSPa4yDYme 7onEtczK2TSE8Vm8wrUnZ+mS+6KMIvDdgshGt+ABKG0xCx+8dH1ekhyIdJYfcLQLyzeg Y2iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=v+4ydFp20gfehTB/QwfCWmMDLcbizE3XcA7eYRrUGT0=; b=V0ykEBAL6dBTVEHjCwxDxyJjPYq58SjKZr1bulNma7WbZIFz5vdKNB/rTyNDaPzjaY wOZw5heyQDGFJ6ayFhQQD5uevNku8+zNBTRBzzA13L4mr316uBz3DKOsaUCYR4WKuUHc TaaqxylEhyq04y/NFPttz8bsUQUA66L2vRK7T2rflxXF9yq9yMSktvMuuwRa/XR0BJwi ohIe/ZAbA7U9DjOhTSW8/dARVty2skr4GYY7Tvz6JMyvicase7TT3LU8bF4wqOPrJT0p EnzF5eiiwayThpzfttAPvaTETOJvg2kzy99LdyoLdP00Bm37maYrn/PrM/LxdyhnfVaU UJAQ== X-Gm-Message-State: AOAM530t1+BWwc+0BlitgMlHCWeVYkIKBoKlxnUb0FsbkDBiwtbSRn5J 55/hmU/c05Xqn0QAlaeQKVMvoIE5qYU= X-Google-Smtp-Source: ABdhPJwlbXDX326D0vhZ8+GuIcsjn3q+wTeoncPXufoZ7bzpy96uphxFcLGzOfusYiaFcRkHMcncag== X-Received: by 2002:adf:f711:: with SMTP id r17mr10493675wrp.358.1617291674502; Thu, 01 Apr 2021 08:41:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a14sm11308451wrg.84.2021.04.01.08.41.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:14 -0700 (PDT) Message-Id: <451563314d84f9d6dee29a4899b5d18033aa227d.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:52 +0000 Subject: [PATCH 10/23] fsmonitor--daemon: add pathname classification Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to classify relative and absolute pathnames and decide how they should be handled. This will be used by the platform-specific backend to respond to each filesystem event. When we register for filesystem notifications on a directory, we get events for everything (recursively) in the directory. We want to report to clients changes to tracked and untracked paths within the working directory. We do not want to report changes within the .git directory, for example. This classification will be used in a later commit by the different backends to classify paths as events are received. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 81 +++++++++++++++++++++++++++++++++++++ fsmonitor--daemon.h | 61 ++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 23a063707972..16252487240a 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -168,6 +168,87 @@ static int handle_client(void *data, const char *command, return result; } +#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" + +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *rel) +{ + if (fspathncmp(rel, ".git", 4)) + return IS_WORKDIR_PATH; + rel += 4; + + if (!*rel) + return IS_DOT_GIT; + if (*rel != '/') + return IS_WORKDIR_PATH; /* e.g. .gitignore */ + rel++; + + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX; + + return IS_INSIDE_DOT_GIT; +} + +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *rel) +{ + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX; + + return IS_INSIDE_GITDIR; +} + +static enum fsmonitor_path_type try_classify_workdir_abs_path( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + + if (fspathncmp(path, state->path_worktree_watch.buf, + state->path_worktree_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_worktree_watch.len; + + if (!*rel) + return IS_WORKDIR_PATH; /* it is the root dir exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_workdir_relative(rel); +} + +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + enum fsmonitor_path_type t; + + t = try_classify_workdir_abs_path(state, path); + if (state->nr_paths_watching == 1) + return t; + if (t != IS_OUTSIDE_CONE) + return t; + + if (fspathncmp(path, state->path_gitdir_watch.buf, + state->path_gitdir_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_gitdir_watch.len; + + if (!*rel) + return IS_GITDIR; /* it is the exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_gitdir_relative(rel); +} + static void *fsmonitor_fs_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 09e4a6fb6675..97ea3766e900 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -32,5 +32,66 @@ struct fsmonitor_daemon_state { int test_client_delay_ms; }; +/* + * Pathname classifications. + * + * The daemon classifies the pathnames that it receives from file + * system notification events into the following categories and uses + * that to decide whether clients are told about them. (And to watch + * for file system synchronization events.) + * + * The client should only care about paths within the working + * directory proper (inside the working directory and not ".git" nor + * inside of ".git/"). That is, the client has read the index and is + * asking for a list of any paths in the working directory that have + * been modified since the last token. The client does not care about + * file system changes within the .git directory (such as new loose + * objects or packfiles). So the client will only receive paths that + * are classified as IS_WORKDIR_PATH. + * + * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the + * exact ".git" directory or GITDIR. If the daemon receives a delete + * event for either of these directories, it will automatically + * shutdown, for example. + * + * Note that the daemon DOES NOT explicitly watch nor special case the + * ".git/index" file. The daemon does not read the index and does not + * have any internal index-relative state. The daemon only collects + * the set of modified paths within the working directory. + */ +enum fsmonitor_path_type { + IS_WORKDIR_PATH = 0, + + IS_DOT_GIT, + IS_INSIDE_DOT_GIT, + IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX, + + IS_GITDIR, + IS_INSIDE_GITDIR, + IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX, + + IS_OUTSIDE_CONE, +}; + +/* + * Classify a pathname relative to the root of the working directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *relative_path); + +/* + * Classify a pathname relative to a that is external to the + * worktree directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *relative_path); + +/* + * Classify an absolute pathname received from a filesystem event. + */ +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Thu Apr 1 15:40:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178673 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7867C433B4 for ; Thu, 1 Apr 2021 17:48:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A4EAD60234 for ; Thu, 1 Apr 2021 17:48:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234861AbhDARsz (ORCPT ); Thu, 1 Apr 2021 13:48:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235119AbhDARqZ (ORCPT ); Thu, 1 Apr 2021 13:46:25 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4E14DC02D541 for ; Thu, 1 Apr 2021 08:41:16 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id d8-20020a1c1d080000b029010f15546281so3065893wmd.4 for ; Thu, 01 Apr 2021 08:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=A48rjjmi3yLdIeRpj2+x6awEGfsdEaz2VsqzaND5D1I=; b=Ckbmd/NS2vRFWi/umYmnlO2dKtSmJQk4XgqQT8DgaU8/WCdFl1xwbIsNzurU68B9Zh +n2JWqzlupH3EjKLb8P7o5thi0tvl8Y29C0LY+5Gs9ohvVce0J1+30cmkOF7sPc29T9c /2fLEGC9U1WuQjDTd9wXw9y23iu+WmmOK89taJxLfmPbEYK6v7y4A4AH0M8bIl5pQgkT lAZ+Ttrzqq2kJ9wtSVf+srtAAU6UVmyJt3zBKKR8vndq2guxEpiJEYJlLEUyz4197bSR TxfiNn4i9EKyVei7ftKQjiv4GqEdnMrqpHMNtj2mBlm3fqqbSuM2jIm+84vJ/IwSG+ix /SeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=A48rjjmi3yLdIeRpj2+x6awEGfsdEaz2VsqzaND5D1I=; b=iNV8BJlgl7E3IB9fnfJYTUYbzr4EMlaL78IcaKjLumeJt7K6bqQWsop7ZFe+BGWnKe NdTGXIOUCRz8d8Kzg3crCjAD2UROI7BhvHxB2y25DcAvuufg+Tc8sqxLWuQq3fodapAv Z2TowpRvbTyF0qej+IGSHVdEAiKo37pv2m0XpaSmXbLzlEvS/8y0/uJ0zRwDzx5/vy84 UItjkGpzFXhpetR35QSvpC9fwi674KatjE+7lZWxApR84rtfqoqJclOksrBjcpKwEbUH zIFOkd2pIv2MzXJNn5KauwyGqYfsWRyluNQF7TYEE8zqsAQsXJUqnk4H9A4K1jYyMxR4 lnRg== X-Gm-Message-State: AOAM530PoGoCVk9DG/gzpgP7FEvAcncuFaNCUe9WZqXXkMDZfN/kJiWl ZjuZvWr3DuWMkbK7jUukEhV4tNx47I4= X-Google-Smtp-Source: ABdhPJznvEfZYdyhOgE0n5uoRNF6OV5V9vCU1pdlaTUeuXJf/x79lZ0OgTRJRf1tVmq67U28ThiImQ== X-Received: by 2002:a1c:4145:: with SMTP id o66mr8846002wma.68.1617291675041; Thu, 01 Apr 2021 08:41:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r1sm13538513wrj.63.2021.04.01.08.41.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:14 -0700 (PDT) Message-Id: <304fe03034f8622aa8728d8872cc9bc539bab861.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:53 +0000 Subject: [PATCH 11/23] fsmonitor--daemon: define token-ids Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to create token-ids and define the overall token naming scheme. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 108 +++++++++++++++++++++++++++++++++++- 1 file changed, 107 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 16252487240a..2d25e36601fe 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -149,6 +149,112 @@ static int do_as_client__send_flush(void) return 0; } +/* + * Requests to and from a FSMonitor Protocol V2 provider use an opaque + * "token" as a virtual timestamp. Clients can request a summary of all + * created/deleted/modified files relative to a token. In the response, + * clients receive a new token for the next (relative) request. + * + * + * Token Format + * ============ + * + * The contents of the token are private and provider-specific. + * + * For the built-in fsmonitor--daemon, we define a token as follows: + * + * "builtin" ":" ":" + * + * The is an arbitrary OPAQUE string, such as a GUID, + * UUID, or {timestamp,pid}. It is used to group all filesystem + * events that happened while the daemon was monitoring (and in-sync + * with the filesystem). + * + * Unlike FSMonitor Protocol V1, it is not defined as a timestamp + * and does not define less-than/greater-than relationships. + * (There are too many race conditions to rely on file system + * event timestamps.) + * + * The is a simple integer incremented for each event + * received. When a new is created, the is + * reset to zero. + * + * + * About Token Ids + * =============== + * + * A new token_id is created: + * + * [1] each time the daemon is started. + * + * [2] any time that the daemon must re-sync with the filesystem + * (such as when the kernel drops or we miss events on a very + * active volume). + * + * [3] in response to a client "flush" command (for dropped event + * testing). + * + * [4] MAYBE We might want to change the token_id after very complex + * filesystem operations are performed, such as a directory move + * sequence that affects many files within. It might be simpler + * to just give up and fake a re-sync (and let the client do a + * full scan) than try to enumerate the effects of such a change. + * + * When a new token_id is created, the daemon is free to discard all + * cached filesystem events associated with any previous token_ids. + * Events associated with a non-current token_id will never be sent + * to a client. A token_id change implicitly means that the daemon + * has gap in its event history. + * + * Therefore, clients that present a token with a stale (non-current) + * token_id will always be given a trivial response. + */ +struct fsmonitor_token_data { + struct strbuf token_id; + struct fsmonitor_batch *batch_head; + struct fsmonitor_batch *batch_tail; + uint64_t client_ref_count; +}; + +static struct fsmonitor_token_data *fsmonitor_new_token_data(void) +{ + static int test_env_value = -1; + static uint64_t flush_count = 0; + struct fsmonitor_token_data *token; + + token = (struct fsmonitor_token_data *)xcalloc(1, sizeof(*token)); + + strbuf_init(&token->token_id, 0); + token->batch_head = NULL; + token->batch_tail = NULL; + token->client_ref_count = 0; + + if (test_env_value < 0) + test_env_value = git_env_bool("GIT_TEST_FSMONITOR_TOKEN", 0); + + if (!test_env_value) { + struct timeval tv; + struct tm tm; + time_t secs; + + gettimeofday(&tv, NULL); + secs = tv.tv_sec; + gmtime_r(&secs, &tm); + + strbuf_addf(&token->token_id, + "%"PRIu64".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ", + flush_count++, + getpid(), + tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, + tm.tm_hour, tm.tm_min, tm.tm_sec, + (long)tv.tv_usec); + } else { + strbuf_addf(&token->token_id, "test_%08x", test_env_value++); + } + + return token; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, const char *command, @@ -330,7 +436,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); state.error_code = 0; - state.current_token_data = NULL; + state.current_token_data = fsmonitor_new_token_data(); state.test_client_delay_ms = 0; /* Prepare to (recursively) watch the directory. */ From patchwork Thu Apr 1 15:40:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6AB8C43617 for ; Thu, 1 Apr 2021 17:48:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A557360FF3 for ; Thu, 1 Apr 2021 17:48:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235874AbhDARr7 (ORCPT ); Thu, 1 Apr 2021 13:47:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58392 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235879AbhDARnR (ORCPT ); Thu, 1 Apr 2021 13:43:17 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D53D4C02D542 for ; Thu, 1 Apr 2021 08:41:16 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id u5-20020a7bcb050000b029010e9316b9d5so1131720wmj.2 for ; Thu, 01 Apr 2021 08:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vKFtf8iU/88j++X/TQINz0X5kBgggdAz2WOyyunvvEs=; b=gYFGN3I9pf2R1hV4ottALnrGTfk+f/yrPlxFUZYEpvOIndDoJxs6zUZ0tcXRi6aD/Z YVShIOnB3U0+wqvoDBiXl0IQ2ICPTs7HZRXPFw26p057gyAP/TgeGG4EQY66ANmP2sLV 3MslXiHABYbpuQyF0/GEl5z4rntXugVjMH0qckpk/HT3AFmQzp7ByaqaL/HPFBrSfjya c+nAIfXcIDw7ZtNn7+AJqqDKzdfhqVdnZhJd90MlqWws33T6VaczhAn+JYMboiKBhtaS jwRFQ7sr9XU43aeKUkgwjhPXouJhAK2SsKcF5JIq90y9RSAIi7MMXAaDeSxyZNozHhnj O5Lw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vKFtf8iU/88j++X/TQINz0X5kBgggdAz2WOyyunvvEs=; b=uJmJY6q6k2YdjnQGxIvw+u9Gw6jdV4mRSQ/33kZ+2MREkidywV/QGmbacpw/qgULq3 Ra/BFfZAtI0pQ/Codqan7eXXKDYNd4L8rUYhv+cTa3hfGpCW3L97LBfFGT9ujjSmX3b4 iho6TKTdPRYxz3s0LhqEaC0PwGf5H1JEQhdVIPorsnTGwBLztNE2Tz7XJVKq4I6DB+ZO 07nl4VF2sRSOXj3EQzicRnmIFkvaxoM3o97BbkPgK+D/8ldJet3k7W4zB8UW9Elir8Ep RlB7LpuUcgkVmXRaSxaihBqdAKIAt0aP2zE8mmQvxJmRjass43cTgnuxGZlW+aUpE0/o V3BQ== X-Gm-Message-State: AOAM533eq0NwKPjLi8rIAFlbLbs7wIStLCtNcOdlhwsljL/Q3avRMKAp EfWsiv5pEN6XalDQehwNv04b718Agts= X-Google-Smtp-Source: ABdhPJxd81Uj+F5Bhrf4UXgjx01ivL3Fkvb0MQdBSUu56QRLMp8LB+bzmEcyeDsmGd56Ftar2dfWZQ== X-Received: by 2002:a7b:cb90:: with SMTP id m16mr8651825wmi.162.1617291675611; Thu, 01 Apr 2021 08:41:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v18sm10737642wru.85.2021.04.01.08.41.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:54 +0000 Subject: [PATCH 12/23] fsmonitor--daemon: create token-based changed path cache Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to build lists of changed paths and associate them with a token-id. This will be used by the platform-specific backends to accumulate changed paths in response to filesystem events. The platform-specific event loops receive batches containing one or more changed paths. Their fs listener thread will accumulate them in a `fsmonitor_batch` (and without locking) and then "publish" them to associate them with the current token and to make them visible to the client worker threads. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 192 ++++++++++++++++++++++++++++++++++++ fsmonitor--daemon.h | 40 ++++++++ 2 files changed, 232 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 2d25e36601fe..48071d445c49 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -255,6 +255,120 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void) return token; } +struct fsmonitor_batch { + struct fsmonitor_batch *next; + uint64_t batch_seq_nr; + const char **interned_paths; + size_t nr, alloc; + time_t pinned_time; +}; + +struct fsmonitor_batch *fsmonitor_batch__new(void) +{ + struct fsmonitor_batch *batch = xcalloc(1, sizeof(*batch)); + + return batch; +} + +struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch) +{ + struct fsmonitor_batch *next; + + if (!batch) + return NULL; + + next = batch->next; + + /* + * The actual strings within the array are interned, so we don't + * own them. + */ + free(batch->interned_paths); + + return next; +} + +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, + const char *path) +{ + const char *interned_path = strintern(path); + + trace_printf_key(&trace_fsmonitor, "event: %s", interned_path); + + ALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc); + batch->interned_paths[batch->nr++] = interned_path; +} + +static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, + const struct fsmonitor_batch *batch_src) +{ + /* assert state->main_lock */ + + size_t k; + + ALLOC_GROW(batch_dest->interned_paths, + batch_dest->nr + batch_src->nr + 1, + batch_dest->alloc); + + for (k = 0; k < batch_src->nr; k++) + batch_dest->interned_paths[batch_dest->nr++] = + batch_src->interned_paths[k]; +} + +static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) +{ + struct fsmonitor_batch *p; + + if (!token) + return; + + assert(token->client_ref_count == 0); + + strbuf_release(&token->token_id); + + for (p = token->batch_head; p; p = fsmonitor_batch__free(p)) + ; + + free(token); +} + +/* + * Flush all of our cached data about the filesystem. Call this if we + * lose sync with the filesystem and miss some notification events. + * + * [1] If we are missing events, then we no longer have a complete + * history of the directory (relative to our current start token). + * We should create a new token and start fresh (as if we just + * booted up). + * + * If there are no readers of the the current token data series, we + * can free it now. Otherwise, let the last reader free it. Either + * way, the old token data series is no longer associated with our + * state data. + */ +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_token_data *free_me = NULL; + struct fsmonitor_token_data *new_one = NULL; + + new_one = fsmonitor_new_token_data(); + + pthread_mutex_lock(&state->main_lock); + + trace_printf_key(&trace_fsmonitor, + "force resync [old '%s'][new '%s']", + state->current_token_data->token_id.buf, + new_one->token_id.buf); + + if (state->current_token_data->client_ref_count == 0) + free_me = state->current_token_data; + state->current_token_data = new_one; + + pthread_mutex_unlock(&state->main_lock); + + fsmonitor_free_token_data(free_me); +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, const char *command, @@ -355,6 +469,77 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( return fsmonitor_classify_path_gitdir_relative(rel); } +/* + * We try to combine small batches at the front of the batch-list to avoid + * having a long list. This hopefully makes it a little easier when we want + * to truncate and maintain the list. However, we don't want the paths array + * to just keep growing and growing with realloc, so we insert an arbitrary + * limit. + */ +#define MY_COMBINE_LIMIT (1024) + +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names) +{ + if (!batch && !cookie_names->nr) + return; + + pthread_mutex_lock(&state->main_lock); + + if (batch) { + struct fsmonitor_batch *head; + + head = state->current_token_data->batch_head; + if (!head) { + batch->batch_seq_nr = 0; + batch->next = NULL; + state->current_token_data->batch_head = batch; + state->current_token_data->batch_tail = batch; + } else if (head->pinned_time) { + /* + * We cannot alter the current batch list + * because: + * + * [a] it is being transmitted to at least one + * client and the handle_client() thread has a + * ref-count, but not a lock on the batch list + * starting with this item. + * + * [b] it has been transmitted in the past to + * at least one client such that future + * requests are relative to this head batch. + * + * So, we can only prepend a new batch onto + * the front of the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else if (head->nr + batch->nr > MY_COMBINE_LIMIT) { + /* + * The head batch in the list has never been + * transmitted to a client, but folding the + * contents of the new batch onto it would + * exceed our arbitrary limit, so just prepend + * the new batch onto the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else { + /* + * We are free to append the paths in the given + * batch onto the end of the current head batch. + */ + fsmonitor_batch__combine(head, batch); + fsmonitor_batch__free(batch); + } + } + + pthread_mutex_unlock(&state->main_lock); +} + static void *fsmonitor_fs_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -369,6 +554,13 @@ static void *fsmonitor_fs_listen__thread_proc(void *_state) fsmonitor_fs_listen__loop(state); + pthread_mutex_lock(&state->main_lock); + if (state->current_token_data && + state->current_token_data->client_ref_count == 0) + fsmonitor_free_token_data(state->current_token_data); + state->current_token_data = NULL; + pthread_mutex_unlock(&state->main_lock); + trace2_thread_exit(); return NULL; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 97ea3766e900..06563b6ed56c 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -12,6 +12,27 @@ struct fsmonitor_batch; struct fsmonitor_token_data; +/* + * Create a new batch of path(s). The returned batch is considered + * private and not linked into the fsmonitor daemon state. The caller + * should fill this batch with one or more paths and then publish it. + */ +struct fsmonitor_batch *fsmonitor_batch__new(void); + +/* + * Free this batch and return the value of the batch->next field. + */ +struct fsmonitor_batch *fsmonitor_batch__free(struct fsmonitor_batch *batch); + +/* + * Add this path to this batch of modified files. + * + * The batch should be private and NOT (yet) linked into the fsmonitor + * daemon state and therefore not yet visible to worker threads and so + * no locking is required. + */ +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); + struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ struct fsmonitor_daemon_state { @@ -93,5 +114,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( struct fsmonitor_daemon_state *state, const char *path); +/* + * Prepend the this batch of path(s) onto the list of batches associated + * with the current token. This makes the batch visible to worker threads. + * + * The caller no longer owns the batch and must not free it. + * + * Wake up the client threads waiting on these cookies. + */ +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names); + +/* + * If the platform-specific layer loses sync with the filesystem, + * it should call this to invalidate cached data and abort waiting + * threads. + */ +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Thu Apr 1 15:40:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178915 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E18A7C43462 for ; Thu, 1 Apr 2021 18:04:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B581361001 for ; Thu, 1 Apr 2021 18:04:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237671AbhDASEG (ORCPT ); Thu, 1 Apr 2021 14:04:06 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33010 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237012AbhDAR7U (ORCPT ); Thu, 1 Apr 2021 13:59:20 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72F90C02D543 for ; Thu, 1 Apr 2021 08:41:17 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id r10-20020a05600c35cab029010c946c95easo1125316wmq.4 for ; Thu, 01 Apr 2021 08:41:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pShdBCkC3E4HzgZsf0TnBiqhjkrMoEvuj5pmqjUft8s=; b=LE2rNy3a4hG6yDWCvq06AfMfQMeW7pbVJGffoiyFi1aX59sJF04IYLiZTO2RO+uGH/ 7UwbBycOSC+oOjPZEntdyaXPR8Pb+gscaMKP8nlCimSznOLN81E1GCj+f02Qxp4BK472 2XQp8wEdMkwuWDWNoxG0XMkfOCozo/sBJfOJwS/ms5pz+I5BomtpgHB+iUgjSjHo2IYW 00QiGzbtlNpbFQ/s/i+OW/5vWDEPtXVgqAf99mZZbbpcTlcVwaM+UuO2g3BUmQVHWyC8 /YG6ymcnhaUOwmdQSDFzwyzzDbGKlrGPohdI9ALBhbfuPK3n0pAUOCjgNxEnJYtgTL/a XhSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pShdBCkC3E4HzgZsf0TnBiqhjkrMoEvuj5pmqjUft8s=; b=lZCAoqgyJiBBcM/WsK7jfZelsnb7QWTenH5/iyQVzTEjFDis91qvkKepq4F8y1Svw3 HeRpL5CACuw0+EN1KmOFcfKynpD7U1C0WK9GwuQmOsz45Y5Gqfkq39Fr1XidXPirjGgE oY35M/Ca/SVnFDrBErWDAGn8z5gbFyZGI6FJLzHWVuuBUFj4oTW6Tcl7hLkXgS22s/kw cTUOaTegUL8kibQavsDgRmj7nx8lm0ThFFbXvF0qcTUgUoAlPu/caiJ2OCcChhoCK3b8 Km82m2TbeE2DssYTQjYKIh0zQANHNexmuKEuhVgatFY+M44K0FJcH7xTPmLHYDyY/9sC GrUw== X-Gm-Message-State: AOAM531LI1Ph0Fj7j3SP3fTLVRQbvCP/BYw+Qq+bva2ZOVO/PBDyDhzt NLwUnUtBDY3l82ixwe0Mxj3HzKNPpQE= X-Google-Smtp-Source: ABdhPJzNl6AoR9bOAxzeTFPgVFdDuKGW+YJi9eUjV1Yj63sF83v+inCqBhMEbIgOu8I+u0QyoOaEHA== X-Received: by 2002:a7b:c047:: with SMTP id u7mr8765854wmc.98.1617291676155; Thu, 01 Apr 2021 08:41:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h9sm8797603wmb.35.2021.04.01.08.41.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:55 +0000 Subject: [PATCH 13/23] fsmonitor-fs-listen-win32: implement FSMonitor backend on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the win32 backend to register a watch on the working tree root directory (recursively). Also watch the if it is not inside the working tree. And to collect path change notifications into batches and publish. Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-win32.c | 493 +++++++++++++++++++ 1 file changed, 493 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-win32.c b/compat/fsmonitor/fsmonitor-fs-listen-win32.c index 880446b49e35..2f1fcf85a0a4 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-win32.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-win32.c @@ -2,20 +2,513 @@ #include "config.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +/* + * The documentation of ReadDirectoryChangesW() states that the maximum + * buffer size is 64K when the monitored directory is remote. + * + * Larger buffers may be used when the monitored directory is local and + * will help us receive events faster from the kernel and avoid dropped + * events. + * + * So we try to use a very large buffer and silently fallback to 64K if + * we get an error. + */ +#define MAX_RDCW_BUF_FALLBACK (65536) +#define MAX_RDCW_BUF (65536 * 8) + +struct one_watch +{ + char buffer[MAX_RDCW_BUF]; + DWORD buf_len; + DWORD count; + + struct strbuf path; + HANDLE hDir; + HANDLE hEvent; + OVERLAPPED overlapped; + + /* + * Is there an active ReadDirectoryChangesW() call pending. If so, we + * need to later call GetOverlappedResult() and possibly CancelIoEx(). + */ + BOOL is_active; +}; + +struct fsmonitor_daemon_backend_data +{ + struct one_watch *watch_worktree; + struct one_watch *watch_gitdir; + + HANDLE hEventShutdown; + + HANDLE hListener[3]; /* we don't own these handles */ +#define LISTENER_SHUTDOWN 0 +#define LISTENER_HAVE_DATA_WORKTREE 1 +#define LISTENER_HAVE_DATA_GITDIR 2 + int nr_listener_handles; +}; + +/* + * Convert the WCHAR path from the notification into UTF8 and + * then normalize it. + */ +static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, + struct strbuf *normalized_path) +{ + int reserve; + int len = 0; + + strbuf_reset(normalized_path); + if (!info->FileNameLength) + goto normalize; + + /* + * Pre-reserve enough space in the UTF8 buffer for + * each Unicode WCHAR character to be mapped into a + * sequence of 2 UTF8 characters. That should let us + * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. + */ + reserve = info->FileNameLength + 1; + strbuf_grow(normalized_path, reserve); + + for (;;) { + len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, + info->FileNameLength / sizeof(WCHAR), + normalized_path->buf, + strbuf_avail(normalized_path) - 1, + NULL, NULL); + if (len > 0) + goto normalize; + if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { + error("[GLE %ld] could not convert path to UTF-8: '%.*ls'", + GetLastError(), + (int)(info->FileNameLength / sizeof(WCHAR)), + info->FileName); + return -1; + } + + strbuf_grow(normalized_path, + strbuf_avail(normalized_path) + reserve); + } + +normalize: + strbuf_setlen(normalized_path, len); + return strbuf_normalize_path(normalized_path); +} void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) { + SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); +} + +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, + const char *path) +{ + struct one_watch *watch = NULL; + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + + hDir = CreateFileA(path, + desired_access, share_mode, NULL, OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, + NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] could not watch '%s'"), + GetLastError(), path); + return NULL; + } + + watch = xcalloc(1, sizeof(*watch)); + + watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ + + strbuf_init(&watch->path, 0); + strbuf_addstr(&watch->path, path); + + watch->hDir = hDir; + watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); + + return watch; +} + +static void destroy_watch(struct one_watch *watch) +{ + if (!watch) + return; + + strbuf_release(&watch->path); + if (watch->hDir != INVALID_HANDLE_VALUE) + CloseHandle(watch->hDir); + if (watch->hEvent != INVALID_HANDLE_VALUE) + CloseHandle(watch->hEvent); + + free(watch); +} + +static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, + struct one_watch *watch) +{ + DWORD dwNotifyFilter = + FILE_NOTIFY_CHANGE_FILE_NAME | + FILE_NOTIFY_CHANGE_DIR_NAME | + FILE_NOTIFY_CHANGE_ATTRIBUTES | + FILE_NOTIFY_CHANGE_SIZE | + FILE_NOTIFY_CHANGE_LAST_WRITE | + FILE_NOTIFY_CHANGE_CREATION; + + ResetEvent(watch->hEvent); + + memset(&watch->overlapped, 0, sizeof(watch->overlapped)); + watch->overlapped.hEvent = watch->hEvent; + +start_watch: + watch->is_active = ReadDirectoryChangesW( + watch->hDir, watch->buffer, watch->buf_len, TRUE, + dwNotifyFilter, &watch->count, &watch->overlapped, NULL); + + if (!watch->is_active && + GetLastError() == ERROR_INVALID_PARAMETER && + watch->buf_len > MAX_RDCW_BUF_FALLBACK) { + watch->buf_len = MAX_RDCW_BUF_FALLBACK; + goto start_watch; + } + + if (watch->is_active) + return 0; + + error("ReadDirectoryChangedW failed on '%s' [GLE %ld]", + watch->path.buf, GetLastError()); + return -1; +} + +static int recv_rdcw_watch(struct one_watch *watch) +{ + watch->is_active = FALSE; + + if (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count, + TRUE)) + return 0; + + // TODO If an external is deleted, the above returns an error. + // TODO I'm not sure that there's anything that we can do here other + // TODO than failing -- the /.git link file would be broken + // TODO anyway. We might try to check for that and return a better + // TODO error message. + + error("GetOverlappedResult failed on '%s' [GLE %ld]", + watch->path.buf, GetLastError()); + return -1; +} + +static void cancel_rdcw_watch(struct one_watch *watch) +{ + DWORD count; + + if (!watch || !watch->is_active) + return; + + CancelIoEx(watch->hDir, &watch->overlapped); + GetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE); + watch->is_active = FALSE; +} + +/* + * Process filesystem events that happen anywhere (recursively) under the + * root directory. For a normal working directory, this includes + * both version controlled files and the contents of the .git/ directory. + * + * If /.git is a file, then we only see events for the file + * itself. + */ +static int process_worktree_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_worktree; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + struct fsmonitor_batch *batch = NULL; + const char *p = watch->buffer; + + /* + * If the kernel gets more events than will fit in the kernel + * buffer associated with our RDCW handle, it drops them and + * returns a count of zero. (A successful call, but with + * length zero.) + */ + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_WORKTREE; + } + + /* + * On Windows, `info` contains an "array" of paths that are + * relative to the root of whichever directory handle received + * the event. + */ + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_workdir_relative(path.buf); + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info->Action == FILE_ACTION_REMOVED) || + (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, path.buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + goto skip_this_path; + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, batch, &cookie_list); + batch = NULL; + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_WORKTREE; + +force_shutdown: + fsmonitor_batch__free(batch); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_SHUTDOWN; +} + +/* + * Process filesystem events that happend anywhere (recursively) under the + * external (such as non-primary worktrees or submodules). + * We only care about cookie files that our client threads created here. + * + * Note that we DO NOT get filesystem events on the external + * itself (it is not inside something that we are watching). In particular, + * we do not get an event if the external is deleted. + */ +static int process_gitdir_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_gitdir; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *p = watch->buffer; + + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_GITDIR; + } + + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_gitdir_relative(path.buf); + + trace_printf_key(&trace_fsmonitor, "BBB: %s", path.buf); + + switch (t) { + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_GITDIR: + goto skip_this_path; + + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + goto skip_this_path; + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, NULL, &cookie_list); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_GITDIR; } void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data = state->backend_data; + DWORD dwWait; + + state->error_code = 0; + + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + + if (data->watch_gitdir && + start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + + for (;;) { + dwWait = WaitForMultipleObjects(data->nr_listener_handles, + data->hListener, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) { + if (recv_rdcw_watch(data->watch_worktree) == -1) + goto force_error_stop; + if (process_worktree_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) { + if (recv_rdcw_watch(data->watch_gitdir) == -1) + goto force_error_stop; + if (process_gitdir_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN) + goto clean_shutdown; + + error(_("could not read directory changes [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->error_code = -1; + +force_shutdown: + /* + * Tell the IPC thead pool to stop (which completes the await + * in the main thread (which will also signal this thread (if + * we are still alive))). + */ + ipc_server_stop_async(state->ipc_server_data); + +clean_shutdown: + cancel_rdcw_watch(data->watch_worktree); + cancel_rdcw_watch(data->watch_gitdir); } int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = xcalloc(1, sizeof(*data)); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->watch_worktree = create_watch(state, + state->path_worktree_watch.buf); + if (!data->watch_worktree) + goto failed; + + if (state->nr_paths_watching > 1) { + data->watch_gitdir = create_watch(state, + state->path_gitdir_watch.buf); + if (!data->watch_gitdir) + goto failed; + } + + data->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown; + data->nr_listener_handles++; + + data->hListener[LISTENER_HAVE_DATA_WORKTREE] = + data->watch_worktree->hEvent; + data->nr_listener_handles++; + + if (data->watch_gitdir) { + data->hListener[LISTENER_HAVE_DATA_GITDIR] = + data->watch_gitdir->hEvent; + data->nr_listener_handles++; + } + + state->backend_data = data; + return 0; + +failed: + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + return -1; } void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + + FREE_AND_NULL(state->backend_data); } From patchwork Thu Apr 1 15:40:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7191EC43611 for ; Thu, 1 Apr 2021 17:43:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1192B613B5 for ; Thu, 1 Apr 2021 17:43:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235893AbhDARnS (ORCPT ); Thu, 1 Apr 2021 13:43:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234808AbhDARkT (ORCPT ); Thu, 1 Apr 2021 13:40:19 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0E674C02D544 for ; Thu, 1 Apr 2021 08:41:18 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id b2-20020a7bc2420000b029010be1081172so1136055wmj.1 for ; Thu, 01 Apr 2021 08:41:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=edBI/UfAdUANph40tPYHwXApM7BGoIz/01tZ6bP76xw=; b=DhIhqghFlINC+pvk7RCv09N6IYN/hdZNh2FSs5VmvuwHxSb/1SMJ1bw96J5W3GGEOC UMPyCw4vycKrXVLSgJ4J/HhGmaMegoZb6eQnZIlUBs2ZMZdCIEB8DiiBUu79I0/5CzhH 8vyL71ZMy8ctHNmLTCKQx7jwDWsgab7FpkAMyBYImj331gceB7Vjd1NgjBICFnJfOiyg T7OuZIlYfiODgYjqNx7Ry/3hbLGcQBL0m3mSCTTUZ4j699j+zNOV6BqMRgBbJ4YvGrvA 1EoaKnPIW8SROe+DKS1fd5fWCgbK/lFB9HIQjtMAo3W8HXh1XVJaUSr7lhz9CipQMcxV a8ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=edBI/UfAdUANph40tPYHwXApM7BGoIz/01tZ6bP76xw=; b=ofgt4HsC9Y5R5xk1r3PRdMqUN/qP1xIxgR3BrRZJrWVD+XsJUkXoUqPJaQZl3TXU/L 9CtzacL5X9cFVEx4sbnYFJF6PJIkb4b65YiV+IzHbwcuRJj3KhlItglRXL1lD4QJ7/oR SlP1HSMBzowH/QMWlhevo/i0lmiZKCd3Sl+MChyvUkz26Y1rRsyALiHkehOjWeUdyc/k PjA0rPDZOAiEzBc5+MUy70MnboLNRbtnbPppsHXdKQMvPgu+r9eAeVh9639R0Ue42w4m DBLcSN7DDD+Yloh/1+skqbv9PebEvxKweyrRzG1LzA70E1+fRi3NzpJzB0zc+69sK93H Jfug== X-Gm-Message-State: AOAM5339gmt543B90O3q/q4xp0adCnpEkTv6pt7XybjUMI5/5mI97d20 LwL99+f0eEYMAeULHPSzOh0tZrehl1s= X-Google-Smtp-Source: ABdhPJyaewqJJospHN4hgv187lTdMBHa9KbqBVbIldPPomDmWXrJ9RjkYPQ7P5dq+ShkbfdtKw6svg== X-Received: by 2002:a7b:cb05:: with SMTP id u5mr8737568wmj.21.1617291676795; Thu, 01 Apr 2021 08:41:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e17sm12423221wra.65.2021.04.01.08.41.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:16 -0700 (PDT) Message-Id: <67fa7c7b8ac79b05d0773dc0affbe39b77aa4893.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:56 +0000 Subject: [PATCH 14/23] fsmonitor-fs-listen-macos: add macos header files for FSEvent Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Include MacOS system declarations to allow us to use FSEvent and CoreFoundation APIs. We need GCC and clang versions because of compiler and header file conflicts. While it is quite possible to #include Apple's CoreServices.h when compiling C source code with clang, trying to build it with GCC currently fails with this error: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/AuthSession.h:32, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Security.h:42, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/CSIdentity.h:43, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/OSServices.framework/Headers/OSServices.h:29, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/IconsCore.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Headers/LaunchServices.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45, /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope 193 | char bytes[kAuthorizationExternalFormLength]; | ^~~~~ The underlying reason is that GCC (rightfully) objects that an `enum` value such as `kAuthorizationExternalFormLength` is not a constant (because it is not, the preprocessor has no knowledge of it, only the actual C compiler does) and can therefore not be used to define the size of a C array. This is a known problem and tracked in GCC's bug tracker: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 In the meantime, let's not block things and go the slightly ugly route of declaring/defining the FSEvents constants, data structures and functions that we need, so that we can avoid above-mentioned issue. Let's do this _only_ for GCC, though, so that the CI/PR builds (which build both with clang and with GCC) can guarantee that we _are_ using the correct data types. Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 96 ++++++++++++++++++++ 1 file changed, 96 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c index b91058d1c4f8..bec5130d9e1d 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -1,3 +1,99 @@ +#if defined(__GNUC__) +/* + * It is possible to #include CoreFoundation/CoreFoundation.h when compiling + * with clang, but not with GCC as of time of writing. + * + * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details. + */ +typedef unsigned int FSEventStreamCreateFlags; +#define kFSEventStreamEventFlagNone 0x00000000 +#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001 +#define kFSEventStreamEventFlagUserDropped 0x00000002 +#define kFSEventStreamEventFlagKernelDropped 0x00000004 +#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008 +#define kFSEventStreamEventFlagHistoryDone 0x00000010 +#define kFSEventStreamEventFlagRootChanged 0x00000020 +#define kFSEventStreamEventFlagMount 0x00000040 +#define kFSEventStreamEventFlagUnmount 0x00000080 +#define kFSEventStreamEventFlagItemCreated 0x00000100 +#define kFSEventStreamEventFlagItemRemoved 0x00000200 +#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400 +#define kFSEventStreamEventFlagItemRenamed 0x00000800 +#define kFSEventStreamEventFlagItemModified 0x00001000 +#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000 +#define kFSEventStreamEventFlagItemChangeOwner 0x00004000 +#define kFSEventStreamEventFlagItemXattrMod 0x00008000 +#define kFSEventStreamEventFlagItemIsFile 0x00010000 +#define kFSEventStreamEventFlagItemIsDir 0x00020000 +#define kFSEventStreamEventFlagItemIsSymlink 0x00040000 +#define kFSEventStreamEventFlagOwnEvent 0x00080000 +#define kFSEventStreamEventFlagItemIsHardlink 0x00100000 +#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000 +#define kFSEventStreamEventFlagItemCloned 0x00400000 + +typedef struct __FSEventStream *FSEventStreamRef; +typedef const FSEventStreamRef ConstFSEventStreamRef; + +typedef unsigned int CFStringEncoding; +#define kCFStringEncodingUTF8 0x08000100 + +typedef const struct __CFString *CFStringRef; +typedef const struct __CFArray *CFArrayRef; +typedef const struct __CFRunLoop *CFRunLoopRef; + +struct FSEventStreamContext { + long long version; + void *cb_data, *retain, *release, *copy_description; +}; + +typedef struct FSEventStreamContext FSEventStreamContext; +typedef unsigned int FSEventStreamEventFlags; +#define kFSEventStreamCreateFlagNoDefer 0x02 +#define kFSEventStreamCreateFlagWatchRoot 0x04 +#define kFSEventStreamCreateFlagFileEvents 0x10 + +typedef unsigned long long FSEventStreamEventId; +#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL + +typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef, + void *context, + __SIZE_TYPE__ num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]); +typedef double CFTimeInterval; +FSEventStreamRef FSEventStreamCreate(void *allocator, + FSEventStreamCallback callback, + FSEventStreamContext *context, + CFArrayRef paths_to_watch, + FSEventStreamEventId since_when, + CFTimeInterval latency, + FSEventStreamCreateFlags flags); +CFStringRef CFStringCreateWithCString(void *allocator, const char *string, + CFStringEncoding encoding); +CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count, + void *callbacks); +void CFRunLoopRun(void); +void CFRunLoopStop(CFRunLoopRef run_loop); +CFRunLoopRef CFRunLoopGetCurrent(void); +extern CFStringRef kCFRunLoopDefaultMode; +void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream, + CFRunLoopRef run_loop, + CFStringRef run_loop_mode); +unsigned char FSEventStreamStart(FSEventStreamRef stream); +void FSEventStreamStop(FSEventStreamRef stream); +void FSEventStreamInvalidate(FSEventStreamRef stream); +void FSEventStreamRelease(FSEventStreamRef stream); +#else +/* + * Let Apple's headers declare `isalnum()` first, before + * Git's headers override it via a constant + */ +#include +#include +#include +#endif + #include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" From patchwork Thu Apr 1 15:40:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C05C7C43611 for ; Thu, 1 Apr 2021 17:47:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 77AF161284 for ; Thu, 1 Apr 2021 17:47:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236308AbhDARr5 (ORCPT ); Thu, 1 Apr 2021 13:47:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58360 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234533AbhDARmN (ORCPT ); Thu, 1 Apr 2021 13:42:13 -0400 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C2AB1C02D545 for ; Thu, 1 Apr 2021 08:41:18 -0700 (PDT) Received: by mail-wm1-x330.google.com with SMTP id p19so1272316wmq.1 for ; Thu, 01 Apr 2021 08:41:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=YeAUI6u+REX12lsdfeZEptcjViUJU8DCgHxwyEQq4Vw=; b=SrM6rpOu3Qvgm7qOT9bxgupXxzH8NW4CHvl3Wza5TWpp2cAmKef61gDuKg1R9N+k7B UazXWVrZYliLnS1kPoUlTXQKBcIdS6caxTLXq9Q3Y2F99AG/n9dqHKs2vJ50dqlhXX8L G8cPw9rsi1Yo7MmTJ3VpxtYOZxXP9VK8RJr4sikf6/p4c2RCL54NrrefSqYl/yi+fV1j ptFyaScc6ikfkKyH8yPRnJGqYi2+jLwV6kOwY499LWPWnxzlQ2cJ/pB16h1gfL0b7lN3 aP19MfpBO+vo/BGVII4VWy8KZpfBCe7auaAmFYR+emEcD2SBdK1BVjtYKujLPQTfYqlE XsuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=YeAUI6u+REX12lsdfeZEptcjViUJU8DCgHxwyEQq4Vw=; b=GxA0XH6id7BxzdTIitJHz8yrT93Q7gx9glBH/0v2hLQWDXtXdcyrZBivr9OzGGY/6D oyh9qrzNuyXGL38PpoZaXHNdNFWkAZqdWr56U5kuzvr2D1qavE2zcNved7ZcqNNd00ru zadJL+bOJ2VCIEEfnRQCkD4gqw8mAlolvLPOferMgZlhtAnLHDj05ebFLHHf/7tz60XF VPVU9ph9ALdtdoV/OBC8Un2R2Y2TkqYhMbqk2Ry9x2LblP4/REGqhkOzSNgIE33aSKfd sRtx40ac6pldIbLf2Mvj1pkyZLDhGA/lJcEWSVw9V7Deso5XurrE535hvKmd7Msj1HST NzVA== X-Gm-Message-State: AOAM532M85wHK0Bdu3SVWsNs61z70md1dusq4ArtJMhdHSoWdolUzTBc z8Atd3iqX0bK04KkRkDsxx91KVOYQlo= X-Google-Smtp-Source: ABdhPJyjd7u2uyp85hNUxnA+0KvCHbzqJX2yMnMs9SA/6Am4YADySHSPSV4CAAoZrbUrw8n0+RK60g== X-Received: by 2002:a05:600c:252:: with SMTP id 18mr8487924wmj.67.1617291677431; Thu, 01 Apr 2021 08:41:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n4sm8106090wmq.40.2021.04.01.08.41.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:17 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:57 +0000 Subject: [PATCH 15/23] fsmonitor-fs-listen-macos: implement FSEvent listener on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement file system event listener on MacOS using FSEvent, CoreFoundation, and CoreServices. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- compat/fsmonitor/fsmonitor-fs-listen-macos.c | 368 +++++++++++++++++++ 1 file changed, 368 insertions(+) diff --git a/compat/fsmonitor/fsmonitor-fs-listen-macos.c b/compat/fsmonitor/fsmonitor-fs-listen-macos.c index bec5130d9e1d..e055fb579cc4 100644 --- a/compat/fsmonitor/fsmonitor-fs-listen-macos.c +++ b/compat/fsmonitor/fsmonitor-fs-listen-macos.c @@ -97,20 +97,388 @@ void FSEventStreamRelease(FSEventStreamRef stream); #include "cache.h" #include "fsmonitor.h" #include "fsmonitor-fs-listen.h" +#include "fsmonitor--daemon.h" + +struct fsmonitor_daemon_backend_data +{ + CFStringRef cfsr_worktree_path; + CFStringRef cfsr_gitdir_path; + + CFArrayRef cfar_paths_to_watch; + int nr_paths_watching; + + FSEventStreamRef stream; + + CFRunLoopRef rl; + + enum shutdown_style { + SHUTDOWN_EVENT = 0, + FORCE_SHUTDOWN, + FORCE_ERROR_STOP, + } shutdown_style; + + unsigned int stream_scheduled:1; + unsigned int stream_started:1; +}; + +static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) +{ + struct strbuf msg = STRBUF_INIT; + + if (flag & kFSEventStreamEventFlagMustScanSubDirs) + strbuf_addstr(&msg, "MustScanSubDirs|"); + if (flag & kFSEventStreamEventFlagUserDropped) + strbuf_addstr(&msg, "UserDropped|"); + if (flag & kFSEventStreamEventFlagKernelDropped) + strbuf_addstr(&msg, "KernelDropped|"); + if (flag & kFSEventStreamEventFlagEventIdsWrapped) + strbuf_addstr(&msg, "EventIdsWrapped|"); + if (flag & kFSEventStreamEventFlagHistoryDone) + strbuf_addstr(&msg, "HistoryDone|"); + if (flag & kFSEventStreamEventFlagRootChanged) + strbuf_addstr(&msg, "RootChanged|"); + if (flag & kFSEventStreamEventFlagMount) + strbuf_addstr(&msg, "Mount|"); + if (flag & kFSEventStreamEventFlagUnmount) + strbuf_addstr(&msg, "Unmount|"); + if (flag & kFSEventStreamEventFlagItemChangeOwner) + strbuf_addstr(&msg, "ItemChangeOwner|"); + if (flag & kFSEventStreamEventFlagItemCreated) + strbuf_addstr(&msg, "ItemCreated|"); + if (flag & kFSEventStreamEventFlagItemFinderInfoMod) + strbuf_addstr(&msg, "ItemFinderInfoMod|"); + if (flag & kFSEventStreamEventFlagItemInodeMetaMod) + strbuf_addstr(&msg, "ItemInodeMetaMod|"); + if (flag & kFSEventStreamEventFlagItemIsDir) + strbuf_addstr(&msg, "ItemIsDir|"); + if (flag & kFSEventStreamEventFlagItemIsFile) + strbuf_addstr(&msg, "ItemIsFile|"); + if (flag & kFSEventStreamEventFlagItemIsHardlink) + strbuf_addstr(&msg, "ItemIsHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsLastHardlink) + strbuf_addstr(&msg, "ItemIsLastHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsSymlink) + strbuf_addstr(&msg, "ItemIsSymlink|"); + if (flag & kFSEventStreamEventFlagItemModified) + strbuf_addstr(&msg, "ItemModified|"); + if (flag & kFSEventStreamEventFlagItemRemoved) + strbuf_addstr(&msg, "ItemRemoved|"); + if (flag & kFSEventStreamEventFlagItemRenamed) + strbuf_addstr(&msg, "ItemRenamed|"); + if (flag & kFSEventStreamEventFlagItemXattrMod) + strbuf_addstr(&msg, "ItemXattrMod|"); + if (flag & kFSEventStreamEventFlagOwnEvent) + strbuf_addstr(&msg, "OwnEvent|"); + if (flag & kFSEventStreamEventFlagItemCloned) + strbuf_addstr(&msg, "ItemCloned|"); + + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + path, flag, msg.buf); + + strbuf_release(&msg); +} + +static int ef_is_root_delete(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRemoved); +} + +static int ef_is_root_renamed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRenamed); +} + +static void fsevent_callback(ConstFSEventStreamRef streamRef, + void *ctx, + size_t num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]) +{ + struct fsmonitor_daemon_state *state = ctx; + struct fsmonitor_daemon_backend_data *data = state->backend_data; + char **paths = (char **)event_paths; + struct fsmonitor_batch *batch = NULL; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *path_k; + const char *slash; + int k; + + /* + * Build a list of all filesystem changes into a private/local + * list and without holding any locks. + */ + for (k = 0; k < num_of_events; k++) { + /* + * On Mac, we receive an array of absolute paths. + */ + path_k = paths[k]; + + /* + * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR. + * Please don't log them to Trace2. + * + * trace_printf_key(&trace_fsmonitor, "XXX '%s'", path_k); + */ + + /* + * If event[k] is marked as dropped, we assume that we have + * lost sync with the filesystem and should flush our cached + * data. We need to: + * + * [1] Abort/wake any client threads waiting for a cookie and + * flush the cached state data (the current token), and + * create a new token. + * + * [2] Discard the batch that we were locally building (since + * they are conceptually relative to the just flushed + * token). + */ + if ((event_flags[k] & kFSEventStreamEventFlagKernelDropped) || + (event_flags[k] & kFSEventStreamEventFlagUserDropped)) { + /* + * see also kFSEventStreamEventFlagMustScanSubDirs + */ + trace2_data_string("fsmonitor", NULL, + "fsm-listen/kernel", "dropped"); + + fsmonitor_force_resync(state); + + if (fsmonitor_batch__free(batch)) + BUG("batch should not have a next"); + string_list_clear(&cookie_list, 0); + + /* + * We assume that any events that we received + * in this callback after this dropped event + * may still be valid, so we continue rather + * than break. (And just in case there is a + * delete of ".git" hiding in there.) + */ + continue; + } + + switch (fsmonitor_classify_path_absolute(state, path_k)) { + + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within .git or gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path_k); + string_list_append(&cookie_list, + slash ? slash + 1 : path_k); + break; + + case IS_INSIDE_DOT_GIT: + case IS_INSIDE_GITDIR: + /* ignore all other paths inside of .git or gitdir */ + break; + + case IS_DOT_GIT: + case IS_GITDIR: + /* + * If .git directory is deleted or renamed away, + * we have to quit. + */ + if (ef_is_root_delete(event_flags[k])) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/gitdir", + "removed"); + goto force_shutdown; + } + if (ef_is_root_renamed(event_flags[k])) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/gitdir", + "renamed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* try to queue normal pathnames */ + + if (trace_pass_fl(&trace_fsmonitor)) + log_flags_set(path_k, event_flags[k]); + + /* fsevent could be marked as both a file and directory */ + + if (event_flags[k] & kFSEventStreamEventFlagItemIsFile) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, rel); + } + + if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + char *p = xstrfmt("%s/", rel); + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, p); + + free(p); + } + + break; + + case IS_OUTSIDE_CONE: + default: + trace_printf_key(&trace_fsmonitor, + "ignoring '%s'", path_k); + break; + } + } + + fsmonitor_publish(state, batch, &cookie_list); + string_list_clear(&cookie_list, 0); + return; + +force_shutdown: + if (fsmonitor_batch__free(batch)) + BUG("batch should not have a next"); + string_list_clear(&cookie_list, 0); + + data->shutdown_style = FORCE_SHUTDOWN; + CFRunLoopStop(data->rl); + return; +} + +/* + * TODO Investigate the proper value for the `latency` argument in the call + * TODO to `FSEventStreamCreate()`. I'm not sure that this needs to be a + * TODO config setting or just something that we tune after some testing. + * TODO + * TODO With a latency of 0.1, I was seeing lots of dropped events during + * TODO the "touch 100000" files test within t/perf/p7519, but with a + * TODO latency of 0.001 I did not see any dropped events. So the "correct" + * TODO value may be somewhere in between. + * TODO + * TODO https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate + */ int fsmonitor_fs_listen__ctor(struct fsmonitor_daemon_state *state) { + FSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer | + kFSEventStreamCreateFlagWatchRoot | + kFSEventStreamCreateFlagFileEvents; + FSEventStreamContext ctx = { + 0, + state, + NULL, + NULL, + NULL + }; + struct fsmonitor_daemon_backend_data *data; + const void *dir_array[2]; + + data = xcalloc(1, sizeof(*data)); + state->backend_data = data; + + data->cfsr_worktree_path = CFStringCreateWithCString( + NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_worktree_path; + + if (state->nr_paths_watching > 1) { + data->cfsr_gitdir_path = CFStringCreateWithCString( + NULL, state->path_gitdir_watch.buf, + kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path; + } + + data->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array, + data->nr_paths_watching, + NULL); + data->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx, + data->cfar_paths_to_watch, + kFSEventStreamEventIdSinceNow, + 0.001, flags); + if (data->stream == NULL) + goto failed; + + /* + * `data->rl` needs to be set inside the listener thread. + */ + + return 0; + +failed: + error("Unable to create FSEventStream."); + + FREE_AND_NULL(state->backend_data); return -1; } void fsmonitor_fs_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + if (data->stream) { + if (data->stream_started) + FSEventStreamStop(data->stream); + if (data->stream_scheduled) + FSEventStreamInvalidate(data->stream); + FSEventStreamRelease(data->stream); + } + + FREE_AND_NULL(state->backend_data); } void fsmonitor_fs_listen__stop_async(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + data->shutdown_style = SHUTDOWN_EVENT; + + CFRunLoopStop(data->rl); } void fsmonitor_fs_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + + data->rl = CFRunLoopGetCurrent(); + + FSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode); + data->stream_scheduled = 1; + + if (!FSEventStreamStart(data->stream)) { + error("Failed to start the FSEventStream"); + goto force_error_stop_without_loop; + } + data->stream_started = 1; + + CFRunLoopRun(); + + switch (data->shutdown_style) { + case FORCE_ERROR_STOP: + state->error_code = -1; + /* fall thru */ + case FORCE_SHUTDOWN: + ipc_server_stop_async(state->ipc_server_data); + /* fall thru */ + case SHUTDOWN_EVENT: + default: + break; + } + return; + +force_error_stop_without_loop: + state->error_code = -1; + ipc_server_stop_async(state->ipc_server_data); + return; } From patchwork Thu Apr 1 15:40:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178707 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF55FC433ED for ; Thu, 1 Apr 2021 17:58:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9321561001 for ; Thu, 1 Apr 2021 17:58:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235674AbhDAR6H (ORCPT ); Thu, 1 Apr 2021 13:58:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32870 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236501AbhDARyj (ORCPT ); Thu, 1 Apr 2021 13:54:39 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 638FEC02D546 for ; Thu, 1 Apr 2021 08:41:19 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id g20so1264874wmk.3 for ; Thu, 01 Apr 2021 08:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sUFSww/LIdrk0EXQfp6x5iXujoa+SRXqPGQmdJewi9Y=; b=et9V1fpxW2AGy3ffRBNOd4/Tg4SVmu7VIa7Ca2LmD3Z7ieTNQP7Xhp1pQoVkmmuj4A bXRYS7a6KKPRSpgeWT6lYYot++lkMfqMwGlyElKJlr8IrVuknsKxFFIKnGRerDsL7Iuy 0wC1Jagx1uHz4YAzgC9YMEoj0BFpApPuW0NlLA/SRzdPLIStmDGvhbIuVNGfMh64cYPj UUjsz7IEblC9eiu6gsAPKITCn1AbJR+8DKo7yDU1/GBcGtBdQU8m4orezWHLzB59b1Es ABAS6ihTHDyxz1TbQy2T9RDzdY/+gf1ThC4Gw9EHviMth9FucgXt9LVM4oIrezTeYosr bDRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sUFSww/LIdrk0EXQfp6x5iXujoa+SRXqPGQmdJewi9Y=; b=aM4nVjNRlXd/n5c7wl5Dm9ekuUG4Wbxe7n5R0z9AhatRiW+J60DuRtPKlCe+POoMNt FagWb00nWCmasVGi95Es7CebBIgHuFWyZM3O2wolY0AKdCYkHuVP7xHeVUeUc6LOSYez fCpAz/we2oq3cBkkJgiRaCAhoWb6aRD20W0JDTDtI4n8FuVubvtFbyPsZMeHMq5UHjnM GGX05/IGC8SQ5XkciWOWc2OtTrKquC+j+R6odyJXHwH9BQhfa1IQW8Cz7VVVXwp1C14b 8kIauHnIkV096IT216sNJMcBw2TBkcmzOzMugA0kVVow33beK7tLfLaWKJCBWKJkE4aj cVVg== X-Gm-Message-State: AOAM532lNjJ7CNVMvJxY3Gi7/q2L1VIQhqzN2nuGbhW26gzBop6vh+VB aXt62x9u3cea9qbrBooZfxk9qgbeC5Q= X-Google-Smtp-Source: ABdhPJylYrBWnRdEIHebMjavyUdh8iEwHCOdvk1WR6T9bAk7CIOQGS16CxEn+XX9gFGnyBlff9bZig== X-Received: by 2002:a1c:e18b:: with SMTP id y133mr8681866wmg.22.1617291678081; Thu, 01 Apr 2021 08:41:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p14sm8533546wmc.30.2021.04.01.08.41.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:17 -0700 (PDT) Message-Id: <2b4ae4fc3d622f0fee8eb5e527a51467e13acfc5.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:58 +0000 Subject: [PATCH 16/23] fsmonitor--daemon: implement handle_client callback Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to respond to IPC requests from client Git processes and respond with a list of modified pathnames relative to the provided token. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 335 +++++++++++++++++++++++++++++++++++- 1 file changed, 333 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 48071d445c49..32df392b25d3 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,6 +7,7 @@ #include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" +#include "pkt-line.h" static const char * const builtin_fsmonitor__daemon_usage[] = { N_("git fsmonitor--daemon --start []"), @@ -369,19 +370,349 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) fsmonitor_free_token_data(free_me); } +/* + * Format an opaque token string to send to the client. + */ +static void fsmonitor_format_response_token( + struct strbuf *response_token, + const struct strbuf *response_token_id, + const struct fsmonitor_batch *batch) +{ + uint64_t seq_nr = (batch) ? batch->batch_seq_nr + 1 : 0; + + strbuf_reset(response_token); + strbuf_addf(response_token, "builtin:%s:%"PRIu64, + response_token_id->buf, seq_nr); +} + +/* + * Parse an opaque token from the client. + */ +static int fsmonitor_parse_client_token(const char *buf_token, + struct strbuf *requested_token_id, + uint64_t *seq_nr) +{ + const char *p; + char *p_end; + + strbuf_reset(requested_token_id); + *seq_nr = 0; + + if (!skip_prefix(buf_token, "builtin:", &p)) + return 1; + + while (*p && *p != ':') + strbuf_addch(requested_token_id, *p++); + if (!*p++) + return 1; + + *seq_nr = (uint64_t)strtoumax(p, &p_end, 10); + if (*p_end) + return 1; + + return 0; +} + +KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal); + +static int do_handle_client(struct fsmonitor_daemon_state *state, + const char *command, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + struct fsmonitor_token_data *token_data = NULL; + struct strbuf response_token = STRBUF_INIT; + struct strbuf requested_token_id = STRBUF_INIT; + struct strbuf payload = STRBUF_INIT; + uint64_t requested_oldest_seq_nr = 0; + uint64_t total_response_len = 0; + const char *p; + const struct fsmonitor_batch *batch_head; + const struct fsmonitor_batch *batch; + intmax_t count = 0, duplicates = 0; + kh_str_t *shown; + int hash_ret; + int result; + + /* + * We expect `command` to be of the form: + * + * := quit NUL + * | flush NUL + * | NUL + * | NUL + */ + + if (!strcmp(command, "quit")) { + /* + * A client has requested over the socket/pipe that the + * daemon shutdown. + * + * Tell the IPC thread pool to shutdown (which completes + * the await in the main thread (which can stop the + * fsmonitor listener thread)). + * + * There is no reply to the client. + */ + return SIMPLE_IPC_QUIT; + } + + if (!strcmp(command, "flush")) { + /* + * Flush all of our cached data and generate a new token + * just like if we lost sync with the filesystem. + * + * Then send a trivial response using the new token. + */ + fsmonitor_force_resync(state); + result = 0; + goto send_trivial_response; + } + + if (!skip_prefix(command, "builtin:", &p)) { + /* assume V1 timestamp or garbage */ + + char *p_end; + + strtoumax(command, &p_end, 10); + trace_printf_key(&trace_fsmonitor, + ((*p_end) ? + "fsmonitor: invalid command line '%s'" : + "fsmonitor: unsupported V1 protocol '%s'"), + command); + result = -1; + goto send_trivial_response; + } + + /* try V2 token */ + + if (fsmonitor_parse_client_token(command, &requested_token_id, + &requested_oldest_seq_nr)) { + trace_printf_key(&trace_fsmonitor, + "fsmonitor: invalid V2 protocol token '%s'", + command); + result = -1; + goto send_trivial_response; + } + + pthread_mutex_lock(&state->main_lock); + + if (!state->current_token_data) { + /* + * We don't have a current token. This may mean that + * the listener thread has not yet started. + */ + pthread_mutex_unlock(&state->main_lock); + result = 0; + goto send_trivial_response; + } + if (strcmp(requested_token_id.buf, + state->current_token_data->token_id.buf)) { + /* + * The client last spoke to a different daemon + * instance -OR- the daemon had to resync with + * the filesystem (and lost events), so reject. + */ + pthread_mutex_unlock(&state->main_lock); + result = 0; + trace2_data_string("fsmonitor", the_repository, + "response/token", "different"); + goto send_trivial_response; + } + if (!state->current_token_data->batch_tail) { + /* + * The listener has not received any filesystem + * events yet since we created the current token. + * We can respond with an empty list, since the + * client has already seen the current token and + * we have nothing new to report. (This is + * instead of sending a trivial response.) + */ + pthread_mutex_unlock(&state->main_lock); + result = 0; + goto send_empty_response; + } + if (requested_oldest_seq_nr < + state->current_token_data->batch_tail->batch_seq_nr) { + /* + * The client wants older events than we have for + * this token_id. This means that the end of our + * batch list was truncated and we cannot give the + * client a complete snapshot relative to their + * request. + */ + pthread_mutex_unlock(&state->main_lock); + + trace_printf_key(&trace_fsmonitor, + "client requested truncated data"); + result = 0; + goto send_trivial_response; + } + + /* + * We're going to hold onto a pointer to the current + * token-data while we walk the list of batches of files. + * During this time, we will NOT be under the lock. + * So we ref-count it. + * + * This allows the listener thread to continue prepending + * new batches of items to the token-data (which we'll ignore). + * + * AND it allows the listener thread to do a token-reset + * (and install a new `current_token_data`). + * + * We mark the current head of the batch list as "pinned" so + * that the listener thread will treat this item as read-only + * (and prevent any more paths from being added to it) from + * now on. + */ + token_data = state->current_token_data; + token_data->client_ref_count++; + + batch_head = token_data->batch_head; + ((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL); + + pthread_mutex_unlock(&state->main_lock); + + /* + * FSMonitor Protocol V2 requires that we send a response header + * with a "new current token" and then all of the paths that changed + * since the "requested token". + */ + fsmonitor_format_response_token(&response_token, + &token_data->token_id, + batch_head); + + reply(reply_data, response_token.buf, response_token.len + 1); + total_response_len += response_token.len + 1; + + trace2_data_string("fsmonitor", the_repository, "response/token", + response_token.buf); + trace_printf_key(&trace_fsmonitor, "response token: %s", response_token.buf); + + shown = kh_init_str(); + for (batch = batch_head; + batch && batch->batch_seq_nr >= requested_oldest_seq_nr; + batch = batch->next) { + size_t k; + + for (k = 0; k < batch->nr; k++) { + const char *s = batch->interned_paths[k]; + size_t s_len; + + if (kh_get_str(shown, s) != kh_end(shown)) + duplicates++; + else { + kh_put_str(shown, s, &hash_ret); + + trace_printf_key(&trace_fsmonitor, + "send[%"PRIuMAX"]: %s", + count, s); + + /* Each path gets written with a trailing NUL */ + s_len = strlen(s) + 1; + + if (payload.len + s_len >= + LARGE_PACKET_DATA_MAX) { + reply(reply_data, payload.buf, + payload.len); + total_response_len += payload.len; + strbuf_reset(&payload); + } + + strbuf_add(&payload, s, s_len); + count++; + } + } + } + + if (payload.len) { + reply(reply_data, payload.buf, payload.len); + total_response_len += payload.len; + } + + kh_release_str(shown); + + pthread_mutex_lock(&state->main_lock); + if (token_data->client_ref_count > 0) + token_data->client_ref_count--; + + if (token_data->client_ref_count == 0) { + if (token_data != state->current_token_data) { + /* + * The listener thread did a token-reset while we were + * walking the batch list. Therefore, this token is + * stale and can be discarded completely. If we are + * the last reader thread using this token, we own + * that work. + */ + fsmonitor_free_token_data(token_data); + } + } + + pthread_mutex_unlock(&state->main_lock); + + trace2_data_intmax("fsmonitor", the_repository, "response/length", total_response_len); + trace2_data_intmax("fsmonitor", the_repository, "response/count/files", count); + trace2_data_intmax("fsmonitor", the_repository, "response/count/duplicates", duplicates); + + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + strbuf_release(&payload); + + return 0; + +send_trivial_response: + pthread_mutex_lock(&state->main_lock); + fsmonitor_format_response_token(&response_token, + &state->current_token_data->token_id, + state->current_token_data->batch_head); + pthread_mutex_unlock(&state->main_lock); + + reply(reply_data, response_token.buf, response_token.len + 1); + trace2_data_string("fsmonitor", the_repository, "response/token", + response_token.buf); + reply(reply_data, "/", 2); + trace2_data_intmax("fsmonitor", the_repository, "response/trivial", 1); + + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + + return result; + +send_empty_response: + pthread_mutex_lock(&state->main_lock); + fsmonitor_format_response_token(&response_token, + &state->current_token_data->token_id, + NULL); + pthread_mutex_unlock(&state->main_lock); + + reply(reply_data, response_token.buf, response_token.len + 1); + trace2_data_string("fsmonitor", the_repository, "response/token", + response_token.buf); + trace2_data_intmax("fsmonitor", the_repository, "response/empty", 1); + + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + + return 0; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, const char *command, ipc_server_reply_cb *reply, struct ipc_server_reply_data *reply_data) { - /* struct fsmonitor_daemon_state *state = data; */ + struct fsmonitor_daemon_state *state = data; int result; + trace_printf_key(&trace_fsmonitor, "requested token: %s", command); + trace2_region_enter("fsmonitor", "handle_client", the_repository); trace2_data_string("fsmonitor", the_repository, "request", command); - result = 0; /* TODO Do something here. */ + result = do_handle_client(state, command, reply, reply_data); trace2_region_leave("fsmonitor", "handle_client", the_repository); From patchwork Thu Apr 1 15:40:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF272C001B8 for ; Thu, 1 Apr 2021 17:43:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 23ACF61374 for ; Thu, 1 Apr 2021 17:43:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235684AbhDARnH (ORCPT ); Thu, 1 Apr 2021 13:43:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234447AbhDARhk (ORCPT ); Thu, 1 Apr 2021 13:37:40 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3FD2C02D547 for ; Thu, 1 Apr 2021 08:41:19 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id j9so2255151wrx.12 for ; Thu, 01 Apr 2021 08:41:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=d7ttphpVx4NoDbT7tJHRrMQZ4D87byK9jMPJVinO5j0=; b=oki13NIsHZOef6SPkmFGkfJKfJftENn2DIVtpz48kn1Uz+MvxUu/HSucQ7AxG3M+rI JpOMjXuzR0CbKCZu+k7+GN6ovFUSxrRF8XfjFT1gv+ljxomrlcY7TG7L+f/2ruMQsi9f Jk++PSh3K7xtsPll0R6YIK73dxcge80AhNxqPXfCPNE8ytFtiomY56afO1O7+fWghSmx mQj69MOX5Ba20H9JX84L8Cu6fUXD7rKWo1q5fUcV3CqmY49JV/WBXJcngfygKzfm/hBq zMlwAdj4/fnFWJFB0H0M4G9/oEa0ukiCzQR31jEzCOwSYHS+mXhOZsA49X2PowayqtIY bR7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=d7ttphpVx4NoDbT7tJHRrMQZ4D87byK9jMPJVinO5j0=; b=nqIr9g8ASbhnLkNkHsCU5QUbZxuTqP+RGt+Wb25DddzYvNpqZlIl4inLndfi1sHw4S Qg0kuSxDGAHb3SSgESDhqcUKCg3k57eKUIFSFihXXt2rK2iZC+zba6EQWAFvzN3e4V+i Z5PdxRaagmV41GaHk7eYDEWc/UaKIuV0WxSIfRd3qZ/l6pQFjwEYrlxhMawsovBBaEv5 VuO3TgkI767IihfJdBbIGx2qweWgSHfRqw5/YgHb+x5FHDeEUaBjrU8yKA3OFM2/pL1v hGmrZhdJ96pBsnk9qE/aZqipizrO4XvLEaSID2DK8r5QSqo1GezBwJ+WT2H8UHR/CnKx NPvA== X-Gm-Message-State: AOAM530C9HTlvFj7M0LHTgpiStG828lVb8BST19i1sRt1V7QxUcuGdP4 ofERHYOEfD+O1CTH8USHsSaL+MYypn8= X-Google-Smtp-Source: ABdhPJxRygJvPv8S+E39Wm+p9iB14O+FY7+20DAM/WDtEMqKC0kSm/K1+lXcy9yrFSX8dMdavE48QA== X-Received: by 2002:adf:c587:: with SMTP id m7mr10219563wrg.369.1617291678677; Thu, 01 Apr 2021 08:41:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l6sm9963338wrt.56.2021.04.01.08.41.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:18 -0700 (PDT) Message-Id: <9f263e70c7245a01210ac743ce3dfda4dcc08308.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:40:59 +0000 Subject: [PATCH 17/23] fsmonitor--daemon: periodically truncate list of modified files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to periodically truncate the list of modified files to save some memory. Clients will ask for the set of changes relative to a token that they found in the FSMN index extension in the index. (This token is like a point in time, but different). Clients will then update the index to contain the response token (so that subsequent commands will be relative to this new token). Therefore, the daemon can gradually truncate the in-memory list of changed paths as they become obsolete (older that the previous token). Since we may have multiple clients making concurrent requests with a skew of tokens and clients may be racing to the talk to the daemon, we lazily truncate the list. We introduce a 5 minute delay and truncate batches 5 minutes after they are considered obsolete. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 78 +++++++++++++++++++++++++++++++++++++ 1 file changed, 78 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 32df392b25d3..e9a9aea59ad6 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -316,6 +316,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, batch_src->interned_paths[k]; } +/* + * To keep the batch list from growing unbounded in response to filesystem + * activity, we try to truncate old batches from the end of the list as + * they become irrelevant. + * + * We assume that the .git/index will be updated with the most recent token + * any time the index is updated. And future commands will only ask for + * recent changes *since* that new token. So as tokens advance into the + * future, older batch items will never be requested/needed. So we can + * truncate them without loss of functionality. + * + * However, multiple commands may be talking to the daemon concurrently + * or perform a slow command, so a little "token skew" is possible. + * Therefore, we want this to be a little bit lazy and have a generous + * delay. + * + * The current reader thread walked backwards in time from `token->batch_head` + * back to `batch_marker` somewhere in the middle of the batch list. + * + * Let's walk backwards in time from that marker an arbitrary delay + * and truncate the list there. Note that these timestamps are completely + * artificial (based on when we pinned the batch item) and not on any + * filesystem activity. + */ +#define MY_TIME_DELAY (5 * 60) /* seconds */ + +static void fsmonitor_batch__truncate(struct fsmonitor_daemon_state *state, + const struct fsmonitor_batch *batch_marker) +{ + /* assert state->main_lock */ + + const struct fsmonitor_batch *batch; + struct fsmonitor_batch *rest; + struct fsmonitor_batch *p; + time_t t; + + if (!batch_marker) + return; + + trace_printf_key(&trace_fsmonitor, "TRNC mark (%"PRIu64",%"PRIu64")", + batch_marker->batch_seq_nr, + (uint64_t)batch_marker->pinned_time); + + for (batch = batch_marker; batch; batch = batch->next) { + if (!batch->pinned_time) /* an overflow batch */ + continue; + + t = batch->pinned_time + MY_TIME_DELAY; + if (t > batch_marker->pinned_time) /* too close to marker */ + continue; + + goto truncate_past_here; + } + + return; + +truncate_past_here: + state->current_token_data->batch_tail = (struct fsmonitor_batch *)batch; + + rest = ((struct fsmonitor_batch *)batch)->next; + ((struct fsmonitor_batch *)batch)->next = NULL; + + for (p = rest; p; p = fsmonitor_batch__free(p)) { + trace_printf_key(&trace_fsmonitor, + "TRNC kill (%"PRIu64",%"PRIu64")", + p->batch_seq_nr, (uint64_t)p->pinned_time); + } +} + static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) { struct fsmonitor_batch *p; @@ -647,6 +716,15 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, * that work. */ fsmonitor_free_token_data(token_data); + } else if (batch) { + /* + * This batch is the first item in the list + * that is older than the requested sequence + * number and might be considered to be + * obsolete. See if we can truncate the list + * and save some memory. + */ + fsmonitor_batch__truncate(state, batch); } } From patchwork Thu Apr 1 15:41:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 44367C433B4 for ; Thu, 1 Apr 2021 18:04:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D0626112E for ; Thu, 1 Apr 2021 18:04:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237599AbhDASEA (ORCPT ); Thu, 1 Apr 2021 14:04:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:33038 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237008AbhDAR7U (ORCPT ); Thu, 1 Apr 2021 13:59:20 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 742A3C02D548 for ; Thu, 1 Apr 2021 08:41:20 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id b9so2265844wrt.8 for ; Thu, 01 Apr 2021 08:41:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Fq+75wZ+1cIqBx465IViCteahZviGOVpai2pd5+YxWU=; b=ALf+zPQ4ovCKG+BwNGhlzG+5p7egK+RyugLnR7N0GZYhv4+H6In/SrFeTYfy12E3Hj 9w6ASVyBOuy6R5b9hICyG0PeO5XKoSLNlVgbPx9T5ByRNH9WAF1X2YJnzadS31lVdMc+ agI7KJSCLu1dfuFxkXMoKshWuwUScqCwOQvDVS++R1cTUj9onkIcE7iJ/Sc0HXWbV5Wn 5iLK4ecQUuc1tq7DjkCfCCc9rmw5L3BC3H15RNAZ9Dxeo4Dvc6jVjq8SgmY7iZIUM+A8 vujlfXIswUfXBqlGJz8dIMF/h4hdVMI2gyijKyCCFj7PeuYgwu82iBREJy6nPmjEOxjG YM9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Fq+75wZ+1cIqBx465IViCteahZviGOVpai2pd5+YxWU=; b=mQ9ARxBFAfq5zJZIJlzpvewZBb0UKcSUGODng+2kI1976E97V0v4bfWF0JfnKoGPQz bghppsKTy2sAWzGE0X6E3oBDAlZCt+eJ0sGUfr5PxkQVeR/0IZxrcXOlE3kL2eBNbVlV 4XoFF5Ceh3rwFidQChHD6HO1tJ7+plv5FjFqMgzt7omEVTFHb3imYSMBGF+75r0jogRO g+QNttkikTTE4/seJBY8Ojarc+GLc1WrVoAoz/LLZpYKMHAPe1kO9XrVuj2Bx6uu0cjk 74pHVfAMkC3R6XCVo8N2MfAWGiR8x76CTe+wuLydhPbklcL282EMxdHIRaAhN0C4l5MS GAYA== X-Gm-Message-State: AOAM530IhQOmUGo8k10ocahOI5VW6GmKMCdcsV8ukw27S82Mk3hAa6pD 077zpbK64XUce+NcHqXYSx/SGtUJmiM= X-Google-Smtp-Source: ABdhPJwQ/cOvUD/MIZCv+9Cr/VYU3RF8UcDXfS2NJpzSsJnN6YJh3DppmzWrd0nKjwWEOFdm4nM/8g== X-Received: by 2002:adf:cd04:: with SMTP id w4mr10646130wrm.148.1617291679248; Thu, 01 Apr 2021 08:41:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c11sm10867359wrm.67.2021.04.01.08.41.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:18 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Apr 2021 15:41:00 +0000 Subject: [PATCH 18/23] fsmonitor--daemon:: introduce client delay for testing Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Define GIT_TEST_FSMONITOR_CLIENT_DELAY as a millisecond delay. Introduce an artificial delay when processing client requests. This make the CI/PR test suite a little more stable and avoids the need to load up test scripts with sleep statements to avoid racy failures. This was mostly seen on 1 or 2 core CI build machines where the test script would create a file and quickly try to confirm that the daemon had seen it *before* the daemon had received the kernel event and causing a test failure. Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 38 ++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index e9a9aea59ad6..0cb09ef0b984 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -150,6 +150,30 @@ static int do_as_client__send_flush(void) return 0; } +static int lookup_client_test_delay(void) +{ + static int delay_ms = -1; + + const char *s; + int ms; + + if (delay_ms >= 0) + return delay_ms; + + delay_ms = 0; + + s = getenv("GIT_TEST_FSMONITOR_CLIENT_DELAY"); + if (!s) + return delay_ms; + + ms = atoi(s); + if (ms < 0) + return delay_ms; + + delay_ms = ms; + return delay_ms; +} + /* * Requests to and from a FSMonitor Protocol V2 provider use an opaque * "token" as a virtual timestamp. Clients can request a summary of all @@ -526,6 +550,18 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, return SIMPLE_IPC_QUIT; } + /* + * For testing purposes, introduce an artificial delay in this + * worker to allow the filesystem listener thread to receive + * any fs events that may have been generated by the client + * process on the other end of the pipe/socket. This helps + * make the CI/PR test suite runs a little more predictable + * and hopefully eliminates the need to introduce `sleep` + * commands in the test scripts. + */ + if (state->test_client_delay_ms) + sleep_millisec(state->test_client_delay_ms); + if (!strcmp(command, "flush")) { /* * Flush all of our cached data and generate a new token @@ -1038,7 +1074,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); state.error_code = 0; state.current_token_data = fsmonitor_new_token_data(); - state.test_client_delay_ms = 0; + state.test_client_delay_ms = lookup_client_test_delay(); /* Prepare to (recursively) watch the directory. */ strbuf_init(&state.path_worktree_watch, 0); From patchwork Thu Apr 1 15:41:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92F98C001EA for ; Thu, 1 Apr 2021 17:43:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A9BF761284 for ; Thu, 1 Apr 2021 17:43:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235834AbhDARnP (ORCPT ); Thu, 1 Apr 2021 13:43:15 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234744AbhDARjk (ORCPT ); Thu, 1 Apr 2021 13:39:40 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 22B37C02D549 for ; Thu, 1 Apr 2021 08:41:21 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id r10-20020a05600c35cab029010c946c95easo1125419wmq.4 for ; Thu, 01 Apr 2021 08:41:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CnoTzJHVAuf871k/UDSVW/LJi9azU3ymhIRK0wnLjGI=; b=NKBBA8Lvp3ABGuM7fVb+z3V367lL7uubSNMT9o+n7mmE9XO9Hm7IzypRZ2WqDOtc6T QW0nYBe7MKBQONyZurQyXyRQKusv9aVrWYAgEacAoO+M6+HRcYfR1JmquAAnv3XwHfOP i7aOeEp1XADLe3uHn85vnZ6zrEfmiSJDYEm+AxRQzu3JNh2Z5vE5J8tcEQFgrg7hao1m 0ebRKuVKSCGoaMMsb1Xim+ZNxDmViWk96z72tcLQIkP795m11QgaZDgNYPESj69DXbKh C0w7KVa6bvlxyjQfg8qeC4/75GCZFQVeE12QoCHMA9gGvUz4V8p0a9JuEbA2FEG3bUfG JP1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CnoTzJHVAuf871k/UDSVW/LJi9azU3ymhIRK0wnLjGI=; b=jDUmWh80vGK/knmLeMVyQl8z/u1V7UTEe7c6Retj7xqzb5X0AJ1t2FYztqINiAsSJP xhn2s2h01kM0hIYv/wVRuSjaMDNiYaW2OgvWZUhqPx8URZ8w4Mg4/3IrwqGKBlCjoD7b BKzHmWBsSc1hqPmbsBRZ0ylXJk6XjE/U3lbqIzfEzFwVxXFtDSQR9+xY/DGh+1skID42 UA/FGy6qFlhg4pJ+yJOl9cQmP53GAoC7aTBvVCn0OUnvoq/wlr3492YUZgP9UJUHvlBU TO/3uIHeuJRaEhMSs0lvL4IHoEg7Lz+D9iUe0bdVEadv+/4v0kLY7D3DaEgQztFAhcOx C+bg== X-Gm-Message-State: AOAM531oh3kmDPII2z5ca/a1dTWTyoe4NEUf7gwFd3ukuCV35abUxI8B zGoDizedzS2AVJsz3RT+JMQ+SaEWsW0= X-Google-Smtp-Source: ABdhPJxo12OlRmrooRgedRL64mkuImYPCfRdFrX4dRDVKlZCRQ36kvQpCDsGtxzCMYSw4O/wmbIjpA== X-Received: by 2002:a7b:c75a:: with SMTP id w26mr8905703wmk.49.1617291679866; Thu, 01 Apr 2021 08:41:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n9sm10157966wrx.46.2021.04.01.08.41.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:19 -0700 (PDT) Message-Id: <038b62dc6744e8d8004f3f8423a40066821c0f1b.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:41:01 +0000 Subject: [PATCH 19/23] fsmonitor--daemon: use a cookie file to sync with file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon client threads to create a cookie file inside the .git directory and then wait until FS events for the cookie are observed by the FS listener thread. This helps address the racy nature of file system events by blocking the client response until the kernel has drained any event backlog. This is especially important on MacOS where kernel events are only issued with a limited frequency. See the `latency` argument of `FSeventStreamCreate()`. The kernel only signals every `latency` seconds, but does not guarantee that the kernel queue is completely drained, so we may have to wait more than one interval. If we increase the frequency, the system is more likely to drop events. We avoid these issues by having each client thread create a unique cookie file and then wait until it is seen in the event stream. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler --- builtin/fsmonitor--daemon.c | 198 ++++++++++++++++++++++++++++++++++++ fsmonitor--daemon.h | 5 + 2 files changed, 203 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 0cb09ef0b984..d6b59a98cedd 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -150,6 +150,149 @@ static int do_as_client__send_flush(void) return 0; } +enum fsmonitor_cookie_item_result { + FCIR_ERROR = -1, /* could not create cookie file ? */ + FCIR_INIT = 0, + FCIR_SEEN, + FCIR_ABORT, +}; + +struct fsmonitor_cookie_item { + struct hashmap_entry entry; + const char *name; + enum fsmonitor_cookie_item_result result; +}; + +static int cookies_cmp(const void *data, const struct hashmap_entry *he1, + const struct hashmap_entry *he2, const void *keydata) +{ + const struct fsmonitor_cookie_item *a = + container_of(he1, const struct fsmonitor_cookie_item, entry); + const struct fsmonitor_cookie_item *b = + container_of(he2, const struct fsmonitor_cookie_item, entry); + + return strcmp(a->name, keydata ? keydata : b->name); +} + +static enum fsmonitor_cookie_item_result fsmonitor_wait_for_cookie( + struct fsmonitor_daemon_state *state) +{ + int fd; + struct fsmonitor_cookie_item cookie; + struct strbuf cookie_pathname = STRBUF_INIT; + struct strbuf cookie_filename = STRBUF_INIT; + const char *slash; + int my_cookie_seq; + + pthread_mutex_lock(&state->main_lock); + + my_cookie_seq = state->cookie_seq++; + + strbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix); + strbuf_addf(&cookie_pathname, "%i-%i", getpid(), my_cookie_seq); + + slash = find_last_dir_sep(cookie_pathname.buf); + if (slash) + strbuf_addstr(&cookie_filename, slash + 1); + else + strbuf_addbuf(&cookie_filename, &cookie_pathname); + cookie.name = strbuf_detach(&cookie_filename, NULL); + cookie.result = FCIR_INIT; + // TODO should we have case-insenstive hash (and in cookie_cmp()) ?? + hashmap_entry_init(&cookie.entry, strhash(cookie.name)); + + /* + * Warning: we are putting the address of a stack variable into a + * global hashmap. This feels dodgy. We must ensure that we remove + * it before this thread and stack frame returns. + */ + hashmap_add(&state->cookies, &cookie.entry); + + trace_printf_key(&trace_fsmonitor, "cookie-wait: '%s' '%s'", + cookie.name, cookie_pathname.buf); + + /* + * Create the cookie file on disk and then wait for a notification + * that the listener thread has seen it. + */ + fd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600); + if (fd >= 0) { + close(fd); + unlink_or_warn(cookie_pathname.buf); + + while (cookie.result == FCIR_INIT) + pthread_cond_wait(&state->cookies_cond, + &state->main_lock); + + hashmap_remove(&state->cookies, &cookie.entry, NULL); + } else { + error_errno(_("could not create fsmonitor cookie '%s'"), + cookie.name); + + cookie.result = FCIR_ERROR; + hashmap_remove(&state->cookies, &cookie.entry, NULL); + } + + pthread_mutex_unlock(&state->main_lock); + + free((char*)cookie.name); + strbuf_release(&cookie_pathname); + return cookie.result; +} + +/* + * Mark these cookies as _SEEN and wake up the corresponding client threads. + */ +static void fsmonitor_cookie_mark_seen(struct fsmonitor_daemon_state *state, + const struct string_list *cookie_names) +{ + /* assert state->main_lock */ + + int k; + int nr_seen = 0; + + for (k = 0; k < cookie_names->nr; k++) { + struct fsmonitor_cookie_item key; + struct fsmonitor_cookie_item *cookie; + + key.name = cookie_names->items[k].string; + hashmap_entry_init(&key.entry, strhash(key.name)); + + cookie = hashmap_get_entry(&state->cookies, &key, entry, NULL); + if (cookie) { + trace_printf_key(&trace_fsmonitor, "cookie-seen: '%s'", + cookie->name); + cookie->result = FCIR_SEEN; + nr_seen++; + } + } + + if (nr_seen) + pthread_cond_broadcast(&state->cookies_cond); +} + +/* + * Set _ABORT on all pending cookies and wake up all client threads. + */ +static void fsmonitor_cookie_abort_all(struct fsmonitor_daemon_state *state) +{ + /* assert state->main_lock */ + + struct hashmap_iter iter; + struct fsmonitor_cookie_item *cookie; + int nr_aborted = 0; + + hashmap_for_each_entry(&state->cookies, &iter, cookie, entry) { + trace_printf_key(&trace_fsmonitor, "cookie-abort: '%s'", + cookie->name); + cookie->result = FCIR_ABORT; + nr_aborted++; + } + + if (nr_aborted) + pthread_cond_broadcast(&state->cookies_cond); +} + static int lookup_client_test_delay(void) { static int delay_ms = -1; @@ -435,6 +578,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) * We should create a new token and start fresh (as if we just * booted up). * + * [2] Some of those lost events may have been for cookie files. We + * should assume the worst and abort them rather letting them starve. + * * If there are no readers of the the current token data series, we * can free it now. Otherwise, let the last reader free it. Either * way, the old token data series is no longer associated with our @@ -454,6 +600,8 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) state->current_token_data->token_id.buf, new_one->token_id.buf); + fsmonitor_cookie_abort_all(state); + if (state->current_token_data->client_ref_count == 0) free_me = state->current_token_data; state->current_token_data = new_one; @@ -526,6 +674,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, kh_str_t *shown; int hash_ret; int result; + enum fsmonitor_cookie_item_result cookie_result; /* * We expect `command` to be of the form: @@ -654,6 +803,39 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, goto send_trivial_response; } + pthread_mutex_unlock(&state->main_lock); + + /* + * Write a cookie file inside the directory being watched in an + * effort to flush out existing filesystem events that we actually + * care about. Suspend this client thread until we see the filesystem + * events for this cookie file. + */ + cookie_result = fsmonitor_wait_for_cookie(state); + if (cookie_result != FCIR_SEEN) { + error(_("fsmonitor: cookie_result '%d' != SEEN"), + cookie_result); + result = 0; + goto send_trivial_response; + } + + pthread_mutex_lock(&state->main_lock); + + if (strcmp(requested_token_id.buf, + state->current_token_data->token_id.buf)) { + /* + * Ack! The listener thread lost sync with the filesystem + * and created a new token while we were waiting for the + * cookie file to be created! Just give up. + */ + pthread_mutex_unlock(&state->main_lock); + + trace_printf_key(&trace_fsmonitor, + "lost filesystem sync"); + result = 0; + goto send_trivial_response; + } + /* * We're going to hold onto a pointer to the current * token-data while we walk the list of batches of files. @@ -982,6 +1164,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, } } + if (cookie_names->nr) + fsmonitor_cookie_mark_seen(state, cookie_names); + pthread_mutex_unlock(&state->main_lock); } @@ -1071,7 +1256,9 @@ static int fsmonitor_run_daemon(void) memset(&state, 0, sizeof(state)); + hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); + pthread_cond_init(&state.cookies_cond, NULL); state.error_code = 0; state.current_token_data = fsmonitor_new_token_data(); state.test_client_delay_ms = lookup_client_test_delay(); @@ -1094,6 +1281,15 @@ static int fsmonitor_run_daemon(void) state.nr_paths_watching = 2; } + /* + * We will write filesystem syncing cookie files into + * /-. + */ + strbuf_init(&state.path_cookie_prefix, 0); + strbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch); + strbuf_addch(&state.path_cookie_prefix, '/'); + strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_PREFIX); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1106,6 +1302,7 @@ static int fsmonitor_run_daemon(void) err = fsmonitor_run_daemon_1(&state); done: + pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsmonitor_fs_listen__dtor(&state); @@ -1113,6 +1310,7 @@ static int fsmonitor_run_daemon(void) strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); + strbuf_release(&state.path_cookie_prefix); return err; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 06563b6ed56c..4e580e285ed6 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -45,6 +45,11 @@ struct fsmonitor_daemon_state { struct fsmonitor_token_data *current_token_data; + struct strbuf path_cookie_prefix; + pthread_cond_t cookies_cond; + int cookie_seq; + struct hashmap cookies; + int error_code; struct fsmonitor_daemon_backend_data *backend_data; From patchwork Thu Apr 1 15:41:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09E73C47065 for ; Thu, 1 Apr 2021 17:43:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E43C061393 for ; Thu, 1 Apr 2021 17:43:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235758AbhDARnJ (ORCPT ); Thu, 1 Apr 2021 13:43:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57186 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234542AbhDARh5 (ORCPT ); Thu, 1 Apr 2021 13:37:57 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F354C02D54A for ; Thu, 1 Apr 2021 08:41:21 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id v11so2259595wro.7 for ; Thu, 01 Apr 2021 08:41:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0wIqWvl3C7SoxxKv0Hgug46sR+HfmyuK7Ceh9eUHOwg=; b=kMRsp09fuTncNCjx195kTSOAEH2xXntP1iUGF22TQaz9lAjUWXzXqIR+idpAWEM4lw 9/RypKJwvDRi5SCWKcWRyrpRtlhqY/tpYJX0GFo6SP46S3g/1ZcIrcL8saKXclyoOsMJ tyNS0jW4WJHNGK4sCWL74POa1+sSuDjNEqCe7ohBkPMb+/r5eZKkseLQJIHK+iaOHJgv ZZXPjhX6tX9Rd+GJnZ/4po4HfgN1A8O6zaK0nLfcuMz+bb/6+VrTnQ42KOP7WvWFPmh6 lk4FboN0gABXyd1+D9n24AbE+7Q+MISEFiT0FcZ8Q2ZjC1MU4J3M3IHtpU5NM7ELMZSa ZI6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0wIqWvl3C7SoxxKv0Hgug46sR+HfmyuK7Ceh9eUHOwg=; b=kMfiojjLSAg3hErxQ7I3IIBgEQCov6ULwZ42shgLgyojI6mO9k+25bmJxCQ5PRJx+8 2rHD1ut/dDGV56eNREjYYSzIZdCF2hqWaB9HuWtwUoO/oof367jWGS/PuPd8lIFzYsnb oK1VbsRlYhXr41EXXi5ptLMeKuktSUaoe2yEMNmiVZZNF8gSwKsEWdhW3spVJ7UFbhqZ xeu4JhMakQDflBupCJGXSZf3bYlVtuvttsAxw0OSBHeyEqJPZvbbjolElQA2buKaZYM9 JVdQcxrYRH4HoT4s0PPYflqca18wZVCsbkOcrkK4Fk5gcIpZky9jdD/AWGL9NYTSS8Tg i2lw== X-Gm-Message-State: AOAM53048RNYlsqu/JP8DqXP8vC3eGoz8WtxupOY51V5p+OHNzTtP6Ym NwSEKqua+S3rveTzLQ9oUPQy7XFGPtY= X-Google-Smtp-Source: ABdhPJylMJAM8jj1iaPBsMbqGasc8Vc199VsS0rFgoNS7AaFRvICfgYF/pj8//6y0yHbRdh+ON+vmQ== X-Received: by 2002:a5d:4105:: with SMTP id l5mr10895187wrp.105.1617291680406; Thu, 01 Apr 2021 08:41:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g11sm10229169wrw.89.2021.04.01.08.41.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:20 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Apr 2021 15:41:02 +0000 Subject: [PATCH 20/23] fsmonitor: force update index when fsmonitor token advances Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Set the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when the fsmonitor response contains a different token to ensure that the index is written to disk. Normally, when the fsmonitor response includes a tracked file, the index is always updated. Similarly, the index might be updated when the response alters the untracked-cache (when enabled). However, in cases where neither of those cause the index to be considered changed, the fsmonitor response is wasted. And subsequent commands will continue to make requests with the same token and if there have not been any changes in the working directory, they will receive the same response. This was observed on Windows after a large checkout. On Windows, the kernel emits events for the files that are changed as they are changed. However, it might delay events for the containing directories until the system is more idle (or someone scans the directory (so it seems)). The first status following a checkout would get the list of files. The subsequent status commands would get the list of directories as the events trickled out. But they would never catch up because the token was not advanced because the index wasn't updated. This list of directories caused `wt_status_collect_untracked()` to unnecessarily spend time actually scanning them during each command. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fsmonitor.c b/fsmonitor.c index d7e18fc8cd47..8b544e31f29f 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -353,6 +353,16 @@ void refresh_fsmonitor(struct index_state *istate) } strbuf_release(&query_result); + /* + * If the fsmonitor response and the subsequent scan of the disk + * did not cause the in-memory index to be marked dirty, then force + * it so that we advance the fsmonitor token in our extension, so + * that future requests don't keep re-requesting the same range. + */ + if (istate->fsmonitor_last_update && + strcmp(istate->fsmonitor_last_update, last_update_token.buf)) + istate->cache_changed |= FSMONITOR_CHANGED; + /* Now that we've updated istate, save the last_update_token */ FREE_AND_NULL(istate->fsmonitor_last_update); istate->fsmonitor_last_update = strbuf_detach(&last_update_token, NULL); From patchwork Thu Apr 1 15:41:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178705 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 18852C43460 for ; Thu, 1 Apr 2021 17:53:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E34E161130 for ; Thu, 1 Apr 2021 17:53:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236016AbhDARxk (ORCPT ); Thu, 1 Apr 2021 13:53:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59754 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237117AbhDARua (ORCPT ); Thu, 1 Apr 2021 13:50:30 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3D9B7C02D54B for ; Thu, 1 Apr 2021 08:41:22 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id c8so2257618wrq.11 for ; Thu, 01 Apr 2021 08:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=BEeFqLDIv6plqXsqBDn5z4EEnMESEOm+rnS6UnhW4j4=; b=F15Bhq0rb1nZfyHT6X4kf7YmUhunMDb4dn/0oamPj/m6mQ+bAINRe2/FlmL5YnQXmR AyoM9JnSg5F/jGVXN7UfUOLTNvKz8GIyHkWib1T/YrBTvHyXlLDOFZOy2I6blzZQQhf0 y6DPmpAP8YNbql+68IECYNUcYlp3La9OJuRCd9uyQK2pQYUJ2c8EOBPYqBgUvDJi8yOA qhatcbr0spw+k46JOfSDMXVsH9VYGK9VZCFoYAsFJOc0xcKp8BSP5YynhI1ILvtxIPZX HynJCSWSl2g1GCdQjjDWSN9rAayUiMuyM4806prm9KBTKWPgLCuex1RI5/KBTAvttvvd EFlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=BEeFqLDIv6plqXsqBDn5z4EEnMESEOm+rnS6UnhW4j4=; b=RmcMqpfId94W2viKltsHgBsfi+TOoRCV6fxQNSWSCyCZVgLJJ2q44pN2iDIWiH75E+ NcU9ydyzMhCeWg7zPSOtrHP2PARbLvozzUj24ewvE+3AmwGOiuigKtz1PQOkgD/0pFwa AgopJGWiv0irqdMIfA5/VHdWQsdT0jprf7uXOrLPec6eI1PftbCi/L7Bw+ma3UTMW7tE yNWc1Y+nD5xLlITq3ARlQB271FteXILAz2SVYvZsoMDpDkMuPniY86xAXetFsW4ywPB0 k1If4+zFOD8uIJFVWloVtT+B0wzf9IJnK7en9tdz2Y92OFXgN1YOQsKovdt+aD2PHROr Vd+g== X-Gm-Message-State: AOAM5328n7PW9GlWIHfOBQ3sFmC1xe7BzdD1/xe4GGscN/zSEQlTkdFm JqJ/mvwAcKj1S2BMS/5KtKtqxmeS1jI= X-Google-Smtp-Source: ABdhPJwzmNDRFb6DtuOqtu00YxxlLUwy2rKagfjd0x7yqjFVoEu9hDmnDPHQ50S4/dwhD7J0gCv+zg== X-Received: by 2002:adf:ed49:: with SMTP id u9mr10529829wro.337.1617291680932; Thu, 01 Apr 2021 08:41:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n9sm10158048wrx.46.2021.04.01.08.41.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:20 -0700 (PDT) Message-Id: <8b2280e5c4d2cec1fe39e90bfc93f059a1d0eb05.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:41:03 +0000 Subject: [PATCH 21/23] t7527: create test for fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 485 +++++++++++++++++++++++++++++++++++ 1 file changed, 485 insertions(+) create mode 100755 t/t7527-builtin-fsmonitor.sh diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh new file mode 100755 index 000000000000..1fd230f1d4c6 --- /dev/null +++ b/t/t7527-builtin-fsmonitor.sh @@ -0,0 +1,485 @@ +#!/bin/sh + +test_description='built-in file system watcher' + +. ./test-lib.sh + +# Ask the fsmonitor daemon to insert a little delay before responding to +# client commands like `git status` and `git fsmonitor--daemon --query` to +# allow recent filesystem events to be received by the daemon. This helps +# the CI/PR builds be more stable. +# +# An arbitrary millisecond value. +# +GIT_TEST_FSMONITOR_CLIENT_DELAY=1000 +export GIT_TEST_FSMONITOR_CLIENT_DELAY + +git version --build-options | grep "feature:" | grep "fsmonitor--daemon" || { + skip_all="The built-in FSMonitor is not supported on this platform" + test_done +} + +kill_repo () { + r=$1 + git -C $r fsmonitor--daemon --stop >/dev/null 2>/dev/null + rm -rf $1 + return 0 +} + +start_daemon () { + case "$#" in + 1) r="-C $1";; + *) r=""; + esac + + git $r fsmonitor--daemon --start || return $? + git $r fsmonitor--daemon --is-running || return $? + + return 0 +} + +test_expect_success 'explicit daemon start and stop' ' + test_when_finished "kill_repo test_explicit" && + + git init test_explicit && + start_daemon test_explicit && + + git -C test_explicit fsmonitor--daemon --stop && + test_must_fail git -C test_explicit fsmonitor--daemon --is-running +' + +test_expect_success 'implicit daemon start' ' + test_when_finished "kill_repo test_implicit" && + + git init test_implicit && + test_must_fail git -C test_implicit fsmonitor--daemon --is-running && + + # query will implicitly start the daemon. + # + # for test-script simplicity, we send a V1 timestamp rather than + # a V2 token. either way, the daemon response to any query contains + # a new V2 token. (the daemon may complain that we sent a V1 request, + # but this test case is only concerned with whether the daemon was + # implicitly started.) + + GIT_TRACE2_EVENT="$PWD/.git/trace" \ + git -C test_implicit fsmonitor--daemon --query 0 >actual && + nul_to_q actual.filtered && + grep "builtin:" actual.filtered && + + # confirm that a daemon was started in the background. + # + # since the mechanism for starting the background daemon is platform + # dependent, just confirm that the foreground command received a + # response from the daemon. + + grep :\"query/response-length\" .git/trace && + + git -C test_implicit fsmonitor--daemon --is-running && + git -C test_implicit fsmonitor--daemon --stop && + test_must_fail git -C test_implicit fsmonitor--daemon --is-running +' + +test_expect_success 'implicit daemon stop (delete .git)' ' + test_when_finished "kill_repo test_implicit_1" && + + git init test_implicit_1 && + + start_daemon test_implicit_1 && + + # deleting the .git directory will implicitly stop the daemon. + rm -rf test_implicit_1/.git && + + # Create an empty .git directory so that the following Git command + # will stay relative to the `-C` directory. Without this, the Git + # command will (override the requested -C argument) and crawl out + # to the containing Git source tree. This would make the test + # result dependent upon whether we were using fsmonitor on our + # development worktree. + + sleep 1 && + mkdir test_implicit_1/.git && + + test_must_fail git -C test_implicit_1 fsmonitor--daemon --is-running +' + +test_expect_success 'implicit daemon stop (rename .git)' ' + test_when_finished "kill_repo test_implicit_2" && + + git init test_implicit_2 && + + start_daemon test_implicit_2 && + + # renaming the .git directory will implicitly stop the daemon. + mv test_implicit_2/.git test_implicit_2/.xxx && + + # Create an empty .git directory so that the following Git command + # will stay relative to the `-C` directory. Without this, the Git + # command will (override the requested -C argument) and crawl out + # to the containing Git source tree. This would make the test + # result dependent upon whether we were using fsmonitor on our + # development worktree. + + sleep 1 && + mkdir test_implicit_2/.git && + + test_must_fail git -C test_implicit_2 fsmonitor--daemon --is-running +' + +test_expect_success 'cannot start multiple daemons' ' + test_when_finished "kill_repo test_multiple" && + + git init test_multiple && + + start_daemon test_multiple && + + test_must_fail git -C test_multiple fsmonitor--daemon --start 2>actual && + grep "fsmonitor--daemon is already running" actual && + + git -C test_multiple fsmonitor--daemon --stop && + test_must_fail git -C test_multiple fsmonitor--daemon --is-running +' + +test_expect_success 'setup' ' + >tracked && + >modified && + >delete && + >rename && + mkdir dir1 && + >dir1/tracked && + >dir1/modified && + >dir1/delete && + >dir1/rename && + mkdir dir2 && + >dir2/tracked && + >dir2/modified && + >dir2/delete && + >dir2/rename && + mkdir dirtorename && + >dirtorename/a && + >dirtorename/b && + + cat >.gitignore <<-\EOF && + .gitignore + expect* + actual* + EOF + + git -c core.useBuiltinFSMonitor= add . && + test_tick && + git -c core.useBuiltinFSMonitor= commit -m initial && + + git config core.useBuiltinFSMonitor true +' + +test_expect_success 'update-index implicitly starts daemon' ' + test_must_fail git fsmonitor--daemon --is-running && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_1" \ + git update-index --fsmonitor && + + git fsmonitor--daemon --is-running && + test_might_fail git fsmonitor--daemon --stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_1 +' + +test_expect_success 'status implicitly starts daemon' ' + test_must_fail git fsmonitor--daemon --is-running && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_2" \ + git status >actual && + + git fsmonitor--daemon --is-running && + test_might_fail git fsmonitor--daemon --stop && + + grep \"event\":\"start\".*\"fsmonitor--daemon\" .git/trace_implicit_2 +' + +edit_files() { + echo 1 >modified + echo 2 >dir1/modified + echo 3 >dir2/modified + >dir1/untracked +} + +delete_files() { + rm -f delete + rm -f dir1/delete + rm -f dir2/delete +} + +create_files() { + echo 1 >new + echo 2 >dir1/new + echo 3 >dir2/new +} + +rename_files() { + mv rename renamed + mv dir1/rename dir1/renamed + mv dir2/rename dir2/renamed +} + +file_to_directory() { + rm -f delete + mkdir delete + echo 1 >delete/new +} + +directory_to_file() { + rm -rf dir1 + echo 1 >dir1 +} + +verify_status() { + git status >actual && + GIT_INDEX_FILE=.git/fresh-index git read-tree master && + GIT_INDEX_FILE=.git/fresh-index git -c core.useBuiltinFSMonitor= status >expect && + test_cmp expect actual && + echo HELLO AFTER && + cat .git/trace && + echo HELLO AFTER +} + +# The next few test cases confirm that our fsmonitor daemon sees each type +# of OS filesystem notification that we care about. At this layer we just +# ensure we are getting the OS notifications and do not try to confirm what +# is reported by `git status`. +# +# We run a simple query after modifying the filesystem just to introduce +# a bit of a delay so that the trace logging from the daemon has time to +# get flushed to disk. +# +# We `reset` and `clean` at the bottom of each test (and before stopping the +# daemon) because these commands might implicitly restart the daemon. + +clean_up_repo_and_stop_daemon () { + git reset --hard HEAD + git clean -fd + git fsmonitor--daemon --stop + rm -f .git/trace +} + +test_expect_success 'edit some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + edit_files && + + git fsmonitor--daemon --query 0 >/dev/null 2>&1 && + + grep "^event: dir1/modified$" .git/trace && + grep "^event: dir2/modified$" .git/trace && + grep "^event: modified$" .git/trace && + grep "^event: dir1/untracked$" .git/trace +' + +test_expect_success 'create some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + create_files && + + git fsmonitor--daemon --query 0 >/dev/null 2>&1 && + + grep "^event: dir1/new$" .git/trace && + grep "^event: dir2/new$" .git/trace && + grep "^event: new$" .git/trace +' + +test_expect_success 'delete some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + delete_files && + + git fsmonitor--daemon --query 0 >/dev/null 2>&1 && + + grep "^event: dir1/delete$" .git/trace && + grep "^event: dir2/delete$" .git/trace && + grep "^event: delete$" .git/trace +' + +test_expect_success 'rename some files' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + rename_files && + + git fsmonitor--daemon --query 0 >/dev/null 2>&1 && + + grep "^event: dir1/rename$" .git/trace && + grep "^event: dir2/rename$" .git/trace && + grep "^event: rename$" .git/trace && + grep "^event: dir1/renamed$" .git/trace && + grep "^event: dir2/renamed$" .git/trace && + grep "^event: renamed$" .git/trace +' + +test_expect_success 'rename directory' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + mv dirtorename dirrenamed && + + git fsmonitor--daemon --query 0 >/dev/null 2>&1 && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace +' + +test_expect_success 'file changes to directory' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + file_to_directory && + + git fsmonitor--daemon --query 0 >/dev/null 2>&1 && + + grep "^event: delete$" .git/trace && + grep "^event: delete/new$" .git/trace +' + +test_expect_success 'directory changes to a file' ' + test_when_finished "clean_up_repo_and_stop_daemon" && + + ( + GIT_TRACE_FSMONITOR="$PWD/.git/trace" && + export GIT_TRACE_FSMONITOR && + + start_daemon + ) && + + directory_to_file && + + git fsmonitor--daemon --query 0 >/dev/null 2>&1 && + + grep "^event: dir1$" .git/trace +' + +# The next few test cases exercise the token-resync code. When filesystem +# drops events (because of filesystem velocity or because the daemon isn't +# polling fast enough), we need to discard the cached data (relative to the +# current token) and start collecting events under a new token. +# +# the 'git fsmonitor--daemon --flush' command can be used to send a "flush" +# message to a running daemon and ask it to do a flush/resync. + +test_expect_success 'flush cached data' ' + test_when_finished "kill_repo test_flush" && + + git init test_flush && + + ( + GIT_TEST_FSMONITOR_TOKEN=true && + export GIT_TEST_FSMONITOR_TOKEN && + + GIT_TRACE_FSMONITOR="$PWD/.git/trace_daemon" && + export GIT_TRACE_FSMONITOR && + + start_daemon test_flush + ) && + + # The daemon should have an initial token with no events in _0 and + # then a few (probably platform-specific number of) events in _1. + # These should both have the same . + + git -C test_flush fsmonitor--daemon --query "builtin:test_00000001:0" >actual_0 && + nul_to_q actual_q0 && + + touch test_flush/file_1 && + touch test_flush/file_2 && + + git -C test_flush fsmonitor--daemon --query "builtin:test_00000001:0" >actual_1 && + nul_to_q actual_q1 && + + grep "file_1" actual_q1 && + + # Force a flush. This will change the , reset the , and + # flush the file data. Then create some events and ensure that the file + # again appears in the cache. It should have the new . + + git -C test_flush fsmonitor--daemon --flush >flush_0 && + nul_to_q flush_q0 && + grep "^builtin:test_00000002:0Q/Q$" flush_q0 && + + git -C test_flush fsmonitor--daemon --query "builtin:test_00000002:0" >actual_2 && + nul_to_q actual_q2 && + + grep "^builtin:test_00000002:0Q$" actual_q2 && + + touch test_flush/file_3 && + + git -C test_flush fsmonitor--daemon --query "builtin:test_00000002:0" >actual_3 && + nul_to_q actual_q3 && + + grep "file_3" actual_q3 +' + +# The next few test cases create repos where the .git directory is NOT +# inside the one of the working directory. That is, where .git is a file +# that points to a directory elsewhere. This happens for submodules and +# non-primary worktrees. + +test_expect_success 'setup worktree base' ' + git init wt-base && + echo 1 >wt-base/file1 && + git -C wt-base add file1 && + git -C wt-base commit -m "c1" +' + +test_expect_success 'worktree with .git file' ' + git -C wt-base worktree add ../wt-secondary && + + ( + GIT_TRACE2_PERF="$PWD/trace2_wt_secondary" && + export GIT_TRACE2_PERF && + + GIT_TRACE_FSMONITOR="$PWD/trace_wt_secondary" && + export GIT_TRACE_FSMONITOR && + + start_daemon wt-secondary + ) && + + git -C wt-secondary fsmonitor--daemon --stop && + test_must_fail git -C wt-secondary fsmonitor--daemon --is-running +' + +test_done From patchwork Thu Apr 1 15:41:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12179035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17EF6C43460 for ; Thu, 1 Apr 2021 18:10:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EC8166112E for ; Thu, 1 Apr 2021 18:10:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S238113AbhDASJR (ORCPT ); Thu, 1 Apr 2021 14:09:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34416 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235300AbhDASDi (ORCPT ); Thu, 1 Apr 2021 14:03:38 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A661CC02D54C for ; Thu, 1 Apr 2021 08:41:22 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id e18so2267093wrt.6 for ; Thu, 01 Apr 2021 08:41:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=pjP4NA613LnanIGMR60vr8n0/EEt96pfhyIvjLJC8jU=; b=TSkjy+56g/uUW4wC4UWKbW4Uk+0xgJUxFxNJi3bZntwDxzFv6gXqhAFEVwDJkl6/96 Kbs/n88Scb3N1PZ4CUEgpMAbrQBsUahZzQS6kmGvUu5eh4BIicwNjDNGz050waODpyrn 3Kn8cAM07pMnZkxPEloV9HS9kAJ+/gcrVlOZUcPU/nfQWsnnyLobh+Jo7gljS0wo77vv uWDUx+x48nd7WzDhwRVgo3mCoxE0MefKTGNV7qC+HUTtSbopdvOJ/bp+a2iu6mWeB0Je xN2kqsh9WPyHoxz/IwrsJb03vN+G1mnQWsVARSoFi5yOp9UBVWHuK4/M0Qjt/eY3YF0a 3j7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=pjP4NA613LnanIGMR60vr8n0/EEt96pfhyIvjLJC8jU=; b=CmCkKAleSxvPMxr1RkvX6+OmcH/5GRhfCosLDQ7eJ/tQMy+Y5Xi8+3kzuDtSfnSoUe lVxfEsSNn2RbSZmYktFdguEqnsHZDM5n6+lOnwi/QXqfKXrAf2nAV50ha/M588k0LNp9 w3dCXX1/j9YU7578F8ZG/1ToK4S96XtLNnDbRP6rLNOjFQ8ORg3oU85gCHwFwLTgZo+N 1Xm6bI9VSp8cHBy7rwqUU3fjGkBmBRzRbyVi35Im6cE9ERvsiDQX+tkDAMoehvhXF4d1 7eK48Gga5yfGdtJAHv+GrLr61B+lpta3bv3Jnyz1lK5vNEbjHMQNnEUinP8NBmCB92y/ ktEA== X-Gm-Message-State: AOAM5320i70MfDF78X3LrnJH5LL0ohFx0hqrWy+fETmh6dEs05j3zMHw 47Y8v1IEeOx1d6vpLfUGsz+9rD0Hs3U= X-Google-Smtp-Source: ABdhPJycqP6yM8lO+9pSIa0n4J0gDG9vmP3GVuFRF43KoQ8SG8uQSo8FaWJiYFr4d5b6qiWb3XMYaA== X-Received: by 2002:adf:d207:: with SMTP id j7mr10420576wrh.150.1617291681497; Thu, 01 Apr 2021 08:41:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a6sm10646158wmm.0.2021.04.01.08.41.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:21 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 01 Apr 2021 15:41:04 +0000 Subject: [PATCH 22/23] p7519: add fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and the "Simple IPC" interface. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 37 +++++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 5eb5044a103c..2d018bc7d589 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -24,7 +24,8 @@ test_description="Test core.fsmonitor" # GIT_PERF_7519_SPLIT_INDEX: used to configure core.splitIndex # GIT_PERF_7519_FSMONITOR: used to configure core.fsMonitor. May be an # absolute path to an integration. May be a space delimited list of -# absolute paths to integrations. +# absolute paths to integrations. (This hook or list of hooks does not +# include the built-in fsmonitor--daemon.) # # The big win for using fsmonitor is the elimination of the need to scan the # working directory looking for changed and untracked files. If the file @@ -135,10 +136,16 @@ test_expect_success "one time repo setup" ' setup_for_fsmonitor() { # set INTEGRATION_SCRIPT depending on the environment - if test -n "$INTEGRATION_PATH" + if test -n "$USE_FSMONITOR_DAEMON" then + git config core.useBuiltinFSMonitor true && + INTEGRATION_SCRIPT=false + elif test -n "$INTEGRATION_PATH" + then + git config core.useBuiltinFSMonitor false && INTEGRATION_SCRIPT="$INTEGRATION_PATH" else + git config core.useBuiltinFSMonitor false && # # Choose integration script based on existence of Watchman. # Fall back to an empty integration script. @@ -285,4 +292,30 @@ test_expect_success "setup without fsmonitor" ' test_fsmonitor_suite trace_stop +# +# Run a full set of perf tests using the built-in fsmonitor--daemon. +# It does not use the Hook API, so it has a different setup. +# Explicitly start the daemon here and before we start client commands +# so that we can later add custom tracing. +# + +test_lazy_prereq HAVE_FSMONITOR_DAEMON ' + git version --build-options | grep "feature:" | grep "fsmonitor--daemon" +' + +if test_have_prereq HAVE_FSMONITOR_DAEMON +then + USE_FSMONITOR_DAEMON=t + + trace_start fsmonitor--daemon--server + git fsmonitor--daemon --start + + trace_start fsmonitor--daemon--client + test_expect_success "setup for fsmonitor--daemon" 'setup_for_fsmonitor' + test_fsmonitor_suite + + git fsmonitor--daemon --stop + trace_stop +fi + test_done From patchwork Thu Apr 1 15:41:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12178701 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C621C43461 for ; Thu, 1 Apr 2021 17:53:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 780C86112E for ; Thu, 1 Apr 2021 17:53:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234236AbhDARxf (ORCPT ); Thu, 1 Apr 2021 13:53:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59766 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237113AbhDARu3 (ORCPT ); Thu, 1 Apr 2021 13:50:29 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3C6D9C02D54D for ; Thu, 1 Apr 2021 08:41:23 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id b9so2265968wrt.8 for ; Thu, 01 Apr 2021 08:41:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tUFUaBwZpxy4xLgtQxuoUKmW481VJeosYXJytJus27E=; b=YMkW3J/gK9W2l0WseD22dnmv51vLx7TWPywe4k0teYR3/6m89IQTFPtfVTHosXPhBP fWvpplu025+6Xhsnq1lplPa16q4DJScoXglXDpY8PJhPIr3lSbYtI6okOPjQwpqbSnCl FXhRIFdNL8cO4sjbo9zNw/GzySQpZex0vmhGB9KecvKBpB/ZDhZ9bHp5khkb6mwF4y2z PLMsK1B3e7RcFRh68Dx+hono6zyZOpqZuAtouM2nQ41cl7qMS3CptBC0E2bTGJptsX9Z U91CauaWVMfEOESf1Z6QbyRdcvZjC9tgZCm180rxwQkGdx4FiBASumxwHWTv5ybQEHpd GvsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tUFUaBwZpxy4xLgtQxuoUKmW481VJeosYXJytJus27E=; b=WVd5E2fCn1u8mygA1ZhhouoOZrMtb4aH0cajnZV0rm2jmW4Vj3gfYwifbyG3TTfOFt 7BBFOcnG/S5/foYE2epJXCw8QCRm0G8OnDB+eDy0LGSnT1iscCnEOynoGF3LHRXVYI2t cNCsu6eshuOosRyd2JuRYtrk3PLebkbMPvEL5i382ugVdmu9MRibuM8B1yGZd2Bhrfu4 PcGcBDh2sH8U/fmz9TGQ9lK/nae8YbJx1QwCBTtbZmlEOsLvoM7BtvVBbzmOqVpdt9MC E7+cNypFUE/lAUpkqTK0tldIEptR9VRKdJxl7kVhOAzTLbYCJA/C2oW0ASkGJk2jXVVq 5WVw== X-Gm-Message-State: AOAM531ReANqlzJC8L5BZTiNSDeUFeBnZ3fjdctBJCCI3kRRwrirHcsO EffCH94kISxNdygJ5MUZSeOYxec9nGI= X-Google-Smtp-Source: ABdhPJzUtQJmO7k9ZrgffRkB7THUa4/+SPBldp7G5Ee8OAbZGIu2YIIr95upyU5AgaEUlulgiEw/gA== X-Received: by 2002:a5d:6a87:: with SMTP id s7mr10434368wru.312.1617291682022; Thu, 01 Apr 2021 08:41:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m10sm8815883wmh.13.2021.04.01.08.41.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Apr 2021 08:41:21 -0700 (PDT) Message-Id: <3eafd0b5cb09dbba8af2abf1a8d602ad27ad3bc4.1617291666.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 01 Apr 2021 15:41:05 +0000 Subject: [PATCH 23/23] t7527: test status with untracked-cache and fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon features and a series of edits and verify that status output is identical. Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 97 ++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 1fd230f1d4c6..ad2188169db7 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -163,6 +163,8 @@ test_expect_success 'setup' ' .gitignore expect* actual* + flush* + trace* EOF git -c core.useBuiltinFSMonitor= add . && @@ -482,4 +484,99 @@ test_expect_success 'worktree with .git file' ' test_must_fail git -C wt-secondary fsmonitor--daemon --is-running ' +# TODO Repeat one of the "edit" tests on wt-secondary and confirm that +# TODO we get the same events and behavior -- that is, that fsmonitor--daemon +# TODO correctly listens to events on both the working directory and to the +# TODO referenced GITDIR. + +test_expect_success 'cleanup worktrees' ' + kill_repo wt-secondary && + kill_repo wt-base +' + +# The next few tests perform arbitrary/contrived file operations and +# confirm that status is correct. That is, that the data (or lack of +# data) from fsmonitor doesn't cause incorrect results. And doesn't +# cause incorrect results when the untracked-cache is enabled. + +test_lazy_prereq UNTRACKED_CACHE ' + { git update-index --test-untracked-cache; ret=$?; } && + test $ret -ne 1 +' + +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon --stop +' + +matrix_clean_up_repo () { + git reset --hard HEAD + git clean -fd +} + +matrix_try () { + uc=$1 + fsm=$2 + fn=$3 + + test_expect_success "Matrix[uc:$uc][fsm:$fsm] $fn" ' + matrix_clean_up_repo && + $fn && + if test $uc = false -a $fsm = false + then + git status --porcelain=v1 >.git/expect.$fn + else + git status --porcelain=v1 >.git/actual.$fn + test_cmp .git/expect.$fn .git/actual.$fn + fi + ' + + return $? +} + +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" +for uc_val in $uc_values +do + if test $uc_val = false + then + test_expect_success "Matrix[uc:$uc_val] disable untracked cache" ' + git config core.untrackedcache false && + git update-index --no-untracked-cache + ' + else + test_expect_success "Matrix[uc:$uc_val] enable untracked cache" ' + git config core.untrackedcache true && + git update-index --untracked-cache + ' + fi + + fsm_values="false true" + for fsm_val in $fsm_values + do + if test $fsm_val = false + then + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor" ' + test_might_fail git config --unset core.useBuiltinFSMonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon --stop 2>/dev/null + ' + else + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor" ' + git config core.useBuiltinFSMonitor true && + git fsmonitor--daemon --start && + git update-index --fsmonitor + ' + fi + + matrix_try $uc_val $fsm_val edit_files + matrix_try $uc_val $fsm_val delete_files + matrix_try $uc_val $fsm_val create_files + matrix_try $uc_val $fsm_val rename_files + matrix_try $uc_val $fsm_val file_to_directory + matrix_try $uc_val $fsm_val directory_to_file + done +done + test_done