From patchwork Fri Mar 25 18:02:44 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791956 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E39D4C4332F for ; Fri, 25 Mar 2022 19:32:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230224AbiCYTdo (ORCPT ); Fri, 25 Mar 2022 15:33:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54536 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230468AbiCYTc4 (ORCPT ); Fri, 25 Mar 2022 15:32:56 -0400 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6F8DE216FB9 for ; Fri, 25 Mar 2022 12:07:30 -0700 (PDT) Received: by mail-wm1-x332.google.com with SMTP id r204-20020a1c44d5000000b0038ccb70e239so135675wma.3 for ; Fri, 25 Mar 2022 12:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=GykMGxH+vcSxKSOuzIgVgE2plWrsxxaHhbdq7JLSrgU=; b=GUs1Y8GTmGubx9lG1XYDb21uEL1LwpQ5SgXwwNvcJ0DgYtW2ditw4hk/AMAVfOfFz4 SxRJSrJgadvfuX5s/M5y1c6vkNK2gJDi+RV1f0X7vDxoAyjsl81L/aHweYa+2sGI0Kcu wja0zfZXy+i8Kwz2lqj2xil9dXDThujWgBeiI9cRUwXOp3q7wPyWmRlIWxl7HRJs62Wo d2yiPAbiRAq7Gfz2b8BLqUxbaGUJwN1IJviuIozCiQMcfzLKhG7U2wBe1UIuZ5+QUpIi H5AUR7ltD8z3uzJs6YV46Y5VdpT2LFE7BCX00JwzsJ8yqydKwylOFyAV0p6FIyNydpwL NDrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=GykMGxH+vcSxKSOuzIgVgE2plWrsxxaHhbdq7JLSrgU=; b=O6uOTv39Z4MaY6yIz6KjCvfDsXQyj1FiS9unNix7st15vBlU470RFYpEkl12jm1x/C gEc+tcJMzOMQGQ2MPrwKFI6m+PGuz+SryHBdx3USLR35DBCfbAQ61QdRcVHuOktgK6Yd waQXK4ba5j3Dgc9LwbNGbitR1ak2OCJLwAgrgTFz5lC50cg5hsjCfybRik5Q+0COpMGV TnCV8MGRuwm0JItHy1zaAeVqLTDBcdgyzKrd1nZQuwinZdZSFKGl9FKzM9KTiVV7G5n9 l2MI+EMF/NUBKfPrNAoTT6k1IraDaTUOUVlNicM/g6L6s7CTFNUXO/igHnhk4+SPYifo PzPA== X-Gm-Message-State: AOAM533/B+3XausFuSU6CsE1Pe9hJbzoTUZTqj1okDAtDTyCD0megs0l 1VEJveGSqQkWBqftNWS5JDyICu2srks= X-Google-Smtp-Source: ABdhPJx89/8i6IWtXFSBmisqvkun0yqR9y+bzYmfxpnCxaOZMPIIlgoE3IYeZCoO85VRpD8JFEZaLA== X-Received: by 2002:a7b:c74c:0:b0:38c:7645:9896 with SMTP id w12-20020a7bc74c000000b0038c76459896mr11337272wmk.83.1648231395923; Fri, 25 Mar 2022 11:03:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s17-20020adfdb11000000b001f02d5fea43sm6096493wri.98.2022.03.25.11.03.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:15 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:44 +0000 Subject: [PATCH v9 01/30] fsmonitor: enhance existing comments, clarify trivial response handling Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- fsmonitor.c | 64 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 41 insertions(+), 23 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index ab9bfc60b34..448d0ee33f5 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -168,29 +168,15 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf * if (result) trace2_data_intmax("fsm_hook", NULL, "query/failed", result); - else { + else trace2_data_intmax("fsm_hook", NULL, "query/response-length", query_result->len); - if (fsmonitor_is_trivial_response(query_result)) - trace2_data_intmax("fsm_hook", NULL, - "query/trivial-response", 1); - } - trace2_region_leave("fsm_hook", "query", NULL); return result; } -int fsmonitor_is_trivial_response(const struct strbuf *query_result) -{ - static char trivial_response[3] = { '\0', '/', '\0' }; - - return query_result->len >= 3 && - !memcmp(trivial_response, - &query_result->buf[query_result->len - 3], 3); -} - static void fsmonitor_refresh_callback(struct index_state *istate, char *name) { int i, len = strlen(name); @@ -238,6 +224,7 @@ void refresh_fsmonitor(struct index_state *istate) struct strbuf last_update_token = STRBUF_INIT; char *buf; unsigned int i; + int is_trivial = 0; if (!core_fsmonitor || istate->fsmonitor_has_run_once) return; @@ -283,6 +270,7 @@ void refresh_fsmonitor(struct index_state *istate) query_success = 0; } else { bol = last_update_token.len + 1; + is_trivial = query_result.buf[bol] == '/'; } } else if (hook_version < 0) { hook_version = HOOK_INTERFACE_VERSION1; @@ -294,16 +282,38 @@ void refresh_fsmonitor(struct index_state *istate) if (hook_version == HOOK_INTERFACE_VERSION1) { query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1, istate->fsmonitor_last_update, &query_result); + if (query_success) + is_trivial = query_result.buf[0] == '/'; } + if (is_trivial) + trace2_data_intmax("fsm_hook", NULL, + "query/trivial-response", 1); + trace_performance_since(last_update, "fsmonitor process '%s'", core_fsmonitor); trace_printf_key(&trace_fsmonitor, "fsmonitor process '%s' returned %s", core_fsmonitor, query_success ? "success" : "failure"); } - /* a fsmonitor process can return '/' to indicate all entries are invalid */ - if (query_success && query_result.buf[bol] != '/') { - /* Mark all entries returned by the monitor as dirty */ + /* + * The response from FSMonitor (excluding the header token) is + * either: + * + * [a] a (possibly empty) list of NUL delimited relative + * pathnames of changed paths. This list can contain + * files and directories. Directories have a trailing + * slash. + * + * [b] a single '/' to indicate the provider had no + * information and that we should consider everything + * invalid. We call this a trivial response. + */ + if (query_success && !is_trivial) { + /* + * Mark all pathnames returned by the monitor as dirty. + * + * This updates both the cache-entries and the untracked-cache. + */ buf = query_result.buf; for (i = bol; i < query_result.len; i++) { if (buf[i] != '\0') @@ -318,11 +328,16 @@ void refresh_fsmonitor(struct index_state *istate) if (istate->untracked) istate->untracked->use_fsmonitor = 1; } else { - - /* We only want to run the post index changed hook if we've actually changed entries, so keep track - * if we actually changed entries or not */ + /* + * We failed to get a response or received a trivial response, + * so invalidate everything. + * + * We only want to run the post index changed hook if + * we've actually changed entries, so keep track if we + * actually changed entries or not. + */ int is_cache_changed = 0; - /* Mark all entries invalid */ + for (i = 0; i < istate->cache_nr; i++) { if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID) { is_cache_changed = 1; @@ -330,7 +345,10 @@ void refresh_fsmonitor(struct index_state *istate) } } - /* If we're going to check every file, ensure we save the results */ + /* + * If we're going to check every file, ensure we save + * the results. + */ if (is_cache_changed) istate->cache_changed |= FSMONITOR_CHANGED; From patchwork Fri Mar 25 18:02:45 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791954 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E6F0C433FE for ; Fri, 25 Mar 2022 19:32:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230170AbiCYTdj (ORCPT ); Fri, 25 Mar 2022 15:33:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51120 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230426AbiCYTcx (ORCPT ); Fri, 25 Mar 2022 15:32:53 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2AC592DC005 for ; Fri, 25 Mar 2022 12:07:25 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id r64so4963262wmr.4 for ; Fri, 25 Mar 2022 12:07:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tTgHuXtnMPt1qW3RcIDoocZO6h35fSaRPRvfUmgg3B0=; b=RcCBQPQcg4mj9t4Fu2bKFOTwPhEpXvLtUecvXBX9p3gQ72QBsHgeQRhhsNDvfXP+h3 qwXfawXhwK49eGdiVXq8s7FmnCv84EmdDmMPkSAoZFfB8OiRAsxEX1OX7lnd/5cUvoGa 6PxPN7l2pgk6CvOHL4Vp1JbnJI4/1rfd2uB8Js2ygEfBpp/+i+skQB27QZWtDvER1pYw GjL9oESTo03CsL0EQC5nstfPG6z3SDT1WWveePSWOQd4Pfd98fBO7u0YgrKEas2LN/nw JWR6owMoGiKvRjvy4321WAKwo02tT6VsK6hWnstmX9vjd77oRu+PYmA9oiCz4nmQUhXY sQhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tTgHuXtnMPt1qW3RcIDoocZO6h35fSaRPRvfUmgg3B0=; b=b7rkrFWGYgVEollx50/Yp1QNQOfOUOffeJAqMM00p4dMCsa0IDqX21UGV+plPDFPLM wfNeNEo/afBdRZAZfVTiDdIvO1+myoffSlO13pZ4DNxicukjka5O011VRADBO0s+MCk0 FIQLr2dCzwVDUMPs4W0AKpBm+Na6hrPJc4WI2q3JAfCmN0m3XS0enA09nVWSmjZ74Q5V 7fC7st9WxrD6HWeUWN8mxhceStghmIGnoY0GyvBsVfMsE18IN6vlXGBkTYEweb+vajAg cnl5n9MlfVOK0DxSKvOByfexlQoUXWRrwA68cpASeyQkr2SancSGhE7Hflner7P1IFVQ pDXQ== X-Gm-Message-State: AOAM531DFaR2NQnXkTNrHH1Jl7H9sWdR/vSAsPdREMEHUxPSGxYit0KJ NISZcDJ8wqo0e7en2BEaNEjyKaQ05h4= X-Google-Smtp-Source: ABdhPJx1/nELUvCZWF5bcM9gyoqz8ZwJi1pSjX7E1Ej6YVUuWn9UTF6XKO4tcpqW02Fh50XZJNBgKg== X-Received: by 2002:a7b:c40f:0:b0:389:f3ad:5166 with SMTP id k15-20020a7bc40f000000b00389f3ad5166mr11102849wmi.63.1648231397134; Fri, 25 Mar 2022 11:03:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z5-20020a05600c0a0500b0037bb8df81a2sm10773643wmp.13.2022.03.25.11.03.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:16 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:45 +0000 Subject: [PATCH v9 02/30] fsmonitor-ipc: create client routines for git-fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create fsmonitor_ipc__*() client routines to spawn the built-in file system monitor daemon and send it an IPC request using the `Simple IPC` API. Stub in empty fsmonitor_ipc__*() functions for unsupported platforms. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- Makefile | 1 + fsmonitor-ipc.c | 171 ++++++++++++++++++++++++++++++++++++++++++++++++ fsmonitor-ipc.h | 48 ++++++++++++++ 3 files changed, 220 insertions(+) create mode 100644 fsmonitor-ipc.c create mode 100644 fsmonitor-ipc.h diff --git a/Makefile b/Makefile index 6f0b4b775fe..a19d850e716 100644 --- a/Makefile +++ b/Makefile @@ -907,6 +907,7 @@ LIB_OBJS += fetch-pack.o LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o +LIB_OBJS += fsmonitor-ipc.o LIB_OBJS += gettext.o LIB_OBJS += gpg-interface.o LIB_OBJS += graph.o diff --git a/fsmonitor-ipc.c b/fsmonitor-ipc.c new file mode 100644 index 00000000000..789e7397baa --- /dev/null +++ b/fsmonitor-ipc.c @@ -0,0 +1,171 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "simple-ipc.h" +#include "fsmonitor-ipc.h" +#include "run-command.h" +#include "strbuf.h" +#include "trace2.h" + +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND + +/* + * A trivial implementation of the fsmonitor_ipc__ API for unsupported + * platforms. + */ + +int fsmonitor_ipc__is_supported(void) +{ + return 0; +} + +const char *fsmonitor_ipc__get_path(void) +{ + return NULL; +} + +enum ipc_active_state fsmonitor_ipc__get_state(void) +{ + return IPC_STATE__OTHER_ERROR; +} + +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer) +{ + return -1; +} + +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer) +{ + return -1; +} + +#else + +int fsmonitor_ipc__is_supported(void) +{ + return 1; +} + +GIT_PATH_FUNC(fsmonitor_ipc__get_path, "fsmonitor--daemon.ipc") + +enum ipc_active_state fsmonitor_ipc__get_state(void) +{ + return ipc_get_active_state(fsmonitor_ipc__get_path()); +} + +static int spawn_daemon(void) +{ + const char *args[] = { "fsmonitor--daemon", "start", NULL }; + + return run_command_v_opt_tr2(args, RUN_COMMAND_NO_STDIN | RUN_GIT_CMD, + "fsmonitor"); +} + +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer) +{ + int ret = -1; + int tried_to_spawn = 0; + enum ipc_active_state state = IPC_STATE__OTHER_ERROR; + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + const char *tok = since_token ? since_token : ""; + size_t tok_len = since_token ? strlen(since_token) : 0; + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + trace2_region_enter("fsm_client", "query", NULL); + trace2_data_string("fsm_client", NULL, "query/command", tok); + +try_again: + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + + switch (state) { + case IPC_STATE__LISTENING: + ret = ipc_client_send_command_to_connection( + connection, tok, tok_len, answer); + ipc_client_close_connection(connection); + + trace2_data_intmax("fsm_client", NULL, + "query/response-length", answer->len); + goto done; + + case IPC_STATE__NOT_LISTENING: + case IPC_STATE__PATH_NOT_FOUND: + if (tried_to_spawn) + goto done; + + tried_to_spawn++; + if (spawn_daemon()) + goto done; + + /* + * Try again, but this time give the daemon a chance to + * actually create the pipe/socket. + * + * Granted, the daemon just started so it can't possibly have + * any FS cached yet, so we'll always get a trivial answer. + * BUT the answer should include a new token that can serve + * as the basis for subsequent requests. + */ + options.wait_if_not_found = 1; + goto try_again; + + case IPC_STATE__INVALID_PATH: + ret = error(_("fsmonitor_ipc__send_query: invalid path '%s'"), + fsmonitor_ipc__get_path()); + goto done; + + case IPC_STATE__OTHER_ERROR: + default: + ret = error(_("fsmonitor_ipc__send_query: unspecified error on '%s'"), + fsmonitor_ipc__get_path()); + goto done; + } + +done: + trace2_region_leave("fsm_client", "query", NULL); + + return ret; +} + +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer) +{ + struct ipc_client_connection *connection = NULL; + struct ipc_client_connect_options options + = IPC_CLIENT_CONNECT_OPTIONS_INIT; + int ret; + enum ipc_active_state state; + const char *c = command ? command : ""; + size_t c_len = command ? strlen(command) : 0; + + strbuf_reset(answer); + + options.wait_if_busy = 1; + options.wait_if_not_found = 0; + + state = ipc_client_try_connect(fsmonitor_ipc__get_path(), &options, + &connection); + if (state != IPC_STATE__LISTENING) { + die(_("fsmonitor--daemon is not running")); + return -1; + } + + ret = ipc_client_send_command_to_connection(connection, c, c_len, + answer); + ipc_client_close_connection(connection); + + if (ret == -1) { + die(_("could not send '%s' command to fsmonitor--daemon"), c); + return -1; + } + + return 0; +} + +#endif diff --git a/fsmonitor-ipc.h b/fsmonitor-ipc.h new file mode 100644 index 00000000000..b6a7067c3af --- /dev/null +++ b/fsmonitor-ipc.h @@ -0,0 +1,48 @@ +#ifndef FSMONITOR_IPC_H +#define FSMONITOR_IPC_H + +#include "simple-ipc.h" + +/* + * Returns true if built-in file system monitor daemon is defined + * for this platform. + */ +int fsmonitor_ipc__is_supported(void); + +/* + * Returns the pathname to the IPC named pipe or Unix domain socket + * where a `git-fsmonitor--daemon` process will listen. This is a + * per-worktree value. + * + * Returns NULL if the daemon is not supported on this platform. + */ +const char *fsmonitor_ipc__get_path(void); + +/* + * Try to determine whether there is a `git-fsmonitor--daemon` process + * listening on the IPC pipe/socket. + */ +enum ipc_active_state fsmonitor_ipc__get_state(void); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc + * and ask for the set of changed files since the given token. + * + * Spawn a daemon process in the background if necessary. + * + * Returns -1 on error; 0 on success. + */ +int fsmonitor_ipc__send_query(const char *since_token, + struct strbuf *answer); + +/* + * Connect to a `git-fsmonitor--daemon` process via simple-ipc and + * send a command verb. If no daemon is available, we DO NOT try to + * start one. + * + * Returns -1 on error; 0 on success. + */ +int fsmonitor_ipc__send_command(const char *command, + struct strbuf *answer); + +#endif /* FSMONITOR_IPC_H */ From patchwork Fri Mar 25 18:02:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 976F9C433EF for ; Fri, 25 Mar 2022 19:25:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229648AbiCYT0n (ORCPT ); Fri, 25 Mar 2022 15:26:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229683AbiCYT0W (ORCPT ); Fri, 25 Mar 2022 15:26:22 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E807E1DEC20 for ; Fri, 25 Mar 2022 11:59:04 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id p12-20020a05600c430c00b0038cbdf52227so4942661wme.2 for ; Fri, 25 Mar 2022 11:59:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cVZrxHwMCRQEqt+37fwGPBYBrCcn6hveyc9yl93zpfY=; b=ZDRDpOMyhR3aUCntnyu3tiD5bBkwXOE188wJ6Ia+kSuIknQSLzd7qH+JNVcnNe6KKH RjceHPHM9vKfytV0Dvf1y1myX0QiaizgKmsPPAlVBWW+BjgnOr/H9pKNTR6hvzV6W05C s6odV9c4qwU2hVg00E2xcvIdwoxJ2F9lKNQEhBKq3fRxudyDGTJlBKVNBdgEM3N/X7bj qs4sNN+Rmnsh8MXK9hAKgs1U5E0aPIYrFitGHrQ7lg5UmF3TqNR4QeeuYogsLrePsSlE wvyGolyV1q5E9HeG75Ccj+oU5FmFNSjE0gEKqEknJ1hm3oBB7sDWGDu05YcPUCV0PRQk r3Nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cVZrxHwMCRQEqt+37fwGPBYBrCcn6hveyc9yl93zpfY=; b=Tdg2HQRPt8CneDZNjuaBSukAS/6vmnb3vsJB5FoKBX1QV1OschSnvxd/He/3zNWMl+ tnWcnHh32uml/MnzfXfdXHUsSCgNyWu/5hfAH+S1ORJC9UjvCe6hN/1yf6784y1XVBhN ERapD62O45sOzYQwYK6iD/gXWs2N2YgVNLZGghOxlk2RiGjQ13sy7ygwFQZHOgCfkYOA Q3+1GIQKKDQgKIpgz2atHjG1GVPJ1Yv7P8CWdVWUsfc7vESzClvPJ0F9kmjVNoFO+74T tFfJ6+fanTB6aUsOPdq+6DfEJ6eIKRBA8OhT3nobN95zniZwpp/M2fDkv0bZRon00Etb 6J5Q== X-Gm-Message-State: AOAM532UFzcikWcFJTqPhwFTOdM2WppLTg0rWIDBivYgFh8a9WPgaFxd J9kAXogSF9l5xLybFf+NCKE9UdNs5DQ= X-Google-Smtp-Source: ABdhPJw4zgNEOcn+/m2N9N1tkLDlhjlb0jvoenFmvTC3A74ei6xAftlFfY+NWv1CJZrNllUSJ6A9Lg== X-Received: by 2002:a1c:f219:0:b0:38c:782c:3bb with SMTP id s25-20020a1cf219000000b0038c782c03bbmr20373437wmc.94.1648231398126; Fri, 25 Mar 2022 11:03:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o5-20020a5d4a85000000b00205a8bb9c0dsm2242118wrq.90.2022.03.25.11.03.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:17 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:46 +0000 Subject: [PATCH v9 03/30] fsmonitor: config settings are repository-specific Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Move fsmonitor config settings to a new and opaque `struct fsmonitor_settings` structure. Add a lazily-loaded pointer to this into `struct repo_settings` Create an `enum fsmonitor_mode` type in `struct fsmonitor_settings` to represent the state of fsmonitor. This lets us represent which, if any, fsmonitor provider (hook or IPC) is enabled. Create `fsm_settings__get_*()` getters to lazily look up fsmonitor- related config settings. Get rid of the `core_fsmonitor` global variable. Move the code to lookup the existing `core.fsmonitor` config value into the fsmonitor settings. Create a hook pathname variable in `struct fsmonitor-settings` and only set it when in hook mode. Extend the definition of `core.fsmonitor` to be either a boolean or a hook pathname. When true, the builtin FSMonitor is used. When false or unset, no FSMonitor (neither builtin nor hook) is used. The existing `core_fsmonitor` global variable was used to store the pathname to the fsmonitor hook *and* it was used as a boolean to see if fsmonitor was enabled. This dual usage and global visibility leads to confusion when we add the IPC-based provider. So lets hide the details in fsmonitor-settings.c and let it decide which provider to use in the case of multiple settings. This avoids cluttering up repo-settings.c with these private details. A future commit in builtin-fsmonitor series will add the ability to disqualify worktrees for various reasons, such as being mounted from a remote volume, where fsmonitor should not be started. Having the config settings hidden in fsmonitor-settings.c allows such worktree restrictions to override the config values used. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- Makefile | 1 + builtin/update-index.c | 7 ++- cache.h | 1 - config.c | 14 ----- config.h | 1 - environment.c | 1 - fsmonitor-settings.c | 114 +++++++++++++++++++++++++++++++++++++++++ fsmonitor-settings.h | 21 ++++++++ fsmonitor.c | 63 ++++++++++++++--------- fsmonitor.h | 15 ++++-- repository.h | 3 ++ t/README | 4 +- 12 files changed, 196 insertions(+), 49 deletions(-) create mode 100644 fsmonitor-settings.c create mode 100644 fsmonitor-settings.h diff --git a/Makefile b/Makefile index a19d850e716..707a56d4c11 100644 --- a/Makefile +++ b/Makefile @@ -908,6 +908,7 @@ LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o LIB_OBJS += fsmonitor-ipc.o +LIB_OBJS += fsmonitor-settings.o LIB_OBJS += gettext.o LIB_OBJS += gpg-interface.o LIB_OBJS += graph.o diff --git a/builtin/update-index.c b/builtin/update-index.c index aafe7eeac2a..876112abb21 100644 --- a/builtin/update-index.c +++ b/builtin/update-index.c @@ -1236,14 +1236,17 @@ int cmd_update_index(int argc, const char **argv, const char *prefix) } if (fsmonitor > 0) { - if (git_config_get_fsmonitor() == 0) + enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r); + if (fsm_mode == FSMONITOR_MODE_DISABLED) { warning(_("core.fsmonitor is unset; " "set it if you really want to " "enable fsmonitor")); + } add_fsmonitor(&the_index); report(_("fsmonitor enabled")); } else if (!fsmonitor) { - if (git_config_get_fsmonitor() == 1) + enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r); + if (fsm_mode > FSMONITOR_MODE_DISABLED) warning(_("core.fsmonitor is set; " "remove it if you really want to " "disable fsmonitor")); diff --git a/cache.h b/cache.h index 04d4d2db25c..aaf334e2aa4 100644 --- a/cache.h +++ b/cache.h @@ -999,7 +999,6 @@ extern int core_preload_index; extern int precomposed_unicode; extern int protect_hfs; extern int protect_ntfs; -extern const char *core_fsmonitor; extern int core_apply_sparse_checkout; extern int core_sparse_checkout_cone; diff --git a/config.c b/config.c index 383b1a4885b..3f9b0739a78 100644 --- a/config.c +++ b/config.c @@ -2626,20 +2626,6 @@ int git_config_get_max_percent_split_change(void) return -1; /* default value */ } -int git_config_get_fsmonitor(void) -{ - if (git_config_get_pathname("core.fsmonitor", &core_fsmonitor)) - core_fsmonitor = getenv("GIT_TEST_FSMONITOR"); - - if (core_fsmonitor && !*core_fsmonitor) - core_fsmonitor = NULL; - - if (core_fsmonitor) - return 1; - - return 0; -} - int git_config_get_index_threads(int *dest) { int is_bool, val; diff --git a/config.h b/config.h index bb49baf1ee0..7654f61c634 100644 --- a/config.h +++ b/config.h @@ -597,7 +597,6 @@ int git_config_get_pathname(const char *key, const char **dest); int git_config_get_index_threads(int *dest); int git_config_get_split_index(void); int git_config_get_max_percent_split_change(void); -int git_config_get_fsmonitor(void); /* This dies if the configured or default date is in the future */ int git_config_get_expiry(const char *key, const char **output); diff --git a/environment.c b/environment.c index fd0501e77a5..00682e638d7 100644 --- a/environment.c +++ b/environment.c @@ -84,7 +84,6 @@ int protect_hfs = PROTECT_HFS_DEFAULT; #define PROTECT_NTFS_DEFAULT 1 #endif int protect_ntfs = PROTECT_NTFS_DEFAULT; -const char *core_fsmonitor; /* * The character that begins a commented line in user-editable file diff --git a/fsmonitor-settings.c b/fsmonitor-settings.c new file mode 100644 index 00000000000..757d230d538 --- /dev/null +++ b/fsmonitor-settings.c @@ -0,0 +1,114 @@ +#include "cache.h" +#include "config.h" +#include "repository.h" +#include "fsmonitor-settings.h" + +/* + * We keep this structure defintion private and have getters + * for all fields so that we can lazy load it as needed. + */ +struct fsmonitor_settings { + enum fsmonitor_mode mode; + char *hook_path; +}; + +static void lookup_fsmonitor_settings(struct repository *r) +{ + struct fsmonitor_settings *s; + const char *const_str; + int bool_value; + + if (r->settings.fsmonitor) + return; + + CALLOC_ARRAY(s, 1); + s->mode = FSMONITOR_MODE_DISABLED; + + r->settings.fsmonitor = s; + + /* + * Overload the existing "core.fsmonitor" config setting (which + * has historically been either unset or a hook pathname) to + * now allow a boolean value to enable the builtin FSMonitor + * or to turn everything off. (This does imply that you can't + * use a hook script named "true" or "false", but that's OK.) + */ + switch (repo_config_get_maybe_bool(r, "core.fsmonitor", &bool_value)) { + + case 0: /* config value was set to */ + if (bool_value) + fsm_settings__set_ipc(r); + return; + + case 1: /* config value was unset */ + const_str = getenv("GIT_TEST_FSMONITOR"); + break; + + case -1: /* config value set to an arbitrary string */ + if (repo_config_get_pathname(r, "core.fsmonitor", &const_str)) + return; /* should not happen */ + break; + + default: /* should not happen */ + return; + } + + if (!const_str || !*const_str) + return; + + fsm_settings__set_hook(r, const_str); +} + +enum fsmonitor_mode fsm_settings__get_mode(struct repository *r) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + return r->settings.fsmonitor->mode; +} + +const char *fsm_settings__get_hook_path(struct repository *r) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + return r->settings.fsmonitor->hook_path; +} + +void fsm_settings__set_ipc(struct repository *r) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + r->settings.fsmonitor->mode = FSMONITOR_MODE_IPC; + FREE_AND_NULL(r->settings.fsmonitor->hook_path); +} + +void fsm_settings__set_hook(struct repository *r, const char *path) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + r->settings.fsmonitor->mode = FSMONITOR_MODE_HOOK; + FREE_AND_NULL(r->settings.fsmonitor->hook_path); + r->settings.fsmonitor->hook_path = strdup(path); +} + +void fsm_settings__set_disabled(struct repository *r) +{ + if (!r) + r = the_repository; + + lookup_fsmonitor_settings(r); + + r->settings.fsmonitor->mode = FSMONITOR_MODE_DISABLED; + FREE_AND_NULL(r->settings.fsmonitor->hook_path); +} diff --git a/fsmonitor-settings.h b/fsmonitor-settings.h new file mode 100644 index 00000000000..a4c5d7b4889 --- /dev/null +++ b/fsmonitor-settings.h @@ -0,0 +1,21 @@ +#ifndef FSMONITOR_SETTINGS_H +#define FSMONITOR_SETTINGS_H + +struct repository; + +enum fsmonitor_mode { + FSMONITOR_MODE_DISABLED = 0, + FSMONITOR_MODE_HOOK = 1, /* core.fsmonitor= */ + FSMONITOR_MODE_IPC = 2, /* core.fsmonitor= */ +}; + +void fsm_settings__set_ipc(struct repository *r); +void fsm_settings__set_hook(struct repository *r, const char *path); +void fsm_settings__set_disabled(struct repository *r); + +enum fsmonitor_mode fsm_settings__get_mode(struct repository *r); +const char *fsm_settings__get_hook_path(struct repository *r); + +struct fsmonitor_settings; + +#endif /* FSMONITOR_SETTINGS_H */ diff --git a/fsmonitor.c b/fsmonitor.c index 448d0ee33f5..0e961b74d82 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -3,6 +3,7 @@ #include "dir.h" #include "ewah/ewok.h" #include "fsmonitor.h" +#include "fsmonitor-ipc.h" #include "run-command.h" #include "strbuf.h" @@ -148,15 +149,18 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate) /* * Call the query-fsmonitor hook passing the last update token of the saved results. */ -static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result) +static int query_fsmonitor_hook(struct repository *r, + int version, + const char *last_update, + struct strbuf *query_result) { struct child_process cp = CHILD_PROCESS_INIT; int result; - if (!core_fsmonitor) + if (fsm_settings__get_mode(r) != FSMONITOR_MODE_HOOK) return -1; - strvec_push(&cp.args, core_fsmonitor); + strvec_push(&cp.args, fsm_settings__get_hook_path(r)); strvec_pushf(&cp.args, "%d", version); strvec_pushf(&cp.args, "%s", last_update); cp.use_shell = 1; @@ -225,17 +229,28 @@ void refresh_fsmonitor(struct index_state *istate) char *buf; unsigned int i; int is_trivial = 0; + struct repository *r = istate->repo ? istate->repo : the_repository; + enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(r); - if (!core_fsmonitor || istate->fsmonitor_has_run_once) + if (fsm_mode <= FSMONITOR_MODE_DISABLED || + istate->fsmonitor_has_run_once) return; - hook_version = fsmonitor_hook_version(); - istate->fsmonitor_has_run_once = 1; trace_printf_key(&trace_fsmonitor, "refresh fsmonitor"); + + if (fsm_mode == FSMONITOR_MODE_IPC) { + /* TODO */ + return; + } + + assert(fsm_mode == FSMONITOR_MODE_HOOK); + + hook_version = fsmonitor_hook_version(); + /* - * This could be racy so save the date/time now and query_fsmonitor + * This could be racy so save the date/time now and query_fsmonitor_hook * should be inclusive to ensure we don't miss potential changes. */ last_update = getnanotime(); @@ -243,13 +258,14 @@ void refresh_fsmonitor(struct index_state *istate) strbuf_addf(&last_update_token, "%"PRIu64"", last_update); /* - * If we have a last update token, call query_fsmonitor for the set of + * If we have a last update token, call query_fsmonitor_hook for the set of * changes since that token, else assume everything is possibly dirty * and check it all. */ if (istate->fsmonitor_last_update) { if (hook_version == -1 || hook_version == HOOK_INTERFACE_VERSION2) { - query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION2, + query_success = !query_fsmonitor_hook( + r, HOOK_INTERFACE_VERSION2, istate->fsmonitor_last_update, &query_result); if (query_success) { @@ -280,7 +296,8 @@ void refresh_fsmonitor(struct index_state *istate) } if (hook_version == HOOK_INTERFACE_VERSION1) { - query_success = !query_fsmonitor(HOOK_INTERFACE_VERSION1, + query_success = !query_fsmonitor_hook( + r, HOOK_INTERFACE_VERSION1, istate->fsmonitor_last_update, &query_result); if (query_success) is_trivial = query_result.buf[0] == '/'; @@ -290,9 +307,12 @@ void refresh_fsmonitor(struct index_state *istate) trace2_data_intmax("fsm_hook", NULL, "query/trivial-response", 1); - trace_performance_since(last_update, "fsmonitor process '%s'", core_fsmonitor); - trace_printf_key(&trace_fsmonitor, "fsmonitor process '%s' returned %s", - core_fsmonitor, query_success ? "success" : "failure"); + trace_performance_since(last_update, "fsmonitor process '%s'", + fsm_settings__get_hook_path(r)); + trace_printf_key(&trace_fsmonitor, + "fsmonitor process '%s' returned %s", + fsm_settings__get_hook_path(r), + query_success ? "success" : "failure"); } /* @@ -429,7 +449,8 @@ void remove_fsmonitor(struct index_state *istate) void tweak_fsmonitor(struct index_state *istate) { unsigned int i; - int fsmonitor_enabled = git_config_get_fsmonitor(); + int fsmonitor_enabled = (fsm_settings__get_mode(istate->repo) + > FSMONITOR_MODE_DISABLED); if (istate->fsmonitor_dirty) { if (fsmonitor_enabled) { @@ -449,16 +470,8 @@ void tweak_fsmonitor(struct index_state *istate) istate->fsmonitor_dirty = NULL; } - switch (fsmonitor_enabled) { - case -1: /* keep: do nothing */ - break; - case 0: /* false */ - remove_fsmonitor(istate); - break; - case 1: /* true */ + if (fsmonitor_enabled) add_fsmonitor(istate); - break; - default: /* unknown value: do nothing */ - break; - } + else + remove_fsmonitor(istate); } diff --git a/fsmonitor.h b/fsmonitor.h index f20d72631d7..3f41f653691 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -3,6 +3,7 @@ #include "cache.h" #include "dir.h" +#include "fsmonitor-settings.h" extern struct trace_key trace_fsmonitor; @@ -57,7 +58,10 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result); */ static inline int is_fsmonitor_refreshed(const struct index_state *istate) { - return !core_fsmonitor || istate->fsmonitor_has_run_once; + enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo); + + return fsm_mode <= FSMONITOR_MODE_DISABLED || + istate->fsmonitor_has_run_once; } /* @@ -67,7 +71,10 @@ static inline int is_fsmonitor_refreshed(const struct index_state *istate) */ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache_entry *ce) { - if (core_fsmonitor && !(ce->ce_flags & CE_FSMONITOR_VALID)) { + enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo); + + if (fsm_mode > FSMONITOR_MODE_DISABLED && + !(ce->ce_flags & CE_FSMONITOR_VALID)) { istate->cache_changed = 1; ce->ce_flags |= CE_FSMONITOR_VALID; trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_clean '%s'", ce->name); @@ -83,7 +90,9 @@ static inline void mark_fsmonitor_valid(struct index_state *istate, struct cache */ static inline void mark_fsmonitor_invalid(struct index_state *istate, struct cache_entry *ce) { - if (core_fsmonitor) { + enum fsmonitor_mode fsm_mode = fsm_settings__get_mode(istate->repo); + + if (fsm_mode > FSMONITOR_MODE_DISABLED) { ce->ce_flags &= ~CE_FSMONITOR_VALID; untracked_cache_invalidate_path(istate, ce->name, 1); trace_printf_key(&trace_fsmonitor, "mark_fsmonitor_invalid '%s'", ce->name); diff --git a/repository.h b/repository.h index ca837cb9e91..9bbb4659cc8 100644 --- a/repository.h +++ b/repository.h @@ -4,6 +4,7 @@ #include "path.h" struct config_set; +struct fsmonitor_settings; struct git_hash_algo; struct index_state; struct lock_file; @@ -35,6 +36,8 @@ struct repo_settings { int command_requires_full_index; int sparse_index; + struct fsmonitor_settings *fsmonitor; /* lazily loaded */ + int index_version; enum untracked_cache_setting core_untracked_cache; diff --git a/t/README b/t/README index f48e0542cdc..9ffea1d3147 100644 --- a/t/README +++ b/t/README @@ -405,8 +405,8 @@ every 'git commit-graph write', as if the `--changed-paths` option was passed in. GIT_TEST_FSMONITOR=$PWD/t7519/fsmonitor-all exercises the fsmonitor -code path for utilizing a file system monitor to speed up detecting -new or changed files. +code paths for utilizing a (hook based) file system monitor to speed up +detecting new or changed files. GIT_TEST_INDEX_VERSION= exercises the index read/write code path for the index version specified. Can be set to any valid version From patchwork Fri Mar 25 18:02:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791957 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA19AC433EF for ; Fri, 25 Mar 2022 19:32:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230303AbiCYTd5 (ORCPT ); Fri, 25 Mar 2022 15:33:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229838AbiCYTcf (ORCPT ); Fri, 25 Mar 2022 15:32:35 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC72415B050 for ; Fri, 25 Mar 2022 12:07:11 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id h16so4978465wmd.0 for ; Fri, 25 Mar 2022 12:07:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=oGDRFTIGvSWmNGonDgOtutzgzNzlHM+LdHJYE6AYmv0=; b=aDBD1My7da2KppfsTSqA+LV5wm8c02H13qLg6Uxy6oCLHGAxp7u0bXmMVL+KlDama8 INhuOaPbbWyHSvt+ytQgX6IZVIony+/9hVzzORNDDyPW0ZowES7/T8YzPzP6c8i8DfPn cnh312r3/OmiRvBW2M5AeKr5Dy05LQcYjoKGKk3DY5SxS0V5XaO44SCR06JtCU3fpxxI hxaKbfd6MEpXFudZumojW2qOtJl0WEoDHFe6W6f6l/dKGnCJohQ2uQt3TK8FB3ivGGIr TDpEH9Oanc5rFXluct8jfyb2ALEkSIDr2i6ixc0R6nVQXrnYRXgIkE0ooukAqyaO2rPb erRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=oGDRFTIGvSWmNGonDgOtutzgzNzlHM+LdHJYE6AYmv0=; b=Jvk1akyqZQQlSYjFzaklnwvPqPKYyLVOZycPZusxkuoLRGKR/JaldyzHmplEp2xzrj AULjlOdnn+Vpf7qcPRkb6oqNUm6WWM1NfnyoWDSi40CEXDVh0whlgG8rrp2hBgVGy+mH 7bgoKqNtWqRoPAPMW1yN3PpSp3CLOAoa99ogtXAE/WPBdrTQvpqpu/MLIKSh1z34HXMD 4WAn04HEmyK9J7LNmNt2/VrIFJOwuOumIG1c/QivXgoOlO4F4eO7JugkgvcXIb6V3wxc Ca8IHp+EJ2wA5zR418bgeNNCS3eZwLAVYMb7e6VPMeks+XSpTvp1nzwjJH9XRidL7Ka+ OLAw== X-Gm-Message-State: AOAM532IkLtswxcjh4JHCoSVoGl5vRX5d3pRKxajpQZzHg1jopF20W+J g3r64QBt35aTnErdC0nf1oldmvJj7Js= X-Google-Smtp-Source: ABdhPJxaxpUawYuvHrkkONLz5eM+lnYIgKwWjvjLMBdPCmxnMHLKpK7q31t2m72vbWPZIA1ZdSc4Zg== X-Received: by 2002:a7b:c922:0:b0:383:e7e2:4a1a with SMTP id h2-20020a7bc922000000b00383e7e24a1amr11362661wml.51.1648231399656; Fri, 25 Mar 2022 11:03:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p23-20020a1c5457000000b0038c98c12ea9sm4983399wmi.1.2022.03.25.11.03.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:19 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:47 +0000 Subject: [PATCH v9 04/30] fsmonitor: use IPC to query the builtin FSMonitor daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Use simple IPC to directly communicate with the new builtin file system monitor daemon when `core.fsmonitor` is set to true. Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- fsmonitor.c | 38 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 36 insertions(+), 2 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 0e961b74d82..a38b5710eb3 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -241,8 +241,41 @@ void refresh_fsmonitor(struct index_state *istate) trace_printf_key(&trace_fsmonitor, "refresh fsmonitor"); if (fsm_mode == FSMONITOR_MODE_IPC) { - /* TODO */ - return; + query_success = !fsmonitor_ipc__send_query( + istate->fsmonitor_last_update ? + istate->fsmonitor_last_update : "builtin:fake", + &query_result); + if (query_success) { + /* + * The response contains a series of nul terminated + * strings. The first is the new token. + * + * Use `char *buf` as an interlude to trick the CI + * static analysis to let us use `strbuf_addstr()` + * here (and only copy the token) rather than + * `strbuf_addbuf()`. + */ + buf = query_result.buf; + strbuf_addstr(&last_update_token, buf); + bol = last_update_token.len + 1; + is_trivial = query_result.buf[bol] == '/'; + if (is_trivial) + trace2_data_intmax("fsm_client", NULL, + "query/trivial-response", 1); + } else { + /* + * The builtin daemon is not available on this + * platform -OR- we failed to get a response. + * + * Generate a fake token (rather than a V1 + * timestamp) for the index extension. (If + * they switch back to the hook API, we don't + * want ambiguous state.) + */ + strbuf_addstr(&last_update_token, "builtin:fake"); + } + + goto apply_results; } assert(fsm_mode == FSMONITOR_MODE_HOOK); @@ -315,6 +348,7 @@ void refresh_fsmonitor(struct index_state *istate) query_success ? "success" : "failure"); } +apply_results: /* * The response from FSMonitor (excluding the header token) is * either: From patchwork Fri Mar 25 18:02:48 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12792007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 25112C433EF for ; Fri, 25 Mar 2022 19:43:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231547AbiCYTpc (ORCPT ); Fri, 25 Mar 2022 15:45:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40464 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231584AbiCYTot (ORCPT ); Fri, 25 Mar 2022 15:44:49 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A20336BA2C for ; Fri, 25 Mar 2022 12:16:09 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id 123-20020a1c1981000000b0038b3616a71aso4948669wmz.4 for ; Fri, 25 Mar 2022 12:16:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=C/9cpq+/7LFgJqbFs52lTGJrXKzcUu9HQhf3f89AGVk=; b=oqt2QaZKUjP8BWVNQfzKsGQB9qjqOA2aEBZxKy/HULh9LiocsHQ9+I6yI7Qsdi2ykw 4CraPXPioY7yFZzZYTxPakHOYlJF9U+LdStup5FLRjDfBtJnDONH65ovwjtt3NCstHz9 dxlf6zz1wrdWBWOIzZie0Z5K5A8Vm3I7h43LRXoCreymDhcn0qS3N0BsH8UVgRUNG/z3 71CVQS6Tge1NHOLBDQWxZ9BsvGTiwbjkbxua8zbbI7D8+0SkJwPoO7M3t8fl5FMGsY/Z 6Vumlga7BLXreEpYiYs0fqYk7GFe4GbPkgYWXRbwR8AVIvMKuC0nx7scYk+hBD9Yxert iiow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=C/9cpq+/7LFgJqbFs52lTGJrXKzcUu9HQhf3f89AGVk=; b=II0JmW2briqkDps653bUnTCPG5+9JJhq3AJVs3tQD3LB6mr7OwJvGUQdr4H3HxavRQ xldE3t3LbK3p5/PdUHV4apPxojczMcBW4atO8rQKETeY/FMYAY6BbMiU3WAGGAgt7QJl luR73WS+qRZDIp2tc4Wo7xzmSMnz41Wd71Bo+IyMXm/l0ja4vPTa+ib25e+uBABrbDXl zs6pP1g5Ggbq10sUCGu7TyGbSWZiAwjZPMslZrVBvv4Y7FnqBNweTvAgwAVy8QsUX6u+ jBL4lEoLJ7stUfMdvDGvElsc/WcPDxmQFq/L2WGaQoW57vuDnrwwbhxaxFwlzZjeNSGy zO7A== X-Gm-Message-State: AOAM533H00i4jlU4OPM3gwT0VeudeXD3p8p6pbG3/iIL1q3nYr301KOm Xf6lXAXWhetC8UqPLxh34hM3PXvdYxU= X-Google-Smtp-Source: ABdhPJzWWol4t2smtV4PP6qduFWnFN94k9fYYh4Dfs600SdRiXmhA1Si82X/JMHZfsvh2HI5kmPeWw== X-Received: by 2002:a7b:ce83:0:b0:37b:f1f1:3a0c with SMTP id q3-20020a7bce83000000b0037bf1f13a0cmr11156411wmj.10.1648231400682; Fri, 25 Mar 2022 11:03:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n8-20020adf8b08000000b001f046cc8891sm5729259wra.24.2022.03.25.11.03.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:20 -0700 (PDT) Message-Id: <6ab7db9cb769fe44db2c76e82da0b5d898538ec0.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:48 +0000 Subject: [PATCH v9 05/30] fsmonitor: document builtin fsmonitor Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Document how `core.fsmonitor` can be set to a boolean to enable or disable the builtin FSMonitor. Update references to `core.fsmonitor` and `core.fsmonitorHookVersion` and pointers to `Watchman` to refer to it. Create `git-fsmonitor--daemon` manual page and describe its features. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- Documentation/config/core.txt | 60 +++++++++++++++----- Documentation/git-fsmonitor--daemon.txt | 75 +++++++++++++++++++++++++ Documentation/git-update-index.txt | 8 ++- 3 files changed, 126 insertions(+), 17 deletions(-) create mode 100644 Documentation/git-fsmonitor--daemon.txt diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt index c04f62a54a1..6303c36c7ed 100644 --- a/Documentation/config/core.txt +++ b/Documentation/config/core.txt @@ -62,22 +62,54 @@ core.protectNTFS:: Defaults to `true` on Windows, and `false` elsewhere. core.fsmonitor:: - If set, the value of this variable is used as a command which - will identify all files that may have changed since the - requested date/time. This information is used to speed up git by - avoiding unnecessary processing of files that have not changed. - See the "fsmonitor-watchman" section of linkgit:githooks[5]. + If set to true, enable the built-in file system monitor + daemon for this working directory (linkgit:git-fsmonitor--daemon[1]). ++ +Like hook-based file system monitors, the built-in file system monitor +can speed up Git commands that need to refresh the Git index +(e.g. `git status`) in a working directory with many files. The +built-in monitor eliminates the need to install and maintain an +external third-party tool. ++ +The built-in file system monitor is currently available only on a +limited set of supported platforms. Currently, this includes Windows +and MacOS. ++ + Otherwise, this variable contains the pathname of the "fsmonitor" + hook command. ++ +This hook command is used to identify all files that may have changed +since the requested date/time. This information is used to speed up +git by avoiding unnecessary scanning of files that have not changed. ++ +See the "fsmonitor-watchman" section of linkgit:githooks[5]. ++ +Note that if you concurrently use multiple versions of Git, such +as one version on the command line and another version in an IDE +tool, that the definition of `core.fsmonitor` was extended to +allow boolean values in addition to hook pathnames. Git versions +2.35.1 and prior will not understand the boolean values and will +consider the "true" or "false" values as hook pathnames to be +invoked. Git versions 2.26 thru 2.35.1 default to hook protocol +V2 and will fall back to no fsmonitor (full scan). Git versions +prior to 2.26 default to hook protocol V1 and will silently +assume there were no changes to report (no scan), so status +commands may report incomplete results. For this reason, it is +best to upgrade all of your Git versions before using the built-in +file system monitor. core.fsmonitorHookVersion:: - Sets the version of hook that is to be used when calling fsmonitor. - There are currently versions 1 and 2. When this is not set, - version 2 will be tried first and if it fails then version 1 - will be tried. Version 1 uses a timestamp as input to determine - which files have changes since that time but some monitors - like watchman have race conditions when used with a timestamp. - Version 2 uses an opaque string so that the monitor can return - something that can be used to determine what files have changed - without race conditions. + Sets the protocol version to be used when invoking the + "fsmonitor" hook. ++ +There are currently versions 1 and 2. When this is not set, +version 2 will be tried first and if it fails then version 1 +will be tried. Version 1 uses a timestamp as input to determine +which files have changes since that time but some monitors +like Watchman have race conditions when used with a timestamp. +Version 2 uses an opaque string so that the monitor can return +something that can be used to determine what files have changed +without race conditions. core.trustctime:: If false, the ctime differences between the index and the diff --git a/Documentation/git-fsmonitor--daemon.txt b/Documentation/git-fsmonitor--daemon.txt new file mode 100644 index 00000000000..0fedf5a4565 --- /dev/null +++ b/Documentation/git-fsmonitor--daemon.txt @@ -0,0 +1,75 @@ +git-fsmonitor--daemon(1) +======================== + +NAME +---- +git-fsmonitor--daemon - A Built-in File System Monitor + +SYNOPSIS +-------- +[verse] +'git fsmonitor--daemon' start +'git fsmonitor--daemon' run +'git fsmonitor--daemon' stop +'git fsmonitor--daemon' status + +DESCRIPTION +----------- + +A daemon to watch the working directory for file and directory +changes using platform-specific file system notification facilities. + +This daemon communicates directly with commands like `git status` +using the link:technical/api-simple-ipc.html[simple IPC] interface +instead of the slower linkgit:githooks[5] interface. + +This daemon is built into Git so that no third-party tools are +required. + +OPTIONS +------- + +start:: + Starts a daemon in the background. + +run:: + Runs a daemon in the foreground. + +stop:: + Stops the daemon running in the current working + directory, if present. + +status:: + Exits with zero status if a daemon is watching the + current working directory. + +REMARKS +------- + +This daemon is a long running process used to watch a single working +directory and maintain a list of the recently changed files and +directories. Performance of commands such as `git status` can be +increased if they just ask for a summary of changes to the working +directory and can avoid scanning the disk. + +When `core.fsmonitor` is set to `true` (see linkgit:git-config[1]) +commands, such as `git status`, will ask the daemon for changes and +automatically start it (if necessary). + +For more information see the "File System Monitor" section in +linkgit:git-update-index[1]. + +CAVEATS +------- + +The fsmonitor daemon does not currently know about submodules and does +not know to filter out file system events that happen within a +submodule. If fsmonitor daemon is watching a super repo and a file is +modified within the working directory of a submodule, it will report +the change (as happening against the super repo). However, the client +will properly ignore these extra events, so performance may be affected +but it will not cause an incorrect result. + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Documentation/git-update-index.txt b/Documentation/git-update-index.txt index 2853f168d97..53ea48a04e2 100644 --- a/Documentation/git-update-index.txt +++ b/Documentation/git-update-index.txt @@ -498,7 +498,9 @@ FILE SYSTEM MONITOR This feature is intended to speed up git operations for repos that have large working directories. -It enables git to work together with a file system monitor (see the +It enables git to work together with a file system monitor (see +linkgit:git-fsmonitor--daemon[1] +and the "fsmonitor-watchman" section of linkgit:githooks[5]) that can inform it as to what files have been modified. This enables git to avoid having to lstat() every file to find modified files. @@ -509,8 +511,8 @@ looking for new files. If you want to enable (or disable) this feature, it is easier to use the `core.fsmonitor` configuration variable (see -linkgit:git-config[1]) than using the `--fsmonitor` option to -`git update-index` in each repository, especially if you want to do so +linkgit:git-config[1]) than using the `--fsmonitor` option to `git +update-index` in each repository, especially if you want to do so across all repositories you use, because you can set the configuration variable in your `$HOME/.gitconfig` just once and have it affect all repositories you touch. From patchwork Fri Mar 25 18:02:49 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9744C433EF for ; Fri, 25 Mar 2022 19:32:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229976AbiCYTe3 (ORCPT ); Fri, 25 Mar 2022 15:34:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55874 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229977AbiCYTde (ORCPT ); Fri, 25 Mar 2022 15:33:34 -0400 Received: from mail-lf1-x134.google.com (mail-lf1-x134.google.com [IPv6:2a00:1450:4864:20::134]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D66E0228D2D for ; Fri, 25 Mar 2022 12:08:36 -0700 (PDT) Received: by mail-lf1-x134.google.com with SMTP id 5so14938116lfp.1 for ; Fri, 25 Mar 2022 12:08:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=qHwl1k+mT7kL/vCpcTSTxB1s0NWzJtpH1afNGYRMp+8=; b=ga6Gcxxfj/lJznBqbyVFgb81z7t8ZrWzwRBh1qZZqBBtklJapiLxdcpXFfiIpY4H9U 04BwrcZvdFbO4pYkXP/2uwrcInWJlxCX7scIWMcXA40ssCMZSSppVSgH15a91qzk6s64 /CJ+Y1cWml7s79uv/YLnPV+tvNasmOprs3II5B2ndqGu4zZ3GHnTcBmit7q4ZWTzEQr7 Xx8kxOE1wDRYjVTMLPMK5ZDuPMOFnxW2x4nBps//VZ/q1pSdP28AHf1GFQLySG3ogeAZ D3S6OAmpTi+yWJvt1s3ceSJF7PtsrzLSGGODocUanx/SvU3n50KzQuI/XqauMGMHkPvC +zqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=qHwl1k+mT7kL/vCpcTSTxB1s0NWzJtpH1afNGYRMp+8=; b=VJXgsdxkxmbPc+zaaGrpH/3ifY87JvKHK67APDE7CixfiCmY9czo/lOTssL5W5kciJ R8vmg/phI7Z+r1y8Mld4loeKiseBxMleqtqh6GSHWWGb0Bv3XtmAPyk6rADVaicOCQmX JqU56JD7Nt7Oskjnwjxo1f2ToprajD1vo+4Y/BWL/yYptS7fZK7fAPvjkZ5W+pP4A42/ 3UGpGzGquXQczQC7N5ArVFm8cmBpp0SGIO5dXFTA+m36EWxXv87DGzVm6OaMdyzcmpE9 igQgnLs4w7z0hFsFPvi+RbZBcT7Vc3zMMnSFgpAR0HTo6nZEWusP9KFnfwfAKBpVgMPr bTgQ== X-Gm-Message-State: AOAM53216GGAqhOTDTJKvM6K3IonMSRmAqUF3fpz2G3BFu5r8Hs4emrL pPOeGWB4vsUhLrSNYeONPQjuEmojXDo= X-Google-Smtp-Source: ABdhPJwFw9hjKPw+v935NCRsy6mb0oWQ0vu/0rXhs7sFH1ft/hYAgng8W18u2xKhPsmTF/N/64A5Qw== X-Received: by 2002:a05:6000:1888:b0:204:1a8b:2999 with SMTP id a8-20020a056000188800b002041a8b2999mr10288723wri.221.1648231402234; Fri, 25 Mar 2022 11:03:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a11-20020a056000188b00b00204109f7826sm6472903wri.28.2022.03.25.11.03.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:21 -0700 (PDT) Message-Id: <0ce8ae3f2cf07bfffdb1cb14454768d2b6d84bf2.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:49 +0000 Subject: [PATCH v9 06/30] fsmonitor--daemon: add a built-in fsmonitor daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create a built-in file system monitoring daemon that can be used by the existing `fsmonitor` feature (protocol API and index extension) to improve the performance of various Git commands, such as `status`. The `fsmonitor--daemon` feature builds upon the `Simple IPC` API and provides an alternative to hook access to existing fsmonitors such as `watchman`. This commit merely adds the new command without any functionality. Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- .gitignore | 1 + Makefile | 1 + builtin.h | 1 + builtin/fsmonitor--daemon.c | 46 +++++++++++++++++++++++++++++++++++++ git.c | 1 + 5 files changed, 50 insertions(+) create mode 100644 builtin/fsmonitor--daemon.c diff --git a/.gitignore b/.gitignore index f817c509ec0..e81de1063a4 100644 --- a/.gitignore +++ b/.gitignore @@ -72,6 +72,7 @@ /git-format-patch /git-fsck /git-fsck-objects +/git-fsmonitor--daemon /git-gc /git-get-tar-commit-id /git-grep diff --git a/Makefile b/Makefile index 707a56d4c11..5af1d5b112e 100644 --- a/Makefile +++ b/Makefile @@ -1114,6 +1114,7 @@ BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o +BUILTIN_OBJS += builtin/fsmonitor--daemon.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o BUILTIN_OBJS += builtin/grep.o diff --git a/builtin.h b/builtin.h index 83379f3832c..40e9ecc8485 100644 --- a/builtin.h +++ b/builtin.h @@ -159,6 +159,7 @@ int cmd_for_each_ref(int argc, const char **argv, const char *prefix); int cmd_for_each_repo(int argc, const char **argv, const char *prefix); int cmd_format_patch(int argc, const char **argv, const char *prefix); int cmd_fsck(int argc, const char **argv, const char *prefix); +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix); int cmd_gc(int argc, const char **argv, const char *prefix); int cmd_get_tar_commit_id(int argc, const char **argv, const char *prefix); int cmd_grep(int argc, const char **argv, const char *prefix); diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c new file mode 100644 index 00000000000..f0498793379 --- /dev/null +++ b/builtin/fsmonitor--daemon.c @@ -0,0 +1,46 @@ +#include "builtin.h" +#include "config.h" +#include "parse-options.h" +#include "fsmonitor.h" +#include "fsmonitor-ipc.h" +#include "simple-ipc.h" +#include "khash.h" + +static const char * const builtin_fsmonitor__daemon_usage[] = { + NULL +}; + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + const char *subcmd; + + struct option options[] = { + OPT_END() + }; + + git_config(git_default_config, NULL); + + argc = parse_options(argc, argv, prefix, options, + builtin_fsmonitor__daemon_usage, 0); + if (argc != 1) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + subcmd = argv[0]; + + die(_("Unhandled subcommand '%s'"), subcmd); +} + +#else +int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) +{ + struct option options[] = { + OPT_END() + }; + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_fsmonitor__daemon_usage, options); + + die(_("fsmonitor--daemon not supported on this platform")); +} +#endif diff --git a/git.c b/git.c index a25940d72e8..3d8e48cf555 100644 --- a/git.c +++ b/git.c @@ -537,6 +537,7 @@ static struct cmd_struct commands[] = { { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, + { "fsmonitor--daemon", cmd_fsmonitor__daemon, RUN_SETUP }, { "gc", cmd_gc, RUN_SETUP }, { "get-tar-commit-id", cmd_get_tar_commit_id, NO_PARSEOPT }, { "grep", cmd_grep, RUN_SETUP_GENTLY }, From patchwork Fri Mar 25 18:02:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791977 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1E566C433F5 for ; Fri, 25 Mar 2022 19:35:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230193AbiCYThI (ORCPT ); Fri, 25 Mar 2022 15:37:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59488 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230460AbiCYTg7 (ORCPT ); Fri, 25 Mar 2022 15:36:59 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9783E205964 for ; Fri, 25 Mar 2022 12:09:55 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id 123-20020a1c1981000000b0038b3616a71aso4940178wmz.4 for ; Fri, 25 Mar 2022 12:09:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3PLwCteJ9y2IKK4YgyfW1TpJLAFrB0rwJ1mFNfWqUuM=; b=XmMONFA/Qg/+RsMZf8jGLI7IZY+EIvoYTBzkq6ObRE/Bf7CcId3pRX25Estf7YgSTE iiFq9puvjsnX2F2QmEcRpiD8Viqe0sSoYEH8XnrNeVddoHBlKrr8ezuMRNQeKvgICnTj uXmP13Qfw1BHRDQrQKhQDwCdeOiP9eUD1E6+VpgzxHuSzsBLNHiC+9Dl9THJTcxYemWw e8ooG93jFhxGKnpmy0369PDvsEg6KqJYc5hoqtZBRn3+oVQD43wkfz/0u3DhCFv8RK7V uZn/jr4lDOnjXYBDpKQHvZ3AhOTRAVnN3mOZhxT4SK/plHpV+JAjNAO3velrCrsa0/T8 tqzg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3PLwCteJ9y2IKK4YgyfW1TpJLAFrB0rwJ1mFNfWqUuM=; b=pdPBL1iKAnPVnZpHiJa6lbxMsCKykrmjMoT5GTNoct2C7pKxs1cvTIM9j7qXnjZHzd 5epGOiIzqVfSMyivVYSEaC1VUyIGkvdFIDO7Ded7s6IhKYBKwhAGo1Or2sPKVRQT45cS rtn+Wdrsyc8L7tlZpCO/lRoBEHWeaKwhyxbGgBoJH5cuZ6w2PLEeqR43nz/ni80DW4uu d5om9uRfWoQCYb4lCnzhoAazHveOd18DtfRaGXYKGdnPYDXAliIOVLnEBKaaMgMd31yI 6nVR6XQLgdBtvNCUVxBXffkcHMpSgP6SuWUzlixXcx0apuwitoJPdybZcdWdr4TqTkO4 uz6g== X-Gm-Message-State: AOAM531uPYZAnEcqqOsVniKN1zkuPVMTQPsUFIK9pHYdv5kas7mLKcN3 Dp4D1vAmw933P+pvl8Ud7Mm3zipLC1Y= X-Google-Smtp-Source: ABdhPJw6AOZ3SkVIUR+TnZVRj90Dt12qetkSALYHFZNI7q9iZX7M2SS4V7I2eJHccdltUB3uzodDmA== X-Received: by 2002:a7b:c841:0:b0:389:864c:e715 with SMTP id c1-20020a7bc841000000b00389864ce715mr11047079wml.72.1648231403187; Fri, 25 Mar 2022 11:03:23 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m13-20020a05600c3b0d00b0038cda9f92fasm3833846wms.0.2022.03.25.11.03.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:22 -0700 (PDT) Message-Id: <4624ce2fa471b3ebb3a08baf0de2609447eb9fe3.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:50 +0000 Subject: [PATCH v9 07/30] fsmonitor--daemon: implement 'stop' and 'status' commands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement `stop` and `status` client commands to control and query the status of a `fsmonitor--daemon` server process (and implicitly start a server process if necessary). Later commits will implement the actual server and monitor the file system. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 51 +++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index f0498793379..5e3178b8bdd 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,10 +7,55 @@ #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon stop"), + N_("git fsmonitor--daemon status"), NULL }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Acting as a CLIENT. + * + * Send a "quit" command to the `git-fsmonitor--daemon` (if running) + * and wait for it to shutdown. + */ +static int do_as_client__send_stop(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("quit", &answer); + + /* The quit command does not return any response data. */ + strbuf_release(&answer); + + if (ret) + return ret; + + trace2_region_enter("fsm_client", "polling-for-daemon-exit", NULL); + while (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + sleep_millisec(50); + trace2_region_leave("fsm_client", "polling-for-daemon-exit", NULL); + + return 0; +} + +static int do_as_client__status(void) +{ + enum ipc_active_state state = fsmonitor_ipc__get_state(); + + switch (state) { + case IPC_STATE__LISTENING: + printf(_("fsmonitor-daemon is watching '%s'\n"), + the_repository->worktree); + return 0; + + default: + printf(_("fsmonitor-daemon is not watching '%s'\n"), + the_repository->worktree); + return 1; + } +} int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { @@ -28,6 +73,12 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) usage_with_options(builtin_fsmonitor__daemon_usage, options); subcmd = argv[0]; + if (!strcmp(subcmd, "stop")) + return !!do_as_client__send_stop(); + + if (!strcmp(subcmd, "status")) + return !!do_as_client__status(); + die(_("Unhandled subcommand '%s'"), subcmd); } From patchwork Fri Mar 25 18:02:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791996 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2784C433F5 for ; Fri, 25 Mar 2022 19:39:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231259AbiCYTlD (ORCPT ); Fri, 25 Mar 2022 15:41:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42768 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232599AbiCYTjC (ORCPT ); Fri, 25 Mar 2022 15:39:02 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4812B14FFDB for ; Fri, 25 Mar 2022 12:11:34 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id p12-20020a05600c430c00b0038cbdf52227so4960191wme.2 for ; Fri, 25 Mar 2022 12:11:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sSAmNlfeMp/SuWaNgnRs8TTLxikird+cAhYFRFbfq6s=; b=KaSWzqtzXXm80JGITBxzg25jHC2Hmb3WKEWzMgIhF7R+HVzeY+ZZFyfW/SEWrIyTRd t8/dtFEczDwmMZdd5KJn/G8Tl3OQV3F+5R3hUlm8kLyO19K8ZLFslFsdSF0hzMlDptKS qaOAWs3eAiuia6yHpH9pS21KYvv7wYfOE29hyteupb+l0RBbzl7ti6Zp3EVCMUxglCtH Q7WX6QZ6kJ6YlfnOP+3lIYduh0Vk1jw4glbqVrb9rdxHxmdjTjmujGw60BFj+PM0kNbS LCEyliAnCL7Wa1jdzzlPL4SxEbtdbgR8/u+7w3Rtgqg1Fr+FB9qogqu9Ts/JagOYtXhK ZMsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sSAmNlfeMp/SuWaNgnRs8TTLxikird+cAhYFRFbfq6s=; b=R7Tdq+92Bh/44f8Af276fz5sVNw4I9Mt8kEo+cv1YLcgMVUT0gbayYwGX0kHIbsThF 1zPHUfhGIkXF3QiBUuwIrK/AQv4VL/Gpm63qFEgj4227Raz0dENKTiyvZZQBjhS68CuH fZQsb1IZQBhD7tGL43+BgyoA54+P6GDgMl74rqZVcDBBGo56xzeP5G4V2NBTh38h/MdT xbUphrr50kdW4U8mW/2tB7dAjAe+VKDHIgXu+yQbCJF130sVsReHSBzZTP+g+iyaEfr0 i9ps/EHCty4/OkiCPomx4Lz3J7N6QrCdWs3smPvEgy//7Kj6Co21fCs9Q7MCcEbeZ4Do 1zJQ== X-Gm-Message-State: AOAM532ejPeMmP6eo7jBARt2FR4xVPlTxl62X+eHhtNGQ3tbyCz3r9lP Cy14q5VpWpC7JhYjIqeQq2rQrbbyQDE= X-Google-Smtp-Source: ABdhPJzvSC++BrE4GUimKBI0rqEHrbfcQptNCvd1MCvQl5coMMR9y1huJ3lPRAAp/J6naPNgqi/VzA== X-Received: by 2002:a05:600c:2e4c:b0:38c:7088:afd0 with SMTP id q12-20020a05600c2e4c00b0038c7088afd0mr20818506wmf.70.1648231404121; Fri, 25 Mar 2022 11:03:24 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m10-20020adfe94a000000b002059e530da1sm2998193wrn.1.2022.03.25.11.03.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:23 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:51 +0000 Subject: [PATCH v9 08/30] compat/fsmonitor/fsm-listen-win32: stub in backend for Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty filesystem listener backend for fsmonitor--daemon on Windows. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- Makefile | 13 ++++++++ compat/fsmonitor/fsm-listen-win32.c | 21 +++++++++++++ compat/fsmonitor/fsm-listen.h | 49 +++++++++++++++++++++++++++++ config.mak.uname | 10 ++++++ contrib/buildsystems/CMakeLists.txt | 7 +++++ repo-settings.c | 1 + 6 files changed, 101 insertions(+) create mode 100644 compat/fsmonitor/fsm-listen-win32.c create mode 100644 compat/fsmonitor/fsm-listen.h diff --git a/Makefile b/Makefile index 5af1d5b112e..26567d4f772 100644 --- a/Makefile +++ b/Makefile @@ -470,6 +470,11 @@ all:: # directory, and the JSON compilation database 'compile_commands.json' will be # created at the root of the repository. # +# If your platform supports a built-in fsmonitor backend, set +# FSMONITOR_DAEMON_BACKEND to the "" of the corresponding +# `compat/fsmonitor/fsm-listen-.c` that implements the +# `fsm_listen__*()` routines. +# # Define DEVELOPER to enable more compiler warnings. Compiler version # and family are auto detected, but could be overridden by defining # COMPILER_FEATURES (see config.mak.dev). You can still set @@ -1968,6 +1973,11 @@ ifdef NEED_ACCESS_ROOT_HANDLER COMPAT_OBJS += compat/access.o endif +ifdef FSMONITOR_DAEMON_BACKEND + COMPAT_CFLAGS += -DHAVE_FSMONITOR_DAEMON_BACKEND + COMPAT_OBJS += compat/fsmonitor/fsm-listen-$(FSMONITOR_DAEMON_BACKEND).o +endif + ifeq ($(TCLTK_PATH),) NO_TCLTK = NoThanks endif @@ -2887,6 +2897,9 @@ GIT-BUILD-OPTIONS: FORCE @echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+ @echo SANITIZE_LEAK=\''$(subst ','\'',$(subst ','\'',$(SANITIZE_LEAK)))'\' >>$@+ @echo X=\'$(X)\' >>$@+ +ifdef FSMONITOR_DAEMON_BACKEND + @echo FSMONITOR_DAEMON_BACKEND=\''$(subst ','\'',$(subst ','\'',$(FSMONITOR_DAEMON_BACKEND)))'\' >>$@+ +endif ifdef TEST_OUTPUT_DIRECTORY @echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+ endif diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c new file mode 100644 index 00000000000..916cbea254f --- /dev/null +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -0,0 +1,21 @@ +#include "cache.h" +#include "config.h" +#include "fsmonitor.h" +#include "fsm-listen.h" + +void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsm_listen__loop(struct fsmonitor_daemon_state *state) +{ +} + +int fsm_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsm_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} diff --git a/compat/fsmonitor/fsm-listen.h b/compat/fsmonitor/fsm-listen.h new file mode 100644 index 00000000000..f0539349baf --- /dev/null +++ b/compat/fsmonitor/fsm-listen.h @@ -0,0 +1,49 @@ +#ifndef FSM_LISTEN_H +#define FSM_LISTEN_H + +/* This needs to be implemented by each backend */ + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +struct fsmonitor_daemon_state; + +/* + * Initialize platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread PRIOR to staring the + * fsmonitor_fs_listener thread. + * + * Returns 0 if successful. + * Returns -1 otherwise. + */ +int fsm_listen__ctor(struct fsmonitor_daemon_state *state); + +/* + * Cleanup platform-specific data for the fsmonitor listener thread. + * This will be called from the main thread AFTER joining the listener. + */ +void fsm_listen__dtor(struct fsmonitor_daemon_state *state); + +/* + * The main body of the platform-specific event loop to watch for + * filesystem events. This will run in the fsmonitor_fs_listen thread. + * + * It should call `ipc_server_stop_async()` if the listener thread + * prematurely terminates (because of a filesystem error or if it + * detects that the .git directory has been deleted). (It should NOT + * do so if the listener thread receives a normal shutdown signal from + * the IPC layer.) + * + * It should set `state->error_code` to -1 if the daemon should exit + * with an error. + */ +void fsm_listen__loop(struct fsmonitor_daemon_state *state); + +/* + * Gently request that the fsmonitor listener thread shutdown. + * It does not wait for it to stop. The caller should do a JOIN + * to wait for it. + */ +void fsm_listen__stop_async(struct fsmonitor_daemon_state *state); + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSM_LISTEN_H */ diff --git a/config.mak.uname b/config.mak.uname index 4352ea39e9b..26074f56bed 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -435,6 +435,11 @@ ifeq ($(uname_S),Windows) # so we don't need this: # # SNPRINTF_RETURNS_BOGUS = YesPlease + + # The builtin FSMonitor requires Named Pipes and Threads on Windows. + # These are always available, so we do not have to conditionally + # support it. + FSMONITOR_DAEMON_BACKEND = win32 NO_SVN_TESTS = YesPlease RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo @@ -619,6 +624,11 @@ ifeq ($(uname_S),MINGW) NO_STRTOUMAX = YesPlease NO_MKDTEMP = YesPlease NO_SVN_TESTS = YesPlease + + # The builtin FSMonitor requires Named Pipes and Threads on Windows. + # These are always available, so we do not have to conditionally + # support it. + FSMONITOR_DAEMON_BACKEND = win32 RUNTIME_PREFIX = YesPlease HAVE_WPGMPTR = YesWeDo NO_ST_BLOCKS_IN_STRUCT_STAT = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index e44232f85d3..0963629db7f 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -285,6 +285,13 @@ else() endif() endif() +if(SUPPORTS_SIMPLE_IPC) + if(CMAKE_SYSTEM_NAME STREQUAL "Windows") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + endif() +endif() + set(EXE_EXTENSION ${CMAKE_EXECUTABLE_SUFFIX}) #header checks diff --git a/repo-settings.c b/repo-settings.c index b4fbd16cdcc..2dfcb2b6542 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -2,6 +2,7 @@ #include "config.h" #include "repository.h" #include "midx.h" +#include "compat/fsmonitor/fsm-listen.h" static void repo_cfg_bool(struct repository *r, const char *key, int *dest, int def) From patchwork Fri Mar 25 18:02:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21CB2C433F5 for ; Fri, 25 Mar 2022 19:32:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230056AbiCYTeD (ORCPT ); Fri, 25 Mar 2022 15:34:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54580 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231722AbiCYTdW (ORCPT ); Fri, 25 Mar 2022 15:33:22 -0400 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A21C4199532 for ; Fri, 25 Mar 2022 12:08:16 -0700 (PDT) Received: by mail-wm1-x335.google.com with SMTP id q20so4976949wmq.1 for ; Fri, 25 Mar 2022 12:08:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JoyUcWYc53GOiCIk4ZkiWSvvHTSDt24PFwNxe1AnEQQ=; b=a+oVr66FK/2EZrWnjOdpNbe4cugLjbu7MslzyO6DxNXWvclfXh2zXTE91LGQGBQGwg THNN/QJe9xGToPqkU/UNmih+Hole0MvGqFsGoMrrvNsFa/1pOJPqIah8xM2q4T/aTwiz eBbyxPPNIzkKEth98YQSbT34XswvirmhDyKaU24DwvXeVYm65meEQuDTIoKjyHRAhg8h PciHT8BsQUeV2xjUw6Ytg/N4sf89nA+ZhvHQwinvuD7zUF542rFrlMR23Ra4zugjG2r7 MqJVqw0UvNkiVDCwNn2KilObdjLiya9UT8omBwMBgFQkD72ITPOvGn0zuDqhOy8COsIE TFUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JoyUcWYc53GOiCIk4ZkiWSvvHTSDt24PFwNxe1AnEQQ=; b=Oszt/xUrN9KosE/jLOJT47JW5ptUcqMCu5a0y5zEJc8bIWozrsYV1u2gyfSzlH5x6t +eZ35TwdZr4PH5HHq5QkyEr+TLDISPJib0Y4GkB0v2/D3nqFfz1wGtfZuZbTk63Zjqsl kj5qjCqgh3Po6DkDZ3jJn3pamAa96+MaCEkSlPHzyLxT0lN+AgLVK5hNCBkuA6INzfFE ys+v0EAP+Y8XVHoYU55+mjWPPZ9sPKnYpAW7iy6WK+gdnGG47HtpxGvBAHiIrsEy1Lab gA+yWIpM2JUyu4OaA+ZzHT/qJD4CYqyprYoeoBYflnxfCpjcZH1o3FIgT3QK3NErA/j1 /kpg== X-Gm-Message-State: AOAM530JPUH7iqwyJrhpn+dM/snoD1CqsHb4iDDkKubbCHyebrNMu2Mz wq9FiBc974MMcA4KM8VMWBAr/XIPW7Y= X-Google-Smtp-Source: ABdhPJzwrH7lH6upRV1rXL68wjFMqse0jBButbnfscWzr/qeh/NY74vDtgkodEGusuJRX+los3A9hQ== X-Received: by 2002:a05:600c:3b02:b0:38c:9148:70dc with SMTP id m2-20020a05600c3b0200b0038c914870dcmr11123419wms.104.1648231405106; Fri, 25 Mar 2022 11:03:25 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 20-20020a05600c22d400b0038c8dbdc1a3sm4839917wmg.38.2022.03.25.11.03.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:24 -0700 (PDT) Message-Id: <2f8a42fdb93623053256939fdbb45cf02fb1c5c9.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:52 +0000 Subject: [PATCH v9 09/30] compat/fsmonitor/fsm-listen-darwin: stub in backend for Darwin Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Stub in empty implementation of fsmonitor--daemon backend for Darwin (aka MacOS). Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- compat/fsmonitor/fsm-listen-darwin.c | 20 ++++++++++++++++++++ config.mak.uname | 10 ++++++++++ contrib/buildsystems/CMakeLists.txt | 3 +++ 3 files changed, 33 insertions(+) create mode 100644 compat/fsmonitor/fsm-listen-darwin.c diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c new file mode 100644 index 00000000000..c84e3344ab9 --- /dev/null +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -0,0 +1,20 @@ +#include "cache.h" +#include "fsmonitor.h" +#include "fsm-listen.h" + +int fsm_listen__ctor(struct fsmonitor_daemon_state *state) +{ + return -1; +} + +void fsm_listen__dtor(struct fsmonitor_daemon_state *state) +{ +} + +void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) +{ +} + +void fsm_listen__loop(struct fsmonitor_daemon_state *state) +{ +} diff --git a/config.mak.uname b/config.mak.uname index 26074f56bed..501970902da 100644 --- a/config.mak.uname +++ b/config.mak.uname @@ -157,6 +157,16 @@ ifeq ($(uname_S),Darwin) MSGFMT = /usr/local/opt/gettext/bin/msgfmt endif endif + + # The builtin FSMonitor on MacOS builds upon Simple-IPC. Both require + # Unix domain sockets and PThreads. + ifndef NO_PTHREADS + ifndef NO_UNIX_SOCKETS + FSMONITOR_DAEMON_BACKEND = darwin + endif + endif + + BASIC_LDFLAGS += -framework CoreServices endif ifeq ($(uname_S),SunOS) NEEDS_SOCKET = YesPlease diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt index 0963629db7f..ee0d7257b77 100644 --- a/contrib/buildsystems/CMakeLists.txt +++ b/contrib/buildsystems/CMakeLists.txt @@ -289,6 +289,9 @@ if(SUPPORTS_SIMPLE_IPC) if(CMAKE_SYSTEM_NAME STREQUAL "Windows") add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-win32.c) + elseif(CMAKE_SYSTEM_NAME STREQUAL "Darwin") + add_compile_definitions(HAVE_FSMONITOR_DAEMON_BACKEND) + list(APPEND compat_SOURCES compat/fsmonitor/fsm-listen-darwin.c) endif() endif() From patchwork Fri Mar 25 18:02:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791998 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BC0BC433F5 for ; Fri, 25 Mar 2022 19:39:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231330AbiCYTlH (ORCPT ); Fri, 25 Mar 2022 15:41:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41662 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231883AbiCYTi0 (ORCPT ); Fri, 25 Mar 2022 15:38:26 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7EF3B241B7C for ; Fri, 25 Mar 2022 12:20:42 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id h16so4994667wmd.0 for ; Fri, 25 Mar 2022 12:20:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=27pKE5W50Xq6AWPWel8FWJuFD8Gh8CWrIgNTdeQM9Z0=; b=SG53KU8vKCaiMNsFuJxsnrHiWaJOKCTuymWUv6rpqqSF2jJwnmlmw0lQI+yQB5sR/6 Xozd6dgg5ZETmJi2t2VO3yY8NyYZFYrniSC8ipDk33rMTHCQ6jQ8Q+h/WZfElKXLGv8R 45pAUDiMeeW7D+wyXrnJnY3L7/yk6m8+oPtPh3F2nojGgaLcn3FCgdz3+PcxC3WDHDji 0xpqjLsyk2orjVRKBSh5qFSig9YYQgj1G4SAEmBOeDrQhucbNPxtWvo4WlwU4EkkCs4N w9I9ComvIkEQEJOfETXx05GBF4HDTpgGyvbRBHtjmVx2aOYlojTJncPHDVu/r8B/5Azp ZgJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=27pKE5W50Xq6AWPWel8FWJuFD8Gh8CWrIgNTdeQM9Z0=; b=DB/Be1bKwMM3fSVUSygIubzh9/Vy7psFuOESBSVCz9OmJqwZK0qxe8qw1XHcUIPs78 zNsNs6N8yB34MOMGCOvCXzZUIn5dw9mloE0k809Yp1N7AxKl/ucCCZ4pyUS1YXSvNfj7 wAns/FZCFKmxRXjGzB76nmHtvKpVaPDEcf94Att2miu/vR1+T9cE4UCuVF26SznDw0s0 xNjKb0F63a+AqmhD0YQ+sB2aSFUivNM6nZPxkxiUfSwTOotFq5cDEfTO/MKn+zKOyKsC kViymFVfQ250/hzJS2qMM0MlnuaGD5ApFiU74vGvBZYHQDtQ3EHrgiKKa6Ya/Or3uJoi ePMA== X-Gm-Message-State: AOAM532AmGKhsqSPWxA4NT+nIvE+yMsPq1uWJZxHZe8p1fxGtLeA8nk4 m1glOXta6WZ7Cvc5/Np7Uq5pLPc/NuQ= X-Google-Smtp-Source: ABdhPJziJRhzw7Ef772db0f4nyqTZLs2x9QGoXUIEp1FK5aCz4VpuklSvfQTGl8DTzrIts82BcqcgA== X-Received: by 2002:a1c:e908:0:b0:38c:782c:2a62 with SMTP id q8-20020a1ce908000000b0038c782c2a62mr20314407wmc.135.1648231406311; Fri, 25 Mar 2022 11:03:26 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e9-20020a5d6d09000000b00203ecdca5b7sm6251033wrq.33.2022.03.25.11.03.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:25 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:53 +0000 Subject: [PATCH v9 10/30] fsmonitor--daemon: implement 'run' command Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement `run` command to try to begin listening for file system events. This version defines the thread structure with a single fsmonitor_fs_listen thread to watch for file system events and a simple IPC thread pool to watch for connection from Git clients over a well-known named pipe or Unix domain socket. This commit does not actually do anything yet because the platform backends are still just stubs. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 228 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 34 ++++++ 2 files changed, 261 insertions(+), 1 deletion(-) create mode 100644 fsmonitor--daemon.h diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 5e3178b8bdd..5591339399a 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -3,16 +3,52 @@ #include "parse-options.h" #include "fsmonitor.h" #include "fsmonitor-ipc.h" +#include "compat/fsmonitor/fsm-listen.h" +#include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon run []"), N_("git fsmonitor--daemon stop"), N_("git fsmonitor--daemon status"), NULL }; #ifdef HAVE_FSMONITOR_DAEMON_BACKEND +/* + * Global state loaded from config. + */ +#define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" +static int fsmonitor__ipc_threads = 8; + +#define FSMONITOR__ANNOUNCE_STARTUP "fsmonitor.announcestartup" +static int fsmonitor__announce_startup = 0; + +static int fsmonitor_config(const char *var, const char *value, void *cb) +{ + if (!strcmp(var, FSMONITOR__IPC_THREADS)) { + int i = git_config_int(var, value); + if (i < 1) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__IPC_THREADS, i); + fsmonitor__ipc_threads = i; + return 0; + } + + if (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) { + int is_bool; + int i = git_config_bool_or_int(var, value, &is_bool); + if (i < 0) + return error(_("value of '%s' not bool or int: %d"), + var, i); + fsmonitor__announce_startup = i; + return 0; + } + + return git_default_config(var, value, cb); +} + /* * Acting as a CLIENT. * @@ -57,15 +93,198 @@ static int do_as_client__status(void) } } +static ipc_server_application_cb handle_client; + +static int handle_client(void *data, + const char *command, size_t command_len, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + /* struct fsmonitor_daemon_state *state = data; */ + int result; + + /* + * The Simple IPC API now supports {char*, len} arguments, but + * FSMonitor always uses proper null-terminated strings, so + * we can ignore the command_len argument. (Trust, but verify.) + */ + if (command_len != strlen(command)) + BUG("FSMonitor assumes text messages"); + + trace2_region_enter("fsmonitor", "handle_client", the_repository); + trace2_data_string("fsmonitor", the_repository, "request", command); + + result = 0; /* TODO Do something here. */ + + trace2_region_leave("fsmonitor", "handle_client", the_repository); + + return result; +} + +static void *fsm_listen__thread_proc(void *_state) +{ + struct fsmonitor_daemon_state *state = _state; + + trace2_thread_start("fsm-listen"); + + trace_printf_key(&trace_fsmonitor, "Watching: worktree '%s'", + state->path_worktree_watch.buf); + if (state->nr_paths_watching > 1) + trace_printf_key(&trace_fsmonitor, "Watching: gitdir '%s'", + state->path_gitdir_watch.buf); + + fsm_listen__loop(state); + + trace2_thread_exit(); + return NULL; +} + +static int fsmonitor_run_daemon_1(struct fsmonitor_daemon_state *state) +{ + struct ipc_server_opts ipc_opts = { + .nr_threads = fsmonitor__ipc_threads, + + /* + * We know that there are no other active threads yet, + * so we can let the IPC layer temporarily chdir() if + * it needs to when creating the server side of the + * Unix domain socket. + */ + .uds_disallow_chdir = 0 + }; + + /* + * Start the IPC thread pool before the we've started the file + * system event listener thread so that we have the IPC handle + * before we need it. + */ + if (ipc_server_run_async(&state->ipc_server_data, + fsmonitor_ipc__get_path(), &ipc_opts, + handle_client, state)) + return error_errno( + _("could not start IPC thread pool on '%s'"), + fsmonitor_ipc__get_path()); + + /* + * Start the fsmonitor listener thread to collect filesystem + * events. + */ + if (pthread_create(&state->listener_thread, NULL, + fsm_listen__thread_proc, state) < 0) { + ipc_server_stop_async(state->ipc_server_data); + ipc_server_await(state->ipc_server_data); + + return error(_("could not start fsmonitor listener thread")); + } + + /* + * The daemon is now fully functional in background threads. + * Wait for the IPC thread pool to shutdown (whether by client + * request or from filesystem activity). + */ + ipc_server_await(state->ipc_server_data); + + /* + * The fsmonitor listener thread may have received a shutdown + * event from the IPC thread pool, but it doesn't hurt to tell + * it again. And wait for it to shutdown. + */ + fsm_listen__stop_async(state); + pthread_join(state->listener_thread, NULL); + + return state->error_code; +} + +static int fsmonitor_run_daemon(void) +{ + struct fsmonitor_daemon_state state; + int err; + + memset(&state, 0, sizeof(state)); + + pthread_mutex_init(&state.main_lock, NULL); + state.error_code = 0; + state.current_token_data = NULL; + + /* Prepare to (recursively) watch the directory. */ + strbuf_init(&state.path_worktree_watch, 0); + strbuf_addstr(&state.path_worktree_watch, absolute_path(get_git_work_tree())); + state.nr_paths_watching = 1; + + /* + * We create and delete cookie files somewhere inside the .git + * directory to help us keep sync with the file system. If + * ".git" is not a directory, then is not inside the + * cone of , so set up a second watch to watch + * the so that we get events for the cookie files. + */ + strbuf_init(&state.path_gitdir_watch, 0); + strbuf_addbuf(&state.path_gitdir_watch, &state.path_worktree_watch); + strbuf_addstr(&state.path_gitdir_watch, "/.git"); + if (!is_directory(state.path_gitdir_watch.buf)) { + strbuf_reset(&state.path_gitdir_watch); + strbuf_addstr(&state.path_gitdir_watch, absolute_path(get_git_dir())); + state.nr_paths_watching = 2; + } + + /* + * Confirm that we can create platform-specific resources for the + * filesystem listener before we bother starting all the threads. + */ + if (fsm_listen__ctor(&state)) { + err = error(_("could not initialize listener thread")); + goto done; + } + + err = fsmonitor_run_daemon_1(&state); + +done: + pthread_mutex_destroy(&state.main_lock); + fsm_listen__dtor(&state); + + ipc_server_free(state.ipc_server_data); + + strbuf_release(&state.path_worktree_watch); + strbuf_release(&state.path_gitdir_watch); + + return err; +} + +static int try_to_run_foreground_daemon(void) +{ + /* + * Technically, we don't need to probe for an existing daemon + * process, since we could just call `fsmonitor_run_daemon()` + * and let it fail if the pipe/socket is busy. + * + * However, this method gives us a nicer error message for a + * common error case. + */ + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + die(_("fsmonitor--daemon is already running '%s'"), + the_repository->worktree); + + if (fsmonitor__announce_startup) { + fprintf(stderr, _("running fsmonitor-daemon in '%s'\n"), + the_repository->worktree); + fflush(stderr); + } + + return !!fsmonitor_run_daemon(); +} + int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { const char *subcmd; struct option options[] = { + OPT_INTEGER(0, "ipc-threads", + &fsmonitor__ipc_threads, + N_("use ipc worker threads")), OPT_END() }; - git_config(git_default_config, NULL); + git_config(fsmonitor_config, NULL); argc = parse_options(argc, argv, prefix, options, builtin_fsmonitor__daemon_usage, 0); @@ -73,6 +292,13 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) usage_with_options(builtin_fsmonitor__daemon_usage, options); subcmd = argv[0]; + if (fsmonitor__ipc_threads < 1) + die(_("invalid 'ipc-threads' value (%d)"), + fsmonitor__ipc_threads); + + if (!strcmp(subcmd, "run")) + return !!try_to_run_foreground_daemon(); + if (!strcmp(subcmd, "stop")) return !!do_as_client__send_stop(); diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h new file mode 100644 index 00000000000..3009c1a83de --- /dev/null +++ b/fsmonitor--daemon.h @@ -0,0 +1,34 @@ +#ifndef FSMONITOR_DAEMON_H +#define FSMONITOR_DAEMON_H + +#ifdef HAVE_FSMONITOR_DAEMON_BACKEND + +#include "cache.h" +#include "dir.h" +#include "run-command.h" +#include "simple-ipc.h" +#include "thread-utils.h" + +struct fsmonitor_batch; +struct fsmonitor_token_data; + +struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ + +struct fsmonitor_daemon_state { + pthread_t listener_thread; + pthread_mutex_t main_lock; + + struct strbuf path_worktree_watch; + struct strbuf path_gitdir_watch; + int nr_paths_watching; + + struct fsmonitor_token_data *current_token_data; + + int error_code; + struct fsmonitor_daemon_backend_data *backend_data; + + struct ipc_server_data *ipc_server_data; +}; + +#endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ +#endif /* FSMONITOR_DAEMON_H */ From patchwork Fri Mar 25 18:02:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791984 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD28EC43217 for ; Fri, 25 Mar 2022 19:38:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230376AbiCYTk2 (ORCPT ); Fri, 25 Mar 2022 15:40:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46160 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231143AbiCYThq (ORCPT ); Fri, 25 Mar 2022 15:37:46 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 83B121C551E for ; Fri, 25 Mar 2022 12:10:21 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id h16so4982370wmd.0 for ; Fri, 25 Mar 2022 12:10:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=TgVyyzQx+jhjmRIkDS0iZCqgyqnaNEg2KAffy6/1bAM=; b=ncIOj08jymICF19/X1KKE16oFI6hsESDIVKVr9fnI98M+mJzD9SHo6PESXz8oy7dHR sbeWfwRbYZhdRjLSXXLvz8u3aMb/6M/nTuPrIL16i+Ejv26sVsHv3olHYgkGqrPkOvvO +4fhZG3aLyg5wSqhs8jFTSPzWbuYJiVbqkTIbTPz7AmRNSGvWS/P9klCViDD8klyTN32 yZeJnxgOxe6A7S0bkZIid0DCxjEcu53SesNF5QIL8fEYK2cFCo3uQ5L/gWfU3Ay5S5X2 PjC8ziTwUW0cPgxL1mRUGpjsgcrS4GF9OdfioWIcvnHB0otVSOPt3PpNdgBqydLOOFL7 xcrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=TgVyyzQx+jhjmRIkDS0iZCqgyqnaNEg2KAffy6/1bAM=; b=l6UB9z4i8wgUARZmE+zT+jC5m5rArMHJsSfrDidmQNm7WGRAHfYSPJldgwpYxQzSZ/ hPqey2K9kxlYYBbPU/0GKzeOnRAln/V5wd+5+/1TzyyLpUmkqS1XuGGH+UpCeM8gO0+m tWW908Iiqv8y4J9Arb0AWFvOx8S2du/vAgyYJMXzmVP6yn9K9x1suDLsttFwFFAF20ja ACRchK84PS1Z2Jkr94ZrgkBXMyVDXGjKPXsG354xf/u8OYMox5zYB6DWlI0Ef3SGAYGf p68MMaaEYfKqPxpVQuwbCbMxnWo2wBAl2Z3OBlvnsBe677J5us/ofpcB0PxFY7ZGq9xK 5JwA== X-Gm-Message-State: AOAM53365Vm/01i+/OCcqtd64zQjM5NJNsQSgFE9ZY/Wgl1Wu6Obwp0S 62bOrH4U7IQhzFtZTVqpoX0VsQRuQHM= X-Google-Smtp-Source: ABdhPJwQKHiwrasUaZiQlbl/A1omOqUo5eh6q9agEECy89nsiXCb5u+8FFjxsS77eQ5nBtc1MUQ/Wg== X-Received: by 2002:a05:600c:252:b0:38c:cd19:fbff with SMTP id 18-20020a05600c025200b0038ccd19fbffmr10264062wmj.65.1648231407305; Fri, 25 Mar 2022 11:03:27 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n124-20020a1ca482000000b0038c9cf6e296sm7255949wme.14.2022.03.25.11.03.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:26 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:54 +0000 Subject: [PATCH v9 11/30] fsmonitor--daemon: implement 'start' command Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement 'git fsmonitor--daemon start' command. This command starts an instance of 'git fsmonitor--daemon run' in the background using the new 'start_bg_command()' function. We avoid the fork-and-call technique on Unix systems in favor of a fork-and-exec technique. This gives us more uniform Trace2 child-* events. It also makes our usage more consistent with Windows usage. On Windows, teach 'git fsmonitor--daemon run' to optionally call 'FreeConsole()' to release handles to the inherited Win32 console (despite being passed invalid handles for stdin/out/err). Without this, command prompts and powershell terminal windows could hang in "exit" until the last background child process exited or released their Win32 console handle. (This was not seen with git-bash shells because they don't have a Win32 console attached to them.) Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 109 +++++++++++++++++++++++++++++++++++- 1 file changed, 107 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 5591339399a..69dd39121a3 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -9,6 +9,7 @@ #include "khash.h" static const char * const builtin_fsmonitor__daemon_usage[] = { + N_("git fsmonitor--daemon start []"), N_("git fsmonitor--daemon run []"), N_("git fsmonitor--daemon stop"), N_("git fsmonitor--daemon status"), @@ -22,6 +23,9 @@ static const char * const builtin_fsmonitor__daemon_usage[] = { #define FSMONITOR__IPC_THREADS "fsmonitor.ipcthreads" static int fsmonitor__ipc_threads = 8; +#define FSMONITOR__START_TIMEOUT "fsmonitor.starttimeout" +static int fsmonitor__start_timeout_sec = 60; + #define FSMONITOR__ANNOUNCE_STARTUP "fsmonitor.announcestartup" static int fsmonitor__announce_startup = 0; @@ -36,6 +40,15 @@ static int fsmonitor_config(const char *var, const char *value, void *cb) return 0; } + if (!strcmp(var, FSMONITOR__START_TIMEOUT)) { + int i = git_config_int(var, value); + if (i < 0) + return error(_("value of '%s' out of range: %d"), + FSMONITOR__START_TIMEOUT, i); + fsmonitor__start_timeout_sec = i; + return 0; + } + if (!strcmp(var, FSMONITOR__ANNOUNCE_STARTUP)) { int is_bool; int i = git_config_bool_or_int(var, value, &is_bool); @@ -250,7 +263,7 @@ done: return err; } -static int try_to_run_foreground_daemon(void) +static int try_to_run_foreground_daemon(int detach_console) { /* * Technically, we don't need to probe for an existing daemon @@ -270,17 +283,106 @@ static int try_to_run_foreground_daemon(void) fflush(stderr); } +#ifdef GIT_WINDOWS_NATIVE + if (detach_console) + FreeConsole(); +#endif + return !!fsmonitor_run_daemon(); } +static start_bg_wait_cb bg_wait_cb; + +static int bg_wait_cb(const struct child_process *cp, void *cb_data) +{ + enum ipc_active_state s = fsmonitor_ipc__get_state(); + + switch (s) { + case IPC_STATE__LISTENING: + /* child is "ready" */ + return 0; + + case IPC_STATE__NOT_LISTENING: + case IPC_STATE__PATH_NOT_FOUND: + /* give child more time */ + return 1; + + default: + case IPC_STATE__INVALID_PATH: + case IPC_STATE__OTHER_ERROR: + /* all the time in world won't help */ + return -1; + } +} + +static int try_to_start_background_daemon(void) +{ + struct child_process cp = CHILD_PROCESS_INIT; + enum start_bg_result sbgr; + + /* + * Before we try to create a background daemon process, see + * if a daemon process is already listening. This makes it + * easier for us to report an already-listening error to the + * console, since our spawn/daemon can only report the success + * of creating the background process (and not whether it + * immediately exited). + */ + if (fsmonitor_ipc__get_state() == IPC_STATE__LISTENING) + die(_("fsmonitor--daemon is already running '%s'"), + the_repository->worktree); + + if (fsmonitor__announce_startup) { + fprintf(stderr, _("starting fsmonitor-daemon in '%s'\n"), + the_repository->worktree); + fflush(stderr); + } + + cp.git_cmd = 1; + + strvec_push(&cp.args, "fsmonitor--daemon"); + strvec_push(&cp.args, "run"); + strvec_push(&cp.args, "--detach"); + strvec_pushf(&cp.args, "--ipc-threads=%d", fsmonitor__ipc_threads); + + cp.no_stdin = 1; + cp.no_stdout = 1; + cp.no_stderr = 1; + + sbgr = start_bg_command(&cp, bg_wait_cb, NULL, + fsmonitor__start_timeout_sec); + + switch (sbgr) { + case SBGR_READY: + return 0; + + default: + case SBGR_ERROR: + case SBGR_CB_ERROR: + return error(_("daemon failed to start")); + + case SBGR_TIMEOUT: + return error(_("daemon not online yet")); + + case SBGR_DIED: + return error(_("daemon terminated")); + } +} + int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) { const char *subcmd; + int detach_console = 0; struct option options[] = { + OPT_BOOL(0, "detach", &detach_console, N_("detach from console")), OPT_INTEGER(0, "ipc-threads", &fsmonitor__ipc_threads, N_("use ipc worker threads")), + OPT_INTEGER(0, "start-timeout", + &fsmonitor__start_timeout_sec, + N_("max seconds to wait for background daemon startup")), + OPT_END() }; @@ -296,8 +398,11 @@ int cmd_fsmonitor__daemon(int argc, const char **argv, const char *prefix) die(_("invalid 'ipc-threads' value (%d)"), fsmonitor__ipc_threads); + if (!strcmp(subcmd, "start")) + return !!try_to_start_background_daemon(); + if (!strcmp(subcmd, "run")) - return !!try_to_run_foreground_daemon(); + return !!try_to_run_foreground_daemon(detach_console); if (!strcmp(subcmd, "stop")) return !!do_as_client__send_stop(); From patchwork Fri Mar 25 18:02:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8F59C433FE for ; Fri, 25 Mar 2022 19:28:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229655AbiCYTaX (ORCPT ); Fri, 25 Mar 2022 15:30:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58008 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229570AbiCYTaK (ORCPT ); Fri, 25 Mar 2022 15:30:10 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 87D8C1DC98D for ; Fri, 25 Mar 2022 12:04:50 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id v130-20020a1cac88000000b00389d0a5c511so9510871wme.5 for ; Fri, 25 Mar 2022 12:04:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8pCLbO4RGMt9FhNM1rARjgkn1qpV57NBDORbLzqjDVc=; b=WZjiHWbiP7Ct5fsqdFWscLCduan9E+R3Pb+KQPOVjIpDgVLuUKoXjEZzPaFdM0a9dL 6YVisaYLMptPTUyMTmnxC97xU89oDhfnmsyhPATR56d1k9/hETT/i7wK3PpahQ1GgqNJ d80veZXYOsZGeRuxZAk7N4I1QcTTf0bgmTmU/cxiE5HP9gYL3vHEIX2vYp2oauEF5Itd 3XO0R1VQgZ+tBKN3kReYJKliE54GAURxiK9fUj/X0uu5fCBLJ7jhphReDTAApfot22AI T19akO1/cytNtlAdEq/Y/cQk01z4jaIi3RNFUdsms0V5qHwZNTszKtX5eO0iyAKRRHB9 WW2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8pCLbO4RGMt9FhNM1rARjgkn1qpV57NBDORbLzqjDVc=; b=jj7MgjS3oJyL/AeLKVA1uzoPcqnQiWTs6uDq1lqyc22F47S/IRzlXnXt/qb0SA1E56 h9VJlaBkdkorrmVzSCfYGKcbDI8rqg+ecRTQY9qbavaA1ota/mhSr095xQZHp6sP+Ddb Gi6mitKrrjRR9D0yVi8YIw4ol/i8qgGUCGsh+7bv0AxP1aCbYsaup+gwdMs9phHF3i8w pw1YHBt1OgoXOJ8RFhC1Qpm12oykNhZgL9RhwvOZGioHAG65kkU9RK+LyH63ypB3ErM7 0Xb13+As/CLFZBBOPW+UR8pa26lfcg1bPmsRc1OG5/nPmdIgFgB1usBUWGbOWDZ+H3ju rZ8A== X-Gm-Message-State: AOAM5328YD3xEeEfhksw5+MvkP0nuZ14IETDjVVBeRxRQUkHdwEuYCI0 sdO1k4giXskL1nFcXftrQQ2ORh6tIgc= X-Google-Smtp-Source: ABdhPJzEz107nKTkDp/MvNE5WrZN9OSAbT4xjwsaYjKz3dJQEkrzv5U9a6yV7tl1SdgTbd7VnygHbA== X-Received: by 2002:a05:600c:1f11:b0:38c:6ce7:69c5 with SMTP id bd17-20020a05600c1f1100b0038c6ce769c5mr11334341wmb.64.1648231408369; Fri, 25 Mar 2022 11:03:28 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v188-20020a1cacc5000000b00384b71a50d5sm5048240wme.24.2022.03.25.11.03.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:27 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:55 +0000 Subject: [PATCH v9 12/30] fsmonitor--daemon: add pathname classification Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to classify relative and absolute pathnames and decide how they should be handled. This will be used by the platform-specific backend to respond to each filesystem event. When we register for filesystem notifications on a directory, we get events for everything (recursively) in the directory. We want to report to clients changes to tracked and untracked paths within the working directory proper. We do not want to report changes within the .git directory, for example. This classification will be used in a later commit by the different backends to classify paths as events are received. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 81 ++++++++++++++++++++++++++++++++++ fsmonitor--daemon.h | 87 +++++++++++++++++++++++++++++++++++++ 2 files changed, 168 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 69dd39121a3..1ce00b7c150 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -134,6 +134,87 @@ static int handle_client(void *data, return result; } +#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" + +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *rel) +{ + if (fspathncmp(rel, ".git", 4)) + return IS_WORKDIR_PATH; + rel += 4; + + if (!*rel) + return IS_DOT_GIT; + if (*rel != '/') + return IS_WORKDIR_PATH; /* e.g. .gitignore */ + rel++; + + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX; + + return IS_INSIDE_DOT_GIT; +} + +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *rel) +{ + if (!fspathncmp(rel, FSMONITOR_COOKIE_PREFIX, + strlen(FSMONITOR_COOKIE_PREFIX))) + return IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX; + + return IS_INSIDE_GITDIR; +} + +static enum fsmonitor_path_type try_classify_workdir_abs_path( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + + if (fspathncmp(path, state->path_worktree_watch.buf, + state->path_worktree_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_worktree_watch.len; + + if (!*rel) + return IS_WORKDIR_PATH; /* it is the root dir exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_workdir_relative(rel); +} + +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path) +{ + const char *rel; + enum fsmonitor_path_type t; + + t = try_classify_workdir_abs_path(state, path); + if (state->nr_paths_watching == 1) + return t; + if (t != IS_OUTSIDE_CONE) + return t; + + if (fspathncmp(path, state->path_gitdir_watch.buf, + state->path_gitdir_watch.len)) + return IS_OUTSIDE_CONE; + + rel = path + state->path_gitdir_watch.len; + + if (!*rel) + return IS_GITDIR; /* it is the exactly */ + if (*rel != '/') + return IS_OUTSIDE_CONE; + rel++; + + return fsmonitor_classify_path_gitdir_relative(rel); +} + static void *fsm_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 3009c1a83de..8c3a71a48bd 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -30,5 +30,92 @@ struct fsmonitor_daemon_state { struct ipc_server_data *ipc_server_data; }; +/* + * Pathname classifications. + * + * The daemon classifies the pathnames that it receives from file + * system notification events into the following categories and uses + * that to decide whether clients are told about them. (And to watch + * for file system synchronization events.) + * + * The daemon only collects and reports on the set of modified paths + * within the working directory (proper). + * + * The client should only care about paths within the working + * directory proper (inside the working directory and not ".git" nor + * inside of ".git/"). That is, the client has read the index and is + * asking for a list of any paths in the working directory that have + * been modified since the last token. The client does not care about + * file system changes within the ".git/" directory (such as new loose + * objects or packfiles). So the client will only receive paths that + * are classified as IS_WORKDIR_PATH. + * + * Note that ".git" is usually a directory and is therefore inside + * the cone of the FS watch that we have on the working directory root, + * so we will also get FS events for disk activity on and within ".git/" + * that we need to respond to or filter from the client. + * + * But Git also allows ".git" to be a *file* that points to a GITDIR + * outside of the working directory. When this happens, we need to + * create FS watches on both the working directory root *and* on the + * (external) GITDIR root. (The latter is required because we put + * cookie files inside it and use them to sync with the FS event + * stream.) + * + * Note that in the context of this discussion, I'm using "GITDIR" + * to only mean an external GITDIR referenced by a ".git" file. + * + * The platform FS event backends will receive watch-specific + * relative paths (except for those OS's that always emit absolute + * paths). We use the following enum and routines to classify each + * path so that we know how to handle it. There is a slight asymmetry + * here because ".git/" is inside the working directory and the + * (external) GITDIR is not, and therefore how we handle events may + * vary slightly, so I have different enums for "IS...DOT_GIT..." and + * "IS...GITDIR...". + * + * The daemon uses the IS_DOT_GIT and IS_GITDIR internally to mean the + * exact ".git" file/directory or GITDIR directory. If the daemon + * receives a delete event for either of these paths, it will + * automatically shutdown, for example. + * + * Note that the daemon DOES NOT explicitly watch nor special case the + * index. The daemon does not read the index nor have any internal + * index-relative state, so there are no "IS...INDEX..." enum values. + */ +enum fsmonitor_path_type { + IS_WORKDIR_PATH = 0, + + IS_DOT_GIT, + IS_INSIDE_DOT_GIT, + IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX, + + IS_GITDIR, + IS_INSIDE_GITDIR, + IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX, + + IS_OUTSIDE_CONE, +}; + +/* + * Classify a pathname relative to the root of the working directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( + const char *relative_path); + +/* + * Classify a pathname relative to a that is external to the + * worktree directory. + */ +enum fsmonitor_path_type fsmonitor_classify_path_gitdir_relative( + const char *relative_path); + +/* + * Classify an absolute pathname received from a filesystem event. + */ +enum fsmonitor_path_type fsmonitor_classify_path_absolute( + struct fsmonitor_daemon_state *state, + const char *path); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Fri Mar 25 18:02:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AD55C433F5 for ; Fri, 25 Mar 2022 19:30:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229939AbiCYTbz (ORCPT ); Fri, 25 Mar 2022 15:31:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230004AbiCYTbr (ORCPT ); Fri, 25 Mar 2022 15:31:47 -0400 Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5037F20DB2F for ; Fri, 25 Mar 2022 12:06:34 -0700 (PDT) Received: by mail-ej1-x632.google.com with SMTP id pv16so17245975ejb.0 for ; Fri, 25 Mar 2022 12:06:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=diJ2kF8fCXK8fuWsFs2qoSTo98Y5oZz+kxAWbs2iN34=; b=pRYQXi2+9XRJ6ar6k5xapqu7x1y7F9D+ZOiuOtG4yNVidyWQsQjuWT6YVwQvD1+mlC GE9VcqmOawBDYslznFGfwt9/hyUoRMi2sTSJH0VTVsaW7WFFDjQwCsks/xiSMXPdbNJW HMaezf2ilxkste4uCAvis2DqkwbdaTFCR/R5DgAwFevJKmVxBPTrW0WagZMfCani7OVB 4Vlr4k/uKuiq6LIQNsTNJhPGNr+YLIyAZ+K2nloePUK32KZmudhOF9MN1ecc9D7qLDwO 6sqa3PDkDjKPkX7rJBVo2rK3N6Km+LTeOcwZ6f+3+TPTm+wTk9+QgP+0tu8SujYNKBPr uZEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=diJ2kF8fCXK8fuWsFs2qoSTo98Y5oZz+kxAWbs2iN34=; b=5HCSlZslVWC9mRD7dabd2hC0OVpPyEnUAlCQi5Q+2IjQPndpb+ObLSDcP6YyiD4mOE 69IRWwmu9/xpmlGTVZr0hYgw6Rd8L+YawTwg2E1Vahc05xvc2zillXIV+ELIRo8yOZB8 JRKwe40z8GmSdtOKFhQ0WUqtcpxpuxQPcwq7ab3FrMHS/ry3GocomQd4BTDJsOiW071a ray0mxBscQ70WahCmyMzg4gWoDVfdo8E4Z+7GdCmAjkFqKDNvaxmCLnhopVfOzr7savF AKnyBheFoyFQz+HuXKmeYv8HYluaM3T9pbsnwxscIL/VSIkI9vA8d+9KVXQBc2iQQMvF HIPw== X-Gm-Message-State: AOAM530IRdEg2HjCD94jt7YECbXfoAdq6qFeyGYcdG20gZRq5nO0Idwa 2QizHb/eLG/J7WSnJ6dNjuT/I02+cDU= X-Google-Smtp-Source: ABdhPJzCWJSikK/lGO1UYeTf0E1YqrSYdFi3eZkH8kfaHhAyfLa68jRbO/cMgRCB5q5rN+NHascYOg== X-Received: by 2002:adf:ba8f:0:b0:1e9:4afb:179b with SMTP id p15-20020adfba8f000000b001e94afb179bmr10366594wrg.57.1648231409296; Fri, 25 Mar 2022 11:03:29 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f16-20020a05600c155000b0038ced37c182sm1713789wmg.35.2022.03.25.11.03.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:28 -0700 (PDT) Message-Id: <53e06b4ae5d0eb1d3c2cb6a61ffd14abc36f6534.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:56 +0000 Subject: [PATCH v9 13/30] fsmonitor--daemon: define token-ids Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to create token-ids and define the overall token naming scheme. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 116 +++++++++++++++++++++++++++++++++++- 1 file changed, 115 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 1ce00b7c150..1c7c156288d 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -106,6 +106,120 @@ static int do_as_client__status(void) } } +/* + * Requests to and from a FSMonitor Protocol V2 provider use an opaque + * "token" as a virtual timestamp. Clients can request a summary of all + * created/deleted/modified files relative to a token. In the response, + * clients receive a new token for the next (relative) request. + * + * + * Token Format + * ============ + * + * The contents of the token are private and provider-specific. + * + * For the built-in fsmonitor--daemon, we define a token as follows: + * + * "builtin" ":" ":" + * + * The "builtin" prefix is used as a namespace to avoid conflicts + * with other providers (such as Watchman). + * + * The is an arbitrary OPAQUE string, such as a GUID, + * UUID, or {timestamp,pid}. It is used to group all filesystem + * events that happened while the daemon was monitoring (and in-sync + * with the filesystem). + * + * Unlike FSMonitor Protocol V1, it is not defined as a timestamp + * and does not define less-than/greater-than relationships. + * (There are too many race conditions to rely on file system + * event timestamps.) + * + * The is a simple integer incremented whenever the + * daemon needs to make its state public. For example, if 1000 file + * system events come in, but no clients have requested the data, + * the daemon can continue to accumulate file changes in the same + * bin and does not need to advance the sequence number. However, + * as soon as a client does arrive, the daemon needs to start a new + * bin and increment the sequence number. + * + * The sequence number serves as the boundary between 2 sets + * of bins -- the older ones that the client has already seen + * and the newer ones that it hasn't. + * + * When a new is created, the is reset to + * zero. + * + * + * About Token Ids + * =============== + * + * A new token_id is created: + * + * [1] each time the daemon is started. + * + * [2] any time that the daemon must re-sync with the filesystem + * (such as when the kernel drops or we miss events on a very + * active volume). + * + * [3] in response to a client "flush" command (for dropped event + * testing). + * + * When a new token_id is created, the daemon is free to discard all + * cached filesystem events associated with any previous token_ids. + * Events associated with a non-current token_id will never be sent + * to a client. A token_id change implicitly means that the daemon + * has gap in its event history. + * + * Therefore, clients that present a token with a stale (non-current) + * token_id will always be given a trivial response. + */ +struct fsmonitor_token_data { + struct strbuf token_id; + struct fsmonitor_batch *batch_head; + struct fsmonitor_batch *batch_tail; + uint64_t client_ref_count; +}; + +static struct fsmonitor_token_data *fsmonitor_new_token_data(void) +{ + static int test_env_value = -1; + static uint64_t flush_count = 0; + struct fsmonitor_token_data *token; + + CALLOC_ARRAY(token, 1); + + strbuf_init(&token->token_id, 0); + token->batch_head = NULL; + token->batch_tail = NULL; + token->client_ref_count = 0; + + if (test_env_value < 0) + test_env_value = git_env_bool("GIT_TEST_FSMONITOR_TOKEN", 0); + + if (!test_env_value) { + struct timeval tv; + struct tm tm; + time_t secs; + + gettimeofday(&tv, NULL); + secs = tv.tv_sec; + gmtime_r(&secs, &tm); + + strbuf_addf(&token->token_id, + "%"PRIu64".%d.%4d%02d%02dT%02d%02d%02d.%06ldZ", + flush_count++, + getpid(), + tm.tm_year + 1900, tm.tm_mon + 1, tm.tm_mday, + tm.tm_hour, tm.tm_min, tm.tm_sec, + (long)tv.tv_usec); + } else { + strbuf_addf(&token->token_id, "test_%08x", test_env_value++); + } + + return token; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -298,7 +412,7 @@ static int fsmonitor_run_daemon(void) pthread_mutex_init(&state.main_lock, NULL); state.error_code = 0; - state.current_token_data = NULL; + state.current_token_data = fsmonitor_new_token_data(); /* Prepare to (recursively) watch the directory. */ strbuf_init(&state.path_worktree_watch, 0); From patchwork Fri Mar 25 18:02:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C8D7C4332F for ; Fri, 25 Mar 2022 19:31:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230011AbiCYTci (ORCPT ); Fri, 25 Mar 2022 15:32:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51088 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229993AbiCYTcT (ORCPT ); Fri, 25 Mar 2022 15:32:19 -0400 Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBE68DA090 for ; Fri, 25 Mar 2022 12:07:03 -0700 (PDT) Received: by mail-ej1-x62d.google.com with SMTP id yy13so17205760ejb.2 for ; Fri, 25 Mar 2022 12:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CagY9EkHGPSukr9Gcxdb9icbGtAz+5el1oCyBPufJIs=; b=LLgtNcSvXyPAzIVG6DChQvCy7KvNP4mb7psEa8bnX0C9hqL3KYT85ZW+5v8WGlv2mz C7k4Cu7QF+TkvSWsPybzgjR86lQ7zhJwdZUyBf7ippdXBD90EJjtmEHRqvkZ4uoeBHdV +0uyLPSBosjlf4q/AKKEXlYVu9dopQ3k+5LOzg/zGiGqFpEJqrjKu5E9HJ7gt/8JS55v JFBGu0Ax/xySkm53OsJSZo0GqT5/Xi0sYQWltCHmGSV+nxR9riPRAmr5dZKOQ0xP/NnO UNA4GE1S2C3cndvuz5havTfZum6EZTqJZJyRh01BLQIQ4MBVZSEBse2GPbumybDnjVZw 5RKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CagY9EkHGPSukr9Gcxdb9icbGtAz+5el1oCyBPufJIs=; b=p60H+5Xi48yCmwUvcgb0vcHtr2hDZNoeyYlH5zLgWMGVJpGYhLNWv+QfYQHKI6BqEm Hci0EqCFltFbPCOjJP/XnsFNBGNP+apPrTWIunyXnhEEcd4KGi4USV8odNWlXPTwT7Nk SXdFHl88V2IAuQ5LttVFZW0NoU2NspOD0DV1htnQmi1qqUmZsbebak5j7gSbhhON+30E jy9E3pzYrd2jxeGZO7es4fMqsUg08dIlo1IYuqLjXUoJf+5voau2Op434iPSOhVuaJEu f3UpkB9CrPO4kuEnXf0vZcxEhP+MQvPLiUpOnzUBzGrTKYrf/rdlbVFCa0CbhE+2MRnY tqbw== X-Gm-Message-State: AOAM531TcXMnkkSLc97O7wDzJ/1LX8pKoRj7IynLSAiXyPJibnwzCdom HSU9n8hdpwPwhk6DhUNrM+THbgplK+M= X-Google-Smtp-Source: ABdhPJyUu4Vv9sxO2nPG/l7mkXZ3rizJT55bjPLVPl2AfWR9JCKs6LxdREIz5HC5LrY0+NIql0blDg== X-Received: by 2002:a5d:47a6:0:b0:205:97fc:8e98 with SMTP id 6-20020a5d47a6000000b0020597fc8e98mr7987385wrb.103.1648231410533; Fri, 25 Mar 2022 11:03:30 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p16-20020a5d6390000000b00203ffebddf3sm7652126wru.99.2022.03.25.11.03.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:29 -0700 (PDT) Message-Id: <39f43fabe024917e6c84b8916197b0dc0add97fc.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:57 +0000 Subject: [PATCH v9 14/30] fsmonitor--daemon: create token-based changed path cache Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to build a list of changed paths and associate them with a token-id. This will be used by the platform-specific backends to accumulate changed paths in response to filesystem events. The platform-specific file system listener thread receives file system events containing one or more changed pathnames (with whatever bucketing or grouping that is convenient for the file system). These paths are accumulated (without locking) by the file system layer into a `fsmonitor_batch`. When the file system layer has drained the kernel event queue, it will "publish" them to our token queue and make them visible to concurrent client worker threads. The token layer is free to combine and/or de-dup paths within these batches for efficient presentation to clients. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 230 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 40 +++++++ 2 files changed, 268 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 1c7c156288d..69312119b07 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -181,17 +181,27 @@ struct fsmonitor_token_data { uint64_t client_ref_count; }; +struct fsmonitor_batch { + struct fsmonitor_batch *next; + uint64_t batch_seq_nr; + const char **interned_paths; + size_t nr, alloc; + time_t pinned_time; +}; + static struct fsmonitor_token_data *fsmonitor_new_token_data(void) { static int test_env_value = -1; static uint64_t flush_count = 0; struct fsmonitor_token_data *token; + struct fsmonitor_batch *batch; CALLOC_ARRAY(token, 1); + batch = fsmonitor_batch__new(); strbuf_init(&token->token_id, 0); - token->batch_head = NULL; - token->batch_tail = NULL; + token->batch_head = batch; + token->batch_tail = batch; token->client_ref_count = 0; if (test_env_value < 0) @@ -217,9 +227,143 @@ static struct fsmonitor_token_data *fsmonitor_new_token_data(void) strbuf_addf(&token->token_id, "test_%08x", test_env_value++); } + /* + * We created a new and are starting a new series + * of tokens with a zero . + * + * Since clients cannot guess our new (non test) + * they will always receive a trivial response (because of the + * mismatch on the ). The trivial response will + * tell them our new so that subsequent requests + * will be relative to our new series. (And when sending that + * response, we pin the current head of the batch list.) + * + * Even if the client correctly guesses the , their + * request of "builtin::0" asks for all changes MORE + * RECENT than batch/bin 0. + * + * This implies that it is a waste to accumulate paths in the + * initial batch/bin (because they will never be transmitted). + * + * So the daemon could be running for days and watching the + * file system, but doesn't need to actually accumulate any + * paths UNTIL we need to set a reference point for a later + * relative request. + * + * However, it is very useful for testing to always have a + * reference point set. Pin batch 0 to force early file system + * events to accumulate. + */ + if (test_env_value) + batch->pinned_time = time(NULL); + return token; } +struct fsmonitor_batch *fsmonitor_batch__new(void) +{ + struct fsmonitor_batch *batch; + + CALLOC_ARRAY(batch, 1); + + return batch; +} + +void fsmonitor_batch__free_list(struct fsmonitor_batch *batch) +{ + while (batch) { + struct fsmonitor_batch *next = batch->next; + + /* + * The actual strings within the array of this batch + * are interned, so we don't own them. We only own + * the array. + */ + free(batch->interned_paths); + free(batch); + + batch = next; + } +} + +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, + const char *path) +{ + const char *interned_path = strintern(path); + + trace_printf_key(&trace_fsmonitor, "event: %s", interned_path); + + ALLOC_GROW(batch->interned_paths, batch->nr + 1, batch->alloc); + batch->interned_paths[batch->nr++] = interned_path; +} + +static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, + const struct fsmonitor_batch *batch_src) +{ + size_t k; + + ALLOC_GROW(batch_dest->interned_paths, + batch_dest->nr + batch_src->nr + 1, + batch_dest->alloc); + + for (k = 0; k < batch_src->nr; k++) + batch_dest->interned_paths[batch_dest->nr++] = + batch_src->interned_paths[k]; +} + +static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) +{ + if (!token) + return; + + assert(token->client_ref_count == 0); + + strbuf_release(&token->token_id); + + fsmonitor_batch__free_list(token->batch_head); + + free(token); +} + +/* + * Flush all of our cached data about the filesystem. Call this if we + * lose sync with the filesystem and miss some notification events. + * + * [1] If we are missing events, then we no longer have a complete + * history of the directory (relative to our current start token). + * We should create a new token and start fresh (as if we just + * booted up). + * + * If there are no concurrent threads reading the current token data + * series, we can free it now. Otherwise, let the last reader free + * it. + * + * Either way, the old token data series is no longer associated with + * our state data. + */ +static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + struct fsmonitor_token_data *free_me = NULL; + struct fsmonitor_token_data *new_one = NULL; + + new_one = fsmonitor_new_token_data(); + + if (state->current_token_data->client_ref_count == 0) + free_me = state->current_token_data; + state->current_token_data = new_one; + + fsmonitor_free_token_data(free_me); +} + +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) +{ + pthread_mutex_lock(&state->main_lock); + with_lock__do_force_resync(state); + pthread_mutex_unlock(&state->main_lock); +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -329,6 +473,81 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( return fsmonitor_classify_path_gitdir_relative(rel); } +/* + * We try to combine small batches at the front of the batch-list to avoid + * having a long list. This hopefully makes it a little easier when we want + * to truncate and maintain the list. However, we don't want the paths array + * to just keep growing and growing with realloc, so we insert an arbitrary + * limit. + */ +#define MY_COMBINE_LIMIT (1024) + +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names) +{ + if (!batch && !cookie_names->nr) + return; + + pthread_mutex_lock(&state->main_lock); + + if (batch) { + struct fsmonitor_batch *head; + + head = state->current_token_data->batch_head; + if (!head) { + BUG("token does not have batch"); + } else if (head->pinned_time) { + /* + * We cannot alter the current batch list + * because: + * + * [a] it is being transmitted to at least one + * client and the handle_client() thread has a + * ref-count, but not a lock on the batch list + * starting with this item. + * + * [b] it has been transmitted in the past to + * at least one client such that future + * requests are relative to this head batch. + * + * So, we can only prepend a new batch onto + * the front of the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else if (!head->batch_seq_nr) { + /* + * Batch 0 is unpinned. See the note in + * `fsmonitor_new_token_data()` about why we + * don't need to accumulate these paths. + */ + fsmonitor_batch__free_list(batch); + } else if (head->nr + batch->nr > MY_COMBINE_LIMIT) { + /* + * The head batch in the list has never been + * transmitted to a client, but folding the + * contents of the new batch onto it would + * exceed our arbitrary limit, so just prepend + * the new batch onto the list. + */ + batch->batch_seq_nr = head->batch_seq_nr + 1; + batch->next = head; + state->current_token_data->batch_head = batch; + } else { + /* + * We are free to add the paths in the given + * batch onto the end of the current head batch. + */ + fsmonitor_batch__combine(head, batch); + fsmonitor_batch__free_list(batch); + } + } + + pthread_mutex_unlock(&state->main_lock); +} + static void *fsm_listen__thread_proc(void *_state) { struct fsmonitor_daemon_state *state = _state; @@ -343,6 +562,13 @@ static void *fsm_listen__thread_proc(void *_state) fsm_listen__loop(state); + pthread_mutex_lock(&state->main_lock); + if (state->current_token_data && + state->current_token_data->client_ref_count == 0) + fsmonitor_free_token_data(state->current_token_data); + state->current_token_data = NULL; + pthread_mutex_unlock(&state->main_lock); + trace2_thread_exit(); return NULL; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 8c3a71a48bd..010fbfe60e9 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -12,6 +12,27 @@ struct fsmonitor_batch; struct fsmonitor_token_data; +/* + * Create a new batch of path(s). The returned batch is considered + * private and not linked into the fsmonitor daemon state. The caller + * should fill this batch with one or more paths and then publish it. + */ +struct fsmonitor_batch *fsmonitor_batch__new(void); + +/* + * Free the list of batches starting with this one. + */ +void fsmonitor_batch__free_list(struct fsmonitor_batch *batch); + +/* + * Add this path to this batch of modified files. + * + * The batch should be private and NOT (yet) linked into the fsmonitor + * daemon state and therefore not yet visible to worker threads and so + * no locking is required. + */ +void fsmonitor_batch__add_path(struct fsmonitor_batch *batch, const char *path); + struct fsmonitor_daemon_backend_data; /* opaque platform-specific data */ struct fsmonitor_daemon_state { @@ -117,5 +138,24 @@ enum fsmonitor_path_type fsmonitor_classify_path_absolute( struct fsmonitor_daemon_state *state, const char *path); +/* + * Prepend the this batch of path(s) onto the list of batches associated + * with the current token. This makes the batch visible to worker threads. + * + * The caller no longer owns the batch and must not free it. + * + * Wake up the client threads waiting on these cookies. + */ +void fsmonitor_publish(struct fsmonitor_daemon_state *state, + struct fsmonitor_batch *batch, + const struct string_list *cookie_names); + +/* + * If the platform-specific layer loses sync with the filesystem, + * it should call this to invalidate cached data and abort waiting + * threads. + */ +void fsmonitor_force_resync(struct fsmonitor_daemon_state *state); + #endif /* HAVE_FSMONITOR_DAEMON_BACKEND */ #endif /* FSMONITOR_DAEMON_H */ From patchwork Fri Mar 25 18:02:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12792006 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72323C433F5 for ; Fri, 25 Mar 2022 19:43:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231523AbiCYTpb (ORCPT ); Fri, 25 Mar 2022 15:45:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42434 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231527AbiCYTpL (ORCPT ); Fri, 25 Mar 2022 15:45:11 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6CE0326ED0 for ; Fri, 25 Mar 2022 12:15:27 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id p189so4981403wmp.3 for ; Fri, 25 Mar 2022 12:15:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=27EYsw0sQz+mI4HvJWY1VfC1P39Tg+tZZLnW70cporg=; b=bDfPhwbIfOb84Qk2XdrFl3Wjr8XxzIKCdEOLsAR8CfMJfzS8HfpH3jNo7PXhi901Nc YA2ob2x3miLhjP7W9ERhbH/j66uLu+w9U3BryBUXfCbJD5vYZVANhfsK+bDuOUk99ah2 c2EWDyKK6nAyCRePUqBSNC5rgtdZ+WXSRCLDAIgwYOsRjBFODrm09aDawO+v66b0xcTx kCrq3Pm50ywJNknhvCcksHqT1EbgBo2Sg1Ada0y8DQ8dP68BH26RKRHzGshNmQFbA04l qPJ1v3neFQ14HxUm1xyWeCFomI9ei7O87tj1xzAkENLzEFilHshCXA0PofoseV5PSaNA oUig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=27EYsw0sQz+mI4HvJWY1VfC1P39Tg+tZZLnW70cporg=; b=wmin3mNUnLo24tN0o2YQRJochj8bv/fIkRsczrjlnEuGBQTTwvW3wJIUEPFzN6eXed T2fHJnBtJQdcelGAgnzHNRALzNi+qWQSEBpa5bwphyjbNo9N9GNG8AenCJfom7ZtKQTM MHH3NYGsj/7IydYPpnYn4Ug/RSzwfkumhmC4RIEZ/Dvt4N8AYoCBZqEUfz6QLQKgZLqp LbotrMVG9Y3UEHJVfWvvqmkZYDoPRihT+Cr4SIkHDYIcYrSOYGkfDkY08qO/MqmwzC1p CgJBfiy434bJNwKNPzxo2ciC4ZviO9EIbqPr/DcIteH+uoov9goiMTJ8IxPJUbQUmkt/ pmeA== X-Gm-Message-State: AOAM530ChH7I5HDhhDBS5HI4xpLT/Er8weEsQjp0MuCJKcHhAGOSYsQ+ Ke26xYCem3Vf/7uCgbDIYebE1kvCy4A= X-Google-Smtp-Source: ABdhPJwb/jvTsXtHhhVIcXVZe/2kVEchQ4tzwIQKZc+zdTvvHB9xqycz/iO5jTku9SIF50vX11KDBQ== X-Received: by 2002:a05:600c:4f08:b0:38c:93fd:570f with SMTP id l8-20020a05600c4f0800b0038c93fd570fmr11210777wmq.136.1648231411452; Fri, 25 Mar 2022 11:03:31 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p2-20020a5d4582000000b00203f51aa12asm5598641wrq.55.2022.03.25.11.03.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:31 -0700 (PDT) Message-Id: <239558e34ffea4240de8699707cd3b404b682488.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:58 +0000 Subject: [PATCH v9 15/30] compat/fsmonitor/fsm-listen-win32: implement FSMonitor backend on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach the win32 backend to register a watch on the working tree root directory (recursively). Also watch the if it is not inside the working tree. And to collect path change notifications into batches and publish. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- compat/fsmonitor/fsm-listen-win32.c | 565 ++++++++++++++++++++++++++++ 1 file changed, 565 insertions(+) diff --git a/compat/fsmonitor/fsm-listen-win32.c b/compat/fsmonitor/fsm-listen-win32.c index 916cbea254f..5b928ab66e5 100644 --- a/compat/fsmonitor/fsm-listen-win32.c +++ b/compat/fsmonitor/fsm-listen-win32.c @@ -2,20 +2,585 @@ #include "config.h" #include "fsmonitor.h" #include "fsm-listen.h" +#include "fsmonitor--daemon.h" + +/* + * The documentation of ReadDirectoryChangesW() states that the maximum + * buffer size is 64K when the monitored directory is remote. + * + * Larger buffers may be used when the monitored directory is local and + * will help us receive events faster from the kernel and avoid dropped + * events. + * + * So we try to use a very large buffer and silently fallback to 64K if + * we get an error. + */ +#define MAX_RDCW_BUF_FALLBACK (65536) +#define MAX_RDCW_BUF (65536 * 8) + +struct one_watch +{ + char buffer[MAX_RDCW_BUF]; + DWORD buf_len; + DWORD count; + + struct strbuf path; + HANDLE hDir; + HANDLE hEvent; + OVERLAPPED overlapped; + + /* + * Is there an active ReadDirectoryChangesW() call pending. If so, we + * need to later call GetOverlappedResult() and possibly CancelIoEx(). + */ + BOOL is_active; +}; + +struct fsmonitor_daemon_backend_data +{ + struct one_watch *watch_worktree; + struct one_watch *watch_gitdir; + + HANDLE hEventShutdown; + + HANDLE hListener[3]; /* we don't own these handles */ +#define LISTENER_SHUTDOWN 0 +#define LISTENER_HAVE_DATA_WORKTREE 1 +#define LISTENER_HAVE_DATA_GITDIR 2 + int nr_listener_handles; +}; + +/* + * Convert the WCHAR path from the notification into UTF8 and + * then normalize it. + */ +static int normalize_path_in_utf8(FILE_NOTIFY_INFORMATION *info, + struct strbuf *normalized_path) +{ + int reserve; + int len = 0; + + strbuf_reset(normalized_path); + if (!info->FileNameLength) + goto normalize; + + /* + * Pre-reserve enough space in the UTF8 buffer for + * each Unicode WCHAR character to be mapped into a + * sequence of 2 UTF8 characters. That should let us + * avoid ERROR_INSUFFICIENT_BUFFER 99.9+% of the time. + */ + reserve = info->FileNameLength + 1; + strbuf_grow(normalized_path, reserve); + + for (;;) { + len = WideCharToMultiByte(CP_UTF8, 0, info->FileName, + info->FileNameLength / sizeof(WCHAR), + normalized_path->buf, + strbuf_avail(normalized_path) - 1, + NULL, NULL); + if (len > 0) + goto normalize; + if (GetLastError() != ERROR_INSUFFICIENT_BUFFER) { + error(_("[GLE %ld] could not convert path to UTF-8: '%.*ls'"), + GetLastError(), + (int)(info->FileNameLength / sizeof(WCHAR)), + info->FileName); + return -1; + } + + strbuf_grow(normalized_path, + strbuf_avail(normalized_path) + reserve); + } + +normalize: + strbuf_setlen(normalized_path, len); + return strbuf_normalize_path(normalized_path); +} void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { + SetEvent(state->backend_data->hListener[LISTENER_SHUTDOWN]); +} + +static struct one_watch *create_watch(struct fsmonitor_daemon_state *state, + const char *path) +{ + struct one_watch *watch = NULL; + DWORD desired_access = FILE_LIST_DIRECTORY; + DWORD share_mode = + FILE_SHARE_WRITE | FILE_SHARE_READ | FILE_SHARE_DELETE; + HANDLE hDir; + wchar_t wpath[MAX_PATH]; + + if (xutftowcs_path(wpath, path) < 0) { + error(_("could not convert to wide characters: '%s'"), path); + return NULL; + } + + hDir = CreateFileW(wpath, + desired_access, share_mode, NULL, OPEN_EXISTING, + FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OVERLAPPED, + NULL); + if (hDir == INVALID_HANDLE_VALUE) { + error(_("[GLE %ld] could not watch '%s'"), + GetLastError(), path); + return NULL; + } + + CALLOC_ARRAY(watch, 1); + + watch->buf_len = sizeof(watch->buffer); /* assume full MAX_RDCW_BUF */ + + strbuf_init(&watch->path, 0); + strbuf_addstr(&watch->path, path); + + watch->hDir = hDir; + watch->hEvent = CreateEvent(NULL, TRUE, FALSE, NULL); + + return watch; +} + +static void destroy_watch(struct one_watch *watch) +{ + if (!watch) + return; + + strbuf_release(&watch->path); + if (watch->hDir != INVALID_HANDLE_VALUE) + CloseHandle(watch->hDir); + if (watch->hEvent != INVALID_HANDLE_VALUE) + CloseHandle(watch->hEvent); + + free(watch); +} + +static int start_rdcw_watch(struct fsmonitor_daemon_backend_data *data, + struct one_watch *watch) +{ + DWORD dwNotifyFilter = + FILE_NOTIFY_CHANGE_FILE_NAME | + FILE_NOTIFY_CHANGE_DIR_NAME | + FILE_NOTIFY_CHANGE_ATTRIBUTES | + FILE_NOTIFY_CHANGE_SIZE | + FILE_NOTIFY_CHANGE_LAST_WRITE | + FILE_NOTIFY_CHANGE_CREATION; + + ResetEvent(watch->hEvent); + + memset(&watch->overlapped, 0, sizeof(watch->overlapped)); + watch->overlapped.hEvent = watch->hEvent; + + /* + * Queue an async call using Overlapped IO. This returns immediately. + * Our event handle will be signalled when the real result is available. + * + * The return value here just means that we successfully queued it. + * We won't know if the Read...() actually produces data until later. + */ + watch->is_active = ReadDirectoryChangesW( + watch->hDir, watch->buffer, watch->buf_len, TRUE, + dwNotifyFilter, &watch->count, &watch->overlapped, NULL); + + if (watch->is_active) + return 0; + + error(_("ReadDirectoryChangedW failed on '%s' [GLE %ld]"), + watch->path.buf, GetLastError()); + return -1; +} + +static int recv_rdcw_watch(struct one_watch *watch) +{ + DWORD gle; + + watch->is_active = FALSE; + + /* + * The overlapped result is ready. If the Read...() was successful + * we finally receive the actual result into our buffer. + */ + if (GetOverlappedResult(watch->hDir, &watch->overlapped, &watch->count, + TRUE)) + return 0; + + gle = GetLastError(); + if (gle == ERROR_INVALID_PARAMETER && + /* + * The kernel throws an invalid parameter error when our + * buffer is too big and we are pointed at a remote + * directory (and possibly for other reasons). Quietly + * set it down and try again. + * + * See note about MAX_RDCW_BUF at the top. + */ + watch->buf_len > MAX_RDCW_BUF_FALLBACK) { + watch->buf_len = MAX_RDCW_BUF_FALLBACK; + return -2; + } + + /* + * NEEDSWORK: If an external is deleted, the above + * returns an error. I'm not sure that there's anything that + * we can do here other than failing -- the /.git + * link file would be broken anyway. We might try to check + * for that and return a better error message, but I'm not + * sure it is worth it. + */ + + error(_("GetOverlappedResult failed on '%s' [GLE %ld]"), + watch->path.buf, gle); + return -1; +} + +static void cancel_rdcw_watch(struct one_watch *watch) +{ + DWORD count; + + if (!watch || !watch->is_active) + return; + + /* + * The calls to ReadDirectoryChangesW() and GetOverlappedResult() + * form a "pair" (my term) where we queue an IO and promise to + * hang around and wait for the kernel to give us the result. + * + * If for some reason after we queue the IO, we have to quit + * or otherwise not stick around for the second half, we must + * tell the kernel to abort the IO. This prevents the kernel + * from writing to our buffer and/or signalling our event + * after we free them. + * + * (Ask me how much fun it was to track that one down). + */ + CancelIoEx(watch->hDir, &watch->overlapped); + GetOverlappedResult(watch->hDir, &watch->overlapped, &count, TRUE); + watch->is_active = FALSE; +} + +/* + * Process filesystem events that happen anywhere (recursively) under the + * root directory. For a normal working directory, this includes + * both version controlled files and the contents of the .git/ directory. + * + * If /.git is a file, then we only see events for the file + * itself. + */ +static int process_worktree_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_worktree; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + struct fsmonitor_batch *batch = NULL; + const char *p = watch->buffer; + + /* + * If the kernel gets more events than will fit in the kernel + * buffer associated with our RDCW handle, it drops them and + * returns a count of zero. + * + * Yes, the call returns WITHOUT error and with length zero. + * This is the documented behavior. (My testing has confirmed + * that it also sets the last error to ERROR_NOTIFY_ENUM_DIR, + * but we do not rely on that since the function did not + * return an error and it is not documented.) + * + * (The "overflow" case is not ambiguous with the "no data" case + * because we did an INFINITE wait.) + * + * This means we have a gap in coverage. Tell the daemon layer + * to resync. + */ + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_WORKTREE; + } + + /* + * On Windows, `info` contains an "array" of paths that are + * relative to the root of whichever directory handle received + * the event. + */ + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_workdir_relative(path.buf); + + switch (t) { + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + /* special case cookie files within .git */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_DOT_GIT: + /* ignore everything inside of "/.git/" */ + break; + + case IS_DOT_GIT: + /* "/.git" was deleted (or renamed away) */ + if ((info->Action == FILE_ACTION_REMOVED) || + (info->Action == FILE_ACTION_RENAMED_OLD_NAME)) { + trace2_data_string("fsmonitor", NULL, + "fsm-listen/dotgit", + "removed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* queue normal pathname */ + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, path.buf); + break; + + case IS_GITDIR: + case IS_INSIDE_GITDIR: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, batch, &cookie_list); + batch = NULL; + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_WORKTREE; + +force_shutdown: + fsmonitor_batch__free_list(batch); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_SHUTDOWN; +} + +/* + * Process filesystem events that happened anywhere (recursively) under the + * external (such as non-primary worktrees or submodules). + * We only care about cookie files that our client threads created here. + * + * Note that we DO NOT get filesystem events on the external + * itself (it is not inside something that we are watching). In particular, + * we do not get an event if the external is deleted. + */ +static int process_gitdir_events(struct fsmonitor_daemon_state *state) +{ + struct fsmonitor_daemon_backend_data *data = state->backend_data; + struct one_watch *watch = data->watch_gitdir; + struct strbuf path = STRBUF_INIT; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *p = watch->buffer; + + if (!watch->count) { + trace2_data_string("fsmonitor", NULL, "fsm-listen/kernel", + "overflow"); + fsmonitor_force_resync(state); + return LISTENER_HAVE_DATA_GITDIR; + } + + for (;;) { + FILE_NOTIFY_INFORMATION *info = (void *)p; + const char *slash; + enum fsmonitor_path_type t; + + strbuf_reset(&path); + if (normalize_path_in_utf8(info, &path) == -1) + goto skip_this_path; + + t = fsmonitor_classify_path_gitdir_relative(path.buf); + + switch (t) { + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path.buf); + string_list_append(&cookie_list, + slash ? slash + 1 : path.buf); + break; + + case IS_INSIDE_GITDIR: + goto skip_this_path; + + default: + BUG("unexpected path classification '%d' for '%s'", + t, path.buf); + } + +skip_this_path: + if (!info->NextEntryOffset) + break; + p += info->NextEntryOffset; + } + + fsmonitor_publish(state, NULL, &cookie_list); + string_list_clear(&cookie_list, 0); + strbuf_release(&path); + return LISTENER_HAVE_DATA_GITDIR; } void fsm_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data = state->backend_data; + DWORD dwWait; + int result; + + state->error_code = 0; + + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + + if (data->watch_gitdir && + start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + + for (;;) { + dwWait = WaitForMultipleObjects(data->nr_listener_handles, + data->hListener, + FALSE, INFINITE); + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_WORKTREE) { + result = recv_rdcw_watch(data->watch_worktree); + if (result == -1) { + /* hard error */ + goto force_error_stop; + } + if (result == -2) { + /* retryable error */ + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + continue; + } + + /* have data */ + if (process_worktree_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_worktree) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_HAVE_DATA_GITDIR) { + result = recv_rdcw_watch(data->watch_gitdir); + if (result == -1) { + /* hard error */ + goto force_error_stop; + } + if (result == -2) { + /* retryable error */ + if (start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + continue; + } + + /* have data */ + if (process_gitdir_events(state) == LISTENER_SHUTDOWN) + goto force_shutdown; + if (start_rdcw_watch(data, data->watch_gitdir) == -1) + goto force_error_stop; + continue; + } + + if (dwWait == WAIT_OBJECT_0 + LISTENER_SHUTDOWN) + goto clean_shutdown; + + error(_("could not read directory changes [GLE %ld]"), + GetLastError()); + goto force_error_stop; + } + +force_error_stop: + state->error_code = -1; + +force_shutdown: + /* + * Tell the IPC thead pool to stop (which completes the await + * in the main thread (which will also signal this thread (if + * we are still alive))). + */ + ipc_server_stop_async(state->ipc_server_data); + +clean_shutdown: + cancel_rdcw_watch(data->watch_worktree); + cancel_rdcw_watch(data->watch_gitdir); } int fsm_listen__ctor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + CALLOC_ARRAY(data, 1); + + data->hEventShutdown = CreateEvent(NULL, TRUE, FALSE, NULL); + + data->watch_worktree = create_watch(state, + state->path_worktree_watch.buf); + if (!data->watch_worktree) + goto failed; + + if (state->nr_paths_watching > 1) { + data->watch_gitdir = create_watch(state, + state->path_gitdir_watch.buf); + if (!data->watch_gitdir) + goto failed; + } + + data->hListener[LISTENER_SHUTDOWN] = data->hEventShutdown; + data->nr_listener_handles++; + + data->hListener[LISTENER_HAVE_DATA_WORKTREE] = + data->watch_worktree->hEvent; + data->nr_listener_handles++; + + if (data->watch_gitdir) { + data->hListener[LISTENER_HAVE_DATA_GITDIR] = + data->watch_gitdir->hEvent; + data->nr_listener_handles++; + } + + state->backend_data = data; + return 0; + +failed: + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + return -1; } void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + CloseHandle(data->hEventShutdown); + destroy_watch(data->watch_worktree); + destroy_watch(data->watch_gitdir); + + FREE_AND_NULL(state->backend_data); } From patchwork Fri Mar 25 18:02:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791958 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C8AF3C433F5 for ; Fri, 25 Mar 2022 19:32:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230258AbiCYTeA (ORCPT ); Fri, 25 Mar 2022 15:34:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50506 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231691AbiCYTdV (ORCPT ); Fri, 25 Mar 2022 15:33:21 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0DB48220B02 for ; Fri, 25 Mar 2022 12:08:14 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id w21so7527469wra.2 for ; Fri, 25 Mar 2022 12:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fAa8hrpVrno1BmdEVVVe0xU+KaqvnFCI1tVOqmc3Cac=; b=ZP0dmdaSysFUOYlITRzQQ3grQA71OffRCR0DWQm+snSfGp/jfIPMveMTcPiwDZohfO rVYW3aGtNofFJ/xuzgU6EYKaJyKmu1+boq56HZEmawcymysRUI6ZWbHPBj48ws3kb1xE ppkLz1pmRA3v65w73wGu047BhMS+F8CvAOcf61BQCXD6dLli5FjEu3w6F4uvjaqRSafk ReuTiSa+R2VMqC4OIX+yMA+BgQR8+8aGNIv0x03aVajMLPfajHZwlyahoQdUyYl9dNZG 9bO63/WY6VRKwQ4wEIQI6zbLRHd3qk5cJkRqS5RtZAsoLpgcuoX0ReFCM9MhJxgHaG0y aBAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fAa8hrpVrno1BmdEVVVe0xU+KaqvnFCI1tVOqmc3Cac=; b=6aLJflZ1cpF+/yIOc8vO3UjfhbshDC3duBCjFTmjuaVWdtfJaCFlPn1LWaI0+ItKHm u/HVarkemo64QBo0AGWdhgkYh9NBlnXhaJAkIeCRdxyr2owDUC1d/NLDevdMsHl6rXIG B28Rj0MKJhA7/q0e5UEWlJG2dKXYTiQOh45x7Z/zAM5cbnIrFA2L746ElFxoetKUPNZc vDMdcLkrYfnRCWE7I2ZJeqxTDUmeHRc/JtpsDx6WwnwbPFxq65ixJRrlClLkVYlbPF5w /eD7pyD++liTIbAaHyiUsAO/89+xYM9i2JRypqPZagshoH5bSmiWGGI7w/0SnSj5sjME EpMw== X-Gm-Message-State: AOAM533g/jpJ1KM41zpcyJMvyNbkowrUSfTadudRQHyrBy4Qu7x8a7pN bM8aqkh9lSjw9PSyYhIm/d4kJGgnAzw= X-Google-Smtp-Source: ABdhPJzwbUflr3nQfE22NY1ON+BDDAaDrPFxfPK9A7Ap0p6qqEBGHTV6QFNJi5EHocArsHUBjkA8bw== X-Received: by 2002:a5d:59ad:0:b0:205:a66c:3c2a with SMTP id p13-20020a5d59ad000000b00205a66c3c2amr3663577wrr.289.1648231412640; Fri, 25 Mar 2022 11:03:32 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q14-20020a1cf30e000000b0038986a18ec8sm4876616wmq.46.2022.03.25.11.03.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:31 -0700 (PDT) Message-Id: <14b775e9d8b1a4672f8175a546eb70e2790c1b23.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:02:59 +0000 Subject: [PATCH v9 16/30] compat/fsmonitor/fsm-listen-darwin: add MacOS header files for FSEvent Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Include MacOS system declarations to allow us to use FSEvent and CoreFoundation APIs. We need different versions of the declarations for GCC vs. clang because of compiler and header file conflicts. While it is quite possible to #include Apple's CoreServices.h when compiling C source code with clang, trying to build it with GCC currently fails with this error: In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/Security.framework/Headers/AuthSession.h:32, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/Security.framework/Headers/Security.h:42, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/CoreServices.framework/Frameworks/... ...OSServices.framework/Headers/CSIdentity.h:43, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/CoreServices.framework/Frameworks/... ...OSServices.framework/Headers/OSServices.h:29, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/CoreServices.framework/Frameworks/... ...LaunchServices.framework/Headers/IconsCore.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/CoreServices.framework/Frameworks/... ...LaunchServices.framework/Headers/LaunchServices.h:23, from /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/CoreServices.framework/Headers/CoreServices.h:45, /Library/Developer/CommandLineTools/SDKs/MacOSX10.14.sdk/System/... ...Library/Frameworks/Security.framework/Headers/Authorization.h:193:7: error: variably modified 'bytes' at file scope 193 | char bytes[kAuthorizationExternalFormLength]; | ^~~~~ The underlying reason is that GCC (rightfully) objects that an `enum` value such as `kAuthorizationExternalFormLength` is not a constant (because it is not, the preprocessor has no knowledge of it, only the actual C compiler does) and can therefore not be used to define the size of a C array. This is a known problem and tracked in GCC's bug tracker: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 In the meantime, let's not block things and go the slightly ugly route of declaring/defining the FSEvents constants, data structures and functions that we need, so that we can avoid above-mentioned issue. Let's do this _only_ for GCC, though, so that the CI/PR builds (which build both with clang and with GCC) can guarantee that we _are_ using the correct data types. Signed-off-by: Johannes Schindelin Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- compat/fsmonitor/fsm-darwin-gcc.h | 92 ++++++++++++++++++++++++++++ compat/fsmonitor/fsm-listen-darwin.c | 24 ++++++++ 2 files changed, 116 insertions(+) create mode 100644 compat/fsmonitor/fsm-darwin-gcc.h diff --git a/compat/fsmonitor/fsm-darwin-gcc.h b/compat/fsmonitor/fsm-darwin-gcc.h new file mode 100644 index 00000000000..1c75c3d48e7 --- /dev/null +++ b/compat/fsmonitor/fsm-darwin-gcc.h @@ -0,0 +1,92 @@ +#ifndef FSM_DARWIN_GCC_H +#define FSM_DARWIN_GCC_H + +#ifndef __clang__ +/* + * It is possible to #include CoreFoundation/CoreFoundation.h when compiling + * with clang, but not with GCC as of time of writing. + * + * See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=93082 for details. + */ +typedef unsigned int FSEventStreamCreateFlags; +#define kFSEventStreamEventFlagNone 0x00000000 +#define kFSEventStreamEventFlagMustScanSubDirs 0x00000001 +#define kFSEventStreamEventFlagUserDropped 0x00000002 +#define kFSEventStreamEventFlagKernelDropped 0x00000004 +#define kFSEventStreamEventFlagEventIdsWrapped 0x00000008 +#define kFSEventStreamEventFlagHistoryDone 0x00000010 +#define kFSEventStreamEventFlagRootChanged 0x00000020 +#define kFSEventStreamEventFlagMount 0x00000040 +#define kFSEventStreamEventFlagUnmount 0x00000080 +#define kFSEventStreamEventFlagItemCreated 0x00000100 +#define kFSEventStreamEventFlagItemRemoved 0x00000200 +#define kFSEventStreamEventFlagItemInodeMetaMod 0x00000400 +#define kFSEventStreamEventFlagItemRenamed 0x00000800 +#define kFSEventStreamEventFlagItemModified 0x00001000 +#define kFSEventStreamEventFlagItemFinderInfoMod 0x00002000 +#define kFSEventStreamEventFlagItemChangeOwner 0x00004000 +#define kFSEventStreamEventFlagItemXattrMod 0x00008000 +#define kFSEventStreamEventFlagItemIsFile 0x00010000 +#define kFSEventStreamEventFlagItemIsDir 0x00020000 +#define kFSEventStreamEventFlagItemIsSymlink 0x00040000 +#define kFSEventStreamEventFlagOwnEvent 0x00080000 +#define kFSEventStreamEventFlagItemIsHardlink 0x00100000 +#define kFSEventStreamEventFlagItemIsLastHardlink 0x00200000 +#define kFSEventStreamEventFlagItemCloned 0x00400000 + +typedef struct __FSEventStream *FSEventStreamRef; +typedef const FSEventStreamRef ConstFSEventStreamRef; + +typedef unsigned int CFStringEncoding; +#define kCFStringEncodingUTF8 0x08000100 + +typedef const struct __CFString *CFStringRef; +typedef const struct __CFArray *CFArrayRef; +typedef const struct __CFRunLoop *CFRunLoopRef; + +struct FSEventStreamContext { + long long version; + void *cb_data, *retain, *release, *copy_description; +}; + +typedef struct FSEventStreamContext FSEventStreamContext; +typedef unsigned int FSEventStreamEventFlags; +#define kFSEventStreamCreateFlagNoDefer 0x02 +#define kFSEventStreamCreateFlagWatchRoot 0x04 +#define kFSEventStreamCreateFlagFileEvents 0x10 + +typedef unsigned long long FSEventStreamEventId; +#define kFSEventStreamEventIdSinceNow 0xFFFFFFFFFFFFFFFFULL + +typedef void (*FSEventStreamCallback)(ConstFSEventStreamRef streamRef, + void *context, + __SIZE_TYPE__ num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]); +typedef double CFTimeInterval; +FSEventStreamRef FSEventStreamCreate(void *allocator, + FSEventStreamCallback callback, + FSEventStreamContext *context, + CFArrayRef paths_to_watch, + FSEventStreamEventId since_when, + CFTimeInterval latency, + FSEventStreamCreateFlags flags); +CFStringRef CFStringCreateWithCString(void *allocator, const char *string, + CFStringEncoding encoding); +CFArrayRef CFArrayCreate(void *allocator, const void **items, long long count, + void *callbacks); +void CFRunLoopRun(void); +void CFRunLoopStop(CFRunLoopRef run_loop); +CFRunLoopRef CFRunLoopGetCurrent(void); +extern CFStringRef kCFRunLoopDefaultMode; +void FSEventStreamScheduleWithRunLoop(FSEventStreamRef stream, + CFRunLoopRef run_loop, + CFStringRef run_loop_mode); +unsigned char FSEventStreamStart(FSEventStreamRef stream); +void FSEventStreamStop(FSEventStreamRef stream); +void FSEventStreamInvalidate(FSEventStreamRef stream); +void FSEventStreamRelease(FSEventStreamRef stream); + +#endif /* !clang */ +#endif /* FSM_DARWIN_GCC_H */ diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index c84e3344ab9..d2ce942cade 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -1,3 +1,27 @@ +#ifndef __clang__ +#include "fsm-darwin-gcc.h" +#else +#include +#include + +#ifndef AVAILABLE_MAC_OS_X_VERSION_10_13_AND_LATER +/* + * This enum value was added in 10.13 to: + * + * /Applications/Xcode.app/Contents/Developer/Platforms/ \ + * MacOSX.platform/Developer/SDKs/MacOSX.sdk/System/ \ + * Library/Frameworks/CoreServices.framework/Frameworks/ \ + * FSEvents.framework/Versions/Current/Headers/FSEvents.h + * + * If we're compiling against an older SDK, this symbol won't be + * present. Silently define it here so that we don't have to ifdef + * the logging or masking below. This should be harmless since older + * versions of macOS won't ever emit this FS event anyway. + */ +#define kFSEventStreamEventFlagItemCloned 0x00400000 +#endif +#endif + #include "cache.h" #include "fsmonitor.h" #include "fsm-listen.h" From patchwork Fri Mar 25 18:03:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1EA7C433EF for ; Fri, 25 Mar 2022 19:29:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229954AbiCYTal (ORCPT ); Fri, 25 Mar 2022 15:30:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:56882 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229732AbiCYTaQ (ORCPT ); Fri, 25 Mar 2022 15:30:16 -0400 Received: from mail-ed1-x52a.google.com (mail-ed1-x52a.google.com [IPv6:2a00:1450:4864:20::52a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B303B1F37B7 for ; Fri, 25 Mar 2022 12:05:17 -0700 (PDT) Received: by mail-ed1-x52a.google.com with SMTP id z92so10252909ede.13 for ; Fri, 25 Mar 2022 12:05:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2p4/vhqS6tNu14et/Mxk5tLaAWlRCUTexkK7bM5otC8=; b=lL6xIaFY1qjffFxUiY7hk2Lj5ZmSj+/nsHP1Sf18f8RhZJFFMZbLADkaQyce3lKDT5 IzRqecn0Ge1+eqiBRD/oDLTs1SnZ2wpdnTwFqU85QoqUcs3D+B+fJdLowYv0+QZKjaF/ Hf7igruWGUY9xz77Xe3iCtLuYp8JjVyGgffCfuhpsbKgaU6s1u8lEIfWj9N0md2JqY3/ ifkvVP7pDXofiYZunDfJ1dBqi8uJ+gymD0ohJqNDwPeuo73wNFcuTqmu8RTynuS9Tbkh 6rb1rJ84HBZkKj8un60xHb7WntFHax/31NJUegRrEOKMwHP/L5YKywkZBF9d3B6+z9pj zTLA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2p4/vhqS6tNu14et/Mxk5tLaAWlRCUTexkK7bM5otC8=; b=F3W8+0sLpsGsMYxwiW5eQ4NVTQNOSXmogPdz/CXvOvT24ZoVRTtGYTqJx7mPg9TiTd Yw5ouCCFO4Y8AMmPRwjyXgQmw1KyThBZ+KNsuFjsJERkDKt8pLTRl/4AJx5DDZmadl6h jKH/5yhakPeL/k34QF9bQqjZ5/8bV2sbhQmxEwv4agOxUL/KFpMcov10e6nyAeBsseMR 9mCmFKHixxlB6nImZ/7VAEAZUhipEKqTuGE1G5DVqd6c3YI45mV1TVTmhsrSY20P+dxi Mehm1WooaEfgyeLm6Q0TikYqUK0dEllWj7xjx3+QGIEveWk919EMEW+kH5+6S1sF7+HG cF0w== X-Gm-Message-State: AOAM530s1MY9bqIZvTWUakQvrMk/bB/A3dtHs+R7ewjpXOLYQBpOeopM 7LzGbzjxm/thkm5BYcO7AVjpV8eE6F8= X-Google-Smtp-Source: ABdhPJw3i3EmGSlk90HBEgLbRLo4DfEp/DSUNvxovSQxfPrU6oWXDettiz22kGH/Br23DIulO6KRXg== X-Received: by 2002:a05:6000:1847:b0:204:1928:166a with SMTP id c7-20020a056000184700b002041928166amr10239052wri.122.1648231413906; Fri, 25 Mar 2022 11:03:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t6-20020a05600c198600b0038cafe3d47dsm5733405wmq.42.2022.03.25.11.03.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:33 -0700 (PDT) Message-Id: <55bd7aee06ca04e6ae9010700103a261d1255113.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:00 +0000 Subject: [PATCH v9 17/30] compat/fsmonitor/fsm-listen-darwin: implement FSEvent listener on MacOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Implement file system event listener on MacOS using FSEvent, CoreFoundation, and CoreServices. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- compat/fsmonitor/fsm-listen-darwin.c | 383 +++++++++++++++++++++++++++ 1 file changed, 383 insertions(+) diff --git a/compat/fsmonitor/fsm-listen-darwin.c b/compat/fsmonitor/fsm-listen-darwin.c index d2ce942cade..0741fe834c3 100644 --- a/compat/fsmonitor/fsm-listen-darwin.c +++ b/compat/fsmonitor/fsm-listen-darwin.c @@ -25,20 +25,403 @@ #include "cache.h" #include "fsmonitor.h" #include "fsm-listen.h" +#include "fsmonitor--daemon.h" + +struct fsmonitor_daemon_backend_data +{ + CFStringRef cfsr_worktree_path; + CFStringRef cfsr_gitdir_path; + + CFArrayRef cfar_paths_to_watch; + int nr_paths_watching; + + FSEventStreamRef stream; + + CFRunLoopRef rl; + + enum shutdown_style { + SHUTDOWN_EVENT = 0, + FORCE_SHUTDOWN, + FORCE_ERROR_STOP, + } shutdown_style; + + unsigned int stream_scheduled:1; + unsigned int stream_started:1; +}; + +static void log_flags_set(const char *path, const FSEventStreamEventFlags flag) +{ + struct strbuf msg = STRBUF_INIT; + + if (flag & kFSEventStreamEventFlagMustScanSubDirs) + strbuf_addstr(&msg, "MustScanSubDirs|"); + if (flag & kFSEventStreamEventFlagUserDropped) + strbuf_addstr(&msg, "UserDropped|"); + if (flag & kFSEventStreamEventFlagKernelDropped) + strbuf_addstr(&msg, "KernelDropped|"); + if (flag & kFSEventStreamEventFlagEventIdsWrapped) + strbuf_addstr(&msg, "EventIdsWrapped|"); + if (flag & kFSEventStreamEventFlagHistoryDone) + strbuf_addstr(&msg, "HistoryDone|"); + if (flag & kFSEventStreamEventFlagRootChanged) + strbuf_addstr(&msg, "RootChanged|"); + if (flag & kFSEventStreamEventFlagMount) + strbuf_addstr(&msg, "Mount|"); + if (flag & kFSEventStreamEventFlagUnmount) + strbuf_addstr(&msg, "Unmount|"); + if (flag & kFSEventStreamEventFlagItemChangeOwner) + strbuf_addstr(&msg, "ItemChangeOwner|"); + if (flag & kFSEventStreamEventFlagItemCreated) + strbuf_addstr(&msg, "ItemCreated|"); + if (flag & kFSEventStreamEventFlagItemFinderInfoMod) + strbuf_addstr(&msg, "ItemFinderInfoMod|"); + if (flag & kFSEventStreamEventFlagItemInodeMetaMod) + strbuf_addstr(&msg, "ItemInodeMetaMod|"); + if (flag & kFSEventStreamEventFlagItemIsDir) + strbuf_addstr(&msg, "ItemIsDir|"); + if (flag & kFSEventStreamEventFlagItemIsFile) + strbuf_addstr(&msg, "ItemIsFile|"); + if (flag & kFSEventStreamEventFlagItemIsHardlink) + strbuf_addstr(&msg, "ItemIsHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsLastHardlink) + strbuf_addstr(&msg, "ItemIsLastHardlink|"); + if (flag & kFSEventStreamEventFlagItemIsSymlink) + strbuf_addstr(&msg, "ItemIsSymlink|"); + if (flag & kFSEventStreamEventFlagItemModified) + strbuf_addstr(&msg, "ItemModified|"); + if (flag & kFSEventStreamEventFlagItemRemoved) + strbuf_addstr(&msg, "ItemRemoved|"); + if (flag & kFSEventStreamEventFlagItemRenamed) + strbuf_addstr(&msg, "ItemRenamed|"); + if (flag & kFSEventStreamEventFlagItemXattrMod) + strbuf_addstr(&msg, "ItemXattrMod|"); + if (flag & kFSEventStreamEventFlagOwnEvent) + strbuf_addstr(&msg, "OwnEvent|"); + if (flag & kFSEventStreamEventFlagItemCloned) + strbuf_addstr(&msg, "ItemCloned|"); + + trace_printf_key(&trace_fsmonitor, "fsevent: '%s', flags=%u %s", + path, flag, msg.buf); + + strbuf_release(&msg); +} + +static int ef_is_root_delete(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRemoved); +} + +static int ef_is_root_renamed(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagItemIsDir && + ef & kFSEventStreamEventFlagItemRenamed); +} + +static int ef_is_dropped(const FSEventStreamEventFlags ef) +{ + return (ef & kFSEventStreamEventFlagMustScanSubDirs || + ef & kFSEventStreamEventFlagKernelDropped || + ef & kFSEventStreamEventFlagUserDropped); +} + +static void fsevent_callback(ConstFSEventStreamRef streamRef, + void *ctx, + size_t num_of_events, + void *event_paths, + const FSEventStreamEventFlags event_flags[], + const FSEventStreamEventId event_ids[]) +{ + struct fsmonitor_daemon_state *state = ctx; + struct fsmonitor_daemon_backend_data *data = state->backend_data; + char **paths = (char **)event_paths; + struct fsmonitor_batch *batch = NULL; + struct string_list cookie_list = STRING_LIST_INIT_DUP; + const char *path_k; + const char *slash; + int k; + struct strbuf tmp = STRBUF_INIT; + + /* + * Build a list of all filesystem changes into a private/local + * list and without holding any locks. + */ + for (k = 0; k < num_of_events; k++) { + /* + * On Mac, we receive an array of absolute paths. + */ + path_k = paths[k]; + + /* + * If you want to debug FSEvents, log them to GIT_TRACE_FSMONITOR. + * Please don't log them to Trace2. + * + * trace_printf_key(&trace_fsmonitor, "Path: '%s'", path_k); + */ + + /* + * If event[k] is marked as dropped, we assume that we have + * lost sync with the filesystem and should flush our cached + * data. We need to: + * + * [1] Abort/wake any client threads waiting for a cookie and + * flush the cached state data (the current token), and + * create a new token. + * + * [2] Discard the batch that we were locally building (since + * they are conceptually relative to the just flushed + * token). + */ + if (ef_is_dropped(event_flags[k])) { + if (trace_pass_fl(&trace_fsmonitor)) + log_flags_set(path_k, event_flags[k]); + + fsmonitor_force_resync(state); + fsmonitor_batch__free_list(batch); + string_list_clear(&cookie_list, 0); + + /* + * We assume that any events that we received + * in this callback after this dropped event + * may still be valid, so we continue rather + * than break. (And just in case there is a + * delete of ".git" hiding in there.) + */ + continue; + } + + switch (fsmonitor_classify_path_absolute(state, path_k)) { + + case IS_INSIDE_DOT_GIT_WITH_COOKIE_PREFIX: + case IS_INSIDE_GITDIR_WITH_COOKIE_PREFIX: + /* special case cookie files within .git or gitdir */ + + /* Use just the filename of the cookie file. */ + slash = find_last_dir_sep(path_k); + string_list_append(&cookie_list, + slash ? slash + 1 : path_k); + break; + + case IS_INSIDE_DOT_GIT: + case IS_INSIDE_GITDIR: + /* ignore all other paths inside of .git or gitdir */ + break; + + case IS_DOT_GIT: + case IS_GITDIR: + /* + * If .git directory is deleted or renamed away, + * we have to quit. + */ + if (ef_is_root_delete(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "event: gitdir removed"); + goto force_shutdown; + } + if (ef_is_root_renamed(event_flags[k])) { + trace_printf_key(&trace_fsmonitor, + "event: gitdir renamed"); + goto force_shutdown; + } + break; + + case IS_WORKDIR_PATH: + /* try to queue normal pathnames */ + + if (trace_pass_fl(&trace_fsmonitor)) + log_flags_set(path_k, event_flags[k]); + + /* + * Because of the implicit "binning" (the + * kernel calls us at a given frequency) and + * de-duping (the kernel is free to combine + * multiple events for a given pathname), an + * individual fsevent could be marked as both + * a file and directory. Add it to the queue + * with both spellings so that the client will + * know how much to invalidate/refresh. + */ + + if (event_flags[k] & kFSEventStreamEventFlagItemIsFile) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, rel); + } + + if (event_flags[k] & kFSEventStreamEventFlagItemIsDir) { + const char *rel = path_k + + state->path_worktree_watch.len + 1; + + strbuf_reset(&tmp); + strbuf_addstr(&tmp, rel); + strbuf_addch(&tmp, '/'); + + if (!batch) + batch = fsmonitor_batch__new(); + fsmonitor_batch__add_path(batch, tmp.buf); + } + + break; + + case IS_OUTSIDE_CONE: + default: + trace_printf_key(&trace_fsmonitor, + "ignoring '%s'", path_k); + break; + } + } + + fsmonitor_publish(state, batch, &cookie_list); + string_list_clear(&cookie_list, 0); + strbuf_release(&tmp); + return; + +force_shutdown: + fsmonitor_batch__free_list(batch); + string_list_clear(&cookie_list, 0); + + data->shutdown_style = FORCE_SHUTDOWN; + CFRunLoopStop(data->rl); + strbuf_release(&tmp); + return; +} + +/* + * In the call to `FSEventStreamCreate()` to setup our watch, the + * `latency` argument determines the frequency of calls to our callback + * with new FS events. Too slow and events get dropped; too fast and + * we burn CPU unnecessarily. Since it is rather obscure, I don't + * think this needs to be a config setting. I've done extensive + * testing on my systems and chosen the value below. It gives good + * results and I've not seen any dropped events. + * + * With a latency of 0.1, I was seeing lots of dropped events during + * the "touch 100000" files test within t/perf/p7519, but with a + * latency of 0.001 I did not see any dropped events. So I'm going + * to assume that this is the "correct" value. + * + * https://developer.apple.com/documentation/coreservices/1443980-fseventstreamcreate + */ int fsm_listen__ctor(struct fsmonitor_daemon_state *state) { + FSEventStreamCreateFlags flags = kFSEventStreamCreateFlagNoDefer | + kFSEventStreamCreateFlagWatchRoot | + kFSEventStreamCreateFlagFileEvents; + FSEventStreamContext ctx = { + 0, + state, + NULL, + NULL, + NULL + }; + struct fsmonitor_daemon_backend_data *data; + const void *dir_array[2]; + + CALLOC_ARRAY(data, 1); + state->backend_data = data; + + data->cfsr_worktree_path = CFStringCreateWithCString( + NULL, state->path_worktree_watch.buf, kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_worktree_path; + + if (state->nr_paths_watching > 1) { + data->cfsr_gitdir_path = CFStringCreateWithCString( + NULL, state->path_gitdir_watch.buf, + kCFStringEncodingUTF8); + dir_array[data->nr_paths_watching++] = data->cfsr_gitdir_path; + } + + data->cfar_paths_to_watch = CFArrayCreate(NULL, dir_array, + data->nr_paths_watching, + NULL); + data->stream = FSEventStreamCreate(NULL, fsevent_callback, &ctx, + data->cfar_paths_to_watch, + kFSEventStreamEventIdSinceNow, + 0.001, flags); + if (data->stream == NULL) + goto failed; + + /* + * `data->rl` needs to be set inside the listener thread. + */ + + return 0; + +failed: + error(_("Unable to create FSEventStream.")); + + FREE_AND_NULL(state->backend_data); return -1; } void fsm_listen__dtor(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + if (!state || !state->backend_data) + return; + + data = state->backend_data; + + if (data->stream) { + if (data->stream_started) + FSEventStreamStop(data->stream); + if (data->stream_scheduled) + FSEventStreamInvalidate(data->stream); + FSEventStreamRelease(data->stream); + } + + FREE_AND_NULL(state->backend_data); } void fsm_listen__stop_async(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + data->shutdown_style = SHUTDOWN_EVENT; + + CFRunLoopStop(data->rl); } void fsm_listen__loop(struct fsmonitor_daemon_state *state) { + struct fsmonitor_daemon_backend_data *data; + + data = state->backend_data; + + data->rl = CFRunLoopGetCurrent(); + + FSEventStreamScheduleWithRunLoop(data->stream, data->rl, kCFRunLoopDefaultMode); + data->stream_scheduled = 1; + + if (!FSEventStreamStart(data->stream)) { + error(_("Failed to start the FSEventStream")); + goto force_error_stop_without_loop; + } + data->stream_started = 1; + + CFRunLoopRun(); + + switch (data->shutdown_style) { + case FORCE_ERROR_STOP: + state->error_code = -1; + /* fall thru */ + case FORCE_SHUTDOWN: + ipc_server_stop_async(state->ipc_server_data); + /* fall thru */ + case SHUTDOWN_EVENT: + default: + break; + } + return; + +force_error_stop_without_loop: + state->error_code = -1; + ipc_server_stop_async(state->ipc_server_data); + return; } From patchwork Fri Mar 25 18:03:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791926 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA957C4332F for ; Fri, 25 Mar 2022 19:26:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229766AbiCYT2J (ORCPT ); Fri, 25 Mar 2022 15:28:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58096 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229769AbiCYT1s (ORCPT ); Fri, 25 Mar 2022 15:27:48 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CD451C4063 for ; Fri, 25 Mar 2022 12:01:16 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id w21so7507250wra.2 for ; Fri, 25 Mar 2022 12:01:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=p0teLAsHTxbSC0+vtDh7IakcwEoThRbRMdCYsFOPVt4=; b=ViR8XaGfsSMTgPUkDGDjQu6F+kR9jghQew65bpG2OMf8IlMewqWGgK6Pm3xPsIJ9+Y tt0W5QBGBN0Vw+3unuT975OtHLwverMpAEMP5IW7vve5DIpGxCLe8QLezy2nD1gGTTbH aBeS9M2nYzCLp9yZpw3DeM4JupHGT3oS8cwruzHXCDNFdiSLY+Dnh/j0rCZexyBlTkS1 YuD6l9j83qLP6U5u79goyQq4QS/+cfPvyl6axIHgNdPFrLjoLpKsphxjVaz9lyRd25r+ Ebzn8GGusatpI+e0z8EBhnTXgZcqVa4LcGXZnV6EGI7svYUTrkbNocEyNugFWzADKHUl Htjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=p0teLAsHTxbSC0+vtDh7IakcwEoThRbRMdCYsFOPVt4=; b=16Gu5h0c6QuyD7UmP9WtZMAMuZ2ovgcxnFvf95PuXL+YHc4aWwKbcc0jMtT5cBUGlI 7gV3U6WW1MRmFU8D8yptT0WuRrYbcd6rO0HaipvHlBAxpZvCpsbJML3g+j13fD2wDjbX nhi0VXB+LFSfANwZgcwwMoBVsqJYdeF9vArsrBlXWcE8y8iA8hfEVIMSvQlEyShtLwE6 UvobdSSX+helCU1eg5970rp1ZHW90jgDrgKyfZUFHCgkHA0dp6MxRDIIwZThk3uVzwLx grgwJ6pTetfb4P5R/m1K1EiGzP9IyPp9JS2xOmwjqjmiPs5YfHSaeiUIyNCv09U9hm+g xgaw== X-Gm-Message-State: AOAM531GUo9THKJw3lyclvPkJL8SrpFdK+VWsg9kj9U4co8NXMfAw6BG p4wpRUn9E7TD1E93AYNR9dXXvNnOmlg= X-Google-Smtp-Source: ABdhPJwtiNTI5jqpIqdtT2DlE+2jhx7UasnJdEoM7vxR0JbUi7obZaaIWK4oFLHfK2msZE6BYIU/XQ== X-Received: by 2002:a5d:51c5:0:b0:203:decf:8fbf with SMTP id n5-20020a5d51c5000000b00203decf8fbfmr10134531wrv.440.1648231414872; Fri, 25 Mar 2022 11:03:34 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l126-20020a1c2584000000b00387d4f35651sm9215312wml.10.2022.03.25.11.03.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:34 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:01 +0000 Subject: [PATCH v9 18/30] fsmonitor--daemon: implement handle_client callback Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to respond to IPC requests from client Git processes and respond with a list of modified pathnames relative to the provided token. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 311 +++++++++++++++++++++++++++++++++++- 1 file changed, 309 insertions(+), 2 deletions(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index 69312119b07..eafaafb45b1 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -7,6 +7,7 @@ #include "fsmonitor--daemon.h" #include "simple-ipc.h" #include "khash.h" +#include "pkt-line.h" static const char * const builtin_fsmonitor__daemon_usage[] = { N_("git fsmonitor--daemon start []"), @@ -364,6 +365,310 @@ void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) pthread_mutex_unlock(&state->main_lock); } +/* + * Format an opaque token string to send to the client. + */ +static void with_lock__format_response_token( + struct strbuf *response_token, + const struct strbuf *response_token_id, + const struct fsmonitor_batch *batch) +{ + /* assert current thread holding state->main_lock */ + + strbuf_reset(response_token); + strbuf_addf(response_token, "builtin:%s:%"PRIu64, + response_token_id->buf, batch->batch_seq_nr); +} + +/* + * Parse an opaque token from the client. + * Returns -1 on error. + */ +static int fsmonitor_parse_client_token(const char *buf_token, + struct strbuf *requested_token_id, + uint64_t *seq_nr) +{ + const char *p; + char *p_end; + + strbuf_reset(requested_token_id); + *seq_nr = 0; + + if (!skip_prefix(buf_token, "builtin:", &p)) + return -1; + + while (*p && *p != ':') + strbuf_addch(requested_token_id, *p++); + if (!*p++) + return -1; + + *seq_nr = (uint64_t)strtoumax(p, &p_end, 10); + if (*p_end) + return -1; + + return 0; +} + +KHASH_INIT(str, const char *, int, 0, kh_str_hash_func, kh_str_hash_equal) + +static int do_handle_client(struct fsmonitor_daemon_state *state, + const char *command, + ipc_server_reply_cb *reply, + struct ipc_server_reply_data *reply_data) +{ + struct fsmonitor_token_data *token_data = NULL; + struct strbuf response_token = STRBUF_INIT; + struct strbuf requested_token_id = STRBUF_INIT; + struct strbuf payload = STRBUF_INIT; + uint64_t requested_oldest_seq_nr = 0; + uint64_t total_response_len = 0; + const char *p; + const struct fsmonitor_batch *batch_head; + const struct fsmonitor_batch *batch; + intmax_t count = 0, duplicates = 0; + kh_str_t *shown; + int hash_ret; + int do_trivial = 0; + int do_flush = 0; + + /* + * We expect `command` to be of the form: + * + * := quit NUL + * | flush NUL + * | NUL + * | NUL + */ + + if (!strcmp(command, "quit")) { + /* + * A client has requested over the socket/pipe that the + * daemon shutdown. + * + * Tell the IPC thread pool to shutdown (which completes + * the await in the main thread (which can stop the + * fsmonitor listener thread)). + * + * There is no reply to the client. + */ + return SIMPLE_IPC_QUIT; + + } else if (!strcmp(command, "flush")) { + /* + * Flush all of our cached data and generate a new token + * just like if we lost sync with the filesystem. + * + * Then send a trivial response using the new token. + */ + do_flush = 1; + do_trivial = 1; + + } else if (!skip_prefix(command, "builtin:", &p)) { + /* assume V1 timestamp or garbage */ + + char *p_end; + + strtoumax(command, &p_end, 10); + trace_printf_key(&trace_fsmonitor, + ((*p_end) ? + "fsmonitor: invalid command line '%s'" : + "fsmonitor: unsupported V1 protocol '%s'"), + command); + do_trivial = 1; + + } else { + /* We have "builtin:*" */ + if (fsmonitor_parse_client_token(command, &requested_token_id, + &requested_oldest_seq_nr)) { + trace_printf_key(&trace_fsmonitor, + "fsmonitor: invalid V2 protocol token '%s'", + command); + do_trivial = 1; + + } else { + /* + * We have a V2 valid token: + * "builtin::" + */ + } + } + + pthread_mutex_lock(&state->main_lock); + + if (!state->current_token_data) + BUG("fsmonitor state does not have a current token"); + + if (do_flush) + with_lock__do_force_resync(state); + + /* + * We mark the current head of the batch list as "pinned" so + * that the listener thread will treat this item as read-only + * (and prevent any more paths from being added to it) from + * now on. + */ + token_data = state->current_token_data; + batch_head = token_data->batch_head; + ((struct fsmonitor_batch *)batch_head)->pinned_time = time(NULL); + + /* + * FSMonitor Protocol V2 requires that we send a response header + * with a "new current token" and then all of the paths that changed + * since the "requested token". We send the seq_nr of the just-pinned + * head batch so that future requests from a client will be relative + * to it. + */ + with_lock__format_response_token(&response_token, + &token_data->token_id, batch_head); + + reply(reply_data, response_token.buf, response_token.len + 1); + total_response_len += response_token.len + 1; + + trace2_data_string("fsmonitor", the_repository, "response/token", + response_token.buf); + trace_printf_key(&trace_fsmonitor, "response token: %s", + response_token.buf); + + if (!do_trivial) { + if (strcmp(requested_token_id.buf, token_data->token_id.buf)) { + /* + * The client last spoke to a different daemon + * instance -OR- the daemon had to resync with + * the filesystem (and lost events), so reject. + */ + trace2_data_string("fsmonitor", the_repository, + "response/token", "different"); + do_trivial = 1; + + } else if (requested_oldest_seq_nr < + token_data->batch_tail->batch_seq_nr) { + /* + * The client wants older events than we have for + * this token_id. This means that the end of our + * batch list was truncated and we cannot give the + * client a complete snapshot relative to their + * request. + */ + trace_printf_key(&trace_fsmonitor, + "client requested truncated data"); + do_trivial = 1; + } + } + + if (do_trivial) { + pthread_mutex_unlock(&state->main_lock); + + reply(reply_data, "/", 2); + + trace2_data_intmax("fsmonitor", the_repository, + "response/trivial", 1); + + goto cleanup; + } + + /* + * We're going to hold onto a pointer to the current + * token-data while we walk the list of batches of files. + * During this time, we will NOT be under the lock. + * So we ref-count it. + * + * This allows the listener thread to continue prepending + * new batches of items to the token-data (which we'll ignore). + * + * AND it allows the listener thread to do a token-reset + * (and install a new `current_token_data`). + */ + token_data->client_ref_count++; + + pthread_mutex_unlock(&state->main_lock); + + /* + * The client request is relative to the token that they sent, + * so walk the batch list backwards from the current head back + * to the batch (sequence number) they named. + * + * We use khash to de-dup the list of pathnames. + * + * NEEDSWORK: each batch contains a list of interned strings, + * so we only need to do pointer comparisons here to build the + * hash table. Currently, we're still comparing the string + * values. + */ + shown = kh_init_str(); + for (batch = batch_head; + batch && batch->batch_seq_nr > requested_oldest_seq_nr; + batch = batch->next) { + size_t k; + + for (k = 0; k < batch->nr; k++) { + const char *s = batch->interned_paths[k]; + size_t s_len; + + if (kh_get_str(shown, s) != kh_end(shown)) + duplicates++; + else { + kh_put_str(shown, s, &hash_ret); + + trace_printf_key(&trace_fsmonitor, + "send[%"PRIuMAX"]: %s", + count, s); + + /* Each path gets written with a trailing NUL */ + s_len = strlen(s) + 1; + + if (payload.len + s_len >= + LARGE_PACKET_DATA_MAX) { + reply(reply_data, payload.buf, + payload.len); + total_response_len += payload.len; + strbuf_reset(&payload); + } + + strbuf_add(&payload, s, s_len); + count++; + } + } + } + + if (payload.len) { + reply(reply_data, payload.buf, payload.len); + total_response_len += payload.len; + } + + kh_release_str(shown); + + pthread_mutex_lock(&state->main_lock); + + if (token_data->client_ref_count > 0) + token_data->client_ref_count--; + + if (token_data->client_ref_count == 0) { + if (token_data != state->current_token_data) { + /* + * The listener thread did a token-reset while we were + * walking the batch list. Therefore, this token is + * stale and can be discarded completely. If we are + * the last reader thread using this token, we own + * that work. + */ + fsmonitor_free_token_data(token_data); + } + } + + pthread_mutex_unlock(&state->main_lock); + + trace2_data_intmax("fsmonitor", the_repository, "response/length", total_response_len); + trace2_data_intmax("fsmonitor", the_repository, "response/count/files", count); + trace2_data_intmax("fsmonitor", the_repository, "response/count/duplicates", duplicates); + +cleanup: + strbuf_release(&response_token); + strbuf_release(&requested_token_id); + strbuf_release(&payload); + + return 0; +} + static ipc_server_application_cb handle_client; static int handle_client(void *data, @@ -371,7 +676,7 @@ static int handle_client(void *data, ipc_server_reply_cb *reply, struct ipc_server_reply_data *reply_data) { - /* struct fsmonitor_daemon_state *state = data; */ + struct fsmonitor_daemon_state *state = data; int result; /* @@ -382,10 +687,12 @@ static int handle_client(void *data, if (command_len != strlen(command)) BUG("FSMonitor assumes text messages"); + trace_printf_key(&trace_fsmonitor, "requested token: %s", command); + trace2_region_enter("fsmonitor", "handle_client", the_repository); trace2_data_string("fsmonitor", the_repository, "request", command); - result = 0; /* TODO Do something here. */ + result = do_handle_client(state, command, reply, reply_data); trace2_region_leave("fsmonitor", "handle_client", the_repository); From patchwork Fri Mar 25 18:03:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791999 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99D02C433F5 for ; Fri, 25 Mar 2022 19:39:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231345AbiCYTlK (ORCPT ); Fri, 25 Mar 2022 15:41:10 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231177AbiCYThx (ORCPT ); Fri, 25 Mar 2022 15:37:53 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5B2611FDFEA for ; Fri, 25 Mar 2022 12:10:24 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id a1so12082337wrh.10 for ; Fri, 25 Mar 2022 12:10:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=6IMoORViExxPGOqXM0jiWHBPtgaquTlkmrPOCTiquVM=; b=a03UnjZf1EDHR2ilcfsWF/9iYP/4LOm5hqTOC0MQVRNBkjZR/r45lPQw2LJjHYnNre zMagfivtjPP0GWz8f0VF5HWJq3WgonGaPCyiUmNX3YJOY9jAMOZRQvPy7ivzxNbkfq7T JEc2ECc1OZ6J2Zq2I2VAT4VzSdN5E7EClGMoJvSbGNyTfG67MxpEbDBT3+43I7TGQsQK nPRbmx+9DBCRIR6FKMLxegxHt3AD9ZVELnZkL0yYmrB/EMcfmd4NVEBzbOo6LFkWpwx8 yA6xwjBf/Tazwu0Z3ZYb40wZ0/JHEu9uXb+3gQefwYoNkFKFlv6UYOF7laGGf0sK1vWA I9xQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=6IMoORViExxPGOqXM0jiWHBPtgaquTlkmrPOCTiquVM=; b=KWkLmWEMet8WGeYc+661KtkHNMpdDu2Fg1EN4r+GfiDe7Oog3lEwemFFMpXTMqhY9e yCFuFwefTgdJMFvBYS2l8EdbBC8DgEOAITlcOPW1cmwwbh6lmyEk716m9Wct3HCZseq5 IVSUL018WpgzUj8RYn7Qi3ltEeJPGmZ+uanH/Q01Yzsg+j/O2IyyAOWH1GVeTsp+0FQ2 MGHF4qF3SJxXGEkD3mjbezXFMrucUN3PZupLqJuIEaWJc/+mhA5cEsznTOBDiJ8OR2QD hWneWFq15Sus1CxqtS6Aldj78c+WsXnp/no+YI/2NVgbjNswRklEjEmCj4ybt81c9b29 Pedg== X-Gm-Message-State: AOAM533pac9bbXz/uZUijBNnqu2FkO2QgfqGP6vnP67N3qhfAX/zagrU j0CsQ5axPnJW/KrjqxL3ZZSH1PvfzWU= X-Google-Smtp-Source: ABdhPJzYb/b+jtV9oLWyBt693l7Fojn3Np1Vt7Bg/zzZoA/ppsx1j4z9zj66nCR6GglxkD16FvGODA== X-Received: by 2002:a5d:47cc:0:b0:204:1c9d:2157 with SMTP id o12-20020a5d47cc000000b002041c9d2157mr9896774wrc.294.1648231416166; Fri, 25 Mar 2022 11:03:36 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m3-20020a5d6243000000b001e33760776fsm5574119wrv.10.2022.03.25.11.03.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:35 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:02 +0000 Subject: [PATCH v9 19/30] help: include fsmonitor--daemon feature flag in version info Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Add the "feature: fsmonitor--daemon" message to the output of `git version --build-options`. The builtin FSMonitor is only available on certain platforms and even then only when certain Makefile flags are enabled, so print a message in the verbose version output when it is available. This can be used by test scripts for prereq testing. Granted, tests could just try `git fsmonitor--daemon status` and look for a 128 exit code or grep for a "not supported" message on stderr, but these methods are rather obscure. The main advantage is that the feature message will automatically appear in bug reports and other support requests. This concept was also used during the development of Scalar for similar reasons. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- help.c | 4 ++++ t/test-lib.sh | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/help.c b/help.c index 71444906ddf..9112a51e84b 100644 --- a/help.c +++ b/help.c @@ -12,6 +12,7 @@ #include "refs.h" #include "parse-options.h" #include "prompt.h" +#include "fsmonitor-ipc.h" struct category_description { uint32_t category; @@ -695,6 +696,9 @@ void get_version_info(struct strbuf *buf, int show_build_options) strbuf_addf(buf, "sizeof-size_t: %d\n", (int)sizeof(size_t)); strbuf_addf(buf, "shell-path: %s\n", SHELL_PATH); /* NEEDSWORK: also save and output GIT-BUILD_OPTIONS? */ + + if (fsmonitor_ipc__is_supported()) + strbuf_addstr(buf, "feature: fsmonitor--daemon\n"); } } diff --git a/t/test-lib.sh b/t/test-lib.sh index e4716b0b867..5d819c1bc11 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1799,3 +1799,10 @@ test_lazy_prereq SHA1 ' # Tests that verify the scheduler integration must set this locally # to avoid errors. GIT_TEST_MAINT_SCHEDULER="none:exit 1" + +# Does this platform support `git fsmonitor--daemon` +# +test_lazy_prereq FSMONITOR_DAEMON ' + git version --build-options >output && + grep "feature: fsmonitor--daemon" output +' From patchwork Fri Mar 25 18:03:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791946 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E1C3C433EF for ; Fri, 25 Mar 2022 19:29:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229827AbiCYTay (ORCPT ); Fri, 25 Mar 2022 15:30:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229577AbiCYTah (ORCPT ); Fri, 25 Mar 2022 15:30:37 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 418E61DF87C for ; Fri, 25 Mar 2022 12:05:35 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id r64so4960884wmr.4 for ; Fri, 25 Mar 2022 12:05:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ekpkFZZsg/xn+/xY3K+YVOvurMdWK/1l1/dNkxZdexU=; b=GIo4MBogx8A44rU/Y1Y25lv4Upd2Hilvd68C0AN4Ydu216hrNoM72kbr/eTB65Chfg 03hiCIO+M747ezQvk6esTt9G3RyrxPZSYFEB9y2jGHz3vdnZevcREZXwl2sbigRiCWDH xh83Gyzxhy9cF2ppb+9KTbHuB2LsrMyj7QAntJrrpRvewCR9l4XbiE8WVFc+KizXy809 f0bZUaL3cu38dK+32oe+zphYdaQh1OOsz7prwCTcwynEa5z1rv75CvJRqY3vBrjahWqg 9MysYDp/e9jchVMrgfIWxNAu6XGmySRdh7ECPvzUyQvLS7n7PfKbYjjD1/MRvs8x7CCD RLlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ekpkFZZsg/xn+/xY3K+YVOvurMdWK/1l1/dNkxZdexU=; b=6rMEbm+xLDTUpjuovSPi4PYNKA5RRBcX4GOq25bPTGOqYQKCgWijLrSWOvppXFpVFn G30I2eHYTe75GZlDGG6LkLIDGjtR5ImOEbaAF59rKHq/Ndr2sUVPTJDFzV/ala5nZyJj mnSJ/GGT/UUMYgiTcz3gOw5zOnwRT/ksZpjuTepJHit+OS8FslCZo7WKVo8IQ3QIVVzl kgZcrgIE4ZmasMDG6WtSFu4l/xccflsGK6q0f5iyXYjalMhII+OSxrpK3snylNnEMwPK nDs/hiO/Gcf4FylRZUCm+K9xOfIaRzCZuAj0FuM2fz/aTeHtNtV7TPrhF1MVLWUQDk2t cVMg== X-Gm-Message-State: AOAM532PqBkf5uN6VeMTC2BEp6S9e3fua78ouw9S27e3qqiBCZ3PI56B ZP+rByfygdwapKBPxQr/0X9Ayv/L9RA= X-Google-Smtp-Source: ABdhPJxGn6S2ctcrKY+z8v0B478x3oQVgMN2b8aKiBteoy2DZ7Q7hR/qEmm4/X4yhRUgXXFYNDJmaw== X-Received: by 2002:a05:600c:3509:b0:38c:b03b:b3c2 with SMTP id h9-20020a05600c350900b0038cb03bb3c2mr21132399wmq.81.1648231417266; Fri, 25 Mar 2022 11:03:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a4-20020a05600c348400b0038cd743830esm3776984wmq.29.2022.03.25.11.03.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:36 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:03 +0000 Subject: [PATCH v9 20/30] t/helper/fsmonitor-client: create IPC client to talk to FSMonitor Daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create an IPC client to send query and flush commands to the daemon. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- Makefile | 1 + t/helper/test-fsmonitor-client.c | 116 +++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + 4 files changed, 119 insertions(+) create mode 100644 t/helper/test-fsmonitor-client.c diff --git a/Makefile b/Makefile index 26567d4f772..daa21bed6c3 100644 --- a/Makefile +++ b/Makefile @@ -716,6 +716,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o TEST_BUILTINS_OBJS += test-dump-untracked-cache.o TEST_BUILTINS_OBJS += test-example-decorate.o TEST_BUILTINS_OBJS += test-fast-rebase.o +TEST_BUILTINS_OBJS += test-fsmonitor-client.o TEST_BUILTINS_OBJS += test-genrandom.o TEST_BUILTINS_OBJS += test-genzeros.o TEST_BUILTINS_OBJS += test-getcwd.o diff --git a/t/helper/test-fsmonitor-client.c b/t/helper/test-fsmonitor-client.c new file mode 100644 index 00000000000..3062c8a3c2b --- /dev/null +++ b/t/helper/test-fsmonitor-client.c @@ -0,0 +1,116 @@ +/* + * test-fsmonitor-client.c: client code to send commands/requests to + * a `git fsmonitor--daemon` daemon. + */ + +#include "test-tool.h" +#include "cache.h" +#include "parse-options.h" +#include "fsmonitor-ipc.h" + +#ifndef HAVE_FSMONITOR_DAEMON_BACKEND +int cmd__fsmonitor_client(int argc, const char **argv) +{ + die("fsmonitor--daemon not available on this platform"); +} +#else + +/* + * Read the `.git/index` to get the last token written to the + * FSMonitor Index Extension. + */ +static const char *get_token_from_index(void) +{ + struct index_state *istate = the_repository->index; + + if (do_read_index(istate, the_repository->index_file, 0) < 0) + die("unable to read index file"); + if (!istate->fsmonitor_last_update) + die("index file does not have fsmonitor extension"); + + return istate->fsmonitor_last_update; +} + +/* + * Send an IPC query to a `git-fsmonitor--daemon` daemon and + * ask for the changes since the given token or from the last + * token in the index extension. + * + * This will implicitly start a daemon process if necessary. The + * daemon process will persist after we exit. + */ +static int do_send_query(const char *token) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + if (!token || !*token) + token = get_token_from_index(); + + ret = fsmonitor_ipc__send_query(token, &answer); + if (ret < 0) + die("could not query fsmonitor--daemon"); + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +/* + * Send a "flush" command to the `git-fsmonitor--daemon` (if running) + * and tell it to flush its cache. + * + * This feature is primarily used by the test suite to simulate a loss of + * sync with the filesystem where we miss kernel events. + */ +static int do_send_flush(void) +{ + struct strbuf answer = STRBUF_INIT; + int ret; + + ret = fsmonitor_ipc__send_command("flush", &answer); + if (ret) + return ret; + + write_in_full(1, answer.buf, answer.len); + strbuf_release(&answer); + + return 0; +} + +int cmd__fsmonitor_client(int argc, const char **argv) +{ + const char *subcmd; + const char *token = NULL; + + const char * const fsmonitor_client_usage[] = { + "test-tool fsmonitor-client query []", + "test-tool fsmonitor-client flush", + NULL, + }; + + struct option options[] = { + OPT_STRING(0, "token", &token, "token", + "command token to send to the server"), + OPT_END() + }; + + argc = parse_options(argc, argv, NULL, options, fsmonitor_client_usage, 0); + + if (argc != 1) + usage_with_options(fsmonitor_client_usage, options); + + subcmd = argv[0]; + + setup_git_directory(); + + if (!strcmp(subcmd, "query")) + return !!do_send_query(token); + + if (!strcmp(subcmd, "flush")) + return !!do_send_flush(); + + die("Unhandled subcommand: '%s'", subcmd); +} +#endif diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index e6ec69cf326..0424f7adf5d 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -32,6 +32,7 @@ static struct test_cmd cmds[] = { { "dump-untracked-cache", cmd__dump_untracked_cache }, { "example-decorate", cmd__example_decorate }, { "fast-rebase", cmd__fast_rebase }, + { "fsmonitor-client", cmd__fsmonitor_client }, { "genrandom", cmd__genrandom }, { "genzeros", cmd__genzeros }, { "getcwd", cmd__getcwd }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 20756eefdda..c876e8246fb 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -23,6 +23,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv); int cmd__dump_reftable(int argc, const char **argv); int cmd__example_decorate(int argc, const char **argv); int cmd__fast_rebase(int argc, const char **argv); +int cmd__fsmonitor_client(int argc, const char **argv); int cmd__genrandom(int argc, const char **argv); int cmd__genzeros(int argc, const char **argv); int cmd__getcwd(int argc, const char **argv); From patchwork Fri Mar 25 18:03:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791927 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95FCCC433FE for ; Fri, 25 Mar 2022 19:27:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229780AbiCYT2c (ORCPT ); Fri, 25 Mar 2022 15:28:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229825AbiCYT20 (ORCPT ); Fri, 25 Mar 2022 15:28:26 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E50011EA5C3 for ; Fri, 25 Mar 2022 12:01:52 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id n35so4951905wms.5 for ; Fri, 25 Mar 2022 12:01:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=PzYiX4KZSbUNaf88GIbxOE1hs0cGSW6c+HsGJS4mMWY=; b=mrFAtsagPaLDR2LMTSGqfGNLkKId/YluN3oKCrYLc1kniLYBcSpGNLMUz+/qq/6TTY mbVe2tVzeDNjVw9zeFgLnP/SWxR/NwNX31PZj2+bKn23fNfZIbpTKP0RLOwQAAz32Et6 NqwEdLiCQNjQI5EhF2pgj6N0+MQrKwBCzFaH1qqwFl+G38ioSLbylqDcpE+QPWnDLbr+ /qq8akVU+j39JVSE7o8fKwmG1YGSNFH4ANvPD5s0DF4G8PKAhGHS0ySHZCy0VbyGgnOP fiQoiDnEvZhYkWwq/UCL+oL/Nw/SEu8yMusVOGVVg9bqL/m6YODFVdYr/B1OXFUA0OI4 RtAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=PzYiX4KZSbUNaf88GIbxOE1hs0cGSW6c+HsGJS4mMWY=; b=KGMZvvhEwajB4r37QFXVy9e4069my0XqbXaYHZEfhjilAEafjgesMqbcJPr5/DvAOF J0MlemtvaTMEB1v3RAMoRRAkxt8OnIT2tO8QBs48aVFOKKVkPC6W7NTaK4Anz+8fam70 aYVZbOZ5yAynuaGgD2t8qj0zAiI+Cw5nQyAVGT0/uMx6tcIg43sUtw+OuV0L67E+6LAJ qaiEOCDgSO4C6YLyTP1NhqWD1cF+/98VbDUUwQ+BeNyGyf3DgXQB/tLWy7ZSrDJ+6xOx Z3uvWQbYRFY1jfYNQnCWZLNKqph3LE7lx3qfJoOlDtsJj/83hAjDStOPbXSiHqYCckuq Eadg== X-Gm-Message-State: AOAM531sRJ6IVpZfrts6Id5iU9MHR9+LlfzYaIUvyCo8TkpNZ817Aoj/ UvdLdf3OyWOopAofGbIO4EQUoukO7dY= X-Google-Smtp-Source: ABdhPJyJEU2VZajejUv5EHzTvUNhjCVNQoNhwQXlSgISl7ISlbtDCnDb3MOS9VnR0XD2v5ft2X5BvA== X-Received: by 2002:a05:600c:600a:b0:38c:6c9e:f9c7 with SMTP id az10-20020a05600c600a00b0038c6c9ef9c7mr20229007wmb.26.1648231418259; Fri, 25 Mar 2022 11:03:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v124-20020a1cac82000000b0037c3d08e0e7sm9020097wme.29.2022.03.25.11.03.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:37 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:04 +0000 Subject: [PATCH v9 21/30] t7527: create test for fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/t7527-builtin-fsmonitor.sh | 494 +++++++++++++++++++++++++++++++++++ 1 file changed, 494 insertions(+) create mode 100755 t/t7527-builtin-fsmonitor.sh diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh new file mode 100755 index 00000000000..062e01c0dfc --- /dev/null +++ b/t/t7527-builtin-fsmonitor.sh @@ -0,0 +1,494 @@ +#!/bin/sh + +test_description='built-in file system watcher' + +. ./test-lib.sh + +if ! test_have_prereq FSMONITOR_DAEMON +then + skip_all="fsmonitor--daemon is not supported on this platform" + test_done +fi + +stop_daemon_delete_repo () { + r=$1 && + test_might_fail git -C $r fsmonitor--daemon stop && + rm -rf $1 +} + +start_daemon () { + r= tf= t2= tk= && + + while test "$#" -ne 0 + do + case "$1" in + -C) + r="-C ${2?}" + shift + ;; + --tf) + tf="${2?}" + shift + ;; + --t2) + t2="${2?}" + shift + ;; + --tk) + tk="${2?}" + shift + ;; + -*) + BUG "error: unknown option: '$1'" + ;; + *) + BUG "error: unbound argument: '$1'" + ;; + esac + shift + done && + + ( + if test -n "$tf" + then + GIT_TRACE_FSMONITOR="$tf" + export GIT_TRACE_FSMONITOR + fi && + + if test -n "$t2" + then + GIT_TRACE2_PERF="$t2" + export GIT_TRACE2_PERF + fi && + + if test -n "$tk" + then + GIT_TEST_FSMONITOR_TOKEN="$tk" + export GIT_TEST_FSMONITOR_TOKEN + fi && + + git $r fsmonitor--daemon start && + git $r fsmonitor--daemon status + ) +} + +# Is a Trace2 data event present with the given catetory and key? +# We do not care what the value is. +# +have_t2_data_event () { + c=$1 && + k=$2 && + + grep -e '"event":"data".*"category":"'"$c"'".*"key":"'"$k"'"' +} + +test_expect_success 'explicit daemon start and stop' ' + test_when_finished "stop_daemon_delete_repo test_explicit" && + + git init test_explicit && + start_daemon -C test_explicit && + + git -C test_explicit fsmonitor--daemon stop && + test_must_fail git -C test_explicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon start' ' + test_when_finished "stop_daemon_delete_repo test_implicit" && + + git init test_implicit && + test_must_fail git -C test_implicit fsmonitor--daemon status && + + # query will implicitly start the daemon. + # + # for test-script simplicity, we send a V1 timestamp rather than + # a V2 token. either way, the daemon response to any query contains + # a new V2 token. (the daemon may complain that we sent a V1 request, + # but this test case is only concerned with whether the daemon was + # implicitly started.) + + GIT_TRACE2_EVENT="$PWD/.git/trace" \ + test-tool -C test_implicit fsmonitor-client query --token 0 >actual && + nul_to_q actual.filtered && + grep "builtin:" actual.filtered && + + # confirm that a daemon was started in the background. + # + # since the mechanism for starting the background daemon is platform + # dependent, just confirm that the foreground command received a + # response from the daemon. + + have_t2_data_event fsm_client query/response-length <.git/trace && + + git -C test_implicit fsmonitor--daemon status && + git -C test_implicit fsmonitor--daemon stop && + test_must_fail git -C test_implicit fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (delete .git)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_1" && + + git init test_implicit_1 && + + start_daemon -C test_implicit_1 && + + # deleting the .git directory will implicitly stop the daemon. + rm -rf test_implicit_1/.git && + + # [1] Create an empty .git directory so that the following Git + # command will stay relative to the `-C` directory. + # + # Without this, the Git command will override the requested + # -C argument and crawl out to the containing Git source tree. + # This would make the test result dependent upon whether we + # were using fsmonitor on our development worktree. + # + sleep 1 && + mkdir test_implicit_1/.git && + + test_must_fail git -C test_implicit_1 fsmonitor--daemon status +' + +test_expect_success 'implicit daemon stop (rename .git)' ' + test_when_finished "stop_daemon_delete_repo test_implicit_2" && + + git init test_implicit_2 && + + start_daemon -C test_implicit_2 && + + # renaming the .git directory will implicitly stop the daemon. + mv test_implicit_2/.git test_implicit_2/.xxx && + + # See [1] above. + # + sleep 1 && + mkdir test_implicit_2/.git && + + test_must_fail git -C test_implicit_2 fsmonitor--daemon status +' + +test_expect_success 'cannot start multiple daemons' ' + test_when_finished "stop_daemon_delete_repo test_multiple" && + + git init test_multiple && + + start_daemon -C test_multiple && + + test_must_fail git -C test_multiple fsmonitor--daemon start 2>actual && + grep "fsmonitor--daemon is already running" actual && + + git -C test_multiple fsmonitor--daemon stop && + test_must_fail git -C test_multiple fsmonitor--daemon status +' + +# These tests use the main repo in the trash directory + +test_expect_success 'setup' ' + >tracked && + >modified && + >delete && + >rename && + mkdir dir1 && + >dir1/tracked && + >dir1/modified && + >dir1/delete && + >dir1/rename && + mkdir dir2 && + >dir2/tracked && + >dir2/modified && + >dir2/delete && + >dir2/rename && + mkdir dirtorename && + >dirtorename/a && + >dirtorename/b && + + cat >.gitignore <<-\EOF && + .gitignore + expect* + actual* + EOF + + git -c core.fsmonitor=false add . && + test_tick && + git -c core.fsmonitor=false commit -m initial && + + git config core.fsmonitor true +' + +# The test already explicitly stopped (or tried to stop) the daemon. +# This is here in case something else fails first. +# +redundant_stop_daemon () { + test_might_fail git fsmonitor--daemon stop +} + +test_expect_success 'update-index implicitly starts daemon' ' + test_when_finished redundant_stop_daemon && + + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_1" \ + git update-index --fsmonitor && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + # Confirm that the trace2 log contains a record of the + # daemon starting. + test_subcommand git fsmonitor--daemon start <.git/trace_implicit_1 +' + +test_expect_success 'status implicitly starts daemon' ' + test_when_finished redundant_stop_daemon && + + test_must_fail git fsmonitor--daemon status && + + GIT_TRACE2_EVENT="$PWD/.git/trace_implicit_2" \ + git status >actual && + + git fsmonitor--daemon status && + test_might_fail git fsmonitor--daemon stop && + + # Confirm that the trace2 log contains a record of the + # daemon starting. + test_subcommand git fsmonitor--daemon start <.git/trace_implicit_2 +' + +edit_files () { + echo 1 >modified && + echo 2 >dir1/modified && + echo 3 >dir2/modified && + >dir1/untracked +} + +delete_files () { + rm -f delete && + rm -f dir1/delete && + rm -f dir2/delete +} + +create_files () { + echo 1 >new && + echo 2 >dir1/new && + echo 3 >dir2/new +} + +rename_files () { + mv rename renamed && + mv dir1/rename dir1/renamed && + mv dir2/rename dir2/renamed +} + +file_to_directory () { + rm -f delete && + mkdir delete && + echo 1 >delete/new +} + +directory_to_file () { + rm -rf dir1 && + echo 1 >dir1 +} + +# The next few test cases confirm that our fsmonitor daemon sees each type +# of OS filesystem notification that we care about. At this layer we just +# ensure we are getting the OS notifications and do not try to confirm what +# is reported by `git status`. +# +# We run a simple query after modifying the filesystem just to introduce +# a bit of a delay so that the trace logging from the daemon has time to +# get flushed to disk. +# +# We `reset` and `clean` at the bottom of each test (and before stopping the +# daemon) because these commands might implicitly restart the daemon. + +clean_up_repo_and_stop_daemon () { + git reset --hard HEAD && + git clean -fd && + test_might_fail git fsmonitor--daemon stop && + rm -f .git/trace +} + +test_expect_success 'edit some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + edit_files && + + test-tool fsmonitor-client query --token 0 && + + grep "^event: dir1/modified$" .git/trace && + grep "^event: dir2/modified$" .git/trace && + grep "^event: modified$" .git/trace && + grep "^event: dir1/untracked$" .git/trace +' + +test_expect_success 'create some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + create_files && + + test-tool fsmonitor-client query --token 0 && + + grep "^event: dir1/new$" .git/trace && + grep "^event: dir2/new$" .git/trace && + grep "^event: new$" .git/trace +' + +test_expect_success 'delete some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + delete_files && + + test-tool fsmonitor-client query --token 0 && + + grep "^event: dir1/delete$" .git/trace && + grep "^event: dir2/delete$" .git/trace && + grep "^event: delete$" .git/trace +' + +test_expect_success 'rename some files' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + rename_files && + + test-tool fsmonitor-client query --token 0 && + + grep "^event: dir1/rename$" .git/trace && + grep "^event: dir2/rename$" .git/trace && + grep "^event: rename$" .git/trace && + grep "^event: dir1/renamed$" .git/trace && + grep "^event: dir2/renamed$" .git/trace && + grep "^event: renamed$" .git/trace +' + +test_expect_success 'rename directory' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + mv dirtorename dirrenamed && + + test-tool fsmonitor-client query --token 0 && + + grep "^event: dirtorename/*$" .git/trace && + grep "^event: dirrenamed/*$" .git/trace +' + +test_expect_success 'file changes to directory' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + file_to_directory && + + test-tool fsmonitor-client query --token 0 && + + grep "^event: delete$" .git/trace && + grep "^event: delete/new$" .git/trace +' + +test_expect_success 'directory changes to a file' ' + test_when_finished clean_up_repo_and_stop_daemon && + + start_daemon --tf "$PWD/.git/trace" && + + directory_to_file && + + test-tool fsmonitor-client query --token 0 && + + grep "^event: dir1$" .git/trace +' + +# The next few test cases exercise the token-resync code. When filesystem +# drops events (because of filesystem velocity or because the daemon isn't +# polling fast enough), we need to discard the cached data (relative to the +# current token) and start collecting events under a new token. +# +# the 'test-tool fsmonitor-client flush' command can be used to send a +# "flush" message to a running daemon and ask it to do a flush/resync. + +test_expect_success 'flush cached data' ' + test_when_finished "stop_daemon_delete_repo test_flush" && + + git init test_flush && + + start_daemon -C test_flush --tf "$PWD/.git/trace_daemon" --tk true && + + # The daemon should have an initial token with no events in _0 and + # then a few (probably platform-specific number of) events in _1. + # These should both have the same . + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_0 && + nul_to_q actual_q0 && + + >test_flush/file_1 && + >test_flush/file_2 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000001:0" >actual_1 && + nul_to_q actual_q1 && + + grep "file_1" actual_q1 && + + # Force a flush. This will change the , reset the , and + # flush the file data. Then create some events and ensure that the file + # again appears in the cache. It should have the new . + + test-tool -C test_flush fsmonitor-client flush >flush_0 && + nul_to_q flush_q0 && + grep "^builtin:test_00000002:0Q/Q$" flush_q0 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_2 && + nul_to_q actual_q2 && + + grep "^builtin:test_00000002:0Q$" actual_q2 && + + >test_flush/file_3 && + + test-tool -C test_flush fsmonitor-client query --token "builtin:test_00000002:0" >actual_3 && + nul_to_q actual_q3 && + + grep "file_3" actual_q3 +' + +# The next few test cases create repos where the .git directory is NOT +# inside the one of the working directory. That is, where .git is a file +# that points to a directory elsewhere. This happens for submodules and +# non-primary worktrees. + +test_expect_success 'setup worktree base' ' + git init wt-base && + echo 1 >wt-base/file1 && + git -C wt-base add file1 && + git -C wt-base commit -m "c1" +' + +test_expect_success 'worktree with .git file' ' + git -C wt-base worktree add ../wt-secondary && + + start_daemon -C wt-secondary \ + --tf "$PWD/trace_wt_secondary" \ + --t2 "$PWD/trace2_wt_secondary" && + + git -C wt-secondary fsmonitor--daemon stop && + test_must_fail git -C wt-secondary fsmonitor--daemon status +' + +# NEEDSWORK: Repeat one of the "edit" tests on wt-secondary and +# confirm that we get the same events and behavior -- that is, that +# fsmonitor--daemon correctly watches BOTH the working directory and +# the external GITDIR directory and behaves the same as when ".git" +# is a directory inside the working directory. + +test_expect_success 'cleanup worktrees' ' + stop_daemon_delete_repo wt-secondary && + stop_daemon_delete_repo wt-base +' + +test_done From patchwork Fri Mar 25 18:03:05 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791925 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 234CEC433EF for ; Fri, 25 Mar 2022 19:26:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229516AbiCYT1t (ORCPT ); Fri, 25 Mar 2022 15:27:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58588 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229789AbiCYT1W (ORCPT ); Fri, 25 Mar 2022 15:27:22 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AC9032B5ADC for ; Fri, 25 Mar 2022 12:00:50 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id a1so12054626wrh.10 for ; Fri, 25 Mar 2022 12:00:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MEG3piFdmt+jIkWUnne4CEZ7uXu+O+UAM85s3uqrVgA=; b=OEapDRwuPMWfK7CzBIMeZ8xhlRkfwKkJlXswvtJaIuvPVzXz8DkKqq2NOsJzR5RouH /04DTPEmqUHyUZD9xb9WSZ0o4M+n4MmAHD2JkjYR7/aZKT7tU3vTtZEKbO1fG/4f8RaW FBwgkiM/f3bArCN6bafkHH7AlETRWlWHWRuz0ZtD4ZBxnWl1DRdUt5Z48gtrdUV65O4c Y1XGWis6ihXkGk7uFTgXNC7HcM99or/r5zAB8dhqFrGbpwiawkWC0oOTqvVd9XO4Af02 ZgTP5p4MoL16zgfTXn6iBtuAOOehx74Gtd352fraSWWILtKWQmDcMn8AFikFhB7dDn6Q GJSA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MEG3piFdmt+jIkWUnne4CEZ7uXu+O+UAM85s3uqrVgA=; b=L6d0ADygSHCzpxXIAMOqTxqFn5a6+Lb5xTiQmyUo/RJ98SG+NqqulsqsKP5Zh9qQeR IPsO7X4TYh7hx6eoR9vxIXpti7Hxer4esRJAfDHH/4q5whlb0xEZhtl55NLa1t+t0c2c EnWnmnRt2zMcpEUXJir3pNVudl6IA2Wf1DiDjB1KoKCSmQPhIpqO8gqY5ybjxd8r5j41 PS2u4syUu50PA5D3eDPLdHokUDaBkOWFJA1fuyzAuDhlebmqEK2ACrhUcAEk006IXrj1 WusyBUkjN/qBujlcv/iB+WhegJD00e5ITAy1zAE5qMXbiMldmo9MUG9nq2GA6qVVvhmc iVog== X-Gm-Message-State: AOAM531CBji4iRJiwRCNO08Yo5jEoFbqm+fMh3pcAUOvBJ6TqbI5oq41 3+KrZfeQWmZQMPhkupxheIUy0f4CvXk= X-Google-Smtp-Source: ABdhPJz8ESI+Io36sQXiQ/DSPiz/Qjq6JZ5Ea37pbjGtMcpCZ4/+7Mt5QuLUPDQX+F0Yuourc6TdAg== X-Received: by 2002:adf:e744:0:b0:204:213:ae40 with SMTP id c4-20020adfe744000000b002040213ae40mr10445974wrn.344.1648231419303; Fri, 25 Mar 2022 11:03:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v20-20020a7bcb54000000b0037fa63db8aasm9115215wmj.5.2022.03.25.11.03.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:38 -0700 (PDT) Message-Id: <06d56d3a733f4fc5ae7fdfa90a667ae99bda1a0e.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:05 +0000 Subject: [PATCH v9 22/30] t/perf: avoid copying builtin fsmonitor files into test repo Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Do not copy any of the various fsmonitor--daemon files from the .git directory of the (GIT_PREF_REPO or GIT_PERF_LARGE_REPO) source repo into the test's trash directory. When perf tests start, they copy the contents of the source repo into the test's trash directory. If fsmonitor is running in the source repo, there may be control files, such as the IPC socket and/or fsmonitor cookie files. These should not be copied into the test repo. Unix domain sockets cannot be copied in the manner used by the test setup, so if present, the test setup fails. Cookie files are harmless, but we should avoid them. The builtin fsmonitor keeps all such control files/sockets in .git/fsmonitor--daemon*, so it is simple to exclude them. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- t/perf/perf-lib.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/perf-lib.sh b/t/perf/perf-lib.sh index 407252bac70..932105cd12c 100644 --- a/t/perf/perf-lib.sh +++ b/t/perf/perf-lib.sh @@ -78,7 +78,7 @@ test_perf_copy_repo_contents () { for stuff in "$1"/* do case "$stuff" in - */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees) + */objects|*/hooks|*/config|*/commondir|*/gitdir|*/worktrees|*/fsmonitor--daemon*) ;; *) cp -R "$stuff" "$repo/.git/" || exit 1 From patchwork Fri Mar 25 18:03:06 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12792002 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89B46C433EF for ; Fri, 25 Mar 2022 19:42:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231445AbiCYTnj (ORCPT ); Fri, 25 Mar 2022 15:43:39 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42530 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231657AbiCYTnL (ORCPT ); Fri, 25 Mar 2022 15:43:11 -0400 Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6FE0E3ED607 for ; Fri, 25 Mar 2022 12:17:12 -0700 (PDT) Received: by mail-ej1-x62c.google.com with SMTP id pv16so17293556ejb.0 for ; Fri, 25 Mar 2022 12:17:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ytZGPKQBsgePTFk8nBNZCqEs9yPPwA8MYf3sa3pnTvQ=; b=XNQYwPODTukke8vD75zE8tb9Z8KnTEMgVs30Tm+KW6n6TbjDPo8C46Kn6D+bQeLPpb oeN3llg1IjQYNbATrIpx5o10tudQAmGNs3+rvdIU9CDbtsd+ytbZkr3sd/RXdLQCQo1L kwzKYzUKj5HF1A39osDdbKRQkvuMWrmkE2Z9gf5c7jpoOC6EtRTECDyz1rsifmVhzj7K aPzUW8CLxxHsj/wH3nP8eGoIFjjYKXQX1boxS1B8ZdR0wR8qj4KhIxpFXfkWtOA3oPtY H4086OQ4YBdCRFhGU29Ey3oBrJqZ0B0Fk4wA3fKmxWC0viap9olhJtemxR6YqXAoRlEP TYFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ytZGPKQBsgePTFk8nBNZCqEs9yPPwA8MYf3sa3pnTvQ=; b=xICaMcbXu+rWj1CGQwM2IrV/Ci0WPKdqvzXzrJ1LCF2W5zSn6MetI2AdqFnxDARZp3 VVm6GbfLIats8WkA7+SizIpUaBOcihwTPDxf4j5BifG1rfBLXwtaFPgm+PN+QCVSYnw5 i9/1bQt0/1vzNziTITUBiDfxaqPSLaviuWvFt5QX5kdk8GIUBVLZ7fWFweNSJt1Oj+/0 b72LJW4hPjcejpdZTGXDMUmgAZ7PIlfel1RtQsmxlkS8QCZa5IjQwZCC5suDfoAqKcxL 5GSCccWQ1BLuTraUyDSBm1+ITXx14wK2iu4ddZ7WU4ANPj6bE06cqzqo0iS8iM/cHCj2 8BRQ== X-Gm-Message-State: AOAM532eeyQBPjW/3MdWCvz1sYVwgoJJydTFo1tpzpFhTOwgLHkpNHER DIsLIw+4AYIXhMTmgUDJY3tEyMHhP6U= X-Google-Smtp-Source: ABdhPJwRY6tmTkoIBhvHLMTPCX4GuSlZa3gTyOrDlF1oJf8AODSOnZvnZPAK5V5yXYaCgx5L/qYjJQ== X-Received: by 2002:adf:9581:0:b0:1ed:c341:4ed1 with SMTP id p1-20020adf9581000000b001edc3414ed1mr9846738wrp.299.1648231420242; Fri, 25 Mar 2022 11:03:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n10-20020a5d588a000000b002052e4aaf89sm5727942wrf.80.2022.03.25.11.03.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:39 -0700 (PDT) Message-Id: <2dd021512781a25ca3b6bc0c4d035b4fd4fc0456.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:06 +0000 Subject: [PATCH v9 23/30] t/helper/test-chmtime: skip directories on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach `test-tool.exe chmtime` to ignore errors when setting the mtime on a directory on Windows. NEEDSWORK: The Windows version of `utime()` (aka `mingw_utime()`) does not properly handle directories because it uses `_wopen()`. It should be converted to using `CreateFileW()` and backup semantics at a minimum. Since I'm already in the middle of a large patch series, I did not want to destabilize other callers of `utime()` right now. The problem has only been observed in the t/perf/p7519 test when the test repo contains an empty directory on disk. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- t/helper/test-chmtime.c | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/t/helper/test-chmtime.c b/t/helper/test-chmtime.c index 524b55ca496..dc28890a183 100644 --- a/t/helper/test-chmtime.c +++ b/t/helper/test-chmtime.c @@ -134,6 +134,21 @@ int cmd__chmtime(int argc, const char **argv) } if (utb.modtime != sb.st_mtime && utime(argv[i], &utb) < 0) { +#ifdef GIT_WINDOWS_NATIVE + if (S_ISDIR(sb.st_mode)) { + /* + * NEEDSWORK: The Windows version of `utime()` + * (aka `mingw_utime()`) does not correctly + * handle directory arguments, since it uses + * `_wopen()`. Ignore it for now since this + * is just a test. + */ + fprintf(stderr, + ("Failed to modify time on directory %s. " + "Skipping\n"), argv[i]); + continue; + } +#endif fprintf(stderr, "Failed to modify time on %s: %s\n", argv[i], strerror(errno)); return 1; From patchwork Fri Mar 25 18:03:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791988 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C29FC433FE for ; Fri, 25 Mar 2022 19:39:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231147AbiCYTki (ORCPT ); Fri, 25 Mar 2022 15:40:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232943AbiCYTje (ORCPT ); Fri, 25 Mar 2022 15:39:34 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 18DD83F8FE2 for ; Fri, 25 Mar 2022 12:18:30 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id r13so12095503wrr.9 for ; Fri, 25 Mar 2022 12:18:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=csSTLLvOXXHEVIyF+4iSBIHVXXVa6fmIaZ1XIryAyhM=; b=Kee4e8FBFIsguN6Et2/8ZllSh12te/RsOK2fWq00nysg6wglgx18h1X5JkgeFi17SY +Pmg8ST9lhOveoVEVa6tYsrtPb5Z5Y5NgzDoKpfhNDAlhjTtvEBW7YDKXAW2iI2RY3Gf XUiB59YmsWuacss11MjkTqE5Fkh20G/ozj2pUsXMDD8EjSuJRSo+Q0CSU5+O8+4bkCiv 5oEyhPU9Va+MVXwGdW44Uhh0ENqVuPo7s3te524eHP56/KUtqKxdmPx5TAUcu7pHVI4M 4egi3VcBrjp7lZJATkQ9QD88kJEMnpDRXD6tWwUK43ucm2qz/uzbAntdr7uGRhrRmh8Y OhNQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=csSTLLvOXXHEVIyF+4iSBIHVXXVa6fmIaZ1XIryAyhM=; b=q0frxotejBuETeLsSpXJjNrKYrIOR65xF5gaEyBLy3oPcoYFRh8eFMfoBYROk8yioZ haGksYRUkCQpts2uUHaTbbYSYRC7Gf7mYJPHByYV7ejfZdunI1hm/NajZnDSlDrEvv/Z gC9TKSEaabb6kUNeup9OWL6xWddWl6DbJLqlvwtPXKOhwjYKyW1J/gq5ld3xUb8y9CCv D/8HJyTuMu8fvq30Y87EHTuWrMww1vjcF/Tyw9kAPgDAj6pMdpKLwS6QZAE32xp05/7W 3jFbnWWr9sY88xOoOb0IeBoxP8e+nqLixuMo3Kt+1JZQVWbG+innms9IAuRQMpEGvGzX k6UQ== X-Gm-Message-State: AOAM532wNAFHdox3a3uCn/qsGOpXe1cYtTDJS3avh93ju/zy9JjNC/kf OWHMC2un4+eHBQ0h5e0pY/EEGZ98xQE= X-Google-Smtp-Source: ABdhPJzWoG7NHkjRrHwWY4AVmKDIPYzu6x5Lhz2Fx7uqNTZqKqr7mKnkDCoyuEFZKJta88JvAAS/Mg== X-Received: by 2002:a5d:40c8:0:b0:205:2a3b:c2c with SMTP id b8-20020a5d40c8000000b002052a3b0c2cmr10119047wrq.13.1648231421240; Fri, 25 Mar 2022 11:03:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m2-20020a056000024200b00205718e3a3csm5526603wrz.2.2022.03.25.11.03.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:40 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:07 +0000 Subject: [PATCH v9 24/30] t/perf/p7519: fix coding style Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index c8be58f3c76..5241eb6c4e5 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -72,7 +72,7 @@ then fi fi -trace_start() { +trace_start () { if test -n "$GIT_PERF_7519_TRACE" then name="$1" @@ -91,7 +91,7 @@ trace_start() { fi } -trace_stop() { +trace_stop () { if test -n "$GIT_PERF_7519_TRACE" then unset GIT_TRACE2_PERF @@ -133,7 +133,7 @@ test_expect_success "one time repo setup" ' fi ' -setup_for_fsmonitor() { +setup_for_fsmonitor () { # set INTEGRATION_SCRIPT depending on the environment if test -n "$INTEGRATION_PATH" then @@ -173,7 +173,7 @@ test_perf_w_drop_caches () { test_perf "$@" } -test_fsmonitor_suite() { +test_fsmonitor_suite () { if test -n "$INTEGRATION_SCRIPT"; then DESC="fsmonitor=$(basename $INTEGRATION_SCRIPT)" else From patchwork Fri Mar 25 18:03:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791922 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 435B4C433EF for ; Fri, 25 Mar 2022 19:24:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229488AbiCYT0Z (ORCPT ); Fri, 25 Mar 2022 15:26:25 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229557AbiCYT0Q (ORCPT ); Fri, 25 Mar 2022 15:26:16 -0400 Received: from mail-ej1-x62b.google.com (mail-ej1-x62b.google.com [IPv6:2a00:1450:4864:20::62b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8BC013E7E05 for ; Fri, 25 Mar 2022 11:58:33 -0700 (PDT) Received: by mail-ej1-x62b.google.com with SMTP id bq8so3176274ejb.10 for ; Fri, 25 Mar 2022 11:58:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=GBSYoKFMiXxL47tbCdtlqToK9/DpNmOfcvl8sunsaH4=; b=MdTUweNTxdVszcXzayefYqy9KQRo9uEkQ7yKpteQmj1GniDl/LT6fPBSbPUI6if2Oz OaolxhIqWNUl5aOaat7lLnH/WwLf+IaiCTXCRGrwIwBdbNN2oCcf35S1HSWjUwO8pwbA T/d1iGVCJYAX3hSUTzd9VFvD2fTGTBrxl5syA3U1vWuQzXSDSjmKMkaD/TZCyb8GaFx0 rvmAK+1ny0qmwAEiatmQ5treljdSEWuhOhYPkPlATLc2fdxMp0EsuOKiM89CFdctnorj 6VwPH1pWYUn51XwfZ3sMLNB0jKVSzyOfvn5XfBpG3//2LyCWF5AX0HHfZr0nvMsfJ/lk tTng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=GBSYoKFMiXxL47tbCdtlqToK9/DpNmOfcvl8sunsaH4=; b=71id213jAh42nr6htGB06Lry6P4bCLkViNcHptlnJ2X/a/J5ByRorZeIuLVLizQAiD Z2YOI7m2hkc+7xby7osU3dWViDR87Cs4kGbEKbARIC0Hn2pHKcX9ewF2n1L3YQMaVBiV mbEqTkjRv3XqzJT1eBermLkqV+CRRId5Q2mQhkGkrpCTsea55DfgxcYZ6eqdSimHy1oE kD4Fi0o6SFpXyK+dLfWoNKa1ai4fmo3irMQYr0wF//u/g690O4+86bwP25TKkMoWS5Pk Ak9aIC0kZxaJ8OsQt2SpsOwv6zSjV48Qt2Wz/ECveWwF4sxiBgOs5/+grPFMdxGwiuFy 9XaQ== X-Gm-Message-State: AOAM5331uIYJ+RdCf00WC6rtKcRqXoO0APTKYRbNyhA41JWgnc+G2wGj Swu5yIcTKLYsN1mm21vOIjg1VNbjluk= X-Google-Smtp-Source: ABdhPJzVtoemCJYl7rqMIsnDJsW0mm6GtwHb2TijCqglnKOKFj5YXZqlr+smCasySs/0KKz1juPigA== X-Received: by 2002:a5d:4ec7:0:b0:203:de83:6f44 with SMTP id s7-20020a5d4ec7000000b00203de836f44mr10193107wrv.56.1648231422559; Fri, 25 Mar 2022 11:03:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u11-20020a5d6acb000000b002058148822bsm7914537wrw.63.2022.03.25.11.03.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:41 -0700 (PDT) Message-Id: <50c2afaa49e20f2b19824acb2ff6a64439e9207c.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:08 +0000 Subject: [PATCH v9 25/30] t/perf/p7519: speed up test on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Change p7519 to use `test_seq` and `xargs` rather than a `for` loop to touch thousands of files. This takes minutes off of test runs on Windows because of process creation overhead. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- t/perf/p7519-fsmonitor.sh | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 5241eb6c4e5..a6c2a910e70 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -98,6 +98,13 @@ trace_stop () { fi } +touch_files () { + n=$1 && + d="$n"_files && + + (cd $d && test_seq 1 $n | xargs touch ) +} + test_expect_success "one time repo setup" ' # set untrackedCache depending on the environment if test -n "$GIT_PERF_7519_UNTRACKED_CACHE" @@ -119,10 +126,11 @@ test_expect_success "one time repo setup" ' fi && mkdir 1_file 10_files 100_files 1000_files 10000_files && - for i in $(test_seq 1 10); do touch 10_files/$i || return 1; done && - for i in $(test_seq 1 100); do touch 100_files/$i || return 1; done && - for i in $(test_seq 1 1000); do touch 1000_files/$i || return 1; done && - for i in $(test_seq 1 10000); do touch 10000_files/$i || return 1; done && + : 1_file directory should be left empty && + touch_files 10 && + touch_files 100 && + touch_files 1000 && + touch_files 10000 && git add 1_file 10_files 100_files 1000_files 10000_files && git commit -qm "Add files" && @@ -199,15 +207,15 @@ test_fsmonitor_suite () { # Update the mtimes on upto 100k files to make status think # that they are dirty. For simplicity, omit any files with - # LFs (i.e. anything that ls-files thinks it needs to dquote). - # Then fully backslash-quote the paths to capture any - # whitespace so that they pass thru xargs properly. + # LFs (i.e. anything that ls-files thinks it needs to dquote) + # and any files with whitespace so that they pass thru xargs + # properly. # test_perf_w_drop_caches "status (dirty) ($DESC)" ' git ls-files | \ head -100000 | \ grep -v \" | \ - sed '\''s/\(.\)/\\\1/g'\'' | \ + grep -v " ." | \ xargs test-tool chmtime -300 && git status ' From patchwork Fri Mar 25 18:03:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB1A4C433EF for ; Fri, 25 Mar 2022 19:39:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231236AbiCYTk7 (ORCPT ); Fri, 25 Mar 2022 15:40:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46108 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231975AbiCYTi3 (ORCPT ); Fri, 25 Mar 2022 15:38:29 -0400 Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14C8E1DA8CE for ; Fri, 25 Mar 2022 12:11:00 -0700 (PDT) Received: by mail-ej1-x62c.google.com with SMTP id a8so17160014ejc.8 for ; Fri, 25 Mar 2022 12:11:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=i8AAWLIhTvBsJbUUWF9sVC+yslqp1BHv6rf37oWZ/I8=; b=T+1rvMWFXy5lRUKR2nQm4RGbAJVKszNtv6ROqNqgv/ILoIqVVBFnnLdlzPIhLYevkT Z+rGezA1lRT3D1D3WhmKhYOXjM3ThtN+iLrVmLufSs/wd7l5NkOwd5YSm22k2Bk8pDBa hoUqu6etURBSqyEYcUqUTja/Ct47y4wBxyaSrc8ApUBtoHTTMw7HT17uzmz03yrEg2cq 7cOZVEUR05cWJJauBN1VSfjACCfVj056O6MY11hRA1bFKGSZBTrgBPX1ta+yw6QSYv1K CSZg1MckxV70jvim/iA+mtBAaI7kQ34aH9TIOkYHqTLr4kU3eZZEmmt5sUbQeCUn/Lft w/qw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=i8AAWLIhTvBsJbUUWF9sVC+yslqp1BHv6rf37oWZ/I8=; b=oLgd1DwyAoO5W8VdP5TAI1Z+28pk6GSI3CpOhR6sPXcc35sTCgOPbQ4kl9wB/otMyJ 4JVlaX/46wN1Z7VvhcZnPFfWbfrI8sJ+ML7Trt6Qx1qG/xYe1VnWEF3JYZtGiixLkpf9 2YSc0BqLXo1NlB4V5BtiD7mCH1DZcKLl3qH2jD+0f/NH4eF9aWSM2pLOMnjTKu0onr+T JexhgMq+yqVXDB9xzXGwdHKvNuMdxiDt3e9QTrqRVSrsehgHzeqyHuBneG8bxrUyGUvx EGx+ZLPLacIt2km7uEGJmDj8U6ySOjpo/5W2aa0pBzb814mz6MQjgDGDsQoVOC8UUO90 0hFw== X-Gm-Message-State: AOAM532aGCl253swSIWyCvW4B4TMZDBarIiPg1h/wsS4JDF/JWRK1krR Izq+pbt0ymIJkzKAYrGSsOe0oES6TWw= X-Google-Smtp-Source: ABdhPJx/rg4cW6nz8o0HdJvqRf1VFGpKg/EzPMKOqhRjVnlG/PNcjxaV+zr1tskQhcSZTBqkDc7O/g== X-Received: by 2002:adf:d4c2:0:b0:203:da50:12c5 with SMTP id w2-20020adfd4c2000000b00203da5012c5mr10425516wrk.100.1648231423486; Fri, 25 Mar 2022 11:03:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t9-20020a05600c198900b0038cb8b38f9fsm8662165wmq.21.2022.03.25.11.03.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:43 -0700 (PDT) Message-Id: <5b18e3b692661010387b3438e86238dbf0fa2eb2.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:09 +0000 Subject: [PATCH v9 26/30] t/perf/p7519: add fsmonitor--daemon test cases Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Repeat all of the fsmonitor perf tests using `git fsmonitor--daemon` and the "Simple IPC" interface. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- t/perf/p7519-fsmonitor.sh | 38 ++++++++++++++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 4 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index a6c2a910e70..0b9129ca7bc 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -141,7 +141,7 @@ test_expect_success "one time repo setup" ' fi ' -setup_for_fsmonitor () { +setup_for_fsmonitor_hook () { # set INTEGRATION_SCRIPT depending on the environment if test -n "$INTEGRATION_PATH" then @@ -182,7 +182,11 @@ test_perf_w_drop_caches () { } test_fsmonitor_suite () { - if test -n "$INTEGRATION_SCRIPT"; then + if test -n "$USE_FSMONITOR_DAEMON" + then + DESC="builtin fsmonitor--daemon" + elif test -n "$INTEGRATION_SCRIPT" + then DESC="fsmonitor=$(basename $INTEGRATION_SCRIPT)" else DESC="fsmonitor=disabled" @@ -261,11 +265,11 @@ test_fsmonitor_suite () { trace_start fsmonitor-watchman if test -n "$GIT_PERF_7519_FSMONITOR"; then for INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do - test_expect_success "setup for fsmonitor $INTEGRATION_PATH" 'setup_for_fsmonitor' + test_expect_success "setup for fsmonitor $INTEGRATION_PATH" 'setup_for_fsmonitor_hook' test_fsmonitor_suite done else - test_expect_success "setup for fsmonitor" 'setup_for_fsmonitor' + test_expect_success "setup for fsmonitor hook" 'setup_for_fsmonitor_hook' test_fsmonitor_suite fi @@ -293,4 +297,30 @@ test_expect_success "setup without fsmonitor" ' test_fsmonitor_suite trace_stop +# +# Run a full set of perf tests using the built-in fsmonitor--daemon. +# It does not use the Hook API, so it has a different setup. +# Explicitly start the daemon here and before we start client commands +# so that we can later add custom tracing. +# +if test_have_prereq FSMONITOR_DAEMON +then + USE_FSMONITOR_DAEMON=t + + test_expect_success "setup for builtin fsmonitor" ' + trace_start fsmonitor--daemon--server && + git fsmonitor--daemon start && + + trace_start fsmonitor--daemon--client && + + git config core.fsmonitor true && + git update-index --fsmonitor + ' + + test_fsmonitor_suite + + git fsmonitor--daemon stop + trace_stop +fi + test_done From patchwork Fri Mar 25 18:03:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90D28C433FE for ; Fri, 25 Mar 2022 19:29:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230039AbiCYTbT (ORCPT ); Fri, 25 Mar 2022 15:31:19 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58054 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229838AbiCYTbA (ORCPT ); Fri, 25 Mar 2022 15:31:00 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9DF461F89C3 for ; Fri, 25 Mar 2022 12:05:55 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id p184-20020a1c29c1000000b0037f76d8b484so4936629wmp.5 for ; Fri, 25 Mar 2022 12:05:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WTJ7J8sDnh/K0NYj2eZlDClERl3b/H70L1CBaOQEcxo=; b=BObbSdK4FdPQZjGX1wzSD0yoUznOPw3Id/bcIcpCSkEQv8Ysl88p+d4gLZfGxk/m+g pXSi9xckRxKLCx5TlFo0JDr7C5GebsTAJcrg+xe+B0b6Otdu+0j+V2+bohqwxi0IOF6Z sA8IXvBF8BSZv0qIAP+kvlCoPnIwa6ioSDA1zET40WBp+fMObEUhTnER50a01yphh+oa dxvqce4dDEhstJ50Je5EGx+iN8o1K3OAjoL60vbXj29pkp79s7AVeMIkmfyOgr5HG97j GdiGUYNmMr3Dk2GaGb0Ck6Ce3P+JpgjwoAJrs2Y0PuEQ3ZJhmK5fYQ7QWWr6gn3PwWq+ S7fA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WTJ7J8sDnh/K0NYj2eZlDClERl3b/H70L1CBaOQEcxo=; b=VnNEU9/a3dMYLV6UKwTYSRcgLnQab9gBBXfSgqgArhdJ9Y1R3wkIiEekTaZm1uPwZr 8xTpvJkNfKgQWhXa+c1+Qk5REZ0BWk3KgtrO4i6ztb9pe14r2+i9ZUMUicJw/glhxO10 VCaboBqevxkhCGGkS7r/wXHl7o/JlKFpfrF6pHYfbF7sKDBiqGNT6F1ymkA9xv/UQyl/ asHjcWjj5r+8hesw/4NjrZYjs9lMqyT4Pb6tASaPlnMPImmRkuOV8rnTuoGmPrI3S8mI sVv87JN8mTC09vEbT9cU12EWcbhckm1OU3oJAg3L5DNiyoXYwONSt/yurRPi3aR6nVaW r0ZA== X-Gm-Message-State: AOAM530X4v3ZORaDLBloVkbd2BHTJsqcABoNCg1hoDB9t6m9uQ81SU2v YxcbW51Gtvq3yR7RYd5JKF7p8BybSM0= X-Google-Smtp-Source: ABdhPJw1Q+Aq91YFYL3Gn9Ksg5aQXNCCxlBC2qOgGiCiITzyexuBN54GuLY68mcHGHpmu0VgydYepQ== X-Received: by 2002:a1c:6a15:0:b0:38b:57e8:dd5c with SMTP id f21-20020a1c6a15000000b0038b57e8dd5cmr20826242wmc.160.1648231424754; Fri, 25 Mar 2022 11:03:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k5-20020a5d66c5000000b00203fd25f4ecsm5392208wrw.16.2022.03.25.11.03.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:43 -0700 (PDT) Message-Id: <899c23f63c375438f060e9ee81440bee8c6613de.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:10 +0000 Subject: [PATCH v9 27/30] fsmonitor--daemon: periodically truncate list of modified files Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon to periodically truncate the list of modified files to save some memory. Clients will ask for the set of changes relative to a token that they found in the FSMN index extension in the index. (This token is like a point in time, but different). Clients will then update the index to contain the response token (so that subsequent commands will be relative to this new token). Therefore, the daemon can gradually truncate the in-memory list of changed paths as they become obsolete (older than the previous token). Since we may have multiple clients making concurrent requests with a skew of tokens and clients may be racing to the talk to the daemon, we lazily truncate the list. We introduce a 5 minute delay and truncate batches 5 minutes after they are considered obsolete. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 88 +++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index eafaafb45b1..ab9cc09f7ce 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -312,6 +312,75 @@ static void fsmonitor_batch__combine(struct fsmonitor_batch *batch_dest, batch_src->interned_paths[k]; } +/* + * To keep the batch list from growing unbounded in response to filesystem + * activity, we try to truncate old batches from the end of the list as + * they become irrelevant. + * + * We assume that the .git/index will be updated with the most recent token + * any time the index is updated. And future commands will only ask for + * recent changes *since* that new token. So as tokens advance into the + * future, older batch items will never be requested/needed. So we can + * truncate them without loss of functionality. + * + * However, multiple commands may be talking to the daemon concurrently + * or perform a slow command, so a little "token skew" is possible. + * Therefore, we want this to be a little bit lazy and have a generous + * delay. + * + * The current reader thread walked backwards in time from `token->batch_head` + * back to `batch_marker` somewhere in the middle of the batch list. + * + * Let's walk backwards in time from that marker an arbitrary delay + * and truncate the list there. Note that these timestamps are completely + * artificial (based on when we pinned the batch item) and not on any + * filesystem activity. + * + * Return the obsolete portion of the list after we have removed it from + * the official list so that the caller can free it after leaving the lock. + */ +#define MY_TIME_DELAY_SECONDS (5 * 60) /* seconds */ + +static struct fsmonitor_batch *with_lock__truncate_old_batches( + struct fsmonitor_daemon_state *state, + const struct fsmonitor_batch *batch_marker) +{ + /* assert current thread holding state->main_lock */ + + const struct fsmonitor_batch *batch; + struct fsmonitor_batch *remainder; + + if (!batch_marker) + return NULL; + + trace_printf_key(&trace_fsmonitor, "Truncate: mark (%"PRIu64",%"PRIu64")", + batch_marker->batch_seq_nr, + (uint64_t)batch_marker->pinned_time); + + for (batch = batch_marker; batch; batch = batch->next) { + time_t t; + + if (!batch->pinned_time) /* an overflow batch */ + continue; + + t = batch->pinned_time + MY_TIME_DELAY_SECONDS; + if (t > batch_marker->pinned_time) /* too close to marker */ + continue; + + goto truncate_past_here; + } + + return NULL; + +truncate_past_here: + state->current_token_data->batch_tail = (struct fsmonitor_batch *)batch; + + remainder = ((struct fsmonitor_batch *)batch)->next; + ((struct fsmonitor_batch *)batch)->next = NULL; + + return remainder; +} + static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) { if (!token) @@ -425,6 +494,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, const char *p; const struct fsmonitor_batch *batch_head; const struct fsmonitor_batch *batch; + struct fsmonitor_batch *remainder = NULL; intmax_t count = 0, duplicates = 0; kh_str_t *shown; int hash_ret; @@ -652,11 +722,29 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, * that work. */ fsmonitor_free_token_data(token_data); + } else if (batch) { + /* + * We are holding the lock and are the only + * reader of the ref-counted portion of the + * list, so we get the honor of seeing if the + * list can be truncated to save memory. + * + * The main loop did not walk to the end of the + * list, so this batch is the first item in the + * batch-list that is older than the requested + * end-point sequence number. See if the tail + * end of the list is obsolete. + */ + remainder = with_lock__truncate_old_batches(state, + batch); } } pthread_mutex_unlock(&state->main_lock); + if (remainder) + fsmonitor_batch__free_list(remainder); + trace2_data_intmax("fsmonitor", the_repository, "response/length", total_response_len); trace2_data_intmax("fsmonitor", the_repository, "response/count/files", count); trace2_data_intmax("fsmonitor", the_repository, "response/count/duplicates", duplicates); From patchwork Fri Mar 25 18:03:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CC25C433F5 for ; Fri, 25 Mar 2022 19:39:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231245AbiCYTlB (ORCPT ); Fri, 25 Mar 2022 15:41:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46880 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233086AbiCYTjk (ORCPT ); Fri, 25 Mar 2022 15:39:40 -0400 Received: from mail-ed1-x535.google.com (mail-ed1-x535.google.com [IPv6:2a00:1450:4864:20::535]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7940234553 for ; Fri, 25 Mar 2022 12:18:09 -0700 (PDT) Received: by mail-ed1-x535.google.com with SMTP id r23so10356544edb.0 for ; Fri, 25 Mar 2022 12:18:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=a2Hil1sjIGQyOaFUifrlQ4ChPXW9UVOHP1tlrqtxbDE=; b=CXNBMUfY/X1SxkZtK98UYhRA6TeDpmYJbbQAEevPZ8VThN8wW1Y3HIXpPXgGM59ITv 7logI/3S7WMpNOvgXG4jUPtbZQfOsWj0/cVR1qc44V+kQ9dwjLFYExEQ8qMzMwmXuBeH 4uzdBS+V3Tm70pAOnJWPv0LRiWs2HCAFOLRelmSodSI1BXdFAQ7QRnkQ9NX/jyVzFbYd AGlpL9w2i74j8ncyM0YJ1YtDvwWPaVZ4+x3dtWKwYaEU9WGXgQAksCk4ptjZDt1acOq6 cueCU8dNOO2hEgiQoSfw2XSHTxH3aDk0M7nFvPdzNyuAMqiC9Or0AUAFUSJshuX1xNGY riug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=a2Hil1sjIGQyOaFUifrlQ4ChPXW9UVOHP1tlrqtxbDE=; b=S/x06VLwDNQlQDqt/9lXjNatDIf4vewFVVs7yfMBPIQZDQRrW8kkMSWTC6A/Pj3NN3 lDNV0EhobaLps3pNGyd1m4N4JK4yRNtQr0hty9u+CAmM1fkZZzQTlmtHpP8JoYxFBD1G X2YjuJfQGMRxRbQRm1K2XOWM0Cs40z+Y0lBHcvapuc3u6/QwCw7vud7dhXhW9QN7Smd5 beZcP9Kt8/tDXCmc7qhXmbxC0J46+fYqmBnaQL1gWRh6MggtqJNVuaVD5VI2TQpXbzpR r9H29wCErjf4YwpZCh144hTKv9G9+2dYS/MkTv/VhAGFMZWB9XqNk9ThKOy/KaGKRF90 AyNw== X-Gm-Message-State: AOAM5316yTp5hukc6PoZF/5JzTI9TIt1dUSaXPZEtV6Hgipz0kt5QVay QNfmSoZnEQNl67vXORR/cPes4nxkNfM= X-Google-Smtp-Source: ABdhPJx9ag0Cf2viBMTlcQAp1uKaefdNlONRVvUqLdkQAw/Kvm7ZMjbhBluB07eSOQsQQqFy3MvqCQ== X-Received: by 2002:adf:d214:0:b0:204:2a98:a166 with SMTP id j20-20020adfd214000000b002042a98a166mr10111635wrh.406.1648231425747; Fri, 25 Mar 2022 11:03:45 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 3-20020a5d47a3000000b0020412ba45f6sm6686893wrb.8.2022.03.25.11.03.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:45 -0700 (PDT) Message-Id: <597a7192f94dd9c5c5101f5f92495565efdfe34f.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:11 +0000 Subject: [PATCH v9 28/30] fsmonitor--daemon: use a cookie file to sync with file system Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Teach fsmonitor--daemon client threads to create a cookie file inside the .git directory and then wait until FS events for the cookie are observed by the FS listener thread. This helps address the racy nature of file system events by blocking the client response until the kernel has drained any event backlog. This is especially important on MacOS where kernel events are only issued with a limited frequency. See the `latency` argument of `FSeventStreamCreate()`. The kernel only signals every `latency` seconds, but does not guarantee that the kernel queue is completely drained, so we may have to wait more than one interval. If we increase the latency, the system is more likely to drop events. We avoid these issues by having each client thread create a unique cookie file and then wait until it is seen in the event stream. Co-authored-by: Kevin Willford Co-authored-by: Johannes Schindelin Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- builtin/fsmonitor--daemon.c | 237 +++++++++++++++++++++++++++++++++++- fsmonitor--daemon.h | 5 + 2 files changed, 241 insertions(+), 1 deletion(-) diff --git a/builtin/fsmonitor--daemon.c b/builtin/fsmonitor--daemon.c index ab9cc09f7ce..46be55a4618 100644 --- a/builtin/fsmonitor--daemon.c +++ b/builtin/fsmonitor--daemon.c @@ -107,6 +107,162 @@ static int do_as_client__status(void) } } +enum fsmonitor_cookie_item_result { + FCIR_ERROR = -1, /* could not create cookie file ? */ + FCIR_INIT, + FCIR_SEEN, + FCIR_ABORT, +}; + +struct fsmonitor_cookie_item { + struct hashmap_entry entry; + char *name; + enum fsmonitor_cookie_item_result result; +}; + +static int cookies_cmp(const void *data, const struct hashmap_entry *he1, + const struct hashmap_entry *he2, const void *keydata) +{ + const struct fsmonitor_cookie_item *a = + container_of(he1, const struct fsmonitor_cookie_item, entry); + const struct fsmonitor_cookie_item *b = + container_of(he2, const struct fsmonitor_cookie_item, entry); + + return strcmp(a->name, keydata ? keydata : b->name); +} + +static enum fsmonitor_cookie_item_result with_lock__wait_for_cookie( + struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + int fd; + struct fsmonitor_cookie_item *cookie; + struct strbuf cookie_pathname = STRBUF_INIT; + struct strbuf cookie_filename = STRBUF_INIT; + enum fsmonitor_cookie_item_result result; + int my_cookie_seq; + + CALLOC_ARRAY(cookie, 1); + + my_cookie_seq = state->cookie_seq++; + + strbuf_addf(&cookie_filename, "%i-%i", getpid(), my_cookie_seq); + + strbuf_addbuf(&cookie_pathname, &state->path_cookie_prefix); + strbuf_addbuf(&cookie_pathname, &cookie_filename); + + cookie->name = strbuf_detach(&cookie_filename, NULL); + cookie->result = FCIR_INIT; + hashmap_entry_init(&cookie->entry, strhash(cookie->name)); + + hashmap_add(&state->cookies, &cookie->entry); + + trace_printf_key(&trace_fsmonitor, "cookie-wait: '%s' '%s'", + cookie->name, cookie_pathname.buf); + + /* + * Create the cookie file on disk and then wait for a notification + * that the listener thread has seen it. + */ + fd = open(cookie_pathname.buf, O_WRONLY | O_CREAT | O_EXCL, 0600); + if (fd < 0) { + error_errno(_("could not create fsmonitor cookie '%s'"), + cookie->name); + + cookie->result = FCIR_ERROR; + goto done; + } + + /* + * Technically, close() and unlink() can fail, but we don't + * care here. We only created the file to trigger a watch + * event from the FS to know that when we're up to date. + */ + close(fd); + unlink(cookie_pathname.buf); + + /* + * Technically, this is an infinite wait (well, unless another + * thread sends us an abort). I'd like to change this to + * use `pthread_cond_timedwait()` and return an error/timeout + * and let the caller do the trivial response thing, but we + * don't have that routine in our thread-utils. + * + * After extensive beta testing I'm not really worried about + * this. Also note that the above open() and unlink() calls + * will cause at least two FS events on that path, so the odds + * of getting stuck are pretty slim. + */ + while (cookie->result == FCIR_INIT) + pthread_cond_wait(&state->cookies_cond, + &state->main_lock); + +done: + hashmap_remove(&state->cookies, &cookie->entry, NULL); + + result = cookie->result; + + free(cookie->name); + free(cookie); + strbuf_release(&cookie_pathname); + + return result; +} + +/* + * Mark these cookies as _SEEN and wake up the corresponding client threads. + */ +static void with_lock__mark_cookies_seen(struct fsmonitor_daemon_state *state, + const struct string_list *cookie_names) +{ + /* assert current thread holding state->main_lock */ + + int k; + int nr_seen = 0; + + for (k = 0; k < cookie_names->nr; k++) { + struct fsmonitor_cookie_item key; + struct fsmonitor_cookie_item *cookie; + + key.name = cookie_names->items[k].string; + hashmap_entry_init(&key.entry, strhash(key.name)); + + cookie = hashmap_get_entry(&state->cookies, &key, entry, NULL); + if (cookie) { + trace_printf_key(&trace_fsmonitor, "cookie-seen: '%s'", + cookie->name); + cookie->result = FCIR_SEEN; + nr_seen++; + } + } + + if (nr_seen) + pthread_cond_broadcast(&state->cookies_cond); +} + +/* + * Set _ABORT on all pending cookies and wake up all client threads. + */ +static void with_lock__abort_all_cookies(struct fsmonitor_daemon_state *state) +{ + /* assert current thread holding state->main_lock */ + + struct hashmap_iter iter; + struct fsmonitor_cookie_item *cookie; + int nr_aborted = 0; + + hashmap_for_each_entry(&state->cookies, &iter, cookie, entry) { + trace_printf_key(&trace_fsmonitor, "cookie-abort: '%s'", + cookie->name); + cookie->result = FCIR_ABORT; + nr_aborted++; + } + + if (nr_aborted) + pthread_cond_broadcast(&state->cookies_cond); +} + /* * Requests to and from a FSMonitor Protocol V2 provider use an opaque * "token" as a virtual timestamp. Clients can request a summary of all @@ -404,6 +560,9 @@ static void fsmonitor_free_token_data(struct fsmonitor_token_data *token) * We should create a new token and start fresh (as if we just * booted up). * + * [2] Some of those lost events may have been for cookie files. We + * should assume the worst and abort them rather letting them starve. + * * If there are no concurrent threads reading the current token data * series, we can free it now. Otherwise, let the last reader free * it. @@ -425,6 +584,8 @@ static void with_lock__do_force_resync(struct fsmonitor_daemon_state *state) state->current_token_data = new_one; fsmonitor_free_token_data(free_me); + + with_lock__abort_all_cookies(state); } void fsmonitor_force_resync(struct fsmonitor_daemon_state *state) @@ -500,6 +661,8 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, int hash_ret; int do_trivial = 0; int do_flush = 0; + int do_cookie = 0; + enum fsmonitor_cookie_item_result cookie_result; /* * We expect `command` to be of the form: @@ -560,6 +723,7 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, * We have a V2 valid token: * "builtin::" */ + do_cookie = 1; } } @@ -568,6 +732,30 @@ static int do_handle_client(struct fsmonitor_daemon_state *state, if (!state->current_token_data) BUG("fsmonitor state does not have a current token"); + /* + * Write a cookie file inside the directory being watched in + * an effort to flush out existing filesystem events that we + * actually care about. Suspend this client thread until we + * see the filesystem events for this cookie file. + * + * Creating the cookie lets us guarantee that our FS listener + * thread has drained the kernel queue and we are caught up + * with the kernel. + * + * If we cannot create the cookie (or otherwise guarantee that + * we are caught up), we send a trivial response. We have to + * assume that there might be some very, very recent activity + * on the FS still in flight. + */ + if (do_cookie) { + cookie_result = with_lock__wait_for_cookie(state); + if (cookie_result != FCIR_SEEN) { + error(_("fsmonitor: cookie_result '%d' != SEEN"), + cookie_result); + do_trivial = 1; + } + } + if (do_flush) with_lock__do_force_resync(state); @@ -787,7 +975,9 @@ static int handle_client(void *data, return result; } -#define FSMONITOR_COOKIE_PREFIX ".fsmonitor-daemon-" +#define FSMONITOR_DIR "fsmonitor--daemon" +#define FSMONITOR_COOKIE_DIR "cookies" +#define FSMONITOR_COOKIE_PREFIX (FSMONITOR_DIR "/" FSMONITOR_COOKIE_DIR "/") enum fsmonitor_path_type fsmonitor_classify_path_workdir_relative( const char *rel) @@ -940,6 +1130,9 @@ void fsmonitor_publish(struct fsmonitor_daemon_state *state, } } + if (cookie_names->nr) + with_lock__mark_cookies_seen(state, cookie_names); + pthread_mutex_unlock(&state->main_lock); } @@ -1031,7 +1224,9 @@ static int fsmonitor_run_daemon(void) memset(&state, 0, sizeof(state)); + hashmap_init(&state.cookies, cookies_cmp, NULL, 0); pthread_mutex_init(&state.main_lock, NULL); + pthread_cond_init(&state.cookies_cond, NULL); state.error_code = 0; state.current_token_data = fsmonitor_new_token_data(); @@ -1056,6 +1251,44 @@ static int fsmonitor_run_daemon(void) state.nr_paths_watching = 2; } + /* + * We will write filesystem syncing cookie files into + * ///-. + * + * The extra layers of subdirectories here keep us from + * changing the mtime on ".git/" or ".git/foo/" when we create + * or delete cookie files. + * + * There have been problems with some IDEs that do a + * non-recursive watch of the ".git/" directory and run a + * series of commands any time something happens. + * + * For example, if we place our cookie files directly in + * ".git/" or ".git/foo/" then a `git status` (or similar + * command) from the IDE will cause a cookie file to be + * created in one of those dirs. This causes the mtime of + * those dirs to change. This triggers the IDE's watch + * notification. This triggers the IDE to run those commands + * again. And the process repeats and the machine never goes + * idle. + * + * Adding the extra layers of subdirectories prevents the + * mtime of ".git/" and ".git/foo" from changing when a + * cookie file is created. + */ + strbuf_init(&state.path_cookie_prefix, 0); + strbuf_addbuf(&state.path_cookie_prefix, &state.path_gitdir_watch); + + strbuf_addch(&state.path_cookie_prefix, '/'); + strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_DIR); + mkdir(state.path_cookie_prefix.buf, 0777); + + strbuf_addch(&state.path_cookie_prefix, '/'); + strbuf_addstr(&state.path_cookie_prefix, FSMONITOR_COOKIE_DIR); + mkdir(state.path_cookie_prefix.buf, 0777); + + strbuf_addch(&state.path_cookie_prefix, '/'); + /* * Confirm that we can create platform-specific resources for the * filesystem listener before we bother starting all the threads. @@ -1068,6 +1301,7 @@ static int fsmonitor_run_daemon(void) err = fsmonitor_run_daemon_1(&state); done: + pthread_cond_destroy(&state.cookies_cond); pthread_mutex_destroy(&state.main_lock); fsm_listen__dtor(&state); @@ -1075,6 +1309,7 @@ done: strbuf_release(&state.path_worktree_watch); strbuf_release(&state.path_gitdir_watch); + strbuf_release(&state.path_cookie_prefix); return err; } diff --git a/fsmonitor--daemon.h b/fsmonitor--daemon.h index 010fbfe60e9..bd09fffc176 100644 --- a/fsmonitor--daemon.h +++ b/fsmonitor--daemon.h @@ -45,6 +45,11 @@ struct fsmonitor_daemon_state { struct fsmonitor_token_data *current_token_data; + struct strbuf path_cookie_prefix; + pthread_cond_t cookies_cond; + int cookie_seq; + struct hashmap cookies; + int error_code; struct fsmonitor_daemon_backend_data *backend_data; From patchwork Fri Mar 25 18:03:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4BB8AC433EF for ; Fri, 25 Mar 2022 19:39:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231206AbiCYTk4 (ORCPT ); Fri, 25 Mar 2022 15:40:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44376 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232055AbiCYTih (ORCPT ); Fri, 25 Mar 2022 15:38:37 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B80C3FE414 for ; Fri, 25 Mar 2022 12:11:07 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id d7so12091479wrb.7 for ; Fri, 25 Mar 2022 12:11:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=OqCROC5s98+c4tM5gHnUYLicOFeDXUaj+atK5YqejLE=; b=GKYBNUqsB5dS8tYeLAYdxSe0x+1vomxazwWdjn6QFFibIo1W3OUHwEGneW5bV0eV+p eoRrzbiCH+GrgUfbxjeFl363tMJb35HuKAEf2r9VBVP1d8JxqJ//CeidqPFiD2crkvT8 aTQCoAQkb/hgIkqD+I71e1SbEN4+nch6MHHcrVDO9qoufWGzHmbQjt8iXS1oxRnMd5/S C7mhUWt3aLz2pZi/NkOMKqUztiSkRII/rYHfFa9lEjWD8KaP3U1xZX2E6bV53o057g3X gJiYsHzeI4zEsHACWMWvAIe3EaLcfjZrve8/FyB/GQVlheGQffHvvjKrTHO/KEBnB4UG aW3g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=OqCROC5s98+c4tM5gHnUYLicOFeDXUaj+atK5YqejLE=; b=r3eqSIOb8IVChUcbpWoFZMdGjhbOEWrSmTBTXuAC0PJKK3rhxrDtWaTZQisb6367TY blH5mg8DjFtIFI8U0w1ZUzKJU5p4bGiJ0J1WF6QLukAVQL3WJm/IkGbpvTA8ItpddE9c 2YxvmQX8MaZ88dqDP0h1Q5+n3h/X/YasnmrRKr3APzAQR/LEVKBmWQLw7oPsSEVkhH4W z9rBhiTnBuDyi0ZUzLZR8SkGAsTPmd+N/a46OEzyydGty6gTDZoySbQU75BYyIwWcFfj o60zqbLciQXCBg0b7afx3/YrxuGOpO1IuTf3/vw5rLx+wqbVQJSlnDwTM+QsEjwiePdh rGEA== X-Gm-Message-State: AOAM532H7ojQrOo/P2JUzHedJ0q7G3+8jA3mQ/EaLVPRimOF5BaWIpFO h5bdKSge1xRPqNDVioPzeiWfugSx7ew= X-Google-Smtp-Source: ABdhPJxhLDfBoUQH9wyDhIFdvvaNhkLMvPjlMP6tCfNmPCVVK6J2NgszG9UlWly29cZfRH71fC68Qw== X-Received: by 2002:a5d:64c4:0:b0:205:8280:73f6 with SMTP id f4-20020a5d64c4000000b00205828073f6mr10152433wri.358.1648231426713; Fri, 25 Mar 2022 11:03:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p11-20020a5d638b000000b002048a77636dsm5358890wru.97.2022.03.25.11.03.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:46 -0700 (PDT) Message-Id: <68a05fd2892b5d3aef6b3e390926e6d3c6b29c2d.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:12 +0000 Subject: [PATCH v9 29/30] fsmonitor: force update index after large responses Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Measure the time taken to apply the FSMonitor query result to the index and the untracked-cache. Set the `FSMONITOR_CHANGED` bit on `istate->cache_changed` when FSMonitor returns a very large repsonse to ensure that the index is written to disk. Normally, when the FSMonitor response includes a tracked file, the index is always updated. Similarly, the index might be updated when the response alters the untracked-cache (when enabled). However, in cases where neither of those cause the index to be considered changed, the FSMonitor response is wasted. Subsequent Git commands will make requests with the same token and receive the same response. If that response is very large, performance may suffer. It would be more efficient to force update the index now (and the token in the index extension) in order to reduce the size of the response received by future commands. This was observed on Windows after a large checkout. On Windows, the kernel emits events for the files that are changed as they are changed. However, it might delay events for the containing directories until the system is more idle (or someone scans the directory (so it seems)). The first status following a checkout would get the list of files. The subsequent status commands would get the list of directories as the events trickled out. But they would never catch up because the token was not advanced because the index wasn't updated. This list of directories caused `wt_status_collect_untracked()` to unnecessarily spend time actually scanning them during each command. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- fsmonitor.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/fsmonitor.c b/fsmonitor.c index a38b5710eb3..292a6742b4f 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -219,6 +219,43 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name) untracked_cache_invalidate_path(istate, name, 0); } +/* + * The number of pathnames that we need to receive from FSMonitor + * before we force the index to be updated. + * + * Note that any pathname within the set of received paths MAY cause + * cache-entry or istate flag bits to be updated and thus cause the + * index to be updated on disk. + * + * However, the response may contain many paths (such as ignored + * paths) that will not update any flag bits. And thus not force the + * index to be updated. (This is fine and normal.) It also means + * that the token will not be updated in the FSMonitor index + * extension. So the next Git command will find the same token in the + * index, make the same token-relative request, and receive the same + * response (plus any newly changed paths). If this response is large + * (and continues to grow), performance could be impacted. + * + * For example, if the user runs a build and it writes 100K object + * files but doesn't modify any source files, the index would not need + * to be updated. The FSMonitor response (after the build and + * relative to a pre-build token) might be 5MB. Each subsequent Git + * command will receive that same 100K/5MB response until something + * causes the index to be updated. And `refresh_fsmonitor()` will + * have to iterate over those 100K paths each time. + * + * Performance could be improved if we optionally force update the + * index after a very large response and get an updated token into + * the FSMonitor index extension. This should allow subsequent + * commands to get smaller and more current responses. + * + * The value chosen here does not need to be precise. The index + * will be updated automatically the first time the user touches + * a tracked file and causes a command like `git status` to + * update an mtime to be updated and/or set a flag bit. + */ +static int fsmonitor_force_update_threshold = 100; + void refresh_fsmonitor(struct index_state *istate) { struct strbuf query_result = STRBUF_INIT; @@ -362,25 +399,39 @@ apply_results: * information and that we should consider everything * invalid. We call this a trivial response. */ + trace2_region_enter("fsmonitor", "apply_results", istate->repo); + if (query_success && !is_trivial) { /* * Mark all pathnames returned by the monitor as dirty. * * This updates both the cache-entries and the untracked-cache. */ + int count = 0; + buf = query_result.buf; for (i = bol; i < query_result.len; i++) { if (buf[i] != '\0') continue; fsmonitor_refresh_callback(istate, buf + bol); bol = i + 1; + count++; } - if (bol < query_result.len) + if (bol < query_result.len) { fsmonitor_refresh_callback(istate, buf + bol); + count++; + } /* Now mark the untracked cache for fsmonitor usage */ if (istate->untracked) istate->untracked->use_fsmonitor = 1; + + if (count > fsmonitor_force_update_threshold) + istate->cache_changed |= FSMONITOR_CHANGED; + + trace2_data_intmax("fsmonitor", istate->repo, "apply_count", + count); + } else { /* * We failed to get a response or received a trivial response, @@ -409,6 +460,8 @@ apply_results: if (istate->untracked) istate->untracked->use_fsmonitor = 0; } + trace2_region_leave("fsmonitor", "apply_results", istate->repo); + strbuf_release(&query_result); /* Now that we've updated istate, save the last_update_token */ From patchwork Fri Mar 25 18:03:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12791939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1F38C433EF for ; Fri, 25 Mar 2022 19:28:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229872AbiCYT3u (ORCPT ); Fri, 25 Mar 2022 15:29:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:34164 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229885AbiCYT3f (ORCPT ); Fri, 25 Mar 2022 15:29:35 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA84BF55F8 for ; Fri, 25 Mar 2022 12:04:02 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id b19so12043517wrh.11 for ; Fri, 25 Mar 2022 12:04:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tPukjc2YCWWKLgm+y0vzYwCeZbE+rpSJVnltmQJirJA=; b=mUFWqCyOYa6oTUlDM2GjTy+X7vAgEhh/Il/qGrGCVAuV38GDUazHZeqMLdWHF+W+ip HyAigA2x/9/voqtLjPthIYtyC9LCbMyZW/JIjJyVvyQY2/2O3Bj8ZOveEALOm3P1LAQS vZbq/EI1Zd6Xbmc0ZhXVK68t7eAwyMbJIxxgTG5SMFO+00/p8Ur44WVs/Hez41baB27c E5DOLYGgC6CXuYykiop/i+id41MzTsq/P2ZcsZsA5JvYIn1ggkkyLyNPxI03zgkpUJd2 3jE588Gy48NGLJVovZq6bqiPJ/sYVkPl+McU+LS6hYkCxnA9wNnv8QBqZv05wU5ct+wp jI7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tPukjc2YCWWKLgm+y0vzYwCeZbE+rpSJVnltmQJirJA=; b=GL6JlfSGNMehF8sHW0b6LHbotXzop91tHEVIwFLOCOgat94aKzSqZyAed4Q55g/MX2 HqUnNRuspyeKnDjFdWJLMYz3Upx31cG3hbm35iwZLnK8XQgTuSsKuXc1bptf+xOMmU1J 8xU937eQ+LQC4UEfEDvzRGLRF0MsRjCxuYO6Efw9oQcqBhRLY6wmtmpZRKm5S2KfDAOt P8p2OZav0dseSqm3FqVTpCts9MaXHQW4RZ1gu6cilcSXJOg6WM5Fp2jsR9KlZfpF7IeS KtjoWcBCcs5HsS1Z31aZX8aXQZYSgWFk73vpg0Zt3UJ3sLnERHZje3fMNLBw5ZitHG6n LSWw== X-Gm-Message-State: AOAM533pAcHFNn15eGYURd0YIw1PjgPHuVysr5EqiWQRHYGfhM1JFrns 7zcoMW4OLPRlsvWLN0S5iGwyYRNmlpA= X-Google-Smtp-Source: ABdhPJxgwuOTLgboq4OMf2efZhLbyfkG6YpiX3lll68xNQ1Hl5MtCbGzRK/iY3BBF+DhDgUEAmtX1w== X-Received: by 2002:adf:e348:0:b0:1f0:537:1807 with SMTP id n8-20020adfe348000000b001f005371807mr10136376wrj.11.1648231427880; Fri, 25 Mar 2022 11:03:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g6-20020a5d5406000000b001f049726044sm5575743wrv.79.2022.03.25.11.03.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Mar 2022 11:03:47 -0700 (PDT) Message-Id: <5eb696daba2fe108d4d9ba2ccf4b357447ef9946.1648231393.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Mar 2022 18:03:13 +0000 Subject: [PATCH v9 30/30] t7527: test status with untracked-cache and fsmonitor--daemon Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Bagas Sanjaya , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , Jeff Hostetler , Eric Sunshine , Johannes Schindelin , Tao Klerks , rsbecker@nexbridge.com, Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Create 2x2 test matrix with the untracked-cache and fsmonitor--daemon features and a series of edits and verify that status output is identical. Signed-off-by: Jeff Hostetler Signed-off-by: Junio C Hamano --- t/t7527-builtin-fsmonitor.sh | 115 +++++++++++++++++++++++++++++++++++ 1 file changed, 115 insertions(+) diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh index 062e01c0dfc..bd0c952a116 100755 --- a/t/t7527-builtin-fsmonitor.sh +++ b/t/t7527-builtin-fsmonitor.sh @@ -205,6 +205,8 @@ test_expect_success 'setup' ' .gitignore expect* actual* + flush* + trace* EOF git -c core.fsmonitor=false add . && @@ -491,4 +493,117 @@ test_expect_success 'cleanup worktrees' ' stop_daemon_delete_repo wt-base ' +# The next few tests perform arbitrary/contrived file operations and +# confirm that status is correct. That is, that the data (or lack of +# data) from fsmonitor doesn't cause incorrect results. And doesn't +# cause incorrect results when the untracked-cache is enabled. + +test_lazy_prereq UNTRACKED_CACHE ' + git update-index --test-untracked-cache +' + +test_expect_success 'Matrix: setup for untracked-cache,fsmonitor matrix' ' + test_unconfig core.fsmonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop +' + +matrix_clean_up_repo () { + git reset --hard HEAD && + git clean -fd +} + +matrix_try () { + uc=$1 && + fsm=$2 && + fn=$3 && + + if test $uc = true && test $fsm = false + then + # The untracked-cache is buggy when FSMonitor is + # DISABLED, so skip the tests for this matrix + # combination. + # + # We've observed random, occasional test failures on + # Windows and MacOS when the UC is turned on and FSM + # is turned off. These are rare, but they do happen + # indicating that it is probably a race condition within + # the untracked cache itself. + # + # It usually happens when a test does F/D trickery and + # then the NEXT test fails because of extra status + # output from stale UC data from the previous test. + # + # Since FSMonitor is not involved in the error, skip + # the tests for this matrix combination. + # + return 0 + fi && + + test_expect_success "Matrix[uc:$uc][fsm:$fsm] $fn" ' + matrix_clean_up_repo && + $fn && + if test $uc = false && test $fsm = false + then + git status --porcelain=v1 >.git/expect.$fn + else + git status --porcelain=v1 >.git/actual.$fn && + test_cmp .git/expect.$fn .git/actual.$fn + fi + ' +} + +uc_values="false" +test_have_prereq UNTRACKED_CACHE && uc_values="false true" +for uc_val in $uc_values +do + if test $uc_val = false + then + test_expect_success "Matrix[uc:$uc_val] disable untracked cache" ' + git config core.untrackedcache false && + git update-index --no-untracked-cache + ' + else + test_expect_success "Matrix[uc:$uc_val] enable untracked cache" ' + git config core.untrackedcache true && + git update-index --untracked-cache + ' + fi + + fsm_values="false true" + for fsm_val in $fsm_values + do + if test $fsm_val = false + then + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor" ' + test_unconfig core.fsmonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop + ' + else + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] enable fsmonitor" ' + git config core.fsmonitor true && + git fsmonitor--daemon start && + git update-index --fsmonitor + ' + fi + + matrix_try $uc_val $fsm_val edit_files + matrix_try $uc_val $fsm_val delete_files + matrix_try $uc_val $fsm_val create_files + matrix_try $uc_val $fsm_val rename_files + matrix_try $uc_val $fsm_val file_to_directory + matrix_try $uc_val $fsm_val directory_to_file + + if test $fsm_val = true + then + test_expect_success "Matrix[uc:$uc_val][fsm:$fsm_val] disable fsmonitor at end" ' + test_unconfig core.fsmonitor && + git update-index --no-fsmonitor && + test_might_fail git fsmonitor--daemon stop + ' + fi + done +done + test_done