From patchwork Wed Feb 3 15:34:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 83C5BC433E0 for ; Wed, 3 Feb 2021 15:38:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 420BA64F58 for ; Wed, 3 Feb 2021 15:38:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234440AbhBCPij (ORCPT ); Wed, 3 Feb 2021 10:38:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234463AbhBCPgR (ORCPT ); Wed, 3 Feb 2021 10:36:17 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 40ABDC06178A for ; Wed, 3 Feb 2021 07:34:55 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id c12so24873424wrc.7 for ; Wed, 03 Feb 2021 07:34:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=lqKy4hVMWrTskZKeFtm7FEKoU1VgE+JBqwUCMTIqmOc=; b=Hikr5YIWfEm8+My0Mr97/7LizCoq9h9FYhJFUT1zg9jkLhHKI3n7R8oKI/o3sEjvbK zjQPImB/nC5BYdNk40YFPf+DaPHb9yZ3TcR29rw2m7yV5qwo4LTc+CmRBEWSARYKKQig XruYR1IeOMhOPg1hA84VPoVg2Kd4FlHh2zrvj9MW+MBih3p7b5IELxKXC58WlysKOqtv IFMhK8ttswfYCgv/iX8h1X4xmfDD+Pb6GjqLov5OvH9AheqpHGVm9SmlOkQgK8SUSIzp UObSZCprAcIvFoGqe2POcrigZnVzs9gGa5nUPi2qwNpgYX2e2L4wPc2cqHwwgCqHS3a8 6HWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=lqKy4hVMWrTskZKeFtm7FEKoU1VgE+JBqwUCMTIqmOc=; b=Fzn18xUs973JVn9TeiLorTdDznN5qq4KEn5ToYFobGFFsJWcaPvfbN6haNT8iQhBja +q2pbdDQTInAYScCazCnsp8PfFKpQn7vthHdSnAD0BM6wZX5490gVcNGgynut0z53AYl nojHZosbLxVvVFh/Yby43lecpgJm0Ji1pcnjlaxAKrm6/DckU4aCQVckKyFodgiww5W5 pOOEYK2ARLo7utI8+Q40ut1hTTtlaSxfJZCHu+JW5Y7ao3oj+zC+U2GlM6eF0KwxQU24 H/LDAK5TcqJS5WtPBZuOnVWdNLaTgDOhc6hnsJVNytVUFKQX2VmAkmEVri+fMMS5F0tq k1Vw== X-Gm-Message-State: AOAM531Bb4Mrhd7XxmUQd8/8N0UpbUt3ieVn+fKFUwFbgcoVMPBRwa6i 4tsErDxH01APkrAV+N7chTMVjYvp6l4= X-Google-Smtp-Source: ABdhPJyhbGQLLi7kaxs7HmSEZCCwMkA6SIaS1k+1GnxV2gQ/ZMN3yqWrC+4hxewf0RgI5UnSZB1mSA== X-Received: by 2002:a5d:6282:: with SMTP id k2mr4307182wru.159.1612366493880; Wed, 03 Feb 2021 07:34:53 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c62sm3162762wmd.43.2021.02.03.07.34.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:34:53 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:40 +0000 Subject: [PATCH v2 01/11] p7519: do not rely on "xargs -d" in test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Convert the test to use a more portable method to update the mtime on a large number of files under version control. The Mac version of xargs does not support the "-d" option. Likewise, the "-0" and "--null" options are not portable. Furthermore, use `test-tool chmtime` rather than `touch` to update the mtime to ensure that it is actually updated (especially on file systems with only whole second resolution). Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 9b43342806b..6677e0ef7ab 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -164,8 +164,18 @@ test_fsmonitor_suite() { git status -uall ' + # Update the mtimes on upto 100k files to make status think + # that they are dirty. For simplicity, omit any files with + # LFs (i.e. anything that ls-files thinks it needs to dquote). + # Then fully backslash-quote the paths to capture any + # whitespace so that they pass thru xargs properly. + # test_perf_w_drop_caches "status (dirty) ($DESC)" ' - git ls-files | head -100000 | xargs -d "\n" touch -h && + git ls-files | \ + head -100000 | \ + grep -v \" | \ + sed '\''s/\(.\)/\\\1/g'\'' | \ + xargs test-tool chmtime -300 && git status ' From patchwork Wed Feb 3 15:34:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70D11C433E0 for ; Wed, 3 Feb 2021 15:39:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 17D6E64DF6 for ; Wed, 3 Feb 2021 15:39:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234598AbhBCPiu (ORCPT ); Wed, 3 Feb 2021 10:38:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234467AbhBCPgR (ORCPT ); Wed, 3 Feb 2021 10:36:17 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 255D0C06178B for ; Wed, 3 Feb 2021 07:34:56 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id a1so24872580wrq.6 for ; Wed, 03 Feb 2021 07:34:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=33gxJd1Jklf/uB/iz/5tsYS9j18tcal8oRvrrvPdTOw=; b=hbdLGhzWf+qvVV+xb9EL17bjnfs6OTh/lBsRqArzvse+gMganj1SYoU0kdRIebqs3r x7SVRIyXPIjfbdaEydM7YvyaAtr9y0BXcr2Lkmbfr3rDwZ38Ju4LaWE3uoBbzbWLcfx8 v9L57gK7ifAFae5MmGO6DnenmKZWvGa/UGv03lSmAjOFB90zH4KT6CzUhmaf1OQrSb2r r8d01h+U1HDF4qeC5aYtR7Fq09OFgCEAceqRCtkrlod0qzpT4nS1ej+zGZyGR9grX2ti b5DG558AcgoyBLlvxwrhJjwshalxbb2tQao59jwrw0te0gnVHSGaNcABLUFRTjHHSULU lpdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=33gxJd1Jklf/uB/iz/5tsYS9j18tcal8oRvrrvPdTOw=; b=cB3q3FGjml8iWPUG1wNLlJFux46QeC2w6vN+bH3zDj8ONgSaIDV+qqDSyPXvTBkPZA f/1pBjcmh1oHPhrnm7yfydiDEl1nqb6splg9H3bju4AE1Gr0rqm1iWBhzcBM06tVE794 35/pHFXiah8otV/qCTuywQNeMri4Sh/BafRUzVgjCX2SvLY2XPazZpuUey6JbL7WNPBX GHlQxWCsNMBOSIs/YmCsuGGQ0OHmHr+4HMoLiJdvpks3JHQrbMjEfuachccbhhLp0s5w bPBDycGJZGXtgOaFcNknTikl3sYg8BbKtyAQL87sNUyQXEazkmIUEHHc6J3mzXFO4hSJ IzTQ== X-Gm-Message-State: AOAM533GGeAqYbGofmCeyQtEFCVsk1+XVDJWmw6SbtZMS2yujf6emM7V XSiVsXv4E/xXXcnHk9lLpAxIhJ/976k= X-Google-Smtp-Source: ABdhPJzOnPM/HHj0cGdkWeT85o5vZ2c/Y9onb1zTZmVTp/+k5BB9dDl6rbxpsgUWIMiJZuSsx2IWZw== X-Received: by 2002:a5d:4a0d:: with SMTP id m13mr4192434wrq.395.1612366494773; Wed, 03 Feb 2021 07:34:54 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a24sm2269797wmj.23.2021.02.03.07.34.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:34:54 -0800 (PST) Message-Id: <3042fc92fe6d5ff05fd2e4ac96ef936a69b2a2ff.1612366490.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:41 +0000 Subject: [PATCH v2 02/11] p7519: fix watchman watch-list test on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Only use the final portion of the test trash directory file name when verifying that Watchman was started. On Windows and under the SDK, $GIT_WORKTREE is a cygwin-style path with forward slashes and a "/c/" drive name. However `watchman watch-list` reports a proper Windows-style pathname with drive letters and backslashes. This causes the grep to fail. Since we don't really care about the full pathname (and we really don't want to bother with normalizaing them), just see if the test-name portion of the path is found. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 6677e0ef7ab..21d525541d5 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -101,7 +101,7 @@ test_expect_success "one time repo setup" ' # If Watchman exists, watch the work tree and attempt a query. if test_have_prereq WATCHMAN; then watchman watch "$GIT_WORK_TREE" && - watchman watch-list | grep -q -F "$GIT_WORK_TREE" + watchman watch-list | grep -q -F "p7519-fsmonitor" fi ' From patchwork Wed Feb 3 15:34:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C78C433E0 for ; Wed, 3 Feb 2021 15:38:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6DE9F64F8D for ; Wed, 3 Feb 2021 15:38:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233847AbhBCPhh (ORCPT ); Wed, 3 Feb 2021 10:37:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41474 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234466AbhBCPga (ORCPT ); Wed, 3 Feb 2021 10:36:30 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 413FEC06178C for ; Wed, 3 Feb 2021 07:34:57 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id z6so24798518wrq.10 for ; Wed, 03 Feb 2021 07:34:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=4JdvzMteIxlWXUcVb8uLqJSuNeM/NiZeS3hpk/UWicg=; b=orT0VTJqnGGKZVLjb53Jn1EVLDYcrKjoOEUWl3P4B0ZluSJPrs7xCuHa3HABzqibMD hYIUy2OeMNHXttmUxvc/W4EP+ilN0tUSEzmbjzOmqq6ZQVwpMTobrahKIbWzPoTbZimg nFkQkisFPsYy/IeTXsSqHvZnvM214Y9542wq0wU0Eu51+i6r51bxwSbBmleIJB0ts4+s qK9w9fU/ZHWm0+2gitWB0HkBetbbL8S8ap1Xf8qXaanhp/2xN/Hd95IGE1qxH11G2RM3 R0Rr8aTUz4fnyGRGT6UU/a2edDD9PCHe8SvqXOOn/NzQSVCyuaSDeyyJJx/DJrevFf0q /YbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=4JdvzMteIxlWXUcVb8uLqJSuNeM/NiZeS3hpk/UWicg=; b=DhLcK6QXEXtfiWTk9eSh4LMZ1PCLLsxbjCsZ2/5M/vuTZ2w/WAw93fOxUpmI1RHKVY vDt0HNTQlRgxzmPNM2bU2PKYIEgT7G1QYscWsR3CqJ9iQprZRPxLgyB8161aqjozJIsw C7KkrReWFkDyKHiOGn5fc7mOl2Y6DNJbVaABOkpMfJoWj9gyNMYMVfrpEw8ZKQLMY7PE eIgorO58sbEfnTN0W1AhoKHdWT9ywQslthLPdyouQWDzdO2psrZbhg4jHb6zONV3Izz1 jnwyvYlURc6z+9bZu1Yd1v3TFVEGqSAAeycx9L0UMlNj6GDrv43gRhTvfOY/C4NuHTP2 gFCw== X-Gm-Message-State: AOAM532/MIkeCeHug0SeUDnYFzHXfwe28ujpMq81Sces056ZMlZiKVwa dOicbQRwaVkAE86E9Zbx7qPabAlimk4= X-Google-Smtp-Source: ABdhPJxnr5QZZehbTrhe49fFf/mg24zob9FZwtwch4vd1P8bVS9YG+X+74LRHMrt8ZasYdd3S8Pskg== X-Received: by 2002:adf:92a6:: with SMTP id 35mr4228898wrn.193.1612366495743; Wed, 03 Feb 2021 07:34:55 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m18sm4244878wrx.17.2021.02.03.07.34.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:34:55 -0800 (PST) Message-Id: <9ceba5e69420f12eeb834f387372520b061e28a7.1612366490.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:42 +0000 Subject: [PATCH v2 03/11] p7519: move watchman cleanup earlier in the test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Shutdown Watchman after the Watchman-based tests and before the block of "no fsmonitor" tests. This helps ensure that Watchman cannot affect the test results for the other. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 21d525541d5..78e7d2c03f5 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -208,6 +208,11 @@ test_fsmonitor_suite() { ' } +# +# Run a full set of perf tests using each Hook-based fsmonitor provider, +# such as Watchman. +# + if test -n "$GIT_PERF_7519_FSMONITOR"; then for INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do test_expect_success "setup for fsmonitor $INTEGRATION_PATH" 'setup_for_fsmonitor' @@ -218,14 +223,6 @@ else test_fsmonitor_suite fi -test_expect_success "setup without fsmonitor" ' - unset INTEGRATION_SCRIPT && - git config --unset core.fsmonitor && - git update-index --no-fsmonitor -' - -test_fsmonitor_suite - if test_have_prereq WATCHMAN then watchman watch-del "$GIT_WORK_TREE" >/dev/null 2>&1 && @@ -235,4 +232,16 @@ then watchman shutdown-server >/dev/null 2>&1 fi +# +# Run a full set of perf tests with the fsmonitor feature disabled. +# + +test_expect_success "setup without fsmonitor" ' + unset INTEGRATION_SCRIPT && + git config --unset core.fsmonitor && + git update-index --no-fsmonitor +' + +test_fsmonitor_suite + test_done From patchwork Wed Feb 3 15:34:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B648C433DB for ; Wed, 3 Feb 2021 15:38:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 217B064F5D for ; Wed, 3 Feb 2021 15:38:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233520AbhBCPi2 (ORCPT ); Wed, 3 Feb 2021 10:38:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41476 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234483AbhBCPga (ORCPT ); Wed, 3 Feb 2021 10:36:30 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 410C0C061793 for ; Wed, 3 Feb 2021 07:34:58 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id u14so2992263wri.3 for ; Wed, 03 Feb 2021 07:34:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uXeiDFAzN5lHNKXhs/cPKyCyGEjIOX0UeKbHlVFvS6U=; b=BZHA7l4OwGze3QNT+R6iBYyOP6teTBC40PMk0cB3Ac+cWPOurJ7dEu5n8sbdWnageb 3/EjjLCm6iyrd3Nveev20SuSPVetsaZuHmGbb9BubTjuVcnhv4gbIFHU+7RuvxnQiYqF jY+oOCqkylJtttf9N3F1pHPpygo3zx/WEFb+lUXlHv/RgzxIKDhAna98vcM8hVrW+4dt kGQ0KPLeC2/qPfISCJzaPZzFJ+M6ya+iOf7LceZXdhOaLmbYvjo23jNvLc9H5Q/HGMtX efO0UNkQJd81gOVEAWCIDoHbhyrsZNJV6yb7vbV0rlW+EwU0sWspEahfaTy1OJRiXZPk UT4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uXeiDFAzN5lHNKXhs/cPKyCyGEjIOX0UeKbHlVFvS6U=; b=IZEKgpnDjvcTNoX0KDwvl33SSh1Cy66urwWRZQJBZ7+QIgtBJH51aG+Pt0EVsTTueN PDhJbQVWguufSWqRcO0cBpcs1E/pERKw07EjZodr6p3d9NNh2bWzTb7OkoxReMgioE6V ico6svBF7dIhViK0YoF4yNPgsYF+b6wjuHXOJ7t0+kwlpsoRbNUigvRZ39ufFYiSvYsM 4V2Wo9dPTsjT5o04w9vyeYefZzdqxPgZzWQht9zFFD+O9K1Ir+w/BnXTXckd05KumHO8 D3WcBPPMtCY7U4DuR4F1nCPIM0mMtu62LUONfO2/yNoLG7FkQmROeMDZnFVU4dsDcJ5O u5pw== X-Gm-Message-State: AOAM530GHAH0Wlnub9KUFfoyyPktPRD7nLtT30sLy2+yYpR9zb9SG7jI StcAi+oWaYoAY76F5jhVib+plOklt7Q= X-Google-Smtp-Source: ABdhPJzWqomelP+AQcBFGGZ85x/Yo9wP1GSglARYaFfVzFUyVYx/GpwQr85GBPkaOTDnEKLOS9PEcw== X-Received: by 2002:a5d:6d06:: with SMTP id e6mr4237825wrq.425.1612366496790; Wed, 03 Feb 2021 07:34:56 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n16sm3244070wmi.5.2021.02.03.07.34.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:34:56 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:43 +0000 Subject: [PATCH v2 04/11] p7519: add trace logging during perf test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Add optional trace logging to allow us to better compare performance of various fsmonitor providers and compare results with non-fsmonitor runs. Currently, this includes Trace2 logging, but may be extended to include other trace targets, such as GIT_TRACE_FSMONITOR if desired. Using this logging helped me explain an odd behavior on MacOS where the kernel was dropping events and causing the hook to Watchman to timeout. Signed-off-by: Jeff Hostetler --- t/perf/.gitignore | 1 + t/perf/Makefile | 4 ++-- t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/t/perf/.gitignore b/t/perf/.gitignore index 982eb8e3a94..72f5d0d3148 100644 --- a/t/perf/.gitignore +++ b/t/perf/.gitignore @@ -1,3 +1,4 @@ /build/ /test-results/ +/test-trace/ /trash directory*/ diff --git a/t/perf/Makefile b/t/perf/Makefile index fcb0e8865e4..2465770a782 100644 --- a/t/perf/Makefile +++ b/t/perf/Makefile @@ -7,10 +7,10 @@ perf: pre-clean ./run pre-clean: - rm -rf test-results + rm -rf test-results test-trace clean: - rm -rf build "trash directory".* test-results + rm -rf build "trash directory".* test-results test-trace test-lint: $(MAKE) -C .. test-lint diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 78e7d2c03f5..aa0ea0e2ec8 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -32,6 +32,8 @@ test_description="Test core.fsmonitor" # # GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests # +# GIT_PERF_7519_TRACE: if set, enable trace logging during the test. +# Trace logs will be grouped by fsmonitor provider. test_perf_large_repo test_checkout_worktree @@ -70,6 +72,32 @@ then fi fi +trace_start() { + if test -n "$GIT_PERF_7519_TRACE" + then + name="$1" + TEST_TRACE_DIR="$TEST_OUTPUT_DIRECTORY/test-trace/p7519/" + echo "Writing trace logging to $TEST_TRACE_DIR" + + mkdir -p "$TEST_TRACE_DIR" + + # Start Trace2 logging and any other GIT_TRACE_* logs that you + # want for this named test case. + + GIT_TRACE2_PERF="$TEST_TRACE_DIR/$name.trace2perf" + export GIT_TRACE2_PERF + + >"$GIT_TRACE2_PERF" + fi +} + +trace_stop() { + if test -n "$GIT_PERF_7519_TRACE" + then + unset GIT_TRACE2_PERF + fi +} + test_expect_success "one time repo setup" ' # set untrackedCache depending on the environment if test -n "$GIT_PERF_7519_UNTRACKED_CACHE" @@ -213,6 +241,7 @@ test_fsmonitor_suite() { # such as Watchman. # +trace_start fsmonitor-watchman if test -n "$GIT_PERF_7519_FSMONITOR"; then for INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do test_expect_success "setup for fsmonitor $INTEGRATION_PATH" 'setup_for_fsmonitor' @@ -231,11 +260,13 @@ then # preventing the removal of the trash directory watchman shutdown-server >/dev/null 2>&1 fi +trace_stop # # Run a full set of perf tests with the fsmonitor feature disabled. # +trace_start fsmonitor-disabled test_expect_success "setup without fsmonitor" ' unset INTEGRATION_SCRIPT && git config --unset core.fsmonitor && @@ -243,5 +274,6 @@ test_expect_success "setup without fsmonitor" ' ' test_fsmonitor_suite +trace_stop test_done From patchwork Wed Feb 3 15:34:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064651 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1347AC43381 for ; Wed, 3 Feb 2021 15:52:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BA7F364F55 for ; Wed, 3 Feb 2021 15:52:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234638AbhBCPwf (ORCPT ); Wed, 3 Feb 2021 10:52:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41632 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234550AbhBCPhN (ORCPT ); Wed, 3 Feb 2021 10:37:13 -0500 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C2D1C061794 for ; Wed, 3 Feb 2021 07:34:59 -0800 (PST) Received: by mail-wr1-x432.google.com with SMTP id z6so24798719wrq.10 for ; Wed, 03 Feb 2021 07:34:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JewujSwXcnrJrGpQL58DN1s7LbKqF3XLqwqHhLqPPEA=; b=heqe3bwy+ZKUhqXD0bSjNerrcFsOcJXUuzuZTO0GU2qzypk4FfTT0rTpVvVzC46eAl sO2xLtqiTYs7gJuEvSXavlU6YDvVH7a3OQCrJy2ENM1eNfnhW19k1i0p+q158hvC3uhr 2MCQkzfmaLtCYs/bmYKsdrJL876sR6Lw9vQYeByaydXVFqvohDfnPdOoF40+0nJXyqPm Zr9uU2asIgQq5WsQz/iLT9jcxCkwCW0YZAQsnrJJA2uyqhSzKeG75yqBf5RnLi71fZsK Vk0VNxPFr1A06w1MUy9RYzgy+Zef2XSGLpQ4jqXj2BXeKoODk0Rylc9HAV75LWjVj1lF wpOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JewujSwXcnrJrGpQL58DN1s7LbKqF3XLqwqHhLqPPEA=; b=Dz8uFKh3ZTWlh7eCWCXj3+zQGD21+o1FZlUfajZGYUvkNB+PI/IDbumr83BVDWqEGI UkTc178In8OfglMTweRWh1GIvBJgA12Ww0lGoNnFegPrZu9xuaSFzPMg+iQh9NR0eQSZ Ln1kEzecigG6cKFmoxXPvz8VlBdKjSiOjkiGpICJjATVhrYJFTO0+aYZ1C3vsQS2ZSqm CHtESkhoV5/qguG5L0blulI5OEMX1Qnz97FbT6SI9pUD1JsvJyglIXV1eR6+rMCemqkL DXT2JsjAGwQi8mIpRPAfVh4VuTPblHH2pc+9ojhE60iQ+yTy/H5HykTrl1ZXzFyvjQNb vPHQ== X-Gm-Message-State: AOAM532z2bW9SM0phnh2+Da4QqUiAUN310j+NbEnMjYAWvuMFaeox0r7 xsAX6sAKzkqTkOlRZUEbD2mhv2HluWM= X-Google-Smtp-Source: ABdhPJzTTlp3Cd43395vAAp2iZKmrHVFLE/d8th86iry4XD4AtOPslS5O61q7gn+HiAL7vWdGSfh8Q== X-Received: by 2002:a5d:554e:: with SMTP id g14mr4211734wrw.305.1612366497930; Wed, 03 Feb 2021 07:34:57 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v25sm3355487wmh.4.2021.02.03.07.34.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:34:57 -0800 (PST) Message-Id: <3c5035e4649d18d96fc0206c2ceacc6ffe99deff.1612366490.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:44 +0000 Subject: [PATCH v2 05/11] preload-index: log the number of lstat calls to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Report the total number of calls made to lstat() inside preload_index(). FSMonitor improves the performance of commands like `git status` by avoiding scanning the disk for changed files. This can be seen in `preload_index()`. Let's measure this. Signed-off-by: Jeff Hostetler --- preload-index.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/preload-index.c b/preload-index.c index ed6eaa47388..e5529a58636 100644 --- a/preload-index.c +++ b/preload-index.c @@ -31,6 +31,7 @@ struct thread_data { struct pathspec pathspec; struct progress_data *progress; int offset, nr; + int t2_nr_lstat; }; static void *preload_thread(void *_data) @@ -73,6 +74,7 @@ static void *preload_thread(void *_data) continue; if (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce))) continue; + p->t2_nr_lstat++; if (lstat(ce->name, &st)) continue; if (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR)) @@ -98,6 +100,7 @@ void preload_index(struct index_state *index, int threads, i, work, offset; struct thread_data data[MAX_PARALLEL]; struct progress_data pd; + int t2_sum_lstat = 0; if (!HAVE_THREADS || !core_preload_index) return; @@ -107,6 +110,9 @@ void preload_index(struct index_state *index, threads = 2; if (threads < 2) return; + + trace2_region_enter("index", "preload", NULL); + trace_performance_enter(); if (threads > MAX_PARALLEL) threads = MAX_PARALLEL; @@ -141,10 +147,14 @@ void preload_index(struct index_state *index, struct thread_data *p = data+i; if (pthread_join(p->pthread, NULL)) die("unable to join threaded lstat"); + t2_sum_lstat += p->t2_nr_lstat; } stop_progress(&pd.progress); trace_performance_leave("preload index"); + + trace2_data_intmax("index", NULL, "preload/sum_lstat", t2_sum_lstat); + trace2_region_leave("index", "preload", NULL); } int repo_read_index_preload(struct repository *repo, From patchwork Wed Feb 3 15:34:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A21C2C433DB for ; Wed, 3 Feb 2021 15:40:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5261564DF6 for ; Wed, 3 Feb 2021 15:40:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234609AbhBCPj0 (ORCPT ); Wed, 3 Feb 2021 10:39:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41634 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234551AbhBCPhN (ORCPT ); Wed, 3 Feb 2021 10:37:13 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4177BC061797 for ; Wed, 3 Feb 2021 07:35:00 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id m13so24880449wro.12 for ; Wed, 03 Feb 2021 07:35:00 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fkxMogoHr5h4CV44ECW6UTWZy4JBxE27oGdnxv+/Kac=; b=SS0P3BjWIdF+bHUn47e6NMVo6vJKuTzFbzsq1YHzNAzmtNO92UJmM2zGW9TmOgGT6D ZneZlienjn/0efIhYkRzRC/VGIDywVZi53g26ArBkWJCLQirRzY7Ab9B7BaXSS3+0xPP YUr+j1sqkpzlhARcpKekZzoRIVhri3ebypbc9ZS9KuI2fZ6hFWnVaEpOhsNIv14LZM5g b1Sq91GNScwXfyXLqPGtrsuyYal1UYH5VoVVHTy6/PhznecznkdtPsKzr987awSWvAR7 7uyIFfDwJxRH+U3NG2puzUcH81ZgGNfeu98jRGNtbLpE+NPc8Q0zecusFQbm7S7toz30 0PgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fkxMogoHr5h4CV44ECW6UTWZy4JBxE27oGdnxv+/Kac=; b=cUocNaQOFjEnlhyxEvpGz77zTqsIV26bkT8J9ZZ6w/qHqftxtsFT2iKpbNo3lkOKbX z9z094mUIkTbKlkR/dgqKEftSaorN7jPIYllwkjc1ZnZEeloUwCZVTSGoN0GDgVQoBD4 XwQgMlVCa07zNVezzepuV70hFkXxwWgPpmm3T/IrTbVBgdQ4w70Ya8tkLlu82Io1b91C oMk9vK7oMsNYFtRDosMHmBkzaRjm3QMs84BcpKoAl4U+QrymQ2hVBkLMLSrpvZhcVqEW xdZgMfE0+HuF2wpgEEdyYoJhEFCo+LO1Iqwzc1nmJNieuVerbHlY9USGjihiXKXKqpXp Dw3w== X-Gm-Message-State: AOAM532l6nfxlmY5S+Us1jc587Z4o+YYQfjW54FF50vfRXUh1A0kUGkp tBd0RGVfABf5J1fhqZFPzzYRS6IVYSo= X-Google-Smtp-Source: ABdhPJz1iWB37dDjXRd2BeEwDeytstcglRB9Pdz0Bsq1m0nu3MSIqFqVLv5T9yoBuc1xuA8i51SkkQ== X-Received: by 2002:a5d:6092:: with SMTP id w18mr4169446wrt.75.1612366498883; Wed, 03 Feb 2021 07:34:58 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c20sm3053871wmb.38.2021.02.03.07.34.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:34:58 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:45 +0000 Subject: [PATCH v2 06/11] read-cache: log the number of lstat calls to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Report the total number of calls made to lstat() inside of refresh_index(). FSMonitor improves the performance of commands like `git status` by avoiding scanning the disk for changed files. This can be seen in `refresh_index()`. Let's measure this. Signed-off-by: Jeff Hostetler --- read-cache.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/read-cache.c b/read-cache.c index ecf6f689940..893cc41e1d9 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1364,7 +1364,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti static struct cache_entry *refresh_cache_ent(struct index_state *istate, struct cache_entry *ce, unsigned int options, int *err, - int *changed_ret) + int *changed_ret, + int *t2_did_lstat) { struct stat st; struct cache_entry *updated; @@ -1406,6 +1407,8 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate, return NULL; } + if (t2_did_lstat) + *t2_did_lstat = 1; if (lstat(ce->name, &st) < 0) { if (ignore_missing && errno == ENOENT) return ce; @@ -1519,6 +1522,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, const char *added_fmt; const char *unmerged_fmt; struct progress *progress = NULL; + int t2_sum_lstat = 0; if (flags & REFRESH_PROGRESS && isatty(2)) progress = start_delayed_progress(_("Refresh index"), @@ -1536,11 +1540,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, * we only have to do the special cases that are left. */ preload_index(istate, pathspec, 0); + trace2_region_enter("index", "refresh", NULL); for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; int changed = 0; int filtered = 0; + int t2_did_lstat = 0; ce = istate->cache[i]; if (ignore_submodules && S_ISGITLINK(ce->ce_mode)) @@ -1566,7 +1572,10 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (filtered) continue; - new_entry = refresh_cache_ent(istate, ce, options, &cache_errno, &changed); + new_entry = refresh_cache_ent(istate, ce, options, + &cache_errno, &changed, + &t2_did_lstat); + t2_sum_lstat += t2_did_lstat; if (new_entry == ce) continue; if (progress) @@ -1602,6 +1611,8 @@ int refresh_index(struct index_state *istate, unsigned int flags, replace_index_entry(istate, i, new_entry); } + trace2_data_intmax("index", NULL, "refresh/sum_lstat", t2_sum_lstat); + trace2_region_leave("index", "refresh", NULL); if (progress) { display_progress(progress, istate->cache_nr); stop_progress(&progress); @@ -1614,7 +1625,7 @@ struct cache_entry *refresh_cache_entry(struct index_state *istate, struct cache_entry *ce, unsigned int options) { - return refresh_cache_ent(istate, ce, options, NULL, NULL); + return refresh_cache_ent(istate, ce, options, NULL, NULL, NULL); } From patchwork Wed Feb 3 15:34:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48C18C433DB for ; Wed, 3 Feb 2021 15:40:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E0E0164F58 for ; Wed, 3 Feb 2021 15:40:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233482AbhBCPju (ORCPT ); Wed, 3 Feb 2021 10:39:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234580AbhBCPhY (ORCPT ); Wed, 3 Feb 2021 10:37:24 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48C58C0617A7 for ; Wed, 3 Feb 2021 07:35:01 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id f16so38293wmq.5 for ; Wed, 03 Feb 2021 07:35:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EwY+RPdKlOlJBlCoDOs+KUc/NkIJWwGIjdTt04+8VTw=; b=FwCTxLrAfmFKlb2PZ+ixgcn7lo8RjYxdRLBnSMw+7nNYi8WzrvYVbgFoxD89OaT/h6 j90wyUVLiHuDln4gLLIGOdIgkGIBGiJaUHnRZAjsZnLgwxDezQHx5xEm3+kFjC5EZ8+C EfaTrU3j1DZmMUofRbF7iFgzB66eTzoIIYFpuNlGvJ/gLgf41nfgUS7uOe7ZDqctG7bB Ti/q70CINpeTG18K71FDzX11aqs9iTeCqpGQ3BYRhm6FwbfSh1ULOdS/BtybfU4yCJjL 3/D1jXO4q+fKYcOT9qym6rFXZ+GMF1ldeO8eQF2UuOO7/wwTNy+HKo+sEISjDvWLzrZa b20Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EwY+RPdKlOlJBlCoDOs+KUc/NkIJWwGIjdTt04+8VTw=; b=uUw7QdxmlIz55V7TG0Egyl3juzYmc6c0b916XxrzS1lYJnM33q3kYqVPH+7jcaLoIp p/qmlHeK0mENnz6zHjI/28X7OszIE9bQZQplPkJl3HLFfH9VcuAZi0L/c/fFS49fJp2m iJeCmX/+YNSGPBbu6zrKmvBOP//ivrm8E7L7SdLIAEx6HBwif06Fh5OYLIPbm8FXlZz1 RoyPPh2x1L1dpfX+vLNkjbWMcwUiodl3XOafkFCYVNFFPtJJ1jQyBzauI3LCTNExaZAU q6bHYgCpn5OpBrmakvxrpTWFNH/AbS1WwkezBorsSyzsqFkoo4PrwY2evk5ss3Yr5j/6 RISQ== X-Gm-Message-State: AOAM531PiRnugh/O5dOVWHSCC3N8mVswiHtOcoE0KeFWWFT2DKNZ2LL6 SkSq/JMI9XX9/s1HS5BZzc40Sg1m54I= X-Google-Smtp-Source: ABdhPJycqrYtXZRaXjkrBHMUPkvClP5eeaYALe+afygsB/sFP3OuhkmVZgwtv0szub9h1i4foiJd6Q== X-Received: by 2002:a1c:25c2:: with SMTP id l185mr3352026wml.62.1612366499830; Wed, 03 Feb 2021 07:34:59 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 143sm1602467wmb.47.2021.02.03.07.34.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:34:59 -0800 (PST) Message-Id: <33cc0b838fadcfd8fc9cf769d77b83f3775e363f.1612366490.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:46 +0000 Subject: [PATCH v2 07/11] read-cache: log the number of scanned files to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Report the number of files in the working directory that were read and their hashes verified in `refresh_index()`. FSMonitor improves the performance of commands like `git status` by avoiding scanning the disk for changed files. Let's measure this. Signed-off-by: Jeff Hostetler --- read-cache.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/read-cache.c b/read-cache.c index 893cc41e1d9..c9dd7f4015e 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1365,7 +1365,8 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate, struct cache_entry *ce, unsigned int options, int *err, int *changed_ret, - int *t2_did_lstat) + int *t2_did_lstat, + int *t2_did_scan) { struct stat st; struct cache_entry *updated; @@ -1445,6 +1446,8 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate, } } + if (t2_did_scan) + *t2_did_scan = 1; if (ie_modified(istate, ce, &st, options)) { if (err) *err = EINVAL; @@ -1523,6 +1526,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, const char *unmerged_fmt; struct progress *progress = NULL; int t2_sum_lstat = 0; + int t2_sum_scan = 0; if (flags & REFRESH_PROGRESS && isatty(2)) progress = start_delayed_progress(_("Refresh index"), @@ -1547,6 +1551,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, int changed = 0; int filtered = 0; int t2_did_lstat = 0; + int t2_did_scan = 0; ce = istate->cache[i]; if (ignore_submodules && S_ISGITLINK(ce->ce_mode)) @@ -1574,8 +1579,9 @@ int refresh_index(struct index_state *istate, unsigned int flags, new_entry = refresh_cache_ent(istate, ce, options, &cache_errno, &changed, - &t2_did_lstat); + &t2_did_lstat, &t2_did_scan); t2_sum_lstat += t2_did_lstat; + t2_sum_scan += t2_did_scan; if (new_entry == ce) continue; if (progress) @@ -1612,6 +1618,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, replace_index_entry(istate, i, new_entry); } trace2_data_intmax("index", NULL, "refresh/sum_lstat", t2_sum_lstat); + trace2_data_intmax("index", NULL, "refresh/sum_scan", t2_sum_scan); trace2_region_leave("index", "refresh", NULL); if (progress) { display_progress(progress, istate->cache_nr); @@ -1625,7 +1632,7 @@ struct cache_entry *refresh_cache_entry(struct index_state *istate, struct cache_entry *ce, unsigned int options) { - return refresh_cache_ent(istate, ce, options, NULL, NULL, NULL); + return refresh_cache_ent(istate, ce, options, NULL, NULL, NULL, NULL); } From patchwork Wed Feb 3 15:34:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064637 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 445B2C433DB for ; Wed, 3 Feb 2021 15:40:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EB9E164F5D for ; Wed, 3 Feb 2021 15:40:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234552AbhBCPku (ORCPT ); Wed, 3 Feb 2021 10:40:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41674 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234581AbhBCPhY (ORCPT ); Wed, 3 Feb 2021 10:37:24 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6D2C4C0617A9 for ; Wed, 3 Feb 2021 07:35:02 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id v15so24895998wrx.4 for ; Wed, 03 Feb 2021 07:35:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+KteHt2BWR//iqZgex9DKa/luwnSNm9uhzmwsnCG0fE=; b=i8jLYMoH9MAF7drPuad5Nz4gOnE5G+n0V4L4HvLw4+tq/+43k2tiKdXIuMZtKXz7cS S6FUxbVxZ2f9Mws9meFYd3uJtRaHUTIWMPkB1xdiBh4asWy6srZEkoRqTmWTXBr0izM6 hRPY4P8CeYX+4QuiQpeN4BQM+bb+/k+gBK+NB8390s86+pJk7J10mL6SvzcU6nF21Z84 QSzXm33Lh5wDbvcXSWrvpt+SdTOKDlrxKmAyzhd7H0k9MYTcvKuz9jfsVsAxEy/vFIKh BnBfeHAWLoI0b1Kul18fRVOHEBNBNLRBklvWbR3PiWLfl/lr73OugEYe0OmRYtEh0B6t gAUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+KteHt2BWR//iqZgex9DKa/luwnSNm9uhzmwsnCG0fE=; b=FsRXfSmiGGeSTRZ5lJSuWUwryxOuOJopnra80T6e60Zp/vgWlqn+cCrQsSzfHd0OR4 5xeTJf9Fl1XKLytp1F3mx55HZKS0j3iyGFNlaUua1N+S7CoFWUzUNoEzo1MdKCJGmQEN hmIYlaIP89+u38dzbbI9lFuHloDpKOfTohzXkk05nDlraq5AHiejIejzrKF5IEI57wMG DM7K6PuJw33Toen5RjFLp+vzL4/adTLD1K6TNLNVSwZPVvcZJmzcS5nVO8zB2SB9nFsi /V2O2B690mI/2RMnMo4Avi1rxQSKjxKHLa0k+QXQtnrQN6a0FomPgeuBemzpEIUVn5UJ SxFw== X-Gm-Message-State: AOAM533YJ4rf6e0eEMT2w6S576dXN2IdziSea1t/oJ7BqpB4JVRnFtHh wyzYaPK+gYfdvJlO0C/Exo4s6bKT9as= X-Google-Smtp-Source: ABdhPJySfk2DRJSCKE0M3J8cFgibvXTJQjw3DvIPbhN1cQT0jlX6a2nYOeVLbaFa1W8ziB9OzV4nsw== X-Received: by 2002:adf:f149:: with SMTP id y9mr4175152wro.144.1612366501024; Wed, 03 Feb 2021 07:35:01 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t18sm4190535wrr.56.2021.02.03.07.35.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:35:00 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:47 +0000 Subject: [PATCH v2 08/11] fsmonitor: log invocation of FSMonitor hook to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Let's measure the time taken to request and receive FSMonitor data via the hook API and the size of the response. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 29 ++++++++++++++++++++++++++++- fsmonitor.h | 5 +++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/fsmonitor.c b/fsmonitor.c index ca031c3abb8..7a2be24cd43 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -142,6 +142,7 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate) static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result) { struct child_process cp = CHILD_PROCESS_INIT; + int result; if (!core_fsmonitor) return -1; @@ -152,7 +153,33 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf * cp.use_shell = 1; cp.dir = get_git_work_tree(); - return capture_command(&cp, query_result, 1024); + trace2_region_enter("fsm_hook", "query", NULL); + + result = capture_command(&cp, query_result, 1024); + + if (result) + trace2_data_intmax("fsm_hook", NULL, "query/failed", result); + else { + trace2_data_intmax("fsm_hook", NULL, "query/response-length", + query_result->len); + + if (fsmonitor_is_trivial_response(query_result)) + trace2_data_intmax("fsm_hook", NULL, + "query/trivial-response", 1); + } + + trace2_region_leave("fsm_hook", "query", NULL); + + return result; +} + +int fsmonitor_is_trivial_response(const struct strbuf *query_result) +{ + static char trivial_response[3] = { '\0', '/', '\0' }; + int is_trivial = !memcmp(trivial_response, + &query_result->buf[query_result->len - 3], 3); + + return is_trivial; } static void fsmonitor_refresh_callback(struct index_state *istate, const char *name) diff --git a/fsmonitor.h b/fsmonitor.h index 739318ab6d1..7f1794b90b0 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -44,6 +44,11 @@ void tweak_fsmonitor(struct index_state *istate); */ void refresh_fsmonitor(struct index_state *istate); +/* + * Does the received result contain the "trivial" response? + */ +int fsmonitor_is_trivial_response(const struct strbuf *query_result); + /* * Set the given cache entries CE_FSMONITOR_VALID bit. This should be * called any time the cache entry has been updated to reflect the From patchwork Wed Feb 3 15:34:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064633 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7007CC433DB for ; Wed, 3 Feb 2021 15:40:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 26A7964F5D for ; Wed, 3 Feb 2021 15:40:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234302AbhBCPk2 (ORCPT ); Wed, 3 Feb 2021 10:40:28 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41690 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233001AbhBCPh3 (ORCPT ); Wed, 3 Feb 2021 10:37:29 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AAAAC0617AA for ; Wed, 3 Feb 2021 07:35:03 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id a1so24873170wrq.6 for ; Wed, 03 Feb 2021 07:35:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tUd9omKSffFRsXV7K6/6DEhbnGyQp25YksuiGbwNKKg=; b=Wk+4k6gqM67HFDWRj4rWCuMLoipzuAs9oPJEh9BHmr/RobAgg+YDtvjd5Xk3NPdoJM XCdqJUiwNA0tveW/QaFWrQIyCWqc3AIgIT9IIZ1Y5qUYBv8IVctwYasIiECUBKudQQAd 6Z+cVLKU0REQFD7+L0ka7LLHHC1Ljoctf6RpeneS1AbKrffdKnYRpidjMi9chkBohIY6 kXpEADddWbWpGpep2Bg9xyRVzSri0y0FJXX/Uipv0j0lJgQOiCryT2FR1wJrYrJEhG7t TxcsOnYOGPniLw/pCajMXwtOa4TSl8vvuGJF5wjnfzju7yTXTmKzb06zSxcOV8V6O8av 0c4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tUd9omKSffFRsXV7K6/6DEhbnGyQp25YksuiGbwNKKg=; b=Avt8bHdPAC3jLKGH043qQDPG4mZ16p2hCg+wzg7vW2epl3D9vEqZ0Bd58IgCb4AyTX MEiDJSw+SlWtO5wK/BKvnvBnP/SesnLCYbiEUkjWp5FqpC6f8YhqUySt0cCNcHvOhog7 1bUcWXW7Jzhp8QLqrQg7gmlC/R6uLnjohCpZei8yVkk3pub1DlGWsxjRpJZNBa7k72uK zymRwwQY/UVtgr5UTD1Bxf0EFm8y81f3scLykXXB3v/eHbJz1p0u0suxhPSUmEx9v8Mn zK/xYL5dV3ZytSB3c+MwPJ9QLIq5h4btPxglsmXX9kE7ej74HwU1hq9za5omPYMiYL1n 3gjA== X-Gm-Message-State: AOAM531vW9oaaGTRvyQKSXthVfqwTZIlAl+KyGFVL0dwCthyz5e8bYyG NU6LRXW09IIderk69+2Gbhfpm3/SGrc= X-Google-Smtp-Source: ABdhPJyFM/hkWPDVPMhIgrGsXyhPSSXjPrDu/0AUHwAzFM6lcAZr4nPC8SJMm2mi3v3o9qzyw+XDuA== X-Received: by 2002:a5d:5708:: with SMTP id a8mr4145523wrv.211.1612366501939; Wed, 03 Feb 2021 07:35:01 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m16sm2746765wmq.36.2021.02.03.07.35.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:35:01 -0800 (PST) Message-Id: <6ec4a4468f6a068291c2ad6b1ddb75f68c13c3a8.1612366490.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:48 +0000 Subject: [PATCH v2 09/11] fsmonitor: log FSMN token when reading and writing the index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- fsmonitor.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 7a2be24cd43..3105dc370ab 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -87,7 +87,11 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data, BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)", (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr); - trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful"); + trace2_data_string("index", NULL, "extension/fsmn/read/token", + istate->fsmonitor_last_update); + trace_printf_key(&trace_fsmonitor, + "read fsmonitor extension successful '%s'", + istate->fsmonitor_last_update); return 0; } @@ -133,7 +137,11 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate) put_be32(&ewah_size, sb->len - ewah_start); memcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t)); - trace_printf_key(&trace_fsmonitor, "write fsmonitor extension successful"); + trace2_data_string("index", NULL, "extension/fsmn/write/token", + istate->fsmonitor_last_update); + trace_printf_key(&trace_fsmonitor, + "write fsmonitor extension successful '%s'", + istate->fsmonitor_last_update); } /* From patchwork Wed Feb 3 15:34:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Willford X-Patchwork-Id: 12064635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7EF0C433E0 for ; Wed, 3 Feb 2021 15:40:49 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5501B64DF6 for ; Wed, 3 Feb 2021 15:40:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233962AbhBCPkh (ORCPT ); Wed, 3 Feb 2021 10:40:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41692 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233759AbhBCPh3 (ORCPT ); Wed, 3 Feb 2021 10:37:29 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 56B1FC0617AB for ; Wed, 3 Feb 2021 07:35:04 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id 7so24889484wrz.0 for ; Wed, 03 Feb 2021 07:35:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MgDPgj7GedDHVjjkk/EtAODaaSn6gWGGMNTT5xfwao8=; b=V+mKj7cJJyNtxgGouSqUoTiX7DRys5A6SO1HnXnj5SJuYTVVZNCaqSOYO5Fn2Ydgol xW0bX/iDw3soTZzEk8+YeVLYfL/PGW2zzxNNKGVlCDSEQRj9lSDSGEtNOxcJlfdhOUeM B2Rl2T6gp1qZgDYpIXHou+rixr/W/8Csf0hm7tb7myRcSX4P3leZy+C5XLSLpPEgSgim fdciuKA+xLQjOM7dQOpo9hAolK6DQnM/qzbxKSXpNIj4iAAG5I90C2G+otKAbdaVM1FV o/uZwbxQvSmaQzSiOLwYgB5pciP2GFWt9ZMUARM4goaolJHleRdngKw2uE+7ibZZZQGA vHdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MgDPgj7GedDHVjjkk/EtAODaaSn6gWGGMNTT5xfwao8=; b=uOKeLQi4TL5JpiVJQXWSgn8XXVgznvvTdYNwRpx4kfmZyLVoBccFZTUNj8P8Zoyyi1 Upukpb5Z7RGT6cDD7WBfKjJBYz52XpxCLz7v9aBDyBo16lRTubXIN73aaX6hbAD40sTb wurGHpvAslPldYksLSt67yIrhOTJUmF/gYxxAORbUZ8aOvsVmFllpi7WG/gJbU8QmNLb EZ0uuQUWrMFp/6hxYW0FE1/5uexTgO5RostYkXPnMY4SDC50Kx8Q5b8jrkMT3Oi65uIp bYMMOEAe1Dd+it58MYXZbQ8ilRg+3V77anXBi+neO9tCQSi5Cj350bfAeC0ZmMrm0yBe rWiQ== X-Gm-Message-State: AOAM531Bk2VjG0UmUlIa2mdHk+JJiUKB0eGNgHCK/SlSpt7zf8/p/FuT CIAslXxImSMy1wRuajUGWM5Zr97wa5w= X-Google-Smtp-Source: ABdhPJwjGhcmZwlLENWbnmKe5WjJqoVuD3bJ7Mc/vy5ThvAumzmgfVaRQ2+HEDiotgE7oenr76ZQOg== X-Received: by 2002:a5d:4051:: with SMTP id w17mr4136063wrp.186.1612366502972; Wed, 03 Feb 2021 07:35:02 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q63sm3212479wma.43.2021.02.03.07.35.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:35:02 -0800 (PST) Message-Id: <2ac66f07a59abe84c0105d5d97560fc3b22a9955.1612366490.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:49 +0000 Subject: [PATCH v2 10/11] fsmonitor: allow all entries for a folder to be invalidated Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Kevin Willford Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Kevin Willford From: Kevin Willford Allow fsmonitor to report directory changes by reporting paths with a trailing slash. Signed-off-by: Jeff Hostetler Signed-off-by: Kevin Willford Signed-off-by: Johannes Schindelin --- fsmonitor.c | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 3105dc370ab..64deeda597e 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -190,13 +190,34 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result) return is_trivial; } -static void fsmonitor_refresh_callback(struct index_state *istate, const char *name) +static void fsmonitor_refresh_callback(struct index_state *istate, char *name) { - int pos = index_name_pos(istate, name, strlen(name)); + int i, len = strlen(name); + if (name[len - 1] == '/') { + + /* + * TODO We should binary search to find the first path with + * TODO this directory prefix. Then linearly update entries + * TODO while the prefix matches. Taking care to search without + * TODO the trailing slash -- because '/' sorts after a few + * TODO interesting special chars, like '.' and ' '. + */ + + /* Mark all entries for the folder invalid */ + for (i = 0; i < istate->cache_nr; i++) { + if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID && + starts_with(istate->cache[i]->name, name)) + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; + } + /* Need to remove the / from the path for the untracked cache */ + name[len - 1] = '\0'; + } else { + int pos = index_name_pos(istate, name, strlen(name)); - if (pos >= 0) { - struct cache_entry *ce = istate->cache[pos]; - ce->ce_flags &= ~CE_FSMONITOR_VALID; + if (pos >= 0) { + struct cache_entry *ce = istate->cache[pos]; + ce->ce_flags &= ~CE_FSMONITOR_VALID; + } } /* From patchwork Wed Feb 3 15:34:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12064631 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 816DAC433DB for ; Wed, 3 Feb 2021 15:40:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 40BDC64F5D for ; Wed, 3 Feb 2021 15:40:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234185AbhBCPkY (ORCPT ); Wed, 3 Feb 2021 10:40:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234271AbhBCPhg (ORCPT ); Wed, 3 Feb 2021 10:37:36 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37666C061351 for ; Wed, 3 Feb 2021 07:35:06 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id 7so24889637wrz.0 for ; Wed, 03 Feb 2021 07:35:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZpeyCC6Uj8x/EgLF5U4HUvxXJe+rqCgzupKMUyrx2y0=; b=V9i6qDXCCWN/HmXbbiIhry93nqdKsEyX+hw3EcB1VxU2hPcrerskSIB9G//5Q4Itco m8LiahhukSCWrTKLo4qByaLKsM5XqrHjBYXsFQeLQU4vRBGoYFUdKRfEN8yi9bAUB7sA Zxd1S9T58J949cBp+yxEQro/ZpGKkhOM7pD1lzjj/5B1t54204CrvhLYuJn0vdo0/O0n v+NolhFteMVBxUlv6xSK8AiA1V50z8+MKdG2SbyhBqt1DQE0A1NJH0xnK10e7Rc/Bft0 YUumn7nZv7VtSnHNP8vY/zIiv8H+o1a41gNG4q82eV0JR3iTKcbdTLJZGpK4FAskg9P1 hLzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZpeyCC6Uj8x/EgLF5U4HUvxXJe+rqCgzupKMUyrx2y0=; b=hfrpYjVPmVt6o/lntNSOJpY4d/f1AMr5f4/nt2tZxPZz/+++xluvOxTT7EQFCXCfMe ytW8oZMhxc+ZHoCyVRDdMcqB3+OMURRZumsGPo2kb7lt0kwGEBl3AigVa+va8kINBWxi 6sIkrZ7e8WN0qj9yxPfybJVF9FDhOrSTWh1aSItqpNpdMKbuYg7S2qyp49n8mj0AfSlz MP9+vLZhQfBE3Lnnj2Nu6Y5kDXaXIszbWN86qwLN9OTc8eIQZ2fmtqr/Cgy2d4FngMko AtS1HN+dBM3xWoKBWkSS1Vl1r7gzoynLs3pbsDFslEM7weF2dZeREt/w+IL9vMHrrlUL viYg== X-Gm-Message-State: AOAM530wgEOaMG7ImNWm6wOhNc1Vy9mHfvCYIpyCQGttqeo8iZ1Jz7nJ YQ09qODEgbCnuctvBUpdnQbnwxRxpsg= X-Google-Smtp-Source: ABdhPJy3oSs9xShpoaJLDsCmzGAqypQi9pxyJiA2yozdHO0duVGLxGPkgcqKY7LUMaws4kN+W56uCw== X-Received: by 2002:a5d:4d8d:: with SMTP id b13mr4028341wru.178.1612366504799; Wed, 03 Feb 2021 07:35:04 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s4sm3397057wme.38.2021.02.03.07.35.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 07:35:04 -0800 (PST) Message-Id: <5410d3ab61d1c9bb2a4d737bca8a81625e333da0.1612366490.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 03 Feb 2021 15:34:50 +0000 Subject: [PATCH v2 11/11] fsmonitor: refactor initialization of fsmonitor_last_update token Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Isolate and document initialization of `istate->fsmonitor_last_update`. This field should contain a fsmonitor-specific opaque token, but we need to initialize it before we can actually talk to a fsmonitor process, so we create a generic default value. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 64deeda597e..e12214b3007 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -343,16 +343,45 @@ void refresh_fsmonitor(struct index_state *istate) istate->fsmonitor_last_update = strbuf_detach(&last_update_token, NULL); } +/* + * The caller wants to turn on FSMonitor. And when the caller writes + * the index to disk, a FSMonitor extension should be included. This + * requires that `istate->fsmonitor_last_update` not be NULL. But we + * have not actually talked to a FSMonitor process yet, so we don't + * have an initial value for this field. + * + * For a protocol V1 FSMonitor process, this field is a formatted + * "nanoseconds since epoch" field. However, for a protocol V2 + * FSMonitor process, this field is an opaque token. + * + * Historically, `add_fsmonitor()` has initialized this field to the + * current time for protocol V1 processes. There are lots of race + * conditions here, but that code has shipped... + * + * The only true solution is to use a V2 FSMonitor and get a current + * or default token value (that it understands), but we cannot do that + * until we have actually talked to an instance of the FSMonitor process + * (but the protocol requires that we send a token first...). + * + * For simplicity, just initialize like we have a V1 process and require + * that V2 processes adapt. + */ +static void initialize_fsmonitor_last_update(struct index_state *istate) +{ + struct strbuf last_update = STRBUF_INIT; + + strbuf_addf(&last_update, "%"PRIu64"", getnanotime()); + istate->fsmonitor_last_update = strbuf_detach(&last_update, NULL); +} + void add_fsmonitor(struct index_state *istate) { unsigned int i; - struct strbuf last_update = STRBUF_INIT; if (!istate->fsmonitor_last_update) { trace_printf_key(&trace_fsmonitor, "add fsmonitor"); istate->cache_changed |= FSMONITOR_CHANGED; - strbuf_addf(&last_update, "%"PRIu64"", getnanotime()); - istate->fsmonitor_last_update = strbuf_detach(&last_update, NULL); + initialize_fsmonitor_last_update(istate); /* reset the fsmonitor state */ for (i = 0; i < istate->cache_nr; i++)