From patchwork Mon Feb 1 22:02:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE3F8C4332B for ; Mon, 1 Feb 2021 22:03:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9536364E2A for ; Mon, 1 Feb 2021 22:03:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229957AbhBAWDJ (ORCPT ); Mon, 1 Feb 2021 17:03:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41804 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229909AbhBAWDG (ORCPT ); Mon, 1 Feb 2021 17:03:06 -0500 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ED61BC06174A for ; Mon, 1 Feb 2021 14:02:25 -0800 (PST) Received: by mail-wr1-x42e.google.com with SMTP id l12so18303012wry.2 for ; Mon, 01 Feb 2021 14:02:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uzW/eYpOmlA1Eubawh6EjttB9/Fs0aZO+Ql5LOejaVs=; b=Oz3M34XGC0pIKC/NSv5Bd2+Rhc8zO6m9cAqCOmjYXuek0w2R2xG6ycY+FjogP0rcaq IPileSwpdAZrzd542ulwyV203cHXdtESDtTfz5fl1nlcqKHSZpHJrqXy5Ls9A1eOTcUY fQY4F8fH1r2iACItivL91WcZkjc4aHOPkRMghwopo5rmjDTpHL44Cb4tx4wJMdnDohsK x61lHUnvIGAd7HXbd/Hmwu7B30vifx0X2Ul4yN3QkEQBr/Ved3j+9fiyd4gTANJ162Xm GYWva2IXrEerdp+Z+FUhCQuma8TrGs0ecw1xRrT2EFeWjDEbM0jRLMult+9MiTote1p+ JeEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uzW/eYpOmlA1Eubawh6EjttB9/Fs0aZO+Ql5LOejaVs=; b=KCu7A+rKxXv8TadOfLXuhI3d3tuUDItwTwr9p6yFHuR4sbj83MrBdHRz3xk4X9ALjf jHU7DAyanb1za6Aw0ZCoDsjfUsmqxdF8/I67mETgA2001/9PID28IayLeFX5O+1Re+np ddjqpf2eHw+Y+1j2YufcwZfIsijnkrp8V+2jQmbokrfx3vouuvjy44AESn86gQ/53aml nJmAygGLUwKV8UdLCXIZZ0OOr771LROPgwnoYbkDOp/8/n06nETnBhDGf9ZThvnFh5PM +2sTRDyb7bszQcIosAQszjujHWs85cM4saNxV8hThS9X88jDS5yM4dbQrWhnQ7cVvllv 74SA== X-Gm-Message-State: AOAM533gByfXQo/dtCFLCnDCmPRpM2SKNAwzAdw8WuHKE8Nbbdm8EFtE bprnBkk8q/j4d6FdpvmQ96o+rhWG22g= X-Google-Smtp-Source: ABdhPJwCnweZjor7R0SQahBoLug+SSpabPbaFHRyt+IjdY2PMVJqMa7KXjUj75/0MrVGxUGq76SzpQ== X-Received: by 2002:a05:6000:143:: with SMTP id r3mr19073243wrx.357.1612216944521; Mon, 01 Feb 2021 14:02:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c9sm578339wmb.33.2021.02.01.14.02.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:24 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:10 +0000 Subject: [PATCH 01/11] p7519: use xargs -0 rather than -d in test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler The Mac version of xargs does not support the "-d" option. Convert the test setup to pipe the data set thru `lf_to_nul | xargs -0` instead. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 9b43342806b..7bb37e9a6c1 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -165,7 +165,7 @@ test_fsmonitor_suite() { ' test_perf_w_drop_caches "status (dirty) ($DESC)" ' - git ls-files | head -100000 | xargs -d "\n" touch -h && + git ls-files | head -100000 | lf_to_nul | xargs -0 touch -h && git status ' From patchwork Mon Feb 1 22:02:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3B46C4332D for ; Mon, 1 Feb 2021 22:03:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BA45E64ED2 for ; Mon, 1 Feb 2021 22:03:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229959AbhBAWDK (ORCPT ); Mon, 1 Feb 2021 17:03:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41810 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229913AbhBAWDH (ORCPT ); Mon, 1 Feb 2021 17:03:07 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CED67C061788 for ; Mon, 1 Feb 2021 14:02:26 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id c12so18284964wrc.7 for ; Mon, 01 Feb 2021 14:02:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Frnu9dWaKlmfKQihAMvHEZiZRPFqefcONv9mu4Qbx8c=; b=vdNf6+jpTo8e1KwGuu7U2ogWmUYlPTnl5ow5QQav0JbKTEe9zgNHBAe9GguKQqxA6O m9Weysa6L/h4Kd8kDnCwhg3QSzxU7QZmXUKu656IUBcewt7p0jELaF/0rOJ5qs/iigs8 zFE+N5hq0oaij7aT1ZHZWPjra9ImBqOJ2iXYk7OXzlpBr7Va/sB2dkkCkvKEtuARgZjk XNicte5BczciZFxAs2mnNWnJ+DHq3xsvTeTa84SuFkCbQEr00CL+dthXeeNMT1RtzaKU N07m1ZAi995FmpuE9oOdvJp7Ls8kF9fKR/iZ/80V93gmuRLhj5oZSYBeUaVV5qLJm/9F RADA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Frnu9dWaKlmfKQihAMvHEZiZRPFqefcONv9mu4Qbx8c=; b=HxcxmMqHBDuG5Y85sEpoLeFDaXCKdN2aZ4IIsLqPqA3Das+FgWqTTu+yKEsgWeXvgF ymEy14kLuat2fa2xA4lOiIRD9mzuqM7CI9FmESJ8ywqdYdPHvllqHbuo5ex8q9aj0iSO wuH3A5EvpfmwKV3AZNRGcSOfo4LwIudzbKHobk5ppwFocqX9vXoDdmB4PN05HglxacHn ejgW13d1KX3h8FHwteP1wsC9QiLOlhQqg9mTuO2bvqbcz5WgtdIUthhnjNGvAxb6j1kU dgGkBIuGlnPCUT8sN5fg9T7P1f3Xitep5Vq2gu2SsH19fNqSh16C3NBRFS573ahpk0Nm 2chw== X-Gm-Message-State: AOAM533F/fmS100JWH7TtDyoCGrXCfB315NlvuLR8yhEKNRAnpA2tT3C ngpo4Ap2Db52NY7vmCqWvset5zIf+V4= X-Google-Smtp-Source: ABdhPJyN4xNwdBUGzm/WrDl46Np07M2oNokOktxGBA/4XUB5TOPsiwQAkjZ44lJiRhp7wKxPFn//Hg== X-Received: by 2002:adf:f68e:: with SMTP id v14mr13544549wrp.273.1612216945424; Mon, 01 Feb 2021 14:02:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b4sm28547050wrn.12.2021.02.01.14.02.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:24 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:11 +0000 Subject: [PATCH 02/11] p7519: fix watchman watch-list test on Windows Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Only use the final portion of the test trash directory file name when verifying that Watchman was started. On Windows and under the SDK, $GIT_WORKTREE is a cygwin-style path with forward slashes and a "/c/" drive name. However `watchman watch-list` reports a proper Windows-style pathname with drive letters and backslashes. This causes the grep to fail. Since we don't really care about the full pathname (and we really don't want to bother with normalizaing them), just see if the test-name portion of the path is found. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 7bb37e9a6c1..e5a4b0582fb 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -101,7 +101,7 @@ test_expect_success "one time repo setup" ' # If Watchman exists, watch the work tree and attempt a query. if test_have_prereq WATCHMAN; then watchman watch "$GIT_WORK_TREE" && - watchman watch-list | grep -q -F "$GIT_WORK_TREE" + watchman watch-list | grep -q -F "p7519-fsmonitor" fi ' From patchwork Mon Feb 1 22:02:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77252C433DB for ; Mon, 1 Feb 2021 22:03:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 52D1664ED0 for ; Mon, 1 Feb 2021 22:03:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229609AbhBAWDl (ORCPT ); Mon, 1 Feb 2021 17:03:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41920 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229554AbhBAWDg (ORCPT ); Mon, 1 Feb 2021 17:03:36 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1FDEC06178A for ; Mon, 1 Feb 2021 14:02:27 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id 7so18308259wrz.0 for ; Mon, 01 Feb 2021 14:02:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=QaBEdmofcLQH4ouMaDZ+kun2sLA/BizWGuUSgFtOJtI=; b=qdg9kpzsfjhTaSpcjdZS/kFr1dIem+6CFwQ5C4si1PThR3t0bFs3OQFX7ti32D9S4q bBNWfjTwaGvkVlsV4EJ+yGOb0vwCMMDj0t3HN2IGuhQDczwukyRr3rxL/o7pc9ndfcvl 9emZxDvXgd4REhVNk2f/Sn+kbhEWpSqtkwpD25V5n3UPH7AMhaRJRnlVVUIJLa0RmsoD t3WC1zlqSC48ePzwgDA7lO4yfWlBFPIn2ExpEuZwG1Wv+d6UW9f2NyTEo4CcRGsZzZk8 icLCz0qHHmFlVSC+hUYi/N0feuE1XgjHUjXXiwWji/PRz4kv+KJeaKTtCDKtO7k3yicy YNQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=QaBEdmofcLQH4ouMaDZ+kun2sLA/BizWGuUSgFtOJtI=; b=AWibb48IrM5eqOO4M/1wmLnyMPRzwSJyFsGhvC9prS+BN0+rMGyDoeIcSJVg+0bnWk U4K4DC+lL7c7iW72umtmA4ZX+nnPMtYXPJJXO4AFITMXZ2ZMSYgfUjjfB/0uLNVnTnBr O9sw3iB1jpCBHt80AjfEOF1KRmgD5uAx71v+PMffNDxyiRISo9lUjCayHzx/aOcSC35T DxTMVA7NO1lezHNOBsmn6qXqr+fXpS/Fy/+r7beKROpW0aIIEHcexOmnB30yb0XLlo/X rAZ8YfoOatVcYpXfgZkGf07mSU2dkY2FZ2zuVVU+ykDnCvJODaYtaAW68a4+I8V62gOy I18g== X-Gm-Message-State: AOAM533TbTV5qUFCyiAdy8RdhvWIpMFwoL2pGfkJCqcNnLKTlIvvHMBP LUAW0NTJ8wRuSDNPvf0K8/z/lc9oG14= X-Google-Smtp-Source: ABdhPJw5KKotNFRQhTJ3EyIUWqFFbcwIgg9yFCy61Tr1S13yJ8hWKfW5nJ3noEIXusRnd8PhMwf0cw== X-Received: by 2002:a5d:4206:: with SMTP id n6mr20209855wrq.213.1612216946400; Mon, 01 Feb 2021 14:02:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d13sm28454438wrx.93.2021.02.01.14.02.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:25 -0800 (PST) Message-Id: <2af6858716f161a12acda7f07e74b0dfed03101b.1612216941.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:12 +0000 Subject: [PATCH 03/11] p7519: move watchman cleanup earlier in the test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Shutdown Watchman after the Watchman-based tests and before the block of "no fsmonitor" tests. This helps ensure that Watchman cannot affect the test results for the other. Signed-off-by: Jeff Hostetler --- t/perf/p7519-fsmonitor.sh | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index e5a4b0582fb..45bbba3c92f 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -198,6 +198,11 @@ test_fsmonitor_suite() { ' } +# +# Run a full set of perf tests using each Hook-based fsmonitor provider, +# such as Watchman. +# + if test -n "$GIT_PERF_7519_FSMONITOR"; then for INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do test_expect_success "setup for fsmonitor $INTEGRATION_PATH" 'setup_for_fsmonitor' @@ -208,14 +213,6 @@ else test_fsmonitor_suite fi -test_expect_success "setup without fsmonitor" ' - unset INTEGRATION_SCRIPT && - git config --unset core.fsmonitor && - git update-index --no-fsmonitor -' - -test_fsmonitor_suite - if test_have_prereq WATCHMAN then watchman watch-del "$GIT_WORK_TREE" >/dev/null 2>&1 && @@ -225,4 +222,16 @@ then watchman shutdown-server >/dev/null 2>&1 fi +# +# Run a full set of perf tests with the fsmonitor feature disabled. +# + +test_expect_success "setup without fsmonitor" ' + unset INTEGRATION_SCRIPT && + git config --unset core.fsmonitor && + git update-index --no-fsmonitor +' + +test_fsmonitor_suite + test_done From patchwork Mon Feb 1 22:02:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9DDAC433E0 for ; Mon, 1 Feb 2021 22:03:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BB63464E2C for ; Mon, 1 Feb 2021 22:03:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229810AbhBAWDp (ORCPT ); Mon, 1 Feb 2021 17:03:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41922 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229973AbhBAWDg (ORCPT ); Mon, 1 Feb 2021 17:03:36 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E20ABC06178B for ; Mon, 1 Feb 2021 14:02:28 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id 6so18285357wri.3 for ; Mon, 01 Feb 2021 14:02:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=qDQXfUw79WQ7Mw00tvloo6yvrULHW0pLb4QJBzNWryw=; b=hIeWy0f/9E/01GJEPEts5iR7e1AdERKpXI6d7C4Go6QUloubMrNVW5ZuHBC7P2WGB1 93BODEKLNPDcBMnK22G4lhUncy/SNeTvnjfBssMYWJ8VU7/pboag8OD/sTLNU8pvghtr T67OGXeMSmnabwkkGeWZ3A/IPdZOLLX1yG3wT0TlfG9EafLx4Q0bJHRcF0vD/xQDDZjW fR3lYJOt2uU9aDAQeDjMffQKLuqvA87e0Njq/cD06ctHpKIyRXY9JZ+84Fgz6e3piR3j EzB+7EoCBU9lOBpIB0Vj4Kdr117EBS6WGv/XiiLbOqopt9VP+rD7HVeYbxEE8Er9e/Ep vlMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=qDQXfUw79WQ7Mw00tvloo6yvrULHW0pLb4QJBzNWryw=; b=LttKEkjR9ajjjnxxq5pd0UhFjB8guoN7We0LDYmq7sT4z6y35zcJojbCqiRy5bta8f tTUI2TGzcncXGfhcjSqCsJX/rVuoybE3QWP3cpAXq3sXL3zgZ4ZfbABpNza5hYYE+8G9 IKnNKTd0jUkJvRbjgtPWvQATClbaV8ZKXPQ5PmTamnL5f7IdiL4m1Di6dljh91NhOkDy mva376Qidim5nnnUqagrJUeZzMXuElBz5wfJLT6lUzcxCfw2V1ShSj7jwEz3khJ90QuJ IcULN+ZybNQCCPJdNIJOL5eH8Q02eP+LhGU+NhXCH7Dyedkf8Rc1GR9WAELdtZzb82do 1dAA== X-Gm-Message-State: AOAM532FZvrebanymoFwHOZA0/lYAL9B2cyEdX5Q9jmB2Yjt2rIPSkB7 RwfFmwyRuyqnSE/a/c5cvGW8i70A4rI= X-Google-Smtp-Source: ABdhPJyahUumIiYlQ3MVrcNl9040vVX9f5LsDnXtbfbt+YloBwl+3PHazoBEJVBR+9xnryUU5CQVsA== X-Received: by 2002:adf:814f:: with SMTP id 73mr20270174wrm.368.1612216947323; Mon, 01 Feb 2021 14:02:27 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r11sm621335wmh.9.2021.02.01.14.02.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:26 -0800 (PST) Message-Id: <8de9985a706675ff41a3549ce00dd42a9dc92c88.1612216941.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:13 +0000 Subject: [PATCH 04/11] p7519: add trace logging during perf test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Add optional trace logging to allow us to better compare performance of various fsmonitor providers and compare results with non-fsmonitor runs. Currently, this includes Trace2 logging, but may be extended to include other trace targets, such as GIT_TRACE_FSMONITOR if desired. Using this logging helped me explain an odd behavior on MacOS where the kernel was dropping events and causing the hook to Watchman to timeout. Signed-off-by: Jeff Hostetler --- t/perf/.gitignore | 1 + t/perf/Makefile | 4 ++-- t/perf/p7519-fsmonitor.sh | 32 ++++++++++++++++++++++++++++++++ 3 files changed, 35 insertions(+), 2 deletions(-) diff --git a/t/perf/.gitignore b/t/perf/.gitignore index 982eb8e3a94..72f5d0d3148 100644 --- a/t/perf/.gitignore +++ b/t/perf/.gitignore @@ -1,3 +1,4 @@ /build/ /test-results/ +/test-trace/ /trash directory*/ diff --git a/t/perf/Makefile b/t/perf/Makefile index fcb0e8865e4..2465770a782 100644 --- a/t/perf/Makefile +++ b/t/perf/Makefile @@ -7,10 +7,10 @@ perf: pre-clean ./run pre-clean: - rm -rf test-results + rm -rf test-results test-trace clean: - rm -rf build "trash directory".* test-results + rm -rf build "trash directory".* test-results test-trace test-lint: $(MAKE) -C .. test-lint diff --git a/t/perf/p7519-fsmonitor.sh b/t/perf/p7519-fsmonitor.sh index 45bbba3c92f..e6724d3604b 100755 --- a/t/perf/p7519-fsmonitor.sh +++ b/t/perf/p7519-fsmonitor.sh @@ -32,6 +32,8 @@ test_description="Test core.fsmonitor" # # GIT_PERF_7519_DROP_CACHE: if set, the OS caches are dropped between tests # +# GIT_PERF_7519_TRACE: if set, enable trace logging during the test. +# Trace logs will be grouped by fsmonitor provider. test_perf_large_repo test_checkout_worktree @@ -70,6 +72,32 @@ then fi fi +trace_start() { + if test -n "$GIT_PERF_7519_TRACE" + then + name="$1" + TEST_TRACE_DIR="$TEST_OUTPUT_DIRECTORY/test-trace/p7519/" + echo "Writing trace logging to $TEST_TRACE_DIR" + + mkdir -p "$TEST_TRACE_DIR" + + # Start Trace2 logging and any other GIT_TRACE_* logs that you + # want for this named test case. + + GIT_TRACE2_PERF="$TEST_TRACE_DIR/$name.trace2perf" + export GIT_TRACE2_PERF + + >"$GIT_TRACE2_PERF" + fi +} + +trace_stop() { + if test -n "$GIT_PERF_7519_TRACE" + then + unset GIT_TRACE2_PERF + fi +} + test_expect_success "one time repo setup" ' # set untrackedCache depending on the environment if test -n "$GIT_PERF_7519_UNTRACKED_CACHE" @@ -203,6 +231,7 @@ test_fsmonitor_suite() { # such as Watchman. # +trace_start fsmonitor-watchman if test -n "$GIT_PERF_7519_FSMONITOR"; then for INTEGRATION_PATH in $GIT_PERF_7519_FSMONITOR; do test_expect_success "setup for fsmonitor $INTEGRATION_PATH" 'setup_for_fsmonitor' @@ -221,11 +250,13 @@ then # preventing the removal of the trash directory watchman shutdown-server >/dev/null 2>&1 fi +trace_stop # # Run a full set of perf tests with the fsmonitor feature disabled. # +trace_start fsmonitor-disabled test_expect_success "setup without fsmonitor" ' unset INTEGRATION_SCRIPT && git config --unset core.fsmonitor && @@ -233,5 +264,6 @@ test_expect_success "setup without fsmonitor" ' ' test_fsmonitor_suite +trace_stop test_done From patchwork Mon Feb 1 22:02:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6B2EC433DB for ; Mon, 1 Feb 2021 22:03:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A1EC764ED3 for ; Mon, 1 Feb 2021 22:03:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229822AbhBAWDv (ORCPT ); Mon, 1 Feb 2021 17:03:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229977AbhBAWDh (ORCPT ); Mon, 1 Feb 2021 17:03:37 -0500 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB8C4C06178C for ; Mon, 1 Feb 2021 14:02:29 -0800 (PST) Received: by mail-wm1-x32f.google.com with SMTP id o5so629820wmq.2 for ; Mon, 01 Feb 2021 14:02:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JewujSwXcnrJrGpQL58DN1s7LbKqF3XLqwqHhLqPPEA=; b=eykLUXBzWnv9rhOpD+B6HZ1XrdKr/qUFOtXZ+mKbh7zMsedk5A/F2i21/qgL7EX6XX aFiQEYeRV7kWXEFESdd3tIQnqWxUsGsXw92c1/CQHDywXUSA2po4XT9sD73wvNpZg0RY N+ZRY0dkd6vGBuFjQihia2Q6lfQ8eEKXLGfE6tJQWV3h0508e1+USMUZe3c4ALA9v5rF tMaBRpwJVfuFDhmFzC4VvyupQ5Jl7VQ5Cfv8HDk21HD27hACQg0HJlrwDrZITntnM9KZ zFg7Y+hBsVtokSWVxB5wvvyCONxLEQ+oSk7zP2ye1An4lCm/4ccG4suI4gonbY55PjjW u9nw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JewujSwXcnrJrGpQL58DN1s7LbKqF3XLqwqHhLqPPEA=; b=Hf3flYqkFDXWKKRCGZr1kRNUR+1clG4d4tKcbKFo06gpjiPrWBxsw7ZZLMEbnMhiQ5 r3F+uMx4Yf/ILeJ/awc9eciwoQgGNS4f7J/HhNdWaYgmnzp6S7+g8H0Ak5qyTMAIXjpr UxID4FDu1zPKYMJHrpLrBPxjfLD8sGYT4NkQf0UCcgeG1z/Go4ddoglQUpXk8GEm63SW lNVKTh+B42QPg2V3Gb+ymk/q+Td/xAaXAmMGpR+nAzAbG+B38cFBPAaf6bJCNzK6igk/ 2u/4O0ZGM5VUMRWh+8WKEJpn+udmXxr4V/2zaGkZwQOmzE5q+nW9ajbybXPVbvNF2Yd7 vXhw== X-Gm-Message-State: AOAM533HiG1wXIeo9xz7uPJ7nKZRPcRvHMmuDOIWMrqFUQ4M8b3RLpvG qf6bCInPpeR+Tt4yUuZghHvcPTuDL9A= X-Google-Smtp-Source: ABdhPJzkPuRpZprcjKP2LpapQ65vIvSTlxsa1KjBA/YQ72JJZsDOfcNFIFDDf4Knbtd/nsnI8OiNfA== X-Received: by 2002:a7b:cb8f:: with SMTP id m15mr789038wmi.65.1612216948534; Mon, 01 Feb 2021 14:02:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u6sm29982898wro.75.2021.02.01.14.02.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:27 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:14 +0000 Subject: [PATCH 05/11] preload-index: log the number of lstat calls to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Report the total number of calls made to lstat() inside preload_index(). FSMonitor improves the performance of commands like `git status` by avoiding scanning the disk for changed files. This can be seen in `preload_index()`. Let's measure this. Signed-off-by: Jeff Hostetler --- preload-index.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/preload-index.c b/preload-index.c index ed6eaa47388..e5529a58636 100644 --- a/preload-index.c +++ b/preload-index.c @@ -31,6 +31,7 @@ struct thread_data { struct pathspec pathspec; struct progress_data *progress; int offset, nr; + int t2_nr_lstat; }; static void *preload_thread(void *_data) @@ -73,6 +74,7 @@ static void *preload_thread(void *_data) continue; if (threaded_has_symlink_leading_path(&cache, ce->name, ce_namelen(ce))) continue; + p->t2_nr_lstat++; if (lstat(ce->name, &st)) continue; if (ie_match_stat(index, ce, &st, CE_MATCH_RACY_IS_DIRTY|CE_MATCH_IGNORE_FSMONITOR)) @@ -98,6 +100,7 @@ void preload_index(struct index_state *index, int threads, i, work, offset; struct thread_data data[MAX_PARALLEL]; struct progress_data pd; + int t2_sum_lstat = 0; if (!HAVE_THREADS || !core_preload_index) return; @@ -107,6 +110,9 @@ void preload_index(struct index_state *index, threads = 2; if (threads < 2) return; + + trace2_region_enter("index", "preload", NULL); + trace_performance_enter(); if (threads > MAX_PARALLEL) threads = MAX_PARALLEL; @@ -141,10 +147,14 @@ void preload_index(struct index_state *index, struct thread_data *p = data+i; if (pthread_join(p->pthread, NULL)) die("unable to join threaded lstat"); + t2_sum_lstat += p->t2_nr_lstat; } stop_progress(&pd.progress); trace_performance_leave("preload index"); + + trace2_data_intmax("index", NULL, "preload/sum_lstat", t2_sum_lstat); + trace2_region_leave("index", "preload", NULL); } int repo_read_index_preload(struct repository *repo, From patchwork Mon Feb 1 22:02:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 845C1C433E6 for ; Mon, 1 Feb 2021 22:04:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 518E864EC3 for ; Mon, 1 Feb 2021 22:04:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230057AbhBAWE3 (ORCPT ); Mon, 1 Feb 2021 17:04:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229975AbhBAWDh (ORCPT ); Mon, 1 Feb 2021 17:03:37 -0500 Received: from mail-wm1-x332.google.com (mail-wm1-x332.google.com [IPv6:2a00:1450:4864:20::332]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E3B65C061793 for ; Mon, 1 Feb 2021 14:02:30 -0800 (PST) Received: by mail-wm1-x332.google.com with SMTP id o10so745451wmc.1 for ; Mon, 01 Feb 2021 14:02:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fkxMogoHr5h4CV44ECW6UTWZy4JBxE27oGdnxv+/Kac=; b=N1IujNPiS87yitCLyOV80brW7kkdbs56Pa1eOML1Cabr+JH8FjkJe1tEVDPAWuda6n P2BFXZq6f4kJGwixY1C0noOUEFebHkoNU/S5WTNXuR7QTXrZnb0gMVx/M/1+y/WcyIT+ NgvTTyKP7g1QBMtmQwjgz5/hH/gnteebM69b4uogseWBDA/2uBBZUHRGxm5VZP8M7doJ h+FuH5kxN2233wbqW6W/HO3g90Gu8PeNDEctnbtVeiGu5t+y/zQwqthKsQpuytn4E48Z X5hOTpdVYJbKIzXkJ2gZgGlnfV4fR8T0ZHXpWVoJecp4abLvjo7E8q0vJX0nH2o7eQQ+ DW7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fkxMogoHr5h4CV44ECW6UTWZy4JBxE27oGdnxv+/Kac=; b=klstmC/peRr2RJPZtbNAvUvn3/sTdDHTNrOacwKwMMrawBa1P/MhrhQYaXi5GHa6GI 8E4RsA9KuBJAapvAfU49UrTbX++D/P9rfTJH9V76Q4Cdfp+qMyofgRDX1m18EfC+WsV0 9mbnzKw5NYLjjBP+8eVfkFR1dh79x1nNSMSU7rUHGa9NY1I+ibVv0Eyf4ARFcrRc1F2G 9Ev6pDbzb62RGijRKRE238dPLQbncwBdvNELIuw0aZnM25xQxI/as+e8/lHTM85RsQ8W yuUndDCOzUj0WWa5wxTnz/r9tKjGE6iKrSe9grvjJiqaqYwJl+t+H05XN2pZa22Htv1P nQOg== X-Gm-Message-State: AOAM532JyJoQd2EAvQtyb62S2mbU6LIZkwWlm/cheJn4cfSeIxzGhE8w YluKRG7AMvYUZB1Yhnjc4a4w7KxaW+4= X-Google-Smtp-Source: ABdhPJyRKODBAGwDWm0VY5VT1TAOSzjqvxb9EI2Pulj+IXp1jbLjH8PYzH2EX9w5FGhFqDEx6ptRyA== X-Received: by 2002:a05:600c:4e92:: with SMTP id f18mr791155wmq.126.1612216949410; Mon, 01 Feb 2021 14:02:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m205sm30951wmf.40.2021.02.01.14.02.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:28 -0800 (PST) Message-Id: <65488f7a1bfa8313a7b7d59e208e54cb7f7c18ae.1612216941.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:15 +0000 Subject: [PATCH 06/11] read-cache: log the number of lstat calls to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Report the total number of calls made to lstat() inside of refresh_index(). FSMonitor improves the performance of commands like `git status` by avoiding scanning the disk for changed files. This can be seen in `refresh_index()`. Let's measure this. Signed-off-by: Jeff Hostetler --- read-cache.c | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/read-cache.c b/read-cache.c index ecf6f689940..893cc41e1d9 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1364,7 +1364,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti static struct cache_entry *refresh_cache_ent(struct index_state *istate, struct cache_entry *ce, unsigned int options, int *err, - int *changed_ret) + int *changed_ret, + int *t2_did_lstat) { struct stat st; struct cache_entry *updated; @@ -1406,6 +1407,8 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate, return NULL; } + if (t2_did_lstat) + *t2_did_lstat = 1; if (lstat(ce->name, &st) < 0) { if (ignore_missing && errno == ENOENT) return ce; @@ -1519,6 +1522,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, const char *added_fmt; const char *unmerged_fmt; struct progress *progress = NULL; + int t2_sum_lstat = 0; if (flags & REFRESH_PROGRESS && isatty(2)) progress = start_delayed_progress(_("Refresh index"), @@ -1536,11 +1540,13 @@ int refresh_index(struct index_state *istate, unsigned int flags, * we only have to do the special cases that are left. */ preload_index(istate, pathspec, 0); + trace2_region_enter("index", "refresh", NULL); for (i = 0; i < istate->cache_nr; i++) { struct cache_entry *ce, *new_entry; int cache_errno = 0; int changed = 0; int filtered = 0; + int t2_did_lstat = 0; ce = istate->cache[i]; if (ignore_submodules && S_ISGITLINK(ce->ce_mode)) @@ -1566,7 +1572,10 @@ int refresh_index(struct index_state *istate, unsigned int flags, if (filtered) continue; - new_entry = refresh_cache_ent(istate, ce, options, &cache_errno, &changed); + new_entry = refresh_cache_ent(istate, ce, options, + &cache_errno, &changed, + &t2_did_lstat); + t2_sum_lstat += t2_did_lstat; if (new_entry == ce) continue; if (progress) @@ -1602,6 +1611,8 @@ int refresh_index(struct index_state *istate, unsigned int flags, replace_index_entry(istate, i, new_entry); } + trace2_data_intmax("index", NULL, "refresh/sum_lstat", t2_sum_lstat); + trace2_region_leave("index", "refresh", NULL); if (progress) { display_progress(progress, istate->cache_nr); stop_progress(&progress); @@ -1614,7 +1625,7 @@ struct cache_entry *refresh_cache_entry(struct index_state *istate, struct cache_entry *ce, unsigned int options) { - return refresh_cache_ent(istate, ce, options, NULL, NULL); + return refresh_cache_ent(istate, ce, options, NULL, NULL, NULL); } From patchwork Mon Feb 1 22:02:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FED0C433E0 for ; Mon, 1 Feb 2021 22:04:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3EE7964E2C for ; Mon, 1 Feb 2021 22:04:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229515AbhBAWD5 (ORCPT ); Mon, 1 Feb 2021 17:03:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41928 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229981AbhBAWDh (ORCPT ); Mon, 1 Feb 2021 17:03:37 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E87F9C061794 for ; Mon, 1 Feb 2021 14:02:31 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id u14so613251wml.4 for ; Mon, 01 Feb 2021 14:02:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=EwY+RPdKlOlJBlCoDOs+KUc/NkIJWwGIjdTt04+8VTw=; b=f3riSPceJ2i5bAT5uZ8XuYicl97OZSFVB31s5zNlvauZ6L3DuZcsJxi6wpK/lXjybu dRF6ksUx+wj2WehJAvjX9qVldw/5mzsRXqAjsu6yAO9nKQZKR5eiMD6LRoOfqRHlbcms izwpKXbHcArAXfRECKvyoGXxFKWdz8yhCRgwYwAyZr9AocPIk8ff2BKsNVqhg3mDRN2F GZFzh+zmRIYLgFgoU6G2b/nStEZSTH8wXI7oLGJPS+SHIaIxY+1kacOZewn2lqjKJX/C iM9JzCBD4i5fOgigdIdpgtE3yiQJnz05E0ldtyLOWdv2lTjmWGcshVrljOorMJd3i8Wf glqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=EwY+RPdKlOlJBlCoDOs+KUc/NkIJWwGIjdTt04+8VTw=; b=a99nU6j+mKuHk4y4PtwBXLhF5jScnqd9g71FWE4OdvPrp+IuAkJX+VM15EPjCd1jeo seYJTJSnMpZbSSiy/iX2oOW6NYXWxudN0sIigNDKi/YSLHYKMYM49kWat02Vh5i9/eiM xCdIwZi03SN3WiWRtb9Ng8mK9TVlnO7p5Z3mLpHsqJUjceZ+VHuYJABDNHa/FRr3hm67 HeiGZzb2RXqdDoBnDl6CCsh3F/I2WsCd8st35zravJHIxhYCDQRz24cFbW10mG4pOUD6 tSrggmh8ZJIi5PqBAUd8jJOVRKcvoIKZwkmct7Cs80pg3wGnOnOKF3bjTrSPpYAXdiYe qReg== X-Gm-Message-State: AOAM531tMmsbSkKhAiPJhgtxmrWjZavyvErCsM0F7+4cydaE9BpF2Yqc QVhaeaKHnLEqJCH/wx3C5H3HzFQXCMw= X-Google-Smtp-Source: ABdhPJwf3ZJKZ2UKzNZzvUFfk4r8kKAmHOUnhE7fGXMOExguoR2im18nWqyGOlBcIXoXh0HbSVq0sA== X-Received: by 2002:a1c:5454:: with SMTP id p20mr786731wmi.128.1612216950364; Mon, 01 Feb 2021 14:02:30 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y18sm27738530wrt.19.2021.02.01.14.02.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:29 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:16 +0000 Subject: [PATCH 07/11] read-cache: log the number of scanned files to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Report the number of files in the working directory that were read and their hashes verified in `refresh_index()`. FSMonitor improves the performance of commands like `git status` by avoiding scanning the disk for changed files. Let's measure this. Signed-off-by: Jeff Hostetler --- read-cache.c | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/read-cache.c b/read-cache.c index 893cc41e1d9..c9dd7f4015e 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1365,7 +1365,8 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate, struct cache_entry *ce, unsigned int options, int *err, int *changed_ret, - int *t2_did_lstat) + int *t2_did_lstat, + int *t2_did_scan) { struct stat st; struct cache_entry *updated; @@ -1445,6 +1446,8 @@ static struct cache_entry *refresh_cache_ent(struct index_state *istate, } } + if (t2_did_scan) + *t2_did_scan = 1; if (ie_modified(istate, ce, &st, options)) { if (err) *err = EINVAL; @@ -1523,6 +1526,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, const char *unmerged_fmt; struct progress *progress = NULL; int t2_sum_lstat = 0; + int t2_sum_scan = 0; if (flags & REFRESH_PROGRESS && isatty(2)) progress = start_delayed_progress(_("Refresh index"), @@ -1547,6 +1551,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, int changed = 0; int filtered = 0; int t2_did_lstat = 0; + int t2_did_scan = 0; ce = istate->cache[i]; if (ignore_submodules && S_ISGITLINK(ce->ce_mode)) @@ -1574,8 +1579,9 @@ int refresh_index(struct index_state *istate, unsigned int flags, new_entry = refresh_cache_ent(istate, ce, options, &cache_errno, &changed, - &t2_did_lstat); + &t2_did_lstat, &t2_did_scan); t2_sum_lstat += t2_did_lstat; + t2_sum_scan += t2_did_scan; if (new_entry == ce) continue; if (progress) @@ -1612,6 +1618,7 @@ int refresh_index(struct index_state *istate, unsigned int flags, replace_index_entry(istate, i, new_entry); } trace2_data_intmax("index", NULL, "refresh/sum_lstat", t2_sum_lstat); + trace2_data_intmax("index", NULL, "refresh/sum_scan", t2_sum_scan); trace2_region_leave("index", "refresh", NULL); if (progress) { display_progress(progress, istate->cache_nr); @@ -1625,7 +1632,7 @@ struct cache_entry *refresh_cache_entry(struct index_state *istate, struct cache_entry *ce, unsigned int options) { - return refresh_cache_ent(istate, ce, options, NULL, NULL, NULL); + return refresh_cache_ent(istate, ce, options, NULL, NULL, NULL, NULL); } From patchwork Mon Feb 1 22:02:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F34D4C433E0 for ; Mon, 1 Feb 2021 22:04:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A85B964E50 for ; Mon, 1 Feb 2021 22:04:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229999AbhBAWEC (ORCPT ); Mon, 1 Feb 2021 17:04:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41930 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229983AbhBAWDh (ORCPT ); Mon, 1 Feb 2021 17:03:37 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E519EC061797 for ; Mon, 1 Feb 2021 14:02:32 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id j18so620295wmi.3 for ; Mon, 01 Feb 2021 14:02:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=+KteHt2BWR//iqZgex9DKa/luwnSNm9uhzmwsnCG0fE=; b=s5dY7oOnZbZ9ZZ177l8XIMNHsMduXKWxJSkbUDZVjTTvaANEH3NkqUcuTf78IVxVWt wJq7LzpGkCbK3lYYPnQQsqE9UIZ8IL1RxAs+Bj6NPMpgXD7VWg3v6y9gJa4v6u5d3tvK meCRYq/GayeH7LGZF31hpVELLQPmGhRA2VDIBoxX6io4EvnJJhesnVDlgGtoIsEaSaOs KBUtDdNZrYmtZjWbZOEh8zJMSBiuchwTHfH3VsBs4bTnCPiD/3rmYkk+AOPk7eRmbDmL 5g+Tk8QBKazu/T9vHp9/Uf15ntDOon6TR+gGrpATgcrFNNtD5u03c1FN8FQmlRxurgnH klng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=+KteHt2BWR//iqZgex9DKa/luwnSNm9uhzmwsnCG0fE=; b=N6Dct8wS7vuXpl7IXE9oQJJyhuXlcpDzfgZpnBvw/6WW7s78V912EIEZ0q7BuRapTM 0cl1ZDsIBhiONT90GvtCRHXuaHCa7+0NXBfaSNxbc0IBPAyXhiVEAF8BfFs5dQNO9YYh a016RJyEm/06yIccmnljQm4V5Nks4CKzvG318d8eyX61TNWQb8QOgmIDwPo/SulG2fKW +y1lu3SOwGQ9y6V/FQ+raD5U0FIjstWL58NDkwea/nMAURI4P/xYG76jEo6P5gkRF+2d a1DU9WCR1q8R09PMKyPeBCFxy5mQ2DHjOFPtHya0zNUGrboSClDHd93LhNmETkSJ2Nkz kZIA== X-Gm-Message-State: AOAM530Xf1xPSsdxVN+DGqrKf3NXh0k6mjIZARyEmXUlZC3cD6/SM+DR 3j6yTebDBoSOkHk484v/LVaScirVczk= X-Google-Smtp-Source: ABdhPJzUJ3cymq5llfAQxRPWrGNQ7GsEHSWaefIlmbHgAyawTWLo41msRWcan864n6tdANGRQVFimQ== X-Received: by 2002:a1c:356:: with SMTP id 83mr810738wmd.31.1612216951494; Mon, 01 Feb 2021 14:02:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r13sm646481wmh.9.2021.02.01.14.02.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:30 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:17 +0000 Subject: [PATCH 08/11] fsmonitor: log invocation of FSMonitor hook to trace2 Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Let's measure the time taken to request and receive FSMonitor data via the hook API and the size of the response. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 29 ++++++++++++++++++++++++++++- fsmonitor.h | 5 +++++ 2 files changed, 33 insertions(+), 1 deletion(-) diff --git a/fsmonitor.c b/fsmonitor.c index ca031c3abb8..7a2be24cd43 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -142,6 +142,7 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate) static int query_fsmonitor(int version, const char *last_update, struct strbuf *query_result) { struct child_process cp = CHILD_PROCESS_INIT; + int result; if (!core_fsmonitor) return -1; @@ -152,7 +153,33 @@ static int query_fsmonitor(int version, const char *last_update, struct strbuf * cp.use_shell = 1; cp.dir = get_git_work_tree(); - return capture_command(&cp, query_result, 1024); + trace2_region_enter("fsm_hook", "query", NULL); + + result = capture_command(&cp, query_result, 1024); + + if (result) + trace2_data_intmax("fsm_hook", NULL, "query/failed", result); + else { + trace2_data_intmax("fsm_hook", NULL, "query/response-length", + query_result->len); + + if (fsmonitor_is_trivial_response(query_result)) + trace2_data_intmax("fsm_hook", NULL, + "query/trivial-response", 1); + } + + trace2_region_leave("fsm_hook", "query", NULL); + + return result; +} + +int fsmonitor_is_trivial_response(const struct strbuf *query_result) +{ + static char trivial_response[3] = { '\0', '/', '\0' }; + int is_trivial = !memcmp(trivial_response, + &query_result->buf[query_result->len - 3], 3); + + return is_trivial; } static void fsmonitor_refresh_callback(struct index_state *istate, const char *name) diff --git a/fsmonitor.h b/fsmonitor.h index 739318ab6d1..7f1794b90b0 100644 --- a/fsmonitor.h +++ b/fsmonitor.h @@ -44,6 +44,11 @@ void tweak_fsmonitor(struct index_state *istate); */ void refresh_fsmonitor(struct index_state *istate); +/* + * Does the received result contain the "trivial" response? + */ +int fsmonitor_is_trivial_response(const struct strbuf *query_result); + /* * Set the given cache entries CE_FSMONITOR_VALID bit. This should be * called any time the cache entry has been updated to reflect the From patchwork Mon Feb 1 22:02:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D933C433DB for ; Mon, 1 Feb 2021 22:03:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 702F164EC3 for ; Mon, 1 Feb 2021 22:03:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229969AbhBAWDX (ORCPT ); Mon, 1 Feb 2021 17:03:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41772 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229913AbhBAWDN (ORCPT ); Mon, 1 Feb 2021 17:03:13 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D20A3C0617A7 for ; Mon, 1 Feb 2021 14:02:33 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id l12so18303302wry.2 for ; Mon, 01 Feb 2021 14:02:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=tUd9omKSffFRsXV7K6/6DEhbnGyQp25YksuiGbwNKKg=; b=nCRfBZqF3NGu82P8kUXz3MIX9DNfsAQwEMkrVAU9/FCcTSAuqctBhFd2OE2c6bnqeb pxXPBcC+CWLbrJPKNDCd+6aDLConU+JuN1R+iaZU1m7zpfZuVSgy9KNi9mEeJ1mOdULQ dNO9n/K7QoxzG+ZjwNyUKxRv+wTEZ4pJpKtGRzbcNw9UU7tzFBBpkReUpNr8Jo7Dp7xV tk2aFQ3a4HyFumf9lf38PID2CQjei+us86RKs0XYEEL2Hu4CsVlX0onkeqGqXkReKYHC diKNF6TN9CswZHu35tJm9h06Pt37+GABrPXCeU50TAYRNzmjYOTLwVH6qyzrIiO96WkI LKiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=tUd9omKSffFRsXV7K6/6DEhbnGyQp25YksuiGbwNKKg=; b=Ca7dQu45KES+NBRI4fiB6aTnJ6ONg0PQuZVce38FqM+STBiJb2HQmxoy0B8aGrekDc UmbDALcwKzZnDXWUk/dNGdX4DYSyexHZsI+s+RuwBZivDME7aV40KjxfxS9FOSh3zkq8 Hj8fbd5vHj1z4ZSyTqpD+OUf4vsZQ3F+OOncjpA5SvL7AdsHatCTJDuNx1vk9y5beyYJ idhPy7J59OY0MYntZr3oEWo9d2mmXSIaXlGMHQSg6A5cJZRpJyHLxUnlZySRyYgcQ6eJ o1El7Mcr7P78eCguE3iSnJ2TZe4oLA4Ms8yb4edD12ImsAinQiEzYv6v5h2CvXLZx1Ic jeyw== X-Gm-Message-State: AOAM533GnXK/2byG230Ky8gRukpsQUaaN8P46ff2+R2nV5/emT+aBZGX 2yt2lON5gU6CQ+qJIPn+RYTibhbzzgE= X-Google-Smtp-Source: ABdhPJxis1lSKH347647RJoxcGxBIeVZeT99gLyprveNkDQ4tdhh4JVQADBBkN6PMeeLAt0A/rV3Vg== X-Received: by 2002:adf:dd10:: with SMTP id a16mr20358364wrm.207.1612216952495; Mon, 01 Feb 2021 14:02:32 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t18sm24688046wrr.56.2021.02.01.14.02.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:31 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:18 +0000 Subject: [PATCH 09/11] fsmonitor: log FSMN token when reading and writing the index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Signed-off-by: Jeff Hostetler --- fsmonitor.c | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 7a2be24cd43..3105dc370ab 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -87,7 +87,11 @@ int read_fsmonitor_extension(struct index_state *istate, const void *data, BUG("fsmonitor_dirty has more entries than the index (%"PRIuMAX" > %u)", (uintmax_t)istate->fsmonitor_dirty->bit_size, istate->cache_nr); - trace_printf_key(&trace_fsmonitor, "read fsmonitor extension successful"); + trace2_data_string("index", NULL, "extension/fsmn/read/token", + istate->fsmonitor_last_update); + trace_printf_key(&trace_fsmonitor, + "read fsmonitor extension successful '%s'", + istate->fsmonitor_last_update); return 0; } @@ -133,7 +137,11 @@ void write_fsmonitor_extension(struct strbuf *sb, struct index_state *istate) put_be32(&ewah_size, sb->len - ewah_start); memcpy(sb->buf + fixup, &ewah_size, sizeof(uint32_t)); - trace_printf_key(&trace_fsmonitor, "write fsmonitor extension successful"); + trace2_data_string("index", NULL, "extension/fsmn/write/token", + istate->fsmonitor_last_update); + trace_printf_key(&trace_fsmonitor, + "write fsmonitor extension successful '%s'", + istate->fsmonitor_last_update); } /* From patchwork Mon Feb 1 22:02:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kevin Willford X-Patchwork-Id: 12060147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C4AFC433E0 for ; Mon, 1 Feb 2021 22:04:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5F9FE64E2C for ; Mon, 1 Feb 2021 22:04:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229986AbhBAWEA (ORCPT ); Mon, 1 Feb 2021 17:04:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41940 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229984AbhBAWDi (ORCPT ); Mon, 1 Feb 2021 17:03:38 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA70DC0617A9 for ; Mon, 1 Feb 2021 14:02:34 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id g10so18295081wrx.1 for ; Mon, 01 Feb 2021 14:02:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MgDPgj7GedDHVjjkk/EtAODaaSn6gWGGMNTT5xfwao8=; b=rQ9ox0YpyQsKf88pXDu/z8EqvFTTKGIiCsKzNsHfjSj611kyoP4If86abkOTMb+8r9 cuTjWxcopjvS4tuQgDeJlU+UEmtGiDdj9nnSvYjwI3Cz8UzlXTl8PVOSfvlrJ6v540uN vkIFsUNW+3MWRmukwFV69h/flDca0EpZh4WOQIltYS7YysC3kXzRVy3ZklPAI3lC1pi9 kINPDH0J+YR02CtlFZxR1GXm3DPi9y3tHxJmU+nxEHFXNi0OSBfp8aCqwcz55wzTV2W4 S7QPq1i73UJhdNAFVTeBvrHQeVFI/ObceM7pMwL51UZhBDnsn02hZ6Hxh1f94BzzezFV tDqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MgDPgj7GedDHVjjkk/EtAODaaSn6gWGGMNTT5xfwao8=; b=jOrg64ljiHqzB1MUJsjkj6Dp0BzYilWFG8RYIotz4BIn9yaCzPFaJnhXzt0wVFVsJC AdM7wVRPqp0SyDqAR78jrdILyzBv23OvwrZGmMNUS5TH0veUn9B7BNSCTSxzvJfFx6mk SZ04p1Gsak2rfl/ZnvESM2oBBEswr1QwOWaYSCtQ85HljV8erfhnXiP4ogUhJXKrSE32 JdGoEwwMF5Yt79DKKbiMofb2hRsyL0CgRWNStigrtbC05Jz+wjf17nvFPMUBNBgWmKgX P/DobnjSU8gRC0gurqOWKbTpDWq6iBIvB+vCAAw1YZk/hkKJb7x8T68ivs0OVO7EBGce wtnA== X-Gm-Message-State: AOAM53041lDPElrbTWMZzklFGuyd8O9f0vfhewdhah6sBzF4cw5ISvzA G6B9GJZkdQ3PhGodof8hxnto7ppc+yM= X-Google-Smtp-Source: ABdhPJyLLwckQadFXbOnITja8HInfo2W1Z2k4kHizSLr0ScCaMVR+z4w7zyEQOe9MCvqmYsdXr1n2w== X-Received: by 2002:a05:6000:186f:: with SMTP id d15mr20465937wri.237.1612216953346; Mon, 01 Feb 2021 14:02:33 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w14sm28255307wro.86.2021.02.01.14.02.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:32 -0800 (PST) Message-Id: <384d2eff8637b7dd4cfbb116dd4b3ce30de5e381.1612216941.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:19 +0000 Subject: [PATCH 10/11] fsmonitor: allow all entries for a folder to be invalidated Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Kevin Willford Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Kevin Willford From: Kevin Willford Allow fsmonitor to report directory changes by reporting paths with a trailing slash. Signed-off-by: Jeff Hostetler Signed-off-by: Kevin Willford Signed-off-by: Johannes Schindelin --- fsmonitor.c | 31 ++++++++++++++++++++++++++----- 1 file changed, 26 insertions(+), 5 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 3105dc370ab..64deeda597e 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -190,13 +190,34 @@ int fsmonitor_is_trivial_response(const struct strbuf *query_result) return is_trivial; } -static void fsmonitor_refresh_callback(struct index_state *istate, const char *name) +static void fsmonitor_refresh_callback(struct index_state *istate, char *name) { - int pos = index_name_pos(istate, name, strlen(name)); + int i, len = strlen(name); + if (name[len - 1] == '/') { + + /* + * TODO We should binary search to find the first path with + * TODO this directory prefix. Then linearly update entries + * TODO while the prefix matches. Taking care to search without + * TODO the trailing slash -- because '/' sorts after a few + * TODO interesting special chars, like '.' and ' '. + */ + + /* Mark all entries for the folder invalid */ + for (i = 0; i < istate->cache_nr; i++) { + if (istate->cache[i]->ce_flags & CE_FSMONITOR_VALID && + starts_with(istate->cache[i]->name, name)) + istate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID; + } + /* Need to remove the / from the path for the untracked cache */ + name[len - 1] = '\0'; + } else { + int pos = index_name_pos(istate, name, strlen(name)); - if (pos >= 0) { - struct cache_entry *ce = istate->cache[pos]; - ce->ce_flags &= ~CE_FSMONITOR_VALID; + if (pos >= 0) { + struct cache_entry *ce = istate->cache[pos]; + ce->ce_flags &= ~CE_FSMONITOR_VALID; + } } /* From patchwork Mon Feb 1 22:02:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Hostetler X-Patchwork-Id: 12060137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B37D8C433E0 for ; Mon, 1 Feb 2021 22:03:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8D37964ED0 for ; Mon, 1 Feb 2021 22:03:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229972AbhBAWD0 (ORCPT ); Mon, 1 Feb 2021 17:03:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41798 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229554AbhBAWDV (ORCPT ); Mon, 1 Feb 2021 17:03:21 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E596C0617AA for ; Mon, 1 Feb 2021 14:02:35 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id e15so597596wme.0 for ; Mon, 01 Feb 2021 14:02:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZpeyCC6Uj8x/EgLF5U4HUvxXJe+rqCgzupKMUyrx2y0=; b=cD8St/00TiYx10ap75dZzgrq1m3NVSd4Lijs+atmictHb6MmloeXKTjvTyUh/dsjNj n7RZgFIIGTVYV6nxVDxy8GROq8tMh3/xxRTtLPqA0QUBsPLUr4IQ979+cbJMTwIHSCMZ 19rtXHYQtxOervfHyKIfa4/3auXskrwDXW50VgiE+O1jpJfGdXeqpY2sC3s6f3tDwtQ2 KNcuDIZ4BK9yAxY0Qm6VAs4sQz1aY9686X1v8IjW7LBCkh7+r6YAoRXKFJKUJvf15mYB GuQTzt9xKLBXRKWbX2auqp8cEOFakrfnl0m2xjgViJqNZbFZCevgDxYimbejZ21scPBb wVHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZpeyCC6Uj8x/EgLF5U4HUvxXJe+rqCgzupKMUyrx2y0=; b=fYHOupN+Bf97QCt4ZUxvCgnVAMPRuir9mWLoPMgP5BOE+W/QSdRdy0A7kuQmywJs5S +R1Zig794xcwSYoWZxvzOzD3qQ5nFvsj3WlpMxmA1jqWnricUM1dEqUtYmDyXBjWBp6U Zqis/CQ5Izb+mlHlFEwnPQNgN3/eCw6yTWca9l4FSl9EcMCEwcDC15ZeVPjx8rEk8erc iPoEyCeohEW32Uk15Waqtu4HNDcOGw1ZRFmrkpHHzx1VplQvTz1WTqTN1CvipRvZFZ5c s/xh93Igei84xHTG0drf0L2OgSDdnCzjOC391dQyM1gf1yJXdXvUKH0CtBT4J+YlF+Zr 0+fw== X-Gm-Message-State: AOAM532q+YOJerr3yuKEVm6AxDE9N7Q0Lx7T6/MLfROaHqif/p94ntQc p9t1vJ5daqeB8xR4wbJzFA65idENX0U= X-Google-Smtp-Source: ABdhPJyTp8vQyYlPzEOJOWf04DhcreBDPRQoc0Cnwqr3jGojUnmlGQLa4UCDATV9XeL3TtiMsOiJqA== X-Received: by 2002:a1c:7f94:: with SMTP id a142mr757548wmd.145.1612216954221; Mon, 01 Feb 2021 14:02:34 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g194sm529378wme.39.2021.02.01.14.02.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 14:02:33 -0800 (PST) Message-Id: <4686196bbc6a583eda16d2f2274e29df98284a22.1612216941.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 01 Feb 2021 22:02:20 +0000 Subject: [PATCH 11/11] fsmonitor: refactor initialization of fsmonitor_last_update token Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Jeff Hostetler , Jeff Hostetler Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff Hostetler From: Jeff Hostetler Isolate and document initialization of `istate->fsmonitor_last_update`. This field should contain a fsmonitor-specific opaque token, but we need to initialize it before we can actually talk to a fsmonitor process, so we create a generic default value. Signed-off-by: Jeff Hostetler --- fsmonitor.c | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/fsmonitor.c b/fsmonitor.c index 64deeda597e..e12214b3007 100644 --- a/fsmonitor.c +++ b/fsmonitor.c @@ -343,16 +343,45 @@ void refresh_fsmonitor(struct index_state *istate) istate->fsmonitor_last_update = strbuf_detach(&last_update_token, NULL); } +/* + * The caller wants to turn on FSMonitor. And when the caller writes + * the index to disk, a FSMonitor extension should be included. This + * requires that `istate->fsmonitor_last_update` not be NULL. But we + * have not actually talked to a FSMonitor process yet, so we don't + * have an initial value for this field. + * + * For a protocol V1 FSMonitor process, this field is a formatted + * "nanoseconds since epoch" field. However, for a protocol V2 + * FSMonitor process, this field is an opaque token. + * + * Historically, `add_fsmonitor()` has initialized this field to the + * current time for protocol V1 processes. There are lots of race + * conditions here, but that code has shipped... + * + * The only true solution is to use a V2 FSMonitor and get a current + * or default token value (that it understands), but we cannot do that + * until we have actually talked to an instance of the FSMonitor process + * (but the protocol requires that we send a token first...). + * + * For simplicity, just initialize like we have a V1 process and require + * that V2 processes adapt. + */ +static void initialize_fsmonitor_last_update(struct index_state *istate) +{ + struct strbuf last_update = STRBUF_INIT; + + strbuf_addf(&last_update, "%"PRIu64"", getnanotime()); + istate->fsmonitor_last_update = strbuf_detach(&last_update, NULL); +} + void add_fsmonitor(struct index_state *istate) { unsigned int i; - struct strbuf last_update = STRBUF_INIT; if (!istate->fsmonitor_last_update) { trace_printf_key(&trace_fsmonitor, "add fsmonitor"); istate->cache_changed |= FSMONITOR_CHANGED; - strbuf_addf(&last_update, "%"PRIu64"", getnanotime()); - istate->fsmonitor_last_update = strbuf_detach(&last_update, NULL); + initialize_fsmonitor_last_update(istate); /* reset the fsmonitor state */ for (i = 0; i < istate->cache_nr; i++)