From patchwork Fri Feb 25 17:10:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tao Klerks X-Patchwork-Id: 12760586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2BCBC433EF for ; Fri, 25 Feb 2022 17:10:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243486AbiBYRKv (ORCPT ); Fri, 25 Feb 2022 12:10:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48326 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237751AbiBYRKv (ORCPT ); Fri, 25 Feb 2022 12:10:51 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB99A186B95 for ; Fri, 25 Feb 2022 09:10:18 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id s13so5481704wrb.6 for ; Fri, 25 Feb 2022 09:10:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Mv2jZjUK6lbjXJZCOXpjHPeNtKrW1epocKS0+GVZpxE=; b=Y9CnQ/ym4teKpzJfhbmM0SLWgOPev5x1Xz1jnn/WjcigqtSDCys3/x/kuf6RFupx0D 665ivfE0oWfMFvB5zdWmoMyLV06uqXrdlHkqJI1X5Hz0xae6wmg0PvB4r0TH9XvDsYc/ Y2yoJyj/8NPfPEEFoWg/RM6P72aUd3zt4fncJE1PNfney5eJTjFUJ33k5fbR8O6D5OKF UYYbL5DI8PvvL+WxOi6P141f5qhiMm/UlzPIY0D0D2GWGZgeE8z0eAFRFeylMfDqmXOF V6Semez0K8LWQEv0Otm3YNDAHBFMQWPEwfwLpNAzzU0s8jXg1Ywt7ANCLwOSe8t9m1/F P5iQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Mv2jZjUK6lbjXJZCOXpjHPeNtKrW1epocKS0+GVZpxE=; b=B21TB5LIjhot/6tBplf6gZeh3naFVOqaaJL4Q29QfjZT72bs0bdP/0zcduVgbpEo5t mgPhz6aeJ6/npORH4wX/iQOvSeGd35P2Kml09lPR8HUW/WGptLgUDEqTJVQAwU9x4Ef/ 2NGMZ7GdWawjpaPxP8IxK6bqN5A4UKrLyESXNfcKK3gA0chPOROfiGx3WO+VgqmZHWee ZJTZvjx+sz06vp3Mu9yYmaVoGCO4AJuBm7XQNqB5WX3c1sdsQc+JKrTDlUUxNyOCcvN2 mDdBIa9e3zieSu+PlIYYaZvJd/PWuNRH61C/Cwe24TZA1tASQ+0lVSror7TSCYK19MGV Xv1A== X-Gm-Message-State: AOAM533KN+3wx1mVt5c/oXx0zYOqA3tHDZAUEXT1hqVv76V9Y2Tl1Pa7 LZzJr2CXCKam0LsA/gAvTVCXtYIVhGE= X-Google-Smtp-Source: ABdhPJxDq1KKpR2q0u9nqZEUe6jNnI2cEuppb+9PXv7kui1tBRr3IO/z5z4b6PPrjxb/PcXi6Gbxqg== X-Received: by 2002:a5d:6e0f:0:b0:1ed:a72f:8842 with SMTP id h15-20020a5d6e0f000000b001eda72f8842mr6995138wrz.568.1645809017226; Fri, 25 Feb 2022 09:10:17 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q7-20020adfc507000000b001ed9f31afdcsm2749255wrf.3.2022.02.25.09.10.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Feb 2022 09:10:16 -0800 (PST) Message-Id: <9421b71540d1f1764db6931d0781576d8a710866.1645809015.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Feb 2022 17:10:12 +0000 Subject: [PATCH v2 1/3] t7519: avoid file to index mtime race for untracked cache Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Tao Klerks , Tao Klerks Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tao Klerks From: Tao Klerks In t7519 there is a test that writes files to disk, and immediately writes the untracked index. Because of mtime-comparison logic that uses a 1-second resolution, this means the cached entries are not trusted/used under some circumstances (see read-cache.c#is_racy_stat()). Untracked cache tests in t7063 use a 1-second delay to avoid this issue, but we don't want to introduce arbitrary slowdowns, so instead use test-tool chmtime to backdate the files slightly. This change doesn't actually affect the outcome of the test, but does enhance its validity, and becomes relevant after later changes. Signed-off-by: Tao Klerks --- t/t7519-status-fsmonitor.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index a6308acf006..3f984136ea9 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -324,13 +324,19 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR' cd dot-git && mkdir -p .git/hooks && : >tracked && + test-tool chmtime =-60 tracked && : >modified && + test-tool chmtime =-60 modified && mkdir dir1 && : >dir1/tracked && + test-tool chmtime =-60 dir1/tracked && : >dir1/modified && + test-tool chmtime =-60 dir1/modified && mkdir dir2 && : >dir2/tracked && + test-tool chmtime =-60 dir2/tracked && : >dir2/modified && + test-tool chmtime =-60 dir2/modified && write_integration_script && git config core.fsmonitor .git/hooks/fsmonitor-test && git update-index --untracked-cache && From patchwork Fri Feb 25 17:10:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tao Klerks X-Patchwork-Id: 12760588 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82991C433EF for ; Fri, 25 Feb 2022 17:10:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243496AbiBYRKy (ORCPT ); Fri, 25 Feb 2022 12:10:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48342 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237751AbiBYRKw (ORCPT ); Fri, 25 Feb 2022 12:10:52 -0500 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B77EB1FED92 for ; Fri, 25 Feb 2022 09:10:19 -0800 (PST) Received: by mail-wm1-x32a.google.com with SMTP id l1-20020a7bcf01000000b0037f881182a8so2080213wmg.2 for ; Fri, 25 Feb 2022 09:10:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=raZeHnz0d+CORTk9WwiMcBzUr8yYuX4fJZnnYzX4to8=; b=BvSZ0W7hTJAhBNGt+N0H/vIkov9vcdpdfnSkpPQMcFx+jh0esiwD3gjzcABRVwhiRQ d+HcDANHXw7Tn6Bs+7k0NOR4vTEwpJg/SqR31zJF1IrvcSK3W6kZGZfi5cy1Lps8VC54 YzFrDSmNLPxHJddauvDTV0QaWI8YKfnpjvVn5EDjQ8iJ7U/cRYihOYJSpORQzXcaZ6Cy fmKKg8UGW/UVDVMiQoKLCHtRwuDUOKXYDDlHN2+TWpZ4Vi+1l4pSfKO+8EOH8XNb+Q3w t5HHjSKqbvKjjD44DE2S+x6qO0+XxFya+H5TvXrM/P4pAdOVrsoDi/7ceR4mUyCX8jE2 f7pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=raZeHnz0d+CORTk9WwiMcBzUr8yYuX4fJZnnYzX4to8=; b=iJnNW99yTxCRs96YdAWXBKO35LB0HOG/a9qKHj+sWUNROjitRfOpPopATCHebmPlAR AZY3+J9HrCUkJ+4pwSiWiFRx1bklv02YJvqL36kaenaz2I8WKZJ7NUdbTwachFkrjzG5 xUgNvyl4df48c94A+UZYQcDGRVBmkzCZbFLPUXxa6T6iORN5+BNCsYee9Oe1XFm481GP GRucafiqLJI+o88mMnnqsuv3wN40Od3iSwP9WhCnlpf1hD9ax7vuEwmQXibkjlYjReRD VaqyKUHs/KiN4tG8Vcj02xt7IWvriB7IOyixn21Hg9MlDccLiUdDCCQk5u0bmpc53m2v 4h2Q== X-Gm-Message-State: AOAM533uFMbllhlGvh1d35oRTe2C9WjA/2KhHtRkNBBDb8DVwmRh9PXY Bnq4obz3REbmTuUTDR1tr4prEc8X+ak= X-Google-Smtp-Source: ABdhPJwlAwqTsXL5Pq03evK0qejBtJdGDMUYeymSemDTS+U83BKb2C6gu2+Cglh7LEViaZc1t9Zz3Q== X-Received: by 2002:a05:600c:4b87:b0:381:fe:3c5e with SMTP id e7-20020a05600c4b8700b0038100fe3c5emr3527342wmp.47.1645809018013; Fri, 25 Feb 2022 09:10:18 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bh7-20020a05600c3d0700b00380f6ec4d5asm6030065wmb.16.2022.02.25.09.10.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Feb 2022 09:10:17 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 25 Feb 2022 17:10:13 +0000 Subject: [PATCH v2 2/3] t7519: populate untracked cache before test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Tao Klerks , Tao Klerks Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tao Klerks From: Tao Klerks In its current state, the t7519 test dealing with untracked cache assumes that "git update-index --untracked-cache" will *populate* the untracked cache. This is not correct - it will only add an empty untracked cache structure to the index. If we're going to compare two git status runs with something interesting happening in-between, we need to ensure that the index is in a stable/steady state *before* that first run. Achieve this by adding another prior "git status" run. At this stage this change does nothing, because there is a bug, addressed in the next patch, whereby once the empty untracked cache structure is added by the update-index invocation, the untracked cache gets updated in every subsequent "git status" call, but the index with these updates does not get written down. That bug actually invalidates this entire test case - but we're fixing that next. Signed-off-by: Tao Klerks --- t/t7519-status-fsmonitor.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index 3f984136ea9..fffc57120d6 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -341,6 +341,7 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR' git config core.fsmonitor .git/hooks/fsmonitor-test && git update-index --untracked-cache && git update-index --fsmonitor && + git status && GIT_TRACE2_PERF="$TRASH_DIRECTORY/trace-before" \ git status && test-tool dump-untracked-cache >../before From patchwork Fri Feb 25 17:10:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tao Klerks X-Patchwork-Id: 12760589 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 626BEC433F5 for ; Fri, 25 Feb 2022 17:10:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S243504AbiBYRKz (ORCPT ); Fri, 25 Feb 2022 12:10:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48352 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S243493AbiBYRKw (ORCPT ); Fri, 25 Feb 2022 12:10:52 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9890A1688CC for ; Fri, 25 Feb 2022 09:10:20 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id n25-20020a05600c3b9900b00380f41e51e6so2086351wms.2 for ; Fri, 25 Feb 2022 09:10:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Em19lLcMqW9ZtBoxwCezKUkfBKndxDcsrBMBl5tkRY0=; b=bQE5Oobl1RQZVb50G4s39PQxGsL4lfhWTNYHgt5wU05QT/u9dm5c8NSma+paAu9BVb 4c7asyRxuZW3/0GKOug9OyluuuGl+aNyNURRWbkh8rw+iRc2t+WlovSCRL9xED3Bq9J9 MYvZE2D+5lFOgNmRRnVpXRtE2LLrM1IcLmbrsCNCpaUJ4W8yiHBHkxT8OgCZVIGthUcq ahkT6EkDxkuggLQv1A7uH6oGJBP0Fe3Fip3wb0BbRTqzg02aUrjzKSnB/Nqr2XNVRArG JLEUsv9bohxth1J1GWJHhhoWSu3aGt/UAF0PgQUkIcx+0H+X8wWc6gIfXfGk/EECIKR/ bOOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Em19lLcMqW9ZtBoxwCezKUkfBKndxDcsrBMBl5tkRY0=; b=dFEg6PMDWJrr3rZyPdKLe617uwNUvpdXcwkzdMsZT3+eQQkfGBS4fFqOzIzbCzvBdm Sh2Btp5IM/eCJGR+wX3SHVr/AOX5A36d2YSOtI+MH5Es28mFcJFOfQApBBt3SijU7YUo jjbtUbtI/TCvDfpkAwwgx5xVrubcBWFZ0ttNExeGgf0ueEq4yDzoy16EPsAaB1s5BSZG BAyz3FUdrQG/OmYZZOx2wLe48x787S6gfhcoVLK1hLe9j6urjTl/QObumizQ9lBU5J3N Vnl1fYSI/x8948WuEvpQK2Mc3gNvsBSHMwKAqh9509Gb+4yDH9qXCzRkS2o3ybDrLWVl xFwg== X-Gm-Message-State: AOAM5310MXKBD3F2GYlE19XBzJ2Jue0NRbiSM1z6484ZSfArhwl65uet adgWfwaXFFWBZNt6i4D8nMhtGgt5uN4= X-Google-Smtp-Source: ABdhPJx6qPEH3aXfwX1Uqi0Ens/Joct2Kgs7hYr4OV7VGH35KDKlKw1+Ej7vgeXUXv4DeyATUu+btA== X-Received: by 2002:a05:600c:2d4c:b0:381:3f3a:4999 with SMTP id a12-20020a05600c2d4c00b003813f3a4999mr2009610wmg.13.1645809018958; Fri, 25 Feb 2022 09:10:18 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m10-20020adfe94a000000b001ef57f562ccsm2764249wrn.51.2022.02.25.09.10.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Feb 2022 09:10:18 -0800 (PST) Message-Id: <190b27e518a22e2253b318e8087446d761e918b2.1645809015.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 25 Feb 2022 17:10:14 +0000 Subject: [PATCH v2 3/3] untracked-cache: write index when populating empty untracked cache Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Tao Klerks , Tao Klerks Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Tao Klerks From: Tao Klerks It is expected that an empty/unpopulated untracked cache structure can be written to the index - by update-index, or by a "git status" call that sees the untracked cache should be enabled and is not, but is running with options that make the untracked cache non-applicable in that run (eg a pathspec). Currently, if that happens, then subsequent "git status" calls end up populating the untracked cache, but not writing the index (not saving their work) - so the performance outcome is almost identical to the cache being altogether disabled. This continues until the index gets written with the untracked cache populated, for some *other* reason, such as a working tree change. Detect the condition where an empty untracked cache exists in the index and we will collect the list of untracked paths, and queue an index write under that condition, so that the collected untracked paths can be written out to the untracked cache extension in the index. This change depends on previous fixes to t7519 for the "ignore .git changes when invalidating UNTR" test case to pass - before this fix, the test never actually did anything as it was not set up correctly. Signed-off-by: Tao Klerks --- dir.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/dir.c b/dir.c index d91295f2bcd..4eee45dec91 100644 --- a/dir.c +++ b/dir.c @@ -2781,7 +2781,8 @@ void remove_untracked_cache(struct index_state *istate) static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *dir, int base_len, - const struct pathspec *pathspec) + const struct pathspec *pathspec, + struct index_state *istate) { struct untracked_cache_dir *root; static int untracked_cache_disabled = -1; @@ -2845,8 +2846,11 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d return NULL; } - if (!dir->untracked->root) + if (!dir->untracked->root) { + /* Untracked cache existed but is not initialized; fix that */ FLEX_ALLOC_STR(dir->untracked->root, name, ""); + istate->cache_changed |= UNTRACKED_CHANGED; + } /* Validate $GIT_DIR/info/exclude and core.excludesfile */ root = dir->untracked->root; @@ -2916,7 +2920,7 @@ int read_directory(struct dir_struct *dir, struct index_state *istate, return dir->nr; } - untracked = validate_untracked_cache(dir, len, pathspec); + untracked = validate_untracked_cache(dir, len, pathspec, istate); if (!untracked) /* * make sure untracked cache code path is disabled,