From patchwork Wed Jul 21 21:06:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12392177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BD7BCC6377B for ; Wed, 21 Jul 2021 21:07:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9E6B3613CF for ; Wed, 21 Jul 2021 21:07:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229764AbhGUU00 (ORCPT ); Wed, 21 Jul 2021 16:26:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229870AbhGUU0Z (ORCPT ); Wed, 21 Jul 2021 16:26:25 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 03FACC061575 for ; Wed, 21 Jul 2021 14:07:02 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id t5so3575671wrw.12 for ; Wed, 21 Jul 2021 14:07:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=H841QFdSJazHTsJiCphF34tuE+VZ8kvrCGBiwT2/RrU=; b=oVDJ3toq1TJXWwIFTEKH/aZBRjcnuD9I8ts5Br253jGTwsUhqOQBfzT9zcXMP/Pq2n 7X49IXC73EgVuUH7FzhVemFn/oUGSb0haQFG9HkxKRNHHMBQAGENm4LzGSKCkYsgq79t 0ir1CqS1LhVJqz8crQNOLw/irbXwievP+/UuabbFxj4imveSoqt6VB58WWrceCvIU63m kWYJI2dRxd7fqQYskneBYtFWTcb41HnZlNEgQUTQdgqXyg7nPfxvzkbuehOBr3K2z6C1 WuPQsE/S9d7oIcCFodC3HftZorRe0PWI+lYzrm8Hwrzp9IExqEc+70Y33N+yU2ThPBdq 4w2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=H841QFdSJazHTsJiCphF34tuE+VZ8kvrCGBiwT2/RrU=; b=E/H5kUu7/yn0MoJkX3MLciiXBDtasG3vJ1yywUaPHYrA1cpyu+hjoX7D9AzJsPmlvB I1tNopqsypPmB4nDCjaXD+29QzVfCDhV4xubsU1zM1k3VAHcPWKq652oW118nt5HyMqn 6xPEJ5wY/3/bUVL7VKVi+z4ZsXr8N8cyIqpSG15Ek50FZg6d99x2gkf10Gj/nbo/wptf xB2uLw68DiEX38TjoJjiX8TGXuFIeN5xTQnKyexhqPL2ECxKXBNntuWX4Tt6pHO2S5PA asjx9oQgXR8+IrYTuX+aRuKXzMj4TSaXSC69EPQ/ppa9INoW+1WmaUcbxPmt2aBkfvsL e+oQ== X-Gm-Message-State: AOAM5303dwjO2ucNe1O7JBkPUtJhFYwGp4m28mvInsuos3HlMpQk/iQa MZALyUXGmUq7H2Cntm/Qw9EVwCqbWvg= X-Google-Smtp-Source: ABdhPJyCfp5RBCorxr68yVWwEBJp72Oc12pPHxJbfFMLyGh8pmDVFL8OdadubsRO8X5lKkf5QKUTAw== X-Received: by 2002:a5d:6da3:: with SMTP id u3mr44643763wrs.394.1626901620670; Wed, 21 Jul 2021 14:07:00 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e6sm32696130wrg.18.2021.07.21.14.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Jul 2021 14:07:00 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 21 Jul 2021 21:06:55 +0000 Subject: [PATCH 1/5] t1092: test merge conflicts outside cone Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 37 ++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 91e30d6ec22..a3c01d588d8 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -114,6 +114,16 @@ test_expect_success 'setup' ' git add . && git commit -m "file to dir" && + for side in left right + do + git checkout -b merge-$side base && + echo $side >>deep/deeper2/a && + echo $side >>folder1/a && + echo $side >>folder2/a && + git add . && + git commit -m "$side" || return 1 + done && + git checkout -b deepest base && echo "updated deepest" >deep/deeper1/deepest/a && git commit -a -m "update deepest" && @@ -482,6 +492,33 @@ test_expect_success 'merge' ' test_all_match git rev-parse HEAD^{tree} ' +test_expect_success 'merge with conflict outside cone' ' + init_repos && + + test_all_match git checkout -b merge-tip merge-left && + test_all_match git status --porcelain=v2 && + test_all_match test_must_fail git merge -m merge merge-right && + test_all_match git status --porcelain=v2 && + + # resolve the conflict in different ways: + # 1. revert to the base + test_all_match git checkout base -- deep/deeper2/a && + test_all_match git status --porcelain=v2 && + + # 2. add the file with conflict markers + test_all_match git add folder1/a && + test_all_match git status --porcelain=v2 && + + # 3. rename the file to another sparse filename + run_on_all mv folder2/a folder2/z && + test_all_match git add folder2 && + test_all_match git status --porcelain=v2 && + + test_all_match git merge --continue && + test_all_match git status --porcelain=v2 && + test_all_match git rev-parse HEAD^{tree} +' + test_expect_success 'merge with outside renames' ' init_repos && From patchwork Wed Jul 21 21:06:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12392179 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3C11CC6377B for ; Wed, 21 Jul 2021 21:07:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 28B6F613CF for ; Wed, 21 Jul 2021 21:07:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229885AbhGUU03 (ORCPT ); Wed, 21 Jul 2021 16:26:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35328 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229915AbhGUU02 (ORCPT ); Wed, 21 Jul 2021 16:26:28 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4C6B7C061575 for ; Wed, 21 Jul 2021 14:07:03 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id 14-20020a05600c028eb0290228f19cb433so337807wmk.0 for ; Wed, 21 Jul 2021 14:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ueE21I85gUWcxNadFee1ccr3Z6nBX8ycBGkFpABYXrY=; b=n89H8jN+ScWY5budL/Fp1iL9Lhta5sybM9h4cPxhrle0/a2KYDfOFDcbTrZU1Uw4iN aj655TFIe1wMonPFkV/WOx0zZQa5tcoYYdqRGjbbrZXv5pifFuAZ/g9VLXcj5OaA3gTR cYRzp+WTrIPAd/9ZvzA3QRmbN7ntcu/UVxkSWpsCLPOcsu8NXWMHodlGBsJHPsyAnjcZ 8zhY2pvhCQPKGSsvPW2UzZOH/EzAoM8+iPsazh2Gqo6+VftAxm763BKnsEPOMyHFPSEM ZguuoeUpjVkPnJ7TpqqGuxl43bBEGIMQHJxqAEA9+F/fG1yUvHMkR6gwGxx/mVbQUyea yOyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ueE21I85gUWcxNadFee1ccr3Z6nBX8ycBGkFpABYXrY=; b=eJ4xwxC9pfpPgzyfaDZa3uuwMj9Ki2hLRTxZRnzIwx67OimVVbJZ+cLrMM1e2k3JcO iipmNBOhNxkoztoy8B9P+MphhzlH4XVy1bAQdv93zzUyRd9pvGNICvgD4PxtHBIrKT2q Z/PSvzsYBSq2h1LQcuUboXBsGyxFXalX21pA9fzznCdtfOWHIBr68LE9lutKv2nk/WIr Tnkz+GAscI4PfjqSLnHQzcWCzKf2mClFbffjRZqKjzXdsHxEyZyCiQ9Ap6PkwOpO3skq oVPtJhi0eaJ96PVOUnuQBVUy2YJYFBo2ojWxJDvwoeTG6zc83EPzUwjmatg0cmMS2hiX EoJw== X-Gm-Message-State: AOAM533sG6uKN6/DdGYPVR9f574Hku1vgu6Supp7+ac3PQNZIgxyPW1f o3b3Ui3PFbTTyAvVSRSu5F6TXiFo7Gg= X-Google-Smtp-Source: ABdhPJzYcd7uaYcJ0asx3/RGrbOlbwsF6hKmhF6+xfKgmiNyrIsGRSdspWditRFioJtHlHR9Zzrr6A== X-Received: by 2002:a1c:4b0a:: with SMTP id y10mr39128011wma.178.1626901621331; Wed, 21 Jul 2021 14:07:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g15sm22957627wmh.44.2021.07.21.14.07.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Jul 2021 14:07:01 -0700 (PDT) Message-Id: <791c6c2c9ade5b065fc0f367e00c52a493d086ef.1626901619.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 21 Jul 2021 21:06:56 +0000 Subject: [PATCH 2/5] add: allow operating on a sparse-only index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Disable command_requires_full_index for 'git add'. This does not require any additional removals of ensure_full_index(). The main reason is that 'git add' discovers changes based on the pathspec and the worktree itself. These are then inserted into the index directly, and calls to index_name_pos() or index_file_exists() already call expand_to_path() at the appropriate time to support a sparse-index. Add a test to check that 'git add -A' and 'git add ' does not expand the index at all, as long as is not within a sparse directory. This does not help the global 'git add .' case. We can measure the improvement using p2000-sparse-operations.sh with these results: Test HEAD~1 HEAD ------------------------------------------------------------------------------ 2000.6: git add -A (full-index-v3) 0.35(0.30+0.05) 0.37(0.29+0.06) +5.7% 2000.7: git add -A (full-index-v4) 0.31(0.26+0.06) 0.33(0.27+0.06) +6.5% 2000.8: git add -A (sparse-index-v3) 0.57(0.53+0.07) 0.05(0.04+0.08) -91.2% 2000.9: git add -A (sparse-index-v4) 0.58(0.55+0.06) 0.05(0.05+0.06) -91.4% While the 91% improvement seems impressive, it's important to recognize that previously we had significant overhead for expanding the sparse-index. Comparing to the full index case, 'git add -A' goes from 0.37s to 0.05s, which is "only" an 86% improvement. Signed-off-by: Derrick Stolee --- builtin/add.c | 3 +++ t/t1092-sparse-checkout-compatibility.sh | 14 ++++++-------- 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index b773b5a4993..c76e6ddd359 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -528,6 +528,9 @@ int cmd_add(int argc, const char **argv, const char *prefix) add_new_files = !take_worktree_changes && !refresh_only && !add_renormalize; require_pathspec = !(take_worktree_changes || (0 < addremove_explicit)); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); /* diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index a3c01d588d8..a11d9d7f35d 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -340,13 +340,6 @@ test_expect_success 'status/add: outside sparse cone' ' test_sparse_match git status --porcelain=v2 && - # This "git add folder1/a" fails with a warning - # in the sparse repos, differing from the full - # repo. This is intentional. - test_sparse_match test_must_fail git add folder1/a && - test_sparse_match test_must_fail git add --refresh folder1/a && - test_all_match git status --porcelain=v2 && - test_all_match git add . && test_all_match git status --porcelain=v2 && test_all_match git commit -m folder1/new && @@ -635,7 +628,12 @@ test_expect_success 'sparse-index is not expanded' ' git -C sparse-index reset --hard && ensure_not_expanded checkout rename-out-to-out -- deep/deeper1 && git -C sparse-index reset --hard && - ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 + ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 && + + echo >>sparse-index/README.md && + ensure_not_expanded add -A && + echo >>sparse-index/extra.txt && + ensure_not_expanded add extra.txt ' # NEEDSWORK: a sparse-checkout behaves differently from a full checkout From patchwork Wed Jul 21 21:06:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12392181 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C8EBC6377A for ; Wed, 21 Jul 2021 21:07:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 54AA7613FD for ; Wed, 21 Jul 2021 21:07:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229933AbhGUU03 (ORCPT ); Wed, 21 Jul 2021 16:26:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35330 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229878AbhGUU02 (ORCPT ); Wed, 21 Jul 2021 16:26:28 -0400 Received: from mail-wm1-x32a.google.com (mail-wm1-x32a.google.com [IPv6:2a00:1450:4864:20::32a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F26CC061757 for ; Wed, 21 Jul 2021 14:07:03 -0700 (PDT) Received: by mail-wm1-x32a.google.com with SMTP id l11-20020a7bc34b0000b029021f84fcaf75so316933wmj.1 for ; Wed, 21 Jul 2021 14:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=J9N9Y3TTk0oY7Us5Osz7APqG+lo0OhkBPQ1D46Jl8tk=; b=Ob8DJPsr3ORl/J3YvnvjRQuDl1UTWCQ8friYMpZ6VsFOBW4AY/BjxRGjikHnjH3gi/ 8TXbmlxK3XMbYHP76A8x3obJGw+0Hyet/Z1T+nmCQX6tLkPSYs52y6hVP9ubFbSkTX2a F1FfgOhFuog48OkHKe2zmHY0V8H94p8gY+2GQeXKKt7Pr3uFr0zGhwheEUsmUDz1GD0t f82N/lTaZ0+BMgQ+tIdpdRlrsbPwQCuQKGellrNnhs+gCErsgKVCMtdMP1YjphHt107j qZMmMSMcgsAMkulTvyK64H8XuVEOfdk+7RHuCHeLF5YvJeZ+yMvIKjIJ5FcRf5qFUN31 ZGFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=J9N9Y3TTk0oY7Us5Osz7APqG+lo0OhkBPQ1D46Jl8tk=; b=beacc1zLe7W7YQXYpBn6XM1fjIEP0/Fo3qb0jou629TYPlBXQmf6lKXf4bcmOuhMPT GOOA7xLQk3kPyNYGzKklKQ9JMtnlfn7Ym08pgYZCuu2rrKfVD0B5OgTxXDiFyH14XwvG 0COdFEnbT/a5zmac/yrXbBIUCKiV2LroLYdB7Z4+BROG6E7VZrympp1Kyhf5gWPCFkr8 8dMPSUvEoYReCDhk6M7atF265eBy8SaHH2IfTLw1Kc5VPzZ+uMh+vvr7vaJp0ZFZA/HZ b1fm9DiQAZKyqXutsV6b08qY7MeALqZVClE0yE4hY7VtBVUt5BgX8bY+BNi6LHRtLWv3 RakA== X-Gm-Message-State: AOAM530Q41+KmI8RI2MCj1DvWHeSrgkQztzFqhMt3m3Zu+jHvWgt1X2B gC/PP8IJdIK7ursvodeSWPOaquYM7jE= X-Google-Smtp-Source: ABdhPJz+XJlWDmcxTmlsONY62whCZOVNqjBUAS15h46tStmBXdcbk3dH8IEttMSDMzaABPU8brNGeg== X-Received: by 2002:a05:600c:3509:: with SMTP id h9mr1063958wmq.81.1626901621969; Wed, 21 Jul 2021 14:07:01 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d18sm22009294wmp.46.2021.07.21.14.07.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Jul 2021 14:07:01 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Wed, 21 Jul 2021 21:06:57 +0000 Subject: [PATCH 3/5] pathspec: stop calling ensure_full_index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The add_pathspec_matches_against_index() focuses on matching a pathspec to file entries in the index. This already works correctly for its only use: checking if untracked files exist in the index. The compatibility checks in t1092 already test that 'git add ' works for a directory outside of the sparse cone. That provides coverage for removing this guard. This finalizes our ability to run 'git add .' without expanding a sparse index to a full one. This is evidenced by an update to t1092 and by these performance numbers for p2000-sparse-operations.sh: Test HEAD~1 HEAD -------------------------------------------------------------------------------- 2000.10: git add . (full-index-v3) 0.37(0.28+0.07) 0.36(0.27+0.06) -2.7% 2000.11: git add . (full-index-v4) 0.33(0.26+0.06) 0.32(0.28+0.05) -3.0% 2000.12: git add . (sparse-index-v3) 0.57(0.53+0.07) 0.06(0.06+0.07) -89.5% 2000.13: git add . (sparse-index-v4) 0.57(0.53+0.07) 0.05(0.03+0.09) -91.2% While the ~90% improvement is shown by the test results, it is worth noting that expanding the sparse index was adding overhead in previous commits. Comparing to the full index case, we see the performance go from 0.33s to 0.05s, an 85% improvement. Signed-off-by: Derrick Stolee --- pathspec.c | 2 -- t/t1092-sparse-checkout-compatibility.sh | 7 +++---- 2 files changed, 3 insertions(+), 6 deletions(-) diff --git a/pathspec.c b/pathspec.c index 08f8d3eedc3..44306fdaca2 100644 --- a/pathspec.c +++ b/pathspec.c @@ -37,8 +37,6 @@ void add_pathspec_matches_against_index(const struct pathspec *pathspec, num_unmatched++; if (!num_unmatched) return; - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(istate); for (i = 0; i < istate->cache_nr; i++) { const struct cache_entry *ce = istate->cache[i]; if (sw_action == PS_IGNORE_SKIP_WORKTREE && ce_skip_worktree(ce)) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index a11d9d7f35d..f9e2f5f4aa1 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -322,9 +322,6 @@ test_expect_success 'commit including unstaged changes' ' test_expect_success 'status/add: outside sparse cone' ' init_repos && - # adding a "missing" file outside the cone should fail - test_sparse_match test_must_fail git add folder1/a && - # folder1 is at HEAD, but outside the sparse cone run_on_sparse mkdir folder1 && cp initial-repo/folder1/a sparse-checkout/folder1/a && @@ -633,7 +630,9 @@ test_expect_success 'sparse-index is not expanded' ' echo >>sparse-index/README.md && ensure_not_expanded add -A && echo >>sparse-index/extra.txt && - ensure_not_expanded add extra.txt + ensure_not_expanded add extra.txt && + echo >>sparse-index/untracked.txt && + ensure_not_expanded add . ' # NEEDSWORK: a sparse-checkout behaves differently from a full checkout From patchwork Wed Jul 21 21:06:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12392183 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A955C6377C for ; Wed, 21 Jul 2021 21:07:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 606C6613CF for ; Wed, 21 Jul 2021 21:07:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229987AbhGUU0b (ORCPT ); Wed, 21 Jul 2021 16:26:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229669AbhGUU02 (ORCPT ); Wed, 21 Jul 2021 16:26:28 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CDA0FC0613C1 for ; Wed, 21 Jul 2021 14:07:03 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id f17so3590962wrt.6 for ; Wed, 21 Jul 2021 14:07:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZbZn9ZoRmasPgRxw/gERFUjtdCVlPbEWarvAwxum0j0=; b=sgMlKoWPhNy76K8s4tO+Zi4ZRX37PO9Spy9TJw+hldeMBdoKdHzvUbvavLK/hYPxy3 5N1T6PmuIlsJnYklf+uEu2qT3VQxBo7xScE/e/XdA5tPAytQi4wK8hKkxdaj2z/6SaNT jzzHR8nn3Y42Rm9hJ5rheR6jMifmqA8KSaHinKApiLnanwBu0yoskcQT/K8H4t1g+w3p PDyenIjEynjUBK91ul/0ms5N71IcaRnbaa/2ZB7paUyCjpCFh76mOhPYQCQ+gO9GKkvf B7S0PKIQOrys1KbdXcC8gRBc51LbMaUNqHKgyyIHnu8q3AwTD//v7FbfI36qmNFg8/oa K0Iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZbZn9ZoRmasPgRxw/gERFUjtdCVlPbEWarvAwxum0j0=; b=RkQa4y3edQBMDKyInw2C6UiF1+q3xAXgGIELxILgrgorQXLWXTZ/uzoYdRcQbDbJKA Bu4DQJACVUwZyE4lyxm/IxL4P7K6wQTtOkpHTJ5umvElK5mrPeDtIhLlLrOZOowbYJms 73sf8WssfjsO5bvWbirh3k9PewHTkP7UKriudHZTes+bV5FEdhwfulLpsHD5ZCcSqPNq +b4yEpZuE9+RI9plEvSphiN5hWLgllG1SixdT6PxJ4zmzJW4lrKADasPXSMA1JD4bU/k XLX2MKR1pr3BDDc8FqkZLT9zpO5vNnPztgqLmilFX9aneyUneZMb5WtPkMujeoJHKjI5 6r3w== X-Gm-Message-State: AOAM532ICwwei/fgsAbqQVNfrRp5uE3bKt968GsIPjdgr+rH665Gli6s H+wEdeGDoC6w858JwVpqHiYLngGJ8kQ= X-Google-Smtp-Source: ABdhPJzzzCoR/iD2DYoASW31Ur/d/hxANeHjPYcja10+yr7a5FvFxLeYX7p2KyHdZzP6nD/7pDvfOg== X-Received: by 2002:adf:ce8d:: with SMTP id r13mr46016866wrn.304.1626901622514; Wed, 21 Jul 2021 14:07:02 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o7sm32601181wrv.72.2021.07.21.14.07.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Jul 2021 14:07:02 -0700 (PDT) Message-Id: <89ec6a7ce677815cdd9889fb90c8c19f8f6dd28f.1626901619.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 21 Jul 2021 21:06:58 +0000 Subject: [PATCH 4/5] t1092: 'git add --refresh' difference with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When running 'git add --refresh ', Git does not actually stage the change, but instead updates the index based on the stat() information in the working directory. This typically does not make sense in a sparse-checkout scenario, where Git wants this file to not exist. However, sometimes the file can exist on-disk for other reasons, such as a user manually adding the file or a merge conflict outside of the sparse cone. Since b243012 (refresh_index(): add flag to ignore SKIP_WORKTREE entries, 2021-04-08), 'git add --refresh ' will output an advice message to indicate that this is not allowed when is outside the sparse cone. The check goes around the sparse index protections at the moment, so it does not find a match when contained in a sparse directory entry. We will update this behavior with a later change, but want to demonstrate the failure now. Signed-off-by: Derrick Stolee --- t/t1092-sparse-checkout-compatibility.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index f9e2f5f4aa1..73c48a71d89 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -347,6 +347,25 @@ test_expect_success 'status/add: outside sparse cone' ' test_all_match git commit -m folder1/newer ' +test_expect_failure 'add: pathspec within sparse directory' ' + init_repos && + + run_on_sparse mkdir folder1 && + run_on_sparse ../edit-contents folder1/a && + run_on_all ../edit-contents folder1/new && + + # This "git add folder1/a" fails with a warning + # in the sparse repos, differing from the full + # repo. This is intentional. + # + # However, in the sparse-index, folder1/a does not + # match any cache entry and fails with a different + # error message. This needs work. + test_sparse_match test_must_fail git add folder1/a && + test_sparse_match test_must_fail git add --refresh folder1/a && + test_all_match git status --porcelain=v2 +' + test_expect_success 'checkout and reset --hard' ' init_repos && From patchwork Wed Jul 21 21:06:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12392185 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F00CC6377B for ; Wed, 21 Jul 2021 21:07:09 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 14B5D613CF for ; Wed, 21 Jul 2021 21:07:09 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230059AbhGUU0c (ORCPT ); Wed, 21 Jul 2021 16:26:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:35338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229897AbhGUU03 (ORCPT ); Wed, 21 Jul 2021 16:26:29 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6C2A5C0613CF for ; Wed, 21 Jul 2021 14:07:04 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id m2so3614369wrq.2 for ; Wed, 21 Jul 2021 14:07:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XsQ49rBsJnlZxaHMiHDwuW+fcg5F8UkoyDC6X7QSgWU=; b=ZSubUMQZEXT2omTEE9Iuahtmetp9chFmzAuMK92NgeOdbGWzRqA44zEY2/L/R8jrmO fCQyMGISJOTlLAmSStqUJ75k2xLyMNDR/lu8Tdtpw1gokVUCyYXJga3/yNS0wvU8vfWf +DJ3zJ8vCvkf8xFYUzFhloVXQ7u2oT9QEJu0m3AU2lzVGmRu6Pmd4erDRWWSJf2p82DI o7757715S/mozUNoskRkMl7KSEQX/C4SrBoSUpJjGHxWNESMn9esLFjMrKN3/yKGbh8G DP1nuZSdR+R7uecZLRz2QAKCkcapINP1SA+1B+f1SzCnflhunR6YYfKBtSq11J+iPNQE 0Dsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XsQ49rBsJnlZxaHMiHDwuW+fcg5F8UkoyDC6X7QSgWU=; b=FtBeq+KyQ7NsYf+y47P9t/7p+VUvDHGax2Iv5W9wi+4XqzA1TwUmX3zafz1clW8tTB HAqz17Fhs1xJpKcaGg7JowvjIQt9v61CWi0L/a3WO3vznOVtPwlLNqkCeDLrMgPfTuvP MFOTA/5XbDv7GFW1ny5vzP78XaeHjEHUZ1FDXhSfgTrydeskB5HXXGEy+Wyx4/WYxwaC MFEgl5Z6e93+cVDGk4QtySjQZTVQQ5PEKZFsW0yGosf3t1vu4vnLRd9UnVTl8qPh8mam rJO6+9CfTViKGAhbmQK2vL6GNj/YcLIdL6WKYSDnXABJAgy/7pxZIQ/Y+NsZa7+LRAsi X6QQ== X-Gm-Message-State: AOAM530Ol0Y/BmMLeanuN00OOJCFb0OqB6BiXMiWiHDBPyr2wr19D2Nl anUPZOcOEtXOkwMfRk6mSh3CsFipuO8= X-Google-Smtp-Source: ABdhPJzUoOfJHgC1YERgM05lcNLvbv/XZwdaoyqsZHpuHuFG6u1HgExyGdPHPJ39Z2Ljw9r0ppVkwg== X-Received: by 2002:adf:ce83:: with SMTP id r3mr45237461wrn.204.1626901623090; Wed, 21 Jul 2021 14:07:03 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o3sm28091688wrm.5.2021.07.21.14.07.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 21 Jul 2021 14:07:02 -0700 (PDT) Message-Id: <76066a78ce077add4bc69819b017516e12ac0d45.1626901619.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 21 Jul 2021 21:06:59 +0000 Subject: [PATCH 5/5] add: ignore outside the sparse-checkout in refresh() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Since b243012 (refresh_index(): add flag to ignore SKIP_WORKTREE entries, 2021-04-08), 'git add --refresh ' will output a warning message when the path is outside the sparse-checkout definition. The implementation of this warning happened in parallel with the sparse-index work to add ensure_full_index() calls throughout the codebase. Update this loop to have the proper logic that checks to see if the pathspec is outside the sparse-checkout definition. This avoids the need to expand the sparse directory entry and determine if the path is tracked, untracked, or ignored. We simply avoid updating the stat() information because there isn't even an entry that matches the path! Signed-off-by: Derrick Stolee --- builtin/add.c | 10 +++++++++- t/t1092-sparse-checkout-compatibility.sh | 6 +----- 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index c76e6ddd359..d512ece655b 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -192,13 +192,21 @@ static int refresh(int verbose, const struct pathspec *pathspec) struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP; int flags = REFRESH_IGNORE_SKIP_WORKTREE | (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET); + struct pattern_list pl = { 0 }; + int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl); seen = xcalloc(pathspec->nr, 1); refresh_index(&the_index, flags, pathspec, seen, _("Unstaged changes after refreshing the index:")); for (i = 0; i < pathspec->nr; i++) { if (!seen[i]) { - if (matches_skip_worktree(pathspec, i, &skip_worktree_seen)) { + const char *path = pathspec->items[i].original; + int dtype = DT_REG; + + if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) || + (sparse_checkout_enabled && + !path_matches_pattern_list(path, strlen(path), NULL, + &dtype, &pl, &the_index))) { string_list_append(&only_match_skip_worktree, pathspec->items[i].original); } else { diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 73c48a71d89..c61424e2074 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -347,7 +347,7 @@ test_expect_success 'status/add: outside sparse cone' ' test_all_match git commit -m folder1/newer ' -test_expect_failure 'add: pathspec within sparse directory' ' +test_expect_success 'add: pathspec within sparse directory' ' init_repos && run_on_sparse mkdir folder1 && @@ -357,10 +357,6 @@ test_expect_failure 'add: pathspec within sparse directory' ' # This "git add folder1/a" fails with a warning # in the sparse repos, differing from the full # repo. This is intentional. - # - # However, in the sparse-index, folder1/a does not - # match any cache entry and fails with a different - # error message. This needs work. test_sparse_match test_must_fail git add folder1/a && test_sparse_match test_must_fail git add --refresh folder1/a && test_all_match git status --porcelain=v2