From patchwork Tue Aug 24 21:51:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455933 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6353C432BE for ; Tue, 24 Aug 2021 21:51:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80638611C8 for ; Tue, 24 Aug 2021 21:51:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235739AbhHXVwd (ORCPT ); Tue, 24 Aug 2021 17:52:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39118 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235551AbhHXVwc (ORCPT ); Tue, 24 Aug 2021 17:52:32 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88518C061764 for ; Tue, 24 Aug 2021 14:51:47 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id x12so33316353wrr.11 for ; Tue, 24 Aug 2021 14:51:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2790nSllAGE+Jryr1ZNUmiwSlcYNs/tn/3C9+QWeo8U=; b=bHoC/cFeawuCurYakzmgnfS9Z+WYf9JT0lZSKLSTu7c2S6t4fSfdergCSiPSanWlsj +GXH+Mq4ecVgKLbveBiFwa7RamELYmNt5Q+SA+DHhNuoVsnV2OQ5E66ozSGl0Ndj2O/l yP9dTPCPWZNHcDyoEkHygT9P1cStYzxf8hYCMRjauAuvVKCX81bzLaocCx8TTscXCERB h/gc2HzA8fNvu5CDj2lUaLBgKi7BWlH6Iqxp9WSWTd0+nFehKRV8zHBwmzNn+jbSZGSA 5IN/UvRfrjud0s+frNMc+O3HKlfIXAP9awXFqjI4j/4j1qq6CaJivNqfVM0TTBEWDDDm 1Ltg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2790nSllAGE+Jryr1ZNUmiwSlcYNs/tn/3C9+QWeo8U=; b=gMLwW38K+pAb/5zv434IMaKM7yIGVUHrhh6cWg4NlDRUCgrnCFZJ2gEbah8oVF09NW TAd3iJf4W4HqS5dUG/y8C4sBForbiEXixHEThVOQapuxiHSE6l1md62XmExsLXDuRR9+ wkg0il3KjekewUuQYQ3iGrgqFQevuut7qK9B3dZAxSBXKqibZjs4Dj5xr+fXsXxmokxG MD3rCS4yRVTOne10NBA6nyAAN4GS5qtxQAKnIZAqMNC8QnqUj+rxPMjZnKM3Km5raUsM kEX1LcGTpMnMnwPo/kOllTstbv3ziHpEvll7MVxcYSBAp9HscBMkebtMAxyHXrxKsmv8 3CWA== X-Gm-Message-State: AOAM530oJImvvZwypDbmbdq6VxzzHHDOsQ1FePJWuiizG34YJ8oJ88HY xTxeALOlXtGuH4Yplb8r2vcJpI7EabI= X-Google-Smtp-Source: ABdhPJxDvfWgvOda4S8iVjsFfKSV93nGauN2Vo6f0RdisFVmCXvvnJyyaldKg/OJJBKlYoDSVChOwQ== X-Received: by 2002:adf:f188:: with SMTP id h8mr5482705wro.218.1629841906148; Tue, 24 Aug 2021 14:51:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n15sm3134802wmq.7.2021.08.24.14.51.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:45 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:35 +0000 Subject: [PATCH v4 01/10] t7519: rewrite sparse index test Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The sparse index is tested with the FS Monitor hook and extension since f8fe49e (fsmonitor: integrate with sparse index, 2021-07-14). This test was very fragile because it shared an index across sparse and non-sparse behavior. Since that expansion and contraction could cause the index to lose its FS Monitor bitmap and token, behavior is fragile to changes in 'git sparse-checkout set'. Rewrite the test to use two clones of the original repo: full and sparse. This allows us to also keep the test files (actual, expect, trace2.txt) out of the repos we are testing with 'git status'. Signed-off-by: Derrick Stolee --- t/t7519-status-fsmonitor.sh | 38 ++++++++++++++++++++----------------- 1 file changed, 21 insertions(+), 17 deletions(-) diff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh index deea88d4431..f1463197b99 100755 --- a/t/t7519-status-fsmonitor.sh +++ b/t/t7519-status-fsmonitor.sh @@ -389,43 +389,47 @@ test_expect_success 'status succeeds after staging/unstaging' ' # If "!" is supplied, then we verify that we do not call ensure_full_index # during a call to 'git status'. Otherwise, we verify that we _do_ call it. check_sparse_index_behavior () { - git status --porcelain=v2 >expect && - git sparse-checkout init --cone --sparse-index && - git sparse-checkout set dir1 dir2 && + git -C full status --porcelain=v2 >expect && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git status --porcelain=v2 >actual && + git -C sparse status --porcelain=v2 >actual && test_region $1 index ensure_full_index trace2.txt && test_region fsm_hook query trace2.txt && test_cmp expect actual && - rm trace2.txt && - git sparse-checkout disable + rm trace2.txt } test_expect_success 'status succeeds with sparse index' ' - git reset --hard && + git clone . full && + git clone --sparse . sparse && + git -C sparse sparse-checkout init --cone --sparse-index && + git -C sparse sparse-checkout set dir1 dir2 && - test_config core.fsmonitor "$TEST_DIRECTORY/t7519/fsmonitor-all" && - check_sparse_index_behavior ! && - - write_script .git/hooks/fsmonitor-test<<-\EOF && + write_script .git/hooks/fsmonitor-test <<-\EOF && printf "last_update_token\0" EOF - git config core.fsmonitor .git/hooks/fsmonitor-test && + git -C full config core.fsmonitor ../.git/hooks/fsmonitor-test && + git -C sparse config core.fsmonitor ../.git/hooks/fsmonitor-test && check_sparse_index_behavior ! && - write_script .git/hooks/fsmonitor-test<<-\EOF && + write_script .git/hooks/fsmonitor-test <<-\EOF && printf "last_update_token\0" printf "dir1/modified\0" EOF check_sparse_index_behavior ! && - cp -r dir1 dir1a && - git add dir1a && - git commit -m "add dir1a" && + git -C sparse sparse-checkout add dir1a && + + for repo in full sparse + do + cp -r $repo/dir1 $repo/dir1a && + git -C $repo add dir1a && + git -C $repo commit -m "add dir1a" || return 1 + done && + git -C sparse sparse-checkout set dir1 dir2 && # This one modifies outside the sparse-checkout definition # and hence we expect to expand the sparse-index. - write_script .git/hooks/fsmonitor-test<<-\EOF && + write_script .git/hooks/fsmonitor-test <<-\EOF && printf "last_update_token\0" printf "dir1a/modified\0" EOF From patchwork Tue Aug 24 21:51:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455937 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C812C4320A for ; Tue, 24 Aug 2021 21:51:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 04A5D613AB for ; Tue, 24 Aug 2021 21:51:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235806AbhHXVwk (ORCPT ); Tue, 24 Aug 2021 17:52:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235738AbhHXVwd (ORCPT ); Tue, 24 Aug 2021 17:52:33 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 554D1C061757 for ; Tue, 24 Aug 2021 14:51:48 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id k20-20020a05600c0b5400b002e87ad6956eso1562845wmr.1 for ; Tue, 24 Aug 2021 14:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NRMabsLggRiXlRQv0kIMAy+bZxcG89mSHktSlIoVFk4=; b=i4wdarE4D7u6L05AXBAkvBME1uGVb99mH2kPwkYH3CvbNyymXBvU1jdLTBQl+nel2X ftC8lhS/5+7xhqMSQWBe5CQxiVAe4biPxoHgtacGso7P+oioY5RqaAgVSVDwgyXr4jen NNRpWc0IwIh1stO/QuQdmNQtrJq1H9kb5NLkjp+CSGqvOX1seXhcmlARJwHIVQl42TwZ WzKnt/hVMFit75JKBK3118p1mK/IRDlTys5W33gFLd7xu7gdgJ1Qs+vDHzxGI7xCivGp iKmNNN8EH6wtLXKiHTF50A5S63OJ9X3pYAqxd5An1WQ4lpg1bpZWKL6cOp7ffyUJrjsU IKuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NRMabsLggRiXlRQv0kIMAy+bZxcG89mSHktSlIoVFk4=; b=AFEzxVjnn1CplECGpAWC5537shO9Je2UgFeaL7CQ9WmgADpxP9G4lBxdTHDDOup5SS DYjAszPZZ4C5OpIfRYqa/j5KvWeYOgDaR/YmLFphHTVd4OltkWD0RJhop2iwSi4tofFJ dFmxeCjrhp5obOEZW42lz2+JfmS10EdJqPANKJYLolhm1mS+A3PHKaszZvjcKbBaY3fG K80MIz7+ulxEx0NWSQ3ApYs/E3gqGFMqpJIE6mbMyqRL8uWXy0oyGHu1z7EJTcO2MEtV P+B+KvfYnnARPyfUSYHLbz+zsgH4HfcP0g3guRCeItTZhF1Ar3qlZnvb5bn27kNCcbb1 dVjg== X-Gm-Message-State: AOAM533jziIPj8cbj+ux6CUTFzht+zCROedzcIN0Yx/1aokbHncUR3rP q8Zpugji+UxbXt4nG2Z9caPITYxvugQ= X-Google-Smtp-Source: ABdhPJy4jvrKm9T32JJbW6pW1nRuFgtL//UD4sIrwa60Tn2jSSoUYsZFTUQufzsU5E+tS0BNgtuVgA== X-Received: by 2002:a05:600c:350d:: with SMTP id h13mr5776445wmq.38.1629841907022; Tue, 24 Aug 2021 14:51:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r12sm19958131wrv.96.2021.08.24.14.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:46 -0700 (PDT) Message-Id: <8660877ba7a06971bb5c443bbc1e07825e950a60.1629841904.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:36 +0000 Subject: [PATCH v4 02/10] sparse-index: silently return when not using cone-mode patterns Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee While the sparse-index is only enabled when core.sparseCheckoutCone is also enabled, it is possible for the user to modify the sparse-checkout file manually in a way that does not match cone-mode patterns. In this case, we should refuse to convert an index into a sparse index, since the sparse_checkout_patterns will not be initialized with recursive and parent path hashsets. Also silently return if there are no cache entries, which is a simple case: there are no paths to make sparse! Signed-off-by: Derrick Stolee --- sparse-index.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/sparse-index.c b/sparse-index.c index c6b4feec413..cd6e0d5f408 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -130,7 +130,7 @@ static int index_has_unmerged_entries(struct index_state *istate) int convert_to_sparse(struct index_state *istate) { int test_env; - if (istate->split_index || istate->sparse_index || + if (istate->split_index || istate->sparse_index || !istate->cache_nr || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; @@ -158,10 +158,16 @@ int convert_to_sparse(struct index_state *istate) return 0; } - if (!istate->sparse_checkout_patterns->use_cone_patterns) { - warning(_("attempting to use sparse-index without cone mode")); - return -1; - } + /* + * We need cone-mode patterns to use sparse-index. If a user edits + * their sparse-checkout file manually, then we can detect during + * parsing that they are not actually using cone-mode patterns and + * hence we need to abort this conversion _without error_. Warnings + * already exist in the pattern parsing to inform the user of their + * bad patterns. + */ + if (!istate->sparse_checkout_patterns->use_cone_patterns) + return 0; /* * NEEDSWORK: If we have unmerged entries, then stay full. From patchwork Tue Aug 24 21:51:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BEC4EC432BE for ; Tue, 24 Aug 2021 21:51:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A95F4613AB for ; Tue, 24 Aug 2021 21:51:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236738AbhHXVwl (ORCPT ); Tue, 24 Aug 2021 17:52:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39128 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235746AbhHXVwd (ORCPT ); Tue, 24 Aug 2021 17:52:33 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E6A47C061764 for ; Tue, 24 Aug 2021 14:51:48 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id q11-20020a7bce8b0000b02902e6880d0accso2889948wmj.0 for ; Tue, 24 Aug 2021 14:51:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=0n87QV9Z/mw0KmAZmpsOxn8gQVw5CMem+cn2Z07qKf8=; b=QBxnBQJlPb/ybEJsZmjxz9b0zFFWHp6l+uJPQ7FaVnlGwKjsDazSR6ARqSlW3UdDHD rZhSXmkrUyLI97BBP18BJsEPPO5i06RbPpdwbbHCMdcDKuQlRqezhgOgGEY0PK/sJV6B desjPoKyalTbvbudEIgr9YfmQdLdKOLVPgF/vlgzmv62iAbOs2sgCbSnggYNudHFb8WP QjDGeNmGeO9JLWqttQOP7JusUOaNGxWdfUAYpUvQMn3/jyIJQfnDCgeqSYN9zYPYMBLs P7eiz2p6maWNJB7cN4/hGCYHeaG1UwAQlYgSy8h8SeKQ2NWPuZ02AMy8fV91fMiD6SkB 0Qmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=0n87QV9Z/mw0KmAZmpsOxn8gQVw5CMem+cn2Z07qKf8=; b=Ou1fEtQBD+ucLGtdb7QEQs2zLhidgBL2Ova1uqlP0yelEMcwuwYe8qtnopCd1voMio /PQYa1RhPH+Qy5tmDUSPjH3hdkfezR0TD4WXFoNIQr2E+F36yhIbZZRKGM4QWhaaFAN2 pmNKoCVoh5ZlemwdDI7oWMOjlEBsgr+23wwBuHsI+dc3+BmwQqZuUTmKjt5/3gArRblq mO9CIpRKDb6/dQJwQbxD20QgH7cLDbYHlbrLYwuBn2uzaG+HtlcZzkaDYFsyY+0+PRMX kVN0K4Fp/J/jmQAjj8XN2SAUvENbxniefy1at29Kha8auZ/pjugSszaOkAJ842ViqyA/ 1Z/w== X-Gm-Message-State: AOAM5307czGwykxotpWA5JA2BNqtFNthaILKtqSpiZ9X/lRsU6N/omdm ihKLRUGLEuBG9lmNaXuvjBokOYNvDEQ= X-Google-Smtp-Source: ABdhPJxfl9OGCVn1FZg+KKIypyZVqhtbsVcSzTL5t6SzOl5d5fasywL6omGqEokRr+0CwyNcma9K1A== X-Received: by 2002:a05:600c:354d:: with SMTP id i13mr5973480wmq.89.1629841907618; Tue, 24 Aug 2021 14:51:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q22sm3378792wmj.32.2021.08.24.14.51.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:47 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:37 +0000 Subject: [PATCH v4 03/10] sparse-index: silently return when cache tree fails Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee If cache_tree_update() returns a non-zero value, then it could not create the cache tree. This is likely due to a path having a merge conflict. Since we are already returning early, let's return silently to avoid making it seem like we failed to write the index at all. If we remove our dependence on the cache tree within convert_to_sparse(), then we could still recover from this scenario and have a sparse index. Signed-off-by: Derrick Stolee --- sparse-index.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/sparse-index.c b/sparse-index.c index cd6e0d5f408..d9b07695953 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -178,10 +178,12 @@ int convert_to_sparse(struct index_state *istate) /* Clear and recompute the cache-tree */ cache_tree_free(&istate->cache_tree); - if (cache_tree_update(istate, 0)) { - warning(_("unable to update cache-tree, staying full")); - return -1; - } + /* + * Silently return if there is a problem with the cache tree update, + * which might just be due to a conflict state in some entry. + */ + if (cache_tree_update(istate, 0)) + return 0; remove_fsmonitor(istate); From patchwork Tue Aug 24 21:51:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5B80C4320E for ; Tue, 24 Aug 2021 21:51:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8761A611C8 for ; Tue, 24 Aug 2021 21:51:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236910AbhHXVwm (ORCPT ); Tue, 24 Aug 2021 17:52:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39132 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235551AbhHXVwe (ORCPT ); Tue, 24 Aug 2021 17:52:34 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 91996C0613C1 for ; Tue, 24 Aug 2021 14:51:49 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id i6so9602629wrv.2 for ; Tue, 24 Aug 2021 14:51:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=MbU88O7tq8/LF2BlKKzEtM7WyKkge6STN6O1kXeP2X4=; b=tJkm2aojw2RytfQ8/b5ODRnins96LjP8D7/cTcB4BM/pqhh0DvVJWylxI24WCn1IzG oJrvjAaKmBeFiu7rEaO8643F5cMyP/4rKKQIvLCGxlRf4nss4f5Rg4jFomzX6rOW8baz p1HwDtNgflOW1vRc8oVovP3Wybq0hNjAFRxWsyNLtkJI2fGFLhT+wCLy5sVQe/46ZJc+ AycWG/1qv8bEghsDHiX0f3zSBhSYJDDw4OJHGPynydiF26q5/u7QiEDfuGUnA6XfNpl+ g9oBFml1zH3SHnc5/iqYfR08GdCDZLxcvI6tTBHoJWDFfJs2dvE/u67IsAOzXQgYUtJP mJTw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=MbU88O7tq8/LF2BlKKzEtM7WyKkge6STN6O1kXeP2X4=; b=hpjMhAmIq4FgdVveBZBHimIPSS+9i+WyprFH2hhOs0l5WV4dzQ62fb0qGFU6pkGUGR 7lslpGDbFS2mL7saHFYQNIuN8Zfa3K34gJUxiGQgOwKrzhlTX+hmetdquqy/STVZxKZY IyACzZVfW/EFSIlTsm+KnN2952k+D5qWpdwrpnAQzAgd4ZuGeP7ZOuDGtVU1axJ59wD1 ygrQzL4cI7v9pIRI6JwmU50TdmKadFjvqXNOx9sc5bHRVuIcaucFUPVkTHMldS077rHT G1Rp5LESSpVWh4mCazm6eApgNRCZ107VZVw0ITkm7wg75OvgM1EQQ8uqk9A6iSC8XFpJ eUCA== X-Gm-Message-State: AOAM5307lkAVBR6Zo8lpHzanx7u9PJFTRTA+u3cm4q+MrNavR3uGEwTg TGv55dIAg5bSzMOhd3/htVJtEFHfg0Y= X-Google-Smtp-Source: ABdhPJyQUi6mKX+4xDQlphXs/E3lx+FvJJOA4Lie/aIEBxmZJ1ZSFQS5Wv1SplamxspHrUz8QtYZUw== X-Received: by 2002:adf:fbc9:: with SMTP id d9mr21635764wrs.152.1629841908247; Tue, 24 Aug 2021 14:51:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 17sm3356362wmj.20.2021.08.24.14.51.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:47 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:38 +0000 Subject: [PATCH v4 04/10] sparse-index: use WRITE_TREE_MISSING_OK Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When updating the cache tree in convert_to_sparse(), the WRITE_TREE_MISSING_OK flag indicates that trees might be computed that do not already exist within the object database. This happens in cases such as 'git add' creating new trees that it wants to store in anticipation of a following 'git commit'. If this flag is not specified, then it might trigger a promisor fetch or a failure due to the object not existing locally. Use WRITE_TREE_MISSING_OK during convert_to_sparse() to avoid these possible reasons for the cache_tree_update() to fail. Signed-off-by: Derrick Stolee --- sparse-index.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sparse-index.c b/sparse-index.c index d9b07695953..880c5f72338 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -181,8 +181,11 @@ int convert_to_sparse(struct index_state *istate) /* * Silently return if there is a problem with the cache tree update, * which might just be due to a conflict state in some entry. + * + * This might create new tree objects, so be sure to use + * WRITE_TREE_MISSING_OK. */ - if (cache_tree_update(istate, 0)) + if (cache_tree_update(istate, WRITE_TREE_MISSING_OK)) return 0; remove_fsmonitor(istate); From patchwork Tue Aug 24 21:51:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92B3EC4338F for ; Tue, 24 Aug 2021 21:52:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7C147611C8 for ; Tue, 24 Aug 2021 21:52:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236340AbhHXVwn (ORCPT ); Tue, 24 Aug 2021 17:52:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39138 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235855AbhHXVwf (ORCPT ); Tue, 24 Aug 2021 17:52:35 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 48AE9C0613CF for ; Tue, 24 Aug 2021 14:51:50 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id u16so33361004wrn.5 for ; Tue, 24 Aug 2021 14:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=odmU7NgB1ACRlSceIxyK1EcqD7Mo50/XC06Fcs7zjso=; b=XyaSo/QgO4RrX2yFb0GXQj1Y1qA+xdXUpBlvgL9FCxQaPCTW4UirJW1F3qumPgzmPj NZQ1BjhwWnkuaA5NrWKZ5nztmUX6c+0XQX4Qpxj+ohXRn2QD27XqqUIX6rjW6u8hxNd2 p5dpPPj5+2vSEy+icYsMnecu3/cnZ0zvpaiQod9OYoFA1qyVfiXa1jTbRcdpyrB7j0Jc tyTjJtVlzLHBhVKszHoTrnICFrKOZwl0xOUUAcVHIJD6i7MntXibEY+9c553f1DBoHQD 7+x2nTy1FH/LrfCGbwHmJ4PTGqK1N9orw7nxdL0g2QWBxnjqao+TUMCfO+T8nBUxV5fG 3xBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=odmU7NgB1ACRlSceIxyK1EcqD7Mo50/XC06Fcs7zjso=; b=HfAhWhF3m3zYzOYE+3+T2ocgk+XM7FTu3/60a1ub8l14ujHLf/ir8nKFW3Ck8P5iYW aMXEglIi7FF+hMKQUQNiX7VG88dfWpUhG7zGENP3h9POAFwZIk+3Ic7Ukq9Qxa4ngKV+ nuM0fepzBJ4UgteBwDclZ4KzC331+OAxnpr5XYvOnIZkiBSXaOpbyP/alQYWe9Ilnkhv o93Ays/tJMCX7GJu0pSBmgMx7L5S444Hh0co59U/Tv1l4NmSNegj2LAwkZRYhSztc1tn x9Zg2N7bRPqAMrTt2GGkYvBnI7/lcIKu+j4HpTAR/1SaPzKPisjKDr1/B5E5tWOLnVil msRA== X-Gm-Message-State: AOAM531iMPVkFdeVS0FYvLwfXCor7UVge++NPl3tAvM2/iR3jTvL9OsF o4X1mS1g2DTuNSYWDNDPybOHq5f7+5w= X-Google-Smtp-Source: ABdhPJzZlr7Wpo8IG8zatmDqDMfRTGCzfdB3GIBk1c/GesBuwG1zz3LfjuaaFeKpSDyOJbdv5F4oxw== X-Received: by 2002:a05:6000:1081:: with SMTP id y1mr20266066wrw.415.1629841908897; Tue, 24 Aug 2021 14:51:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y4sm3480421wmi.22.2021.08.24.14.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:48 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:39 +0000 Subject: [PATCH v4 05/10] unpack-trees: fix nested sparse-dir search MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The iterated search in find_cache_entry() was recently modified to include a loop that searches backwards for a sparse directory entry that matches the given traverse_info and name_entry. However, the string comparison failed to actually concatenate those two strings, so this failed to find a sparse directory when it was not a top-level directory. This caused some errors in rare cases where a 'git checkout' spanned a diff that modified files within the sparse directory entry, but we could not correctly find the entry. Helped-by: Johannes Schindelin Helped-by: René Scharfe Signed-off-by: Derrick Stolee --- unpack-trees.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/unpack-trees.c b/unpack-trees.c index 5786645f315..47ef0cf4c53 100644 --- a/unpack-trees.c +++ b/unpack-trees.c @@ -1255,7 +1255,8 @@ static int sparse_dir_matches_path(const struct cache_entry *ce, static struct cache_entry *find_cache_entry(struct traverse_info *info, const struct name_entry *p) { - struct cache_entry *ce; + const char *path; + struct cache_entry *ce = NULL; int pos = find_cache_pos(info, p->path, p->pathlen); struct unpack_trees_options *o = info->data; @@ -1283,7 +1284,9 @@ static struct cache_entry *find_cache_entry(struct traverse_info *info, while (pos >= 0) { ce = o->src_index->cache[pos]; - if (strncmp(ce->name, p->path, p->pathlen)) + if (!skip_prefix(ce->name, info->traverse_path, &path) || + strncmp(path, p->path, p->pathlen) || + path[p->pathlen] != '/') return NULL; if (S_ISSPARSEDIR(ce->ce_mode) && From patchwork Tue Aug 24 21:51:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 705BDC432BE for ; Tue, 24 Aug 2021 21:52:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 54B8C6138B for ; Tue, 24 Aug 2021 21:52:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237383AbhHXVwp (ORCPT ); Tue, 24 Aug 2021 17:52:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235905AbhHXVwj (ORCPT ); Tue, 24 Aug 2021 17:52:39 -0400 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id ECAA4C061796 for ; Tue, 24 Aug 2021 14:51:50 -0700 (PDT) Received: by mail-wm1-x329.google.com with SMTP id 79-20020a1c0452000000b002e6cf79e572so2871199wme.1 for ; Tue, 24 Aug 2021 14:51:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=xgVFZpCgEwxSR63IhFeNQZTeJ2cnt9VEz2UJ67AnEPM=; b=IuWj03W4epfDI5t1SDaFu7hHtX3UL+FJAUHFF50Ce53MTt3gue0wA4KKIPaFWZMiP9 d2uKrzbFNYQKkqCCgOZkK4G1JgzuhAgHogFFt8HN+cLw0IzQhGQzoBaqqMNYuO+e7xu7 qZ4hqnGOWeMT3/3Pz3wtqIer5eK1w8CEAKVidFT9MTt7OhHLN+KkveTth5AsRhA+Tt/U hDwsIEEHJ75LyTHO+DQoecag1i8wIZE36YYnjRcU9mjEks1AcqTfN4GfzCIz7CRlP4x9 eO7+Ci37sampxBaRnmo/1fxNhMAUoGMzMmes+a+2H1KwK39pIiO0x4xDYZp92xpgov8+ Lh3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=xgVFZpCgEwxSR63IhFeNQZTeJ2cnt9VEz2UJ67AnEPM=; b=mhLmhP6EWfP34prnU0PCi7iG2LZndvJFgsJ/x7xR8/9gKu2wHr7GKXXoLVFUbXpCFu 4bSaM4rvVxrtu9F6Ub7hcJLoP53xsI7m9HE4OpToRhZ43zKt0a2/4ZXBWWT/glorYF7R OoddgztM3NGhkS7tKa5H+pGh3B+SVkwyA+n+hxayEj9X25vDVqc/8rt64rvwzoqH/aTn k8+KqmnzhoxIwSTXeoY9MNEZW9Er8cvEDZu5N7VamINcRrpgKbO4jxgE7qiCfjowBuNj NfKU9OU8weyQFwr7brHYxuCfqDtIDe0D5zC9yl+e0bxBBHq5+GW+ctYrT446Q5Zt8gSM pbgA== X-Gm-Message-State: AOAM5313e4w4mUwPiu02JktxAPy7De6fNk+JiFnHuANBDSLl4wklRnrz z2idiZfouh+5lGD7BqJr0Y3jvfmPM6A= X-Google-Smtp-Source: ABdhPJzqr8U5lQWQv5c45/8cfM++XsmCET3QbRjloEeHwNvE7PdC/r9dcknCSdoxXoOt188Sa5iwdg== X-Received: by 2002:a7b:c0c6:: with SMTP id s6mr5975899wmh.161.1629841909585; Tue, 24 Aug 2021 14:51:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k17sm4082794wmj.0.2021.08.24.14.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:49 -0700 (PDT) Message-Id: <1958751aa0ef84ec18ddc7363d1f0c9e6c3f8d4d.1629841904.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:40 +0000 Subject: [PATCH v4 06/10] sparse-checkout: create helper methods Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we integrate the sparse index into more builtins, we occasionally need to check the sparse-checkout patterns to see if a path is within the sparse-checkout cone. Create some helper methods that help initialize the patterns and check for pattern matching to make this easier. The existing callers of commands like get_sparse_checkout_patterns() use a custom 'struct pattern_list' that is not necessarily the one in the 'struct index_state', so there are not many previous uses that could adopt these helpers. There are just two in builtin/add.c and sparse-index.c that can use path_in_sparse_checkout(). We add a path_in_cone_mode_sparse_checkout() as well that will only return false if the path is outside of the sparse-checkout definition _and_ the sparse-checkout patterns are in cone mode. Signed-off-by: Derrick Stolee --- builtin/add.c | 7 +------ dir.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++ dir.h | 8 ++++++++ sparse-index.c | 14 +++----------- 4 files changed, 64 insertions(+), 17 deletions(-) diff --git a/builtin/add.c b/builtin/add.c index 17528e8f922..88a6c0c69fb 100644 --- a/builtin/add.c +++ b/builtin/add.c @@ -190,8 +190,6 @@ static int refresh(int verbose, const struct pathspec *pathspec) struct string_list only_match_skip_worktree = STRING_LIST_INIT_NODUP; int flags = REFRESH_IGNORE_SKIP_WORKTREE | (verbose ? REFRESH_IN_PORCELAIN : REFRESH_QUIET); - struct pattern_list pl = { 0 }; - int sparse_checkout_enabled = !get_sparse_checkout_patterns(&pl); seen = xcalloc(pathspec->nr, 1); refresh_index(&the_index, flags, pathspec, seen, @@ -199,12 +197,9 @@ static int refresh(int verbose, const struct pathspec *pathspec) for (i = 0; i < pathspec->nr; i++) { if (!seen[i]) { const char *path = pathspec->items[i].original; - int dtype = DT_REG; if (matches_skip_worktree(pathspec, i, &skip_worktree_seen) || - (sparse_checkout_enabled && - !path_matches_pattern_list(path, strlen(path), NULL, - &dtype, &pl, &the_index))) { + !path_in_sparse_checkout(path, &the_index)) { string_list_append(&only_match_skip_worktree, pathspec->items[i].original); } else { diff --git a/dir.c b/dir.c index 03c4d212672..86afa2eae00 100644 --- a/dir.c +++ b/dir.c @@ -1439,6 +1439,58 @@ done: return result; } +int init_sparse_checkout_patterns(struct index_state *istate) +{ + if (!core_apply_sparse_checkout) + return 1; + if (istate->sparse_checkout_patterns) + return 0; + + CALLOC_ARRAY(istate->sparse_checkout_patterns, 1); + + if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) { + FREE_AND_NULL(istate->sparse_checkout_patterns); + return -1; + } + + return 0; +} + +static int path_in_sparse_checkout_1(const char *path, + struct index_state *istate, + int require_cone_mode) +{ + const char *base; + int dtype = DT_REG; + + /* + * We default to accepting a path if there are no patterns or + * they are of the wrong type. + */ + if (init_sparse_checkout_patterns(istate) || + (require_cone_mode && + !istate->sparse_checkout_patterns->use_cone_patterns)) + return 1; + + base = strrchr(path, '/'); + return path_matches_pattern_list(path, strlen(path), base ? base + 1 : path, + &dtype, + istate->sparse_checkout_patterns, + istate) > 0; +} + +int path_in_sparse_checkout(const char *path, + struct index_state *istate) +{ + return path_in_sparse_checkout_1(path, istate, 0); +} + +int path_in_cone_mode_sparse_checkout(const char *path, + struct index_state *istate) +{ + return path_in_sparse_checkout_1(path, istate, 1); +} + static struct path_pattern *last_matching_pattern_from_lists( struct dir_struct *dir, struct index_state *istate, const char *pathname, int pathlen, diff --git a/dir.h b/dir.h index b3e1a54a971..6823312521e 100644 --- a/dir.h +++ b/dir.h @@ -394,6 +394,14 @@ enum pattern_match_result path_matches_pattern_list(const char *pathname, const char *basename, int *dtype, struct pattern_list *pl, struct index_state *istate); + +int init_sparse_checkout_patterns(struct index_state *state); + +int path_in_sparse_checkout(const char *path, + struct index_state *istate); +int path_in_cone_mode_sparse_checkout(const char *path, + struct index_state *istate); + struct dir_entry *dir_add_ignored(struct dir_struct *dir, struct index_state *istate, const char *pathname, int len); diff --git a/sparse-index.c b/sparse-index.c index 880c5f72338..23f7c3bd361 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -33,19 +33,14 @@ static int convert_to_sparse_rec(struct index_state *istate, { int i, can_convert = 1; int start_converted = num_converted; - enum pattern_match_result match; - int dtype = DT_UNKNOWN; struct strbuf child_path = STRBUF_INIT; - struct pattern_list *pl = istate->sparse_checkout_patterns; /* * Is the current path outside of the sparse cone? * Then check if the region can be replaced by a sparse * directory entry (everything is sparse and merged). */ - match = path_matches_pattern_list(ct_path, ct_pathlen, - NULL, &dtype, pl, istate); - if (match != NOT_MATCHED) + if (path_in_sparse_checkout(ct_path, istate)) can_convert = 0; for (i = start; can_convert && i < end; i++) { @@ -152,11 +147,8 @@ int convert_to_sparse(struct index_state *istate) if (!istate->repo->settings.sparse_index) return 0; - if (!istate->sparse_checkout_patterns) { - istate->sparse_checkout_patterns = xcalloc(1, sizeof(struct pattern_list)); - if (get_sparse_checkout_patterns(istate->sparse_checkout_patterns) < 0) - return 0; - } + if (init_sparse_checkout_patterns(istate)) + return 0; /* * We need cone-mode patterns to use sparse-index. If a user edits From patchwork Tue Aug 24 21:51:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A5B3C4320E for ; Tue, 24 Aug 2021 21:52:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 54A86611C8 for ; Tue, 24 Aug 2021 21:52:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236674AbhHXVwq (ORCPT ); Tue, 24 Aug 2021 17:52:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39162 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236318AbhHXVwj (ORCPT ); Tue, 24 Aug 2021 17:52:39 -0400 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8EF0AC0617AD for ; Tue, 24 Aug 2021 14:51:51 -0700 (PDT) Received: by mail-wm1-x331.google.com with SMTP id j17-20020a05600c1c1100b002e754875260so3220619wms.4 for ; Tue, 24 Aug 2021 14:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=i9HtyxcyL/rFpqLzUK1JHZNiZLF1vUp90EnsiHFgFOg=; b=nw7A4dwpQi1pz+4s6K++BF9XvLrgP6iYten5Jd4c3Jh/sLtwIgzvxPcM8jHOPS9Tn2 ogF0MOi5dqF8/TijbOnSXukBl7buw0N3ukKXkgu5ivByTofKJz4VI6EMFDJimMM7vIlZ F5wIT3NWZBHvMrXRhUpGtVVGenf74SfNk45SzYASrOZoEY89L7tdQguru0wjMEXSe8ro ro6AsX82XQsX7L+2qyRkUNAVN1iZJ8iU7QYnJG3jpSyuCwqHexFVYbmdgs7d1aRud7E3 HFawKUEE4+lYeaxMycJX5Dz94hTxlLb+Qh8FUQ7PSKBgDh9bd/A0XFOalpEo5gm5S3FB /Niw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=i9HtyxcyL/rFpqLzUK1JHZNiZLF1vUp90EnsiHFgFOg=; b=EmUeUTX1m2Fb/9bWxzkCStazqswPO+CpYOn3sM3xAUZrUkEgyGh5p+gPUD2CSUyzVo 4dABa1ZADhd/BQS4yriexT5pDBGHaL8mr3oWEFOu3G15KCbGNQhvhqF+2oUyoJlFGiBW stSnRU/Nca2SjzI8MLZn1CVR0QtRSFwbZMHa34JKDx1HA91mvMHWvCDmUtLKOUKOSGkx tc0mKAUqjWgWr7SIM1qQDexsfDx9jdoupo4vKNQQ8z0FmEVQzKYFDQeoLO3Vz/u9/8qV wd+AcUhyD77mjJhDKXCV/h60pIYkjaUQHbOXxUDFswuiTZaPTwslkPGYgVbK0P1iMIMh 1TGQ== X-Gm-Message-State: AOAM5327VOTJ07PY4KsWQka1gauHknZKjYEc7sTee8FiIB86s7IbgUmH JxU7a2TOcpSixOlN7/iZP0UipzryLS4= X-Google-Smtp-Source: ABdhPJySd0Zb0VTyEvHCs7X0OZJLbnHJY1NKTmfNnusOS0m3eAjVS4w2awOwaiADkREBKBJz9Le6ZA== X-Received: by 2002:a05:600c:3b98:: with SMTP id n24mr5879707wms.11.1629841910172; Tue, 24 Aug 2021 14:51:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s14sm3355989wmc.25.2021.08.24.14.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:49 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:41 +0000 Subject: [PATCH v4 07/10] attr: be careful about sparse directories Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Signed-off-by: Derrick Stolee --- attr.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/attr.c b/attr.c index d029e681f28..62db4aed443 100644 --- a/attr.c +++ b/attr.c @@ -14,6 +14,7 @@ #include "utf8.h" #include "quote.h" #include "thread-utils.h" +#include "dir.h" const char git_attr__true[] = "(builtin)true"; const char git_attr__false[] = "\0(builtin)false"; @@ -744,6 +745,19 @@ static struct attr_stack *read_attr_from_index(struct index_state *istate, if (!istate) return NULL; + /* + * The .gitattributes file only applies to files within its + * parent directory. In the case of cone-mode sparse-checkout, + * the .gitattributes file is sparse if and only if all paths + * within that directory are also sparse. Thus, don't load the + * .gitattributes file since it will not matter. + * + * In the case of a sparse index, it is critical that we don't go + * looking for a .gitattributes file, as the index will expand. + */ + if (!path_in_cone_mode_sparse_checkout(path, istate)) + return NULL; + buf = read_blob_data_from_index(istate, path, NULL); if (!buf) return NULL; From patchwork Tue Aug 24 21:51:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72AFDC4338F for ; Tue, 24 Aug 2021 21:52:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 57D826127B for ; Tue, 24 Aug 2021 21:52:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235659AbhHXVwq (ORCPT ); Tue, 24 Aug 2021 17:52:46 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39168 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236374AbhHXVwj (ORCPT ); Tue, 24 Aug 2021 17:52:39 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CEE1C0617AE for ; Tue, 24 Aug 2021 14:51:52 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id z9so33286433wrh.10 for ; Tue, 24 Aug 2021 14:51:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=M+K0Z9WIqh5h1pKa505Bywi5J0nM3YeVcKgYGu5P63g=; b=AMYHVr6NQj08noxdS7ORl6QxRGWCBfqJ++ZdkX+i31WK8q9clWjysa1/C5NxTAsN6W yAmU4RJAkDk7zYobyDbJlaH72TOzZ59sOsOvchhIxGrtOF+8CbxKN24eMoqhkBKxjItI MfQ7xUKu9ztclNY4Mq9TUeebvghh3zRWbw/Z20UmRMtEow7AK98JcXe3GuCrz7HpLzBZ S6vA9Qddxg/tLlBoDc33/UqhS2S3YAr6w+OnW3yxVtGcLPVORRdka8pLI5BUkfhNpqnc 8e33HMJpL4VtDA5DnRxOlNQ4z6elZXJ0D6PnReabje67KkmHbAzB3pagnZRRCjmDRj5V UCEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=M+K0Z9WIqh5h1pKa505Bywi5J0nM3YeVcKgYGu5P63g=; b=eUNR7qeTcrLTJRMyRieHJGdcmXu2UG8PmCgrZ252aIU75xG6lGX271Jr445IqMDdDo Hki8UX+kTZS2tejizZ6jXlbV2yUm+tOHGn9md4ExTETt0Wf38uc3S7wOMuMEu5Q+K8uS IzEoXUpi95yI4FMe2RvKp1BxdINyyLcchJghKwhobohb7+/qCPfDVuQ239EczEaDBXpR uYz/dlWrusrL82q3y3GM8K/F+/vchjulajHgUtXIDd12lanPQOhB7+c4cOrZPouXFiPG MSxYY5I+z14BTsKnVobld77eZ0Gtl52HFQQQTLSB7Ere2U45CrnV2UuBDf8MuIem9qYb PGmw== X-Gm-Message-State: AOAM533ceCJcgtrj2J2GdJ/wz73Ye8PRjpLNjzDnm8oQaGFbaVZCq9kU QsfLvfpv8wZnL1iEm9oP/Zw2/FCD7w0= X-Google-Smtp-Source: ABdhPJwui8AS9RqXkQaCy487J36LCxhz3v4dHdJKcCZSCivie6jClOTMFozfLoQZOLMWDivAdX8FlQ== X-Received: by 2002:adf:e604:: with SMTP id p4mr5290708wrm.37.1629841910913; Tue, 24 Aug 2021 14:51:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s13sm3507246wmc.47.2021.08.24.14.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:50 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:42 +0000 Subject: [PATCH v4 08/10] sparse-index: add SPARSE_INDEX_MEMORY_ONLY flag Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The convert_to_sparse() method checks for the GIT_TEST_SPARSE_INDEX environment variable or the "index.sparse" config setting before converting the index to a sparse one. This is for ease of use since all current consumers are preparing to compress the index before writing it to disk. If these settings are not enabled, then convert_to_sparse() silently returns without doing anything. We will add a consumer in the next change that wants to use the sparse index as an in-memory data structure, regardless of whether the on-disk format should be sparse. To that end, create the SPARSE_INDEX_MEMORY_ONLY flag that will skip these config checks when enabled. All current consumers are modified to pass '0' in the new 'flags' parameter. Signed-off-by: Derrick Stolee --- read-cache.c | 4 ++-- sparse-index.c | 37 ++++++++++++++++++++++--------------- sparse-index.h | 3 ++- 3 files changed, 26 insertions(+), 18 deletions(-) diff --git a/read-cache.c b/read-cache.c index 9048ef9e905..f5d4385c408 100644 --- a/read-cache.c +++ b/read-cache.c @@ -3069,7 +3069,7 @@ static int do_write_locked_index(struct index_state *istate, struct lock_file *l int ret; int was_full = !istate->sparse_index; - ret = convert_to_sparse(istate); + ret = convert_to_sparse(istate, 0); if (ret) { warning(_("failed to convert to a sparse-index")); @@ -3182,7 +3182,7 @@ static int write_shared_index(struct index_state *istate, int ret, was_full = !istate->sparse_index; move_cache_to_base_index(istate); - convert_to_sparse(istate); + convert_to_sparse(istate, 0); trace2_region_enter_printf("index", "shared/do_write_index", the_repository, "%s", get_tempfile_path(*temp)); diff --git a/sparse-index.c b/sparse-index.c index 23f7c3bd361..0bc45f60ac5 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -122,30 +122,37 @@ static int index_has_unmerged_entries(struct index_state *istate) return 0; } -int convert_to_sparse(struct index_state *istate) +int convert_to_sparse(struct index_state *istate, int flags) { int test_env; - if (istate->split_index || istate->sparse_index || !istate->cache_nr || + if (istate->sparse_index || !istate->cache_nr || !core_apply_sparse_checkout || !core_sparse_checkout_cone) return 0; if (!istate->repo) istate->repo = the_repository; - /* - * The GIT_TEST_SPARSE_INDEX environment variable triggers the - * index.sparse config variable to be on. - */ - test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1); - if (test_env >= 0) - set_sparse_index_config(istate->repo, test_env); + if (!(flags & SPARSE_INDEX_MEMORY_ONLY)) { + /* + * The sparse index is not (yet) integrated with a split index. + */ + if (istate->split_index) + return 0; + /* + * The GIT_TEST_SPARSE_INDEX environment variable triggers the + * index.sparse config variable to be on. + */ + test_env = git_env_bool("GIT_TEST_SPARSE_INDEX", -1); + if (test_env >= 0) + set_sparse_index_config(istate->repo, test_env); - /* - * Only convert to sparse if index.sparse is set. - */ - prepare_repo_settings(istate->repo); - if (!istate->repo->settings.sparse_index) - return 0; + /* + * Only convert to sparse if index.sparse is set. + */ + prepare_repo_settings(istate->repo); + if (!istate->repo->settings.sparse_index) + return 0; + } if (init_sparse_checkout_patterns(istate)) return 0; diff --git a/sparse-index.h b/sparse-index.h index 1115a0d7dd9..9f3d7bc7faf 100644 --- a/sparse-index.h +++ b/sparse-index.h @@ -2,7 +2,8 @@ #define SPARSE_INDEX_H__ struct index_state; -int convert_to_sparse(struct index_state *istate); +#define SPARSE_INDEX_MEMORY_ONLY (1 << 0) +int convert_to_sparse(struct index_state *istate, int flags); /* * Some places in the codebase expect to search for a specific path. From patchwork Tue Aug 24 21:51:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455951 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 283C2C432BE for ; Tue, 24 Aug 2021 21:52:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 12F566127B for ; Tue, 24 Aug 2021 21:52:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237854AbhHXVws (ORCPT ); Tue, 24 Aug 2021 17:52:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236380AbhHXVwk (ORCPT ); Tue, 24 Aug 2021 17:52:40 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A98FFC0613A4 for ; Tue, 24 Aug 2021 14:51:53 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id e5so16776011wrp.8 for ; Tue, 24 Aug 2021 14:51:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=6BhipxpdLHCjlk4DMmYt4bLPZa+qCBdpfxVEfdEvEY0=; b=reoyDRmJ+9yAM3OUOBU5YwAc1OY5goeArhBv93ipTXr6mrwhb/J56vcRPTZpe3vUKg hyuU4+Sie6/VTwnYc3WdN6wxSeseSbh1mbtPib0hQVj2B/V5iBBbvR8Qpfd1HtXN3gw1 aiRKAR6aPLQsMHUKdkSMIey4OLCVjg80d7J0hjf/C+6ejNfnNEHg18fJnOgAyK38FYnE nPklPlIvPv5Z4K7XYu2tgveU5ol/COtEMbR5+Cjq3vJA7zgSiEABWyJdZCEdwUXO3HSJ NPKPBFLap3LGZh8pNHaJBFY7FtW3GGaCRRIr2oGj1bgEYc84UU5y7by3JVEklQj/K7yR 1O0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=6BhipxpdLHCjlk4DMmYt4bLPZa+qCBdpfxVEfdEvEY0=; b=s7W1cbUR7riQb2UV+bTaKeFzlI3uEjIgBJitIRiRRFpsnf+Wg3D/HbBo2dghAaGmcq mYeMEb5ua3gqjdD97NJK0cYrozvEdCgph5UVJ/WMDhjTv5tp+ZSmyDGWeXzrRxRnTR/X hlhEGSvTI0A1BTQPCTrzLSYsWTJc2x0Mvtm3zZIlW1Fb3rpPbqkMAVeB+Q7KKSPF4bvE goy5KFIc2EHubgEWtQ6PQ4gHiOUCI9Qf4bD7oRLJHRmsW1m0P6Xozr0IHUeVFjzQTXfi yVDthwYPcw+QzS96D++D/2zt2ctv2Swxbycqc7FAPL728Kvd+F7XrABdPowkhddmhTdZ PH1g== X-Gm-Message-State: AOAM533VAFty7jnz90/gnmWJ+bQlmA6vbdS/mDPfrTE1jx5r03Wxv0fd voP6VS7mVjUcnu8shDvs4Mb/v0emeQw= X-Google-Smtp-Source: ABdhPJxejw9VHhCZS9C+7W56NIl3oppHtSpGjtD9lZ0SUkCmzItq3w5lPSfdVzpGjrHD9F9gDHvQWA== X-Received: by 2002:a5d:47af:: with SMTP id 15mr21492230wrb.35.1629841912282; Tue, 24 Aug 2021 14:51:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o21sm3655648wms.32.2021.08.24.14.51.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:51 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:43 +0000 Subject: [PATCH v4 09/10] sparse-checkout: clear tracked sparse dirs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When changing the scope of a sparse-checkout using cone mode, we might have some tracked directories go out of scope. The current logic removes the tracked files from within those directories, but leaves the ignored files within those directories. This is a bit unexpected to users who have given input to Git saying they don't need those directories anymore. This is something that is new to the cone mode pattern type: the user has explicitly said "I want these directories and _not_ those directories." The typical sparse-checkout patterns more generally apply to "I want files with with these patterns" so it is natural to leave ignored files as they are. This focus on directories in cone mode provides us an opportunity to change the behavior. Leaving these ignored files in the sparse directories makes it impossible to gain performance benefits in the sparse index. When we track into these directories, we need to know if the files are ignored or not, which might depend on the _tracked_ .gitignore file(s) within the sparse directory. This depends on the indexed version of the file, so the sparse directory must be expanded. We must take special care to look for untracked, non-ignored files in these directories before deleting them. We do not want to delete any meaningful work that the users were doing in those directories and perhaps forgot to add and commit before switching sparse-checkout definitions. Since those untracked files might be code files that generated ignored build output, also do not delete any ignored files from these directories in that case. The users can recover their state by resetting their sparse-checkout definition to include that directory and continue. Alternatively, they can see the warning that is presented and delete the directory themselves to regain the performance they expect. By deleting the sparse directories when changing scope (or running 'git sparse-checkout reapply') we regain these performance benefits as if the repository was in a clean state. Since these ignored files are frequently build output or helper files from IDEs, the users should not need the files now that the tracked files are removed. If the tracked files reappear, then they will have newer timestamps than the build artifacts, so the artifacts will need to be regenerated anyway. Use the sparse-index as a data structure in order to find the sparse directories that can be safely deleted. Re-expand the index to a full one if it was full before. Signed-off-by: Derrick Stolee --- Documentation/git-sparse-checkout.txt | 10 +++ builtin/sparse-checkout.c | 94 +++++++++++++++++++++++++++ t/t1091-sparse-checkout-builtin.sh | 59 +++++++++++++++++ 3 files changed, 163 insertions(+) diff --git a/Documentation/git-sparse-checkout.txt b/Documentation/git-sparse-checkout.txt index fdcf43f87cb..42056ee9ff9 100644 --- a/Documentation/git-sparse-checkout.txt +++ b/Documentation/git-sparse-checkout.txt @@ -210,6 +210,16 @@ case-insensitive check. This corrects for case mismatched filenames in the 'git sparse-checkout set' command to reflect the expected cone in the working directory. +When changing the sparse-checkout patterns in cone mode, Git will inspect each +tracked directory that is not within the sparse-checkout cone to see if it +contains any untracked files. If all of those files are ignored due to the +`.gitignore` patterns, then the directory will be deleted. If any of the +untracked files within that directory is not ignored, then no deletions will +occur within that directory and a warning message will appear. If these files +are important, then reset your sparse-checkout definition so they are included, +use `git add` and `git commit` to store them, then remove any remaining files +manually to ensure Git can behave optimally. + SUBMODULES ---------- diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index 8ba9f13787b..d0f5c4702be 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -100,6 +100,98 @@ static int sparse_checkout_list(int argc, const char **argv) return 0; } +static void clean_tracked_sparse_directories(struct repository *r) +{ + int i, was_full = 0; + struct strbuf path = STRBUF_INIT; + size_t pathlen; + struct string_list_item *item; + struct string_list sparse_dirs = STRING_LIST_INIT_DUP; + + /* + * If we are not using cone mode patterns, then we cannot + * delete directories outside of the sparse cone. + */ + if (!r || !r->index || !r->worktree) + return; + if (init_sparse_checkout_patterns(r->index) || + !r->index->sparse_checkout_patterns->use_cone_patterns) + return; + + /* + * Use the sparse index as a data structure to assist finding + * directories that are safe to delete. This conversion to a + * sparse index will not delete directories that contain + * conflicted entries or submodules. + */ + if (!r->index->sparse_index) { + /* + * If something, such as a merge conflict or other concern, + * prevents us from converting to a sparse index, then do + * not try deleting files. + */ + if (convert_to_sparse(r->index, SPARSE_INDEX_MEMORY_ONLY)) + return; + was_full = 1; + } + + strbuf_addstr(&path, r->worktree); + strbuf_complete(&path, '/'); + pathlen = path.len; + + /* + * Collect directories that have gone out of scope but also + * exist on disk, so there is some work to be done. We need to + * store the entries in a list before exploring, since that might + * expand the sparse-index again. + */ + for (i = 0; i < r->index->cache_nr; i++) { + struct cache_entry *ce = r->index->cache[i]; + + if (S_ISSPARSEDIR(ce->ce_mode) && + repo_file_exists(r, ce->name)) + string_list_append(&sparse_dirs, ce->name); + } + + for_each_string_list_item(item, &sparse_dirs) { + struct dir_struct dir = DIR_INIT; + struct pathspec p = { 0 }; + struct strvec s = STRVEC_INIT; + + strbuf_setlen(&path, pathlen); + strbuf_addstr(&path, item->string); + + dir.flags |= DIR_SHOW_IGNORED_TOO; + + setup_standard_excludes(&dir); + strvec_push(&s, path.buf); + + parse_pathspec(&p, PATHSPEC_GLOB, 0, NULL, s.v); + fill_directory(&dir, r->index, &p); + + if (dir.nr) { + warning(_("directory '%s' contains untracked files," + " but is not in the sparse-checkout cone"), + item->string); + } else if (remove_dir_recursively(&path, 0)) { + /* + * Removal is "best effort". If something blocks + * the deletion, then continue with a warning. + */ + warning(_("failed to remove directory '%s'"), + item->string); + } + + dir_clear(&dir); + } + + string_list_clear(&sparse_dirs, 0); + strbuf_release(&path); + + if (was_full) + ensure_full_index(r->index); +} + static int update_working_directory(struct pattern_list *pl) { enum update_sparsity_result result; @@ -141,6 +233,8 @@ static int update_working_directory(struct pattern_list *pl) else rollback_lock_file(&lock_file); + clean_tracked_sparse_directories(r); + r->index->sparse_checkout_patterns = NULL; return result; } diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 38fc8340f5c..71236981e64 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -642,4 +642,63 @@ test_expect_success MINGW 'cone mode replaces backslashes with slashes' ' check_files repo/deep a deeper1 ' +test_expect_success 'cone mode clears ignored subdirectories' ' + rm repo/.git/info/sparse-checkout && + + git -C repo sparse-checkout init --cone && + git -C repo sparse-checkout set deep/deeper1 && + + cat >repo/.gitignore <<-\EOF && + obj/ + *.o + EOF + + git -C repo add .gitignore && + git -C repo commit -m ".gitignore" && + + mkdir -p repo/obj repo/folder1/obj repo/deep/deeper2/obj && + for file in folder1/obj/a obj/a folder1/file.o folder1.o \ + deep/deeper2/obj/a deep/deeper2/file.o file.o + do + echo ignored >repo/$file || return 1 + done && + + git -C repo status --porcelain=v2 >out && + test_must_be_empty out && + + git -C repo sparse-checkout reapply && + test_path_is_missing repo/folder1 && + test_path_is_missing repo/deep/deeper2 && + test_path_is_dir repo/obj && + test_path_is_file repo/file.o && + + git -C repo status --porcelain=v2 >out && + test_must_be_empty out && + + git -C repo sparse-checkout set deep/deeper2 && + test_path_is_missing repo/deep/deeper1 && + test_path_is_dir repo/deep/deeper2 && + test_path_is_dir repo/obj && + test_path_is_file repo/file.o && + + >repo/deep/deeper2/ignored.o && + >repo/deep/deeper2/untracked && + + # When an untracked file is in the way, all untracked files + # (even ignored files) are preserved. + git -C repo sparse-checkout set folder1 2>err && + grep "contains untracked files" err && + test_path_is_file repo/deep/deeper2/ignored.o && + test_path_is_file repo/deep/deeper2/untracked && + + # The rest of the cone matches expectation + test_path_is_missing repo/deep/deeper1 && + test_path_is_dir repo/obj && + test_path_is_file repo/file.o && + + git -C repo status --porcelain=v2 >out && + echo "? deep/deeper2/untracked" >expect && + test_cmp expect out +' + test_done From patchwork Tue Aug 24 21:51:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12455953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F14D5C4338F for ; Tue, 24 Aug 2021 21:52:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DDA346127B for ; Tue, 24 Aug 2021 21:52:05 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S236828AbhHXVwt (ORCPT ); Tue, 24 Aug 2021 17:52:49 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39174 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S236513AbhHXVwk (ORCPT ); Tue, 24 Aug 2021 17:52:40 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32C69C0612E7 for ; Tue, 24 Aug 2021 14:51:54 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id d22-20020a1c1d16000000b002e7777970f0so2848194wmd.3 for ; Tue, 24 Aug 2021 14:51:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uirt+/Fd9vyJ7GknotIoX/RPYOmGJD0KnjJarGrZ0yQ=; b=V9lhlbwBjJ/x2bBL7TW9vTa+7sWTlgz4oag0sDK1UEjrq9qn4BZ5d2tzdVntKaWAIJ X164Xq3jYrC0QN4eRX3h2zyxFm6ndxm3bRzE94HuK1vppmdRiXkT6H0A4vxqZ4gt1HY4 qKq/uugdzLC72MzRMo7yJvcnVQVT8NMOQ15bBPwrGIQZu21Wj1oaFDwH8/I/3Xvz7cVJ 6sluToZJ3JIZmVBBpaftuiPYaiUcToVxGqEbLYRRZBABhjZezIcc2C1NCbkiqeEy8Vdg R2eKs7oj22V7Oh93+jEnxn5kDlUAi69UPxzWkUy2Mz19NLEAlTwsUCtnBJGlW+uHvSJ/ qJ3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uirt+/Fd9vyJ7GknotIoX/RPYOmGJD0KnjJarGrZ0yQ=; b=BnnbgF391cfIZr/Mi4PFxD18Vi0QTI5p24NKWgWSHnT1luBpATaKusA+pG73Oe9h9M QTS6S0BTmV9O1qPl6iN9dBD8tHbKBzGk+Gg0/26P5AMg2ftzDhr1sIoaOH+X6yi5RQ0R eas2G0Dexemf8Ms8k3ijhAPf1Zrzwj7MRpWnBBOzQOQGTSvqb+KuLv2oq55GWWyB5cRE 6zhSfBUhOvgR+d9PoWXGRmLJaOeWcfb6f0XBliA7vmzgxry/tKopoP1XlQGcJuTvOGYA SaBKdycoD6dBkGFRUuX7C18H/8W0pibVfR/uKwSHI2lPlMovyODWUrCS5W9x+mS/SsqS nClw== X-Gm-Message-State: AOAM531jCxmRdLngK9KEm5/3c93hDi/wRRChL+CsC72XoOEUUyFd8X1I 3UbU58T4OgZDtPC6MTud+PU+JtY9yPA= X-Google-Smtp-Source: ABdhPJwR/2OCx5+XmycwRbTDKbn8xK2F9s7T7jNOIyCz5rCdQchvo0WGYAiqLEO+CO4eSy3K61c6mA== X-Received: by 2002:a05:600c:1911:: with SMTP id j17mr5966958wmq.164.1629841912813; Tue, 24 Aug 2021 14:51:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p12sm3098607wmq.44.2021.08.24.14.51.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Aug 2021 14:51:52 -0700 (PDT) Message-Id: <8d55a6ba2fdf64cee4eb51f3cb6f9808bd0b7505.1629841904.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 24 Aug 2021 21:51:44 +0000 Subject: [PATCH v4 10/10] sparse-checkout: add config to disable deleting dirs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Johannes Schindelin , Eric Sunshine , =?utf-8?b?UmVuw6k=?= Scharfe , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The clean_tracked_sparse_directories() method deletes the tracked directories that go out of scope when the sparse-checkout cone changes, at least in cone mode. This is new behavior, but is recommended based on our understanding of how users are interacting with the feature in most cases. It is possible that some users will object to the new behavior, so create a new configuration option 'index.deleteSparseDirectories' that can be set to 'false' to make clean_tracked_sparse_directories() do nothing. This will keep all untracked files in the working tree and cause performance problems with the sparse index, but those trade-offs are for the user to decide. Signed-off-by: Derrick Stolee --- Documentation/config/index.txt | 6 ++++++ builtin/sparse-checkout.c | 9 ++++++++- t/t1091-sparse-checkout-builtin.sh | 4 ++++ 3 files changed, 18 insertions(+), 1 deletion(-) diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt index 75f3a2d1054..c65da20a931 100644 --- a/Documentation/config/index.txt +++ b/Documentation/config/index.txt @@ -1,3 +1,9 @@ +index.deleteSparseDirectories:: + When enabled, the cone mode sparse-checkout feature will delete + directories that are outside of the sparse-checkout cone, unless + such a directory contains an untracked, non-ignored file. Defaults + to true. + index.recordEndOfIndexEntries:: Specifies whether the index file should include an "End Of Index Entry" section. This reduces index load time on multiprocessor diff --git a/builtin/sparse-checkout.c b/builtin/sparse-checkout.c index d0f5c4702be..33ec729d9de 100644 --- a/builtin/sparse-checkout.c +++ b/builtin/sparse-checkout.c @@ -102,7 +102,7 @@ static int sparse_checkout_list(int argc, const char **argv) static void clean_tracked_sparse_directories(struct repository *r) { - int i, was_full = 0; + int i, value, was_full = 0; struct strbuf path = STRBUF_INIT; size_t pathlen; struct string_list_item *item; @@ -118,6 +118,13 @@ static void clean_tracked_sparse_directories(struct repository *r) !r->index->sparse_checkout_patterns->use_cone_patterns) return; + /* + * Users can disable this behavior. + */ + if (!repo_config_get_bool(r, "index.deletesparsedirectories", &value) && + !value) + return; + /* * Use the sparse index as a data structure to assist finding * directories that are safe to delete. This conversion to a diff --git a/t/t1091-sparse-checkout-builtin.sh b/t/t1091-sparse-checkout-builtin.sh index 71236981e64..e0f31186d89 100755 --- a/t/t1091-sparse-checkout-builtin.sh +++ b/t/t1091-sparse-checkout-builtin.sh @@ -666,6 +666,10 @@ test_expect_success 'cone mode clears ignored subdirectories' ' git -C repo status --porcelain=v2 >out && test_must_be_empty out && + git -C repo -c index.deleteSparseDirectories=false sparse-checkout reapply && + test_path_is_dir repo/folder1 && + test_path_is_dir repo/deep/deeper2 && + git -C repo sparse-checkout reapply && test_path_is_missing repo/folder1 && test_path_is_missing repo/deep/deeper2 &&