From patchwork Sun Aug 7 04:13:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shaoxuan Yuan X-Patchwork-Id: 12938122 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5D5EC19F2A for ; Sun, 7 Aug 2022 04:14:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231771AbiHGEN6 (ORCPT ); Sun, 7 Aug 2022 00:13:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231655AbiHGEN5 (ORCPT ); Sun, 7 Aug 2022 00:13:57 -0400 Received: from mail-pg1-x52e.google.com (mail-pg1-x52e.google.com [IPv6:2607:f8b0:4864:20::52e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5D52BB482 for ; Sat, 6 Aug 2022 21:13:56 -0700 (PDT) Received: by mail-pg1-x52e.google.com with SMTP id f65so5878355pgc.12 for ; Sat, 06 Aug 2022 21:13:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1glDHHPs34NyLI2qDVY1GS9geIHtpVd4Lz4mGVfXKm0=; b=ecKlQmLnjENqQvWK5qe8g8ifwUT2lFKJ7j155JWTO9ULE8Mmq+z79jdS8gTDo5/Sc8 2D8/7PEQxph+KDGDNu/1C2jCl+7up22fjTonOPLYNwHaUhHMRlzyL8g61VA8MnbGpgTZ 4CeK+ZlOU40WLovxEaSpiQ3uFUB927S9bqROBsUnhDS+jmblEvD4dBqs0KkpDyez4jwq JA+GH1ApeUip/O49I6Ri1wfbAWJjkpi3ODHnCsGtpM1rPVq8rkdtobJJreY2NL4LCgRN 7kCkP4DMhTZRpjxOQoXpwv2YglNl8R5Q3EINbRyAgEutprDDNNrydQlEwlZkUHXYwE0z NWGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1glDHHPs34NyLI2qDVY1GS9geIHtpVd4Lz4mGVfXKm0=; b=QFSNS+qfHaP+O/s6HDMftrGyj6JVbIBBerdDL8ca87X2JzvfcdqThNAkzt5/rKDVMv KBxsT8Go1NBe9MaArih7Waj2ppZM8WtstvvsKbuzslZ/7ESBI3clqoimn2ls0gbmxASK zbEDf3YrUjVq3/Onytliglj7H7/MAk1BY7yccfq6vk7BJsw8DMDviHpMdd9GOTmvqLCv OoGpnqQjeY/dHLnDIaQVcrGpTx7gDgIr2MMXO3BW9fP0mtiIrfkd4swZYILZyXvWsgDr Wlx6XpTHnJQA9BrrqlyLsDv2L2M+Uns8p/NkaXDEYKsW88xKucGpK53SKxBVjmg9VN2G 5uNA== X-Gm-Message-State: ACgBeo1dQArIDawwXaOFlxWst2ER81SeqzypVD8fPOo/RLpxe1TL4+9j R5FwnzOdRlxewgPN9LArNbu4eUXVNJZ2Mw== X-Google-Smtp-Source: AA6agR6JA0xytcJlO8DRIQ5mswAjQj0yxE9Cd4bw53PQlMQASGeWxq6h8i9YEuKWhMlhJw8dzN2KCQ== X-Received: by 2002:a65:4388:0:b0:41b:c071:d21f with SMTP id m8-20020a654388000000b0041bc071d21fmr10937161pgp.335.1659845635718; Sat, 06 Aug 2022 21:13:55 -0700 (PDT) Received: from ffyuanda.localdomain ([113.65.209.111]) by smtp.gmail.com with ESMTPSA id v22-20020a17090a0e1600b001f333fab3d6sm8057027pje.18.2022.08.06.21.13.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Aug 2022 21:13:55 -0700 (PDT) From: Shaoxuan Yuan To: git@vger.kernel.org Cc: vdye@github.com, derrickstolee@github.com, Shaoxuan Yuan Subject: [PATCH v2 1/4] t1092: add tests for `git-rm` Date: Sun, 7 Aug 2022 12:13:32 +0800 Message-Id: <20220807041335.1790658-2-shaoxuan.yuan02@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> References: <20220803045118.1243087-1-shaoxuan.yuan02@gmail.com> <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add tests for `git-rm`, make sure it behaves as expected when is both inside or outside of sparse-checkout definition. Helped-by: Victoria Dye Helped-by: Derrick Stolee Signed-off-by: Shaoxuan Yuan --- t/t1092-sparse-checkout-compatibility.sh | 57 ++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 763c6cc684..c9300b77dd 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -1853,4 +1853,61 @@ test_expect_success 'mv directory from out-of-cone to in-cone' ' grep -e "H deep/0/1" actual ' +test_expect_success 'rm pathspec inside sparse definition' ' + init_repos && + + test_all_match git rm deep/a && + test_all_match git status --porcelain=v2 && + + # test wildcard + run_on_all git reset --hard && + test_all_match git rm deep/* && + test_all_match git status --porcelain=v2 && + + # test recursive rm + run_on_all git reset --hard && + test_all_match git rm -r deep && + test_all_match git status --porcelain=v2 +' + +test_expect_failure 'rm pathspec outside sparse definition' ' + init_repos && + + for file in folder1/a folder1/0/1 + do + test_sparse_match test_must_fail git rm $file && + test_sparse_match test_must_fail git rm --cached $file && + test_sparse_match git rm --sparse $file && + test_sparse_match git status --porcelain=v2 + done && + + cat >folder1-full <<-EOF && + rm ${SQ}folder1/0/0/0${SQ} + rm ${SQ}folder1/0/1${SQ} + rm ${SQ}folder1/a${SQ} + EOF + + cat >folder1-sparse <<-EOF && + rm ${SQ}folder1/${SQ} + EOF + + # test wildcard + run_on_sparse git reset --hard && + run_on_sparse git sparse-checkout reapply && + test_sparse_match test_must_fail git rm folder1/* && + run_on_sparse git rm --sparse folder1/* && + test_cmp folder1-full sparse-checkout-out && + test_cmp folder1-sparse sparse-index-out && + test_sparse_match git status --porcelain=v2 && + + # test recursive rm + run_on_sparse git reset --hard && + run_on_sparse git sparse-checkout reapply && + test_sparse_match test_must_fail git rm --sparse folder1 && + run_on_sparse git rm --sparse -r folder1 && + test_cmp folder1-full sparse-checkout-out && + test_cmp folder1-sparse sparse-index-out && + test_sparse_match git status --porcelain=v2 +' + test_done From patchwork Sun Aug 7 04:13:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shaoxuan Yuan X-Patchwork-Id: 12938123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12305C19F2A for ; Sun, 7 Aug 2022 04:14:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232255AbiHGEOL (ORCPT ); Sun, 7 Aug 2022 00:14:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45760 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231655AbiHGEN7 (ORCPT ); Sun, 7 Aug 2022 00:13:59 -0400 Received: from mail-pj1-x102e.google.com (mail-pj1-x102e.google.com [IPv6:2607:f8b0:4864:20::102e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1ECF4EE18 for ; Sat, 6 Aug 2022 21:13:58 -0700 (PDT) Received: by mail-pj1-x102e.google.com with SMTP id t2-20020a17090a4e4200b001f21572f3a4so6397070pjl.0 for ; Sat, 06 Aug 2022 21:13:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=M80OBkqBbPP5TxkPa4NVySG9d6BNEtfyUuRLa0Heys8=; b=pGzhxYGB0tvpPU9Aovb3XWECig6B5rva68U9Gqz9BS1DrUpS8E7MxGOAscKZ9xOTRg AYkoi/6c/BLIdoAcih4TGJqz6QemJzdUWloN+peDI95jDEAqmV06dgXkLFTDyK4kpVxD FhBWYTC4p+USFJ/q6mMce+ozmvkjA5rUzVbrbdVjF23JLOJwCi58LLPZ1az/+GFssqXD P4D6KbSu/Qrd1Skk1KesOgNEO4cqwPCpPMeZLMPHEtaslZXGZocJYzgrfWPctCquD2lu LYwltsZwUsXPCcL3zJ8gDf8KlHHbc6E/i8VLY9xxb3atU9fqbAR6MbpXGO2wTv5K4vWA qmLg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M80OBkqBbPP5TxkPa4NVySG9d6BNEtfyUuRLa0Heys8=; b=Pwi+wpP97Nct2Rm2TcELZq0HSXIj7HwHCSqUGwk4/ACtGSuJvdZILWB55epv9y8MLu hS6Ryanr2D+fQSn9NnP4t5j/XCEbkGAjfdmSvBv0Ql1It6mCSLobFbxrf8esVkf0s+im /Zp1tds8u6il8r2t4VPeQDepoeGPSe+15TgGszLLe0/sAIjc1FWePTlnlty514fKppeo YTNn6L9rXzl/Fv7A1Y2EbI/IymZOKuZwOuWI+taU2go9RdY6hBzGdUBtJcxEOacnfHo6 k0oqwzybmjVKb0wy4xL+vztVCQX7AdercnrmzWXMYk36G/Sa+XgVlb1t6QZqvA+sQWES 1HkA== X-Gm-Message-State: ACgBeo27UNEHZzdkHTyObYY8UZp39EZPy/6ffCTSnZzyCLXOEfR4pqlx NGPuwCfoX38kQxgb479RmN2ZwXx0W4cMyQ== X-Google-Smtp-Source: AA6agR4HCimskbT0xBv0jlejjtAlZYb7N+rp0tYwvtOEAVMpKaCMMFV2t0jxFACu5jt0RDzVhM51aQ== X-Received: by 2002:a17:903:120e:b0:16f:2e9:8c41 with SMTP id l14-20020a170903120e00b0016f02e98c41mr13560134plh.55.1659845637352; Sat, 06 Aug 2022 21:13:57 -0700 (PDT) Received: from ffyuanda.localdomain ([113.65.209.111]) by smtp.gmail.com with ESMTPSA id v22-20020a17090a0e1600b001f333fab3d6sm8057027pje.18.2022.08.06.21.13.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Aug 2022 21:13:57 -0700 (PDT) From: Shaoxuan Yuan To: git@vger.kernel.org Cc: vdye@github.com, derrickstolee@github.com, Shaoxuan Yuan Subject: [PATCH v2 2/4] pathspec.h: move pathspec_needs_expanded_index() from reset.c to here Date: Sun, 7 Aug 2022 12:13:33 +0800 Message-Id: <20220807041335.1790658-3-shaoxuan.yuan02@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> References: <20220803045118.1243087-1-shaoxuan.yuan02@gmail.com> <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Method pathspec_needs_expanded_index() in reset.c from 4d1cfc1351 (reset: make --mixed sparse-aware, 2021-11-29) is reusable when we need to verify if the index needs to be expanded when the command is utilizing a pathspec rather than a literal path. Move it to pathspec.h for reusability. Add a few items to the function so it can better serve its purpose as a standalone public function: * Add a check in front so if the index is not sparse, return early since no expansion is needed. * It now takes an arbitrary 'struct index_state' pointer instead of using `the_index` and `active_cache`. * Add documentation to the function. Helped-by: Victoria Dye Helped-by: Derrick Stolee Signed-off-by: Shaoxuan Yuan --- builtin/reset.c | 84 +--------------------------------------------- pathspec.c | 89 +++++++++++++++++++++++++++++++++++++++++++++++++ pathspec.h | 12 +++++++ 3 files changed, 102 insertions(+), 83 deletions(-) diff --git a/builtin/reset.c b/builtin/reset.c index 344fff8f3a..fdce6f8c85 100644 --- a/builtin/reset.c +++ b/builtin/reset.c @@ -174,88 +174,6 @@ static void update_index_from_diff(struct diff_queue_struct *q, } } -static int pathspec_needs_expanded_index(const struct pathspec *pathspec) -{ - unsigned int i, pos; - int res = 0; - char *skip_worktree_seen = NULL; - - /* - * When using a magic pathspec, assume for the sake of simplicity that - * the index needs to be expanded to match all matchable files. - */ - if (pathspec->magic) - return 1; - - for (i = 0; i < pathspec->nr; i++) { - struct pathspec_item item = pathspec->items[i]; - - /* - * If the pathspec item has a wildcard, the index should be expanded - * if the pathspec has the possibility of matching a subset of entries inside - * of a sparse directory (but not the entire directory). - * - * If the pathspec item is a literal path, the index only needs to be expanded - * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't - * expand for in-cone files) and b) it doesn't match any sparse directories - * (since we can reset whole sparse directories without expanding them). - */ - if (item.nowildcard_len < item.len) { - /* - * Special case: if the pattern is a path inside the cone - * followed by only wildcards, the pattern cannot match - * partial sparse directories, so we know we don't need to - * expand the index. - * - * Examples: - * - in-cone/foo***: doesn't need expanded index - * - not-in-cone/bar*: may need expanded index - * - **.c: may need expanded index - */ - if (strspn(item.original + item.nowildcard_len, "*") == item.len - item.nowildcard_len && - path_in_cone_mode_sparse_checkout(item.original, &the_index)) - continue; - - for (pos = 0; pos < active_nr; pos++) { - struct cache_entry *ce = active_cache[pos]; - - if (!S_ISSPARSEDIR(ce->ce_mode)) - continue; - - /* - * If the pre-wildcard length is longer than the sparse - * directory name and the sparse directory is the first - * component of the pathspec, need to expand the index. - */ - if (item.nowildcard_len > ce_namelen(ce) && - !strncmp(item.original, ce->name, ce_namelen(ce))) { - res = 1; - break; - } - - /* - * If the pre-wildcard length is shorter than the sparse - * directory and the pathspec does not match the whole - * directory, need to expand the index. - */ - if (!strncmp(item.original, ce->name, item.nowildcard_len) && - wildmatch(item.original, ce->name, 0)) { - res = 1; - break; - } - } - } else if (!path_in_cone_mode_sparse_checkout(item.original, &the_index) && - !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) - res = 1; - - if (res > 0) - break; - } - - free(skip_worktree_seen); - return res; -} - static int read_from_tree(const struct pathspec *pathspec, struct object_id *tree_oid, int intent_to_add) @@ -273,7 +191,7 @@ static int read_from_tree(const struct pathspec *pathspec, opt.change = diff_change; opt.add_remove = diff_addremove; - if (pathspec->nr && the_index.sparse_index && pathspec_needs_expanded_index(pathspec)) + if (pathspec->nr && pathspec_needs_expanded_index(&the_index, pathspec)) ensure_full_index(&the_index); if (do_diff_cache(tree_oid, &opt)) diff --git a/pathspec.c b/pathspec.c index 84ad9c73cf..46e77a85fe 100644 --- a/pathspec.c +++ b/pathspec.c @@ -759,3 +759,92 @@ int match_pathspec_attrs(struct index_state *istate, return 1; } + +int pathspec_needs_expanded_index(struct index_state *istate, + const struct pathspec *pathspec) +{ + unsigned int i, pos; + int res = 0; + char *skip_worktree_seen = NULL; + + /* + * If index is not sparse, no index expansion is needed. + */ + if (!istate->sparse_index) + return 0; + + /* + * When using a magic pathspec, assume for the sake of simplicity that + * the index needs to be expanded to match all matchable files. + */ + if (pathspec->magic) + return 1; + + for (i = 0; i < pathspec->nr; i++) { + struct pathspec_item item = pathspec->items[i]; + + /* + * If the pathspec item has a wildcard, the index should be expanded + * if the pathspec has the possibility of matching a subset of entries inside + * of a sparse directory (but not the entire directory). + * + * If the pathspec item is a literal path, the index only needs to be expanded + * if a) the pathspec isn't in the sparse checkout cone (to make sure we don't + * expand for in-cone files) and b) it doesn't match any sparse directories + * (since we can reset whole sparse directories without expanding them). + */ + if (item.nowildcard_len < item.len) { + /* + * Special case: if the pattern is a path inside the cone + * followed by only wildcards, the pattern cannot match + * partial sparse directories, so we know we don't need to + * expand the index. + * + * Examples: + * - in-cone/foo***: doesn't need expanded index + * - not-in-cone/bar*: may need expanded index + * - **.c: may need expanded index + */ + if (strspn(item.original + item.nowildcard_len, "*") == item.len - item.nowildcard_len && + path_in_cone_mode_sparse_checkout(item.original, istate)) + continue; + + for (pos = 0; pos < istate->cache_nr; pos++) { + struct cache_entry *ce = istate->cache[pos]; + + if (!S_ISSPARSEDIR(ce->ce_mode)) + continue; + + /* + * If the pre-wildcard length is longer than the sparse + * directory name and the sparse directory is the first + * component of the pathspec, need to expand the index. + */ + if (item.nowildcard_len > ce_namelen(ce) && + !strncmp(item.original, ce->name, ce_namelen(ce))) { + res = 1; + break; + } + + /* + * If the pre-wildcard length is shorter than the sparse + * directory and the pathspec does not match the whole + * directory, need to expand the index. + */ + if (!strncmp(item.original, ce->name, item.nowildcard_len) && + wildmatch(item.original, ce->name, 0)) { + res = 1; + break; + } + } + } else if (!path_in_cone_mode_sparse_checkout(item.original, istate) && + !matches_skip_worktree(pathspec, i, &skip_worktree_seen)) + res = 1; + + if (res > 0) + break; + } + + free(skip_worktree_seen); + return res; +} diff --git a/pathspec.h b/pathspec.h index 402ebb8080..41f6adfbb4 100644 --- a/pathspec.h +++ b/pathspec.h @@ -171,4 +171,16 @@ int match_pathspec_attrs(struct index_state *istate, const char *name, int namelen, const struct pathspec_item *item); +/* + * Determine whether a pathspec will match only entire index entries (non-sparse + * files and/or entire sparse directories). If the pathspec has the potential to + * match partial contents of a sparse directory, return 1 to indicate the index + * should be expanded to match the appropriate index entries. + * + * For the sake of simplicity, always return 1 if using a more complex "magic" + * pathspec. + */ +int pathspec_needs_expanded_index(struct index_state *istate, + const struct pathspec *pathspec); + #endif /* PATHSPEC_H */ From patchwork Sun Aug 7 04:13:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shaoxuan Yuan X-Patchwork-Id: 12938124 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E8EAC25B06 for ; Sun, 7 Aug 2022 04:14:14 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231888AbiHGEON (ORCPT ); Sun, 7 Aug 2022 00:14:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45830 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231944AbiHGEOA (ORCPT ); Sun, 7 Aug 2022 00:14:00 -0400 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DC4C310FCB for ; Sat, 6 Aug 2022 21:13:59 -0700 (PDT) Received: by mail-pl1-x62d.google.com with SMTP id iw1so5893243plb.6 for ; Sat, 06 Aug 2022 21:13:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=w14YtX5hbdcv5nO+DH51KgoD08YYAKivWfr4n0Nepjk=; b=gKButyYxX07QfbvAADKmsSh8wjmd/sgNWtmgo2HB3vYfYb6T0srScKc/dMWTIFY/5e Tk3MMDRSKNX0P64IkBU533RXjKcUNWZXOqjQCecBGKzc4jy9CYfrwA7dBd95jd1db14e Fn+XpFBvrLKpuCssdH2JVDds+FAA1fRouIeQJ4mIiO8YlEVnCbxRuUFPB8bREsegWe6N 3/VV6lkPb0h9qoqEKDuVWFn7nr/YxUGCpzqn9MCMq4YRTzfJGHau6GFngMsMTFqmOA9C 9pFtITKdmSUTJU4gI7kzuGfbtEnsdqiyCpciPsW/yOEqBJKgC5eR7lkyvNU/kg0DLwpC k3PA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=w14YtX5hbdcv5nO+DH51KgoD08YYAKivWfr4n0Nepjk=; b=QhZYypwpDIhB12DE24KbDGbdtXmV0/+wAfABZNj6yDJV+M8XjQprSTGuGGq0WShNhb 97MRqn8AhQfR0Xds96C9m5LLn6MpOGhw5KbbJYtgIKaSpjwwGmGPyaYLwEcTXkV/pB3k OfdrEshlZ0faSZgCzbRMMSIAPgP9CeKCzaAodEo/6ZfscCZN1aX1+zql69sFmbeXDQxK 5NvVEvCxxLmC0OF3/LaSYzrGbv8jn78/aFMr2jzYX4KFT8CbTwIN//EP9xc9UFvKzXhq zsdDXXAlPkId2c2UMw29CYvJJwtItomswL7qDmGPFRazD7Ub5ueOOP/OKUB+knyaSsHT Otgg== X-Gm-Message-State: ACgBeo01fmzlc5A6t6MTVQtZ9zsfD2bXRo6P1WSEaEWWbei1jgE8jwvK K6mBbOBWBbc43uYtNVT+Z9O6PnlzEUtPvA== X-Google-Smtp-Source: AA6agR61Bkhl0jpO/aCgWe4tGmTZoAB6YFkQHoCcDgAeQpVFu8l7nEtNOaxjPtr+Be+NDIVNQf9L6A== X-Received: by 2002:a17:90b:1c0d:b0:1f5:4ddf:1607 with SMTP id oc13-20020a17090b1c0d00b001f54ddf1607mr20879735pjb.90.1659845639120; Sat, 06 Aug 2022 21:13:59 -0700 (PDT) Received: from ffyuanda.localdomain ([113.65.209.111]) by smtp.gmail.com with ESMTPSA id v22-20020a17090a0e1600b001f333fab3d6sm8057027pje.18.2022.08.06.21.13.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Aug 2022 21:13:58 -0700 (PDT) From: Shaoxuan Yuan To: git@vger.kernel.org Cc: vdye@github.com, derrickstolee@github.com, Shaoxuan Yuan Subject: [PATCH v2 3/4] rm: expand the index only when necessary Date: Sun, 7 Aug 2022 12:13:34 +0800 Message-Id: <20220807041335.1790658-4-shaoxuan.yuan02@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> References: <20220803045118.1243087-1-shaoxuan.yuan02@gmail.com> <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove the `ensure_full_index()` method so `git-rm` does not always expand the index when the expansion is unnecessary, i.e. when does not have any possibilities to match anything outside of sparse-checkout definition. Expand the index when the needs an expanded index, i.e. the contains wildcard that may need a full-index or the is simply outside of sparse-checkout definition. Notice that the test 'rm pathspec expands index when necessary' in t1092 *is* testing this code change behavior, though it will be marked as 'test_expect_success' only in the next patch, where we officially mark `command_requires_full_index = 0`, so the index does not expand unless we tell it to do so. Notice that because we also want `ensure_full_index` to record the stdout and stderr from Git command, a corresponding modification is also included in this patch. The reason we want the "sparse-index-out" and "sparse-index-err", is that we need to make sure there is no error from Git command itself, so we can rely on the `test_region` result and determine if the index is expanded or not. Helped-by: Victoria Dye Helped-by: Derrick Stolee Signed-off-by: Shaoxuan Yuan --- builtin/rm.c | 5 +++-- t/t1092-sparse-checkout-compatibility.sh | 27 ++++++++++++++++++++++-- 2 files changed, 28 insertions(+), 4 deletions(-) diff --git a/builtin/rm.c b/builtin/rm.c index 84a935a16e..58ed924f0d 100644 --- a/builtin/rm.c +++ b/builtin/rm.c @@ -296,8 +296,9 @@ int cmd_rm(int argc, const char **argv, const char *prefix) seen = xcalloc(pathspec.nr, 1); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(&the_index); + if (pathspec_needs_expanded_index(&the_index, &pathspec)) + ensure_full_index(&the_index); + for (i = 0; i < active_nr; i++) { const struct cache_entry *ce = active_cache[i]; diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index c9300b77dd..94464cf911 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -1340,10 +1340,14 @@ ensure_not_expanded () { shift && test_must_fail env \ GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \ - git -C sparse-index "$@" || return 1 + git -C sparse-index "$@" \ + >sparse-index-out \ + 2>sparse-index-error || return 1 else GIT_TRACE2_EVENT="$(pwd)/trace2.txt" \ - git -C sparse-index "$@" || return 1 + git -C sparse-index "$@" \ + >sparse-index-out \ + 2>sparse-index-error || return 1 fi && test_region ! index ensure_full_index trace2.txt } @@ -1910,4 +1914,23 @@ test_expect_failure 'rm pathspec outside sparse definition' ' test_sparse_match git status --porcelain=v2 ' +test_expect_failure 'rm pathspec expands index when necessary' ' + init_repos && + + # in-cone pathspec (do not expand) + ensure_not_expanded rm "deep/deep*" && + test_must_be_empty sparse-index-err && + + # out-of-cone pathspec (expand) + ! ensure_not_expanded rm --sparse "folder1/a*" && + test_must_be_empty sparse-index-err && + + # pathspec that should expand index + ! ensure_not_expanded rm "*/a" && + test_must_be_empty sparse-index-err && + + ! ensure_not_expanded rm "**a" && + test_must_be_empty sparse-index-err +' + test_done From patchwork Sun Aug 7 04:13:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Shaoxuan Yuan X-Patchwork-Id: 12938125 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88651C19F2A for ; Sun, 7 Aug 2022 04:14:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232129AbiHGEOO (ORCPT ); Sun, 7 Aug 2022 00:14:14 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:45988 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232010AbiHGEOJ (ORCPT ); Sun, 7 Aug 2022 00:14:09 -0400 Received: from mail-pf1-x434.google.com (mail-pf1-x434.google.com [IPv6:2607:f8b0:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F77D10FC1 for ; Sat, 6 Aug 2022 21:14:01 -0700 (PDT) Received: by mail-pf1-x434.google.com with SMTP id f192so5519441pfa.9 for ; Sat, 06 Aug 2022 21:14:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QwQSIqSJQY91YX7HKnfyjKUWJ4xfGmIOD5cfPv6OaBU=; b=TK9h8Y5ZI+zTznl/mTDDgLCTcNpaE1fz28mkABzLyP0Tdzcf7lAyTK5eIrS/YKFLq8 cTiDSTR9qrSpgknh6QlPzHr9CBMW+negkQNw+p9HcDD1urzwAj9vg6ryMR7OVZgZmnza ipeVw37eri494Ps93oyNHdCBe28CdGpK3D/1ggp2ijQ0c2tejxoyqNVK26k9dIfVmCa8 1sKhJx3S7kSyf0bR/dtQR/kEhylfiGm/etzEmg8wyu8scOmqd1Yu1NXYps44HvqkqqvA QQKTd+EHof41R3tCPVOvRXphZl8rdnTcbk9W823L3VLpSgH9sYkhN8DQTNcelyLXDtmE 3Y4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QwQSIqSJQY91YX7HKnfyjKUWJ4xfGmIOD5cfPv6OaBU=; b=iBwgc/Y2+pBJX7VV2/aZb5Acg7A+v2H1C1PVc9e9ro1tMkjCkl5Fx72GPtKOE0AyQI X3E7oIKJlvEByFaRoFncACjVVcT44P3+9VubA2u2TYhQLPAl3gjtwCkyW7Bm3m5Ipq2T juAS4ej1Qp5cxUlFws5P5NxAG+M3CLr/wMHsJNiD7UZWS49uabUs9yvexDzOMmTib5mk czzVqCQWc7xkXq+v+gtcMoW66w7AQoXZmBdcrXktNFGRvB1TqQpk5aNGG3VMdiB3cfJW Z5nq5eDb5xchs2QfQtGU3MF9kCt7qxZQOXx9/1gwB0pJLC261DrKNMEsXG9xgWpsbPRf Uugg== X-Gm-Message-State: ACgBeo1AzNpjgXG5B1MhUhcveTKRQtnicZrb53ELvOd4XYJN258kNhT9 L61tbWOml6AsezttjKsP7aNOzznZa/Aihg== X-Google-Smtp-Source: AA6agR66dyxuxiOgzSDKMLzhRbQ6zXEgspsAIcUL/c+DICvO4X+4wyTqbDgoOK9qpJnietMrlF91eQ== X-Received: by 2002:a05:6a00:1d9e:b0:52d:aa13:67fc with SMTP id z30-20020a056a001d9e00b0052daa1367fcmr13374786pfw.73.1659845640789; Sat, 06 Aug 2022 21:14:00 -0700 (PDT) Received: from ffyuanda.localdomain ([113.65.209.111]) by smtp.gmail.com with ESMTPSA id v22-20020a17090a0e1600b001f333fab3d6sm8057027pje.18.2022.08.06.21.13.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 06 Aug 2022 21:14:00 -0700 (PDT) From: Shaoxuan Yuan To: git@vger.kernel.org Cc: vdye@github.com, derrickstolee@github.com, Shaoxuan Yuan Subject: [PATCH v2 4/4] rm: integrate with sparse-index Date: Sun, 7 Aug 2022 12:13:35 +0800 Message-Id: <20220807041335.1790658-5-shaoxuan.yuan02@gmail.com> X-Mailer: git-send-email 2.37.0 In-Reply-To: <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> References: <20220803045118.1243087-1-shaoxuan.yuan02@gmail.com> <20220807041335.1790658-1-shaoxuan.yuan02@gmail.com> MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Enable the sparse index within the `git-rm` command. The `p2000` tests demonstrate a ~92% execution time reduction for 'git rm' using a sparse index. Test HEAD~1 HEAD -------------------------------------------------------------------------- 2000.74: git rm ... (full-v3) 0.41(0.37+0.05) 0.43(0.36+0.07) +4.9% 2000.75: git rm ... (full-v4) 0.38(0.34+0.05) 0.39(0.35+0.05) +2.6% 2000.76: git rm ... (sparse-v3) 0.57(0.56+0.01) 0.05(0.05+0.00) -91.2% 2000.77: git rm ... (sparse-v4) 0.57(0.55+0.02) 0.03(0.03+0.00) -94.7% ---- Also, normalize a behavioral difference of `git-rm` under sparse-index. See related discussion [1]. `git-rm` a sparse-directory entry within a sparse-index enabled repo behaves differently from a sparse directory within a sparse-checkout enabled repo. For example, in a sparse-index repo, where 'folder1' is a sparse-directory entry, `git rm -r --sparse folder1` provides this: rm 'folder1/' Whereas in a sparse-checkout repo *without* sparse-index, doing so provides this: rm 'folder1/0/0/0' rm 'folder1/0/1' rm 'folder1/a' Because `git rm` a sparse-directory entry does not need to expand the index, therefore we should accept the current behavior, which is faster than "expand the sparse-directory entry to match the sparse-checkout situation". Modify a previous test so such difference is not considered as an error. [1] https://github.com/ffyuanda/git/pull/6#discussion_r934861398 Helped-by: Victoria Dye Helped-by: Derrick Stolee Signed-off-by: Shaoxuan Yuan --- builtin/rm.c | 2 ++ t/perf/p2000-sparse-operations.sh | 1 + t/t1092-sparse-checkout-compatibility.sh | 20 +++++++++++++++++--- 3 files changed, 20 insertions(+), 3 deletions(-) diff --git a/builtin/rm.c b/builtin/rm.c index 58ed924f0d..b6ba859fe4 100644 --- a/builtin/rm.c +++ b/builtin/rm.c @@ -287,6 +287,8 @@ int cmd_rm(int argc, const char **argv, const char *prefix) if (!index_only) setup_work_tree(); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; hold_locked_index(&lock_file, LOCK_DIE_ON_ERROR); if (read_cache() < 0) diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh index c181110a43..fce8151d41 100755 --- a/t/perf/p2000-sparse-operations.sh +++ b/t/perf/p2000-sparse-operations.sh @@ -123,5 +123,6 @@ test_perf_on_all git blame $SPARSE_CONE/f3/a test_perf_on_all git read-tree -mu HEAD test_perf_on_all git checkout-index -f --all test_perf_on_all git update-index --add --remove $SPARSE_CONE/a +test_perf_on_all "git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a" test_done diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 94464cf911..68ded9063b 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -912,7 +912,7 @@ test_expect_success 'read-tree --prefix' ' test_all_match git read-tree --prefix=deep/deeper1/deepest -u deepest && test_all_match git status --porcelain=v2 && - test_all_match git rm -rf --sparse folder1/ && + run_on_all git rm -rf --sparse folder1/ && test_all_match git read-tree --prefix=folder1/ -u update-folder1 && test_all_match git status --porcelain=v2 && @@ -1874,7 +1874,7 @@ test_expect_success 'rm pathspec inside sparse definition' ' test_all_match git status --porcelain=v2 ' -test_expect_failure 'rm pathspec outside sparse definition' ' +test_expect_success 'rm pathspec outside sparse definition' ' init_repos && for file in folder1/a folder1/0/1 @@ -1914,7 +1914,7 @@ test_expect_failure 'rm pathspec outside sparse definition' ' test_sparse_match git status --porcelain=v2 ' -test_expect_failure 'rm pathspec expands index when necessary' ' +test_expect_success 'rm pathspec expands index when necessary' ' init_repos && # in-cone pathspec (do not expand) @@ -1933,4 +1933,18 @@ test_expect_failure 'rm pathspec expands index when necessary' ' test_must_be_empty sparse-index-err ' +test_expect_success 'sparse index is not expanded: rm' ' + init_repos && + + ensure_not_expanded rm deep/a && + + # test in-cone wildcard + git -C sparse-index reset --hard && + ensure_not_expanded rm deep/* && + + # test recursive rm + git -C sparse-index reset --hard && + ensure_not_expanded rm -r deep +' + test_done