From patchwork Mon Apr 5 13:04:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12182917 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 807C6C433ED for ; Mon, 5 Apr 2021 13:04:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5A941611BE for ; Mon, 5 Apr 2021 13:04:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240748AbhDENE3 (ORCPT ); Mon, 5 Apr 2021 09:04:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229681AbhDENE0 (ORCPT ); Mon, 5 Apr 2021 09:04:26 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4098DC061788 for ; Mon, 5 Apr 2021 06:04:20 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id i18so7412722wrm.5 for ; Mon, 05 Apr 2021 06:04:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RxYhwUVO73QD53gXj34Ia2LkRQxnBglVimbVxWy9/Ic=; b=WnRZkYf/pXmiStz/coRa+0Jnq814xR3uU5OKWhqM7osohLVB0awP1gYNGMerw3hZ7x aK6xJMBAtA3F7FSNMtrJSbOhk3eTmrrjbdpxrz9Qyv6Y1xhnHdvBiejAFwCBHoS5+POf zKgHQEo6Ll0/rZgaM8jGxdaYS7M1qJT6P9aqwrHE7Z/uFXDV0MUNkU7LVIEWufO+x7+w 3b1SV4gGAKK6lymnbLS0L/RUW20QTsix7k8LBltCH3qgjJOCW2Ayr0Br91ybmwiogmip uPeB5aMCYpYF9+gKvmsKzLk6s6kAY6GkdJXItBrpEwkM2rFcDSbWpH5TYRnik0tahZ+j XcDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RxYhwUVO73QD53gXj34Ia2LkRQxnBglVimbVxWy9/Ic=; b=RAPAg5cywsSMMsoJiFZPqFpdyT9GS2Xof7CH1xR/EcijI43uxRxm5XPozJXyTzMrrP 0Q653guDac93IL8CATUvFXA+OAfw0cBA4FKyGO7qnA6rekj1einVg23wFPuDJkGshMTW ZNW2pHpumyDzhG3UUz4E3Txvk6xsM68mqmY48iO/xX7IwdeMp73zFywbBTwcls9p52fl Y/Htwd4zRONr3IAt7WNEQwygAzaGUYHx8FJGkC3vMIzEVKqKUWQMLxAnfzd2USgFZEIy 2NXOuEi5bvfBzkjgWAuPgnQT/lGoTKAXdK44DxoRyfgHcQMSGo1yM6lyNQDE3+9loCgK id1g== X-Gm-Message-State: AOAM533/FQj/SG6GTft639Mh89lVvD78DoiIIAUNRslwgM2MGp9MWIU4 bW9NolXTgT5U2DnfwqQ683e5RUECsi0= X-Google-Smtp-Source: ABdhPJxZ5VbLUdRe8JvC414Rk8Iu+rxgyp0+MrWVjZB/X+Z4Jf7CFR09YAWsEyHrrsryZXX3yjZMRg== X-Received: by 2002:adf:ee0a:: with SMTP id y10mr27968796wrn.177.1617627858757; Mon, 05 Apr 2021 06:04:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y1sm23320033wmq.29.2021.04.05.06.04.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Apr 2021 06:04:18 -0700 (PDT) Message-Id: <3a94ff80657cd8e0142c1007be4f183c6a18587e.1617627856.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Apr 2021 13:04:11 +0000 Subject: [PATCH 1/5] maintenance: simplify prefetch logic Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: tom.saeger@oracle.com, gitster@pobox.com, sunshine@sunshineco.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The previous logic filled a string list with the names of each remote, but instead we could simply run the appropriate 'git fetch' data directly in the remote iterator. Do this for reduced code size, but also becuase it sets up an upcoming change to use the remote's refspec. This data is accessible from the 'struct remote' data that is now accessible in fetch_remote(). Signed-off-by: Derrick Stolee --- builtin/gc.c | 33 ++++++++------------------------- 1 file changed, 8 insertions(+), 25 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index ef7226d7bca4..fa8128de9ae1 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -873,55 +873,38 @@ static int maintenance_task_commit_graph(struct maintenance_run_opts *opts) return 0; } -static int fetch_remote(const char *remote, struct maintenance_run_opts *opts) +static int fetch_remote(struct remote *remote, void *cbdata) { + struct maintenance_run_opts *opts = cbdata; struct child_process child = CHILD_PROCESS_INIT; child.git_cmd = 1; - strvec_pushl(&child.args, "fetch", remote, "--prune", "--no-tags", + strvec_pushl(&child.args, "fetch", remote->name, "--prune", "--no-tags", "--no-write-fetch-head", "--recurse-submodules=no", "--refmap=", NULL); if (opts->quiet) strvec_push(&child.args, "--quiet"); - strvec_pushf(&child.args, "+refs/heads/*:refs/prefetch/%s/*", remote); + strvec_pushf(&child.args, "+refs/heads/*:refs/prefetch/%s/*", remote->name); return !!run_command(&child); } -static int append_remote(struct remote *remote, void *cbdata) -{ - struct string_list *remotes = (struct string_list *)cbdata; - - string_list_append(remotes, remote->name); - return 0; -} - static int maintenance_task_prefetch(struct maintenance_run_opts *opts) { - int result = 0; - struct string_list_item *item; - struct string_list remotes = STRING_LIST_INIT_DUP; - git_config_set_multivar_gently("log.excludedecoration", "refs/prefetch/", "refs/prefetch/", CONFIG_FLAGS_FIXED_VALUE | CONFIG_FLAGS_MULTI_REPLACE); - if (for_each_remote(append_remote, &remotes)) { - error(_("failed to fill remotes")); - result = 1; - goto cleanup; + if (for_each_remote(fetch_remote, opts)) { + error(_("failed to prefetch remotes")); + return 1; } - for_each_string_list_item(item, &remotes) - result |= fetch_remote(item->string, opts); - -cleanup: - string_list_clear(&remotes, 0); - return result; + return 0; } static int maintenance_task_gc(struct maintenance_run_opts *opts) From patchwork Mon Apr 5 13:04:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12182913 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FE54C433ED for ; Mon, 5 Apr 2021 13:04:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 15C83611BE for ; Mon, 5 Apr 2021 13:04:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S237078AbhDENE2 (ORCPT ); Mon, 5 Apr 2021 09:04:28 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235661AbhDENE0 (ORCPT ); Mon, 5 Apr 2021 09:04:26 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7F46CC06178C for ; Mon, 5 Apr 2021 06:04:20 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id j18so10861112wra.2 for ; Mon, 05 Apr 2021 06:04:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=wuRuYYWVThwVuKMtFPkVF2Ilw+IL9JMWGp2mFqIegZM=; b=eb95RImDZQuiUourk2hqgPzAJwJYFF25zcRR+SfynHpoj6eNbR7hz5/tjNWOpgcicJ nVLOu6oJOoiRiM+9zZ0B2a9rSkCuPL3uQBAMUyBqnvOdUu5XmkzDGw5cW9iIHC70vvmM V4+nuj4roz52M5G1rtxIMeFCuTQrugyrqw0V+lRHzgciEzLCC+25OYIz9yQThqp3NppH wRxudNxc4Zk8vO/Lcge2dW4l0GkEvs9ZwvSQFXlC0CwPpnKoaqz5iHm1E5psh5xT589p dThwJAz8kDBfT/Z72SUyTTXa6sxUWjdGtrOovoJpeoXotZexyin/7NC4oACQnaVXV8pP qtGA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=wuRuYYWVThwVuKMtFPkVF2Ilw+IL9JMWGp2mFqIegZM=; b=f+Tu/dxFv09wqJuiNgmFtyoCD5D97ZXPDc64t6+3SanYbJ8lfVyIoHtNgsO0ATa7oR Yoq+i6YwskF0GMLqVopukZEsC1GAKRhAKsxnNIT9kLW8WXtFd72ft3fh3QMN1kcvyXuR mqbvUjfPTfE0uqETW2hFCY/WVMgJTcpLq82BpGgCafABSz9vQFF5B94JYS68ePDSEK87 IyM5v/I+c6HIkzOphqZkqYA3MXRfRdvOkH4cmhIHIj7q+RyTkwvFowwIcQ+p6HF51vkk v458RBlC3F2Lsy7CWh1hWbUKH4CYotu64gKxvryFgdPFSDvkuDJKSzL56oM4Z4NjrGm4 kNcQ== X-Gm-Message-State: AOAM532Ptuo1covV0nkRaBHOhJHYo2GVvE55heueaF1Ms7Jk66ePKqsf ZFnnv5GfjGnPNnnT9ogSn9oaPfXUwoA= X-Google-Smtp-Source: ABdhPJymM5Y88g6uGWyWEadNqvtV2P8F0xEBMDvluKKBHjlGffPuDZaTPULdg9ASPI3GfCIlugbhSw== X-Received: by 2002:adf:e68e:: with SMTP id r14mr28700027wrm.273.1617627859304; Mon, 05 Apr 2021 06:04:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q10sm12125205wrx.35.2021.04.05.06.04.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Apr 2021 06:04:19 -0700 (PDT) Message-Id: <2b74889c2a323f03be477ffdf9ff388405779c3b.1617627856.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Apr 2021 13:04:12 +0000 Subject: [PATCH 2/5] test-lib: use exact match for test_subcommand Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: tom.saeger@oracle.com, gitster@pobox.com, sunshine@sunshineco.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The use of 'grep' inside test_subcommand uses general patterns, leading to sometimes needing escape characters to avoid incorrect matches. Further, some platforms interpret different glob characters differently. Use 'grep -F' to use an exact match. This requires removing escape characters from existing callers. Luckily, this is only one test that expects refspecs as part of the subcommand. Reported-by: Eric Sunshine Signed-off-by: Derrick Stolee --- t/t7900-maintenance.sh | 4 ++-- t/test-lib-functions.sh | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 2412d8c5c006..fc2315edec11 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -142,8 +142,8 @@ test_expect_success 'prefetch multiple remotes' ' test_commit -C clone2 two && GIT_TRACE2_EVENT="$(pwd)/run-prefetch.txt" git maintenance run --task=prefetch 2>/dev/null && fetchargs="--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet" && - test_subcommand git fetch remote1 $fetchargs +refs/heads/\\*:refs/prefetch/remote1/\\* X-Patchwork-Id: 12182919 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA834C43461 for ; Mon, 5 Apr 2021 13:04:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8985B613AD for ; Mon, 5 Apr 2021 13:04:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240923AbhDENEd (ORCPT ); Mon, 5 Apr 2021 09:04:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50220 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235731AbhDENE2 (ORCPT ); Mon, 5 Apr 2021 09:04:28 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07010C061793 for ; Mon, 5 Apr 2021 06:04:21 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id a12so1092155wrq.13 for ; Mon, 05 Apr 2021 06:04:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=P0CSrGG0xl0UZrDRJ2dSC6xgIbyMmNQRJEdbeKxUd2w=; b=eCpwGhxYRWuIhx7A2B+7uD7mIsIETD41zMxklbrVAbKh3Q+lK0j2kClOng74tmjZpr d21Fh51boX10pnzER/rvb6OJvVBC1whVAsQ2viMGMSNv9f530bYLGA11zfHbsjyclZOx LKO8aSBgHKf0ibrVYMl8kkeN6evb3Iv07M3aHdWQ4p4aodYNCy9xt5bFuYu+Ez2e193u wIe3Oka8sktClO9SD9cG/O3cXYKGbqJCIHmJBddG5RI/twKaIRw6w2XXk35XnCBDeGYb KNwJf31Nh76kf8gIBUgii5qrsX2168fncJ4Z3GU2o7mv7+X9Z+iegCmW+JKxA+WYClN3 /Waw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=P0CSrGG0xl0UZrDRJ2dSC6xgIbyMmNQRJEdbeKxUd2w=; b=TV0ghSgjrmvgQk7PrSo/f1rM/PIKMNAXaV/1Me0g+9Lri3ixXX6+1LABGDyQEmzerP JbLjUlNVP7TcA2WuoKZd8L/ubOH8k+vl3lbAutS2py/HwXLpx3irN6xyJhkyZ60i944J US7qGVwO8+8248y/No+PNA2Y89Kq8YCZLBuA0lYIQJv595/TK0vQGLjXDcRHSBNQ3qvT d2K8+I1ZttYwQsepo4tCpa+ks/XpdIqTkU0pAgf5yY1eEydXIyGxxjH3mbsJmmZrqCko dkin9Nhi5wCSqqHDG5GWlbChIgYes8hVQTg5wgDkNDNtQARm8RLnfURjnzTGLtTSjfdS 4U7w== X-Gm-Message-State: AOAM531ZR98awFbPUeSvVrs24JUnp598j0OMaA9GDyLjNvkEwHI0jop8 /+fBTb18+YOSAR1enprFMFYZeT3Qo6Q= X-Google-Smtp-Source: ABdhPJys0Ny0jnAVLtqCHtQrJAvLm24vpxfTizmke5BooxLWhUj7yZMnpo/94PXIFnYku/RRYaIjcw== X-Received: by 2002:a5d:6a92:: with SMTP id s18mr9905813wru.200.1617627859864; Mon, 05 Apr 2021 06:04:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u19sm11082665wml.28.2021.04.05.06.04.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Apr 2021 06:04:19 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 05 Apr 2021 13:04:13 +0000 Subject: [PATCH 3/5] refspec: output a refspec item Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: tom.saeger@oracle.com, gitster@pobox.com, sunshine@sunshineco.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Add a new method, refspec_item_format(), that takes a 'struct refspec_item' pointer as input and returns a string for how that refspec item should be written to Git's config or a subcommand, such as 'git fetch'. There are several subtleties regarding special-case refspecs that can occur and are represented in t5511-refspec.sh. These cases will be explored in new tests in the following change. It requires adding a new test helper in order to test this format directly, so that is saved for a separate change to keep this one focused on the logic of the format method. A future change will consume this method when translating refspecs in the 'prefetch' task of the 'git maintenance' builtin. Signed-off-by: Derrick Stolee --- refspec.c | 25 +++++++++++++++++++++++++ refspec.h | 5 +++++ 2 files changed, 30 insertions(+) diff --git a/refspec.c b/refspec.c index e3d852c0bfec..ca65ba01bfe6 100644 --- a/refspec.c +++ b/refspec.c @@ -180,6 +180,31 @@ void refspec_item_clear(struct refspec_item *item) item->exact_sha1 = 0; } +const char *refspec_item_format(const struct refspec_item *rsi) +{ + static struct strbuf buf = STRBUF_INIT; + + strbuf_reset(&buf); + + if (rsi->matching) + return ":"; + + if (rsi->negative) + strbuf_addch(&buf, '^'); + else if (rsi->force) + strbuf_addch(&buf, '+'); + + if (rsi->src) + strbuf_addstr(&buf, rsi->src); + + if (rsi->dst) { + strbuf_addch(&buf, ':'); + strbuf_addstr(&buf, rsi->dst); + } + + return buf.buf; +} + void refspec_init(struct refspec *rs, int fetch) { memset(rs, 0, sizeof(*rs)); diff --git a/refspec.h b/refspec.h index 8b79891d3218..92a312f5b4e6 100644 --- a/refspec.h +++ b/refspec.h @@ -56,6 +56,11 @@ int refspec_item_init(struct refspec_item *item, const char *refspec, void refspec_item_init_or_die(struct refspec_item *item, const char *refspec, int fetch); void refspec_item_clear(struct refspec_item *item); +/* + * Output a given refspec item to a string. + */ +const char *refspec_item_format(const struct refspec_item *rsi); + void refspec_init(struct refspec *rs, int fetch); void refspec_append(struct refspec *rs, const char *refspec); __attribute__((format (printf,2,3))) From patchwork Mon Apr 5 13:04:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12182921 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1955C43460 for ; Mon, 5 Apr 2021 13:04:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 79E58613A3 for ; Mon, 5 Apr 2021 13:04:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240821AbhDENEc (ORCPT ); Mon, 5 Apr 2021 09:04:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50230 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235704AbhDENE2 (ORCPT ); Mon, 5 Apr 2021 09:04:28 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E0E50C061756 for ; Mon, 5 Apr 2021 06:04:21 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id x7so10862626wrw.10 for ; Mon, 05 Apr 2021 06:04:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5L7gpGfJ37qujqIwEpILGJe44hZo32Ph3clz45KGZjY=; b=VbGxMv8LhwRMKZUYcu6ozGYMUmn1f0omcMYV4MFKFcueq7RFmzJH5t79d6jYm/SnIN FhsSGm31lgIdfE+AMKeZZVX3sGP8PEleKtocy+faoXgySZhbPFYg3jnnla8S3kBc6noE UZacba21vW5IQLNw78glZbpY0pdRAcMfDwCIHEFDZfk+rb2K1yNncRDMXhA8pEfPwWG0 BKTy6iFZ5Ksa6VVR1vUBaEhuxZpivQdY2CRYv3xNpGfM0PE/fP31mBwoOkxCXebPVFT4 Mf48Ey1wd+HkBmGxOXR5JOg5TYinY+pYqYYUm5/apBWS86W87Heqrpjf41m7QtpmHfk2 lpuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5L7gpGfJ37qujqIwEpILGJe44hZo32Ph3clz45KGZjY=; b=EPWzv+bXRC6uyWvj6KDytZlCqaEpv7O92gJu2WzM5U+NB77CjaY6Ru5bu8hvFkjabE 9gfKXEKqHfQIrihb0i7xZTWNCIO5AYNk/4CDH9bOCnmQBgQ7DYSpmfgmKlzS+xg1So/c ObVXNveJ2THa8bH2virNzFxRoEHL1o59dKTMyP2dJJJWj6eHw7dnAZzDlsckyjkf+PO5 FliQ3UY2DVmH41CgQKESxgaDeYugEKeyzUYT04i4UtkiEbw0r2YtpDo9Dmm8ZQPvxbBb mhVyv/zyt8bk2L41Kv+9FFJIUMeFhG4ksTd26LCiFG02akxbbM2gl57nOLh27SiOt0e+ GGRQ== X-Gm-Message-State: AOAM533ciHO+t5MNb+cc33Gja4dwA5yhNq2O4E0n7urqbhDnGIjF9zO0 xT2ammDbLj/7+YKQdnYQO9FtjlCbuV0= X-Google-Smtp-Source: ABdhPJzDMSoKGk8PDA8OUjTpC6MTl910SHjrtThJfAy2bqSBpjxMLmkbf5n7kYr/Nlgm0VvlgeAdYw== X-Received: by 2002:adf:f14b:: with SMTP id y11mr1901235wro.79.1617627860601; Mon, 05 Apr 2021 06:04:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h6sm15704676wrr.21.2021.04.05.06.04.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Apr 2021 06:04:20 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Mon, 05 Apr 2021 13:04:14 +0000 Subject: [PATCH 4/5] test-tool: test refspec input/output Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: tom.saeger@oracle.com, gitster@pobox.com, sunshine@sunshineco.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Add a new test-helper, 'test-tool refspec', that currently reads stdin line-by-line and translates the refspecs using the parsing logic of refspec_item_init() and writes them to output. Create a test in t5511-refspec.sh that uses this helper to test several known special cases. This includes all of the special properties of the 'struct refspec_item', including: * force: The refspec starts with '+'. * pattern: Each side of the refspec has a glob character ('*') * matching: The refspec is simply the string ":". * exact_sha1: The 'src' string is a 40-character hex string. * negative: The refspec starts with '^' and 'dst' is NULL. While the exact_sha1 property doesn't require special logic in refspec_item_format, it is still tested here for completeness. There is also the special-case refspec "@" which translates to "HEAD". Note that if a refspec does not start with "refs/", then that is not incorporated as part of the 'struct refspec_item'. This behavior is confirmed by these tests. These refspecs still work in the wild because the refs layer interprets them appropriately as branches, prepending "refs/" or "refs/heads/" as necessary. I spent some time attempting to insert these prefixes explicitly in parse_refspec(), but these are several subtleties I was unable to overcome. If such a change were to be made, then this new test in t5511-refspec.sh will need to be updated with new output. For example, the input lines ending with "translated" are designed to demonstrate these subtleties. Signed-off-by: Derrick Stolee --- Makefile | 1 + t/helper/test-refspec.c | 39 +++++++++++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t5511-refspec.sh | 41 +++++++++++++++++++++++++++++++++++++++++ 5 files changed, 83 insertions(+) create mode 100644 t/helper/test-refspec.c diff --git a/Makefile b/Makefile index a6a73c574191..f858c9f25976 100644 --- a/Makefile +++ b/Makefile @@ -734,6 +734,7 @@ TEST_BUILTINS_OBJS += test-reach.o TEST_BUILTINS_OBJS += test-read-cache.o TEST_BUILTINS_OBJS += test-read-graph.o TEST_BUILTINS_OBJS += test-read-midx.o +TEST_BUILTINS_OBJS += test-refspec.o TEST_BUILTINS_OBJS += test-ref-store.o TEST_BUILTINS_OBJS += test-regex.o TEST_BUILTINS_OBJS += test-repository.o diff --git a/t/helper/test-refspec.c b/t/helper/test-refspec.c new file mode 100644 index 000000000000..08cf441a0a06 --- /dev/null +++ b/t/helper/test-refspec.c @@ -0,0 +1,39 @@ +#include "cache.h" +#include "parse-options.h" +#include "refspec.h" +#include "strbuf.h" +#include "test-tool.h" + +static const char * const refspec_usage[] = { + N_("test-tool refspec [--fetch]"), + NULL +}; + +int cmd__refspec(int argc, const char **argv) +{ + struct strbuf line = STRBUF_INIT; + int fetch = 0; + + struct option refspec_options [] = { + OPT_BOOL(0, "fetch", &fetch, + N_("enable the 'fetch' option for parsing refpecs")), + OPT_END() + }; + + argc = parse_options(argc, argv, NULL, refspec_options, + refspec_usage, 0); + + while (strbuf_getline(&line, stdin) != EOF) { + struct refspec_item rsi; + + if (!refspec_item_init(&rsi, line.buf, fetch)) { + printf("failed to parse %s\n", line.buf); + continue; + } + + printf("%s\n", refspec_item_format(&rsi)); + refspec_item_clear(&rsi); + } + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 287aa6002307..f534ad1731a9 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -55,6 +55,7 @@ static struct test_cmd cmds[] = { { "read-cache", cmd__read_cache }, { "read-graph", cmd__read_graph }, { "read-midx", cmd__read_midx }, + { "refspec", cmd__refspec }, { "ref-store", cmd__ref_store }, { "regex", cmd__regex }, { "repository", cmd__repository }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 9ea4b31011dd..46a0b8850f17 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -44,6 +44,7 @@ int cmd__reach(int argc, const char **argv); int cmd__read_cache(int argc, const char **argv); int cmd__read_graph(int argc, const char **argv); int cmd__read_midx(int argc, const char **argv); +int cmd__refspec(int argc, const char **argv); int cmd__ref_store(int argc, const char **argv); int cmd__regex(int argc, const char **argv); int cmd__repository(int argc, const char **argv); diff --git a/t/t5511-refspec.sh b/t/t5511-refspec.sh index be025b90f989..7614b6adf932 100755 --- a/t/t5511-refspec.sh +++ b/t/t5511-refspec.sh @@ -93,4 +93,45 @@ test_refspec fetch "refs/heads/${good}" bad=$(printf '\011tab') test_refspec fetch "refs/heads/${bad}" invalid +test_expect_success 'test input/output round trip' ' + cat >input <<-\EOF && + +refs/heads/*:refs/remotes/origin/* + refs/heads/*:refs/remotes/origin/* + refs/heads/main:refs/remotes/frotz/xyzzy + :refs/remotes/frotz/deleteme + ^refs/heads/secrets + refs/heads/secret:refs/heads/translated + refs/heads/secret:heads/translated + refs/heads/secret:remotes/translated + secret:translated + refs/heads/*:remotes/xxy/* + refs/heads*/for-linus:refs/remotes/mine/* + 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed + HEAD + @ + : + EOF + cat >expect <<-\EOF && + +refs/heads/*:refs/remotes/origin/* + refs/heads/*:refs/remotes/origin/* + refs/heads/main:refs/remotes/frotz/xyzzy + :refs/remotes/frotz/deleteme + ^refs/heads/secrets + refs/heads/secret:refs/heads/translated + refs/heads/secret:heads/translated + refs/heads/secret:remotes/translated + secret:translated + refs/heads/*:remotes/xxy/* + refs/heads*/for-linus:refs/remotes/mine/* + 2e36527f23b7f6ae15e6f21ac3b08bf3fed6ee48:refs/heads/fixed + HEAD + HEAD + : + EOF + test-tool refspec output && + test_cmp expect output && + test-tool refspec --fetch output && + test_cmp expect output +' + test_done From patchwork Mon Apr 5 13:04:15 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12182923 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8B7DC43462 for ; Mon, 5 Apr 2021 13:04:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B608B613A9 for ; Mon, 5 Apr 2021 13:04:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240945AbhDENEf (ORCPT ); Mon, 5 Apr 2021 09:04:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S237176AbhDENE2 (ORCPT ); Mon, 5 Apr 2021 09:04:28 -0400 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 54792C061788 for ; Mon, 5 Apr 2021 06:04:22 -0700 (PDT) Received: by mail-wm1-x32c.google.com with SMTP id u5-20020a7bcb050000b029010e9316b9d5so5605630wmj.2 for ; Mon, 05 Apr 2021 06:04:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=YJFPKjbf38fYJcW/14HvAHQpRrYbhNHkMU/HO3rFRFs=; b=R9QjggiyYVuiNEij9mWQK9I9Ys3NztdLt4XLFXtvmpkiO1JRg9t28xuy7ckGUfxt9v /D2aQ7lJ63RU6M/ewnqvQlML0txvVKy9/iw+2WUXw3Ano9eR+dlSNnrrmsA0crhLPSjJ pYstsF4m3bGsNnDITKXfDmx4mDJAdBrIhnscrNZWgfEVwi0tlFGEotfQCHe3xPVgHUwq sryLxnM+Z2N4ow/R7JLO817d/xxmLHDLmeYzZuUg0YgAnRtV0Q3OFeb/xPhHTvFvtIiL dRMMchlEmaWcQdRtpn76tVIHb0mKeg2/5uqf5hxHNR3MXnNYB117+niNBKnEcefFGJXk IsXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=YJFPKjbf38fYJcW/14HvAHQpRrYbhNHkMU/HO3rFRFs=; b=bdvHElL8GwbJ31flErclS2xCUwy4Vp9gac+K2jJ9saYZs5+LRihMbP5jtbkLuvovPU vyqHf7gDfq9xqUYB+Tt70BzmY/pdQf2T+zHlwvRG3WOo2MBXrmI4WtQ00SUdKPvD+IRP go8iBUEDUbIui5TfKqsidA6GfgIT4PgyWBlZiVmcLISx40rEgA5Iw05qY7t7/TKyLHUQ kiWWPycXWlYnJPnfFptV33v0wXtFMbzlfVW1bzkLojf8n8PjgrDLFqdc1j2CU8U7utGq bv7QN1GwgWKtnq3fEOzbxUZiEV8poqMJBvVlVsIDmsS0cHekn3S4ymWsBNglFtQHtIE2 7Dfg== X-Gm-Message-State: AOAM531vrZP5QKFc08nS7madXSjmPVe6cJSHy7Ey2IGuUDCC8CZqNh3C qbxv1qxgxeVhIuey7f1GLvgqhL6PDOU= X-Google-Smtp-Source: ABdhPJzHFgn/cEIA1rAkl6BnemK4eW7+ZcWbDFkt/kRAvdYe1TtEYVyHuuLruNXtEKfBkLYcnChkdg== X-Received: by 2002:a1c:9a51:: with SMTP id c78mr14216249wme.160.1617627861099; Mon, 05 Apr 2021 06:04:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id j9sm23086496wmi.24.2021.04.05.06.04.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Apr 2021 06:04:20 -0700 (PDT) Message-Id: <7f6c127dac48409ddc8d30ad236182bee21c1957.1617627856.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Apr 2021 13:04:15 +0000 Subject: [PATCH 5/5] maintenance: allow custom refspecs during prefetch Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: tom.saeger@oracle.com, gitster@pobox.com, sunshine@sunshineco.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The prefetch task previously used the default refspec source plus a custom refspec destination to avoid colliding with remote refs: +refs/heads/*:refs/prefetch//* However, some users customize their refspec to reduce how much data they download from specific remotes. This can involve restrictive patterns for fetching or negative patterns to avoid downloading some refs. Modify fetch_remote() to iterate over the remote's refspec list and translate that into the appropriate prefetch scenario. Specifically, re-parse the raw form of the refspec into a new 'struct refspec' and modify the 'dst' member to replace a leading "refs/" substring with "refs/prefetch/", or prepend "refs/prefetch/" to 'dst' otherwise. Negative refspecs do not have a 'dst' so they can be transferred to the 'git fetch' command unmodified. This prefix change provides the benefit of keeping whatever collisions may exist in the custom refspecs, if that is a desirable outcome. This changes the names of the refs that would be fetched by the default refspec. Instead of "refs/prefetch//" they will now go to "refs/prefetch/remotes//". While this is a change, it is not a seriously breaking one: these refs are intended to be hidden and not used. Update the documentation to be more generic about the destination refs. Do not mention custom refpecs explicitly, as that does not need to be highlighted in this documentation. The important part of placing refs in refs/prefetch remains. Reported-by: Tom Saeger Signed-off-by: Derrick Stolee Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 3 +-- builtin/gc.c | 34 +++++++++++++++++++++++- t/t7900-maintenance.sh | 43 ++++++++++++++++++++++++++----- 3 files changed, 71 insertions(+), 9 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 80ddd33ceba0..95a24264eb10 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -94,8 +94,7 @@ prefetch:: objects from all registered remotes. For each remote, a `git fetch` command is run. The refmap is custom to avoid updating local or remote branches (those in `refs/heads` or `refs/remotes`). Instead, the - remote refs are stored in `refs/prefetch//`. Also, tags are - not updated. + refs are stored in `refs/prefetch/`. Also, tags are not updated. + This is done to avoid disrupting the remote-tracking branches. The end users expect these refs to stay unmoved unless they initiate a fetch. With prefetch diff --git a/builtin/gc.c b/builtin/gc.c index fa8128de9ae1..92cb8b4e0bfa 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -32,6 +32,7 @@ #include "remote.h" #include "object-store.h" #include "exec-cmd.h" +#include "refspec.h" #define FAILED_RUN "failed to run %s" @@ -877,6 +878,7 @@ static int fetch_remote(struct remote *remote, void *cbdata) { struct maintenance_run_opts *opts = cbdata; struct child_process child = CHILD_PROCESS_INIT; + int i; child.git_cmd = 1; strvec_pushl(&child.args, "fetch", remote->name, "--prune", "--no-tags", @@ -886,7 +888,37 @@ static int fetch_remote(struct remote *remote, void *cbdata) if (opts->quiet) strvec_push(&child.args, "--quiet"); - strvec_pushf(&child.args, "+refs/heads/*:refs/prefetch/%s/*", remote->name); + for (i = 0; i < remote->fetch.nr; i++) { + struct refspec_item replace; + struct refspec_item *rsi = &remote->fetch.items[i]; + struct strbuf new_dst = STRBUF_INIT; + size_t ignore_len = 0; + + if (rsi->negative) { + strvec_push(&child.args, remote->fetch.raw[i]); + continue; + } + + refspec_item_init(&replace, remote->fetch.raw[i], 1); + + /* + * If a refspec dst starts with "refs/" at the start, + * then we will replace "refs/" with "refs/prefetch/". + * Otherwise, we will prepend the dst string with + * "refs/prefetch/". + */ + if (!strncmp(replace.dst, "refs/", 5)) + ignore_len = 5; + + strbuf_addstr(&new_dst, "refs/prefetch/"); + strbuf_addstr(&new_dst, replace.dst + ignore_len); + free(replace.dst); + replace.dst = strbuf_detach(&new_dst, NULL); + + strvec_push(&child.args, refspec_item_format(&replace)); + + refspec_item_clear(&replace); + } return !!run_command(&child); } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index fc2315edec11..3366ea188782 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -142,20 +142,51 @@ test_expect_success 'prefetch multiple remotes' ' test_commit -C clone2 two && GIT_TRACE2_EVENT="$(pwd)/run-prefetch.txt" git maintenance run --task=prefetch 2>/dev/null && fetchargs="--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet" && - test_subcommand git fetch remote1 $fetchargs +refs/heads/*:refs/prefetch/remote1/* log && ! grep "prefetch" log ' +test_expect_success 'prefetch custom refspecs' ' + git -C clone1 branch -f special/fetched HEAD && + git -C clone1 branch -f special/secret/not-fetched HEAD && + + # create multiple refspecs for remote1 + git config --add remote.remote1.fetch +refs/heads/special/fetched:refs/heads/fetched && + git config --add remote.remote1.fetch ^refs/heads/special/secret/not-fetched && + + GIT_TRACE2_EVENT="$(pwd)/prefetch-refspec.txt" git maintenance run --task=prefetch 2>/dev/null && + + fetchargs="--prune --no-tags --no-write-fetch-head --recurse-submodules=no --refmap= --quiet" && + + # skips second refspec because it is not a pattern type + rs1="+refs/heads/*:refs/prefetch/remotes/remote1/*" && + rs2="+refs/heads/special/fetched:refs/prefetch/heads/fetched" && + rs3="^refs/heads/special/secret/not-fetched" && + + test_subcommand git fetch remote1 $fetchargs $rs1 $rs2 $rs3