From patchwork Tue Feb 9 13:42:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12078203 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 45577C433DB for ; Tue, 9 Feb 2021 13:45:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F00E964ED7 for ; Tue, 9 Feb 2021 13:45:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231723AbhBINp0 (ORCPT ); Tue, 9 Feb 2021 08:45:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59616 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231503AbhBINnP (ORCPT ); Tue, 9 Feb 2021 08:43:15 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B36BAC061788 for ; Tue, 9 Feb 2021 05:42:34 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id g10so21857118wrx.1 for ; Tue, 09 Feb 2021 05:42:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=yFp1XTjvHQxcMPHbhdWE9yRT59wXZWm/2ZDVLjNmWi8=; b=fJZFlDM4h379Aer8MtDMKOxD/1lOMlPyfO7f4fk4i8J9gKz4AjnlpEQoTu9ebyCkQo qWqxYBpZxzXrXAxdcuhkpBBmbiQlxsOsyD8MNDtg6uzh0iiDpDOSOvVqbdqrmzw8Co/s Kr3WuBBfS20MgKG+efcTKfObj5DkbHTrynXIU1wccLnFaPMczpGI3pmjPj0UXKEp5WMc dU9MuVGg4rVrJxF3Uhpn9lhB3FNke4uMe32R977iCOLMKLE1SG1rU/tDngy9MMd6j+YN 1zKE5HOmcnoUW1vivS+ZEM1mRNnlSGOfCPDPjvaKbNYNUkoQqmkMHGq/6iErlQsiLoEF inYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=yFp1XTjvHQxcMPHbhdWE9yRT59wXZWm/2ZDVLjNmWi8=; b=m/Moz5H6/CodpUmVV+pM7CvrGeZlLpGBoj02M9I6GPbAR8EGepZVll6K5ZQ3iGJ70H 3VHYUYJTzqp0C0hgrxjivQLn05V/LbnQdX247vEAkSuNPvcejd9GhYuparoIJbIk1zcJ QmxwiilRJuUcfrxlMnljZcyTe0565Dbq4d5A1+X8b0l4HxWJHPwBIVw5aOo3WWeN1H8o +j7TdHaro0mcd5qv8DDJpeTTMynyX1+VDfao0JC5Nwvy/oCfeSmIZlhPJ69VYp0zXb2w prQKaLNztuSCZhTR5Bt0NZvFy8AYmPOzFtGt74G5DLIvMO0AMZX9rkGZk5rmyqBswozW +wdQ== X-Gm-Message-State: AOAM532fIpbLda6tVjoCLriQy7/S0OYBbXegHb51WaTmP1zqOAfw8oCL 3CSGVoz8MG1hAAFewD1rsbLRpSDs/jc= X-Google-Smtp-Source: ABdhPJz6ik09bjomG7I3jFZlTruwfMcfVE3LCMio56syVtVZI3WMNK1IBJjXTMLDb+nPOVX+be/RZQ== X-Received: by 2002:adf:f8c1:: with SMTP id f1mr25450979wrq.76.1612878153280; Tue, 09 Feb 2021 05:42:33 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d16sm35619065wrr.59.2021.02.09.05.42.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 05:42:32 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 09 Feb 2021 13:42:28 +0000 Subject: [PATCH v2 1/2] maintenance: add pack-refs task Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee It is valuable to collect loose refs into a more compressed form. This is typically the packed-refs file, although this could be the reftable in the future. Having packed refs can be extremely valuable in repos with many tags or remote branches that are not modified by the local user, but still are necessary for other queries. For instance, with many exploded refs, commands such as git describe --tags --exact-match HEAD can be very slow (multiple seconds). This command in particular is used by terminal prompts to show when a detatched HEAD is pointing to an existing tag, so having it be slow causes significant delays for users. Add a new 'pack-refs' maintenance task. It runs 'git pack-refs --all --prune' to move loose refs into a packed form. For now, that is the packed-refs file, but could adjust to other file formats in the future. This is the first of several sub-tasks of the 'gc' task that could be extracted to their own tasks. In this process, we should not change the behavior of the 'gc' task since that remains the default way to keep repositories maintained. Creating a new task for one of these sub-tasks only provides more customization options for those choosing to not use the 'gc' task. It is certainly possible to have both the 'gc' and 'pack-refs' tasks enabled and run regularly. While they may repeat effort, they do not conflict in a destructive way. The 'auto_condition' function pointer is left NULL for now. We could extend this in the future to have a condition check if pack-refs should be run during 'git maintenance run --auto'. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 6 ++++++ builtin/gc.c | 21 +++++++++++++++++---- t/t7900-maintenance.sh | 12 ++++++++++++ 3 files changed, 35 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 3b432171d608..80ddd33ceba0 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -145,6 +145,12 @@ incremental-repack:: which is a special case that attempts to repack all pack-files into a single pack-file. +pack-refs:: + The `pack-refs` task collects the loose reference files and + collects them into a single file. This speeds up operations that + need to iterate across many references. See linkgit:git-pack-refs[1] + for more information. + OPTIONS ------- --auto:: diff --git a/builtin/gc.c b/builtin/gc.c index 4c40594d660e..41bec4f177b3 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -54,7 +54,6 @@ static const char *prune_worktrees_expire = "3.months.ago"; static unsigned long big_pack_threshold; static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE; -static struct strvec pack_refs_cmd = STRVEC_INIT; static struct strvec reflog = STRVEC_INIT; static struct strvec repack = STRVEC_INIT; static struct strvec prune = STRVEC_INIT; @@ -163,6 +162,15 @@ static void gc_config(void) git_config(git_default_config, NULL); } +struct maintenance_run_opts; +static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts) +{ + struct strvec pack_refs_cmd = STRVEC_INIT; + strvec_pushl(&pack_refs_cmd, "pack-refs", "--all", "--prune", NULL); + + return run_command_v_opt(pack_refs_cmd.v, RUN_GIT_CMD); +} + static int too_many_loose_objects(void) { /* @@ -518,8 +526,8 @@ static void gc_before_repack(void) if (done++) return; - if (pack_refs && run_command_v_opt(pack_refs_cmd.v, RUN_GIT_CMD)) - die(FAILED_RUN, pack_refs_cmd.v[0]); + if (pack_refs && maintenance_task_pack_refs(NULL)) + die(FAILED_RUN, "pack-refs"); if (prune_reflogs && run_command_v_opt(reflog.v, RUN_GIT_CMD)) die(FAILED_RUN, reflog.v[0]); @@ -556,7 +564,6 @@ int cmd_gc(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_gc_usage, builtin_gc_options); - strvec_pushl(&pack_refs_cmd, "pack-refs", "--all", "--prune", NULL); strvec_pushl(&reflog, "reflog", "expire", "--all", NULL); strvec_pushl(&repack, "repack", "-d", "-l", NULL); strvec_pushl(&prune, "prune", "--expire", NULL); @@ -1224,6 +1231,7 @@ enum maintenance_task_label { TASK_INCREMENTAL_REPACK, TASK_GC, TASK_COMMIT_GRAPH, + TASK_PACK_REFS, /* Leave as final value */ TASK__COUNT @@ -1255,6 +1263,11 @@ static struct maintenance_task tasks[] = { maintenance_task_commit_graph, should_write_commit_graph, }, + [TASK_PACK_REFS] = { + "pack-refs", + maintenance_task_pack_refs, + NULL, + }, }; static int compare_tasks_by_selection(const void *a_, const void *b_) diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 78ccf4b33f87..4a8a78769bd6 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -343,6 +343,18 @@ test_expect_success 'maintenance.incremental-repack.auto' ' test_subcommand git multi-pack-index write --no-progress after && + test_must_be_empty after && + test_subcommand git pack-refs --all --prune err && test_i18ngrep "at most one" err From patchwork Tue Feb 9 13:42:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12078205 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6AA3DC433E6 for ; Tue, 9 Feb 2021 13:45:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2C4D464ED2 for ; Tue, 9 Feb 2021 13:45:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231585AbhBINpb (ORCPT ); Tue, 9 Feb 2021 08:45:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59624 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231249AbhBINnQ (ORCPT ); Tue, 9 Feb 2021 08:43:16 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E96A5C06178A for ; Tue, 9 Feb 2021 05:42:35 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id 7so21874255wrz.0 for ; Tue, 09 Feb 2021 05:42:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=D+M6pr6oWKUqk3Ewhp2PVkO4UkRC3UrEK0yzXOdMM78=; b=s5RFiL+dSdUBm+scdk11SpOm7iehtoK3GLMbuZldc7ZknlCyMloiLiE6xYRoz9FHLj uC3niZurN1Rf+ltj4iFajYYC84w/oqo1DT1HvXLB7WJe4lpS6PZcHmerKeSw/V212O7d HqOVhIsqKmL8iH/mWru0rp4ny1aXia+kLcTqObjHd3/WU0+yGY/a8ARda/s2TGpqiXDj JRO17d1y5dKGIt39qR+e8sIiIXou9DQFIozm4zAsfRAZZQl8Es96gInm003VBKppP3kB i9JeRdyhRkDtng7RzQks9k6uvIXZdojV1u01FEWKwEbxQAEUoPt2KxITIlgNFOgoN05T jvlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=D+M6pr6oWKUqk3Ewhp2PVkO4UkRC3UrEK0yzXOdMM78=; b=TP6Ls3MjByiSkYK7TjpTL4TFglXC8+LONmZukns+9vEYtG2qAPHCHAtP6Wfj18HWLV RhLw1+pn69XyutwscmfPk66kANf6To8DV6g0KsuEpkw1CdJ9zaWvNMqb4ePk7NsWrwaT ImS50wse7Q0FZRBthS9l9fFbgtgRxFh6q6iMk/vEfmkQwS2ab0KkQ+g+kRBrw0ydE2H5 +IXMJ4W3b55ov6Y/UPjos02pzWO3IG9wfVQ6qmfbdf5hX5Pfhsp1hX58ESF2QqHfI47P W9xICt5xo7D9+CFLfbqE7hIF2KuCuvruKqM3SDdYEdynBsoMUTHIh8ERY0b4sBTp6cLy mSRg== X-Gm-Message-State: AOAM531QarbBHVaV34GmWPzQThbdpDW2+AxL/DfSyrGvF3LAHMqfImED Ed7S98NPrqRjFCxytAZYMzVGgGzfvvk= X-Google-Smtp-Source: ABdhPJzcVxPmzHEQpVIJAX+TqajDQenNHrV0ja4zOvw2gtOt17/USNRsHZNl2R7b/8V6Y75RxPp9xg== X-Received: by 2002:a5d:554a:: with SMTP id g10mr16428215wrw.383.1612878154489; Tue, 09 Feb 2021 05:42:34 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d23sm5003668wmd.11.2021.02.09.05.42.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 09 Feb 2021 05:42:33 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 09 Feb 2021 13:42:29 +0000 Subject: [PATCH v2 2/2] maintenance: incremental strategy runs pack-refs weekly Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Taylor Blau , Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When the 'maintenance.strategy' config option is set to 'incremental', a default maintenance schedule is enabled. Add the 'pack-refs' task to that strategy at the weekly cadence. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 5 +++-- builtin/gc.c | 2 ++ t/t7900-maintenance.sh | 14 ++++++++++++++ 3 files changed, 19 insertions(+), 2 deletions(-) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index a5ead09e4bc2..18f056213145 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -15,8 +15,9 @@ maintenance.strategy:: * `none`: This default setting implies no task are run at any schedule. * `incremental`: This setting optimizes for performing small maintenance activities that do not delete any data. This does not schedule the `gc` - task, but runs the `prefetch` and `commit-graph` tasks hourly and the - `loose-objects` and `incremental-repack` tasks daily. + task, but runs the `prefetch` and `commit-graph` tasks hourly, the + `loose-objects` and `incremental-repack` tasks daily, and the `pack-refs` + task weekly. maintenance..enabled:: This boolean config option controls whether the maintenance task diff --git a/builtin/gc.c b/builtin/gc.c index 41bec4f177b3..6db9cb39e679 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1352,6 +1352,8 @@ static void initialize_maintenance_strategy(void) tasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY; tasks[TASK_LOOSE_OBJECTS].enabled = 1; tasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY; + tasks[TASK_PACK_REFS].enabled = 1; + tasks[TASK_PACK_REFS].schedule = SCHEDULE_WEEKLY; } } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 4a8a78769bd6..286b18db3cc2 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -408,18 +408,32 @@ test_expect_success 'maintenance.strategy inheritance' ' git maintenance run --schedule=hourly --quiet && GIT_TRACE2_EVENT="$(pwd)/incremental-daily.txt" \ git maintenance run --schedule=daily --quiet && + GIT_TRACE2_EVENT="$(pwd)/incremental-weekly.txt" \ + git maintenance run --schedule=weekly --quiet && test_subcommand git commit-graph write --split --reachable \ --no-progress