From patchwork Tue Aug 18 14:22:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721047 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A9A121575 for ; Tue, 18 Aug 2020 14:23:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 86CE42086A for ; Tue, 18 Aug 2020 14:23:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dtBbgBQz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726844AbgHROXc (ORCPT ); Tue, 18 Aug 2020 10:23:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726542AbgHROXO (ORCPT ); Tue, 18 Aug 2020 10:23:14 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 804DFC061342 for ; Tue, 18 Aug 2020 07:23:13 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id p14so16404196wmg.1 for ; Tue, 18 Aug 2020 07:23:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=rz0elcoRUQhrWP42+CgdqapPZCpm/iX374uh59Y88Gc=; b=dtBbgBQz3oCExgbZwuzBd4yBlduH1EE6uAk+3wVFQsBAXwlH52F2aJHB4+p9FG92uh U2Wk4odcqjt5I+fELvhs+w7rU8XtXXPhCUlazv19MoT+576BOs6LWP1sHtyuMqe6x3S9 rnOseJwdae+zEdrsL2zZhOAwrHQgkNRHNgctMjq9RgsA68Vpjs7uuuAQgEQsbtQeZfj+ YByQE63XXHwtyV3gOEAhFEaXIUy3Ud3G0C9Hl2KOIqAjweYUQLAca6vdiKlqCYcPFgGu fH9E7qLJ/rc41KmX0yXgjAO7kVNMdLLcjUSkDLgmYTjkGZbTO87ODjnpn/3npgwNY8vq pOIw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=rz0elcoRUQhrWP42+CgdqapPZCpm/iX374uh59Y88Gc=; b=JvM6sKp0AebWKNU/NgiQRxRITL379fSDkkaEJ0l/V4FEmuMGAKZjyk7LE11RtjlpIx mhipRhaZ6IhgJFKYmq7/SP8OCj9NwQc1bre9ApkItvLYA7ghGP9S6dptHrb/YqNAIKDt M3NPxhFFW16QxEPZf2V1LNL1AVotyROefMyP4jSsZhM6v4HW+fDUQ+pUFFJ+0GetUgXA ZAarfE9+wxn3RxRih6x4/OoG2YhBNDWe2CzNQ23XXS67as/YrK8iniEX6+CEH0TGeP5+ 9IFtr1xjd6pA5t308stx/9QjtvxjKPiQ6CQc2wynAa72EuAb+hWS9YmRIiRMjcD87B0I ndMQ== X-Gm-Message-State: AOAM532jHIAS1qirXgNxLxRe9HA7MJ93ZLcUkPqw9NVBWVeNzC7l96Ej uGyv6Mng9dYnRnhLz/rkqaa6sVkesb4= X-Google-Smtp-Source: ABdhPJzVuoU1DQD4D8gOIc/ctUoStydVnSOT5L/rR5LBxdfa0kUoHpDpLSHmLP7oBUnSLCdrrvSj0Q== X-Received: by 2002:a7b:c197:: with SMTP id y23mr149273wmi.165.1597760591747; Tue, 18 Aug 2020 07:23:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 145sm155571wma.20.2020.08.18.07.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:11 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:22:58 +0000 Subject: [PATCH v2 01/11] maintenance: create basic maintenance runner Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'gc' builtin is our current entrypoint for automatically maintaining a repository. This one tool does many operations, such as repacking the repository, packing refs, and rewriting the commit-graph file. The name implies it performs "garbage collection" which means several different things, and some users may not want to use this operation that rewrites the entire object database. Create a new 'maintenance' builtin that will become a more general- purpose command. To start, it will only support the 'run' subcommand, but will later expand to add subcommands for scheduling maintenance in the background. For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin. In fact, the only option is the '--auto' toggle, which is handed directly to the 'gc' builtin. The current change is isolated to this simple operation to prevent more interesting logic from being lost in all of the boilerplate of adding a new builtin. Use existing builtin/gc.c file because we want to share code between the two builtins. It is possible that we will have 'maintenance' replace the 'gc' builtin entirely at some point, leaving 'git gc' as an alias for some specific arguments to 'git maintenance run'. Create a new test_subcommand helper that allows us to test if a certain subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a file. A negation mode is available that will be used in later tests. Helped-by: Jonathan Nieder Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-maintenance.txt | 57 ++++++++++++++++++++++++++++++ builtin.h | 1 + builtin/gc.c | 58 +++++++++++++++++++++++++++++++ git.c | 1 + t/t7900-maintenance.sh | 21 +++++++++++ t/test-lib-functions.sh | 33 ++++++++++++++++++ 7 files changed, 172 insertions(+) create mode 100644 Documentation/git-maintenance.txt create mode 100755 t/t7900-maintenance.sh diff --git a/.gitignore b/.gitignore index ee509a2ad2..a5808fa30d 100644 --- a/.gitignore +++ b/.gitignore @@ -90,6 +90,7 @@ /git-ls-tree /git-mailinfo /git-mailsplit +/git-maintenance /git-merge /git-merge-base /git-merge-index diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt new file mode 100644 index 0000000000..ff47fb3641 --- /dev/null +++ b/Documentation/git-maintenance.txt @@ -0,0 +1,57 @@ +git-maintenance(1) +================== + +NAME +---- +git-maintenance - Run tasks to optimize Git repository data + + +SYNOPSIS +-------- +[verse] +'git maintenance' run [] + + +DESCRIPTION +----------- +Run tasks to optimize Git repository data, speeding up other Git commands +and reducing storage requirements for the repository. + +Git commands that add repository data, such as `git add` or `git fetch`, +are optimized for a responsive user experience. These commands do not take +time to optimize the Git data, since such optimizations scale with the full +size of the repository while these user commands each perform a relatively +small action. + +The `git maintenance` command provides flexibility for how to optimize the +Git repository. + +SUBCOMMANDS +----------- + +run:: + Run one or more maintenance tasks. + +TASKS +----- + +gc:: + Clean up unnecessary files and optimize the local repository. "GC" + stands for "garbage collection," but this task performs many + smaller tasks. This task can be expensive for large repositories, + as it repacks all Git objects into a single pack-file. It can also + be disruptive in some situations, as it deletes stale data. See + linkgit:git-gc[1] for more details on garbage collection in Git. + +OPTIONS +------- +--auto:: + When combined with the `run` subcommand, run maintenance tasks + only if certain thresholds are met. For example, the `gc` task + runs when the number of loose objects exceeds the number stored + in the `gc.auto` config setting, or when the number of pack-files + exceeds the `gc.autoPackLimit` config setting. + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/builtin.h b/builtin.h index a5ae15bfe5..17c1c0ce49 100644 --- a/builtin.h +++ b/builtin.h @@ -167,6 +167,7 @@ int cmd_ls_tree(int argc, const char **argv, const char *prefix); int cmd_ls_remote(int argc, const char **argv, const char *prefix); int cmd_mailinfo(int argc, const char **argv, const char *prefix); int cmd_mailsplit(int argc, const char **argv, const char *prefix); +int cmd_maintenance(int argc, const char **argv, const char *prefix); int cmd_merge(int argc, const char **argv, const char *prefix); int cmd_merge_base(int argc, const char **argv, const char *prefix); int cmd_merge_index(int argc, const char **argv, const char *prefix); diff --git a/builtin/gc.c b/builtin/gc.c index aafa0946f5..ce91c754ee 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -699,3 +699,61 @@ int cmd_gc(int argc, const char **argv, const char *prefix) return 0; } + +static const char * const builtin_maintenance_usage[] = { + N_("git maintenance run []"), + NULL +}; + +struct maintenance_opts { + int auto_flag; +}; + +static int maintenance_task_gc(struct maintenance_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_push(&child.args, "gc"); + + if (opts->auto_flag) + strvec_push(&child.args, "--auto"); + + close_object_store(the_repository->objects); + return run_command(&child); +} + +static int maintenance_run(struct maintenance_opts *opts) +{ + return maintenance_task_gc(opts); +} + +int cmd_maintenance(int argc, const char **argv, const char *prefix) +{ + struct maintenance_opts opts; + struct option builtin_maintenance_options[] = { + OPT_BOOL(0, "auto", &opts.auto_flag, + N_("run tasks based on the state of the repository")), + OPT_END() + }; + + memset(&opts, 0, sizeof(opts)); + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_maintenance_usage, + builtin_maintenance_options); + + argc = parse_options(argc, argv, prefix, + builtin_maintenance_options, + builtin_maintenance_usage, + PARSE_OPT_KEEP_UNKNOWN); + + if (argc != 1) + usage_with_options(builtin_maintenance_usage, + builtin_maintenance_options); + + if (!strcmp(argv[0], "run")) + return maintenance_run(&opts); + + die(_("invalid subcommand: %s"), argv[0]); +} diff --git a/git.c b/git.c index 8bd1d7551d..24f250d29a 100644 --- a/git.c +++ b/git.c @@ -529,6 +529,7 @@ static struct cmd_struct commands[] = { { "ls-tree", cmd_ls_tree, RUN_SETUP }, { "mailinfo", cmd_mailinfo, RUN_SETUP_GENTLY | NO_PARSEOPT }, { "mailsplit", cmd_mailsplit, NO_PARSEOPT }, + { "maintenance", cmd_maintenance, RUN_SETUP_GENTLY | NO_PARSEOPT }, { "merge", cmd_merge, RUN_SETUP | NEED_WORK_TREE }, { "merge-base", cmd_merge_base, RUN_SETUP }, { "merge-file", cmd_merge_file, RUN_SETUP_GENTLY }, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh new file mode 100755 index 0000000000..14aa691f19 --- /dev/null +++ b/t/t7900-maintenance.sh @@ -0,0 +1,21 @@ +#!/bin/sh + +test_description='git maintenance builtin' + +. ./test-lib.sh + +test_expect_success 'help text' ' + test_expect_code 129 git maintenance -h 2>err && + test_i18ngrep "usage: git maintenance run" err && + test_expect_code 128 git maintenance barf 2>err && + test_i18ngrep "invalid subcommand: barf" err +' + +test_expect_success 'run [--auto]' ' + GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run && + GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto && + test_subcommand git gc ... < +# +# For example, to look for an invocation of "git upload-pack +# /path/to/repo" +# +# GIT_TRACE2_EVENT=event.log git fetch ... && +# test_subcommand git upload-pack "$PATH" X-Patchwork-Id: 11721043 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 214521575 for ; Tue, 18 Aug 2020 14:23:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 039F5207DA for ; Tue, 18 Aug 2020 14:23:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="P7JCF3BN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726630AbgHROX3 (ORCPT ); Tue, 18 Aug 2020 10:23:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50604 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726592AbgHROXO (ORCPT ); Tue, 18 Aug 2020 10:23:14 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EB888C061343 for ; Tue, 18 Aug 2020 07:23:13 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id c15so18432903wrs.11 for ; Tue, 18 Aug 2020 07:23:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=L/FClNfAsjRYer0xPyUbGOAjl73XOyq8VX1msCNRm+Y=; b=P7JCF3BNkGIWrQHqS9r9ne5wYYKGwZD3O1jkmFTTn43+VUf8YRSmsH77vJR0XXs0Ia Ldk1BzRLFOwThFvJ8HkIeypaAVHNYGN3zey6TWlyW24WFbZyVNKCUTjwF5tQ8COSahn+ VohgrlvrUMbZHH7MeXu+qcEqjTA2ms7/eYrZndLFYxYnsGhGOHvpQl8D9evx/SI2tfuG pHV54ZSsB/gi4gt4OLrypHom5TenAT0MMDZTRHXRUy4X5R9zLONaR9uoD/YHDc5q0mtt cXi6YvBz7Cidvl9EKIpHzGuJ2Z6oQZ036Zk5HF6tKSuWJ96P7jobt56xNp2WM2omtEJk z07w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=L/FClNfAsjRYer0xPyUbGOAjl73XOyq8VX1msCNRm+Y=; b=rnA2oQqESntOMjRMKlNI6TcAjaNpkUd7CZ7rxbiLGRL6sGMH4JImTtoBoi9ay6Uowy xldy5R3aTBxIYVfUfBXn1xC9vAHgTf2fdP5O7rcTWSoZsIV1+6gvEAVw1w2IRE7xhjqu LIx5qb0c4u3fZGWWTNNFKkjhCaA2KB0YQYI6cyRIajENgXwinnerU2sxMmgSx2f4rGkm +V1/NhF9jLFeH6sXPHybE6+yLDGq2dWQJ0g4V8FGM08uy0xHjhuxlR9nYq80nvA2pDpQ z8T6AojvW5wpQJ8lsLKnXG3BmZta7KnAACnyKbTplmqkvAkXjehZd8izzxVTWqwNID0M vD6g== X-Gm-Message-State: AOAM531LrAePvXShWEVqdHpT4qmV9kxpgiVvKekhOQ79xumXrHUC35WV uMEniE7argF0CGfN1MGgGdeCDWY0GDA= X-Google-Smtp-Source: ABdhPJyxh95jXZNkgdx4eDSCKUhblcOwkVez8zhPwet63+i6vZEyWWnHzDJEMRVvz70VSqgf7zSJGQ== X-Received: by 2002:adf:c108:: with SMTP id r8mr21153577wre.350.1597760592492; Tue, 18 Aug 2020 07:23:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g188sm174548wma.5.2020.08.18.07.23.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:12 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:22:59 +0000 Subject: [PATCH v2 02/11] maintenance: add --quiet option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Maintenance activities are commonly used as steps in larger scripts. Providing a '--quiet' option allows those scripts to be less noisy when run on a terminal window. Turn this mode on by default when stderr is not a terminal. Pipe the option to the 'git gc' child process. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 3 +++ builtin/gc.c | 9 +++++++++ t/t7900-maintenance.sh | 15 ++++++++++----- 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index ff47fb3641..04fa0fe329 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -52,6 +52,9 @@ OPTIONS in the `gc.auto` config setting, or when the number of pack-files exceeds the `gc.autoPackLimit` config setting. +--quiet:: + Do not report progress or other information over `stderr`. + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index ce91c754ee..b75ddb780b 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -707,6 +707,7 @@ static const char * const builtin_maintenance_usage[] = { struct maintenance_opts { int auto_flag; + int quiet; }; static int maintenance_task_gc(struct maintenance_opts *opts) @@ -718,6 +719,10 @@ static int maintenance_task_gc(struct maintenance_opts *opts) if (opts->auto_flag) strvec_push(&child.args, "--auto"); + if (opts->quiet) + strvec_push(&child.args, "--quiet"); + else + strvec_push(&child.args, "--no-quiet"); close_object_store(the_repository->objects); return run_command(&child); @@ -734,6 +739,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) struct option builtin_maintenance_options[] = { OPT_BOOL(0, "auto", &opts.auto_flag, N_("run tasks based on the state of the repository")), + OPT_BOOL(0, "quiet", &opts.quiet, + N_("do not report progress or other information over stderr")), OPT_END() }; @@ -743,6 +750,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) usage_with_options(builtin_maintenance_usage, builtin_maintenance_options); + opts.quiet = !isatty(2); + argc = parse_options(argc, argv, prefix, builtin_maintenance_options, builtin_maintenance_usage, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 14aa691f19..47d512464c 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -11,11 +11,16 @@ test_expect_success 'help text' ' test_i18ngrep "invalid subcommand: barf" err ' -test_expect_success 'run [--auto]' ' - GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run && - GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto && - test_subcommand git gc /dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" \ + git maintenance run --auto 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-no-quiet.txt" \ + git maintenance run --no-quiet 2>/dev/null && + test_subcommand git gc --quiet X-Patchwork-Id: 11721049 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E7B4B722 for ; Tue, 18 Aug 2020 14:23:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CA850207DA for ; Tue, 18 Aug 2020 14:23:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HzD2gurr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726862AbgHROXg (ORCPT ); Tue, 18 Aug 2020 10:23:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50606 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726605AbgHROXP (ORCPT ); Tue, 18 Aug 2020 10:23:15 -0400 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E86A9C061344 for ; Tue, 18 Aug 2020 07:23:14 -0700 (PDT) Received: by mail-wm1-x343.google.com with SMTP id p14so16404267wmg.1 for ; Tue, 18 Aug 2020 07:23:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=haPdcuSHV4uqg9oVGnsbasRCEzOuPwsFJKkmpIY23OQ=; b=HzD2gurr2xcL1/yc33QTeMjcXpEna7J6ccZN7hg/jwD4I16Wm9aIdFv9o2GGLIU01b CLNeNkNd2/mjqYFzY6GDZdTVVxBY2TVpn3rf5bY22zp6LB0qJ9ByAI03tszZHPyyvUne W0WSe5JmS+fXdhNjy1w5N1xZO+MmvfDK7eqYJJN1tmSzvm7e1B8spaNnIA6Qo+zIVxla EqANQKcPF9ByGb33FZZwd+2lzrDlLdSDOj05fwimMtsf2pMYAy12pb6oi+ElZmZXBQ68 ecELvloAA+cL8xf94PJ6CabJdhBB+YzSGzLqr+7/D3A8Dy7nOHkU/DdStgmkRf0NNqGD 8ssQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=haPdcuSHV4uqg9oVGnsbasRCEzOuPwsFJKkmpIY23OQ=; b=LyeEKq811+9xt3ZAvTucvZheMtF40PnK3TNiEduIAVox/xn2dqqecByvrSa+ERIhbz 6jkShvy35VRVjhUSgH5RxQkXHmu6+swy1IMOndJcqH7pJt5sQOKCPiyZCtcW6ROWJvsn 32Da6DIzeoXT0SoG5EAb2Xl2iHGhuhJW+6EBCXMkWJb8Nhr95UDeOCsYt0WvHsjF5+80 mgeelfSYKB41y8k9xqqMKX5QDkohZ0yiS0lL7L4GxUI44JJ+IfrH0O00B0/+q+WfAjIa flR5Ez9/3u3lYAgyfFHEEVAlS9zgKb9ZSik+DJH0wtk3WzJIaNxhfHI5v/T9MO9p6JLV KlbA== X-Gm-Message-State: AOAM533M2eVwTs9oHWXIYQuLyLRBrBFXAJ06vYnWdxuHs8/dzM8mnh9H fYicDKsG5XCeQbTUjg687Nzo6toGTaM= X-Google-Smtp-Source: ABdhPJzZh9KysedY6G+Q/EBu8RPiz12SI7whXKDzkKNhREZG15Tmn+KCQbvfhqABLgdOwdDaUqG3zQ== X-Received: by 2002:a7b:cf08:: with SMTP id l8mr133041wmg.183.1597760593340; Tue, 18 Aug 2020 07:23:13 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d11sm35213460wrw.77.2020.08.18.07.23.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:12 -0700 (PDT) Message-Id: <91741a0cfc44009f6792c8d3d8d7cc086b1a0669.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:00 +0000 Subject: [PATCH v2 03/11] maintenance: replace run_auto_gc() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The run_auto_gc() method is used in several places to trigger a check for repo maintenance after some Git commands, such as 'git commit' or 'git fetch'. To allow for extra customization of this maintenance activity, replace the 'git gc --auto [--quiet]' call with one to 'git maintenance run --auto [--quiet]'. As we extend the maintenance builtin with other steps, users will be able to select different maintenance activities. Rename run_auto_gc() to run_auto_maintenance() to be clearer what is happening on this call, and to expose all callers in the current diff. Rewrite the method to use a struct child_process to simplify the calls slightly. Since 'git fetch' already allows disabling the 'git gc --auto' subprocess, add an equivalent option with a different name to be more descriptive of the new behavior: '--[no-]maintenance'. Update the documentation to include these options at the same time. Signed-off-by: Derrick Stolee --- Documentation/fetch-options.txt | 6 ++++-- Documentation/git-clone.txt | 6 +++--- builtin/am.c | 2 +- builtin/commit.c | 2 +- builtin/fetch.c | 6 ++++-- builtin/merge.c | 2 +- builtin/rebase.c | 4 ++-- run-command.c | 16 +++++++--------- run-command.h | 2 +- t/t5510-fetch.sh | 2 +- 10 files changed, 25 insertions(+), 23 deletions(-) diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index 6e2a160a47..495bc8ab5a 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -86,9 +86,11 @@ ifndef::git-pull[] Allow several and arguments to be specified. No s may be specified. +--[no-]auto-maintenance:: --[no-]auto-gc:: - Run `git gc --auto` at the end to perform garbage collection - if needed. This is enabled by default. + Run `git maintenance run --auto` at the end to perform automatic + repository maintenance if needed. (`--[no-]auto-gc` is a synonym.) + This is enabled by default. --[no-]write-commit-graph:: Write a commit-graph after fetching. This overrides the config diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt index c898310099..097e6a86c5 100644 --- a/Documentation/git-clone.txt +++ b/Documentation/git-clone.txt @@ -78,9 +78,9 @@ repository using this option and then delete branches (or use any other Git command that makes any existing commit unreferenced) in the source repository, some objects may become unreferenced (or dangling). These objects may be removed by normal Git operations (such as `git commit`) -which automatically call `git gc --auto`. (See linkgit:git-gc[1].) -If these objects are removed and were referenced by the cloned repository, -then the cloned repository will become corrupt. +which automatically call `git maintenance run --auto`. (See +linkgit:git-maintenance[1].) If these objects are removed and were referenced +by the cloned repository, then the cloned repository will become corrupt. + Note that running `git repack` without the `--local` option in a repository cloned with `--shared` will copy objects from the source repository into a pack diff --git a/builtin/am.c b/builtin/am.c index dd4e6c2d9b..68e9d17379 100644 --- a/builtin/am.c +++ b/builtin/am.c @@ -1795,7 +1795,7 @@ static void am_run(struct am_state *state, int resume) if (!state->rebasing) { am_destroy(state); close_object_store(the_repository->objects); - run_auto_gc(state->quiet); + run_auto_maintenance(state->quiet); } } diff --git a/builtin/commit.c b/builtin/commit.c index 69ac78d5e5..f9b0a0c05d 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1702,7 +1702,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix) git_test_write_commit_graph_or_die(); repo_rerere(the_repository, 0); - run_auto_gc(quiet); + run_auto_maintenance(quiet); run_commit_hook(use_editor, get_index_file(), "post-commit", NULL); if (amend && !no_post_rewrite) { commit_post_rewrite(the_repository, current_head, &oid); diff --git a/builtin/fetch.c b/builtin/fetch.c index c49f0e9752..c8b9366d3c 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = { OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"), N_("report that we have only objects reachable from this object")), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options), + OPT_BOOL(0, "auto-maintenance", &enable_auto_gc, + N_("run 'maintenance --auto' after fetching")), OPT_BOOL(0, "auto-gc", &enable_auto_gc, - N_("run 'gc --auto' after fetching")), + N_("run 'maintenance --auto' after fetching")), OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates, N_("check for forced-updates on all updated branches")), OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph, @@ -1882,7 +1884,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) close_object_store(the_repository->objects); if (enable_auto_gc) - run_auto_gc(verbosity < 0); + run_auto_maintenance(verbosity < 0); return result; } diff --git a/builtin/merge.c b/builtin/merge.c index 7da707bf55..c068e73037 100644 --- a/builtin/merge.c +++ b/builtin/merge.c @@ -457,7 +457,7 @@ static void finish(struct commit *head_commit, * user should see them. */ close_object_store(the_repository->objects); - run_auto_gc(verbosity < 0); + run_auto_maintenance(verbosity < 0); } } if (new_head && show_diffstat) { diff --git a/builtin/rebase.c b/builtin/rebase.c index dadb52fa92..9ffba9009d 100644 --- a/builtin/rebase.c +++ b/builtin/rebase.c @@ -728,10 +728,10 @@ static int finish_rebase(struct rebase_options *opts) apply_autostash(state_dir_path("autostash", opts)); close_object_store(the_repository->objects); /* - * We ignore errors in 'gc --auto', since the + * We ignore errors in 'git maintenance run --auto', since the * user should see them. */ - run_auto_gc(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE))); + run_auto_maintenance(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE))); if (opts->type == REBASE_MERGE) { struct replay_opts replay = REPLAY_OPTS_INIT; diff --git a/run-command.c b/run-command.c index cc9c3296ba..2ee59acdc8 100644 --- a/run-command.c +++ b/run-command.c @@ -1866,15 +1866,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task, return result; } -int run_auto_gc(int quiet) +int run_auto_maintenance(int quiet) { - struct strvec argv_gc_auto = STRVEC_INIT; - int status; + struct child_process maint = CHILD_PROCESS_INIT; - strvec_pushl(&argv_gc_auto, "gc", "--auto", NULL); - if (quiet) - strvec_push(&argv_gc_auto, "--quiet"); - status = run_command_v_opt(argv_gc_auto.v, RUN_GIT_CMD); - strvec_clear(&argv_gc_auto); - return status; + maint.git_cmd = 1; + strvec_pushl(&maint.args, "maintenance", "run", "--auto", NULL); + strvec_push(&maint.args, quiet ? "--quiet" : "--no-quiet"); + + return run_command(&maint); } diff --git a/run-command.h b/run-command.h index 8b9bfaef16..6472b38bde 100644 --- a/run-command.h +++ b/run-command.h @@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args); /* * Trigger an auto-gc */ -int run_auto_gc(int quiet); +int run_auto_maintenance(int quiet); #define RUN_COMMAND_NO_STDIN 1 #define RUN_GIT_CMD 2 /*If this is to be git sub-command */ diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh index a66dbe0bde..9850ecde5d 100755 --- a/t/t5510-fetch.sh +++ b/t/t5510-fetch.sh @@ -919,7 +919,7 @@ test_expect_success 'fetching with auto-gc does not lock up' ' git config fetch.unpackLimit 1 && git config gc.autoPackLimit 1 && git config gc.autoDetach false && - GIT_ASK_YESNO="$D/askyesno" git fetch >fetch.out 2>&1 && + GIT_ASK_YESNO="$D/askyesno" git fetch --verbose >fetch.out 2>&1 && test_i18ngrep "Auto packing the repository" fetch.out && ! grep "Should I try again" fetch.out ) From patchwork Tue Aug 18 14:23:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721051 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 92233722 for ; Tue, 18 Aug 2020 14:23:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7AD6F207FF for ; Tue, 18 Aug 2020 14:23:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qrWjN+5o" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726868AbgHROXn (ORCPT ); Tue, 18 Aug 2020 10:23:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50614 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726612AbgHROXQ (ORCPT ); Tue, 18 Aug 2020 10:23:16 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AAFA2C061345 for ; Tue, 18 Aug 2020 07:23:15 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id f7so18478874wrw.1 for ; Tue, 18 Aug 2020 07:23:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=esQK9tq9GqpY+vSwmwnemBsEpJ5f4uldkNOirXjhIgI=; b=qrWjN+5ogl8cDZriT7ERC4fE5lKmiP+Fc4pSK404zOV16eAsePP6FX1aP36yg/dmjW okrvGbF/48nwp1vDWYc1SGwkyzJqrD/VXlj6ohr9pY8badaLbY9EOxi8G3Lmo97dPRt7 XROG6oazA2F+X+niyIL9vg4pxmnCi/gnXHDboIORtKy4dvtHkHB4VIm+YX2akQr6uqtI bGLqldOoOOf34iCbVV4UojrWKUPImjBF++GOwMNj+i8nqC5LcNRswNCEeOlE9B71AMz/ aEpJbmgTzCpq7Os8naaCAJUEd7mQ6uoE9IDJa63FouOpFzuNy4yVZ/6FiqGQqYIIqZAD uyzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=esQK9tq9GqpY+vSwmwnemBsEpJ5f4uldkNOirXjhIgI=; b=pBOlquCLQwJAbmjgaMO20QOssJ4xmvNlSIu1UBuoaKXfQS+pbxyRpzKa8cHquJC9HR 35Arc5fW5XN03ChsPsmeQWkWvctnhjUvRiYogPuzbLAt7nYyZMT57iF6fsgrlx2cP7la 2Awbx6dSkWFZT9CUNHVLAocG+c8g6NY3NX98KxZss6PkfQmDZ7jtWhIDIdzms+gyvbUV zr1NBd50JZabftrwQlIVm1fk6BpsKwpLHYPB5KQqHvSIbs4VrL3RAGb2+eTE5EsXnHD9 gGlIvCIRq0FP+Btf2JlI038y6CKoo0YVo1pzhPTD3gbhttTzCNVLVEnHzrGuRS0P12sj HgFQ== X-Gm-Message-State: AOAM533i3uSOhoS8ajKsyrEYazHZVuvdNVKiXirEe53eqjcVvq9eT8dM LILzIzQS26QsDDAy11NiCLmGsYjweS8= X-Google-Smtp-Source: ABdhPJygWgv2CfmwwKOEPU4qHQ8lgcafnYd36OGVXqepdkXBD5j1gNj3y2kgsCeQ53NwYT+l77HJGg== X-Received: by 2002:adf:d84f:: with SMTP id k15mr19666480wrl.176.1597760594256; Tue, 18 Aug 2020 07:23:14 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 32sm38180934wrh.18.2020.08.18.07.23.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:13 -0700 (PDT) Message-Id: <1db3b96280d9bf7908b7dc9fa13b80c445164a99.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:01 +0000 Subject: [PATCH v2 04/11] maintenance: initialize task array Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee In anticipation of implementing multiple maintenance tasks inside the 'maintenance' builtin, use a list of structs to describe the work to be done. The struct maintenance_task stores the name of the task (as given by a future command-line argument) along with a function pointer to its implementation and a boolean for whether the step is enabled. A list these structs are initialized with the full list of implemented tasks along with a default order. For now, this list only contains the "gc" task. This task is also the only task enabled by default. The run subcommand will return a nonzero exit code if any task fails. However, it will attempt all tasks in its loop before returning with the failure. Also each failed task will send an error message. Helped-by: Taylor Blau Helped-by: Junio C Hamano Signed-off-by: Derrick Stolee --- builtin/gc.c | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/builtin/gc.c b/builtin/gc.c index b75ddb780b..946d871d54 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -728,9 +728,45 @@ static int maintenance_task_gc(struct maintenance_opts *opts) return run_command(&child); } +typedef int maintenance_task_fn(struct maintenance_opts *opts); + +struct maintenance_task { + const char *name; + maintenance_task_fn *fn; + unsigned enabled:1; +}; + +enum maintenance_task_label { + TASK_GC, + + /* Leave as final value */ + TASK__COUNT +}; + +static struct maintenance_task tasks[] = { + [TASK_GC] = { + "gc", + maintenance_task_gc, + 1, + }, +}; + static int maintenance_run(struct maintenance_opts *opts) { - return maintenance_task_gc(opts); + int i; + int result = 0; + + for (i = 0; i < TASK__COUNT; i++) { + if (!tasks[i].enabled) + continue; + + if (tasks[i].fn(opts)) { + error(_("task '%s' failed"), tasks[i].name); + result = 1; + } + } + + return result; } int cmd_maintenance(int argc, const char **argv, const char *prefix) From patchwork Tue Aug 18 14:23:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721055 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1034722 for ; Tue, 18 Aug 2020 14:23:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A374C207DA for ; Tue, 18 Aug 2020 14:23:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="iIB/HaUr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726880AbgHROXr (ORCPT ); Tue, 18 Aug 2020 10:23:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50626 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726617AbgHROXU (ORCPT ); Tue, 18 Aug 2020 10:23:20 -0400 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BFE80C061346 for ; Tue, 18 Aug 2020 07:23:16 -0700 (PDT) Received: by mail-wm1-x343.google.com with SMTP id x5so16399487wmi.2 for ; Tue, 18 Aug 2020 07:23:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=M+gCXYRrWbptNZftxgMrKXm1wDae0qyaoex08Gzk68k=; b=iIB/HaUrE7onpPf+5XZIdBj1k5D+NNg3epji1GOdqkq31UA0/af3KqzqcmF8xaWZLv 8HpePqLsTPczUkV4myYqwu1LMH/xTrXhAYjYWwI+gr/gH+aaTes6MQJSy16GlJoX0OA7 8NKZzDzS6u6wXV2vDJW9MVuTEfS/EgrFUkv/1fmisXgVtHbVX7PqfncukKt/DE6YX/r8 w56w5mXrpeVGBR6sUxs9z6CgG6nKhuIERvt+K5wn5+Py4f6Zb6CjZYt3a8xK6WuzbrjK azqJpALbaDXNWAVTgZo8ak3U5VtESCqgYskVOw/imJicsXMjlvVSTdnHQD1WstieFjpy VhpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=M+gCXYRrWbptNZftxgMrKXm1wDae0qyaoex08Gzk68k=; b=OjCqdBJuBOz+p37b1GzYDovb4oqQyNfrRySZenjR/QxJ/imWMsqgGMjpwDV8rBgghG iZwHQEz8PtFyUFg7R384WeOwRxIocvxWdhTTyVOia1FZrYVb6/gDO0EodYnx085Z0pd/ S/m9TQO3a29Xzcnn/uJEd7FvDoPxWGPmdU+apT+w0a7KNCB+2Dr3q4nQYr1EnT8LhEww cZOgyCuo9FO6Md3nbhO64E6MS4G0ywgpJ7JUhlv85PVRlkvQ3pBg4VX4h+RomL4ws9VF A/4JKghdezvyJoct0NgoqPwfchzc1fbn9CQjKOqprVmyfW7Ek91INJPoJHGxEnZYe0oB bWyQ== X-Gm-Message-State: AOAM531GyOI4fWSm5i9ZMfSdpV3cLQIEYL7yoddZUGbj9zdSiY9gA61K Sz5P+j2Qt5IloRYeKh1FJkZPcXGl4HA= X-Google-Smtp-Source: ABdhPJx2qt8nuqMM9SoARG8EsZwBP8fjr70w61kTueW7l7d1xif/qZXE4vLnZOjJj5y/JUps+x1cvw== X-Received: by 2002:a05:600c:2189:: with SMTP id e9mr152420wme.171.1597760595174; Tue, 18 Aug 2020 07:23:15 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p6sm167085wmg.0.2020.08.18.07.23.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:14 -0700 (PDT) Message-Id: <50b457fd57aef4e9ac6a15549561936dc962ef36.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:02 +0000 Subject: [PATCH v2 05/11] maintenance: add commit-graph task Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The first new task in the 'git maintenance' builtin is the 'commit-graph' job. It is based on the sequence of events in the 'commit-graph' job in Scalar [1]. This sequence is as follows: 1. git commit-graph write --reachable --split 2. git commit-graph verify --shallow 3. If the verify succeeds, stop. 4. Delete the commit-graph-chain file. 5. git commit-graph write --reachable --split By writing an incremental commit-graph file using the "--split" option we minimize the disruption from this operation. The default behavior is to merge layers until the new "top" layer is less than half the size of the layer below. This provides quick writes most of the time, with the longer writes following a power law distribution. Most importantly, concurrent Git processes only look at the commit-graph-chain file for a very short amount of time, so they will verly likely not be holding a handle to the file when we try to replace it. (This only matters on Windows.) If a concurrent process reads the old commit-graph-chain file, but our job expires some of the .graph files before they can be read, then those processes will see a warning message (but not fail). This could be avoided by a future update to use the --expire-time argument when writing the commit-graph. By using 'git commit-graph verify --shallow' we can ensure that the file we just wrote is valid. This is an extra safety precaution that is faster than our 'write' subcommand. In the rare situation that the newest layer of the commit-graph is corrupt, we can "fix" the corruption by deleting the commit-graph-chain file and rewrite the full commit-graph as a new one-layer commit graph. This does not completely prevent _that_ file from being corrupt, but it does recompute the commit-graph by parsing commits from the object database. In our use of this step in Scalar and VFS for Git, we have only seen this issue arise because our microsoft/git fork reverted 43d3561 ("commit-graph write: don't die if the existing graph is corrupt" 2019-03-25) for a while to keep commit-graph writes very fast. We dropped the revert when updating to v2.23.0. The verify still has potential for catching corrupt data across the layer boundary: if the new file has commit X with parent Y in an old file but the commit ID for Y in the old file had a bitswap, then we will notice that in the 'verify' command. [1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/CommitGraphStep.cs Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 11 ++++++ builtin/gc.c | 63 +++++++++++++++++++++++++++++++ commit-graph.c | 8 ++-- commit-graph.h | 1 + t/t7900-maintenance.sh | 2 + 5 files changed, 81 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 04fa0fe329..c816fa1dcd 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -35,6 +35,17 @@ run:: TASKS ----- +commit-graph:: + The `commit-graph` job updates the `commit-graph` files incrementally, + then verifies that the written data is correct. If the new layer has an + issue, then the chain file is removed and the `commit-graph` is + rewritten from scratch. ++ +The incremental write is safe to run alongside concurrent Git processes +since it will not expire `.graph` files that were in the previous +`commit-graph-chain` file. They will be deleted by a later run based on +the expiration delay. + gc:: Clean up unnecessary files and optimize the local repository. "GC" stands for "garbage collection," but this task performs many diff --git a/builtin/gc.c b/builtin/gc.c index 946d871d54..4f9352b9d0 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -710,6 +710,64 @@ struct maintenance_opts { int quiet; }; +static int run_write_commit_graph(struct maintenance_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_pushl(&child.args, "commit-graph", "write", + "--split", "--reachable", NULL); + + if (opts->quiet) + strvec_push(&child.args, "--no-progress"); + + return !!run_command(&child); +} + +static int run_verify_commit_graph(struct maintenance_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_pushl(&child.args, "commit-graph", "verify", + "--shallow", NULL); + + if (opts->quiet) + strvec_push(&child.args, "--no-progress"); + + return !!run_command(&child); +} + +static int maintenance_task_commit_graph(struct maintenance_opts *opts) +{ + struct repository *r = the_repository; + char *chain_path; + + close_object_store(r->objects); + if (run_write_commit_graph(opts)) { + error(_("failed to write commit-graph")); + return 1; + } + + if (!run_verify_commit_graph(opts)) + return 0; + + warning(_("commit-graph verify caught error, rewriting")); + + chain_path = get_commit_graph_chain_filename(r->objects->odb); + if (unlink(chain_path)) { + UNLEAK(chain_path); + die(_("failed to remove commit-graph at %s"), chain_path); + } + free(chain_path); + + if (!run_write_commit_graph(opts)) + return 0; + + error(_("failed to rewrite commit-graph")); + return 1; +} + static int maintenance_task_gc(struct maintenance_opts *opts) { struct child_process child = CHILD_PROCESS_INIT; @@ -738,6 +796,7 @@ struct maintenance_task { enum maintenance_task_label { TASK_GC, + TASK_COMMIT_GRAPH, /* Leave as final value */ TASK__COUNT @@ -749,6 +808,10 @@ static struct maintenance_task tasks[] = { maintenance_task_gc, 1, }, + [TASK_COMMIT_GRAPH] = { + "commit-graph", + maintenance_task_commit_graph, + }, }; static int maintenance_run(struct maintenance_opts *opts) diff --git a/commit-graph.c b/commit-graph.c index 1af68c297d..9705d237e4 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -172,7 +172,7 @@ static char *get_split_graph_filename(struct object_directory *odb, oid_hex); } -static char *get_chain_filename(struct object_directory *odb) +char *get_commit_graph_chain_filename(struct object_directory *odb) { return xstrfmt("%s/info/commit-graphs/commit-graph-chain", odb->path); } @@ -521,7 +521,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, struct stat st; struct object_id *oids; int i = 0, valid = 1, count; - char *chain_name = get_chain_filename(odb); + char *chain_name = get_commit_graph_chain_filename(odb); FILE *fp; int stat_res; @@ -1619,7 +1619,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } if (ctx->split) { - char *lock_name = get_chain_filename(ctx->odb); + char *lock_name = get_commit_graph_chain_filename(ctx->odb); hold_lock_file_for_update_mode(&lk, lock_name, LOCK_DIE_ON_ERROR, 0444); @@ -1996,7 +1996,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx) if (ctx->split_opts && ctx->split_opts->expire_time) expire_time = ctx->split_opts->expire_time; if (!ctx->split) { - char *chain_file_name = get_chain_filename(ctx->odb); + char *chain_file_name = get_commit_graph_chain_filename(ctx->odb); unlink(chain_file_name); free(chain_file_name); ctx->num_commit_graphs_after = 0; diff --git a/commit-graph.h b/commit-graph.h index 28f89cdf3e..3c202748c3 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -25,6 +25,7 @@ struct commit; struct bloom_filter_settings; char *get_commit_graph_filename(struct object_directory *odb); +char *get_commit_graph_chain_filename(struct object_directory *odb); int open_commit_graph(const char *graph_file, int *fd, struct stat *st); /* diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 47d512464c..c0c4e6846e 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -4,6 +4,8 @@ test_description='git maintenance builtin' . ./test-lib.sh +GIT_TEST_COMMIT_GRAPH=0 + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance run" err && From patchwork Tue Aug 18 14:23:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721053 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C048F722 for ; Tue, 18 Aug 2020 14:23:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9CDD0207D3 for ; Tue, 18 Aug 2020 14:23:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="hZuing9T" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726803AbgHROXp (ORCPT ); Tue, 18 Aug 2020 10:23:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50628 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726636AbgHROXU (ORCPT ); Tue, 18 Aug 2020 10:23:20 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C80F3C061347 for ; Tue, 18 Aug 2020 07:23:17 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id g75so17174338wme.4 for ; Tue, 18 Aug 2020 07:23:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JHjJU4VSqmYw0QkLeFL1p9zHlz8jMEG2I2KKfue5Smg=; b=hZuing9TJu0ZZtsw211wk+X9TiYUcW/2bcJ2JUsfQZnZqpPoMrZLROQ4f23SvgC/+d 5mUVHU4pjQ2QWoaS5gh573bA+XeVZ1jjxiCoo2v1wcIsDCXfGbH8ZqnpKzM4XyBkHEfm vrJHpHKr0NwBC4nfOETXGrl0ckdGZvkLTrMY6z6cU+Ms/aDtB4zRnlRMoCcalI+wVk66 v7R2uh0arr5Jkqu/QsK7BE6BzUndlzL5OFwkeQ3PQMG1+fCSew998MLg5ErtWmcsMWiz 4cGzprn8tBgVKqCAdZHgsOxn11/jAkL11mVH3o1QD94UK6hpuqma4H7fiJ8VJWSzzsjA YREw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JHjJU4VSqmYw0QkLeFL1p9zHlz8jMEG2I2KKfue5Smg=; b=V/gpqWB5tWPyTYA78n3eea/Sauu2f66tImuG0zX7zlkexNT90QePCT7cjy7BibQb6/ 1Ju/rzexL51B6dRrbFE2ObCeMhJvGRWadHYeoYEeEvR0w9MOLqVDj2Pqr6ARGnomzmUT 6XHqS2Nk1kBGJ6LbMYhkH9RsWl5Tw6978jYEjIvdrwAgGRpcnAdjIm5dsd1vu5yxyRQv TG8YrgvWTHYjojKr+xXc5J2kOpYJFBsr7HVkYdsCkHVs+lxGxTce2mfRUyrPdjSlPugC R69yCkGYaLmOTwmrk9C8cBNoQqfAkm9DAsQq1sihH9KyWEgTS8CDh5YNLym2shH1v1JX SMTA== X-Gm-Message-State: AOAM530dR48efcfZ81o5KAlTlZji8QYy5NPuDvrO7Gp4Mxq7wB7QLLDh 5hO5VAiVNehcKMsB3Y6UiW+on9Lyp2M= X-Google-Smtp-Source: ABdhPJyugO2uH3ln7qxa9vbQK8TRNOxFHGHr4GpeoRWah8bpD4+fyT3RaoR6KZaIxFUnX9TFmtlUKg== X-Received: by 2002:a05:600c:2146:: with SMTP id v6mr161023wml.87.1597760596178; Tue, 18 Aug 2020 07:23:16 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v11sm34437150wrr.10.2020.08.18.07.23.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:15 -0700 (PDT) Message-Id: <85268bd53ef7f4e7b3d97a8ae091290e8e74441d.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:03 +0000 Subject: [PATCH v2 06/11] maintenance: add --task option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee A user may want to only run certain maintenance tasks in a certain order. Add the --task= option, which allows a user to specify an ordered list of tasks to run. These cannot be run multiple times, however. Here is where our array of maintenance_task pointers becomes critical. We can sort the array of pointers based on the task order, but we do not want to move the struct data itself in order to preserve the hashmap references. We use the hashmap to match the --task= arguments into the task struct data. Keep in mind that the 'enabled' member of the maintenance_task struct is a placeholder for a future 'maintenance..enabled' config option. Thus, we use the 'enabled' member to specify which tasks are run when the user does not specify any --task= arguments. The 'enabled' member should be ignored if --task= appears. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 8 +++- builtin/gc.c | 61 +++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 27 ++++++++++++++ 3 files changed, 92 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index c816fa1dcd..cf59eb258c 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -30,7 +30,9 @@ SUBCOMMANDS ----------- run:: - Run one or more maintenance tasks. + Run one or more maintenance tasks. If one or more `--task=` + options are specified, then those tasks are run in the provided + order. Otherwise, only the `gc` task is run. TASKS ----- @@ -66,6 +68,10 @@ OPTIONS --quiet:: Do not report progress or other information over `stderr`. +--task=:: + If this option is specified one or more times, then only run the + specified tasks in the specified order. + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 4f9352b9d0..0231d2f8c1 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -791,7 +791,9 @@ typedef int maintenance_task_fn(struct maintenance_opts *opts); struct maintenance_task { const char *name; maintenance_task_fn *fn; - unsigned enabled:1; + unsigned enabled:1, + selected:1; + int selected_order; }; enum maintenance_task_label { @@ -814,13 +816,32 @@ static struct maintenance_task tasks[] = { }, }; +static int compare_tasks_by_selection(const void *a_, const void *b_) +{ + const struct maintenance_task *a, *b; + + a = (const struct maintenance_task *)&a_; + b = (const struct maintenance_task *)&b_; + + return b->selected_order - a->selected_order; +} + static int maintenance_run(struct maintenance_opts *opts) { - int i; + int i, found_selected = 0; int result = 0; + for (i = 0; !found_selected && i < TASK__COUNT; i++) + found_selected = tasks[i].selected; + + if (found_selected) + QSORT(tasks, TASK__COUNT, compare_tasks_by_selection); + for (i = 0; i < TASK__COUNT; i++) { - if (!tasks[i].enabled) + if (found_selected && !tasks[i].selected) + continue; + + if (!found_selected && !tasks[i].enabled) continue; if (tasks[i].fn(opts)) { @@ -832,6 +853,37 @@ static int maintenance_run(struct maintenance_opts *opts) return result; } +static int task_option_parse(const struct option *opt, + const char *arg, int unset) +{ + int i, num_selected = 0; + struct maintenance_task *task = NULL; + + BUG_ON_OPT_NEG(unset); + + for (i = 0; i < TASK__COUNT; i++) { + num_selected += tasks[i].selected; + if (!strcasecmp(tasks[i].name, arg)) { + task = &tasks[i]; + } + } + + if (!task) { + error(_("'%s' is not a valid task"), arg); + return 1; + } + + if (task->selected) { + error(_("task '%s' cannot be selected multiple times"), arg); + return 1; + } + + task->selected = 1; + task->selected_order = num_selected + 1; + + return 0; +} + int cmd_maintenance(int argc, const char **argv, const char *prefix) { struct maintenance_opts opts; @@ -840,6 +892,9 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) N_("run tasks based on the state of the repository")), OPT_BOOL(0, "quiet", &opts.quiet, N_("do not report progress or other information over stderr")), + OPT_CALLBACK_F(0, "task", NULL, N_("task"), + N_("run a specific task"), + PARSE_OPT_NONEG, task_option_parse), OPT_END() }; diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index c0c4e6846e..792765aff7 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -25,4 +25,31 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet ' ' + GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ + git maintenance run --task=commit-graph 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-gc.txt" \ + git maintenance run --task=gc 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ + git maintenance run --task=commit-graph 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-both.txt" \ + git maintenance run --task=commit-graph --task=gc 2>/dev/null && + test_subcommand ! git gc --quiet err && + test_i18ngrep "is not a valid task" err +' + +test_expect_success 'run --task duplicate' ' + test_must_fail git maintenance run --task=gc --task=gc 2>err && + test_i18ngrep "cannot be selected multiple times" err +' + test_done From patchwork Tue Aug 18 14:23:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721065 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DA5BD1392 for ; Tue, 18 Aug 2020 14:24:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C1E4B207FF for ; Tue, 18 Aug 2020 14:24:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ax0r2OC+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726932AbgHROXw (ORCPT ); Tue, 18 Aug 2020 10:23:52 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50640 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726681AbgHROXY (ORCPT ); Tue, 18 Aug 2020 10:23:24 -0400 Received: from mail-wm1-x341.google.com (mail-wm1-x341.google.com [IPv6:2a00:1450:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98F83C061348 for ; Tue, 18 Aug 2020 07:23:18 -0700 (PDT) Received: by mail-wm1-x341.google.com with SMTP id d190so16393358wmd.4 for ; Tue, 18 Aug 2020 07:23:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=NoV3mdz7adJNLz7R5IzVkTZi/pFBlVIFLmIKQrOHX3s=; b=ax0r2OC+T5H4yBVETo8GBMPP6hjmQqzyEzCEXp9ohgxmeMpqhcckB9LH6Fd3R1wCJ/ iuvdMMqgaOViIlXdmr9LOfRR7L8sQytMer0I+g/w2N1CedgxaBV6cRvbP0Hu0UfnOSVS XZnzeQuiH7iMk7oSjQ7N+bszr3xvLfnxiWKfgi4xyxH+TG4lHu1L90R9Ii/VCVL0Ethd Ewq2gXaPiwiFlNdjeMLP7HGtYLRee+YFYt3/QDMNSra61yi9mRVg0o1TSiMkSB1JGIb6 dgEIpTSdNjAd9KT1B1a3jT/98dmCfy5mn3uXyBuVFUi+u71Jg3mdGByR3bhHdRjSbS2R rB1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=NoV3mdz7adJNLz7R5IzVkTZi/pFBlVIFLmIKQrOHX3s=; b=gJAD4//fV9i89LOgaSWdvsypMNjm0Roe0K5O9uOiV7S6quwfT74/InKRgPkUL8DxjQ LH5y+dIPKIvxs+WNO0wD3botGkX+VpimRldyQ9LGDnTqQPWfwB0fY10cOCvdTZ27R2HL +jBNvlOZASmeqaabV9vlMf5ZQ0ikJ5W/3wZ2PZweB3yM/VMkx5nCEzoycOOfGstPMsLF 2+JrdnuD13r7V36dFBUNbhCS7hAMDKcWDU//mgWWCT9kZjn+MKhFP9LZopZFZjaRxz0V Y5xM2AAUBz0Ferpb+lrukFnp+/w9J2V7RlLXJLuT64uEV487TCpPJUQ4/SEc0FZ/A+aA dHbQ== X-Gm-Message-State: AOAM533PfYWtsBwnGEu1dANVXAqt/3QJz3PBZLgAlZjvMUhYGnDh52UU AUGMXOHICSB1FiZcZHXCSlrLL4vgzZ0= X-Google-Smtp-Source: ABdhPJzJqmbyuImoFogrQUbq3E5qcQxT2E1nkI5WVq05cHYldtpM1VT45DqYd3K+eX9yMOO9/hUu6A== X-Received: by 2002:a1c:4844:: with SMTP id v65mr152890wma.149.1597760597132; Tue, 18 Aug 2020 07:23:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 31sm35415925wrj.94.2020.08.18.07.23.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:16 -0700 (PDT) Message-Id: <6f86cfaa94cfeaf7a2af417991ca07e41b6b0c3d.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:04 +0000 Subject: [PATCH v2 07/11] maintenance: take a lock on the objects directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Performing maintenance on a Git repository involves writing data to the .git directory, which is not safe to do with multiple writers attempting the same operation. Ensure that only one 'git maintenance' process is running at a time by holding a file-based lock. Simply the presence of the .git/maintenance.lock file will prevent future maintenance. This lock is never committed, since it does not represent meaningful data. Instead, it is only a placeholder. If the lock file already exists, then fail silently. This will become very important later when we implement the 'fetch' task, as this is our stop-gap from creating a recursive process loop between 'git fetch' and 'git maintenance run'. Signed-off-by: Derrick Stolee --- builtin/gc.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/builtin/gc.c b/builtin/gc.c index 0231d2f8c1..a47bf64623 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -830,6 +830,25 @@ static int maintenance_run(struct maintenance_opts *opts) { int i, found_selected = 0; int result = 0; + struct lock_file lk; + struct repository *r = the_repository; + char *lock_path = xstrfmt("%s/maintenance", r->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) { + /* + * Another maintenance command is running. + * + * If --auto was provided, then it is likely due to a + * recursive process stack. Do not report an error in + * that case. + */ + if (!opts->auto_flag && !opts->quiet) + error(_("lock file '%s' exists, skipping maintenance"), + lock_path); + free(lock_path); + return 0; + } + free(lock_path); for (i = 0; !found_selected && i < TASK__COUNT; i++) found_selected = tasks[i].selected; @@ -850,6 +869,7 @@ static int maintenance_run(struct maintenance_opts *opts) } } + rollback_lock_file(&lk); return result; } From patchwork Tue Aug 18 14:23:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721057 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 017FD1575 for ; Tue, 18 Aug 2020 14:23:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DC9A5207DA for ; Tue, 18 Aug 2020 14:23:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KOa4FWvu" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726899AbgHROXv (ORCPT ); Tue, 18 Aug 2020 10:23:51 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726660AbgHROXY (ORCPT ); Tue, 18 Aug 2020 10:23:24 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89BB5C061349 for ; Tue, 18 Aug 2020 07:23:20 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id a14so18474642wra.5 for ; Tue, 18 Aug 2020 07:23:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=WCpVK7NB7Xk3fBXFwbSH1b/7+huidYMfIphjudO3fu0=; b=KOa4FWvuajXt3O6DKST+9SLABp4YHTqfzVGnlwx6ViffAoS0n3ZHfrUuFbDWiNPyHs vXvQ3eAKx+eoTLruOlCV1/8WxpadqveNHxjHgV/Oe+w2qzikHw6l2Ujqc88CtvZtm7P5 rC6ZG7XTwmpvB5F3FKww4MmOW57qtTVfObR8vxisZ4IOzW68OdjP6rQiIdE9iF6UoC6Y egBCfkiQllAIXutTPHCdWVQg46yMQ6duJjhwan6s4BYOdknNVTrp4D5A3PHlsCdjl7bp 7N91pFWP2D9zVFfE5ekdkEL2ghgu54CQLONgjQM/5c4Z1tFVKoFFD/ujkmaD82o3LIlj L4uw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=WCpVK7NB7Xk3fBXFwbSH1b/7+huidYMfIphjudO3fu0=; b=My18iZmYO9uYXVhLWQ9JQPhQJkGY52Z9keKrHUYo/QLFPqN3VjL9k5rDKJX6C/SCvk jf1btcOWT294/7XE/NqfOEto/9j+jcDgW64hrVAB6jd3kzSZWM+H1AHr3a50UR/u8bFE xr/4u2Xd/p675omA9jpYteqnkVMVlP4hY9nzlyeS+lRmUpWFfUOENlWkWHmXn/QwBqg8 SxlnnNoqc1FiEUCzkQgh+IVJIBDWXMIrlISTEO8o8olsHwySp9/+rN3RKRQGpY4YQvFo vD+6tALehs7W30FV9ALfqf8R7GnGSDwMOthKeaRMeHIq09OgtUFL/+cdUwl4W+klAiv7 UEmQ== X-Gm-Message-State: AOAM530+ohrvs71mx6dGhktmwYo4SLjO08VMAXLGLzbYBBy/HklAudrg ujRzgSgF4YvZ6zFNoSOdwRGCt37p4NM= X-Google-Smtp-Source: ABdhPJwXHMnkjlohutN9xmOTbTgZTPHt3fUAgIPzTIdeCuVFrbWD3DM23y+oa2RCCMPl7uRWufn4gQ== X-Received: by 2002:a5d:6348:: with SMTP id b8mr20199655wrw.362.1597760598122; Tue, 18 Aug 2020 07:23:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 68sm35237409wra.39.2020.08.18.07.23.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:17 -0700 (PDT) Message-Id: <5c0f9d69d18d51f19fcd76fca647c1df6711d70f.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:05 +0000 Subject: [PATCH v2 08/11] maintenance: create maintenance..enabled config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Currently, a normal run of "git maintenance run" will only run the 'gc' task, as it is the only one enabled. This is mostly for backwards- compatible reasons since "git maintenance run --auto" commands replaced previous "git gc --auto" commands after some Git processes. Users could manually run specific maintenance tasks by calling "git maintenance run --task=" directly. Allow users to customize which steps are run automatically using config. The 'maintenance..enabled' option then can turn on these other tasks (or turn off the 'gc' task). Signed-off-by: Derrick Stolee --- Documentation/config.txt | 2 ++ Documentation/config/maintenance.txt | 4 ++++ Documentation/git-maintenance.txt | 8 +++++--- builtin/gc.c | 19 +++++++++++++++++++ t/t7900-maintenance.sh | 8 ++++++++ 5 files changed, 38 insertions(+), 3 deletions(-) create mode 100644 Documentation/config/maintenance.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index ef0768b91a..2783b825f9 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -396,6 +396,8 @@ include::config/mailinfo.txt[] include::config/mailmap.txt[] +include::config/maintenance.txt[] + include::config/man.txt[] include::config/merge.txt[] diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt new file mode 100644 index 0000000000..370cbfb42f --- /dev/null +++ b/Documentation/config/maintenance.txt @@ -0,0 +1,4 @@ +maintenance..enabled:: + This boolean config option controls whether the maintenance task + with name `` is run when no `--task` option is specified. + By default, only `maintenance.gc.enabled` is true. diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index cf59eb258c..9af08c644f 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -30,9 +30,11 @@ SUBCOMMANDS ----------- run:: - Run one or more maintenance tasks. If one or more `--task=` - options are specified, then those tasks are run in the provided - order. Otherwise, only the `gc` task is run. + Run one or more maintenance tasks. If one or more `--task` options + are specified, then those tasks are run in that order. Otherwise, + the tasks are determined by which `maintenance..enabled` + config options are true. By default, only `maintenance.gc.enabled` + is true. TASKS ----- diff --git a/builtin/gc.c b/builtin/gc.c index a47bf64623..e66326ede9 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -873,6 +873,24 @@ static int maintenance_run(struct maintenance_opts *opts) return result; } +static void initialize_task_config(void) +{ + int i; + struct strbuf config_name = STRBUF_INIT; + for (i = 0; i < TASK__COUNT; i++) { + int config_value; + + strbuf_setlen(&config_name, 0); + strbuf_addf(&config_name, "maintenance.%s.enabled", + tasks[i].name); + + if (!git_config_get_bool(config_name.buf, &config_value)) + tasks[i].enabled = config_value; + } + + strbuf_release(&config_name); +} + static int task_option_parse(const struct option *opt, const char *arg, int unset) { @@ -925,6 +943,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) builtin_maintenance_options); opts.quiet = !isatty(2); + initialize_task_config(); argc = parse_options(argc, argv, prefix, builtin_maintenance_options, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 792765aff7..290abb7832 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -25,6 +25,14 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet .enabled' ' + git config maintenance.gc.enabled false && + git config maintenance.commit-graph.enabled true && + GIT_TRACE2_EVENT="$(pwd)/run-config.txt" git maintenance run 2>err && + test_subcommand ! git gc --quiet ' ' GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ git maintenance run --task=commit-graph 2>/dev/null && From patchwork Tue Aug 18 14:23:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721063 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 594FA1575 for ; Tue, 18 Aug 2020 14:24:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3FD1520786 for ; Tue, 18 Aug 2020 14:24:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="OhcaOkX7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726957AbgHROXy (ORCPT ); Tue, 18 Aug 2020 10:23:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50642 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726689AbgHROXY (ORCPT ); Tue, 18 Aug 2020 10:23:24 -0400 Received: from mail-wm1-x341.google.com (mail-wm1-x341.google.com [IPv6:2a00:1450:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9EE72C061389 for ; Tue, 18 Aug 2020 07:23:20 -0700 (PDT) Received: by mail-wm1-x341.google.com with SMTP id k8so17199154wma.2 for ; Tue, 18 Aug 2020 07:23:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=xNVqN3VkYN9KqK6raUGF/QrF7C9o4aABkMukeCp8YFU=; b=OhcaOkX7c1Ns6kcXvyb/cBU/7VAs27Ar4LNMaOiMLZZm7e0tIABx8zZkmOvqkiWMBO oN98nzEJ3wB+8FGYZUmSqNHrpb1NeJdug9Msx10mP8mVH24XN6GOyh5MbSt+4bTlX4Jw a6jlJG8I72QYlp98+FsRN8DGq592CRXnYYf+zT06/aEuoE03AER0DjYPZW7fxyJtBLAS C/qKVdOBfAkdMBhv5LTyk261Q2xWWIVBhAz5yIz0V/WjyzyutNGHinsSgB9PWDbvD37y m8qnJQU24uBcZ+RfOlyhVkMFR5D4x2rmCd/V2gN+VtsiHUg9IdxWtoJ7f8n7RAJp7oY2 PI5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=xNVqN3VkYN9KqK6raUGF/QrF7C9o4aABkMukeCp8YFU=; b=MPq4CSmu7XNxC44T2eNCIFu8AMvJPNPmZDc5UQnjaUguIUwj/lYp+tq086njjmf3ar DCVdhE7MA1jSp30Cgz4Mj2R6JNKtejEJHgnQ2nQ8DoKc2SL4PmW18Ll4ZSu2bR/9E84N Qbw+nXa4WTj4nQbMj6yk80D3qV92Q8+nVPTtdKWcRskiFMB3bmLfaq+0QujHe+3MdyEb xhVu6eo9TwhzyzbHif96vVJAiIIrnirPHkk7ZLmKGcNeylCyHbJqFwJtmM9XOOGTXJiY ORfrFYcnq1b7Qq551ATd2Rp+mP9iTfpn45Ia11VNlKodOLmMlVZEsuYbbe2lBFbUxdKk frhg== X-Gm-Message-State: AOAM533pwOpa7PyKCfgVyliErJPxZvJFrcl/sXB4Kjah1wusWdDy8DuE jVazrQHLaKi+Jvn6u4PdrP/ByKNC5k8= X-Google-Smtp-Source: ABdhPJwE2WgiAoteS3I1EI78DnkY0sKUZRqXRehTFJ2NEgRL+HWYa7DZcNl1GjOQP26Q3iAh/gMghg== X-Received: by 2002:a1c:cc0c:: with SMTP id h12mr158042wmb.57.1597760599052; Tue, 18 Aug 2020 07:23:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g16sm31613734wrs.88.2020.08.18.07.23.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:18 -0700 (PDT) Message-Id: <68bf5bef4bebea48234b6f5c75c7a07d38ca15c0.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:06 +0000 Subject: [PATCH v2 09/11] maintenance: use pointers to check --auto Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git maintenance run' command has an '--auto' option. This is used by other Git commands such as 'git commit' or 'git fetch' to check if maintenance should be run after adding data to the repository. Previously, this --auto option was only used to add the argument to the 'git gc' command as part of the 'gc' task. We will be expanding the other tasks to perform a check to see if they should do work as part of the --auto flag, when they are enabled by config. First, update the 'gc' task to perform the auto check inside the maintenance process. This prevents running an extra 'git gc --auto' command when not needed. It also shows a model for other tasks. Second, use the 'auto_condition' function pointer as a signal for whether we enable the maintenance task under '--auto'. For instance, we do not want to enable the 'fetch' task in '--auto' mode, so that function pointer will remain NULL. Now that we are not automatically calling 'git gc', a test in t5514-fetch-multiple.sh must be changed to watch for 'git maintenance' instead. We continue to pass the '--auto' option to the 'git gc' command when necessary, because of the gc.autoDetach config option changes behavior. Likely, we will want to absorb the daemonizing behavior implied by gc.autoDetach as a maintenance.autoDetach config option. Signed-off-by: Derrick Stolee --- builtin/gc.c | 16 ++++++++++++++++ t/t5514-fetch-multiple.sh | 2 +- t/t7900-maintenance.sh | 2 +- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e66326ede9..c54c3070b8 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -788,9 +788,17 @@ static int maintenance_task_gc(struct maintenance_opts *opts) typedef int maintenance_task_fn(struct maintenance_opts *opts); +/* + * An auto condition function returns 1 if the task should run + * and 0 if the task should NOT run. See needs_to_gc() for an + * example. + */ +typedef int maintenance_auto_fn(void); + struct maintenance_task { const char *name; maintenance_task_fn *fn; + maintenance_auto_fn *auto_condition; unsigned enabled:1, selected:1; int selected_order; @@ -808,6 +816,7 @@ static struct maintenance_task tasks[] = { [TASK_GC] = { "gc", maintenance_task_gc, + need_to_gc, 1, }, [TASK_COMMIT_GRAPH] = { @@ -863,6 +872,11 @@ static int maintenance_run(struct maintenance_opts *opts) if (!found_selected && !tasks[i].enabled) continue; + if (opts->auto_flag && + (!tasks[i].auto_condition || + !tasks[i].auto_condition())) + continue; + if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); result = 1; @@ -877,6 +891,8 @@ static void initialize_task_config(void) { int i; struct strbuf config_name = STRBUF_INIT; + gc_config(); + for (i = 0; i < TASK__COUNT; i++) { int config_value; diff --git a/t/t5514-fetch-multiple.sh b/t/t5514-fetch-multiple.sh index de8e2f1531..bd202ec6f3 100755 --- a/t/t5514-fetch-multiple.sh +++ b/t/t5514-fetch-multiple.sh @@ -108,7 +108,7 @@ test_expect_success 'git fetch --multiple (two remotes)' ' GIT_TRACE=1 git fetch --multiple one two 2>trace && git branch -r > output && test_cmp ../expect output && - grep "built-in: git gc" trace >gc && + grep "built-in: git maintenance" trace >gc && test_line_count = 1 gc ) ' diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 290abb7832..4f6a04ddb1 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -21,7 +21,7 @@ test_expect_success 'run [--auto|--quiet]' ' GIT_TRACE2_EVENT="$(pwd)/run-no-quiet.txt" \ git maintenance run --no-quiet 2>/dev/null && test_subcommand git gc --quiet X-Patchwork-Id: 11721059 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 995721392 for ; Tue, 18 Aug 2020 14:24:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7BC4A20825 for ; Tue, 18 Aug 2020 14:24:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="N0o09gjO" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726972AbgHROX5 (ORCPT ); Tue, 18 Aug 2020 10:23:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50644 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726727AbgHROXY (ORCPT ); Tue, 18 Aug 2020 10:23:24 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 90A4BC06134A for ; Tue, 18 Aug 2020 07:23:21 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id x5so16399693wmi.2 for ; Tue, 18 Aug 2020 07:23:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=065lcNmEdoUu44wwh69X/sCzMfLXKMLC1F0QAzASnbs=; b=N0o09gjOFh/cXgJm/aLAC5oPL6DAzdkmKY7c36gqG9x8Ajgle9lRPh2snxkX6JCrC7 PGU7G0myLzEWYBimJzRBFyc5WFUcHz1MP+8geCPtd56+StwJGLX0ctAhW2TDrmLQzFKZ A7zKUCcV9WV/J258VnluuitbgfWedNADBcgTcBOhGUkz0RWAxxMxnMmMhMRlhEzFB1sX /9MhgmmOBoN8zdpW4MAdIEVMkCZF5hLzSyihmOAQHaYF1FsbAIQDOpnyuUeimxL7K/sY UEOrUzyXlfLopidUAFo4WB9uFuyMifKAW0cppLVwuLD5zZdfsYqiJcehJAx0Tts6EEqg wxxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=065lcNmEdoUu44wwh69X/sCzMfLXKMLC1F0QAzASnbs=; b=Z0I7F1gsybpZlTXfQTfU59wbIsq+vDnFQc+Z2l0ZYKEXzGCKF6LOE2fBi0r/nWvk0t UPWwcdeVc79YnBfQtmMf3R6tw2419HyBV4xlaj0BxMGsQISRjpv2Lhq9YHMxSxl5qKym L4mJcT76Uwr98KpvQ4kLH/Rrh8z1lxiCnyuigMfLS3NwfMUvhFSCYYxw3TWa70o67t3F Ev8mz7Vakl+KzK5sWqpVgJhRIchCvvi+toR5cs9Yv4BhTu//L0eesPqElO/ANUUIx1jg ZXNBdk06QX0cMZz24IDgqhOvA/a2x4VydRaqpaUaU+5L3BUCTNbzED46vQo1HnrlKqhU kTJw== X-Gm-Message-State: AOAM532NZ5pP19P5G0Km+iInFuxWbhuEmyhAGF3DK6hTZfwkWzyZo+S9 HkwBsdLzQmbSgmYgN20QMid2C9UbME8= X-Google-Smtp-Source: ABdhPJwTFghG8KNDb1l3dikbiWbYS5O3eSOb9vADpG1MVC+DgV0BQiNuwEkHed3yGChNO0MYT5VObw== X-Received: by 2002:a7b:c185:: with SMTP id y5mr160077wmi.95.1597760600012; Tue, 18 Aug 2020 07:23:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id u66sm128341wmu.37.2020.08.18.07.23.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:19 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:07 +0000 Subject: [PATCH v2 10/11] maintenance: add auto condition for commit-graph task Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Instead of writing a new commit-graph in every 'git maintenance run --auto' process (when maintenance.commit-graph.enalbed is configured to be true), only write when there are "enough" commits not in a commit-graph file. This count is controlled by the maintenance.commit-graph.auto config option. To compute the count, use a depth-first search starting at each ref, and leaving markers using the PARENT1 flag. If this count reaches the limit, then terminate early and start the task. Otherwise, this operation will peel every ref and parse the commit it points to. If these are all in the commit-graph, then this is typically a very fast operation. Users with many refs might feel a slow-down, and hence could consider updating their limit to be very small. A negative value will force the step to run every time. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 10 ++++ builtin/gc.c | 76 ++++++++++++++++++++++++++++ object.h | 1 + 3 files changed, 87 insertions(+) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index 370cbfb42f..9bd69b9df3 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -2,3 +2,13 @@ maintenance..enabled:: This boolean config option controls whether the maintenance task with name `` is run when no `--task` option is specified. By default, only `maintenance.gc.enabled` is true. + +maintenance.commit-graph.auto:: + This integer config option controls how often the `commit-graph` task + should be run as part of `git maintenance run --auto`. If zero, then + the `commit-graph` task will not run with the `--auto` option. A + negative value will force the task to run every time. Otherwise, a + positive value implies the command should run when the number of + reachable commits that are not in the commit-graph file is at least + the value of `maintenance.commit-graph.auto`. The default value is + 100. diff --git a/builtin/gc.c b/builtin/gc.c index c54c3070b8..73e485bfc0 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -28,6 +28,7 @@ #include "blob.h" #include "tree.h" #include "promisor-remote.h" +#include "refs.h" #define FAILED_RUN "failed to run %s" @@ -710,6 +711,80 @@ struct maintenance_opts { int quiet; }; +/* Remember to update object flag allocation in object.h */ +#define PARENT1 (1u<<16) + +static int num_commits_not_in_graph = 0; +static int limit_commits_not_in_graph = 100; + +static int dfs_on_ref(const char *refname, + const struct object_id *oid, int flags, + void *cb_data) +{ + int result = 0; + struct object_id peeled; + struct commit_list *stack = NULL; + struct commit *commit; + + if (!peel_ref(refname, &peeled)) + oid = &peeled; + if (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT) + return 0; + + commit = lookup_commit(the_repository, oid); + if (!commit) + return 0; + if (parse_commit(commit)) + return 0; + + commit_list_append(commit, &stack); + + while (!result && stack) { + struct commit_list *parent; + + commit = pop_commit(&stack); + + for (parent = commit->parents; parent; parent = parent->next) { + if (parse_commit(parent->item) || + commit_graph_position(parent->item) != COMMIT_NOT_FROM_GRAPH || + parent->item->object.flags & PARENT1) + continue; + + parent->item->object.flags |= PARENT1; + num_commits_not_in_graph++; + + if (num_commits_not_in_graph >= limit_commits_not_in_graph) { + result = 1; + break; + } + + commit_list_append(parent->item, &stack); + } + } + + free_commit_list(stack); + return result; +} + +static int should_write_commit_graph(void) +{ + int result; + + git_config_get_int("maintenance.commit-graph.auto", + &limit_commits_not_in_graph); + + if (!limit_commits_not_in_graph) + return 0; + if (limit_commits_not_in_graph < 0) + return 1; + + result = for_each_ref(dfs_on_ref, NULL); + + clear_commit_marks_all(PARENT1); + + return result; +} + static int run_write_commit_graph(struct maintenance_opts *opts) { struct child_process child = CHILD_PROCESS_INIT; @@ -822,6 +897,7 @@ static struct maintenance_task tasks[] = { [TASK_COMMIT_GRAPH] = { "commit-graph", maintenance_task_commit_graph, + should_write_commit_graph, }, }; diff --git a/object.h b/object.h index 96a2105859..30d4e0f6a0 100644 --- a/object.h +++ b/object.h @@ -74,6 +74,7 @@ struct object_array { * list-objects-filter.c: 21 * builtin/fsck.c: 0--3 * builtin/index-pack.c: 2021 + * builtin/maintenance.c: 16 * builtin/pack-objects.c: 20 * builtin/reflog.c: 10--12 * builtin/show-branch.c: 0-------------------------------------------26 From patchwork Tue Aug 18 14:23:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11721061 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B70A5722 for ; Tue, 18 Aug 2020 14:24:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9EE4320825 for ; Tue, 18 Aug 2020 14:24:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="RO7hsi/+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726845AbgHROX5 (ORCPT ); Tue, 18 Aug 2020 10:23:57 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:50656 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726731AbgHROX1 (ORCPT ); Tue, 18 Aug 2020 10:23:27 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 72577C06134B for ; Tue, 18 Aug 2020 07:23:22 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id g75so17174586wme.4 for ; Tue, 18 Aug 2020 07:23:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CnLjXOJlzWTb8DhmLa2BUbqj9NUjX6DalUVKu7VYcx0=; b=RO7hsi/+uwvX1SZDrRSBsGLEkjDvl1BvXnqWRRC8o0213u8qkTyOUp0kU75d+sy4It rEhfhgwLX2iWqaPZ1cMx5JCsM0mImhALoRybb4KnM8JUUEwyyt+YLln9iMZeB4XI28FQ mIvIYa4N2Eyc9oVisVpREFbh9Bd0opxtE3dRxrOjqSQGEuqBg/Hl6tfuGC8lWeQFjW7q a9ywV6iNhmXN8KfZ8EXc6WEQ3RKXNK22LlNsl/NAfFjuf8YxXd+3qn3MCxxdSqJQuFM+ CbP3WtnOJbCTZsQqVn1s15vHmnnjAO1U8qL3YPccApcwYuwtbMg3/ire/hSZpDQm+b0d NuWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CnLjXOJlzWTb8DhmLa2BUbqj9NUjX6DalUVKu7VYcx0=; b=UnDIeKht66MbwYsFFLxkUPt8ZiLNfDuAvjqipg1eVM62beLJhY4Huq+np8OStyLIP/ QxvfvZUFOqLQhukhvAylj/30qhZZD2wuUiddvrrl2gGTeG2Gxyzlj0UyI0gPyHd6AY8h AwpNOQdvXWDp/saNYRY53TT0xTbjGaipyJQ2/Pu1ocwHesNR4NGW9nQkb6/zP8Hb/fee PA1wrfq1DI8zo0V2d2WFMZHcwjjIFimGi+5xp534SVLJbO03tJ0FRkCgRrN5Y82oO0Hj PlemeLOscJZdf71w/VvDzz16t62a24YhA3sI/AEFKNoL1Oo8mvTKE/EXuIwBy4c/SX5t XHTw== X-Gm-Message-State: AOAM531xVoMyQuX6YJ3bAjXDTbFCRZ+C/fy5RGWzmH1PKGjnd86SOHjv UCLjr54ieOz8LdhcQyOnBYNXvpubtaw= X-Google-Smtp-Source: ABdhPJwrV46J7wv8qHoKUUF3WdvlPixKXUNVDb9Su3kNLPLxFdgCJXXQ/aJ8iPHqMxomBFefBDPCIg== X-Received: by 2002:a1c:cc0c:: with SMTP id h12mr158168wmb.57.1597760600846; Tue, 18 Aug 2020 07:23:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w132sm126175wma.32.2020.08.18.07.23.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 18 Aug 2020 07:23:20 -0700 (PDT) Message-Id: <46fbe161aa154de9406117ee916b1bd5f549905c.1597760589.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 18 Aug 2020 14:23:08 +0000 Subject: [PATCH v2 11/11] maintenance: add trace2 regions for task execution Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Signed-off-by: Derrick Stolee --- builtin/gc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builtin/gc.c b/builtin/gc.c index 73e485bfc0..3fdb08655c 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -953,10 +953,12 @@ static int maintenance_run(struct maintenance_opts *opts) !tasks[i].auto_condition())) continue; + trace2_region_enter("maintenance", tasks[i].name, r); if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); result = 1; } + trace2_region_leave("maintenance", tasks[i].name, r); } rollback_lock_file(&lk);