From patchwork Tue Aug 25 18:33:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736169 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7069F159A for ; Tue, 25 Aug 2020 18:34:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4EAB42071E for ; Tue, 25 Aug 2020 18:34:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pwCNE+8t" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726432AbgHYSd6 (ORCPT ); Tue, 25 Aug 2020 14:33:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726119AbgHYSdw (ORCPT ); Tue, 25 Aug 2020 14:33:52 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4DBC9C061756 for ; Tue, 25 Aug 2020 11:33:52 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id f7so13874234wrw.1 for ; Tue, 25 Aug 2020 11:33:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZfkgU/gH4H5W+CFiXAoafxfJn1w3A8mviczuXnMlgbE=; b=pwCNE+8t53SMw9u5ahi9nYOe5R8baqzuvpOohH7Te2kTNsjwpcT+xhhrfsQ2qpQGF3 toCDKlVuD4r2IBRWBHkBc+IOR3t9xKH3T5+DwgENf+xxuH2nrESuj4ITEtCD6yz/ybaT sFvQGDpPAy7eCwTBZkSUioLOHumn2czhCT12ShKrNym90FLA/pg250TSw0DaScEUy7/n cCtXukAwatItd0WaeZctaXsD02T88kHVJtxCAzHVm943xJHWmQs8w4oyWQ6ZjkOKEgVY Da1JKKT+NmyBKJqckZ5gIOj9iYhG1jG9N0q7vn5VU3IIklo5TTebcm3EoIAbG8RH+euy nHgw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZfkgU/gH4H5W+CFiXAoafxfJn1w3A8mviczuXnMlgbE=; b=L8zo5iTuZY5Z04O1O7r16jxI11d8F4jxZnLtzpPt8QLc868v54WnPkfet6oV7iuQx/ 4eOgqG6ltGKZTRDKMOy8j8F9jIZv90opguu/Kucm27siQ/Yaxnxo8CdA0rKqHxa2VAye brsZq1yw0HYGMdKdnKcXFM2kkAGgXV96MQ9wYejKLN97mNIQ0PNHy4UsdNYarSZ2aqph d6Gq5JJqzBRJGrMzRstgabJtbt65yCZ4xYlKJN5sTUL5mMWVJW74PkL1s5kaoAUhLxcn 8lkSym5y8mlIZiWFCUMjI8BKFsG61+32M0lGN1cQ1tTcnApDiQS1PYLZ8eosnx7rEo7r V7cw== X-Gm-Message-State: AOAM533nTo2JQnODgVZgKdqAJLwHFnnaHs2yoi3oA7z5eGfCQG1AGUSr 6ExZdnPuG+TME5X3qbeW7ROGYD6u2f4= X-Google-Smtp-Source: ABdhPJx1Sjjh5yUES6g/CP8zVkLjW+t/J4LzC3DLR83hVzm4baUwnCnZzhoj8PjgSdxKezxXvkHs0A== X-Received: by 2002:adf:8405:: with SMTP id 5mr11776862wrf.393.1598380429315; Tue, 25 Aug 2020 11:33:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm33272499wrm.52.2020.08.25.11.33.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:48 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:36 +0000 Subject: [PATCH v3 01/11] maintenance: create basic maintenance runner Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'gc' builtin is our current entrypoint for automatically maintaining a repository. This one tool does many operations, such as repacking the repository, packing refs, and rewriting the commit-graph file. The name implies it performs "garbage collection" which means several different things, and some users may not want to use this operation that rewrites the entire object database. Create a new 'maintenance' builtin that will become a more general- purpose command. To start, it will only support the 'run' subcommand, but will later expand to add subcommands for scheduling maintenance in the background. For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin. In fact, the only option is the '--auto' toggle, which is handed directly to the 'gc' builtin. The current change is isolated to this simple operation to prevent more interesting logic from being lost in all of the boilerplate of adding a new builtin. Use existing builtin/gc.c file because we want to share code between the two builtins. It is possible that we will have 'maintenance' replace the 'gc' builtin entirely at some point, leaving 'git gc' as an alias for some specific arguments to 'git maintenance run'. Create a new test_subcommand helper that allows us to test if a certain subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a file. A negation mode is available that will be used in later tests. Helped-by: Jonathan Nieder Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-maintenance.txt | 57 +++++++++++++++++++++++++++++++ builtin.h | 1 + builtin/gc.c | 57 +++++++++++++++++++++++++++++++ command-list.txt | 1 + git.c | 1 + t/t7900-maintenance.sh | 21 ++++++++++++ t/test-lib-functions.sh | 33 ++++++++++++++++++ 8 files changed, 172 insertions(+) create mode 100644 Documentation/git-maintenance.txt create mode 100755 t/t7900-maintenance.sh diff --git a/.gitignore b/.gitignore index ee509a2ad2..a5808fa30d 100644 --- a/.gitignore +++ b/.gitignore @@ -90,6 +90,7 @@ /git-ls-tree /git-mailinfo /git-mailsplit +/git-maintenance /git-merge /git-merge-base /git-merge-index diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt new file mode 100644 index 0000000000..ff47fb3641 --- /dev/null +++ b/Documentation/git-maintenance.txt @@ -0,0 +1,57 @@ +git-maintenance(1) +================== + +NAME +---- +git-maintenance - Run tasks to optimize Git repository data + + +SYNOPSIS +-------- +[verse] +'git maintenance' run [] + + +DESCRIPTION +----------- +Run tasks to optimize Git repository data, speeding up other Git commands +and reducing storage requirements for the repository. + +Git commands that add repository data, such as `git add` or `git fetch`, +are optimized for a responsive user experience. These commands do not take +time to optimize the Git data, since such optimizations scale with the full +size of the repository while these user commands each perform a relatively +small action. + +The `git maintenance` command provides flexibility for how to optimize the +Git repository. + +SUBCOMMANDS +----------- + +run:: + Run one or more maintenance tasks. + +TASKS +----- + +gc:: + Clean up unnecessary files and optimize the local repository. "GC" + stands for "garbage collection," but this task performs many + smaller tasks. This task can be expensive for large repositories, + as it repacks all Git objects into a single pack-file. It can also + be disruptive in some situations, as it deletes stale data. See + linkgit:git-gc[1] for more details on garbage collection in Git. + +OPTIONS +------- +--auto:: + When combined with the `run` subcommand, run maintenance tasks + only if certain thresholds are met. For example, the `gc` task + runs when the number of loose objects exceeds the number stored + in the `gc.auto` config setting, or when the number of pack-files + exceeds the `gc.autoPackLimit` config setting. + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/builtin.h b/builtin.h index a5ae15bfe5..17c1c0ce49 100644 --- a/builtin.h +++ b/builtin.h @@ -167,6 +167,7 @@ int cmd_ls_tree(int argc, const char **argv, const char *prefix); int cmd_ls_remote(int argc, const char **argv, const char *prefix); int cmd_mailinfo(int argc, const char **argv, const char *prefix); int cmd_mailsplit(int argc, const char **argv, const char *prefix); +int cmd_maintenance(int argc, const char **argv, const char *prefix); int cmd_merge(int argc, const char **argv, const char *prefix); int cmd_merge_base(int argc, const char **argv, const char *prefix); int cmd_merge_index(int argc, const char **argv, const char *prefix); diff --git a/builtin/gc.c b/builtin/gc.c index aafa0946f5..7f2771e786 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -699,3 +699,60 @@ int cmd_gc(int argc, const char **argv, const char *prefix) return 0; } + +static const char * const builtin_maintenance_run_usage[] = { + N_("git maintenance run [--auto]"), + NULL +}; + +struct maintenance_run_opts { + int auto_flag; +}; + +static int maintenance_task_gc(struct maintenance_run_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_push(&child.args, "gc"); + + if (opts->auto_flag) + strvec_push(&child.args, "--auto"); + + close_object_store(the_repository->objects); + return run_command(&child); +} + +static int maintenance_run(int argc, const char **argv, const char *prefix) +{ + struct maintenance_run_opts opts; + struct option builtin_maintenance_run_options[] = { + OPT_BOOL(0, "auto", &opts.auto_flag, + N_("run tasks based on the state of the repository")), + OPT_END() + }; + memset(&opts, 0, sizeof(opts)); + + argc = parse_options(argc, argv, prefix, + builtin_maintenance_run_options, + builtin_maintenance_run_usage, + PARSE_OPT_STOP_AT_NON_OPTION); + + if (argc != 0) + usage_with_options(builtin_maintenance_run_usage, + builtin_maintenance_run_options); + return maintenance_task_gc(&opts); +} + +static const char builtin_maintenance_usage[] = N_("git maintenance run []"); + +int cmd_maintenance(int argc, const char **argv, const char *prefix) +{ + if (argc == 2 && !strcmp(argv[1], "-h")) + usage(builtin_maintenance_usage); + + if (!strcmp(argv[1], "run")) + return maintenance_run(argc - 1, argv + 1, prefix); + + die(_("invalid subcommand: %s"), argv[1]); +} diff --git a/command-list.txt b/command-list.txt index e5901f2213..0e3204e7d1 100644 --- a/command-list.txt +++ b/command-list.txt @@ -117,6 +117,7 @@ git-ls-remote plumbinginterrogators git-ls-tree plumbinginterrogators git-mailinfo purehelpers git-mailsplit purehelpers +git-maintenance mainporcelain git-merge mainporcelain history git-merge-base plumbinginterrogators git-merge-file plumbingmanipulators diff --git a/git.c b/git.c index 8bd1d7551d..24f250d29a 100644 --- a/git.c +++ b/git.c @@ -529,6 +529,7 @@ static struct cmd_struct commands[] = { { "ls-tree", cmd_ls_tree, RUN_SETUP }, { "mailinfo", cmd_mailinfo, RUN_SETUP_GENTLY | NO_PARSEOPT }, { "mailsplit", cmd_mailsplit, NO_PARSEOPT }, + { "maintenance", cmd_maintenance, RUN_SETUP_GENTLY | NO_PARSEOPT }, { "merge", cmd_merge, RUN_SETUP | NEED_WORK_TREE }, { "merge-base", cmd_merge_base, RUN_SETUP }, { "merge-file", cmd_merge_file, RUN_SETUP_GENTLY }, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh new file mode 100755 index 0000000000..14aa691f19 --- /dev/null +++ b/t/t7900-maintenance.sh @@ -0,0 +1,21 @@ +#!/bin/sh + +test_description='git maintenance builtin' + +. ./test-lib.sh + +test_expect_success 'help text' ' + test_expect_code 129 git maintenance -h 2>err && + test_i18ngrep "usage: git maintenance run" err && + test_expect_code 128 git maintenance barf 2>err && + test_i18ngrep "invalid subcommand: barf" err +' + +test_expect_success 'run [--auto]' ' + GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run && + GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto && + test_subcommand git gc ... < +# +# For example, to look for an invocation of "git upload-pack +# /path/to/repo" +# +# GIT_TRACE2_EVENT=event.log git fetch ... && +# test_subcommand git upload-pack "$PATH" X-Patchwork-Id: 11736175 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2D5FD109B for ; Tue, 25 Aug 2020 18:34:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 154FA20782 for ; Tue, 25 Aug 2020 18:34:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CBi6UTBF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726617AbgHYSeI (ORCPT ); Tue, 25 Aug 2020 14:34:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43398 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726180AbgHYSdx (ORCPT ); Tue, 25 Aug 2020 14:33:53 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B767FC061757 for ; Tue, 25 Aug 2020 11:33:52 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id x9so3102917wmi.2 for ; Tue, 25 Aug 2020 11:33:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=TNkZA0YzlqgpbaFMJJADNK04OIR3HUTxPD/4oyKldqI=; b=CBi6UTBFA+OOn+WezSOLvG6/V8IUtOxlPKH8n+NwhUtL9Q2XRlT51r+mRwrNFwWnQD lplNJ7JeFpDddVABZqe7OpPdlZ0Wl+Stoxjz+SIXJGaAnawiJ9TJNEMD8+R4+EbJzbeN 9fuOGxeKvOIQRPMYDXm+nsLIfU29UYEzpAtcedotQZauwnjmK/B77Ds+ZQyLdvnc6Pbz cdoKO3kznIDC6C7HZRWFEbmi92/72/vLEYYRK/ka8HgxzdEyFKYyD7Zez/C0YIHiDuhp iNMgOW+LYVRpwiEiD/IY4XnGGX7gnUl3gxTRbBhpVQyyVk+n8YPgjSXoQ2/ZzUo0kijH uWxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=TNkZA0YzlqgpbaFMJJADNK04OIR3HUTxPD/4oyKldqI=; b=R2CQd9TGRsL1XVbeHgkcoaKudbQkNSSSd42ZVRv47Btl9T6aUDH0049tcns+pJ9Llw twZT3SJ6x1wwAgKM8M7Xr2AfPGbSJev5L/RBbwX99H+6YDQMNdfqSjzfURo/DdsQO9Nl PRFtb1ZYeFTS9diQe3KjEv/rY8QzNqocL3EBPmT3iK+Q9+Xi4TT9qxw6zxbmE860Dj5d N0enDtoaA4ZWvazxTgUTczjGV03TBHwTXCdUpKYaHQ9H1vFeY4SBCCp70no/0hE504CV Vvh+GUUm8adH3/YLER2trSdT2r+8cdrCMXN3ja5qfhqpu9zD6blA7/CjBlfxsG9sdj93 Amtw== X-Gm-Message-State: AOAM532XQTUH9luwyk1HA5Do4oVNuuoTRclXusITIrjLPnOsDIZbUMds OrWeEIiIFf1bKdY1ch5A+zyAiXx3mx0= X-Google-Smtp-Source: ABdhPJyxWt1+KcQQSEWy/RUPycsydFKdxxCfWrjiXSFunKEdas8RgyieG19sJ3K8eCtbrXKDQA/zNA== X-Received: by 2002:a05:600c:22d6:: with SMTP id 22mr3178790wmg.120.1598380430382; Tue, 25 Aug 2020 11:33:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v16sm7615589wmj.14.2020.08.25.11.33.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:49 -0700 (PDT) Message-Id: <5386d8a6283d814c0d0d2b7a180409f064cea709.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:37 +0000 Subject: [PATCH v3 02/11] maintenance: add --quiet option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Maintenance activities are commonly used as steps in larger scripts. Providing a '--quiet' option allows those scripts to be less noisy when run on a terminal window. Turn this mode on by default when stderr is not a terminal. Pipe the option to the 'git gc' child process. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 3 +++ builtin/gc.c | 11 ++++++++++- t/t7900-maintenance.sh | 15 ++++++++++----- 3 files changed, 23 insertions(+), 6 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index ff47fb3641..04fa0fe329 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -52,6 +52,9 @@ OPTIONS in the `gc.auto` config setting, or when the number of pack-files exceeds the `gc.autoPackLimit` config setting. +--quiet:: + Do not report progress or other information over `stderr`. + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 7f2771e786..da11f1edcd 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -701,12 +701,13 @@ int cmd_gc(int argc, const char **argv, const char *prefix) } static const char * const builtin_maintenance_run_usage[] = { - N_("git maintenance run [--auto]"), + N_("git maintenance run [--auto] [--[no-]quiet]"), NULL }; struct maintenance_run_opts { int auto_flag; + int quiet; }; static int maintenance_task_gc(struct maintenance_run_opts *opts) @@ -718,6 +719,10 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts) if (opts->auto_flag) strvec_push(&child.args, "--auto"); + if (opts->quiet) + strvec_push(&child.args, "--quiet"); + else + strvec_push(&child.args, "--no-quiet"); close_object_store(the_repository->objects); return run_command(&child); @@ -729,10 +734,14 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) struct option builtin_maintenance_run_options[] = { OPT_BOOL(0, "auto", &opts.auto_flag, N_("run tasks based on the state of the repository")), + OPT_BOOL(0, "quiet", &opts.quiet, + N_("do not report progress or other information over stderr")), OPT_END() }; memset(&opts, 0, sizeof(opts)); + opts.quiet = !isatty(2); + argc = parse_options(argc, argv, prefix, builtin_maintenance_run_options, builtin_maintenance_run_usage, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 14aa691f19..47d512464c 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -11,11 +11,16 @@ test_expect_success 'help text' ' test_i18ngrep "invalid subcommand: barf" err ' -test_expect_success 'run [--auto]' ' - GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run && - GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto && - test_subcommand git gc /dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" \ + git maintenance run --auto 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-no-quiet.txt" \ + git maintenance run --no-quiet 2>/dev/null && + test_subcommand git gc --quiet X-Patchwork-Id: 11736173 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 88502109B for ; Tue, 25 Aug 2020 18:34:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 68E452071E for ; Tue, 25 Aug 2020 18:34:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LOApDveB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726593AbgHYSeE (ORCPT ); Tue, 25 Aug 2020 14:34:04 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726158AbgHYSdx (ORCPT ); Tue, 25 Aug 2020 14:33:53 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 02A2DC061574 for ; Tue, 25 Aug 2020 11:33:53 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id g75so3089729wme.4 for ; Tue, 25 Aug 2020 11:33:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ZLD9qTyuUt057D6hwair//y9YNofza2ZxPWALwJ/KXI=; b=LOApDveBHnCyRrnZIoI1kqcXauTP88B3MFRWCcZ94uWF+4xFT5pVRbkOz8oxQWbejN wWqBWYzNuddrxuc54GYOMBg9yFdJjkRHymC73+h4h4qNjwMMItCEvoHHdn69+DU35FA6 RRODqA2Zw+XzEkSHIgIY1l7tmB0RHMMqiLhGEgZOnQxU+XTA8rNHeifgtmEPXaDsnBaG IwiiJJ6zPbx8plmunAibaU3XcpeTqLwe57HMHijmu/X12m/JGPvWJ0Rvi9yVoyv8aPET V0+DvzjYAeMOLIZifNZE9TVIc/ggSCzny6PwlIcJRAFV/hKtOAIUXNUFTFjBkDJY9atg islA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ZLD9qTyuUt057D6hwair//y9YNofza2ZxPWALwJ/KXI=; b=rfP5lMZZo+mwf0nh/P0vVyPX5z19GljNm9Y5YQF49W1xQmEqApLNBybZt6xJWK1Vnj sIj9obNwgcmuM2qnK2pFQroREuVcaxTWtLuihO9ppBNk0+gYTA93HohMxGwD+VWsFyBk 8G1Oy05ABknEagyHpmD+XADvOroNVYAVOF1RUHbiAPDMuqnUKsMRSiR7e4Bnd3NGkGVD hlqPXQDidvc5ti1XrTNQqPwenIq+N4s0Ke1zsGJWveivyEF4Wo+IFiczw18GkFDei1g3 7MqggwWf3xfoUcBJ7eITMf81pjGtm+6htdZzhcpNe7iZ3YOcH1MqZOQFU7lmC7pC3NCz 09OQ== X-Gm-Message-State: AOAM531m1rURCzUoYDiSnoKSb5YNjn2EoPfvr4GjSQpI+z2SpNTiEV6V axVU0dBEFN0F/M8oj/CUEcibv57A2+c= X-Google-Smtp-Source: ABdhPJyBb6qRPCaTEe0c1qJrNxUKoh9etI230U3ggAPXQR+8cZqLgCZzFSlwHLgBMH+HFC+cIrpMow== X-Received: by 2002:a1c:28c1:: with SMTP id o184mr3398654wmo.91.1598380431340; Tue, 25 Aug 2020 11:33:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s21sm947621wmj.2.2020.08.25.11.33.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:50 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:38 +0000 Subject: [PATCH v3 03/11] maintenance: replace run_auto_gc() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The run_auto_gc() method is used in several places to trigger a check for repo maintenance after some Git commands, such as 'git commit' or 'git fetch'. To allow for extra customization of this maintenance activity, replace the 'git gc --auto [--quiet]' call with one to 'git maintenance run --auto [--quiet]'. As we extend the maintenance builtin with other steps, users will be able to select different maintenance activities. Rename run_auto_gc() to run_auto_maintenance() to be clearer what is happening on this call, and to expose all callers in the current diff. Rewrite the method to use a struct child_process to simplify the calls slightly. Since 'git fetch' already allows disabling the 'git gc --auto' subprocess, add an equivalent option with a different name to be more descriptive of the new behavior: '--[no-]maintenance'. Update the documentation to include these options at the same time. Signed-off-by: Derrick Stolee --- Documentation/fetch-options.txt | 6 ++++-- Documentation/git-clone.txt | 6 +++--- builtin/am.c | 2 +- builtin/commit.c | 2 +- builtin/fetch.c | 6 ++++-- builtin/merge.c | 2 +- builtin/rebase.c | 4 ++-- run-command.c | 16 +++++++--------- run-command.h | 2 +- t/t5510-fetch.sh | 2 +- 10 files changed, 25 insertions(+), 23 deletions(-) diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index 07d06ff445..b65a758661 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -95,9 +95,11 @@ ifndef::git-pull[] Allow several and arguments to be specified. No s may be specified. +--[no-]auto-maintenance:: --[no-]auto-gc:: - Run `git gc --auto` at the end to perform garbage collection - if needed. This is enabled by default. + Run `git maintenance run --auto` at the end to perform automatic + repository maintenance if needed. (`--[no-]auto-gc` is a synonym.) + This is enabled by default. --[no-]write-commit-graph:: Write a commit-graph after fetching. This overrides the config diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt index c898310099..097e6a86c5 100644 --- a/Documentation/git-clone.txt +++ b/Documentation/git-clone.txt @@ -78,9 +78,9 @@ repository using this option and then delete branches (or use any other Git command that makes any existing commit unreferenced) in the source repository, some objects may become unreferenced (or dangling). These objects may be removed by normal Git operations (such as `git commit`) -which automatically call `git gc --auto`. (See linkgit:git-gc[1].) -If these objects are removed and were referenced by the cloned repository, -then the cloned repository will become corrupt. +which automatically call `git maintenance run --auto`. (See +linkgit:git-maintenance[1].) If these objects are removed and were referenced +by the cloned repository, then the cloned repository will become corrupt. + Note that running `git repack` without the `--local` option in a repository cloned with `--shared` will copy objects from the source repository into a pack diff --git a/builtin/am.c b/builtin/am.c index dd4e6c2d9b..68e9d17379 100644 --- a/builtin/am.c +++ b/builtin/am.c @@ -1795,7 +1795,7 @@ static void am_run(struct am_state *state, int resume) if (!state->rebasing) { am_destroy(state); close_object_store(the_repository->objects); - run_auto_gc(state->quiet); + run_auto_maintenance(state->quiet); } } diff --git a/builtin/commit.c b/builtin/commit.c index 69ac78d5e5..f9b0a0c05d 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1702,7 +1702,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix) git_test_write_commit_graph_or_die(); repo_rerere(the_repository, 0); - run_auto_gc(quiet); + run_auto_maintenance(quiet); run_commit_hook(use_editor, get_index_file(), "post-commit", NULL); if (amend && !no_post_rewrite) { commit_post_rewrite(the_repository, current_head, &oid); diff --git a/builtin/fetch.c b/builtin/fetch.c index 2eb8d6a5a5..cb38e6f5ec 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -199,8 +199,10 @@ static struct option builtin_fetch_options[] = { OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"), N_("report that we have only objects reachable from this object")), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options), + OPT_BOOL(0, "auto-maintenance", &enable_auto_gc, + N_("run 'maintenance --auto' after fetching")), OPT_BOOL(0, "auto-gc", &enable_auto_gc, - N_("run 'gc --auto' after fetching")), + N_("run 'maintenance --auto' after fetching")), OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates, N_("check for forced-updates on all updated branches")), OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph, @@ -1891,7 +1893,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) close_object_store(the_repository->objects); if (enable_auto_gc) - run_auto_gc(verbosity < 0); + run_auto_maintenance(verbosity < 0); return result; } diff --git a/builtin/merge.c b/builtin/merge.c index 74829a838e..8f31bfab56 100644 --- a/builtin/merge.c +++ b/builtin/merge.c @@ -456,7 +456,7 @@ static void finish(struct commit *head_commit, * user should see them. */ close_object_store(the_repository->objects); - run_auto_gc(verbosity < 0); + run_auto_maintenance(verbosity < 0); } } if (new_head && show_diffstat) { diff --git a/builtin/rebase.c b/builtin/rebase.c index dadb52fa92..9ffba9009d 100644 --- a/builtin/rebase.c +++ b/builtin/rebase.c @@ -728,10 +728,10 @@ static int finish_rebase(struct rebase_options *opts) apply_autostash(state_dir_path("autostash", opts)); close_object_store(the_repository->objects); /* - * We ignore errors in 'gc --auto', since the + * We ignore errors in 'git maintenance run --auto', since the * user should see them. */ - run_auto_gc(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE))); + run_auto_maintenance(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE))); if (opts->type == REBASE_MERGE) { struct replay_opts replay = REPLAY_OPTS_INIT; diff --git a/run-command.c b/run-command.c index cc9c3296ba..2ee59acdc8 100644 --- a/run-command.c +++ b/run-command.c @@ -1866,15 +1866,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task, return result; } -int run_auto_gc(int quiet) +int run_auto_maintenance(int quiet) { - struct strvec argv_gc_auto = STRVEC_INIT; - int status; + struct child_process maint = CHILD_PROCESS_INIT; - strvec_pushl(&argv_gc_auto, "gc", "--auto", NULL); - if (quiet) - strvec_push(&argv_gc_auto, "--quiet"); - status = run_command_v_opt(argv_gc_auto.v, RUN_GIT_CMD); - strvec_clear(&argv_gc_auto); - return status; + maint.git_cmd = 1; + strvec_pushl(&maint.args, "maintenance", "run", "--auto", NULL); + strvec_push(&maint.args, quiet ? "--quiet" : "--no-quiet"); + + return run_command(&maint); } diff --git a/run-command.h b/run-command.h index 8b9bfaef16..6472b38bde 100644 --- a/run-command.h +++ b/run-command.h @@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args); /* * Trigger an auto-gc */ -int run_auto_gc(int quiet); +int run_auto_maintenance(int quiet); #define RUN_COMMAND_NO_STDIN 1 #define RUN_GIT_CMD 2 /*If this is to be git sub-command */ diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh index 2a1abe91f0..a9682c5669 100755 --- a/t/t5510-fetch.sh +++ b/t/t5510-fetch.sh @@ -934,7 +934,7 @@ test_expect_success 'fetching with auto-gc does not lock up' ' git config fetch.unpackLimit 1 && git config gc.autoPackLimit 1 && git config gc.autoDetach false && - GIT_ASK_YESNO="$D/askyesno" git fetch >fetch.out 2>&1 && + GIT_ASK_YESNO="$D/askyesno" git fetch --verbose >fetch.out 2>&1 && test_i18ngrep "Auto packing the repository" fetch.out && ! grep "Should I try again" fetch.out ) From patchwork Tue Aug 25 18:33:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736177 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 061C213B1 for ; Tue, 25 Aug 2020 18:34:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DE49F2076C for ; Tue, 25 Aug 2020 18:34:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jd6qQQjN" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726619AbgHYSeL (ORCPT ); Tue, 25 Aug 2020 14:34:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43396 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726187AbgHYSdz (ORCPT ); Tue, 25 Aug 2020 14:33:55 -0400 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98626C061755 for ; Tue, 25 Aug 2020 11:33:54 -0700 (PDT) Received: by mail-wm1-x343.google.com with SMTP id 83so3567581wme.4 for ; Tue, 25 Aug 2020 11:33:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Vl2kTUl6dSX+CyNd8PSqD1JASPC5be8B3MIBf7geL1k=; b=jd6qQQjN2AqRVqXs9ny+/FWPxS/xws/GI+t06iAYfIB4foDpulwmqOTbx/6Qo7aI7A fohY4RwtiHVn++G1erUFEhVi4wbE19v8qL5j3CgFakyxr7vLtpTGt44x01bcJdWPfQ7Q O4gH5DnP5DTa6Y/8l6kvzsivczNSMlrfhStYw7q9C/Tzqo9cXPB2sZTbdEE9mxZ7f+A1 fAwJYqbhCnQOXFM1HPMLdZAwYeiepCTkr6/uvTX7cG6G53T9NrHtcvpxCSzAjfVPiy6H xuQ35WKMh+z0NepUo0qddPhAWEJqIBvP1imt/RU2u0qx3VXJabV5nUzFoofQPA294t2n Me0A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Vl2kTUl6dSX+CyNd8PSqD1JASPC5be8B3MIBf7geL1k=; b=LvOX8vTXFhc3kZorUekHEAE3V8mleh3sBMthl4I4QIVE7XaKUAh2azGiJKNABAxV2r 0zzPt0Dd4PzS7L9UBr31C0Aw/5x12M8VxxU2nqtPincHac6ErfY6ANw9qQ093vTVSzsB ItK2mI1CZleLdy1QgCUTelBaCpaIsnoZfZsVLR3YqJzIYuMh61lxWdCZfrDGWsAfsXNC vDV1GDRW0dQyObsjb8OKFHlMkrHh5bDCwnoA5ylezkFf0v5Wevtq4mEwXnFFMJqxTcC8 L7AztEUwpLp/69keHMydO8i8bTvD9tW5fhlUhS2/v4CMkx11EVGtLhkv9lIzDOQ7zEPR 2XTA== X-Gm-Message-State: AOAM530Qgk6C1r1+LWpc6fTE0Hz0qoiHmuI8HzENbuwqn6HmdkKJ8bJN aVnSRSc0pkG6RR+VtekKSdnRxlEZ8RY= X-Google-Smtp-Source: ABdhPJwFWH/upkL81154p79lXpjzycJjHoh+tQb7+NNTVXTWJCJhzmFf3vOcrPP/2RjYGf9nxw3DZg== X-Received: by 2002:a1c:3886:: with SMTP id f128mr3221665wma.121.1598380432325; Tue, 25 Aug 2020 11:33:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n5sm8758605wme.9.2020.08.25.11.33.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:51 -0700 (PDT) Message-Id: <82326c1c38ca6d7ec71bddb50d33c1e5bc3421fb.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:39 +0000 Subject: [PATCH v3 04/11] maintenance: initialize task array Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee In anticipation of implementing multiple maintenance tasks inside the 'maintenance' builtin, use a list of structs to describe the work to be done. The struct maintenance_task stores the name of the task (as given by a future command-line argument) along with a function pointer to its implementation and a boolean for whether the step is enabled. A list these structs are initialized with the full list of implemented tasks along with a default order. For now, this list only contains the "gc" task. This task is also the only task enabled by default. The run subcommand will return a nonzero exit code if any task fails. However, it will attempt all tasks in its loop before returning with the failure. Also each failed task will print an error message. Helped-by: Taylor Blau Helped-by: Junio C Hamano Signed-off-by: Derrick Stolee --- builtin/gc.c | 43 ++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/builtin/gc.c b/builtin/gc.c index da11f1edcd..81643515e5 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -728,6 +728,47 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts) return run_command(&child); } +typedef int maintenance_task_fn(struct maintenance_run_opts *opts); + +struct maintenance_task { + const char *name; + maintenance_task_fn *fn; + unsigned enabled:1; +}; + +enum maintenance_task_label { + TASK_GC, + + /* Leave as final value */ + TASK__COUNT +}; + +static struct maintenance_task tasks[] = { + [TASK_GC] = { + "gc", + maintenance_task_gc, + 1, + }, +}; + +static int maintenance_run_tasks(struct maintenance_run_opts *opts) +{ + int i; + int result = 0; + + for (i = 0; i < TASK__COUNT; i++) { + if (!tasks[i].enabled) + continue; + + if (tasks[i].fn(opts)) { + error(_("task '%s' failed"), tasks[i].name); + result = 1; + } + } + + return result; +} + static int maintenance_run(int argc, const char **argv, const char *prefix) { struct maintenance_run_opts opts; @@ -750,7 +791,7 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) if (argc != 0) usage_with_options(builtin_maintenance_run_usage, builtin_maintenance_run_options); - return maintenance_task_gc(&opts); + return maintenance_run_tasks(&opts); } static const char builtin_maintenance_usage[] = N_("git maintenance run []"); From patchwork Tue Aug 25 18:33:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736181 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A81513B1 for ; Tue, 25 Aug 2020 18:34:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 284E42074D for ; Tue, 25 Aug 2020 18:34:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IshS51jV" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726645AbgHYSeS (ORCPT ); Tue, 25 Aug 2020 14:34:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43400 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726090AbgHYSd4 (ORCPT ); Tue, 25 Aug 2020 14:33:56 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E66BBC0613ED for ; Tue, 25 Aug 2020 11:33:55 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id a5so13854135wrm.6 for ; Tue, 25 Aug 2020 11:33:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=DYDzdATbEtsle88tIWHcnPVQI2YF/b2jAlzqwzreowg=; b=IshS51jVytdvJ6hgPu1aYhBxsWZfeciZO0NUmGjvFVGJ4R9tXuIFe2ZczpQcPq0b1a pfXsI1Xrf4VLZtyBSx3lTSRwEm11sDKsjdz4x4oMfFpquNGaDwJQfKzCk90bIa9fxofV S4QWwtHyU6tma0MI5AgHa7BD820XX4oJAA8CQvuRPxJPXso5xSeatgqCczOtdHc9zKGi KpPgaNS2qZ2Rd4J2tPf+fJuLHr2X/ijsMLsoIF2OL8/6IgbjeYJbvgoXOf2R0u0ZFATL zhm8BfTWm9EOEq1YH2IbyBdCXG2olqU0k7UQLDHV4GS1V4+3HMxhZ5E6hpRU1Cw1ZcaR hhvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=DYDzdATbEtsle88tIWHcnPVQI2YF/b2jAlzqwzreowg=; b=VpEvG1C6oSbQXwbJhIcgW23wVGF0r9VcIgdqZm2LEwUfcXBJ9kmCq2MMluvmbbeOBa l2LcR+7WMYWZmQ9pgyPKiU31bwAnqtAU+/qjsA1KWKYzLTtdUwwkY5+uBqDHQssv7xd5 nqQsj2KsJ8vyhLUwvrLuGU0PcTXcjiVNVLQ9+NjWkMbUwKWXt1UGAr/vxiPp8FbXckSR RewCP1cdZiwZ4BXGKSrSHVTeQ2+nqQjfyIio9ySY2FXjy/W4GaFeWN4OlpbORSV41xoa IhOppTejIfydG7t5K+Bguyfk/EZasTzL62IEw040PJs3m6iuYzqGulteFrfl7Tnq7+Bt l1ww== X-Gm-Message-State: AOAM5324ZrmBiYxSFkV/VB3FZ8bhuwIPX1BijmqETOgW0O0YpFs8NPy+ 3UqgWE0Vfd+jN3GlQR/TzMI0I97H0vM= X-Google-Smtp-Source: ABdhPJyF3wsTu+UDWsRmOBhlnIenNJW7ySrFT4c2iqCS1106+EAOU3jaxdKiYKHW2y5isjaefGtiVQ== X-Received: by 2002:a5d:5592:: with SMTP id i18mr8674867wrv.212.1598380433176; Tue, 25 Aug 2020 11:33:53 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v16sm7615870wmj.14.2020.08.25.11.33.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:52 -0700 (PDT) Message-Id: <06984a42bfc892b82f7b39a1aa366158fcf3fd20.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:40 +0000 Subject: [PATCH v3 05/11] maintenance: add commit-graph task Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The first new task in the 'git maintenance' builtin is the 'commit-graph' task. This updates the commit-graph file incrementally with the command git commit-graph write --reachable --split By writing an incremental commit-graph file using the "--split" option we minimize the disruption from this operation. The default behavior is to merge layers until the new "top" layer is less than half the size of the layer below. This provides quick writes most of the time, with the longer writes following a power law distribution. Most importantly, concurrent Git processes only look at the commit-graph-chain file for a very short amount of time, so they will verly likely not be holding a handle to the file when we try to replace it. (This only matters on Windows.) If a concurrent process reads the old commit-graph-chain file, but our job expires some of the .graph files before they can be read, then those processes will see a warning message (but not fail). This could be avoided by a future update to use the --expire-time argument when writing the commit-graph. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 8 ++++++++ builtin/gc.c | 30 ++++++++++++++++++++++++++++++ commit-graph.c | 8 ++++---- commit-graph.h | 1 + t/t7900-maintenance.sh | 2 ++ 5 files changed, 45 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 04fa0fe329..fc5dbcf0b9 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -35,6 +35,14 @@ run:: TASKS ----- +commit-graph:: + The `commit-graph` job updates the `commit-graph` files incrementally, + then verifies that the written data is correct. The incremental + write is safe to run alongside concurrent Git processes since it + will not expire `.graph` files that were in the previous + `commit-graph-chain` file. They will be deleted by a later run based + on the expiration delay. + gc:: Clean up unnecessary files and optimize the local repository. "GC" stands for "garbage collection," but this task performs many diff --git a/builtin/gc.c b/builtin/gc.c index 81643515e5..16e567992e 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -710,6 +710,31 @@ struct maintenance_run_opts { int quiet; }; +static int run_write_commit_graph(struct maintenance_run_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_pushl(&child.args, "commit-graph", "write", + "--split", "--reachable", NULL); + + if (opts->quiet) + strvec_push(&child.args, "--no-progress"); + + return !!run_command(&child); +} + +static int maintenance_task_commit_graph(struct maintenance_run_opts *opts) +{ + close_object_store(the_repository->objects); + if (run_write_commit_graph(opts)) { + error(_("failed to write commit-graph")); + return 1; + } + + return 0; +} + static int maintenance_task_gc(struct maintenance_run_opts *opts) { struct child_process child = CHILD_PROCESS_INIT; @@ -738,6 +763,7 @@ struct maintenance_task { enum maintenance_task_label { TASK_GC, + TASK_COMMIT_GRAPH, /* Leave as final value */ TASK__COUNT @@ -749,6 +775,10 @@ static struct maintenance_task tasks[] = { maintenance_task_gc, 1, }, + [TASK_COMMIT_GRAPH] = { + "commit-graph", + maintenance_task_commit_graph, + }, }; static int maintenance_run_tasks(struct maintenance_run_opts *opts) diff --git a/commit-graph.c b/commit-graph.c index e51c91dd5b..a9b0db7d4a 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -172,7 +172,7 @@ static char *get_split_graph_filename(struct object_directory *odb, oid_hex); } -static char *get_chain_filename(struct object_directory *odb) +char *get_commit_graph_chain_filename(struct object_directory *odb) { return xstrfmt("%s/info/commit-graphs/commit-graph-chain", odb->path); } @@ -516,7 +516,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, struct stat st; struct object_id *oids; int i = 0, valid = 1, count; - char *chain_name = get_chain_filename(odb); + char *chain_name = get_commit_graph_chain_filename(odb); FILE *fp; int stat_res; @@ -1668,7 +1668,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } if (ctx->split) { - char *lock_name = get_chain_filename(ctx->odb); + char *lock_name = get_commit_graph_chain_filename(ctx->odb); hold_lock_file_for_update_mode(&lk, lock_name, LOCK_DIE_ON_ERROR, 0444); @@ -2038,7 +2038,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx) if (ctx->split_opts && ctx->split_opts->expire_time) expire_time = ctx->split_opts->expire_time; if (!ctx->split) { - char *chain_file_name = get_chain_filename(ctx->odb); + char *chain_file_name = get_commit_graph_chain_filename(ctx->odb); unlink(chain_file_name); free(chain_file_name); ctx->num_commit_graphs_after = 0; diff --git a/commit-graph.h b/commit-graph.h index 09a97030dc..765221cdcc 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -25,6 +25,7 @@ struct raw_object_store; struct string_list; char *get_commit_graph_filename(struct object_directory *odb); +char *get_commit_graph_chain_filename(struct object_directory *odb); int open_commit_graph(const char *graph_file, int *fd, struct stat *st); /* diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 47d512464c..c0c4e6846e 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -4,6 +4,8 @@ test_description='git maintenance builtin' . ./test-lib.sh +GIT_TEST_COMMIT_GRAPH=0 + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance run" err && From patchwork Tue Aug 25 18:33:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736179 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4AF67109B for ; Tue, 25 Aug 2020 18:34:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2DEFE2074D for ; Tue, 25 Aug 2020 18:34:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EupoLdo4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726630AbgHYSeN (ORCPT ); Tue, 25 Aug 2020 14:34:13 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43406 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726306AbgHYSd4 (ORCPT ); Tue, 25 Aug 2020 14:33:56 -0400 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FE1DC061756 for ; Tue, 25 Aug 2020 11:33:55 -0700 (PDT) Received: by mail-wr1-x441.google.com with SMTP id y3so13813453wrl.4 for ; Tue, 25 Aug 2020 11:33:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=305uUdjHb+4q6rX/VtEHpGfhWHIvwXgJSbT7kEvsmqc=; b=EupoLdo4ld3zcGKoqlPp2ycby7dhXPLpxyopDYvNABlgH7GLODujFarZJ1iZ+IOWA8 uOOUfdwBSDs2WDK0b1h0AOui5uilM36bURFk2Q3t1tqOFvOSm8ivz3nYaoKQl/3R8wla V+v3LTlyusufs53h1XAySOgYNNXVNwGpyIVFezY55vOQX4wOEkrcjNX1mvT28gtpp3bd L62YEXSWpETT23ouw1lwAFuYoN3sKihL4sBuVvqHSn9fqmfpvjSLtWOLj54BAOxGvISV cjlM+A3nDC+e9qfEEOHm0126w06sagr+4Pl+KMpqcgiEfkydc8FJKpbR6gPVIwRZEyiw w2sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=305uUdjHb+4q6rX/VtEHpGfhWHIvwXgJSbT7kEvsmqc=; b=Xe5JV/Nx2MVLLfbJ+KP691nuaI4nRybScY7N9BOctt28E1y1VcEQy6HlldTohaDoSk /hNTRvjGAIB5uegxhKVO4fHmtVN8AlXovGvVXdbjkwwCa1Bj4MYquLppLElHDBRM5k5m +51p5nu9WmV1spjP3jHA3qOfrK0Y/YgupzC52Wt+7TxEh0ZiYWVao59tAqiscRSRzFnN NxZ0mOCenFLeMZ1HsYnsfEnmWERRs5ssJc4ezhxqY/AaaWFyPBL+Z4znyMoak/T+KasU YAWAB0xa+/9j4akVAUnkBIaXuMJ2ya60DX8LuMraAmcPM4b0dMQ/tDX0tRP+VYvSyANZ c9Dw== X-Gm-Message-State: AOAM532g1Qlv0e0uP9Ox/qRsQlXM2IZlMXLwyOgSaDLbmS/54Y3EdnKu iVVB2UKc13+eNR7kZ7Z55cxqLlf2mFA= X-Google-Smtp-Source: ABdhPJx9xVX3ASg+WeencdrZzkgzWGbqqQnRW5BGCfq/uk40JCSaXjpwqJpGcMA8YXrpcy9ggfibng== X-Received: by 2002:adf:97d5:: with SMTP id t21mr8467827wrb.376.1598380434052; Tue, 25 Aug 2020 11:33:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o5sm6888193wmc.33.2020.08.25.11.33.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:53 -0700 (PDT) Message-Id: <69298aee24f317b03c39eef341f4d6ebebd88bc7.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:41 +0000 Subject: [PATCH v3 06/11] maintenance: add --task option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee A user may want to only run certain maintenance tasks in a certain order. Add the --task= option, which allows a user to specify an ordered list of tasks to run. These cannot be run multiple times, however. Here is where our array of maintenance_task pointers becomes critical. We can sort the array of pointers based on the task order, but we do not want to move the struct data itself in order to preserve the hashmap references. We use the hashmap to match the --task= arguments into the task struct data. Keep in mind that the 'enabled' member of the maintenance_task struct is a placeholder for a future 'maintenance..enabled' config option. Thus, we use the 'enabled' member to specify which tasks are run when the user does not specify any --task= arguments. The 'enabled' member should be ignored if --task= appears. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 9 ++++- builtin/gc.c | 66 +++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 27 +++++++++++++ 3 files changed, 98 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index fc5dbcf0b9..819ca41ab6 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -30,7 +30,9 @@ SUBCOMMANDS ----------- run:: - Run one or more maintenance tasks. + Run one or more maintenance tasks. If one or more `--task=` + options are specified, then those tasks are run in the provided + order. Otherwise, only the `gc` task is run. TASKS ----- @@ -63,6 +65,11 @@ OPTIONS --quiet:: Do not report progress or other information over `stderr`. +--task=:: + If this option is specified one or more times, then only run the + specified tasks in the specified order. See the 'TASKS' section + for the list of accepted `` values. + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 16e567992e..e94f263c77 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -701,7 +701,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix) } static const char * const builtin_maintenance_run_usage[] = { - N_("git maintenance run [--auto] [--[no-]quiet]"), + N_("git maintenance run [--auto] [--[no-]quiet] [--task=]"), NULL }; @@ -759,6 +759,9 @@ struct maintenance_task { const char *name; maintenance_task_fn *fn; unsigned enabled:1; + + /* -1 if not selected. */ + int selected_order; }; enum maintenance_task_label { @@ -781,13 +784,32 @@ static struct maintenance_task tasks[] = { }, }; +static int compare_tasks_by_selection(const void *a_, const void *b_) +{ + const struct maintenance_task *a, *b; + + a = (const struct maintenance_task *)&a_; + b = (const struct maintenance_task *)&b_; + + return b->selected_order - a->selected_order; +} + static int maintenance_run_tasks(struct maintenance_run_opts *opts) { - int i; + int i, found_selected = 0; int result = 0; + for (i = 0; !found_selected && i < TASK__COUNT; i++) + found_selected = tasks[i].selected_order >= 0; + + if (found_selected) + QSORT(tasks, TASK__COUNT, compare_tasks_by_selection); + for (i = 0; i < TASK__COUNT; i++) { - if (!tasks[i].enabled) + if (found_selected && tasks[i].selected_order < 0) + continue; + + if (!found_selected && !tasks[i].enabled) continue; if (tasks[i].fn(opts)) { @@ -799,20 +821,58 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) return result; } +static int task_option_parse(const struct option *opt, + const char *arg, int unset) +{ + int i, num_selected = 0; + struct maintenance_task *task = NULL; + + BUG_ON_OPT_NEG(unset); + + for (i = 0; i < TASK__COUNT; i++) { + if (tasks[i].selected_order >= 0) + num_selected++; + if (!strcasecmp(tasks[i].name, arg)) { + task = &tasks[i]; + } + } + + if (!task) { + error(_("'%s' is not a valid task"), arg); + return 1; + } + + if (task->selected_order >= 0) { + error(_("task '%s' cannot be selected multiple times"), arg); + return 1; + } + + task->selected_order = num_selected + 1; + + return 0; +} + static int maintenance_run(int argc, const char **argv, const char *prefix) { + int i; struct maintenance_run_opts opts; struct option builtin_maintenance_run_options[] = { OPT_BOOL(0, "auto", &opts.auto_flag, N_("run tasks based on the state of the repository")), OPT_BOOL(0, "quiet", &opts.quiet, N_("do not report progress or other information over stderr")), + OPT_CALLBACK_F(0, "task", NULL, N_("task"), + N_("run a specific task"), + PARSE_OPT_NONEG, task_option_parse), OPT_END() }; memset(&opts, 0, sizeof(opts)); opts.quiet = !isatty(2); + for (i = 0; i < TASK__COUNT; i++) + tasks[i].selected_order = -1; + argc = parse_options(argc, argv, prefix, builtin_maintenance_run_options, builtin_maintenance_run_usage, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index c0c4e6846e..792765aff7 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -25,4 +25,31 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet ' ' + GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ + git maintenance run --task=commit-graph 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-gc.txt" \ + git maintenance run --task=gc 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ + git maintenance run --task=commit-graph 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-both.txt" \ + git maintenance run --task=commit-graph --task=gc 2>/dev/null && + test_subcommand ! git gc --quiet err && + test_i18ngrep "is not a valid task" err +' + +test_expect_success 'run --task duplicate' ' + test_must_fail git maintenance run --task=gc --task=gc 2>err && + test_i18ngrep "cannot be selected multiple times" err +' + test_done From patchwork Tue Aug 25 18:33:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736183 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C50FD109B for ; Tue, 25 Aug 2020 18:34:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id ACF2F2076C for ; Tue, 25 Aug 2020 18:34:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cLWfivSX" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726656AbgHYSeV (ORCPT ); Tue, 25 Aug 2020 14:34:21 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43414 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726374AbgHYSd5 (ORCPT ); Tue, 25 Aug 2020 14:33:57 -0400 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 874C8C061574 for ; Tue, 25 Aug 2020 11:33:56 -0700 (PDT) Received: by mail-wm1-x343.google.com with SMTP id x9so3103236wmi.2 for ; Tue, 25 Aug 2020 11:33:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ve9lqwr3rt4c0jBLwiGFjmwc1D6qMi9ssw1Sq551o4Q=; b=cLWfivSXUejnsqhyN+321LDgKqfR5hfWNvRQJaDhyZqKBXXRh3aOFZZSes+XE6JdZi jCBfQCJphUdg4aUfhXw6OZR+9SWknj3j9awR++2uj7ey8eKNqX5IKixyxxewO30VTKhQ EKxTlPKmc23fg4LcrIbn6dJzG2ngFHEY0tI7+NfG4sGQUpfamAPlWINbgA4K1mrG686h IsqlwUORzZ9QDroHx4Jq813Yw+NeB4DFFwAE8VWvRM+lvrm93HlhdzD+Oii6jJ3DZErW XnTyfRiKsezPWcK/yhrC1lEe6bBrkRJm2zjXprHOwZZ5qiwYcXWOt6J2MOrbZvjfNF9Y dHuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ve9lqwr3rt4c0jBLwiGFjmwc1D6qMi9ssw1Sq551o4Q=; b=AL7tEZB0deAa1Qolktod8yiTrcVw5jBNRKvaCGYmYQWYJMcyQ9FPAVRfbFTTG9xkJ2 aZInkCLEWtoDjOUw8k3tKMhzgDJBmyfU5VuBs4JJ1ySYeehnqi2R2n8/o7wp1dwfcoNh 7Hxa6v+EC3D422XDoMs0adTcY6a/BalDj8qyQvc6Ryzgz4L+IbAFSXZNIX3aTr/IspyS PU1FpP6jAMGAOSq+Vvh1qiCuuylzPR4grUK1HR9AdiU/7M0VQ+GwtOyijHkz0GLZfoKS URkxqa01Y01tbfNLKlt51Q7EdVHq/6yM+4Ne+WxrMl4tuJmxlhyypuMGD5EKIGVTVK/V cBnA== X-Gm-Message-State: AOAM5334L9mfkg+CJMIinyjCgtT8wvUwqeKTzLtEpvHgudR0gTAQwpvq dbYElaT8CM6oVLJgB9NNbnINNhgom8s= X-Google-Smtp-Source: ABdhPJyoERXotOUb4PH6m9otYv6v3S8+s/JS1SYIGHj/nTjSWZ7aUcEm3D3oadGCn9r8cVL1nXOXbQ== X-Received: by 2002:a1c:2045:: with SMTP id g66mr3091791wmg.184.1598380434964; Tue, 25 Aug 2020 11:33:54 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 36sm12417324wrs.25.2020.08.25.11.33.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:54 -0700 (PDT) Message-Id: <3d513acdd8885d90393cbfa847c38e802ffddac9.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:42 +0000 Subject: [PATCH v3 07/11] maintenance: take a lock on the objects directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Performing maintenance on a Git repository involves writing data to the .git directory, which is not safe to do with multiple writers attempting the same operation. Ensure that only one 'git maintenance' process is running at a time by holding a file-based lock. Simply the presence of the .git/maintenance.lock file will prevent future maintenance. This lock is never committed, since it does not represent meaningful data. Instead, it is only a placeholder. If the lock file already exists, then fail with a warning. If '--auto' is specified, then instead no warning is shown and no tasks are attempted. This will become very important later when we implement the 'prefetch' task, as this is our stop-gap from creating a recursive process loop between 'git fetch' and 'git maintenance run --auto'. Signed-off-by: Derrick Stolee --- builtin/gc.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/builtin/gc.c b/builtin/gc.c index e94f263c77..1cebb7282d 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -798,6 +798,25 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) { int i, found_selected = 0; int result = 0; + struct lock_file lk; + struct repository *r = the_repository; + char *lock_path = xstrfmt("%s/maintenance", r->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) { + /* + * Another maintenance command is running. + * + * If --auto was provided, then it is likely due to a + * recursive process stack. Do not report an error in + * that case. + */ + if (!opts->auto_flag && !opts->quiet) + warning(_("lock file '%s' exists, skipping maintenance"), + lock_path); + free(lock_path); + return 0; + } + free(lock_path); for (i = 0; !found_selected && i < TASK__COUNT; i++) found_selected = tasks[i].selected_order >= 0; @@ -818,6 +837,7 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) } } + rollback_lock_file(&lk); return result; } From patchwork Tue Aug 25 18:33:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736185 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 327FD109B for ; Tue, 25 Aug 2020 18:34:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 165B42076C for ; Tue, 25 Aug 2020 18:34:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IcNynvpC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726666AbgHYSeX (ORCPT ); Tue, 25 Aug 2020 14:34:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726429AbgHYSd6 (ORCPT ); Tue, 25 Aug 2020 14:33:58 -0400 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A981FC061786 for ; Tue, 25 Aug 2020 11:33:57 -0700 (PDT) Received: by mail-wr1-x441.google.com with SMTP id 2so2663136wrj.10 for ; Tue, 25 Aug 2020 11:33:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vK6iK0+RvphJs0HeXxn440+72BWzbhU3tPC7lZ3S1Hw=; b=IcNynvpCeH0xX9Gr6zDB5almb5EcDBrkt9TcsRoWJ5kuBgM+vslTwXSkgdDvFf/AMa kSi4QAqdHYBkAwwiOlXJO6K6un8DhWtfOgUazjKjtFzc3693BkdFfbt0NqSAMABzzNNE CHfWyrEHbyKTR6FFIn2JBMulGTbqtIFSN5gV92XWTNVhUWEgfjJiKcUdi5rcgdnsAROh +ET32I2IDYMcJ4vHAHlrUJGbD71oGr6AwYySDA75ye+4uX/gyNKW7FufFFMTss3WCKMd QbeC1xWMHvCnXnxuUQUL4RgBLWwMY/ruQ2+NfHGLE7VMxUGT4s91hL0rHVwiKVb3KO+h miZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vK6iK0+RvphJs0HeXxn440+72BWzbhU3tPC7lZ3S1Hw=; b=gAAYLw/X+WpEKbeKgGVdERVWU4ZWurLFAr7mcbY2DD+RcLAekomJDhEDog3grmstoR /SlUzDXDudUqI3k4/LcEZFZ+YxJyfjw6loqQRFHTy+LxzMfyKUffwnwCaWl7jUplDhps AFaHZ4a22zqWh5Eo6P+mcLPREs32xLh0ehSgTqzXJL7Gx11oZQ06z9/yxL6fKfXjYPA/ H6jiI9VpxOYRTUeFvH+QvYMmTa7IQnYQlYThQun1nSaFNuHuThXnSxNKmjennmH/f4cu lsVexEoYyEht+AQ8+PjHbc5pERH8CJ3m7d1wyO8TbgH3RQUsAKSoTCKtCUNC9AKJUiAP D+AQ== X-Gm-Message-State: AOAM531mvkG41nTFsAWEt3Jb18LWNj3p6XmX1P2ZV/7+W8bi2u6+5kVc hJpDxa3bp3eQ9Txdaip1k6fw2JLc0C0= X-Google-Smtp-Source: ABdhPJzjFxcRgwpHAekJ+6B14zqK4Rz/oTVCw2TtNDy/+TmH96CDBuWvUXnEFYbImlb6VbmlATDARw== X-Received: by 2002:a05:6000:12c1:: with SMTP id l1mr11301403wrx.270.1598380436120; Tue, 25 Aug 2020 11:33:56 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h7sm7578222wmf.43.2020.08.25.11.33.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:55 -0700 (PDT) Message-Id: <712f5f2d8ec3548c152c20b50b75a58b773ce3f4.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:43 +0000 Subject: [PATCH v3 08/11] maintenance: create maintenance..enabled config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Currently, a normal run of "git maintenance run" will only run the 'gc' task, as it is the only one enabled. This is mostly for backwards- compatible reasons since "git maintenance run --auto" commands replaced previous "git gc --auto" commands after some Git processes. Users could manually run specific maintenance tasks by calling "git maintenance run --task=" directly. Allow users to customize which steps are run automatically using config. The 'maintenance..enabled' option then can turn on these other tasks (or turn off the 'gc' task). Signed-off-by: Derrick Stolee --- Documentation/config.txt | 2 ++ Documentation/config/maintenance.txt | 6 ++++++ Documentation/git-maintenance.txt | 14 +++++++++----- builtin/gc.c | 19 +++++++++++++++++++ t/t7900-maintenance.sh | 8 ++++++++ 5 files changed, 44 insertions(+), 5 deletions(-) create mode 100644 Documentation/config/maintenance.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index 3042d80978..f93b6837e4 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -398,6 +398,8 @@ include::config/mailinfo.txt[] include::config/mailmap.txt[] +include::config/maintenance.txt[] + include::config/man.txt[] include::config/merge.txt[] diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt new file mode 100644 index 0000000000..4402b8b49f --- /dev/null +++ b/Documentation/config/maintenance.txt @@ -0,0 +1,6 @@ +maintenance..enabled:: + This boolean config option controls whether the maintenance task + with name `` is run when no `--task` option is specified to + `git maintenance run`. These config values are ignored if a + `--task` option exists. By default, only `maintenance.gc.enabled` + is true. diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 819ca41ab6..6abcb8255a 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -30,9 +30,11 @@ SUBCOMMANDS ----------- run:: - Run one or more maintenance tasks. If one or more `--task=` - options are specified, then those tasks are run in the provided - order. Otherwise, only the `gc` task is run. + Run one or more maintenance tasks. If one or more `--task` options + are specified, then those tasks are run in that order. Otherwise, + the tasks are determined by which `maintenance..enabled` + config options are true. By default, only `maintenance.gc.enabled` + is true. TASKS ----- @@ -67,8 +69,10 @@ OPTIONS --task=:: If this option is specified one or more times, then only run the - specified tasks in the specified order. See the 'TASKS' section - for the list of accepted `` values. + specified tasks in the specified order. If no `--task=` + arguments are specified, then only the tasks with + `maintenance..enabled` configured as `true` are considered. + See the 'TASKS' section for the list of accepted `` values. GIT --- diff --git a/builtin/gc.c b/builtin/gc.c index 1cebb7282d..67a8d405a1 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -841,6 +841,24 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) return result; } +static void initialize_task_config(void) +{ + int i; + struct strbuf config_name = STRBUF_INIT; + for (i = 0; i < TASK__COUNT; i++) { + int config_value; + + strbuf_setlen(&config_name, 0); + strbuf_addf(&config_name, "maintenance.%s.enabled", + tasks[i].name); + + if (!git_config_get_bool(config_name.buf, &config_value)) + tasks[i].enabled = config_value; + } + + strbuf_release(&config_name); +} + static int task_option_parse(const struct option *opt, const char *arg, int unset) { @@ -889,6 +907,7 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) memset(&opts, 0, sizeof(opts)); opts.quiet = !isatty(2); + initialize_task_config(); for (i = 0; i < TASK__COUNT; i++) tasks[i].selected_order = -1; diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 792765aff7..290abb7832 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -25,6 +25,14 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet .enabled' ' + git config maintenance.gc.enabled false && + git config maintenance.commit-graph.enabled true && + GIT_TRACE2_EVENT="$(pwd)/run-config.txt" git maintenance run 2>err && + test_subcommand ! git gc --quiet ' ' GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ git maintenance run --task=commit-graph 2>/dev/null && From patchwork Tue Aug 25 18:33:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736191 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01AD6109B for ; Tue, 25 Aug 2020 18:34:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D83922074D for ; Tue, 25 Aug 2020 18:34:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qLqyi1zZ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726633AbgHYSei (ORCPT ); Tue, 25 Aug 2020 14:34:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43430 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726556AbgHYSeA (ORCPT ); Tue, 25 Aug 2020 14:34:00 -0400 Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50C43C061796 for ; Tue, 25 Aug 2020 11:34:00 -0700 (PDT) Received: by mail-wm1-x336.google.com with SMTP id 83so3567962wme.4 for ; Tue, 25 Aug 2020 11:34:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=vM0TuBEUY0vrapOf9Ut/0RUxanAqe66yuP0tCUJZssI=; b=qLqyi1zZ93D0wuHFjTV+O6aArs2RPob3NsCrqEZQeAU6wasFLp5lRIVluZeLkiNZ0s OWVMlfcqj1qQPTTPLujvRZE3G1TaPwhZQjp4+oEKZg7KkMWsXt/tMSKVgUJEjhfyXMBH Ge+5iCVuvvbxffsu5SPyTMS1hWBsLrGConS+klqWX2XgVsqO3T5bAvYZANpQR1oOROSd de65bq0tH0EVJFIzIXzDKZYGNSasTcDY6Yp+GaIwiKW3Ul31wVaoCQiGv31L7w0r6nO6 VHXepYv/tvsYrj9BYi/OhGcxtol3o9J7aQmWTqS8+xphbDk4xjkDDRHFos4C871bYpML VUOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=vM0TuBEUY0vrapOf9Ut/0RUxanAqe66yuP0tCUJZssI=; b=DEINnVtyrEJ6K4eRb/4HilJ1ow6bR21D0OC+I1bmfmMZXDWevyYsJpvbarOfwwZyMA hxDeVlJHtX5Udk0Gl8OJHhRe05HoyQaDJFhc9LW3Bu201e4CqTfGIicLOHYBEgqrLhMr OgzMAHxSWaYJ2KvZDWSAslhwdTWorWyMgGuWSGE6WAvSEUOlKCT7iR409HrqFDIxtFjG /eIJ304+6p6IuGnMfS+ZfZbf/GuB8tr2oMsRWNl4nu46oEuffCaBs3qxYDnotQNUBMGH el+i+gZM3HGBqb58JZ0TCKxg7/O2I7a2xj7OcYzq7jPv+7M9DL/wF34Gtfo/LlNqQ89L PBGA== X-Gm-Message-State: AOAM532CVwrN1QZwHzAPdDs6nZPv/x3QMlqVPUBzvo9LQd0wMVNqGvYa j8uiLTmaiTgYuzKAbi5ruA5zLH7Kado= X-Google-Smtp-Source: ABdhPJzXBTrkJ3CBg4i5ATBhFbbjhmnD77tcBYe/NnHNbOIZDd6ze3pCvjUNGvz3pkfK9bFc3nVixw== X-Received: by 2002:a7b:c00c:: with SMTP id c12mr3486700wmb.54.1598380437045; Tue, 25 Aug 2020 11:33:57 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f6sm10785252wme.32.2020.08.25.11.33.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:56 -0700 (PDT) Message-Id: <69d3b48fd4474783579297d3d25f04dd8f1e7673.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:44 +0000 Subject: [PATCH v3 09/11] maintenance: use pointers to check --auto Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git maintenance run' command has an '--auto' option. This is used by other Git commands such as 'git commit' or 'git fetch' to check if maintenance should be run after adding data to the repository. Previously, this --auto option was only used to add the argument to the 'git gc' command as part of the 'gc' task. We will be expanding the other tasks to perform a check to see if they should do work as part of the --auto flag, when they are enabled by config. First, update the 'gc' task to perform the auto check inside the maintenance process. This prevents running an extra 'git gc --auto' command when not needed. It also shows a model for other tasks. Second, use the 'auto_condition' function pointer as a signal for whether we enable the maintenance task under '--auto'. For instance, we do not want to enable the 'fetch' task in '--auto' mode, so that function pointer will remain NULL. Now that we are not automatically calling 'git gc', a test in t5514-fetch-multiple.sh must be changed to watch for 'git maintenance' instead. We continue to pass the '--auto' option to the 'git gc' command when necessary, because of the gc.autoDetach config option changes behavior. Likely, we will want to absorb the daemonizing behavior implied by gc.autoDetach as a maintenance.autoDetach config option. Signed-off-by: Derrick Stolee --- builtin/gc.c | 16 ++++++++++++++++ t/t5514-fetch-multiple.sh | 2 +- t/t7900-maintenance.sh | 2 +- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index 67a8d405a1..709d13553b 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -755,9 +755,17 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts) typedef int maintenance_task_fn(struct maintenance_run_opts *opts); +/* + * An auto condition function returns 1 if the task should run + * and 0 if the task should NOT run. See needs_to_gc() for an + * example. + */ +typedef int maintenance_auto_fn(void); + struct maintenance_task { const char *name; maintenance_task_fn *fn; + maintenance_auto_fn *auto_condition; unsigned enabled:1; /* -1 if not selected. */ @@ -776,6 +784,7 @@ static struct maintenance_task tasks[] = { [TASK_GC] = { "gc", maintenance_task_gc, + need_to_gc, 1, }, [TASK_COMMIT_GRAPH] = { @@ -831,6 +840,11 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) if (!found_selected && !tasks[i].enabled) continue; + if (opts->auto_flag && + (!tasks[i].auto_condition || + !tasks[i].auto_condition())) + continue; + if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); result = 1; @@ -845,6 +859,8 @@ static void initialize_task_config(void) { int i; struct strbuf config_name = STRBUF_INIT; + gc_config(); + for (i = 0; i < TASK__COUNT; i++) { int config_value; diff --git a/t/t5514-fetch-multiple.sh b/t/t5514-fetch-multiple.sh index de8e2f1531..bd202ec6f3 100755 --- a/t/t5514-fetch-multiple.sh +++ b/t/t5514-fetch-multiple.sh @@ -108,7 +108,7 @@ test_expect_success 'git fetch --multiple (two remotes)' ' GIT_TRACE=1 git fetch --multiple one two 2>trace && git branch -r > output && test_cmp ../expect output && - grep "built-in: git gc" trace >gc && + grep "built-in: git maintenance" trace >gc && test_line_count = 1 gc ) ' diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 290abb7832..4f6a04ddb1 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -21,7 +21,7 @@ test_expect_success 'run [--auto|--quiet]' ' GIT_TRACE2_EVENT="$(pwd)/run-no-quiet.txt" \ git maintenance run --no-quiet 2>/dev/null && test_subcommand git gc --quiet X-Patchwork-Id: 11736187 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0F4A0109B for ; Tue, 25 Aug 2020 18:34:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E5C562074D for ; Tue, 25 Aug 2020 18:34:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="c2BmGxBr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726706AbgHYSeY (ORCPT ); Tue, 25 Aug 2020 14:34:24 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43398 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726466AbgHYSeA (ORCPT ); Tue, 25 Aug 2020 14:34:00 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 562FDC061757 for ; Tue, 25 Aug 2020 11:33:59 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id 83so3568044wme.4 for ; Tue, 25 Aug 2020 11:33:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=2uAnxf2B5tiekxp+tVpWDPB8CQkwx46JIztlNDXM5IM=; b=c2BmGxBrYPp4lYJz1u8zrtxuq21ZRB6CWcEM0h7K5zEyqQ3GIv7uY5eGUdaecn7lqo JdOFxC1e2f4PyZ9TnBPS4L5LEMmzYFiiX8ezgefPNIW9GV+k/u+aMKNXrLLB8G+ai2+4 sZG2O7/QjlWewj44ycM2g6uExGyrbH+kKVUy29inc/JYJ6XF3uKeC9TLcNNIcV2PCngK M/f8P7zxYria8ApHhQ342YZng63WjD4kVS6J7/+/lNtAXco1u8vb7EUpCXTeA6kvkSqp XFDjouDhnESjB9omtIn9+K1BxQMj6wSDfg5FT4eYLUnByqaM4Pm2ANAtLVh6kV1HVeu4 WQJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=2uAnxf2B5tiekxp+tVpWDPB8CQkwx46JIztlNDXM5IM=; b=L6RnkRg+9zba50cjwearnfnW7cR53LyY8Gcc+BIdlcrKeujDfhErlPL9LMqowRBRMc 7dGvd/JR3kNk6NmFkSwIwjTMSEbevzrQ2LK00cPmBJa8tlRYcei6g+bQGPEGG8DBzBZl xIRWfhxZ1XHcQNZYqP5K4SHy96Z04r6q6y+tAt/SXKjpi/urCk6FKI7MK3TCneoTk/kc ttoYInuPJwUiGpKjeNhcaChCx7RkLqPAEURne/KEmtmkS5ktkAyUtVRW4YxlXLlhHH08 0JHM1dxwreG+6dLPJXQTNztaWLImwB9kluh2jXY4DF3BQVG2LWIXkk6jz2DyE9R/q4tf g2JA== X-Gm-Message-State: AOAM533EoFC7dgiAyu2LEddmT8bUgwCDS0cFuPXCKSryS4J8LHkT/Zfa LQKrDtSncbftAaEe4L0Lv0APFpDWHm8= X-Google-Smtp-Source: ABdhPJy/fYfMpploBYXiyo7c0HIr5sTNuOHiFF53d9oArFrnxbqd33i5VlhcXSEM7+60EBIuLLN6KQ== X-Received: by 2002:a1c:a585:: with SMTP id o127mr3399226wme.102.1598380437826; Tue, 25 Aug 2020 11:33:57 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f6sm10785358wme.32.2020.08.25.11.33.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:57 -0700 (PDT) Message-Id: <4c3115fe3522bee47ba1f8f5e847e99ad7e56d40.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:45 +0000 Subject: [PATCH v3 10/11] maintenance: add auto condition for commit-graph task Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Instead of writing a new commit-graph in every 'git maintenance run --auto' process (when maintenance.commit-graph.enalbed is configured to be true), only write when there are "enough" commits not in a commit-graph file. This count is controlled by the maintenance.commit-graph.auto config option. To compute the count, use a depth-first search starting at each ref, and leaving markers using the PARENT1 flag. If this count reaches the limit, then terminate early and start the task. Otherwise, this operation will peel every ref and parse the commit it points to. If these are all in the commit-graph, then this is typically a very fast operation. Users with many refs might feel a slow-down, and hence could consider updating their limit to be very small. A negative value will force the step to run every time. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 10 ++++ builtin/gc.c | 82 ++++++++++++++++++++++++++++ object.h | 1 + 3 files changed, 93 insertions(+) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index 4402b8b49f..7cc6700d57 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -4,3 +4,13 @@ maintenance..enabled:: `git maintenance run`. These config values are ignored if a `--task` option exists. By default, only `maintenance.gc.enabled` is true. + +maintenance.commit-graph.auto:: + This integer config option controls how often the `commit-graph` task + should be run as part of `git maintenance run --auto`. If zero, then + the `commit-graph` task will not run with the `--auto` option. A + negative value will force the task to run every time. Otherwise, a + positive value implies the command should run when the number of + reachable commits that are not in the commit-graph file is at least + the value of `maintenance.commit-graph.auto`. The default value is + 100. diff --git a/builtin/gc.c b/builtin/gc.c index 709d13553b..8c4edf19ba 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -28,6 +28,7 @@ #include "blob.h" #include "tree.h" #include "promisor-remote.h" +#include "refs.h" #define FAILED_RUN "failed to run %s" @@ -710,6 +711,86 @@ struct maintenance_run_opts { int quiet; }; +/* Remember to update object flag allocation in object.h */ +#define SEEN (1u<<0) + +struct cg_auto_data { + int num_not_in_graph; + int limit; +}; + +static int dfs_on_ref(const char *refname, + const struct object_id *oid, int flags, + void *cb_data) +{ + struct cg_auto_data *data = (struct cg_auto_data *)cb_data; + int result = 0; + struct object_id peeled; + struct commit_list *stack = NULL; + struct commit *commit; + + if (!peel_ref(refname, &peeled)) + oid = &peeled; + if (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT) + return 0; + + commit = lookup_commit(the_repository, oid); + if (!commit) + return 0; + if (parse_commit(commit)) + return 0; + + commit_list_append(commit, &stack); + + while (!result && stack) { + struct commit_list *parent; + + commit = pop_commit(&stack); + + for (parent = commit->parents; parent; parent = parent->next) { + if (parse_commit(parent->item) || + commit_graph_position(parent->item) != COMMIT_NOT_FROM_GRAPH || + parent->item->object.flags & SEEN) + continue; + + parent->item->object.flags |= SEEN; + data->num_not_in_graph++; + + if (data->num_not_in_graph >= data->limit) { + result = 1; + break; + } + + commit_list_append(parent->item, &stack); + } + } + + free_commit_list(stack); + return result; +} + +static int should_write_commit_graph(void) +{ + int result; + struct cg_auto_data data; + + data.num_not_in_graph = 0; + data.limit = 100; + git_config_get_int("maintenance.commit-graph.auto", + &data.limit); + + if (!data.limit) + return 0; + if (data.limit < 0) + return 1; + + result = for_each_ref(dfs_on_ref, &data); + + clear_commit_marks_all(SEEN); + + return result; +} + static int run_write_commit_graph(struct maintenance_run_opts *opts) { struct child_process child = CHILD_PROCESS_INIT; @@ -790,6 +871,7 @@ static struct maintenance_task tasks[] = { [TASK_COMMIT_GRAPH] = { "commit-graph", maintenance_task_commit_graph, + should_write_commit_graph, }, }; diff --git a/object.h b/object.h index 96a2105859..20b18805f0 100644 --- a/object.h +++ b/object.h @@ -73,6 +73,7 @@ struct object_array { * sha1-name.c: 20 * list-objects-filter.c: 21 * builtin/fsck.c: 0--3 + * builtin/gc.c: 0 * builtin/index-pack.c: 2021 * builtin/pack-objects.c: 20 * builtin/reflog.c: 10--12 From patchwork Tue Aug 25 18:33:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11736189 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 64620109B for ; Tue, 25 Aug 2020 18:34:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 456F52071E for ; Tue, 25 Aug 2020 18:34:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cGKnO3OW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726391AbgHYSed (ORCPT ); Tue, 25 Aug 2020 14:34:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726541AbgHYSeA (ORCPT ); Tue, 25 Aug 2020 14:34:00 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 10458C061795 for ; Tue, 25 Aug 2020 11:34:00 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id p17so8155322wrj.8 for ; Tue, 25 Aug 2020 11:33:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3BP+V47fkAW0XOrc+qRv9CzrUpLDeEi54vnAws/gvxo=; b=cGKnO3OWGxDmGFVmOOclzON5s9MChrvgP1DIL/G+xcCu9vjMv9v/XX0I0ZDQtjllc/ m/BPfDEr84EQQu3nomvQAvSq7GJkhKbdSdkRHElXoqcPhgEPf+ueEY5g/TQV/uoLMXTe +EinptKagd5tm/IpOyFjzJ3AYZax4pDj9FgzfsNPQo7m4gQSqXJjOwETNZkLcXFApX5V WihvqpphjujzlHDcnOg51Jq2hzHYjUd2EjNRyNEj7k9hKoRf39LUnA4Jf68iBjErsmI5 4058PkDl3hZ9gBjp4pjf0HizhOqe2yeiT1gj/mA4GVnp3SPsBKUNgFFmBKrlWqWWj+bZ 6lEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3BP+V47fkAW0XOrc+qRv9CzrUpLDeEi54vnAws/gvxo=; b=UsODAy317HcKgYMkfjmXB8s9iJU2UogFyM+yS2ULlmR1AZhG/peHs+CZQsJkzDgCjU 4zsgfemUlRdS8228wFRl6wngvxd2dDKyFpWqMDN03s5zfSctFPTrIZ0nvB6RyCFCLKqF oz/K3a9FVrFSaZdLWLclt3uDjN2//qH9074Kww9XHIdtRtjLrAtNL/dPHW6Bhf4a00K9 bnc2j1/9ohx5+rrh9m+m3Y28fKSgXLJaxjKINUnFA18hiU1xWkrG4ZafNyhIJCPDEjxB JBFxt6FDY5YlXr2EhlbxcnUnD48T6bSI3uQSP8EEqaH/7P/JP2KtQgAOLd6eR7XhQjyI zMKA== X-Gm-Message-State: AOAM533dxW2/xFAL1/+CDRbY1ozx9xSpADs3SX1PMn/sHOiNgk9mMeKK bqW/x458oGIZEcqP6NsgR4RVkHrlF2s= X-Google-Smtp-Source: ABdhPJyTqUZQuZtFNElpR/NtbK3UPLyvBezBimQE//2KA5dnLIgouv+V2wh0mkcEefJW/dVepkklyQ== X-Received: by 2002:adf:e90f:: with SMTP id f15mr11684090wrm.310.1598380438676; Tue, 25 Aug 2020 11:33:58 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f3sm7355228wmb.35.2020.08.25.11.33.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:33:58 -0700 (PDT) Message-Id: <652a8eac57d04a51820c7a5b45031b50c5188e7b.1598380427.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:33:46 +0000 Subject: [PATCH v3 11/11] maintenance: add trace2 regions for task execution Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Signed-off-by: Derrick Stolee --- builtin/gc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builtin/gc.c b/builtin/gc.c index 8c4edf19ba..c3bcdc1167 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -927,10 +927,12 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) !tasks[i].auto_condition())) continue; + trace2_region_enter("maintenance", tasks[i].name, r); if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); result = 1; } + trace2_region_leave("maintenance", tasks[i].name, r); } rollback_lock_file(&lk);