From patchwork Thu Aug 6 15:48:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703797 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB21D138C for ; Thu, 6 Aug 2020 17:56:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 50F11206C3 for ; Thu, 6 Aug 2020 17:56:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="rD+eQZ8c" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728016AbgHFRzr (ORCPT ); Thu, 6 Aug 2020 13:55:47 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38080 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728012AbgHFQSn (ORCPT ); Thu, 6 Aug 2020 12:18:43 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B4A5C0F26CC for ; Thu, 6 Aug 2020 08:48:46 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id r2so39385054wrs.8 for ; Thu, 06 Aug 2020 08:48:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=esMNGJkQluoXLB0Pzy2j+iWmXD3pMw8Y0EySjVmWBhU=; b=rD+eQZ8cA5KCns6b4AGtswuS+ZrxADVhYVVDbf2SUxbjXJnmzF7lIEbD5GYqjbv5wv IeY+lUSy7QOtQpIRVVCk4yFzjXboNgyT+m8vaY5oCjoISbusB2LCnQ/KVpm8Z6fiuYj2 fR4ZH5goQQVl3vnDzw5jnya44FeyGyJGW76wtzDjwqL40mYzRrAXsuAGzsre2ABG9JiF Wi/D0GlUF4OpRIyt0JHY6TVy1RtrHSxcVh/j+YSAWAn5RjUAolBZrDENU6wCaeJD3FRo JuZ8zCmGR7NI8yWDwz0Sv04ffAJ7w93HCIs04nUO9RaXsO5f9D8gYn1abK8DbSbFps7B 21RA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=esMNGJkQluoXLB0Pzy2j+iWmXD3pMw8Y0EySjVmWBhU=; b=aVUJQXJ1av3lhujkrj4iGFkHUixndNSTkSntGH/zaobGkyMiuRv/2oGRMEuCnf7Pt2 gDfvlMkBhXYh3o7BTnhp5hb8ULwV8k0IltXxdsspDBXKaN9IPTVmBHOn8cuajZeYGsyd Ve9QLyvk1d6ZnCOAGH5by2Ldo52Si+wxdS6Gf6Oi3ziUcVH2CzRjBz9Q29jVIyYt9o4k 37KvW/2tNFak+NWSWpdnyB6rdtyuOTKLjfQ/1nwfqneJZV6zzanwg4umreZWoeZ/djgv bnTpKFG0oBctNzCL2UYzYAqlGUMHB3D230axHJ21sNXLjq41lSgP0k3znCE8s3YZZUC5 xNcg== X-Gm-Message-State: AOAM530PFeJq+D4mFwjNMXDsDrPA+vO6uTjODdYvqd/NIHGIt/Uf0vnA 2x1aRMd87epUFq6l66cFevmU9Z0y X-Google-Smtp-Source: ABdhPJzD8+pU1qu9oAHtKS8hwdBBXjCVmzE2+egYfqulEOF8t39kInlWwnuch8L0h3jivhCMHQCQ/Q== X-Received: by 2002:a5d:4401:: with SMTP id z1mr7787657wrq.305.1596728924126; Thu, 06 Aug 2020 08:48:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b142sm7286267wmd.19.2020.08.06.08.48.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:43 -0700 (PDT) Message-Id: <2b9deb6d6a23e53bec75e109f2e3ef9217420425.1596728921.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:31 +0000 Subject: [PATCH 01/11] maintenance: create basic maintenance runner Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'gc' builtin is our current entrypoint for automatically maintaining a repository. This one tool does many operations, such as repacking the repository, packing refs, and rewriting the commit-graph file. The name implies it performs "garbage collection" which means several different things, and some users may not want to use this operation that rewrites the entire object database. Create a new 'maintenance' builtin that will become a more general- purpose command. To start, it will only support the 'run' subcommand, but will later expand to add subcommands for scheduling maintenance in the background. For now, the 'maintenance' builtin is a thin shim over the 'gc' builtin. In fact, the only option is the '--auto' toggle, which is handed directly to the 'gc' builtin. The current change is isolated to this simple operation to prevent more interesting logic from being lost in all of the boilerplate of adding a new builtin. Use existing builtin/gc.c file because we want to share code between the two builtins. It is possible that we will have 'maintenance' replace the 'gc' builtin entirely at some point, leaving 'git gc' as an alias for some specific arguments to 'git maintenance run'. Create a new test_subcommand helper that allows us to test if a certain subcommand was run. It requires storing the GIT_TRACE2_EVENT logs in a file. A negation mode is available that will be used in later tests. Helped-by: Jonathan Nieder Signed-off-by: Derrick Stolee Reviewed-by: Jonathan Nieder --- .gitignore | 1 + Documentation/git-maintenance.txt | 57 +++++++++++++++++++++++++++++++ builtin.h | 1 + builtin/gc.c | 57 +++++++++++++++++++++++++++++++ git.c | 1 + t/t7900-maintenance.sh | 19 +++++++++++ t/test-lib-functions.sh | 33 ++++++++++++++++++ 7 files changed, 169 insertions(+) create mode 100644 Documentation/git-maintenance.txt create mode 100755 t/t7900-maintenance.sh diff --git a/.gitignore b/.gitignore index ee509a2ad2..a5808fa30d 100644 --- a/.gitignore +++ b/.gitignore @@ -90,6 +90,7 @@ /git-ls-tree /git-mailinfo /git-mailsplit +/git-maintenance /git-merge /git-merge-base /git-merge-index diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt new file mode 100644 index 0000000000..34cd2b4417 --- /dev/null +++ b/Documentation/git-maintenance.txt @@ -0,0 +1,57 @@ +git-maintenance(1) +================== + +NAME +---- +git-maintenance - Run tasks to optimize Git repository data + + +SYNOPSIS +-------- +[verse] +'git maintenance' run [] + + +DESCRIPTION +----------- +Run tasks to optimize Git repository data, speeding up other Git commands +and reducing storage requirements for the repository. ++ +Git commands that add repository data, such as `git add` or `git fetch`, +are optimized for a responsive user experience. These commands do not take +time to optimize the Git data, since such optimizations scale with the full +size of the repository while these user commands each perform a relatively +small action. ++ +The `git maintenance` command provides flexibility for how to optimize the +Git repository. + +SUBCOMMANDS +----------- + +run:: + Run one or more maintenance tasks. + +TASKS +----- + +gc:: + Cleanup unnecessary files and optimize the local repository. "GC" + stands for "garbage collection," but this task performs many + smaller tasks. This task can be rather expensive for large + repositories, as it repacks all Git objects into a single pack-file. + It can also be disruptive in some situations, as it deletes stale + data. + +OPTIONS +------- +--auto:: + When combined with the `run` subcommand, run maintenance tasks + only if certain thresholds are met. For example, the `gc` task + runs when the number of loose objects exceeds the number stored + in the `gc.auto` config setting, or when the number of pack-files + exceeds the `gc.autoPackLimit` config setting. + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/builtin.h b/builtin.h index a5ae15bfe5..17c1c0ce49 100644 --- a/builtin.h +++ b/builtin.h @@ -167,6 +167,7 @@ int cmd_ls_tree(int argc, const char **argv, const char *prefix); int cmd_ls_remote(int argc, const char **argv, const char *prefix); int cmd_mailinfo(int argc, const char **argv, const char *prefix); int cmd_mailsplit(int argc, const char **argv, const char *prefix); +int cmd_maintenance(int argc, const char **argv, const char *prefix); int cmd_merge(int argc, const char **argv, const char *prefix); int cmd_merge_base(int argc, const char **argv, const char *prefix); int cmd_merge_index(int argc, const char **argv, const char *prefix); diff --git a/builtin/gc.c b/builtin/gc.c index aafa0946f5..e4f0ce1c86 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -699,3 +699,60 @@ int cmd_gc(int argc, const char **argv, const char *prefix) return 0; } + +static const char * const builtin_maintenance_usage[] = { + N_("git maintenance run []"), + NULL +}; + +struct maintenance_opts { + int auto_flag; +}; + +static int maintenance_task_gc(struct maintenance_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_push(&child.args, "gc"); + + if (opts->auto_flag) + strvec_push(&child.args, "--auto"); + + close_object_store(the_repository->objects); + return run_command(&child); +} + +static int maintenance_run(struct maintenance_opts *opts) +{ + return maintenance_task_gc(opts); +} + +int cmd_maintenance(int argc, const char **argv, const char *prefix) +{ + static struct maintenance_opts opts; + static struct option builtin_maintenance_options[] = { + OPT_BOOL(0, "auto", &opts.auto_flag, + N_("run tasks based on the state of the repository")), + OPT_END() + }; + + memset(&opts, 0, sizeof(opts)); + + if (argc == 2 && !strcmp(argv[1], "-h")) + usage_with_options(builtin_maintenance_usage, + builtin_maintenance_options); + + argc = parse_options(argc, argv, prefix, + builtin_maintenance_options, + builtin_maintenance_usage, + PARSE_OPT_KEEP_UNKNOWN); + + if (argc == 1) { + if (!strcmp(argv[0], "run")) + return maintenance_run(&opts); + } + + usage_with_options(builtin_maintenance_usage, + builtin_maintenance_options); +} diff --git a/git.c b/git.c index 8bd1d7551d..24f250d29a 100644 --- a/git.c +++ b/git.c @@ -529,6 +529,7 @@ static struct cmd_struct commands[] = { { "ls-tree", cmd_ls_tree, RUN_SETUP }, { "mailinfo", cmd_mailinfo, RUN_SETUP_GENTLY | NO_PARSEOPT }, { "mailsplit", cmd_mailsplit, NO_PARSEOPT }, + { "maintenance", cmd_maintenance, RUN_SETUP_GENTLY | NO_PARSEOPT }, { "merge", cmd_merge, RUN_SETUP | NEED_WORK_TREE }, { "merge-base", cmd_merge_base, RUN_SETUP }, { "merge-file", cmd_merge_file, RUN_SETUP_GENTLY }, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh new file mode 100755 index 0000000000..c4b9b4a6fe --- /dev/null +++ b/t/t7900-maintenance.sh @@ -0,0 +1,19 @@ +#!/bin/sh + +test_description='git maintenance builtin' + +. ./test-lib.sh + +test_expect_success 'help text' ' + test_expect_code 129 git maintenance -h 2>err && + test_i18ngrep "usage: git maintenance run" err +' + +test_expect_success 'run [--auto]' ' + GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run && + GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto && + test_subcommand git gc ... < +# +# For example, to look for an invocation of "git upload-pack +# /path/to/repo" +# +# GIT_TRACE2_EVENT=event.log git fetch ... && +# test_subcommand git upload-pack "$PATH" X-Patchwork-Id: 11703591 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 14EFC13B1 for ; Thu, 6 Aug 2020 16:54:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6BB792311A for ; Thu, 6 Aug 2020 16:54:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jVUV9IVk" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729768AbgHFQyg (ORCPT ); Thu, 6 Aug 2020 12:54:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729087AbgHFQun (ORCPT ); Thu, 6 Aug 2020 12:50:43 -0400 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FB0FC0F26CE for ; Thu, 6 Aug 2020 08:48:46 -0700 (PDT) Received: by mail-wr1-x441.google.com with SMTP id l2so33935959wrc.7 for ; Thu, 06 Aug 2020 08:48:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RwtwfO3iJG5DwWl3yJq5hteOnX7vVssVEJSokU49/uk=; b=jVUV9IVk8jh1pAxCp1NX6X2P3ewg/qVCpRpll8+9tE1VHmukQG0dOtzoKBNtmf7Qf3 j4V6WZpUD20fwqKcYudmTYY7mahSUqceDjyfsBprLmlFQNO6k84UFhiuf84TeaN879vB PxItYZwXNK2+T9U1zKwrIKE4HotU4FfPIVFg54rmeJqQuJEn5jTLRIUVIz2HBrFsA7bi D29b8BWPFS59C/zjTnYs1XY1XX0DW1VWoa6qE7x6MmtZ8ft4jC2eQ44aEJDN5MN8fVWp BNFVcC9/PPoNlqYQ3Y51PCCJkZstEodlCa5wIskd4DXFrwAgqyJUoK9guh8U4PU6G2o2 +PWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RwtwfO3iJG5DwWl3yJq5hteOnX7vVssVEJSokU49/uk=; b=QN4Oe005Ns8MFvA2r4XFMa3ZO0Fgn6+IKGUA8sgWcBBpvf81JVSxkMmA8rDcYfwtFl pOI/XNNshvUISkpYRM+7HCUmiPOaIcy4JXpE1To2/TBE5FfE74NVNODWcw34KY+x4KAk qT3SDz8R1m85TjDI0yUfJbO0JeKGue1Z5ZKpn4+cnwVmAX2GWDNC170ZG2b/F5mk5jBl d5Ovm6YbH3K6MxgT66S1nMkgkO4pEWP3UudEExTggY4vfesdzhp7wASEFSF/MOh8yXxW T9xwYOX1pckBS2ZBAvTrEBtvPNzdgiDMf2NELNi8AMNFS5jHy0ENx1ZVNDQou6V6054Y AW6A== X-Gm-Message-State: AOAM532+aUhB9XnjecpJgzpg/yEYXoSeVam6H5iT8KoZLiPNtJzRa+b8 9r0CBwCKwGvuTxIwON8hTFATrkkc X-Google-Smtp-Source: ABdhPJyv9vIShaz6cA3v7wRSQaC4jdMLbDfvNvCIIqsnn31OgJnqBE8GmhBF0mHZKl1gG96EXJOyMg== X-Received: by 2002:adf:812a:: with SMTP id 39mr7728430wrm.137.1596728924986; Thu, 06 Aug 2020 08:48:44 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r22sm7142436wmh.45.2020.08.06.08.48.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:44 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:32 +0000 Subject: [PATCH 02/11] maintenance: add --quiet option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Maintenance activities are commonly used as steps in larger scripts. Providing a '--quiet' option allows those scripts to be less noisy when run on a terminal window. Turn this mode on by default when stderr is not a terminal. Pipe the option to the 'git gc' child process. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 3 +++ builtin/gc.c | 9 +++++++++ t/t7900-maintenance.sh | 15 ++++++++++----- 3 files changed, 22 insertions(+), 5 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 34cd2b4417..089fa4cedc 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -52,6 +52,9 @@ OPTIONS in the `gc.auto` config setting, or when the number of pack-files exceeds the `gc.autoPackLimit` config setting. +--quiet:: + Do not report progress or other information over `stderr`. + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index e4f0ce1c86..a060ba8424 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -707,6 +707,7 @@ static const char * const builtin_maintenance_usage[] = { struct maintenance_opts { int auto_flag; + int quiet; }; static int maintenance_task_gc(struct maintenance_opts *opts) @@ -718,6 +719,10 @@ static int maintenance_task_gc(struct maintenance_opts *opts) if (opts->auto_flag) strvec_push(&child.args, "--auto"); + if (opts->quiet) + strvec_push(&child.args, "--quiet"); + else + strvec_push(&child.args, "--no-quiet"); close_object_store(the_repository->objects); return run_command(&child); @@ -734,6 +739,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) static struct option builtin_maintenance_options[] = { OPT_BOOL(0, "auto", &opts.auto_flag, N_("run tasks based on the state of the repository")), + OPT_BOOL(0, "quiet", &opts.quiet, + N_("do not report progress or other information over stderr")), OPT_END() }; @@ -743,6 +750,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) usage_with_options(builtin_maintenance_usage, builtin_maintenance_options); + opts.quiet = !isatty(2); + argc = parse_options(argc, argv, prefix, builtin_maintenance_options, builtin_maintenance_usage, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index c4b9b4a6fe..7b63b4ec0c 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -9,11 +9,16 @@ test_expect_success 'help text' ' test_i18ngrep "usage: git maintenance run" err ' -test_expect_success 'run [--auto]' ' - GIT_TRACE2_EVENT="$(pwd)/run-no-auto.txt" git maintenance run && - GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" git maintenance run --auto && - test_subcommand git gc /dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-auto.txt" \ + git maintenance run --auto 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-no-quiet.txt" \ + git maintenance run --no-quiet 2>/dev/null && + test_subcommand git gc --quiet X-Patchwork-Id: 11703799 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D82D8722 for ; Thu, 6 Aug 2020 17:56:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 413B2206C3 for ; Thu, 6 Aug 2020 17:56:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kUnOU+0J" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730659AbgHFR4b (ORCPT ); Thu, 6 Aug 2020 13:56:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37822 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726927AbgHFQRX (ORCPT ); Thu, 6 Aug 2020 12:17:23 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2B784C002140 for ; Thu, 6 Aug 2020 08:48:47 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id r4so41498977wrx.9 for ; Thu, 06 Aug 2020 08:48:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=haPdcuSHV4uqg9oVGnsbasRCEzOuPwsFJKkmpIY23OQ=; b=kUnOU+0JRbS6XAwYWZw7+V9JxzxmobZeQN8P4unwnyUxJNQk8sgOIA4/DJ53MWv8Hh VnUr38A4Ei37OGzbEOl+DBLp8k/zKtp3wFF0uxeNbBeEXbCSAL4zbE3/Ysp291nwfepO QPU6FaXQgPxWw3HLnK6wOxyzwp6uzPnDHrHbFotXDdX9NATdC3sRegzZlx+uvtkyQzS9 Gk47fGnOZYlyX+7f0yuGpWWzqw2YmYEdB0di/E1pMTx29mf1+jk6VkVsnZV7gHHl6y8P qx/Q16Bl47UI2DR89rqd8cpKbi/nGOY7gCOi0yGtM536w1hxydnlWKbRYIa7dJ5IZal8 NBWg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=haPdcuSHV4uqg9oVGnsbasRCEzOuPwsFJKkmpIY23OQ=; b=lHyBhRO3FCesCcGtsudHPgulDsCf/IbyxBYzndYOYcIKF+8y8PEOh/DAV+78WxWbeX ZSEU9YcQi7byURgX16PeIeM3E6Cx5olSZ9Gvp9MHP7xNTruFYGhz8Ls4ddi6iC50G7uI EV+3o4+WEiP1oD50bLSj8ZkKEZeI9cvu5HvN6PykOrSUPc5gNDDb98QIOoSjzrtqxMUB IyRZfROqeWB7Dtysgvii4IxGTBrdxMvMduwP/ak+M75AK172fD/WsLZIKdjMXQeebG0b KWf9b4ddG1zCxDB5yg1f+4MXBMUM3Jnuzx9T0zTPSkAE9Bd2UFNCR1WVZeJpA/UC/wi+ dqCw== X-Gm-Message-State: AOAM5332oRUlhzw8vDq1tf0OgkrFFknFc6ayxDua8awNtYp6nwgTsCSQ NlMuazjQxPFDuDAlQHVLZltgmKvq X-Google-Smtp-Source: ABdhPJxJtNZd1aH9NDwlsAJDEe6XxMTAC54fPYWoLqeWrQ+KpNux/maC5PEpBIXPYDvj0Sz8tNzNmw== X-Received: by 2002:adf:ab0d:: with SMTP id q13mr7533334wrc.134.1596728926251; Thu, 06 Aug 2020 08:48:46 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v16sm3928373wmj.14.2020.08.06.08.48.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:45 -0700 (PDT) Message-Id: <233811310b6536f4287d0ebb0ab869a526dc04c6.1596728921.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:33 +0000 Subject: [PATCH 03/11] maintenance: replace run_auto_gc() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The run_auto_gc() method is used in several places to trigger a check for repo maintenance after some Git commands, such as 'git commit' or 'git fetch'. To allow for extra customization of this maintenance activity, replace the 'git gc --auto [--quiet]' call with one to 'git maintenance run --auto [--quiet]'. As we extend the maintenance builtin with other steps, users will be able to select different maintenance activities. Rename run_auto_gc() to run_auto_maintenance() to be clearer what is happening on this call, and to expose all callers in the current diff. Rewrite the method to use a struct child_process to simplify the calls slightly. Since 'git fetch' already allows disabling the 'git gc --auto' subprocess, add an equivalent option with a different name to be more descriptive of the new behavior: '--[no-]maintenance'. Update the documentation to include these options at the same time. Signed-off-by: Derrick Stolee --- Documentation/fetch-options.txt | 6 ++++-- Documentation/git-clone.txt | 6 +++--- builtin/am.c | 2 +- builtin/commit.c | 2 +- builtin/fetch.c | 6 ++++-- builtin/merge.c | 2 +- builtin/rebase.c | 4 ++-- run-command.c | 16 +++++++--------- run-command.h | 2 +- t/t5510-fetch.sh | 2 +- 10 files changed, 25 insertions(+), 23 deletions(-) diff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt index 6e2a160a47..495bc8ab5a 100644 --- a/Documentation/fetch-options.txt +++ b/Documentation/fetch-options.txt @@ -86,9 +86,11 @@ ifndef::git-pull[] Allow several and arguments to be specified. No s may be specified. +--[no-]auto-maintenance:: --[no-]auto-gc:: - Run `git gc --auto` at the end to perform garbage collection - if needed. This is enabled by default. + Run `git maintenance run --auto` at the end to perform automatic + repository maintenance if needed. (`--[no-]auto-gc` is a synonym.) + This is enabled by default. --[no-]write-commit-graph:: Write a commit-graph after fetching. This overrides the config diff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt index c898310099..097e6a86c5 100644 --- a/Documentation/git-clone.txt +++ b/Documentation/git-clone.txt @@ -78,9 +78,9 @@ repository using this option and then delete branches (or use any other Git command that makes any existing commit unreferenced) in the source repository, some objects may become unreferenced (or dangling). These objects may be removed by normal Git operations (such as `git commit`) -which automatically call `git gc --auto`. (See linkgit:git-gc[1].) -If these objects are removed and were referenced by the cloned repository, -then the cloned repository will become corrupt. +which automatically call `git maintenance run --auto`. (See +linkgit:git-maintenance[1].) If these objects are removed and were referenced +by the cloned repository, then the cloned repository will become corrupt. + Note that running `git repack` without the `--local` option in a repository cloned with `--shared` will copy objects from the source repository into a pack diff --git a/builtin/am.c b/builtin/am.c index dd4e6c2d9b..68e9d17379 100644 --- a/builtin/am.c +++ b/builtin/am.c @@ -1795,7 +1795,7 @@ static void am_run(struct am_state *state, int resume) if (!state->rebasing) { am_destroy(state); close_object_store(the_repository->objects); - run_auto_gc(state->quiet); + run_auto_maintenance(state->quiet); } } diff --git a/builtin/commit.c b/builtin/commit.c index 69ac78d5e5..f9b0a0c05d 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1702,7 +1702,7 @@ int cmd_commit(int argc, const char **argv, const char *prefix) git_test_write_commit_graph_or_die(); repo_rerere(the_repository, 0); - run_auto_gc(quiet); + run_auto_maintenance(quiet); run_commit_hook(use_editor, get_index_file(), "post-commit", NULL); if (amend && !no_post_rewrite) { commit_post_rewrite(the_repository, current_head, &oid); diff --git a/builtin/fetch.c b/builtin/fetch.c index c49f0e9752..c8b9366d3c 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -196,8 +196,10 @@ static struct option builtin_fetch_options[] = { OPT_STRING_LIST(0, "negotiation-tip", &negotiation_tip, N_("revision"), N_("report that we have only objects reachable from this object")), OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options), + OPT_BOOL(0, "auto-maintenance", &enable_auto_gc, + N_("run 'maintenance --auto' after fetching")), OPT_BOOL(0, "auto-gc", &enable_auto_gc, - N_("run 'gc --auto' after fetching")), + N_("run 'maintenance --auto' after fetching")), OPT_BOOL(0, "show-forced-updates", &fetch_show_forced_updates, N_("check for forced-updates on all updated branches")), OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph, @@ -1882,7 +1884,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) close_object_store(the_repository->objects); if (enable_auto_gc) - run_auto_gc(verbosity < 0); + run_auto_maintenance(verbosity < 0); return result; } diff --git a/builtin/merge.c b/builtin/merge.c index 7da707bf55..c068e73037 100644 --- a/builtin/merge.c +++ b/builtin/merge.c @@ -457,7 +457,7 @@ static void finish(struct commit *head_commit, * user should see them. */ close_object_store(the_repository->objects); - run_auto_gc(verbosity < 0); + run_auto_maintenance(verbosity < 0); } } if (new_head && show_diffstat) { diff --git a/builtin/rebase.c b/builtin/rebase.c index dadb52fa92..9ffba9009d 100644 --- a/builtin/rebase.c +++ b/builtin/rebase.c @@ -728,10 +728,10 @@ static int finish_rebase(struct rebase_options *opts) apply_autostash(state_dir_path("autostash", opts)); close_object_store(the_repository->objects); /* - * We ignore errors in 'gc --auto', since the + * We ignore errors in 'git maintenance run --auto', since the * user should see them. */ - run_auto_gc(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE))); + run_auto_maintenance(!(opts->flags & (REBASE_NO_QUIET|REBASE_VERBOSE))); if (opts->type == REBASE_MERGE) { struct replay_opts replay = REPLAY_OPTS_INIT; diff --git a/run-command.c b/run-command.c index cc9c3296ba..2ee59acdc8 100644 --- a/run-command.c +++ b/run-command.c @@ -1866,15 +1866,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task, return result; } -int run_auto_gc(int quiet) +int run_auto_maintenance(int quiet) { - struct strvec argv_gc_auto = STRVEC_INIT; - int status; + struct child_process maint = CHILD_PROCESS_INIT; - strvec_pushl(&argv_gc_auto, "gc", "--auto", NULL); - if (quiet) - strvec_push(&argv_gc_auto, "--quiet"); - status = run_command_v_opt(argv_gc_auto.v, RUN_GIT_CMD); - strvec_clear(&argv_gc_auto); - return status; + maint.git_cmd = 1; + strvec_pushl(&maint.args, "maintenance", "run", "--auto", NULL); + strvec_push(&maint.args, quiet ? "--quiet" : "--no-quiet"); + + return run_command(&maint); } diff --git a/run-command.h b/run-command.h index 8b9bfaef16..6472b38bde 100644 --- a/run-command.h +++ b/run-command.h @@ -221,7 +221,7 @@ int run_hook_ve(const char *const *env, const char *name, va_list args); /* * Trigger an auto-gc */ -int run_auto_gc(int quiet); +int run_auto_maintenance(int quiet); #define RUN_COMMAND_NO_STDIN 1 #define RUN_GIT_CMD 2 /*If this is to be git sub-command */ diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh index a66dbe0bde..9850ecde5d 100755 --- a/t/t5510-fetch.sh +++ b/t/t5510-fetch.sh @@ -919,7 +919,7 @@ test_expect_success 'fetching with auto-gc does not lock up' ' git config fetch.unpackLimit 1 && git config gc.autoPackLimit 1 && git config gc.autoDetach false && - GIT_ASK_YESNO="$D/askyesno" git fetch >fetch.out 2>&1 && + GIT_ASK_YESNO="$D/askyesno" git fetch --verbose >fetch.out 2>&1 && test_i18ngrep "Auto packing the repository" fetch.out && ! grep "Should I try again" fetch.out ) From patchwork Thu Aug 6 15:48:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703557 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 05CC1722 for ; Thu, 6 Aug 2020 16:52:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 601BA2086A for ; Thu, 6 Aug 2020 16:52:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kXauzGEB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729618AbgHFQwB (ORCPT ); Thu, 6 Aug 2020 12:52:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43370 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729562AbgHFQvF (ORCPT ); Thu, 6 Aug 2020 12:51:05 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BFE3C002141 for ; Thu, 6 Aug 2020 08:48:48 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id f7so44502202wrw.1 for ; Thu, 06 Aug 2020 08:48:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=XnWOKAq/9nd+xoGuHVxK9oDozWhfGFERuP9RHAy5J0s=; b=kXauzGEBbzlzsICunbz3vi9vLVesg5rEP+0wz4NP6eSKf2EVZu2bBxdJzMOhIdkoKe Vf0ZQyorhB2QTOLPE+KlKcm4Zi85WuS6vWSeIokvAp/1tuv6DT7VRKhfIHoiPMgDVimk GB01of+kU5WetPzk3SrJKTtQ+wiEKgxT3S30xiNmZefKLqx7rIyJbZ1zw3FKxFFhoUiy b+lcaFxCQ9JGgjNnnk4uP35uRJq3bRrHBCcBpJEcyMX4WMdc8ve3eNpWPe90nUT73yj7 uXONqoczLzUw7KlpFNYm0WUJU44Tb/hrNOnfG98eC/IUZ8yFmXYgJSeLBNUH543EIKWo xAdg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=XnWOKAq/9nd+xoGuHVxK9oDozWhfGFERuP9RHAy5J0s=; b=tzhMqvyEzc7Z0SgBVkwDMg32Kre2oDTzRuT7zHJP/qVLiky8r9nGpoWSEUZd/SIKpA K+TFL/57gGbUdy/MZwFMOyJD1cqMAQGO9QUqlXsFpAVnSoJ+PU3TuT6z9HFvm6/gXIpj oUkKGcnQD1tFqzUpnxGOjGUvpD2BZGFPyDq5OSj7mXjd3bXySYt2pycRVWrvonrkpaMD i+b+RdRTmTYCKTP+nyrKDVAGvzbfBcn5RbORySexSgPguVWHzTHjFoBraBoN7WvB4o/4 ALUN61vqA8hN2r1Tlsw+zPjfqbvmlkkkfTGfQfgmuTdeFx/0lMG+wfidqm7sqSFesHSY 1vpg== X-Gm-Message-State: AOAM531/F+DR96A4KCsi5+E/HI6NHtfiVpIwI0vewkL598OcXCJNOE25 fc36cbbJFYQyAxJ7YxLIF4ozUtxM X-Google-Smtp-Source: ABdhPJzRpPIHChfaHnLZR8mLd8INA4orY5MJ/87oEsh8uvJxK/nCVyiNDFPHOyICERX081/2ph2t3w== X-Received: by 2002:adf:e904:: with SMTP id f4mr7978128wrm.300.1596728927078; Thu, 06 Aug 2020 08:48:47 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o3sm7002965wru.64.2020.08.06.08.48.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:46 -0700 (PDT) Message-Id: <7efa23abc85bead9713c34d5f56b2f308dde7bf8.1596728921.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:34 +0000 Subject: [PATCH 04/11] maintenance: initialize task array Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee In anticipation of implementing multiple maintenance tasks inside the 'maintenance' builtin, use a list of structs to describe the work to be done. The struct maintenance_task stores the name of the task (as given by a future command-line argument) along with a function pointer to its implementation and a boolean for whether the step is enabled. A list these structs are initialized with the full list of implemented tasks along with a default order. For now, this list only contains the "gc" task. This task is also the only task enabled by default. The run subcommand will return a nonzero exit code if any task fails. However, it will attempt all tasks in its loop before returning with the failure. Also each failed task will send an error message. Helped-by: Taylor Blau Helped-by: Junio C Hamano Signed-off-by: Derrick Stolee --- builtin/gc.c | 38 +++++++++++++++++++++++++++++++++++++- 1 file changed, 37 insertions(+), 1 deletion(-) diff --git a/builtin/gc.c b/builtin/gc.c index a060ba8424..150dce4301 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -728,9 +728,45 @@ static int maintenance_task_gc(struct maintenance_opts *opts) return run_command(&child); } +typedef int maintenance_task_fn(struct maintenance_opts *opts); + +struct maintenance_task { + const char *name; + maintenance_task_fn *fn; + unsigned enabled:1; +}; + +enum maintenance_task_label { + TASK_GC, + + /* Leave as final value */ + TASK__COUNT +}; + +static struct maintenance_task tasks[] = { + [TASK_GC] = { + "gc", + maintenance_task_gc, + 1, + }, +}; + static int maintenance_run(struct maintenance_opts *opts) { - return maintenance_task_gc(opts); + int i; + int result = 0; + + for (i = 0; i < TASK__COUNT; i++) { + if (!tasks[i].enabled) + continue; + + if (tasks[i].fn(opts)) { + error(_("task '%s' failed"), tasks[i].name); + result = 1; + } + } + + return result; } int cmd_maintenance(int argc, const char **argv, const char *prefix) From patchwork Thu Aug 6 15:48:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703539 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A3F8B13B1 for ; Thu, 6 Aug 2020 16:48:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F026B2311A for ; Thu, 6 Aug 2020 16:48:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oE9uO0Mt" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729459AbgHFQrm (ORCPT ); Thu, 6 Aug 2020 12:47:42 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728720AbgHFQpH (ORCPT ); Thu, 6 Aug 2020 12:45:07 -0400 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8F7AAC002142 for ; Thu, 6 Aug 2020 08:48:49 -0700 (PDT) Received: by mail-wm1-x343.google.com with SMTP id t14so10151477wmi.3 for ; Thu, 06 Aug 2020 08:48:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=303SZoeGqCA0672IdnCKWTl2RMdleMaL49S3kRHa8CY=; b=oE9uO0Mt0t5TloYkjvoOMiKFdp9jk5j4TwOhbYZ1EOhrJyuSACQiJzq0pDOChhrxC6 vhOmlo9A18qla20Ru3Tlo19sG3sIDTvBTGvawu0i83/3HQLm39FU0BOryEDeV4YovuX8 7VyQsgIqQDdMBG2EJlfWCSh1j+xN3crlNWG+HhcXdB42IW+ps+N2THSli2vfT+oHhLwq x7PRWkpNrE7QzCEzfgPAMyJIvVbXHNwsqGUQaIFnalBamnFGZhk8xCrROAA6KX5tDtff UahxFMVhB9KKmF/zjdk9lWuJCoy7k+rgtdkscwoU0E40dXgrwMYfLw5hiyTKw17ceSdM ysDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=303SZoeGqCA0672IdnCKWTl2RMdleMaL49S3kRHa8CY=; b=nJF4T4xgt9ilwS1bTgpnJyo13lih5c8zuKx7R0XFkktgDLHSNUdjZ3H6DC0yaD1wwA yh6Z6hNbeYk7Gq1A2mpoXfpmE/wwINcxN3yHmXOCJRKTqdYb0Sh+X6OI4LfRwXkbTxit j4ETjs1DyHUiPktxOaLIN9S2ysHJtXqL54Ws66JVnQjhfegYux+8q9NNPGsEGomBLY2I osjbLjbSxKB7MpPvy1RZF9fYLh0GP3RwJHsDFq+Yf7dASi2MPr5bocnrQ/nT+LNXdeYJ +rgNXe5LtSFmBOOGqDR4WzZkf4hbO5lx8lP40HhBD5+zO69o5V77V7JynMIcsH+DOz2G vQUw== X-Gm-Message-State: AOAM531edXdwubnln1K64T1U+kcfRbEChbmmy1nd+pCh3hKizhTCDUYi I/C+ufGiRUk5rVdNx6ECN5bfXrkn X-Google-Smtp-Source: ABdhPJxzWm4GGCV8rrDC4OHeKMRI8H4AXXwSAuel2TMtc2Uj31/0Ii7ns3Zgh8kYLQYp81DzVSxN0g== X-Received: by 2002:a1c:2d95:: with SMTP id t143mr8276889wmt.44.1596728928031; Thu, 06 Aug 2020 08:48:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 31sm6880784wrj.94.2020.08.06.08.48.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:47 -0700 (PDT) Message-Id: <902b742032ae19087392538936cc81768a59e0e1.1596728921.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:35 +0000 Subject: [PATCH 05/11] maintenance: add commit-graph task Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The first new task in the 'git maintenance' builtin is the 'commit-graph' job. It is based on the sequence of events in the 'commit-graph' job in Scalar [1]. This sequence is as follows: 1. git commit-graph write --reachable --split 2. git commit-graph verify --shallow 3. If the verify succeeds, stop. 4. Delete the commit-graph-chain file. 5. git commit-graph write --reachable --split By writing an incremental commit-graph file using the "--split" option we minimize the disruption from this operation. The default behavior is to merge layers until the new "top" layer is less than half the size of the layer below. This provides quick writes most of the time, with the longer writes following a power law distribution. Most importantly, concurrent Git processes only look at the commit-graph-chain file for a very short amount of time, so they will verly likely not be holding a handle to the file when we try to replace it. (This only matters on Windows.) If a concurrent process reads the old commit-graph-chain file, but our job expires some of the .graph files before they can be read, then those processes will see a warning message (but not fail). This could be avoided by a future update to use the --expire-time argument when writing the commit-graph. By using 'git commit-graph verify --shallow' we can ensure that the file we just wrote is valid. This is an extra safety precaution that is faster than our 'write' subcommand. In the rare situation that the newest layer of the commit-graph is corrupt, we can "fix" the corruption by deleting the commit-graph-chain file and rewrite the full commit-graph as a new one-layer commit graph. This does not completely prevent _that_ file from being corrupt, but it does recompute the commit-graph by parsing commits from the object database. In our use of this step in Scalar and VFS for Git, we have only seen this issue arise because our microsoft/git fork reverted 43d3561 ("commit-graph write: don't die if the existing graph is corrupt" 2019-03-25) for a while to keep commit-graph writes very fast. We dropped the revert when updating to v2.23.0. The verify still has potential for catching corrupt data across the layer boundary: if the new file has commit X with parent Y in an old file but the commit ID for Y in the old file had a bitswap, then we will notice that in the 'verify' command. [1] https://github.com/microsoft/scalar/blob/master/Scalar.Common/Maintenance/CommitGraphStep.cs Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 18 +++++++++ builtin/gc.c | 63 +++++++++++++++++++++++++++++++ commit-graph.c | 8 ++-- commit-graph.h | 1 + t/t7900-maintenance.sh | 2 + 5 files changed, 88 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 089fa4cedc..35b0be7d40 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -35,6 +35,24 @@ run:: TASKS ----- +commit-graph:: + The `commit-graph` job updates the `commit-graph` files incrementally, + then verifies that the written data is correct. If the new layer has an + issue, then the chain file is removed and the `commit-graph` is + rewritten from scratch. ++ +The verification only checks the top layer of the `commit-graph` chain. +If the incremental write merged the new commits with at least one +existing layer, then there is potential for on-disk corruption being +carried forward into the new file. This will be noticed and the new +commit-graph file will be clean as Git reparses the commit data from +the object database. ++ +The incremental write is safe to run alongside concurrent Git processes +since it will not expire `.graph` files that were in the previous +`commit-graph-chain` file. They will be deleted by a later run based on +the expiration delay. + gc:: Cleanup unnecessary files and optimize the local repository. "GC" stands for "garbage collection," but this task performs many diff --git a/builtin/gc.c b/builtin/gc.c index 150dce4301..3b7b914d60 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -710,6 +710,64 @@ struct maintenance_opts { int quiet; }; +static int run_write_commit_graph(struct maintenance_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_pushl(&child.args, "commit-graph", "write", + "--split", "--reachable", NULL); + + if (opts->quiet) + strvec_push(&child.args, "--no-progress"); + + return !!run_command(&child); +} + +static int run_verify_commit_graph(struct maintenance_opts *opts) +{ + struct child_process child = CHILD_PROCESS_INIT; + + child.git_cmd = 1; + strvec_pushl(&child.args, "commit-graph", "verify", + "--shallow", NULL); + + if (opts->quiet) + strvec_push(&child.args, "--no-progress"); + + return !!run_command(&child); +} + +static int maintenance_task_commit_graph(struct maintenance_opts *opts) +{ + struct repository *r = the_repository; + char *chain_path; + + close_object_store(r->objects); + if (run_write_commit_graph(opts)) { + error(_("failed to write commit-graph")); + return 1; + } + + if (!run_verify_commit_graph(opts)) + return 0; + + warning(_("commit-graph verify caught error, rewriting")); + + chain_path = get_commit_graph_chain_filename(r->objects->odb); + if (unlink(chain_path)) { + UNLEAK(chain_path); + die(_("failed to remove commit-graph at %s"), chain_path); + } + free(chain_path); + + if (!run_write_commit_graph(opts)) + return 0; + + error(_("failed to rewrite commit-graph")); + return 1; +} + static int maintenance_task_gc(struct maintenance_opts *opts) { struct child_process child = CHILD_PROCESS_INIT; @@ -738,6 +796,7 @@ struct maintenance_task { enum maintenance_task_label { TASK_GC, + TASK_COMMIT_GRAPH, /* Leave as final value */ TASK__COUNT @@ -749,6 +808,10 @@ static struct maintenance_task tasks[] = { maintenance_task_gc, 1, }, + [TASK_COMMIT_GRAPH] = { + "commit-graph", + maintenance_task_commit_graph, + }, }; static int maintenance_run(struct maintenance_opts *opts) diff --git a/commit-graph.c b/commit-graph.c index 1af68c297d..9705d237e4 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -172,7 +172,7 @@ static char *get_split_graph_filename(struct object_directory *odb, oid_hex); } -static char *get_chain_filename(struct object_directory *odb) +char *get_commit_graph_chain_filename(struct object_directory *odb) { return xstrfmt("%s/info/commit-graphs/commit-graph-chain", odb->path); } @@ -521,7 +521,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, struct stat st; struct object_id *oids; int i = 0, valid = 1, count; - char *chain_name = get_chain_filename(odb); + char *chain_name = get_commit_graph_chain_filename(odb); FILE *fp; int stat_res; @@ -1619,7 +1619,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) } if (ctx->split) { - char *lock_name = get_chain_filename(ctx->odb); + char *lock_name = get_commit_graph_chain_filename(ctx->odb); hold_lock_file_for_update_mode(&lk, lock_name, LOCK_DIE_ON_ERROR, 0444); @@ -1996,7 +1996,7 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx) if (ctx->split_opts && ctx->split_opts->expire_time) expire_time = ctx->split_opts->expire_time; if (!ctx->split) { - char *chain_file_name = get_chain_filename(ctx->odb); + char *chain_file_name = get_commit_graph_chain_filename(ctx->odb); unlink(chain_file_name); free(chain_file_name); ctx->num_commit_graphs_after = 0; diff --git a/commit-graph.h b/commit-graph.h index 28f89cdf3e..3c202748c3 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -25,6 +25,7 @@ struct commit; struct bloom_filter_settings; char *get_commit_graph_filename(struct object_directory *odb); +char *get_commit_graph_chain_filename(struct object_directory *odb); int open_commit_graph(const char *graph_file, int *fd, struct stat *st); /* diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 7b63b4ec0c..384294d111 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -4,6 +4,8 @@ test_description='git maintenance builtin' . ./test-lib.sh +GIT_TEST_COMMIT_GRAPH=0 + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance run" err From patchwork Thu Aug 6 15:48:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703477 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9A6AB1392 for ; Thu, 6 Aug 2020 16:23:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E58FC23117 for ; Thu, 6 Aug 2020 16:23:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Vuw+85v3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727946AbgHFQVo (ORCPT ); Thu, 6 Aug 2020 12:21:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727890AbgHFQSr (ORCPT ); Thu, 6 Aug 2020 12:18:47 -0400 Received: from mail-wm1-x341.google.com (mail-wm1-x341.google.com [IPv6:2a00:1450:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6BEBAC002143 for ; Thu, 6 Aug 2020 08:48:50 -0700 (PDT) Received: by mail-wm1-x341.google.com with SMTP id 9so9247110wmj.5 for ; Thu, 06 Aug 2020 08:48:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=BTbUl878qRwWso7jC4UFCTBzi45fQN029QdP1WegGYQ=; b=Vuw+85v3SGUzyHsbxZNKKjMxcXkjJrgb0LEkJ+XYaIy7h0F2Du3KbJvmhrw9X4msF9 Sc46yybNkFYwYQ3iJ/Z7f4tG4dJfyOmPk4/I5d9cYMbGJ3DKJdAeyBf0XyXE2lQvPR6n AYy72EnuhIQKYGsOPFZlYq+fxkWAqlmTERN5NsgDEyMdQYKM+l564cbeto/Hs9gTv+A7 hMNjYxaZm5Jbw5xDxlYXmNkZ5n7ZqG/FGSrYAd1DvhJF2hg0MVLDQ5KGxbrx+XRqRg8N jxHHDBS7RPxtynRCidg3g0of2VNbVwVQBE6bwLqF/OyflwFlhEnTt6Sq/QGAUYG1cRG4 dU1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=BTbUl878qRwWso7jC4UFCTBzi45fQN029QdP1WegGYQ=; b=c6xzSrTu0FgsEIlYFcuH2+GBzXpMw8WM94ZaG0iZHgdvEA6cxdFCF4B17OoDRH3THH vnsNQrQ0Vg0F2jyeiRqssDiES7VVRE5DiR0eNY1o7wDX0KtXemIFjmblpLnhBwct9cv0 E8OyedL/1ECNDeNfxR0UNOB3ln2iVF6jck0G8jvuDbcORkGQub/7KyjLuYYSDMOOozpl Qpo3X7Bzq4b6LU4U0AKZ/MWDPX64cs/ZEsEtJNt+XQ5Nwqo4Eb3cqGm7cpu1YoFtnVvg HqBy/vUG5WLlKw1NTp8zcK321908PUfYwaYkcB9Dv8gQzansP4FdcmA13V9m5hYeatpl OPQA== X-Gm-Message-State: AOAM533NSl4Rt4iv7rI46m4vZWFzbwT4Yhkn9jLIlS3DYvtkHMmNUppI scUgdA0PZEs0HhevHW7JqvMzyn0z X-Google-Smtp-Source: ABdhPJxBsuYFe4DpOXkNd2cWxPv3faXwqUDdClbO5JIsY3WloH+CfIsrad2YbPK6L75qVLV+1G1uOw== X-Received: by 2002:a7b:c4d5:: with SMTP id g21mr9261870wmk.185.1596728928921; Thu, 06 Aug 2020 08:48:48 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w64sm7228436wmb.26.2020.08.06.08.48.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:48 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:36 +0000 Subject: [PATCH 06/11] maintenance: add --task option Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee A user may want to only run certain maintenance tasks in a certain order. Add the --task= option, which allows a user to specify an ordered list of tasks to run. These cannot be run multiple times, however. Here is where our array of maintenance_task pointers becomes critical. We can sort the array of pointers based on the task order, but we do not want to move the struct data itself in order to preserve the hashmap references. We use the hashmap to match the --task= arguments into the task struct data. Keep in mind that the 'enabled' member of the maintenance_task struct is a placeholder for a future 'maintenance..enabled' config option. Thus, we use the 'enabled' member to specify which tasks are run when the user does not specify any --task= arguments. The 'enabled' member should be ignored if --task= appears. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 8 +++- builtin/gc.c | 61 +++++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 27 ++++++++++++++ 3 files changed, 92 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 35b0be7d40..5911213c9c 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -30,7 +30,9 @@ SUBCOMMANDS ----------- run:: - Run one or more maintenance tasks. + Run one or more maintenance tasks. If one or more `--task=` + options are specified, then those tasks are run in the provided + order. Otherwise, only the `gc` task is run. TASKS ----- @@ -73,6 +75,10 @@ OPTIONS --quiet:: Do not report progress or other information over `stderr`. +--task=:: + If this option is specified one or more times, then only run the + specified tasks in the specified order. + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 3b7b914d60..3d50ab7ac9 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -791,7 +791,9 @@ typedef int maintenance_task_fn(struct maintenance_opts *opts); struct maintenance_task { const char *name; maintenance_task_fn *fn; - unsigned enabled:1; + unsigned enabled:1, + selected:1; + int selected_order; }; enum maintenance_task_label { @@ -814,13 +816,32 @@ static struct maintenance_task tasks[] = { }, }; +static int compare_tasks_by_selection(const void *a_, const void *b_) +{ + const struct maintenance_task *a, *b; + + a = (const struct maintenance_task *)&a_; + b = (const struct maintenance_task *)&b_; + + return b->selected_order - a->selected_order; +} + static int maintenance_run(struct maintenance_opts *opts) { - int i; + int i, found_selected = 0; int result = 0; + for (i = 0; !found_selected && i < TASK__COUNT; i++) + found_selected = tasks[i].selected; + + if (found_selected) + QSORT(tasks, TASK__COUNT, compare_tasks_by_selection); + for (i = 0; i < TASK__COUNT; i++) { - if (!tasks[i].enabled) + if (found_selected && !tasks[i].selected) + continue; + + if (!found_selected && !tasks[i].enabled) continue; if (tasks[i].fn(opts)) { @@ -832,6 +853,37 @@ static int maintenance_run(struct maintenance_opts *opts) return result; } +static int task_option_parse(const struct option *opt, + const char *arg, int unset) +{ + int i, num_selected = 0; + struct maintenance_task *task = NULL; + + BUG_ON_OPT_NEG(unset); + + for (i = 0; i < TASK__COUNT; i++) { + num_selected += tasks[i].selected; + if (!strcasecmp(tasks[i].name, arg)) { + task = &tasks[i]; + } + } + + if (!task) { + error(_("'%s' is not a valid task"), arg); + return 1; + } + + if (task->selected) { + error(_("task '%s' cannot be selected multiple times"), arg); + return 1; + } + + task->selected = 1; + task->selected_order = num_selected + 1; + + return 0; +} + int cmd_maintenance(int argc, const char **argv, const char *prefix) { static struct maintenance_opts opts; @@ -840,6 +892,9 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) N_("run tasks based on the state of the repository")), OPT_BOOL(0, "quiet", &opts.quiet, N_("do not report progress or other information over stderr")), + OPT_CALLBACK_F(0, "task", NULL, N_("task"), + N_("run a specific task"), + PARSE_OPT_NONEG, task_option_parse), OPT_END() }; diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 384294d111..bcc7131818 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -23,4 +23,31 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet ' ' + GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ + git maintenance run --task=commit-graph 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-gc.txt" \ + git maintenance run --task=gc 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ + git maintenance run --task=commit-graph 2>/dev/null && + GIT_TRACE2_EVENT="$(pwd)/run-both.txt" \ + git maintenance run --task=commit-graph --task=gc 2>/dev/null && + test_subcommand ! git gc --quiet err && + test_i18ngrep "is not a valid task" err +' + +test_expect_success 'run --task duplicate' ' + test_must_fail git maintenance run --task=gc --task=gc 2>err && + test_i18ngrep "cannot be selected multiple times" err +' + test_done From patchwork Thu Aug 6 15:48:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703801 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 86D62722 for ; Thu, 6 Aug 2020 17:56:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E9DE6206C3 for ; Thu, 6 Aug 2020 17:56:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gP6xkdpC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730663AbgHFR4c (ORCPT ); Thu, 6 Aug 2020 13:56:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37638 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726375AbgHFQRX (ORCPT ); Thu, 6 Aug 2020 12:17:23 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 255ABC002144 for ; Thu, 6 Aug 2020 08:48:51 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id r2so39385341wrs.8 for ; Thu, 06 Aug 2020 08:48:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ieFpWj9cIIdQEgz9tTei7UAmyp4BbdrmUtoTOZ5t5X0=; b=gP6xkdpCeDGqCbeplSvNM9o+Kwz6W0bBcydRQVZY5Lt3IrwxjfFlVH5R/awUZksnBy uogQKVAJ3CUQ/llmtgHW3/zp9Obu12FZUNmqNMNJvLxS2fvyiCoONect/Pzq7YKSFmaD rKLWTjO5h2RCOx+13hl/t69F6tymuJ65Qrjlp+J/R90LPMKaRa8TKJ+R2Ro3Yl8xKXZK GMO5U53x34EI3608YBH/y/ZLkZo1QMb6FLLmp5IvHBZg7furj14bqOwNh2CZ50iVzCrQ sj4hpbIza0w5YHX6U0J5cDQ94GRuzGe30LumKuFbPFbzVhxBw5MVSDPkUZJs+fz6fYRT /pfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ieFpWj9cIIdQEgz9tTei7UAmyp4BbdrmUtoTOZ5t5X0=; b=dTLpiE+3s3q9Ntlbzj89TBng+xmXAr4fE35CSt7mK4vH0G853ak93qYY6e3es8sQcC x9ix8/v/PHDRMVnIUx06Ez9Uy/Jxu9OsQt4/MfSm/XRCT+nAj2GGooIJfGTDuQyZD9Ui s12kFGma5r+FcscT2qDFSjdSy5gobsFai2Jfu1xLHVy7cbKVm63ySs9o/hD6uTeeaet1 ooHXY3EY3reVuau2T1iMGaBX34jgf5nKMLc1E2vczSjfTwI9F268DRyMSGgdfHd9Ftc3 46jLUNa6Y/X6C8UAD87Y8OiqUYynJvpWaeLkGToPIYJ1V1mAAHEIiBZYBGk7zcJxnCmW vsYg== X-Gm-Message-State: AOAM531l4bg6YT/HmuJs01nhHOYbbmX+T9cOOCImTIdWwiscQ1KykdCf comeRUkTV2gqDi3+yzXinWhL0B/D X-Google-Smtp-Source: ABdhPJy1OsA4K1NBEiQOfZb74c++x9wWQo/OFnKv4TObSJTJ1f1oxQy+lxlVGM5hT50+eJbzownWqw== X-Received: by 2002:a5d:5273:: with SMTP id l19mr7905183wrc.257.1596728929745; Thu, 06 Aug 2020 08:48:49 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e4sm6978500wru.55.2020.08.06.08.48.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:49 -0700 (PDT) Message-Id: <79af39be13701267a094362032cbdfa13aa6a6f1.1596728921.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:37 +0000 Subject: [PATCH 07/11] maintenance: take a lock on the objects directory Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Performing maintenance on a Git repository involves writing data to the .git directory, which is not safe to do with multiple writers attempting the same operation. Ensure that only one 'git maintenance' process is running at a time by holding a file-based lock. Simply the presence of the .git/maintenance.lock file will prevent future maintenance. This lock is never committed, since it does not represent meaningful data. Instead, it is only a placeholder. If the lock file already exists, then fail silently. This will become very important later when we implement the 'fetch' task, as this is our stop-gap from creating a recursive process loop between 'git fetch' and 'git maintenance run'. Signed-off-by: Derrick Stolee --- builtin/gc.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/builtin/gc.c b/builtin/gc.c index 3d50ab7ac9..1b7d502b87 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -830,6 +830,25 @@ static int maintenance_run(struct maintenance_opts *opts) { int i, found_selected = 0; int result = 0; + struct lock_file lk; + struct repository *r = the_repository; + char *lock_path = xstrfmt("%s/maintenance", r->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) { + /* + * Another maintenance command is running. + * + * If --auto was provided, then it is likely due to a + * recursive process stack. Do not report an error in + * that case. + */ + if (!opts->auto_flag && !opts->quiet) + error(_("lock file '%s' exists, skipping maintenance"), + lock_path); + free(lock_path); + return 0; + } + free(lock_path); for (i = 0; !found_selected && i < TASK__COUNT; i++) found_selected = tasks[i].selected; @@ -850,6 +869,7 @@ static int maintenance_run(struct maintenance_opts *opts) } } + rollback_lock_file(&lk); return result; } From patchwork Thu Aug 6 15:48:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703481 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A0EB13B1 for ; Thu, 6 Aug 2020 16:27:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A2B7F2311B for ; Thu, 6 Aug 2020 16:27:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Rky2w6co" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728255AbgHFQ1s (ORCPT ); Thu, 6 Aug 2020 12:27:48 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37958 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727115AbgHFQSr (ORCPT ); Thu, 6 Aug 2020 12:18:47 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3DB2BC002146 for ; Thu, 6 Aug 2020 08:48:52 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id a5so34506705wrm.6 for ; Thu, 06 Aug 2020 08:48:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=lCIavgK26TxWY6o7Jx/OsmHH6/0nLwidyNPcYgO+hw8=; b=Rky2w6co/0XqDC4inCbBZu3uGTS6Pbgo+L6Ro0Cx94qPBPlza6nVzikiX8dyL6S4uM yTyYfr00woBWjwtWahV2b8I2/xnLCE777LgVZn4Vio4LkpD+H/jjSGrnwwX6ZHfARgJe 0npi00Tk2EnI27+RrPAqKywVJoT6nTTYpe+tmI0kAl9K0dG6UitibgbtrQWTh6q5K6mK udc28+pXUk7VuXtwPjp4mfp5AsU3a8KY6SDUJKbW2GrSsYnDmjP5ppgNrV7H+rPggsP6 gFV9ReCbpY1ektwxwCR6uIJm2OlHERGrS8WOPK5bHF+C04NFNBiBd2dwjMrYhI8/S+XR a0uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=lCIavgK26TxWY6o7Jx/OsmHH6/0nLwidyNPcYgO+hw8=; b=ZRWvf4KF53yYzVFM5t/zml3EEnUxPgUvrHgOXcT1wqTKffstdO32Di/ZFXFXUdQ0ae 4StQksR438/PIY8XhZrC6qfxk5XVLN2FAigqDVxha3Gdwd1kV2A8V9+fP0yBefRMNhzW qzMvF1+3QIfVx6vNgbLk/GvQY65SandYNxIQIYk17mp64M6GeYdyzsdGb9rZI/5b5Drz CkaNP5rxdpfKD+yV/dIqUo2pi4/y6iiO1bsWQP375LBRXXtfDW0pkFilfGiGIBK7fjO1 TgHg9P1tCAcc2prCZwEFH7CnVrDOgASifghPE+jtxAw1QCK3gnnpg9awayYmJ5r1YHMd ONOQ== X-Gm-Message-State: AOAM530E8B/68/UJPhAbjTF+hXD6k4W5+R2zSUq/obRI4v5ZPsFK7lfb KI6lBcpw//JVKgyIcgTEU+RK2rKv X-Google-Smtp-Source: ABdhPJyyIbXkRJuakIECYrDWBnI+Y08Cpvolc+MdUoB6Oil5HDhIG1D+273VLxtqV/eCauJjiZlThw== X-Received: by 2002:a5d:5712:: with SMTP id a18mr7857534wrv.184.1596728930753; Thu, 06 Aug 2020 08:48:50 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 3sm6672146wms.36.2020.08.06.08.48.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:50 -0700 (PDT) Message-Id: <69bfc6a4b2bb1a86788e69a2a3ac6134085aa798.1596728921.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:38 +0000 Subject: [PATCH 08/11] maintenance: create maintenance..enabled config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Currently, a normal run of "git maintenance run" will only run the 'gc' task, as it is the only one enabled. This is mostly for backwards- compatible reasons since "git maintenance run --auto" commands replaced previous "git gc --auto" commands after some Git processes. Users could manually run specific maintenance tasks by calling "git maintenance run --task=" directly. Allow users to customize which steps are run automatically using config. The 'maintenance..enabled' option then can turn on these other tasks (or turn off the 'gc' task). Signed-off-by: Derrick Stolee --- Documentation/config.txt | 2 ++ Documentation/config/maintenance.txt | 4 ++++ Documentation/git-maintenance.txt | 8 +++++--- builtin/gc.c | 19 +++++++++++++++++++ t/t7900-maintenance.sh | 8 ++++++++ 5 files changed, 38 insertions(+), 3 deletions(-) create mode 100644 Documentation/config/maintenance.txt diff --git a/Documentation/config.txt b/Documentation/config.txt index ef0768b91a..2783b825f9 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -396,6 +396,8 @@ include::config/mailinfo.txt[] include::config/mailmap.txt[] +include::config/maintenance.txt[] + include::config/man.txt[] include::config/merge.txt[] diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt new file mode 100644 index 0000000000..370cbfb42f --- /dev/null +++ b/Documentation/config/maintenance.txt @@ -0,0 +1,4 @@ +maintenance..enabled:: + This boolean config option controls whether the maintenance task + with name `` is run when no `--task` option is specified. + By default, only `maintenance.gc.enabled` is true. diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 5911213c9c..e1a2a8902c 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -30,9 +30,11 @@ SUBCOMMANDS ----------- run:: - Run one or more maintenance tasks. If one or more `--task=` - options are specified, then those tasks are run in the provided - order. Otherwise, only the `gc` task is run. + Run one or more maintenance tasks. If one or more `--task` options + are specified, then those tasks are run in that order. Otherwise, + the tasks are determined by which `maintenance..enabled` + config options are true. By default, only `maintenance.gc.enabled` + is true. TASKS ----- diff --git a/builtin/gc.c b/builtin/gc.c index 1b7d502b87..dc1d260858 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -873,6 +873,24 @@ static int maintenance_run(struct maintenance_opts *opts) return result; } +static void initialize_task_config(void) +{ + int i; + struct strbuf config_name = STRBUF_INIT; + for (i = 0; i < TASK__COUNT; i++) { + int config_value; + + strbuf_setlen(&config_name, 0); + strbuf_addf(&config_name, "maintenance.%s.enabled", + tasks[i].name); + + if (!git_config_get_bool(config_name.buf, &config_value)) + tasks[i].enabled = config_value; + } + + strbuf_release(&config_name); +} + static int task_option_parse(const struct option *opt, const char *arg, int unset) { @@ -925,6 +943,7 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) builtin_maintenance_options); opts.quiet = !isatty(2); + initialize_task_config(); argc = parse_options(argc, argv, prefix, builtin_maintenance_options, diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index bcc7131818..cdf0bf2e60 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -23,6 +23,14 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet .enabled' ' + git config maintenance.gc.enabled false && + git config maintenance.commit-graph.enabled true && + GIT_TRACE2_EVENT="$(pwd)/run-config.txt" git maintenance run 2>err && + test_subcommand ! git gc --quiet ' ' GIT_TRACE2_EVENT="$(pwd)/run-commit-graph.txt" \ git maintenance run --task=commit-graph 2>/dev/null && From patchwork Thu Aug 6 15:48:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703803 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B9E8722 for ; Thu, 6 Aug 2020 17:56:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AA5F1221E2 for ; Thu, 6 Aug 2020 17:56:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="O/7XwcrE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730619AbgHFR4a (ORCPT ); Thu, 6 Aug 2020 13:56:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37824 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726977AbgHFQRX (ORCPT ); Thu, 6 Aug 2020 12:17:23 -0400 Received: from mail-wr1-x444.google.com (mail-wr1-x444.google.com [IPv6:2a00:1450:4864:20::444]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 24EA7C002147 for ; Thu, 6 Aug 2020 08:48:53 -0700 (PDT) Received: by mail-wr1-x444.google.com with SMTP id a5so34506752wrm.6 for ; Thu, 06 Aug 2020 08:48:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=YMZSxoX6uVDMUjg146cWLxHkPLGIA7mmqY6I5YeqTsw=; b=O/7XwcrEV+N1HBZWwepgS0mAiOp5WLLHQoA5YZnMaC9mLJZNb56NRq4jvCkXJ4yNMb qvM6J+ucvf4sqhPZUdwyHKLoNVmJ7pmMjhKbuUBhrGGa4YcDeh9KxDJvo3fimohmS2S2 g2FVQgGLFXL3iP0NhxFXSO2YEEXXkZIQMGA54SlXFW45M7bXMDaNqXY/xJ8fsgwqPF/f GhPbVBgWpByyJ/dYTORTqvts+bhRhxoGmHzN7B9JgEflr0aM0iKXP1xC9xnqgUlRcnKI bpodeqrKV8pWeVtMZea/1KKxO4n5zAG5igwGppEiImAYpMB1jaZABTQdXiDe6Az/LESp WRjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=YMZSxoX6uVDMUjg146cWLxHkPLGIA7mmqY6I5YeqTsw=; b=OQe3+DoRJdgTqmawDUHF4hlMZ4ok2wmn3LFSFW1MKEZ6WyvPEW+LoIkMCqSklrAZny g0oMS3IdBO4A0cBrICdI9PM4aEXVn7q0kOAhjqE0WwJDYmjeAe4L3dmWEVMkwEuLnT9Z xcskEvDLqQG1MZaRYDb6zTrTunUINK6tHvfoeXw4ujj1P9SY3ado1bCnR1sLPi97z3Qk TvyFjVyo201lHzVuQYVAybKJ/kPWKsGw5dTJ1ip1zAp1d24ekO0Z/OXxp2ggFiFSlPCz AP+kUmUY/FV9GJDs5viY/bACbp4H/B4yChT5wMoqL9f48SITs6h2qlFK5sQbQyhbUPQC gWCQ== X-Gm-Message-State: AOAM530RFz165iB4sUq58iiRxf4X1Rg5tMrO7uGJbNpIkBHRiG6m4ws2 Cjjx2U4/vvsPbBNYqJdlb/F25W3q X-Google-Smtp-Source: ABdhPJwIZyIV7tVdSdsUGhYW0y8htvB9qZRN9cjmgrAUimA1/mhqceQb4JFhK+/d8BP6bJw7M+r9ww== X-Received: by 2002:a5d:5710:: with SMTP id a16mr8593487wrv.217.1596728931654; Thu, 06 Aug 2020 08:48:51 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m1sm6479115wmc.28.2020.08.06.08.48.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:51 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:39 +0000 Subject: [PATCH 09/11] maintenance: use pointers to check --auto Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git maintenance run' command has an '--auto' option. This is used by other Git commands such as 'git commit' or 'git fetch' to check if maintenance should be run after adding data to the repository. Previously, this --auto option was only used to add the argument to the 'git gc' command as part of the 'gc' task. We will be expanding the other tasks to perform a check to see if they should do work as part of the --auto flag, when they are enabled by config. First, update the 'gc' task to perform the auto check inside the maintenance process. This prevents running an extra 'git gc --auto' command when not needed. It also shows a model for other tasks. Second, use the 'auto_condition' function pointer as a signal for whether we enable the maintenance task under '--auto'. For instance, we do not want to enable the 'fetch' task in '--auto' mode, so that function pointer will remain NULL. Now that we are not automatically calling 'git gc', a test in t5514-fetch-multiple.sh must be changed to watch for 'git maintenance' instead. We continue to pass the '--auto' option to the 'git gc' command when necessary, because of the gc.autoDetach config option changes behavior. Likely, we will want to absorb the daemonizing behavior implied by gc.autoDetach as a maintenance.autoDetach config option. Signed-off-by: Derrick Stolee --- builtin/gc.c | 16 ++++++++++++++++ t/t5514-fetch-multiple.sh | 2 +- t/t7900-maintenance.sh | 2 +- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index dc1d260858..2b3e45eb71 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -788,9 +788,17 @@ static int maintenance_task_gc(struct maintenance_opts *opts) typedef int maintenance_task_fn(struct maintenance_opts *opts); +/* + * An auto condition function returns 1 if the task should run + * and 0 if the task should NOT run. See needs_to_gc() for an + * example. + */ +typedef int maintenance_auto_fn(void); + struct maintenance_task { const char *name; maintenance_task_fn *fn; + maintenance_auto_fn *auto_condition; unsigned enabled:1, selected:1; int selected_order; @@ -808,6 +816,7 @@ static struct maintenance_task tasks[] = { [TASK_GC] = { "gc", maintenance_task_gc, + need_to_gc, 1, }, [TASK_COMMIT_GRAPH] = { @@ -863,6 +872,11 @@ static int maintenance_run(struct maintenance_opts *opts) if (!found_selected && !tasks[i].enabled) continue; + if (opts->auto_flag && + (!tasks[i].auto_condition || + !tasks[i].auto_condition())) + continue; + if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); result = 1; @@ -877,6 +891,8 @@ static void initialize_task_config(void) { int i; struct strbuf config_name = STRBUF_INIT; + gc_config(); + for (i = 0; i < TASK__COUNT; i++) { int config_value; diff --git a/t/t5514-fetch-multiple.sh b/t/t5514-fetch-multiple.sh index de8e2f1531..bd202ec6f3 100755 --- a/t/t5514-fetch-multiple.sh +++ b/t/t5514-fetch-multiple.sh @@ -108,7 +108,7 @@ test_expect_success 'git fetch --multiple (two remotes)' ' GIT_TRACE=1 git fetch --multiple one two 2>trace && git branch -r > output && test_cmp ../expect output && - grep "built-in: git gc" trace >gc && + grep "built-in: git maintenance" trace >gc && test_line_count = 1 gc ) ' diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index cdf0bf2e60..406dc7c303 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -19,7 +19,7 @@ test_expect_success 'run [--auto|--quiet]' ' GIT_TRACE2_EVENT="$(pwd)/run-no-quiet.txt" \ git maintenance run --no-quiet 2>/dev/null && test_subcommand git gc --quiet X-Patchwork-Id: 11703581 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8F6B5722 for ; Thu, 6 Aug 2020 16:53:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E1AE92086A for ; Thu, 6 Aug 2020 16:53:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tktImup2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729686AbgHFQxe (ORCPT ); Thu, 6 Aug 2020 12:53:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43332 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729436AbgHFQu5 (ORCPT ); Thu, 6 Aug 2020 12:50:57 -0400 Received: from mail-wm1-x341.google.com (mail-wm1-x341.google.com [IPv6:2a00:1450:4864:20::341]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB4E1C002148 for ; Thu, 6 Aug 2020 08:48:53 -0700 (PDT) Received: by mail-wm1-x341.google.com with SMTP id c80so9281572wme.0 for ; Thu, 06 Aug 2020 08:48:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=LVcw63hOu0h+rjT/ZXCO/mP9lk9UyKpe3tfyE2hUF/A=; b=tktImup2KX4BwFGvvdAugUWDxNMzkRoyhzlRuU9Dv//nMYoIZDf0JE4RLUta7Wfs+G WC26FJeZn9i+RR9THwmlaXYB6QodzzRJClYUHbiRma1BCOvc8z7hS039UwohqcJlSdj1 CvmKNwC71Z8PQXS5oBvfdIcAfN0QSBRnoSCbURSk5cRp/03EseOJMv+Y7n5IXfAgSMO+ tJwN1cNhK/QdQnCsHlyWNNJj3ujO6+4EeW7V+HqguRVxGlVg2OyUPiSywiCX810k+jJi aFkplL2dNJyLVRxGwx4XLPHMgrne9hm+ACnUqccbRg/K0VgjOG4T6P47AK6mzsQw3YCG 6E1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=LVcw63hOu0h+rjT/ZXCO/mP9lk9UyKpe3tfyE2hUF/A=; b=Krq1a+ZKa0lw/RlxMq00h5bUd/5Pk5PYtzSa6ideLZ4vrXXy5ps4LMUMTdKcfDpLLI kE2+EHzMKlF3/alymo9jQQrir5Mb15JMoVp7+3rfrhD6Bg2YtJBxabEqaDA0Yc6Pgi5B 9I2RE/QIR1CyizTkl6sP+kkgO8CczPZQza+CZOW4/hRdHxbqFT63v/NWphHOYhf7SrbR qW8dKADLZRz7kBxpOEmQhy3MPTD9kBVk8TOq2uO2zkuWOBXmi+RKs4vyTn5h2puQp2Vt bEAyKw8HlCnM4ZrN/XxCEpCOUA+v4MdMdUC+lHkoE8g9yaXpuPB8i9+RrKnyopjuqSlW BTxw== X-Gm-Message-State: AOAM530nIzcGLeJJTaXuOT5APRGIqUow4Pn6ZtZ9a/ZgADK7IzBq7wDo px0gHwSMgVFz9AASfTE2LyHUX7UK X-Google-Smtp-Source: ABdhPJzqXQINtXvEvAzIuIpkPgtPVhlIGNgnmGGYgS8eKUahezjCLl1bnhf6rrTNCkLgFZFy3WDDWg== X-Received: by 2002:a05:600c:25cc:: with SMTP id 12mr8444617wml.120.1596728932433; Thu, 06 Aug 2020 08:48:52 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p14sm7536081wrx.90.2020.08.06.08.48.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:51 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:40 +0000 Subject: [PATCH 10/11] maintenance: add auto condition for commit-graph task Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Instead of writing a new commit-graph in every 'git maintenance run --auto' process (when maintenance.commit-graph.enalbed is configured to be true), only write when there are "enough" commits not in a commit-graph file. This count is controlled by the maintenance.commit-graph.auto config option. To compute the count, use a depth-first search starting at each ref, and leaving markers using the PARENT1 flag. If this count reaches the limit, then terminate early and start the task. Otherwise, this operation will peel every ref and parse the commit it points to. If these are all in the commit-graph, then this is typically a very fast operation. Users with many refs might feel a slow-down, and hence could consider updating their limit to be very small. A negative value will force the step to run every time. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 10 ++++ builtin/gc.c | 76 ++++++++++++++++++++++++++++ object.h | 1 + 3 files changed, 87 insertions(+) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index 370cbfb42f..9bd69b9df3 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -2,3 +2,13 @@ maintenance..enabled:: This boolean config option controls whether the maintenance task with name `` is run when no `--task` option is specified. By default, only `maintenance.gc.enabled` is true. + +maintenance.commit-graph.auto:: + This integer config option controls how often the `commit-graph` task + should be run as part of `git maintenance run --auto`. If zero, then + the `commit-graph` task will not run with the `--auto` option. A + negative value will force the task to run every time. Otherwise, a + positive value implies the command should run when the number of + reachable commits that are not in the commit-graph file is at least + the value of `maintenance.commit-graph.auto`. The default value is + 100. diff --git a/builtin/gc.c b/builtin/gc.c index 2b3e45eb71..20d85f6f4e 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -28,6 +28,7 @@ #include "blob.h" #include "tree.h" #include "promisor-remote.h" +#include "refs.h" #define FAILED_RUN "failed to run %s" @@ -710,6 +711,80 @@ struct maintenance_opts { int quiet; }; +/* Remember to update object flag allocation in object.h */ +#define PARENT1 (1u<<16) + +static int num_commits_not_in_graph = 0; +static int limit_commits_not_in_graph = 100; + +static int dfs_on_ref(const char *refname, + const struct object_id *oid, int flags, + void *cb_data) +{ + int result = 0; + struct object_id peeled; + struct commit_list *stack = NULL; + struct commit *commit; + + if (!peel_ref(refname, &peeled)) + oid = &peeled; + if (oid_object_info(the_repository, oid, NULL) != OBJ_COMMIT) + return 0; + + commit = lookup_commit(the_repository, oid); + if (!commit) + return 0; + if (parse_commit(commit)) + return 0; + + commit_list_append(commit, &stack); + + while (!result && stack) { + struct commit_list *parent; + + commit = pop_commit(&stack); + + for (parent = commit->parents; parent; parent = parent->next) { + if (parse_commit(parent->item) || + commit_graph_position(parent->item) != COMMIT_NOT_FROM_GRAPH || + parent->item->object.flags & PARENT1) + continue; + + parent->item->object.flags |= PARENT1; + num_commits_not_in_graph++; + + if (num_commits_not_in_graph >= limit_commits_not_in_graph) { + result = 1; + break; + } + + commit_list_append(parent->item, &stack); + } + } + + free_commit_list(stack); + return result; +} + +static int should_write_commit_graph(void) +{ + int result; + + git_config_get_int("maintenance.commit-graph.auto", + &limit_commits_not_in_graph); + + if (!limit_commits_not_in_graph) + return 0; + if (limit_commits_not_in_graph < 0) + return 1; + + result = for_each_ref(dfs_on_ref, NULL); + + clear_commit_marks_all(PARENT1); + + return result; +} + static int run_write_commit_graph(struct maintenance_opts *opts) { struct child_process child = CHILD_PROCESS_INIT; @@ -822,6 +897,7 @@ static struct maintenance_task tasks[] = { [TASK_COMMIT_GRAPH] = { "commit-graph", maintenance_task_commit_graph, + should_write_commit_graph, }, }; diff --git a/object.h b/object.h index 96a2105859..30d4e0f6a0 100644 --- a/object.h +++ b/object.h @@ -74,6 +74,7 @@ struct object_array { * list-objects-filter.c: 21 * builtin/fsck.c: 0--3 * builtin/index-pack.c: 2021 + * builtin/maintenance.c: 16 * builtin/pack-objects.c: 20 * builtin/reflog.c: 10--12 * builtin/show-branch.c: 0-------------------------------------------26 From patchwork Thu Aug 6 15:48:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Philippe Blain via GitGitGadget X-Patchwork-Id: 11703589 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BBE9A722 for ; Thu, 6 Aug 2020 16:54:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 246532311A for ; Thu, 6 Aug 2020 16:54:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="XeunsKqz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729764AbgHFQyf (ORCPT ); Thu, 6 Aug 2020 12:54:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43354 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729321AbgHFQut (ORCPT ); Thu, 6 Aug 2020 12:50:49 -0400 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0796FC002149 for ; Thu, 6 Aug 2020 08:49:00 -0700 (PDT) Received: by mail-wr1-x429.google.com with SMTP id r2so39385545wrs.8 for ; Thu, 06 Aug 2020 08:48:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=behzd+8DIw6EK2CfZW2SoL/+CxmkiSwDfqrky3nrMWY=; b=XeunsKqzqB+yYRpBXU4nZNWUC172Ey+hTjRHIOg6ThAjESfSar/MrswD11hfv8oilB DunuEZ+whLK6g8JFjFrYrtNI1zpDnHCKasvkSQeaQhaMGKCOXqwUz54eehMqFVVxizGt 389mq8ArN+zw1eSzCXWDdXr6TZh+3omir0ilCJfGd7rtXEi4xqjU57FVEzwpD18mpEsZ P1eP1RAs+tb+wL6GONHjb0gBvfNdNfID85/Yq9F2k4L5qfN3Zvui7d1bgXIzhjM+KMfG CJLT0U+3xIEAMmQvwJrHGnFfai5/sY0k7oM7vMZ65o6zUSbT8uqWo/jluXyZii83spEt C87A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=behzd+8DIw6EK2CfZW2SoL/+CxmkiSwDfqrky3nrMWY=; b=kzO1Nvz48L2hzFjD7Z2HEVicXiD0J4JZhhPndX3ca4w9/wZ8aoqn42u8VqdfLtu/m7 Ux/Wb1fMgy+gaGKwdtOf2ZB+tc4ovMf0MDlQLAuAm0AS1G++MU1nvkEqSfWWP/k6ALZH ophJi0S1c/3GSoYtL0fOzBb2J1DmJKaZS6vmO73daDxBFtwfKlvN/U167op4aP7o8mqq Mby/EW10SfJFAb7yghCWBfo9SA03lfaGbvV9vRS5oTLcPdwztjHz69+S1OIsEZWSCjQ4 z70A5HKZtE64HAliwnaYfmbRoBGya2U1B2fg6waGoyHp3B3UH5BdtB+5R3e8eK6ShxBE VsyA== X-Gm-Message-State: AOAM5330GfnCJiME7U0K3pDUkhrhY7EcIsiyVOj5YNCqXTcw3NmnWdPO vmlH2vcwysPRj3OUYe5UaXTe8jgK X-Google-Smtp-Source: ABdhPJy2hTl7WgqHVlnaTXX5d8e281dHgYRLCK05Q/+SEXCEaLLviSMmx+EIM8uPL/KJ+G8M+TMqzQ== X-Received: by 2002:adf:fe12:: with SMTP id n18mr7952617wrr.295.1596728933210; Thu, 06 Aug 2020 08:48:53 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n11sm6453718wmi.15.2020.08.06.08.48.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 08:48:52 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Thu, 06 Aug 2020 15:48:41 +0000 Subject: [PATCH 11/11] maintenance: add trace2 regions for task execution Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Signed-off-by: Derrick Stolee --- builtin/gc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/builtin/gc.c b/builtin/gc.c index 20d85f6f4e..8162bca974 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -953,10 +953,12 @@ static int maintenance_run(struct maintenance_opts *opts) !tasks[i].auto_condition())) continue; + trace2_region_enter("maintenance", tasks[i].name, r); if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); result = 1; } + trace2_region_leave("maintenance", tasks[i].name, r); } rollback_lock_file(&lk);