From patchwork Tue Aug 25 18:39:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11736213 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2FE92913 for ; Tue, 25 Aug 2020 18:40:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 17C362074D for ; Tue, 25 Aug 2020 18:40:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IyUeLqRz" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726716AbgHYSkL (ORCPT ); Tue, 25 Aug 2020 14:40:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726119AbgHYSkJ (ORCPT ); Tue, 25 Aug 2020 14:40:09 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A77D6C061755 for ; Tue, 25 Aug 2020 11:40:08 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id y3so13838926wrl.4 for ; Tue, 25 Aug 2020 11:40:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5lURrJo2phQFyTaCa9VzL15Edu5uwyU3aVspJO/+d8Q=; b=IyUeLqRzNxy08lQGHSiYrY/oykI2rKIKiWeApAdD7YzlwENc9tFNTed8jQeB+mijpZ qQDPjVting8zlQH5NbL4M/COmn5BIXxF2dcqTzqGmxau1atVd11W1GmSU/IF2EfPr2Ln WLcHi+BwAR11jbHqyNj5KdciZluxTh10kYmw5C0Tm8oIYrQx36Mk2zIShBD9h+UG6oza QhGzHcXIKqnfGesz/bDN5b5gvJbL4iN2P7bCauGV1qAeDBQsozl0IoZPoqtyliOQfzxE AhvSlWJjfxUwBZF/eAfzvPr4RfzOVpsiNA12x0yGRTYMs6ddoCIFD+X27id2cIITmQ8u iJQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5lURrJo2phQFyTaCa9VzL15Edu5uwyU3aVspJO/+d8Q=; b=RbKtnK4eCiElCaRk1QPEgNB9+WyxnFUvvL8ItylX6c9UzB7M3faH6qFsBx79+5KuVy NkW+D2gieJVm3/LSDS13gamNnnmPR6snLssLfFkyjjuIGm25olxdbzC29rbhqVIMlWL7 XHI50Nzp9pNn8ADDYMnaTX2hfSrJI14RBitYBoVjfshIYJz2VH3LeNuVhkpzxWXEMVVO rooERC+nlG3ggqGvV2epGZmR6HhU6RUvLZF7LQRaS2MhEj8NMsARh7vnZ1/S0dnjSmno sQFWVAl3zCCp7jTggUcju5kn3a/FpHwN2Gbk2BTC/Jj2qXhnKVwvbm1gOSoKbfkDXIoW 7pWA== X-Gm-Message-State: AOAM531XXDuNz5USnvBp8AlGO2v0b1sqjIumPltxBQWhoiOzi8AFqa/M LNfxbvMgfV0wSM+kJRv/Owh0jwHMaOg= X-Google-Smtp-Source: ABdhPJxV/zLJ3mmoZFyHwzGEZXYCD/2EE2QlHgxH/t/QppCyC4g1t/31HcUMr3Lgxj/DxBJeb7CBAQ== X-Received: by 2002:adf:c182:: with SMTP id x2mr375124wre.400.1598380807024; Tue, 25 Aug 2020 11:40:07 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n21sm7138855wmc.11.2020.08.25.11.40.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:40:06 -0700 (PDT) Message-Id: <5fdd8188b1d9b6efc2803b557b3ba344e184d22e.1598380805.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:39:58 +0000 Subject: [PATCH v2 1/7] maintenance: optionally skip --auto process Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Some commands run 'git maintenance run --auto --[no-]quiet' after doing their normal work, as a way to keep repositories clean as they are used. Currently, users who do not want this maintenance to occur would set the 'gc.auto' config option to 0 to avoid the 'gc' task from running. However, this does not stop the extra process invocation. On Windows, this extra process invocation can be more expensive than necessary. Allow users to drop this extra process by setting 'maintenance.auto' to 'false'. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 5 +++++ run-command.c | 6 ++++++ t/t7900-maintenance.sh | 13 +++++++++++++ 3 files changed, 24 insertions(+) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index a0706d8f09..06db758172 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -1,3 +1,8 @@ +maintenance.auto:: + This boolean config option controls whether some commands run + `git maintenance run --auto` after doing their normal work. Defaults + to true. + maintenance..enabled:: This boolean config option controls whether the maintenance task with name `` is run when no `--task` option is specified to diff --git a/run-command.c b/run-command.c index 2ee59acdc8..ea4d0fb4b1 100644 --- a/run-command.c +++ b/run-command.c @@ -7,6 +7,7 @@ #include "strbuf.h" #include "string-list.h" #include "quote.h" +#include "config.h" void child_process_init(struct child_process *child) { @@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task, int run_auto_maintenance(int quiet) { + int enabled; struct child_process maint = CHILD_PROCESS_INIT; + if (!git_config_get_bool("maintenance.auto", &enabled) && + !enabled) + return 0; + maint.git_cmd = 1; strvec_pushl(&maint.args, "maintenance", "run", "--auto", NULL); strvec_push(&maint.args, quiet ? "--quiet" : "--no-quiet"); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 6f878b0141..e0ba19e1ff 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -26,6 +26,19 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet .enabled' ' git config maintenance.gc.enabled false && git config maintenance.commit-graph.enabled true && From patchwork Tue Aug 25 18:39:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11736217 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87B22109B for ; Tue, 25 Aug 2020 18:40:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6D88C2076C for ; Tue, 25 Aug 2020 18:40:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Y8xkz5E6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726752AbgHYSkM (ORCPT ); Tue, 25 Aug 2020 14:40:12 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44404 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726158AbgHYSkJ (ORCPT ); Tue, 25 Aug 2020 14:40:09 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A831C061756 for ; Tue, 25 Aug 2020 11:40:09 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id u18so3603537wmc.3 for ; Tue, 25 Aug 2020 11:40:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ablqq3jT9ZyD0iEPR8c7GLfn5zx1HtHOn/rEKNsk8Lo=; b=Y8xkz5E6x73AuzfQIMJPHN9VXoiay1JLIhL8UJlDhiaV8WaULafdtMx0AN6k+PaOLN /hxLjRi5q8tnMWmW6jwcx5MRKbANiN3fJmnm6GCOUYhHSwiDV+6Vn3o/9SczV/fRg8w/ oJFR9/x4OEeAIvMjY2zeEPAX+dwMSP3xCjvubEbPxLREl2FV1YPOgvBg1dAtGS4DOp0P Maq3TLf4dBD1gXMr66zC1VbhQCUk9EfVwjn5X3i5BGPzFUEKD79GFTv8V1NjwCbGNCxE BGkwKG2cSoeWfsNCt+/e60kvIBQvK8GAD4sfVEIinEFTPEW8U9Kg+18SWxp51oR0xSq+ YoXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ablqq3jT9ZyD0iEPR8c7GLfn5zx1HtHOn/rEKNsk8Lo=; b=TZ6/zV7KASb2lOC7j+aOyZviA5vqBUGrE9pUmcUhyHuom/suvGgZ6byjVSesv6ijNq BO1up2HFhmhUR1mIi3WkOz1Ae4LPzjUYM/idq0DSZDU7mz20aHDxPHr9D0Zd4X5gOQcJ uaanIFoP0NhTG51fBUQ9P3pVTSw2fnMYPNMkvxVhpppfPO88+PFlt0hmX+dntjvO9iPL G1u9OG2xCyXEs6d4KEPTMTMi1eNN7qmuHqP/jypp78rWMTNZVJybJ8vK37FUaVDDVjVd YeKCBr9iYsfyFF++2O+wfiTTbKjfxURJ+VP3vFJHnBzHbhhHG/wXVNNUZOTFbAhKHe+d lxBw== X-Gm-Message-State: AOAM5330ZyUDEEdrfJKtYixwAeuDzV2gzN/qvY/AGHDCic0ONqnej6lM FKRQKXP4PPHStpwpjUgNbsEXaPBBs1M= X-Google-Smtp-Source: ABdhPJyC9W9wFkEdRNwzQHoUi5fECqpiy93qWAPyiq5zr52/Zux3NLm+sV6RC59YQleCJ3bgPVvbkg== X-Received: by 2002:a1c:c906:: with SMTP id f6mr3440267wmb.5.1598380807846; Tue, 25 Aug 2020 11:40:07 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a7sm7115641wmj.24.2020.08.25.11.40.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:40:07 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:39:59 +0000 Subject: [PATCH v2 2/7] maintenance: store the "last run" time in config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Users may want to run certain maintenance tasks only so often. Update the local config with a new 'maintenance..lastRun' config option that stores the timestamp just before running the maintenance task. I selected the timestamp before the task, as opposed to after the task, for a couple reasons: 1. The time the task takes to execute should not contribute to the interval between running the tasks. If a daily task takes 10 minutes to run, then every day the execution will drift by at least 10 minutes. 2. If the task fails for some unforseen reason, it would be good to indicate that we _attempted_ the task at a certain timestamp. This will avoid spamming a repository that is in a bad state. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 5 +++++ builtin/gc.c | 16 ++++++++++++++++ t/t7900-maintenance.sh | 10 ++++++++++ 3 files changed, 31 insertions(+) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index 06db758172..8dd34169da 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -10,6 +10,11 @@ maintenance..enabled:: `--task` option exists. By default, only `maintenance.gc.enabled` is true. +maintenance..lastRun:: + This config value is automatically updated by Git when the task + `` is run. It stores a timestamp representing the most-recent + run of the ``. + maintenance.commit-graph.auto:: This integer config option controls how often the `commit-graph` task should be run as part of `git maintenance run --auto`. If zero, then diff --git a/builtin/gc.c b/builtin/gc.c index f8459df04c..fb6f231a5c 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1212,6 +1212,20 @@ static int compare_tasks_by_selection(const void *a_, const void *b_) return b->selected_order - a->selected_order; } +static void update_last_run(struct maintenance_task *task) +{ + timestamp_t now = approxidate("now"); + struct strbuf config = STRBUF_INIT; + struct strbuf value = STRBUF_INIT; + strbuf_addf(&config, "maintenance.%s.lastrun", task->name); + strbuf_addf(&value, "%"PRItime"", now); + + git_config_set(config.buf, value.buf); + + strbuf_release(&config); + strbuf_release(&value); +} + static int maintenance_run_tasks(struct maintenance_run_opts *opts) { int i, found_selected = 0; @@ -1254,6 +1268,8 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) !tasks[i].auto_condition())) continue; + update_last_run(&tasks[i]); + trace2_region_enter("maintenance", tasks[i].name, r); if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index e0ba19e1ff..a985ce3674 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -264,4 +264,14 @@ test_expect_success 'maintenance.incremental-repack.auto' ' done ' +test_expect_success 'tasks update maintenance..lastRun' ' + git config --unset maintenance.commit-graph.lastrun && + GIT_TRACE2_EVENT="$(pwd)/run.txt" \ + GIT_TEST_DATE_NOW=1595000000 \ + git maintenance run --task=commit-graph 2>/dev/null && + test_subcommand git commit-graph write --split --reachable \ + --no-progress X-Patchwork-Id: 11736219 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F3E9E913 for ; Tue, 25 Aug 2020 18:40:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D92DD2076C for ; Tue, 25 Aug 2020 18:40:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="BGpvMS6X" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726757AbgHYSkS (ORCPT ); Tue, 25 Aug 2020 14:40:18 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44410 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726709AbgHYSkK (ORCPT ); Tue, 25 Aug 2020 14:40:10 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6221BC061574 for ; Tue, 25 Aug 2020 11:40:10 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id b17so13064879wru.2 for ; Tue, 25 Aug 2020 11:40:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=OdzOCvRhhYnmY+eXGmtafDY7myv9Ro1QU1DnL1wqlv8=; b=BGpvMS6XtNzU/0ga8QVZw1dpeBlgBpbcp5enUB5Wj3qXh8HFWJExWJYTh47XJkKBGd EEe6Xg3ApbbpInoNYA1IAEiAM1pwxPvt1XKGzjN17DgecShoxwJaeyLCWD6KzUQLQliU GewoYALhdksnJQW0JupmdxlnDkqrTZYaKZ/DqJfgwGhN386//qaAtb+291bSK8q+Ynlg i3oXNvvAOw9fW0mOCMNiZdIHi8Ga7gt/wNhA10/9s2neqOyhYBDwQ25Qyn2dzXWOjsyk P84yoEzpGlqGB2D7C+MR6thXv1f5ObggZQlfQKCIBHJTGXlTknA23tZa5genk8Pc1Yr7 w0Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=OdzOCvRhhYnmY+eXGmtafDY7myv9Ro1QU1DnL1wqlv8=; b=BsLCUnkTbDKyVcHEWVM6jHXTbutuBBwCTM5d1krXCxeOvTRbZZiQxQdZkG48tJBO/2 x1vmIHNdNP2M41rG2LNPxADVlqZpQF7XW4NNRTBqZP7uJ3g5hH+kxxBbxhgc+UHpwyRD zpjTtbm5lqrq3mQA+4ik9jFiUN3heElGS3oVVa7Mn+yIeLjUHV34sks2DP8fGiM4qDTw 9WNArQnB7iE2VO9RfamZ44UFGx12dSC57piK08NnFPthNyUoBDQTgwvq2GITkhRfnuxU rwkyyPXpsCrgNQPVvd2QwoW3IPtTAxPnNep/YfrJYY43B/3S0X3JBSOxOoZEtWfjFG9m 43xQ== X-Gm-Message-State: AOAM530YiyedUjKONRNo53+XLJv+bfQGdg09sJ6Gn08YM3Y9phEStX43 DWyyPVwUYB2D3lYyWLd86ESg4VxXVX8= X-Google-Smtp-Source: ABdhPJycGLTAwsFETHl26pdznc09Q6cl+KuSxDgXQ4s5+4zJzvsEEwPJBNMw7feRjQOZw796PIdG0Q== X-Received: by 2002:adf:ff89:: with SMTP id j9mr4497840wrr.179.1598380808683; Tue, 25 Aug 2020 11:40:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m1sm7182940wmc.28.2020.08.25.11.40.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:40:08 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:40:00 +0000 Subject: [PATCH v2 3/7] maintenance: add --scheduled option and config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee A user may want to run certain maintenance tasks based on frequency, not conditions given in the repository. For example, the user may want to perform a 'prefetch' task every hour, or 'gc' task every day. To assist, update the 'git maintenance run --scheduled' command to check the config for the last run of that task and add a number of seconds. The task would then run only if the current time is beyond that minimum timestamp. Add a '--scheduled' option to 'git maintenance run' to only run tasks that have had enough time pass since their last run. This is done for each enabled task by checking if the current timestamp is at least as large as the sum of 'maintenance..lastRun' and 'maintenance..schedule' in the Git config. This second value is new to this commit, storing a number of seconds intended between runs. A user could then set up an hourly maintenance run with the following cron table: 0 * * * * git -C maintenance run --scheduled Then, the user could configure the repository with the following config values: maintenance.prefetch.schedule 3000 maintenance.gc.schedule 86000 These numbers are slightly lower than one hour and one day (in seconds). The cron schedule will enforce the hourly run rate, but we can use these schedules to ensure the 'gc' task runs once a day. The error is given because the *.lastRun config option is specified at the _start_ of the task run. Otherwise, a slow task run could shift the "daily" job of 'gc' from a 10:00pm run to 11:00pm run, or later. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 9 +++++ Documentation/git-maintenance.txt | 13 ++++++- builtin/gc.c | 54 ++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 20 +++++++++++ 4 files changed, 92 insertions(+), 4 deletions(-) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index 8dd34169da..caacacd322 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -15,6 +15,15 @@ maintenance..lastRun:: `` is run. It stores a timestamp representing the most-recent run of the ``. +maintenance..schedule:: + This config option controls whether or not the given `` runs + during a `git maintenance run --scheduled` command. If the option + is an integer value `S`, then the `` is run when the current + time is `S` seconds after the timestamp stored in + `maintenance..lastRun`. If the option has no value or a + non-integer value, then the task will never run with the `--scheduled` + option. + maintenance.commit-graph.auto:: This integer config option controls how often the `commit-graph` task should be run as part of `git maintenance run --auto`. If zero, then diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index b44efb05a3..2bc02c65e4 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -107,7 +107,18 @@ OPTIONS only if certain thresholds are met. For example, the `gc` task runs when the number of loose objects exceeds the number stored in the `gc.auto` config setting, or when the number of pack-files - exceeds the `gc.autoPackLimit` config setting. + exceeds the `gc.autoPackLimit` config setting. Not compatible with + the `--scheduled` option. + +--scheduled:: + When combined with the `run` subcommand, run maintenance tasks + only if certain time conditions are met, as specified by the + `maintenance..schedule` config value for each ``. + This config value specifies a number of seconds since the last + time that task ran, according to the `maintenance..lastRun` + config value. The tasks that are tested are those provided by + the `--task=` option(s) or those with + `maintenance..enabled` set to true. --quiet:: Do not report progress or other information over `stderr`. diff --git a/builtin/gc.c b/builtin/gc.c index fb6f231a5c..5726a9a3b3 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -705,12 +705,13 @@ int cmd_gc(int argc, const char **argv, const char *prefix) } static const char * const builtin_maintenance_run_usage[] = { - N_("git maintenance run [--auto] [--[no-]quiet] [--task=]"), + N_("git maintenance run [--auto] [--[no-]quiet] [--task=] [--scheduled]"), NULL }; struct maintenance_run_opts { int auto_flag; + int scheduled; int quiet; }; @@ -1157,7 +1158,8 @@ struct maintenance_task { const char *name; maintenance_task_fn *fn; maintenance_auto_fn *auto_condition; - unsigned enabled:1; + unsigned enabled:1, + scheduled:1; /* -1 if not selected. */ int selected_order; @@ -1268,6 +1270,9 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) !tasks[i].auto_condition())) continue; + if (opts->scheduled && !tasks[i].scheduled) + continue; + update_last_run(&tasks[i]); trace2_region_enter("maintenance", tasks[i].name, r); @@ -1282,6 +1287,29 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) return result; } +static void fill_schedule_info(struct maintenance_task *task, + const char *config_name, + timestamp_t schedule_delay) +{ + timestamp_t now = approxidate("now"); + char *value = NULL; + struct strbuf last_run = STRBUF_INIT; + int64_t previous_run; + + strbuf_addf(&last_run, "maintenance.%s.lastrun", task->name); + + if (git_config_get_string(last_run.buf, &value)) + task->scheduled = 1; + else { + previous_run = git_config_int64(last_run.buf, value); + if (now >= previous_run + schedule_delay) + task->scheduled = 1; + } + + free(value); + strbuf_release(&last_run); +} + static void initialize_task_config(void) { int i; @@ -1290,13 +1318,28 @@ static void initialize_task_config(void) for (i = 0; i < TASK__COUNT; i++) { int config_value; + char *config_str; - strbuf_setlen(&config_name, 0); + strbuf_reset(&config_name); strbuf_addf(&config_name, "maintenance.%s.enabled", tasks[i].name); if (!git_config_get_bool(config_name.buf, &config_value)) tasks[i].enabled = config_value; + + strbuf_reset(&config_name); + strbuf_addf(&config_name, "maintenance.%s.schedule", + tasks[i].name); + + if (!git_config_get_string(config_name.buf, &config_str)) { + timestamp_t schedule_delay = git_config_int64( + config_name.buf, + config_str); + fill_schedule_info(&tasks[i], + config_name.buf, + schedule_delay); + free(config_str); + } } strbuf_release(&config_name); @@ -1340,6 +1383,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) struct option builtin_maintenance_run_options[] = { OPT_BOOL(0, "auto", &opts.auto_flag, N_("run tasks based on the state of the repository")), + OPT_BOOL(0, "scheduled", &opts.scheduled, + N_("run tasks based on time intervals")), OPT_BOOL(0, "quiet", &opts.quiet, N_("do not report progress or other information over stderr")), OPT_CALLBACK_F(0, "task", NULL, N_("task"), @@ -1360,6 +1405,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) builtin_maintenance_run_usage, PARSE_OPT_STOP_AT_NON_OPTION); + if (opts.auto_flag + opts.scheduled > 1) + die(_("use at most one of the --auto and --scheduled options")); + if (argc != 0) usage_with_options(builtin_maintenance_run_usage, builtin_maintenance_run_options); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index a985ce3674..3e0c5f1ca8 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -264,6 +264,11 @@ test_expect_success 'maintenance.incremental-repack.auto' ' done ' +test_expect_success '--auto and --scheduled incompatible' ' + test_must_fail git maintenance run --auto --scheduled 2>err && + test_i18ngrep "at most one" err +' + test_expect_success 'tasks update maintenance..lastRun' ' git config --unset maintenance.commit-graph.lastrun && GIT_TRACE2_EVENT="$(pwd)/run.txt" \ @@ -274,4 +279,19 @@ test_expect_success 'tasks update maintenance..lastRun' ' test_cmp_config 1595000000 maintenance.commit-graph.lastrun ' +test_expect_success '--scheduled with specific time' ' + git config maintenance.commit-graph.schedule 100 && + GIT_TRACE2_EVENT="$(pwd)/too-soon.txt" \ + GIT_TEST_DATE_NOW=1595000099 \ + git maintenance run --scheduled 2>/dev/null && + test_subcommand ! git commit-graph write --split --reachable \ + --no-progress /dev/null && + test_subcommand git commit-graph write --split --reachable \ + --no-progress X-Patchwork-Id: 11736221 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 70A8E913 for ; Tue, 25 Aug 2020 18:40:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 52DBB2074D for ; Tue, 25 Aug 2020 18:40:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="sdYgwFQX" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726374AbgHYSkU (ORCPT ); Tue, 25 Aug 2020 14:40:20 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44412 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726737AbgHYSkL (ORCPT ); Tue, 25 Aug 2020 14:40:11 -0400 Received: from mail-wr1-x441.google.com (mail-wr1-x441.google.com [IPv6:2a00:1450:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 53AFAC061755 for ; Tue, 25 Aug 2020 11:40:11 -0700 (PDT) Received: by mail-wr1-x441.google.com with SMTP id a5so13880436wrm.6 for ; Tue, 25 Aug 2020 11:40:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=LJaF+KBq6DinuUhew6phz3UG9LLuWbKh+8DAyadGgBc=; b=sdYgwFQX/DTcYdRf7O0+M2uMBJSZDssmvdXznVZt7thLlZNjhZTNResKxaujJqnatM KrwRHSlFsDv/wdvDckpz7YBNpoLHHp3gI5Lf+5NJsW/RxolqTphg9PPgb5+ue22l87R7 UM0SVMjN3GblLiXp1ltaiV7+r1eeTCawGGvO13mYlsc1m9mtWgVZzpxG1GjWS/phDgsg aeb4ucvhTtcjwtO6ywyd86lr7vuRc9+74Pq79xKlM1k7iOEOIye7cqaLalhi9nbvdX8s jQt2luFNd39miV2EsYmDDbahszxWSbLv55KvaEUIMN3oxFNdRcQnjCzTXvrl44pfBhn6 IV+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=LJaF+KBq6DinuUhew6phz3UG9LLuWbKh+8DAyadGgBc=; b=O3tvET2yYvbomv2DKBuKB5xpc3oy+dFUqqrXPBfrQQPBrLWhLmlpn+ak5TINAOkDW2 oukLlK1yRPNWbcQghAKACmhN195Fmx7GDMJ2PC1SatAr7PyJkLY4CxWDnoqzPFctezCW aqm+KiCNxkr6oM/9yWC1A24ZgRacufu8JUXOYZ1KJON8kbn4lQd/HkFJyRW6Z+kvc35a r24JgkoYvmFJ8leiXNfL9/gBqU9bc5BYDgHhVrOvptkRTYhLpoQ+WNceHkVjpY9g7ol+ Ih9hiLVfWppmXTUTVVBC0ovp3YhSaCME8vJ4gViAvowawCgjyQl7WG1k9VNdgYNZRR78 NGTw== X-Gm-Message-State: AOAM533vFnQY4FCH0t1ctcdn+TsxWJOL8SZF2e8BMCAxVCYcgHdHnnSi wWVj6y6xCSjMeKW0hqLcT6AkUn1cZjE= X-Google-Smtp-Source: ABdhPJxq8JexgBzbHrs0grGuqShjc5ImkeedCtRZt0p8A8MLQbqO0k4gAOMRdr2p/2rOwsGSSvl2QA== X-Received: by 2002:adf:ea0f:: with SMTP id q15mr11652313wrm.113.1598380809658; Tue, 25 Aug 2020 11:40:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n4sm18279439wrv.8.2020.08.25.11.40.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:40:09 -0700 (PDT) Message-Id: <0314258c5cbb8fd771c35e433bf6be95297c4597.1598380805.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:40:01 +0000 Subject: [PATCH v2 4/7] for-each-repo: run subcommands on configured repos Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee It can be helpful to store a list of repositories in global or system config and then iterate Git commands on that list. Create a new builtin that makes this process simple for experts. We will use this builtin to run scheduled maintenance on all configured repositories in a future change. The test is very simple, but does highlight that the "--" argument is optional. Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++ Makefile | 1 + builtin.h | 1 + builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++ command-list.txt | 1 + git.c | 1 + t/t0068-for-each-repo.sh | 30 +++++++++++++++ 8 files changed, 152 insertions(+) create mode 100644 Documentation/git-for-each-repo.txt create mode 100644 builtin/for-each-repo.c create mode 100755 t/t0068-for-each-repo.sh diff --git a/.gitignore b/.gitignore index a5808fa30d..5eb2a2be71 100644 --- a/.gitignore +++ b/.gitignore @@ -67,6 +67,7 @@ /git-filter-branch /git-fmt-merge-msg /git-for-each-ref +/git-for-each-repo /git-format-patch /git-fsck /git-fsck-objects diff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt new file mode 100644 index 0000000000..94bd19da26 --- /dev/null +++ b/Documentation/git-for-each-repo.txt @@ -0,0 +1,59 @@ +git-for-each-repo(1) +==================== + +NAME +---- +git-for-each-repo - Run a Git command on a list of repositories + + +SYNOPSIS +-------- +[verse] +'git for-each-repo' --config= [--] + + +DESCRIPTION +----------- +Run a Git command on a list of repositories. The arguments after the +known options or `--` indicator are used as the arguments for the Git +subprocess. + +THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE. + +For example, we could run maintenance on each of a list of repositories +stored in a `maintenance.repo` config variable using + +------------- +git for-each-repo --config=maintenance.repo maintenance run +------------- + +This will run `git -C maintenance run` for each value `` +in the multi-valued config variable `maintenance.repo`. + + +OPTIONS +------- +--config=:: + Use the given config variable as a multi-valued list storing + absolute path names. Iterate on that list of paths to run + the given arguments. ++ +These config values are loaded from system, global, and local Git config, +as available. If `git for-each-repo` is run in a directory that is not a +Git repository, then only the system and global config is used. + + +SUBPROCESS BEHAVIOR +------------------- + +If any `git -C ` subprocess returns a non-zero exit code, +then the `git for-each-repo` process returns that exit code without running +more subprocesses. + +Each `git -C ` subprocess inherits the standard file +descriptors `stdin`, `stdout`, and `stderr`. + + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Makefile b/Makefile index 65f8cfb236..7c588ff036 100644 --- a/Makefile +++ b/Makefile @@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o BUILTIN_OBJS += builtin/fetch.o BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o +BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o diff --git a/builtin.h b/builtin.h index 17c1c0ce49..ff7c6e5aa9 100644 --- a/builtin.h +++ b/builtin.h @@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix); int cmd_fetch_pack(int argc, const char **argv, const char *prefix); int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix); int cmd_for_each_ref(int argc, const char **argv, const char *prefix); +int cmd_for_each_repo(int argc, const char **argv, const char *prefix); int cmd_format_patch(int argc, const char **argv, const char *prefix); int cmd_fsck(int argc, const char **argv, const char *prefix); int cmd_gc(int argc, const char **argv, const char *prefix); diff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c new file mode 100644 index 0000000000..5bba623ff1 --- /dev/null +++ b/builtin/for-each-repo.c @@ -0,0 +1,58 @@ +#include "cache.h" +#include "config.h" +#include "builtin.h" +#include "parse-options.h" +#include "run-command.h" +#include "string-list.h" + +static const char * const for_each_repo_usage[] = { + N_("git for-each-repo --config= "), + NULL +}; + +static int run_command_on_repo(const char *path, + void *cbdata) +{ + int i; + struct child_process child = CHILD_PROCESS_INIT; + struct strvec *args = (struct strvec *)cbdata; + + child.git_cmd = 1; + strvec_pushl(&child.args, "-C", path, NULL); + + for (i = 0; i < args->nr; i++) + strvec_push(&child.args, args->v[i]); + + return run_command(&child); +} + +int cmd_for_each_repo(int argc, const char **argv, const char *prefix) +{ + static const char *config_key = NULL; + int i, result = 0; + const struct string_list *values; + struct strvec args = STRVEC_INIT; + + const struct option options[] = { + OPT_STRING(0, "config", &config_key, N_("config"), + N_("config key storing a list of repository paths")), + OPT_END() + }; + + argc = parse_options(argc, argv, prefix, options, for_each_repo_usage, + PARSE_OPT_STOP_AT_NON_OPTION); + + if (!config_key) + die(_("missing --config=")); + + for (i = 0; i < argc; i++) + strvec_push(&args, argv[i]); + + values = repo_config_get_value_multi(the_repository, + config_key); + + for (i = 0; !result && i < values->nr; i++) + result = run_command_on_repo(values->items[i].string, &args); + + return result; +} diff --git a/command-list.txt b/command-list.txt index 0e3204e7d1..581499be82 100644 --- a/command-list.txt +++ b/command-list.txt @@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories git-filter-branch ancillarymanipulators git-fmt-merge-msg purehelpers git-for-each-ref plumbinginterrogators +git-for-each-repo plumbinginterrogators git-format-patch mainporcelain git-fsck ancillaryinterrogators complete git-gc mainporcelain diff --git a/git.c b/git.c index 24f250d29a..1cab64b5d1 100644 --- a/git.c +++ b/git.c @@ -511,6 +511,7 @@ static struct cmd_struct commands[] = { { "fetch-pack", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT }, { "fmt-merge-msg", cmd_fmt_merge_msg, RUN_SETUP }, { "for-each-ref", cmd_for_each_ref, RUN_SETUP }, + { "for-each-repo", cmd_for_each_repo, RUN_SETUP_GENTLY }, { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, diff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh new file mode 100755 index 0000000000..136b4ec839 --- /dev/null +++ b/t/t0068-for-each-repo.sh @@ -0,0 +1,30 @@ +#!/bin/sh + +test_description='git for-each-repo builtin' + +. ./test-lib.sh + +test_expect_success 'run based on configured value' ' + git init one && + git init two && + git init three && + git -C two commit --allow-empty -m "DID NOT RUN" && + git config run.key "$TRASH_DIRECTORY/one" && + git config --add run.key "$TRASH_DIRECTORY/three" && + git for-each-repo --config=run.key commit --allow-empty -m "ran" && + git -C one log -1 --pretty=format:%s >message && + grep ran message && + git -C two log -1 --pretty=format:%s >message && + ! grep ran message && + git -C three log -1 --pretty=format:%s >message && + grep ran message && + git for-each-repo --config=run.key -- commit --allow-empty -m "ran again" && + git -C one log -1 --pretty=format:%s >message && + grep again message && + git -C two log -1 --pretty=format:%s >message && + ! grep again message && + git -C three log -1 --pretty=format:%s >message && + grep again message +' + +test_done From patchwork Tue Aug 25 18:40:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11736227 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F1E3913 for ; Tue, 25 Aug 2020 18:40:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F12932071E for ; Tue, 25 Aug 2020 18:40:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A8pbRyCr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726783AbgHYSk0 (ORCPT ); Tue, 25 Aug 2020 14:40:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44420 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726740AbgHYSkM (ORCPT ); Tue, 25 Aug 2020 14:40:12 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 28DC3C061574 for ; Tue, 25 Aug 2020 11:40:12 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id b17so13064989wru.2 for ; Tue, 25 Aug 2020 11:40:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=3PhVGgGEqdc/N+TszgKfIi4lzL1p/ldjNnJzfUXIgec=; b=A8pbRyCrYF7YFx1AfEaT+O9XjW4rSEvsRDRg57vuGtM8WNFYvChMujz9zq33gSUEEX a4AvJfXp191c/SHybo4PfKWO5i833oma475yaNFCa+981h1h37DbZib6BDI8CQMHl3aG mhahffdIMX7k63E/I/PwmYM3O0RvSb71OuqaDz0U0KKRzOg7qXjM4l1z89cbOJD/O4eN vz6n+9xQnL1y3SMIOQ9IRX4qmsGPx+iXxbAOb99x1nr3UOuhKNOtAm8pkQ+HBdfFDG7E rjOhlpk0FHQ1JYAz1NX2ZL1PRJkOhokVsGATwleQCtty62+UdsNyXp8qFshw8gEZc7VW zcnQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=3PhVGgGEqdc/N+TszgKfIi4lzL1p/ldjNnJzfUXIgec=; b=a0/j3esx+lXuTmRpz8ZeW0r30+zfCzb5lA0Ii0WebOT3w3eccN+iYTcRS1uTtjNUDA WY6pIT0CxvMdDzr8EH5ALy2457i8Cv1OCA/YDvqroZ7sBuMKXnMOA2kGFWIMgbOFRdpG AJFYPnpO9h9xNecNHmfjsv5bKZptHcjKsIIEHwR8lS740MmjfodM9xMNUsVxItWck4vd DgN1riR6Yda0Okk7nuZVJmj5t41ET1Xa4wcGkT1TAxCneG9hLnl5IpJZ0p6U/YCOzgxu Fot+h/5PZZxz5EbqPOdq17zczBGz+kSKNZTRiyN6tuu1aQL6dNNFjust8yzhjpUnTz6p 1FhQ== X-Gm-Message-State: AOAM531G8bzj0nA5EJle95gD6XoB2X/pHg3+5yXSIECy7In6yB7BB+/q lwRj+zvHBDDKohpPhhbz+hih1i7YhGY= X-Google-Smtp-Source: ABdhPJx6qJhiuClVUI92Eljzv+y9mkelLHsp/ztzI1OjQxKPWH3jEQUrogqLTznkJapKQDCFcpEXRA== X-Received: by 2002:adf:e982:: with SMTP id h2mr12723759wrm.394.1598380810614; Tue, 25 Aug 2020 11:40:10 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q12sm21285172wrn.94.2020.08.25.11.40.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:40:09 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:40:02 +0000 Subject: [PATCH v2 5/7] maintenance: add [un]register subcommands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee In preparation for launching background maintenance from the 'git maintenance' builtin, create register/unregister subcommands. These commands update the new 'maintenance.repos' config option in the global config so the background maintenance job knows which repositories to maintain. These commands allow users to add a repository to the background maintenance list without disrupting the actual maintenance mechanism. For example, a user can run 'git maintenance register' when no background maintenance is running and it will not start the background maintenance. A later update to start running background maintenance will then pick up this repository automatically. The opposite example is that a user can run 'git maintenance unregister' to remove the current repository from background maintenance without halting maintenance for other repositories. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 14 ++++++++ builtin/gc.c | 55 ++++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 17 +++++++++- 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 2bc02c65e4..c42a176a95 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -29,6 +29,15 @@ Git repository. SUBCOMMANDS ----------- +register:: + Initialize Git config values so any scheduled maintenance will + start running on this repository. This adds the repository to the + `maintenance.repo` config variable in the current user's global + config and enables some recommended configuration values for + `maintenance..schedule`. The tasks that are enabled are safe + for running in the background without disrupting foreground + processes. + run:: Run one or more maintenance tasks. If one or more `--task` options are specified, then those tasks are run in that order. Otherwise, @@ -36,6 +45,11 @@ run:: config options are true. By default, only `maintenance.gc.enabled` is true. +unregister:: + Remove the current repository from background maintenance. This + only removes the repository from the configured list. It does not + stop the background maintenance processes from running. + TASKS ----- diff --git a/builtin/gc.c b/builtin/gc.c index 5726a9a3b3..5218d52cb7 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1414,7 +1414,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) return maintenance_run_tasks(&opts); } -static const char builtin_maintenance_usage[] = N_("git maintenance run []"); +static int maintenance_register(void) +{ + struct child_process config_set = CHILD_PROCESS_INIT; + struct child_process config_get = CHILD_PROCESS_INIT; + + /* There is no current repository, so skip registering it */ + if (!the_repository || !the_repository->gitdir) + return 0; + + config_get.git_cmd = 1; + strvec_pushl(&config_get.args, "config", "--global", "--get", "maintenance.repo", + the_repository->worktree ? the_repository->worktree + : the_repository->gitdir, + NULL); + config_get.out = -1; + + if (start_command(&config_get)) + return error(_("failed to run 'git config'")); + + /* We already have this value in our config! */ + if (!finish_command(&config_get)) + return 0; + + config_set.git_cmd = 1; + strvec_pushl(&config_set.args, "config", "--add", "--global", "maintenance.repo", + the_repository->worktree ? the_repository->worktree + : the_repository->gitdir, + NULL); + + return run_command(&config_set); +} + +static int maintenance_unregister(void) +{ + struct child_process config_unset = CHILD_PROCESS_INIT; + + if (!the_repository || !the_repository->gitdir) + return error(_("no current repository to unregister")); + + config_unset.git_cmd = 1; + strvec_pushl(&config_unset.args, "config", "--global", "--unset", + "maintenance.repo", + the_repository->worktree ? the_repository->worktree + : the_repository->gitdir, + NULL); + + return run_command(&config_unset); +} + +static const char builtin_maintenance_usage[] = N_("git maintenance []"); int cmd_maintenance(int argc, const char **argv, const char *prefix) { @@ -1423,6 +1472,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) if (!strcmp(argv[1], "run")) return maintenance_run(argc - 1, argv + 1, prefix); + if (!strcmp(argv[1], "register")) + return maintenance_register(); + if (!strcmp(argv[1], "unregister")) + return maintenance_unregister(); die(_("invalid subcommand: %s"), argv[1]); } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 3e0c5f1ca8..b20ee2d542 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0 test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && - test_i18ngrep "usage: git maintenance run" err && + test_i18ngrep "usage: git maintenance " err && test_expect_code 128 git maintenance barf 2>err && test_i18ngrep "invalid subcommand: barf" err ' @@ -294,4 +294,19 @@ test_expect_success '--scheduled with specific time' ' test_cmp_config 1595000100 maintenance.commit-graph.lastrun ' +test_expect_success 'register and unregister' ' + test_when_finished git config --global --unset-all maintenance.repo && + git config --global --add maintenance.repo /existing1 && + git config --global --add maintenance.repo /existing2 && + git config --global --get-all maintenance.repo >before && + git maintenance register && + git config --global --get-all maintenance.repo >actual && + cp before after && + pwd >>after && + test_cmp after actual && + git maintenance unregister && + git config --global --get-all maintenance.repo >actual && + test_cmp before actual +' + test_done From patchwork Tue Aug 25 18:40:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11736223 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CE391109B for ; Tue, 25 Aug 2020 18:40:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A379520782 for ; Tue, 25 Aug 2020 18:40:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LCI8Zgl4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726730AbgHYSkc (ORCPT ); Tue, 25 Aug 2020 14:40:32 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726763AbgHYSkR (ORCPT ); Tue, 25 Aug 2020 14:40:17 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38127C061756 for ; Tue, 25 Aug 2020 11:40:14 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id t2so3630257wma.0 for ; Tue, 25 Aug 2020 11:40:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=7rrg2QEevl+XyyeHkHjnyuPdQSrw3GebymizIEuEhcY=; b=LCI8Zgl4muqJFZiVz9gNMHzfgueCZFkGuY36ayOGgByQtAi+CkgZPb8myeu4iZvVmd /CngHurNdinezdxMFLtZ0VHXP3b8ctlGiMEWGkDTluDl1bMImPk41c/h06o7Y3pSmXMu FWjWyGz6RzCTefyrf0KBJgd352z9BRD1tS2KYSBtIayYlIbFTqtfR5xej+waXXDLCASK LkiaYvMu4EQ+U0N7UDynQGUSDrrVv04Iewd1ixHAhgAV7Ib4rSG2J7s67GAhjE3dkiEP WJZ0XLhV4C2sMA/N/3W+Jl9XVwh9+fPqOJIaKK5bpphiYcYPk5GxeTiO7BiGq3gRpUat dCYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=7rrg2QEevl+XyyeHkHjnyuPdQSrw3GebymizIEuEhcY=; b=nuqWzYF+wtW1lIE9tx7s2v1ECmI8GY+Hd+uTlaUcqDWYVvBtevIZDBIP1/Vm8pyYVg mIoUCmr9xPURrw4Qn1KewuVBlvFaRU4mm6H+VVGfdkutlHh8/52wtWH1unWh7VerFzKL JgE+c53muebmPqntvbh5nCJGSxnof5XCArGhHmAH5cDpUinJWiQP5+pAdH5rMkqlxdRF epY0Il87ediuboKqDWbpeBnGzmXgs/rEhmfgSIUfzvGYKtv0Z6ZBkMESXTgasOK21WlB XVw1mDY7urQGs0ME06inmFL9wxkT1JNrC0q3G1T19bzEvKbA24pdIHo3ebIsRfL9VPLQ rpLA== X-Gm-Message-State: AOAM5305Ali1R2QCwTzGuBQuZaXizgOfa1bKMrcXtMkKVaVfqsLtN6dH RRkogjRmfoC24KZkOYVXjEJD5QyaDos= X-Google-Smtp-Source: ABdhPJxUaqbhlwfXVpn+cgF8DZI+4qXNzpM52R0fXy4GrQP/xr9+5h0hNOvEKbxDj3+qqzOGRJ0lwQ== X-Received: by 2002:a1c:b4c1:: with SMTP id d184mr3441708wmf.26.1598380811536; Tue, 25 Aug 2020 11:40:11 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p14sm36183979wrg.96.2020.08.25.11.40.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:40:10 -0700 (PDT) Message-Id: <8a7c34035ac8a86c82dc2bdeee8a9a76fc5626ed.1598380805.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:40:03 +0000 Subject: [PATCH v2 6/7] maintenance: add start/stop subcommands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Add new subcommands to 'git maintenance' that start or stop background maintenance using 'cron', when available. This integration is as simple as I could make it, barring some implementation complications. For now, the background maintenance is scheduled to run hourly via the following cron table row (ignore line breaks): 0 * * * * $p/git --exec-path=$p for-each-repo --config=maintenance.repo maintenance run --scheduled Future extensions may want to add more complex schedules or some form of logging. For now, hourly runs seem frequent enough to satisfy the needs of tasks like 'prefetch' without being so frequent that users would complain about many no-op commands. Here, "$p" is a placeholder for the path to the current Git executable. This is critical for systems with multiple versions of Git. Specifically, macOS has a system version at '/usr/bin/git' while the version that users can install resides at '/usr/local/bin/git' (symlinked to '/usr/local/libexec/git-core/git'). This will also use your locally-built version if you build and run this in your development environment without installing first. The GIT_TEST_CRONTAB environment variable is not intended for users to edit, but instead as a way to mock the 'crontab [-l]' command. This variable is set in test-lib.sh to avoid a future test from accidentally running anything with the cron integration from modifying the user's schedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our tests to check how the schedule is modified in 'git maintenance (start|stop)' commands. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 11 +++ Makefile | 1 + builtin/gc.c | 117 ++++++++++++++++++++++++++++++ t/helper/test-crontab.c | 35 +++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t7900-maintenance.sh | 30 ++++++++ t/test-lib.sh | 6 ++ 8 files changed, 202 insertions(+) create mode 100644 t/helper/test-crontab.c diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index c42a176a95..d0316db5ae 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -45,6 +45,17 @@ run:: config options are true. By default, only `maintenance.gc.enabled` is true. +start:: + Start running maintenance on the current repository. This performs + the same config updates as the `register` subcommand, then updates + the background scheduler to run `git maintenance run --scheduled` + on an hourly basis. + +stop:: + Halt the background maintenance schedule. The current repository + is not removed from the list of maintained repositories, in case + the background maintenance is restarted later. + unregister:: Remove the current repository from background maintenance. This only removes the repository from the configured list. It does not diff --git a/Makefile b/Makefile index 7c588ff036..c39b39bd7d 100644 --- a/Makefile +++ b/Makefile @@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o TEST_BUILTINS_OBJS += test-bloom.o TEST_BUILTINS_OBJS += test-chmtime.o TEST_BUILTINS_OBJS += test-config.o +TEST_BUILTINS_OBJS += test-crontab.o TEST_BUILTINS_OBJS += test-ctype.o TEST_BUILTINS_OBJS += test-date.o TEST_BUILTINS_OBJS += test-delta.o diff --git a/builtin/gc.c b/builtin/gc.c index 5218d52cb7..d97af4e546 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -32,6 +32,7 @@ #include "remote.h" #include "midx.h" #include "object-store.h" +#include "exec-cmd.h" #define FAILED_RUN "failed to run %s" @@ -1463,6 +1464,118 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +#define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" +#define END_LINE "# END GIT MAINTENANCE SCHEDULE" + +static int update_background_schedule(int run_maintenance) +{ + int result = 0; + int in_old_region = 0; + struct child_process crontab_list = CHILD_PROCESS_INIT; + struct child_process crontab_edit = CHILD_PROCESS_INIT; + FILE *cron_list, *cron_in; + const char *crontab_name; + struct strbuf line = STRBUF_INIT; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + crontab_name = getenv("GIT_TEST_CRONTAB"); + if (!crontab_name) + crontab_name = "crontab"; + + strvec_split(&crontab_list.args, crontab_name); + strvec_push(&crontab_list.args, "-l"); + crontab_list.in = -1; + crontab_list.out = dup(lk.tempfile->fd); + crontab_list.git_cmd = 0; + + if (start_command(&crontab_list)) { + result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); + goto cleanup; + } + + /* Ignore exit code, as an empty crontab will return error. */ + finish_command(&crontab_list); + + /* + * Read from the .lock file, filtering out the old + * schedule while appending the new schedule. + */ + cron_list = fdopen(lk.tempfile->fd, "r"); + rewind(cron_list); + + strvec_split(&crontab_edit.args, crontab_name); + crontab_edit.in = -1; + crontab_edit.git_cmd = 0; + + if (start_command(&crontab_edit)) { + result = error(_("failed to run 'crontab'; your system might not support 'cron'")); + goto cleanup; + } + + cron_in = fdopen(crontab_edit.in, "w"); + if (!cron_in) { + result = error(_("failed to open stdin of 'crontab'")); + goto done_editing; + } + + while (!strbuf_getline_lf(&line, cron_list)) { + if (!in_old_region && !strcmp(line.buf, BEGIN_LINE)) + in_old_region = 1; + if (in_old_region) + continue; + fprintf(cron_in, "%s\n", line.buf); + if (in_old_region && !strcmp(line.buf, END_LINE)) + in_old_region = 0; + } + + if (run_maintenance) { + const char *exec_path = git_exec_path(); + + fprintf(cron_in, "\n%s\n", BEGIN_LINE); + fprintf(cron_in, "# The following schedule was created by Git\n"); + fprintf(cron_in, "# Any edits made in this region might be\n"); + fprintf(cron_in, "# replaced in the future by a Git command.\n\n"); + + fprintf(cron_in, + "0 * * * * \"%s/git\" --exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --scheduled\n", + exec_path, exec_path); + + fprintf(cron_in, "\n%s\n", END_LINE); + } + + fflush(cron_in); + fclose(cron_in); + close(crontab_edit.in); + +done_editing: + if (finish_command(&crontab_edit)) { + result = error(_("'crontab' died")); + goto cleanup; + } + fclose(cron_list); + +cleanup: + rollback_lock_file(&lk); + return result; +} + +static int maintenance_start(void) +{ + if (maintenance_register()) + warning(_("failed to add repo to global config")); + + return update_background_schedule(1); +} + +static int maintenance_stop(void) +{ + return update_background_schedule(0); +} + static const char builtin_maintenance_usage[] = N_("git maintenance []"); int cmd_maintenance(int argc, const char **argv, const char *prefix) @@ -1472,6 +1585,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) if (!strcmp(argv[1], "run")) return maintenance_run(argc - 1, argv + 1, prefix); + if (!strcmp(argv[1], "start")) + return maintenance_start(); + if (!strcmp(argv[1], "stop")) + return maintenance_stop(); if (!strcmp(argv[1], "register")) return maintenance_register(); if (!strcmp(argv[1], "unregister")) diff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c new file mode 100644 index 0000000000..f5db6319c6 --- /dev/null +++ b/t/helper/test-crontab.c @@ -0,0 +1,35 @@ +#include "test-tool.h" +#include "cache.h" + +/* + * Usage: test-tool cron [-l] + * + * If -l is specified, then write the contents of to stdou. + * Otherwise, write from stdin into . + */ +int cmd__crontab(int argc, const char **argv) +{ + char a; + FILE *from, *to; + + if (argc == 3 && !strcmp(argv[2], "-l")) { + from = fopen(argv[1], "r"); + if (!from) + return 0; + to = stdout; + } else if (argc == 2) { + from = stdin; + to = fopen(argv[1], "w"); + } else + return error("unknown arguments"); + + while ((a = fgetc(from)) != EOF) + fputc(a, to); + + if (argc == 3) + fclose(from); + else + fclose(to); + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 590b2efca7..432b49d948 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -18,6 +18,7 @@ static struct test_cmd cmds[] = { { "bloom", cmd__bloom }, { "chmtime", cmd__chmtime }, { "config", cmd__config }, + { "crontab", cmd__crontab }, { "ctype", cmd__ctype }, { "date", cmd__date }, { "delta", cmd__delta }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index ddc8e990e9..7c3281e071 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv); int cmd__bloom(int argc, const char **argv); int cmd__chmtime(int argc, const char **argv); int cmd__config(int argc, const char **argv); +int cmd__crontab(int argc, const char **argv); int cmd__ctype(int argc, const char **argv); int cmd__date(int argc, const char **argv); int cmd__delta(int argc, const char **argv); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index b20ee2d542..6491031be8 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -309,4 +309,34 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' +test_expect_success 'start from empty cron table' ' + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + grep "for-each-repo --config=maintenance.repo maintenance run --scheduled" cron.txt +' + +test_expect_success 'stop from existing schedule' ' + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + # The newline is preserved + echo >empty && + test_cmp empty cron.txt && + + # Operation is idempotent + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + test_cmp empty cron.txt +' + +test_expect_success 'start preserves existing schedule' ' + echo "Important information!" >cron.txt && + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + grep "Important information!" cron.txt +' + test_done diff --git a/t/test-lib.sh b/t/test-lib.sh index ef31f40037..4a60d1ed76 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 ' test_lazy_prereq REBASE_P ' test -z "$GIT_TEST_SKIP_REBASE_P" ' + +# Ensure that no test accidentally triggers a Git command +# that runs 'crontab', affecting a user's cron schedule. +# Tests that verify the cron integration must set this locally +# to avoid errors. +GIT_TEST_CRONTAB="exit 1" From patchwork Tue Aug 25 18:40:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Phillip Wood via GitGitGadget X-Patchwork-Id: 11736225 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 77DD5109B for ; Tue, 25 Aug 2020 18:40:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5BC072074D for ; Tue, 25 Aug 2020 18:40:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Z48YzOCW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726799AbgHYSkb (ORCPT ); Tue, 25 Aug 2020 14:40:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44426 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726759AbgHYSkO (ORCPT ); Tue, 25 Aug 2020 14:40:14 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE3D5C061755 for ; Tue, 25 Aug 2020 11:40:13 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id c19so1685536wmd.1 for ; Tue, 25 Aug 2020 11:40:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=zei/QNl/f24ejb0jKsnwgS7KGDd4U08Mr0i0BWWzX+w=; b=Z48YzOCWW8TVOpz+gslLOu/UZBVZa17hTcW8ktscRu+bI5/cDGRGQmzHQ871KDPqXh JFgAtSaFAORHenesRhOzgxy4OLkm181ISRTQcg679ZQ2XYLLJJR5fIvbfnDjOgbcoIUe HJdP0XgQGrSWkIPcGLx0Zwltm56Z3jZw9u0VoUIZ5igc/s/wM+RUUgkO14XabPMVA+yY p/KRXr6gVDLtK64ARkLq1UWW3hCFzGADhJl8XN7roGEYX8R4URB9Boggl9lU4waPLoqd k9HqfdH0Gj5KzeKJHO+taZHWqGQqZGaZdBtbyYwUk5lYSV5kceTJ1PoOBlvA6WTzjgFf x+tQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=zei/QNl/f24ejb0jKsnwgS7KGDd4U08Mr0i0BWWzX+w=; b=VIv0dFWtOCQLW8YWMcWTf5ucpH7XNGkeCc+gr6s3jgmYfEhkUzLfpWeSpPYcPpKHqy 21n+7LfLxCBXcjuoYVdEIbalc9PRnpUdm6QEcvXvzoZrqRY6WtwZel0pQBWA7AtjJFxN KzbxkOY8TrhMAWq9QVzdyKjcHICNIJKnAlpboQalTl4gnEUj44KRc3ijb0ojKuvP8urx 7zcMLBRPCSmdLJrdNEycVRrH3fmZIXidoHh6V2kiPvDM7A5inR9eoqOpIOULmd0QbkRy AuKMNQMi29r+LdWRSGE1z80nq/LiPTs59Cn0fd7n2vIuiuihL6zdHG50N9z3CW30+vJy U23A== X-Gm-Message-State: AOAM533YaJNL8S78atKmcoIJnR0n/pULknZxiFMnbZxiwl1dVEtA0i4d UM6s6REWqSGE/XloImoavMrbwBRlOMo= X-Google-Smtp-Source: ABdhPJyoWNR73fMpAt4CnyhPElBxx000HZahJMcYBcv7DUmE024YESxnwYHn4H/MapbVCcWT93tKzg== X-Received: by 2002:a1c:2543:: with SMTP id l64mr3060412wml.96.1598380812332; Tue, 25 Aug 2020 11:40:12 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f3sm7395755wmb.35.2020.08.25.11.40.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 11:40:11 -0700 (PDT) Message-Id: <9ecabeb0551653835d4992a8ac984b9995bd92f0.1598380805.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Tue, 25 Aug 2020 18:40:04 +0000 Subject: [PATCH v2 7/7] maintenance: recommended schedule in register/start Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: sandals@crustytoothpaste.net, steadmon@google.com, jrnieder@gmail.com, peff@peff.net, congdanhqx@gmail.com, phillip.wood123@gmail.com, emilyshaffer@google.com, sluongng@gmail.com, jonathantanmy@google.com, Derrick Stolee , Derrick Stolee Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git maintenance (register|start)' subcommands add the current repository to the global Git config so maintenance will operate on that repository. It does not specify what maintenance should occur or how often. If a user sets any 'maintenance..scheduled' config value, then they have chosen a specific schedule for themselves and Git should respect that. However, in an effort to recommend a good schedule for repositories of all sizes, set new config values for recommended tasks that are safe to run in the background while users run foreground Git commands. These commands are generally everything but the 'gc' task. Author's Note: I feel we should do _something_ to recommend a good schedule to users, but I'm not 100% set on this schedule. This is the schedule we use in Scalar and VFS for Git for very large repositories using the GVFS protocol. While the schedule works in that environment, it is possible that "normal" Git repositories could benefit from something more obvious (such as running 'gc' once a day). However, this patch gives us a place to start a conversation on what we should recommend. For my purposes, Scalar will set these config values so we can always differ from core Git's recommendations. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 6 +++++ builtin/gc.c | 44 +++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 5 ++++ 3 files changed, 55 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index d0316db5ae..bba76f0b0d 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -37,6 +37,12 @@ register:: `maintenance..schedule`. The tasks that are enabled are safe for running in the background without disrupting foreground processes. ++ +If your repository has no 'maintenance..schedule' configuration +values set, then Git will set configuration values to some recommended +settings. These settings disable foreground maintenance while performing +maintenance tasks in the background that will not interrupt foreground Git +operations. run:: Run one or more maintenance tasks. If one or more `--task` options diff --git a/builtin/gc.c b/builtin/gc.c index d97af4e546..037402e47f 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1415,6 +1415,47 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) return maintenance_run_tasks(&opts); } +static int has_schedule_config(void) +{ + int i, found = 0; + struct strbuf config_name = STRBUF_INIT; + size_t prefix; + + strbuf_addstr(&config_name, "maintenance."); + prefix = config_name.len; + + for (i = 0; !found && i < TASK__COUNT; i++) { + int value; + + strbuf_setlen(&config_name, prefix); + strbuf_addf(&config_name, "%s.schedule", tasks[i].name); + + if (!git_config_get_int(config_name.buf, &value)) + found = 1; + } + + strbuf_release(&config_name); + return found; +} + +static void set_recommended_schedule(void) +{ + git_config_set("maintenance.auto", "false"); + git_config_set("maintenance.gc.enabled", "false"); + + git_config_set("maintenance.prefetch.enabled", "true"); + git_config_set("maintenance.prefetch.schedule", "3500"); + + git_config_set("maintenance.commit-graph.enabled", "true"); + git_config_set("maintenance.commit-graph.schedule", "3500"); + + git_config_set("maintenance.loose-objects.enabled", "true"); + git_config_set("maintenance.loose-objects.schedule", "86000"); + + git_config_set("maintenance.incremental-repack.enabled", "true"); + git_config_set("maintenance.incremental-repack.schedule", "86000"); +} + static int maintenance_register(void) { struct child_process config_set = CHILD_PROCESS_INIT; @@ -1424,6 +1465,9 @@ static int maintenance_register(void) if (!the_repository || !the_repository->gitdir) return 0; + if (has_schedule_config()) + set_recommended_schedule(); + config_get.git_cmd = 1; strvec_pushl(&config_get.args, "config", "--global", "--get", "maintenance.repo", the_repository->worktree ? the_repository->worktree diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 6491031be8..7417e5858a 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -300,6 +300,11 @@ test_expect_success 'register and unregister' ' git config --global --add maintenance.repo /existing2 && git config --global --get-all maintenance.repo >before && git maintenance register && + test_cmp_config false maintenance.auto && + test_cmp_config false maintenance.gc.enabled && + test_cmp_config true maintenance.prefetch.enabled && + test_cmp_config 3500 maintenance.commit-graph.schedule && + test_cmp_config 86000 maintenance.incremental-repack.schedule && git config --global --get-all maintenance.repo >actual && cp before after && pwd >>after &&