From patchwork Mon Oct 5 12:57:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11816641 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D5020112E for ; Mon, 5 Oct 2020 13:04:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B73D120874 for ; Mon, 5 Oct 2020 13:04:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="NTvPPHmm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726765AbgJENE3 (ORCPT ); Mon, 5 Oct 2020 09:04:29 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726197AbgJENEK (ORCPT ); Mon, 5 Oct 2020 09:04:10 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C5D8DC0613B1 for ; Mon, 5 Oct 2020 05:57:19 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id 13so8512569wmf.0 for ; Mon, 05 Oct 2020 05:57:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=i23re7jsKeggERrBXtJjshBZV8NteQPTfOTlXWmSBic=; b=NTvPPHmmpRxX7aWhgH1sUI2QgK8hzK57FqAW0XKteFf30MgL2oGVSvRG1GUge2j5f/ h3j9QBod64tbp+SzeyhIeJg7HQW1/mqmdsuMuPJM4KpMnyqnGrfMHhUR6LGJwyRA3LSI XO+Dl36LeBCfxu4F1MptXNMvW7YqGCpsyJzj0iHgX4uScErEQWhb9SV6ZnIukEJwE0A/ tj+eWxN5wv220jGxNBNRgFDwSAqohjugPeDAb+FQk7LLAyqlacZE7vdpLJEuy575ZKFk 0LNgnNoU0/wYjivyTLkAi8yQc6Up4+6ODEkFf3rHfSXeXM6jiJArY37ClSdBhkyatxs1 LVAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=i23re7jsKeggERrBXtJjshBZV8NteQPTfOTlXWmSBic=; b=pzH4K+BB2S0PmIVoI+ymEDMDUDJKPOtfa6jgEQUiamt3PhmLIpO7cOd/ABe5HQ2Mkn MpcSRyxfNO12wGUl41pWbEjoSpKnq9PI5J8j2+FPV9FryeVnbUn1iZQOa0Y7iEgdOZrg +3J45dStOTdj1nEPGHREP16JjkJApTufoKzbEu7maSAcxoC6noAqKc255sTGM9g8CtT6 m689+hJh5wnMBNFI07ObNKZBY/rBk86rRmhBqKuqAKIevHgG8/q83pQmGTzQF5T6Ac8H Ix3PeRGQuv2MQJvu4pLtYGd+BTUBb8u6weuZC21tCGJbnVNd9WQM5TsD6Rlbt+s6rew9 zR3A== X-Gm-Message-State: AOAM5303ZcA1YrzwKdJ4K1MdO+I/W1zHMDafrYwDWkwAf4DaIlnQ25W1 vH3lMayNpWq4NI6/0Zk9AmbZy6uCSWA= X-Google-Smtp-Source: ABdhPJzo+XQUkWYvy6FNmPbd8jVQ5LNsu8a+XEkNpyuXKAfhBZ0ErH0r8CC9MfafFxOYc8OaOObnNg== X-Received: by 2002:a1c:dd0b:: with SMTP id u11mr16703577wmg.186.1601902637307; Mon, 05 Oct 2020 05:57:17 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d30sm6081569wrc.19.2020.10.05.05.57.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 05:57:16 -0700 (PDT) Message-Id: <02e7286dba906f6cd729537d1f22e98ba4154c55.1601902635.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 05 Oct 2020 12:57:08 +0000 Subject: [PATCH v3 1/7] maintenance: optionally skip --auto process Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, SZEDER =?utf-8?b?R8OhYm9y?= , Derrick Stolee , =?utf-8?b?xJBvw6BuIFRy4bqnbiBDw7RuZw==?= Danh , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Some commands run 'git maintenance run --auto --[no-]quiet' after doing their normal work, as a way to keep repositories clean as they are used. Currently, users who do not want this maintenance to occur would set the 'gc.auto' config option to 0 to avoid the 'gc' task from running. However, this does not stop the extra process invocation. On Windows, this extra process invocation can be more expensive than necessary. Allow users to drop this extra process by setting 'maintenance.auto' to 'false'. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 5 +++++ run-command.c | 6 ++++++ t/t7900-maintenance.sh | 13 +++++++++++++ 3 files changed, 24 insertions(+) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index a0706d8f09..06db758172 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -1,3 +1,8 @@ +maintenance.auto:: + This boolean config option controls whether some commands run + `git maintenance run --auto` after doing their normal work. Defaults + to true. + maintenance..enabled:: This boolean config option controls whether the maintenance task with name `` is run when no `--task` option is specified to diff --git a/run-command.c b/run-command.c index 2ee59acdc8..ea4d0fb4b1 100644 --- a/run-command.c +++ b/run-command.c @@ -7,6 +7,7 @@ #include "strbuf.h" #include "string-list.h" #include "quote.h" +#include "config.h" void child_process_init(struct child_process *child) { @@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task, int run_auto_maintenance(int quiet) { + int enabled; struct child_process maint = CHILD_PROCESS_INIT; + if (!git_config_get_bool("maintenance.auto", &enabled) && + !enabled) + return 0; + maint.git_cmd = 1; strvec_pushl(&maint.args, "maintenance", "run", "--auto", NULL); strvec_push(&maint.args, quiet ? "--quiet" : "--no-quiet"); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 55116c2f04..c7caaa7a55 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -28,6 +28,19 @@ test_expect_success 'run [--auto|--quiet]' ' test_subcommand git gc --no-quiet .enabled' ' git config maintenance.gc.enabled false && git config maintenance.commit-graph.enabled true && From patchwork Mon Oct 5 12:57:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11816625 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 00EBD112E for ; Mon, 5 Oct 2020 13:04:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CD8C2208A9 for ; Mon, 5 Oct 2020 13:04:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="oan35E6l" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726749AbgJENEL (ORCPT ); Mon, 5 Oct 2020 09:04:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54454 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726248AbgJENEK (ORCPT ); Mon, 5 Oct 2020 09:04:10 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DE9E5C0613B2 for ; Mon, 5 Oct 2020 05:57:19 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id d4so8499522wmd.5 for ; Mon, 05 Oct 2020 05:57:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Sy1fNp0IvDLU7VY072G0Tf6lU7VIhiL8U5UrQvwJNbo=; b=oan35E6lLiPLgytYdz9+UwLx92MVJRbSpPoof5zIuHMrdXY/VKKc6tNiWEzYKq4eKi FSlGmKQYJMwqxF3oeX6Er8Y9xsPCmyeS29odAZSTIfRKPHdlhtKeh1cWKVTgt4dr+4Yj 18cX2BRoo6rctsNDyd/0eDYmL7FHBKqqjyOs1TMBfhPxwlyV/nU240vcrxZpniMDlBJJ 3W14RJ+aD8sytCfzpxYslYIrLbPdWsPim0RKVtWFKFQbj09Q6ZpzjaOymrDYhayoUBgH XG0VA4nDHjLIZn2c6t1ZxfaCxbvGojzrASzExYCxVuXXn6pI5G/qXDfJcn5uY1YzXc3I ueBw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Sy1fNp0IvDLU7VY072G0Tf6lU7VIhiL8U5UrQvwJNbo=; b=klB3V261dxO9mHQeH1VsAsLj9u9AgacZ5Wxr9G0kHGnOkGV98PW1F8hrTyP0haqW/o dKEr1wqjsxFrvOJDimzYo6mojljVbehtg3dWfEx2d51U09iy5hvvf6aQ/AVHdPGI5+bY a4Q3SJy99/AgVB4L5HWZoJG+mb8KPfULbOgOp/xUljWCxIm6yHaGXQonfgXIbr8v3gGt UD0IghQhxrjjU1qSzD9wU7/RdC5Z1hUiFLrOCJtlJSsyGddPFWEdnemGb6d11JbXpr5v BQyq4e9LVzqv5rR93SvIhT+LVhDp1AFS0s8eEhozciM8pKZ8r9amkYkYzGqCuqiIGENG UEHg== X-Gm-Message-State: AOAM5332jMpjbCPVhJ7o7bT9wdC/4FdzQ2pzYjAt7C9BK3zKwIorRkrs S2eKp56fz5qaI0qL06l9+VBylKWJwuM= X-Google-Smtp-Source: ABdhPJyNtLHU7DmRdn6mgIidq45CYHnpCM1S9zcLphAMXk1piS4+XEFyDOXghaNBVxNMlBjRbsxAVQ== X-Received: by 2002:a1c:1d89:: with SMTP id d131mr4331508wmd.119.1601902638182; Mon, 05 Oct 2020 05:57:18 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m11sm12908469wmf.10.2020.10.05.05.57.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 05:57:17 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 05 Oct 2020 12:57:09 +0000 Subject: [PATCH v3 2/7] maintenance: add --schedule option and config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, SZEDER =?utf-8?b?R8OhYm9y?= , Derrick Stolee , =?utf-8?b?xJBvw6BuIFRy4bqnbiBDw7RuZw==?= Danh , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Maintenance currently triggers when certain data-size thresholds are met, such as number of pack-files or loose objects. Users may want to run certain maintenance tasks based on frequency instead. For example, a user may want to perform a 'prefetch' task every hour, or 'gc' task every day. To help these users, update the 'git maintenance run' command to include a '--schedule=' option. The allowed frequencies are 'hourly', 'daily', and 'weekly'. These values are also allowed in a new config value 'maintenance..schedule'. The 'git maintenance run --schedule=' checks the '*.schedule' config value for each enabled task to see if the configured frequency is at least as frequent as the frequency from the '--schedule' argument. We use the following order, for full clarity: 'hourly' > 'daily' > 'weekly' Use new 'enum schedule_priority' to track these values numerically. The following cron table would run the scheduled tasks with the correct frequencies: 0 1-23 * * * git -C maintenance run --schedule=hourly 0 0 * * 1-6 git -C maintenance run --schedule=daily 0 0 * * 0 git -C maintenance run --schedule=weekly This cron schedule will run --schedule=hourly every hour except at midnight. This avoids a concurrent run with the --schedule=daily that runs at midnight every day except the first day of the week. This avoids a concurrent run with the --schedule=weekly that runs at midnight on the first day of the week. Since --schedule=daily also runs the 'hourly' tasks and --schedule=weekly runs the 'hourly' and 'daily' tasks, we will still see all tasks run with the proper frequencies. Signed-off-by: Derrick Stolee --- Documentation/config/maintenance.txt | 5 +++ Documentation/git-maintenance.txt | 13 +++++- builtin/gc.c | 64 ++++++++++++++++++++++++++-- t/t7900-maintenance.sh | 40 +++++++++++++++++ 4 files changed, 118 insertions(+), 4 deletions(-) diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt index 06db758172..70585564fa 100644 --- a/Documentation/config/maintenance.txt +++ b/Documentation/config/maintenance.txt @@ -10,6 +10,11 @@ maintenance..enabled:: `--task` option exists. By default, only `maintenance.gc.enabled` is true. +maintenance..schedule:: + This config option controls whether or not the given `` runs + during a `git maintenance run --schedule=` command. The + value must be one of "hourly", "daily", or "weekly". + maintenance.commit-graph.auto:: This integer config option controls how often the `commit-graph` task should be run as part of `git maintenance run --auto`. If zero, then diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 3f5d8946b4..ed94f66e36 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -110,7 +110,18 @@ OPTIONS only if certain thresholds are met. For example, the `gc` task runs when the number of loose objects exceeds the number stored in the `gc.auto` config setting, or when the number of pack-files - exceeds the `gc.autoPackLimit` config setting. + exceeds the `gc.autoPackLimit` config setting. Not compatible with + the `--schedule` option. + +--schedule:: + When combined with the `run` subcommand, run maintenance tasks + only if certain time conditions are met, as specified by the + `maintenance..schedule` config value for each ``. + This config value specifies a number of seconds since the last + time that task ran, according to the `maintenance..lastRun` + config value. The tasks that are tested are those provided by + the `--task=` option(s) or those with + `maintenance..enabled` set to true. --quiet:: Do not report progress or other information over `stderr`. diff --git a/builtin/gc.c b/builtin/gc.c index 2b99596ec8..03b24ea0db 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -703,14 +703,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix) return 0; } -static const char * const builtin_maintenance_run_usage[] = { - N_("git maintenance run [--auto] [--[no-]quiet] [--task=]"), +static const char *const builtin_maintenance_run_usage[] = { + N_("git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]"), NULL }; +enum schedule_priority { + SCHEDULE_NONE = 0, + SCHEDULE_WEEKLY = 1, + SCHEDULE_DAILY = 2, + SCHEDULE_HOURLY = 3, +}; + +static enum schedule_priority parse_schedule(const char *value) +{ + if (!value) + return SCHEDULE_NONE; + if (!strcasecmp(value, "hourly")) + return SCHEDULE_HOURLY; + if (!strcasecmp(value, "daily")) + return SCHEDULE_DAILY; + if (!strcasecmp(value, "weekly")) + return SCHEDULE_WEEKLY; + return SCHEDULE_NONE; +} + +static int maintenance_opt_schedule(const struct option *opt, const char *arg, + int unset) +{ + enum schedule_priority *priority = opt->value; + + if (unset) + die(_("--no-schedule is not allowed")); + + *priority = parse_schedule(arg); + + if (!*priority) + die(_("unrecognized --schedule argument '%s'"), arg); + + return 0; +} + struct maintenance_run_opts { int auto_flag; int quiet; + enum schedule_priority schedule; }; /* Remember to update object flag allocation in object.h */ @@ -1158,6 +1195,8 @@ struct maintenance_task { maintenance_auto_fn *auto_condition; unsigned enabled:1; + enum schedule_priority schedule; + /* -1 if not selected. */ int selected_order; }; @@ -1253,6 +1292,9 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) !tasks[i].auto_condition())) continue; + if (opts->schedule && tasks[i].schedule < opts->schedule) + continue; + trace2_region_enter("maintenance", tasks[i].name, r); if (tasks[i].fn(opts)) { error(_("task '%s' failed"), tasks[i].name); @@ -1273,13 +1315,23 @@ static void initialize_task_config(void) for (i = 0; i < TASK__COUNT; i++) { int config_value; + char *config_str; - strbuf_setlen(&config_name, 0); + strbuf_reset(&config_name); strbuf_addf(&config_name, "maintenance.%s.enabled", tasks[i].name); if (!git_config_get_bool(config_name.buf, &config_value)) tasks[i].enabled = config_value; + + strbuf_reset(&config_name); + strbuf_addf(&config_name, "maintenance.%s.schedule", + tasks[i].name); + + if (!git_config_get_string(config_name.buf, &config_str)) { + tasks[i].schedule = parse_schedule(config_str); + free(config_str); + } } strbuf_release(&config_name); @@ -1323,6 +1375,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) struct option builtin_maintenance_run_options[] = { OPT_BOOL(0, "auto", &opts.auto_flag, N_("run tasks based on the state of the repository")), + OPT_CALLBACK(0, "schedule", &opts.schedule, N_("frequency"), + N_("run tasks based on frequency"), + maintenance_opt_schedule), OPT_BOOL(0, "quiet", &opts.quiet, N_("do not report progress or other information over stderr")), OPT_CALLBACK_F(0, "task", NULL, N_("task"), @@ -1343,6 +1398,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) builtin_maintenance_run_usage, PARSE_OPT_STOP_AT_NON_OPTION); + if (opts.auto_flag && opts.schedule) + die(_("use at most one of --auto and --schedule=")); + if (argc != 0) usage_with_options(builtin_maintenance_run_usage, builtin_maintenance_run_options); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index c7caaa7a55..33d73cd01c 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -260,4 +260,44 @@ test_expect_success 'maintenance.incremental-repack.auto' ' test_subcommand git multi-pack-index write --no-progress err && + test_i18ngrep "at most one" err +' + +test_expect_success 'invalid --schedule value' ' + test_must_fail git maintenance run --schedule=annually 2>err && + test_i18ngrep "unrecognized --schedule" err +' + +test_expect_success '--schedule inheritance weekly -> daily -> hourly' ' + git config maintenance.loose-objects.enabled true && + git config maintenance.loose-objects.schedule hourly && + git config maintenance.commit-graph.enabled true && + git config maintenance.commit-graph.schedule daily && + git config maintenance.incremental-repack.enabled true && + git config maintenance.incremental-repack.schedule weekly && + + GIT_TRACE2_EVENT="$(pwd)/hourly.txt" \ + git maintenance run --schedule=hourly 2>/dev/null && + test_subcommand git prune-packed --quiet /dev/null && + test_subcommand git prune-packed --quiet /dev/null && + test_subcommand git prune-packed --quiet X-Patchwork-Id: 11816623 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C23A2618 for ; Mon, 5 Oct 2020 13:04:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 942A920874 for ; Mon, 5 Oct 2020 13:04:12 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="aQntGy2t" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726742AbgJENEL (ORCPT ); Mon, 5 Oct 2020 09:04:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54448 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726073AbgJENEK (ORCPT ); Mon, 5 Oct 2020 09:04:10 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 97639C0613B3 for ; Mon, 5 Oct 2020 05:57:20 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id p15so4035159wmi.4 for ; Mon, 05 Oct 2020 05:57:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=LJaF+KBq6DinuUhew6phz3UG9LLuWbKh+8DAyadGgBc=; b=aQntGy2tMrEy3oBONC3gGiu/wsfA6GHRHlq+pgL0YmPLgO98myJtoo45d4AT/vpL4V LByNEFHYz2rRN5gVyuf3WrGkxsK+l0pB6te1hRjeFTsSNIvmRcfKfjPcuCRChJQdTuFB 5hnbCIRlXwRIjiIvi3C/2p12ffiS4WnY6ep+7uX8NDK9Mu1BQAkuJaD6DCYsxW26MR5y JmjGbPIfUJuixOFbCugLhD2YH5dsCtim+rULy3Nq5cXCqLA3JT8c/r8RexAfZ94lqg20 MdEodxpOrbkK0JO4QkqyiBIItGOa2dj7MGDx3cDa5hbsLJPeU+gYHr9gmbzrN7XYOT6R cSdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=LJaF+KBq6DinuUhew6phz3UG9LLuWbKh+8DAyadGgBc=; b=M9Cfe4BbCZjWBwn1vv3TAYhPEIZMlmxvRC5vGrDYPGYuSZ6W38+RmvTvdKnJhIt2b3 /O3PynpPNqUTTG1zR+s4oQ7jWlRhyH00an586ImUQPoEhGa0L3z0Nem0fNKDcGE1qUCO +V173xECXGMD/6rem/2UNaIlOkQu9vllkPN9xDNTFBSJEDT3Qi1Iw/RPxUrfCTcu6xUg UbUp7EkC+X1N0rSRx6YbHDuZslJS97G91ws/oFTgHfwnZlavGpRKaxS8QdiqsyQ0vacG vVfMnYw199Ez1cQDs4fY2LIWfxnH5Mxjjf7oLFYgEMqcuQeWVa4thilGaJ3BFQBJ/8QY lGEA== X-Gm-Message-State: AOAM5313qtjcjSsvKClO4MeWlvXBm9/uh+kwVs2upeNQv2ek0ChufV5k CXCq2s90VRdqd2ew9dOkA24i1kq4BmM= X-Google-Smtp-Source: ABdhPJwVXkQzBPPbDxMeO+24HMMmhXGSsoyrKzrh0VwU6g9KcguR+BhMbzDe//SOLYpRf6YXfe51cg== X-Received: by 2002:a1c:a7cc:: with SMTP id q195mr7047892wme.8.1601902639028; Mon, 05 Oct 2020 05:57:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l18sm13126367wrp.84.2020.10.05.05.57.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 05:57:18 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 05 Oct 2020 12:57:10 +0000 Subject: [PATCH v3 3/7] for-each-repo: run subcommands on configured repos Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, SZEDER =?utf-8?b?R8OhYm9y?= , Derrick Stolee , =?utf-8?b?xJBvw6BuIFRy4bqnbiBDw7RuZw==?= Danh , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee It can be helpful to store a list of repositories in global or system config and then iterate Git commands on that list. Create a new builtin that makes this process simple for experts. We will use this builtin to run scheduled maintenance on all configured repositories in a future change. The test is very simple, but does highlight that the "--" argument is optional. Signed-off-by: Derrick Stolee --- .gitignore | 1 + Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++ Makefile | 1 + builtin.h | 1 + builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++ command-list.txt | 1 + git.c | 1 + t/t0068-for-each-repo.sh | 30 +++++++++++++++ 8 files changed, 152 insertions(+) create mode 100644 Documentation/git-for-each-repo.txt create mode 100644 builtin/for-each-repo.c create mode 100755 t/t0068-for-each-repo.sh diff --git a/.gitignore b/.gitignore index a5808fa30d..5eb2a2be71 100644 --- a/.gitignore +++ b/.gitignore @@ -67,6 +67,7 @@ /git-filter-branch /git-fmt-merge-msg /git-for-each-ref +/git-for-each-repo /git-format-patch /git-fsck /git-fsck-objects diff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt new file mode 100644 index 0000000000..94bd19da26 --- /dev/null +++ b/Documentation/git-for-each-repo.txt @@ -0,0 +1,59 @@ +git-for-each-repo(1) +==================== + +NAME +---- +git-for-each-repo - Run a Git command on a list of repositories + + +SYNOPSIS +-------- +[verse] +'git for-each-repo' --config= [--] + + +DESCRIPTION +----------- +Run a Git command on a list of repositories. The arguments after the +known options or `--` indicator are used as the arguments for the Git +subprocess. + +THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE. + +For example, we could run maintenance on each of a list of repositories +stored in a `maintenance.repo` config variable using + +------------- +git for-each-repo --config=maintenance.repo maintenance run +------------- + +This will run `git -C maintenance run` for each value `` +in the multi-valued config variable `maintenance.repo`. + + +OPTIONS +------- +--config=:: + Use the given config variable as a multi-valued list storing + absolute path names. Iterate on that list of paths to run + the given arguments. ++ +These config values are loaded from system, global, and local Git config, +as available. If `git for-each-repo` is run in a directory that is not a +Git repository, then only the system and global config is used. + + +SUBPROCESS BEHAVIOR +------------------- + +If any `git -C ` subprocess returns a non-zero exit code, +then the `git for-each-repo` process returns that exit code without running +more subprocesses. + +Each `git -C ` subprocess inherits the standard file +descriptors `stdin`, `stdout`, and `stderr`. + + +GIT +--- +Part of the linkgit:git[1] suite diff --git a/Makefile b/Makefile index 65f8cfb236..7c588ff036 100644 --- a/Makefile +++ b/Makefile @@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o BUILTIN_OBJS += builtin/fetch.o BUILTIN_OBJS += builtin/fmt-merge-msg.o BUILTIN_OBJS += builtin/for-each-ref.o +BUILTIN_OBJS += builtin/for-each-repo.o BUILTIN_OBJS += builtin/fsck.o BUILTIN_OBJS += builtin/gc.o BUILTIN_OBJS += builtin/get-tar-commit-id.o diff --git a/builtin.h b/builtin.h index 17c1c0ce49..ff7c6e5aa9 100644 --- a/builtin.h +++ b/builtin.h @@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix); int cmd_fetch_pack(int argc, const char **argv, const char *prefix); int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix); int cmd_for_each_ref(int argc, const char **argv, const char *prefix); +int cmd_for_each_repo(int argc, const char **argv, const char *prefix); int cmd_format_patch(int argc, const char **argv, const char *prefix); int cmd_fsck(int argc, const char **argv, const char *prefix); int cmd_gc(int argc, const char **argv, const char *prefix); diff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c new file mode 100644 index 0000000000..5bba623ff1 --- /dev/null +++ b/builtin/for-each-repo.c @@ -0,0 +1,58 @@ +#include "cache.h" +#include "config.h" +#include "builtin.h" +#include "parse-options.h" +#include "run-command.h" +#include "string-list.h" + +static const char * const for_each_repo_usage[] = { + N_("git for-each-repo --config= "), + NULL +}; + +static int run_command_on_repo(const char *path, + void *cbdata) +{ + int i; + struct child_process child = CHILD_PROCESS_INIT; + struct strvec *args = (struct strvec *)cbdata; + + child.git_cmd = 1; + strvec_pushl(&child.args, "-C", path, NULL); + + for (i = 0; i < args->nr; i++) + strvec_push(&child.args, args->v[i]); + + return run_command(&child); +} + +int cmd_for_each_repo(int argc, const char **argv, const char *prefix) +{ + static const char *config_key = NULL; + int i, result = 0; + const struct string_list *values; + struct strvec args = STRVEC_INIT; + + const struct option options[] = { + OPT_STRING(0, "config", &config_key, N_("config"), + N_("config key storing a list of repository paths")), + OPT_END() + }; + + argc = parse_options(argc, argv, prefix, options, for_each_repo_usage, + PARSE_OPT_STOP_AT_NON_OPTION); + + if (!config_key) + die(_("missing --config=")); + + for (i = 0; i < argc; i++) + strvec_push(&args, argv[i]); + + values = repo_config_get_value_multi(the_repository, + config_key); + + for (i = 0; !result && i < values->nr; i++) + result = run_command_on_repo(values->items[i].string, &args); + + return result; +} diff --git a/command-list.txt b/command-list.txt index 0e3204e7d1..581499be82 100644 --- a/command-list.txt +++ b/command-list.txt @@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories git-filter-branch ancillarymanipulators git-fmt-merge-msg purehelpers git-for-each-ref plumbinginterrogators +git-for-each-repo plumbinginterrogators git-format-patch mainporcelain git-fsck ancillaryinterrogators complete git-gc mainporcelain diff --git a/git.c b/git.c index 24f250d29a..1cab64b5d1 100644 --- a/git.c +++ b/git.c @@ -511,6 +511,7 @@ static struct cmd_struct commands[] = { { "fetch-pack", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT }, { "fmt-merge-msg", cmd_fmt_merge_msg, RUN_SETUP }, { "for-each-ref", cmd_for_each_ref, RUN_SETUP }, + { "for-each-repo", cmd_for_each_repo, RUN_SETUP_GENTLY }, { "format-patch", cmd_format_patch, RUN_SETUP }, { "fsck", cmd_fsck, RUN_SETUP }, { "fsck-objects", cmd_fsck, RUN_SETUP }, diff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh new file mode 100755 index 0000000000..136b4ec839 --- /dev/null +++ b/t/t0068-for-each-repo.sh @@ -0,0 +1,30 @@ +#!/bin/sh + +test_description='git for-each-repo builtin' + +. ./test-lib.sh + +test_expect_success 'run based on configured value' ' + git init one && + git init two && + git init three && + git -C two commit --allow-empty -m "DID NOT RUN" && + git config run.key "$TRASH_DIRECTORY/one" && + git config --add run.key "$TRASH_DIRECTORY/three" && + git for-each-repo --config=run.key commit --allow-empty -m "ran" && + git -C one log -1 --pretty=format:%s >message && + grep ran message && + git -C two log -1 --pretty=format:%s >message && + ! grep ran message && + git -C three log -1 --pretty=format:%s >message && + grep ran message && + git for-each-repo --config=run.key -- commit --allow-empty -m "ran again" && + git -C one log -1 --pretty=format:%s >message && + grep again message && + git -C two log -1 --pretty=format:%s >message && + ! grep again message && + git -C three log -1 --pretty=format:%s >message && + grep again message +' + +test_done From patchwork Mon Oct 5 12:57:11 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11816631 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C9EC13B2 for ; Mon, 5 Oct 2020 13:04:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB377207BC for ; Mon, 5 Oct 2020 13:04:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bhPSBzhv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726444AbgJENE0 (ORCPT ); Mon, 5 Oct 2020 09:04:26 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54452 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726730AbgJENEK (ORCPT ); Mon, 5 Oct 2020 09:04:10 -0400 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6245DC0613B4 for ; Mon, 5 Oct 2020 05:57:21 -0700 (PDT) Received: by mail-wr1-x442.google.com with SMTP id j2so9461324wrx.7 for ; Mon, 05 Oct 2020 05:57:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=YZRvSJQSR01ZgZPGmy5Oy6QQ1M778dWiXzW4kSk/4UU=; b=bhPSBzhvCGd2Ho+hyBSyItI1GeCXwkq4wVEHZd5vK4hKL+9b4IvtBViKffs0Cr/v+b o9Aa8nGpMe0GbCbWmxT56SOPxuOrl0vrY82B26vgPN9VzrxMNvVDKZQU/Zq9SbOTfj51 Wd/OPRY7su+Iiw/KgqeOB+fxPHB/bc4EIcaW07CwgJzIvv62BwU3eYaCK/dQQIsor1CB h+tcAw6fqJ8kb8V4AEt65UdLn9bpKVoU2F4JcR42UMRXQGgBAZJAs3A4Z2UjFZgq7KSy aumMg8le7wuictpJpZyUcKClslw8a0Iy5jepy5SCda+AfwaSIKS7Vw4SuD42biX0N/rs WS+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=YZRvSJQSR01ZgZPGmy5Oy6QQ1M778dWiXzW4kSk/4UU=; b=iS9Aw95qtKtQ2ySy337PjcNbGiCzzhIlsW5lii2jMKolvbeDTDl0/quqBYJIkMjohV xpZ3n+A1i/5H0CE0OWo54y6PuYymZSH96uE9OpgSHqeacU7g/mQdmozU7ERg4rUpAOwi qgsCCX+d6EL9pxjoEVC4XXa0NzL/nD7GlhBaW/O52MkWk2x/Aw+C7CY6FbVvHKaEm1Er 8vRyjLc9MaVf3Ya5Weyk5aXyqTQ15hZx5Tiolls5ZQmqcQbX8zqwcAvi/mzyqJWpQ7Q2 fD8zZ3LdohWD0nIcl/wGavWpQt1R+OywAAjvp0lEVGbtWWDjg1EdJREeFvk3hiTj2hQ0 GKeA== X-Gm-Message-State: AOAM530po1dwRLb+0UvmyOWpNET745A3pPFUdQiH2AJ9x56ukkHG3gMP aSXgXY6SStfiMrhcTdUWKbPA/vb8jPg= X-Google-Smtp-Source: ABdhPJzHoxAu2QwDjJRzXHI5xeJqVcgXduEfh5rpEIe+E+CIKqRLZmSbbYTHMgwn44YQ7fc5qw/ZOQ== X-Received: by 2002:a5d:61c7:: with SMTP id q7mr17815824wrv.343.1601902639869; Mon, 05 Oct 2020 05:57:19 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x10sm12923547wmi.37.2020.10.05.05.57.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 05:57:19 -0700 (PDT) Message-Id: <922b984c8aa4217a2f60307513dfa43f16475a00.1601902635.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 05 Oct 2020 12:57:11 +0000 Subject: [PATCH v3 4/7] maintenance: add [un]register subcommands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, SZEDER =?utf-8?b?R8OhYm9y?= , Derrick Stolee , =?utf-8?b?xJBvw6BuIFRy4bqnbiBDw7RuZw==?= Danh , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee In preparation for launching background maintenance from the 'git maintenance' builtin, create register/unregister subcommands. These commands update the new 'maintenance.repos' config option in the global config so the background maintenance job knows which repositories to maintain. These commands allow users to add a repository to the background maintenance list without disrupting the actual maintenance mechanism. For example, a user can run 'git maintenance register' when no background maintenance is running and it will not start the background maintenance. A later update to start running background maintenance will then pick up this repository automatically. The opposite example is that a user can run 'git maintenance unregister' to remove the current repository from background maintenance without halting maintenance for other repositories. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 14 ++++++++ builtin/gc.c | 55 ++++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 17 +++++++++- 3 files changed, 84 insertions(+), 2 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index ed94f66e36..1c59fd0cb5 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -29,6 +29,15 @@ Git repository. SUBCOMMANDS ----------- +register:: + Initialize Git config values so any scheduled maintenance will + start running on this repository. This adds the repository to the + `maintenance.repo` config variable in the current user's global + config and enables some recommended configuration values for + `maintenance..schedule`. The tasks that are enabled are safe + for running in the background without disrupting foreground + processes. + run:: Run one or more maintenance tasks. If one or more `--task` options are specified, then those tasks are run in that order. Otherwise, @@ -36,6 +45,11 @@ run:: config options are true. By default, only `maintenance.gc.enabled` is true. +unregister:: + Remove the current repository from background maintenance. This + only removes the repository from the configured list. It does not + stop the background maintenance processes from running. + TASKS ----- diff --git a/builtin/gc.c b/builtin/gc.c index 03b24ea0db..edf1d35ce5 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1407,7 +1407,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix) return maintenance_run_tasks(&opts); } -static const char builtin_maintenance_usage[] = N_("git maintenance run []"); +static int maintenance_register(void) +{ + struct child_process config_set = CHILD_PROCESS_INIT; + struct child_process config_get = CHILD_PROCESS_INIT; + + /* There is no current repository, so skip registering it */ + if (!the_repository || !the_repository->gitdir) + return 0; + + config_get.git_cmd = 1; + strvec_pushl(&config_get.args, "config", "--global", "--get", "maintenance.repo", + the_repository->worktree ? the_repository->worktree + : the_repository->gitdir, + NULL); + config_get.out = -1; + + if (start_command(&config_get)) + return error(_("failed to run 'git config'")); + + /* We already have this value in our config! */ + if (!finish_command(&config_get)) + return 0; + + config_set.git_cmd = 1; + strvec_pushl(&config_set.args, "config", "--add", "--global", "maintenance.repo", + the_repository->worktree ? the_repository->worktree + : the_repository->gitdir, + NULL); + + return run_command(&config_set); +} + +static int maintenance_unregister(void) +{ + struct child_process config_unset = CHILD_PROCESS_INIT; + + if (!the_repository || !the_repository->gitdir) + return error(_("no current repository to unregister")); + + config_unset.git_cmd = 1; + strvec_pushl(&config_unset.args, "config", "--global", "--unset", + "maintenance.repo", + the_repository->worktree ? the_repository->worktree + : the_repository->gitdir, + NULL); + + return run_command(&config_unset); +} + +static const char builtin_maintenance_usage[] = N_("git maintenance []"); int cmd_maintenance(int argc, const char **argv, const char *prefix) { @@ -1417,6 +1466,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) if (!strcmp(argv[1], "run")) return maintenance_run(argc - 1, argv + 1, prefix); + if (!strcmp(argv[1], "register")) + return maintenance_register(); + if (!strcmp(argv[1], "unregister")) + return maintenance_unregister(); die(_("invalid subcommand: %s"), argv[1]); } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 33d73cd01c..8f383d01d9 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0 test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && - test_i18ngrep "usage: git maintenance run" err && + test_i18ngrep "usage: git maintenance " err && test_expect_code 128 git maintenance barf 2>err && test_i18ngrep "invalid subcommand: barf" err && test_expect_code 129 git maintenance 2>err && @@ -300,4 +300,19 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' ' test_subcommand git multi-pack-index write --no-progress before && + git maintenance register && + git config --global --get-all maintenance.repo >actual && + cp before after && + pwd >>after && + test_cmp after actual && + git maintenance unregister && + git config --global --get-all maintenance.repo >actual && + test_cmp before actual +' + test_done From patchwork Mon Oct 5 12:57:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11816629 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9A01C618 for ; Mon, 5 Oct 2020 13:04:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 689BC207BC for ; Mon, 5 Oct 2020 13:04:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eP9LyA3f" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726747AbgJENER (ORCPT ); Mon, 5 Oct 2020 09:04:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726731AbgJENEK (ORCPT ); Mon, 5 Oct 2020 09:04:10 -0400 Received: from mail-wm1-x344.google.com (mail-wm1-x344.google.com [IPv6:2a00:1450:4864:20::344]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 79385C0613B5 for ; Mon, 5 Oct 2020 05:57:22 -0700 (PDT) Received: by mail-wm1-x344.google.com with SMTP id l15so5838648wmh.1 for ; Mon, 05 Oct 2020 05:57:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=fa90TYFI5LjZNfWJEszlEKRKMQdHQSZaM6H+f7lq+KY=; b=eP9LyA3f5gVw1KS2pnPwy67Yk4CHR+yKMEkR6wcnEeCucGrjiGR13UKIhBxdR+uBit Kv9/aI/vTmV2mXdmX5WK8gEBBCcxLkFYV7betbten5QtLD2cBmXI2zKLSCjcAw35WF/5 7Fo88eNw/j77MDNzatK1D617EdusagTgjK3Jh9AS3a+x5odARW+ggogzmKo1LQO0eo4B 9QNbklKGoUl0Dv3ouDX7VzGSs6LxmYxjRkR5Fmi/IatD9g0MXufgID5ikS4dARdHpeqq PY2EpJHK08xZb6gWSwGx28J9Gf2eLXxsyHE+k6hL7P1vSt7QKdJmDtEb/OX17LgYCYiE 3Sqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=fa90TYFI5LjZNfWJEszlEKRKMQdHQSZaM6H+f7lq+KY=; b=Rml4grR/i0h1JBvbl9qqgHJPsvkOnGf0GueKeF06lOeC2xfZFPdSY/p2GR7yXnChTM m4UmBALdWTJYRhcuv2xPcXlpP2N9Z2EzRsGay93CWiX7o7Jb38uO1jT6a9E0Km3qhTue fH5j30Nd1ddt7d6ke8cyv/srX5C0e2+iBf+dP0iW60I/X8CyMOqLyKihTDCObTWD+zMF pndkZo5T9uzuGGp+HFCeLicyLm6If085pyGpyXhNknMIrkAsgVTk7/WE1Pjxd0QwTpga fMTp4JcWcZQsdcCThaaiOmTXzifvHUtnDkLsKP5Tv0ldrzbdiaa0Qgbt3uYWpx/0oODM m+RQ== X-Gm-Message-State: AOAM533OY9v4mYi5vYLMUXiAv/rO6JKFQhuugFy7AzcLuEmtcedbyYQ8 7OFhFgMgL2qyJlqjHnvF8U331QZwqEw= X-Google-Smtp-Source: ABdhPJwjv2+6kCKPcNL9nHGvX8e+U00HylPTXUWssUKPlvvGETcW0nrzXOAaXfedNqrESc4IZvThug== X-Received: by 2002:a1c:9a0c:: with SMTP id c12mr17313088wme.85.1601902640663; Mon, 05 Oct 2020 05:57:20 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z11sm9327168wrh.70.2020.10.05.05.57.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 05:57:20 -0700 (PDT) Message-Id: <5194f6b1facbd14cc17eea0337c0cc397a2a51fc.1601902635.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 05 Oct 2020 12:57:12 +0000 Subject: [PATCH v3 5/7] maintenance: add start/stop subcommands Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, SZEDER =?utf-8?b?R8OhYm9y?= , Derrick Stolee , =?utf-8?b?xJBvw6BuIFRy4bqnbiBDw7RuZw==?= Danh , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee Add new subcommands to 'git maintenance' that start or stop background maintenance using 'cron', when available. This integration is as simple as I could make it, barring some implementation complications. The schedule is laid out as follows: 0 1-23 * * * $cmd maintenance run --schedule=hourly 0 0 * * 1-6 $cmd maintenance run --schedule=daily 0 0 * * 0 $cmd maintenance run --schedule=weekly where $cmd is a properly-qualified 'git for-each-repo' execution: $cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo where $path points to the location of the Git executable running 'git maintenance start'. This is critical for systems with multiple versions of Git. Specifically, macOS has a system version at '/usr/bin/git' while the version that users can install resides at '/usr/local/bin/git' (symlinked to '/usr/local/libexec/git-core/git'). This will also use your locally-built version if you build and run this in your development environment without installing first. This conditional schedule avoids having cron launch multiple 'git for-each-repo' commands in parallel. Such parallel commands would likely lead to the 'hourly' and 'daily' tasks competing over the object database lock. This could lead to to some tasks never being run! Since the --schedule= argument will run all tasks with _at least_ the given frequency, the daily runs will also run the hourly tasks. Similarly, the weekly runs will also run the daily and hourly tasks. The GIT_TEST_CRONTAB environment variable is not intended for users to edit, but instead as a way to mock the 'crontab [-l]' command. This variable is set in test-lib.sh to avoid a future test from accidentally running anything with the cron integration from modifying the user's schedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our tests to check how the schedule is modified in 'git maintenance (start|stop)' commands. Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 11 +++ Makefile | 1 + builtin/gc.c | 124 ++++++++++++++++++++++++++++++ t/helper/test-crontab.c | 35 +++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t7900-maintenance.sh | 28 +++++++ t/test-lib.sh | 6 ++ 8 files changed, 207 insertions(+) create mode 100644 t/helper/test-crontab.c diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 1c59fd0cb5..7628a6d157 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -45,6 +45,17 @@ run:: config options are true. By default, only `maintenance.gc.enabled` is true. +start:: + Start running maintenance on the current repository. This performs + the same config updates as the `register` subcommand, then updates + the background scheduler to run `git maintenance run --scheduled` + on an hourly basis. + +stop:: + Halt the background maintenance schedule. The current repository + is not removed from the list of maintained repositories, in case + the background maintenance is restarted later. + unregister:: Remove the current repository from background maintenance. This only removes the repository from the configured list. It does not diff --git a/Makefile b/Makefile index 7c588ff036..c39b39bd7d 100644 --- a/Makefile +++ b/Makefile @@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o TEST_BUILTINS_OBJS += test-bloom.o TEST_BUILTINS_OBJS += test-chmtime.o TEST_BUILTINS_OBJS += test-config.o +TEST_BUILTINS_OBJS += test-crontab.o TEST_BUILTINS_OBJS += test-ctype.o TEST_BUILTINS_OBJS += test-date.o TEST_BUILTINS_OBJS += test-delta.o diff --git a/builtin/gc.c b/builtin/gc.c index edf1d35ce5..a387f46585 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -31,6 +31,7 @@ #include "refs.h" #include "remote.h" #include "object-store.h" +#include "exec-cmd.h" #define FAILED_RUN "failed to run %s" @@ -1456,6 +1457,125 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +#define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" +#define END_LINE "# END GIT MAINTENANCE SCHEDULE" + +static int update_background_schedule(int run_maintenance) +{ + int result = 0; + int in_old_region = 0; + struct child_process crontab_list = CHILD_PROCESS_INIT; + struct child_process crontab_edit = CHILD_PROCESS_INIT; + FILE *cron_list, *cron_in; + const char *crontab_name; + struct strbuf line = STRBUF_INIT; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + crontab_name = getenv("GIT_TEST_CRONTAB"); + if (!crontab_name) + crontab_name = "crontab"; + + strvec_split(&crontab_list.args, crontab_name); + strvec_push(&crontab_list.args, "-l"); + crontab_list.in = -1; + crontab_list.out = dup(lk.tempfile->fd); + crontab_list.git_cmd = 0; + + if (start_command(&crontab_list)) { + result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); + goto cleanup; + } + + /* Ignore exit code, as an empty crontab will return error. */ + finish_command(&crontab_list); + + /* + * Read from the .lock file, filtering out the old + * schedule while appending the new schedule. + */ + cron_list = fdopen(lk.tempfile->fd, "r"); + rewind(cron_list); + + strvec_split(&crontab_edit.args, crontab_name); + crontab_edit.in = -1; + crontab_edit.git_cmd = 0; + + if (start_command(&crontab_edit)) { + result = error(_("failed to run 'crontab'; your system might not support 'cron'")); + goto cleanup; + } + + cron_in = fdopen(crontab_edit.in, "w"); + if (!cron_in) { + result = error(_("failed to open stdin of 'crontab'")); + goto done_editing; + } + + while (!strbuf_getline_lf(&line, cron_list)) { + if (!in_old_region && !strcmp(line.buf, BEGIN_LINE)) + in_old_region = 1; + if (in_old_region) + continue; + fprintf(cron_in, "%s\n", line.buf); + if (in_old_region && !strcmp(line.buf, END_LINE)) + in_old_region = 0; + } + + if (run_maintenance) { + struct strbuf line_format = STRBUF_INIT; + const char *exec_path = git_exec_path(); + + fprintf(cron_in, "%s\n", BEGIN_LINE); + fprintf(cron_in, + "# The following schedule was created by Git\n"); + fprintf(cron_in, "# Any edits made in this region might be\n"); + fprintf(cron_in, + "# replaced in the future by a Git command.\n\n"); + + strbuf_addf(&line_format, + "%%s %%s * * %%s \"%s/git\" --exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%%s\n", + exec_path, exec_path); + fprintf(cron_in, line_format.buf, "0", "1-23", "*", "hourly"); + fprintf(cron_in, line_format.buf, "0", "0", "1-6", "daily"); + fprintf(cron_in, line_format.buf, "0", "0", "0", "weekly"); + strbuf_release(&line_format); + + fprintf(cron_in, "\n%s\n", END_LINE); + } + + fflush(cron_in); + fclose(cron_in); + close(crontab_edit.in); + +done_editing: + if (finish_command(&crontab_edit)) { + result = error(_("'crontab' died")); + goto cleanup; + } + fclose(cron_list); + +cleanup: + rollback_lock_file(&lk); + return result; +} + +static int maintenance_start(void) +{ + if (maintenance_register()) + warning(_("failed to add repo to global config")); + + return update_background_schedule(1); +} + +static int maintenance_stop(void) +{ + return update_background_schedule(0); +} + static const char builtin_maintenance_usage[] = N_("git maintenance []"); int cmd_maintenance(int argc, const char **argv, const char *prefix) @@ -1466,6 +1586,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix) if (!strcmp(argv[1], "run")) return maintenance_run(argc - 1, argv + 1, prefix); + if (!strcmp(argv[1], "start")) + return maintenance_start(); + if (!strcmp(argv[1], "stop")) + return maintenance_stop(); if (!strcmp(argv[1], "register")) return maintenance_register(); if (!strcmp(argv[1], "unregister")) diff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c new file mode 100644 index 0000000000..e7c0137a47 --- /dev/null +++ b/t/helper/test-crontab.c @@ -0,0 +1,35 @@ +#include "test-tool.h" +#include "cache.h" + +/* + * Usage: test-tool cron [-l] + * + * If -l is specified, then write the contents of to stdout. + * Otherwise, write from stdin into . + */ +int cmd__crontab(int argc, const char **argv) +{ + int a; + FILE *from, *to; + + if (argc == 3 && !strcmp(argv[2], "-l")) { + from = fopen(argv[1], "r"); + if (!from) + return 0; + to = stdout; + } else if (argc == 2) { + from = stdin; + to = fopen(argv[1], "w"); + } else + return error("unknown arguments"); + + while ((a = fgetc(from)) != EOF) + fputc(a, to); + + if (argc == 3) + fclose(from); + else + fclose(to); + + return 0; +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 590b2efca7..432b49d948 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -18,6 +18,7 @@ static struct test_cmd cmds[] = { { "bloom", cmd__bloom }, { "chmtime", cmd__chmtime }, { "config", cmd__config }, + { "crontab", cmd__crontab }, { "ctype", cmd__ctype }, { "date", cmd__date }, { "delta", cmd__delta }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index ddc8e990e9..7c3281e071 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv); int cmd__bloom(int argc, const char **argv); int cmd__chmtime(int argc, const char **argv); int cmd__config(int argc, const char **argv); +int cmd__crontab(int argc, const char **argv); int cmd__ctype(int argc, const char **argv); int cmd__date(int argc, const char **argv); int cmd__delta(int argc, const char **argv); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 8f383d01d9..7715e40391 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -315,4 +315,32 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' +test_expect_success 'start from empty cron table' ' + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + grep "for-each-repo --config=maintenance.repo maintenance run --schedule=daily" cron.txt && + grep "for-each-repo --config=maintenance.repo maintenance run --schedule=hourly" cron.txt && + grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt +' + +test_expect_success 'stop from existing schedule' ' + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + # Operation is idempotent + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + test_must_be_empty cron.txt +' + +test_expect_success 'start preserves existing schedule' ' + echo "Important information!" >cron.txt && + GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + grep "Important information!" cron.txt +' + test_done diff --git a/t/test-lib.sh b/t/test-lib.sh index ef31f40037..4a60d1ed76 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 ' test_lazy_prereq REBASE_P ' test -z "$GIT_TEST_SKIP_REBASE_P" ' + +# Ensure that no test accidentally triggers a Git command +# that runs 'crontab', affecting a user's cron schedule. +# Tests that verify the cron integration must set this locally +# to avoid errors. +GIT_TEST_CRONTAB="exit 1" From patchwork Mon Oct 5 12:57:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11816627 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 869711752 for ; Mon, 5 Oct 2020 13:04:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6108E20874 for ; Mon, 5 Oct 2020 13:04:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="vM1S+VeX" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726746AbgJENEL (ORCPT ); Mon, 5 Oct 2020 09:04:11 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54450 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726165AbgJENEK (ORCPT ); Mon, 5 Oct 2020 09:04:10 -0400 Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 049DCC0613B6 for ; Mon, 5 Oct 2020 05:57:23 -0700 (PDT) Received: by mail-wm1-x342.google.com with SMTP id z22so1711214wmi.0 for ; Mon, 05 Oct 2020 05:57:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=45zsat4+nWUk+8NCAMYKXXJVlAnGS7QiFYr3T9GBVpc=; b=vM1S+VeXL/D/0O8Chmriu9Du9F4vfTyU1e0GOOmEQnzzXm3+wMEu3vEHD9iav9jNpE J01Bv5ZbJRQxKivS9WFBH6CfDyfXCWz81ou/fYpRs6h3CsCMhhu7rd8IdUgv6/rWTCUB nol0aB/Ba8PHUSMAKGYfgW6P3dBDWB3QMtkHT5jdy/lxleE5+DNYgfhxxJ06bq41Nc+N VH/hvAGGDYHU97dZmba4rPiK9MEGKz+7uNukEaOLxpB0QBoum8KHPN/+FTocQyA+KfQz LTXqSJdVHY2JheSapoPEMRqOAD5lUHxeTnIFOb+gerxpq8+kwgmc50JZO3yLNuIWDVg4 J3cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=45zsat4+nWUk+8NCAMYKXXJVlAnGS7QiFYr3T9GBVpc=; b=k7YY8pxVX8f80qmP7LSlkxrKixAHLnsMJvCNSJ1I56qPbKcWFdELoIWEpsEvTDc8ho 7AZR69xpZBONVX+7aLFpK01l8X3qfWeJ01+1yCwKyN9NRNvKctyqXnO3nfXYywa3FhU6 gHseiyb3t19Wrmyf/hYiEsv2dPgBW8NFiXRh1TOz5tFCuHb7erj2dsXR8Ao43Vhr4A/f 0Bn4NbJMkd7v7w/2k28GpWOM/asx8iThPlbkfEq/EajOWqIsZU+bsjU6xAULT9pXx7XP RVsyj3LHllFOZ+awFKQGQOxXwTNmSsRSTUiSl7zbcT7TyzKnOAw6QPGBxAz44oxExg/8 C8bA== X-Gm-Message-State: AOAM530obxas2Qxsc7I/w022CTv0LzulUjIL47cvzer4S5FpOOpcLkpY i3kXv/VSDYc9Rtex3FPudThKfA9KICk= X-Google-Smtp-Source: ABdhPJzkjrIDPXCl5Kj7AbkISJMlDe+VSUJ4dNL70wAXnGGlxVGJRkj+A1Sp2/ZxmHsuy1MLRqgoew== X-Received: by 2002:a1c:bcd5:: with SMTP id m204mr8920350wmf.26.1601902641434; Mon, 05 Oct 2020 05:57:21 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f6sm12934618wro.5.2020.10.05.05.57.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 05:57:20 -0700 (PDT) Message-Id: In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 05 Oct 2020 12:57:13 +0000 Subject: [PATCH v3 6/7] maintenance: use default schedule if not configured MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, SZEDER =?utf-8?b?R8OhYm9y?= , Derrick Stolee , =?utf-8?b?xJBvw6BuIFRy4bqnbiBDw7RuZw==?= Danh , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git maintenance (register|start)' subcommands add the current repository to the global Git config so maintenance will operate on that repository. It does not specify what maintenance should occur or how often. If a user sets any 'maintenance..schedule' config value, then they have chosen a specific schedule for themselves and Git should respect that when running 'git maintenance run --schedule='. To make this process extremely simple for users, assume a default schedule when no 'maintenance..schedule' or '...enabled' config settings are concretely set. This is only an in-process assumption, so future versions of Git could adjust this expected schedule. Helped-by: Martin Ågren Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 15 ++++++++ builtin/gc.c | 58 +++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 11 +++--- 3 files changed, 80 insertions(+), 4 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 7628a6d157..52fff86844 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -37,6 +37,21 @@ register:: `maintenance..schedule`. The tasks that are enabled are safe for running in the background without disrupting foreground processes. ++ +If your repository has no `maintenance..schedule` configuration +values set, then Git will use a recommended default schedule that performs +background maintenance that will not interrupt foreground commands. The +default schedule is as follows: ++ +* `gc`: disabled. +* `commit-graph`: hourly. +* `prefetch`: hourly. +* `loose-objects`: daily. +* `incremental-repack`: daily. ++ +`git maintenance register` will also disable foreground maintenance by +setting `maintenance.auto = false` in the current repository. This config +setting will remain after a `git maintenance unregister` command. run:: Run one or more maintenance tasks. If one or more `--task` options diff --git a/builtin/gc.c b/builtin/gc.c index a387f46585..965690704b 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1251,6 +1251,59 @@ static int compare_tasks_by_selection(const void *a_, const void *b_) return b->selected_order - a->selected_order; } +static int has_schedule_config(void) +{ + int i, found = 0; + struct strbuf config_name = STRBUF_INIT; + size_t prefix; + + strbuf_addstr(&config_name, "maintenance."); + prefix = config_name.len; + + for (i = 0; !found && i < TASK__COUNT; i++) { + char *value; + + strbuf_setlen(&config_name, prefix); + strbuf_addf(&config_name, "%s.schedule", tasks[i].name); + + if (!git_config_get_string(config_name.buf, &value)) { + found = 1; + FREE_AND_NULL(value); + } + + strbuf_setlen(&config_name, prefix); + strbuf_addf(&config_name, "%s.enabled", tasks[i].name); + + if (!git_config_get_string(config_name.buf, &value)) { + found = 1; + FREE_AND_NULL(value); + } + } + + strbuf_release(&config_name); + return found; +} + +static void set_recommended_schedule(void) +{ + if (has_schedule_config()) + return; + + tasks[TASK_GC].enabled = 0; + + tasks[TASK_PREFETCH].enabled = 1; + tasks[TASK_PREFETCH].schedule = SCHEDULE_HOURLY; + + tasks[TASK_COMMIT_GRAPH].enabled = 1; + tasks[TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY; + + tasks[TASK_LOOSE_OBJECTS].enabled = 1; + tasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY; + + tasks[TASK_INCREMENTAL_REPACK].enabled = 1; + tasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY; +} + static int maintenance_run_tasks(struct maintenance_run_opts *opts) { int i, found_selected = 0; @@ -1280,6 +1333,8 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts) if (found_selected) QSORT(tasks, TASK__COUNT, compare_tasks_by_selection); + else if (opts->schedule != SCHEDULE_NONE) + set_recommended_schedule(); for (i = 0; i < TASK__COUNT; i++) { if (found_selected && tasks[i].selected_order < 0) @@ -1417,6 +1472,9 @@ static int maintenance_register(void) if (!the_repository || !the_repository->gitdir) return 0; + /* Disable foreground maintenance */ + git_config_set("maintenance.auto", "false"); + config_get.git_cmd = 1; strvec_pushl(&config_get.args, "config", "--global", "--get", "maintenance.repo", the_repository->worktree ? the_repository->worktree diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 7715e40391..7154987fd2 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -305,11 +305,14 @@ test_expect_success 'register and unregister' ' git config --global --add maintenance.repo /existing1 && git config --global --add maintenance.repo /existing2 && git config --global --get-all maintenance.repo >before && + git maintenance register && - git config --global --get-all maintenance.repo >actual && - cp before after && - pwd >>after && - test_cmp after actual && + test_cmp_config false maintenance.auto && + git config --global --get-all maintenance.repo >between && + cp before expect && + pwd >>expect && + test_cmp expect between && + git maintenance unregister && git config --global --get-all maintenance.repo >actual && test_cmp before actual From patchwork Mon Oct 5 12:57:14 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Linus Arver via GitGitGadget X-Patchwork-Id: 11816639 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B45F618 for ; Mon, 5 Oct 2020 13:04:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D75FB20848 for ; Mon, 5 Oct 2020 13:04:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="A3lBKu9T" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726768AbgJENEa (ORCPT ); Mon, 5 Oct 2020 09:04:30 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54456 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726232AbgJENEK (ORCPT ); Mon, 5 Oct 2020 09:04:10 -0400 Received: from mail-wr1-x443.google.com (mail-wr1-x443.google.com [IPv6:2a00:1450:4864:20::443]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D85D9C0613B7 for ; Mon, 5 Oct 2020 05:57:23 -0700 (PDT) Received: by mail-wr1-x443.google.com with SMTP id n18so1633965wrs.5 for ; Mon, 05 Oct 2020 05:57:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=lHGs8NOv3ROvd3brhi9use7jNjcKsphiI2pW+nWzM4I=; b=A3lBKu9TK6DprEr3IGJeJLVc73KSHCwn9DWEVBiJ1z8sVb5C4yvdsI3nrnOkNysUaY /Zx0f4mn+OuOZH6oYFzZ8kyQImbgiveIkNjJoi/huEicKwq4ZwsOqY6mXPGCPlZYQQVO TFh5XCXAesXCgXJJaquskWCZycBIQ51EuFesVv16/Wpsq/Up4bP1044h0NIHMgetjgrz ulpkD/0SOrODJsRQ2LmSmaZFYAlS/mUCh960XMrVOulI/v8GaVHhqYJDS9cVgp/NaxDY 3pzbL9lC6jaHlIKGQYH77ZL+b9+Q+bRIYpmhPJD6ct3ndsHIK2M1XrSwqdtP2eX15cq6 1+ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=lHGs8NOv3ROvd3brhi9use7jNjcKsphiI2pW+nWzM4I=; b=ANpzzeyS5HrCCV/y1x9Sec1u4CyW2PeB6t1GF/+feOXOan75XCtk5YQ/fCUREjpjSl rGr/52xY6Yce5chCuXuYwr+aykaQXm/8RyMGE7qQCrbkh1jlzxtGmCNgfpF6az161Ref KrP1/mhA/bRpZ1XvCPg+VL8zo8te4QRfdgUVon2L+aaz2TRQ/rjhipW7HFmPKB/fXmCg rlcQ5pV1sxwC9A+6VRQGBXqaQRMylM/7AU7U0HGNj2SmsHqXQ+2Em8fNW9oEFtzs5ZnU I6r/Dx2mbc/SA2ysSZpHoxkNFJ8nA42QavNGPe+Tm70YCaDFkzbQ0ivMJQUwkTgW5xfs HyVw== X-Gm-Message-State: AOAM533hgrXNncrLDsJZlgiEyBUcKlvvQATiwneccs9sEZUkWF9njHTP TR5ghEMFK4GWXWBDuZULCi3z9XNluYI= X-Google-Smtp-Source: ABdhPJzAhubFu5eYcwbgUmJ28KLDHZ3+HPLIUw+B43Jt6+z7PFAymZnCJHyVvVL9zAvFu1Un9QQjmA== X-Received: by 2002:a5d:4311:: with SMTP id h17mr18031541wrq.398.1601902642360; Mon, 05 Oct 2020 05:57:22 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z83sm13001757wmb.4.2020.10.05.05.57.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Oct 2020 05:57:21 -0700 (PDT) Message-Id: <8e42ff44ce9b28a070648cf760f94e2543cf0812.1601902635.git.gitgitgadget@gmail.com> In-Reply-To: References: From: "Derrick Stolee via GitGitGadget" Date: Mon, 05 Oct 2020 12:57:14 +0000 Subject: [PATCH v3 7/7] maintenance: add troubleshooting guide to docs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: jrnieder@gmail.com, jonathantanmy@google.com, sluongng@gmail.com, congdanhqx@gmail.com, SZEDER =?utf-8?b?R8OhYm9y?= , Derrick Stolee , =?utf-8?b?xJBvw6BuIFRy4bqnbiBDw7RuZw==?= Danh , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee The 'git maintenance run' subcommand takes a lock on the object database to prevent concurrent processes from competing for resources. This is an important safety measure to prevent possible repository corruption and data loss. This feature can lead to confusing behavior if a user is not aware of it. Add a TROUBLESHOOTING section to the 'git maintenance' builtin documentation that discusses these tradeoffs. The short version of this section is that Git will not corrupt your repository, but if the list of scheduled tasks takes longer than an hour then some scheduled tasks may be dropped due to this object database collision. For example, a long-running "daily" task at midnight might prevent an "hourly" task from running at 1AM. The opposite is also possible, but less likely as long as the "hourly" tasks are much faster than the "daily" and "weekly" tasks. Helped-by: Junio C Hamano Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 44 +++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 52fff86844..738a4c7ebd 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -173,6 +173,50 @@ OPTIONS `maintenance..enabled` configured as `true` are considered. See the 'TASKS' section for the list of accepted `` values. + +TROUBLESHOOTING +--------------- +The `git maintenance` command is designed to simplify the repository +maintenance patterns while minimizing user wait time during Git commands. +A variety of configuration options are available to allow customizing this +process. The default maintenance options focus on operations that complete +quickly, even on large repositories. + +Users may find some cases where scheduled maintenance tasks do not run as +frequently as intended. Each `git maintenance run` command takes a lock on +the repository's object database, and this prevents other concurrent +`git maintenance run` commands from running on the same repository. Without +this safeguard, competing processes could leave the repository in an +unpredictable state. + +The background maintenance schedule runs `git maintenance run` processes +on an hourly basis. Each run executes the "hourly" tasks. At midnight, +that process also executes the "daily" tasks. At midnight on the first day +of the week, that process also executes the "weekly" tasks. A single +process iterates over each registered repository, performing the scheduled +tasks for that frequency. Depending on the number of registered +repositories and their sizes, this process may take longer than an hour. +In this case, multiple `git maintenance run` commands may run on the same +repository at the same time, colliding on the object database lock. This +results in one of the two tasks not running. + +If you find that some maintenance windows are taking longer than one hour +to complete, then consider reducing the complexity of your maintenance +tasks. For example, the `gc` task is much slower than the +`incremental-repack` task. However, this comes at a cost of a slightly +larger object database. Consider moving more expensive tasks to be run +less frequently. + +Expert users may consider scheduling their own maintenance tasks using a +different schedule than is available through `git maintenance start` and +Git configuration options. These users should be aware of the object +database lock and how concurrent `git maintenance run` commands behave. +Further, the `git gc` command should not be combined with +`git maintenance run` commands. `git gc` modifies the object database +but does not take the lock in the same way as `git maintenance run`. If +possible, use `git maintenance run --task=gc` instead of `git gc`. + + GIT --- Part of the linkgit:git[1] suite