From patchwork Tue Jan 5 13:08:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11998835 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6230EC433E0 for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 28D6922AED for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729337AbhAENJP (ORCPT ); Tue, 5 Jan 2021 08:09:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44204 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727397AbhAENJN (ORCPT ); Tue, 5 Jan 2021 08:09:13 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07150C061793 for ; Tue, 5 Jan 2021 05:08:33 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id 3so2995385wmg.4 for ; Tue, 05 Jan 2021 05:08:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=RodYAdkQd8R+l8FdZHezwZH9UXeD13YYCk5O1JOaM8I=; b=WstGXo6+Nf2mOMPRZMPY21nqP66PsKOVJWKLRIEiPwck2qZ0u1U91R/FN4nNGbNZky pIebapi0An5M4NnHXHsiNs8PkdL2ewH3Sx0dFim+mjVrTQUgKHoWJgVrJT0OSFc/tPpk r6Jkk3EV+PBsmaWb4kxzp3ZNYDNp08LqE+2yaC9yqb9CQtqnBkXPED7kVQP/s9pv3s7k oBgdVYmLHHyYZyuNeUekhQkKd/xiqm9bMkpArgX7FfVU6Ezp9wp9oLrRFy0MUcgSo7Rw KsyjN7Kp1AlMNKGZtCIk8+my/fC/9vnu2zb5nRYDXLZRKGzIl73VPGuVIDEJ1ikp8RTE an4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=RodYAdkQd8R+l8FdZHezwZH9UXeD13YYCk5O1JOaM8I=; b=fFM86l8uUhXWFlD2J/wOlIS7p6Tf1tC6ccA12mK/qVxIaZmrcc4QAltI/0cCS6OrTA FNMynRUGdu+jR4WvA7lxZ1g3n8s1JPKHm3yIcVgGjpaioEUxy6Z3mxXIl3G5l1BNIOtz 5Q3nef6xM3RHLsvk0moRRG5V4JqhWTsPvLTy7xu0A24xCOyyHS+pWqAw7Xr79P08a/Vm hTRvb3X1NZCqwjtFkfI2VEoG+zSiUqzAilxugNjprRx36uzd9xkDspIqsuXnHPWqxb7l bq0T8C4W8EsxAfH3Xiitl+QkwyC9eV6kAA6NNVS49yGHL6LVVcS3B9/obCslHS8Gmdp4 /9Ng== X-Gm-Message-State: AOAM530K53Jf0xHgXz61I83F4g0OVLEd7L3hp1WN16Iv9f/q9m/yfHyC O11hn/Eyl2bGXTqWVxjvxN3+9f7Qkns= X-Google-Smtp-Source: ABdhPJwFT8oCexhtni/dXzxZ47/PcmYaGWRRhOZ4l56oJQfE1zggH1U/8j7JWWZfe4wWePa6jP5znQ== X-Received: by 2002:a1c:7d94:: with SMTP id y142mr3461108wmc.105.1609852111460; Tue, 05 Jan 2021 05:08:31 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e15sm96544077wrx.86.2021.01.05.05.08.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Jan 2021 05:08:30 -0800 (PST) Message-Id: <4807342b0019be29bb369ed3403a485f0ce9c15d.1609852108.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Jan 2021 13:08:25 +0000 Subject: [PATCH v7 1/4] maintenance: extract platform-specific scheduling Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. As we add this generality, rename GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as ":" so we can test platform-specific scheduling logic even when not on the correct platform. By specifying the in this string, we will be able to test all three sets of Git logic from a Linux machine. Co-authored-by: Eric Sunshine Signed-off-by: Eric Sunshine Signed-off-by: Derrick Stolee --- builtin/gc.c | 70 ++++++++++++++++++++++++++---------------- t/t7900-maintenance.sh | 8 ++--- t/test-lib.sh | 7 +++-- 3 files changed, 51 insertions(+), 34 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a12..18ae7f7138a 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,35 +1494,23 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; struct child_process crontab_list = CHILD_PROCESS_INIT; struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); - - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) - crontab_name = "crontab"; - - strvec_split(&crontab_list.args, crontab_name); + strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_list)) + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); /* Ignore exit code, as an empty crontab will return error. */ finish_command(&crontab_list); @@ -1531,17 +1519,15 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); + strvec_split(&crontab_edit.args, cmd); crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_edit)) + return error(_("failed to run 'crontab'; your system might not support 'cron'")); cron_in = fdopen(crontab_edit.in, "w"); if (!cron_in) { @@ -1586,14 +1572,44 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; + else + fclose(cron_list); + return result; +} + +static const char platform_scheduler[] = "crontab"; + +static int update_background_schedule(int enable) +{ + int result; + const char *scheduler = platform_scheduler; + const char *cmd = scheduler; + char *testing; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); + if (testing) { + char *sep = strchr(testing, ':'); + if (!sep) + die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); + *sep = '\0'; + scheduler = testing; + cmd = sep + 1; } - fclose(cron_list); -cleanup: + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); + rollback_lock_file(&lk); + free(testing); return result; } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1a..eeb939168da 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -368,7 +368,7 @@ test_expect_success 'register and unregister' ' ' test_expect_success 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -379,19 +379,19 @@ test_expect_success 'start from empty cron table' ' ' test_expect_success 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && # Operation is idempotent - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && test_must_be_empty cron.txt ' test_expect_success 'start preserves existing schedule' ' echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed766..ddbeee1f5eb 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1704,7 +1704,8 @@ test_lazy_prereq REBASE_P ' ' # Ensure that no test accidentally triggers a Git command -# that runs 'crontab', affecting a user's cron schedule. -# Tests that verify the cron integration must set this locally +# that runs the actual maintenance scheduler, affecting a user's +# system permanently. +# Tests that verify the scheduler integration must set this locally # to avoid errors. -GIT_TEST_CRONTAB="exit 1" +GIT_TEST_MAINT_SCHEDULER="none:exit 1" From patchwork Tue Jan 5 13:08:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11998833 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82E7FC433DB for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 437B922B37 for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729536AbhAENJQ (ORCPT ); Tue, 5 Jan 2021 08:09:16 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44206 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729176AbhAENJO (ORCPT ); Tue, 5 Jan 2021 08:09:14 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8467C061795 for ; Tue, 5 Jan 2021 05:08:33 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id t16so36130341wra.3 for ; Tue, 05 Jan 2021 05:08:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=gtvdyGfTmRDfT1qs0ZCAbuyHuiIlM022mJ22jXXFyKo=; b=JWsFbt0D3BmnlP+luXS5F63pV6nhtIrLrJvY7940lR/g/XotHPjceFiPFOrhY90noK GPds+oVokLWfvgF6jfPP7nL+VU9YByoXlyWhn41FIFly92/mYj1a4YtR/cdg87EvJtKp no6r+qWPLz+TO02o3HeRkPQJS1W6oClT7e17U0qfvSeOlp2HtwTIzC9+mNh1OMdE3onu YW/mtwUv8h7/+iiPSr4V/FRC0VKaiNYXCZGtOmeBqBRQSMPJumnpSFRHoe+XxB5n5+Oj Mo+H/gvcaOuFhrFTpdniK2MLPt3SDgCLDl7BwAjEoIzROA2uUSwaRPHLDWICKSR01vID M9NA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=gtvdyGfTmRDfT1qs0ZCAbuyHuiIlM022mJ22jXXFyKo=; b=X3/Oaj1wV8seuB4KxRhB7du1crQFhBhqPfz0SPh5j+thQeiFNTi0XXZ65uBR1iXvbA 5yzQQvQuWOL+0tFNsTiXpqysj05hSR1nJPr3rZ91TkMNuwyhWjGKP4rLW4hg7dSeojrV KroXmcXuf+mWt+iriOdsIZqU+tQqNlkOqwPwz/fmdGLuMZ0103MtarrpWutMPBf9ZURd SMf6oWrsM8OVdAHkU+mQfj6z5BQJkwIhrJjg9OGDucRWgijl/gPmAPdQF+JItXXXOUkr tcLgGB3er/q6wIKKPauGWtWAG4Jy6vMC9qlskYlPwlaDrYg++YIEBdsNW8pTtK1vy7NN 9qvg== X-Gm-Message-State: AOAM531WA/aNUlBbSJQdr4RGyfIqJjAu7omh6zjQMLMekJlcq+Qy3tjZ 6qbavucctTt2fql2wUBmg37adIfs3e0= X-Google-Smtp-Source: ABdhPJxNzB2sjn3rkbpGsRxnSqa0SLDPK2KSuiAMO0zircyPCvu2DXmMZyi3ngUNlv2CnnuWY4MQoA== X-Received: by 2002:adf:f707:: with SMTP id r7mr87713775wrp.113.1609852112478; Tue, 05 Jan 2021 05:08:32 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y7sm3873600wmb.37.2021.01.05.05.08.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Jan 2021 05:08:31 -0800 (PST) Message-Id: <7cc70a8fe7bad92179961b31520147af39d5353c.1609852109.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Jan 2021 13:08:26 +0000 Subject: [PATCH v7 2/4] maintenance: include 'cron' details in docs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc2..1aa11124186 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is cron(8). This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "//git" --exec-path="/" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "//git" --exec-path="/" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "//git" --exec-path="/" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `crontab` entry specifies the full path of the `git` executable to +ensure that the executed `git` command is the same one with which +`git maintenance start` was issued independent of `PATH`. If the same user +runs `git maintenance start` with multiple Git executables, then only the +latest executable is used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config. The `git maintenance` process +then determines which maintenance tasks are configured to run on each +repository with each `` using the `maintenance..schedule` +config options. These values are loaded from the global or repository +config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +the crontab(5) documentation for advanced scheduling techniques. Please +do use the full path and `--exec-path` techniques from the default +schedule to ensure you are executing the correct binaries in your +schedule. + GIT --- From patchwork Tue Jan 5 13:08:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11998839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4B51C433E9 for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 76D8122B30 for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729730AbhAENJR (ORCPT ); Tue, 5 Jan 2021 08:09:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44212 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727397AbhAENJP (ORCPT ); Tue, 5 Jan 2021 08:09:15 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32A39C061796 for ; Tue, 5 Jan 2021 05:08:35 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id i9so36125317wrc.4 for ; Tue, 05 Jan 2021 05:08:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:mime-version :content-transfer-encoding:fcc:to:cc; bh=SfeZtHvAZcdKjf4/1ooSPlrX3oyoViFaD4G679ZrPiE=; b=CbUqx0Ar0Lmmnf9N57GryIrSBIrQlaQ7rG9wIgwlX8Kl1U9rDTrj4d06+/TePW7o4L 7NQVB8PLv58YS6MeJlMtd0Y9ZDAXYcVZ23mEDR1NQsVYIud+QSvOZn3HigLWx8caaNe3 UppYVkZd2GTRg8iZ1rjwy9IKFg5D52vQPh/U4Btzu5A7GM6hhXFOCqHmu6bPpd8JHvR4 fDj0Oo9u6SV1X4YJ6vF9gdR0o3PGpfO8o3gP2qVlXuvJapGf1A1KrI+NKcoIAezZOZgx fWn54WGDrJLZyd+ZVYmpejRx8Uka5qxDYBRyXGHiuADCCklg4+K+9BBhUrICwiRE4aAM 5iMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:mime-version:content-transfer-encoding:fcc:to:cc; bh=SfeZtHvAZcdKjf4/1ooSPlrX3oyoViFaD4G679ZrPiE=; b=nuyU4mlGzoRGvizP6E6lFYwNLirij8eRXQhhAivXvdhzUzjZEp7+SoG1Zgk8UXshSW jZZKEdKftcdp0riEG23YAO5x+xMrwwSkThjVRXNyNPOPmowdOA+aivb3y7Qjy4JZ/j5c pf/0R08MUwGkO/F4fKmzgGipMPrNvNbRteu/IzbhMdY49yA3+oUSAQPx4xLVM96DJgxO sy1x+L5StWRMhabKrcDRxbzDdL2fHdj1HRQP9tkxt3lCLJqoEtOwYy3AL7mKf2ppYPEv zplBPMsgNxsXkr0huNoFc4MhwO/T2XUfklm6Yjr+gcGdpXegWHRQE9M2g5LJTnHV9huq QfWA== X-Gm-Message-State: AOAM530OBsEePU5k8sXJcOXIG+3yBf31yk8ODwmyIOnJ76UfdaDu1jGU EyhSAXDAyqgCjNlSO15Zbfn7jNSoEEk= X-Google-Smtp-Source: ABdhPJxnu3VR/87c3HhuKbXPO2zm9Lge7iRAP//rDUV2AtLtzu3nTJKnPJfQI03PJxJI47m5TW15xg== X-Received: by 2002:a5d:47a5:: with SMTP id 5mr86587262wrb.109.1609852113340; Tue, 05 Jan 2021 05:08:33 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id d191sm3764556wmd.24.2021.01.05.05.08.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Jan 2021 05:08:32 -0800 (PST) Message-Id: <3576c7aa54e08c11ac86675ca9bc11b80366d390.1609852109.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Jan 2021 13:08:27 +0000 Subject: [PATCH v7 3/4] maintenance: use launchctl on macOS MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of tags. For example, here is my plist file for the weekly scheduled tasks: Labelorg.git-scm.git.weekly ProgramArguments /usr/local/libexec/git-core/git --exec-path=/usr/local/libexec/git-core for-each-repo --config=maintenance.repo maintenance run --schedule=weekly StartCalendarInterval Day0 Hour0 Minute0 The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. The need for a user id requires us to run 'id -u' which works on POSIX systems but not Windows. Further, the need for fully-qualitifed path names including $HOME behaves differently in the Git internals and the external test suite. The $HOME variable starts with "C:\..." instead of the "/c/..." that is provided by Git in these subcommands. The test therefore has a prerequisite that we are not on Windows. The cross- platform logic still allows us to test the macOS logic on a Linux machine. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_MAINT_SCHEDULER. An earlier version of this patch accidentally had an opening "" tag when it should have had a closing "" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format. This is useful for any system that might contain the tool, so use it whenever it is available. We strive to make these tests work on all platforms, but Windows caused some headaches. In particular, the value of getuid() called by the C code is not guaranteed to be the same as `$(id -u)` invoked by a test. This is because `git.exe` is a native Windows program, whereas the utility programs run by the test script mostly utilize the MSYS2 runtime, which emulates a POSIX-like environment. Since the purpose of the test is to check that the input to the hook is well-formed, the actual user ID is immaterial, thus we can work around the problem by making the the test UID-agnostic. Another subtle issue is the $HOME environment variable being a Windows-style path instead of a Unix-style path. We can be more flexible here instead of expecting exact path matches. Helped-by: Ævar Arnfjörð Bjarmason Co-authored-by: Eric Sunshine Signed-off-by: Eric Sunshine Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 40 +++++++ builtin/gc.c | 188 +++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 59 ++++++++++ 3 files changed, 286 insertions(+), 1 deletion(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 1aa11124186..5f8f63f0988 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,46 @@ schedule to ensure you are executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to schedule timed jobs in macOS. Scheduling +maintenance through `git maintenance (start|stop)` requires some +`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `` element following +the `StartCalendarInterval` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +launchctl.plist(5) for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 18ae7f7138a..782769f2438 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,186 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *launchctl_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *launchctl_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static char *launchctl_get_uid(void) +{ + return xstrfmt("gui/%d", getuid()); +} + +static int launchctl_boot_plist(int enable, const char *filename, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + char *uid = launchctl_get_uid(); + + strvec_split(&child.args, cmd); + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); + strvec_push(&child.args, uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; + child.no_stdout = 1; + + if (start_command(&child)) + die(_("failed to start launchctl")); + + result = finish_command(&child); + + free(uid); + return result; +} + +static int launchctl_remove_plist(enum schedule_priority schedule, const char *cmd) +{ + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + int result = launchctl_boot_plist(0, filename, cmd); + unlink(filename); + free(filename); + free(name); + return result; +} + +static int launchctl_remove_plists(const char *cmd) +{ + return launchctl_remove_plist(SCHEDULE_HOURLY, cmd) || + launchctl_remove_plist(SCHEDULE_DAILY, cmd) || + launchctl_remove_plist(SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_schedule_plist(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = launchctl_service_name(frequency); + char *filename = launchctl_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + + preamble = "\n" + "\n" + "" + "\n" + "Label%s\n" + "ProgramArguments\n" + "\n" + "%s/git\n" + "--exec-path=%s\n" + "for-each-repo\n" + "--config=maintenance.repo\n" + "maintenance\n" + "run\n" + "--schedule=%s\n" + "\n" + "StartCalendarInterval\n" + "\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "\n" + "Hour%d\n" + "Minute0\n" + "\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "\n" + "Day%d\n" + "Hour0\n" + "Minute0\n" + "\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "\n" + "Day0\n" + "Hour0\n" + "Minute0\n" + "\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "\n\n\n"); + fclose(plist); + + /* bootout might fail if not already running, so ignore */ + launchctl_boot_plist(0, filename, cmd); + if (launchctl_boot_plist(1, filename, cmd)) + die(_("failed to bootstrap service %s"), filename); + + free(filename); + free(name); + return 0; +} + +static int launchctl_add_plists(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return launchctl_schedule_plist(exec_path, SCHEDULE_HOURLY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_DAILY, cmd) || + launchctl_schedule_plist(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int launchctl_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return launchctl_add_plists(cmd); + else + return launchctl_remove_plists(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1579,7 +1759,11 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) return result; } +#if defined(__APPLE__) +static const char platform_scheduler[] = "launchctl"; +#else static const char platform_scheduler[] = "crontab"; +#endif static int update_background_schedule(int enable) { @@ -1603,7 +1787,9 @@ static int update_background_schedule(int enable) if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) return error(_("another process is scheduling background maintenance")); - if (!strcmp(scheduler, "crontab")) + if (!strcmp(scheduler, "launchctl")) + result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else die("unknown background scheduler: %s", scheduler); diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index eeb939168da..adf24dee72d 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -7,6 +7,19 @@ test_description='git maintenance builtin' GIT_TEST_COMMIT_GRAPH=0 GIT_TEST_MULTI_PACK_INDEX=0 +test_lazy_prereq XMLLINT ' + xmllint --version +' + +test_xmllint () { + if test_have_prereq XMLLINT + then + xmllint --noout "$@" + else + true + fi +} + test_expect_success 'help text' ' test_expect_code 129 git maintenance -h 2>err && test_i18ngrep "usage: git maintenance " err && @@ -395,6 +408,52 @@ test_expect_success 'start preserves existing schedule' ' grep "Important information!" cron.txt ' +test_expect_success 'start and stop macOS maintenance' ' + # ensure $HOME can be compared against hook arguments on all platforms + pfx=$(cd "$HOME" && pwd) && + + write_script print-args <<-\EOF && + echo $* | sed "s:gui/[0-9][0-9]*:gui/[UID]:" >>args + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm -f expect && + for frequency in hourly daily weekly + do + PLIST="$pfx/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + test_xmllint "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/[UID] $PLIST" >>expect && + echo "bootstrap gui/[UID] $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER=launchctl:./print-args git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "bootout gui/[UID] $pfx/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && + test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && From patchwork Tue Jan 5 13:08:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11998837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8C685C433E6 for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5F0A922B3B for ; Tue, 5 Jan 2021 13:09:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729684AbhAENJR (ORCPT ); Tue, 5 Jan 2021 08:09:17 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729513AbhAENJQ (ORCPT ); Tue, 5 Jan 2021 08:09:16 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6DB7C061798 for ; Tue, 5 Jan 2021 05:08:35 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id t30so36153869wrb.0 for ; Tue, 05 Jan 2021 05:08:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=uqUiNtL73v9qRcMKJ/7YcDUUHeK1b32DpcQz9BCr8GA=; b=l1s3IzGZ3Y2/0sbOPEMWMMF6i8/a4maX+DwCpkbeze5qMdlQbId2jWVkT53V2rgVTD lVI4ctVo+ya2MDeghe6PJiLTZ1C5YVQwRCzHmXla/HfDxpkBSCNTzWirIm2EUEWXEfck SUgr1lHj1bMvo68tTEY/vsPhbWD64qFh9KNbCez8KJf40iSLTf5pJb79LgAkZpb6RBDe c6QOGnUcS48NiYrfkDhUVOerwQM3Ha8gYBxVNPjs2vyBCCjFbqrDleM4muOmVWUMJFNr 1IDn05kiru4052EuVrLJy7rHpw12HSjld9Mh2xFBpCyREEsrfrtKb3O3KGi1b5emidZ8 gMrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=uqUiNtL73v9qRcMKJ/7YcDUUHeK1b32DpcQz9BCr8GA=; b=cNgjsnReFg2HDLNEiGjazzcSn3Hri0QyCEmuZTeMqEQ1N96kGrFgknG5vrqJOHYSFr MEQAaixhKOd4pQA4uPUU14wugw5WeXk4N6jeKiwHBXTktuptGzkFnP4OSP5I4DgtETcg j7yTPG4ykPg4BZ9ETbG6Cj6bRX2XPhjxpNcvFxjLlWdpeJL5rA0jlz4AIm7tfhxiW4eT J9OTMGd3ebPeYFR07kE/wOCPCd9y+1xcR6YGtJjoSCHmRw/8Mch0FbgFCyyth3+ofQcm JQQiyD23whL4tqcw9ETvFwNcB5z/p+RU1wfr8cl4eeLqEL/KYG0IQtHHCeIWd1vpzirz Tc2g== X-Gm-Message-State: AOAM530/ni2PdF0cp6D67CiyTNJsLbEUbScbFRbh5Sx00xFbo+gO6J6x BU4sRb4Es3ohNj1aKeaWvH1jK5ls8Rg= X-Google-Smtp-Source: ABdhPJy1uVdph0mHSMD48hA8UAiTf1gLodPxeMv9jt+uwou7ojclMwDhJDBPwIB4od3HXIvWn3zdyw== X-Received: by 2002:adf:f401:: with SMTP id g1mr84535424wro.258.1609852114122; Tue, 05 Jan 2021 05:08:34 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a17sm99608464wrs.20.2021.01.05.05.08.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Jan 2021 05:08:33 -0800 (PST) Message-Id: <68f5013dee3762d5ad2fdc4bb229b522881efdf7.1609852109.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 05 Jan 2021 13:08:28 +0000 Subject: [PATCH v7 4/4] maintenance: use Windows scheduled tasks Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific logic on other platforms. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. Since we use a temporary file for the XML files sent to 'schtasks', we prefix the random characters with the frequency so it is easier to examine the proper file during tests. Instead of an exact match on the 'args' file, we 'grep' for the arguments other than the filename. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Co-authored-by: Eric Sunshine Signed-off-by: Eric Sunshine Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 168 +++++++++++++++++++++++++++++- t/t7900-maintenance.sh | 37 +++++++ 3 files changed, 226 insertions(+), 1 deletion(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 5f8f63f0988..6970f2b8983 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -313,6 +313,28 @@ To create more advanced customizations to your background tasks, see launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance ()`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 782769f2438..fdc95d9e99f 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1589,7 +1589,7 @@ static int launchctl_schedule_plist(const char *exec_path, enum schedule_priorit die(_("failed to create directories for '%s'"), filename); plist = xfopen(filename, "w"); - preamble = "\n" + preamble = "\n" "\n" "" "\n" @@ -1671,6 +1671,168 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm return launchctl_remove_plists(cmd); } +static char *schtasks_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + strvec_split(&args, cmd); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int schtasks_remove_tasks(const char *cmd) +{ + return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || + schtasks_remove_task(SCHEDULE_DAILY, cmd) || + schtasks_remove_task(SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml; + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + struct strbuf tfilename = STRBUF_INIT; + + strbuf_addf(&tfilename, "%s/schedule_%s_XXXXXX", + get_git_common_dir(), frequency); + tfile = xmks_tempfile(tfilename.buf); + strbuf_release(&tfilename); + + if (!fdopen_tempfile(tfile, "w")) + die(_("failed to create temp xml file")); + + xml = "\n" + "\n" + "\n" + "\n"; + fputs(xml, tfile->fp); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(tfile->fp, + "2020-01-01T01:00:00\n" + "true\n" + "\n" + "1\n" + "\n" + "\n" + "PT1H\n" + "PT23H\n" + "false\n" + "\n"); + break; + + case SCHEDULE_DAILY: + fprintf(tfile->fp, + "2020-01-01T00:00:00\n" + "true\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "1\n" + "\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(tfile->fp, + "2020-01-01T00:00:00\n" + "true\n" + "\n" + "\n" + "\n" + "\n" + "1\n" + "\n"); + break; + + default: + break; + } + + xml = "\n" + "\n" + "\n" + "\n" + "InteractiveToken\n" + "LeastPrivilege\n" + "\n" + "\n" + "\n" + "IgnoreNew\n" + "true\n" + "true\n" + "true\n" + "false\n" + "PT72H\n" + "7\n" + "\n" + "\n" + "\n" + "\"%s\\git.exe\"\n" + "--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s\n" + "\n" + "\n" + "\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", + get_tempfile_path(tfile), NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; + child.no_stderr = 1; + + if (start_command(&child)) + die(_("failed to start schtasks")); + result = finish_command(&child); + + delete_tempfile(&tfile); + free(name); + return result; +} + +static int schtasks_schedule_tasks(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return schtasks_schedule_tasks(cmd); + else + return schtasks_remove_tasks(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1761,6 +1923,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) #if defined(__APPLE__) static const char platform_scheduler[] = "launchctl"; +#elif defined(GIT_WINDOWS_NATIVE) +static const char platform_scheduler[] = "schtasks"; #else static const char platform_scheduler[] = "crontab"; #endif @@ -1789,6 +1953,8 @@ static int update_background_schedule(int enable) if (!strcmp(scheduler, "launchctl")) result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "schtasks")) + result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index adf24dee72d..135505f6195 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -454,6 +454,43 @@ test_expect_success 'start and stop macOS maintenance' ' test_line_count = 0 actual ' +test_expect_success 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + while test $# -gt 0 + do + case "$1" in + /xml) shift; xmlfile=$1; break ;; + *) shift ;; + esac + done + test -z "$xmlfile" || cp "$xmlfile" "$xmlfile.xml" + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + for frequency in hourly daily weekly + do + grep "/create /tn Git Maintenance ($frequency) /f /xml" args && + file=$(ls .git/schedule_${frequency}*.xml) && + test_xmllint "$file" || return 1 + done && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register &&