From patchwork Tue Nov 17 21:13:15 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11913555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 02D54C2D0E4 for ; Tue, 17 Nov 2020 21:13:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8E76F24248 for ; Tue, 17 Nov 2020 21:13:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="FPMUKHEK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726904AbgKQVNX (ORCPT ); Tue, 17 Nov 2020 16:13:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48272 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725823AbgKQVNW (ORCPT ); Tue, 17 Nov 2020 16:13:22 -0500 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 532A8C0613CF for ; Tue, 17 Nov 2020 13:13:22 -0800 (PST) Received: by mail-wr1-x442.google.com with SMTP id m6so7643778wrg.7 for ; Tue, 17 Nov 2020 13:13:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=GeHMVAQUD8d6JoZr3OOSyoRumStUgEOOmfk4MjdgbTM=; b=FPMUKHEKuoj7aLVCJE18eqglKz2NDpUKdd7Hm8tlmkiV/5Qhr0SaWR1KJisb4UVRTy KlrvcZ2NHrgXNHf7FXhIvdfSCPqMJHJLUBQQsJsuVDR5iECkxZZdvswBw2RJTpbONQvb vORbCzcJAL32nNTUf2W2Q7RMb4k+bdmbYt4yo6nSyC2cfKEwR2F/xDV8zEJ0n6q5ibUp 1V/AEvOJmpZcFcIexKvJCMNbzQAVuP3Er8Hyf/Te4V9VhMcudrRHmx1QgvqHCMyNAfSd llV54yBaxZJRzE1dqCZYwPTcItO7goAEeTGQbSqEHgblAqu+gJgCvSrqN1CLOnkRsuH1 1hqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=GeHMVAQUD8d6JoZr3OOSyoRumStUgEOOmfk4MjdgbTM=; b=gYLjd/+mpbMO4l6RU8T8EQunZpsWJor/P/o5pwCu5w0WcYnToQ9T2CfZdi+bcqG0wy h/VElR5jE2XXyrIv3QuGqW8O7uXnFbqBVujNOivDPzFuuW++0exmw1jK5Y/SHMlf7175 PTtrCtz2paIlO2GgyhO5RVAxO8EapHU3j4kuy9CJDu7ILovcDbnBsMkQbppHV05KgQ+c l24SJzxPms3k+2sWh1mG0rIc7u/DDQFlYre/lPf+hgQO90fLq5aRWlipHoYQbZfGxo+N RrgATohwlgoQObVGpAGasXzVPEY1xpRWOLm1xoA3vNB35Vg2tZ/9TlTDNE1NB17ImV0S PeBQ== X-Gm-Message-State: AOAM533VLKqdGit8MaQVqqlIksVc570GK2orl8FE37j8wnA3WP7DyXYX /44+7tlL0rPDVAG6s93GFRIFAy4UM/o= X-Google-Smtp-Source: ABdhPJxid0qPyqNtwpB61xAXIATk+OqDdC+Z//Zcf6+/Jd0kTbJ+LT6zHdVXny794Y5uni82XC7xKQ== X-Received: by 2002:a5d:6050:: with SMTP id j16mr1598492wrt.158.1605647600878; Tue, 17 Nov 2020 13:13:20 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g4sm29927656wrp.0.2020.11.17.13.13.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Nov 2020 13:13:20 -0800 (PST) Message-Id: <4807342b0019be29bb369ed3403a485f0ce9c15d.1605647598.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 17 Nov 2020 21:13:15 +0000 Subject: [PATCH v4 1/4] maintenance: extract platform-specific scheduling Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The existing schedule mechanism using 'cron' is supported by POSIX platforms, but not Windows. It also works slightly differently on macOS to significant detriment of the user experience. To allow for new implementations on these platforms, extract a method that performs the platform-specific scheduling mechanism. This will be swapped at compile time with new implementations on specialized platforms. As we add this generality, rename GIT_TEST_CRONTAB to GIT_TEST_MAINT_SCHEDULER. Further, this variable is now parsed as ":" so we can test platform-specific scheduling logic even when not on the correct platform. By specifying the in this string, we will be able to test all three sets of Git logic from a Linux machine. Co-authored-by: Eric Sunshine Signed-off-by: Eric Sunshine Signed-off-by: Derrick Stolee --- builtin/gc.c | 70 ++++++++++++++++++++++++++---------------- t/t7900-maintenance.sh | 8 ++--- t/test-lib.sh | 7 +++-- 3 files changed, 51 insertions(+), 34 deletions(-) diff --git a/builtin/gc.c b/builtin/gc.c index e3098ef6a1..18ae7f7138 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1494,35 +1494,23 @@ static int maintenance_unregister(void) #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" -static int update_background_schedule(int run_maintenance) +static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) { int result = 0; int in_old_region = 0; struct child_process crontab_list = CHILD_PROCESS_INIT; struct child_process crontab_edit = CHILD_PROCESS_INIT; FILE *cron_list, *cron_in; - const char *crontab_name; struct strbuf line = STRBUF_INIT; - struct lock_file lk; - char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); - if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) - return error(_("another process is scheduling background maintenance")); - - crontab_name = getenv("GIT_TEST_CRONTAB"); - if (!crontab_name) - crontab_name = "crontab"; - - strvec_split(&crontab_list.args, crontab_name); + strvec_split(&crontab_list.args, cmd); strvec_push(&crontab_list.args, "-l"); crontab_list.in = -1; - crontab_list.out = dup(lk.tempfile->fd); + crontab_list.out = dup(fd); crontab_list.git_cmd = 0; - if (start_command(&crontab_list)) { - result = error(_("failed to run 'crontab -l'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_list)) + return error(_("failed to run 'crontab -l'; your system might not support 'cron'")); /* Ignore exit code, as an empty crontab will return error. */ finish_command(&crontab_list); @@ -1531,17 +1519,15 @@ static int update_background_schedule(int run_maintenance) * Read from the .lock file, filtering out the old * schedule while appending the new schedule. */ - cron_list = fdopen(lk.tempfile->fd, "r"); + cron_list = fdopen(fd, "r"); rewind(cron_list); - strvec_split(&crontab_edit.args, crontab_name); + strvec_split(&crontab_edit.args, cmd); crontab_edit.in = -1; crontab_edit.git_cmd = 0; - if (start_command(&crontab_edit)) { - result = error(_("failed to run 'crontab'; your system might not support 'cron'")); - goto cleanup; - } + if (start_command(&crontab_edit)) + return error(_("failed to run 'crontab'; your system might not support 'cron'")); cron_in = fdopen(crontab_edit.in, "w"); if (!cron_in) { @@ -1586,14 +1572,44 @@ static int update_background_schedule(int run_maintenance) close(crontab_edit.in); done_editing: - if (finish_command(&crontab_edit)) { + if (finish_command(&crontab_edit)) result = error(_("'crontab' died")); - goto cleanup; + else + fclose(cron_list); + return result; +} + +static const char platform_scheduler[] = "crontab"; + +static int update_background_schedule(int enable) +{ + int result; + const char *scheduler = platform_scheduler; + const char *cmd = scheduler; + char *testing; + struct lock_file lk; + char *lock_path = xstrfmt("%s/schedule", the_repository->objects->odb->path); + + testing = xstrdup_or_null(getenv("GIT_TEST_MAINT_SCHEDULER")); + if (testing) { + char *sep = strchr(testing, ':'); + if (!sep) + die("GIT_TEST_MAINT_SCHEDULER unparseable: %s", testing); + *sep = '\0'; + scheduler = testing; + cmd = sep + 1; } - fclose(cron_list); -cleanup: + if (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0) + return error(_("another process is scheduling background maintenance")); + + if (!strcmp(scheduler, "crontab")) + result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); + else + die("unknown background scheduler: %s", scheduler); + rollback_lock_file(&lk); + free(testing); return result; } diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1..eeb939168d 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -368,7 +368,7 @@ test_expect_success 'register and unregister' ' ' test_expect_success 'start from empty cron table' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && # start registers the repo git config --get --global maintenance.repo "$(pwd)" && @@ -379,19 +379,19 @@ test_expect_success 'start from empty cron table' ' ' test_expect_success 'stop from existing schedule' ' - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo git config --get --global maintenance.repo "$(pwd)" && # Operation is idempotent - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance stop && test_must_be_empty cron.txt ' test_expect_success 'start preserves existing schedule' ' echo "Important information!" >cron.txt && - GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && + GIT_TEST_MAINT_SCHEDULER="crontab:test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..ddbeee1f5e 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1704,7 +1704,8 @@ test_lazy_prereq REBASE_P ' ' # Ensure that no test accidentally triggers a Git command -# that runs 'crontab', affecting a user's cron schedule. -# Tests that verify the cron integration must set this locally +# that runs the actual maintenance scheduler, affecting a user's +# system permanently. +# Tests that verify the scheduler integration must set this locally # to avoid errors. -GIT_TEST_CRONTAB="exit 1" +GIT_TEST_MAINT_SCHEDULER="none:exit 1" From patchwork Tue Nov 17 21:13:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11913557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F3F8C63697 for ; Tue, 17 Nov 2020 21:13:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D5FFB2463D for ; Tue, 17 Nov 2020 21:13:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="fM+c03+p" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727254AbgKQVNY (ORCPT ); Tue, 17 Nov 2020 16:13:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48280 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727012AbgKQVNX (ORCPT ); Tue, 17 Nov 2020 16:13:23 -0500 Received: from mail-wm1-x343.google.com (mail-wm1-x343.google.com [IPv6:2a00:1450:4864:20::343]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1CD03C0613CF for ; Tue, 17 Nov 2020 13:13:23 -0800 (PST) Received: by mail-wm1-x343.google.com with SMTP id p19so2839492wmg.0 for ; Tue, 17 Nov 2020 13:13:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=sp0JJOb8Xo2lvjjF5FxDXSrq7tug88l9uB9y6jDD9mM=; b=fM+c03+p8d8LYSQ1b49tICJ3Q5QlPpFbMgLBB9xtcnD9/83sXK97328DNSiaraUNAl SxbIjpbNxG/Mef3do75dfajCjrEA95P7xDNLMS16UUwarZi1ULT2GjQm25U0hSXoIjLG Fu6bNIccENrak4+5uZRYIO3PLkTgTH0iMbLB2YgcNIxeP5tN5IJVHyGfO5zfp6Qih8b3 YWK7UJgMiay+yfxNVj/1lOi1Htjc8DabbFo9untoUIFEH4nrcltnOpNr1feuAxEnnE1r uZxXZvwpegCCPvpB2Br78dUBLnULbpgCFzjJg+pDpd4+Rj8e4SWP4RGynsSKZui3qZpP isiQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=sp0JJOb8Xo2lvjjF5FxDXSrq7tug88l9uB9y6jDD9mM=; b=PIRFrXuiyYyShqk/nFKLTPSnuTe5YoK3K40mtXOZfdo5YuAPuOnifkoR342f6uHRxy wSKkjRf57UCILYwuxw+P8TytLwF2qhIfkT4taUmpHmO5PpIb89IuuOZ7f9234oSMoVDw 3rthsL5EI4RaErutvIGxhERU2kDRP/oMRQ2nPQjelKoEzkqvresoLfdiFKrYBXmE9WK/ 95WHirmaG0iIjPWs4iKe/F7Sny9Hb3LwqVaYTW/23mMb3dJMN6/TwnPD7z4bxUg5W5ZF 4s/8pNkaP6wxXsxMJL2qXeAR0sMTTQhv/+LVwQttTxplcptGMnaTJ8wJNIN0DHuvacuh p8HQ== X-Gm-Message-State: AOAM531rpToF4n1DDv18m+1+LfHZdW8jWw68ydRzQa7o1RPqlJtM0ySO 2srLKtgY2W2I/779o+0g6ug2UFoBXd4= X-Google-Smtp-Source: ABdhPJxdbuPMD/4aGf2o832/98UKvSo3IP0wGAeMEgpSdkhR1WMCw9B2AnB3oppizmeKxpVqbGL0LA== X-Received: by 2002:a7b:c8c5:: with SMTP id f5mr978326wml.174.1605647601622; Tue, 17 Nov 2020 13:13:21 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t13sm30465628wru.67.2020.11.17.13.13.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Nov 2020 13:13:21 -0800 (PST) Message-Id: <99170df4626544c1dc26d2e188b215a776140a32.1605647598.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 17 Nov 2020 21:13:16 +0000 Subject: [PATCH v4 2/4] maintenance: include 'cron' details in docs Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Advanced and expert users may want to know how 'git maintenance start' schedules background maintenance in order to customize their own schedules beyond what the maintenance.* config values allow. Start a new set of sections in git-maintenance.txt that describe how 'cron' is used to run these tasks. This is particularly valuable for users who want to inspect what Git is doing or for users who want to customize the schedule further. Having a baseline can provide a way forward for users who have never worked with cron schedules. Helped-by: Eric Sunshine Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 54 +++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 6fec1eb8dc..4c7aac877d 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -218,6 +218,60 @@ Further, the `git gc` command should not be combined with but does not take the lock in the same way as `git maintenance run`. If possible, use `git maintenance run --task=gc` instead of `git gc`. +The following sections describe the mechanisms put in place to run +background maintenance by `git maintenance start` and how to customize +them. + +BACKGROUND MAINTENANCE ON POSIX SYSTEMS +--------------------------------------- + +The standard mechanism for scheduling background tasks on POSIX systems +is `cron`. This tool executes commands based on a given schedule. The +current list of user-scheduled tasks can be found by running `crontab -l`. +The schedule written by `git maintenance start` is similar to this: + +----------------------------------------------------------------------- +# BEGIN GIT MAINTENANCE SCHEDULE +# The following schedule was created by Git +# Any edits made in this region might be +# replaced in the future by a Git command. + +0 1-23 * * * "//git" --exec-path="/" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly +0 0 * * 1-6 "//git" --exec-path="/" for-each-repo --config=maintenance.repo maintenance run --schedule=daily +0 0 * * 0 "//git" --exec-path="/" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly + +# END GIT MAINTENANCE SCHEDULE +----------------------------------------------------------------------- + +The comments are used as a region to mark the schedule as written by Git. +Any modifications within this region will be completely deleted by +`git maintenance stop` or overwritten by `git maintenance start`. + +The `` string is loaded to specifically use the location for the +`git` executable used in the `git maintenance start` command. This allows +for multiple versions to be compatible. However, if the same user runs +`git maintenance start` with multiple Git executables, then only the +latest executable will be used. + +These commands use `git for-each-repo --config=maintenance.repo` to run +`git maintenance run --schedule=` on each repository listed in +the multi-valued `maintenance.repo` config option. These are typically +loaded from the user-specific global config located at `~/.gitconfig`. +The `git maintenance` process then determines which maintenance tasks +are configured to run on each repository with each `` using +the `maintenance..schedule` config options. These values are loaded +from the global or repository config values. + +If the config values are insufficient to achieve your desired background +maintenance schedule, then you can create your own schedule. If you run +`crontab -e`, then an editor will load with your user-specific `cron` +schedule. In that editor, you can add your own schedule lines. You could +start by adapting the default schedule listed earlier, or you could read +https://man7.org/linux/man-pages/man5/crontab.5.html[the `crontab` documentation] +for advanced scheduling techniques. Please do use the full path and +`--exec-path` techniques from the default schedule to ensure you are +executing the correct binaries in your schedule. + GIT --- From patchwork Fri Nov 13 14:00:23 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11903555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 189CCC61DD8 for ; Fri, 13 Nov 2020 14:00:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C1D7F2222F for ; Fri, 13 Nov 2020 14:00:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="qfCw4lwm" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726890AbgKMOAf (ORCPT ); Fri, 13 Nov 2020 09:00:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:54198 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726498AbgKMOAc (ORCPT ); Fri, 13 Nov 2020 09:00:32 -0500 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7605EC0617A7 for ; Fri, 13 Nov 2020 06:00:31 -0800 (PST) Received: by mail-wr1-x442.google.com with SMTP id 23so10002112wrc.8 for ; Fri, 13 Nov 2020 06:00:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=nrdt/ZGVHPlOL7C2hXQVtktTOEYxhIq5MfM1pca3jXs=; b=qfCw4lwmzr7zBL4Q97K999mf6wedCq4KFxLTYPxbv9z4+DmfcUY9ZC9RX+KJirga3K jXciddW3ZcVM2QTNUAGfgqttYG0V1Z18487u2UjT02ZZRORT3IHl5XCbGYYlfIbClfS5 l33n7+sHzc+8K6QgrT10xBAKWk2jzVS+3Z2u/pGYk7kvbILYk3tgLgkVOT4cc6L1FUOH 2RSaKvq/lZzX5qJv5p6Qy6vNtaeHj4mMY+OIInFyzvVvJwbMnvPvKlJjPRAVjRYG93Zp Fq5bXpz5MWk6N01j/GYBzSKxQ3jsie3RJGFCtzHf1UjzI/Mq0APgQubxqGRM8HUxvYWA CHDA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=nrdt/ZGVHPlOL7C2hXQVtktTOEYxhIq5MfM1pca3jXs=; b=Dkuwv2cM9rjOfINRI3Q7jz4RvPh0qORULnTJxW+wJ5genZxoT5fGIe2ImVhybSopgA IfutQ82KuJ6DTxlKUsAVqJB13XAxYBN4OKpdqmkC/4xgkSe0bjIJt2oa4C6vZLAcoN7V iJdUO4kesVckzQQU7/BGkyVbGDhIMxt5zJoh2Yu9T0G05qmgE8k0rchJyjxZ2ZnMRZV6 +tZHkb5KuSd7hNz0BE/cd0YqbO15rBbM3+LDos5xoxuagX+tMwBP+Wv+oBz8gRc1m8s2 8E73cq6m2zrZhJRZ1wci9cJjm0ItdlhNisNBrQgYHQ8V/+EHFaIVmbUfckFTTis1v1P1 rUQw== X-Gm-Message-State: AOAM530MnJtSpd/DDUNwk94P7rG17xQvC8JpTSWGj/3ZMpMseEZuR8lD GcSHYHyk1hMUyEQEeC+qQTO4t0G505s= X-Google-Smtp-Source: ABdhPJySD0xUsmLehNWMkfATtchHkyjGuqjDgJb44GbE/93zvm2iwERUIPlJm223kyJGZ5Bd2g5B9g== X-Received: by 2002:a5d:42c9:: with SMTP id t9mr3509709wrr.13.1605276029645; Fri, 13 Nov 2020 06:00:29 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id i33sm11548429wri.79.2020.11.13.06.00.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Nov 2020 06:00:29 -0800 (PST) Message-Id: <1629bcfcf82dbc2ed9889a0e9ea2d08427901c4e.1605276024.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 13 Nov 2020 14:00:23 +0000 Subject: [PATCH v3 3/4] maintenance: use launchctl on macOS Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The existing mechanism for scheduling background maintenance is done through cron. The 'crontab -e' command allows updating the schedule while cron itself runs those commands. While this is technically supported by macOS, it has some significant deficiencies: 1. Every run of 'crontab -e' must request elevated privileges through the user interface. When running 'git maintenance start' from the Terminal app, it presents a dialog box saying "Terminal.app would like to administer your computer. Administration can include modifying passwords, networking, and system settings." This is more alarming than what we are hoping to achieve. If this alert had some information about how "git" is trying to run "crontab" then we would have some reason to believe that this dialog might be fine. However, it also doesn't help that some scenarios just leave Git waiting for a response without presenting anything to the user. I experienced this when executing the command from a Bash terminal view inside Visual Studio Code. 2. While cron initializes a user environment enough for "git config --global --show-origin" to show the correct config file information, it does not set up the environment enough for Git Credential Manager Core to load credentials during a 'prefetch' task. My prefetches against private repositories required re-authenticating through UI pop-ups in a way that should not be required. The solution is to switch from cron to the Apple-recommended [1] 'launchd' tool. [1] https://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPSystemStartup/Chapters/ScheduledJobs.html The basics of this tool is that we need to create XML-formatted "plist" files inside "~/Library/LaunchAgents/" and then use the 'launchctl' tool to make launchd aware of them. The plist files include all of the scheduling information, along with the command-line arguments split across an array of tags. For example, here is my plist file for the weekly scheduled tasks: Labelorg.git-scm.git.weekly ProgramArguments /usr/local/libexec/git-core/git --exec-path=/usr/local/libexec/git-core for-each-repo --config=maintenance.repo maintenance run --schedule=weekly StartCalendarInterval Day0 Hour0 Minute0 The schedules for the daily and hourly tasks are more complicated since we need to use an array for the StartCalendarInterval with an entry for each of the six days other than the 0th day (to avoid colliding with the weekly task), and each of the 23 hours other than the 0th hour (to avoid colliding with the daily task). The "Label" value is currently filled with "org.git-scm.git.X" where X is the frequency. We need a different plist file for each frequency. The launchctl command needs to be aligned with a user id in order to initialize the command environment. This must be done using the 'launchctl bootstrap' subcommand. This subcommand is new as of macOS 10.11, which was released in September 2015. Before that release the 'launchctl load' subcommand was recommended. The best source of information on this transition I have seen is available at [2]. The current design does not preclude a future version that detects the available fatures of 'launchctl' to use the older commands. However, it is best to rely on the newest version since Apple might completely remove the deprecated version on short notice. [2] https://babodee.wordpress.com/2016/04/09/launchctl-2-0-syntax/ To remove a schedule, we must run 'launchctl bootout' with a valid plist file. We also need to 'bootout' a task before the 'bootstrap' subcommand will succeed, if such a task already exists. We can verify the commands that were run by 'git maintenance start' and 'git maintenance stop' by injecting a script that writes the command-line arguments into GIT_TEST_CRONTAB. An earlier version of this patch accidentally had an opening "" tag when it should have had a closing "" tag. This was caught during manual testing with actual 'launchctl' commands, but we do not want to update developers' tasks when running tests. It appears that macOS includes the "xmllint" tool which can verify the XML format, so call it from the macOS-specific tests to ensure the .plist files are well-formatted. Helped-by: Eric Sunshine Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 40 ++++++ builtin/gc.c | 195 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 48 +++++++- t/test-lib.sh | 4 + 4 files changed, 284 insertions(+), 3 deletions(-) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index 1aa1112418..5f8f63f098 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -273,6 +273,46 @@ schedule to ensure you are executing the correct binaries in your schedule. +BACKGROUND MAINTENANCE ON MACOS SYSTEMS +--------------------------------------- + +While macOS technically supports `cron`, using `crontab -e` requires +elevated privileges and the executed process does not have a full user +context. Without a full user context, Git and its credential helpers +cannot access stored credentials, so some maintenance tasks are not +functional. + +Instead, `git maintenance start` interacts with the `launchctl` tool, +which is the recommended way to schedule timed jobs in macOS. Scheduling +maintenance through `git maintenance (start|stop)` requires some +`launchctl` features available only in macOS 10.11 or later. + +Your user-specific scheduled tasks are stored as XML-formatted `.plist` +files in `~/Library/LaunchAgents/`. You can see the currently-registered +tasks using the following command: + +----------------------------------------------------------------------- +$ ls ~/Library/LaunchAgents/org.git-scm.git* +org.git-scm.git.daily.plist +org.git-scm.git.hourly.plist +org.git-scm.git.weekly.plist +----------------------------------------------------------------------- + +One task is registered for each `--schedule=` option. To +inspect how the XML format describes each schedule, open one of these +`.plist` files in an editor and inspect the `` element following +the `StartCalendarInterval` element. + +`git maintenance start` will overwrite these files and register the +tasks again with `launchctl`, so any customizations should be done by +creating your own `.plist` files with distinct names. Similarly, the +`git maintenance stop` command will unregister the tasks with `launchctl` +and delete the `.plist` files. + +To create more advanced customizations to your background tasks, see +launchctl.plist(5) for more information. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index c1f7d9bdc2..da2c892f68 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1491,6 +1491,200 @@ static int maintenance_unregister(void) return run_command(&config_unset); } +#if defined(__APPLE__) + +static char *get_service_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "org.git-scm.git.%s", frequency); + return strbuf_detach(&label, NULL); +} + +static char *get_service_filename(const char *name) +{ + char *expanded; + struct strbuf filename = STRBUF_INIT; + strbuf_addf(&filename, "~/Library/LaunchAgents/%s.plist", name); + + expanded = expand_user_path(filename.buf, 1); + if (!expanded) + die(_("failed to expand path '%s'"), filename.buf); + + strbuf_release(&filename); + return expanded; +} + +static const char *get_frequency(enum schedule_priority schedule) +{ + switch (schedule) { + case SCHEDULE_HOURLY: + return "hourly"; + case SCHEDULE_DAILY: + return "daily"; + case SCHEDULE_WEEKLY: + return "weekly"; + default: + BUG("invalid schedule %d", schedule); + } +} + +static char *get_uid(void) +{ + struct strbuf output = STRBUF_INIT; + struct child_process id = CHILD_PROCESS_INIT; + + strvec_pushl(&id.args, "/usr/bin/id", "-u", NULL); + if (capture_command(&id, &output, 0)) + die(_("failed to discover user id")); + + strbuf_trim_trailing_newline(&output); + return strbuf_detach(&output, NULL); +} + +static int boot_plist(int enable, const char *filename) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + char *uid = get_uid(); + const char *launchctl = getenv("GIT_TEST_CRONTAB"); + if (!launchctl) + launchctl = "/bin/launchctl"; + + strvec_split(&child.args, launchctl); + + if (enable) + strvec_push(&child.args, "bootstrap"); + else + strvec_push(&child.args, "bootout"); + strvec_pushf(&child.args, "gui/%s", uid); + strvec_push(&child.args, filename); + + child.no_stderr = 1; + child.no_stdout = 1; + + if (start_command(&child)) + die(_("failed to start launchctl")); + + result = finish_command(&child); + + free(uid); + return result; +} + +static int remove_plist(enum schedule_priority schedule) +{ + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + int result = boot_plist(0, filename); + unlink(filename); + free(filename); + free(name); + return result; +} + +static int remove_plists(void) +{ + return remove_plist(SCHEDULE_HOURLY) || + remove_plist(SCHEDULE_DAILY) || + remove_plist(SCHEDULE_WEEKLY); +} + +static int schedule_plist(const char *exec_path, enum schedule_priority schedule) +{ + FILE *plist; + int i; + const char *preamble, *repeat; + const char *frequency = get_frequency(schedule); + char *name = get_service_name(frequency); + char *filename = get_service_filename(name); + + if (safe_create_leading_directories(filename)) + die(_("failed to create directories for '%s'"), filename); + plist = xfopen(filename, "w"); + + preamble = "\n" + "\n" + "" + "\n" + "Label%s\n" + "ProgramArguments\n" + "\n" + "%s/git\n" + "--exec-path=%s\n" + "for-each-repo\n" + "--config=maintenance.repo\n" + "maintenance\n" + "run\n" + "--schedule=%s\n" + "\n" + "StartCalendarInterval\n" + "\n"; + fprintf(plist, preamble, name, exec_path, exec_path, frequency); + + switch (schedule) { + case SCHEDULE_HOURLY: + repeat = "\n" + "Hour%d\n" + "Minute0\n" + "\n"; + for (i = 1; i <= 23; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_DAILY: + repeat = "\n" + "Day%d\n" + "Hour0\n" + "Minute0\n" + "\n"; + for (i = 1; i <= 6; i++) + fprintf(plist, repeat, i); + break; + + case SCHEDULE_WEEKLY: + fprintf(plist, + "\n" + "Day0\n" + "Hour0\n" + "Minute0\n" + "\n"); + break; + + default: + /* unreachable */ + break; + } + fprintf(plist, "\n\n\n"); + + /* bootout might fail if not already running, so ignore */ + boot_plist(0, filename); + if (boot_plist(1, filename)) + die(_("failed to bootstrap service %s"), filename); + + fclose(plist); + free(filename); + free(name); + return 0; +} + +static int add_plists(void) +{ + const char *exec_path = git_exec_path(); + + return schedule_plist(exec_path, SCHEDULE_HOURLY) || + schedule_plist(exec_path, SCHEDULE_DAILY) || + schedule_plist(exec_path, SCHEDULE_WEEKLY); +} + +static int platform_update_schedule(int run_maintenance, int fd) +{ + if (run_maintenance) + return add_plists(); + else + return remove_plists(); +} +#else #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1585,6 +1779,7 @@ static int platform_update_schedule(int run_maintenance, int fd) fclose(cron_list); return result; } +#endif static int update_background_schedule(int run_maintenance) { diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 20184e96e1..29d340a828 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -367,7 +367,7 @@ test_expect_success 'register and unregister' ' test_cmp before actual ' -test_expect_success 'start from empty cron table' ' +test_expect_success !MACOS_MAINTENANCE 'start from empty cron table' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && # start registers the repo @@ -378,7 +378,7 @@ test_expect_success 'start from empty cron table' ' grep "for-each-repo --config=maintenance.repo maintenance run --schedule=weekly" cron.txt ' -test_expect_success 'stop from existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'stop from existing schedule' ' GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance stop && # stop does not unregister the repo @@ -389,12 +389,54 @@ test_expect_success 'stop from existing schedule' ' test_must_be_empty cron.txt ' -test_expect_success 'start preserves existing schedule' ' +test_expect_success !MACOS_MAINTENANCE 'start preserves existing schedule' ' echo "Important information!" >cron.txt && GIT_TEST_CRONTAB="test-tool crontab cron.txt" git maintenance start && grep "Important information!" cron.txt ' +test_expect_success MACOS_MAINTENANCE 'start and stop macOS maintenance' ' + write_script print-args "#!/bin/sh\necho \$* >>args" && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + # ~/Library/LaunchAgents + ls "$HOME/Library/LaunchAgents" >actual && + cat >expect <<-\EOF && + org.git-scm.git.daily.plist + org.git-scm.git.hourly.plist + org.git-scm.git.weekly.plist + EOF + test_cmp expect actual && + + rm expect && + for frequency in hourly daily weekly + do + PLIST="$HOME/Library/LaunchAgents/org.git-scm.git.$frequency.plist" && + xmllint --noout "$PLIST" && + grep schedule=$frequency "$PLIST" && + echo "bootout gui/$UID $PLIST" >>expect && + echo "bootstrap gui/$UID $PLIST" >>expect || return 1 + done && + test_cmp expect args && + + rm -f args && + GIT_TEST_CRONTAB="./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "bootout gui/$UID $HOME/Library/LaunchAgents/org.git-scm.git.%s.plist\n" \ + hourly daily weekly >expect && + test_cmp expect args && + ls "$HOME/Library/LaunchAgents" >actual && + test_line_count = 0 actual +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register && diff --git a/t/test-lib.sh b/t/test-lib.sh index 4a60d1ed76..620ffbf3af 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1703,6 +1703,10 @@ test_lazy_prereq REBASE_P ' test -z "$GIT_TEST_SKIP_REBASE_P" ' +test_lazy_prereq MACOS_MAINTENANCE ' + launchctl list +' + # Ensure that no test accidentally triggers a Git command # that runs 'crontab', affecting a user's cron schedule. # Tests that verify the cron integration must set this locally From patchwork Tue Nov 17 21:13:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 11913559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D61FC6379D for ; Tue, 17 Nov 2020 21:13:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AB93A24655 for ; Tue, 17 Nov 2020 21:13:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Eeb6yH3D" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727330AbgKQVN1 (ORCPT ); Tue, 17 Nov 2020 16:13:27 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725823AbgKQVNZ (ORCPT ); Tue, 17 Nov 2020 16:13:25 -0500 Received: from mail-wr1-x442.google.com (mail-wr1-x442.google.com [IPv6:2a00:1450:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CBF34C0617A6 for ; Tue, 17 Nov 2020 13:13:24 -0800 (PST) Received: by mail-wr1-x442.google.com with SMTP id s8so24811002wrw.10 for ; Tue, 17 Nov 2020 13:13:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=UWSxa6R0gDljhiKN78pr8Y3ZBB5V5ZzzfVcYtAc4nSQ=; b=Eeb6yH3DXfR9x+8uVMwHBpxtc42PvheZtuBS4TjyBpcbI0WMgnzYORBPGQAhliHC+R iZtnJxSnJGdTSBc1KA9/XMxJy+cEH3DoOk8Mp26bCplYaI8KidB60AuSCa+aDjqTD0Vk VgJdCw2GxUzlF1jU+Af8ar3RqFPqRY225ZLhAg/kxiQYDGZ1AmW1oGYrA9WUD+SIllWO uJkUmVjP/tt2P4PGdJ4cQgGSUAzDehgbqNj/AzvowKk+XdQ3gwehsl3ir7vBNxNzjIZO FlWZ6lFbB8bArTrSs27QaM1Ae26CWvHBBZ0sQqDuE+XSIkYJ7Wimb3Ke3OteKkKMqWT/ gvJg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=UWSxa6R0gDljhiKN78pr8Y3ZBB5V5ZzzfVcYtAc4nSQ=; b=rpTyVi1EieLpdY6A4s4GRQrHef2nN6PTIlbfLad8zytOZJOPg4VA4jm3rR9m2c8M5M QaBIFTlXFkqQozxIj0gCUnKRT8+f3DeE7H0Ksf4O57HgAXY6RQO1MYzFnGfokSc2lP51 mh7QixD1fKuNqzqyXYA6yNNn3zsLUGbBhwe8ue2DPVKC0ePW71v5HrIoEEYiPAlY1rP3 WuF8pKVZHxAZ389g/zaHeWD/w0dCCU9wUhvhxEr0I45b+WPKM4kNeS4bKjIEGTAHG765 TtfTZM3Gqph5hSshbqcipr2vNCbKpJjkzUxEHOF41Z2rr7eKaN8352smb4gqIWq24stX 0TTw== X-Gm-Message-State: AOAM531jyuknnysBDFFXVADaqVf6IumdKVwHGSP98hS5OhF10jC9knUs wGxUhaXbzXqXdNa+MoMULc/pNO72ebU= X-Google-Smtp-Source: ABdhPJzKhWyf61CVGiezCFC+XhuJltgBYSAyejbPj8EpbPAxQTvXFcxHUC8/P81aNeUQ3RpvXdMsHQ== X-Received: by 2002:a5d:514a:: with SMTP id u10mr1449222wrt.312.1605647603250; Tue, 17 Nov 2020 13:13:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id m18sm28102835wru.37.2020.11.17.13.13.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Nov 2020 13:13:22 -0800 (PST) Message-Id: In-Reply-To: References: Date: Tue, 17 Nov 2020 21:13:18 +0000 Subject: [PATCH v4 4/4] maintenance: use Windows scheduled tasks Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Eric Sunshine , Derrick Stolee , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Git's background maintenance uses cron by default, but this is not available on Windows. Instead, integrate with Task Scheduler. Tasks can be scheduled using the 'schtasks' command. There are several command-line options that can allow for some advanced scheduling, but unfortunately these seem to all require authenticating using a password. Instead, use the "/xml" option to pass an XML file that contains the configuration for the necessary schedule. These XML files are based on some that I exported after constructing a schedule in the Task Scheduler GUI. These options only run background maintenance when the user is logged in, and more fields are populated with the current username and SID at run-time by 'schtasks'. Since the GIT_TEST_MAINT_SCHEDULER environment variable allows us to specify 'schtasks' as the scheduler, we can test the Windows-specific logic on a macOS platform. Thus, add a check that the XML file written by Git is valid when xmllint exists on the system. There is a deficiency in the current design. Windows has two kinds of applications: GUI applications that start by "winmain()" and console applications that start by "main()". Console applications are attached to a new Console window if they are not already associated with a GUI application. This means that every hour the scheudled task launches a command window for the scheduled tasks. Not only is this visually obtrusive, but it also takes focus from whatever else the user is doing! A simple fix would be to insert a GUI application that acts as a shim between the scheduled task and Git. This is currently possible in Git for Windows by setting the tag equal to C:\Program Files\Git\git-bash.exe with options "--hide --no-needs-console --command=cmd\git.exe" followed by the arguments currently used. Since git-bash.exe is not included in Windows builds of core Git, I chose to leave out this feature. My plan is to submit a small patch to Git for Windows that converts the use of git.exe with this use of git-bash.exe in the short term. In the long term, we can consider creating this GUI shim application within core Git, perhaps in contrib/. Co-authored-by: Eric Sunshine Signed-off-by: Eric Sunshine Signed-off-by: Derrick Stolee --- Documentation/git-maintenance.txt | 22 ++++ builtin/gc.c | 165 ++++++++++++++++++++++++++++++ t/t7900-maintenance.sh | 40 ++++++++ 3 files changed, 227 insertions(+) diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt index f2d59f2bcc..e1adfff6db 100644 --- a/Documentation/git-maintenance.txt +++ b/Documentation/git-maintenance.txt @@ -313,6 +313,28 @@ To create more advanced customizations to your background tasks, see launchctl.plist(5) for more information. +BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS +----------------------------------------- + +Windows does not support `cron` and instead has its own system for +scheduling background tasks. The `git maintenance start` command uses +the `schtasks` command to submit tasks to this system. You can inspect +all background tasks using the Task Scheduler application. The tasks +added by Git have names of the form `Git Maintenance ()`. +The Task Scheduler GUI has ways to inspect these tasks, but you can also +export the tasks to XML files and view the details there. + +Note that since Git is a console application, these background tasks +create a console window visible to the current user. This can be changed +manually by selecting the "Run whether user is logged in or not" option +in Task Scheduler. This change requires a password input, which is why +`git maintenance start` does not select it by default. + +If you want to customize the background tasks, please rename the tasks +so future calls to `git maintenance (start|stop)` do not overwrite your +custom tasks. + + GIT --- Part of the linkgit:git[1] suite diff --git a/builtin/gc.c b/builtin/gc.c index 782769f243..f6c42f96c1 100644 --- a/builtin/gc.c +++ b/builtin/gc.c @@ -1671,6 +1671,167 @@ static int launchctl_update_schedule(int run_maintenance, int fd, const char *cm return launchctl_remove_plists(cmd); } +static char *schtasks_task_name(const char *frequency) +{ + struct strbuf label = STRBUF_INIT; + strbuf_addf(&label, "Git Maintenance (%s)", frequency); + return strbuf_detach(&label, NULL); +} + +static int schtasks_remove_task(enum schedule_priority schedule, const char *cmd) +{ + int result; + struct strvec args = STRVEC_INIT; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + strvec_split(&args, cmd); + strvec_pushl(&args, "/delete", "/tn", name, "/f", NULL); + + result = run_command_v_opt(args.v, 0); + + strvec_clear(&args); + free(name); + return result; +} + +static int schtasks_remove_tasks(const char *cmd) +{ + return schtasks_remove_task(SCHEDULE_HOURLY, cmd) || + schtasks_remove_task(SCHEDULE_DAILY, cmd) || + schtasks_remove_task(SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_schedule_task(const char *exec_path, enum schedule_priority schedule, const char *cmd) +{ + int result; + struct child_process child = CHILD_PROCESS_INIT; + const char *xml; + char *xmlpath; + struct tempfile *tfile; + const char *frequency = get_frequency(schedule); + char *name = schtasks_task_name(frequency); + + xmlpath = xstrfmt("%s/schedule-%s.xml", + the_repository->objects->odb->path, + frequency); + tfile = create_tempfile(xmlpath); + if (!tfile || !fdopen_tempfile(tfile, "w")) + die(_("failed to create '%s'"), xmlpath); + + xml = "\n" + "\n" + "\n" + "\n"; + fputs(xml, tfile->fp); + + switch (schedule) { + case SCHEDULE_HOURLY: + fprintf(tfile->fp, + "2020-01-01T01:00:00\n" + "true\n" + "\n" + "1\n" + "\n" + "\n" + "PT1H\n" + "PT23H\n" + "false\n" + "\n"); + break; + + case SCHEDULE_DAILY: + fprintf(tfile->fp, + "2020-01-01T00:00:00\n" + "true\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "\n" + "1\n" + "\n"); + break; + + case SCHEDULE_WEEKLY: + fprintf(tfile->fp, + "2020-01-01T00:00:00\n" + "true\n" + "\n" + "\n" + "\n" + "\n" + "1\n" + "\n"); + break; + + default: + break; + } + + xml = "\n" + "\n" + "\n" + "\n" + "InteractiveToken\n" + "LeastPrivilege\n" + "\n" + "\n" + "\n" + "IgnoreNew\n" + "true\n" + "true\n" + "true\n" + "false\n" + "PT72H\n" + "7\n" + "\n" + "\n" + "\n" + "\"%s\\git.exe\"\n" + "--exec-path=\"%s\" for-each-repo --config=maintenance.repo maintenance run --schedule=%s\n" + "\n" + "\n" + "\n"; + fprintf(tfile->fp, xml, exec_path, exec_path, frequency); + strvec_split(&child.args, cmd); + strvec_pushl(&child.args, "/create", "/tn", name, "/f", "/xml", xmlpath, NULL); + close_tempfile_gently(tfile); + + child.no_stdout = 1; + child.no_stderr = 1; + + if (start_command(&child)) + die(_("failed to start schtasks")); + result = finish_command(&child); + + delete_tempfile(&tfile); + free(xmlpath); + free(name); + return result; +} + +static int schtasks_schedule_tasks(const char *cmd) +{ + const char *exec_path = git_exec_path(); + + return schtasks_schedule_task(exec_path, SCHEDULE_HOURLY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_DAILY, cmd) || + schtasks_schedule_task(exec_path, SCHEDULE_WEEKLY, cmd); +} + +static int schtasks_update_schedule(int run_maintenance, int fd, const char *cmd) +{ + if (run_maintenance) + return schtasks_schedule_tasks(cmd); + else + return schtasks_remove_tasks(cmd); +} + #define BEGIN_LINE "# BEGIN GIT MAINTENANCE SCHEDULE" #define END_LINE "# END GIT MAINTENANCE SCHEDULE" @@ -1761,6 +1922,8 @@ static int crontab_update_schedule(int run_maintenance, int fd, const char *cmd) #if defined(__APPLE__) static const char platform_scheduler[] = "launchctl"; +#elif defined(GIT_WINDOWS_NATIVE) +static const char platform_scheduler[] = "schtasks"; #else static const char platform_scheduler[] = "crontab"; #endif @@ -1789,6 +1952,8 @@ static int update_background_schedule(int enable) if (!strcmp(scheduler, "launchctl")) result = launchctl_update_schedule(enable, lk.tempfile->fd, cmd); + else if (!strcmp(scheduler, "schtasks")) + result = schtasks_update_schedule(enable, lk.tempfile->fd, cmd); else if (!strcmp(scheduler, "crontab")) result = crontab_update_schedule(enable, lk.tempfile->fd, cmd); else diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh index 6d37312901..a26ff22541 100755 --- a/t/t7900-maintenance.sh +++ b/t/t7900-maintenance.sh @@ -453,6 +453,46 @@ test_expect_success !MINGW 'start and stop macOS maintenance' ' test_line_count = 0 actual ' +test_expect_success 'start and stop Windows maintenance' ' + write_script print-args <<-\EOF && + echo $* >>args + while test $# -gt 0 + do + case "$1" in + /xml) shift; xmlfile=$1; break ;; + *) shift ;; + esac + done + test -z "$xmlfile" || cp "$xmlfile" . + EOF + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance start && + + # start registers the repo + git config --get --global maintenance.repo "$(pwd)" && + + printf "/create /tn Git Maintenance (%s) /f /xml .git/objects/schedule-%s.xml\n" \ + hourly hourly daily daily weekly weekly >expect && + test_cmp expect args && + + for frequency in hourly daily weekly + do + test_xmllint "schedule-$frequency.xml" + done && + + rm -f args && + GIT_TEST_MAINT_SCHEDULER="schtasks:./print-args" git maintenance stop && + + # stop does not unregister the repo + git config --get --global maintenance.repo "$(pwd)" && + + rm expect && + printf "/delete /tn Git Maintenance (%s) /f\n" \ + hourly daily weekly >expect && + test_cmp expect args +' + test_expect_success 'register preserves existing strategy' ' git config maintenance.strategy none && git maintenance register &&