From patchwork Thu Mar 2 22:02:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13157943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05646C6FA8E for ; Thu, 2 Mar 2023 22:03:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229728AbjCBWD3 (ORCPT ); Thu, 2 Mar 2023 17:03:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229614AbjCBWD0 (ORCPT ); Thu, 2 Mar 2023 17:03:26 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F80059E65 for ; Thu, 2 Mar 2023 14:03:05 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id x9-20020a1709028ec900b0019cad25ecf5so325118plo.10 for ; Thu, 02 Mar 2023 14:03:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ub7DW9x6qpUF2zPgVvwHVDQqvKzP7eX/hRfcTS2AcKU=; b=BvGakBEMnT/VRL5SSqcspWbmFPBgc++BZ541zlZE6A7IxzwsZcYXlbze2BCngmD2w2 d2j8LFyeW3DNmJPQKaJ3algogZL+0VwZw4ki2wPhQXwl1aU4N5NurJOKCtFByxPJv+md 3wynVroXy9wGLA13VdJikHzSCROqioi5egXPhNC73YJpEDeFppKPUu+fZpHyfMby7Ryx 13rel8LejEe7LqhaP8yIwG0XAGzUI/898/9FhmcAxVCtypFkYeqIKqv9Gt/44yWMDuoU Xvz7VOabI6HrnkBhpFeuY9b76bE1+R7j1aeStOKLys+Ni8GC2QY98/RanD8Cn1enCpOw ueOw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ub7DW9x6qpUF2zPgVvwHVDQqvKzP7eX/hRfcTS2AcKU=; b=5o1uJ7VB+EFhfT/kRxiJX2ju+ZYtbgA1dDWDVDG8p03Aljkt5gflAHpyQnLMxCg10p 6v6L8F7q1dHP0edRaRd2hBylFJAUktdMETwz47eFssVX2GyA05y+GFIbjd1mXZgwbdDJ dvtSOo6IKyR5NJqfbS3PQje86b9Rm5rAF2DvUqOs8SksvjoJZeyDUvUILk1dp9oVeCqk uL/Op+JHpPmMs4uh3qTpp5bmuIOW4y5yD5CC6xydgwBL58Htz03gCcKGPBl9cqmmmFac n304+Oir1tbRmeI0LWKxlJjKcYdWl3f1oa+DoWR/k9tCNgxYf559Ux+Tt0tf+jj2WkLR MIYw== X-Gm-Message-State: AO0yUKUALcdwktXselkpCC0+W049Nn0Tyt8gYK03w9zsk7/v8Wbv26Ca 04hbYAeVG1AhSQ7JksyrAOW5LmitOvzd5GDXG3mRA+YaafDMU9O9oq7lmOWISG0CXikbIaloNx6 uM/JYjleBeF2EUY1ZVdVSQ4g4k6XpKg2cY487CazopPELOGiIIBREzkw5QZ9IeXfVCQ== X-Google-Smtp-Source: AK7set+eQk8TFpaj4iZC0gqPDX1eGLh/r6e1Wt71Upikfh0/CTp9327RHUJDZ+wlJ7NAArD0aMAeoYAElTFNPZg= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:903:2988:b0:19c:df4e:f3c2 with SMTP id lm8-20020a170903298800b0019cdf4ef3c2mr1655660plb.6.1677794579553; Thu, 02 Mar 2023 14:02:59 -0800 (PST) Date: Thu, 2 Mar 2023 22:02:46 +0000 In-Reply-To: <20230302215237.1473444-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230209000212.1892457-1-calvinwan@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog Message-ID: <20230302220251.1474923-1-calvinwan@google.com> Subject: [PATCH v9 1/6] run-command: add on_stderr_output_fn to run_processes_parallel_opts From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add duplicate_output_fn as an optionally set function in run_process_parallel_opts. If set, output from each child process is copied and passed to the callback function whenever output from the child process is buffered to allow for separate parsing. Fix two items in pp_buffer_stderr: * strbuf_read_once returns a ssize_t but the variable it is set to is an int so fix that. * Add missing brackets to "else if" statement The ungroup/duplicate_output incompatibility check is nested to prepare for future imcompatibles modes with ungroup. Signed-off-by: Calvin Wan --- run-command.c | 16 ++++++++++++--- run-command.h | 25 ++++++++++++++++++++++++ t/helper/test-run-command.c | 20 +++++++++++++++++++ t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++ 4 files changed, 97 insertions(+), 3 deletions(-) diff --git a/run-command.c b/run-command.c index 756f1839aa..7eed4e98c2 100644 --- a/run-command.c +++ b/run-command.c @@ -1526,6 +1526,11 @@ static void pp_init(struct parallel_processes *pp, if (!opts->get_next_task) BUG("you need to specify a get_next_task function"); + if (opts->ungroup) { + if (opts->on_stderr_output) + BUG("on_stderr_output and ungroup are incompatible with each other"); + } + CALLOC_ARRAY(pp->children, n); if (!opts->ungroup) CALLOC_ARRAY(pp->pfd, n); @@ -1645,14 +1650,19 @@ static void pp_buffer_stderr(struct parallel_processes *pp, for (size_t i = 0; i < opts->processes; i++) { if (pp->children[i].state == GIT_CP_WORKING && pp->pfd[i].revents & (POLLIN | POLLHUP)) { - int n = strbuf_read_once(&pp->children[i].err, - pp->children[i].process.err, 0); + ssize_t n = strbuf_read_once(&pp->children[i].err, + pp->children[i].process.err, 0); if (n == 0) { close(pp->children[i].process.err); pp->children[i].state = GIT_CP_WAIT_CLEANUP; - } else if (n < 0) + } else if (n < 0) { if (errno != EAGAIN) die_errno("read"); + } else if (opts->on_stderr_output) { + opts->on_stderr_output(&pp->children[i].err, + pp->children[i].err.len - n, + opts->data, pp->children[i].data); + } } } } diff --git a/run-command.h b/run-command.h index 072db56a4d..8f08e41fae 100644 --- a/run-command.h +++ b/run-command.h @@ -408,6 +408,25 @@ typedef int (*start_failure_fn)(struct strbuf *out, void *pp_cb, void *pp_task_cb); +/** + * This callback is called whenever output from a child process is buffered + * + * See run_processes_parallel() below for a discussion of the "struct + * strbuf *out" parameter. + * + * The offset refers to the number of bytes originally in "out" before + * the output from the child process was buffered. Therefore, the buffer + * range, "out + buf" to the end of "out", would contain the buffer of + * the child process output. + * + * pp_cb is the callback cookie as passed into run_processes_parallel, + * pp_task_cb is the callback cookie as passed into get_next_task_fn. + * + * This function is incompatible with "ungroup" + */ +typedef void (*on_stderr_output_fn)(struct strbuf *out, size_t offset, + void *pp_cb, void *pp_task_cb); + /** * This callback is called on every child process that finished processing. * @@ -461,6 +480,12 @@ struct run_process_parallel_opts */ start_failure_fn start_failure; + /** + * on_stderr_output: See on_stderr_output_fn() above. Unless you need + * to capture output from child processes, leave this as NULL. + */ + on_stderr_output_fn on_stderr_output; + /** * task_finished: See task_finished_fn() above. This can be * NULL to omit any special handling. diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c index 3ecb830f4a..a2fac6f762 100644 --- a/t/helper/test-run-command.c +++ b/t/helper/test-run-command.c @@ -52,6 +52,20 @@ static int no_job(struct child_process *cp, return 0; } +static void on_stderr_output(struct strbuf *out, + size_t offset, + void *pp_cb UNUSED, + void *pp_task_cb UNUSED) +{ + struct string_list list = STRING_LIST_INIT_DUP; + struct string_list_item *item; + + string_list_split(&list, out->buf + offset, '\n', -1); + for_each_string_list_item(item, &list) + fprintf(stderr, "on_stderr_output: %s\n", item->string); + string_list_clear(&list, 0); +} + static int task_finished(int result, struct strbuf *err, void *pp_cb, @@ -439,6 +453,12 @@ int cmd__run_command(int argc, const char **argv) opts.ungroup = 1; } + if (!strcmp(argv[1], "--on-stderr-output")) { + argv += 1; + argc -= 1; + opts.on_stderr_output = on_stderr_output; + } + jobs = atoi(argv[2]); strvec_clear(&proc.args); strvec_pushv(&proc.args, (const char **)argv + 3); diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh index e2411f6a9b..883d871dfb 100755 --- a/t/t0061-run-command.sh +++ b/t/t0061-run-command.sh @@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more jobs available than tasks --on-stderr-output' ' + test-tool run-command --on-stderr-output run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "on_stderr_output: Hello" err) && + test 4 = $(grep -c "on_stderr_output: World" err) && + sed "/on_stderr_output/d" err >err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' ' test-tool run-command --ungroup run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' ' test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with as many jobs as tasks --on-stderr-output' ' + test-tool run-command --on-stderr-output run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "on_stderr_output: Hello" err) && + test 4 = $(grep -c "on_stderr_output: World" err) && + sed "/on_stderr_output/d" err >err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' ' test-tool run-command --ungroup run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more tasks than jobs available --on-stderr-output' ' + test-tool run-command --on-stderr-output run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "on_stderr_output: Hello" err) && + test 4 = $(grep -c "on_stderr_output: World" err) && + sed "/on_stderr_output/d" err >err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' ' test-tool run-command --ungroup run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' ' test_cmp expect actual ' +test_expect_success 'run_command is asked to abort gracefully --on-stderr-output' ' + test-tool run-command --on-stderr-output run-command-abort 3 false >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command is asked to abort gracefully (ungroup)' ' test-tool run-command --ungroup run-command-abort 3 false >out 2>err && test_must_be_empty out && @@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' ' test_cmp expect actual ' +test_expect_success 'run_command outputs --on-stderr-output' ' + test-tool run-command --on-stderr-output run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command outputs (ungroup) ' ' test-tool run-command --ungroup run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_must_be_empty out && From patchwork Thu Mar 2 22:02:47 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13157944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 67D45C678D4 for ; Thu, 2 Mar 2023 22:03:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229748AbjCBWDd (ORCPT ); Thu, 2 Mar 2023 17:03:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229738AbjCBWD3 (ORCPT ); Thu, 2 Mar 2023 17:03:29 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 915B05A6E6 for ; Thu, 2 Mar 2023 14:03:06 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id z2-20020a170903018200b0019cfc0a566eso315903plg.15 for ; Thu, 02 Mar 2023 14:03:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=0w0JSky0wue+bAdzXqkRUkKqNS3LfA2qZNiuvavmOKs=; b=FT8LU7CDVcvOCheoqOsPARuU6Vlf61QiwYXNte3tKd2qScPPDuJLSvUW/SOZHUN6Ao SVGr33ybXEiqnuv1JpiLnNAGw6sYdw52m4NLtveUU+ej+TLnWcW6YCC6cwCBNAlJqD0i 21Z/mBsDjkr7cfYj71PnsW6j06Ij5VE+nEDoo9QNK2p57622XfYEh5df4AmJE4qca3BZ PoFgOYD2Qz6koE5R+K2mjRcx7yPfkeLUGUGWLiVtVyCKv0gUkV+7KgRXIDmxod90Zeye hBTli82brVXrsEA2/+PiXrmRcOWw+QgWfYXSgyrQImEcWLXCMNGi+JjDHnZ232eEFOnr VaVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0w0JSky0wue+bAdzXqkRUkKqNS3LfA2qZNiuvavmOKs=; b=kXeE5d9CNgQeAj0NZWOgfCfmOFvH20gTW/qZrEdWO9wWoRApEGDYhWDt/yhUHmTTuE ZSI+kYgCIxz7LGNr++nar5W9eZ2V7aoJnjOYplKcrPO6XeDrCbz6LIVw/KTgOx3M8lIr kAqHwFFTn6dC5SyBEXI3wEC6dTHix2EDEwakNQjbp+HE6Mf4+1kYaAS0yWazmQt5raUo 9CAEYhBVtHlD4XJ71B3W7KRx/zHPTfBau+GXETwnyUi8hvZmXEYnCthTDCWeWU+82G4i +2PNUCc9RDM6F2L+1dGVuwABss5WA+94rYfHXpEvfkAlO1eSprulKmae8gcCCLMKPH/N nEZg== X-Gm-Message-State: AO0yUKU0v4Q+YZxsKAbFHs5ZGprlnYI8xZgb+uH9BuXH6gS4DImGRyvd RLaLCI2Wu+U7LGwp8t0oyP/OjdIG9UKX+qpRIm6lNBGdvk+gLu1BQr9QZeWOFR/YEl+7k6qjZbn YeN7BuVrPBcrjTR1mQKUlvl5/ZhqD211uKaASkFsptVCMTu6fXkRTw/OFHtZ3IjLbGw== X-Google-Smtp-Source: AK7set8FSf7pLL0qSkgoDPz606uOzrHp7AVPqrY94XEK9IbnY/32KazuH7w5sbjo7ZXb/PqSFTVw8fumGcehkaQ= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a62:dbc5:0:b0:5df:9809:621f with SMTP id f188-20020a62dbc5000000b005df9809621fmr22598pfg.0.1677794581418; Thu, 02 Mar 2023 14:03:01 -0800 (PST) Date: Thu, 2 Mar 2023 22:02:47 +0000 In-Reply-To: <20230302215237.1473444-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230209000212.1892457-1-calvinwan@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog Message-ID: <20230302220251.1474923-2-calvinwan@google.com> Subject: [PATCH v9 2/6] submodule: rename strbuf variable From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A prepatory change for a future patch that moves the status parsing logic to a separate function. Signed-off-by: Calvin Wan --- submodule.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/submodule.c b/submodule.c index fae24ef34a..faf37c1101 100644 --- a/submodule.c +++ b/submodule.c @@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) fp = xfdopen(cp.out, "r"); while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { + char *str = buf.buf; + const size_t len = buf.len; + /* regular untracked files */ - if (buf.buf[0] == '?') + if (str[0] == '?') dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '1' || - buf.buf[0] == '2') { + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { /* T = line type, XY = status, SSSS = submodule state */ - if (buf.len < strlen("T XY SSSS")) + if (len < strlen("T XY SSSS")) BUG("invalid status --porcelain=2 line %s", - buf.buf); + str); - if (buf.buf[5] == 'S' && buf.buf[8] == 'U') + if (str[5] == 'S' && str[8] == 'U') /* nested untracked file */ dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '2' || - memcmp(buf.buf + 5, "S..U", 4)) + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) /* other change */ dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; } From patchwork Thu Mar 2 22:02:48 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13157945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D1C2AC678D4 for ; Thu, 2 Mar 2023 22:03:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229854AbjCBWDg (ORCPT ); Thu, 2 Mar 2023 17:03:36 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46840 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229754AbjCBWD3 (ORCPT ); Thu, 2 Mar 2023 17:03:29 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F0D945ADC5 for ; Thu, 2 Mar 2023 14:03:06 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id 10-20020a63030a000000b004fb64e929f2so99264pgd.7 for ; Thu, 02 Mar 2023 14:03:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1CK1+XmaOIwxsuu8wDVNqSp47NjVbJnFfkOEMkQvjBI=; b=lZdnDAsh/sK4GlhEaYZslpw9UQJOzZv5RceHTo4eCjxBq+WQd5pMV1wiZ2GbEnBgmw cQo3HZ/Eyq8dbpVXbIyZlr/einxQWpJFrrAvZ1Uzxuva1D16HKmhdq+AULFFbrbFNIOj BniFkhqwdXk1UPf967xvEZ5nqLIRB2m81SN/HPCMEjXysjF8Ylfktkmf3e4BPmuaWuQl eVTyRaBxtRIpvQMdjjAIplLos1nUKx+2MLXamuj2v89Vfd45QMpLujy7fNFLcuBbOSMm 7ci7t7bQvYfspJf4zNXAch5/s+zwLT3LEjBao7sLZPgg/+QWry692tUv5i0BHeIFXbq+ GRaw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1CK1+XmaOIwxsuu8wDVNqSp47NjVbJnFfkOEMkQvjBI=; b=AxnB3kIuAxW1I8nWmPxY5AsAW7gMHkaj8k2ZUGFolOhd/4aBhZQmmnPx9HBpW6E2UI rN57wsqov9HBJQ/3jdm+5Cf4LiSCXv9wHhowT0l6SpbbBao7CJm5eP+iOkF4ZND8Gtnv zmMBzANLOQ9klLlsOQ0UH9pV96+VmNiPeCMkQwO0i9cUpIfJ4IA+jsVMhLTF30og9JsS XYTFeIDunhmyKwZWsb8PRlrCHvJsjmpwmIglnZjWMed6bFbXPVSZC95CxW0pCq0d4ASs pVlPhMzHMrqklVWTtNA+NnJIExs+fDXUH3JbI4qTlsrInaVO02kcKjzOWopPVa5qbUHK KFZg== X-Gm-Message-State: AO0yUKXX3oawHfqhSYh7eTWhOlI4FuKSO05r4lUMUqTQJI7fEKNqlQ01 xmD4qirbO/9qyvZBbHtO9jJpxVI/9iU6w7KDQMU3HC4RempXLkhwJOBhFfJ05pvNC8pfiNVtw4i 2L+XYUdmjnfLvTY3KofJv5d1hB/cXsDtJPE5cnfTD9NLzMS1yIW/HLIUYj3AxrKUIpA== X-Google-Smtp-Source: AK7set9T+BPC6d7LmWiCjmN/lqHWbfogfe4jdmv+Qy+AkYGnh1WOplNv/4QtaHBHQNhXhmQtRqy73/8ZaeMM9MM= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a05:6a00:2254:b0:5de:ece4:2674 with SMTP id i20-20020a056a00225400b005deece42674mr4058pfu.3.1677794582912; Thu, 02 Mar 2023 14:03:02 -0800 (PST) Date: Thu, 2 Mar 2023 22:02:48 +0000 In-Reply-To: <20230302215237.1473444-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230209000212.1892457-1-calvinwan@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog Message-ID: <20230302220251.1474923-3-calvinwan@google.com> Subject: [PATCH v9 3/6] submodule: move status parsing into function From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A future patch requires the ability to parse the output of git status --porcelain=2. Move parsing code from is_submodule_modified to parse_status_porcelain. Signed-off-by: Calvin Wan --- submodule.c | 74 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 32 deletions(-) diff --git a/submodule.c b/submodule.c index faf37c1101..768d4b4cd7 100644 --- a/submodule.c +++ b/submodule.c @@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r, return spf.result; } +static int parse_status_porcelain(char *str, size_t len, + unsigned *dirty_submodule, + int ignore_untracked) +{ + /* regular untracked files */ + if (str[0] == '?') + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { + /* T = line type, XY = status, SSSS = submodule state */ + if (len < strlen("T XY SSSS")) + BUG("invalid status --porcelain=2 line %s", + str); + + if (str[5] == 'S' && str[8] == 'U') + /* nested untracked file */ + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) + /* other change */ + *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; + } + + if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && + ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || + ignore_untracked)) { + /* + * We're not interested in any further information from + * the child any more, neither output nor its exit code. + */ + return 1; + } + return 0; +} + unsigned is_submodule_modified(const char *path, int ignore_untracked) { struct child_process cp = CHILD_PROCESS_INIT; @@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) char *str = buf.buf; const size_t len = buf.len; - /* regular untracked files */ - if (str[0] == '?') - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '1' || - str[0] == '2') { - /* T = line type, XY = status, SSSS = submodule state */ - if (len < strlen("T XY SSSS")) - BUG("invalid status --porcelain=2 line %s", - str); - - if (str[5] == 'S' && str[8] == 'U') - /* nested untracked file */ - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '2' || - memcmp(str + 5, "S..U", 4)) - /* other change */ - dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; - } - - if ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && - ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || - ignore_untracked)) { - /* - * We're not interested in any further information from - * the child any more, neither output nor its exit code. - */ - ignore_cp_exit_code = 1; + ignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule, + ignore_untracked); + if (ignore_cp_exit_code) break; - } } fclose(fp); From patchwork Thu Mar 2 22:02:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13157947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3F706C678D4 for ; Thu, 2 Mar 2023 22:03:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229849AbjCBWDj (ORCPT ); Thu, 2 Mar 2023 17:03:39 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46916 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229803AbjCBWDb (ORCPT ); Thu, 2 Mar 2023 17:03:31 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 80F4A5942B for ; Thu, 2 Mar 2023 14:03:07 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id s21-20020a632155000000b004fc1f5c8a4dso98885pgm.11 for ; Thu, 02 Mar 2023 14:03:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RVcHRxuNeHDNfJVOwxlgeyrSJLnyl430+ZUW4qkzWu8=; b=YWTLwywQ8hmVaq3dZ+pyQUDBX1Vqo9QRv13ZzLZQ+Z19W6aAt3XEIsIAr4ro+I6G4f D4mdgfgITnjmq4OLHOGai5Wq56OwZD0MaYHWLc+Av+eK+MCnmqPs0HAxBo2liIcMRa07 UyUQO44LlZhO2ys6t7+ylB5g3qCEoQiogJruWD82C8L0ZwG4luh5XdjHQXNxKM4ylkUd 8nRvMBLGy6+Qi7RA+BU4F3Vg2uKlaG4/t/VITYebT102sNhPRc1SfRiiDA4rKAkukVmA i654W2tP2ljOLJOUJjjvPMNaDIf0wdIKLrwS1ycIdjkeyQbOYDp2UkOa6x98Wa+rp61Y TyPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RVcHRxuNeHDNfJVOwxlgeyrSJLnyl430+ZUW4qkzWu8=; b=At5oVQo1MkaRnPywrDCe1dL9Lazy0wvWuZguZt4e5320JdqdKlYSb7eSE5nluA/5pq GzmrdYzLkWl8AOYT7Be68wAWgfVJTM5PXO8XXhaQTy3I6Sq0mLvfRTlW/nHAzq839GRn TgJM/p5/m5Upx6ssx0kfMnnJeN+SrbQj7WKAVr1Ry8I+Bsof4jrgtp6XpqyEtYWvqjjU ttDb+0GYNSyVW1ygInrOgppIj9O0e0j44IutYFGuOuUn7STnRBIlBAG8gbz74zAT5IwP 0DPx3TJxKVCxInhcpVkbnsHwsdIj8SiaYBgAg+iQZCla6sp/jy07nfoDFrqP5r1IFCAU BSmQ== X-Gm-Message-State: AO0yUKXMw2Om8MZ7w+uGQGLbGyg+ZbigCi4MbReSVW4vCJU45cOXReR8 mSJdViQcErXewe2qJFki62hau0zEcW8OsOqs0IOskYLUCmSF1BRCsw6NTzzI3dMdaZ8ram36kvw 40hQ9eXdQ2LGzgX0kCL30jBR5Sttxcsy+tbOu5RJKiheQvBOmF5yro9c+T9hl4iHTQA== X-Google-Smtp-Source: AK7set/7VSfxIxjtbPHxwH3p3n6nrbgFLzKLnzIJDWPi1A0Dwh9gTQKN5jHYxC4MRj+UvpyypIQeoZRwHZ7ogbU= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a63:788c:0:b0:503:20b9:5d88 with SMTP id t134-20020a63788c000000b0050320b95d88mr3932945pgc.4.1677794584449; Thu, 02 Mar 2023 14:03:04 -0800 (PST) Date: Thu, 2 Mar 2023 22:02:49 +0000 In-Reply-To: <20230302215237.1473444-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230209000212.1892457-1-calvinwan@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog Message-ID: <20230302220251.1474923-4-calvinwan@google.com> Subject: [PATCH v9 4/6] submodule: refactor is_submodule_modified() From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor out submodule status logic and error messages that will be used in a future patch. Signed-off-by: Calvin Wan --- submodule.c | 65 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 42 insertions(+), 23 deletions(-) diff --git a/submodule.c b/submodule.c index 768d4b4cd7..426074cebb 100644 --- a/submodule.c +++ b/submodule.c @@ -28,6 +28,10 @@ static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF; static int initialized_fetch_ref_tips; static struct oid_array ref_tips_before_fetch; static struct oid_array ref_tips_after_fetch; +#define STATUS_PORCELAIN_START_ERROR \ + N_("could not run 'git status --porcelain=2' in submodule %s") +#define STATUS_PORCELAIN_FAIL_ERROR \ + N_("'git status --porcelain=2' failed in submodule %s") /* * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file @@ -1870,6 +1874,40 @@ int fetch_submodules(struct repository *r, return spf.result; } +static int verify_submodule_git_directory(const char *path) +{ + const char *git_dir; + struct strbuf buf = STRBUF_INIT; + + strbuf_addf(&buf, "%s/.git", path); + git_dir = read_gitfile(buf.buf); + if (!git_dir) + git_dir = buf.buf; + if (!is_git_directory(git_dir)) { + if (is_directory(git_dir)) + die(_("'%s' not recognized as a git repository"), git_dir); + strbuf_release(&buf); + /* The submodule is not checked out, so it is not modified */ + return 0; + } + strbuf_release(&buf); + return 1; +} + +static void prepare_status_porcelain(struct child_process *cp, + const char *path, int ignore_untracked) +{ + strvec_pushl(&cp->args, "status", "--porcelain=2", NULL); + if (ignore_untracked) + strvec_push(&cp->args, "-uno"); + + prepare_submodule_repo_env(&cp->env); + cp->git_cmd = 1; + cp->no_stdin = 1; + cp->out = -1; + cp->dir = path; +} + static int parse_status_porcelain(char *str, size_t len, unsigned *dirty_submodule, int ignore_untracked) @@ -1915,33 +1953,14 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) struct strbuf buf = STRBUF_INIT; FILE *fp; unsigned dirty_submodule = 0; - const char *git_dir; int ignore_cp_exit_code = 0; - strbuf_addf(&buf, "%s/.git", path); - git_dir = read_gitfile(buf.buf); - if (!git_dir) - git_dir = buf.buf; - if (!is_git_directory(git_dir)) { - if (is_directory(git_dir)) - die(_("'%s' not recognized as a git repository"), git_dir); - strbuf_release(&buf); - /* The submodule is not checked out, so it is not modified */ + if (!verify_submodule_git_directory(path)) return 0; - } - strbuf_reset(&buf); - - strvec_pushl(&cp.args, "status", "--porcelain=2", NULL); - if (ignore_untracked) - strvec_push(&cp.args, "-uno"); - prepare_submodule_repo_env(&cp.env); - cp.git_cmd = 1; - cp.no_stdin = 1; - cp.out = -1; - cp.dir = path; + prepare_status_porcelain(&cp, path, ignore_untracked); if (start_command(&cp)) - die(_("Could not run 'git status --porcelain=2' in submodule %s"), path); + die(_(STATUS_PORCELAIN_START_ERROR), path); fp = xfdopen(cp.out, "r"); while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { @@ -1956,7 +1975,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) fclose(fp); if (finish_command(&cp) && !ignore_cp_exit_code) - die(_("'git status --porcelain=2' failed in submodule %s"), path); + die(_(STATUS_PORCELAIN_FAIL_ERROR), path); strbuf_release(&buf); return dirty_submodule; From patchwork Thu Mar 2 22:02:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13157946 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D474C7EE30 for ; Thu, 2 Mar 2023 22:03:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229867AbjCBWDi (ORCPT ); Thu, 2 Mar 2023 17:03:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46926 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229819AbjCBWDc (ORCPT ); Thu, 2 Mar 2023 17:03:32 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B9C045ADCC for ; Thu, 2 Mar 2023 14:03:07 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id u8-20020a17090ae00800b00237e4f46c8bso108395pjy.7 for ; Thu, 02 Mar 2023 14:03:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=c+ESOsaWLLSTKe9690Yw7UnD2ULze1EdKUa4KMNoPRg=; b=m20VBXnZBWux03ONvmIWOtcg9PDYjX457i/4vnnIR7LxJk3K+0JyiFfnQVRj+RaNlw iSUtL90C0WHAcK/w4mgR+lyOkmNay8PrNbbHjhuzf9o6vHNGCdsf3rLhujll+vdn2Jgm 6C7Of+87I7YFCVmv1rBpZM7E41+14Aq/aFQtntf43MaxIkVPGKh1eSD7h6C3TAyhzgYc BvjEnVZbh7+NoHfM/j1U40ipdYRpmGpKsuLDk4tNBWs/mW8fwOWgc7bss252eGY5g8En m3lwXLH4Rxx8ui/K+9usiC1A0Nn1KsF/hCfS6vpz8EoRhwXsdDc7SlR6SxNWhSxR/pPx soFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=c+ESOsaWLLSTKe9690Yw7UnD2ULze1EdKUa4KMNoPRg=; b=HpZuek3H4WuP+vCmc1FcTSqbce6De5fHT3SZ/fXpW3TvSEQTkMANA8tSfpjobuX4x7 fyqYed8eVKUcPsTc326JorA2DUKeeXqoXNTR/AUCORjtbmnPgsCz9LDjKdfxpgmRRBwG V5/jUJxMiufEbBSkFLVh73gbwLTuCkSghd2B7I9xXyO2jGwBT0INsujtpXVsayM4uQzj ZKf5PE1oODTJP1h6MnX5crrZavmWlNBk1l91/XBPH6H+dPjpWKwF5lhvkQNhS+xIZ+hY sYnW2vuPV2aSJzcHv9BBsFEsRw9RfHTmqDtEFs/e1SlbD8TIJ/sXBSYNJ/aYTBT4TVof CHjg== X-Gm-Message-State: AO0yUKUV1cvv/+BzTBhMBws6rpBoa8GzyUOApTT0GSV7DvVhea4FcOPx ClzhSCJHt878uwehOecqFLdQnfdBR+dEGCjZfWVwANI0li1y3PquLjLWjcNV5aDrQbbcl3BTkdT fMqi/gojjyyg+eyLHYmXVVob/4zHWcVKOfVxLXy3WjrhZ721rywKbjpn1vyRDiURb/w== X-Google-Smtp-Source: AK7set/ktoPJtAo1LIajyg9KFL+LdZr43oBgWtQAYK2fSz5P4+q6+oKH7+xLq06LQ37UJYrUDFu7+q04qYs5zgQ= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a63:6fce:0:b0:4fb:b3d4:864d with SMTP id k197-20020a636fce000000b004fbb3d4864dmr3847272pgc.4.1677794586269; Thu, 02 Mar 2023 14:03:06 -0800 (PST) Date: Thu, 2 Mar 2023 22:02:50 +0000 In-Reply-To: <20230302215237.1473444-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230209000212.1892457-1-calvinwan@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog Message-ID: <20230302220251.1474923-5-calvinwan@google.com> Subject: [PATCH v9 5/6] diff-lib: refactor out diff_change logic From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In run_diff_files, there is logic that records the diff and updates relevant bits at the end of each entry iteration. Refactor out that logic into a helper function so a future patch can call it. Signed-off-by: Calvin Wan --- diff-lib.c | 48 +++++++++++++++++++++++++++++++----------------- 1 file changed, 31 insertions(+), 17 deletions(-) diff --git a/diff-lib.c b/diff-lib.c index dec040c366..744ae98a69 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -88,6 +88,34 @@ static int match_stat_with_submodule(struct diff_options *diffopt, return changed; } +/** + * Records diff_change if there is a change in the entry from run_diff_files. + * If there is no change, then the cache entry is marked CE_UPTODATE and + * CE_FSMONITOR_VALID. If there is no change and the find_copies_harder flag + * is not set, then the function returns early. + */ +static void record_file_diff(struct diff_options *options, unsigned newmode, + unsigned dirty_submodule, int changed, + struct index_state *istate, + struct cache_entry *ce) +{ + unsigned int oldmode; + const struct object_id *old_oid, *new_oid; + + if (!changed && !dirty_submodule) { + ce_mark_uptodate(ce); + mark_fsmonitor_valid(istate, ce); + if (!options->flags.find_copies_harder) + return; + } + oldmode = ce->ce_mode; + old_oid = &ce->oid; + new_oid = changed ? null_oid() : &ce->oid; + diff_change(options, oldmode, newmode, old_oid, new_oid, + !is_null_oid(old_oid), !is_null_oid(new_oid), + ce->name, 0, dirty_submodule); +} + int run_diff_files(struct rev_info *revs, unsigned int option) { int entries, i; @@ -105,11 +133,10 @@ int run_diff_files(struct rev_info *revs, unsigned int option) diff_unmerged_stage = 2; entries = istate->cache_nr; for (i = 0; i < entries; i++) { - unsigned int oldmode, newmode; + unsigned int newmode; struct cache_entry *ce = istate->cache[i]; int changed; unsigned dirty_submodule = 0; - const struct object_id *old_oid, *new_oid; if (diff_can_quit_early(&revs->diffopt)) break; @@ -245,21 +272,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option) newmode = ce_mode_from_stat(ce, st.st_mode); } - if (!changed && !dirty_submodule) { - ce_mark_uptodate(ce); - mark_fsmonitor_valid(istate, ce); - if (!revs->diffopt.flags.find_copies_harder) - continue; - } - oldmode = ce->ce_mode; - old_oid = &ce->oid; - new_oid = changed ? null_oid() : &ce->oid; - diff_change(&revs->diffopt, oldmode, newmode, - old_oid, new_oid, - !is_null_oid(old_oid), - !is_null_oid(new_oid), - ce->name, 0, dirty_submodule); - + record_file_diff(&revs->diffopt, newmode, dirty_submodule, + changed, istate, ce); } diffcore_std(&revs->diffopt); diff_flush(&revs->diffopt); From patchwork Thu Mar 2 22:02:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13157948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9C9FEC678D4 for ; Thu, 2 Mar 2023 22:03:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229886AbjCBWDt (ORCPT ); Thu, 2 Mar 2023 17:03:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:46952 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229862AbjCBWDh (ORCPT ); Thu, 2 Mar 2023 17:03:37 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2EA2C65AA for ; Thu, 2 Mar 2023 14:03:11 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-536be78056eso4648497b3.1 for ; Thu, 02 Mar 2023 14:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TBQtc7oLap20tvuHKP0ZsC95XKy4EHg1LfrEHNOYvL4=; b=cG54kCIQIXARVsj0oJ+zjeq7TmlF7/iRVQj0yjx9s4tW8vleUnWd9BUnMq1poqS9x2 LXCI374+vpXVyyESSExSgziEX/dNknGMa71gTPtXeYQl1Ktet8NFInh1Ng+jdXn4B4IV FL8mL4y6UcORz+yBlAH5tpYFzaSuQCmprj9G37S4vN1XaDod7alNGwDoga4yFSv2aXRz KPibE1mjojGrBsgozfoF7lgiyUXPhbCPh4jZuTJoKt1m6hvwpX2oEnddwwdJ1ix/+pEK 8tpUX1/+4EIgQ7Kr8grdxnEBxZda6rk8ouXKBSZXvw0pwPpqYilSPhnVzb35AeOEoPlf YxUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TBQtc7oLap20tvuHKP0ZsC95XKy4EHg1LfrEHNOYvL4=; b=FWLiyFQ7O/2rMS0VQqqQp1i57Dk/XXBEJ+WtE2/noC9lOu8kHb3q86F1zQWNAJ5Ukd EV2OBLCmBv8j6Znt+YwviT3inRqrOt4VfpbPcDq6iklOqwtnh4qT0N5j4OvIrHIasZKD yqb+xntaaWiFjAqImvAMovIRffVoA5UkVjsr4SMWVazrM1Ef2OlyH6jL5aBFgpRnPASM HHWhPmWPjxtSJS0Msbj0PDnYpZ/JvDeDy1IBYNEoL/nzZj6P6Vz9fqa/qHcnnXytFSld HkWJB41tpwStutFD+HMs8gYZ3f0mOl0n3eICWXh5omdva1wK0VD+XyBzDbESEW6IHGht W4nA== X-Gm-Message-State: AO0yUKV0awdX4FRDsBpT9PAqZuIdZLezqvSzIGxBu45Rf42caYn8aEPR widhYM2pY7V2qs3jzIWqBNqXrJyXqGtzU0WowGgOqKaA8WhmP8+iNdDyyMopdqj6Zd85B7dSdPV 4NqbnMJDN/+A2oTSWoDdLLGSGuoLJCH6ds4Plz2FxQmOEW3K1lRZVzwkSCEFIy3UMZA== X-Google-Smtp-Source: AK7set9gLruO9PDg73uetafY97F079sUJrI5V6sR1xDOjFaZOsgHky+XAmQ9Xx677xTVNMLpNns27htu0DKhts8= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a81:4416:0:b0:533:9ffb:cb13 with SMTP id r22-20020a814416000000b005339ffbcb13mr7139045ywa.3.1677794587894; Thu, 02 Mar 2023 14:03:07 -0800 (PST) Date: Thu, 2 Mar 2023 22:02:51 +0000 In-Reply-To: <20230302215237.1473444-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230209000212.1892457-1-calvinwan@google.com> X-Mailer: git-send-email 2.40.0.rc0.216.gc4246ad0f0-goog Message-ID: <20230302220251.1474923-6-calvinwan@google.com> Subject: [PATCH v9 6/6] diff-lib: parallelize run_diff_files for submodules From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During the iteration of the index entries in run_diff_files, whenever a submodule is found and needs its status checked, a subprocess is spawned for it. Instead of spawning the subprocess immediately and waiting for its completion to continue, hold onto all submodules and relevant information in a list. Then use that list to create tasks for run_processes_parallel. Subprocess output is passed to status_on_stderr_output which stores it to be parsed on completion of the subprocess. Add config option submodule.diffJobs to set the maximum number of parallel jobs. The option defaults to 1 if unset. If set to 0, the number of jobs is set to online_cpus(). Since run_diff_files is called from many different commands, I chose to grab the config option in the function rather than adding variables to every git command and then figuring out how to pass them all in. Signed-off-by: Calvin Wan --- Documentation/config/submodule.txt | 12 +++ diff-lib.c | 81 +++++++++++++++--- submodule.c | 128 +++++++++++++++++++++++++++++ submodule.h | 9 ++ t/t4027-diff-submodule.sh | 31 +++++++ t/t7506-status-submodule.sh | 25 ++++++ 6 files changed, 274 insertions(+), 12 deletions(-) diff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt index 6490527b45..3209eb8117 100644 --- a/Documentation/config/submodule.txt +++ b/Documentation/config/submodule.txt @@ -93,6 +93,18 @@ submodule.fetchJobs:: in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1. +submodule.diffJobs:: + Specifies how many submodules are diffed at the same time. A + positive integer allows up to that number of submodules diffed + in parallel. A value of 0 will give some reasonable default. + If unset, it defaults to 1. The diff operation is used by many + other git commands such as add, merge, diff, status, stash and + more. Note that the expensive part of the diff operation is + reading the index from cache or memory. Therefore multiple jobs + may be detrimental to performance if your hardware does not + support parallel reads or if the number of jobs greatly exceeds + the amount of supported reads. + submodule.alternateLocation:: Specifies how the submodules obtain alternates when submodules are cloned. Possible values are `no`, `superproject`. diff --git a/diff-lib.c b/diff-lib.c index 744ae98a69..7fe6ced950 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -14,6 +14,7 @@ #include "dir.h" #include "fsmonitor.h" #include "commit-reach.h" +#include "config.h" /* * diff-files @@ -65,26 +66,41 @@ static int check_removed(const struct index_state *istate, const struct cache_en * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES * option is set, the caller does not only want to know if a submodule is * modified at all but wants to know all the conditions that are met (new - * commits, untracked content and/or modified content). + * commits, untracked content and/or modified content). If + * defer_submodule_status bit is set, dirty_submodule will be left to the + * caller to set. defer_submodule_status can also be set to 0 in this + * function if there is no need to check if the submodule is modified. */ static int match_stat_with_submodule(struct diff_options *diffopt, const struct cache_entry *ce, struct stat *st, unsigned ce_option, - unsigned *dirty_submodule) + unsigned *dirty_submodule, int *defer_submodule_status, + unsigned *ignore_untracked) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); + int defer = 0; + if (S_ISGITLINK(ce->ce_mode)) { struct diff_flags orig_flags = diffopt->flags; if (!diffopt->flags.override_submodule_config) set_diffopt_flags_from_submodule_config(diffopt, ce->name); - if (diffopt->flags.ignore_submodules) + if (diffopt->flags.ignore_submodules) { changed = 0; - else if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, - diffopt->flags.ignore_untracked_in_submodules); + } else if (!diffopt->flags.ignore_dirty_submodules && + (!changed || diffopt->flags.dirty_submodules)) { + if (defer_submodule_status && *defer_submodule_status) { + defer = 1; + *ignore_untracked = diffopt->flags.ignore_untracked_in_submodules; + } else { + *dirty_submodule = is_submodule_modified(ce->name, + diffopt->flags.ignore_untracked_in_submodules); + } + } diffopt->flags = orig_flags; } + + if (defer_submodule_status) + *defer_submodule_status = defer; return changed; } @@ -124,6 +140,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ? CE_MATCH_RACY_IS_DIRTY : 0); uint64_t start = getnanotime(); struct index_state *istate = revs->diffopt.repo->index; + struct string_list submodules = STRING_LIST_INIT_NODUP; diff_set_mnemonic_prefix(&revs->diffopt, "i/", "w/"); @@ -136,7 +153,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option) unsigned int newmode; struct cache_entry *ce = istate->cache[i]; int changed; - unsigned dirty_submodule = 0; + int defer_submodule_status = 1; if (diff_can_quit_early(&revs->diffopt)) break; @@ -247,6 +264,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option) newmode = ce->ce_mode; } else { struct stat st; + unsigned ignore_untracked = 0; changed = check_removed(istate, ce, &st); if (changed) { @@ -268,13 +286,52 @@ int run_diff_files(struct rev_info *revs, unsigned int option) } changed = match_stat_with_submodule(&revs->diffopt, ce, &st, - ce_option, &dirty_submodule); + ce_option, NULL, + &defer_submodule_status, + &ignore_untracked); newmode = ce_mode_from_stat(ce, st.st_mode); + if (defer_submodule_status) { + struct submodule_status_util tmp = { + .changed = changed, + .dirty_submodule = 0, + .ignore_untracked = ignore_untracked, + .newmode = newmode, + .ce = ce, + .path = ce->name, + }; + struct string_list_item *item; + + item = string_list_append(&submodules, ce->name); + item->util = xmalloc(sizeof(tmp)); + memcpy(item->util, &tmp, sizeof(tmp)); + continue; + } } - record_file_diff(&revs->diffopt, newmode, dirty_submodule, - changed, istate, ce); + if (!defer_submodule_status) + record_file_diff(&revs->diffopt, newmode, 0, + changed,istate, ce); + } + if (submodules.nr) { + unsigned long parallel_jobs; + struct string_list_item *item; + + if (git_config_get_ulong("submodule.diffjobs", ¶llel_jobs)) + parallel_jobs = 1; + else if (!parallel_jobs) + parallel_jobs = online_cpus(); + + if (get_submodules_status(&submodules, parallel_jobs)) + die(_("submodule status failed")); + for_each_string_list_item(item, &submodules) { + struct submodule_status_util *util = item->util; + + record_file_diff(&revs->diffopt, util->newmode, + util->dirty_submodule, util->changed, + istate, util->ce); + } } + string_list_clear(&submodules, 1); diffcore_std(&revs->diffopt); diff_flush(&revs->diffopt); trace_performance_since(start, "diff-files"); @@ -322,7 +379,7 @@ static int get_stat_data(const struct index_state *istate, return -1; } changed = match_stat_with_submodule(diffopt, ce, &st, - 0, dirty_submodule); + 0, dirty_submodule, NULL, NULL); if (changed) { mode = ce_mode_from_stat(ce, st.st_mode); oid = null_oid(); diff --git a/submodule.c b/submodule.c index 426074cebb..6f6e150a3f 100644 --- a/submodule.c +++ b/submodule.c @@ -1373,6 +1373,13 @@ int submodule_touches_in_range(struct repository *r, return ret; } +struct submodule_parallel_status { + size_t index_count; + int result; + + struct string_list *submodule_names; +}; + struct submodule_parallel_fetch { /* * The index of the last index entry processed by @@ -1455,6 +1462,12 @@ struct fetch_task { struct oid_array *commits; /* Ensure these commits are fetched */ }; +struct status_task { + const char *path; + struct strbuf out; + int ignore_untracked; +}; + /** * When a submodule is not defined in .gitmodules, we cannot access it * via the regular submodule-config. Create a fake submodule, which we can @@ -1981,6 +1994,121 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) return dirty_submodule; } +static struct status_task * +get_status_task_from_index(struct submodule_parallel_status *sps, + struct strbuf *err) +{ + for (; sps->index_count < sps->submodule_names->nr; sps->index_count++) { + struct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util; + struct status_task *task; + + if (!verify_submodule_git_directory(util->path)) + continue; + + task = xmalloc(sizeof(*task)); + task->path = util->path; + task->ignore_untracked = util->ignore_untracked; + strbuf_init(&task->out, 0); + sps->index_count++; + return task; + } + return NULL; +} + +static int get_next_submodule_status(struct child_process *cp, + struct strbuf *err, void *data, + void **task_cb) +{ + struct submodule_parallel_status *sps = data; + struct status_task *task = get_status_task_from_index(sps, err); + + if (!task) + return 0; + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env); + prepare_status_porcelain(cp, task->path, task->ignore_untracked); + *task_cb = task; + return 1; +} + +static int status_start_failure(struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + + sps->result = 1; + strbuf_addf(err, _(STATUS_PORCELAIN_START_ERROR), task->path); + return 0; +} + +static void status_on_stderr_output(struct strbuf *out, + size_t offset, + void *cb, void *task_cb) +{ + struct status_task *task = task_cb; + + strbuf_add(&task->out, out->buf + offset, out->len - offset); + strbuf_setlen(out, offset); +} + +static int status_finish(int retvalue, struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + struct string_list_item *it = + string_list_lookup(sps->submodule_names, task->path); + struct submodule_status_util *util = it->util; + struct string_list list = STRING_LIST_INIT_DUP; + struct string_list_item *item; + + if (retvalue) { + sps->result = 1; + strbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path); + } + + string_list_split(&list, task->out.buf, '\n', -1); + for_each_string_list_item(item, &list) { + if (parse_status_porcelain(item->string, + strlen(item->string), + &util->dirty_submodule, + util->ignore_untracked)) + break; + } + string_list_clear(&list, 0); + strbuf_release(&task->out); + free(task); + + return 0; +} + +int get_submodules_status(struct string_list *submodules, + int max_parallel_jobs) +{ + struct submodule_parallel_status sps = { + .submodule_names = submodules, + }; + const struct run_process_parallel_opts opts = { + .tr2_category = "submodule", + .tr2_label = "parallel/status", + + .processes = max_parallel_jobs, + + .get_next_task = get_next_submodule_status, + .start_failure = status_start_failure, + .on_stderr_output = status_on_stderr_output, + .task_finished = status_finish, + .data = &sps, + }; + + string_list_sort(sps.submodule_names); + run_processes_parallel(&opts); + + return sps.result; +} + int submodule_uses_gitfile(const char *path) { struct child_process cp = CHILD_PROCESS_INIT; diff --git a/submodule.h b/submodule.h index b52a4ff1e7..08d278a414 100644 --- a/submodule.h +++ b/submodule.h @@ -41,6 +41,13 @@ struct submodule_update_strategy { .type = SM_UPDATE_UNSPECIFIED, \ } +struct submodule_status_util { + int changed, ignore_untracked; + unsigned dirty_submodule, newmode; + struct cache_entry *ce; + const char *path; +}; + int is_gitmodules_unmerged(struct index_state *istate); int is_writing_gitmodules_ok(void); int is_staging_gitmodules_ok(struct index_state *istate); @@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r, int command_line_option, int default_option, int quiet, int max_parallel_jobs); +int get_submodules_status(struct string_list *submodules, + int max_parallel_jobs); unsigned is_submodule_modified(const char *path, int ignore_untracked); int submodule_uses_gitfile(const char *path); diff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh index 40164ae07d..1c747cc325 100755 --- a/t/t4027-diff-submodule.sh +++ b/t/t4027-diff-submodule.sh @@ -34,6 +34,25 @@ test_expect_success setup ' subtip=$3 subprev=$2 ' +test_expect_success 'diff in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git diff && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git diff && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_expect_success 'git diff --raw HEAD' ' hexsz=$(test_oid hexsz) && git diff --raw --abbrev=$hexsz HEAD >actual && @@ -70,6 +89,18 @@ test_expect_success 'git diff HEAD with dirty submodule (work tree)' ' test_cmp expect.body actual.body ' +test_expect_success 'git diff HEAD with dirty submodule (work tree, parallel)' ' + ( + cd sub && + git reset --hard && + echo >>world + ) && + git -c submodule.diffJobs=8 diff HEAD >actual && + sed -e "1,/^@@/d" actual >actual.body && + expect_from_to >expect.body $subtip $subprev-dirty && + test_cmp expect.body actual.body +' + test_expect_success 'git diff HEAD with dirty submodule (index)' ' ( cd sub && diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh index d050091345..7da64e4c4c 100755 --- a/t/t7506-status-submodule.sh +++ b/t/t7506-status-submodule.sh @@ -412,4 +412,29 @@ test_expect_success 'status with added file in nested submodule (short)' ' EOF ' +test_expect_success 'status in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git status && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git status && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + +test_expect_success 'status in superproject with submodules (parallel)' ' + git -C super status --porcelain >output && + git -C super -c submodule.diffJobs=8 status --porcelain >output_parallel && + diff output output_parallel +' + test_done