From patchwork Tue Oct 11 23:26:01 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13004539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CC59C433FE for ; Tue, 11 Oct 2022 23:26:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229618AbiJKX0e (ORCPT ); Tue, 11 Oct 2022 19:26:34 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58394 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229540AbiJKX0c (ORCPT ); Tue, 11 Oct 2022 19:26:32 -0400 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C903371718 for ; Tue, 11 Oct 2022 16:26:31 -0700 (PDT) Received: by mail-pg1-x54a.google.com with SMTP id y71-20020a638a4a000000b0046014b2258dso5068318pgd.19 for ; Tue, 11 Oct 2022 16:26:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ceeTMB+fvef5OOtMwQxHt1ya7sLwCsHHOc9Au/3yZw0=; b=LMq5sCjBG1ipU26JEKvpBm8PcFjjdSMBmP3EP7wkfs107NL8urwx9r0KERJCHnh0vj bCwTw507UOTVXm4kxVg8+SEJVHbiHxM5kYcW6UsZTzD1hoXKRJIuvjBTtD8pq9xKuXMA iwZ4PCuTbCu88b5VWC637AHm1AI/uwOOkiQiv3i/XFaQINOTBxPlMgTXO4VuzCyZBL27 rIqLrs24l7xCZWmenTOEpsGyiCyzqaTxvXy7IlPASF3BiXvh13u2+me1LPJxkcy7E83o X25ks54Iqc5uS2+4QQrC3Jbk0/ecVJptdnF46hnHdo5Cy4/HlHncoqo6m+IiBLQ867If z0CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ceeTMB+fvef5OOtMwQxHt1ya7sLwCsHHOc9Au/3yZw0=; b=W9CD/j0Nh4LXNT/tQt16Xr2UFKOI/wD/dNVI9faM7NchepEkGEyCXa58hJ2OE5sdl/ EpOu9iImTX1mcDrwlzZ9M8gzYGOBcdk3QCO94rZkHDlESmxzS4CM1X0C9PT69/N8VzBb z6sBqHHyvPN3VJ/X7RFVZ4HyOKM51gES+bCZ3Ex9Pj0Tv4G8MFS7mF0cYZeBRssNbGai mNowPFra8HEeJz/nNJiqyWpvw6BaKjlevb/K8p5zuLs5mdTCRbtvBWzjvVWp6Jgfie6Z flEdiWemDxaY2TGnSqopf0ma+oNW5dTYxNrNhSQtXHJucnUx6oLW5af8zgtwb7XcZoiq 91xA== X-Gm-Message-State: ACrzQf0POTDUSpMmXWFxbzIKi1rIQWFJJrAfDLGUiK9skwON4jvA1Zf0 RArVCbJlwydfkTUnz8zkvzUs0xMQmdmLxStu+AkZC+L2k+irOgL9X/vPlvuT/jOEKM9Cw52vltD wfCv8wrvVaqOBuBj+ogEP3X+OIVKneA3BluCFjjREHq8oy3C5EVIF8ZlZhKVVWoNEqw== X-Google-Smtp-Source: AMsMyM5E21BxzFE8BoqbxSIpzD8Np5RJoB+4mxjo4TwvDc8YRC5T+fFglXCGpO5Yon1mphVbekpkEJHH0+nu4uI= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a05:6a00:1707:b0:562:e790:dfc3 with SMTP id h7-20020a056a00170700b00562e790dfc3mr21638075pfc.59.1665530791191; Tue, 11 Oct 2022 16:26:31 -0700 (PDT) Date: Tue, 11 Oct 2022 23:26:01 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221011232604.839941-2-calvinwan@google.com> Subject: [PATCH v2 1/4] run-command: add pipe_output_fn to run_processes_parallel_opts From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add pipe_output_fn as an optionally set function in run_process_parallel_opts. If set, output from each child process is piped to the callback function to allow for separate parsing. Signed-off-by: Calvin Wan --- run-command.c | 12 +++++-- run-command.h | 22 +++++++++++++ t/helper/test-run-command.c | 18 ++++++++++ t/t0061-run-command.sh | 65 +++++++++++++++++++++++-------------- 4 files changed, 90 insertions(+), 27 deletions(-) diff --git a/run-command.c b/run-command.c index da02631933..c6090e4cb8 100644 --- a/run-command.c +++ b/run-command.c @@ -1689,12 +1689,16 @@ static void pp_buffer_stderr(struct parallel_processes *pp, int output_timeout) } } -static void pp_output(struct parallel_processes *pp) +static void pp_output(struct parallel_processes *pp, + const struct run_process_parallel_opts *opts) { int i = pp->output_owner; if (pp->children[i].state == GIT_CP_WORKING && pp->children[i].err.len) { + if (opts->pipe_output) + opts->pipe_output(&pp->children[i].err, pp->data, + pp->children[i].data); strbuf_write(&pp->children[i].err, stderr); strbuf_reset(&pp->children[i].err); } @@ -1716,6 +1720,10 @@ static int pp_collect_finished(struct parallel_processes *pp, code = finish_command(&pp->children[i].process); + if (opts->pipe_output) + opts->pipe_output(&pp->children[i].err, pp->data, + pp->children[i].data); + if (opts->task_finished) code = opts->task_finished(code, opts->ungroup ? NULL : &pp->children[i].err, pp->data, @@ -1803,7 +1811,7 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts) pp.children[i].state = GIT_CP_WAIT_CLEANUP; } else { pp_buffer_stderr(&pp, output_timeout); - pp_output(&pp); + pp_output(&pp, opts); } code = pp_collect_finished(&pp, opts); if (code) { diff --git a/run-command.h b/run-command.h index 075bd9b9de..cb51c56ea6 100644 --- a/run-command.h +++ b/run-command.h @@ -440,6 +440,22 @@ typedef int (*start_failure_fn)(struct strbuf *out, void *pp_cb, void *pp_task_cb); +/** + * This callback is periodically called while child processes are + * running and also when a child process finishes. + * + * "struct strbuf *out" contains the output collected from pp_task_cb + * since the last call of this function. + * + * pp_cb is the callback cookie as passed into run_processes_parallel, + * pp_task_cb is the callback cookie as passed into get_next_task_fn. + * + * This function is incompatible with "ungroup" + */ +typedef void (*pipe_output_fn)(struct strbuf *out, + void *pp_cb, + void *pp_task_cb); + /** * This callback is called on every child process that finished processing. * @@ -493,6 +509,12 @@ struct run_process_parallel_opts */ start_failure_fn start_failure; + /** + * pipe_output: See pipe_output_fn() above. This should be + * NULL unless process specific output is needed + */ + pipe_output_fn pipe_output; + /** * task_finished: See task_finished_fn() above. This can be * NULL to omit any special handling. diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c index 46bac2bb70..d3c3df7960 100644 --- a/t/helper/test-run-command.c +++ b/t/helper/test-run-command.c @@ -52,6 +52,18 @@ static int no_job(struct child_process *cp, return 0; } +static void pipe_output(struct strbuf *out, + void *pp_cb, + void *pp_task_cb) +{ + fprintf(stderr, "%s", out->buf); + /* + * Resetting output to show that piped output would print the + * same as other tests without the pipe_output() function set + */ + strbuf_reset(out); +} + static int task_finished(int result, struct strbuf *err, void *pp_cb, @@ -439,6 +451,12 @@ int cmd__run_command(int argc, const char **argv) opts.ungroup = 1; } + if (!strcmp(argv[1], "--pipe-output")) { + argv += 1; + argc -= 1; + opts.pipe_output = pipe_output; + } + jobs = atoi(argv[2]); strvec_clear(&proc.args); strvec_pushv(&proc.args, (const char **)argv + 3); diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh index 19af082750..feabb3717b 100755 --- a/t/t0061-run-command.sh +++ b/t/t0061-run-command.sh @@ -129,11 +129,14 @@ Hello World EOF -test_expect_success 'run_command runs in parallel with more jobs available than tasks' ' - test-tool run-command run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && - test_must_be_empty out && - test_cmp expect err -' +for opt in '' '--pipe-output' +do + test_expect_success "run_command runs in parallel with more jobs available than tasks $opt" ' + test-tool run-command run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err + ' +done test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' ' test-tool run-command --ungroup run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && @@ -141,11 +144,14 @@ test_expect_success 'run_command runs ungrouped in parallel with more jobs avail test_line_count = 4 err ' -test_expect_success 'run_command runs in parallel with as many jobs as tasks' ' - test-tool run-command run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && - test_must_be_empty out && - test_cmp expect err -' +for opt in '' '--pipe-output' +do + test_expect_success "run_command runs in parallel with as many jobs as tasks $opt" ' + test-tool run-command run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err + ' +done test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' ' test-tool run-command --ungroup run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && @@ -153,11 +159,14 @@ test_expect_success 'run_command runs ungrouped in parallel with as many jobs as test_line_count = 4 err ' -test_expect_success 'run_command runs in parallel with more tasks than jobs available' ' - test-tool run-command run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && - test_must_be_empty out && - test_cmp expect err -' +for opt in '' '--pipe-output' +do + test_expect_success "run_command runs in parallel with more tasks than jobs available $opt" ' + test-tool run-command run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err + ' +done test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' ' test-tool run-command --ungroup run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && @@ -174,11 +183,14 @@ preloaded output of a child asking for a quick stop EOF -test_expect_success 'run_command is asked to abort gracefully' ' - test-tool run-command run-command-abort 3 false >out 2>err && - test_must_be_empty out && - test_cmp expect err -' +for opt in '' '--pipe-output' +do + test_expect_success "run_command is asked to abort gracefully $opt" ' + test-tool run-command run-command-abort 3 false >out 2>err && + test_must_be_empty out && + test_cmp expect err + ' +done test_expect_success 'run_command is asked to abort gracefully (ungroup)' ' test-tool run-command --ungroup run-command-abort 3 false >out 2>err && @@ -190,11 +202,14 @@ cat >expect <<-EOF no further jobs available EOF -test_expect_success 'run_command outputs ' ' - test-tool run-command run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && - test_must_be_empty out && - test_cmp expect err -' +for opt in '' '--pipe-output' +do + test_expect_success "run_command outputs $opt" ' + test-tool run-command run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err + ' +done test_expect_success 'run_command outputs (ungroup) ' ' test-tool run-command --ungroup run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && From patchwork Tue Oct 11 23:26:02 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13004540 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 427C5C4332F for ; Tue, 11 Oct 2022 23:26:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229633AbiJKX0f (ORCPT ); Tue, 11 Oct 2022 19:26:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58402 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229627AbiJKX0e (ORCPT ); Tue, 11 Oct 2022 19:26:34 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7A4C86F27F for ; Tue, 11 Oct 2022 16:26:33 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id q62-20020a17090a17c400b0020d334fbe19so208434pja.0 for ; Tue, 11 Oct 2022 16:26:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GBf67QctDHdO8OijIMWU7F0vySW5T0Zas2xcFHPJo7o=; b=MZnBaGRS0w5LWAAYQ6QT4fNWso78H6sp45GXO/l6Ozfu3AMjcoLuFSsZP/UKogRpJ1 PafLdWthNHCUclWUFObqKcH7nVaQASxCZLaZdekSp1EJryAAa9g77Zey7RvC6FFGTv/W ds4LzfJToHni0csmFGv29vbvupoPA71+JdN2XD5ncRzqARvpS01LBjDLb1I/bgtNuBki 5QS0Yp+VP3MuLhye5Gr8rSh7A90duwgPxQN7RL6LyVHEbHGbZbCQWHapQhkkje0UwaRU QjR6yT4juZE1KCQygbbgubKUhQhXNX0zuX1jtAl66IgRp5y45doKD5UHI20dekW9tWZW r5CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GBf67QctDHdO8OijIMWU7F0vySW5T0Zas2xcFHPJo7o=; b=gpdIZaFJ7YQJxUd7TTnsLfIMY5iSGDKbaYsuC5JyhX/EqIzIpEBqlOjumjDzFqIN+D a45RwzBfv+2RgInjpbqa8lGg2ilbS86sWG5QRjTiSoneHVcfkE0q/cqdNR7o3fReWa/r Dn2onpHgYLjPsr7zlOFl0Q8Dj0CxX8bD7ynEQ7sRNVsJq4vm3LhVeL2dj5rKVWjU4yS/ aHomoPNY/cS3qHQcZ2gzP6OqsOLYhprPMY8hhCtxcvBNDJ31QsB2e0fsScDBHgK42sy6 ve1KN02E08aGnF0aGKg2oPRzQgTkQFMvsB92bYfMcMvAtK3ZzjHqzNdF8GgeKiSzYfyR UsxQ== X-Gm-Message-State: ACrzQf2mHKaXmthvVFRiLGDMxwNUKga9P0oxVPc/ym13IZW1w0w4JgdZ qkq/7cL9qhWUVvFWv+iptJjubdy5RSiJGbIzmTvSm9i2RBcjeNXCCJa/e8ykjtUcuU0mchpwozj z0eM46OTgGxh1WFuCs6qZfsyLjeTT2Wukpp4r1h/8BGMvJCxKzq0+nba3VWwKP8qOnA== X-Google-Smtp-Source: AMsMyM6p/JY4ZMW8ElJAvkpA6rWXRcbqyM3NaPNmDxkvKjdjPcRUc0tG7ko/GW0MzIzR6pQ+EGp5r0wsZ2poANk= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a05:6a00:891:b0:565:85a7:a6e with SMTP id q17-20020a056a00089100b0056585a70a6emr3042509pfj.21.1665530792985; Tue, 11 Oct 2022 16:26:32 -0700 (PDT) Date: Tue, 11 Oct 2022 23:26:02 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221011232604.839941-3-calvinwan@google.com> Subject: [PATCH v2 2/4] submodule: move status parsing into function From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A future patch requires the ability to parse the output of git status --porcelain=2. Move parsing code from is_submodule_modified to parse_status_porcelain. Signed-off-by: Calvin Wan --- submodule.c | 71 +++++++++++++++++++++++++++++------------------------ 1 file changed, 39 insertions(+), 32 deletions(-) diff --git a/submodule.c b/submodule.c index 72b295b87b..a3410ed8f0 100644 --- a/submodule.c +++ b/submodule.c @@ -1864,6 +1864,43 @@ int fetch_submodules(struct repository *r, return spf.result; } +static int parse_status_porcelain(char *buf, unsigned *dirty_submodule, int ignore_untracked) +{ + /* regular untracked files */ + if (buf[0] == '?') + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (buf[0] == 'u' || + buf[0] == '1' || + buf[0] == '2') { + /* T = line type, XY = status, SSSS = submodule state */ + if (strlen(buf) < strlen("T XY SSSS")) + BUG("invalid status --porcelain=2 line %s", + buf); + + if (buf[5] == 'S' && buf[8] == 'U') + /* nested untracked file */ + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (buf[0] == 'u' || + buf[0] == '2' || + memcmp(buf + 5, "S..U", 4)) + /* other change */ + *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; + } + + if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && + ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || + ignore_untracked)) { + /* + * We're not interested in any further information from + * the child any more, neither output nor its exit code. + */ + return 1; + } + return 0; +} + unsigned is_submodule_modified(const char *path, int ignore_untracked) { struct child_process cp = CHILD_PROCESS_INIT; @@ -1900,39 +1937,9 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) fp = xfdopen(cp.out, "r"); while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { - /* regular untracked files */ - if (buf.buf[0] == '?') - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (buf.buf[0] == 'u' || - buf.buf[0] == '1' || - buf.buf[0] == '2') { - /* T = line type, XY = status, SSSS = submodule state */ - if (buf.len < strlen("T XY SSSS")) - BUG("invalid status --porcelain=2 line %s", - buf.buf); - - if (buf.buf[5] == 'S' && buf.buf[8] == 'U') - /* nested untracked file */ - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (buf.buf[0] == 'u' || - buf.buf[0] == '2' || - memcmp(buf.buf + 5, "S..U", 4)) - /* other change */ - dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; - } - - if ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && - ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || - ignore_untracked)) { - /* - * We're not interested in any further information from - * the child any more, neither output nor its exit code. - */ - ignore_cp_exit_code = 1; + ignore_cp_exit_code = parse_status_porcelain(buf.buf, &dirty_submodule, ignore_untracked); + if (ignore_cp_exit_code) break; - } } fclose(fp); From patchwork Tue Oct 11 23:26:03 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13004541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CF20C4332F for ; Tue, 11 Oct 2022 23:26:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229699AbiJKX0l (ORCPT ); Tue, 11 Oct 2022 19:26:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229627AbiJKX0g (ORCPT ); Tue, 11 Oct 2022 19:26:36 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F157C6F27F for ; Tue, 11 Oct 2022 16:26:34 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id z24-20020a17090abd9800b0020d43dcc8c3so218001pjr.9 for ; Tue, 11 Oct 2022 16:26:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=im4HlctZZKrR9Un2oLKgHczK0bYCIv5ClqJwS46AMOM=; b=caWc/XnvV2KgFs+aiywJd7vKS0ysHRDqA9A8366T0kZOOsq2Huuebkd0c2CttYFxtI +PwkoAWwzGWSeovYyk439oq6W1kXPDFu1twb4zyCKG0MoYbsc0kg0KlmmYZ3fDp8Mv0E 0ub8szAb22T+wTH7e5nIPXWq7i2XnbQkybODMY8HwXMhdQ5X8IUg/prQ/aFFyc/32Nob j+BxcskQYr3oMaHAgUk6Ir3KJP3EKZPrWe6TbPA5xlLixLeaXBjkNbBtVXhZArAvcHBf 7V3dBl8ViCihY/AsTD6nDGPsMkbdoJe1EpwohH5QSpIUq6HvzBbdA1tMNub7aev44yHg i2sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=im4HlctZZKrR9Un2oLKgHczK0bYCIv5ClqJwS46AMOM=; b=IVGGLLi8xIIEBrw5ABY49aDnlBZo1Si2DXu7USEn4VCXaT3aGRcOO8l6qKNdpQkHA1 4A/Tc7UD62xXDQw6gwUA27klysyNDvcTLOUg1qwHQkZBg+Mal3Lp/0T2Xq13lkqi/Ryo orpFxyEAmdswu3/lHzZIRe4u9PLEGouzZ8XDYqShZheEyShcnTSRr/ATuTunKjnBgDaP wipX/SCu9zfkX/LZbBp99OI5RCvOrkREetj/HG2wYS869djwy1n1y6A/wVEP/6bDjSHn UEf+AxX7gpmBYxNPn9b1BtxSH1RuE630ZiWaPFAcL/Eu3qSxTavf1XwuzDkhrP97Y8J7 0QUA== X-Gm-Message-State: ACrzQf01o8jeIi9imIyBN1aDLJ94c9RnIftHTtlas721ZOYBVYDQ9Tlg tGpl3/1wRi5tu8/2bXfIuKnN1NKCbUtSNVG8YUQI4VeHQOLSb4BWSp+Wqsyr1KlaFF6vYt0NMRr kVSev413gxob7hstj8ge/XWURqFJmwTNoejukVkGelImMOq6wco+6slmgwJOC2s07LA== X-Google-Smtp-Source: AMsMyM5W6ySAfFyKWOJjqBA5hCpnVVHcJUyIOP0CIa/TDT9tlimiISpzunsonNIQk5NHgszltYXZzP/CsmrPad4= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:90b:4c02:b0:20a:7d26:149 with SMTP id na2-20020a17090b4c0200b0020a7d260149mr1822820pjb.134.1665530794534; Tue, 11 Oct 2022 16:26:34 -0700 (PDT) Date: Tue, 11 Oct 2022 23:26:03 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221011232604.839941-4-calvinwan@google.com> Subject: [PATCH v2 3/4] diff-lib: refactor match_stat_with_submodule From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Flatten out the if statements in match_stat_with_submodule so the logic is more readable and easier for future patches to add to. orig_flags didn't need to be set if the cache entry wasn't a GITLINK so defer setting it. Signed-off-by: Calvin Wan --- diff-lib.c | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/diff-lib.c b/diff-lib.c index 2edea41a23..e249322141 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -73,18 +73,25 @@ static int match_stat_with_submodule(struct diff_options *diffopt, unsigned *dirty_submodule) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); - if (S_ISGITLINK(ce->ce_mode)) { - struct diff_flags orig_flags = diffopt->flags; - if (!diffopt->flags.override_submodule_config) - set_diffopt_flags_from_submodule_config(diffopt, ce->name); - if (diffopt->flags.ignore_submodules) - changed = 0; - else if (!diffopt->flags.ignore_dirty_submodules && + struct diff_flags orig_flags; + + if (!S_ISGITLINK(ce->ce_mode)) + goto ret; + + orig_flags = diffopt->flags; + if (!diffopt->flags.override_submodule_config) + set_diffopt_flags_from_submodule_config(diffopt, ce->name); + if (diffopt->flags.ignore_submodules) { + changed = 0; + goto cleanup; + } + if (!diffopt->flags.ignore_dirty_submodules && (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, - diffopt->flags.ignore_untracked_in_submodules); + *dirty_submodule = is_submodule_modified(ce->name, + diffopt->flags.ignore_untracked_in_submodules); +cleanup: diffopt->flags = orig_flags; - } +ret: return changed; } From patchwork Tue Oct 11 23:26:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13004542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7EB2C433FE for ; Tue, 11 Oct 2022 23:26:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229733AbiJKX0p (ORCPT ); Tue, 11 Oct 2022 19:26:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58514 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229695AbiJKX0k (ORCPT ); Tue, 11 Oct 2022 19:26:40 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9598372B56 for ; Tue, 11 Oct 2022 16:26:36 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id f9-20020a17090a664900b0020ada57adaeso187985pjm.7 for ; Tue, 11 Oct 2022 16:26:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=kdNGMyS5lHvrbNMzDNMWJAAfM3H9B0uRpEYTTcfXHQE=; b=IrrEaNbZOIghFeh7GZQVRZistVaQ3Q9TawUcKDulr1sNv7R0MSMJCCAV66hH24Orwp ED0z4sB0Lv7z/qF7ugYo+b5vvM1qIbBH2/IjxcN+FJqz8VfjuJ2yogcZHB4VCDxnhQ3c 8/2EXsvcRsY+h7P1MRV7W7e6QvHeWm9/Tocm8eakpKmwMux57uOk11TW7O9GsLR8AR8+ EW7sa2gMvQ8HoOBzDtOHkBtslnPxZSE5+cpU7AFDuY7EJ5xdY2WCIpR1RlOHPRXZ1N7q fds/ojNMnqY5ylKZMBll7yB9/VDbbc7J9xzaKb8PnaxwzNFGRvblHMRhZquZGqFCI4iq 0eug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kdNGMyS5lHvrbNMzDNMWJAAfM3H9B0uRpEYTTcfXHQE=; b=KFxFYNSSLJyq0zIt+dInaNsDn1BwyrmNkkgtFU0TPQIoaaVnga7A9roEX0NRNqIkPf J4hisiE52taQiVlUhELm2B5uEOpepZiLzLlNg89KtKou5K8Rg8/v1gOIdqnilBR/e+Vd Dspfjrs31xrQUFEWr2Q5VeAVRGwHKVS4KzwW7tcn0AS2uMFtIzkCvkNT0q9Mp+PgsGQv dGA7i4uAXI0SFMGlLtyRDyUN8BweO67NF+cMNrsBe5cs+QTP1dFDKyeKVado9Yw9si3r Vl1BDOcNJqFKguOcvGU0meuS7hg7ugeAQ6Uce5j1eia0LrrwgKi2SaxioZnq4MArQ979 BAgQ== X-Gm-Message-State: ACrzQf359HTGkuMI5HtAqaQm2BmU5lktzsaOTKOq0/D64GZwx7yz3Kn/ 56zsKNn0njzZhNvQt3R6TIM5jQbDymg727qcYVFn0UYec9T8ISldxGVO/voQghgKwQlizAPEjeY AJzCB933PdP4vrEqgUET4c1JCvW/XE48tDws21NHkuXnSzx3v16DyQ20b4nTF/nuFXg== X-Google-Smtp-Source: AMsMyM7IJXjGfTT+NqQwvYmNk/A78rBkDWoafXhmN0FjoWWT4WXZbR8RZqeZaJlV8juYIN9/5GnaqUGFz2RnM90= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:902:e944:b0:179:dee4:f115 with SMTP id b4-20020a170902e94400b00179dee4f115mr27576642pll.141.1665530795955; Tue, 11 Oct 2022 16:26:35 -0700 (PDT) Date: Tue, 11 Oct 2022 23:26:04 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.0.rc1.362.ged0d419d3c-goog Message-ID: <20221011232604.839941-5-calvinwan@google.com> Subject: [PATCH v2 4/4] diff-lib: parallelize run_diff_files for submodules From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During the iteration of the index entries in run_diff_files, whenever a submodule is found and needs its status checked, a subprocess is spawned for it. Instead of spawning the subprocess immediately and waiting for its completion to continue, hold onto all submodules and relevant information in a list. Then use that list to create tasks for run_processes_parallel. Subprocess output is piped to status_pipe_output which parses it. Add config option submodule.diffJobs to set the maximum number of parallel jobs. The option defaults to 1 if unset. If set to 0, the number of jobs is set to online_cpus(). I added a TODO here, regarding defaults -- please see the cover letter for the discussion. Since run_diff_files is called from many different commands, I chose to grab the config option in the function rather than adding variables to every git command and then figuring out how to pass them all in. Signed-off-by: Calvin Wan --- Documentation/config/submodule.txt | 12 +++ diff-lib.c | 89 ++++++++++++++-- submodule.c | 159 +++++++++++++++++++++++++++++ submodule.h | 9 ++ t/t4027-diff-submodule.sh | 19 ++++ t/t7506-status-submodule.sh | 19 ++++ 6 files changed, 299 insertions(+), 8 deletions(-) diff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt index 6490527b45..3209eb8117 100644 --- a/Documentation/config/submodule.txt +++ b/Documentation/config/submodule.txt @@ -93,6 +93,18 @@ submodule.fetchJobs:: in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1. +submodule.diffJobs:: + Specifies how many submodules are diffed at the same time. A + positive integer allows up to that number of submodules diffed + in parallel. A value of 0 will give some reasonable default. + If unset, it defaults to 1. The diff operation is used by many + other git commands such as add, merge, diff, status, stash and + more. Note that the expensive part of the diff operation is + reading the index from cache or memory. Therefore multiple jobs + may be detrimental to performance if your hardware does not + support parallel reads or if the number of jobs greatly exceeds + the amount of supported reads. + submodule.alternateLocation:: Specifies how the submodules obtain alternates when submodules are cloned. Possible values are `no`, `superproject`. diff --git a/diff-lib.c b/diff-lib.c index e249322141..44bb5558b3 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -14,6 +14,7 @@ #include "dir.h" #include "fsmonitor.h" #include "commit-reach.h" +#include "config.h" /* * diff-files @@ -65,15 +66,20 @@ static int check_removed(const struct index_state *istate, const struct cache_en * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES * option is set, the caller does not only want to know if a submodule is * modified at all but wants to know all the conditions that are met (new - * commits, untracked content and/or modified content). + * commits, untracked content and/or modified content). If + * defer_submodule_status bit is set, dirty_submodule will be left to the + * caller to set. defer_submodule_status can also be set to 0 in this + * function if there is no need to check if the submodule is modified. */ static int match_stat_with_submodule(struct diff_options *diffopt, const struct cache_entry *ce, struct stat *st, unsigned ce_option, - unsigned *dirty_submodule) + unsigned *dirty_submodule, int *defer_submodule_status, + int *ignore_untracked_in_submodules) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); struct diff_flags orig_flags; + int defer = 0; if (!S_ISGITLINK(ce->ce_mode)) goto ret; @@ -86,12 +92,21 @@ static int match_stat_with_submodule(struct diff_options *diffopt, goto cleanup; } if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, - diffopt->flags.ignore_untracked_in_submodules); + (!changed || diffopt->flags.dirty_submodules)) { + if (defer_submodule_status && *defer_submodule_status) { + defer = 1; + *ignore_untracked_in_submodules = + diffopt->flags.ignore_untracked_in_submodules; + } else { + *dirty_submodule = is_submodule_modified(ce->name, + diffopt->flags.ignore_untracked_in_submodules); + } + } cleanup: - diffopt->flags = orig_flags; + diffopt->flags = orig_flags; ret: + if (defer_submodule_status) + *defer_submodule_status = defer; return changed; } @@ -103,6 +118,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ? CE_MATCH_RACY_IS_DIRTY : 0); uint64_t start = getnanotime(); struct index_state *istate = revs->diffopt.repo->index; + struct string_list submodules = STRING_LIST_INIT_NODUP; diff_set_mnemonic_prefix(&revs->diffopt, "i/", "w/"); @@ -227,6 +243,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option) newmode = ce->ce_mode; } else { struct stat st; + int ignore_untracked_in_submodules = 0; + int defer_submodule_status = !!revs->repo; changed = check_removed(istate, ce, &st); if (changed) { @@ -248,8 +266,22 @@ int run_diff_files(struct rev_info *revs, unsigned int option) } changed = match_stat_with_submodule(&revs->diffopt, ce, &st, - ce_option, &dirty_submodule); + ce_option, &dirty_submodule, + &defer_submodule_status, + &ignore_untracked_in_submodules); newmode = ce_mode_from_stat(ce, st.st_mode); + if (defer_submodule_status) { + struct string_list_item *item = + string_list_append(&submodules, ce->name); + struct submodule_status_util *util = xmalloc(sizeof(*util)); + util->changed = changed; + util->dirty_submodule = 0; + util->ignore_untracked = ignore_untracked_in_submodules; + util->newmode = newmode; + util->ce = ce; + item->util = util; + continue; + } } if (!changed && !dirty_submodule) { @@ -268,6 +300,47 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ce->name, 0, dirty_submodule); } + if (submodules.nr > 0) { + int i; + int parallel_jobs = 1; + git_config_get_int("submodule.diffjobs", ¶llel_jobs); + if (parallel_jobs < 0) { + die(_("submodule.diffjobs cannot be negative")); + } + else if (!parallel_jobs) { + /* + * TODO: Decide what a reasonable default for parallel_jobs + * is. Currently mimics what other parallel config options + * default to. + */ + parallel_jobs = online_cpus(); + } + + if (get_submodules_status(revs->repo, &submodules, parallel_jobs)) + BUG("Submodule status failed"); + for (i = 0; i < submodules.nr; i++) { + struct submodule_status_util *util = submodules.items[i].util; + struct cache_entry *ce = util->ce; + unsigned int oldmode; + const struct object_id *old_oid, *new_oid; + + if (!util->changed && !util->dirty_submodule) { + ce_mark_uptodate(ce); + mark_fsmonitor_valid(istate, ce); + if (!revs->diffopt.flags.find_copies_harder) + continue; + } + oldmode = ce->ce_mode; + old_oid = &ce->oid; + new_oid = util->changed ? null_oid() : &ce->oid; + diff_change(&revs->diffopt, oldmode, util->newmode, + old_oid, new_oid, + !is_null_oid(old_oid), + !is_null_oid(new_oid), + ce->name, 0, util->dirty_submodule); + } + } + string_list_clear(&submodules, 1); diffcore_std(&revs->diffopt); diff_flush(&revs->diffopt); trace_performance_since(start, "diff-files"); @@ -315,7 +388,7 @@ static int get_stat_data(const struct index_state *istate, return -1; } changed = match_stat_with_submodule(diffopt, ce, &st, - 0, dirty_submodule); + 0, dirty_submodule, NULL, NULL); if (changed) { mode = ce_mode_from_stat(ce, st.st_mode); oid = null_oid(); diff --git a/submodule.c b/submodule.c index a3410ed8f0..72051fda81 100644 --- a/submodule.c +++ b/submodule.c @@ -1363,6 +1363,20 @@ int submodule_touches_in_range(struct repository *r, return ret; } +struct submodule_parallel_status { + int index_count; + int result; + + struct string_list *submodule_names; + struct repository *r; + + /* Pending statuses by OIDs */ + struct status_task **oid_status_tasks; + int oid_status_tasks_nr, oid_status_tasks_alloc; +}; + +#define SPS_INIT { 0 } + struct submodule_parallel_fetch { /* * The index of the last index entry processed by @@ -1445,6 +1459,13 @@ struct fetch_task { struct oid_array *commits; /* Ensure these commits are fetched */ }; +struct status_task { + struct repository *repo; + const char *path; + unsigned dirty_submodule; + int ignore_untracked; +}; + /** * When a submodule is not defined in .gitmodules, we cannot access it * via the regular submodule-config. Create a fake submodule, which we can @@ -1950,6 +1971,144 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) return dirty_submodule; } +static struct status_task * +get_status_task_from_index(struct submodule_parallel_status *sps, + struct strbuf *err) +{ + for (; sps->index_count < sps->submodule_names->nr; sps->index_count++) { + struct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util; + const struct cache_entry *ce = util->ce; + struct status_task *task; + struct strbuf buf = STRBUF_INIT; + const char *git_dir; + + strbuf_addf(&buf, "%s/.git", ce->name); + git_dir = read_gitfile(buf.buf); + if (!git_dir) + git_dir = buf.buf; + if (!is_git_directory(git_dir)) { + if (is_directory(git_dir)) + die(_("'%s' not recognized as a git repository"), git_dir); + strbuf_release(&buf); + /* The submodule is not checked out, so it is not modified */ + util->dirty_submodule = 0; + continue; + } + strbuf_release(&buf); + + task = xmalloc(sizeof(*task)); + memset(task, 0, sizeof(*task)); + task->path = ce->name; + task->dirty_submodule = util->dirty_submodule; + task->ignore_untracked = util->ignore_untracked; + sps->index_count++; + return task; + } + return NULL; +} + + +static int get_next_submodule_status(struct child_process *cp, + struct strbuf *err, void *data, void **task_cb) +{ + struct submodule_parallel_status *sps = data; + struct status_task *task = get_status_task_from_index(sps, err); + int ignore_untracked; + + if (!task) { + return 0; + } + + ignore_untracked = task->ignore_untracked; + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env); + + strvec_init(&cp->args); + strvec_pushl(&cp->args, "status", "--porcelain=2", NULL); + if (ignore_untracked) + strvec_push(&cp->args, "-uno"); + + prepare_submodule_repo_env(&cp->env); + cp->git_cmd = 1; + cp->no_stdin = 1; + cp->dir = task->path; + *task_cb = task; + return 1; +} + +static int status_start_failure(struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + + sps->result = 1; + return 0; +} + +static void status_pipe_output(struct strbuf *out, + void *cb, void *task_cb) +{ + struct status_task *task = task_cb; + struct string_list list = STRING_LIST_INIT_DUP; + int i; + + string_list_split(&list, out->buf, '\n', -1); + + for (i = 0; i < list.nr; i++) { + if (parse_status_porcelain(list.items[i].string, + &task->dirty_submodule, task->ignore_untracked)) + break; + } + string_list_clear(&list, 0); + strbuf_reset(out); +} + +static int status_finish(int retvalue, struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + struct string_list_item *it = + string_list_lookup(sps->submodule_names, task->path); + struct submodule_status_util *util = it->util; + + util->dirty_submodule = task->dirty_submodule; + free(task); + + return 0; +} + +int get_submodules_status(struct repository *r, + struct string_list *submodules, + int max_parallel_jobs) +{ + struct submodule_parallel_status sps = SPS_INIT; + const struct run_process_parallel_opts opts = { + .tr2_category = "submodule", + .tr2_label = "parallel/status", + + .jobs = max_parallel_jobs, + + .get_next_task = get_next_submodule_status, + .start_failure = status_start_failure, + .pipe_output = status_pipe_output, + .task_finished = status_finish, + .data = &sps, + }; + + sps.r = r; + + if (repo_read_index(r) < 0) + die(_("index file corrupt")); + + sps.submodule_names = submodules; + string_list_sort(sps.submodule_names); + run_processes_parallel(&opts); + + return sps.result; +} + int submodule_uses_gitfile(const char *path) { struct child_process cp = CHILD_PROCESS_INIT; diff --git a/submodule.h b/submodule.h index 6a9fec6de1..c829d37b9f 100644 --- a/submodule.h +++ b/submodule.h @@ -41,6 +41,12 @@ struct submodule_update_strategy { .type = SM_UPDATE_UNSPECIFIED, \ } +struct submodule_status_util { + int changed, ignore_untracked; + unsigned dirty_submodule, newmode; + struct cache_entry *ce; +}; + int is_gitmodules_unmerged(struct index_state *istate); int is_writing_gitmodules_ok(void); int is_staging_gitmodules_ok(struct index_state *istate); @@ -94,6 +100,9 @@ int fetch_submodules(struct repository *r, int command_line_option, int default_option, int quiet, int max_parallel_jobs); +int get_submodules_status(struct repository *r, + struct string_list *submodules, + int max_parallel_jobs); unsigned is_submodule_modified(const char *path, int ignore_untracked); int submodule_uses_gitfile(const char *path); diff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh index 40164ae07d..e08ee315a7 100755 --- a/t/t4027-diff-submodule.sh +++ b/t/t4027-diff-submodule.sh @@ -34,6 +34,25 @@ test_expect_success setup ' subtip=$3 subprev=$2 ' +test_expect_success 'diff in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git diff && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git diff && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_expect_success 'git diff --raw HEAD' ' hexsz=$(test_oid hexsz) && git diff --raw --abbrev=$hexsz HEAD >actual && diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh index f5426a8e58..dfdfb58bfc 100755 --- a/t/t7506-status-submodule.sh +++ b/t/t7506-status-submodule.sh @@ -410,4 +410,23 @@ test_expect_success 'status with added file in nested submodule (short)' ' EOF ' +test_expect_success 'status in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git status && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git status && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_done