From patchwork Tue Nov 8 18:41:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13036719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id F050BC4332F for ; Tue, 8 Nov 2022 18:42:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231468AbiKHSmY (ORCPT ); Tue, 8 Nov 2022 13:42:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52720 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230235AbiKHSmV (ORCPT ); Tue, 8 Nov 2022 13:42:21 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E98751EAC7 for ; Tue, 8 Nov 2022 10:42:19 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id c7-20020a170903234700b0018729febd96so11730774plh.19 for ; Tue, 08 Nov 2022 10:42:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GMNLWZfPj6vsTp6+ALJfFkGBMyKCWALQeo2lsroj8+c=; b=EwBlWs99Q6gGbDndFLDpvlgh1QTP6IS+6g+6ReYfxPRfpWBFpBHdXyihSrU26M+XNQ YAQ+qWVC9BP+F2AcEsKoRiMTH3+db8IZXkgwkpQBnp54I1HvzEex1IYplUJqAhStRTav KM9KujpqepJt3Wythoh5xIYfjV026ksl3TrP1ZlzOZU7PIUZmqXKBMAUk8aAjHC4ijbX LqoTvy9/10qqegCJBzwWJuXWlN7z98zS54RDSzzxd9kfIzDNpTQ9g0Won2sOaDPOLo7q VNEJFspKdwOnsuQ3Gq2N5SjZhRsyd0seovLZcXy1RI10o484wrJaVJ6pEkEcH1lkDDo6 jG9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GMNLWZfPj6vsTp6+ALJfFkGBMyKCWALQeo2lsroj8+c=; b=wxCOYP1KMNL3K4EiXgMlOInnVeRSN5jw73egug6V+c3Te+h0d8BEgHCzB/NpqDjsPz rabQXq+ynOBfku5GQ7uPGEQyC92duyLBeWZRPOzAOclFWF/BPoS8UVVfbh5NppZsdMrc pD9tt3yUKxy3pCg2YNE865khOxMWTj5n/nFmI1xflClefOLqOi/UxQi1SFM8ZX4uUYyJ 0urMOyxCuYvavaWNIIHJ1ilU0Rs3rJ+P/mwfhnjlPAIkHOrCLDiRHINNWS8G6HXklsqT cL8bdrqVp+TJkMqmP7v/2P9HTkoIMipRuiGPHI+fUub8EnQ4zECg4js2G5TGDQ08LhP0 nQQw== X-Gm-Message-State: ACrzQf1KXxioCo32mlFNFQ6v6uAMnrU1YIzJMXOsf7VOxGSUKtAOf8DK 9A0RScv14DcX50pYZogUjQ1YESRAaMPTTcNeNgHtGf2qCeuPled2k/f4wYzBFuCulL/WeNuqY/P Zc/HFKpP04kgAiMHXBNe/AZE7fuVS7I9lbhOzU/Sc3pIgHJU8VbyizQy7RM+USbLX+A== X-Google-Smtp-Source: AMsMyM5a94dly9xrMhZ51HrKeuR3i1/5D4q8A7lVcHn9Vh8bDvtFxP8SvhZFQeWKJ57rQ62P6Zcq5+iDCGDqdmw= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:90a:d30c:b0:213:1035:f914 with SMTP id p12-20020a17090ad30c00b002131035f914mr75627036pju.196.1667932939319; Tue, 08 Nov 2022 10:42:19 -0800 (PST) Date: Tue, 8 Nov 2022 18:41:56 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221108184200.2813458-2-calvinwan@google.com> Subject: [PATCH v4 1/5] run-command: add duplicate_output_fn to run_processes_parallel_opts From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, myriamanis@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add duplicate_output_fn as an optionally set function in run_process_parallel_opts. If set, output from each child process is copied and passed to the callback function whenever output from the child process is buffered to allow for separate parsing. Signed-off-by: Calvin Wan --- run-command.c | 13 +++++++++++-- run-command.h | 24 +++++++++++++++++++++++ t/helper/test-run-command.c | 21 ++++++++++++++++++++ t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++ 4 files changed, 95 insertions(+), 2 deletions(-) diff --git a/run-command.c b/run-command.c index c772acd743..b8f430eb03 100644 --- a/run-command.c +++ b/run-command.c @@ -1560,6 +1560,9 @@ static void pp_init(struct parallel_processes *pp, if (!opts->get_next_task) BUG("you need to specify a get_next_task function"); + + if (opts->duplicate_output && opts->ungroup) + BUG("duplicate_output and ungroup are incompatible with each other"); CALLOC_ARRAY(pp->children, n); if (!opts->ungroup) @@ -1680,8 +1683,14 @@ static void pp_buffer_stderr(struct parallel_processes *pp, for (size_t i = 0; i < opts->processes; i++) { if (pp->children[i].state == GIT_CP_WORKING && pp->pfd[i].revents & (POLLIN | POLLHUP)) { - int n = strbuf_read_once(&pp->children[i].err, - pp->children[i].process.err, 0); + struct strbuf buf = STRBUF_INIT; + int n = strbuf_read_once(&buf, pp->children[i].process.err, 0); + strbuf_addbuf(&pp->children[i].err, &buf); + if (opts->duplicate_output) + opts->duplicate_output(&buf, &pp->children[i].err, + opts->data, + pp->children[i].data); + strbuf_release(&buf); if (n == 0) { close(pp->children[i].process.err); pp->children[i].state = GIT_CP_WAIT_CLEANUP; diff --git a/run-command.h b/run-command.h index e3e1ea01ad..dd6d6a86c2 100644 --- a/run-command.h +++ b/run-command.h @@ -440,6 +440,24 @@ typedef int (*start_failure_fn)(struct strbuf *out, void *pp_cb, void *pp_task_cb); +/** + * This callback is called whenever output from a child process is buffered + * + * "struct strbuf *process_out" contains the output from the child process + * + * See run_processes_parallel() below for a discussion of the "struct + * strbuf *out" parameter. + * + * pp_cb is the callback cookie as passed into run_processes_parallel, + * pp_task_cb is the callback cookie as passed into get_next_task_fn. + * + * This function is incompatible with "ungroup" + */ +typedef void (*duplicate_output_fn)(struct strbuf *process_out, + struct strbuf *out, + void *pp_cb, + void *pp_task_cb); + /** * This callback is called on every child process that finished processing. * @@ -493,6 +511,12 @@ struct run_process_parallel_opts */ start_failure_fn start_failure; + /** + * duplicate_output: See duplicate_output_fn() above. This should be + * NULL unless process specific output is needed + */ + duplicate_output_fn duplicate_output; + /** * task_finished: See task_finished_fn() above. This can be * NULL to omit any special handling. diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c index 3ecb830f4a..40dd329e02 100644 --- a/t/helper/test-run-command.c +++ b/t/helper/test-run-command.c @@ -52,6 +52,21 @@ static int no_job(struct child_process *cp, return 0; } +static void duplicate_output(struct strbuf *process_out, + struct strbuf *out, + void *pp_cb, + void *pp_task_cb) +{ + struct string_list list = STRING_LIST_INIT_DUP; + + string_list_split(&list, process_out->buf, '\n', -1); + for (size_t i = 0; i < list.nr; i++) { + if (strlen(list.items[i].string) > 0) + fprintf(stderr, "duplicate_output: %s\n", list.items[i].string); + } + string_list_clear(&list, 0); +} + static int task_finished(int result, struct strbuf *err, void *pp_cb, @@ -439,6 +454,12 @@ int cmd__run_command(int argc, const char **argv) opts.ungroup = 1; } + if (!strcmp(argv[1], "--duplicate-output")) { + argv += 1; + argc -= 1; + opts.duplicate_output = duplicate_output; + } + jobs = atoi(argv[2]); strvec_clear(&proc.args); strvec_pushv(&proc.args, (const char **)argv + 3); diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh index 7b5423eebd..130aec7c68 100755 --- a/t/t0061-run-command.sh +++ b/t/t0061-run-command.sh @@ -134,6 +134,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' ' test-tool run-command --ungroup run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -145,6 +154,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' ' test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' ' test-tool run-command --ungroup run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -156,6 +174,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' ' test-tool run-command --ungroup run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -176,6 +203,12 @@ test_expect_success 'run_command is asked to abort gracefully' ' test_cmp expect actual ' +test_expect_success 'run_command is asked to abort gracefully --duplicate-output' ' + test-tool run-command --duplicate-output run-command-abort 3 false >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command is asked to abort gracefully (ungroup)' ' test-tool run-command --ungroup run-command-abort 3 false >out 2>err && test_must_be_empty out && @@ -191,6 +224,12 @@ test_expect_success 'run_command outputs ' ' test_cmp expect actual ' +test_expect_success 'run_command outputs --duplicate-output' ' + test-tool run-command --duplicate-output run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command outputs (ungroup) ' ' test-tool run-command --ungroup run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_must_be_empty out && From patchwork Tue Nov 8 18:41:57 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13036720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55C00C4332F for ; Tue, 8 Nov 2022 18:42:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231868AbiKHSma (ORCPT ); Tue, 8 Nov 2022 13:42:30 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52732 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230522AbiKHSmV (ORCPT ); Tue, 8 Nov 2022 13:42:21 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B6402181B for ; Tue, 8 Nov 2022 10:42:21 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id c1-20020a170902d48100b0018723580343so11794161plg.15 for ; Tue, 08 Nov 2022 10:42:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=e2R3o2Am0MFayMeKl5FASeV6fHInGYw140p84pmEosA=; b=bSXrv+7fV0JVca2F4NOvkHxRE8Tq59OSO2efBKkl6xW8NDqxXMHp8QVittCzh8g3i+ een58+0whrMsY8gLCMommFGS28TR04jR5hKpkrQnhdS7IiWeVA7PgCiGL/u65+VGZv1l 5Nw1Z0m8EAkNGUMa18y4IMgFKn4loVPj7/fPVYfIu4F7u3eICS6DDqeNVj/0mzLHAAXp AX3yCclDsMDoXb8NWP5e0DfE95c1JhJ/zebIe+tri01JPkvQ0fshwqPo1+YxrZ5TJAtC 7EximOB38P+jdH2dRYuTT4YkNW/X7ShfgeZWj2KE7RDASQqRDOuqN5Y2Q4JPbBT/N+hk FCJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=e2R3o2Am0MFayMeKl5FASeV6fHInGYw140p84pmEosA=; b=4UyR+2XbzSqnaA31/NTx5ZE2t1K1oWQ6xZIwm3t4mp2GjdBThIdHKshWNZLrcbpmCg WWGBZe78R6Ivd8Ij2V9dBMvTdvzN624IT8n+6j+59SjFggShS8PMBfQIaIhBzipqMHNa fIbGgfI6Bpu6RYZHPtE/BBTjkKuJc14KjsSpHrCJC3DOwq8bFIPdHF2rKIwRS6ysy8Dy E7eedSHmSROZGdmF9Z7Ntu3Uvf19JafBfef5B7CPiCRK1CxSlo5vyoA6t8B6g9rEbTfa 7EHi1CHcO9Pqko0TWhW10dDUZ1KODhly2pUbNLBvay+zXMPJ64LDyiP/TsilNaxHHwyH ki2w== X-Gm-Message-State: ACrzQf0XkGUIusFIQkY/kY1iHtiG/iilJYA5afqmP7rC2psxhA9I4rDR 7o2xL1J6scdTrbZXftWDD7Y63ytN6fk/7E2ove9M76bmHXv3GJWbOybT/U6TJA2W4TDDu2h9dT1 NNMvcqE3GHA32RV+gJxRfMurvwAAy161jp0V06baM/yUPMzIHIfVrsFfhbvKLx23rmQ== X-Google-Smtp-Source: AMsMyM4Jbp22DoA3GTt3GLvtEtFlE0TOUSkOT9wxSw2CJqiIO5Y12vVf9bavISaYv165vFYadwvgh2dFZ7weSl0= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:903:22d2:b0:187:1f4a:6593 with SMTP id y18-20020a17090322d200b001871f4a6593mr49539614plg.138.1667932940685; Tue, 08 Nov 2022 10:42:20 -0800 (PST) Date: Tue, 8 Nov 2022 18:41:57 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221108184200.2813458-3-calvinwan@google.com> Subject: [PATCH v4 2/5] submodule: strbuf variable rename From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, myriamanis@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A prepatory change for a future patch that moves the status parsing logic to a separate function. Signed-off-by: Calvin Wan --- submodule.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/submodule.c b/submodule.c index b958162d28..31ee53bd57 100644 --- a/submodule.c +++ b/submodule.c @@ -1900,25 +1900,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) fp = xfdopen(cp.out, "r"); while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { + char *str = buf.buf; + const size_t len = buf.len; + /* regular untracked files */ - if (buf.buf[0] == '?') + if (str[0] == '?') dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '1' || - buf.buf[0] == '2') { + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { /* T = line type, XY = status, SSSS = submodule state */ - if (buf.len < strlen("T XY SSSS")) + if (len < strlen("T XY SSSS")) BUG("invalid status --porcelain=2 line %s", - buf.buf); + str); - if (buf.buf[5] == 'S' && buf.buf[8] == 'U') + if (str[5] == 'S' && str[8] == 'U') /* nested untracked file */ dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '2' || - memcmp(buf.buf + 5, "S..U", 4)) + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) /* other change */ dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; } From patchwork Tue Nov 8 18:41:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13036721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78036C433FE for ; Tue, 8 Nov 2022 18:42:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230522AbiKHSmd (ORCPT ); Tue, 8 Nov 2022 13:42:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52776 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231509AbiKHSmY (ORCPT ); Tue, 8 Nov 2022 13:42:24 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EA02266C8C for ; Tue, 8 Nov 2022 10:42:22 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id y65-20020a25c844000000b006bb773548d5so14654356ybf.5 for ; Tue, 08 Nov 2022 10:42:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=mXvsqKgSINLop8HJijHfOtFPSIYo6zDFwIipieqNbTE=; b=iD5KhW5rHtiRzg+DFXB6YkpqAG+hXFgUMcqskDGF5fd2bWYzITwJRsoVr1IHVKUsBA btXyQAgcP0qR70vGALysvw6hHg1ByvK9Wk1sQehvqMvG/7Jdu8nUqOKplTHnH5rzOm6l HiJ36/hfOurB5kKImmEaRc3VIClSfcAIAfLP/dqohnOUfWxEsMzNP7tCMSkrdTq8dU7i E1cMr0AHnr506aAS++9CgMPtKj3jRorbHU+w739OhOsWcwT6iXCix4BtWfKLQ4vSimo+ I0zUgJ5IL3cOzkGJ4ZijDpbKuG9d+DDfPBjxalhUCmBIx/N5oXdW+TdfgNTasbv7+jyx cStw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=mXvsqKgSINLop8HJijHfOtFPSIYo6zDFwIipieqNbTE=; b=aHEIALwScdCJ0O5URhe2lmrg1lQQWoEsSsTQrWzU5mty8Yl0FsDLBz21ajA6Cw1/+k yVv8a89yOQO6zdhjizpxmrxYWyKVb4ZROCPCKbaX2nmHRZn45eOwHN1/CI9BREbmiGsh gUj7IcOXD8UQOqqpG7PkgJO6eDBOBbWkx3I8rhBJ/37IptCQ7oKaQbtG57uI+hMc6HpC tJm+R814hBCYbB1Kl8pBImEiFk0cu7wq+h/kwHm760KrVY0hBCz/1mlkwd2O1NpvStEw nYWvXmWLXd8KozhDYAgWKT6qZSTA2HIZbSJ4W5ha6LYyPF6B4zD5GlCp7lvNF8tw6xhY AUpg== X-Gm-Message-State: ACrzQf2OQ/81RnArfG8I+LBJ610E28iWx2YvsuESn2Vs2GRkVu3Zqnxo 2WsiNph2AkJ1minjRQFlTVnIpdsqrgBMpe/5oez87t5I32izAyusC/uVk/VoM9/YpjHjQq1Cbqj 1NBWToXCP/AGUQBMgacfsRqelZM3dRy0fUyKBrzcrY2c5LcY3mYKAUEFj5CurjAJ+gA== X-Google-Smtp-Source: AMsMyM5g9vhyU8Msk6FD0RliVBAgpfhY4XNsgrl13mtg4Ldw/M10/VgUtxv4UjzzKHh3rITfUlbYZERUJgLFybA= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a81:4d5:0:b0:370:b2b:c52a with SMTP id 204-20020a8104d5000000b003700b2bc52amr999972ywe.342.1667932942136; Tue, 08 Nov 2022 10:42:22 -0800 (PST) Date: Tue, 8 Nov 2022 18:41:58 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221108184200.2813458-4-calvinwan@google.com> Subject: [PATCH v4 3/5] submodule: move status parsing into function From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, myriamanis@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A future patch requires the ability to parse the output of git status --porcelain=2. Move parsing code from is_submodule_modified to parse_status_porcelain. Signed-off-by: Calvin Wan --- submodule.c | 74 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 32 deletions(-) diff --git a/submodule.c b/submodule.c index 31ee53bd57..fd3385620c 100644 --- a/submodule.c +++ b/submodule.c @@ -1864,6 +1864,45 @@ int fetch_submodules(struct repository *r, return spf.result; } +static int parse_status_porcelain(char *str, size_t len, + unsigned *dirty_submodule, + int ignore_untracked) +{ + /* regular untracked files */ + if (str[0] == '?') + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { + /* T = line type, XY = status, SSSS = submodule state */ + if (len < strlen("T XY SSSS")) + BUG("invalid status --porcelain=2 line %s", + str); + + if (str[5] == 'S' && str[8] == 'U') + /* nested untracked file */ + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) + /* other change */ + *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; + } + + if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && + ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || + ignore_untracked)) { + /* + * We're not interested in any further information from + * the child any more, neither output nor its exit code. + */ + return 1; + } + return 0; +} + unsigned is_submodule_modified(const char *path, int ignore_untracked) { struct child_process cp = CHILD_PROCESS_INIT; @@ -1903,39 +1942,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) char *str = buf.buf; const size_t len = buf.len; - /* regular untracked files */ - if (str[0] == '?') - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '1' || - str[0] == '2') { - /* T = line type, XY = status, SSSS = submodule state */ - if (len < strlen("T XY SSSS")) - BUG("invalid status --porcelain=2 line %s", - str); - - if (str[5] == 'S' && str[8] == 'U') - /* nested untracked file */ - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '2' || - memcmp(str + 5, "S..U", 4)) - /* other change */ - dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; - } - - if ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && - ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || - ignore_untracked)) { - /* - * We're not interested in any further information from - * the child any more, neither output nor its exit code. - */ - ignore_cp_exit_code = 1; + ignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule, + ignore_untracked); + if (ignore_cp_exit_code) break; - } } fclose(fp); From patchwork Tue Nov 8 18:41:59 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13036722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 07226C4332F for ; Tue, 8 Nov 2022 18:42:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231688AbiKHSmf (ORCPT ); Tue, 8 Nov 2022 13:42:35 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52782 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231543AbiKHSmY (ORCPT ); Tue, 8 Nov 2022 13:42:24 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 051045985F for ; Tue, 8 Nov 2022 10:42:24 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id l63-20020a639142000000b0046f5bbb7372so8314696pge.23 for ; Tue, 08 Nov 2022 10:42:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=NY3XPH0EpuzSaGwctQvhV14t2wGRIiH6gSU5FViSu9I=; b=ZolB8mRJBHzfB+y4w6Mm3kuak7qg3CrRz/XovYezAtY9+eYY/yqm6+7MNw1rDX5J9A gspskSaKBXGzK5i94AMycD4reBS5/dzeX6EqjCl+T+bpx+BgUUdLagW7FTtTBq62ansG 9V7Omd1KmIQIaQvOncqFuBwb9IeXh7ZG/nkCELuBQzAs5IsUE9qvNct8mefZXsjXv3F7 rjy+O/qd/fbJDr8KvsJXh/HVHMn7ZTd9vj4LaJWQC41WRP7J1+H8SIQqEeK8eIHuiCYc O2r3vC9Fkk84xCYF4NetnzRMgZiW29UcxJiY5DdqQMOLhyV41Ij7GH51JqFNuB0dnDBh dxZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NY3XPH0EpuzSaGwctQvhV14t2wGRIiH6gSU5FViSu9I=; b=QdxIPSFEa4pLy8wzTQlEer3DRVRhnWpcSS3/vh7iN6bblWpjYlB2S/NSj+SLyFCPO0 WJcWvMi3DyYT2kVBZPVOBciBCEEUXKXCjfBwikpmroWMHpFA6DPktAjIMIC73X6/f4/B Gu4tClHej59rhjFDsYqfdpo+EtNfmY1iFkc8pDSFQmB55RtQepgQQ+xbG1/HJ+30+h95 cZsixXwFPQizXfWPHTO2rUMEmyEhR7D893VOG2YL2s+e/iSYbB9w/atZIZ2u53764CMi 4jy9U8FT9rF+9CGZQAPAVVe59Tp/Tl22pFB0YtHdZZ4bpLAbQiOwzavixKoGsZa+jI0d YR5g== X-Gm-Message-State: ACrzQf16kbTZUwlrSbyQJ1+hw0X3HE6X6fiV5/ddQrivXApwIFLs2M40 Js6954FyMtwepoIAIjID60nK4X4CfQjrPj6uE/vkbzJ82iyzth52QZgUo5cehlHfbh490bK+7ec HrOpTLjmMVs2Si6FDj7HsDRFyhwrn+FHs/STyM/kKsklYYLVPhmSZEvoX+K7v3QFXaQ== X-Google-Smtp-Source: AMsMyM7CQK3cU0j5r5pLFlApeRvetYJP7O33u8qZQ78vcgBqsQ5U8hafwmdnBxXJEMPqKkRhJZiJNkiiwJY74Xc= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a62:3281:0:b0:56d:385:149b with SMTP id y123-20020a623281000000b0056d0385149bmr57246455pfy.29.1667932943463; Tue, 08 Nov 2022 10:42:23 -0800 (PST) Date: Tue, 8 Nov 2022 18:41:59 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221108184200.2813458-5-calvinwan@google.com> Subject: [PATCH v4 4/5] diff-lib: refactor match_stat_with_submodule From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, myriamanis@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Flatten out the if statements in match_stat_with_submodule so the logic is more readable and easier for future patches to add to. orig_flags didn't need to be set if the cache entry wasn't a GITLINK so defer setting it. Signed-off-by: Calvin Wan --- diff-lib.c | 29 ++++++++++++++++++----------- 1 file changed, 18 insertions(+), 11 deletions(-) diff --git a/diff-lib.c b/diff-lib.c index 2edea41a23..f5257c0c71 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -73,18 +73,25 @@ static int match_stat_with_submodule(struct diff_options *diffopt, unsigned *dirty_submodule) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); - if (S_ISGITLINK(ce->ce_mode)) { - struct diff_flags orig_flags = diffopt->flags; - if (!diffopt->flags.override_submodule_config) - set_diffopt_flags_from_submodule_config(diffopt, ce->name); - if (diffopt->flags.ignore_submodules) - changed = 0; - else if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, - diffopt->flags.ignore_untracked_in_submodules); - diffopt->flags = orig_flags; + struct diff_flags orig_flags; + + if (!S_ISGITLINK(ce->ce_mode)) + goto ret; + + orig_flags = diffopt->flags; + if (!diffopt->flags.override_submodule_config) + set_diffopt_flags_from_submodule_config(diffopt, ce->name); + if (diffopt->flags.ignore_submodules) { + changed = 0; + goto cleanup; } + if (!diffopt->flags.ignore_dirty_submodules && + (!changed || diffopt->flags.dirty_submodules)) + *dirty_submodule = is_submodule_modified(ce->name, + diffopt->flags.ignore_untracked_in_submodules); +cleanup: + diffopt->flags = orig_flags; +ret: return changed; } From patchwork Tue Nov 8 18:42:00 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13036723 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE486C433FE for ; Tue, 8 Nov 2022 18:42:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231709AbiKHSmo (ORCPT ); Tue, 8 Nov 2022 13:42:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:52770 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231678AbiKHSm1 (ORCPT ); Tue, 8 Nov 2022 13:42:27 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 982A05F86E for ; Tue, 8 Nov 2022 10:42:25 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id l63-20020a639142000000b0046f5bbb7372so8314727pge.23 for ; Tue, 08 Nov 2022 10:42:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=g2t4oC9JaydK5pH9v+DX5+suSIU5L79UtLx8XGatGfU=; b=qFx7/uqUEcS+DRaLxHSx0S7ZLSS6Cu+chhSKKkTbUFJE87ycacRampAhv2vvHpV88T MTvmvIO9E1L/IjHoE4mXG7kL+cbj9vMMIulpdZa84gWSVMH/i699rCKOOFu/NorCVJ2M sXC+wG0/J7B9rVF7eUbYcppepfx05bMdMqUK+RQczgbaFMJ7Dc5JncGKiKv80pda0T3E UD5GFEcB3425e7RVFJ/kfvEnGMP8Qrpc3FrUhu8HShnMaMZWtVDGJpONUxHmjUGyNCq7 2eeD09R9APqqJAnUIegevTL9aXVe+thxW9c/uIpP4Xebxks2z61dpc7WH1aJoq6RtKxM Erxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=g2t4oC9JaydK5pH9v+DX5+suSIU5L79UtLx8XGatGfU=; b=z6boYxa4HoUqYdeF/HbDj58mDn9p4ik+1tsloEj1ABp8WuP+MwX4a+Ri/PMq8P+w/I leY/hIPL7fnG3em1y/ia/TpHIyrr+sPI+uR6PkR/fXKBsMMXKRbzzPVaTwuDWwmAt6s5 /r2sz/Ekf+3kDTgIWjIUsj+/kYH/ivyCt/ja0/XEHAv/S+yjQc+ssHENCmWN6zUOpbOP QEDB9rFb/lE6LRpsm2bNXsg/LXEKomdSEsTvv8wl7BAcAuuPI0/wNNaIs8wDgTK5MhRK skgWwhS6+Vt4QmNJ85OChNrZ7Z07AwQ4i83MMbt+wSLmzEJt8lpahMGs/Vbn6Ddzi6el cfpg== X-Gm-Message-State: ACrzQf2h0LBg3GJhp2dlwX/bzpDCwkmcIDsKmA0qYZ+5zCH5tHkG0X14 R4640bhCKiGswJZy2OttauO4owRdWU32VqwaYXbXkVP5DjohfU6Xj8vG4R6HvSF+MafRetlMJbY Cv8NMrgmjDfn7eiEu94m+V1LbH2px3ThELKkFxOzBT6XKFfK0C3kQfjDIqvHr4e46dQ== X-Google-Smtp-Source: AMsMyM59rO7BZfzmFDHZpe+0o4mhN4PS8msnwBIislUd3NdWXXS31PtZGRd/bfq9R1/vMfoyrxi9qHogozpYnvQ= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a05:6a00:2489:b0:56d:2d20:d01e with SMTP id c9-20020a056a00248900b0056d2d20d01emr54909897pfv.65.1667932945264; Tue, 08 Nov 2022 10:42:25 -0800 (PST) Date: Tue, 8 Nov 2022 18:42:00 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.38.1.431.g37b22c650d-goog Message-ID: <20221108184200.2813458-6-calvinwan@google.com> Subject: [PATCH v4 5/5] diff-lib: parallelize run_diff_files for submodules From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, myriamanis@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During the iteration of the index entries in run_diff_files, whenever a submodule is found and needs its status checked, a subprocess is spawned for it. Instead of spawning the subprocess immediately and waiting for its completion to continue, hold onto all submodules and relevant information in a list. Then use that list to create tasks for run_processes_parallel. Subprocess output is duplicated and passed to status_pipe_output which parses it. Add config option submodule.diffJobs to set the maximum number of parallel jobs. The option defaults to 1 if unset. If set to 0, the number of jobs is set to online_cpus(). Since run_diff_files is called from many different commands, I chose to grab the config option in the function rather than adding variables to every git command and then figuring out how to pass them all in. Signed-off-by: Calvin Wan --- Documentation/config/submodule.txt | 12 +++ diff-lib.c | 80 +++++++++++++-- submodule.c | 154 +++++++++++++++++++++++++++++ submodule.h | 9 ++ t/t4027-diff-submodule.sh | 19 ++++ t/t7506-status-submodule.sh | 19 ++++ 6 files changed, 287 insertions(+), 6 deletions(-) diff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt index 6490527b45..1144a5ad74 100644 --- a/Documentation/config/submodule.txt +++ b/Documentation/config/submodule.txt @@ -93,6 +93,18 @@ submodule.fetchJobs:: in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1. +submodule.diffJobs:: + Specifies how many submodules are diffed at the same time. A + positive integer allows up to that number of submodules diffed + in parallel. A value of 0 will give the number of logical cores. + If unset, it defaults to 1. The diff operation is used by many + other git commands such as add, merge, diff, status, stash and + more. Note that the expensive part of the diff operation is + reading the index from cache or memory. Therefore multiple jobs + may be detrimental to performance if your hardware does not + support parallel reads or if the number of jobs greatly exceeds + the amount of supported reads. + submodule.alternateLocation:: Specifies how the submodules obtain alternates when submodules are cloned. Possible values are `no`, `superproject`. diff --git a/diff-lib.c b/diff-lib.c index f5257c0c71..30a3d9a2b5 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -14,6 +14,7 @@ #include "dir.h" #include "fsmonitor.h" #include "commit-reach.h" +#include "config.h" /* * diff-files @@ -65,15 +66,20 @@ static int check_removed(const struct index_state *istate, const struct cache_en * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES * option is set, the caller does not only want to know if a submodule is * modified at all but wants to know all the conditions that are met (new - * commits, untracked content and/or modified content). + * commits, untracked content and/or modified content). If + * defer_submodule_status bit is set, dirty_submodule will be left to the + * caller to set. defer_submodule_status can also be set to 0 in this + * function if there is no need to check if the submodule is modified. */ static int match_stat_with_submodule(struct diff_options *diffopt, const struct cache_entry *ce, struct stat *st, unsigned ce_option, - unsigned *dirty_submodule) + unsigned *dirty_submodule, int *defer_submodule_status, + unsigned *ignore_untracked) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); struct diff_flags orig_flags; + int defer = 0; if (!S_ISGITLINK(ce->ce_mode)) goto ret; @@ -86,12 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt, goto cleanup; } if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, + (!changed || diffopt->flags.dirty_submodules)) { + if (defer_submodule_status && *defer_submodule_status) { + defer = 1; + *ignore_untracked = diffopt->flags.ignore_untracked_in_submodules; + } else { + *dirty_submodule = is_submodule_modified(ce->name, diffopt->flags.ignore_untracked_in_submodules); + } + } cleanup: diffopt->flags = orig_flags; ret: + if (defer_submodule_status) + *defer_submodule_status = defer; return changed; } @@ -103,6 +117,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ? CE_MATCH_RACY_IS_DIRTY : 0); uint64_t start = getnanotime(); struct index_state *istate = revs->diffopt.repo->index; + struct string_list submodules = STRING_LIST_INIT_NODUP; diff_set_mnemonic_prefix(&revs->diffopt, "i/", "w/"); @@ -227,6 +242,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option) newmode = ce->ce_mode; } else { struct stat st; + unsigned ignore_untracked = 0; + int defer_submodule_status = !!revs->repo; changed = check_removed(istate, ce, &st); if (changed) { @@ -248,8 +265,25 @@ int run_diff_files(struct rev_info *revs, unsigned int option) } changed = match_stat_with_submodule(&revs->diffopt, ce, &st, - ce_option, &dirty_submodule); + ce_option, &dirty_submodule, + &defer_submodule_status, + &ignore_untracked); newmode = ce_mode_from_stat(ce, st.st_mode); + if (defer_submodule_status) { + struct submodule_status_util tmp = { + .changed = changed, + .dirty_submodule = 0, + .ignore_untracked = ignore_untracked, + .newmode = newmode, + .ce = ce, + }; + struct string_list_item *item; + + item = string_list_append(&submodules, ce->name); + item->util = xmalloc(sizeof(tmp)); + memcpy(item->util, &tmp, sizeof(tmp)); + continue; + } } if (!changed && !dirty_submodule) { @@ -268,6 +302,40 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ce->name, 0, dirty_submodule); } + if (submodules.nr > 0) { + int parallel_jobs; + if (git_config_get_int("submodule.diffjobs", ¶llel_jobs)) + parallel_jobs = 1; + else if (!parallel_jobs) + parallel_jobs = online_cpus(); + else if (parallel_jobs < 0) + die(_("submodule.diffjobs cannot be negative")); + + if (get_submodules_status(revs->repo, &submodules, parallel_jobs)) + die(_("submodule status failed")); + for (size_t i = 0; i < submodules.nr; i++) { + struct submodule_status_util *util = submodules.items[i].util; + struct cache_entry *ce = util->ce; + unsigned int oldmode; + const struct object_id *old_oid, *new_oid; + + if (!util->changed && !util->dirty_submodule) { + ce_mark_uptodate(ce); + mark_fsmonitor_valid(istate, ce); + if (!revs->diffopt.flags.find_copies_harder) + continue; + } + oldmode = ce->ce_mode; + old_oid = &ce->oid; + new_oid = util->changed ? null_oid() : &ce->oid; + diff_change(&revs->diffopt, oldmode, util->newmode, + old_oid, new_oid, + !is_null_oid(old_oid), + !is_null_oid(new_oid), + ce->name, 0, util->dirty_submodule); + } + } + string_list_clear(&submodules, 1); diffcore_std(&revs->diffopt); diff_flush(&revs->diffopt); trace_performance_since(start, "diff-files"); @@ -315,7 +383,7 @@ static int get_stat_data(const struct index_state *istate, return -1; } changed = match_stat_with_submodule(diffopt, ce, &st, - 0, dirty_submodule); + 0, dirty_submodule, NULL, NULL); if (changed) { mode = ce_mode_from_stat(ce, st.st_mode); oid = null_oid(); diff --git a/submodule.c b/submodule.c index fd3385620c..763a05d523 100644 --- a/submodule.c +++ b/submodule.c @@ -1363,6 +1363,18 @@ int submodule_touches_in_range(struct repository *r, return ret; } +struct submodule_parallel_status { + size_t index_count; + int result; + + struct string_list *submodule_names; + struct repository *r; + + /* Pending statuses by OIDs */ + struct status_task **oid_status_tasks; + int oid_status_tasks_nr, oid_status_tasks_alloc; +}; + struct submodule_parallel_fetch { /* * The index of the last index entry processed by @@ -1445,6 +1457,12 @@ struct fetch_task { struct oid_array *commits; /* Ensure these commits are fetched */ }; +struct status_task { + const char *path; + unsigned dirty_submodule; + int ignore_untracked; +}; + /** * When a submodule is not defined in .gitmodules, we cannot access it * via the regular submodule-config. Create a fake submodule, which we can @@ -1956,6 +1974,142 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) return dirty_submodule; } +static struct status_task * +get_status_task_from_index(struct submodule_parallel_status *sps, + struct strbuf *err) +{ + for (; sps->index_count < sps->submodule_names->nr; sps->index_count++) { + struct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util; + const struct cache_entry *ce = util->ce; + struct status_task *task; + struct status_task tmp = { + .path = ce->name, + .dirty_submodule = util->dirty_submodule, + .ignore_untracked = util->ignore_untracked, + }; + struct strbuf buf = STRBUF_INIT; + const char *git_dir; + + strbuf_addf(&buf, "%s/.git", ce->name); + git_dir = read_gitfile(buf.buf); + if (!git_dir) + git_dir = buf.buf; + if (!is_git_directory(git_dir)) { + if (is_directory(git_dir)) + die(_("'%s' not recognized as a git repository"), git_dir); + strbuf_release(&buf); + /* The submodule is not checked out, so it is not modified */ + util->dirty_submodule = 0; + continue; + } + strbuf_release(&buf); + + task = xmalloc(sizeof(*task)); + memcpy(task, &tmp, sizeof(*task)); + sps->index_count++; + return task; + } + return NULL; +} + + +static int get_next_submodule_status(struct child_process *cp, + struct strbuf *err, void *data, + void **task_cb) +{ + struct submodule_parallel_status *sps = data; + struct status_task *task = get_status_task_from_index(sps, err); + + if (!task) + return 0; + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env); + + strvec_init(&cp->args); + strvec_pushl(&cp->args, "status", "--porcelain=2", NULL); + if (task->ignore_untracked) + strvec_push(&cp->args, "-uno"); + + prepare_submodule_repo_env(&cp->env); + cp->git_cmd = 1; + cp->dir = task->path; + *task_cb = task; + return 1; +} + +static int status_start_failure(struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + + sps->result = 1; + return 0; +} + +static void status_duplicate_output(struct strbuf *process_out, + struct strbuf *out, + void *cb, void *task_cb) +{ + struct status_task *task = task_cb; + struct string_list list = STRING_LIST_INIT_DUP; + struct string_list_item *item; + + string_list_split(&list, process_out->buf, '\n', -1); + + for_each_string_list_item(item, &list) { + if (parse_status_porcelain(item->string, + strlen(item->string), + &task->dirty_submodule, + task->ignore_untracked)) + break; + } + string_list_clear(&list, 0); + strbuf_reset(out); +} + +static int status_finish(int retvalue, struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + struct string_list_item *it = + string_list_lookup(sps->submodule_names, task->path); + struct submodule_status_util *util = it->util; + + util->dirty_submodule = task->dirty_submodule; + free(task); + + return 0; +} + +int get_submodules_status(struct repository *r, + struct string_list *submodules, + int max_parallel_jobs) +{ + struct submodule_parallel_status sps = { + .r = r, + .submodule_names = submodules, + }; + const struct run_process_parallel_opts opts = { + .tr2_category = "submodule", + .tr2_label = "parallel/status", + + .processes = max_parallel_jobs, + + .get_next_task = get_next_submodule_status, + .start_failure = status_start_failure, + .duplicate_output = status_duplicate_output, + .task_finished = status_finish, + .data = &sps, + }; + + string_list_sort(sps.submodule_names); + run_processes_parallel(&opts); + + return sps.result; +} + int submodule_uses_gitfile(const char *path) { struct child_process cp = CHILD_PROCESS_INIT; diff --git a/submodule.h b/submodule.h index 6a9fec6de1..cbb7231a5d 100644 --- a/submodule.h +++ b/submodule.h @@ -41,6 +41,12 @@ struct submodule_update_strategy { .type = SM_UPDATE_UNSPECIFIED, \ } +struct submodule_status_util { + int changed, ignore_untracked; + unsigned dirty_submodule, newmode; + struct cache_entry *ce; +}; + int is_gitmodules_unmerged(struct index_state *istate); int is_writing_gitmodules_ok(void); int is_staging_gitmodules_ok(struct index_state *istate); @@ -94,6 +100,9 @@ int fetch_submodules(struct repository *r, int command_line_option, int default_option, int quiet, int max_parallel_jobs); +int get_submodules_status(struct repository *r, + struct string_list *submodules, + int max_parallel_jobs); unsigned is_submodule_modified(const char *path, int ignore_untracked); int submodule_uses_gitfile(const char *path); diff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh index 40164ae07d..e08ee315a7 100755 --- a/t/t4027-diff-submodule.sh +++ b/t/t4027-diff-submodule.sh @@ -34,6 +34,25 @@ test_expect_success setup ' subtip=$3 subprev=$2 ' +test_expect_success 'diff in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git diff && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git diff && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_expect_success 'git diff --raw HEAD' ' hexsz=$(test_oid hexsz) && git diff --raw --abbrev=$hexsz HEAD >actual && diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh index d050091345..52a82b703f 100755 --- a/t/t7506-status-submodule.sh +++ b/t/t7506-status-submodule.sh @@ -412,4 +412,23 @@ test_expect_success 'status with added file in nested submodule (short)' ' EOF ' +test_expect_success 'status in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git status && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git status && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_done