From patchwork Tue Feb 7 18:17:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13131959 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7C4B1C6379F for ; Tue, 7 Feb 2023 18:18:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231929AbjBGSSB (ORCPT ); Tue, 7 Feb 2023 13:18:01 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40338 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231309AbjBGSRe (ORCPT ); Tue, 7 Feb 2023 13:17:34 -0500 Received: from mail-yw1-x1149.google.com (mail-yw1-x1149.google.com [IPv6:2607:f8b0:4864:20::1149]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 210F3EB71 for ; Tue, 7 Feb 2023 10:17:18 -0800 (PST) Received: by mail-yw1-x1149.google.com with SMTP id 00721157ae682-5261de2841fso110457767b3.7 for ; Tue, 07 Feb 2023 10:17:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=vUpIJZH3ue95F84uqowtHCl+qgA9d0kRegn+UN/ED78=; b=IOknm6hlGm7nvrBM6PP8rNVgh8vZtKWcm5Axm0cICbhp5QReQfs7E/j61dnKcFctm0 xKMC100q0onlxq4XEzLzwE3Y8XB6hBzKa+YKEADiXMUs9hk0odXS2vv2lVZ+Y7Q5amnx aV09w9lO+GSMf86SglqTJ4HZNr2dC3DfQd3uw7ey+F1RnMQ4YTMwlJ1iLSP54rceF2KM 3SOrjncKWNJi3Jg3tkorCyxKPkDOJipE7sf/gu8SJlko/Um7oEBPCV2yUE/C18qyV675 cgBbqlYkNQbP6c1OPnaBpdr7yueFaaoexG1uGvNoktNS9POaRDM94xDpXv4p0YeStKdW lTxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=vUpIJZH3ue95F84uqowtHCl+qgA9d0kRegn+UN/ED78=; b=xF7mxSPO+rRihUGZ++U7pGdWNqm8FMXG/8r1J7x2xePxht7IcDQIy0/2BRBsOjJSqJ 1Y4oPxWE6CYPiqsgbL3Ycf1eUM5TdkcazIYM969CXkzjR9EYOdo/3QvheSAcOPH15pXc V5mT08G2/1CPd9sX52hrfgYZYtnxSVXAfKzRPjXkpUXzJQFP1957vbpCbdF8iPE+HV+9 YBEOe5rnHaRAZ5/v7iGNp9A3WCihgvAvWzh8dILVbcrWTtdblPO3iWaVoqYAVGHeicFu wv9IuleB7ZZH2gnxg+3ktdw9E6nsCZilsuI3+Kkavr6r6Z0PVq3AaDNrM9Xo7it98Juc gd/Q== X-Gm-Message-State: AO0yUKWH1+ESNTYVCLzQhPa/C+qaWt+mZLsMxlyQSjX4Q8uLDMTxK5lI O/miFBvql2bCaVQ9XvyDu23Dt/krbP+GNod2SKQ8KSnbXCds76UQGv5oRotJMh+AWqqGjyJaRKg hMa+ytIZUlPri5Us2sjYMopG5EM1lPpp4iHNckXzBZt1SifHEmRlSG6piSORi7Ajvdw== X-Google-Smtp-Source: AK7set93r17bqz4gAbqtHp9NLXuH2bnCKYeTTQCZ+1j5Ui4aFg2zEOEDvPjeK5Gi2nGJ/jnmT3F3Nan31w/saY8= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a5b:c12:0:b0:8b6:a289:24a6 with SMTP id f18-20020a5b0c12000000b008b6a28924a6mr129953ybq.569.1675793836947; Tue, 07 Feb 2023 10:17:16 -0800 (PST) Date: Tue, 7 Feb 2023 18:17:00 +0000 In-Reply-To: <20230117193041.708692-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230117193041.708692-1-calvinwan@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230207181706.363453-2-calvinwan@google.com> Subject: [PATCH v7 1/7] run-command: add duplicate_output_fn to run_processes_parallel_opts From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add duplicate_output_fn as an optionally set function in run_process_parallel_opts. If set, output from each child process is copied and passed to the callback function whenever output from the child process is buffered to allow for separate parsing. Signed-off-by: Calvin Wan --- run-command.c | 16 ++++++++++++--- run-command.h | 27 +++++++++++++++++++++++++ t/helper/test-run-command.c | 21 ++++++++++++++++++++ t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++ 4 files changed, 100 insertions(+), 3 deletions(-) diff --git a/run-command.c b/run-command.c index 756f1839aa..cad88befe0 100644 --- a/run-command.c +++ b/run-command.c @@ -1526,6 +1526,9 @@ static void pp_init(struct parallel_processes *pp, if (!opts->get_next_task) BUG("you need to specify a get_next_task function"); + if (opts->duplicate_output && opts->ungroup) + BUG("duplicate_output and ungroup are incompatible with each other"); + CALLOC_ARRAY(pp->children, n); if (!opts->ungroup) CALLOC_ARRAY(pp->pfd, n); @@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp, for (size_t i = 0; i < opts->processes; i++) { if (pp->children[i].state == GIT_CP_WORKING && pp->pfd[i].revents & (POLLIN | POLLHUP)) { - int n = strbuf_read_once(&pp->children[i].err, - pp->children[i].process.err, 0); + ssize_t n = strbuf_read_once(&pp->children[i].err, + pp->children[i].process.err, 0); if (n == 0) { close(pp->children[i].process.err); pp->children[i].state = GIT_CP_WAIT_CLEANUP; - } else if (n < 0) + } else if (n < 0) { if (errno != EAGAIN) die_errno("read"); + } else { + if (opts->duplicate_output) + opts->duplicate_output(&pp->children[i].err, + strlen(pp->children[i].err.buf) - n, + opts->data, + pp->children[i].data); + } } } } diff --git a/run-command.h b/run-command.h index 072db56a4d..6dcf999f6c 100644 --- a/run-command.h +++ b/run-command.h @@ -408,6 +408,27 @@ typedef int (*start_failure_fn)(struct strbuf *out, void *pp_cb, void *pp_task_cb); +/** + * This callback is called whenever output from a child process is buffered + * + * See run_processes_parallel() below for a discussion of the "struct + * strbuf *out" parameter. + * + * The offset refers to the number of bytes originally in "out" before + * the output from the child process was buffered. Therefore, the buffer + * range, "out + buf" to the end of "out", would contain the buffer of + * the child process output. + * + * pp_cb is the callback cookie as passed into run_processes_parallel, + * pp_task_cb is the callback cookie as passed into get_next_task_fn. + * + * This function is incompatible with "ungroup" + */ +typedef void (*duplicate_output_fn)(struct strbuf *out, + size_t offset, + void *pp_cb, + void *pp_task_cb); + /** * This callback is called on every child process that finished processing. * @@ -461,6 +482,12 @@ struct run_process_parallel_opts */ start_failure_fn start_failure; + /** + * duplicate_output: See duplicate_output_fn() above. This should be + * NULL unless process specific output is needed + */ + duplicate_output_fn duplicate_output; + /** * task_finished: See task_finished_fn() above. This can be * NULL to omit any special handling. diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c index 3ecb830f4a..ffd3cd0045 100644 --- a/t/helper/test-run-command.c +++ b/t/helper/test-run-command.c @@ -52,6 +52,21 @@ static int no_job(struct child_process *cp, return 0; } +static void duplicate_output(struct strbuf *out, + size_t offset, + void *pp_cb UNUSED, + void *pp_task_cb UNUSED) +{ + struct string_list list = STRING_LIST_INIT_DUP; + + string_list_split(&list, out->buf + offset, '\n', -1); + for (size_t i = 0; i < list.nr; i++) { + if (strlen(list.items[i].string) > 0) + fprintf(stderr, "duplicate_output: %s\n", list.items[i].string); + } + string_list_clear(&list, 0); +} + static int task_finished(int result, struct strbuf *err, void *pp_cb, @@ -439,6 +454,12 @@ int cmd__run_command(int argc, const char **argv) opts.ungroup = 1; } + if (!strcmp(argv[1], "--duplicate-output")) { + argv += 1; + argc -= 1; + opts.duplicate_output = duplicate_output; + } + jobs = atoi(argv[2]); strvec_clear(&proc.args); strvec_pushv(&proc.args, (const char **)argv + 3); diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh index e2411f6a9b..879e536638 100755 --- a/t/t0061-run-command.sh +++ b/t/t0061-run-command.sh @@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' ' test-tool run-command --ungroup run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' ' test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' ' test-tool run-command --ungroup run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' ' test-tool run-command --ungroup run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' ' test_cmp expect actual ' +test_expect_success 'run_command is asked to abort gracefully --duplicate-output' ' + test-tool run-command --duplicate-output run-command-abort 3 false >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command is asked to abort gracefully (ungroup)' ' test-tool run-command --ungroup run-command-abort 3 false >out 2>err && test_must_be_empty out && @@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' ' test_cmp expect actual ' +test_expect_success 'run_command outputs --duplicate-output' ' + test-tool run-command --duplicate-output run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command outputs (ungroup) ' ' test-tool run-command --ungroup run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_must_be_empty out && From patchwork Tue Feb 7 18:17:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13131960 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7778C636CC for ; Tue, 7 Feb 2023 18:18:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232026AbjBGSSF (ORCPT ); Tue, 7 Feb 2023 13:18:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40340 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232038AbjBGSRe (ORCPT ); Tue, 7 Feb 2023 13:17:34 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BF8B63B65E for ; Tue, 7 Feb 2023 10:17:19 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-527b1358200so68143987b3.13 for ; Tue, 07 Feb 2023 10:17:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=K46XHE2BLqdDbnyWSR0bU7t4XlcU/9xs6xJD00qC73Y=; b=at7YRqjz3SHiQJe8AZ1SoSxptRkeLigsj5bmrPaSgdELCZHGMLdJ9SjI/DMzWvVMrY nYPCKaBffsCAxPKs73flxXti49DBeQ8Cvea2e6FtqC/PhS+7+7FSsx76xd16SVeeIWd1 lHFrVTIfIAs67FMfJClzvfWlVkFmTsWcMZbYDVxNNx5pomiq7Ony0q7xsLC7qxaz0pE4 dUG1zy9otfLzADqG2K3kBeOWfTbiLzlMh2Utjvq91aLZQraNIgjaZHCe7fHrrfD6Ip2k FbBx91xLRmGaYwqiJhHEIo3WbrEPIsVa3/JPZMz2ADsevh80HdLc9tQ1OyfJb8cfZmDK mPWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=K46XHE2BLqdDbnyWSR0bU7t4XlcU/9xs6xJD00qC73Y=; b=JK/mS/GnTAafEGDviGSACnKgMSyghBbdQdzESuiFrDc481Snh0hVgy5pKDmWWkIOWS b22RG8BgZZQML7DckM/2gZzn2VhRvrZWXKz/VvYnq90h2bLPPjBuUb2/B8i8/usYzVzz n6JdFI2CZvqpIF/dxlNPrIxS9P1+pDk/xVikgxywfxQjQ4eLEjGOo5XMjj7IQuwSvX7X LSP+KtaxdC3HcxZsK3VEj8Jw2o8rgxRpRFp80PYbcxPxi80C73Ze9S0FxhDSULixboNL LzUcivr9H3VkgBV5ubLauv0FmTuc2aZe+A5VpLn6Oco3GqR4bY5SclOVvagRwb77hbj9 KXWw== X-Gm-Message-State: AO0yUKU+/bUtKVm5mcsrMjuCZAzu5STXq+xzRZ12yU9bikXCRVCd49VI WnBT0ZWbJhe00qgo5cejgM9qXMuY6yUDrG7Gn6swaloOlR6vsvd1/zlqGaLPtwpdkHrggUucdX7 5I/iiGLWlCL8wv869XMXJxCEmcJR2D/QjtbnwDC9PhLm5U9lk/hhO8e7tTfTWqdpmkw== X-Google-Smtp-Source: AK7set8WEgskc8fZUL5wQDI3umWME55Bv0N5HySKKy7v6mgvGT2r9bpgSFnUK5qTnwAOhAmU8rAbHlmvqwJJJ+Q= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a5b:c12:0:b0:8b6:a289:24a6 with SMTP id f18-20020a5b0c12000000b008b6a28924a6mr129966ybq.569.1675793838705; Tue, 07 Feb 2023 10:17:18 -0800 (PST) Date: Tue, 7 Feb 2023 18:17:01 +0000 In-Reply-To: <20230117193041.708692-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230117193041.708692-1-calvinwan@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230207181706.363453-3-calvinwan@google.com> Subject: [PATCH v7 2/7] submodule: strbuf variable rename From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A prepatory change for a future patch that moves the status parsing logic to a separate function. Signed-off-by: Calvin Wan --- submodule.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/submodule.c b/submodule.c index fae24ef34a..faf37c1101 100644 --- a/submodule.c +++ b/submodule.c @@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) fp = xfdopen(cp.out, "r"); while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { + char *str = buf.buf; + const size_t len = buf.len; + /* regular untracked files */ - if (buf.buf[0] == '?') + if (str[0] == '?') dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '1' || - buf.buf[0] == '2') { + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { /* T = line type, XY = status, SSSS = submodule state */ - if (buf.len < strlen("T XY SSSS")) + if (len < strlen("T XY SSSS")) BUG("invalid status --porcelain=2 line %s", - buf.buf); + str); - if (buf.buf[5] == 'S' && buf.buf[8] == 'U') + if (str[5] == 'S' && str[8] == 'U') /* nested untracked file */ dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '2' || - memcmp(buf.buf + 5, "S..U", 4)) + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) /* other change */ dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; } From patchwork Tue Feb 7 18:17:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13131961 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B5FCC636D3 for ; Tue, 7 Feb 2023 18:18:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232138AbjBGSSJ (ORCPT ); Tue, 7 Feb 2023 13:18:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230260AbjBGSRh (ORCPT ); Tue, 7 Feb 2023 13:17:37 -0500 Received: from mail-yb1-xb4a.google.com (mail-yb1-xb4a.google.com [IPv6:2607:f8b0:4864:20::b4a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B97583D080 for ; Tue, 7 Feb 2023 10:17:21 -0800 (PST) Received: by mail-yb1-xb4a.google.com with SMTP id q133-20020a25d98b000000b0089ab1a9f0fdso5973294ybg.1 for ; Tue, 07 Feb 2023 10:17:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=xcCDJ/9InDiBfm3IVYXcLEw2c7g9o9VhW+yu1jBDgF0=; b=XINfUpgxlb7xYljqJyK6pZRvLj0USdtSA2nfKtCkBUWfn+v5url1e7WHO3zFHkfk6a Cj7M8AqiWUoPuwe2ykMnEUFaw9rP3qJjFTXRYxF7zjoLhXO2R9i9YaRFjnu4X9Ml1Xmw 8qdoGcIFgoaM0f82vPEra5X0SyUq44kTpX+zl5+FWORa+dBekrzTcecfcTkYDGuL/J/x hRnu7SyPpI4DNSWNe8gE+Db9aS9+YTcQxAO1Ig/BCqOsdia62fpAzMxALMnfmozSWDra IcExwDKqh5Oo+VAYdOLly1JGhHr1dpTsgAkbl1oiTCP0Mv7P0mt40qqmUbzEBVKPwAHa EGHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=xcCDJ/9InDiBfm3IVYXcLEw2c7g9o9VhW+yu1jBDgF0=; b=wvOyNJoWGUE9mLAWar4H8Ef9jAByRAZLEMRAHA8ZGKhz9gGGQf15p3Kqqh7HW9MUy2 vUZKSjFzvXgG3cuSYMlHiw0ckbkdOpugWLIL5cpAnonZ2jPYEjmkDCvoNsWr+RaeSJRY wrrnIzGLHWszEe4Wxj8Gi4lWfS5DWaNg8UCfVzhrPOUys2EPrVgJ1SL3qKm4qgos+IHh p9buqLVykWfyKPozo6WpojliZgtKk63Zcbb0oPduMKj+NECszq5wLevzQ2p0eU3f/Ia1 4Cu6/6LT4LYkoO1gxy8idfN4Oyr++TK48fTQvWoe2RuGuezw1udGC8px3EMBnhMgXV/d SlxA== X-Gm-Message-State: AO0yUKWlkzywB376uA6KOYrbsEkxRxEDjsVfPwxYMfKUKwDEFAxxwPTm dkWGQH/fjADDu3E5xmONJQmfWrS0y2f419T4MfFnZG7SKZMq93eMwGqG3ohGYLrEqUa0mrP9v0f o9kHtrfW1BC/ufPlf4Gp/ZlF5dSUFq/jWpOqyFfOxBfu/Sb/rmFSpG6zSqxRBFaD12A== X-Google-Smtp-Source: AK7set9PQyEL9U2mmPK6KITBOFAM5DMEN01xKmkP+1YiZ1y60X/1Upz2WEgR7fVWROsFUOx51wRnkoO2tvwfXlM= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a05:6902:3d0:b0:837:b2e9:9cfc with SMTP id g16-20020a05690203d000b00837b2e99cfcmr445764ybs.637.1675793840532; Tue, 07 Feb 2023 10:17:20 -0800 (PST) Date: Tue, 7 Feb 2023 18:17:02 +0000 In-Reply-To: <20230117193041.708692-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230117193041.708692-1-calvinwan@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230207181706.363453-4-calvinwan@google.com> Subject: [PATCH v7 3/7] submodule: move status parsing into function From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A future patch requires the ability to parse the output of git status --porcelain=2. Move parsing code from is_submodule_modified to parse_status_porcelain. Signed-off-by: Calvin Wan --- submodule.c | 74 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 32 deletions(-) diff --git a/submodule.c b/submodule.c index faf37c1101..768d4b4cd7 100644 --- a/submodule.c +++ b/submodule.c @@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r, return spf.result; } +static int parse_status_porcelain(char *str, size_t len, + unsigned *dirty_submodule, + int ignore_untracked) +{ + /* regular untracked files */ + if (str[0] == '?') + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { + /* T = line type, XY = status, SSSS = submodule state */ + if (len < strlen("T XY SSSS")) + BUG("invalid status --porcelain=2 line %s", + str); + + if (str[5] == 'S' && str[8] == 'U') + /* nested untracked file */ + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) + /* other change */ + *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; + } + + if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && + ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || + ignore_untracked)) { + /* + * We're not interested in any further information from + * the child any more, neither output nor its exit code. + */ + return 1; + } + return 0; +} + unsigned is_submodule_modified(const char *path, int ignore_untracked) { struct child_process cp = CHILD_PROCESS_INIT; @@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) char *str = buf.buf; const size_t len = buf.len; - /* regular untracked files */ - if (str[0] == '?') - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '1' || - str[0] == '2') { - /* T = line type, XY = status, SSSS = submodule state */ - if (len < strlen("T XY SSSS")) - BUG("invalid status --porcelain=2 line %s", - str); - - if (str[5] == 'S' && str[8] == 'U') - /* nested untracked file */ - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '2' || - memcmp(str + 5, "S..U", 4)) - /* other change */ - dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; - } - - if ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && - ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || - ignore_untracked)) { - /* - * We're not interested in any further information from - * the child any more, neither output nor its exit code. - */ - ignore_cp_exit_code = 1; + ignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule, + ignore_untracked); + if (ignore_cp_exit_code) break; - } } fclose(fp); From patchwork Tue Feb 7 18:17:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13131962 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C642C636D3 for ; Tue, 7 Feb 2023 18:18:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231390AbjBGSSb (ORCPT ); Tue, 7 Feb 2023 13:18:31 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231843AbjBGSRo (ORCPT ); Tue, 7 Feb 2023 13:17:44 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 88B833E60A for ; Tue, 7 Feb 2023 10:17:24 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id x3-20020a654143000000b004f2f2dafb21so5932971pgp.0 for ; Tue, 07 Feb 2023 10:17:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UTXNmyRG9NzKoSWvWYTY4/IA2uob49GSwxXBgqWy+Mk=; b=ZmqRARlbakQ0MqP+VqGksXrURUd4BGdFhbM0RITepbAYAmJNObGDD2EJfU2lT6UzkV QiBvFktkRgAT/aqUspKyO9NddzU7QP9KNEiOre6xIcBWBqIAzEmiPZ0ywrxo6sPNMtOc rnd33QeP6Kw9Y2wUd5lhzDC3eAY1hTE6h/e4dbVUNGOSRknySazegpNA50ab8Pfzsh60 5NbHmlC4J4BBB6YXifUOLIoyVhzMKW8F77729QSY0DwvMiE4YfXIl/JUj7Wo39t04J+i f6tfybgbZJY6YOrE45qgRSL3+nV4pmVONUP8MCTdENI3vMZNvAsWT1/Z38Qrz7l7ybnO a+uA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UTXNmyRG9NzKoSWvWYTY4/IA2uob49GSwxXBgqWy+Mk=; b=00cUP03JVeXcLc7bql5SnNhFFcu+jKJczFQV9sRHbOwac7QYckFQHZw5yqKB0v1IXC +n8RHFrot8H7chjJmbulfrAttLQhJ/V/Hdi0DQLFC4TBS2vBwna5ZmLqRqC2RYkSeG0m UbpQTH4yVDSFd3DsvD98l9fWkEMcRl/NYTMu9UjgSRmffXx5STF4Dj5rjxZUUKC4eenf /BQRq+b7lny96Kn4FBSZ+h0HcxTfW3jqSH4Lgl+xoigZHiUGfp3BiElabYaw95QZhQyZ C810mGgngyuKf5fY2aexIuRjBAyCPRd5M+j/AiST9SH+TMC7PnQLC6W8a2hAv1BjMqDa rOCw== X-Gm-Message-State: AO0yUKVmLz7VTRYBVVF7rJ8ZbZg8IhmjVYDXeT0claE4LR7PovXmZmfB unZ1rZCrQDReqFwqg7FALJ4O8cLX1Fe+ruO4O4b/yXEcvIWLPbt8htQx++LOthnfJFbWuLmcaMZ ybg7+mnsRc93ApQM6HQPGUMrE3vHm1zjHESClrxeOH8F8B687thpGNCCptQIPxZsZQw== X-Google-Smtp-Source: AK7set+eGLNYRZCNv4cDhy1epV4VUrb06FHgVDWWTC5OY7PQHJtZw6dJTjGAdMn4hQCtMdwFzEsAaCI2R+Un4WQ= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:90a:f684:b0:230:ddd8:869b with SMTP id cl4-20020a17090af68400b00230ddd8869bmr1366563pjb.87.1675793842549; Tue, 07 Feb 2023 10:17:22 -0800 (PST) Date: Tue, 7 Feb 2023 18:17:03 +0000 In-Reply-To: <20230117193041.708692-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230117193041.708692-1-calvinwan@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230207181706.363453-5-calvinwan@google.com> Subject: [PATCH v7 4/7] submodule: refactor is_submodule_modified() From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor out submodule status logic and error messages that will be used in a future patch. Signed-off-by: Calvin Wan --- submodule.c | 65 ++++++++++++++++++++++++++++++++++------------------- 1 file changed, 42 insertions(+), 23 deletions(-) diff --git a/submodule.c b/submodule.c index 768d4b4cd7..d88aa2c573 100644 --- a/submodule.c +++ b/submodule.c @@ -28,6 +28,10 @@ static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF; static int initialized_fetch_ref_tips; static struct oid_array ref_tips_before_fetch; static struct oid_array ref_tips_after_fetch; +static const char *status_porcelain_start_error = + N_("could not run 'git status --porcelain=2' in submodule %s"); +static const char *status_porcelain_fail_error = + N_("'git status --porcelain=2' failed in submodule %s"); /* * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file @@ -1870,6 +1874,40 @@ int fetch_submodules(struct repository *r, return spf.result; } +static int verify_submodule_git_directory(const char *path) +{ + const char *git_dir; + struct strbuf buf = STRBUF_INIT; + + strbuf_addf(&buf, "%s/.git", path); + git_dir = read_gitfile(buf.buf); + if (!git_dir) + git_dir = buf.buf; + if (!is_git_directory(git_dir)) { + if (is_directory(git_dir)) + die(_("'%s' not recognized as a git repository"), git_dir); + strbuf_release(&buf); + /* The submodule is not checked out, so it is not modified */ + return 0; + } + strbuf_release(&buf); + return 1; +} + +static void prepare_status_porcelain(struct child_process *cp, + const char *path, int ignore_untracked) +{ + strvec_pushl(&cp->args, "status", "--porcelain=2", NULL); + if (ignore_untracked) + strvec_push(&cp->args, "-uno"); + + prepare_submodule_repo_env(&cp->env); + cp->git_cmd = 1; + cp->no_stdin = 1; + cp->out = -1; + cp->dir = path; +} + static int parse_status_porcelain(char *str, size_t len, unsigned *dirty_submodule, int ignore_untracked) @@ -1915,33 +1953,14 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) struct strbuf buf = STRBUF_INIT; FILE *fp; unsigned dirty_submodule = 0; - const char *git_dir; int ignore_cp_exit_code = 0; - strbuf_addf(&buf, "%s/.git", path); - git_dir = read_gitfile(buf.buf); - if (!git_dir) - git_dir = buf.buf; - if (!is_git_directory(git_dir)) { - if (is_directory(git_dir)) - die(_("'%s' not recognized as a git repository"), git_dir); - strbuf_release(&buf); - /* The submodule is not checked out, so it is not modified */ + if (!verify_submodule_git_directory(path)) return 0; - } - strbuf_reset(&buf); - - strvec_pushl(&cp.args, "status", "--porcelain=2", NULL); - if (ignore_untracked) - strvec_push(&cp.args, "-uno"); - prepare_submodule_repo_env(&cp.env); - cp.git_cmd = 1; - cp.no_stdin = 1; - cp.out = -1; - cp.dir = path; + prepare_status_porcelain(&cp, path, ignore_untracked); if (start_command(&cp)) - die(_("Could not run 'git status --porcelain=2' in submodule %s"), path); + die(_(status_porcelain_start_error), path); fp = xfdopen(cp.out, "r"); while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { @@ -1956,7 +1975,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) fclose(fp); if (finish_command(&cp) && !ignore_cp_exit_code) - die(_("'git status --porcelain=2' failed in submodule %s"), path); + die(_(status_porcelain_fail_error), path); strbuf_release(&buf); return dirty_submodule; From patchwork Tue Feb 7 18:17:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13131963 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 12B99C636CC for ; Tue, 7 Feb 2023 18:18:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232185AbjBGSSd (ORCPT ); Tue, 7 Feb 2023 13:18:33 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41596 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230204AbjBGSRm (ORCPT ); Tue, 7 Feb 2023 13:17:42 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 93A433D083 for ; Tue, 7 Feb 2023 10:17:24 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id s20-20020a17090b071400b00230ecf76407so1280765pjz.3 for ; Tue, 07 Feb 2023 10:17:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZYwZxXuZCKch3x+Vy3R4rbmwqBPB6ajawKXYoGOyBuA=; b=MdDbQNiq7IjnFetyJLPTQXKwkT+WH9LieGM10L/jyQrPHKa12f9xJtP1PieAd6tKGR /5ojHFfbWKBjXsCHKDMwW5X35MrhvMnZOvQonIBaZ9yuXFwrPo5sq4Yw4DcB7eLBS91G KouRxIsa3K/P1eBmxMqy5yvaYCcflBFW7v8wyioCnVScWXWEeiBL4SEdbQA/JN8mcKNP p8iixey7zjmI7BwlphD4RTvDSa9ZlJgvmDZJe+kQYPvhncwPnyRWGmuLkEulvkkjHFc+ 2V+TzMtTDkSUFRvAY1zuoRrml6yvCG8ym3zU/KwHuy+tFlfuOIEa4BXEM50ckX6lUtFU PrSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZYwZxXuZCKch3x+Vy3R4rbmwqBPB6ajawKXYoGOyBuA=; b=2kphz1VYug+6SbQiDflm0BGLxiyQ8cY1xq5t1/QW5qMYGbJMzbGrjrqgTyf/ARbTJi nE4cd/xZjTarilA4woGsPlmM3dWnaADfQ6sXHC88K1qb2YjfjecmJGONCEBwLrjytY5p HRpDMoViGrdAoxWEk5uie1c+izunjB+O3jkje9mWb6sqjvHq5KDG4Cy5XGeMu7kHnNpO 6ipGXHkyq9LZnEkMXSwymEDMv7s1lLOixHr+r0C+gH4NHANKt8epdKH97aPcuGe3dthh gnL8ALQLNZBD+dmQ3+PEDa5OagRRWndBFeFgdwvJrVXhiXG3Ed8wluWuxuccMtqfrsBI A7Zw== X-Gm-Message-State: AO0yUKVybWgktT/GQzI3/R3TEi07GYsXO3N+yH4Sd3lSf4dRT4YY3dFh WBMkDk5P0y/vqrG7OFt11EZOGq1Q0eR/C+CdsNnosGKWydn2aERFe9N1ZWA492yXLkVNJQJc5xM N64lhTNQ1Pj14ZoVNBTXffTbPU/zbvRObMVXeCmemms5BcLhQty/go3Rg5HYPxbiukA== X-Google-Smtp-Source: AK7set/na8FLmR3HRtWtdWgJzoG1aLZFn+IW8/UVY9Mo9icHQKU2nH6xusQQPQYqQDcRH/faaYiDbKDRife/e4k= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a63:935a:0:b0:4cd:1e13:2866 with SMTP id w26-20020a63935a000000b004cd1e132866mr790927pgm.117.1675793843804; Tue, 07 Feb 2023 10:17:23 -0800 (PST) Date: Tue, 7 Feb 2023 18:17:04 +0000 In-Reply-To: <20230117193041.708692-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230117193041.708692-1-calvinwan@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230207181706.363453-6-calvinwan@google.com> Subject: [PATCH v7 5/7] diff-lib: refactor out diff_change logic From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor out logic that sets up the diff_change call into a helper function for a future patch. Signed-off-by: Calvin Wan --- diff-lib.c | 46 +++++++++++++++++++++++++++++----------------- 1 file changed, 29 insertions(+), 17 deletions(-) diff --git a/diff-lib.c b/diff-lib.c index dec040c366..7101cfda3f 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -88,6 +88,31 @@ static int match_stat_with_submodule(struct diff_options *diffopt, return changed; } +static int diff_change_helper(struct diff_options *options, + unsigned newmode, unsigned dirty_submodule, + int changed, struct index_state *istate, + struct cache_entry *ce) +{ + unsigned int oldmode; + const struct object_id *old_oid, *new_oid; + + if (!changed && !dirty_submodule) { + ce_mark_uptodate(ce); + mark_fsmonitor_valid(istate, ce); + if (!options->flags.find_copies_harder) + return 1; + } + oldmode = ce->ce_mode; + old_oid = &ce->oid; + new_oid = changed ? null_oid() : &ce->oid; + diff_change(options, oldmode, newmode, + old_oid, new_oid, + !is_null_oid(old_oid), + !is_null_oid(new_oid), + ce->name, 0, dirty_submodule); + return 0; +} + int run_diff_files(struct rev_info *revs, unsigned int option) { int entries, i; @@ -105,11 +130,10 @@ int run_diff_files(struct rev_info *revs, unsigned int option) diff_unmerged_stage = 2; entries = istate->cache_nr; for (i = 0; i < entries; i++) { - unsigned int oldmode, newmode; + unsigned int newmode; struct cache_entry *ce = istate->cache[i]; int changed; unsigned dirty_submodule = 0; - const struct object_id *old_oid, *new_oid; if (diff_can_quit_early(&revs->diffopt)) break; @@ -245,21 +269,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option) newmode = ce_mode_from_stat(ce, st.st_mode); } - if (!changed && !dirty_submodule) { - ce_mark_uptodate(ce); - mark_fsmonitor_valid(istate, ce); - if (!revs->diffopt.flags.find_copies_harder) - continue; - } - oldmode = ce->ce_mode; - old_oid = &ce->oid; - new_oid = changed ? null_oid() : &ce->oid; - diff_change(&revs->diffopt, oldmode, newmode, - old_oid, new_oid, - !is_null_oid(old_oid), - !is_null_oid(new_oid), - ce->name, 0, dirty_submodule); - + if (diff_change_helper(&revs->diffopt, newmode, dirty_submodule, + changed, istate, ce)) + continue; } diffcore_std(&revs->diffopt); diff_flush(&revs->diffopt); From patchwork Tue Feb 7 18:17:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13131964 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1049C636CC for ; Tue, 7 Feb 2023 18:18:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232214AbjBGSSe (ORCPT ); Tue, 7 Feb 2023 13:18:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:42172 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232181AbjBGSRs (ORCPT ); Tue, 7 Feb 2023 13:17:48 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 31BB33FF18 for ; Tue, 7 Feb 2023 10:17:27 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id p15-20020a170902a40f00b00192b2bbb7f8so8534031plq.14 for ; Tue, 07 Feb 2023 10:17:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UGKOv7lTtPMgeVgfzyGwAjJPuXhc/OIK+sP+8kf5fMw=; b=isfw44oQ6V9MiymwI3aX6sowOFoRuP+jyVkQji2I4P7ZX6J1rrMVnt8hfkzTTxrIGa bgWg6EJfj2CTg2jiPTwmW0jnoAbsBkT++/8l/PlRDkUhlDOcrjolUXEcseH0WOQfzL/l Ksnn6fU1eIHStZFHEWL4kDqJaG1lYpa2yUUMW/5gs1GC0eI+JMUSn3Ewuw25LJ/sEksg SSMkcJATH4RCxGuH8Yaf1cu2AtPH27HIaKbqBJWfgai12jJvq0p6Z6YetuxXT5eftjf3 RK66sLDuoUl4S5LBoK4U7T+UFITzCxgCTqIEFvLdZ05inVHWM6rJvlVSk0J/XlJI3K0R onSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UGKOv7lTtPMgeVgfzyGwAjJPuXhc/OIK+sP+8kf5fMw=; b=h3PA23vQpVUxTDd46lmxu1wej+gF/69+5iStpXw4qKZFaw9Kb25ufGtWyskq7wrf1k 8riKqoKGdwZ/HEmqhOWcUdued5IjTFZSP7M6hhsgDTewvFPJycakmIdDhVslPhKZlVDB diTTB5GW1Pex4I8ZLLd+BBWJa3qHy6m+wPus5gu7oRwlfDM/fBzpJOBYSTvLrzy+Id3k MuZO+05GdWaQ/9RZOsXA6f7xo8bMHN0nLUxFJijE2JdGhYCHekyme6OnGJLDOsBoG1oX UeiFI5m3CA/jM7h9OfcbkfZ7ThCrc4/MgLE6jcMiQAn5Ky0TJojh629Ga95gQ1P93qIh FMcg== X-Gm-Message-State: AO0yUKX+ReoHLZs0NiDRS4PTxmvqejCeGXInfMK1jtuy2ys+8yw+xUox bz5DM29eW3vR/E76Zh9aONgfdfinft/nxm7YJkA9JTyUbQifxtbSmGg1VOhuhBwsJ8jVgqzoa6G ZbmiNg5oWWihmEKIv9NSB9lC0QfWlYdKh4ue0QT2Cak5LGdsMhw2OrzExNw1b1UEHvg== X-Google-Smtp-Source: AK7set9tQYBVk1sh84lpoqvzWCF8seOVHBm6/RVPRmLKBMKHVpWl7rXGOj1FrwECdEOnsksPXofzTYO2QFsWrGs= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:903:234e:b0:196:8445:56d5 with SMTP id c14-20020a170903234e00b00196844556d5mr1048546plh.13.1675793845478; Tue, 07 Feb 2023 10:17:25 -0800 (PST) Date: Tue, 7 Feb 2023 18:17:05 +0000 In-Reply-To: <20230117193041.708692-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230117193041.708692-1-calvinwan@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230207181706.363453-7-calvinwan@google.com> Subject: [PATCH v7 6/7] diff-lib: refactor match_stat_with_submodule From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Flatten out the if statements in match_stat_with_submodule so the logic is more readable and easier for future patches to add to. orig_flags didn't need to be set if the cache entry wasn't a GITLINK so defer setting it. Signed-off-by: Calvin Wan --- diff-lib.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/diff-lib.c b/diff-lib.c index 7101cfda3f..e18c886a80 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -73,18 +73,24 @@ static int match_stat_with_submodule(struct diff_options *diffopt, unsigned *dirty_submodule) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); - if (S_ISGITLINK(ce->ce_mode)) { - struct diff_flags orig_flags = diffopt->flags; - if (!diffopt->flags.override_submodule_config) - set_diffopt_flags_from_submodule_config(diffopt, ce->name); - if (diffopt->flags.ignore_submodules) - changed = 0; - else if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, - diffopt->flags.ignore_untracked_in_submodules); - diffopt->flags = orig_flags; + struct diff_flags orig_flags; + + if (!S_ISGITLINK(ce->ce_mode)) + return changed; + + orig_flags = diffopt->flags; + if (!diffopt->flags.override_submodule_config) + set_diffopt_flags_from_submodule_config(diffopt, ce->name); + if (diffopt->flags.ignore_submodules) { + changed = 0; + goto cleanup; } + if (!diffopt->flags.ignore_dirty_submodules && + (!changed || diffopt->flags.dirty_submodules)) + *dirty_submodule = is_submodule_modified(ce->name, + diffopt->flags.ignore_untracked_in_submodules); +cleanup: + diffopt->flags = orig_flags; return changed; } From patchwork Tue Feb 7 18:17:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13131965 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 617CAC636D4 for ; Tue, 7 Feb 2023 18:18:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232231AbjBGSSl (ORCPT ); Tue, 7 Feb 2023 13:18:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:41022 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232267AbjBGSR4 (ORCPT ); Tue, 7 Feb 2023 13:17:56 -0500 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 285293C2A7 for ; Tue, 7 Feb 2023 10:17:28 -0800 (PST) Received: by mail-pl1-x649.google.com with SMTP id p18-20020a170902ead200b0019909279002so4289198pld.3 for ; Tue, 07 Feb 2023 10:17:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QCBZoKWC9mkk+iJ00p8tn1PIaGfT0oYjiTiNxEbfLPE=; b=hLRfJK+0sjvRcWDsCPmwORmcFOBrQjJNfMuBqYkovVKoVqd54fIcgPzv/4gG0Rvncq 7G5WDvQxSqhiOqmrJcgQotDG/EGIFUzcp1TjEGi66PRV3moIQwCBolSfM9lrsXu0gwWB Rymlf/k1HC9KUsoo9ii1/oFlXIDAZd9lu8aeKD+0jYHpGABobZggAEKoRn/GVKuJMcw+ fhM077zSrPX3bSbvsHEjSAKzFb/WoYUdCx0quBctgKfMPJZzjYwjOLrHF89jU7uSojt9 LqjldEVJT+UVN+sLd8sLU+1oWK2lDWujVRP4rWSebnxlIOfv+BJ1zBE+mYDu9lVqOtzN w4qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QCBZoKWC9mkk+iJ00p8tn1PIaGfT0oYjiTiNxEbfLPE=; b=aIgmM6n4MJ/rRMkKIuGureTKZowhQ4J5I7DA4xqrRMIn73n1LRlj5T36FaxldU1dfC sKuBdUOfF3ekdHfhCa0iUsMf/9INl1ixg/p08syr1gpsfi1CM+WaaXrupDhIjQO1/b+z K5E4MBOfBpsDnP/z1/KFdblpKzEMt0Ppf9a+NsYm87E8OOVqxz8UD3Baii68hk2nCPAc 0Y1p2JeIUmTtVDyPVMnQdaCHAL4wvAU3WgGUica8rCd/sF5NqddYgb3m8vIiapaGi19Z kA9hzmpQ9TTwoAjVw3UKf7FKvVRohiMud0OoBWiOO9ppO2ldDA7suvPlQPuHoowtFuoO 0ayw== X-Gm-Message-State: AO0yUKVz+zHuzjbQnG/h6hWFf4A0sW/bVYzmqDuQE4MAYrFM3hfTO8r0 71q77gkvekgvtl8KKXSByMej43xys9IBDqB0OftmFcNLCj+b0bFnjjeHS3YUZ7led4/3Nw5klq6 5qlIDidTdgqEqqmgDTu2Ynw1v1IA31avMDZwMZ8O8pGKJH48qjycz0gpk40o1bx6wig== X-Google-Smtp-Source: AK7set9FSRjPMftnqGwNww+iMZJRtVy56BH2hFmlCjU6WN2j5wO1Cq2Iz4GF/VDp6U/gCu/S+5cxaelObnIZu/U= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:90a:170c:b0:22c:5369:8a36 with SMTP id z12-20020a17090a170c00b0022c53698a36mr1439pjd.0.1675793847059; Tue, 07 Feb 2023 10:17:27 -0800 (PST) Date: Tue, 7 Feb 2023 18:17:06 +0000 In-Reply-To: <20230117193041.708692-1-calvinwan@google.com> Mime-Version: 1.0 References: <20230117193041.708692-1-calvinwan@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog Message-ID: <20230207181706.363453-8-calvinwan@google.com> Subject: [PATCH v7 7/7] diff-lib: parallelize run_diff_files for submodules From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , avarab@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During the iteration of the index entries in run_diff_files, whenever a submodule is found and needs its status checked, a subprocess is spawned for it. Instead of spawning the subprocess immediately and waiting for its completion to continue, hold onto all submodules and relevant information in a list. Then use that list to create tasks for run_processes_parallel. Subprocess output is duplicated and passed to status_pipe_output which stores it to be parsed on completion of the subprocess. Add config option submodule.diffJobs to set the maximum number of parallel jobs. The option defaults to 1 if unset. If set to 0, the number of jobs is set to online_cpus(). Since run_diff_files is called from many different commands, I chose to grab the config option in the function rather than adding variables to every git command and then figuring out how to pass them all in. Signed-off-by: Calvin Wan --- Documentation/config/submodule.txt | 12 +++ diff-lib.c | 71 ++++++++++++-- submodule.c | 148 +++++++++++++++++++++++++++++ submodule.h | 9 ++ t/t4027-diff-submodule.sh | 31 ++++++ t/t7506-status-submodule.sh | 25 +++++ 6 files changed, 289 insertions(+), 7 deletions(-) diff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt index 6490527b45..3209eb8117 100644 --- a/Documentation/config/submodule.txt +++ b/Documentation/config/submodule.txt @@ -93,6 +93,18 @@ submodule.fetchJobs:: in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1. +submodule.diffJobs:: + Specifies how many submodules are diffed at the same time. A + positive integer allows up to that number of submodules diffed + in parallel. A value of 0 will give some reasonable default. + If unset, it defaults to 1. The diff operation is used by many + other git commands such as add, merge, diff, status, stash and + more. Note that the expensive part of the diff operation is + reading the index from cache or memory. Therefore multiple jobs + may be detrimental to performance if your hardware does not + support parallel reads or if the number of jobs greatly exceeds + the amount of supported reads. + submodule.alternateLocation:: Specifies how the submodules obtain alternates when submodules are cloned. Possible values are `no`, `superproject`. diff --git a/diff-lib.c b/diff-lib.c index e18c886a80..f91cd73ae7 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -14,6 +14,7 @@ #include "dir.h" #include "fsmonitor.h" #include "commit-reach.h" +#include "config.h" /* * diff-files @@ -65,18 +66,23 @@ static int check_removed(const struct index_state *istate, const struct cache_en * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES * option is set, the caller does not only want to know if a submodule is * modified at all but wants to know all the conditions that are met (new - * commits, untracked content and/or modified content). + * commits, untracked content and/or modified content). If + * defer_submodule_status bit is set, dirty_submodule will be left to the + * caller to set. defer_submodule_status can also be set to 0 in this + * function if there is no need to check if the submodule is modified. */ static int match_stat_with_submodule(struct diff_options *diffopt, const struct cache_entry *ce, struct stat *st, unsigned ce_option, - unsigned *dirty_submodule) + unsigned *dirty_submodule, int *defer_submodule_status, + unsigned *ignore_untracked) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); struct diff_flags orig_flags; + int defer = 0; if (!S_ISGITLINK(ce->ce_mode)) - return changed; + goto ret; orig_flags = diffopt->flags; if (!diffopt->flags.override_submodule_config) @@ -86,11 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt, goto cleanup; } if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, + (!changed || diffopt->flags.dirty_submodules)) { + if (defer_submodule_status && *defer_submodule_status) { + defer = 1; + *ignore_untracked = diffopt->flags.ignore_untracked_in_submodules; + } else { + *dirty_submodule = is_submodule_modified(ce->name, diffopt->flags.ignore_untracked_in_submodules); + } + } cleanup: diffopt->flags = orig_flags; +ret: + if (defer_submodule_status) + *defer_submodule_status = defer; return changed; } @@ -127,6 +142,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ? CE_MATCH_RACY_IS_DIRTY : 0); uint64_t start = getnanotime(); struct index_state *istate = revs->diffopt.repo->index; + struct string_list submodules = STRING_LIST_INIT_NODUP; diff_set_mnemonic_prefix(&revs->diffopt, "i/", "w/"); @@ -250,6 +266,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option) newmode = ce->ce_mode; } else { struct stat st; + unsigned ignore_untracked = 0; + int defer_submodule_status = 1; changed = check_removed(istate, ce, &st); if (changed) { @@ -271,14 +289,53 @@ int run_diff_files(struct rev_info *revs, unsigned int option) } changed = match_stat_with_submodule(&revs->diffopt, ce, &st, - ce_option, &dirty_submodule); + ce_option, &dirty_submodule, + &defer_submodule_status, + &ignore_untracked); newmode = ce_mode_from_stat(ce, st.st_mode); + if (defer_submodule_status) { + struct submodule_status_util tmp = { + .changed = changed, + .dirty_submodule = 0, + .ignore_untracked = ignore_untracked, + .newmode = newmode, + .ce = ce, + .path = ce->name, + }; + struct string_list_item *item; + + item = string_list_append(&submodules, ce->name); + item->util = xmalloc(sizeof(tmp)); + memcpy(item->util, &tmp, sizeof(tmp)); + continue; + } } if (diff_change_helper(&revs->diffopt, newmode, dirty_submodule, changed, istate, ce)) continue; } + if (submodules.nr > 0) { + int parallel_jobs; + if (git_config_get_int("submodule.diffjobs", ¶llel_jobs)) + parallel_jobs = 1; + else if (!parallel_jobs) + parallel_jobs = online_cpus(); + else if (parallel_jobs < 0) + die(_("submodule.diffjobs cannot be negative")); + + if (get_submodules_status(&submodules, parallel_jobs)) + die(_("submodule status failed")); + for (size_t i = 0; i < submodules.nr; i++) { + struct submodule_status_util *util = submodules.items[i].util; + + if (diff_change_helper(&revs->diffopt, util->newmode, + util->dirty_submodule, util->changed, + istate, util->ce)) + continue; + } + } + string_list_clear(&submodules, 1); diffcore_std(&revs->diffopt); diff_flush(&revs->diffopt); trace_performance_since(start, "diff-files"); @@ -326,7 +383,7 @@ static int get_stat_data(const struct index_state *istate, return -1; } changed = match_stat_with_submodule(diffopt, ce, &st, - 0, dirty_submodule); + 0, dirty_submodule, NULL, NULL); if (changed) { mode = ce_mode_from_stat(ce, st.st_mode); oid = null_oid(); diff --git a/submodule.c b/submodule.c index d88aa2c573..3e1811691a 100644 --- a/submodule.c +++ b/submodule.c @@ -1373,6 +1373,17 @@ int submodule_touches_in_range(struct repository *r, return ret; } +struct submodule_parallel_status { + size_t index_count; + int result; + + struct string_list *submodule_names; + + /* Pending statuses by OIDs */ + struct status_task **oid_status_tasks; + int oid_status_tasks_nr, oid_status_tasks_alloc; +}; + struct submodule_parallel_fetch { /* * The index of the last index entry processed by @@ -1455,6 +1466,12 @@ struct fetch_task { struct oid_array *commits; /* Ensure these commits are fetched */ }; +struct status_task { + const char *path; + struct strbuf out; + int ignore_untracked; +}; + /** * When a submodule is not defined in .gitmodules, we cannot access it * via the regular submodule-config. Create a fake submodule, which we can @@ -1947,6 +1964,25 @@ static int parse_status_porcelain(char *str, size_t len, return 0; } +static void parse_status_porcelain_strbuf(struct strbuf *buf, + unsigned *dirty_submodule, + int ignore_untracked) +{ + struct string_list list = STRING_LIST_INIT_DUP; + struct string_list_item *item; + + string_list_split(&list, buf->buf, '\n', -1); + + for_each_string_list_item(item, &list) { + if (parse_status_porcelain(item->string, + strlen(item->string), + dirty_submodule, + ignore_untracked)) + break; + } + string_list_clear(&list, 0); +} + unsigned is_submodule_modified(const char *path, int ignore_untracked) { struct child_process cp = CHILD_PROCESS_INIT; @@ -1981,6 +2017,118 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) return dirty_submodule; } +static struct status_task * +get_status_task_from_index(struct submodule_parallel_status *sps, + struct strbuf *err) +{ + for (; sps->index_count < sps->submodule_names->nr; sps->index_count++) { + struct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util; + struct status_task *task; + + if (!verify_submodule_git_directory(util->path)) + continue; + + task = xmalloc(sizeof(*task)); + task->path = util->path; + task->ignore_untracked = util->ignore_untracked; + strbuf_init(&task->out, 0); + sps->index_count++; + return task; + } + return NULL; +} + +static int get_next_submodule_status(struct child_process *cp, + struct strbuf *err, void *data, + void **task_cb) +{ + struct submodule_parallel_status *sps = data; + struct status_task *task = get_status_task_from_index(sps, err); + + if (!task) + return 0; + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env); + prepare_status_porcelain(cp, task->path, task->ignore_untracked); + *task_cb = task; + return 1; +} + +static int status_start_failure(struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + + sps->result = 1; + strbuf_addf(err, + _(status_porcelain_start_error), + task->path); + return 0; +} + +static void status_duplicate_output(struct strbuf *out, + size_t offset, + void *cb, void *task_cb) +{ + struct status_task *task = task_cb; + + strbuf_add(&task->out, out->buf + offset, out->len - offset); + strbuf_setlen(out, offset); +} + +static int status_finish(int retvalue, struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + struct string_list_item *it = + string_list_lookup(sps->submodule_names, task->path); + struct submodule_status_util *util = it->util; + + if (retvalue) { + sps->result = 1; + strbuf_addf(err, + _(status_porcelain_fail_error), + task->path); + } + + parse_status_porcelain_strbuf(&task->out, + &util->dirty_submodule, + util->ignore_untracked); + + strbuf_release(&task->out); + free(task); + + return 0; +} + +int get_submodules_status(struct string_list *submodules, + int max_parallel_jobs) +{ + struct submodule_parallel_status sps = { + .submodule_names = submodules, + }; + const struct run_process_parallel_opts opts = { + .tr2_category = "submodule", + .tr2_label = "parallel/status", + + .processes = max_parallel_jobs, + + .get_next_task = get_next_submodule_status, + .start_failure = status_start_failure, + .duplicate_output = status_duplicate_output, + .task_finished = status_finish, + .data = &sps, + }; + + string_list_sort(sps.submodule_names); + run_processes_parallel(&opts); + + return sps.result; +} + int submodule_uses_gitfile(const char *path) { struct child_process cp = CHILD_PROCESS_INIT; diff --git a/submodule.h b/submodule.h index b52a4ff1e7..08d278a414 100644 --- a/submodule.h +++ b/submodule.h @@ -41,6 +41,13 @@ struct submodule_update_strategy { .type = SM_UPDATE_UNSPECIFIED, \ } +struct submodule_status_util { + int changed, ignore_untracked; + unsigned dirty_submodule, newmode; + struct cache_entry *ce; + const char *path; +}; + int is_gitmodules_unmerged(struct index_state *istate); int is_writing_gitmodules_ok(void); int is_staging_gitmodules_ok(struct index_state *istate); @@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r, int command_line_option, int default_option, int quiet, int max_parallel_jobs); +int get_submodules_status(struct string_list *submodules, + int max_parallel_jobs); unsigned is_submodule_modified(const char *path, int ignore_untracked); int submodule_uses_gitfile(const char *path); diff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh index 40164ae07d..1c747cc325 100755 --- a/t/t4027-diff-submodule.sh +++ b/t/t4027-diff-submodule.sh @@ -34,6 +34,25 @@ test_expect_success setup ' subtip=$3 subprev=$2 ' +test_expect_success 'diff in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git diff && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git diff && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_expect_success 'git diff --raw HEAD' ' hexsz=$(test_oid hexsz) && git diff --raw --abbrev=$hexsz HEAD >actual && @@ -70,6 +89,18 @@ test_expect_success 'git diff HEAD with dirty submodule (work tree)' ' test_cmp expect.body actual.body ' +test_expect_success 'git diff HEAD with dirty submodule (work tree, parallel)' ' + ( + cd sub && + git reset --hard && + echo >>world + ) && + git -c submodule.diffJobs=8 diff HEAD >actual && + sed -e "1,/^@@/d" actual >actual.body && + expect_from_to >expect.body $subtip $subprev-dirty && + test_cmp expect.body actual.body +' + test_expect_success 'git diff HEAD with dirty submodule (index)' ' ( cd sub && diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh index d050091345..7da64e4c4c 100755 --- a/t/t7506-status-submodule.sh +++ b/t/t7506-status-submodule.sh @@ -412,4 +412,29 @@ test_expect_success 'status with added file in nested submodule (short)' ' EOF ' +test_expect_success 'status in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git status && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git status && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + +test_expect_success 'status in superproject with submodules (parallel)' ' + git -C super status --porcelain >output && + git -C super -c submodule.diffJobs=8 status --porcelain >output_parallel && + diff output output_parallel +' + test_done