From patchwork Wed Jan 4 21:54:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13089082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96434C53210 for ; Wed, 4 Jan 2023 21:54:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240357AbjADVyu (ORCPT ); Wed, 4 Jan 2023 16:54:50 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58422 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240349AbjADVyo (ORCPT ); Wed, 4 Jan 2023 16:54:44 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 81B121EAD6 for ; Wed, 4 Jan 2023 13:54:43 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id u6-20020a170903124600b00188cd4769bcso24515359plh.0 for ; Wed, 04 Jan 2023 13:54:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=asvscKm9OeP3iNTMrV1XEEibhSePzCPPG6+7mAFVIIY=; b=D9L6POOWaQfOeRGUWHEcnWWFtRH9/d1OFMT6uSWBlmSBj9hlnGX3F9PzSrELkYyRiH RryqIsNrUwHC/HxjVxOyyrTzbsJwpuA25rSWyvn3YSMePfFpQkZqLPw0haRApUqbw0EB hM4uqqWSxVHXLiMo+pA21rUpkDOj5vyop8ocDX0MlcjN87IeXL28ITBRpJHCIH3LRHn2 WUIMKExlyjpY9XmVbh5zP2CTh+fhQ0qQo63LatguPdkPemhden5Lh0q74PNvA9NFpZOH EtGneyUTcYhyHmmctAhu+3k9XlEEbwIgrU+WvrvRoxEtqUORcW9iMUdjHisDBJYS4KHK ZGqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=asvscKm9OeP3iNTMrV1XEEibhSePzCPPG6+7mAFVIIY=; b=mhxHoEQJl0j5AnB4bZFJ7gzCvc9K/V63LiS5xWhzfJdCAsfZ3YdYyDWxGUwxVdt0pF OBVRYcZTEXkcMHvEmqKc1r1Pfh7U8yh7Qpw+VEXTJ0SeUqrkj9mlpTzvnOW7xKuD2Ixl hpl/Nmt8ztHaDg96fUy+x3uTXnQqXbOUOeaba3q2YtXkifii/J4HKldxRo1IYjFC44/O SIy28cKFU5Rqv1sVMLMFuT1OEL5FYTJHy4kCryHSL/vkvY1L8I8AXNphej7MXa5C/9KI ccYwimOp7Ao2MpWydCPVnKhc7DGphD8OI1vWgkQik41CuWZVYJFL2byj3qEcgDm5pCmt L4VA== X-Gm-Message-State: AFqh2krPrTwSJ97Y8LCx9dIruPPYq03MnAe3RCzx2NBVXVVokjMmbR8l AyhVbJRcR0s0psdKu57LZSoxee8MT1QSvIDfkXFj79LIDj9JK9fmPsO1UU2T58oMIocdE65ihKB nmKMWzPM/7nU8ASM1NWydSY5xZ5FGQRKZYp7VBmeu8sntbcXa7Q/2LMSSyqQWe/zGhQ== X-Google-Smtp-Source: AMrXdXv6d16fdgHqTBpmgyo2O0wQX23vR8z4E6B1JL1436l/aYza9FWSYVz8ncE8Z4cduwcEzpf7GzzhRFsBT0A= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:902:e744:b0:192:9aeb:d47 with SMTP id p4-20020a170902e74400b001929aeb0d47mr1874440plf.65.1672869282618; Wed, 04 Jan 2023 13:54:42 -0800 (PST) Date: Wed, 4 Jan 2023 21:54:10 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230104215415.1083526-2-calvinwan@google.com> Subject: [PATCH v5 1/6] run-command: add duplicate_output_fn to run_processes_parallel_opts From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add duplicate_output_fn as an optionally set function in run_process_parallel_opts. If set, output from each child process is copied and passed to the callback function whenever output from the child process is buffered to allow for separate parsing. Signed-off-by: Calvin Wan --- run-command.c | 16 ++++++++++++--- run-command.h | 27 +++++++++++++++++++++++++ t/helper/test-run-command.c | 21 ++++++++++++++++++++ t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++ 4 files changed, 100 insertions(+), 3 deletions(-) diff --git a/run-command.c b/run-command.c index 756f1839aa..cad88befe0 100644 --- a/run-command.c +++ b/run-command.c @@ -1526,6 +1526,9 @@ static void pp_init(struct parallel_processes *pp, if (!opts->get_next_task) BUG("you need to specify a get_next_task function"); + if (opts->duplicate_output && opts->ungroup) + BUG("duplicate_output and ungroup are incompatible with each other"); + CALLOC_ARRAY(pp->children, n); if (!opts->ungroup) CALLOC_ARRAY(pp->pfd, n); @@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp, for (size_t i = 0; i < opts->processes; i++) { if (pp->children[i].state == GIT_CP_WORKING && pp->pfd[i].revents & (POLLIN | POLLHUP)) { - int n = strbuf_read_once(&pp->children[i].err, - pp->children[i].process.err, 0); + ssize_t n = strbuf_read_once(&pp->children[i].err, + pp->children[i].process.err, 0); if (n == 0) { close(pp->children[i].process.err); pp->children[i].state = GIT_CP_WAIT_CLEANUP; - } else if (n < 0) + } else if (n < 0) { if (errno != EAGAIN) die_errno("read"); + } else { + if (opts->duplicate_output) + opts->duplicate_output(&pp->children[i].err, + strlen(pp->children[i].err.buf) - n, + opts->data, + pp->children[i].data); + } } } } diff --git a/run-command.h b/run-command.h index 072db56a4d..6dcf999f6c 100644 --- a/run-command.h +++ b/run-command.h @@ -408,6 +408,27 @@ typedef int (*start_failure_fn)(struct strbuf *out, void *pp_cb, void *pp_task_cb); +/** + * This callback is called whenever output from a child process is buffered + * + * See run_processes_parallel() below for a discussion of the "struct + * strbuf *out" parameter. + * + * The offset refers to the number of bytes originally in "out" before + * the output from the child process was buffered. Therefore, the buffer + * range, "out + buf" to the end of "out", would contain the buffer of + * the child process output. + * + * pp_cb is the callback cookie as passed into run_processes_parallel, + * pp_task_cb is the callback cookie as passed into get_next_task_fn. + * + * This function is incompatible with "ungroup" + */ +typedef void (*duplicate_output_fn)(struct strbuf *out, + size_t offset, + void *pp_cb, + void *pp_task_cb); + /** * This callback is called on every child process that finished processing. * @@ -461,6 +482,12 @@ struct run_process_parallel_opts */ start_failure_fn start_failure; + /** + * duplicate_output: See duplicate_output_fn() above. This should be + * NULL unless process specific output is needed + */ + duplicate_output_fn duplicate_output; + /** * task_finished: See task_finished_fn() above. This can be * NULL to omit any special handling. diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c index 3ecb830f4a..ffd3cd0045 100644 --- a/t/helper/test-run-command.c +++ b/t/helper/test-run-command.c @@ -52,6 +52,21 @@ static int no_job(struct child_process *cp, return 0; } +static void duplicate_output(struct strbuf *out, + size_t offset, + void *pp_cb UNUSED, + void *pp_task_cb UNUSED) +{ + struct string_list list = STRING_LIST_INIT_DUP; + + string_list_split(&list, out->buf + offset, '\n', -1); + for (size_t i = 0; i < list.nr; i++) { + if (strlen(list.items[i].string) > 0) + fprintf(stderr, "duplicate_output: %s\n", list.items[i].string); + } + string_list_clear(&list, 0); +} + static int task_finished(int result, struct strbuf *err, void *pp_cb, @@ -439,6 +454,12 @@ int cmd__run_command(int argc, const char **argv) opts.ungroup = 1; } + if (!strcmp(argv[1], "--duplicate-output")) { + argv += 1; + argc -= 1; + opts.duplicate_output = duplicate_output; + } + jobs = atoi(argv[2]); strvec_clear(&proc.args); strvec_pushv(&proc.args, (const char **)argv + 3); diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh index e2411f6a9b..879e536638 100755 --- a/t/t0061-run-command.sh +++ b/t/t0061-run-command.sh @@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' ' test-tool run-command --ungroup run-command-parallel 5 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' ' test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' ' test-tool run-command --ungroup run-command-parallel 4 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai test_cmp expect actual ' +test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' ' + test-tool run-command --duplicate-output run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test 4 = $(grep -c "duplicate_output: Hello" err) && + test 4 = $(grep -c "duplicate_output: World" err) && + sed "/duplicate_output/d" err > err1 && + test_cmp expect err1 +' + test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' ' test-tool run-command --ungroup run-command-parallel 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_line_count = 8 out && @@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' ' test_cmp expect actual ' +test_expect_success 'run_command is asked to abort gracefully --duplicate-output' ' + test-tool run-command --duplicate-output run-command-abort 3 false >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command is asked to abort gracefully (ungroup)' ' test-tool run-command --ungroup run-command-abort 3 false >out 2>err && test_must_be_empty out && @@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' ' test_cmp expect actual ' +test_expect_success 'run_command outputs --duplicate-output' ' + test-tool run-command --duplicate-output run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && + test_must_be_empty out && + test_cmp expect err +' + test_expect_success 'run_command outputs (ungroup) ' ' test-tool run-command --ungroup run-command-no-jobs 3 sh -c "printf \"%s\n%s\n\" Hello World" >out 2>err && test_must_be_empty out && From patchwork Wed Jan 4 21:54:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13089083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1447EC46467 for ; Wed, 4 Jan 2023 21:54:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240365AbjADVyy (ORCPT ); Wed, 4 Jan 2023 16:54:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58428 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229610AbjADVyp (ORCPT ); Wed, 4 Jan 2023 16:54:45 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 50AD31E3FD for ; Wed, 4 Jan 2023 13:54:45 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id y2-20020a17090a784200b00225c0839b80so11557697pjl.5 for ; Wed, 04 Jan 2023 13:54:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=DwwyDRcEc/QahaSm8IQ8wFaNXRSbSZzT+SIlbiS7D3Q=; b=OHUDQ0/6H2yINksptjngFM9jfRCRK8Nt+kN0bBVsaGpK8r6KvDq4+n87OqD4o/x2G8 EBebywDpRDI59Yot/GIcyx6q+lZjXcFQXpfpKASdA+/s2BhqynrJuriX+Fts/Sijn0H8 4P49+9pljF8o7quiEZOWP7DW9AujCQgRw3HtX3XmLgF0uNanwtN22cCVLj9e3GYB+MQA OPDiPjWqYJ4wodVTC5C+Wfi+sOPDasqK+c3lZRDLcdvf65T/H9TufVBZ81b+StcEr4ic AC8z58noRGKRodV19+Ng461sxNacGbg0B3MWGqEbWm0fC5oXZKcId9am4AYLGVUJn1wf a8BQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DwwyDRcEc/QahaSm8IQ8wFaNXRSbSZzT+SIlbiS7D3Q=; b=5Wt3xXwhXXKRgSBV0G1eavUS2w5Am0JtTQFkxqXR0sc3E1/fCSUjx/PPm2AY62ZPcA JMERVT4U0zBupjffnqIEXnZRFAAJzQ+hCP7McfZ6GZ9tJIYhnGet2g1JLwW/ouqmRTVt Nspu3mnfqzJv4S1y81MekSa9r/5Nu9P8jFX6UYodU5A2LdpXrIKSeGbDd6AubXO2wyHF yLYtmgPyItxY31wNyxuA8uH3b+bTC0fwrLXmfjWXedJTzu7dlIMGP8qfvDsDQ2sO1Oqu 2KF9W5kd+BJ4DBN2MpnLMKWlJwvre9V3S5ZOsAB+pDox9Voqe/bXohdj3++B1faK0/fy KxXw== X-Gm-Message-State: AFqh2koUtlF247SSGZ1rQ3psPvv6y9P4X4ze5O8vVkTHdtAjqE4SYFnY ong/BlJ0Z8cDevVXCUcqHCFWJxhO48yO1kGs1DdOvLESs5sqU7CekCQ538AIUp5D/TiaudpjJ8R BTHVIempyPNOu4ScskKxShF74HrC91OuOrfhNoG+AkAa0QhiriSkj/bEsoILafyH0fw== X-Google-Smtp-Source: AMrXdXt6ioubDvWQT/SKVqc4kcJxMFQ2T6BcIMdySbSZNM2G8CKQEVbIL4PFsBTIB92bzVwAYKEROSu9XMkS20k= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:aa7:9188:0:b0:582:687f:83e8 with SMTP id x8-20020aa79188000000b00582687f83e8mr810520pfa.79.1672869284734; Wed, 04 Jan 2023 13:54:44 -0800 (PST) Date: Wed, 4 Jan 2023 21:54:11 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230104215415.1083526-3-calvinwan@google.com> Subject: [PATCH v5 2/6] submodule: strbuf variable rename From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A prepatory change for a future patch that moves the status parsing logic to a separate function. Signed-off-by: Calvin Wan --- submodule.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/submodule.c b/submodule.c index fae24ef34a..faf37c1101 100644 --- a/submodule.c +++ b/submodule.c @@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) fp = xfdopen(cp.out, "r"); while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { + char *str = buf.buf; + const size_t len = buf.len; + /* regular untracked files */ - if (buf.buf[0] == '?') + if (str[0] == '?') dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '1' || - buf.buf[0] == '2') { + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { /* T = line type, XY = status, SSSS = submodule state */ - if (buf.len < strlen("T XY SSSS")) + if (len < strlen("T XY SSSS")) BUG("invalid status --porcelain=2 line %s", - buf.buf); + str); - if (buf.buf[5] == 'S' && buf.buf[8] == 'U') + if (str[5] == 'S' && str[8] == 'U') /* nested untracked file */ dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - if (buf.buf[0] == 'u' || - buf.buf[0] == '2' || - memcmp(buf.buf + 5, "S..U", 4)) + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) /* other change */ dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; } From patchwork Wed Jan 4 21:54:12 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13089084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A82CFC53210 for ; Wed, 4 Jan 2023 21:55:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240355AbjADVzD (ORCPT ); Wed, 4 Jan 2023 16:55:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58436 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240333AbjADVys (ORCPT ); Wed, 4 Jan 2023 16:54:48 -0500 Received: from mail-yw1-x114a.google.com (mail-yw1-x114a.google.com [IPv6:2607:f8b0:4864:20::114a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12AF61EAD6 for ; Wed, 4 Jan 2023 13:54:47 -0800 (PST) Received: by mail-yw1-x114a.google.com with SMTP id 00721157ae682-434eb7c6fa5so360700847b3.14 for ; Wed, 04 Jan 2023 13:54:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6rZ466JhVR5XM57QkszJq3kGwqA/2fSr9kRJYjghQas=; b=RTmjeRr+Uh3+Kre+Afxv1DkO/Gj0KrndnRZIqN50ZIUZsxoTVp8x2yhf0VharQM0Uo F9U5OhG1wGB/m+MHBfTuzbQG2qrQ3U4TRQXuYwH9MVVjjGFlpdotg80c2Gl8xzYgIaOW 98S//4cq/hf+rM4hJdwI7zah19K6Dn/nekfs/Olp6d7rsNZHEuQiasMOZy1u61AEvPOX USVVjYlHFOyAyQ1+EPtr55raxrgO+f2XxeAvkZ9SwVEp7Wx4pj9ZLdkElbVhN+6fkWCO /4L0hYJkK4Eo9tuTXjNi0AbiryMxFjZ+fsW3LyeahnwhVg6q0n99KwWpDnGBQDmKGmB5 pBRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6rZ466JhVR5XM57QkszJq3kGwqA/2fSr9kRJYjghQas=; b=bP4K1fpyxBX+w0f+nDLo2HgiWewYPtamRf7rAnb1bWTOnIr8etfydvrs74ajtum7F/ quipICdwT30j8eTCZX+nRCZLBL0yHzAJfYQqoOUkUnFjuwyNM4A0+BVgRDHMjIoGgx7g TlIpsLxp3UnlDn/LY35bttKjJMcM9MSMoc2OXj3X4iscx1hLQrL1AmkrIYxpL/sqrzUF wQrdWg2vS9sNLn9/XHXWMfBfpisB1C1lYwFAMiWJebd2+WZmtYALcAbzBoFuy9HLXUma ZnV4+PGMhWidYQuqEm9zXXZQkDaU3PFgwIgijcxrqCzjB0qdYcDmc1vI7S4eHDRMZN48 qjqA== X-Gm-Message-State: AFqh2kqsELYAU9ACW5Z9MfPhUYqCyzS553+MEqxpdiTwj00GuvVQKYkJ sNFr9iHjotQBSB+nF0jvNdnTK+GyWTHkagNfxnBrX/gW2glVAiDqU8GgCCSyGb0AwwDwepaTga/ 0UANhQkVBJkLFtnt3gw8vrW51eP4xZid5k0dglDbGgl7cK3bJQVbgTkA7SGR6DfyCyw== X-Google-Smtp-Source: AMrXdXvrVpOl6R7fAvYvMK8+zfINGGNin0yyjEEFKM4gJoWqqrQ5fPRxs+RpQzoDXmW0LT3Sf+A5E193HmoL6FU= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a05:690c:444:b0:3fe:dd73:15df with SMTP id bj4-20020a05690c044400b003fedd7315dfmr6506213ywb.331.1672869286285; Wed, 04 Jan 2023 13:54:46 -0800 (PST) Date: Wed, 4 Jan 2023 21:54:12 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230104215415.1083526-4-calvinwan@google.com> Subject: [PATCH v5 3/6] submodule: move status parsing into function From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org A future patch requires the ability to parse the output of git status --porcelain=2. Move parsing code from is_submodule_modified to parse_status_porcelain. Signed-off-by: Calvin Wan --- submodule.c | 74 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 42 insertions(+), 32 deletions(-) diff --git a/submodule.c b/submodule.c index faf37c1101..768d4b4cd7 100644 --- a/submodule.c +++ b/submodule.c @@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r, return spf.result; } +static int parse_status_porcelain(char *str, size_t len, + unsigned *dirty_submodule, + int ignore_untracked) +{ + /* regular untracked files */ + if (str[0] == '?') + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { + /* T = line type, XY = status, SSSS = submodule state */ + if (len < strlen("T XY SSSS")) + BUG("invalid status --porcelain=2 line %s", + str); + + if (str[5] == 'S' && str[8] == 'U') + /* nested untracked file */ + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) + /* other change */ + *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; + } + + if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && + ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || + ignore_untracked)) { + /* + * We're not interested in any further information from + * the child any more, neither output nor its exit code. + */ + return 1; + } + return 0; +} + unsigned is_submodule_modified(const char *path, int ignore_untracked) { struct child_process cp = CHILD_PROCESS_INIT; @@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) char *str = buf.buf; const size_t len = buf.len; - /* regular untracked files */ - if (str[0] == '?') - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '1' || - str[0] == '2') { - /* T = line type, XY = status, SSSS = submodule state */ - if (len < strlen("T XY SSSS")) - BUG("invalid status --porcelain=2 line %s", - str); - - if (str[5] == 'S' && str[8] == 'U') - /* nested untracked file */ - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '2' || - memcmp(str + 5, "S..U", 4)) - /* other change */ - dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; - } - - if ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && - ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || - ignore_untracked)) { - /* - * We're not interested in any further information from - * the child any more, neither output nor its exit code. - */ - ignore_cp_exit_code = 1; + ignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule, + ignore_untracked); + if (ignore_cp_exit_code) break; - } } fclose(fp); From patchwork Wed Jan 4 21:54:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13089085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5A691C54E76 for ; Wed, 4 Jan 2023 21:55:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240368AbjADVzF (ORCPT ); Wed, 4 Jan 2023 16:55:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58446 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240340AbjADVys (ORCPT ); Wed, 4 Jan 2023 16:54:48 -0500 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4B253D124 for ; Wed, 4 Jan 2023 13:54:48 -0800 (PST) Received: by mail-pf1-x449.google.com with SMTP id j1-20020aa78001000000b0057d28e11cb6so16407264pfi.11 for ; Wed, 04 Jan 2023 13:54:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=RUdisve4gZWbsO7A2yUrvfD/yW1XxoGAj1HL6Sg3nEw=; b=cPobUe75znJ4wV/tcN55OXEYgws6ySy/GjbJuM7LpUXIQ9TsRPldLATpN2SdkEwS5D mSETIBqs/wQZXQPxIn6FtVqkTtTgHpzpJTS1i8JmOr+TlSD2WyIB6oABLgGG90yCnPau OZyuyeMlS+2pMHQI5mEC6C3IwgJSoVQENw+fTig0AWjWt5N/1G+AlHJTpT55FPQ2f/G7 97qYx9PS9miba6nNhjdJdjQLc6tAVnpHtU0Q+v0Ql5VHwSaRP9IG7wwCs1yzYbAbSxyv W1y2jD9qZskKBwodQixbkQcIsyOg6BXF7m51oLtWY7mLcfb3AGiENpGlR6Uke1oJhN7q QKiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=RUdisve4gZWbsO7A2yUrvfD/yW1XxoGAj1HL6Sg3nEw=; b=0jFFxTeOaIyBDuZdqHQQDi3ZsoUotuioLORWdaI8tXT+VcUlvxKsX54a+n/TsH8cwo UXt1LLElhAUTbGW3ZaZWHR9LUIHBqing5buh0FdIpMzo/W0NGw/N3MYUwlU6DP2JDBQg eX3aJlF46A8qm9qPsoMs4tqrq7YVvLVDuTkcq/oX23B7tZGMU3bxdbArzWqXU9hgFtyb XPDTwjtD0Hcre7ciO5jM53ECf3ppeR3maYeA7KAAxmLm+hu52yFBscoYw466wkHJsd1m BiWXkDkXwf3WJnk2WVeLA0fdmvWBT5y+rA+CxyyJWyFtwGrckuiNk27uERq1urxjxPPL pFqg== X-Gm-Message-State: AFqh2ko3VZRsoPsQGlWhqXa/hCV84eQv0SFhGFhZ82tT8/vFhX0UPR1c vdJN3p9IsPl4mOk9b4edT39TbUOEpFIWW6KhZA7EHD9Cw0mZhDKGFn/jkyz3kMAl6kxwfE4y1eo fZOPFtwNbgoyAQKpz6D1nGaZGoq4ByBTTOVqvgC91NWbhZbasIQQSKg5GpBXKdBXrwA== X-Google-Smtp-Source: AMrXdXsKWcxiskND/XKZVURIInJoIJ/1pgvwW5+ZkplaFah8L/vST5Vl/vHN4Ozq5lCRR+EloU++sBC8dUFbcT8= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a62:1558:0:b0:581:aeb8:4316 with SMTP id 85-20020a621558000000b00581aeb84316mr1713748pfv.33.1672869287693; Wed, 04 Jan 2023 13:54:47 -0800 (PST) Date: Wed, 4 Jan 2023 21:54:13 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230104215415.1083526-5-calvinwan@google.com> Subject: [PATCH v5 4/6] diff-lib: refactor match_stat_with_submodule From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Flatten out the if statements in match_stat_with_submodule so the logic is more readable and easier for future patches to add to. orig_flags didn't need to be set if the cache entry wasn't a GITLINK so defer setting it. Signed-off-by: Calvin Wan --- diff-lib.c | 28 +++++++++++++++++----------- 1 file changed, 17 insertions(+), 11 deletions(-) diff --git a/diff-lib.c b/diff-lib.c index dec040c366..64583fded0 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -73,18 +73,24 @@ static int match_stat_with_submodule(struct diff_options *diffopt, unsigned *dirty_submodule) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); - if (S_ISGITLINK(ce->ce_mode)) { - struct diff_flags orig_flags = diffopt->flags; - if (!diffopt->flags.override_submodule_config) - set_diffopt_flags_from_submodule_config(diffopt, ce->name); - if (diffopt->flags.ignore_submodules) - changed = 0; - else if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, - diffopt->flags.ignore_untracked_in_submodules); - diffopt->flags = orig_flags; + struct diff_flags orig_flags; + + if (!S_ISGITLINK(ce->ce_mode)) + return changed; + + orig_flags = diffopt->flags; + if (!diffopt->flags.override_submodule_config) + set_diffopt_flags_from_submodule_config(diffopt, ce->name); + if (diffopt->flags.ignore_submodules) { + changed = 0; + goto cleanup; } + if (!diffopt->flags.ignore_dirty_submodules && + (!changed || diffopt->flags.dirty_submodules)) + *dirty_submodule = is_submodule_modified(ce->name, + diffopt->flags.ignore_untracked_in_submodules); +cleanup: + diffopt->flags = orig_flags; return changed; } From patchwork Wed Jan 4 21:54:14 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13089086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 521CBC53210 for ; Wed, 4 Jan 2023 21:55:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240375AbjADVzG (ORCPT ); Wed, 4 Jan 2023 16:55:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58462 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240358AbjADVyv (ORCPT ); Wed, 4 Jan 2023 16:54:51 -0500 Received: from mail-pj1-x1049.google.com (mail-pj1-x1049.google.com [IPv6:2607:f8b0:4864:20::1049]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 152941EAEC for ; Wed, 4 Jan 2023 13:54:50 -0800 (PST) Received: by mail-pj1-x1049.google.com with SMTP id o21-20020a17090aac1500b00226349b1e06so9810916pjq.4 for ; Wed, 04 Jan 2023 13:54:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1BNqSfQOU3Hx8scQJxkrLIOVgoTWnsofjmodcNJQEKw=; b=enuydBnchezqfZNg+wjYabfVRM+pHK/2zMwJvbgRrBsczp/yqJ1dBjXNMFdA599kM+ RF6z/CR6hgyXvI/sobfKDelkP3aat1AEKLkcAdH1PLibTy5l39EwKhvIuyJSzyCoUK3+ 6Cak2+SYnpLMZ7YN0WmN+0okUpdDO5kzkxCgbWnRtTu2Wmx0EC2Q1oI9n7lIemc7Jtck rOfvBUkLnX/tNqKjJ/guKXnp3+pKKgfuJ03hh1nNKJKT+bFoTMLYYkJUGUvvTv+vdScL Tsz4jGPf6lN93lgjK1JfVb+eHJU2CbjYvN8NVdCfIR3DjsobTfTBEBoq8soKG7WNYRVn W7Ew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1BNqSfQOU3Hx8scQJxkrLIOVgoTWnsofjmodcNJQEKw=; b=L2lXE6h3y5oJOE+LwoTuZ2lcuCQ/dBMy91cfdMK7HPfGYSPcF5GBXts7TEufzHLuhB loCu3qcffP8P5IjUagfmYmixsEmb55ubrRKCj81KVGXdr7whjBsX+E7GDn8Y8l5v+Avn MAX4q3U4NHReXxmF8HHwezIfUqhQrt0EHbwccWeNvAzb4pB4+rfPbMRgpuZey8xC3HZ8 a/Udb0YSQ74EG5vogk2fnCLEXURhb2yB8cS0EH4D6xrQVE3LLKS+AUUWPRqshGWNO5Iy /Ur5GckUYzgHEGQs4MfxccvP0euEsv+eTcBHa50bYfXxFen9H5Bk1uYjI5HXHvqkddL1 ZQkQ== X-Gm-Message-State: AFqh2koBv0YfLfU/swG8A0FfBdkkljUwTkV+TubCoRhqCwXdTfZVQhny mMn36xdICTkPJqgnvZbhlfO36ajZ0ljfz63Tp4ZAqELL7AKvoXXiHOistebfPDGwnHDGGjU0ef2 0eRwQOxvlZGDEyCCN5e/BbcjvLyPFXE/SsacpppS6iLlQFwxVWhvR8l70DPewjj+zXw== X-Google-Smtp-Source: AMrXdXvrj1Rnv4iv6HZt29HrpXcC1ABw4U+glGq32bdbzxpvDs3fq5wq3Yq4Ug+tciasHG72DySXSn4t+slWOxw= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a05:6a00:4483:b0:57f:bdba:6d25 with SMTP id cu3-20020a056a00448300b0057fbdba6d25mr2821619pfb.48.1672869289313; Wed, 04 Jan 2023 13:54:49 -0800 (PST) Date: Wed, 4 Jan 2023 21:54:14 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230104215415.1083526-6-calvinwan@google.com> Subject: [PATCH v5 5/6] diff-lib: parallelize run_diff_files for submodules From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org During the iteration of the index entries in run_diff_files, whenever a submodule is found and needs its status checked, a subprocess is spawned for it. Instead of spawning the subprocess immediately and waiting for its completion to continue, hold onto all submodules and relevant information in a list. Then use that list to create tasks for run_processes_parallel. Subprocess output is duplicated and passed to status_pipe_output which stores it to be parsed on completion of the subprocess. Add config option submodule.diffJobs to set the maximum number of parallel jobs. The option defaults to 1 if unset. If set to 0, the number of jobs is set to online_cpus(). Since run_diff_files is called from many different commands, I chose to grab the config option in the function rather than adding variables to every git command and then figuring out how to pass them all in. Signed-off-by: Calvin Wan --- Documentation/config/submodule.txt | 12 +++ diff-lib.c | 84 +++++++++++++-- submodule.c | 168 +++++++++++++++++++++++++++++ submodule.h | 9 ++ t/t4027-diff-submodule.sh | 19 ++++ t/t7506-status-submodule.sh | 19 ++++ 6 files changed, 304 insertions(+), 7 deletions(-) diff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt index 6490527b45..3209eb8117 100644 --- a/Documentation/config/submodule.txt +++ b/Documentation/config/submodule.txt @@ -93,6 +93,18 @@ submodule.fetchJobs:: in parallel. A value of 0 will give some reasonable default. If unset, it defaults to 1. +submodule.diffJobs:: + Specifies how many submodules are diffed at the same time. A + positive integer allows up to that number of submodules diffed + in parallel. A value of 0 will give some reasonable default. + If unset, it defaults to 1. The diff operation is used by many + other git commands such as add, merge, diff, status, stash and + more. Note that the expensive part of the diff operation is + reading the index from cache or memory. Therefore multiple jobs + may be detrimental to performance if your hardware does not + support parallel reads or if the number of jobs greatly exceeds + the amount of supported reads. + submodule.alternateLocation:: Specifies how the submodules obtain alternates when submodules are cloned. Possible values are `no`, `superproject`. diff --git a/diff-lib.c b/diff-lib.c index 64583fded0..f51ea07f36 100644 --- a/diff-lib.c +++ b/diff-lib.c @@ -14,6 +14,7 @@ #include "dir.h" #include "fsmonitor.h" #include "commit-reach.h" +#include "config.h" /* * diff-files @@ -65,18 +66,23 @@ static int check_removed(const struct index_state *istate, const struct cache_en * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES * option is set, the caller does not only want to know if a submodule is * modified at all but wants to know all the conditions that are met (new - * commits, untracked content and/or modified content). + * commits, untracked content and/or modified content). If + * defer_submodule_status bit is set, dirty_submodule will be left to the + * caller to set. defer_submodule_status can also be set to 0 in this + * function if there is no need to check if the submodule is modified. */ static int match_stat_with_submodule(struct diff_options *diffopt, const struct cache_entry *ce, struct stat *st, unsigned ce_option, - unsigned *dirty_submodule) + unsigned *dirty_submodule, int *defer_submodule_status, + unsigned *ignore_untracked) { int changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option); struct diff_flags orig_flags; + int defer = 0; if (!S_ISGITLINK(ce->ce_mode)) - return changed; + goto ret; orig_flags = diffopt->flags; if (!diffopt->flags.override_submodule_config) @@ -86,11 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt, goto cleanup; } if (!diffopt->flags.ignore_dirty_submodules && - (!changed || diffopt->flags.dirty_submodules)) - *dirty_submodule = is_submodule_modified(ce->name, + (!changed || diffopt->flags.dirty_submodules)) { + if (defer_submodule_status && *defer_submodule_status) { + defer = 1; + *ignore_untracked = diffopt->flags.ignore_untracked_in_submodules; + } else { + *dirty_submodule = is_submodule_modified(ce->name, diffopt->flags.ignore_untracked_in_submodules); + } + } cleanup: diffopt->flags = orig_flags; +ret: + if (defer_submodule_status) + *defer_submodule_status = defer; return changed; } @@ -102,6 +117,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ? CE_MATCH_RACY_IS_DIRTY : 0); uint64_t start = getnanotime(); struct index_state *istate = revs->diffopt.repo->index; + struct string_list submodules = STRING_LIST_INIT_NODUP; diff_set_mnemonic_prefix(&revs->diffopt, "i/", "w/"); @@ -226,6 +242,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option) newmode = ce->ce_mode; } else { struct stat st; + unsigned ignore_untracked = 0; + int defer_submodule_status = !!revs->repo; changed = check_removed(istate, ce, &st); if (changed) { @@ -247,8 +265,26 @@ int run_diff_files(struct rev_info *revs, unsigned int option) } changed = match_stat_with_submodule(&revs->diffopt, ce, &st, - ce_option, &dirty_submodule); + ce_option, &dirty_submodule, + &defer_submodule_status, + &ignore_untracked); newmode = ce_mode_from_stat(ce, st.st_mode); + if (defer_submodule_status) { + struct submodule_status_util tmp = { + .changed = changed, + .dirty_submodule = 0, + .ignore_untracked = ignore_untracked, + .newmode = newmode, + .ce = ce, + .path = ce->name, + }; + struct string_list_item *item; + + item = string_list_append(&submodules, ce->name); + item->util = xmalloc(sizeof(tmp)); + memcpy(item->util, &tmp, sizeof(tmp)); + continue; + } } if (!changed && !dirty_submodule) { @@ -267,6 +303,40 @@ int run_diff_files(struct rev_info *revs, unsigned int option) ce->name, 0, dirty_submodule); } + if (submodules.nr > 0) { + int parallel_jobs; + if (git_config_get_int("submodule.diffjobs", ¶llel_jobs)) + parallel_jobs = 1; + else if (!parallel_jobs) + parallel_jobs = online_cpus(); + else if (parallel_jobs < 0) + die(_("submodule.diffjobs cannot be negative")); + + if (get_submodules_status(&submodules, parallel_jobs)) + die(_("submodule status failed")); + for (size_t i = 0; i < submodules.nr; i++) { + struct submodule_status_util *util = submodules.items[i].util; + struct cache_entry *ce = util->ce; + unsigned int oldmode; + const struct object_id *old_oid, *new_oid; + + if (!util->changed && !util->dirty_submodule) { + ce_mark_uptodate(ce); + mark_fsmonitor_valid(istate, ce); + if (!revs->diffopt.flags.find_copies_harder) + continue; + } + oldmode = ce->ce_mode; + old_oid = &ce->oid; + new_oid = util->changed ? null_oid() : &ce->oid; + diff_change(&revs->diffopt, oldmode, util->newmode, + old_oid, new_oid, + !is_null_oid(old_oid), + !is_null_oid(new_oid), + ce->name, 0, util->dirty_submodule); + } + } + string_list_clear(&submodules, 1); diffcore_std(&revs->diffopt); diff_flush(&revs->diffopt); trace_performance_since(start, "diff-files"); @@ -314,7 +384,7 @@ static int get_stat_data(const struct index_state *istate, return -1; } changed = match_stat_with_submodule(diffopt, ce, &st, - 0, dirty_submodule); + 0, dirty_submodule, NULL, NULL); if (changed) { mode = ce_mode_from_stat(ce, st.st_mode); oid = null_oid(); diff --git a/submodule.c b/submodule.c index 768d4b4cd7..a0ca646d9b 100644 --- a/submodule.c +++ b/submodule.c @@ -1369,6 +1369,17 @@ int submodule_touches_in_range(struct repository *r, return ret; } +struct submodule_parallel_status { + size_t index_count; + int result; + + struct string_list *submodule_names; + + /* Pending statuses by OIDs */ + struct status_task **oid_status_tasks; + int oid_status_tasks_nr, oid_status_tasks_alloc; +}; + struct submodule_parallel_fetch { /* * The index of the last index entry processed by @@ -1451,6 +1462,12 @@ struct fetch_task { struct oid_array *commits; /* Ensure these commits are fetched */ }; +struct status_task { + const char *path; + struct strbuf out; + int ignore_untracked; +}; + /** * When a submodule is not defined in .gitmodules, we cannot access it * via the regular submodule-config. Create a fake submodule, which we can @@ -1909,6 +1926,25 @@ static int parse_status_porcelain(char *str, size_t len, return 0; } +static void parse_status_porcelain_strbuf(struct strbuf *buf, + unsigned *dirty_submodule, + int ignore_untracked) +{ + struct string_list list = STRING_LIST_INIT_DUP; + struct string_list_item *item; + + string_list_split(&list, buf->buf, '\n', -1); + + for_each_string_list_item(item, &list) { + if (parse_status_porcelain(item->string, + strlen(item->string), + dirty_submodule, + ignore_untracked)) + break; + } + string_list_clear(&list, 0); +} + unsigned is_submodule_modified(const char *path, int ignore_untracked) { struct child_process cp = CHILD_PROCESS_INIT; @@ -1962,6 +1998,138 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked) return dirty_submodule; } +static struct status_task * +get_status_task_from_index(struct submodule_parallel_status *sps, + struct strbuf *err) +{ + for (; sps->index_count < sps->submodule_names->nr; sps->index_count++) { + struct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util; + struct status_task *task; + struct strbuf buf = STRBUF_INIT; + const char *git_dir; + + strbuf_addf(&buf, "%s/.git", util->path); + git_dir = read_gitfile(buf.buf); + if (!git_dir) + git_dir = buf.buf; + if (!is_git_directory(git_dir)) { + if (is_directory(git_dir)) + die(_("'%s' not recognized as a git repository"), git_dir); + strbuf_release(&buf); + /* The submodule is not checked out, so it is not modified */ + util->dirty_submodule = 0; + continue; + } + strbuf_release(&buf); + + task = xmalloc(sizeof(*task)); + task->path = util->path; + task->ignore_untracked = util->ignore_untracked; + strbuf_init(&task->out, 0); + sps->index_count++; + return task; + } + return NULL; +} + +static int get_next_submodule_status(struct child_process *cp, + struct strbuf *err, void *data, + void **task_cb) +{ + struct submodule_parallel_status *sps = data; + struct status_task *task = get_status_task_from_index(sps, err); + + if (!task) + return 0; + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env); + + strvec_init(&cp->args); + strvec_pushl(&cp->args, "status", "--porcelain=2", NULL); + if (task->ignore_untracked) + strvec_push(&cp->args, "-uno"); + + prepare_submodule_repo_env(&cp->env); + cp->git_cmd = 1; + cp->dir = task->path; + *task_cb = task; + return 1; +} + +static int status_start_failure(struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + + sps->result = 1; + strbuf_addf(err, + _("could not run 'git status --porcelain=2' in submodule %s"), + task->path); + return 0; +} + +static void status_duplicate_output(struct strbuf *out, + size_t offset, + void *cb, void *task_cb) +{ + struct status_task *task = task_cb; + + strbuf_add(&task->out, out->buf + offset, out->len - offset); + strbuf_setlen(out, offset); +} + +static int status_finish(int retvalue, struct strbuf *err, + void *cb, void *task_cb) +{ + struct submodule_parallel_status *sps = cb; + struct status_task *task = task_cb; + struct string_list_item *it = + string_list_lookup(sps->submodule_names, task->path); + struct submodule_status_util *util = it->util; + + if (retvalue) { + sps->result = 1; + strbuf_addf(err, + _("'git status --porcelain=2' failed in submodule %s"), + task->path); + } + + parse_status_porcelain_strbuf(&task->out, + &util->dirty_submodule, + util->ignore_untracked); + + free(task); + + return 0; +} + +int get_submodules_status(struct string_list *submodules, + int max_parallel_jobs) +{ + struct submodule_parallel_status sps = { + .submodule_names = submodules, + }; + const struct run_process_parallel_opts opts = { + .tr2_category = "submodule", + .tr2_label = "parallel/status", + + .processes = max_parallel_jobs, + + .get_next_task = get_next_submodule_status, + .start_failure = status_start_failure, + .duplicate_output = status_duplicate_output, + .task_finished = status_finish, + .data = &sps, + }; + + string_list_sort(sps.submodule_names); + run_processes_parallel(&opts); + + return sps.result; +} + int submodule_uses_gitfile(const char *path) { struct child_process cp = CHILD_PROCESS_INIT; diff --git a/submodule.h b/submodule.h index b52a4ff1e7..08d278a414 100644 --- a/submodule.h +++ b/submodule.h @@ -41,6 +41,13 @@ struct submodule_update_strategy { .type = SM_UPDATE_UNSPECIFIED, \ } +struct submodule_status_util { + int changed, ignore_untracked; + unsigned dirty_submodule, newmode; + struct cache_entry *ce; + const char *path; +}; + int is_gitmodules_unmerged(struct index_state *istate); int is_writing_gitmodules_ok(void); int is_staging_gitmodules_ok(struct index_state *istate); @@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r, int command_line_option, int default_option, int quiet, int max_parallel_jobs); +int get_submodules_status(struct string_list *submodules, + int max_parallel_jobs); unsigned is_submodule_modified(const char *path, int ignore_untracked); int submodule_uses_gitfile(const char *path); diff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh index 40164ae07d..e08ee315a7 100755 --- a/t/t4027-diff-submodule.sh +++ b/t/t4027-diff-submodule.sh @@ -34,6 +34,25 @@ test_expect_success setup ' subtip=$3 subprev=$2 ' +test_expect_success 'diff in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git diff && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git diff && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_expect_success 'git diff --raw HEAD' ' hexsz=$(test_oid hexsz) && git diff --raw --abbrev=$hexsz HEAD >actual && diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh index d050091345..52a82b703f 100755 --- a/t/t7506-status-submodule.sh +++ b/t/t7506-status-submodule.sh @@ -412,4 +412,23 @@ test_expect_success 'status with added file in nested submodule (short)' ' EOF ' +test_expect_success 'status in superproject with submodules respects parallel settings' ' + test_when_finished "rm -f trace.out" && + ( + GIT_TRACE=$(pwd)/trace.out git status && + grep "1 tasks" trace.out && + >trace.out && + + git config submodule.diffJobs 8 && + GIT_TRACE=$(pwd)/trace.out git status && + grep "8 tasks" trace.out && + >trace.out && + + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status && + grep "preparing to run up to [0-9]* tasks" trace.out && + ! grep "up to 0 tasks" trace.out && + >trace.out + ) +' + test_done From patchwork Wed Jan 4 21:54:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Calvin Wan X-Patchwork-Id: 13089087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6038C46467 for ; Wed, 4 Jan 2023 21:55:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S240392AbjADVzI (ORCPT ); Wed, 4 Jan 2023 16:55:08 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58502 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S240366AbjADVzB (ORCPT ); Wed, 4 Jan 2023 16:55:01 -0500 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84FB51EEEA for ; Wed, 4 Jan 2023 13:54:51 -0800 (PST) Received: by mail-pj1-x104a.google.com with SMTP id om16-20020a17090b3a9000b002216006cbffso23176717pjb.3 for ; Wed, 04 Jan 2023 13:54:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=9/vkAOAnT96FKvWsQTiyEgfPodVet54xzExY6d4HiUs=; b=CozFRcG3AVjIKnNyThp0xWnPU8Blif2uOZZUCUs0oWCno8M6PVDheJPSyBf5OxAerE L0PBd3ayzmlVEuKt9SsleduCkJuBxKqDr5evzhysP2e3DznLS22W4P72aPdSYA9ZnpfD R8ElWybAPHVNVS8obDIQ2pZVZrtsIZSztFJan40+D7dMBcThVco8vTgXHBTtD1Qzc4H+ Ak6k7LSqYe4rJL54tTiIUq0HeNUHhbBRGwiKcAXXB4QYYjIaWucl2W6ezVuY7MIL7esN gQVwOA+cgu7DcWRR5HwwLgyrvyWwV5//XHZtROyEi0VrAC8OjAK+JNoVAwxuuCF1oZe+ Gwjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9/vkAOAnT96FKvWsQTiyEgfPodVet54xzExY6d4HiUs=; b=2WD2csHMTF4uH8UFiUD2eS0Bc6BbpsQa1gbd7TOmw6+4BZYciqCf4zUzRddfPfmx31 4mgNuC971EBnEyM2KCK90AIlrDPaXzoOEdzdRawjs75YsTViQm9UL2/2k9nTw64ps6vX co66efJCslbp+qUnVwpLnFUHDiEzzy4ShV79AW/XYb5ttV9gz61jhlTge3ojS2FxC3aS o2RDBfMNmr91y4xJSgBUWKSSIDWVxudT6MoPtDQAC7pZAiIINsmIMN/C/5/qm/bYqrYb kZrAgxKatCNblVmXF9v2saBwDLq4e2qq6YG19qAo2/ASvpBw8/tFbLtHvOe9+AUENB/C M46g== X-Gm-Message-State: AFqh2kppUU1Vqp9B6Wv+vpzglVCN6GhO37/9HEM/MB8sOhkOSL5++75E aX/+mkvOXfy/Ksb513dfjvrt1twO0OBbJllC3ddLX8srN6ei5yGwyGY2vczWowYAbMrMGnzJ1Pn iyzR/aZM2xJOc/Ietcz6dqEL9izJelQH8cHstIGde2Zn0WBi0XwJiwJ9A92o9URbEfA== X-Google-Smtp-Source: AMrXdXtbDL72GdrP6ZgJlUO4ARvocrdpKeO19Zty4AmZnOfCtZHSBezcepWwfwl4a9SekGDLGA42cKZFourSjRY= X-Received: from barleywine.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:3bd4]) (user=calvinwan job=sendgmr) by 2002:a17:902:f314:b0:192:f1fc:e8b8 with SMTP id c20-20020a170902f31400b00192f1fce8b8mr279004ple.37.1672869290989; Wed, 04 Jan 2023 13:54:50 -0800 (PST) Date: Wed, 4 Jan 2023 21:54:15 +0000 In-Reply-To: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.39.0.314.g84b9a713c41-goog Message-ID: <20230104215415.1083526-7-calvinwan@google.com> Subject: [PATCH v5 6/6] submodule: call parallel code from serial status From: Calvin Wan To: git@vger.kernel.org Cc: Calvin Wan , emilyshaffer@google.com, avarab@gmail.com, phillip.wood123@gmail.com, chooglen@google.com, newren@gmail.com, jonathantanmy@google.com Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Remove the serial implementation of status inside of is_submodule_modified since the parallel implementation of status with one job accomplishes the same task. Combine parse_status_porcelain and parse_status_porcelain_strbuf since the only other caller of parse_status_porcelain was in is_submodule_modified Signed-off-by: Calvin Wan --- submodule.c | 143 ++++++++++++++++++---------------------------------- 1 file changed, 48 insertions(+), 95 deletions(-) diff --git a/submodule.c b/submodule.c index a0ca646d9b..042e26137f 100644 --- a/submodule.c +++ b/submodule.c @@ -1887,46 +1887,7 @@ int fetch_submodules(struct repository *r, return spf.result; } -static int parse_status_porcelain(char *str, size_t len, - unsigned *dirty_submodule, - int ignore_untracked) -{ - /* regular untracked files */ - if (str[0] == '?') - *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '1' || - str[0] == '2') { - /* T = line type, XY = status, SSSS = submodule state */ - if (len < strlen("T XY SSSS")) - BUG("invalid status --porcelain=2 line %s", - str); - - if (str[5] == 'S' && str[8] == 'U') - /* nested untracked file */ - *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; - - if (str[0] == 'u' || - str[0] == '2' || - memcmp(str + 5, "S..U", 4)) - /* other change */ - *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; - } - - if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && - ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || - ignore_untracked)) { - /* - * We're not interested in any further information from - * the child any more, neither output nor its exit code. - */ - return 1; - } - return 0; -} - -static void parse_status_porcelain_strbuf(struct strbuf *buf, +static void parse_status_porcelain(struct strbuf *buf, unsigned *dirty_submodule, int ignore_untracked) { @@ -1936,66 +1897,58 @@ static void parse_status_porcelain_strbuf(struct strbuf *buf, string_list_split(&list, buf->buf, '\n', -1); for_each_string_list_item(item, &list) { - if (parse_status_porcelain(item->string, - strlen(item->string), - dirty_submodule, - ignore_untracked)) + char *str = item->string; + /* regular untracked files */ + if (str[0] == '?') + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '1' || + str[0] == '2') { + /* T = line type, XY = status, SSSS = submodule state */ + if (strlen(str) < strlen("T XY SSSS")) + BUG("invalid status --porcelain=2 line %s", + str); + + if (str[5] == 'S' && str[8] == 'U') + /* nested untracked file */ + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED; + + if (str[0] == 'u' || + str[0] == '2' || + memcmp(str + 5, "S..U", 4)) + /* other change */ + *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED; + } + + if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) && + ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) || + ignore_untracked)) { + /* + * We're not interested in any further information from + * the child any more, neither output nor its exit code. + */ break; + } } string_list_clear(&list, 0); } unsigned is_submodule_modified(const char *path, int ignore_untracked) { - struct child_process cp = CHILD_PROCESS_INIT; - struct strbuf buf = STRBUF_INIT; - FILE *fp; - unsigned dirty_submodule = 0; - const char *git_dir; - int ignore_cp_exit_code = 0; - - strbuf_addf(&buf, "%s/.git", path); - git_dir = read_gitfile(buf.buf); - if (!git_dir) - git_dir = buf.buf; - if (!is_git_directory(git_dir)) { - if (is_directory(git_dir)) - die(_("'%s' not recognized as a git repository"), git_dir); - strbuf_release(&buf); - /* The submodule is not checked out, so it is not modified */ - return 0; - } - strbuf_reset(&buf); - - strvec_pushl(&cp.args, "status", "--porcelain=2", NULL); - if (ignore_untracked) - strvec_push(&cp.args, "-uno"); - - prepare_submodule_repo_env(&cp.env); - cp.git_cmd = 1; - cp.no_stdin = 1; - cp.out = -1; - cp.dir = path; - if (start_command(&cp)) - die(_("Could not run 'git status --porcelain=2' in submodule %s"), path); - - fp = xfdopen(cp.out, "r"); - while (strbuf_getwholeline(&buf, fp, '\n') != EOF) { - char *str = buf.buf; - const size_t len = buf.len; - - ignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule, - ignore_untracked); - if (ignore_cp_exit_code) - break; - } - fclose(fp); - - if (finish_command(&cp) && !ignore_cp_exit_code) - die(_("'git status --porcelain=2' failed in submodule %s"), path); + struct submodule_status_util util = { + .dirty_submodule = 0, + .ignore_untracked = ignore_untracked, + .path = path, + }; + struct string_list sub = STRING_LIST_INIT_NODUP; + struct string_list_item *item; - strbuf_release(&buf); - return dirty_submodule; + item = string_list_append(&sub, path); + item->util = &util; + if (get_submodules_status(&sub, 1)) + die(_("submodule status failed")); + return util.dirty_submodule; } static struct status_task * @@ -2096,9 +2049,9 @@ static int status_finish(int retvalue, struct strbuf *err, task->path); } - parse_status_porcelain_strbuf(&task->out, - &util->dirty_submodule, - util->ignore_untracked); + parse_status_porcelain(&task->out, + &util->dirty_submodule, + util->ignore_untracked); free(task);