From patchwork Tue Sep 25 19:47:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614751 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD9C015E8 for ; Tue, 25 Sep 2018 19:48:11 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9ED822ABB8 for ; Tue, 25 Sep 2018 19:48:11 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 904672ABD1; Tue, 25 Sep 2018 19:48:11 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60A6A2ABB8 for ; Tue, 25 Sep 2018 19:48:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727511AbeIZB5O (ORCPT ); Tue, 25 Sep 2018 21:57:14 -0400 Received: from mail-it1-f202.google.com ([209.85.166.202]:40645 "EHLO mail-it1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB5O (ORCPT ); Tue, 25 Sep 2018 21:57:14 -0400 Received: by mail-it1-f202.google.com with SMTP id k143-v6so9130735ite.5 for ; Tue, 25 Sep 2018 12:48:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8Pj1l1qpHHAI3KTncoubtJ8GKbEYHB+cWW0tdRyGZrg=; b=ijvi20nbbbVY1gW74LUli6N31tRaOsMJ85Epq0rBCM5gNBCX04orOSSA0NJCnf05od QXdcn11QI7ct/TobbHGFDshZS+T8zS/F1MqB+GfI2WD77JqimNkJkpMmF2sfkR4Ko+gY 28VZgrFyg3YjA9zOCtlSuPX5t06+Srgt9+ITSYYA2qKHPewBCA9SyNKM/BuIYUuBEy9e eSChdP3acEQn//o5EHOSyNmw3c8qAbfHZlvunrEwRidaqubzOkOc/spVhw/Hl5QoQqnF T0DLHTt6v7Qrm5WPVfFVnYvtCBRC6kDLdIGBEn8bNQxNEJnwbQDpLmZfIwF5pKkJcwOI uMPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8Pj1l1qpHHAI3KTncoubtJ8GKbEYHB+cWW0tdRyGZrg=; b=GwzkzV0+7t4ikdl4rfdAnlaM9Z1SNrS8PfPMAf9oP35rEjoAh8cJSFOt+0BYKLDucF TdEkYex1/PSv2XpWmRedgpb6dh6lVrxpvRN2qhu1OWcymnqydxTQ3ZesGMCXbZizQmzz S4GgL/ZrouGVW94mwaq+iIS4fuDvhMIW+20M4XYcYNQIjYa0L2Y5doLUJ3aKiLHTC2xO 11lEBe1keDiTd9tqumSEx66GfY7c0Mjx+kp3JY442dSYo9DeC513Q7aAVJQkDztgWvZ9 E2jD4WEB7tOuQaVhdGxKJ1e7cMNbs2zb26fbOstM5ezfnLZBZQ2W+h1jdP9BO4uwEqF4 Fjtg== X-Gm-Message-State: ABuFfoibx2CAo4qgoOsKCrWBZdPHCENFIRZ6bHttvGXek45wb0ltLZiw C2eD14Fyis+5xdWb3+JsJnobj9j0MoaJ9V9lUOVw62YL2w13pgAFnFS4AD3khj56LjZ2Z7Y6+o2 T1gffRtAhmJm0wW937AoxeBKoZnMfegUh6XzcSKllciYRTGOMYCWUdKFrOj7d X-Google-Smtp-Source: ACcGV60hjg2MDwj8KMB0IsltPnx2NTaENCUSsY/bsfhhqVrTYy+yNt84B23lb5cQcRmymS0gGMIg91GihGFP X-Received: by 2002:a24:d49:: with SMTP id 70-v6mr1884334itx.5.1537904885357; Tue, 25 Sep 2018 12:48:05 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:47 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-2-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 1/9] sha1-array: provide oid_array_filter From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Helped-by: Junio C Hamano Signed-off-by: Stefan Beller --- Documentation/technical/api-oid-array.txt | 5 +++++ sha1-array.c | 17 +++++++++++++++++ sha1-array.h | 3 +++ 3 files changed, 25 insertions(+) diff --git a/Documentation/technical/api-oid-array.txt b/Documentation/technical/api-oid-array.txt index 9febfb1d528..c97428c2c34 100644 --- a/Documentation/technical/api-oid-array.txt +++ b/Documentation/technical/api-oid-array.txt @@ -48,6 +48,11 @@ Functions is not sorted, this function has the side effect of sorting it. +`oid_array_filter`:: + Apply the callback function `want` to each entry in the array, + retaining only the entries for which the function returns true. + Preserve the order of the entries that are retained. + Examples -------- diff --git a/sha1-array.c b/sha1-array.c index b94e0ec0f5e..d922e94e3fc 100644 --- a/sha1-array.c +++ b/sha1-array.c @@ -77,3 +77,20 @@ int oid_array_for_each_unique(struct oid_array *array, } return 0; } + +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cb_data) +{ + unsigned nr = array->nr, src, dst; + struct object_id *oids = array->oid; + + for (src = dst = 0; src < nr; src++) { + if (want(&oids[src], cb_data)) { + if (src != dst) + oidcpy(&oids[dst], &oids[src]); + dst++; + } + } + array->nr = dst; +} diff --git a/sha1-array.h b/sha1-array.h index 232bf950172..55d016c4bf7 100644 --- a/sha1-array.h +++ b/sha1-array.h @@ -22,5 +22,8 @@ int oid_array_for_each(struct oid_array *array, int oid_array_for_each_unique(struct oid_array *array, for_each_oid_fn fn, void *data); +void oid_array_filter(struct oid_array *array, + for_each_oid_fn want, + void *cbdata); #endif /* SHA1_ARRAY_H */ From patchwork Tue Sep 25 19:47:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614757 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 74EF615E8 for ; Tue, 25 Sep 2018 19:48:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 65DFA2A940 for ; Tue, 25 Sep 2018 19:48:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5A03D2ABB8; Tue, 25 Sep 2018 19:48:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 493AB2ABC8 for ; Tue, 25 Sep 2018 19:48:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727594AbeIZB5Q (ORCPT ); Tue, 25 Sep 2018 21:57:16 -0400 Received: from mail-ot1-f74.google.com ([209.85.210.74]:38039 "EHLO mail-ot1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB5Q (ORCPT ); Tue, 25 Sep 2018 21:57:16 -0400 Received: by mail-ot1-f74.google.com with SMTP id j65-v6so28164181otc.5 for ; Tue, 25 Sep 2018 12:48:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Np6Su2BdsDTx+mOYUhDKyUAc/s/nHx2bm7iMRHYO0hI=; b=p1+hU03AZu2eriI50o9fi4wgUvzOWJWUG+/l4F64je4vh8o1E6CmGL8MKANPDk0ujn /ZP6nDZreEUOqj6M0aELKIQRePm5kc1fK96DE45SHvh/E40WbmaLeVrn/M/Tqt+GXxu0 kOllBkBqbxvZUTuRQ/7AWXAzT5t2eTLh+3YBwS9Xq/IENSz1YrzcYa6E5lpVALK6WTW5 Rh7uxWHnA8LyAq3xsOl2gYU6Ztx0Ytcvcl3A1mZRHHeMQunGvXsSFhG1wGMkfgoaN+Up 7OcuLypkuW4G/5SJDyCM8dZerGex16S01/Jnm8XTFZzGZlJUdTYV5eiA1oEHlaQwBt6m 8dAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Np6Su2BdsDTx+mOYUhDKyUAc/s/nHx2bm7iMRHYO0hI=; b=e1nYtruK+Un52eX4nyinsfuVD1zgNYZQ+nkwLnW8jsrEcstuJd1lSYYHLied1JVAEv 7UC1zeajerighmj0ZYAAbTYpufAMPzUB+SO51sFTzWW+/3LwdrwpJhjGR5DkHcnMGw7Z YxWxMl0XWsOXJSlLsYqTmep1Dj4VuYV8+9DXuy5xqfsn83RbDlZjtZcuETy2lhdgJIp+ 4PeDCrDmqHQ9Ygum2LpZtfwuttp4oOmo2uWHK+epgM07M2GgWqPxNbJzphYFO1YlNnL+ 2I1L1yBTThnkOon8jgwhkZ9hd5T5fH/PLiW5DtvZUjLMwzLPIUFKAv4iWF0ZLaXUPfCm QfKA== X-Gm-Message-State: ABuFfoh+NrLQDjl2c1PuGIGSbZxtspw94Waj6zjzB7dQQkNFNCOjpVrr R1ZBxxrZwvT8nNkRx1UkZvegu09bqz4/tVrvLHCbFHk9/SytNSE/WYk44B4HyR3JMJ+oG+nWvQW DNHaros5GxyhTeUoZ0Iu09sZJI5mjaH3Ci4r8UDhLbJPgsQcexxvDT8HXf6SH X-Google-Smtp-Source: ACcGV62oDDo/qH5txF3y7itZ0DXVPnEWCs6o+zCtuAbP6fpsS8M8joNJB1yPvC/GiTlia3+JWcgg6LPZwWuz X-Received: by 2002:a9d:701a:: with SMTP id k26-v6mr3256278otj.65.1537904887651; Tue, 25 Sep 2018 12:48:07 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:48 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-3-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 2/9] submodule.c: fix indentation From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The submodule subsystem is really bad at staying within 80 characters. Fix it while we are here. Signed-off-by: Stefan Beller --- submodule.c | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/submodule.c b/submodule.c index b53cb6e9c47..0de9e2800ad 100644 --- a/submodule.c +++ b/submodule.c @@ -1244,7 +1244,8 @@ static int get_next_submodule(struct child_process *cp, if (!submodule) { const char *name = default_name_or_path(ce->name); if (name) { - default_submodule.path = default_submodule.name = name; + default_submodule.path = name; + default_submodule.name = name; submodule = &default_submodule; } } @@ -1254,8 +1255,10 @@ static int get_next_submodule(struct child_process *cp, default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || !unsorted_string_list_lookup(&changed_submodule_names, - submodule->name)) + if (!submodule || + !unsorted_string_list_lookup( + &changed_submodule_names, + submodule->name)) continue; default_argv = "on-demand"; break; From patchwork Tue Sep 25 19:47:49 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614753 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A19B0174A for ; Tue, 25 Sep 2018 19:48:12 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 930422ABB8 for ; Tue, 25 Sep 2018 19:48:12 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 873532ABD3; Tue, 25 Sep 2018 19:48:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE3CF2ABB8 for ; Tue, 25 Sep 2018 19:48:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728067AbeIZB5S (ORCPT ); Tue, 25 Sep 2018 21:57:18 -0400 Received: from mail-ot1-f74.google.com ([209.85.210.74]:44617 "EHLO mail-ot1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB5S (ORCPT ); Tue, 25 Sep 2018 21:57:18 -0400 Received: by mail-ot1-f74.google.com with SMTP id v62-v6so27953637ota.11 for ; Tue, 25 Sep 2018 12:48:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MazBY761I0QXmSzjxhMEG0YAD7M94SZQLpZC0Ra4T8A=; b=ipOz00xKCZKRQFlmMaZQAWFPWwARz1usFX7A0CLfCB5Vby4w06rpd8CtM59TSw3WXS CrL8wov56fSNydn9o7yQBEV6YowPTuWUkpZ+HTZsTKv6HUjbSpQc+u7fisYEWKSh9PAC bHyP9SysnXt3YCi5fEfq0iM0fV0Y7du01NenJ2jfcu8Fr19lTDT6nUtunyObhdR+Y/Xf Arx8GCSuzLFac+XNrPN8JeuRy25mmxr6OjaIgVhQaQVX7VOcui9T1Bl3NfPtVixUNG9R FYvKur6larOncVQTfp3ixnJgFZUONS1Q7iH/mIoEDsOYlF8VpL1b2kX+DMcTkm2jg++m eoKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MazBY761I0QXmSzjxhMEG0YAD7M94SZQLpZC0Ra4T8A=; b=RZ0SBdPcQbHgDRMVyjo0V+3Sii1fgXerzSc8EJczz2ZOtfMHZ5uiXIILpkrc4FmzAR tis+U4+6nOnZWvjDT2xvFEo5gK+bZAu169gyBVJS6kFhiOAq9G0VLLl4Mi3p5o+DS3PP hS1V6b4NnFnRMXWe3M/1kLrqy/bmD3QAO9YH4/suMtMJ9rDuPbY35wJBvIPfIhyLac01 wkyn+UBChJEjcjecgjXa0wTohnANmdsrZo8uJCQnFFpX8VRbXIf08H6uzLqoSpUrRTpT 2Pa6ZWs2vspin7LSaJVite58CUaKhjlFcqJJ33HrGq/RFHjdspMY0HQG3KXHtIdRwjCv tPkQ== X-Gm-Message-State: ABuFfojxT0z1E48Vx0rrKAjN5gsPV2o9u8Cq5zbq/faosPWEtPRlzDWx mcHbDYuxGUJ9TojIbmYaE+lGHGOnmR3ELPpKqr1MNVQ0LlFN7EFfRUQyhIhsbScYu5ohMmefrNe UAXgPnBEFhiw1V3Lg4mHaw6L397cBY6+gP5z464W6JEgdTJwgMRUW7vh92wOr X-Google-Smtp-Source: ACcGV63yCHWy4IxpO6LO67y38jzIdhfgqA3m89j+c/IktZlW1pHgxopbBijBRLIbvpsnB7Lil409f8tEUCl5 X-Received: by 2002:aca:ce05:: with SMTP id e5-v6mr3054554oig.9.1537904889977; Tue, 25 Sep 2018 12:48:09 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:49 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-4-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 3/9] submodule.c: sort changed_submodule_names before searching it From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP We can string_list_insert() to maintain sorted-ness of the list as we find new items, or we can string_list_append() to build an unsorted list and sort it at the end just once. To pick which one is more appropriate, we notice the fact that we discover new items more or less in the already sorted order. That makes "append then sort" more appropriate. Signed-off-by: Stefan Beller --- submodule.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/submodule.c b/submodule.c index 0de9e2800ad..22c64bd8559 100644 --- a/submodule.c +++ b/submodule.c @@ -1256,7 +1256,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || - !unsorted_string_list_lookup( + !string_list_lookup( &changed_submodule_names, submodule->name)) continue; @@ -1350,6 +1350,7 @@ int fetch_populated_submodules(struct repository *r, /* default value, "--submodule-prefix" and its value are added later */ calculate_changed_submodule_paths(); + string_list_sort(&changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, From patchwork Tue Sep 25 19:47:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614761 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A8DDB174A for ; Tue, 25 Sep 2018 19:48:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 976122ABBA for ; Tue, 25 Sep 2018 19:48:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8B8A92ABC8; Tue, 25 Sep 2018 19:48:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4A2252ABB8 for ; Tue, 25 Sep 2018 19:48:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728077AbeIZB5V (ORCPT ); Tue, 25 Sep 2018 21:57:21 -0400 Received: from mail-qt1-f201.google.com ([209.85.160.201]:50770 "EHLO mail-qt1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB5V (ORCPT ); Tue, 25 Sep 2018 21:57:21 -0400 Received: by mail-qt1-f201.google.com with SMTP id e14-v6so9442390qtp.17 for ; Tue, 25 Sep 2018 12:48:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=stcmNKivRrqfXeP6KQWIfEL9itU1hFxTUcZ7vo1OUjQ=; b=Qnfo/yBQp83b8zZavrCjxAhMFGOtQD6sYjRLBUEzEnGEtkF8QWv310U6vkCReIlvwr 3hPVvd/rCrVafD7me/5WIO5PeBAHLGfLH/gbstp6rH0WJZdaA7bw20GsffqRZYO5pAs/ E2t9Hdcf/B/AtGsotmUf63F7jZUg/cmq6A0VYBvA2SzgHIjpjABdDNHwQmQY7VTyxEQE gUFcyJbtDBJAuYfvFzQOoG5lraNw5JBAvaDPhl9nPUj2eMgAhptoWN7R7L78/qqx74Om R2he4KFvSaQBzNhN4hdiOVrKhjau6ftZKKZsTtRLJ7w0qH2cUGCxgjRU1LphGgPurh6F 7pcA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=stcmNKivRrqfXeP6KQWIfEL9itU1hFxTUcZ7vo1OUjQ=; b=AIeDfXR4q6w4+gLZ8QFD/1V8nMGgiT+77++XXytMXbSE9XE5Qx0KyvB6knNKnDQWFi hSjCeUCanG83Ex+XDoNm0AbmDDeUq3IJ/ujxwvzfyP6SrTQx3sRveYY/pDV/NfTZLdQc GUKQbJP57gEpyEbc2hAf64XRZ0EZpZdJvZI3BspgJBTeXrbIiImtglZZ/2GjFyBg3bS4 Rg8z0x3q6ZpOnaHGGpCFnGc8S514S6s53gZdZESDTgPUlEuOC8MbHJZrYcZtGvuXlbO4 cTNVKYxuzx6j07VvOp0SsPmSMOsbFfC4zeOEHUeKIqwxUq9BBrIxai51ez9ZrSe0MLWP NNWg== X-Gm-Message-State: ABuFfojfqGUkxXWeu+y+8F3ByPfexI3P+ulQpeCByyI+lZIHteD7d9g7 ISk5RHQXF5fu/T+y+r7+rOa+jBhm5yhpwY6psznf4IQ0sPGTawBtBai+f2Xh0lhj6j0xjC7qR6I yboWDE8Ra8QlTPSENfRPGXq7Y44Y+IzMNDKeSchJtD1nnwbBHUjjo+bPztS5+ X-Google-Smtp-Source: ACcGV63vU2AnUGDJmKDBS/WHLT51RSA0S+MQ81CiDo/v3IPUk59R3u/HG8qtHvwHZUV9I4Nm2MMx51hOch6Z X-Received: by 2002:ad4:4152:: with SMTP id z18-v6mr848765qvp.40.1537904892377; Tue, 25 Sep 2018 12:48:12 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:50 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-5-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 4/9] submodule: move global changed_submodule_names into fetch submodule struct From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The `changed_submodule_names` are only used for fetching, so let's make it part of the struct that is passed around for fetching submodules. Signed-off-by: Stefan Beller --- submodule.c | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/submodule.c b/submodule.c index 22c64bd8559..17103379ba4 100644 --- a/submodule.c +++ b/submodule.c @@ -25,7 +25,7 @@ #include "commit-reach.h" static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF; -static struct string_list changed_submodule_names = STRING_LIST_INIT_DUP; + static int initialized_fetch_ref_tips; static struct oid_array ref_tips_before_fetch; static struct oid_array ref_tips_after_fetch; @@ -1110,7 +1110,22 @@ void check_for_new_submodule_commits(struct object_id *oid) oid_array_append(&ref_tips_after_fetch, oid); } -static void calculate_changed_submodule_paths(void) +struct submodule_parallel_fetch { + int count; + struct argv_array args; + struct repository *r; + const char *prefix; + int command_line_option; + int default_option; + int quiet; + int result; + + struct string_list changed_submodule_names; +}; +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } + +static void calculate_changed_submodule_paths( + struct submodule_parallel_fetch *spf) { struct argv_array argv = ARGV_ARRAY_INIT; struct string_list changed_submodules = STRING_LIST_INIT_DUP; @@ -1148,7 +1163,8 @@ static void calculate_changed_submodule_paths(void) continue; if (!submodule_has_commits(path, commits)) - string_list_append(&changed_submodule_names, name->string); + string_list_append(&spf->changed_submodule_names, + name->string); } free_submodules_oids(&changed_submodules); @@ -1185,18 +1201,6 @@ int submodule_touches_in_range(struct object_id *excl_oid, return ret; } -struct submodule_parallel_fetch { - int count; - struct argv_array args; - struct repository *r; - const char *prefix; - int command_line_option; - int default_option; - int quiet; - int result; -}; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0} - static int get_fetch_recurse_config(const struct submodule *submodule, struct submodule_parallel_fetch *spf) { @@ -1257,7 +1261,7 @@ static int get_next_submodule(struct child_process *cp, case RECURSE_SUBMODULES_ON_DEMAND: if (!submodule || !string_list_lookup( - &changed_submodule_names, + &spf->changed_submodule_names, submodule->name)) continue; default_argv = "on-demand"; @@ -1349,8 +1353,8 @@ int fetch_populated_submodules(struct repository *r, argv_array_push(&spf.args, "--recurse-submodules-default"); /* default value, "--submodule-prefix" and its value are added later */ - calculate_changed_submodule_paths(); - string_list_sort(&changed_submodule_names); + calculate_changed_submodule_paths(&spf); + string_list_sort(&spf.changed_submodule_names); run_processes_parallel(max_parallel_jobs, get_next_submodule, fetch_start_failure, @@ -1359,7 +1363,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&changed_submodule_names, 1); + string_list_clear(&spf.changed_submodule_names, 1); return spf.result; } From patchwork Tue Sep 25 19:47:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614759 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5F88F174A for ; Tue, 25 Sep 2018 19:48:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 519F02ABBA for ; Tue, 25 Sep 2018 19:48:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4435C2ABCF; Tue, 25 Sep 2018 19:48:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF7392ABBA for ; Tue, 25 Sep 2018 19:48:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728090AbeIZB5X (ORCPT ); Tue, 25 Sep 2018 21:57:23 -0400 Received: from mail-yb1-f201.google.com ([209.85.219.201]:50186 "EHLO mail-yb1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB5X (ORCPT ); Tue, 25 Sep 2018 21:57:23 -0400 Received: by mail-yb1-f201.google.com with SMTP id z8-v6so10384824ybo.17 for ; Tue, 25 Sep 2018 12:48:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OovX9oELGzUKOdvo4/EwToobohXdU2DiOOONq/EmaAw=; b=matiTTXb/eGpF1m5H26IcWdCOsSf+Twg7AkmVxTV/EE9RPFLbL/hE52+LV7mK/Yyal m6YfKl0YpHA7MxOHiv2Dfwj6EjMNPic8m3MzqswTRWp0yH3cmFtFVZI4LE0nchuFUXgb oKMq/7NIfcLvSkIAmGqwPLkoR0kwfpx2Oe3c1OQd3iR4pZaQrHQkiyjXv1+lRZPcb2hC 2tDqIrJuT7Fvvz+GJDaw7gZ/wyjwznfVZ/qKdwsbfbf6MggXq1r4Biu61aKAfNwAJdKO LmwxBDcWF9+XbsU+zpA8XOHA4KUpS4ZzECXDBJOOycuMiFLQUEtgFrBQn/pmKp+m+rUQ CFrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=OovX9oELGzUKOdvo4/EwToobohXdU2DiOOONq/EmaAw=; b=rgNDziRbBfatIHX4qNrWLlfEPhBGQRFP5nQzDuXJXJ05PPZt//UnW82mEgmVNBCCOI GbrLxWMC7KVnGzlrRf6Jzl2SsHjYPG60saUgoYXllm8f8KApGh5lfjPgnl8weozlN997 eM7x2v5bQ606iSc4K4clmtK1/6Qq8ubydSOOvNslFHkSwzfEPTNP8HWYxo75ZQCem6wf fBcC4Jc5/zwHFrgtaS+2/YNAOBZZvpY+ea3RrieyqKU/a6Jas052etaGUGjbhiTz/EeC WwiyXUEpRULNxvrhtRK0LtteeK/HULtMxXD+69xirLVObW6bhaXUSgK2CT1auYRc3jNA Z7NA== X-Gm-Message-State: ABuFfoinBDeqFkCwEgUr9VdULuKypNXXfoPHxHvhZMuTETEQIEGbA5tD hADjUPEVC6SCSkt6OWVazmOrDgw0w190xB3u3CT6K6OqRKCdVm7HBIis7H/SLo3K+WEoEd7fKw4 TiVcSwoYRjXeUoJcktE3CkyPkm55JptaHhB9q8t9F7eRyLiVIbl/6jJ7P1rFX X-Google-Smtp-Source: ACcGV61LtpCg2z5UChn94wg+013obmozJ7+tTGJo8sxo44SPmLokheS73spvw3DlHpAHcAYRRHb5S2A1N6wi X-Received: by 2002:a81:260b:: with SMTP id m11-v6mr132282ywm.178.1537904894582; Tue, 25 Sep 2018 12:48:14 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:51 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-6-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 5/9] submodule.c: do not copy around submodule list From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP 'calculate_changed_submodule_paths' uses a local list to compute the changed submodules, and then produces the result by copying appropriate items into the result list. Instead use the result list directly and prune items afterwards using string_list_remove_empty_items. By doing so we'll have access to the util pointer for longer that contains the commits that we need to fetch, which will be useful in a later patch. Signed-off-by: Stefan Beller --- submodule.c | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/submodule.c b/submodule.c index 17103379ba4..dd478ed70bf 100644 --- a/submodule.c +++ b/submodule.c @@ -1128,8 +1128,7 @@ static void calculate_changed_submodule_paths( struct submodule_parallel_fetch *spf) { struct argv_array argv = ARGV_ARRAY_INIT; - struct string_list changed_submodules = STRING_LIST_INIT_DUP; - const struct string_list_item *name; + struct string_list_item *name; /* No need to check if there are no submodules configured */ if (!submodule_from_path(the_repository, NULL, NULL)) @@ -1146,9 +1145,9 @@ static void calculate_changed_submodule_paths( * Collect all submodules (whether checked out or not) for which new * commits have been recorded upstream in "changed_submodule_names". */ - collect_changed_submodules(&changed_submodules, &argv); + collect_changed_submodules(&spf->changed_submodule_names, &argv); - for_each_string_list_item(name, &changed_submodules) { + for_each_string_list_item(name, &spf->changed_submodule_names) { struct oid_array *commits = name->util; const struct submodule *submodule; const char *path = NULL; @@ -1162,12 +1161,14 @@ static void calculate_changed_submodule_paths( if (!path) continue; - if (!submodule_has_commits(path, commits)) - string_list_append(&spf->changed_submodule_names, - name->string); + if (submodule_has_commits(path, commits)) { + oid_array_clear(commits); + *name->string = '\0'; + } } - free_submodules_oids(&changed_submodules); + string_list_remove_empty_items(&spf->changed_submodule_names, 1); + argv_array_clear(&argv); oid_array_clear(&ref_tips_before_fetch); oid_array_clear(&ref_tips_after_fetch); @@ -1363,7 +1364,7 @@ int fetch_populated_submodules(struct repository *r, argv_array_clear(&spf.args); out: - string_list_clear(&spf.changed_submodule_names, 1); + free_submodules_oids(&spf.changed_submodule_names); return spf.result; } From patchwork Tue Sep 25 19:47:52 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614763 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 543FD15E8 for ; Tue, 25 Sep 2018 19:48:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 45DD32ABB8 for ; Tue, 25 Sep 2018 19:48:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3A7402ABBA; Tue, 25 Sep 2018 19:48:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 984E62ABD3 for ; Tue, 25 Sep 2018 19:48:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728101AbeIZB50 (ORCPT ); Tue, 25 Sep 2018 21:57:26 -0400 Received: from mail-ot1-f74.google.com ([209.85.210.74]:52317 "EHLO mail-ot1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB50 (ORCPT ); Tue, 25 Sep 2018 21:57:26 -0400 Received: by mail-ot1-f74.google.com with SMTP id a12-v6so3557328otl.19 for ; Tue, 25 Sep 2018 12:48:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=FOJIYkOnYkPNIGjmy6xUMk0e00HZIIc94locFSkFtUM=; b=cml4ZJOaikhGDKyJOFBZdnLwGaF46U6Ajf5+7A+uBAOPPlEwgkEVYPmFMnkjQ+XwmG hWjivlB3fY2r+9Kwij2cPArjoIhQuC0h4URSEX90ljAEVJ/We+zoWXKwhlXwtoAG2j9c awGINcV4yfGOlDOMVcg7I0XTgelmRF+feyrf+KqFyRQ4auw0jDQbdXg+XGMjig4EVOW6 GYrDK0i6aHIf2ot183xBdeXFx90IVsNNr+/5VR0YIXEZchvcidY5N6RxUd6tlCVJ9e4K uyGzFr2bYfzpNUHeha22FFjWBeMdpdtDk714/NByKCm6AN9RqZfzZ7bj7MO792n29bln Krlw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=FOJIYkOnYkPNIGjmy6xUMk0e00HZIIc94locFSkFtUM=; b=CuxA0W10YSs6w924ZUAVVhvphKlXm3Z/Ib91Y94M1PBF/6A31uuetNW3R7F88gQxr/ DEik4IIQ2qU3lTTVmVg2xD6EcZekJn9aNI+MJpNZC1tvuDWPH9sj0kSgs9L+8TRZ4xul l5U0d9IDySgj5ZucUJoh1R/fF9y5ShdR3H+6hCRba0aR5tWQgJjaiywG45I3BMRAPKdR dk7VwmH4l/30una01O8Me8gfCbJWM9UABlWs38X33TPBrpefGnRofNRWyji2o6FgsVsy hsP9yHfXzW7aOVetMljVWLwdBs8YVoBDKyldsm8JTamSulefn0nvajzOuTj1gIkdqlJe 5SSA== X-Gm-Message-State: ABuFfoirEP+/Ads80AjDI0yjS6sst+nh4atWv5VtDbQCKyWDCJCQ+2+I cJkh2uSeWKTBhDNd1p1dHelPjsbHjdPr/fZF8wnVnLWZ0F6+87LegxFF8URr6hVWgAT0RIGWWm7 kcQG2ELfaqXTDnYDtsEIP8L6IwWyx/j7ArT5hpXYQEnGV6z330T50N/Wg5WpY X-Google-Smtp-Source: ACcGV62Gr7QOThLWw55dRatx+ylm3gNrxnStusidsJ4sK1yxiXQHEJD+nmh6jwD3cybAnCnEeylk1GhAOO/P X-Received: by 2002:aca:ded5:: with SMTP id v204-v6mr2890433oig.7.1537904897001; Tue, 25 Sep 2018 12:48:17 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:52 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-7-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 6/9] repository: repo_submodule_init to take a submodule struct From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP When constructing a struct repository for a submodule for some revision of the superproject where the submodule is not contained in the index, it may not be present in the working tree currently either. In that siutation giving a 'path' argument is not useful. Upgrade the repo_submodule_init function to take a struct submodule instead. While we are at it, overhaul the repo_submodule_init function by renaming the submodule repository struct, which is to be initialized, to a name that is not confused with the struct submodule as easily. Also move its documentation into the header file. Signed-off-by: Stefan Beller --- builtin/grep.c | 17 ++++++++++------- builtin/ls-files.c | 12 +++++++----- builtin/submodule--helper.c | 2 +- repository.c | 27 ++++++++++----------------- repository.h | 11 +++++++++-- 5 files changed, 37 insertions(+), 32 deletions(-) diff --git a/builtin/grep.c b/builtin/grep.c index 601f801158f..81c53c862b1 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -418,16 +418,19 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, const struct object_id *oid, const char *filename, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); + int hit; if (!is_submodule_active(superproject, path)) return 0; - if (repo_submodule_init(&submodule, superproject, path)) + if (repo_submodule_init(&subrepo, superproject, sub)) return 0; - repo_read_gitmodules(&submodule); + repo_read_gitmodules(&subrepo); /* * NEEDSWORK: This adds the submodule's object directory to the list of @@ -440,7 +443,7 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, * object. */ grep_read_lock(); - add_to_alternates_memory(submodule.objects->objectdir); + add_to_alternates_memory(subrepo.objects->objectdir); grep_read_unlock(); if (oid) { @@ -465,14 +468,14 @@ static int grep_submodule(struct grep_opt *opt, struct repository *superproject, init_tree_desc(&tree, data, size); hit = grep_tree(opt, pathspec, &tree, &base, base.len, - object->type == OBJ_COMMIT, &submodule); + object->type == OBJ_COMMIT, &subrepo); strbuf_release(&base); free(data); } else { - hit = grep_cache(opt, &submodule, pathspec, 1); + hit = grep_cache(opt, &subrepo, pathspec, 1); } - repo_clear(&submodule); + repo_clear(&subrepo); return hit; } diff --git a/builtin/ls-files.c b/builtin/ls-files.c index 7f9919a3623..4d1649c1b3a 100644 --- a/builtin/ls-files.c +++ b/builtin/ls-files.c @@ -206,17 +206,19 @@ static void show_files(struct repository *repo, struct dir_struct *dir); static void show_submodule(struct repository *superproject, struct dir_struct *dir, const char *path) { - struct repository submodule; + struct repository subrepo; + const struct submodule *sub = submodule_from_path(superproject, + &null_oid, path); - if (repo_submodule_init(&submodule, superproject, path)) + if (repo_submodule_init(&subrepo, superproject, sub)) return; - if (repo_read_index(&submodule) < 0) + if (repo_read_index(&subrepo) < 0) die("index file corrupt"); - show_files(&submodule, dir); + show_files(&subrepo, dir); - repo_clear(&submodule); + repo_clear(&subrepo); } static void show_ce(struct repository *repo, struct dir_struct *dir, diff --git a/builtin/submodule--helper.c b/builtin/submodule--helper.c index 40844870cfb..1164e0979bd 100644 --- a/builtin/submodule--helper.c +++ b/builtin/submodule--helper.c @@ -2037,7 +2037,7 @@ static int ensure_core_worktree(int argc, const char **argv, const char *prefix) if (!sub) BUG("We could get the submodule handle before?"); - if (repo_submodule_init(&subrepo, the_repository, path)) + if (repo_submodule_init(&subrepo, the_repository, sub)) die(_("could not get a repository handle for submodule '%s'"), path); if (!repo_config_get_string(&subrepo, "core.worktree", &cw)) { diff --git a/repository.c b/repository.c index 5dd14867181..aabe64ee5d9 100644 --- a/repository.c +++ b/repository.c @@ -166,30 +166,23 @@ int repo_init(struct repository *repo, return -1; } -/* - * Initialize 'submodule' as the submodule given by 'path' in parent repository - * 'superproject'. - * Return 0 upon success and a non-zero value upon failure. - */ -int repo_submodule_init(struct repository *submodule, +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path) + const struct submodule *sub) { - const struct submodule *sub; struct strbuf gitdir = STRBUF_INIT; struct strbuf worktree = STRBUF_INIT; int ret = 0; - sub = submodule_from_path(superproject, &null_oid, path); if (!sub) { ret = -1; goto out; } - strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", path); - strbuf_repo_worktree_path(&worktree, superproject, "%s", path); + strbuf_repo_worktree_path(&gitdir, superproject, "%s/.git", sub->path); + strbuf_repo_worktree_path(&worktree, superproject, "%s", sub->path); - if (repo_init(submodule, gitdir.buf, worktree.buf)) { + if (repo_init(subrepo, gitdir.buf, worktree.buf)) { /* * If initilization fails then it may be due to the submodule * not being populated in the superproject's worktree. Instead @@ -201,16 +194,16 @@ int repo_submodule_init(struct repository *submodule, strbuf_repo_git_path(&gitdir, superproject, "modules/%s", sub->name); - if (repo_init(submodule, gitdir.buf, NULL)) { + if (repo_init(subrepo, gitdir.buf, NULL)) { ret = -1; goto out; } } - submodule->submodule_prefix = xstrfmt("%s%s/", - superproject->submodule_prefix ? - superproject->submodule_prefix : - "", path); + subrepo->submodule_prefix = xstrfmt("%s%s/", + superproject->submodule_prefix ? + superproject->submodule_prefix : + "", sub->path); out: strbuf_release(&gitdir); diff --git a/repository.h b/repository.h index 9f16c42c1ed..a9c7a5baa54 100644 --- a/repository.h +++ b/repository.h @@ -116,9 +116,16 @@ void repo_set_worktree(struct repository *repo, const char *path); void repo_set_hash_algo(struct repository *repo, int algo); void initialize_the_repository(void); int repo_init(struct repository *r, const char *gitdir, const char *worktree); -int repo_submodule_init(struct repository *submodule, + +/* + * Initialize the repository 'subrepo' as the submodule given by the + * struct submodule 'sub' in parent repository 'superproject'. + * Return 0 upon success and a non-zero value upon failure. + */ +struct submodule; +int repo_submodule_init(struct repository *subrepo, struct repository *superproject, - const char *path); + const struct submodule *sub); void repo_clear(struct repository *repo); /* From patchwork Tue Sep 25 19:47:53 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614765 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 90F8815E8 for ; Tue, 25 Sep 2018 19:48:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 826432ABB8 for ; Tue, 25 Sep 2018 19:48:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 76BD62ABC8; Tue, 25 Sep 2018 19:48:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F256D2ABB8 for ; Tue, 25 Sep 2018 19:48:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728138AbeIZB52 (ORCPT ); Tue, 25 Sep 2018 21:57:28 -0400 Received: from mail-qk1-f201.google.com ([209.85.222.201]:44892 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB52 (ORCPT ); Tue, 25 Sep 2018 21:57:28 -0400 Received: by mail-qk1-f201.google.com with SMTP id w126-v6so26910001qka.11 for ; Tue, 25 Sep 2018 12:48:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=CYGPtAr3eVWOKxYZzJk/YrFgQ87n0oDXdwFbMavaH7s=; b=IZIozHXp9imK/40nGiegCUFBN47hnq/yTlOoxSpPRmYdTG2pN9UwPFhldxZE+f4bWa X8MlILZtLxYUbOJSltcxTakS/Rro/NlFly/XgQjMpQVqMM7i6+jjceZfnWL7O0l+rLfp j5HBOa3fU8mBiLJlJ8IOF5GFoDXf8nJHu32Hp/KxxebxhxLgBAkUPSksLafBB13Zzdur kwCzreHPrAmlSe7ut/znpfkfSN4B7xCOwdswuOFr3FmfdSyPhytgQHD0oUz0m4Lrrzds SuJiSFWCFUr9JiA/CK5oZgUpt1DjyavkpWoAl1Hv8CzQrOdiKUnbHyiPLeMRunKsBLjV x42g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=CYGPtAr3eVWOKxYZzJk/YrFgQ87n0oDXdwFbMavaH7s=; b=L/cM7j/XlPy+RWpyFJYptbvQU62BcG4LZyVSjkqn00v9bkkoJMeUwuGKVWAMFw62UV OzL8MGQXLEaIjmJEUbQPS1bGtwBZS9F+ezRp49urhXinJmMePhOCvPCCuX6JSe10Cy0S Bsro4UPMb3nV8sc3fC0FGtUNWbvrVnA0xpmH8mK9bjpTb3TLqsmXbdixPulmWdAVqqZj oImSh5g5Z2Ux72H8DRmkXjFsQW77JHhF/sTMJxjUuAm9PhJsHHEwuxtvY4QTEVAF/W7g PY2ktVMYm024xFW9Z/wOV2or0aUdpjvYBQqUfyCEyFFHsklzcGb+zIKF46D13Cnvn2R5 ncKQ== X-Gm-Message-State: ABuFfojmNN8ELSWYXoC7mupxiZdpqpplBd5tWdq772hVMC5O5YvwUKmc FcKc1GDIcCYBM8ehNL//U2T06QF5BARUAaTeeKBgMV54QY1u3EM25t+90GW7VLInn0Abqr9fkok XQqiWGHMSwmsMzeQsWoHoGAyJvZ0+cXcATc4Yf8Hk/q8fd9HxOeCn6XtAumg+ X-Google-Smtp-Source: ACcGV61VrF71ZnCgnIbo4lZtJ+WUwjO32jzmx/ztKRVUWWlbcprKZbyD2sXjfd1m35bENEP+LcozuS5r5ojB X-Received: by 2002:aed:3665:: with SMTP id e92-v6mr838301qtb.41.1537904899341; Tue, 25 Sep 2018 12:48:19 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:53 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-8-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 7/9] submodule: fetch in submodules git directory instead of in worktree From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This patch started as a refactoring to make 'get_next_submodule' more readable, but upon doing so, I realized that "git fetch" of the submodule actually doesn't need to be run in the submodules worktree. So let's run it in its git dir instead. That should pave the way towards fetching submodules that are currently not checked out. This patch leaks the cp->dir in get_next_submodule, as any further callback in run_processes_parallel doesn't have access to the child process any more. In an early iteration of this patch, the function get_submodule_repo_for directly returned the string containing the git directory, which would be a better design choice for this patch. However the next patch both fixes the memory leak of cp->dir and also has a use case for using the full repository handle of the submodule, so it makes sense to introduce the get_submodule_repo_for here already. Signed-off-by: Stefan Beller --- submodule.c | 51 +++++++++++++++++++++++++++---------- t/t5526-fetch-submodules.sh | 7 ++++- 2 files changed, 44 insertions(+), 14 deletions(-) diff --git a/submodule.c b/submodule.c index dd478ed70bf..3f791f22771 100644 --- a/submodule.c +++ b/submodule.c @@ -481,6 +481,12 @@ void prepare_submodule_repo_env(struct argv_array *out) DEFAULT_GIT_DIR_ENVIRONMENT); } +static void prepare_submodule_repo_env_in_gitdir(struct argv_array *out) +{ + prepare_submodule_repo_env_no_git_dir(out); + argv_array_pushf(out, "%s=.", GIT_DIR_ENVIRONMENT); +} + /* Helper function to display the submodule header line prior to the full * summary output. If it can locate the submodule objects directory it will * attempt to lookup both the left and right commits and put them into the @@ -1227,6 +1233,29 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +static struct repository *get_submodule_repo_for(struct repository *r, + const struct submodule *sub) +{ + struct repository *ret = xmalloc(sizeof(*ret)); + + if (repo_submodule_init(ret, r, sub)) { + /* + * No entry in .gitmodules? Technically not a submodule, + * but historically we supported repositories that happen to be + * in-place where a gitlink is. Keep supporting them. + */ + struct strbuf gitdir = STRBUF_INIT; + strbuf_repo_worktree_path(&gitdir, r, "%s/.git", sub->path); + if (repo_init(ret, gitdir.buf, NULL)) { + strbuf_release(&gitdir); + return NULL; + } + strbuf_release(&gitdir); + } + + return ret; +} + static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { @@ -1234,12 +1263,11 @@ static int get_next_submodule(struct child_process *cp, struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_path = STRBUF_INIT; - struct strbuf submodule_git_dir = STRBUF_INIT; struct strbuf submodule_prefix = STRBUF_INIT; const struct cache_entry *ce = spf->r->index->cache[spf->count]; - const char *git_dir, *default_argv; + const char *default_argv; const struct submodule *submodule; + struct repository *repo; struct submodule default_submodule = SUBMODULE_INIT; if (!S_ISGITLINK(ce->ce_mode)) @@ -1274,16 +1302,12 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_repo_worktree_path(&submodule_path, spf->r, "%s", ce->name); - strbuf_addf(&submodule_git_dir, "%s/.git", submodule_path.buf); strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - git_dir = read_gitfile(submodule_git_dir.buf); - if (!git_dir) - git_dir = submodule_git_dir.buf; - if (is_directory(git_dir)) { + repo = get_submodule_repo_for(spf->r, submodule); + if (repo) { child_process_init(cp); - cp->dir = strbuf_detach(&submodule_path, NULL); - prepare_submodule_repo_env(&cp->env_array); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); + cp->dir = xstrdup(repo->gitdir); cp->git_cmd = 1; if (!spf->quiet) strbuf_addf(err, "Fetching submodule %s%s\n", @@ -1293,10 +1317,11 @@ static int get_next_submodule(struct child_process *cp, argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); argv_array_push(&cp->args, submodule_prefix.buf); + + repo_clear(repo); + free(repo); ret = 1; } - strbuf_release(&submodule_path); - strbuf_release(&submodule_git_dir); strbuf_release(&submodule_prefix); if (ret) { spf->count++; diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index 6c2f9b2ba26..42692219a1a 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -566,7 +566,12 @@ test_expect_success 'fetching submodule into a broken repository' ' test_must_fail git -C dst status && test_must_fail git -C dst diff && - test_must_fail git -C dst fetch --recurse-submodules + + # git-fetch cannot find the git directory of the submodule, + # so it will do nothing, successfully, as it cannot distinguish between + # this broken submodule and a submodule that was just set active but + # not cloned yet + git -C dst fetch --recurse-submodules ' test_expect_success "fetch new commits when submodule got renamed" ' From patchwork Tue Sep 25 19:47:54 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614767 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 27408174A for ; Tue, 25 Sep 2018 19:48:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 167B52ABB8 for ; Tue, 25 Sep 2018 19:48:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0AD922ABC8; Tue, 25 Sep 2018 19:48:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 17DA02ABB8 for ; Tue, 25 Sep 2018 19:48:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728161AbeIZB5a (ORCPT ); Tue, 25 Sep 2018 21:57:30 -0400 Received: from mail-pf1-f202.google.com ([209.85.210.202]:38543 "EHLO mail-pf1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB5a (ORCPT ); Tue, 25 Sep 2018 21:57:30 -0400 Received: by mail-pf1-f202.google.com with SMTP id e15-v6so13167816pfi.5 for ; Tue, 25 Sep 2018 12:48:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Wza86QGRfJjlfu3C65NECDr3K9wHp8NXevfN8ppXQZk=; b=bUrDQZCrOPMxKoh8J4ACjhzmQMIn+iJHt7oWLlErEIsbVySpoHIs6t3f8I7IjxTLzj iByHsNKYvvdCg6xnD0oeIGNm7pzGpmAxVZUbUuK5UXnXGG0vKg/7BbawGvobhX6e2PjN /8FxXK0q6q9sCCbtX7SJc5DZsMa2a7P+uGsz5yc6cQciZSPuWA9l7tBVUFDbmtMhcQuT 8WSo5b2l751ikV9guebAQGAqHeJV2VjOsxK3kXZ5I9UpFhN8g06naxbUhzz2h4+HtxWD NlDE0zQXvYIJcl/C9RNVFgRl3FzmGGgcuV8SHXCSuFyaG65y3Ptigi05kxGmJyzG8uO1 UqOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Wza86QGRfJjlfu3C65NECDr3K9wHp8NXevfN8ppXQZk=; b=JjnP3nPGQ1wTYqWh9Hii75A6LREZ7XWJfn8ZaaiaPs/jd6Z3M8fObVcXyOUXFkRUWv ZqNLteqejzmZ0dUSpWW5THDGDNmkUS65t6IOahqqPheH/kwa8nbJg+Fro9Pnfi9MKJ9O RsEkl/jpkCLmo3bJlMJbJexMDD8RrKR52UpYfAgYgI8oYcSX5MmRF6cIfYGoGt4Brm5o YClPHWKdMhKBsKlJ9hd7nDrkAvl97i+YqbuERiqFhh07eVFZdsJBJ1Tg5WYU65tWhDFR gXd6dIdyWSOjhqDleyWobRm3q714Em1uxjGoa0GVwcfK8u/flnaSdhv6I41faQW9nKfY lb8Q== X-Gm-Message-State: ABuFfojkDNyfltg4wGsGjjFOceSvgLI2F9E+3AD3SEOwFyom7TDFAu2H n2DH5OZRnZXhaLNC+KO3Xi7xZHKBQFg0IEBkxTeWah7nfahHyN2jNPJRIAj93tRjzsnkyj2XAwN +2xtkAcEBsWJ6hyBVhNKrFb2N6KNG4a0CAVG1A7VxmPjuJTe48iPvOmur5YMH X-Google-Smtp-Source: ACcGV60HnhaRn1BPc55X8TCI7c+JKsKxQJ69i+4wVDM7QtN5179p8We55rrl7Svoo2wIjq1YBFVme4looJAF X-Received: by 2002:a63:354d:: with SMTP id c74-v6mr96031pga.2.1537904901434; Tue, 25 Sep 2018 12:48:21 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:54 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-9-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 8/9] fetch: retry fetching submodules if needed objects were not fetched From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently when git-fetch is asked to recurse into submodules, it dispatches a plain "git-fetch -C " (with some submodule related options such as prefix and recusing strategy, but) without any information of the remote or the tip that should be fetched. This works surprisingly well in some workflows (such as using submodules as a third party library), while not so well in other scenarios, such as in a Gerrit topic-based workflow, that can tie together changes (potentially across repositories) on the server side. One of the parts of such a Gerrit workflow is to download a change when wanting to examine it, and you'd want to have its submodule changes that are in the same topic downloaded as well. However these submodule changes reside in their own repository in their own ref (refs/changes/). Retry fetching a submodule by object name if the object id that the superproject points to, cannot be found. This retrying does not happen when the "git fetch" done at the superproject is not storing the fetched results in remote tracking branches (i.e. instead just recording them to FETCH_HEAD) in this step. A later patch will fix this. builtin/fetch used to only inspect submodules when they were fetched "on-demand", as in either on/off case it was clear whether the submodule needs to be fetched. However to know whether we need to try fetching the object ids, we need to identify the object names, which is done in this function check_for_new_submodule_commits(), so we'll also run that code in case the submodule recursion is set to "on". Signed-off-by: Stefan Beller --- builtin/fetch.c | 9 +- submodule.c | 185 ++++++++++++++++++++++++++++++------ t/t5526-fetch-submodules.sh | 16 ++++ 3 files changed, 177 insertions(+), 33 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 0696abfc2a1..e3b03ad3bd3 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -707,8 +707,7 @@ static int update_local_ref(struct ref *ref, what = _("[new ref]"); } - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref(msg, ref, 0); format_display(display, r ? '!' : '*', what, @@ -723,8 +722,7 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, ".."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("fast-forward", ref, 1); format_display(display, r ? '!' : ' ', quickref.buf, @@ -738,8 +736,7 @@ static int update_local_ref(struct ref *ref, strbuf_add_unique_abbrev(&quickref, ¤t->object.oid, DEFAULT_ABBREV); strbuf_addstr(&quickref, "..."); strbuf_add_unique_abbrev(&quickref, &ref->new_oid, DEFAULT_ABBREV); - if ((recurse_submodules != RECURSE_SUBMODULES_OFF) && - (recurse_submodules != RECURSE_SUBMODULES_ON)) + if (recurse_submodules != RECURSE_SUBMODULES_OFF) check_for_new_submodule_commits(&ref->new_oid); r = s_update_ref("forced-update", ref, 1); format_display(display, r ? '!' : '+', quickref.buf, diff --git a/submodule.c b/submodule.c index 3f791f22771..05799362e05 100644 --- a/submodule.c +++ b/submodule.c @@ -1127,8 +1127,12 @@ struct submodule_parallel_fetch { int result; struct string_list changed_submodule_names; + struct get_next_submodule_task **retry; + int retry_nr, retry_alloc; }; -#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, STRING_LIST_INIT_DUP } +#define SPF_INIT {0, ARGV_ARRAY_INIT, NULL, NULL, 0, 0, 0, 0, \ + STRING_LIST_INIT_DUP, \ + NULL, 0, 0} static void calculate_changed_submodule_paths( struct submodule_parallel_fetch *spf) @@ -1233,6 +1237,56 @@ static int get_fetch_recurse_config(const struct submodule *submodule, return spf->default_option; } +struct get_next_submodule_task { + struct repository *repo; + const struct submodule *sub; + unsigned free_sub : 1; /* Do we need to free the submodule? */ + struct oid_array *commits; +}; + +static const struct submodule *get_default_submodule(const char *path) +{ + struct submodule *ret = NULL; + const char *name = default_name_or_path(path); + + if (!name) + return NULL; + + ret = xmalloc(sizeof(*ret)); + memset(ret, 0, sizeof(*ret)); + ret->path = name; + ret->name = name; + + return (const struct submodule *) ret; +} + +static struct get_next_submodule_task *get_next_submodule_task_create( + struct repository *r, const char *path) +{ + struct get_next_submodule_task *task = xmalloc(sizeof(*task)); + memset(task, 0, sizeof(*task)); + + task->sub = submodule_from_path(r, &null_oid, path); + if (!task->sub) { + task->sub = get_default_submodule(path); + task->free_sub = 1; + } + + return task; +} + +static void get_next_submodule_task_release(struct get_next_submodule_task *p) +{ + if (p->free_sub) + free((void*)p->sub); + p->free_sub = 0; + p->sub = NULL; + + if (p->repo) + repo_clear(p->repo); + FREE_AND_NULL(p->repo); +} + static struct repository *get_submodule_repo_for(struct repository *r, const struct submodule *sub) { @@ -1259,39 +1313,35 @@ static struct repository *get_submodule_repo_for(struct repository *r, static int get_next_submodule(struct child_process *cp, struct strbuf *err, void *data, void **task_cb) { - int ret = 0; struct submodule_parallel_fetch *spf = data; for (; spf->count < spf->r->index->cache_nr; spf->count++) { - struct strbuf submodule_prefix = STRBUF_INIT; + int recurse_config; const struct cache_entry *ce = spf->r->index->cache[spf->count]; const char *default_argv; - const struct submodule *submodule; - struct repository *repo; - struct submodule default_submodule = SUBMODULE_INIT; + struct get_next_submodule_task *task; if (!S_ISGITLINK(ce->ce_mode)) continue; - submodule = submodule_from_path(spf->r, &null_oid, ce->name); - if (!submodule) { - const char *name = default_name_or_path(ce->name); - if (name) { - default_submodule.path = name; - default_submodule.name = name; - submodule = &default_submodule; - } + task = get_next_submodule_task_create(spf->r, ce->name); + + if (!task->sub) { + free(task); + continue; } - switch (get_fetch_recurse_config(submodule, spf)) + recurse_config = get_fetch_recurse_config(task->sub, spf); + + switch (recurse_config) { default: case RECURSE_SUBMODULES_DEFAULT: case RECURSE_SUBMODULES_ON_DEMAND: - if (!submodule || + if (!task->sub || !string_list_lookup( &spf->changed_submodule_names, - submodule->name)) + task->sub->name)) continue; default_argv = "on-demand"; break; @@ -1302,12 +1352,12 @@ static int get_next_submodule(struct child_process *cp, continue; } - strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, ce->name); - repo = get_submodule_repo_for(spf->r, submodule); - if (repo) { + task->repo = get_submodule_repo_for(spf->r, task->sub); + if (task->repo) { + struct strbuf submodule_prefix = STRBUF_INIT; child_process_init(cp); prepare_submodule_repo_env_in_gitdir(&cp->env_array); - cp->dir = xstrdup(repo->gitdir); + cp->dir = task->repo->gitdir; cp->git_cmd = 1; if (!spf->quiet) strbuf_addf(err, "Fetching submodule %s%s\n", @@ -1316,18 +1366,51 @@ static int get_next_submodule(struct child_process *cp, argv_array_pushv(&cp->args, spf->args.argv); argv_array_push(&cp->args, default_argv); argv_array_push(&cp->args, "--submodule-prefix"); + + strbuf_addf(&submodule_prefix, "%s%s/", + spf->prefix, + task->sub->path); argv_array_push(&cp->args, submodule_prefix.buf); - repo_clear(repo); - free(repo); - ret = 1; - } - strbuf_release(&submodule_prefix); - if (ret) { spf->count++; + *task_cb = task; + + strbuf_release(&submodule_prefix); return 1; + } else { + get_next_submodule_task_release(task); + free(task); } } + + if (spf->retry_nr) { + struct get_next_submodule_task *task = spf->retry[spf->retry_nr - 1]; + struct strbuf submodule_prefix = STRBUF_INIT; + spf->retry_nr--; + + strbuf_addf(&submodule_prefix, "%s%s/", spf->prefix, task->sub->path); + + child_process_init(cp); + prepare_submodule_repo_env_in_gitdir(&cp->env_array); + cp->git_cmd = 1; + cp->dir = task->repo->gitdir; + + argv_array_init(&cp->args); + argv_array_pushv(&cp->args, spf->args.argv); + argv_array_push(&cp->args, "on-demand"); + argv_array_push(&cp->args, "--submodule-prefix"); + argv_array_push(&cp->args, submodule_prefix.buf); + + /* NEEDSWORK: have get_default_remote from s--h */ + argv_array_push(&cp->args, "origin"); + oid_array_for_each_unique(task->commits, + append_oid_to_argv, &cp->args); + + *task_cb = task; + strbuf_release(&submodule_prefix); + return 1; + } + return 0; } @@ -1335,20 +1418,68 @@ static int fetch_start_failure(struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct get_next_submodule_task *task = task_cb; spf->result = 1; + get_next_submodule_task_release(task); return 0; } +static int commit_exists_in_sub(const struct object_id *oid, void *data) +{ + struct repository *subrepo = data; + + enum object_type type = oid_object_info(subrepo, oid, NULL); + + return type != OBJ_COMMIT; +} + static int fetch_finish(int retvalue, struct strbuf *err, void *cb, void *task_cb) { struct submodule_parallel_fetch *spf = cb; + struct get_next_submodule_task *task = task_cb; + const struct submodule *sub; + + struct string_list_item *it; + struct oid_array *commits; if (retvalue) spf->result = 1; + if (!task) + return 0; + + sub = task->sub; + if (!sub) + goto out; + + it = string_list_lookup(&spf->changed_submodule_names, sub->name); + if (!it) + goto out; + + commits = it->util; + oid_array_filter(commits, + commit_exists_in_sub, + task->repo); + + /* Are there commits that do not exist? */ + if (commits->nr) { + /* We already tried fetching them, do not try again. */ + if (task->commits) + return 0; + + task->commits = commits; + ALLOC_GROW(spf->retry, spf->retry_nr + 1, spf->retry_alloc); + spf->retry[spf->retry_nr] = task; + spf->retry_nr++; + return 0; + } + +out: + get_next_submodule_task_release(task); + return 0; } diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index 42692219a1a..af12c50e7dd 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -605,4 +605,20 @@ test_expect_success "fetch new commits when submodule got renamed" ' test_cmp expect actual ' +test_expect_success "fetch new commits on-demand when they are not reachable" ' + git checkout --detach && + C=$(git -C submodule commit-tree -m "new change outside refs/heads" HEAD^{tree}) && + git -C submodule update-ref refs/changes/1 $C && + git update-index --cacheinfo 160000 $C submodule && + git commit -m "updated submodule outside of refs/heads" && + D=$(git rev-parse HEAD) && + git update-ref refs/changes/2 $D && + ( + cd downstream && + git fetch --recurse-submodules --recurse-submodules-default on-demand origin refs/changes/2:refs/heads/my_branch && + git -C submodule cat-file -t $C && + git checkout --recurse-submodules FETCH_HEAD + ) +' + test_done From patchwork Tue Sep 25 19:47:55 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Stefan Beller X-Patchwork-Id: 10614769 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5ACD5174A for ; Tue, 25 Sep 2018 19:48:27 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 49C6F2ABB8 for ; Tue, 25 Sep 2018 19:48:27 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3E1B32ABCF; Tue, 25 Sep 2018 19:48:27 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C11BD2ABB8 for ; Tue, 25 Sep 2018 19:48:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728178AbeIZB5d (ORCPT ); Tue, 25 Sep 2018 21:57:33 -0400 Received: from mail-qt1-f201.google.com ([209.85.160.201]:45107 "EHLO mail-qt1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726878AbeIZB5d (ORCPT ); Tue, 25 Sep 2018 21:57:33 -0400 Received: by mail-qt1-f201.google.com with SMTP id t17-v6so9471508qtq.12 for ; Tue, 25 Sep 2018 12:48:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=yXjkJf91n9Rs3zQy4wj5xsWoG/1W5BjZgk3IKxvBYCs=; b=f4WxLKLsRgLKj4bLg6cL5C+A+6h94EAnhBfQVuZgu0G17O6JyNDQn+HMX6kRyGiVXp dSllwuYSPxqZ+Y8950Pd6mWtotxhdGU3e8iWPQI+HUO1jRnBy8jkJMR6f9Pm76zYWp4m pvjmG5IdrB8+SyCUUjyiJWMnFAHeQ7sN0/y8KRnYBnMjhLeill2fr5mj77s9qENyrvrs 85o82o2tdMoJy5vISBPcr3qjkTUwjt//Uja7zYkNxcGwRMWFUtKiknwgalG3vbzEv8xK 677s3pplDZNzbfn4SXq1DNAO0vTAYv4zfJBBha98+shC2tseMQ/ULV8aw7D5PjCagAS+ WADg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yXjkJf91n9Rs3zQy4wj5xsWoG/1W5BjZgk3IKxvBYCs=; b=gFPWZCWiGwPIlBS3rrWI/RdXDXsZdMYE9nIIjE3lSrYROCoyfWknGYlI7NJIJiQoO2 ELg3loHRMzvcHGbgHtfWAAKQhq8ewqHugO2j3RH/6Ns0Ni2UTD2AW67WkfDQY/WgIkcc /paW/xNDC3nV087Hkhk7S/IfQkpyGW9RH/JYZeUNb/yPJZ1VYpMBH5wsuekvgbyeh7Bq jwdKHRUhAe3V628u3HUhSylNaBHdLNNgL+8x8/Krsg6t5SL8w4pWMKW6aEbVWsJydqcf UUwhRbvOGqdSoNmi6ic9HoW+w0CqArwBXrTjb8vpCLYHEMNXPjzJRedjbeuA6WwCZJ2I zSuA== X-Gm-Message-State: ABuFfogtyLOThSlOyIS9g3kB7NYd9kMAuK1RUCaqwDOBznLEfQ5EUntf XcCFBjSctrfEHxuOKPE6tvl2XUm2TkK8WjaiHRjjgyi81zDm72rM42ldE/fgkfviaHkD6vJ39+8 /eVWnZpsZ6OOVSbisiNBa2skZShRLyekj6JLptpkTZCOKsVwOSygu2uh1uy7l X-Google-Smtp-Source: ACcGV61guH+wdAZlmLb6d+MyZyWRlkDL4tL/G5IparfPjorlY935OSQ7lkaOHk8AamlHFndyrgVWe13lvPTH X-Received: by 2002:a0c:fa83:: with SMTP id o3-v6mr935797qvn.2.1537904904320; Tue, 25 Sep 2018 12:48:24 -0700 (PDT) Date: Tue, 25 Sep 2018 12:47:55 -0700 In-Reply-To: <20180925194755.105578-1-sbeller@google.com> Message-Id: <20180925194755.105578-10-sbeller@google.com> Mime-Version: 1.0 References: <20180925194755.105578-1-sbeller@google.com> X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v4 9/9] builtin/fetch: check for submodule updates for non branch fetches From: Stefan Beller To: git@vger.kernel.org Cc: Stefan Beller Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Gerrit, the code review tool, has a different workflow than our mailing list based approach. Usually users upload changes to a Gerrit server and continuous integration and testing happens by bots. Sometimes however a user wants to checkout a change locally and look at it locally. For this use case, Gerrit offers a command line snippet to copy and paste to your terminal, which looks like git fetch https:///gerrit refs/changes/ && git checkout FETCH_HEAD For Gerrit changes that contain changing submodule gitlinks, it would be easy to extend both the fetch and checkout with the '--recurse-submodules' flag, such that this command line snippet would produce the state of a change locally. However the functionality added in the previous patch, which would ensure that we fetch the objects in the submodule that the gitlink pointed at, only works for remote tracking branches so far, not for FETCH_HEAD. Make sure that fetching a superproject to its FETCH_HEAD, also respects the existence checks for objects in the submodule recursion. Signed-off-by: Stefan Beller --- builtin/fetch.c | 5 ++++- t/t5526-fetch-submodules.sh | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index e3b03ad3bd3..f2d9e548bf0 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -894,11 +894,14 @@ static int store_updated_refs(const char *raw_url, const char *remote_name, rc |= update_local_ref(ref, what, rm, ¬e, summary_width); free(ref); - } else + } else { + check_for_new_submodule_commits(&rm->old_oid); format_display(¬e, '*', *kind ? kind : "branch", NULL, *what ? what : "HEAD", "FETCH_HEAD", summary_width); + } + if (note.len) { if (verbosity >= 0 && !shown_url) { fprintf(stderr, _("From %.*s\n"), diff --git a/t/t5526-fetch-submodules.sh b/t/t5526-fetch-submodules.sh index af12c50e7dd..a509eabb044 100755 --- a/t/t5526-fetch-submodules.sh +++ b/t/t5526-fetch-submodules.sh @@ -615,7 +615,7 @@ test_expect_success "fetch new commits on-demand when they are not reachable" ' git update-ref refs/changes/2 $D && ( cd downstream && - git fetch --recurse-submodules --recurse-submodules-default on-demand origin refs/changes/2:refs/heads/my_branch && + git fetch --recurse-submodules origin refs/changes/2 && git -C submodule cat-file -t $C && git checkout --recurse-submodules FETCH_HEAD )