From patchwork Sun Jan 24 02:34:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12041995 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FEDFC433E0 for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0B9D822C7C for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726412AbhAXCfW (ORCPT ); Sat, 23 Jan 2021 21:35:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36276 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726288AbhAXCfS (ORCPT ); Sat, 23 Jan 2021 21:35:18 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B46EBC061786 for ; Sat, 23 Jan 2021 18:34:37 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id 127so10242060ybn.5 for ; Sat, 23 Jan 2021 18:34:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=qGrcsnbMq9etB26uWF+H0pfkFEtafEwsttDBXCGjtuo=; b=HGpAi26j3oHbwJbdgcq+pPgYQq86m4xDzqLpwIAtfthLthfWY6pHaWM0oSP4Fwqwcb cJsovzootVosAzl2lo3SmS1GuXdggPXwLiKIDDF4+OwCN2GDUuwkpZRkR4LtaO0lghwY sMJeOwpBADBUMuYGO4r6SU6XzqDe/U+PaeCXvMP10zaCFszD0r/vBEQmsUBm815ks7Tq DkuIsAWfXytoF7bc104m+sYVqiRMEkx2OmTLZEyc8Mp2bOYrl2I7lIa/sELdaZclYiID Krk7yo8U/C4SlQtZXR30dgoqNZsJx/Te+ECHkaNrGHx2lmK2Eut8JjxpvcNyDOrygTy+ zx7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qGrcsnbMq9etB26uWF+H0pfkFEtafEwsttDBXCGjtuo=; b=Cz/KO0YfU5eOrVDlU/bqEcUpEyUCUzjyjwEn/u43pa9A0SM2530c0nVSTQP+LlHZLO zBDP9cFaX+ZhVGhg4e8FhL0XHlhhQpBJEMmaOnNx8npUufYpJQJuDIiB+vXvW/bAU+3r 6BNpGYRFt6btYg5dSiz5ioMTZKoYN44qHlpp+dXsv3JMjuvPCK1LhFmfyCgpzbf1RdCZ yp1VlUwtX1816+SJQ/UbYK0GtaCM5Ll9Ed0hICtnTdz/brEcy12MXNbkKZ5aY/tQVBf1 aYx71DOw0dQeQcXGNbklWEKacl8xB0ka7icdqpvd0BFJgtIqqT1IUbbkTifeFWLDob8M Sufg== X-Gm-Message-State: AOAM530k8Q4x7SRNUEm/0MJtXmpXS88UikrofZq5JMsDYmWzAd1+livy PBi+H/XOxQMr6T3XQlbukiHISDSgHcb23RCfSp3Q+e6SB8Dub8l4TWNUuVXrC0er8t3h7jYGnXM FhOyFh1KRdmrDvU4SIxt8vRlC2fijJxvx8c/AZIraDDQ1xettWB0I0LeCOu9v1JYav25Z8NeHF+ i/ X-Google-Smtp-Source: ABdhPJy99+X0s7+g+gNgW3LbmWnnArVR7pSdVSTdU03B0u8WbeMp3TrAiucIjY2pQCXAQGKlTOxlT7bZ3iyLLtef0CVt Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a25:7b82:: with SMTP id w124mr15350851ybc.492.1611455676832; Sat, 23 Jan 2021 18:34:36 -0800 (PST) Date: Sat, 23 Jan 2021 18:34:27 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.280.ga3ce27912f-goog Subject: [PATCH 1/4] http: allow custom index-pack args From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Currently, when fetching, packfiles referenced by URIs are run through index-pack without any arguments other than --stdin and --keep, no matter what arguments are used for the packfile that is inline in the fetch response. As a preparation for ensuring that all packs (whether inline or not) use the same index-pack arguments, teach the http subsystem to allow custom index-pack arguments. http-fetch has been updated to use the new API. For now, it passes --keep alone instead of --keep with a process ID, but this is only temporary because http-fetch itself will be taught to accept index-pack parameters (instead of using a hardcoded constant) in a subsequent commit. Signed-off-by: Jonathan Tan --- http-fetch.c | 6 +++++- http.c | 15 ++++++++------- http.h | 10 +++++----- 3 files changed, 18 insertions(+), 13 deletions(-) diff --git a/http-fetch.c b/http-fetch.c index c4ccc5fea9..2d1d9d054f 100644 --- a/http-fetch.c +++ b/http-fetch.c @@ -43,6 +43,9 @@ static int fetch_using_walker(const char *raw_url, int get_verbosely, return rc; } +static const char *index_pack_args[] = + {"index-pack", "--stdin", "--keep", NULL}; + static void fetch_single_packfile(struct object_id *packfile_hash, const char *url) { struct http_pack_request *preq; @@ -55,7 +58,8 @@ static void fetch_single_packfile(struct object_id *packfile_hash, if (preq == NULL) die("couldn't create http pack request"); preq->slot->results = &results; - preq->generate_keep = 1; + preq->index_pack_args = index_pack_args; + preq->preserve_index_pack_stdout = 1; if (start_active_slot(preq->slot)) { run_active_slot(preq->slot); diff --git a/http.c b/http.c index 8b23a546af..f8ea28bb2e 100644 --- a/http.c +++ b/http.c @@ -2259,6 +2259,9 @@ void release_http_pack_request(struct http_pack_request *preq) free(preq); } +static const char *default_index_pack_args[] = + {"index-pack", "--stdin", NULL}; + int finish_http_pack_request(struct http_pack_request *preq) { struct child_process ip = CHILD_PROCESS_INIT; @@ -2270,17 +2273,15 @@ int finish_http_pack_request(struct http_pack_request *preq) tmpfile_fd = xopen(preq->tmpfile.buf, O_RDONLY); - strvec_push(&ip.args, "index-pack"); - strvec_push(&ip.args, "--stdin"); ip.git_cmd = 1; ip.in = tmpfile_fd; - if (preq->generate_keep) { - strvec_pushf(&ip.args, "--keep=git %"PRIuMAX, - (uintmax_t)getpid()); + ip.argv = preq->index_pack_args ? preq->index_pack_args + : default_index_pack_args; + + if (preq->preserve_index_pack_stdout) ip.out = 0; - } else { + else ip.no_stdout = 1; - } if (run_command(&ip)) { ret = -1; diff --git a/http.h b/http.h index 5de792ef3f..bf3d1270ad 100644 --- a/http.h +++ b/http.h @@ -218,12 +218,12 @@ struct http_pack_request { char *url; /* - * If this is true, finish_http_pack_request() will pass "--keep" to - * index-pack, resulting in the creation of a keep file, and will not - * suppress its stdout (that is, the "keep\t\n" line will be - * printed to stdout). + * index-pack command to run. Must be terminated by NULL. + * + * If NULL, defaults to {"index-pack", "--stdin", NULL}. */ - unsigned generate_keep : 1; + const char **index_pack_args; + unsigned preserve_index_pack_stdout : 1; FILE *packfile; struct strbuf tmpfile; From patchwork Sun Jan 24 02:34:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12041997 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 72681C433E9 for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 403B422B51 for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726433AbhAXCfX (ORCPT ); Sat, 23 Jan 2021 21:35:23 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726367AbhAXCfT (ORCPT ); Sat, 23 Jan 2021 21:35:19 -0500 Received: from mail-pg1-x54a.google.com (mail-pg1-x54a.google.com [IPv6:2607:f8b0:4864:20::54a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13B94C061788 for ; Sat, 23 Jan 2021 18:34:39 -0800 (PST) Received: by mail-pg1-x54a.google.com with SMTP id 26so5841980pgl.2 for ; Sat, 23 Jan 2021 18:34:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=RPJ3i+iJP0I/CGazHQ7sdb+EtTxh6LvmX8o5+0li5/E=; b=dChiFjrL8K0dZTHJCu4IO5ftiqLTp8XoTCdd2eaozvIdYP+OlHSpqpXbRFsMauqhVE KIgELECB60bumhJ2dqxAsq9+8bwt1/iGG/TuIKzlqIRzDw5uSi+pLlJp4KNH/8krgKZh /FgplscftupNPksTytujXImw17CSKdbY/zGywznZqKqtSFOd2YeDt8bmjdOwixLNLfLv oFE6Exkqj3BGBWE1SWlXMSxKno9UPUEHKOgp9lNc8jmqfBBQxwXEthaTpQ1yh7hSUli5 5tOdq9z7AVXImN5gT8OnwNOg9wBxgDAckOoK0wCESsW6jZyiUDGoInla8Wb+VsnVpl8W Iubg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=RPJ3i+iJP0I/CGazHQ7sdb+EtTxh6LvmX8o5+0li5/E=; b=sS0YeERrWAgjmUVFz7OGjz3m9bE6D4UA8ojeAOs9Q4YdFMLnu6D8MxlKiJy/eVjrPI 40My69Ftbj3yg77+AjsMUhU4yk8/9A6quFd87wviV/rIengn4uO7aRRRu55iNZyRy9Q9 clpMAbLk5APZ1w4cRuvX30v8u0qJaa5jGhF0uAcKlHgAtmDKvIMvt670E6b5VdDrpocu Imh6zHXjd2nqvRBwQQrUK/66LEPw/wqslHyV1CyENawDWznYoU+t1X7k5Hb/s7cAULfn RKuyLIFIjrdIA8MjzS6zMpTgYDjhck0deeoeveSHeFOoXI4fyv3RJQz+PCP32TqkvzkJ KYqQ== X-Gm-Message-State: AOAM533ETQVSVNI0KNgy9P/RExT0rWwyzi97kpag7tobewosVeRamFkH rTjJWAffyr3ZZkM6kx/X/D3hjQEPufhJLFTeakdNAx8FGnxSoiyj9RWiuc3CDsYiw1uYcR7RBQD T44m4nbl6ndNXPCuOArjc02rvjHDcBPslnCAsEY+c/IOl28kuosOnq09rxbkZAYkpHowNHfBBam u6 X-Google-Smtp-Source: ABdhPJxj48OgDVgMX/ytCcJDQMMlQQG7L2j4udnrMiexpnaS6yEpN10oLbCHjGn6XjMEwOpU4DVHY667WEJ1/ClVsUaM Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a62:cd49:0:b029:1b5:4e48:6f1a with SMTP id o70-20020a62cd490000b02901b54e486f1amr5461011pfg.14.1611455678370; Sat, 23 Jan 2021 18:34:38 -0800 (PST) Date: Sat, 23 Jan 2021 18:34:28 -0800 In-Reply-To: Message-Id: <3032117ee747e72f6208eab7f960737d25f8a82a.1611455251.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.280.ga3ce27912f-goog Subject: [PATCH 2/4] http-fetch: allow custom index-pack args From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org This is the next step in teaching fetch-pack to pass its index-pack arguments when processing packfiles referenced by URIs. The "--keep" in fetch-pack.c will be replaced with a full message in a subsequent commit. Signed-off-by: Jonathan Tan --- Documentation/git-http-fetch.txt | 9 ++++++-- fetch-pack.c | 1 + http-fetch.c | 35 +++++++++++++++++++++++++++----- t/t5550-http-fetch-dumb.sh | 3 ++- 4 files changed, 40 insertions(+), 8 deletions(-) diff --git a/Documentation/git-http-fetch.txt b/Documentation/git-http-fetch.txt index 4deb4893f5..aa171088e8 100644 --- a/Documentation/git-http-fetch.txt +++ b/Documentation/git-http-fetch.txt @@ -41,11 +41,16 @@ commit-id:: ['\t'] --packfile=:: - Instead of a commit id on the command line (which is not expected in + For internal use only. Instead of a commit id on the command line (which is not expected in this case), 'git http-fetch' fetches the packfile directly at the given URL and uses index-pack to generate corresponding .idx and .keep files. The hash is used to determine the name of the temporary file and is - arbitrary. The output of index-pack is printed to stdout. + arbitrary. The output of index-pack is printed to stdout. Requires + --index-pack-args. + +--index-pack-args=:: + For internal use only. The command to run on the contents of the + downloaded pack. Arguments are URL-encoded separated by spaces. --recover:: Verify that everything reachable from target is fetched. Used after diff --git a/fetch-pack.c b/fetch-pack.c index 876f90c759..274ae602f7 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1645,6 +1645,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, strvec_pushf(&cmd.args, "--packfile=%.*s", (int) the_hash_algo->hexsz, packfile_uris.items[i].string); + strvec_push(&cmd.args, "--index-pack-args=index-pack --stdin --keep"); strvec_push(&cmd.args, uri); cmd.git_cmd = 1; cmd.no_stdin = 1; diff --git a/http-fetch.c b/http-fetch.c index 2d1d9d054f..12feb84e71 100644 --- a/http-fetch.c +++ b/http-fetch.c @@ -3,6 +3,7 @@ #include "exec-cmd.h" #include "http.h" #include "walker.h" +#include "strvec.h" static const char http_fetch_usage[] = "git http-fetch " "[-c] [-t] [-a] [-v] [--recover] [-w ref] [--stdin | --packfile=hash | commit-id] url"; @@ -43,11 +44,9 @@ static int fetch_using_walker(const char *raw_url, int get_verbosely, return rc; } -static const char *index_pack_args[] = - {"index-pack", "--stdin", "--keep", NULL}; - static void fetch_single_packfile(struct object_id *packfile_hash, - const char *url) { + const char *url, + const char **index_pack_args) { struct http_pack_request *preq; struct slot_results results; int ret; @@ -90,6 +89,7 @@ int cmd_main(int argc, const char **argv) int packfile = 0; int nongit; struct object_id packfile_hash; + const char *index_pack_args = NULL; setup_git_directory_gently(&nongit); @@ -116,6 +116,8 @@ int cmd_main(int argc, const char **argv) packfile = 1; if (parse_oid_hex(p, &packfile_hash, &end) || *end) die(_("argument to --packfile must be a valid hash (got '%s')"), p); + } else if (skip_prefix(argv[arg], "--index-pack-args=", &p)) { + index_pack_args = p; } arg++; } @@ -128,10 +130,33 @@ int cmd_main(int argc, const char **argv) git_config(git_default_config, NULL); if (packfile) { - fetch_single_packfile(&packfile_hash, argv[arg]); + struct strvec encoded = STRVEC_INIT; + char **raw; + int i; + + if (!index_pack_args) + die(_("--packfile requires --index-pack-args")); + + strvec_split(&encoded, index_pack_args); + + CALLOC_ARRAY(raw, encoded.nr + 1); + for (i = 0; i < encoded.nr; i++) + raw[i] = url_percent_decode(encoded.v[i]); + + fetch_single_packfile(&packfile_hash, argv[arg], + (const char **) raw); + + for (i = 0; i < encoded.nr; i++) + free(raw[i]); + free(raw); + strvec_clear(&encoded); + return 0; } + if (index_pack_args) + die(_("--index-pack-args can only be used with --packfile")); + if (commits_on_stdin) { commits = walker_targets_stdin(&commit_id, &write_ref); } else { diff --git a/t/t5550-http-fetch-dumb.sh b/t/t5550-http-fetch-dumb.sh index 483578b2d7..af90e7efed 100755 --- a/t/t5550-http-fetch-dumb.sh +++ b/t/t5550-http-fetch-dumb.sh @@ -224,7 +224,8 @@ test_expect_success 'http-fetch --packfile' ' git init packfileclient && p=$(cd "$HTTPD_DOCUMENT_ROOT_PATH"/repo_pack.git && ls objects/pack/pack-*.pack) && - git -C packfileclient http-fetch --packfile=$ARBITRARY "$HTTPD_URL"/dumb/repo_pack.git/$p >out && + git -C packfileclient http-fetch --packfile=$ARBITRARY \ + --index-pack-args="index-pack --stdin --keep" "$HTTPD_URL"/dumb/repo_pack.git/$p >out && grep "^keep.[0-9a-f]\{16,\}$" out && cut -c6- out >packhash && From patchwork Sun Jan 24 02:34:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12041999 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E0A3C433E6 for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2280222CA0 for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726440AbhAXCfY (ORCPT ); Sat, 23 Jan 2021 21:35:24 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36288 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726398AbhAXCfV (ORCPT ); Sat, 23 Jan 2021 21:35:21 -0500 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4038C06178B for ; Sat, 23 Jan 2021 18:34:40 -0800 (PST) Received: by mail-pl1-x64a.google.com with SMTP id w22so5273686pll.6 for ; Sat, 23 Jan 2021 18:34:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=L5G3YxsqDIJmFyDCs25WvIUAb2GYtYzYmGV7CT5cWmg=; b=Cxq9uWe2fU8ZWzCM0YN1ya2BgwM/+I0xymnvqUccS3XzYUcT+iEVDIurK0HXK/HJXq I/dkZ4dSbYIicDi3DlnKSyoNkJgmMHP/hxN+U1KvUYXcuUzKx2caE3Qfo3p8z8/0LPqO MLLagggXnTfQl40OC6rjgA1NL2CoXa1Kya5cPkia9JoOvITu6JjcxoLOGA0rxs5pTn/t obzQCZtSQXmUlp72Pi5fkrCqo+fuykuNuIwp1XxvxHk+b7rgxi3vWQ/ZO7iYo/Qi2iKr vo95jOjDFeeaGxgfScFPSYXAcZJz6La8I8e5Ap+kFHqpe0gvlb+3y2ndbsJ8cA3FlWWj uENQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=L5G3YxsqDIJmFyDCs25WvIUAb2GYtYzYmGV7CT5cWmg=; b=s7d4a1pm5XPuF4K+x1GUya6RIYPqQ19bSN1kd4L6SVzWc7kHeoHUwLUGFWRnbOxi4A iNTRG9RgTOf8XFFYs+mThz5jDh6Fn5d8YBn8BishBzy6CBI0DBrGYis+0fvq+4bxGZr0 eTXHefKtj8mVjXVcFE9R4dSAlmgwdNMQrkqu1DFIxiHrCNfCJ1/3wk/yqbwz085/MwBd ut19q4BwqBnNF3EoA5RzGD8wqB0TCw3d5yID2g/7XltluhIGR69DilXscXg8E/7WN3Tv GTummqT8cCChPzhtdwkAnrNoAW3t0K8PBnWzbpB/TbVKeBDNwXVOk+fjN0odR0TlZjiS YCmA== X-Gm-Message-State: AOAM53102V6lFrK9tjqbaPkgCBntUItb/0ilTIQLcjwQPqWw72QQrBmL UlSeSrI00qcpajupr7lhnp2iTlzhnJUXcV9Id4IhRIE8Yp+9NL+4oqrTsnVTqtIoJyexXQbGdu3 joLYc++CvD0aB2oApfP9hOOpaTsPmaCzRg1pNU0QNIrXbJxjDrxfsfYUm2SZNM+R7Hlwc/ylkEm L4 X-Google-Smtp-Source: ABdhPJyp2H3rN2TNzR5qQItcy/LjR1K0RC0yApU7keGoyTEqg7WXqE0TV6PeVyD8aM2jAYx5bU70xsTN9LIHEROziEpb Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a17:90a:8b94:: with SMTP id z20mr1562530pjn.1.1611455680023; Sat, 23 Jan 2021 18:34:40 -0800 (PST) Date: Sat, 23 Jan 2021 18:34:29 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.280.ga3ce27912f-goog Subject: [PATCH 3/4] fetch-pack: with packfile URIs, use index-pack arg From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Unify the index-pack arguments used when processing the inline pack and when downloading packfiles referenced by URIs. This is done by teaching get_pack() to also store the index-pack arguments whenever at least one packfile URI is given, and then when processing the packfile URI(s), using the stored arguments. Signed-off-by: Jonathan Tan --- fetch-pack.c | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 274ae602f7..fe69635eb5 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -797,12 +797,13 @@ static void write_promisor_file(const char *keep_name, } /* - * Pass 1 as "only_packfile" if the pack received is the only pack in this - * fetch request (that is, if there were no packfile URIs provided). + * If packfile URIs were provided, pass a non-NULL pointer to index_pack_args. + * The string to pass as the --index-pack-args argument to http-fetch will be + * stored there. (It must be freed by the caller.) */ static int get_pack(struct fetch_pack_args *args, int xd[2], struct string_list *pack_lockfiles, - int only_packfile, + char **index_pack_args, struct ref **sought, int nr_sought) { struct async demux; @@ -845,7 +846,7 @@ static int get_pack(struct fetch_pack_args *args, strvec_push(&cmd.args, alternate_shallow_file); } - if (do_keep || args->from_promisor) { + if (do_keep || args->from_promisor || index_pack_args) { if (pack_lockfiles) cmd.out = -1; cmd_name = "index-pack"; @@ -863,7 +864,7 @@ static int get_pack(struct fetch_pack_args *args, "--keep=fetch-pack %"PRIuMAX " on %s", (uintmax_t)getpid(), hostname); } - if (only_packfile && args->check_self_contained_and_connected) + if (!index_pack_args && args->check_self_contained_and_connected) strvec_push(&cmd.args, "--check-self-contained-and-connected"); else /* @@ -901,7 +902,7 @@ static int get_pack(struct fetch_pack_args *args, : transfer_fsck_objects >= 0 ? transfer_fsck_objects : 0) { - if (args->from_promisor || !only_packfile) + if (args->from_promisor || index_pack_args) /* * We cannot use --strict in index-pack because it * checks both broken objects and links, but we only @@ -913,6 +914,19 @@ static int get_pack(struct fetch_pack_args *args, fsck_msg_types.buf); } + if (index_pack_args) { + struct strbuf joined = STRBUF_INIT; + int i; + + for (i = 0; i < cmd.args.nr; i++) { + if (i) + strbuf_addch(&joined, ' '); + strbuf_addstr_urlencode(&joined, cmd.args.v[i], + is_rfc3986_unreserved); + } + *index_pack_args = strbuf_detach(&joined, NULL); + } + cmd.in = demux.out; cmd.git_cmd = 1; if (start_command(&cmd)) @@ -1084,7 +1098,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, alternate_shallow_file = setup_temporary_shallow(si->shallow); else alternate_shallow_file = NULL; - if (get_pack(args, fd, pack_lockfiles, 1, sought, nr_sought)) + if (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought)) die(_("git fetch-pack: fetch failed.")); all_done: @@ -1535,6 +1549,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, int seen_ack = 0; struct string_list packfile_uris = STRING_LIST_INIT_DUP; int i; + char *index_pack_args = NULL; negotiator = &negotiator_alloc; fetch_negotiator_init(r, negotiator); @@ -1624,7 +1639,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, receive_packfile_uris(&reader, &packfile_uris); process_section_header(&reader, "packfile", 0); if (get_pack(args, fd, pack_lockfiles, - !packfile_uris.nr, sought, nr_sought)) + packfile_uris.nr ? &index_pack_args : NULL, + sought, nr_sought)) die(_("git fetch-pack: fetch failed.")); do_check_stateless_delimiter(args, &reader); @@ -1645,7 +1661,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, strvec_pushf(&cmd.args, "--packfile=%.*s", (int) the_hash_algo->hexsz, packfile_uris.items[i].string); - strvec_push(&cmd.args, "--index-pack-args=index-pack --stdin --keep"); + strvec_pushf(&cmd.args, "--index-pack-args=%s", index_pack_args); strvec_push(&cmd.args, uri); cmd.git_cmd = 1; cmd.no_stdin = 1; @@ -1681,6 +1697,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packname)); } string_list_clear(&packfile_uris, 0); + FREE_AND_NULL(index_pack_args); if (negotiator) negotiator->release(negotiator); From patchwork Sun Jan 24 02:34:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 12042001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-31.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B863C43381 for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 55AF022C7C for ; Sun, 24 Jan 2021 02:35:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726451AbhAXCfh (ORCPT ); Sat, 23 Jan 2021 21:35:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36296 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726408AbhAXCfW (ORCPT ); Sat, 23 Jan 2021 21:35:22 -0500 Received: from mail-yb1-xb49.google.com (mail-yb1-xb49.google.com [IPv6:2607:f8b0:4864:20::b49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 84A22C061793 for ; Sat, 23 Jan 2021 18:34:42 -0800 (PST) Received: by mail-yb1-xb49.google.com with SMTP id p80so10326146ybg.10 for ; Sat, 23 Jan 2021 18:34:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=ttB1HOiRN9uAenxGYEWvX8qTvvPet+djWbyNtQp9318=; b=K1HF4NtMzwZe9H8CUOx53ISuQKv5j6ELaY/TNju3ghRM+AeuaEJ9Pf18yxP5DaNd7a lF4am/tmTJNcoVlxSWQBeBKLzZWCD5ZfiZVAAD7QusxXMLH+SmHk8Z8MQFz+UkhaHO0V ZjGztBgTCN/RepgJTtombcAh5IFGqJTvefDKpyaHx38WVhIppGXAivOb7YBdo3MckrjC 9l5BCO3aVWBbUmaq8u1JYjA05+GBi/6ZZCBKUf55GNq5x94j6R3M0GMb2cQRsFCAGWQq uKWNwonVWbM2sZNFb4A9gZQW3F0FL88lz3Mvv/izaMQk01A1xmHvcKw045FJjrjqGpoP lguA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ttB1HOiRN9uAenxGYEWvX8qTvvPet+djWbyNtQp9318=; b=drDVeQGgY80cdruEr+f0324XSDeLqfjk7QJFS55ZgNAy/OKHSncEW+K+dekEKzCA7H ioRsWH+/g1jlAoBF8G/HdOFwB5DQse0jcOQezAwmxVAFwQ1jRuyoxz46/K4wGpPt5n+L KmD2Wcolw+yP6Mdh+CcERoJeSlUPbpmSDlW5JryD00TQtU+empjPvXUe6Tngpat1OwVI rVBS1axthI1+C163BQ7QFFXtZRxOH141EWyjqKZkiUVEIyfjKr8CyTGnNHOrhzqdRmfz XJsLCcg9w83G6Z214Hq31+c6N8IBQisFqZdk2sgRfRkUpR9yUgaCd40fu69txl1iN2TR BPQg== X-Gm-Message-State: AOAM531iKqS4O5jSiCrd39DY5HIr1R++P6bIpfgi/hcQyK2XCeXeFoSq 7lesxUdfL1CetqHIoHv5YinSmIR85XHc7JXv14r6HVaf84g5e1oBeQ0S3+siLVspVqxwKv4lKcj n1GRhs9cepi7Cxq6504OkQZAJs+tuvfIXDPvoDlNutEhnTbu81rYcv5aKtno7UIt38PqTmg3RU7 EC X-Google-Smtp-Source: ABdhPJxjDLu56eiMw0tzfZiO/pLDaGfGgeXY4zVqXrrgB1IpY0UL0vbUp5NzDZHF2jquJ0evpwIdmb50kx/xdOwg+RDH Sender: "jonathantanmy via sendgmr" X-Received: from twelve4.c.googlers.com ([fda3:e722:ac3:10:24:72f4:c0a8:437a]) (user=jonathantanmy job=sendgmr) by 2002:a25:2606:: with SMTP id m6mr16867314ybm.213.1611455681749; Sat, 23 Jan 2021 18:34:41 -0800 (PST) Date: Sat, 23 Jan 2021 18:34:30 -0800 In-Reply-To: Message-Id: <2d6d8c2671fe424c752994dcb5277d4d923e17a0.1611455251.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20210115234300.350442-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.30.0.280.ga3ce27912f-goog Subject: [PATCH 4/4] fetch-pack: print and use dangling .gitmodules From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Teach index-pack to print dangling .gitmodules links after its "keep" or "pack" line instead of declaring an error, and teach fetch-pack to check such lines printed. This allows the tree side of the .gitmodules link to be in one packfile and the blob side to be in another without failing the fsck check, because it is now fetch-pack which checks such objects after all packfiles have been downloaded and indexed (and not index-pack on an individual packfile, as it is before this commit). Signed-off-by: Jonathan Tan --- Documentation/git-index-pack.txt | 7 ++- builtin/index-pack.c | 9 +++- builtin/receive-pack.c | 2 +- fetch-pack.c | 78 +++++++++++++++++++++++++++----- fsck.c | 16 +++++-- fsck.h | 8 ++++ pack-write.c | 8 +++- pack.h | 2 +- t/t5702-protocol-v2.sh | 47 +++++++++++++++++++ 9 files changed, 155 insertions(+), 22 deletions(-) diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index af0c26232c..e74a4a1eda 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -78,7 +78,12 @@ OPTIONS Die if the pack contains broken links. For internal use only. --fsck-objects:: - Die if the pack contains broken objects. For internal use only. + For internal use only. ++ +Die if the pack contains broken objects. If the pack contains a tree +pointing to a .gitmodules blob that does not exist, prints the hash of +that blob (for the caller to check) after the hash that goes into the +name of the pack/idx file (see "Notes"). --threads=:: Specifies the number of threads to spawn when resolving diff --git a/builtin/index-pack.c b/builtin/index-pack.c index 557bd2f348..f995c15115 100644 --- a/builtin/index-pack.c +++ b/builtin/index-pack.c @@ -1888,8 +1888,13 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix) else close(input_fd); - if (do_fsck_object && fsck_finish(&fsck_options)) - die(_("fsck error in pack objects")); + if (do_fsck_object) { + struct fsck_options fo = FSCK_OPTIONS_STRICT; + + fo.print_dangling_gitmodules = 1; + if (fsck_finish(&fo)) + die(_("fsck error in pack objects")); + } free(objects); strbuf_release(&index_name_buf); diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c index d49d050e6e..ed2c9b42e9 100644 --- a/builtin/receive-pack.c +++ b/builtin/receive-pack.c @@ -2275,7 +2275,7 @@ static const char *unpack(int err_fd, struct shallow_info *si) status = start_command(&child); if (status) return "index-pack fork failed"; - pack_lockfile = index_pack_lockfile(child.out); + pack_lockfile = index_pack_lockfile(child.out, NULL); close(child.out); status = finish_command(&child); if (status) diff --git a/fetch-pack.c b/fetch-pack.c index fe69635eb5..128362e0ba 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -796,6 +796,26 @@ static void write_promisor_file(const char *keep_name, strbuf_release(&promisor_name); } +static void parse_gitmodules_oids(int fd, struct oidset *gitmodules_oids) +{ + int len = the_hash_algo->hexsz + 1; /* hash + NL */ + + do { + char hex_hash[GIT_MAX_HEXSZ + 1]; + int read_len = read_in_full(fd, hex_hash, len); + struct object_id oid; + const char *end; + + if (!read_len) + return; + if (read_len != len) + die("invalid length read %d", read_len); + if (parse_oid_hex(hex_hash, &oid, &end) || *end != '\n') + die("invalid hash"); + oidset_insert(gitmodules_oids, &oid); + } while (1); +} + /* * If packfile URIs were provided, pass a non-NULL pointer to index_pack_args. * The string to pass as the --index-pack-args argument to http-fetch will be @@ -804,7 +824,8 @@ static void write_promisor_file(const char *keep_name, static int get_pack(struct fetch_pack_args *args, int xd[2], struct string_list *pack_lockfiles, char **index_pack_args, - struct ref **sought, int nr_sought) + struct ref **sought, int nr_sought, + struct oidset *gitmodules_oids) { struct async demux; int do_keep = args->keep_pack; @@ -812,6 +833,7 @@ static int get_pack(struct fetch_pack_args *args, struct pack_header header; int pass_header = 0; struct child_process cmd = CHILD_PROCESS_INIT; + int fsck_objects = 0; int ret; memset(&demux, 0, sizeof(demux)); @@ -846,8 +868,15 @@ static int get_pack(struct fetch_pack_args *args, strvec_push(&cmd.args, alternate_shallow_file); } - if (do_keep || args->from_promisor || index_pack_args) { - if (pack_lockfiles) + if (fetch_fsck_objects >= 0 + ? fetch_fsck_objects + : transfer_fsck_objects >= 0 + ? transfer_fsck_objects + : 0) + fsck_objects = 1; + + if (do_keep || args->from_promisor || index_pack_args || fsck_objects) { + if (pack_lockfiles || fsck_objects) cmd.out = -1; cmd_name = "index-pack"; strvec_push(&cmd.args, cmd_name); @@ -897,11 +926,7 @@ static int get_pack(struct fetch_pack_args *args, strvec_pushf(&cmd.args, "--pack_header=%"PRIu32",%"PRIu32, ntohl(header.hdr_version), ntohl(header.hdr_entries)); - if (fetch_fsck_objects >= 0 - ? fetch_fsck_objects - : transfer_fsck_objects >= 0 - ? transfer_fsck_objects - : 0) { + if (fsck_objects) { if (args->from_promisor || index_pack_args) /* * We cannot use --strict in index-pack because it @@ -931,10 +956,15 @@ static int get_pack(struct fetch_pack_args *args, cmd.git_cmd = 1; if (start_command(&cmd)) die(_("fetch-pack: unable to fork off %s"), cmd_name); - if (do_keep && pack_lockfiles) { - char *pack_lockfile = index_pack_lockfile(cmd.out); + if (do_keep && (pack_lockfiles || fsck_objects)) { + int is_well_formed; + char *pack_lockfile = index_pack_lockfile(cmd.out, &is_well_formed); + + if (!is_well_formed) + die(_("fetch-pack: invalid index-pack output")); if (pack_lockfile) string_list_append_nodup(pack_lockfiles, pack_lockfile); + parse_gitmodules_oids(cmd.out, gitmodules_oids); close(cmd.out); } @@ -969,6 +999,22 @@ static int cmp_ref_by_name(const void *a_, const void *b_) return strcmp(a->name, b->name); } +static void fsck_gitmodules_oids(struct oidset *gitmodules_oids) +{ + struct oidset_iter iter; + const struct object_id *oid; + struct fsck_options fo = FSCK_OPTIONS_STRICT; + + if (!oidset_size(gitmodules_oids)) + return; + + oidset_iter_init(gitmodules_oids, &iter); + while ((oid = oidset_iter_next(&iter))) + register_found_gitmodules(oid); + if (fsck_finish(&fo)) + die("fsck failed"); +} + static struct ref *do_fetch_pack(struct fetch_pack_args *args, int fd[2], const struct ref *orig_ref, @@ -983,6 +1029,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, int agent_len; struct fetch_negotiator negotiator_alloc; struct fetch_negotiator *negotiator; + struct oidset gitmodules_oids = OIDSET_INIT; negotiator = &negotiator_alloc; fetch_negotiator_init(r, negotiator); @@ -1098,8 +1145,10 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, alternate_shallow_file = setup_temporary_shallow(si->shallow); else alternate_shallow_file = NULL; - if (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought)) + if (get_pack(args, fd, pack_lockfiles, NULL, sought, nr_sought, + &gitmodules_oids)) die(_("git fetch-pack: fetch failed.")); + fsck_gitmodules_oids(&gitmodules_oids); all_done: if (negotiator) @@ -1550,6 +1599,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, struct string_list packfile_uris = STRING_LIST_INIT_DUP; int i; char *index_pack_args = NULL; + struct oidset gitmodules_oids = OIDSET_INIT; negotiator = &negotiator_alloc; fetch_negotiator_init(r, negotiator); @@ -1640,7 +1690,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, process_section_header(&reader, "packfile", 0); if (get_pack(args, fd, pack_lockfiles, packfile_uris.nr ? &index_pack_args : NULL, - sought, nr_sought)) + sought, nr_sought, &gitmodules_oids)) die(_("git fetch-pack: fetch failed.")); do_check_stateless_delimiter(args, &reader); @@ -1680,6 +1730,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packname[the_hash_algo->hexsz] = '\0'; + parse_gitmodules_oids(cmd.out, &gitmodules_oids); + close(cmd.out); if (finish_command(&cmd)) @@ -1699,6 +1751,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, string_list_clear(&packfile_uris, 0); FREE_AND_NULL(index_pack_args); + fsck_gitmodules_oids(&gitmodules_oids); + if (negotiator) negotiator->release(negotiator); diff --git a/fsck.c b/fsck.c index f82e2fe9e3..04f3d342af 100644 --- a/fsck.c +++ b/fsck.c @@ -1243,6 +1243,11 @@ int fsck_error_function(struct fsck_options *o, return 1; } +void register_found_gitmodules(const struct object_id *oid) +{ + oidset_insert(&gitmodules_found, oid); +} + int fsck_finish(struct fsck_options *options) { int ret = 0; @@ -1262,10 +1267,13 @@ int fsck_finish(struct fsck_options *options) if (!buf) { if (is_promisor_object(oid)) continue; - ret |= report(options, - oid, OBJ_BLOB, - FSCK_MSG_GITMODULES_MISSING, - "unable to read .gitmodules blob"); + if (options->print_dangling_gitmodules) + printf("%s\n", oid_to_hex(oid)); + else + ret |= report(options, + oid, OBJ_BLOB, + FSCK_MSG_GITMODULES_MISSING, + "unable to read .gitmodules blob"); continue; } diff --git a/fsck.h b/fsck.h index 69cf715e79..4b8cf03445 100644 --- a/fsck.h +++ b/fsck.h @@ -41,6 +41,12 @@ struct fsck_options { int *msg_type; struct oidset skiplist; kh_oid_map_t *object_names; + + /* + * If 1, print the hashes of missing .gitmodules blobs instead of + * considering them to be errors. + */ + unsigned print_dangling_gitmodules:1; }; #define FSCK_OPTIONS_DEFAULT { NULL, fsck_error_function, 0, NULL, OIDSET_INIT } @@ -62,6 +68,8 @@ int fsck_walk(struct object *obj, void *data, struct fsck_options *options); int fsck_object(struct object *obj, void *data, unsigned long size, struct fsck_options *options); +void register_found_gitmodules(const struct object_id *oid); + /* * Some fsck checks are context-dependent, and may end up queued; run this * after completing all fsck_object() calls in order to resolve any remaining diff --git a/pack-write.c b/pack-write.c index 3513665e1e..f66ea8e5a1 100644 --- a/pack-write.c +++ b/pack-write.c @@ -272,7 +272,7 @@ void fixup_pack_header_footer(int pack_fd, fsync_or_die(pack_fd, pack_name); } -char *index_pack_lockfile(int ip_out) +char *index_pack_lockfile(int ip_out, int *is_well_formed) { char packname[GIT_MAX_HEXSZ + 6]; const int len = the_hash_algo->hexsz + 6; @@ -286,11 +286,17 @@ char *index_pack_lockfile(int ip_out) */ if (read_in_full(ip_out, packname, len) == len && packname[len-1] == '\n') { const char *name; + + if (is_well_formed) + *is_well_formed = 1; packname[len-1] = 0; if (skip_prefix(packname, "keep\t", &name)) return xstrfmt("%s/pack/pack-%s.keep", get_object_directory(), name); + return NULL; } + if (is_well_formed) + *is_well_formed = 0; return NULL; } diff --git a/pack.h b/pack.h index 9fc0945ac9..09cffec395 100644 --- a/pack.h +++ b/pack.h @@ -85,7 +85,7 @@ int verify_pack_index(struct packed_git *); int verify_pack(struct repository *, struct packed_git *, verify_fn fn, struct progress *, uint32_t); off_t write_pack_header(struct hashfile *f, uint32_t); void fixup_pack_header_footer(int, unsigned char *, const char *, uint32_t, unsigned char *, off_t); -char *index_pack_lockfile(int fd); +char *index_pack_lockfile(int fd, int *is_well_formed); /* * The "hdr" output buffer should be at least this big, which will handle sizes diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 7d5b17909b..8b8fb43dbc 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -936,6 +936,53 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails on bad object' test_i18ngrep "invalid author/committer line - missing email" error ' +test_expect_success 'packfile-uri with transfer.fsckobjects succeeds when .gitmodules is separate from tree' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + rm -rf "$P" http_child && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + echo "[submodule libfoo]" >"$P/.gitmodules" && + echo "path = include/foo" >>"$P/.gitmodules" && + echo "url = git://example.com/git/lib.git" >>"$P/.gitmodules" && + git -C "$P" add .gitmodules && + git -C "$P" commit -m x && + + configure_exclusion "$P" .gitmodules >h && + + sane_unset GIT_TEST_SIDEBAND_ALL && + git -c protocol.version=2 -c transfer.fsckobjects=1 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + + # Ensure that there are exactly 4 files (2 .pack and 2 .idx). + ls http_child/.git/objects/pack/* >filelist && + test_line_count = 4 filelist +' + +test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodules separate from tree is invalid' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + rm -rf "$P" http_child err && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + echo "[submodule \"..\"]" >"$P/.gitmodules" && + echo "path = include/foo" >>"$P/.gitmodules" && + echo "url = git://example.com/git/lib.git" >>"$P/.gitmodules" && + git -C "$P" add .gitmodules && + git -C "$P" commit -m x && + + configure_exclusion "$P" .gitmodules >h && + + sane_unset GIT_TEST_SIDEBAND_ALL && + test_must_fail git -c protocol.version=2 -c transfer.fsckobjects=1 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child 2>err && + test_i18ngrep "disallowed submodule name" err +' + # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled.