From patchwork Fri Mar 11 16:24:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12778351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51AF6C433EF for ; Fri, 11 Mar 2022 16:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350273AbiCKQZt (ORCPT ); Fri, 11 Mar 2022 11:25:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36954 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350250AbiCKQZi (ORCPT ); Fri, 11 Mar 2022 11:25:38 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8CD73107D02 for ; Fri, 11 Mar 2022 08:24:33 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id i8so13720232wrr.8 for ; Fri, 11 Mar 2022 08:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=rqxJNJYD78ta3kVwBh7gAzBuMvN0MJDx+7M+5V/Ljoo=; b=SiIRuxzDP9jGFHrJWbicXyO6u/jpghH4zIJpz5TWzSiD4RGMO4PauX3KYBdgKsOSTo KCVAneZW6KoAY4wTuVI4IgTenJeKqERKsCka8jrwe+QQrWbQm8ByHH8Pj2RoPvh2F/oZ RRUZYnVCLn6+pRFVatkB6Ff5hCz+vmVlCzsx5PBoOaa0cEfjUhdz2SgiPbpWviW85CJt BTaS1XWqlz2pzQPfQvuZDMFgxzve9nu8Wjh1RpVv93ugHwp7r2RoX76FXWScryCBwBgL QRyvHLBs2ckh28v2scHj3iJTnsboXIHrXhMDKKuRYfw1Ut4XfwyABTBUt1M13YTN480Q FmLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=rqxJNJYD78ta3kVwBh7gAzBuMvN0MJDx+7M+5V/Ljoo=; b=cf8vRqq3IywHMn6rPGUjM7ILhcUzL25tg2NAvRJoLcSdmCRxdKwZgw9R65khlAbAk3 96PuQOxbFTFMg7AjTRuhWtKHjcW9v9YsA+3oj7HVK5Wtkm36FbpsnOw1XZwgjEhuIvdG JXC2sMCIKIFxQuaoZrCft5XRqyf4U7MstoEU8LiOkCf6Zn3TcAK0vhjDr0s4JjdYX7qF alMMMmA/xf96dhHh7XdHxHwLZX667COA8beVM+taidtqK7BDLMGXe/gDFCVwqxxgQW0q e2AyQ7hjEq2LNbEqqyknXna7/oYvThABN0aDUm9/P0wWTtV2+K/FXtp3cvZeDb0Czlmz e2KA== X-Gm-Message-State: AOAM531PCJuxgHz2Nt76ZCAVgvnjhoHgQJCPMJU1NPnlvqqCeobfwNzE ypDW/n+pjbMUy4ELDWPHZutu8Tt6XnnxXQ== X-Google-Smtp-Source: ABdhPJyYyASzuNf8ZkYPQfZGJt1UqixmLI/x8NmhCsZeRXGboGB9iTL8EgiO61bZ44me+6GXymb/vg== X-Received: by 2002:adf:ebd0:0:b0:1e3:f9b:7b77 with SMTP id v16-20020adfebd0000000b001e30f9b7b77mr7662448wrn.691.1647015871323; Fri, 11 Mar 2022 08:24:31 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id f22-20020a1cc916000000b00380d3e49e89sm7318667wmb.22.2022.03.11.08.24.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:24:30 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Derrick Stolee , Jonathan Tan , Jonathan Nieder , Albert Cui , "Robin H . Johnson" , Teng Long , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH v2 01/13] protocol v2: add server-side "bundle-uri" skeleton Date: Fri, 11 Mar 2022 17:24:13 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1337.g7e32d794afe In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". The use-cases are similar to those of the existing "Packfile URIs", and the two feature can be combined within a single request, but "bundle-uri" has a few advantages over packfile-uris in some some common scenarios, discussed below. This change does not give us a working "bundle-uri" client, subsequent commits will do that. Let's first establish what the protocol for this should be like first. The client implementation will then implement this specification. With this change when the uploadpack.bundleURI config is set to a URI (or URIs, if set >1 times), advertise a "bundle-uri" command. Then when the client requests "bundle-uri" emit those URIs back at them. Differences between this and the existing packfile-uri facility: A. There is no "real" support for packfile-uri in git.git. The uploadpack.blobPackfileUri setting allows carving out a list of blobs (actually any OIDs), but as alluded to in bfc2a36ff2a (Doc: clarify contents of packfile sent as URI, 2021-01-20) the only "real" implementation is JGit based. B. The uploadpack.blobPackfileUri is a MUST where this is a "CAN". I.e. once a client says they support packfile-uri of given list of protocols the server will send them a PACK response assuming they've downloaded the URI they client was sent, if the client doesn't do that they don't have a valid repository. Pointing at a bundle and having the client send us "have" lines (or not, maybe they couldn't fetch it, or decided they didn't want to) is more flexible, and can gracefully recover e.g. if the CDN isn't reachable (maybe you do support "https", but the CDN provider is down, or blocked your whole country). C. The client, after executing "ls-refs" will disconnect if it has also grabbed the "bundle-uris" and knows the server won't send it anything it doesn't already have (or expect to have, if it's downloading the bundles concurrent to an early disconnect). This is in (small) contrast to packfile-uri where a client would enter a negotiation dialog, which may or may not result in a packfile-uri and/or an inline PACK. D. Because of "C" clients can, if the bundles are up-to-date, get an up-to-date repository with just "bundle-uri" and "ls-refs" commands, with no need to enter a dialog with "git upload-pack". That small dialog is unlikely to matter for performance purposes, this section is just noting differences between "bundle-uri" and "packfile-uri". As noted above the features are compatible, a client that supports "bundle-uri" and "packfile-uri" might download a bundle, and then proceed with a "fetch" dialog, that dialog might then result in "packfile-uri" response. In practice server operators are unlikely to want to mix the two, since the main benefit of either approach is the ability to offload large "clone" responses to CDNs. A server operator would have little reason not to go with one approach or the other. There was a suggestion of implementing a similar feature long ago[1] by Jeff King. The main difference between it and this approach is that we've since gained protocol v2, so we can add this as an optional path in the dialog between client and server. The 2011 implementation hooked into the transport mechanism to try to clone from a bundle directly. See also [2] and [3] for some later mentions of that approach. See also [4] for the series that implemented uploadpack.blobPackfileUri, and [5] for a series on top that did the .gitmodules check in that context. See [6] for the "ls-refs unborn" feature which modified code in similar areas of the request flow. Finally, there's currently a concurrent (submitted after the v1 of this commit, but before the subsequent client parts of this implementation) RFC of a somewhat similar "bundle-uri" facility at [7]. 1. https://lore.kernel.org/git/20111110074330.GA27925@sigill.intra.peff.net/ 2. https://lore.kernel.org/git/20190514092900.GA11679@sigill.intra.peff.net/ 3. https://lore.kernel.org/git/YFJWz5yIGng+a16k@coredump.intra.peff.net/ 4. https://lore.kernel.org/git/cover.1591821067.git.jonathantanmy@google.com/ Merged as 34e849b05a4 (Merge branch 'jt/cdn-offload', 2020-06-25) 5. https://lore.kernel.org/git/cover.1614021092.git.jonathantanmy@google.com/ Merged as 6ee353d42f3 (Merge branch 'jt/transfer-fsck-across-packs', 2021-03-01) 6. 69571dfe219 (Merge branch 'jt/clone-unborn-head', 2021-02-17) 7. https://lore.kernel.org/git/pull.1160.git.1645641063.gitgitgadget@gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/technical/protocol-v2.txt | 209 ++++++++++++++++++++++++ Makefile | 1 + bundle-uri.c | 55 +++++++ bundle-uri.h | 13 ++ serve.c | 6 + t/t5701-git-serve.sh | 124 +++++++++++++- 6 files changed, 407 insertions(+), 1 deletion(-) create mode 100644 bundle-uri.c create mode 100644 bundle-uri.h diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 8a877d27e23..3ea96add398 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -566,3 +566,212 @@ and associated requested information, each separated by a single space. attr = "size" obj-info = obj-id SP obj-size + +bundle-uri +~~~~~~~~~~ + +If the 'bundle-uri' capability is advertised, the server supports the +`bundle-uri' command. + +The capability is currently advertised with no value (i.e. not +"bundle-uri=somevalue"), a value may be added in the future for +supporting command-wide extensions. Clients MUST ignore any unknown +capability values and proceed with the 'bundle-uri` dialog they +support. + +The 'bundle-uri' command is intended to be issued before `fetch` to +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and +inform the subsequent `fetch` command. + +The client CAN issue `bundle-uri` before or after any other valid +command. To be useful to clients it's expected that it'll be issued +after an `ls-refs` and before `fetch`, but CAN be issued at any time +in the dialog. + +DISCUSSION of bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^^ + +The intent of the feature is optimize for server resource consumption +in the common case by changing the common case of fetching a very +large PACK during linkgit:git-clone[1] into a smaller incremental +fetch. + +It also allows servers to achieve better caching in combination with +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]). + +By having new clones or fetches be a more predictable and common +negotiation against the tips of recently produces *.bundle file(s). +Servers might even pre-generate the results of such negotiations for +the `uploadpack.packObjectsHook` as new pushes come in. + +I.e. the server would anticipate that fresh clones will download a +known bundle, followed by catching up to the current state of the +repository using ref tips found in that bundle (or bundles). + +PROTOCOL for bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^ + +A `bundle-uri` request takes no arguments, and as noted above does not +currently advertise a capability value. Both may be added in the +future. + +When the client issues a `command=bundle-uri` the response is a list +of URIs the server would like the client to fetch out-of-bounds before +proceeding with the `fetch` request in this format: + + output = bundle-uri-line + bundle-uri-line* flush-pkt + + bundle-uri-line = PKT-LINE(bundle-uri) + *(SP bundle-feature-key *(=bundle-feature-val)) + LF + + bundle-uri = A URI such as a https://, ssh:// etc. URI + + bundle-feature-key = Any printable ASCII characters except SP or "=" + bundle-feature-val = Any printable ASCII characters except SP or "=" + +No `bundle-feature-key`=`bundle-feature-value` fields are currently +defined. See the discussion of features below. + +Clients are still expected to fully parse the line according to the +above format, lines that do not conform to the format SHOULD be +discarded. The user MAY be warned in such a case. + +bundle-uri CLIENT AND SERVER EXPECTATIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +".bundle" FORMAT +++++++++++++++++ + +The advertised bundle(s) MUST be in a format that "git bundle verify" +would accept. I.e. they MUST contain one or more reference tips for +use by the client, MUST indicate prerequisites (in any) with standard +"-" prefixes, and MUST indicate their "object-format", if +applicable. Create "*.bundle" files with "git bundle create". + +bundle-uri CLIENT ERROR RECOVERY +++++++++++++++++++++++++++++++++ + +A client MUST above all gracefully degrade on errors, whether that +error is because of bad missing/data in the bundle URI(s), because +that client is too dumb to e.g. understand and fully parse out bundle +headers and their prerequisite relationships, or something else. + +Server operators should feel confident in turning on "bundle-uri" and +not worry if e.g. their CDN goes down that clones or fetches will run +into hard failures. Even if the server bundle bundle(s) are +incomplete, or bad in some way the client should still end up with a +functioning repository, just as if it had chosen not to use this +protocol extension. + +All subsequent discussion on client and server interaction MUST keep +this in mind. + +bundle-uri SERVER TO CLIENT ++++++++++++++++++++++++++++ + +The ordering of the returned bundle uris is not significant. Clients +MUST parse their headers to discover their contained OIDS and +prerequisites. A client MUST consider the content of the bundle(s) +themselves and their header as the ultimate source of truth. + +A server MAY even return bundle(s) that don't have any direct +relationship to the repository being cloned (either through accident, +or intentional "clever" configuration), and expect a client to sort +out what data they'd like from the bundle(s), if any. + +bundle-uri CLIENT TO SERVER ++++++++++++++++++++++++++++ + +The client SHOULD provide reference tips found in the bundle header(s) +as 'have' lines in any subsequent `fetch` request. A client MAY also +ignore the bundle(s) entirely if doing so is deemed worse for some +reason, e.g. if the bundles can't be downloaded, it doesn't like the +tips it finds etc. + +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s) +of the bundle(s) the client finds that the ref tips it wants can be +retrieved entirety from advertised bundle(s), it MAY disconnect. The +results of such a 'clone' or 'fetch' should be indistinguishable from +the state attained without using bundle-uri. + +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY +++++++++++++++++++++++++++++++++++++++++++++++ + +A client MAY perform an early disconnect while still downloading the +bundle(s) (having streamed and parsed their headers). In such a case +the client MUST gracefully recover from any errors related to +finishing the download and validation of the bundle(s). + +I.e. a client might need to re-connect and issue a 'fetch' command, +and possibly fall back to not making use of 'bundle-uri' at all. + +This "MAY" behavior is specified as such (and not a "SHOULD") on the +assumption that a server advertising bundle uris is more likely than +not to be serving up a relatively large repository, and to be pointing +to URIs that have a good chance of being in working order. A client +MAY e.g. look at the payload size of the bundles as a heuristic to see +if an early disconnect is worth it, should falling back on a full +"fetch" dialog be necessary. + +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION ++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +A client SHOULD commence a negotiation of a PACK from the server via +the "fetch" command using the OID tips found in advertised bundles, +even if's still in the process of downloading those bundle(s). + +This allows for aggressive early disconnects from any interactive +server dialog. The client blindly trusts that the advertised OID tips +are relevant, and issues them as 'have' lines, it then requests any +tips it would like (usually from the "ls-refs" advertisement) via +'want' lines. The server will then compute a (hopefully small) PACK +with the expected difference between the tips from the bundle(s) and +the data requested. + +The only connection the client then needs to keep active is to the +concurrently downloading static bundle(s), when those and the +incremental PACK are retrieved they should be inflated and +validated. Any errors at this point should be gracefully recovered +from, see above. + +bundle-uri PROTOCOL FEATURES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As noted above no `bundle-feature-key`=`bundle-feature-value` fields +are currently defined. + +They are intended for future per-URI metadata which older clients MUST +ignore and gracefully degrade on. Any fields they do recognize they +CAN also ignore. + +Any backwards-incompatible addition of pre-URI key-value will be +guarded by a new value or values in 'bundle-uri' capability +advertisement itself, and/or by new future `bundle-uri` request +arguments. + +While no per-URI key-value are currently supported currently they're +intended to support future features such as: + + * Add a "hash=" or "size=" advertise the expected hash or + size of the bundle file. + + * Advertise that one or more bundle files are the same (to e.g. have + clients round-robin or otherwise choose one of N possible files). + + * A "oid=" shortcut and "prerequisite=" shortcut. For + expressing the common case of a bundle with one tip and no + prerequisites, or one tip and one prerequisite. ++ +This would allow for optimizing the common case of servers who'd like +to provide one "big bundle" containing only their "main" branch, +and/or incremental updates thereof. ++ +A client receiving such a a response MAY assume that they can skip +retrieving the header from a bundle at the indicated URI, and thus +save themselves and the server(s) the request(s) needed to inspect the +headers of that bundle or bundles. diff --git a/Makefile b/Makefile index 6f0b4b775fe..5a3d35109a1 100644 --- a/Makefile +++ b/Makefile @@ -855,6 +855,7 @@ LIB_OBJS += blob.o LIB_OBJS += bloom.o LIB_OBJS += branch.o LIB_OBJS += bulk-checkin.o +LIB_OBJS += bundle-uri.o LIB_OBJS += bundle.o LIB_OBJS += cache-tree.o LIB_OBJS += cbtree.o diff --git a/bundle-uri.c b/bundle-uri.c new file mode 100644 index 00000000000..ff054ddc690 --- /dev/null +++ b/bundle-uri.c @@ -0,0 +1,55 @@ +#include "cache.h" +#include "bundle-uri.h" +#include "pkt-line.h" +#include "config.h" + +static void send_bundle_uris(struct packet_writer *writer, + struct string_list *uris) +{ + struct string_list_item *item; + + for_each_string_list_item(item, uris) + packet_writer_write(writer, "%s", item->string); +} + +static int advertise_bundle_uri = -1; +static struct string_list bundle_uris = STRING_LIST_INIT_DUP; +static int bundle_uri_config(const char *var, const char *value, void *data) +{ + if (!strcmp(var, "uploadpack.bundleuri")) { + advertise_bundle_uri = 1; + string_list_append(&bundle_uris, value); + } + + return 0; +} + +int bundle_uri_advertise(struct repository *r, struct strbuf *value) +{ + if (advertise_bundle_uri != -1) + goto cached; + + git_config(bundle_uri_config, NULL); + advertise_bundle_uri = !!bundle_uris.nr; + +cached: + return advertise_bundle_uri; +} + +int bundle_uri_command(struct repository *r, + struct packet_reader *request) +{ + struct packet_writer writer; + packet_writer_init(&writer, 1); + + while (packet_reader_read(request) == PACKET_READ_NORMAL) + die(_("bundle-uri: unexpected argument: '%s'"), request->line); + if (request->status != PACKET_READ_FLUSH) + die(_("bundle-uri: expected flush after arguments")); + + send_bundle_uris(&writer, &bundle_uris); + + packet_writer_flush(&writer); + + return 0; +} diff --git a/bundle-uri.h b/bundle-uri.h new file mode 100644 index 00000000000..5a7e556a0ba --- /dev/null +++ b/bundle-uri.h @@ -0,0 +1,13 @@ +#ifndef BUNDLE_URI_H +#define BUNDLE_URI_H +#include "repository.h" +#include "pkt-line.h" +#include "strbuf.h" + +/** + * API used by serve.[ch]. + */ +int bundle_uri_advertise(struct repository *r, struct strbuf *value); +int bundle_uri_command(struct repository *r, struct packet_reader *request); + +#endif /* BUNDLE_URI_H */ diff --git a/serve.c b/serve.c index b3fe9b5126a..f3e0203d2c6 100644 --- a/serve.c +++ b/serve.c @@ -8,6 +8,7 @@ #include "protocol-caps.h" #include "serve.h" #include "upload-pack.h" +#include "bundle-uri.h" static int advertise_sid = -1; static int client_hash_algo = GIT_HASH_SHA1; @@ -136,6 +137,11 @@ static struct protocol_capability capabilities[] = { .advertise = always_advertise, .command = cap_object_info, }, + { + .name = "bundle-uri", + .advertise = bundle_uri_advertise, + .command = bundle_uri_command, + }, }; void protocol_v2_advertise_capabilities(void) diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index 1896f671cb3..9d053f77a93 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' ' wrong_algo sha1:sha256 wrong_algo sha256:sha1 EOF - cat >expect <<-EOF && + cat >expect.base <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) ls-refs=unborn @@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' ' server-option object-format=$(test_oid algo) object-info + EOF + cat >expect.trailer <<-EOF && 0000 EOF + cat expect.base expect.trailer >expect && GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ --advertise-capabilities >out && @@ -342,4 +345,123 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +# Test the basics of bundle-uri +# +test_expect_success 'test capability advertisement with uploadpack.bundleURI' ' + test_config uploadpack.bundleURI FAKE && + + cat >expect.extra <<-EOF && + bundle-uri + EOF + cat expect.base \ + expect.extra \ + expect.trailer >expect && + + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ + --advertise-capabilities >out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: dies if not enabled' ' + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: invalid command '"'"'bundle-uri'"'"' + EOF + + cat >expect <<-\EOF && + ERR serve: invalid command '"'"'bundle-uri'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + + +test_expect_success 'basics of bundle-uri: enabled with single URI' ' + test_config uploadpack.bundleURI https://cdn.example.com/repo.bdl && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0000 + EOF + + cat >expect <<-EOF && + https://cdn.example.com/repo.bdl + 0000 + EOF + + test-tool serve-v2 --stateless-rpc out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: enabled with single URI' ' + test_config uploadpack.bundleURI https://cdn.example.com/repo.bdl && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0000 + EOF + + cat >expect <<-EOF && + https://cdn.example.com/repo.bdl + 0000 + EOF + + test-tool serve-v2 --stateless-rpc out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: enabled with two URIs' ' + test_config uploadpack.bundleURI https://cdn.example.com/repo.bdl && + test_config uploadpack.bundleURI https://cdn.example.com/recent.bdl --add && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0000 + EOF + + cat >expect <<-EOF && + https://cdn.example.com/repo.bdl + https://cdn.example.com/recent.bdl + 0000 + EOF + + test-tool serve-v2 --stateless-rpc out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: unknown future feature(s)' ' + test_config uploadpack.bundleURI https://cdn.example.com/fake.bdl && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0001 + some-feature + we-do-not + know=about + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: bundle-uri: unexpected argument: '"'"'some-feature'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + test_done From patchwork Fri Mar 11 16:24:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12778350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2ADC5C433F5 for ; Fri, 11 Mar 2022 16:24:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350287AbiCKQZs (ORCPT ); Fri, 11 Mar 2022 11:25:48 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36924 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350242AbiCKQZi (ORCPT ); Fri, 11 Mar 2022 11:25:38 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0F840107D9F for ; Fri, 11 Mar 2022 08:24:34 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id j26so13748125wrb.1 for ; Fri, 11 Mar 2022 08:24:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=hW5j2D8aiAs8OcWGVTQCf1AGBhVI0YUZtaUfyCwlOGE=; b=knnh7oJW46wlGOjU0LrsvEPHDuplgz/qxBPQGHW7nafC1Duj3cpJQkOrlxDf56ITGs 8GNq7gjWyMt0NHEEkQkgqsB0nCY2/cEHQD483PySigorB4PRVWIj/741M/Qu3gNZQORK 4pS4OAvKAJ8ccFTp/Uy7Pr9NHsjLPwPMT/jW9cgcmyJeJkYqEy+wkfP0M2a6Bt5DH6wW kxyFk/UE6K/YoEhGNSnnwXQ6qZZLxXdnjlXgZ680VTp8eD9VNITUEedZQLAu3KlzJ08c PJr+m6GFwm7dZcxc/CeGdctHZWZNc52lLmGk9FlPnV9AXyucRIKRP7MI5HH7WUDXLINT 5bmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=hW5j2D8aiAs8OcWGVTQCf1AGBhVI0YUZtaUfyCwlOGE=; b=FtDaNaJK0HuTQF8OUxwz8+dv80r0yXwbSM+ZgmPrS6q+Vz1s+mUAi1tSO3B/H1Di2W I+cmqtwssSTkIL9VGgorgVxoHJM9fVFlhqi8oRmUR0nF5XAVXSC3adeWKRYQtdTjLS4Q vyh14yFvYeaUtyTfqbNEZC545JMVCkJMW/uw+Cx64QARl/Ai10h6ZPaJuKzm2pka1/wU sUkKDd7cY3Rdb/MWMCGaovX3R+Nss+Z9gyQ0L35HrpamoXEAtZExabs2lYZo0w7I6gfa EMJTx/z9WBCPregLtu+rJX0dv9nCWQ8DFeh7RdrZkhXZ2ODVC4UsUUalb4IJvkHHuqWO 3wUQ== X-Gm-Message-State: AOAM531CDxzzTzXvWPt0wO8NXr+W7SZi515LCZV7excw9dC7AW0G3Q1A u+HPg0yTGxYMbHqGhAESY00h8JX/XSOsBA== X-Google-Smtp-Source: ABdhPJwtwbrlkXbsV2Ae2rgOoWD7KUJ3KiZxthQr9X8xkaZPXE/SaOt3GAlY6JeefRq7Es66iNQTLA== X-Received: by 2002:a5d:598b:0:b0:203:95c0:7b72 with SMTP id n11-20020a5d598b000000b0020395c07b72mr3292580wri.172.1647015872225; Fri, 11 Mar 2022 08:24:32 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id f22-20020a1cc916000000b00380d3e49e89sm7318667wmb.22.2022.03.11.08.24.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:24:31 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Derrick Stolee , Jonathan Tan , Jonathan Nieder , Albert Cui , "Robin H . Johnson" , Teng Long , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH v2 02/13] bundle-uri docs: add design notes Date: Fri, 11 Mar 2022 17:24:14 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1337.g7e32d794afe In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a design doc for the bundle-uri protocol extension to go along with the packfile-uri extension added in cd8402e0fd8 (Documentation: add Packfile URIs design doc, 2020-06-10). Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/technical/bundle-uri.txt | 119 ++++++++++++++++++++++++ Documentation/technical/protocol-v2.txt | 5 + 2 files changed, 124 insertions(+) create mode 100644 Documentation/technical/bundle-uri.txt diff --git a/Documentation/technical/bundle-uri.txt b/Documentation/technical/bundle-uri.txt new file mode 100644 index 00000000000..5ae9a15eafe --- /dev/null +++ b/Documentation/technical/bundle-uri.txt @@ -0,0 +1,119 @@ +Bundle URI Design Notes +======================= + +Protocol +-------- + +See `bundle-uri` in the link:protocol-v2.html[protocol-v2] +documentation for a discussion of the bundle-uri command, and the +expectations of clients and servers. + +This document is a a more general discussion of how the `bundle-uri` +command fits in with the rest of the git ecosystem, its design goals +and non-goals, comparison to alternatives etc. + +Comparison with Packfile URIs +----------------------------- + +There is a similar "Packfile URIs" facility, see the +link:packfile-uri.html[packfile-uri] documentation for details. + +The Packfile URIs facility requires a much closer cooperation between +CDN and server than the bundle URI facility. + +I.e. the server MUST know what objects exist in the packfile URI it's +pointing to, as well as its pack checksum. Failure to do so will not +only result in a client error (the packfile hash won't match), but +even if it got past that would likely result in a corrupt repository +with tips pointing to unreachable objects. + +By comparison the bundle URIs are meant to be a "dumb" solution +friendly to e.g. having a weekly cronjob take a snapshot of a git +repository, that snapshot being uploaded to a network of FTP mirrors +(which may be inconsistent or out of date). + +The server does not need to know what state the side-channel download +is at, because the client will first validate it, and then optionally +negotiate with the server using what it discovers there. + +Using the local `transfer.injectBundleURI` configuration variable (see +linkgit:git-config[1]) the `bundle-uri` mechanism doesn't even need +the server to support it. + +Security +-------- + +The omission of something equivalent to the packfile in the +Packfile URIs protocol is intentional, as having it would require +closer server and CDN cooperation than some server operators are +comfortable with. + +Furthermore, it is not needed for security. The server doesn't need to +trust its CDN. If the server were to attempt to send harmful content +to the client, the result would not validate against the server's +provided ref tips gotten from ls-refs. + +The lack of a such a hash does leave room open to a malicious CDN +operation to be annoying however. E.g. they could inject irrelevant +objects into the bundles, which would enlarge the downloaded +repository until a "gc" would eventually throw them away. + +In practice the lack of a hash is considered to be a non-issue. Anyone +concerned about such security problems between their server and their +CDN is going to be pointing to a "https" URL under their control. For +a client the "threat" is the same as without bundle-uri, i.e. a server +is free to be annoying today and send you garbage in the PACK that you +won't need. + +Security issues peculiar to bundle-uri +-------------------------------------- + +Both packfile-uri and bundle-uri use the `fetch.uriProtocols` +configuration variable (see linkgit:git-config[1]) to configure which +protocols they support. + +By default this is set to "http,https" for both, but bundle-uri +supports adding "file" to that list. The server can thus point to +"file://" URIs it expects the client to have access to. + +This is primarily intended for use with the `transfer.injectBundleURI` +mechanism, but can also be useful e.g. in a centralized environment +where a server might point to a "file:///mnt/bundles/big-repo.bdl" it +knows to be mounted on the local machine (e.g. a racked server), +points to it in its "bundle-uri" response. + +The client can then add "file" to the `fetch.uriProtocols` list to +obey such responses. That does mean that a malicious server can point +to any arbitrary file on the local machine. The threat of this is +considered minimal, since anyone adding `file` to `fetch.uriProtocols` +likely knows what they're doing and controls both ands, and the worst +they can do is make a curl(1) pipe garbage into "index-pack" (which +will likely promptly die on the non-PACK-file). + +Security comparison with packfile-uri +------------------------------------- + +The initial implementation of packfile-uri needed special adjusting to +run "git fsck" on incoming .gitmodules files, this was to deal with a +general security issue in git, See CVE-2018-17456. + +The current packfile-uri mechanism requires special handling around +"fsck" to do such cross-PACK fsck's, this is because it first indexes +the "incremental" PACK, and then any PACK(s) provided via +packfile-uri, before finally doing a full connectivity check. + +This is effect doing the fsck one might do via "clone" and "fetch" in +reverse, or the equivalent of starting with the incremental "fetch", +followed by the "clone". + +Since the packfile-uri mechanism can result in the .gitmodules blob +referenced by such a "fetch" to be in the pack for the "clone" the +fetch-pack process needs to keep state between the indexing of +multiple packs, to remember to fsck the blob (via the "clone") later +after seeing it in a tree (from the "fetch). + +There are no known security issues with the way packfile-uri does +this, but since bundle-uri effectively emulates what a which doesn't +support either "bundle-uri" or "packfile-uri" would do on clone/fetch, +any future security issues peculiar to the packfile-uri approach are +unlikely to be shared by it. diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 3ea96add398..3a51492049f 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -775,3 +775,8 @@ A client receiving such a a response MAY assume that they can skip retrieving the header from a bundle at the indicated URI, and thus save themselves and the server(s) the request(s) needed to inspect the headers of that bundle or bundles. + +bundle-uri SEE ALSO +^^^^^^^^^^^^^^^^^^^ + +See the link:bundle-uri.html[Bundle URI Design Notes] for more. From patchwork Fri Mar 11 16:24:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12778356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05B56C433F5 for ; Fri, 11 Mar 2022 16:24:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350300AbiCKQZ6 (ORCPT ); Fri, 11 Mar 2022 11:25:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37572 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350255AbiCKQZr (ORCPT ); Fri, 11 Mar 2022 11:25:47 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1B4A010856D for ; Fri, 11 Mar 2022 08:24:35 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id r10so13742699wrp.3 for ; Fri, 11 Mar 2022 08:24:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=VkP407ey7sSKihQ8LsbKfHv3jstOQBCZaBhwp07qZIU=; b=NXwnVU51GJzaQT61LEnHs9xU39W5ZNyxjcDXLacr6fs6Q/LTduP5WGtZLrA5YK9zSG KR4GLMC/nE8qKQI3cDV+nM0TGJRq8tp+B6zacaSDX6x+vbKT1goRDtIWjvjkkeX0aONG IHw3ZPyYKZfx2Q2vcTZgvypAqZu1jTCZ32kTxVESvJiqpw2VpCD09+erAVLja4iSjNj4 zZIcSoeHlGtIuimIaUg+gZLgSF1myFJQeaYt09m/3p8+KrSGwkdseEOWXMo+77unwJlM Ph2WsRpvK97i2gVOtoyJJm8npUKFeIAxKoZUQxNj/sP7EskWyO9ty23kxpfS58qxGq+D o5Fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=VkP407ey7sSKihQ8LsbKfHv3jstOQBCZaBhwp07qZIU=; b=4dnDCDLPylxxIhh7gKD/rUpnb3GG4BRTxe+Pmw+8Qay2ao/xskXc2ZOrDCF3uhXFe4 89AWNduQYIj0t7kZkhq+OU3lG5StaNDyHV0x4TizVYjOBDzu65ymYc2QAbQBN/QkJb6i HyRG/yBH8VcBt/1DZrWpHRa8svObjNUS+F1uCDgejuAPak4VjAnE3AIcr/wtzliMxr2S cybAC8r9sG0/8kVU1+E9DWn7AonmzQGooV3ftTVQEtFYMyasJaYBjf6WSP1q7VpRfECL iC9xnwYCAJdi+GXIhtO9iQIGNapSp9lEEqN9Y4kNNXZuF6D7xpKoi+u4TAJgiK+4cA9e N06w== X-Gm-Message-State: AOAM530eqkOmt7v0kOEkpcToVNmCiRo4meRnnvKvnbXeomi4/ixQy3Gu rYIHqrvRKSfmOvlGOZZ5qE34Pa6S4LdCcg== X-Google-Smtp-Source: ABdhPJxzN/Xtmnr9ssvL7iqp5/d+jKlD3LwscWw5KHNy8I6QuK4wLG7lAFVX/vCtvD+Aq+SeRB1Zxw== X-Received: by 2002:adf:dc91:0:b0:1f0:728c:8faf with SMTP id r17-20020adfdc91000000b001f0728c8fafmr8046192wrj.287.1647015873266; Fri, 11 Mar 2022 08:24:33 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id f22-20020a1cc916000000b00380d3e49e89sm7318667wmb.22.2022.03.11.08.24.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:24:32 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Derrick Stolee , Jonathan Tan , Jonathan Nieder , Albert Cui , "Robin H . Johnson" , Teng Long , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH v2 03/13] bundle-uri client: add "bundle-uri" parsing + tests Date: Fri, 11 Mar 2022 17:24:15 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1337.g7e32d794afe In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a "test-tool bundle-uri parse" which parses the format defined in the newly specified "bundle-uri" command. As note in the "bundle-uri" section in protocol-v2.txt we haven't specified any key-values yet, just URI lines, but we should parse their format for conformity with the spec. We need to make sure our future client doesn't die if this optional data is ever provided by the server, and that we've covered all the edge cases with these key-values in our specification. Let's add and test a bundle_uri_parse_line() to do that. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 1 + bundle-uri.c | 124 +++++++++++++++++++++++++++++ bundle-uri.h | 16 ++++ t/helper/test-bundle-uri.c | 83 +++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t5750-bundle-uri-parse.sh | 153 ++++++++++++++++++++++++++++++++++++ 7 files changed, 379 insertions(+) create mode 100644 t/helper/test-bundle-uri.c create mode 100755 t/t5750-bundle-uri-parse.sh diff --git a/Makefile b/Makefile index 5a3d35109a1..4fec8e5af09 100644 --- a/Makefile +++ b/Makefile @@ -696,6 +696,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS)) TEST_BUILTINS_OBJS += test-advise.o TEST_BUILTINS_OBJS += test-bitmap.o TEST_BUILTINS_OBJS += test-bloom.o +TEST_BUILTINS_OBJS += test-bundle-uri.o TEST_BUILTINS_OBJS += test-chmtime.o TEST_BUILTINS_OBJS += test-config.o TEST_BUILTINS_OBJS += test-crontab.o diff --git a/bundle-uri.c b/bundle-uri.c index ff054ddc690..9827fc5da17 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -53,3 +53,127 @@ int bundle_uri_command(struct repository *r, return 0; } + +/** + * General API for {transport,connect}.c etc. + */ +int bundle_uri_parse_line(struct string_list *bundle_uri, const char *line) +{ + size_t i; + struct string_list columns = STRING_LIST_INIT_DUP; + const char *uri; + struct string_list *uri_columns = NULL; + int ret = 0; + + if (!strlen(line)) + return error(_("bundle-uri: got an empty line")); + + /* + * Right now we don't understand anything beyond the first SP, + * but let's be tolerant and ignore any future unknown + * fields. See the "MUST" note about "bundle-feature-key" in + * Documentation/technical/protocol-v2.txt + */ + if (string_list_split(&columns, line, ' ', -1) < 1) + return error(_("bundle-uri: line not in SP-delimited format: %s"), line); + + /* + * We represent a "[ ...]" line with the URI + * being the .string in a string list, and the .util being an + * optional string list of key (.string) and values + * (.util). If the top-level .util is NULL there's no + * key-value pairs.... + */ + uri = columns.items[0].string; + if (!strlen(uri)) { + ret = error(_("bundle-uri: got an empty URI component")); + goto cleanup; + } + + /* + * ... we're going to need that non-NULL .util . + */ + if (columns.nr > 1) { + uri_columns = xcalloc(1, sizeof(struct string_list)); + string_list_init_dup(uri_columns); + } + + /* + * Let's parse the optional "kv" format, even if we don't + * understand any of the keys or values yet. + */ + for (i = 1; i < columns.nr; i++) { + struct string_list kv = STRING_LIST_INIT_DUP; + const char *arg = columns.items[i].string; + int fields = string_list_split(&kv, arg, '=', 2); + int err = 0; + + switch (fields) { + case 0: + BUG("should have no fields=0"); + case 1: + if (!strlen(arg)) { + err = error("bundle-uri: column %lu: got an empty attribute (full line was '%s')", + i, line); + break; + } + /* + * We could dance around with + * string_list_append_nodup() and skip + * string_list_clear(&kv, 0) here, but let's + * keep it simple. + */ + string_list_append(uri_columns, arg); + break; + case 2: + { + const char *k = kv.items[0].string; + const char *v = kv.items[1].string; + + string_list_append(uri_columns, k)->util = xstrdup(v); + break; + } + default: + err = error("bundle-uri: column %lu: '%s' more than one '=' character (full line was '%s')", + i, arg, line); + break; + } + + string_list_clear(&kv, 0); + if (err) { + ret = err; + break; + } + } + + + /* + * Per the spec we'll only consider bundle-uri lines OK if + * there were no parsing problems, even if the problems were + * with attributes whose content we don't understand. + */ + if (ret && uri_columns) { + string_list_clear(uri_columns, 1); + free(uri_columns); + } else if (!ret) { + string_list_append(bundle_uri, uri)->util = uri_columns; + } + +cleanup: + string_list_clear(&columns, 0); + return ret; +} + +static void bundle_uri_string_list_clear_cb(void *util, const char *string) +{ + struct string_list *fields = util; + if (!fields) + return; + string_list_clear(fields, 1); + free(fields); +} + +void bundle_uri_string_list_clear(struct string_list *bundle_uri) +{ + string_list_clear_func(bundle_uri, bundle_uri_string_list_clear_cb); +} diff --git a/bundle-uri.h b/bundle-uri.h index 5a7e556a0ba..be6d1df97ff 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -3,6 +3,7 @@ #include "repository.h" #include "pkt-line.h" #include "strbuf.h" +#include "string-list.h" /** * API used by serve.[ch]. @@ -10,4 +11,19 @@ int bundle_uri_advertise(struct repository *r, struct strbuf *value); int bundle_uri_command(struct repository *r, struct packet_reader *request); +/** + * General API for {transport,connect}.c etc. + */ + +/** + * bundle_uri_parse_line() returns 0 when a valid bundle-uri has been + * added to `bundle_uri`, <0 on error. + */ +int bundle_uri_parse_line(struct string_list *bundle_uri, const char *line); + +/** + * Clear the `bundle_uri` list. Just a very thin wrapper on + * string_list_clear(). + */ +void bundle_uri_string_list_clear(struct string_list *bundle_uri); #endif /* BUNDLE_URI_H */ diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c new file mode 100644 index 00000000000..805a86c0130 --- /dev/null +++ b/t/helper/test-bundle-uri.c @@ -0,0 +1,83 @@ +#include "test-tool.h" +#include "parse-options.h" +#include "bundle-uri.h" +#include "strbuf.h" +#include "string-list.h" + +static int cmd__bundle_uri_parse(int argc, const char **argv) +{ + const char *usage[] = { + "test-tool bundle-uri parse util; + + fprintf(stdout, "%s", item->string); + if (!kv) { + fprintf(stdout, "\n"); + continue; + } + for_each_string_list_item(kv_item, kv) { + const char *k = kv_item->string; + const char *v = kv_item->util; + + if (v) + fprintf(stdout, " [kv: %s => %s]", k, v); + else + fprintf(stdout, " [attr: %s]", k); + } + fprintf(stdout, "\n"); + } + strbuf_release(&sb); + + bundle_uri_string_list_clear(&list); + + return err < 0 ? 1 : 0; +usage: + usage_with_options(usage, options); +} + +int cmd__bundle_uri(int argc, const char **argv) +{ + const char *usage[] = { + "test-tool bundle-uri []", + NULL + }; + struct option options[] = { + OPT_END(), + }; + + argc = parse_options(argc, argv, NULL, options, usage, + PARSE_OPT_STOP_AT_NON_OPTION | + PARSE_OPT_KEEP_ARGV0); + if (argc == 1) + goto usage; + + if (!strcmp(argv[1], "parse")) + return cmd__bundle_uri_parse(argc - 1, argv + 1); + error("there is no test-tool bundle-uri tool '%s'", argv[1]); + +usage: + usage_with_options(usage, options); +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index e6ec69cf326..dc73e68f329 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -17,6 +17,7 @@ static struct test_cmd cmds[] = { { "advise", cmd__advise_if_enabled }, { "bitmap", cmd__bitmap }, { "bloom", cmd__bloom }, + { "bundle-uri", cmd__bundle_uri }, { "chmtime", cmd__chmtime }, { "config", cmd__config }, { "crontab", cmd__crontab }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 20756eefdda..927b6b418cd 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -7,6 +7,7 @@ int cmd__advise_if_enabled(int argc, const char **argv); int cmd__bitmap(int argc, const char **argv); int cmd__bloom(int argc, const char **argv); +int cmd__bundle_uri(int argc, const char **argv); int cmd__chmtime(int argc, const char **argv); int cmd__config(int argc, const char **argv); int cmd__crontab(int argc, const char **argv); diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh new file mode 100755 index 00000000000..70fd1b398e9 --- /dev/null +++ b/t/t5750-bundle-uri-parse.sh @@ -0,0 +1,153 @@ +#!/bin/sh + +test_description="Test bundle-uri bundle_uri_parse_line()" + +TEST_NO_CREATE_REPO=1 +TEST_PASSES_SANITIZE_LEAK=true +. ./test-lib.sh + +test_expect_success 'bundle_uri_parse_line() just URIs' ' + cat >in <<-\EOF && + http://example.com/bundle.bdl + https://example.com/bundle.bdl + file:///usr/share/git/bundle.bdl + EOF + + # For the simple case + cp in expect && + + test-tool bundle-uri parse actual 2>err && + test_must_be_empty err && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() with attributes' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl attr + http://example.com/bundle2.bdl ibute + EOF + + cat >expect <<-\EOF && + http://example.com/bundle1.bdl [attr: attr] + http://example.com/bundle2.bdl [attr: ibute] + EOF + + test-tool bundle-uri parse actual 2>err && + test_must_be_empty err && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() with attributes and key-value attributes' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl x a=b y c=d z e=f a=b + EOF + + + cat >expect <<-\EOF && + http://example.com/bundle1.bdl [attr: x] [kv: a => b] [attr: y] [kv: c => d] [attr: z] [kv: e => f] [kv: a => b] + EOF + + test-tool bundle-uri parse actual 2>err && + test_must_be_empty err && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: extra SP' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl one-space + http://example.com/bundle2.bdl two-space + http://example.com/bundle3.bdl three-space + EOF + + cat >err.expect <<-\EOF && + error: bundle-uri: column 1: got an empty attribute (full line was '\''http://example.com/bundle2.bdl two-space'\'') + error: bad line: '\''http://example.com/bundle2.bdl two-space'\'' + error: bundle-uri: column 1: got an empty attribute (full line was '\''http://example.com/bundle3.bdl three-space'\'') + error: bad line: '\''http://example.com/bundle3.bdl three-space'\'' + EOF + + cat >expect <<-\EOF && + http://example.com/bundle1.bdl [attr: one-space] + EOF + + test_must_fail test-tool bundle-uri parse actual 2>err.actual && + test_cmp err.expect err.actual && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty lines' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl + + http://example.com/bundle2.bdl a=b + + http://example.com/bundle3.bdl + EOF + + cat >err.expect <<-\EOF && + error: bundle-uri: got an empty line + error: bad line: '\'''\'' + error: bundle-uri: got an empty line + error: bad line: '\'''\'' + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + http://example.com/bundle1.bdl + http://example.com/bundle2.bdl [kv: a => b] + http://example.com/bundle3.bdl + EOF + + test_must_fail test-tool bundle-uri parse actual 2>err.actual && + test_cmp err.expect err.actual && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty URIs' ' + sed "s/> //" >in <<-\EOF && + http://example.com/bundle1.bdl + > a=b + http://example.com/bundle3.bdl a=b + EOF + + cat >err.expect <<-\EOF && + error: bundle-uri: got an empty URI component + error: bad line: '\'' a=b'\'' + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + http://example.com/bundle1.bdl + http://example.com/bundle3.bdl [kv: a => b] + EOF + + test_must_fail test-tool bundle-uri parse actual 2>err.actual && + test_cmp err.expect err.actual && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: multiple = in key-values' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl k=v=extra + http://example.com/bundle2.bdl a=b k=v=extra c=d + http://example.com/bundle3.bdl kv=ok + EOF + + cat >err.expect <<-\EOF && + error: bundle-uri: column 1: '\''k=v=extra'\'' more than one '\''='\'' character (full line was '\''http://example.com/bundle1.bdl k=v=extra'\'') + error: bad line: '\''http://example.com/bundle1.bdl k=v=extra'\'' + error: bundle-uri: column 2: '\''k=v=extra'\'' more than one '\''='\'' character (full line was '\''http://example.com/bundle2.bdl a=b k=v=extra c=d'\'') + error: bad line: '\''http://example.com/bundle2.bdl a=b k=v=extra c=d'\'' + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + http://example.com/bundle3.bdl [kv: kv => ok] + EOF + + test_must_fail test-tool bundle-uri parse actual 2>err.actual && + test_cmp err.expect err.actual && + test_cmp expect actual +' + +test_done From patchwork Fri Mar 11 16:24:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12778352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35098C433FE for ; Fri, 11 Mar 2022 16:24:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350278AbiCKQZv (ORCPT ); Fri, 11 Mar 2022 11:25:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37574 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350254AbiCKQZr (ORCPT ); Fri, 11 Mar 2022 11:25:47 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E2702108761 for ; Fri, 11 Mar 2022 08:24:35 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id u1so13694062wrg.11 for ; Fri, 11 Mar 2022 08:24:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=njeOfpQ5Pl+tiJg5qLKUagkvcUMPqLWuYzEZN9f+OTQ=; b=AT1WtpV51AyciAdvpNTQ+Gp3UdQ7pPyz0E6LafW5c2YPK0VkbpekUHVYuzR2FyT6/6 DMN79RXkHGkGlYTAwxSC2ErD8bNSqFTZqUVMk+7rRZprLgd39m6G/hMXqBKgsE144a8M Y1ODGkvRtK34sSRBYzdx+iyxys9ZWszIvRy3truCsJkhHLV+rBv1HTIUXBzgasF1uxo2 rG9ml4B3V5ckRgLOoCmmiXLBuuLTxPrLzGTsWy3XP79L2TkYetXeR9i3sB1NFPi8XK36 pclic9Nv9mGvq1+0ayeKVa7cmudQrQP9AViVZdMqJc4Vr7x8S/vv51lN+CxQS0AL1nro YnkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=njeOfpQ5Pl+tiJg5qLKUagkvcUMPqLWuYzEZN9f+OTQ=; b=glRKxlSr58AJfqofZ+js7n4Vx1qjXHs0+LPCTsJXjmv75DqdDZC2NYesX1g7vtVui3 jX6NlRbBiPiQT+8+wYWAuG8dmEBC2NnOsM1CsgeVzG9uM+ZQkqUn2S0cgp0byBNN7VmZ 4J5NS47oYceoV9D8pTOivHDjN341/n7QEZpIOgycMYwmrCREeAiIFbRtn0xz5FsBL5Qp bs8iyFC+5gr8NFDz/4KW87nDuPMP8oAuKR+TT3R44o1FI00MXc8xO0IkmsQnKDGmp1Vi WHjdqWhxJnhWSuLjl46NBD6HMRVbZ++ht31MrnH+LwGlALscfpYlWOHxQtPVLJZp38a7 w1gw== X-Gm-Message-State: AOAM530Tchb/4L8k8UtSj4X8q2hT5NXOvYkoWOmwvckovsxWkYa9algL fKr6+7Ymr3b884/T9WBfIZojA/TeHp8spg== X-Google-Smtp-Source: ABdhPJxmk10Y6o7VLAebtOBt1ocwnwNV2dWOP44o7xUm6y35dz7wE/FG/k0dqIGm3xUgd16VPo+5Fw== X-Received: by 2002:adf:b74b:0:b0:1ed:e1d3:e053 with SMTP id n11-20020adfb74b000000b001ede1d3e053mr7986395wre.131.1647015874252; Fri, 11 Mar 2022 08:24:34 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id f22-20020a1cc916000000b00380d3e49e89sm7318667wmb.22.2022.03.11.08.24.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:24:33 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Derrick Stolee , Jonathan Tan , Jonathan Nieder , Albert Cui , "Robin H . Johnson" , Teng Long , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH v2 04/13] connect.c: refactor sending of agent & object-format Date: Fri, 11 Mar 2022 17:24:16 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1337.g7e32d794afe In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor the sending of the "agent" and "object-format" capabilities into a function. This was added in its current form in ab67235bc4 (connect: parse v2 refs with correct hash algorithm, 2020-05-25). When we connect to a v2 server we need to know about its object-format, and it needs to know about ours. Since most things in connect.c and transport.c piggy-back on the eager getting of remote refs via the handshake() those commands can make use of the just-sent-over object-format by ls-refs. But I'm about to add a command that may come after ls-refs, and may not, but we need the server to know about our user-agent and object-format. So let's split this into a function. Signed-off-by: Ævar Arnfjörð Bjarmason --- connect.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/connect.c b/connect.c index afc79a6236e..e6d0b1d34bd 100644 --- a/connect.c +++ b/connect.c @@ -473,6 +473,24 @@ void check_stateless_delimiter(int stateless_rpc, die("%s", error); } +static void send_capabilities(int fd_out, struct packet_reader *reader) +{ + const char *hash_name; + + if (server_supports_v2("agent", 0)) + packet_write_fmt(fd_out, "agent=%s", git_user_agent_sanitized()); + + if (server_feature_v2("object-format", &hash_name)) { + int hash_algo = hash_algo_by_name(hash_name); + if (hash_algo == GIT_HASH_UNKNOWN) + die(_("unknown object format '%s' specified by server"), hash_name); + reader->hash_algo = &hash_algos[hash_algo]; + packet_write_fmt(fd_out, "object-format=%s", reader->hash_algo->name); + } else { + reader->hash_algo = &hash_algos[GIT_HASH_SHA1]; + } +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, @@ -480,7 +498,6 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, int stateless_rpc) { int i; - const char *hash_name; struct strvec *ref_prefixes = transport_options ? &transport_options->ref_prefixes : NULL; const char **unborn_head_target = transport_options ? @@ -490,18 +507,8 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, if (server_supports_v2("ls-refs", 1)) packet_write_fmt(fd_out, "command=ls-refs\n"); - if (server_supports_v2("agent", 0)) - packet_write_fmt(fd_out, "agent=%s", git_user_agent_sanitized()); - - if (server_feature_v2("object-format", &hash_name)) { - int hash_algo = hash_algo_by_name(hash_name); - if (hash_algo == GIT_HASH_UNKNOWN) - die(_("unknown object format '%s' specified by server"), hash_name); - reader->hash_algo = &hash_algos[hash_algo]; - packet_write_fmt(fd_out, "object-format=%s", reader->hash_algo->name); - } else { - reader->hash_algo = &hash_algos[GIT_HASH_SHA1]; - } + /* Send capabilities */ + send_capabilities(fd_out, reader); if (server_options && server_options->nr && server_supports_v2("server-option", 1)) From patchwork Fri Mar 11 16:24:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12778354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 207FFC433EF for ; Fri, 11 Mar 2022 16:24:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350256AbiCKQZy (ORCPT ); Fri, 11 Mar 2022 11:25:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37576 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350261AbiCKQZr (ORCPT ); Fri, 11 Mar 2022 11:25:47 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2CC7310A7CC for ; Fri, 11 Mar 2022 08:24:37 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id r10so13742864wrp.3 for ; Fri, 11 Mar 2022 08:24:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Y62alKyHdEQTCNrRjgFAPKPu891DwMg9ckjj4cSa4Ds=; b=VOv4tcNZARqf503UBuipGOUztP5DV23aQcUGysWA96YPP1oeOj3uWJwdtD1cfAbiX0 lmuM04VNyREDiDx3qup00xXlX0plct9zp/JmiWknt6y1YhpV+oGe5wnF//B6Mip1PaQ4 0VXC/ua6NkBO4J+HIQwYauLCBZ5M6S48rXwdFDwv8eBhz8f57QI1SDOm0CFHKUv6gvO9 BhitL1h2NfVNya0z7KDLcwARV2HEz2euM2DYbSoy8ss5E/MmEp9hgPzKxwzI7m/F9CUB Kk0PpRHn548IgH/rBVeEH7RGnzPdV7O2NFedkYxwT/yW+s9q5/vWCyfOiHHoESDdURX3 P7mw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Y62alKyHdEQTCNrRjgFAPKPu891DwMg9ckjj4cSa4Ds=; b=IB4R6qYwkGVL/ksIU6wPvgjHUYsHRuz8xsIBP6GHp7B8z2tPUZu9iL3GWq1AZAvsEo MgQXEAoTVLQ4xoT3LP2zTD2m/sdqGJVrXSqaFyysTRC70Ulm7FqslCWZy1TwosmCwM2u WJx2qiiiywbShFU27t9MXEYh3egupBMGbz60ou2urvV6ol3pwyQSmxD+EBeFSTtlCW0c 2N1mWRarTWrNwNXAG5E20H51VuZCcC8Ldu2invU4YKw0t0PLrnfei85rS1dXxVn2UVdN P55//5yQh60s235K67eKU9eebgX+QZ4l/jmeiTSY/zBDxNOx9iaDLprqJzfgNkrwRm02 BAiA== X-Gm-Message-State: AOAM533/fkJm8+JdIyyrdrVlW7Xouaak0kj9zdx6IJ3BPKefXG6sKEcN MR2UR9xnL+XPtPWxZ5+LYNJTJSogmZeyKw== X-Google-Smtp-Source: ABdhPJwEdlRwbqMR3Kr4fIoCk4iTJMXrUPOIJdg17k7WeoqmHikTqaROUowIqnQo7+jdtGWrpSBdYw== X-Received: by 2002:adf:fa45:0:b0:203:954d:d5e3 with SMTP id y5-20020adffa45000000b00203954dd5e3mr3171190wrr.533.1647015875151; Fri, 11 Mar 2022 08:24:35 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id f22-20020a1cc916000000b00380d3e49e89sm7318667wmb.22.2022.03.11.08.24.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:24:34 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Derrick Stolee , Jonathan Tan , Jonathan Nieder , Albert Cui , "Robin H . Johnson" , Teng Long , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH v2 05/13] bundle-uri client: add minimal NOOP client Date: Fri, 11 Mar 2022 17:24:17 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1337.g7e32d794afe In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Set up all the needed client parts of the "bundle-uri" protocol extension, without actually doing anything with the bundle URIs. I.e. if the server says it supports "bundle-uri" we'll issue a command=bundle-uri after command=ls-refs when we're cloning. We'll parse the returned output using the code already tested for in t5750-bundle-uri-parse.sh. What we aren't doing is actually acting on that data, i.e. downloading the bundle(s) before we get to doing the command=fetch, and adjusting our negotiation dialog appropriately. I'll do that in subsequent commits. There's a question of what level of encapsulation we should use here, I've opted to use connect.h in clone.c, but we could also e.g. make transport_get_remote_refs() invoke this, i.e. make it implicitly get the bundle-uri list for later steps. This approach means that we don't "support" this in "git fetch" for now. I'm starting with the case of initial clones, although as noted in preceding commits to the protocol documentation nothing about this approach precludes getting bundles on incremental fetches. For the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set environment variables for git-upload-pack (it's started by Apache), so let's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT} prerequisites for consistency and future use. Signed-off-by: Ævar Arnfjörð Bjarmason --- builtin/clone.c | 7 ++ bundle-uri.c | 4 + connect.c | 47 +++++++++ remote.h | 4 + t/lib-t5730-protocol-v2-bundle-uri.sh | 141 +++++++++++++++++++++++++ t/t5730-protocol-v2-bundle-uri-file.sh | 36 +++++++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++ t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++ transport-helper.c | 13 +++ transport-internal.h | 7 ++ transport.c | 48 +++++++++ transport.h | 18 ++++ 12 files changed, 359 insertions(+) create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh diff --git a/builtin/clone.c b/builtin/clone.c index a572cda5030..b2c1b4142ee 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -27,6 +27,7 @@ #include "iterator.h" #include "sigchain.h" #include "branch.h" +#include "connect.h" #include "remote.h" #include "run-command.h" #include "connected.h" @@ -1220,6 +1221,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/bundle-uri.c b/bundle-uri.c index 9827fc5da17..c503ed51ca8 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -26,6 +26,10 @@ static int bundle_uri_config(const char *var, const char *value, void *data) int bundle_uri_advertise(struct repository *r, struct strbuf *value) { + if (value && + git_env_bool("GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE", 0)) + strbuf_addstr(value, "test-unknown-capability-value"); + if (advertise_bundle_uri != -1) goto cached; diff --git a/connect.c b/connect.c index e6d0b1d34bd..a8fdb5255f7 100644 --- a/connect.c +++ b/connect.c @@ -15,6 +15,7 @@ #include "version.h" #include "protocol.h" #include "alias.h" +#include "bundle-uri.h" static char *server_capabilities_v1; static struct strvec server_capabilities_v2 = STRVEC_INIT; @@ -491,6 +492,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader) } } +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct string_list *bundle_uri, int stateless_rpc) +{ + int line_nr = 1; + + /* Assert bundle-uri support */ + server_supports_v2("bundle-uri", 1); + + /* (Re-)send capabilities */ + send_capabilities(fd_out, reader); + + /* Send command */ + packet_write_fmt(fd_out, "command=bundle-uri\n"); + packet_delim(fd_out); + + /* Send options */ + if (git_env_bool("GIT_TEST_PROTOCOL_BAD_BUNDLE_URI", 0)) + packet_write_fmt(fd_out, "test-bad-client\n"); + packet_flush(fd_out); + + /* Process response from server */ + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { + const char *line = reader->line; + line_nr++; + + if (!bundle_uri_parse_line(bundle_uri, line)) + continue; + + return error(_("error on bundle-uri response line %d: %s"), + line_nr, line); + } + + if (reader->status != PACKET_READ_FLUSH) + return error(_("expected flush after bundle-uri listing")); + + /* + * Might die(), but obscure enough that that's OK, e.g. in + * serve.c we'll call BUG() on its equivalent (the + * PACKET_READ_RESPONSE_END check). + */ + check_stateless_delimiter(stateless_rpc, reader, + _("expected response end packet after ref listing")); + + return 0; +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, diff --git a/remote.h b/remote.h index 4a1209ae2c8..ca820b60948 100644 --- a/remote.h +++ b/remote.h @@ -236,6 +236,10 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, const struct string_list *server_options, int stateless_rpc); +/* Used for protocol v2 in order to retrieve refs from a remote */ +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct string_list *bundle_uri, int stateless_rpc); + int resolve_remote_symref(struct ref *ref, struct ref *list); /* diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh new file mode 100644 index 00000000000..724acecda66 --- /dev/null +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -0,0 +1,141 @@ +# Included from t573*-protocol-v2-bundle-uri-*.sh + +T5730_PARENT= +T5730_URI= +T5730_BUNDLE_URI= +case "$T5730_PROTOCOL" in +file) + T5730_PARENT=file_parent + T5730_URI="file://$PWD/file_parent" + T5730_BUNDLE_URI="$T5730_URI/fake.bdl" + test_set_prereq T5730_FILE + ;; +git) + . "$TEST_DIRECTORY"/lib-git-daemon.sh + start_git_daemon --export-all --enable=receive-pack + T5730_PARENT="$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent" + T5730_URI="$GIT_DAEMON_URL/parent" + T5730_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq T5730_GIT + ;; +http) + . "$TEST_DIRECTORY"/lib-httpd.sh + start_httpd + T5730_PARENT="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" + T5730_URI="$HTTPD_URL/smart/http_parent" + T5730_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq T5730_HTTP + ;; +*) + BUG "Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)" + ;; +esac + +test_expect_success "setup protocol v2 $T5730_PROTOCOL:// tests" ' + git init "$T5730_PARENT" && + test_commit -C "$T5730_PARENT" one +' + +# Poor man's URI escaping. Good enough for the test suite whose trash +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art. +test_uri_escape() { + sed 's/ /%20/g' +} + +case "$T5730_PROTOCOL" in +http) + test_expect_success "setup config for $T5730_PROTOCOL:// tests" ' + git -C "$T5730_PARENT" config http.receivepack true + ' + ;; +*) + ;; +esac +T5730_BUNDLE_URI_ESCAPED=$(echo "$T5730_BUNDLE_URI" | test_uri_escape) + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri" ' + test_when_finished "rm -f log" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + ! grep bundle-uri log +' + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep bundle-uri log +' + +test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" uploadpack.bundleURI \ + "$T5730_BUNDLE_URI_ESCAPED" && + + cat >err.expect <<-\EOF && + Cloning into '"'"'child'"'"'... + EOF + case "$T5730_PROTOCOL" in + file) + cat >fatal-bundle-uri.expect <<-\EOF + fatal: bundle-uri: unexpected argument: '"'"'test-bad-client'"'"' + EOF + ;; + *) + cat >fatal.expect <<-\EOF + fatal: read error: Connection reset by peer + EOF + ;; + esac && + + test_when_finished "rm -rf child" && + test_must_fail ok=sigpipe env \ + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \ + git -c protocol.version=2 \ + clone "$T5730_URI" child \ + >out 2>err && + test_must_be_empty out && + + grep -v -e ^fatal: -e ^error: err >err.actual && + test_cmp err.expect err.actual && + + case "$T5730_PROTOCOL" in + file) + # Due to general race conditions with client/server replies we + # may or may not get "fatal: the remote end hung up + # expectedly" here + grep "^fatal: bundle-uri:" err >fatal-bundle-uri.actual && + test_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual + ;; + *) + grep "^fatal:" err >fatal.actual && + test_cmp fatal.expect fatal.actual + ;; + esac && + + grep "clone> test-bad-client$" log >sent-bad-request && + test_file_not_empty sent-bad-request +' diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh new file mode 100755 index 00000000000..89203d3a23c --- /dev/null +++ b/t/t5730-protocol-v2-bundle-uri-file.sh @@ -0,0 +1,36 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'file://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'file://' transport +# +T5730_PROTOCOL=file +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_expect_success "unknown capability value with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + grep "> bundle-uri=test-unknown-capability-value" log +' + +test_done diff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh new file mode 100755 index 00000000000..282847b311f --- /dev/null +++ b/t/t5731-protocol-v2-bundle-uri-git.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=git +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh new file mode 100755 index 00000000000..fcc1cf3faef --- /dev/null +++ b/t/t5732-protocol-v2-bundle-uri-http.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=http +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/transport-helper.c b/transport-helper.c index a0297b0986c..4ddf2a4be2a 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1264,9 +1264,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport, return ret; } +static int get_bundle_uri(struct transport *transport) +{ + get_helper(transport); + + if (process_connect(transport, 0)) { + do_take_over(transport); + return transport->vtable->get_bundle_uri(transport); + } + + return -1; +} + static struct transport_vtable vtable = { .set_option = set_helper_option, .get_refs_list = get_refs_list, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs, .push_refs = push_refs, .connect = connect_helper, diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733ac..90ea749e5cf 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -26,6 +26,13 @@ struct transport_vtable { struct ref *(*get_refs_list)(struct transport *transport, int for_push, struct transport_ls_refs_options *transport_options); + /** + * Populates the remote side's bundle-uri under protocol v2, + * if the "bundle-uri" capability was advertised. Returns 0 if + * OK, negative values on error. + */ + int (*get_bundle_uri)(struct transport *transport); + /** * Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. diff --git a/transport.c b/transport.c index 253d6671b1f..7c9a371ed7f 100644 --- a/transport.c +++ b/transport.c @@ -22,6 +22,7 @@ #include "protocol.h" #include "object-store.h" #include "color.h" +#include "bundle-uri.h" static int transport_use_color = -1; static char transport_colors[][COLOR_MAXLEN] = { @@ -349,6 +350,21 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus return handshake(transport, for_push, options, 1); } +static int get_bundle_uri(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct packet_reader reader; + int stateless_rpc = transport->stateless_rpc; + string_list_init_dup(&transport->bundle_uri); + + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF); + + return get_remote_bundle_uri(data->fd[1], &reader, + &transport->bundle_uri, stateless_rpc); +} + static int fetch_refs_via_pack(struct transport *transport, int nr_heads, struct ref **to_fetch) { @@ -888,6 +904,7 @@ static int disconnect_git(struct transport *transport) static struct transport_vtable taken_over_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .disconnect = disconnect_git @@ -1041,6 +1058,7 @@ static struct transport_vtable bundle_vtable = { static struct transport_vtable builtin_smart_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .connect = connect_git, @@ -1054,6 +1072,7 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->progress = isatty(2); string_list_init_dup(&ret->pack_lockfiles); + string_list_init_dup(&ret->bundle_uri); if (!remote) BUG("No remote provided to transport_get()"); @@ -1462,6 +1481,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } +int transport_get_remote_bundle_uri(struct transport *transport) +{ + const struct transport_vtable *vtable = transport->vtable; + + /* Lazily configured */ + if (transport->got_remote_bundle_uri++) + return 0; + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + + /* + * This is intentionally below the transport.injectBundleURI, + * we want to be able to inject into protocol v0, or into the + * dialog of a server who doesn't support this. + */ + if (!vtable->get_bundle_uri) + return error(_("bundle-uri operation not supported by protocol")); + + if (vtable->get_bundle_uri(transport) < 0) + return error(_("could not retrieve server-advertised bundle-uri list")); + return 0; +} + void transport_unlock_pack(struct transport *transport, unsigned int flags) { int in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER); @@ -1492,6 +1539,7 @@ int transport_disconnect(struct transport *transport) ret = transport->vtable->disconnect(transport); if (transport->got_remote_refs) free_refs((void *)transport->remote_refs); + bundle_uri_string_list_clear(&transport->bundle_uri); free(transport); return ret; } diff --git a/transport.h b/transport.h index a0bc6a1e9eb..7740306b850 100644 --- a/transport.h +++ b/transport.h @@ -75,6 +75,18 @@ struct transport { */ unsigned got_remote_refs : 1; + /** + * Indicates whether we already called get_bundle_uri_list(); set by + * transport.c::transport_get_remote_bundle_uri(). + */ + unsigned got_remote_bundle_uri : 1; + + /* + * The results of "command=bundle-uri", if both sides support + * the "bundle-uri" capability. + */ + struct string_list bundle_uri; + /* * Transports that call take-over destroys the data specific to * the transport type while doing so, and cannot be reused. @@ -276,6 +288,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts); const struct ref *transport_get_remote_refs(struct transport *transport, struct transport_ls_refs_options *transport_options); +/** + * Retrieve bundle URI(s) from a remote. Populates "struct + * transport"'s "bundle_uri" and "got_remote_bundle_uri". + */ +int transport_get_remote_bundle_uri(struct transport *transport); + /* * Fetch the hash algorithm used by a remote. * From patchwork Fri Mar 11 16:24:18 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12778353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8941EC433F5 for ; Fri, 11 Mar 2022 16:24:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350283AbiCKQZx (ORCPT ); Fri, 11 Mar 2022 11:25:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37578 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350258AbiCKQZr (ORCPT ); Fri, 11 Mar 2022 11:25:47 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A67D210C511 for ; Fri, 11 Mar 2022 08:24:38 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id j17so13796565wrc.0 for ; Fri, 11 Mar 2022 08:24:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JLd2YVw5tmx/BJCz73mvp3eFaJ4ngzpmjCzNGRitZkw=; b=gUm2ZQQhsM/zYIO3uCjHHXT3DIR+3DmDn6P3H9uYGh56/sLKnuQ5gfkOCTtD4sMcdW Zis040K8EShgGyiuWIFPGp0EvFIxRrgPmD8BuDaZiGNfU/iDk1X1kVgbNBteGPQxs3Vh zlnLg3uH3VtZrSRMxQqzEyhpnuMl9q6m3Bjx4Al/Sl8K3K5LaR4LEYib1nE1aSEmRXGH Kk9YS9kttYzlrSe+KY/mn435JtMzqYvhznqSdgc1UodWAWqEStQR5eRzy+C7HMf2+PT+ 7eUO0KVDD+H/deB4QPbEL+JcHRPufInl+WazMzZQKWomMhsfkL+JJnpqcUE4gYkyMZQt WWpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JLd2YVw5tmx/BJCz73mvp3eFaJ4ngzpmjCzNGRitZkw=; b=Uj4GWVbGX4/cs4IK3qITFGLJZEQeZtdwt+V89ThozVA33jLlWR6hq/asGtEp+XxPzo B2BmbTZo+fzs4Y12JvGZSj2nMBQojVNNCSbmBlP4c3kTgfwXf66GPyt85/rtBOhZ/mH/ dxxRQpzB+z1r4HnoviB3xdHVQ77LG3xAx8KwKgiKNS+Ch1M9OGTfpBrCorY+k8g+VLLY KQSpQiwz+nnu2M/m9FIlLYRpHwUa2qkKmX565cKhceBU2xgxzvIA2cjHebJGy+1ZWKWk pztGPueaPFjZImdIV1rVyByTjVJ1AtpR72sjUZyyesq+PRGnYQCKtyLVgPwuiOaJ03F1 rDIQ== X-Gm-Message-State: AOAM532Qsq+S9AVkcAKzFyhzKrUDgkEmgf+vdJ2NLMf97QmjHvHv1O37 fnz40qqCmtROF6sg+rDt6Of1o3cdPPjL3w== X-Google-Smtp-Source: ABdhPJzy6PAD+9FEeZhhB1m3TpSSi9rqMqadSPSDbY60bgdntf7ESVp/r8wE6LqeBc5tSGvem8wJzA== X-Received: by 2002:adf:dd8e:0:b0:1f0:22f1:aed8 with SMTP id x14-20020adfdd8e000000b001f022f1aed8mr7917776wrl.269.1647015876651; Fri, 11 Mar 2022 08:24:36 -0800 (PST) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id f22-20020a1cc916000000b00380d3e49e89sm7318667wmb.22.2022.03.11.08.24.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Mar 2022 08:24:35 -0800 (PST) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Junio C Hamano , Derrick Stolee , Jonathan Tan , Jonathan Nieder , Albert Cui , "Robin H . Johnson" , Teng Long , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH v2 06/13] bundle-uri client: add "git ls-remote-bundle-uri" Date: Fri, 11 Mar 2022 17:24:18 +0100 Message-Id: X-Mailer: git-send-email 2.35.1.1337.g7e32d794afe In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a git-ls-remote-bundle-uri command, this is a thin wrapper for issuing protocol v2 "bundle-uri" commands to a server, and to the parsing routines in bundle-uri.c. Since in the "git clone" case we'll have already done the handshake(), but not here, introduce a "got_advertisement" state along with "got_remote_heads". It seems to me that the "got_remote_heads" is badly named in the first place, and the whole logic of eagerly getting ls-refs on handshake() or not could be refactored somewhat, but let's not do that now, and instead just add another self-documenting state variable. Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/git-ls-remote-bundle-uri.txt | 62 ++++++++++ Documentation/git-ls-remote.txt | 1 + Makefile | 1 + builtin.h | 1 + builtin/clone.c | 2 +- builtin/ls-remote-bundle-uri.c | 90 ++++++++++++++ command-list.txt | 1 + git.c | 1 + t/lib-t5730-protocol-v2-bundle-uri.sh | 132 +++++++++++++++++++++ transport.c | 43 +++++-- transport.h | 6 +- 11 files changed, 329 insertions(+), 11 deletions(-) create mode 100644 Documentation/git-ls-remote-bundle-uri.txt create mode 100644 builtin/ls-remote-bundle-uri.c diff --git a/Documentation/git-ls-remote-bundle-uri.txt b/Documentation/git-ls-remote-bundle-uri.txt new file mode 100644 index 00000000000..793d7677f2f --- /dev/null +++ b/Documentation/git-ls-remote-bundle-uri.txt @@ -0,0 +1,62 @@ +git-ls-remote-bundle-uri(1) +=========================== + +NAME +---- +git-ls-remote-bundle-uri - List 'bundle-uri' in a remote repository + +SYNOPSIS +-------- +[verse] +'git ls-remote-bundle-uri' [-q |--quiet] [--uri] [--upload-pack=] + [[-o | --server-option=]