From patchwork Tue Nov 1 01:07:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13026565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFCF4FA3740 for ; Tue, 1 Nov 2022 01:07:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229782AbiKABHo (ORCPT ); Mon, 31 Oct 2022 21:07:44 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49724 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229469AbiKABHk (ORCPT ); Mon, 31 Oct 2022 21:07:40 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 761031658D for ; Mon, 31 Oct 2022 18:07:38 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id j15so18260626wrq.3 for ; Mon, 31 Oct 2022 18:07:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=oNoP/XiagNYJqY1gtmMDna01LlRJft9mBpsTdeWfLTk=; b=bECG95dI6ZL3I2Sj62xHV2TSOrx1WEvmESXZ2rcosluHA+rJCGJ5ZK0s57lM/JgzW/ gatknl0cXuITybqyfYxKNQaxygoNZJRBVjlrFUo0vaZXTsDDfl7Jx4Fh9Wpf3N/E+O1d VhbcMT6eMkywUf9n8ukHUGSpz8cjh+dl+AkFYHb3i4uFw96YL2Om0z3JN0dL8bVX0zZ6 Io0zRwa8IZY+41t07hzY34iSANiqTbBQeunT7QRcAwWUFoyyaA/c2Md0vvH7p0PgmjZm gkTPfMeCMGThCHl7X/N4yicE6GlLRYGPg71u3A+8hcJxoQfDsW2s+LqAQm7NDwt/o0C9 UMUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oNoP/XiagNYJqY1gtmMDna01LlRJft9mBpsTdeWfLTk=; b=Sxfnng4r5Wo7MJ4hVG/bhLjpyVlQVJ0K4mqKutg2g4+UGG28rQcgkzO+5IiONtiE+n mXvEMmS8R78QwXwPkl/k99SqWHZ0ENNV4xUUyQ7j14zE+Wjo4eReRaY8foXfcoQgSd6R RXSzjO4YzdhUXInApr2IwziDsBgekFSbfOmPtLCN8Iol2hUgR26HgHnKU8lIhvq/vYKO NCt+ebudXjLdqWUgiXwT7UDP/JCp5aj0ocQrxdIImDxZ6gsHKAoaN+b2PZULRYjoAxEx FkAalmx/pkfMPHlpMWSO0kReTEEWM4tkWBS/kkWCuURxw/N0SGvfnsL66GD3tHCjRH4Q t9PA== X-Gm-Message-State: ACrzQf2AQoo/hlr/YHYzqRnwcgVm53r+LMXQFLH8nvDaapAGiBzQelsi ap7fNPH+k3le1HixZn2TA0ysjAeS9RQ= X-Google-Smtp-Source: AMsMyM6yc5eijpgdCqqyNB1Sz+YKD5Z6KKvYOVzRg1XdWngVw2AUt7dq90CeiPhK0iaVR8frc4e54Q== X-Received: by 2002:a5d:64e9:0:b0:22e:7631:bcab with SMTP id g9-20020a5d64e9000000b0022e7631bcabmr10207586wri.36.1667264856604; Mon, 31 Oct 2022 18:07:36 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ck7-20020a5d5e87000000b00235da296623sm8576383wrb.31.2022.10.31.18.07.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:36 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:26 +0000 Subject: [PATCH 1/9] protocol v2: add server-side "bundle-uri" skeleton MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 verb with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/gitprotocol-v2.txt | 193 +++++++++++++++++++++++++++++++ bundle-uri.c | 36 ++++++ bundle-uri.h | 7 ++ serve.c | 6 + t/t5701-git-serve.sh | 40 ++++++- 5 files changed, 281 insertions(+), 1 deletion(-) diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt index 59bf41cefb9..57642b4a415 100644 --- a/Documentation/gitprotocol-v2.txt +++ b/Documentation/gitprotocol-v2.txt @@ -578,6 +578,199 @@ and associated requested information, each separated by a single space. obj-info = obj-id SP obj-size +bundle-uri +~~~~~~~~~~ + +If the 'bundle-uri' capability is advertised, the server supports the +`bundle-uri' command. + +The capability is currently advertised with no value (i.e. not +"bundle-uri=somevalue"), a value may be added in the future for +supporting command-wide extensions. Clients MUST ignore any unknown +capability values and proceed with the 'bundle-uri` dialog they +support. + +The 'bundle-uri' command is intended to be issued before `fetch` to +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and +inform the subsequent `fetch` command. + +The client CAN issue `bundle-uri` before or after any other valid +command. To be useful to clients it's expected that it'll be issued +after an `ls-refs` and before `fetch`, but CAN be issued at any time +in the dialog. + +DISCUSSION of bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^^ + +The intent of the feature is optimize for server resource consumption +in the common case by changing the common case of fetching a very +large PACK during linkgit:git-clone[1] into a smaller incremental +fetch. + +It also allows servers to achieve better caching in combination with +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]). + +By having new clones or fetches be a more predictable and common +negotiation against the tips of recently produces *.bundle file(s). +Servers might even pre-generate the results of such negotiations for +the `uploadpack.packObjectsHook` as new pushes come in. + +One way that servers could take advantage of these bundles is that the +server would anticipate that fresh clones will download a known bundle, +followed by catching up to the current state of the repository using ref +tips found in that bundle (or bundles). + +PROTOCOL for bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^ + +A `bundle-uri` request takes no arguments, and as noted above does not +currently advertise a capability value. Both may be added in the +future. + +When the client issues a `command=bundle-uri` the response is a list of +key-value pairs provided as packet lines with value `=`. The +meaning of these key-value pairs are provided by the config keys in the +`bundle.*` namespace (see linkgit:git-config[1]). + +Clients are still expected to fully parse the line according to the +above format, lines that do not conform to the format SHOULD be +discarded. The user MAY be warned in such a case. + +bundle-uri CLIENT AND SERVER EXPECTATIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +URI CONTENTS:: +The advertised URIs MUST be in one of two possible formats. ++ +The first possible format is a bundle file that `git bundle verify` +would accept. I.e. they MUST contain one or more reference tips for +use by the client, MUST indicate prerequisites (in any) with standard +"-" prefixes, and MUST indicate their "object-format", if +applicable. Create "*.bundle" files with `git bundle create`. ++ +The second possible format is a plaintext file that `git config --list` +would accept (with the `--file` option). The key-value pairs in this list +are in the `bundle.*` namespace (see linkgit:git-config[1]). + +bundle-uri CLIENT ERROR RECOVERY:: +A client MUST above all gracefully degrade on errors, whether that +error is because of bad missing/data in the bundle URI(s), because +that client is too dumb to e.g. understand and fully parse out bundle +headers and their prerequisite relationships, or something else. ++ +Server operators should feel confident in turning on "bundle-uri" and +not worry if e.g. their CDN goes down that clones or fetches will run +into hard failures. Even if the server bundle bundle(s) are +incomplete, or bad in some way the client should still end up with a +functioning repository, just as if it had chosen not to use this +protocol extension. ++ +All subsequent discussion on client and server interaction MUST keep +this in mind. + +bundle-uri SERVER TO CLIENT:: +The ordering of the returned bundle uris is not significant. Clients +MUST parse their headers to discover their contained OIDS and +prerequisites. A client MUST consider the content of the bundle(s) +themselves and their header as the ultimate source of truth. ++ +A server MAY even return bundle(s) that don't have any direct +relationship to the repository being cloned (either through accident, +or intentional "clever" configuration), and expect a client to sort +out what data they'd like from the bundle(s), if any. + +bundle-uri CLIENT TO SERVER:: +The client SHOULD provide reference tips found in the bundle header(s) +as 'have' lines in any subsequent `fetch` request. A client MAY also +ignore the bundle(s) entirely if doing so is deemed worse for some +reason, e.g. if the bundles can't be downloaded, it doesn't like the +tips it finds etc. + +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION:: +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s) +of the bundle(s) the client finds that the ref tips it wants can be +retrieved entirety from advertised bundle(s), it MAY disconnect. The +results of such a 'clone' or 'fetch' should be indistinguishable from +the state attained without using bundle-uri. + +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY:: +A client MAY perform an early disconnect while still downloading the +bundle(s) (having streamed and parsed their headers). In such a case +the client MUST gracefully recover from any errors related to +finishing the download and validation of the bundle(s). ++ +I.e. a client might need to re-connect and issue a 'fetch' command, +and possibly fall back to not making use of 'bundle-uri' at all. ++ +This "MAY" behavior is specified as such (and not a "SHOULD") on the +assumption that a server advertising bundle uris is more likely than +not to be serving up a relatively large repository, and to be pointing +to URIs that have a good chance of being in working order. A client +MAY e.g. look at the payload size of the bundles as a heuristic to see +if an early disconnect is worth it, should falling back on a full +"fetch" dialog be necessary. + +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION:: +A client SHOULD commence a negotiation of a PACK from the server via +the "fetch" command using the OID tips found in advertised bundles, +even if's still in the process of downloading those bundle(s). ++ +This allows for aggressive early disconnects from any interactive +server dialog. The client blindly trusts that the advertised OID tips +are relevant, and issues them as 'have' lines, it then requests any +tips it would like (usually from the "ls-refs" advertisement) via +'want' lines. The server will then compute a (hopefully small) PACK +with the expected difference between the tips from the bundle(s) and +the data requested. ++ +The only connection the client then needs to keep active is to the +concurrently downloading static bundle(s), when those and the +incremental PACK are retrieved they should be inflated and +validated. Any errors at this point should be gracefully recovered +from, see above. + +bundle-uri PROTOCOL FEATURES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As noted above the `=` definitions are documented by the +`bundle.*` config namespace. + +In particular, the `bundle.version` key specifies an integer value. The +only accepted value at the moment is `1`, but if the client sees an +unexpected value here then the client MUST ignore the bundle list. + +As long as `bundle.version` is understood, all other unknown keys MAY be +ignored by the client. The server will guarantee compatibility with older +clients, though newer clients may be better able to use the extra keys to +minimize downloads. + +Any backwards-incompatible addition of pre-URI key-value will be +guarded by a new `bundle.version` value or values in 'bundle-uri' +capability advertisement itself, and/or by new future `bundle-uri` +request arguments. + +Some example key-value pairs that are not currently implemented but could +be implemented in the future include: + + * Add a "hash=" or "size=" advertise the expected hash or + size of the bundle file. + + * Advertise that one or more bundle files are the same (to e.g. have + clients round-robin or otherwise choose one of N possible files). + + * A "oid=" shortcut and "prerequisite=" shortcut. For + expressing the common case of a bundle with one tip and no + prerequisites, or one tip and one prerequisite. ++ +This would allow for optimizing the common case of servers who'd like +to provide one "big bundle" containing only their "main" branch, +and/or incremental updates thereof. ++ +A client receiving such a a response MAY assume that they can skip +retrieving the header from a bundle at the indicated URI, and thus +save themselves and the server(s) the request(s) needed to inspect the +headers of that bundle or bundles. + GIT --- Part of the linkgit:git[1] suite diff --git a/bundle-uri.c b/bundle-uri.c index 79a914f961b..32022595964 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -563,6 +563,42 @@ cleanup: return result; } +/** + * API for serve.c. + */ + +int bundle_uri_advertise(struct repository *r, struct strbuf *value) +{ + static int advertise_bundle_uri = -1; + + if (advertise_bundle_uri != -1) + goto cached; + + advertise_bundle_uri = 0; + git_config_get_maybe_bool("uploadpack.advertisebundleuris", &advertise_bundle_uri); + +cached: + return advertise_bundle_uri; +} + +int bundle_uri_command(struct repository *r, + struct packet_reader *request) +{ + struct packet_writer writer; + packet_writer_init(&writer, 1); + + while (packet_reader_read(request) == PACKET_READ_NORMAL) + die(_("bundle-uri: unexpected argument: '%s'"), request->line); + if (request->status != PACKET_READ_FLUSH) + die(_("bundle-uri: expected flush after arguments")); + + /* TODO: Implement the communication */ + + packet_writer_flush(&writer); + + return 0; +} + /** * General API for {transport,connect}.c etc. */ diff --git a/bundle-uri.h b/bundle-uri.h index 4dbc269823c..357111ecce8 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -4,6 +4,7 @@ #include "hashmap.h" #include "strbuf.h" +struct packet_reader; struct repository; struct string_list; @@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * API for serve.c. + */ +int bundle_uri_advertise(struct repository *r, struct strbuf *value); +int bundle_uri_command(struct repository *r, struct packet_reader *request); + /** * General API for {transport,connect}.c etc. */ diff --git a/serve.c b/serve.c index 733347f602a..cbf4a143cfe 100644 --- a/serve.c +++ b/serve.c @@ -7,6 +7,7 @@ #include "protocol-caps.h" #include "serve.h" #include "upload-pack.h" +#include "bundle-uri.h" static int advertise_sid = -1; static int client_hash_algo = GIT_HASH_SHA1; @@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = { .advertise = always_advertise, .command = cap_object_info, }, + { + .name = "bundle-uri", + .advertise = bundle_uri_advertise, + .command = bundle_uri_command, + }, }; void protocol_v2_advertise_capabilities(void) diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index 1896f671cb3..f21e5e9d33d 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' ' wrong_algo sha1:sha256 wrong_algo sha256:sha1 EOF - cat >expect <<-EOF && + cat >expect.base <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) ls-refs=unborn @@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' ' server-option object-format=$(test_oid algo) object-info + EOF + cat >expect.trailer <<-EOF && 0000 EOF + cat expect.base expect.trailer >expect && GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ --advertise-capabilities >out && @@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' ' + test_config uploadpack.advertiseBundleURIs true && + + cat >expect.extra <<-EOF && + bundle-uri + EOF + cat expect.base \ + expect.extra \ + expect.trailer >expect && + + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ + --advertise-capabilities >out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: dies if not enabled' ' + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: invalid command '"'"'bundle-uri'"'"' + EOF + + cat >expect <<-\EOF && + ERR serve: invalid command '"'"'bundle-uri'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + test_done From patchwork Tue Nov 1 01:07:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13026566 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2199AFA3741 for ; Tue, 1 Nov 2022 01:07:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229822AbiKABHy (ORCPT ); Mon, 31 Oct 2022 21:07:54 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49740 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229757AbiKABHl (ORCPT ); Mon, 31 Oct 2022 21:07:41 -0400 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7FF6A15725 for ; Mon, 31 Oct 2022 18:07:39 -0700 (PDT) Received: by mail-wm1-x32d.google.com with SMTP id p16so2517867wmc.3 for ; Mon, 31 Oct 2022 18:07:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=3UI49Ux3Lhj27XScB9qQIdWE3hSbDUtOzn9Zu6tf32w=; b=FX7dgqXbD2ghwJcbEpX7Qf1tWxZEclMhiewOUhdsVwAL7YmyIBTZm2eWu65wZgG1j4 rZ8C01M9IbiwPNK1KTU2C4ELW6CAMiCgccfWWXwyauPIdOIDO9wWUj9M+83wmK26stCN 5VmMprl3PDkv2RLbCEGTpwVIVs4sP2JgReDhLCp+gGmGr2FOv/JpYsRDpwkabNc2JTzZ LA0u6SZfwl1K0khyYsEltHW7EtbX/MxUhRpvWWq2vXRttXBF8duVGCN7eG+M9jU624I+ P5OzFDKkuSDCDgUaA5kSfSKLceDwWS81Zkz+hjhiFxMbBqXf473EsndLeK3GgkFgdpxW EymA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3UI49Ux3Lhj27XScB9qQIdWE3hSbDUtOzn9Zu6tf32w=; b=FFOOuQU8Dlq1SN0SOxXJaF0wkXQdLW5Qw4NJIUffHsDxhveQuwMaenkumOoE55jshs 8dsjD8qBGBABTTDRkKGGsR4qVqs66CtNH6q845KSZqtbpYhHkcLqMStNJD7Dwuno3qKR W7m1I4SaZlJGNQxaCnKHIrqSWOxXX69jmsYHvc+C7WzRzVVq3POpFkLgSNQq+0UgzHaB yHjF7xMFWlDLpH8N5fc3KZGO37skowcmZHb86qYMp2d8BbyWMzscvuHnKFet5ec6HAB4 grNA02UPMqXT5up+h4kJMgglqealtKD0d4Mr39O1Z+/Pt6Qnxjk/wahFC9i0JBr+nyR6 +ZUQ== X-Gm-Message-State: ACrzQf27tfFIeAOp/csaOyVzRqgvH65yL7nno/TCmslwwgYFbH5tjzOO EbEMruckhehGqGseHTFxiYH0eyCcZug= X-Google-Smtp-Source: AMsMyM4EKvvUFr6mZvLmvyUngzFIEqkdwTkROmnSuOWE8pJIWqH9Zu1G8Yrf2175e/GZtDR7Dha+zw== X-Received: by 2002:a05:600c:154f:b0:3cc:767a:4653 with SMTP id f15-20020a05600c154f00b003cc767a4653mr20560149wmg.70.1667264857461; Mon, 31 Oct 2022 18:07:37 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r10-20020a05600c35ca00b003c6f1732f65sm6716825wmq.38.2022.10.31.18.07.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:36 -0700 (PDT) Message-Id: <64dd9bf41de0310bbed5e3623870819f1c876f83.1667264854.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:27 +0000 Subject: [PATCH 2/9] bundle-uri client: add minimal NOOP client MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Set up all the needed client parts of the "bundle-uri" protocol extension, without actually doing anything with the bundle URIs. I.e. if the server says it supports "bundle-uri" we'll issue a command=bundle-uri after command=ls-refs when we're cloning. We'll parse the returned output using the code already tested for in t5750-bundle-uri-parse.sh. What we aren't doing is actually acting on that data, i.e. downloading the bundle(s) before we get to doing the command=fetch, and adjusting our negotiation dialog appropriately. I'll do that in subsequent commits. There's a question of what level of encapsulation we should use here, I've opted to use connect.h in clone.c, but we could also e.g. make transport_get_remote_refs() invoke this, i.e. make it implicitly get the bundle-uri list for later steps. This approach means that we don't "support" this in "git fetch" for now. I'm starting with the case of initial clones, although as noted in preceding commits to the protocol documentation nothing about this approach precludes getting bundles on incremental fetches. For the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set environment variables for git-upload-pack (it's started by Apache), so let's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT} prerequisites for consistency and future use. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- builtin/clone.c | 7 ++ bundle-uri.c | 4 + connect.c | 47 ++++++++ remote.h | 5 + t/lib-t5730-protocol-v2-bundle-uri.sh | 148 +++++++++++++++++++++++++ t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++++++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++ t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++ transport-helper.c | 13 +++ transport-internal.h | 7 ++ transport.c | 55 +++++++++ transport.h | 19 ++++ 12 files changed, 375 insertions(+) create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh diff --git a/builtin/clone.c b/builtin/clone.c index 547d6464b3c..edf98295af2 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -27,6 +27,7 @@ #include "iterator.h" #include "sigchain.h" #include "branch.h" +#include "connect.h" #include "remote.h" #include "run-command.h" #include "connected.h" @@ -1266,6 +1267,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/bundle-uri.c b/bundle-uri.c index 32022595964..2201b604b11 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -571,6 +571,10 @@ int bundle_uri_advertise(struct repository *r, struct strbuf *value) { static int advertise_bundle_uri = -1; + if (value && + git_env_bool("GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE", 0)) + strbuf_addstr(value, "test-unknown-capability-value"); + if (advertise_bundle_uri != -1) goto cached; diff --git a/connect.c b/connect.c index 5ea53deda23..d39effb7492 100644 --- a/connect.c +++ b/connect.c @@ -15,6 +15,7 @@ #include "version.h" #include "protocol.h" #include "alias.h" +#include "bundle-uri.h" static char *server_capabilities_v1; static struct strvec server_capabilities_v2 = STRVEC_INIT; @@ -491,6 +492,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader) } } +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc) +{ + int line_nr = 1; + + /* Assert bundle-uri support */ + server_supports_v2("bundle-uri", 1); + + /* (Re-)send capabilities */ + send_capabilities(fd_out, reader); + + /* Send command */ + packet_write_fmt(fd_out, "command=bundle-uri\n"); + packet_delim(fd_out); + + /* Send options */ + if (git_env_bool("GIT_TEST_PROTOCOL_BAD_BUNDLE_URI", 0)) + packet_write_fmt(fd_out, "test-bad-client\n"); + packet_flush(fd_out); + + /* Process response from server */ + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { + const char *line = reader->line; + line_nr++; + + if (!bundle_uri_parse_line(bundles, line)) + continue; + + return error(_("error on bundle-uri response line %d: %s"), + line_nr, line); + } + + if (reader->status != PACKET_READ_FLUSH) + return error(_("expected flush after bundle-uri listing")); + + /* + * Might die(), but obscure enough that that's OK, e.g. in + * serve.c we'll call BUG() on its equivalent (the + * PACKET_READ_RESPONSE_END check). + */ + check_stateless_delimiter(stateless_rpc, reader, + _("expected response end packet after ref listing")); + + return 0; +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, diff --git a/remote.h b/remote.h index 1c4621b414b..1ebbe42792e 100644 --- a/remote.h +++ b/remote.h @@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, const struct string_list *server_options, int stateless_rpc); +/* Used for protocol v2 in order to retrieve refs from a remote */ +struct bundle_list; +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc); + int resolve_remote_symref(struct ref *ref, struct ref *list); /* diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh new file mode 100644 index 00000000000..27294e9c976 --- /dev/null +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -0,0 +1,148 @@ +# Included from t573*-protocol-v2-bundle-uri-*.sh + +T5730_PARENT= +T5730_URI= +T5730_BUNDLE_URI= +case "$T5730_PROTOCOL" in +file) + T5730_PARENT=file_parent + T5730_URI="file://$PWD/file_parent" + T5730_BUNDLE_URI="$T5730_URI/fake.bdl" + test_set_prereq T5730_FILE + ;; +git) + . "$TEST_DIRECTORY"/lib-git-daemon.sh + start_git_daemon --export-all --enable=receive-pack + T5730_PARENT="$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent" + T5730_URI="$GIT_DAEMON_URL/parent" + T5730_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq T5730_GIT + ;; +http) + . "$TEST_DIRECTORY"/lib-httpd.sh + start_httpd + T5730_PARENT="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" + T5730_URI="$HTTPD_URL/smart/http_parent" + T5730_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq T5730_HTTP + ;; +*) + BUG "Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)" + ;; +esac + +test_expect_success "setup protocol v2 $T5730_PROTOCOL:// tests" ' + git init "$T5730_PARENT" && + test_commit -C "$T5730_PARENT" one && + git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs true +' + +# Poor man's URI escaping. Good enough for the test suite whose trash +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art. +test_uri_escape() { + sed 's/ /%20/g' +} + +case "$T5730_PROTOCOL" in +http) + test_expect_success "setup config for $T5730_PROTOCOL:// tests" ' + git -C "$T5730_PARENT" config http.receivepack true + ' + ;; +*) + ;; +esac +T5730_BUNDLE_URI_ESCAPED=$(echo "$T5730_BUNDLE_URI" | test_uri_escape) + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri" ' + test_when_finished "rm -f log" && + test_when_finished "git -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true" && + git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs false && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + ! grep bundle-uri log +' + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep bundle-uri log +' + +test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" uploadpack.bundleURI \ + "$T5730_BUNDLE_URI_ESCAPED" && + + cat >err.expect <<-\EOF && + Cloning into '"'"'child'"'"'... + EOF + case "$T5730_PROTOCOL" in + file) + cat >fatal-bundle-uri.expect <<-\EOF + fatal: bundle-uri: unexpected argument: '"'"'test-bad-client'"'"' + EOF + ;; + *) + cat >fatal.expect <<-\EOF + fatal: read error: Connection reset by peer + EOF + ;; + esac && + + test_when_finished "rm -rf child" && + test_must_fail ok=sigpipe env \ + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \ + git -c protocol.version=2 \ + clone "$T5730_URI" child \ + >out 2>err && + test_must_be_empty out && + + grep -v -e ^fatal: -e ^error: err >err.actual && + test_cmp err.expect err.actual && + + case "$T5730_PROTOCOL" in + file) + # Due to general race conditions with client/server replies we + # may or may not get "fatal: the remote end hung up + # expectedly" here + grep "^fatal: bundle-uri:" err >fatal-bundle-uri.actual && + test_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual + ;; + *) + grep "^fatal:" err >fatal.actual && + # Due to the same race conditions this might be + # "fatal: read error: Connection reset by peer", "fatal: the remote end + # hung up unexpectedly" etc. + cat fatal.actual && + test_file_not_empty fatal.actual + ;; + esac && + + grep "clone> test-bad-client$" log >sent-bad-request && + test_file_not_empty sent-bad-request +' diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh new file mode 100755 index 00000000000..89203d3a23c --- /dev/null +++ b/t/t5730-protocol-v2-bundle-uri-file.sh @@ -0,0 +1,36 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'file://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'file://' transport +# +T5730_PROTOCOL=file +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_expect_success "unknown capability value with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + grep "> bundle-uri=test-unknown-capability-value" log +' + +test_done diff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh new file mode 100755 index 00000000000..282847b311f --- /dev/null +++ b/t/t5731-protocol-v2-bundle-uri-git.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=git +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh new file mode 100755 index 00000000000..fcc1cf3faef --- /dev/null +++ b/t/t5732-protocol-v2-bundle-uri-http.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=http +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/transport-helper.c b/transport-helper.c index e95267a4ab5..3ea7c2bb5ad 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport, return ret; } +static int get_bundle_uri(struct transport *transport) +{ + get_helper(transport); + + if (process_connect(transport, 0)) { + do_take_over(transport); + return transport->vtable->get_bundle_uri(transport); + } + + return -1; +} + static struct transport_vtable vtable = { .set_option = set_helper_option, .get_refs_list = get_refs_list, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs, .push_refs = push_refs, .connect = connect_helper, diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733ac..90ea749e5cf 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -26,6 +26,13 @@ struct transport_vtable { struct ref *(*get_refs_list)(struct transport *transport, int for_push, struct transport_ls_refs_options *transport_options); + /** + * Populates the remote side's bundle-uri under protocol v2, + * if the "bundle-uri" capability was advertised. Returns 0 if + * OK, negative values on error. + */ + int (*get_bundle_uri)(struct transport *transport); + /** * Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. diff --git a/transport.c b/transport.c index e7b97194c10..a020adc1f56 100644 --- a/transport.c +++ b/transport.c @@ -22,6 +22,7 @@ #include "protocol.h" #include "object-store.h" #include "color.h" +#include "bundle-uri.h" static int transport_use_color = -1; static char transport_colors[][COLOR_MAXLEN] = { @@ -359,6 +360,25 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus return handshake(transport, for_push, options, 1); } +static int get_bundle_uri(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct packet_reader reader; + int stateless_rpc = transport->stateless_rpc; + + if (!transport->bundles) { + CALLOC_ARRAY(transport->bundles, 1); + init_bundle_list(transport->bundles); + } + + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF); + + return get_remote_bundle_uri(data->fd[1], &reader, + transport->bundles, stateless_rpc); +} + static int fetch_refs_via_pack(struct transport *transport, int nr_heads, struct ref **to_fetch) { @@ -902,6 +922,7 @@ static int disconnect_git(struct transport *transport) static struct transport_vtable taken_over_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .disconnect = disconnect_git @@ -1054,6 +1075,7 @@ static struct transport_vtable bundle_vtable = { static struct transport_vtable builtin_smart_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .connect = connect_git, @@ -1068,6 +1090,9 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->progress = isatty(2); string_list_init_dup(&ret->pack_lockfiles); + CALLOC_ARRAY(ret->bundles, 1); + init_bundle_list(ret->bundles); + if (!remote) BUG("No remote provided to transport_get()"); @@ -1482,6 +1507,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } +int transport_get_remote_bundle_uri(struct transport *transport) +{ + const struct transport_vtable *vtable = transport->vtable; + + /* Check config only once. */ + if (transport->got_remote_bundle_uri++) + return 0; + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + + /* + * This is intentionally below the transport.injectBundleURI, + * we want to be able to inject into protocol v0, or into the + * dialog of a server who doesn't support this. + */ + if (!vtable->get_bundle_uri) + return error(_("bundle-uri operation not supported by protocol")); + + if (vtable->get_bundle_uri(transport) < 0) + return error(_("could not retrieve server-advertised bundle-uri list")); + return 0; +} + void transport_unlock_pack(struct transport *transport, unsigned int flags) { int in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER); @@ -1512,6 +1565,8 @@ int transport_disconnect(struct transport *transport) ret = transport->vtable->disconnect(transport); if (transport->got_remote_refs) free_refs((void *)transport->remote_refs); + clear_bundle_list(transport->bundles); + free(transport->bundles); free(transport); return ret; } diff --git a/transport.h b/transport.h index b5bf7b3e704..85150f504fb 100644 --- a/transport.h +++ b/transport.h @@ -62,6 +62,7 @@ enum transport_family { TRANSPORT_FAMILY_IPV6 }; +struct bundle_list; struct transport { const struct transport_vtable *vtable; @@ -76,6 +77,18 @@ struct transport { */ unsigned got_remote_refs : 1; + /** + * Indicates whether we already called get_bundle_uri_list(); set by + * transport.c::transport_get_remote_bundle_uri(). + */ + unsigned got_remote_bundle_uri : 1; + + /* + * The results of "command=bundle-uri", if both sides support + * the "bundle-uri" capability. + */ + struct bundle_list *bundles; + /* * Transports that call take-over destroys the data specific to * the transport type while doing so, and cannot be reused. @@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts); const struct ref *transport_get_remote_refs(struct transport *transport, struct transport_ls_refs_options *transport_options); +/** + * Retrieve bundle URI(s) from a remote. Populates "struct + * transport"'s "bundle_uri" and "got_remote_bundle_uri". + */ +int transport_get_remote_bundle_uri(struct transport *transport); + /* * Fetch the hash algorithm used by a remote. * From patchwork Tue Nov 1 01:07:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13026567 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9D73FA3740 for ; Tue, 1 Nov 2022 01:07:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229842AbiKABHz (ORCPT ); Mon, 31 Oct 2022 21:07:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229763AbiKABHl (ORCPT ); Mon, 31 Oct 2022 21:07:41 -0400 Received: from mail-wm1-x334.google.com (mail-wm1-x334.google.com [IPv6:2a00:1450:4864:20::334]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 5E0E715FF9 for ; Mon, 31 Oct 2022 18:07:40 -0700 (PDT) Received: by mail-wm1-x334.google.com with SMTP id f16-20020a05600c491000b003cf66a2e7c0so4415489wmp.5 for ; Mon, 31 Oct 2022 18:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=MUdRbpUXZ/e0h09r4Q7akeFJHmNHo+hT1vq/WO9qf70=; b=VBiS7mgTuq/lf9lZk1fNNRW0XQHHNDtOcXELCwEaxYUrUnyKx7E1D8qf0FunL4Xs7e sSPkQ8VJxZ0y0wz+JQyOftWi2p3JimKX1Q4/xqKy5L0VQM56yMWf/uETbNm9YtfdcHve mH7saKd04qjnBmJTRh7UF7ff9F96b34lWUkw4g3YeSD6bJFv4RMQZWkU1vn4wVJQqWM8 K4oSihadoR2e4YKhRELT20eQeflG7jqGNxcOJYuHFDs9UafLzuVs3RWLWtUjfrEhQHTk zZcHRsG8JO5Gh/l1i6tOhYecVxdP2hvhykd/4pS4CDu2lWPTa6h9s2C6KTAEVeHYdcO8 a8GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MUdRbpUXZ/e0h09r4Q7akeFJHmNHo+hT1vq/WO9qf70=; b=H5uz67ifGf+zwOdaIKsk/8/s07JOickOnignKoEWtKjhTSDkvYcBwFqMNxp3yZVtmR vmMt/htTJpdwYUBnLLb2IHr/Z5eIBCvXyMmOvms4i6TJTCA7yZGr7tpTv40CbOB6yBDs XKCRGo8mxZn8luximkt3D8DKzZTrKVhfyzxd+osLmytSZ3LXFOppNv5vOyQ3rTdvGIEj D6ksAm7HpoHS6IGpnJvPCtaWMJrUqcTnvs+vH7WD2f4o/FVwe4MCgWK98k9SeAG0zLAF gjTEaXfkIYFyCEmKxFplN7okL3eAilRyJV/+w8wY/4JhJzVZ0l2xg1FAKbC1UF84ez4U T3fA== X-Gm-Message-State: ACrzQf1dPkyRIshurExDET+IsWcC2S/m7du4bR0tiJxI2ipqjz4LUUBI j2ALvU6OVciuMe3xtkiXyZK09vhZXSY= X-Google-Smtp-Source: AMsMyM5g7X8p96DCnFNsQrHZRydqJE8PDhW8HJPYsNQHrL1Hq1HaXiSyHqx5vZlDZTY4evfTK6bsPQ== X-Received: by 2002:a05:600c:3d8a:b0:3c6:f241:cb36 with SMTP id bi10-20020a05600c3d8a00b003c6f241cb36mr9946774wmb.115.1667264858565; Mon, 31 Oct 2022 18:07:38 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t128-20020a1c4686000000b003cf7055c057sm4987808wma.3.2022.10.31.18.07.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:37 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:28 +0000 Subject: [PATCH 3/9] bundle-uri client: add helper for testing server MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper for issuing protocol v2 "bundle-uri" commands to a server, and to the parsing routines in bundle-uri.c. Since in the "git clone" case we'll have already done the handshake(), but not here, introduce a "got_advertisement" state along with "got_remote_heads". It seems to me that the "got_remote_heads" is badly named in the first place, and the whole logic of eagerly getting ls-refs on handshake() or not could be refactored somewhat, but let's not do that now, and instead just add another self-documenting state variable. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- builtin/clone.c | 2 +- t/helper/test-bundle-uri.c | 46 +++++++++++++++++++ t/lib-t5730-protocol-v2-bundle-uri.sh | 63 ++++++++++++++++++++++----- transport.c | 43 ++++++++++++++---- transport.h | 6 ++- 5 files changed, 139 insertions(+), 21 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index edf98295af2..22b1e506452 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1271,7 +1271,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) * Populate transport->got_remote_bundle_uri and * transport->bundle_uri. We might get nothing. */ - transport_get_remote_bundle_uri(transport); + transport_get_remote_bundle_uri(transport, 1); if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index 25afd393428..ffb975b7b4f 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -3,6 +3,10 @@ #include "bundle-uri.h" #include "strbuf.h" #include "string-list.h" +#include "transport.h" +#include "ref-filter.h" +#include "remote.h" +#include "refs.h" enum input_mode { KEY_VALUE_PAIRS, @@ -68,6 +72,46 @@ usage: usage_with_options(usage, options); } +static int cmd_ls_remote(int argc, const char **argv) +{ + const char *uploadpack = NULL; + struct string_list server_options = STRING_LIST_INIT_DUP; + const char *dest; + struct remote *remote; + struct transport *transport; + int status = 0; + + dest = argc > 1 ? argv[1] : NULL; + + remote = remote_get(dest); + if (!remote) { + if (dest) + die(_("bad repository '%s'"), dest); + die(_("no remote configured to get bundle URIs from")); + } + if (!remote->url_nr) + die(_("remote '%s' has no configured URL"), dest); + + transport = transport_get(remote, NULL); + if (uploadpack) + transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack); + if (server_options.nr) + transport->server_options = &server_options; + + if (transport_get_remote_bundle_uri(transport, 0) < 0) { + error(_("could not get the bundle-uri list")); + status = 1; + goto cleanup; + } + + print_bundle_list(stdout, transport->bundles); + +cleanup: + if (transport_disconnect(transport)) + return 1; + return status; +} + int cmd__bundle_uri(int argc, const char **argv) { const char *usage[] = { @@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv) return cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS); if (!strcmp(argv[1], "parse-config")) return cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE); + if (!strcmp(argv[1], "ls-remote")) + return cmd_ls_remote(argc - 1, argv + 1); error("there is no test-tool bundle-uri tool '%s'", argv[1]); usage: diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh index 27294e9c976..c327544641b 100644 --- a/t/lib-t5730-protocol-v2-bundle-uri.sh +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -34,7 +34,9 @@ esac test_expect_success "setup protocol v2 $T5730_PROTOCOL:// tests" ' git init "$T5730_PARENT" && test_commit -C "$T5730_PARENT" one && - git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs true + git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs true && + git -C "$T5730_PARENT" config bundle.version 1 && + git -C "$T5730_PARENT" config bundle.mode all ' # Poor man's URI escaping. Good enough for the test suite whose trash @@ -61,9 +63,8 @@ test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: no bundl git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs false && GIT_TRACE_PACKET="$PWD/log" \ - git \ - -c protocol.version=2 \ - ls-remote --symref "$T5730_URI" \ + test-tool bundle-uri \ + ls-remote "$T5730_URI" \ >actual 2>err && # Server responded using protocol v2 @@ -76,12 +77,11 @@ test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bun test_when_finished "rm -f log" && test_config -C "$T5730_PARENT" \ - uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && GIT_TRACE_PACKET="$PWD/log" \ - git \ - -c protocol.version=2 \ - ls-remote --symref "$T5730_URI" \ + test-tool bundle-uri \ + ls-remote "$T5730_URI" \ >actual 2>err && # Server responded using protocol v2 @@ -94,8 +94,8 @@ test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bun test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protocol v2" ' test_when_finished "rm -f log" && - test_config -C "$T5730_PARENT" uploadpack.bundleURI \ - "$T5730_BUNDLE_URI_ESCAPED" && + test_config -C "$T5730_PARENT" \ + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && cat >err.expect <<-\EOF && Cloning into '"'"'child'"'"'... @@ -146,3 +146,46 @@ test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protoc grep "clone> test-bad-client$" log >sent-bad-request && test_file_not_empty sent-bad-request ' + +test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + GIT_TRACE_PACKET="$PWD/log" \ + test-tool bundle-uri \ + ls-remote \ + "$T5730_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2 and extra data" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && + + # Extra data should be ignored + test_config -C "$T5730_PARENT" bundle.only.extra bogus && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + GIT_TRACE_PACKET="$PWD/log" \ + test-tool bundle-uri \ + ls-remote \ + "$T5730_URI" \ + >actual && + test_cmp_config_output expect actual +' diff --git a/transport.c b/transport.c index a020adc1f56..86460f5be28 100644 --- a/transport.c +++ b/transport.c @@ -198,6 +198,7 @@ struct git_transport_data { struct git_transport_options options; struct child_process *conn; int fd[2]; + unsigned got_advertisement : 1; unsigned got_remote_heads : 1; enum protocol_version version; struct oid_array extra_have; @@ -346,6 +347,7 @@ static struct ref *handshake(struct transport *transport, int for_push, BUG("unknown protocol version"); } data->got_remote_heads = 1; + data->got_advertisement = 1; transport->hash_algo = reader.hash_algo; if (reader.line_peeked) @@ -371,6 +373,33 @@ static int get_bundle_uri(struct transport *transport) init_bundle_list(transport->bundles); } + if (!data->got_advertisement) { + struct ref *refs; + struct git_transport_data *data = transport->data; + enum protocol_version version; + + refs = handshake(transport, 0, NULL, 0); + version = data->version; + + switch (version) { + case protocol_v2: + assert(!refs); + break; + case protocol_v0: + case protocol_v1: + case protocol_unknown_version: + assert(refs); + break; + } + } + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + packet_reader_init(&reader, data->fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_GENTLE_ON_EOF); @@ -1507,7 +1536,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } -int transport_get_remote_bundle_uri(struct transport *transport) +int transport_get_remote_bundle_uri(struct transport *transport, int quiet) { const struct transport_vtable *vtable = transport->vtable; @@ -1515,20 +1544,16 @@ int transport_get_remote_bundle_uri(struct transport *transport) if (transport->got_remote_bundle_uri++) return 0; - /* - * "Support" protocol v0 and v2 without bundle-uri support by - * silently degrading to a NOOP. - */ - if (!server_supports_v2("bundle-uri", 0)) - return 0; - /* * This is intentionally below the transport.injectBundleURI, * we want to be able to inject into protocol v0, or into the * dialog of a server who doesn't support this. */ - if (!vtable->get_bundle_uri) + if (!vtable->get_bundle_uri) { + if (quiet) + return -1; return error(_("bundle-uri operation not supported by protocol")); + } if (vtable->get_bundle_uri(transport) < 0) return error(_("could not retrieve server-advertised bundle-uri list")); diff --git a/transport.h b/transport.h index 85150f504fb..dd0115b83bf 100644 --- a/transport.h +++ b/transport.h @@ -297,8 +297,12 @@ const struct ref *transport_get_remote_refs(struct transport *transport, /** * Retrieve bundle URI(s) from a remote. Populates "struct * transport"'s "bundle_uri" and "got_remote_bundle_uri". + * + * With `quiet=1` it will not complain if the serve doesn't support + * the protocol, but only if we discover the server uses it, and + * encounter issues then. */ -int transport_get_remote_bundle_uri(struct transport *transport); +int transport_get_remote_bundle_uri(struct transport *transport, int quiet); /* * Fetch the hash algorithm used by a remote. From patchwork Tue Nov 1 01:07:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13026568 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 33B40FA3744 for ; Tue, 1 Nov 2022 01:07:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229819AbiKABH4 (ORCPT ); Mon, 31 Oct 2022 21:07:56 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49758 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229766AbiKABHm (ORCPT ); Mon, 31 Oct 2022 21:07:42 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0B5211658D for ; Mon, 31 Oct 2022 18:07:41 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id o4so18259292wrq.6 for ; Mon, 31 Oct 2022 18:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=tC4MklC4EotCdkubgLTySzDpazds/cBLtZSjsLhluuw=; b=jFj/oobU+Ea475WPUcASevBTkI5tHEalNRAwtjMNVwRsuIpFxRiO7OIKTUqZWuSlnd IqDxM2keWy9/hQhCL9KWo16tdnZ+LpbpRAR1206i7ScDYtnk/hJiJkLqTE2uR4mQMvpw beWmiFbcXzm02UGU/e5/12hDJ4pFXFjJI6iJ8kVuQvASnNZM3TIJb3rHQ+fGQKKR+y0Q uGQYuHrxyJW2GTwiL23lb2FLfsNZDYPtHoQpEn8vsJityWdacF8vx6k/iv4Tifr1ARWn Sm7yR4WC+pxtwWDtyqyQdzzX6eHOvo+1SaND/fCFHrb3QKQ8o6si/VeFuPm1okTrtM1N NfgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tC4MklC4EotCdkubgLTySzDpazds/cBLtZSjsLhluuw=; b=OFQX+3yp03tPCJEdZGTnfgj9wQECpRRXIoGhDPapGwor6xc1UJV/4fWH+Kf8DK+a7q VEc1i0CjeeYXy+jE4ydpaStcPG+FDpi27SqSQTATQr51/FSq4jCMX4CXlLdOjokgWfLE nKE41RIBujuVr0yL8YH+qpyskfK/TZOxRH/op7VzHh6Hl0Ztz9EzZVegDMr48q3ursJm csUU2IqwCAHqhtxXLD940kJGqrTfsVcZJ1GXsVlVrfRkpttwwpB11YYicYuHp6MChRZD 7T5adRV3BhNOOhqHvwGZQJu17rp93epa4lvyibrF1BKTXumNpR9/t/Jl1gXz85Bqa/Ck AD1w== X-Gm-Message-State: ACrzQf1Hy+5d8ZH2i9w1X2OMTWFFDuIgTJhOv7o3r/W/Ag1Q6szU85Gq 5Cnp1NO5GE83CiGvRdMws/LaIcPEMYo= X-Google-Smtp-Source: AMsMyM4ywJhYgP+F7zu+pnA4EMDfDlXSq8b5OlIDXHvPBGGrEb70sMSUn7j5S8STbKS1mf+jA9GkfQ== X-Received: by 2002:a5d:47c7:0:b0:236:64a4:6d5a with SMTP id o7-20020a5d47c7000000b0023664a46d5amr9649209wrc.666.1667264859334; Mon, 31 Oct 2022 18:07:39 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id b9-20020a5d45c9000000b0022e36c1113fsm8168864wrs.13.2022.10.31.18.07.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:38 -0700 (PDT) Message-Id: <431cd585184b29f981247a6fa298c5a555937728.1667264854.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:29 +0000 Subject: [PATCH 4/9] bundle-uri: serve bundle.* keys from config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Implement the "bundle-uri" protocol v2 capability by populating the key=value packet lines from the local Git config. The list of bundles is provided from the keys beginning with "bundle.". In the future, we may want to filter this list to be more specific to the exact known keys that the server intends to share, but for flexibility at the moment we will assume that the config values are well-formed. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 +++++++++++- t/lib-t5730-protocol-v2-bundle-uri.sh | 35 +++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 2201b604b11..3469f1aaa98 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -585,6 +585,16 @@ cached: return advertise_bundle_uri; } +static int config_to_packet_line(const char *key, const char *value, void *data) +{ + struct packet_reader *writer = data; + + if (!strncmp(key, "bundle.", 7)) + packet_write_fmt(writer->fd, "%s=%s", key, value); + + return 0; +} + int bundle_uri_command(struct repository *r, struct packet_reader *request) { @@ -596,7 +606,11 @@ int bundle_uri_command(struct repository *r, if (request->status != PACKET_READ_FLUSH) die(_("bundle-uri: expected flush after arguments")); - /* TODO: Implement the communication */ + /* + * Read all "bundle.*" config lines to the client as key=value + * packet lines. + */ + git_config(config_to_packet_line, &writer); packet_writer_flush(&writer); diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh index c327544641b..000fcc5e20b 100644 --- a/t/lib-t5730-protocol-v2-bundle-uri.sh +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -158,6 +158,8 @@ test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2" ' [bundle] version = 1 mode = all + [bundle "only"] + uri = $T5730_BUNDLE_URI_ESCAPED EOF GIT_TRACE_PACKET="$PWD/log" \ test-tool bundle-uri \ @@ -181,6 +183,39 @@ test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2 and ext [bundle] version = 1 mode = all + [bundle "only"] + uri = $T5730_BUNDLE_URI_ESCAPED + EOF + GIT_TRACE_PACKET="$PWD/log" \ + test-tool bundle-uri \ + ls-remote \ + "$T5730_URI" \ + >actual && + test_cmp_config_output expect actual +' + + +test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2 with list" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + bundle.bundle1.uri "$T5730_BUNDLE_URI_ESCAPED-1.bdl" && + test_config -C "$T5730_PARENT" \ + bundle.bundle2.uri "$T5730_BUNDLE_URI_ESCAPED-2.bdl" && + test_config -C "$T5730_PARENT" \ + bundle.bundle3.uri "$T5730_BUNDLE_URI_ESCAPED-3.bdl" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + [bundle "bundle1"] + uri = $T5730_BUNDLE_URI_ESCAPED-1.bdl + [bundle "bundle2"] + uri = $T5730_BUNDLE_URI_ESCAPED-2.bdl + [bundle "bundle3"] + uri = $T5730_BUNDLE_URI_ESCAPED-3.bdl EOF GIT_TRACE_PACKET="$PWD/log" \ test-tool bundle-uri \ From patchwork Tue Nov 1 01:07:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13026569 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9A13FA3740 for ; Tue, 1 Nov 2022 01:07:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229866AbiKABH6 (ORCPT ); Mon, 31 Oct 2022 21:07:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49792 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229776AbiKABHn (ORCPT ); Mon, 31 Oct 2022 21:07:43 -0400 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D1CF31659C for ; Mon, 31 Oct 2022 18:07:41 -0700 (PDT) Received: by mail-wm1-x32b.google.com with SMTP id b20-20020a05600c4e1400b003cc28585e2fso9078330wmq.1 for ; Mon, 31 Oct 2022 18:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=tLooz02+SsovFOZcXwMGlFj9MxU6sRX+bGtaH84Oqg8=; b=OrKSGtCa3mJBYad3UE0XGbzkR8zDjwk/Je+7tO6MXsiGbfvtyJxfVRucocaUeoBWAF rKJIc7BeFXe/xVjkEA2ACSiynOCivE1KDA1flG+Wi9L0cNuYew2GTQ1l9Ho64VWjVxc+ /XIWjJtZNtnbwX4gAVuV1UxFMVJ1z/G7fAXGdR8/JOAh6Fc9yMSjn65HfSO5WdTXPcDo KEu/QnnTeWTNqZqP63MlLMxYzpLj9ArtqdDuVg1S8RUjfRSFnuKTS1bFdJmdrYihzrDZ BOAG7DLl6LHgs7PycCGiCsuDaCJZNutUtNoyL0zVEMQx0IO5ffYTmPdbDMipDssUaKTV +MsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tLooz02+SsovFOZcXwMGlFj9MxU6sRX+bGtaH84Oqg8=; b=QWsUqntBtcXDo0ytGIkp8igzPBPpSeEnnqYjlLXTcrPAhf2qBQsohdyaPdxb4rDEme TduB7TX2wdmeD/rKLX7Ql1yVjheRO7QjQhiaXdqrMjTbaymfuOKP0z3rC7cNztNjOi4O 7PekNjO3Ir0wGY6SLhO10XL9swRD9rdR8+qtyx/9cBh/rg0H34gNHi+Lw7XciekLPFSY y4b1xIBu60CqSrQe/VibHutZwU79NV2t3xszcGfcWVLJTeRtKxC3vztNXDaf/4M+VuHL DQ8ox2GVzjAOsNxpdOqGQd6QPexUCaFhFC2kOHb52I+Dzr/H/vzZPCzaQMNSXEznVYAF /sSw== X-Gm-Message-State: ACrzQf1zcb+3bKzfzL79xCujDjFl/k+y/Iei27BiVL2OfysHRQU+fuRQ f6dGI5vtE6jWvi7E/pkwaYhRPCYYcvM= X-Google-Smtp-Source: AMsMyM4fVlqjvMT38TewIMkx9HAagxSzwEAEsmjIm+SH1TheijpIcgBGB7PlZiuAV9nU0VxtAUMmmQ== X-Received: by 2002:a05:600c:15c9:b0:3cf:6054:3b3b with SMTP id v9-20020a05600c15c900b003cf60543b3bmr12495092wmf.167.1667264860139; Mon, 31 Oct 2022 18:07:40 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y10-20020a5d620a000000b0023657e1b97esm8544447wru.11.2022.10.31.18.07.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:39 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:30 +0000 Subject: [PATCH 5/9] bundle-uri client: add boolean transfer.bundleURI setting MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= The yet-to-be introduced client support for bundle-uri will always fall back on a full clone, but we'd still like to be able to ignore a server's bundle-uri advertisement entirely. The new transfer.bundleURI config option defaults to 'false', but a user can set it to 'true' to enable checking for bundle URIs from the origin Git server using protocol v2. To enable this setting by default in the correct tests, add a GIT_TEST_BUNDLE_URI environment variable. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/config/transfer.txt | 6 ++++++ t/lib-t5730-protocol-v2-bundle-uri.sh | 3 +++ transport.c | 10 +++++++--- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt index 264812cca4d..c3ac767d1e4 100644 --- a/Documentation/config/transfer.txt +++ b/Documentation/config/transfer.txt @@ -115,3 +115,9 @@ transfer.unpackLimit:: transfer.advertiseSID:: Boolean. When true, client and server processes will advertise their unique session IDs to their remote counterpart. Defaults to false. + +transfer.bundleURI:: + When `true`, local `git clone` commands will request bundle + information from the remote server (if advertised) and download + bundles before continuing the clone through the Git protocol. + Defaults to `false`. diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh index 000fcc5e20b..872bc39ad1b 100644 --- a/t/lib-t5730-protocol-v2-bundle-uri.sh +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -1,5 +1,8 @@ # Included from t573*-protocol-v2-bundle-uri-*.sh +GIT_TEST_BUNDLE_URI=1 +export GIT_TEST_BUNDLE_URI + T5730_PARENT= T5730_URI= T5730_BUNDLE_URI= diff --git a/transport.c b/transport.c index 86460f5be28..b33180226ae 100644 --- a/transport.c +++ b/transport.c @@ -1538,6 +1538,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) int transport_get_remote_bundle_uri(struct transport *transport, int quiet) { + int value = 0; const struct transport_vtable *vtable = transport->vtable; /* Check config only once. */ @@ -1545,10 +1546,13 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet) return 0; /* - * This is intentionally below the transport.injectBundleURI, - * we want to be able to inject into protocol v0, or into the - * dialog of a server who doesn't support this. + * Don't use bundle-uri at all, if configured not to. Only proceed + * if GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true. */ + if (!git_env_bool("GIT_TEST_BUNDLE_URI", 0) && + (git_config_get_bool("transfer.bundleuri", &value) || !value)) + return 0; + if (!vtable->get_bundle_uri) { if (quiet) return -1; From patchwork Tue Nov 1 01:07:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13026570 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 24C9BFA3740 for ; Tue, 1 Nov 2022 01:08:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229868AbiKABH7 (ORCPT ); Mon, 31 Oct 2022 21:07:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49972 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229561AbiKABHw (ORCPT ); Mon, 31 Oct 2022 21:07:52 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A0D4A16595 for ; Mon, 31 Oct 2022 18:07:42 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id y16so18215800wrt.12 for ; Mon, 31 Oct 2022 18:07:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=x0WcloxmUBLoHeCAmzYfJFJ2dAabXIl2IzKQa0WH0r0=; b=KpYRYaN+ertlGZtWDWPoAHVvltzHUUL1ZtDlI/QXgT9ntjGBeDo7eC6lgXui4q8uWE EpnpJrIbxVewAej+rVS8H3+DLjNiien9+KIA9P5LRFkQ+CtXeXYPSeZsqLOk9scz0364 mcE/oWrGi94wjJwFwovWw6FlY9pY4q7Wy6dYYSR8n97OT5ilPrHWY6ZbG7uc6iDUjfUx 5wG946pBGRml5fjbDn2YynydcGWycPj03Ajd0OORj4b5GsOX81nKYlMbqF+RJCFB0mvz 07qhS677pEbPDGa+CYhfwYtBlpD04akj1xqaqperKhHK6e8zTXcTIlzJJJRgiR7ZwHtq TEmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=x0WcloxmUBLoHeCAmzYfJFJ2dAabXIl2IzKQa0WH0r0=; b=bLsX76XHClhUue6Jv1xJcEbDAEDnVz18D8gEwzjxpL/StOaBCO1nd6L4uLVbinMBvE Bw3fXyUFdVrPdACOMpqzvjSV8+b/FSzo+dUchXLuecjcpCVkhk3v9lZe2NaUenid2NxY U0uqKeJm+IaDyTJRy7MFCCF+YpYYryAzYIIZcZ6VeFmz/tb7DJa6Vx8oRVzneGE0r8T8 YFKy7IoqPCFaE+RF/7y/kH1zu7NSIwJ7Uk1nl53BZLPjy1G5Vb5dESP8oJEa5wPAgF/R HMOcQTyMwI/bW4ZgFetsCAbiuE1JbHbbjNNali6R+7x0uht4itdCxM7gOB1iEranuQV2 ON+g== X-Gm-Message-State: ACrzQf0k3fPbaUCaBkscvNH65DY10c8AV8tM0dZwNoaFJtlfHYRqQbYR vq1TYEwFKxeruUcgNx/uwQ7JhGRU7j4= X-Google-Smtp-Source: AMsMyM7l7njlltPMFc13utS7LIRrVBoiDkYnt2AWgD34i1E7f/qQiFEmL/xX/dU/T9NHNnD5zRARgQ== X-Received: by 2002:adf:f081:0:b0:236:5e7c:4ec2 with SMTP id n1-20020adff081000000b002365e7c4ec2mr9521962wro.641.1667264861002; Mon, 31 Oct 2022 18:07:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bp7-20020a5d5a87000000b00236545edc91sm8573348wrb.76.2022.10.31.18.07.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:40 -0700 (PDT) Message-Id: <2200a70d279adcc6460dcb7bef7b0e681d19a414.1667264854.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:31 +0000 Subject: [PATCH 6/9] strbuf: reintroduce strbuf_parent_directory() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The strbuf_parent_directory() method was added as a static method in contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended config and starts maintenance, 2021-12-03) and then removed in 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now there is a need for a similar method in the bundle URI feature. Re-add the method, this time in strbuf.c. The method requirements are slightly modified to allow a trailing slash, in which case nothing is done. The return value is the number of byte removed. Signed-off-by: Derrick Stolee --- strbuf.c | 9 +++++++++ strbuf.h | 7 +++++++ 2 files changed, 16 insertions(+) diff --git a/strbuf.c b/strbuf.c index 0890b1405c5..b5cb324c431 100644 --- a/strbuf.c +++ b/strbuf.c @@ -1200,3 +1200,12 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path, free(path2); return res; } + +size_t strbuf_parent_directory(struct strbuf *buf) +{ + size_t len = buf->len; + size_t offset = offset_1st_component(buf->buf); + char *path_sep = find_last_dir_sep(buf->buf + offset); + strbuf_setlen(buf, path_sep ? path_sep - buf->buf : offset); + return len - buf->len; +} diff --git a/strbuf.h b/strbuf.h index 76965a17d44..8a964a08c31 100644 --- a/strbuf.h +++ b/strbuf.h @@ -664,6 +664,13 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer, int strbuf_edit_interactively(struct strbuf *buffer, const char *path, const char *const *env); +/* + * Remove the deepest subdirectory in the provided path string. If path + * contains a trailing separator, then the path is considered a directory + * and nothing is modified. + */ +size_t strbuf_parent_directory(struct strbuf *buf); + void strbuf_add_lines(struct strbuf *sb, const char *prefix, const char *buf, From patchwork Tue Nov 1 01:07:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13026572 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1464FA3741 for ; Tue, 1 Nov 2022 01:08:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229875AbiKABID (ORCPT ); Mon, 31 Oct 2022 21:08:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49990 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229792AbiKABHx (ORCPT ); Mon, 31 Oct 2022 21:07:53 -0400 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6E53B165AC for ; Mon, 31 Oct 2022 18:07:43 -0700 (PDT) Received: by mail-wm1-x32e.google.com with SMTP id b20-20020a05600c4e1400b003cc28585e2fso9078363wmq.1 for ; Mon, 31 Oct 2022 18:07:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=lIT1c16Xd/NJcClYTegONT4v1Q64EvFWgUorOXXJwK0=; b=PYyr0/hCi6/CA3l6EpU6HJRRKFqI0L8YeRXlGD9zN12VizEhuamSL6ibvgVvLNFj0c rQNgiIQF3uMvJy8v94cD16QHP0Q2KNr3ivLZedUu7LBIN9KNHtOe/1wWtoeYm7wY7y2l EPuMKx3rAhTRsaBf9FKFs5F1cedwNVhgP2xQXtSYLUJqCc9R9Qiv9hEKDrzu2NZuL3pT /dPoxR7fI2qAxUMqTVwA7jY71D1ZfXw7uxLThrCwtcIUc4Lt2RCqFj8d8cyMiLbm33Iu L4VGk5z6DCPLwps3GpazPI7QkmxmXAH2WfT01Z9yaa2zBluL/kjlatezbLywAcr8xGpP c9XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lIT1c16Xd/NJcClYTegONT4v1Q64EvFWgUorOXXJwK0=; b=M/TGXOP4rXrDSUueQZN2FcRJ1jPP+CR4+A4NLeIhRNpyuz+LvmiytK4z61ybKc/Gt7 5VU3s38d/OkJ7pXLFViUUzS9siaJvq4RgErtRIBNf3ErspUiipPSrcg5lQtby7NZdYMZ NUeok6pm1BrHSAnZEVVXKK9Civ0TMtvHY30TbBohi7+Xxh51kYu/IGwm8IGU6UfokFcj P6m1ddku6LRQX2qsXzgiPVc9R4zKWVc8sdY0+6PWhuX0Cq+tprw5EbCkaeHHZtAYBbby xBgPNaVH11ospcouYHHzW5gTFY9QuVegLf4x0v9xhBRhIY0LAMYtpOE4RD31lwVtt0oT 94SQ== X-Gm-Message-State: ACrzQf23YoW7Yx1lgrNKGnXLcB37OaKYVX78VdgvWkwnuB3dRFq1S7Yt /RxlVgJeekRTYQRWbRh5L3UyeMOIA3M= X-Google-Smtp-Source: AMsMyM4r5VVBhdm0F/72emil3asKxBrNVRBcVUA/kEzaJsWskSpMYu3993HPihIlIEPEf7BryApDOA== X-Received: by 2002:a05:600c:5119:b0:3c6:cae1:1512 with SMTP id o25-20020a05600c511900b003c6cae11512mr20682008wms.80.1667264861830; Mon, 31 Oct 2022 18:07:41 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ay16-20020a05600c1e1000b003cf537ec2efsm10176657wmb.36.2022.10.31.18.07.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:41 -0700 (PDT) Message-Id: <3550f6fb91b0214c459cbc0dd9bb8c6d7c677f85.1667264854.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:32 +0000 Subject: [PATCH 7/9] bundle-uri: allow relative URLs in bundle lists Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Bundle providers may want to distribute that data across multiple CDNs. This might require a change in the base URI, all the way to the domain name. If all bundles require an absolute URI in their 'uri' value, then every push to a CDN would require altering the table of contents to match the expected domain and exact location within it. Allow a bundle list to specify a relative URI for the bundles. This allows easier distribution of bundle data. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 ++++++++++- bundle-uri.h | 9 +++++++ t/helper/test-bundle-uri.c | 2 ++ t/t5750-bundle-uri-parse.sh | 54 +++++++++++++++++++++++++++++++++++++ transport.c | 3 +++ 5 files changed, 83 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 3469f1aaa98..0f3902bbd2b 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -7,6 +7,7 @@ #include "hashmap.h" #include "pkt-line.h" #include "config.h" +#include "remote.h" static int compare_bundles(const void *hashmap_cmp_fn_data, const struct hashmap_entry *he1, @@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list) for_all_bundles_in_list(list, clear_remote_bundle_info, NULL); hashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent); + free(list->baseURI); } int for_all_bundles_in_list(struct bundle_list *list, @@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value, if (!strcmp(subkey, "uri")) { if (bundle->uri) return -1; - bundle->uri = xstrdup(value); + bundle->uri = relative_url(list->baseURI, value, NULL); return 0; } @@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri, .error_action = CONFIG_ERROR_ERROR, }; + if (!list->baseURI) { + struct strbuf baseURI = STRBUF_INIT; + strbuf_addstr(&baseURI, uri); + + /* + * If the URI does not end with a trailing slash, then + * remove the filename portion of the path. This is + * important for relative URIs. + */ + strbuf_parent_directory(&baseURI); + list->baseURI = strbuf_detach(&baseURI, NULL); + } result = git_config_from_file_with_options(config_to_bundle_list, filename, list, &opts); diff --git a/bundle-uri.h b/bundle-uri.h index 357111ecce8..7905e56732c 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -61,6 +61,15 @@ struct bundle_list { int version; enum bundle_list_mode mode; struct hashmap bundles; + + /** + * The baseURI of a bundle_list is used as the base for any + * relative URIs advertised by the bundle list at that location. + * + * When the list is generated from a Git server, then use that + * server's location. + */ + char *baseURI; }; void init_bundle_list(struct bundle_list *list); diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index ffb975b7b4f..5aa0b494ce3 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo init_bundle_list(&list); + list.baseURI = xstrdup(""); + switch (mode) { case KEY_VALUE_PAIRS: if (argc != 1) diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh index c2fe3f9c5a5..ed5262a8d2b 100755 --- a/t/t5750-bundle-uri-parse.sh +++ b/t/t5750-bundle-uri-parse.sh @@ -30,6 +30,30 @@ test_expect_success 'bundle_uri_parse_line() just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'bundle_uri_parse_line(): relative URIs' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-key-values in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' ' cat >in <<-\EOF && =bogus-value @@ -136,6 +160,36 @@ test_expect_success 'parse config format: just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'parse config format: relative URIs' ' + cat >in <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = bundle.bdl + [bundle "two"] + uri = ../bundle.bdl + [bundle "three"] + uri = sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-config in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + test_expect_success 'parse config format edge cases: empty key or value' ' cat >in1 <<-\EOF && = bogus-value diff --git a/transport.c b/transport.c index b33180226ae..2c4ff0c2023 100644 --- a/transport.c +++ b/transport.c @@ -1553,6 +1553,9 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet) (git_config_get_bool("transfer.bundleuri", &value) || !value)) return 0; + if (!transport->bundles->baseURI) + transport->bundles->baseURI = xstrdup(transport->url); + if (!vtable->get_bundle_uri) { if (quiet) return -1; From patchwork Tue Nov 1 01:07:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13026571 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CDA9FA3741 for ; Tue, 1 Nov 2022 01:08:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229872AbiKABIB (ORCPT ); Mon, 31 Oct 2022 21:08:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49984 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229785AbiKABHx (ORCPT ); Mon, 31 Oct 2022 21:07:53 -0400 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 37C77165A9 for ; Mon, 31 Oct 2022 18:07:43 -0700 (PDT) Received: by mail-wr1-x431.google.com with SMTP id j15so18260822wrq.3 for ; Mon, 31 Oct 2022 18:07:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=0deZTLd7Q4hhlFR48unTZr9maHwfrTk4i/2Pp8BqAsQ=; b=iOw58SUwGpC4MN3Z4wGx2T7dmULAtF/DvxOxvQDOOobWhu/mbfKuUxOAqluYENCc9F sY/8+XC4bCqPR4YlSOv3HElrYedilV5KplKcYoq4BKsd6zYQ+Il1AW/uKuveLVwOYjRZ X5Es1Ywnu+KMnMDZqjRVIB/FKbflLK7MwLz1sLiPf2taL9PjDYMMlR/GwLKrpBja2OB8 EFJf6L22b0STujSnEbGSH6vrVkBm1YMZZklaIZ6Cx8ufK/RRXvNDEsobcMFGuVErDJje TlXJ51dy+q0lBix/ayP5MI9D1CZyKitIKdJVfPZWOBFE8doi7duSvXdZW8C0Nq+ErkQ0 HMtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0deZTLd7Q4hhlFR48unTZr9maHwfrTk4i/2Pp8BqAsQ=; b=ixMlpNXdXfIfkdJ3DWeLtJInlvqnl90/FH57uCYtmaPKQG8rbPnikMC9N9ND+A6YX+ LDlQNJOao3kUnIM5VzT8XhyW0yaa5GfZ1AUkCWKC1l5GGCWPjsL9xsW30Xz239P/tQ8S cddlV9yYiE21oJxcXl5UzD2nAY+YUlPIQd2jKWK+XiJ7J8l0Mvl58nzYYHo4eZyO1kXz mHgYE1VyCCFopmOu0EG+GzHwmdRB+P/7ukJPLEABsVmLPy0daQZ+qT7u7JYbZUsruScd sO6gNYbgjMzhmLgf+SdG37hYEM1NpICKYoubaGOwAB9qfMpZiYNV4ZXg3cCXrjNUCMH/ Z0zg== X-Gm-Message-State: ACrzQf0ChNfqdnMvu6enBO79IPZUjqEzqzNIsiMgGZuu/cPJUEWk3g9D C0984UVtELahR30VqnJ6FW54qby4H+w= X-Google-Smtp-Source: AMsMyM7tYHcTbY6eugH8GVIY3v1IfUVz22NwQogM+CIrSWZ09i66CURbcyNW8zgVTx1aDaarDrN1WQ== X-Received: by 2002:a5d:5710:0:b0:236:cdd4:4cf4 with SMTP id a16-20020a5d5710000000b00236cdd44cf4mr4548089wrv.376.1667264862512; Mon, 31 Oct 2022 18:07:42 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h8-20020a05600c350800b003c6b70a4d69sm1453491wmq.42.2022.10.31.18.07.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:42 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:33 +0000 Subject: [PATCH 8/9] bundle-uri: download bundles from an advertised list Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The logic in fetch_bundle_uri() is useful for the --bundle-uri option of 'git clone', but is not helpful when the clone operation discovers a list of URIs from the bundle-uri protocol v2 verb. To actually download and unbundle the advertised bundles, we need a different mechanism. Create the new fetch_bundle_list() method which is very similar to fetch_bundle_uri() except that it relies on download_bundle_list() instead of fetch_bundle_uri_internal(). The download_bundle_list() method will recursively call fetch_bundle_uri_internal() if any of the advertised URIs serve a bundle list instead of a bundle. This will also follow the bundle.list.mode setting from the input list: "any" will download only one such URI while "all" will download data from all of the URIs. In an identical way to fetch_bundle_uri(), the bundles are unbundled after all of the bundle lists have been expanded and all necessary URIs. Signed-off-by: Derrick Stolee --- bundle-uri.c | 21 +++++++++++++++++++++ bundle-uri.h | 11 +++++++++++ 2 files changed, 32 insertions(+) diff --git a/bundle-uri.c b/bundle-uri.c index 0f3902bbd2b..0b0081ed269 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -577,6 +577,27 @@ cleanup: return result; } +int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list) +{ + int result; + struct bundle_list global_list; + + init_bundle_list(&global_list); + + /* If a bundle is added to this global list, then it is required. */ + global_list.mode = BUNDLE_MODE_ALL; + + if ((result = download_bundle_list(r, list, &global_list, 0))) + goto cleanup; + + result = unbundle_all_bundles(r, &global_list); + +cleanup: + for_all_bundles_in_list(&global_list, unlink_bundle, NULL); + clear_bundle_list(&global_list); + return result; +} + /** * API for serve.c. */ diff --git a/bundle-uri.h b/bundle-uri.h index 7905e56732c..a75b68d2f5a 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -102,6 +102,17 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * Given a bundle list that was already advertised (likely by the + * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the + * bundles according to the bundle strategy of that list. + * + * Returns non-zero if no bundle information is found at the given 'uri'. + */ +int fetch_bundle_list(struct repository *r, + const char *uri, + struct bundle_list *list); + /** * API for serve.c. */ From patchwork Tue Nov 1 01:07:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13026573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0417AFA3740 for ; Tue, 1 Nov 2022 01:08:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229757AbiKABIF (ORCPT ); Mon, 31 Oct 2022 21:08:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:49974 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229823AbiKABHy (ORCPT ); Mon, 31 Oct 2022 21:07:54 -0400 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E4A6A165B5 for ; Mon, 31 Oct 2022 18:07:44 -0700 (PDT) Received: by mail-wr1-x430.google.com with SMTP id w14so18220220wru.8 for ; Mon, 31 Oct 2022 18:07:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=k3/06B+tN8VDLYpoq51dkxKW8dwQ9YorBO/1Gu/cg8g=; b=MP+pMItKZ8k7kNLYfcI/TUrqcXYKYqhDeUcpiFjuQ4NmdLxB7Hbo0uaucWNo770Ndu zfKCIk9rqQaqqDb5pHyU5XTN7N7Nqo/NY33mt806cNr2bkIT8473CZfQbl2sL8KGjRHl b15remwRJIYlnkMwn0EI8vKksq5P78L/uQglrS0ka2p0iKn2LLru7njb254P4VGZo9j+ VvY+TCH6G4ET+Bx3MXyYuwOR6W9R7c0eCsSB5eDfSXOAgxKu8DT+t8eH7i36QAFqHQ7E IqaHgnkcM3llYXARNPL3xvIh6riMcRuaqp7Q+UeYxeUZHJTk9CAZfG+mX3EFEFXCDg/N 4GFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k3/06B+tN8VDLYpoq51dkxKW8dwQ9YorBO/1Gu/cg8g=; b=vbukw2aaFFpHrIFztktEP6VI8lJPkWOOL59WauJneQBesD/fyOGEEgMxAyCxdkeCaz E7CI/VMxbUL7AgWu68mO3MLJykf/OuZ5VF8KaEcRT5iabgVd54cTKjOZUJfpbYaKKLbP 9N2BLOAopF2TU5FZ86WuZatcBusMAhucMenaLQaxNfnM3R1JxajjIRES891HMONgkzZ4 zCLQ/clS+iB3MrTRlVOybxwNp9sOECp6DixYIr2jVLIZ1qTr4udT3GJZexCmBTt85p0U 34FBTLAaqx0D+XALXR/UA+B1++sUUAj8s73aPaDMrEoQsu/gH7vXQ7I7NluNrPfPNJU+ FeuA== X-Gm-Message-State: ACrzQf21+m58lK61HnQFwhHmOG0Jqh4UaDpMeo9KPIK9GH046jQQr8cU smdRuWCteZQ69Uhp3cXTst0emoKQK14= X-Google-Smtp-Source: AMsMyM6qw03mnMhFCJLh1YkrMgB7gefr1OwCV9W44zHuXg/v5qXPdyb3lnE+qb96KOSZohTPmSywSA== X-Received: by 2002:adf:fc07:0:b0:236:6f04:2010 with SMTP id i7-20020adffc07000000b002366f042010mr9739982wrr.227.1667264863298; Mon, 31 Oct 2022 18:07:43 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id az5-20020a05600c600500b003b435c41103sm9795860wmb.0.2022.10.31.18.07.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 31 Oct 2022 18:07:42 -0700 (PDT) Message-Id: <1c034bba744716b03c6ce4422a378001c9c7e39f.1667264854.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 01 Nov 2022 01:07:34 +0000 Subject: [PATCH 9/9] clone: unbundle the advertised bundles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A previous change introduced the transport methods to acquire a bundle list from the 'bundle-uri' protocol v2 verb, when advertised _and_ when the client has chosen to enable the feature. Teach Git to download and unbundle the data advertised by those bundles during 'git clone'. Also, since the --bundle-uri option exists, we do not want to mix the advertised bundles with the user-specified bundles. Signed-off-by: Derrick Stolee --- builtin/clone.c | 26 +++++++++++++++++---- t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 5 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 22b1e506452..09f10477ed6 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1267,11 +1267,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); - /* - * Populate transport->got_remote_bundle_uri and - * transport->bundle_uri. We might get nothing. - */ - transport_get_remote_bundle_uri(transport, 1); + if (!bundle_uri) { + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport, 1); + + if (transport->bundles && + hashmap_get_size(&transport->bundles->bundles)) { + /* At this point, we need the_repository to match the cloned repo. */ + if (repo_init(the_repository, git_dir, work_tree)) + warning(_("failed to initialize the repo, skipping bundle URI")); + if (fetch_bundle_list(the_repository, + remote->url[0], + transport->bundles)) + warning(_("failed to fetch advertised bundles")); + } else { + clear_bundle_list(transport->bundles); + FREE_AND_NULL(transport->bundles); + } + } if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 45f0803ed4d..d1d8139751e 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' ' git clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo ' +test_expect_success 'auto-discover bundle URI from HTTP clone' ' + test_when_finished rm -rf trace.txt repo2 "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle" --all && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.mode all && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + + GIT_TEST_BUNDLE_URI=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 clone \ + $HTTPD_URL/smart/repo2.git repo2 && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt +' + +test_expect_success 'auto-discover multiple bundles from HTTP clone' ' + test_when_finished rm -rf trace.txt repo3 "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + test_commit -C src new && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/new.bundle" HEAD~1..HEAD && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.mode all && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.new.uri "$HTTPD_URL/new.bundle" && + + GIT_TEST_BUNDLE_URI=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 clone \ + $HTTPD_URL/smart/repo3.git repo3 && + + # We should fetch _both_ bundles + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/new.bundle"\] + EOF + grep -f pattern trace.txt +' + # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled.