From patchwork Mon Dec 5 17:50:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13064941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 759F7C4321E for ; Mon, 5 Dec 2022 17:52:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232825AbiLERw3 (ORCPT ); Mon, 5 Dec 2022 12:52:29 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44284 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232640AbiLERvk (ORCPT ); Mon, 5 Dec 2022 12:51:40 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 1A15C3898 for ; Mon, 5 Dec 2022 09:50:44 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id o7-20020a05600c510700b003cffc0b3374so10725869wms.0 for ; Mon, 05 Dec 2022 09:50:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=/vixHV679y33VgTXhSIXK+evywACQByl7SjY6sOsa8o=; b=OKrmdk1peatCN0zvF8dxyabkVE7w8hQ0vOBat2ivn1M0gAhGagGKRjhruCUx58kjfg 3Lqb4D9FR/OrpXsjKWiyOFRR9zMhG0E9qjWBbEKwzbD5sc3Kt6zHVbVZzo/7oAO0eMKk /sRx7yYRNldasbJvVceLOitFttMO7mmqqwuVRfv8gvzi2Vj5voc4jCWGmq7WVujBPQEG 25Sm5asp9aCpukaqpTIo3l2a6FwX2XuHnarqrVCQaYJ11MoTA6+lbirPZ65dOqzBkd/C xNzhSL+ZKkRChkNCZ5/+LOqTaas1cy5u4cBpZvD+/ctH2KoHqcNGv+C1tRQBWKjCW33y UzjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/vixHV679y33VgTXhSIXK+evywACQByl7SjY6sOsa8o=; b=Jr0r7O7RY5ngLYIMwFrGAV1w/D+SVOy0099DRmrLoxQ7TolB9IwvyA6vkGWvlH5Px2 GHbb4i9XKO2aLBNgxyDxNKKSBf1/lhTpw8Hug64hPL9My6dKHkpHRu81YxKw4nibYdxp dijSzkEwCiHdfy+R9SQro8LulFkZ3hU0ZbimH/mCnAnXs9DX5jT+88GlIACiX7v3nM8Y uL6bCBZLLKZlul+KzC1MpTF+L3lF8UYkY3a6FzpqqQXp0S8jNpYT5ZQoQgZh83C0ZFuI HdSwAhBE8JPOWsSB4lZl249uohY1VL0UdMI95QnGmwms/sAYyos+OlosbP2qpXKDuX/t PXlA== X-Gm-Message-State: ANoB5pmsXxmfiOPrW1rT22CavjgV+WlUjDDlgWdX0vJ2yWfoF2xKZfNM oOVycIRaIfVp0uGqqBCdgU8tF74ZVig= X-Google-Smtp-Source: AA0mqf5xKZXwiyyhO/zo9eK4dRjP4rgOVoVKHQa9JmqD0voRoKd8SKwLBzSPFyKtmFKyJFubisc+Qg== X-Received: by 2002:a7b:c046:0:b0:3cf:d58f:f66e with SMTP id u6-20020a7bc046000000b003cfd58ff66emr49743817wmc.165.1670262642269; Mon, 05 Dec 2022 09:50:42 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bg2-20020a05600c3c8200b003a3170a7af9sm21231928wmb.4.2022.12.05.09.50.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:41 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:29 +0000 Subject: [PATCH v3 01/11] protocol v2: add server-side "bundle-uri" skeleton MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 command with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++++++++++ bundle-uri.c | 36 ++++++ bundle-uri.h | 7 ++ serve.c | 6 + t/t5701-git-serve.sh | 40 +++++- 5 files changed, 289 insertions(+), 1 deletion(-) diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt index 59bf41cefb9..10bd2d40cec 100644 --- a/Documentation/gitprotocol-v2.txt +++ b/Documentation/gitprotocol-v2.txt @@ -578,6 +578,207 @@ and associated requested information, each separated by a single space. obj-info = obj-id SP obj-size +bundle-uri +~~~~~~~~~~ + +If the 'bundle-uri' capability is advertised, the server supports the +`bundle-uri' command. + +The capability is currently advertised with no value (i.e. not +"bundle-uri=somevalue"), a value may be added in the future for +supporting command-wide extensions. Clients MUST ignore any unknown +capability values and proceed with the 'bundle-uri` dialog they +support. + +The 'bundle-uri' command is intended to be issued before `fetch` to +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and +inform the subsequent `fetch` command. + +The client CAN issue `bundle-uri` before or after any other valid +command. To be useful to clients it's expected that it'll be issued +after an `ls-refs` and before `fetch`, but CAN be issued at any time +in the dialog. + +DISCUSSION of bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^^ + +The intent of the feature is optimize for server resource consumption +in the common case by changing the common case of fetching a very +large PACK during linkgit:git-clone[1] into a smaller incremental +fetch. + +It also allows servers to achieve better caching in combination with +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]). + +By having new clones or fetches be a more predictable and common +negotiation against the tips of recently produces *.bundle file(s). +Servers might even pre-generate the results of such negotiations for +the `uploadpack.packObjectsHook` as new pushes come in. + +One way that servers could take advantage of these bundles is that the +server would anticipate that fresh clones will download a known bundle, +followed by catching up to the current state of the repository using ref +tips found in that bundle (or bundles). + +PROTOCOL for bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^ + +A `bundle-uri` request takes no arguments, and as noted above does not +currently advertise a capability value. Both may be added in the +future. + +When the client issues a `command=bundle-uri` request, the response is a +list of key-value pairs provided as packet lines with value +`=`. Each `` should be interpreted as a config key from +the `bundle.*` namespace to construct a list of bundles. These keys are +grouped by a `bundle..` subsection, where each key corresponding to a +given `` contributes attributes to the bundle defined by that ``. +See linkgit:git-config[1] for the specific details of these keys and how +the Git client will interpret their values. + +Clients MUST parse the line according to the above format, lines that do +not conform to the format SHOULD be discarded. The user MAY be warned in +such a case. + +bundle-uri CLIENT AND SERVER EXPECTATIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +URI CONTENTS:: +The content at the advertised URIs MUST be one of two types. ++ +The advertised URI may contain a bundle file that `git bundle verify` +would accept. I.e. they MUST contain one or more reference tips for +use by the client, MUST indicate prerequisites (in any) with standard +"-" prefixes, and MUST indicate their "object-format", if +applicable. ++ +The advertised URI may alternatively contain a plaintext file that `git +config --list` would accept (with the `--file` option). The key-value +pairs in this list are in the `bundle.*` namespace (see +linkgit:git-config[1]). + +bundle-uri CLIENT ERROR RECOVERY:: +A client MUST above all gracefully degrade on errors, whether that +error is because of bad missing/data in the bundle URI(s), because +that client is too dumb to e.g. understand and fully parse out bundle +headers and their prerequisite relationships, or something else. ++ +Server operators should feel confident in turning on "bundle-uri" and +not worry if e.g. their CDN goes down that clones or fetches will run +into hard failures. Even if the server bundle bundle(s) are +incomplete, or bad in some way the client should still end up with a +functioning repository, just as if it had chosen not to use this +protocol extension. ++ +All subsequent discussion on client and server interaction MUST keep +this in mind. + +bundle-uri SERVER TO CLIENT:: +The ordering of the returned bundle uris is not significant. Clients +MUST parse their headers to discover their contained OIDS and +prerequisites. A client MUST consider the content of the bundle(s) +themselves and their header as the ultimate source of truth. ++ +A server MAY even return bundle(s) that don't have any direct +relationship to the repository being cloned (either through accident, +or intentional "clever" configuration), and expect a client to sort +out what data they'd like from the bundle(s), if any. + +bundle-uri CLIENT TO SERVER:: +The client SHOULD provide reference tips found in the bundle header(s) +as 'have' lines in any subsequent `fetch` request. A client MAY also +ignore the bundle(s) entirely if doing so is deemed worse for some +reason, e.g. if the bundles can't be downloaded, it doesn't like the +tips it finds etc. + +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION:: +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s) +of the bundle(s) the client finds that the ref tips it wants can be +retrieved entirely from advertised bundle(s), the client MAY disconnect +from the Git server. The results of such a 'clone' or 'fetch' should be +indistinguishable from the state attained without using bundle-uri. + +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY:: +A client MAY perform an early disconnect while still downloading the +bundle(s) (having streamed and parsed their headers). In such a case +the client MUST gracefully recover from any errors related to +finishing the download and validation of the bundle(s). ++ +I.e. a client might need to re-connect and issue a 'fetch' command, +and possibly fall back to not making use of 'bundle-uri' at all. ++ +This "MAY" behavior is specified as such (and not a "SHOULD") on the +assumption that a server advertising bundle uris is more likely than +not to be serving up a relatively large repository, and to be pointing +to URIs that have a good chance of being in working order. A client +MAY e.g. look at the payload size of the bundles as a heuristic to see +if an early disconnect is worth it, should falling back on a full +"fetch" dialog be necessary. + +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION:: +A client SHOULD commence a negotiation of a PACK from the server via +the "fetch" command using the OID tips found in advertised bundles, +even if's still in the process of downloading those bundle(s). ++ +This allows for aggressive early disconnects from any interactive +server dialog. The client blindly trusts that the advertised OID tips +are relevant, and issues them as 'have' lines, it then requests any +tips it would like (usually from the "ls-refs" advertisement) via +'want' lines. The server will then compute a (hopefully small) PACK +with the expected difference between the tips from the bundle(s) and +the data requested. ++ +The only connection the client then needs to keep active is to the +concurrently downloading static bundle(s), when those and the +incremental PACK are retrieved they should be inflated and +validated. Any errors at this point should be gracefully recovered +from, see above. + +bundle-uri PROTOCOL FEATURES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The client constructs a bundle list from the `=` pairs +provided by the server. These pairs are part of the `bundle.*` namespace +as documented in linkgit:git-config[1]. In this section, we discuss some +of these keys and describe the actions the client will do in response to +this information. + +In particular, the `bundle.version` key specifies an integer value. The +only accepted value at the moment is `1`, but if the client sees an +unexpected value here then the client MUST ignore the bundle list. + +As long as `bundle.version` is understood, all other unknown keys MAY be +ignored by the client. The server will guarantee compatibility with older +clients, though newer clients may be better able to use the extra keys to +minimize downloads. + +Any backwards-incompatible addition of pre-URI key-value will be +guarded by a new `bundle.version` value or values in 'bundle-uri' +capability advertisement itself, and/or by new future `bundle-uri` +request arguments. + +Some example key-value pairs that are not currently implemented but could +be implemented in the future include: + + * Add a "hash=" or "size=" advertise the expected hash or + size of the bundle file. + + * Advertise that one or more bundle files are the same (to e.g. have + clients round-robin or otherwise choose one of N possible files). + + * A "oid=" shortcut and "prerequisite=" shortcut. For + expressing the common case of a bundle with one tip and no + prerequisites, or one tip and one prerequisite. ++ +This would allow for optimizing the common case of servers who'd like +to provide one "big bundle" containing only their "main" branch, +and/or incremental updates thereof. ++ +A client receiving such a a response MAY assume that they can skip +retrieving the header from a bundle at the indicated URI, and thus +save themselves and the server(s) the request(s) needed to inspect the +headers of that bundle or bundles. + GIT --- Part of the linkgit:git[1] suite diff --git a/bundle-uri.c b/bundle-uri.c index 79a914f961b..32022595964 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -563,6 +563,42 @@ cleanup: return result; } +/** + * API for serve.c. + */ + +int bundle_uri_advertise(struct repository *r, struct strbuf *value) +{ + static int advertise_bundle_uri = -1; + + if (advertise_bundle_uri != -1) + goto cached; + + advertise_bundle_uri = 0; + git_config_get_maybe_bool("uploadpack.advertisebundleuris", &advertise_bundle_uri); + +cached: + return advertise_bundle_uri; +} + +int bundle_uri_command(struct repository *r, + struct packet_reader *request) +{ + struct packet_writer writer; + packet_writer_init(&writer, 1); + + while (packet_reader_read(request) == PACKET_READ_NORMAL) + die(_("bundle-uri: unexpected argument: '%s'"), request->line); + if (request->status != PACKET_READ_FLUSH) + die(_("bundle-uri: expected flush after arguments")); + + /* TODO: Implement the communication */ + + packet_writer_flush(&writer); + + return 0; +} + /** * General API for {transport,connect}.c etc. */ diff --git a/bundle-uri.h b/bundle-uri.h index 4dbc269823c..357111ecce8 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -4,6 +4,7 @@ #include "hashmap.h" #include "strbuf.h" +struct packet_reader; struct repository; struct string_list; @@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * API for serve.c. + */ +int bundle_uri_advertise(struct repository *r, struct strbuf *value); +int bundle_uri_command(struct repository *r, struct packet_reader *request); + /** * General API for {transport,connect}.c etc. */ diff --git a/serve.c b/serve.c index 733347f602a..cbf4a143cfe 100644 --- a/serve.c +++ b/serve.c @@ -7,6 +7,7 @@ #include "protocol-caps.h" #include "serve.h" #include "upload-pack.h" +#include "bundle-uri.h" static int advertise_sid = -1; static int client_hash_algo = GIT_HASH_SHA1; @@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = { .advertise = always_advertise, .command = cap_object_info, }, + { + .name = "bundle-uri", + .advertise = bundle_uri_advertise, + .command = bundle_uri_command, + }, }; void protocol_v2_advertise_capabilities(void) diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index 1896f671cb3..f21e5e9d33d 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' ' wrong_algo sha1:sha256 wrong_algo sha256:sha1 EOF - cat >expect <<-EOF && + cat >expect.base <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) ls-refs=unborn @@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' ' server-option object-format=$(test_oid algo) object-info + EOF + cat >expect.trailer <<-EOF && 0000 EOF + cat expect.base expect.trailer >expect && GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ --advertise-capabilities >out && @@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' ' + test_config uploadpack.advertiseBundleURIs true && + + cat >expect.extra <<-EOF && + bundle-uri + EOF + cat expect.base \ + expect.extra \ + expect.trailer >expect && + + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ + --advertise-capabilities >out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: dies if not enabled' ' + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: invalid command '"'"'bundle-uri'"'"' + EOF + + cat >expect <<-\EOF && + ERR serve: invalid command '"'"'bundle-uri'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + test_done From patchwork Mon Dec 5 17:50:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13064944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FD1CC4332F for ; Mon, 5 Dec 2022 17:52:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232829AbiLERwl (ORCPT ); Mon, 5 Dec 2022 12:52:41 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44554 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232397AbiLERvk (ORCPT ); Mon, 5 Dec 2022 12:51:40 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AA45A60CB for ; Mon, 5 Dec 2022 09:50:44 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id w15so19802051wrl.9 for ; Mon, 05 Dec 2022 09:50:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=HDKpSmBn+STnbwuijw/6VlfTrIRU0b/XAk8TQ+voHE0=; b=eJdCJ0cIwRHv/iZvrB/6NkRESnsJC8+CqHifnqVx6JfTvPcL3y0i/GwdoB53oG7IaX 3qvEmQBr1Zui+P8OlZFWnaW8XkUq9fZwq8SIF1q541xLd3SEj8LOm0lzKhsw/vi9VJZ1 +IxhOL3fhchloHr7txPccUDROgCNwfw03rCS89M/pKMd9Mnvi2EjUHKuWCiWCkldIpXe SKlb3SVd7mcERX7iTZzwY6Y/0jXSMRFjJDh5zP85hGAtFEpswQrPD15Eu45xtlZIFC6l rfAhX97amMm87dao3y7tT/6Ar4zTBwnd7sC42dip1NEl6602u+3arLVCVo4+2qv7MkSG hURg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HDKpSmBn+STnbwuijw/6VlfTrIRU0b/XAk8TQ+voHE0=; b=lvsWQjZNY50FxsiHQzadgM9yWNUjQEwrzY/NpAzYpOQh3JROrRya6wW/vC+pC9h6K4 syEaPWM6Mi7ncjXqaSyNDEVI/wrfWbOIYePJygnFyXhoBkwTgD5GlCwhsoUr4DmUhabZ HjJd6chkDkZWrMZmQRF2/aSeBJgTxVKUkQaLJ+CeZ6KdAwrGPBVrwEgEUnOwmTX5BooT MO9glm3+0nXJw1YBKzWwOIJ11bbjmE0QENhGFYTuQhPLy3KK13mXxeLI2AAY73qfZBRY jcqYDk3S04wCXVZeevfZ4/aHOeRbCRk6ZL0UQ0NoAO424lpJxZGAddYum9mYfefYIKnc ow8A== X-Gm-Message-State: ANoB5plFuqdz6ns9t2+6U6ST7rzHXc4yy79HML5mriiotXKfRVWHqLNb K6MoZlAAELJizV9GppP3OFR+rXIsWU4= X-Google-Smtp-Source: AA0mqf7c4k1j/WjJg0jFYvtjm88P2YT8t/pHaMx1SAFRoEbiaajEHp6w+2AHI+UlWWkLfl21KUO/CQ== X-Received: by 2002:a5d:4a4c:0:b0:242:322c:c098 with SMTP id v12-20020a5d4a4c000000b00242322cc098mr5669670wrs.210.1670262643028; Mon, 05 Dec 2022 09:50:43 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id bi18-20020a05600c3d9200b003c65c9a36dfsm18098934wmb.48.2022.12.05.09.50.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:42 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:30 +0000 Subject: [PATCH v3 02/11] t: create test harness for 'bundle-uri' command MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= The previous change allowed for a Git server to advertise the 'bundle-uri' command as a capability based on the uploadPack.advertiseBundleURIs config option. Create a set of tests that check that this capability is advertised using 'git ls-remote'. In order to test this functionality across three protocols (file, git, and http), create lib-bundle-uri-protocol.sh to generalize the tests, allowing the other test scripts to set an environment variable and otherwise inherit the setup and tests from this script. The tests currently only test that the 'bundle-uri' command is advertised or not. Other actions will be tested as the Git client learns to request the 'bundle-uri' command and parse its response. To help with URI escaping, specifically for file paths with a space in them, extract a 'sed' invocation from t9199-git-svn-info.sh into a helper function for use here, too. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- t/lib-bundle-uri-protocol.sh | 85 ++++++++++++++++++++++++++ t/t5730-protocol-v2-bundle-uri-file.sh | 17 ++++++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++++++ t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++++++ t/t9119-git-svn-info.sh | 2 +- t/test-lib-functions.sh | 7 +++ 6 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 t/lib-bundle-uri-protocol.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh new file mode 100644 index 00000000000..2da22a39cb8 --- /dev/null +++ b/t/lib-bundle-uri-protocol.sh @@ -0,0 +1,85 @@ +# Set up and run tests of the 'bundle-uri' command in protocol v2 +# +# The test that includes this script should set BUNDLE_URI_PROTOCOL +# to one of "file", "git", or "http". + +BUNDLE_URI_TEST_PARENT= +BUNDLE_URI_TEST_URI= +BUNDLE_URI_TEST_BUNDLE_URI= +case "$BUNDLE_URI_PROTOCOL" in +file) + BUNDLE_URI_PARENT=file_parent + BUNDLE_URI_REPO_URI="file://$PWD/file_parent" + BUNDLE_URI_BUNDLE_URI="$BUNDLE_URI_REPO_URI/fake.bdl" + test_set_prereq BUNDLE_URI_FILE + ;; +git) + . "$TEST_DIRECTORY"/lib-git-daemon.sh + start_git_daemon --export-all --enable=receive-pack + BUNDLE_URI_PARENT="$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent" + BUNDLE_URI_REPO_URI="$GIT_DAEMON_URL/parent" + BUNDLE_URI_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq BUNDLE_URI_GIT + ;; +http) + . "$TEST_DIRECTORY"/lib-httpd.sh + start_httpd + BUNDLE_URI_PARENT="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" + BUNDLE_URI_REPO_URI="$HTTPD_URL/smart/http_parent" + BUNDLE_URI_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq BUNDLE_URI_HTTP + ;; +*) + BUG "Need to pass valid BUNDLE_URI_PROTOCOL (was \"$BUNDLE_URI_PROTOCOL\")" + ;; +esac + +test_expect_success "setup protocol v2 $BUNDLE_URI_PROTOCOL:// tests" ' + git init "$BUNDLE_URI_PARENT" && + test_commit -C "$BUNDLE_URI_PARENT" one && + git -C "$BUNDLE_URI_PARENT" config uploadpack.advertiseBundleURIs true +' + +case "$BUNDLE_URI_PROTOCOL" in +http) + test_expect_success "setup config for $BUNDLE_URI_PROTOCOL:// tests" ' + git -C "$BUNDLE_URI_PARENT" config http.receivepack true + ' + ;; +*) + ;; +esac +BUNDLE_URI_BUNDLE_URI_ESCAPED=$(echo "$BUNDLE_URI_BUNDLE_URI" | test_uri_escape) + +test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: no bundle-uri" ' + test_when_finished "rm -f log" && + test_when_finished "git -C \"$BUNDLE_URI_PARENT\" config uploadpack.advertiseBundleURIs true" && + git -C "$BUNDLE_URI_PARENT" config uploadpack.advertiseBundleURIs false && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$BUNDLE_URI_REPO_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + ! grep bundle-uri log +' + +test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: have bundle-uri" ' + test_when_finished "rm -f log" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$BUNDLE_URI_REPO_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log +' diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh new file mode 100755 index 00000000000..37bdb725bca --- /dev/null +++ b/t/t5730-protocol-v2-bundle-uri-file.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'file://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'file://' transport +# +BUNDLE_URI_PROTOCOL=file +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh new file mode 100755 index 00000000000..8add1b37abc --- /dev/null +++ b/t/t5731-protocol-v2-bundle-uri-git.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +BUNDLE_URI_PROTOCOL=git +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh new file mode 100755 index 00000000000..129daa02269 --- /dev/null +++ b/t/t5732-protocol-v2-bundle-uri-http.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'http://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'http://' transport +# +BUNDLE_URI_PROTOCOL=http +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t9119-git-svn-info.sh b/t/t9119-git-svn-info.sh index 8201c3e808a..088d1c57a88 100755 --- a/t/t9119-git-svn-info.sh +++ b/t/t9119-git-svn-info.sh @@ -28,7 +28,7 @@ test_cmp_info () { rm -f tmp.expect tmp.actual } -quoted_svnrepo="$(echo $svnrepo | sed 's/ /%20/')" +quoted_svnrepo="$(echo $svnrepo | test_uri_escape)" test_expect_success 'setup repository and import' ' mkdir info && diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index 29d914a12ba..5f6966a404b 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1755,6 +1755,13 @@ test_path_is_hidden () { return 1 } +# Poor man's URI escaping. Good enough for the test suite whose trash +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art. +test_uri_escape() { + sed 's/ /%20/g' +} + # Check that the given command was invoked as part of the # trace2-format trace on stdin. # From patchwork Mon Dec 5 17:50:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13064947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FAF1C4332F for ; Mon, 5 Dec 2022 17:52:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232896AbiLERwt (ORCPT ); Mon, 5 Dec 2022 12:52:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44558 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232745AbiLERvk (ORCPT ); Mon, 5 Dec 2022 12:51:40 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 729C060E3 for ; Mon, 5 Dec 2022 09:50:45 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id c65-20020a1c3544000000b003cfffd00fc0so12341328wma.1 for ; Mon, 05 Dec 2022 09:50:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=kqYBa7AYck1KXjiGuWcjJXZHxBlNJrFfsuzq+UJ+C08=; b=Fpwfg35vk2eri9tQ4QTY8Q8lIqZF5yswAtRSHH8cQGvW64oZXeZf0L/9H/P47PYyXY UhKfk8PfDsBcI6801eCXSeggds5nyc3JhJDN/r0TZjqrI8D2uGMQf/snJ7i4c8yeX5yU aqXfHNICHmC/pTd69KRIYTuA8ur/ZOe6p/Ft976TpQ+bE/7BgQS0NBHWDQP453iU8+zd TENTGWuouOMqoi2CgsWyoB7eRh7qMUaG5gUCbZgBMbjKUelEV3FTExEivAps/1qemX6C uNB30KBQGCkywTo+fFIBmT5AfMX3EuvjBOWrpjYx26Zq0QwYmjae+d2QU0zmtQN4MBcY WThA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kqYBa7AYck1KXjiGuWcjJXZHxBlNJrFfsuzq+UJ+C08=; b=BGiHnNdopOSoPLx82uZu77pdM8U2IfSwgvlRusr+Ob5XKzfzZKS1BXrFhl73dKTuh+ QRtgaG0DhdR29lSHDoHLvhTNiCMP23pRa/k+bp4MCr3HoS6C26H4MtjprSMC4YrMp2eS P8ZpVqHzAsw2r21lX8odHh+OLKUFBLvXF0+EWVBR+HDPMhD1SELT2kuisHS7WzrU2iW3 jyRjExkjhJAX3n8rKf5RY7VDphS2BZcG0DHPdZKQiTTW6LXc8uC88DVn3Yh6FbvUU+tQ +BzHCnQYg10uLif44IFjT3JIZa0fkgE9yZFkTk0aOeKUykREoJIaBplEYWG5GlN9r3mU hjUw== X-Gm-Message-State: ANoB5pmfz7XHlVJvkIm8+HFpBnJkgeY+6aNskXdxqSqgtgepsr89tUSx 6dIpFVvwi4wqEOsKZAGLQtvlo7G5e/c= X-Google-Smtp-Source: AA0mqf6sTkMBMCFappqIbr5nauXen2esGOrcJdn9fbfowi6zvHVh+DsE25b5Sp2OFbwITpWWx/0aBg== X-Received: by 2002:a05:600c:3109:b0:3cf:5731:53db with SMTP id g9-20020a05600c310900b003cf573153dbmr65537575wmo.85.1670262643799; Mon, 05 Dec 2022 09:50:43 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g9-20020a05600c000900b003cfd58409desm21016480wmc.13.2022.12.05.09.50.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:43 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:31 +0000 Subject: [PATCH v3 03/11] clone: request the 'bundle-uri' command when available MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Set up all the needed client parts of the 'bundle-uri' protocol v2 command, without actually doing anything with the bundle URIs. If the server says it supports 'bundle-uri' teach Git to issue the 'bundle-uri' command after the 'ls-refs' during 'git clone'. The returned key=value pairs are passed to the bundle list code which is tested using a different ingest mechanism in t5750-bundle-uri-parse.sh. At this point, Git does nothing with that bundle list. It will not download any of the bundles. That will come in a later change after these protocol bits are finalized. The no-op client is initially used only by 'git clone' to test the basic functionality, and eventually will bootstrap the initial download of Git objects during a fresh clone. The bundle URI client will not be integrated into other fetches until a mechanism is created to select a subset of bundles for download. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- builtin/clone.c | 6 +++++ connect.c | 44 +++++++++++++++++++++++++++++++ remote.h | 5 ++++ t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++ transport-helper.c | 13 +++++++++ transport-internal.h | 7 +++++ transport.c | 51 ++++++++++++++++++++++++++++++++++++ transport.h | 19 ++++++++++++++ 8 files changed, 164 insertions(+) diff --git a/builtin/clone.c b/builtin/clone.c index 547d6464b3c..39364c25b15 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1266,6 +1266,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/connect.c b/connect.c index 5ea53deda23..624a10f18ee 100644 --- a/connect.c +++ b/connect.c @@ -15,6 +15,7 @@ #include "version.h" #include "protocol.h" #include "alias.h" +#include "bundle-uri.h" static char *server_capabilities_v1; static struct strvec server_capabilities_v2 = STRVEC_INIT; @@ -491,6 +492,49 @@ static void send_capabilities(int fd_out, struct packet_reader *reader) } } +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc) +{ + int line_nr = 1; + + /* Assert bundle-uri support */ + server_supports_v2("bundle-uri", 1); + + /* (Re-)send capabilities */ + send_capabilities(fd_out, reader); + + /* Send command */ + packet_write_fmt(fd_out, "command=bundle-uri\n"); + packet_delim(fd_out); + + packet_flush(fd_out); + + /* Process response from server */ + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { + const char *line = reader->line; + line_nr++; + + if (!bundle_uri_parse_line(bundles, line)) + continue; + + return error(_("error on bundle-uri response line %d: %s"), + line_nr, line); + } + + if (reader->status != PACKET_READ_FLUSH) + return error(_("expected flush after bundle-uri listing")); + + /* + * Might die(), but obscure enough that that's OK, e.g. in + * serve.c we'll call BUG() on its equivalent (the + * PACKET_READ_RESPONSE_END check). + */ + check_stateless_delimiter(stateless_rpc, reader, + _("expected response end packet after ref listing")); + + return 0; +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, diff --git a/remote.h b/remote.h index 1c4621b414b..1ebbe42792e 100644 --- a/remote.h +++ b/remote.h @@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, const struct string_list *server_options, int stateless_rpc); +/* Used for protocol v2 in order to retrieve refs from a remote */ +struct bundle_list; +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc); + int resolve_remote_symref(struct ref *ref, struct ref *list); /* diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 2da22a39cb8..d44c6e10f9e 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -83,3 +83,22 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav # Server advertised bundle-uri capability grep "< bundle-uri" log ' + +test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' + test_when_finished "rm -rf log cloned" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + clone "$BUNDLE_URI_REPO_URI" cloned \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log && + + # Client issued bundle-uri command + grep "> command=bundle-uri" log +' diff --git a/transport-helper.c b/transport-helper.c index e95267a4ab5..3ea7c2bb5ad 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport, return ret; } +static int get_bundle_uri(struct transport *transport) +{ + get_helper(transport); + + if (process_connect(transport, 0)) { + do_take_over(transport); + return transport->vtable->get_bundle_uri(transport); + } + + return -1; +} + static struct transport_vtable vtable = { .set_option = set_helper_option, .get_refs_list = get_refs_list, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs, .push_refs = push_refs, .connect = connect_helper, diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733ac..90ea749e5cf 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -26,6 +26,13 @@ struct transport_vtable { struct ref *(*get_refs_list)(struct transport *transport, int for_push, struct transport_ls_refs_options *transport_options); + /** + * Populates the remote side's bundle-uri under protocol v2, + * if the "bundle-uri" capability was advertised. Returns 0 if + * OK, negative values on error. + */ + int (*get_bundle_uri)(struct transport *transport); + /** * Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. diff --git a/transport.c b/transport.c index e7b97194c10..b6f279e92cb 100644 --- a/transport.c +++ b/transport.c @@ -22,6 +22,7 @@ #include "protocol.h" #include "object-store.h" #include "color.h" +#include "bundle-uri.h" static int transport_use_color = -1; static char transport_colors[][COLOR_MAXLEN] = { @@ -359,6 +360,32 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus return handshake(transport, for_push, options, 1); } +static int get_bundle_uri(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct packet_reader reader; + int stateless_rpc = transport->stateless_rpc; + + if (!transport->bundles) { + CALLOC_ARRAY(transport->bundles, 1); + init_bundle_list(transport->bundles); + } + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF); + + return get_remote_bundle_uri(data->fd[1], &reader, + transport->bundles, stateless_rpc); +} + static int fetch_refs_via_pack(struct transport *transport, int nr_heads, struct ref **to_fetch) { @@ -902,6 +929,7 @@ static int disconnect_git(struct transport *transport) static struct transport_vtable taken_over_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .disconnect = disconnect_git @@ -1054,6 +1082,7 @@ static struct transport_vtable bundle_vtable = { static struct transport_vtable builtin_smart_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .connect = connect_git, @@ -1068,6 +1097,9 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->progress = isatty(2); string_list_init_dup(&ret->pack_lockfiles); + CALLOC_ARRAY(ret->bundles, 1); + init_bundle_list(ret->bundles); + if (!remote) BUG("No remote provided to transport_get()"); @@ -1482,6 +1514,23 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } +int transport_get_remote_bundle_uri(struct transport *transport) +{ + const struct transport_vtable *vtable = transport->vtable; + + /* Check config only once. */ + if (transport->got_remote_bundle_uri) + return 0; + transport->got_remote_bundle_uri = 1; + + if (!vtable->get_bundle_uri) + return error(_("bundle-uri operation not supported by protocol")); + + if (vtable->get_bundle_uri(transport) < 0) + return error(_("could not retrieve server-advertised bundle-uri list")); + return 0; +} + void transport_unlock_pack(struct transport *transport, unsigned int flags) { int in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER); @@ -1512,6 +1561,8 @@ int transport_disconnect(struct transport *transport) ret = transport->vtable->disconnect(transport); if (transport->got_remote_refs) free_refs((void *)transport->remote_refs); + clear_bundle_list(transport->bundles); + free(transport->bundles); free(transport); return ret; } diff --git a/transport.h b/transport.h index b5bf7b3e704..85150f504fb 100644 --- a/transport.h +++ b/transport.h @@ -62,6 +62,7 @@ enum transport_family { TRANSPORT_FAMILY_IPV6 }; +struct bundle_list; struct transport { const struct transport_vtable *vtable; @@ -76,6 +77,18 @@ struct transport { */ unsigned got_remote_refs : 1; + /** + * Indicates whether we already called get_bundle_uri_list(); set by + * transport.c::transport_get_remote_bundle_uri(). + */ + unsigned got_remote_bundle_uri : 1; + + /* + * The results of "command=bundle-uri", if both sides support + * the "bundle-uri" capability. + */ + struct bundle_list *bundles; + /* * Transports that call take-over destroys the data specific to * the transport type while doing so, and cannot be reused. @@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts); const struct ref *transport_get_remote_refs(struct transport *transport, struct transport_ls_refs_options *transport_options); +/** + * Retrieve bundle URI(s) from a remote. Populates "struct + * transport"'s "bundle_uri" and "got_remote_bundle_uri". + */ +int transport_get_remote_bundle_uri(struct transport *transport); + /* * Fetch the hash algorithm used by a remote. * From patchwork Mon Dec 5 17:50:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13064943 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8111AC4332F for ; Mon, 5 Dec 2022 17:52:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232284AbiLERwi (ORCPT ); Mon, 5 Dec 2022 12:52:38 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44300 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232748AbiLERvk (ORCPT ); Mon, 5 Dec 2022 12:51:40 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3B8DB55A3 for ; Mon, 5 Dec 2022 09:50:46 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id l26so3118465wms.4 for ; Mon, 05 Dec 2022 09:50:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=R9sGu4mNVPiGzjCsbbEkFX5eqQNP2vNJY/ftRlETupo=; b=PpTJDRAvDEAQ6yBbAbYxrrCiwJax6lRBp3zCGeOQb0axs+jwpZy+ycBSe2E+Z9f9Zl NN5ujuzYh6SRGfbpJvGg9+qYq84cwBt0ao5Ook0fOK9TXmX708M60d8JrJELl5qct028 3FAaGfGhISE37y0NFCZMaPZsDk3iBQJ0aqRMNNP8WN2mjoc094G//M11FmXGOjWN+Osq rKX+LvWuSA77p9ycLLgA/s7ST2g93gRY1yB8Dxbq4oh3gI3cN+dsAAEVT1MyFqxoUkUu MLPw2eyNZ/3QZ8MPLl6mD/W+07ivTIfpP2+8po168pB65Wohc4uXkcFuKZNMPCn//L1e j2sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=R9sGu4mNVPiGzjCsbbEkFX5eqQNP2vNJY/ftRlETupo=; b=CLRwsyxiWle0ic3VLe33MShdG0AJQ13/8RQceXbzIzw4UwzNCSKWtKM5KQEBsNdoTR LAe2Al/glEVlkbGaS7zZo8ffXJIQfzhyQu68bQu3j2MCVRa20OvahDNBJEDEaWNS2RcL gPm4S/LImHPgeHFliflmCaAFolRZRE1YeKxj71JOVw0xZ3TB58GD+05MkBDDHRfDbOLQ 0xPByYsu1KM5+QH9nBWDG2Mg0WJklZtjkj06EKuzE5BA67bYgtlXystffrKcCZgUIZUI hRq1QQqYA9TK6ZYBvC5/j1MxZhogBDwrpQbwB3oxGOSOuAlEgNh4yPpc45fpm2amnIhR 4Rkg== X-Gm-Message-State: ANoB5pkYDlmsQFpH3Hu4pySrJfzEW8gvxp41P6fuSrSpR4RvhAAokRLG nIrQO7zSDiHkoMaaBlL7OdE1MlGcADI= X-Google-Smtp-Source: AA0mqf7YTN3rj+1XmprmN4S2Z0wZlgzzVjRr8T5wtviO+np6xzRlSkbZw3xHcudZxE9hWZVbeMgkfg== X-Received: by 2002:a05:600c:1e0f:b0:3cf:e9a4:3261 with SMTP id ay15-20020a05600c1e0f00b003cfe9a43261mr65589428wmb.41.1670262644634; Mon, 05 Dec 2022 09:50:44 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s11-20020a5d424b000000b00241cfe6e286sm15186020wrr.98.2022.12.05.09.50.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:44 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:32 +0000 Subject: [PATCH v3 04/11] bundle-uri client: add boolean transfer.bundleURI setting MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= The yet-to-be introduced client support for bundle-uri will always fall back on a full clone, but we'd still like to be able to ignore a server's bundle-uri advertisement entirely. The new transfer.bundleURI config option defaults to 'false', but a user can set it to 'true' to enable checking for bundle URIs from the origin Git server using protocol v2. To enable this setting by default in the correct tests, add a GIT_TEST_BUNDLE_URI environment variable. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/config/transfer.txt | 6 ++++++ t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++++++- transport.c | 9 +++++++++ 3 files changed, 33 insertions(+), 1 deletion(-) diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt index 264812cca4d..c3ac767d1e4 100644 --- a/Documentation/config/transfer.txt +++ b/Documentation/config/transfer.txt @@ -115,3 +115,9 @@ transfer.unpackLimit:: transfer.advertiseSID:: Boolean. When true, client and server processes will advertise their unique session IDs to their remote counterpart. Defaults to false. + +transfer.bundleURI:: + When `true`, local `git clone` commands will request bundle + information from the remote server (if advertised) and download + bundles before continuing the clone through the Git protocol. + Defaults to `false`. diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index d44c6e10f9e..77bfd4f0119 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -85,9 +85,10 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav ' test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' - test_when_finished "rm -rf log cloned" && + test_when_finished "rm -rf log cloned cloned2" && GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_BUNDLE_URI=0 \ git \ -c protocol.version=2 \ clone "$BUNDLE_URI_REPO_URI" cloned \ @@ -99,6 +100,22 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque # Server advertised bundle-uri capability grep "< bundle-uri" log && + # Client did not issue bundle-uri command + ! grep "> command=bundle-uri" log && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c transfer.bundleURI=true \ + -c protocol.version=2 \ + clone "$BUNDLE_URI_REPO_URI" cloned2 \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log && + # Client issued bundle-uri command grep "> command=bundle-uri" log ' diff --git a/transport.c b/transport.c index b6f279e92cb..9f9e38d66dd 100644 --- a/transport.c +++ b/transport.c @@ -1516,6 +1516,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) int transport_get_remote_bundle_uri(struct transport *transport) { + int value = 0; const struct transport_vtable *vtable = transport->vtable; /* Check config only once. */ @@ -1523,6 +1524,14 @@ int transport_get_remote_bundle_uri(struct transport *transport) return 0; transport->got_remote_bundle_uri = 1; + /* + * Don't request bundle-uri from the server unless configured to + * do so by GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true. + */ + if (!git_env_bool("GIT_TEST_BUNDLE_URI", 0) && + (git_config_get_bool("transfer.bundleuri", &value) || !value)) + return 0; + if (!vtable->get_bundle_uri) return error(_("bundle-uri operation not supported by protocol")); From patchwork Mon Dec 5 17:50:33 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13064949 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 167B8C4332F for ; Mon, 5 Dec 2022 17:52:58 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231862AbiLERwz (ORCPT ); Mon, 5 Dec 2022 12:52:55 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44294 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232754AbiLERvk (ORCPT ); Mon, 5 Dec 2022 12:51:40 -0500 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 13BC01081 for ; Mon, 5 Dec 2022 09:50:47 -0800 (PST) Received: by mail-wr1-x42d.google.com with SMTP id bs21so19828723wrb.4 for ; Mon, 05 Dec 2022 09:50:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=HCR2+6odN8ReYZxWt0ja68XfOzXoM3qdriRpywDUfYw=; b=nJ0Ta9Pmt/4okxbum/xu3BiaClWqd1rzCJi1FocfP9cXW1q99wgN5n42PXUkmanO41 pV4lxpsZHIv9HHYFHevj8XB/7PLFFUe3TD+GIaYKwn/tHT8ws7hKGDwHEUlxWGF/5vtl FbemQmlCHmPrjywwHh8PFvdrEtIGoXUx1K4wVMItR6cymhKyJkhmvGY8Swzkcfq1xFu/ aNaEIGCZ7ThIk3+T8Ny39EtVldL+8N63MX33lZw+5lb9GKU7+MhB8wzs6E/44nk6lae/ aTvfWQWpsu7NlzXwg8mbjmLJjMCcs632F4Av2Zm96xpFLIhvxlhR/9OqQZm4CnfAQKiG 2y9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HCR2+6odN8ReYZxWt0ja68XfOzXoM3qdriRpywDUfYw=; b=39mVgzCncHCFHnLUV20hgRlHsuF+mq3laSxnxeaMTvk6mkVoD7HU6b9tiGLXnRDNM7 ky6VUClD6RjCSRvfTDPoz1MtmUdXd+NbOdcehsZolGj4uotfk5fgRzui5wmqCGZNBiAW 762esbORLl/02Q58pQCeYzQBomYhQnwZ5tvXx+4U79PsN5fdMWh5LLLez595FYzeKcFB TkJoNsIBm23KpU5YFQ8B6glvqOvE2z+duwGtCpRTFLzKhYnNqzWD7QgY+Y1dYiYtdO+s v7L1MQle2VNZ5KG6lUS3BCigWESEElTqWaz0CtadJNVOhDk+T+a+lYpA0Tmboouq7hfA dGgg== X-Gm-Message-State: ANoB5pnjRr9EXlQv3EXL9l/vN95zjMRKpfeJLnywrJRcvyos8wCT15Mq OeWPEjAGIrno17cuDs8MNsxG38BaJq0= X-Google-Smtp-Source: AA0mqf7adApuABaNTGK7/bprMl9WZGOEAOX0eEKyO3Bq5hYv4rjaIuYNDKo+KVdC1SGh0QwKsq4IBQ== X-Received: by 2002:adf:eb0c:0:b0:236:6deb:4498 with SMTP id s12-20020adfeb0c000000b002366deb4498mr52274538wrn.52.1670262645409; Mon, 05 Dec 2022 09:50:45 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l41-20020a05600c1d2900b003d1e4f3ac8esm626969wms.33.2022.12.05.09.50.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:45 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:33 +0000 Subject: [PATCH v3 05/11] transport: rename got_remote_heads Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The 'got_remote_heads' member of 'struct git_transport_data' was used historically to indicate that the initial server connection was made and the ref advertisement was returned. With protocol v2, that initial handshake does not necessarily include the ref advertisement, so this member is not an accurate name. Thankfully, all uses of the member are only checking to see if the handshake should take place, not whether or not some local data has the ref advertisement. Rename the member to 'finished_handshake' to represent the proper state. Note that the variable is only set to 1 during the handshake() method. Signed-off-by: Derrick Stolee --- transport.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/transport.c b/transport.c index 9f9e38d66dd..a2281d95262 100644 --- a/transport.c +++ b/transport.c @@ -198,7 +198,7 @@ struct git_transport_data { struct git_transport_options options; struct child_process *conn; int fd[2]; - unsigned got_remote_heads : 1; + unsigned finished_handshake : 1; enum protocol_version version; struct oid_array extra_have; struct oid_array shallow; @@ -345,7 +345,7 @@ static struct ref *handshake(struct transport *transport, int for_push, case protocol_unknown_version: BUG("unknown protocol version"); } - data->got_remote_heads = 1; + data->finished_handshake = 1; transport->hash_algo = reader.hash_algo; if (reader.line_peeked) @@ -421,7 +421,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.negotiation_tips = data->options.negotiation_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow; - if (!data->got_remote_heads) { + if (!data->finished_handshake) { int i; int must_list_refs = 0; for (i = 0; i < nr_heads; i++) { @@ -461,7 +461,7 @@ static int fetch_refs_via_pack(struct transport *transport, to_fetch, nr_heads, &data->shallow, &transport->pack_lockfiles, data->version); - data->got_remote_heads = 0; + data->finished_handshake = 0; data->options.self_contained_and_connected = args.self_contained_and_connected; data->options.connectivity_checked = args.connectivity_checked; @@ -846,7 +846,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re if (transport_color_config() < 0) return -1; - if (!data->got_remote_heads) + if (!data->finished_handshake) get_refs_via_connect(transport, 1, NULL); memset(&args, 0, sizeof(args)); @@ -894,7 +894,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re else ret = finish_connect(data->conn); data->conn = NULL; - data->got_remote_heads = 0; + data->finished_handshake = 0; return ret; } @@ -914,7 +914,7 @@ static int disconnect_git(struct transport *transport) { struct git_transport_data *data = transport->data; if (data->conn) { - if (data->got_remote_heads && !transport->stateless_rpc) + if (data->finished_handshake && !transport->stateless_rpc) packet_flush(data->fd[1]); close(data->fd[0]); if (data->fd[1] >= 0) @@ -949,7 +949,7 @@ void transport_take_over(struct transport *transport, data->conn = child; data->fd[0] = data->conn->out; data->fd[1] = data->conn->in; - data->got_remote_heads = 0; + data->finished_handshake = 0; transport->data = data; transport->vtable = &taken_over_vtable; @@ -1150,7 +1150,7 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->smart_options = &(data->options); data->conn = NULL; - data->got_remote_heads = 0; + data->finished_handshake = 0; } else { /* Unknown protocol in URL. Pass to external handler. */ int len = external_specification_len(url); From patchwork Mon Dec 5 17:50:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13064946 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61AC8C4332F for ; Mon, 5 Dec 2022 17:52:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232855AbiLERwq (ORCPT ); Mon, 5 Dec 2022 12:52:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44302 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232332AbiLERvk (ORCPT ); Mon, 5 Dec 2022 12:51:40 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D11C260D4 for ; Mon, 5 Dec 2022 09:50:47 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id i131-20020a1c3b89000000b003d1cb516ce0so2915192wma.4 for ; Mon, 05 Dec 2022 09:50:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=KJx+nKCpe9ZLXTw+nG9R54gmSIYf+jjrN1a4A9Y4e08=; b=nFKQeLNnwE+KkIt98kdDRhpspIZuoNtpM3biFBeO7uSdEmBAhIkhE1ZIqjIOD5d0oX 9OP72mV+mN2I8fIjpWDk8PxFbNOG7HzcSfmxwjaYdc9VRrpUgm/l13QDOK0Oxi5bmQxD RLjGnG1xvqrAIL33qRYGBrP6QLi4i+5y/PV3bQ7dIcozCTQpE0zFIIAKIHm9ZbF6y+bw WYNuiuWG8s8jUVLN2R/B6iqMyMQpnhqRSEx8PXj3sqfuVmn8wxA4HGA70jmxQAIwUad5 gD5WQ3Ir48e54OmjShG8ze9zhF03LbaE11XST2I1nTr5n2861V1pXooTs2HuyAcJemxT PZew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KJx+nKCpe9ZLXTw+nG9R54gmSIYf+jjrN1a4A9Y4e08=; b=UwSt1GmTTLXY6RaJmH1uKxca4LRk7k+/NWhMZQrJ13ZtJAEA8APMSyzKacl3o3DVMb 98gY8L1lA/serruFrFUdxT9UrYNBgpUrzh/mlwJJQqPglI4LFhl8MCAvfndjZbAU7f6e DFV8tItj4KhuvASFw5x0L4F1r2vhlhKLMHoFxQYjRAVFlt1XfW2d4qXwkuJSQpIBknFq Ogsb1v1Nu/EMvpcfolexFsUlAZ234Qg6YDSr9znwkZmQ+n9JMp5iey3QbRXU85aYv03U hMBmyqQIdOFousq3OxfzQbOmeu7uAM0p3feBe2JgBlQBR1C4WduMHF7eps7C+iRRLMCA Ll+w== X-Gm-Message-State: ANoB5pke3a8yDmru164UzFtOJLDiYs3sjLWTHZVXL6nlkb6AyozHazmi MiyVnUBvUCjySvcK8qA3+5eFTLlbPIo= X-Google-Smtp-Source: AA0mqf7bQ7jKcOFR8zjqPEoKJiC0EIWeJqhIB2TNL0ssddh4EAzUDWndVKjKeVtu+dHGQ03SldvUHQ== X-Received: by 2002:a05:600c:35c8:b0:3cf:cf89:2f02 with SMTP id r8-20020a05600c35c800b003cfcf892f02mr49961763wmq.2.1670262646196; Mon, 05 Dec 2022 09:50:46 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id p15-20020a5d638f000000b0024207478de3sm14409685wru.93.2022.12.05.09.50.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:45 -0800 (PST) Message-Id: <46a58e83caf6d37e5a62b39d3c2e35e278211297.1670262639.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:34 +0000 Subject: [PATCH v3 06/11] bundle-uri client: add helper for testing server MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper for issuing protocol v2 "bundle-uri" commands to a server, and to the parsing routines in bundle-uri.c. In the "git clone" case we'll have already done the handshake(), but not here. Add an extra case to check for this handshake in get_bundle_uri() for ease of use for future callers. Rename the existing 'got_remote_heads' to 'finished_handshake' to make it more clear what that bit represents. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- t/helper/test-bundle-uri.c | 46 ++++++++++++++++++++++++++++++++++++ t/lib-bundle-uri-protocol.sh | 39 ++++++++++++++++++++++++++++++ transport.c | 7 ++++++ 3 files changed, 92 insertions(+) diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index 25afd393428..f8159187014 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -3,6 +3,10 @@ #include "bundle-uri.h" #include "strbuf.h" #include "string-list.h" +#include "transport.h" +#include "ref-filter.h" +#include "remote.h" +#include "refs.h" enum input_mode { KEY_VALUE_PAIRS, @@ -68,6 +72,46 @@ usage: usage_with_options(usage, options); } +static int cmd_ls_remote(int argc, const char **argv) +{ + const char *uploadpack = NULL; + struct string_list server_options = STRING_LIST_INIT_DUP; + const char *dest; + struct remote *remote; + struct transport *transport; + int status = 0; + + dest = argc > 1 ? argv[1] : NULL; + + remote = remote_get(dest); + if (!remote) { + if (dest) + die(_("bad repository '%s'"), dest); + die(_("no remote configured to get bundle URIs from")); + } + if (!remote->url_nr) + die(_("remote '%s' has no configured URL"), dest); + + transport = transport_get(remote, NULL); + if (uploadpack) + transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack); + if (server_options.nr) + transport->server_options = &server_options; + + if (transport_get_remote_bundle_uri(transport) < 0) { + error(_("could not get the bundle-uri list")); + status = 1; + goto cleanup; + } + + print_bundle_list(stdout, transport->bundles); + +cleanup: + if (transport_disconnect(transport)) + return 1; + return status; +} + int cmd__bundle_uri(int argc, const char **argv) { const char *usage[] = { @@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv) return cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS); if (!strcmp(argv[1], "parse-config")) return cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE); + if (!strcmp(argv[1], "ls-remote")) + return cmd_ls_remote(argc - 1, argv + 1); error("there is no test-tool bundle-uri tool '%s'", argv[1]); usage: diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 77bfd4f0119..88e339ae9ad 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -119,3 +119,42 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque # Client issued bundle-uri command grep "> command=bundle-uri" log ' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.only.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 and extra data" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.only.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED" && + + # Extra data should be ignored + test_config -C "$BUNDLE_URI_PARENT" bundle.only.extra bogus && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' diff --git a/transport.c b/transport.c index a2281d95262..97d395e10a3 100644 --- a/transport.c +++ b/transport.c @@ -371,6 +371,13 @@ static int get_bundle_uri(struct transport *transport) init_bundle_list(transport->bundles); } + if (!data->finished_handshake) { + struct ref *refs = handshake(transport, 0, NULL, 0); + + if (refs) + free_refs(refs); + } + /* * "Support" protocol v0 and v2 without bundle-uri support by * silently degrading to a NOOP. From patchwork Mon Dec 5 17:50:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13064942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47836C4332F for ; Mon, 5 Dec 2022 17:52:37 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229471AbiLERwe (ORCPT ); Mon, 5 Dec 2022 12:52:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44568 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232766AbiLERvl (ORCPT ); Mon, 5 Dec 2022 12:51:41 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A6CD7625E for ; Mon, 5 Dec 2022 09:50:48 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id h7so13796424wrs.6 for ; Mon, 05 Dec 2022 09:50:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=CnzbZmXRCbIZiehAtyBacIrTHxr9bywfhJLnH/5Lu34=; b=YPKpt9YFyKFYvLhkHusCQkhW1QiOdA+XqwmzB23ps0SXsQMHjRicZABLSCy6PT7Dxb fbrEdx7AJCNXafMKHpdJgxRpixore5YuRxazlbGOtaahQ7JmzG+vJeEwvMiA5bJ7f//Z W8KDTeGsZsuukssuvT2UpyoZuCZcDgudsVNOn/FZ4p2ePulrD+6NQUI145Rjmxkh0QfU acwtDAaEgotzJuEe1G0iscgRjF0BYCJCFPM/J862Psd7gzehXUUHIVlgNuCloO0xrbCe 2EhNvzBLnMsuQZbxTqU1mfppGXt9saKUOrtvg7TH8cKE9/ZM8Bxwx11pQtaMu4eDKGIQ DtYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CnzbZmXRCbIZiehAtyBacIrTHxr9bywfhJLnH/5Lu34=; b=g3Qi7OmPlMnVjy28fBEHvF0VXwLna+tV561oK9aBHwn6RXMmBP9xeqCKgJmey6gNpp D7n6UkK8b0c0Ru3oe8JdJnd8+1FLOsC5Os5xssGJ/Jn6WNV88tCrNQMWikcrg8+NHwBE /nTjk8zF1Lq9xDi6WZK8ZEwhxPP2X06FVIzP6RV2LaKqi8Y5Uf0idaW7zidLO1YtW/Ut f+JXxQzTkhjigbITGzHjAv6KhPOpDCyyPWft2aUmMtnbxppwvnhiLhQvMOn38gr9p91v DUz500R9ooxyt01cxbRFTAgQiNOYsWlmbYttYr0C7qK42RQKv2o770JA+gQULoapxqQF LIJw== X-Gm-Message-State: ANoB5pkrwaG1uQHnL0Skui/V8kML/yVhPej9Yvp7oacqLEKIwsVLarcV Lom5wBXeFQ6IdrjMT7/FzPNfDU5k0C8= X-Google-Smtp-Source: AA0mqf744g28B8P4z1DxGQIOp8dBh73g8SIHCWBHfGkKYK0BzeQ+tb2VUsDTBcvsseHdBkkkfKCxAg== X-Received: by 2002:adf:ce06:0:b0:242:436e:c69a with SMTP id p6-20020adfce06000000b00242436ec69amr9424661wrn.631.1670262647006; Mon, 05 Dec 2022 09:50:47 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id v6-20020adfe286000000b00241f467f888sm14758952wri.74.2022.12.05.09.50.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:46 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:35 +0000 Subject: [PATCH v3 07/11] bundle-uri: serve bundle.* keys from config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Implement the "bundle-uri" protocol v2 capability by populating the key=value packet lines from the local Git config. The list of bundles is provided from the keys beginning with "bundle.". In the future, we may want to filter this list to be more specific to the exact known keys that the server intends to share, but for flexibility at the moment we will assume that the config values are well-formed. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 +++++++++++++++- t/lib-bundle-uri-protocol.sh | 35 +++++++++++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 32022595964..6919f541085 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -581,6 +581,16 @@ cached: return advertise_bundle_uri; } +static int config_to_packet_line(const char *key, const char *value, void *data) +{ + struct packet_reader *writer = data; + + if (!strncmp(key, "bundle.", 7)) + packet_write_fmt(writer->fd, "%s=%s", key, value); + + return 0; +} + int bundle_uri_command(struct repository *r, struct packet_reader *request) { @@ -592,7 +602,11 @@ int bundle_uri_command(struct repository *r, if (request->status != PACKET_READ_FLUSH) die(_("bundle-uri: expected flush after arguments")); - /* TODO: Implement the communication */ + /* + * Read all "bundle.*" config lines to the client as key=value + * packet lines. + */ + git_config(config_to_packet_line, &writer); packet_writer_flush(&writer); diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 88e339ae9ad..6d3f871fa0f 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -129,8 +129,11 @@ test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol [bundle] version = 1 mode = all + [bundle "only"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED EOF + GIT_TEST_BUNDLE_URI=1 \ test-tool bundle-uri \ ls-remote \ "$BUNDLE_URI_REPO_URI" \ @@ -150,8 +153,40 @@ test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol [bundle] version = 1 mode = all + [bundle "only"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED EOF + GIT_TEST_BUNDLE_URI=1 \ + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 with list" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle1.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl" && + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle2.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl" && + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle3.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + [bundle "bundle1"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl + [bundle "bundle2"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl + [bundle "bundle3"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl + EOF + + GIT_TEST_BUNDLE_URI=1 \ test-tool bundle-uri \ ls-remote \ "$BUNDLE_URI_REPO_URI" \ From patchwork Mon Dec 5 17:50:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13064948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78B2CC47089 for ; Mon, 5 Dec 2022 17:52:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231213AbiLERwy (ORCPT ); Mon, 5 Dec 2022 12:52:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44560 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232618AbiLERvl (ORCPT ); Mon, 5 Dec 2022 12:51:41 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 482186153 for ; Mon, 5 Dec 2022 09:50:48 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id w15so19802360wrl.9 for ; Mon, 05 Dec 2022 09:50:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=0CJhC9zf/tPjJhvDDmpb9psUAlq9xPX1O0pWhY3soLg=; b=WrxqBQa2FRgZhQufNZj07BQlw4FbnD8Ojwzl0fyLabaAebAx7U3xAdw5P+0ne35Ql7 7BEQ7Iq2kWkQLnEoVR8sroxy5NcMGQf1PdsqT9AkMduAFITvYl6B4kYDobAxuA5U0h5O sdMVPnxHOvRhupYv3X9kg/dAHpEtyXTpHcdpEWd/f69BzQzbRArFxB5uh6MrXAoWKe/Z 0iyQcPIYI1QTEiPALl+cGsx0fB5Selb2aouQaZf8MiKC6i5dkijBX7o45CTMWartVFpY vH5aEyG8jYJYmadc/URN/VrIgPyduXCmSvyYbDR7phwFYIg28hSUbzqSsp5RdZXbYGTy ZIkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0CJhC9zf/tPjJhvDDmpb9psUAlq9xPX1O0pWhY3soLg=; b=Ms7WEOCArIXaUhyi3mHSV405dVmjDrRyR5uyGp39Vjg7+tcIuJNIvbyht/Dzb1qWVA PovIgBAccWcVu4BW+OTQt+4Qv9BsANABMzJwbul5VdFOHnMr5i/jKC0oCP2KQV9yaGfk sd3zQIHJWuOVWDOES5RBTUvopVZ+UvHlshInjB2CBWpsxsWTh3DG1LLln1qx5aDSdjyo /GR705Oe7IpPR+5dTJmGHhimYLJvZEoRq03L68fH2FQffsKPOu/t9yXPALx+i5iFc41C R7+Khr2Qyfc2pyipphQu6uIjasdo8ZTtQ7A7mDealh52dgadhoBX8klDuwQpTketxN5+ TCtA== X-Gm-Message-State: ANoB5plH9H5RkD8bR84rn99Tw9hgTlnozfi5mRsYeGG/uT8mFPhIw9ti v2oIyYX+ZDJaHHb5vD4QRA3pblifpVA= X-Google-Smtp-Source: AA0mqf4jEYSjw5w3yUfgTMeKEnVF8D3wyNikLaTTkJcIUWHY/vo8ELD/XfFslaukAqjqLmovozZ1Rw== X-Received: by 2002:a5d:444e:0:b0:242:32e9:3a74 with SMTP id x14-20020a5d444e000000b0024232e93a74mr12954341wrr.14.1670262647747; Mon, 05 Dec 2022 09:50:47 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ba16-20020a0560001c1000b002365254ea42sm16027157wrb.1.2022.12.05.09.50.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:47 -0800 (PST) Message-Id: <1eec3426aee12bbd674ebd646075f0d4c0b1f5af.1670262639.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:36 +0000 Subject: [PATCH v3 08/11] strbuf: introduce strbuf_strip_file_from_path() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The strbuf_parent_directory() method was added as a static method in contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended config and starts maintenance, 2021-12-03) and then removed in 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now there is a need for a similar method in the bundle URI feature. Re-add the method, this time in strbuf.c, but with a new name: strbuf_strip_file_from_path(). The method requirements are slightly modified to allow a trailing slash, in which case nothing is done, which makes the name change valuable. Signed-off-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason --- strbuf.c | 6 ++++++ strbuf.h | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/strbuf.c b/strbuf.c index 0890b1405c5..c383f41a3c5 100644 --- a/strbuf.c +++ b/strbuf.c @@ -1200,3 +1200,9 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path, free(path2); return res; } + +void strbuf_strip_file_from_path(struct strbuf *sb) +{ + char *path_sep = find_last_dir_sep(sb->buf); + strbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0); +} diff --git a/strbuf.h b/strbuf.h index 76965a17d44..f6dbb9681ee 100644 --- a/strbuf.h +++ b/strbuf.h @@ -664,6 +664,17 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer, int strbuf_edit_interactively(struct strbuf *buffer, const char *path, const char *const *env); +/* + * Remove the filename from the provided path string. If the path + * contains a trailing separator, then the path is considered a directory + * and nothing is modified. + * + * Examples: + * - "/path/to/file" -> "/path/to/" + * - "/path/to/dir/" -> "/path/to/dir/" + */ +void strbuf_strip_file_from_path(struct strbuf *sb); + void strbuf_add_lines(struct strbuf *sb, const char *prefix, const char *buf, From patchwork Mon Dec 5 17:50:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13064940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7545C4332F for ; Mon, 5 Dec 2022 17:52:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232649AbiLERwZ (ORCPT ); Mon, 5 Dec 2022 12:52:25 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44562 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232772AbiLERvl (ORCPT ); Mon, 5 Dec 2022 12:51:41 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14C8D62C8 for ; Mon, 5 Dec 2022 09:50:50 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id bg10so9364629wmb.1 for ; Mon, 05 Dec 2022 09:50:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=CywwVsO/oG6poD1gJfQFbKhOmz6urWmUyEVt2g0nh7U=; b=iZnt3QPRU9Iao71/r6qTNU8y/iofDvBPV2AUOvKHzC3uTdkDO5oOJyGgA3kaXVpGVL fPzlWsJ+OKQwdppYlWDzZjqoy/yVJV/ptr2SEZvmC9/z67tb2k8bfONQBZIwbtL1/+is 7ZL/2Fi+k4P4UkBWxLHb3cp8YRbFc4RxNYfXXyFNP6lnsqdSG6ct3QqXVr1xIj+lqm4g gfjzCvaw2a4x7MHYWtq/Cyl03syzWd2//yiG2kqAbBJNCdYcFwlCy1s6eWuhJ+B38YYR MKBCSIvGgb1PE5ykFfx5Q1Di34rW8bWHp6osjOWv5utqgl7FtWAK1aP1XIgs1l3zIAIL 3r1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CywwVsO/oG6poD1gJfQFbKhOmz6urWmUyEVt2g0nh7U=; b=28NNIWUpc55h+GOxd5TjcbSXRY42Q3F8kpcqjV+k3qnZcFCd8SG+mVKzt+pTPFbAWO YIDv8/xEvx0AeUZzlf/hueAmKRRh4lcYh4MmHJkqBEr9H6xDq/xRPn7RSojifj2MBFXg GbAVa8ct9qNvnhHKC34A2W2vRlv6xcjDavhwZlu0bb+Tzs0DMN3B1LCXmSl5RvuUCfl1 8VgcxUrRDM9PnHmxEDZfoek5YrIoQC4cgDhP5UL3dcWdWdiwhoTdWOwGYFt9wKfjK8H6 3quQ5E1ZesJjIq3qyThnt+OMuo1aAbvrnn0EuXhBkJlCSv3KRWWKYXadRf9ld32GQEJX 6K0w== X-Gm-Message-State: ANoB5plCkQL+g3QLClwd9SBHnnlbc2Yq0UKBsSFcMoDq806jp55h1T6d 4dwa1sgO+G3qWsZnJP9OhjVXsi4CPSw= X-Google-Smtp-Source: AA0mqf7/k3aPfXxQbriWqakBe62mIb04hfCUsnmTv/QM+VZfpgAPht7NS0pz3cL/yIc7hKuUNmg+5g== X-Received: by 2002:a7b:c8c3:0:b0:3cf:b49e:1638 with SMTP id f3-20020a7bc8c3000000b003cfb49e1638mr50101922wml.50.1670262648476; Mon, 05 Dec 2022 09:50:48 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r15-20020adfdc8f000000b002425787c5easm6270402wrj.96.2022.12.05.09.50.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:48 -0800 (PST) Message-Id: <48731438d6a289129a5768b80af856fb49692426.1670262639.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:37 +0000 Subject: [PATCH v3 09/11] bundle-uri: allow relative URLs in bundle lists Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Bundle providers may want to distribute that data across multiple CDNs. This might require a change in the base URI, all the way to the domain name. If all bundles require an absolute URI in their 'uri' value, then every push to a CDN would require altering the table of contents to match the expected domain and exact location within it. Allow a bundle list to specify a relative URI for the bundles. This URI is based on where the client received the bundle list. For a list provided in the 'bundle-uri' protocol v2 command, the Git remote URI is the base URI. Otherwise, the bundle list was provided from an HTTP URI not using the Git protocol, and that URI is the base URI. This allows easier distribution of bundle data. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 +++++++- bundle-uri.h | 14 +++++++ t/helper/test-bundle-uri.c | 2 + t/t5750-bundle-uri-parse.sh | 82 +++++++++++++++++++++++++++++++++++++ transport.c | 3 ++ 5 files changed, 116 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 6919f541085..80370992773 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -7,6 +7,7 @@ #include "hashmap.h" #include "pkt-line.h" #include "config.h" +#include "remote.h" static int compare_bundles(const void *hashmap_cmp_fn_data, const struct hashmap_entry *he1, @@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list) for_all_bundles_in_list(list, clear_remote_bundle_info, NULL); hashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent); + free(list->baseURI); } int for_all_bundles_in_list(struct bundle_list *list, @@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value, if (!strcmp(subkey, "uri")) { if (bundle->uri) return -1; - bundle->uri = xstrdup(value); + bundle->uri = relative_url(list->baseURI, value, NULL); return 0; } @@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri, .error_action = CONFIG_ERROR_ERROR, }; + if (!list->baseURI) { + struct strbuf baseURI = STRBUF_INIT; + strbuf_addstr(&baseURI, uri); + + /* + * If the URI does not end with a trailing slash, then + * remove the filename portion of the path. This is + * important for relative URIs. + */ + strbuf_strip_file_from_path(&baseURI); + list->baseURI = strbuf_detach(&baseURI, NULL); + } result = git_config_from_file_with_options(config_to_bundle_list, filename, list, &opts); diff --git a/bundle-uri.h b/bundle-uri.h index 357111ecce8..e7e90a5f088 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -61,6 +61,20 @@ struct bundle_list { int version; enum bundle_list_mode mode; struct hashmap bundles; + + /** + * The baseURI of a bundle_list is the URI that provided the list. + * + * In the case of the 'bundle-uri' protocol v2 command, the base + * URI is the URI of the Git remote. + * + * Otherewise, the bundle list was downloaded over HTTP from some + * known URI. + * + * The baseURI is used as the base for any relative URIs + * advertised by the bundle list at that location. + */ + char *baseURI; }; void init_bundle_list(struct bundle_list *list); diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index f8159187014..5df5bc3b89e 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo init_bundle_list(&list); + list.baseURI = xstrdup(""); + switch (mode) { case KEY_VALUE_PAIRS: if (argc != 1) diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh index c2fe3f9c5a5..7b4f930e532 100755 --- a/t/t5750-bundle-uri-parse.sh +++ b/t/t5750-bundle-uri-parse.sh @@ -30,6 +30,58 @@ test_expect_success 'bundle_uri_parse_line() just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'bundle_uri_parse_line(): relative URIs' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-key-values in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line(): relative URIs and parent paths' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=../../bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /../bundle.bdl + EOF + + # TODO: We would prefer if parsing a bundle list would not cause + # a die() and instead would give a warning and allow the rest of + # a Git command to continue. This test_must_fail is necessary for + # now until the interface for relative_url() allows for reporting + # an error instead of die()ing. + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + grep "fatal: cannot strip one component off url" err +' + test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' ' cat >in <<-\EOF && =bogus-value @@ -136,6 +188,36 @@ test_expect_success 'parse config format: just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'parse config format: relative URIs' ' + cat >in <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = bundle.bdl + [bundle "two"] + uri = ../bundle.bdl + [bundle "three"] + uri = sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-config in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + test_expect_success 'parse config format edge cases: empty key or value' ' cat >in1 <<-\EOF && = bogus-value diff --git a/transport.c b/transport.c index 97d395e10a3..957dca4923c 100644 --- a/transport.c +++ b/transport.c @@ -1539,6 +1539,9 @@ int transport_get_remote_bundle_uri(struct transport *transport) (git_config_get_bool("transfer.bundleuri", &value) || !value)) return 0; + if (!transport->bundles->baseURI) + transport->bundles->baseURI = xstrdup(transport->url); + if (!vtable->get_bundle_uri) return error(_("bundle-uri operation not supported by protocol")); From patchwork Mon Dec 5 17:50:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13064939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DCDCC4708D for ; Mon, 5 Dec 2022 17:52:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230519AbiLERwV (ORCPT ); Mon, 5 Dec 2022 12:52:21 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44196 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231923AbiLERvl (ORCPT ); Mon, 5 Dec 2022 12:51:41 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A7F5263E1 for ; Mon, 5 Dec 2022 09:50:51 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id m14so19797665wrh.7 for ; Mon, 05 Dec 2022 09:50:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=xHJA4BH3cwSREt24TlkbDVnBn5j0+GPdetbayNtBX+0=; b=Eb/2JHKlN5nTPLwoUaZQKvnneXpyuppfw9NlDnDgoWcrjbyd4v4qlIS17ItYux18co vZr9A81ZcpHPLdDG9HnNMszLkfP4mwuYCHr27brwg3KQdLo+YwFZpSiaCUGWS5g/rhBf UXhJk8yzMcS15oSb5SSHqt+Ghj9rcBUp4/sOWrp9zkf+QUd1nKJvePdOgKj/gOsNrrf6 gxipNkXlmXyguMRa28PZ0ffHx1e7ghePR7vVTl200jQCz0Fnih6+gvFivqgur4FVuxKc /4o8axZwyiQTaQDLPP99mMHoCSDoLG+Vj0nCSGNdLQAM3yP6VsZ7zz/yKCU1KvxcfMW4 soHQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xHJA4BH3cwSREt24TlkbDVnBn5j0+GPdetbayNtBX+0=; b=iMEqxSK0vQQYp98QDJAARu8upuVw+O/9mEHeasgXavIrZNzt83y7b5P8n4r3mGYPLV YujW/Fyl0+kvca8UEgg0qt2CFC7L2W87heH1q+Kr6A1lxu6YSl5uNY0tuWWoDlVa9OU5 TeOKyHDVaTMeUKXWbMKZHtQUm7tz65KwEXVvZfnTJfR6Nk6Nl5IY3dkchP0ppAMRf+Ld E7mcR+bIHLCnEeJOwy+pS0n8GyhdqF6kGJEX0M9QmFlMBuN8EQOQ/XCjHPvHw8ccvLva TPTvnEf4gQf0C2+XFA7u5Yb+Z3Sk7IdWWjBmWH0pgiaQVRQx94zbiVEXATVxoK9oARJS djFA== X-Gm-Message-State: ANoB5pl4DAQbCFQthZoUsrJgNeVFEfLZlZq3j7a+solF5CqEZYkDil1O 2pdMnsbkr4vC34a9ha1P7j0SeoFhUWk= X-Google-Smtp-Source: AA0mqf6edIpKZEgPsMFFlniaObYwEI6Ll0/W6fwEd62tYQSSxDMFmci6YILenv8+YugNtg4ZAOAn+g== X-Received: by 2002:adf:eb92:0:b0:236:80a8:485e with SMTP id t18-20020adfeb92000000b0023680a8485emr43539867wrn.362.1670262649187; Mon, 05 Dec 2022 09:50:49 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n3-20020a05600c3b8300b003cfbbd54178sm29564680wms.2.2022.12.05.09.50.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:48 -0800 (PST) Message-Id: <69bf154bec63a22df8e5eac89f975625ce73c8ac.1670262639.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:38 +0000 Subject: [PATCH v3 10/11] bundle-uri: download bundles from an advertised list Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The logic in fetch_bundle_uri() is useful for the --bundle-uri option of 'git clone', but is not helpful when the clone operation discovers a list of URIs from the bundle-uri protocol v2 command. To actually download and unbundle the advertised bundles, we need a different mechanism. Create the new fetch_bundle_list() method which is very similar to fetch_bundle_uri() except that it relies on download_bundle_list() instead of fetch_bundle_uri_internal(). The download_bundle_list() method will recursively call fetch_bundle_uri_internal() if any of the advertised URIs serve a bundle list instead of a bundle. This will also follow the bundle.list.mode setting from the input list: "any" will download only one such URI while "all" will download data from all of the URIs. In an identical way to fetch_bundle_uri(), the bundles are unbundled after all of the bundle lists have been expanded and all necessary URIs. Signed-off-by: Derrick Stolee --- bundle-uri.c | 21 +++++++++++++++++++++ bundle-uri.h | 11 +++++++++++ 2 files changed, 32 insertions(+) diff --git a/bundle-uri.c b/bundle-uri.c index 80370992773..c411b871bdd 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -577,6 +577,27 @@ cleanup: return result; } +int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list) +{ + int result; + struct bundle_list global_list; + + init_bundle_list(&global_list); + + /* If a bundle is added to this global list, then it is required. */ + global_list.mode = BUNDLE_MODE_ALL; + + if ((result = download_bundle_list(r, list, &global_list, 0))) + goto cleanup; + + result = unbundle_all_bundles(r, &global_list); + +cleanup: + for_all_bundles_in_list(&global_list, unlink_bundle, NULL); + clear_bundle_list(&global_list); + return result; +} + /** * API for serve.c. */ diff --git a/bundle-uri.h b/bundle-uri.h index e7e90a5f088..b2c9c160a52 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -107,6 +107,17 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * Given a bundle list that was already advertised (likely by the + * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the + * bundles according to the bundle strategy of that list. + * + * Returns non-zero if no bundle information is found at the given 'uri'. + */ +int fetch_bundle_list(struct repository *r, + const char *uri, + struct bundle_list *list); + /** * API for serve.c. */ From patchwork Mon Dec 5 17:50:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13064945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 819EBC47089 for ; Mon, 5 Dec 2022 17:52:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232839AbiLERwp (ORCPT ); Mon, 5 Dec 2022 12:52:45 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44316 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232409AbiLERvl (ORCPT ); Mon, 5 Dec 2022 12:51:41 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8033E657B for ; Mon, 5 Dec 2022 09:50:52 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id n7so9345020wms.3 for ; Mon, 05 Dec 2022 09:50:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=mGfncvEIZ1IJH1viufNLa/kn8K2n+QVCVUQHa5ASzVU=; b=e0bX4kSgvzNpeZHLAyHP0dtVCkbpNaGexhfzAUMBloO2kcYC1JSyGBfMbP1ZIwZxm4 ZiXmYR3+//89Nvkzzr3Y20pMcq7csta7EvmlTLIaPHXk8KOR5DYRS6WHo18+8ehGjXgS ES9FC2o2uEdNQzvyX4fMl895O/+N0t3D4IZFltQ09zBznpK3dZFtqPeQ7/ft7iuoL/0Y 0GuN8BpJL0fj+NziA7L72xE9symqZvD48m97qtkQosBuUeWEks9LMpTKyhjt8vEOP1OO fRi+w2z2B75RyW2bgYaeTPe6/3tOMQeRMMu+i7LryPRTWehLEMhXpoZRRELXknSz1LzC ocXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=mGfncvEIZ1IJH1viufNLa/kn8K2n+QVCVUQHa5ASzVU=; b=Gi+hyOtJs+Z1M9B6FNw1sFNjmfNgrY1bPE9u3+TcEnZPgqSJx4/2lt6788piinOECm hKCqxO9teRT4Ri8DX46ynP1YFjuvjWRDeAccgWgCYHd6HzG5Yk57NJWTJNlVDwLCtJmj kOsXzUoRM1bTO5u15m4RnaE6a2+59ujSAHCRANNviH8Li0sfW3No5D70fjN3rQDl1ETI WfZ4x1eM0f7e3FKqHrPEB5Y0NUmkElro8uDLXRtcVr5UBKFAXCT6KOIF7X9FZAOdo6rI VDEsbg45Q0xQjlTrypjwSZyJr7JSzbGHmMOmB1yj8JuV0xr8yYTlZhbcKTFBZ/TDLsbZ Fimg== X-Gm-Message-State: ANoB5pmk8kOdWoJ94GOQmAewVFeFxPtgpbdAhHheD7oD0Li1UEaq5Fg3 TcTITsDj8HpUCYvYHUnxVntSYBtZAM4= X-Google-Smtp-Source: AA0mqf6ewFrPMqhHcUo1zTEGJ4iSLwnsvJzQXt2u5klt2N5/j+/wf9DDyTaPnG0HTU60CqCtQNoFOA== X-Received: by 2002:a05:600c:1c1b:b0:3d1:f04:d1db with SMTP id j27-20020a05600c1c1b00b003d10f04d1dbmr6073073wms.44.1670262650857; Mon, 05 Dec 2022 09:50:50 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r8-20020a05600c424800b003c5571c27a1sm21468011wmm.32.2022.12.05.09.50.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 05 Dec 2022 09:50:50 -0800 (PST) Message-Id: <7e1819162b6129371168d89dd60a1bd3cb7de299.1670262639.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 05 Dec 2022 17:50:39 +0000 Subject: [PATCH v3 11/11] clone: unbundle the advertised bundles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A previous change introduced the transport methods to acquire a bundle list from the 'bundle-uri' protocol v2 command, when advertised _and_ when the client has chosen to enable the feature. Teach Git to download and unbundle the data advertised by those bundles during 'git clone'. Also, since the --bundle-uri option exists, we do not want to mix the advertised bundles with the user-specified bundles. Signed-off-by: Derrick Stolee --- builtin/clone.c | 26 +++++++++++++--- t/lib-bundle-uri-protocol.sh | 21 +++++++++++-- t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++ 3 files changed, 99 insertions(+), 7 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 39364c25b15..af8b2a4df66 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1266,11 +1266,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); - /* - * Populate transport->got_remote_bundle_uri and - * transport->bundle_uri. We might get nothing. - */ - transport_get_remote_bundle_uri(transport); + if (!bundle_uri) { + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + + if (transport->bundles && + hashmap_get_size(&transport->bundles->bundles)) { + /* At this point, we need the_repository to match the cloned repo. */ + if (repo_init(the_repository, git_dir, work_tree)) + warning(_("failed to initialize the repo, skipping bundle URI")); + else if (fetch_bundle_list(the_repository, + remote->url[0], + transport->bundles)) + warning(_("failed to fetch advertised bundles")); + } else { + clear_bundle_list(transport->bundles); + FREE_AND_NULL(transport->bundles); + } + } if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 6d3f871fa0f..73e2d45bc8b 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -85,7 +85,7 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav ' test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' - test_when_finished "rm -rf log cloned cloned2" && + test_when_finished "rm -rf log* cloned*" && GIT_TRACE_PACKET="$PWD/log" \ GIT_TEST_BUNDLE_URI=0 \ @@ -117,7 +117,24 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque grep "< bundle-uri" log && # Client issued bundle-uri command - grep "> command=bundle-uri" log + grep "> command=bundle-uri" log && + + GIT_TRACE_PACKET="$PWD/log3" \ + git \ + -c transfer.bundleURI=true \ + -c protocol.version=2 \ + clone --bundle-uri="$BUNDLE_URI_BUNDLE_URI" \ + "$BUNDLE_URI_REPO_URI" cloned3 \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log3 && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log3 && + + # Client did not issue bundle-uri command (--bundle-uri override) + ! grep "> command=bundle-uri" log3 ' test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2" ' diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 45f0803ed4d..d1d8139751e 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' ' git clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo ' +test_expect_success 'auto-discover bundle URI from HTTP clone' ' + test_when_finished rm -rf trace.txt repo2 "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle" --all && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.mode all && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + + GIT_TEST_BUNDLE_URI=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 clone \ + $HTTPD_URL/smart/repo2.git repo2 && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt +' + +test_expect_success 'auto-discover multiple bundles from HTTP clone' ' + test_when_finished rm -rf trace.txt repo3 "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + test_commit -C src new && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/new.bundle" HEAD~1..HEAD && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.mode all && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.new.uri "$HTTPD_URL/new.bundle" && + + GIT_TEST_BUNDLE_URI=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 clone \ + $HTTPD_URL/smart/repo3.git repo3 && + + # We should fetch _both_ bundles + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/new.bundle"\] + EOF + grep -f pattern trace.txt +' + # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled.