From patchwork Thu Dec 22 15:14:07 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13080128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C101C4332F for ; Thu, 22 Dec 2022 15:14:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235627AbiLVPOk (ORCPT ); Thu, 22 Dec 2022 10:14:40 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235473AbiLVPOZ (ORCPT ); Thu, 22 Dec 2022 10:14:25 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BB5CC2A25D for ; Thu, 22 Dec 2022 07:14:22 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id h7so1957514wrs.6 for ; Thu, 22 Dec 2022 07:14:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=UvaqDapRgefZLwc1Xafomhv3j+nEWPerru2X0eZ8srI=; b=og96MGAkA8Ogw6ifSe7cL5VIKWPfZBMGRRKVEQFCeOWnYIJfbjL4E5yAXbLBhJOI0v CC0x7H8lRiHMRxH+1+Yfp46OaI7ZNSdyBssH7xT2nRVyL0kAqqd8f+y/z+sLpygTRYne SyHS1tv24pCoqgNfZteYulqmxxmjGRagxr20r0znWlbM2YjOzFSc5F0BLlvNhwjvnvuh BEjkfCjsO1iFw5agWSIe/NfsDPXndlNKXCVKYGxYLTK9KP2WQQl0sV3adXIdLBDIyBha t9Jh3jLsxLIDhHah6YykJdF0Y7ymrFXD7FMkubqC5eJD+MItppDK/xZ65V+9vhakt5GD QZuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=UvaqDapRgefZLwc1Xafomhv3j+nEWPerru2X0eZ8srI=; b=rtlE4w+JmfFrf049Yv7Tbu84XfgO82LcW9Yb7si9gcjbv6QUJsQS2RAf5jbnnWQsmE IS3sZovan5sTcqCBHpYogTjc3VkXNVWkXiRD7e02Wfj+e0I4bGBLhIIjXzLCKqbgh6jo i+OhAxgLCOLzzWykwGLfL1VVqJOpX7bkN1hS3EW0g8qPUBvg1Fc0sUwY2kTBQ58zsTz/ YQFo6VwS12SBW7SmI6hxkTS8ggskk/5MS2MMa2verNtgzl+u7rhI88edtbsjicPHN2xf J5qj1Hv9yOJB5Fa2PohlfBg/u+SrKsamcY8vHtBoYnUJNfKlBnNA4qwihL3T6XNjaNA3 RJ8A== X-Gm-Message-State: AFqh2kpOeu22Qc1u6mrBbCg86W8aPPm+TYUucC6ZSaAItDdJBjWoHVRz hy+E8MlFMpIqD8zLptmL+na3VsZNS7Y= X-Google-Smtp-Source: AMrXdXsQn4GHXa/8uHwzEgGtrUlWO/aD1TbiktzevUirD9hiYLcmUPEqVf2anj1S1y8gnFwY3YrlzA== X-Received: by 2002:adf:f2c9:0:b0:242:1522:24a7 with SMTP id d9-20020adff2c9000000b00242152224a7mr3669659wrp.32.1671722060853; Thu, 22 Dec 2022 07:14:20 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t13-20020adfe44d000000b002365b759b65sm723793wrm.86.2022.12.22.07.14.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:20 -0800 (PST) Message-Id: <5ba91813de38d2b6cb2018517460623d94003926.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:07 +0000 Subject: [PATCH v4 01/11] protocol v2: add server-side "bundle-uri" skeleton MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 command with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++++++++++ bundle-uri.c | 36 ++++++ bundle-uri.h | 7 ++ serve.c | 6 + t/t5701-git-serve.sh | 40 +++++- 5 files changed, 289 insertions(+), 1 deletion(-) diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt index 59bf41cefb9..10bd2d40cec 100644 --- a/Documentation/gitprotocol-v2.txt +++ b/Documentation/gitprotocol-v2.txt @@ -578,6 +578,207 @@ and associated requested information, each separated by a single space. obj-info = obj-id SP obj-size +bundle-uri +~~~~~~~~~~ + +If the 'bundle-uri' capability is advertised, the server supports the +`bundle-uri' command. + +The capability is currently advertised with no value (i.e. not +"bundle-uri=somevalue"), a value may be added in the future for +supporting command-wide extensions. Clients MUST ignore any unknown +capability values and proceed with the 'bundle-uri` dialog they +support. + +The 'bundle-uri' command is intended to be issued before `fetch` to +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and +inform the subsequent `fetch` command. + +The client CAN issue `bundle-uri` before or after any other valid +command. To be useful to clients it's expected that it'll be issued +after an `ls-refs` and before `fetch`, but CAN be issued at any time +in the dialog. + +DISCUSSION of bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^^ + +The intent of the feature is optimize for server resource consumption +in the common case by changing the common case of fetching a very +large PACK during linkgit:git-clone[1] into a smaller incremental +fetch. + +It also allows servers to achieve better caching in combination with +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]). + +By having new clones or fetches be a more predictable and common +negotiation against the tips of recently produces *.bundle file(s). +Servers might even pre-generate the results of such negotiations for +the `uploadpack.packObjectsHook` as new pushes come in. + +One way that servers could take advantage of these bundles is that the +server would anticipate that fresh clones will download a known bundle, +followed by catching up to the current state of the repository using ref +tips found in that bundle (or bundles). + +PROTOCOL for bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^ + +A `bundle-uri` request takes no arguments, and as noted above does not +currently advertise a capability value. Both may be added in the +future. + +When the client issues a `command=bundle-uri` request, the response is a +list of key-value pairs provided as packet lines with value +`=`. Each `` should be interpreted as a config key from +the `bundle.*` namespace to construct a list of bundles. These keys are +grouped by a `bundle..` subsection, where each key corresponding to a +given `` contributes attributes to the bundle defined by that ``. +See linkgit:git-config[1] for the specific details of these keys and how +the Git client will interpret their values. + +Clients MUST parse the line according to the above format, lines that do +not conform to the format SHOULD be discarded. The user MAY be warned in +such a case. + +bundle-uri CLIENT AND SERVER EXPECTATIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +URI CONTENTS:: +The content at the advertised URIs MUST be one of two types. ++ +The advertised URI may contain a bundle file that `git bundle verify` +would accept. I.e. they MUST contain one or more reference tips for +use by the client, MUST indicate prerequisites (in any) with standard +"-" prefixes, and MUST indicate their "object-format", if +applicable. ++ +The advertised URI may alternatively contain a plaintext file that `git +config --list` would accept (with the `--file` option). The key-value +pairs in this list are in the `bundle.*` namespace (see +linkgit:git-config[1]). + +bundle-uri CLIENT ERROR RECOVERY:: +A client MUST above all gracefully degrade on errors, whether that +error is because of bad missing/data in the bundle URI(s), because +that client is too dumb to e.g. understand and fully parse out bundle +headers and their prerequisite relationships, or something else. ++ +Server operators should feel confident in turning on "bundle-uri" and +not worry if e.g. their CDN goes down that clones or fetches will run +into hard failures. Even if the server bundle bundle(s) are +incomplete, or bad in some way the client should still end up with a +functioning repository, just as if it had chosen not to use this +protocol extension. ++ +All subsequent discussion on client and server interaction MUST keep +this in mind. + +bundle-uri SERVER TO CLIENT:: +The ordering of the returned bundle uris is not significant. Clients +MUST parse their headers to discover their contained OIDS and +prerequisites. A client MUST consider the content of the bundle(s) +themselves and their header as the ultimate source of truth. ++ +A server MAY even return bundle(s) that don't have any direct +relationship to the repository being cloned (either through accident, +or intentional "clever" configuration), and expect a client to sort +out what data they'd like from the bundle(s), if any. + +bundle-uri CLIENT TO SERVER:: +The client SHOULD provide reference tips found in the bundle header(s) +as 'have' lines in any subsequent `fetch` request. A client MAY also +ignore the bundle(s) entirely if doing so is deemed worse for some +reason, e.g. if the bundles can't be downloaded, it doesn't like the +tips it finds etc. + +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION:: +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s) +of the bundle(s) the client finds that the ref tips it wants can be +retrieved entirely from advertised bundle(s), the client MAY disconnect +from the Git server. The results of such a 'clone' or 'fetch' should be +indistinguishable from the state attained without using bundle-uri. + +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY:: +A client MAY perform an early disconnect while still downloading the +bundle(s) (having streamed and parsed their headers). In such a case +the client MUST gracefully recover from any errors related to +finishing the download and validation of the bundle(s). ++ +I.e. a client might need to re-connect and issue a 'fetch' command, +and possibly fall back to not making use of 'bundle-uri' at all. ++ +This "MAY" behavior is specified as such (and not a "SHOULD") on the +assumption that a server advertising bundle uris is more likely than +not to be serving up a relatively large repository, and to be pointing +to URIs that have a good chance of being in working order. A client +MAY e.g. look at the payload size of the bundles as a heuristic to see +if an early disconnect is worth it, should falling back on a full +"fetch" dialog be necessary. + +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION:: +A client SHOULD commence a negotiation of a PACK from the server via +the "fetch" command using the OID tips found in advertised bundles, +even if's still in the process of downloading those bundle(s). ++ +This allows for aggressive early disconnects from any interactive +server dialog. The client blindly trusts that the advertised OID tips +are relevant, and issues them as 'have' lines, it then requests any +tips it would like (usually from the "ls-refs" advertisement) via +'want' lines. The server will then compute a (hopefully small) PACK +with the expected difference between the tips from the bundle(s) and +the data requested. ++ +The only connection the client then needs to keep active is to the +concurrently downloading static bundle(s), when those and the +incremental PACK are retrieved they should be inflated and +validated. Any errors at this point should be gracefully recovered +from, see above. + +bundle-uri PROTOCOL FEATURES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The client constructs a bundle list from the `=` pairs +provided by the server. These pairs are part of the `bundle.*` namespace +as documented in linkgit:git-config[1]. In this section, we discuss some +of these keys and describe the actions the client will do in response to +this information. + +In particular, the `bundle.version` key specifies an integer value. The +only accepted value at the moment is `1`, but if the client sees an +unexpected value here then the client MUST ignore the bundle list. + +As long as `bundle.version` is understood, all other unknown keys MAY be +ignored by the client. The server will guarantee compatibility with older +clients, though newer clients may be better able to use the extra keys to +minimize downloads. + +Any backwards-incompatible addition of pre-URI key-value will be +guarded by a new `bundle.version` value or values in 'bundle-uri' +capability advertisement itself, and/or by new future `bundle-uri` +request arguments. + +Some example key-value pairs that are not currently implemented but could +be implemented in the future include: + + * Add a "hash=" or "size=" advertise the expected hash or + size of the bundle file. + + * Advertise that one or more bundle files are the same (to e.g. have + clients round-robin or otherwise choose one of N possible files). + + * A "oid=" shortcut and "prerequisite=" shortcut. For + expressing the common case of a bundle with one tip and no + prerequisites, or one tip and one prerequisite. ++ +This would allow for optimizing the common case of servers who'd like +to provide one "big bundle" containing only their "main" branch, +and/or incremental updates thereof. ++ +A client receiving such a a response MAY assume that they can skip +retrieving the header from a bundle at the indicated URI, and thus +save themselves and the server(s) the request(s) needed to inspect the +headers of that bundle or bundles. + GIT --- Part of the linkgit:git[1] suite diff --git a/bundle-uri.c b/bundle-uri.c index 79a914f961b..28d8966005e 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -563,6 +563,42 @@ cleanup: return result; } +/** + * API for serve.c. + */ + +int bundle_uri_advertise(struct repository *r, struct strbuf *value UNUSED) +{ + static int advertise_bundle_uri = -1; + + if (advertise_bundle_uri != -1) + goto cached; + + advertise_bundle_uri = 0; + repo_config_get_maybe_bool(r, "uploadpack.advertisebundleuris", &advertise_bundle_uri); + +cached: + return advertise_bundle_uri; +} + +int bundle_uri_command(struct repository *r, + struct packet_reader *request) +{ + struct packet_writer writer; + packet_writer_init(&writer, 1); + + while (packet_reader_read(request) == PACKET_READ_NORMAL) + die(_("bundle-uri: unexpected argument: '%s'"), request->line); + if (request->status != PACKET_READ_FLUSH) + die(_("bundle-uri: expected flush after arguments")); + + /* TODO: Implement the communication */ + + packet_writer_flush(&writer); + + return 0; +} + /** * General API for {transport,connect}.c etc. */ diff --git a/bundle-uri.h b/bundle-uri.h index 4dbc269823c..357111ecce8 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -4,6 +4,7 @@ #include "hashmap.h" #include "strbuf.h" +struct packet_reader; struct repository; struct string_list; @@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * API for serve.c. + */ +int bundle_uri_advertise(struct repository *r, struct strbuf *value); +int bundle_uri_command(struct repository *r, struct packet_reader *request); + /** * General API for {transport,connect}.c etc. */ diff --git a/serve.c b/serve.c index 733347f602a..cbf4a143cfe 100644 --- a/serve.c +++ b/serve.c @@ -7,6 +7,7 @@ #include "protocol-caps.h" #include "serve.h" #include "upload-pack.h" +#include "bundle-uri.h" static int advertise_sid = -1; static int client_hash_algo = GIT_HASH_SHA1; @@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = { .advertise = always_advertise, .command = cap_object_info, }, + { + .name = "bundle-uri", + .advertise = bundle_uri_advertise, + .command = bundle_uri_command, + }, }; void protocol_v2_advertise_capabilities(void) diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index 1896f671cb3..f21e5e9d33d 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' ' wrong_algo sha1:sha256 wrong_algo sha256:sha1 EOF - cat >expect <<-EOF && + cat >expect.base <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) ls-refs=unborn @@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' ' server-option object-format=$(test_oid algo) object-info + EOF + cat >expect.trailer <<-EOF && 0000 EOF + cat expect.base expect.trailer >expect && GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ --advertise-capabilities >out && @@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' ' + test_config uploadpack.advertiseBundleURIs true && + + cat >expect.extra <<-EOF && + bundle-uri + EOF + cat expect.base \ + expect.extra \ + expect.trailer >expect && + + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ + --advertise-capabilities >out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: dies if not enabled' ' + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: invalid command '"'"'bundle-uri'"'"' + EOF + + cat >expect <<-\EOF && + ERR serve: invalid command '"'"'bundle-uri'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + test_done From patchwork Thu Dec 22 15:14:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13080129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 098CFC10F1D for ; Thu, 22 Dec 2022 15:14:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235400AbiLVPOo (ORCPT ); Thu, 22 Dec 2022 10:14:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235515AbiLVPOZ (ORCPT ); Thu, 22 Dec 2022 10:14:25 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 654A01007D for ; Thu, 22 Dec 2022 07:14:23 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id r126-20020a1c4484000000b003d6b8e8e07fso3914438wma.0 for ; Thu, 22 Dec 2022 07:14:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=HDKpSmBn+STnbwuijw/6VlfTrIRU0b/XAk8TQ+voHE0=; b=eWbcsnaqR6Jrjvt81VTbQgq4A0YTDgIPZLYZjbkFn3RXTJkfi9sMITCllV/ANHpmsa yXnZKKMMLgo64wXWUfM8KXI0SUecrCKJ45m+4MZL2WW033SnIu0dXRwwWBOhV9nFU4gq 4Oi8p9+fZpVy+ScsfUjqdxiIBkINqq9/Zw+VhnZwoBi2rlwJPxrrBH2fUIV/Zn0kQyED LEPIUckL+fc8dNwya8qyRukxnnT+LUx4XMT19kC2/X91V07fIiphXFioA8rbPaKOujmV UUCQkNd0w8PaIecTafIgzHX0OYW7K2K+UZp9aNEiAmWsB5qB998r/8JmxIXtLhzRYniP M2ZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HDKpSmBn+STnbwuijw/6VlfTrIRU0b/XAk8TQ+voHE0=; b=4Onfod2mbu9VhWdpPYkYTVZhzYVpd2CMGymXnax11l0ACEkYMbw3mBuuX7liaL8cI+ w6BzzROllWweOxaDnSMq/1/jXvZlIYNKJUhvC1fGJrWZ+fFBFdEExiEDAe04DlRMaw2t FVn5FeFNBUGaRB/3efNhg/Hfb7kbO3xnuPLKvoLIMD9ULEye/Gs0k3H6bv8uwRJt5isH 6NqG7G7P0EvKzQrMkeeh4lhGoTzcHKm0EBazcfMxx0krvExScpq3UrDguhbX1MmEM3cw 2llbyi71hYEho7XJw3HF7O5w7exOE+fBsjcFgnYWIW1yAq+tIX72KBMH3kKUShzpwiz4 DbvA== X-Gm-Message-State: AFqh2korx5Aya+Wqdq6zu5JrL9HIy8wVV+7A2ii0K5HodjLSJMKoyCiJ D3s6lcdlxGLVuBAUiHd26kqyHLYGu8c= X-Google-Smtp-Source: AMrXdXst77UHpLJqCmQb9AJXcKHvoNkTjCkoalZ4IjIuxaP38Wsbu5XItkGHI6RyDU4YHtWaPx6wVQ== X-Received: by 2002:a05:600c:1e09:b0:3cf:b73f:bf8f with SMTP id ay9-20020a05600c1e0900b003cfb73fbf8fmr4554727wmb.7.1671722061694; Thu, 22 Dec 2022 07:14:21 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id c12-20020a05600c0a4c00b003cfa3a12660sm12017792wmq.1.2022.12.22.07.14.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:21 -0800 (PST) Message-Id: <3267a5b3b372d9b33ad3afd22bde7ba5559331ae.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:08 +0000 Subject: [PATCH v4 02/11] t: create test harness for 'bundle-uri' command MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= The previous change allowed for a Git server to advertise the 'bundle-uri' command as a capability based on the uploadPack.advertiseBundleURIs config option. Create a set of tests that check that this capability is advertised using 'git ls-remote'. In order to test this functionality across three protocols (file, git, and http), create lib-bundle-uri-protocol.sh to generalize the tests, allowing the other test scripts to set an environment variable and otherwise inherit the setup and tests from this script. The tests currently only test that the 'bundle-uri' command is advertised or not. Other actions will be tested as the Git client learns to request the 'bundle-uri' command and parse its response. To help with URI escaping, specifically for file paths with a space in them, extract a 'sed' invocation from t9199-git-svn-info.sh into a helper function for use here, too. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- t/lib-bundle-uri-protocol.sh | 85 ++++++++++++++++++++++++++ t/t5730-protocol-v2-bundle-uri-file.sh | 17 ++++++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 ++++++ t/t5732-protocol-v2-bundle-uri-http.sh | 17 ++++++ t/t9119-git-svn-info.sh | 2 +- t/test-lib-functions.sh | 7 +++ 6 files changed, 144 insertions(+), 1 deletion(-) create mode 100644 t/lib-bundle-uri-protocol.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh new file mode 100644 index 00000000000..2da22a39cb8 --- /dev/null +++ b/t/lib-bundle-uri-protocol.sh @@ -0,0 +1,85 @@ +# Set up and run tests of the 'bundle-uri' command in protocol v2 +# +# The test that includes this script should set BUNDLE_URI_PROTOCOL +# to one of "file", "git", or "http". + +BUNDLE_URI_TEST_PARENT= +BUNDLE_URI_TEST_URI= +BUNDLE_URI_TEST_BUNDLE_URI= +case "$BUNDLE_URI_PROTOCOL" in +file) + BUNDLE_URI_PARENT=file_parent + BUNDLE_URI_REPO_URI="file://$PWD/file_parent" + BUNDLE_URI_BUNDLE_URI="$BUNDLE_URI_REPO_URI/fake.bdl" + test_set_prereq BUNDLE_URI_FILE + ;; +git) + . "$TEST_DIRECTORY"/lib-git-daemon.sh + start_git_daemon --export-all --enable=receive-pack + BUNDLE_URI_PARENT="$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent" + BUNDLE_URI_REPO_URI="$GIT_DAEMON_URL/parent" + BUNDLE_URI_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq BUNDLE_URI_GIT + ;; +http) + . "$TEST_DIRECTORY"/lib-httpd.sh + start_httpd + BUNDLE_URI_PARENT="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" + BUNDLE_URI_REPO_URI="$HTTPD_URL/smart/http_parent" + BUNDLE_URI_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq BUNDLE_URI_HTTP + ;; +*) + BUG "Need to pass valid BUNDLE_URI_PROTOCOL (was \"$BUNDLE_URI_PROTOCOL\")" + ;; +esac + +test_expect_success "setup protocol v2 $BUNDLE_URI_PROTOCOL:// tests" ' + git init "$BUNDLE_URI_PARENT" && + test_commit -C "$BUNDLE_URI_PARENT" one && + git -C "$BUNDLE_URI_PARENT" config uploadpack.advertiseBundleURIs true +' + +case "$BUNDLE_URI_PROTOCOL" in +http) + test_expect_success "setup config for $BUNDLE_URI_PROTOCOL:// tests" ' + git -C "$BUNDLE_URI_PARENT" config http.receivepack true + ' + ;; +*) + ;; +esac +BUNDLE_URI_BUNDLE_URI_ESCAPED=$(echo "$BUNDLE_URI_BUNDLE_URI" | test_uri_escape) + +test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: no bundle-uri" ' + test_when_finished "rm -f log" && + test_when_finished "git -C \"$BUNDLE_URI_PARENT\" config uploadpack.advertiseBundleURIs true" && + git -C "$BUNDLE_URI_PARENT" config uploadpack.advertiseBundleURIs false && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$BUNDLE_URI_REPO_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + ! grep bundle-uri log +' + +test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: have bundle-uri" ' + test_when_finished "rm -f log" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$BUNDLE_URI_REPO_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log +' diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh new file mode 100755 index 00000000000..37bdb725bca --- /dev/null +++ b/t/t5730-protocol-v2-bundle-uri-file.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'file://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'file://' transport +# +BUNDLE_URI_PROTOCOL=file +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh new file mode 100755 index 00000000000..8add1b37abc --- /dev/null +++ b/t/t5731-protocol-v2-bundle-uri-git.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +BUNDLE_URI_PROTOCOL=git +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh new file mode 100755 index 00000000000..129daa02269 --- /dev/null +++ b/t/t5732-protocol-v2-bundle-uri-http.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'http://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'http://' transport +# +BUNDLE_URI_PROTOCOL=http +. "$TEST_DIRECTORY"/lib-bundle-uri-protocol.sh + +test_done diff --git a/t/t9119-git-svn-info.sh b/t/t9119-git-svn-info.sh index 8201c3e808a..088d1c57a88 100755 --- a/t/t9119-git-svn-info.sh +++ b/t/t9119-git-svn-info.sh @@ -28,7 +28,7 @@ test_cmp_info () { rm -f tmp.expect tmp.actual } -quoted_svnrepo="$(echo $svnrepo | sed 's/ /%20/')" +quoted_svnrepo="$(echo $svnrepo | test_uri_escape)" test_expect_success 'setup repository and import' ' mkdir info && diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh index 29d914a12ba..5f6966a404b 100644 --- a/t/test-lib-functions.sh +++ b/t/test-lib-functions.sh @@ -1755,6 +1755,13 @@ test_path_is_hidden () { return 1 } +# Poor man's URI escaping. Good enough for the test suite whose trash +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art. +test_uri_escape() { + sed 's/ /%20/g' +} + # Check that the given command was invoked as part of the # trace2-format trace on stdin. # From patchwork Thu Dec 22 15:14:09 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13080130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88A97C4332F for ; Thu, 22 Dec 2022 15:14:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235647AbiLVPOq (ORCPT ); Thu, 22 Dec 2022 10:14:46 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59156 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235516AbiLVPOZ (ORCPT ); Thu, 22 Dec 2022 10:14:25 -0500 Received: from mail-wm1-x32c.google.com (mail-wm1-x32c.google.com [IPv6:2a00:1450:4864:20::32c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 488B611A08 for ; Thu, 22 Dec 2022 07:14:24 -0800 (PST) Received: by mail-wm1-x32c.google.com with SMTP id i82-20020a1c3b55000000b003d1e906ca23so1422798wma.3 for ; Thu, 22 Dec 2022 07:14:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=kqYBa7AYck1KXjiGuWcjJXZHxBlNJrFfsuzq+UJ+C08=; b=dfhC0cd6MjtqS7eKkGn0cctp6IeAx0nBfIeNIA1p+bUwT5zlS+kn852Gz9WwA1lgN0 DgN7urNwe6SAeNN3wvtsaZgpZbbhijkN3CB0teNtFhXjmssJ8T8PVFM811qKyPul3nMD BlDCt+D/qCSH2P+OgBcLSe6F+MXfLdpmmOm9fT7J9kuWHi+XtN7wl+fOgW/QDPMIFLsO 1dJWi5znVO+M6ltI42mWRRZYxF1y9No6+W4eBArn4CtVIAUuJZQvM+tVQ74sjBXnYaBg 45jSw950TIjYPI0TrUp6w1H4tYiXoFNDYSPuoRta6OcyDaoV6py7+RHMTDbiSQWkHwt4 uVcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kqYBa7AYck1KXjiGuWcjJXZHxBlNJrFfsuzq+UJ+C08=; b=58OMYesEJWanJ99oPjtNVNXBr5Zch8CjF7PSB8yUM4zNgvOUuKkwaHmS1geL4+9OAG tYKz74QGgIkd2zI3YiUDKbvEof9ZGkAs5SXmNo7s0uQaVnOCEJLzHf3jD2K1n8mdhL6f su7vXDyDgYO4QYdn+HlQmJn8/Y9bGIP08WLhMYbrRDhOHZRxPtoBwFeDsYu9J0sVLFv5 Wrs6zOMC857RAUumTaJ/hc3W9Ef476wVllx4a76qEHqYyXCMzmPyFl2ngj5o7Hk2RvZR NGRRxrTyWYSRN4XAvFlKplmO2aBBKRZ3vaiP+yf594M9W3zkS1bIvYzDxAuW8Uh8Q4PI 9kEw== X-Gm-Message-State: AFqh2koALBlORta+hHut2jPi1y8OLE64rWL5qWjdrYVMBOx1xtQk555O lUz2DB5njBeTOtEHMpI5xw7o94dA4Fo= X-Google-Smtp-Source: AMrXdXvQrltfKfcIlXGOQHFMOr+WGSwbGdPhXZU8BGkBDU4GvqfLmGPwYfog+zlx3VzgEJ8m/4sI+g== X-Received: by 2002:a05:600c:22d9:b0:3d3:56e6:4f0d with SMTP id 25-20020a05600c22d900b003d356e64f0dmr4563669wmg.7.1671722062613; Thu, 22 Dec 2022 07:14:22 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h6-20020adffa86000000b0024246991121sm782777wrr.116.2022.12.22.07.14.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:22 -0800 (PST) Message-Id: <5bf6fe55771a5687d711c59cf0460baa087603a3.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:09 +0000 Subject: [PATCH v4 03/11] clone: request the 'bundle-uri' command when available MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Set up all the needed client parts of the 'bundle-uri' protocol v2 command, without actually doing anything with the bundle URIs. If the server says it supports 'bundle-uri' teach Git to issue the 'bundle-uri' command after the 'ls-refs' during 'git clone'. The returned key=value pairs are passed to the bundle list code which is tested using a different ingest mechanism in t5750-bundle-uri-parse.sh. At this point, Git does nothing with that bundle list. It will not download any of the bundles. That will come in a later change after these protocol bits are finalized. The no-op client is initially used only by 'git clone' to test the basic functionality, and eventually will bootstrap the initial download of Git objects during a fresh clone. The bundle URI client will not be integrated into other fetches until a mechanism is created to select a subset of bundles for download. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- builtin/clone.c | 6 +++++ connect.c | 44 +++++++++++++++++++++++++++++++ remote.h | 5 ++++ t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++ transport-helper.c | 13 +++++++++ transport-internal.h | 7 +++++ transport.c | 51 ++++++++++++++++++++++++++++++++++++ transport.h | 19 ++++++++++++++ 8 files changed, 164 insertions(+) diff --git a/builtin/clone.c b/builtin/clone.c index 547d6464b3c..39364c25b15 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1266,6 +1266,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/connect.c b/connect.c index 5ea53deda23..624a10f18ee 100644 --- a/connect.c +++ b/connect.c @@ -15,6 +15,7 @@ #include "version.h" #include "protocol.h" #include "alias.h" +#include "bundle-uri.h" static char *server_capabilities_v1; static struct strvec server_capabilities_v2 = STRVEC_INIT; @@ -491,6 +492,49 @@ static void send_capabilities(int fd_out, struct packet_reader *reader) } } +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc) +{ + int line_nr = 1; + + /* Assert bundle-uri support */ + server_supports_v2("bundle-uri", 1); + + /* (Re-)send capabilities */ + send_capabilities(fd_out, reader); + + /* Send command */ + packet_write_fmt(fd_out, "command=bundle-uri\n"); + packet_delim(fd_out); + + packet_flush(fd_out); + + /* Process response from server */ + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { + const char *line = reader->line; + line_nr++; + + if (!bundle_uri_parse_line(bundles, line)) + continue; + + return error(_("error on bundle-uri response line %d: %s"), + line_nr, line); + } + + if (reader->status != PACKET_READ_FLUSH) + return error(_("expected flush after bundle-uri listing")); + + /* + * Might die(), but obscure enough that that's OK, e.g. in + * serve.c we'll call BUG() on its equivalent (the + * PACKET_READ_RESPONSE_END check). + */ + check_stateless_delimiter(stateless_rpc, reader, + _("expected response end packet after ref listing")); + + return 0; +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, diff --git a/remote.h b/remote.h index 1c4621b414b..1ebbe42792e 100644 --- a/remote.h +++ b/remote.h @@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, const struct string_list *server_options, int stateless_rpc); +/* Used for protocol v2 in order to retrieve refs from a remote */ +struct bundle_list; +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc); + int resolve_remote_symref(struct ref *ref, struct ref *list); /* diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 2da22a39cb8..d44c6e10f9e 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -83,3 +83,22 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav # Server advertised bundle-uri capability grep "< bundle-uri" log ' + +test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' + test_when_finished "rm -rf log cloned" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + clone "$BUNDLE_URI_REPO_URI" cloned \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log && + + # Client issued bundle-uri command + grep "> command=bundle-uri" log +' diff --git a/transport-helper.c b/transport-helper.c index e95267a4ab5..3ea7c2bb5ad 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport, return ret; } +static int get_bundle_uri(struct transport *transport) +{ + get_helper(transport); + + if (process_connect(transport, 0)) { + do_take_over(transport); + return transport->vtable->get_bundle_uri(transport); + } + + return -1; +} + static struct transport_vtable vtable = { .set_option = set_helper_option, .get_refs_list = get_refs_list, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs, .push_refs = push_refs, .connect = connect_helper, diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733ac..90ea749e5cf 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -26,6 +26,13 @@ struct transport_vtable { struct ref *(*get_refs_list)(struct transport *transport, int for_push, struct transport_ls_refs_options *transport_options); + /** + * Populates the remote side's bundle-uri under protocol v2, + * if the "bundle-uri" capability was advertised. Returns 0 if + * OK, negative values on error. + */ + int (*get_bundle_uri)(struct transport *transport); + /** * Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. diff --git a/transport.c b/transport.c index e7b97194c10..b6f279e92cb 100644 --- a/transport.c +++ b/transport.c @@ -22,6 +22,7 @@ #include "protocol.h" #include "object-store.h" #include "color.h" +#include "bundle-uri.h" static int transport_use_color = -1; static char transport_colors[][COLOR_MAXLEN] = { @@ -359,6 +360,32 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus return handshake(transport, for_push, options, 1); } +static int get_bundle_uri(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct packet_reader reader; + int stateless_rpc = transport->stateless_rpc; + + if (!transport->bundles) { + CALLOC_ARRAY(transport->bundles, 1); + init_bundle_list(transport->bundles); + } + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF); + + return get_remote_bundle_uri(data->fd[1], &reader, + transport->bundles, stateless_rpc); +} + static int fetch_refs_via_pack(struct transport *transport, int nr_heads, struct ref **to_fetch) { @@ -902,6 +929,7 @@ static int disconnect_git(struct transport *transport) static struct transport_vtable taken_over_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .disconnect = disconnect_git @@ -1054,6 +1082,7 @@ static struct transport_vtable bundle_vtable = { static struct transport_vtable builtin_smart_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .connect = connect_git, @@ -1068,6 +1097,9 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->progress = isatty(2); string_list_init_dup(&ret->pack_lockfiles); + CALLOC_ARRAY(ret->bundles, 1); + init_bundle_list(ret->bundles); + if (!remote) BUG("No remote provided to transport_get()"); @@ -1482,6 +1514,23 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } +int transport_get_remote_bundle_uri(struct transport *transport) +{ + const struct transport_vtable *vtable = transport->vtable; + + /* Check config only once. */ + if (transport->got_remote_bundle_uri) + return 0; + transport->got_remote_bundle_uri = 1; + + if (!vtable->get_bundle_uri) + return error(_("bundle-uri operation not supported by protocol")); + + if (vtable->get_bundle_uri(transport) < 0) + return error(_("could not retrieve server-advertised bundle-uri list")); + return 0; +} + void transport_unlock_pack(struct transport *transport, unsigned int flags) { int in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER); @@ -1512,6 +1561,8 @@ int transport_disconnect(struct transport *transport) ret = transport->vtable->disconnect(transport); if (transport->got_remote_refs) free_refs((void *)transport->remote_refs); + clear_bundle_list(transport->bundles); + free(transport->bundles); free(transport); return ret; } diff --git a/transport.h b/transport.h index b5bf7b3e704..85150f504fb 100644 --- a/transport.h +++ b/transport.h @@ -62,6 +62,7 @@ enum transport_family { TRANSPORT_FAMILY_IPV6 }; +struct bundle_list; struct transport { const struct transport_vtable *vtable; @@ -76,6 +77,18 @@ struct transport { */ unsigned got_remote_refs : 1; + /** + * Indicates whether we already called get_bundle_uri_list(); set by + * transport.c::transport_get_remote_bundle_uri(). + */ + unsigned got_remote_bundle_uri : 1; + + /* + * The results of "command=bundle-uri", if both sides support + * the "bundle-uri" capability. + */ + struct bundle_list *bundles; + /* * Transports that call take-over destroys the data specific to * the transport type while doing so, and cannot be reused. @@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts); const struct ref *transport_get_remote_refs(struct transport *transport, struct transport_ls_refs_options *transport_options); +/** + * Retrieve bundle URI(s) from a remote. Populates "struct + * transport"'s "bundle_uri" and "got_remote_bundle_uri". + */ +int transport_get_remote_bundle_uri(struct transport *transport); + /* * Fetch the hash algorithm used by a remote. * From patchwork Thu Dec 22 15:14:10 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13080132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3996C4332F for ; Thu, 22 Dec 2022 15:14:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235712AbiLVPOx (ORCPT ); Thu, 22 Dec 2022 10:14:53 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59962 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235574AbiLVPO0 (ORCPT ); Thu, 22 Dec 2022 10:14:26 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 27494FE2 for ; Thu, 22 Dec 2022 07:14:25 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id ja17so1626648wmb.3 for ; Thu, 22 Dec 2022 07:14:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=8U8zfWN47or1CWj7YTDiNX6qhV2CR343X60qq8TUyW4=; b=Kr7G1DMLEsCV7Aa2Uyg/xKYc/YEqY5FNKQbKNA0MpHYPsr0LBjy4oxrg97Sc4uJDlR NzdgN/enNWyMWO37vz2D8GXwYLVhNV6EwGSGa2IdwO/CSqq4XC+AoU+Hq0MD+Kltha6Z SmSnRnQdxpAsoq1oEAgmhR+hOUZVHitY6INJcglX2VixLNpLd/wmheQPwRP1pjFiGm9h rG3siwGpQAvdNfIb/hIYZxh0dSfg3EWTA6BYpIXGY1u59ov1Jq6bDHLjbljHXttT+Scd +uQfWPfq5o9ukqBIN2XrfWFUjPvuKBuZhwiCAYBabvwadYIRxkHOmIqIxQiDbWT0HsF/ f8Yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8U8zfWN47or1CWj7YTDiNX6qhV2CR343X60qq8TUyW4=; b=s52geKItygIYC51T+va0wRi+c7PJZ4zYCb7zK0HSEcUeI6F59SotrlYxunx6zFIQaT FK+SZyDUplZOoJKGMYEPWQu0i0HhpzBriOwrnzCNDk8e3bYaJn8Z5mGDZfcJoLR8eYjo KH0H3ipZSPbzatT6YlYLqYhKNT1dLWmaR0u3N5XkTD37ejqmBnqezSCfwfC5T4UnXZQG BZY3LVSamqjjepODOT5pe4sB9l+D0P1rlOkFpLJ+/0aMEzt+PbLjGizgsDU8M/hwBgHC A0GwuJP7yaN7cMGMH58Q7EMM2LzkSBuwNEFg+WvCi4eSqeLETrhpGgJGd+IAc8CbAp2F 5gaA== X-Gm-Message-State: AFqh2kpUyC0s8tp5VEOcyD3+QZLtYOr6BXUQfLBHhweOUCsesoqrtX9C dHrpr7J8eFGv6tC5scj4UcI/ESdGQf0= X-Google-Smtp-Source: AMrXdXvK6XfhauN0xjiF8blgQdwEsoXI6WKRmzGy+PTbS0ySGvni/MtyuKbAedvKqkaziN0o3D0YUw== X-Received: by 2002:a05:600c:1d03:b0:3d3:4aa6:4fe6 with SMTP id l3-20020a05600c1d0300b003d34aa64fe6mr4679516wms.3.1671722063385; Thu, 22 Dec 2022 07:14:23 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o15-20020a05600c4fcf00b003c6f3f6675bsm6770532wmq.26.2022.12.22.07.14.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:23 -0800 (PST) Message-Id: <876dd3f221fe575dc0407983e15280de4e9e18e0.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:10 +0000 Subject: [PATCH v4 04/11] bundle-uri client: add boolean transfer.bundleURI setting MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= The yet-to-be introduced client support for bundle-uri will always fall back on a full clone, but we'd still like to be able to ignore a server's bundle-uri advertisement entirely. The new transfer.bundleURI config option defaults to 'false', but a user can set it to 'true' to enable checking for bundle URIs from the origin Git server using protocol v2. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/config/transfer.txt | 6 ++++++ t/lib-bundle-uri-protocol.sh | 19 ++++++++++++++++++- transport.c | 8 ++++++++ 3 files changed, 32 insertions(+), 1 deletion(-) diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt index 264812cca4d..c3ac767d1e4 100644 --- a/Documentation/config/transfer.txt +++ b/Documentation/config/transfer.txt @@ -115,3 +115,9 @@ transfer.unpackLimit:: transfer.advertiseSID:: Boolean. When true, client and server processes will advertise their unique session IDs to their remote counterpart. Defaults to false. + +transfer.bundleURI:: + When `true`, local `git clone` commands will request bundle + information from the remote server (if advertised) and download + bundles before continuing the clone through the Git protocol. + Defaults to `false`. diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index d44c6e10f9e..75ea8c4418f 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -85,10 +85,11 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav ' test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' - test_when_finished "rm -rf log cloned" && + test_when_finished "rm -rf log cloned cloned2" && GIT_TRACE_PACKET="$PWD/log" \ git \ + -c transfer.bundleURI=false \ -c protocol.version=2 \ clone "$BUNDLE_URI_REPO_URI" cloned \ >actual 2>err && @@ -99,6 +100,22 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque # Server advertised bundle-uri capability grep "< bundle-uri" log && + # Client did not issue bundle-uri command + ! grep "> command=bundle-uri" log && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c transfer.bundleURI=true \ + -c protocol.version=2 \ + clone "$BUNDLE_URI_REPO_URI" cloned2 \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log && + # Client issued bundle-uri command grep "> command=bundle-uri" log ' diff --git a/transport.c b/transport.c index b6f279e92cb..b4cf2c0252e 100644 --- a/transport.c +++ b/transport.c @@ -1516,6 +1516,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) int transport_get_remote_bundle_uri(struct transport *transport) { + int value = 0; const struct transport_vtable *vtable = transport->vtable; /* Check config only once. */ @@ -1523,6 +1524,13 @@ int transport_get_remote_bundle_uri(struct transport *transport) return 0; transport->got_remote_bundle_uri = 1; + /* + * Don't request bundle-uri from the server unless configured to + * do so by the transfer.bundleURI=true config option. + */ + if (git_config_get_bool("transfer.bundleuri", &value) || !value) + return 0; + if (!vtable->get_bundle_uri) return error(_("bundle-uri operation not supported by protocol")); From patchwork Thu Dec 22 15:14:11 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13080131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E196C10F1D for ; Thu, 22 Dec 2022 15:14:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235673AbiLVPOv (ORCPT ); Thu, 22 Dec 2022 10:14:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59672 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235345AbiLVPO1 (ORCPT ); Thu, 22 Dec 2022 10:14:27 -0500 Received: from mail-wr1-x430.google.com (mail-wr1-x430.google.com [IPv6:2a00:1450:4864:20::430]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D629B23157 for ; Thu, 22 Dec 2022 07:14:25 -0800 (PST) Received: by mail-wr1-x430.google.com with SMTP id w15so1950397wrl.9 for ; Thu, 22 Dec 2022 07:14:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=AXP7ZJ7T/+kkP8EsJl/0WuAJ4TG9ORZbUaFHwLWlnYQ=; b=aaeknxAaHcFZlX5DT7xqol49wp62IV3pS5bUuaz7P2HWdJmyZ2/f5r6990VgIrWtiB SvYGapynZUWo8N8cVBTw3G5d/5iIChIZsK4j9PvLmOYSHwrs/4gLPbARs08i7+9S51Wc GJFhwpqLMNOjcT0qu1jgBdEw8JWLD3V3S3GuXu5suoqPrXAnXnVRHHdSJ0IJIaEcW47T dUAzmFxlTxE8tOQpczYxByR1RcaquqFgFviuU2UEbHpqSCLW6XY4SatIOq5YlWdq1oD7 qLTOQCCkLhblOQVYn45rdul8cqigv3ll34V5Ai8ZrEdRy6gxnk+a30QL18QPX4cc4u1S 8wpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AXP7ZJ7T/+kkP8EsJl/0WuAJ4TG9ORZbUaFHwLWlnYQ=; b=iEFxuSxKGPuT5CRncM+j4fcOrXZv5Ql1q/ed2egJWQwrllKhbLD0Z6rHiDJlAldbUP 7S5mycgfSxZfCnZFtgYPWg21D9ikGxzkTxC5yrCw4wMNGBGmGxu2nfuknIAxP9DdTVDx Ist8qyB3tw1KaQh05ojosXMTnyn/Qs8Meyr59WoNOBv1tzya4XEsEIUWEJVwKFverAwj 91iiBK4d7WE/0x/0FWVNu7jUIMRgvAeFiddFwfqB9uoqCH+i8RsAPESlY4cIoGDKPMA+ GYgTzdoY9KxIssWMk0ZAlTjfqzXP83LXq2L5ZC1/51aZC2Dv/d+cuUmvhXOL15QJhxMl 7wiQ== X-Gm-Message-State: AFqh2kpEWSp1bYDI31PNc25IgdylWHxro1Mn38PkchTzyH0ZgIYZIP5L aalk69DyerOmlUReAGWt8d+LGKyQsY0= X-Google-Smtp-Source: AMrXdXtV0y+/g54OpNhEEiD+s9z0I3LAVVsZo2kWXUrKd1aHIdsfUqBrKYRhsk/ZbyXRQrMV1mm9xg== X-Received: by 2002:adf:e8c1:0:b0:242:782c:f397 with SMTP id k1-20020adfe8c1000000b00242782cf397mr3568243wrn.25.1671722064243; Thu, 22 Dec 2022 07:14:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id w14-20020adfee4e000000b002422bc69111sm984577wro.9.2022.12.22.07.14.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:23 -0800 (PST) Message-Id: <8f5a483c32958b8ab2ac6deeef383165e7214260.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:11 +0000 Subject: [PATCH v4 05/11] transport: rename got_remote_heads Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The 'got_remote_heads' member of 'struct git_transport_data' was used historically to indicate that the initial server connection was made and the ref advertisement was returned. With protocol v2, that initial handshake does not necessarily include the ref advertisement, so this member is not an accurate name. Thankfully, all uses of the member are only checking to see if the handshake should take place, not whether or not some local data has the ref advertisement. Rename the member to 'finished_handshake' to represent the proper state. Note that the variable is only set to 1 during the handshake() method. Signed-off-by: Derrick Stolee --- transport.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/transport.c b/transport.c index b4cf2c0252e..757ad552bf3 100644 --- a/transport.c +++ b/transport.c @@ -198,7 +198,7 @@ struct git_transport_data { struct git_transport_options options; struct child_process *conn; int fd[2]; - unsigned got_remote_heads : 1; + unsigned finished_handshake : 1; enum protocol_version version; struct oid_array extra_have; struct oid_array shallow; @@ -345,7 +345,7 @@ static struct ref *handshake(struct transport *transport, int for_push, case protocol_unknown_version: BUG("unknown protocol version"); } - data->got_remote_heads = 1; + data->finished_handshake = 1; transport->hash_algo = reader.hash_algo; if (reader.line_peeked) @@ -421,7 +421,7 @@ static int fetch_refs_via_pack(struct transport *transport, args.negotiation_tips = data->options.negotiation_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow; - if (!data->got_remote_heads) { + if (!data->finished_handshake) { int i; int must_list_refs = 0; for (i = 0; i < nr_heads; i++) { @@ -461,7 +461,7 @@ static int fetch_refs_via_pack(struct transport *transport, to_fetch, nr_heads, &data->shallow, &transport->pack_lockfiles, data->version); - data->got_remote_heads = 0; + data->finished_handshake = 0; data->options.self_contained_and_connected = args.self_contained_and_connected; data->options.connectivity_checked = args.connectivity_checked; @@ -846,7 +846,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re if (transport_color_config() < 0) return -1; - if (!data->got_remote_heads) + if (!data->finished_handshake) get_refs_via_connect(transport, 1, NULL); memset(&args, 0, sizeof(args)); @@ -894,7 +894,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re else ret = finish_connect(data->conn); data->conn = NULL; - data->got_remote_heads = 0; + data->finished_handshake = 0; return ret; } @@ -914,7 +914,7 @@ static int disconnect_git(struct transport *transport) { struct git_transport_data *data = transport->data; if (data->conn) { - if (data->got_remote_heads && !transport->stateless_rpc) + if (data->finished_handshake && !transport->stateless_rpc) packet_flush(data->fd[1]); close(data->fd[0]); if (data->fd[1] >= 0) @@ -949,7 +949,7 @@ void transport_take_over(struct transport *transport, data->conn = child; data->fd[0] = data->conn->out; data->fd[1] = data->conn->in; - data->got_remote_heads = 0; + data->finished_handshake = 0; transport->data = data; transport->vtable = &taken_over_vtable; @@ -1150,7 +1150,7 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->smart_options = &(data->options); data->conn = NULL; - data->got_remote_heads = 0; + data->finished_handshake = 0; } else { /* Unknown protocol in URL. Pass to external handler. */ int len = external_specification_len(url); From patchwork Thu Dec 22 15:14:12 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13080134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88A53C10F1D for ; Thu, 22 Dec 2022 15:15:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235477AbiLVPPA (ORCPT ); Thu, 22 Dec 2022 10:15:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59848 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235447AbiLVPO2 (ORCPT ); Thu, 22 Dec 2022 10:14:28 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E09A822B0A for ; Thu, 22 Dec 2022 07:14:26 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id c65-20020a1c3544000000b003cfffd00fc0so4123414wma.1 for ; Thu, 22 Dec 2022 07:14:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=4qP5ArxwgVVOvZLxFNIjuay1SnGTTd27mCoyuwFDBro=; b=auAYIsjaLEpMIMcGr/fm8B2xrddwK7W/Nh6JGYCmhWctbtyuxTrcS/alJjJnsrL0zw /c9aUwCKFWmaoWKbZXfaULcJFx1v2339TU3TtEpW+E2M5A5B9pgynQ0zUB3jUzPFQudd wUjF+YOxELZqjT6Cun1xKsfB3R551u4vw+0MwqfqjKjfKXzXxEGgdbFVXx1mBMBpgqFd 0733ngVnPYvz/Ik3dyuqPWLj1z6Pm2vy9nHUhFc3teLKFpCs8pYwwpybuHA3PacbxFB7 heqL2bml/Zb2ygPn1q0T5eU4Mn7YNOy/dB7l2kbDUMaCjFbtagmir0PMMhVkTN9FkWTK SM7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4qP5ArxwgVVOvZLxFNIjuay1SnGTTd27mCoyuwFDBro=; b=4WX/4AzrqGjO+pUPpypcVHPmkAG5Ats2ao+3vnT1fsKsPvJc1MriWYhzrDHu3I57YS qXeGEGQ2zX5PAtQeWq+WMwnJuSwioREWHzXDAvMa6mN0GN4MMyegq7W8whZ1jmsSnscW yWQx15h0fegJTJoh54rxhg2w1CNzg4LKp6yG/waIYs1Ga9OFofT7Gn7exHbYk8+mg0PA gVGN5/FD2F7L9w/pI7a7L9HxG32W3cknzMmPTWxY7iTnObbbHXqKqhr1/bhQRkXr2/8n td6TjeReFMT65be0r/SKvVk6lqdLEk3PLDvQHjtN5qZpIlKOqA7FgHOBFjyfl+MoYOIb 8FAA== X-Gm-Message-State: AFqh2krI8stmqyXXEEclS7RTJPrp9cxBhh6v554uo3C88dmjyDFNXD3H 3IrtjsIl2MGtE9BmCoxgovWId/+Hw1g= X-Google-Smtp-Source: AMrXdXtXeC/j5gp1tuEdX+ZvqmcEziPnmGrzv41mMyV5I4PeUq3g4qALcr3MHZUdqRPOSSvdmTjd6A== X-Received: by 2002:a05:600c:3d86:b0:3d1:e04f:9bfa with SMTP id bi6-20020a05600c3d8600b003d1e04f9bfamr4853970wmb.28.1671722064953; Thu, 22 Dec 2022 07:14:24 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n9-20020a05600c4f8900b003d1b4d957aasm1548253wmq.36.2022.12.22.07.14.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:24 -0800 (PST) Message-Id: <13e4c82e3380d8b91583550e61671bd8eac69ab1.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:12 +0000 Subject: [PATCH v4 06/11] bundle-uri client: add helper for testing server MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper for issuing protocol v2 "bundle-uri" commands to a server, and to the parsing routines in bundle-uri.c. In the "git clone" case we'll have already done the handshake(), but not here. Add an extra case to check for this handshake in get_bundle_uri() for ease of use for future callers. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- t/helper/test-bundle-uri.c | 46 ++++++++++++++++++++++++++++++++++++ t/lib-bundle-uri-protocol.sh | 46 ++++++++++++++++++++++++++++++++++++ transport.c | 7 ++++++ 3 files changed, 99 insertions(+) diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index 25afd393428..f8159187014 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -3,6 +3,10 @@ #include "bundle-uri.h" #include "strbuf.h" #include "string-list.h" +#include "transport.h" +#include "ref-filter.h" +#include "remote.h" +#include "refs.h" enum input_mode { KEY_VALUE_PAIRS, @@ -68,6 +72,46 @@ usage: usage_with_options(usage, options); } +static int cmd_ls_remote(int argc, const char **argv) +{ + const char *uploadpack = NULL; + struct string_list server_options = STRING_LIST_INIT_DUP; + const char *dest; + struct remote *remote; + struct transport *transport; + int status = 0; + + dest = argc > 1 ? argv[1] : NULL; + + remote = remote_get(dest); + if (!remote) { + if (dest) + die(_("bad repository '%s'"), dest); + die(_("no remote configured to get bundle URIs from")); + } + if (!remote->url_nr) + die(_("remote '%s' has no configured URL"), dest); + + transport = transport_get(remote, NULL); + if (uploadpack) + transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack); + if (server_options.nr) + transport->server_options = &server_options; + + if (transport_get_remote_bundle_uri(transport) < 0) { + error(_("could not get the bundle-uri list")); + status = 1; + goto cleanup; + } + + print_bundle_list(stdout, transport->bundles); + +cleanup: + if (transport_disconnect(transport)) + return 1; + return status; +} + int cmd__bundle_uri(int argc, const char **argv) { const char *usage[] = { @@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv) return cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS); if (!strcmp(argv[1], "parse-config")) return cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE); + if (!strcmp(argv[1], "ls-remote")) + return cmd_ls_remote(argc - 1, argv + 1); error("there is no test-tool bundle-uri tool '%s'", argv[1]); usage: diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 75ea8c4418f..5620e230387 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -119,3 +119,49 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque # Client issued bundle-uri command grep "> command=bundle-uri" log ' + +# The remaining tests will all assume transfer.bundleURI=true +# +# This test can be removed when transfer.bundleURI is enabled by default. +test_expect_success 'enable transfer.bundleURI for remaining tests' ' + git config --global transfer.bundleURI true +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.only.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 and extra data" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.only.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED" && + + # Extra data should be ignored + test_config -C "$BUNDLE_URI_PARENT" bundle.only.extra bogus && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' diff --git a/transport.c b/transport.c index 757ad552bf3..0f35114a13e 100644 --- a/transport.c +++ b/transport.c @@ -371,6 +371,13 @@ static int get_bundle_uri(struct transport *transport) init_bundle_list(transport->bundles); } + if (!data->finished_handshake) { + struct ref *refs = handshake(transport, 0, NULL, 0); + + if (refs) + free_refs(refs); + } + /* * "Support" protocol v0 and v2 without bundle-uri support by * silently degrading to a NOOP. From patchwork Thu Dec 22 15:14:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13080133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B39DC4332F for ; Thu, 22 Dec 2022 15:14:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230134AbiLVPO6 (ORCPT ); Thu, 22 Dec 2022 10:14:58 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59846 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235389AbiLVPO1 (ORCPT ); Thu, 22 Dec 2022 10:14:27 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 52E892DD5 for ; Thu, 22 Dec 2022 07:14:26 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id h7so1957811wrs.6 for ; Thu, 22 Dec 2022 07:14:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=HwCw5MWgk/AvEUrGuGpC4uxSq9gDladWOjROHmwfeS4=; b=WQ03jQOW/sUrvFX3zeDOCExVqpPzTojgkVuy6EbmwbHjo9ca2Y/I1s6+Foi5rHNkZq DaMPcAUQwDFHyZ2t4xzCWeM9XOSO1X8P/chntMgrrXlJPTx1ztNsSsDU4wO0ifoEJNHf ay8lzsQ5P89pzgPUYUyyrTAacpkxUWziByTHeIvtcoQ6F+IG+ynGQE1xeg/45SJ2cMmE +Nlmoeo75a4MxhyQqA9wGkSmQLTsToYwBQd5nfX9F/0X0tp5+JGvZoKJS1y5bVSFI+sb /whCUyT+nze//zu8ppcK7xY/kzNOZ+X7KGTS6hdiHdPJj+ZZNOxYZGQBsto1BlW8k9Ew IBPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HwCw5MWgk/AvEUrGuGpC4uxSq9gDladWOjROHmwfeS4=; b=HZTacapGLzaPN7QVAeiL/jEDwhxLe8QTT1pF+XaDdBEN/iRPju2I3HnU5gtde7HOIg mkpk7ymkbherAHyjk4A5Y1FND5oOjUA7R4nIrWql7i5AaUh1dCCcnsy27c5pmxgE1pjb h/U1OKqzLN3v3mjxhaWGvdcjM91mGTCUM1Zq31XHczfw0iI/PhsnsHWzJ7MX9sbFclit jFGdomuz74p3pc2YERCZ0HjggdTUYXpsmyvZ2PuOyakOrqGH3QHRrr2lCkIIFaBHQOi9 K00Yr313dmBkVlVuhlGzCzEbOz3Ju0NZoHMpj0JAZbO7luKukQFEu38+r6QOQG1lPBoZ 9bAg== X-Gm-Message-State: AFqh2kpqxB5+QBv4+qhk8daeP/VcZ1171JiFkzd/DSbvIT2jGw5VrXns KalmvGKYGBIHCmh5R8AEVBrsZrV1yb4= X-Google-Smtp-Source: AMrXdXsh1v2U0Hd8bzlVtgqxlve7IZ7l9btMb6fS1tJST5f3r22884m9xzTqoDEpI9YWwLcGbfAA6g== X-Received: by 2002:a5d:58e6:0:b0:26f:aaff:e98c with SMTP id f6-20020a5d58e6000000b0026faaffe98cmr1842742wrd.27.1671722065781; Thu, 22 Dec 2022 07:14:25 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e13-20020a5d500d000000b0024194bba380sm787405wrt.22.2022.12.22.07.14.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:25 -0800 (PST) Message-Id: In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:13 +0000 Subject: [PATCH v4 07/11] bundle-uri: serve bundle.* keys from config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Implement the "bundle-uri" protocol v2 capability by populating the key=value packet lines from the local Git config. The list of bundles is provided from the keys beginning with "bundle.". In the future, we may want to filter this list to be more specific to the exact known keys that the server intends to share, but for flexibility at the moment we will assume that the config values are well-formed. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 +++++++++++++++- t/lib-bundle-uri-protocol.sh | 32 ++++++++++++++++++++++++++++++++ 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 28d8966005e..26ff4b062d7 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -581,6 +581,16 @@ cached: return advertise_bundle_uri; } +static int config_to_packet_line(const char *key, const char *value, void *data) +{ + struct packet_reader *writer = data; + + if (!strncmp(key, "bundle.", 7)) + packet_write_fmt(writer->fd, "%s=%s", key, value); + + return 0; +} + int bundle_uri_command(struct repository *r, struct packet_reader *request) { @@ -592,7 +602,11 @@ int bundle_uri_command(struct repository *r, if (request->status != PACKET_READ_FLUSH) die(_("bundle-uri: expected flush after arguments")); - /* TODO: Implement the communication */ + /* + * Read all "bundle.*" config lines to the client as key=value + * packet lines. + */ + git_config(config_to_packet_line, &writer); packet_writer_flush(&writer); diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 5620e230387..3022ea4a95b 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -136,6 +136,8 @@ test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol [bundle] version = 1 mode = all + [bundle "only"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED EOF test-tool bundle-uri \ @@ -157,6 +159,36 @@ test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol [bundle] version = 1 mode = all + [bundle "only"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED + EOF + + test-tool bundle-uri \ + ls-remote \ + "$BUNDLE_URI_REPO_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "test bundle-uri with $BUNDLE_URI_PROTOCOL:// using protocol v2 with list" ' + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle1.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl" && + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle2.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl" && + test_config -C "$BUNDLE_URI_PARENT" \ + bundle.bundle3.uri "$BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + [bundle "bundle1"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-1.bdl + [bundle "bundle2"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-2.bdl + [bundle "bundle3"] + uri = $BUNDLE_URI_BUNDLE_URI_ESCAPED-3.bdl EOF test-tool bundle-uri \ From patchwork Thu Dec 22 15:14:14 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13080135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7A94C4332F for ; Thu, 22 Dec 2022 15:15:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235501AbiLVPPC (ORCPT ); Thu, 22 Dec 2022 10:15:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59684 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229665AbiLVPO3 (ORCPT ); Thu, 22 Dec 2022 10:14:29 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 11F9B233AB for ; Thu, 22 Dec 2022 07:14:28 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id bx10so1996359wrb.0 for ; Thu, 22 Dec 2022 07:14:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=0CJhC9zf/tPjJhvDDmpb9psUAlq9xPX1O0pWhY3soLg=; b=PHQKjY6bv7//fGYmweVktbm+BJMKMdgw6Un7VFMl/NygMb9EeJnKEca70KzagaqIkc ZZgjg6hAnO7Vx8+Z8lQj6cCoVX3VrPKXgHsZmhRGlMMrXphF18o1KWTSt6+l08DJ9X9w YxO5SOkeFqP+zGRlbBL8katzDNlOjNY/f9gr1X1HeTlECEiyrJQ3YJ1Jka8dJVDK+J28 IdDrdvfUcvR6B0oYfp2IOChuGT/yrXuxJLEMMNOomq1GVCMscziTsT6wxKKcVbn6zbby mae/I6MFK/s0zoiZL+pOkzWXb4+ruyb91bQqHngVuhbobD+Y8aFk1XfGK/vKQhunsurR neNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0CJhC9zf/tPjJhvDDmpb9psUAlq9xPX1O0pWhY3soLg=; b=vX73A1ejCLYbaVD7veYZtiixCMzcSdJJ0NSozPSmwtBQWAxFc7tLg7V2ix2Bx56Aa3 iirdcxgiT4XvioMn4BBLiRBNJ4pjgYDThG7znEypXP66+9XMLcoBPdqyje1oDjg3N+Mc 63lZlzk6Oz1zrrGaxYMV8DQpF9yum2ll+KvP7haeoV5TfwCur29FPGVHTxH8guw9fEY0 FskWOPUi5Lnrvy3DuIUGBeL64WmZVp870p9Me1hxfACyTH9vcma2D9MzBa3+jSHxKAZi u1pWolr+gdk2X3Y8pEXre430GYQWoXtblkHABX533q0vGRNw6i9Q2XLzDGftyPfaSTJ6 87dg== X-Gm-Message-State: AFqh2kqkcpwTcsI0y/Z3AWTmYLWY/WJyY39mE+0jet4QWKDy6NJda8RU SXcXPeZ4Y4XG3IrbQ5VZiZsT70jwrC4= X-Google-Smtp-Source: AMrXdXvxMmeUxfebtrQBOILgkmdijxzUwnqxPiOp84K5YVP7IehiAbqtWIwe02LOY9TVhbfOZkFaMA== X-Received: by 2002:adf:f3d0:0:b0:242:6f93:d558 with SMTP id g16-20020adff3d0000000b002426f93d558mr3887831wrp.67.1671722066502; Thu, 22 Dec 2022 07:14:26 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r13-20020a5d52cd000000b002425be3c9e2sm804330wrv.60.2022.12.22.07.14.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:26 -0800 (PST) Message-Id: In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:14 +0000 Subject: [PATCH v4 08/11] strbuf: introduce strbuf_strip_file_from_path() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The strbuf_parent_directory() method was added as a static method in contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended config and starts maintenance, 2021-12-03) and then removed in 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now there is a need for a similar method in the bundle URI feature. Re-add the method, this time in strbuf.c, but with a new name: strbuf_strip_file_from_path(). The method requirements are slightly modified to allow a trailing slash, in which case nothing is done, which makes the name change valuable. Signed-off-by: Derrick Stolee --- strbuf.c | 6 ++++++ strbuf.h | 11 +++++++++++ 2 files changed, 17 insertions(+) diff --git a/strbuf.c b/strbuf.c index 0890b1405c5..c383f41a3c5 100644 --- a/strbuf.c +++ b/strbuf.c @@ -1200,3 +1200,9 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path, free(path2); return res; } + +void strbuf_strip_file_from_path(struct strbuf *sb) +{ + char *path_sep = find_last_dir_sep(sb->buf); + strbuf_setlen(sb, path_sep ? path_sep - sb->buf + 1 : 0); +} diff --git a/strbuf.h b/strbuf.h index 76965a17d44..f6dbb9681ee 100644 --- a/strbuf.h +++ b/strbuf.h @@ -664,6 +664,17 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer, int strbuf_edit_interactively(struct strbuf *buffer, const char *path, const char *const *env); +/* + * Remove the filename from the provided path string. If the path + * contains a trailing separator, then the path is considered a directory + * and nothing is modified. + * + * Examples: + * - "/path/to/file" -> "/path/to/" + * - "/path/to/dir/" -> "/path/to/dir/" + */ +void strbuf_strip_file_from_path(struct strbuf *sb); + void strbuf_add_lines(struct strbuf *sb, const char *prefix, const char *buf, From patchwork Thu Dec 22 15:14:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13080136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 62A16C4167B for ; Thu, 22 Dec 2022 15:15:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235480AbiLVPPF (ORCPT ); Thu, 22 Dec 2022 10:15:05 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59528 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229670AbiLVPOa (ORCPT ); Thu, 22 Dec 2022 10:14:30 -0500 Received: from mail-wm1-x329.google.com (mail-wm1-x329.google.com [IPv6:2a00:1450:4864:20::329]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CFDCF6324 for ; Thu, 22 Dec 2022 07:14:28 -0800 (PST) Received: by mail-wm1-x329.google.com with SMTP id z8-20020a05600c220800b003d33b0bda11so4209531wml.0 for ; Thu, 22 Dec 2022 07:14:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=HKM5SMCqgoswVsdSQ/vi3+puAeU0kvN+GIO+4TQjQfw=; b=fBcBfSVA0uFTlfuYb6Bn45xKP51C9Bstfjq0DKchqI1t0M+pEOwk8CD6eV9u+vK/bp pecg78ZgAhFhK/GbHxaio2twMmd2t/Ac/OHOiMK1jZjfIxslU1mXcDdDxnf1BIm4SNlE NzlIdLtAsWqugqvNSORca3doNsHUpOiMdYZz23sR3DS4rqSEHSy2p7fERpqnc5ycrOpv A4VnFxl4MjhmHONwHNELzBqXVFKxV5vID197wEH7GE1ssoyAVpK94MX3RtnijnAN47F0 wxQ5T5A2iDBD6gKtvvBmBaYOBv9HSmwq1s3Ft33YClGJ7Vs7Jtl8RSM4kMMP35kLRuBF T2iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HKM5SMCqgoswVsdSQ/vi3+puAeU0kvN+GIO+4TQjQfw=; b=cSzmj7OhcWXNyM8ScJykLWpg4NPOS/tYjO2703NbO1Y9wKu+Q0J54gTYNpIupQYMxK 0c4KUFHmZWyxBfYnqLbN+Lq+fRt9RRGXRXQc92zaGKV7KkVGX19q3ozCV3oeV6X+XXnD G8AMBTMZCkKAMXqbm+HDtUmIUX3clk8ZpdC5OvwLN3d8PuhF2U+ewUszFc17ycUyeEQi 38NxtMHJiVf2XpuzhpY80S2r65Y6Rl66kXyfWbfMP/02uDqt7F1e1K1izIHKOFdZGB9i DBnOUvAWdqueM+W8W5oDubEUJ/mY/56MiuI2Luadg2qFQj6fR0/CmIe5JX970pGKQedG byUg== X-Gm-Message-State: AFqh2krzzFofpttcyw9i6cLswoHJxAGoA1YTP4nanZ68R59mMR+33GPz 6e1DTVZUcoGKeTdYLNUWJ3WWcfnejxA= X-Google-Smtp-Source: AMrXdXsv1jCAIA4JjIzU8vNDZpSdTF/amOjx1ZaPLAlJ5CL9kC+MHdw3j38vPWSuk21inW3UnmuzwQ== X-Received: by 2002:a05:600c:3b90:b0:3d1:f0f1:ceb4 with SMTP id n16-20020a05600c3b9000b003d1f0f1ceb4mr4629875wms.19.1671722067194; Thu, 22 Dec 2022 07:14:27 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l1-20020a05600c4f0100b003cf5ec79bf9sm1412228wmq.40.2022.12.22.07.14.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:26 -0800 (PST) Message-Id: In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:15 +0000 Subject: [PATCH v4 09/11] bundle-uri: allow relative URLs in bundle lists Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Bundle providers may want to distribute that data across multiple CDNs. This might require a change in the base URI, all the way to the domain name. If all bundles require an absolute URI in their 'uri' value, then every push to a CDN would require altering the table of contents to match the expected domain and exact location within it. Allow a bundle list to specify a relative URI for the bundles. This URI is based on where the client received the bundle list. For a list provided in the 'bundle-uri' protocol v2 command, the Git remote URI is the base URI. Otherwise, the bundle list was provided from an HTTP URI not using the Git protocol, and that URI is the base URI. This allows easier distribution of bundle data. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 +++++++- bundle-uri.h | 14 +++++++ t/helper/test-bundle-uri.c | 2 + t/t5750-bundle-uri-parse.sh | 82 +++++++++++++++++++++++++++++++++++++ transport.c | 3 ++ 5 files changed, 116 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 26ff4b062d7..69929d363cc 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -7,6 +7,7 @@ #include "hashmap.h" #include "pkt-line.h" #include "config.h" +#include "remote.h" static int compare_bundles(const void *hashmap_cmp_fn_data, const struct hashmap_entry *he1, @@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list) for_all_bundles_in_list(list, clear_remote_bundle_info, NULL); hashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent); + free(list->baseURI); } int for_all_bundles_in_list(struct bundle_list *list, @@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value, if (!strcmp(subkey, "uri")) { if (bundle->uri) return -1; - bundle->uri = xstrdup(value); + bundle->uri = relative_url(list->baseURI, value, NULL); return 0; } @@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri, .error_action = CONFIG_ERROR_ERROR, }; + if (!list->baseURI) { + struct strbuf baseURI = STRBUF_INIT; + strbuf_addstr(&baseURI, uri); + + /* + * If the URI does not end with a trailing slash, then + * remove the filename portion of the path. This is + * important for relative URIs. + */ + strbuf_strip_file_from_path(&baseURI); + list->baseURI = strbuf_detach(&baseURI, NULL); + } result = git_config_from_file_with_options(config_to_bundle_list, filename, list, &opts); diff --git a/bundle-uri.h b/bundle-uri.h index 357111ecce8..c505444bc75 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -61,6 +61,20 @@ struct bundle_list { int version; enum bundle_list_mode mode; struct hashmap bundles; + + /** + * The baseURI of a bundle_list is the URI that provided the list. + * + * In the case of the 'bundle-uri' protocol v2 command, the base + * URI is the URI of the Git remote. + * + * Otherwise, the bundle list was downloaded over HTTP from some + * known URI. 'baseURI' is set to that value. + * + * The baseURI is used as the base for any relative URIs + * advertised by the bundle list at that location. + */ + char *baseURI; }; void init_bundle_list(struct bundle_list *list); diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index f8159187014..5df5bc3b89e 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo init_bundle_list(&list); + list.baseURI = xstrdup(""); + switch (mode) { case KEY_VALUE_PAIRS: if (argc != 1) diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh index c2fe3f9c5a5..7b4f930e532 100755 --- a/t/t5750-bundle-uri-parse.sh +++ b/t/t5750-bundle-uri-parse.sh @@ -30,6 +30,58 @@ test_expect_success 'bundle_uri_parse_line() just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'bundle_uri_parse_line(): relative URIs' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-key-values in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + +test_expect_success 'bundle_uri_parse_line(): relative URIs and parent paths' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=../../bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /../bundle.bdl + EOF + + # TODO: We would prefer if parsing a bundle list would not cause + # a die() and instead would give a warning and allow the rest of + # a Git command to continue. This test_must_fail is necessary for + # now until the interface for relative_url() allows for reporting + # an error instead of die()ing. + test_must_fail test-tool bundle-uri parse-key-values in >actual 2>err && + grep "fatal: cannot strip one component off url" err +' + test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' ' cat >in <<-\EOF && =bogus-value @@ -136,6 +188,36 @@ test_expect_success 'parse config format: just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'parse config format: relative URIs' ' + cat >in <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = bundle.bdl + [bundle "two"] + uri = ../bundle.bdl + [bundle "three"] + uri = sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-config in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + test_expect_success 'parse config format edge cases: empty key or value' ' cat >in1 <<-\EOF && = bogus-value diff --git a/transport.c b/transport.c index 0f35114a13e..241f8a6ba2d 100644 --- a/transport.c +++ b/transport.c @@ -1538,6 +1538,9 @@ int transport_get_remote_bundle_uri(struct transport *transport) if (git_config_get_bool("transfer.bundleuri", &value) || !value) return 0; + if (!transport->bundles->baseURI) + transport->bundles->baseURI = xstrdup(transport->url); + if (!vtable->get_bundle_uri) return error(_("bundle-uri operation not supported by protocol")); From patchwork Thu Dec 22 15:14:16 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13080137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8319CC4332F for ; Thu, 22 Dec 2022 15:15:08 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229658AbiLVPPG (ORCPT ); Thu, 22 Dec 2022 10:15:06 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59582 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235321AbiLVPOa (ORCPT ); Thu, 22 Dec 2022 10:14:30 -0500 Received: from mail-wm1-x32b.google.com (mail-wm1-x32b.google.com [IPv6:2a00:1450:4864:20::32b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A523426ACE for ; Thu, 22 Dec 2022 07:14:29 -0800 (PST) Received: by mail-wm1-x32b.google.com with SMTP id p1-20020a05600c1d8100b003d8c9b191e0so1492639wms.4 for ; Thu, 22 Dec 2022 07:14:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=Nl+CWSiRwwdslFulwFO0CZwAJW3T7ygqWyaq3xDlAoo=; b=DQDKJ+GDyUnE5weLe1PnVc57bDpbsKzgimgg3M1pdfcXtwBjFLHVqltQYWRsvRLXtG GA/7h6cpjKDEJnn6oCSZG3qUMhYoTmKgz1x5lJtE9kuadyRfSnwGRqVsG6ry9nOrLPsG HM5X1v/CGojGjzN+OqyB/wqEiJcYcdsRdlVUoUnTzpjsqsq6A4iTP98/vbL2/VzJ3HPQ VHcg9k6kqNpC0uKvD9Lu6HKuJg8UNFa2bjM5a89/yHu2IuqMbMW2W4GQwnvJhQV/LJCL 7sdtQkxn5UCs1MYOa7SPOov0kMGI3DWA57cwyY6wR5kS+1JBCOVeAkWRvkvECpqg6WAY 4Gbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nl+CWSiRwwdslFulwFO0CZwAJW3T7ygqWyaq3xDlAoo=; b=fQFk4Kbs3kxPaTHxWyjMGUHxma6BSBnbCUHzIpMwKpNAZ+Ec+fepg0kauQgjDANxUN MqA/0FUCVG6Y7DgqZ5Y15dZR3MYq6zdH7n/YWhA2371Et9hyGT63UMV9+W0WESfjKlOI VaKpnkgaQIMlnDH4VAbaY8X73hoYvK3fC0CrADrbXJ0sIe025ve0w7Ol2Od3f1ZgEGrP QhNq9bhjEnylbDpFH5Y0Cfue5UlkDtgL9MRRYg2DDkw5NDY+axWjHYxTq7XdOOBrrGV1 GbFwxeL/ZyxHXrlKMLi/FqKkvvQU6MX6kRjLhHNGad6SDDwrHbXhXYcvZw9Q8f8xboyL y1Bw== X-Gm-Message-State: AFqh2kpBDk7L+oN/4L1LbufJC5kQvX0sg3ni8tzKjGWdoG6g9e4473gF DwvCQnPaOgVrncPGOwg5VwpbYxcVcbM= X-Google-Smtp-Source: AMrXdXuN1h1KImxrgub4BjEuaKyozzekQDyS8E4Rrgr3IGSk0/cTg20RgEupKZ2jicwh+VrogNs4lA== X-Received: by 2002:a05:600c:3b21:b0:3d0:d177:cadf with SMTP id m33-20020a05600c3b2100b003d0d177cadfmr4447907wms.32.1671722067999; Thu, 22 Dec 2022 07:14:27 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id t8-20020a05600c198800b003d220ef3232sm1439844wmq.34.2022.12.22.07.14.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:27 -0800 (PST) Message-Id: <72ca6f4254fd29edaba9f201324e22892b4dfb2d.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:16 +0000 Subject: [PATCH v4 10/11] bundle-uri: download bundles from an advertised list Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The logic in fetch_bundle_uri() is useful for the --bundle-uri option of 'git clone', but is not helpful when the clone operation discovers a list of URIs from the bundle-uri protocol v2 command. To actually download and unbundle the advertised bundles, we need a different mechanism. Create the new fetch_bundle_list() method which is very similar to fetch_bundle_uri() except that it relies on download_bundle_list() instead of fetch_bundle_uri_internal(). The download_bundle_list() method will recursively call fetch_bundle_uri_internal() if any of the advertised URIs serve a bundle list instead of a bundle. This will also follow the bundle.list.mode setting from the input list: "any" will download only one such URI while "all" will download data from all of the URIs. In an identical way to fetch_bundle_uri(), the bundles are unbundled after all of the bundle lists have been expanded and all necessary URIs. Signed-off-by: Derrick Stolee --- bundle-uri.c | 21 +++++++++++++++++++++ bundle-uri.h | 14 ++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/bundle-uri.c b/bundle-uri.c index 69929d363cc..36268dda172 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -577,6 +577,27 @@ cleanup: return result; } +int fetch_bundle_list(struct repository *r, struct bundle_list *list) +{ + int result; + struct bundle_list global_list; + + init_bundle_list(&global_list); + + /* If a bundle is added to this global list, then it is required. */ + global_list.mode = BUNDLE_MODE_ALL; + + if ((result = download_bundle_list(r, list, &global_list, 0))) + goto cleanup; + + result = unbundle_all_bundles(r, &global_list); + +cleanup: + for_all_bundles_in_list(&global_list, unlink_bundle, NULL); + clear_bundle_list(&global_list); + return result; +} + /** * API for serve.c. */ diff --git a/bundle-uri.h b/bundle-uri.h index c505444bc75..d5e89f1671c 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -107,6 +107,20 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * Given a bundle list that was already advertised (likely by the + * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the + * bundles according to the bundle strategy of that list. + * + * It is expected that the given 'list' is initialized, including its + * 'baseURI' value. + * + * Returns non-zero if there was an error trying to download the list + * or any of its advertised bundles. + */ +int fetch_bundle_list(struct repository *r, + struct bundle_list *list); + /** * API for serve.c. */ From patchwork Thu Dec 22 15:14:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13080138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 857CAC4332F for ; Thu, 22 Dec 2022 15:15:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235429AbiLVPPJ (ORCPT ); Thu, 22 Dec 2022 10:15:09 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:59762 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230407AbiLVPOc (ORCPT ); Thu, 22 Dec 2022 10:14:32 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 61C4838A7 for ; Thu, 22 Dec 2022 07:14:30 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id m14so1954247wrh.7 for ; Thu, 22 Dec 2022 07:14:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=St9c68m1U2pVma0sTvdcQQataF+S+J7SYqQBoIWNBDw=; b=dlLYd+dwQKZnLvrUPcbTAXeGXl6oyMs64jzTrfllSbJAts7Zz6UjPTixy4piBFAgvB 0cps7OfMUgcV2w34ntOF2z/aPlqvnntxzHPsHWUD/v8fOgjOBQ1Qm2eqh6CVfUmTwUL4 jR3SbtnMR4ZsSlCwQKIDuAJCSJE6rz9fOJFvCMv+xSAj2gPYcK0rCWKYdPUnguNc/9Ag +C0yRomDqeFw/WKIwI55nXUkFjFZp+eOsxbsu2QxU5k1OY0Rchhvq18PNKsJRDUxvaGH 2ACypQhX8YiMd4Ej0fKBLzF67BBtZko7+A5j/Paqbrh00J8S+g0VkY9NwH/FWocTuIMx opbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=St9c68m1U2pVma0sTvdcQQataF+S+J7SYqQBoIWNBDw=; b=nFEQ9PeTVuH59thjq+BTdIwEB/IrGXclqPkA+oejEMFqyT/Y6tRNzXHh5c73lP1aCF WbxGy1Wv39olydYRhtxYUxM5qxpORCv7ieQMWLdfKLuPz/Nfm51tckqdSkL5a8bV9EGG Qp2iM4uHTiN2srEdGfd+JJMAXz7zzlmkTtu0g/w0UvONuXGDPXJLx1lveD1dVB280+/X w4bpJQgcm/UMApEWjvhVCyQDdE4QtEQJcG2wGodaD/BRstUa03jrhzL3LCWo0VM5D/fQ PTo18lI/EsFsImmfXvG342mwA3YIZxjFzGpK6sAKO3oacWJFJygJ53vzFk2o//jpd4iq FXXg== X-Gm-Message-State: AFqh2koB00qHnbxxY5Ix7IlJHCZtGv4arZERzU5O5d8b+Lq7EUPqDbZc AAvUIPZEJzVhD+Ls/Fzs256yq6rX7wE= X-Google-Smtp-Source: AMrXdXtJ4JxuUKqzgy1W6G8EXzROv0/F45l1OLr167D5C+3Il4TnD8WmdohrBYRLdAqr1t+i1qzcww== X-Received: by 2002:a05:6000:1e0a:b0:242:67f2:9af7 with SMTP id bj10-20020a0560001e0a00b0024267f29af7mr4080495wrb.69.1671722068755; Thu, 22 Dec 2022 07:14:28 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x2-20020adfec02000000b0025e86026866sm1070035wrn.0.2022.12.22.07.14.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 22 Dec 2022 07:14:28 -0800 (PST) Message-Id: <46ab2b05b1559bf289d28ab3353ea9cecf84875d.1671722058.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 22 Dec 2022 15:14:17 +0000 Subject: [PATCH v4 11/11] clone: unbundle the advertised bundles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Victoria Dye , Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A previous change introduced the transport methods to acquire a bundle list from the 'bundle-uri' protocol v2 command, when advertised _and_ when the client has chosen to enable the feature. Teach Git to download and unbundle the data advertised by those bundles during 'git clone'. This takes place between the ref advertisement and the object data download, and stateful connections will linger while the client downloads bundles. In the future, we should consider closing the remote connection during this process. Also, since the --bundle-uri option exists, we do not want to mix the advertised bundles with the user-specified bundles. Signed-off-by: Derrick Stolee --- builtin/clone.c | 25 ++++++++++++--- t/lib-bundle-uri-protocol.sh | 21 +++++++++++-- t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++ 3 files changed, 98 insertions(+), 7 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 39364c25b15..527839662b0 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1266,11 +1266,26 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); - /* - * Populate transport->got_remote_bundle_uri and - * transport->bundle_uri. We might get nothing. - */ - transport_get_remote_bundle_uri(transport); + if (!bundle_uri) { + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + + if (transport->bundles && + hashmap_get_size(&transport->bundles->bundles)) { + /* At this point, we need the_repository to match the cloned repo. */ + if (repo_init(the_repository, git_dir, work_tree)) + warning(_("failed to initialize the repo, skipping bundle URI")); + else if (fetch_bundle_list(the_repository, + transport->bundles)) + warning(_("failed to fetch advertised bundles")); + } else { + clear_bundle_list(transport->bundles); + FREE_AND_NULL(transport->bundles); + } + } if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/t/lib-bundle-uri-protocol.sh b/t/lib-bundle-uri-protocol.sh index 3022ea4a95b..a4a1af8d029 100644 --- a/t/lib-bundle-uri-protocol.sh +++ b/t/lib-bundle-uri-protocol.sh @@ -85,7 +85,7 @@ test_expect_success "connect with $BUNDLE_URI_PROTOCOL:// using protocol v2: hav ' test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: request bundle-uris" ' - test_when_finished "rm -rf log cloned cloned2" && + test_when_finished "rm -rf log* cloned*" && GIT_TRACE_PACKET="$PWD/log" \ git \ @@ -117,7 +117,24 @@ test_expect_success "clone with $BUNDLE_URI_PROTOCOL:// using protocol v2: reque grep "< bundle-uri" log && # Client issued bundle-uri command - grep "> command=bundle-uri" log + grep "> command=bundle-uri" log && + + GIT_TRACE_PACKET="$PWD/log3" \ + git \ + -c transfer.bundleURI=true \ + -c protocol.version=2 \ + clone --bundle-uri="$BUNDLE_URI_BUNDLE_URI" \ + "$BUNDLE_URI_REPO_URI" cloned3 \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log3 && + + # Server advertised bundle-uri capability + grep "< bundle-uri" log3 && + + # Client did not issue bundle-uri command (--bundle-uri override) + ! grep "> command=bundle-uri" log3 ' # The remaining tests will all assume transfer.bundleURI=true diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 45f0803ed4d..00d4fae5136 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' ' git clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo ' +test_expect_success 'auto-discover bundle URI from HTTP clone' ' + test_when_finished rm -rf trace.txt repo2 "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle" --all && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.mode all && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 \ + -c transfer.bundleURI=true clone \ + $HTTPD_URL/smart/repo2.git repo2 && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt +' + +test_expect_success 'auto-discover multiple bundles from HTTP clone' ' + test_when_finished rm -rf trace.txt repo3 "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + test_commit -C src new && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/new.bundle" HEAD~1..HEAD && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.mode all && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.new.uri "$HTTPD_URL/new.bundle" && + + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 \ + -c transfer.bundleURI=true clone \ + $HTTPD_URL/smart/repo3.git repo3 && + + # We should fetch _both_ bundles + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/new.bundle"\] + EOF + grep -f pattern trace.txt +' + # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled.