From patchwork Thu Aug 5 15:07:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12421391 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5B55C432BE for ; Thu, 5 Aug 2021 15:07:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8A5FB6104F for ; Thu, 5 Aug 2021 15:07:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241797AbhHEPHx (ORCPT ); Thu, 5 Aug 2021 11:07:53 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51032 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241788AbhHEPHx (ORCPT ); Thu, 5 Aug 2021 11:07:53 -0400 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BAD6C061765 for ; Thu, 5 Aug 2021 08:07:38 -0700 (PDT) Received: by mail-wr1-x436.google.com with SMTP id h14so6982311wrx.10 for ; Thu, 05 Aug 2021 08:07:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=op9PMwrZGHenow0iKHSaxGmGiBEb30e6FhbfZYWo6S8=; b=MBCEQs4BSt4/krPkX/3Rf5jwOS5YQyT1CMIXtBzivoRY+uTnXEMEqtBQrHEWvGHb8t oTYHkjGr2XlhYOpEqEfw4NLyvVdPRkc0X2SPneL1HZVEA43XTOWtVd5Ch3u2VnS5jE0K qOCCidiINECn79c0+WtE8RVF4H99jyeWiJFxhVfUYXDPFwGf6lLw/hp7kgkC/gRrGiaG Husk6RV1Xmd+ZZXT6Jvr3rd63U8wlwnhVzOJRIcj/2c7IFO6v+51aaNOddIwXiKWjbuS lBF7ZmzkMyBW4Q8S6M0caCiJ3aeXRCK/aSOHOCwflsgHPhY8mZ+O/+TCbhSeSJohA6UL uUGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=op9PMwrZGHenow0iKHSaxGmGiBEb30e6FhbfZYWo6S8=; b=gt92rqenDu+5HlmKO6O5cA8YHVTUFa0mqzuEBaKq4+wr6VS7j8GmwluDLOA/IlbzIh ljT44acyFZXY3nsx1tM3itzsVOI7UMEUQLpz1ik+JyiTpuWxGnh0jzcPyGkJjJMJ37XP xiOfWd0s8xDs6RO2JqTq1nR2HyufrvD6oybhstYdpA79XVH7xoyP1K9RTCuS6MwQKkYj H4R8UJwp+swa0cqYMUlkTyeif1osu02syNw3m/cDpJBHnTHrjxGEldfqg8nbafbgCiJD V/PmIi8cw7Tqb9HxxO13ZUdbzphAEM49LD8dTzD5wCAi0AU+mONO0HPo7+lgE9qxyiD7 lF7g== X-Gm-Message-State: AOAM533DHQbvhBeHkyT24KVyCd1OeX7sNMQhlV7eQkMBJyOe4GDcCCsR teDqkR4TUnTXVJyGrT9XCBZInUYMdzzxrA== X-Google-Smtp-Source: ABdhPJzTUdtSwn5iT8tbX4RSWuEWBhI6VUgKeX6EAo5ZQx2g+pSXgOssjoByTwtn9eleTsf6C28KJA== X-Received: by 2002:adf:dfcb:: with SMTP id q11mr5895565wrn.16.1628176056716; Thu, 05 Aug 2021 08:07:36 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id w4sm6856340wrm.24.2021.08.05.08.07.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Aug 2021 08:07:36 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Christian Couder , Albert Cui , Jonathan Tan , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH 01/13] serve: add command to advertise bundle URIs Date: Thu, 5 Aug 2021 17:07:17 +0200 Message-Id: X-Mailer: git-send-email 2.33.0.rc0.646.g585563e77f In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When the uploadpack.bundleURI config is set to a URI (or URIs, if set >1 times), advertise a "bundle-uri" command, then when the client requests "bundle-uri" emit those URIs back at them. The client CAN then request those URIs out of bounds, and after they've done (after either disconnecting & coming back, or leaving us hanging), proceed with the rest of request flow. I.e. issuing a "ls-refs" followed by a "fetch". The client MAY then send us "have" lines with the tips they've unpacked from their newly acquired bundle(s). This commit doesn't implement any of that required client behavior, only the trivial server behavior of spewing a list of URLs at the client on request. There is already a uploadpack.blobPackfileUri setting for the server, so why is this needed? The Documentation/technical/bundle-uri.txt added in a preceding commit discusses that in more detail, but in summary: 1. There is no "real" support for in git.git. The uploadpack.blobPackfileUri setting allows carving out a list of blobs (actually any OIDs), but as alluded to in bfc2a36ff2a (Doc: clarify contents of packfile sent as URI, 2021-01-20) the only "real" implementation is JGit based. 2. The uploadpack.blobPackfileUri is a MUST where this is a "CAN". I.e. once a client says they support packfile-uri of given list of protocols the server will send them a PACK response assuming they've downloaded the URI they client was sent, if the client doesn't do that they don't have a valid repository. Pointing at a bundle and having the client send us "have" lines (or not, maybe they couldn't fetch it, or decided they didn't want to) is more flexible, and can gracefully recover e.g. if the CDN isn't reachable (maybe you do support "https", but the CDN provider is down, or blocked your whole country). 3. Because of the disconnect in #2 "dumb" servers can seed pre-clients, e.g. we might point to a repo.bundle whose exact state we aren't sure of (a cronjob updates it, sometimes). The client will discover its contents, and give us the "have" lines, the "packfile-uri" method effectively requires the server to have those exact "have" lines (or rather, it will produce a similar PACK using give-or-take the same exclusions). 4. This provides an easy way to the long sought after "resumable clones". I.e. since we can assume that it's in the server's interest to keep their bundle(s) as up-to-date as possible, most or all of the history we need to fetch will be in the bundle. If we fail midway through the "clone" we can offload the problem of resuming to wget/curl/rsync/whatever, instead of (as has been suggested, but not implemented for the "normal" dialog) "repairing" a partial PACK response or something. There was a suggestion of implementing a similar feature long ago[1] by Jeff King. The main difference between it and this approach is that we've since gained protocol v2, so we can add this as an optional path in the dialog between client and server. The 2011 implementation hooked into the transport mechanism to try to clone from a bundle directly. See also [2] and [3] for some later mentions of that approach. See also [4] for the series that implemented uploadpack.blobPackfileUri, and [5] for a series on top that did the .gitmodules check in that context. See [6] for the "ls-refs unborn" feature which modified code in similar areas of the request flow. 1. https://lore.kernel.org/git/20111110074330.GA27925@sigill.intra.peff.net/ 2. https://lore.kernel.org/git/20190514092900.GA11679@sigill.intra.peff.net/ 3. https://lore.kernel.org/git/YFJWz5yIGng+a16k@coredump.intra.peff.net/ 4. https://lore.kernel.org/git/cover.1591821067.git.jonathantanmy@google.com/ Merged as 34e849b05a4 (Merge branch 'jt/cdn-offload', 2020-06-25) 5. https://lore.kernel.org/git/cover.1614021092.git.jonathantanmy@google.com/ Merged as 6ee353d42f3 (Merge branch 'jt/transfer-fsck-across-packs', 2021-03-01) 6. 69571dfe219 (Merge branch 'jt/clone-unborn-head', 2021-02-17) Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/technical/protocol-v2.txt | 140 ++++++++++++++++++++++++ Makefile | 1 + bundle-uri.c | 65 +++++++++++ bundle-uri.h | 14 +++ serve.c | 6 + t/t5701-git-serve.sh | 124 ++++++++++++++++++++- 6 files changed, 349 insertions(+), 1 deletion(-) create mode 100644 bundle-uri.c create mode 100644 bundle-uri.h diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 213538f1d0..d10d5e9ef6 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -556,3 +556,143 @@ and associated requested information, each separated by a single space. attr = "size" obj-info = obj-id SP obj-size + +bundle-uri +~~~~~~~~~~ + +If the 'bundle-uri' capability is advertised, the server supports the +`bundle-uri' command. + +The capability is currently advertised with no value (i.e. not +"bundle-uri=somevalue"), a value may be added in the future for +supporting command-wide extensions. Clients MUST ignore any unknown +capability values and proceed with the 'bundle-uri` dialog they +support. + +The 'bundle-uri' command is intended to be issued before `fetch` to +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and +inform the subsequent `fetch` command. + +The client CAN issue `bundle-uri` before or after any other valid +command. It's expected that it'll be issued after an `ls-refs` and +before `fetch`. + +DISCUSSION of bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^^ + +The intent of the feature is optimize for server resource consumption +in the common case by changing the common case of fetching a very +large PACK during linkgit:git-clone[1] into a smaller incremental +fetch. + +It also allows servers to achieve better caching in combination with +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]). + +By having new clones or fetches be a more predictable and common +negotiation against the tips of recently produces *.bundle file(s). +Servers might even pre-generate the results of such negotiations for +the `uploadpack.packObjectsHook` as new pushes come in. + +I.e. the server would anticipate that fresh clones will download a +known bundle, followed by catching up to the current state of the +repository using ref tips found in that bundle (or bundles). + +PROTOCOL for bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^ + +A `bundle-uri` request takes no arguments, and as noted above does not +currently advertise a capability value. Both may be added in the +future. + +When the client issues a `command=bundle-uri` the response is a list +of URIs the server would like the client to fetch out-of-bounds before +proceeding with the `fetch` request in this format: + + output = bundle-uri-line + bundle-uri-line* flush-pkt + + bundle-uri-line = PKT-LINE(bundle-uri) + *(SP bundle-feature-key *(=bundle-feature-val)) + LF + + bundle-uri = A URI such as a https://, ssh:// etc. URI + + bundle-feature-key = Any printable ASCII characters except SP or "=" + bundle-feature-val = Any printable ASCII characters except SP or "=" + +No `bundle-feature-key`=`bundle-feature-value` fields are currently +defined. See the discussion of features below. + +bundle-uri CLIENT AND SERVER EXPECTATIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The advertised bundles MUST contain one or more reference tips for use +by the client. Bundles that are not self-contained MUST use the +standard "-" prefixes in the bundle format to indicate their +prerequisites. I.e. they must be in the standard format "git bundle +create" would create. + +If after an `ls-refs` the client finds that the ref tips it wants can +be retrieved entirety from advertised bundle(s), it MAY +disconnect. The results of such a "clone" or "fetch" should be +indistinguishable from the state attained without using bundle-uri. + +The client MAY also keep the connection open pending download of the +bundle-uris, e.g. should on or more downloads (or their validation) +fail. + +The client MAY provide reference tips found in the bundle(s) to be +downloaded out-of-bounds as `have` lines in the `fetch` request. They +MAY also ignore the bundle(s) entirely (e.g. if they can't be +downloaded) or some combination of the two. + +For the convenience of clients bundles SHOULD be provided in the order +that they must be unpacked in if processed one-at-a-time by a dumber client. + +That usually means a "big bundle" first with most of the history +that's self-contained, optionally followed by incremental updates on +that "big bundle". + +This ordering is a mere convention and not a MUST, e.g. a repository +with N branches with disconnected histories might have N "big +bundles", each with their own self-contained history. A server might +also only provide "incremental updates". + +A client MUST consider the content of the bundles themselves and their +header as the ultimate source of truth. Servers MAY be tolerant of +simpler clients by using the convention outlined above. + +As noted before a client MUST gracefully degrade on errors, whether +that error is because of bad missing/data in the bundle URI(s), or +because that client is too dumb to e.g. understand and fully parse out +bundle headers and their prerequisite relationships. + +bundle-uri PROTOCOL FEATURES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +As noted above no `bundle-feature-key`=`bundle-feature-value` fields +are currently defined. + +They are intended for future per-URI metadata which older clients MUST +ignore and gracefully degrade on. Any fields they do recognize they +CAN also ignore. + +Any backwards-incompatible addition of pre-URI key-value will be +guarded by a new value or values in 'bundle-uri' capability +advertisement itself, and/or by new future `bundle-uri` request +arguments. + +While no per-URI key-value are currently supported currently they're +intended to support future features such as: + + * Add a "hash=" or "size=" advertise the expected hash or + size of the bundle file. + + * Advertise that one or more bundle files are the same (to e.g. have + clients round-robin or otherwise choose one of N possible files). + + * A "tip=" shortcut. A client who'd have the advertised + would know there was no need to download the relevant bundle(s), + they've got that OID already (for multi-tips the client would need + to fetch the bundle, or do e.g. HTTP range requests to get its + header). diff --git a/Makefile b/Makefile index 9573190f1d..877c6c47b6 100644 --- a/Makefile +++ b/Makefile @@ -850,6 +850,7 @@ LIB_OBJS += blob.o LIB_OBJS += bloom.o LIB_OBJS += branch.o LIB_OBJS += bulk-checkin.o +LIB_OBJS += bundle-uri.o LIB_OBJS += bundle.o LIB_OBJS += cache-tree.o LIB_OBJS += cbtree.o diff --git a/bundle-uri.c b/bundle-uri.c new file mode 100644 index 0000000000..2d93e8b003 --- /dev/null +++ b/bundle-uri.c @@ -0,0 +1,65 @@ +#include "cache.h" +#include "bundle-uri.h" +#include "pkt-line.h" +#include "config.h" + +/** + * serve.[ch] API. + */ + +/* + * "uploadpack.bundleURI" is advertised only if there's URIs to serve + * up per the config. + */ +static int advertise_bundle_uri = -1; + +static void send_bundle_uris(struct packet_writer *writer, + struct string_list *uris) +{ + struct string_list_item *item; + for_each_string_list_item(item, uris) { + const char *uri = item->string; + + packet_writer_write(writer, "%s", uri); + } +} + +static struct string_list bundle_uris = STRING_LIST_INIT_DUP; + +static int bundle_uri_startup_config(const char *var, const char *value, + void *data) +{ + if (!strcmp(var, "uploadpack.bundleuri")) { + advertise_bundle_uri = 1; + string_list_append(&bundle_uris, value); + } + return 0; +} + +int bundle_uri_advertise(struct repository *r, struct strbuf *value) +{ + if (advertise_bundle_uri == -1) { + git_config(bundle_uri_startup_config, NULL); + if (advertise_bundle_uri == -1) + advertise_bundle_uri = 0; + } + return advertise_bundle_uri; +} + +int bundle_uri_command(struct repository *r, + struct packet_reader *request) +{ + struct packet_writer writer; + packet_writer_init(&writer, 1); + + while (packet_reader_read(request) == PACKET_READ_NORMAL) + die("bundle-uri: unexpected argument: '%s'", request->line); + if (request->status != PACKET_READ_FLUSH) + die("bundle-uri: expected flush after arguments"); + + send_bundle_uris(&writer, &bundle_uris); + + packet_writer_flush(&writer); + + return 0; +} diff --git a/bundle-uri.h b/bundle-uri.h new file mode 100644 index 0000000000..6a40efeb39 --- /dev/null +++ b/bundle-uri.h @@ -0,0 +1,14 @@ +#ifndef BUNDLE_URI_H +#define BUNDLE_URI_H + +struct repository; +struct packet_reader; +struct packet_writer; + +/** + * serve.[ch] API. + */ +int bundle_uri_advertise(struct repository *r, struct strbuf *value); +int bundle_uri_command(struct repository *r, struct packet_reader *request); + +#endif /* BUNDLE_URI_H */ diff --git a/serve.c b/serve.c index 1817edc7f5..789bf5fc38 100644 --- a/serve.c +++ b/serve.c @@ -8,6 +8,7 @@ #include "protocol-caps.h" #include "serve.h" #include "upload-pack.h" +#include "bundle-uri.h" static int advertise_sid = -1; @@ -104,6 +105,11 @@ static struct protocol_capability capabilities[] = { .advertise = always_advertise, .command = cap_object_info, }, + { + .name = "bundle-uri", + .advertise = bundle_uri_advertise, + .command = bundle_uri_command, + }, }; void protocol_v2_advertise_capabilities(void) diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index 930721f053..21d5314d83 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -12,7 +12,7 @@ test_expect_success 'test capability advertisement' ' wrong_algo sha1:sha256 wrong_algo sha256:sha1 EOF - cat >expect <<-EOF && + cat >expect.base <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) ls-refs=unborn @@ -20,8 +20,11 @@ test_expect_success 'test capability advertisement' ' server-option object-format=$(test_oid algo) object-info + EOF + cat >expect.trailer <<-EOF && 0000 EOF + cat expect.base expect.trailer >expect && GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ --advertise-capabilities >out && @@ -266,4 +269,123 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +# Test the basics of bundle-uri +# +test_expect_success 'test capability advertisement with uploadpack.bundleURI' ' + test_config uploadpack.bundleURI FAKE && + + cat >expect.extra <<-EOF && + bundle-uri + EOF + cat expect.base \ + expect.extra \ + expect.trailer >expect && + + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ + --advertise-capabilities >out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: dies if not enabled' ' + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: invalid command '"'"'bundle-uri'"'"' + EOF + + cat >expect <<-\EOF && + ERR serve: invalid command '"'"'bundle-uri'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + + +test_expect_success 'basics of bundle-uri: enabled with single URI' ' + test_config uploadpack.bundleURI https://cdn.example.com/repo.bdl && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0000 + EOF + + cat >expect <<-EOF && + https://cdn.example.com/repo.bdl + 0000 + EOF + + test-tool serve-v2 --stateless-rpc out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: enabled with single URI' ' + test_config uploadpack.bundleURI https://cdn.example.com/repo.bdl && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0000 + EOF + + cat >expect <<-EOF && + https://cdn.example.com/repo.bdl + 0000 + EOF + + test-tool serve-v2 --stateless-rpc out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: enabled with two URIs' ' + test_config uploadpack.bundleURI https://cdn.example.com/repo.bdl && + test_config uploadpack.bundleURI https://cdn.example.com/recent.bdl --add && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0000 + EOF + + cat >expect <<-EOF && + https://cdn.example.com/repo.bdl + https://cdn.example.com/recent.bdl + 0000 + EOF + + test-tool serve-v2 --stateless-rpc out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: unknown future feature(s)' ' + test_config uploadpack.bundleURI https://cdn.example.com/fake.bdl && + + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + object-format=$(test_oid algo) + 0001 + some-feature + we-do-not + know=about + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: bundle-uri: unexpected argument: '"'"'some-feature'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + test_done From patchwork Thu Aug 5 15:07:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12421393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F075CC4338F for ; Thu, 5 Aug 2021 15:07:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DA03F60EE5 for ; Thu, 5 Aug 2021 15:07:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241811AbhHEPIA (ORCPT ); Thu, 5 Aug 2021 11:08:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51034 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241788AbhHEPHy (ORCPT ); Thu, 5 Aug 2021 11:07:54 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2D0C1C0613D5 for ; Thu, 5 Aug 2021 08:07:39 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id l18so7017063wrv.5 for ; Thu, 05 Aug 2021 08:07:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ymcFBiqCpg2n5b6B373QBftuPLgXHZVVsLtynEOn+iY=; b=LXGhtMgryiK7OJ9DJ/JCj2X+rMAAajnxrw2kRfUV/86i1Zm5Hthaev+hKaYFrACpDU nZKuMst05h21FBUMFly63oUiAAhYr2Zgm9trf1BqC917VL5K6hlfuoSs35W9L+w0qCxs jNfAWLZSO2CF2thebtgkMY918HfKda6OvakbdES51ihQeyg0VDq7zDZmoow8KN5J61vQ bcxJ6XgKms3LqoLpZdTBUtdN/bqqXH9ZF+vP2S02ZxGHQnXdYMYrm0FWswuJonnnHhYh SN+LMFq3OXP+Xhvt9jgTirhtB2nc97q1cOP9B1gx16lA8jMtpmqFirtEp8fo+SbNPgvd 7f5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ymcFBiqCpg2n5b6B373QBftuPLgXHZVVsLtynEOn+iY=; b=da/q2AeJZw2sHpbHKzKexvQ1BxJFrv639UDGKxmuAJGn3MPBERa6fhd7zQC8FskZHB 3x0/5bMnXPRgGLAQK/D+/rENIP3EUWF/SUivvuFi4ZUM1/1yEGSjKFN2SNXTuId1iREV XZBXL5y2xbfRBKJmBS4f134cXceTjFvS2C+fdhtB0Bk1T8lYLg6gNo37E5Jt+yQqz+nY QCjzq9ZU6eMVcdRelXY9oG/V9YlyS3msn0vbYZlacTuVTUqSnBKbUCsCJ8R5QQa6XIPQ XK+MjLgzj2kAT0N/Pd99E60xIa4wfBHqytXp4pMcvU0px6uCKpF/p1fynZIQaco9n1Tq y6lQ== X-Gm-Message-State: AOAM531IrSgxwoEjHy2igW7hbMUx6hPxbhyMXMiC33Mx+lttiwLVSO2S kmcm9wMrP4ACRH35ZplvxDX0N+ZdBBzLJQ== X-Google-Smtp-Source: ABdhPJwrGSUyZ/Iin0IcbGxaaub2UNfUY46B6gwm55Ds6maCcPYGwl/aCQcNoB8A1JncBtErOWGGZg== X-Received: by 2002:adf:f287:: with SMTP id k7mr5705256wro.206.1628176057548; Thu, 05 Aug 2021 08:07:37 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id w4sm6856340wrm.24.2021.08.05.08.07.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Aug 2021 08:07:37 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Christian Couder , Albert Cui , Jonathan Tan , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH 02/13] bundle-uri client: add "bundle-uri" parsing + tests Date: Thu, 5 Aug 2021 17:07:18 +0200 Message-Id: X-Mailer: git-send-email 2.33.0.rc0.646.g585563e77f In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a "test-tool bundle-uri parse" which parses the format defined in the newly specified "bundle-uri" command. As note in the "bundle-uri" section in protocol-v2.txt we haven't specified any key-values yet, just URI lines, but we need to make sure our client doesn't die if this optional data is provided by the server. Let's add a bundle_uri_parse_line() to do that, in subsequent commits the actual client code in {connect,transport}.c will make use of it. Signed-off-by: Ævar Arnfjörð Bjarmason --- Makefile | 1 + bundle-uri.c | 80 ++++++++++++++++++++++++++++++ bundle-uri.h | 16 ++++++ t/helper/test-bundle-uri.c | 80 ++++++++++++++++++++++++++++++ t/helper/test-tool.c | 1 + t/helper/test-tool.h | 1 + t/t5750-bundle-uri-parse.sh | 98 +++++++++++++++++++++++++++++++++++++ 7 files changed, 277 insertions(+) create mode 100644 t/helper/test-bundle-uri.c create mode 100755 t/t5750-bundle-uri-parse.sh diff --git a/Makefile b/Makefile index 877c6c47b6..6d3612b962 100644 --- a/Makefile +++ b/Makefile @@ -699,6 +699,7 @@ PROGRAMS += $(patsubst %.o,git-%$X,$(PROGRAM_OBJS)) TEST_BUILTINS_OBJS += test-advise.o TEST_BUILTINS_OBJS += test-bitmap.o TEST_BUILTINS_OBJS += test-bloom.o +TEST_BUILTINS_OBJS += test-bundle-uri.o TEST_BUILTINS_OBJS += test-chmtime.o TEST_BUILTINS_OBJS += test-config.o TEST_BUILTINS_OBJS += test-crontab.o diff --git a/bundle-uri.c b/bundle-uri.c index 2d93e8b003..d48bb78012 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -63,3 +63,83 @@ int bundle_uri_command(struct repository *r, return 0; } + +/** + * General API for {transport,connect}.c etc. + */ +int bundle_uri_parse_line(struct string_list *bundle_uri, const char *line) +{ + int i; + struct string_list uri = STRING_LIST_INIT_DUP; + struct string_list_item *item = NULL; + int err = 0; + + /* + * Right now we don't understand anything beyond the first SP, + * but let's be tolerant and ignore any future unknown + * fields. See the "MUST" note about "bundle-feature-key" in + * technical/protocol-v2.txt + */ + if (string_list_split(&uri, line, ' ', -1) < 1) + return error(_("bundle-uri line not in SP-delimited format: %s"), line); + + for (i = 0; i < uri.nr; i++) { + struct string_list kv = STRING_LIST_INIT_DUP; + struct string_list_item *kv_item = NULL; + const char *arg = uri.items[i].string; + int fields; + + /* + * The "string" for each list item is the parsed URI + * at the start of the line + */ + if (i == 0) { + item = string_list_append(bundle_uri, arg); + continue; + } + + /* + * Anything else on the line is keys or key-value + * pairs separated by "=". + * + * Let's parse the format, even if we don't understand + * any of the keys or values yet. + */ + assert(item); + arg = uri.items[i].string; + if (i == 1) { + item->util = xcalloc(1, sizeof(struct string_list)); + string_list_init(item->util, 1); + } + + fields = string_list_split(&kv, arg, '=', 2); + if (fields < 1 || fields > 2) { + err = error("expected `k` or `k=v` in column %d of bundle-uri line '%s', got '%s'", + i, line, arg); + string_list_clear(&kv, 0); + continue; + } + + kv_item = string_list_append(item->util, kv.items[0].string); + if (kv.nr == 2) + kv_item->util = xstrdup(kv.items[1].string); + + string_list_clear(&kv, 0); + } + string_list_clear(&uri, 0); + return err; +} + +static void bundle_uri_string_list_clear_cb(void *util, const char *string) +{ + struct string_list *fields = util; + if (!fields) + return; + string_list_clear(fields, 1); + free(fields); +} + +void bundle_uri_string_list_clear(struct string_list *bundle_uri) +{ + string_list_clear_func(bundle_uri, bundle_uri_string_list_clear_cb); +} diff --git a/bundle-uri.h b/bundle-uri.h index 6a40efeb39..2d271801b8 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -4,6 +4,7 @@ struct repository; struct packet_reader; struct packet_writer; +struct string_list; /** * serve.[ch] API. @@ -11,4 +12,19 @@ struct packet_writer; int bundle_uri_advertise(struct repository *r, struct strbuf *value); int bundle_uri_command(struct repository *r, struct packet_reader *request); +/** + * General API for {transport,connect}.c etc. + */ + +/** + * bundle_uri_parse_line() returns 0 when a valid bundle-uri has been + * added to `bundle_uri`, <0 on error. + */ +int bundle_uri_parse_line(struct string_list *bundle_uri, const char *line); + +/** + * Clear the `bundle_uri` list. Just a very thin wrapper on + * string_list_clear(). + */ +void bundle_uri_string_list_clear(struct string_list *bundle_uri); #endif /* BUNDLE_URI_H */ diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c new file mode 100644 index 0000000000..014da1b6ab --- /dev/null +++ b/t/helper/test-bundle-uri.c @@ -0,0 +1,80 @@ +#include "test-tool.h" +#include "parse-options.h" +#include "bundle-uri.h" +#include "strbuf.h" +#include "string-list.h" + +static int cmd__bundle_uri_parse(int argc, const char **argv) +{ + const char *usage[] = { + "test-tool bundle-uri parse util; + + fprintf(stdout, "%s", item->string); + if (!kv) { + fprintf(stdout, "\n"); + continue; + } + for_each_string_list_item(kv_item, kv) { + const char *k = kv_item->string; + const char *v = kv_item->util; + + if (v) + fprintf(stdout, " [kv: %s => %s]", k, v); + else + fprintf(stdout, " [attr: %s]", k); + } + fprintf(stdout, "\n"); + } + strbuf_release(&sb); + + bundle_uri_string_list_clear(&list); + + return err < 0 ? 1 : 0; +usage: + usage_with_options(usage, options); +} + +int cmd__bundle_uri(int argc, const char **argv) +{ + const char *usage[] = { + "test-tool bundle-uri []", + NULL + }; + struct option options[] = { + OPT_END(), + }; + + argc = parse_options(argc, argv, NULL, options, usage, + PARSE_OPT_STOP_AT_NON_OPTION | + PARSE_OPT_KEEP_ARGV0); + if (argc == 1) + goto usage; + + if (!strcmp(argv[1], "parse")) + return cmd__bundle_uri_parse(argc - 1, argv + 1); + error("there is no test-tool bundle-uri tool '%s'", argv[1]); + +usage: + usage_with_options(usage, options); +} diff --git a/t/helper/test-tool.c b/t/helper/test-tool.c index 3ce5585e53..b6e1ee7b25 100644 --- a/t/helper/test-tool.c +++ b/t/helper/test-tool.c @@ -17,6 +17,7 @@ static struct test_cmd cmds[] = { { "advise", cmd__advise_if_enabled }, { "bitmap", cmd__bitmap }, { "bloom", cmd__bloom }, + { "bundle-uri", cmd__bundle_uri }, { "chmtime", cmd__chmtime }, { "config", cmd__config }, { "crontab", cmd__crontab }, diff --git a/t/helper/test-tool.h b/t/helper/test-tool.h index 9f0f522850..ef839ac726 100644 --- a/t/helper/test-tool.h +++ b/t/helper/test-tool.h @@ -7,6 +7,7 @@ int cmd__advise_if_enabled(int argc, const char **argv); int cmd__bitmap(int argc, const char **argv); int cmd__bloom(int argc, const char **argv); +int cmd__bundle_uri(int argc, const char **argv); int cmd__chmtime(int argc, const char **argv); int cmd__config(int argc, const char **argv); int cmd__crontab(int argc, const char **argv); diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh new file mode 100755 index 0000000000..ef1b9fedea --- /dev/null +++ b/t/t5750-bundle-uri-parse.sh @@ -0,0 +1,98 @@ +#!/bin/sh + +test_description="Test bundle-uri bundle_uri_parse_line()" + +TEST_NO_CREATE_REPO=1 +. ./test-lib.sh + +test_expect_success 'bundle_uri_parse_line() just URIs' ' + cat >in <<-\EOF && + http://example.com/bundle.bdl + https://example.com/bundle.bdl + file:///usr/share/git/bundle.bdl + EOF + + # For the simple case + cp in expect && + + test-tool bundle-uri parse actual 2>err && + test_must_be_empty err && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() with attributes' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl attr + http://example.com/bundle2.bdl ibute + EOF + + + cat >expect <<-\EOF && + http://example.com/bundle1.bdl [attr: attr] + http://example.com/bundle2.bdl [attr: ibute] + EOF + + test-tool bundle-uri parse actual 2>err && + test_must_be_empty err && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() with attributes and key-value attributes' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl x a=b y c=d z e=f a=b + EOF + + + cat >expect <<-\EOF && + http://example.com/bundle1.bdl [attr: x] [kv: a => b] [attr: y] [kv: c => d] [attr: z] [kv: e => f] [kv: a => b] + EOF + + test-tool bundle-uri parse actual 2>err && + test_must_be_empty err && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: extra SP' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl one-space + http://example.com/bundle1.bdl two-space + http://example.com/bundle1.bdl three-space + EOF + + # We are anal just the one SP + cat >expect <<-\EOF && + http://example.com/bundle1.bdl [attr: one-space] + http://example.com/bundle1.bdl [attr: ] [attr: two-space] + http://example.com/bundle1.bdl [attr: ] [attr: ] [attr: three-space] + EOF + + test-tool bundle-uri parse actual 2>err && + test_must_be_empty err && + test_cmp expect actual +' + +test_expect_success 'bundle_uri_parse_line() parsing edge cases: multiple = in key-values' ' + cat >in <<-\EOF && + http://example.com/bundle1.bdl k=v=extra + http://example.com/bundle2.bdl a=b k=v=extra c=d + EOF + + cat >err.expect <<-\EOF && + error: expected `k` or `k=v` in column 1 of bundle-uri line '"'"'http://example.com/bundle1.bdl k=v=extra'"'"', got '"'"'k=v=extra'"'"' + error: bad line: http://example.com/bundle1.bdl k=v=extra + error: expected `k` or `k=v` in column 2 of bundle-uri line '"'"'http://example.com/bundle2.bdl a=b k=v=extra c=d'"'"', got '"'"'k=v=extra'"'"' + error: bad line: http://example.com/bundle2.bdl a=b k=v=extra c=d + EOF + + # We fail, but try to continue parsing regardless + cat >expect <<-\EOF && + http://example.com/bundle1.bdl + http://example.com/bundle2.bdl [kv: a => b] [kv: c => d] + EOF + + test_must_fail test-tool bundle-uri parse actual 2>err.actual && + test_cmp err.expect err.actual && + test_cmp expect actual +' + +test_done From patchwork Thu Aug 5 15:07:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12421397 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 96610C432BE for ; Thu, 5 Aug 2021 15:07:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 815BC6104F for ; Thu, 5 Aug 2021 15:07:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241824AbhHEPIB (ORCPT ); Thu, 5 Aug 2021 11:08:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51040 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241800AbhHEPHz (ORCPT ); Thu, 5 Aug 2021 11:07:55 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 12D83C061798 for ; Thu, 5 Aug 2021 08:07:40 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id b13so7023774wrs.3 for ; Thu, 05 Aug 2021 08:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=9CR5UmrmRUux5yP3B4QxqMrRhlY/t/mCix8RY9b8Pv8=; b=L9/HMiJJi3UFR3mwYZChp6OPy0op8lquWbw0A0bLzIkR20DfgdAh5KkR2Az0YX+R/Q 9faP0QP4K3Eb1rC2+zztev9v7Rv2+hmNNU63sn+S/8Kw1Fl8hvCO7kT3xT0Kg8WqW97/ rAJvPZQzqdvJQS/fJ+mS1S4xczEG539VqFEuXYj7OGQtHgoSM/JYzkmSr1lihpHvRXUe azTCJKVYEJUf9qZ0nKv3PSLPXX/fZ1fCY/+iiIUyWEUjFQ+lT5iqomwCrtngLsoQNnj/ k2MltLexFtVV91RDD1JDW3DbP6LjQ3w1CT0K3Lov7v/J6sZnSFZrkkcDiW4+Z/jPJWSs o9yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=9CR5UmrmRUux5yP3B4QxqMrRhlY/t/mCix8RY9b8Pv8=; b=OxMs1GMhyZCwIgI2/1X8NtKXine6s+LZrEGcq5Tq9lTMwMQQVUfi/LUgaxK3ffCZjv naRIYInWvhT7PPaISCYPOsn6KT9JwOp1WnS41kiaxVMtNz9Rdlr6jTw6LwOX8eoC9D0j TJVZGB4ZFwTBkpULzu3FtEl8O0mFOKv5dqD2m+sMFxdLQsU99YOA4HjK+14ickVevdUV OJinTkiuF5cWUFCHSp1V0LP0cPqQvmFmeRhmkb2hI8AgR0j4e4yWFKNB8TgstoAYfPtI EL5g7YkfxFowJ2CIGhMOmXgEGh2oOQTmdLlJMdexKc5msF//npSwKXEDOiV2meDojDRT qUwg== X-Gm-Message-State: AOAM530cHws9n/qvqT+cMmsmxaGwnP6WrMn/Ws1odI10uRMizFflxXzy lCBMrm8SsEnUBkTBIfwe9hsGxha4eS/FeA== X-Google-Smtp-Source: ABdhPJwohvmmtvzzCZ/C43GyAuRdmKN62OB2iUy6GiKXHP1kLGpGuBrJEYpXU8NLE5FavL1oBEqd+w== X-Received: by 2002:a5d:5381:: with SMTP id d1mr5739119wrv.418.1628176058484; Thu, 05 Aug 2021 08:07:38 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id w4sm6856340wrm.24.2021.08.05.08.07.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Aug 2021 08:07:37 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Christian Couder , Albert Cui , Jonathan Tan , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH 03/13] connect.c: refactor sending of agent & object-format Date: Thu, 5 Aug 2021 17:07:19 +0200 Message-Id: X-Mailer: git-send-email 2.33.0.rc0.646.g585563e77f In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Refactor the sending of the "agent" and "object-format" capabilities into a function. This was added in its current form in ab67235bc4 (connect: parse v2 refs with correct hash algorithm, 2020-05-25). When we connect to a v2 server we need to know about its object-format, and it needs to know about ours. Since most things in connect.c and transport.c piggy-back on the eager getting of remote refs via the handshake() those commands can make use of the just-sent-over object-format by ls-refs. But I'm about to add a command that may come after ls-refs, and may not, but we need the server to know about our user-agent and object-format. So let's split this into a function. Signed-off-by: Ævar Arnfjörð Bjarmason --- connect.c | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/connect.c b/connect.c index 70b13389ba..18dfeae2b8 100644 --- a/connect.c +++ b/connect.c @@ -471,6 +471,24 @@ void check_stateless_delimiter(int stateless_rpc, die("%s", error); } +static void send_capabilities(int fd_out, struct packet_reader *reader) +{ + const char *hash_name; + + if (server_supports_v2("agent", 0)) + packet_write_fmt(fd_out, "agent=%s", git_user_agent_sanitized()); + + if (server_feature_v2("object-format", &hash_name)) { + int hash_algo = hash_algo_by_name(hash_name); + if (hash_algo == GIT_HASH_UNKNOWN) + die(_("unknown object format '%s' specified by server"), hash_name); + reader->hash_algo = &hash_algos[hash_algo]; + packet_write_fmt(fd_out, "object-format=%s", reader->hash_algo->name); + } else { + reader->hash_algo = &hash_algos[GIT_HASH_SHA1]; + } +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, @@ -478,7 +496,6 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, int stateless_rpc) { int i; - const char *hash_name; struct strvec *ref_prefixes = transport_options ? &transport_options->ref_prefixes : NULL; char **unborn_head_target = transport_options ? @@ -488,18 +505,8 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, if (server_supports_v2("ls-refs", 1)) packet_write_fmt(fd_out, "command=ls-refs\n"); - if (server_supports_v2("agent", 0)) - packet_write_fmt(fd_out, "agent=%s", git_user_agent_sanitized()); - - if (server_feature_v2("object-format", &hash_name)) { - int hash_algo = hash_algo_by_name(hash_name); - if (hash_algo == GIT_HASH_UNKNOWN) - die(_("unknown object format '%s' specified by server"), hash_name); - reader->hash_algo = &hash_algos[hash_algo]; - packet_write_fmt(fd_out, "object-format=%s", reader->hash_algo->name); - } else { - reader->hash_algo = &hash_algos[GIT_HASH_SHA1]; - } + /* Send capabilities */ + send_capabilities(fd_out, reader); if (server_options && server_options->nr && server_supports_v2("server-option", 1)) From patchwork Thu Aug 5 15:07:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12421395 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-20.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F284C4320A for ; Thu, 5 Aug 2021 15:07:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EBFFE60EEA for ; Thu, 5 Aug 2021 15:07:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241821AbhHEPIA (ORCPT ); Thu, 5 Aug 2021 11:08:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51046 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241799AbhHEPHz (ORCPT ); Thu, 5 Aug 2021 11:07:55 -0400 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id EE0F3C061799 for ; Thu, 5 Aug 2021 08:07:40 -0700 (PDT) Received: by mail-wr1-x433.google.com with SMTP id k4so7049267wrc.0 for ; Thu, 05 Aug 2021 08:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DJBWBNRJ22Ea59m490UARLLFkDsa+NdsEYVAWGP4px0=; b=jK/MvXxj8pqMwi+GTwe8RhZ7I0g4T5K2XbSVMJjuNDdiHSgzLbWD/5ZWqlxpd+DR66 hh7OEn2Oe4nSpotrcJ673kTYug5BNwFcPzt2Exsq0zb5q6YzPNDcr2dHjv1dhZQg62lz +EyShORRKQyKmWHC3ksk4NS3Via7/rJ0U8b7t9buFMG7qol4F0Z/B5a/byXmqNkVUblo +bkQGp60nOFKlFDA8NoNvAi41jdRlh3yA9oXkGqH4r2gXiurgzPdojSbIVoYLE04dmZr nWWEwVamIvIPkPEY/FDIaS+Wb8hsy0bvNdOp9tdjJ8G/qs7FIsKHP4GJE0hqC0mMvBt0 zmtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DJBWBNRJ22Ea59m490UARLLFkDsa+NdsEYVAWGP4px0=; b=gQ8wajWU1Kymjn8ZThPOcOuiQhbKaebyo0lUCtJT52QP21YSdX4PW54mJ8HL4IoUuh qAFYQOjz1gjI5+FHnkq/Vpu112XgghaieKzIB+O/x7eTZDEnv3L5RJ0NEYQQWz1Rk8Cq i1BKIiLorBNzIeQAEruKzFaV31casG4GEccs2VMAJWZ7d/cqn/QvEqGLYKpIOOq9ZDVu OgN+TgSgegNkEcSoIdCRz3tDse/JX+cBa8EtfonkQO1IL+09sLnB8B1evWXuBsfzVC6o XJjpCvPeQUO4HrvvENq0gkr5ngLAoJnCYD0E446tnaioBDfHLMaZ3MfV18DdzF0sl83J V8Og== X-Gm-Message-State: AOAM530aLnX904sfspkUjAyTK5cnhfd/R6V+xKCtnkx+HLUntK1JKl3q IYQ9WFf+sK7uB+f/ugq7eUk2nzJ34yUGSg== X-Google-Smtp-Source: ABdhPJzSl8PzPVxHyvzzoeAvTZymbRfiDCirhu45qnw/kbKX6/hc1Cpbd44+Wj8qBKJkjo6c9S5/Kg== X-Received: by 2002:adf:d4c6:: with SMTP id w6mr5866160wrk.353.1628176059213; Thu, 05 Aug 2021 08:07:39 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id w4sm6856340wrm.24.2021.08.05.08.07.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Aug 2021 08:07:38 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Christian Couder , Albert Cui , Jonathan Tan , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH 04/13] bundle-uri client: add minimal NOOP client Date: Thu, 5 Aug 2021 17:07:20 +0200 Message-Id: X-Mailer: git-send-email 2.33.0.rc0.646.g585563e77f In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Set up all the needed client parts of the "bundle-uri" protocol extension, without actually doing anything with the bundle URIs. I.e. if the server says it supports "bundle-uri" we'll issue a command=bundle-uri after command=ls-refs when we're cloning. We'll parse the returned output using the code already tested for in t5750-bundle-uri-parse.sh. What we aren't doing is actually acting on that data, i.e. downloading the bundle(s) before we get to doing the command=fetch, and adjusting our negotiation dialog appropriately. I'll do that in subsequent commits. There's a question of what level of encapsulation we should use here, I've opted to use connect.h in clone.c, but we could also e.g. make transport_get_remote_refs() invoke this, i.e. make it implicitly get the bundle-uri list for later steps. This approach means that we don't "support" this in "git fetch" for now. I'm starting with the case of initial clones, although as noted in preceding commits to the protocol documentation nothing about this approach precludes getting bundles on incremental fetches. For the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set environment variables for git-upload-pack (it's started by Apache), so let's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT} prerequisites for consistency and future use. Signed-off-by: Ævar Arnfjörð Bjarmason --- builtin/clone.c | 7 ++ bundle-uri.c | 10 +- connect.c | 47 +++++++++ remote.h | 4 + t/lib-t5730-protocol-v2-bundle-uri.sh | 140 +++++++++++++++++++++++++ t/t5730-protocol-v2-bundle-uri-file.sh | 36 +++++++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++ t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++ transport-helper.c | 13 +++ transport-internal.h | 7 ++ transport.c | 48 +++++++++ transport.h | 18 ++++ 12 files changed, 362 insertions(+), 2 deletions(-) create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh diff --git a/builtin/clone.c b/builtin/clone.c index 66fe66679c..1e4e6be57d 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -27,6 +27,7 @@ #include "iterator.h" #include "sigchain.h" #include "branch.h" +#include "connect.h" #include "remote.h" #include "run-command.h" #include "connected.h" @@ -1292,6 +1293,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix) refs = transport_get_remote_refs(transport, &transport_ls_refs_options); + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + if (refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/bundle-uri.c b/bundle-uri.c index d48bb78012..56619bde22 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -38,11 +38,16 @@ static int bundle_uri_startup_config(const char *var, const char *value, int bundle_uri_advertise(struct repository *r, struct strbuf *value) { + if (value && + git_env_bool("GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE", 0)) + strbuf_addstr(value, "test-unknown-capability-value"); + if (advertise_bundle_uri == -1) { git_config(bundle_uri_startup_config, NULL); if (advertise_bundle_uri == -1) advertise_bundle_uri = 0; } + return advertise_bundle_uri; } @@ -53,9 +58,10 @@ int bundle_uri_command(struct repository *r, packet_writer_init(&writer, 1); while (packet_reader_read(request) == PACKET_READ_NORMAL) - die("bundle-uri: unexpected argument: '%s'", request->line); + die(_("bundle-uri: unexpected argument: '%s'"), request->line); + if (request->status != PACKET_READ_FLUSH) - die("bundle-uri: expected flush after arguments"); + die(_("bundle-uri: expected flush after arguments")); send_bundle_uris(&writer, &bundle_uris); diff --git a/connect.c b/connect.c index 18dfeae2b8..c7601ffd83 100644 --- a/connect.c +++ b/connect.c @@ -15,6 +15,7 @@ #include "version.h" #include "protocol.h" #include "alias.h" +#include "bundle-uri.h" static char *server_capabilities_v1; static struct strvec server_capabilities_v2 = STRVEC_INIT; @@ -489,6 +490,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader) } } +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct string_list *bundle_uri, int stateless_rpc) +{ + int line_nr = 1; + + /* Assert bundle-uri support */ + server_supports_v2("bundle-uri", 1); + + /* (Re-)send capabilities */ + send_capabilities(fd_out, reader); + + /* Send command */ + packet_write_fmt(fd_out, "command=bundle-uri\n"); + packet_delim(fd_out); + + /* Send options */ + if (git_env_bool("GIT_TEST_PROTOCOL_BAD_BUNDLE_URI", 0)) + packet_write_fmt(fd_out, "test-bad-client\n"); + packet_flush(fd_out); + + /* Process response from server */ + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { + const char *line = reader->line; + line_nr++; + + if (!bundle_uri_parse_line(bundle_uri, line)) + continue; + + return error(_("error on bundle-uri response line %d: %s"), + line_nr, line); + } + + if (reader->status != PACKET_READ_FLUSH) + return error(_("expected flush after bundle-uri listing")); + + /* + * Might die(), but obscure enough that that's OK, e.g. in + * serve.c we'll call BUG() on its equivalent (the + * PACKET_READ_RESPONSE_END check). + */ + check_stateless_delimiter(stateless_rpc, reader, + _("expected response end packet after ref listing")); + + return 0; +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, diff --git a/remote.h b/remote.h index 5a59198252..a0823fe4e9 100644 --- a/remote.h +++ b/remote.h @@ -202,6 +202,10 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, const struct string_list *server_options, int stateless_rpc); +/* Used for protocol v2 in order to retrieve refs from a remote */ +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct string_list *bundle_uri, int stateless_rpc); + int resolve_remote_symref(struct ref *ref, struct ref *list); /* diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh new file mode 100644 index 0000000000..6bd9d2dcfb --- /dev/null +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -0,0 +1,140 @@ +# Included from t573*-protocol-v2-bundle-uri-*.sh + +T5730_PARENT= +T5730_URI= +T5730_BUNDLE_URI= +case "$T5730_PROTOCOL" in +file) + T5730_PARENT=file_parent + T5730_URI="file://$PWD/file_parent" + T5730_BUNDLE_URI="$T5730_URI/fake.bdl" + test_set_prereq T5730_FILE + ;; +git) + . "$TEST_DIRECTORY"/lib-git-daemon.sh + start_git_daemon --export-all --enable=receive-pack + T5730_PARENT="$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent" + T5730_URI="$GIT_DAEMON_URL/parent" + T5730_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq T5730_GIT + ;; +http) + . "$TEST_DIRECTORY"/lib-httpd.sh + start_httpd + T5730_PARENT="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" + T5730_URI="$HTTPD_URL/smart/http_parent" + test_set_prereq T5730_HTTP + ;; +*) + BUG "Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)" + ;; +esac + +test_expect_success "setup protocol v2 $T5730_PROTOCOL:// tests" ' + git init "$T5730_PARENT" && + test_commit -C "$T5730_PARENT" one +' + +# Poor man's URI escaping. Good enough for the test suite whose trash +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art. +test_uri_escape() { + sed 's/ /%20/g' +} + +case "$T5730_PROTOCOL" in +http) + test_expect_success "setup config for $T5730_PROTOCOL:// tests" ' + git -C "$T5730_PARENT" config http.receivepack true + ' + ;; +*) + ;; +esac +T5730_BUNDLE_URI_ESCAPED=$(echo "$T5730_BUNDLE_URI" | test_uri_escape) + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri" ' + test_when_finished "rm -f log" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + ! grep bundle-uri log +' + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep bundle-uri log +' + +test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" uploadpack.bundleURI \ + "$T5730_BUNDLE_URI_ESCAPED" && + + cat >err.expect <<-\EOF && + Cloning into '"'"'child'"'"'... + EOF + case "$T5730_PROTOCOL" in + file) + cat >fatal-bundle-uri.expect <<-\EOF + fatal: bundle-uri: unexpected argument: '"'"'test-bad-client'"'"' + EOF + ;; + *) + cat >fatal.expect <<-\EOF + fatal: read error: Connection reset by peer + EOF + ;; + esac && + + test_when_finished "rm -rf child" && + test_must_fail ok=sigpipe env \ + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \ + git -c protocol.version=2 \ + clone "$T5730_URI" child \ + >out 2>err && + test_must_be_empty out && + + grep -v -e ^fatal: -e ^error: err >err.actual && + test_cmp err.expect err.actual && + + case "$T5730_PROTOCOL" in + file) + # Due to general race conditions with client/server replies we + # may or may not get "fatal: the remote end hung up + # expectedly" here + grep "^fatal: bundle-uri:" err >fatal-bundle-uri.actual && + test_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual + ;; + *) + grep "^fatal:" err >fatal.actual && + test_cmp fatal.expect fatal.actual + ;; + esac && + + grep "clone> test-bad-client$" log >sent-bad-request && + test_file_not_empty sent-bad-request +' diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh new file mode 100755 index 0000000000..89203d3a23 --- /dev/null +++ b/t/t5730-protocol-v2-bundle-uri-file.sh @@ -0,0 +1,36 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'file://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'file://' transport +# +T5730_PROTOCOL=file +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_expect_success "unknown capability value with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + grep "> bundle-uri=test-unknown-capability-value" log +' + +test_done diff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh new file mode 100755 index 0000000000..282847b311 --- /dev/null +++ b/t/t5731-protocol-v2-bundle-uri-git.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=git +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh new file mode 100755 index 0000000000..fcc1cf3fae --- /dev/null +++ b/t/t5732-protocol-v2-bundle-uri-http.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=http +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/transport-helper.c b/transport-helper.c index e8dbdd1153..aff27bcf78 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1260,9 +1260,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport, return ret; } +static int get_bundle_uri(struct transport *transport) +{ + get_helper(transport); + + if (process_connect(transport, 0)) { + do_take_over(transport); + return transport->vtable->get_bundle_uri(transport); + } + + return -1; +} + static struct transport_vtable vtable = { .set_option = set_helper_option, .get_refs_list = get_refs_list, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs, .push_refs = push_refs, .connect = connect_helper, diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733a..90ea749e5c 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -26,6 +26,13 @@ struct transport_vtable { struct ref *(*get_refs_list)(struct transport *transport, int for_push, struct transport_ls_refs_options *transport_options); + /** + * Populates the remote side's bundle-uri under protocol v2, + * if the "bundle-uri" capability was advertised. Returns 0 if + * OK, negative values on error. + */ + int (*get_bundle_uri)(struct transport *transport); + /** * Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. diff --git a/transport.c b/transport.c index f9400b9b0b..444cf74756 100644 --- a/transport.c +++ b/transport.c @@ -22,6 +22,7 @@ #include "protocol.h" #include "object-store.h" #include "color.h" +#include "bundle-uri.h" static int transport_use_color = -1; static char transport_colors[][COLOR_MAXLEN] = { @@ -345,6 +346,21 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus return handshake(transport, for_push, options, 1); } +static int get_bundle_uri(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct packet_reader reader; + int stateless_rpc = transport->stateless_rpc; + string_list_init(&transport->bundle_uri, 1); + + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF); + + return get_remote_bundle_uri(data->fd[1], &reader, + &transport->bundle_uri, stateless_rpc); +} + static int fetch_refs_via_pack(struct transport *transport, int nr_heads, struct ref **to_fetch) { @@ -884,6 +900,7 @@ static int disconnect_git(struct transport *transport) static struct transport_vtable taken_over_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .disconnect = disconnect_git @@ -1037,6 +1054,7 @@ static struct transport_vtable bundle_vtable = { static struct transport_vtable builtin_smart_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .connect = connect_git, @@ -1050,6 +1068,7 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->progress = isatty(2); string_list_init_dup(&ret->pack_lockfiles); + string_list_init_dup(&ret->bundle_uri); if (!remote) BUG("No remote provided to transport_get()"); @@ -1453,6 +1472,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } +int transport_get_remote_bundle_uri(struct transport *transport) +{ + const struct transport_vtable *vtable = transport->vtable; + + /* Lazily configured */ + if (transport->got_remote_bundle_uri++) + return 0; + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + + /* + * This is intentionally below the transport.injectBundleURI, + * we want to be able to inject into protocol v0, or into the + * dialog of a server who doesn't support this. + */ + if (!vtable->get_bundle_uri) + return error(_("bundle-uri operation not supported by protocol")); + + if (vtable->get_bundle_uri(transport) < 0) + return error(_("could not retrieve server-advertised bundle-uri list")); + return 0; +} + void transport_unlock_pack(struct transport *transport) { int i; @@ -1478,6 +1525,7 @@ int transport_disconnect(struct transport *transport) ret = transport->vtable->disconnect(transport); if (transport->got_remote_refs) free_refs((void *)transport->remote_refs); + bundle_uri_string_list_clear(&transport->bundle_uri); free(transport); return ret; } diff --git a/transport.h b/transport.h index 1cbab11373..fb9f89f956 100644 --- a/transport.h +++ b/transport.h @@ -75,6 +75,18 @@ struct transport { */ unsigned got_remote_refs : 1; + /** + * Indicates whether we already called get_bundle_uri_list(); set by + * transport.c::transport_get_remote_bundle_uri(). + */ + unsigned got_remote_bundle_uri : 1; + + /* + * The results of "command=bundle-uri", if both sides support + * the "bundle-uri" capability. + */ + struct string_list bundle_uri; + /* * Transports that call take-over destroys the data specific to * the transport type while doing so, and cannot be reused. @@ -270,6 +282,12 @@ struct transport_ls_refs_options { const struct ref *transport_get_remote_refs(struct transport *transport, struct transport_ls_refs_options *transport_options); +/** + * Retrieve bundle URI(s) from a remote. Populates "struct + * transport"'s "bundle_uri" and "got_remote_bundle_uri". + */ +int transport_get_remote_bundle_uri(struct transport *transport); + /* * Fetch the hash algorithm used by a remote. * From patchwork Thu Aug 5 15:07:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 12421399 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D5F6C4338F for ; Thu, 5 Aug 2021 15:07:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0D2106104F for ; Thu, 5 Aug 2021 15:07:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S241833AbhHEPID (ORCPT ); Thu, 5 Aug 2021 11:08:03 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S241802AbhHEPH4 (ORCPT ); Thu, 5 Aug 2021 11:07:56 -0400 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00116C06179A for ; Thu, 5 Aug 2021 08:07:41 -0700 (PDT) Received: by mail-wr1-x42a.google.com with SMTP id c16so6973588wrp.13 for ; Thu, 05 Aug 2021 08:07:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qBDOnItBFnb7alf2HtrbVDLCOI8LvSqiBHDXY/lTj3w=; b=eHmw4bkRCnRs1ryr0xOA8ONuYRj6ZV51aJwf/+MubLzUZB8+FauVtFZihnYBPXcGpi sdS6PPQK714iVIQbYPdPHZJzUzIG1LiGUzcsBrSG+gc1CihtQA1se6dPfxFhDfinB+P4 HvOhrkmcZzYp6F5Y0YWEGYEGGV/oR32eboT3lsSjPWAfIYGqLIn5hsAAf814cfCKNDev 7iSoOeUVARknJg+dGokqAPu4mE0D9UbJmhU185xmrbKjY8LOkBF7xBjJhFFlYFYfi2Ir e26l339Qd+P8m8JEU2SWz3lyOUjy1rjrh6/psdzk4fFfpTzF0KpZk5bTaahep098EHPj SjCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qBDOnItBFnb7alf2HtrbVDLCOI8LvSqiBHDXY/lTj3w=; b=Fy2jqWvF+v5aEspmJgO10ha0V6EigtskqBNFU3/jqzaYXnSdgRDDSVaBmYLKS+53YP cltqwt55aa8yxH3MryjcQRsARAnJKFQAKFGqftx8Y5BIyXzkY9A7z1OPmIukoCqTaKK+ BjTK94Wwo1az5EpNPKmtvjTC4WMhG72IorPHUV6eaI7hTfSgmfW+iD+ORQgt2/JcQ19O f84KNb6lub0zxWzKGWT8tvhRYNLW4hGshSnrr76yQc5NsNfP6WAPdCxrcK9hsUX1fHJ8 yvJ3Hvfa1M5NyrTb1zhTA1ikBrHx5ocuoex1tsCLEF+HozPnw1HLXQ8IbqPDprrjSZ9F XbZw== X-Gm-Message-State: AOAM533YMmkFEPjo1jhR3sB8z87WveJnz2A+HbgUCtKT1Sf4SwTAMkYW EPSOhCHKGURICbReL+LXMtqEenmugGqkSg== X-Google-Smtp-Source: ABdhPJwT9aTiB8Kz1wwq/S0H+a4Fps3l0A4oDgclYwOlrh3uR7C+sf/7xNQpy7Q8s9H7ViatY9hQaQ== X-Received: by 2002:a5d:4d0d:: with SMTP id z13mr5688140wrt.34.1628176060233; Thu, 05 Aug 2021 08:07:40 -0700 (PDT) Received: from vm.nix.is (vm.nix.is. [2a01:4f8:120:2468::2]) by smtp.gmail.com with ESMTPSA id w4sm6856340wrm.24.2021.08.05.08.07.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 05 Aug 2021 08:07:39 -0700 (PDT) From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Christian Couder , Albert Cui , Jonathan Tan , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= Subject: [RFC PATCH 05/13] bundle-uri client: add "git ls-remote-bundle-uri" Date: Thu, 5 Aug 2021 17:07:21 +0200 Message-Id: X-Mailer: git-send-email 2.33.0.rc0.646.g585563e77f In-Reply-To: References: MIME-Version: 1.0 Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a git-ls-remote-bundle-uri command, this is a thin wrapper for issuing protocol v2 "bundle-uri" commands to a server, and to the parsing routines in bundle-uri.c. Since in the "git clone" case we'll have already done the handshake(), but not here, introduce a "got_advertisement" state along with "got_remote_heads". It seems to me that the "got_remote_heads" is badly named in the first place, and the whole logic of eagerly getting ls-refs on handshake() or not could be refactored somewhat, but let's not do that now, and instead just add another self-documenting state variable. Signed-off-by: Ævar Arnfjörð Bjarmason --- Documentation/git-ls-remote-bundle-uri.txt | 63 ++++++++++ Documentation/git-ls-remote.txt | 1 + Makefile | 1 + builtin.h | 1 + builtin/clone.c | 2 +- builtin/ls-remote-bundle-uri.c | 90 ++++++++++++++ command-list.txt | 1 + git.c | 1 + t/lib-t5730-protocol-v2-bundle-uri.sh | 132 +++++++++++++++++++++ transport.c | 43 +++++-- transport.h | 6 +- 11 files changed, 330 insertions(+), 11 deletions(-) create mode 100644 Documentation/git-ls-remote-bundle-uri.txt create mode 100644 builtin/ls-remote-bundle-uri.c diff --git a/Documentation/git-ls-remote-bundle-uri.txt b/Documentation/git-ls-remote-bundle-uri.txt new file mode 100644 index 0000000000..7d3d16a36f --- /dev/null +++ b/Documentation/git-ls-remote-bundle-uri.txt @@ -0,0 +1,63 @@ +git-ls-remote-bundle-uri(1) +=========================== + +NAME +---- +git-ls-remote-bundle-uri - List 'bundle-uri' in a remote repository + + +SYNOPSIS +-------- +[verse] +'git ls-remote-bundle-uri [--quiet] [--uri] [--upload-pack=] + [[-o | --server-option=]