From patchwork Wed Aug 21 22:20:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11108219 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4C7271813 for ; Wed, 21 Aug 2019 22:20:18 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 20D8F205C9 for ; Wed, 21 Aug 2019 22:20:18 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="HBFk1vY/" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731078AbfHUWUR (ORCPT ); Wed, 21 Aug 2019 18:20:17 -0400 Received: from mail-qt1-f201.google.com ([209.85.160.201]:54876 "EHLO mail-qt1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731036AbfHUWUR (ORCPT ); Wed, 21 Aug 2019 18:20:17 -0400 Received: by mail-qt1-f201.google.com with SMTP id t5so4327074qtd.21 for ; Wed, 21 Aug 2019 15:20:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=voJ5YlbZ80ykeqUOFPGRPrkTlLG6UGzqlEmIieYtmOw=; b=HBFk1vY/NPrLkb7c/eXLPgUXgNeaQy310/6eMBWFASnXmFUrtID6Foy0awJ27Fe4i7 WRYqLIiGP31Tcb24XqQaBQ6UrT6YuFCLvnUMDLsTnQBZr8rAPtxEwArXQvAWCCr9CdNO tMw15yxdTCmnpT3x+/Yi7sGaaj4T2/naDJMwLFZfXo54gWwooMlB3vO2Mon/4i6ISl9P +XRzXNsQzjgioy9ah8FnUKQMhkpcakhinxNoDkThYqmGhgq3MPj7CiHxlU4KubHMIjzG KT9jfWRNu7DWnCohlyw9FgVx8lir3Xse47Us1PUQvdbxBpTbwXBWQ5ctwNIz6mL7BscD V3aQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=voJ5YlbZ80ykeqUOFPGRPrkTlLG6UGzqlEmIieYtmOw=; b=LfJQQ9lz3DzPD9a+Mt9UjEPCuGp/CdLFtCqekKcYt1leiF/jFDsTepXeg2FiukdmHV aBrFjYB5V+NbJgs/CBTfXHwHgNccWn887MvFzrvSa3UWYfM7r3qyAT+9zb1mV3Fae7Gn oKJaBkv3arbPL8zE2Hd/KgZ4ZcOtR91zT4bPw+MEsiGhq4ULDegMhVI1jlFsEBjssM2X GtuC/U66AN3TAk+tALn5Xx2lIfvdtVve7UEAbaTzt52ORdUZHwn+Equ6tCpJTWv9kACL nqpnnIS89xSV1Ojf4O/xOjRn0+LZhPhk0WjO851N1McfpFZKMo/y5UzOHpuSCTOfXhch dT2A== X-Gm-Message-State: APjAAAXgFdb9MR0u4ooohKNNBaWPLXnpVSorqxgmbLSXb1ViUDvP+mxU WhJRJ7YyV6RQ0VxVvVzonhtKf9AFCN74TJxKFUoqaXza0+0SZuFT0OjcB1ZnzIuBwyqtkLRdDqN eh2jNIxNQWR24Ycn6TNh/xChClZhi7xrUdz5ZFWrhA4pzsM9yxemnpyp0IeCymBz4ffRILRu7Bk 4W X-Google-Smtp-Source: APXvYqz3ebdpLVrcTgdy70tvWEChFrdPdePFEa2W+S1mxK2gm07YAcq4OLuwqeaaqFHNS9p186Hw066ZKak7t4nFMKzw X-Received: by 2002:a37:a484:: with SMTP id n126mr33828982qke.73.1566426015545; Wed, 21 Aug 2019 15:20:15 -0700 (PDT) Date: Wed, 21 Aug 2019 15:20:09 -0700 In-Reply-To: Message-Id: <9e0971d10117651a9a475c3b5c183a09e617a67b.1566425828.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.23.0.187.g17f5b7556c-goog Subject: [PATCH 1/2] transport-helper: skip ls-refs if unnecessary From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Commit e70a3030e7 ("fetch: do not list refs if fetching only hashes", 2018-10-07) and its ancestors taught Git, as an optimization, to skip the ls-refs step when it is not necessary during a protocol v2 fetch (for example, when lazy fetching a missing object in a partial clone, or when running "git fetch --no-tags "). But that was only done for natively supported protocols; in particular, HTTP was not supported. Teach Git to skip ls-refs when using remote helpers that support connect or stateless-connect. To do this, fetch() is made an acceptable entry point. Because fetch() can now be the first function in the vtable called, "get_helper(transport);" has to be added to the beginning of that function to set the transport up (if not yet set up) before process_connect() is invoked. When fetch() is called, the transport could be taken over (this happens if "connect" or "stateless-connect" is successfully run without any "fallback" response), or not. If the transport is taken over, execution continues like execution for natively supported protocols (fetch_refs_via_pack() is executed, which will fetch refs using ls-refs if needed). If not, the remote helper interface will invoke get_refs_list() if it hasn't been invoked yet, preserving existing behavior. Signed-off-by: Jonathan Tan --- t/t5702-protocol-v2.sh | 13 +++++++++++++ transport-helper.c | 39 +++++++++++++++++++++++++++++++++------ 2 files changed, 46 insertions(+), 6 deletions(-) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 011b81d4fc..61f5080de0 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -631,6 +631,19 @@ test_expect_success 'fetch with http:// using protocol v2' ' grep "git< version 2" log ' +test_expect_success 'fetch with http:// by hash without tag following with protocol v2 does not list refs' ' + test_when_finished "rm -f log" && + + test_commit -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" two_a && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" rev-parse two_a >two_a_hash && + + GIT_TRACE_PACKET="$(pwd)/log" git -C http_child -c protocol.version=2 \ + fetch --no-tags origin $(cat two_a_hash) && + + grep "fetch< version 2" log && + ! grep "fetch> command=ls-refs" log +' + test_expect_success 'fetch from namespaced repo respects namespaces' ' test_when_finished "rm -f log" && diff --git a/transport-helper.c b/transport-helper.c index 6b05a88faf..1fb31e1a6e 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -33,6 +33,16 @@ struct helper_data { check_connectivity : 1, no_disconnect_req : 1, no_private_update : 1; + + /* + * As an optimization, the transport code may invoke fetch before + * get_refs_list. If this happens, and if the transport helper doesn't + * support connect or stateless_connect, we need to invoke + * get_refs_list ourselves if we haven't already done so. Keep track of + * whether we have invoked get_refs_list. + */ + unsigned get_refs_list_called : 1; + char *export_marks; char *import_marks; /* These go from remote name (as in "list") to private name */ @@ -652,17 +662,25 @@ static int connect_helper(struct transport *transport, const char *name, return 0; } +static struct ref *get_refs_list_using_list(struct transport *transport, + int for_push); + static int fetch(struct transport *transport, int nr_heads, struct ref **to_fetch) { struct helper_data *data = transport->data; int i, count; + get_helper(transport); + if (process_connect(transport, 0)) { do_take_over(transport); return transport->vtable->fetch(transport, nr_heads, to_fetch); } + if (!data->get_refs_list_called) + get_refs_list_using_list(transport, 0); + count = 0; for (i = 0; i < nr_heads; i++) if (!(to_fetch[i]->status & REF_STATUS_UPTODATE)) @@ -1058,6 +1076,19 @@ static int has_attribute(const char *attrs, const char *attr) static struct ref *get_refs_list(struct transport *transport, int for_push, const struct argv_array *ref_prefixes) +{ + get_helper(transport); + + if (process_connect(transport, for_push)) { + do_take_over(transport); + return transport->vtable->get_refs_list(transport, for_push, ref_prefixes); + } + + return get_refs_list_using_list(transport, for_push); +} + +static struct ref *get_refs_list_using_list(struct transport *transport, + int for_push) { struct helper_data *data = transport->data; struct child_process *helper; @@ -1066,13 +1097,9 @@ static struct ref *get_refs_list(struct transport *transport, int for_push, struct ref *posn; struct strbuf buf = STRBUF_INIT; + data->get_refs_list_called = 1; helper = get_helper(transport); - if (process_connect(transport, for_push)) { - do_take_over(transport); - return transport->vtable->get_refs_list(transport, for_push, ref_prefixes); - } - if (data->push && for_push) write_str_in_full(helper->in, "list for-push\n"); else @@ -1119,7 +1146,7 @@ static struct ref *get_refs_list(struct transport *transport, int for_push, } static struct transport_vtable vtable = { - 0, + 1, set_helper_option, get_refs_list, fetch, From patchwork Wed Aug 21 22:20:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11108221 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CFC04112C for ; Wed, 21 Aug 2019 22:20:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id AD775205C9 for ; Wed, 21 Aug 2019 22:20:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="L67SPCIc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731082AbfHUWUT (ORCPT ); Wed, 21 Aug 2019 18:20:19 -0400 Received: from mail-vk1-f202.google.com ([209.85.221.202]:47363 "EHLO mail-vk1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1731036AbfHUWUT (ORCPT ); Wed, 21 Aug 2019 18:20:19 -0400 Received: by mail-vk1-f202.google.com with SMTP id n185so1473708vkf.14 for ; Wed, 21 Aug 2019 15:20:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=NOIdeejJSo+ld3TRqxPdwXhoZelO4Zah8uc6JoL/wdQ=; b=L67SPCIcj5M0WzqMRdKrl9MX4gPpFTBKaTDezidCJomApsla7/ix7doLSlmK+8wvus f1xLZ0Tyiay5Qn69LRf0uYJNLUOl8XB8xcZrgdhIIlAIBEMicb9gO+kqrqi1b9XiULtB vDTDih/Yzzz+ww7vcWzF8bvjlxFM7rNIf1SYzYXwzgd4NgX7btqurEAyRZDNLyzoouPT gxcC/fAmbs9lIUp0Au9J/5TjOthyTT44bElplPZDx0QnGjJvEIqHrRbqHo3c4CcmpgM2 KL0v+D2m/K1LmYudz+UVbuOX1p7rFwwdsEYql5ornpBgCTNkAKBxMvEP8scR4KT6XG0z L2IA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=NOIdeejJSo+ld3TRqxPdwXhoZelO4Zah8uc6JoL/wdQ=; b=kTQupYF3mvcl30IFvimLpPSS726DLIxJhNlbN1OWC96T0Nm7EfaV8rxFkWdpphVFYk i71UFtlrGU6v6pNIgBXDOa74sfdAAITSAjf2zXouHI8FdPZtvU/yJDBqGn78sKRTlmU5 wXmfldTfaUsb0BJrRTtXZHZobr0qq+kfbn95PqUjZ+xI94zxmKXlQ79KFCtQpK4DO52y LuvDKrVYF2FFJHJhbB9UzgmzNqHnqSiiYd93SZwf07INNh/e/ZsD6+1yyw057kIfGsel hUg/+mGsKqQ7KB3ZsdB/mJ5+5QDJeYKlWBef6NCAGW+TnlJoETbuJxCzOBzel2v09K7g yjqg== X-Gm-Message-State: APjAAAVdNFz1iQUhYGbsy1k6numPGLDUA1OYyaraGvMLdhsi34uSMRYj xxw627cOwUGvA1EEhAZLnZshIU/VC164VUnvbUcxIz8aEwiBaQLyEuvFoMVu1jQhokz23+Z/wN3 vvKE0QEB2ELVwRBCBI0awUbIwjaieyZsfa4AGu0H4jPQ/7eYq17GS49pNdy0qExcs/JdHzBC5TP HF X-Google-Smtp-Source: APXvYqya9Kir74YvkJi6vw63QLaEZCVoYwXef1zv78C2lMXthRUNfkvY40QCv1+lzwLqBZSMJc8mixz4z+0NCKzKZ70l X-Received: by 2002:a67:eb4e:: with SMTP id x14mr13861284vso.103.1566426018142; Wed, 21 Aug 2019 15:20:18 -0700 (PDT) Date: Wed, 21 Aug 2019 15:20:10 -0700 In-Reply-To: Message-Id: <5795a2a156af01966168176f9540ed1a8a7bb222.1566425828.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.23.0.187.g17f5b7556c-goog Subject: [PATCH 2/2] transport: teach all vtables to allow fetch first From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The only transport that does not allow fetch() to be called before get_refs_list() is the bundle transport. Clean up the code by teaching the bundle transport the ability to do this, and removing support for transports that don't support this order of invocation. Signed-off-by: Jonathan Tan --- t/t5607-clone-bundle.sh | 11 +++++++++++ transport-helper.c | 1 - transport-internal.h | 6 ------ transport.c | 18 ++++++------------ 4 files changed, 17 insertions(+), 19 deletions(-) diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh index 2a0fb15cf1..b7a3fdf02d 100755 --- a/t/t5607-clone-bundle.sh +++ b/t/t5607-clone-bundle.sh @@ -83,4 +83,15 @@ test_expect_success 'failed bundle creation does not leave cruft' ' test_path_is_missing fail.bundle.lock ' +test_expect_success 'fetch SHA-1 from bundle' ' + test_create_repo foo && + test_commit -C foo x && + git -C foo bundle create tip.bundle -1 master && + git -C foo rev-parse HEAD >hash && + + # Exercise to ensure that fetching a SHA-1 from a bundle works with no + # errors + git fetch --no-tags foo/tip.bundle "$(cat hash)" +' + test_done diff --git a/transport-helper.c b/transport-helper.c index 1fb31e1a6e..96955d4004 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1146,7 +1146,6 @@ static struct ref *get_refs_list_using_list(struct transport *transport, } static struct transport_vtable vtable = { - 1, set_helper_option, get_refs_list, fetch, diff --git a/transport-internal.h b/transport-internal.h index 004bee5e36..1cde6258a7 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -6,12 +6,6 @@ struct transport; struct argv_array; struct transport_vtable { - /** - * This transport supports the fetch() function being called - * without get_refs_list() first being called. - */ - unsigned fetch_without_list : 1; - /** * Returns 0 if successful, positive if the option is not * recognized or is inapplicable, and negative if the option diff --git a/transport.c b/transport.c index 778c60bf57..662a2d9ae0 100644 --- a/transport.c +++ b/transport.c @@ -122,6 +122,7 @@ static void set_upstreams(struct transport *transport, struct ref *refs, struct bundle_transport_data { int fd; struct bundle_header header; + unsigned get_refs_from_bundle_called : 1; }; static struct ref *get_refs_from_bundle(struct transport *transport, @@ -135,6 +136,8 @@ static struct ref *get_refs_from_bundle(struct transport *transport, if (for_push) return NULL; + data->get_refs_from_bundle_called = 1; + if (data->fd > 0) close(data->fd); data->fd = read_bundle_header(transport->url, &data->header); @@ -154,6 +157,9 @@ static int fetch_refs_from_bundle(struct transport *transport, int nr_heads, struct ref **to_fetch) { struct bundle_transport_data *data = transport->data; + + if (!data->get_refs_from_bundle_called) + get_refs_from_bundle(transport, 0, NULL); return unbundle(the_repository, &data->header, data->fd, transport->progress ? BUNDLE_VERBOSE : 0); } @@ -742,7 +748,6 @@ static int disconnect_git(struct transport *transport) } static struct transport_vtable taken_over_vtable = { - 1, NULL, get_refs_via_connect, fetch_refs_via_pack, @@ -892,7 +897,6 @@ void transport_check_allowed(const char *type) } static struct transport_vtable bundle_vtable = { - 0, NULL, get_refs_from_bundle, fetch_refs_from_bundle, @@ -902,7 +906,6 @@ static struct transport_vtable bundle_vtable = { }; static struct transport_vtable builtin_smart_vtable = { - 1, NULL, get_refs_via_connect, fetch_refs_via_pack, @@ -1285,15 +1288,6 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) struct ref **heads = NULL; struct ref *rm; - if (!transport->vtable->fetch_without_list) - /* - * Some transports (e.g. the built-in bundle transport and the - * transport helper interface) do not work when fetching is - * done immediately after transport creation. List the remote - * refs anyway (if not already listed) as a workaround. - */ - transport_get_remote_refs(transport, NULL); - for (rm = refs; rm; rm = rm->next) { nr_refs++; if (rm->peer_ref &&