From patchwork Tue Aug 11 22:52:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11709775 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95AD4159A for ; Tue, 11 Aug 2020 22:52:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 77A6920782 for ; Tue, 11 Aug 2020 22:52:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="JK5DU3Dh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726453AbgHKWwb (ORCPT ); Tue, 11 Aug 2020 18:52:31 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43806 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726023AbgHKWwb (ORCPT ); Tue, 11 Aug 2020 18:52:31 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 00F3EC06174A for ; Tue, 11 Aug 2020 15:52:31 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id a6so317335pjd.1 for ; Tue, 11 Aug 2020 15:52:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=bfnczTnXw8CEsvNQ+zFRvEgwXldfKc0Uajp2AQ2rAfg=; b=JK5DU3Dh26NHRxPEl1I7eUTC8jCJl+6smuVuUe9abriHzWsiFVDmrrOa4erqLQd/Py PYX8JxHcIzv9AKn5ESHBtk/l5WH1B0gnQ9VA7Fgs8tjY/ZCNdV9aLIvDqf8X8STBmiyK 7eS8Piw5ir4c1aVG3PTEnWVOLblYhlEa2F7yOJVG9ylYb3Ne8TiQE+G7oe4SkCr9B0la wS2MlWVcRv+2/yZ1gyVKtLqduhVOO6wUq2mWnxR+y06ISvXjL76uqn5eGPMcUUj9GSBT G6+Xnjim/TLADivBlpap7tOvrjJmNGV54nJmBt/qF2wHWAQoZ7ZrSZCViLB3/BSrmzrt KNYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=bfnczTnXw8CEsvNQ+zFRvEgwXldfKc0Uajp2AQ2rAfg=; b=ZhNf/XziBbTuGyHkn5mjkiu0UK9qmXm9bgG/Nn9ofOCF8W1agHzKjenqpnjwUO3RqC CeJ70eqa+MUKnFv4ZO1+EJtUnk6E4UBsuVld6jtE9Q9Mi+LimYE6Gx7GxMswVRz4agEz RDM1UUzCaOfk3e6or+kBs66PkOLLPymMf/kXRVpbrNpZ+SkSZ8FfDw/9UF/qUNG7u6re tuFuUT/NYZM/He6zN0R/fm25JTE78ztn3ijgvRmZj3h+lVE3pXEawli8ai1o4pnbIthR JrPDFOS3+bFhOEsZTkJS2WdAdRHFI2pFz3Nuv3+QqEFWYPesV9WoRWfjNSYxDjRVmsvx 2XTw== X-Gm-Message-State: AOAM531GBcHKVCseVQLkHMcklzSilssBmTRdDwjWYjUQ2RZQ2ZMsTR0e hN0OcN+TJ9qwLJfVwCqEeZGewFGJ82Sfe72/9g+GLezxyz2xCAl4YLQoeUQZGbLR6bEGtZ99fhI zpvOlNpXaHq0RWPnThSMK9+y+dsDKWSAoQVb1vY5JEuknExo6Gy0mv/1p1yUxHuu78rHYj6pmzj Jr X-Google-Smtp-Source: ABdhPJxp1wakPQk7SUBK0wS0HekcSCxybSkae5csFTjmdajs12h5wQi2RoPCzYRZ9eXsqqHqx+8JVVJ4zunnF93EkiO9 X-Received: by 2002:a17:90a:f2c7:: with SMTP id gt7mr3066130pjb.204.1597186349952; Tue, 11 Aug 2020 15:52:29 -0700 (PDT) Date: Tue, 11 Aug 2020 15:52:16 -0700 In-Reply-To: Message-Id: <35bdd372ae3691f54775dd616576e8ed6d68f1d3.1597184949.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog Subject: [PATCH v2 1/7] negotiator/null: add null fetch negotiator From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Add a null fetch negotiator. This is introduced to allow partial clones to skip the unneeded negotiation step when fetching missing objects using a "git fetch" subprocess. (The implementation of spawning a "git fetch" subprocess will be done in a subsequent patch.) But this can also be useful for end users, e.g. as a blunt fix for object corruption. Signed-off-by: Jonathan Tan --- Documentation/config/fetch.txt | 5 +++- Makefile | 1 + fetch-negotiator.c | 5 ++++ negotiator/null.c | 44 ++++++++++++++++++++++++++++++++ negotiator/null.h | 8 ++++++ repo-settings.c | 2 ++ repository.h | 1 + t/t5554-null-fetch-negotiator.sh | 22 ++++++++++++++++ 8 files changed, 87 insertions(+), 1 deletion(-) create mode 100644 negotiator/null.c create mode 100644 negotiator/null.h create mode 100755 t/t5554-null-fetch-negotiator.sh diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt index 0aaa05e8c0..09aff404be 100644 --- a/Documentation/config/fetch.txt +++ b/Documentation/config/fetch.txt @@ -60,7 +60,10 @@ fetch.negotiationAlgorithm:: sent when negotiating the contents of the packfile to be sent by the server. Set to "skipping" to use an algorithm that skips commits in an effort to converge faster, but may result in a larger-than-necessary - packfile; The default is "default" which instructs Git to use the default algorithm + packfile; or set to "null" to not send any information at all, which + will almost certainly result in a larger-than-necessary packfile, but + will skip the negotiation step. + The default is "default" which instructs Git to use the default algorithm that never skips commits (unless the server has acknowledged it or one of its descendants). If `feature.experimental` is enabled, then this setting defaults to "skipping". diff --git a/Makefile b/Makefile index 372139f1f2..297ea5e517 100644 --- a/Makefile +++ b/Makefile @@ -917,6 +917,7 @@ LIB_OBJS += mergesort.o LIB_OBJS += midx.o LIB_OBJS += name-hash.o LIB_OBJS += negotiator/default.o +LIB_OBJS += negotiator/null.o LIB_OBJS += negotiator/skipping.o LIB_OBJS += notes-cache.o LIB_OBJS += notes-merge.o diff --git a/fetch-negotiator.c b/fetch-negotiator.c index 0a1357dc9d..e2ecbe4367 100644 --- a/fetch-negotiator.c +++ b/fetch-negotiator.c @@ -2,6 +2,7 @@ #include "fetch-negotiator.h" #include "negotiator/default.h" #include "negotiator/skipping.h" +#include "negotiator/null.h" #include "repository.h" void fetch_negotiator_init(struct repository *r, @@ -13,6 +14,10 @@ void fetch_negotiator_init(struct repository *r, skipping_negotiator_init(negotiator); return; + case FETCH_NEGOTIATION_NULL: + null_negotiator_init(negotiator); + return; + case FETCH_NEGOTIATION_DEFAULT: default: default_negotiator_init(negotiator); diff --git a/negotiator/null.c b/negotiator/null.c new file mode 100644 index 0000000000..1bd834f121 --- /dev/null +++ b/negotiator/null.c @@ -0,0 +1,44 @@ +#include "cache.h" +#include "null.h" +#include "../commit.h" +#include "../fetch-negotiator.h" + +static void known_common(struct fetch_negotiator *n, struct commit *c) +{ + /* do nothing */ +} + +static void add_tip(struct fetch_negotiator *n, struct commit *c) +{ + /* do nothing */ +} + +static const struct object_id *next(struct fetch_negotiator *n) +{ + return NULL; +} + +static int ack(struct fetch_negotiator *n, struct commit *c) +{ + /* + * This negotiator does not emit any commits, so there is no commit to + * be acknowledged. If there is any ack, there is a bug. + */ + BUG("ack with null negotiator, which does not emit any commits"); + return 0; +} + +static void release(struct fetch_negotiator *n) +{ + /* nothing to release */ +} + +void null_negotiator_init(struct fetch_negotiator *negotiator) +{ + negotiator->known_common = known_common; + negotiator->add_tip = add_tip; + negotiator->next = next; + negotiator->ack = ack; + negotiator->release = release; + negotiator->data = NULL; +} diff --git a/negotiator/null.h b/negotiator/null.h new file mode 100644 index 0000000000..96713f280e --- /dev/null +++ b/negotiator/null.h @@ -0,0 +1,8 @@ +#ifndef NEGOTIATOR_NULL_H +#define NEGOTIATOR_NULL_H + +struct fetch_negotiator; + +void null_negotiator_init(struct fetch_negotiator *negotiator); + +#endif diff --git a/repo-settings.c b/repo-settings.c index 0918408b34..a8c7e1edd7 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -39,6 +39,8 @@ void prepare_repo_settings(struct repository *r) if (!repo_config_get_string(r, "fetch.negotiationalgorithm", &strval)) { if (!strcasecmp(strval, "skipping")) r->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_SKIPPING; + else if (!strcasecmp(strval, "null")) + r->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_NULL; else r->settings.fetch_negotiation_algorithm = FETCH_NEGOTIATION_DEFAULT; } diff --git a/repository.h b/repository.h index 3c1f7d54bd..f72911a185 100644 --- a/repository.h +++ b/repository.h @@ -23,6 +23,7 @@ enum fetch_negotiation_setting { FETCH_NEGOTIATION_NONE = 0, FETCH_NEGOTIATION_DEFAULT = 1, FETCH_NEGOTIATION_SKIPPING = 2, + FETCH_NEGOTIATION_NULL = 3, }; struct repo_settings { diff --git a/t/t5554-null-fetch-negotiator.sh b/t/t5554-null-fetch-negotiator.sh new file mode 100755 index 0000000000..09a8f0d608 --- /dev/null +++ b/t/t5554-null-fetch-negotiator.sh @@ -0,0 +1,22 @@ +#!/bin/sh + +test_description='test null fetch negotiator' +. ./test-lib.sh + +test_expect_success 'null negotiator does not emit any "have"' ' + rm -f trace && + + test_create_repo server && + test_commit -C server to_fetch && + + test_create_repo client && + test_commit -C client we_have && + + test_config -C client fetch.negotiationalgorithm null && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch "$(pwd)/server" && + + ! grep "fetch> have" trace && + grep "fetch> done" trace +' + +test_done From patchwork Tue Aug 11 22:52:17 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11709777 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6021B159A for ; Tue, 11 Aug 2020 22:52:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 472F520774 for ; Tue, 11 Aug 2020 22:52:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Lz2ZOpxi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726479AbgHKWwd (ORCPT ); Tue, 11 Aug 2020 18:52:33 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43814 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726023AbgHKWwc (ORCPT ); Tue, 11 Aug 2020 18:52:32 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 945B8C06174A for ; Tue, 11 Aug 2020 15:52:32 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id bg5so336358plb.18 for ; Tue, 11 Aug 2020 15:52:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Kat1I1RCRdoIRw/u2zLFnXz4xICtec5tGi3//jdxym4=; b=Lz2ZOpxiiXkDGC1eONA4HBlCQStMSX/dKIuklvD8v3JnbmyN9Zo2/RS2TarmjN9CZK f7L8+aOgk3extpAjEEEZB5DBoNSIPwXG7IDZHiPKuEvtl9d+ehVultMdnoIrH7yz2bYr o5NHDhLVj0XL8saUEXFJBJ+vrgJMsZmjaWy7Lm90h8Fj9uOHn795v1mD+LFfmTlP4RFq FaKgmtGGw0Urv8Ol0CP8UbQryEslZ8fio+vNRvrc33l2h3HMh4/o7ljNRFlB9MyTNAx/ wVPcnslh+CQ61vlTJcybtCooMFac8ZkxnO1UTNb0z4Ka8tzD99cZgO6FGNnj8A1GmcKm 3a0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Kat1I1RCRdoIRw/u2zLFnXz4xICtec5tGi3//jdxym4=; b=SgHYCP09w7zsL285OUGOfkAW9U/uUAzApB8BkrwTDhCvSZ7c85rlwzYr8sqeVGedRG QKBV0MJeGolXd0KdC2U40VY+oOwZH9E1wmcymYp6zjwJ6/+JfMlg85PhhWVd22KLAFSg s6SR9QGf8PUvT9UYIkoLElFS0V1nnEAEFBOY6sG9mKsVAJ7s6RBoSAGJRRX/6WF+I+2X 3ERmSTTm1rRkeEwZ4Z3LZXBFH5Dk5n/67hsvOr4LA4XZNhCjHKbRcum8HaV4YjQa7bH2 fmgFDAq6g55njuXuZMuSi4wh44h0BGqqgW6kz3fIYjUH/DaC9sE14s3btg76scmxl0cm XBAQ== X-Gm-Message-State: AOAM530p4JaLR4Y0615MdIgEZgi6te0PSxUZpukcWbGW9IjTpiPwuzyE 6Z7vj0PsfkuJ1yIGmnv61ZMn3T+xVs/N5Xfpf0giLjdXIyI+0C7Gos5KQLdD21+zl3MNkiJ3zhz 7qnh4LdLG3hYwh53CDasPi3q4+rrYiaHmEku87Vb0xxGNxzPZXEK23Qn6B3vgXsOZawWHzfL3YS cw X-Google-Smtp-Source: ABdhPJzyqZI2FBbrdm3plG1I6B6Tjyk8OJHICFSkMtb8yJe/1xCELsfb07vHEb+qwVMDR0Gg2pS50EzZ5VAkKiIcECDY X-Received: by 2002:a17:90a:3488:: with SMTP id p8mr341034pjb.1.1597186351532; Tue, 11 Aug 2020 15:52:31 -0700 (PDT) Date: Tue, 11 Aug 2020 15:52:17 -0700 In-Reply-To: Message-Id: <00ad7dd875d8a6754c0ccf6f2789cec27aaa1d1d.1597184949.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog Subject: [PATCH v2 2/7] fetch: allow refspecs specified through stdin From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a subsequent patch, partial clones will be taught to fetch missing objects using a "git fetch" subprocess. Because the number of objects fetched may be too numerous to fit on the command line, teach "fetch" to accept refspecs passed through stdin. Signed-off-by: Jonathan Tan --- Documentation/git-fetch.txt | 4 ++++ builtin/fetch.c | 19 +++++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index 45b6d8e633..9067c2079e 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -48,6 +48,10 @@ include::fetch-options.txt[] include::pull-fetch-param.txt[] +--stdin:: + Read refspecs, one per line, from stdin in addition to those provided + as arguments. The "tag " format is not supported. + include::urls-remotes.txt[] diff --git a/builtin/fetch.c b/builtin/fetch.c index 3ccf69753f..a5498646bf 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -80,6 +80,7 @@ static struct list_objects_filter_options filter_options; static struct string_list server_options = STRING_LIST_INIT_DUP; static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP; static int fetch_write_commit_graph = -1; +static int stdin_refspecs = 0; static int git_fetch_config(const char *k, const char *v, void *cb) { @@ -209,6 +210,8 @@ static struct option builtin_fetch_options[] = { N_("check for forced-updates on all updated branches")), OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph, N_("write the commit-graph after fetching")), + OPT_BOOL(0, "stdin", &stdin_refspecs, + N_("accept refspecs from stdin")), OPT_END() }; @@ -1684,7 +1687,8 @@ static inline void fetch_one_setup_partial(struct remote *remote) return; } -static int fetch_one(struct remote *remote, int argc, const char **argv, int prune_tags_ok) +static int fetch_one(struct remote *remote, int argc, const char **argv, + int prune_tags_ok, int use_stdin_refspecs) { struct refspec rs = REFSPEC_INIT_FETCH; int i; @@ -1741,6 +1745,13 @@ static int fetch_one(struct remote *remote, int argc, const char **argv, int pru } } + if (use_stdin_refspecs) { + struct strbuf line = STRBUF_INIT; + while (strbuf_getline_lf(&line, stdin) != EOF) + refspec_append(&rs, line.buf); + strbuf_release(&line); + } + if (server_options.nr) gtransport->server_options = &server_options; @@ -1841,7 +1852,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (remote) { if (filter_options.choice || has_promisor_remote()) fetch_one_setup_partial(remote); - result = fetch_one(remote, argc, argv, prune_tags_ok); + result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs); } else { int max_children = max_jobs; @@ -1849,6 +1860,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) die(_("--filter can only be used with the remote " "configured in extensions.partialclone")); + if (stdin_refspecs) + die(_("--stdin can only be used when fetching " + "from one remote")); + if (max_children < 0) max_children = fetch_parallel_config; From patchwork Tue Aug 11 22:52:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11709781 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9BDFD13B6 for ; Tue, 11 Aug 2020 22:52:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 835AF20774 for ; Tue, 11 Aug 2020 22:52:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="SK6vSAjP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726489AbgHKWwf (ORCPT ); Tue, 11 Aug 2020 18:52:35 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43818 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726023AbgHKWwd (ORCPT ); Tue, 11 Aug 2020 18:52:33 -0400 Received: from mail-pl1-x64a.google.com (mail-pl1-x64a.google.com [IPv6:2607:f8b0:4864:20::64a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D6A0DC06174A for ; Tue, 11 Aug 2020 15:52:33 -0700 (PDT) Received: by mail-pl1-x64a.google.com with SMTP id o6so352863pll.9 for ; Tue, 11 Aug 2020 15:52:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=zMoE8u83hGjGVbJdMIkSCp21xQm7MB2j2LIx1Cd3GJo=; b=SK6vSAjPsFg7bx7tMXvFK2puTK6HF7ed63ATvHjNjXB2cyXdcL06CpxbARgdtc+QwE NXwvMM4goNF4kseK0d3MHqkzfvfN1rNhWJiZVmpHR7MYK3M5phk5rV5XoAxSbp/SgFpv na6M/SW2nX7gdTiLVWc7/Gv9z1CAcNUwMUvb9rFwOmlHKDh5nBIRehLHoZRDDTlF8dc5 f8QqP7yDmJh6nsgNg8VaOVGy8KuFn9hz3S2iSAe5ib5DpnFVz30l4jBP7jKAAJw+DbcR oo+gQYEWpshTLqpvssnryA6bOeOMD54IKCGl+SEnwQzzGPB495tWOkSEsdwdFiozzU9/ ry8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=zMoE8u83hGjGVbJdMIkSCp21xQm7MB2j2LIx1Cd3GJo=; b=q2zBpY2n55n6SU75QSEzt14P0Kaa8Mycu1ITUTzhv8jWj94eGmPsZoUV+XlkSQ/sHa 3PvnJcD6iwgb5j/YAfJ9TzFILR9mcc9XxvyVmv9eTmm6ZZIPJ1ATqdK3ttiG74bhNLAN iVzZTmUvZVGqGd1ZipgBESIIhy82wbgdhyH2d7N+mCelcrNLfqCtGAsb54nSTiwPRUIN ne7OlMVr49aBiKYQ9bzIeudfHje9AAH/soDJWC3MvggSr4Yn4W+A3gpobsEsEm3y0B0a 4lN6IYi6nitep0LpCxswJpFgY7tJWCptJVffknrwRtWpqo7iCbjgLv/dMOUGnpr7cvh2 /FRw== X-Gm-Message-State: AOAM530ht08ZduDtXGMDEDSRvq8SdUnkkqkHXfVitbQhKcNYuKVbaPUJ rYda1BMe+Q+Vi3snsNi05lTAqp5H30Fsa90wuIazP488ZEG7u7R/wzwIBaLKqdRbv51oOJ19c7x vVjmJATUMJrkoIP2eVf5qF3+QO0swxGaOXoJgsme+kXGQ/KlOJ7ujAYCcVuxgxdk0edI/NrHBs2 8f X-Google-Smtp-Source: ABdhPJz0hG4rGCs30A+hTSDhd2GRZvRoyixpkBC95Mwj0JvHOjaUNA/8m38T8TmOC0jX8MGtK9z9gvlo621KfJR55Ecs X-Received: by 2002:a17:90a:fc86:: with SMTP id ci6mr3407977pjb.160.1597186353277; Tue, 11 Aug 2020 15:52:33 -0700 (PDT) Date: Tue, 11 Aug 2020 15:52:18 -0700 In-Reply-To: Message-Id: <8b4a522a13089ea144e922280f1c71a4ec3afa1f.1597184949.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog Subject: [PATCH v2 3/7] fetch: avoid reading submodule config until needed From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Teach "git fetch" to avoid reading the submodule config until necessary. This allows users to avoid the lazy-fetching of this potentially missing config file by specifying the --recurse-submodules=no command line option. Signed-off-by: Jonathan Tan --- builtin/fetch.c | 10 ++++++++-- submodule-config.c | 5 +++-- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index a5498646bf..29db219c68 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1786,12 +1786,18 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) free(anon); } - fetch_config_from_gitmodules(&submodule_fetch_jobs_config, - &recurse_submodules); git_config(git_fetch_config, NULL); argc = parse_options(argc, argv, prefix, builtin_fetch_options, builtin_fetch_usage, 0); + if (recurse_submodules != RECURSE_SUBMODULES_OFF) { + int *sfjc = submodule_fetch_jobs_config == -1 + ? &submodule_fetch_jobs_config : NULL; + int *rs = recurse_submodules == RECURSE_SUBMODULES_DEFAULT + ? &recurse_submodules : NULL; + + fetch_config_from_gitmodules(sfjc, rs); + } if (deepen_relative) { if (deepen_relative < 0) diff --git a/submodule-config.c b/submodule-config.c index e175dfbc38..8d65273ed2 100644 --- a/submodule-config.c +++ b/submodule-config.c @@ -776,10 +776,11 @@ struct fetch_config { static int gitmodules_fetch_config(const char *var, const char *value, void *cb) { struct fetch_config *config = cb; - if (!strcmp(var, "submodule.fetchjobs")) { + if (!strcmp(var, "submodule.fetchjobs") && config->max_children) { *(config->max_children) = parse_submodule_fetchjobs(var, value); return 0; - } else if (!strcmp(var, "fetch.recursesubmodules")) { + } else if (!strcmp(var, "fetch.recursesubmodules") && + config->recurse_submodules) { *(config->recurse_submodules) = parse_fetch_recurse_submodules_arg(var, value); return 0; } From patchwork Tue Aug 11 22:52:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11709785 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A6237159A for ; Tue, 11 Aug 2020 22:52:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9091C20782 for ; Tue, 11 Aug 2020 22:52:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sRYtzt/4" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726501AbgHKWwg (ORCPT ); Tue, 11 Aug 2020 18:52:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43826 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726485AbgHKWwf (ORCPT ); Tue, 11 Aug 2020 18:52:35 -0400 Received: from mail-pg1-x549.google.com (mail-pg1-x549.google.com [IPv6:2607:f8b0:4864:20::549]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 45969C06174A for ; Tue, 11 Aug 2020 15:52:35 -0700 (PDT) Received: by mail-pg1-x549.google.com with SMTP id h2so123775pgc.19 for ; Tue, 11 Aug 2020 15:52:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=jwRjL2mpTqfC1FMsfWpfqXN6M+5vuTAImcsjzmY1qOg=; b=sRYtzt/4chinj+e7sN7SKss06ChfANGdUPR3EPDRlkd+Gn1hlzaqUMZkuc0Dw4bKgt ztl9jRku1l/t1IFBJQEnfmkh7GFRWUJGR2TJ7/wvWCojwU5kteEew9hFSexlAZlb3yNu NaN8oZ/IStMU3aHx8EYpzC8eFSQPTdm+keOp+0zrzEY8wNEILUgPywi1dLVyixraC7Jo fzRg1/7qBBGhln+3jGN8/TqXrIv6/3E/lgETpMlOMXYIvjMrILyAstP7UkObyGnrIN3U nx6uidI4UL88TyOVzthvPj+Kbzp0zCV2X+tiPBAm6SIgJuymGWOyCgugPa7ju8LcAl6F Kqlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=jwRjL2mpTqfC1FMsfWpfqXN6M+5vuTAImcsjzmY1qOg=; b=XQ5y8mabhwBsGoqKaY2eselG5VuVsqQsKwyh0jHrpMwE5s0TCsh8/+rbBtFPjrM0Zn taNOSfDTQ/U1r9Q3HsnFxLF/igiBQJPDFvL4Khfq0yj46N7fZKGn+c0CKyMlUCjGobkI MRakJh6FRcCpxWZnckETZEr/y8Ul+I/KhaY3rOB6X/M0uRXEWYSv7mQNyQ1ZDmgDKsA6 48mw5v3Eh63/njzKRdwD2SWTs/4PIZfv/zIR7CBotQ1v/by6qi7W8supQ+SdJnNH91hL foUGtu8/SpgCjfk4E/5AA0mI8bMRw6CTDHevaOZuXFGZLaL+6EdlZye5Ev+4csiV9+mf 034g== X-Gm-Message-State: AOAM532V8yP4DBNmiH2ZvPl0ySlA0J3R/IgkIB+9LxRLzQftwx/rTsDH 6J2Itf6fqj7H0jloLNTIXvM6qZ+rDVfSzZgPPt5jaRg6kRcro3R54AmFKzQ5wAwFxndIRQS9KVF PC1pwyfMCBDVxtoOaPfvCoZTGGl2nuWt2E4KLl/9+6bw+a/BJrkdzwvVFL/1hQR2Px+U5sdG/y8 7F X-Google-Smtp-Source: ABdhPJyXXn01pW2JgPNOzRv/bYzbLVL+BitnVVvaBAmukcDu2LIEfXZu+MowA8UX31EM2zbIjF0YFtEeP2Pu7UZFndXa X-Received: by 2002:a17:90b:3684:: with SMTP id mj4mr3184949pjb.195.1597186354690; Tue, 11 Aug 2020 15:52:34 -0700 (PDT) Date: Tue, 11 Aug 2020 15:52:19 -0700 In-Reply-To: Message-Id: <77bc83e7f2c0a9c95e2ff31aa7a11295bbdf054c.1597184949.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog Subject: [PATCH v2 4/7] fetch: only populate existing_refs if needed From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When fetching tags, Git only writes tags that do not already exist in the client repository. This necessitates an iteration over all the refs, but fetch performs this iteration even if no tags are fetched. This issue is more severe in a partial clone because the iteration over refs also checks that the targets of those refs are present, necessitating a lazy fetch if the target is missing. Therefore, iterate over the refs only when necessary. The user can avoid this iteration by refraining from fetching tags, for example, by passing --no-tags as an argument. A subsequent patch will also teach Git to use "git fetch" to lazy-fetch missing objects in a partial clone, thus also making use of this change. Signed-off-by: Jonathan Tan --- builtin/fetch.c | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/builtin/fetch.c b/builtin/fetch.c index 29db219c68..6460ce3f4e 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -449,6 +449,7 @@ static struct ref *get_ref_map(struct remote *remote, struct ref *orefs = NULL, **oref_tail = &orefs; struct hashmap existing_refs; + int existing_refs_populated = 0; if (rs->nr) { struct refspec *fetch_refspec; @@ -542,15 +543,18 @@ static struct ref *get_ref_map(struct remote *remote, ref_map = ref_remove_duplicates(ref_map); - refname_hash_init(&existing_refs); - for_each_ref(add_one_refname, &existing_refs); - for (rm = ref_map; rm; rm = rm->next) { if (rm->peer_ref) { const char *refname = rm->peer_ref->name; struct refname_hash_entry *peer_item; unsigned int hash = strhash(refname); + if (!existing_refs_populated) { + refname_hash_init(&existing_refs); + for_each_ref(add_one_refname, &existing_refs); + existing_refs_populated = 1; + } + peer_item = hashmap_get_entry_from_hash(&existing_refs, hash, refname, struct refname_hash_entry, ent); @@ -560,7 +564,8 @@ static struct ref *get_ref_map(struct remote *remote, } } } - hashmap_free_entries(&existing_refs, struct refname_hash_entry, ent); + if (existing_refs_populated) + hashmap_free_entries(&existing_refs, struct refname_hash_entry, ent); return ref_map; } From patchwork Tue Aug 11 22:52:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11709787 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2B45813B1 for ; Tue, 11 Aug 2020 22:52:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1073F20782 for ; Tue, 11 Aug 2020 22:52:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="sdoovPh8" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726506AbgHKWwi (ORCPT ); Tue, 11 Aug 2020 18:52:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43832 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726485AbgHKWwh (ORCPT ); Tue, 11 Aug 2020 18:52:37 -0400 Received: from mail-qv1-xf49.google.com (mail-qv1-xf49.google.com [IPv6:2607:f8b0:4864:20::f49]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3423CC06174A for ; Tue, 11 Aug 2020 15:52:37 -0700 (PDT) Received: by mail-qv1-xf49.google.com with SMTP id k17so140114qvj.12 for ; Tue, 11 Aug 2020 15:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=O1Bh6BhVn8SUBJWD0lp1O2C74Y3Om2bQhNlICQdu+dE=; b=sdoovPh8vUdgtsh3USxq5gk90PDmyGmwenOF5Ljtb1BoZ/x0DBFHDFQdHUUhbNpzEd SSZz06ZtUHtouLpL2oNiJ2l5H3oamx5t2GjFVHuHTSugJoKwVTuVoBJ00Ut/x/WsSCzC bioKuKq6YXiB9VXdD90X5YiJSV7tHe6t+H9nem4kLHGmUkSj8l1rxrzjEb6eP94ODpLb TWMzflPyrwPc+zRZbPoKQWkJgtWjVJ3bMQsHYWhvs99ZV2Ioy1ydwwZUTSsUL+CZgS1q Pl+za1hLxedS1h3+5FRwp2QnQ+dhyVWfrpCZBzyjwnJxYAYsaH/1/7X7fOBr0f4ozLpE ctcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=O1Bh6BhVn8SUBJWD0lp1O2C74Y3Om2bQhNlICQdu+dE=; b=FPsXTI29l+gGrvtLoWLpnMKTQ9kVc8K7I3VW4w93RnZrKOHlnSDl2RxdE2MBLGLZif eylPaMvxZv2WjeWbebb7dPbaVxbBxzAGbOo2Wr3abZxGrlmYiCEOZpLINha5ln5eeVnX u/G21tpMdTZtZ5MbPzAgavCW8zNZh1mgVtXG+KuSxn9wZr5T97jE2mDKAam0JyxmbjcM zbBvqtN4BXgl0s7ePDUKz1kqi8ySy4SqeKVSAH1gXZe/zWCT9/y3uDm4WDWC2fnV04mq MujdmSKHPEyn5SCBs/8Lq+T0WGJPWcTu/9rQymU+9iSwDPkzH2lfhE6y0B829fg0RO0E QWeg== X-Gm-Message-State: AOAM531aY78L8JrPY3j+foZEpICYxzT5PQtRn5345uzIIfEVEyOdQYxi gmZKdPY0EgqCfsrZHAKeP0hfUnu+6R0B2PNQVYA6it9bAuhYyv3ccv823cIa1wHL19QkDtkYvFr c6/fFFarP60ihUcRVCwOpfZ9ztTvCm0iYSAi1ubelgyB6z9IsNS6IXw8jf5amIsdy0h2MKZt/0N KF X-Google-Smtp-Source: ABdhPJy4hfzeNr26DUw7qVSjG/B8RPFH9xGrKecJtepWanTHcixbhZ6YGIlclMDFQgvcXc4o7DtSWciauuwNvJ3VB231 X-Received: by 2002:ad4:51d1:: with SMTP id p17mr3772080qvq.14.1597186356146; Tue, 11 Aug 2020 15:52:36 -0700 (PDT) Date: Tue, 11 Aug 2020 15:52:20 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog Subject: [PATCH v2 5/7] fetch-pack: do not lazy-fetch during ref iteration From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Teach "fetch-pack" not to lazy fetch whenever iterating over refs. This is done by using the raw form of ref iteration and by dereferencing tags ourselves. Signed-off-by: Jonathan Tan --- fetch-pack.c | 79 ++++++++++++++++++++++------------------ t/t5616-partial-clone.sh | 20 ++++++++++ 2 files changed, 64 insertions(+), 35 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 80fb3bd899..707bbc31fd 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -108,24 +108,48 @@ static void for_each_cached_alternate(struct fetch_negotiator *negotiator, cb(negotiator, cache.items[i]); } +static struct commit *deref_without_lazy_fetch(const struct object_id *oid, + int mark_tags_complete) +{ + enum object_type type; + struct object_info info = { .typep = &type }; + + while (1) { + if (oid_object_info_extended(the_repository, oid, &info, + OBJECT_INFO_SKIP_FETCH_OBJECT | OBJECT_INFO_QUICK)) + return NULL; + if (type == OBJ_TAG) { + struct tag *tag = (struct tag *) + parse_object(the_repository, oid); + + if (!tag->tagged) + return NULL; + if (mark_tags_complete) + tag->object.flags |= COMPLETE; + oid = &tag->tagged->oid; + } else { + break; + } + } + if (type == OBJ_COMMIT) + return (struct commit *) parse_object(the_repository, oid); + return NULL; +} + static int rev_list_insert_ref(struct fetch_negotiator *negotiator, - const char *refname, const struct object_id *oid) { - struct object *o = deref_tag(the_repository, - parse_object(the_repository, oid), - refname, 0); - - if (o && o->type == OBJ_COMMIT) - negotiator->add_tip(negotiator, (struct commit *)o); + struct commit *c = deref_without_lazy_fetch(oid, 0); + if (c) + negotiator->add_tip(negotiator, c); return 0; } static int rev_list_insert_ref_oid(const char *refname, const struct object_id *oid, int flag, void *cb_data) { - return rev_list_insert_ref(cb_data, refname, oid); + return rev_list_insert_ref(cb_data, oid); } enum ack_type { @@ -201,7 +225,7 @@ static void send_request(struct fetch_pack_args *args, static void insert_one_alternate_object(struct fetch_negotiator *negotiator, struct object *obj) { - rev_list_insert_ref(negotiator, NULL, &obj->oid); + rev_list_insert_ref(negotiator, &obj->oid); } #define INITIAL_FLUSH 16 @@ -230,13 +254,12 @@ static void mark_tips(struct fetch_negotiator *negotiator, int i; if (!negotiation_tips) { - for_each_ref(rev_list_insert_ref_oid, negotiator); + for_each_rawref(rev_list_insert_ref_oid, negotiator); return; } for (i = 0; i < negotiation_tips->nr; i++) - rev_list_insert_ref(negotiator, NULL, - &negotiation_tips->oid[i]); + rev_list_insert_ref(negotiator, &negotiation_tips->oid[i]); return; } @@ -503,21 +526,11 @@ static struct commit_list *complete; static int mark_complete(const struct object_id *oid) { - struct object *o = parse_object(the_repository, oid); - - while (o && o->type == OBJ_TAG) { - struct tag *t = (struct tag *) o; - if (!t->tagged) - break; /* broken repository */ - o->flags |= COMPLETE; - o = parse_object(the_repository, &t->tagged->oid); - } - if (o && o->type == OBJ_COMMIT) { - struct commit *commit = (struct commit *)o; - if (!(commit->object.flags & COMPLETE)) { - commit->object.flags |= COMPLETE; - commit_list_insert(commit, &complete); - } + struct commit *commit = deref_without_lazy_fetch(oid, 1); + + if (commit && !(commit->object.flags & COMPLETE)) { + commit->object.flags |= COMPLETE; + commit_list_insert(commit, &complete); } return 0; } @@ -702,7 +715,7 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator, */ trace2_region_enter("fetch-pack", "mark_complete_local_refs", NULL); if (!args->deepen) { - for_each_ref(mark_complete_oid, NULL); + for_each_rawref(mark_complete_oid, NULL); for_each_cached_alternate(NULL, mark_alternate_complete); commit_list_sort_by_date(&complete); if (cutoff) @@ -716,16 +729,12 @@ static void mark_complete_and_common_ref(struct fetch_negotiator *negotiator, */ trace2_region_enter("fetch-pack", "mark_common_remote_refs", NULL); for (ref = *refs; ref; ref = ref->next) { - struct object *o = deref_tag(the_repository, - lookup_object(the_repository, - &ref->old_oid), - NULL, 0); + struct commit *c = deref_without_lazy_fetch(&ref->old_oid, 0); - if (!o || o->type != OBJ_COMMIT || !(o->flags & COMPLETE)) + if (!c || !(c->object.flags & COMPLETE)) continue; - negotiator->known_common(negotiator, - (struct commit *)o); + negotiator->known_common(negotiator, c); } trace2_region_leave("fetch-pack", "mark_common_remote_refs", NULL); diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 8a27452a51..e53492d595 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -384,6 +384,26 @@ test_expect_success 'fetch lazy-fetches only to resolve deltas, protocol v2' ' grep "want $(cat hash)" trace ' +test_expect_success 'fetch does not lazy-fetch missing targets of its refs' ' + rm -rf server client trace && + + test_create_repo server && + test_config -C server uploadpack.allowfilter 1 && + test_config -C server uploadpack.allowanysha1inwant 1 && + test_commit -C server foo && + + git clone --filter=blob:none "file://$(pwd)/server" client && + # Make all refs point to nothing by deleting all objects. + rm client/.git/objects/pack/* && + + test_commit -C server bar && + GIT_TRACE_PACKET="$(pwd)/trace" git -C client fetch \ + --no-tags --recurse-submodules=no \ + origin refs/tags/bar && + FOO_HASH=$(git -C server rev-parse foo) && + ! grep "want $FOO_HASH" trace +' + # The following two tests must be in this order. It is important that # the srv.bare repository did not have tags during clone, but has tags # in the fetch. From patchwork Tue Aug 11 22:52:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11709789 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 800A713B1 for ; Tue, 11 Aug 2020 22:52:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6474620774 for ; Tue, 11 Aug 2020 22:52:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="dH94yxJD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726518AbgHKWwk (ORCPT ); Tue, 11 Aug 2020 18:52:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43836 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726485AbgHKWwi (ORCPT ); Tue, 11 Aug 2020 18:52:38 -0400 Received: from mail-pj1-x104a.google.com (mail-pj1-x104a.google.com [IPv6:2607:f8b0:4864:20::104a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6A4B9C06174A for ; Tue, 11 Aug 2020 15:52:38 -0700 (PDT) Received: by mail-pj1-x104a.google.com with SMTP id a6so317537pjd.1 for ; Tue, 11 Aug 2020 15:52:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+cRds6fzGxdmuNFa4B3c2k1Y9LNrCIYbxcvjT81setQ=; b=dH94yxJDTe5aoO4de3r613s4D2YEIiyl7vQlQhB8FRzoOeYmqkbfJAlGIltvPn3ZES 3crYLkmZE5vhCbnLKp1WLRPCCQcHZoLD2+kBvZKXr37ltvujIIxRec31AQls5RvixIal tOWgtInKmDgwVtbFA33tEwU3K34IKN/a5N3f2IbjZLnzpHn0cai1gKaPjYkRexLzwICh 2GFkFF9v0nLZbf8FH3SxpJCfugeaJWpwrk68mPU//meEoek3tsrsEkQOa6uBIow/heVT jJka1xQzU/a5CEMwatAYDjYb6wlNkzL5S01XgRxm175CZiTWWoCeNc4Q5thHqJ8H5P3f +JIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+cRds6fzGxdmuNFa4B3c2k1Y9LNrCIYbxcvjT81setQ=; b=UCtImmw6jQpxoCwnV5rvvKZQiA3o7ybSxVMungvC9gb9z8YJKeTdMgkjjkjUV371Pi 8z7Ylvun8PSzEvTglWnzMwMchDQLyxDB3iDWBP7TSGQIhiUUVdxOD/iPy/JoXlUxP8YX 6tEODUz6gB3RHeC6i4RhjVyR1L8zC21AkFIa31xn8QUKdBMehf+xaLUT1lrXpy7iKrgb SY9pwrG3Nt9xrWtPG9Iuq/E2/u5ZV0h1e1q/LaWHP9XfkYPimOfezDmbLaYrcHkLikNj LMyh15raVlWjsEex67jAl1bou6WeQxhW04ZUsQPbp+VAupT/+BgG/fAS5SLmW4OuIzon X++A== X-Gm-Message-State: AOAM531d8gtkO1Vwm7bOZeWwcjVY1OAkvEj9Ew/02VbkxUOCax+9YLlm EfLENCVnLcrpsYLMuyyHqn8G1VKTChjDbAm+W7LrsZErHxiQdKuXbKhAeNx9QQZcBsKDkPHYgYe FTlCVzgaVuOXCKSfmAhwdGI++b6lkNfNv5Tki8HWB+uJYxN2fCGVL6cTN4cFEuuIE9OabmeOL1I K/ X-Google-Smtp-Source: ABdhPJzgO6PUiHwPEq3rl7eNVS5/0P2jBN3vwwiZtOmX/MDNh7tzx8j+EzBsuTtZ/IGU5626WJvypyib6jHViBDwvZcK X-Received: by 2002:a17:902:262:: with SMTP id 89mr2638328plc.31.1597186357740; Tue, 11 Aug 2020 15:52:37 -0700 (PDT) Date: Tue, 11 Aug 2020 15:52:21 -0700 In-Reply-To: Message-Id: <55d2e5a4cccee0ae719f4210c2cbeeb6a691cf2f.1597184949.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog Subject: [PATCH v2 6/7] promisor-remote: lazy-fetch objects in subprocess From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Teach Git to lazy-fetch missing objects in a subprocess instead of doing it in-process. This allows any fatal errors that occur during the fetch to be isolated and converted into an error return value, instead of causing the current command being run to terminate. Signed-off-by: Jonathan Tan --- Documentation/technical/partial-clone.txt | 13 ++----- promisor-remote.c | 46 +++++++++++------------ t/t0410-partial-clone.sh | 2 +- t/t4067-diff-partial-clone.sh | 8 ++-- t/t5601-clone.sh | 2 +- 5 files changed, 30 insertions(+), 41 deletions(-) diff --git a/Documentation/technical/partial-clone.txt b/Documentation/technical/partial-clone.txt index b9e17e7a28..0780d30cac 100644 --- a/Documentation/technical/partial-clone.txt +++ b/Documentation/technical/partial-clone.txt @@ -171,20 +171,13 @@ additional flag. Fetching Missing Objects ------------------------ -- Fetching of objects is done using the existing transport mechanism using - transport_fetch_refs(), setting a new transport option - TRANS_OPT_NO_DEPENDENTS to indicate that only the objects themselves are - desired, not any object that they refer to. -+ -Because some transports invoke fetch_pack() in the same process, fetch_pack() -has been updated to not use any object flags when the corresponding argument -(no_dependents) is set. +- Fetching of objects is done by invoking a "git fetch" subprocess. - The local repository sends a request with the hashes of all requested - objects as "want" lines, and does not perform any packfile negotiation. + objects, and does not perform any packfile negotiation. It then receives a packfile. -- Because we are reusing the existing fetch-pack mechanism, fetching +- Because we are reusing the existing fetch mechanism, fetching currently fetches all objects referred to by the requested objects, even though they are not necessary. diff --git a/promisor-remote.c b/promisor-remote.c index baaea12fd6..6e647610e9 100644 --- a/promisor-remote.c +++ b/promisor-remote.c @@ -3,6 +3,7 @@ #include "promisor-remote.h" #include "config.h" #include "transport.h" +#include "argv-array.h" static char *repository_format_partial_clone; static const char *core_partial_clone_filter_default; @@ -12,39 +13,34 @@ void set_repository_format_partial_clone(char *partial_clone) repository_format_partial_clone = xstrdup_or_null(partial_clone); } -static int fetch_refs(const char *remote_name, struct ref *ref) -{ - struct remote *remote; - struct transport *transport; - int res; - - remote = remote_get(remote_name); - if (!remote->url[0]) - die(_("Remote with no URL")); - transport = transport_get(remote, remote->url[0]); - - transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); - transport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, "1"); - res = transport_fetch_refs(transport, ref); - - return res; -} - static int fetch_objects(const char *remote_name, const struct object_id *oids, int oid_nr) { - struct ref *ref = NULL; + struct child_process child = CHILD_PROCESS_INIT; int i; + FILE *child_in; + + child.git_cmd = 1; + child.in = -1; + argv_array_pushl(&child.args, "-c", "fetch.negotiationAlgorithm=null", + "fetch", remote_name, "--no-tags", + "--no-write-fetch-head", "--recurse-submodules=no", + "--filter=blob:none", "--stdin", NULL); + if (start_command(&child)) + die(_("promisor-remote: unable to fork off fetch subprocess")); + child_in = xfdopen(child.in, "w"); for (i = 0; i < oid_nr; i++) { - struct ref *new_ref = alloc_ref(oid_to_hex(&oids[i])); - oidcpy(&new_ref->old_oid, &oids[i]); - new_ref->exact_oid = 1; - new_ref->next = ref; - ref = new_ref; + if (fputs(oid_to_hex(&oids[i]), child_in) < 0) + die_errno(_("promisor-remote: could not write to fetch subprocess")); + if (fputc('\n', child_in) < 0) + die_errno(_("promisor-remote: could not write to fetch subprocess")); } - return fetch_refs(remote_name, ref); + + if (fclose(child_in) < 0) + die_errno(_("promisor-remote: could not close stdin to fetch subprocess")); + return finish_command(&child) ? -1 : 0; } static struct promisor_remote *promisors; diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 463dc3a8be..3e454f934e 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -203,7 +203,7 @@ test_expect_success 'fetching of missing objects works with ref-in-want enabled' rm -rf repo/.git/objects/* && rm -f trace && GIT_TRACE_PACKET="$(pwd)/trace" git -C repo cat-file -p "$HASH" && - grep "git< fetch=.*ref-in-want" trace + grep "fetch< fetch=.*ref-in-want" trace ' test_expect_success 'fetching of missing objects from another promisor remote' ' diff --git a/t/t4067-diff-partial-clone.sh b/t/t4067-diff-partial-clone.sh index ef8e0e9cb0..804f2a82e8 100755 --- a/t/t4067-diff-partial-clone.sh +++ b/t/t4067-diff-partial-clone.sh @@ -20,7 +20,7 @@ test_expect_success 'git show batches blobs' ' # Ensure that there is exactly 1 negotiation by checking that there is # only 1 "done" line sent. ("done" marks the end of negotiation.) GIT_TRACE_PACKET="$(pwd)/trace" git -C client show HEAD && - grep "git> done" trace >done_lines && + grep "fetch> done" trace >done_lines && test_line_count = 1 done_lines ' @@ -44,7 +44,7 @@ test_expect_success 'diff batches blobs' ' # Ensure that there is exactly 1 negotiation by checking that there is # only 1 "done" line sent. ("done" marks the end of negotiation.) GIT_TRACE_PACKET="$(pwd)/trace" git -C client diff HEAD^ HEAD && - grep "git> done" trace >done_lines && + grep "fetch> done" trace >done_lines && test_line_count = 1 done_lines ' @@ -127,7 +127,7 @@ test_expect_success 'diff with rename detection batches blobs' ' # only 1 "done" line sent. ("done" marks the end of negotiation.) GIT_TRACE_PACKET="$(pwd)/trace" git -C client diff --raw -M HEAD^ HEAD >out && grep ":100644 100644.*R[0-9][0-9][0-9].*b.*c" out && - grep "git> done" trace >done_lines && + grep "fetch> done" trace >done_lines && test_line_count = 1 done_lines ' @@ -175,7 +175,7 @@ test_expect_success 'diff --break-rewrites fetches only if necessary, and batche # by checking that there is only 1 "done" line sent. ("done" marks the # end of negotiation.) GIT_TRACE_PACKET="$(pwd)/trace" git -C client diff --break-rewrites --raw -M HEAD^ HEAD && - grep "git> done" trace >done_lines && + grep "fetch> done" trace >done_lines && test_line_count = 1 done_lines ' diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 84ea2a3eb7..f82b0dbb5a 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -702,7 +702,7 @@ test_expect_success 'batch missing blob request during checkout' ' # Ensure that there is only one negotiation by checking that there is # only "done" line sent. ("done" marks the end of negotiation.) GIT_TRACE_PACKET="$(pwd)/trace" git -C client checkout HEAD^ && - grep "git> done" trace >done_lines && + grep "fetch> done" trace >done_lines && test_line_count = 1 done_lines ' From patchwork Tue Aug 11 22:52:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11709791 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4B0A413B1 for ; Tue, 11 Aug 2020 22:52:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2E5DB20774 for ; Tue, 11 Aug 2020 22:52:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="t8AwfNTW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726523AbgHKWwn (ORCPT ); Tue, 11 Aug 2020 18:52:43 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:43844 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726515AbgHKWwk (ORCPT ); Tue, 11 Aug 2020 18:52:40 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 43865C06174A for ; Tue, 11 Aug 2020 15:52:40 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id k12so321911pfu.19 for ; Tue, 11 Aug 2020 15:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=JLTNJIdPFGHLYtuov9coTWsHN6K98zxqpsaIAmq3saU=; b=t8AwfNTWjVzS9/jV+hYafK11zPDWq9ZE2BrBQu0dCe8M/geU5rrtlKSBvHj6DCMJV4 24Ga4hMS7MG3j6t4NsLcanB7sKQvOSqlmfmpZ0AQos9LXXP34DWlE9VgD0dCaQ/X5Eco FwokfaLvG+v2FGfDpOTFxhu5SdnxChz9+ZE9agBzGsBODA7CcLUsDwao3w/rEfQ9cTOc uknM87YkZxhp2j0sTRRSp6tjnAybY8/aavk5LarLmw6UbfJpBbywZ3DZs8ZyWV/QZV0V bklPlJItl8uZMIJItJUHv+js0KyI2pH0G7jEcGze08b7zH2fyWayuFkxC8Q0VTJkYtss x6Zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=JLTNJIdPFGHLYtuov9coTWsHN6K98zxqpsaIAmq3saU=; b=aVj3hPSIJ5l/EfLUT5D1SlDGdXdNEle8xgzmS7NOpBW2pIapZ5iqUXcpiKMuVRAmOF L4v6Wj8nDtEnZMqSiFeEgKOb0g0RG0pYP+d4c1kWmIHmwOHsOwdYx4VP0MZ/awvXtCBy 444zk6xp4+Vf5eKJ1Dn71QZpIwjcVdcn9jox3/3bwQcru3Q9g90gA+xERmlB7ZvddNKe XIv8XEA+Eix60R190faiXzbQbIerxCQ9FtR+BwXUZBTAo+pnq0ujoPKxenzbYjWKBhsR WcVbnuKYKWbrl6Gm4vddr2apW8WMGv+kVN7/a1xGWGtCG3uovFifGeFY1ShSMqbq8dad 3KUA== X-Gm-Message-State: AOAM530TbLB+OAZG9Ws4ojH+N6VJcE4UBIBtA4j4uMbtdlzzfg5QwN1Y CR4EZW4DMNBNCC44nzw0WbXCvrywK8sjArDJwJc9yis+YhdfydNwcHZ6G5jfZElPq2mxMRNNuUW PkjhPUO1QoHtZ/zGOt9mEG+wCjP09EaTSvjG9oJSeTj0eOzdZ/dmCGyn+E2YA1SugYJI+brs7+W Q3 X-Google-Smtp-Source: ABdhPJzAEF7RedwnN2kjOkhhooFw4X2FHR4TJZWN16xIwcuWmCChAwxvmZO/M7xLEr2YMnwxKM2ViW1XzD89L72sw51C X-Received: by 2002:a17:90b:3684:: with SMTP id mj4mr3185121pjb.195.1597186359421; Tue, 11 Aug 2020 15:52:39 -0700 (PDT) Date: Tue, 11 Aug 2020 15:52:22 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.236.gb10cc79966-goog Subject: [PATCH v2 7/7] fetch-pack: remove no_dependents code From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Now that Git has switched to using a subprocess to lazy-fetch missing objects, remove the no_dependents code as it is no longer used. Signed-off-by: Jonathan Tan --- builtin/fetch-pack.c | 4 -- fetch-pack.c | 110 ++++++++++++------------------------------- fetch-pack.h | 14 ------ remote-curl.c | 6 --- transport.c | 4 -- transport.h | 7 --- 6 files changed, 30 insertions(+), 115 deletions(-) diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c index bbb5c96167..58b7c1fbdc 100644 --- a/builtin/fetch-pack.c +++ b/builtin/fetch-pack.c @@ -153,10 +153,6 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix) args.from_promisor = 1; continue; } - if (!strcmp("--no-dependents", arg)) { - args.no_dependents = 1; - continue; - } if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) { parse_list_objects_filter(&args.filter_options, arg); continue; diff --git a/fetch-pack.c b/fetch-pack.c index 707bbc31fd..3212957dae 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -285,10 +285,8 @@ static int find_common(struct fetch_negotiator *negotiator, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); - if (!args->no_dependents) { - mark_tips(negotiator, args->negotiation_tips); - for_each_cached_alternate(negotiator, insert_one_alternate_object); - } + mark_tips(negotiator, args->negotiation_tips); + for_each_cached_alternate(negotiator, insert_one_alternate_object); fetching = 0; for ( ; refs ; refs = refs->next) { @@ -305,12 +303,8 @@ static int find_common(struct fetch_negotiator *negotiator, * We use lookup_object here because we are only * interested in the case we *know* the object is * reachable and we have already scanned it. - * - * Do this only if args->no_dependents is false (if it is true, - * we cannot trust the object flags). */ - if (!args->no_dependents && - ((o = lookup_object(the_repository, remote)) != NULL) && + if (((o = lookup_object(the_repository, remote)) != NULL) && (o->flags & COMPLETE)) { continue; } @@ -410,8 +404,6 @@ static int find_common(struct fetch_negotiator *negotiator, trace2_region_enter("fetch-pack", "negotiation_v0_v1", the_repository); flushes = 0; retval = -1; - if (args->no_dependents) - goto done; while ((oid = negotiator->next(negotiator))) { packet_buf_write(&req_buf, "have %s\n", oid_to_hex(oid)); print_verbose(args, "have %s", oid_to_hex(oid)); @@ -666,9 +658,7 @@ struct loose_object_iter { /* * Mark recent commits available locally and reachable from a local ref as - * COMPLETE. If args->no_dependents is false, also mark COMPLETE remote refs as - * COMMON_REF (otherwise, we are not planning to participate in negotiation, and - * thus do not need COMMON_REF marks). + * COMPLETE. * * The cutoff time for recency is determined by this heuristic: it is the * earliest commit time of the objects in refs that are commits and that we know @@ -969,12 +959,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, struct fetch_negotiator negotiator_alloc; struct fetch_negotiator *negotiator; - if (args->no_dependents) { - negotiator = NULL; - } else { - negotiator = &negotiator_alloc; - fetch_negotiator_init(r, negotiator); - } + negotiator = &negotiator_alloc; + fetch_negotiator_init(r, negotiator); sort_ref_list(&ref, ref_compare_name); QSORT(sought, nr_sought, cmp_ref_by_name); @@ -1062,15 +1048,11 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args, if (!server_supports_hash(the_hash_algo->name, NULL)) die(_("Server does not support this repository's object format")); - if (!args->no_dependents) { - mark_complete_and_common_ref(negotiator, args, &ref); - filter_refs(args, &ref, sought, nr_sought); - if (everything_local(args, &ref)) { - packet_flush(fd[1]); - goto all_done; - } - } else { - filter_refs(args, &ref, sought, nr_sought); + mark_complete_and_common_ref(negotiator, args, &ref); + filter_refs(args, &ref, sought, nr_sought); + if (everything_local(args, &ref)) { + packet_flush(fd[1]); + goto all_done; } if (find_common(negotiator, args, fd, &oid, ref) < 0) if (!args->keep_pack) @@ -1119,7 +1101,7 @@ static void add_shallow_requests(struct strbuf *req_buf, packet_buf_write(req_buf, "deepen-relative\n"); } -static void add_wants(int no_dependents, const struct ref *wants, struct strbuf *req_buf) +static void add_wants(const struct ref *wants, struct strbuf *req_buf) { int use_ref_in_want = server_supports_feature("fetch", "ref-in-want", 0); @@ -1136,12 +1118,8 @@ static void add_wants(int no_dependents, const struct ref *wants, struct strbuf * We use lookup_object here because we are only * interested in the case we *know* the object is * reachable and we have already scanned it. - * - * Do this only if args->no_dependents is false (if it is true, - * we cannot trust the object flags). */ - if (!no_dependents && - ((o = lookup_object(the_repository, remote)) != NULL) && + if (((o = lookup_object(the_repository, remote)) != NULL) && (o->flags & COMPLETE)) { continue; } @@ -1275,19 +1253,14 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, } /* add wants */ - add_wants(args->no_dependents, wants, &req_buf); + add_wants(wants, &req_buf); - if (args->no_dependents) { - packet_buf_write(&req_buf, "done"); - ret = 1; - } else { - /* Add all of the common commits we've found in previous rounds */ - add_common(&req_buf, common); + /* Add all of the common commits we've found in previous rounds */ + add_common(&req_buf, common); - /* Add initial haves */ - ret = add_haves(negotiator, seen_ack, &req_buf, - haves_to_send, in_vain); - } + /* Add initial haves */ + ret = add_haves(negotiator, seen_ack, &req_buf, + haves_to_send, in_vain); /* Send request */ packet_buf_flush(&req_buf); @@ -1547,12 +1520,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, struct string_list packfile_uris = STRING_LIST_INIT_DUP; int i; - if (args->no_dependents) { - negotiator = NULL; - } else { - negotiator = &negotiator_alloc; - fetch_negotiator_init(r, negotiator); - } + negotiator = &negotiator_alloc; + fetch_negotiator_init(r, negotiator); packet_reader_init(&reader, fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | @@ -1576,21 +1545,16 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, args->deepen = 1; /* Filter 'ref' by 'sought' and those that aren't local */ - if (!args->no_dependents) { - mark_complete_and_common_ref(negotiator, args, &ref); - filter_refs(args, &ref, sought, nr_sought); - if (everything_local(args, &ref)) - state = FETCH_DONE; - else - state = FETCH_SEND_REQUEST; - - mark_tips(negotiator, args->negotiation_tips); - for_each_cached_alternate(negotiator, - insert_one_alternate_object); - } else { - filter_refs(args, &ref, sought, nr_sought); + mark_complete_and_common_ref(negotiator, args, &ref); + filter_refs(args, &ref, sought, nr_sought); + if (everything_local(args, &ref)) + state = FETCH_DONE; + else state = FETCH_SEND_REQUEST; - } + + mark_tips(negotiator, args->negotiation_tips); + for_each_cached_alternate(negotiator, + insert_one_alternate_object); break; case FETCH_SEND_REQUEST: if (!negotiation_started) { @@ -1911,20 +1875,6 @@ struct ref *fetch_pack(struct fetch_pack_args *args, if (nr_sought) nr_sought = remove_duplicates_in_refs(sought, nr_sought); - if (args->no_dependents && !args->filter_options.choice) { - /* - * The protocol does not support requesting that only the - * wanted objects be sent, so approximate this by setting a - * "blob:none" filter if no filter is already set. This works - * for all object types: note that wanted blobs will still be - * sent because they are directly specified as a "want". - * - * NEEDSWORK: Add an option in the protocol to request that - * only the wanted objects be sent, and implement it. - */ - parse_list_objects_filter(&args->filter_options, "blob:none"); - } - if (version != protocol_v2 && !ref) { packet_flush(fd[1]); die(_("no matching remote head")); diff --git a/fetch-pack.h b/fetch-pack.h index 85d1e39fe7..bbe2938059 100644 --- a/fetch-pack.h +++ b/fetch-pack.h @@ -42,20 +42,6 @@ struct fetch_pack_args { unsigned deepen:1; unsigned from_promisor:1; - /* - * Attempt to fetch only the wanted objects, and not any objects - * referred to by them. Due to protocol limitations, extraneous - * objects may still be included. (When fetching non-blob - * objects, only blobs are excluded; when fetching a blob, the - * blob itself will still be sent. The client does not need to - * know whether a wanted object is a blob or not.) - * - * If 1, fetch_pack() will also not modify any object flags. - * This allows fetch_pack() to safely be called by any function, - * regardless of which object flags it uses (if any). - */ - unsigned no_dependents:1; - /* * Because fetch_pack() overwrites the shallow file upon a * successful deepening non-clone fetch, if this struct diff --git a/remote-curl.c b/remote-curl.c index 5cbc6e5002..a0c81a64bc 100644 --- a/remote-curl.c +++ b/remote-curl.c @@ -40,7 +40,6 @@ struct options { push_cert : 2, deepen_relative : 1, from_promisor : 1, - no_dependents : 1, atomic : 1, object_format : 1; const struct git_hash_algo *hash_algo; @@ -186,9 +185,6 @@ static int set_option(const char *name, const char *value) } else if (!strcmp(name, "from-promisor")) { options.from_promisor = 1; return 0; - } else if (!strcmp(name, "no-dependents")) { - options.no_dependents = 1; - return 0; } else if (!strcmp(name, "filter")) { options.filter = xstrdup(value); return 0; @@ -1171,8 +1167,6 @@ static int fetch_git(struct discovery *heads, argv_array_push(&args, "--deepen-relative"); if (options.from_promisor) argv_array_push(&args, "--from-promisor"); - if (options.no_dependents) - argv_array_push(&args, "--no-dependents"); if (options.filter) argv_array_pushf(&args, "--filter=%s", options.filter); argv_array_push(&args, url.buf); diff --git a/transport.c b/transport.c index b41386eccb..32e1f21f0c 100644 --- a/transport.c +++ b/transport.c @@ -232,9 +232,6 @@ static int set_git_option(struct git_transport_options *opts, } else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) { opts->from_promisor = !!value; return 0; - } else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) { - opts->no_dependents = !!value; - return 0; } else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) { list_objects_filter_die_if_populated(&opts->filter_options); parse_list_objects_filter(&opts->filter_options, value); @@ -359,7 +356,6 @@ static int fetch_refs_via_pack(struct transport *transport, args.cloning = transport->cloning; args.update_shallow = data->options.update_shallow; args.from_promisor = data->options.from_promisor; - args.no_dependents = data->options.no_dependents; args.filter_options = data->options.filter_options; args.stateless_rpc = transport->stateless_rpc; args.server_options = transport->server_options; diff --git a/transport.h b/transport.h index b3c30133ea..7aa1f33145 100644 --- a/transport.h +++ b/transport.h @@ -16,7 +16,6 @@ struct git_transport_options { unsigned update_shallow : 1; unsigned deepen_relative : 1; unsigned from_promisor : 1; - unsigned no_dependents : 1; /* * If this transport supports connect or stateless-connect, @@ -201,12 +200,6 @@ void transport_check_allowed(const char *type); /* Indicate that these objects are being fetched by a promisor */ #define TRANS_OPT_FROM_PROMISOR "from-promisor" -/* - * Indicate that only the objects wanted need to be fetched, not their - * dependents - */ -#define TRANS_OPT_NO_DEPENDENTS "no-dependents" - /* Filter objects for partial clone and fetch */ #define TRANS_OPT_LIST_OBJECTS_FILTER "filter"