From patchwork Fri Jul 24 22:38:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11684663 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E84E413B4 for ; Fri, 24 Jul 2020 22:38:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C8297206EB for ; Fri, 24 Jul 2020 22:38:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="gvd8VvkX" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726652AbgGXWiu (ORCPT ); Fri, 24 Jul 2020 18:38:50 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726154AbgGXWit (ORCPT ); Fri, 24 Jul 2020 18:38:49 -0400 Received: from mail-pl1-x649.google.com (mail-pl1-x649.google.com [IPv6:2607:f8b0:4864:20::649]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30664C0619D3 for ; Fri, 24 Jul 2020 15:38:49 -0700 (PDT) Received: by mail-pl1-x649.google.com with SMTP id ba2so6417870plb.0 for ; Fri, 24 Jul 2020 15:38:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=oGGmANiWYRz2cfeqYz1QWzU9UB5XnVP0RJTEch6uElQ=; b=gvd8VvkXpB4MAipBgXsxQvFUnnA9TcxeWBKmgnnB51HbgEeflfP+O/84fE2/LMle4w hJh1tf6QhRCkMgRcpI0HILsohov0jCpLNW9TGv8/WxecXAXf8Zb3NnF3b6PWwpayc4kT Yfd21a+NgQrAGZxPiYE/kTwSTIBb7MdNlwCSYcDfENykrVcoCUIMS+ykgMlY2+r2TLoe ijq188gqseKuvN6VkYQguqjJ3MyT8t3/SKeU/3awcm1GNY/XVNTDFE084R6UjpPsEU5q Zoj7wnEFJevSAkQTOHz2O1H4Rd7LYg1HffS/+6eHMHVzZA85EDmdRMYgGrmGMpFAduy+ 3H2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=oGGmANiWYRz2cfeqYz1QWzU9UB5XnVP0RJTEch6uElQ=; b=Jp6vac6BtgyhdjpeXwu1eJuOZ/IzGRvRDtPSPEPAsmDEyMdFr5ERpoQ+NSeS6y5NNL 4v+xj3P1LSuPnlJqCJAeWx82+06YgJJ619cxAieUd+Oe1/HcpzxyNbtMYp67lBrnvKDq 9BJiDq50sWSvJX0SFxaiBQBSFNtQ3nLvx5DZzbYgSPIe6/DON14zoEpmmPSNyJ5uUGDs Bcje+3UzlcgFMv1Dsu+4FJsAuSHOk84qpTR4bifiuyQnS2V/yZ+49J8RRuqV7/oqBpI/ CNLjDF/9segUdJzUKlyczJJ7wEOyHChWnhupOaJo2jYBPTn+SqzqsrSDNQhT5jTOtUXe FnNw== X-Gm-Message-State: AOAM530JFQNG1P5PYR7kroS2qJzrpHWrzndiBeelk2iDzqsnqAn9Ajy6 D/uX0+kkak0yean6AJhqiP0HF6XUgUD+iZ7x/jlydIKj2us6T+mXhfWVChFZxa0RVgI2MhRo1Z4 b/CdWDtnmSVmCHT2ugaitMAwm4M6/kWqeYPlE3DDbSivd+D2n1QP1zFOCaZgVa7mxVkmLZE+A88 AF X-Google-Smtp-Source: ABdhPJzTcdYayUxSubzQdWlECRAVxbEaKt79h3d7CbsZxQIhlwXeZxhc/g3WXcuosm7unlSkpC2/N/0i8a1laZT2WwYp X-Received: by 2002:a62:7650:: with SMTP id r77mr10588213pfc.235.1595630328435; Fri, 24 Jul 2020 15:38:48 -0700 (PDT) Date: Fri, 24 Jul 2020 15:38:44 -0700 Message-Id: <20200724223844.2723397-1-jonathantanmy@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.28.0.rc0.142.g3c755180ce-goog Subject: [RFC PATCH] Modify fetch-pack to no longer die on error? From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org We've had a few instances where a lazy fetch in a partial clone fails, leading to a fatal error, when the calling code could have easily recovered - in other words, the severity of the bug should have just a wasted fetch instead of stopping the whole command. Part of the issue (and possibly the whole issue - I haven't looked at this beyond fetch-pack yet) is that fetch-pack dies whenever it encounters an error, so I took a look at fixing that. (Note that fetch-pack is sometimes run through a remote helper, meaning that we could leave the die() invocations in and just make sure that we handle failure in the separate process correctly. But when the promisor remote is HTTP protocol v2 or SSH protocol v0/v2, this is not true - fetch_pack() is run in-process.) I think the best way for easy authorship and review is to convert each possibly-dying function into a function that either returns a possibly-null error string or returns success/failure and writes the error string into an "out" parameter. In this way, the change is rather mechanical and should be easy to review. In the patch below I chose the former approach, and I modified 2 functions (one that returns no value and one that returns a value) to demonstrate what it would look like. We could also take this further and have a "struct error" for type safety and macros - e.g. THROW() to return a "struct error", TRY() to execute what's inside the parentheses and return the error if there is one, and OR_DIE() to execute what's inside the parentheses and die if there is an error. Any opinions before I continue working on this? Signed-off-by: Jonathan Tan --- fetch-pack.c | 78 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 53 insertions(+), 25 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 80fb3bd899..20a7e05ea8 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -24,6 +24,8 @@ #include "fsck.h" #include "shallow.h" +typedef char * error_string; + static int transfer_unpack_limit = -1; static int fetch_unpack_limit = -1; static int unpack_limit = 100; @@ -136,8 +138,8 @@ enum ack_type { ACK_ready }; -static void consume_shallow_list(struct fetch_pack_args *args, - struct packet_reader *reader) +static error_string consume_shallow_list(struct fetch_pack_args *args, + struct packet_reader *reader) { if (args->stateless_rpc && args->deepen) { /* If we sent a depth we will get back "duplicate" @@ -149,41 +151,54 @@ static void consume_shallow_list(struct fetch_pack_args *args, continue; if (starts_with(reader->line, "unshallow ")) continue; - die(_("git fetch-pack: expected shallow list")); + return xstrdup(_("git fetch-pack: expected shallow list")); } if (reader->status != PACKET_READ_FLUSH) - die(_("git fetch-pack: expected a flush packet after shallow list")); + return xstrdup(_("git fetch-pack: expected a flush packet after shallow list")); } + return NULL; } -static enum ack_type get_ack(struct packet_reader *reader, - struct object_id *result_oid) +static error_string get_ack(struct packet_reader *reader, + enum ack_type *result_ack, + struct object_id *result_oid) { int len; const char *arg; if (packet_reader_read(reader) != PACKET_READ_NORMAL) - die(_("git fetch-pack: expected ACK/NAK, got a flush packet")); + return xstrdup(_("git fetch-pack: expected ACK/NAK, got a flush packet")); len = reader->pktlen; - if (!strcmp(reader->line, "NAK")) - return NAK; + if (!strcmp(reader->line, "NAK")) { + *result_ack = NAK; + return NULL; + } if (skip_prefix(reader->line, "ACK ", &arg)) { const char *p; if (!parse_oid_hex(arg, result_oid, &p)) { len -= p - reader->line; - if (len < 1) - return ACK; - if (strstr(p, "continue")) - return ACK_continue; - if (strstr(p, "common")) - return ACK_common; - if (strstr(p, "ready")) - return ACK_ready; - return ACK; + if (len < 1) { + *result_ack = ACK; + return NULL; + } + if (strstr(p, "continue")) { + *result_ack = ACK_continue; + return NULL; + } + if (strstr(p, "common")) { + *result_ack = ACK_common; + return NULL; + } + if (strstr(p, "ready")) { + *result_ack = ACK_ready; + return NULL; + } + *result_ack = ACK; + return NULL; } } - die(_("git fetch-pack: expected ACK/NAK, got '%s'"), reader->line); + return xstrfmt(_("git fetch-pack: expected ACK/NAK, got '%s'"), reader->line); } static void send_request(struct fetch_pack_args *args, @@ -394,7 +409,8 @@ static int find_common(struct fetch_negotiator *negotiator, print_verbose(args, "have %s", oid_to_hex(oid)); in_vain++; if (flush_at <= ++count) { - int ack; + enum ack_type ack; + error_string err; packet_buf_flush(&req_buf); send_request(args, fd[1], &req_buf); @@ -409,9 +425,11 @@ static int find_common(struct fetch_negotiator *negotiator, if (!args->stateless_rpc && count == INITIAL_FLUSH) continue; - consume_shallow_list(args, &reader); + if ((err = consume_shallow_list(args, &reader))) + die("%s", err); do { - ack = get_ack(&reader, result_oid); + if ((err = get_ack(&reader, &ack, result_oid))) + die("%s", err); if (ack) print_verbose(args, _("got %s %d %s"), "ack", ack, oid_to_hex(result_oid)); @@ -457,6 +475,9 @@ static int find_common(struct fetch_negotiator *negotiator, got_ready = 1; break; } + case NAK: + /* nothing */ + break; } } while (ack); flushes--; @@ -481,10 +502,17 @@ static int find_common(struct fetch_negotiator *negotiator, } strbuf_release(&req_buf); - if (!got_ready || !no_done) - consume_shallow_list(args, &reader); + if (!got_ready || !no_done) { + error_string err; + if ((err = consume_shallow_list(args, &reader))) + die("%s", err); + } while (flushes || multi_ack) { - int ack = get_ack(&reader, result_oid); + error_string err; + enum ack_type ack; + + if ((err = get_ack(&reader, &ack, result_oid))) + die("%s", err); if (ack) { print_verbose(args, _("got %s (%d) %s"), "ack", ack, oid_to_hex(result_oid)); From patchwork Tue Aug 18 04:01:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 11719845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 805F41744 for ; Tue, 18 Aug 2020 04:02:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 58D922072A for ; Tue, 18 Aug 2020 04:02:16 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="n4V6FTLD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726306AbgHREBz (ORCPT ); Tue, 18 Aug 2020 00:01:55 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:37710 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725554AbgHREBy (ORCPT ); Tue, 18 Aug 2020 00:01:54 -0400 Received: from mail-pf1-x449.google.com (mail-pf1-x449.google.com [IPv6:2607:f8b0:4864:20::449]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 14CF8C061389 for ; Mon, 17 Aug 2020 21:01:54 -0700 (PDT) Received: by mail-pf1-x449.google.com with SMTP id 4so12044658pfd.23 for ; Mon, 17 Aug 2020 21:01:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=KMR/f/lp74dQGVAR9jgtjWRJABwbGdpjUZiaHz1Lz3s=; b=n4V6FTLDWlxhMvNy4eu2M02H47UALIXfKTy5tSxKFJI9DkUpwaRUqylNgHimwCvfla 1F14eyUvIW/SUKVvPlFQHF94UFBBU9BLO6uG9H9rmaYGKfqW/8NKP21GwTPCtAlq4NU3 BcJ7g5Hl62gPFhK5vh4CD32zbpt5Mjd6hM/h6xSRTNhxhPzoUHxceA7Ttp8k9ZfA2quU 39LVMzMFU6BvFIsTTY6/zFpXCrzFHwGf/NwFNDom7Mxv1TFk62hvkgGF6sVdvQUZWZ1A 0BJW0oWi65jUngO+BmTmTJacOVLVqURK4vH4YBf/2UnGBgC3sosS+b+vmU3FKic88r/J oLsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=KMR/f/lp74dQGVAR9jgtjWRJABwbGdpjUZiaHz1Lz3s=; b=Am4LDVs4uFuxXufrWjd2LCq02NeB18ud4rPuilGAK2uLc/XoU3BnUb1PspuDmTRzUW 4z+0nkWXa3FTu8ELA3tUIzVL49GdzahxcuiSrq2hDsEPDBvADWlGCMeHLm9i9gR6OlBT 3aCin9vgghj4YTBw2WyDxV4eMQ4LH5hQm7SfwiAZUeAMt11I3/ZP4fME/ZSSDEa8YVhe JRbh48uo+RJxCdpZq3n1Brm/POECx+TD9InQvv99jv2e0bMwWPFupc4l/a3phuTIi/yE RHfmsnY+v4hNFtsoyJrNM6iTKod5kv6HorYX177Y6YDK1K//aNsXcbRJpJvT7d3uQ7ha NARg== X-Gm-Message-State: AOAM530HtkHJvW+/qHMyWh3KrTtsiWcXnWuX1XjRrTBQ6POfNhGBHC0s M9gQiOGa63SPk2TNrOAPEQfkAkxmrWIxMvOdCavGf8TIOoxehmhLc/cgeyfSj5yeAeTNP1VOry/ KcL8+VeSTIQxcKUjncEbqBCGRbxUDKG9LrF9wHlN3bsEDdTByaGvdt2Kl0W8E6+m30T9i1t/6Au 7Y X-Google-Smtp-Source: ABdhPJzGlmvgGneVpiAty2O1NXjAlWlVcYkPRivSRAwNL1Pgum/D9NYZGyXBOLAkQIht6rk0EJw9C8oN1XmdoR45qbkR X-Received: by 2002:a17:90a:7485:: with SMTP id p5mr9120979pjk.130.1597723313379; Mon, 17 Aug 2020 21:01:53 -0700 (PDT) Date: Mon, 17 Aug 2020 21:01:32 -0700 In-Reply-To: Message-Id: <9f277f1631a37785dcd72a552dd86738d639ea59.1597722942.git.jonathantanmy@google.com> Mime-Version: 1.0 References: <20200724223844.2723397-1-jonathantanmy@google.com> X-Mailer: git-send-email 2.28.0.220.ged08abb693-goog Subject: [PATCH v3 2/7] fetch: allow refspecs specified through stdin From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , stolee@gmail.com, gitster@pobox.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In a subsequent patch, partial clones will be taught to fetch missing objects using a "git fetch" subprocess. Because the number of objects fetched may be too numerous to fit on the command line, teach "fetch" to accept refspecs passed through stdin. Signed-off-by: Jonathan Tan --- Documentation/git-fetch.txt | 4 ++++ builtin/fetch.c | 19 +++++++++++++++++-- 2 files changed, 21 insertions(+), 2 deletions(-) diff --git a/Documentation/git-fetch.txt b/Documentation/git-fetch.txt index 45b6d8e633..9067c2079e 100644 --- a/Documentation/git-fetch.txt +++ b/Documentation/git-fetch.txt @@ -48,6 +48,10 @@ include::fetch-options.txt[] include::pull-fetch-param.txt[] +--stdin:: + Read refspecs, one per line, from stdin in addition to those provided + as arguments. The "tag " format is not supported. + include::urls-remotes.txt[] diff --git a/builtin/fetch.c b/builtin/fetch.c index 3ccf69753f..a5498646bf 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -80,6 +80,7 @@ static struct list_objects_filter_options filter_options; static struct string_list server_options = STRING_LIST_INIT_DUP; static struct string_list negotiation_tip = STRING_LIST_INIT_NODUP; static int fetch_write_commit_graph = -1; +static int stdin_refspecs = 0; static int git_fetch_config(const char *k, const char *v, void *cb) { @@ -209,6 +210,8 @@ static struct option builtin_fetch_options[] = { N_("check for forced-updates on all updated branches")), OPT_BOOL(0, "write-commit-graph", &fetch_write_commit_graph, N_("write the commit-graph after fetching")), + OPT_BOOL(0, "stdin", &stdin_refspecs, + N_("accept refspecs from stdin")), OPT_END() }; @@ -1684,7 +1687,8 @@ static inline void fetch_one_setup_partial(struct remote *remote) return; } -static int fetch_one(struct remote *remote, int argc, const char **argv, int prune_tags_ok) +static int fetch_one(struct remote *remote, int argc, const char **argv, + int prune_tags_ok, int use_stdin_refspecs) { struct refspec rs = REFSPEC_INIT_FETCH; int i; @@ -1741,6 +1745,13 @@ static int fetch_one(struct remote *remote, int argc, const char **argv, int pru } } + if (use_stdin_refspecs) { + struct strbuf line = STRBUF_INIT; + while (strbuf_getline_lf(&line, stdin) != EOF) + refspec_append(&rs, line.buf); + strbuf_release(&line); + } + if (server_options.nr) gtransport->server_options = &server_options; @@ -1841,7 +1852,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) if (remote) { if (filter_options.choice || has_promisor_remote()) fetch_one_setup_partial(remote); - result = fetch_one(remote, argc, argv, prune_tags_ok); + result = fetch_one(remote, argc, argv, prune_tags_ok, stdin_refspecs); } else { int max_children = max_jobs; @@ -1849,6 +1860,10 @@ int cmd_fetch(int argc, const char **argv, const char *prefix) die(_("--filter can only be used with the remote " "configured in extensions.partialclone")); + if (stdin_refspecs) + die(_("--stdin can only be used when fetching " + "from one remote")); + if (max_children < 0) max_children = fetch_parallel_config;