From patchwork Fri Jan 11 22:18:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760743 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1D8D13BF for ; Fri, 11 Jan 2019 22:18:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8FCF72A0D5 for ; Fri, 11 Jan 2019 22:18:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 82C792A0E2; Fri, 11 Jan 2019 22:18:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 61DBC2A0D5 for ; Fri, 11 Jan 2019 22:18:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726410AbfAKWSk (ORCPT ); Fri, 11 Jan 2019 17:18:40 -0500 Received: from mail-pf1-f201.google.com ([209.85.210.201]:34400 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWSj (ORCPT ); Fri, 11 Jan 2019 17:18:39 -0500 Received: by mail-pf1-f201.google.com with SMTP id u20so11386684pfa.1 for ; Fri, 11 Jan 2019 14:18:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=6+0q0gQ23/SFGeoicC5zoMmtHFtn2A0fuBIL3tZ4MLU=; b=MqzeoYqbswfCTpZnRylkkwNtcpVyX55nuJTqyCI2Xo/sWIlNJzfknK3OTpD/D+u72J gPpAiCFKg0Ps4MZHYiyk4jL7ETPTBk5I3VDuEZh7sKVdjgv6OaEM4oMxijFfbviYVqPa Nx95j1Ti/Kp1FSXO3u7rklPOldEcmacON5jL+JuSwAkbr89Z9kXZ/vx+iq1DkzYAbKUv MxjQ43Rc0wTUrDKcpKxyiLCTULzvEhgf2P7kOrBKwtGOS1oVPouSdfi84ogOWMRXvlvM J9LdQbOxjZNYP5pJy0DCCeygaOh4weq6geJVEHPTg03VrvdXwzySZkiKzyCksAsxVfDF DFyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6+0q0gQ23/SFGeoicC5zoMmtHFtn2A0fuBIL3tZ4MLU=; b=OGKFKET4+MD9OdU9jHrNRajTTGcElz/001/atqcs1wqYRukD/Xcj8fSuprPKBFynX4 UovDq6v+F6CuCRVMsiilJRQgwuieCyhy8UOmvhZBk2uPvJVlX3be+zgVJxNhlO6Y6LfN 4RJWx9X9USYONWkzahGs/oJXlUP2JTJHlo/chrFqWPs2vL4HTN6imuabXhqQmn32gbck +o1yZ8/Spc2HWTQw0CbHOzWZj89R6TU8caPqCgRlwYfxBDNiQvku2ExDKpMP02N5YslN fACqBt3Y7fjQ+zvbcNIDrCi4TApSfd62D6i3ffLyaRckCSa6GzL6YHxShLdRTF/6XclN meDQ== X-Gm-Message-State: AJcUukc4QIC37Z+tZTHng4e8cM2v1WSauuiFK7v47me4dJu7zvuJR2Dn fsvezYMIZhFA4L19cUg64QLcuKc+hfP7BPRxynS6kaqjDj4QndXpLLp8MLqHpKmulCw1KOruqFy g1JJd8+c7eSi7dxkjTUa1ZprwtoeXc7b3j6Cs4GYxu0R00trXVv30AJIBAcn1q01h9D00bv6bqo Wk X-Google-Smtp-Source: ALg8bN5x1n4fxN0VcpqR8/9qjhiFDI/oumi8G/qWrB6uCnim/Oi1G80elxbO3pFYJLB+cCjSbguGxyCri8QOfOLk1jSb X-Received: by 2002:a65:5c07:: with SMTP id u7mr1138034pgr.36.1547245118334; Fri, 11 Jan 2019 14:18:38 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:14 -0800 In-Reply-To: Message-Id: <9b37e68e2f0d83ea133f6662efb780466c518170.1547244620.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH 1/4] pkt-line: introduce struct packet_writer From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It will be convenient for a future patch if writing options (specifically, whether the written data is to be multiplexed) could be controlled from a single place, so create struct packet_writer to serve as that place, and modify upload-pack to use it. Currently, it only stores the output fd, but a subsequent patch will (as described above) introduce an option to determine if the written data is to be multiplexed. Signed-off-by: Jonathan Tan --- pkt-line.c | 47 ++++++++++++++++++--- pkt-line.h | 14 +++++++ upload-pack.c | 112 +++++++++++++++++++++++++++----------------------- 3 files changed, 115 insertions(+), 58 deletions(-) diff --git a/pkt-line.c b/pkt-line.c index e70ea6d88f..9d3e402d41 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -129,12 +129,14 @@ static void set_packet_header(char *buf, const int size) #undef hex } -static void format_packet(struct strbuf *out, const char *fmt, va_list args) +static void format_packet(struct strbuf *out, const char *prefix, + const char *fmt, va_list args) { size_t orig_len, n; orig_len = out->len; strbuf_addstr(out, "0000"); + strbuf_addstr(out, prefix); strbuf_vaddf(out, fmt, args); n = out->len - orig_len; @@ -145,13 +147,13 @@ static void format_packet(struct strbuf *out, const char *fmt, va_list args) packet_trace(out->buf + orig_len + 4, n - 4, 1); } -static int packet_write_fmt_1(int fd, int gently, +static int packet_write_fmt_1(int fd, int gently, const char *prefix, const char *fmt, va_list args) { static struct strbuf buf = STRBUF_INIT; strbuf_reset(&buf); - format_packet(&buf, fmt, args); + format_packet(&buf, prefix, fmt, args); if (write_in_full(fd, buf.buf, buf.len) < 0) { if (!gently) { check_pipe(errno); @@ -168,7 +170,7 @@ void packet_write_fmt(int fd, const char *fmt, ...) va_list args; va_start(args, fmt); - packet_write_fmt_1(fd, 0, fmt, args); + packet_write_fmt_1(fd, 0, "", fmt, args); va_end(args); } @@ -178,7 +180,7 @@ int packet_write_fmt_gently(int fd, const char *fmt, ...) va_list args; va_start(args, fmt); - status = packet_write_fmt_1(fd, 1, fmt, args); + status = packet_write_fmt_1(fd, 1, "", fmt, args); va_end(args); return status; } @@ -211,7 +213,7 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...) va_list args; va_start(args, fmt); - format_packet(buf, fmt, args); + format_packet(buf, "", fmt, args); va_end(args); } @@ -486,3 +488,36 @@ enum packet_read_status packet_reader_peek(struct packet_reader *reader) reader->line_peeked = 1; return reader->status; } + +void packet_writer_init(struct packet_writer *writer, int dest_fd) +{ + writer->dest_fd = dest_fd; +} + +void packet_writer_write(struct packet_writer *writer, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + packet_write_fmt_1(writer->dest_fd, 0, "", fmt, args); + va_end(args); +} + +void packet_writer_error(struct packet_writer *writer, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + packet_write_fmt_1(writer->dest_fd, 0, "ERR ", fmt, args); + va_end(args); +} + +void packet_writer_delim(struct packet_writer *writer) +{ + packet_delim(writer->dest_fd); +} + +void packet_writer_flush(struct packet_writer *writer) +{ + packet_flush(writer->dest_fd); +} diff --git a/pkt-line.h b/pkt-line.h index d7e1dbc047..023ad2951d 100644 --- a/pkt-line.h +++ b/pkt-line.h @@ -183,4 +183,18 @@ extern enum packet_read_status packet_reader_peek(struct packet_reader *reader); #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4) extern char packet_buffer[LARGE_PACKET_MAX]; +struct packet_writer { + int dest_fd; +}; + +void packet_writer_init(struct packet_writer *writer, int dest_fd); + +/* These functions die upon failure. */ +__attribute__((format (printf, 2, 3))) +void packet_writer_write(struct packet_writer *writer, const char *fmt, ...); +__attribute__((format (printf, 2, 3))) +void packet_writer_error(struct packet_writer *writer, const char *fmt, ...); +void packet_writer_delim(struct packet_writer *writer); +void packet_writer_flush(struct packet_writer *writer); + #endif diff --git a/upload-pack.c b/upload-pack.c index 08b547cf01..60a26bbbfd 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -613,13 +613,14 @@ static void check_non_tip(struct object_array *want_obj) } } -static void send_shallow(struct commit_list *result) +static void send_shallow(struct packet_writer *writer, + struct commit_list *result) { while (result) { struct object *object = &result->item->object; if (!(object->flags & (CLIENT_SHALLOW|NOT_SHALLOW))) { - packet_write_fmt(1, "shallow %s", - oid_to_hex(&object->oid)); + packet_writer_write(writer, "shallow %s", + oid_to_hex(&object->oid)); register_shallow(the_repository, &object->oid); shallow_nr++; } @@ -627,7 +628,8 @@ static void send_shallow(struct commit_list *result) } } -static void send_unshallow(const struct object_array *shallows, +static void send_unshallow(struct packet_writer *writer, + const struct object_array *shallows, struct object_array *want_obj) { int i; @@ -636,8 +638,8 @@ static void send_unshallow(const struct object_array *shallows, struct object *object = shallows->objects[i].item; if (object->flags & NOT_SHALLOW) { struct commit_list *parents; - packet_write_fmt(1, "unshallow %s", - oid_to_hex(&object->oid)); + packet_writer_write(writer, "unshallow %s", + oid_to_hex(&object->oid)); object->flags &= ~CLIENT_SHALLOW; /* * We want to _register_ "object" as shallow, but we @@ -662,7 +664,7 @@ static void send_unshallow(const struct object_array *shallows, } } -static void deepen(int depth, int deepen_relative, +static void deepen(struct packet_writer *writer, int depth, int deepen_relative, struct object_array *shallows, struct object_array *want_obj) { if (depth == INFINITE_DEPTH && !is_repository_shallow(the_repository)) { @@ -680,7 +682,7 @@ static void deepen(int depth, int deepen_relative, result = get_shallow_commits(&reachable_shallows, depth + 1, SHALLOW, NOT_SHALLOW); - send_shallow(result); + send_shallow(writer, result); free_commit_list(result); object_array_clear(&reachable_shallows); } else { @@ -688,14 +690,15 @@ static void deepen(int depth, int deepen_relative, result = get_shallow_commits(want_obj, depth, SHALLOW, NOT_SHALLOW); - send_shallow(result); + send_shallow(writer, result); free_commit_list(result); } - send_unshallow(shallows, want_obj); + send_unshallow(writer, shallows, want_obj); } -static void deepen_by_rev_list(int ac, const char **av, +static void deepen_by_rev_list(struct packet_writer *writer, int ac, + const char **av, struct object_array *shallows, struct object_array *want_obj) { @@ -703,13 +706,14 @@ static void deepen_by_rev_list(int ac, const char **av, close_commit_graph(the_repository); result = get_shallow_commits_by_rev_list(ac, av, SHALLOW, NOT_SHALLOW); - send_shallow(result); + send_shallow(writer, result); free_commit_list(result); - send_unshallow(shallows, want_obj); + send_unshallow(writer, shallows, want_obj); } /* Returns 1 if a shallow list is sent or 0 otherwise */ -static int send_shallow_list(int depth, int deepen_rev_list, +static int send_shallow_list(struct packet_writer *writer, + int depth, int deepen_rev_list, timestamp_t deepen_since, struct string_list *deepen_not, struct object_array *shallows, @@ -720,7 +724,7 @@ static int send_shallow_list(int depth, int deepen_rev_list, if (depth > 0 && deepen_rev_list) die("git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together"); if (depth > 0) { - deepen(depth, deepen_relative, shallows, want_obj); + deepen(writer, depth, deepen_relative, shallows, want_obj); ret = 1; } else if (deepen_rev_list) { struct argv_array av = ARGV_ARRAY_INIT; @@ -741,7 +745,7 @@ static int send_shallow_list(int depth, int deepen_rev_list, struct object *o = want_obj->objects[i].item; argv_array_push(&av, oid_to_hex(&o->oid)); } - deepen_by_rev_list(av.argc, av.argv, shallows, want_obj); + deepen_by_rev_list(writer, av.argc, av.argv, shallows, want_obj); argv_array_clear(&av); ret = 1; } else { @@ -834,8 +838,10 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan int has_non_tip = 0; timestamp_t deepen_since = 0; int deepen_rev_list = 0; + struct packet_writer writer; shallow_nr = 0; + packet_writer_init(&writer, 1); for (;;) { struct object *o; const char *features; @@ -892,9 +898,9 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan o = parse_object(the_repository, &oid_buf); if (!o) { - packet_write_fmt(1, - "ERR upload-pack: not our ref %s", - oid_to_hex(&oid_buf)); + packet_writer_error(&writer, + "upload-pack: not our ref %s", + oid_to_hex(&oid_buf)); die("git upload-pack: not our ref %s", oid_to_hex(&oid_buf)); } @@ -923,7 +929,7 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan if (depth == 0 && !deepen_rev_list && shallows.nr == 0) return; - if (send_shallow_list(depth, deepen_rev_list, deepen_since, + if (send_shallow_list(&writer, depth, deepen_rev_list, deepen_since, &deepen_not, &shallows, want_obj)) packet_flush(1); object_array_clear(&shallows); @@ -1102,6 +1108,8 @@ struct upload_pack_data { int deepen_rev_list; int deepen_relative; + struct packet_writer writer; + unsigned stateless_rpc : 1; unsigned use_thin_pack : 1; @@ -1125,6 +1133,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->haves = haves; data->shallows = shallows; data->deepen_not = deepen_not; + packet_writer_init(&data->writer, 1); } static void upload_pack_data_clear(struct upload_pack_data *data) @@ -1136,7 +1145,8 @@ static void upload_pack_data_clear(struct upload_pack_data *data) string_list_clear(&data->deepen_not, 0); } -static int parse_want(const char *line, struct object_array *want_obj) +static int parse_want(struct packet_writer *writer, const char *line, + struct object_array *want_obj) { const char *arg; if (skip_prefix(line, "want ", &arg)) { @@ -1149,9 +1159,9 @@ static int parse_want(const char *line, struct object_array *want_obj) o = parse_object(the_repository, &oid); if (!o) { - packet_write_fmt(1, - "ERR upload-pack: not our ref %s", - oid_to_hex(&oid)); + packet_writer_error(writer, + "upload-pack: not our ref %s", + oid_to_hex(&oid)); die("git upload-pack: not our ref %s", oid_to_hex(&oid)); } @@ -1167,7 +1177,8 @@ static int parse_want(const char *line, struct object_array *want_obj) return 0; } -static int parse_want_ref(const char *line, struct string_list *wanted_refs, +static int parse_want_ref(struct packet_writer *writer, const char *line, + struct string_list *wanted_refs, struct object_array *want_obj) { const char *arg; @@ -1177,7 +1188,7 @@ static int parse_want_ref(const char *line, struct string_list *wanted_refs, struct object *o; if (read_ref(arg, &oid)) { - packet_write_fmt(1, "ERR unknown ref %s", arg); + packet_writer_error(writer, "unknown ref %s", arg); die("unknown ref %s", arg); } @@ -1220,10 +1231,11 @@ static void process_args(struct packet_reader *request, const char *p; /* process want */ - if (parse_want(arg, want_obj)) + if (parse_want(&data->writer, arg, want_obj)) continue; if (allow_ref_in_want && - parse_want_ref(arg, &data->wanted_refs, want_obj)) + parse_want_ref(&data->writer, arg, &data->wanted_refs, + want_obj)) continue; /* process have line */ if (parse_have(arg, &data->haves)) @@ -1317,26 +1329,26 @@ static int process_haves(struct oid_array *haves, struct oid_array *common, return 0; } -static int send_acks(struct oid_array *acks, struct strbuf *response, +static int send_acks(struct packet_writer *writer, struct oid_array *acks, const struct object_array *have_obj, struct object_array *want_obj) { int i; - packet_buf_write(response, "acknowledgments\n"); + packet_writer_write(writer, "acknowledgments\n"); /* Send Acks */ if (!acks->nr) - packet_buf_write(response, "NAK\n"); + packet_writer_write(writer, "NAK\n"); for (i = 0; i < acks->nr; i++) { - packet_buf_write(response, "ACK %s\n", - oid_to_hex(&acks->oid[i])); + packet_writer_write(writer, "ACK %s\n", + oid_to_hex(&acks->oid[i])); } if (ok_to_give_up(have_obj, want_obj)) { /* Send Ready */ - packet_buf_write(response, "ready\n"); + packet_writer_write(writer, "ready\n"); return 1; } @@ -1348,25 +1360,20 @@ static int process_haves_and_send_acks(struct upload_pack_data *data, struct object_array *want_obj) { struct oid_array common = OID_ARRAY_INIT; - struct strbuf response = STRBUF_INIT; int ret = 0; process_haves(&data->haves, &common, have_obj); if (data->done) { ret = 1; - } else if (send_acks(&common, &response, have_obj, want_obj)) { - packet_buf_delim(&response); + } else if (send_acks(&data->writer, &common, have_obj, want_obj)) { + packet_writer_delim(&data->writer); ret = 1; } else { /* Add Flush */ - packet_buf_flush(&response); + packet_writer_flush(&data->writer); ret = 0; } - /* Send response */ - write_or_die(1, response.buf, response.len); - strbuf_release(&response); - oid_array_clear(&data->haves); oid_array_clear(&common); return ret; @@ -1379,15 +1386,15 @@ static void send_wanted_ref_info(struct upload_pack_data *data) if (!data->wanted_refs.nr) return; - packet_write_fmt(1, "wanted-refs\n"); + packet_writer_write(&data->writer, "wanted-refs\n"); for_each_string_list_item(item, &data->wanted_refs) { - packet_write_fmt(1, "%s %s\n", - oid_to_hex(item->util), - item->string); + packet_writer_write(&data->writer, "%s %s\n", + oid_to_hex(item->util), + item->string); } - packet_delim(1); + packet_writer_delim(&data->writer); } static void send_shallow_info(struct upload_pack_data *data, @@ -1398,14 +1405,15 @@ static void send_shallow_info(struct upload_pack_data *data, !is_repository_shallow(the_repository)) return; - packet_write_fmt(1, "shallow-info\n"); + packet_writer_write(&data->writer, "shallow-info\n"); - if (!send_shallow_list(data->depth, data->deepen_rev_list, + if (!send_shallow_list(&data->writer, data->depth, + data->deepen_rev_list, data->deepen_since, &data->deepen_not, &data->shallows, want_obj) && is_repository_shallow(the_repository)) - deepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows, - want_obj); + deepen(&data->writer, INFINITE_DEPTH, data->deepen_relative, + &data->shallows, want_obj); packet_delim(1); } @@ -1467,7 +1475,7 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys, send_wanted_ref_info(&data); send_shallow_info(&data, &want_obj); - packet_write_fmt(1, "packfile\n"); + packet_writer_write(&data.writer, "packfile\n"); create_pack_file(&have_obj, &want_obj); state = FETCH_DONE; break; From patchwork Fri Jan 11 22:18:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760745 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8DD9891E for ; Fri, 11 Jan 2019 22:18:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7D3F82A0CB for ; Fri, 11 Jan 2019 22:18:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 71A902A0D8; Fri, 11 Jan 2019 22:18:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B622C2A0CB for ; Fri, 11 Jan 2019 22:18:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726451AbfAKWSm (ORCPT ); Fri, 11 Jan 2019 17:18:42 -0500 Received: from mail-qt1-f202.google.com ([209.85.160.202]:40465 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWSm (ORCPT ); Fri, 11 Jan 2019 17:18:42 -0500 Received: by mail-qt1-f202.google.com with SMTP id 42so17792556qtr.7 for ; Fri, 11 Jan 2019 14:18:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Sv1aZi3YF+NGNzBUA3ciALmHzEru+YxmhOo45rDzMTw=; b=jBpvSFtO4Ohhj/cA4KXcyNa9J52TH6VHDCok8jIfz0LG921vklyPBLc6LJJ/kkNSgu 0lXXQ7ldByHiQtXPtgF06acvs4cC8V/4X880v6KtwkZkkxGgsTNlWs7wjC/byKb/wv7I 6yZikqHwctGD0IEI5T+DuaK3RKhfYuLI95GqdjwYi3sWl7zIWYPVj/olj5zMf2A22khU pI8MiW871/Pn3vmsUeEudl/F3SpTR/AA+61duV4A/oqjU31Hm/ka0HpooXyYWdCjus6Z HizrRShQk6zDZTfWPxpXXsY6czkVFrUe1kHZDHSXOe9oWk6vrbYLWzrL8f+hIsPggjUW CkWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Sv1aZi3YF+NGNzBUA3ciALmHzEru+YxmhOo45rDzMTw=; b=WSzsso0cLX9QBSjBAutcA//gR5+RhGIbB96T91TG1wCOhQnsvwSjdEJcWiDUnztcgk WRgdmB++oICRLzqYC33BO6tgG3tgQe8rrVIVlfPXBBfoHwkKNs31oxCptmacvk2CmLSD r1ptdNPj0RbMA3o2YSiINMoGZHgCPHmZpaH3nGC4s/kTYJ5625/zVcBzVmFJM6VdmEl4 +O3GzC1meBoOmUqovfyplXPz4ugl7BlDqe7ptsh/jKd/9Oulg78hPRkg1C486Y5zaRZX Uo37DkAULoKA0bFE3TC61OjUg5Jz+wmqfUxqGir2juGtFk44qqnJBO6ucocT4qU8PI54 VXvQ== X-Gm-Message-State: AJcUukcR8IdFKrA9NhqYRuwqBUeWL/GUkk7YwFQWq0nAckT7sxjigTUM 71hHZL7nw3G+X32qi39vfcRlLmidsNtkjq/OA7CZ1RTQ+IAZyTSVCML9EflMvkSOs3Puw8/fwSn MSN80Fl7cXvA/zsmHtbMf9izQx/JP6l2VNVwjyQ55g4tIfdf/Pa9B3yQY3+wS4RAWHKincPbNvm Wp X-Google-Smtp-Source: ALg8bN4a/Uo8fBP3hjd+HIj3OGIVWX97T3pbWeIt1U0DkawwkNXIA71eIq4Hpg0cslMex01giaU3Cap5jbGLeM909o3J X-Received: by 2002:ac8:3364:: with SMTP id u33mr1528939qta.56.1547245120964; Fri, 11 Jan 2019 14:18:40 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:15 -0800 In-Reply-To: Message-Id: <4079a1f110086bd412fb4f56bc8b7f8a6a7a8f9f.1547244620.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH 2/4] sideband: reverse its dependency on pkt-line From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A subsequent patch will teach struct packet_reader a new field that, if set, instructs it to interpret read data as multiplexed. This will create a dependency from pkt-line to sideband. To avoid a circular dependency, split recv_sideband() into 2 parts: the reading loop (left in recv_sideband()) and the processing of the contents (in diagnose_sideband()), and move the former into pkt-line. This reverses the direction of dependency: sideband no longer depends on pkt-line, and pkt-line now depends on sideband. Signed-off-by: Jonathan Tan --- pkt-line.c | 22 ++++++++ pkt-line.h | 16 ++++++ sideband.c | 156 +++++++++++++++++++++++++---------------------------- sideband.h | 15 +++++- 4 files changed, 125 insertions(+), 84 deletions(-) diff --git a/pkt-line.c b/pkt-line.c index 9d3e402d41..ebdc6c2530 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -439,6 +439,28 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out) return sb_out->len - orig_len; } +int recv_sideband(const char *me, int in_stream, int out) +{ + char buf[LARGE_PACKET_MAX + 1]; + int retval = 0; + int len; + + while (1) { + len = packet_read(in_stream, NULL, NULL, buf, LARGE_PACKET_MAX, 0); + retval = diagnose_sideband(me, buf, len); + switch (retval) { + case SIDEBAND_PRIMARY: + write_or_die(out, buf + 1, len - 1); + break; + case SIDEBAND_PROGRESS: + /* already written by diagnose_sideband() */ + break; + default: /* flush or error */ + return retval; + } + } +} + /* Packet Reader Functions */ void packet_reader_init(struct packet_reader *reader, int fd, char *src_buffer, size_t src_len, diff --git a/pkt-line.h b/pkt-line.h index 023ad2951d..a8e92a4b63 100644 --- a/pkt-line.h +++ b/pkt-line.h @@ -3,6 +3,7 @@ #include "git-compat-util.h" #include "strbuf.h" +#include "sideband.h" /* * Write a packetized stream, where each line is preceded by @@ -120,6 +121,21 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size); */ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out); +/* + * Receive multiplexed output stream over git native protocol. + * in_stream is the input stream from the remote, which carries data + * in pkt_line format with band designator. Demultiplex it into out + * and err and return error appropriately. Band #1 carries the + * primary payload. Things coming over band #2 is not necessarily + * error; they are usually informative message on the standard error + * stream, aka "verbose"). A message over band #3 is a signal that + * the remote died unexpectedly. A flush() concludes the stream. + * + * Returns SIDEBAND_FLUSH upon a normal conclusion, and SIDEBAND_PROTOCOL_ERROR + * or SIDEBAND_REMOTE_ERROR if an error occurred. + */ +int recv_sideband(const char *me, int in_stream, int out); + struct packet_reader { /* source file descriptor */ int fd; diff --git a/sideband.c b/sideband.c index 368647acf8..842a92e975 100644 --- a/sideband.c +++ b/sideband.c @@ -1,7 +1,6 @@ #include "cache.h" #include "color.h" #include "config.h" -#include "pkt-line.h" #include "sideband.h" #include "help.h" @@ -109,103 +108,94 @@ static void maybe_colorize_sideband(struct strbuf *dest, const char *src, int n) } -/* - * Receive multiplexed output stream over git native protocol. - * in_stream is the input stream from the remote, which carries data - * in pkt_line format with band designator. Demultiplex it into out - * and err and return error appropriately. Band #1 carries the - * primary payload. Things coming over band #2 is not necessarily - * error; they are usually informative message on the standard error - * stream, aka "verbose"). A message over band #3 is a signal that - * the remote died unexpectedly. A flush() concludes the stream. - */ - #define DISPLAY_PREFIX "remote: " #define ANSI_SUFFIX "\033[K" #define DUMB_SUFFIX " " -int recv_sideband(const char *me, int in_stream, int out) +int diagnose_sideband(const char *me, char *buf, int len) { - const char *suffix; - char buf[LARGE_PACKET_MAX + 1]; + static const char *suffix; struct strbuf outbuf = STRBUF_INIT; int retval = 0; + const char *b, *brk; + int band; + + if (!suffix) { + if (isatty(2) && !is_terminal_dumb()) + suffix = ANSI_SUFFIX; + else + suffix = DUMB_SUFFIX; + } - if (isatty(2) && !is_terminal_dumb()) - suffix = ANSI_SUFFIX; - else - suffix = DUMB_SUFFIX; + if (len == 0) { + retval = SIDEBAND_FLUSH; + goto cleanup; + } + if (len < 1) { + strbuf_addf(&outbuf, + "%s%s: protocol error: no band designator", + outbuf.len ? "\n" : "", me); + retval = SIDEBAND_PROTOCOL_ERROR; + goto cleanup; + } + band = buf[0] & 0xff; + buf[len] = '\0'; + len--; + switch (band) { + case 3: + strbuf_addf(&outbuf, "%s%s", outbuf.len ? "\n" : "", + DISPLAY_PREFIX); + maybe_colorize_sideband(&outbuf, buf + 1, len); + + retval = SIDEBAND_REMOTE_ERROR; + break; + case 2: + b = buf + 1; - while (!retval) { - const char *b, *brk; - int band, len; - len = packet_read(in_stream, NULL, NULL, buf, LARGE_PACKET_MAX, 0); - if (len == 0) - break; - if (len < 1) { - strbuf_addf(&outbuf, - "%s%s: protocol error: no band designator", - outbuf.len ? "\n" : "", me); - retval = SIDEBAND_PROTOCOL_ERROR; - break; - } - band = buf[0] & 0xff; - buf[len] = '\0'; - len--; - switch (band) { - case 3: - strbuf_addf(&outbuf, "%s%s", outbuf.len ? "\n" : "", - DISPLAY_PREFIX); - maybe_colorize_sideband(&outbuf, buf + 1, len); - - retval = SIDEBAND_REMOTE_ERROR; - break; - case 2: - b = buf + 1; - - /* - * Append a suffix to each nonempty line to clear the - * end of the screen line. - * - * The output is accumulated in a buffer and - * each line is printed to stderr using - * write(2) to ensure inter-process atomicity. - */ - while ((brk = strpbrk(b, "\n\r"))) { - int linelen = brk - b; - - if (!outbuf.len) - strbuf_addstr(&outbuf, DISPLAY_PREFIX); - if (linelen > 0) { - maybe_colorize_sideband(&outbuf, b, linelen); - strbuf_addstr(&outbuf, suffix); - } - - strbuf_addch(&outbuf, *brk); - xwrite(2, outbuf.buf, outbuf.len); - strbuf_reset(&outbuf); - - b = brk + 1; + /* + * Append a suffix to each nonempty line to clear the + * end of the screen line. + * + * The output is accumulated in a buffer and + * each line is printed to stderr using + * write(2) to ensure inter-process atomicity. + */ + while ((brk = strpbrk(b, "\n\r"))) { + int linelen = brk - b; + + if (!outbuf.len) + strbuf_addstr(&outbuf, DISPLAY_PREFIX); + if (linelen > 0) { + maybe_colorize_sideband(&outbuf, b, linelen); + strbuf_addstr(&outbuf, suffix); } - if (*b) { - strbuf_addstr(&outbuf, outbuf.len ? - "" : DISPLAY_PREFIX); - maybe_colorize_sideband(&outbuf, b, strlen(b)); - } - break; - case 1: - write_or_die(out, buf + 1, len); - break; - default: - strbuf_addf(&outbuf, "%s%s: protocol error: bad band #%d", - outbuf.len ? "\n" : "", me, band); - retval = SIDEBAND_PROTOCOL_ERROR; - break; + strbuf_addch(&outbuf, *brk); + xwrite(2, outbuf.buf, outbuf.len); + strbuf_reset(&outbuf); + + b = brk + 1; + } + + if (*b) { + strbuf_addstr(&outbuf, outbuf.len ? + "" : DISPLAY_PREFIX); + maybe_colorize_sideband(&outbuf, b, strlen(b)); } + retval = SIDEBAND_PROGRESS; + break; + case 1: + retval = SIDEBAND_PRIMARY; + break; + default: + strbuf_addf(&outbuf, "%s%s: protocol error: bad band #%d", + outbuf.len ? "\n" : "", me, band); + retval = SIDEBAND_PROTOCOL_ERROR; + break; } +cleanup: if (outbuf.len) { strbuf_addch(&outbuf, '\n'); xwrite(2, outbuf.buf, outbuf.len); diff --git a/sideband.h b/sideband.h index 7a8146f161..a56cd86287 100644 --- a/sideband.h +++ b/sideband.h @@ -3,8 +3,21 @@ #define SIDEBAND_PROTOCOL_ERROR -2 #define SIDEBAND_REMOTE_ERROR -1 +#define SIDEBAND_FLUSH 0 +#define SIDEBAND_PRIMARY 1 +#define SIDEBAND_PROGRESS 2 + +/* + * buf and len should be the result of reading a line from a remote sending + * multiplexed data. + * + * Determines the nature of the result and returns it. If + * SIDEBAND_PROTOCOL_ERROR, SIDEBAND_REMOTE_ERROR, or SIDEBAND_PROGRESS, also + * prints a message (or the formatted contents of the notice in the case of + * SIDEBAND_PROGRESS) to stderr. + */ +int diagnose_sideband(const char *me, char *buf, int len); -int recv_sideband(const char *me, int in_stream, int out); void send_sideband(int fd, int band, const char *data, ssize_t sz, int packet_max); #endif From patchwork Fri Jan 11 22:18:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760747 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F97B13BF for ; Fri, 11 Jan 2019 22:18:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1B62C29BFF for ; Fri, 11 Jan 2019 22:18:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 086B329C1B; Fri, 11 Jan 2019 22:18:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 510DE2A0CB for ; Fri, 11 Jan 2019 22:18:46 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726475AbfAKWSp (ORCPT ); Fri, 11 Jan 2019 17:18:45 -0500 Received: from mail-qk1-f201.google.com ([209.85.222.201]:48237 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWSp (ORCPT ); Fri, 11 Jan 2019 17:18:45 -0500 Received: by mail-qk1-f201.google.com with SMTP id p79so10737392qki.15 for ; Fri, 11 Jan 2019 14:18:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=n4H8iGSWXpqzX+fstyFb2cmBl2ExCJnOpQet/+NtrPw=; b=naVOF7iyDHb6OUOhWZM3LA8SZez3idGbowrt1oooqb1jOuyuhg83+1OEg7UH+NyWto SPgSlksU4DIB9ItqeX5Si7R5bgTe9K44lPH32vokCMZQnUWLJfyFAkRVJ2s2OKtAXqfv iMug5/2sMKg6ogNrauReQklKO2+bW09o3CdJUk6erxE53rFUhlybqy8P/Qd8i31lMkYa b6NRq/B2ctfIhfuz8YBp+ifh5ZBSZM0uwIvtv2URIIUQIz8VGElZNUCK5jKvjedsO/Ks Iiz6mQrAtQ+Nl5xLHmW1cq2OXFjRjIb97QjjyOL9/Yp1++KX4RTquIRCRQpv9IhS4nq5 Vcbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=n4H8iGSWXpqzX+fstyFb2cmBl2ExCJnOpQet/+NtrPw=; b=DfuOfTrmbHh+Xn5HukudsS/zinzh9c+d7BR3WPLhvuHeymGLUYgr7dQ4r4Dc4AG9hV 695J5zKbUCApCkVl/HTpEpnuK0qMVH4uWB4fGwKYSwC3i/MF4WMixhNM98QU/ecwyZ0E mZaYYHW6225g8UwILkXeSMw8BG52xjmM3ERJ/Un+ZJWyiaS7rDfYx6/MvsftCetnE/YW b/GS0kiE6QlZ2MxvHQvFa6q9cYQCkb64OThw6MuDv5zFwbXQggPKCNULmScK21pf/8gl Xa+U2fc5FShNW6q4a7A2sLWgoB3/e24bMYwNI7/cmUqvon040FUo5jigL3l+bPSKJrok ooZA== X-Gm-Message-State: AJcUukettm+BKl4rM033UU5GJvSw46+/Z5Lq4LYZZbQ+Gw0b2y59yFoK FH7khunl1IoHgff5TolgjuA5qna2Xd7vEo4l3looycS5NnBN0BvQZFW4ubLViZAf2hVlziwqBaT 4pl4UkluSmfwf1SojpQJm4c5rKpgUHesAKzBinFoFRQIm3m4fIDQzdNA9BetLtfEEDrCgDyoXRy 84 X-Google-Smtp-Source: ALg8bN4NHtbzmYR9OwIlwdgCrGwHn8oCBLrioUlVPEYTPrCLo6nNOOceLY0SUu+vKTZMCqzxWiO2UyJ87d/MID/f89Q+ X-Received: by 2002:ac8:225d:: with SMTP id p29mr1562484qtp.48.1547245123468; Fri, 11 Jan 2019 14:18:43 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:16 -0800 In-Reply-To: Message-Id: <673b928b4158b0aecca843302483b48da5ed8d35.1547244620.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH 3/4] {fetch,upload}-pack: sideband v2 fetch response From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, a response to a fetch request has sideband support only while the packfile is being sent, meaning that the server cannot send notices until the start of the packfile. Extend sideband support in protocol v2 fetch responses to the whole response. upload-pack will advertise it if the uploadpack.allowsidebandall configuration variable is set, and fetch-pack will automatically request it if advertised. If the sideband is to be used throughout the whole response, upload-pack will use it to send errors instead of prefixing a PKT-LINE payload with "ERR ". This will be tested in a subsequent patch. Signed-off-by: Jonathan Tan --- Documentation/technical/protocol-v2.txt | 10 +++++ fetch-pack.c | 12 +++++- pkt-line.c | 51 +++++++++++++++++++------ pkt-line.h | 4 ++ sideband.c | 7 +++- sideband.h | 2 +- upload-pack.c | 16 ++++++++ 7 files changed, 87 insertions(+), 15 deletions(-) diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 09e4e0273f..1b0633f59f 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -307,6 +307,16 @@ the 'wanted-refs' section in the server's response as explained below. particular ref, where is the full name of a ref on the server. +If the 'sideband-all' feature is advertised, the following argument can be +included in the client's request: + + sideband-all + Instruct the server to send the whole response multiplexed, not just + the packfile section. All non-flush and non-delim PKT-LINE in the + response (not only in the packfile section) will then start with a byte + indicating its sideband (1, 2, or 3), and the server may send "0005\1" + (a PKT-LINE of sideband 1 with no payload) as a keepalive packet. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. diff --git a/fetch-pack.c b/fetch-pack.c index 3f9626dbf7..b89199891d 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1125,7 +1125,8 @@ static int add_haves(struct fetch_negotiator *negotiator, static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, const struct fetch_pack_args *args, const struct ref *wants, struct oidset *common, - int *haves_to_send, int *in_vain) + int *haves_to_send, int *in_vain, + int sideband_all) { int ret = 0; struct strbuf req_buf = STRBUF_INIT; @@ -1151,6 +1152,8 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, packet_buf_write(&req_buf, "include-tag"); if (prefer_ofs_delta) packet_buf_write(&req_buf, "ofs-delta"); + if (sideband_all) + packet_buf_write(&req_buf, "sideband-all"); /* Add shallow-info and deepen request */ if (server_supports_feature("fetch", "shallow", 0)) @@ -1359,6 +1362,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packet_reader_init(&reader, fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); + if (server_supports_feature("fetch", "sideband-all", 0)) { + reader.use_sideband = 1; + reader.me = "fetch-pack"; + } while (state != FETCH_DONE) { switch (state) { @@ -1392,7 +1399,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, case FETCH_SEND_REQUEST: if (send_fetch_request(&negotiator, fd[1], args, ref, &common, - &haves_to_send, &in_vain)) + &haves_to_send, &in_vain, + reader.use_sideband)) state = FETCH_GET_PACK; else state = FETCH_PROCESS_ACKS; diff --git a/pkt-line.c b/pkt-line.c index ebdc6c2530..71b17e6b93 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -447,7 +447,7 @@ int recv_sideband(const char *me, int in_stream, int out) while (1) { len = packet_read(in_stream, NULL, NULL, buf, LARGE_PACKET_MAX, 0); - retval = diagnose_sideband(me, buf, len); + retval = diagnose_sideband(me, buf, len, 0); switch (retval) { case SIDEBAND_PRIMARY: write_or_die(out, buf + 1, len - 1); @@ -483,16 +483,42 @@ enum packet_read_status packet_reader_read(struct packet_reader *reader) return reader->status; } - reader->status = packet_read_with_status(reader->fd, - &reader->src_buffer, - &reader->src_len, - reader->buffer, - reader->buffer_size, - &reader->pktlen, - reader->options); + while (1) { + int retval; + reader->status = packet_read_with_status(reader->fd, + &reader->src_buffer, + &reader->src_len, + reader->buffer, + reader->buffer_size, + &reader->pktlen, + reader->options); + if (!reader->use_sideband) + break; + retval = diagnose_sideband(reader->me, reader->buffer, + reader->pktlen, 1); + switch (retval) { + case SIDEBAND_PROTOCOL_ERROR: + case SIDEBAND_REMOTE_ERROR: + BUG("should have died in diagnose_sideband"); + case SIDEBAND_FLUSH: + goto nonprogress; + case SIDEBAND_PRIMARY: + if (reader->pktlen != 1) + goto nonprogress; + /* + * Since pktlen is 1, this is a keepalive + * packet. Wait for the next one. + */ + break; + default: /* SIDEBAND_PROGRESS */ + ; + } + } +nonprogress: if (reader->status == PACKET_READ_NORMAL) - reader->line = reader->buffer; + reader->line = reader->use_sideband ? + reader->buffer + 1 : reader->buffer; else reader->line = NULL; @@ -514,6 +540,7 @@ enum packet_read_status packet_reader_peek(struct packet_reader *reader) void packet_writer_init(struct packet_writer *writer, int dest_fd) { writer->dest_fd = dest_fd; + writer->use_sideband = 0; } void packet_writer_write(struct packet_writer *writer, const char *fmt, ...) @@ -521,7 +548,8 @@ void packet_writer_write(struct packet_writer *writer, const char *fmt, ...) va_list args; va_start(args, fmt); - packet_write_fmt_1(writer->dest_fd, 0, "", fmt, args); + packet_write_fmt_1(writer->dest_fd, 0, + writer->use_sideband ? "\1" : "", fmt, args); va_end(args); } @@ -530,7 +558,8 @@ void packet_writer_error(struct packet_writer *writer, const char *fmt, ...) va_list args; va_start(args, fmt); - packet_write_fmt_1(writer->dest_fd, 0, "ERR ", fmt, args); + packet_write_fmt_1(writer->dest_fd, 0, + writer->use_sideband ? "\3" : "ERR ", fmt, args); va_end(args); } diff --git a/pkt-line.h b/pkt-line.h index a8e92a4b63..ad9a4a2cd7 100644 --- a/pkt-line.h +++ b/pkt-line.h @@ -162,6 +162,9 @@ struct packet_reader { /* indicates if a line has been peeked */ int line_peeked; + + unsigned use_sideband : 1; + const char *me; }; /* @@ -201,6 +204,7 @@ extern char packet_buffer[LARGE_PACKET_MAX]; struct packet_writer { int dest_fd; + unsigned use_sideband : 1; }; void packet_writer_init(struct packet_writer *writer, int dest_fd); diff --git a/sideband.c b/sideband.c index 842a92e975..cbeab380ae 100644 --- a/sideband.c +++ b/sideband.c @@ -113,7 +113,7 @@ static void maybe_colorize_sideband(struct strbuf *dest, const char *src, int n) #define ANSI_SUFFIX "\033[K" #define DUMB_SUFFIX " " -int diagnose_sideband(const char *me, char *buf, int len) +int diagnose_sideband(const char *me, char *buf, int len, int die_on_error) { static const char *suffix; struct strbuf outbuf = STRBUF_INIT; @@ -144,6 +144,9 @@ int diagnose_sideband(const char *me, char *buf, int len) len--; switch (band) { case 3: + if (die_on_error) + die("remote error: %s", buf + 1); + strbuf_addf(&outbuf, "%s%s", outbuf.len ? "\n" : "", DISPLAY_PREFIX); maybe_colorize_sideband(&outbuf, buf + 1, len); @@ -196,6 +199,8 @@ int diagnose_sideband(const char *me, char *buf, int len) } cleanup: + if (die_on_error && retval == SIDEBAND_PROTOCOL_ERROR) + die("%s", outbuf.buf); if (outbuf.len) { strbuf_addch(&outbuf, '\n'); xwrite(2, outbuf.buf, outbuf.len); diff --git a/sideband.h b/sideband.h index a56cd86287..3786670694 100644 --- a/sideband.h +++ b/sideband.h @@ -16,7 +16,7 @@ * prints a message (or the formatted contents of the notice in the case of * SIDEBAND_PROGRESS) to stderr. */ -int diagnose_sideband(const char *me, char *buf, int len); +int diagnose_sideband(const char *me, char *buf, int len, int die_on_error); void send_sideband(int fd, int band, const char *data, ssize_t sz, int packet_max); diff --git a/upload-pack.c b/upload-pack.c index 60a26bbbfd..765b7695d2 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -71,6 +71,8 @@ static int allow_filter; static int allow_ref_in_want; static struct list_objects_filter_options filter_options; +static int allow_sideband_all; + static void reset_timeout(void) { alarm(timeout); @@ -1046,6 +1048,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused) allow_filter = git_config_bool(var, value); } else if (!strcmp("uploadpack.allowrefinwant", var)) { allow_ref_in_want = git_config_bool(var, value); + } else if (!strcmp("uploadpack.allowsidebandall", var)) { + allow_sideband_all = git_config_bool(var, value); } if (current_config_scope() != CONFIG_SCOPE_REPO) { @@ -1284,6 +1288,11 @@ static void process_args(struct packet_reader *request, continue; } + if (allow_sideband_all && !strcmp(arg, "sideband-all")) { + data->writer.use_sideband = 1; + continue; + } + /* ignore unknown lines maybe? */ die("unexpected line: '%s'", arg); } @@ -1496,6 +1505,7 @@ int upload_pack_advertise(struct repository *r, if (value) { int allow_filter_value; int allow_ref_in_want; + int allow_sideband_all_value; strbuf_addstr(value, "shallow"); @@ -1510,6 +1520,12 @@ int upload_pack_advertise(struct repository *r, &allow_ref_in_want) && allow_ref_in_want) strbuf_addstr(value, " ref-in-want"); + + if (!repo_config_get_bool(the_repository, + "uploadpack.allowsidebandall", + &allow_sideband_all_value) && + allow_sideband_all_value) + strbuf_addstr(value, " sideband-all"); } return 1; From patchwork Fri Jan 11 22:18:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760751 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6492391E for ; Fri, 11 Jan 2019 22:18:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5383228A40 for ; Fri, 11 Jan 2019 22:18:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 47BCB29BFF; Fri, 11 Jan 2019 22:18:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3FD2D28A40 for ; Fri, 11 Jan 2019 22:18:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726488AbfAKWSr (ORCPT ); Fri, 11 Jan 2019 17:18:47 -0500 Received: from mail-pl1-f201.google.com ([209.85.214.201]:53257 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWSq (ORCPT ); Fri, 11 Jan 2019 17:18:46 -0500 Received: by mail-pl1-f201.google.com with SMTP id ay11so9077864plb.20 for ; Fri, 11 Jan 2019 14:18:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=fQrZtSR2ysfXRBWArswhh+utOdeCl3lzeOx/B9vvsCk=; b=UWQYoRZ3IA0YzuosgCdYv37CpNl8Up3Pl8sgTaXrOrVkNtTXF8hM8sD4geik1qgHJH 4Nh9xMsHOXGVuoxbCpTxnKQrKLm9DlpRp6Rs0iEhSNl03QhiFeJRvymy8AS+54brsQnF ngtBO8OMvd87cqWLkmRfn0IIwwfsroz+6GHtGLSOWoA29kvNY7sMpGg0XKbWxyQEWCtC gJKUpgQSoYuWfaJz7c9coGTrYFeHAnr7VByBiZgdItxBhamLf0A4kSdOO1mZNWFGpGUA Q67RvdNkZzBoKEy2Sl/wmZ4eSVBPoZDlS7ccS2NTCKXxgNaoSdG8q8NMJAN2129Lt/OL Pr+g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=fQrZtSR2ysfXRBWArswhh+utOdeCl3lzeOx/B9vvsCk=; b=abwUr2FUsA4UdbyU+qJO7Kd1k7/QH8RE0Pg1ZjhmDdWH6YxtzpTL4bT/1JsK2AnzmB kB/HviMKx85/D8SZTV9LMVuqYW0Hj3VkDQZuxIK9RUvJGLDgFTBuuXd9xuFtU+iO2RjV MdFOhh/5bLjyHJOAAJOhT0Bf/nqk1GYupn5BspCofBW+nOgSVc2dx7RBNlZvKzmuUT3H 7SdzXGKuAQcKVVkHHKIGzqjogdDq7TgpOzQ/0Sj4DFDiIZBA7UR/tCBm/d8lFMq+b85R JUQA1PkzqJ9jAAIiS4yGqLDeGzs9unc7iriPUhBKe3dJPO6wue/f1TsCVhAwIuxmzAck y+OA== X-Gm-Message-State: AJcUukfvQBn/SXgkB14iDyNcyxkb1RazHW4CfdNKyr8KeAyYULs6/+Io wgBqa0frkmBVpJU9CKVNN9bJswQ+w951EO1WOMhhkgqrn4MMpgUpjJ9JzOVhT6Wzk/tQf616tiW 6COL5SPGnbM42xEcX3DOKJwbEHKEY/7iGgFRcFx0dQB/kJrXOO+EvMkDGpmH/aveJzN9D0JhgYF v6 X-Google-Smtp-Source: ALg8bN4XHS55iypf9aqiSPGqAedzrkkVLIIPaWl6wP+rSWVfM9rLkA7gacTMNGF6MO2Ave2Jn9JM3kAno+b7NYEC8Dx+ X-Received: by 2002:a17:902:8e88:: with SMTP id bg8mr843896plb.110.1547245125643; Fri, 11 Jan 2019 14:18:45 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:17 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH 4/4] tests: define GIT_TEST_SIDEBAND_ALL From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Define a GIT_TEST_SIDEBAND_ALL environment variable meant to be used from tests. When set to true, this overrides uploadpack.allowsidebandall to true, allowing the entire test suite to be run as if this configuration is in place for all repositories. As of this patch, all tests pass whether GIT_TEST_SIDEBAND_ALL is unset or set to 1. Signed-off-by: Jonathan Tan --- fetch-pack.c | 3 ++- t/README | 5 +++++ t/lib-httpd/apache.conf | 1 + t/t5537-fetch-shallow.sh | 3 ++- t/t5701-git-serve.sh | 2 +- t/t5702-protocol-v2.sh | 4 ++-- upload-pack.c | 13 ++++++++----- 7 files changed, 21 insertions(+), 10 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index b89199891d..4618568fee 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1362,7 +1362,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packet_reader_init(&reader, fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); - if (server_supports_feature("fetch", "sideband-all", 0)) { + if (git_env_bool("GIT_TEST_SIDEBAND_ALL", 1) && + server_supports_feature("fetch", "sideband-all", 0)) { reader.use_sideband = 1; reader.me = "fetch-pack"; } diff --git a/t/README b/t/README index 28711cc508..b275c883b8 100644 --- a/t/README +++ b/t/README @@ -358,6 +358,11 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. +GIT_TEST_SIDEBAND_ALL=, when true, overrides the +'uploadpack.allowSidebandAll' setting to true, and when false, forces +fetch-pack to not request sideband-all (even if the server advertises +sideband-all). + Naming Tests ------------ diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf index 581c010d8f..9a6d368247 100644 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@ -78,6 +78,7 @@ PassEnv GNUPGHOME PassEnv ASAN_OPTIONS PassEnv GIT_TRACE PassEnv GIT_CONFIG_NOSYSTEM +PassEnv GIT_TEST_SIDEBAND_ALL SetEnvIf Git-Protocol ".*" GIT_PROTOCOL=$0 diff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh index 6faf17e17a..6caf628efa 100755 --- a/t/t5537-fetch-shallow.sh +++ b/t/t5537-fetch-shallow.sh @@ -243,7 +243,8 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f "$(git -C "$REPO" rev-parse HEAD)" \ "$(git -C "$REPO" rev-parse HEAD^)" \ >"$HTTPD_ROOT_PATH/one-time-sed" && - test_must_fail git -C client fetch --depth=1 "$HTTPD_URL/one_time_sed/repo" \ + test_must_fail env GIT_TEST_SIDEBAND_ALL=0 git -C client \ + fetch --depth=1 "$HTTPD_URL/one_time_sed/repo" \ master:a_branch && # Ensure that the one-time-sed script was used. diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index ae79c6bbc0..fe45bf828d 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -14,7 +14,7 @@ test_expect_success 'test capability advertisement' ' 0000 EOF - git serve --advertise-capabilities >out && + GIT_TEST_SIDEBAND_ALL=0 git serve --advertise-capabilities >out && test-tool pkt-line unpack actual && test_cmp expect actual ' diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 0f2b09ebb8..b491c62e3e 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -583,8 +583,8 @@ test_expect_success 'when server does not send "ready", expect FLUSH' ' test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" git -C http_child \ -c protocol.version=2 \ fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err && - grep "fetch< acknowledgments" log && - ! grep "fetch< ready" log && + grep "fetch< .*acknowledgments" log && + ! grep "fetch< .*ready" log && test_i18ngrep "expected no other sections to be sent after no .ready." err ' diff --git a/upload-pack.c b/upload-pack.c index 765b7695d2..0c1feccaab 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1288,7 +1288,9 @@ static void process_args(struct packet_reader *request, continue; } - if (allow_sideband_all && !strcmp(arg, "sideband-all")) { + if ((git_env_bool("GIT_TEST_SIDEBAND_ALL", 0) || + allow_sideband_all) && + !strcmp(arg, "sideband-all")) { data->writer.use_sideband = 1; continue; } @@ -1521,10 +1523,11 @@ int upload_pack_advertise(struct repository *r, allow_ref_in_want) strbuf_addstr(value, " ref-in-want"); - if (!repo_config_get_bool(the_repository, - "uploadpack.allowsidebandall", - &allow_sideband_all_value) && - allow_sideband_all_value) + if (git_env_bool("GIT_TEST_SIDEBAND_ALL", 0) || + (!repo_config_get_bool(the_repository, + "uploadpack.allowsidebandall", + &allow_sideband_all_value) && + allow_sideband_all_value)) strbuf_addstr(value, " sideband-all"); } From patchwork Fri Jan 11 22:18:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760749 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6492314E5 for ; Fri, 11 Jan 2019 22:18:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5446629B60 for ; Fri, 11 Jan 2019 22:18:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4893429C1B; Fri, 11 Jan 2019 22:18:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DA9BB29B60 for ; Fri, 11 Jan 2019 22:18:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726506AbfAKWSt (ORCPT ); Fri, 11 Jan 2019 17:18:49 -0500 Received: from mail-it1-f202.google.com ([209.85.166.202]:59892 "EHLO mail-it1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWSt (ORCPT ); Fri, 11 Jan 2019 17:18:49 -0500 Received: by mail-it1-f202.google.com with SMTP id t143so3434071itc.9 for ; Fri, 11 Jan 2019 14:18:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=AzD2Oa7ggtjnDkmZ1k8bM8FTJ9o6vAOnxmwmhxXUOx0=; b=WqaeiZ8WXyy+4XzgV0jS78CzQsmnQk+PWrCFyQPnHbmLDzh2y8fxOW5915QxQ4vcap QX/QrXKZcDtq8GpI0yq4YjZFkbO5g2TDfiSAsD47IvUZxdbpq5oq4b6VimuJQKtXrbFw mU6psWVbvJ/ulITwCQ79C5soAPe2nSdlGXHMambeyqUBJLY5M4NrMj7K5ev4/3cn9zVm Q+pEBEf9vuBTO5imvcuHAPngkdLAClJGKf/RcUgPKN6LfLa0BUmygFyjiVwfRDSh+HAP akFdLsfjrJ3nnJ4C2r57N3998VQGR+fRvJHKpPB2RMHpFGYOuWPjbmJQZSdJKdzLe1bC 2yrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=AzD2Oa7ggtjnDkmZ1k8bM8FTJ9o6vAOnxmwmhxXUOx0=; b=Dq9b0DyZsthdT9Mt1MIWeLqiUnDSChILkTQ6JgOy0Xs2Xne6+Q52jr/D/diTybjCbn dnBfwHwhmByg12EBgUNrZAFwstHdl+v3RMUxJcoeFdR7KhHo2cWLiIDENZ1kIi/iJoIe UIo89HHWwneyh+gA7CF+ncvs2fO2p4hHCjSUUaR+aAnfeiBsUn2l8ruElrc1wRRBjjv3 TmAh2vHbx7kX+ELO4+pdn4DY1Wf0CIl96crHFdZUjMiSSRnMwYcYOqH1c8BCVEG8sYXQ IWmvHUuatlNxg7WMPb2ifxfaoNuDVgbrGRHbWe7grmwcHUw7kR50hu6MrWfZtw9khqYh yuiA== X-Gm-Message-State: AJcUukfgB8Rj5ZyHuY4deQ4eZRsBsc/L3uFo3QJrRcTIE17jZcUAO/kD EyaIfbfJIuQ/z2xT8qUKA8R59jPGtAujMyGILv8PNkOI4WPUXer0dB1NRT1JgjwTx6cnW/B4Qx+ fyogi6kcJ25e6c1pm9AmPFZI7m7Gml1J+JBcSs0UspsMRHKSFAS0Z8nCtFK/G2WHMVFuTSV4igX NU X-Google-Smtp-Source: ALg8bN4npS9PCgGSDYHJINjr2a37U8Ka7VQ9oM7Ve6wXYgTKacTesMe8C+EyNja62YsyrhPsfw8zaQKVyKXMLR9UTTyQ X-Received: by 2002:a24:5411:: with SMTP id t17mr1471404ita.10.1547245128345; Fri, 11 Jan 2019 14:18:48 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:18 -0800 In-Reply-To: Message-Id: <7e5d8a6ba3ae4211e18973f1549986c2e7640c82.1547244620.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP 5/4] Documentation: order protocol v2 sections From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The current C Git implementation expects Git servers to follow a specific order of sections when transmitting protocol v2 responses, but this is not explicit in the documentation. Make the order explicit. Signed-off-by: Jonathan Tan --- Documentation/technical/protocol-v2.txt | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 1b0633f59f..c76f2311c2 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -319,11 +319,11 @@ included in the client's request: The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section -header. +header. Most sections are sent only when the packfile is sent. - output = *section - section = (acknowledgments | shallow-info | wanted-refs | packfile) - (flush-pkt | delim-pkt) + output = acknowledgements flush-pkt | + [acknowledgments delim-pkt] [shallow-info delim-pkt] + [wanted-refs delim-pkt] packfile flush-pkt acknowledgments = PKT-LINE("acknowledgments" LF) (nak | *ack) @@ -345,9 +345,10 @@ header. *PKT-LINE(%x01-03 *%x00-ff) acknowledgments section - * If the client determines that it is finished with negotiations - by sending a "done" line, the acknowledgments sections MUST be - omitted from the server's response. + * If the client determines that it is finished with negotiations by + sending a "done" line (thus requiring the server to send a packfile), + the acknowledgments sections MUST be omitted from the server's + response. * Always begins with the section header "acknowledgments" @@ -398,9 +399,6 @@ header. which the client has not indicated was shallow as a part of its request. - * This section is only included if a packfile section is also - included in the response. - wanted-refs section * This section is only included if the client has requested a ref using a 'want-ref' line and if a packfile section is also From patchwork Fri Jan 11 22:18:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760753 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C7A513BF for ; Fri, 11 Jan 2019 22:18:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id F0CF128A40 for ; Fri, 11 Jan 2019 22:18:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E560529C1B; Fri, 11 Jan 2019 22:18:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D374E28A40 for ; Fri, 11 Jan 2019 22:18:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726525AbfAKWSw (ORCPT ); Fri, 11 Jan 2019 17:18:52 -0500 Received: from mail-vs1-f73.google.com ([209.85.217.73]:35409 "EHLO mail-vs1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWSw (ORCPT ); Fri, 11 Jan 2019 17:18:52 -0500 Received: by mail-vs1-f73.google.com with SMTP id w206so6974898vsc.2 for ; Fri, 11 Jan 2019 14:18:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=2tCSrD+5toMh2iFm7DXOsPJNOV8SzEwoo0+gn/L2xdY=; b=Cjbdva/GIWFpPIOW2FjJ2eD4V4XnkcnVKwY/WeZgjmyUxlxWNISRHpIpvY3+J122WL fY8IE/XPLjMAJmG5nK+tN4yXq0D2eA7psvcGG2XhXRH9agY9w4MAF/++aXq9kGuzp0XO m9GidcSMvaR2q1JcJHLJnp+PawDh4PXjoJ28suKogHthXNQGCxSqoRKWrTAOMwwvIsqX Rf2nHFl3OjS7LsqU6JyeD5DkhpNmibz8zkdlss7Y8sW3HujXBJb9b3es/YUi0+PPw2aV rWZWGM8Y3jLIBAJyi1xvSmDfNuidANpVo1s9H4CtspaFWLDSwLS+PpIdth9bPyG/1euD 7WFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=2tCSrD+5toMh2iFm7DXOsPJNOV8SzEwoo0+gn/L2xdY=; b=ku18Fh8r0JYgNslOy9mIo2LOg6lTG6n4ta8MuLxKWzXiIHFeHwML+CdVTv09x4VS0l r7SifMP/ORNpGPjKasNursKAxV7EgsxnVfIUiHA8vYZqsDNT3S60YfGShTCg/UGUn85m a9kjkGsvCzAUKOsr+3ggsZt6OVm14RXoRfcOSiL8+7xEzNvaQsAH2Qxi1gQoWXoBAljR 2sPW+iat0JkAz7j0WheijZmifx+q/83V4rmO4vUM9eRHcV7pX2jw0pWcPuS4b8IUuDSF TypxW9KRZ0GPf+kj5G4I8q29Gd6ouNYJkxePpCEo8EtE9mNX19KmzIyUi+gsQjMtC1kg HlUg== X-Gm-Message-State: AJcUukeeyiXDnu2FQcDKwRJcbK6fmrSwvAV24lzFxXrmBCeeC+qUgyzn VSABAOJhmLUkJi3euhR49SOS2SbY9+N1+yWWE12B/lclBPbpv7T7cj4fgyRhiONmk/7Td3hPey7 L0fHGDDdBPvXWy5HHOYxT4EmTLAVQ7yqmyNLVkfAkdy/z4quJmZPypFZj8BVSI4Jb1ajC/THJFS IM X-Google-Smtp-Source: ALg8bN7VZN1Bi2g3Yz7xHL+N0lIauVUe/SUuQuo03C42zRsEWL3Sbp7RHz6yFN9ohu/OlxQYlKmVWM1QGuTa1e9FeLRX X-Received: by 2002:a67:3e58:: with SMTP id l85mr1730800vsa.14.1547245131001; Fri, 11 Jan 2019 14:18:51 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:19 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP 6/4] Documentation: add Packfile URIs design doc From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Jonathan Tan --- Documentation/technical/packfile-uri.txt | 83 ++++++++++++++++++++++++ Documentation/technical/protocol-v2.txt | 6 +- 2 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 Documentation/technical/packfile-uri.txt diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt new file mode 100644 index 0000000000..6535801486 --- /dev/null +++ b/Documentation/technical/packfile-uri.txt @@ -0,0 +1,83 @@ +Packfile URIs +============= + +This feature allows servers to serve part of their packfile response as URIs. +This allows server designs that improve scalability in bandwidth and CPU usage +(for example, by serving some data through a CDN), and (in the future) provides +some measure of resumability to clients. + +This feature is available only in protocol version 2. + +Protocol +-------- + +The server advertises `packfile-uris`. + +If the client replies with the following arguments: + + * packfile-uris + * thin-pack + * ofs-delta + +when the server sends the packfile, it MAY send a `packfile-uris` section +directly before the `packfile` section (right after `wanted-refs` if it is +sent) containing HTTP(S) URIs. See protocol-v2.txt for the documentation of +this section. + +Clients then should understand that the returned packfile could be incomplete, +and that it needs to download all the given URIs before the fetch or clone is +complete. Each URI should point to a Git packfile (which may be a thin pack and +which may contain offset deltas). + +Server design +------------- + +The server can be trivially made compatible with the proposed protocol by +having it advertise `packfile-uris`, tolerating the client sending +`packfile-uris`, and never sending any `packfile-uris` section. But we should +include some sort of non-trivial implementation in the Minimum Viable Product, +at least so that we can test the client. + +This is the implementation: a feature, marked experimental, that allows the +server to be configured by one or more `uploadpack.blobPackfileUri= +` entries. Whenever the list of objects to be sent is assembled, a blob +with the given sha1 can be replaced by the given URI. This allows, for example, +servers to delegate serving of large blobs to CDNs. + +Client design +------------- + +While fetching, the client needs to remember the list of URIs and cannot +declare that the fetch is complete until all URIs have been downloaded as +packfiles. + +The division of work (initial fetch + additional URIs) introduces convenient +points for resumption of an interrupted clone - such resumption can be done +after the Minimum Viable Product (see "Future work"). + +The client can inhibit this feature (i.e. refrain from sending the +`packfile-urls` parameter) by passing --no-packfile-urls to `git fetch`. + +Future work +----------- + +The protocol design allows some evolution of the server and client without any +need for protocol changes, so only a small-scoped design is included here to +form the MVP. For example, the following can be done: + + * On the server, a long-running process that takes in entire requests and + outputs a list of URIs and the corresponding inclusion and exclusion sets of + objects. This allows, e.g., signed URIs to be used and packfiles for common + requests to be cached. + * On the client, resumption of clone. If a clone is interrupted, information + could be recorded in the repository's config and a "clone-resume" command + can resume the clone in progress. (Resumption of subsequent fetches is more + difficult because that must deal with the user wanting to use the repository + even after the fetch was interrupted.) + +There are some possible features that will require a change in protocol: + + * Additional HTTP headers (e.g. authentication) + * Byte range support + * Different file formats referenced by URIs (e.g. raw object) + diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index c76f2311c2..de5c63bc9b 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -323,7 +323,8 @@ header. Most sections are sent only when the packfile is sent. output = acknowledgements flush-pkt | [acknowledgments delim-pkt] [shallow-info delim-pkt] - [wanted-refs delim-pkt] packfile flush-pkt + [wanted-refs delim-pkt] [packfile-uris delim-pkt] + packfile flush-pkt acknowledgments = PKT-LINE("acknowledgments" LF) (nak | *ack) @@ -341,6 +342,9 @@ header. Most sections are sent only when the packfile is sent. *PKT-LINE(wanted-ref LF) wanted-ref = obj-id SP refname + packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri + packfile-uri = PKT-LINE("uri" SP *%x20-ff LF) + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff) From patchwork Fri Jan 11 22:18:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760755 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EC9A013BF for ; Fri, 11 Jan 2019 22:18:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DCB7A28A40 for ; Fri, 11 Jan 2019 22:18:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D107529BFF; Fri, 11 Jan 2019 22:18:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6829528A40 for ; Fri, 11 Jan 2019 22:18:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726530AbfAKWSz (ORCPT ); Fri, 11 Jan 2019 17:18:55 -0500 Received: from mail-qk1-f201.google.com ([209.85.222.201]:53868 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWSz (ORCPT ); Fri, 11 Jan 2019 17:18:55 -0500 Received: by mail-qk1-f201.google.com with SMTP id r145so10749181qke.20 for ; Fri, 11 Jan 2019 14:18:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=/7twxnn7Iz4wc2JC/kdGYLyArr4JAVsEg+FhUY4JsXE=; b=vi7Be1eWhOfc5kGRceYd5SZlWLrlHdz1680OH0bTaGIyK9tW75TKhaz7npXvPcM/+p 0xM/HgGpCWwp8Hpw0+H7MDbPaQnm7f/YYRrrnBRS1/oyjfZQLa2csber2Wfhtaek6LzW 6Dz8ziOoogLIpa151ES4QqW07QvvC1MarAAUOj8uzgoPs1xHyrEEkPq3VuQRAzHWlyxq cTYQweqe5ZjIgNOIWnaIWHNCHOp9vszDLH6HMRcMYwRSbK8L1LnwFUkTSCX/UxLP36MJ 3kBia1AIFWddiRl9dzSMtV9CoQpoH/sGl6gOVd5hsCUab5AHwzj7CGvEOV3ql4dLUCB+ MxFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=/7twxnn7Iz4wc2JC/kdGYLyArr4JAVsEg+FhUY4JsXE=; b=hKuCSvGVuNcqyYKdgZhKch09VMlhhCZHP7mYi5xI5oJI4Ad0W0IweudurqS3A6r+3i DOebSEr3piLnvktyiopg5hVxY6An1zgzcs5aYBk03twoyNP7WfL1HwiI5oR8YAZHGGKz gNcDcawR3FS/Q9nxYTK3PuKEpve689z/PRwJ/v+k97PjyAjl8y9vHK0pPFW7bF2oSDY+ Y81UFXQrFsAc45kTLr6UgYPuaGWwvZB+kb6pAYROULQPLsU4lMxdfDAQsh4SzB0EUVsg OOJ+xvu3bwPHTtXVaVTJgbCS5HAq3lUe+SFqPKI6ma+oDPTbwAPTu+bL/wYU866WyEex l5ZQ== X-Gm-Message-State: AJcUukevRPLgLoQFECkqlr9invwy6i52nrmL/+/PgOxWPFhYKoHCB+Va /2436037gt7eL0ZJDNScgPW0laSIjea88wWzZICsf1Oc6ap1xFVQfNSIvkI5U0Ria3nZybLtx0o wMssI6urWdpkhtJR6dtLJBOicdvsWlXyNO35dLmfYG+wiqqZTTRMvY1uPQsGLxvBI4rQu0tDEHO wh X-Google-Smtp-Source: ALg8bN71zmbGif5iKb1Eq8yST5MjyocDdhcaqCFfrgH+O2ROrqjMslPGEDebj+dr/+O7ScLX+9eX3HxmmTjB81BCqnFR X-Received: by 2002:a0c:ae41:: with SMTP id z1mr1545403qvc.11.1547245133777; Fri, 11 Jan 2019 14:18:53 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:20 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP 7/4] upload-pack: refactor reading of pack-objects out From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Subsequent patches will change how the output of pack-objects is processed, so extract that processing into its own function. Currently, at most 1 character can be buffered (in the "buffered" local variable). One of those patches will require a larger buffer, so replace that "buffered" local variable with a buffer array. Signed-off-by: Jonathan Tan --- upload-pack.c | 80 +++++++++++++++++++++++++++++---------------------- 1 file changed, 46 insertions(+), 34 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index 0c1feccaab..c87b752550 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -103,14 +103,51 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data) return 0; } +struct output_state { + char buffer[8193]; + int used; +}; + +static int read_pack_objects_stdout(int outfd, struct output_state *os) +{ + /* Data ready; we keep the last byte to ourselves + * in case we detect broken rev-list, so that we + * can leave the stream corrupted. This is + * unfortunate -- unpack-objects would happily + * accept a valid packdata with trailing garbage, + * so appending garbage after we pass all the + * pack data is not good enough to signal + * breakage to downstream. + */ + ssize_t readsz; + + readsz = xread(outfd, os->buffer + os->used, + sizeof(os->buffer) - os->used); + if (readsz < 0) { + return readsz; + } + os->used += readsz; + + if (os->used > 1) { + send_client_data(1, os->buffer, os->used - 1); + os->buffer[0] = os->buffer[os->used - 1]; + os->used = 1; + } else { + send_client_data(1, os->buffer, os->used); + os->used = 0; + } + + return readsz; +} + static void create_pack_file(const struct object_array *have_obj, const struct object_array *want_obj) { struct child_process pack_objects = CHILD_PROCESS_INIT; - char data[8193], progress[128]; + struct output_state output_state = {0}; + char progress[128]; char abort_msg[] = "aborting due to possible repository " "corruption on the remote side."; - int buffered = -1; ssize_t sz; int i; FILE *pipe_fd; @@ -237,39 +274,15 @@ static void create_pack_file(const struct object_array *have_obj, continue; } if (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) { - /* Data ready; we keep the last byte to ourselves - * in case we detect broken rev-list, so that we - * can leave the stream corrupted. This is - * unfortunate -- unpack-objects would happily - * accept a valid packdata with trailing garbage, - * so appending garbage after we pass all the - * pack data is not good enough to signal - * breakage to downstream. - */ - char *cp = data; - ssize_t outsz = 0; - if (0 <= buffered) { - *cp++ = buffered; - outsz++; - } - sz = xread(pack_objects.out, cp, - sizeof(data) - outsz); - if (0 < sz) - ; - else if (sz == 0) { + int result = read_pack_objects_stdout(pack_objects.out, + &output_state); + + if (result == 0) { close(pack_objects.out); pack_objects.out = -1; - } - else + } else if (result < 0) { goto fail; - sz += outsz; - if (1 < sz) { - buffered = data[sz-1] & 0xFF; - sz--; } - else - buffered = -1; - send_client_data(1, data, sz); } /* @@ -294,9 +307,8 @@ static void create_pack_file(const struct object_array *have_obj, } /* flush the data */ - if (0 <= buffered) { - data[0] = buffered; - send_client_data(1, data, 1); + if (output_state.used > 0) { + send_client_data(1, output_state.buffer, output_state.used); fprintf(stderr, "flushed.\n"); } if (use_sideband) From patchwork Fri Jan 11 22:18:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10760757 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 09ED291E for ; Fri, 11 Jan 2019 22:19:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED93828A40 for ; Fri, 11 Jan 2019 22:18:59 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E267F29BFF; Fri, 11 Jan 2019 22:18:59 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EB7A28A40 for ; Fri, 11 Jan 2019 22:18:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726532AbfAKWS6 (ORCPT ); Fri, 11 Jan 2019 17:18:58 -0500 Received: from mail-yw1-f74.google.com ([209.85.161.74]:42115 "EHLO mail-yw1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725767AbfAKWS5 (ORCPT ); Fri, 11 Jan 2019 17:18:57 -0500 Received: by mail-yw1-f74.google.com with SMTP id d72so8613661ywe.9 for ; Fri, 11 Jan 2019 14:18:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=kvlaPdpwiLzKFBWMBG4yUcpuRyhe9W7l3KzSblqzlsY=; b=HHIRTpXAlv4bTS7hxFtt+c6CTSFA+VfVwgVaer7fcyTcXMxGuNLXV+dT980+eBuqj1 Hla6mgYHMyJBohNbE8tNa3S9QoL198ZwsTy2cvcE60JTuhVoHnj/KcyOFKk8Mp6Bho0M h4nqylwBQGcPr6pRJQgJLbp+fltfI2omOUzSf/m+vDDYLhI2RZsvVFRSm31sQp+6cl5Z FInCNIcs187QZkX+c80q12rBh+6AnOv3bZBK5yh5Jv56gpox8KqKpGeTH5slqEKPP9H0 TISSHJ2lqWM9aeNknTGhWrWbY9gaThvu0YtmuhopllgvKLiCX7MOX+Hl60sLsh9mO5OA uPiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=kvlaPdpwiLzKFBWMBG4yUcpuRyhe9W7l3KzSblqzlsY=; b=RP9UTYBGadqcSYedSiABDtU4U+pSrkWthVluL24zOCVEBza6Oy4sQntyv2P9ta5pQh R3ximLkNplYmys1qUG0Sm3UDp4KdYlDDBO06b3pmcyCFJdU8g4TubtxBkB9AFIEqa0s5 y0lesCr15AbQ0eVaRgNDsNB1O+cNv1EyXSUi6+uzFs/e1cqW1Mu5oq7bSvdhoC82kQxx BgVTv4kCJk9MHcbj3iKaeHuD0CcPmtw+EtcslOCAlEByhfMxix/hVfwzPq9d67RKkzLa NqlBKj/EcrkxPlDRizOzsIdCTguyA7Z261S2FYsqrMBzS/Vw3UVu4P2NKg5AqLUbmclH Hpaw== X-Gm-Message-State: AJcUukcPcUcMfFME4WkccsyHpkh0PlDZdFbMSW+FUkwMJG42OVKYnHWE xXMs7sv58QQTWhCHUI8MmzT4YCQzzQA6kICE4aB3WcRTmDRnOdZBgr7jkZbT+8oA3ARGPGvAeon eTG7chpgk3EvaSFD/rmq8YMZ2losj1bYgt4zqe4c2RMJxUcU15l8wxbjGo/+pootcXrhdvMZM1W cv X-Google-Smtp-Source: ALg8bN6HgV06IgW+wyO3CfoRjyVn8GNmJBm7Akx3GHyc/ZEXLKZt2wwtRHesahv2IL30MTh6bToZQLwEjuVDnaEQyVaq X-Received: by 2002:a25:af04:: with SMTP id a4mr1119352ybh.70.1547245136050; Fri, 11 Jan 2019 14:18:56 -0800 (PST) Date: Fri, 11 Jan 2019 14:18:21 -0800 In-Reply-To: Message-Id: <7a6c111a9ad1c808fb3f0b562ac8b351e3fb1832.1547244620.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP 8/4] upload-pack: send part of packfile response as uri From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is a partial implementation of upload-pack sending part of its packfile response as URIs. The client is not fully implemented - it knows to ignore the "packfile-uris" section, but because it does not actually fetch those URIs, the returned packfile is incomplete. A test is included to show that the appropriate URI is indeed transmitted, and that the returned packfile is lacking exactly the expected object. Signed-off-by: Jonathan Tan --- builtin/pack-objects.c | 48 ++++++++++++++++++++++++++++++++++++ fetch-pack.c | 9 +++++++ t/t5702-protocol-v2.sh | 27 ++++++++++++++++++++ upload-pack.c | 56 ++++++++++++++++++++++++++++++++++++------ 4 files changed, 133 insertions(+), 7 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 411aefd687..448c42a666 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -117,6 +117,15 @@ enum missing_action { static enum missing_action arg_missing_action; static show_object_fn fn_show_object; +struct configured_exclusion { + struct oidmap_entry e; + char *uri; +}; +static struct oidmap configured_exclusions; + +static int exclude_configured_blobs; +static struct oidset excluded_by_config; + /* * stats */ @@ -831,6 +840,23 @@ static off_t write_reused_pack(struct hashfile *f) return reuse_packfile_offset - sizeof(struct pack_header); } +static void write_excluded_by_configs(void) +{ + struct oidset_iter iter; + const struct object_id *oid; + + oidset_iter_init(&excluded_by_config, &iter); + while ((oid = oidset_iter_next(&iter))) { + struct configured_exclusion *ex = + oidmap_get(&configured_exclusions, oid); + + if (!ex) + BUG("configured exclusion wasn't configured"); + write_in_full(1, ex->uri, strlen(ex->uri)); + write_in_full(1, "\n", 1); + } +} + static const char no_split_warning[] = N_( "disabling bitmap writing, packs are split due to pack.packSizeLimit" ); @@ -1124,6 +1150,12 @@ static int want_object_in_pack(const struct object_id *oid, } } + if (exclude_configured_blobs && + oidmap_get(&configured_exclusions, oid)) { + oidset_insert(&excluded_by_config, oid); + return 0; + } + return 1; } @@ -2728,6 +2760,19 @@ static int git_pack_config(const char *k, const char *v, void *cb) pack_idx_opts.version); return 0; } + if (!strcmp(k, "uploadpack.blobpackfileuri")) { + struct configured_exclusion *ex = xmalloc(sizeof(*ex)); + const char *end; + + if (parse_oid_hex(v, &ex->e.oid, &end) || *end != ' ') + die(_("value of uploadpack.blobpackfileuri must be " + "of the form ' ' (got '%s')"), v); + if (oidmap_get(&configured_exclusions, &ex->e.oid)) + die(_("object already configured in another " + "uploadpack.blobpackfileuri (got '%s')"), v); + ex->uri = xstrdup(end + 1); + oidmap_put(&configured_exclusions, ex); + } return git_default_config(k, v, cb); } @@ -3316,6 +3361,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) N_("do not pack objects in promisor packfiles")), OPT_BOOL(0, "delta-islands", &use_delta_islands, N_("respect islands during delta compression")), + OPT_BOOL(0, "exclude-configured-blobs", &exclude_configured_blobs, + N_("respect uploadpack.blobpackfileuri")), OPT_END(), }; @@ -3489,6 +3536,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) return 0; if (nr_result) prepare_pack(window, depth); + write_excluded_by_configs(); write_pack_file(); if (progress) fprintf_ln(stderr, diff --git a/fetch-pack.c b/fetch-pack.c index 4618568fee..79af87b2cf 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1429,6 +1429,15 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, receive_wanted_refs(&reader, sought, nr_sought); /* get the pack */ + if (process_section_header(&reader, "packfile-uris", 1)) { + /* skip the whole section */ + process_section_header(&reader, "packfile-uris", 0); + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) { + /* do nothing */ + } + if (reader.status != PACKET_READ_DELIM) + die("expected DELIM"); + } process_section_header(&reader, "packfile", 0); if (get_pack(args, fd, pack_lockfile)) die(_("git fetch-pack: fetch failed.")); diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index b491c62e3e..ba85ee4dd9 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -588,6 +588,33 @@ test_expect_success 'when server does not send "ready", expect FLUSH' ' test_i18ngrep "expected no other sections to be sent after no .ready." err ' +test_expect_success 'part of packfile response provided as URI' ' + rm -rf "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" http_child log && + + git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + echo my-blob >"$HTTPD_DOCUMENT_ROOT_PATH/http_parent/my-blob" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" add my-blob && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" commit -m x && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" hash-object my-blob >h && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" config \ + "uploadpack.blobpackfileuri" \ + "$(cat h) https://example.com/a-uri" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" config \ + "uploadpack.allowsidebandall" "true" && + + # NEEDSWORK: "git clone" fails here because it ignores the URI provided + # instead of fetching it. + test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" \ + git -c protocol.version=2 clone \ + "$HTTPD_URL/smart/http_parent" http_child 2>err && + # Although "git clone" fails, we can still check that the server + # provided the URI we requested and that the error message pinpoints + # the object that is missing. + grep "clone< .*uri https://example.com/a-uri" log && + test_i18ngrep "did not receive expected object $(cat h)" err +' + stop_httpd test_done diff --git a/upload-pack.c b/upload-pack.c index c87b752550..fe6e2300e3 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -106,9 +106,12 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data) struct output_state { char buffer[8193]; int used; + unsigned packfile_uris_started : 1; + unsigned packfile_started : 1; }; -static int read_pack_objects_stdout(int outfd, struct output_state *os) +static int read_pack_objects_stdout(int outfd, struct output_state *os, + int write_packfile_line) { /* Data ready; we keep the last byte to ourselves * in case we detect broken rev-list, so that we @@ -128,6 +131,37 @@ static int read_pack_objects_stdout(int outfd, struct output_state *os) } os->used += readsz; + while (!os->packfile_started) { + char *p; + if (os->used >= 4 && !memcmp(os->buffer, "PACK", 4)) { + os->packfile_started = 1; + if (write_packfile_line) { + if (os->packfile_uris_started) + packet_delim(1); + packet_write_fmt(1, "\1packfile\n"); + } + break; + } + if ((p = memchr(os->buffer, '\n', os->used))) { + if (!os->packfile_uris_started) { + os->packfile_uris_started = 1; + if (!write_packfile_line) + BUG("packfile_uris requires sideband-all"); + packet_write_fmt(1, "\1packfile-uris\n"); + } + *p = '\0'; + packet_write_fmt(1, "\1uri %s\n", os->buffer); + + os->used -= p - os->buffer + 1; + memmove(os->buffer, p, os->used); + } else { + /* + * Incomplete line. + */ + return readsz; + } + } + if (os->used > 1) { send_client_data(1, os->buffer, os->used - 1); os->buffer[0] = os->buffer[os->used - 1]; @@ -141,7 +175,8 @@ static int read_pack_objects_stdout(int outfd, struct output_state *os) } static void create_pack_file(const struct object_array *have_obj, - const struct object_array *want_obj) + const struct object_array *want_obj, + int write_packfile_line) { struct child_process pack_objects = CHILD_PROCESS_INIT; struct output_state output_state = {0}; @@ -189,6 +224,9 @@ static void create_pack_file(const struct object_array *have_obj, filter_options.filter_spec); } } + if (write_packfile_line) + argv_array_push(&pack_objects.args, + "--exclude-configured-blobs"); pack_objects.in = -1; pack_objects.out = -1; @@ -275,8 +313,8 @@ static void create_pack_file(const struct object_array *have_obj, } if (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) { int result = read_pack_objects_stdout(pack_objects.out, - &output_state); - + &output_state, + write_packfile_line); if (result == 0) { close(pack_objects.out); pack_objects.out = -1; @@ -1108,7 +1146,7 @@ void upload_pack(struct upload_pack_options *options) if (want_obj.nr) { struct object_array have_obj = OBJECT_ARRAY_INIT; get_common_commits(&reader, &have_obj, &want_obj); - create_pack_file(&have_obj, &want_obj); + create_pack_file(&have_obj, &want_obj, 0); } } @@ -1498,8 +1536,12 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys, send_wanted_ref_info(&data); send_shallow_info(&data, &want_obj); - packet_writer_write(&data.writer, "packfile\n"); - create_pack_file(&have_obj, &want_obj); + if (data.writer.use_sideband) { + create_pack_file(&have_obj, &want_obj, 1); + } else { + packet_write_fmt(1, "packfile\n"); + create_pack_file(&have_obj, &want_obj, 0); + } state = FETCH_DONE; break; case FETCH_DONE: