From patchwork Wed Jan 16 19:28:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10766853 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 51794746 for ; Wed, 16 Jan 2019 19:28:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 413822E8DD for ; Wed, 16 Jan 2019 19:28:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 355342EBB6; Wed, 16 Jan 2019 19:28:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0EE7C2E8DD for ; Wed, 16 Jan 2019 19:28:25 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730889AbfAPT2Y (ORCPT ); Wed, 16 Jan 2019 14:28:24 -0500 Received: from mail-oi1-f202.google.com ([209.85.167.202]:49928 "EHLO mail-oi1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730511AbfAPT2X (ORCPT ); Wed, 16 Jan 2019 14:28:23 -0500 Received: by mail-oi1-f202.google.com with SMTP id t83so2148080oie.16 for ; Wed, 16 Jan 2019 11:28:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=SThs+kw/htl5JYykr122hBT5BJAaU7cA/0v6+eWqhkw=; b=pnhEEnzRwmqZeJvXRhJ3YalIM3ngBPrZ9u1wty2gJpH5oH/cjJJs02AxLTdIBlUX5Q wd96Ek1n6DuSsNlN2rIzBfg6pjpgMu9roewGRRERtzwAyxcS/mAwqL3F8dU7rKIoDxVs ztJkkyKhrP0eN96ixemYYjkQrnKLj+SNGhCtVj/C+JTn2SutAY6YHgBwGK/leujUxTzt zmgdJ/1PEyIRkEYV+miYvqaOe5VNIVASEolcKsTZ3XEwit2TsjkAL06ManSfrUpqizTL Ojabh/9HWweeo/B7GoUn2/sxbCIEp89JT3TnhJ/1e41aXH1BlDMSB//RBNmiEkMHsUog llDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=SThs+kw/htl5JYykr122hBT5BJAaU7cA/0v6+eWqhkw=; b=tXO3uAmdA1uGSQgn512LICUtV4IJT/4QjcF469y1lBpg0G9yZoggyqsUVTY7ocrGPB IRMhenRQDpG3QjrBVxbRml85QVDcnorFBMuZZ0+QIV95H29d+Af8OFfa5TfBRkufyFi7 hO+pcA8JimX4/6RfIyHIROq7QU24M4ON3NRSrOv6BN9GQEzffYw87XmCdMoMoncve9aq dMBAZDfAKq6HYZuF19yTTKsXjsJ+NKGGW8GbqONDv6l6L/co4tWf5/aDq0d7V9KykED4 ge1qSz4k3DATioOTn4ZEvflp+jpDiOewiiEqUlKZ2Z38wKPu+b2AAnNgxZmrhguJAJwN iotA== X-Gm-Message-State: AJcUukfo0JaPajhNGf+LeVxBPzBWcN0lT0LiLt8fZRln+tQHym1qgj/5 XmTE8jYUUrGne3MkqqE3IRy4yDuu6UTlQfVjU6s2VQtu3q4zjgEDDVCqsCYaIaVbV39buvm3mj4 pAVysM04ijNvVPXetmXkyG33EsiVWJWBwqecaeKJ3g8XyxvvovjX8+57i9xIcamOHOW2sCf1omL UJ X-Google-Smtp-Source: ALg8bN6LNN/uzDcsuzx70TRaRco2o0jnQLuX16DBVh1jY8j5eTPES40RjCvoWKU39Py44oAY6tX9eqzV7rucNxLrFvOK X-Received: by 2002:aca:f448:: with SMTP id s69mr2632655oih.52.1547666902041; Wed, 16 Jan 2019 11:28:22 -0800 (PST) Date: Wed, 16 Jan 2019 11:28:12 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH v3 1/4] pkt-line: introduce struct packet_writer From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com, szeder.dev@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A future patch will allow the client to request multiplexing of the entire fetch response (and not only during packfile transmission), which in turn allows the server to send progress and keepalive messages at any time during the response. It will be convenient for a future patch if writing options (specifically, whether the written data is to be multiplexed) could be controlled from a single place, so create struct packet_writer to serve as that place, and modify upload-pack to use it. Currently, it only stores the output fd, but a subsequent patch will (as described above) introduce an option to determine if the written data is to be multiplexed. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- pkt-line.c | 47 ++++++++++++++++++--- pkt-line.h | 14 +++++++ upload-pack.c | 112 +++++++++++++++++++++++++++----------------------- 3 files changed, 115 insertions(+), 58 deletions(-) diff --git a/pkt-line.c b/pkt-line.c index e70ea6d88f..9d3e402d41 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -129,12 +129,14 @@ static void set_packet_header(char *buf, const int size) #undef hex } -static void format_packet(struct strbuf *out, const char *fmt, va_list args) +static void format_packet(struct strbuf *out, const char *prefix, + const char *fmt, va_list args) { size_t orig_len, n; orig_len = out->len; strbuf_addstr(out, "0000"); + strbuf_addstr(out, prefix); strbuf_vaddf(out, fmt, args); n = out->len - orig_len; @@ -145,13 +147,13 @@ static void format_packet(struct strbuf *out, const char *fmt, va_list args) packet_trace(out->buf + orig_len + 4, n - 4, 1); } -static int packet_write_fmt_1(int fd, int gently, +static int packet_write_fmt_1(int fd, int gently, const char *prefix, const char *fmt, va_list args) { static struct strbuf buf = STRBUF_INIT; strbuf_reset(&buf); - format_packet(&buf, fmt, args); + format_packet(&buf, prefix, fmt, args); if (write_in_full(fd, buf.buf, buf.len) < 0) { if (!gently) { check_pipe(errno); @@ -168,7 +170,7 @@ void packet_write_fmt(int fd, const char *fmt, ...) va_list args; va_start(args, fmt); - packet_write_fmt_1(fd, 0, fmt, args); + packet_write_fmt_1(fd, 0, "", fmt, args); va_end(args); } @@ -178,7 +180,7 @@ int packet_write_fmt_gently(int fd, const char *fmt, ...) va_list args; va_start(args, fmt); - status = packet_write_fmt_1(fd, 1, fmt, args); + status = packet_write_fmt_1(fd, 1, "", fmt, args); va_end(args); return status; } @@ -211,7 +213,7 @@ void packet_buf_write(struct strbuf *buf, const char *fmt, ...) va_list args; va_start(args, fmt); - format_packet(buf, fmt, args); + format_packet(buf, "", fmt, args); va_end(args); } @@ -486,3 +488,36 @@ enum packet_read_status packet_reader_peek(struct packet_reader *reader) reader->line_peeked = 1; return reader->status; } + +void packet_writer_init(struct packet_writer *writer, int dest_fd) +{ + writer->dest_fd = dest_fd; +} + +void packet_writer_write(struct packet_writer *writer, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + packet_write_fmt_1(writer->dest_fd, 0, "", fmt, args); + va_end(args); +} + +void packet_writer_error(struct packet_writer *writer, const char *fmt, ...) +{ + va_list args; + + va_start(args, fmt); + packet_write_fmt_1(writer->dest_fd, 0, "ERR ", fmt, args); + va_end(args); +} + +void packet_writer_delim(struct packet_writer *writer) +{ + packet_delim(writer->dest_fd); +} + +void packet_writer_flush(struct packet_writer *writer) +{ + packet_flush(writer->dest_fd); +} diff --git a/pkt-line.h b/pkt-line.h index d7e1dbc047..023ad2951d 100644 --- a/pkt-line.h +++ b/pkt-line.h @@ -183,4 +183,18 @@ extern enum packet_read_status packet_reader_peek(struct packet_reader *reader); #define LARGE_PACKET_DATA_MAX (LARGE_PACKET_MAX - 4) extern char packet_buffer[LARGE_PACKET_MAX]; +struct packet_writer { + int dest_fd; +}; + +void packet_writer_init(struct packet_writer *writer, int dest_fd); + +/* These functions die upon failure. */ +__attribute__((format (printf, 2, 3))) +void packet_writer_write(struct packet_writer *writer, const char *fmt, ...); +__attribute__((format (printf, 2, 3))) +void packet_writer_error(struct packet_writer *writer, const char *fmt, ...); +void packet_writer_delim(struct packet_writer *writer); +void packet_writer_flush(struct packet_writer *writer); + #endif diff --git a/upload-pack.c b/upload-pack.c index 08b547cf01..60a26bbbfd 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -613,13 +613,14 @@ static void check_non_tip(struct object_array *want_obj) } } -static void send_shallow(struct commit_list *result) +static void send_shallow(struct packet_writer *writer, + struct commit_list *result) { while (result) { struct object *object = &result->item->object; if (!(object->flags & (CLIENT_SHALLOW|NOT_SHALLOW))) { - packet_write_fmt(1, "shallow %s", - oid_to_hex(&object->oid)); + packet_writer_write(writer, "shallow %s", + oid_to_hex(&object->oid)); register_shallow(the_repository, &object->oid); shallow_nr++; } @@ -627,7 +628,8 @@ static void send_shallow(struct commit_list *result) } } -static void send_unshallow(const struct object_array *shallows, +static void send_unshallow(struct packet_writer *writer, + const struct object_array *shallows, struct object_array *want_obj) { int i; @@ -636,8 +638,8 @@ static void send_unshallow(const struct object_array *shallows, struct object *object = shallows->objects[i].item; if (object->flags & NOT_SHALLOW) { struct commit_list *parents; - packet_write_fmt(1, "unshallow %s", - oid_to_hex(&object->oid)); + packet_writer_write(writer, "unshallow %s", + oid_to_hex(&object->oid)); object->flags &= ~CLIENT_SHALLOW; /* * We want to _register_ "object" as shallow, but we @@ -662,7 +664,7 @@ static void send_unshallow(const struct object_array *shallows, } } -static void deepen(int depth, int deepen_relative, +static void deepen(struct packet_writer *writer, int depth, int deepen_relative, struct object_array *shallows, struct object_array *want_obj) { if (depth == INFINITE_DEPTH && !is_repository_shallow(the_repository)) { @@ -680,7 +682,7 @@ static void deepen(int depth, int deepen_relative, result = get_shallow_commits(&reachable_shallows, depth + 1, SHALLOW, NOT_SHALLOW); - send_shallow(result); + send_shallow(writer, result); free_commit_list(result); object_array_clear(&reachable_shallows); } else { @@ -688,14 +690,15 @@ static void deepen(int depth, int deepen_relative, result = get_shallow_commits(want_obj, depth, SHALLOW, NOT_SHALLOW); - send_shallow(result); + send_shallow(writer, result); free_commit_list(result); } - send_unshallow(shallows, want_obj); + send_unshallow(writer, shallows, want_obj); } -static void deepen_by_rev_list(int ac, const char **av, +static void deepen_by_rev_list(struct packet_writer *writer, int ac, + const char **av, struct object_array *shallows, struct object_array *want_obj) { @@ -703,13 +706,14 @@ static void deepen_by_rev_list(int ac, const char **av, close_commit_graph(the_repository); result = get_shallow_commits_by_rev_list(ac, av, SHALLOW, NOT_SHALLOW); - send_shallow(result); + send_shallow(writer, result); free_commit_list(result); - send_unshallow(shallows, want_obj); + send_unshallow(writer, shallows, want_obj); } /* Returns 1 if a shallow list is sent or 0 otherwise */ -static int send_shallow_list(int depth, int deepen_rev_list, +static int send_shallow_list(struct packet_writer *writer, + int depth, int deepen_rev_list, timestamp_t deepen_since, struct string_list *deepen_not, struct object_array *shallows, @@ -720,7 +724,7 @@ static int send_shallow_list(int depth, int deepen_rev_list, if (depth > 0 && deepen_rev_list) die("git upload-pack: deepen and deepen-since (or deepen-not) cannot be used together"); if (depth > 0) { - deepen(depth, deepen_relative, shallows, want_obj); + deepen(writer, depth, deepen_relative, shallows, want_obj); ret = 1; } else if (deepen_rev_list) { struct argv_array av = ARGV_ARRAY_INIT; @@ -741,7 +745,7 @@ static int send_shallow_list(int depth, int deepen_rev_list, struct object *o = want_obj->objects[i].item; argv_array_push(&av, oid_to_hex(&o->oid)); } - deepen_by_rev_list(av.argc, av.argv, shallows, want_obj); + deepen_by_rev_list(writer, av.argc, av.argv, shallows, want_obj); argv_array_clear(&av); ret = 1; } else { @@ -834,8 +838,10 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan int has_non_tip = 0; timestamp_t deepen_since = 0; int deepen_rev_list = 0; + struct packet_writer writer; shallow_nr = 0; + packet_writer_init(&writer, 1); for (;;) { struct object *o; const char *features; @@ -892,9 +898,9 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan o = parse_object(the_repository, &oid_buf); if (!o) { - packet_write_fmt(1, - "ERR upload-pack: not our ref %s", - oid_to_hex(&oid_buf)); + packet_writer_error(&writer, + "upload-pack: not our ref %s", + oid_to_hex(&oid_buf)); die("git upload-pack: not our ref %s", oid_to_hex(&oid_buf)); } @@ -923,7 +929,7 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan if (depth == 0 && !deepen_rev_list && shallows.nr == 0) return; - if (send_shallow_list(depth, deepen_rev_list, deepen_since, + if (send_shallow_list(&writer, depth, deepen_rev_list, deepen_since, &deepen_not, &shallows, want_obj)) packet_flush(1); object_array_clear(&shallows); @@ -1102,6 +1108,8 @@ struct upload_pack_data { int deepen_rev_list; int deepen_relative; + struct packet_writer writer; + unsigned stateless_rpc : 1; unsigned use_thin_pack : 1; @@ -1125,6 +1133,7 @@ static void upload_pack_data_init(struct upload_pack_data *data) data->haves = haves; data->shallows = shallows; data->deepen_not = deepen_not; + packet_writer_init(&data->writer, 1); } static void upload_pack_data_clear(struct upload_pack_data *data) @@ -1136,7 +1145,8 @@ static void upload_pack_data_clear(struct upload_pack_data *data) string_list_clear(&data->deepen_not, 0); } -static int parse_want(const char *line, struct object_array *want_obj) +static int parse_want(struct packet_writer *writer, const char *line, + struct object_array *want_obj) { const char *arg; if (skip_prefix(line, "want ", &arg)) { @@ -1149,9 +1159,9 @@ static int parse_want(const char *line, struct object_array *want_obj) o = parse_object(the_repository, &oid); if (!o) { - packet_write_fmt(1, - "ERR upload-pack: not our ref %s", - oid_to_hex(&oid)); + packet_writer_error(writer, + "upload-pack: not our ref %s", + oid_to_hex(&oid)); die("git upload-pack: not our ref %s", oid_to_hex(&oid)); } @@ -1167,7 +1177,8 @@ static int parse_want(const char *line, struct object_array *want_obj) return 0; } -static int parse_want_ref(const char *line, struct string_list *wanted_refs, +static int parse_want_ref(struct packet_writer *writer, const char *line, + struct string_list *wanted_refs, struct object_array *want_obj) { const char *arg; @@ -1177,7 +1188,7 @@ static int parse_want_ref(const char *line, struct string_list *wanted_refs, struct object *o; if (read_ref(arg, &oid)) { - packet_write_fmt(1, "ERR unknown ref %s", arg); + packet_writer_error(writer, "unknown ref %s", arg); die("unknown ref %s", arg); } @@ -1220,10 +1231,11 @@ static void process_args(struct packet_reader *request, const char *p; /* process want */ - if (parse_want(arg, want_obj)) + if (parse_want(&data->writer, arg, want_obj)) continue; if (allow_ref_in_want && - parse_want_ref(arg, &data->wanted_refs, want_obj)) + parse_want_ref(&data->writer, arg, &data->wanted_refs, + want_obj)) continue; /* process have line */ if (parse_have(arg, &data->haves)) @@ -1317,26 +1329,26 @@ static int process_haves(struct oid_array *haves, struct oid_array *common, return 0; } -static int send_acks(struct oid_array *acks, struct strbuf *response, +static int send_acks(struct packet_writer *writer, struct oid_array *acks, const struct object_array *have_obj, struct object_array *want_obj) { int i; - packet_buf_write(response, "acknowledgments\n"); + packet_writer_write(writer, "acknowledgments\n"); /* Send Acks */ if (!acks->nr) - packet_buf_write(response, "NAK\n"); + packet_writer_write(writer, "NAK\n"); for (i = 0; i < acks->nr; i++) { - packet_buf_write(response, "ACK %s\n", - oid_to_hex(&acks->oid[i])); + packet_writer_write(writer, "ACK %s\n", + oid_to_hex(&acks->oid[i])); } if (ok_to_give_up(have_obj, want_obj)) { /* Send Ready */ - packet_buf_write(response, "ready\n"); + packet_writer_write(writer, "ready\n"); return 1; } @@ -1348,25 +1360,20 @@ static int process_haves_and_send_acks(struct upload_pack_data *data, struct object_array *want_obj) { struct oid_array common = OID_ARRAY_INIT; - struct strbuf response = STRBUF_INIT; int ret = 0; process_haves(&data->haves, &common, have_obj); if (data->done) { ret = 1; - } else if (send_acks(&common, &response, have_obj, want_obj)) { - packet_buf_delim(&response); + } else if (send_acks(&data->writer, &common, have_obj, want_obj)) { + packet_writer_delim(&data->writer); ret = 1; } else { /* Add Flush */ - packet_buf_flush(&response); + packet_writer_flush(&data->writer); ret = 0; } - /* Send response */ - write_or_die(1, response.buf, response.len); - strbuf_release(&response); - oid_array_clear(&data->haves); oid_array_clear(&common); return ret; @@ -1379,15 +1386,15 @@ static void send_wanted_ref_info(struct upload_pack_data *data) if (!data->wanted_refs.nr) return; - packet_write_fmt(1, "wanted-refs\n"); + packet_writer_write(&data->writer, "wanted-refs\n"); for_each_string_list_item(item, &data->wanted_refs) { - packet_write_fmt(1, "%s %s\n", - oid_to_hex(item->util), - item->string); + packet_writer_write(&data->writer, "%s %s\n", + oid_to_hex(item->util), + item->string); } - packet_delim(1); + packet_writer_delim(&data->writer); } static void send_shallow_info(struct upload_pack_data *data, @@ -1398,14 +1405,15 @@ static void send_shallow_info(struct upload_pack_data *data, !is_repository_shallow(the_repository)) return; - packet_write_fmt(1, "shallow-info\n"); + packet_writer_write(&data->writer, "shallow-info\n"); - if (!send_shallow_list(data->depth, data->deepen_rev_list, + if (!send_shallow_list(&data->writer, data->depth, + data->deepen_rev_list, data->deepen_since, &data->deepen_not, &data->shallows, want_obj) && is_repository_shallow(the_repository)) - deepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows, - want_obj); + deepen(&data->writer, INFINITE_DEPTH, data->deepen_relative, + &data->shallows, want_obj); packet_delim(1); } @@ -1467,7 +1475,7 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys, send_wanted_ref_info(&data); send_shallow_info(&data, &want_obj); - packet_write_fmt(1, "packfile\n"); + packet_writer_write(&data.writer, "packfile\n"); create_pack_file(&have_obj, &want_obj); state = FETCH_DONE; break; From patchwork Wed Jan 16 19:28:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10766855 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B26D9746 for ; Wed, 16 Jan 2019 19:28:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A31412E8DD for ; Wed, 16 Jan 2019 19:28:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 974412EBB6; Wed, 16 Jan 2019 19:28:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D54192E8DD for ; Wed, 16 Jan 2019 19:28:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730898AbfAPT21 (ORCPT ); Wed, 16 Jan 2019 14:28:27 -0500 Received: from mail-vs1-f73.google.com ([209.85.217.73]:36022 "EHLO mail-vs1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730883AbfAPT21 (ORCPT ); Wed, 16 Jan 2019 14:28:27 -0500 Received: by mail-vs1-f73.google.com with SMTP id r133so3177270vsc.3 for ; Wed, 16 Jan 2019 11:28:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=emSsA6+gXaQI4sDm6nTtOEvDON6WrazAVIAG/1x/3uM=; b=vEitrxD9eWwmJlkwv+AnKwZp+cXQsOMCtDaMe0lvwNEGgF3Wn2qWVox31GPSos0Jw9 5+Qz1u1TrSgj6Pcr35cH7fBpQX/b+pPH7387jTRU/K/2OHt45kNSzJoeJXztMQJCx9TX nz+cmijdA7eMfqqtjDK3VdkYXSOZCro28Z7H6G75rUhBf2LRs8t8PmOi9vgbCrQwExMW DG3h7hUbqAcXkOra+T6Chq9tjRJ1cgSFBC9LCn8oMcdw5TvWLQFG9pvKr8KBvVM6RPjl kJ+iq4ll37GsMs8c0bsto3rYyftBstJWsGBmeXvSFoAgtExV45nRikyBY0ne6Jj1BksE kgAQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=emSsA6+gXaQI4sDm6nTtOEvDON6WrazAVIAG/1x/3uM=; b=NJYvkPNO5xPlwRmi8XDKHrzHvIK8dFQZMfOxhIg6vaxnYAc7r0VVF6Sf3cR19swqlO SMEVwyPdXsSBpx8yzFUUzB3XMA8bncGFPkDc8yFdKIlGJwR065YIv6GTE6m4CGxTwPIz FINF0lWHtTKh64e6Wj+Lx5SnCcYgc070Orba2HBv2uVsUlZoERVmrE/qWUlgBgh3L+OG xMtS3XxWyFSvUxevCArCWhM1eO9FPCbGbfrC8XAfpD6zKIIJOJfGfDh5Vj7ctEBPSxr4 peiOuS8JpBIqEBgU8TM6IgFiLgmJf2fd1bzUT1m8i8v250uLyK/BLM1bL4zt4Fd6tBuc zWLA== X-Gm-Message-State: AJcUukfS4AomEiZbl3NUTxp1F41rpbFKpc6ThwvXbDT9Qx5nolhB8oec CE3Wa0blIxKoulF6zTF7H2fTLoSL2sKSSDQQcM16aFyVAA1WcGpHHunECArL7w3CxXU5YrrLdeh ZnfMctMLPrOQyfgbgbXHGRLaFS8RyxKIC+1OYHvZkj168NhL+sS/snxADd3Xh4rdpiAtFzU4lEC WE X-Google-Smtp-Source: ALg8bN7yJ7LpPag/Iwt91uxz5WtFqLevnVAfuAWTjjqaTfAbkPC+0llJzheOeKfUJd4PY9VE+o5/l3KUtP7AhJ+WsrRf X-Received: by 2002:ab0:c0d:: with SMTP id a13mr6491255uak.19.1547666904931; Wed, 16 Jan 2019 11:28:24 -0800 (PST) Date: Wed, 16 Jan 2019 11:28:13 -0800 In-Reply-To: Message-Id: <1a8f645c19fffd2ecd53c5c15be48a1e436729ba.1547666330.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH v3 2/4] sideband: reverse its dependency on pkt-line From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com, szeder.dev@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A subsequent patch will teach struct packet_reader a new field that, if set, instructs it to interpret read data as multiplexed. This will create a dependency from pkt-line to sideband. To avoid a circular dependency, split recv_sideband() into 2 parts: the reading loop (left in recv_sideband()) and the processing of the contents (in demultiplex_sideband()), and move the former into pkt-line. This reverses the direction of dependency: sideband no longer depends on pkt-line, and pkt-line now depends on sideband. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- pkt-line.c | 23 +++++++ pkt-line.h | 16 +++++ sideband.c | 173 +++++++++++++++++++++++++---------------------------- sideband.h | 24 +++++++- 4 files changed, 141 insertions(+), 95 deletions(-) diff --git a/pkt-line.c b/pkt-line.c index 9d3e402d41..321ff632a5 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -439,6 +439,29 @@ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out) return sb_out->len - orig_len; } +int recv_sideband(const char *me, int in_stream, int out) +{ + char buf[LARGE_PACKET_MAX + 1]; + int len; + struct strbuf scratch = STRBUF_INIT; + enum sideband_type sideband_type; + + while (1) { + len = packet_read(in_stream, NULL, NULL, buf, LARGE_PACKET_MAX, + 0); + if (!demultiplex_sideband(me, buf, len, &scratch, + &sideband_type)) + continue; + switch (sideband_type) { + case SIDEBAND_PRIMARY: + write_or_die(out, buf + 1, len - 1); + break; + default: /* errors: message already written */ + return sideband_type; + } + } +} + /* Packet Reader Functions */ void packet_reader_init(struct packet_reader *reader, int fd, char *src_buffer, size_t src_len, diff --git a/pkt-line.h b/pkt-line.h index 023ad2951d..a8e92a4b63 100644 --- a/pkt-line.h +++ b/pkt-line.h @@ -3,6 +3,7 @@ #include "git-compat-util.h" #include "strbuf.h" +#include "sideband.h" /* * Write a packetized stream, where each line is preceded by @@ -120,6 +121,21 @@ char *packet_read_line_buf(char **src_buf, size_t *src_len, int *size); */ ssize_t read_packetized_to_strbuf(int fd_in, struct strbuf *sb_out); +/* + * Receive multiplexed output stream over git native protocol. + * in_stream is the input stream from the remote, which carries data + * in pkt_line format with band designator. Demultiplex it into out + * and err and return error appropriately. Band #1 carries the + * primary payload. Things coming over band #2 is not necessarily + * error; they are usually informative message on the standard error + * stream, aka "verbose"). A message over band #3 is a signal that + * the remote died unexpectedly. A flush() concludes the stream. + * + * Returns SIDEBAND_FLUSH upon a normal conclusion, and SIDEBAND_PROTOCOL_ERROR + * or SIDEBAND_REMOTE_ERROR if an error occurred. + */ +int recv_sideband(const char *me, int in_stream, int out); + struct packet_reader { /* source file descriptor */ int fd; diff --git a/sideband.c b/sideband.c index 368647acf8..e89d677626 100644 --- a/sideband.c +++ b/sideband.c @@ -1,7 +1,6 @@ #include "cache.h" #include "color.h" #include "config.h" -#include "pkt-line.h" #include "sideband.h" #include "help.h" @@ -109,109 +108,99 @@ static void maybe_colorize_sideband(struct strbuf *dest, const char *src, int n) } -/* - * Receive multiplexed output stream over git native protocol. - * in_stream is the input stream from the remote, which carries data - * in pkt_line format with band designator. Demultiplex it into out - * and err and return error appropriately. Band #1 carries the - * primary payload. Things coming over band #2 is not necessarily - * error; they are usually informative message on the standard error - * stream, aka "verbose"). A message over band #3 is a signal that - * the remote died unexpectedly. A flush() concludes the stream. - */ - #define DISPLAY_PREFIX "remote: " #define ANSI_SUFFIX "\033[K" #define DUMB_SUFFIX " " -int recv_sideband(const char *me, int in_stream, int out) +int demultiplex_sideband(const char *me, char *buf, int len, + struct strbuf *scratch, + enum sideband_type *sideband_type) { - const char *suffix; - char buf[LARGE_PACKET_MAX + 1]; - struct strbuf outbuf = STRBUF_INIT; - int retval = 0; - - if (isatty(2) && !is_terminal_dumb()) - suffix = ANSI_SUFFIX; - else - suffix = DUMB_SUFFIX; - - while (!retval) { - const char *b, *brk; - int band, len; - len = packet_read(in_stream, NULL, NULL, buf, LARGE_PACKET_MAX, 0); - if (len == 0) - break; - if (len < 1) { - strbuf_addf(&outbuf, - "%s%s: protocol error: no band designator", - outbuf.len ? "\n" : "", me); - retval = SIDEBAND_PROTOCOL_ERROR; - break; - } - band = buf[0] & 0xff; - buf[len] = '\0'; - len--; - switch (band) { - case 3: - strbuf_addf(&outbuf, "%s%s", outbuf.len ? "\n" : "", - DISPLAY_PREFIX); - maybe_colorize_sideband(&outbuf, buf + 1, len); - - retval = SIDEBAND_REMOTE_ERROR; - break; - case 2: - b = buf + 1; - - /* - * Append a suffix to each nonempty line to clear the - * end of the screen line. - * - * The output is accumulated in a buffer and - * each line is printed to stderr using - * write(2) to ensure inter-process atomicity. - */ - while ((brk = strpbrk(b, "\n\r"))) { - int linelen = brk - b; - - if (!outbuf.len) - strbuf_addstr(&outbuf, DISPLAY_PREFIX); - if (linelen > 0) { - maybe_colorize_sideband(&outbuf, b, linelen); - strbuf_addstr(&outbuf, suffix); - } - - strbuf_addch(&outbuf, *brk); - xwrite(2, outbuf.buf, outbuf.len); - strbuf_reset(&outbuf); - - b = brk + 1; - } + static const char *suffix; + const char *b, *brk; + int band; + + if (!suffix) { + if (isatty(2) && !is_terminal_dumb()) + suffix = ANSI_SUFFIX; + else + suffix = DUMB_SUFFIX; + } - if (*b) { - strbuf_addstr(&outbuf, outbuf.len ? - "" : DISPLAY_PREFIX); - maybe_colorize_sideband(&outbuf, b, strlen(b)); + if (len == 0) { + *sideband_type = SIDEBAND_FLUSH; + goto cleanup; + } + if (len < 1) { + strbuf_addf(scratch, + "%s%s: protocol error: no band designator", + scratch->len ? "\n" : "", me); + *sideband_type = SIDEBAND_PROTOCOL_ERROR; + goto cleanup; + } + band = buf[0] & 0xff; + buf[len] = '\0'; + len--; + switch (band) { + case 3: + strbuf_addf(scratch, "%s%s", scratch->len ? "\n" : "", + DISPLAY_PREFIX); + maybe_colorize_sideband(scratch, buf + 1, len); + + *sideband_type = SIDEBAND_REMOTE_ERROR; + break; + case 2: + b = buf + 1; + + /* + * Append a suffix to each nonempty line to clear the + * end of the screen line. + * + * The output is accumulated in a buffer and + * each line is printed to stderr using + * write(2) to ensure inter-process atomicity. + */ + while ((brk = strpbrk(b, "\n\r"))) { + int linelen = brk - b; + + if (!scratch->len) + strbuf_addstr(scratch, DISPLAY_PREFIX); + if (linelen > 0) { + maybe_colorize_sideband(scratch, b, linelen); + strbuf_addstr(scratch, suffix); } - break; - case 1: - write_or_die(out, buf + 1, len); - break; - default: - strbuf_addf(&outbuf, "%s%s: protocol error: bad band #%d", - outbuf.len ? "\n" : "", me, band); - retval = SIDEBAND_PROTOCOL_ERROR; - break; + + strbuf_addch(scratch, *brk); + xwrite(2, scratch->buf, scratch->len); + strbuf_reset(scratch); + + b = brk + 1; + } + + if (*b) { + strbuf_addstr(scratch, scratch->len ? + "" : DISPLAY_PREFIX); + maybe_colorize_sideband(scratch, b, strlen(b)); } + return 0; + case 1: + *sideband_type = SIDEBAND_PRIMARY; + break; + default: + strbuf_addf(scratch, "%s%s: protocol error: bad band #%d", + scratch->len ? "\n" : "", me, band); + *sideband_type = SIDEBAND_PROTOCOL_ERROR; + break; } - if (outbuf.len) { - strbuf_addch(&outbuf, '\n'); - xwrite(2, outbuf.buf, outbuf.len); +cleanup: + if (scratch->len) { + strbuf_addch(scratch, '\n'); + xwrite(2, scratch->buf, scratch->len); } - strbuf_release(&outbuf); - return retval; + strbuf_release(scratch); + return 1; } /* diff --git a/sideband.h b/sideband.h index 7a8146f161..2c4f021645 100644 --- a/sideband.h +++ b/sideband.h @@ -1,10 +1,28 @@ #ifndef SIDEBAND_H #define SIDEBAND_H -#define SIDEBAND_PROTOCOL_ERROR -2 -#define SIDEBAND_REMOTE_ERROR -1 +enum sideband_type { + SIDEBAND_PROTOCOL_ERROR = -2, + SIDEBAND_REMOTE_ERROR = -1, + SIDEBAND_FLUSH = 0, + SIDEBAND_PRIMARY = 1 +}; + +/* + * Inspects a multiplexed packet read from the remote. If this packet is a + * progress packet and thus should not be processed by the caller, returns 0. + * Otherwise, returns 1, releases scratch, and sets sideband_type. + * + * If this packet is SIDEBAND_PROTOCOL_ERROR, SIDEBAND_REMOTE_ERROR, or a + * progress packet, also prints a message to stderr. + * + * scratch must be a struct strbuf allocated by the caller. It is used to store + * progress messages split across multiple packets. + */ +int demultiplex_sideband(const char *me, char *buf, int len, + struct strbuf *scratch, + enum sideband_type *sideband_type); -int recv_sideband(const char *me, int in_stream, int out); void send_sideband(int fd, int band, const char *data, ssize_t sz, int packet_max); #endif From patchwork Wed Jan 16 19:28:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10766857 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 15B436C2 for ; Wed, 16 Jan 2019 19:28:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 02B2A2E8DD for ; Wed, 16 Jan 2019 19:28:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E58742EC08; Wed, 16 Jan 2019 19:28:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E59862E8DD for ; Wed, 16 Jan 2019 19:28:30 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730937AbfAPT23 (ORCPT ); Wed, 16 Jan 2019 14:28:29 -0500 Received: from mail-oi1-f201.google.com ([209.85.167.201]:39653 "EHLO mail-oi1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730883AbfAPT23 (ORCPT ); Wed, 16 Jan 2019 14:28:29 -0500 Received: by mail-oi1-f201.google.com with SMTP id v184so2159442oie.6 for ; Wed, 16 Jan 2019 11:28:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OQWztLw+DEhHfKH1ViIgKUEbGm+kTom50YytAkQyp+w=; b=eTcaX/AigInnAlzS5FIFF25aYC3rpySgUt0EUNRDBXJkMGm5aQ9XLCnHMWVUrSf9ug loAOryuQAY4Di29zxYDSgijCYyMmu3XJVmbMB+dzoF1aWJxcp6uIOxOoztKVuvrdeW+X Gd2e1U3PGNg6UftW9T6MVr8cWxg29qZ3YEzZJG51662ncKLylr8N/o7hk4kg3EAreHwv OsqNiA/kmEDoKlrDrhpDQIcxhBjJ4WrjzeMA57VU7+GzdLQfV43o5I9nNUKTGDa0fzMC G45/AoPHV6D0awNRSlA8UVZyenp9tjzYzF9/lZ1mYvysExjzLoqmY8BHcPu9fNfITEaf 5tXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=OQWztLw+DEhHfKH1ViIgKUEbGm+kTom50YytAkQyp+w=; b=VO7B1wbzSHaXrpT7wyt7wZ7ZxbrBUz9izQt9WqVFzqT3Ya0xmxpxTgPR+BxgMdCGEm mbDQAHIPzfOvaGvAUBZLW/wFPnZ8TUYky5qLHByR0sN9OXOlXkSW1omuHevM/KorwwZ+ aor1YzBMnVW90iUFBqm02Ku9O6yfpO9fH7idwDtae92BeDo3s0rFxNafq8JZ80nmuTzc 9EviR+5TLs2J/8kLM2exQjCAZ9OyoRhZNpXValCSOpH8PeiW+t92n81TNoFNfgwkki8f LSPf3mdlZQebkgvzOsCmD8Rs4afCaWpm1rU4I0okLpIFVYKNCQMjFUwYezgeL3WDBG1f g9sQ== X-Gm-Message-State: AJcUukdZy5yHQKept5ngCUadWbqbrw7Jod1Xe0KW6ASLGBR5rZdNflWI V27J4Zj5A6p6HpIdy7NMT+clxM6yqJJL9Z37oliqreKcMr5eImqAYzsoPJBJKabPluKQwvrUl5I DYP90JcTWMbof+SfbNP/xJNcWF4cosMDa8UAX2Af217fFYT8YKBhSjDUOVrZkSy/mp4a+Fgod4E j8 X-Google-Smtp-Source: ALg8bN6gMLBgPbiQhbOrfQ2uBGwTuCPys3SnehTU4uOpgrNAPEgVmZiKSJSaPXTjOtjSWihryXCAXM47LFLbrvz2YjpP X-Received: by 2002:aca:504c:: with SMTP id e73mr5977261oib.21.1547666907949; Wed, 16 Jan 2019 11:28:27 -0800 (PST) Date: Wed, 16 Jan 2019 11:28:14 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH v3 3/4] {fetch,upload}-pack: sideband v2 fetch response From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com, szeder.dev@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, a response to a fetch request has sideband support only while the packfile is being sent, meaning that the server cannot send notices until the start of the packfile. Extend sideband support in protocol v2 fetch responses to the whole response. upload-pack will advertise it if the uploadpack.allowsidebandall configuration variable is set, and fetch-pack will automatically request it if advertised. If the sideband is to be used throughout the whole response, upload-pack will use it to send errors instead of prefixing a PKT-LINE payload with "ERR ". This will be tested in a subsequent patch. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- Documentation/technical/protocol-v2.txt | 10 ++++++ fetch-pack.c | 12 +++++-- pkt-line.c | 43 ++++++++++++++++++------- pkt-line.h | 4 +++ sideband.c | 5 +++ sideband.h | 1 + upload-pack.c | 16 +++++++++ 7 files changed, 78 insertions(+), 13 deletions(-) diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 09e4e0273f..39b40c0dc1 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -307,6 +307,16 @@ the 'wanted-refs' section in the server's response as explained below. particular ref, where is the full name of a ref on the server. +If the 'sideband-all' feature is advertised, the following argument can be +included in the client's request: + + sideband-all + Instruct the server to send the whole response multiplexed, not just + the packfile section. All non-flush and non-delim PKT-LINE in the + response (not only in the packfile section) will then start with a byte + indicating its sideband (1, 2, or 3), and the server may send "0005\2" + (a PKT-LINE of sideband 2 with no payload) as a keepalive packet. + The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section header. diff --git a/fetch-pack.c b/fetch-pack.c index ee0847e6ae..86e9e18901 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1090,7 +1090,8 @@ static int add_haves(struct fetch_negotiator *negotiator, static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, const struct fetch_pack_args *args, const struct ref *wants, struct oidset *common, - int *haves_to_send, int *in_vain) + int *haves_to_send, int *in_vain, + int sideband_all) { int ret = 0; struct strbuf req_buf = STRBUF_INIT; @@ -1116,6 +1117,8 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, packet_buf_write(&req_buf, "include-tag"); if (prefer_ofs_delta) packet_buf_write(&req_buf, "ofs-delta"); + if (sideband_all) + packet_buf_write(&req_buf, "sideband-all"); /* Add shallow-info and deepen request */ if (server_supports_feature("fetch", "shallow", 0)) @@ -1324,6 +1327,10 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packet_reader_init(&reader, fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); + if (server_supports_feature("fetch", "sideband-all", 0)) { + reader.use_sideband = 1; + reader.me = "fetch-pack"; + } while (state != FETCH_DONE) { switch (state) { @@ -1357,7 +1364,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, case FETCH_SEND_REQUEST: if (send_fetch_request(&negotiator, fd[1], args, ref, &common, - &haves_to_send, &in_vain)) + &haves_to_send, &in_vain, + reader.use_sideband)) state = FETCH_GET_PACK; else state = FETCH_PROCESS_ACKS; diff --git a/pkt-line.c b/pkt-line.c index 321ff632a5..d4b71d3e82 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -449,7 +449,7 @@ int recv_sideband(const char *me, int in_stream, int out) while (1) { len = packet_read(in_stream, NULL, NULL, buf, LARGE_PACKET_MAX, 0); - if (!demultiplex_sideband(me, buf, len, &scratch, + if (!demultiplex_sideband(me, buf, len, 0, &scratch, &sideband_type)) continue; switch (sideband_type) { @@ -475,25 +475,43 @@ void packet_reader_init(struct packet_reader *reader, int fd, reader->buffer = packet_buffer; reader->buffer_size = sizeof(packet_buffer); reader->options = options; + reader->me = "git"; } enum packet_read_status packet_reader_read(struct packet_reader *reader) { + struct strbuf scratch = STRBUF_INIT; + if (reader->line_peeked) { reader->line_peeked = 0; return reader->status; } - reader->status = packet_read_with_status(reader->fd, - &reader->src_buffer, - &reader->src_len, - reader->buffer, - reader->buffer_size, - &reader->pktlen, - reader->options); + /* + * Consume all progress packets until a primary payload packet is + * received + */ + while (1) { + enum sideband_type sideband_type; + reader->status = packet_read_with_status(reader->fd, + &reader->src_buffer, + &reader->src_len, + reader->buffer, + reader->buffer_size, + &reader->pktlen, + reader->options); + if (!reader->use_sideband) + break; + if (demultiplex_sideband(reader->me, reader->buffer, + reader->pktlen, 1, &scratch, + &sideband_type)) + break; + } if (reader->status == PACKET_READ_NORMAL) - reader->line = reader->buffer; + /* Skip the sideband designator if sideband is used */ + reader->line = reader->use_sideband ? + reader->buffer + 1 : reader->buffer; else reader->line = NULL; @@ -515,6 +533,7 @@ enum packet_read_status packet_reader_peek(struct packet_reader *reader) void packet_writer_init(struct packet_writer *writer, int dest_fd) { writer->dest_fd = dest_fd; + writer->use_sideband = 0; } void packet_writer_write(struct packet_writer *writer, const char *fmt, ...) @@ -522,7 +541,8 @@ void packet_writer_write(struct packet_writer *writer, const char *fmt, ...) va_list args; va_start(args, fmt); - packet_write_fmt_1(writer->dest_fd, 0, "", fmt, args); + packet_write_fmt_1(writer->dest_fd, 0, + writer->use_sideband ? "\001" : "", fmt, args); va_end(args); } @@ -531,7 +551,8 @@ void packet_writer_error(struct packet_writer *writer, const char *fmt, ...) va_list args; va_start(args, fmt); - packet_write_fmt_1(writer->dest_fd, 0, "ERR ", fmt, args); + packet_write_fmt_1(writer->dest_fd, 0, + writer->use_sideband ? "\003" : "ERR ", fmt, args); va_end(args); } diff --git a/pkt-line.h b/pkt-line.h index a8e92a4b63..ad9a4a2cd7 100644 --- a/pkt-line.h +++ b/pkt-line.h @@ -162,6 +162,9 @@ struct packet_reader { /* indicates if a line has been peeked */ int line_peeked; + + unsigned use_sideband : 1; + const char *me; }; /* @@ -201,6 +204,7 @@ extern char packet_buffer[LARGE_PACKET_MAX]; struct packet_writer { int dest_fd; + unsigned use_sideband : 1; }; void packet_writer_init(struct packet_writer *writer, int dest_fd); diff --git a/sideband.c b/sideband.c index e89d677626..6a16feb262 100644 --- a/sideband.c +++ b/sideband.c @@ -114,6 +114,7 @@ static void maybe_colorize_sideband(struct strbuf *dest, const char *src, int n) #define DUMB_SUFFIX " " int demultiplex_sideband(const char *me, char *buf, int len, + int die_on_error, struct strbuf *scratch, enum sideband_type *sideband_type) { @@ -144,6 +145,8 @@ int demultiplex_sideband(const char *me, char *buf, int len, len--; switch (band) { case 3: + if (die_on_error) + die("remote error: %s", buf + 1); strbuf_addf(scratch, "%s%s", scratch->len ? "\n" : "", DISPLAY_PREFIX); maybe_colorize_sideband(scratch, buf + 1, len); @@ -195,6 +198,8 @@ int demultiplex_sideband(const char *me, char *buf, int len, } cleanup: + if (die_on_error && *sideband_type == SIDEBAND_PROTOCOL_ERROR) + die("%s", scratch->buf); if (scratch->len) { strbuf_addch(scratch, '\n'); xwrite(2, scratch->buf, scratch->len); diff --git a/sideband.h b/sideband.h index 2c4f021645..227740a58e 100644 --- a/sideband.h +++ b/sideband.h @@ -20,6 +20,7 @@ enum sideband_type { * progress messages split across multiple packets. */ int demultiplex_sideband(const char *me, char *buf, int len, + int die_on_error, struct strbuf *scratch, enum sideband_type *sideband_type); diff --git a/upload-pack.c b/upload-pack.c index 60a26bbbfd..765b7695d2 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -71,6 +71,8 @@ static int allow_filter; static int allow_ref_in_want; static struct list_objects_filter_options filter_options; +static int allow_sideband_all; + static void reset_timeout(void) { alarm(timeout); @@ -1046,6 +1048,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused) allow_filter = git_config_bool(var, value); } else if (!strcmp("uploadpack.allowrefinwant", var)) { allow_ref_in_want = git_config_bool(var, value); + } else if (!strcmp("uploadpack.allowsidebandall", var)) { + allow_sideband_all = git_config_bool(var, value); } if (current_config_scope() != CONFIG_SCOPE_REPO) { @@ -1284,6 +1288,11 @@ static void process_args(struct packet_reader *request, continue; } + if (allow_sideband_all && !strcmp(arg, "sideband-all")) { + data->writer.use_sideband = 1; + continue; + } + /* ignore unknown lines maybe? */ die("unexpected line: '%s'", arg); } @@ -1496,6 +1505,7 @@ int upload_pack_advertise(struct repository *r, if (value) { int allow_filter_value; int allow_ref_in_want; + int allow_sideband_all_value; strbuf_addstr(value, "shallow"); @@ -1510,6 +1520,12 @@ int upload_pack_advertise(struct repository *r, &allow_ref_in_want) && allow_ref_in_want) strbuf_addstr(value, " ref-in-want"); + + if (!repo_config_get_bool(the_repository, + "uploadpack.allowsidebandall", + &allow_sideband_all_value) && + allow_sideband_all_value) + strbuf_addstr(value, " sideband-all"); } return 1; From patchwork Wed Jan 16 19:28:15 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10766859 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 32DA46C2 for ; Wed, 16 Jan 2019 19:28:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 24A052E8DD for ; Wed, 16 Jan 2019 19:28:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 190832EBB6; Wed, 16 Jan 2019 19:28:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 50DEA2E8FD for ; Wed, 16 Jan 2019 19:28:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730943AbfAPT2c (ORCPT ); Wed, 16 Jan 2019 14:28:32 -0500 Received: from mail-pl1-f201.google.com ([209.85.214.201]:45085 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730883AbfAPT2b (ORCPT ); Wed, 16 Jan 2019 14:28:31 -0500 Received: by mail-pl1-f201.google.com with SMTP id h10so4415087plk.12 for ; Wed, 16 Jan 2019 11:28:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=V3vxRzDnCRitF8AAV8x+8Xaxh6NQUSoxEQAcFW2sY/Q=; b=D3cSI6mcxWuZrYRK5VS9zArMHGamCiKp6kVYA2XgJbDEnf1bvqU94t2OT1ZVI/qpQ6 Qe28kgUdLSyciDzZfjnbPRgTmLLlQ7VX/K6HHXFYfWR6u7Uv7ygccJIunHNK50JhMUd8 VDx8V6XdoueXJU5GUhVo1CKbTGO37KFQPv8HOohkllVGrHbrKv6Fj/yC9fzkStu6dJaU P6FVvhtp/OtjfRaNUQhq/q5WRuB9+O+ecsF5ySehFui2LYqXyfhIpHvfO39lm6IyF0BQ tS1KpAPzrQR+MwDpxN8MaRYnMqZ26Y5RbriVPFpXBKvZfQE+xZhyL7Cf5C3TyUF4eqQT Sxlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=V3vxRzDnCRitF8AAV8x+8Xaxh6NQUSoxEQAcFW2sY/Q=; b=Zoqr2WLAuWsGKnGxef3rMvqqJTkDJ0H7C38rEmfwHJUpu4mC2GtjAt9bljUi10UxEl byp/ZqYkdloAYeWLRFSKdakP7GM6kjVf7/I5A1mmLsd+gIz1fbNStLENDvMW+m7dr6DG NCf1UA6Li3SC1wt8STFUot9y9a4B8AcN9YSU9xBzKpiqwl5KV+GN2X92ZGf1/VpESkuR C3vIO5lXrWjwudeYGxBRNy5CiVfr6MyYCRh0Sr7Np2G7rX/3pZ76BMcHi/DR9i5PS6MR 74ViFVVOICyCCwOIhUJ5l32Kc2YgSmJAZmLd6CTbp93N03Fa/kcQg0lYyPwLOK8xjxK7 SoJg== X-Gm-Message-State: AJcUukeKJRVTqQWc9jk9uykiYFmhMFD8fWFyZlqLlEtV6Hz64NcoYFCA SYb+8GcQhK7b54KKxSzGEhetEB3n1KFFP05fWs8g9R+rRgpajc1rfS28cAiORXjqkjhLYi41zSZ DwL5Z3Atq0eUJExg3qT5xpx6zp7ncwGDwttKG8GtOCK9fAcyBsmlZNPFZnRBDx6EZm5GedugBkK dD X-Google-Smtp-Source: ALg8bN5t0jIzpGhdGmRgrxSrl2ylSh1I66DjnDVTZYwVqIRu+lMEZbLIqpD32EJ+DmtSiY5Q/4o1CVMAdpqTzytgCOYB X-Received: by 2002:a17:902:2a6b:: with SMTP id i98mr3280946plb.67.1547666910430; Wed, 16 Jan 2019 11:28:30 -0800 (PST) Date: Wed, 16 Jan 2019 11:28:15 -0800 In-Reply-To: Message-Id: <47a98b67113869aa6a887ced52560c8306e55bc0.1547666330.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [PATCH v3 4/4] tests: define GIT_TEST_SIDEBAND_ALL From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan , gitster@pobox.com, szeder.dev@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Define a GIT_TEST_SIDEBAND_ALL environment variable meant to be used from tests. When set to true, this overrides uploadpack.allowsidebandall to true, allowing the entire test suite to be run as if this configuration is in place for all repositories. As of this patch, all tests pass whether GIT_TEST_SIDEBAND_ALL is unset or set to 1. Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano --- fetch-pack.c | 3 ++- t/README | 5 +++++ t/lib-httpd/apache.conf | 1 + t/t5537-fetch-shallow.sh | 3 ++- t/t5701-git-serve.sh | 2 +- t/t5702-protocol-v2.sh | 4 ++-- upload-pack.c | 13 ++++++++----- 7 files changed, 21 insertions(+), 10 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 86e9e18901..e98294d918 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1327,7 +1327,8 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, packet_reader_init(&reader, fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_DIE_ON_ERR_PACKET); - if (server_supports_feature("fetch", "sideband-all", 0)) { + if (git_env_bool("GIT_TEST_SIDEBAND_ALL", 1) && + server_supports_feature("fetch", "sideband-all", 0)) { reader.use_sideband = 1; reader.me = "fetch-pack"; } diff --git a/t/README b/t/README index 28711cc508..b275c883b8 100644 --- a/t/README +++ b/t/README @@ -358,6 +358,11 @@ GIT_TEST_MULTI_PACK_INDEX=, when true, forces the multi-pack- index to be written after every 'git repack' command, and overrides the 'core.multiPackIndex' setting to true. +GIT_TEST_SIDEBAND_ALL=, when true, overrides the +'uploadpack.allowSidebandAll' setting to true, and when false, forces +fetch-pack to not request sideband-all (even if the server advertises +sideband-all). + Naming Tests ------------ diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf index 581c010d8f..9a6d368247 100644 --- a/t/lib-httpd/apache.conf +++ b/t/lib-httpd/apache.conf @@ -78,6 +78,7 @@ PassEnv GNUPGHOME PassEnv ASAN_OPTIONS PassEnv GIT_TRACE PassEnv GIT_CONFIG_NOSYSTEM +PassEnv GIT_TEST_SIDEBAND_ALL SetEnvIf Git-Protocol ".*" GIT_PROTOCOL=$0 diff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh index 6faf17e17a..6caf628efa 100755 --- a/t/t5537-fetch-shallow.sh +++ b/t/t5537-fetch-shallow.sh @@ -243,7 +243,8 @@ test_expect_success 'shallow fetches check connectivity before writing shallow f "$(git -C "$REPO" rev-parse HEAD)" \ "$(git -C "$REPO" rev-parse HEAD^)" \ >"$HTTPD_ROOT_PATH/one-time-sed" && - test_must_fail git -C client fetch --depth=1 "$HTTPD_URL/one_time_sed/repo" \ + test_must_fail env GIT_TEST_SIDEBAND_ALL=0 git -C client \ + fetch --depth=1 "$HTTPD_URL/one_time_sed/repo" \ master:a_branch && # Ensure that the one-time-sed script was used. diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index ae79c6bbc0..fe45bf828d 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -14,7 +14,7 @@ test_expect_success 'test capability advertisement' ' 0000 EOF - git serve --advertise-capabilities >out && + GIT_TEST_SIDEBAND_ALL=0 git serve --advertise-capabilities >out && test-tool pkt-line unpack actual && test_cmp expect actual ' diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 0f2b09ebb8..b491c62e3e 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -583,8 +583,8 @@ test_expect_success 'when server does not send "ready", expect FLUSH' ' test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" git -C http_child \ -c protocol.version=2 \ fetch "$HTTPD_URL/one_time_sed/http_parent" 2> err && - grep "fetch< acknowledgments" log && - ! grep "fetch< ready" log && + grep "fetch< .*acknowledgments" log && + ! grep "fetch< .*ready" log && test_i18ngrep "expected no other sections to be sent after no .ready." err ' diff --git a/upload-pack.c b/upload-pack.c index 765b7695d2..0c1feccaab 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -1288,7 +1288,9 @@ static void process_args(struct packet_reader *request, continue; } - if (allow_sideband_all && !strcmp(arg, "sideband-all")) { + if ((git_env_bool("GIT_TEST_SIDEBAND_ALL", 0) || + allow_sideband_all) && + !strcmp(arg, "sideband-all")) { data->writer.use_sideband = 1; continue; } @@ -1521,10 +1523,11 @@ int upload_pack_advertise(struct repository *r, allow_ref_in_want) strbuf_addstr(value, " ref-in-want"); - if (!repo_config_get_bool(the_repository, - "uploadpack.allowsidebandall", - &allow_sideband_all_value) && - allow_sideband_all_value) + if (git_env_bool("GIT_TEST_SIDEBAND_ALL", 0) || + (!repo_config_get_bool(the_repository, + "uploadpack.allowsidebandall", + &allow_sideband_all_value) && + allow_sideband_all_value)) strbuf_addstr(value, " sideband-all"); }