From patchwork Fri Oct 8 16:03:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ivan Frade X-Patchwork-Id: 12545667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 194B8C433F5 for ; Fri, 8 Oct 2021 16:03:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EE88A60F4F for ; Fri, 8 Oct 2021 16:03:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229644AbhJHQFH (ORCPT ); Fri, 8 Oct 2021 12:05:07 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40714 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229534AbhJHQFG (ORCPT ); Fri, 8 Oct 2021 12:05:06 -0400 Received: from mail-wr1-x42f.google.com (mail-wr1-x42f.google.com [IPv6:2a00:1450:4864:20::42f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D4BD5C061755 for ; Fri, 8 Oct 2021 09:03:10 -0700 (PDT) Received: by mail-wr1-x42f.google.com with SMTP id i12so18644779wrb.7 for ; Fri, 08 Oct 2021 09:03:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=ECq2nyO7lTU5+WTnAphxxRmzj4lbuyMk4pOSvbwXdvU=; b=RPbhNXHd66kXVdL1834DRnS7nlKRsUo5M76QpzwFoTldAfIpEbJa1gsBv7JEmwSTUQ rk2eb2H/avUSFN6enYIV2Uf+3+3ylzXwFP3Ghh6PLut73wJ3zFqo1RzxxmryqoBiW4vD y5QT7L/rnNMRqxkTatCydBPpd8fm1ZaKG9TuFyMcuhZ45SdFrWmZetz+ynX4poXb6mX8 eGnk6ENoclYDCcwtHdOJutROWW6y8n3S4e71VWVBZZckwd99e7sSc1aRMuMm6COsXSCU 04/aN9gx1XVKyd0Hmcq/QV3YBJpLsz/B0B67pTXeCBfwxDuqWmDIjSowGi/kLD1U1ESq El1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=ECq2nyO7lTU5+WTnAphxxRmzj4lbuyMk4pOSvbwXdvU=; b=JHhWSHiuL9qP4r89Iqc/05HQDY5+idLKiejedRgH7dUNkhwehiEmtDwPQZ841GY01B 5G0XweKxq6qszicaJ/YENwOvdvYpslzw+uOtB83d+6ucT4C5UJqHgQ+6E7WXR53UeBCP pV8Cub3sLRuUKEELMX9GYQfbs8IbuZjovjw2ZS/Jo2Fznaer8MuUFvyTmk/Mxjvaeshz VHQgVBK46+1bKud152D67bg97iYD4ng7qwqpaAX97UKaHH0ssROtP5B+5ahA82YhrtDb q9LWQy0wHoqcws9e9OSQdeaGIKirHpUcXWN+FIMAdn/EajEiOhBgnhe4LdmyouIuB+IG U1gQ== X-Gm-Message-State: AOAM533zEdfqdsCTtJDkYMQDchG1pwbIgvgrKxwwg1QnkIEC/DOxVrLl w8G2yeGHvUT673ZBkcdxGAVW0xJg3xI= X-Google-Smtp-Source: ABdhPJxCTKL6VKGw5C9oAXrDa0h4cBL+uHiGvIq0cwR6k/NKfEXioUgnQYbN+88rJPAkZx18RHQV7A== X-Received: by 2002:adf:aa4e:: with SMTP id q14mr5169419wrd.100.1633708989238; Fri, 08 Oct 2021 09:03:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id y6sm2892306wro.63.2021.10.08.09.03.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Oct 2021 09:03:09 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Fri, 08 Oct 2021 16:03:05 +0000 Subject: [PATCH 1/2] fetch-pack: redact packfile urls in traces Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ivan Frade , Ivan Frade Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ivan Frade From: Ivan Frade In some setups, packfile uris act as bearer token. It is not recommended to expose them plainly in logs, although in special circunstances (e.g. debug) it makes sense to write them. Redact the packfile-uri lines by default, unless the GIT_TRACE_REDACT variable is set to false. This mimics the redacting of the Authorization header in HTTP. Signed-off-by: Ivan Frade --- fetch-pack.c | 11 +++++++++++ http-fetch.c | 4 +++- pkt-line.c | 7 ++++++- pkt-line.h | 1 + t/t5702-protocol-v2.sh | 43 ++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 64 insertions(+), 2 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index a9604f35a3e..05c85eeafa1 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1518,7 +1518,16 @@ static void receive_wanted_refs(struct packet_reader *reader, static void receive_packfile_uris(struct packet_reader *reader, struct string_list *uris) { + int original_options; process_section_header(reader, "packfile-uris", 0); + /* + * In some setups, packfile-uris act as bearer tokens, + * redact them by default. + */ + original_options = reader->options; + if (git_env_bool("GIT_TRACE_REDACT", 1)) + reader->options |= PACKET_READ_REDACT_ON_TRACE; + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { if (reader->pktlen < the_hash_algo->hexsz || reader->line[the_hash_algo->hexsz] != ' ') @@ -1526,6 +1535,8 @@ static void receive_packfile_uris(struct packet_reader *reader, string_list_append(uris, reader->line); } + reader->options = original_options; + if (reader->status != PACKET_READ_DELIM) die("expected DELIM"); } diff --git a/http-fetch.c b/http-fetch.c index fa642462a9e..d35e33e4f65 100644 --- a/http-fetch.c +++ b/http-fetch.c @@ -63,7 +63,9 @@ static void fetch_single_packfile(struct object_id *packfile_hash, if (start_active_slot(preq->slot)) { run_active_slot(preq->slot); if (results.curl_result != CURLE_OK) { - die("Unable to get pack file %s\n%s", preq->url, + int showUrl = git_env_bool("GIT_TRACE_REDACT", 1); + die("Unable to get offloaded pack file %s\n%s", + showUrl ? preq->url : "", curl_errorstr); } } else { diff --git a/pkt-line.c b/pkt-line.c index de4a94b437e..8da8ed88ccf 100644 --- a/pkt-line.c +++ b/pkt-line.c @@ -443,7 +443,12 @@ enum packet_read_status packet_read_with_status(int fd, char **src_buffer, len--; buffer[len] = 0; - packet_trace(buffer, len, 0); + if (options & PACKET_READ_REDACT_ON_TRACE) { + const char *redacted = ""; + packet_trace(redacted, strlen(redacted), 0); + } else { + packet_trace(buffer, len, 0); + } if ((options & PACKET_READ_DIE_ON_ERR_PACKET) && starts_with(buffer, "ERR ")) diff --git a/pkt-line.h b/pkt-line.h index 82b95e4bdd3..44c02f3bc6e 100644 --- a/pkt-line.h +++ b/pkt-line.h @@ -88,6 +88,7 @@ void packet_fflush(FILE *f); #define PACKET_READ_CHOMP_NEWLINE (1u<<1) #define PACKET_READ_DIE_ON_ERR_PACKET (1u<<2) #define PACKET_READ_GENTLE_ON_READ_ERROR (1u<<3) +#define PACKET_READ_REDACT_ON_TRACE (1u<<4) int packet_read(int fd, char **src_buffer, size_t *src_len, char *buffer, unsigned size, int options); diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index d527cf6c49f..a620a678a56 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -1107,6 +1107,49 @@ test_expect_success 'packfile-uri with transfer.fsckobjects fails when .gitmodul test_i18ngrep "disallowed submodule name" err ' +test_expect_success 'packfile-uri redacted in trace' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + rm -rf "$P" http_child log && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + echo my-blob >"$P/my-blob" && + git -C "$P" add my-blob && + git -C "$P" commit -m x && + + configure_exclusion "$P" my-blob >h && + + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + + grep -A1 "clone<\ ..packfile-uris" log | grep "clone<\ " +' + +test_expect_success 'packfile-uri not redacted in trace when GIT_TRACE_REDACT=0' ' + P="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + rm -rf "$P" http_child log && + + git init "$P" && + git -C "$P" config "uploadpack.allowsidebandall" "true" && + + echo my-blob >"$P/my-blob" && + git -C "$P" add my-blob && + git -C "$P" commit -m x && + + configure_exclusion "$P" my-blob >h && + + GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \ + GIT_TRACE_REDACT=0 \ + git -c protocol.version=2 \ + -c fetch.uriprotocols=http,https \ + clone "$HTTPD_URL/smart/http_parent" http_child && + + grep -A1 "clone<\ ..packfile-uris" log | grep -E "clone<\ ..[[:alnum:]]{40,64}\ http" +' + test_expect_success 'http:// --negotiate-only' ' SERVER="$HTTPD_DOCUMENT_ROOT_PATH/server" && URI="$HTTPD_URL/smart/server" && From patchwork Fri Oct 8 16:03:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ivan Frade X-Patchwork-Id: 12545669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 10824C433FE for ; Fri, 8 Oct 2021 16:03:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E026961037 for ; Fri, 8 Oct 2021 16:03:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229664AbhJHQFI (ORCPT ); Fri, 8 Oct 2021 12:05:08 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:40718 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229606AbhJHQFG (ORCPT ); Fri, 8 Oct 2021 12:05:06 -0400 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2F0E8C061570 for ; Fri, 8 Oct 2021 09:03:11 -0700 (PDT) Received: by mail-wr1-x434.google.com with SMTP id t8so31455825wri.1 for ; Fri, 08 Oct 2021 09:03:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=x9Ve1tZewj6K6PlM1PVUHZyHW/GWlN0ZnaPrifWlv5s=; b=BGPPA2Lp6l6AG66DFXLtkSZpkPz8gtSh+NCgnymujkagoPpXJy0tQ4EO1M7VNvciX5 x4cZegV2skwVsB9CIiDJaYx+GTvAUSl9TXecuMmWnXrXqLKcIsQvxXy6DlvX+G7/tYOl sjiNNWoOr4+kt+ravEEp1LsCWFlGR5Z2jfYCU1Vo5eonxb4qf9B3Djwv7asXn5f9WAJD JbYiFCWFJLHuLaXgc5IkbadKIaePYnHHDzXgRfSn1xoFgp2WEenG9gz3DSQ8AgQG6C4e CPzbcGfIkU0DCZPgt+/VMtTQmKD9Ni+i2hH6/7jgbiOnDbhXY3ZAKUb8b6cr0tPUf4AT pi5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=x9Ve1tZewj6K6PlM1PVUHZyHW/GWlN0ZnaPrifWlv5s=; b=atIN91Gjq1hlw61EzUcfC8vToltSj25yzaQeGcjW/vmNKpKC6IhYRJtIRRxVXJ1Ann 7UUkl5sAQGPh7ZREcAC3cju4qAQ3C6fXOiIHWK/buecGkKLsL1bIdF6OIfXverhLhuRp r2GrH7pU8cgcyn2RzWz/OK+lyAdy0XFAGOEUflBjNLRpio2XjBFeZgetNyUPAh6Yzohe D1tXK+ba0W9ENhcgq6Czh01gsL923oS+/ITLEC5ez7edwEpMxEb1Ru0vgZcYk0NWAEe3 5vllDs2gDXGbaxP3rdXl+d8SrESknNv6O5BkoJYmodlwhI9K9P4tzhgcKClOet7OtIrK evcw== X-Gm-Message-State: AOAM530LY55/rsdnQjZSn9xBP8b+pB5XNnR+UEej2bmPWQ++UQWkRG0L u2YXgCkeeJe6jJx2x21hb8RocwpUml8= X-Google-Smtp-Source: ABdhPJyPVmfPQAbZ6OyZ4Bcv6FLzp5F7lRB+GynllQdXywMdpCl3pvNYM8O7FSYWT6vPPwnhck1yIQ== X-Received: by 2002:a1c:4c17:: with SMTP id z23mr4444808wmf.61.1633708989829; Fri, 08 Oct 2021 09:03:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z79sm3018927wmc.17.2021.10.08.09.03.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Oct 2021 09:03:09 -0700 (PDT) Message-Id: <497c5fd18d7206c137d8a62d229d2f295c9fe4fa.1633708986.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 08 Oct 2021 16:03:06 +0000 Subject: [PATCH 2/2] Documentation: packfile-uri hash can be longer than 40 hex chars Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Ivan Frade , Ivan Frade Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ivan Frade From: Ivan Frade Packfile-uri line specifies a hash of 40 hex character, but with SHA256 this hash size is 64. There are already tests using SHA256 (e.g. in ubuntu-latest/linux-clang). Update protocol-v2 documentation to indicate that the hash size depends on the hash algorithm in use. Signed-off-by: Ivan Frade --- Documentation/technical/protocol-v2.txt | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 21e8258ccf3..a23f12d6c2b 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -393,7 +393,7 @@ header. Most sections are sent only when the packfile is sent. wanted-ref = obj-id SP refname packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri - packfile-uri = PKT-LINE(40*(HEXDIGIT) SP *%x20-ff LF) + packfile-uri = PKT-LINE((40|64)*(HEXDIGIT) SP *%x20-ff LF) packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff) @@ -476,9 +476,9 @@ header. Most sections are sent only when the packfile is sent. * For each URI the server sends, it sends a hash of the pack's contents (as output by git index-pack) followed by the URI. - * The hashes are 40 hex characters long. When Git upgrades to a new - hash algorithm, this might need to be updated. (It should match - whatever index-pack outputs after "pack\t" or "keep\t". + * The hashes length is defined by the hash algorithm (40 hex + characters in SHA-1, 64 in SHA-256). It should match whatever + index-pack outputs after "pack\t" or "keep\t". packfile section * This section is only included if the client has sent 'want'