From patchwork Fri Feb 5 18:22:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Ackermann X-Patchwork-Id: 12070695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E90C0C433E6 for ; Fri, 5 Feb 2021 18:27:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 85D5664EFE for ; Fri, 5 Feb 2021 18:27:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231398AbhBEQoe (ORCPT ); Fri, 5 Feb 2021 11:44:34 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230064AbhBEQkx (ORCPT ); Fri, 5 Feb 2021 11:40:53 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 07E98C061786 for ; Fri, 5 Feb 2021 10:22:34 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id b3so8722810wrj.5 for ; Fri, 05 Feb 2021 10:22:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=73PanmKylEvqjb3EGhj/cA5vVvduH5Ze/XXY8XVnxuQ=; b=nsaWo0Acznz94RcuDVOkVHR7bwQq/9yxFQKHhZ1rrd4eIvg4AGTQklmSTdh6ehKQ4U DHEQoVF8YH56hubuTL/GFdzIXzAmUQc+BzZtvLCfTxHRytIuF82NkVSy99gYjeNBrm1Z +QXuMPgubIXUlNzzMyxDls3GPwrmyw990f0EYuw/MD5drCj0QNNh5kKftpO9ahXWIw/f 4EbL8D0Aa9y5JsPFKgdTMjz8czGhVFCxoDvtYslccUAqZNXSVt82gQADXAyIERj5Ddy9 rAbAtxgEfSR47crLj2QdAI1QRoXL2wb3kJDpTI/7vb6cdU+sF2YTgTqFsoFTE1kLsahC MWQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=73PanmKylEvqjb3EGhj/cA5vVvduH5Ze/XXY8XVnxuQ=; b=trumabbJQVrrb2FgDQMurOftNs07+/wETMAcUnlLWeAKf5RSpo0PjXARrcYgyXxYjz 1zwN0XXrLFD5VIt6QJlj501ejOFUQRp601MEHErGTvlQsZ+A7ecIiZmjBD+2fLQb7s01 VkWS2SgF64nei0+X+XL+ZqDSAsirJJNfmJH+JMe3MFNY3MjQuJVBGyT7OIx6AkFKurld M5h5c3v6EYltdtO27S7b+9WRlQnNhGajMbB/0M9LccNsEnSG1IfZmWHzNQeE+MqWY/Zb 1ZY+NZ0qnhN25zm4GAmM59nucfjhpcJLpr6l7XAA+2eC+1MbMUM7lcIUIGAAqGa13wvI 1thw== X-Gm-Message-State: AOAM533g1VJNmcN4wXUDuBJkgs2gi240L2oF6VeiUM+NWWqJ2qpFcfgc ws7fnSkxfXjsyrS7NQEm85cwFQAOr9w= X-Google-Smtp-Source: ABdhPJx1DrnkaMcj1oof76n15eFFn2AO67dNVamv4vP8/r0VhXX61EouiBSEKBMzzBtnDGfktxyt1A== X-Received: by 2002:a5d:6c66:: with SMTP id r6mr6343594wrz.86.1612549352368; Fri, 05 Feb 2021 10:22:32 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id z1sm13194619wru.70.2021.02.05.10.22.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 10:22:31 -0800 (PST) Message-Id: <7c78d0c1c30add2a8937e0a6fb725509738a858c.1612549349.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 05 Feb 2021 18:22:24 +0000 Subject: [PATCH v3 1/6] doc hash-function-transition: fix asciidoc output Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "brian m. carlson" , Thomas Ackermann , Thomas Ackermann Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Thomas Ackermann From: Thomas Ackermann Asciidoc requires lists to start with an empty line and uses different characters for indentation levels ("-", "*", "**", ...). For special symbols like a dash "--" has to be used and there is no double arrow "<->", so a left and right arrow "<-->" has to be combined for that. Lastly for verbatim output a newline followed by an indentation has to be used. Fix asciidoc output for lists, special characters and verbatim text while retaining the readabilty of the original text file. Signed-off-by: Thomas Ackermann --- .../technical/hash-function-transition.txt | 79 +++++++++++-------- 1 file changed, 45 insertions(+), 34 deletions(-) diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index 6fd20ebbc254..b23d23151a57 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -94,7 +94,7 @@ Overview -------- We introduce a new repository format extension. Repositories with this extension enabled use SHA-256 instead of SHA-1 to name their objects. -This affects both object names and object content --- both the names +This affects both object names and object content -- both the names of objects and all references to other objects within an object are switched to the new hash function. @@ -191,21 +191,21 @@ hash functions. They have the following format (all integers are in network byte order): - A header appears at the beginning and consists of the following: - - The 4-byte pack index signature: '\377t0c' - - 4-byte version number: 3 - - 4-byte length of the header section, including the signature and + * The 4-byte pack index signature: '\377t0c' + * 4-byte version number: 3 + * 4-byte length of the header section, including the signature and version number - - 4-byte number of objects contained in the pack - - 4-byte number of object formats in this pack index: 2 - - For each object format: - - 4-byte format identifier (e.g., 'sha1' for SHA-1) - - 4-byte length in bytes of shortened object names. This is the + * 4-byte number of objects contained in the pack + * 4-byte number of object formats in this pack index: 2 + * For each object format: + ** 4-byte format identifier (e.g., 'sha1' for SHA-1) + ** 4-byte length in bytes of shortened object names. This is the shortest possible length needed to make names in the shortened object name table unambiguous. - - 4-byte integer, recording where tables relating to this format + ** 4-byte integer, recording where tables relating to this format are stored in this index file, as an offset from the beginning. - - 4-byte offset to the trailer from the beginning of this file. - - Zero or more additional key/value pairs (4-byte key, 4-byte + * 4-byte offset to the trailer from the beginning of this file. + * Zero or more additional key/value pairs (4-byte key, 4-byte value). Only one key is supported: 'PSRC'. See the "Loose objects and unreachable objects" section for supported values and how this is used. All other keys are reserved. Readers must ignore @@ -213,37 +213,36 @@ network byte order): - Zero or more NUL bytes. This can optionally be used to improve the alignment of the full object name table below. - Tables for the first object format: - - A sorted table of shortened object names. These are prefixes of + * A sorted table of shortened object names. These are prefixes of the names of all objects in this pack file, packed together without offset values to reduce the cache footprint of the binary search for a specific object name. - - A table of full object names in pack order. This allows resolving + * A table of full object names in pack order. This allows resolving a reference to "the nth object in the pack file" (from a reachability bitmap or from the next table of another object format) to its object name. - - A table of 4-byte values mapping object name order to pack order. + * A table of 4-byte values mapping object name order to pack order. For an object in the table of sorted shortened object names, the value at the corresponding index in this table is the index in the previous table for that same object. - This can be used to look up the object in reachability bitmaps or to look up its name in another object format. - - A table of 4-byte CRC32 values of the packed object data, in the + * A table of 4-byte CRC32 values of the packed object data, in the order that the objects appear in the pack file. This is to allow compressed data to be copied directly from pack to pack during repacking without undetected data corruption. - - A table of 4-byte offset values. For an object in the table of + * A table of 4-byte offset values. For an object in the table of sorted shortened object names, the value at the corresponding index in this table indicates where that object can be found in the pack file. These are usually 31-bit pack file offsets, but large offsets are encoded as an index into the next table with the most significant bit set. - - A table of 8-byte offset entries (empty for pack files less than + * A table of 8-byte offset entries (empty for pack files less than 2 GiB). Pack files are organized with heavily used objects toward the front, so most object references should not need to refer to this table. @@ -252,10 +251,10 @@ network byte order): up to and not including the table of CRC32 values. - Zero or more NUL bytes. - The trailer consists of the following: - - A copy of the 20-byte SHA-256 checksum at the end of the + * A copy of the 20-byte SHA-256 checksum at the end of the corresponding packfile. - - 20-byte SHA-256 checksum of all of the above. + * 20-byte SHA-256 checksum of all of the above. Loose object index ~~~~~~~~~~~~~~~~~~ @@ -351,7 +350,7 @@ the following steps: 3. convert to sha256: open a new (sha256) packfile. Read the topologically sorted list just generated. For each object, inflate its sha1-content, convert to sha256-content, and write it to the sha256 - pack. Record the new sha1<->sha256 mapping entry for use in the idx. + pack. Record the new sha1<-->sha256 mapping entry for use in the idx. 4. sort: reorder entries in the new pack to match the order of objects in the pack the server generated and include blobs. Write a sha256 idx file @@ -391,6 +390,7 @@ existing "gpgsig" field. Its signed payload is the sha256-content of the commit object with any "gpgsig" and "gpgsig-sha256" fields removed. This means commits can be signed + 1. using SHA-1 only, as in existing signed commit objects 2. using both SHA-1 and SHA-256, by using both gpgsig-sha256 and gpgsig fields. @@ -408,6 +408,7 @@ sha256-content of the tag with its gpgsig-sha256 field and "-----BEGIN PGP SIGNATURE-----" delimited in-body signature removed. This means tags can be signed + 1. using SHA-1 only, as in existing signed tag objects 2. using both SHA-1 and SHA-256, by using gpgsig-sha256 and an in-body signature. @@ -598,7 +599,7 @@ The user can also explicitly specify which format to use for a particular revision specifier and for output, overriding the mode. For example: -git --output-format=sha1 log abac87a^{sha1}..f787cac^{sha256} + git --output-format=sha1 log abac87a^{sha1}..f787cac^{sha256} Choice of Hash -------------- @@ -636,6 +637,7 @@ We choose SHA-256. Transition plan --------------- Some initial steps can be implemented independently of one another: + - adding a hash function API (vtable) - teaching fsck to tolerate the gpgsig-sha256 field - excluding gpgsig-* from the fields copied by "git commit --amend" @@ -647,9 +649,9 @@ Some initial steps can be implemented independently of one another: - introducing index v3 - adding support for the PSRC field and safer object pruning - The first user-visible change is the introduction of the objectFormat extension (without compatObjectFormat). This requires: + - teaching fsck about this mode of operation - using the hash function API (vtable) when computing object names - signing objects and verifying signatures @@ -657,6 +659,7 @@ extension (without compatObjectFormat). This requires: repository Next comes introduction of compatObjectFormat: + - implementing the loose-object-idx - translating object names between object formats - translating object content between object formats @@ -669,6 +672,7 @@ Next comes introduction of compatObjectFormat: "Object names on the command line" above) The next step is supporting fetches and pushes to SHA-1 repositories: + - allow pushes to a repository using the compat format - generate a topologically sorted list of the SHA-1 names of fetched objects @@ -734,6 +738,7 @@ Using hash functions in parallel Objects newly created would be addressed by the new hash, but inside such an object (e.g. commit) it is still possible to address objects using the old hash function. + * You cannot trust its history (needed for bisectability) in the future without further work * Maintenance burden as the number of supported hash functions grows @@ -749,6 +754,7 @@ sha1-content based signatures. In other words, a single signature was used to attest to the object content using both hash functions. This had some advantages: + * Using one signature instead of two speeds up the signing process. * Having one signed payload with both hashes allows the signer to attest to the sha1-name and sha256-name referring to the same object. @@ -756,6 +762,7 @@ content using both hash functions. This had some advantages: to be detected quickly using current versions of git. However, it also came with disadvantages: + * Verifying a signed object requires access to the sha1-names of all objects it references, even after the transition is complete and translation table is no longer needed for anything else. To support @@ -782,16 +789,17 @@ Document History bmwill@google.com, jonathantanmy@google.com, jrnieder@gmail.com, sbeller@google.com -Initial version sent to -http://lore.kernel.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com +* Initial version sent to http://lore.kernel.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com 2017-03-03 jrnieder@gmail.com Incorporated suggestions from jonathantanmy and sbeller: + * describe purpose of signed objects with each hash type * redefine signed object verification using object content under the first hash function 2017-03-06 jrnieder@gmail.com + * Use SHA3-256 instead of SHA2 (thanks, Linus and brian m. carlson).[1][2] * Make sha3-based signatures a separate field, avoiding the need for "hash" and "nohash" fields (thanks to peff[3]). @@ -805,6 +813,7 @@ Incorporated suggestions from jonathantanmy and sbeller: especially Junio). 2017-09-27 jrnieder@gmail.com, sbeller@google.com + * use placeholder NewHash instead of SHA3-256 * describe criteria for picking a hash function. * include a transition plan (thanks especially to Brandon Williams @@ -816,12 +825,14 @@ Incorporated suggestions from jonathantanmy and sbeller: Later history: - See the history of this file in git.git for the history of subsequent - edits. This document history is no longer being maintained as it - would now be superfluous to the commit log +* See the history of this file in git.git for the history of subsequent + edits. This document history is no longer being maintained as it + would now be superfluous to the commit log + +References: -[1] http://lore.kernel.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/ -[2] http://lore.kernel.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/ -[3] http://lore.kernel.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/ -[4] http://lore.kernel.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net -[5] https://lore.kernel.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/ + [1] http://lore.kernel.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/ + [2] http://lore.kernel.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/ + [3] http://lore.kernel.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/ + [4] http://lore.kernel.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net + [5] https://lore.kernel.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/ From patchwork Fri Feb 5 18:22:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Ackermann X-Patchwork-Id: 12070697 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97CC0C433E0 for ; Fri, 5 Feb 2021 18:27:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 04A2564EE8 for ; Fri, 5 Feb 2021 18:27:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233000AbhBEQon (ORCPT ); Fri, 5 Feb 2021 11:44:43 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44126 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233308AbhBEQky (ORCPT ); Fri, 5 Feb 2021 11:40:54 -0500 Received: from mail-wr1-x42a.google.com (mail-wr1-x42a.google.com [IPv6:2a00:1450:4864:20::42a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 62FA5C06178A for ; Fri, 5 Feb 2021 10:22:35 -0800 (PST) Received: by mail-wr1-x42a.google.com with SMTP id p15so8693940wrq.8 for ; Fri, 05 Feb 2021 10:22:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=cUIxkiOWcZNSGyVaoSleR4SOVvfUesYE0h+OiU8ZPts=; b=ZFK5gPjb/0fvMc4sNuv93cyyjrTM/TDXgyThGNWVXJz9mrDj43cVIu/YZC//PIDcMC Tmk2RcOVtk1cWYKIzfzGk8m++78qRPe6iHgIumgSf99lc6zNlgYgzECzYpldPcsl0BFQ ltI8N+fPk6RxkR0+CGH7VYRNGEk8JyGPr/fL6z5CWq38JIwVYnGRjn02Il0QnnHawE4F 3QGYwP20gnoURblSgvqedifbi5uV6egsw4WdFk7sfIQRnXFeSrCBlBOyerwcobRa8Mew +xAhG1SZf3D04DfbJH2isBeJFYL/W3YUh3HKTK98qLpHPoW4Yq1i3mEZiNRQbv7EY/QZ 7nKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=cUIxkiOWcZNSGyVaoSleR4SOVvfUesYE0h+OiU8ZPts=; b=lM5I0pltn9EorvU/gtrvJmGHiN2vN9ujsRsVyGs/6YCCt6Ujju4FZoxx4NnAAVgN/3 T1pUo877F91H5Fp+wxNscmKFJo/O8EaW7H+6ayT03DlKxY3rQu0pdznL2l3LcKgdZxxM 5IpuQcIxopxdXJ7T2jHYU7saqvGY5kb6sHFt5AtNTJW+Cx6DECOb6ITsln7OCdVzv9aR MHOQS5uxVWfwgfPcV4nF5O5SE14b1xQ/gD4Rcn5YWHTInLrrKmrEHzQrMCMqvfV7ZdFL clpQb6HprMEOSws3pdWYU//w1mAJhxFV+NRetU+4W8JBrTRfJDMB8c04yZEvLkcNdcyO Mm/w== X-Gm-Message-State: AOAM532l0JhGvFq0FUJ4yyxUvRYG0sgLV1GZJOpRcK2hiIg+Vle3Sknt IvZ9dN3ZkqAyNnrO62Bpu+SU814f7Lg= X-Google-Smtp-Source: ABdhPJxmYTz09tlO3FLpY5mSKlnusSQGs032uYTqFDPTYemVhfRTlclnXxs5QYs0VwJdTTdD65wVoQ== X-Received: by 2002:adf:ea51:: with SMTP id j17mr6732799wrn.382.1612549353610; Fri, 05 Feb 2021 10:22:33 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o18sm1509760wmp.19.2021.02.05.10.22.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 10:22:32 -0800 (PST) Message-Id: <69ebc9a8f19ac6f37415a47dc315ca3a2f6938a7.1612549349.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 05 Feb 2021 18:22:25 +0000 Subject: [PATCH v3 2/6] doc hash-function-transition: use SHA-1 and SHA-256 consistently Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "brian m. carlson" , Thomas Ackermann , Thomas Ackermann Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Thomas Ackermann From: Thomas Ackermann Use SHA-1 and SHA-256 instead of sha1 and sha256 when referring to the hash type. Signed-off-by: Thomas Ackermann --- .../technical/hash-function-transition.txt | 126 +++++++++--------- 1 file changed, 63 insertions(+), 63 deletions(-) diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index b23d23151a57..8c01608cbfa0 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -107,7 +107,7 @@ mapping to allow naming objects using either their SHA-1 and SHA-256 names interchangeably. "git cat-file" and "git hash-object" gain options to display an object -in its sha1 form and write an object given its sha1 form. This +in its SHA-1 form and write an object given its SHA-1 form. This requires all objects referenced by that object to be present in the object database so that they can be named using the appropriate name (using the bidirectional hash mapping). @@ -115,7 +115,7 @@ object database so that they can be named using the appropriate name Fetches from a SHA-1 based server convert the fetched objects into SHA-256 form and record the mapping in the bidirectional mapping table (see below for details). Pushes to a SHA-1 based server convert the -objects being pushed into sha1 form so the server does not have to be +objects being pushed into SHA-1 form so the server does not have to be aware of the hash function the client is using. Detailed Design @@ -151,38 +151,38 @@ repository extensions. Object names ~~~~~~~~~~~~ -Objects can be named by their 40 hexadecimal digit sha1-name or 64 -hexadecimal digit sha256-name, plus names derived from those (see +Objects can be named by their 40 hexadecimal digit SHA-1 name or 64 +hexadecimal digit SHA-256 name, plus names derived from those (see gitrevisions(7)). -The sha1-name of an object is the SHA-1 of the concatenation of its -type, length, a nul byte, and the object's sha1-content. This is the +The SHA-1 name of an object is the SHA-1 of the concatenation of its +type, length, a nul byte, and the object's SHA-1 content. This is the traditional used in Git to name objects. -The sha256-name of an object is the SHA-256 of the concatenation of its -type, length, a nul byte, and the object's sha256-content. +The SHA-256 name of an object is the SHA-256 of the concatenation of its +type, length, a nul byte, and the object's SHA-256 content. Object format ~~~~~~~~~~~~~ The content as a byte sequence of a tag, commit, or tree object named -by sha1 and sha256 differ because an object named by sha256-name refers to -other objects by their sha256-names and an object named by sha1-name -refers to other objects by their sha1-names. +by SHA-1 and SHA-256 differ because an object named by SHA-256 name refers to +other objects by their SHA-256 names and an object named by SHA-1 name +refers to other objects by their SHA-1 names. -The sha256-content of an object is the same as its sha1-content, except -that objects referenced by the object are named using their sha256-names -instead of sha1-names. Because a blob object does not refer to any -other object, its sha1-content and sha256-content are the same. +The SHA-256 content of an object is the same as its SHA-1 content, except +that objects referenced by the object are named using their SHA-256 names +instead of SHA-1 names. Because a blob object does not refer to any +other object, its SHA-1 content and SHA-256 content are the same. -The format allows round-trip conversion between sha256-content and -sha1-content. +The format allows round-trip conversion between SHA-256 content and +SHA-1 content. Object storage ~~~~~~~~~~~~~~ Loose objects use zlib compression and packed objects use the packed format described in Documentation/technical/pack-format.txt, just like -today. The content that is compressed and stored uses sha256-content -instead of sha1-content. +today. The content that is compressed and stored uses SHA-256 content +instead of SHA-1 content. Pack index ~~~~~~~~~~ @@ -287,18 +287,18 @@ To remove entries (e.g. in "git pack-refs" or "git-prune"): Translation table ~~~~~~~~~~~~~~~~~ -The index files support a bidirectional mapping between sha1-names -and sha256-names. The lookup proceeds similarly to ordinary object -lookups. For example, to convert a sha1-name to a sha256-name: +The index files support a bidirectional mapping between SHA-1 names +and SHA-256 names. The lookup proceeds similarly to ordinary object +lookups. For example, to convert a SHA-1 name to a SHA-256 name: 1. Look for the object in idx files. If a match is present in the - idx's sorted list of truncated sha1-names, then: - a. Read the corresponding entry in the sha1-name order to pack + idx's sorted list of truncated SHA-1 names, then: + a. Read the corresponding entry in the SHA-1 name order to pack name order mapping. - b. Read the corresponding entry in the full sha1-name table to + b. Read the corresponding entry in the full SHA-1 name table to verify we found the right object. If it is, then - c. Read the corresponding entry in the full sha256-name table. - That is the object's sha256-name. + c. Read the corresponding entry in the full SHA-256 name table. + That is the object's SHA-256 name. 2. Check for a loose object. Read lines from loose-object-idx until we find a match. @@ -312,10 +312,10 @@ Since all operations that make new objects (e.g., "git commit") add the new objects to the corresponding index, this mapping is possible for all objects in the object store. -Reading an object's sha1-content -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -The sha1-content of an object can be read by converting all sha256-names -its sha256-content references to sha1-names using the translation table. +Reading an object's SHA-1 content +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The SHA-1 content of an object can be read by converting all SHA-256 names +its SHA-256 content references to SHA-1 names using the translation table. Fetch ~~~~~ @@ -338,7 +338,7 @@ the following steps: 1. index-pack: inflate each object in the packfile and compute its SHA-1. Objects can contain deltas in OBJ_REF_DELTA format against objects the client has locally. These objects can be looked up - using the translation table and their sha1-content read as + using the translation table and their SHA-1 content read as described above to resolve the deltas. 2. topological sort: starting at the "want"s from the negotiation phase, walk through objects in the pack and emit a list of them, @@ -347,12 +347,12 @@ the following steps: (This list only contains objects reachable from the "wants". If the pack from the server contained additional extraneous objects, then they will be discarded.) -3. convert to sha256: open a new (sha256) packfile. Read the topologically +3. convert to SHA-256: open a new SHA-256 packfile. Read the topologically sorted list just generated. For each object, inflate its - sha1-content, convert to sha256-content, and write it to the sha256 - pack. Record the new sha1<-->sha256 mapping entry for use in the idx. + SHA-1 content, convert to SHA-256 content, and write it to the SHA-256 + pack. Record the new SHA-1<-->SHA-256 mapping entry for use in the idx. 4. sort: reorder entries in the new pack to match the order of objects - in the pack the server generated and include blobs. Write a sha256 idx + in the pack the server generated and include blobs. Write a SHA-256 idx file 5. clean up: remove the SHA-1 based pack file, index, and topologically sorted list obtained from the server in steps 1 @@ -377,16 +377,16 @@ experimenting to get this to perform well. Push ~~~~ Push is simpler than fetch because the objects referenced by the -pushed objects are already in the translation table. The sha1-content +pushed objects are already in the translation table. The SHA-1 content of each object being pushed can be read as described in the "Reading -an object's sha1-content" section to generate the pack written by git +an object's SHA-1 content" section to generate the pack written by git send-pack. Signed Commits ~~~~~~~~~~~~~~ We add a new field "gpgsig-sha256" to the commit object format to allow signing commits without relying on SHA-1. It is similar to the -existing "gpgsig" field. Its signed payload is the sha256-content of the +existing "gpgsig" field. Its signed payload is the SHA-256 content of the commit object with any "gpgsig" and "gpgsig-sha256" fields removed. This means commits can be signed @@ -404,7 +404,7 @@ Signed Tags ~~~~~~~~~~~ We add a new field "gpgsig-sha256" to the tag object format to allow signing tags without relying on SHA-1. Its signed payload is the -sha256-content of the tag with its gpgsig-sha256 field and "-----BEGIN PGP +SHA-256 content of the tag with its gpgsig-sha256 field and "-----BEGIN PGP SIGNATURE-----" delimited in-body signature removed. This means tags can be signed @@ -416,11 +416,11 @@ This means tags can be signed Mergetag embedding ~~~~~~~~~~~~~~~~~~ -The mergetag field in the sha1-content of a commit contains the -sha1-content of a tag that was merged by that commit. +The mergetag field in the SHA-1 content of a commit contains the +SHA-1 content of a tag that was merged by that commit. -The mergetag field in the sha256-content of the same commit contains the -sha256-content of the same tag. +The mergetag field in the SHA-256 content of the same commit contains the +SHA-256 content of the same tag. Submodules ~~~~~~~~~~ @@ -495,7 +495,7 @@ Caveats ------- Invalid objects ~~~~~~~~~~~~~~~ -The conversion from sha1-content to sha256-content retains any +The conversion from SHA-1 content to SHA-256 content retains any brokenness in the original object (e.g., tree entry modes encoded with leading 0, tree objects whose paths are not sorted correctly, and commit objects without an author or committer). This is a deliberate @@ -514,15 +514,15 @@ allow lifting this restriction. Alternates ~~~~~~~~~~ -For the same reason, a sha256 repository cannot borrow objects from a -sha1 repository using objects/info/alternates or +For the same reason, a SHA-256 repository cannot borrow objects from a +SHA-1 repository using objects/info/alternates or $GIT_ALTERNATE_OBJECT_REPOSITORIES. git notes ~~~~~~~~~ -The "git notes" tool annotates objects using their sha1-name as key. +The "git notes" tool annotates objects using their SHA-1 name as key. This design does not describe a way to migrate notes trees to use -sha256-names. That migration is expected to happen separately (for +SHA-256 names. That migration is expected to happen separately (for example using a file at the root of the notes tree to describe which hash it uses). @@ -556,7 +556,7 @@ unclear: Git 2.12 -Does this mean Git v2.12.0 is the commit with sha1-name +Does this mean Git v2.12.0 is the commit with SHA-1 name e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7 or the commit with new-40-digit-hash-name e7e07d5a4fcc2a203d9873968ad3e6bd4d7419d7? @@ -676,7 +676,7 @@ The next step is supporting fetches and pushes to SHA-1 repositories: - allow pushes to a repository using the compat format - generate a topologically sorted list of the SHA-1 names of fetched objects -- convert the fetched packfile to sha256 format and generate an idx +- convert the fetched packfile to SHA-256 format and generate an idx file - re-sort to match the order of objects in the fetched packfile @@ -748,38 +748,38 @@ using the old hash function. Signed objects with multiple hashes ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Instead of introducing the gpgsig-sha256 field in commit and tag objects -for sha256-content based signatures, an earlier version of this design -added "hash sha256 " fields to strengthen the existing -sha1-content based signatures. +for SHA-256 content based signatures, an earlier version of this design +added "hash sha256 " fields to strengthen the existing +SHA-1 content based signatures. In other words, a single signature was used to attest to the object content using both hash functions. This had some advantages: * Using one signature instead of two speeds up the signing process. * Having one signed payload with both hashes allows the signer to - attest to the sha1-name and sha256-name referring to the same object. + attest to the SHA-1 name and SHA-256 name referring to the same object. * All users consume the same signature. Broken signatures are likely to be detected quickly using current versions of git. However, it also came with disadvantages: -* Verifying a signed object requires access to the sha1-names of all +* Verifying a signed object requires access to the SHA-1 names of all objects it references, even after the transition is complete and translation table is no longer needed for anything else. To support - this, the design added fields such as "hash sha1 tree " - and "hash sha1 parent " to the sha256-content of a signed + this, the design added fields such as "hash sha1 tree " + and "hash sha1 parent " to the SHA-256 content of a signed commit, complicating the conversion process. -* Allowing signed objects without a sha1 (for after the transition is +* Allowing signed objects without a SHA-1 (for after the transition is complete) complicated the design further, requiring a "nohash sha1" - field to suppress including "hash sha1" fields in the sha256-content + field to suppress including "hash sha1" fields in the SHA-256 content and signed payload. Lazily populated translation table ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Some of the work of building the translation table could be deferred to push time, but that would significantly complicate and slow down pushes. -Calculating the sha1-name at object creation time at the same time it is -being streamed to disk and having its sha256-name calculated should be +Calculating the SHA-1 name at object creation time at the same time it is +being streamed to disk and having its SHA-256 name calculated should be an acceptable cost. Document History @@ -801,7 +801,7 @@ Incorporated suggestions from jonathantanmy and sbeller: 2017-03-06 jrnieder@gmail.com * Use SHA3-256 instead of SHA2 (thanks, Linus and brian m. carlson).[1][2] -* Make sha3-based signatures a separate field, avoiding the need for +* Make SHA3-based signatures a separate field, avoiding the need for "hash" and "nohash" fields (thanks to peff[3]). * Add a sorting phase to fetch (thanks to Junio for noticing the need for this). From patchwork Fri Feb 5 18:22:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Ackermann X-Patchwork-Id: 12070689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id F2580C433E0 for ; Fri, 5 Feb 2021 18:26:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9F6EA64E75 for ; Fri, 5 Feb 2021 18:26:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231630AbhBEQo0 (ORCPT ); Fri, 5 Feb 2021 11:44:26 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44134 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233269AbhBEQkz (ORCPT ); Fri, 5 Feb 2021 11:40:55 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4CA42C06178B for ; Fri, 5 Feb 2021 10:22:36 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id c4so8682809wru.9 for ; Fri, 05 Feb 2021 10:22:36 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=8+g1/GHHS722X/SiBgQQpJwcszLIHLP4uLMT0hi6i/c=; b=NnAZM7MVK19a6QTyg0uDYQOhLK+2DdozsvlkEmcpCVQ5aqZ9iEv19f4B/MYT2ktZWh s+nlYfWy+n4T/Zj3mB4yjcXXNjcdk3+qfpKhVptXTAp8mcWMAM8eekjSyy1Uj9pUW8IO 1ALKL+QyAk2h8umGNHVf/iUvaEfKwxKs4NKZ07qNGkis99yh55F52Y0Wi4wvAwmmA5Zd NDLjOZe9HdFMx+PDiWypDpB7tIcgQUTcs3u3hkUpZL8OMQMI1mEUAUQhjKXN1iI3SnvW op5ur43AZzKBp4q6n7F4YjIp8ywi9xN9UZ/Gb6xnGkm+ucsMpFQsqsV0gwD8/yQ9BQnp 75pw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=8+g1/GHHS722X/SiBgQQpJwcszLIHLP4uLMT0hi6i/c=; b=HoobnCbsrPYWlGuUodT4NLBOFKcMvv+lLPsDf4L9VJBhG8N3Gnyg0aFKKJZGjOpnjz 5FnRHeSPi+lAJ1D4Y0/7ec7nZpa992/dzFcHzCNLAgFh2tJV08tUnJrIXcqLmEmSlr8o 42q4F9UCvKYVoH0Sx2d+O0kt4pFgflhpU9UgZ4Lu5xATRkyI+YjRvtk7XtfabnExWXTc M6j3RIo6wc/WwEAXgxk33DOwiF7AIRvOzK4zAjoIrlLs/DR1kpeOICf6zUiMyrT2um6F 91/bUHkc2VBsbbFJWveVhljN55v/NBtuFx/XlHe7gCkpMNd0jD7HyM8OYz/DnRzBdfaP 0Dpg== X-Gm-Message-State: AOAM532ueupjST8XCahjykD6Up59SCOhY7i0UQIpG6pYn9Xzxji/8Nlb esk7zYXHgzSgVJd5UBRgj+mdq/xNNVY= X-Google-Smtp-Source: ABdhPJy3URAV1IEj2pETpkddni16Lz5JuvFYofy5i+3lMKhOgLqUcwHBaJE4XDggH+77/+YitALPKA== X-Received: by 2002:adf:e381:: with SMTP id e1mr6316066wrm.22.1612549354874; Fri, 05 Feb 2021 10:22:34 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id e11sm13022702wrt.35.2021.02.05.10.22.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 10:22:33 -0800 (PST) Message-Id: <06b781206e4cd5cfe23747f77d9228e5dc7e42c4.1612549349.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 05 Feb 2021 18:22:26 +0000 Subject: [PATCH v3 3/6] doc hash-function-transition: use upper case consistently Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "brian m. carlson" , Thomas Ackermann , Thomas Ackermann Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Thomas Ackermann From: Thomas Ackermann Use upper case consistently in Document History. Signed-off-by: Thomas Ackermann --- .../technical/hash-function-transition.txt | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index 8c01608cbfa0..9e13919a0e5b 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -794,8 +794,8 @@ sbeller@google.com 2017-03-03 jrnieder@gmail.com Incorporated suggestions from jonathantanmy and sbeller: -* describe purpose of signed objects with each hash type -* redefine signed object verification using object content under the +* Describe purpose of signed objects with each hash type +* Redefine signed object verification using object content under the first hash function 2017-03-06 jrnieder@gmail.com @@ -814,13 +814,13 @@ Incorporated suggestions from jonathantanmy and sbeller: 2017-09-27 jrnieder@gmail.com, sbeller@google.com -* use placeholder NewHash instead of SHA3-256 -* describe criteria for picking a hash function. -* include a transition plan (thanks especially to Brandon Williams +* Use placeholder NewHash instead of SHA3-256 +* Describe criteria for picking a hash function. +* Include a transition plan (thanks especially to Brandon Williams for fleshing these ideas out) -* define the translation table (thanks, Shawn Pearce[5], Jonathan +* Define the translation table (thanks, Shawn Pearce[5], Jonathan Tan, and Masaya Suzuki) -* avoid loose object overhead by packing more aggressively in +* Avoid loose object overhead by packing more aggressively in "git gc --auto" Later history: From patchwork Fri Feb 5 18:22:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Ackermann X-Patchwork-Id: 12070685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86742C433DB for ; Fri, 5 Feb 2021 18:26:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3D0B464DA1 for ; Fri, 5 Feb 2021 18:26:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233343AbhBEQnS (ORCPT ); Fri, 5 Feb 2021 11:43:18 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44136 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233317AbhBEQkz (ORCPT ); Fri, 5 Feb 2021 11:40:55 -0500 Received: from mail-wr1-x433.google.com (mail-wr1-x433.google.com [IPv6:2a00:1450:4864:20::433]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 599A6C06178C for ; Fri, 5 Feb 2021 10:22:37 -0800 (PST) Received: by mail-wr1-x433.google.com with SMTP id c12so8712908wrc.7 for ; Fri, 05 Feb 2021 10:22:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=5DIQw4oiFcBFObujiOKsv1mveVboPnS40X7HNxioWC0=; b=JpQx6dbsKKEIqSGulCkzC0hACcluFSBmIQ+F/hXTel+breYrzagh6f35pKGA0OKeWM hpiU7gK4pGD0Lixnk7UCp3oZ6dopHtqXShTELz+KuU7tBjvkQMQ0b0gSuuftAHGC7Yc6 pcGFXuhVDanZ1I2KasXCEeo9nP4dhRIj5/gn+8F/AAU87O57iaGp4UVCFW+KVLCkqd0c QOKoV2OEdVOAQci5T8/ei5skA2U9bBvRzUbHHjm0y79MOk0f3gFzwKX+HML/zosqNwHS gPXU0otXUj/mp6hVk5ZO0/pu3EdP2aoTHysRdbawYFhhJzgCHGVlccEyRDQvnTgrihyH VXbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=5DIQw4oiFcBFObujiOKsv1mveVboPnS40X7HNxioWC0=; b=Hhb6FA9LjJvW75Ts0YfqvIwttvouXO9yTXP+zGRr8kKNVUmbHTpbhs9gci+JRC9SrT kD5w0CBsXzSu3Mgfzpuc9DqIcUTT2iu9bBKm+Lokv92NYxruY02cO83lsIyZ+n9KcrGh ntWzwFh/WaCnLT018aZil4PPr5Vl0wCtSXjVLkee/tBEsqxMziO406ZPELTMCjZn80TS HsNnu7QJ+RcZfrXROJe4QJKUwowMNRmpq7e9ox9G/1pniL4PTcpbSerIFcTVBbK5F0CD pB3xkib5u771+rOfHgIOVcPZt20Gp/kfsi5zCBCHhvfasQMkKFTSsU08QJ/w50k4/HNv VuLg== X-Gm-Message-State: AOAM533dJYbULdUTHuUTYrOkNVE0nl737VYZyZDg8aOOhogn1BIJnOXu 8uf/vMaqZX5Xa5I6egW565a3MCbEvMs= X-Google-Smtp-Source: ABdhPJwH5chv9AK6cJAq24fFDAUAcUyztEarBU7oezb48m9VKBEsVdz8sTs8xpgUDoiPbStf3OhrJA== X-Received: by 2002:adf:f4c1:: with SMTP id h1mr6495063wrp.102.1612549355995; Fri, 05 Feb 2021 10:22:35 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 35sm14735921wrn.42.2021.02.05.10.22.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 10:22:35 -0800 (PST) Message-Id: <7a29f06c3f2569188cf73ca6ae55ab93c1dd1893.1612549349.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Fri, 05 Feb 2021 18:22:27 +0000 Subject: [PATCH v3 4/6] doc hash-function-transition: fix incomplete sentence Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "brian m. carlson" , Thomas Ackermann , Thomas Ackermann Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Thomas Ackermann From: Thomas Ackermann Signed-off-by: Thomas Ackermann --- Documentation/technical/hash-function-transition.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index 9e13919a0e5b..5ff9ee027cff 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -315,7 +315,7 @@ for all objects in the object store. Reading an object's SHA-1 content ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The SHA-1 content of an object can be read by converting all SHA-256 names -its SHA-256 content references to SHA-1 names using the translation table. +of its SHA-256 content references to SHA-1 names using the translation table. Fetch ~~~~~ From patchwork Fri Feb 5 18:22:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Ackermann X-Patchwork-Id: 12070687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4E69C433DB for ; Fri, 5 Feb 2021 18:26:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8955D64E75 for ; Fri, 5 Feb 2021 18:26:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233313AbhBEQno (ORCPT ); Fri, 5 Feb 2021 11:43:44 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233331AbhBEQk5 (ORCPT ); Fri, 5 Feb 2021 11:40:57 -0500 Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 98E8EC061793 for ; Fri, 5 Feb 2021 10:22:38 -0800 (PST) Received: by mail-wm1-x331.google.com with SMTP id w4so6747780wmi.4 for ; Fri, 05 Feb 2021 10:22:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=l6FtnOpglT3mK3SEVJz/3kN9LAgIBU9mWvZcmxVv3ck=; b=EVhjuSDASBveU775niA0EpN7HnMVg7LkMv0M6NWCctC7CayY8PaBsKPgFGGewLKCde aVIEV+FMKJ2v5oiIYZtYQ5GpdjrWCs6o+PV0CaDG208Zdru13dk8gkju8qx4Aaaj/UmX C8MwqMBGCYkZ3dLYAZdigg/M1FF6Jy4PddvWjgOyKYT9Bzxxf3EaF1ZCTs1Us/2hvcyR dyX6dC2fsHB3yRcoBVdJ+UG1lujNtNMJXWe7zjrGo9Ef1cPjuvFt1YzsOhpjrD4Gcrll TW3pKzn4QuYTMP0l20LYDBpuFuZAZwLL6CiDwoueZufa/wiPOKQTTqF/30u7tWBeLfzG 5Rvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=l6FtnOpglT3mK3SEVJz/3kN9LAgIBU9mWvZcmxVv3ck=; b=L8ah97NuQOafPA/wu0E7EP46455RJiGY69267nxiTm3WpbaPBwR5xvU2oM3BHEyaGj 31RxQgdpzl10Aqmc6GEYIweTehhHMFesj0bl5GcjI4ZEdZvN4ArCisMfPMGT+jbTqTOR azrGKLqEdtcZShvLAIg6RyJ4AwKfErp7DCgNFJXH0MMOMt0xTdJBrehRoCvuUKANbkiS BpCYF+tfSy/a0/5ejCKoovwNv7pRqb0rJtuHAimecfUoYr7WtaR8lkkhsk7Y4Kv4kc8I Bw9Fnwt3Z0wPN4hYOLuaFvSRNyLfGsqEhFsCDVsheok8OnuCNLDrkBQc23bxD9hDm7ck pAbw== X-Gm-Message-State: AOAM533ymuQO9L9l4s6erAZiKDreg8kHF7VVdMTkWQtKn1t1eNL5hbAH AQettLiOUogsEglCBqPtN/7sQptBBks= X-Google-Smtp-Source: ABdhPJxq/E12JfVxTkSEL7muewB7+1qbRrjWW8LF2I2gbyBimBeWvKea1PCDTSTs4oUi77/74FU4VA== X-Received: by 2002:a1c:ab57:: with SMTP id u84mr4807717wme.115.1612549357092; Fri, 05 Feb 2021 10:22:37 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o18sm9413286wmh.20.2021.02.05.10.22.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 10:22:36 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 05 Feb 2021 18:22:28 +0000 Subject: [PATCH v3 5/6] doc hash-function-transition: move rationale upwards Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "brian m. carlson" , Thomas Ackermann , Thomas Ackermann Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Thomas Ackermann From: Thomas Ackermann Move rationale for new hash function to beginning of document so that it appears before the concrete move to SHA-256 is described. Remove some of the details about SHA-1 weaknesses and add references to the details on how the new hash function was chosen instead. Signed-off-by: Thomas Ackermann --- .../technical/hash-function-transition.txt | 76 +++++++++---------- 1 file changed, 34 insertions(+), 42 deletions(-) diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index 5ff9ee027cff..0c4cb98cd4e9 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -33,16 +33,9 @@ researchers. On 23 February 2017 the SHAttered attack Git v2.13.0 and later subsequently moved to a hardened SHA-1 implementation by default, which isn't vulnerable to the SHAttered -attack. +attack, but SHA-1 is still weak. -Thus Git has in effect already migrated to a new hash that isn't SHA-1 -and doesn't share its vulnerabilities, its new hash function just -happens to produce exactly the same output for all known inputs, -except two PDFs published by the SHAttered researchers, and the new -implementation (written by those researchers) claims to detect future -cryptanalytic collision attacks. - -Regardless, it's considered prudent to move past any variant of SHA-1 +Thus it's considered prudent to move past any variant of SHA-1 to a new hash. There's no guarantee that future attacks on SHA-1 won't be published in the future, and those attacks may not have viable mitigations. @@ -57,6 +50,38 @@ SHA-1 still possesses the other properties such as fast object lookup and safe error checking, but other hash functions are equally suitable that are believed to be cryptographically secure. +Choice of Hash +-------------- +The hash to replace the hardened SHA-1 should be stronger than SHA-1 +was: we would like it to be trustworthy and useful in practice for at +least 10 years. + +Some other relevant properties: + +1. A 256-bit hash (long enough to match common security practice; not + excessively long to hurt performance and disk usage). + +2. High quality implementations should be widely available (e.g., in + OpenSSL and Apple CommonCrypto). + +3. The hash function's properties should match Git's needs (e.g. Git + requires collision and 2nd preimage resistance and does not require + length extension resistance). + +4. As a tiebreaker, the hash should be fast to compute (fortunately + many contenders are faster than SHA-1). + +There were several contenders for a successor hash to SHA-1, including +SHA-256, SHA-512/256, SHA-256x16, K12, and BLAKE2bp-256. + +In late 2018 the project picked SHA-256 as its successor hash. + +See 0ed8d8da374 (doc hash-function-transition: pick SHA-256 as +NewHash, 2018-08-04) and numerous mailing list threads at the time, +particularly the one starting at +https://lore.kernel.org/git/20180609224913.GC38834@genre.crustytoothpaste.net/ +for more information. + Goals ----- 1. The transition to SHA-256 can be done one local repository at a time. @@ -601,39 +626,6 @@ example: git --output-format=sha1 log abac87a^{sha1}..f787cac^{sha256} -Choice of Hash --------------- -In early 2005, around the time that Git was written, Xiaoyun Wang, -Yiqun Lisa Yin, and Hongbo Yu announced an attack finding SHA-1 -collisions in 2^69 operations. In August they published details. -Luckily, no practical demonstrations of a collision in full SHA-1 were -published until 10 years later, in 2017. - -Git v2.13.0 and later subsequently moved to a hardened SHA-1 -implementation by default that mitigates the SHAttered attack, but -SHA-1 is still believed to be weak. - -The hash to replace this hardened SHA-1 should be stronger than SHA-1 -was: we would like it to be trustworthy and useful in practice for at -least 10 years. - -Some other relevant properties: - -1. A 256-bit hash (long enough to match common security practice; not - excessively long to hurt performance and disk usage). - -2. High quality implementations should be widely available (e.g., in - OpenSSL and Apple CommonCrypto). - -3. The hash function's properties should match Git's needs (e.g. Git - requires collision and 2nd preimage resistance and does not require - length extension resistance). - -4. As a tiebreaker, the hash should be fast to compute (fortunately - many contenders are faster than SHA-1). - -We choose SHA-256. - Transition plan --------------- Some initial steps can be implemented independently of one another: From patchwork Fri Feb 5 18:22:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Ackermann X-Patchwork-Id: 12070683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 340F0C433DB for ; Fri, 5 Feb 2021 18:25:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DF5E164EE8 for ; Fri, 5 Feb 2021 18:25:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233232AbhBEQnL (ORCPT ); Fri, 5 Feb 2021 11:43:11 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44148 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233343AbhBEQk6 (ORCPT ); Fri, 5 Feb 2021 11:40:58 -0500 Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id BA72BC061794 for ; Fri, 5 Feb 2021 10:22:39 -0800 (PST) Received: by mail-wm1-x335.google.com with SMTP id f16so6737621wmq.5 for ; Fri, 05 Feb 2021 10:22:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=C6R78wCggJig/b9jMpwT9VHryu0qKW6zI6S7/k2ERS4=; b=PZprj07iqf6pTn6uscHDygKpNSPThfQunFbb1lq1QG8Y5GBLRmD/EpU7W8xlAvRKdU 6vQSY+4OGXPHUdU3lmVpnQDLEzqYpTW3R/6DFqkWjokK8h+wkjtwVJ9Tgn0+G7LDJMPv Gh/5s8Tt6AR5OZpDPIOdUJISMJ3cWziQ2asvjmfJcOtwoCSONvsOXb8SldAiI7u+oRMZ /ZlupNjBM0hP6igor3glnCOo85RckQFdNxTRZymtOWBNrpVBIrWGAaIa2PJhRorJekL3 knL4vR6ysyxQ3VLH1I6SBKjJVAuUeNaM5eWk1k2RFiVWH64v8HfwxgJ6LKwWj84QKUDp XBVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=C6R78wCggJig/b9jMpwT9VHryu0qKW6zI6S7/k2ERS4=; b=DcukTjN1z6sdwtnJsTbCBYaMN1mV5tczSZZ5plN2cbMZa7FitNrk9n1KlMmfXuYaVS Y3F2dTneQ69jMNKChInRNNpm1H6i43y5rUp8nZ6zAaYt6TAaEqeeW5S8HeSiEN5Ab992 folGf8KAmFk0d+0Xn3PSkCt7l2y05nsjP9TJGHOMbyKgzy3KGtcfP+7O5dNvJfBSSPmM IU0uzLBgarll/2RQ15y5uAZv8DRZBjwuzov23YWVwGOd+5xp3/utlpUxkekjH6QfeeCY pk6I1atoUvww0nVR6LL2D3Lfale6CTFFTLuYLR7YF5b8Y4oVw8XMmQkyMcxvDhUkAwC1 Mehw== X-Gm-Message-State: AOAM5302p4ZsHkNxT0+/8Pg7IFABHCfYkZXmMKakrplBTzTwjVGs6xho cIh5MN4c64hBHZeI0rj9VT7KrhgjySc= X-Google-Smtp-Source: ABdhPJyxWgkDQGhfQuy/shFUxpWR5F9cuwkSLN2PU3mnPPkSvGFEa2cMwFF7AC/VC4Xh/6DY31Lk9w== X-Received: by 2002:a1c:e903:: with SMTP id q3mr4663355wmc.100.1612549358245; Fri, 05 Feb 2021 10:22:38 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s64sm10054802wms.21.2021.02.05.10.22.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 05 Feb 2021 10:22:37 -0800 (PST) Message-Id: In-Reply-To: References: Date: Fri, 05 Feb 2021 18:22:29 +0000 Subject: [PATCH v3 6/6] doc: use https links Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: Junio C Hamano , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsA==?= Bjarmason , "brian m. carlson" , Thomas Ackermann , Thomas Ackermann Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Thomas Ackermann From: Thomas Ackermann Use only https links for lore.kernel.org. Signed-off-by: Thomas Ackermann --- Documentation/technical/hash-function-transition.txt | 10 +++++----- t/t0021-conversion.sh | 4 ++-- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt index 0c4cb98cd4e9..7c1630bf8324 100644 --- a/Documentation/technical/hash-function-transition.txt +++ b/Documentation/technical/hash-function-transition.txt @@ -781,7 +781,7 @@ Document History bmwill@google.com, jonathantanmy@google.com, jrnieder@gmail.com, sbeller@google.com -* Initial version sent to http://lore.kernel.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com +* Initial version sent to https://lore.kernel.org/git/20170304011251.GA26789@aiede.mtv.corp.google.com 2017-03-03 jrnieder@gmail.com Incorporated suggestions from jonathantanmy and sbeller: @@ -823,8 +823,8 @@ Later history: References: - [1] http://lore.kernel.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/ - [2] http://lore.kernel.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/ - [3] http://lore.kernel.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/ - [4] http://lore.kernel.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net + [1] https://lore.kernel.org/git/CA+55aFzJtejiCjV0e43+9oR3QuJK2PiFiLQemytoLpyJWe6P9w@mail.gmail.com/ + [2] https://lore.kernel.org/git/CA+55aFz+gkAsDZ24zmePQuEs1XPS9BP_s8O7Q4wQ7LV7X5-oDA@mail.gmail.com/ + [3] https://lore.kernel.org/git/20170306084353.nrns455dvkdsfgo5@sigill.intra.peff.net/ + [4] https://lore.kernel.org/git/20170304224936.rqqtkdvfjgyezsht@genre.crustytoothpaste.net [5] https://lore.kernel.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/ diff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh index e4c4de5c7456..e828ee964c1b 100755 --- a/t/t0021-conversion.sh +++ b/t/t0021-conversion.sh @@ -34,7 +34,7 @@ filter_git () { # Compare two files and ensure that `clean` and `smudge` respectively are # called at least once if specified in the `expect` file. The actual # invocation count is not relevant because their number can vary. -# c.f. http://lore.kernel.org/git/xmqqshv18i8i.fsf@gitster.mtv.corp.google.com/ +# c.f. https://lore.kernel.org/git/xmqqshv18i8i.fsf@gitster.mtv.corp.google.com/ test_cmp_count () { expect=$1 actual=$2 @@ -49,7 +49,7 @@ test_cmp_count () { # Compare two files but exclude all `clean` invocations because Git can # call `clean` zero or more times. -# c.f. http://lore.kernel.org/git/xmqqshv18i8i.fsf@gitster.mtv.corp.google.com/ +# c.f. https://lore.kernel.org/git/xmqqshv18i8i.fsf@gitster.mtv.corp.google.com/ test_cmp_exclude_clean () { expect=$1 actual=$2