[00/44] SHA-256 part 2/3: protocol functionality

Message

brian m. carlson May 13, 2020, 12:53 a.m. UTC

This is part 2 of 3 of the SHA-256 work.  This series, which is
unfortunately longer than I'd like, adds all of the protocol logic to
work with SHA-256 repositories.

It was originally planned that we would not upgrade the protocol and
would use SHA-1 for all protocol functionality until some point in the
future.  However, doing that requires a huge amount of additional work
(probably incorporating several hundred more patches which are not yet
written) and it's not possible to get the test suite to even come close
to passing without a way to fetch and push repositories.  I therefore
decided that implementing an object-format extension was the best way
forward.

This series provides object-format extensions for both the original and
v2 protocols, including full documentation.  Helpers, such as
git-remote-https, also learn capabilities to pass the object-format
extension back and forth, and to query its state.  The code is designed
to allow multiple object-format extensions to be provided if the server
supports multiple algorithms for one repo and to default to SHA-1 if no
object-format extension is provided.

The other two cases are the dumb HTTP protocol and bundles, both of
which have no object-format extension (because they provide no
capabilities) and are therefore distinguished solely by their hash
length.  We will have problems if in the future we need to use another
256-bit algorithm, but I plan to be improvident and hope that we'll move
to longer algorithms in the future to cover ourselves for post-quantum
security.

Clone support is necessarily a little tricky because we are initializing
a repository and then fetching refs, at which point we learn what hash
algorithm the remote side supports.  We work around this by calling the
code that updates the hash algorithm and repository version a second
time to rewrite that data once we know what version we're using.  This
is the most robust way I could approach this problem, but it is still a
little ugly.

As mentioned, this series is longer than I'd like, but it is complete:
this is all the SHA-256 protocol work.  Additional future series include
one last series of test fixes (28 patches) plus six final patches in the
series that enables SHA-256 support.

brian m. carlson (44):
  t1050: match object ID paths in a hash-insensitive way
  Documentation: document v1 protocol object-format capability
  connect: have ref processing code take struct packet_reader
  wrapper: add function to compare strings with different NUL
    termination
  remote: advertise the object-format capability on the server side
  connect: add function to parse multiple v1 capability values
  connect: add function to fetch value of a v2 server capability
  pkt-line: add a member for hash algorithm
  transport: add a hash algorithm member
  connect: add function to detect supported v1 hash functions
  send-pack: detect when the server doesn't support our hash
  connect: make parse_feature_value extern
  fetch-pack: detect when the server doesn't support our hash
  connect: detect algorithm when fetching refs
  builtin/receive-pack: detect when the server doesn't support our hash
  docs: update remote helper docs for object-format extensions
  transport-helper: implement object-format extensions
  remote-curl: implement object-format extensions
  builtin/clone: initialize hash algorithm properly
  t5562: pass object-format in synthesized test data
  t5704: send object-format capability with SHA-256
  fetch-pack: parse and advertise the object-format capability
  setup: set the_repository's hash algo when checking format
  t3200: mark assertion with SHA1 prerequisite
  packfile: compute and use the index CRC offset
  t5302: modernize test formatting
  builtin/show-index: provide options to determine hash algo
  t1302: expect repo format version 1 for SHA-256
  Documentation/technical: document object-format for protocol v2
  connect: pass full packet reader when parsing v2 refs
  connect: parse v2 refs with correct hash algorithm
  serve: advertise object-format capability for protocol v2
  t5500: make hash independent
  builtin/ls-remote: initialize repository based on fetch
  remote-curl: detect algorithm for dumb HTTP by size
  builtin/index-pack: add option to specify hash algorithm
  t1050: pass algorithm to index-pack when outside repo
  remote-curl: avoid truncating refs with ls-remote
  t/helper: initialize the repository for test-sha1-array
  t5702: offer an object-format capability in the test
  t5703: use object-format serve option
  t5300: pass --object-format to git index-pack
  bundle: detect hash algorithm when reading refs
  remote-testgit: adapt for object-format

 Documentation/gitremote-helpers.txt           |  33 +-
 .../technical/protocol-capabilities.txt       |  16 +-
 Documentation/technical/protocol-v2.txt       |   9 +
 builtin/clone.c                               |   9 +
 builtin/index-pack.c                          |  11 +-
 builtin/ls-remote.c                           |   4 +
 builtin/receive-pack.c                        |  10 +
 builtin/show-index.c                          |  29 +-
 bundle.c                                      |  22 +-
 bundle.h                                      |   1 +
 connect.c                                     | 136 +++++--
 connect.h                                     |   3 +
 fetch-pack.c                                  |  14 +
 git-compat-util.h                             |   2 +
 git.c                                         |   2 +-
 object-store.h                                |   1 +
 packfile.c                                    |   1 +
 pkt-line.c                                    |   1 +
 pkt-line.h                                    |   3 +
 remote-curl.c                                 |  46 ++-
 send-pack.c                                   |   6 +
 serve.c                                       |  27 ++
 setup.c                                       |   1 +
 t/helper/test-oid-array.c                     |   3 +
 t/t1050-large.sh                              |   6 +-
 t/t1302-repo-version.sh                       |   6 +-
 t/t3200-branch.sh                             |   2 +-
 t/t5300-pack-object.sh                        |   9 +-
 t/t5302-pack-index.sh                         | 360 +++++++++---------
 t/t5500-fetch-pack.sh                         |   5 +-
 t/t5562-http-backend-content-length.sh        |  14 +-
 t/t5701-git-serve.sh                          |  28 +-
 t/t5702-protocol-v2.sh                        |   2 +
 t/t5703-upload-pack-ref-in-want.sh            |  19 +-
 t/t5704-protocol-violations.sh                |  12 +
 t/t5801/git-remote-testgit                    |   6 +
 t/test-lib.sh                                 |   1 +
 transport-helper.c                            |  24 +-
 transport.c                                   |  18 +-
 transport.h                                   |   8 +
 upload-pack.c                                 |   3 +-
 wrapper.c                                     |  12 +
 42 files changed, 670 insertions(+), 255 deletions(-)