[0/8] repack: introduce `--write-midx`

Message

Taylor Blau Sept. 11, 2021, 3:32 a.m. UTC

This series implements the final component of multi-pack reachability bitmaps:
the ability to write them from `git repack`.

The gist is that it is currently impossible to write a MIDX based on the result
of a `git repack` without the possibility that an existing MIDX may go away.
That's because `git repack` may have deleted a pack that was included in the
MIDX (either because it was rolled up when doing a `--geometric` repack, or the
repack was done all-into-one), and MIDXs cannot refer to deleted packs.

To solve this, `git repack` is taught a new option to write a MIDX between the
critical section: right after the new packs are written and moved into place,
but right before any old packs are deleted.

Unfortunately, there is quite a bit of auxiliary tooling we need to implement
before this is possible. The series is laid out like so:

  - The first two patches lay the ground work for and expose `git
    multi-pack-index --stdin-packs`, which allows callers to write a MIDX
    containing only certain packs.

  - The next patch implements preliminary support for `--refs-snapshot`, which
    allows the MIDX bitmap selection code to reference the prior state of
    reference tips, preventing a race.

  - The next two patches prepare the `repack` builtin to write a MIDX, and the
    following patch actually implements `--write-midx`.

  - The final two patches make sure the MIDX generated has an appropriately
    chosen preferred pack, and that the refs-snapshot is provided.

It is based on `tb/multi-pack-bitmaps`. Thanks in advance for your review!

Taylor Blau (8):
  midx: expose 'write_midx_file_only()' publicly
  builtin/multi-pack-index.c: support --stdin-packs mode
  midx: preliminary support for `--refs-snapshot`
  builtin/repack.c: keep track of existing packs unconditionally
  builtin/repack.c: extract showing progress to a variable
  builtin/repack.c: support writing a MIDX while repacking
  builtin/repack.c: make largest pack preferred
  builtin/repack.c: pass `--refs-snapshot` when writing bitmaps

 Documentation/git-multi-pack-index.txt |  19 ++
 Documentation/git-repack.txt           |  18 +-
 builtin/multi-pack-index.c             |  35 +++-
 builtin/repack.c                       | 251 ++++++++++++++++++++++---
 midx.c                                 | 102 ++++++++--
 midx.h                                 |   4 +-
 pack-bitmap.c                          |   2 +-
 pack-bitmap.h                          |   1 +
 t/helper/test-read-midx.c              |  25 ++-
 t/lib-midx.sh                          |   8 +
 t/t5319-multi-pack-index.sh            |  15 ++
 t/t5326-multi-pack-bitmaps.sh          |  82 ++++++++
 t/t7700-repack.sh                      |  96 ++++++++++
 t/t7703-repack-geometric.sh            |  23 +++
 14 files changed, 630 insertions(+), 51 deletions(-)
 create mode 100644 t/lib-midx.sh