[v2,0/9] refs: ref storage format migrations

Message

Patrick Steinhardt May 24, 2024, 10:14 a.m. UTC

Hi,

this patch series implements support for migrating between ref storage
formats in a repository via a new `git refs migrate` command. The scope
of this command is currently limited to repositories without worktrees
and reflogs. Furthermore, the user needs to make sure that there are no
concurrent writes.

Changes compared to v1:

  - Improve commit messages.

  - Mention that `--dry-run` mode will print the path to the migrated
    directory to stdout.

  - Mention that the repository should be deregistered from maintenance
    before running the migration.

Thanks!

Patrick

Patrick Steinhardt (9):
  setup: unset ref storage when reinitializing repository version
  refs: convert ref storage format to an enum
  refs: pass storage format to `ref_store_init()` explicitly
  refs: allow to skip creation of reflog entries
  refs/files: refactor `add_pseudoref_and_head_entries()`
  refs/files: extract function to iterate through root refs
  refs: implement removal of ref storages
  refs: implement logic to migrate between ref storage formats
  builtin/refs: new command to migrate ref storage formats

 .gitignore                 |   1 +
 Documentation/git-refs.txt |  62 +++++++
 Makefile                   |   1 +
 builtin.h                  |   1 +
 builtin/clone.c            |   2 +-
 builtin/init-db.c          |   2 +-
 builtin/refs.c             |  75 +++++++++
 git.c                      |   1 +
 refs.c                     | 319 +++++++++++++++++++++++++++++++++++--
 refs.h                     |  41 ++++-
 refs/files-backend.c       | 121 ++++++++++++--
 refs/packed-backend.c      |  15 ++
 refs/refs-internal.h       |   7 +
 refs/reftable-backend.c    |  37 ++++-
 repository.c               |   3 +-
 repository.h               |  10 +-
 setup.c                    |  10 +-
 setup.h                    |   9 +-
 t/helper/test-ref-store.c  |   1 +
 t/t1460-refs-migrate.sh    | 243 ++++++++++++++++++++++++++++
 20 files changed, 916 insertions(+), 45 deletions(-)
 create mode 100644 Documentation/git-refs.txt
 create mode 100644 builtin/refs.c
 create mode 100755 t/t1460-refs-migrate.sh

Range-diff against v1:
 1:  8b11127daf =  1:  8b11127daf setup: unset ref storage when reinitializing repository version
 2:  25f740f395 =  2:  25f740f395 refs: convert ref storage format to an enum
 3:  6e7b9764f6 =  3:  6e7b9764f6 refs: pass storage format to `ref_store_init()` explicitly
 4:  03f4ac6ee7 =  4:  03f4ac6ee7 refs: allow to skip creation of reflog entries
 5:  71f31fe66c =  5:  71f31fe66c refs/files: refactor `add_pseudoref_and_head_entries()`
 6:  6b696690ca =  6:  6b696690ca refs/files: extract function to iterate through root refs
 7:  b758c419c6 =  7:  b758c419c6 refs: implement removal of ref storages
 8:  4e0edda6d3 !  8:  4d3eb5ea89 refs: implement logic to migrate between ref storage formats
    @@ Commit message
         whole repository. Add the logic to do so.
     
         The implementation is generic and works with arbitrary ref storage
    -    formats because we only use. It does have a few limitations though:
    +    formats so that a backend does not need to implement any migration
    +    logic. It does have a few limitations though:
     
           - We do not migrate repositories with worktrees, because worktrees
             have separate ref storages. It makes the overall affair more complex
    @@ Commit message
     
         In other words, this version is a minimum viable product for migrating a
         repository's ref storage format. It works alright for bare repos, which
    -    typically have neither worktrees nor reflogs. But it will not work for
    -    many other repositories without some preparations. These limitations are
    -    not set into stone though, and ideally we will eventually address them
    -    over time.
    +    often have neither worktrees nor reflogs. But it will not work for many
    +    other repositories without some preparations. These limitations are not
    +    set into stone though, and ideally we will eventually address them over
    +    time.
     
         The logic is not yet used by anything, and thus there are no tests for
         it. Those will be added in the next commit.
 9:  2ebcc0db65 !  9:  0df17a51b4 builtin/refs: new command to migrate ref storage formats
    @@ Documentation/git-refs.txt (new)
     +--dry-run::
     +	Perform the migration, but do not modify the repository. The migrated
     +	refs will be written into a separate directory that can be inspected
    -+	separately. This can be used to double check that the migration works
    -+	as expected before doing performing the actual migration.
    ++	separately. The name of the directory will be reported on stdout. This
    ++	can be used to double check that the migration works as expected doing
    ++	performing the actual migration.
     +
     +KNOWN LIMITATIONS
     +-----------------
    @@ Documentation/git-refs.txt (new)
     +
     +* There is no way to block concurrent writes to the repository during an
     +  ongoing migration. Concurrent writes can lead to an inconsistent migrated
    -+  state. Users are expected to block writes on a higher level.
    ++  state. Users are expected to block writes on a higher level. If your
    ++  repository is registered for scheduled maintenance, it is recommended to
    ++  unregister it first with git-maintenance(1).
     +
     +These limitations may eventually be lifted.
     +