| Message ID | cover.1716451672.git.ps@pks.im (mailing list archive) |
|---|---|
| Headers | show |
| Series | refs: ref storage format migrations | expand |
Patrick Steinhardt <ps@pks.im> writes: > - It is not safe with concurrent writers. This is the limitation that > ... > none at all, as it may cause users to be less mindful. That's why I > decided to just have no solution at all and document the limitation > accordingly. Documenting the limitation is a good place to start. For normal users, would it be sufficient to (1) tell your colleagues that this repository is currently closed and do not push into it; (2) configure "git gc --auto" to never kick in; (3) delist the repository from "git maintenance" schedule. before they try this feature out?
On Thu, May 23, 2024 at 09:09:41AM -0700, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > > - It is not safe with concurrent writers. This is the limitation that > > ... > > none at all, as it may cause users to be less mindful. That's why I > > decided to just have no solution at all and document the limitation > > accordingly. > > Documenting the limitation is a good place to start. For normal > users, would it be sufficient to > > (1) tell your colleagues that this repository is currently closed > and do not push into it; > > (2) configure "git gc --auto" to never kick in; > > (3) delist the repository from "git maintenance" schedule. > > before they try this feature out? I think (2) wouldn't even be needed. Auto-GC only kicks in when there is a write in the repository, and if both (1) and (3) are true then there are none. But other than that yes, (1) and (3) should be sufficient. Patrick
Patrick Steinhardt <ps@pks.im> writes: >> Documenting the limitation is a good place to start. For normal >> users, would it be sufficient to >> >> (1) tell your colleagues that this repository is currently closed >> and do not push into it; >> >> (2) configure "git gc --auto" to never kick in; >> >> (3) delist the repository from "git maintenance" schedule. >> >> before they try this feature out? > > I think (2) wouldn't even be needed. Auto-GC only kicks in when there is > a write in the repository, and if both (1) and (3) are true then there > are none. But other than that yes, (1) and (3) should be sufficient. So it may make sense to document something like that at least? Thanks.
On Fri, May 24, 2024 at 09:28:13AM -0700, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > >> Documenting the limitation is a good place to start. For normal > >> users, would it be sufficient to > >> > >> (1) tell your colleagues that this repository is currently closed > >> and do not push into it; > >> > >> (2) configure "git gc --auto" to never kick in; > >> > >> (3) delist the repository from "git maintenance" schedule. > >> > >> before they try this feature out? > > > > I think (2) wouldn't even be needed. Auto-GC only kicks in when there is > > a write in the repository, and if both (1) and (3) are true then there > > are none. But other than that yes, (1) and (3) should be sufficient. > > So it may make sense to document something like that at least? It is documented as part of the new git-refs(1) man page, in the "Known limitations" section: * There is no way to block concurrent writes to the repository during an ongoing migration. Concurrent writes can lead to an inconsistent migrated state. Users are expected to block writes on a higher level. If your repository is registered for scheduled maintenance, it is recommended to unregister it first with git-maintenance(1). Is that sufficient, or do you think we need to expand on this? Patrick
Patrick Steinhardt <ps@pks.im> writes: > It is documented as part of the new git-refs(1) man page, in the "Known > limitations" section: > > * There is no way to block concurrent writes to the repository during an > ongoing migration. Concurrent writes can lead to an inconsistent migrated > state. Users are expected to block writes on a higher level. If your > repository is registered for scheduled maintenance, it is recommended to > unregister it first with git-maintenance(1). Good. Thanks.
Hi, this patch series implements migration of ref storage formats such that it is for example possible to migrate a repository with "files" format to the "reftable" format. The migration logic comes in the form of a new `git refs migrate` subcommand. As mentioned in [1], I do have plans to extend the new git-refs(1) over time. In the current form, the migration logic has three major limitations: - It does not work with repositories that have worktrees as we have to migrate multiple ref stores here, one for every worktree. I wanted to avoid making this series too complex right from the start. - It does not migrate reflogs, because we have no interfaces to do so. I want to eventually address this by adding log-only updates to transactions. - It is not safe with concurrent writers. This is the limitation that is most critical in my eyes. The root cause here is that it is inherently impossible to lock the "files" backend for writes. I have been thinking about this issue a lot and have not found any solution that works. There are partial solutions here: - Create a central lockfile for the "files" backend -- if there, the backend will refuse to write. If that lock needs to be acquired during the "commit" phase of transaction though then we would essentially start to sequentialize all writes in the "files" backend, which is a non-starter. If not, then processes which are running already may not have seen it, and thus the issue still exists. - Pack all loose refs, remove "refs/" and create a "refs.lock" file. This isn't safe though because root refs can still be written. - Create a log of concurrent writes and apply that to the migrated refs once done. This is a lot of complexity, and it's unclear whether it even solves the issue with already-running writers. - Create a temporary "extensions.refMigration" extension that is unhandled by Git. New processes will refuse to run in such a repo and thus cannot write to it. Again, unsafe with running writers. Another alternative is that we could just make this a best effort. The "reftable" backend supports locking, and for the "files" backend we could just lock "HEAD" and call it a day. I'm not sure whether it is preferable though to have a "partial" solution compared to having none at all, as it may cause users to be less mindful. That's why I decided to just have no solution at all and document the limitation accordingly. If anybody has ideas here I'd be very happy to hear them. Anyway. The current state of this patch series is sufficient to migrate repos without reflogs and worktrees, and thus mostly applies to bare repositories, only. This is somewhat intentional -- as a server operator this is of course our primary usecase at GitLab. We do plan to also upstream support for writing reflogs though, but in a later step. We do not plan to implement support for migrating repositories with worktrees, but I'd be happy to help out with the effort in case somebody else wants to. The series is built on top of 4365c6fcf9 (The sixth batch, 2024-05-20). It pulls in the following two dependencies: - ps/refs-without-the-repository-updates at 00892786b8 (refs/packed: remove references to `the_hash_algo`, 2024-05-17). This is mostly to avoid conflicts. - ps/pseudo-ref-terminology at 8e4f5c2dc2 (refs: refuse to write pseudorefs, 2024-05-15). This is a functional prerequisite because the migration logic relies on the new definition of pseudorefs. There are two minor textual conflicts when merged to "next" or "seen". One is a conflicting added header in "refs/reftable-backend.c", and one is a conflict with added functions in "refs.c". Both of these conflicts are trivial to solve by accepting both sides. Thanks! Patrick [1]: https://lore.kernel.org/git/ZkNJaaKTTKbns8wo@tanuki/#t Patrick Steinhardt (9): setup: unset ref storage when reinitializing repository version refs: convert ref storage format to an enum refs: pass storage format to `ref_store_init()` explicitly refs: allow to skip creation of reflog entries refs/files: refactor `add_pseudoref_and_head_entries()` refs/files: extract function to iterate through root refs refs: implement removal of ref storages refs: implement logic to migrate between ref storage formats builtin/refs: new command to migrate ref storage formats .gitignore | 1 + Documentation/git-refs.txt | 59 +++++++ Makefile | 1 + builtin.h | 1 + builtin/clone.c | 2 +- builtin/init-db.c | 2 +- builtin/refs.c | 75 +++++++++ git.c | 1 + refs.c | 319 +++++++++++++++++++++++++++++++++++-- refs.h | 41 ++++- refs/files-backend.c | 121 ++++++++++++-- refs/packed-backend.c | 15 ++ refs/refs-internal.h | 7 + refs/reftable-backend.c | 37 ++++- repository.c | 3 +- repository.h | 10 +- setup.c | 10 +- setup.h | 9 +- t/helper/test-ref-store.c | 1 + t/t1460-refs-migrate.sh | 243 ++++++++++++++++++++++++++++ 20 files changed, 913 insertions(+), 45 deletions(-) create mode 100644 Documentation/git-refs.txt create mode 100644 builtin/refs.c create mode 100755 t/t1460-refs-migrate.sh