[0/3] compat/mingw: implement POSIX-style atomic renames

Message

Patrick Steinhardt Oct. 23, 2024, 3:04 p.m. UTC

Hi,

so after chatting with Johannes recently I decided to have another stab
at the failures we've seen on Windows with the reftable backend. A quick
refresher: reftables use a central "tables.lock" file that gets updated
via renames such that the update itself is atomic. This causes issues on
Windows when there are many concurrent readers/writers because Windows
does not allow you to rename a file over another file that is held open
by another process by default. This issue is surfaced by t0610, where we
spawn a 100 concurrent git-update-ref(1) processes.

This patch series aims to fix the issue on Windows by implementing
POSIX-style atomic renames. The first two patches refactor the code such
that we use `FILE_SHARE_DELETE` to open files, which allows other
processes to delete the file even while we have an open handle. This
sharing mode is also important in the context of renames, which are
treated like a deletion of the replaced file. The third patch then
reimplements renames via `SetFileAttributesByHandle(FileRenameInfoEx)`,
which has a flag `FILE_RENAME_FLAG_POSIX_SEMANTICS`. When set, Windows
allows us to replace files during a rename which still have an open
handle.

There's one catch: this is only supported in Windows 10 and newer. On
one hand this means that we have to provide the required definitions
ourselves, as we cannot bump the Windows SDK version. On the other hand
we also have to support the case where older Windows versions now fail
at runtime due to `FileRenameInfoEx` being unsupported. But overall this
isn't too bad, I'd say. Johannes has also kindly tested the fallback
logic for me on Windows 8.1.

With these changes t0610 now passes on my Windows 10 machine, but this
may also have a positive impact on other parts of Git where we might
have previously failed.

Thanks!

Patrick

Patrick Steinhardt (3):
  compat/mingw: share file handles created via `CreateFileW()`
  compat/mingw: allow deletion of most opened files
  compat/mingw: support POSIX semantics for atomic renames

 compat/mingw.c             | 146 +++++++++++++++++++++++++++++++++++--
 t/t0610-reftable-basics.sh |   8 +-
 2 files changed, 145 insertions(+), 9 deletions(-)

Comments

Taylor Blau Oct. 23, 2024, 3:36 p.m. UTC | #1

On Wed, Oct 23, 2024 at 05:04:55PM +0200, Patrick Steinhardt wrote:
> With these changes t0610 now passes on my Windows 10 machine, but this
> may also have a positive impact on other parts of Git where we might
> have previously failed.

Ooo. Thank you both for collaborating and working on this.

Thanks,
Taylor