| Message ID | 20190628070211.hfweqcons6c6gy52@dcvr (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | repack: disable bitmaps-by-default if .keep files exist | expand |
On Fri, Jun 28 2019, Eric Wong wrote: > Jeff King <peff@peff.net> wrote: >> On Sun, Jun 23, 2019 at 06:08:25PM +0000, Eric Wong wrote: >> >> > > I'm not sure of the right solution. For maximal backwards-compatibility, >> > > the default for bitmaps could become "if not bare and if there are no >> > > .keep files". But that would mean bitmaps sometimes not getting >> > > generated because of the problems that ee34a2bead was trying to solve. >> > > >> > > That's probably OK, though; you can always flip the bitmap config to >> > > "true" yourself if you _must_ have bitmaps. >> > >> > What about something like this? Needs tests but I need to leave, now. >> >> Yeah, I think that's the right direction. > > OK. I have a real patch with one additional test, below. > (don't have a lot of time for hacking) > >> Though... >> >> > +static int has_pack_keep_file(void) >> > +{ >> > + DIR *dir; >> > + struct dirent *e; >> > + int found = 0; >> > + >> > + if (!(dir = opendir(packdir))) >> > + return found; >> > + >> > + while ((e = readdir(dir)) != NULL) { >> > + if (ends_with(e->d_name, ".keep")) { >> > + found = 1; >> > + break; >> > + } >> > + } >> > + closedir(dir); >> > + return found; >> > +} >> >> I think this can be replaced with just checking p->pack_keep for each >> item in the packed_git list. > > Good point, I tend to forget git C API internals as soon as I > learn them :x > >> That's racy, but then so is your code here, since it's really the child >> pack-objects which is going to deal with the .keep. I don't think we >> need to care much about the race, though. Either: > > Agreed. <snip> > > --------8<------- > Subject: [PATCH] repack: disable bitmaps-by-default if .keep files exist > > Bitmaps aren't useful with multiple packs, and users with > .keep files ended up with redundant packs when bitmaps > got enabled by default in bare repos. > > So detect when .keep files exist and stop enabling bitmaps > by default in that case. > > Wasteful (but otherwise harmless) race conditions with .keep files > documented by Jeff King still apply and there's a chance we'd > still end up with redundant data on the FS: > > https://public-inbox.org/git/20190623224244.GB1100@sigill.intra.peff.net/ > > Fixes: 36eba0323d3288a8 ("repack: enable bitmaps by default on bare repos") > Signed-off-by: Eric Wong <e@80x24.org> > Helped-by: Jeff King <peff@peff.net> > Reported-by: Janos Farkas <chexum@gmail.com> > --- > builtin/repack.c | 18 ++++++++++++++++-- > t/t7700-repack.sh | 10 ++++++++++ > 2 files changed, 26 insertions(+), 2 deletions(-) > > diff --git a/builtin/repack.c b/builtin/repack.c > index caca113927..a9529d1afc 100644 > --- a/builtin/repack.c > +++ b/builtin/repack.c > @@ -89,6 +89,17 @@ static void remove_pack_on_signal(int signo) > raise(signo); > } > > +static int has_pack_keep_file(void) > +{ > + struct packed_git *p; > + > + for (p = get_packed_git(the_repository); p; p = p->next) { > + if (p->pack_keep) > + return 1; > + } > + return 0; > +} > + > /* > * Adds all packs hex strings to the fname list, which do not > * have a corresponding .keep file. These packs are not to > @@ -343,9 +354,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix) > (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE))) > die(_("--keep-unreachable and -A are incompatible")); > > - if (write_bitmaps < 0) > + if (write_bitmaps < 0) { > write_bitmaps = (pack_everything & ALL_INTO_ONE) && > - is_bare_repository(); > + is_bare_repository() && > + keep_pack_list.nr == 0 && > + !has_pack_keep_file(); > + } > if (pack_kept_objects < 0) > pack_kept_objects = write_bitmaps; > > diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh > index 86d05160a3..0acde3b1f8 100755 > --- a/t/t7700-repack.sh > +++ b/t/t7700-repack.sh > @@ -239,4 +239,14 @@ test_expect_success 'bitmaps can be disabled on bare repos' ' > test -z "$bitmap" > ' I have the feedback I posted before this patch in https://public-inbox.org/git/874l4f8h4c.fsf@evledraar.gmail.com/ In particular "b" there since "a" is clearly more work. I.e. shouldn't we at least in interactive mode on a "gc" print something about skipping what we'd otherwise do. Maybe that's tricky with the gc.log functionality, but I think we should at least document this before the next guy shows up with "sometimes my .bitmap files aren't generated...". > +test_expect_success 'no bitmaps created if .keep files present' ' > + pack=$(ls bare.git/objects/pack/*.pack) && > + test_path_is_file "$pack" && > + keep=${pack%.pack}.keep && > + >"$keep" && > + git -C bare.git repack -ad && > + bitmap=$(ls bare.git/objects/pack/*.bitmap 2>/dev/null || :) && > + test -z "$bitmap" Maybe more readable as: find .git/objects/pack/ -type f -name '*.bitmap' >actual && test_must_be_empty actual I.e. it avoids the "|| :" & subshell. > +' > + > test_done
On Fri, Jun 28, 2019 at 07:02:11AM +0000, Eric Wong wrote: > Jeff King <peff@peff.net> wrote: > > On Sun, Jun 23, 2019 at 06:08:25PM +0000, Eric Wong wrote: > > > > > > I'm not sure of the right solution. For maximal backwards-compatibility, > > > > the default for bitmaps could become "if not bare and if there are no > > > > .keep files". But that would mean bitmaps sometimes not getting > > > > generated because of the problems that ee34a2bead was trying to solve. > > > > > > > > That's probably OK, though; you can always flip the bitmap config to > > > > "true" yourself if you _must_ have bitmaps. > > > > > > What about something like this? Needs tests but I need to leave, now. > > > > Yeah, I think that's the right direction. > > OK. I have a real patch with one additional test, below. > (don't have a lot of time for hacking) > > > Though... > > > > > +static int has_pack_keep_file(void) > > > +{ > > > + DIR *dir; > > > + struct dirent *e; > > > + int found = 0; > > > + > > > + if (!(dir = opendir(packdir))) > > > + return found; > > > + > > > + while ((e = readdir(dir)) != NULL) { > > > + if (ends_with(e->d_name, ".keep")) { > > > + found = 1; > > > + break; > > > + } > > > + } > > > + closedir(dir); > > > + return found; > > > +} > > > > I think this can be replaced with just checking p->pack_keep for each > > item in the packed_git list. > > Good point, I tend to forget git C API internals as soon as I > learn them :x > > > That's racy, but then so is your code here, since it's really the child > > pack-objects which is going to deal with the .keep. I don't think we > > need to care much about the race, though. Either: > > Agreed. <snip> > > --------8<------- > Subject: [PATCH] repack: disable bitmaps-by-default if .keep files exist > > Bitmaps aren't useful with multiple packs, and users with > .keep files ended up with redundant packs when bitmaps > got enabled by default in bare repos. > > So detect when .keep files exist and stop enabling bitmaps > by default in that case. > > Wasteful (but otherwise harmless) race conditions with .keep files > documented by Jeff King still apply and there's a chance we'd > still end up with redundant data on the FS: > > https://public-inbox.org/git/20190623224244.GB1100@sigill.intra.peff.net/ > > Fixes: 36eba0323d3288a8 ("repack: enable bitmaps by default on bare repos") > Signed-off-by: Eric Wong <e@80x24.org> > Helped-by: Jeff King <peff@peff.net> > Reported-by: Janos Farkas <chexum@gmail.com> > --- > builtin/repack.c | 18 ++++++++++++++++-- > t/t7700-repack.sh | 10 ++++++++++ > 2 files changed, 26 insertions(+), 2 deletions(-) > > diff --git a/builtin/repack.c b/builtin/repack.c > index caca113927..a9529d1afc 100644 > --- a/builtin/repack.c > +++ b/builtin/repack.c > @@ -89,6 +89,17 @@ static void remove_pack_on_signal(int signo) > raise(signo); > } > > +static int has_pack_keep_file(void) > +{ > + struct packed_git *p; > + > + for (p = get_packed_git(the_repository); p; p = p->next) { > + if (p->pack_keep) > + return 1; > + } > + return 0; > +} > + > /* > * Adds all packs hex strings to the fname list, which do not > * have a corresponding .keep file. These packs are not to > @@ -343,9 +354,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix) > (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE))) > die(_("--keep-unreachable and -A are incompatible")); > > - if (write_bitmaps < 0) > + if (write_bitmaps < 0) { > write_bitmaps = (pack_everything & ALL_INTO_ONE) && > - is_bare_repository(); > + is_bare_repository() && > + keep_pack_list.nr == 0 && > + !has_pack_keep_file(); > + } > if (pack_kept_objects < 0) > pack_kept_objects = write_bitmaps; > > diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh > index 86d05160a3..0acde3b1f8 100755 > --- a/t/t7700-repack.sh > +++ b/t/t7700-repack.sh > @@ -239,4 +239,14 @@ test_expect_success 'bitmaps can be disabled on bare repos' ' > test -z "$bitmap" > ' > > +test_expect_success 'no bitmaps created if .keep files present' ' > + pack=$(ls bare.git/objects/pack/*.pack) && > + test_path_is_file "$pack" && > + keep=${pack%.pack}.keep && > + >"$keep" && > + git -C bare.git repack -ad && > + bitmap=$(ls bare.git/objects/pack/*.bitmap 2>/dev/null || :) && > + test -z "$bitmap" This test fails when run with 'GIT_TEST_MULTI_PACK_INDEX=1': + ls bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.pack + pack=bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.pack + test_path_is_file bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.pack + test -f bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.pack + keep=bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.keep + + git -C bare.git repack -ad + ls bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.bitmap + bitmap=bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.bitmap + test -z bare.git/objects/pack/pack-51a6a758692f03f9e1f31fd087bba30584afad2f.bitmap error: last command exited with $?=1 not ok 15 - no bitmaps created if .keep files present When the new has_pack_keep_file() helper function calls get_packed_git(the_repository) in the loop it returns with NULL already on the first iteration, so the keep file goes unnoticed. > +' > + > test_done > -- > EW
diff --git a/builtin/repack.c b/builtin/repack.c index caca113927..a9529d1afc 100644 --- a/builtin/repack.c +++ b/builtin/repack.c @@ -89,6 +89,17 @@ static void remove_pack_on_signal(int signo) raise(signo); } +static int has_pack_keep_file(void) +{ + struct packed_git *p; + + for (p = get_packed_git(the_repository); p; p = p->next) { + if (p->pack_keep) + return 1; + } + return 0; +} + /* * Adds all packs hex strings to the fname list, which do not * have a corresponding .keep file. These packs are not to @@ -343,9 +354,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix) (unpack_unreachable || (pack_everything & LOOSEN_UNREACHABLE))) die(_("--keep-unreachable and -A are incompatible")); - if (write_bitmaps < 0) + if (write_bitmaps < 0) { write_bitmaps = (pack_everything & ALL_INTO_ONE) && - is_bare_repository(); + is_bare_repository() && + keep_pack_list.nr == 0 && + !has_pack_keep_file(); + } if (pack_kept_objects < 0) pack_kept_objects = write_bitmaps; diff --git a/t/t7700-repack.sh b/t/t7700-repack.sh index 86d05160a3..0acde3b1f8 100755 --- a/t/t7700-repack.sh +++ b/t/t7700-repack.sh @@ -239,4 +239,14 @@ test_expect_success 'bitmaps can be disabled on bare repos' ' test -z "$bitmap" ' +test_expect_success 'no bitmaps created if .keep files present' ' + pack=$(ls bare.git/objects/pack/*.pack) && + test_path_is_file "$pack" && + keep=${pack%.pack}.keep && + >"$keep" && + git -C bare.git repack -ad && + bitmap=$(ls bare.git/objects/pack/*.bitmap 2>/dev/null || :) && + test -z "$bitmap" +' + test_done