[v3,3/3] repack -ad: prune the list of shallow commits

Commit Message

Linus Arver via GitGitGadget Oct. 22, 2018, 10:05 p.m. UTC

From: Johannes Schindelin <[e-mail]>

`git repack` can drop unreachable commits without further warning,
making the corresponding entries in `.git/shallow` invalid, which causes
serious problems when deepening the branches.

One scenario where unreachable commits are dropped by `git repack` is
when a `git fetch --prune` (or even a `git fetch` when a ref was
force-pushed in the meantime) can make a commit unreachable that was
reachable before.

Therefore it is not safe to assume that a `git repack -adlf` will keep
unreachable commits alone (under the assumption that they had not been
packed in the first place, which is an assumption at least some of Git's
code seems to make).

This is particularly important to keep in mind when looking at the
`.git/shallow` file: if any commits listed in that file become
unreachable, it is not a problem, but if they go missing, it *is* a
problem. One symptom of this problem is that a deepening fetch may now
fail with

	fatal: error in object: unshallow <commit-hash>

To avoid this problem, let's prune the shallow list in `git repack` when
the `-d` option is passed, unless `-A` is passed, too (which would force
the now-unreachable objects to be turned into loose objects instead of
being deleted). Additionally, we also need to take `--keep-reachable`
and `--unpack-unreachable=<date>` into account.

Note: an alternative solution discussed during the review of this patch
was to teach `git fetch` to simply ignore entries in .git/shallow if the
corresponding commits do not exist locally. A quick test, however,
revealed that the .git/shallow file is written during a shallow *clone*,
in which case the commits do not exist, either, but the "shallow" line
*does* need to be sent. Therefore, this approach would be a lot more
finicky than the approach presented by the this patch.

Signed-off-by: Johannes Schindelin <[e-mail]>
---
 builtin/repack.c         | 6 ++++++
 t/t5537-fetch-shallow.sh | 2 +-
 2 files changed, 7 insertions(+), 1 deletion(-)

Comments

Junio C Hamano Oct. 24, 2018, 3:56 a.m. UTC | #1

"Johannes Schindelin via GitGitGadget" <[e-mail]>
writes:

> diff --git a/builtin/repack.c b/builtin/repack.c
> index c6a7943d5..9217fc832 100644
> --- a/builtin/repack.c
> +++ b/builtin/repack.c
> @@ -549,6 +549,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
>  		if (!po_args.quiet && isatty(2))
>  			opts |= PRUNE_PACKED_VERBOSE;
>  		prune_packed_objects(opts);
> +
> +		if (!keep_unreachable &&
> +		    (!(pack_everything & LOOSEN_UNREACHABLE) ||
> +		     unpack_unreachable) &&
> +		    is_repository_shallow(the_repository))
> +			prune_shallow(0, 1);

The logic looks correct (and the reasoning behind it, which was
explained in the log message, was sound).  prune_shallow(0, 1)
however is not easily understandable.

There are only two callsites of this function after these three
patches, and the areas of the code that have these calls are in
relatively quiescent parts in the whole tree, so it shouldn't be too
distracting to do an update to make the function take a flag word,
so that we can make callsites more immediately obvious, i.e.

	prune_shallow(PRUNE_SHALLOW_QUICK)

It of course can be left as a low-hanging fruit loose-end.

> diff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh
> index 2d0031703..777c9d1dc 100755
> --- a/t/t5537-fetch-shallow.sh
> +++ b/t/t5537-fetch-shallow.sh
> @@ -186,7 +186,7 @@ EOF
>  	test_cmp expect actual
>  '
>  
> -test_expect_failure '.git/shallow is edited by repack' '
> +test_expect_success '.git/shallow is edited by repack' '
>  	git init shallow-server &&
>  	test_commit -C shallow-server A &&
>  	test_commit -C shallow-server B &&

Johannes Schindelin Oct. 24, 2018, 8:02 a.m. UTC | #2

Hi Junio,

On Wed, 24 Oct 2018, Junio C Hamano wrote:

> "Johannes Schindelin via GitGitGadget" <[e-mail]>
> writes:
> 
> > diff --git a/builtin/repack.c b/builtin/repack.c
> > index c6a7943d5..9217fc832 100644
> > --- a/builtin/repack.c
> > +++ b/builtin/repack.c
> > @@ -549,6 +549,12 @@ int cmd_repack(int argc, const char **argv, const char *prefix)
> >  		if (!po_args.quiet && isatty(2))
> >  			opts |= PRUNE_PACKED_VERBOSE;
> >  		prune_packed_objects(opts);
> > +
> > +		if (!keep_unreachable &&
> > +		    (!(pack_everything & LOOSEN_UNREACHABLE) ||
> > +		     unpack_unreachable) &&
> > +		    is_repository_shallow(the_repository))
> > +			prune_shallow(0, 1);
> 
> The logic looks correct (and the reasoning behind it, which was
> explained in the log message, was sound).  prune_shallow(0, 1)
> however is not easily understandable.
> 
> There are only two callsites of this function after these three
> patches, and the areas of the code that have these calls are in
> relatively quiescent parts in the whole tree, so it shouldn't be too
> distracting to do an update to make the function take a flag word,
> so that we can make callsites more immediately obvious, i.e.
> 
> 	prune_shallow(PRUNE_SHALLOW_QUICK)
> 
> It of course can be left as a low-hanging fruit loose-end.

I almost did that, but then decided not to clutter the previous patch with
this change (and global constant).

Having said that, since you also had this idea, I will make that change.
It will read nicer, you are right.

Ciao,
Dscho

> 
> > diff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh
> > index 2d0031703..777c9d1dc 100755
> > --- a/t/t5537-fetch-shallow.sh
> > +++ b/t/t5537-fetch-shallow.sh
> > @@ -186,7 +186,7 @@ EOF
> >  	test_cmp expect actual
> >  '
> >  
> > -test_expect_failure '.git/shallow is edited by repack' '
> > +test_expect_success '.git/shallow is edited by repack' '
> >  	git init shallow-server &&
> >  	test_commit -C shallow-server A &&
> >  	test_commit -C shallow-server B &&
>

Patch

diff --git a/builtin/repack.c b/builtin/repack.c
index c6a7943d5..9217fc832 100644
--- a/builtin/repack.c
+++ b/builtin/repack.c
@@ -549,6 +549,12 @@  int cmd_repack(int argc, const char **argv, const char *prefix)
 		if (!po_args.quiet && isatty(2))
 			opts |= PRUNE_PACKED_VERBOSE;
 		prune_packed_objects(opts);
+
+		if (!keep_unreachable &&
+		    (!(pack_everything & LOOSEN_UNREACHABLE) ||
+		     unpack_unreachable) &&
+		    is_repository_shallow(the_repository))
+			prune_shallow(0, 1);
 	}
 
 	if (!no_update_server_info)
diff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh
index 2d0031703..777c9d1dc 100755
--- a/t/t5537-fetch-shallow.sh
+++ b/t/t5537-fetch-shallow.sh
@@ -186,7 +186,7 @@  EOF
 	test_cmp expect actual
 '
 
-test_expect_failure '.git/shallow is edited by repack' '
+test_expect_success '.git/shallow is edited by repack' '
 	git init shallow-server &&
 	test_commit -C shallow-server A &&
 	test_commit -C shallow-server B &&