fetch: show progress for packfile uri downloads

Commit Message

Albert Cui March 17, 2021, 7:03 p.m. UTC

From: Albert Cui <[e-mail]>

Git appears to hang when downloading packfiles as this part of the
fetch is silent, causing user confusion. This change implements
progress for the number of packfiles downloaded; a progress display
for bytes would involve deeper changes at the http-fetch layer
instead of fetch-pack, the caller.

Signed-off-by: Albert Cui <[e-mail]>
---
    fetch: show progress for packfile uri downloads
    
    Git appears to hang when downloading packfiles as this part of the fetch
    is silent, causing user confusion. This change implements progress for
    the number of packfiles downloaded; a progress display for bytes would
    involve deeper changes at the http-fetch layer instead of fetch-pack,
    the caller.

Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-907%2Falbertcui%2Fprogress-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-907/albertcui/progress-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/907

 fetch-pack.c           | 8 ++++++++
 t/t5702-protocol-v2.sh | 5 ++++-
 2 files changed, 12 insertions(+), 1 deletion(-)


base-commit: a5828ae6b52137b913b978e16cd2334482eb4c1f

Comments

Jeff King March 17, 2021, 7:21 p.m. UTC | #1

On Wed, Mar 17, 2021 at 07:03:13PM +0000, Albert Cui via GitGitGadget wrote:

> From: Albert Cui <[e-mail]>
> 
> Git appears to hang when downloading packfiles as this part of the
> fetch is silent, causing user confusion. This change implements
> progress for the number of packfiles downloaded; a progress display
> for bytes would involve deeper changes at the http-fetch layer
> instead of fetch-pack, the caller.

I think this is an improvement, but I agree that a real byte display
would be much better. I actually worked on this a long time ago for a
very similar feature that we never quite pushed over the finish line.
See patches 11 and 12 here:

  https://lore.kernel.org/git/[e-mail]/

(it might need some of the earlier refactoring, too, I'm not sure; and
quite likely will need forward-porting as it has been 10 years).

-Peff

Junio C Hamano March 17, 2021, 7:31 p.m. UTC | #2

"Albert Cui via GitGitGadget" <[e-mail]> writes:

> From: Albert Cui <[e-mail]>
>
> Git appears to hang when downloading packfiles as this part of the
> fetch is silent, causing user confusion. This change implements
> progress for the number of packfiles downloaded; a progress display
> for bytes would involve deeper changes at the http-fetch layer
> instead of fetch-pack, the caller.

... "hence we do not do so in this patch"?  

That's probably a very sensible way to go.

I expect that http-fetch will in the longer term become a mere
fallback default used by those who do not have anything better.
Because we are not in the business of writing a performant HTTP
downloader, we would be better off if we make it easy to plug an
external HTTP downloader other people write in to this codepath.

> +	packfile_uri_progress = start_progress(_("Downloading packs"), packfile_uris.nr);

OK, so we plan to count from 0 up to .nr; and the message is made
localizable.  Good.

> @@ -1696,6 +1700,7 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
>  		const char *uri = packfile_uris.items[i].string +
>  			the_hash_algo->hexsz + 1;
>  
> +		display_progress(packfile_uri_progress, i+1);

		display_progress(packfile_uri_progress, i + 1);

> diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
> index 2e1243ca40b0..8964a4003678 100755
> --- a/t/t5702-protocol-v2.sh
> +++ b/t/t5702-protocol-v2.sh
> @@ -851,7 +851,8 @@ test_expect_success 'part of packfile response provided as URI' '
>  	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
>  	git -c protocol.version=2 \
>  		-c fetch.uriprotocols=http,https \
> -		clone "$HTTPD_URL/smart/http_parent" http_child &&
> +		clone "$HTTPD_URL/smart/http_parent" http_child \
> +		--progress 2>progress &&

Some existing tests use GIT_PROGRESS_DELAY to protect against an
operation that is too quick to complete.  Don't we need to do the
same?  If not, then perhaps we need to allow delaying the progress
meter we add with this patch for "too quick" case, perhaps?

>  	# Ensure that my-blob and other-blob are in separate packfiles.
>  	for idx in http_child/.git/objects/pack/*.idx
> @@ -875,6 +876,8 @@ test_expect_success 'part of packfile response provided as URI' '
>  	test -f hfound &&
>  	test -f h2found &&
>  
> +	test_i18ngrep "Downloading packs" progress &&

Also, I am not sure with all the terminal control junk, 'grep'
should be expected to reliably pick this substring in the output.
Are we expecting any other output to the standard error stream?
Some tests in t5318 seem to just see if the output is non-empty, and
I am wondering if that is an approach more appropriate here (not
rhetorical---I simply do not know the answer).

>  	# Ensure that there are exactly 3 packfiles with associated .idx
>  	ls http_child/.git/objects/pack/*.pack \
>  	    http_child/.git/objects/pack/*.idx >filelist &&
>
> base-commit: a5828ae6b52137b913b978e16cd2334482eb4c1f

Ævar Arnfjörð Bjarmason April 10, 2021, 8:31 a.m. UTC | #3

On Wed, Mar 17 2021, Albert Cui via GitGitGadget wrote:

> From: Albert Cui <[e-mail]>
> [...]
> @@ -875,6 +876,8 @@ test_expect_success 'part of packfile response provided as URI' '
>  	test -f hfound &&
>  	test -f h2found &&
>  
> +	test_i18ngrep "Downloading packs" progress &&

This can just be "grep"

Patch

diff --git a/fetch-pack.c b/fetch-pack.c
index 6a61a464283e..5646d151293d 100644
--- a/fetch-pack.c
+++ b/fetch-pack.c
@@ -23,6 +23,7 @@ 
 #include "fetch-negotiator.h"
 #include "fsck.h"
 #include "shallow.h"
+#include "progress.h"
 
 static int transfer_unpack_limit = -1;
 static int fetch_unpack_limit = -1;
@@ -1585,6 +1586,7 @@  static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 	struct fetch_negotiator *negotiator;
 	int seen_ack = 0;
 	struct string_list packfile_uris = STRING_LIST_INIT_DUP;
+	struct progress *packfile_uri_progress;
 	int i;
 	struct strvec index_pack_args = STRVEC_INIT;
 	struct oidset gitmodules_oids = OIDSET_INIT;
@@ -1689,6 +1691,8 @@  static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		}
 	}
 
+	packfile_uri_progress = start_progress(_("Downloading packs"), packfile_uris.nr);
+
 	for (i = 0; i < packfile_uris.nr; i++) {
 		int j;
 		struct child_process cmd = CHILD_PROCESS_INIT;
@@ -1696,6 +1700,7 @@  static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 		const char *uri = packfile_uris.items[i].string +
 			the_hash_algo->hexsz + 1;
 
+		display_progress(packfile_uri_progress, i+1);
 		strvec_push(&cmd.args, "http-fetch");
 		strvec_pushf(&cmd.args, "--packfile=%.*s",
 			     (int) the_hash_algo->hexsz,
@@ -1739,6 +1744,9 @@  static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args,
 						 get_object_directory(),
 						 packname));
 	}
+
+	stop_progress(&packfile_uri_progress);
+
 	string_list_clear(&packfile_uris, 0);
 	strvec_clear(&index_pack_args);
 
diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh
index 2e1243ca40b0..8964a4003678 100755
--- a/t/t5702-protocol-v2.sh
+++ b/t/t5702-protocol-v2.sh
@@ -851,7 +851,8 @@  test_expect_success 'part of packfile response provided as URI' '
 	GIT_TRACE=1 GIT_TRACE_PACKET="$(pwd)/log" GIT_TEST_SIDEBAND_ALL=1 \
 	git -c protocol.version=2 \
 		-c fetch.uriprotocols=http,https \
-		clone "$HTTPD_URL/smart/http_parent" http_child &&
+		clone "$HTTPD_URL/smart/http_parent" http_child \
+		--progress 2>progress &&
 
 	# Ensure that my-blob and other-blob are in separate packfiles.
 	for idx in http_child/.git/objects/pack/*.idx
@@ -875,6 +876,8 @@  test_expect_success 'part of packfile response provided as URI' '
 	test -f hfound &&
 	test -f h2found &&
 
+	test_i18ngrep "Downloading packs" progress &&
+
 	# Ensure that there are exactly 3 packfiles with associated .idx
 	ls http_child/.git/objects/pack/*.pack \
 	    http_child/.git/objects/pack/*.idx >filelist &&