| Message ID | 20181218212435.201641-1-jonathantanmy@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | upload-pack: teach deepen-relative in protocol v2 | expand |
On 2018.12.18 13:24, Jonathan Tan wrote: > Commit 685fbd3291 ("fetch-pack: perform a fetch using v2", 2018-03-15) > attempted to teach Git deepen-relative in protocol v2 (among other > things), but it didn't work: > > (1) fetch-pack.c needs to emit "deepen-relative". > (2) upload-pack.c needs to ensure that the correct deepen_relative > variable is passed to deepen() (there are two - the static variable > and the one in struct upload_pack_data). > (3) Before deepen() computes the list of reachable shallows, it first > needs to mark all "our" refs as OUR_REF. v2 currently does not do > this, because unlike v0, it is not needed otherwise. > > Fix all this and include a test demonstrating that it works now. For > (2), the static variable deepen_relative is also eliminated, removing a > source of confusion. > > Signed-off-by: Jonathan Tan <jonathantanmy@google.com> > --- > Similar to my first fix [1], another fix for an issue discovered by > Aevar's GIT_TEST_PROTOCOL_VERSION patches. This one is more > straightforward. > > With this fix and my first fix [1], t5500 no longer reveals any more > bugs. (There might be more in other test files.) > > [1] https://public-inbox.org/git/20181218010811.143608-1-jonathantanmy@google.com/ > --- > fetch-pack.c | 2 ++ > t/t5702-protocol-v2.sh | 29 +++++++++++++++++++++++++++++ > upload-pack.c | 17 +++++++++++++++-- > 3 files changed, 46 insertions(+), 2 deletions(-) > > diff --git a/fetch-pack.c b/fetch-pack.c > index 9691046e64..c383ea46b3 100644 > --- a/fetch-pack.c > +++ b/fetch-pack.c > @@ -1042,6 +1042,8 @@ static void add_shallow_requests(struct strbuf *req_buf, > packet_buf_write(req_buf, "deepen-not %s", s->string); > } > } > + if (args->deepen_relative) > + packet_buf_write(req_buf, "deepen-relative\n"); > } > > static void add_wants(int no_dependents, const struct ref *wants, struct strbuf *req_buf) > diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh > index 0f2b09ebb8..340953f01c 100755 > --- a/t/t5702-protocol-v2.sh > +++ b/t/t5702-protocol-v2.sh > @@ -471,6 +471,35 @@ test_expect_success 'upload-pack respects client shallows' ' > grep "fetch< version 2" trace > ' > > +test_expect_success 'deepen-relative' ' > + rm -rf server client trace && > + > + test_create_repo server && > + test_commit -C server one && > + test_commit -C server two && > + test_commit -C server three && > + git clone --depth 1 "file://$(pwd)/server" client && > + test_commit -C server four && > + > + # Sanity check that only "three" is downloaded > + git -C client log --pretty=tformat:%s master >actual && > + echo three >expected && > + test_cmp expected actual && > + > + GIT_TRACE_PACKET="$(pwd)/trace" git -C client -c protocol.version=2 \ > + fetch --deepen=1 origin && > + # Ensure that protocol v2 is used > + grep "fetch< version 2" trace && > + > + git -C client log --pretty=tformat:%s origin/master >actual && > + cat >expected <<-\EOF && > + four > + three > + two > + EOF > + test_cmp expected actual > +' > + > # Test protocol v2 with 'http://' transport > # > . "$TEST_DIRECTORY"/lib-httpd.sh > diff --git a/upload-pack.c b/upload-pack.c > index 5e81f1ff24..9df27b55a0 100644 > --- a/upload-pack.c > +++ b/upload-pack.c > @@ -43,7 +43,6 @@ > > static timestamp_t oldest_have; > > -static int deepen_relative; > static int multi_ack; > static int no_done; > static int use_thin_pack, use_ofs_delta, use_include_tag; > @@ -662,6 +661,9 @@ static void send_unshallow(const struct object_array *shallows, > } > } > > +static int check_ref(const char *refname_full, const struct object_id *oid, > + int flag, void *cb_data); > + > static void deepen(int depth, int deepen_relative, > struct object_array *shallows, struct object_array *want_obj) > { > @@ -676,6 +678,13 @@ static void deepen(int depth, int deepen_relative, > struct object_array reachable_shallows = OBJECT_ARRAY_INIT; > struct commit_list *result; > > + /* > + * Checking for reachable shallows requires that our refs be > + * marked with OUR_REF. > + */ > + head_ref_namespaced(check_ref, NULL); > + for_each_namespaced_ref(check_ref, NULL); > + > get_reachable_list(shallows, &reachable_shallows); > result = get_shallow_commits(&reachable_shallows, > depth + 1, > @@ -712,6 +721,7 @@ static void deepen_by_rev_list(int ac, const char **av, > static int send_shallow_list(int depth, int deepen_rev_list, > timestamp_t deepen_since, > struct string_list *deepen_not, > + int deepen_relative, > struct object_array *shallows, > struct object_array *want_obj) > { > @@ -834,6 +844,7 @@ static void receive_needs(struct object_array *want_obj) > int has_non_tip = 0; > timestamp_t deepen_since = 0; > int deepen_rev_list = 0; > + int deepen_relative = 0; > > shallow_nr = 0; > for (;;) { > @@ -925,7 +936,8 @@ static void receive_needs(struct object_array *want_obj) > return; > > if (send_shallow_list(depth, deepen_rev_list, deepen_since, > - &deepen_not, &shallows, want_obj)) > + &deepen_not, deepen_relative, &shallows, > + want_obj)) > packet_flush(1); > object_array_clear(&shallows); > } > @@ -1398,6 +1410,7 @@ static void send_shallow_info(struct upload_pack_data *data, > > if (!send_shallow_list(data->depth, data->deepen_rev_list, > data->deepen_since, &data->deepen_not, > + data->deepen_relative, > &data->shallows, want_obj) && > is_repository_shallow(the_repository)) > deepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows, > -- > 2.19.0.271.gfe8321ec05.dirty > Looks good to me. Reviewed-by: Josh Steadmon <steadmon@google.com>
Josh Steadmon <steadmon@google.com> writes: > Looks good to me. > Reviewed-by: Josh Steadmon <steadmon@google.com> Thanks, both.
Junio C Hamano <gitster@pobox.com> writes: > Josh Steadmon <steadmon@google.com> writes: > >> Looks good to me. >> Reviewed-by: Josh Steadmon <steadmon@google.com> > > Thanks, both. This, when applied on top of ecbdaf08991, seems to break t5702.30, at least for me. Tip of 'pu' in today's push-out will be broken as it has this patch merged. Can somebody take a look? expecting success: rm -rf server client trace && test_create_repo server && test_commit -C server one && test_commit -C server two && test_commit -C server three && git clone --depth 1 "file://$(pwd)/server" client && test_commit -C server four && # Sanity check that only "three" is downloaded git -C client log --pretty=tformat:%s master >actual && echo three >expected && test_cmp expected actual && GIT_TRACE_PACKET="$(pwd)/trace" git -C client -c protocol.version=2 \ fetch --deepen=1 origin && # Ensure that protocol v2 is used grep "fetch< version 2" trace && git -C client log --pretty=tformat:%s origin/master >actual && cat >expected <<-\EOF && four three two EOF test_cmp expected actual Initialized empty Git repository in /home/gitster/w/git.git/t/trash directory.t5702-protocol-v2/server/.git/ [master (root-commit) 1581e3e] one Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 one.t [master 5680d21] two Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 two.t [master 94705d7] three Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 three.t Cloning into 'client'... remote: Enumerating objects: 5, done. remote: Counting objects: 100% (5/5), done. remote: Compressing objects: 100% (2/2), done. remote: Total 5 (delta 0), reused 0 (delta 0) Receiving objects: 100% (5/5), done. [master 0b417c7] four Author: A U Thor <author@example.com> 1 file changed, 1 insertion(+) create mode 100644 four.t remote: Enumerating objects: 6, done. remote: Counting objects: 100% (6/6), done. remote: Compressing objects: 100% (4/4), done. remote: Total 5 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (5/5), done. From file:///home/gitster/w/git.git/t/trash directory.t5702-protocol-v2/server 94705d7..0b417c7 master -> origin/master * [new tag] four -> four fatal: shallow file has changed since we read it not ok 30 - deepen-relative
> Junio C Hamano <gitster@pobox.com> writes: > > > Josh Steadmon <steadmon@google.com> writes: > > > >> Looks good to me. > >> Reviewed-by: Josh Steadmon <steadmon@google.com> > > > > Thanks, both. > > This, when applied on top of ecbdaf08991, seems to break t5702.30, > at least for me. > > Tip of 'pu' in today's push-out will be broken as it has this patch > merged. > > Can somebody take a look? I can reproduce this. I'll investigate this further. In the meantime, feel free to remove this from pu.
> > This, when applied on top of ecbdaf08991, seems to break t5702.30, > > at least for me. > > > > Tip of 'pu' in today's push-out will be broken as it has this patch > > merged. > > > > Can somebody take a look? > > I can reproduce this. I'll investigate this further. In the meantime, > feel free to remove this from pu. The test failure is caused by a bug I fixed in [1]. I've verified that if I apply [1] first, then this, all tests pass. If you do include this patch in some branch, it should be dependent on [1], but I understand if you'd like reviewers to look at both this and [1] first. I thought I tested this without something that fixes the issue fixed by [1] and it passed, but apparently that's not true. [1] https://public-inbox.org/git/20190110193645.34080-1-jonathantanmy@google.com/
Jonathan Tan <jonathantanmy@google.com> writes: >> > This, when applied on top of ecbdaf08991, seems to break t5702.30, >> > at least for me. >> > >> > Tip of 'pu' in today's push-out will be broken as it has this patch >> > merged. >> > >> > Can somebody take a look? >> >> I can reproduce this. I'll investigate this further. In the meantime, >> feel free to remove this from pu. > > The test failure is caused by a bug I fixed in [1]. I've verified that > if I apply [1] first, then this, all tests pass. If you do include this > patch in some branch, it should be dependent on [1], but I understand if > you'd like reviewers to look at both this and [1] first. > > I thought I tested this without something that fixes the issue fixed by > [1] and it passed, but apparently that's not true. > > [1] https://public-inbox.org/git/20190110193645.34080-1-jonathantanmy@google.com/ OK, that means that it would be the cleanest to make them into two-patch series, I guess. Let's requeue these two and see what happens. Thanks.
diff --git a/fetch-pack.c b/fetch-pack.c index 9691046e64..c383ea46b3 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1042,6 +1042,8 @@ static void add_shallow_requests(struct strbuf *req_buf, packet_buf_write(req_buf, "deepen-not %s", s->string); } } + if (args->deepen_relative) + packet_buf_write(req_buf, "deepen-relative\n"); } static void add_wants(int no_dependents, const struct ref *wants, struct strbuf *req_buf) diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 0f2b09ebb8..340953f01c 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -471,6 +471,35 @@ test_expect_success 'upload-pack respects client shallows' ' grep "fetch< version 2" trace ' +test_expect_success 'deepen-relative' ' + rm -rf server client trace && + + test_create_repo server && + test_commit -C server one && + test_commit -C server two && + test_commit -C server three && + git clone --depth 1 "file://$(pwd)/server" client && + test_commit -C server four && + + # Sanity check that only "three" is downloaded + git -C client log --pretty=tformat:%s master >actual && + echo three >expected && + test_cmp expected actual && + + GIT_TRACE_PACKET="$(pwd)/trace" git -C client -c protocol.version=2 \ + fetch --deepen=1 origin && + # Ensure that protocol v2 is used + grep "fetch< version 2" trace && + + git -C client log --pretty=tformat:%s origin/master >actual && + cat >expected <<-\EOF && + four + three + two + EOF + test_cmp expected actual +' + # Test protocol v2 with 'http://' transport # . "$TEST_DIRECTORY"/lib-httpd.sh diff --git a/upload-pack.c b/upload-pack.c index 5e81f1ff24..9df27b55a0 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -43,7 +43,6 @@ static timestamp_t oldest_have; -static int deepen_relative; static int multi_ack; static int no_done; static int use_thin_pack, use_ofs_delta, use_include_tag; @@ -662,6 +661,9 @@ static void send_unshallow(const struct object_array *shallows, } } +static int check_ref(const char *refname_full, const struct object_id *oid, + int flag, void *cb_data); + static void deepen(int depth, int deepen_relative, struct object_array *shallows, struct object_array *want_obj) { @@ -676,6 +678,13 @@ static void deepen(int depth, int deepen_relative, struct object_array reachable_shallows = OBJECT_ARRAY_INIT; struct commit_list *result; + /* + * Checking for reachable shallows requires that our refs be + * marked with OUR_REF. + */ + head_ref_namespaced(check_ref, NULL); + for_each_namespaced_ref(check_ref, NULL); + get_reachable_list(shallows, &reachable_shallows); result = get_shallow_commits(&reachable_shallows, depth + 1, @@ -712,6 +721,7 @@ static void deepen_by_rev_list(int ac, const char **av, static int send_shallow_list(int depth, int deepen_rev_list, timestamp_t deepen_since, struct string_list *deepen_not, + int deepen_relative, struct object_array *shallows, struct object_array *want_obj) { @@ -834,6 +844,7 @@ static void receive_needs(struct object_array *want_obj) int has_non_tip = 0; timestamp_t deepen_since = 0; int deepen_rev_list = 0; + int deepen_relative = 0; shallow_nr = 0; for (;;) { @@ -925,7 +936,8 @@ static void receive_needs(struct object_array *want_obj) return; if (send_shallow_list(depth, deepen_rev_list, deepen_since, - &deepen_not, &shallows, want_obj)) + &deepen_not, deepen_relative, &shallows, + want_obj)) packet_flush(1); object_array_clear(&shallows); } @@ -1398,6 +1410,7 @@ static void send_shallow_info(struct upload_pack_data *data, if (!send_shallow_list(data->depth, data->deepen_rev_list, data->deepen_since, &data->deepen_not, + data->deepen_relative, &data->shallows, want_obj) && is_repository_shallow(the_repository)) deepen(INFINITE_DEPTH, data->deepen_relative, &data->shallows,
Commit 685fbd3291 ("fetch-pack: perform a fetch using v2", 2018-03-15) attempted to teach Git deepen-relative in protocol v2 (among other things), but it didn't work: (1) fetch-pack.c needs to emit "deepen-relative". (2) upload-pack.c needs to ensure that the correct deepen_relative variable is passed to deepen() (there are two - the static variable and the one in struct upload_pack_data). (3) Before deepen() computes the list of reachable shallows, it first needs to mark all "our" refs as OUR_REF. v2 currently does not do this, because unlike v0, it is not needed otherwise. Fix all this and include a test demonstrating that it works now. For (2), the static variable deepen_relative is also eliminated, removing a source of confusion. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> --- Similar to my first fix [1], another fix for an issue discovered by Aevar's GIT_TEST_PROTOCOL_VERSION patches. This one is more straightforward. With this fix and my first fix [1], t5500 no longer reveals any more bugs. (There might be more in other test files.) [1] https://public-inbox.org/git/20181218010811.143608-1-jonathantanmy@google.com/ --- fetch-pack.c | 2 ++ t/t5702-protocol-v2.sh | 29 +++++++++++++++++++++++++++++ upload-pack.c | 17 +++++++++++++++-- 3 files changed, 46 insertions(+), 2 deletions(-)