Message ID | 20200707062039.GC784740@google.com (mailing list archive) |
---|---|
State | New, archived |
Headers | show |
Series | experimental: default to fetch.writeCommitGraph=false | expand |
On 7/7/2020 2:20 AM, Jonathan Nieder wrote: > The fetch.writeCommitGraph feature makes fetches write out a commit > graph file for the newly downloaded pack on fetch. This improves the > performance of various commands that would perform a revision walk and > eventually ought to be the default for everyone. To prepare for that > future, it's enabled by default for users that set > feature.experimental=true to experience such future defaults. > > Alas, for --unshallow fetches from a shallow clone it runs into a > snag: by the time Git has fetched the new objects and is writing a > commit graph, it has performed a revision walk and r->parsed_objects > contains information about the shallow boundary from *before* the > fetch. The commit graph writing code is careful to avoid writing a > commit graph file in shallow repositories, but the new state is not > shallow, and the result is that from that point on, commands like "git > log" make use of a newly written commit graph file representing a > fictional history with the old shallow boundary. > > We could fix this by making the commit graph writing code more careful > to avoid writing a commit graph that could have used any grafts or > shallow state, but it is possible that there are other pieces of > mutated state that fetch's commit graph writing code may be relying > on. So disable it in the feature.experimental configuration. > > Google developers have been running in this configuration (by setting > fetch.writeCommitGraph=false in the system config) to work around this > bug since it was discovered in April. Once the fix lands, we'll > enable fetch.writeCommitGraph=true again to give it some early testing > before rolling out to a wider audience. > > In other words: > > - this patch only affects behavior with feature.experimental=true > > - it makes feature.experimental match the configuration Google has > been using for the last few months, meaning it would leave users in > a better tested state than without it > > - this should improve testing for other features guarded by > feature.experimental, by making feature.experimental safer to use > > Reported-by: Jay Conrod <jayconrod@google.com> > Helped-by: Taylor Blau <me@ttaylorr.com> > Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> > --- > I realize this is late to send. That said, as described above, I > think it's a good way to buy time by minimizing user exposure to > fetch.writeCommitGraph=true until a fix for it is well cooked. While it would certainly be better to fix the commit_graph_compatible() method, I understand that that is a more complicated problem. > In other words, I'd like to see this patch in Git 2.28-rc0. This patch is 100% correct. Normally, I would say "this is experimental, and a user can always disable fetch.writeCommitGraph manually if they are running into this." This is especially true because unshallowing a repo is (probably) a rare operation. At least, I expect that very few users actually do it, and those who do are expert users. But, it is best to reduce user pain, even in rare cases like this. Further, I hope to submit new maintenance tasks soon which can replace fetch.writeCommitGraph. Thanks, -Stolee
Jonathan Nieder <jrnieder@gmail.com> writes: > The fetch.writeCommitGraph feature makes fetches write out a commit > In other words: > > - this patch only affects behavior with feature.experimental=true > > - it makes feature.experimental match the configuration Google has > been using for the last few months, meaning it would leave users in > a better tested state than without it > > - this should improve testing for other features guarded by > feature.experimental, by making feature.experimental safer to use In other words, fetch.writeCommitGraph in its current form is too broken to be recommended even for brave souls with "experimental" bit on. I wonder if we perhaps wnat to add to the documentation for writeCommitGraph configuration that its use is currently not recommended in a shallow clone or something (I know it is not a problem just to use it with shallow but the breakage needs to involve unshallowing, but by definition those who do not use shallow would not hit the unshallowing bug, so...). > Reported-by: Jay Conrod <jayconrod@google.com> > Helped-by: Taylor Blau <me@ttaylorr.com> > Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> > --- > I realize this is late to send. That said, as described above, I > think it's a good way to buy time by minimizing user exposure to > fetch.writeCommitGraph=true until a fix for it is well cooked. > > In other words, I'd like to see this patch in Git 2.28-rc0. Yes, I do, too. Thanks. > diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt > index b1a9b1461d3..b20394038d1 100644 > --- a/Documentation/config/fetch.txt > +++ b/Documentation/config/fetch.txt > @@ -90,5 +90,4 @@ fetch.writeCommitGraph:: > the existing commit-graph file(s). Occasionally, these files will > merge and the write may take longer. Having an updated commit-graph > file helps performance of many Git commands, including `git merge-base`, > - `git push -f`, and `git log --graph`. Defaults to false, unless > - `feature.experimental` is true. > + `git push -f`, and `git log --graph`. Defaults to false. > diff --git a/repo-settings.c b/repo-settings.c > index dc6817daa95..0918408b344 100644 > --- a/repo-settings.c > +++ b/repo-settings.c > @@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r) > UPDATE_DEFAULT_BOOL(r->settings.index_version, 4); > UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE); > } > + > if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value)) > r->settings.fetch_write_commit_graph = value; > - if (!repo_config_get_bool(r, "feature.experimental", &value) && value) { > - UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING); > - UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1); > - } > UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0); > > + if (!repo_config_get_bool(r, "feature.experimental", &value) && value) > + UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING); > + > /* Hack for test programs like test-dump-untracked-cache */ > if (ignore_untracked_cache_config) > r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;
Hi Junio, On Tue, Jul 07, 2020 at 08:09:36AM -0700, Junio C Hamano wrote: > Jonathan Nieder <jrnieder@gmail.com> writes: > > > The fetch.writeCommitGraph feature makes fetches write out a commit > > > In other words: > > > > - this patch only affects behavior with feature.experimental=true > > > > - it makes feature.experimental match the configuration Google has > > been using for the last few months, meaning it would leave users in > > a better tested state than without it > > > > - this should improve testing for other features guarded by > > feature.experimental, by making feature.experimental safer to use > > > In other words, fetch.writeCommitGraph in its current form is too > broken to be recommended even for brave souls with "experimental" > bit on. > > I wonder if we perhaps wnat to add to the documentation for > writeCommitGraph configuration that its use is currently not > recommended in a shallow clone or something (I know it is not > a problem just to use it with shallow but the breakage needs > to involve unshallowing, but by definition those who do not > use shallow would not hit the unshallowing bug, so...). I think this is a good direction if you don't want to take the patch I sent in [1] for v2.28.0. If you do, though, I don't think that this would be necessary. > > Reported-by: Jay Conrod <jayconrod@google.com> > > Helped-by: Taylor Blau <me@ttaylorr.com> > > Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> > > --- > > I realize this is late to send. That said, as described above, I > > think it's a good way to buy time by minimizing user exposure to > > fetch.writeCommitGraph=true until a fix for it is well cooked. > > > > In other words, I'd like to see this patch in Git 2.28-rc0. > > Yes, I do, too. > > Thanks. > > > diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt > > index b1a9b1461d3..b20394038d1 100644 > > --- a/Documentation/config/fetch.txt > > +++ b/Documentation/config/fetch.txt > > @@ -90,5 +90,4 @@ fetch.writeCommitGraph:: > > the existing commit-graph file(s). Occasionally, these files will > > merge and the write may take longer. Having an updated commit-graph > > file helps performance of many Git commands, including `git merge-base`, > > - `git push -f`, and `git log --graph`. Defaults to false, unless > > - `feature.experimental` is true. > > + `git push -f`, and `git log --graph`. Defaults to false. > > diff --git a/repo-settings.c b/repo-settings.c > > index dc6817daa95..0918408b344 100644 > > --- a/repo-settings.c > > +++ b/repo-settings.c > > @@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r) > > UPDATE_DEFAULT_BOOL(r->settings.index_version, 4); > > UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE); > > } > > + > > if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value)) > > r->settings.fetch_write_commit_graph = value; > > - if (!repo_config_get_bool(r, "feature.experimental", &value) && value) { > > - UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING); > > - UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1); > > - } > > UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0); > > > > + if (!repo_config_get_bool(r, "feature.experimental", &value) && value) > > + UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING); > > + > > /* Hack for test programs like test-dump-untracked-cache */ > > if (ignore_untracked_cache_config) > > r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP; Thanks, Taylor [1]: https://lore.kernel.org/git/20200707144338.GA26342@syl.lan/T/#t
Taylor Blau <me@ttaylorr.com> writes: >> I wonder if we perhaps wnat to add to the documentation for >> writeCommitGraph configuration that its use is currently not >> recommended in a shallow clone or something (I know it is not >> a problem just to use it with shallow but the breakage needs >> to involve unshallowing, but by definition those who do not >> use shallow would not hit the unshallowing bug, so...). > > I think this is a good direction if you don't want to take the patch I > sent in [1] for v2.28.0. If you do, though, I don't think that this > would be necessary. Good timing. I didn't know a "fix" was already being worked on ([1] is the patch from this morning, right? I haven't seen it except for its subject). We could obviously do both excluding it from the usual experimental set and applying your fix, so that those who are really curious can help us make sure your fix would be all that is needed. Let's see what Jonathan says... Thanks.
On Tue, Jul 07, 2020 at 09:50:00AM -0700, Junio C Hamano wrote: > Taylor Blau <me@ttaylorr.com> writes: > > >> I wonder if we perhaps wnat to add to the documentation for > >> writeCommitGraph configuration that its use is currently not > >> recommended in a shallow clone or something (I know it is not > >> a problem just to use it with shallow but the breakage needs > >> to involve unshallowing, but by definition those who do not > >> use shallow would not hit the unshallowing bug, so...). > > > > I think this is a good direction if you don't want to take the patch I > > sent in [1] for v2.28.0. If you do, though, I don't think that this > > would be necessary. > > Good timing. I didn't know a "fix" was already being worked on ([1] > is the patch from this morning, right? I haven't seen it except for > its subject). [1] is the fix. Jonathan wrote it a month or so ago, I just added a test on top. (Independently, I tested it with the reproduction in the original bug report, and it worked properly). > We could obviously do both excluding it from the usual experimental > set and applying your fix, so that those who are really curious can > help us make sure your fix would be all that is needed. Let's see > what Jonathan says... Either of those sound good to me. > Thanks. Thanks, Taylor
Junio C Hamano wrote: > We could obviously do both excluding it from the usual experimental > set and applying your fix, so that those who are really curious can > help us make sure your fix would be all that is needed. Let's see > what Jonathan says... Yes, that would be my preference. That is, both: * applying the fix to provide a good experience to users interested in fetch.writeCommitGraph * disabling fetch.writeCommitGraph in the experimental set, since it has not had much production exposure yet. The experimental set is relatively young, so I want to ensure people's initial experiences with it are positive so that they stick with it if they're interested in experimental features (or in other words, I think there's still a place for features that are not yet proven enough to go in the experimental set). Regardless of what is put in the experimental set, at $DAYJOB we will run with the fix applied and with fetch.writeCommitGraph=true starting next week. I'd encourage anyone else with a similarly controlled setup to try the same. Thanks, Jonathan
diff --git a/Documentation/config/feature.txt b/Documentation/config/feature.txt index 28c33602d52..c0cbf2bb1cd 100644 --- a/Documentation/config/feature.txt +++ b/Documentation/config/feature.txt @@ -15,14 +15,6 @@ feature.experimental:: * `fetch.negotiationAlgorithm=skipping` may improve fetch negotiation times by skipping more commits at a time, reducing the number of round trips. + -* `fetch.writeCommitGraph=true` writes a commit-graph after every `git fetch` -command that downloads a pack-file from a remote. Using the `--split` option, -most executions will create a very small commit-graph file on top of the -existing commit-graph file(s). Occasionally, these files will merge and the -write may take longer. Having an updated commit-graph file helps performance -of many Git commands, including `git merge-base`, `git push -f`, and -`git log --graph`. -+ * `protocol.version=2` speeds up fetches from repositories with many refs by allowing the client to specify which refs to list before the server lists them. diff --git a/Documentation/config/fetch.txt b/Documentation/config/fetch.txt index b1a9b1461d3..b20394038d1 100644 --- a/Documentation/config/fetch.txt +++ b/Documentation/config/fetch.txt @@ -90,5 +90,4 @@ fetch.writeCommitGraph:: the existing commit-graph file(s). Occasionally, these files will merge and the write may take longer. Having an updated commit-graph file helps performance of many Git commands, including `git merge-base`, - `git push -f`, and `git log --graph`. Defaults to false, unless - `feature.experimental` is true. + `git push -f`, and `git log --graph`. Defaults to false. diff --git a/repo-settings.c b/repo-settings.c index dc6817daa95..0918408b344 100644 --- a/repo-settings.c +++ b/repo-settings.c @@ -51,14 +51,14 @@ void prepare_repo_settings(struct repository *r) UPDATE_DEFAULT_BOOL(r->settings.index_version, 4); UPDATE_DEFAULT_BOOL(r->settings.core_untracked_cache, UNTRACKED_CACHE_WRITE); } + if (!repo_config_get_bool(r, "fetch.writecommitgraph", &value)) r->settings.fetch_write_commit_graph = value; - if (!repo_config_get_bool(r, "feature.experimental", &value) && value) { - UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING); - UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 1); - } UPDATE_DEFAULT_BOOL(r->settings.fetch_write_commit_graph, 0); + if (!repo_config_get_bool(r, "feature.experimental", &value) && value) + UPDATE_DEFAULT_BOOL(r->settings.fetch_negotiation_algorithm, FETCH_NEGOTIATION_SKIPPING); + /* Hack for test programs like test-dump-untracked-cache */ if (ignore_untracked_cache_config) r->settings.core_untracked_cache = UNTRACKED_CACHE_KEEP;
The fetch.writeCommitGraph feature makes fetches write out a commit graph file for the newly downloaded pack on fetch. This improves the performance of various commands that would perform a revision walk and eventually ought to be the default for everyone. To prepare for that future, it's enabled by default for users that set feature.experimental=true to experience such future defaults. Alas, for --unshallow fetches from a shallow clone it runs into a snag: by the time Git has fetched the new objects and is writing a commit graph, it has performed a revision walk and r->parsed_objects contains information about the shallow boundary from *before* the fetch. The commit graph writing code is careful to avoid writing a commit graph file in shallow repositories, but the new state is not shallow, and the result is that from that point on, commands like "git log" make use of a newly written commit graph file representing a fictional history with the old shallow boundary. We could fix this by making the commit graph writing code more careful to avoid writing a commit graph that could have used any grafts or shallow state, but it is possible that there are other pieces of mutated state that fetch's commit graph writing code may be relying on. So disable it in the feature.experimental configuration. Google developers have been running in this configuration (by setting fetch.writeCommitGraph=false in the system config) to work around this bug since it was discovered in April. Once the fix lands, we'll enable fetch.writeCommitGraph=true again to give it some early testing before rolling out to a wider audience. In other words: - this patch only affects behavior with feature.experimental=true - it makes feature.experimental match the configuration Google has been using for the last few months, meaning it would leave users in a better tested state than without it - this should improve testing for other features guarded by feature.experimental, by making feature.experimental safer to use Reported-by: Jay Conrod <jayconrod@google.com> Helped-by: Taylor Blau <me@ttaylorr.com> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> --- I realize this is late to send. That said, as described above, I think it's a good way to buy time by minimizing user exposure to fetch.writeCommitGraph=true until a fix for it is well cooked. In other words, I'd like to see this patch in Git 2.28-rc0. Thanks of all kinds welcome, as always. Previous discussion: https://lore.kernel.org/git/20200603034213.GB253041@google.com/ Documentation/config/feature.txt | 8 -------- Documentation/config/fetch.txt | 3 +-- repo-settings.c | 8 ++++---- 3 files changed, 5 insertions(+), 14 deletions(-)