| Message ID | 9d7f484907e2bd2492e6676238579e9f0c6ed374.1628162156.git.ps@pks.im (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | Speed up connectivity checks | expand |
Patrick Steinhardt <ps@pks.im> writes: > + } else if (!strcmp(arg, "--unsorted-input")) { > + if (revs->no_walk && !revs->unsorted_input) > + die(_("--unsorted-input is incompatible with --no-walk and --no-walk=sorted")); > + revs->unsorted_input = 1; So this can be used with --no-walk=unsorted, even though doing so would be redundant and meaningless. OK. > @@ -2651,6 +2655,8 @@ static int handle_revision_pseudo_opt(const char *submodule, > } else if (!strcmp(arg, "--not")) { > *flags ^= UNINTERESTING | BOTTOM; > } else if (!strcmp(arg, "--no-walk")) { > + if (revs->unsorted_input) > + die(_("--no-walk is incompatible with --no-walk=unsorted and --unsorted-input")); And likewise, --no-walk is --no-walk=sorted so we do not allow it to be used with --unsorted-input or --no=walk=unsorted. OK. > revs->no_walk = 1; > } else if (skip_prefix(arg, "--no-walk=", &optarg)) { > /* > @@ -2658,9 +2664,12 @@ static int handle_revision_pseudo_opt(const char *submodule, > * not allowed, since the argument is optional. > */ > revs->no_walk = 1; > - if (!strcmp(optarg, "sorted")) > + if (!strcmp(optarg, "sorted")) { > + if (revs->unsorted_input) > + die(_("--no-walk=sorted is incompatible with --no-walk=unsorted " > + "and --unsorted-input")); OK. > revs->unsorted_input = 0; > - else if (!strcmp(optarg, "unsorted")) > + } else if (!strcmp(optarg, "unsorted")) > revs->unsorted_input = 1; This is --no-walk=unsorted; could it have been given after --no-walk or --no-walk=unsorted? The application of the incompatibility rules seems a bit uneven. An earlier piece of code will reject "--no-walk=unsorted --no-walk" given in this order (see "And likewise" above). But here, this part of the code will happily take "--no-walk --no-walk=unsorted". Of course these details can be fixed with more careful code design, but I wonder if it may be result in the code and behaviour that is far simpler to explain (and probably implement) if we declare that * --no-walk is not a synonym to --no-walk=sorted; it just flips .no_walk member on. * --no-walk=sorted and --no-walk=unsorted flip .no_walk member on, and then flips .unsorted_input member off or on, respectively. and define that the usual last-one-wins rule would apply? Thanks.
On Thu, Aug 05, 2021 at 11:44:05AM -0700, Junio C Hamano wrote: > Patrick Steinhardt <ps@pks.im> writes: > > revs->unsorted_input = 0; > > - else if (!strcmp(optarg, "unsorted")) > > + } else if (!strcmp(optarg, "unsorted")) > > revs->unsorted_input = 1; > > This is --no-walk=unsorted; could it have been given after --no-walk > or --no-walk=unsorted? > > The application of the incompatibility rules seems a bit uneven. An > earlier piece of code will reject "--no-walk=unsorted --no-walk" given > in this order (see "And likewise" above). But here, this part of > the code will happily take "--no-walk --no-walk=unsorted". > > Of course these details can be fixed with more careful code design, > but I wonder if it may be result in the code and behaviour that is > far simpler to explain (and probably implement) if we declare that > > * --no-walk is not a synonym to --no-walk=sorted; it just flips > .no_walk member on. > > * --no-walk=sorted and --no-walk=unsorted flip .no_walk member on, > and then flips .unsorted_input member off or on, respectively. > > and define that the usual last-one-wins rule would apply? > > Thanks. Wouldn't that effectively change semantics though? If the user passes `git rev-list --no-walk=unsorted --no-walk`, then the result is a sorted revwalk right now. One may argue that most likely, nobody is doing that, but you never really know. An easier approach which keeps existing semantics is to just make `--no-walk` and `--unsorted-input` mutually exclusive: - If the `unsorted_input` bit is set and `no_walk` isn't, and we observe any `--no-walk` option, then we bail. - Likewise, if the `no_walk` bit is set, then we bail when we see `--unsorted-input` regardless of the value of `unsorted_input`. This would keep current semantics of `--no-walk`, but prohobit using it together with the new option. Patrick
Patrick Steinhardt <ps@pks.im> writes: > Wouldn't that effectively change semantics though? If the user passes > `git rev-list --no-walk=unsorted --no-walk`, then the result is a sorted > revwalk right now. One may argue that most likely, nobody is doing that, > but you never really know. True. > An easier approach which keeps existing semantics is to just make > `--no-walk` and `--unsorted-input` mutually exclusive: > > - If the `unsorted_input` bit is set and `no_walk` isn't, and we > observe any `--no-walk` option, then we bail. > > - Likewise, if the `no_walk` bit is set, then we bail when we see > `--unsorted-input` regardless of the value of `unsorted_input`. > This would keep current semantics of `--no-walk`, but prohobit > using it together with the new option. True again. As I said, I do prefer going in the "start tight to forbid combination" route. But as I pointed out, the coverage by the posted patch seemed to have gap(s). Thanks.
diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index 24569b06d1..b7bd27e171 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -968,6 +968,11 @@ list of the missing objects. Object IDs are prefixed with a ``?'' character. objects. endif::git-rev-list[] +--unsorted-input:: + Show commits in the order they were given on the command line instead + of sorting them in reverse chronological order by commit time. Cannot + be combined with `--no-walk` or `--no-walk=sorted`. + --no-walk[=(sorted|unsorted)]:: Only show the given commits, but do not traverse their ancestors. This has no effect if a range is specified. If the argument @@ -975,7 +980,8 @@ endif::git-rev-list[] given on the command line. Otherwise (if `sorted` or no argument was given), the commits are shown in reverse chronological order by commit time. - Cannot be combined with `--graph`. + Cannot be combined with `--graph`. Cannot be combined with + `--unsorted-input` if `sorted` or no argument was given. --do-walk:: Overrides a previous `--no-walk`. diff --git a/connected.c b/connected.c index b18299fdf0..b5f9523a5f 100644 --- a/connected.c +++ b/connected.c @@ -106,6 +106,7 @@ int check_connected(oid_iterate_fn fn, void *cb_data, if (opt->progress) strvec_pushf(&rev_list.args, "--progress=%s", _("Checking connectivity")); + strvec_push(&rev_list.args, "--unsorted-input"); rev_list.git_cmd = 1; rev_list.env = opt->env; diff --git a/revision.c b/revision.c index 86bbcd10d2..793f76a509 100644 --- a/revision.c +++ b/revision.c @@ -2256,6 +2256,10 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg } else if (!strcmp(arg, "--author-date-order")) { revs->sort_order = REV_SORT_BY_AUTHOR_DATE; revs->topo_order = 1; + } else if (!strcmp(arg, "--unsorted-input")) { + if (revs->no_walk && !revs->unsorted_input) + die(_("--unsorted-input is incompatible with --no-walk and --no-walk=sorted")); + revs->unsorted_input = 1; } else if (!strcmp(arg, "--early-output")) { revs->early_output = 100; revs->topo_order = 1; @@ -2651,6 +2655,8 @@ static int handle_revision_pseudo_opt(const char *submodule, } else if (!strcmp(arg, "--not")) { *flags ^= UNINTERESTING | BOTTOM; } else if (!strcmp(arg, "--no-walk")) { + if (revs->unsorted_input) + die(_("--no-walk is incompatible with --no-walk=unsorted and --unsorted-input")); revs->no_walk = 1; } else if (skip_prefix(arg, "--no-walk=", &optarg)) { /* @@ -2658,9 +2664,12 @@ static int handle_revision_pseudo_opt(const char *submodule, * not allowed, since the argument is optional. */ revs->no_walk = 1; - if (!strcmp(optarg, "sorted")) + if (!strcmp(optarg, "sorted")) { + if (revs->unsorted_input) + die(_("--no-walk=sorted is incompatible with --no-walk=unsorted " + "and --unsorted-input")); revs->unsorted_input = 0; - else if (!strcmp(optarg, "unsorted")) + } else if (!strcmp(optarg, "unsorted")) revs->unsorted_input = 1; else return error("invalid argument to --no-walk"); diff --git a/t/t6000-rev-list-misc.sh b/t/t6000-rev-list-misc.sh index 12def7bcbf..8e213eb413 100755 --- a/t/t6000-rev-list-misc.sh +++ b/t/t6000-rev-list-misc.sh @@ -169,4 +169,42 @@ test_expect_success 'rev-list --count --objects' ' test_line_count = $count actual ' +test_expect_success 'rev-list --unsorted-input results in different sorting' ' + git rev-list --unsorted-input HEAD HEAD~ >first && + git rev-list --unsorted-input HEAD~ HEAD >second && + ! test_cmp first second && + sort first >first.sorted && + sort second >second.sorted && + test_cmp first.sorted second.sorted +' + +test_expect_success 'rev-list --unsorted-input compatible with --no-walk=unsorted' ' + git rev-list --unsorted-input --no-walk=unsorted HEAD HEAD~ >actual && + git rev-parse HEAD >expect && + git rev-parse HEAD~ >>expect && + test_cmp expect actual +' + +test_expect_success 'rev-list --unsorted-input incompatible with --no-walk=sorted' ' + cat >expect <<-EOF && + fatal: --no-walk is incompatible with --no-walk=unsorted and --unsorted-input + EOF + test_must_fail git rev-list --unsorted-input --no-walk HEAD 2>error && + test_cmp expect error && + + cat >expect <<-EOF && + fatal: --no-walk=sorted is incompatible with --no-walk=unsorted and --unsorted-input + EOF + test_must_fail git rev-list --unsorted-input --no-walk=sorted HEAD 2>error && + test_cmp expect error && + + cat >expect <<-EOF && + fatal: --unsorted-input is incompatible with --no-walk and --no-walk=sorted + EOF + test_must_fail git rev-list --no-walk --unsorted-input HEAD 2>error && + test_cmp expect error && + test_must_fail git rev-list --no-walk=sorted --unsorted-input HEAD 2>error && + test_cmp expect error +' + test_done
In order to compute whether objects reachable from a set of tips are all connected, we do a revision walk with these tips as positive references and `--not --all`. `--not --all` will cause the revision walk to load all preexisting references as uninteresting, which can be very expensive in repositories with many references. Benchmarking the git-rev-list(1) command highlights that by far the most expensive single phase is initial sorting of the input revisions: after all references have been loaded, we first sort commits by author date. In a real-world repository with about 2.2 million references, it makes up about 40% of the total runtime of git-rev-list(1). Ultimately, the connectivity check shouldn't really bother about the order of input revisions at all. We only care whether we can actually walk all objects until we hit the cut-off point. So sorting the input is a complete waste of time. Introduce a new "--unsorted-input" flag to git-rev-list(1) which will cause it to not sort the commits and adjust the connectivity check to always pass the flag. This results in the following speedups, executed in a clone of gitlab-org/gitlab [1]: Benchmark #1: git rev-list --objects --quiet --not --all --not $(cat newrev) Time (mean ± σ): 7.639 s ± 0.065 s [User: 7.304 s, System: 0.335 s] Range (min … max): 7.543 s … 7.742 s 10 runs Benchmark #2: git rev-list --unsorted-input --objects --quiet --not --all --not $newrev Time (mean ± σ): 4.995 s ± 0.044 s [User: 4.657 s, System: 0.337 s] Range (min … max): 4.909 s … 5.048 s 10 runs Summary 'git rev-list --unsorted-input --objects --quiet --not --all --not $(cat newrev)' ran 1.53 ± 0.02 times faster than 'git rev-list --objects --quiet --not --all --not $newrev' [1]: https://gitlab.com/gitlab-org/gitlab.git. Note that not all refs are visible to clients. Signed-off-by: Patrick Steinhardt <ps@pks.im> --- Documentation/rev-list-options.txt | 8 ++++++- connected.c | 1 + revision.c | 13 ++++++++-- t/t6000-rev-list-misc.sh | 38 ++++++++++++++++++++++++++++++ 4 files changed, 57 insertions(+), 3 deletions(-)