| Message ID | 20220903003623.64750-4-shaoxuan.yuan02@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | grep: integrate with sparse index | expand |
Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> writes: > @@ -537,8 +534,20 @@ static int grep_cache(struct grep_opt *opt, > > strbuf_setlen(&name, name_base_len); > strbuf_addstr(&name, ce->name); > + if (S_ISSPARSEDIR(ce->ce_mode)) { > + enum object_type type; > + struct tree_desc tree; > + void *data; > + unsigned long size; > + > + data = read_object_file(&ce->oid, &type, &size); > + init_tree_desc(&tree, data, size); > > - if (S_ISREG(ce->ce_mode) && > + hit |= grep_tree(opt, pathspec, &tree, &name, 0, 0); > + strbuf_reset(&name); Is this correct? I would have expected that this would chomp to name_base_len, just like what the code before this if/elseif cascade did. There needs a test that is run with repo->submodule_prefix != NULL to uncover issues like this, perhaps? > + strbuf_addstr(&name, ce->name); > + free(data); > + } else if (S_ISREG(ce->ce_mode) && > match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL, > S_ISDIR(ce->ce_mode) || > S_ISGITLINK(ce->ce_mode))) { > diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh > index fce8151d41..3242cfe91a 100755 > --- a/t/perf/p2000-sparse-operations.sh > +++ b/t/perf/p2000-sparse-operations.sh > @@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD > test_perf_on_all git checkout-index -f --all > test_perf_on_all git update-index --add --remove $SPARSE_CONE/a > test_perf_on_all "git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a" > +test_perf_on_all git grep --cached --sparse bogus -- "f2/f1/f1/*" > > test_done > diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh > index 63becc3138..56e4614276 100755 > --- a/t/t1092-sparse-checkout-compatibility.sh > +++ b/t/t1092-sparse-checkout-compatibility.sh > @@ -1987,7 +1987,15 @@ test_expect_success 'grep is not expanded' ' > > # All files within the folder1/* pathspec are sparse, > # so this command does not find any matches > - ensure_not_expanded ! grep a -- folder1/* > + ensure_not_expanded ! grep a -- folder1/* && > + > + # test out-of-cone pathspec with or without wildcard > + ensure_not_expanded grep --sparse --cached a -- "folder1/a" && > + ensure_not_expanded grep --sparse --cached a -- "folder1/*" && > + > + # test in-cone pathspec with or without wildcard > + ensure_not_expanded grep --sparse --cached a -- "deep/a" && > + ensure_not_expanded grep --sparse --cached a -- "deep/*" > ' > > test_done
On 9/2/2022 9:39 PM, Junio C Hamano wrote: > Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> writes: > >> @@ -537,8 +534,20 @@ static int grep_cache(struct grep_opt *opt, >> >> strbuf_setlen(&name, name_base_len); >> strbuf_addstr(&name, ce->name); >> + if (S_ISSPARSEDIR(ce->ce_mode)) { >> + enum object_type type; >> + struct tree_desc tree; >> + void *data; >> + unsigned long size; >> + >> + data = read_object_file(&ce->oid, &type, &size); >> + init_tree_desc(&tree, data, size); >> >> - if (S_ISREG(ce->ce_mode) && >> + hit |= grep_tree(opt, pathspec, &tree, &name, 0, 0); >> + strbuf_reset(&name); > > Is this correct? > > I would have expected that this would chomp to name_base_len, just > like what the code before this if/elseif cascade did. OK. > > There needs a test that is run with repo->submodule_prefix != NULL > to uncover issues like this, perhaps? I'm sorry that I forgot to directly reply to this. But I have sent a v5 [1] based on your suggestions here. Thanks for the review! [1] https://lore.kernel.org/git/20220908001854.206789-1-shaoxuan.yuan02@gmail.com/ Thanks, Shaoxuan
diff --git a/builtin/grep.c b/builtin/grep.c index a0b4dbc1dc..d8c086abff 100644 --- a/builtin/grep.c +++ b/builtin/grep.c @@ -522,9 +522,6 @@ static int grep_cache(struct grep_opt *opt, if (repo_read_index(repo) < 0) die(_("index file corrupt")); - if (grep_sparse) - ensure_full_index(repo->index); - for (nr = 0; nr < repo->index->cache_nr; nr++) { const struct cache_entry *ce = repo->index->cache[nr]; @@ -537,8 +534,20 @@ static int grep_cache(struct grep_opt *opt, strbuf_setlen(&name, name_base_len); strbuf_addstr(&name, ce->name); + if (S_ISSPARSEDIR(ce->ce_mode)) { + enum object_type type; + struct tree_desc tree; + void *data; + unsigned long size; + + data = read_object_file(&ce->oid, &type, &size); + init_tree_desc(&tree, data, size); - if (S_ISREG(ce->ce_mode) && + hit |= grep_tree(opt, pathspec, &tree, &name, 0, 0); + strbuf_reset(&name); + strbuf_addstr(&name, ce->name); + free(data); + } else if (S_ISREG(ce->ce_mode) && match_pathspec(repo->index, pathspec, name.buf, name.len, 0, NULL, S_ISDIR(ce->ce_mode) || S_ISGITLINK(ce->ce_mode))) { diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh index fce8151d41..3242cfe91a 100755 --- a/t/perf/p2000-sparse-operations.sh +++ b/t/perf/p2000-sparse-operations.sh @@ -124,5 +124,6 @@ test_perf_on_all git read-tree -mu HEAD test_perf_on_all git checkout-index -f --all test_perf_on_all git update-index --add --remove $SPARSE_CONE/a test_perf_on_all "git rm -f $SPARSE_CONE/a && git checkout HEAD -- $SPARSE_CONE/a" +test_perf_on_all git grep --cached --sparse bogus -- "f2/f1/f1/*" test_done diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 63becc3138..56e4614276 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -1987,7 +1987,15 @@ test_expect_success 'grep is not expanded' ' # All files within the folder1/* pathspec are sparse, # so this command does not find any matches - ensure_not_expanded ! grep a -- folder1/* + ensure_not_expanded ! grep a -- folder1/* && + + # test out-of-cone pathspec with or without wildcard + ensure_not_expanded grep --sparse --cached a -- "folder1/a" && + ensure_not_expanded grep --sparse --cached a -- "folder1/*" && + + # test in-cone pathspec with or without wildcard + ensure_not_expanded grep --sparse --cached a -- "deep/a" && + ensure_not_expanded grep --sparse --cached a -- "deep/*" ' test_done
Before this patch, whenever --sparse is used, `git-grep` utilizes the ensure_full_index() method to expand the index and search all the entries. Because this method requires walking all the trees and constructing the index, it is the slow part within the whole command. To achieve better performance, this patch uses grep_tree() to search the sparse directory entries and get rid of the ensure_full_index() method. Why grep_tree() is a better choice over ensure_full_index()? 1) grep_tree() is as correct as ensure_full_index(). grep_tree() looks into every sparse-directory entry (represented by a tree) recursively when looping over the index, and the result of doing so matches the result of expanding the index. 2) grep_tree() utilizes pathspecs to limit the scope of searching. ensure_full_index() always expands the index when --sparse is used, that means it will always walk all the trees and blobs in the repo without caring if the user only wants a subset of the content, i.e. using a pathspec. On the other hand, grep_tree() will only search the contents that match the pathspec, and thus possibly walking fewer trees. 3) grep_tree() does not construct and copy back a new index, while ensure_full_index() does. This also saves some time. ---------------- Performance test - Summary: p2000 tests demonstrate a ~71% execution time reduction for `git grep --cached --sparse bogus -- "f2/f1/f1/*"` using tree-walking logic. However, notice that this result varies depending on the pathspec given. See below "Command used for testing" for more details. Test HEAD~ HEAD ------------------------------------------------------- 2000.78: git grep ... (full-v3) 0.35 0.39 (≈) 2000.79: git grep ... (full-v4) 0.36 0.30 (≈) 2000.80: git grep ... (sparse-v3) 0.88 0.23 (-73.8%) 2000.81: git grep ... (sparse-v4) 0.83 0.26 (-68.6%) - Command used for testing: git grep --cached --sparse bogus -- "f2/f1/f1/*" The reason for specifying a pathspec is that, if we don't specify a pathspec, then grep_tree() will walk all the trees and blobs to find the pattern, and the time consumed doing so is not too different from using the original ensure_full_index() method, which also spends most of the time walking trees. However, when a pathspec is specified, this latest logic will only walk the area of trees enclosed by the pathspec, and the time consumed is reasonably a lot less. Generally speaking, because the performance gain is acheived by walking less trees, which are specified by the pathspec, the HEAD time v.s. HEAD~ time in sparse-v[3|4], should be proportional to "pathspec enclosed area" v.s. "all area", respectively. Namely, the wider the <pathspec> is encompassing, the less the performance difference between HEAD~ and HEAD, and vice versa. That is, if we don't specify a pathspec, the performance difference [1] is indistinguishable: both methods walk all the trees and take generally same amount of time (even with the index construction time included for ensure_full_index()). [1] Performance test result without pathspec (hence walking all trees): Command used: git grep --cached --sparse bogus Test HEAD~ HEAD --------------------------------------------------- 2000.78: git grep ... (full-v3) 6.17 5.19 (≈) 2000.79: git grep ... (full-v4) 6.19 5.46 (≈) 2000.80: git grep ... (sparse-v3) 6.57 6.44 (≈) 2000.81: git grep ... (sparse-v4) 6.65 6.28 (≈) Suggested-by: Derrick Stolee <derrickstolee@github.com> Helped-by: Derrick Stolee <derrickstolee@github.com> Helped-by: Victoria Dye <vdye@github.com> Signed-off-by: Shaoxuan Yuan <shaoxuan.yuan02@gmail.com> --- builtin/grep.c | 17 +++++++++++++---- t/perf/p2000-sparse-operations.sh | 1 + t/t1092-sparse-checkout-compatibility.sh | 10 +++++++++- 3 files changed, 23 insertions(+), 5 deletions(-)