| Message ID | fe4cf0841d500a46ad8f10c00818b467023d0ffc.1539298957.git.matvore@google.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | support for filtering trees and blobs based on depth | expand |
Matthew DeVore <matvore@google.com> writes: > The tree:0 filter does not need to traverse the trees that it has > filtered out, so optimize list-objects and list-objects-filter to skip > traversing the trees entirely. Before this patch, we iterated over all > children of the tree, and did nothing for all of them, which was > wasteful. > > Signed-off-by: Matthew DeVore <matvore@google.com> > --- > list-objects-filter.c | 11 +++++++++-- > list-objects-filter.h | 6 ++++++ > list-objects.c | 5 ++++- > t/t6112-rev-list-filters-objects.sh | 10 ++++++++++ > 4 files changed, 29 insertions(+), 3 deletions(-) This step looks more like "ow, we could have done the tree:0 support that is in 'next' better" than a part of "here is a series to do tree:N for non zero value of N". If that is the case, I'd prefer to see this step polished enough before [2-3/3] of this RFC is worked on, so that we can merge the tree:0 (but not yet tree:N) support that is solid down to 'master' soonish. > diff --git a/list-objects-filter.c b/list-objects-filter.c > index 09b2b05d5..37fba456d 100644 > --- a/list-objects-filter.c > +++ b/list-objects-filter.c > @@ -102,9 +102,16 @@ static enum list_objects_filter_result filter_trees_none( > > case LOFS_BEGIN_TREE: > case LOFS_BLOB: > - if (filter_data->omits) > + if (filter_data->omits) { > oidset_insert(filter_data->omits, &obj->oid); > - return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ > + /* _MARK_SEEN but not _DO_SHOW (hard omit) */ > + return LOFR_MARK_SEEN; > + } > + else > + /* > + * Not collecting omits so no need to to traverse tree. > + */ > + return LOFR_SKIP_TREE | LOFR_MARK_SEEN; OK, so "not collecting omits" is synonymous to "N==0, we aren't doing tree at any level", and at this point in the series before the support for N>0 is introduced, we'd always take this "else" clause because tree:0 is the only thing we support? Style: our modern style is to use {} around the body which is a single statement on the else clause when the body of the corresponding if clause needs {} around (and vice versa), so if (filter_data->omits) { oidset_isnert(...); return ...; } else { return ...; } > diff --git a/list-objects-filter.h b/list-objects-filter.h > index a6f6b4990..52b4a84da 100644 > --- a/list-objects-filter.h > +++ b/list-objects-filter.h > @@ -24,6 +24,11 @@ struct oidset; > * In general, objects should only be shown once, but > * this result DOES NOT imply that we mark it SEEN. > * > + * _SKIP_TREE : Used in LOFS_BEGIN_TREE situation - indicates that > + * the tree's children should not be iterated over. This > + * is used as an optimization when all children will > + * definitely be ignored. > + * > * Most of the time, you want the combination (_MARK_SEEN | _DO_SHOW) > * but they can be used independently, such as when sparse-checkout > * pattern matching is being applied. > @@ -45,6 +50,7 @@ enum list_objects_filter_result { > LOFR_ZERO = 0, > LOFR_MARK_SEEN = 1<<0, > LOFR_DO_SHOW = 1<<1, > + LOFR_SKIP_TREE = 1<<2, > }; > > enum list_objects_filter_situation { > diff --git a/list-objects.c b/list-objects.c > index 7a1a0929d..d1e3d217c 100644 > --- a/list-objects.c > +++ b/list-objects.c > @@ -11,6 +11,7 @@ > #include "list-objects-filter-options.h" > #include "packfile.h" > #include "object-store.h" > +#include "trace.h" > > struct traversal_context { > struct rev_info *revs; > @@ -184,7 +185,9 @@ static void process_tree(struct traversal_context *ctx, > if (base->len) > strbuf_addch(base, '/'); > > - if (!failed_parse) > + if (r & LOFR_SKIP_TREE) > + trace_printf("Skipping contents of tree %s...\n", base->buf); > + else if (!failed_parse) > process_tree_contents(ctx, tree, base); Even when failed_parse==true, i.e. we found that the tree object's data cannot be understood, if we have skip-tree bit set, that means we do not care---we won't be descending into its children anyway. Makes sense. > diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh > index 08e0c7db6..efb1bee2e 100755 > --- a/t/t6112-rev-list-filters-objects.sh > +++ b/t/t6112-rev-list-filters-objects.sh > @@ -244,6 +244,16 @@ test_expect_success 'verify tree:0 includes trees in "filtered" output' ' > test_cmp expected filtered_types > ' > > +# Make sure tree:0 does not iterate through any trees. > + > +test_expect_success 'filter a GIANT tree through tree:0' ' > + GIT_TRACE=1 git -C r3 rev-list \ > + --objects --filter=tree:0 HEAD 2>filter_trace && > + grep "Skipping contents of tree [.][.][.]" filter_trace >actual && Here you are not jus tmaking sure SKIP_TREE bit is set for some tree, but it is set when base->buf is an empty string (i.e. the top level tree)? Which makes sense, and the next text makes sure that between the two commits, the number of total "top level" trees is 2, but I wonder if it is more direct to also make sure that the code is not even seeing or skipping any tree inside these top level trees (i.e. the same message but for ""!=base->buf should never appear in the trace). > + # One line for each commit traversed. > + test_line_count = 2 actual > +' > + > # Delete some loose objects and use rev-list, but WITHOUT any filtering. > # This models previously omitted objects that we did not receive. Thanks.
On Sun, Oct 14, 2018 at 4:15 PM Junio C Hamano <gitster@pobox.com> wrote: > > This step looks more like "ow, we could have done the tree:0 support > that is in 'next' better" than a part of "here is a series to do > tree:N for non zero value of N". > > If that is the case, I'd prefer to see this step polished enough > before [2-3/3] of this RFC is worked on, so that we can merge the > tree:0 (but not yet tree:N) support that is solid down to 'master' > soonish. That's fair. So I will prioritize this patch above the rest of the patchset and send it separately in the future. > OK, so "not collecting omits" is synonymous to "N==0, we aren't > doing tree at any level", and at this point in the series before the > support for N>0 is introduced, we'd always take this "else" clause > because tree:0 is the only thing we support? Actually "not collecting omits" and depth==0 are orthogonal concepts. "Collect omits" is the logic needed to implement the --filter-print-omitted flag on git rev-list. You can do this when depth==0 (it will actually print all the trees and blobs recursively). "Collect omits" is tested *with* tree:0 in test t6112 in the test labeled 'verify tree:0 includes trees in "filtered" output' IOW, both branches here are important even in this patch. If we are collecting omits, then we can't skip the tree because omits collecting is recursive. Otherwise, we *can* skip the tree. But maybe printing omits should not be recursive? The decision was never discussed. The code to not be recursive is simpler, because we don't need this if/else. Recursiveness is counter-intuitive since we would "skip" a tree and at the same time print its contents. > > Style: our modern style is to use {} around the body which is a > single statement on the else clause when the body of the > corresponding if clause needs {} around (and vice versa), so > Fixed, and I didn't realize I was supposed to be hugging "else" with the curly braces. What you say is what CodingGuidelines says to do. Thanks for pointing that out. > > Even when failed_parse==true, i.e. we found that the tree object's > data cannot be understood, if we have skip-tree bit set, that means > we do not care---we won't be descending into its children anyway. > Yes. > > +# Make sure tree:0 does not iterate through any trees. > > + > > +test_expect_success 'filter a GIANT tree through tree:0' ' > > + GIT_TRACE=1 git -C r3 rev-list \ > > + --objects --filter=tree:0 HEAD 2>filter_trace && > > + grep "Skipping contents of tree [.][.][.]" filter_trace >actual && > > Here you are not jus tmaking sure SKIP_TREE bit is set for some > tree, but it is set when base->buf is an empty string (i.e. the top > level tree)? Which makes sense, and the next text makes sure that > between the two commits, the number of total "top level" trees is 2, > but I wonder if it is more direct to also make sure that the code is > not even seeing or skipping any tree inside these top level trees > (i.e. the same message but for ""!=base->buf should never appear in > the trace). Makes sense. I added another check in this test for other "Skipping" messages. Thank you for reviewing.
diff --git a/list-objects-filter.c b/list-objects-filter.c index 09b2b05d5..37fba456d 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -102,9 +102,16 @@ static enum list_objects_filter_result filter_trees_none( case LOFS_BEGIN_TREE: case LOFS_BLOB: - if (filter_data->omits) + if (filter_data->omits) { oidset_insert(filter_data->omits, &obj->oid); - return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ + /* _MARK_SEEN but not _DO_SHOW (hard omit) */ + return LOFR_MARK_SEEN; + } + else + /* + * Not collecting omits so no need to to traverse tree. + */ + return LOFR_SKIP_TREE | LOFR_MARK_SEEN; case LOFS_END_TREE: assert(obj->type == OBJ_TREE); diff --git a/list-objects-filter.h b/list-objects-filter.h index a6f6b4990..52b4a84da 100644 --- a/list-objects-filter.h +++ b/list-objects-filter.h @@ -24,6 +24,11 @@ struct oidset; * In general, objects should only be shown once, but * this result DOES NOT imply that we mark it SEEN. * + * _SKIP_TREE : Used in LOFS_BEGIN_TREE situation - indicates that + * the tree's children should not be iterated over. This + * is used as an optimization when all children will + * definitely be ignored. + * * Most of the time, you want the combination (_MARK_SEEN | _DO_SHOW) * but they can be used independently, such as when sparse-checkout * pattern matching is being applied. @@ -45,6 +50,7 @@ enum list_objects_filter_result { LOFR_ZERO = 0, LOFR_MARK_SEEN = 1<<0, LOFR_DO_SHOW = 1<<1, + LOFR_SKIP_TREE = 1<<2, }; enum list_objects_filter_situation { diff --git a/list-objects.c b/list-objects.c index 7a1a0929d..d1e3d217c 100644 --- a/list-objects.c +++ b/list-objects.c @@ -11,6 +11,7 @@ #include "list-objects-filter-options.h" #include "packfile.h" #include "object-store.h" +#include "trace.h" struct traversal_context { struct rev_info *revs; @@ -184,7 +185,9 @@ static void process_tree(struct traversal_context *ctx, if (base->len) strbuf_addch(base, '/'); - if (!failed_parse) + if (r & LOFR_SKIP_TREE) + trace_printf("Skipping contents of tree %s...\n", base->buf); + else if (!failed_parse) process_tree_contents(ctx, tree, base); if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) { diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index 08e0c7db6..efb1bee2e 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -244,6 +244,16 @@ test_expect_success 'verify tree:0 includes trees in "filtered" output' ' test_cmp expected filtered_types ' +# Make sure tree:0 does not iterate through any trees. + +test_expect_success 'filter a GIANT tree through tree:0' ' + GIT_TRACE=1 git -C r3 rev-list \ + --objects --filter=tree:0 HEAD 2>filter_trace && + grep "Skipping contents of tree [.][.][.]" filter_trace >actual && + # One line for each commit traversed. + test_line_count = 2 actual +' + # Delete some loose objects and use rev-list, but WITHOUT any filtering. # This models previously omitted objects that we did not receive.
The tree:0 filter does not need to traverse the trees that it has filtered out, so optimize list-objects and list-objects-filter to skip traversing the trees entirely. Before this patch, we iterated over all children of the tree, and did nothing for all of them, which was wasteful. Signed-off-by: Matthew DeVore <matvore@google.com> --- list-objects-filter.c | 11 +++++++++-- list-objects-filter.h | 6 ++++++ list-objects.c | 5 ++++- t/t6112-rev-list-filters-objects.sh | 10 ++++++++++ 4 files changed, 29 insertions(+), 3 deletions(-)