From patchwork Sat Jun 1 00:35:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971137 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C2D36C5 for ; Sat, 1 Jun 2019 00:36:21 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EF52C28D7B for ; Sat, 1 Jun 2019 00:36:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E2DFD28D7E; Sat, 1 Jun 2019 00:36:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A1AB628D7B for ; Sat, 1 Jun 2019 00:36:19 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726888AbfFAAgS (ORCPT ); Fri, 31 May 2019 20:36:18 -0400 Received: from mail-pf1-f201.google.com ([209.85.210.201]:49372 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAgS (ORCPT ); Fri, 31 May 2019 20:36:18 -0400 Received: by mail-pf1-f201.google.com with SMTP id r4so8646573pfh.16 for ; Fri, 31 May 2019 17:36:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=fJbwBri9DIQMQcXztSj91a/rwzKE2HReivm+tqz5K24=; b=NIZLiMiOXgMDOUHnNN/lhlDlX8HOe/UoH921l5lK+yQK+tp72ijtbyGz1YlXEn6Q6H R9aDAxsg+D7/tUHDQD9fOLqzGaUsMAm4ZjGwkPORd92L38Oo0YBf9DZADDFfTy3XoXq+ lPHMcrCwRrA6JrCLjLNcOo8VNZEufmBoo3FQzJBnPTJU/8shzPgtavHuvVz0lt2AAL4/ N252K6BtoUwSgGjtowO0gkDYoaqdPHnkVqp8nmZi/HT8FVpUmkYX9niD5W6a8vdVL4mF +6jt+4Jd5gICnMk7igrrZJU/zUVc57XgcfEQtNroAGw+Oigm9SeW5YnmX0Z1BAuNtuSv uvJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=fJbwBri9DIQMQcXztSj91a/rwzKE2HReivm+tqz5K24=; b=ZU/oFci/mGdTy6CG/36XSl8X0g1f/UiSJYmbZYDqoI8R8H97BnOU9xbeciHF3AWEV4 CGEpH3ykr3yLS5J/n+bm7bx6SJQlDMpTl1uJLycjzZkwk7nBox57VUvBJ04z+K6qCm2c yF2xiypMlwVrb71HRQbzY1Lc2076ZrAVgJlAZXWeKK/WsP+y5+CNDwTgUYAq2nlElXPB 71q61UqvNztGrOs4MKaJ5/r3saA5ry/r8b40rtNqd8N4aJdtkFt/lfDID4N2qUAzxoFs rfQoDxpvIQPtdYFanaC2rymkXntVJ/53sZFFfUl4lP4WcmBpElEdtkGq58EIe586RFNu tUig== X-Gm-Message-State: APjAAAXDkrZZnprCdb3LZR8f3uxG+Y92kW8Wm4eCB7vCLWIl6aImy/Eh M7uua3VzWUrn3T9d/ityRK2lr+zPhjRdU2LCC/AzDOkVnFS4rRrlaHzPtarLXDVuYOc5d1R5LFa +UAwLlYGJY/WBy50KnReBLoopIAZIRPkjOlYcNP5VnTNoLS1ZhesaepuBDsI= X-Google-Smtp-Source: APXvYqxR6jQpwoisbJ4XNWaOPrSPl5oIeRkMlUx0Gnc4aVMVVqbrWhuPrGmcdSV2oy5PwDnU7oD113T6v2ed X-Received: by 2002:a63:5d54:: with SMTP id o20mr11854418pgm.97.1559349376481; Fri, 31 May 2019 17:36:16 -0700 (PDT) Date: Fri, 31 May 2019 17:35:55 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-2-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 1/9] list-objects-filter: make API easier to use From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Jeff Hostetler , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make the list-objects-filter.h API more opaque and easier to use. This prepares for combined filter support, where filters will be created and used in a new context. Helped-by: Jeff Hostetler Helped-by: Junio C Hamano Signed-off-by: Matthew DeVore --- list-objects-filter.c | 122 +++++++++++++++++++++++++++--------------- list-objects-filter.h | 35 ++++++------ list-objects.c | 55 ++++++++----------- 3 files changed, 117 insertions(+), 95 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index ee449de3f7..35e0bbe123 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -19,20 +19,34 @@ * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects * that have been shown, but should be revisited if they appear * in the traversal (until we mark it SEEN). This is a way to * let us silently de-dup calls to show() in the caller. This * is subtly different from the "revision.h:SHOWN" and the * "sha1-name.c:ONELINE_SEEN" bits. And also different from * the non-de-dup usage in pack-bitmap.c */ #define FILTER_SHOWN_BUT_REVISIT (1<<21) +struct filter { + enum list_objects_filter_result (*filter_object_fn)( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + void *filter_data); + + void (*free_fn)(void *filter_data); + + void *filter_data; +}; + /* * A filter for list-objects to omit ALL blobs from the traversal. * And to OPTIONALLY collect a list of the omitted OIDs. */ struct filter_blobs_none_data { struct oidset *omits; }; static enum list_objects_filter_result filter_blobs_none( struct repository *r, @@ -60,32 +74,31 @@ static enum list_objects_filter_result filter_blobs_none( case LOFS_BLOB: assert(obj->type == OBJ_BLOB); assert((obj->flags & SEEN) == 0); if (filter_data->omits) oidset_insert(filter_data->omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } } -static void *filter_blobs_none__init( +static void filter_blobs_none__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_blobs_none_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; - *filter_fn = filter_blobs_none; - *filter_free_fn = free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_blobs_none; + filter->free_fn = free; } /* * A filter for list-objects to omit ALL trees and blobs from the traversal. * Can OPTIONALLY collect a list of the omitted OIDs. */ struct filter_trees_depth_data { struct oidset *omits; /* @@ -194,35 +207,34 @@ static enum list_objects_filter_result filter_trees_depth( } static void filter_trees_free(void *filter_data) { struct filter_trees_depth_data *d = filter_data; if (!d) return; oidmap_free(&d->seen_at_depth, 1); free(d); } -static void *filter_trees_depth__init( +static void filter_trees_depth__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_trees_depth_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; oidmap_init(&d->seen_at_depth, 0); d->exclude_depth = filter_options->tree_exclude_depth; d->current_depth = 0; - *filter_fn = filter_trees_depth; - *filter_free_fn = filter_trees_free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_trees_depth; + filter->free_fn = filter_trees_free; } /* * A filter for list-objects to omit large blobs. * And to OPTIONALLY collect a list of the omitted OIDs. */ struct filter_blobs_limit_data { struct oidset *omits; unsigned long max_bytes; }; @@ -274,33 +286,32 @@ static enum list_objects_filter_result filter_blobs_limit( oidset_insert(filter_data->omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } include_it: if (filter_data->omits) oidset_remove(filter_data->omits, &obj->oid); return LOFR_MARK_SEEN | LOFR_DO_SHOW; } -static void *filter_blobs_limit__init( +static void filter_blobs_limit__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; d->max_bytes = filter_options->blob_limit_value; - *filter_fn = filter_blobs_limit; - *filter_free_fn = free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_blobs_limit; + filter->free_fn = free; } /* * A filter driven by a sparse-checkout specification to only * include blobs that a sparse checkout would populate. * * The sparse-checkout spec can be loaded from a blob with the * given OID or from a local pathname. We allow an OID because * the repo may be bare or we may be doing the filtering on the * server. @@ -450,92 +461,117 @@ static enum list_objects_filter_result filter_sparse( } static void filter_sparse_free(void *filter_data) { struct filter_sparse_data *d = filter_data; /* TODO free contents of 'd' */ free(d); } -static void *filter_sparse_oid__init( +static void filter_sparse_oid__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_sparse_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; if (add_excludes_from_blob_to_list(filter_options->sparse_oid_value, NULL, 0, &d->el) < 0) die("could not load filter specification"); ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; - *filter_fn = filter_sparse; - *filter_free_fn = filter_sparse_free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_sparse; + filter->free_fn = filter_sparse_free; } -static void *filter_sparse_path__init( +static void filter_sparse_path__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_sparse_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; if (add_excludes_from_file_to_list(filter_options->sparse_path_value, NULL, 0, &d->el, NULL) < 0) die("could not load filter specification"); ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; - *filter_fn = filter_sparse; - *filter_free_fn = filter_sparse_free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_sparse; + filter->free_fn = filter_sparse_free; } -typedef void *(*filter_init_fn)( +typedef void (*filter_init_fn)( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn); + struct filter *filter); /* * Must match "enum list_objects_filter_choice". */ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, filter_trees_depth__init, filter_sparse_oid__init, filter_sparse_path__init, }; -void *list_objects_filter__init( +struct filter *list_objects_filter__init( struct oidset *omitted, - struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct list_objects_filter_options *filter_options) { + struct filter *filter; filter_init_fn init_fn; assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT); if (filter_options->choice >= LOFC__COUNT) BUG("invalid list-objects filter choice: %d", filter_options->choice); init_fn = s_filters[filter_options->choice]; - if (init_fn) - return init_fn(omitted, filter_options, - filter_fn, filter_free_fn); - *filter_fn = NULL; - *filter_free_fn = NULL; - return NULL; + if (!init_fn) + return NULL; + + filter = xcalloc(1, sizeof(*filter)); + init_fn(omitted, filter_options, filter); + return filter; +} + +enum list_objects_filter_result list_objects_filter__filter_object( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct filter *filter) +{ + if (filter && (obj->flags & NOT_USER_GIVEN)) + return filter->filter_object_fn(r, filter_situation, obj, + pathname, filename, + filter->filter_data); + /* + * No filter is active or user gave object explicitly. Choose default + * behavior based on filter situation. + */ + if (filter_situation == LOFS_END_TREE) + return 0; + return LOFR_MARK_SEEN | LOFR_DO_SHOW; +} + +void list_objects_filter__free(struct filter *filter) +{ + if (!filter) + return; + filter->free_fn(filter->filter_data); + free(filter); } diff --git a/list-objects-filter.h b/list-objects-filter.h index 1d45a4ad57..6908954266 100644 --- a/list-objects-filter.h +++ b/list-objects-filter.h @@ -53,37 +53,34 @@ enum list_objects_filter_result { LOFR_DO_SHOW = 1<<1, LOFR_SKIP_TREE = 1<<2, }; enum list_objects_filter_situation { LOFS_BEGIN_TREE, LOFS_END_TREE, LOFS_BLOB }; -typedef enum list_objects_filter_result (*filter_object_fn)( +struct filter; + +/* Constructor for the set of defined list-objects filters. */ +struct filter *list_objects_filter__init( + struct oidset *omitted, + struct list_objects_filter_options *filter_options); + +/* + * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this + * function behaves as expected if no filter is configured: all objects are + * included. + */ +enum list_objects_filter_result list_objects_filter__filter_object( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, - void *filter_data); - -typedef void (*filter_free_fn)(void *filter_data); + struct filter *filter); -/* - * Constructor for the set of defined list-objects filters. - * Returns a generic "void *filter_data". - * - * The returned "filter_fn" will be used by traverse_commit_list() - * to filter the results. - * - * The returned "filter_free_fn" is a destructor for the - * filter_data. - */ -void *list_objects_filter__init( - struct oidset *omitted, - struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn); +/* Destroys `filter`. Does nothing if `filter` is null. */ +void list_objects_filter__free(struct filter *filter); #endif /* LIST_OBJECTS_FILTER_H */ diff --git a/list-objects.c b/list-objects.c index b5651ddd5b..9307d91fb3 100644 --- a/list-objects.c +++ b/list-objects.c @@ -11,32 +11,31 @@ #include "list-objects-filter-options.h" #include "packfile.h" #include "object-store.h" #include "trace.h" struct traversal_context { struct rev_info *revs; show_object_fn show_object; show_commit_fn show_commit; void *show_data; - filter_object_fn filter_fn; - void *filter_data; + struct filter *filter; }; static void process_blob(struct traversal_context *ctx, struct blob *blob, struct strbuf *path, const char *name) { struct object *obj = &blob->object; size_t pathlen; - enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; + enum list_objects_filter_result r; if (!ctx->revs->blob_objects) return; if (!obj) die("bad blob object"); if (obj->flags & (UNINTERESTING | SEEN)) return; /* * Pre-filter known-missing objects when explicitly requested. @@ -47,25 +46,24 @@ static void process_blob(struct traversal_context *ctx, * may cause the actual filter to report an incomplete list * of missing objects. */ if (ctx->revs->exclude_promisor_objects && !has_object_file(&obj->oid) && is_promisor_object(&obj->oid)) return; pathlen = path->len; strbuf_addstr(path, name); - if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) - r = ctx->filter_fn(ctx->revs->repo, - LOFS_BLOB, obj, - path->buf, &path->buf[pathlen], - ctx->filter_data); + r = list_objects_filter__filter_object(ctx->revs->repo, + LOFS_BLOB, obj, + path->buf, &path->buf[pathlen], + ctx->filter); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) ctx->show_object(obj, path->buf, ctx->show_data); strbuf_setlen(path, pathlen); } /* * Processing a gitlink entry currently does nothing, since * we do not recurse into the subproject. @@ -150,21 +148,21 @@ static void process_tree_contents(struct traversal_context *ctx, } static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, const char *name) { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; int baselen = base->len; - enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; + enum list_objects_filter_result r; int failed_parse; if (!revs->tree_objects) return; if (!obj) die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; failed_parse = parse_tree_gently(tree, 1); @@ -179,47 +177,44 @@ static void process_tree(struct traversal_context *ctx, */ if (revs->exclude_promisor_objects && is_promisor_object(&obj->oid)) return; if (!revs->do_not_die_on_missing_tree) die("bad tree object %s", oid_to_hex(&obj->oid)); } strbuf_addstr(base, name); - if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) - r = ctx->filter_fn(ctx->revs->repo, - LOFS_BEGIN_TREE, obj, - base->buf, &base->buf[baselen], - ctx->filter_data); + r = list_objects_filter__filter_object(ctx->revs->repo, + LOFS_BEGIN_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) ctx->show_object(obj, base->buf, ctx->show_data); if (base->len) strbuf_addch(base, '/'); if (r & LOFR_SKIP_TREE) trace_printf("Skipping contents of tree %s...\n", base->buf); else if (!failed_parse) process_tree_contents(ctx, tree, base); - if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) { - r = ctx->filter_fn(ctx->revs->repo, - LOFS_END_TREE, obj, - base->buf, &base->buf[baselen], - ctx->filter_data); - if (r & LOFR_MARK_SEEN) - obj->flags |= SEEN; - if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); - } + r = list_objects_filter__filter_object(ctx->revs->repo, + LOFS_END_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter); + if (r & LOFR_MARK_SEEN) + obj->flags |= SEEN; + if (r & LOFR_DO_SHOW) + ctx->show_object(obj, base->buf, ctx->show_data); strbuf_setlen(base, baselen); free_tree_buffer(tree); } static void mark_edge_parents_uninteresting(struct commit *commit, struct rev_info *revs, show_edge_fn show_edge) { struct commit_list *parents; @@ -395,38 +390,32 @@ static void do_traverse(struct traversal_context *ctx) void traverse_commit_list(struct rev_info *revs, show_commit_fn show_commit, show_object_fn show_object, void *show_data) { struct traversal_context ctx; ctx.revs = revs; ctx.show_commit = show_commit; ctx.show_object = show_object; ctx.show_data = show_data; - ctx.filter_fn = NULL; - ctx.filter_data = NULL; + ctx.filter = NULL; do_traverse(&ctx); } void traverse_commit_list_filtered( struct list_objects_filter_options *filter_options, struct rev_info *revs, show_commit_fn show_commit, show_object_fn show_object, void *show_data, struct oidset *omitted) { struct traversal_context ctx; - filter_free_fn filter_free_fn = NULL; ctx.revs = revs; ctx.show_object = show_object; ctx.show_commit = show_commit; ctx.show_data = show_data; - ctx.filter_fn = NULL; - - ctx.filter_data = list_objects_filter__init(omitted, filter_options, - &ctx.filter_fn, &filter_free_fn); + ctx.filter = list_objects_filter__init(omitted, filter_options); do_traverse(&ctx); - if (ctx.filter_data && filter_free_fn) - filter_free_fn(ctx.filter_data); + list_objects_filter__free(ctx.filter); } From patchwork Sat Jun 1 00:35:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971139 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 758B76C5 for ; Sat, 1 Jun 2019 00:36:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 659A928D7B for ; Sat, 1 Jun 2019 00:36:22 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 582E428D7E; Sat, 1 Jun 2019 00:36:22 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6AD9928D7B for ; Sat, 1 Jun 2019 00:36:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726924AbfFAAgU (ORCPT ); Fri, 31 May 2019 20:36:20 -0400 Received: from mail-ot1-f73.google.com ([209.85.210.73]:51843 "EHLO mail-ot1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAgU (ORCPT ); Fri, 31 May 2019 20:36:20 -0400 Received: by mail-ot1-f73.google.com with SMTP id 68so5330475otu.18 for ; Fri, 31 May 2019 17:36:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=hdl2BsVLXiyOnyHCPj1YS7p/NDUxBd2TjLElrke5JQA=; b=M8xKJU2/lt3jXkHatWCtByUEOQ8B4csFEvWX9Mmd2Ps7VkqRPTRvVfRCNM+n3GJsvH FEiRMJ5cZwsEHRcD+Zsij87bmqoO10uahpQUCBwS503hayXOIOVNWxeLlSPD6EXVJPvb 7BIZBAPBhgAoxKaZ/fsmL3rtco1Gt4nZAHRzgfiGr7PojkAWLfzvYV94v0GhWkEkOZJs 76QzZi+TLTTE75lHDueIt1XNPgKedwxPF2Oof/ak07nSf1tY5jOUXDOKGBwl/stBvCOU JhsG2djRnmOQthliQxTruqRIlscKxrOJAzkFzEv7RLfZ7Prk1llRHL9/v9KTLbNHv4eu jBYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=hdl2BsVLXiyOnyHCPj1YS7p/NDUxBd2TjLElrke5JQA=; b=I98qIX7m9sQEGL6RoUbGXNFoTqcpdD4nRy/Mu4nOYvxqF5EYsosxrGHPBcPMl/LYqA Zct82fGhDq56tl4xj2rtgPnuii9aihL0gVyKXHJfrVTzUJd5qlulfLYw2MPkzH9fp4wa pdTCKeBypikUAL5XRaL8m25uL15riZQRFo/09Gzwp1cM4h3vJDJeYJneQxbTdO9DBEme TwLNLev8fcBsDwMn9AvL/B9aF79fC4WPNbmA0ie8RaLUIhrXi3WMp6w0VsRdJe/oGPMF z4acciNPx0hKB406NOZWmVJDeZiimgGkgkG9m2TDxLESHaq97UFy6V+7pZ4XHMkJ7qHf PVOA== X-Gm-Message-State: APjAAAW8w9h5BwokVy8FqNCFxfKmDAPA8Pmynjq3foWlVG7nu1qPUwbX 9ff+050VFHOsNk582I/mtlRmopMQc7q/6uaRTgHgor//tWQNquRzw1RihwjSFYKFQK8yIb1ui/H yfiZOndzTcyAUsSm3gw9aqLG5hQOQFd2Ft8F7AL5KD195yl/20hbyJVo+qok= X-Google-Smtp-Source: APXvYqwW5CHQ902RCvwTRLxJ+EKJCSxUj8WI9Rp1HjLvQOj1IXG8ByVOnyF9zfyzvAsGasI7uXDmhebkc4fw X-Received: by 2002:aca:3c1:: with SMTP id 184mr884929oid.170.1559349379086; Fri, 31 May 2019 17:36:19 -0700 (PDT) Date: Fri, 31 May 2019 17:35:56 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-3-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 2/9] list-objects-filter: put omits set in filter struct From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The oidset *omits pointer must be accessed by the combine filter in a type-agnostic way once the graph traversal is over. Store that pointer in the general `filter` struct. This will be used in a follow-up patch to implement the combine filter. Signed-off-by: Matthew DeVore --- list-objects-filter.c | 70 ++++++++++++++++--------------------------- 1 file changed, 26 insertions(+), 44 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index 35e0bbe123..57bbf6ec1c 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -26,88 +26,76 @@ */ #define FILTER_SHOWN_BUT_REVISIT (1<<21) struct filter { enum list_objects_filter_result (*filter_object_fn)( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data); void (*free_fn)(void *filter_data); void *filter_data; -}; -/* - * A filter for list-objects to omit ALL blobs from the traversal. - * And to OPTIONALLY collect a list of the omitted OIDs. - */ -struct filter_blobs_none_data { + /* If non-NULL, the filter collects a list of the omitted OIDs here. */ struct oidset *omits; }; static enum list_objects_filter_result filter_blobs_none( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { - struct filter_blobs_none_data *filter_data = filter_data_; - switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); /* always include all tree objects */ return LOFR_MARK_SEEN | LOFR_DO_SHOW; case LOFS_END_TREE: assert(obj->type == OBJ_TREE); return LOFR_ZERO; case LOFS_BLOB: assert(obj->type == OBJ_BLOB); assert((obj->flags & SEEN) == 0); - if (filter_data->omits) - oidset_insert(filter_data->omits, &obj->oid); + if (omits) + oidset_insert(omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } } static void filter_blobs_none__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { - struct filter_blobs_none_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; - - filter->filter_data = d; filter->filter_object_fn = filter_blobs_none; filter->free_fn = free; } /* * A filter for list-objects to omit ALL trees and blobs from the traversal. * Can OPTIONALLY collect a list of the omitted OIDs. */ struct filter_trees_depth_data { - struct oidset *omits; - /* * Maps trees to the minimum depth at which they were seen. It is not * necessary to re-traverse a tree at deeper or equal depths than it has * already been traversed. * * We can't use LOFR_MARK_SEEN for tree objects since this will prevent * it from being traversed at shallower depths. */ struct oidmap seen_at_depth; @@ -116,38 +104,39 @@ struct filter_trees_depth_data { }; struct seen_map_entry { struct oidmap_entry base; size_t depth; }; /* Returns 1 if the oid was in the omits set before it was invoked. */ static int filter_trees_update_omits( struct object *obj, - struct filter_trees_depth_data *filter_data, + struct oidset *omits, int include_it) { - if (!filter_data->omits) + if (!omits) return 0; if (include_it) - return oidset_remove(filter_data->omits, &obj->oid); + return oidset_remove(omits, &obj->oid); else - return oidset_insert(filter_data->omits, &obj->oid); + return oidset_insert(omits, &obj->oid); } static enum list_objects_filter_result filter_trees_depth( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { struct filter_trees_depth_data *filter_data = filter_data_; struct seen_map_entry *seen_info; int include_it = filter_data->current_depth < filter_data->exclude_depth; int filter_res; int already_seen; /* @@ -158,47 +147,47 @@ static enum list_objects_filter_result filter_trees_depth( switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); case LOFS_END_TREE: assert(obj->type == OBJ_TREE); filter_data->current_depth--; return LOFR_ZERO; case LOFS_BLOB: - filter_trees_update_omits(obj, filter_data, include_it); + filter_trees_update_omits(obj, omits, include_it); return include_it ? LOFR_MARK_SEEN | LOFR_DO_SHOW : LOFR_ZERO; case LOFS_BEGIN_TREE: seen_info = oidmap_get( &filter_data->seen_at_depth, &obj->oid); if (!seen_info) { seen_info = xcalloc(1, sizeof(*seen_info)); oidcpy(&seen_info->base.oid, &obj->oid); seen_info->depth = filter_data->current_depth; oidmap_put(&filter_data->seen_at_depth, seen_info); already_seen = 0; } else { already_seen = filter_data->current_depth >= seen_info->depth; } if (already_seen) { filter_res = LOFR_SKIP_TREE; } else { int been_omitted = filter_trees_update_omits( - obj, filter_data, include_it); + obj, omits, include_it); seen_info->depth = filter_data->current_depth; if (include_it) filter_res = LOFR_DO_SHOW; - else if (filter_data->omits && !been_omitted) + else if (omits && !been_omitted) /* * Must update omit information of children * recursively; they have not been omitted yet. */ filter_res = LOFR_ZERO; else filter_res = LOFR_SKIP_TREE; } filter_data->current_depth++; @@ -208,50 +197,48 @@ static enum list_objects_filter_result filter_trees_depth( static void filter_trees_free(void *filter_data) { struct filter_trees_depth_data *d = filter_data; if (!d) return; oidmap_free(&d->seen_at_depth, 1); free(d); } static void filter_trees_depth__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { struct filter_trees_depth_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; oidmap_init(&d->seen_at_depth, 0); d->exclude_depth = filter_options->tree_exclude_depth; d->current_depth = 0; filter->filter_data = d; filter->filter_object_fn = filter_trees_depth; filter->free_fn = filter_trees_free; } /* * A filter for list-objects to omit large blobs. * And to OPTIONALLY collect a list of the omitted OIDs. */ struct filter_blobs_limit_data { - struct oidset *omits; unsigned long max_bytes; }; static enum list_objects_filter_result filter_blobs_limit( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { struct filter_blobs_limit_data *filter_data = filter_data_; unsigned long object_length; enum object_type t; switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); @@ -275,38 +262,36 @@ static enum list_objects_filter_result filter_blobs_limit( * apply the size filter criteria. Be conservative * and force show it (and let the caller deal with * the ambiguity). */ goto include_it; } if (object_length < filter_data->max_bytes) goto include_it; - if (filter_data->omits) - oidset_insert(filter_data->omits, &obj->oid); + if (omits) + oidset_insert(omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } include_it: - if (filter_data->omits) - oidset_remove(filter_data->omits, &obj->oid); + if (omits) + oidset_remove(omits, &obj->oid); return LOFR_MARK_SEEN | LOFR_DO_SHOW; } static void filter_blobs_limit__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { struct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; d->max_bytes = filter_options->blob_limit_value; filter->filter_data = d; filter->filter_object_fn = filter_blobs_limit; filter->free_fn = free; } /* * A filter driven by a sparse-checkout specification to only * include blobs that a sparse checkout would populate. @@ -330,33 +315,33 @@ struct frame { * omitted objects. * * 0 if everything (recursively) contained in this directory * has been explicitly included (SHOWN) in the result and * the directory may be short-cut later in the traversal. */ unsigned child_prov_omit : 1; }; struct filter_sparse_data { - struct oidset *omits; struct exclude_list el; size_t nr, alloc; struct frame *array_frame; }; static enum list_objects_filter_result filter_sparse( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { struct filter_sparse_data *filter_data = filter_data_; int val, dtype; struct frame *frame; switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); @@ -425,98 +410,93 @@ static enum list_objects_filter_result filter_sparse( frame = &filter_data->array_frame[filter_data->nr]; dtype = DT_REG; val = is_excluded_from_list(pathname, strlen(pathname), filename, &dtype, &filter_data->el, r->index); if (val < 0) val = frame->defval; if (val > 0) { - if (filter_data->omits) - oidset_remove(filter_data->omits, &obj->oid); + if (omits) + oidset_remove(omits, &obj->oid); return LOFR_MARK_SEEN | LOFR_DO_SHOW; } /* * Provisionally omit it. We've already established that * this pathname is not in the sparse-checkout specification * with the CURRENT pathname, so we *WANT* to omit this blob. * * However, a pathname elsewhere in the tree may also * reference this same blob, so we cannot reject it yet. * Leave the LOFR_ bits unset so that if the blob appears * again in the traversal, we will be asked again. */ - if (filter_data->omits) - oidset_insert(filter_data->omits, &obj->oid); + if (omits) + oidset_insert(omits, &obj->oid); /* * Remember that at least 1 blob in this tree was * provisionally omitted. This prevents us from short * cutting the tree in future iterations. */ frame->child_prov_omit = 1; return LOFR_ZERO; } } static void filter_sparse_free(void *filter_data) { struct filter_sparse_data *d = filter_data; /* TODO free contents of 'd' */ free(d); } static void filter_sparse_oid__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { struct filter_sparse_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; if (add_excludes_from_blob_to_list(filter_options->sparse_oid_value, NULL, 0, &d->el) < 0) die("could not load filter specification"); ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; filter->filter_data = d; filter->filter_object_fn = filter_sparse; filter->free_fn = filter_sparse_free; } static void filter_sparse_path__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { struct filter_sparse_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; if (add_excludes_from_file_to_list(filter_options->sparse_path_value, NULL, 0, &d->el, NULL) < 0) die("could not load filter specification"); ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; filter->filter_data = d; filter->filter_object_fn = filter_sparse; filter->free_fn = filter_sparse_free; } typedef void (*filter_init_fn)( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter); /* * Must match "enum list_objects_filter_choice". */ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, @@ -536,35 +516,37 @@ struct filter *list_objects_filter__init( if (filter_options->choice >= LOFC__COUNT) BUG("invalid list-objects filter choice: %d", filter_options->choice); init_fn = s_filters[filter_options->choice]; if (!init_fn) return NULL; filter = xcalloc(1, sizeof(*filter)); - init_fn(omitted, filter_options, filter); + filter->omits = omitted; + init_fn(filter_options, filter); return filter; } enum list_objects_filter_result list_objects_filter__filter_object( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, struct filter *filter) { if (filter && (obj->flags & NOT_USER_GIVEN)) return filter->filter_object_fn(r, filter_situation, obj, pathname, filename, + filter->omits, filter->filter_data); /* * No filter is active or user gave object explicitly. Choose default * behavior based on filter situation. */ if (filter_situation == LOFS_END_TREE) return 0; return LOFR_MARK_SEEN | LOFR_DO_SHOW; } From patchwork Sat Jun 1 00:35:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971141 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F3DE13AD for ; Sat, 1 Jun 2019 00:36:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 603F228D7B for ; Sat, 1 Jun 2019 00:36:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 53DBE28D7E; Sat, 1 Jun 2019 00:36:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E518A28D7B for ; Sat, 1 Jun 2019 00:36:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726946AbfFAAgY (ORCPT ); Fri, 31 May 2019 20:36:24 -0400 Received: from mail-vs1-f74.google.com ([209.85.217.74]:52691 "EHLO mail-vs1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAgX (ORCPT ); Fri, 31 May 2019 20:36:23 -0400 Received: by mail-vs1-f74.google.com with SMTP id x10so2338837vsj.19 for ; Fri, 31 May 2019 17:36:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Etk/YcOOQAFgNrjKK70hHfrOn3ZSVae7qoO5uYh1Yjk=; b=Ht/ZzIWBjcj3Q5hqchguGtQLe1sKyLVYlcbg1NKJVBecNCYBHHb/Dc9TCT19L5rwxg 2YGBzsj5fQ3qpW4TkyViPsBtYi9+V8h/FF8X5RVh3xxttMzH5AUVCeqaMpmjJqkq8RoR smKM1Y1L3HDFQGIqx3Z0EJU+RNJaqMyUau9q012wTlQVnERt1H4ZGLi6CUhAGQIiscOo 07zg81bcKrwm3J4U81EEZMwUmY9aYhvy6jiys4OZ2qR4me+dPVkbJ5XzuuqregZwTDiw Gjhq81YcGt2PMYpvQwtDdkvcZNQTpl2gqhTr6oHNV/eW6775yfThLt0GStnbg8xKow5g G40A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Etk/YcOOQAFgNrjKK70hHfrOn3ZSVae7qoO5uYh1Yjk=; b=hNli3srukZHK3+yEF1ED7gj5ndXedV4gF0p0L3FLUtJEo9oRMfcPrk9H8UnUgEx0e+ 2BsuxS7L3x0szJRcR11gmsKZkWhWf6JfLBh7LvXjVWmoAmeHMmxyjzS7YcijX82hpeA4 ltbqTLitMmkfqMW0qHZ0AF/hmpK8hTRq/XWDB4a4iA61tBEuzGW8D4kk4R/MNhcXFOUY 1F88uYbhbxRlP0Ar0gJA+S9mzzo3xfwy6TxW5ThtarGZ1wtV+qbbqjaf2SsH2M5YW2Mm 182FTwPCNLCX53/zVrlpR9HRvuypEngEfhxOtVF3TPrW6vGfGMd0C5ynC2LUFeAN4C5Q TV2Q== X-Gm-Message-State: APjAAAWqjxT7AIKW14u+VFs2IXrBqLaYHaSBf36FqQo1vE5+d2sQn6rP DM31PcLyRSVFEIPQZX6f+BsSvEpFglunE0zIWpsQiYr8iuYm4Bu5uWdhYAPuvh9UDaGcmAU/JG0 J5XClGffWQSE5JZaZPtKLlnn8ndCjtLmdgmfMaqCvETULBYyA3HCY6boUYxI= X-Google-Smtp-Source: APXvYqy9ElqofoyS2kSOmucs+GN6h1RR3Rb4nIoO0NmzRjiTWcP8i/FLClUqp8l+y8KHslY4iqhogJkOEqyU X-Received: by 2002:a67:b30b:: with SMTP id a11mr6884958vsm.86.1559349382624; Fri, 31 May 2019 17:36:22 -0700 (PDT) Date: Fri, 31 May 2019 17:35:57 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-4-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 3/9] list-objects-filter-options: always supply *errbuf From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Making errbuf an optional argument complicates error reporting. Fix this by making all callers supply an errbuf, even if they may ignore it. This will be important in follow-up patches where the filter-spec parsing has more pitfalls and possible errors. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0036f7378..aef24ddae3 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -23,47 +23,40 @@ * convenience of the current command. */ static int gently_parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg, struct strbuf *errbuf) { const char *v0; if (filter_options->choice) { - if (errbuf) { - strbuf_addstr( - errbuf, - _("multiple filter-specs cannot be combined")); - } + strbuf_addstr( + errbuf, _("multiple filter-specs cannot be combined")); return 1; } filter_options->filter_spec = strdup(arg); if (!strcmp(arg, "blob:none")) { filter_options->choice = LOFC_BLOB_NONE; return 0; } else if (skip_prefix(arg, "blob:limit=", &v0)) { if (git_parse_ulong(v0, &filter_options->blob_limit_value)) { filter_options->choice = LOFC_BLOB_LIMIT; return 0; } } else if (skip_prefix(arg, "tree:", &v0)) { if (!git_parse_ulong(v0, &filter_options->tree_exclude_depth)) { - if (errbuf) { - strbuf_addstr( - errbuf, - _("expected 'tree:'")); - } + strbuf_addstr(errbuf, _("expected 'tree:'")); return 1; } filter_options->choice = LOFC_TREE_DEPTH; return 0; } else if (skip_prefix(arg, "sparse:oid=", &v0)) { struct object_context oc; struct object_id sparse_oid; /* @@ -80,22 +73,21 @@ static int gently_parse_list_objects_filter( } else if (skip_prefix(arg, "sparse:path=", &v0)) { filter_options->choice = LOFC_SPARSE_PATH; filter_options->sparse_path_value = strdup(v0); return 0; } /* * Please update _git_fetch() in git-completion.bash when you * add new filters */ - if (errbuf) - strbuf_addf(errbuf, "invalid filter-spec '%s'", arg); + strbuf_addf(errbuf, "invalid filter-spec '%s'", arg); memset(filter_options, 0, sizeof(*filter_options)); return 1; } int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; if (gently_parse_list_objects_filter(filter_options, arg, &buf)) @@ -166,19 +158,22 @@ void partial_clone_register( */ core_partial_clone_filter_default = xstrdup(filter_options->filter_spec); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); } void partial_clone_get_default_filter_spec( struct list_objects_filter_options *filter_options) { + struct strbuf errbuf = STRBUF_INIT; + /* * Parse default value, but silently ignore it if it is invalid. */ if (!core_partial_clone_filter_default) return; gently_parse_list_objects_filter(filter_options, core_partial_clone_filter_default, - NULL); + &errbuf); + strbuf_release(&errbuf); } From patchwork Sat Jun 1 00:35:58 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971143 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8CD4413AD for ; Sat, 1 Jun 2019 00:36:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7C4AF28D7B for ; Sat, 1 Jun 2019 00:36:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6FB7228D7E; Sat, 1 Jun 2019 00:36:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D779228D7B for ; Sat, 1 Jun 2019 00:36:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726960AbfFAAg0 (ORCPT ); Fri, 31 May 2019 20:36:26 -0400 Received: from mail-pf1-f201.google.com ([209.85.210.201]:36269 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAg0 (ORCPT ); Fri, 31 May 2019 20:36:26 -0400 Received: by mail-pf1-f201.google.com with SMTP id d125so8681178pfd.3 for ; Fri, 31 May 2019 17:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9dhVaOCyoakx7Lu2qKrXBfmPmcmSkAN/X4BQfbwySik=; b=s/Qyv/T5MJjhN5So4t+FPEqhijcNTSO1omxuQEHHtwEf2+m977GU8oYgGEi0Y+jL31 fDPkvBv4kobBLhvFFdj4xXX8BwPpLaei9SLVo/cq7CNnfQOmPJ0l6dD8SKyFc6/43maG RsDoeDIH5tRoqdLY1LmOIrn+RvfETWZu5i2B/hJmjjO23RnjtMVhJMiGYi45KlPd3RzZ YDdfDYQJNJTm6EXI4Yv1jseUSu96u2AeHslFzP1jEoPMuN6yDFBLDHODCWwWxgxU22PA 5Nff1Ml2O7n5a/Nz3SA+ippyTQ9vaoNFbXr7L3x0+qp1gygsuHVqqtUYrr6+UtQy7Z8B imKg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9dhVaOCyoakx7Lu2qKrXBfmPmcmSkAN/X4BQfbwySik=; b=ptI5Yud99ANb6+2YneLgnV84oIxjBvqc4U6PD3YgEF2oubAkQV1Av0Og/XSodI0yKX 5L+k6gM0+H7C8mTXCMcpTD0dcgRt0vn37eG5OKeojNvyZgnppVUclOBHEAMo9zpNDr7z EXvLxYMGm9eg//YN1MlyqWE+9ncb4TNKHa55AhNqWLjWGKLMTpdHz8OQmwDIHLZiG+rJ bS68hmHx1qt4mLT+G6EcnIvcsiJWHtmf775PDoQQHXamfAWjMOk78NDfBdcz7RZPH8PN dYHo0yWEAP/epQyyQRtyI0hiD8LsHq8rV/ZU46s9dDgWHGrblfTybnOZFub/tNYsAHVl u9RQ== X-Gm-Message-State: APjAAAW7qyqgQcb2SKiRnxsFPJHi1I64Ug0A0uJT2J8AT8HztQlwd7oB Yj9V+1ddNz1b1+KDZ5U/e0DwIwc8g/G9MBW0CccmqlUucLaz062S791oS32L4BJGLqFam/ftC0O 1pd6XHFzKofvVLpcqldbKxS2bRizOGOFGztG3eLdSvKvWyKB4bKptuQXGSeg= X-Google-Smtp-Source: APXvYqzFSwl3/ATqpWcT7Ytda6HPXfakK7X3UlhJK22sSjqcsnpx/roGSlVsOS9FZSg2fxB3S9DhgutNr5LO X-Received: by 2002:a63:5d45:: with SMTP id o5mr12693574pgm.40.1559349385298; Fri, 31 May 2019 17:36:25 -0700 (PDT) Date: Fri, 31 May 2019 17:35:58 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-5-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 4/9] list-objects-filter: implement composite filters From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Jeff Hostetler , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allow combining filters such that only objects accepted by all filters are shown. The motivation for this is to allow getting directory listings without also fetching blobs. This can be done by combining blob:none with tree:. There are massive repositories that have larger-than-expected trees - even if you include only a single commit. The current usage requires passing the filter to rev-list in the following form: --filter= --filter= ... Such usage is currently an error, so giving it a meaning is backwards- compatible. The URL-encoding method is being implemented before the repeated flag logic, and the user-facing documentation for URL-encoding is being withheld until the repeated flag feature is implemented. The URL-encoding is in general not meant to be used directly by the user, and it is better to describe the URL-encoding feature in terms of the repeated flag. Helped-by: Emily Shaffer Helped-by: Jeff Hostetler Helped-by: Junio C Hamano Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 135 ++++++++++++++++++++++- list-objects-filter-options.h | 17 ++- list-objects-filter.c | 159 +++++++++++++++++++++++++++ t/t6112-rev-list-filters-objects.sh | 163 +++++++++++++++++++++++++++- 4 files changed, 468 insertions(+), 6 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index aef24ddae3..0f1d4181cb 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -1,19 +1,24 @@ #include "cache.h" #include "commit.h" #include "config.h" #include "revision.h" #include "argv-array.h" #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +static int parse_combine_filter( + struct list_objects_filter_options *filter_options, + const char *arg, + struct strbuf *errbuf); + /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: * --filter= * and in the pack protocol as: * "filter" SP * * The filter keyword will be used by many commands. * See Documentation/rev-list-options.txt for allowed values for . * @@ -28,22 +33,20 @@ static int gently_parse_list_objects_filter( struct strbuf *errbuf) { const char *v0; if (filter_options->choice) { strbuf_addstr( errbuf, _("multiple filter-specs cannot be combined")); return 1; } - filter_options->filter_spec = strdup(arg); - if (!strcmp(arg, "blob:none")) { filter_options->choice = LOFC_BLOB_NONE; return 0; } else if (skip_prefix(arg, "blob:limit=", &v0)) { if (git_parse_ulong(v0, &filter_options->blob_limit_value)) { filter_options->choice = LOFC_BLOB_LIMIT; return 0; } @@ -67,36 +70,155 @@ static int gently_parse_list_objects_filter( if (!get_oid_with_context(the_repository, v0, GET_OID_BLOB, &sparse_oid, &oc)) filter_options->sparse_oid_value = oiddup(&sparse_oid); filter_options->choice = LOFC_SPARSE_OID; return 0; } else if (skip_prefix(arg, "sparse:path=", &v0)) { filter_options->choice = LOFC_SPARSE_PATH; filter_options->sparse_path_value = strdup(v0); return 0; + + } else if (skip_prefix(arg, "combine:", &v0)) { + return parse_combine_filter(filter_options, v0, errbuf); + } /* * Please update _git_fetch() in git-completion.bash when you * add new filters */ strbuf_addf(errbuf, "invalid filter-spec '%s'", arg); memset(filter_options, 0, sizeof(*filter_options)); return 1; } +static int url_decode(struct strbuf *s, struct strbuf *errbuf) +{ + char *dest = s->buf; + char *src = s->buf; + size_t new_len; + + while (*src) { + if (src[0] != '%') { + *dest++ = *src++; + continue; + } + + if (hex_to_bytes((unsigned char *)dest, src + 1, 1)) { + strbuf_addstr(errbuf, + "error in filter-spec - " + "invalid hex sequence after %"); + return 1; + } + + if (!*dest) { + strbuf_addstr(errbuf, + "error in filter-spec - unexpected %00"); + return 1; + } + + src += 3; + dest++; + } + new_len = dest - s->buf; + strbuf_remove(s, new_len, s->len - new_len); + + return 0; +} + +static const char *RESERVED_NON_WS = "~`!@#$^&*()[]{}\\;'\",<>?"; + +static int has_reserved_character( + struct strbuf *sub_spec, struct strbuf *errbuf) +{ + const char *c = sub_spec->buf; + while (*c) { + if (*c <= ' ' || strchr(RESERVED_NON_WS, *c)) { + strbuf_addf(errbuf, + "must escape char in sub-filter-spec: '%c'", + *c); + return 1; + } + c++; + } + + return 0; +} + +static int parse_combine_subfilter( + struct list_objects_filter_options *filter_options, + struct strbuf *subspec, + struct strbuf *errbuf) +{ + size_t new_index = filter_options->sub_nr++; + + ALLOC_GROW(filter_options->sub, filter_options->sub_nr, + filter_options->sub_alloc); + memset(&filter_options->sub[new_index], 0, + sizeof(*filter_options->sub)); + + return has_reserved_character(subspec, errbuf) || + url_decode(subspec, errbuf) || + gently_parse_list_objects_filter( + &filter_options->sub[new_index], subspec->buf, errbuf); +} + +static int parse_combine_filter( + struct list_objects_filter_options *filter_options, + const char *arg, + struct strbuf *errbuf) +{ + struct strbuf **subspecs = strbuf_split_str(arg, '+', 0); + size_t sub; + int result; + + if (!subspecs[0]) { + strbuf_addf(errbuf, + _("expected something after combine:")); + result = 1; + goto cleanup; + } + + for (sub = 0; subspecs[sub]; sub++) { + if (subspecs[sub + 1]) { + /* + * This is not the last subspec. Remove trailing "+" so + * we can parse it. + */ + size_t last = subspecs[sub]->len - 1; + assert(subspecs[sub]->buf[last] == '+'); + strbuf_remove(subspecs[sub], last, 1); + } + result = parse_combine_subfilter( + filter_options, subspecs[sub], errbuf); + if (result) + goto cleanup; + } + + filter_options->choice = LOFC_COMBINE; + +cleanup: + strbuf_list_free(subspecs); + if (result) { + list_objects_filter_release(filter_options); + memset(filter_options, 0, sizeof(*filter_options)); + } + return result; +} + int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; + filter_options->filter_spec = strdup(arg); if (gently_parse_list_objects_filter(filter_options, arg, &buf)) die("%s", buf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; @@ -119,23 +241,30 @@ void expand_list_objects_filter_spec( else if (filter->choice == LOFC_TREE_DEPTH) strbuf_addf(expanded_spec, "tree:%lu", filter->tree_exclude_depth); else strbuf_addstr(expanded_spec, filter->filter_spec); } void list_objects_filter_release( struct list_objects_filter_options *filter_options) { + size_t sub; + + if (!filter_options) + return; free(filter_options->filter_spec); free(filter_options->sparse_oid_value); free(filter_options->sparse_path_value); + for (sub = 0; sub < filter_options->sub_nr; sub++) + list_objects_filter_release(&filter_options->sub[sub]); + free(filter_options->sub); memset(filter_options, 0, sizeof(*filter_options)); } void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options) { /* * Record the name of the partial clone remote in the * config and in the global variable -- the latter is @@ -165,15 +294,17 @@ void partial_clone_register( void partial_clone_get_default_filter_spec( struct list_objects_filter_options *filter_options) { struct strbuf errbuf = STRBUF_INIT; /* * Parse default value, but silently ignore it if it is invalid. */ if (!core_partial_clone_filter_default) return; + + filter_options->filter_spec = strdup(core_partial_clone_filter_default); gently_parse_list_objects_filter(filter_options, core_partial_clone_filter_default, &errbuf); strbuf_release(&errbuf); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index e3adc78ebf..8f08ed74a1 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -7,20 +7,21 @@ /* * The list of defined filters for list-objects. */ enum list_objects_filter_choice { LOFC_DISABLED = 0, LOFC_BLOB_NONE, LOFC_BLOB_LIMIT, LOFC_TREE_DEPTH, LOFC_SPARSE_OID, LOFC_SPARSE_PATH, + LOFC_COMBINE, LOFC__COUNT /* must be last */ }; struct list_objects_filter_options { /* * 'filter_spec' is the raw argument value given on the command line * or protocol request. (The part after the "--keyword=".) For * commands that launch filtering sub-processes, or for communication * over the network, don't use this value; use the result of * expand_list_objects_filter_spec() instead. @@ -32,28 +33,38 @@ struct list_objects_filter_options { * the filtering algorithm to use. */ enum list_objects_filter_choice choice; /* * Choice is LOFC_DISABLED because "--no-filter" was requested. */ unsigned int no_filter : 1; /* - * Parsed values (fields) from within the filter-spec. These are - * choice-specific; not all values will be defined for any given - * choice. + * BEGIN choice-specific parsed values from within the filter-spec. Only + * some values will be defined for any given choice. */ + struct object_id *sparse_oid_value; char *sparse_path_value; unsigned long blob_limit_value; unsigned long tree_exclude_depth; + + /* LOFC_COMBINE values */ + + /* This array contains all the subfilters which this filter combines. */ + size_t sub_nr, sub_alloc; + struct list_objects_filter_options *sub; + + /* + * END choice-specific parsed values. + */ }; /* Normalized command line arguments */ #define CL_ARG__FILTER "filter" int parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg); int opt_parse_list_objects_filter(const struct option *opt, diff --git a/list-objects-filter.c b/list-objects-filter.c index 57bbf6ec1c..c8a006edf9 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -19,30 +19,45 @@ * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects * that have been shown, but should be revisited if they appear * in the traversal (until we mark it SEEN). This is a way to * let us silently de-dup calls to show() in the caller. This * is subtly different from the "revision.h:SHOWN" and the * "sha1-name.c:ONELINE_SEEN" bits. And also different from * the non-de-dup usage in pack-bitmap.c */ #define FILTER_SHOWN_BUT_REVISIT (1<<21) +struct subfilter { + struct filter *filter; + struct oidset seen; + struct oidset omits; + struct object_id skip_tree; + unsigned is_skipping_tree : 1; +}; + struct filter { enum list_objects_filter_result (*filter_object_fn)( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, struct oidset *omits, void *filter_data); + /* + * Optional. If this function is supplied and the filter needs to + * collect omits, then this function is called once before free_fn is + * called. + */ + void (*finalize_omits_fn)(struct oidset *omits, void *filter_data); + void (*free_fn)(void *filter_data); void *filter_data; /* If non-NULL, the filter collects a list of the omitted OIDs here. */ struct oidset *omits; }; static enum list_objects_filter_result filter_blobs_none( struct repository *r, @@ -482,34 +497,176 @@ static void filter_sparse_path__init( ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; filter->filter_data = d; filter->filter_object_fn = filter_sparse; filter->free_fn = filter_sparse_free; } +/* A filter which only shows objects shown by all sub-filters. */ +struct combine_filter_data { + struct subfilter *sub; + size_t nr; +}; + +static int should_delegate(enum list_objects_filter_situation filter_situation, + struct object *obj, + struct subfilter *sub) +{ + if (!sub->is_skipping_tree) + return 1; + if (filter_situation == LOFS_END_TREE && + oideq(&obj->oid, &sub->skip_tree)) { + sub->is_skipping_tree = 0; + return 1; + } + return 0; +} + +static enum list_objects_filter_result process_subfilter( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct subfilter *sub) +{ + enum list_objects_filter_result result; + + /* + * Check should_delegate before oidset_contains so that + * is_skipping_tree gets unset even when the object is marked as seen. + * As of this writing, no filter uses LOFR_MARK_SEEN on trees that also + * uses LOFR_SKIP_TREE, so the ordering is only theoretically + * important. Be cautious if you change the order of the below checks + * and more filters have been added! + */ + if (!should_delegate(filter_situation, obj, sub)) + return LOFR_ZERO; + if (oidset_contains(&sub->seen, &obj->oid)) + return LOFR_ZERO; + + result = list_objects_filter__filter_object( + r, filter_situation, obj, pathname, filename, sub->filter); + + if (result & LOFR_MARK_SEEN) + oidset_insert(&sub->seen, &obj->oid); + + if (result & LOFR_SKIP_TREE) { + sub->is_skipping_tree = 1; + sub->skip_tree = obj->oid; + } + + return result; +} + +static enum list_objects_filter_result filter_combine( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct oidset *omits, + void *filter_data) +{ + struct combine_filter_data *d = filter_data; + enum list_objects_filter_result combined_result = + LOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE; + size_t sub; + + for (sub = 0; sub < d->nr; sub++) { + enum list_objects_filter_result sub_result = process_subfilter( + r, filter_situation, obj, pathname, filename, + &d->sub[sub]); + if (!(sub_result & LOFR_DO_SHOW)) + combined_result &= ~LOFR_DO_SHOW; + if (!(sub_result & LOFR_MARK_SEEN)) + combined_result &= ~LOFR_MARK_SEEN; + if (!d->sub[sub].is_skipping_tree) + combined_result &= ~LOFR_SKIP_TREE; + } + + return combined_result; +} + +static void filter_combine__free(void *filter_data) +{ + struct combine_filter_data *d = filter_data; + size_t sub; + for (sub = 0; sub < d->nr; sub++) { + list_objects_filter__free(d->sub[sub].filter); + oidset_clear(&d->sub[sub].seen); + if (d->sub[sub].omits.set.size) + BUG("expected oidset to be cleared already"); + } + free(d->sub); +} + +static void add_all(struct oidset *dest, struct oidset *src) { + struct oidset_iter iter; + struct object_id *src_oid; + + oidset_iter_init(src, &iter); + while ((src_oid = oidset_iter_next(&iter)) != NULL) + oidset_insert(dest, src_oid); +} + +static void filter_combine__finalize_omits( + struct oidset *omits, + void *filter_data) +{ + struct combine_filter_data *d = filter_data; + size_t sub; + + for (sub = 0; sub < d->nr; sub++) { + add_all(omits, &d->sub[sub].omits); + oidset_clear(&d->sub[sub].omits); + } +} + +static void filter_combine__init( + struct list_objects_filter_options *filter_options, + struct filter* filter) +{ + struct combine_filter_data *d = xcalloc(1, sizeof(*d)); + size_t sub; + + d->nr = filter_options->sub_nr; + d->sub = xcalloc(d->nr, sizeof(*d->sub)); + for (sub = 0; sub < d->nr; sub++) + d->sub[sub].filter = list_objects_filter__init( + filter->omits ? &d->sub[sub].omits : NULL, + &filter_options->sub[sub]); + + filter->filter_data = d; + filter->filter_object_fn = filter_combine; + filter->free_fn = filter_combine__free; + filter->finalize_omits_fn = filter_combine__finalize_omits; +} + typedef void (*filter_init_fn)( struct list_objects_filter_options *filter_options, struct filter *filter); /* * Must match "enum list_objects_filter_choice". */ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, filter_trees_depth__init, filter_sparse_oid__init, filter_sparse_path__init, + filter_combine__init, }; struct filter *list_objects_filter__init( struct oidset *omitted, struct list_objects_filter_options *filter_options) { struct filter *filter; filter_init_fn init_fn; assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT); @@ -547,13 +704,15 @@ enum list_objects_filter_result list_objects_filter__filter_object( */ if (filter_situation == LOFS_END_TREE) return 0; return LOFR_MARK_SEEN | LOFR_DO_SHOW; } void list_objects_filter__free(struct filter *filter) { if (!filter) return; + if (filter->finalize_omits_fn && filter->omits) + filter->finalize_omits_fn(filter->omits, filter->filter_data); filter->free_fn(filter->filter_data); free(filter); } diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index 9c11427719..c36199457d 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -284,21 +284,33 @@ test_expect_success 'verify tree:0 includes trees in "filtered" output' ' # Make sure tree:0 does not iterate through any trees. test_expect_success 'verify skipping tree iteration when not collecting omits' ' GIT_TRACE=1 git -C r3 rev-list \ --objects --filter=tree:0 HEAD 2>filter_trace && grep "Skipping contents of tree [.][.][.]" filter_trace >actual && # One line for each commit traversed. test_line_count = 2 actual && # Make sure no other trees were considered besides the root. - ! grep "Skipping contents of tree [^.]" filter_trace + ! grep "Skipping contents of tree [^.]" filter_trace && + + # Try this again with "combine:". If both sub-filters are skipping + # trees, the composite filter should also skip trees. This is not + # important unless the user does combine:tree:X+tree:Y or another filter + # besides "tree:" is implemented in the future which can skip trees. + GIT_TRACE=1 git -C r3 rev-list \ + --objects --filter=combine:tree:1+tree:3 HEAD 2>filter_trace && + + # Only skip the dir1/ tree, which is shared between the two commits. + grep "Skipping contents of tree " filter_trace >actual && + test_write_lines "Skipping contents of tree dir1/..." >expected && + test_cmp expected actual ' # Test tree:# filters. expect_has () { commit=$1 && name=$2 && hash=$(git -C r3 rev-parse $commit:$name) && grep "^$hash $name$" actual @@ -336,20 +348,138 @@ test_expect_success 'verify tree:3 includes everything expected' ' expect_has HEAD dir1/sparse1 && expect_has HEAD dir1/sparse2 && expect_has HEAD pattern && expect_has HEAD sparse1 && expect_has HEAD sparse2 && # There are also 2 commit objects test_line_count = 10 actual ' +test_expect_success 'combine:... for a simple combination' ' + git -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \ + >actual && + + expect_has HEAD "" && + expect_has HEAD~1 "" && + expect_has HEAD dir1 && + + # There are also 2 commit objects + test_line_count = 5 actual +' + +test_expect_success 'combine:... with URL encoding' ' + git -C r3 rev-list --objects \ + --filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual && + + expect_has HEAD "" && + expect_has HEAD~1 "" && + expect_has HEAD dir1 && + + # There are also 2 commit objects + test_line_count = 5 actual +' + +expect_invalid_filter_spec () { + spec="$1" && + err="$2" && + + test_must_fail git -C r3 rev-list --objects --filter="$spec" HEAD \ + >actual 2>actual_stderr && + test_must_be_empty actual && + test_i18ngrep "$err" actual_stderr +} + +test_expect_success 'combine:... while URL-encoding things that should not be' ' + expect_invalid_filter_spec combine%3Atree:2+blob:none \ + "invalid filter-spec" +' + +test_expect_success 'combine: with nothing after the :' ' + expect_invalid_filter_spec combine: "expected something after combine:" +' + +test_expect_success 'parse error in first sub-filter in combine:' ' + expect_invalid_filter_spec combine:tree:asdf+blob:none \ + "expected .tree:." +' + +test_expect_success 'combine:... with invalid URL-encoded sequences' ' + # Not enough hex chars + expect_invalid_filter_spec combine:tree:2+blob:non%a \ + "error in filter-spec - invalid hex sequence after %" && + # Non-hex digit after % + expect_invalid_filter_spec combine:tree:2+blob%G5none \ + "error in filter-spec - invalid hex sequence after %" && + # Null byte encoded by % + expect_invalid_filter_spec combine:tree:2+blob%00none \ + "error in filter-spec - unexpected %00" +' + +test_expect_success 'combine:... with non-encoded reserved chars' ' + expect_invalid_filter_spec combine:tree:2+sparse:@xyz \ + "must escape char in sub-filter-spec: .@." && + expect_invalid_filter_spec combine:tree:2+sparse:\` \ + "must escape char in sub-filter-spec: .\`." && + expect_invalid_filter_spec combine:tree:2+sparse:~abc \ + "must escape char in sub-filter-spec: .\~." +' + +test_expect_success 'validate err msg for "combine:+"' ' + expect_invalid_filter_spec combine:tree:2+ "expected .tree:." +' + +test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' ' + git -C r3 rev-list --objects --filter="combine:tree:2+bl%6Fb:n%6fne" \ + HEAD >actual && + test_line_count = 5 actual && + git -C r3 rev-list --objects --filter="combine:tree%3A2+blob%3anone" \ + HEAD >actual && + test_line_count = 5 actual && + git -C r3 rev-list --objects --filter="combine:tree:%30" HEAD >actual && + test_line_count = 2 actual && + git -C r3 rev-list --objects --filter="combine:tree:%39+blob:none" \ + HEAD >actual && + test_line_count = 5 actual +' + +test_expect_success 'add a sparse pattern blob whose path has reserved chars' ' + cp r3/pattern r3/pattern1+renamed% && + git -C r3 add pattern1+renamed% && + git -C r3 commit -m "add sparse pattern file with reserved chars" +' + +test_expect_success 'combine:... with more than two sub-filters' ' + git -C r3 rev-list --objects \ + --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \ + HEAD >actual && + + expect_has HEAD "" && + expect_has HEAD~1 "" && + expect_has HEAD~2 "" && + expect_has HEAD dir1 && + expect_has HEAD dir1/sparse1 && + expect_has HEAD dir1/sparse2 && + + # Should also have 3 commits + test_line_count = 9 actual && + + # Try again, this time making sure the last sub-filter is only + # URL-decoded once. + cp actual expect && + + git -C r3 rev-list --objects \ + --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \ + HEAD >actual && + test_cmp expect actual +' + # Test provisional omit collection logic with a repo that has objects appearing # at multiple depths - first deeper than the filter's threshold, then shallow. test_expect_success 'setup r4' ' git init r4 && echo foo > r4/foo && mkdir r4/subdir && echo bar > r4/subdir/bar && @@ -379,20 +509,51 @@ test_expect_success 'test tree:# filter provisional omit for blob and tree' ' test_expect_success 'verify skipping tree iteration when collecting omits' ' GIT_TRACE=1 git -C r4 rev-list --filter-print-omitted \ --objects --filter=tree:0 HEAD 2>filter_trace && grep "^Skipping contents of tree " filter_trace >actual && echo "Skipping contents of tree subdir/..." >expect && test_cmp expect actual ' +test_expect_success 'setup r5' ' + git init r5 && + mkdir -p r5/subdir && + + echo 1 >r5/short-root && + echo 12345 >r5/long-root && + echo a >r5/subdir/short-subdir && + echo abcde >r5/subdir/long-subdir && + + git -C r5 add short-root long-root subdir && + git -C r5 commit -m "commit msg" +' + +test_expect_success 'verify collecting omits in combined: filter' ' + # Note that this test guards against the naive implementation of simply + # giving both filters the same "omits" set and expecting it to + # automatically merge them. + git -C r5 rev-list --objects --quiet --filter-print-omitted \ + --filter=combine:tree:2+blob:limit=3 HEAD >actual && + + # Expect 0 trees/commits, 3 blobs omitted (all blobs except short-root) + omitted_1=$(echo 12345 | git hash-object --stdin) && + omitted_2=$(echo a | git hash-object --stdin) && + omitted_3=$(echo abcde | git hash-object --stdin) && + + grep ~$omitted_1 actual && + grep ~$omitted_2 actual && + grep ~$omitted_3 actual && + test_line_count = 3 actual +' + # Test tree: where a tree is iterated to twice - once where a subentry is # too deep to be included, and again where the blob inside it is shallow enough # to be included. This makes sure we don't use LOFR_MARK_SEEN incorrectly (we # can't use it because a tree can be iterated over again at a lower depth). test_expect_success 'tree: where we iterate over tree at two levels' ' git init r5 && mkdir -p r5/a/subdir/b && echo foo > r5/a/subdir/b/foo && From patchwork Sat Jun 1 00:35:59 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971145 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 677F96C5 for ; Sat, 1 Jun 2019 00:36:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 599B528D7B for ; Sat, 1 Jun 2019 00:36:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4D74328D7E; Sat, 1 Jun 2019 00:36:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E90BC28D7B for ; Sat, 1 Jun 2019 00:36:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726988AbfFAAg3 (ORCPT ); Fri, 31 May 2019 20:36:29 -0400 Received: from mail-ot1-f74.google.com ([209.85.210.74]:51845 "EHLO mail-ot1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAg2 (ORCPT ); Fri, 31 May 2019 20:36:28 -0400 Received: by mail-ot1-f74.google.com with SMTP id 68so5330635otu.18 for ; Fri, 31 May 2019 17:36:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=dxOE1UKvycoFri0zbfk/c9+hDBCoXzOEGJxEViG+CT8=; b=MNm7T43PjPJyC28FPkDjXZ4QLreQ3Hpa5qeoNSV95HikVhplkoy2cRFd2xdT9XRNoW TQxKQ6+z6bJ+t2KocvTMPjy8VnZvzWTiqBRj9uFV37V4wY9O3w1epwJS5ZF8Eb0gzM+0 BznYwcChONcjjelOO0HfWktfJOcwovXucdpvFuwamfgzVP3tHlXQnGD7b+NHMwjOdsCX 9kVT655d/QfUOgtxpVXKVlLs8PJLGr7xAd3DDfYDT4xpPfCUEJFWt82mkIoY7YXUNkEc 8YvI2nd02gwt3c+zJXVXnbpITB/QeB1LnKl2FJg9pLrcBgFPSp6oYGTTaVvMotu0wCNF Qbvg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=dxOE1UKvycoFri0zbfk/c9+hDBCoXzOEGJxEViG+CT8=; b=s+vbe+47cIU6zC+NqgF+u0y/u2GwDQxkZgH7C/lNYnsdd7lkwZ1s/wHrTQg0j4zy+K pO9dbWYZGHju4bEiZVwLtbg3pI9fpijLQevs77xut/8JDAGgqQiWGZd4PrDkbQ1n0xYL nWr8ctTHTUWv2LoMPT2T2pqbHhaR+/aGfgprothWcCad+dsh0Ihs87nZqF1uIcgjdCzT 9/QGGd8uzFhRrv3tDMGP0LFU7kVN5XIIxTxTffZyeS7VBcL5KcL3uPmF/6pWZVBsvxDt YrImT1xN08ktj2xKtsaK5hnvo6Rd4mw6EzKeJuzWQfxv55+luWuUiZCihjFuKYlTuQ+S ZUZQ== X-Gm-Message-State: APjAAAXdA6uIgk3kmtu3VYHiMUOYKzK8rXuaCN2R8nWOpPQ1wIzgZpFL 8s+zmvcV/5oHg76MU2eXY2eUC6V7AZA9gO2fCOIkwg1eRIU4nQzE2OP5zwa8hhducQpeRrM/izv 6IGYeDDUX7TZ46jOLu/Q5jplqf1kWk2JaDkSn1I2mJZ9rH2b998GFw/G6bFg= X-Google-Smtp-Source: APXvYqxacP5D0Rk7K7nyzJ4qHYdrtumHxGtWeB3JHQcKQDpaC3jLqyu+Gfi2Nq1F+SjmB6rmZrVasXgXMXW3 X-Received: by 2002:a9d:4041:: with SMTP id o1mr3964315oti.178.1559349387789; Fri, 31 May 2019 17:36:27 -0700 (PDT) Date: Fri, 31 May 2019 17:35:59 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-6-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 5/9] list-objects-filter-options: move error check up From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Move the check that filter_options->choice is set to higher in the call stack. This can only be set when the gentle parse function is called from one of the two call sites. This is important because in an upcoming patch this may or may not be an error, and whether it is an error is only known to the parse_list_objects_filter function. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 0f1d4181cb..e8132b811e 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -27,25 +27,22 @@ static int parse_combine_filter( * expand_list_objects_filter_spec() first). We also "intern" the arg for the * convenience of the current command. */ static int gently_parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg, struct strbuf *errbuf) { const char *v0; - if (filter_options->choice) { - strbuf_addstr( - errbuf, _("multiple filter-specs cannot be combined")); - return 1; - } + if (filter_options->choice) + BUG("filter_options already populated"); if (!strcmp(arg, "blob:none")) { filter_options->choice = LOFC_BLOB_NONE; return 0; } else if (skip_prefix(arg, "blob:limit=", &v0)) { if (git_parse_ulong(v0, &filter_options->blob_limit_value)) { filter_options->choice = LOFC_BLOB_LIMIT; return 0; } @@ -204,20 +201,22 @@ cleanup: list_objects_filter_release(filter_options); memset(filter_options, 0, sizeof(*filter_options)); } return result; } int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; + if (filter_options->choice) + die(_("multiple filter-specs cannot be combined")); filter_options->filter_spec = strdup(arg); if (gently_parse_list_objects_filter(filter_options, arg, &buf)) die("%s", buf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; From patchwork Sat Jun 1 00:36:00 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971147 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6F4FE6C5 for ; Sat, 1 Jun 2019 00:36:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 60B5828D7B for ; Sat, 1 Jun 2019 00:36:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 54C2B28D7E; Sat, 1 Jun 2019 00:36:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BA74228D7B for ; Sat, 1 Jun 2019 00:36:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726994AbfFAAgb (ORCPT ); Fri, 31 May 2019 20:36:31 -0400 Received: from mail-yw1-f73.google.com ([209.85.161.73]:37768 "EHLO mail-yw1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAgb (ORCPT ); Fri, 31 May 2019 20:36:31 -0400 Received: by mail-yw1-f73.google.com with SMTP id j68so10375888ywj.4 for ; Fri, 31 May 2019 17:36:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=4aBa33JqXUmynj90qnc05rPN1ZbGoDsEyb3IfGaHvnk=; b=ey2M/B29ExPpNawGQEGkwI6pVmfuQNh1EwefREvsoNprPYz7LQWZkpTyMWNA8nRNNI 0pb7FnSAeDidAmUtx1kGuRxr531IE/VrEqVcMTr3dolV76dFvcZjqAxJM7iVcw70N54v vPu4U8sWIFJ7WoFRfJWl1/PwRk7irzy6u4RCAUGJRI5t47UYS5DzCjwR6sSWF3Dyol2I aWczWFZ96qkoILf5iDIU3cy8VnFiCspdZNXp5G8uqc5/lG6p54YtrNctxvQ3r270ivqT BtnkqZZH6cNYWAXknfWIJvt5KleAmaVomfvVHO6WrE2hxYXOusq/rC2paIxUWqRojGEa mnIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=4aBa33JqXUmynj90qnc05rPN1ZbGoDsEyb3IfGaHvnk=; b=IpyivLHZqK6yP7pYqHnzg4XdbG2XM3tLyW1mJjCRdpY31QWLE72HgmaZOecPtLQAFN hAkHmWJ1mGepLGcgP56LqAnF2QqA8XMEYSeVPZj314AwBSmMGfbwR5zpNVeWBLI5OEBn 5KC2V4F95ZT0fRJAV9E3R/m5PNZLPhr9NN3TudhvC2mfj+JGu+sf9NmCCUiHnqOvAfTj Q8jzeAJLMfRdrALui9EDfgV9smgc4LCVATAZe5LsRo/Zi0CA0QTAvQp+DI5Mnrvnuhla u+Sp6cOzKyehQer8Fo6PliHYvt+33X/u5KV6Zial0xQeMBtykchMME94U1riHBDppXjr OhAg== X-Gm-Message-State: APjAAAV2wLnGKcsMwyFimFOP0GTsDOnWi2MM6Vz9ykWvnqfxYWEqWhjc mEaaSqamRvFZjRcVsnrY+iRtuEI7VIarVe7Yoz7S86NudNxZb7sRfXQuBBzUktVbx85qHePxi2Y DLtR2zzmJflfmWWPW5OQzXSVUZWJNtL1MeCu9gO1Er03vJ8QGZiucgLtysgc= X-Google-Smtp-Source: APXvYqy8zxUF8i19eM7rtpELy5Qca52pqtky3uace/nIfWrxCikfXGNiCZ+E7Mxe/qEoLIoWkDAt8UOKkT47 X-Received: by 2002:a25:d6:: with SMTP id 205mr6595615yba.180.1559349390315; Fri, 31 May 2019 17:36:30 -0700 (PDT) Date: Fri, 31 May 2019 17:36:00 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-7-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 6/9] list-objects-filter-options: make filter_spec a strbuf From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make the filter_spec string a strbuf rather than a raw C string. A future patch will need to grow this string dynamically. Signed-off-by: Matthew DeVore --- builtin/rev-list.c | 2 +- list-objects-filter-options.c | 16 ++++++++++------ list-objects-filter-options.h | 2 +- upload-pack.c | 2 +- 4 files changed, 13 insertions(+), 9 deletions(-) diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 9f31837d30..7137f13a74 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -460,21 +460,21 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) continue; } if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) { parse_list_objects_filter(&filter_options, arg); if (filter_options.choice && !revs.blob_objects) die(_("object filtering requires --objects")); if (filter_options.choice == LOFC_SPARSE_OID && !filter_options.sparse_oid_value) die(_("invalid sparse value '%s'"), - filter_options.filter_spec); + filter_options.filter_spec.buf); continue; } if (!strcmp(arg, ("--no-" CL_ARG__FILTER))) { list_objects_filter_set_no_filter(&filter_options); continue; } if (!strcmp(arg, "--filter-print-omitted")) { arg_print_omitted = 1; continue; } diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index e8132b811e..5687425847 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -203,21 +203,22 @@ cleanup: } return result; } int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; if (filter_options->choice) die(_("multiple filter-specs cannot be combined")); - filter_options->filter_spec = strdup(arg); + strbuf_init(&filter_options->filter_spec, 0); + strbuf_addstr(&filter_options->filter_spec, arg); if (gently_parse_list_objects_filter(filter_options, arg, &buf)) die("%s", buf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; @@ -226,39 +227,39 @@ int opt_parse_list_objects_filter(const struct option *opt, return 0; } return parse_list_objects_filter(filter_options, arg); } void expand_list_objects_filter_spec( const struct list_objects_filter_options *filter, struct strbuf *expanded_spec) { - strbuf_init(expanded_spec, strlen(filter->filter_spec)); + strbuf_init(expanded_spec, 0); if (filter->choice == LOFC_BLOB_LIMIT) strbuf_addf(expanded_spec, "blob:limit=%lu", filter->blob_limit_value); else if (filter->choice == LOFC_TREE_DEPTH) strbuf_addf(expanded_spec, "tree:%lu", filter->tree_exclude_depth); else - strbuf_addstr(expanded_spec, filter->filter_spec); + strbuf_addstr(expanded_spec, filter->filter_spec.buf); } void list_objects_filter_release( struct list_objects_filter_options *filter_options) { size_t sub; if (!filter_options) return; - free(filter_options->filter_spec); + strbuf_release(&filter_options->filter_spec); free(filter_options->sparse_oid_value); free(filter_options->sparse_path_value); for (sub = 0; sub < filter_options->sub_nr; sub++) list_objects_filter_release(&filter_options->sub[sub]); free(filter_options->sub); memset(filter_options, 0, sizeof(*filter_options)); } void partial_clone_register( const char *remote, @@ -278,32 +279,35 @@ void partial_clone_register( git_config_set("core.repositoryformatversion", "1"); git_config_set("extensions.partialclone", remote); repository_format_partial_clone = xstrdup(remote); /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. */ core_partial_clone_filter_default = - xstrdup(filter_options->filter_spec); + xstrdup(filter_options->filter_spec.buf); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); } void partial_clone_get_default_filter_spec( struct list_objects_filter_options *filter_options) { struct strbuf errbuf = STRBUF_INIT; /* * Parse default value, but silently ignore it if it is invalid. */ if (!core_partial_clone_filter_default) return; - filter_options->filter_spec = strdup(core_partial_clone_filter_default); + if (!filter_options->filter_spec.buf) + strbuf_init(&filter_options->filter_spec, 0); + strbuf_addstr(&filter_options->filter_spec, + core_partial_clone_filter_default); gently_parse_list_objects_filter(filter_options, core_partial_clone_filter_default, &errbuf); strbuf_release(&errbuf); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 8f08ed74a1..e1e23fd191 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -19,21 +19,21 @@ enum list_objects_filter_choice { }; struct list_objects_filter_options { /* * 'filter_spec' is the raw argument value given on the command line * or protocol request. (The part after the "--keyword=".) For * commands that launch filtering sub-processes, or for communication * over the network, don't use this value; use the result of * expand_list_objects_filter_spec() instead. */ - char *filter_spec; + struct strbuf filter_spec; /* * 'choice' is determined by parsing the filter-spec. This indicates * the filtering algorithm to use. */ enum list_objects_filter_choice choice; /* * Choice is LOFC_DISABLED because "--no-filter" was requested. */ diff --git a/upload-pack.c b/upload-pack.c index d2ea5eb20d..2cdd499f28 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -133,21 +133,21 @@ static void create_pack_file(const struct object_array *have_obj, argv_array_push(&pack_objects.args, "--stdout"); if (shallow_nr) argv_array_push(&pack_objects.args, "--shallow"); if (!no_progress) argv_array_push(&pack_objects.args, "--progress"); if (use_ofs_delta) argv_array_push(&pack_objects.args, "--delta-base-offset"); if (use_include_tag) argv_array_push(&pack_objects.args, "--include-tag"); - if (filter_options.filter_spec) { + if (filter_options.filter_spec.len) { struct strbuf expanded_filter_spec = STRBUF_INIT; expand_list_objects_filter_spec(&filter_options, &expanded_filter_spec); if (pack_objects.use_shell) { struct strbuf buf = STRBUF_INIT; sq_quote_buf(&buf, expanded_filter_spec.buf); argv_array_pushf(&pack_objects.args, "--filter=%s", buf.buf); strbuf_release(&buf); } else { argv_array_pushf(&pack_objects.args, "--filter=%s", From patchwork Sat Jun 1 00:36:01 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971149 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0B67C13AD for ; Sat, 1 Jun 2019 00:36:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EE83A28D7B for ; Sat, 1 Jun 2019 00:36:36 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E1B3428D7E; Sat, 1 Jun 2019 00:36:36 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B97C128D7B for ; Sat, 1 Jun 2019 00:36:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727003AbfFAAge (ORCPT ); Fri, 31 May 2019 20:36:34 -0400 Received: from mail-pg1-f202.google.com ([209.85.215.202]:41004 "EHLO mail-pg1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAge (ORCPT ); Fri, 31 May 2019 20:36:34 -0400 Received: by mail-pg1-f202.google.com with SMTP id d7so5808623pgc.8 for ; Fri, 31 May 2019 17:36:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=BMPcOqBP2FWjY9Ze5KWmFGtgnrtud6lGYenFifssv5U=; b=czUPca0SPE2y1c781YycjckqJkVVdXYhd+Top1G3czbLMZzAs+mfL73ZdcV+QOfTnf oQVIibZFXblavyPf73eGXPI7nEckw7dN0RcQAYcDhC7ClsghhhsVbEhC2QbYIJNlQ7lQ Q5O7vYxBW9r8kJCk4Z0NXcj8/y3MX+mH92TOrR80knRWfOsjRhXrSYvpq9nRE33tsqQx 2OIrMcBtwGVijLOifOo7lnJGw4KjZXWL59KyefuZZtjeJWFW15nMi8DLEKE7pEKsHcAi Mzr+nrbccsZKj5FKJNSWTV3rCtEyNyCsHKHnOa9w9YVJo9Z/i5qGBJ1I17v2/4ZMKn5X 4Rjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BMPcOqBP2FWjY9Ze5KWmFGtgnrtud6lGYenFifssv5U=; b=f3N+OuW+Jkbkt2iMHxLnyas18l/pO2aUFZ2tTsr1Y6m/fta/ZTyJjDPZfkqMAXkk/l kiIGIKS4oLEhpAiIZjSjrZ0l8gXckjzijfD/iiOD/cLrOqldhSTlO10QpkKUqFdo4nmJ 70mVHIyU+qoYqR5cptbyo5oJFBgv8EcRgu6dQkGwwKAIGXjqul73IMTUPgxGJvitM7+S 2Fx3Kk3GLIYvFAYL6UXks8zi1blu8/X1HgjLLqP4pDrVmQCBSme7JIa0nogVLITel31T 6QE2rEuWPRzxzF9sNwMYdyIHZgUG58xyOWg0Icvd1e5udfNEx+bUqBXYS8b7cd6JzpG0 WtRg== X-Gm-Message-State: APjAAAVcdt9VazrxMKau5sr1Ynj8ILZOJj2FW416PNNT0+SfzZE6eNcf jhlkDDV2OgpLsquFJ1MxA1vybATub2BuySsUKEtFbIml7Otz+eHEEUT4inXZpCWSnB20Oh6+fBm dSd0fqp/1pooNBDkkT1WZtGKb59BP/j5Jd2NNzTLNwhVvZMQvy1H3C3VNwIg= X-Google-Smtp-Source: APXvYqzleiFjrFmoHAcIX10Rz9F9MiKmIzsJTu/1SL+bekk2rcqUM40kXsYYKrg70xIMc54tmZ/3mSLzD4fg X-Received: by 2002:a63:8449:: with SMTP id k70mr12798812pgd.208.1559349393129; Fri, 31 May 2019 17:36:33 -0700 (PDT) Date: Fri, 31 May 2019 17:36:01 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-8-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 7/9] list-objects-filter-options: allow mult. --filter From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Jeff Hostetler , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allow combining of multiple filters by simply repeating the --filter flag. Before this patch, the user had to combine them in a single flag somewhat awkwardly (e.g. --filter=combine:FOO+BAR), including URL-encoding the individual filters. To make this work, in the --filter flag parsing callback, rather than error out when we detect that the filter_options struct is already populated, we modify it in-place to contain the added sub-filter. The existing sub-filter becomes the lhs of the combined filter, and the next sub-filter becomes the rhs. We also have to URL-encode the LHS and RHS sub-filters. We can simplify the operation if the LHS is already a combine: filter. In that case, we just append the URL-encoded RHS sub-filter to the LHS spec to get the new spec. Helped-by: Emily Shaffer Helped-by: Jeff Hostetler Helped-by: Junio C Hamano Signed-off-by: Matthew DeVore --- Documentation/rev-list-options.txt | 16 +++++ list-objects-filter-options.c | 90 ++++++++++++++++++++++++++--- list-objects-filter-options.h | 11 ++++ t/t5616-partial-clone.sh | 19 ++++++ t/t6112-rev-list-filters-objects.sh | 44 ++++++++++++-- transport.c | 1 + upload-pack.c | 2 + 7 files changed, 171 insertions(+), 12 deletions(-) diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index ddbc1de43f..7b4116f279 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -730,20 +730,36 @@ specification contained in . + The form '--filter=tree:' omits all blobs and trees whose depth from the root tree is >= (minimum depth if an object is located at multiple depths in the commits traversed). =0 will not include any trees or blobs unless included explicitly in the command-line (or standard input when --stdin is used). =1 will include only the tree and blobs which are referenced directly by a commit reachable from or an explicitly-given object. =2 is like =1 while also including trees and blobs one more level removed from an explicitly-given commit or tree. ++ +Multiple '--filter=' flags can be specified to combine filters. Only +objects which are accepted by every filter are included. ++ +The form '--filter=combine:++...' can also be +used to combined several filters, but this is harder than just repeating +the '--filter' flag and is usually not necessary. Filters are joined by +'{plus}' and individual filters are %-encoded (i.e. URL-encoded). +Besides the '{plus}' and '%' characters, the following characters are +reserved and also must be encoded: `~!@#$^&*()[]{}\;",<>?`+'`+ +as well as all characters with ASCII code <= `0x20`, which includes +space and newline. ++ +Other arbitrary characters can also be encoded. For instance, +'combine:tree:3+blob:none' and 'combine:tree%3A3+blob%3Anone' are +equivalent. --no-filter:: Turn off any previous `--filter=` argument. --filter-print-omitted:: Only useful with `--filter=`; prints a list of the objects omitted by the filter. Object IDs are prefixed with a ``~'' character. --missing=:: A debug option to help with future "partial clone" development. diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 5687425847..5e98e4a309 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -1,18 +1,19 @@ #include "cache.h" #include "commit.h" #include "config.h" #include "revision.h" #include "argv-array.h" #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "trace.h" static int parse_combine_filter( struct list_objects_filter_options *filter_options, const char *arg, struct strbuf *errbuf); /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: * --filter= @@ -197,30 +198,105 @@ static int parse_combine_filter( cleanup: strbuf_list_free(subspecs); if (result) { list_objects_filter_release(filter_options); memset(filter_options, 0, sizeof(*filter_options)); } return result; } -int parse_list_objects_filter(struct list_objects_filter_options *filter_options, - const char *arg) +static void add_url_encoded(struct strbuf *dest, const char *s) +{ + while (*s) { + if (*s <= ' ' || strchr(RESERVED_NON_WS, *s) || + *s == '%' || *s == '+') + strbuf_addf(dest, "%%%02X", (int)*s); + else + strbuf_addf(dest, "%c", *s); + s++; + } +} + +/* + * Changes filter_options into an equivalent LOFC_COMBINE filter options + * instance. Does not do anything if filter_options is already LOFC_COMBINE. + */ +static void transform_to_combine_type( + struct list_objects_filter_options *filter_options) +{ + assert(filter_options->choice); + if (filter_options->choice == LOFC_COMBINE) + return; + { + const int initial_sub_alloc = 2; + struct list_objects_filter_options *sub_array = + xcalloc(initial_sub_alloc, sizeof(*sub_array)); + sub_array[0] = *filter_options; + memset(filter_options, 0, sizeof(*filter_options)); + filter_options->sub = sub_array; + filter_options->sub_alloc = initial_sub_alloc; + } + filter_options->sub_nr = 1; + filter_options->choice = LOFC_COMBINE; + strbuf_init(&filter_options->filter_spec, 0); + strbuf_addstr(&filter_options->filter_spec, "combine:"); + add_url_encoded(&filter_options->filter_spec, + filter_options->sub[0].filter_spec.buf); + /* + * We don't need the filter_spec strings for subfilter specs, only the + * top level. + */ + strbuf_release(&filter_options->sub[0].filter_spec); +} + +void list_objects_filter_die_if_populated( + struct list_objects_filter_options *filter_options) { - struct strbuf buf = STRBUF_INIT; if (filter_options->choice) die(_("multiple filter-specs cannot be combined")); - strbuf_init(&filter_options->filter_spec, 0); - strbuf_addstr(&filter_options->filter_spec, arg); - if (gently_parse_list_objects_filter(filter_options, arg, &buf)) - die("%s", buf.buf); +} + +int parse_list_objects_filter( + struct list_objects_filter_options *filter_options, + const char *arg) +{ + struct strbuf errbuf = STRBUF_INIT; + int parse_error; + + if (!filter_options->choice) { + strbuf_init(&filter_options->filter_spec, 0); + strbuf_addstr(&filter_options->filter_spec, arg); + + parse_error = gently_parse_list_objects_filter( + filter_options, arg, &errbuf); + } else { + /* + * Make filter_options an LOFC_COMBINE spec so we can trivially + * add subspecs to it. + */ + transform_to_combine_type(filter_options); + + strbuf_addstr(&filter_options->filter_spec, "+"); + add_url_encoded(&filter_options->filter_spec, arg); + trace_printf("Generated composite filter-spec: %s\n", + filter_options->filter_spec.buf); + ALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1, + filter_options->sub_alloc); + filter_options = &filter_options->sub[filter_options->sub_nr++]; + memset(filter_options, 0, sizeof(*filter_options)); + + parse_error = gently_parse_list_objects_filter( + filter_options, arg, &errbuf); + } + if (parse_error) + die("%s", errbuf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; if (unset || !arg) { list_objects_filter_set_no_filter(filter_options); diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index e1e23fd191..f8c8a624e4 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -56,20 +56,31 @@ struct list_objects_filter_options { struct list_objects_filter_options *sub; /* * END choice-specific parsed values. */ }; /* Normalized command line arguments */ #define CL_ARG__FILTER "filter" +void list_objects_filter_die_if_populated( + struct list_objects_filter_options *filter_options); + +/* + * Parses the filter spec string given by arg and either (1) simply places the + * result in filter_options if it is not yet populated or (2) combines it with + * the filter already in filter_options if it is already populated. In the case + * of (2), the filter specs are combined as if specified with 'combine:'. + * + * Dies and prints a user-facing message if an error occurs. + */ int parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg); int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset); #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \ { OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_("args"), \ N_("object filtering"), 0, \ diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index 9a8f9886b3..11536f4028 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -201,20 +201,39 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr test_line_count = 70 fetched_objects && awk -f print_1.awk fetched_objects | xargs -n1 git -C dst cat-file -t >fetched_types && sort -u fetched_types >unique_types.observed && test_write_lines blob commit tree >unique_types.expected && test_cmp unique_types.expected unique_types.observed ' +test_expect_success 'implicitly construct combine: filter with repeated flags' ' + GIT_TRACE=$(pwd)/trace git clone --bare \ + --filter=blob:none --filter=tree:1 \ + "file://$(pwd)/srv.bare" pc2 && + grep "trace:.* git pack-objects .*--filter=combine:blob:none+tree:1" \ + trace && + git -C pc2 rev-list --objects --missing=allow-any HEAD >objects && + + # We should have gotten some root trees. + grep " $" objects && + # Should not have gotten any non-root trees or blobs. + ! grep " ." objects && + + xargs -n 1 git -C pc2 cat-file -t types && + sort -u types >unique_types.actual && + test_write_lines commit tree >unique_types.expected && + test_cmp unique_types.expected unique_types.actual +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && test_commit -C src x && test_config -C src uploadpack.allowfilter 1 && test_config -C src uploadpack.allowanysha1inwant 1 && # Create a tag pointing to a blob. BLOB=$(echo blob-contents | git -C src hash-object --stdin -w) && git -C src tag myblob "$BLOB" && diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index c36199457d..7fb5e50cde 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -357,21 +357,30 @@ test_expect_success 'verify tree:3 includes everything expected' ' test_expect_success 'combine:... for a simple combination' ' git -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \ >actual && expect_has HEAD "" && expect_has HEAD~1 "" && expect_has HEAD dir1 && # There are also 2 commit objects - test_line_count = 5 actual + test_line_count = 5 actual && + + cp actual expected && + + # Try again using repeated --filter - this is equivalent to a manual + # combine with "combine:...+..." + git -C r3 rev-list --objects --filter=combine:tree:2 \ + --filter=blob:none HEAD >actual && + + test_cmp expected actual ' test_expect_success 'combine:... with URL encoding' ' git -C r3 rev-list --objects \ --filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual && expect_has HEAD "" && expect_has HEAD~1 "" && expect_has HEAD dir1 && @@ -435,24 +444,26 @@ test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' ' git -C r3 rev-list --objects --filter="combine:tree%3A2+blob%3anone" \ HEAD >actual && test_line_count = 5 actual && git -C r3 rev-list --objects --filter="combine:tree:%30" HEAD >actual && test_line_count = 2 actual && git -C r3 rev-list --objects --filter="combine:tree:%39+blob:none" \ HEAD >actual && test_line_count = 5 actual ' -test_expect_success 'add a sparse pattern blob whose path has reserved chars' ' +test_expect_success 'add sparse pattern blobs whose paths have reserved chars' ' cp r3/pattern r3/pattern1+renamed% && - git -C r3 add pattern1+renamed% && - git -C r3 commit -m "add sparse pattern file with reserved chars" + cp r3/pattern "r3/p;at%ter+n" && + cp r3/pattern r3/^~pattern && + git -C r3 add pattern1+renamed% "p;at%ter+n" ^~pattern && + git -C r3 commit -m "add sparse pattern files with reserved chars" ' test_expect_success 'combine:... with more than two sub-filters' ' git -C r3 rev-list --objects \ --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \ HEAD >actual && expect_has HEAD "" && expect_has HEAD~1 "" && expect_has HEAD~2 "" && @@ -463,21 +474,44 @@ test_expect_success 'combine:... with more than two sub-filters' ' # Should also have 3 commits test_line_count = 9 actual && # Try again, this time making sure the last sub-filter is only # URL-decoded once. cp actual expect && git -C r3 rev-list --objects \ --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \ HEAD >actual && - test_cmp expect actual + test_cmp expect actual && + + # Use the same composite filter again, but with a pattern file name that + # requires encoding multiple characters, and use implicit filter + # combining. + GIT_TRACE=$(pwd)/trace git -C r3 rev-list --objects \ + --filter=tree:3 --filter=blob:limit=40 \ + --filter=sparse:oid="master:p;at%ter+n" \ + HEAD >actual && + + test_cmp expect actual && + grep "Generated composite filter-spec: combine:tree:3+blob:limit=40+sparse:oid=master:p%3Bat%25ter%2B" \ + trace && + + # Repeat the above test, but this time, the characters to encode are in + # the LHS of the combined filter. + GIT_TRACE=$(pwd)/trace git -C r3 rev-list --objects \ + --filter=sparse:oid=master:^~pattern \ + --filter=tree:3 --filter=blob:limit=40 \ + HEAD >actual && + + test_cmp expect actual && + grep "Generated composite filter-spec: combine:sparse:oid=master:%5E%7Epattern+tree:3+blob:limit=40" \ + trace ' # Test provisional omit collection logic with a repo that has objects appearing # at multiple depths - first deeper than the filter's threshold, then shallow. test_expect_success 'setup r4' ' git init r4 && echo foo > r4/foo && mkdir r4/subdir && diff --git a/transport.c b/transport.c index f1fcd2c4b0..ee7dd1c062 100644 --- a/transport.c +++ b/transport.c @@ -217,20 +217,21 @@ static int set_git_option(struct git_transport_options *opts, } else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) { opts->deepen_relative = !!value; return 0; } else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) { opts->from_promisor = !!value; return 0; } else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) { opts->no_dependents = !!value; return 0; } else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) { + list_objects_filter_die_if_populated(&opts->filter_options); parse_list_objects_filter(&opts->filter_options, value); return 0; } return 1; } static int connect_setup(struct transport *transport, int for_push) { struct git_transport_data *data = transport->data; int flags = transport->verbose > 0 ? CONNECT_VERBOSE : 0; diff --git a/upload-pack.c b/upload-pack.c index 2cdd499f28..16e748ba58 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -877,20 +877,21 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan if (process_deepen(reader->line, &depth)) continue; if (process_deepen_since(reader->line, &deepen_since, &deepen_rev_list)) continue; if (process_deepen_not(reader->line, &deepen_not, &deepen_rev_list)) continue; if (skip_prefix(reader->line, "filter ", &arg)) { if (!filter_capability_requested) die("git upload-pack: filtering capability not negotiated"); + list_objects_filter_die_if_populated(&filter_options); parse_list_objects_filter(&filter_options, arg); continue; } if (!skip_prefix(reader->line, "want ", &arg) || parse_oid_hex(arg, &oid_buf, &features)) die("git upload-pack: protocol error, " "expected to get object ID, not '%s'", reader->line); if (parse_feature_request(features, "deepen-relative")) @@ -1296,20 +1297,21 @@ static void process_args(struct packet_reader *request, continue; if (process_deepen_not(arg, &data->deepen_not, &data->deepen_rev_list)) continue; if (!strcmp(arg, "deepen-relative")) { data->deepen_relative = 1; continue; } if (allow_filter && skip_prefix(arg, "filter ", &p)) { + list_objects_filter_die_if_populated(&filter_options); parse_list_objects_filter(&filter_options, p); continue; } if ((git_env_bool("GIT_TEST_SIDEBAND_ALL", 0) || allow_sideband_all) && !strcmp(arg, "sideband-all")) { data->writer.use_sideband = 1; continue; } From patchwork Sat Jun 1 00:36:02 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971151 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CF68213AD for ; Sat, 1 Jun 2019 00:36:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C164B28D7B for ; Sat, 1 Jun 2019 00:36:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B568A28D7E; Sat, 1 Jun 2019 00:36:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 447FB28D7B for ; Sat, 1 Jun 2019 00:36:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727026AbfFAAgh (ORCPT ); Fri, 31 May 2019 20:36:37 -0400 Received: from mail-qt1-f202.google.com ([209.85.160.202]:40469 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727012AbfFAAgh (ORCPT ); Fri, 31 May 2019 20:36:37 -0400 Received: by mail-qt1-f202.google.com with SMTP id 37so2081881qtc.7 for ; Fri, 31 May 2019 17:36:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=yqEWilff9PfEDxnWwsAsf5xmQ2hB/9aS1CJfqlFT+BM=; b=ZT5F9EHOtZSzNdfAkZiB585TIXy8xHTBnTnQeHDA4vsrWS9ro4uHvpvOAEgtI0Ie5L Fb+YHxtBblZWO146riTaTpy14IGQAh7diaK/GwnEqYGlO0FKq0XNxO3gqYBtE1OHOwRM 9xHAYJEJQSFnOjfYpe9lMLGbNIZl/tMHB53SCTFE52r589+1RcKW219TPZPbx+UZ06pY BkTvfGZ4OEhtp1ebX/9cPslF9829/O6bt0iC2WYJj5whLYOxdm0Gcw/4u+jpU32mKTgQ NRUjZ6M06Hm0vWc4Vpy0IDPh1Bw//UFvRIeppiI+hBTfHBIS1YtQLR0XVEdyNNvD27xW ZhsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=yqEWilff9PfEDxnWwsAsf5xmQ2hB/9aS1CJfqlFT+BM=; b=rrkSOg1nxTHZ6YDduHh9CI0mrp2PqboW52YlYb1bGEhF3VBg/zxGE2ce1TKmK/OVP7 abYb1iYVgsySjrhYS0vKSqyvi91Czd1YVnBwHyrMEA8mP2vk0Ij3232kDZNNjdAEa+mZ 2Rti+w8bjdGAy6qH3Owxl5nyBYA3gPohYaI2KNqtNcIOL7DpCQno3G1q6nrme2VSRu+k N7ADHmh656tQNzt3G4yLanxUooAapuUM5KGbeOkb8FEEJGwckhAfzHYQGFs2ow5OsVhX XCJMHAzrcmLPJmZKNHgk71o94oMm0T3ZaX+3gN9uH/553WMHeXdyHNtefCuvEaB6E25E d9zA== X-Gm-Message-State: APjAAAWwy8W4y3T6YpNE9A6jnl0PWDLflw9iC07yiqUNrhd9legF1fKu ke52t5ZcbCgGxgwukPrZEcO9pR5U6B7H1FmfZv6QL/8MkA4bGn03lh+iUOlJEOgpFTUHmTLrord YhqauavNebZCm5EdBrDYz2roc+yzJqn6nv9dyQ5tu2tzslAw4m5vp8oArKlc= X-Google-Smtp-Source: APXvYqzjpwDYj51paBVkKjnEph+18kggbIr78pcuogrrFhoqi5GLVeyWcSx/Pp/mreSz1U1PLRTF6uCtbf0q X-Received: by 2002:a0c:9dc4:: with SMTP id p4mr11784434qvf.69.1559349395813; Fri, 31 May 2019 17:36:35 -0700 (PDT) Date: Fri, 31 May 2019 17:36:02 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-9-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 8/9] list-objects-filter-options: clean up use of ALLOC_GROW From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce a new macro ALLOC_GROW_BY which automatically zeros the added array elements and takes care of updating the nr value. Use the macro in code introduced earlier in this patchset. Signed-off-by: Matthew DeVore --- cache.h | 22 ++++++++++++++++++++++ list-objects-filter-options.c | 17 +++++++---------- 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/cache.h b/cache.h index fa8ede9a2d..847fbdeff0 100644 --- a/cache.h +++ b/cache.h @@ -652,33 +652,55 @@ int init_db(const char *git_dir, const char *real_git_dir, void sanitize_stdfds(void); int daemonize(void); #define alloc_nr(x) (((x)+16)*3/2) /* * Realloc the buffer pointed at by variable 'x' so that it can hold * at least 'nr' entries; the number of entries currently allocated * is 'alloc', using the standard growing factor alloc_nr() macro. * + * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some + * added niceties. + * * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'. */ #define ALLOC_GROW(x, nr, alloc) \ do { \ if ((nr) > alloc) { \ if (alloc_nr(alloc) < (nr)) \ alloc = (nr); \ else \ alloc = alloc_nr(alloc); \ REALLOC_ARRAY(x, alloc); \ } \ } while (0) +/* + * Similar to ALLOC_GROW but handles updating of the nr value and + * zeroing the bytes of the newly-grown array elements. + * + * DO NOT USE any expression with side-effect for any of the + * arguments. + */ +#define ALLOC_GROW_BY(x, nr, increase, alloc) \ + do { \ + if (increase) { \ + size_t new_nr = nr + (increase); \ + if (new_nr < nr) \ + BUG("negative growth in ALLOC_GROW_BY"); \ + ALLOC_GROW(x, new_nr, alloc); \ + memset((x) + nr, 0, sizeof(*(x)) * (increase)); \ + nr = new_nr; \ + } \ + } while (0) + /* Initialize and use the cache information */ struct lock_file; void preload_index(struct index_state *index, const struct pathspec *pathspec, unsigned int refresh_flags); int do_read_index(struct index_state *istate, const char *path, int must_exist); /* for testting only! */ int read_index_from(struct index_state *, const char *path, const char *gitdir); int is_index_unborn(struct index_state *); diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 5e98e4a309..d8abe6cfcf 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -142,26 +142,24 @@ static int has_reserved_character( } return 0; } static int parse_combine_subfilter( struct list_objects_filter_options *filter_options, struct strbuf *subspec, struct strbuf *errbuf) { - size_t new_index = filter_options->sub_nr++; + size_t new_index = filter_options->sub_nr; - ALLOC_GROW(filter_options->sub, filter_options->sub_nr, - filter_options->sub_alloc); - memset(&filter_options->sub[new_index], 0, - sizeof(*filter_options->sub)); + ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1, + filter_options->sub_alloc); return has_reserved_character(subspec, errbuf) || url_decode(subspec, errbuf) || gently_parse_list_objects_filter( &filter_options->sub[new_index], subspec->buf, errbuf); } static int parse_combine_filter( struct list_objects_filter_options *filter_options, const char *arg, @@ -273,27 +271,26 @@ int parse_list_objects_filter( /* * Make filter_options an LOFC_COMBINE spec so we can trivially * add subspecs to it. */ transform_to_combine_type(filter_options); strbuf_addstr(&filter_options->filter_spec, "+"); add_url_encoded(&filter_options->filter_spec, arg); trace_printf("Generated composite filter-spec: %s\n", filter_options->filter_spec.buf); - ALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1, - filter_options->sub_alloc); - filter_options = &filter_options->sub[filter_options->sub_nr++]; - memset(filter_options, 0, sizeof(*filter_options)); + ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1, + filter_options->sub_alloc); parse_error = gently_parse_list_objects_filter( - filter_options, arg, &errbuf); + &filter_options->sub[filter_options->sub_nr - 1], arg, + &errbuf); } if (parse_error) die("%s", errbuf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; From patchwork Sat Jun 1 00:36:03 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10971153 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 798946C5 for ; Sat, 1 Jun 2019 00:36:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6BB1A28D7B for ; Sat, 1 Jun 2019 00:36:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5F8F228D7E; Sat, 1 Jun 2019 00:36:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3B3A028D7B for ; Sat, 1 Jun 2019 00:36:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727012AbfFAAgj (ORCPT ); Fri, 31 May 2019 20:36:39 -0400 Received: from mail-ot1-f74.google.com ([209.85.210.74]:41501 "EHLO mail-ot1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726518AbfFAAgi (ORCPT ); Fri, 31 May 2019 20:36:38 -0400 Received: by mail-ot1-f74.google.com with SMTP id z1so5342570oth.8 for ; Fri, 31 May 2019 17:36:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=qiiN5fs489IzftRbt2WCr2seLgQznoLRsCHpc6K/kbQ=; b=YuJtg2dpZn9NphL5+k7iGUgTOuRAaicMX7A3WvvEpwl3ykedgWUfwZ+sMmr701us/1 5TwHvko1vJZSMXDqIrt6N0wDhTNTJ5Nt9YBi4x4b8IaJmuswqLS4YT2ROyxPy+Kz5+Cr yICtD0kwa2dAcacyYj19L8Yq5bpJCzIslKdYK2/cvBgRNgSGDv8jbM4AKrCn/5PvuVMt 8Vw4LqFce3XJ4Rf84vw18gdhUmHQyqqcQNp4YiCll0Iyr9/OYkQ9bWKXuRme9r2ynjBj 3/+gbzCA1PvHGM3oj6ZQuiMt2rQFM3HZFATxmeI5zCE+vkXhXPUKfRQrGI5IAGM+mGhz dzvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=qiiN5fs489IzftRbt2WCr2seLgQznoLRsCHpc6K/kbQ=; b=A6Y+dXfTUEzedpjHIFYXajDORVL3JDHU5Uspo45MrTcZ7Nnq0lwHyJ/ZnpP0m9UYMj a7mquMkNk/Kr3hT0NS9FH/NnioQGTMwZV1oqbNB84jk9ZtkewKzq/mF3wNnlM2L9NT8b 2eYxYACVlUW1CtPgzyFAUVgR7ClKauexZzaKgAO6Q6O5sVUC6FXeJkDXvAwHujDeAC5Q hfD2CEif63nJY1spIyI6HqCBWqoWYBdRZmHsqZ6RwISLPLXie3ilt+LapYNsxgnIwx+s ww6nB9fHMCicHl5m044qS5LtFMPx2k+I1aOt3VH46DPZHgl7k3giH2FHOmoRQTWasN6Y 6Rtg== X-Gm-Message-State: APjAAAUcMvXHPL7x9V1v62BfOPwa4HSIEaIfDPoU2HhYONCSyGN3yb9O CdcXrJyTjOFh+DjI2/NeGI0lLq+PdAROTl0OveggS3eXHk+ctKyy12u2x6P/BEAUI57prPzv6h7 ylUNn069ws/rDSBaCdLyt/y5osC0nhdmIT9cbMtDdRGd1OjFWVepHQcoTDx4= X-Google-Smtp-Source: APXvYqxJ7/WTb1MRdwkhycyZ57q6zdM7Tic4yjcsJ2xBY55EfJ5XjyoKReXpA0xhNVqOBX2CSThVDbAeslik X-Received: by 2002:aca:5209:: with SMTP id g9mr91242oib.35.1559349398139; Fri, 31 May 2019 17:36:38 -0700 (PDT) Date: Fri, 31 May 2019 17:36:03 -0700 In-Reply-To: <20190601003603.90794-1-matvore@google.com> Message-Id: <20190601003603.90794-10-matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.rc1.311.g5d7573a151-goog Subject: [PATCH v2 9/9] list-objects-filter-options: make parser void From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This function always returns 0, so make it return void instead. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 12 +++++------- list-objects-filter-options.h | 2 +- 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index d8abe6cfcf..ed02c88eb6 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -247,21 +247,21 @@ static void transform_to_combine_type( strbuf_release(&filter_options->sub[0].filter_spec); } void list_objects_filter_die_if_populated( struct list_objects_filter_options *filter_options) { if (filter_options->choice) die(_("multiple filter-specs cannot be combined")); } -int parse_list_objects_filter( +void parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf errbuf = STRBUF_INIT; int parse_error; if (!filter_options->choice) { strbuf_init(&filter_options->filter_spec, 0); strbuf_addstr(&filter_options->filter_spec, arg); @@ -280,34 +280,32 @@ int parse_list_objects_filter( filter_options->filter_spec.buf); ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1, filter_options->sub_alloc); parse_error = gently_parse_list_objects_filter( &filter_options->sub[filter_options->sub_nr - 1], arg, &errbuf); } if (parse_error) die("%s", errbuf.buf); - return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; - if (unset || !arg) { + if (unset || !arg) list_objects_filter_set_no_filter(filter_options); - return 0; - } - - return parse_list_objects_filter(filter_options, arg); + else + parse_list_objects_filter(filter_options, arg); + return 0; } void expand_list_objects_filter_spec( const struct list_objects_filter_options *filter, struct strbuf *expanded_spec) { strbuf_init(expanded_spec, 0); if (filter->choice == LOFC_BLOB_LIMIT) strbuf_addf(expanded_spec, "blob:limit=%lu", filter->blob_limit_value); diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index f8c8a624e4..2c0ce6383a 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -67,21 +67,21 @@ void list_objects_filter_die_if_populated( struct list_objects_filter_options *filter_options); /* * Parses the filter spec string given by arg and either (1) simply places the * result in filter_options if it is not yet populated or (2) combines it with * the filter already in filter_options if it is already populated. In the case * of (2), the filter specs are combined as if specified with 'combine:'. * * Dies and prints a user-facing message if an error occurs. */ -int parse_list_objects_filter( +void parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg); int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset); #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \ { OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_("args"), \ N_("object filtering"), 0, \ opt_parse_list_objects_filter }