From patchwork Thu Jun 27 22:54:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020799 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 21DAF112C for ; Thu, 27 Jun 2019 22:54:29 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 190D828708 for ; Thu, 27 Jun 2019 22:54:29 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 0D73228738; Thu, 27 Jun 2019 22:54:29 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CDFC528708 for ; Thu, 27 Jun 2019 22:54:27 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726640AbfF0Wy1 (ORCPT ); Thu, 27 Jun 2019 18:54:27 -0400 Received: from mail-ua1-f74.google.com ([209.85.222.74]:48577 "EHLO mail-ua1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726550AbfF0Wy0 (ORCPT ); Thu, 27 Jun 2019 18:54:26 -0400 Received: by mail-ua1-f74.google.com with SMTP id s14so613459uap.15 for ; Thu, 27 Jun 2019 15:54:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=9HktLtxBfH3lsYSwks8R6VbVlmRoz5arUYbn8F64Ppg=; b=hsbsHJH65Rv84s3NWefN8pv40kntR367ZUN0Dh2yatjBEpwGqw8p0/sUnrVoCbDUje hI6cJ8ZguAtoje06Ziut3UZ7XpqoI08GjcocE4JDBWDMEEhnwlfq4ig6TTLL0lTzek1h fep+iF144j8sZZTfOdViK1J6RwbLRDMnzDTHtDfypkMQDpuofHFNCU9yGzf5G7W/6gyY OMIBntI/1FAfkn1mI5Wdl3LWXgr9D1fLO3vhsSzWxVUPM9s3veKe+frDXyFPw9Yic3bI mOrdvtoxQWTZMYf6KSGKHVKVmIt30EKKenab4CkRnIMuoQnaB+MuNlrgF8Vgxwq+07SU p6LA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=9HktLtxBfH3lsYSwks8R6VbVlmRoz5arUYbn8F64Ppg=; b=NK4lpy/QyXPPVqOmHaFDTi3bdWOX00GwTv+NHyTh3LAsZjU+2xSYoIod1bmCi8KkSw 53vW47Bfi/M2HldFiMj7hMM1gkL0MJ1O47D3rVh3s+6vc/zVkLR7F9ScRn6ltuCOoP93 1sr0yavKsUp5cYRDUO8TjE1u83uix7Gg2z+CeS4eMMZQgm1QAicG+5Xs1mdFE+jnO/4R bq8hdoISoO4lC5bpeYwf9Fzc1l0Zo72j3f5q7VfRY6u+fD8tI4Xs4aPLsPvh1KAgNQgH x6fVXV9PWoAkUGVzeXsdQc8RSzDJ4YpH1BwFadAU4ALU4cZ78iBkXaDQr3t43T66teAz zqdg== X-Gm-Message-State: APjAAAU7sBxmca/V04aHvsI1wbZLlX7MncrDvLtEi9K3FQUY3cRpMhvi ZkI/eCgRnX3XjjBInk6EK1fpburU39tLU5askfTP8bKqaVVu07vIFFIxfjSrcGINbUAiJeJ8TlL eTGU/PFSBrmbXCf7SzXMIm8DMUgwuaPyCt/W53VQxIc1USElUseTXKu3OevI= X-Google-Smtp-Source: APXvYqxjacqHAucnW/tpFqcY4GITXlOPmGfdx7FxAZ7kvz5HuS7+9GVEp6bvjzvrdevjT7KgXvcV+D2sMGT9 X-Received: by 2002:ab0:7782:: with SMTP id x2mr4007164uar.140.1561676064382; Thu, 27 Jun 2019 15:54:24 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:05 -0700 In-Reply-To: Message-Id: <4fe4baf34506a956a53688b7a2d2d6665878cdf6.1561675151.git.matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 01/10] list-objects-filter: encapsulate filter components From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de, Jeff Hostetler , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Encapsulate filter_fn, filter_free_fn, and filter_data into their own opaque struct. Due to opaqueness, filter_fn and filter_free_fn can no longer be accessed directly by users. Currently, all usages of filter_fn are guarded by a necessary check: (obj->flags & NOT_USER_GIVEN) && filter_fn Take the opportunity to include this check into the new function list_objects_filter__filter_object(), so that we no longer need to write this check at every caller of the filter function. Also, the init functions in list-objects-filter.c no longer need to confusingly return the filter constituents in various places (filter_fn and filter_free_fn as out parameters, and filter_data as the function's return value); they can just initialize the "struct filter" passed in. Helped-by: Jeff Hostetler Helped-by: Jonathan Tan Helped-by: Junio C Hamano Signed-off-by: Matthew DeVore --- list-objects-filter.c | 112 ++++++++++++++++++++++++++++-------------- list-objects-filter.h | 35 ++++++------- list-objects.c | 55 +++++++++------------ 3 files changed, 113 insertions(+), 89 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index 36e1f774bc..e06b82def0 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -19,20 +19,34 @@ * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects * that have been shown, but should be revisited if they appear * in the traversal (until we mark it SEEN). This is a way to * let us silently de-dup calls to show() in the caller. This * is subtly different from the "revision.h:SHOWN" and the * "sha1-name.c:ONELINE_SEEN" bits. And also different from * the non-de-dup usage in pack-bitmap.c */ #define FILTER_SHOWN_BUT_REVISIT (1<<21) +struct filter { + enum list_objects_filter_result (*filter_object_fn)( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + void *filter_data); + + void (*free_fn)(void *filter_data); + + void *filter_data; +}; + /* * A filter for list-objects to omit ALL blobs from the traversal. * And to OPTIONALLY collect a list of the omitted OIDs. */ struct filter_blobs_none_data { struct oidset *omits; }; static enum list_objects_filter_result filter_blobs_none( struct repository *r, @@ -60,32 +74,31 @@ static enum list_objects_filter_result filter_blobs_none( case LOFS_BLOB: assert(obj->type == OBJ_BLOB); assert((obj->flags & SEEN) == 0); if (filter_data->omits) oidset_insert(filter_data->omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } } -static void *filter_blobs_none__init( +static void filter_blobs_none__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_blobs_none_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; - *filter_fn = filter_blobs_none; - *filter_free_fn = free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_blobs_none; + filter->free_fn = free; } /* * A filter for list-objects to omit ALL trees and blobs from the traversal. * Can OPTIONALLY collect a list of the omitted OIDs. */ struct filter_trees_depth_data { struct oidset *omits; /* @@ -194,35 +207,34 @@ static enum list_objects_filter_result filter_trees_depth( } static void filter_trees_free(void *filter_data) { struct filter_trees_depth_data *d = filter_data; if (!d) return; oidmap_free(&d->seen_at_depth, 1); free(d); } -static void *filter_trees_depth__init( +static void filter_trees_depth__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_trees_depth_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; oidmap_init(&d->seen_at_depth, 0); d->exclude_depth = filter_options->tree_exclude_depth; d->current_depth = 0; - *filter_fn = filter_trees_depth; - *filter_free_fn = filter_trees_free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_trees_depth; + filter->free_fn = filter_trees_free; } /* * A filter for list-objects to omit large blobs. * And to OPTIONALLY collect a list of the omitted OIDs. */ struct filter_blobs_limit_data { struct oidset *omits; unsigned long max_bytes; }; @@ -274,33 +286,32 @@ static enum list_objects_filter_result filter_blobs_limit( oidset_insert(filter_data->omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } include_it: if (filter_data->omits) oidset_remove(filter_data->omits, &obj->oid); return LOFR_MARK_SEEN | LOFR_DO_SHOW; } -static void *filter_blobs_limit__init( +static void filter_blobs_limit__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; d->max_bytes = filter_options->blob_limit_value; - *filter_fn = filter_blobs_limit; - *filter_free_fn = free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_blobs_limit; + filter->free_fn = free; } /* * A filter driven by a sparse-checkout specification to only * include blobs that a sparse checkout would populate. * * The sparse-checkout spec can be loaded from a blob with the * given OID or from a local pathname. We allow an OID because * the repo may be bare or we may be doing the filtering on the * server. @@ -449,71 +460,98 @@ static enum list_objects_filter_result filter_sparse( } static void filter_sparse_free(void *filter_data) { struct filter_sparse_data *d = filter_data; free(d->array_frame); free(d); } -static void *filter_sparse_oid__init( +static void filter_sparse_oid__init( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct filter *filter) { struct filter_sparse_data *d = xcalloc(1, sizeof(*d)); d->omits = omitted; if (add_excludes_from_blob_to_list(filter_options->sparse_oid_value, NULL, 0, &d->el) < 0) die("could not load filter specification"); ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; d->nr++; - *filter_fn = filter_sparse; - *filter_free_fn = filter_sparse_free; - return d; + filter->filter_data = d; + filter->filter_object_fn = filter_sparse; + filter->free_fn = filter_sparse_free; } -typedef void *(*filter_init_fn)( +typedef void (*filter_init_fn)( struct oidset *omitted, struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn); + struct filter *filter); /* * Must match "enum list_objects_filter_choice". */ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, filter_trees_depth__init, filter_sparse_oid__init, }; -void *list_objects_filter__init( +struct filter *list_objects_filter__init( struct oidset *omitted, - struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn) + struct list_objects_filter_options *filter_options) { + struct filter *filter; filter_init_fn init_fn; assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT); if (filter_options->choice >= LOFC__COUNT) BUG("invalid list-objects filter choice: %d", filter_options->choice); init_fn = s_filters[filter_options->choice]; - if (init_fn) - return init_fn(omitted, filter_options, - filter_fn, filter_free_fn); - *filter_fn = NULL; - *filter_free_fn = NULL; - return NULL; + if (!init_fn) + return NULL; + + filter = xcalloc(1, sizeof(*filter)); + init_fn(omitted, filter_options, filter); + return filter; +} + +enum list_objects_filter_result list_objects_filter__filter_object( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct filter *filter) +{ + if (filter && (obj->flags & NOT_USER_GIVEN)) + return filter->filter_object_fn(r, filter_situation, obj, + pathname, filename, + filter->filter_data); + /* + * No filter is active or user gave object explicitly. In this case, + * always show the object (except when LOFS_END_TREE, since this tree + * had already been shown when LOFS_BEGIN_TREE). + */ + if (filter_situation == LOFS_END_TREE) + return 0; + return LOFR_MARK_SEEN | LOFR_DO_SHOW; +} + +void list_objects_filter__free(struct filter *filter) +{ + if (!filter) + return; + filter->free_fn(filter->filter_data); + free(filter); } diff --git a/list-objects-filter.h b/list-objects-filter.h index 1d45a4ad57..6908954266 100644 --- a/list-objects-filter.h +++ b/list-objects-filter.h @@ -53,37 +53,34 @@ enum list_objects_filter_result { LOFR_DO_SHOW = 1<<1, LOFR_SKIP_TREE = 1<<2, }; enum list_objects_filter_situation { LOFS_BEGIN_TREE, LOFS_END_TREE, LOFS_BLOB }; -typedef enum list_objects_filter_result (*filter_object_fn)( +struct filter; + +/* Constructor for the set of defined list-objects filters. */ +struct filter *list_objects_filter__init( + struct oidset *omitted, + struct list_objects_filter_options *filter_options); + +/* + * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this + * function behaves as expected if no filter is configured: all objects are + * included. + */ +enum list_objects_filter_result list_objects_filter__filter_object( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, - void *filter_data); - -typedef void (*filter_free_fn)(void *filter_data); + struct filter *filter); -/* - * Constructor for the set of defined list-objects filters. - * Returns a generic "void *filter_data". - * - * The returned "filter_fn" will be used by traverse_commit_list() - * to filter the results. - * - * The returned "filter_free_fn" is a destructor for the - * filter_data. - */ -void *list_objects_filter__init( - struct oidset *omitted, - struct list_objects_filter_options *filter_options, - filter_object_fn *filter_fn, - filter_free_fn *filter_free_fn); +/* Destroys `filter`. Does nothing if `filter` is null. */ +void list_objects_filter__free(struct filter *filter); #endif /* LIST_OBJECTS_FILTER_H */ diff --git a/list-objects.c b/list-objects.c index b5651ddd5b..9307d91fb3 100644 --- a/list-objects.c +++ b/list-objects.c @@ -11,32 +11,31 @@ #include "list-objects-filter-options.h" #include "packfile.h" #include "object-store.h" #include "trace.h" struct traversal_context { struct rev_info *revs; show_object_fn show_object; show_commit_fn show_commit; void *show_data; - filter_object_fn filter_fn; - void *filter_data; + struct filter *filter; }; static void process_blob(struct traversal_context *ctx, struct blob *blob, struct strbuf *path, const char *name) { struct object *obj = &blob->object; size_t pathlen; - enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; + enum list_objects_filter_result r; if (!ctx->revs->blob_objects) return; if (!obj) die("bad blob object"); if (obj->flags & (UNINTERESTING | SEEN)) return; /* * Pre-filter known-missing objects when explicitly requested. @@ -47,25 +46,24 @@ static void process_blob(struct traversal_context *ctx, * may cause the actual filter to report an incomplete list * of missing objects. */ if (ctx->revs->exclude_promisor_objects && !has_object_file(&obj->oid) && is_promisor_object(&obj->oid)) return; pathlen = path->len; strbuf_addstr(path, name); - if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) - r = ctx->filter_fn(ctx->revs->repo, - LOFS_BLOB, obj, - path->buf, &path->buf[pathlen], - ctx->filter_data); + r = list_objects_filter__filter_object(ctx->revs->repo, + LOFS_BLOB, obj, + path->buf, &path->buf[pathlen], + ctx->filter); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) ctx->show_object(obj, path->buf, ctx->show_data); strbuf_setlen(path, pathlen); } /* * Processing a gitlink entry currently does nothing, since * we do not recurse into the subproject. @@ -150,21 +148,21 @@ static void process_tree_contents(struct traversal_context *ctx, } static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, const char *name) { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; int baselen = base->len; - enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; + enum list_objects_filter_result r; int failed_parse; if (!revs->tree_objects) return; if (!obj) die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; failed_parse = parse_tree_gently(tree, 1); @@ -179,47 +177,44 @@ static void process_tree(struct traversal_context *ctx, */ if (revs->exclude_promisor_objects && is_promisor_object(&obj->oid)) return; if (!revs->do_not_die_on_missing_tree) die("bad tree object %s", oid_to_hex(&obj->oid)); } strbuf_addstr(base, name); - if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) - r = ctx->filter_fn(ctx->revs->repo, - LOFS_BEGIN_TREE, obj, - base->buf, &base->buf[baselen], - ctx->filter_data); + r = list_objects_filter__filter_object(ctx->revs->repo, + LOFS_BEGIN_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) ctx->show_object(obj, base->buf, ctx->show_data); if (base->len) strbuf_addch(base, '/'); if (r & LOFR_SKIP_TREE) trace_printf("Skipping contents of tree %s...\n", base->buf); else if (!failed_parse) process_tree_contents(ctx, tree, base); - if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) { - r = ctx->filter_fn(ctx->revs->repo, - LOFS_END_TREE, obj, - base->buf, &base->buf[baselen], - ctx->filter_data); - if (r & LOFR_MARK_SEEN) - obj->flags |= SEEN; - if (r & LOFR_DO_SHOW) - ctx->show_object(obj, base->buf, ctx->show_data); - } + r = list_objects_filter__filter_object(ctx->revs->repo, + LOFS_END_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter); + if (r & LOFR_MARK_SEEN) + obj->flags |= SEEN; + if (r & LOFR_DO_SHOW) + ctx->show_object(obj, base->buf, ctx->show_data); strbuf_setlen(base, baselen); free_tree_buffer(tree); } static void mark_edge_parents_uninteresting(struct commit *commit, struct rev_info *revs, show_edge_fn show_edge) { struct commit_list *parents; @@ -395,38 +390,32 @@ static void do_traverse(struct traversal_context *ctx) void traverse_commit_list(struct rev_info *revs, show_commit_fn show_commit, show_object_fn show_object, void *show_data) { struct traversal_context ctx; ctx.revs = revs; ctx.show_commit = show_commit; ctx.show_object = show_object; ctx.show_data = show_data; - ctx.filter_fn = NULL; - ctx.filter_data = NULL; + ctx.filter = NULL; do_traverse(&ctx); } void traverse_commit_list_filtered( struct list_objects_filter_options *filter_options, struct rev_info *revs, show_commit_fn show_commit, show_object_fn show_object, void *show_data, struct oidset *omitted) { struct traversal_context ctx; - filter_free_fn filter_free_fn = NULL; ctx.revs = revs; ctx.show_object = show_object; ctx.show_commit = show_commit; ctx.show_data = show_data; - ctx.filter_fn = NULL; - - ctx.filter_data = list_objects_filter__init(omitted, filter_options, - &ctx.filter_fn, &filter_free_fn); + ctx.filter = list_objects_filter__init(omitted, filter_options); do_traverse(&ctx); - if (ctx.filter_data && filter_free_fn) - filter_free_fn(ctx.filter_data); + list_objects_filter__free(ctx.filter); } From patchwork Thu Jun 27 22:54:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020805 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E463813B4 for ; Thu, 27 Jun 2019 22:54:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D99BE28708 for ; Thu, 27 Jun 2019 22:54:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id CCACF2872E; Thu, 27 Jun 2019 22:54:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E51CD28708 for ; Thu, 27 Jun 2019 22:54:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726655AbfF0Wy2 (ORCPT ); Thu, 27 Jun 2019 18:54:28 -0400 Received: from mail-pl1-f202.google.com ([209.85.214.202]:33674 "EHLO mail-pl1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726647AbfF0Wy2 (ORCPT ); Thu, 27 Jun 2019 18:54:28 -0400 Received: by mail-pl1-f202.google.com with SMTP id f2so2261067plr.0 for ; Thu, 27 Jun 2019 15:54:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=JGksyiK5RmZkf+ytgjSy6zPynu37En4vDcWXMUpqvYY=; b=OIGA3NE7o9TwPaeKNetEBCNYON/cb5IGjjXKMjtPP61QI3ugbmCAaASqCiiArvuYce kNepY6T4HGbGhBy2c5olXSckHATPACf1DRZ5IjMeze0F6e1slzqyFUAiX64bwU4xtA4m oUA1CMqGqT44T0a8S9lRO0/KBOEWnQVmQnJpwlHdcXaevEmrsEcE/J1UHosU/+Ck+OUX KmUyC61A10SG1M5RE3dN0Z3UpkQ3TiiAhG1C4lsMz09KcQSPf5Yqdv54ihy4E+A67cgq 3kUyEVIMjby6coYdIYaM5SECqGHSXNN0S1p+T74NfjaBSxIsUwPq6CasieCxDK1Uz/G6 wlPA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=JGksyiK5RmZkf+ytgjSy6zPynu37En4vDcWXMUpqvYY=; b=AICSIosxYuPZ0wOE4KS7CAVRDU7TG2ZJ8tM0jB5kzU3vLHdgHYzdIbtt4dTt2WUneZ usWpZqmh0aM4R/pvEJ+A8tles7fk3e6UvSJXdfqpAMsEdc/sDgkyyjl+XuZST6mjjvhm 8X3IvvHU7U/apQLmM9ZEMtU9rInHkDuWe7GDkwwPwkE8WOHn/6gZOngax5NuL7YkPBJO WP7Y5bH5IY61mc0eq0p0RUWlVf1Wp5JcStblN4kHbAXgEjcayUrlIjwSjaHMbXqPnwWL p0Bl2vou/nUBIANn9kEW7qhBkTXEnzorodcQsLd45Rvb9W4VlWTA0k0ETXNQ+blKBay3 buqg== X-Gm-Message-State: APjAAAUi34AxyF1SUIDxsjdZqOFBysQ+TCIEFq57mm/tLgMVsnmX3u22 zQaT/WrFyAQpoxu8I6ZX+eETpf+F9IeUqsBqB8jFhGzySkbcPkJp0dD79i7Mf+X0glyhOhNmjcR XWTBdaK+8SSacUrTviJqYW4iDD3UcfaQYWP7V6gfrsRNh5qMLdhXSlYouPFA= X-Google-Smtp-Source: APXvYqydLjykAZ6GWMSLkek9aBKRDo8QraIOX5Ywz3fe3BrZDSreAmQiOCGYbxITHFXaekkCWTPqJleUbf+S X-Received: by 2002:a63:f14:: with SMTP id e20mr6127919pgl.227.1561676067244; Thu, 27 Jun 2019 15:54:27 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:06 -0700 In-Reply-To: Message-Id: <62441a2214528b114341796673c6f8fda14d24b8.1561675151.git.matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 02/10] list-objects-filter: put omits set in filter struct From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The oidset *omits pointer must be accessed by the combine filter in a type-agnostic way once the graph traversal is over. Store that pointer in the general `filter` struct. This will be used in a follow-up patch to implement the combine filter. Signed-off-by: Matthew DeVore --- list-objects-filter.c | 68 +++++++++++++++++-------------------------- 1 file changed, 26 insertions(+), 42 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index e06b82def0..3b4b6764ca 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -26,88 +26,76 @@ */ #define FILTER_SHOWN_BUT_REVISIT (1<<21) struct filter { enum list_objects_filter_result (*filter_object_fn)( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data); void (*free_fn)(void *filter_data); void *filter_data; -}; -/* - * A filter for list-objects to omit ALL blobs from the traversal. - * And to OPTIONALLY collect a list of the omitted OIDs. - */ -struct filter_blobs_none_data { + /* If non-NULL, the filter collects a list of the omitted OIDs here. */ struct oidset *omits; }; static enum list_objects_filter_result filter_blobs_none( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { - struct filter_blobs_none_data *filter_data = filter_data_; - switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); /* always include all tree objects */ return LOFR_MARK_SEEN | LOFR_DO_SHOW; case LOFS_END_TREE: assert(obj->type == OBJ_TREE); return LOFR_ZERO; case LOFS_BLOB: assert(obj->type == OBJ_BLOB); assert((obj->flags & SEEN) == 0); - if (filter_data->omits) - oidset_insert(filter_data->omits, &obj->oid); + if (omits) + oidset_insert(omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } } static void filter_blobs_none__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { - struct filter_blobs_none_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; - - filter->filter_data = d; filter->filter_object_fn = filter_blobs_none; filter->free_fn = free; } /* * A filter for list-objects to omit ALL trees and blobs from the traversal. * Can OPTIONALLY collect a list of the omitted OIDs. */ struct filter_trees_depth_data { - struct oidset *omits; - /* * Maps trees to the minimum depth at which they were seen. It is not * necessary to re-traverse a tree at deeper or equal depths than it has * already been traversed. * * We can't use LOFR_MARK_SEEN for tree objects since this will prevent * it from being traversed at shallower depths. */ struct oidmap seen_at_depth; @@ -116,38 +104,39 @@ struct filter_trees_depth_data { }; struct seen_map_entry { struct oidmap_entry base; size_t depth; }; /* Returns 1 if the oid was in the omits set before it was invoked. */ static int filter_trees_update_omits( struct object *obj, - struct filter_trees_depth_data *filter_data, + struct oidset *omits, int include_it) { - if (!filter_data->omits) + if (!omits) return 0; if (include_it) - return oidset_remove(filter_data->omits, &obj->oid); + return oidset_remove(omits, &obj->oid); else - return oidset_insert(filter_data->omits, &obj->oid); + return oidset_insert(omits, &obj->oid); } static enum list_objects_filter_result filter_trees_depth( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { struct filter_trees_depth_data *filter_data = filter_data_; struct seen_map_entry *seen_info; int include_it = filter_data->current_depth < filter_data->exclude_depth; int filter_res; int already_seen; /* @@ -158,47 +147,47 @@ static enum list_objects_filter_result filter_trees_depth( switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); case LOFS_END_TREE: assert(obj->type == OBJ_TREE); filter_data->current_depth--; return LOFR_ZERO; case LOFS_BLOB: - filter_trees_update_omits(obj, filter_data, include_it); + filter_trees_update_omits(obj, omits, include_it); return include_it ? LOFR_MARK_SEEN | LOFR_DO_SHOW : LOFR_ZERO; case LOFS_BEGIN_TREE: seen_info = oidmap_get( &filter_data->seen_at_depth, &obj->oid); if (!seen_info) { seen_info = xcalloc(1, sizeof(*seen_info)); oidcpy(&seen_info->base.oid, &obj->oid); seen_info->depth = filter_data->current_depth; oidmap_put(&filter_data->seen_at_depth, seen_info); already_seen = 0; } else { already_seen = filter_data->current_depth >= seen_info->depth; } if (already_seen) { filter_res = LOFR_SKIP_TREE; } else { int been_omitted = filter_trees_update_omits( - obj, filter_data, include_it); + obj, omits, include_it); seen_info->depth = filter_data->current_depth; if (include_it) filter_res = LOFR_DO_SHOW; - else if (filter_data->omits && !been_omitted) + else if (omits && !been_omitted) /* * Must update omit information of children * recursively; they have not been omitted yet. */ filter_res = LOFR_ZERO; else filter_res = LOFR_SKIP_TREE; } filter_data->current_depth++; @@ -208,50 +197,48 @@ static enum list_objects_filter_result filter_trees_depth( static void filter_trees_free(void *filter_data) { struct filter_trees_depth_data *d = filter_data; if (!d) return; oidmap_free(&d->seen_at_depth, 1); free(d); } static void filter_trees_depth__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { struct filter_trees_depth_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; oidmap_init(&d->seen_at_depth, 0); d->exclude_depth = filter_options->tree_exclude_depth; d->current_depth = 0; filter->filter_data = d; filter->filter_object_fn = filter_trees_depth; filter->free_fn = filter_trees_free; } /* * A filter for list-objects to omit large blobs. * And to OPTIONALLY collect a list of the omitted OIDs. */ struct filter_blobs_limit_data { - struct oidset *omits; unsigned long max_bytes; }; static enum list_objects_filter_result filter_blobs_limit( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { struct filter_blobs_limit_data *filter_data = filter_data_; unsigned long object_length; enum object_type t; switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); @@ -275,38 +262,36 @@ static enum list_objects_filter_result filter_blobs_limit( * apply the size filter criteria. Be conservative * and force show it (and let the caller deal with * the ambiguity). */ goto include_it; } if (object_length < filter_data->max_bytes) goto include_it; - if (filter_data->omits) - oidset_insert(filter_data->omits, &obj->oid); + if (omits) + oidset_insert(omits, &obj->oid); return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ } include_it: - if (filter_data->omits) - oidset_remove(filter_data->omits, &obj->oid); + if (omits) + oidset_remove(omits, &obj->oid); return LOFR_MARK_SEEN | LOFR_DO_SHOW; } static void filter_blobs_limit__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { struct filter_blobs_limit_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; d->max_bytes = filter_options->blob_limit_value; filter->filter_data = d; filter->filter_object_fn = filter_blobs_limit; filter->free_fn = free; } /* * A filter driven by a sparse-checkout specification to only * include blobs that a sparse checkout would populate. @@ -330,33 +315,33 @@ struct frame { * omitted objects. * * 0 if everything (recursively) contained in this directory * has been explicitly included (SHOWN) in the result and * the directory may be short-cut later in the traversal. */ unsigned child_prov_omit : 1; }; struct filter_sparse_data { - struct oidset *omits; struct exclude_list el; size_t nr, alloc; struct frame *array_frame; }; static enum list_objects_filter_result filter_sparse( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, + struct oidset *omits, void *filter_data_) { struct filter_sparse_data *filter_data = filter_data_; int val, dtype; struct frame *frame; switch (filter_situation) { default: BUG("unknown filter_situation: %d", filter_situation); @@ -424,79 +409,76 @@ static enum list_objects_filter_result filter_sparse( frame = &filter_data->array_frame[filter_data->nr - 1]; dtype = DT_REG; val = is_excluded_from_list(pathname, strlen(pathname), filename, &dtype, &filter_data->el, r->index); if (val < 0) val = frame->defval; if (val > 0) { - if (filter_data->omits) - oidset_remove(filter_data->omits, &obj->oid); + if (omits) + oidset_remove(omits, &obj->oid); return LOFR_MARK_SEEN | LOFR_DO_SHOW; } /* * Provisionally omit it. We've already established that * this pathname is not in the sparse-checkout specification * with the CURRENT pathname, so we *WANT* to omit this blob. * * However, a pathname elsewhere in the tree may also * reference this same blob, so we cannot reject it yet. * Leave the LOFR_ bits unset so that if the blob appears * again in the traversal, we will be asked again. */ - if (filter_data->omits) - oidset_insert(filter_data->omits, &obj->oid); + if (omits) + oidset_insert(omits, &obj->oid); /* * Remember that at least 1 blob in this tree was * provisionally omitted. This prevents us from short * cutting the tree in future iterations. */ frame->child_prov_omit = 1; return LOFR_ZERO; } } static void filter_sparse_free(void *filter_data) { struct filter_sparse_data *d = filter_data; free(d->array_frame); free(d); } static void filter_sparse_oid__init( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter) { struct filter_sparse_data *d = xcalloc(1, sizeof(*d)); - d->omits = omitted; if (add_excludes_from_blob_to_list(filter_options->sparse_oid_value, NULL, 0, &d->el) < 0) die("could not load filter specification"); ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; d->nr++; filter->filter_data = d; filter->filter_object_fn = filter_sparse; filter->free_fn = filter_sparse_free; } typedef void (*filter_init_fn)( - struct oidset *omitted, struct list_objects_filter_options *filter_options, struct filter *filter); /* * Must match "enum list_objects_filter_choice". */ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, @@ -515,35 +497,37 @@ struct filter *list_objects_filter__init( if (filter_options->choice >= LOFC__COUNT) BUG("invalid list-objects filter choice: %d", filter_options->choice); init_fn = s_filters[filter_options->choice]; if (!init_fn) return NULL; filter = xcalloc(1, sizeof(*filter)); - init_fn(omitted, filter_options, filter); + filter->omits = omitted; + init_fn(filter_options, filter); return filter; } enum list_objects_filter_result list_objects_filter__filter_object( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, struct filter *filter) { if (filter && (obj->flags & NOT_USER_GIVEN)) return filter->filter_object_fn(r, filter_situation, obj, pathname, filename, + filter->omits, filter->filter_data); /* * No filter is active or user gave object explicitly. In this case, * always show the object (except when LOFS_END_TREE, since this tree * had already been shown when LOFS_BEGIN_TREE). */ if (filter_situation == LOFS_END_TREE) return 0; return LOFR_MARK_SEEN | LOFR_DO_SHOW; } From patchwork Thu Jun 27 22:54:07 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020803 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 972E6112C for ; Thu, 27 Jun 2019 22:54:33 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8D8ED28708 for ; Thu, 27 Jun 2019 22:54:33 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 80B4B2872E; Thu, 27 Jun 2019 22:54:33 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 20D8028708 for ; Thu, 27 Jun 2019 22:54:33 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726674AbfF0Wyc (ORCPT ); Thu, 27 Jun 2019 18:54:32 -0400 Received: from mail-pg1-f202.google.com ([209.85.215.202]:42984 "EHLO mail-pg1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wyb (ORCPT ); Thu, 27 Jun 2019 18:54:31 -0400 Received: by mail-pg1-f202.google.com with SMTP id d3so2078252pgc.9 for ; Thu, 27 Jun 2019 15:54:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=e0zrX29cXTiPgzh0S2x3jm3vNwOrDTtHpzupGcqrZMw=; b=R1r+wqupxbTlW9Uvlrhe2WOYGgHXFAbu4paLckC1g+NpInA0cStNbPOdxs05Ed90K0 tIpWuUXecJETGUnPI/q9+mv3dvRzr+dMwCOoZulh2o+slGENfo6F3D9Bjn5Ux6V9i0TV zuIVsql1jf7theny26kZlnPOvMhFFB7AdZMblf5+e1IDuIJE61o3CNV5DRmAr2ZCvrmm ircWcpDGrOFDjMA5cg4iRmJv4U44l8Zn2004DiTiCcty2kQkk268lSp+mVGaz7gmZHVb bWUsefugkK2WvAn0TeI14yJpmugno3XDcvxJhyJHlWnxyhjfPfPlOsWdQ14zEHHpQBsY csFw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=e0zrX29cXTiPgzh0S2x3jm3vNwOrDTtHpzupGcqrZMw=; b=LOgwhvZrYOCtCStyGYneu8O8BOKvxZNr/6agLq3yuWVRZTNnrYkslQEG506jfOa7ng RHask1a0wdBZ2IV8umsfiCGvkI0FUuEWNM/JaQ0t+++CKg2eqaDIwHS88WEDxVZyAqZI BwfjFjgKJvs055a2mcnrAoG2OivlnMBXpx8/9Vl43a1YY5V8Ok7GD7rmFqDK6NgBL3ie uqbeSINGi0QS0/d4TTKveioTvQY8FMvm/OyTXxaEf8CDWd4lSOhr7MT9Z+NQ5iVB5GDS Sa+ah2cEURzXBOsObpeUZOQ/jD/0SGCphaR8KZkV6x0exOuVe4wfim4pWPUJSUYZtiOj THgQ== X-Gm-Message-State: APjAAAV918Clsen+1FtWQ5K1hTe0yH0i4A5qHerW5ztCpYXfHzwpciSc 5NmJ3Szqqdzw/wyR9I8sYA84CvmB5LGlq7QlXOMwxkoz6Z5E9XoEEZZQlDASA+/ifaaie9a18xa wtnU6kWqtZSlrYnFadfi6CszP9x5GFsr62FluuRjGhh3Et9cWbC52x9WdTnA= X-Google-Smtp-Source: APXvYqyrrjfAooMu7JQ3TWvtg650K7QlSC8U7Bdm/SPjFTav1eceO/ZDPF017dljA1LO/UHDd5QPYYUfWte8 X-Received: by 2002:a63:c03:: with SMTP id b3mr6236095pgl.68.1561676069908; Thu, 27 Jun 2019 15:54:29 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:07 -0700 In-Reply-To: Message-Id: <14400b99b888d6181ef95220de0b372216e6ef1c.1561675151.git.matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 03/10] list-objects-filter-options: always supply *errbuf From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Making errbuf an optional argument complicates error reporting. Fix this by making all callers supply an errbuf, even if they may ignore it. This will be important in follow-up patches where the filter-spec parsing has more pitfalls and possible errors. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 21 ++++++++------------- 1 file changed, 8 insertions(+), 13 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 1cb20c659c..7c3e397d29 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -23,47 +23,40 @@ * convenience of the current command. */ static int gently_parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg, struct strbuf *errbuf) { const char *v0; if (filter_options->choice) { - if (errbuf) { - strbuf_addstr( - errbuf, - _("multiple filter-specs cannot be combined")); - } + strbuf_addstr( + errbuf, _("multiple filter-specs cannot be combined")); return 1; } filter_options->filter_spec = strdup(arg); if (!strcmp(arg, "blob:none")) { filter_options->choice = LOFC_BLOB_NONE; return 0; } else if (skip_prefix(arg, "blob:limit=", &v0)) { if (git_parse_ulong(v0, &filter_options->blob_limit_value)) { filter_options->choice = LOFC_BLOB_LIMIT; return 0; } } else if (skip_prefix(arg, "tree:", &v0)) { if (!git_parse_ulong(v0, &filter_options->tree_exclude_depth)) { - if (errbuf) { - strbuf_addstr( - errbuf, - _("expected 'tree:'")); - } + strbuf_addstr(errbuf, _("expected 'tree:'")); return 1; } filter_options->choice = LOFC_TREE_DEPTH; return 0; } else if (skip_prefix(arg, "sparse:oid=", &v0)) { struct object_context oc; struct object_id sparse_oid; /* @@ -83,22 +76,21 @@ static int gently_parse_list_objects_filter( errbuf, _("sparse:path filters support has been dropped")); } return 1; } /* * Please update _git_fetch() in git-completion.bash when you * add new filters */ - if (errbuf) - strbuf_addf(errbuf, _("invalid filter-spec '%s'"), arg); + strbuf_addf(errbuf, _("invalid filter-spec '%s'"), arg); memset(filter_options, 0, sizeof(*filter_options)); return 1; } int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; if (gently_parse_list_objects_filter(filter_options, arg, &buf)) @@ -168,19 +160,22 @@ void partial_clone_register( */ core_partial_clone_filter_default = xstrdup(filter_options->filter_spec); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); } void partial_clone_get_default_filter_spec( struct list_objects_filter_options *filter_options) { + struct strbuf errbuf = STRBUF_INIT; + /* * Parse default value, but silently ignore it if it is invalid. */ if (!core_partial_clone_filter_default) return; gently_parse_list_objects_filter(filter_options, core_partial_clone_filter_default, - NULL); + &errbuf); + strbuf_release(&errbuf); } From patchwork Thu Jun 27 22:54:08 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020807 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2E76C13B4 for ; Thu, 27 Jun 2019 22:54:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 21B4628708 for ; Thu, 27 Jun 2019 22:54:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 15D3F2872E; Thu, 27 Jun 2019 22:54:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 69A6C28708 for ; Thu, 27 Jun 2019 22:54:35 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726679AbfF0Wye (ORCPT ); Thu, 27 Jun 2019 18:54:34 -0400 Received: from mail-pg1-f202.google.com ([209.85.215.202]:55570 "EHLO mail-pg1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wye (ORCPT ); Thu, 27 Jun 2019 18:54:34 -0400 Received: by mail-pg1-f202.google.com with SMTP id b10so2059390pgb.22 for ; Thu, 27 Jun 2019 15:54:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Hg2CYnpzSZQK9oaG/ZwdZmg0hFzErkrPR5GR43OFyBg=; b=I5+BQ4PZA2PjtJ7zAzENF1+6dY9dDZaGJkqoB5jXZXfKsWo46P2TB/+WPc/eicPWNT xCClPVrBJU298ITn6iHL7tWIAfU9IbCJn9u1X2CpzNgMEl5umKaw08lZ/JUqKab3hs3j e63ce62hpb/eWpChU1UN2mUTk5MWZn2+XJBNpj5yvurYGywSI+9u3ACV2plshcX8CZ9W pXeyhpk92yGzs85ZrXsYuWJ/DTu9h7FEr4AOCvIw433xZr8eO4iaBBfI6VqI17Q1JICV n6kUtb35l1kt81HX+OZ60PrFGGSSH6f0M7W3tYWoqkl8Ys8qH3F0jV0d4RvacgiEKcU9 L0QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Hg2CYnpzSZQK9oaG/ZwdZmg0hFzErkrPR5GR43OFyBg=; b=MKufRq3WPLXnBpqOql1ay+O9InbDdrDn3llvOFT36QiL0gnQUr1Tx4ajMhp+uTZCE5 IVqL/gpJ9sND67QoDfgcgi6D8V61JRQlcJh2t2QoVT5v04ZXrSCnwznESybBpkB+Gf0k DOGKj44++swjUEUvala9/C1qCdhp6xBk1D/r9xCCr2qm1xTkMOl9piYNzc6GSlwN7nHp 9gKe4fKVqXiHqiYhyId46hb1Eyj6pQUmRF7cA4++FyMsTFkotsbQlM+xTG3QjXK9Eli/ nC608bBX0AYUCxxc1YhXT8YjaGxOZFuETMuwSQss/hizhJL8y4wC3W2qgJ8ER6wET2G4 H8DQ== X-Gm-Message-State: APjAAAWWnXmm/abRvaMByf35iBxI0kIc5JL3lBI00ZsTl90DxcJYhI4V xnmCYfuObeOExodYYTE8InoReUC6wMSW0sRYKCZUlO7ShScbPWXqJqkFWOx2NGaZbHLSfkN9rLK f1Drvk+WAGbLN1/Ege0JTmIu0TO0WxBhAglpn2eXq6RWyNPks2Pn3iJargVs= X-Google-Smtp-Source: APXvYqxRC59SCiIZLTqpv12lvXAB3CbEPM156we9oOnCIj7JX5GlcXHIlowXD6iaBx3tySnwrbpYRnFr7kA6 X-Received: by 2002:a63:7d18:: with SMTP id y24mr6081873pgc.438.1561676072441; Thu, 27 Jun 2019 15:54:32 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:08 -0700 In-Reply-To: Message-Id: <17b525b6104147658504f4e7a9044e3dceee5a99.1561675151.git.matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 04/10] list-objects-filter: implement composite filters From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de, Jeff Hostetler , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allow combining filters such that only objects accepted by all filters are shown. The motivation for this is to allow getting directory listings without also fetching blobs. This can be done by combining blob:none with tree:. There are massive repositories that have larger-than-expected trees - even if you include only a single commit. A combined filter supports any number of subfilters, and is written in the following form: combine:++ Certain non-alphanumeric characters in each filter must be URL-encoded. For now, combined filters must be specified in this form. In a subsequent commit, rev-list will support multiple --filter arguments which will have the same effect as specifying one filter argument starting with "combine:". The documentation will be updated in that commit, as the URL-encoding scheme is in general not meant to be used directly by the user, and it is better to describe the URL-encoding feature in terms of the repeated flag. Helped-by: Emily Shaffer Helped-by: Jeff Hostetler Helped-by: Johannes Schindelin Helped-by: Jonathan Tan Helped-by: Junio C Hamano Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 106 +++++++++++++++++- list-objects-filter-options.h | 17 ++- list-objects-filter.c | 161 ++++++++++++++++++++++++++++ list-objects-filter.h | 13 ++- t/t6112-rev-list-filters-objects.sh | 151 +++++++++++++++++++++++++- url.c | 6 ++ url.h | 8 ++ 7 files changed, 454 insertions(+), 8 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 7c3e397d29..75d0236ee2 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -1,18 +1,24 @@ #include "cache.h" #include "commit.h" #include "config.h" #include "revision.h" #include "argv-array.h" #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "url.h" + +static int parse_combine_filter( + struct list_objects_filter_options *filter_options, + const char *arg, + struct strbuf *errbuf); /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: * --filter= * and in the pack protocol as: * "filter" SP * * The filter keyword will be used by many commands. * See Documentation/rev-list-options.txt for allowed values for . @@ -28,22 +34,20 @@ static int gently_parse_list_objects_filter( struct strbuf *errbuf) { const char *v0; if (filter_options->choice) { strbuf_addstr( errbuf, _("multiple filter-specs cannot be combined")); return 1; } - filter_options->filter_spec = strdup(arg); - if (!strcmp(arg, "blob:none")) { filter_options->choice = LOFC_BLOB_NONE; return 0; } else if (skip_prefix(arg, "blob:limit=", &v0)) { if (git_parse_ulong(v0, &filter_options->blob_limit_value)) { filter_options->choice = LOFC_BLOB_LIMIT; return 0; } @@ -70,36 +74,125 @@ static int gently_parse_list_objects_filter( filter_options->choice = LOFC_SPARSE_OID; return 0; } else if (skip_prefix(arg, "sparse:path=", &v0)) { if (errbuf) { strbuf_addstr( errbuf, _("sparse:path filters support has been dropped")); } return 1; + + } else if (skip_prefix(arg, "combine:", &v0)) { + return parse_combine_filter(filter_options, v0, errbuf); + } /* * Please update _git_fetch() in git-completion.bash when you * add new filters */ strbuf_addf(errbuf, _("invalid filter-spec '%s'"), arg); memset(filter_options, 0, sizeof(*filter_options)); return 1; } +static const char *RESERVED_NON_WS = "~`!@#$^&*()[]{}\\;'\",<>?"; + +static int has_reserved_character( + struct strbuf *sub_spec, struct strbuf *errbuf) +{ + const char *c = sub_spec->buf; + while (*c) { + if (*c <= ' ' || strchr(RESERVED_NON_WS, *c)) { + strbuf_addf( + errbuf, + _("must escape char in sub-filter-spec: '%c'"), + *c); + return 1; + } + c++; + } + + return 0; +} + +static int parse_combine_subfilter( + struct list_objects_filter_options *filter_options, + struct strbuf *subspec, + struct strbuf *errbuf) +{ + size_t new_index = filter_options->sub_nr++; + char *decoded; + int result; + + ALLOC_GROW(filter_options->sub, filter_options->sub_nr, + filter_options->sub_alloc); + memset(&filter_options->sub[new_index], 0, + sizeof(*filter_options->sub)); + + decoded = url_percent_decode(subspec->buf); + + result = has_reserved_character(subspec, errbuf) || + gently_parse_list_objects_filter( + &filter_options->sub[new_index], decoded, errbuf); + + free(decoded); + return result; +} + +static int parse_combine_filter( + struct list_objects_filter_options *filter_options, + const char *arg, + struct strbuf *errbuf) +{ + struct strbuf **subspecs = strbuf_split_str(arg, '+', 0); + size_t sub; + int result = 0; + + if (!subspecs[0]) { + strbuf_addstr(errbuf, _("expected something after combine:")); + result = 1; + goto cleanup; + } + + for (sub = 0; subspecs[sub] && !result; sub++) { + if (subspecs[sub + 1]) { + /* + * This is not the last subspec. Remove trailing "+" so + * we can parse it. + */ + size_t last = subspecs[sub]->len - 1; + assert(subspecs[sub]->buf[last] == '+'); + strbuf_remove(subspecs[sub], last, 1); + } + result = parse_combine_subfilter( + filter_options, subspecs[sub], errbuf); + } + + filter_options->choice = LOFC_COMBINE; + +cleanup: + strbuf_list_free(subspecs); + if (result) { + list_objects_filter_release(filter_options); + memset(filter_options, 0, sizeof(*filter_options)); + } + return result; +} + int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; + filter_options->filter_spec = strdup(arg); if (gently_parse_list_objects_filter(filter_options, arg, &buf)) die("%s", buf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; @@ -122,22 +215,29 @@ void expand_list_objects_filter_spec( else if (filter->choice == LOFC_TREE_DEPTH) strbuf_addf(expanded_spec, "tree:%lu", filter->tree_exclude_depth); else strbuf_addstr(expanded_spec, filter->filter_spec); } void list_objects_filter_release( struct list_objects_filter_options *filter_options) { + size_t sub; + + if (!filter_options) + return; free(filter_options->filter_spec); free(filter_options->sparse_oid_value); + for (sub = 0; sub < filter_options->sub_nr; sub++) + list_objects_filter_release(&filter_options->sub[sub]); + free(filter_options->sub); memset(filter_options, 0, sizeof(*filter_options)); } void partial_clone_register( const char *remote, const struct list_objects_filter_options *filter_options) { /* * Record the name of the partial clone remote in the * config and in the global variable -- the latter is @@ -167,15 +267,17 @@ void partial_clone_register( void partial_clone_get_default_filter_spec( struct list_objects_filter_options *filter_options) { struct strbuf errbuf = STRBUF_INIT; /* * Parse default value, but silently ignore it if it is invalid. */ if (!core_partial_clone_filter_default) return; + + filter_options->filter_spec = strdup(core_partial_clone_filter_default); gently_parse_list_objects_filter(filter_options, core_partial_clone_filter_default, &errbuf); strbuf_release(&errbuf); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index c54f0000fb..789faef1e5 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -6,20 +6,21 @@ /* * The list of defined filters for list-objects. */ enum list_objects_filter_choice { LOFC_DISABLED = 0, LOFC_BLOB_NONE, LOFC_BLOB_LIMIT, LOFC_TREE_DEPTH, LOFC_SPARSE_OID, + LOFC_COMBINE, LOFC__COUNT /* must be last */ }; struct list_objects_filter_options { /* * 'filter_spec' is the raw argument value given on the command line * or protocol request. (The part after the "--keyword=".) For * commands that launch filtering sub-processes, or for communication * over the network, don't use this value; use the result of * expand_list_objects_filter_spec() instead. @@ -31,27 +32,37 @@ struct list_objects_filter_options { * the filtering algorithm to use. */ enum list_objects_filter_choice choice; /* * Choice is LOFC_DISABLED because "--no-filter" was requested. */ unsigned int no_filter : 1; /* - * Parsed values (fields) from within the filter-spec. These are - * choice-specific; not all values will be defined for any given - * choice. + * BEGIN choice-specific parsed values from within the filter-spec. Only + * some values will be defined for any given choice. */ + struct object_id *sparse_oid_value; unsigned long blob_limit_value; unsigned long tree_exclude_depth; + + /* LOFC_COMBINE values */ + + /* This array contains all the subfilters which this filter combines. */ + size_t sub_nr, sub_alloc; + struct list_objects_filter_options *sub; + + /* + * END choice-specific parsed values. + */ }; /* Normalized command line arguments */ #define CL_ARG__FILTER "filter" int parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg); int opt_parse_list_objects_filter(const struct option *opt, diff --git a/list-objects-filter.c b/list-objects-filter.c index 3b4b6764ca..d664264d65 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -19,30 +19,55 @@ * FILTER_SHOWN_BUT_REVISIT -- we set this bit on tree objects * that have been shown, but should be revisited if they appear * in the traversal (until we mark it SEEN). This is a way to * let us silently de-dup calls to show() in the caller. This * is subtly different from the "revision.h:SHOWN" and the * "sha1-name.c:ONELINE_SEEN" bits. And also different from * the non-de-dup usage in pack-bitmap.c */ #define FILTER_SHOWN_BUT_REVISIT (1<<21) +struct subfilter { + struct filter *filter; + struct oidset seen; + struct oidset omits; + struct object_id skip_tree; + unsigned is_skipping_tree : 1; +}; + struct filter { enum list_objects_filter_result (*filter_object_fn)( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, struct oidset *omits, void *filter_data); + /* + * Optional. If this function is supplied and the filter needs + * to collect omits, then this function is called once before + * free_fn is called. + * + * This is required because the following two conditions hold: + * + * a. A tree filter can add and remove objects as an object + * graph is traversed. + * b. A combine filter's omit set is the union of all its + * subfilters, which may include tree: filters. + * + * As such, the omits sets must be separate sets, and can only + * be unioned after the traversal is completed. + */ + void (*finalize_omits_fn)(struct oidset *omits, void *filter_data); + void (*free_fn)(void *filter_data); void *filter_data; /* If non-NULL, the filter collects a list of the omitted OIDs here. */ struct oidset *omits; }; static enum list_objects_filter_result filter_blobs_none( struct repository *r, @@ -464,33 +489,167 @@ static void filter_sparse_oid__init( ALLOC_GROW(d->array_frame, d->nr + 1, d->alloc); d->array_frame[d->nr].defval = 0; /* default to include */ d->array_frame[d->nr].child_prov_omit = 0; d->nr++; filter->filter_data = d; filter->filter_object_fn = filter_sparse; filter->free_fn = filter_sparse_free; } +/* A filter which only shows objects shown by all sub-filters. */ +struct combine_filter_data { + struct subfilter *sub; + size_t nr; +}; + +static enum list_objects_filter_result process_subfilter( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct subfilter *sub) +{ + enum list_objects_filter_result result; + + /* + * Check and update is_skipping_tree before oidset_contains so + * that is_skipping_tree gets unset even when the object is + * marked as seen. As of this writing, no filter uses + * LOFR_MARK_SEEN on trees that also uses LOFR_SKIP_TREE, so the + * ordering is only theoretically important. Be cautious if you + * change the order of the below checks and more filters have + * been added! + */ + if (sub->is_skipping_tree) { + if (filter_situation == LOFS_END_TREE && + oideq(&obj->oid, &sub->skip_tree)) + sub->is_skipping_tree = 0; + else + return LOFR_ZERO; + } + if (oidset_contains(&sub->seen, &obj->oid)) + return LOFR_ZERO; + + result = list_objects_filter__filter_object( + r, filter_situation, obj, pathname, filename, sub->filter); + + if (result & LOFR_MARK_SEEN) + oidset_insert(&sub->seen, &obj->oid); + + if (result & LOFR_SKIP_TREE) { + sub->is_skipping_tree = 1; + sub->skip_tree = obj->oid; + } + + return result; +} + +static enum list_objects_filter_result filter_combine( + struct repository *r, + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + struct oidset *omits, + void *filter_data) +{ + struct combine_filter_data *d = filter_data; + enum list_objects_filter_result combined_result = + LOFR_DO_SHOW | LOFR_MARK_SEEN | LOFR_SKIP_TREE; + size_t sub; + + for (sub = 0; sub < d->nr; sub++) { + enum list_objects_filter_result sub_result = process_subfilter( + r, filter_situation, obj, pathname, filename, + &d->sub[sub]); + if (!(sub_result & LOFR_DO_SHOW)) + combined_result &= ~LOFR_DO_SHOW; + if (!(sub_result & LOFR_MARK_SEEN)) + combined_result &= ~LOFR_MARK_SEEN; + if (!d->sub[sub].is_skipping_tree) + combined_result &= ~LOFR_SKIP_TREE; + } + + return combined_result; +} + +static void filter_combine__free(void *filter_data) +{ + struct combine_filter_data *d = filter_data; + size_t sub; + for (sub = 0; sub < d->nr; sub++) { + list_objects_filter__free(d->sub[sub].filter); + oidset_clear(&d->sub[sub].seen); + if (d->sub[sub].omits.set.size) + BUG("expected oidset to be cleared already"); + } + free(d->sub); +} + +static void add_all(struct oidset *dest, struct oidset *src) { + struct oidset_iter iter; + struct object_id *src_oid; + + oidset_iter_init(src, &iter); + while ((src_oid = oidset_iter_next(&iter)) != NULL) + oidset_insert(dest, src_oid); +} + +static void filter_combine__finalize_omits( + struct oidset *omits, + void *filter_data) +{ + struct combine_filter_data *d = filter_data; + size_t sub; + + for (sub = 0; sub < d->nr; sub++) { + add_all(omits, &d->sub[sub].omits); + oidset_clear(&d->sub[sub].omits); + } +} + +static void filter_combine__init( + struct list_objects_filter_options *filter_options, + struct filter* filter) +{ + struct combine_filter_data *d = xcalloc(1, sizeof(*d)); + size_t sub; + + d->nr = filter_options->sub_nr; + d->sub = xcalloc(d->nr, sizeof(*d->sub)); + for (sub = 0; sub < d->nr; sub++) + d->sub[sub].filter = list_objects_filter__init( + filter->omits ? &d->sub[sub].omits : NULL, + &filter_options->sub[sub]); + + filter->filter_data = d; + filter->filter_object_fn = filter_combine; + filter->free_fn = filter_combine__free; + filter->finalize_omits_fn = filter_combine__finalize_omits; +} + typedef void (*filter_init_fn)( struct list_objects_filter_options *filter_options, struct filter *filter); /* * Must match "enum list_objects_filter_choice". */ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, filter_trees_depth__init, filter_sparse_oid__init, + filter_combine__init, }; struct filter *list_objects_filter__init( struct oidset *omitted, struct list_objects_filter_options *filter_options) { struct filter *filter; filter_init_fn init_fn; assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT); @@ -529,13 +688,15 @@ enum list_objects_filter_result list_objects_filter__filter_object( */ if (filter_situation == LOFS_END_TREE) return 0; return LOFR_MARK_SEEN | LOFR_DO_SHOW; } void list_objects_filter__free(struct filter *filter) { if (!filter) return; + if (filter->finalize_omits_fn && filter->omits) + filter->finalize_omits_fn(filter->omits, filter->filter_data); filter->free_fn(filter->filter_data); free(filter); } diff --git a/list-objects-filter.h b/list-objects-filter.h index 6908954266..cfd784e203 100644 --- a/list-objects-filter.h +++ b/list-objects-filter.h @@ -55,32 +55,41 @@ enum list_objects_filter_result { }; enum list_objects_filter_situation { LOFS_BEGIN_TREE, LOFS_END_TREE, LOFS_BLOB }; struct filter; -/* Constructor for the set of defined list-objects filters. */ +/* + * Constructor for the set of defined list-objects filters. + * The `omitted` set is optional. It is populated with objects that the + * filter excludes. This set should not be considered finalized until + * after list_objects_filter__free is called on the returned `struct + * filter *`. + */ struct filter *list_objects_filter__init( struct oidset *omitted, struct list_objects_filter_options *filter_options); /* * Lets `filter` decide how to handle the `obj`. If `filter` is NULL, this * function behaves as expected if no filter is configured: all objects are * included. */ enum list_objects_filter_result list_objects_filter__filter_object( struct repository *r, enum list_objects_filter_situation filter_situation, struct object *obj, const char *pathname, const char *filename, struct filter *filter); -/* Destroys `filter`. Does nothing if `filter` is null. */ +/* + * Destroys `filter` and finalizes the `omitted` set, if present. Does + * nothing if `filter` is null. + */ void list_objects_filter__free(struct filter *filter); #endif /* LIST_OBJECTS_FILTER_H */ diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index acd7f5ab80..05d4f2e9c2 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -271,21 +271,33 @@ test_expect_success 'verify tree:0 includes trees in "filtered" output' ' # Make sure tree:0 does not iterate through any trees. test_expect_success 'verify skipping tree iteration when not collecting omits' ' GIT_TRACE=1 git -C r3 rev-list \ --objects --filter=tree:0 HEAD 2>filter_trace && grep "Skipping contents of tree [.][.][.]" filter_trace >actual && # One line for each commit traversed. test_line_count = 2 actual && # Make sure no other trees were considered besides the root. - ! grep "Skipping contents of tree [^.]" filter_trace + ! grep "Skipping contents of tree [^.]" filter_trace && + + # Try this again with "combine:". If both sub-filters are skipping + # trees, the composite filter should also skip trees. This is not + # important unless the user does combine:tree:X+tree:Y or another filter + # besides "tree:" is implemented in the future which can skip trees. + GIT_TRACE=1 git -C r3 rev-list \ + --objects --filter=combine:tree:1+tree:3 HEAD 2>filter_trace && + + # Only skip the dir1/ tree, which is shared between the two commits. + grep "Skipping contents of tree " filter_trace >actual && + test_write_lines "Skipping contents of tree dir1/..." >expected && + test_cmp expected actual ' # Test tree:# filters. expect_has () { commit=$1 && name=$2 && hash=$(git -C r3 rev-parse $commit:$name) && grep "^$hash $name$" actual @@ -323,20 +335,126 @@ test_expect_success 'verify tree:3 includes everything expected' ' expect_has HEAD dir1/sparse1 && expect_has HEAD dir1/sparse2 && expect_has HEAD pattern && expect_has HEAD sparse1 && expect_has HEAD sparse2 && # There are also 2 commit objects test_line_count = 10 actual ' +test_expect_success 'combine:... for a simple combination' ' + git -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \ + >actual && + + expect_has HEAD "" && + expect_has HEAD~1 "" && + expect_has HEAD dir1 && + + # There are also 2 commit objects + test_line_count = 5 actual +' + +test_expect_success 'combine:... with URL encoding' ' + git -C r3 rev-list --objects \ + --filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual && + + expect_has HEAD "" && + expect_has HEAD~1 "" && + expect_has HEAD dir1 && + + # There are also 2 commit objects + test_line_count = 5 actual +' + +expect_invalid_filter_spec () { + spec="$1" && + err="$2" && + + test_must_fail git -C r3 rev-list --objects --filter="$spec" HEAD \ + >actual 2>actual_stderr && + test_must_be_empty actual && + test_i18ngrep "$err" actual_stderr +} + +test_expect_success 'combine:... while URL-encoding things that should not be' ' + expect_invalid_filter_spec combine%3Atree:2+blob:none \ + "invalid filter-spec" +' + +test_expect_success 'combine: with nothing after the :' ' + expect_invalid_filter_spec combine: "expected something after combine:" +' + +test_expect_success 'parse error in first sub-filter in combine:' ' + expect_invalid_filter_spec combine:tree:asdf+blob:none \ + "expected .tree:." +' + +test_expect_success 'combine:... with non-encoded reserved chars' ' + expect_invalid_filter_spec combine:tree:2+sparse:@xyz \ + "must escape char in sub-filter-spec: .@." && + expect_invalid_filter_spec combine:tree:2+sparse:\` \ + "must escape char in sub-filter-spec: .\`." && + expect_invalid_filter_spec combine:tree:2+sparse:~abc \ + "must escape char in sub-filter-spec: .\~." +' + +test_expect_success 'validate err msg for "combine:+"' ' + expect_invalid_filter_spec combine:tree:2+ "expected .tree:." +' + +test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' ' + git -C r3 rev-list --objects --filter="combine:tree:2+bl%6Fb:n%6fne" \ + HEAD >actual && + test_line_count = 5 actual && + git -C r3 rev-list --objects --filter="combine:tree%3A2+blob%3anone" \ + HEAD >actual && + test_line_count = 5 actual && + git -C r3 rev-list --objects --filter="combine:tree:%30" HEAD >actual && + test_line_count = 2 actual && + git -C r3 rev-list --objects --filter="combine:tree:%39+blob:none" \ + HEAD >actual && + test_line_count = 5 actual +' + +test_expect_success 'add a sparse pattern blob whose path has reserved chars' ' + cp r3/pattern r3/pattern1+renamed% && + git -C r3 add pattern1+renamed% && + git -C r3 commit -m "add sparse pattern file with reserved chars" +' + +test_expect_success 'combine:... with more than two sub-filters' ' + git -C r3 rev-list --objects \ + --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \ + HEAD >actual && + + expect_has HEAD "" && + expect_has HEAD~1 "" && + expect_has HEAD~2 "" && + expect_has HEAD dir1 && + expect_has HEAD dir1/sparse1 && + expect_has HEAD dir1/sparse2 && + + # Should also have 3 commits + test_line_count = 9 actual && + + # Try again, this time making sure the last sub-filter is only + # URL-decoded once. + cp actual expect && + + git -C r3 rev-list --objects \ + --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \ + HEAD >actual && + test_cmp expect actual +' + # Test provisional omit collection logic with a repo that has objects appearing # at multiple depths - first deeper than the filter's threshold, then shallow. test_expect_success 'setup r4' ' git init r4 && echo foo > r4/foo && mkdir r4/subdir && echo bar > r4/subdir/bar && @@ -366,20 +484,51 @@ test_expect_success 'test tree:# filter provisional omit for blob and tree' ' test_expect_success 'verify skipping tree iteration when collecting omits' ' GIT_TRACE=1 git -C r4 rev-list --filter-print-omitted \ --objects --filter=tree:0 HEAD 2>filter_trace && grep "^Skipping contents of tree " filter_trace >actual && echo "Skipping contents of tree subdir/..." >expect && test_cmp expect actual ' +test_expect_success 'setup r5' ' + git init r5 && + mkdir -p r5/subdir && + + echo 1 >r5/short-root && + echo 12345 >r5/long-root && + echo a >r5/subdir/short-subdir && + echo abcde >r5/subdir/long-subdir && + + git -C r5 add short-root long-root subdir && + git -C r5 commit -m "commit msg" +' + +test_expect_success 'verify collecting omits in combined: filter' ' + # Note that this test guards against the naive implementation of simply + # giving both filters the same "omits" set and expecting it to + # automatically merge them. + git -C r5 rev-list --objects --quiet --filter-print-omitted \ + --filter=combine:tree:2+blob:limit=3 HEAD >actual && + + # Expect 0 trees/commits, 3 blobs omitted (all blobs except short-root) + omitted_1=$(echo 12345 | git hash-object --stdin) && + omitted_2=$(echo a | git hash-object --stdin) && + omitted_3=$(echo abcde | git hash-object --stdin) && + + grep ~$omitted_1 actual && + grep ~$omitted_2 actual && + grep ~$omitted_3 actual && + test_line_count = 3 actual +' + # Test tree: where a tree is iterated to twice - once where a subentry is # too deep to be included, and again where the blob inside it is shallow enough # to be included. This makes sure we don't use LOFR_MARK_SEEN incorrectly (we # can't use it because a tree can be iterated over again at a lower depth). test_expect_success 'tree: where we iterate over tree at two levels' ' git init r5 && mkdir -p r5/a/subdir/b && echo foo > r5/a/subdir/b/foo && diff --git a/url.c b/url.c index 1b8ef78cea..e34e5e7517 100644 --- a/url.c +++ b/url.c @@ -79,20 +79,26 @@ char *url_decode_mem(const char *url, int len) /* Skip protocol part if present */ if (colon && url < colon) { strbuf_add(&out, url, colon - url); len -= colon - url; url = colon; } return url_decode_internal(&url, len, NULL, &out, 0); } +char *url_percent_decode(const char *encoded) +{ + struct strbuf out = STRBUF_INIT; + return url_decode_internal(&encoded, strlen(encoded), NULL, &out, 0); +} + char *url_decode_parameter_name(const char **query) { struct strbuf out = STRBUF_INIT; return url_decode_internal(query, -1, "&=", &out, 1); } char *url_decode_parameter_value(const char **query) { struct strbuf out = STRBUF_INIT; return url_decode_internal(query, -1, "&", &out, 1); diff --git a/url.h b/url.h index 00b7d58c33..2a27c34277 100644 --- a/url.h +++ b/url.h @@ -1,16 +1,24 @@ #ifndef URL_H #define URL_H struct strbuf; int is_url(const char *url); int is_urlschemechar(int first_flag, int ch); char *url_decode(const char *url); char *url_decode_mem(const char *url, int len); + +/* + * Similar to the url_decode_{,mem} methods above, but doesn't assume there + * is a scheme followed by a : at the start of the string. Instead, %-sequences + * before any : are also parsed. + */ +char *url_percent_decode(const char *encoded); + char *url_decode_parameter_name(const char **query); char *url_decode_parameter_value(const char **query); void end_url_with_slash(struct strbuf *buf, const char *url); void str_end_url_with_slash(const char *url, char **dest); #endif /* URL_H */ From patchwork Thu Jun 27 22:54:09 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020809 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6D730112C for ; Thu, 27 Jun 2019 22:54:38 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6499528708 for ; Thu, 27 Jun 2019 22:54:38 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 58C522872E; Thu, 27 Jun 2019 22:54:38 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 01EA828708 for ; Thu, 27 Jun 2019 22:54:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726686AbfF0Wyh (ORCPT ); Thu, 27 Jun 2019 18:54:37 -0400 Received: from mail-pf1-f202.google.com ([209.85.210.202]:52353 "EHLO mail-pf1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wyg (ORCPT ); Thu, 27 Jun 2019 18:54:36 -0400 Received: by mail-pf1-f202.google.com with SMTP id a20so2472169pfn.19 for ; Thu, 27 Jun 2019 15:54:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=S+6irJWpzfC/LlA6nHhdC8F/0EGj3YmAm6cyz5TWzUc=; b=shWfYMsLBN+O3+d7ndH8WtlK7XXSpTw4N6HDg6Zh4yw9dw97aGFZT5YCGfDf7dc4He q3k1zsbW80fOaAy2kP5qxGnpOOW2nQX2hh3YjlDb1WBmKginF6bps1nC7ufD6FK0P6jm K2GjktZsdt6FtpAjRLI2wEJlxk2AxrXEjwCJP26p8RK40NN/q3mKNz31uXtcdyctP0iL 0Wv/myddHAc2nN0PIQgz/6SeWY2l/JDWSddeDskqHtIq8Ngxy2YDdOFOYd3Xre67oHef 628nlQGwSfBHv3Mr9H56sKRyftPdUJXR4cyPDWTMGncU276JkgHQKwc6jUmnxJYxKily DVqA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=S+6irJWpzfC/LlA6nHhdC8F/0EGj3YmAm6cyz5TWzUc=; b=X4aUEBtG7PufjpauxsaJNjaIAF6+KaTuYtQxfNT+RU22iKCznB6pvJPkoTAlFcypKC hN7FKIqza3HHw1rZ0gxYIvAjIo3FkJifR4wH48wZozrBHYWYrC/ZLR7ZkIkNhSW5gRp9 B2zSkvn2K8E/h3mmlb3kOjymxqYWAW9WuquW5oXSv/sbC/shYjAw5Rvk132QO0yw3mXa t4vFIPcJuj51tHubYtOfxLoDYlasJrwdkuJ1oO/6cV+IYnt8mnrbFRyKxX+O0IkDaFv7 Lt2JDThHIkmnnb5h4PJdfx4IHbAcewhddkYlEI+MOtHSnhK/CE00xC9+n/L0Fm6N8Ko1 8MrA== X-Gm-Message-State: APjAAAUodWOD9v3bzL6sMP0S5YRu6EV+5u6f5YxsKHx5QCFtI702xTXc vwDkRTry4c25VnDaUMD4BdkpswrknMNoV8a8WRpWA/rnBWkFFMi6FO5HKrWXreOrIddmxyaGXF5 lQLLhY5TVubZcwuwFdMmhV3UmYuU3wwFb04dbp2iNFdBTRYLZFFJAdBW+5vs= X-Google-Smtp-Source: APXvYqyyOM7sjr9rtj/wCYoy1zIOTDeu4ApPuredSohTkWDdk8SAV4bzN1KkB2WIU5Ts+G2qkjFS+2WnUhTx X-Received: by 2002:a63:221c:: with SMTP id i28mr2281427pgi.114.1561676074996; Thu, 27 Jun 2019 15:54:34 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:09 -0700 In-Reply-To: Message-Id: <32d1d81e5dc3b6ccbaa94266823e9c6c4657c191.1561675151.git.matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 05/10] list-objects-filter-options: move error check up From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Move the check that filter_options->choice is set to higher in the call stack. This can only be set when the gentle parse function is called from one of the two call sites. This is important because in an upcoming patch this may or may not be an error, and whether it is an error is only known to the parse_list_objects_filter function. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 75d0236ee2..5fe2814841 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -28,25 +28,22 @@ static int parse_combine_filter( * expand_list_objects_filter_spec() first). We also "intern" the arg for the * convenience of the current command. */ static int gently_parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg, struct strbuf *errbuf) { const char *v0; - if (filter_options->choice) { - strbuf_addstr( - errbuf, _("multiple filter-specs cannot be combined")); - return 1; - } + if (filter_options->choice) + BUG("filter_options already populated"); if (!strcmp(arg, "blob:none")) { filter_options->choice = LOFC_BLOB_NONE; return 0; } else if (skip_prefix(arg, "blob:limit=", &v0)) { if (git_parse_ulong(v0, &filter_options->blob_limit_value)) { filter_options->choice = LOFC_BLOB_LIMIT; return 0; } @@ -178,20 +175,22 @@ static int parse_combine_filter( list_objects_filter_release(filter_options); memset(filter_options, 0, sizeof(*filter_options)); } return result; } int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; + if (filter_options->choice) + die(_("multiple filter-specs cannot be combined")); filter_options->filter_spec = strdup(arg); if (gently_parse_list_objects_filter(filter_options, arg, &buf)) die("%s", buf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; From patchwork Thu Jun 27 22:54:10 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020811 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 91D06112C for ; Thu, 27 Jun 2019 22:54:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 860C028708 for ; Thu, 27 Jun 2019 22:54:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7A3762872E; Thu, 27 Jun 2019 22:54:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4857928708 for ; Thu, 27 Jun 2019 22:54:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726691AbfF0Wyj (ORCPT ); Thu, 27 Jun 2019 18:54:39 -0400 Received: from mail-pl1-f202.google.com ([209.85.214.202]:43473 "EHLO mail-pl1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wyj (ORCPT ); Thu, 27 Jun 2019 18:54:39 -0400 Received: by mail-pl1-f202.google.com with SMTP id t2so2249216plo.10 for ; Thu, 27 Jun 2019 15:54:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=w8OpjAkgKR4MVt+iYMLS42Uxq9w+oKN6ZPUqPgCyv8c=; b=JLdn0ruHh+3UoOZPBCqaxBCsGg/cmaVvkDZeIcVaKwoxKp4zRaLDArKKZUazSkK60I BvoIC5U6zz2YckdR+I58Iv16reffECAOGPcxcnGeR4bVqvn8lkCP5Q6T5M7Ao+ecW8cp 2mPUhTwwuSRBc8+zvG2GFAHP2m6/HUfrC6txmKtGL3P6VqGeCji5hbYtHBBFeYL7R618 x6wFJ99O+kF8MF1OHnHrhj+sc2yX10btB9yT4Gu5vDh8nReI44eW88byIpa1mrYetC4N Q4g9iI9wUPZTkBlM+B+KpGU7AvLkocveimwj7Tm7CzM5W+SGW12l2HZpyduPLgAYnFGi sk1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=w8OpjAkgKR4MVt+iYMLS42Uxq9w+oKN6ZPUqPgCyv8c=; b=MPWhI4cJZLBEt+W/mLel2IIFui1gw/KEaNKZWdwkF8ktE/4pLtMF99tl5s3veEvoqk kvMrQuEnTACB7ES6EWeE/MRiNT8xOjARoh6x/NVNUSgOrtwQr0c0kW0qfciNP6TW2HSC c21UPA27zcmfto9w9DxpIdQvXMIAV184xtqA5k84sx/zF309kvGn8mPV5c6S4kacEUeh RPC9gLkkXWxilBfuABdkVDR5/Tfi0rx2Y4vNtOGd0FX47SmpnCDJbEG1PiNmyZs+rc4J EsKRIB1qXG6UMlofR8ThiIsuK5hz5D5pyZnnsfkQtTDK3ENuuI7EGkDa9l2UME8NjJXC ZmuA== X-Gm-Message-State: APjAAAXfVmOjo9bZrbGAl8Nj2eHEw8TXSnKV/+/Ld0PI2zLFJsyFXIal /4d3Tp0PJVMthHtrfrezQHoxuzYHiiOdth1AvY38zF/T/H8TTGIs8tKXr1qjG6+b21L/lB3hBl9 4fr25GU2IeqZBcwiGogzH9z91CM4RnMsKBX+b2MqQxKWLxqlP7u0b39Iji2A= X-Google-Smtp-Source: APXvYqyglu2pBtgWv8c44wAb/S+iHzlUVZBnshmeEbRSVYx1tpKlsez3KekKzjDt4yQDZGzlmhmbpX76w3we X-Received: by 2002:a63:c006:: with SMTP id h6mr5940817pgg.285.1561676077697; Thu, 27 Jun 2019 15:54:37 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:10 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 06/10] list-objects-filter-options: make filter_spec a string_list From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de, Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Make the filter_spec string a string_list rather than a raw C string. The list of strings must be concatted together to make a complete filter_spec. A future patch will use this capability to build "combine:" filter specs gradually. A strbuf would seem to be a more natural choice for this object, but it unfortunately requires initialization besides just zero'ing out the memory. This results in all container structs, and all containers of those structs, etc., to also require initialization. Initializing them all would be more cumbersome that simply using a string_list, which behaves properly when its contents are zero'd. For the purposes of code simplification, change behavior in how filter specs are conveyed over the protocol: do not normalize the tree: filter specs since there should be no server in existence that supports tree:# but not tree:#k etc. Helped-by: Junio C Hamano Signed-off-by: Matthew DeVore --- builtin/clone.c | 8 ++--- builtin/fetch.c | 9 ++---- builtin/rev-list.c | 6 ++-- fetch-pack.c | 20 ++++-------- list-objects-filter-options.c | 50 ++++++++++++++++++++--------- list-objects-filter-options.h | 27 +++++++++++----- t/t6112-rev-list-filters-objects.sh | 7 ---- transport-helper.c | 10 ++---- upload-pack.c | 11 +++---- 9 files changed, 78 insertions(+), 70 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 5b9ebe9947..a693e6ca44 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1142,27 +1142,25 @@ int cmd_clone(int argc, const char **argv, const char *prefix) transport_set_option(transport, TRANS_OPT_FOLLOWTAGS, "1"); if (option_upload_pack) transport_set_option(transport, TRANS_OPT_UPLOADPACK, option_upload_pack); if (server_options.nr) transport->server_options = &server_options; if (filter_options.choice) { - struct strbuf expanded_filter_spec = STRBUF_INIT; - expand_list_objects_filter_spec(&filter_options, - &expanded_filter_spec); + const char *spec = + expand_list_objects_filter_spec(&filter_options); transport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, - expanded_filter_spec.buf); + spec); transport_set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); - strbuf_release(&expanded_filter_spec); } if (transport->smart_options && !deepen && !filter_options.choice) transport->smart_options->check_self_contained_and_connected = 1; argv_array_push(&ref_prefixes, "HEAD"); refspec_ref_prefixes(&remote->fetch, &ref_prefixes); if (option_branch) expand_ref_prefix(&ref_prefixes, option_branch); diff --git a/builtin/fetch.c b/builtin/fetch.c index 4ba63d5ac6..dee89e1a19 100644 --- a/builtin/fetch.c +++ b/builtin/fetch.c @@ -1181,27 +1181,24 @@ static struct transport *prepare_transport(struct remote *remote, int deepen) if (deepen && deepen_since) set_option(transport, TRANS_OPT_DEEPEN_SINCE, deepen_since); if (deepen && deepen_not.nr) set_option(transport, TRANS_OPT_DEEPEN_NOT, (const char *)&deepen_not); if (deepen_relative) set_option(transport, TRANS_OPT_DEEPEN_RELATIVE, "yes"); if (update_shallow) set_option(transport, TRANS_OPT_UPDATE_SHALLOW, "yes"); if (filter_options.choice) { - struct strbuf expanded_filter_spec = STRBUF_INIT; - expand_list_objects_filter_spec(&filter_options, - &expanded_filter_spec); - set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, - expanded_filter_spec.buf); + const char *spec = + expand_list_objects_filter_spec(&filter_options); + set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER, spec); set_option(transport, TRANS_OPT_FROM_PROMISOR, "1"); - strbuf_release(&expanded_filter_spec); } if (negotiation_tip.nr) { if (transport->smart_options) add_negotiation_tips(transport->smart_options); else warning("Ignoring --negotiation-tip because the protocol does not support it."); } return transport; } diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 660172b014..68acbe8fd2 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -459,22 +459,24 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) show_progress = arg; continue; } if (skip_prefix(arg, ("--" CL_ARG__FILTER "="), &arg)) { parse_list_objects_filter(&filter_options, arg); if (filter_options.choice && !revs.blob_objects) die(_("object filtering requires --objects")); if (filter_options.choice == LOFC_SPARSE_OID && !filter_options.sparse_oid_value) - die(_("invalid sparse value '%s'"), - filter_options.filter_spec); + die( + _("invalid sparse value '%s'"), + list_objects_filter_spec( + &filter_options)); continue; } if (!strcmp(arg, ("--no-" CL_ARG__FILTER))) { list_objects_filter_set_no_filter(&filter_options); continue; } if (!strcmp(arg, "--filter-print-omitted")) { arg_print_omitted = 1; continue; } diff --git a/fetch-pack.c b/fetch-pack.c index 1c10f54e78..72e13b0a1d 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -332,26 +332,23 @@ static int find_common(struct fetch_negotiator *negotiator, packet_buf_write(&req_buf, "deepen-since %"PRItime, max_age); } if (args->deepen_not) { int i; for (i = 0; i < args->deepen_not->nr; i++) { struct string_list_item *s = args->deepen_not->items + i; packet_buf_write(&req_buf, "deepen-not %s", s->string); } } if (server_supports_filtering && args->filter_options.choice) { - struct strbuf expanded_filter_spec = STRBUF_INIT; - expand_list_objects_filter_spec(&args->filter_options, - &expanded_filter_spec); - packet_buf_write(&req_buf, "filter %s", - expanded_filter_spec.buf); - strbuf_release(&expanded_filter_spec); + const char *spec = + expand_list_objects_filter_spec(&args->filter_options); + packet_buf_write(&req_buf, "filter %s", spec); } packet_buf_flush(&req_buf); state_len = req_buf.len; if (args->deepen) { const char *arg; struct object_id oid; send_request(args, fd[1], &req_buf); while (packet_reader_read(&reader) == PACKET_READ_NORMAL) { @@ -1092,21 +1089,21 @@ static int add_haves(struct fetch_negotiator *negotiator, ret = 1; } /* Increase haves to send on next round */ *haves_to_send = next_flush(1, *haves_to_send); return ret; } static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, - const struct fetch_pack_args *args, + struct fetch_pack_args *args, const struct ref *wants, struct oidset *common, int *haves_to_send, int *in_vain, int sideband_all) { int ret = 0; struct strbuf req_buf = STRBUF_INIT; if (server_supports_v2("fetch", 1)) packet_buf_write(&req_buf, "command=fetch"); if (server_supports_v2("agent", 0)) @@ -1133,27 +1130,24 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, /* Add shallow-info and deepen request */ if (server_supports_feature("fetch", "shallow", 0)) add_shallow_requests(&req_buf, args); else if (is_repository_shallow(the_repository) || args->deepen) die(_("Server does not support shallow requests")); /* Add filter */ if (server_supports_feature("fetch", "filter", 0) && args->filter_options.choice) { - struct strbuf expanded_filter_spec = STRBUF_INIT; + const char *spec = + expand_list_objects_filter_spec(&args->filter_options); print_verbose(args, _("Server supports filter")); - expand_list_objects_filter_spec(&args->filter_options, - &expanded_filter_spec); - packet_buf_write(&req_buf, "filter %s", - expanded_filter_spec.buf); - strbuf_release(&expanded_filter_spec); + packet_buf_write(&req_buf, "filter %s", spec); } else if (args->filter_options.choice) { warning("filtering not recognized by server, ignoring"); } /* add wants */ add_wants(args->no_dependents, wants, &req_buf); if (args->no_dependents) { packet_buf_write(&req_buf, "done"); ret = 1; diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 5fe2814841..01c0f13346 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -177,72 +177,89 @@ static int parse_combine_filter( } return result; } int parse_list_objects_filter(struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf buf = STRBUF_INIT; if (filter_options->choice) die(_("multiple filter-specs cannot be combined")); - filter_options->filter_spec = strdup(arg); + string_list_append(&filter_options->filter_spec, xstrdup(arg)); if (gently_parse_list_objects_filter(filter_options, arg, &buf)) die("%s", buf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; if (unset || !arg) { list_objects_filter_set_no_filter(filter_options); return 0; } return parse_list_objects_filter(filter_options, arg); } -void expand_list_objects_filter_spec( - const struct list_objects_filter_options *filter, - struct strbuf *expanded_spec) +const char *list_objects_filter_spec(struct list_objects_filter_options *filter) { - strbuf_init(expanded_spec, strlen(filter->filter_spec)); - if (filter->choice == LOFC_BLOB_LIMIT) - strbuf_addf(expanded_spec, "blob:limit=%lu", + if (!filter->filter_spec.nr) + BUG("no filter_spec available for this filter"); + if (filter->filter_spec.nr != 1) { + struct strbuf concatted = STRBUF_INIT; + strbuf_add_separated_string_list( + &concatted, "", &filter->filter_spec); + string_list_clear(&filter->filter_spec, /*free_util=*/0); + string_list_append( + &filter->filter_spec, strbuf_detach(&concatted, NULL)); + } + + return filter->filter_spec.items[0].string; +} + +const char *expand_list_objects_filter_spec( + struct list_objects_filter_options *filter) +{ + if (filter->choice == LOFC_BLOB_LIMIT) { + struct strbuf expanded_spec = STRBUF_INIT; + strbuf_addf(&expanded_spec, "blob:limit=%lu", filter->blob_limit_value); - else if (filter->choice == LOFC_TREE_DEPTH) - strbuf_addf(expanded_spec, "tree:%lu", - filter->tree_exclude_depth); - else - strbuf_addstr(expanded_spec, filter->filter_spec); + string_list_clear(&filter->filter_spec, /*free_util=*/0); + string_list_append( + &filter->filter_spec, + strbuf_detach(&expanded_spec, NULL)); + } + + return list_objects_filter_spec(filter); } void list_objects_filter_release( struct list_objects_filter_options *filter_options) { size_t sub; if (!filter_options) return; - free(filter_options->filter_spec); + string_list_clear(&filter_options->filter_spec, /*free_util=*/0); free(filter_options->sparse_oid_value); for (sub = 0; sub < filter_options->sub_nr; sub++) list_objects_filter_release(&filter_options->sub[sub]); free(filter_options->sub); memset(filter_options, 0, sizeof(*filter_options)); } void partial_clone_register( const char *remote, - const struct list_objects_filter_options *filter_options) + struct list_objects_filter_options *filter_options) { /* * Record the name of the partial clone remote in the * config and in the global variable -- the latter is * used throughout to indicate that partial clone is * enabled and to expect missing objects. */ if (repository_format_partial_clone && *repository_format_partial_clone && strcmp(remote, repository_format_partial_clone)) @@ -251,32 +268,33 @@ void partial_clone_register( git_config_set("core.repositoryformatversion", "1"); git_config_set("extensions.partialclone", remote); repository_format_partial_clone = xstrdup(remote); /* * Record the initial filter-spec in the config as * the default for subsequent fetches from this remote. */ core_partial_clone_filter_default = - xstrdup(filter_options->filter_spec); + xstrdup(expand_list_objects_filter_spec(filter_options)); git_config_set("core.partialclonefilter", core_partial_clone_filter_default); } void partial_clone_get_default_filter_spec( struct list_objects_filter_options *filter_options) { struct strbuf errbuf = STRBUF_INIT; /* * Parse default value, but silently ignore it if it is invalid. */ if (!core_partial_clone_filter_default) return; - filter_options->filter_spec = strdup(core_partial_clone_filter_default); + string_list_append(&filter_options->filter_spec, + core_partial_clone_filter_default); gently_parse_list_objects_filter(filter_options, core_partial_clone_filter_default, &errbuf); strbuf_release(&errbuf); } diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 789faef1e5..bb33303f9b 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -1,15 +1,15 @@ #ifndef LIST_OBJECTS_FILTER_OPTIONS_H #define LIST_OBJECTS_FILTER_OPTIONS_H #include "parse-options.h" -#include "strbuf.h" +#include "string-list.h" /* * The list of defined filters for list-objects. */ enum list_objects_filter_choice { LOFC_DISABLED = 0, LOFC_BLOB_NONE, LOFC_BLOB_LIMIT, LOFC_TREE_DEPTH, LOFC_SPARSE_OID, @@ -17,22 +17,24 @@ enum list_objects_filter_choice { LOFC__COUNT /* must be last */ }; struct list_objects_filter_options { /* * 'filter_spec' is the raw argument value given on the command line * or protocol request. (The part after the "--keyword=".) For * commands that launch filtering sub-processes, or for communication * over the network, don't use this value; use the result of * expand_list_objects_filter_spec() instead. + * To get the raw filter spec given by the user, use the result of + * list_objects_filter_spec(). */ - char *filter_spec; + struct string_list filter_spec; /* * 'choice' is determined by parsing the filter-spec. This indicates * the filtering algorithm to use. */ enum list_objects_filter_choice choice; /* * Choice is LOFC_DISABLED because "--no-filter" was requested. */ @@ -69,35 +71,44 @@ int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset); #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \ { OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_("args"), \ N_("object filtering"), 0, \ opt_parse_list_objects_filter } /* * Translates abbreviated numbers in the filter's filter_spec into their * fully-expanded forms (e.g., "limit:blob=1k" becomes "limit:blob=1024"). + * Returns a string owned by the list_objects_filter_options object. * - * This form should be used instead of the raw filter_spec field when - * communicating with a remote process or subprocess. + * This form should be used instead of the raw list_objects_filter_spec() + * value when communicating with a remote process or subprocess. */ -void expand_list_objects_filter_spec( - const struct list_objects_filter_options *filter, - struct strbuf *expanded_spec); +const char *expand_list_objects_filter_spec( + struct list_objects_filter_options *filter); + +/* + * Returns the filter spec string more or less in the form as the user + * entered it. This form of the filter_spec can be used in user-facing + * messages. Returns a string owned by the list_objects_filter_options + * object. + */ +const char *list_objects_filter_spec( + struct list_objects_filter_options *filter); void list_objects_filter_release( struct list_objects_filter_options *filter_options); static inline void list_objects_filter_set_no_filter( struct list_objects_filter_options *filter_options) { list_objects_filter_release(filter_options); filter_options->no_filter = 1; } void partial_clone_register( const char *remote, - const struct list_objects_filter_options *filter_options); + struct list_objects_filter_options *filter_options); void partial_clone_get_default_filter_spec( struct list_objects_filter_options *filter_options); #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */ diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index 05d4f2e9c2..27ba15719a 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -583,18 +583,11 @@ test_expect_success 'rev-list W/ missing=allow-any' ' # Test expansion of filter specs. test_expect_success 'expand blob limit in protocol' ' git -C r2 config --local uploadpack.allowfilter 1 && GIT_TRACE_PACKET="$(pwd)/trace" git -c protocol.version=2 clone \ --filter=blob:limit=1k "file://$(pwd)/r2" limit && ! grep "blob:limit=1k" trace && grep "blob:limit=1024" trace ' -test_expect_success 'expand tree depth limit in protocol' ' - GIT_TRACE_PACKET="$(pwd)/tree_trace" git -c protocol.version=2 clone \ - --filter=tree:0k "file://$(pwd)/r2" tree && - ! grep "tree:0k" tree_trace && - grep "tree:0" tree_trace -' - test_done diff --git a/transport-helper.c b/transport-helper.c index c7e17ec9cb..0a34544df0 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -675,27 +675,23 @@ static int fetch(struct transport *transport, data->transport_options.check_self_contained_and_connected) set_helper_option(transport, "check-connectivity", "true"); if (transport->cloning) set_helper_option(transport, "cloning", "true"); if (data->transport_options.update_shallow) set_helper_option(transport, "update-shallow", "true"); if (data->transport_options.filter_options.choice) { - struct strbuf expanded_filter_spec = STRBUF_INIT; - expand_list_objects_filter_spec( - &data->transport_options.filter_options, - &expanded_filter_spec); - set_helper_option(transport, "filter", - expanded_filter_spec.buf); - strbuf_release(&expanded_filter_spec); + const char *spec = expand_list_objects_filter_spec( + &data->transport_options.filter_options); + set_helper_option(transport, "filter", spec); } if (data->transport_options.negotiation_tips) warning("Ignoring --negotiation-tip because the protocol does not support it."); if (data->fetch) return fetch_with_fetch(transport, nr_heads, to_fetch); if (data->import) return fetch_with_import(transport, nr_heads, to_fetch); diff --git a/upload-pack.c b/upload-pack.c index 4d2129e7fc..d404d88941 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -133,32 +133,31 @@ static void create_pack_file(const struct object_array *have_obj, argv_array_push(&pack_objects.args, "--stdout"); if (shallow_nr) argv_array_push(&pack_objects.args, "--shallow"); if (!no_progress) argv_array_push(&pack_objects.args, "--progress"); if (use_ofs_delta) argv_array_push(&pack_objects.args, "--delta-base-offset"); if (use_include_tag) argv_array_push(&pack_objects.args, "--include-tag"); - if (filter_options.filter_spec) { - struct strbuf expanded_filter_spec = STRBUF_INIT; - expand_list_objects_filter_spec(&filter_options, - &expanded_filter_spec); + if (filter_options.choice) { + const char *spec = + expand_list_objects_filter_spec(&filter_options); if (pack_objects.use_shell) { struct strbuf buf = STRBUF_INIT; - sq_quote_buf(&buf, expanded_filter_spec.buf); + sq_quote_buf(&buf, spec); argv_array_pushf(&pack_objects.args, "--filter=%s", buf.buf); strbuf_release(&buf); } else { argv_array_pushf(&pack_objects.args, "--filter=%s", - expanded_filter_spec.buf); + spec); } } pack_objects.in = -1; pack_objects.out = -1; pack_objects.err = -1; if (start_command(&pack_objects)) die("git upload-pack: unable to fork git-pack-objects"); From patchwork Thu Jun 27 22:54:11 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020813 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3733B13B4 for ; Thu, 27 Jun 2019 22:54:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D95628708 for ; Thu, 27 Jun 2019 22:54:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 21B9E2872E; Thu, 27 Jun 2019 22:54:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 98FC928708 for ; Thu, 27 Jun 2019 22:54:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726702AbfF0Wyl (ORCPT ); Thu, 27 Jun 2019 18:54:41 -0400 Received: from mail-pf1-f202.google.com ([209.85.210.202]:45003 "EHLO mail-pf1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wyl (ORCPT ); Thu, 27 Jun 2019 18:54:41 -0400 Received: by mail-pf1-f202.google.com with SMTP id 5so2472073pff.11 for ; Thu, 27 Jun 2019 15:54:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eU/6vmDcEgz0Mza8pmxvvwqg4Td0kO5wtPJdy8PrWFM=; b=c1CUzd2jia793SEZO7xQOYIs25z0oMpkx16yZFJW4+6TTbeyeWVON1CbcmjzMVih78 I6FBUt/ePZaChxKFeiyTP75Y5V/WYQ8h6mF9b+mBH/CIJndPZEECblLw6bRL93WgKXgq upszl45FaSHTd9nS69z2w+3fL3/7zLpUZAbZySFz082uvv847ndRT6wkMwzEamz3MrJI PghipdmcraoIpndKyJibgk3QKNhTYcpx4R7tNp1oTOq+W3igOit3h6omNdamZjau+BSP DEND988LkyZXeN/849yGUX010/2Yt1SlLNkIq5wZ4eor4L38nR7I0ppHFjMWwTMiLbP3 6vIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eU/6vmDcEgz0Mza8pmxvvwqg4Td0kO5wtPJdy8PrWFM=; b=t4YUXEXQl72ECUlyNs6iaausKmuEVVN4T7loaPYfMKl0HdHlcIJpmk4b02HuM6I4Zn WDJ2QxIXOVCE+FQKOaxxaTnt1QndAMpvhNuBr3m5vzgFqXyGcYQ/2mrSYCrSxI4Oxd7A niqFsoS24if4e+gMfTaTZ0CXgIlViQ+16CYXetW8d1MmYnoQVIAX/FZNGZKxNq2J4j9p ICRT79KEypHJV+2enhx2iz+m9ceoNk8Zw0zjEhSP0KlZkAQ2OOEL71GAvh+m7mMIrOQ6 iSkC1cwT6G3R2VYlf6XPPKnhO44nGD+tzLoHiQwB7eDfx0/U1wCNSMQFJYa32cy16I3h W6qw== X-Gm-Message-State: APjAAAVWAozKY+aRJH9sFvRY5yEl/1MJHnhlYqSPv5XPrGTI0cfgtzgT piMjD/LwsnrlwRl2910vGCZJzcwmHYCYJsRg4GKaXvsZJHgrNLcPIsBdx1fmJlJpOEQXg2fgb67 Um/qKz9MNGKd/rh3rqxc/aoEtS3kSXOS6/cNkpxMYU49FZMzvBkSTTkh2V0M= X-Google-Smtp-Source: APXvYqwU8y5DgjKfvY9RplS6TZTHVEnvMlixD3+ThqGBG0Gfx6IXEHFvfb/t7m5+fjYpWtAKDrzPdC7Ljn/P X-Received: by 2002:a63:78ca:: with SMTP id t193mr6056955pgc.10.1561676080234; Thu, 27 Jun 2019 15:54:40 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:11 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 07/10] strbuf: give URL-encoding API a char predicate fn From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allow callers to specify exactly what characters need to be URL-encoded and which do not. This new API will be taken advantage of in a patch later in this set. Helped-by: Jeff King Signed-off-by: Matthew DeVore --- credential-store.c | 9 +++++---- http.c | 6 ++++-- strbuf.c | 15 ++++++++------- strbuf.h | 7 ++++++- 4 files changed, 23 insertions(+), 14 deletions(-) diff --git a/credential-store.c b/credential-store.c index ac295420dd..c010497cb2 100644 --- a/credential-store.c +++ b/credential-store.c @@ -65,29 +65,30 @@ static void rewrite_credential_file(const char *fn, struct credential *c, parse_credential_file(fn, c, NULL, print_line); if (commit_lock_file(&credential_lock) < 0) die_errno("unable to write credential store"); } static void store_credential_file(const char *fn, struct credential *c) { struct strbuf buf = STRBUF_INIT; strbuf_addf(&buf, "%s://", c->protocol); - strbuf_addstr_urlencode(&buf, c->username, 1); + strbuf_addstr_urlencode(&buf, c->username, is_rfc3986_unreserved); strbuf_addch(&buf, ':'); - strbuf_addstr_urlencode(&buf, c->password, 1); + strbuf_addstr_urlencode(&buf, c->password, is_rfc3986_unreserved); strbuf_addch(&buf, '@'); if (c->host) - strbuf_addstr_urlencode(&buf, c->host, 1); + strbuf_addstr_urlencode(&buf, c->host, is_rfc3986_unreserved); if (c->path) { strbuf_addch(&buf, '/'); - strbuf_addstr_urlencode(&buf, c->path, 0); + strbuf_addstr_urlencode(&buf, c->path, + is_rfc3986_reserved_or_unreserved); } rewrite_credential_file(fn, c, &buf); strbuf_release(&buf); } static void store_credential(const struct string_list *fns, struct credential *c) { struct string_list_item *fn; diff --git a/http.c b/http.c index 27aa0a3192..938b9e55af 100644 --- a/http.c +++ b/http.c @@ -506,23 +506,25 @@ static void var_override(const char **var, char *value) static void set_proxyauth_name_password(CURL *result) { #if LIBCURL_VERSION_NUM >= 0x071301 curl_easy_setopt(result, CURLOPT_PROXYUSERNAME, proxy_auth.username); curl_easy_setopt(result, CURLOPT_PROXYPASSWORD, proxy_auth.password); #else struct strbuf s = STRBUF_INIT; - strbuf_addstr_urlencode(&s, proxy_auth.username, 1); + strbuf_addstr_urlencode(&s, proxy_auth.username, + is_rfc3986_unreserved); strbuf_addch(&s, ':'); - strbuf_addstr_urlencode(&s, proxy_auth.password, 1); + strbuf_addstr_urlencode(&s, proxy_auth.password, + is_rfc3986_unreserved); curl_proxyuserpwd = strbuf_detach(&s, NULL); curl_easy_setopt(result, CURLOPT_PROXYUSERPWD, curl_proxyuserpwd); #endif } static void init_curl_proxy_auth(CURL *result) { if (proxy_auth.username) { if (!proxy_auth.password) credential_fill(&proxy_auth); diff --git a/strbuf.c b/strbuf.c index 0e18b259ce..60ab5144f2 100644 --- a/strbuf.c +++ b/strbuf.c @@ -767,55 +767,56 @@ void strbuf_addstr_xml_quoted(struct strbuf *buf, const char *s) case '&': strbuf_addstr(buf, "&"); break; case 0: return; } s++; } } -static int is_rfc3986_reserved(char ch) +int is_rfc3986_reserved_or_unreserved(char ch) { + if (is_rfc3986_unreserved(ch)) + return 1; switch (ch) { case '!': case '*': case '\'': case '(': case ')': case ';': case ':': case '@': case '&': case '=': case '+': case '$': case ',': case '/': case '?': case '#': case '[': case ']': return 1; } return 0; } -static int is_rfc3986_unreserved(char ch) +int is_rfc3986_unreserved(char ch) { return isalnum(ch) || ch == '-' || ch == '_' || ch == '.' || ch == '~'; } static void strbuf_add_urlencode(struct strbuf *sb, const char *s, size_t len, - int reserved) + char_predicate allow_unencoded_fn) { strbuf_grow(sb, len); while (len--) { char ch = *s++; - if (is_rfc3986_unreserved(ch) || - (!reserved && is_rfc3986_reserved(ch))) + if (allow_unencoded_fn(ch)) strbuf_addch(sb, ch); else strbuf_addf(sb, "%%%02x", (unsigned char)ch); } } void strbuf_addstr_urlencode(struct strbuf *sb, const char *s, - int reserved) + char_predicate allow_unencoded_fn) { - strbuf_add_urlencode(sb, s, strlen(s), reserved); + strbuf_add_urlencode(sb, s, strlen(s), allow_unencoded_fn); } void strbuf_humanise_bytes(struct strbuf *buf, off_t bytes) { if (bytes > 1 << 30) { strbuf_addf(buf, "%u.%2.2u GiB", (unsigned)(bytes >> 30), (unsigned)(bytes & ((1 << 30) - 1)) / 10737419); } else if (bytes > 1 << 20) { unsigned x = bytes + 5243; /* for rounding */ diff --git a/strbuf.h b/strbuf.h index c8d98dfb95..346d722492 100644 --- a/strbuf.h +++ b/strbuf.h @@ -659,22 +659,27 @@ void strbuf_branchname(struct strbuf *sb, const char *name, unsigned allowed); /* * Like strbuf_branchname() above, but confirm that the result is * syntactically valid to be used as a local branch name in refs/heads/. * * The return value is "0" if the result is valid, and "-1" otherwise. */ int strbuf_check_branch_ref(struct strbuf *sb, const char *name); +typedef int (*char_predicate)(char ch); + +int is_rfc3986_unreserved(char ch); +int is_rfc3986_reserved_or_unreserved(char ch); + void strbuf_addstr_urlencode(struct strbuf *sb, const char *name, - int reserved); + char_predicate allow_unencoded_fn); __attribute__((format (printf,1,2))) int printf_ln(const char *fmt, ...); __attribute__((format (printf,2,3))) int fprintf_ln(FILE *fp, const char *fmt, ...); char *xstrdup_tolower(const char *); char *xstrdup_toupper(const char *); /** From patchwork Thu Jun 27 22:54:12 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020815 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA96E13B4 for ; Thu, 27 Jun 2019 22:54:47 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9DBD828708 for ; Thu, 27 Jun 2019 22:54:47 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 915F62872E; Thu, 27 Jun 2019 22:54:47 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8840428708 for ; Thu, 27 Jun 2019 22:54:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726710AbfF0Wyo (ORCPT ); Thu, 27 Jun 2019 18:54:44 -0400 Received: from mail-yw1-f74.google.com ([209.85.161.74]:43028 "EHLO mail-yw1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wyo (ORCPT ); Thu, 27 Jun 2019 18:54:44 -0400 Received: by mail-yw1-f74.google.com with SMTP id b188so5232051ywb.10 for ; Thu, 27 Jun 2019 15:54:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=f+2G8GFv5PcFSKkHO+v21HrYHWYammS3+c5IzsnJjOw=; b=NGSJfJ7ZUOvPBUkbxIII3mhO6vL7AjdGjqyYV0uQLEr4GghBuKlV0GtOlOwd7e6kPp 61YVQWQGO/AlEnH6EkudDBpIKZLcr/Ur0blD/SkvkTuyqQyLAeU1AOi7y3SYrfB14OsU yM4Op3z6Sj8MPvwvEu+8oEHY1mdNpCbEc2YF0KIL8In90D2ffNaeGEAPZHgc2yVpZwl7 s1dLdfS4h0JI+Q8a+U+6YOP20B+BixLutIHlvKgWZl30HCDKma2VseO1ihdu9fWQwhZU vgVfqutSYfdXD7e3IHykH+5c/2gKb17rVRYSfTXaCFICoJ6/1+qxRYX90EDoN22n0MPb advg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=f+2G8GFv5PcFSKkHO+v21HrYHWYammS3+c5IzsnJjOw=; b=fjNqeexPuHPqcWE9gVy9PogjHcd3YeNL1oDlw9xZGP1s0tMoYshhs1UKCZDZP9ILw3 E+F6B00956zN26+ERLxBV4UBrpPFw21RNiHAUB0p0HpXneViqfAgOtlMNLZm+twNw0LZ T/dP3enqK3VCN1+FFthV4EMHx1TyPOUVyq8R3Z4JGevvq1dxmcehX07fnhn2kCMZ7eZA 1hU4GoO4iTrBbKWwe5DR+Tn77mYyc4YmSCWD8V8j6330d41pvJ5PgyuQn8TAS0iUIVCO jEgNg4lDlYkJ/TLvviQIGhyS19OCEnirm3K/JSdaQyGHy/b/jWm2nHbFfZnaiXHCjpBA i4WQ== X-Gm-Message-State: APjAAAWPBd44gshK0pnPzmVac6P1XcDvSOYCjLQjGdTO8+qaqFpJykoY 9jbj6v8h03De/qZePVm3mYCL5ZCykJE+Ww0Fqla3t7F4XGdnd+7go46xLnPkWH6lWdxExQ+/MQx 9zW6odPQw7LuxSJmCzXCVI0NIT1CTs0l+ZBtGDW12HLq70oddMUPMOYpkozI= X-Google-Smtp-Source: APXvYqxZ/TlYrMJpDfMyRGStK5blC/GcaIx7340xa+cqCVhaHQq22veiRAtToWLN+noaK7InMi+u4VJfpueN X-Received: by 2002:a0d:c0c5:: with SMTP id b188mr3792946ywd.271.1561676083012; Thu, 27 Jun 2019 15:54:43 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:12 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 08/10] list-objects-filter-options: allow mult. --filter From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de, Jeff Hostetler , Junio C Hamano Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Allow combining of multiple filters by simply repeating the --filter flag. Before this patch, the user had to combine them in a single flag somewhat awkwardly (e.g. --filter=combine:FOO+BAR), including URL-encoding the individual filters. To make this work, in the --filter flag parsing callback, rather than error out when we detect that the filter_options struct is already populated, we modify it in-place to contain the added sub-filter. The existing sub-filter becomes the lhs of the combined filter, and the next sub-filter becomes the rhs. We also have to URL-encode the LHS and RHS sub-filters. We can simplify the operation if the LHS is already a combine: filter. In that case, we just append the URL-encoded RHS sub-filter to the LHS spec to get the new spec. Helped-by: Emily Shaffer Helped-by: Jeff Hostetler Helped-by: Jeff King Helped-by: Junio C Hamano Signed-off-by: Matthew DeVore --- Documentation/rev-list-options.txt | 16 ++++++ list-objects-filter-options.c | 88 +++++++++++++++++++++++++++-- list-objects-filter-options.h | 11 ++++ t/t5616-partial-clone.sh | 19 +++++++ t/t6112-rev-list-filters-objects.sh | 46 +++++++++++++-- transport.c | 1 + upload-pack.c | 2 + 7 files changed, 173 insertions(+), 10 deletions(-) diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index 71a1fcc093..d1f080bf6d 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -731,20 +731,36 @@ at multiple depths in the commits traversed). =0 will not include any trees or blobs unless included explicitly in the command-line (or standard input when --stdin is used). =1 will include only the tree and blobs which are referenced directly by a commit reachable from or an explicitly-given object. =2 is like =1 while also including trees and blobs one more level removed from an explicitly-given commit or tree. + Note that the form '--filter=sparse:path=' that wants to read from an arbitrary path on the filesystem has been dropped for security reasons. ++ +Multiple '--filter=' flags can be specified to combine filters. Only +objects which are accepted by every filter are included. ++ +The form '--filter=combine:++...' can also be +used to combined several filters, but this is harder than just repeating +the '--filter' flag and is usually not necessary. Filters are joined by +'{plus}' and individual filters are %-encoded (i.e. URL-encoded). +Besides the '{plus}' and '%' characters, the following characters are +reserved and also must be encoded: `~!@#$^&*()[]{}\;",<>?`+'`+ +as well as all characters with ASCII code <= `0x20`, which includes +space and newline. ++ +Other arbitrary characters can also be encoded. For instance, +'combine:tree:3+blob:none' and 'combine:tree%3A3+blob%3Anone' are +equivalent. --no-filter:: Turn off any previous `--filter=` argument. --filter-print-omitted:: Only useful with `--filter=`; prints a list of the objects omitted by the filter. Object IDs are prefixed with a ``~'' character. --missing=:: A debug option to help with future "partial clone" development. diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 01c0f13346..2506dc8327 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -1,18 +1,19 @@ #include "cache.h" #include "commit.h" #include "config.h" #include "revision.h" #include "argv-array.h" #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "trace.h" #include "url.h" static int parse_combine_filter( struct list_objects_filter_options *filter_options, const char *arg, struct strbuf *errbuf); /* * Parse value of the argument to the "filter" keyword. * On the command line this looks like: @@ -171,29 +172,106 @@ static int parse_combine_filter( cleanup: strbuf_list_free(subspecs); if (result) { list_objects_filter_release(filter_options); memset(filter_options, 0, sizeof(*filter_options)); } return result; } -int parse_list_objects_filter(struct list_objects_filter_options *filter_options, - const char *arg) +static int allow_unencoded(char ch) +{ + if (ch <= ' ' || ch == '%' || ch == '+') + return 0; + return !strchr(RESERVED_NON_WS, ch); +} + +static void filter_spec_append_urlencode( + struct list_objects_filter_options *filter, const char *raw) { struct strbuf buf = STRBUF_INIT; + strbuf_addstr_urlencode(&buf, raw, allow_unencoded); + trace_printf("Add to combine filter-spec: %s\n", buf.buf); + string_list_append(&filter->filter_spec, strbuf_detach(&buf, NULL)); +} + +/* + * Changes filter_options into an equivalent LOFC_COMBINE filter options + * instance. Does not do anything if filter_options is already LOFC_COMBINE. + */ +static void transform_to_combine_type( + struct list_objects_filter_options *filter_options) +{ + assert(filter_options->choice); + if (filter_options->choice == LOFC_COMBINE) + return; + { + const int initial_sub_alloc = 2; + struct list_objects_filter_options *sub_array = + xcalloc(initial_sub_alloc, sizeof(*sub_array)); + sub_array[0] = *filter_options; + memset(filter_options, 0, sizeof(*filter_options)); + filter_options->sub = sub_array; + filter_options->sub_alloc = initial_sub_alloc; + } + filter_options->sub_nr = 1; + filter_options->choice = LOFC_COMBINE; + string_list_append(&filter_options->filter_spec, xstrdup("combine:")); + filter_spec_append_urlencode( + filter_options, + list_objects_filter_spec(&filter_options->sub[0])); + /* + * We don't need the filter_spec strings for subfilter specs, only the + * top level. + */ + string_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0); +} + +void list_objects_filter_die_if_populated( + struct list_objects_filter_options *filter_options) +{ if (filter_options->choice) die(_("multiple filter-specs cannot be combined")); - string_list_append(&filter_options->filter_spec, xstrdup(arg)); - if (gently_parse_list_objects_filter(filter_options, arg, &buf)) - die("%s", buf.buf); +} + +int parse_list_objects_filter( + struct list_objects_filter_options *filter_options, + const char *arg) +{ + struct strbuf errbuf = STRBUF_INIT; + int parse_error; + + if (!filter_options->choice) { + string_list_append(&filter_options->filter_spec, xstrdup(arg)); + + parse_error = gently_parse_list_objects_filter( + filter_options, arg, &errbuf); + } else { + /* + * Make filter_options an LOFC_COMBINE spec so we can trivially + * add subspecs to it. + */ + transform_to_combine_type(filter_options); + + string_list_append(&filter_options->filter_spec, xstrdup("+")); + filter_spec_append_urlencode(filter_options, arg); + ALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1, + filter_options->sub_alloc); + filter_options = &filter_options->sub[filter_options->sub_nr++]; + memset(filter_options, 0, sizeof(*filter_options)); + + parse_error = gently_parse_list_objects_filter( + filter_options, arg, &errbuf); + } + if (parse_error) + die("%s", errbuf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; if (unset || !arg) { list_objects_filter_set_no_filter(filter_options); diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index bb33303f9b..d8bc7e946e 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -56,20 +56,31 @@ struct list_objects_filter_options { struct list_objects_filter_options *sub; /* * END choice-specific parsed values. */ }; /* Normalized command line arguments */ #define CL_ARG__FILTER "filter" +void list_objects_filter_die_if_populated( + struct list_objects_filter_options *filter_options); + +/* + * Parses the filter spec string given by arg and either (1) simply places the + * result in filter_options if it is not yet populated or (2) combines it with + * the filter already in filter_options if it is already populated. In the case + * of (2), the filter specs are combined as if specified with 'combine:'. + * + * Dies and prints a user-facing message if an error occurs. + */ int parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg); int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset); #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \ { OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_("args"), \ N_("object filtering"), 0, \ diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index b91ef548f8..32b7d72f3c 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -201,20 +201,39 @@ test_expect_success 'use fsck before and after manually fetching a missing subtr test_line_count = 70 fetched_objects && awk -f print_1.awk fetched_objects | xargs -n1 git -C dst cat-file -t >fetched_types && sort -u fetched_types >unique_types.observed && test_write_lines blob commit tree >unique_types.expected && test_cmp unique_types.expected unique_types.observed ' +test_expect_success 'implicitly construct combine: filter with repeated flags' ' + GIT_TRACE=$(pwd)/trace git clone --bare \ + --filter=blob:none --filter=tree:1 \ + "file://$(pwd)/srv.bare" pc2 && + grep "trace:.* git pack-objects .*--filter=combine:blob:none+tree:1" \ + trace && + git -C pc2 rev-list --objects --missing=allow-any HEAD >objects && + + # We should have gotten some root trees. + grep " $" objects && + # Should not have gotten any non-root trees or blobs. + ! grep " ." objects && + + xargs -n 1 git -C pc2 cat-file -t types && + sort -u types >unique_types.actual && + test_write_lines commit tree >unique_types.expected && + test_cmp unique_types.expected unique_types.actual +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && test_commit -C src x && test_config -C src uploadpack.allowfilter 1 && test_config -C src uploadpack.allowanysha1inwant 1 && # Create a tag pointing to a blob. BLOB=$(echo blob-contents | git -C src hash-object --stdin -w) && git -C src tag myblob "$BLOB" && diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index 27ba15719a..de0e5a5d36 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -344,21 +344,30 @@ test_expect_success 'verify tree:3 includes everything expected' ' test_expect_success 'combine:... for a simple combination' ' git -C r3 rev-list --objects --filter=combine:tree:2+blob:none HEAD \ >actual && expect_has HEAD "" && expect_has HEAD~1 "" && expect_has HEAD dir1 && # There are also 2 commit objects - test_line_count = 5 actual + test_line_count = 5 actual && + + cp actual expected && + + # Try again using repeated --filter - this is equivalent to a manual + # combine with "combine:...+..." + git -C r3 rev-list --objects --filter=combine:tree:2 \ + --filter=blob:none HEAD >actual && + + test_cmp expected actual ' test_expect_success 'combine:... with URL encoding' ' git -C r3 rev-list --objects \ --filter=combine:tree%3a2+blob:%6Eon%65 HEAD >actual && expect_has HEAD "" && expect_has HEAD~1 "" && expect_has HEAD dir1 && @@ -410,24 +419,26 @@ test_expect_success 'combine:... with edge-case hex digits: Ff Aa 0 9' ' git -C r3 rev-list --objects --filter="combine:tree%3A2+blob%3anone" \ HEAD >actual && test_line_count = 5 actual && git -C r3 rev-list --objects --filter="combine:tree:%30" HEAD >actual && test_line_count = 2 actual && git -C r3 rev-list --objects --filter="combine:tree:%39+blob:none" \ HEAD >actual && test_line_count = 5 actual ' -test_expect_success 'add a sparse pattern blob whose path has reserved chars' ' +test_expect_success 'add sparse pattern blobs whose paths have reserved chars' ' cp r3/pattern r3/pattern1+renamed% && - git -C r3 add pattern1+renamed% && - git -C r3 commit -m "add sparse pattern file with reserved chars" + cp r3/pattern "r3/p;at%ter+n" && + cp r3/pattern r3/^~pattern && + git -C r3 add pattern1+renamed% "p;at%ter+n" ^~pattern && + git -C r3 commit -m "add sparse pattern files with reserved chars" ' test_expect_success 'combine:... with more than two sub-filters' ' git -C r3 rev-list --objects \ --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern \ HEAD >actual && expect_has HEAD "" && expect_has HEAD~1 "" && expect_has HEAD~2 "" && @@ -438,21 +449,46 @@ test_expect_success 'combine:... with more than two sub-filters' ' # Should also have 3 commits test_line_count = 9 actual && # Try again, this time making sure the last sub-filter is only # URL-decoded once. cp actual expect && git -C r3 rev-list --objects \ --filter=combine:tree:3+blob:limit=40+sparse:oid=master:pattern1%2brenamed%25 \ HEAD >actual && - test_cmp expect actual + test_cmp expect actual && + + # Use the same composite filter again, but with a pattern file name that + # requires encoding multiple characters, and use implicit filter + # combining. + test_when_finished "rm -f trace1" && + GIT_TRACE=$(pwd)/trace1 git -C r3 rev-list --objects \ + --filter=tree:3 --filter=blob:limit=40 \ + --filter=sparse:oid="master:p;at%ter+n" \ + HEAD >actual && + + test_cmp expect actual && + grep "Add to combine filter-spec: sparse:oid=master:p%3bat%25ter%2bn" \ + trace1 && + + # Repeat the above test, but this time, the characters to encode are in + # the LHS of the combined filter. + test_when_finished "rm -f trace2" && + GIT_TRACE=$(pwd)/trace2 git -C r3 rev-list --objects \ + --filter=sparse:oid=master:^~pattern \ + --filter=tree:3 --filter=blob:limit=40 \ + HEAD >actual && + + test_cmp expect actual && + grep "Add to combine filter-spec: sparse:oid=master:%5e%7epattern" \ + trace2 ' # Test provisional omit collection logic with a repo that has objects appearing # at multiple depths - first deeper than the filter's threshold, then shallow. test_expect_success 'setup r4' ' git init r4 && echo foo > r4/foo && mkdir r4/subdir && diff --git a/transport.c b/transport.c index f1fcd2c4b0..ee7dd1c062 100644 --- a/transport.c +++ b/transport.c @@ -217,20 +217,21 @@ static int set_git_option(struct git_transport_options *opts, } else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) { opts->deepen_relative = !!value; return 0; } else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) { opts->from_promisor = !!value; return 0; } else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) { opts->no_dependents = !!value; return 0; } else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) { + list_objects_filter_die_if_populated(&opts->filter_options); parse_list_objects_filter(&opts->filter_options, value); return 0; } return 1; } static int connect_setup(struct transport *transport, int for_push) { struct git_transport_data *data = transport->data; int flags = transport->verbose > 0 ? CONNECT_VERBOSE : 0; diff --git a/upload-pack.c b/upload-pack.c index d404d88941..f8a76ebda3 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -876,20 +876,21 @@ static void receive_needs(struct packet_reader *reader, struct object_array *wan if (process_deepen(reader->line, &depth)) continue; if (process_deepen_since(reader->line, &deepen_since, &deepen_rev_list)) continue; if (process_deepen_not(reader->line, &deepen_not, &deepen_rev_list)) continue; if (skip_prefix(reader->line, "filter ", &arg)) { if (!filter_capability_requested) die("git upload-pack: filtering capability not negotiated"); + list_objects_filter_die_if_populated(&filter_options); parse_list_objects_filter(&filter_options, arg); continue; } if (!skip_prefix(reader->line, "want ", &arg) || parse_oid_hex(arg, &oid_buf, &features)) die("git upload-pack: protocol error, " "expected to get object ID, not '%s'", reader->line); if (parse_feature_request(features, "deepen-relative")) @@ -1297,20 +1298,21 @@ static void process_args(struct packet_reader *request, continue; if (process_deepen_not(arg, &data->deepen_not, &data->deepen_rev_list)) continue; if (!strcmp(arg, "deepen-relative")) { data->deepen_relative = 1; continue; } if (allow_filter && skip_prefix(arg, "filter ", &p)) { + list_objects_filter_die_if_populated(&filter_options); parse_list_objects_filter(&filter_options, p); continue; } if ((git_env_bool("GIT_TEST_SIDEBAND_ALL", 0) || allow_sideband_all) && !strcmp(arg, "sideband-all")) { data->writer.use_sideband = 1; continue; } From patchwork Thu Jun 27 22:54:13 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020817 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B75FF112C for ; Thu, 27 Jun 2019 22:54:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AE12D28708 for ; Thu, 27 Jun 2019 22:54:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A22532872E; Thu, 27 Jun 2019 22:54:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 32EE428708 for ; Thu, 27 Jun 2019 22:54:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726713AbfF0Wyr (ORCPT ); Thu, 27 Jun 2019 18:54:47 -0400 Received: from mail-pl1-f201.google.com ([209.85.214.201]:33676 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wyq (ORCPT ); Thu, 27 Jun 2019 18:54:46 -0400 Received: by mail-pl1-f201.google.com with SMTP id f2so2261409plr.0 for ; Thu, 27 Jun 2019 15:54:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=w7vpVJ8s7clxjwwCepQCPd9uZ8PX6ZSIFiVtWpSl6Js=; b=vge+IU4wyuDP4g2DBPKqveOG2sOvuNCxX/tvaTAcEnwwIJLQgkoIkoP0hBU2f4bSh4 /TdSwgoeYmSROlc/fEId5Xh32zMEOuR3QqfouJ40X0JeDxCmGenmnVQIKRz1KGJOjHtb ZVcoNF0eARv0EthZraTtiw/EI17DsjlGSRykM+90MeaVRtS3GB9Pev2hT6IREXqk+YoJ mh9NPbmGVvlJyq/3H6+ZYqPvHwbfizDa0oBwmvEjqFTtLeCuinDaA0PkdotjAx5stkxd EJl1x5hU2uYhUzfwXxnYRZVxDq05KzU6ZmNlLXUAobJyMmkQ3hDz7yzoLFDj9hmGDqrV 9jWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=w7vpVJ8s7clxjwwCepQCPd9uZ8PX6ZSIFiVtWpSl6Js=; b=IeK75BNIajin8yM6Vk82+9WtUSVmPxk3+ntH9fBRiKCDEoat3nYlPVcjkBqAHz1izu MFtoL79p8IPykTZvaBur0fu0ryQHU8l6QMR4IZlADANXEWro0fQEdE3Hs+0ngaClmg9l W/sLHM8lY42at3wklQvDyw721pPm7D2WmV0mLh1YOlxmYugCQ2+yEmOji3Fc4YhXKFGd WtHHwrqniIo6rXRnni2R+MAm9suRGXkjyR7dMAuQyLbYX/QJh3SzceBmDKTkDTzmriqO LUvgxtqhsrCw4bgRyNJF9uQGpZAGN/ube1A2+nyER9ap4NcPZhh6FUk3W+dMBG3ZtYpD egtA== X-Gm-Message-State: APjAAAVZAdxKn1bSzY5NLa7I3+kHBXHuRg2d7eEFt0kl3EUd/2Uzcko6 8APlLP70QYKTt98TTHhJJy8k247Lu59FON8WIPBP+1uy3dCZ33MuohHEeJjkgfFdZXrNl+3ENK1 jjDfmxt8sFDkNhak6LPV6fqfyWGIB1c++sQT5CfpDYE80N0OXDj5GyEy/CEk= X-Google-Smtp-Source: APXvYqxgPQj9C3FDSFSntVO5HSuFzpGr4Mln/a9BEohb8mASktI6OGoNFCj65RT3VgntMcUrx/omnxUusOL4 X-Received: by 2002:a63:607:: with SMTP id 7mr3753691pgg.240.1561676085479; Thu, 27 Jun 2019 15:54:45 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:13 -0700 In-Reply-To: Message-Id: <0af31656ccceefbf9eff82506b89c613613cef94.1561675151.git.matvore@google.com> Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 09/10] list-objects-filter-options: clean up use of ALLOC_GROW From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Introduce a new macro ALLOC_GROW_BY which automatically zeros the added array elements and takes care of updating the nr value. Use the macro in code introduced earlier in this patchset. Signed-off-by: Matthew DeVore --- cache.h | 22 ++++++++++++++++++++++ list-objects-filter-options.c | 17 +++++++---------- 2 files changed, 29 insertions(+), 10 deletions(-) diff --git a/cache.h b/cache.h index bf20337ef4..cf5d70c196 100644 --- a/cache.h +++ b/cache.h @@ -653,33 +653,55 @@ int init_db(const char *git_dir, const char *real_git_dir, void sanitize_stdfds(void); int daemonize(void); #define alloc_nr(x) (((x)+16)*3/2) /* * Realloc the buffer pointed at by variable 'x' so that it can hold * at least 'nr' entries; the number of entries currently allocated * is 'alloc', using the standard growing factor alloc_nr() macro. * + * Consider using ALLOC_GROW_BY instead of ALLOC_GROW as it has some + * added niceties. + * * DO NOT USE any expression with side-effect for 'x', 'nr', or 'alloc'. */ #define ALLOC_GROW(x, nr, alloc) \ do { \ if ((nr) > alloc) { \ if (alloc_nr(alloc) < (nr)) \ alloc = (nr); \ else \ alloc = alloc_nr(alloc); \ REALLOC_ARRAY(x, alloc); \ } \ } while (0) +/* + * Similar to ALLOC_GROW but handles updating of the nr value and + * zeroing the bytes of the newly-grown array elements. + * + * DO NOT USE any expression with side-effect for any of the + * arguments. + */ +#define ALLOC_GROW_BY(x, nr, increase, alloc) \ + do { \ + if (increase) { \ + size_t new_nr = nr + (increase); \ + if (new_nr < nr) \ + BUG("negative growth in ALLOC_GROW_BY"); \ + ALLOC_GROW(x, new_nr, alloc); \ + memset((x) + nr, 0, sizeof(*(x)) * (increase)); \ + nr = new_nr; \ + } \ + } while (0) + /* Initialize and use the cache information */ struct lock_file; void preload_index(struct index_state *index, const struct pathspec *pathspec, unsigned int refresh_flags); int do_read_index(struct index_state *istate, const char *path, int must_exist); /* for testting only! */ int read_index_from(struct index_state *, const char *path, const char *gitdir); int is_index_unborn(struct index_state *); diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 2506dc8327..44bc1153d1 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -113,28 +113,26 @@ static int has_reserved_character( } return 0; } static int parse_combine_subfilter( struct list_objects_filter_options *filter_options, struct strbuf *subspec, struct strbuf *errbuf) { - size_t new_index = filter_options->sub_nr++; + size_t new_index = filter_options->sub_nr; char *decoded; int result; - ALLOC_GROW(filter_options->sub, filter_options->sub_nr, - filter_options->sub_alloc); - memset(&filter_options->sub[new_index], 0, - sizeof(*filter_options->sub)); + ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1, + filter_options->sub_alloc); decoded = url_percent_decode(subspec->buf); result = has_reserved_character(subspec, errbuf) || gently_parse_list_objects_filter( &filter_options->sub[new_index], decoded, errbuf); free(decoded); return result; } @@ -248,27 +246,26 @@ int parse_list_objects_filter( filter_options, arg, &errbuf); } else { /* * Make filter_options an LOFC_COMBINE spec so we can trivially * add subspecs to it. */ transform_to_combine_type(filter_options); string_list_append(&filter_options->filter_spec, xstrdup("+")); filter_spec_append_urlencode(filter_options, arg); - ALLOC_GROW(filter_options->sub, filter_options->sub_nr + 1, - filter_options->sub_alloc); - filter_options = &filter_options->sub[filter_options->sub_nr++]; - memset(filter_options, 0, sizeof(*filter_options)); + ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1, + filter_options->sub_alloc); parse_error = gently_parse_list_objects_filter( - filter_options, arg, &errbuf); + &filter_options->sub[filter_options->sub_nr - 1], arg, + &errbuf); } if (parse_error) die("%s", errbuf.buf); return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; From patchwork Thu Jun 27 22:54:14 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 11020819 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 394F213B4 for ; Thu, 27 Jun 2019 22:54:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2F97E28708 for ; Thu, 27 Jun 2019 22:54:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 23BC22872E; Thu, 27 Jun 2019 22:54:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B61A128708 for ; Thu, 27 Jun 2019 22:54:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726720AbfF0Wyt (ORCPT ); Thu, 27 Jun 2019 18:54:49 -0400 Received: from mail-ua1-f74.google.com ([209.85.222.74]:48589 "EHLO mail-ua1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726599AbfF0Wyt (ORCPT ); Thu, 27 Jun 2019 18:54:49 -0400 Received: by mail-ua1-f74.google.com with SMTP id s14so613708uap.15 for ; Thu, 27 Jun 2019 15:54:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=e3qcG3oNQpxCOKE5EH7WjZS4k/Idsi7oJfpaycVLTxc=; b=LnORr4f+Jy+nfosQ7tIYaMeqJwhlsbzGuREUk4HaA94N7nTgOAwYVOELuco0lGsTs6 JX9BlZy4YHuEouGiqi4VD2NL++ZrzXEc5ix52hRqKGmyBBS/OMdQb0T4j9K4sAhppXC2 R+B68mJPqNbBPZLn3BCflI+V3OH5ZaU/FPsrRYLEz4V84qUtxUlXQTXpL8WA8xs4pqLE NOsKg6yFb4rFQbZG0uHlPI41bnUR71UTSsIKix6PwDMTzbs5X4r2wGkNwosN9X4AuhU4 /qWqHF9oVqaMmgPYSDee34Cz7icRhxxQIQHY5cd33W9Gi7fY82Gz0pk0akg7ygvhIt+9 7mag== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=e3qcG3oNQpxCOKE5EH7WjZS4k/Idsi7oJfpaycVLTxc=; b=bz4Ymv4X0CjFkir4FxupBKQNY6flHutxfUYImzxMDpD/04/OEWUoUJ0dzecYgMsNYq J1CAghuh0w9F5+SA8iwU9fxtLfK7ih5A7xDkcgCUie6j0qNC+UIAsnRmIpDBffE/CrYJ LQSEeMG7oHjiQug96m/8YF3OQrErmF7JUod8dxbIwsxwjr4XniL8wKor7TzFkKBIhj8f o+XUQQwlCdaT75pnwZkQK0or46/xiFoWrRrufY9TaTZHcBnKlx5441TRNAh1KVh9mwH1 s6HOFdocJ4WwzXHOaZcxu6cIrOt+gdKL5BFgKrngoaBvWtTMypTT660RVor7ih1r5GeU jg9g== X-Gm-Message-State: APjAAAXkZiitWl2lpHsiwSS9SkCWzbsNz0MbGOK7NJqO9LNxHdizLpps hALhn426pNQ64UBiRs2qCLy1iO7pXmToYRQP2tPcqZBiaIordQuOah9QGtUzKccENo5Chj4HYuJ v8K+7EVDjl6yzizByBKzRgDJOWmX9BF5Oo5gHUL3d1UUbSPjyeekuv2+nbjQ= X-Google-Smtp-Source: APXvYqz8ybpFH295rhV5ax0xF6flDm2k9Bs/3yzRvgpa+wptG7Gov3sfQDUhMbbp1YuNkMHDoxrikL7cil8z X-Received: by 2002:a1f:2909:: with SMTP id p9mr2554669vkp.23.1561676088288; Thu, 27 Jun 2019 15:54:48 -0700 (PDT) Date: Thu, 27 Jun 2019 15:54:14 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: <20190601003603.90794-1-matvore@google.com> X-Mailer: git-send-email 2.22.0.410.gd8fdbe21b5-goog Subject: [PATCH v5 10/10] list-objects-filter-options: make parser void From: Matthew DeVore To: git@vger.kernel.org, jonathantanmy@google.com, jrn@google.com, dstolee@microsoft.com, jeffhost@microsoft.com, jrnieder@gmail.com, pclouds@gmail.com, peff@peff.net, emilyshaffer@google.com Cc: Matthew DeVore , matvore@comcast.net, Johannes.Schindelin@gmx.de Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This function always returns 0, so make it return void instead. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 12 +++++------- list-objects-filter-options.h | 2 +- 2 files changed, 6 insertions(+), 8 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index 44bc1153d1..ba1425cb4a 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -225,21 +225,21 @@ static void transform_to_combine_type( string_list_clear(&filter_options->sub[0].filter_spec, /*free_util=*/0); } void list_objects_filter_die_if_populated( struct list_objects_filter_options *filter_options) { if (filter_options->choice) die(_("multiple filter-specs cannot be combined")); } -int parse_list_objects_filter( +void parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg) { struct strbuf errbuf = STRBUF_INIT; int parse_error; if (!filter_options->choice) { string_list_append(&filter_options->filter_spec, xstrdup(arg)); parse_error = gently_parse_list_objects_filter( @@ -255,34 +255,32 @@ int parse_list_objects_filter( filter_spec_append_urlencode(filter_options, arg); ALLOC_GROW_BY(filter_options->sub, filter_options->sub_nr, 1, filter_options->sub_alloc); parse_error = gently_parse_list_objects_filter( &filter_options->sub[filter_options->sub_nr - 1], arg, &errbuf); } if (parse_error) die("%s", errbuf.buf); - return 0; } int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset) { struct list_objects_filter_options *filter_options = opt->value; - if (unset || !arg) { + if (unset || !arg) list_objects_filter_set_no_filter(filter_options); - return 0; - } - - return parse_list_objects_filter(filter_options, arg); + else + parse_list_objects_filter(filter_options, arg); + return 0; } const char *list_objects_filter_spec(struct list_objects_filter_options *filter) { if (!filter->filter_spec.nr) BUG("no filter_spec available for this filter"); if (filter->filter_spec.nr != 1) { struct strbuf concatted = STRBUF_INIT; strbuf_add_separated_string_list( &concatted, "", &filter->filter_spec); diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index d8bc7e946e..db37dfb34a 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -67,21 +67,21 @@ void list_objects_filter_die_if_populated( struct list_objects_filter_options *filter_options); /* * Parses the filter spec string given by arg and either (1) simply places the * result in filter_options if it is not yet populated or (2) combines it with * the filter already in filter_options if it is already populated. In the case * of (2), the filter specs are combined as if specified with 'combine:'. * * Dies and prints a user-facing message if an error occurs. */ -int parse_list_objects_filter( +void parse_list_objects_filter( struct list_objects_filter_options *filter_options, const char *arg); int opt_parse_list_objects_filter(const struct option *opt, const char *arg, int unset); #define OPT_PARSE_LIST_OBJECTS_FILTER(fo) \ { OPTION_CALLBACK, 0, CL_ARG__FILTER, fo, N_("args"), \ N_("object filtering"), 0, \ opt_parse_list_objects_filter }