From patchwork Wed Oct 3 19:52:12 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625251 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B38B813BB for ; Wed, 3 Oct 2018 19:52:40 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A35BC2904B for ; Wed, 3 Oct 2018 19:52:40 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 97B2F29050; Wed, 3 Oct 2018 19:52:40 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 90B282904E for ; Wed, 3 Oct 2018 19:52:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727180AbeJDCm1 (ORCPT ); Wed, 3 Oct 2018 22:42:27 -0400 Received: from mail-pl1-f201.google.com ([209.85.214.201]:49775 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCm1 (ORCPT ); Wed, 3 Oct 2018 22:42:27 -0400 Received: by mail-pl1-f201.google.com with SMTP id y7-v6so6432663plp.16 for ; Wed, 03 Oct 2018 12:52:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8Uppbwd0luknABY1misofZPsHdAAq25GdpUtk/w+bec=; b=g+VNk3z58kdmQbAeVPHlWOTPZubf2YmsypYWKb5HaiiSH623GhMMPZ3IMHu4xNpAOT Mdsb//AdK7f9l09QIe9+bAKnLGYuonCreFdWzdk5ekfwY3o3NPWmT4It1eZKjUTf5gsD cmeiYbsOhNfK8tXki+xuTHVbn9nIrk7PkulfTjf9pMgCyyqwrGsmm7ldY08FgZGDfvTg dxYKgjFBccpd0V9sg/QDunJW9M2VkRDLdcSAuWmAkeK75520TgfR6ulbTIND0bGOcnpa EXdAIPOVK80G4Lk9YSMyBJYxvznUSESq5tcqAAtcpYZduOrBf4xqDsCN7O7P8e5QWYfJ 85fQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8Uppbwd0luknABY1misofZPsHdAAq25GdpUtk/w+bec=; b=pSyZpVgmY4gDWdKYUJMrNc1XeQhO70aXdDXZvGXzPQS26YSU20rtBmeCbOxnP+nuBg 0tMnUiTUe/bmkmmSguFBtWn7RkF4fNmkSYeSTz+UGSNEx25jwgSQDGFIWn6ul7jcjHcP x9Qlksm9P1n5TF2Da1Q8qAp9flHsruT9cU4SbxYYgdkl+VsAIEZUtQ/QEPNlIGHOpJdE XltuP120EaGrzxVQVsE0VZ2zm8/HUwbUmfMeVoDx1fivzVlICnbBZlQsNXSnUfBiB+Xn g8+NQcHcn8D5/XvmMtD0aa9S61wCat+LtwPX15oLxMnGLdqDjrYxyFJMTtld3qqBQeaG fmWA== X-Gm-Message-State: ABuFfogM30ZRlGTPfQgr9/8SL1eZB3d8sPhTTZgBD00DVErzdkhLNkud osgGMbDUQp+uGl99X77KU6RMrP3Wf/Gdloo6JYlSxMdhgVUAEUFWMcXC8d8O5FdGixMauEfJfVt kS8Hkt8cB9C0JHfPDq9Yg6gwPnJAWQs2TNA/908HCu8STKai459dWY2NumnA= X-Google-Smtp-Source: ACcGV630e4bWQ5snuKljvcSXDQneh/u+jsd8eDujM4e3eq+2wlmtTA73EJNoAAhgr9cuka8XE753fg830rcL X-Received: by 2002:a65:4009:: with SMTP id f9-v6mr1189120pgp.63.1538596357367; Wed, 03 Oct 2018 12:52:37 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:12 -0700 In-Reply-To: Message-Id: <3bff094363c7887bfe6547e520cbe83097cf6f73.1538592829.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 1/8] list-objects: store common func args in struct From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This will make utility functions easier to create, as done by the next patch. Signed-off-by: Matthew DeVore --- list-objects.c | 158 +++++++++++++++++++++++-------------------------- 1 file changed, 74 insertions(+), 84 deletions(-) diff --git a/list-objects.c b/list-objects.c index c99c47ac1..584518a3f 100644 --- a/list-objects.c +++ b/list-objects.c @@ -12,20 +12,25 @@ #include "packfile.h" #include "object-store.h" -static void process_blob(struct rev_info *revs, +struct traversal_context { + struct rev_info *revs; + show_object_fn show_object; + show_commit_fn show_commit; + void *show_data; + filter_object_fn filter_fn; + void *filter_data; +}; + +static void process_blob(struct traversal_context *ctx, struct blob *blob, - show_object_fn show, struct strbuf *path, - const char *name, - void *cb_data, - filter_object_fn filter_fn, - void *filter_data) + const char *name) { struct object *obj = &blob->object; size_t pathlen; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; - if (!revs->blob_objects) + if (!ctx->revs->blob_objects) return; if (!obj) die("bad blob object"); @@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs, * may cause the actual filter to report an incomplete list * of missing objects. */ - if (revs->exclude_promisor_objects && + if (ctx->revs->exclude_promisor_objects && !has_object_file(&obj->oid) && is_promisor_object(&obj->oid)) return; pathlen = path->len; strbuf_addstr(path, name); - if (!(obj->flags & USER_GIVEN) && filter_fn) - r = filter_fn(LOFS_BLOB, obj, - path->buf, &path->buf[pathlen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + r = ctx->filter_fn(LOFS_BLOB, obj, + path->buf, &path->buf[pathlen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, path->buf, cb_data); + ctx->show_object(obj, path->buf, ctx->show_data); strbuf_setlen(path, pathlen); } @@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs, * the link, and how to do it. Whether it necessarily makes * any sense what-so-ever to ever do that is another issue. */ -static void process_gitlink(struct rev_info *revs, +static void process_gitlink(struct traversal_context *ctx, const unsigned char *sha1, - show_object_fn show, struct strbuf *path, - const char *name, - void *cb_data) + const char *name) { /* Nothing to do */ } -static void process_tree(struct rev_info *revs, +static void process_tree(struct traversal_context *ctx, struct tree *tree, - show_object_fn show, struct strbuf *base, - const char *name, - void *cb_data, - filter_object_fn filter_fn, - void *filter_data) + const char *name) { struct object *obj = &tree->object; + struct rev_info *revs = ctx->revs; struct tree_desc desc; struct name_entry entry; enum interesting match = revs->diffopt.pathspec.nr == 0 ? @@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs, } strbuf_addstr(base, name); - if (!(obj->flags & USER_GIVEN) && filter_fn) - r = filter_fn(LOFS_BEGIN_TREE, obj, - base->buf, &base->buf[baselen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + r = ctx->filter_fn(LOFS_BEGIN_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, base->buf, cb_data); + ctx->show_object(obj, base->buf, ctx->show_data); if (base->len) strbuf_addch(base, '/'); @@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs, } if (S_ISDIR(entry.mode)) - process_tree(revs, + process_tree(ctx, lookup_tree(the_repository, entry.oid), - show, base, entry.path, - cb_data, filter_fn, filter_data); + base, entry.path); else if (S_ISGITLINK(entry.mode)) - process_gitlink(revs, entry.oid->hash, - show, base, entry.path, - cb_data); + process_gitlink(ctx, entry.oid->hash, base, entry.path); else - process_blob(revs, + process_blob(ctx, lookup_blob(the_repository, entry.oid), - show, base, entry.path, - cb_data, filter_fn, filter_data); + base, entry.path); } - if (!(obj->flags & USER_GIVEN) && filter_fn) { - r = filter_fn(LOFS_END_TREE, obj, - base->buf, &base->buf[baselen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { + r = ctx->filter_fn(LOFS_END_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, base->buf, cb_data); + ctx->show_object(obj, base->buf, ctx->show_data); } strbuf_setlen(base, baselen); @@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree) add_pending_object(revs, &tree->object, ""); } -static void traverse_trees_and_blobs(struct rev_info *revs, - struct strbuf *base, - show_object_fn show_object, - void *show_data, - filter_object_fn filter_fn, - void *filter_data) +static void traverse_trees_and_blobs(struct traversal_context *ctx, + struct strbuf *base) { int i; assert(base->len == 0); - for (i = 0; i < revs->pending.nr; i++) { - struct object_array_entry *pending = revs->pending.objects + i; + for (i = 0; i < ctx->revs->pending.nr; i++) { + struct object_array_entry *pending = ctx->revs->pending.objects + i; struct object *obj = pending->item; const char *name = pending->name; const char *path = pending->path; @@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs, continue; if (obj->type == OBJ_TAG) { obj->flags |= SEEN; - show_object(obj, name, show_data); + ctx->show_object(obj, name, ctx->show_data); continue; } if (!path) path = ""; if (obj->type == OBJ_TREE) { - process_tree(revs, (struct tree *)obj, show_object, - base, path, show_data, - filter_fn, filter_data); + process_tree(ctx, (struct tree *)obj, base, path); continue; } if (obj->type == OBJ_BLOB) { - process_blob(revs, (struct blob *)obj, show_object, - base, path, show_data, - filter_fn, filter_data); + process_blob(ctx, (struct blob *)obj, base, path); continue; } die("unknown pending object %s (%s)", oid_to_hex(&obj->oid), name); } - object_array_clear(&revs->pending); + object_array_clear(&ctx->revs->pending); } -static void do_traverse(struct rev_info *revs, - show_commit_fn show_commit, - show_object_fn show_object, - void *show_data, - filter_object_fn filter_fn, - void *filter_data) +static void do_traverse(struct traversal_context *ctx) { struct commit *commit; struct strbuf csp; /* callee's scratch pad */ strbuf_init(&csp, PATH_MAX); - while ((commit = get_revision(revs)) != NULL) { + while ((commit = get_revision(ctx->revs)) != NULL) { /* * an uninteresting boundary commit may not have its tree * parsed yet, but we are not going to show them anyway */ if (get_commit_tree(commit)) - add_pending_tree(revs, get_commit_tree(commit)); - show_commit(commit, show_data); + add_pending_tree(ctx->revs, get_commit_tree(commit)); + ctx->show_commit(commit, ctx->show_data); - if (revs->tree_blobs_in_commit_order) + if (ctx->revs->tree_blobs_in_commit_order) /* * NEEDSWORK: Adding the tree and then flushing it here * needs a reallocation for each commit. Can we pass the * tree directory without allocation churn? */ - traverse_trees_and_blobs(revs, &csp, - show_object, show_data, - filter_fn, filter_data); + traverse_trees_and_blobs(ctx, &csp); } - traverse_trees_and_blobs(revs, &csp, - show_object, show_data, - filter_fn, filter_data); + traverse_trees_and_blobs(ctx, &csp); strbuf_release(&csp); } @@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs, show_object_fn show_object, void *show_data) { - do_traverse(revs, show_commit, show_object, show_data, NULL, NULL); + struct traversal_context ctx; + ctx.revs = revs; + ctx.show_commit = show_commit; + ctx.show_object = show_object; + ctx.show_data = show_data; + ctx.filter_fn = NULL; + ctx.filter_data = NULL; + do_traverse(&ctx); } void traverse_commit_list_filtered( @@ -337,14 +323,18 @@ void traverse_commit_list_filtered( void *show_data, struct oidset *omitted) { - filter_object_fn filter_fn = NULL; + struct traversal_context ctx; filter_free_fn filter_free_fn = NULL; - void *filter_data = NULL; - - filter_data = list_objects_filter__init(omitted, filter_options, - &filter_fn, &filter_free_fn); - do_traverse(revs, show_commit, show_object, show_data, - filter_fn, filter_data); - if (filter_data && filter_free_fn) - filter_free_fn(filter_data); + + ctx.revs = revs; + ctx.show_object = show_object; + ctx.show_commit = show_commit; + ctx.show_data = show_data; + ctx.filter_fn = NULL; + + ctx.filter_data = list_objects_filter__init(omitted, filter_options, + &ctx.filter_fn, &filter_free_fn); + do_traverse(&ctx); + if (ctx.filter_data && filter_free_fn) + filter_free_fn(ctx.filter_data); } From patchwork Wed Oct 3 19:52:13 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625253 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F0DBF13BB for ; Wed, 3 Oct 2018 19:52:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DD48F2904B for ; Wed, 3 Oct 2018 19:52:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D150829050; Wed, 3 Oct 2018 19:52:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6F1722904B for ; Wed, 3 Oct 2018 19:52:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727253AbeJDCm3 (ORCPT ); Wed, 3 Oct 2018 22:42:29 -0400 Received: from mail-vs1-f73.google.com ([209.85.217.73]:55834 "EHLO mail-vs1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCm3 (ORCPT ); Wed, 3 Oct 2018 22:42:29 -0400 Received: by mail-vs1-f73.google.com with SMTP id e15-v6so2248635vso.22 for ; Wed, 03 Oct 2018 12:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=w9PL1mndR6OdE97ALIWJ7/eMgY6Fzt1CJXNhuiZ9jBs=; b=arTeUfiaJxcrJSSzA2A7i7Wmj3jJvuFF5VD2EkXX1EQODo/vowkj7kGnO3mZKvPCtH UP6IWqYjQ3g2qowG+3tb/XcXhc0pjxCuxSX2HFMXRcB27gnaoPD5QSECHOncV67hwFtE 3Kt1RjkNJHTvokUaAM64zSCU/FcvHDtOWjyXMAa9+0QA8IWJ85MM617S9ulgDxIAgCJu ymm53VGqoos9ivx5G/Vcubd1BlbFHDj6U3bs4Ba+X7cwPIYY7qgxbjhFukd2Fwx1ZCNz 1Y+m+ArgsDG5jWB7SNX/vooy9L1qA0Cdni3PFA0cH6EgfbWZbMxaCYog3K15AJ0b/bB8 zarw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=w9PL1mndR6OdE97ALIWJ7/eMgY6Fzt1CJXNhuiZ9jBs=; b=pF/I9/Tr1c4qad+t8d59zSfVQb12koX85aBbMkH6ZKs8qNukgaCPxhk0gtzmZAXKh2 J5X3T3yi9VBNGesxVHW582OBFZNVM0Q0+X/GM93GbwAIhy9mo/z3GAG7mGCB5cdok+2T TixqFzyYs4ajwvR76XTxqSOrQRSW0MbGY1GvADAdBtoVqRFgSozaNbVhbBqypMoFsdqq /Uwi0MlwN9AWtq/ERgFWplamQ7MYl3xcLRI4MEy5qpsQhNUHlDiURssqP7AhbmCyPIqj Z+LeLTztpaVHtugcrVsbRI1PkSKXu8jDu1kVUyJxr1NJCW6gmCP8Jbxkl+YHksQRxXM7 vF3A== X-Gm-Message-State: ABuFfogUytpXoj+/tEkaoBin+9+jH0zf/SAEHzuVxfNii+r+BOmDrHed Ua+AxtauJFFuiIrc5NxDkvH2xQ0HqTGTeOlHgJeL8WjyMT8pY73dwfLKYVSMbirpPVsmu/gjaJ6 vsKtsDtcSWFRczf6ycuYSwz6tX8JNz491co5efrx23xDxCMMF/pZYkZEW4q0= X-Google-Smtp-Source: ACcGV62BflcmKwRMGBx4u0eLYHoIe3xdu1/d3YBq/n0B2vMRLQAIdnuiJZuWqEfmi3odQTmXl409HW0l+cFM X-Received: by 2002:a67:7f8f:: with SMTP id a137-v6mr2775580vsd.21.1538596359972; Wed, 03 Oct 2018 12:52:39 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:13 -0700 In-Reply-To: Message-Id: <42ad5623bab52b8207831eab4a4f27e47a318e36.1538592829.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 2/8] list-objects: refactor to process_tree_contents From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This will be used in a follow-up patch to reduce indentation needed when invoking the logic conditionally. i.e. rather than: if (foo) { while (...) { /* this is very indented */ } } we will have: if (foo) process_tree_contents(...); Signed-off-by: Matthew DeVore --- list-objects.c | 68 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 27 deletions(-) diff --git a/list-objects.c b/list-objects.c index 584518a3f..ccc529e5e 100644 --- a/list-objects.c +++ b/list-objects.c @@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx, /* Nothing to do */ } +static void process_tree(struct traversal_context *ctx, + struct tree *tree, + struct strbuf *base, + const char *name); + +static void process_tree_contents(struct traversal_context *ctx, + struct tree *tree, + struct strbuf *base) +{ + struct tree_desc desc; + struct name_entry entry; + enum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ? + all_entries_interesting : entry_not_interesting; + + init_tree_desc(&desc, tree->buffer, tree->size); + + while (tree_entry(&desc, &entry)) { + if (match != all_entries_interesting) { + match = tree_entry_interesting(&entry, base, 0, + &ctx->revs->diffopt.pathspec); + if (match == all_entries_not_interesting) + break; + if (match == entry_not_interesting) + continue; + } + + if (S_ISDIR(entry.mode)) + process_tree(ctx, + lookup_tree(the_repository, entry.oid), + base, entry.path); + else if (S_ISGITLINK(entry.mode)) + process_gitlink(ctx, entry.oid->hash, + base, entry.path); + else + process_blob(ctx, + lookup_blob(the_repository, entry.oid), + base, entry.path); + } +} + static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, @@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx, { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; - struct tree_desc desc; - struct name_entry entry; - enum interesting match = revs->diffopt.pathspec.nr == 0 ? - all_entries_interesting: entry_not_interesting; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; int gently = revs->ignore_missing_links || @@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx, if (base->len) strbuf_addch(base, '/'); - init_tree_desc(&desc, tree->buffer, tree->size); - - while (tree_entry(&desc, &entry)) { - if (match != all_entries_interesting) { - match = tree_entry_interesting(&entry, base, 0, - &revs->diffopt.pathspec); - if (match == all_entries_not_interesting) - break; - if (match == entry_not_interesting) - continue; - } - - if (S_ISDIR(entry.mode)) - process_tree(ctx, - lookup_tree(the_repository, entry.oid), - base, entry.path); - else if (S_ISGITLINK(entry.mode)) - process_gitlink(ctx, entry.oid->hash, base, entry.path); - else - process_blob(ctx, - lookup_blob(the_repository, entry.oid), - base, entry.path); - } + process_tree_contents(ctx, tree, base); if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, From patchwork Wed Oct 3 19:52:14 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625255 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3B7D513BB for ; Wed, 3 Oct 2018 19:52:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 299082904E for ; Wed, 3 Oct 2018 19:52:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1C73729051; Wed, 3 Oct 2018 19:52:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BB2682904E for ; Wed, 3 Oct 2018 19:52:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727353AbeJDCmc (ORCPT ); Wed, 3 Oct 2018 22:42:32 -0400 Received: from mail-vs1-f73.google.com ([209.85.217.73]:51049 "EHLO mail-vs1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCmc (ORCPT ); Wed, 3 Oct 2018 22:42:32 -0400 Received: by mail-vs1-f73.google.com with SMTP id m5-v6so2251113vsn.17 for ; Wed, 03 Oct 2018 12:52:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lRBW4T+Uj1qpBIE5Q/etFfBP2/bdQ1ZPeAB+hIuj5tY=; b=mxWSJI5zSifQ8ENzAXIOo+IT1U/0T7wq27yUga+fyCG7Vrhs3VOOtQ+WJ3q+m0a0HT DacqLpJw4f/PW7L5ODLpmpUai4/VC4Nu4CYPupClfMsKjMn5K5cFkNun3WWUkCAlRH2l H7M9gIUvFFJiafELkMXXIHeqZ5A5QHWLfApGG6xFnT6UuIfSYaFjWWi5OLU9Vbe0fuB5 EaN3zP9XMui+VCZ+RrseWKtUMtj3LP/q5ygf8bD4MlM0DOCVTpPYy9hcCVFbkjwsTRGD 3/SRsAB2aRR2FIsPZg9g2ZbfjWYopAPo2yYEupzcazPmTt/lYmuE7nmb4zzwDbLEJUjL RfkA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lRBW4T+Uj1qpBIE5Q/etFfBP2/bdQ1ZPeAB+hIuj5tY=; b=gNKK9mFncsfMvJYKOZzFwjPOES6UEM2BhOeD+euWcyAhy5c66sStsD3taVADLSWjVI P6ZMQ0/yzKNsen6QtKzldmBHQW3I1Pzmmrj8std2g5VLld++9WrTJzXcJfB09SBGNe6W OS2cIx24RorKH2jGATJH27N27eJ2OEkxcrPqjrqs7VPayKqSqqSri6THuBHaeMlIOSCf DgkDowgo/8NOa1Gu8oSa1AiZuAHS4VfEnHk6N0MoAUgw2bA2I9J9rXW2qcxoZEdVEvcx kjixvvT6tW+xKZf1gUan6pbhD7SwiLEyYqTM5DoMr/vFN5sMuxSzzDo5OzKLNmgJ7cKL v9Sw== X-Gm-Message-State: ABuFfogGn/5asFq3R87PbD8xxO5j4iUSNwr5iXfnrq5AJUq7VX4uhoVa adshe8Mh8t4ACqv5CwQrSlpRzNt6XjakFN5MU7oETfhw5iGDpTEBbE2RrJGQ5qzxjAEmaOEURQm 6OlbbwzT4Q4sOH7kAu2o1wZQf6+sY4cEKN21sonaxnnRURSxNmKBRmaBUxtg= X-Google-Smtp-Source: ACcGV62cZ2yKSCxsg5ZTetJEHoXQ9bGmXqxsPHpTqqhvYLeT8uISHgWEZVOC5gQ2fmhcCFOml/asM6klnohE X-Received: by 2002:a1f:ae4d:: with SMTP id x74-v6mr2736697vke.8.1538596362477; Wed, 03 Oct 2018 12:52:42 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:14 -0700 In-Reply-To: Message-Id: <0d1285a0ec412846e5417ec524a866f6b2c39d82.1538592829.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 3/8] list-objects: always parse trees gently From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If parsing fails when revs->ignore_missing_links and revs->exclude_promisor_objects are both false, we print the OID anyway in the die("bad tree object...") call, so any message printed by parse_tree_gently() is superfluous. Signed-off-by: Matthew DeVore --- list-objects.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/list-objects.c b/list-objects.c index ccc529e5e..f9b51db7a 100644 --- a/list-objects.c +++ b/list-objects.c @@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx, struct rev_info *revs = ctx->revs; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; - int gently = revs->ignore_missing_links || - revs->exclude_promisor_objects; if (!revs->tree_objects) return; @@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx, die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; - if (parse_tree_gently(tree, gently) < 0) { + if (parse_tree_gently(tree, 1) < 0) { if (revs->ignore_missing_links) return; From patchwork Wed Oct 3 19:52:15 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625257 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2183F1515 for ; Wed, 3 Oct 2018 19:52:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D2F252904B for ; Wed, 3 Oct 2018 19:52:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C74F129051; Wed, 3 Oct 2018 19:52:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 064432904B for ; Wed, 3 Oct 2018 19:52:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727357AbeJDCmf (ORCPT ); Wed, 3 Oct 2018 22:42:35 -0400 Received: from mail-qt1-f202.google.com ([209.85.160.202]:51443 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCmf (ORCPT ); Wed, 3 Oct 2018 22:42:35 -0400 Received: by mail-qt1-f202.google.com with SMTP id m1-v6so6107728qtb.18 for ; Wed, 03 Oct 2018 12:52:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=raIGRrhYeEuVmXqyFBN1aZj1x2FMhZt/AnSO+5/JfnY=; b=CWmQGVkvVIgoGJi3IcNxtSkYutXcOsXh2A2N6tYuj3E/Vbs/fniOCypPktbD1cRytj tEGPcCIa4FuwDJlXspbq5i5bpK94fsMfOsXdK9QP1qHDOS7NdKYzG5mLidYcrdn5bLTo Elvw0DsrzGRNf6bT0N2jX9bnMrDdifWVuYfT8rdgop3XMnRnMoNK2uw/CojYpy6L9UGe bD6OsiOBntj6i/kYhvIbSo7xmr2VUTN6GmWjOX6QygnPYkAREOjhgFIMBuLFwifQ6C86 m7SV3pbunhD6k4xeGG+KB85aCZE71+LIAe967OgeG0NuxQMivifsIWyFTwu8IqRepXX3 HyUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=raIGRrhYeEuVmXqyFBN1aZj1x2FMhZt/AnSO+5/JfnY=; b=VY8eVJP8pgV5if+lKCF2kg9PKelXYBzpw/eS0Oyura+k5u0s7mSQCLBZup06mvuFAm KUMUPBkRHTO0HaEgBQDgR2B/DS82+lOrztCnwtWmvxaL6flOqrbMkLMiAc4tpGTxAVjO 54nlRzvDvUDGS75Q4F9gEhhuhYNNEx+m5zN+cO3/vdJuc9Dgoo7vpEBj1hr+obXxBWz+ GSBTYVQPBlfR4r4swvzAtpQBkrYoP6mlJv9pPGl0P2R8GwZFp2CPSbmS/NNrHifGiNjU v1nWWolTg2yh1E37lbl8wSZUcEd99U3LLRZ2PXqX4v5QSI0r2SeFoYtlgYwnyctA038p LkfQ== X-Gm-Message-State: ABuFfohbV0RjkMrcxAFg4bVsipq2Lx9a3UbIFRvjUbq0arXSwfNV6HVh NBNKH3maDczYURyUkvC6omu9CqSnR85sf2zQmwRPvMOohn29nAJLQlAVOzMtP+a8+3t8jXECeoN PVIaMKZ99aRPGLMd1Al09YM64a/x9M9lR77vxATQW3bXHZdg4mhPX6CgQBCY= X-Google-Smtp-Source: ACcGV62fq1E+k0WoqEMmAuSIF1GbkQpZ0w78OYcZ6/HN4ZQBk2uTaTAszqQxKtUCFeOtxDT/lwrINIRmrKpI X-Received: by 2002:a0c:d253:: with SMTP id o19-v6mr2205018qvh.43.1538596365005; Wed, 03 Oct 2018 12:52:45 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:15 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 4/8] rev-list: handle missing tree objects properly From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Previously, we assumed only blob objects could be missing. This patch makes rev-list handle missing trees like missing blobs. The --missing=* and --exclude-promisor-objects flags now work for trees as they already do for blobs. This is demonstrated in t6112. Signed-off-by: Matthew DeVore --- builtin/rev-list.c | 11 ++++--- list-objects.c | 11 +++++-- revision.h | 15 +++++++++ t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++ t/t5317-pack-objects-filter-objects.sh | 13 ++++++++ t/t6112-rev-list-filters-objects.sh | 17 ++++++++++ 6 files changed, 105 insertions(+), 7 deletions(-) diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 5b07f3f4a..49d6deed7 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -6,6 +6,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "object.h" #include "object-store.h" #include "pack.h" #include "pack-bitmap.h" @@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj) */ switch (arg_missing_action) { case MA_ERROR: - die("missing blob object '%s'", oid_to_hex(&obj->oid)); + die("missing %s object '%s'", + type_name(obj->type), oid_to_hex(&obj->oid)); return; case MA_ALLOW_ANY: @@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj) case MA_ALLOW_PROMISOR: if (is_promisor_object(&obj->oid)) return; - die("unexpected missing blob object '%s'", - oid_to_hex(&obj->oid)); + die("unexpected missing %s object '%s'", + type_name(obj->type), oid_to_hex(&obj->oid)); return; default: @@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj) static int finish_object(struct object *obj, const char *name, void *cb_data) { struct rev_list_info *info = cb_data; - if (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) { + if (!has_object_file(&obj->oid)) { finish_object__ma(obj); return 1; } @@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) init_revisions(&revs, prefix); revs.abbrev = DEFAULT_ABBREV; revs.commit_format = CMIT_FMT_UNSPECIFIED; + revs.do_not_die_on_missing_tree = 1; /* * Scan the argument list before invoking setup_revisions(), so that we diff --git a/list-objects.c b/list-objects.c index f9b51db7a..243192af5 100644 --- a/list-objects.c +++ b/list-objects.c @@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx, struct rev_info *revs = ctx->revs; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; + int failed_parse; if (!revs->tree_objects) return; @@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx, die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; - if (parse_tree_gently(tree, 1) < 0) { + + failed_parse = parse_tree_gently(tree, 1); + if (failed_parse) { if (revs->ignore_missing_links) return; @@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx, is_promisor_object(&obj->oid)) return; - die("bad tree object %s", oid_to_hex(&obj->oid)); + if (!revs->do_not_die_on_missing_tree) + die("bad tree object %s", oid_to_hex(&obj->oid)); } strbuf_addstr(base, name); @@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx, if (base->len) strbuf_addch(base, '/'); - process_tree_contents(ctx, tree, base); + if (!failed_parse) + process_tree_contents(ctx, tree, base); if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, diff --git a/revision.h b/revision.h index 007278cc1..5910613cb 100644 --- a/revision.h +++ b/revision.h @@ -126,6 +126,21 @@ struct rev_info { line_level_traverse:1, tree_blobs_in_commit_order:1, + /* + * Blobs are shown without regard for their existence. + * But not so for trees: unless exclude_promisor_objects + * is set and the tree in question is a promisor object; + * OR ignore_missing_links is set, the revision walker + * dies with a "bad tree object HASH" message when + * encountering a missing tree. For callers that can + * handle missing trees and want them to be filterable + * and showable, set this to true. The revision walker + * will filter and show such a missing tree as usual, + * but will not attempt to recurse into this tree + * object. + */ + do_not_die_on_missing_tree:1, + /* for internal use only */ exclude_promisor_objects:1; diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 128130066..5bc5b4445 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' ' ! grep $FOO out ' +test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' ' + rm -rf repo && + test_create_repo repo && + test_commit -C repo foo && + test_commit -C repo bar && + test_commit -C repo baz && + + promise_and_delete $(git -C repo rev-parse bar^{tree}) && + promise_and_delete $(git -C repo rev-parse foo^{tree}) && + + git -C repo config core.repositoryformatversion 1 && + git -C repo config extensions.partialclone "arbitrary string" && + + git -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 3 commits, 3 blobs, and 1 tree + test_line_count = 7 objs && + + # Do the same for --exclude-promisor-objects, but with all trees gone. + promise_and_delete $(git -C repo rev-parse baz^{tree}) && + git -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 3 commits, no blobs or trees + test_line_count = 3 objs +' + +test_expect_success 'missing non-root tree object and rev-list' ' + rm -rf repo && + test_create_repo repo && + mkdir repo/dir && + echo foo >repo/dir/foo && + git -C repo add dir/foo && + git -C repo commit -m "commit dir/foo" && + + promise_and_delete $(git -C repo rev-parse HEAD:dir) && + + git -C repo config core.repositoryformatversion 1 && + git -C repo config extensions.partialclone "arbitrary string" && + + git -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 1 commit and 1 tree + test_line_count = 2 objs +' + test_expect_success 'rev-list stops traversal at missing and promised tree' ' rm -rf repo && test_create_repo repo && diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 6710c8bc8..9839b48c1 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre test_cmp observed expected ' +test_expect_success 'get an error for missing tree object' ' + git init r5 && + echo foo >r5/foo && + git -C r5 add foo && + git -C r5 commit -m "foo" && + del=$(git -C r5 rev-parse HEAD^{tree} | sed "s|..|&/|") && + rm r5/.git/objects/$del && + test_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF && + HEAD + EOF + grep -q "bad tree object" bad_tree +' + # Test blob:limit=[kmg] filter. # We boundary test around the size parameter. The filter is strictly less than # the value, so size 500 and 1000 should have the same results, but 1001 should diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index d4ff0b3be..c662c97db 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -195,6 +195,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' ' test_cmp observed expected ' +test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' ' + TREE=$(git -C r3 rev-parse HEAD:dir1) && + + rm r3/.git/objects/$(echo $TREE | sed "s|^..|&/|") && + + git -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err && + echo "?$TREE" >expected && + test_cmp expected missing_objs && + + # do not complain when a missing tree cannot be parsed + test_must_be_empty rev_list_err && + + git -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err && + ! grep $TREE objs && + test_must_be_empty rev_list_err +' + # Delete some loose objects and use rev-list, but WITHOUT any filtering. # This models previously omitted objects that we did not receive. From patchwork Wed Oct 3 19:52:16 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625259 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0936E1515 for ; Wed, 3 Oct 2018 19:52:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED5A42904B for ; Wed, 3 Oct 2018 19:52:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E07C429051; Wed, 3 Oct 2018 19:52:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3DDD82904B for ; Wed, 3 Oct 2018 19:52:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727432AbeJDCmh (ORCPT ); Wed, 3 Oct 2018 22:42:37 -0400 Received: from mail-it1-f202.google.com ([209.85.166.202]:40627 "EHLO mail-it1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCmh (ORCPT ); Wed, 3 Oct 2018 22:42:37 -0400 Received: by mail-it1-f202.google.com with SMTP id z136-v6so8038696itc.5 for ; Wed, 03 Oct 2018 12:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=tzE4Q/+BwfQHqwpB0245kC9B2C7VotiDFckQfMCMuSI=; b=GAYVWy2Y2RucuA63qdVQQKS8KmydWkMEpLyjWepEzyOYuL8e5cIQFLFNwstT17dOf4 IBz5NiZc0PPPfEarfo/bNuPVlwtFhGnVpPz/4zr2TAgTobS1RFIu0YNhKag6O0CvSoWh fFBwUI0rzAI+AVwgLh0WAekP/OKUcCMzNCmf7hq1Vw4F3Dr3kvdHMT++SjiSBQ4KEeh9 oMnt/jV/iSb53A9yAiBAXl+WWlbkjgxW4WeRmkvS+Qf2oSZi/hq0aSTgovq5B8XzelE8 ABHLFLi6arviIEqo+T//zbpw+uLFQ3xH5ynQikxqeFSso+OQ5EH/cEvcyKfy5y3XZwlr W+0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=tzE4Q/+BwfQHqwpB0245kC9B2C7VotiDFckQfMCMuSI=; b=F9oLkRxeieyIZRGKAsK3RkvU2A+olEbiB3fnxrzopg8E7zfPG8NZO05lLCGK0YpSZu mOPrQZeEyctThxBfq2oxOSGubAAUWnN3oi+dL6AjCDq2WP79GuLyv9AwgEvzXbrmbbgT vG+VsnpSZUUpm9tLvrcODDWi74UYhbmN3YQIlIj5mIrvsuMRrlIqSFmkjrj7oftbFi4Y rGzNnM5v31T3JoMlbdj+NBwmNc6JWtETWuVUMd8nnEI2PjnDjI3jJhkB9cKQdhHIZL5d xsCujCCnsnstkkR7uGQQzhTnqvhfzxY//7XDRp/SNA62sI8sbUj5r6eJ010to2S1Hihg gZ5w== X-Gm-Message-State: ABuFfojch81OS0KCrIqY2widuG4Hk+lh4igwgP6tGrojjXQiLlCmaJCt BoJorL/50utL0/E/oco/XotEnoi6o44ovgJoTZk0Mmg61V3zjmu5ZwmBcOhGMPebipEGUPF/zGX o+AocfWH4k3tD6M/oXyERTzkGEbTAeqMyNvxYq8MfYirwJh0OwfmnnaxigwA= X-Google-Smtp-Source: ACcGV60cBrPh4anp2gR5qvKYl3RYJKwF8fzjDOUYON/isIHPnnwHDuSYL7X9yVAZe9bQNH+uENw16OpT/zzy X-Received: by 2002:a24:1382:: with SMTP id 124-v6mr2511099itz.30.1538596367350; Wed, 03 Oct 2018 12:52:47 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:16 -0700 In-Reply-To: Message-Id: <6fb22953a334327a48881f691481981af925d56a.1538592829.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 5/8] revision: mark non-user-given objects instead From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, list-objects.c incorrectly treats all root trees of commits as USER_GIVEN. Also, it would be easier to mark objects that are non-user-given instead of user-given, since the places in the code where we access an object through a reference are more obvious than the places where we access an object that was given by the user. Resolve these two problems by introducing a flag NOT_USER_GIVEN that marks blobs and trees that are non-user-given, replacing USER_GIVEN. (Only blobs and trees are marked because this mark is only used when filtering objects, and filtering of other types of objects is not supported yet.) This fixes a bug in that git rev-list behaved differently from git pack-objects. pack-objects would *not* filter objects given explicitly on the command line and rev-list would filter. This was because the two commands used a different function to add objects to the rev_info struct. This seems to have been an oversight, and pack-objects has the correct behavior, so I added a test to make sure that rev-list now behaves properly. Signed-off-by: Matthew DeVore --- list-objects.c | 31 +++++++++++++++++------------ revision.c | 1 - revision.h | 11 ++++++++-- t/t6112-rev-list-filters-objects.sh | 12 +++++++++++ 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/list-objects.c b/list-objects.c index 243192af5..7a1a0929d 100644 --- a/list-objects.c +++ b/list-objects.c @@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx, pathlen = path->len; strbuf_addstr(path, name); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) r = ctx->filter_fn(LOFS_BLOB, obj, path->buf, &path->buf[pathlen], ctx->filter_data); @@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx, continue; } - if (S_ISDIR(entry.mode)) - process_tree(ctx, - lookup_tree(the_repository, entry.oid), - base, entry.path); + if (S_ISDIR(entry.mode)) { + struct tree *t = lookup_tree(the_repository, entry.oid); + t->object.flags |= NOT_USER_GIVEN; + process_tree(ctx, t, base, entry.path); + } else if (S_ISGITLINK(entry.mode)) process_gitlink(ctx, entry.oid->hash, base, entry.path); - else - process_blob(ctx, - lookup_blob(the_repository, entry.oid), - base, entry.path); + else { + struct blob *b = lookup_blob(the_repository, entry.oid); + b->object.flags |= NOT_USER_GIVEN; + process_blob(ctx, b, base, entry.path); + } } } @@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx, } strbuf_addstr(base, name); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) r = ctx->filter_fn(LOFS_BEGIN_TREE, obj, base->buf, &base->buf[baselen], ctx->filter_data); @@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx, if (!failed_parse) process_tree_contents(ctx, tree, base); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, base->buf, &base->buf[baselen], ctx->filter_data); @@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx) * an uninteresting boundary commit may not have its tree * parsed yet, but we are not going to show them anyway */ - if (get_commit_tree(commit)) - add_pending_tree(ctx->revs, get_commit_tree(commit)); + if (get_commit_tree(commit)) { + struct tree *tree = get_commit_tree(commit); + tree->object.flags |= NOT_USER_GIVEN; + add_pending_tree(ctx->revs, tree); + } ctx->show_commit(commit, ctx->show_data); if (ctx->revs->tree_blobs_in_commit_order) diff --git a/revision.c b/revision.c index de4dce600..72d48a17f 100644 --- a/revision.c +++ b/revision.c @@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs, strbuf_release(&buf); return; /* do not add the commit itself */ } - obj->flags |= USER_GIVEN; add_object_array_with_path(obj, name, &revs->pending, mode, path); } diff --git a/revision.h b/revision.h index 5910613cb..83e164039 100644 --- a/revision.h +++ b/revision.h @@ -21,9 +21,16 @@ #define SYMMETRIC_LEFT (1u<<8) #define PATCHSAME (1u<<9) #define BOTTOM (1u<<10) -#define USER_GIVEN (1u<<25) /* given directly by the user */ +/* + * Indicates object was reached by traversal. i.e. not given by user on + * command-line or stdin. + * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support + * filtering trees and blobs, but it may be useful to support filtering commits + * in the future. + */ +#define NOT_USER_GIVEN (1u<<25) #define TRACK_LINEAR (1u<<26) -#define ALL_REV_FLAGS (((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR) +#define ALL_REV_FLAGS (((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR) #define DECORATE_SHORT_REFS 1 #define DECORATE_FULL_REFS 2 diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index c662c97db..11186209b 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -30,6 +30,18 @@ test_expect_success 'verify blob:none omits all 5 blobs' ' test_cmp observed expected ' +test_expect_success 'specify blob explicitly prevents filtering' ' + file_3=$(git -C r1 ls-files -s file.3 | + awk -f print_2.awk) && + + file_4=$(git -C r1 ls-files -s file.4 | + awk -f print_2.awk) && + + git -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed && + grep -q "$file_3" observed && + test_must_fail grep -q "$file_4" observed +' + test_expect_success 'verify emitted+omitted == all' ' git -C r1 rev-list HEAD --objects \ | awk -f print_1.awk \ From patchwork Wed Oct 3 19:52:17 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625261 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7EEE813BB for ; Wed, 3 Oct 2018 19:52:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6A79A2904B for ; Wed, 3 Oct 2018 19:52:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5EE0029051; Wed, 3 Oct 2018 19:52:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0B2B02904B for ; Wed, 3 Oct 2018 19:52:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727443AbeJDCmj (ORCPT ); Wed, 3 Oct 2018 22:42:39 -0400 Received: from mail-io1-f73.google.com ([209.85.166.73]:45309 "EHLO mail-io1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCmj (ORCPT ); Wed, 3 Oct 2018 22:42:39 -0400 Received: by mail-io1-f73.google.com with SMTP id g133-v6so6449150ioa.12 for ; Wed, 03 Oct 2018 12:52:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eo+qxEhYTUv3+lsEELI+YrG0h/e6PzpHh4BhzyK/WVc=; b=gAV1pG9qtW4Gz7g/6wt2I0sUyxQwbYdF40PaC/3Ege7UsbSaXONJYdlkBB4jGBQrS3 kEqCVl5vnVGQG0Yk1i83qija1/gft0wkQkTODlKy9n31FUPT45r4h0l3hHOOAPaOwoEh m3I4sEObEZlein2JixHUBGGqd3lw7R45QgEupkt6R1ysu90bn+q2S0Rsx0NW9CWd2s65 Ruy63t47pMMupuJP2lxaM0SVOG4fbaWa2nxxEfVGSsEvQCjxxJrYSskWgol5iyH6+n4Y ojvgar2+4EUdYiJqeSX/5ex7Ean2n1I4oRCvT/2QcSemVXPVFhlTyHSFh+C7CtpWUHo6 9v9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eo+qxEhYTUv3+lsEELI+YrG0h/e6PzpHh4BhzyK/WVc=; b=InXrf0JBSMyFaCS3AkbFeOuHH6SW444WEEkDQpj54b7xpXe0zEiujqzYk+7fXTbp65 n7NGZOu4XsouUTBr8w8uRnoRzxgwPWoIDrDCWhBezZoJTIoEv/cUA2ChZIvJA2jIAznI 8W4f/HvzOCGMTHeoApEqrz3MHqx8NcrNvunoRBd7BQtqs4nIejOlGGFlKZHBU450qUkg G5NEdi9akAYlS0vwSYtaix1pygewVUr4e3AAw5rMmptcAgsrjYpfqyccY2gYL2IMXyzF vNGBvcarBXv+i+Hj6BMLPrvVCunvJ+7LslRgczmZN1D59t3YaW4lbH1C7qh5ptjYGT2f BfWA== X-Gm-Message-State: ABuFfoiaR7uQcgNtikeaD8f/uIrsslWH+n8DuNirP0ND04fRCDn0JQnW G+nAMt9R+w7HulvW7uvUjqaYnBKeLg0lbMjFsUS2XQNdVwSb5aeLIgcsdqZVj0TB+wkpfwZkNvv Zj7h5kYGZW5SlBlyEOjDj7vgRnCSoH9NNby1/wRicNJ390q7w1vMLwkKIlUA= X-Google-Smtp-Source: ACcGV61nyEW7Kc4E6Xn8jc5GLlkiRVYSkr1+iI/QIZS3xeCnNcBBheoD9vnQQR1dHi3ADNA6XCQUoIsSvgGc X-Received: by 2002:a05:660c:b0c:: with SMTP id f12mr2367551itk.22.1538596369764; Wed, 03 Oct 2018 12:52:49 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:17 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 6/8] list-objects-filter: use BUG rather than die From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In some cases in this file, BUG makes more sense than die. In such cases, a we get there from a coding error rather than a user error. 'return' has been removed following some instances of BUG since BUG does not return. Signed-off-by: Matthew DeVore --- list-objects-filter.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index a0ba78b20..5f8b1a002 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -389,7 +386,7 @@ void *list_objects_filter__init( assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT); if (filter_options->choice >= LOFC__COUNT) - die("invalid list-objects filter choice: %d", + BUG("invalid list-objects filter choice: %d", filter_options->choice); init_fn = s_filters[filter_options->choice]; From patchwork Wed Oct 3 19:52:18 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625263 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 924F81515 for ; Wed, 3 Oct 2018 19:52:54 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 841692904B for ; Wed, 3 Oct 2018 19:52:54 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 76FF129051; Wed, 3 Oct 2018 19:52:54 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1FA1F2904B for ; Wed, 3 Oct 2018 19:52:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727319AbeJDCmm (ORCPT ); Wed, 3 Oct 2018 22:42:42 -0400 Received: from mail-io1-f74.google.com ([209.85.166.74]:55212 "EHLO mail-io1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCml (ORCPT ); Wed, 3 Oct 2018 22:42:41 -0400 Received: by mail-io1-f74.google.com with SMTP id l24-v6so6547331iok.21 for ; Wed, 03 Oct 2018 12:52:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=PQw5VNRqY67VEQ0o3B/Q/ISsDEZKysHNiMYTvo+2YUI=; b=Df2HlqngtWvkNC/UoySDVyGsQmmM0/GkEvW+U99QzifwsWzst5C6fkCxBKeyBSwGdz cDd8BwddI3eDFcodaeHDvGqA1FHNdKt9KQVxYrz/7rmEEVhI6Apq+51ssUTzbjmGuZj8 ufriDXLefoL8qkMpgXOAz1Z0E3PEbZ2oPeecpbhqjX5IXohw2LJpqgwXv2QyAejwkwi/ RpcZC/qX1ozKeoiaTY6pukH7CFajQ1n2jgI+nioh4E6khErjeVtPOFdTnSPNs3TKXVS9 yanueRuFg3PrCc9N+IwMQkiahk1fpiuUAq2Kit0JRpPR6DUpKIe8sj6KXzVbohBiLErU i9mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PQw5VNRqY67VEQ0o3B/Q/ISsDEZKysHNiMYTvo+2YUI=; b=KKCYV5Ws8DDyHdHon8CVxDInl3vsvguQzfZVGg2KuaUBL0YxksFPji8OdxE+GYa+wb eGaytDTI0uXBxS386SohRE5PvRb8oAqQ+AgEzVs9xTSVBapXQ7DHyA6son+bYfuGx/5Y 1b5I3dCifeDciMfWNAXqw5+wBY4rrVYtTdQ5gE+INh+4qZ6+dsNIIMEiKTdYk86NZFE9 DzEWGcX3Wwt/kKuPYRR22iiU/qtbRICHZ8zirditZju6qe5eidthjSK/00LRcob9+7Q8 IiV01hE+EqvKkPstTS0dZnb2iYL4gGMTxpKZm51K8lAFayZ/oG8dzPg+XZ/6FhybY3SX ZHdw== X-Gm-Message-State: ABuFfohBbMykb4o6QeMIxUfur+y2GZM4AtYHMq+4qOJsoQ9/+xVhLBhT foIZWT3tppJ6Fp/MHNlFMrJsW8gMTaRLEmOnDFzfggPoLF/AMzMm5XP60p3QQYG0kueNxVknBj6 k7KyUpnwCM9p0laisofk9OUvtk2ehLu1FU7hjOwQFIi1Jn00aQ44vZT27vsw= X-Google-Smtp-Source: ACcGV63MV/JJAz2l4Fwd2NpX91/Tib6o1s5FN3ZUI7HCuSumFmks1EuZPBRzBkd5sP0xy1CMx/T2MBMllOFs X-Received: by 2002:a05:660c:b0c:: with SMTP id f12mr2367615itk.22.1538596372291; Wed, 03 Oct 2018 12:52:52 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:18 -0700 In-Reply-To: Message-Id: <4d9bbf5c8af670a969a5b11876a65f0505b1a1b8.1538592829.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 7/8] list-objects-filter-options: do not over-strbuf_init From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The function gently_parse_list_objects_filter is either called with errbuf=STRBUF_INIT or errbuf=NULL, but that function calls strbuf_init when errbuf is not NULL. strbuf_init is only necessary if errbuf contains garbage, and risks a memory leak if errbuf already has a non-STRBUF_INIT state. It should be the caller's responsibility to make sure errbuf is not garbage, since garbage content is easily avoidable with STRBUF_INIT. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0e2bd6a0..d259bdb2c 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter( if (filter_options->choice) { if (errbuf) { - strbuf_init(errbuf, 0); strbuf_addstr( errbuf, _("multiple filter-specs cannot be combined")); @@ -71,10 +70,9 @@ static int gently_parse_list_objects_filter( return 0; } - if (errbuf) { - strbuf_init(errbuf, 0); + if (errbuf) strbuf_addf(errbuf, "invalid filter-spec '%s'", arg); - } + memset(filter_options, 0, sizeof(*filter_options)); return 1; } From patchwork Wed Oct 3 19:52:19 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10625265 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7B70F1515 for ; Wed, 3 Oct 2018 19:52:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 68FDC2904B for ; Wed, 3 Oct 2018 19:52:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5D7CA29051; Wed, 3 Oct 2018 19:52:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8ACA12904B for ; Wed, 3 Oct 2018 19:52:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727450AbeJDCmp (ORCPT ); Wed, 3 Oct 2018 22:42:45 -0400 Received: from mail-qt1-f202.google.com ([209.85.160.202]:34606 "EHLO mail-qt1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726941AbeJDCmp (ORCPT ); Wed, 3 Oct 2018 22:42:45 -0400 Received: by mail-qt1-f202.google.com with SMTP id e88-v6so6271101qtb.1 for ; Wed, 03 Oct 2018 12:52:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=G/TcuJZJ8E0csjRFvYNE4Kw3ZGMq8kIM7gEPP3kalCc=; b=KHtMg0Pm2CqdUBda+zYuVl80t4Lb2HCG9Gh8eTtgaV9OyKXRV4G0LP+chwe4mv7g7u imFKgt/VFSPP1MyJ1XDlkRPBSb0sbf7eDOryD3eZ6iWBfZH9jQTcder1WxgUra7ihbty +mY4mYynZdaDWIucqwJ/vA/QnohVgZq1YNNQfz9CIVmIZTlbcwNcCorFtTWm72Go6u7I 0A4MqiiYYLKzEynDfLA2e+xtnzpSlecKmYPkFLqi5B5OfZaFvV5GynicV5DIr9oxE37y rSdmfuvWu7Nl9yGaNCrESYTNHVseI4ZsQ4o+yYqkrJs2jpl40Ag5N5Z3XjyAAqWbAHCq lkMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=G/TcuJZJ8E0csjRFvYNE4Kw3ZGMq8kIM7gEPP3kalCc=; b=Jzhn/Jd7dwgYY1K+LLAqV5Bl3QDRp6qMomGwMpsCohoV+KUlOnZso+8MPKCUBmljer 3uQV3YZgCK3Oy/bE5KgxWRwqDvXaGJcFd60CFxdxnzHFbminyFUZ/A+ViBoYqZ1HSEW9 G6wpEfpTKk4urdNE0Qeef2Q8RT7FrTuDhCpYXCtBFAdN8676rvg6WCgVjVp//gK4wbWa nmn6pWIQWTdJVgB/DP3e0l0XmfV+2Yx7YkkW4sh+y2scerdGGuzdqx28Ciq2vBAZLHgH qjbiMsZUORLxKM09ge/JJ6P8TRmb9Xp28frhvhB6KTp+LlGeCPAOA2vElrWzoIsg6Age BdzQ== X-Gm-Message-State: ABuFfogs7DSKDEaVXL7DneHMbaQYgmeokT/YIHAs6n9KFEe6vXkWjujw vwAI66n7CpMTkUhBQlVIeP+BmvDz6OQrxL/67SnhuM5/kmfPM3HkEMQdaN6NN3uCX74qpDe2n8S TtnmdDcV6YmHg2arIRg08oVaUl+FM1gFMfnRfwKJtfMids5eq9lJUBSYXCfM= X-Google-Smtp-Source: ACcGV624TsVNDrv7+sR78+cZuugpB162YLnwUGXzdIdiy2gnWRDpB5fIYnZ0tFpFoInRpi8r401ju7lS0Dj9 X-Received: by 2002:a0c:aee6:: with SMTP id n38-v6mr2269355qvd.17.1538596374911; Wed, 03 Oct 2018 12:52:54 -0700 (PDT) Date: Wed, 3 Oct 2018 12:52:19 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v10 8/8] list-objects-filter: implement filter tree:0 From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Teach list-objects the "tree:0" filter which allows for filtering out all tree and blob objects (unless other objects are explicitly specified by the user). The purpose of this patch is to allow smaller partial clones. The name of this filter - tree:0 - does not explicitly specify that it also filters out all blobs, but this should not cause much confusion because blobs are not at all useful without the trees that refer to them. I also considered only:commits as a name, but this is inaccurate because it suggests that annotated tags are omitted, but actually they are included. The name "tree:0" allows later filtering based on depth, i.e. "tree:1" would filter out all but the root tree and blobs. In order to avoid confusion between 0 and capital O, the documentation was worded in a somewhat round-about way that also hints at this future improvement to the feature. Signed-off-by: Matthew DeVore --- Documentation/rev-list-options.txt | 5 +++ list-objects-filter-options.c | 13 +++++++ list-objects-filter-options.h | 1 + list-objects-filter.c | 49 ++++++++++++++++++++++++++ t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++ t/t5616-partial-clone.sh | 42 ++++++++++++++++++++++ t/t6112-rev-list-filters-objects.sh | 14 ++++++++ 7 files changed, 152 insertions(+) diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index 7b273635d..5f1672913 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -731,6 +731,11 @@ the requested refs. + The form '--filter=sparse:path=' similarly uses a sparse-checkout specification contained in . ++ +The form '--filter=tree:' omits all blobs and trees whose depth +from the root tree is >= (minimum depth if an object is located +at multiple depths in the commits traversed). Currently, only =0 +is supported, which omits all blobs and trees. --no-filter:: Turn off any previous `--filter=` argument. diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index d259bdb2c..e8da2e858 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -49,6 +49,19 @@ static int gently_parse_list_objects_filter( return 0; } + } else if (skip_prefix(arg, "tree:", &v0)) { + unsigned long depth; + if (!git_parse_ulong(v0, &depth) || depth != 0) { + if (errbuf) { + strbuf_addstr( + errbuf, + _("only 'tree:0' is supported")); + } + return 1; + } + filter_options->choice = LOFC_TREE_NONE; + return 0; + } else if (skip_prefix(arg, "sparse:oid=", &v0)) { struct object_context oc; struct object_id sparse_oid; diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 0000a61f8..af64e5c66 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -10,6 +10,7 @@ enum list_objects_filter_choice { LOFC_DISABLED = 0, LOFC_BLOB_NONE, LOFC_BLOB_LIMIT, + LOFC_TREE_NONE, LOFC_SPARSE_OID, LOFC_SPARSE_PATH, LOFC__COUNT /* must be last */ diff --git a/list-objects-filter.c b/list-objects-filter.c index 5f8b1a002..09b2b05d5 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -79,6 +79,54 @@ static void *filter_blobs_none__init( return d; } +/* + * A filter for list-objects to omit ALL trees and blobs from the traversal. + * Can OPTIONALLY collect a list of the omitted OIDs. + */ +struct filter_trees_none_data { + struct oidset *omits; +}; + +static enum list_objects_filter_result filter_trees_none( + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + void *filter_data_) +{ + struct filter_trees_none_data *filter_data = filter_data_; + + switch (filter_situation) { + default: + BUG("unknown filter_situation: %d", filter_situation); + + case LOFS_BEGIN_TREE: + case LOFS_BLOB: + if (filter_data->omits) + oidset_insert(filter_data->omits, &obj->oid); + return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ + + case LOFS_END_TREE: + assert(obj->type == OBJ_TREE); + return LOFR_ZERO; + + } +} + +static void* filter_trees_none__init( + struct oidset *omitted, + struct list_objects_filter_options *filter_options, + filter_object_fn *filter_fn, + filter_free_fn *filter_free_fn) +{ + struct filter_trees_none_data *d = xcalloc(1, sizeof(*d)); + d->omits = omitted; + + *filter_fn = filter_trees_none; + *filter_free_fn = free; + return d; +} + /* * A filter for list-objects to omit large blobs. * And to OPTIONALLY collect a list of the omitted OIDs. @@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, + filter_trees_none__init, filter_sparse_oid__init, filter_sparse_path__init, }; diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 9839b48c1..510d3537f 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' ' grep -q "bad tree object" bad_tree ' +test_expect_success 'setup for tests of tree:0' ' + mkdir r1/subtree && + echo "This is a file in a subtree" >r1/subtree/file && + git -C r1 add subtree/file && + git -C r1 commit -m subtree +' + +test_expect_success 'verify tree:0 packfile has no blobs or trees' ' + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF && + HEAD + EOF + git -C r1 index-pack ../commitsonly.pack && + git -C r1 verify-pack -v ../commitsonly.pack >objs && + ! grep -E "tree|blob" objs +' + +test_expect_success 'grab tree directly when using tree:0' ' + # We should get the tree specified directly but not its blobs or subtrees. + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF && + HEAD: + EOF + git -C r1 index-pack ../commitsonly.pack && + git -C r1 verify-pack -v ../commitsonly.pack >objs && + awk "/tree|blob/{print \$1}" objs >trees_and_blobs && + git -C r1 rev-parse HEAD: >expected && + test_cmp expected trees_and_blobs +' + # Test blob:limit=[kmg] filter. # We boundary test around the size parameter. The filter is strictly less than # the value, so size 500 and 1000 should have the same results, but 1001 should diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index bbbe7537d..53fbf7db8 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -154,6 +154,48 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack - grep "git index-pack.*--fsck-objects" trace ' +test_expect_success 'use fsck before and after manually fetching a missing subtree' ' + # push new commit so server has a subtree + mkdir src/dir && + echo "in dir" >src/dir/file.txt && + git -C src add dir/file.txt && + git -C src commit -m "file in dir" && + git -C src push -u srv master && + SUBTREE=$(git -C src rev-parse HEAD:dir) && + + rm -rf dst && + git clone --no-checkout --filter=tree:0 "file://$(pwd)/srv.bare" dst && + git -C dst fsck && + + # Make sure we only have commits, and all trees and blobs are missing. + git -C dst rev-list --missing=allow-any --objects master \ + >fetched_objects && + awk -f print_1.awk fetched_objects | + xargs -n1 git -C dst cat-file -t >fetched_types && + + sort -u fetched_types >unique_types.observed && + echo commit >unique_types.expected && + test_cmp unique_types.expected unique_types.observed && + + # Auto-fetch a tree with cat-file. + git -C dst cat-file -p $SUBTREE >tree_contents && + grep file.txt tree_contents && + + # fsck still works after an auto-fetch of a tree. + git -C dst fsck && + + # Auto-fetch all remaining trees and blobs with --missing=error + git -C dst rev-list --missing=error --objects master >fetched_objects && + test_line_count = 70 fetched_objects && + + awk -f print_1.awk fetched_objects | + xargs -n1 git -C dst cat-file -t >fetched_types && + + sort -u fetched_types >unique_types.observed && + printf "blob\ncommit\ntree\n" >unique_types.expected && + test_cmp unique_types.expected unique_types.observed +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index 11186209b..5a61614b1 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -224,6 +224,20 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre test_must_be_empty rev_list_err ' +# Test tree:0 filter. + +test_expect_success 'verify tree:0 includes trees in "filtered" output' ' + git -C r3 rev-list --quiet --objects --filter-print-omitted \ + --filter=tree:0 HEAD | + awk -f print_1.awk | + sed s/~// | + xargs -n1 git -C r3 cat-file -t | + sort -u >filtered_types && + + printf "blob\ntree\n" >expected && + test_cmp expected filtered_types +' + # Delete some loose objects and use rev-list, but WITHOUT any filtering. # This models previously omitted objects that we did not receive.