From patchwork Fri Sep 21 20:31:57 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610947 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 58DFA161F for ; Fri, 21 Sep 2018 20:32:30 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 461BB2E7A5 for ; Fri, 21 Sep 2018 20:32:30 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 3976B2E7B4; Fri, 21 Sep 2018 20:32:30 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 33D8A2E7A5 for ; Fri, 21 Sep 2018 20:32:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391496AbeIVCW7 (ORCPT ); Fri, 21 Sep 2018 22:22:59 -0400 Received: from mail-ua1-f74.google.com ([209.85.222.74]:44478 "EHLO mail-ua1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391318AbeIVCW7 (ORCPT ); Fri, 21 Sep 2018 22:22:59 -0400 Received: by mail-ua1-f74.google.com with SMTP id d22-v6so3490768uaq.11 for ; Fri, 21 Sep 2018 13:32:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=TgH/QacDGg7Oj2iQRSFg9H1GJ0tfo5thD1AEE9RxD1A=; b=rXMmbooDoa7fYaceTRXKW8z41wxx15D1uintyLmvs7d3p2zk0XGwW6U9qFelKytNCX O2e6aOYvIvVPxTIt9LN/7ETqJ4NtNnVgirFBg2FPvfljJtvqtA2DPdPLWy18bDEZEg1n KUJaqpJSqhbMjm+HHcQ+QqQ4ozyj7RqZFgeDf/THepC2F23Q3kEZqbw5RxuGc1Io2NXA 8i0NomLNL3G1iMnuAAg6PAXuLuEgwiNTp4BOAImqS01sTL7il9fUIsMmJk/A26d0o2Jk Pl60x2nZ1oylb6cJTXxlTCFIHcw4yeIDYRuKZs0dcmNJZeRHTEjSyFgDxjmRc2c5sw4X 5XRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=TgH/QacDGg7Oj2iQRSFg9H1GJ0tfo5thD1AEE9RxD1A=; b=ZT/EULZsuMMH8KM9R5TuR++saiK2KyrdlGKPMpG4KzrkYRMg/DDM5OMd8zfnDQrU2e oF3snp4Zz/OjEAuKIKP2HlbrK7+hHueT3pSPUzhCoi5zHCI8976KVmq3G1qfCcozmhYG 7okPq4Gb8LAE9UrXw/iKqOoZuCpu7C2SoTN0kjgD+on7BeFq/4VM42yW9Ry3myO8gagB FR3V5kwL+knQh7lZd66Z4sFgQ92yJZ4YYn+fLKWjeazk0AJutDru4UUz4AF9BwNArIWV Zc2fXtQEkk+a0kk972XI4Qwa1tQ+ZaKPymbdIrh4IMG/WQFaDtwkg1Lz0koN15/h8H/L EYkA== X-Gm-Message-State: APzg51A5aX7Mkp+mE5XTFNORf4lsixamNx0qzTMEmIVouBwdSR+L/lG5 MidKpFk3zW+jyeMimbcBQaGA/H490oI1qsRXa+eUpqPHpxxCpy67TE3VqJUyjlrqCbmlQZEa+Rz K4G4wB1r4y8hER9nITik0rFyIVzQKKxoqOs5uHRySOBE5Vb59K1N4y5A0s/8= X-Google-Smtp-Source: ANB0VdYktM9kNbz0tFyhZomNg+cTDoczTdRsq10pPhF30gB64OMqyi0ZG0zPjqfWXNWtZPCf3o4XIN2S3jmy X-Received: by 2002:ab0:2102:: with SMTP id d2-v6mr8986091ual.6.1537561945655; Fri, 21 Sep 2018 13:32:25 -0700 (PDT) Date: Fri, 21 Sep 2018 13:31:57 -0700 In-Reply-To: Message-Id: <3bff094363c7887bfe6547e520cbe83097cf6f73.1537561420.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 1/8] list-objects: store common func args in struct From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This will make utility functions easier to create, as done by the next patch. Signed-off-by: Matthew DeVore --- list-objects.c | 158 +++++++++++++++++++++++-------------------------- 1 file changed, 74 insertions(+), 84 deletions(-) diff --git a/list-objects.c b/list-objects.c index c99c47ac1..584518a3f 100644 --- a/list-objects.c +++ b/list-objects.c @@ -12,20 +12,25 @@ #include "packfile.h" #include "object-store.h" -static void process_blob(struct rev_info *revs, +struct traversal_context { + struct rev_info *revs; + show_object_fn show_object; + show_commit_fn show_commit; + void *show_data; + filter_object_fn filter_fn; + void *filter_data; +}; + +static void process_blob(struct traversal_context *ctx, struct blob *blob, - show_object_fn show, struct strbuf *path, - const char *name, - void *cb_data, - filter_object_fn filter_fn, - void *filter_data) + const char *name) { struct object *obj = &blob->object; size_t pathlen; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; - if (!revs->blob_objects) + if (!ctx->revs->blob_objects) return; if (!obj) die("bad blob object"); @@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs, * may cause the actual filter to report an incomplete list * of missing objects. */ - if (revs->exclude_promisor_objects && + if (ctx->revs->exclude_promisor_objects && !has_object_file(&obj->oid) && is_promisor_object(&obj->oid)) return; pathlen = path->len; strbuf_addstr(path, name); - if (!(obj->flags & USER_GIVEN) && filter_fn) - r = filter_fn(LOFS_BLOB, obj, - path->buf, &path->buf[pathlen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + r = ctx->filter_fn(LOFS_BLOB, obj, + path->buf, &path->buf[pathlen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, path->buf, cb_data); + ctx->show_object(obj, path->buf, ctx->show_data); strbuf_setlen(path, pathlen); } @@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs, * the link, and how to do it. Whether it necessarily makes * any sense what-so-ever to ever do that is another issue. */ -static void process_gitlink(struct rev_info *revs, +static void process_gitlink(struct traversal_context *ctx, const unsigned char *sha1, - show_object_fn show, struct strbuf *path, - const char *name, - void *cb_data) + const char *name) { /* Nothing to do */ } -static void process_tree(struct rev_info *revs, +static void process_tree(struct traversal_context *ctx, struct tree *tree, - show_object_fn show, struct strbuf *base, - const char *name, - void *cb_data, - filter_object_fn filter_fn, - void *filter_data) + const char *name) { struct object *obj = &tree->object; + struct rev_info *revs = ctx->revs; struct tree_desc desc; struct name_entry entry; enum interesting match = revs->diffopt.pathspec.nr == 0 ? @@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs, } strbuf_addstr(base, name); - if (!(obj->flags & USER_GIVEN) && filter_fn) - r = filter_fn(LOFS_BEGIN_TREE, obj, - base->buf, &base->buf[baselen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + r = ctx->filter_fn(LOFS_BEGIN_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, base->buf, cb_data); + ctx->show_object(obj, base->buf, ctx->show_data); if (base->len) strbuf_addch(base, '/'); @@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs, } if (S_ISDIR(entry.mode)) - process_tree(revs, + process_tree(ctx, lookup_tree(the_repository, entry.oid), - show, base, entry.path, - cb_data, filter_fn, filter_data); + base, entry.path); else if (S_ISGITLINK(entry.mode)) - process_gitlink(revs, entry.oid->hash, - show, base, entry.path, - cb_data); + process_gitlink(ctx, entry.oid->hash, base, entry.path); else - process_blob(revs, + process_blob(ctx, lookup_blob(the_repository, entry.oid), - show, base, entry.path, - cb_data, filter_fn, filter_data); + base, entry.path); } - if (!(obj->flags & USER_GIVEN) && filter_fn) { - r = filter_fn(LOFS_END_TREE, obj, - base->buf, &base->buf[baselen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { + r = ctx->filter_fn(LOFS_END_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, base->buf, cb_data); + ctx->show_object(obj, base->buf, ctx->show_data); } strbuf_setlen(base, baselen); @@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree) add_pending_object(revs, &tree->object, ""); } -static void traverse_trees_and_blobs(struct rev_info *revs, - struct strbuf *base, - show_object_fn show_object, - void *show_data, - filter_object_fn filter_fn, - void *filter_data) +static void traverse_trees_and_blobs(struct traversal_context *ctx, + struct strbuf *base) { int i; assert(base->len == 0); - for (i = 0; i < revs->pending.nr; i++) { - struct object_array_entry *pending = revs->pending.objects + i; + for (i = 0; i < ctx->revs->pending.nr; i++) { + struct object_array_entry *pending = ctx->revs->pending.objects + i; struct object *obj = pending->item; const char *name = pending->name; const char *path = pending->path; @@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs, continue; if (obj->type == OBJ_TAG) { obj->flags |= SEEN; - show_object(obj, name, show_data); + ctx->show_object(obj, name, ctx->show_data); continue; } if (!path) path = ""; if (obj->type == OBJ_TREE) { - process_tree(revs, (struct tree *)obj, show_object, - base, path, show_data, - filter_fn, filter_data); + process_tree(ctx, (struct tree *)obj, base, path); continue; } if (obj->type == OBJ_BLOB) { - process_blob(revs, (struct blob *)obj, show_object, - base, path, show_data, - filter_fn, filter_data); + process_blob(ctx, (struct blob *)obj, base, path); continue; } die("unknown pending object %s (%s)", oid_to_hex(&obj->oid), name); } - object_array_clear(&revs->pending); + object_array_clear(&ctx->revs->pending); } -static void do_traverse(struct rev_info *revs, - show_commit_fn show_commit, - show_object_fn show_object, - void *show_data, - filter_object_fn filter_fn, - void *filter_data) +static void do_traverse(struct traversal_context *ctx) { struct commit *commit; struct strbuf csp; /* callee's scratch pad */ strbuf_init(&csp, PATH_MAX); - while ((commit = get_revision(revs)) != NULL) { + while ((commit = get_revision(ctx->revs)) != NULL) { /* * an uninteresting boundary commit may not have its tree * parsed yet, but we are not going to show them anyway */ if (get_commit_tree(commit)) - add_pending_tree(revs, get_commit_tree(commit)); - show_commit(commit, show_data); + add_pending_tree(ctx->revs, get_commit_tree(commit)); + ctx->show_commit(commit, ctx->show_data); - if (revs->tree_blobs_in_commit_order) + if (ctx->revs->tree_blobs_in_commit_order) /* * NEEDSWORK: Adding the tree and then flushing it here * needs a reallocation for each commit. Can we pass the * tree directory without allocation churn? */ - traverse_trees_and_blobs(revs, &csp, - show_object, show_data, - filter_fn, filter_data); + traverse_trees_and_blobs(ctx, &csp); } - traverse_trees_and_blobs(revs, &csp, - show_object, show_data, - filter_fn, filter_data); + traverse_trees_and_blobs(ctx, &csp); strbuf_release(&csp); } @@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs, show_object_fn show_object, void *show_data) { - do_traverse(revs, show_commit, show_object, show_data, NULL, NULL); + struct traversal_context ctx; + ctx.revs = revs; + ctx.show_commit = show_commit; + ctx.show_object = show_object; + ctx.show_data = show_data; + ctx.filter_fn = NULL; + ctx.filter_data = NULL; + do_traverse(&ctx); } void traverse_commit_list_filtered( @@ -337,14 +323,18 @@ void traverse_commit_list_filtered( void *show_data, struct oidset *omitted) { - filter_object_fn filter_fn = NULL; + struct traversal_context ctx; filter_free_fn filter_free_fn = NULL; - void *filter_data = NULL; - - filter_data = list_objects_filter__init(omitted, filter_options, - &filter_fn, &filter_free_fn); - do_traverse(revs, show_commit, show_object, show_data, - filter_fn, filter_data); - if (filter_data && filter_free_fn) - filter_free_fn(filter_data); + + ctx.revs = revs; + ctx.show_object = show_object; + ctx.show_commit = show_commit; + ctx.show_data = show_data; + ctx.filter_fn = NULL; + + ctx.filter_data = list_objects_filter__init(omitted, filter_options, + &ctx.filter_fn, &filter_free_fn); + do_traverse(&ctx); + if (ctx.filter_data && filter_free_fn) + filter_free_fn(ctx.filter_data); } From patchwork Fri Sep 21 20:31:58 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610949 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 63581112B for ; Fri, 21 Sep 2018 20:32:31 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 542782E7A5 for ; Fri, 21 Sep 2018 20:32:31 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 489BC2E7AF; Fri, 21 Sep 2018 20:32:31 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AEA682E7AC for ; Fri, 21 Sep 2018 20:32:29 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391499AbeIVCXA (ORCPT ); Fri, 21 Sep 2018 22:23:00 -0400 Received: from mail-pf1-f202.google.com ([209.85.210.202]:35877 "EHLO mail-pf1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391318AbeIVCW7 (ORCPT ); Fri, 21 Sep 2018 22:22:59 -0400 Received: by mail-pf1-f202.google.com with SMTP id d22-v6so7041490pfn.3 for ; Fri, 21 Sep 2018 13:32:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+jcwrQHG0/7jLAct55sMVM0IYo0kK4SM6zajPikhqMg=; b=qUu6QRBfDsTqltQ7L2mKMQtSTtE+3gEUBElUBZJZBTTw5yNvgq6Ye8QU5BIVKrBDAw H0oc3BV+0uV2lU9REuqy5EBVDueaEUGp3RGvz9QQxxDFocUnJwZzBWfYOWEIbab7O5/9 +8tRTKGIfykfN2P1+XdQjsNMNGO8eVey+HAfegtF7ch/oTLLKm5IsSN2mr8cxWq90+dw ZDIf1wIksbInxn+Fee0x1Se2SHdDtzSyH+rxrfxWVzYvCkhL6a+EE3i0ebuKBXAMWNhz qyYGrKlEZDoMpNpuKB0E4gYzfRG/3wgk2/P+tHjUK+fh1rmuo3bn7v5N9Cw+0i3DV1z9 e6WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=+jcwrQHG0/7jLAct55sMVM0IYo0kK4SM6zajPikhqMg=; b=oYL5Fog1eZ3XDA4R2sEMNt1k5xl4p32+Kew8jIsM4FTARU+r3ZfRoIWSVW6vGkpdNQ JKEkOoW+Rkx+QSB1eWKD+XhugdwmaXQPC0RFMRN4s/IYwdyini0lyrUD0ld9598gO7Y5 cn9rYcbxrRzD1vHZ2tTMcvkUOzPQ/52dmUFPEGpWJzbg0SNAI9Xgiq3j3BQec7/pxLgY X+fnmEt2UBKKRxQ9jqyY22+QXInn//aXo4ciTtlBLQLRaE7Sy3lIeJRpjrocu3AJqL7g XX170Jtxhw8DX89N5SV/Cz3sEkEGy1AuSL4h8AcsY/lH/CFjWQ79u2i9Hw+QRFVrad8l XJjQ== X-Gm-Message-State: APzg51BS9zr+kyf+g2xCHSzRHAX3bB3rJTbYtLWawIdWtiu2ID37vMnk RRBWq0XTcm5JHKgqh1OMz4BRRp/dw2TJyPcxCZndHQLs0xoJMN+TLie6pngXHlhLNg+MyiyIv7e LIq68H5PHPPMUV6AIGTPldnK1kDBqvqUCock89lVYI49ukPbdQclHnBU229c= X-Google-Smtp-Source: ANB0VdZfvJ9OFXuRCl5ewufiGg5OrtdzW/udIkrVB7AssAJWWTICNr8kLmxtbT8xKHAX8jLthn+xTtZa/uYC X-Received: by 2002:a63:8b42:: with SMTP id j63-v6mr1910277pge.19.1537561948018; Fri, 21 Sep 2018 13:32:28 -0700 (PDT) Date: Fri, 21 Sep 2018 13:31:58 -0700 In-Reply-To: Message-Id: <42ad5623bab52b8207831eab4a4f27e47a318e36.1537561420.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 2/8] list-objects: refactor to process_tree_contents From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This will be used in a follow-up patch to reduce indentation needed when invoking the logic conditionally. i.e. rather than: if (foo) { while (...) { /* this is very indented */ } } we will have: if (foo) process_tree_contents(...); Signed-off-by: Matthew DeVore --- list-objects.c | 68 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 27 deletions(-) diff --git a/list-objects.c b/list-objects.c index 584518a3f..ccc529e5e 100644 --- a/list-objects.c +++ b/list-objects.c @@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx, /* Nothing to do */ } +static void process_tree(struct traversal_context *ctx, + struct tree *tree, + struct strbuf *base, + const char *name); + +static void process_tree_contents(struct traversal_context *ctx, + struct tree *tree, + struct strbuf *base) +{ + struct tree_desc desc; + struct name_entry entry; + enum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ? + all_entries_interesting : entry_not_interesting; + + init_tree_desc(&desc, tree->buffer, tree->size); + + while (tree_entry(&desc, &entry)) { + if (match != all_entries_interesting) { + match = tree_entry_interesting(&entry, base, 0, + &ctx->revs->diffopt.pathspec); + if (match == all_entries_not_interesting) + break; + if (match == entry_not_interesting) + continue; + } + + if (S_ISDIR(entry.mode)) + process_tree(ctx, + lookup_tree(the_repository, entry.oid), + base, entry.path); + else if (S_ISGITLINK(entry.mode)) + process_gitlink(ctx, entry.oid->hash, + base, entry.path); + else + process_blob(ctx, + lookup_blob(the_repository, entry.oid), + base, entry.path); + } +} + static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, @@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx, { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; - struct tree_desc desc; - struct name_entry entry; - enum interesting match = revs->diffopt.pathspec.nr == 0 ? - all_entries_interesting: entry_not_interesting; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; int gently = revs->ignore_missing_links || @@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx, if (base->len) strbuf_addch(base, '/'); - init_tree_desc(&desc, tree->buffer, tree->size); - - while (tree_entry(&desc, &entry)) { - if (match != all_entries_interesting) { - match = tree_entry_interesting(&entry, base, 0, - &revs->diffopt.pathspec); - if (match == all_entries_not_interesting) - break; - if (match == entry_not_interesting) - continue; - } - - if (S_ISDIR(entry.mode)) - process_tree(ctx, - lookup_tree(the_repository, entry.oid), - base, entry.path); - else if (S_ISGITLINK(entry.mode)) - process_gitlink(ctx, entry.oid->hash, base, entry.path); - else - process_blob(ctx, - lookup_blob(the_repository, entry.oid), - base, entry.path); - } + process_tree_contents(ctx, tree, base); if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, From patchwork Fri Sep 21 20:31:59 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610951 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 682C6112B for ; Fri, 21 Sep 2018 20:32:32 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 596A62E7A5 for ; Fri, 21 Sep 2018 20:32:32 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4C8102E7AF; Fri, 21 Sep 2018 20:32:32 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 035472E7A5 for ; Fri, 21 Sep 2018 20:32:32 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391502AbeIVCXC (ORCPT ); Fri, 21 Sep 2018 22:23:02 -0400 Received: from mail-pf1-f201.google.com ([209.85.210.201]:35323 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391318AbeIVCXB (ORCPT ); Fri, 21 Sep 2018 22:23:01 -0400 Received: by mail-pf1-f201.google.com with SMTP id c8-v6so7039338pfn.2 for ; Fri, 21 Sep 2018 13:32:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=MBDgC0bSUmOT/SwVohS9qQegpVsmQ26X2ru/G8EyZ6A=; b=CBzhw2nuMUs89/AkFpiZZbvkAMilODF/O+pmPVIagyOt2EWzEYNfH5s69uTCP6PCEm /082FLB3q3RPdRhm1TCTtva2gb/hz2kxMFf8vK31RvKlbsJ7AH0bDxRhoTmyJtqvbFYu t14U5pz7hMsPIG8212F6WH0nvpl9HrQEfP2lHqpK+fNcr8UXgp53j5Btt0Hu5WrSz+05 YHGqJE9/chWPnFKTULBmxHTUNmo6YBz+gamEg3djlDmhute112F8UmNt2UbtYw5tZ81S jZjTDMbD5cu0kd1GeSvoapUQCJAukc1dscHE59wK7QXZVM7TMWdee5rm+1mXUZr0RsbW ZeMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=MBDgC0bSUmOT/SwVohS9qQegpVsmQ26X2ru/G8EyZ6A=; b=PdMpwtZ6dZ20QpcOn4tnW3JOgkluT/msM8IokYY6MvaBNPzXrxHPV1fsV0aZ1IP6lW BM1XeL7aVo57eoAbbilQ9Qtg0s03gmgJcpUIYX9y403ykKeceyM175uBRYyihJzvGWAD baHHS92DitBWGlWjiViIqyacq7qUCe8LXtttJtrfrWa93wfSk3S/0+39Bi1fTB+u9iTr csRAOY1ja6fUBI9c+JvzWKQSCA7sSqMb3sdVKL9F//lpIy4Yrh7grYkLyV91IfZosdZ+ eSHePM9SmST6g/o37Btw6A2ag/nwoJpr0zRDYZq9+8kU4JRC2N6MX6ZRJLK9gFQttjdr SqaA== X-Gm-Message-State: APzg51D/rPNs09uBEl1Ttzu5kHss2Qw38yo+07jRN+vtQ63FkNrOYqP9 aWigWJ/PMvATDmN6jxeq4myMIW2ivWCXmDgH/CusgDXzNc4yToAFjBZu08d2eJPg9OZ+T/vjItD 1P4uhwA06JZj7KL82bI86oJoske0+nHVdgiRwAfLjSCqpixuYNVg/gJoAmxI= X-Google-Smtp-Source: ANB0VdZkMnDdJV2/ajuMoDsB/53vivinVwPJulazwHbIsQIpwtUjZY21kKQkPzQXXoBYxttoJcdEmSdnkGuq X-Received: by 2002:a63:a17:: with SMTP id 23-v6mr303369pgk.140.1537561950137; Fri, 21 Sep 2018 13:32:30 -0700 (PDT) Date: Fri, 21 Sep 2018 13:31:59 -0700 In-Reply-To: Message-Id: <0d1285a0ec412846e5417ec524a866f6b2c39d82.1537561420.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 3/8] list-objects: always parse trees gently From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If parsing fails when revs->ignore_missing_links and revs->exclude_promisor_objects are both false, we print the OID anyway in the die("bad tree object...") call, so any message printed by parse_tree_gently() is superfluous. Signed-off-by: Matthew DeVore --- list-objects.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/list-objects.c b/list-objects.c index ccc529e5e..f9b51db7a 100644 --- a/list-objects.c +++ b/list-objects.c @@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx, struct rev_info *revs = ctx->revs; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; - int gently = revs->ignore_missing_links || - revs->exclude_promisor_objects; if (!revs->tree_objects) return; @@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx, die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; - if (parse_tree_gently(tree, gently) < 0) { + if (parse_tree_gently(tree, 1) < 0) { if (revs->ignore_missing_links) return; From patchwork Fri Sep 21 20:32:00 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610953 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7D2A8161F for ; Fri, 21 Sep 2018 20:32:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 69BA42E7A5 for ; Fri, 21 Sep 2018 20:32:35 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5D9582E7AF; Fri, 21 Sep 2018 20:32:35 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 982052E7A5 for ; Fri, 21 Sep 2018 20:32:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391505AbeIVCXE (ORCPT ); Fri, 21 Sep 2018 22:23:04 -0400 Received: from mail-pl1-f201.google.com ([209.85.214.201]:50085 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391318AbeIVCXE (ORCPT ); Fri, 21 Sep 2018 22:23:04 -0400 Received: by mail-pl1-f201.google.com with SMTP id b6-v6so6586049pls.16 for ; Fri, 21 Sep 2018 13:32:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lpX5ucK9xI9EsxegPosQAeFWufOLxzsCt4RrQALecnk=; b=PMtCNeI30ngqA90BOLGXKpPAkiTt69BQqFQxAADfDTl0/JbbqvdI0m5jRZvp2SFJcF gLhCdaDfmX+r0cAtps+y/AprKwytORXRzdrQBp4GsfCUt3zNi/JR+zsKHn7SD1YXvVkQ 78IUjQHv8WgJvesea1Ixn5V3ex2MwE1ZTJ7ycngZeJjX6ZTtQoUvQVFNRs5xixWc2ZIV sulzLGXaHlqHmRWzsO+bWVzOPh3CaBK6HRRt9NcyaMJHFX7YM29h+Mz2U4tvLqLP+lm8 M0ksGLpqW46lYhzPpBwTPIIoX5bRLEnuzTh6TWWY4bDcyQA1tr+tc1pDDbe8Jm1/1T5D +ZUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lpX5ucK9xI9EsxegPosQAeFWufOLxzsCt4RrQALecnk=; b=BfzmtYesYA0ASwgdSczaEZxmEY5DKqK9RkfV/9wTgN4nFwFOV+vPL8i2D3OkVeQ0DK EOjRbSy97orm6N1++Hb6IztrHaRGt1nYDQnQEMaeVVgylFrdxBbF6FwUMXcgsQXk269w Yx+IxbBulYB3VfFKh53joFZqFPdarIKJg6k6uiXvugbGRIMRdhcbsLpr1sugJ8SttxxE RxQuRBbTMzyOC8w7PywqSwNHXcXIsVhoBeAOT7r+YaUIo9kNeOFRIOrdt7CmHh2eDg6L cpnX8SPAhd5SJWBiCCPOYbafHhBAjYsFqBpMLyc3G8vvoWotKSlbFhVBttw2iYDXKlKU lNag== X-Gm-Message-State: APzg51CY0kLcHX25N0Bv3i7EW9WMqj4D/ZYN3G+La86G5j+VwYIZLDqC AHD+0HPa3j2naTCzXQMJbqaETtzpuM859Fc/+Mz7iQZNEcyOV3OYJspA229kqFmSIxVtlpdbT3R xNvpzmRXAiU5vwLDuM+M90bOBa+lphJKz2RcbjDa4H6TyK3Tdu0E2kgoocnc= X-Google-Smtp-Source: ANB0VdYV/6NkNpXZnu6Y5BKN7i+DY+xU/dv8wKGSuG7o6wVCGSc1cTbQjr/jQGnJabzvYfOmP3DSq0hWTDth X-Received: by 2002:a65:580c:: with SMTP id g12-v6mr1803510pgr.119.1537561952316; Fri, 21 Sep 2018 13:32:32 -0700 (PDT) Date: Fri, 21 Sep 2018 13:32:00 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 4/8] rev-list: handle missing tree objects properly From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Previously, we assumed only blob objects could be missing. This patch makes rev-list handle missing trees like missing blobs. The --missing=* and --exclude-promisor-objects flags now work for trees as they already do for blobs. This is demonstrated in t6112. Signed-off-by: Matthew DeVore --- builtin/rev-list.c | 11 ++++--- list-objects.c | 11 +++++-- revision.h | 15 +++++++++ t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++ t/t5317-pack-objects-filter-objects.sh | 13 ++++++++ t/t6112-rev-list-filters-objects.sh | 17 ++++++++++ 6 files changed, 105 insertions(+), 7 deletions(-) diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 5b07f3f4a..49d6deed7 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -6,6 +6,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "object.h" #include "object-store.h" #include "pack.h" #include "pack-bitmap.h" @@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj) */ switch (arg_missing_action) { case MA_ERROR: - die("missing blob object '%s'", oid_to_hex(&obj->oid)); + die("missing %s object '%s'", + type_name(obj->type), oid_to_hex(&obj->oid)); return; case MA_ALLOW_ANY: @@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj) case MA_ALLOW_PROMISOR: if (is_promisor_object(&obj->oid)) return; - die("unexpected missing blob object '%s'", - oid_to_hex(&obj->oid)); + die("unexpected missing %s object '%s'", + type_name(obj->type), oid_to_hex(&obj->oid)); return; default: @@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj) static int finish_object(struct object *obj, const char *name, void *cb_data) { struct rev_list_info *info = cb_data; - if (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) { + if (!has_object_file(&obj->oid)) { finish_object__ma(obj); return 1; } @@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) init_revisions(&revs, prefix); revs.abbrev = DEFAULT_ABBREV; revs.commit_format = CMIT_FMT_UNSPECIFIED; + revs.do_not_die_on_missing_tree = 1; /* * Scan the argument list before invoking setup_revisions(), so that we diff --git a/list-objects.c b/list-objects.c index f9b51db7a..243192af5 100644 --- a/list-objects.c +++ b/list-objects.c @@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx, struct rev_info *revs = ctx->revs; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; + int failed_parse; if (!revs->tree_objects) return; @@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx, die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; - if (parse_tree_gently(tree, 1) < 0) { + + failed_parse = parse_tree_gently(tree, 1); + if (failed_parse) { if (revs->ignore_missing_links) return; @@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx, is_promisor_object(&obj->oid)) return; - die("bad tree object %s", oid_to_hex(&obj->oid)); + if (!revs->do_not_die_on_missing_tree) + die("bad tree object %s", oid_to_hex(&obj->oid)); } strbuf_addstr(base, name); @@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx, if (base->len) strbuf_addch(base, '/'); - process_tree_contents(ctx, tree, base); + if (!failed_parse) + process_tree_contents(ctx, tree, base); if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, diff --git a/revision.h b/revision.h index 007278cc1..5910613cb 100644 --- a/revision.h +++ b/revision.h @@ -126,6 +126,21 @@ struct rev_info { line_level_traverse:1, tree_blobs_in_commit_order:1, + /* + * Blobs are shown without regard for their existence. + * But not so for trees: unless exclude_promisor_objects + * is set and the tree in question is a promisor object; + * OR ignore_missing_links is set, the revision walker + * dies with a "bad tree object HASH" message when + * encountering a missing tree. For callers that can + * handle missing trees and want them to be filterable + * and showable, set this to true. The revision walker + * will filter and show such a missing tree as usual, + * but will not attempt to recurse into this tree + * object. + */ + do_not_die_on_missing_tree:1, + /* for internal use only */ exclude_promisor_objects:1; diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 128130066..5bc5b4445 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' ' ! grep $FOO out ' +test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' ' + rm -rf repo && + test_create_repo repo && + test_commit -C repo foo && + test_commit -C repo bar && + test_commit -C repo baz && + + promise_and_delete $(git -C repo rev-parse bar^{tree}) && + promise_and_delete $(git -C repo rev-parse foo^{tree}) && + + git -C repo config core.repositoryformatversion 1 && + git -C repo config extensions.partialclone "arbitrary string" && + + git -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 3 commits, 3 blobs, and 1 tree + test_line_count = 7 objs && + + # Do the same for --exclude-promisor-objects, but with all trees gone. + promise_and_delete $(git -C repo rev-parse baz^{tree}) && + git -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 3 commits, no blobs or trees + test_line_count = 3 objs +' + +test_expect_success 'missing non-root tree object and rev-list' ' + rm -rf repo && + test_create_repo repo && + mkdir repo/dir && + echo foo >repo/dir/foo && + git -C repo add dir/foo && + git -C repo commit -m "commit dir/foo" && + + promise_and_delete $(git -C repo rev-parse HEAD:dir) && + + git -C repo config core.repositoryformatversion 1 && + git -C repo config extensions.partialclone "arbitrary string" && + + git -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 1 commit and 1 tree + test_line_count = 2 objs +' + test_expect_success 'rev-list stops traversal at missing and promised tree' ' rm -rf repo && test_create_repo repo && diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 6710c8bc8..9839b48c1 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre test_cmp observed expected ' +test_expect_success 'get an error for missing tree object' ' + git init r5 && + echo foo >r5/foo && + git -C r5 add foo && + git -C r5 commit -m "foo" && + del=$(git -C r5 rev-parse HEAD^{tree} | sed "s|..|&/|") && + rm r5/.git/objects/$del && + test_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF && + HEAD + EOF + grep -q "bad tree object" bad_tree +' + # Test blob:limit=[kmg] filter. # We boundary test around the size parameter. The filter is strictly less than # the value, so size 500 and 1000 should have the same results, but 1001 should diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index d4ff0b3be..c662c97db 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -195,6 +195,23 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' ' test_cmp observed expected ' +test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' ' + TREE=$(git -C r3 rev-parse HEAD:dir1) && + + rm r3/.git/objects/$(echo $TREE | sed "s|^..|&/|") && + + git -C r3 rev-list --quiet --missing=print --objects HEAD >missing_objs 2>rev_list_err && + echo "?$TREE" >expected && + test_cmp expected missing_objs && + + # do not complain when a missing tree cannot be parsed + test_must_be_empty rev_list_err && + + git -C r3 rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err && + ! grep $TREE objs && + test_must_be_empty rev_list_err +' + # Delete some loose objects and use rev-list, but WITHOUT any filtering. # This models previously omitted objects that we did not receive. From patchwork Fri Sep 21 20:32:01 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610955 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 92A16161F for ; Fri, 21 Sep 2018 20:32:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 841752E7A5 for ; Fri, 21 Sep 2018 20:32:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 782FC2E7AF; Fri, 21 Sep 2018 20:32:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id EAF4B2E7A5 for ; Fri, 21 Sep 2018 20:32:36 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391509AbeIVCXH (ORCPT ); Fri, 21 Sep 2018 22:23:07 -0400 Received: from mail-pl1-f202.google.com ([209.85.214.202]:45962 "EHLO mail-pl1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391318AbeIVCXG (ORCPT ); Fri, 21 Sep 2018 22:23:06 -0400 Received: by mail-pl1-f202.google.com with SMTP id k18-v6so6540020pls.12 for ; Fri, 21 Sep 2018 13:32:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=OLWCCwES3bxPoq6mvhnxyJM0hwM3UZJT4UydWB6+HaM=; b=s6xZ5DxcrRxgddgqZ6o8kSU/6p5itnCelgPTHiQ0u9eU6v43W1H7357nqHmoVWadZO B6fvAQlkRv126q4j6ND9a4MQ1Q2gK6Y7AQ1ImIiX4A9NRRRe0FGaMtNcLz37tyRYfqWP 4CJZzwEbKms6upMy8YoyVTse1wtN7VuTL3bUs+MWjkKNjtWNZuu6HOnPQUy9goCQ9s9b 106RZhsa403J55LVxXonlVc7yoMRVjMoOLfmEa/TbWZs/fGLpEecEeB1Tu4SFOY5DYza Ke+y068VPxGtknNCQzqZWGDI9AtLww9Htl/bsF+I5NafvjdELXHS1IqZPn3SUpFmeONu P5BA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=OLWCCwES3bxPoq6mvhnxyJM0hwM3UZJT4UydWB6+HaM=; b=q6k7h0rTxamUO+Wb4rXdLsBDg6L6VV8Z/yxM7WjYfR4pLuuT/wbdxCBXabzPYpV+Aa dEl1O2+IA6I5XgKR0+840X9O/mX8Xx6K4/oO6LOaAu0doZz70kbetYKfilcK806zQ7sv z+KUIwNWw4YE+xJq0vAWLaAl3A1T7grq/LB7Ligj4P2GMxw49qh6DBaqbqtmBMdVLA4p CMzffpZ5XvDkIYopvSDRThZ9RgfxwpOyUR9VOhVsGUvmmsklbEA9zNjnLCwz8lcLYJ/F X7ugX56xa/tc9KkTJiUUirm4412lN8R9N3ADSt9TUF/0Inp0f+Wdgmn///SEjgNDnWGt +h7g== X-Gm-Message-State: ABuFfojegcUzNwAJwTUx6einSNNvIRvafbL6NPEMMUEuw670HUfUDQ/J kRjhW2ksdRIt0jt0OvaNDzTAfjh8Sg1oU6itwSWhdDuUKjQayLvJ1wq1ys+U2Xn6LJewSC3b5ou AASHblhxebprWvVoiz47wlzlGIFpOzKcciq66C+wPYY5CTPz29qyar92KygM= X-Google-Smtp-Source: ACcGV61EcE6p5S5vpTJY8iLFPh992rI89yc81dViVMscOnV2GSanEidILMeIafP5R9wCeCB1VmVjbTetrA+P X-Received: by 2002:a17:902:778a:: with SMTP id o10-v6mr219904pll.40.1537561954525; Fri, 21 Sep 2018 13:32:34 -0700 (PDT) Date: Fri, 21 Sep 2018 13:32:01 -0700 In-Reply-To: Message-Id: <6fb22953a334327a48881f691481981af925d56a.1537561420.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 5/8] revision: mark non-user-given objects instead From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, list-objects.c incorrectly treats all root trees of commits as USER_GIVEN. Also, it would be easier to mark objects that are non-user-given instead of user-given, since the places in the code where we access an object through a reference are more obvious than the places where we access an object that was given by the user. Resolve these two problems by introducing a flag NOT_USER_GIVEN that marks blobs and trees that are non-user-given, replacing USER_GIVEN. (Only blobs and trees are marked because this mark is only used when filtering objects, and filtering of other types of objects is not supported yet.) This fixes a bug in that git rev-list behaved differently from git pack-objects. pack-objects would *not* filter objects given explicitly on the command line and rev-list would filter. This was because the two commands used a different function to add objects to the rev_info struct. This seems to have been an oversight, and pack-objects has the correct behavior, so I added a test to make sure that rev-list now behaves properly. Signed-off-by: Matthew DeVore --- list-objects.c | 31 +++++++++++++++++------------ revision.c | 1 - revision.h | 11 ++++++++-- t/t6112-rev-list-filters-objects.sh | 12 +++++++++++ 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/list-objects.c b/list-objects.c index 243192af5..7a1a0929d 100644 --- a/list-objects.c +++ b/list-objects.c @@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx, pathlen = path->len; strbuf_addstr(path, name); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) r = ctx->filter_fn(LOFS_BLOB, obj, path->buf, &path->buf[pathlen], ctx->filter_data); @@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx, continue; } - if (S_ISDIR(entry.mode)) - process_tree(ctx, - lookup_tree(the_repository, entry.oid), - base, entry.path); + if (S_ISDIR(entry.mode)) { + struct tree *t = lookup_tree(the_repository, entry.oid); + t->object.flags |= NOT_USER_GIVEN; + process_tree(ctx, t, base, entry.path); + } else if (S_ISGITLINK(entry.mode)) process_gitlink(ctx, entry.oid->hash, base, entry.path); - else - process_blob(ctx, - lookup_blob(the_repository, entry.oid), - base, entry.path); + else { + struct blob *b = lookup_blob(the_repository, entry.oid); + b->object.flags |= NOT_USER_GIVEN; + process_blob(ctx, b, base, entry.path); + } } } @@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx, } strbuf_addstr(base, name); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) r = ctx->filter_fn(LOFS_BEGIN_TREE, obj, base->buf, &base->buf[baselen], ctx->filter_data); @@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx, if (!failed_parse) process_tree_contents(ctx, tree, base); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, base->buf, &base->buf[baselen], ctx->filter_data); @@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx) * an uninteresting boundary commit may not have its tree * parsed yet, but we are not going to show them anyway */ - if (get_commit_tree(commit)) - add_pending_tree(ctx->revs, get_commit_tree(commit)); + if (get_commit_tree(commit)) { + struct tree *tree = get_commit_tree(commit); + tree->object.flags |= NOT_USER_GIVEN; + add_pending_tree(ctx->revs, tree); + } ctx->show_commit(commit, ctx->show_data); if (ctx->revs->tree_blobs_in_commit_order) diff --git a/revision.c b/revision.c index de4dce600..72d48a17f 100644 --- a/revision.c +++ b/revision.c @@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs, strbuf_release(&buf); return; /* do not add the commit itself */ } - obj->flags |= USER_GIVEN; add_object_array_with_path(obj, name, &revs->pending, mode, path); } diff --git a/revision.h b/revision.h index 5910613cb..83e164039 100644 --- a/revision.h +++ b/revision.h @@ -21,9 +21,16 @@ #define SYMMETRIC_LEFT (1u<<8) #define PATCHSAME (1u<<9) #define BOTTOM (1u<<10) -#define USER_GIVEN (1u<<25) /* given directly by the user */ +/* + * Indicates object was reached by traversal. i.e. not given by user on + * command-line or stdin. + * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support + * filtering trees and blobs, but it may be useful to support filtering commits + * in the future. + */ +#define NOT_USER_GIVEN (1u<<25) #define TRACK_LINEAR (1u<<26) -#define ALL_REV_FLAGS (((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR) +#define ALL_REV_FLAGS (((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR) #define DECORATE_SHORT_REFS 1 #define DECORATE_FULL_REFS 2 diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index c662c97db..11186209b 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -30,6 +30,18 @@ test_expect_success 'verify blob:none omits all 5 blobs' ' test_cmp observed expected ' +test_expect_success 'specify blob explicitly prevents filtering' ' + file_3=$(git -C r1 ls-files -s file.3 | + awk -f print_2.awk) && + + file_4=$(git -C r1 ls-files -s file.4 | + awk -f print_2.awk) && + + git -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed && + grep -q "$file_3" observed && + test_must_fail grep -q "$file_4" observed +' + test_expect_success 'verify emitted+omitted == all' ' git -C r1 rev-list HEAD --objects \ | awk -f print_1.awk \ From patchwork Fri Sep 21 20:32:02 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610957 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F16E112B for ; Fri, 21 Sep 2018 20:32:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1063C2E7A5 for ; Fri, 21 Sep 2018 20:32:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 043352E7AF; Fri, 21 Sep 2018 20:32:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A49422E7A5 for ; Fri, 21 Sep 2018 20:32:38 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391513AbeIVCXI (ORCPT ); Fri, 21 Sep 2018 22:23:08 -0400 Received: from mail-pf1-f201.google.com ([209.85.210.201]:41036 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391426AbeIVCXI (ORCPT ); Fri, 21 Sep 2018 22:23:08 -0400 Received: by mail-pf1-f201.google.com with SMTP id u13-v6so6965210pfm.8 for ; Fri, 21 Sep 2018 13:32:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Fb1ZOsUg8OZt+qOtOwoizl8zOK6GtkO7XgtEA0jfOxM=; b=sj2PoxYuSaVAuQn4o4DQ2SmpbKzDIcg3LlE4kbGT86pc5i7Jv5C5xrQsTqhu6s4+Ei bTboFXa2xjAmTKWWvpxxdjPVBux/oGJBRhpVU52+DsPnfFp9nV7Z0NObpjl8YoFtOb7e 1ch7ZDfRZ8l7KBDcYx9BbxA5GtgGcNvkzJ7a1evGt4gk1SNRCpClQRthy9MwzvCalQHX bXUbGL8iDeIKV6ATIXPep/rynr9F6+ex0FO7/qguooj44UMz5mopzAr/994+sg8tiNjS e334YcjwMcfSgnsEpu+719E1gIOCGQ5lb25nEvj6P0dz5c0QzqsqCuiOHKc5cJkFhzPn Q4NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Fb1ZOsUg8OZt+qOtOwoizl8zOK6GtkO7XgtEA0jfOxM=; b=uTLj6bVbteIG2x7x8H7phqAwmoR+HRtv/At8l7HY98CYk+RFTuMHYn32GOh+YpvIoJ YT/2xggRYAjUEZLmpNd8AhG15E5PfYOjc+waiGlZll6Rq9Ar2icHYmIRfFIJmxrFpN6l uhDEAOIBZiG2/p/plStkBgFpOexFtYYACmeyk5W94pmcpbbWmIcriX+HegLFqOCy0ecs OTII3EN1WNJ+vzteOmjHOWIWefPP50j4Llb81TVvlgoiFqrguVYSDnSCYBOvpcq6S/bV V6a0+/MVUFiQfiL5oS96nmvj1bmizvdtftLorXBwvTFdr92HN6YlWB+Og+1m9Py/coqQ Ov2w== X-Gm-Message-State: APzg51AGRmWLJiHgwgHOZMNpK59EiZTBktVNDeiXmpSgT+ZkHvnC5KDp VI6833fobCIcaN48UhTpkQeuV3LWdn1ui3vgTE+JazsAQYx/k31qlaC9xFQVPxQvegdXS/W4xXF wl/YIW/h4F0yeO3CPqiC33xHuRj3cEuAH4LlVWqzUsP3u4OjIpUus60NqKxY= X-Google-Smtp-Source: ANB0VdYTj39oEbMyIBWU6lrCjWzJRwfNMer46DP7EDOADlOMwFuy2JEpxNN6O1fDxwIooUzx1avvt7Ucf7e0 X-Received: by 2002:a62:c016:: with SMTP id x22-v6mr2981261pff.56.1537561956555; Fri, 21 Sep 2018 13:32:36 -0700 (PDT) Date: Fri, 21 Sep 2018 13:32:02 -0700 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 6/8] list-objects-filter: use BUG rather than die From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In some cases in this file, BUG makes more sense than die. In such cases, a we get there from a coding error rather than a user error. 'return' has been removed following some instances of BUG since BUG does not return. Signed-off-by: Matthew DeVore --- list-objects-filter.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index a0ba78b20..5f8b1a002 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -389,7 +386,7 @@ void *list_objects_filter__init( assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT); if (filter_options->choice >= LOFC__COUNT) - die("invalid list-objects filter choice: %d", + BUG("invalid list-objects filter choice: %d", filter_options->choice); init_fn = s_filters[filter_options->choice]; From patchwork Fri Sep 21 20:32:03 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610959 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 31FB7161F for ; Fri, 21 Sep 2018 20:32:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2366C2E7A5 for ; Fri, 21 Sep 2018 20:32:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 174D32E7AF; Fri, 21 Sep 2018 20:32:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BD6B22E7A5 for ; Fri, 21 Sep 2018 20:32:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391515AbeIVCXK (ORCPT ); Fri, 21 Sep 2018 22:23:10 -0400 Received: from mail-pl1-f201.google.com ([209.85.214.201]:40308 "EHLO mail-pl1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391318AbeIVCXK (ORCPT ); Fri, 21 Sep 2018 22:23:10 -0400 Received: by mail-pl1-f201.google.com with SMTP id n4-v6so6561685plk.7 for ; Fri, 21 Sep 2018 13:32:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=H1YyALbrZqOs0ZmlI+UqYUl/H9MNngc99B3IfB/8IdA=; b=X0YiNk5wj/zei0p668rebqxOb6olkfVRoNKSZzl0IuZXEn9Qifx95IsCeX4B3SZ40n mtf5n1bfmIm3z3/mfs/A6sxTA1h8j7xQ6cDT70SrmmXS8B4Iuk9CPE+pnr1VesOMyLby JGQWKPsjIVYQ4EGOv9SaYBQvmdwklnhEovX54xi9nHE08KHqxcuwBCYtD8aMd0gY5api shjwgohIIH2FcjDsnOJVXNUOztrrtYe0NL451zVVVSVsKy+pgVOZAgMOcnqKxkp/qWQa v8rjP3MI+FLERhmQNeUspeTDkOtskae6dbdjk04mM7e0RUmvMYPO0kFkBwTzvx5SjAtz YFMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=H1YyALbrZqOs0ZmlI+UqYUl/H9MNngc99B3IfB/8IdA=; b=OtLW3OYYyZ856uMDFOtesZEW+mVLeTLiUUQClHFjy65rznfOgslWVHK660lxfA2JYF +O8RTtXWcMGkdhuTaZEoA4JckrzB7SifNm0FQ7C2rGIf/Lvu3cRHMncjbNmK7CLg6RHG u7blheTbdehiwE064BKO2neUd1zC5aTw1T57YF1QH6EGzFGrW2+Oy0KxhaD7adkp++k/ Lvn7hcyWDdPZXg2aNbEeg9G9g4073wDfdo2EpTtG91JFcA1vJceZfC77kmBvWeG5raJw /pFfOHOIwkW+7dp6NSMc1aMJyesLfTL4nl+h7f3f+1S04qfroGyQ3ErXOOXD8ORnbI/J FsrA== X-Gm-Message-State: APzg51CDTB3Sqrh/lfv3EoFCN+0KY2tFb+KtEnq0YG8MUlYBwzg/HX44 CWWTiQEf2gQ39LCGrIED3JRxgPd6spcjKFXSsxM4V0xH/XsHed10wZlYHzYQZBuxK9ItM4J3Kol ptTdR+2NMVjmrKFpbR/RJoYuNklBk29SZHf27J3WjPtmixQ73sjlB84/YPrU= X-Google-Smtp-Source: ANB0VdYOUM8yXUdDX4yEt+rBEgatB3vtC/WgMC2LJ1kiqX1rw5UsdGvRMV0uoC7H5VmwjYZU2zttx7meG/p0 X-Received: by 2002:a63:7f0c:: with SMTP id a12-v6mr2114787pgd.76.1537561958895; Fri, 21 Sep 2018 13:32:38 -0700 (PDT) Date: Fri, 21 Sep 2018 13:32:03 -0700 In-Reply-To: Message-Id: <4d9bbf5c8af670a969a5b11876a65f0505b1a1b8.1537561420.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 7/8] list-objects-filter-options: do not over-strbuf_init From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The function gently_parse_list_objects_filter is either called with errbuf=STRBUF_INIT or errbuf=NULL, but that function calls strbuf_init when errbuf is not NULL. strbuf_init is only necessary if errbuf contains garbage, and risks a memory leak if errbuf already has a non-STRBUF_INIT state. It should be the caller's responsibility to make sure errbuf is not garbage, since garbage content is easily avoidable with STRBUF_INIT. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0e2bd6a0..d259bdb2c 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter( if (filter_options->choice) { if (errbuf) { - strbuf_init(errbuf, 0); strbuf_addstr( errbuf, _("multiple filter-specs cannot be combined")); @@ -71,10 +70,9 @@ static int gently_parse_list_objects_filter( return 0; } - if (errbuf) { - strbuf_init(errbuf, 0); + if (errbuf) strbuf_addf(errbuf, "invalid filter-spec '%s'", arg); - } + memset(filter_options, 0, sizeof(*filter_options)); return 1; } From patchwork Fri Sep 21 20:32:04 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10610961 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 157AF161F for ; Fri, 21 Sep 2018 20:32:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 067662E7A5 for ; Fri, 21 Sep 2018 20:32:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EF4302E7AF; Fri, 21 Sep 2018 20:32:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3735D2E7A5 for ; Fri, 21 Sep 2018 20:32:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2391519AbeIVCXO (ORCPT ); Fri, 21 Sep 2018 22:23:14 -0400 Received: from mail-pf1-f201.google.com ([209.85.210.201]:37591 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2391517AbeIVCXO (ORCPT ); Fri, 21 Sep 2018 22:23:14 -0400 Received: by mail-pf1-f201.google.com with SMTP id v9-v6so7032902pff.4 for ; Fri, 21 Sep 2018 13:32:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=mgM40qMWfEoIxPqTpoIns8JsKvkKo0kvvCSL68u4r0g=; b=GI5sh+QMhSSSiGQnhot3sWqG6dYMOjL0Z0xpklf673bZGMSYzTqrrPiS88RoCUZX0z 2I/hr3Sg/WWkqb6NwoZe08M22eHRTqTFXtOke8CMtkT4JmpH06RtB8YdJx3L36AIHFB+ v0HGZYH5p8fHUFWxg6XybQVZzbCkbqk77bJpp4OSBM4OgNxIpEKDPz2wNdCobbtgp25z +bE+gTvW85aIzDXHIwRIs2Hy/29VzciDzg47CA8FQ33Shl5czQzK/nQsKM0VNgOgWJYM ZcsdL42Cy+pRF+qTG7PPeLympJwF16uWm3EMYUGiPtve8DyQSBcPGcPrWUxuhgDtL9IL BVtA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=mgM40qMWfEoIxPqTpoIns8JsKvkKo0kvvCSL68u4r0g=; b=J4DkMjxFqlIayXar5agC15KjNxDjfaKoOf8UHSp7wjfuHSzTlP1b4qb6dY4JwIE3OW ieBDDL5lJsDKNhCLafHVtfoIcfpWPz1siY1sWNdj2igyy0aEpPDzeRVGZSPHopjrFa5d 5jyfanXvEEqAG01Q9h+ZspwSVlW0kga3uzk+0Mi2KFnfkkJ2DPt6bMR27HWf/8uWu1FY vVwaQvB8vYP+XAGRhaBTHaW1B+QPd712ieYkgrPyvwWOaYC3Jj8fLYL8wJiC+eYLr2Us xjLIC57OTLciu5w9WjhKV8hI928u//2NWtgxg7MhcBnN81s9a5GH21XON8T0K+dkseuz f6vg== X-Gm-Message-State: APzg51Bq+s2ALDXJxW12RKlrdvWJxxWZNCs5j6wLkXETtP0JNkSBv1KF AJPWBke7wshGUVqLTxCpu23JzioXsblL7eSzf6DIqXJuVrssWjNhArhSTRNLc57EJr/cS2Mm16v /1qi3PGf7yaV2lastPjJURKZ5Vfi6hqhKelfvlAohbWPCURKvkvk9Qk/1W4A= X-Google-Smtp-Source: ANB0VdaH53V5j7YlzSWGKYvYAnxToZCFAEBsR1pkM/Iki3EEBPvwXSpEc2wFPb8/9qhmi7bp7Loi71gi54kN X-Received: by 2002:a65:4981:: with SMTP id r1-v6mr2155803pgs.0.1537561961573; Fri, 21 Sep 2018 13:32:41 -0700 (PDT) Date: Fri, 21 Sep 2018 13:32:04 -0700 In-Reply-To: Message-Id: <5e87a9cfcd096784965892c1f87c78bed2f24367.1537561420.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.444.g18242da7ef-goog Subject: [PATCH v9 8/8] list-objects-filter: implement filter tree:0 From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Teach list-objects the "tree:0" filter which allows for filtering out all tree and blob objects (unless other objects are explicitly specified by the user). The purpose of this patch is to allow smaller partial clones. The name of this filter - tree:0 - does not explicitly specify that it also filters out all blobs, but this should not cause much confusion because blobs are not at all useful without the trees that refer to them. I also considered only:commits as a name, but this is inaccurate because it suggests that annotated tags are omitted, but actually they are included. The name "tree:0" allows later filtering based on depth, i.e. "tree:1" would filter out all but the root tree and blobs. In order to avoid confusion between 0 and capital O, the documentation was worded in a somewhat round-about way that also hints at this future improvement to the feature. Signed-off-by: Matthew DeVore --- Documentation/rev-list-options.txt | 5 +++ list-objects-filter-options.c | 13 +++++++ list-objects-filter-options.h | 1 + list-objects-filter.c | 49 ++++++++++++++++++++++++++ t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++ t/t5616-partial-clone.sh | 41 +++++++++++++++++++++ t/t6112-rev-list-filters-objects.sh | 13 +++++++ 7 files changed, 150 insertions(+) diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index 7b273635d..5f1672913 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -731,6 +731,11 @@ the requested refs. + The form '--filter=sparse:path=' similarly uses a sparse-checkout specification contained in . ++ +The form '--filter=tree:' omits all blobs and trees whose depth +from the root tree is >= (minimum depth if an object is located +at multiple depths in the commits traversed). Currently, only =0 +is supported, which omits all blobs and trees. --no-filter:: Turn off any previous `--filter=` argument. diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index d259bdb2c..e8da2e858 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -49,6 +49,19 @@ static int gently_parse_list_objects_filter( return 0; } + } else if (skip_prefix(arg, "tree:", &v0)) { + unsigned long depth; + if (!git_parse_ulong(v0, &depth) || depth != 0) { + if (errbuf) { + strbuf_addstr( + errbuf, + _("only 'tree:0' is supported")); + } + return 1; + } + filter_options->choice = LOFC_TREE_NONE; + return 0; + } else if (skip_prefix(arg, "sparse:oid=", &v0)) { struct object_context oc; struct object_id sparse_oid; diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 0000a61f8..af64e5c66 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -10,6 +10,7 @@ enum list_objects_filter_choice { LOFC_DISABLED = 0, LOFC_BLOB_NONE, LOFC_BLOB_LIMIT, + LOFC_TREE_NONE, LOFC_SPARSE_OID, LOFC_SPARSE_PATH, LOFC__COUNT /* must be last */ diff --git a/list-objects-filter.c b/list-objects-filter.c index 5f8b1a002..09b2b05d5 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -79,6 +79,54 @@ static void *filter_blobs_none__init( return d; } +/* + * A filter for list-objects to omit ALL trees and blobs from the traversal. + * Can OPTIONALLY collect a list of the omitted OIDs. + */ +struct filter_trees_none_data { + struct oidset *omits; +}; + +static enum list_objects_filter_result filter_trees_none( + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + void *filter_data_) +{ + struct filter_trees_none_data *filter_data = filter_data_; + + switch (filter_situation) { + default: + BUG("unknown filter_situation: %d", filter_situation); + + case LOFS_BEGIN_TREE: + case LOFS_BLOB: + if (filter_data->omits) + oidset_insert(filter_data->omits, &obj->oid); + return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ + + case LOFS_END_TREE: + assert(obj->type == OBJ_TREE); + return LOFR_ZERO; + + } +} + +static void* filter_trees_none__init( + struct oidset *omitted, + struct list_objects_filter_options *filter_options, + filter_object_fn *filter_fn, + filter_free_fn *filter_free_fn) +{ + struct filter_trees_none_data *d = xcalloc(1, sizeof(*d)); + d->omits = omitted; + + *filter_fn = filter_trees_none; + *filter_free_fn = free; + return d; +} + /* * A filter for list-objects to omit large blobs. * And to OPTIONALLY collect a list of the omitted OIDs. @@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, + filter_trees_none__init, filter_sparse_oid__init, filter_sparse_path__init, }; diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 9839b48c1..510d3537f 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' ' grep -q "bad tree object" bad_tree ' +test_expect_success 'setup for tests of tree:0' ' + mkdir r1/subtree && + echo "This is a file in a subtree" >r1/subtree/file && + git -C r1 add subtree/file && + git -C r1 commit -m subtree +' + +test_expect_success 'verify tree:0 packfile has no blobs or trees' ' + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF && + HEAD + EOF + git -C r1 index-pack ../commitsonly.pack && + git -C r1 verify-pack -v ../commitsonly.pack >objs && + ! grep -E "tree|blob" objs +' + +test_expect_success 'grab tree directly when using tree:0' ' + # We should get the tree specified directly but not its blobs or subtrees. + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF && + HEAD: + EOF + git -C r1 index-pack ../commitsonly.pack && + git -C r1 verify-pack -v ../commitsonly.pack >objs && + awk "/tree|blob/{print \$1}" objs >trees_and_blobs && + git -C r1 rev-parse HEAD: >expected && + test_cmp expected trees_and_blobs +' + # Test blob:limit=[kmg] filter. # We boundary test around the size parameter. The filter is strictly less than # the value, so size 500 and 1000 should have the same results, but 1001 should diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index bbbe7537d..7b6294ca5 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -154,6 +154,47 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack - grep "git index-pack.*--fsck-objects" trace ' +test_expect_success 'use fsck before and after manually fetching a missing subtree' ' + # push new commit so server has a subtree + mkdir src/dir && + echo "in dir" >src/dir/file.txt && + git -C src add dir/file.txt && + git -C src commit -m "file in dir" && + git -C src push -u srv master && + SUBTREE=$(git -C src rev-parse HEAD:dir) && + + rm -rf dst && + git clone --no-checkout --filter=tree:0 "file://$(pwd)/srv.bare" dst && + git -C dst fsck && + + # Make sure we only have commits, and all trees and blobs are missing. + git -C dst rev-list master --missing=allow-any --objects >fetched_objects && + awk -f print_1.awk fetched_objects | + xargs -n1 git -C dst cat-file -t >fetched_types && + + sort -u fetched_types >unique_types.observed && + echo commit >unique_types.expected && + test_cmp unique_types.expected unique_types.observed && + + # Auto-fetch a tree with cat-file. + git -C dst cat-file -p $SUBTREE >tree_contents && + grep file.txt tree_contents && + + # fsck still works after an auto-fetch of a tree. + git -C dst fsck && + + # Auto-fetch all remaining trees and blobs with --missing=error + git -C dst rev-list master --missing=error --objects >fetched_objects && + test_line_count = 70 fetched_objects && + + awk -f print_1.awk fetched_objects | + xargs -n1 git -C dst cat-file -t >fetched_types && + + sort -u fetched_types >unique_types.observed && + printf "blob\ncommit\ntree\n" >unique_types.expected && + test_cmp unique_types.expected unique_types.observed +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index 11186209b..6e5c41a68 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -224,6 +224,19 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre test_must_be_empty rev_list_err ' +# Test tree:0 filter. + +test_expect_success 'verify tree:0 includes trees in "filtered" output' ' + git -C r3 rev-list HEAD --quiet --objects --filter-print-omitted --filter=tree:0 | + awk -f print_1.awk | + sed s/~// | + xargs -n1 git -C r3 cat-file -t | + sort -u >filtered_types && + + printf "blob\ntree\n" >expected && + test_cmp expected filtered_types +' + # Delete some loose objects and use rev-list, but WITHOUT any filtering. # This models previously omitted objects that we did not receive.